2024-03-24T01:08:26.481 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-24T01:08:26.481 DEBUG:teuthology.run:Teuthology command: teuthology --name mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi --owner mchangir --archive /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616453 --verbose --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/journal-repair} -- /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616453/orig.config.yaml 2024-03-24T01:08:26.526 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T01:08:26.614 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616453 branch: wip-mchangir-testing-main-20240318.032620 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/journal-repair} email: mchangir@redhat.com first_in_suite: false job_id: '7616453' kernel: flavor: default kdb: true sha1: 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-mchangir-testing-main-20240318.032620 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - bad backtrace on directory inode - error reading table object - Metadata damage detected - slow requests are blocked - Behind on trimming - error reading sessionmap sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 subvolume_version: 1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-mchangir-testing-main-20240318.032620 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 owner: mchangir priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 3550 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 sleep_before_teardown: 0 subset: 613/1024 suite: fs suite_branch: wip-mchangir-testing-main-20240318.032620 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi090.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDPxTpKXePPQNHpR4U7WKV34yOJfd7d/mNOkZ153u5OYNUU/BVD7SyJJJSic+YiPf10ceGPv0TJhVYnK5d2enao= smithi120.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIsD5/kDR1BWRDWfmlrWw9avR8IC90oP1y5hOVOfAiJXGeLToZTQvDj9Ol/SdQ0iTE0wDvajJKkjTfhdzedsbTY= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_journal_repair teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-22_09:49:57 tube: smithi user: mchangir verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-24T01:08:26.614 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa; will attempt to use it 2024-03-24T01:08:26.615 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks 2024-03-24T01:08:26.615 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-24T01:08:26.616 INFO:teuthology.task.internal:Checking packages... 2024-03-24T01:08:26.641 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3be9a5c9bc793e11e2800a8c0c696e8b46742033' 2024-03-24T01:08:26.641 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T01:08:26.641 INFO:teuthology.packaging:ref: None 2024-03-24T01:08:26.641 INFO:teuthology.packaging:tag: None 2024-03-24T01:08:26.641 INFO:teuthology.packaging:branch: wip-mchangir-testing-main-20240318.032620 2024-03-24T01:08:26.641 INFO:teuthology.packaging:sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T01:08:26.641 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-mchangir-testing-main-20240318.032620 2024-03-24T01:08:26.866 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2145.g3be9a5c9 2024-03-24T01:08:26.868 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-24T01:08:26.875 INFO:teuthology.task.internal:no buildpackages task found 2024-03-24T01:08:26.875 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-24T01:08:26.898 INFO:teuthology.task.internal:Saving configuration 2024-03-24T01:08:26.913 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-24T01:08:26.957 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T01:08:26.983 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616453', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 01:02:38.676443', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDPxTpKXePPQNHpR4U7WKV34yOJfd7d/mNOkZ153u5OYNUU/BVD7SyJJJSic+YiPf10ceGPv0TJhVYnK5d2enao='} 2024-03-24T01:08:27.004 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi120.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616453', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 01:02:38.675107', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIsD5/kDR1BWRDWfmlrWw9avR8IC90oP1y5hOVOfAiJXGeLToZTQvDj9Ol/SdQ0iTE0wDvajJKkjTfhdzedsbTY='} 2024-03-24T01:08:27.004 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-24T01:08:27.011 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-03-24T01:08:27.011 INFO:teuthology.task.internal:roles: ubuntu@smithi120.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-03-24T01:08:27.011 INFO:teuthology.run_tasks:Running task console_log... 2024-03-24T01:08:27.108 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb35c7e8f70>, signals=[15]) 2024-03-24T01:08:27.108 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-24T01:08:27.115 INFO:teuthology.task.internal:Opening connections... 2024-03-24T01:08:27.116 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi090.front.sepia.ceph.com 2024-03-24T01:08:27.118 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T01:08:27.207 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi120.front.sepia.ceph.com 2024-03-24T01:08:27.209 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T01:08:27.297 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-24T01:08:27.304 DEBUG:teuthology.orchestra.run.smithi090:> uname -m 2024-03-24T01:08:27.321 INFO:teuthology.orchestra.run.smithi090.stdout:x86_64 2024-03-24T01:08:27.321 DEBUG:teuthology.orchestra.run.smithi090:> cat /etc/os-release 2024-03-24T01:08:27.378 INFO:teuthology.orchestra.run.smithi090.stdout:NAME="CentOS Stream" 2024-03-24T01:08:27.378 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION="9" 2024-03-24T01:08:27.378 INFO:teuthology.orchestra.run.smithi090.stdout:ID="centos" 2024-03-24T01:08:27.379 INFO:teuthology.orchestra.run.smithi090.stdout:ID_LIKE="rhel fedora" 2024-03-24T01:08:27.379 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_ID="9" 2024-03-24T01:08:27.379 INFO:teuthology.orchestra.run.smithi090.stdout:PLATFORM_ID="platform:el9" 2024-03-24T01:08:27.379 INFO:teuthology.orchestra.run.smithi090.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T01:08:27.379 INFO:teuthology.orchestra.run.smithi090.stdout:ANSI_COLOR="0;31" 2024-03-24T01:08:27.379 INFO:teuthology.orchestra.run.smithi090.stdout:LOGO="fedora-logo-icon" 2024-03-24T01:08:27.379 INFO:teuthology.orchestra.run.smithi090.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T01:08:27.379 INFO:teuthology.orchestra.run.smithi090.stdout:HOME_URL="https://centos.org/" 2024-03-24T01:08:27.379 INFO:teuthology.orchestra.run.smithi090.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T01:08:27.379 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T01:08:27.379 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T01:08:27.380 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server 2024-03-24T01:08:27.403 DEBUG:teuthology.orchestra.run.smithi120:> uname -m 2024-03-24T01:08:27.421 INFO:teuthology.orchestra.run.smithi120.stdout:x86_64 2024-03-24T01:08:27.421 DEBUG:teuthology.orchestra.run.smithi120:> cat /etc/os-release 2024-03-24T01:08:27.478 INFO:teuthology.orchestra.run.smithi120.stdout:NAME="CentOS Stream" 2024-03-24T01:08:27.478 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION="9" 2024-03-24T01:08:27.478 INFO:teuthology.orchestra.run.smithi120.stdout:ID="centos" 2024-03-24T01:08:27.478 INFO:teuthology.orchestra.run.smithi120.stdout:ID_LIKE="rhel fedora" 2024-03-24T01:08:27.478 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION_ID="9" 2024-03-24T01:08:27.479 INFO:teuthology.orchestra.run.smithi120.stdout:PLATFORM_ID="platform:el9" 2024-03-24T01:08:27.479 INFO:teuthology.orchestra.run.smithi120.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T01:08:27.479 INFO:teuthology.orchestra.run.smithi120.stdout:ANSI_COLOR="0;31" 2024-03-24T01:08:27.479 INFO:teuthology.orchestra.run.smithi120.stdout:LOGO="fedora-logo-icon" 2024-03-24T01:08:27.479 INFO:teuthology.orchestra.run.smithi120.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T01:08:27.479 INFO:teuthology.orchestra.run.smithi120.stdout:HOME_URL="https://centos.org/" 2024-03-24T01:08:27.479 INFO:teuthology.orchestra.run.smithi120.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T01:08:27.479 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T01:08:27.479 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T01:08:27.479 INFO:teuthology.lock.ops:Updating smithi120.front.sepia.ceph.com on lock server 2024-03-24T01:08:27.503 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-24T01:08:27.512 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-24T01:08:27.556 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-24T01:08:27.556 DEBUG:teuthology.orchestra.run.smithi090:> test '!' -e /home/ubuntu/cephtest 2024-03-24T01:08:27.559 DEBUG:teuthology.orchestra.run.smithi120:> test '!' -e /home/ubuntu/cephtest 2024-03-24T01:08:27.575 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-24T01:08:27.632 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-24T01:08:27.633 DEBUG:teuthology.orchestra.run.smithi090:> test -z $(ls -A /var/lib/ceph) 2024-03-24T01:08:27.635 DEBUG:teuthology.orchestra.run.smithi120:> test -z $(ls -A /var/lib/ceph) 2024-03-24T01:08:27.696 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-24T01:08:27.751 INFO:teuthology.run_tasks:Running task kernel... 2024-03-24T01:08:27.766 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': True, 'sha1': '5ec3e6e88b2e2783e09498588bb1f433ae9c8986'} 2024-03-24T01:08:27.766 INFO:teuthology.task.kernel:config {'mon.a': {'flavor': 'default', 'kdb': True, 'sha1': '5ec3e6e88b2e2783e09498588bb1f433ae9c8986'}, 'mon.b': {'flavor': 'default', 'kdb': True, 'sha1': '5ec3e6e88b2e2783e09498588bb1f433ae9c8986'}}, timeout 300 2024-03-24T01:08:27.766 DEBUG:teuthology.orchestra.run.smithi090:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T01:08:27.767 DEBUG:teuthology.orchestra.run.smithi120:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T01:08:27.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T01:08:27.783 DEBUG:teuthology.task.kernel:sha1 for mon.a is 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-24T01:08:27.784 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5ec3e6e88b2e2783e09498588bb1f433ae9c8986"... 2024-03-24T01:08:27.784 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2024-03-24T01:08:27.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T01:08:27.785 DEBUG:teuthology.task.kernel:sha1 for mon.b is 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-24T01:08:27.785 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5ec3e6e88b2e2783e09498588bb1f433ae9c8986"... 2024-03-24T01:08:27.785 DEBUG:teuthology.orchestra.run.smithi120:> uname -r 2024-03-24T01:08:27.843 INFO:teuthology.orchestra.run.smithi120.stdout:5.14.0-361.el9.x86_64 2024-03-24T01:08:27.844 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-24T01:08:27.845 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-24T01:08:27.845 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-24T01:08:27.850 INFO:teuthology.orchestra.run.smithi090.stdout:5.14.0-361.el9.x86_64 2024-03-24T01:08:27.850 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-24T01:08:27.850 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-24T01:08:27.850 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-24T01:08:27.992 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum upgrade -y linux-firmware 2024-03-24T01:08:27.998 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum upgrade -y linux-firmware 2024-03-24T01:08:30.220 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-24T01:08:30.453 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-24T01:08:31.845 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2024-03-24T01:08:31.929 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-9 - BaseOS 3.2 MB/s | 2.0 MB 00:00 2024-03-24T01:08:33.603 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - BaseOS 5.7 MB/s | 8.0 MB 00:01 2024-03-24T01:08:33.628 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - BaseOS 5.9 MB/s | 8.0 MB 00:01 2024-03-24T01:08:35.446 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-03-24T01:08:40.306 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - CRB 4.4 MB/s | 6.1 MB 00:01 2024-03-24T01:08:41.598 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - Extras packages 98 kB/s | 16 kB 00:00 2024-03-24T01:08:55.418 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - AppStream 930 kB/s | 19 MB 00:20 2024-03-24T01:08:59.394 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.1 MB 00:00 2024-03-24T01:09:01.020 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 16 kB 00:00 2024-03-24T01:09:02.217 INFO:teuthology.orchestra.run.smithi120.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-03-24T01:09:07.114 INFO:teuthology.orchestra.run.smithi120.stdout:lab-extras 18 kB/s | 1.7 kB 00:00 2024-03-24T01:09:09.288 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-24T01:09:09.299 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-24T01:09:09.299 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2024-03-24T01:09:09.299 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-24T01:09:09.299 INFO:teuthology.orchestra.run.smithi120.stdout:Upgrading: 2024-03-24T01:09:09.299 INFO:teuthology.orchestra.run.smithi120.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-24T01:09:09.300 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:09:09.300 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-24T01:09:09.300 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2024-03-24T01:09:09.300 INFO:teuthology.orchestra.run.smithi120.stdout:Upgrade 1 Package 2024-03-24T01:09:09.300 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:09:09.300 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 375 M 2024-03-24T01:09:09.300 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-24T01:09:16.523 INFO:teuthology.orchestra.run.smithi120.stdout:linux-firmware-20240219-143.el9.noarch.rpm 52 MB/s | 375 MB 00:07 2024-03-24T01:09:16.524 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-24T01:09:16.524 INFO:teuthology.orchestra.run.smithi120.stdout:Total 52 MB/s | 375 MB 00:07 2024-03-24T01:09:16.524 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-24T01:09:16.623 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-24T01:09:16.623 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-24T01:09:19.044 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-24T01:09:19.044 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-24T01:09:20.168 INFO:teuthology.orchestra.run.smithi090.stdout:Extra Packages for Enterprise Linux 551 kB/s | 21 MB 00:38 2024-03-24T01:09:25.069 INFO:teuthology.orchestra.run.smithi090.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-24T01:09:25.439 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-24T01:09:25.674 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-24T01:09:25.766 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-24T01:09:27.180 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-24T01:09:27.180 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-24T01:09:27.314 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-24T01:09:27.320 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-24T01:09:27.320 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-03-24T01:09:27.320 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-24T01:09:27.320 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrading: 2024-03-24T01:09:27.320 INFO:teuthology.orchestra.run.smithi090.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-24T01:09:27.320 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:09:27.320 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-24T01:09:27.320 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-24T01:09:27.320 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrade 1 Package 2024-03-24T01:09:27.321 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:09:27.321 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 375 M 2024-03-24T01:09:27.321 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-24T01:09:27.619 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-24T01:09:27.619 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:09:27.619 INFO:teuthology.orchestra.run.smithi120.stdout:Upgraded: 2024-03-24T01:09:27.619 INFO:teuthology.orchestra.run.smithi120.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-24T01:09:27.619 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:09:27.619 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-24T01:09:27.926 INFO:teuthology.task.kernel:Downloading kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on mon.b... 2024-03-24T01:09:27.927 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-24T01:09:28.070 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/ 2024-03-24T01:09:28.070 DEBUG:teuthology.orchestra.run.smithi120:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g5ec3e6e88b2e-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-24T01:09:28.073 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi120.front.sepia.ceph.com to complete... 2024-03-24T01:09:35.421 INFO:teuthology.orchestra.run.smithi090.stdout:linux-firmware-20240219-143.el9.noarch.rpm 46 MB/s | 375 MB 00:08 2024-03-24T01:09:35.422 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-24T01:09:35.422 INFO:teuthology.orchestra.run.smithi090.stdout:Total 46 MB/s | 375 MB 00:08 2024-03-24T01:09:35.422 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-24T01:09:35.541 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-24T01:09:35.541 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-24T01:09:37.911 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-24T01:09:37.911 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-24T01:09:44.439 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24 01:09:44 URL:https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/kernel-6.8.0_g5ec3e6e88b2e-1.x86_64.rpm [336309687/336309687] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-24T01:09:44.439 INFO:teuthology.orchestra.run.smithi120.stderr:FINISHED --2024-03-24 01:09:44-- 2024-03-24T01:09:44.439 INFO:teuthology.orchestra.run.smithi120.stderr:Total wall clock time: 16s 2024-03-24T01:09:44.439 INFO:teuthology.orchestra.run.smithi120.stderr:Downloaded: 1 files, 321M in 16s (20.1 MB/s) 2024-03-24T01:09:44.443 INFO:teuthology.task.kernel:Installing kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on mon.b... 2024-03-24T01:09:44.443 DEBUG:teuthology.orchestra.run.smithi120:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-24T01:09:45.126 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-24T01:09:45.336 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-24T01:09:45.426 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-24T01:09:46.366 INFO:teuthology.orchestra.run.smithi120.stdout:Verifying... ######################################## 2024-03-24T01:09:46.424 INFO:teuthology.orchestra.run.smithi120.stdout:Preparing... ######################################## 2024-03-24T01:09:46.504 INFO:teuthology.orchestra.run.smithi120.stdout:Updating / installing... 2024-03-24T01:09:46.961 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-24T01:09:46.961 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-24T01:09:47.373 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-24T01:09:47.373 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:09:47.373 INFO:teuthology.orchestra.run.smithi090.stdout:Upgraded: 2024-03-24T01:09:47.373 INFO:teuthology.orchestra.run.smithi090.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-24T01:09:47.373 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:09:47.373 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-24T01:09:47.731 INFO:teuthology.task.kernel:Downloading kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on mon.a... 2024-03-24T01:09:47.731 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-24T01:09:47.870 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/ 2024-03-24T01:09:47.871 DEBUG:teuthology.orchestra.run.smithi090:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g5ec3e6e88b2e-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-24T01:09:47.873 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi090.front.sepia.ceph.com to complete... 2024-03-24T01:09:54.028 INFO:teuthology.orchestra.run.smithi120.stdout:kernel-6.8.0_g5ec3e6e88b2e-1 ######################################## 2024-03-24T01:09:59.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24 01:09:59 URL:https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/kernel-6.8.0_g5ec3e6e88b2e-1.x86_64.rpm [336309687/336309687] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-24T01:09:59.510 INFO:teuthology.orchestra.run.smithi090.stderr:FINISHED --2024-03-24 01:09:59-- 2024-03-24T01:09:59.510 INFO:teuthology.orchestra.run.smithi090.stderr:Total wall clock time: 12s 2024-03-24T01:09:59.510 INFO:teuthology.orchestra.run.smithi090.stderr:Downloaded: 1 files, 321M in 11s (28.3 MB/s) 2024-03-24T01:09:59.513 INFO:teuthology.task.kernel:Installing kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on mon.a... 2024-03-24T01:09:59.513 DEBUG:teuthology.orchestra.run.smithi090:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-24T01:10:01.445 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying... ######################################## 2024-03-24T01:10:01.519 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing... ######################################## 2024-03-24T01:10:01.598 INFO:teuthology.orchestra.run.smithi090.stdout:Updating / installing... 2024-03-24T01:10:09.104 INFO:teuthology.orchestra.run.smithi090.stdout:kernel-6.8.0_g5ec3e6e88b2e-1 ######################################## 2024-03-24T01:10:10.906 INFO:teuthology.orchestra.run.smithi120.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T01:10:29.801 INFO:teuthology.orchestra.run.smithi090.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T01:10:33.360 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi120.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-24T01:10:33.361 DEBUG:teuthology.orchestra.run.smithi120:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-24T01:10:33.426 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e 2024-03-24T01:10:33.426 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/System.map 2024-03-24T01:10:33.426 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/config 2024-03-24T01:10:33.426 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel 2024-03-24T01:10:33.426 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch 2024-03-24T01:10:33.426 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86 2024-03-24T01:10:33.426 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto 2024-03-24T01:10:33.427 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-24T01:10:33.427 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-24T01:10:33.427 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-24T01:10:33.427 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-24T01:10:33.427 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-24T01:10:33.427 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-24T01:10:33.427 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-24T01:10:33.427 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-24T01:10:33.427 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-24T01:10:33.427 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-24T01:10:33.427 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-24T01:10:33.427 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-24T01:10:33.427 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-24T01:10:33.428 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-24T01:10:33.428 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-24T01:10:33.428 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-24T01:10:33.428 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-24T01:10:33.428 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-24T01:10:33.428 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-24T01:10:33.428 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-24T01:10:33.428 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel 2024-03-24T01:10:33.428 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu 2024-03-24T01:10:33.428 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce 2024-03-24T01:10:33.428 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-24T01:10:33.428 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm 2024-03-24T01:10:33.428 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-24T01:10:33.428 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-24T01:10:33.429 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm.ko 2024-03-24T01:10:33.429 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto 2024-03-24T01:10:33.429 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ansi_cprng.ko 2024-03-24T01:10:33.429 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/anubis.ko 2024-03-24T01:10:33.429 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/arc4.ko 2024-03-24T01:10:33.429 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx 2024-03-24T01:10:33.429 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_memcpy.ko 2024-03-24T01:10:33.429 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_pq.ko 2024-03-24T01:10:33.429 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-24T01:10:33.429 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_tx.ko 2024-03-24T01:10:33.429 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_xor.ko 2024-03-24T01:10:33.429 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/raid6test.ko 2024-03-24T01:10:33.429 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authenc.ko 2024-03-24T01:10:33.429 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authencesn.ko 2024-03-24T01:10:33.430 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blake2b_generic.ko 2024-03-24T01:10:33.430 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_common.ko 2024-03-24T01:10:33.430 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_generic.ko 2024-03-24T01:10:33.430 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/camellia_generic.ko 2024-03-24T01:10:33.430 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast5_generic.ko 2024-03-24T01:10:33.430 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast6_generic.ko 2024-03-24T01:10:33.430 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast_common.ko 2024-03-24T01:10:33.430 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ccm.ko 2024-03-24T01:10:33.430 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cmac.ko 2024-03-24T01:10:33.430 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crc32_generic.ko 2024-03-24T01:10:33.430 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_engine.ko 2024-03-24T01:10:33.430 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_user.ko 2024-03-24T01:10:33.430 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cts.ko 2024-03-24T01:10:33.431 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/des_generic.ko 2024-03-24T01:10:33.431 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/dh_generic.ko 2024-03-24T01:10:33.431 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/echainiv.ko 2024-03-24T01:10:33.431 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/essiv.ko 2024-03-24T01:10:33.431 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/fcrypt.ko 2024-03-24T01:10:33.431 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/khazad.ko 2024-03-24T01:10:33.431 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4.ko 2024-03-24T01:10:33.431 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4hc.ko 2024-03-24T01:10:33.431 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/md4.ko 2024-03-24T01:10:33.431 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/michael_mic.ko 2024-03-24T01:10:33.431 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcbc.ko 2024-03-24T01:10:33.431 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcrypt.ko 2024-03-24T01:10:33.431 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/rmd160.ko 2024-03-24T01:10:33.432 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/seed.ko 2024-03-24T01:10:33.432 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/serpent_generic.ko 2024-03-24T01:10:33.432 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tcrypt.ko 2024-03-24T01:10:33.432 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tea.ko 2024-03-24T01:10:33.432 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_common.ko 2024-03-24T01:10:33.432 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_generic.ko 2024-03-24T01:10:33.432 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/vmac.ko 2024-03-24T01:10:33.432 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/wp512.ko 2024-03-24T01:10:33.432 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xcbc.ko 2024-03-24T01:10:33.432 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xor.ko 2024-03-24T01:10:33.432 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xxhash_generic.ko 2024-03-24T01:10:33.432 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers 2024-03-24T01:10:33.432 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi 2024-03-24T01:10:33.432 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-24T01:10:33.433 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_pad.ko 2024-03-24T01:10:33.433 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/custom_method.ko 2024-03-24T01:10:33.433 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/ec_sys.ko 2024-03-24T01:10:33.433 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/platform_profile.ko 2024-03-24T01:10:33.433 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbs.ko 2024-03-24T01:10:33.433 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbshc.ko 2024-03-24T01:10:33.433 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/video.ko 2024-03-24T01:10:33.433 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata 2024-03-24T01:10:33.433 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/acard-ahci.ko 2024-03-24T01:10:33.433 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ahci_platform.ko 2024-03-24T01:10:33.433 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ata_generic.ko 2024-03-24T01:10:33.433 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/libahci_platform.ko 2024-03-24T01:10:33.433 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_acpi.ko 2024-03-24T01:10:33.433 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ali.ko 2024-03-24T01:10:33.434 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_amd.ko 2024-03-24T01:10:33.434 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_artop.ko 2024-03-24T01:10:33.434 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atiixp.ko 2024-03-24T01:10:33.434 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atp867x.ko 2024-03-24T01:10:33.434 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd640.ko 2024-03-24T01:10:33.434 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd64x.ko 2024-03-24T01:10:33.434 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cypress.ko 2024-03-24T01:10:33.434 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_efar.ko 2024-03-24T01:10:33.434 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt366.ko 2024-03-24T01:10:33.434 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt37x.ko 2024-03-24T01:10:33.434 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-24T01:10:33.434 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-24T01:10:33.435 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it8213.ko 2024-03-24T01:10:33.435 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it821x.ko 2024-03-24T01:10:33.435 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_jmicron.ko 2024-03-24T01:10:33.435 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_marvell.ko 2024-03-24T01:10:33.435 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_mpiix.ko 2024-03-24T01:10:33.435 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_netcell.ko 2024-03-24T01:10:33.435 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ninja32.ko 2024-03-24T01:10:33.435 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87410.ko 2024-03-24T01:10:33.435 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87415.ko 2024-03-24T01:10:33.435 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_oldpiix.ko 2024-03-24T01:10:33.435 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_opti.ko 2024-03-24T01:10:33.435 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_optidma.ko 2024-03-24T01:10:33.435 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pcmcia.ko 2024-03-24T01:10:33.435 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-24T01:10:33.436 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-24T01:10:33.436 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_piccolo.ko 2024-03-24T01:10:33.436 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_rdc.ko 2024-03-24T01:10:33.436 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sch.ko 2024-03-24T01:10:33.436 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_serverworks.ko 2024-03-24T01:10:33.436 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sil680.ko 2024-03-24T01:10:33.436 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sis.ko 2024-03-24T01:10:33.436 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sl82c105.ko 2024-03-24T01:10:33.436 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_triflex.ko 2024-03-24T01:10:33.436 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_via.ko 2024-03-24T01:10:33.436 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pdc_adma.ko 2024-03-24T01:10:33.436 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_inic162x.ko 2024-03-24T01:10:33.437 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_mv.ko 2024-03-24T01:10:33.437 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_nv.ko 2024-03-24T01:10:33.437 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_promise.ko 2024-03-24T01:10:33.437 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_qstor.ko 2024-03-24T01:10:33.437 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil.ko 2024-03-24T01:10:33.437 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil24.ko 2024-03-24T01:10:33.437 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sis.ko 2024-03-24T01:10:33.437 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_svw.ko 2024-03-24T01:10:33.437 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sx4.ko 2024-03-24T01:10:33.437 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_uli.ko 2024-03-24T01:10:33.437 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_via.ko 2024-03-24T01:10:33.437 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_vsc.ko 2024-03-24T01:10:33.437 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay 2024-03-24T01:10:33.437 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay/ks0108.ko 2024-03-24T01:10:33.438 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base 2024-03-24T01:10:33.438 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap 2024-03-24T01:10:33.438 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-24T01:10:33.438 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-24T01:10:33.438 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma 2024-03-24T01:10:33.438 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma/bcma.ko 2024-03-24T01:10:33.438 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block 2024-03-24T01:10:33.438 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe 2024-03-24T01:10:33.438 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe/aoe.ko 2024-03-24T01:10:33.438 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/brd.ko 2024-03-24T01:10:33.438 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd 2024-03-24T01:10:33.438 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd/drbd.ko 2024-03-24T01:10:33.438 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/floppy.ko 2024-03-24T01:10:33.438 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/loop.ko 2024-03-24T01:10:33.439 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx 2024-03-24T01:10:33.439 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-24T01:10:33.439 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/nbd.ko 2024-03-24T01:10:33.439 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk 2024-03-24T01:10:33.439 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk/null_blk.ko 2024-03-24T01:10:33.439 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/pktcdvd.ko 2024-03-24T01:10:33.439 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/rbd.ko 2024-03-24T01:10:33.439 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/virtio_blk.ko 2024-03-24T01:10:33.439 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback 2024-03-24T01:10:33.439 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-24T01:10:33.439 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkfront.ko 2024-03-24T01:10:33.439 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram 2024-03-24T01:10:33.439 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram/zram.ko 2024-03-24T01:10:33.440 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char 2024-03-24T01:10:33.440 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hangcheck-timer.ko 2024-03-24T01:10:33.440 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random 2024-03-24T01:10:33.440 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-24T01:10:33.440 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-24T01:10:33.440 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-24T01:10:33.440 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/via-rng.ko 2024-03-24T01:10:33.440 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-24T01:10:33.440 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi 2024-03-24T01:10:33.440 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-24T01:10:33.440 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-24T01:10:33.440 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-24T01:10:33.440 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-24T01:10:33.440 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-24T01:10:33.441 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-24T01:10:33.441 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/lp.ko 2024-03-24T01:10:33.441 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave 2024-03-24T01:10:33.441 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave/mwave.ko 2024-03-24T01:10:33.441 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ppdev.ko 2024-03-24T01:10:33.441 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tlclk.ko 2024-03-24T01:10:33.441 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm 2024-03-24T01:10:33.441 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm.ko 2024-03-24T01:10:33.441 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-24T01:10:33.441 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-24T01:10:33.441 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-24T01:10:33.441 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-24T01:10:33.441 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-24T01:10:33.441 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/uv_mmtimer.ko 2024-03-24T01:10:33.442 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/virtio_console.ko 2024-03-24T01:10:33.442 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq 2024-03-24T01:10:33.442 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-24T01:10:33.442 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-24T01:10:33.442 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-24T01:10:33.442 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-24T01:10:33.442 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-24T01:10:33.442 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-24T01:10:33.442 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto 2024-03-24T01:10:33.442 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp 2024-03-24T01:10:33.442 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-24T01:10:33.442 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp.ko 2024-03-24T01:10:33.442 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel 2024-03-24T01:10:33.443 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat 2024-03-24T01:10:33.443 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common 2024-03-24T01:10:33.443 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-24T01:10:33.443 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-24T01:10:33.443 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-24T01:10:33.443 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-aes.ko 2024-03-24T01:10:33.443 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-sha.ko 2024-03-24T01:10:33.443 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio 2024-03-24T01:10:33.443 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-24T01:10:33.443 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca 2024-03-24T01:10:33.443 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca/dca.ko 2024-03-24T01:10:33.443 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/devfreq 2024-03-24T01:10:33.443 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-24T01:10:33.443 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma 2024-03-24T01:10:33.444 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw 2024-03-24T01:10:33.444 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-24T01:10:33.444 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat 2024-03-24T01:10:33.444 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-24T01:10:33.444 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac 2024-03-24T01:10:33.444 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/amd64_edac.ko 2024-03-24T01:10:33.444 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/e752x_edac.ko 2024-03-24T01:10:33.444 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/edac_mce_amd.ko 2024-03-24T01:10:33.444 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3000_edac.ko 2024-03-24T01:10:33.444 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3200_edac.ko 2024-03-24T01:10:33.444 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5100_edac.ko 2024-03-24T01:10:33.444 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5400_edac.ko 2024-03-24T01:10:33.444 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7300_edac.ko 2024-03-24T01:10:33.444 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7core_edac.ko 2024-03-24T01:10:33.445 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i82975x_edac.ko 2024-03-24T01:10:33.445 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/ie31200_edac.ko 2024-03-24T01:10:33.445 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/sb_edac.ko 2024-03-24T01:10:33.445 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/x38_edac.ko 2024-03-24T01:10:33.445 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware 2024-03-24T01:10:33.445 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/edd.ko 2024-03-24T01:10:33.445 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-24T01:10:33.445 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio 2024-03-24T01:10:33.445 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-ich.ko 2024-03-24T01:10:33.445 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-24T01:10:33.445 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid 2024-03-24T01:10:33.445 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-appleir.ko 2024-03-24T01:10:33.445 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-aureal.ko 2024-03-24T01:10:33.446 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-axff.ko 2024-03-24T01:10:33.446 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-dr.ko 2024-03-24T01:10:33.446 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elecom.ko 2024-03-24T01:10:33.446 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elo.ko 2024-03-24T01:10:33.446 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-emsff.ko 2024-03-24T01:10:33.446 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gaff.ko 2024-03-24T01:10:33.446 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gt683r.ko 2024-03-24T01:10:33.446 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gyration.ko 2024-03-24T01:10:33.446 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-24T01:10:33.446 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-24T01:10:33.446 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtekff.ko 2024-03-24T01:10:33.446 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-hyperv.ko 2024-03-24T01:10:33.446 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-icade.ko 2024-03-24T01:10:33.446 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-keytouch.ko 2024-03-24T01:10:33.446 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-kye.ko 2024-03-24T01:10:33.447 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lcpower.ko 2024-03-24T01:10:33.447 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-led.ko 2024-03-24T01:10:33.447 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lenovo.ko 2024-03-24T01:10:33.447 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-24T01:10:33.447 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-24T01:10:33.447 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-multitouch.ko 2024-03-24T01:10:33.447 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-ortek.ko 2024-03-24T01:10:33.447 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-penmount.ko 2024-03-24T01:10:33.447 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-petalynx.ko 2024-03-24T01:10:33.447 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-picolcd.ko 2024-03-24T01:10:33.447 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-pl.ko 2024-03-24T01:10:33.447 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-plantronics.ko 2024-03-24T01:10:33.448 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-primax.ko 2024-03-24T01:10:33.448 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-rmi.ko 2024-03-24T01:10:33.448 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-24T01:10:33.448 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-common.ko 2024-03-24T01:10:33.448 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-24T01:10:33.448 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-24T01:10:33.448 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-24T01:10:33.448 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-24T01:10:33.448 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-24T01:10:33.448 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-24T01:10:33.448 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-24T01:10:33.448 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-24T01:10:33.448 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-24T01:10:33.448 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat.ko 2024-03-24T01:10:33.449 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-saitek.ko 2024-03-24T01:10:33.449 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-samsung.ko 2024-03-24T01:10:33.449 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-24T01:10:33.449 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sjoy.ko 2024-03-24T01:10:33.449 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sony.ko 2024-03-24T01:10:33.449 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-speedlink.ko 2024-03-24T01:10:33.449 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-steelseries.ko 2024-03-24T01:10:33.449 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sunplus.ko 2024-03-24T01:10:33.449 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-24T01:10:33.449 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tivo.ko 2024-03-24T01:10:33.449 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tmff.ko 2024-03-24T01:10:33.449 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-topseed.ko 2024-03-24T01:10:33.449 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-twinhan.ko 2024-03-24T01:10:33.450 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-uclogic.ko 2024-03-24T01:10:33.450 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-waltop.ko 2024-03-24T01:10:33.450 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-wiimote.ko 2024-03-24T01:10:33.450 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-xinmo.ko 2024-03-24T01:10:33.450 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zpff.ko 2024-03-24T01:10:33.450 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zydacron.ko 2024-03-24T01:10:33.450 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/uhid.ko 2024-03-24T01:10:33.450 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/wacom.ko 2024-03-24T01:10:33.450 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv 2024-03-24T01:10:33.450 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_balloon.ko 2024-03-24T01:10:33.450 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_utils.ko 2024-03-24T01:10:33.450 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_vmbus.ko 2024-03-24T01:10:33.450 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon 2024-03-24T01:10:33.450 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru.ko 2024-03-24T01:10:33.451 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru3.ko 2024-03-24T01:10:33.451 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-24T01:10:33.451 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7414.ko 2024-03-24T01:10:33.451 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7418.ko 2024-03-24T01:10:33.451 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adc128d818.ko 2024-03-24T01:10:33.451 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1025.ko 2024-03-24T01:10:33.451 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1026.ko 2024-03-24T01:10:33.451 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1029.ko 2024-03-24T01:10:33.451 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1031.ko 2024-03-24T01:10:33.451 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm9240.ko 2024-03-24T01:10:33.451 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ads7828.ko 2024-03-24T01:10:33.451 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7410.ko 2024-03-24T01:10:33.451 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7411.ko 2024-03-24T01:10:33.452 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7462.ko 2024-03-24T01:10:33.452 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7470.ko 2024-03-24T01:10:33.452 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7475.ko 2024-03-24T01:10:33.452 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7x10.ko 2024-03-24T01:10:33.452 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/amc6821.ko 2024-03-24T01:10:33.452 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/applesmc.ko 2024-03-24T01:10:33.452 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asb100.ko 2024-03-24T01:10:33.452 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asc7621.ko 2024-03-24T01:10:33.452 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-24T01:10:33.452 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/atxp1.ko 2024-03-24T01:10:33.452 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/coretemp.ko 2024-03-24T01:10:33.452 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-24T01:10:33.452 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dme1737.ko 2024-03-24T01:10:33.452 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds1621.ko 2024-03-24T01:10:33.452 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds620.ko 2024-03-24T01:10:33.453 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc1403.ko 2024-03-24T01:10:33.453 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc6w201.ko 2024-03-24T01:10:33.453 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71805f.ko 2024-03-24T01:10:33.453 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71882fg.ko 2024-03-24T01:10:33.453 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f75375s.ko 2024-03-24T01:10:33.453 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fam15h_power.ko 2024-03-24T01:10:33.453 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fschmd.ko 2024-03-24T01:10:33.453 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g760a.ko 2024-03-24T01:10:33.453 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g762.ko 2024-03-24T01:10:33.453 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl518sm.ko 2024-03-24T01:10:33.453 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl520sm.ko 2024-03-24T01:10:33.453 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-24T01:10:33.453 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5500_temp.ko 2024-03-24T01:10:33.454 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5k_amb.ko 2024-03-24T01:10:33.454 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmaem.ko 2024-03-24T01:10:33.454 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmpex.ko 2024-03-24T01:10:33.454 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina209.ko 2024-03-24T01:10:33.454 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina2xx.ko 2024-03-24T01:10:33.454 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/it87.ko 2024-03-24T01:10:33.454 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k10temp.ko 2024-03-24T01:10:33.454 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k8temp.ko 2024-03-24T01:10:33.454 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lineage-pem.ko 2024-03-24T01:10:33.454 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm63.ko 2024-03-24T01:10:33.454 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm73.ko 2024-03-24T01:10:33.454 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm75.ko 2024-03-24T01:10:33.454 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm77.ko 2024-03-24T01:10:33.455 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm78.ko 2024-03-24T01:10:33.455 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm80.ko 2024-03-24T01:10:33.455 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm83.ko 2024-03-24T01:10:33.455 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm85.ko 2024-03-24T01:10:33.455 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm87.ko 2024-03-24T01:10:33.455 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm90.ko 2024-03-24T01:10:33.455 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm92.ko 2024-03-24T01:10:33.455 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm93.ko 2024-03-24T01:10:33.455 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95234.ko 2024-03-24T01:10:33.455 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95241.ko 2024-03-24T01:10:33.455 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95245.ko 2024-03-24T01:10:33.455 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc2945.ko 2024-03-24T01:10:33.455 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4151.ko 2024-03-24T01:10:33.455 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4215.ko 2024-03-24T01:10:33.456 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4222.ko 2024-03-24T01:10:33.456 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4245.ko 2024-03-24T01:10:33.456 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4260.ko 2024-03-24T01:10:33.456 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4261.ko 2024-03-24T01:10:33.456 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max16065.ko 2024-03-24T01:10:33.456 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1619.ko 2024-03-24T01:10:33.456 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1668.ko 2024-03-24T01:10:33.456 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max197.ko 2024-03-24T01:10:33.456 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6639.ko 2024-03-24T01:10:33.456 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6650.ko 2024-03-24T01:10:33.456 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6697.ko 2024-03-24T01:10:33.456 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/mcp3021.ko 2024-03-24T01:10:33.457 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6683.ko 2024-03-24T01:10:33.457 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775-core.ko 2024-03-24T01:10:33.457 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775.ko 2024-03-24T01:10:33.457 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct7802.ko 2024-03-24T01:10:33.457 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87360.ko 2024-03-24T01:10:33.457 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87427.ko 2024-03-24T01:10:33.457 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pcf8591.ko 2024-03-24T01:10:33.457 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus 2024-03-24T01:10:33.457 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-24T01:10:33.457 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-24T01:10:33.457 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-24T01:10:33.457 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-24T01:10:33.457 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-24T01:10:33.457 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-24T01:10:33.458 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-24T01:10:33.458 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-24T01:10:33.458 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-24T01:10:33.458 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-24T01:10:33.458 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-24T01:10:33.458 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-24T01:10:33.458 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/powr1220.ko 2024-03-24T01:10:33.458 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5627.ko 2024-03-24T01:10:33.458 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5636.ko 2024-03-24T01:10:33.458 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-24T01:10:33.458 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht15.ko 2024-03-24T01:10:33.458 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht21.ko 2024-03-24T01:10:33.458 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/shtc1.ko 2024-03-24T01:10:33.458 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sis5595.ko 2024-03-24T01:10:33.458 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47b397.ko 2024-03-24T01:10:33.459 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m1.ko 2024-03-24T01:10:33.459 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m192.ko 2024-03-24T01:10:33.459 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/thmc50.ko 2024-03-24T01:10:33.459 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp102.ko 2024-03-24T01:10:33.459 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp103.ko 2024-03-24T01:10:33.459 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp401.ko 2024-03-24T01:10:33.459 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp421.ko 2024-03-24T01:10:33.459 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via-cputemp.ko 2024-03-24T01:10:33.459 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via686a.ko 2024-03-24T01:10:33.459 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt1211.ko 2024-03-24T01:10:33.459 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt8231.ko 2024-03-24T01:10:33.459 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627ehf.ko 2024-03-24T01:10:33.459 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627hf.ko 2024-03-24T01:10:33.459 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83781d.ko 2024-03-24T01:10:33.460 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83791d.ko 2024-03-24T01:10:33.460 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83792d.ko 2024-03-24T01:10:33.460 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83793.ko 2024-03-24T01:10:33.460 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83795.ko 2024-03-24T01:10:33.460 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l785ts.ko 2024-03-24T01:10:33.460 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l786ng.ko 2024-03-24T01:10:33.460 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c 2024-03-24T01:10:33.460 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos 2024-03-24T01:10:33.460 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-24T01:10:33.460 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-24T01:10:33.460 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses 2024-03-24T01:10:33.460 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-24T01:10:33.460 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-24T01:10:33.460 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-24T01:10:33.461 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-24T01:10:33.461 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-24T01:10:33.461 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-24T01:10:33.461 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-24T01:10:33.461 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-24T01:10:33.461 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-24T01:10:33.461 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-24T01:10:33.461 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-24T01:10:33.461 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-24T01:10:33.461 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-24T01:10:33.461 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-24T01:10:33.461 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-24T01:10:33.461 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-24T01:10:33.462 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-24T01:10:33.462 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-24T01:10:33.462 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-24T01:10:33.462 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-24T01:10:33.462 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-24T01:10:33.462 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-24T01:10:33.462 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-24T01:10:33.462 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-dev.ko 2024-03-24T01:10:33.462 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-mux.ko 2024-03-24T01:10:33.462 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-24T01:10:33.462 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-smbus.ko 2024-03-24T01:10:33.462 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-stub.ko 2024-03-24T01:10:33.462 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband 2024-03-24T01:10:33.463 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core 2024-03-24T01:10:33.463 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-24T01:10:33.463 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_core.ko 2024-03-24T01:10:33.463 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-24T01:10:33.463 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-24T01:10:33.463 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-24T01:10:33.463 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-24T01:10:33.463 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-24T01:10:33.463 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw 2024-03-24T01:10:33.463 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4 2024-03-24T01:10:33.463 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-24T01:10:33.463 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4 2024-03-24T01:10:33.464 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-24T01:10:33.464 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5 2024-03-24T01:10:33.464 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-24T01:10:33.464 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca 2024-03-24T01:10:33.464 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-24T01:10:33.464 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp 2024-03-24T01:10:33.464 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib 2024-03-24T01:10:33.464 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-24T01:10:33.464 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser 2024-03-24T01:10:33.464 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-24T01:10:33.464 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert 2024-03-24T01:10:33.464 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-24T01:10:33.465 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp 2024-03-24T01:10:33.465 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-24T01:10:33.465 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt 2024-03-24T01:10:33.465 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-24T01:10:33.465 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input 2024-03-24T01:10:33.465 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport 2024-03-24T01:10:33.465 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-24T01:10:33.465 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-24T01:10:33.465 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/gameport.ko 2024-03-24T01:10:33.465 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/joydev.ko 2024-03-24T01:10:33.465 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard 2024-03-24T01:10:33.465 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-24T01:10:33.466 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse 2024-03-24T01:10:33.466 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/appletouch.ko 2024-03-24T01:10:33.466 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/bcm5974.ko 2024-03-24T01:10:33.466 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/cyapatp.ko 2024-03-24T01:10:33.466 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-24T01:10:33.466 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/sermouse.ko 2024-03-24T01:10:33.466 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-24T01:10:33.466 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-24T01:10:33.466 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-24T01:10:33.466 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4 2024-03-24T01:10:33.466 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-24T01:10:33.466 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio 2024-03-24T01:10:33.467 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/altera_ps2.ko 2024-03-24T01:10:33.467 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/arc_ps2.ko 2024-03-24T01:10:33.467 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-24T01:10:33.467 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/serio_raw.ko 2024-03-24T01:10:33.467 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/sparse-keymap.ko 2024-03-24T01:10:33.467 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds 2024-03-24T01:10:33.467 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-blinkm.ko 2024-03-24T01:10:33.467 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lm3530.ko 2024-03-24T01:10:33.467 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lp3944.ko 2024-03-24T01:10:33.467 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-ss4200.ko 2024-03-24T01:10:33.467 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger 2024-03-24T01:10:33.467 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-24T01:10:33.468 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-24T01:10:33.468 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-24T01:10:33.468 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-24T01:10:33.468 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-24T01:10:33.468 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-24T01:10:33.468 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-24T01:10:33.468 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-24T01:10:33.468 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-24T01:10:33.468 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md 2024-03-24T01:10:33.468 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache 2024-03-24T01:10:33.468 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache/bcache.ko 2024-03-24T01:10:33.468 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-bio-prison.ko 2024-03-24T01:10:33.469 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache-smq.ko 2024-03-24T01:10:33.469 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache.ko 2024-03-24T01:10:33.469 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-crypt.ko 2024-03-24T01:10:33.469 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-delay.ko 2024-03-24T01:10:33.469 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-flakey.ko 2024-03-24T01:10:33.469 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-log-userspace.ko 2024-03-24T01:10:33.469 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-multipath.ko 2024-03-24T01:10:33.469 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-queue-length.ko 2024-03-24T01:10:33.469 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-raid.ko 2024-03-24T01:10:33.469 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-round-robin.ko 2024-03-24T01:10:33.469 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-service-time.ko 2024-03-24T01:10:33.469 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-switch.ko 2024-03-24T01:10:33.469 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-thin-pool.ko 2024-03-24T01:10:33.470 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-verity.ko 2024-03-24T01:10:33.470 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data 2024-03-24T01:10:33.470 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-24T01:10:33.470 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid0.ko 2024-03-24T01:10:33.470 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid1.ko 2024-03-24T01:10:33.470 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid10.ko 2024-03-24T01:10:33.470 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid456.ko 2024-03-24T01:10:33.470 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message 2024-03-24T01:10:33.470 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion 2024-03-24T01:10:33.470 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptbase.ko 2024-03-24T01:10:33.470 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptctl.ko 2024-03-24T01:10:33.470 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptfc.ko 2024-03-24T01:10:33.471 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptlan.ko 2024-03-24T01:10:33.471 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptsas.ko 2024-03-24T01:10:33.471 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptscsih.ko 2024-03-24T01:10:33.471 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptspi.ko 2024-03-24T01:10:33.471 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd 2024-03-24T01:10:33.471 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_ich.ko 2024-03-24T01:10:33.471 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_sch.ko 2024-03-24T01:10:33.471 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/mfd-core.ko 2024-03-24T01:10:33.471 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/sm501.ko 2024-03-24T01:10:33.471 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/viperboard.ko 2024-03-24T01:10:33.471 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/vx855.ko 2024-03-24T01:10:33.471 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/wl1273-core.ko 2024-03-24T01:10:33.471 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc 2024-03-24T01:10:33.472 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl 2024-03-24T01:10:33.472 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-24T01:10:33.472 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds9802als.ko 2024-03-24T01:10:33.472 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds990x.ko 2024-03-24T01:10:33.472 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/bh1770glc.ko 2024-03-24T01:10:33.472 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710 2024-03-24T01:10:33.472 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710/cb710.ko 2024-03-24T01:10:33.472 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom 2024-03-24T01:10:33.472 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/at24.ko 2024-03-24T01:10:33.472 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-24T01:10:33.472 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/max6875.ko 2024-03-24T01:10:33.472 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/enclosure.ko 2024-03-24T01:10:33.472 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/hpilo.ko 2024-03-24T01:10:33.472 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29003.ko 2024-03-24T01:10:33.473 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29020.ko 2024-03-24T01:10:33.473 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d 2024-03-24T01:10:33.473 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-24T01:10:33.473 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-24T01:10:33.473 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei 2024-03-24T01:10:33.473 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-me.ko 2024-03-24T01:10:33.473 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-txe.ko 2024-03-24T01:10:33.473 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei.ko 2024-03-24T01:10:33.473 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-gru 2024-03-24T01:10:33.473 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-24T01:10:33.473 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp 2024-03-24T01:10:33.473 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-24T01:10:33.473 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-24T01:10:33.474 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-24T01:10:33.474 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_7xx1.ko 2024-03-24T01:10:33.474 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_core.ko 2024-03-24T01:10:33.474 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tsl2550.ko 2024-03-24T01:10:33.474 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_balloon.ko 2024-03-24T01:10:33.474 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci 2024-03-24T01:10:33.474 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-24T01:10:33.474 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net 2024-03-24T01:10:33.474 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding 2024-03-24T01:10:33.474 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding/bonding.ko 2024-03-24T01:10:33.474 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa 2024-03-24T01:10:33.474 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53 2024-03-24T01:10:33.474 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-24T01:10:33.475 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-24T01:10:33.475 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-24T01:10:33.475 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx 2024-03-24T01:10:33.475 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-24T01:10:33.475 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dummy.ko 2024-03-24T01:10:33.475 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/eql.ko 2024-03-24T01:10:33.475 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet 2024-03-24T01:10:33.475 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com 2024-03-24T01:10:33.475 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-24T01:10:33.475 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-24T01:10:33.475 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-24T01:10:33.475 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-24T01:10:33.475 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390 2024-03-24T01:10:33.476 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-24T01:10:33.476 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-24T01:10:33.476 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-24T01:10:33.476 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-24T01:10:33.476 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec 2024-03-24T01:10:33.476 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-24T01:10:33.476 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere 2024-03-24T01:10:33.476 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-24T01:10:33.476 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon 2024-03-24T01:10:33.476 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-24T01:10:33.476 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera 2024-03-24T01:10:33.476 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-24T01:10:33.477 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd 2024-03-24T01:10:33.477 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-24T01:10:33.477 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-24T01:10:33.477 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-24T01:10:33.477 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros 2024-03-24T01:10:33.477 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx 2024-03-24T01:10:33.477 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-24T01:10:33.477 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-24T01:10:33.477 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-24T01:10:33.477 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-24T01:10:33.477 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-24T01:10:33.477 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx 2024-03-24T01:10:33.477 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-24T01:10:33.478 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-24T01:10:33.478 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom 2024-03-24T01:10:33.478 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-24T01:10:33.478 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-24T01:10:33.478 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-24T01:10:33.478 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-24T01:10:33.478 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-24T01:10:33.478 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet 2024-03-24T01:10:33.478 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-24T01:10:33.478 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-24T01:10:33.478 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade 2024-03-24T01:10:33.478 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna 2024-03-24T01:10:33.478 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-24T01:10:33.479 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio 2024-03-24T01:10:33.479 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-24T01:10:33.479 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-24T01:10:33.479 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-24T01:10:33.479 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-24T01:10:33.479 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-24T01:10:33.479 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-24T01:10:33.479 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-24T01:10:33.479 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-24T01:10:33.479 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-24T01:10:33.479 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-24T01:10:33.479 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco 2024-03-24T01:10:33.479 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic 2024-03-24T01:10:33.480 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-24T01:10:33.480 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec 2024-03-24T01:10:33.480 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip 2024-03-24T01:10:33.480 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-24T01:10:33.480 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-24T01:10:33.480 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-24T01:10:33.480 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-24T01:10:33.480 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-24T01:10:33.480 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-24T01:10:33.480 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink 2024-03-24T01:10:33.480 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-24T01:10:33.480 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-24T01:10:33.481 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dnet.ko 2024-03-24T01:10:33.481 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex 2024-03-24T01:10:33.481 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet 2024-03-24T01:10:33.481 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-24T01:10:33.481 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ethoc.ko 2024-03-24T01:10:33.481 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/fealnx.ko 2024-03-24T01:10:33.481 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel 2024-03-24T01:10:33.481 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-24T01:10:33.481 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000 2024-03-24T01:10:33.481 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-24T01:10:33.481 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e 2024-03-24T01:10:33.481 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-24T01:10:33.481 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k 2024-03-24T01:10:33.482 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-24T01:10:33.482 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e 2024-03-24T01:10:33.482 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-24T01:10:33.482 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf 2024-03-24T01:10:33.482 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-24T01:10:33.482 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb 2024-03-24T01:10:33.482 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-24T01:10:33.482 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf 2024-03-24T01:10:33.482 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-24T01:10:33.482 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-24T01:10:33.482 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-24T01:10:33.482 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-24T01:10:33.482 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-24T01:10:33.482 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/jme.ko 2024-03-24T01:10:33.483 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell 2024-03-24T01:10:33.483 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-24T01:10:33.483 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-24T01:10:33.483 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-24T01:10:33.483 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox 2024-03-24T01:10:33.483 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-24T01:10:33.483 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-24T01:10:33.483 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-24T01:10:33.483 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-24T01:10:33.483 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-24T01:10:33.483 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-24T01:10:33.483 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel 2024-03-24T01:10:33.483 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-24T01:10:33.484 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom 2024-03-24T01:10:33.484 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-24T01:10:33.484 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-24T01:10:33.484 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi 2024-03-24T01:10:33.484 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-24T01:10:33.484 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-24T01:10:33.484 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion 2024-03-24T01:10:33.484 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-24T01:10:33.484 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia 2024-03-24T01:10:33.484 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-24T01:10:33.484 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines 2024-03-24T01:10:33.484 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-24T01:10:33.484 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-24T01:10:33.484 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic 2024-03-24T01:10:33.485 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-24T01:10:33.485 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-24T01:10:33.485 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-24T01:10:33.485 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-24T01:10:33.485 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-24T01:10:33.485 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc 2024-03-24T01:10:33.485 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-24T01:10:33.485 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek 2024-03-24T01:10:33.485 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-24T01:10:33.485 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-24T01:10:33.485 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-24T01:10:33.485 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-24T01:10:33.485 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rocker 2024-03-24T01:10:33.486 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-24T01:10:33.486 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc 2024-03-24T01:10:33.486 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-24T01:10:33.486 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan 2024-03-24T01:10:33.486 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-24T01:10:33.486 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis 2024-03-24T01:10:33.486 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-24T01:10:33.486 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-24T01:10:33.486 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc 2024-03-24T01:10:33.486 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-24T01:10:33.486 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-24T01:10:33.486 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-24T01:10:33.486 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-24T01:10:33.486 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro 2024-03-24T01:10:33.487 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-24T01:10:33.487 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-24T01:10:33.487 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-24T01:10:33.487 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun 2024-03-24T01:10:33.487 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-24T01:10:33.487 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-24T01:10:33.487 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-24T01:10:33.487 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-24T01:10:33.487 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti 2024-03-24T01:10:33.487 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-24T01:10:33.487 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti 2024-03-24T01:10:33.487 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-24T01:10:33.487 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via 2024-03-24T01:10:33.488 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-24T01:10:33.488 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-24T01:10:33.488 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet 2024-03-24T01:10:33.488 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-24T01:10:33.488 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-24T01:10:33.488 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom 2024-03-24T01:10:33.488 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-24T01:10:33.488 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/geneve.ko 2024-03-24T01:10:33.488 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv 2024-03-24T01:10:33.488 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-24T01:10:33.488 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154 2024-03-24T01:10:33.488 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-24T01:10:33.488 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ifb.ko 2024-03-24T01:10:33.488 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan 2024-03-24T01:10:33.488 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-24T01:10:33.489 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvlan.ko 2024-03-24T01:10:33.489 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvtap.ko 2024-03-24T01:10:33.489 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio 2024-03-24T01:10:33.489 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio.ko 2024-03-24T01:10:33.489 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-24T01:10:33.489 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-24T01:10:33.489 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-24T01:10:33.489 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mii.ko 2024-03-24T01:10:33.489 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/net_failover.ko 2024-03-24T01:10:33.489 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/netconsole.ko 2024-03-24T01:10:33.489 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/nlmon.ko 2024-03-24T01:10:33.489 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs 2024-03-24T01:10:33.489 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-24T01:10:33.490 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-24T01:10:33.490 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy 2024-03-24T01:10:33.490 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/amd.ko 2024-03-24T01:10:33.490 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-24T01:10:33.490 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-24T01:10:33.490 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-24T01:10:33.490 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm87xx.ko 2024-03-24T01:10:33.490 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/broadcom.ko 2024-03-24T01:10:33.490 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/cicada.ko 2024-03-24T01:10:33.490 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/davicom.ko 2024-03-24T01:10:33.490 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/dp83640.ko 2024-03-24T01:10:33.490 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/et1011c.ko 2024-03-24T01:10:33.490 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/icplus.ko 2024-03-24T01:10:33.490 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/lxt.ko 2024-03-24T01:10:33.491 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/marvell.ko 2024-03-24T01:10:33.491 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/micrel.ko 2024-03-24T01:10:33.491 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/national.ko 2024-03-24T01:10:33.491 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/phylink.ko 2024-03-24T01:10:33.491 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/qsemi.ko 2024-03-24T01:10:33.491 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/realtek.ko 2024-03-24T01:10:33.491 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/smsc.ko 2024-03-24T01:10:33.491 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/ste10Xp.ko 2024-03-24T01:10:33.491 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/vitesse.ko 2024-03-24T01:10:33.491 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp 2024-03-24T01:10:33.491 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-24T01:10:33.491 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_async.ko 2024-03-24T01:10:33.491 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-24T01:10:33.491 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-24T01:10:33.492 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-24T01:10:33.492 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-24T01:10:33.492 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppoe.ko 2024-03-24T01:10:33.492 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppox.ko 2024-03-24T01:10:33.492 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pptp.ko 2024-03-24T01:10:33.492 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip 2024-03-24T01:10:33.492 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/slhc.ko 2024-03-24T01:10:33.492 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/slip.ko 2024-03-24T01:10:33.492 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/sungem_phy.ko 2024-03-24T01:10:33.492 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/tap.ko 2024-03-24T01:10:33.492 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team 2024-03-24T01:10:33.492 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team.ko 2024-03-24T01:10:33.492 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-24T01:10:33.493 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-24T01:10:33.493 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-24T01:10:33.493 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_random.ko 2024-03-24T01:10:33.493 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-24T01:10:33.493 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/tun.ko 2024-03-24T01:10:33.493 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/veth.ko 2024-03-24T01:10:33.493 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/virtio_net.ko 2024-03-24T01:10:33.493 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3 2024-03-24T01:10:33.493 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-24T01:10:33.493 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan 2024-03-24T01:10:33.493 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan/vxlan.ko 2024-03-24T01:10:33.493 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback 2024-03-24T01:10:33.493 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-24T01:10:33.493 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netfront.ko 2024-03-24T01:10:33.494 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb 2024-03-24T01:10:33.494 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb/ntb.ko 2024-03-24T01:10:33.494 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme 2024-03-24T01:10:33.494 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host 2024-03-24T01:10:33.494 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme-core.ko 2024-03-24T01:10:33.494 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme.ko 2024-03-24T01:10:33.494 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport 2024-03-24T01:10:33.494 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport.ko 2024-03-24T01:10:33.494 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_cs.ko 2024-03-24T01:10:33.494 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_pc.ko 2024-03-24T01:10:33.494 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_serial.ko 2024-03-24T01:10:33.494 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci 2024-03-24T01:10:33.494 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug 2024-03-24T01:10:33.494 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-24T01:10:33.495 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/pcie 2024-03-24T01:10:33.495 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-24T01:10:33.495 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/xen-pcifront.ko 2024-03-24T01:10:33.495 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia 2024-03-24T01:10:33.495 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/i82092.ko 2024-03-24T01:10:33.495 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/pd6729.ko 2024-03-24T01:10:33.495 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-24T01:10:33.495 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl 2024-03-24T01:10:33.495 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel 2024-03-24T01:10:33.495 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-24T01:10:33.495 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform 2024-03-24T01:10:33.495 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86 2024-03-24T01:10:33.495 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-24T01:10:33.496 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acerhdf.ko 2024-03-24T01:10:33.496 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-24T01:10:33.496 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-24T01:10:33.496 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-24T01:10:33.496 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-24T01:10:33.496 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-24T01:10:33.496 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-24T01:10:33.496 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-24T01:10:33.496 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-24T01:10:33.496 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-24T01:10:33.496 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/hdaps.ko 2024-03-24T01:10:33.496 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel 2024-03-24T01:10:33.496 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-24T01:10:33.496 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel_ips.ko 2024-03-24T01:10:33.496 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-24T01:10:33.497 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-24T01:10:33.497 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-24T01:10:33.497 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-24T01:10:33.497 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-24T01:10:33.497 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-24T01:10:33.497 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-24T01:10:33.497 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-24T01:10:33.497 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-24T01:10:33.497 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi.ko 2024-03-24T01:10:33.497 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap 2024-03-24T01:10:33.497 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-24T01:10:33.497 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-24T01:10:33.497 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps 2024-03-24T01:10:33.497 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients 2024-03-24T01:10:33.498 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-24T01:10:33.498 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-24T01:10:33.498 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps_parport.ko 2024-03-24T01:10:33.498 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/pps_core.ko 2024-03-24T01:10:33.498 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp 2024-03-24T01:10:33.498 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp.ko 2024-03-24T01:10:33.498 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp_kvm.ko 2024-03-24T01:10:33.498 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc 2024-03-24T01:10:33.498 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-24T01:10:33.498 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-24T01:10:33.498 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-24T01:10:33.498 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-24T01:10:33.498 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-24T01:10:33.498 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-24T01:10:33.499 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-24T01:10:33.499 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-24T01:10:33.499 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-24T01:10:33.499 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-24T01:10:33.499 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-em3027.ko 2024-03-24T01:10:33.499 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-24T01:10:33.499 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-24T01:10:33.499 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-24T01:10:33.499 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-24T01:10:33.499 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-24T01:10:33.499 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-24T01:10:33.499 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-max6900.ko 2024-03-24T01:10:33.499 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-24T01:10:33.499 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-24T01:10:33.500 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-24T01:10:33.500 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-24T01:10:33.500 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-24T01:10:33.500 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-24T01:10:33.500 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-24T01:10:33.500 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-24T01:10:33.500 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-24T01:10:33.500 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-24T01:10:33.500 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-24T01:10:33.500 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-24T01:10:33.500 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-x1205.ko 2024-03-24T01:10:33.500 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi 2024-03-24T01:10:33.500 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-9xxx.ko 2024-03-24T01:10:33.500 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-sas.ko 2024-03-24T01:10:33.501 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-xxxx.ko 2024-03-24T01:10:33.501 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/BusLogic.ko 2024-03-24T01:10:33.501 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/a100u2w.ko 2024-03-24T01:10:33.501 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid 2024-03-24T01:10:33.501 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-24T01:10:33.501 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/advansys.ko 2024-03-24T01:10:33.501 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx 2024-03-24T01:10:33.501 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-24T01:10:33.501 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-24T01:10:33.501 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/am53c974.ko 2024-03-24T01:10:33.501 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr 2024-03-24T01:10:33.501 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-24T01:10:33.501 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/atp870u.ko 2024-03-24T01:10:33.501 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi 2024-03-24T01:10:33.502 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-24T01:10:33.502 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa 2024-03-24T01:10:33.502 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa/bfa.ko 2024-03-24T01:10:33.502 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc 2024-03-24T01:10:33.502 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-24T01:10:33.502 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i 2024-03-24T01:10:33.502 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-24T01:10:33.502 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ch.ko 2024-03-24T01:10:33.502 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor 2024-03-24T01:10:33.502 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-24T01:10:33.502 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi 2024-03-24T01:10:33.502 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-24T01:10:33.502 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-24T01:10:33.503 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-24T01:10:33.503 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-24T01:10:33.503 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-24T01:10:33.503 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dc395x.ko 2024-03-24T01:10:33.503 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler 2024-03-24T01:10:33.503 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-24T01:10:33.503 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-24T01:10:33.503 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-24T01:10:33.503 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-24T01:10:33.503 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dmx3191d.ko 2024-03-24T01:10:33.503 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r 2024-03-24T01:10:33.503 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-24T01:10:33.503 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esp_scsi.ko 2024-03-24T01:10:33.503 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe 2024-03-24T01:10:33.504 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-24T01:10:33.504 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-24T01:10:33.504 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic 2024-03-24T01:10:33.504 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic/fnic.ko 2024-03-24T01:10:33.504 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hpsa.ko 2024-03-24T01:10:33.504 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hptiop.ko 2024-03-24T01:10:33.504 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hv_storvsc.ko 2024-03-24T01:10:33.504 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/initio.ko 2024-03-24T01:10:33.504 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ipr.ko 2024-03-24T01:10:33.504 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ips.ko 2024-03-24T01:10:33.504 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci 2024-03-24T01:10:33.504 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci/isci.ko 2024-03-24T01:10:33.504 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-24T01:10:33.504 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-24T01:10:33.504 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc 2024-03-24T01:10:33.505 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc/libfc.ko 2024-03-24T01:10:33.505 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi.ko 2024-03-24T01:10:33.505 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-24T01:10:33.505 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas 2024-03-24T01:10:33.505 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas/libsas.ko 2024-03-24T01:10:33.505 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc 2024-03-24T01:10:33.505 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-24T01:10:33.505 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid 2024-03-24T01:10:33.505 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid.ko 2024-03-24T01:10:33.505 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-24T01:10:33.505 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-24T01:10:33.505 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-24T01:10:33.505 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas 2024-03-24T01:10:33.506 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-24T01:10:33.506 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas 2024-03-24T01:10:33.506 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-24T01:10:33.506 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvumi.ko 2024-03-24T01:10:33.506 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001 2024-03-24T01:10:33.506 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-24T01:10:33.506 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pmcraid.ko 2024-03-24T01:10:33.506 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla1280.ko 2024-03-24T01:10:33.506 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx 2024-03-24T01:10:33.506 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-24T01:10:33.506 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-24T01:10:33.506 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx 2024-03-24T01:10:33.507 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-24T01:10:33.507 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/raid_class.ko 2024-03-24T01:10:33.507 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_debug.ko 2024-03-24T01:10:33.507 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-24T01:10:33.507 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-24T01:10:33.507 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-24T01:10:33.507 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-24T01:10:33.507 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-24T01:10:33.507 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ses.ko 2024-03-24T01:10:33.507 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/st.ko 2024-03-24T01:10:33.507 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/stex.ko 2024-03-24T01:10:33.507 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2 2024-03-24T01:10:33.507 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-24T01:10:33.508 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/virtio_scsi.ko 2024-03-24T01:10:33.508 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-24T01:10:33.508 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/wd719x.ko 2024-03-24T01:10:33.508 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb 2024-03-24T01:10:33.508 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb/ssb.ko 2024-03-24T01:10:33.508 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target 2024-03-24T01:10:33.508 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi 2024-03-24T01:10:33.508 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-24T01:10:33.508 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback 2024-03-24T01:10:33.508 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-24T01:10:33.508 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_file.ko 2024-03-24T01:10:33.508 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_iblock.ko 2024-03-24T01:10:33.508 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_mod.ko 2024-03-24T01:10:33.509 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_pscsi.ko 2024-03-24T01:10:33.509 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_user.ko 2024-03-24T01:10:33.509 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc 2024-03-24T01:10:33.509 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-24T01:10:33.509 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal 2024-03-24T01:10:33.509 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel 2024-03-24T01:10:33.509 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal 2024-03-24T01:10:33.509 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-24T01:10:33.509 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-24T01:10:33.509 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-24T01:10:33.509 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-24T01:10:33.509 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-24T01:10:33.509 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-24T01:10:33.509 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-24T01:10:33.510 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-24T01:10:33.510 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-24T01:10:33.510 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-24T01:10:33.510 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-24T01:10:33.510 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-24T01:10:33.510 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-24T01:10:33.510 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-24T01:10:33.510 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-24T01:10:33.510 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-24T01:10:33.510 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-24T01:10:33.510 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-24T01:10:33.510 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty 2024-03-24T01:10:33.510 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless 2024-03-24T01:10:33.510 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-24T01:10:33.511 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_gsm.ko 2024-03-24T01:10:33.511 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_hdlc.ko 2024-03-24T01:10:33.511 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/nozomi.ko 2024-03-24T01:10:33.511 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial 2024-03-24T01:10:33.511 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250 2024-03-24T01:10:33.511 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-24T01:10:33.511 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/arc_uart.ko 2024-03-24T01:10:33.511 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm 2024-03-24T01:10:33.511 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-24T01:10:33.511 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs 2024-03-24T01:10:33.511 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core 2024-03-24T01:10:33.511 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-24T01:10:33.511 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host 2024-03-24T01:10:33.511 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-24T01:10:33.512 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio 2024-03-24T01:10:33.512 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio.ko 2024-03-24T01:10:33.512 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_aec.ko 2024-03-24T01:10:33.512 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_cif.ko 2024-03-24T01:10:33.512 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_pci_generic.ko 2024-03-24T01:10:33.512 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_sercos3.ko 2024-03-24T01:10:33.512 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb 2024-03-24T01:10:33.512 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class 2024-03-24T01:10:33.512 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-acm.ko 2024-03-24T01:10:33.512 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-24T01:10:33.512 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usblp.ko 2024-03-24T01:10:33.512 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usbtmc.ko 2024-03-24T01:10:33.512 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host 2024-03-24T01:10:33.512 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-24T01:10:33.512 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image 2024-03-24T01:10:33.513 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/mdc800.ko 2024-03-24T01:10:33.513 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/microtek.ko 2024-03-24T01:10:33.513 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc 2024-03-24T01:10:33.513 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/adutux.ko 2024-03-24T01:10:33.513 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/appledisplay.ko 2024-03-24T01:10:33.513 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi26.ko 2024-03-24T01:10:33.513 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi62.ko 2024-03-24T01:10:33.513 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ezusb.ko 2024-03-24T01:10:33.513 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/idmouse.ko 2024-03-24T01:10:33.513 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/iowarrior.ko 2024-03-24T01:10:33.513 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-24T01:10:33.513 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ldusb.ko 2024-03-24T01:10:33.513 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/legousbtower.ko 2024-03-24T01:10:33.513 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga 2024-03-24T01:10:33.514 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-24T01:10:33.514 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-24T01:10:33.514 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usb3503.ko 2024-03-24T01:10:33.514 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usblcd.ko 2024-03-24T01:10:33.514 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-24T01:10:33.514 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/uss720.ko 2024-03-24T01:10:33.514 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/yurex.ko 2024-03-24T01:10:33.514 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy 2024-03-24T01:10:33.514 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy/phy-generic.ko 2024-03-24T01:10:33.514 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial 2024-03-24T01:10:33.514 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/aircable.ko 2024-03-24T01:10:33.514 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ark3116.ko 2024-03-24T01:10:33.514 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-24T01:10:33.515 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ch341.ko 2024-03-24T01:10:33.515 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cp210x.ko 2024-03-24T01:10:33.515 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cyberjack.ko 2024-03-24T01:10:33.515 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-24T01:10:33.515 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-24T01:10:33.515 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/empeg.ko 2024-03-24T01:10:33.515 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-24T01:10:33.515 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-24T01:10:33.515 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-24T01:10:33.515 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_ti.ko 2024-03-24T01:10:33.515 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipaq.ko 2024-03-24T01:10:33.515 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipw.ko 2024-03-24T01:10:33.515 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ir-usb.ko 2024-03-24T01:10:33.515 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-24T01:10:33.515 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan.ko 2024-03-24T01:10:33.516 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-24T01:10:33.516 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-24T01:10:33.516 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-24T01:10:33.516 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mct_u232.ko 2024-03-24T01:10:33.516 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7720.ko 2024-03-24T01:10:33.516 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7840.ko 2024-03-24T01:10:33.516 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/navman.ko 2024-03-24T01:10:33.516 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/omninet.ko 2024-03-24T01:10:33.516 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/opticon.ko 2024-03-24T01:10:33.516 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/option.ko 2024-03-24T01:10:33.516 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/oti6858.ko 2024-03-24T01:10:33.516 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/pl2303.ko 2024-03-24T01:10:33.516 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcaux.ko 2024-03-24T01:10:33.516 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcserial.ko 2024-03-24T01:10:33.517 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/quatech2.ko 2024-03-24T01:10:33.517 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/safe_serial.ko 2024-03-24T01:10:33.517 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/sierra.ko 2024-03-24T01:10:33.517 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-24T01:10:33.517 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ssu100.ko 2024-03-24T01:10:33.517 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/symbolserial.ko 2024-03-24T01:10:33.517 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-24T01:10:33.517 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-24T01:10:33.517 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_debug.ko 2024-03-24T01:10:33.517 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-24T01:10:33.517 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/visor.ko 2024-03-24T01:10:33.517 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/whiteheat.ko 2024-03-24T01:10:33.517 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-24T01:10:33.517 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage 2024-03-24T01:10:33.517 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/uas.ko 2024-03-24T01:10:33.517 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-24T01:10:33.518 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-24T01:10:33.518 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-24T01:10:33.518 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-24T01:10:33.518 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-24T01:10:33.518 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-24T01:10:33.518 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-24T01:10:33.518 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-karma.ko 2024-03-24T01:10:33.518 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-24T01:10:33.518 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-24T01:10:33.518 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-24T01:10:33.518 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-24T01:10:33.518 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-24T01:10:33.518 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/usb-storage.ko 2024-03-24T01:10:33.518 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip 2024-03-24T01:10:33.518 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-24T01:10:33.519 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-24T01:10:33.519 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-24T01:10:33.519 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio 2024-03-24T01:10:33.519 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci 2024-03-24T01:10:33.519 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-24T01:10:33.519 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-24T01:10:33.519 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio.ko 2024-03-24T01:10:33.519 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-24T01:10:33.519 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost 2024-03-24T01:10:33.519 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost.ko 2024-03-24T01:10:33.519 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-24T01:10:33.519 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_net.ko 2024-03-24T01:10:33.519 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_scsi.ko 2024-03-24T01:10:33.519 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video 2024-03-24T01:10:33.519 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight 2024-03-24T01:10:33.520 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/apple_bl.ko 2024-03-24T01:10:33.520 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/lcd.ko 2024-03-24T01:10:33.520 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-24T01:10:33.520 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio 2024-03-24T01:10:33.520 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio.ko 2024-03-24T01:10:33.520 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_balloon.ko 2024-03-24T01:10:33.520 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_mmio.ko 2024-03-24T01:10:33.520 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci.ko 2024-03-24T01:10:33.520 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-24T01:10:33.520 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-24T01:10:33.520 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_ring.ko 2024-03-24T01:10:33.520 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1 2024-03-24T01:10:33.520 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters 2024-03-24T01:10:33.520 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2482.ko 2024-03-24T01:10:33.521 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2490.ko 2024-03-24T01:10:33.521 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves 2024-03-24T01:10:33.521 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-24T01:10:33.521 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-24T01:10:33.521 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-24T01:10:33.521 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-24T01:10:33.521 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-24T01:10:33.521 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-24T01:10:33.521 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-24T01:10:33.521 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-24T01:10:33.521 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-24T01:10:33.521 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-24T01:10:33.521 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-24T01:10:33.521 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/wire.ko 2024-03-24T01:10:33.521 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog 2024-03-24T01:10:33.522 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-24T01:10:33.522 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-24T01:10:33.522 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-24T01:10:33.522 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/hpwdt.ko 2024-03-24T01:10:33.522 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/i6300esb.ko 2024-03-24T01:10:33.522 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-24T01:10:33.522 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-24T01:10:33.522 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ib700wdt.ko 2024-03-24T01:10:33.522 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ibmasr.ko 2024-03-24T01:10:33.522 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-24T01:10:33.522 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-24T01:10:33.522 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it87_wdt.ko 2024-03-24T01:10:33.522 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/machzwd.ko 2024-03-24T01:10:33.522 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/nv_tco.ko 2024-03-24T01:10:33.522 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-24T01:10:33.523 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-24T01:10:33.523 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-24T01:10:33.523 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-24T01:10:33.523 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/softdog.ko 2024-03-24T01:10:33.523 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-24T01:10:33.523 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/via_wdt.ko 2024-03-24T01:10:33.523 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-24T01:10:33.523 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-24T01:10:33.523 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-24T01:10:33.523 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/wdt_pci.ko 2024-03-24T01:10:33.523 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/xen_wdt.ko 2024-03-24T01:10:33.523 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen 2024-03-24T01:10:33.523 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-24T01:10:33.523 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-evtchn.ko 2024-03-24T01:10:33.524 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntalloc.ko 2024-03-24T01:10:33.524 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntdev.ko 2024-03-24T01:10:33.524 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback 2024-03-24T01:10:33.524 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-24T01:10:33.524 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-privcmd.ko 2024-03-24T01:10:33.524 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs 2024-03-24T01:10:33.524 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-24T01:10:33.524 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs 2024-03-24T01:10:33.524 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p 2024-03-24T01:10:33.524 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p/9p.ko 2024-03-24T01:10:33.524 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs 2024-03-24T01:10:33.524 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs/affs.ko 2024-03-24T01:10:33.525 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs 2024-03-24T01:10:33.525 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs/befs.ko 2024-03-24T01:10:33.525 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/binfmt_misc.ko 2024-03-24T01:10:33.525 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs 2024-03-24T01:10:33.525 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs/btrfs.ko 2024-03-24T01:10:33.525 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles 2024-03-24T01:10:33.525 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles/cachefiles.ko 2024-03-24T01:10:33.525 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph 2024-03-24T01:10:33.525 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph/ceph.ko 2024-03-24T01:10:33.525 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda 2024-03-24T01:10:33.525 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda/coda.ko 2024-03-24T01:10:33.525 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs 2024-03-24T01:10:33.525 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs/cramfs.ko 2024-03-24T01:10:33.525 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm 2024-03-24T01:10:33.526 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm/dlm.ko 2024-03-24T01:10:33.526 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ecryptfs 2024-03-24T01:10:33.526 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-24T01:10:33.526 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs 2024-03-24T01:10:33.526 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs/f2fs.ko 2024-03-24T01:10:33.526 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat 2024-03-24T01:10:33.526 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/fat.ko 2024-03-24T01:10:33.526 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/msdos.ko 2024-03-24T01:10:33.526 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/vfat.ko 2024-03-24T01:10:33.526 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse 2024-03-24T01:10:33.526 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/cuse.ko 2024-03-24T01:10:33.526 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/fuse.ko 2024-03-24T01:10:33.526 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2 2024-03-24T01:10:33.527 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2/gfs2.ko 2024-03-24T01:10:33.527 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs 2024-03-24T01:10:33.527 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs/hfs.ko 2024-03-24T01:10:33.527 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus 2024-03-24T01:10:33.527 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus/hfsplus.ko 2024-03-24T01:10:33.527 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs 2024-03-24T01:10:33.527 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs/isofs.ko 2024-03-24T01:10:33.527 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs 2024-03-24T01:10:33.527 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs/jfs.ko 2024-03-24T01:10:33.527 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd 2024-03-24T01:10:33.527 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd/lockd.ko 2024-03-24T01:10:33.527 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix 2024-03-24T01:10:33.527 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix/minix.ko 2024-03-24T01:10:33.527 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs 2024-03-24T01:10:33.528 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs/netfs.ko 2024-03-24T01:10:33.528 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs 2024-03-24T01:10:33.528 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout 2024-03-24T01:10:33.528 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-24T01:10:33.528 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout 2024-03-24T01:10:33.528 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-24T01:10:33.528 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout 2024-03-24T01:10:33.528 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-24T01:10:33.528 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfs.ko 2024-03-24T01:10:33.528 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv3.ko 2024-03-24T01:10:33.528 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv4.ko 2024-03-24T01:10:33.528 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common 2024-03-24T01:10:33.528 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/grace.ko 2024-03-24T01:10:33.528 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/nfs_acl.ko 2024-03-24T01:10:33.529 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd 2024-03-24T01:10:33.529 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd/nfsd.ko 2024-03-24T01:10:33.529 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2 2024-03-24T01:10:33.529 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2/nilfs2.ko 2024-03-24T01:10:33.529 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls 2024-03-24T01:10:33.529 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-celtic.ko 2024-03-24T01:10:33.529 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-centeuro.ko 2024-03-24T01:10:33.529 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-croatian.ko 2024-03-24T01:10:33.529 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-cyrillic.ko 2024-03-24T01:10:33.529 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-gaelic.ko 2024-03-24T01:10:33.529 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-greek.ko 2024-03-24T01:10:33.529 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-iceland.ko 2024-03-24T01:10:33.529 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-inuit.ko 2024-03-24T01:10:33.530 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-roman.ko 2024-03-24T01:10:33.530 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-romanian.ko 2024-03-24T01:10:33.530 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-turkish.ko 2024-03-24T01:10:33.530 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1250.ko 2024-03-24T01:10:33.530 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1251.ko 2024-03-24T01:10:33.530 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1255.ko 2024-03-24T01:10:33.530 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp737.ko 2024-03-24T01:10:33.530 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp775.ko 2024-03-24T01:10:33.530 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp850.ko 2024-03-24T01:10:33.530 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp852.ko 2024-03-24T01:10:33.530 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp855.ko 2024-03-24T01:10:33.530 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp857.ko 2024-03-24T01:10:33.530 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp860.ko 2024-03-24T01:10:33.530 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp861.ko 2024-03-24T01:10:33.530 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp862.ko 2024-03-24T01:10:33.531 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp863.ko 2024-03-24T01:10:33.531 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp864.ko 2024-03-24T01:10:33.531 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp865.ko 2024-03-24T01:10:33.531 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp866.ko 2024-03-24T01:10:33.531 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp869.ko 2024-03-24T01:10:33.531 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp874.ko 2024-03-24T01:10:33.531 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp932.ko 2024-03-24T01:10:33.531 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp936.ko 2024-03-24T01:10:33.531 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp949.ko 2024-03-24T01:10:33.531 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp950.ko 2024-03-24T01:10:33.531 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_euc-jp.ko 2024-03-24T01:10:33.531 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-1.ko 2024-03-24T01:10:33.531 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-13.ko 2024-03-24T01:10:33.531 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-14.ko 2024-03-24T01:10:33.532 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-15.ko 2024-03-24T01:10:33.532 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-2.ko 2024-03-24T01:10:33.532 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-3.ko 2024-03-24T01:10:33.532 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-4.ko 2024-03-24T01:10:33.532 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-5.ko 2024-03-24T01:10:33.532 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-6.ko 2024-03-24T01:10:33.532 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-7.ko 2024-03-24T01:10:33.532 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-9.ko 2024-03-24T01:10:33.532 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-r.ko 2024-03-24T01:10:33.532 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-ru.ko 2024-03-24T01:10:33.532 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-u.ko 2024-03-24T01:10:33.532 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-24T01:10:33.532 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_utf8.ko 2024-03-24T01:10:33.532 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2 2024-03-24T01:10:33.533 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster 2024-03-24T01:10:33.533 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-24T01:10:33.533 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm 2024-03-24T01:10:33.533 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-24T01:10:33.533 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs 2024-03-24T01:10:33.533 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-24T01:10:33.533 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2.ko 2024-03-24T01:10:33.533 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-24T01:10:33.533 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-24T01:10:33.533 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-24T01:10:33.533 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs 2024-03-24T01:10:33.533 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs/overlay.ko 2024-03-24T01:10:33.533 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore 2024-03-24T01:10:33.534 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore/ramoops.ko 2024-03-24T01:10:33.534 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs 2024-03-24T01:10:33.534 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs/reiserfs.ko 2024-03-24T01:10:33.534 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs 2024-03-24T01:10:33.534 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs/romfs.ko 2024-03-24T01:10:33.534 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb 2024-03-24T01:10:33.534 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client 2024-03-24T01:10:33.534 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client/cifs.ko 2024-03-24T01:10:33.534 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common 2024-03-24T01:10:33.534 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_arc4.ko 2024-03-24T01:10:33.534 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_md4.ko 2024-03-24T01:10:33.534 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs 2024-03-24T01:10:33.534 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs/squashfs.ko 2024-03-24T01:10:33.535 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv 2024-03-24T01:10:33.535 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv/sysv.ko 2024-03-24T01:10:33.535 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf 2024-03-24T01:10:33.535 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf/udf.ko 2024-03-24T01:10:33.535 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs 2024-03-24T01:10:33.535 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs/ufs.ko 2024-03-24T01:10:33.535 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs 2024-03-24T01:10:33.535 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs/xfs.ko 2024-03-24T01:10:33.535 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel 2024-03-24T01:10:33.535 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/rcu 2024-03-24T01:10:33.535 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/rcu/rcutorture.ko 2024-03-24T01:10:33.535 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/torture.ko 2024-03-24T01:10:33.535 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/trace 2024-03-24T01:10:33.536 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-24T01:10:33.536 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib 2024-03-24T01:10:33.536 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/asn1_encoder.ko 2024-03-24T01:10:33.536 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc-itu-t.ko 2024-03-24T01:10:33.536 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc8.ko 2024-03-24T01:10:33.536 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto 2024-03-24T01:10:33.536 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libarc4.ko 2024-03-24T01:10:33.536 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libdes.ko 2024-03-24T01:10:33.536 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/libcrc32c.ko 2024-03-24T01:10:33.536 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lru_cache.ko 2024-03-24T01:10:33.536 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4 2024-03-24T01:10:33.536 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4_compress.ko 2024-03-24T01:10:33.536 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4hc_compress.ko 2024-03-24T01:10:33.537 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math 2024-03-24T01:10:33.537 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math/cordic.ko 2024-03-24T01:10:33.537 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6 2024-03-24T01:10:33.537 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6/raid6_pq.ko 2024-03-24T01:10:33.537 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon 2024-03-24T01:10:33.537 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-24T01:10:33.537 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_bm.ko 2024-03-24T01:10:33.537 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_fsm.ko 2024-03-24T01:10:33.537 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_kmp.ko 2024-03-24T01:10:33.537 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/zstd 2024-03-24T01:10:33.537 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/zstd/zstd_compress.ko 2024-03-24T01:10:33.537 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm 2024-03-24T01:10:33.538 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm/hwpoison-inject.ko 2024-03-24T01:10:33.538 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net 2024-03-24T01:10:33.538 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan 2024-03-24T01:10:33.538 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/6lowpan.ko 2024-03-24T01:10:33.538 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_dest.ko 2024-03-24T01:10:33.538 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_fragment.ko 2024-03-24T01:10:33.538 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_hop.ko 2024-03-24T01:10:33.538 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-24T01:10:33.538 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_mobility.ko 2024-03-24T01:10:33.538 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_routing.ko 2024-03-24T01:10:33.538 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_udp.ko 2024-03-24T01:10:33.538 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802 2024-03-24T01:10:33.538 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/garp.ko 2024-03-24T01:10:33.539 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/mrp.ko 2024-03-24T01:10:33.539 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/p8022.ko 2024-03-24T01:10:33.539 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/psnap.ko 2024-03-24T01:10:33.539 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/stp.ko 2024-03-24T01:10:33.539 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q 2024-03-24T01:10:33.539 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q/8021q.ko 2024-03-24T01:10:33.539 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p 2024-03-24T01:10:33.539 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet.ko 2024-03-24T01:10:33.539 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_fd.ko 2024-03-24T01:10:33.539 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_rdma.ko 2024-03-24T01:10:33.539 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_virtio.ko 2024-03-24T01:10:33.539 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv 2024-03-24T01:10:33.539 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv/batman-adv.ko 2024-03-24T01:10:33.539 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge 2024-03-24T01:10:33.540 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/br_netfilter.ko 2024-03-24T01:10:33.540 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/bridge.ko 2024-03-24T01:10:33.540 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter 2024-03-24T01:10:33.540 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-24T01:10:33.540 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-24T01:10:33.540 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-24T01:10:33.540 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-24T01:10:33.540 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-24T01:10:33.540 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-24T01:10:33.540 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-24T01:10:33.540 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-24T01:10:33.540 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-24T01:10:33.540 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-24T01:10:33.540 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-24T01:10:33.540 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-24T01:10:33.541 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-24T01:10:33.541 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-24T01:10:33.541 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-24T01:10:33.541 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-24T01:10:33.541 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-24T01:10:33.541 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-24T01:10:33.541 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-24T01:10:33.541 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-24T01:10:33.541 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtables.ko 2024-03-24T01:10:33.541 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-24T01:10:33.541 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-24T01:10:33.541 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-24T01:10:33.541 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph 2024-03-24T01:10:33.542 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph/libceph.ko 2024-03-24T01:10:33.542 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core 2024-03-24T01:10:33.542 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/failover.ko 2024-03-24T01:10:33.542 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/pktgen.ko 2024-03-24T01:10:33.542 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp 2024-03-24T01:10:33.542 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp.ko 2024-03-24T01:10:33.542 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_diag.ko 2024-03-24T01:10:33.542 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv4.ko 2024-03-24T01:10:33.542 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv6.ko 2024-03-24T01:10:33.542 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dns_resolver 2024-03-24T01:10:33.542 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dns_resolver/dns_resolver.ko 2024-03-24T01:10:33.542 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa 2024-03-24T01:10:33.542 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/dsa_core.ko 2024-03-24T01:10:33.542 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_brcm.ko 2024-03-24T01:10:33.543 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_dsa.ko 2024-03-24T01:10:33.543 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_none.ko 2024-03-24T01:10:33.543 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_trailer.ko 2024-03-24T01:10:33.543 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154 2024-03-24T01:10:33.543 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan 2024-03-24T01:10:33.543 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-24T01:10:33.543 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154.ko 2024-03-24T01:10:33.543 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-24T01:10:33.543 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4 2024-03-24T01:10:33.543 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ah4.ko 2024-03-24T01:10:33.543 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/esp4.ko 2024-03-24T01:10:33.543 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/fou.ko 2024-03-24T01:10:33.543 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/gre.ko 2024-03-24T01:10:33.544 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/inet_diag.ko 2024-03-24T01:10:33.544 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_gre.ko 2024-03-24T01:10:33.544 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_tunnel.ko 2024-03-24T01:10:33.544 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_vti.ko 2024-03-24T01:10:33.544 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipcomp.ko 2024-03-24T01:10:33.544 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipip.ko 2024-03-24T01:10:33.544 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter 2024-03-24T01:10:33.544 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-24T01:10:33.544 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-24T01:10:33.544 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-24T01:10:33.544 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-24T01:10:33.544 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-24T01:10:33.544 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-24T01:10:33.544 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-24T01:10:33.544 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-24T01:10:33.544 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-24T01:10:33.545 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-24T01:10:33.545 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-24T01:10:33.545 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-24T01:10:33.545 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-24T01:10:33.545 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-24T01:10:33.545 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-24T01:10:33.545 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-24T01:10:33.545 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-24T01:10:33.545 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-24T01:10:33.545 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-24T01:10:33.545 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-24T01:10:33.545 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-24T01:10:33.545 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_bic.ko 2024-03-24T01:10:33.546 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_dctcp.ko 2024-03-24T01:10:33.546 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_diag.ko 2024-03-24T01:10:33.546 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_highspeed.ko 2024-03-24T01:10:33.546 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_htcp.ko 2024-03-24T01:10:33.546 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_hybla.ko 2024-03-24T01:10:33.546 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_illinois.ko 2024-03-24T01:10:33.546 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_lp.ko 2024-03-24T01:10:33.546 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_scalable.ko 2024-03-24T01:10:33.546 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_vegas.ko 2024-03-24T01:10:33.546 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_veno.ko 2024-03-24T01:10:33.546 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_westwood.ko 2024-03-24T01:10:33.546 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_yeah.ko 2024-03-24T01:10:33.546 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tunnel4.ko 2024-03-24T01:10:33.547 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_diag.ko 2024-03-24T01:10:33.547 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_tunnel.ko 2024-03-24T01:10:33.547 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-24T01:10:33.547 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6 2024-03-24T01:10:33.547 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ah6.ko 2024-03-24T01:10:33.547 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/esp6.ko 2024-03-24T01:10:33.547 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/fou6.ko 2024-03-24T01:10:33.547 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_tunnel.ko 2024-03-24T01:10:33.547 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-24T01:10:33.547 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_vti.ko 2024-03-24T01:10:33.547 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ipcomp6.ko 2024-03-24T01:10:33.547 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter 2024-03-24T01:10:33.547 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-24T01:10:33.548 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-24T01:10:33.548 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-24T01:10:33.548 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-24T01:10:33.548 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-24T01:10:33.548 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-24T01:10:33.548 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-24T01:10:33.548 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-24T01:10:33.548 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-24T01:10:33.548 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-24T01:10:33.548 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-24T01:10:33.548 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-24T01:10:33.548 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-24T01:10:33.548 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-24T01:10:33.548 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-24T01:10:33.549 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-24T01:10:33.549 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-24T01:10:33.549 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-24T01:10:33.549 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-24T01:10:33.549 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-24T01:10:33.549 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-24T01:10:33.549 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-24T01:10:33.549 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-24T01:10:33.549 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-24T01:10:33.549 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/sit.ko 2024-03-24T01:10:33.549 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/tunnel6.ko 2024-03-24T01:10:33.549 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-24T01:10:33.550 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key 2024-03-24T01:10:33.550 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key/af_key.ko 2024-03-24T01:10:33.550 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp 2024-03-24T01:10:33.550 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_core.ko 2024-03-24T01:10:33.550 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-24T01:10:33.550 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_eth.ko 2024-03-24T01:10:33.550 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip.ko 2024-03-24T01:10:33.550 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip6.ko 2024-03-24T01:10:33.550 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_netlink.ko 2024-03-24T01:10:33.550 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ppp.ko 2024-03-24T01:10:33.550 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc 2024-03-24T01:10:33.550 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc/llc.ko 2024-03-24T01:10:33.550 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154 2024-03-24T01:10:33.550 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154/mac802154.ko 2024-03-24T01:10:33.551 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls 2024-03-24T01:10:33.551 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls/mpls_gso.ko 2024-03-24T01:10:33.551 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter 2024-03-24T01:10:33.551 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset 2024-03-24T01:10:33.551 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set.ko 2024-03-24T01:10:33.551 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-24T01:10:33.551 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-24T01:10:33.551 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-24T01:10:33.551 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-24T01:10:33.551 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-24T01:10:33.551 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-24T01:10:33.551 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-24T01:10:33.552 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-24T01:10:33.552 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-24T01:10:33.552 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-24T01:10:33.552 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-24T01:10:33.552 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-24T01:10:33.552 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-24T01:10:33.552 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-24T01:10:33.552 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-24T01:10:33.552 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs 2024-03-24T01:10:33.552 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-24T01:10:33.552 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-24T01:10:33.552 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-24T01:10:33.552 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-24T01:10:33.552 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-24T01:10:33.553 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-24T01:10:33.553 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-24T01:10:33.553 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-24T01:10:33.553 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-24T01:10:33.553 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-24T01:10:33.553 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-24T01:10:33.553 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-24T01:10:33.553 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-24T01:10:33.553 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-24T01:10:33.553 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conncount.ko 2024-03-24T01:10:33.553 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack.ko 2024-03-24T01:10:33.553 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-24T01:10:33.553 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-24T01:10:33.553 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-24T01:10:33.553 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-24T01:10:33.554 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-24T01:10:33.554 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-24T01:10:33.554 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-24T01:10:33.554 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-24T01:10:33.554 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-24T01:10:33.554 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-24T01:10:33.554 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-24T01:10:33.554 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-24T01:10:33.554 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-24T01:10:33.554 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table.ko 2024-03-24T01:10:33.554 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-24T01:10:33.554 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_log_syslog.ko 2024-03-24T01:10:33.554 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat.ko 2024-03-24T01:10:33.555 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-24T01:10:33.555 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-24T01:10:33.555 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_irc.ko 2024-03-24T01:10:33.555 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_sip.ko 2024-03-24T01:10:33.555 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-24T01:10:33.555 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-24T01:10:33.555 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_tables.ko 2024-03-24T01:10:33.555 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink.ko 2024-03-24T01:10:33.555 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-24T01:10:33.555 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_log.ko 2024-03-24T01:10:33.555 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-24T01:10:33.555 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-24T01:10:33.555 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_chain_nat.ko 2024-03-24T01:10:33.555 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_compat.ko 2024-03-24T01:10:33.556 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_ct.ko 2024-03-24T01:10:33.556 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-24T01:10:33.556 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fib.ko 2024-03-24T01:10:33.556 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-24T01:10:33.556 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_hash.ko 2024-03-24T01:10:33.556 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_limit.ko 2024-03-24T01:10:33.556 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_log.ko 2024-03-24T01:10:33.556 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_masq.ko 2024-03-24T01:10:33.556 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_nat.ko 2024-03-24T01:10:33.556 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_numgen.ko 2024-03-24T01:10:33.556 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_queue.ko 2024-03-24T01:10:33.556 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_quota.ko 2024-03-24T01:10:33.556 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_redir.ko 2024-03-24T01:10:33.556 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject.ko 2024-03-24T01:10:33.556 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject_inet.ko 2024-03-24T01:10:33.557 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_socket.ko 2024-03-24T01:10:33.557 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_tunnel.ko 2024-03-24T01:10:33.557 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_AUDIT.ko 2024-03-24T01:10:33.557 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-24T01:10:33.557 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-24T01:10:33.557 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-24T01:10:33.557 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CT.ko 2024-03-24T01:10:33.557 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_DSCP.ko 2024-03-24T01:10:33.557 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HL.ko 2024-03-24T01:10:33.557 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HMARK.ko 2024-03-24T01:10:33.557 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-24T01:10:33.557 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LED.ko 2024-03-24T01:10:33.557 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LOG.ko 2024-03-24T01:10:33.557 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-24T01:10:33.558 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NETMAP.ko 2024-03-24T01:10:33.558 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFLOG.ko 2024-03-24T01:10:33.558 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-24T01:10:33.558 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_RATEEST.ko 2024-03-24T01:10:33.558 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-24T01:10:33.558 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_SECMARK.ko 2024-03-24T01:10:33.558 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-24T01:10:33.558 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-24T01:10:33.558 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TEE.ko 2024-03-24T01:10:33.558 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TPROXY.ko 2024-03-24T01:10:33.558 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TRACE.ko 2024-03-24T01:10:33.558 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_addrtype.ko 2024-03-24T01:10:33.558 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_bpf.ko 2024-03-24T01:10:33.558 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cgroup.ko 2024-03-24T01:10:33.558 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cluster.ko 2024-03-24T01:10:33.559 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_comment.ko 2024-03-24T01:10:33.559 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connbytes.ko 2024-03-24T01:10:33.559 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlabel.ko 2024-03-24T01:10:33.559 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlimit.ko 2024-03-24T01:10:33.559 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connmark.ko 2024-03-24T01:10:33.559 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_conntrack.ko 2024-03-24T01:10:33.559 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cpu.ko 2024-03-24T01:10:33.559 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dccp.ko 2024-03-24T01:10:33.559 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_devgroup.ko 2024-03-24T01:10:33.559 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dscp.ko 2024-03-24T01:10:33.559 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ecn.ko 2024-03-24T01:10:33.559 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_esp.ko 2024-03-24T01:10:33.559 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hashlimit.ko 2024-03-24T01:10:33.559 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_helper.ko 2024-03-24T01:10:33.560 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hl.ko 2024-03-24T01:10:33.560 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipcomp.ko 2024-03-24T01:10:33.560 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_iprange.ko 2024-03-24T01:10:33.560 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipvs.ko 2024-03-24T01:10:33.560 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_l2tp.ko 2024-03-24T01:10:33.560 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_length.ko 2024-03-24T01:10:33.560 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_limit.ko 2024-03-24T01:10:33.560 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mac.ko 2024-03-24T01:10:33.560 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mark.ko 2024-03-24T01:10:33.560 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_multiport.ko 2024-03-24T01:10:33.560 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nat.ko 2024-03-24T01:10:33.560 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nfacct.ko 2024-03-24T01:10:33.560 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_osf.ko 2024-03-24T01:10:33.560 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_owner.ko 2024-03-24T01:10:33.560 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_physdev.ko 2024-03-24T01:10:33.561 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_pkttype.ko 2024-03-24T01:10:33.561 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_policy.ko 2024-03-24T01:10:33.561 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_quota.ko 2024-03-24T01:10:33.561 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_rateest.ko 2024-03-24T01:10:33.561 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_realm.ko 2024-03-24T01:10:33.561 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_recent.ko 2024-03-24T01:10:33.561 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_sctp.ko 2024-03-24T01:10:33.561 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_set.ko 2024-03-24T01:10:33.561 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_socket.ko 2024-03-24T01:10:33.561 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_state.ko 2024-03-24T01:10:33.561 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_statistic.ko 2024-03-24T01:10:33.561 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_string.ko 2024-03-24T01:10:33.561 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_tcpmss.ko 2024-03-24T01:10:33.562 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_time.ko 2024-03-24T01:10:33.562 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_u32.ko 2024-03-24T01:10:33.562 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink 2024-03-24T01:10:33.562 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink/netlink_diag.ko 2024-03-24T01:10:33.562 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh 2024-03-24T01:10:33.562 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh/nsh.ko 2024-03-24T01:10:33.562 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch 2024-03-24T01:10:33.562 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/openvswitch.ko 2024-03-24T01:10:33.562 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-geneve.ko 2024-03-24T01:10:33.562 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-gre.ko 2024-03-24T01:10:33.562 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-vxlan.ko 2024-03-24T01:10:33.562 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet 2024-03-24T01:10:33.562 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet/af_packet_diag.ko 2024-03-24T01:10:33.563 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds 2024-03-24T01:10:33.563 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds.ko 2024-03-24T01:10:33.563 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_rdma.ko 2024-03-24T01:10:33.563 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_tcp.ko 2024-03-24T01:10:33.563 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched 2024-03-24T01:10:33.563 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_csum.ko 2024-03-24T01:10:33.563 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_gact.ko 2024-03-24T01:10:33.563 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_mirred.ko 2024-03-24T01:10:33.563 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_nat.ko 2024-03-24T01:10:33.563 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_pedit.ko 2024-03-24T01:10:33.563 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_police.ko 2024-03-24T01:10:33.563 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_simple.ko 2024-03-24T01:10:33.563 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_skbedit.ko 2024-03-24T01:10:33.564 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_vlan.ko 2024-03-24T01:10:33.564 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_basic.ko 2024-03-24T01:10:33.564 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_bpf.ko 2024-03-24T01:10:33.564 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_flow.ko 2024-03-24T01:10:33.564 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_fw.ko 2024-03-24T01:10:33.564 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_route.ko 2024-03-24T01:10:33.564 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_u32.ko 2024-03-24T01:10:33.564 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_cmp.ko 2024-03-24T01:10:33.564 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_ipset.ko 2024-03-24T01:10:33.564 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_meta.ko 2024-03-24T01:10:33.564 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_nbyte.ko 2024-03-24T01:10:33.564 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_text.ko 2024-03-24T01:10:33.564 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_u32.ko 2024-03-24T01:10:33.565 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_choke.ko 2024-03-24T01:10:33.565 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_codel.ko 2024-03-24T01:10:33.565 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_drr.ko 2024-03-24T01:10:33.565 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_fq.ko 2024-03-24T01:10:33.565 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_gred.ko 2024-03-24T01:10:33.565 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hfsc.ko 2024-03-24T01:10:33.565 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hhf.ko 2024-03-24T01:10:33.565 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_htb.ko 2024-03-24T01:10:33.565 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_ingress.ko 2024-03-24T01:10:33.565 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio.ko 2024-03-24T01:10:33.565 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio_lib.ko 2024-03-24T01:10:33.565 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_multiq.ko 2024-03-24T01:10:33.565 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_netem.ko 2024-03-24T01:10:33.565 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_pie.ko 2024-03-24T01:10:33.566 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_plug.ko 2024-03-24T01:10:33.566 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_prio.ko 2024-03-24T01:10:33.566 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_qfq.ko 2024-03-24T01:10:33.566 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_red.ko 2024-03-24T01:10:33.566 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfb.ko 2024-03-24T01:10:33.566 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfq.ko 2024-03-24T01:10:33.566 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_tbf.ko 2024-03-24T01:10:33.566 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_teql.ko 2024-03-24T01:10:33.566 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp 2024-03-24T01:10:33.566 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp.ko 2024-03-24T01:10:33.566 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp_diag.ko 2024-03-24T01:10:33.567 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc 2024-03-24T01:10:33.567 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss 2024-03-24T01:10:33.567 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-24T01:10:33.567 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-24T01:10:33.567 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/sunrpc.ko 2024-03-24T01:10:33.567 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma 2024-03-24T01:10:33.567 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-24T01:10:33.567 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc 2024-03-24T01:10:33.567 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/diag.ko 2024-03-24T01:10:33.567 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/tipc.ko 2024-03-24T01:10:33.567 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix 2024-03-24T01:10:33.567 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix/unix_diag.ko 2024-03-24T01:10:33.567 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock 2024-03-24T01:10:33.567 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-24T01:10:33.568 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-24T01:10:33.568 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock.ko 2024-03-24T01:10:33.568 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-24T01:10:33.568 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-24T01:10:33.568 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm 2024-03-24T01:10:33.568 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-24T01:10:33.568 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security 2024-03-24T01:10:33.568 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys 2024-03-24T01:10:33.568 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/encrypted-keys 2024-03-24T01:10:33.568 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-24T01:10:33.568 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/trusted-keys 2024-03-24T01:10:33.568 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/trusted-keys/trusted.ko 2024-03-24T01:10:33.568 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt 2024-03-24T01:10:33.568 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib 2024-03-24T01:10:33.569 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib/irqbypass.ko 2024-03-24T01:10:33.569 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias 2024-03-24T01:10:33.569 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias.bin 2024-03-24T01:10:33.569 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin 2024-03-24T01:10:33.569 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.alias.bin 2024-03-24T01:10:33.569 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.bin 2024-03-24T01:10:33.569 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.modinfo 2024-03-24T01:10:33.569 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep 2024-03-24T01:10:33.569 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep.bin 2024-03-24T01:10:33.569 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.devname 2024-03-24T01:10:33.569 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.order 2024-03-24T01:10:33.569 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.softdep 2024-03-24T01:10:33.569 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols 2024-03-24T01:10:33.569 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols.bin 2024-03-24T01:10:33.569 INFO:teuthology.orchestra.run.smithi120.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/vmlinuz 2024-03-24T01:10:33.573 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g5ec3e6e88b2e 2024-03-24T01:10:33.574 DEBUG:teuthology.orchestra.run.smithi120:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-24T01:10:33.597 INFO:teuthology.orchestra.run.smithi120.stdout:postinstall scriptlet (using /bin/sh): 2024-03-24T01:10:33.597 INFO:teuthology.orchestra.run.smithi120.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-24T01:10:33.597 INFO:teuthology.orchestra.run.smithi120.stdout: /usr/bin/kernel-install add 6.8.0-g5ec3e6e88b2e /lib/modules/6.8.0-g5ec3e6e88b2e/vmlinuz 2024-03-24T01:10:33.597 INFO:teuthology.orchestra.run.smithi120.stdout:fi 2024-03-24T01:10:33.598 INFO:teuthology.orchestra.run.smithi120.stdout:for file in vmlinuz System.map config; do 2024-03-24T01:10:33.598 INFO:teuthology.orchestra.run.smithi120.stdout: if ! cmp --silent "/lib/modules/6.8.0-g5ec3e6e88b2e/${file}" "/boot/${file}-6.8.0-g5ec3e6e88b2e"; then 2024-03-24T01:10:33.598 INFO:teuthology.orchestra.run.smithi120.stdout: cp "/lib/modules/6.8.0-g5ec3e6e88b2e/${file}" "/boot/${file}-6.8.0-g5ec3e6e88b2e" 2024-03-24T01:10:33.598 INFO:teuthology.orchestra.run.smithi120.stdout: fi 2024-03-24T01:10:33.598 INFO:teuthology.orchestra.run.smithi120.stdout:done 2024-03-24T01:10:33.599 INFO:teuthology.orchestra.run.smithi120.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-24T01:10:33.599 INFO:teuthology.orchestra.run.smithi120.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-24T01:10:33.599 INFO:teuthology.orchestra.run.smithi120.stdout:new-kernel-pkg --remove 6.8.0-g5ec3e6e88b2e --rminitrd --initrdfile=/boot/initramfs-6.8.0-g5ec3e6e88b2e.img 2024-03-24T01:10:33.599 INFO:teuthology.orchestra.run.smithi120.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-24T01:10:33.599 INFO:teuthology.orchestra.run.smithi120.stdout:kernel-install remove 6.8.0-g5ec3e6e88b2e 2024-03-24T01:10:33.600 INFO:teuthology.orchestra.run.smithi120.stdout:fi 2024-03-24T01:10:33.600 INFO:teuthology.orchestra.run.smithi120.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-24T01:10:33.600 INFO:teuthology.orchestra.run.smithi120.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-24T01:10:33.600 INFO:teuthology.orchestra.run.smithi120.stdout:/sbin/update-bootloader --remove 6.8.0-g5ec3e6e88b2e 2024-03-24T01:10:33.600 INFO:teuthology.orchestra.run.smithi120.stdout:fi 2024-03-24T01:10:33.601 DEBUG:teuthology.orchestra.run.smithi120:> sudo rpm -qi grub2-tools 2024-03-24T01:10:33.676 INFO:teuthology.orchestra.run.smithi120.stdout:Name : grub2-tools 2024-03-24T01:10:33.676 INFO:teuthology.orchestra.run.smithi120.stdout:Epoch : 1 2024-03-24T01:10:33.677 INFO:teuthology.orchestra.run.smithi120.stdout:Version : 2.06 2024-03-24T01:10:33.677 INFO:teuthology.orchestra.run.smithi120.stdout:Release : 68.el9 2024-03-24T01:10:33.677 INFO:teuthology.orchestra.run.smithi120.stdout:Architecture: x86_64 2024-03-24T01:10:33.677 INFO:teuthology.orchestra.run.smithi120.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T01:10:33.677 INFO:teuthology.orchestra.run.smithi120.stdout:Group : Unspecified 2024-03-24T01:10:33.677 INFO:teuthology.orchestra.run.smithi120.stdout:Size : 8264254 2024-03-24T01:10:33.677 INFO:teuthology.orchestra.run.smithi120.stdout:License : GPLv3+ 2024-03-24T01:10:33.677 INFO:teuthology.orchestra.run.smithi120.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T01:10:33.677 INFO:teuthology.orchestra.run.smithi120.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T01:10:33.677 INFO:teuthology.orchestra.run.smithi120.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T01:10:33.677 INFO:teuthology.orchestra.run.smithi120.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T01:10:33.677 INFO:teuthology.orchestra.run.smithi120.stdout:Packager : builder@centos.org 2024-03-24T01:10:33.677 INFO:teuthology.orchestra.run.smithi120.stdout:Vendor : CentOS 2024-03-24T01:10:33.678 INFO:teuthology.orchestra.run.smithi120.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T01:10:33.678 INFO:teuthology.orchestra.run.smithi120.stdout:Summary : Support tools for GRUB. 2024-03-24T01:10:33.678 INFO:teuthology.orchestra.run.smithi120.stdout:Description : 2024-03-24T01:10:33.678 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:10:33.678 INFO:teuthology.orchestra.run.smithi120.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T01:10:33.678 INFO:teuthology.orchestra.run.smithi120.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T01:10:33.678 INFO:teuthology.orchestra.run.smithi120.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T01:10:33.678 INFO:teuthology.orchestra.run.smithi120.stdout:hardware devices. 2024-03-24T01:10:33.678 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:10:33.678 INFO:teuthology.orchestra.run.smithi120.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T01:10:33.679 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T01:10:33.679 INFO:teuthology.task.kernel:Updating grub on smithi120 to boot 6.8.0-g5ec3e6e88b2e 2024-03-24T01:10:33.679 DEBUG:teuthology.orchestra.run.smithi120:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T01:10:34.298 INFO:teuthology.orchestra.run.smithi120.stderr:Generating grub configuration file ... 2024-03-24T01:10:35.699 INFO:teuthology.orchestra.run.smithi120.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T01:10:35.731 INFO:teuthology.orchestra.run.smithi120.stderr:done 2024-03-24T01:10:35.733 DEBUG:teuthology.orchestra.run.smithi120:> mktemp 2024-03-24T01:10:35.747 INFO:teuthology.orchestra.run.smithi120.stdout:/tmp/tmp.5BCNCaYMXs 2024-03-24T01:10:35.748 DEBUG:teuthology.orchestra.run.smithi120:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.5BCNCaYMXs 2024-03-24T01:10:35.816 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 0666 /tmp/tmp.5BCNCaYMXs 2024-03-24T01:10:35.959 DEBUG:teuthology.orchestra.remote:smithi120:/tmp/tmp.5BCNCaYMXs is 6KB 2024-03-24T01:10:36.008 DEBUG:teuthology.orchestra.run.smithi120:> rm -fr /tmp/tmp.5BCNCaYMXs 2024-03-24T01:10:36.022 DEBUG:teuthology.orchestra.run.smithi120:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T01:10:36.086 INFO:teuthology.orchestra.run.smithi120.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T01:10:36.087 INFO:teuthology.orchestra.run.smithi120.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T01:10:36.087 INFO:teuthology.orchestra.run.smithi120.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T01:10:36.087 INFO:teuthology.orchestra.run.smithi120.stdout:15949e3c88704300882482eddfd7baae-6.8.0-g5ec3e6e88b2e.conf 2024-03-24T01:10:36.088 DEBUG:teuthology.orchestra.run.smithi120:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-g5ec3e6e88b2e 2024-03-24T01:10:36.239 DEBUG:teuthology.orchestra.run.smithi120:> sudo shutdown -r now 2024-03-24T01:10:53.340 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi090.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-24T01:10:53.341 DEBUG:teuthology.orchestra.run.smithi090:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-24T01:10:53.430 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e 2024-03-24T01:10:53.430 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/System.map 2024-03-24T01:10:53.431 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/config 2024-03-24T01:10:53.431 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel 2024-03-24T01:10:53.431 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch 2024-03-24T01:10:53.431 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86 2024-03-24T01:10:53.431 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto 2024-03-24T01:10:53.431 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-24T01:10:53.431 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-24T01:10:53.431 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-24T01:10:53.431 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-24T01:10:53.431 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-24T01:10:53.431 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-24T01:10:53.431 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-24T01:10:53.431 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-24T01:10:53.431 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-24T01:10:53.432 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-24T01:10:53.432 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-24T01:10:53.432 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-24T01:10:53.432 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-24T01:10:53.432 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-24T01:10:53.432 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-24T01:10:53.432 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-24T01:10:53.432 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-24T01:10:53.432 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-24T01:10:53.432 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-24T01:10:53.432 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-24T01:10:53.432 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel 2024-03-24T01:10:53.432 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu 2024-03-24T01:10:53.432 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce 2024-03-24T01:10:53.433 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-24T01:10:53.433 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm 2024-03-24T01:10:53.433 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-24T01:10:53.433 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-24T01:10:53.433 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm.ko 2024-03-24T01:10:53.433 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto 2024-03-24T01:10:53.433 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ansi_cprng.ko 2024-03-24T01:10:53.433 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/anubis.ko 2024-03-24T01:10:53.433 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/arc4.ko 2024-03-24T01:10:53.433 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx 2024-03-24T01:10:53.433 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_memcpy.ko 2024-03-24T01:10:53.433 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_pq.ko 2024-03-24T01:10:53.433 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-24T01:10:53.433 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_tx.ko 2024-03-24T01:10:53.433 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_xor.ko 2024-03-24T01:10:53.434 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/raid6test.ko 2024-03-24T01:10:53.434 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authenc.ko 2024-03-24T01:10:53.434 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authencesn.ko 2024-03-24T01:10:53.434 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blake2b_generic.ko 2024-03-24T01:10:53.434 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_common.ko 2024-03-24T01:10:53.434 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_generic.ko 2024-03-24T01:10:53.434 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/camellia_generic.ko 2024-03-24T01:10:53.434 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast5_generic.ko 2024-03-24T01:10:53.434 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast6_generic.ko 2024-03-24T01:10:53.434 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast_common.ko 2024-03-24T01:10:53.434 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ccm.ko 2024-03-24T01:10:53.434 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cmac.ko 2024-03-24T01:10:53.434 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crc32_generic.ko 2024-03-24T01:10:53.434 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_engine.ko 2024-03-24T01:10:53.435 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_user.ko 2024-03-24T01:10:53.435 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cts.ko 2024-03-24T01:10:53.435 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/des_generic.ko 2024-03-24T01:10:53.435 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/dh_generic.ko 2024-03-24T01:10:53.435 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/echainiv.ko 2024-03-24T01:10:53.435 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/essiv.ko 2024-03-24T01:10:53.435 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/fcrypt.ko 2024-03-24T01:10:53.435 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/khazad.ko 2024-03-24T01:10:53.435 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4.ko 2024-03-24T01:10:53.435 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4hc.ko 2024-03-24T01:10:53.435 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/md4.ko 2024-03-24T01:10:53.435 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/michael_mic.ko 2024-03-24T01:10:53.435 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcbc.ko 2024-03-24T01:10:53.435 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcrypt.ko 2024-03-24T01:10:53.436 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/rmd160.ko 2024-03-24T01:10:53.436 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/seed.ko 2024-03-24T01:10:53.436 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/serpent_generic.ko 2024-03-24T01:10:53.436 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tcrypt.ko 2024-03-24T01:10:53.436 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tea.ko 2024-03-24T01:10:53.436 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_common.ko 2024-03-24T01:10:53.436 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_generic.ko 2024-03-24T01:10:53.436 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/vmac.ko 2024-03-24T01:10:53.436 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/wp512.ko 2024-03-24T01:10:53.436 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xcbc.ko 2024-03-24T01:10:53.436 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xor.ko 2024-03-24T01:10:53.436 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xxhash_generic.ko 2024-03-24T01:10:53.436 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers 2024-03-24T01:10:53.436 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi 2024-03-24T01:10:53.436 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-24T01:10:53.437 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_pad.ko 2024-03-24T01:10:53.437 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/custom_method.ko 2024-03-24T01:10:53.437 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/ec_sys.ko 2024-03-24T01:10:53.437 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/platform_profile.ko 2024-03-24T01:10:53.437 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbs.ko 2024-03-24T01:10:53.437 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbshc.ko 2024-03-24T01:10:53.437 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/video.ko 2024-03-24T01:10:53.437 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata 2024-03-24T01:10:53.437 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/acard-ahci.ko 2024-03-24T01:10:53.437 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ahci_platform.ko 2024-03-24T01:10:53.437 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ata_generic.ko 2024-03-24T01:10:53.437 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/libahci_platform.ko 2024-03-24T01:10:53.437 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_acpi.ko 2024-03-24T01:10:53.437 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ali.ko 2024-03-24T01:10:53.437 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_amd.ko 2024-03-24T01:10:53.438 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_artop.ko 2024-03-24T01:10:53.438 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atiixp.ko 2024-03-24T01:10:53.438 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atp867x.ko 2024-03-24T01:10:53.438 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd640.ko 2024-03-24T01:10:53.438 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd64x.ko 2024-03-24T01:10:53.438 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cypress.ko 2024-03-24T01:10:53.438 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_efar.ko 2024-03-24T01:10:53.438 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt366.ko 2024-03-24T01:10:53.438 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt37x.ko 2024-03-24T01:10:53.438 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-24T01:10:53.438 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-24T01:10:53.438 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it8213.ko 2024-03-24T01:10:53.438 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it821x.ko 2024-03-24T01:10:53.438 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_jmicron.ko 2024-03-24T01:10:53.439 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_marvell.ko 2024-03-24T01:10:53.439 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_mpiix.ko 2024-03-24T01:10:53.439 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_netcell.ko 2024-03-24T01:10:53.439 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ninja32.ko 2024-03-24T01:10:53.439 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87410.ko 2024-03-24T01:10:53.439 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87415.ko 2024-03-24T01:10:53.439 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_oldpiix.ko 2024-03-24T01:10:53.439 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_opti.ko 2024-03-24T01:10:53.439 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_optidma.ko 2024-03-24T01:10:53.439 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pcmcia.ko 2024-03-24T01:10:53.439 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-24T01:10:53.439 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-24T01:10:53.439 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_piccolo.ko 2024-03-24T01:10:53.439 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_rdc.ko 2024-03-24T01:10:53.440 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sch.ko 2024-03-24T01:10:53.440 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_serverworks.ko 2024-03-24T01:10:53.440 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sil680.ko 2024-03-24T01:10:53.440 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sis.ko 2024-03-24T01:10:53.440 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sl82c105.ko 2024-03-24T01:10:53.440 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_triflex.ko 2024-03-24T01:10:53.440 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_via.ko 2024-03-24T01:10:53.440 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pdc_adma.ko 2024-03-24T01:10:53.440 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_inic162x.ko 2024-03-24T01:10:53.440 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_mv.ko 2024-03-24T01:10:53.440 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_nv.ko 2024-03-24T01:10:53.440 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_promise.ko 2024-03-24T01:10:53.440 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_qstor.ko 2024-03-24T01:10:53.440 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil.ko 2024-03-24T01:10:53.440 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil24.ko 2024-03-24T01:10:53.441 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sis.ko 2024-03-24T01:10:53.441 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_svw.ko 2024-03-24T01:10:53.441 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sx4.ko 2024-03-24T01:10:53.441 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_uli.ko 2024-03-24T01:10:53.441 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_via.ko 2024-03-24T01:10:53.441 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_vsc.ko 2024-03-24T01:10:53.441 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay 2024-03-24T01:10:53.441 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay/ks0108.ko 2024-03-24T01:10:53.441 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base 2024-03-24T01:10:53.441 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap 2024-03-24T01:10:53.441 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-24T01:10:53.441 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-24T01:10:53.441 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma 2024-03-24T01:10:53.441 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma/bcma.ko 2024-03-24T01:10:53.441 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block 2024-03-24T01:10:53.442 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe 2024-03-24T01:10:53.442 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe/aoe.ko 2024-03-24T01:10:53.442 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/brd.ko 2024-03-24T01:10:53.442 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd 2024-03-24T01:10:53.442 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd/drbd.ko 2024-03-24T01:10:53.442 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/floppy.ko 2024-03-24T01:10:53.442 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/loop.ko 2024-03-24T01:10:53.442 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx 2024-03-24T01:10:53.442 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-24T01:10:53.442 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/nbd.ko 2024-03-24T01:10:53.442 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk 2024-03-24T01:10:53.442 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk/null_blk.ko 2024-03-24T01:10:53.442 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/pktcdvd.ko 2024-03-24T01:10:53.442 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/rbd.ko 2024-03-24T01:10:53.442 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/virtio_blk.ko 2024-03-24T01:10:53.443 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback 2024-03-24T01:10:53.443 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-24T01:10:53.443 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkfront.ko 2024-03-24T01:10:53.443 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram 2024-03-24T01:10:53.443 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram/zram.ko 2024-03-24T01:10:53.443 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char 2024-03-24T01:10:53.443 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hangcheck-timer.ko 2024-03-24T01:10:53.443 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random 2024-03-24T01:10:53.443 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-24T01:10:53.443 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-24T01:10:53.443 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-24T01:10:53.443 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/via-rng.ko 2024-03-24T01:10:53.443 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-24T01:10:53.443 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi 2024-03-24T01:10:53.444 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-24T01:10:53.444 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-24T01:10:53.444 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-24T01:10:53.444 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-24T01:10:53.444 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-24T01:10:53.444 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-24T01:10:53.444 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/lp.ko 2024-03-24T01:10:53.444 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave 2024-03-24T01:10:53.444 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave/mwave.ko 2024-03-24T01:10:53.444 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ppdev.ko 2024-03-24T01:10:53.444 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tlclk.ko 2024-03-24T01:10:53.444 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm 2024-03-24T01:10:53.444 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm.ko 2024-03-24T01:10:53.444 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-24T01:10:53.444 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-24T01:10:53.445 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-24T01:10:53.445 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-24T01:10:53.445 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-24T01:10:53.445 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/uv_mmtimer.ko 2024-03-24T01:10:53.445 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/virtio_console.ko 2024-03-24T01:10:53.445 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq 2024-03-24T01:10:53.445 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-24T01:10:53.445 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-24T01:10:53.445 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-24T01:10:53.445 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-24T01:10:53.445 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-24T01:10:53.445 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-24T01:10:53.445 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto 2024-03-24T01:10:53.445 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp 2024-03-24T01:10:53.446 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-24T01:10:53.446 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp.ko 2024-03-24T01:10:53.446 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel 2024-03-24T01:10:53.446 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat 2024-03-24T01:10:53.446 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common 2024-03-24T01:10:53.446 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-24T01:10:53.446 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-24T01:10:53.446 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-24T01:10:53.446 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-aes.ko 2024-03-24T01:10:53.446 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-sha.ko 2024-03-24T01:10:53.446 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio 2024-03-24T01:10:53.446 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-24T01:10:53.446 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca 2024-03-24T01:10:53.446 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca/dca.ko 2024-03-24T01:10:53.446 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/devfreq 2024-03-24T01:10:53.447 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-24T01:10:53.447 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma 2024-03-24T01:10:53.447 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw 2024-03-24T01:10:53.447 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-24T01:10:53.447 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat 2024-03-24T01:10:53.447 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-24T01:10:53.447 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac 2024-03-24T01:10:53.447 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/amd64_edac.ko 2024-03-24T01:10:53.447 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/e752x_edac.ko 2024-03-24T01:10:53.447 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/edac_mce_amd.ko 2024-03-24T01:10:53.447 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3000_edac.ko 2024-03-24T01:10:53.447 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3200_edac.ko 2024-03-24T01:10:53.447 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5100_edac.ko 2024-03-24T01:10:53.447 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5400_edac.ko 2024-03-24T01:10:53.448 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7300_edac.ko 2024-03-24T01:10:53.448 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7core_edac.ko 2024-03-24T01:10:53.448 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i82975x_edac.ko 2024-03-24T01:10:53.448 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/ie31200_edac.ko 2024-03-24T01:10:53.448 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/sb_edac.ko 2024-03-24T01:10:53.448 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/x38_edac.ko 2024-03-24T01:10:53.448 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware 2024-03-24T01:10:53.448 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/edd.ko 2024-03-24T01:10:53.448 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-24T01:10:53.448 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio 2024-03-24T01:10:53.448 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-ich.ko 2024-03-24T01:10:53.448 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-24T01:10:53.448 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid 2024-03-24T01:10:53.448 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-appleir.ko 2024-03-24T01:10:53.448 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-aureal.ko 2024-03-24T01:10:53.449 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-axff.ko 2024-03-24T01:10:53.449 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-dr.ko 2024-03-24T01:10:53.449 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elecom.ko 2024-03-24T01:10:53.449 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elo.ko 2024-03-24T01:10:53.449 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-emsff.ko 2024-03-24T01:10:53.449 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gaff.ko 2024-03-24T01:10:53.449 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gt683r.ko 2024-03-24T01:10:53.449 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gyration.ko 2024-03-24T01:10:53.449 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-24T01:10:53.449 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-24T01:10:53.449 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtekff.ko 2024-03-24T01:10:53.449 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-hyperv.ko 2024-03-24T01:10:53.449 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-icade.ko 2024-03-24T01:10:53.449 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-keytouch.ko 2024-03-24T01:10:53.450 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-kye.ko 2024-03-24T01:10:53.450 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lcpower.ko 2024-03-24T01:10:53.450 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-led.ko 2024-03-24T01:10:53.450 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lenovo.ko 2024-03-24T01:10:53.450 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-24T01:10:53.450 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-24T01:10:53.450 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-multitouch.ko 2024-03-24T01:10:53.450 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-ortek.ko 2024-03-24T01:10:53.450 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-penmount.ko 2024-03-24T01:10:53.450 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-petalynx.ko 2024-03-24T01:10:53.450 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-picolcd.ko 2024-03-24T01:10:53.450 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-pl.ko 2024-03-24T01:10:53.450 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-plantronics.ko 2024-03-24T01:10:53.450 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-primax.ko 2024-03-24T01:10:53.450 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-rmi.ko 2024-03-24T01:10:53.451 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-24T01:10:53.451 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-common.ko 2024-03-24T01:10:53.451 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-24T01:10:53.451 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-24T01:10:53.451 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-24T01:10:53.451 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-24T01:10:53.451 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-24T01:10:53.451 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-24T01:10:53.451 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-24T01:10:53.451 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-24T01:10:53.451 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-24T01:10:53.451 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat.ko 2024-03-24T01:10:53.451 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-saitek.ko 2024-03-24T01:10:53.451 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-samsung.ko 2024-03-24T01:10:53.451 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-24T01:10:53.452 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sjoy.ko 2024-03-24T01:10:53.452 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sony.ko 2024-03-24T01:10:53.452 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-speedlink.ko 2024-03-24T01:10:53.452 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-steelseries.ko 2024-03-24T01:10:53.452 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sunplus.ko 2024-03-24T01:10:53.452 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-24T01:10:53.452 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tivo.ko 2024-03-24T01:10:53.452 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tmff.ko 2024-03-24T01:10:53.452 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-topseed.ko 2024-03-24T01:10:53.452 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-twinhan.ko 2024-03-24T01:10:53.452 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-uclogic.ko 2024-03-24T01:10:53.452 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-waltop.ko 2024-03-24T01:10:53.452 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-wiimote.ko 2024-03-24T01:10:53.452 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-xinmo.ko 2024-03-24T01:10:53.453 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zpff.ko 2024-03-24T01:10:53.453 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zydacron.ko 2024-03-24T01:10:53.453 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/uhid.ko 2024-03-24T01:10:53.453 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/wacom.ko 2024-03-24T01:10:53.453 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv 2024-03-24T01:10:53.453 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_balloon.ko 2024-03-24T01:10:53.453 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_utils.ko 2024-03-24T01:10:53.453 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_vmbus.ko 2024-03-24T01:10:53.453 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon 2024-03-24T01:10:53.453 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru.ko 2024-03-24T01:10:53.453 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru3.ko 2024-03-24T01:10:53.453 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-24T01:10:53.453 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7414.ko 2024-03-24T01:10:53.453 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7418.ko 2024-03-24T01:10:53.453 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adc128d818.ko 2024-03-24T01:10:53.454 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1025.ko 2024-03-24T01:10:53.454 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1026.ko 2024-03-24T01:10:53.454 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1029.ko 2024-03-24T01:10:53.454 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1031.ko 2024-03-24T01:10:53.454 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm9240.ko 2024-03-24T01:10:53.454 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ads7828.ko 2024-03-24T01:10:53.454 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7410.ko 2024-03-24T01:10:53.454 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7411.ko 2024-03-24T01:10:53.454 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7462.ko 2024-03-24T01:10:53.454 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7470.ko 2024-03-24T01:10:53.454 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7475.ko 2024-03-24T01:10:53.454 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7x10.ko 2024-03-24T01:10:53.454 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/amc6821.ko 2024-03-24T01:10:53.454 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/applesmc.ko 2024-03-24T01:10:53.454 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asb100.ko 2024-03-24T01:10:53.455 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asc7621.ko 2024-03-24T01:10:53.455 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-24T01:10:53.455 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/atxp1.ko 2024-03-24T01:10:53.455 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/coretemp.ko 2024-03-24T01:10:53.455 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-24T01:10:53.455 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dme1737.ko 2024-03-24T01:10:53.455 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds1621.ko 2024-03-24T01:10:53.455 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds620.ko 2024-03-24T01:10:53.455 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc1403.ko 2024-03-24T01:10:53.455 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc6w201.ko 2024-03-24T01:10:53.455 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71805f.ko 2024-03-24T01:10:53.455 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71882fg.ko 2024-03-24T01:10:53.455 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f75375s.ko 2024-03-24T01:10:53.455 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fam15h_power.ko 2024-03-24T01:10:53.456 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fschmd.ko 2024-03-24T01:10:53.456 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g760a.ko 2024-03-24T01:10:53.456 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g762.ko 2024-03-24T01:10:53.456 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl518sm.ko 2024-03-24T01:10:53.456 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl520sm.ko 2024-03-24T01:10:53.456 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-24T01:10:53.456 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5500_temp.ko 2024-03-24T01:10:53.456 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5k_amb.ko 2024-03-24T01:10:53.456 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmaem.ko 2024-03-24T01:10:53.456 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmpex.ko 2024-03-24T01:10:53.456 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina209.ko 2024-03-24T01:10:53.456 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina2xx.ko 2024-03-24T01:10:53.456 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/it87.ko 2024-03-24T01:10:53.456 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k10temp.ko 2024-03-24T01:10:53.456 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k8temp.ko 2024-03-24T01:10:53.457 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lineage-pem.ko 2024-03-24T01:10:53.457 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm63.ko 2024-03-24T01:10:53.457 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm73.ko 2024-03-24T01:10:53.457 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm75.ko 2024-03-24T01:10:53.457 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm77.ko 2024-03-24T01:10:53.457 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm78.ko 2024-03-24T01:10:53.457 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm80.ko 2024-03-24T01:10:53.457 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm83.ko 2024-03-24T01:10:53.457 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm85.ko 2024-03-24T01:10:53.457 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm87.ko 2024-03-24T01:10:53.457 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm90.ko 2024-03-24T01:10:53.457 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm92.ko 2024-03-24T01:10:53.457 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm93.ko 2024-03-24T01:10:53.457 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95234.ko 2024-03-24T01:10:53.458 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95241.ko 2024-03-24T01:10:53.458 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95245.ko 2024-03-24T01:10:53.458 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc2945.ko 2024-03-24T01:10:53.458 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4151.ko 2024-03-24T01:10:53.458 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4215.ko 2024-03-24T01:10:53.458 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4222.ko 2024-03-24T01:10:53.458 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4245.ko 2024-03-24T01:10:53.458 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4260.ko 2024-03-24T01:10:53.458 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4261.ko 2024-03-24T01:10:53.458 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max16065.ko 2024-03-24T01:10:53.458 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1619.ko 2024-03-24T01:10:53.458 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1668.ko 2024-03-24T01:10:53.458 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max197.ko 2024-03-24T01:10:53.458 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6639.ko 2024-03-24T01:10:53.458 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6650.ko 2024-03-24T01:10:53.459 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6697.ko 2024-03-24T01:10:53.459 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/mcp3021.ko 2024-03-24T01:10:53.459 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6683.ko 2024-03-24T01:10:53.459 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775-core.ko 2024-03-24T01:10:53.459 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775.ko 2024-03-24T01:10:53.459 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct7802.ko 2024-03-24T01:10:53.459 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87360.ko 2024-03-24T01:10:53.459 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87427.ko 2024-03-24T01:10:53.459 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pcf8591.ko 2024-03-24T01:10:53.459 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus 2024-03-24T01:10:53.459 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-24T01:10:53.459 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-24T01:10:53.459 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-24T01:10:53.459 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-24T01:10:53.460 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-24T01:10:53.460 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-24T01:10:53.460 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-24T01:10:53.460 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-24T01:10:53.460 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-24T01:10:53.460 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-24T01:10:53.460 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-24T01:10:53.460 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-24T01:10:53.460 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/powr1220.ko 2024-03-24T01:10:53.460 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5627.ko 2024-03-24T01:10:53.460 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5636.ko 2024-03-24T01:10:53.460 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-24T01:10:53.460 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht15.ko 2024-03-24T01:10:53.460 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht21.ko 2024-03-24T01:10:53.460 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/shtc1.ko 2024-03-24T01:10:53.461 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sis5595.ko 2024-03-24T01:10:53.461 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47b397.ko 2024-03-24T01:10:53.461 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m1.ko 2024-03-24T01:10:53.461 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m192.ko 2024-03-24T01:10:53.461 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/thmc50.ko 2024-03-24T01:10:53.461 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp102.ko 2024-03-24T01:10:53.461 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp103.ko 2024-03-24T01:10:53.461 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp401.ko 2024-03-24T01:10:53.461 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp421.ko 2024-03-24T01:10:53.461 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via-cputemp.ko 2024-03-24T01:10:53.461 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via686a.ko 2024-03-24T01:10:53.461 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt1211.ko 2024-03-24T01:10:53.461 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt8231.ko 2024-03-24T01:10:53.461 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627ehf.ko 2024-03-24T01:10:53.461 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627hf.ko 2024-03-24T01:10:53.462 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83781d.ko 2024-03-24T01:10:53.462 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83791d.ko 2024-03-24T01:10:53.462 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83792d.ko 2024-03-24T01:10:53.462 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83793.ko 2024-03-24T01:10:53.462 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83795.ko 2024-03-24T01:10:53.462 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l785ts.ko 2024-03-24T01:10:53.462 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l786ng.ko 2024-03-24T01:10:53.462 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c 2024-03-24T01:10:53.462 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos 2024-03-24T01:10:53.462 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-24T01:10:53.462 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-24T01:10:53.462 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses 2024-03-24T01:10:53.462 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-24T01:10:53.462 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-24T01:10:53.462 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-24T01:10:53.463 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-24T01:10:53.463 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-24T01:10:53.463 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-24T01:10:53.463 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-24T01:10:53.463 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-24T01:10:53.463 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-24T01:10:53.463 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-24T01:10:53.463 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-24T01:10:53.463 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-24T01:10:53.463 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-24T01:10:53.463 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-24T01:10:53.463 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-24T01:10:53.463 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-24T01:10:53.463 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-24T01:10:53.464 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-24T01:10:53.464 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-24T01:10:53.464 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-24T01:10:53.464 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-24T01:10:53.464 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-24T01:10:53.464 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-24T01:10:53.464 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-dev.ko 2024-03-24T01:10:53.464 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-mux.ko 2024-03-24T01:10:53.464 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-24T01:10:53.464 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-smbus.ko 2024-03-24T01:10:53.464 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-stub.ko 2024-03-24T01:10:53.464 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband 2024-03-24T01:10:53.464 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core 2024-03-24T01:10:53.464 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-24T01:10:53.464 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_core.ko 2024-03-24T01:10:53.465 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-24T01:10:53.465 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-24T01:10:53.465 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-24T01:10:53.465 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-24T01:10:53.465 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-24T01:10:53.465 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw 2024-03-24T01:10:53.465 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4 2024-03-24T01:10:53.465 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-24T01:10:53.465 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4 2024-03-24T01:10:53.465 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-24T01:10:53.465 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5 2024-03-24T01:10:53.465 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-24T01:10:53.465 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca 2024-03-24T01:10:53.465 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-24T01:10:53.466 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp 2024-03-24T01:10:53.466 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib 2024-03-24T01:10:53.466 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-24T01:10:53.466 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser 2024-03-24T01:10:53.466 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-24T01:10:53.466 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert 2024-03-24T01:10:53.466 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-24T01:10:53.466 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp 2024-03-24T01:10:53.466 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-24T01:10:53.466 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt 2024-03-24T01:10:53.466 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-24T01:10:53.466 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input 2024-03-24T01:10:53.466 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport 2024-03-24T01:10:53.466 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-24T01:10:53.467 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-24T01:10:53.467 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/gameport.ko 2024-03-24T01:10:53.467 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/joydev.ko 2024-03-24T01:10:53.467 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard 2024-03-24T01:10:53.467 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-24T01:10:53.467 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse 2024-03-24T01:10:53.467 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/appletouch.ko 2024-03-24T01:10:53.467 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/bcm5974.ko 2024-03-24T01:10:53.467 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/cyapatp.ko 2024-03-24T01:10:53.467 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-24T01:10:53.467 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/sermouse.ko 2024-03-24T01:10:53.467 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-24T01:10:53.467 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-24T01:10:53.467 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-24T01:10:53.467 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4 2024-03-24T01:10:53.468 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-24T01:10:53.468 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio 2024-03-24T01:10:53.468 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/altera_ps2.ko 2024-03-24T01:10:53.468 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/arc_ps2.ko 2024-03-24T01:10:53.468 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-24T01:10:53.468 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/serio_raw.ko 2024-03-24T01:10:53.468 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/sparse-keymap.ko 2024-03-24T01:10:53.468 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds 2024-03-24T01:10:53.468 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-blinkm.ko 2024-03-24T01:10:53.468 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lm3530.ko 2024-03-24T01:10:53.468 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lp3944.ko 2024-03-24T01:10:53.468 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-ss4200.ko 2024-03-24T01:10:53.468 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger 2024-03-24T01:10:53.468 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-24T01:10:53.468 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-24T01:10:53.469 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-24T01:10:53.469 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-24T01:10:53.469 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-24T01:10:53.469 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-24T01:10:53.469 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-24T01:10:53.469 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-24T01:10:53.469 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-24T01:10:53.469 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md 2024-03-24T01:10:53.469 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache 2024-03-24T01:10:53.469 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache/bcache.ko 2024-03-24T01:10:53.469 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-bio-prison.ko 2024-03-24T01:10:53.469 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache-smq.ko 2024-03-24T01:10:53.469 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache.ko 2024-03-24T01:10:53.469 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-crypt.ko 2024-03-24T01:10:53.470 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-delay.ko 2024-03-24T01:10:53.470 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-flakey.ko 2024-03-24T01:10:53.470 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-log-userspace.ko 2024-03-24T01:10:53.470 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-multipath.ko 2024-03-24T01:10:53.470 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-queue-length.ko 2024-03-24T01:10:53.470 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-raid.ko 2024-03-24T01:10:53.470 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-round-robin.ko 2024-03-24T01:10:53.470 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-service-time.ko 2024-03-24T01:10:53.470 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-switch.ko 2024-03-24T01:10:53.470 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-thin-pool.ko 2024-03-24T01:10:53.470 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-verity.ko 2024-03-24T01:10:53.470 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data 2024-03-24T01:10:53.470 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-24T01:10:53.470 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid0.ko 2024-03-24T01:10:53.470 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid1.ko 2024-03-24T01:10:53.471 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid10.ko 2024-03-24T01:10:53.471 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid456.ko 2024-03-24T01:10:53.471 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message 2024-03-24T01:10:53.471 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion 2024-03-24T01:10:53.471 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptbase.ko 2024-03-24T01:10:53.471 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptctl.ko 2024-03-24T01:10:53.471 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptfc.ko 2024-03-24T01:10:53.471 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptlan.ko 2024-03-24T01:10:53.471 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptsas.ko 2024-03-24T01:10:53.471 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptscsih.ko 2024-03-24T01:10:53.471 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptspi.ko 2024-03-24T01:10:53.471 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd 2024-03-24T01:10:53.471 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_ich.ko 2024-03-24T01:10:53.471 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_sch.ko 2024-03-24T01:10:53.471 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/mfd-core.ko 2024-03-24T01:10:53.472 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/sm501.ko 2024-03-24T01:10:53.472 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/viperboard.ko 2024-03-24T01:10:53.472 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/vx855.ko 2024-03-24T01:10:53.472 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/wl1273-core.ko 2024-03-24T01:10:53.472 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc 2024-03-24T01:10:53.472 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl 2024-03-24T01:10:53.472 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-24T01:10:53.472 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds9802als.ko 2024-03-24T01:10:53.472 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds990x.ko 2024-03-24T01:10:53.472 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/bh1770glc.ko 2024-03-24T01:10:53.472 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710 2024-03-24T01:10:53.472 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710/cb710.ko 2024-03-24T01:10:53.472 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom 2024-03-24T01:10:53.472 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/at24.ko 2024-03-24T01:10:53.472 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-24T01:10:53.473 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/max6875.ko 2024-03-24T01:10:53.473 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/enclosure.ko 2024-03-24T01:10:53.473 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/hpilo.ko 2024-03-24T01:10:53.473 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29003.ko 2024-03-24T01:10:53.473 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29020.ko 2024-03-24T01:10:53.473 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d 2024-03-24T01:10:53.473 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-24T01:10:53.473 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-24T01:10:53.473 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei 2024-03-24T01:10:53.473 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-me.ko 2024-03-24T01:10:53.473 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-txe.ko 2024-03-24T01:10:53.473 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei.ko 2024-03-24T01:10:53.473 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-gru 2024-03-24T01:10:53.473 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-24T01:10:53.474 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp 2024-03-24T01:10:53.474 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-24T01:10:53.474 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-24T01:10:53.474 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-24T01:10:53.474 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_7xx1.ko 2024-03-24T01:10:53.474 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_core.ko 2024-03-24T01:10:53.474 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tsl2550.ko 2024-03-24T01:10:53.474 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_balloon.ko 2024-03-24T01:10:53.474 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci 2024-03-24T01:10:53.474 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-24T01:10:53.474 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net 2024-03-24T01:10:53.474 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding 2024-03-24T01:10:53.474 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding/bonding.ko 2024-03-24T01:10:53.474 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa 2024-03-24T01:10:53.474 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53 2024-03-24T01:10:53.475 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-24T01:10:53.475 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-24T01:10:53.475 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-24T01:10:53.475 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx 2024-03-24T01:10:53.475 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-24T01:10:53.475 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dummy.ko 2024-03-24T01:10:53.475 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/eql.ko 2024-03-24T01:10:53.475 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet 2024-03-24T01:10:53.475 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com 2024-03-24T01:10:53.475 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-24T01:10:53.475 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-24T01:10:53.475 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-24T01:10:53.475 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-24T01:10:53.475 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390 2024-03-24T01:10:53.476 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-24T01:10:53.476 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-24T01:10:53.476 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-24T01:10:53.476 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-24T01:10:53.476 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec 2024-03-24T01:10:53.476 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-24T01:10:53.476 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere 2024-03-24T01:10:53.476 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-24T01:10:53.476 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon 2024-03-24T01:10:53.476 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-24T01:10:53.476 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera 2024-03-24T01:10:53.476 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-24T01:10:53.476 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd 2024-03-24T01:10:53.476 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-24T01:10:53.476 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-24T01:10:53.477 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-24T01:10:53.477 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros 2024-03-24T01:10:53.477 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx 2024-03-24T01:10:53.477 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-24T01:10:53.477 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-24T01:10:53.477 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-24T01:10:53.477 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-24T01:10:53.477 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-24T01:10:53.477 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx 2024-03-24T01:10:53.477 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-24T01:10:53.477 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-24T01:10:53.477 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom 2024-03-24T01:10:53.477 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-24T01:10:53.477 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-24T01:10:53.477 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-24T01:10:53.478 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-24T01:10:53.478 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-24T01:10:53.478 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet 2024-03-24T01:10:53.478 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-24T01:10:53.478 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-24T01:10:53.478 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade 2024-03-24T01:10:53.478 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna 2024-03-24T01:10:53.478 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-24T01:10:53.478 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio 2024-03-24T01:10:53.478 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-24T01:10:53.478 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-24T01:10:53.478 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-24T01:10:53.478 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-24T01:10:53.478 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-24T01:10:53.479 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-24T01:10:53.479 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-24T01:10:53.479 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-24T01:10:53.479 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-24T01:10:53.479 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-24T01:10:53.479 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco 2024-03-24T01:10:53.479 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic 2024-03-24T01:10:53.479 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-24T01:10:53.479 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec 2024-03-24T01:10:53.479 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip 2024-03-24T01:10:53.479 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-24T01:10:53.479 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-24T01:10:53.479 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-24T01:10:53.479 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-24T01:10:53.479 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-24T01:10:53.480 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-24T01:10:53.480 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink 2024-03-24T01:10:53.480 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-24T01:10:53.480 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-24T01:10:53.480 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dnet.ko 2024-03-24T01:10:53.480 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex 2024-03-24T01:10:53.480 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet 2024-03-24T01:10:53.480 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-24T01:10:53.480 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ethoc.ko 2024-03-24T01:10:53.480 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/fealnx.ko 2024-03-24T01:10:53.480 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel 2024-03-24T01:10:53.480 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-24T01:10:53.480 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000 2024-03-24T01:10:53.480 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-24T01:10:53.481 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e 2024-03-24T01:10:53.481 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-24T01:10:53.481 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k 2024-03-24T01:10:53.481 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-24T01:10:53.481 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e 2024-03-24T01:10:53.481 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-24T01:10:53.481 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf 2024-03-24T01:10:53.481 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-24T01:10:53.481 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb 2024-03-24T01:10:53.481 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-24T01:10:53.481 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf 2024-03-24T01:10:53.481 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-24T01:10:53.481 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-24T01:10:53.481 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-24T01:10:53.481 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-24T01:10:53.482 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-24T01:10:53.482 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/jme.ko 2024-03-24T01:10:53.482 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell 2024-03-24T01:10:53.482 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-24T01:10:53.482 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-24T01:10:53.482 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-24T01:10:53.482 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox 2024-03-24T01:10:53.482 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-24T01:10:53.482 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-24T01:10:53.482 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-24T01:10:53.482 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-24T01:10:53.482 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-24T01:10:53.482 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-24T01:10:53.482 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel 2024-03-24T01:10:53.483 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-24T01:10:53.483 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom 2024-03-24T01:10:53.483 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-24T01:10:53.483 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-24T01:10:53.483 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi 2024-03-24T01:10:53.483 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-24T01:10:53.483 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-24T01:10:53.483 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion 2024-03-24T01:10:53.483 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-24T01:10:53.483 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia 2024-03-24T01:10:53.483 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-24T01:10:53.483 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines 2024-03-24T01:10:53.483 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-24T01:10:53.483 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-24T01:10:53.483 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic 2024-03-24T01:10:53.484 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-24T01:10:53.484 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-24T01:10:53.484 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-24T01:10:53.484 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-24T01:10:53.484 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-24T01:10:53.484 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc 2024-03-24T01:10:53.484 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-24T01:10:53.484 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek 2024-03-24T01:10:53.484 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-24T01:10:53.484 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-24T01:10:53.484 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-24T01:10:53.484 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-24T01:10:53.484 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rocker 2024-03-24T01:10:53.484 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-24T01:10:53.485 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc 2024-03-24T01:10:53.485 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-24T01:10:53.485 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan 2024-03-24T01:10:53.485 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-24T01:10:53.485 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis 2024-03-24T01:10:53.485 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-24T01:10:53.485 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-24T01:10:53.485 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc 2024-03-24T01:10:53.485 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-24T01:10:53.485 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-24T01:10:53.485 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-24T01:10:53.485 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-24T01:10:53.485 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro 2024-03-24T01:10:53.485 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-24T01:10:53.485 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-24T01:10:53.486 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-24T01:10:53.486 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun 2024-03-24T01:10:53.486 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-24T01:10:53.486 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-24T01:10:53.486 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-24T01:10:53.486 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-24T01:10:53.486 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti 2024-03-24T01:10:53.486 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-24T01:10:53.486 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti 2024-03-24T01:10:53.486 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-24T01:10:53.486 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via 2024-03-24T01:10:53.486 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-24T01:10:53.486 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-24T01:10:53.486 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet 2024-03-24T01:10:53.486 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-24T01:10:53.487 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-24T01:10:53.487 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom 2024-03-24T01:10:53.487 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-24T01:10:53.487 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/geneve.ko 2024-03-24T01:10:53.487 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv 2024-03-24T01:10:53.487 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-24T01:10:53.487 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154 2024-03-24T01:10:53.487 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-24T01:10:53.487 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ifb.ko 2024-03-24T01:10:53.487 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan 2024-03-24T01:10:53.487 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-24T01:10:53.487 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvlan.ko 2024-03-24T01:10:53.487 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvtap.ko 2024-03-24T01:10:53.487 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio 2024-03-24T01:10:53.488 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio.ko 2024-03-24T01:10:53.488 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-24T01:10:53.488 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-24T01:10:53.488 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-24T01:10:53.488 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mii.ko 2024-03-24T01:10:53.488 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/net_failover.ko 2024-03-24T01:10:53.488 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/netconsole.ko 2024-03-24T01:10:53.488 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/nlmon.ko 2024-03-24T01:10:53.488 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs 2024-03-24T01:10:53.488 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-24T01:10:53.488 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-24T01:10:53.488 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy 2024-03-24T01:10:53.488 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/amd.ko 2024-03-24T01:10:53.488 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-24T01:10:53.488 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-24T01:10:53.489 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-24T01:10:53.489 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm87xx.ko 2024-03-24T01:10:53.489 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/broadcom.ko 2024-03-24T01:10:53.489 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/cicada.ko 2024-03-24T01:10:53.489 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/davicom.ko 2024-03-24T01:10:53.489 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/dp83640.ko 2024-03-24T01:10:53.489 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/et1011c.ko 2024-03-24T01:10:53.489 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/icplus.ko 2024-03-24T01:10:53.489 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/lxt.ko 2024-03-24T01:10:53.489 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/marvell.ko 2024-03-24T01:10:53.489 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/micrel.ko 2024-03-24T01:10:53.489 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/national.ko 2024-03-24T01:10:53.489 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/phylink.ko 2024-03-24T01:10:53.489 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/qsemi.ko 2024-03-24T01:10:53.490 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/realtek.ko 2024-03-24T01:10:53.490 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/smsc.ko 2024-03-24T01:10:53.490 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/ste10Xp.ko 2024-03-24T01:10:53.490 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/vitesse.ko 2024-03-24T01:10:53.490 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp 2024-03-24T01:10:53.490 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-24T01:10:53.490 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_async.ko 2024-03-24T01:10:53.490 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-24T01:10:53.490 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-24T01:10:53.490 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-24T01:10:53.490 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-24T01:10:53.490 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppoe.ko 2024-03-24T01:10:53.490 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppox.ko 2024-03-24T01:10:53.490 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pptp.ko 2024-03-24T01:10:53.490 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip 2024-03-24T01:10:53.491 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/slhc.ko 2024-03-24T01:10:53.491 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/slip.ko 2024-03-24T01:10:53.491 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/sungem_phy.ko 2024-03-24T01:10:53.491 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/tap.ko 2024-03-24T01:10:53.491 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team 2024-03-24T01:10:53.491 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team.ko 2024-03-24T01:10:53.491 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-24T01:10:53.491 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-24T01:10:53.491 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-24T01:10:53.491 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_random.ko 2024-03-24T01:10:53.491 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-24T01:10:53.491 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/tun.ko 2024-03-24T01:10:53.491 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/veth.ko 2024-03-24T01:10:53.491 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/virtio_net.ko 2024-03-24T01:10:53.492 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3 2024-03-24T01:10:53.492 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-24T01:10:53.492 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan 2024-03-24T01:10:53.492 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan/vxlan.ko 2024-03-24T01:10:53.492 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback 2024-03-24T01:10:53.492 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-24T01:10:53.492 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netfront.ko 2024-03-24T01:10:53.492 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb 2024-03-24T01:10:53.492 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb/ntb.ko 2024-03-24T01:10:53.492 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme 2024-03-24T01:10:53.492 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host 2024-03-24T01:10:53.492 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme-core.ko 2024-03-24T01:10:53.492 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme.ko 2024-03-24T01:10:53.492 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport 2024-03-24T01:10:53.492 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport.ko 2024-03-24T01:10:53.493 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_cs.ko 2024-03-24T01:10:53.493 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_pc.ko 2024-03-24T01:10:53.493 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_serial.ko 2024-03-24T01:10:53.493 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci 2024-03-24T01:10:53.493 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug 2024-03-24T01:10:53.493 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-24T01:10:53.493 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/pcie 2024-03-24T01:10:53.493 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-24T01:10:53.493 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/xen-pcifront.ko 2024-03-24T01:10:53.493 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia 2024-03-24T01:10:53.493 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/i82092.ko 2024-03-24T01:10:53.493 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/pd6729.ko 2024-03-24T01:10:53.493 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-24T01:10:53.493 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl 2024-03-24T01:10:53.493 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel 2024-03-24T01:10:53.494 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-24T01:10:53.494 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform 2024-03-24T01:10:53.494 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86 2024-03-24T01:10:53.494 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-24T01:10:53.494 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acerhdf.ko 2024-03-24T01:10:53.494 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-24T01:10:53.494 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-24T01:10:53.494 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-24T01:10:53.494 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-24T01:10:53.494 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-24T01:10:53.494 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-24T01:10:53.494 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-24T01:10:53.494 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-24T01:10:53.494 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-24T01:10:53.495 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/hdaps.ko 2024-03-24T01:10:53.495 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel 2024-03-24T01:10:53.495 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-24T01:10:53.495 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel_ips.ko 2024-03-24T01:10:53.495 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-24T01:10:53.495 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-24T01:10:53.495 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-24T01:10:53.495 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-24T01:10:53.495 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-24T01:10:53.495 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-24T01:10:53.495 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-24T01:10:53.495 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-24T01:10:53.495 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-24T01:10:53.495 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi.ko 2024-03-24T01:10:53.495 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap 2024-03-24T01:10:53.496 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-24T01:10:53.496 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-24T01:10:53.496 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps 2024-03-24T01:10:53.496 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients 2024-03-24T01:10:53.496 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-24T01:10:53.496 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-24T01:10:53.496 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps_parport.ko 2024-03-24T01:10:53.496 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/pps_core.ko 2024-03-24T01:10:53.496 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp 2024-03-24T01:10:53.496 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp.ko 2024-03-24T01:10:53.496 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp_kvm.ko 2024-03-24T01:10:53.496 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc 2024-03-24T01:10:53.496 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-24T01:10:53.496 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-24T01:10:53.496 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-24T01:10:53.497 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-24T01:10:53.497 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-24T01:10:53.497 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-24T01:10:53.497 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-24T01:10:53.497 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-24T01:10:53.497 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-24T01:10:53.497 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-24T01:10:53.497 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-em3027.ko 2024-03-24T01:10:53.497 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-24T01:10:53.497 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-24T01:10:53.497 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-24T01:10:53.497 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-24T01:10:53.497 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-24T01:10:53.497 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-24T01:10:53.498 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-max6900.ko 2024-03-24T01:10:53.498 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-24T01:10:53.498 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-24T01:10:53.498 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-24T01:10:53.498 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-24T01:10:53.498 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-24T01:10:53.498 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-24T01:10:53.498 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-24T01:10:53.498 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-24T01:10:53.498 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-24T01:10:53.498 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-24T01:10:53.498 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-24T01:10:53.498 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-24T01:10:53.498 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-x1205.ko 2024-03-24T01:10:53.498 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi 2024-03-24T01:10:53.499 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-9xxx.ko 2024-03-24T01:10:53.499 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-sas.ko 2024-03-24T01:10:53.499 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-xxxx.ko 2024-03-24T01:10:53.499 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/BusLogic.ko 2024-03-24T01:10:53.499 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/a100u2w.ko 2024-03-24T01:10:53.499 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid 2024-03-24T01:10:53.499 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-24T01:10:53.499 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/advansys.ko 2024-03-24T01:10:53.499 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx 2024-03-24T01:10:53.499 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-24T01:10:53.499 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-24T01:10:53.499 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/am53c974.ko 2024-03-24T01:10:53.499 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr 2024-03-24T01:10:53.499 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-24T01:10:53.500 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/atp870u.ko 2024-03-24T01:10:53.500 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi 2024-03-24T01:10:53.500 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-24T01:10:53.500 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa 2024-03-24T01:10:53.500 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa/bfa.ko 2024-03-24T01:10:53.500 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc 2024-03-24T01:10:53.500 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-24T01:10:53.500 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i 2024-03-24T01:10:53.500 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-24T01:10:53.500 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ch.ko 2024-03-24T01:10:53.500 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor 2024-03-24T01:10:53.500 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-24T01:10:53.500 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi 2024-03-24T01:10:53.500 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-24T01:10:53.500 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-24T01:10:53.501 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-24T01:10:53.501 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-24T01:10:53.501 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-24T01:10:53.501 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dc395x.ko 2024-03-24T01:10:53.501 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler 2024-03-24T01:10:53.501 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-24T01:10:53.501 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-24T01:10:53.501 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-24T01:10:53.501 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-24T01:10:53.501 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dmx3191d.ko 2024-03-24T01:10:53.501 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r 2024-03-24T01:10:53.501 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-24T01:10:53.501 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esp_scsi.ko 2024-03-24T01:10:53.501 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe 2024-03-24T01:10:53.501 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-24T01:10:53.502 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-24T01:10:53.502 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic 2024-03-24T01:10:53.502 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic/fnic.ko 2024-03-24T01:10:53.502 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hpsa.ko 2024-03-24T01:10:53.502 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hptiop.ko 2024-03-24T01:10:53.502 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hv_storvsc.ko 2024-03-24T01:10:53.502 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/initio.ko 2024-03-24T01:10:53.502 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ipr.ko 2024-03-24T01:10:53.502 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ips.ko 2024-03-24T01:10:53.502 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci 2024-03-24T01:10:53.502 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci/isci.ko 2024-03-24T01:10:53.502 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-24T01:10:53.502 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-24T01:10:53.502 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc 2024-03-24T01:10:53.502 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc/libfc.ko 2024-03-24T01:10:53.503 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi.ko 2024-03-24T01:10:53.503 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-24T01:10:53.503 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas 2024-03-24T01:10:53.503 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas/libsas.ko 2024-03-24T01:10:53.503 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc 2024-03-24T01:10:53.503 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-24T01:10:53.503 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid 2024-03-24T01:10:53.503 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid.ko 2024-03-24T01:10:53.503 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-24T01:10:53.503 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-24T01:10:53.503 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-24T01:10:53.503 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas 2024-03-24T01:10:53.503 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-24T01:10:53.503 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas 2024-03-24T01:10:53.504 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-24T01:10:53.504 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvumi.ko 2024-03-24T01:10:53.504 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001 2024-03-24T01:10:53.504 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-24T01:10:53.504 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pmcraid.ko 2024-03-24T01:10:53.504 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla1280.ko 2024-03-24T01:10:53.504 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx 2024-03-24T01:10:53.504 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-24T01:10:53.504 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-24T01:10:53.504 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx 2024-03-24T01:10:53.504 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-24T01:10:53.504 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/raid_class.ko 2024-03-24T01:10:53.504 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_debug.ko 2024-03-24T01:10:53.504 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-24T01:10:53.504 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-24T01:10:53.505 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-24T01:10:53.505 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-24T01:10:53.505 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-24T01:10:53.505 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ses.ko 2024-03-24T01:10:53.505 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/st.ko 2024-03-24T01:10:53.505 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/stex.ko 2024-03-24T01:10:53.505 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2 2024-03-24T01:10:53.505 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-24T01:10:53.505 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/virtio_scsi.ko 2024-03-24T01:10:53.505 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-24T01:10:53.505 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/wd719x.ko 2024-03-24T01:10:53.505 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb 2024-03-24T01:10:53.505 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb/ssb.ko 2024-03-24T01:10:53.505 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target 2024-03-24T01:10:53.506 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi 2024-03-24T01:10:53.506 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-24T01:10:53.506 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback 2024-03-24T01:10:53.506 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-24T01:10:53.506 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_file.ko 2024-03-24T01:10:53.506 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_iblock.ko 2024-03-24T01:10:53.506 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_mod.ko 2024-03-24T01:10:53.506 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_pscsi.ko 2024-03-24T01:10:53.506 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_user.ko 2024-03-24T01:10:53.506 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc 2024-03-24T01:10:53.506 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-24T01:10:53.506 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal 2024-03-24T01:10:53.506 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel 2024-03-24T01:10:53.506 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal 2024-03-24T01:10:53.506 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-24T01:10:53.507 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-24T01:10:53.507 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-24T01:10:53.507 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-24T01:10:53.507 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-24T01:10:53.507 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-24T01:10:53.507 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-24T01:10:53.507 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-24T01:10:53.507 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-24T01:10:53.507 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-24T01:10:53.507 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-24T01:10:53.507 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-24T01:10:53.507 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-24T01:10:53.507 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-24T01:10:53.507 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-24T01:10:53.508 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-24T01:10:53.508 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-24T01:10:53.508 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-24T01:10:53.508 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty 2024-03-24T01:10:53.508 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless 2024-03-24T01:10:53.508 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-24T01:10:53.508 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_gsm.ko 2024-03-24T01:10:53.508 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_hdlc.ko 2024-03-24T01:10:53.508 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/nozomi.ko 2024-03-24T01:10:53.508 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial 2024-03-24T01:10:53.508 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250 2024-03-24T01:10:53.508 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-24T01:10:53.508 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/arc_uart.ko 2024-03-24T01:10:53.508 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm 2024-03-24T01:10:53.508 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-24T01:10:53.509 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs 2024-03-24T01:10:53.509 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core 2024-03-24T01:10:53.509 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-24T01:10:53.509 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host 2024-03-24T01:10:53.509 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-24T01:10:53.509 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio 2024-03-24T01:10:53.509 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio.ko 2024-03-24T01:10:53.509 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_aec.ko 2024-03-24T01:10:53.509 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_cif.ko 2024-03-24T01:10:53.509 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_pci_generic.ko 2024-03-24T01:10:53.509 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_sercos3.ko 2024-03-24T01:10:53.509 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb 2024-03-24T01:10:53.509 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class 2024-03-24T01:10:53.509 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-acm.ko 2024-03-24T01:10:53.509 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-24T01:10:53.510 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usblp.ko 2024-03-24T01:10:53.510 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usbtmc.ko 2024-03-24T01:10:53.510 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host 2024-03-24T01:10:53.510 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-24T01:10:53.510 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image 2024-03-24T01:10:53.510 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/mdc800.ko 2024-03-24T01:10:53.510 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/microtek.ko 2024-03-24T01:10:53.510 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc 2024-03-24T01:10:53.510 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/adutux.ko 2024-03-24T01:10:53.510 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/appledisplay.ko 2024-03-24T01:10:53.510 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi26.ko 2024-03-24T01:10:53.510 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi62.ko 2024-03-24T01:10:53.510 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ezusb.ko 2024-03-24T01:10:53.510 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/idmouse.ko 2024-03-24T01:10:53.511 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/iowarrior.ko 2024-03-24T01:10:53.511 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-24T01:10:53.511 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ldusb.ko 2024-03-24T01:10:53.511 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/legousbtower.ko 2024-03-24T01:10:53.511 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga 2024-03-24T01:10:53.511 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-24T01:10:53.511 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-24T01:10:53.511 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usb3503.ko 2024-03-24T01:10:53.511 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usblcd.ko 2024-03-24T01:10:53.511 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-24T01:10:53.511 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/uss720.ko 2024-03-24T01:10:53.511 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/yurex.ko 2024-03-24T01:10:53.511 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy 2024-03-24T01:10:53.511 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy/phy-generic.ko 2024-03-24T01:10:53.511 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial 2024-03-24T01:10:53.512 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/aircable.ko 2024-03-24T01:10:53.512 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ark3116.ko 2024-03-24T01:10:53.512 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-24T01:10:53.512 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ch341.ko 2024-03-24T01:10:53.512 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cp210x.ko 2024-03-24T01:10:53.512 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cyberjack.ko 2024-03-24T01:10:53.512 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-24T01:10:53.512 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-24T01:10:53.512 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/empeg.ko 2024-03-24T01:10:53.512 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-24T01:10:53.512 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-24T01:10:53.512 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-24T01:10:53.512 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_ti.ko 2024-03-24T01:10:53.512 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipaq.ko 2024-03-24T01:10:53.513 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipw.ko 2024-03-24T01:10:53.513 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ir-usb.ko 2024-03-24T01:10:53.513 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-24T01:10:53.513 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan.ko 2024-03-24T01:10:53.513 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-24T01:10:53.513 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-24T01:10:53.513 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-24T01:10:53.513 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mct_u232.ko 2024-03-24T01:10:53.513 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7720.ko 2024-03-24T01:10:53.513 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7840.ko 2024-03-24T01:10:53.513 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/navman.ko 2024-03-24T01:10:53.513 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/omninet.ko 2024-03-24T01:10:53.513 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/opticon.ko 2024-03-24T01:10:53.513 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/option.ko 2024-03-24T01:10:53.513 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/oti6858.ko 2024-03-24T01:10:53.514 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/pl2303.ko 2024-03-24T01:10:53.514 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcaux.ko 2024-03-24T01:10:53.514 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcserial.ko 2024-03-24T01:10:53.514 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/quatech2.ko 2024-03-24T01:10:53.514 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/safe_serial.ko 2024-03-24T01:10:53.514 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/sierra.ko 2024-03-24T01:10:53.514 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-24T01:10:53.514 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ssu100.ko 2024-03-24T01:10:53.514 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/symbolserial.ko 2024-03-24T01:10:53.514 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-24T01:10:53.514 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-24T01:10:53.514 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_debug.ko 2024-03-24T01:10:53.514 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-24T01:10:53.514 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/visor.ko 2024-03-24T01:10:53.515 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/whiteheat.ko 2024-03-24T01:10:53.515 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-24T01:10:53.515 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage 2024-03-24T01:10:53.515 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/uas.ko 2024-03-24T01:10:53.515 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-24T01:10:53.515 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-24T01:10:53.515 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-24T01:10:53.515 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-24T01:10:53.515 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-24T01:10:53.515 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-24T01:10:53.515 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-24T01:10:53.515 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-karma.ko 2024-03-24T01:10:53.515 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-24T01:10:53.515 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-24T01:10:53.515 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-24T01:10:53.516 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-24T01:10:53.516 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-24T01:10:53.516 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/usb-storage.ko 2024-03-24T01:10:53.516 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip 2024-03-24T01:10:53.516 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-24T01:10:53.516 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-24T01:10:53.516 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-24T01:10:53.516 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio 2024-03-24T01:10:53.516 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci 2024-03-24T01:10:53.516 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-24T01:10:53.516 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-24T01:10:53.516 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio.ko 2024-03-24T01:10:53.516 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-24T01:10:53.516 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost 2024-03-24T01:10:53.517 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost.ko 2024-03-24T01:10:53.517 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-24T01:10:53.517 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_net.ko 2024-03-24T01:10:53.517 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_scsi.ko 2024-03-24T01:10:53.517 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video 2024-03-24T01:10:53.517 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight 2024-03-24T01:10:53.517 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/apple_bl.ko 2024-03-24T01:10:53.517 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/lcd.ko 2024-03-24T01:10:53.517 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-24T01:10:53.517 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio 2024-03-24T01:10:53.517 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio.ko 2024-03-24T01:10:53.517 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_balloon.ko 2024-03-24T01:10:53.517 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_mmio.ko 2024-03-24T01:10:53.517 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci.ko 2024-03-24T01:10:53.517 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-24T01:10:53.518 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-24T01:10:53.518 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_ring.ko 2024-03-24T01:10:53.518 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1 2024-03-24T01:10:53.518 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters 2024-03-24T01:10:53.518 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2482.ko 2024-03-24T01:10:53.518 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2490.ko 2024-03-24T01:10:53.518 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves 2024-03-24T01:10:53.518 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-24T01:10:53.518 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-24T01:10:53.518 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-24T01:10:53.518 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-24T01:10:53.518 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-24T01:10:53.518 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-24T01:10:53.518 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-24T01:10:53.519 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-24T01:10:53.519 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-24T01:10:53.519 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-24T01:10:53.519 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-24T01:10:53.519 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/wire.ko 2024-03-24T01:10:53.519 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog 2024-03-24T01:10:53.519 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-24T01:10:53.519 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-24T01:10:53.519 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-24T01:10:53.519 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/hpwdt.ko 2024-03-24T01:10:53.519 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/i6300esb.ko 2024-03-24T01:10:53.519 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-24T01:10:53.519 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-24T01:10:53.519 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ib700wdt.ko 2024-03-24T01:10:53.519 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ibmasr.ko 2024-03-24T01:10:53.520 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-24T01:10:53.520 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-24T01:10:53.520 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it87_wdt.ko 2024-03-24T01:10:53.520 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/machzwd.ko 2024-03-24T01:10:53.520 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/nv_tco.ko 2024-03-24T01:10:53.520 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-24T01:10:53.520 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-24T01:10:53.520 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-24T01:10:53.520 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-24T01:10:53.520 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/softdog.ko 2024-03-24T01:10:53.520 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-24T01:10:53.520 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/via_wdt.ko 2024-03-24T01:10:53.520 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-24T01:10:53.520 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-24T01:10:53.521 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-24T01:10:53.521 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/wdt_pci.ko 2024-03-24T01:10:53.521 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/xen_wdt.ko 2024-03-24T01:10:53.521 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen 2024-03-24T01:10:53.521 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-24T01:10:53.521 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-evtchn.ko 2024-03-24T01:10:53.521 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntalloc.ko 2024-03-24T01:10:53.521 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntdev.ko 2024-03-24T01:10:53.521 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback 2024-03-24T01:10:53.521 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-24T01:10:53.521 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-privcmd.ko 2024-03-24T01:10:53.521 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs 2024-03-24T01:10:53.521 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-24T01:10:53.521 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs 2024-03-24T01:10:53.521 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p 2024-03-24T01:10:53.522 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p/9p.ko 2024-03-24T01:10:53.522 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs 2024-03-24T01:10:53.522 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs/affs.ko 2024-03-24T01:10:53.522 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs 2024-03-24T01:10:53.522 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs/befs.ko 2024-03-24T01:10:53.522 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/binfmt_misc.ko 2024-03-24T01:10:53.522 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs 2024-03-24T01:10:53.522 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs/btrfs.ko 2024-03-24T01:10:53.522 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles 2024-03-24T01:10:53.522 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles/cachefiles.ko 2024-03-24T01:10:53.522 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph 2024-03-24T01:10:53.522 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph/ceph.ko 2024-03-24T01:10:53.522 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda 2024-03-24T01:10:53.522 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda/coda.ko 2024-03-24T01:10:53.522 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs 2024-03-24T01:10:53.523 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs/cramfs.ko 2024-03-24T01:10:53.523 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm 2024-03-24T01:10:53.523 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm/dlm.ko 2024-03-24T01:10:53.523 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ecryptfs 2024-03-24T01:10:53.523 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-24T01:10:53.523 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs 2024-03-24T01:10:53.523 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs/f2fs.ko 2024-03-24T01:10:53.523 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat 2024-03-24T01:10:53.523 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/fat.ko 2024-03-24T01:10:53.523 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/msdos.ko 2024-03-24T01:10:53.523 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/vfat.ko 2024-03-24T01:10:53.523 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse 2024-03-24T01:10:53.523 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/cuse.ko 2024-03-24T01:10:53.523 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/fuse.ko 2024-03-24T01:10:53.524 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2 2024-03-24T01:10:53.524 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2/gfs2.ko 2024-03-24T01:10:53.524 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs 2024-03-24T01:10:53.524 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs/hfs.ko 2024-03-24T01:10:53.524 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus 2024-03-24T01:10:53.524 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus/hfsplus.ko 2024-03-24T01:10:53.524 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs 2024-03-24T01:10:53.524 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs/isofs.ko 2024-03-24T01:10:53.524 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs 2024-03-24T01:10:53.524 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs/jfs.ko 2024-03-24T01:10:53.524 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd 2024-03-24T01:10:53.524 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd/lockd.ko 2024-03-24T01:10:53.524 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix 2024-03-24T01:10:53.524 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix/minix.ko 2024-03-24T01:10:53.524 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs 2024-03-24T01:10:53.525 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs/netfs.ko 2024-03-24T01:10:53.525 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs 2024-03-24T01:10:53.525 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout 2024-03-24T01:10:53.525 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-24T01:10:53.525 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout 2024-03-24T01:10:53.525 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-24T01:10:53.525 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout 2024-03-24T01:10:53.525 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-24T01:10:53.525 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfs.ko 2024-03-24T01:10:53.525 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv3.ko 2024-03-24T01:10:53.525 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv4.ko 2024-03-24T01:10:53.525 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common 2024-03-24T01:10:53.525 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/grace.ko 2024-03-24T01:10:53.525 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/nfs_acl.ko 2024-03-24T01:10:53.525 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd 2024-03-24T01:10:53.526 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd/nfsd.ko 2024-03-24T01:10:53.526 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2 2024-03-24T01:10:53.526 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2/nilfs2.ko 2024-03-24T01:10:53.526 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls 2024-03-24T01:10:53.526 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-celtic.ko 2024-03-24T01:10:53.526 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-centeuro.ko 2024-03-24T01:10:53.526 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-croatian.ko 2024-03-24T01:10:53.526 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-cyrillic.ko 2024-03-24T01:10:53.526 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-gaelic.ko 2024-03-24T01:10:53.526 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-greek.ko 2024-03-24T01:10:53.526 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-iceland.ko 2024-03-24T01:10:53.526 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-inuit.ko 2024-03-24T01:10:53.526 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-roman.ko 2024-03-24T01:10:53.526 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-romanian.ko 2024-03-24T01:10:53.527 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-turkish.ko 2024-03-24T01:10:53.527 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1250.ko 2024-03-24T01:10:53.527 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1251.ko 2024-03-24T01:10:53.527 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1255.ko 2024-03-24T01:10:53.527 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp737.ko 2024-03-24T01:10:53.527 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp775.ko 2024-03-24T01:10:53.527 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp850.ko 2024-03-24T01:10:53.527 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp852.ko 2024-03-24T01:10:53.527 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp855.ko 2024-03-24T01:10:53.527 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp857.ko 2024-03-24T01:10:53.527 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp860.ko 2024-03-24T01:10:53.527 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp861.ko 2024-03-24T01:10:53.527 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp862.ko 2024-03-24T01:10:53.527 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp863.ko 2024-03-24T01:10:53.527 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp864.ko 2024-03-24T01:10:53.528 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp865.ko 2024-03-24T01:10:53.528 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp866.ko 2024-03-24T01:10:53.528 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp869.ko 2024-03-24T01:10:53.528 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp874.ko 2024-03-24T01:10:53.528 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp932.ko 2024-03-24T01:10:53.528 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp936.ko 2024-03-24T01:10:53.528 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp949.ko 2024-03-24T01:10:53.528 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp950.ko 2024-03-24T01:10:53.528 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_euc-jp.ko 2024-03-24T01:10:53.528 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-1.ko 2024-03-24T01:10:53.528 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-13.ko 2024-03-24T01:10:53.528 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-14.ko 2024-03-24T01:10:53.528 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-15.ko 2024-03-24T01:10:53.528 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-2.ko 2024-03-24T01:10:53.528 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-3.ko 2024-03-24T01:10:53.529 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-4.ko 2024-03-24T01:10:53.529 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-5.ko 2024-03-24T01:10:53.529 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-6.ko 2024-03-24T01:10:53.529 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-7.ko 2024-03-24T01:10:53.529 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-9.ko 2024-03-24T01:10:53.529 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-r.ko 2024-03-24T01:10:53.529 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-ru.ko 2024-03-24T01:10:53.529 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-u.ko 2024-03-24T01:10:53.529 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-24T01:10:53.529 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_utf8.ko 2024-03-24T01:10:53.529 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2 2024-03-24T01:10:53.529 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster 2024-03-24T01:10:53.529 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-24T01:10:53.529 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm 2024-03-24T01:10:53.529 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-24T01:10:53.530 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs 2024-03-24T01:10:53.530 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-24T01:10:53.530 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2.ko 2024-03-24T01:10:53.530 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-24T01:10:53.530 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-24T01:10:53.530 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-24T01:10:53.530 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs 2024-03-24T01:10:53.530 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs/overlay.ko 2024-03-24T01:10:53.530 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore 2024-03-24T01:10:53.530 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore/ramoops.ko 2024-03-24T01:10:53.530 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs 2024-03-24T01:10:53.530 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs/reiserfs.ko 2024-03-24T01:10:53.530 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs 2024-03-24T01:10:53.530 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs/romfs.ko 2024-03-24T01:10:53.531 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb 2024-03-24T01:10:53.531 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client 2024-03-24T01:10:53.531 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client/cifs.ko 2024-03-24T01:10:53.531 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common 2024-03-24T01:10:53.531 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_arc4.ko 2024-03-24T01:10:53.531 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_md4.ko 2024-03-24T01:10:53.531 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs 2024-03-24T01:10:53.531 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs/squashfs.ko 2024-03-24T01:10:53.531 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv 2024-03-24T01:10:53.531 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv/sysv.ko 2024-03-24T01:10:53.531 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf 2024-03-24T01:10:53.531 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf/udf.ko 2024-03-24T01:10:53.531 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs 2024-03-24T01:10:53.531 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs/ufs.ko 2024-03-24T01:10:53.531 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs 2024-03-24T01:10:53.532 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs/xfs.ko 2024-03-24T01:10:53.532 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel 2024-03-24T01:10:53.532 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/rcu 2024-03-24T01:10:53.532 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/rcu/rcutorture.ko 2024-03-24T01:10:53.532 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/torture.ko 2024-03-24T01:10:53.532 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/trace 2024-03-24T01:10:53.532 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-24T01:10:53.532 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib 2024-03-24T01:10:53.532 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/asn1_encoder.ko 2024-03-24T01:10:53.532 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc-itu-t.ko 2024-03-24T01:10:53.532 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc8.ko 2024-03-24T01:10:53.532 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto 2024-03-24T01:10:53.532 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libarc4.ko 2024-03-24T01:10:53.532 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libdes.ko 2024-03-24T01:10:53.532 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/libcrc32c.ko 2024-03-24T01:10:53.533 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lru_cache.ko 2024-03-24T01:10:53.533 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4 2024-03-24T01:10:53.533 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4_compress.ko 2024-03-24T01:10:53.533 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4hc_compress.ko 2024-03-24T01:10:53.533 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math 2024-03-24T01:10:53.533 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math/cordic.ko 2024-03-24T01:10:53.533 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6 2024-03-24T01:10:53.533 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6/raid6_pq.ko 2024-03-24T01:10:53.533 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon 2024-03-24T01:10:53.533 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-24T01:10:53.533 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_bm.ko 2024-03-24T01:10:53.533 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_fsm.ko 2024-03-24T01:10:53.533 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_kmp.ko 2024-03-24T01:10:53.533 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/zstd 2024-03-24T01:10:53.533 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/zstd/zstd_compress.ko 2024-03-24T01:10:53.534 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm 2024-03-24T01:10:53.534 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm/hwpoison-inject.ko 2024-03-24T01:10:53.534 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net 2024-03-24T01:10:53.534 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan 2024-03-24T01:10:53.534 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/6lowpan.ko 2024-03-24T01:10:53.534 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_dest.ko 2024-03-24T01:10:53.534 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_fragment.ko 2024-03-24T01:10:53.534 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_hop.ko 2024-03-24T01:10:53.534 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-24T01:10:53.534 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_mobility.ko 2024-03-24T01:10:53.534 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_routing.ko 2024-03-24T01:10:53.534 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_udp.ko 2024-03-24T01:10:53.534 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802 2024-03-24T01:10:53.534 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/garp.ko 2024-03-24T01:10:53.535 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/mrp.ko 2024-03-24T01:10:53.535 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/p8022.ko 2024-03-24T01:10:53.535 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/psnap.ko 2024-03-24T01:10:53.535 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/stp.ko 2024-03-24T01:10:53.535 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q 2024-03-24T01:10:53.535 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q/8021q.ko 2024-03-24T01:10:53.535 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p 2024-03-24T01:10:53.535 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet.ko 2024-03-24T01:10:53.535 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_fd.ko 2024-03-24T01:10:53.535 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_rdma.ko 2024-03-24T01:10:53.535 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_virtio.ko 2024-03-24T01:10:53.535 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv 2024-03-24T01:10:53.535 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv/batman-adv.ko 2024-03-24T01:10:53.535 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge 2024-03-24T01:10:53.535 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/br_netfilter.ko 2024-03-24T01:10:53.536 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/bridge.ko 2024-03-24T01:10:53.536 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter 2024-03-24T01:10:53.536 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-24T01:10:53.536 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-24T01:10:53.536 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-24T01:10:53.536 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-24T01:10:53.536 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-24T01:10:53.536 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-24T01:10:53.536 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-24T01:10:53.536 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-24T01:10:53.536 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-24T01:10:53.536 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-24T01:10:53.536 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-24T01:10:53.536 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-24T01:10:53.536 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-24T01:10:53.537 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-24T01:10:53.537 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-24T01:10:53.537 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-24T01:10:53.537 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-24T01:10:53.537 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-24T01:10:53.537 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-24T01:10:53.537 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-24T01:10:53.537 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtables.ko 2024-03-24T01:10:53.537 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-24T01:10:53.537 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-24T01:10:53.537 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-24T01:10:53.537 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph 2024-03-24T01:10:53.537 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph/libceph.ko 2024-03-24T01:10:53.537 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core 2024-03-24T01:10:53.537 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/failover.ko 2024-03-24T01:10:53.538 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/pktgen.ko 2024-03-24T01:10:53.538 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp 2024-03-24T01:10:53.538 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp.ko 2024-03-24T01:10:53.538 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_diag.ko 2024-03-24T01:10:53.538 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv4.ko 2024-03-24T01:10:53.538 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv6.ko 2024-03-24T01:10:53.538 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dns_resolver 2024-03-24T01:10:53.538 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dns_resolver/dns_resolver.ko 2024-03-24T01:10:53.538 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa 2024-03-24T01:10:53.538 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/dsa_core.ko 2024-03-24T01:10:53.538 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_brcm.ko 2024-03-24T01:10:53.538 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_dsa.ko 2024-03-24T01:10:53.538 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_none.ko 2024-03-24T01:10:53.538 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_trailer.ko 2024-03-24T01:10:53.538 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154 2024-03-24T01:10:53.539 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan 2024-03-24T01:10:53.539 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-24T01:10:53.539 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154.ko 2024-03-24T01:10:53.539 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-24T01:10:53.539 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4 2024-03-24T01:10:53.539 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ah4.ko 2024-03-24T01:10:53.539 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/esp4.ko 2024-03-24T01:10:53.539 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/fou.ko 2024-03-24T01:10:53.539 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/gre.ko 2024-03-24T01:10:53.539 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/inet_diag.ko 2024-03-24T01:10:53.539 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_gre.ko 2024-03-24T01:10:53.539 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_tunnel.ko 2024-03-24T01:10:53.539 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_vti.ko 2024-03-24T01:10:53.539 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipcomp.ko 2024-03-24T01:10:53.540 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipip.ko 2024-03-24T01:10:53.540 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter 2024-03-24T01:10:53.540 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-24T01:10:53.540 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-24T01:10:53.540 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-24T01:10:53.540 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-24T01:10:53.540 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-24T01:10:53.540 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-24T01:10:53.540 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-24T01:10:53.540 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-24T01:10:53.540 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-24T01:10:53.540 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-24T01:10:53.540 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-24T01:10:53.540 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-24T01:10:53.540 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-24T01:10:53.541 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-24T01:10:53.541 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-24T01:10:53.541 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-24T01:10:53.541 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-24T01:10:53.541 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-24T01:10:53.541 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-24T01:10:53.541 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-24T01:10:53.541 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-24T01:10:53.541 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_bic.ko 2024-03-24T01:10:53.541 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_dctcp.ko 2024-03-24T01:10:53.541 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_diag.ko 2024-03-24T01:10:53.541 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_highspeed.ko 2024-03-24T01:10:53.541 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_htcp.ko 2024-03-24T01:10:53.541 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_hybla.ko 2024-03-24T01:10:53.541 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_illinois.ko 2024-03-24T01:10:53.542 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_lp.ko 2024-03-24T01:10:53.542 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_scalable.ko 2024-03-24T01:10:53.542 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_vegas.ko 2024-03-24T01:10:53.542 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_veno.ko 2024-03-24T01:10:53.542 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_westwood.ko 2024-03-24T01:10:53.542 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_yeah.ko 2024-03-24T01:10:53.542 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tunnel4.ko 2024-03-24T01:10:53.542 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_diag.ko 2024-03-24T01:10:53.542 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_tunnel.ko 2024-03-24T01:10:53.542 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-24T01:10:53.542 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6 2024-03-24T01:10:53.542 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ah6.ko 2024-03-24T01:10:53.542 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/esp6.ko 2024-03-24T01:10:53.542 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/fou6.ko 2024-03-24T01:10:53.543 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_tunnel.ko 2024-03-24T01:10:53.543 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-24T01:10:53.543 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_vti.ko 2024-03-24T01:10:53.543 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ipcomp6.ko 2024-03-24T01:10:53.543 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter 2024-03-24T01:10:53.543 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-24T01:10:53.543 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-24T01:10:53.543 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-24T01:10:53.543 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-24T01:10:53.543 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-24T01:10:53.543 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-24T01:10:53.543 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-24T01:10:53.543 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-24T01:10:53.543 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-24T01:10:53.543 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-24T01:10:53.544 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-24T01:10:53.544 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-24T01:10:53.544 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-24T01:10:53.544 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-24T01:10:53.544 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-24T01:10:53.544 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-24T01:10:53.544 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-24T01:10:53.544 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-24T01:10:53.544 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-24T01:10:53.544 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-24T01:10:53.544 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-24T01:10:53.544 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-24T01:10:53.544 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-24T01:10:53.544 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-24T01:10:53.544 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/sit.ko 2024-03-24T01:10:53.545 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/tunnel6.ko 2024-03-24T01:10:53.545 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-24T01:10:53.545 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key 2024-03-24T01:10:53.545 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key/af_key.ko 2024-03-24T01:10:53.545 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp 2024-03-24T01:10:53.545 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_core.ko 2024-03-24T01:10:53.545 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-24T01:10:53.545 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_eth.ko 2024-03-24T01:10:53.545 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip.ko 2024-03-24T01:10:53.545 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip6.ko 2024-03-24T01:10:53.545 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_netlink.ko 2024-03-24T01:10:53.545 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ppp.ko 2024-03-24T01:10:53.545 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc 2024-03-24T01:10:53.545 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc/llc.ko 2024-03-24T01:10:53.545 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154 2024-03-24T01:10:53.546 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154/mac802154.ko 2024-03-24T01:10:53.546 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls 2024-03-24T01:10:53.546 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls/mpls_gso.ko 2024-03-24T01:10:53.546 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter 2024-03-24T01:10:53.546 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset 2024-03-24T01:10:53.546 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set.ko 2024-03-24T01:10:53.546 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-24T01:10:53.546 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-24T01:10:53.546 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-24T01:10:53.546 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-24T01:10:53.546 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-24T01:10:53.546 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-24T01:10:53.546 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-24T01:10:53.546 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-24T01:10:53.546 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-24T01:10:53.547 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-24T01:10:53.547 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-24T01:10:53.547 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-24T01:10:53.547 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-24T01:10:53.547 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-24T01:10:53.547 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-24T01:10:53.547 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs 2024-03-24T01:10:53.547 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-24T01:10:53.547 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-24T01:10:53.547 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-24T01:10:53.547 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-24T01:10:53.547 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-24T01:10:53.547 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-24T01:10:53.547 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-24T01:10:53.548 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-24T01:10:53.548 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-24T01:10:53.548 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-24T01:10:53.548 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-24T01:10:53.548 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-24T01:10:53.548 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-24T01:10:53.548 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-24T01:10:53.548 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conncount.ko 2024-03-24T01:10:53.548 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack.ko 2024-03-24T01:10:53.548 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-24T01:10:53.548 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-24T01:10:53.548 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-24T01:10:53.548 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-24T01:10:53.548 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-24T01:10:53.548 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-24T01:10:53.549 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-24T01:10:53.549 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-24T01:10:53.549 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-24T01:10:53.549 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-24T01:10:53.549 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-24T01:10:53.549 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-24T01:10:53.549 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-24T01:10:53.549 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table.ko 2024-03-24T01:10:53.549 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-24T01:10:53.549 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_log_syslog.ko 2024-03-24T01:10:53.549 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat.ko 2024-03-24T01:10:53.549 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-24T01:10:53.549 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-24T01:10:53.549 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_irc.ko 2024-03-24T01:10:53.549 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_sip.ko 2024-03-24T01:10:53.550 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-24T01:10:53.550 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-24T01:10:53.550 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_tables.ko 2024-03-24T01:10:53.550 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink.ko 2024-03-24T01:10:53.550 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-24T01:10:53.550 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_log.ko 2024-03-24T01:10:53.550 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-24T01:10:53.550 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-24T01:10:53.550 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_chain_nat.ko 2024-03-24T01:10:53.550 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_compat.ko 2024-03-24T01:10:53.550 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_ct.ko 2024-03-24T01:10:53.550 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-24T01:10:53.550 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fib.ko 2024-03-24T01:10:53.550 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-24T01:10:53.551 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_hash.ko 2024-03-24T01:10:53.551 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_limit.ko 2024-03-24T01:10:53.551 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_log.ko 2024-03-24T01:10:53.551 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_masq.ko 2024-03-24T01:10:53.551 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_nat.ko 2024-03-24T01:10:53.551 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_numgen.ko 2024-03-24T01:10:53.551 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_queue.ko 2024-03-24T01:10:53.551 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_quota.ko 2024-03-24T01:10:53.551 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_redir.ko 2024-03-24T01:10:53.551 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject.ko 2024-03-24T01:10:53.551 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject_inet.ko 2024-03-24T01:10:53.551 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_socket.ko 2024-03-24T01:10:53.551 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_tunnel.ko 2024-03-24T01:10:53.551 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_AUDIT.ko 2024-03-24T01:10:53.551 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-24T01:10:53.552 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-24T01:10:53.552 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-24T01:10:53.552 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CT.ko 2024-03-24T01:10:53.552 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_DSCP.ko 2024-03-24T01:10:53.552 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HL.ko 2024-03-24T01:10:53.552 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HMARK.ko 2024-03-24T01:10:53.552 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-24T01:10:53.552 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LED.ko 2024-03-24T01:10:53.552 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LOG.ko 2024-03-24T01:10:53.552 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-24T01:10:53.552 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NETMAP.ko 2024-03-24T01:10:53.552 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFLOG.ko 2024-03-24T01:10:53.552 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-24T01:10:53.552 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_RATEEST.ko 2024-03-24T01:10:53.552 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-24T01:10:53.553 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_SECMARK.ko 2024-03-24T01:10:53.553 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-24T01:10:53.553 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-24T01:10:53.553 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TEE.ko 2024-03-24T01:10:53.553 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TPROXY.ko 2024-03-24T01:10:53.553 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TRACE.ko 2024-03-24T01:10:53.553 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_addrtype.ko 2024-03-24T01:10:53.553 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_bpf.ko 2024-03-24T01:10:53.553 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cgroup.ko 2024-03-24T01:10:53.553 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cluster.ko 2024-03-24T01:10:53.553 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_comment.ko 2024-03-24T01:10:53.553 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connbytes.ko 2024-03-24T01:10:53.553 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlabel.ko 2024-03-24T01:10:53.553 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlimit.ko 2024-03-24T01:10:53.553 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connmark.ko 2024-03-24T01:10:53.554 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_conntrack.ko 2024-03-24T01:10:53.554 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cpu.ko 2024-03-24T01:10:53.554 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dccp.ko 2024-03-24T01:10:53.554 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_devgroup.ko 2024-03-24T01:10:53.554 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dscp.ko 2024-03-24T01:10:53.554 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ecn.ko 2024-03-24T01:10:53.554 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_esp.ko 2024-03-24T01:10:53.554 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hashlimit.ko 2024-03-24T01:10:53.554 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_helper.ko 2024-03-24T01:10:53.554 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hl.ko 2024-03-24T01:10:53.554 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipcomp.ko 2024-03-24T01:10:53.554 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_iprange.ko 2024-03-24T01:10:53.554 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipvs.ko 2024-03-24T01:10:53.554 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_l2tp.ko 2024-03-24T01:10:53.555 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_length.ko 2024-03-24T01:10:53.555 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_limit.ko 2024-03-24T01:10:53.555 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mac.ko 2024-03-24T01:10:53.555 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mark.ko 2024-03-24T01:10:53.555 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_multiport.ko 2024-03-24T01:10:53.555 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nat.ko 2024-03-24T01:10:53.555 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nfacct.ko 2024-03-24T01:10:53.555 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_osf.ko 2024-03-24T01:10:53.555 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_owner.ko 2024-03-24T01:10:53.555 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_physdev.ko 2024-03-24T01:10:53.555 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_pkttype.ko 2024-03-24T01:10:53.555 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_policy.ko 2024-03-24T01:10:53.555 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_quota.ko 2024-03-24T01:10:53.555 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_rateest.ko 2024-03-24T01:10:53.555 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_realm.ko 2024-03-24T01:10:53.556 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_recent.ko 2024-03-24T01:10:53.556 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_sctp.ko 2024-03-24T01:10:53.556 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_set.ko 2024-03-24T01:10:53.556 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_socket.ko 2024-03-24T01:10:53.556 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_state.ko 2024-03-24T01:10:53.556 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_statistic.ko 2024-03-24T01:10:53.556 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_string.ko 2024-03-24T01:10:53.556 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_tcpmss.ko 2024-03-24T01:10:53.556 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_time.ko 2024-03-24T01:10:53.556 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_u32.ko 2024-03-24T01:10:53.556 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink 2024-03-24T01:10:53.556 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink/netlink_diag.ko 2024-03-24T01:10:53.556 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh 2024-03-24T01:10:53.556 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh/nsh.ko 2024-03-24T01:10:53.556 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch 2024-03-24T01:10:53.557 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/openvswitch.ko 2024-03-24T01:10:53.557 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-geneve.ko 2024-03-24T01:10:53.557 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-gre.ko 2024-03-24T01:10:53.557 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-vxlan.ko 2024-03-24T01:10:53.557 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet 2024-03-24T01:10:53.557 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet/af_packet_diag.ko 2024-03-24T01:10:53.557 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds 2024-03-24T01:10:53.557 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds.ko 2024-03-24T01:10:53.557 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_rdma.ko 2024-03-24T01:10:53.557 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_tcp.ko 2024-03-24T01:10:53.557 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched 2024-03-24T01:10:53.557 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_csum.ko 2024-03-24T01:10:53.557 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_gact.ko 2024-03-24T01:10:53.557 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_mirred.ko 2024-03-24T01:10:53.558 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_nat.ko 2024-03-24T01:10:53.558 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_pedit.ko 2024-03-24T01:10:53.558 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_police.ko 2024-03-24T01:10:53.558 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_simple.ko 2024-03-24T01:10:53.558 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_skbedit.ko 2024-03-24T01:10:53.558 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_vlan.ko 2024-03-24T01:10:53.558 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_basic.ko 2024-03-24T01:10:53.558 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_bpf.ko 2024-03-24T01:10:53.558 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_flow.ko 2024-03-24T01:10:53.558 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_fw.ko 2024-03-24T01:10:53.558 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_route.ko 2024-03-24T01:10:53.558 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_u32.ko 2024-03-24T01:10:53.558 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_cmp.ko 2024-03-24T01:10:53.558 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_ipset.ko 2024-03-24T01:10:53.559 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_meta.ko 2024-03-24T01:10:53.559 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_nbyte.ko 2024-03-24T01:10:53.559 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_text.ko 2024-03-24T01:10:53.559 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_u32.ko 2024-03-24T01:10:53.559 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_choke.ko 2024-03-24T01:10:53.559 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_codel.ko 2024-03-24T01:10:53.559 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_drr.ko 2024-03-24T01:10:53.559 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_fq.ko 2024-03-24T01:10:53.559 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_gred.ko 2024-03-24T01:10:53.559 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hfsc.ko 2024-03-24T01:10:53.559 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hhf.ko 2024-03-24T01:10:53.559 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_htb.ko 2024-03-24T01:10:53.559 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_ingress.ko 2024-03-24T01:10:53.559 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio.ko 2024-03-24T01:10:53.560 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio_lib.ko 2024-03-24T01:10:53.560 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_multiq.ko 2024-03-24T01:10:53.560 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_netem.ko 2024-03-24T01:10:53.560 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_pie.ko 2024-03-24T01:10:53.560 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_plug.ko 2024-03-24T01:10:53.560 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_prio.ko 2024-03-24T01:10:53.560 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_qfq.ko 2024-03-24T01:10:53.560 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_red.ko 2024-03-24T01:10:53.560 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfb.ko 2024-03-24T01:10:53.560 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfq.ko 2024-03-24T01:10:53.560 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_tbf.ko 2024-03-24T01:10:53.560 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_teql.ko 2024-03-24T01:10:53.560 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp 2024-03-24T01:10:53.560 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp.ko 2024-03-24T01:10:53.560 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp_diag.ko 2024-03-24T01:10:53.561 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc 2024-03-24T01:10:53.561 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss 2024-03-24T01:10:53.561 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-24T01:10:53.561 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-24T01:10:53.561 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/sunrpc.ko 2024-03-24T01:10:53.561 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma 2024-03-24T01:10:53.561 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-24T01:10:53.561 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc 2024-03-24T01:10:53.561 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/diag.ko 2024-03-24T01:10:53.561 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/tipc.ko 2024-03-24T01:10:53.561 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix 2024-03-24T01:10:53.561 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix/unix_diag.ko 2024-03-24T01:10:53.561 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock 2024-03-24T01:10:53.561 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-24T01:10:53.561 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-24T01:10:53.562 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock.ko 2024-03-24T01:10:53.562 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-24T01:10:53.562 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-24T01:10:53.562 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm 2024-03-24T01:10:53.562 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-24T01:10:53.562 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security 2024-03-24T01:10:53.562 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys 2024-03-24T01:10:53.562 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/encrypted-keys 2024-03-24T01:10:53.562 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-24T01:10:53.562 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/trusted-keys 2024-03-24T01:10:53.562 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/trusted-keys/trusted.ko 2024-03-24T01:10:53.562 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt 2024-03-24T01:10:53.562 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib 2024-03-24T01:10:53.562 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib/irqbypass.ko 2024-03-24T01:10:53.563 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias 2024-03-24T01:10:53.563 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias.bin 2024-03-24T01:10:53.563 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin 2024-03-24T01:10:53.563 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.alias.bin 2024-03-24T01:10:53.563 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.bin 2024-03-24T01:10:53.563 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.modinfo 2024-03-24T01:10:53.563 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep 2024-03-24T01:10:53.563 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep.bin 2024-03-24T01:10:53.563 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.devname 2024-03-24T01:10:53.563 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.order 2024-03-24T01:10:53.563 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.softdep 2024-03-24T01:10:53.563 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols 2024-03-24T01:10:53.563 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols.bin 2024-03-24T01:10:53.563 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/vmlinuz 2024-03-24T01:10:53.567 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g5ec3e6e88b2e 2024-03-24T01:10:53.567 DEBUG:teuthology.orchestra.run.smithi090:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-24T01:10:53.591 INFO:teuthology.orchestra.run.smithi090.stdout:postinstall scriptlet (using /bin/sh): 2024-03-24T01:10:53.591 INFO:teuthology.orchestra.run.smithi090.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-24T01:10:53.591 INFO:teuthology.orchestra.run.smithi090.stdout: /usr/bin/kernel-install add 6.8.0-g5ec3e6e88b2e /lib/modules/6.8.0-g5ec3e6e88b2e/vmlinuz 2024-03-24T01:10:53.591 INFO:teuthology.orchestra.run.smithi090.stdout:fi 2024-03-24T01:10:53.591 INFO:teuthology.orchestra.run.smithi090.stdout:for file in vmlinuz System.map config; do 2024-03-24T01:10:53.591 INFO:teuthology.orchestra.run.smithi090.stdout: if ! cmp --silent "/lib/modules/6.8.0-g5ec3e6e88b2e/${file}" "/boot/${file}-6.8.0-g5ec3e6e88b2e"; then 2024-03-24T01:10:53.592 INFO:teuthology.orchestra.run.smithi090.stdout: cp "/lib/modules/6.8.0-g5ec3e6e88b2e/${file}" "/boot/${file}-6.8.0-g5ec3e6e88b2e" 2024-03-24T01:10:53.592 INFO:teuthology.orchestra.run.smithi090.stdout: fi 2024-03-24T01:10:53.592 INFO:teuthology.orchestra.run.smithi090.stdout:done 2024-03-24T01:10:53.592 INFO:teuthology.orchestra.run.smithi090.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-24T01:10:53.592 INFO:teuthology.orchestra.run.smithi090.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-24T01:10:53.592 INFO:teuthology.orchestra.run.smithi090.stdout:new-kernel-pkg --remove 6.8.0-g5ec3e6e88b2e --rminitrd --initrdfile=/boot/initramfs-6.8.0-g5ec3e6e88b2e.img 2024-03-24T01:10:53.592 INFO:teuthology.orchestra.run.smithi090.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-24T01:10:53.592 INFO:teuthology.orchestra.run.smithi090.stdout:kernel-install remove 6.8.0-g5ec3e6e88b2e 2024-03-24T01:10:53.592 INFO:teuthology.orchestra.run.smithi090.stdout:fi 2024-03-24T01:10:53.592 INFO:teuthology.orchestra.run.smithi090.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-24T01:10:53.592 INFO:teuthology.orchestra.run.smithi090.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-24T01:10:53.592 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/update-bootloader --remove 6.8.0-g5ec3e6e88b2e 2024-03-24T01:10:53.592 INFO:teuthology.orchestra.run.smithi090.stdout:fi 2024-03-24T01:10:53.593 DEBUG:teuthology.orchestra.run.smithi090:> sudo rpm -qi grub2-tools 2024-03-24T01:10:53.670 INFO:teuthology.orchestra.run.smithi090.stdout:Name : grub2-tools 2024-03-24T01:10:53.670 INFO:teuthology.orchestra.run.smithi090.stdout:Epoch : 1 2024-03-24T01:10:53.670 INFO:teuthology.orchestra.run.smithi090.stdout:Version : 2.06 2024-03-24T01:10:53.670 INFO:teuthology.orchestra.run.smithi090.stdout:Release : 68.el9 2024-03-24T01:10:53.670 INFO:teuthology.orchestra.run.smithi090.stdout:Architecture: x86_64 2024-03-24T01:10:53.670 INFO:teuthology.orchestra.run.smithi090.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T01:10:53.670 INFO:teuthology.orchestra.run.smithi090.stdout:Group : Unspecified 2024-03-24T01:10:53.670 INFO:teuthology.orchestra.run.smithi090.stdout:Size : 8264254 2024-03-24T01:10:53.671 INFO:teuthology.orchestra.run.smithi090.stdout:License : GPLv3+ 2024-03-24T01:10:53.671 INFO:teuthology.orchestra.run.smithi090.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T01:10:53.671 INFO:teuthology.orchestra.run.smithi090.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T01:10:53.671 INFO:teuthology.orchestra.run.smithi090.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T01:10:53.671 INFO:teuthology.orchestra.run.smithi090.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T01:10:53.671 INFO:teuthology.orchestra.run.smithi090.stdout:Packager : builder@centos.org 2024-03-24T01:10:53.671 INFO:teuthology.orchestra.run.smithi090.stdout:Vendor : CentOS 2024-03-24T01:10:53.671 INFO:teuthology.orchestra.run.smithi090.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T01:10:53.671 INFO:teuthology.orchestra.run.smithi090.stdout:Summary : Support tools for GRUB. 2024-03-24T01:10:53.671 INFO:teuthology.orchestra.run.smithi090.stdout:Description : 2024-03-24T01:10:53.671 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:10:53.671 INFO:teuthology.orchestra.run.smithi090.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T01:10:53.671 INFO:teuthology.orchestra.run.smithi090.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T01:10:53.671 INFO:teuthology.orchestra.run.smithi090.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T01:10:53.671 INFO:teuthology.orchestra.run.smithi090.stdout:hardware devices. 2024-03-24T01:10:53.671 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:10:53.672 INFO:teuthology.orchestra.run.smithi090.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T01:10:53.672 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T01:10:53.673 INFO:teuthology.task.kernel:Updating grub on smithi090 to boot 6.8.0-g5ec3e6e88b2e 2024-03-24T01:10:53.673 DEBUG:teuthology.orchestra.run.smithi090:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T01:10:54.308 INFO:teuthology.orchestra.run.smithi090.stderr:Generating grub configuration file ... 2024-03-24T01:10:56.159 INFO:teuthology.orchestra.run.smithi090.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T01:10:56.191 INFO:teuthology.orchestra.run.smithi090.stderr:done 2024-03-24T01:10:56.194 DEBUG:teuthology.orchestra.run.smithi090:> mktemp 2024-03-24T01:10:56.221 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/tmp.ifOjFwSTSe 2024-03-24T01:10:56.221 DEBUG:teuthology.orchestra.run.smithi090:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ifOjFwSTSe 2024-03-24T01:10:56.292 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0666 /tmp/tmp.ifOjFwSTSe 2024-03-24T01:10:56.445 DEBUG:teuthology.orchestra.remote:smithi090:/tmp/tmp.ifOjFwSTSe is 6KB 2024-03-24T01:10:56.493 DEBUG:teuthology.orchestra.run.smithi090:> rm -fr /tmp/tmp.ifOjFwSTSe 2024-03-24T01:10:56.507 DEBUG:teuthology.orchestra.run.smithi090:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T01:10:56.571 INFO:teuthology.orchestra.run.smithi090.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T01:10:56.571 INFO:teuthology.orchestra.run.smithi090.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T01:10:56.572 INFO:teuthology.orchestra.run.smithi090.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T01:10:56.572 INFO:teuthology.orchestra.run.smithi090.stdout:15949e3c88704300882482eddfd7baae-6.8.0-g5ec3e6e88b2e.conf 2024-03-24T01:10:56.573 DEBUG:teuthology.orchestra.run.smithi090:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-g5ec3e6e88b2e 2024-03-24T01:10:56.730 DEBUG:teuthology.orchestra.run.smithi090:> sudo shutdown -r now 2024-03-24T01:11:06.246 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-24T01:11:06.247 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-03-24T01:11:06.247 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T01:11:26.754 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-24T01:11:26.755 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-24T01:11:26.756 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T01:12:01.310 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-24T01:12:06.250 DEBUG:teuthology.orchestra.remote:timed out 2024-03-24T01:12:10.315 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-24T01:12:10.316 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T01:12:13.375 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-24T01:12:15.253 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-03-24T01:12:15.254 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T01:12:22.915 DEBUG:teuthology.orchestra.run.smithi120:> true 2024-03-24T01:12:23.508 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-03-24T01:12:23.509 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "6.8.0-g5ec3e6e88b2e"... 2024-03-24T01:12:23.509 DEBUG:teuthology.orchestra.run.smithi120:> uname -r 2024-03-24T01:12:23.550 INFO:teuthology.orchestra.run.smithi120.stdout:6.8.0-g5ec3e6e88b2e 2024-03-24T01:12:23.550 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g5ec3e6e88b2e vs 6.8.0-g5ec3e6e88b2e 2024-03-24T01:12:23.550 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T01:12:23.550 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T01:12:24.551 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-24T01:12:24.552 DEBUG:teuthology.orchestra.run.smithi120:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T01:12:24.671 INFO:teuthology.orchestra.run.smithi120.stdout:ttyS1 2024-03-24T01:12:24.733 DEBUG:teuthology.parallel:result is None 2024-03-24T01:12:25.376 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-24T01:12:25.378 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T01:12:43.806 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-24T01:12:58.808 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-24T01:12:58.809 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T01:13:02.657 DEBUG:teuthology.orchestra.run.smithi090:> true 2024-03-24T01:13:03.566 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-24T01:13:03.566 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "6.8.0-g5ec3e6e88b2e"... 2024-03-24T01:13:03.567 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2024-03-24T01:13:03.605 INFO:teuthology.orchestra.run.smithi090.stdout:6.8.0-g5ec3e6e88b2e 2024-03-24T01:13:03.605 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g5ec3e6e88b2e vs 6.8.0-g5ec3e6e88b2e 2024-03-24T01:13:03.605 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T01:13:03.606 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T01:13:04.606 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-24T01:13:04.607 DEBUG:teuthology.orchestra.run.smithi090:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T01:13:04.873 INFO:teuthology.orchestra.run.smithi090.stdout:ttyS1 2024-03-24T01:13:04.920 DEBUG:teuthology.parallel:result is None 2024-03-24T01:13:04.920 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-24T01:13:04.928 INFO:teuthology.task.internal:Creating test directory... 2024-03-24T01:13:04.929 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T01:13:04.933 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T01:13:04.992 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-24T01:13:05.023 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-24T01:13:05.034 INFO:teuthology.task.internal:Creating archive directory... 2024-03-24T01:13:05.034 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T01:13:05.039 DEBUG:teuthology.orchestra.run.smithi120:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T01:13:05.112 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-24T01:13:05.128 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-24T01:13:05.128 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T01:13:05.132 DEBUG:teuthology.orchestra.run.smithi120:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T01:13:05.201 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T01:13:05.234 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T01:13:05.239 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T01:13:05.279 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T01:13:05.284 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-24T01:13:05.291 INFO:teuthology.task.internal:Configuring sudo... 2024-03-24T01:13:05.291 DEBUG:teuthology.orchestra.run.smithi090:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T01:13:05.298 DEBUG:teuthology.orchestra.run.smithi120:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T01:13:05.395 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-24T01:13:05.405 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-24T01:13:05.405 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T01:13:05.410 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T01:13:05.462 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T01:13:05.565 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T01:13:05.672 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T01:13:05.749 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T01:13:05.819 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:13:05.819 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T01:13:05.905 DEBUG:teuthology.orchestra.run.smithi120:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T01:13:05.989 DEBUG:teuthology.orchestra.run.smithi120:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T01:13:06.121 DEBUG:teuthology.orchestra.run.smithi120:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T01:13:06.204 DEBUG:teuthology.orchestra.run.smithi120:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T01:13:06.299 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:13:06.299 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T01:13:06.392 DEBUG:teuthology.orchestra.run.smithi090:> sudo service rsyslog restart 2024-03-24T01:13:06.400 DEBUG:teuthology.orchestra.run.smithi120:> sudo service rsyslog restart 2024-03-24T01:13:06.518 INFO:teuthology.orchestra.run.smithi090.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T01:13:06.568 INFO:teuthology.orchestra.run.smithi120.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T01:13:06.951 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-24T01:13:06.962 INFO:teuthology.task.internal:Starting timer... 2024-03-24T01:13:06.963 INFO:teuthology.run_tasks:Running task pcp... 2024-03-24T01:13:06.990 INFO:teuthology.run_tasks:Running task selinux... 2024-03-24T01:13:07.013 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-03-24T01:13:07.013 DEBUG:teuthology.orchestra.run.smithi090:> sudo service auditd rotate 2024-03-24T01:13:07.142 INFO:teuthology.orchestra.run.smithi090.stdout:Rotating logs: 2024-03-24T01:13:07.146 DEBUG:teuthology.orchestra.run.smithi120:> sudo service auditd rotate 2024-03-24T01:13:07.285 INFO:teuthology.orchestra.run.smithi120.stdout:Rotating logs: 2024-03-24T01:13:07.289 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-24T01:13:07.289 DEBUG:teuthology.orchestra.run.smithi090:> /usr/sbin/getenforce 2024-03-24T01:13:07.343 INFO:teuthology.orchestra.run.smithi090.stdout:Permissive 2024-03-24T01:13:07.343 DEBUG:teuthology.orchestra.run.smithi120:> /usr/sbin/getenforce 2024-03-24T01:13:07.396 INFO:teuthology.orchestra.run.smithi120.stdout:Permissive 2024-03-24T01:13:07.396 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi090.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi120.front.sepia.ceph.com': 'permissive'} 2024-03-24T01:13:07.397 DEBUG:teuthology.orchestra.run.smithi090:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T01:13:07.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T01:13:07.484 DEBUG:teuthology.orchestra.run.smithi120:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T01:13:07.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T01:13:07.577 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-24T01:13:07.577 DEBUG:teuthology.orchestra.run.smithi090:> sudo /usr/sbin/setenforce permissive 2024-03-24T01:13:07.661 DEBUG:teuthology.orchestra.run.smithi120:> sudo /usr/sbin/setenforce permissive 2024-03-24T01:13:07.750 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-24T01:13:07.761 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-24T01:13:07.767 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-24T01:13:07.959 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-24T01:13:08.042 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-24T01:13:08.043 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi090.front.sepia.ceph.com,smithi120.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-24T01:20:18.916 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi090.front.sepia.ceph.com'), Remote(name='ubuntu@smithi120.front.sepia.ceph.com')] 2024-03-24T01:20:18.918 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-24T01:20:18.919 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T01:20:19.055 DEBUG:teuthology.orchestra.run.smithi090:> true 2024-03-24T01:20:19.207 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-24T01:20:19.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-03-24T01:20:19.208 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T01:20:19.345 DEBUG:teuthology.orchestra.run.smithi120:> true 2024-03-24T01:20:19.502 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-03-24T01:20:19.503 INFO:teuthology.run_tasks:Running task clock... 2024-03-24T01:20:19.514 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-24T01:20:19.514 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T01:20:19.515 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T01:20:19.522 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T01:20:19.522 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T01:20:19.618 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T01:20:19.621 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T01:20:19.667 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T01:20:19.672 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T01:20:19.751 INFO:teuthology.orchestra.run.smithi120.stderr:sudo: ntpd: command not found 2024-03-24T01:20:19.761 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: ntpd: command not found 2024-03-24T01:20:19.791 INFO:teuthology.orchestra.run.smithi120.stdout:506 Cannot talk to daemon 2024-03-24T01:20:19.805 INFO:teuthology.orchestra.run.smithi090.stdout:506 Cannot talk to daemon 2024-03-24T01:20:19.844 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T01:20:19.857 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T01:20:19.896 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T01:20:19.911 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T01:20:20.023 INFO:teuthology.orchestra.run.smithi120.stderr:bash: line 1: ntpq: command not found 2024-03-24T01:20:20.031 INFO:teuthology.orchestra.run.smithi120.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T01:20:20.031 INFO:teuthology.orchestra.run.smithi120.stdout:=============================================================================== 2024-03-24T01:20:20.032 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T01:20:20.032 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T01:20:20.032 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T01:20:20.032 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T01:20:20.040 INFO:teuthology.orchestra.run.smithi090.stderr:bash: line 1: ntpq: command not found 2024-03-24T01:20:20.047 INFO:teuthology.orchestra.run.smithi090.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T01:20:20.047 INFO:teuthology.orchestra.run.smithi090.stdout:=============================================================================== 2024-03-24T01:20:20.047 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T01:20:20.048 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T01:20:20.048 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T01:20:20.048 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T01:20:20.049 INFO:teuthology.run_tasks:Running task install... 2024-03-24T01:20:20.057 DEBUG:teuthology.task.install:project ceph 2024-03-24T01:20:20.058 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}} 2024-03-24T01:20:20.058 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'} 2024-03-24T01:20:20.058 INFO:teuthology.task.install:Using flavor: default 2024-03-24T01:20:20.064 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-24T01:20:20.064 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-24T01:20:20.064 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-24T01:20:20.065 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T01:20:20.067 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-24T01:20:20.067 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T01:20:20.209 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-24T01:20:20.210 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-24T01:20:20.281 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-24T01:20:20.282 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-24T01:20:20.379 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T01:20:20.379 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:20:20.379 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T01:20:20.389 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T01:20:20.389 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:20:20.389 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T01:20:20.477 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-24T01:20:20.477 DEBUG:teuthology.orchestra.run.smithi090:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T01:20:20.486 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-24T01:20:20.486 DEBUG:teuthology.orchestra.run.smithi120:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T01:20:20.617 DEBUG:teuthology.orchestra.run.smithi090:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T01:20:20.625 DEBUG:teuthology.orchestra.run.smithi120:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T01:20:20.749 DEBUG:teuthology.orchestra.run.smithi090:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T01:20:20.757 DEBUG:teuthology.orchestra.run.smithi120:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T01:20:20.837 INFO:teuthology.orchestra.run.smithi090.stdout:check_obsoletes = 1 2024-03-24T01:20:20.842 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum clean all 2024-03-24T01:20:20.850 INFO:teuthology.orchestra.run.smithi120.stdout:check_obsoletes = 1 2024-03-24T01:20:20.855 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum clean all 2024-03-24T01:20:21.188 INFO:teuthology.orchestra.run.smithi090.stdout:74 files removed 2024-03-24T01:20:21.198 INFO:teuthology.orchestra.run.smithi120.stdout:74 files removed 2024-03-24T01:20:21.227 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-24T01:20:21.236 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-24T01:20:22.259 INFO:teuthology.orchestra.run.smithi090.stdout:ceph packages for x86_64 157 kB/s | 83 kB 00:00 2024-03-24T01:20:22.286 INFO:teuthology.orchestra.run.smithi120.stdout:ceph packages for x86_64 152 kB/s | 83 kB 00:00 2024-03-24T01:20:22.761 INFO:teuthology.orchestra.run.smithi120.stdout:ceph noarch packages 26 kB/s | 10 kB 00:00 2024-03-24T01:20:22.810 INFO:teuthology.orchestra.run.smithi090.stdout:ceph noarch packages 22 kB/s | 10 kB 00:00 2024-03-24T01:20:23.312 INFO:teuthology.orchestra.run.smithi120.stdout:ceph source packages 4.0 kB/s | 1.9 kB 00:00 2024-03-24T01:20:23.319 INFO:teuthology.orchestra.run.smithi090.stdout:ceph source packages 4.4 kB/s | 1.9 kB 00:00 2024-03-24T01:20:24.006 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-24T01:20:24.040 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-24T01:20:25.657 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-9 - BaseOS 5.7 MB/s | 2.0 MB 00:00 2024-03-24T01:20:25.699 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-9 - BaseOS 5.4 MB/s | 2.0 MB 00:00 2024-03-24T01:20:26.754 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-24T01:20:35.420 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - BaseOS 876 kB/s | 8.0 MB 00:09 2024-03-24T01:20:37.698 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-03-24T01:20:49.583 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - CRB 772 kB/s | 6.1 MB 00:08 2024-03-24T01:20:51.217 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - Extras packages 51 kB/s | 16 kB 00:00 2024-03-24T01:20:51.485 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - AppStream 820 kB/s | 19 MB 00:23 2024-03-24T01:20:52.911 INFO:teuthology.orchestra.run.smithi090.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-24T01:20:58.190 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - CRB 2.1 MB/s | 6.1 MB 00:02 2024-03-24T01:20:58.771 INFO:teuthology.orchestra.run.smithi090.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-03-24T01:20:59.728 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - Extras packages 82 kB/s | 16 kB 00:00 2024-03-24T01:21:01.416 INFO:teuthology.orchestra.run.smithi090.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T01:21:01.417 INFO:teuthology.orchestra.run.smithi090.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T01:21:01.441 INFO:teuthology.orchestra.run.smithi090.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-24T01:21:01.442 INFO:teuthology.orchestra.run.smithi090.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-24T01:21:01.444 INFO:teuthology.orchestra.run.smithi090.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-24T01:21:01.445 INFO:teuthology.orchestra.run.smithi090.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-24T01:21:01.448 INFO:teuthology.orchestra.run.smithi090.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-24T01:21:01.453 INFO:teuthology.orchestra.run.smithi090.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-24T01:21:01.453 INFO:teuthology.orchestra.run.smithi090.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-24T01:21:01.541 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-24T01:21:01.569 INFO:teuthology.orchestra.run.smithi090.stdout:============================================================================================ 2024-03-24T01:21:01.569 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-03-24T01:21:01.570 INFO:teuthology.orchestra.run.smithi090.stdout:============================================================================================ 2024-03-24T01:21:01.570 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-24T01:21:01.570 INFO:teuthology.orchestra.run.smithi090.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-24T01:21:01.570 INFO:teuthology.orchestra.run.smithi090.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-24T01:21:01.570 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-24T01:21:01.570 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-24T01:21:01.570 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-24T01:21:01.570 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-24T01:21:01.570 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-24T01:21:01.570 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-24T01:21:01.570 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-24T01:21:01.570 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-24T01:21:01.571 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-24T01:21:01.571 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-24T01:21:01.571 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-24T01:21:01.571 INFO:teuthology.orchestra.run.smithi090.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-24T01:21:01.571 INFO:teuthology.orchestra.run.smithi090.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-24T01:21:01.571 INFO:teuthology.orchestra.run.smithi090.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-24T01:21:01.571 INFO:teuthology.orchestra.run.smithi090.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-24T01:21:01.571 INFO:teuthology.orchestra.run.smithi090.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-24T01:21:01.571 INFO:teuthology.orchestra.run.smithi090.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-24T01:21:01.571 INFO:teuthology.orchestra.run.smithi090.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-24T01:21:01.571 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-24T01:21:01.571 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-24T01:21:01.571 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-24T01:21:01.571 INFO:teuthology.orchestra.run.smithi090.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-24T01:21:01.572 INFO:teuthology.orchestra.run.smithi090.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-24T01:21:01.572 INFO:teuthology.orchestra.run.smithi090.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-24T01:21:01.572 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-24T01:21:01.572 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-24T01:21:01.572 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-24T01:21:01.572 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-24T01:21:01.572 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-24T01:21:01.572 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-24T01:21:01.572 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-24T01:21:01.572 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-24T01:21:01.572 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-24T01:21:01.572 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-24T01:21:01.572 INFO:teuthology.orchestra.run.smithi090.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-24T01:21:01.572 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrading: 2024-03-24T01:21:01.573 INFO:teuthology.orchestra.run.smithi090.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-24T01:21:01.573 INFO:teuthology.orchestra.run.smithi090.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-24T01:21:01.573 INFO:teuthology.orchestra.run.smithi090.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-24T01:21:01.573 INFO:teuthology.orchestra.run.smithi090.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-24T01:21:01.573 INFO:teuthology.orchestra.run.smithi090.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-24T01:21:01.573 INFO:teuthology.orchestra.run.smithi090.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-24T01:21:01.573 INFO:teuthology.orchestra.run.smithi090.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-24T01:21:01.573 INFO:teuthology.orchestra.run.smithi090.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-24T01:21:01.573 INFO:teuthology.orchestra.run.smithi090.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-24T01:21:01.573 INFO:teuthology.orchestra.run.smithi090.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-24T01:21:01.573 INFO:teuthology.orchestra.run.smithi090.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-24T01:21:01.573 INFO:teuthology.orchestra.run.smithi090.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-24T01:21:01.573 INFO:teuthology.orchestra.run.smithi090.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-24T01:21:01.573 INFO:teuthology.orchestra.run.smithi090.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-24T01:21:01.573 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-24T01:21:01.574 INFO:teuthology.orchestra.run.smithi090.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-24T01:21:01.574 INFO:teuthology.orchestra.run.smithi090.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-24T01:21:01.574 INFO:teuthology.orchestra.run.smithi090.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-24T01:21:01.574 INFO:teuthology.orchestra.run.smithi090.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-24T01:21:01.574 INFO:teuthology.orchestra.run.smithi090.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-24T01:21:01.574 INFO:teuthology.orchestra.run.smithi090.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-24T01:21:01.574 INFO:teuthology.orchestra.run.smithi090.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-24T01:21:01.574 INFO:teuthology.orchestra.run.smithi090.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-24T01:21:01.574 INFO:teuthology.orchestra.run.smithi090.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-24T01:21:01.574 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-03-24T01:21:01.574 INFO:teuthology.orchestra.run.smithi090.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-24T01:21:01.574 INFO:teuthology.orchestra.run.smithi090.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-24T01:21:01.574 INFO:teuthology.orchestra.run.smithi090.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T01:21:01.575 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-24T01:21:01.575 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-24T01:21:01.575 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-24T01:21:01.575 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-24T01:21:01.575 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-24T01:21:01.575 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-24T01:21:01.575 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-24T01:21:01.575 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-24T01:21:01.575 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T01:21:01.575 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T01:21:01.575 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T01:21:01.575 INFO:teuthology.orchestra.run.smithi090.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T01:21:01.575 INFO:teuthology.orchestra.run.smithi090.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T01:21:01.576 INFO:teuthology.orchestra.run.smithi090.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-24T01:21:01.576 INFO:teuthology.orchestra.run.smithi090.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T01:21:01.576 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T01:21:01.576 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T01:21:01.576 INFO:teuthology.orchestra.run.smithi090.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-24T01:21:01.576 INFO:teuthology.orchestra.run.smithi090.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T01:21:01.576 INFO:teuthology.orchestra.run.smithi090.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T01:21:01.576 INFO:teuthology.orchestra.run.smithi090.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T01:21:01.576 INFO:teuthology.orchestra.run.smithi090.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T01:21:01.576 INFO:teuthology.orchestra.run.smithi090.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T01:21:01.576 INFO:teuthology.orchestra.run.smithi090.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-24T01:21:01.576 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T01:21:01.576 INFO:teuthology.orchestra.run.smithi090.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-24T01:21:01.576 INFO:teuthology.orchestra.run.smithi090.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T01:21:01.576 INFO:teuthology.orchestra.run.smithi090.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-24T01:21:01.577 INFO:teuthology.orchestra.run.smithi090.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T01:21:01.577 INFO:teuthology.orchestra.run.smithi090.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-24T01:21:01.577 INFO:teuthology.orchestra.run.smithi090.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-24T01:21:01.577 INFO:teuthology.orchestra.run.smithi090.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-24T01:21:01.577 INFO:teuthology.orchestra.run.smithi090.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T01:21:01.577 INFO:teuthology.orchestra.run.smithi090.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T01:21:01.577 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T01:21:01.577 INFO:teuthology.orchestra.run.smithi090.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T01:21:01.577 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-24T01:21:01.577 INFO:teuthology.orchestra.run.smithi090.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-24T01:21:01.577 INFO:teuthology.orchestra.run.smithi090.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-24T01:21:01.577 INFO:teuthology.orchestra.run.smithi090.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T01:21:01.577 INFO:teuthology.orchestra.run.smithi090.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-24T01:21:01.577 INFO:teuthology.orchestra.run.smithi090.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T01:21:01.578 INFO:teuthology.orchestra.run.smithi090.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T01:21:01.578 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T01:21:01.578 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-24T01:21:01.578 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-24T01:21:01.578 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T01:21:01.578 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T01:21:01.578 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T01:21:01.578 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T01:21:01.578 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-24T01:21:01.578 INFO:teuthology.orchestra.run.smithi090.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-24T01:21:01.578 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T01:21:01.578 INFO:teuthology.orchestra.run.smithi090.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T01:21:01.578 INFO:teuthology.orchestra.run.smithi090.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T01:21:01.578 INFO:teuthology.orchestra.run.smithi090.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-24T01:21:01.579 INFO:teuthology.orchestra.run.smithi090.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-24T01:21:01.579 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T01:21:01.579 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T01:21:01.579 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T01:21:01.579 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T01:21:01.579 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T01:21:01.579 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T01:21:01.579 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T01:21:01.579 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T01:21:01.579 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T01:21:01.579 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T01:21:01.579 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T01:21:01.579 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T01:21:01.579 INFO:teuthology.orchestra.run.smithi090.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T01:21:01.579 INFO:teuthology.orchestra.run.smithi090.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T01:21:01.579 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T01:21:01.580 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T01:21:01.580 INFO:teuthology.orchestra.run.smithi090.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T01:21:01.580 INFO:teuthology.orchestra.run.smithi090.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-24T01:21:01.580 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T01:21:01.580 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T01:21:01.580 INFO:teuthology.orchestra.run.smithi090.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T01:21:01.580 INFO:teuthology.orchestra.run.smithi090.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T01:21:01.580 INFO:teuthology.orchestra.run.smithi090.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-24T01:21:01.580 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T01:21:01.580 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T01:21:01.580 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T01:21:01.580 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T01:21:01.580 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-24T01:21:01.580 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T01:21:01.581 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T01:21:01.581 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T01:21:01.581 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T01:21:01.581 INFO:teuthology.orchestra.run.smithi090.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T01:21:01.581 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T01:21:01.581 INFO:teuthology.orchestra.run.smithi090.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T01:21:01.581 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T01:21:01.581 INFO:teuthology.orchestra.run.smithi090.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T01:21:01.581 INFO:teuthology.orchestra.run.smithi090.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T01:21:01.581 INFO:teuthology.orchestra.run.smithi090.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T01:21:01.581 INFO:teuthology.orchestra.run.smithi090.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-24T01:21:01.581 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T01:21:01.581 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T01:21:01.581 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T01:21:01.581 INFO:teuthology.orchestra.run.smithi090.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T01:21:01.582 INFO:teuthology.orchestra.run.smithi090.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-24T01:21:01.582 INFO:teuthology.orchestra.run.smithi090.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-24T01:21:01.582 INFO:teuthology.orchestra.run.smithi090.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T01:21:01.582 INFO:teuthology.orchestra.run.smithi090.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T01:21:01.582 INFO:teuthology.orchestra.run.smithi090.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T01:21:01.582 INFO:teuthology.orchestra.run.smithi090.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-24T01:21:01.582 INFO:teuthology.orchestra.run.smithi090.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T01:21:01.582 INFO:teuthology.orchestra.run.smithi090.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-24T01:21:01.582 INFO:teuthology.orchestra.run.smithi090.stdout:Installing weak dependencies: 2024-03-24T01:21:01.582 INFO:teuthology.orchestra.run.smithi090.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-24T01:21:01.582 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:21:01.582 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-24T01:21:01.582 INFO:teuthology.orchestra.run.smithi090.stdout:============================================================================================ 2024-03-24T01:21:01.582 INFO:teuthology.orchestra.run.smithi090.stdout:Install 150 Packages 2024-03-24T01:21:01.582 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrade 24 Packages 2024-03-24T01:21:01.583 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:21:01.583 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 224 M 2024-03-24T01:21:01.583 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-24T01:21:02.335 INFO:teuthology.orchestra.run.smithi120.stdout:Extra Packages for Enterprise Linux 8.5 MB/s | 21 MB 00:02 2024-03-24T01:21:02.882 INFO:teuthology.orchestra.run.smithi090.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 17 kB/s | 6.6 kB 00:00 2024-03-24T01:21:03.199 INFO:teuthology.orchestra.run.smithi090.stdout:(2/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 7.6 MB/s | 5.4 MB 00:00 2024-03-24T01:21:03.249 INFO:teuthology.orchestra.run.smithi090.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 2.9 MB/s | 145 kB 00:00 2024-03-24T01:21:03.442 INFO:teuthology.orchestra.run.smithi090.stdout:(4/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 12 MB/s | 2.3 MB 00:00 2024-03-24T01:21:03.492 INFO:teuthology.orchestra.run.smithi090.stdout:(5/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.4 MB/s | 890 kB 00:00 2024-03-24T01:21:03.576 INFO:teuthology.orchestra.run.smithi090.stdout:(6/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 11 MB/s | 1.5 MB 00:00 2024-03-24T01:21:04.476 INFO:teuthology.orchestra.run.smithi090.stdout:(7/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 4.8 MB/s | 4.7 MB 00:00 2024-03-24T01:21:04.751 INFO:teuthology.orchestra.run.smithi090.stdout:(8/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x86 14 MB/s | 17 MB 00:01 2024-03-24T01:21:04.818 INFO:teuthology.orchestra.run.smithi090.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 384 kB/s | 25 kB 00:00 2024-03-24T01:21:06.691 INFO:teuthology.orchestra.run.smithi090.stdout:(10/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el 4.5 MB/s | 9.9 MB 00:02 2024-03-24T01:21:06.986 INFO:teuthology.orchestra.run.smithi090.stdout:(11/174): ceph-common-19.0.0-2145.g3be9a5c9.el9 4.6 MB/s | 20 MB 00:04 2024-03-24T01:21:07.028 INFO:teuthology.orchestra.run.smithi090.stdout:(12/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 657 kB/s | 220 kB 00:00 2024-03-24T01:21:07.053 INFO:teuthology.orchestra.run.smithi090.stdout:(13/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 411 kB/s | 27 kB 00:00 2024-03-24T01:21:07.095 INFO:teuthology.orchestra.run.smithi090.stdout:(14/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 473 kB/s | 31 kB 00:00 2024-03-24T01:21:07.229 INFO:teuthology.orchestra.run.smithi090.stdout:(15/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 4.2 MB/s | 746 kB 00:00 2024-03-24T01:21:07.254 INFO:teuthology.orchestra.run.smithi090.stdout:(16/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 1.0 MB/s | 163 kB 00:00 2024-03-24T01:21:07.321 INFO:teuthology.orchestra.run.smithi090.stdout:(17/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.4 MB/s | 127 kB 00:00 2024-03-24T01:21:07.405 INFO:teuthology.orchestra.run.smithi090.stdout:(18/174): libradosstriper1-19.0.0-2145.g3be9a5c 3.3 MB/s | 501 kB 00:00 2024-03-24T01:21:07.505 INFO:teuthology.orchestra.run.smithi090.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 456 kB/s | 45 kB 00:00 2024-03-24T01:21:07.614 INFO:teuthology.orchestra.run.smithi090.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 1.2 MB/s | 131 kB 00:00 2024-03-24T01:21:07.724 INFO:teuthology.orchestra.run.smithi090.stdout:(21/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 1.5 MB/s | 162 kB 00:00 2024-03-24T01:21:07.841 INFO:teuthology.orchestra.run.smithi090.stdout:(22/174): python3-rados-19.0.0-2145.g3be9a5c9.e 2.7 MB/s | 321 kB 00:00 2024-03-24T01:21:07.966 INFO:teuthology.orchestra.run.smithi090.stdout:(23/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 8.0 MB/s | 5.1 MB 00:00 2024-03-24T01:21:08.008 INFO:teuthology.orchestra.run.smithi090.stdout:(24/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 1.8 MB/s | 298 kB 00:00 2024-03-24T01:21:08.074 INFO:teuthology.orchestra.run.smithi090.stdout:(25/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 929 kB/s | 99 kB 00:00 2024-03-24T01:21:08.109 INFO:teuthology.orchestra.run.smithi090.stdout:(26/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 858 kB/s | 85 kB 00:00 2024-03-24T01:21:08.176 INFO:teuthology.orchestra.run.smithi090.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 2.5 MB/s | 170 kB 00:00 2024-03-24T01:21:08.312 INFO:teuthology.orchestra.run.smithi120.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-24T01:21:08.768 INFO:teuthology.orchestra.run.smithi090.stdout:(28/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 12 MB/s | 48 MB 00:03 2024-03-24T01:21:08.793 INFO:teuthology.orchestra.run.smithi090.stdout:(29/174): ceph-grafana-dashboards-19.0.0-2145.g 55 kB/s | 34 kB 00:00 2024-03-24T01:21:08.835 INFO:teuthology.orchestra.run.smithi090.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 2.1 MB/s | 139 kB 00:00 2024-03-24T01:21:08.927 INFO:teuthology.orchestra.run.smithi090.stdout:(31/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 3.6 MB/s | 3.1 MB 00:00 2024-03-24T01:21:09.069 INFO:teuthology.orchestra.run.smithi090.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2145.g3b 1.7 MB/s | 248 kB 00:00 2024-03-24T01:21:09.178 INFO:teuthology.orchestra.run.smithi090.stdout:(33/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 460 kB/s | 49 kB 00:00 2024-03-24T01:21:09.245 INFO:teuthology.orchestra.run.smithi090.stdout:(34/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 4.0 MB/s | 1.8 MB 00:00 2024-03-24T01:21:09.270 INFO:teuthology.orchestra.run.smithi090.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2145.g3 181 kB/s | 16 kB 00:00 2024-03-24T01:21:09.321 INFO:teuthology.orchestra.run.smithi090.stdout:(36/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 3.5 MB/s | 264 kB 00:00 2024-03-24T01:21:09.471 INFO:teuthology.orchestra.run.smithi090.stdout:(37/174): ceph-mgr-diskprediction-local-19.0.0- 12 MB/s | 7.4 MB 00:00 2024-03-24T01:21:09.505 INFO:teuthology.orchestra.run.smithi090.stdout:(38/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 100 kB/s | 18 kB 00:00 2024-03-24T01:21:09.555 INFO:teuthology.orchestra.run.smithi090.stdout:(39/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 2.6 MB/s | 748 kB 00:00 2024-03-24T01:21:09.805 INFO:teuthology.orchestra.run.smithi090.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 797 kB/s | 106 kB 00:00 2024-03-24T01:21:09.847 INFO:teuthology.orchestra.run.smithi090.stdout:(41/174): automake-1.16.2-8.el9.noarch.rpm 1.7 MB/s | 662 kB 00:00 2024-03-24T01:21:09.889 INFO:teuthology.orchestra.run.smithi090.stdout:(42/174): bison-3.7.4-5.el9.x86_64.rpm 2.3 MB/s | 920 kB 00:00 2024-03-24T01:21:09.914 INFO:teuthology.orchestra.run.smithi090.stdout:(43/174): flex-2.6.4-9.el9.x86_64.rpm 3.0 MB/s | 307 kB 00:00 2024-03-24T01:21:09.940 INFO:teuthology.orchestra.run.smithi090.stdout:(44/174): flexiblas-3.0.4-8.el9.x86_64.rpm 358 kB/s | 32 kB 00:00 2024-03-24T01:21:09.965 INFO:teuthology.orchestra.run.smithi090.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 351 kB/s | 17 kB 00:00 2024-03-24T01:21:10.042 INFO:teuthology.orchestra.run.smithi090.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 100 kB/s | 10 kB 00:00 2024-03-24T01:21:10.292 INFO:teuthology.orchestra.run.smithi090.stdout:(47/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 7.9 MB/s | 3.0 MB 00:00 2024-03-24T01:21:10.317 INFO:teuthology.orchestra.run.smithi090.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 470 kB/s | 159 kB 00:00 2024-03-24T01:21:10.342 INFO:teuthology.orchestra.run.smithi090.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 150 kB/s | 44 kB 00:00 2024-03-24T01:21:10.393 INFO:teuthology.orchestra.run.smithi090.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 6.6 MB/s | 661 kB 00:00 2024-03-24T01:21:10.418 INFO:teuthology.orchestra.run.smithi090.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 675 kB/s | 46 kB 00:00 2024-03-24T01:21:10.468 INFO:teuthology.orchestra.run.smithi090.stdout:(52/174): libtool-2.4.6-45.el9.x86_64.rpm 4.7 MB/s | 578 kB 00:00 2024-03-24T01:21:10.494 INFO:teuthology.orchestra.run.smithi090.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 282 kB 00:00 2024-03-24T01:21:10.563 INFO:teuthology.orchestra.run.smithi090.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-24T01:21:10.594 INFO:teuthology.orchestra.run.smithi090.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 350 kB/s | 34 kB 00:00 2024-03-24T01:21:10.637 INFO:teuthology.orchestra.run.smithi090.stdout:(56/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.5 MB/s | 217 kB 00:00 2024-03-24T01:21:10.917 INFO:teuthology.orchestra.run.smithi120.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T01:21:10.919 INFO:teuthology.orchestra.run.smithi120.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T01:21:10.943 INFO:teuthology.orchestra.run.smithi120.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-24T01:21:10.944 INFO:teuthology.orchestra.run.smithi120.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-24T01:21:10.946 INFO:teuthology.orchestra.run.smithi120.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-24T01:21:10.947 INFO:teuthology.orchestra.run.smithi120.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-24T01:21:10.950 INFO:teuthology.orchestra.run.smithi120.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-24T01:21:10.954 INFO:teuthology.orchestra.run.smithi120.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-24T01:21:10.955 INFO:teuthology.orchestra.run.smithi120.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-24T01:21:10.970 INFO:teuthology.orchestra.run.smithi090.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 12 MB/s | 4.7 MB 00:00 2024-03-24T01:21:11.037 INFO:teuthology.orchestra.run.smithi090.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 2.0 MB/s | 811 kB 00:00 2024-03-24T01:21:11.046 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-24T01:21:11.082 INFO:teuthology.orchestra.run.smithi120.stdout:============================================================================================ 2024-03-24T01:21:11.082 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2024-03-24T01:21:11.083 INFO:teuthology.orchestra.run.smithi120.stdout:============================================================================================ 2024-03-24T01:21:11.083 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-24T01:21:11.083 INFO:teuthology.orchestra.run.smithi120.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-24T01:21:11.083 INFO:teuthology.orchestra.run.smithi120.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-24T01:21:11.083 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-24T01:21:11.083 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-24T01:21:11.083 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-24T01:21:11.083 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-24T01:21:11.083 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-24T01:21:11.083 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-24T01:21:11.083 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-24T01:21:11.084 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-24T01:21:11.084 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-24T01:21:11.084 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-24T01:21:11.084 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-24T01:21:11.084 INFO:teuthology.orchestra.run.smithi120.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-24T01:21:11.084 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-24T01:21:11.084 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-24T01:21:11.084 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-24T01:21:11.084 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-24T01:21:11.084 INFO:teuthology.orchestra.run.smithi120.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-24T01:21:11.084 INFO:teuthology.orchestra.run.smithi120.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-24T01:21:11.084 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-24T01:21:11.084 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-24T01:21:11.085 INFO:teuthology.orchestra.run.smithi120.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-24T01:21:11.085 INFO:teuthology.orchestra.run.smithi120.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-24T01:21:11.085 INFO:teuthology.orchestra.run.smithi120.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-24T01:21:11.085 INFO:teuthology.orchestra.run.smithi120.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-24T01:21:11.085 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-24T01:21:11.085 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-24T01:21:11.085 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-24T01:21:11.085 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-24T01:21:11.085 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-24T01:21:11.085 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-24T01:21:11.085 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-24T01:21:11.085 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-24T01:21:11.085 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-24T01:21:11.085 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-24T01:21:11.086 INFO:teuthology.orchestra.run.smithi120.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-24T01:21:11.086 INFO:teuthology.orchestra.run.smithi120.stdout:Upgrading: 2024-03-24T01:21:11.086 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-24T01:21:11.086 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-24T01:21:11.086 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-24T01:21:11.086 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-24T01:21:11.086 INFO:teuthology.orchestra.run.smithi120.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-24T01:21:11.086 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-24T01:21:11.086 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-24T01:21:11.086 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-24T01:21:11.086 INFO:teuthology.orchestra.run.smithi120.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-24T01:21:11.086 INFO:teuthology.orchestra.run.smithi120.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-24T01:21:11.086 INFO:teuthology.orchestra.run.smithi120.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-24T01:21:11.087 INFO:teuthology.orchestra.run.smithi120.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-24T01:21:11.087 INFO:teuthology.orchestra.run.smithi120.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-24T01:21:11.087 INFO:teuthology.orchestra.run.smithi120.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-24T01:21:11.087 INFO:teuthology.orchestra.run.smithi120.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-24T01:21:11.087 INFO:teuthology.orchestra.run.smithi120.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-24T01:21:11.087 INFO:teuthology.orchestra.run.smithi120.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-24T01:21:11.087 INFO:teuthology.orchestra.run.smithi120.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-24T01:21:11.087 INFO:teuthology.orchestra.run.smithi120.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-24T01:21:11.087 INFO:teuthology.orchestra.run.smithi120.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-24T01:21:11.087 INFO:teuthology.orchestra.run.smithi120.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-24T01:21:11.087 INFO:teuthology.orchestra.run.smithi120.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-24T01:21:11.087 INFO:teuthology.orchestra.run.smithi120.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-24T01:21:11.087 INFO:teuthology.orchestra.run.smithi120.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-24T01:21:11.087 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2024-03-24T01:21:11.087 INFO:teuthology.orchestra.run.smithi120.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-24T01:21:11.088 INFO:teuthology.orchestra.run.smithi120.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-24T01:21:11.088 INFO:teuthology.orchestra.run.smithi120.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T01:21:11.088 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-24T01:21:11.088 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-24T01:21:11.088 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-24T01:21:11.088 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-24T01:21:11.088 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-24T01:21:11.088 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-24T01:21:11.088 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-24T01:21:11.088 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-24T01:21:11.088 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T01:21:11.088 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T01:21:11.088 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T01:21:11.089 INFO:teuthology.orchestra.run.smithi120.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T01:21:11.089 INFO:teuthology.orchestra.run.smithi120.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T01:21:11.089 INFO:teuthology.orchestra.run.smithi120.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-24T01:21:11.089 INFO:teuthology.orchestra.run.smithi120.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T01:21:11.089 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T01:21:11.089 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T01:21:11.089 INFO:teuthology.orchestra.run.smithi120.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-24T01:21:11.089 INFO:teuthology.orchestra.run.smithi120.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T01:21:11.089 INFO:teuthology.orchestra.run.smithi120.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T01:21:11.089 INFO:teuthology.orchestra.run.smithi120.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T01:21:11.089 INFO:teuthology.orchestra.run.smithi120.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T01:21:11.089 INFO:teuthology.orchestra.run.smithi120.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T01:21:11.089 INFO:teuthology.orchestra.run.smithi120.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-24T01:21:11.089 INFO:teuthology.orchestra.run.smithi120.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T01:21:11.090 INFO:teuthology.orchestra.run.smithi120.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-24T01:21:11.090 INFO:teuthology.orchestra.run.smithi120.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T01:21:11.090 INFO:teuthology.orchestra.run.smithi120.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-24T01:21:11.090 INFO:teuthology.orchestra.run.smithi120.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T01:21:11.090 INFO:teuthology.orchestra.run.smithi120.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-24T01:21:11.090 INFO:teuthology.orchestra.run.smithi120.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-24T01:21:11.090 INFO:teuthology.orchestra.run.smithi120.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-24T01:21:11.090 INFO:teuthology.orchestra.run.smithi120.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T01:21:11.090 INFO:teuthology.orchestra.run.smithi120.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T01:21:11.090 INFO:teuthology.orchestra.run.smithi120.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T01:21:11.090 INFO:teuthology.orchestra.run.smithi120.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T01:21:11.090 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-24T01:21:11.090 INFO:teuthology.orchestra.run.smithi120.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-24T01:21:11.090 INFO:teuthology.orchestra.run.smithi120.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-24T01:21:11.091 INFO:teuthology.orchestra.run.smithi120.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T01:21:11.091 INFO:teuthology.orchestra.run.smithi120.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-24T01:21:11.091 INFO:teuthology.orchestra.run.smithi120.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T01:21:11.091 INFO:teuthology.orchestra.run.smithi120.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T01:21:11.091 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T01:21:11.091 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-24T01:21:11.091 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-24T01:21:11.091 INFO:teuthology.orchestra.run.smithi120.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T01:21:11.091 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T01:21:11.091 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T01:21:11.091 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T01:21:11.091 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-24T01:21:11.091 INFO:teuthology.orchestra.run.smithi120.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-24T01:21:11.091 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T01:21:11.092 INFO:teuthology.orchestra.run.smithi120.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T01:21:11.092 INFO:teuthology.orchestra.run.smithi120.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T01:21:11.092 INFO:teuthology.orchestra.run.smithi120.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-24T01:21:11.092 INFO:teuthology.orchestra.run.smithi120.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-24T01:21:11.092 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T01:21:11.092 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T01:21:11.092 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T01:21:11.092 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T01:21:11.092 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T01:21:11.092 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T01:21:11.092 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T01:21:11.092 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T01:21:11.092 INFO:teuthology.orchestra.run.smithi120.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T01:21:11.092 INFO:teuthology.orchestra.run.smithi120.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T01:21:11.093 INFO:teuthology.orchestra.run.smithi120.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T01:21:11.093 INFO:teuthology.orchestra.run.smithi120.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T01:21:11.093 INFO:teuthology.orchestra.run.smithi120.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T01:21:11.093 INFO:teuthology.orchestra.run.smithi120.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T01:21:11.093 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T01:21:11.093 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T01:21:11.093 INFO:teuthology.orchestra.run.smithi120.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T01:21:11.093 INFO:teuthology.orchestra.run.smithi120.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-24T01:21:11.093 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T01:21:11.093 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T01:21:11.093 INFO:teuthology.orchestra.run.smithi120.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T01:21:11.093 INFO:teuthology.orchestra.run.smithi120.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T01:21:11.093 INFO:teuthology.orchestra.run.smithi120.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-24T01:21:11.093 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T01:21:11.094 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T01:21:11.094 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T01:21:11.094 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T01:21:11.094 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-24T01:21:11.094 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T01:21:11.094 INFO:teuthology.orchestra.run.smithi120.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T01:21:11.094 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T01:21:11.094 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T01:21:11.094 INFO:teuthology.orchestra.run.smithi120.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T01:21:11.094 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T01:21:11.094 INFO:teuthology.orchestra.run.smithi120.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T01:21:11.094 INFO:teuthology.orchestra.run.smithi120.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T01:21:11.094 INFO:teuthology.orchestra.run.smithi120.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T01:21:11.095 INFO:teuthology.orchestra.run.smithi120.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T01:21:11.095 INFO:teuthology.orchestra.run.smithi120.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T01:21:11.095 INFO:teuthology.orchestra.run.smithi120.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-24T01:21:11.095 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T01:21:11.095 INFO:teuthology.orchestra.run.smithi120.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T01:21:11.095 INFO:teuthology.orchestra.run.smithi120.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T01:21:11.095 INFO:teuthology.orchestra.run.smithi120.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T01:21:11.095 INFO:teuthology.orchestra.run.smithi120.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-24T01:21:11.095 INFO:teuthology.orchestra.run.smithi120.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-24T01:21:11.095 INFO:teuthology.orchestra.run.smithi120.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T01:21:11.095 INFO:teuthology.orchestra.run.smithi120.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T01:21:11.095 INFO:teuthology.orchestra.run.smithi120.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T01:21:11.095 INFO:teuthology.orchestra.run.smithi120.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-24T01:21:11.095 INFO:teuthology.orchestra.run.smithi120.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T01:21:11.096 INFO:teuthology.orchestra.run.smithi120.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-24T01:21:11.096 INFO:teuthology.orchestra.run.smithi120.stdout:Installing weak dependencies: 2024-03-24T01:21:11.096 INFO:teuthology.orchestra.run.smithi120.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-24T01:21:11.096 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:21:11.096 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-24T01:21:11.096 INFO:teuthology.orchestra.run.smithi120.stdout:============================================================================================ 2024-03-24T01:21:11.096 INFO:teuthology.orchestra.run.smithi120.stdout:Install 150 Packages 2024-03-24T01:21:11.096 INFO:teuthology.orchestra.run.smithi120.stdout:Upgrade 24 Packages 2024-03-24T01:21:11.096 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:21:11.097 INFO:teuthology.orchestra.run.smithi090.stdout:(59/174): postgresql-13.11-1.el9.x86_64.rpm 3.1 MB/s | 1.5 MB 00:00 2024-03-24T01:21:11.097 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 224 M 2024-03-24T01:21:11.097 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-24T01:21:11.129 INFO:teuthology.orchestra.run.smithi090.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 856 kB/s | 134 kB 00:00 2024-03-24T01:21:11.313 INFO:teuthology.orchestra.run.smithi090.stdout:(61/174): postgresql-server-13.11-1.el9.x86_64. 21 MB/s | 5.7 MB 00:00 2024-03-24T01:21:11.347 INFO:teuthology.orchestra.run.smithi090.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 393 kB/s | 78 kB 00:00 2024-03-24T01:21:11.414 INFO:teuthology.orchestra.run.smithi090.stdout:(63/174): protobuf-3.14.0-13.el9.x86_64.rpm 3.8 MB/s | 1.0 MB 00:00 2024-03-24T01:21:11.573 INFO:teuthology.orchestra.run.smithi090.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-24T01:21:11.598 INFO:teuthology.orchestra.run.smithi090.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 105 kB/s | 24 kB 00:00 2024-03-24T01:21:11.623 INFO:teuthology.orchestra.run.smithi090.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 94 kB/s | 18 kB 00:00 2024-03-24T01:21:11.878 INFO:teuthology.orchestra.run.smithi090.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 124 kB/s | 32 kB 00:00 2024-03-24T01:21:11.949 INFO:teuthology.orchestra.run.smithi090.stdout:(68/174): python3-mako-1.1.4-6.el9.noarch.rpm 445 kB/s | 144 kB 00:00 2024-03-24T01:21:12.074 INFO:teuthology.orchestra.run.smithi090.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 4.8 MB 00:00 2024-03-24T01:21:12.108 INFO:teuthology.orchestra.run.smithi090.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 968 kB/s | 217 kB 00:00 2024-03-24T01:21:12.133 INFO:teuthology.orchestra.run.smithi090.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.0 MB/s | 174 kB 00:00 2024-03-24T01:21:12.184 INFO:teuthology.orchestra.run.smithi090.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 830 kB/s | 41 kB 00:00 2024-03-24T01:21:12.218 INFO:teuthology.orchestra.run.smithi090.stdout:(73/174): python3-packaging-20.9-5.el9.noarch.r 821 kB/s | 69 kB 00:00 2024-03-24T01:21:12.252 INFO:teuthology.orchestra.run.smithi090.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 2.0 MB/s | 238 kB 00:00 2024-03-24T01:21:12.287 INFO:teuthology.orchestra.run.smithi090.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 132 kB 00:00 2024-03-24T01:21:12.310 INFO:teuthology.orchestra.run.smithi090.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 841 kB/s | 47 kB 00:00 2024-03-24T01:21:12.344 INFO:teuthology.orchestra.run.smithi090.stdout:(77/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.3 MB/s | 211 kB 00:00 2024-03-24T01:21:12.406 INFO:teuthology.orchestra.run.smithi090.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 360 kB/s | 43 kB 00:00 2024-03-24T01:21:12.570 INFO:teuthology.orchestra.run.smithi090.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 259 kB/s | 39 kB 00:00 2024-03-24T01:21:12.595 INFO:teuthology.orchestra.run.smithi090.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 235 kB/s | 41 kB 00:00 2024-03-24T01:21:12.829 INFO:teuthology.orchestra.run.smithi090.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 30 MB/s | 16 MB 00:00 2024-03-24T01:21:12.879 INFO:teuthology.orchestra.run.smithi090.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 977 kB/s | 300 kB 00:00 2024-03-24T01:21:12.904 INFO:teuthology.orchestra.run.smithi090.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 189 kB/s | 56 kB 00:00 2024-03-24T01:21:12.930 INFO:teuthology.orchestra.run.smithi090.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.3 MB/s | 63 kB 00:00 2024-03-24T01:21:12.955 INFO:teuthology.orchestra.run.smithi090.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 599 kB/s | 44 kB 00:00 2024-03-24T01:21:12.980 INFO:teuthology.orchestra.run.smithi090.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-03-24T01:21:13.022 INFO:teuthology.orchestra.run.smithi090.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.6 MB/s | 241 kB 00:00 2024-03-24T01:21:13.072 INFO:teuthology.orchestra.run.smithi090.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 9.9 MB/s | 1.1 MB 00:00 2024-03-24T01:21:13.098 INFO:teuthology.orchestra.run.smithi090.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 997 kB/s | 103 kB 00:00 2024-03-24T01:21:13.123 INFO:teuthology.orchestra.run.smithi090.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 2.5 MB/s | 124 kB 00:00 2024-03-24T01:21:13.148 INFO:teuthology.orchestra.run.smithi090.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.5 MB/s | 187 kB 00:00 2024-03-24T01:21:13.274 INFO:teuthology.orchestra.run.smithi090.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-24T01:21:13.341 INFO:teuthology.orchestra.run.smithi090.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-03-24T01:21:13.374 INFO:teuthology.orchestra.run.smithi090.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.2 MB/s | 125 kB 00:00 2024-03-24T01:21:13.405 INFO:teuthology.orchestra.run.smithi120.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 23 kB/s | 6.6 kB 00:00 2024-03-24T01:21:13.425 INFO:teuthology.orchestra.run.smithi090.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.8 MB/s | 803 kB 00:00 2024-03-24T01:21:13.483 INFO:teuthology.orchestra.run.smithi090.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 152 kB/s | 21 kB 00:00 2024-03-24T01:21:13.509 INFO:teuthology.orchestra.run.smithi090.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 191 kB/s | 25 kB 00:00 2024-03-24T01:21:13.642 INFO:teuthology.orchestra.run.smithi090.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.8 MB/s | 245 kB 00:00 2024-03-24T01:21:13.677 INFO:teuthology.orchestra.run.smithi090.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 1.5 MB/s | 248 kB 00:00 2024-03-24T01:21:13.736 INFO:teuthology.orchestra.run.smithi090.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 607 kB/s | 35 kB 00:00 2024-03-24T01:21:13.817 INFO:teuthology.orchestra.run.smithi090.stdout:(101/174): qatlib-23.11.0-1.el9.x86_64.rpm 1.2 MB/s | 216 kB 00:00 2024-03-24T01:21:13.870 INFO:teuthology.orchestra.run.smithi090.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 335 kB/s | 44 kB 00:00 2024-03-24T01:21:13.912 INFO:teuthology.orchestra.run.smithi090.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.0 MB/s | 94 kB 00:00 2024-03-24T01:21:13.981 INFO:teuthology.orchestra.run.smithi120.stdout:(2/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.5 MB/s | 890 kB 00:00 2024-03-24T01:21:14.029 INFO:teuthology.orchestra.run.smithi090.stdout:(104/174): openssl-devel-3.0.7-27.el9.x86_64.rp 6.9 MB/s | 4.1 MB 00:00 2024-03-24T01:21:14.097 INFO:teuthology.orchestra.run.smithi120.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 1.2 MB/s | 145 kB 00:00 2024-03-24T01:21:14.179 INFO:teuthology.orchestra.run.smithi090.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 72 kB/s | 22 kB 00:00 2024-03-24T01:21:14.346 INFO:teuthology.orchestra.run.smithi090.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 1.9 MB/s | 863 kB 00:00 2024-03-24T01:21:14.465 INFO:teuthology.orchestra.run.smithi120.stdout:(4/174): ceph-common-19.0.0-2145.g3be9a5c9.el9. 15 MB/s | 20 MB 00:01 2024-03-24T01:21:14.538 INFO:teuthology.orchestra.run.smithi090.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-03-24T01:21:14.598 INFO:teuthology.orchestra.run.smithi120.stdout:(5/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 11 MB/s | 1.5 MB 00:00 2024-03-24T01:21:14.656 INFO:teuthology.orchestra.run.smithi090.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 649 kB/s | 309 kB 00:00 2024-03-24T01:21:14.681 INFO:teuthology.orchestra.run.smithi090.stdout:(109/174): grpc-data-1.46.7-10.el9.noarch.rpm 59 kB/s | 19 kB 00:00 2024-03-24T01:21:14.740 INFO:teuthology.orchestra.run.smithi090.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 313 kB/s | 26 kB 00:00 2024-03-24T01:21:14.874 INFO:teuthology.orchestra.run.smithi090.stdout:(111/174): libarrow-9.0.0-10.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-03-24T01:21:14.898 INFO:teuthology.orchestra.run.smithi120.stdout:(6/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 16 MB/s | 4.7 MB 00:00 2024-03-24T01:21:14.908 INFO:teuthology.orchestra.run.smithi090.stdout:(112/174): liboath-2.6.7-2.el9.x86_64.rpm 215 kB/s | 49 kB 00:00 2024-03-24T01:21:14.933 INFO:teuthology.orchestra.run.smithi090.stdout:(113/174): libunwind-1.6.2-1.el9.x86_64.rpm 350 kB/s | 67 kB 00:00 2024-03-24T01:21:14.973 INFO:teuthology.orchestra.run.smithi120.stdout:(7/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 2.6 MB/s | 2.3 MB 00:00 2024-03-24T01:21:14.975 INFO:teuthology.orchestra.run.smithi090.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-03-24T01:21:15.059 INFO:teuthology.orchestra.run.smithi090.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 523 kB/s | 43 kB 00:00 2024-03-24T01:21:15.109 INFO:teuthology.orchestra.run.smithi090.stdout:(116/174): python3-asyncssh-2.13.2-5.el9.noarch 3.1 MB/s | 548 kB 00:00 2024-03-24T01:21:15.185 INFO:teuthology.orchestra.run.smithi090.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 257 kB/s | 32 kB 00:00 2024-03-24T01:21:15.218 INFO:teuthology.orchestra.run.smithi090.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 138 kB/s | 15 kB 00:00 2024-03-24T01:21:15.261 INFO:teuthology.orchestra.run.smithi090.stdout:(119/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.3 MB/s | 838 kB 00:00 2024-03-24T01:21:15.290 INFO:teuthology.orchestra.run.smithi120.stdout:(8/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 2.5 MB/s | 5.4 MB 00:02 2024-03-24T01:21:15.294 INFO:teuthology.orchestra.run.smithi090.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.6 MB/s | 172 kB 00:00 2024-03-24T01:21:15.336 INFO:teuthology.orchestra.run.smithi090.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-03-24T01:21:15.357 INFO:teuthology.orchestra.run.smithi120.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 384 kB/s | 25 kB 00:00 2024-03-24T01:21:15.370 INFO:teuthology.orchestra.run.smithi090.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.7 MB/s | 303 kB 00:00 2024-03-24T01:21:15.395 INFO:teuthology.orchestra.run.smithi090.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 353 kB/s | 35 kB 00:00 2024-03-24T01:21:15.446 INFO:teuthology.orchestra.run.smithi090.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 1.8 MB/s | 203 kB 00:00 2024-03-24T01:21:15.496 INFO:teuthology.orchestra.run.smithi090.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-03-24T01:21:15.521 INFO:teuthology.orchestra.run.smithi090.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 143 kB/s | 11 kB 00:00 2024-03-24T01:21:15.565 INFO:teuthology.orchestra.run.smithi090.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 262 kB/s | 18 kB 00:00 2024-03-24T01:21:15.590 INFO:teuthology.orchestra.run.smithi090.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 342 kB/s | 23 kB 00:00 2024-03-24T01:21:15.690 INFO:teuthology.orchestra.run.smithi090.stdout:(129/174): python3-grpcio-1.46.7-10.el9.x86_64. 6.4 MB/s | 2.0 MB 00:00 2024-03-24T01:21:15.716 INFO:teuthology.orchestra.run.smithi090.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 129 kB/s | 19 kB 00:00 2024-03-24T01:21:15.741 INFO:teuthology.orchestra.run.smithi090.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 131 kB/s | 20 kB 00:00 2024-03-24T01:21:15.808 INFO:teuthology.orchestra.run.smithi090.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 8.8 MB/s | 1.0 MB 00:00 2024-03-24T01:21:15.833 INFO:teuthology.orchestra.run.smithi090.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 508 kB/s | 46 kB 00:00 2024-03-24T01:21:15.859 INFO:teuthology.orchestra.run.smithi090.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 678 kB/s | 79 kB 00:00 2024-03-24T01:21:15.890 INFO:teuthology.orchestra.run.smithi120.stdout:(10/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x8 17 MB/s | 17 MB 00:00 2024-03-24T01:21:15.901 INFO:teuthology.orchestra.run.smithi090.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 632 kB/s | 58 kB 00:00 2024-03-24T01:21:15.946 INFO:teuthology.orchestra.run.smithi090.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.5 MB/s | 272 kB 00:00 2024-03-24T01:21:15.957 INFO:teuthology.orchestra.run.smithi120.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 3.3 MB/s | 220 kB 00:00 2024-03-24T01:21:15.985 INFO:teuthology.orchestra.run.smithi090.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 131 kB/s | 16 kB 00:00 2024-03-24T01:21:16.008 INFO:teuthology.orchestra.run.smithi120.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 549 kB/s | 27 kB 00:00 2024-03-24T01:21:16.018 INFO:teuthology.orchestra.run.smithi090.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 770 kB/s | 90 kB 00:00 2024-03-24T01:21:16.044 INFO:teuthology.orchestra.run.smithi090.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 281 kB/s | 28 kB 00:00 2024-03-24T01:21:16.075 INFO:teuthology.orchestra.run.smithi120.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 475 kB/s | 31 kB 00:00 2024-03-24T01:21:16.127 INFO:teuthology.orchestra.run.smithi090.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 218 kB/s | 31 kB 00:00 2024-03-24T01:21:16.161 INFO:teuthology.orchestra.run.smithi090.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 508 kB/s | 59 kB 00:00 2024-03-24T01:21:16.195 INFO:teuthology.orchestra.run.smithi090.stdout:(142/174): python3-routes-2.5.1-5.el9.noarch.rp 1.0 MB/s | 188 kB 00:00 2024-03-24T01:21:16.217 INFO:teuthology.orchestra.run.smithi120.stdout:(14/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 5.2 MB/s | 746 kB 00:00 2024-03-24T01:21:16.229 INFO:teuthology.orchestra.run.smithi090.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 355 kB/s | 36 kB 00:00 2024-03-24T01:21:16.271 INFO:teuthology.orchestra.run.smithi090.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 470 kB/s | 51 kB 00:00 2024-03-24T01:21:16.275 INFO:teuthology.orchestra.run.smithi120.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 2.8 MB/s | 163 kB 00:00 2024-03-24T01:21:16.304 INFO:teuthology.orchestra.run.smithi090.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-03-24T01:21:16.338 INFO:teuthology.orchestra.run.smithi090.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 827 kB/s | 90 kB 00:00 2024-03-24T01:21:16.350 INFO:teuthology.orchestra.run.smithi120.stdout:(16/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.7 MB/s | 127 kB 00:00 2024-03-24T01:21:16.380 INFO:teuthology.orchestra.run.smithi090.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.9 MB/s | 427 kB 00:00 2024-03-24T01:21:16.414 INFO:teuthology.orchestra.run.smithi090.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 184 kB/s | 20 kB 00:00 2024-03-24T01:21:16.447 INFO:teuthology.orchestra.run.smithi090.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-03-24T01:21:16.450 INFO:teuthology.orchestra.run.smithi120.stdout:(17/174): libradosstriper1-19.0.0-2145.g3be9a5c 4.9 MB/s | 501 kB 00:00 2024-03-24T01:21:16.531 INFO:teuthology.orchestra.run.smithi090.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-24T01:21:16.591 INFO:teuthology.orchestra.run.smithi090.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 5.3 MB/s | 321 kB 00:00 2024-03-24T01:21:16.825 INFO:teuthology.orchestra.run.smithi090.stdout:(152/174): NetworkManager-1.47.2-1.el9.x86_64.r 9.9 MB/s | 2.3 MB 00:00 2024-03-24T01:21:16.925 INFO:teuthology.orchestra.run.smithi090.stdout:(153/174): NetworkManager-libnm-1.47.2-1.el9.x8 19 MB/s | 1.8 MB 00:00 2024-03-24T01:21:16.959 INFO:teuthology.orchestra.run.smithi120.stdout:(18/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 10 MB/s | 5.1 MB 00:00 2024-03-24T01:21:16.976 INFO:teuthology.orchestra.run.smithi090.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 855 kB/s | 42 kB 00:00 2024-03-24T01:21:17.018 INFO:teuthology.orchestra.run.smithi120.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 789 kB/s | 45 kB 00:00 2024-03-24T01:21:17.051 INFO:teuthology.orchestra.run.smithi090.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 3.3 MB/s | 248 kB 00:00 2024-03-24T01:21:17.085 INFO:teuthology.orchestra.run.smithi120.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 1.9 MB/s | 131 kB 00:00 2024-03-24T01:21:17.101 INFO:teuthology.orchestra.run.smithi090.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2024-03-24T01:21:17.152 INFO:teuthology.orchestra.run.smithi090.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 740 kB/s | 36 kB 00:00 2024-03-24T01:21:17.160 INFO:teuthology.orchestra.run.smithi120.stdout:(21/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 2.1 MB/s | 162 kB 00:00 2024-03-24T01:21:17.202 INFO:teuthology.orchestra.run.smithi090.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 3.8 MB/s | 192 kB 00:00 2024-03-24T01:21:17.261 INFO:teuthology.orchestra.run.smithi090.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 4.3 MB/s | 256 kB 00:00 2024-03-24T01:21:17.275 INFO:teuthology.orchestra.run.smithi120.stdout:(22/174): python3-rados-19.0.0-2145.g3be9a5c9.e 2.7 MB/s | 321 kB 00:00 2024-03-24T01:21:17.312 INFO:teuthology.orchestra.run.smithi090.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 465 kB/s | 23 kB 00:00 2024-03-24T01:21:17.352 INFO:teuthology.orchestra.run.smithi120.stdout:(23/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 3.8 MB/s | 298 kB 00:00 2024-03-24T01:21:17.370 INFO:teuthology.orchestra.run.smithi090.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 108 kB 00:00 2024-03-24T01:21:17.419 INFO:teuthology.orchestra.run.smithi120.stdout:(24/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 1.5 MB/s | 99 kB 00:00 2024-03-24T01:21:17.503 INFO:teuthology.orchestra.run.smithi090.stdout:(162/174): librados2-19.0.0-2145.g3be9a5c9.el9. 3.1 MB/s | 3.4 MB 00:01 2024-03-24T01:21:17.538 INFO:teuthology.orchestra.run.smithi090.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 931 kB/s | 155 kB 00:00 2024-03-24T01:21:17.586 INFO:teuthology.orchestra.run.smithi120.stdout:(25/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el 3.8 MB/s | 9.9 MB 00:02 2024-03-24T01:21:17.588 INFO:teuthology.orchestra.run.smithi090.stdout:(164/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.3 MB/s | 64 kB 00:00 2024-03-24T01:21:17.611 INFO:teuthology.orchestra.run.smithi120.stdout:(26/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 447 kB/s | 85 kB 00:00 2024-03-24T01:21:17.622 INFO:teuthology.orchestra.run.smithi090.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 136 kB 00:00 2024-03-24T01:21:17.656 INFO:teuthology.orchestra.run.smithi090.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 433 kB/s | 29 kB 00:00 2024-03-24T01:21:17.670 INFO:teuthology.orchestra.run.smithi120.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 2.9 MB/s | 170 kB 00:00 2024-03-24T01:21:17.720 INFO:teuthology.orchestra.run.smithi120.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 691 kB/s | 34 kB 00:00 2024-03-24T01:21:17.781 INFO:teuthology.orchestra.run.smithi090.stdout:(167/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 17 MB/s | 2.2 MB 00:00 2024-03-24T01:21:17.787 INFO:teuthology.orchestra.run.smithi120.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 2.0 MB/s | 139 kB 00:00 2024-03-24T01:21:17.840 INFO:teuthology.orchestra.run.smithi090.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 5.6 MB/s | 1.2 MB 00:00 2024-03-24T01:21:17.908 INFO:teuthology.orchestra.run.smithi090.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 6.8 MB/s | 465 kB 00:00 2024-03-24T01:21:17.992 INFO:teuthology.orchestra.run.smithi090.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 11 MB/s | 2.3 MB 00:00 2024-03-24T01:21:18.017 INFO:teuthology.orchestra.run.smithi090.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 288 kB/s | 31 kB 00:00 2024-03-24T01:21:18.021 INFO:teuthology.orchestra.run.smithi120.stdout:(30/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 7.7 MB/s | 1.8 MB 00:00 2024-03-24T01:21:18.101 INFO:teuthology.orchestra.run.smithi090.stdout:(172/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 1.9 MB/s | 3.1 MB 00:01 2024-03-24T01:21:18.169 INFO:teuthology.orchestra.run.smithi090.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 6.1 MB/s | 1.1 MB 00:00 2024-03-24T01:21:18.193 INFO:teuthology.orchestra.run.smithi090.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 426 kB/s | 74 kB 00:00 2024-03-24T01:21:18.211 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-24T01:21:18.211 INFO:teuthology.orchestra.run.smithi090.stdout:Total 13 MB/s | 224 MB 00:16 2024-03-24T01:21:18.362 INFO:teuthology.orchestra.run.smithi120.stdout:(31/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 4.0 MB/s | 3.1 MB 00:00 2024-03-24T01:21:18.471 INFO:teuthology.orchestra.run.smithi120.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2145.g3b 2.3 MB/s | 248 kB 00:00 2024-03-24T01:21:18.563 INFO:teuthology.orchestra.run.smithi120.stdout:(33/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 544 kB/s | 49 kB 00:00 2024-03-24T01:21:18.647 INFO:teuthology.orchestra.run.smithi120.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2145.g3 200 kB/s | 16 kB 00:00 2024-03-24T01:21:18.780 INFO:teuthology.orchestra.run.smithi120.stdout:(35/174): ceph-mgr-diskprediction-local-19.0.0- 9.7 MB/s | 7.4 MB 00:00 2024-03-24T01:21:18.814 INFO:teuthology.orchestra.run.smithi120.stdout:(36/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 1.6 MB/s | 264 kB 00:00 2024-03-24T01:21:18.898 INFO:teuthology.orchestra.run.smithi120.stdout:(37/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 6.3 MB/s | 748 kB 00:00 2024-03-24T01:21:18.923 INFO:teuthology.orchestra.run.smithi120.stdout:(38/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 169 kB/s | 18 kB 00:00 2024-03-24T01:21:19.566 INFO:teuthology.orchestra.run.smithi120.stdout:(39/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 11 MB/s | 48 MB 00:04 2024-03-24T01:21:19.638 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-24T01:21:19.657 INFO:teuthology.orchestra.run.smithi120.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 1.2 MB/s | 920 kB 00:00 2024-03-24T01:21:19.699 INFO:teuthology.orchestra.run.smithi120.stdout:(41/174): automake-1.16.2-8.el9.noarch.rpm 827 kB/s | 662 kB 00:00 2024-03-24T01:21:19.724 INFO:teuthology.orchestra.run.smithi120.stdout:(42/174): boost-program-options-1.75.0-8.el9.x8 687 kB/s | 106 kB 00:00 2024-03-24T01:21:19.758 INFO:teuthology.orchestra.run.smithi120.stdout:(43/174): flex-2.6.4-9.el9.x86_64.rpm 3.1 MB/s | 307 kB 00:00 2024-03-24T01:21:19.800 INFO:teuthology.orchestra.run.smithi120.stdout:(44/174): flexiblas-3.0.4-8.el9.x86_64.rpm 442 kB/s | 32 kB 00:00 2024-03-24T01:21:19.900 INFO:teuthology.orchestra.run.smithi120.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 17 MB/s | 3.0 MB 00:00 2024-03-24T01:21:19.925 INFO:teuthology.orchestra.run.smithi120.stdout:(46/174): flexiblas-openblas-openmp-3.0.4-8.el9 113 kB/s | 17 kB 00:00 2024-03-24T01:21:19.950 INFO:teuthology.orchestra.run.smithi120.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 73 kB/s | 10 kB 00:00 2024-03-24T01:21:19.975 INFO:teuthology.orchestra.run.smithi120.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.2 MB/s | 159 kB 00:00 2024-03-24T01:21:20.001 INFO:teuthology.orchestra.run.smithi120.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 597 kB/s | 44 kB 00:00 2024-03-24T01:21:20.013 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-24T01:21:20.013 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-24T01:21:20.042 INFO:teuthology.orchestra.run.smithi120.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 7.2 MB/s | 661 kB 00:00 2024-03-24T01:21:20.068 INFO:teuthology.orchestra.run.smithi120.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 768 kB/s | 46 kB 00:00 2024-03-24T01:21:20.109 INFO:teuthology.orchestra.run.smithi120.stdout:(52/174): libtool-2.4.6-45.el9.x86_64.rpm 5.5 MB/s | 578 kB 00:00 2024-03-24T01:21:20.135 INFO:teuthology.orchestra.run.smithi120.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-03-24T01:21:20.168 INFO:teuthology.orchestra.run.smithi120.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 2.0 MB/s | 187 kB 00:00 2024-03-24T01:21:20.202 INFO:teuthology.orchestra.run.smithi120.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 528 kB/s | 34 kB 00:00 2024-03-24T01:21:20.253 INFO:teuthology.orchestra.run.smithi120.stdout:(56/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.8 MB/s | 217 kB 00:00 2024-03-24T01:21:20.394 INFO:teuthology.orchestra.run.smithi120.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 21 MB/s | 4.7 MB 00:00 2024-03-24T01:21:20.445 INFO:teuthology.orchestra.run.smithi120.stdout:(58/174): postgresql-13.11-1.el9.x86_64.rpm 6.8 MB/s | 1.5 MB 00:00 2024-03-24T01:21:20.478 INFO:teuthology.orchestra.run.smithi120.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 3.7 MB/s | 811 kB 00:00 2024-03-24T01:21:20.512 INFO:teuthology.orchestra.run.smithi120.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.1 MB/s | 134 kB 00:00 2024-03-24T01:21:20.603 INFO:teuthology.orchestra.run.smithi120.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 8.3 MB/s | 1.0 MB 00:00 2024-03-24T01:21:20.779 INFO:teuthology.orchestra.run.smithi120.stdout:(62/174): postgresql-server-13.11-1.el9.x86_64. 19 MB/s | 5.7 MB 00:00 2024-03-24T01:21:20.804 INFO:teuthology.orchestra.run.smithi120.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 267 kB/s | 78 kB 00:00 2024-03-24T01:21:20.955 INFO:teuthology.orchestra.run.smithi120.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2024-03-24T01:21:20.971 INFO:teuthology.orchestra.run.smithi120.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 161 kB/s | 24 kB 00:00 2024-03-24T01:21:20.988 INFO:teuthology.orchestra.run.smithi120.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 108 kB/s | 18 kB 00:00 2024-03-24T01:21:21.022 INFO:teuthology.orchestra.run.smithi120.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 1.0 MB/s | 32 kB 00:00 2024-03-24T01:21:21.072 INFO:teuthology.orchestra.run.smithi120.stdout:(68/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 144 kB 00:00 2024-03-24T01:21:21.206 INFO:teuthology.orchestra.run.smithi120.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 22 MB/s | 4.8 MB 00:00 2024-03-24T01:21:21.231 INFO:teuthology.orchestra.run.smithi120.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.1 MB/s | 217 kB 00:00 2024-03-24T01:21:21.256 INFO:teuthology.orchestra.run.smithi120.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.0 MB/s | 174 kB 00:00 2024-03-24T01:21:21.299 INFO:teuthology.orchestra.run.smithi120.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 1.7 MB/s | 69 kB 00:00 2024-03-24T01:21:21.323 INFO:teuthology.orchestra.run.smithi120.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 631 kB/s | 41 kB 00:00 2024-03-24T01:21:21.348 INFO:teuthology.orchestra.run.smithi120.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 2.6 MB/s | 238 kB 00:00 2024-03-24T01:21:21.390 INFO:teuthology.orchestra.run.smithi120.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.2 MB/s | 132 kB 00:00 2024-03-24T01:21:21.415 INFO:teuthology.orchestra.run.smithi120.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 728 kB/s | 47 kB 00:00 2024-03-24T01:21:21.449 INFO:teuthology.orchestra.run.smithi120.stdout:(77/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.1 MB/s | 211 kB 00:00 2024-03-24T01:21:21.499 INFO:teuthology.orchestra.run.smithi120.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 528 kB/s | 43 kB 00:00 2024-03-24T01:21:21.650 INFO:teuthology.orchestra.run.smithi120.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 199 kB/s | 39 kB 00:00 2024-03-24T01:21:21.850 INFO:teuthology.orchestra.run.smithi120.stdout:(80/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 36 MB/s | 16 MB 00:00 2024-03-24T01:21:21.875 INFO:teuthology.orchestra.run.smithi120.stdout:(81/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 114 kB/s | 41 kB 00:00 2024-03-24T01:21:21.908 INFO:teuthology.orchestra.run.smithi120.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.2 MB/s | 300 kB 00:00 2024-03-24T01:21:21.933 INFO:teuthology.orchestra.run.smithi120.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 1.0 MB/s | 56 kB 00:00 2024-03-24T01:21:21.960 INFO:teuthology.orchestra.run.smithi120.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 785 kB/s | 63 kB 00:00 2024-03-24T01:21:21.984 INFO:teuthology.orchestra.run.smithi120.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 591 kB/s | 44 kB 00:00 2024-03-24T01:21:22.017 INFO:teuthology.orchestra.run.smithi120.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.1 MB/s | 241 kB 00:00 2024-03-24T01:21:22.043 INFO:teuthology.orchestra.run.smithi120.stdout:(87/174): fuse-2.9.9-15.el9.x86_64.rpm 957 kB/s | 79 kB 00:00 2024-03-24T01:21:22.085 INFO:teuthology.orchestra.run.smithi120.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.1 MB 00:00 2024-03-24T01:21:22.110 INFO:teuthology.orchestra.run.smithi120.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-24T01:21:22.144 INFO:teuthology.orchestra.run.smithi120.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 2.1 MB/s | 124 kB 00:00 2024-03-24T01:21:22.168 INFO:teuthology.orchestra.run.smithi120.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2024-03-24T01:21:22.577 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-24T01:21:22.578 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-24T01:21:22.594 INFO:teuthology.orchestra.run.smithi120.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 390 kB/s | 187 kB 00:00 2024-03-24T01:21:22.802 INFO:teuthology.orchestra.run.smithi120.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 309 kB/s | 195 kB 00:00 2024-03-24T01:21:22.853 INFO:teuthology.orchestra.run.smithi120.stdout:(94/174): libgfortran-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2024-03-24T01:21:22.928 INFO:teuthology.orchestra.run.smithi120.stdout:(95/174): python3-requests-2.25.1-8.el9.noarch. 376 kB/s | 125 kB 00:00 2024-03-24T01:21:23.470 INFO:teuthology.orchestra.run.smithi120.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 32 kB/s | 21 kB 00:00 2024-03-24T01:21:23.570 INFO:teuthology.orchestra.run.smithi120.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 36 kB/s | 25 kB 00:00 2024-03-24T01:21:24.471 INFO:teuthology.orchestra.run.smithi120.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 245 kB/s | 245 kB 00:00 2024-03-24T01:21:24.512 INFO:teuthology.orchestra.run.smithi120.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 264 kB/s | 248 kB 00:00 2024-03-24T01:21:25.196 INFO:teuthology.orchestra.run.smithi120.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 51 kB/s | 35 kB 00:00 2024-03-24T01:21:25.379 INFO:teuthology.orchestra.run.smithi120.stdout:(101/174): qatlib-23.11.0-1.el9.x86_64.rpm 239 kB/s | 216 kB 00:00 2024-03-24T01:21:25.946 INFO:teuthology.orchestra.run.smithi120.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 59 kB/s | 44 kB 00:00 2024-03-24T01:21:26.063 INFO:teuthology.orchestra.run.smithi120.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 194 kB/s | 22 kB 00:00 2024-03-24T01:21:26.214 INFO:teuthology.orchestra.run.smithi120.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 112 kB/s | 94 kB 00:00 2024-03-24T01:21:26.322 INFO:teuthology.orchestra.run.smithi120.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 3.3 MB/s | 863 kB 00:00 2024-03-24T01:21:27.431 INFO:teuthology.orchestra.run.smithi120.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 297 kB/s | 309 kB 00:01 2024-03-24T01:21:28.406 INFO:teuthology.orchestra.run.smithi120.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 250 kB/s | 548 kB 00:02 2024-03-24T01:21:28.531 INFO:teuthology.orchestra.run.smithi120.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 18 kB/s | 19 kB 00:01 2024-03-24T01:21:28.739 INFO:teuthology.orchestra.run.smithi120.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 125 kB/s | 26 kB 00:00 2024-03-24T01:21:28.956 INFO:teuthology.orchestra.run.smithi120.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 225 kB/s | 49 kB 00:00 2024-03-24T01:21:29.182 INFO:teuthology.orchestra.run.smithi120.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 300 kB/s | 67 kB 00:00 2024-03-24T01:21:29.390 INFO:teuthology.orchestra.run.smithi120.stdout:(112/174): openssl-devel-3.0.7-27.el9.x86_64.rp 655 kB/s | 4.1 MB 00:06 2024-03-24T01:21:29.532 INFO:teuthology.orchestra.run.smithi120.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 433 kB/s | 151 kB 00:00 2024-03-24T01:21:29.943 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-24T01:21:30.068 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-24T01:21:30.159 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-24T01:21:30.206 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T01:21:30.249 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T01:21:30.337 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-24T01:21:30.432 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-24T01:21:30.482 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-24T01:21:30.514 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-24T01:21:30.608 INFO:teuthology.orchestra.run.smithi120.stdout:(114/174): python3-asyncssh-2.13.2-5.el9.noarch 545 kB/s | 548 kB 00:01 2024-03-24T01:21:30.637 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-24T01:21:30.758 INFO:teuthology.orchestra.run.smithi120.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 291 kB/s | 43 kB 00:00 2024-03-24T01:21:30.891 INFO:teuthology.orchestra.run.smithi120.stdout:(116/174): python3-cachetools-4.2.4-1.el9.noarc 243 kB/s | 32 kB 00:00 2024-03-24T01:21:31.025 INFO:teuthology.orchestra.run.smithi120.stdout:(117/174): python3-certifi-2021.10.8-2.el9.noar 113 kB/s | 15 kB 00:00 2024-03-24T01:21:31.050 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-24T01:21:31.155 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T01:21:31.319 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T01:21:31.347 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T01:21:31.367 INFO:teuthology.orchestra.run.smithi120.stdout:(118/174): python3-cheroot-8.6.0-4.el9.noarch.r 504 kB/s | 172 kB 00:00 2024-03-24T01:21:31.472 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T01:21:31.550 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-24T01:21:31.592 INFO:teuthology.orchestra.run.smithi120.stdout:(119/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 381 kB/s | 838 kB 00:02 2024-03-24T01:21:31.642 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-24T01:21:31.909 INFO:teuthology.orchestra.run.smithi120.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 663 kB/s | 358 kB 00:00 2024-03-24T01:21:31.933 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-24T01:21:32.017 INFO:teuthology.orchestra.run.smithi120.stdout:(121/174): python3-colorama-0.4.4-12.el9.noarch 328 kB/s | 35 kB 00:00 2024-03-24T01:21:32.024 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-24T01:21:32.051 INFO:teuthology.orchestra.run.smithi120.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 662 kB/s | 303 kB 00:00 2024-03-24T01:21:32.055 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T01:21:32.231 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T01:21:32.264 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-24T01:21:32.351 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-24T01:21:32.360 INFO:teuthology.orchestra.run.smithi120.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 594 kB/s | 203 kB 00:00 2024-03-24T01:21:32.400 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-24T01:21:32.480 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T01:21:32.541 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T01:21:32.627 INFO:teuthology.orchestra.run.smithi120.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 543 kB/s | 144 kB 00:00 2024-03-24T01:21:32.735 INFO:teuthology.orchestra.run.smithi120.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 99 kB/s | 11 kB 00:00 2024-03-24T01:21:32.836 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T01:21:32.844 INFO:teuthology.orchestra.run.smithi120.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 164 kB/s | 18 kB 00:00 2024-03-24T01:21:32.885 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T01:21:32.944 INFO:teuthology.orchestra.run.smithi120.stdout:(127/174): python3-jaraco-collections-3.0.0-8.e 234 kB/s | 23 kB 00:00 2024-03-24T01:21:32.960 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-24T01:21:33.044 INFO:teuthology.orchestra.run.smithi120.stdout:(128/174): python3-jaraco-functools-3.5.0-2.el9 195 kB/s | 19 kB 00:00 2024-03-24T01:21:33.145 INFO:teuthology.orchestra.run.smithi120.stdout:(129/174): python3-jaraco-text-3.2.0-6.el9.noar 198 kB/s | 20 kB 00:00 2024-03-24T01:21:33.416 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-24T01:21:33.510 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T01:21:33.600 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T01:21:33.902 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T01:21:33.947 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T01:21:34.314 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T01:21:34.510 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-24T01:21:34.553 INFO:teuthology.orchestra.run.smithi120.stdout:(130/174): python3-kubernetes-26.1.0-2.el9.noar 743 kB/s | 1.0 MB 00:01 2024-03-24T01:21:34.637 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-24T01:21:34.653 INFO:teuthology.orchestra.run.smithi120.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 468 kB/s | 46 kB 00:00 2024-03-24T01:21:34.688 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-24T01:21:34.711 INFO:teuthology.orchestra.run.smithi120.stdout:(132/174): python3-grpcio-1.46.7-10.el9.x86_64. 784 kB/s | 2.0 MB 00:02 2024-03-24T01:21:34.762 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-24T01:21:34.812 INFO:teuthology.orchestra.run.smithi120.stdout:(133/174): python3-natsort-7.1.1-5.el9.noarch.r 578 kB/s | 58 kB 00:00 2024-03-24T01:21:34.834 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-24T01:21:34.838 INFO:teuthology.orchestra.run.smithi120.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 430 kB/s | 79 kB 00:00 2024-03-24T01:21:34.938 INFO:teuthology.orchestra.run.smithi120.stdout:(135/174): python3-portend-3.1.0-2.el9.noarch.r 166 kB/s | 16 kB 00:00 2024-03-24T01:21:34.952 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-24T01:21:35.012 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-24T01:21:35.064 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-24T01:21:35.146 INFO:teuthology.orchestra.run.smithi120.stdout:(136/174): libarrow-9.0.0-10.el9.x86_64.rpm 671 kB/s | 4.4 MB 00:06 2024-03-24T01:21:35.166 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-24T01:21:35.175 INFO:teuthology.orchestra.run.smithi120.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 380 kB/s | 90 kB 00:00 2024-03-24T01:21:35.213 INFO:teuthology.orchestra.run.smithi120.stdout:(138/174): python3-pecan-1.4.2-3.el9.noarch.rpm 681 kB/s | 272 kB 00:00 2024-03-24T01:21:35.247 INFO:teuthology.orchestra.run.smithi120.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 283 kB/s | 28 kB 00:00 2024-03-24T01:21:35.259 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-24T01:21:35.272 INFO:teuthology.orchestra.run.smithi120.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 319 kB/s | 31 kB 00:00 2024-03-24T01:21:35.331 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-24T01:21:35.348 INFO:teuthology.orchestra.run.smithi120.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 593 kB/s | 59 kB 00:00 2024-03-24T01:21:35.373 INFO:teuthology.orchestra.run.smithi120.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 359 kB/s | 36 kB 00:00 2024-03-24T01:21:35.377 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-24T01:21:35.417 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-24T01:21:35.448 INFO:teuthology.orchestra.run.smithi120.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 511 kB/s | 51 kB 00:00 2024-03-24T01:21:35.490 INFO:teuthology.orchestra.run.smithi120.stdout:(144/174): python3-routes-2.5.1-5.el9.noarch.rp 683 kB/s | 188 kB 00:00 2024-03-24T01:21:35.538 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-24T01:21:35.578 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-24T01:21:35.632 INFO:teuthology.orchestra.run.smithi120.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 490 kB/s | 90 kB 00:00 2024-03-24T01:21:35.651 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-24T01:21:35.711 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-24T01:21:35.732 INFO:teuthology.orchestra.run.smithi120.stdout:(146/174): python3-webob-1.8.7-6.el9.noarch.rpm 640 kB/s | 230 kB 00:00 2024-03-24T01:21:35.766 INFO:teuthology.orchestra.run.smithi120.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 150 kB/s | 20 kB 00:00 2024-03-24T01:21:35.973 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-24T01:21:36.008 INFO:teuthology.orchestra.run.smithi120.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 697 kB/s | 192 kB 00:00 2024-03-24T01:21:36.037 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-24T01:21:36.041 INFO:teuthology.orchestra.run.smithi120.stdout:(149/174): python3-werkzeug-2.0.3-3.el9.1.noarc 776 kB/s | 427 kB 00:00 2024-03-24T01:21:36.146 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-24T01:21:36.328 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-24T01:21:36.393 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-24T01:21:36.475 INFO:teuthology.orchestra.run.smithi120.stdout:(150/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 7.3 MB/s | 3.1 MB 00:00 2024-03-24T01:21:36.517 INFO:teuthology.orchestra.run.smithi120.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.7 MB/s | 321 kB 00:00 2024-03-24T01:21:36.725 INFO:teuthology.orchestra.run.smithi120.stdout:(152/174): librados2-19.0.0-2145.g3be9a5c9.el9. 4.7 MB/s | 3.4 MB 00:00 2024-03-24T01:21:37.087 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-24T01:21:37.134 INFO:teuthology.orchestra.run.smithi120.stdout:(153/174): NetworkManager-1.47.2-1.el9.x86_64.r 3.7 MB/s | 2.3 MB 00:00 2024-03-24T01:21:37.190 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-24T01:21:37.360 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-24T01:21:37.401 INFO:teuthology.orchestra.run.smithi120.stdout:(154/174): NetworkManager-libnm-1.47.2-1.el9.x8 2.7 MB/s | 1.8 MB 00:00 2024-03-24T01:21:37.411 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T01:21:37.438 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T01:21:37.500 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-24T01:21:37.560 INFO:teuthology.orchestra.run.smithi120.stdout:(155/174): NetworkManager-team-1.47.2-1.el9.x86 99 kB/s | 42 kB 00:00 2024-03-24T01:21:37.592 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-24T01:21:37.650 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-24T01:21:37.695 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-24T01:21:37.785 INFO:teuthology.orchestra.run.smithi120.stdout:(156/174): NetworkManager-tui-1.47.2-1.el9.x86_ 655 kB/s | 248 kB 00:00 2024-03-24T01:21:37.918 INFO:teuthology.orchestra.run.smithi120.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 201 kB/s | 71 kB 00:00 2024-03-24T01:21:38.060 INFO:teuthology.orchestra.run.smithi120.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 133 kB/s | 36 kB 00:00 2024-03-24T01:21:38.176 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-24T01:21:38.294 INFO:teuthology.orchestra.run.smithi120.stdout:(159/174): thrift-0.15.0-2.el9.x86_64.rpm 641 kB/s | 1.6 MB 00:02 2024-03-24T01:21:38.327 INFO:teuthology.orchestra.run.smithi120.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 471 kB/s | 192 kB 00:00 2024-03-24T01:21:38.447 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-24T01:21:38.528 INFO:teuthology.orchestra.run.smithi120.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 548 kB/s | 256 kB 00:00 2024-03-24T01:21:38.540 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-24T01:21:38.570 INFO:teuthology.orchestra.run.smithi120.stdout:(162/174): libacl-2.3.1-4.el9.x86_64.rpm 84 kB/s | 23 kB 00:00 2024-03-24T01:21:38.612 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-24T01:21:38.786 INFO:teuthology.orchestra.run.smithi120.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 237 kB/s | 108 kB 00:00 2024-03-24T01:21:38.846 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-24T01:21:38.892 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-24T01:21:38.970 INFO:teuthology.orchestra.run.smithi120.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 353 kB/s | 155 kB 00:00 2024-03-24T01:21:38.996 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-24T01:21:39.004 INFO:teuthology.orchestra.run.smithi120.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 371 kB/s | 136 kB 00:00 2024-03-24T01:21:39.092 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-24T01:21:39.134 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-24T01:21:39.162 INFO:teuthology.orchestra.run.smithi120.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 172 kB/s | 64 kB 00:00 2024-03-24T01:21:39.182 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-24T01:21:39.253 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-24T01:21:39.296 INFO:teuthology.orchestra.run.smithi120.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 89 kB/s | 29 kB 00:00 2024-03-24T01:21:39.300 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-24T01:21:39.362 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-24T01:21:39.445 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-24T01:21:39.641 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-24T01:21:39.662 INFO:teuthology.orchestra.run.smithi120.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 1.8 MB/s | 1.2 MB 00:00 2024-03-24T01:21:39.698 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-24T01:21:39.736 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-24T01:21:39.763 INFO:teuthology.orchestra.run.smithi120.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 3.6 MB/s | 2.2 MB 00:00 2024-03-24T01:21:39.792 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-24T01:21:39.914 INFO:teuthology.orchestra.run.smithi120.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 3.7 MB/s | 2.3 MB 00:00 2024-03-24T01:21:39.924 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-24T01:21:39.970 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-24T01:21:40.024 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-24T01:21:40.052 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-24T01:21:40.077 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-24T01:21:40.204 INFO:teuthology.orchestra.run.smithi120.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 82 kB/s | 31 kB 00:00 2024-03-24T01:21:40.239 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-24T01:21:40.247 INFO:teuthology.orchestra.run.smithi120.stdout:(172/174): util-linux-core-2.37.4-18.el9.x86_64 797 kB/s | 465 kB 00:00 2024-03-24T01:21:40.417 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-24T01:21:40.440 INFO:teuthology.orchestra.run.smithi120.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 2.3 MB/s | 1.1 MB 00:00 2024-03-24T01:21:40.554 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-24T01:21:40.606 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-24T01:21:40.675 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-24T01:21:40.753 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-24T01:21:40.956 INFO:teuthology.orchestra.run.smithi120.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 99 kB/s | 74 kB 00:00 2024-03-24T01:21:40.974 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-24T01:21:40.974 INFO:teuthology.orchestra.run.smithi120.stdout:Total 7.5 MB/s | 224 MB 00:29 2024-03-24T01:21:41.025 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-24T01:21:41.086 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-24T01:21:41.270 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-24T01:21:41.325 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-24T01:21:41.365 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-24T01:21:41.396 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-24T01:21:41.436 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-24T01:21:41.484 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-24T01:21:41.544 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-24T01:21:41.602 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-24T01:21:41.651 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-24T01:21:41.826 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-24T01:21:41.864 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-24T01:21:41.898 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-24T01:21:41.954 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-24T01:21:41.994 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-24T01:21:42.137 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-24T01:21:42.256 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-24T01:21:42.309 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-24T01:21:42.345 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-24T01:21:42.402 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-24T01:21:42.405 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-24T01:21:42.540 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-24T01:21:42.603 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-24T01:21:42.781 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-24T01:21:42.781 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-24T01:21:43.560 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-24T01:21:43.621 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-24T01:21:43.656 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-24T01:21:43.714 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-24T01:21:43.835 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-24T01:21:43.872 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-24T01:21:44.102 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-24T01:21:44.136 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-24T01:21:44.237 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-24T01:21:45.023 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-24T01:21:45.120 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-24T01:21:45.120 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-24T01:21:45.158 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-24T01:21:46.689 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-24T01:21:46.818 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-24T01:21:46.859 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-24T01:21:46.893 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-24T01:21:46.925 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-24T01:21:47.304 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-24T01:21:47.340 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-24T01:21:47.461 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-24T01:21:47.497 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-24T01:21:47.565 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-24T01:21:48.085 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-24T01:21:48.120 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-24T01:21:48.235 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-24T01:21:48.263 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-24T01:21:50.813 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T01:21:50.854 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T01:21:51.479 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T01:21:51.533 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-24T01:21:51.711 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-24T01:21:51.711 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T01:21:51.711 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T01:21:51.711 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:21:51.753 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-24T01:21:52.410 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-24T01:21:52.505 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-24T01:21:52.552 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-24T01:21:52.591 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T01:21:52.633 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T01:21:52.724 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-24T01:21:52.827 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-24T01:21:52.883 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-24T01:21:52.923 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-24T01:21:53.018 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-24T01:21:53.442 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-24T01:21:53.490 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T01:21:53.645 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T01:21:53.671 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T01:21:53.792 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T01:21:53.838 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-24T01:21:53.910 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-24T01:21:54.211 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-24T01:21:54.267 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-24T01:21:54.292 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T01:21:54.468 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T01:21:54.493 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-24T01:21:54.582 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-24T01:21:54.628 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-24T01:21:54.645 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T01:21:54.708 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T01:21:55.019 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T01:21:55.064 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T01:21:55.097 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-24T01:21:55.403 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-24T01:21:55.639 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T01:21:55.719 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T01:21:55.906 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T01:21:55.967 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T01:21:56.270 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T01:21:56.451 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-24T01:21:56.566 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-24T01:21:56.606 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-24T01:21:56.642 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-24T01:21:56.702 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-24T01:21:56.821 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-24T01:21:56.880 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-24T01:21:56.922 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-24T01:21:57.003 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-24T01:21:57.053 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-24T01:21:57.113 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-24T01:21:57.158 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-24T01:21:57.190 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-24T01:21:57.303 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-24T01:21:57.335 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-24T01:21:57.397 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-24T01:21:57.435 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-24T01:21:57.554 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-24T01:21:57.636 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-24T01:21:57.758 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-24T01:21:58.190 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-24T01:21:58.257 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-24T01:21:58.925 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-24T01:21:59.030 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-24T01:21:59.160 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-24T01:21:59.223 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T01:21:59.248 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T01:21:59.286 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-24T01:21:59.365 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-24T01:21:59.416 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-24T01:21:59.455 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-24T01:21:59.825 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-24T01:22:00.048 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-24T01:22:00.137 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-24T01:22:00.192 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-24T01:22:00.392 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-24T01:22:00.442 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-24T01:22:00.518 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-24T01:22:00.612 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-24T01:22:00.650 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-24T01:22:00.690 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-24T01:22:00.757 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-24T01:22:00.798 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-24T01:22:00.860 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-24T01:22:00.928 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-24T01:22:01.117 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-24T01:22:01.170 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-24T01:22:01.209 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-24T01:22:01.267 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-24T01:22:01.395 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-24T01:22:01.433 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-24T01:22:01.478 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-24T01:22:01.507 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-24T01:22:01.532 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-24T01:22:01.667 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-24T01:22:01.835 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-24T01:22:01.966 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-24T01:22:02.018 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-24T01:22:02.081 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-24T01:22:02.154 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-24T01:22:02.399 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-24T01:22:02.464 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-24T01:22:02.632 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-24T01:22:02.686 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-24T01:22:02.726 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-24T01:22:02.757 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-24T01:22:02.797 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-24T01:22:02.843 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-24T01:22:02.903 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-24T01:22:02.952 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-24T01:22:02.994 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-24T01:22:03.157 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-24T01:22:03.199 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-24T01:22:03.242 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-24T01:22:03.308 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-24T01:22:03.354 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-24T01:22:03.499 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-24T01:22:03.613 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-24T01:22:03.673 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-24T01:22:03.727 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-24T01:22:03.785 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-24T01:22:03.926 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-24T01:22:04.001 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-24T01:22:04.956 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-24T01:22:05.008 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-24T01:22:05.042 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-24T01:22:05.099 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-24T01:22:05.209 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-24T01:22:05.252 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-24T01:22:05.482 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-24T01:22:05.515 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-24T01:22:05.607 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-24T01:22:05.824 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-24T01:22:05.824 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /sys 2024-03-24T01:22:05.825 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /proc 2024-03-24T01:22:05.825 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /mnt 2024-03-24T01:22:05.825 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /var/tmp 2024-03-24T01:22:05.825 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /home 2024-03-24T01:22:05.825 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /root 2024-03-24T01:22:05.825 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /tmp 2024-03-24T01:22:05.825 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:06.058 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-24T01:22:06.113 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-24T01:22:06.114 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T01:22:06.114 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-24T01:22:06.114 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:06.390 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-24T01:22:06.526 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-24T01:22:07.738 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-24T01:22:07.813 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-24T01:22:07.813 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T01:22:07.813 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T01:22:07.813 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T01:22:07.813 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T01:22:07.813 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:08.019 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-24T01:22:08.071 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-24T01:22:08.112 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-24T01:22:08.140 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-24T01:22:08.140 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T01:22:08.140 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T01:22:08.140 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T01:22:08.140 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T01:22:08.140 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:08.159 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-24T01:22:08.190 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-24T01:22:08.222 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-24T01:22:08.598 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-24T01:22:08.621 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-24T01:22:08.637 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-24T01:22:08.687 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-24T01:22:08.688 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T01:22:08.688 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T01:22:08.688 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T01:22:08.688 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T01:22:08.688 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:08.740 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-24T01:22:08.768 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-24T01:22:08.799 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-24T01:22:08.822 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-24T01:22:09.347 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-24T01:22:09.374 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-24T01:22:09.449 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-24T01:22:09.462 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-24T01:22:09.476 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-24T01:22:09.493 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-24T01:22:09.543 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-24T01:22:09.680 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-24T01:22:09.714 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-24T01:22:10.045 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-24T01:22:10.099 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-24T01:22:11.186 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-24T01:22:11.224 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-24T01:22:11.366 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-24T01:22:11.586 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-24T01:22:11.620 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-24T01:22:11.705 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-24T01:22:11.705 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T01:22:11.705 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T01:22:11.705 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T01:22:11.706 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T01:22:11.706 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:11.762 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-24T01:22:11.800 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-24T01:22:11.897 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-24T01:22:11.999 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T01:22:12.052 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T01:22:12.675 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T01:22:12.719 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-24T01:22:12.869 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-24T01:22:12.898 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-24T01:22:12.905 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-24T01:22:12.905 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T01:22:12.905 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T01:22:12.906 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:12.953 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-24T01:22:12.991 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-24T01:22:12.991 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T01:22:12.991 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T01:22:12.992 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T01:22:12.992 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T01:22:12.992 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:13.049 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-24T01:22:13.118 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-24T01:22:13.118 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T01:22:13.119 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T01:22:13.119 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:13.200 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T01:22:13.275 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T01:22:13.275 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T01:22:13.275 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-24T01:22:13.276 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T01:22:13.276 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T01:22:13.276 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:13.667 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-24T01:22:13.738 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-24T01:22:13.738 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T01:22:13.738 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T01:22:13.738 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T01:22:13.738 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T01:22:13.738 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:18.527 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-24T01:22:18.581 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-24T01:22:18.616 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-24T01:22:18.655 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-24T01:22:18.939 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-24T01:22:19.071 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-24T01:22:19.119 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-24T01:22:19.196 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-24T01:22:19.282 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-24T01:22:19.336 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-24T01:22:19.379 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-24T01:22:19.423 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-24T01:22:19.483 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-24T01:22:19.576 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-24T01:22:19.684 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-24T01:22:19.740 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-24T01:22:20.667 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T01:22:20.724 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T01:22:20.779 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T01:22:22.107 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-24T01:22:22.218 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-24T01:22:22.286 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-24T01:22:22.426 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-24T01:22:22.472 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-24T01:22:22.530 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-24T01:22:22.565 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-24T01:22:22.607 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-24T01:22:22.608 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T01:22:22.752 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T01:22:22.794 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-24T01:22:22.794 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T01:22:22.932 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T01:22:22.996 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-24T01:22:23.048 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-24T01:22:23.089 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-24T01:22:23.122 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-24T01:22:23.210 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-24T01:22:23.289 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-24T01:22:23.346 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-24T01:22:23.379 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-24T01:22:23.446 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-24T01:22:23.480 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-24T01:22:23.513 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-24T01:22:23.548 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T01:22:23.549 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T01:22:23.914 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T01:22:23.914 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T01:22:24.054 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T01:22:24.093 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-24T01:22:24.147 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-24T01:22:24.179 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-24T01:22:24.228 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-24T01:22:24.355 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T01:22:24.622 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T01:22:24.622 INFO:teuthology.orchestra.run.smithi090.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-24T01:22:24.622 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:25.977 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-24T01:22:25.977 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /sys 2024-03-24T01:22:25.977 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /proc 2024-03-24T01:22:25.977 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /mnt 2024-03-24T01:22:25.977 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /var/tmp 2024-03-24T01:22:25.977 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /home 2024-03-24T01:22:25.977 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /root 2024-03-24T01:22:25.977 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /tmp 2024-03-24T01:22:25.977 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:26.195 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-24T01:22:26.244 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-24T01:22:26.244 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T01:22:26.244 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-24T01:22:26.244 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:27.849 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-24T01:22:27.929 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-24T01:22:27.929 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T01:22:27.929 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T01:22:27.929 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T01:22:27.929 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T01:22:27.929 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:28.191 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-24T01:22:28.263 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-24T01:22:28.263 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T01:22:28.263 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T01:22:28.263 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T01:22:28.263 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T01:22:28.263 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:28.783 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-24T01:22:28.858 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-24T01:22:28.858 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T01:22:28.858 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T01:22:28.858 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T01:22:28.858 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T01:22:28.858 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:28.985 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-24T01:22:29.630 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-24T01:22:29.662 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-24T01:22:29.713 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-24T01:22:29.825 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-24T01:22:29.857 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-24T01:22:30.186 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-24T01:22:30.217 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-24T01:22:31.117 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-24T01:22:31.117 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-24T01:22:31.117 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-24T01:22:31.117 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-24T01:22:31.117 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-24T01:22:31.117 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-24T01:22:31.117 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-24T01:22:31.117 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-24T01:22:31.117 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-24T01:22:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T01:22:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T01:22:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-24T01:22:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-24T01:22:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-24T01:22:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-24T01:22:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T01:22:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-24T01:22:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-24T01:22:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-24T01:22:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-24T01:22:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-24T01:22:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-24T01:22:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-24T01:22:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-24T01:22:31.119 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-24T01:22:31.119 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-24T01:22:31.119 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-24T01:22:31.119 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-24T01:22:31.119 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-24T01:22:31.119 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-24T01:22:31.134 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-24T01:22:31.134 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-24T01:22:31.134 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-24T01:22:31.134 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-24T01:22:31.134 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-24T01:22:31.134 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-24T01:22:31.134 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-24T01:22:31.134 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-24T01:22:31.135 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-24T01:22:31.135 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-24T01:22:31.135 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-24T01:22:31.135 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-24T01:22:31.135 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-24T01:22:31.135 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-24T01:22:31.135 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-24T01:22:31.135 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-24T01:22:31.135 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-24T01:22:31.135 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-24T01:22:31.135 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-24T01:22:31.135 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-24T01:22:31.135 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-24T01:22:31.135 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-24T01:22:31.135 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-24T01:22:31.136 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-24T01:22:31.136 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-24T01:22:31.136 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-24T01:22:31.136 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-24T01:22:31.136 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-24T01:22:31.136 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-24T01:22:31.136 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-24T01:22:31.136 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-24T01:22:31.136 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-24T01:22:31.136 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-24T01:22:31.136 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-24T01:22:31.136 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-24T01:22:31.136 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-24T01:22:31.136 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-24T01:22:31.137 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-24T01:22:31.137 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-24T01:22:31.137 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-24T01:22:31.137 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-24T01:22:31.137 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-24T01:22:31.137 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-24T01:22:31.137 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-24T01:22:31.137 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-24T01:22:31.137 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-24T01:22:31.137 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-24T01:22:31.137 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-24T01:22:31.137 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-24T01:22:31.137 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-24T01:22:31.137 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-24T01:22:31.137 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-24T01:22:31.138 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-24T01:22:31.138 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-24T01:22:31.138 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-24T01:22:31.138 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-24T01:22:31.138 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-24T01:22:31.138 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-24T01:22:31.138 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-24T01:22:31.138 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-24T01:22:31.138 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-24T01:22:31.138 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-24T01:22:31.138 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-24T01:22:31.138 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-24T01:22:31.138 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-24T01:22:31.139 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-24T01:22:31.139 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-24T01:22:31.139 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-24T01:22:31.139 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-24T01:22:31.139 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-24T01:22:31.139 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-24T01:22:31.139 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-24T01:22:31.139 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-24T01:22:31.139 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-24T01:22:31.139 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-24T01:22:31.139 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-24T01:22:31.139 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-24T01:22:31.139 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-24T01:22:31.139 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-24T01:22:31.139 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-24T01:22:31.140 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-24T01:22:31.140 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-24T01:22:31.140 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-24T01:22:31.140 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-24T01:22:31.140 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-24T01:22:31.140 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-24T01:22:31.140 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-24T01:22:31.140 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-24T01:22:31.140 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-24T01:22:31.140 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-24T01:22:31.140 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-24T01:22:31.140 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-24T01:22:31.140 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-24T01:22:31.141 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-24T01:22:31.141 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-24T01:22:31.141 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-24T01:22:31.141 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-24T01:22:31.141 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-24T01:22:31.141 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-24T01:22:31.141 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-24T01:22:31.141 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-24T01:22:31.141 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-24T01:22:31.141 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-24T01:22:31.141 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-24T01:22:31.141 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-24T01:22:31.141 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-24T01:22:31.141 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-24T01:22:31.141 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-24T01:22:31.142 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-24T01:22:31.142 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-24T01:22:31.142 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-24T01:22:31.142 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-24T01:22:31.142 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-24T01:22:31.142 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-24T01:22:31.142 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-24T01:22:31.142 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-24T01:22:31.142 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-24T01:22:31.142 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-24T01:22:31.142 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-24T01:22:31.142 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-24T01:22:31.142 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T01:22:31.142 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-24T01:22:31.142 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-24T01:22:31.143 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-24T01:22:31.143 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-24T01:22:31.143 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-24T01:22:31.143 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-24T01:22:31.143 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-24T01:22:31.143 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-24T01:22:31.143 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-24T01:22:31.143 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-24T01:22:31.143 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-24T01:22:31.143 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-24T01:22:31.143 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-24T01:22:31.143 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-24T01:22:31.143 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-24T01:22:31.143 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-24T01:22:31.144 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-24T01:22:31.144 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-24T01:22:31.144 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-24T01:22:31.144 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-24T01:22:31.144 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-24T01:22:31.144 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-24T01:22:31.144 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-24T01:22:31.144 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-24T01:22:31.144 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-24T01:22:31.144 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-24T01:22:31.144 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-24T01:22:31.144 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-24T01:22:31.144 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-24T01:22:31.144 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-24T01:22:31.145 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-24T01:22:31.145 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-24T01:22:31.145 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-24T01:22:31.145 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-24T01:22:31.145 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-24T01:22:31.145 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-24T01:22:31.145 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-24T01:22:31.145 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-24T01:22:31.145 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-24T01:22:31.145 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-24T01:22:31.145 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-24T01:22:31.145 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-24T01:22:31.145 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-24T01:22:31.145 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-24T01:22:31.145 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-24T01:22:31.146 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-24T01:22:31.331 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-24T01:22:31.367 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-24T01:22:31.508 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-24T01:22:31.688 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-24T01:22:31.719 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-24T01:22:31.792 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-24T01:22:31.793 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T01:22:31.793 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T01:22:31.793 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T01:22:31.793 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T01:22:31.793 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:31.854 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-24T01:22:31.891 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-24T01:22:31.961 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-24T01:22:32.906 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-24T01:22:32.939 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-24T01:22:33.004 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-24T01:22:33.004 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T01:22:33.004 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T01:22:33.004 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T01:22:33.005 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T01:22:33.005 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:33.056 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-24T01:22:33.125 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-24T01:22:33.125 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T01:22:33.125 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T01:22:33.125 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:33.190 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T01:22:33.265 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T01:22:33.265 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T01:22:33.265 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-24T01:22:33.265 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T01:22:33.265 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T01:22:33.265 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:33.624 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-24T01:22:33.695 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-24T01:22:33.695 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T01:22:33.695 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T01:22:33.695 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T01:22:33.696 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T01:22:33.696 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:33.988 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-24T01:22:33.988 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:33.988 INFO:teuthology.orchestra.run.smithi090.stdout:Upgraded: 2024-03-24T01:22:33.988 INFO:teuthology.orchestra.run.smithi090.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-24T01:22:33.988 INFO:teuthology.orchestra.run.smithi090.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-24T01:22:33.988 INFO:teuthology.orchestra.run.smithi090.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-24T01:22:33.988 INFO:teuthology.orchestra.run.smithi090.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-24T01:22:33.988 INFO:teuthology.orchestra.run.smithi090.stdout: acl-2.3.1-4.el9.x86_64 2024-03-24T01:22:33.989 INFO:teuthology.orchestra.run.smithi090.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-24T01:22:33.989 INFO:teuthology.orchestra.run.smithi090.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-24T01:22:33.989 INFO:teuthology.orchestra.run.smithi090.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-24T01:22:33.989 INFO:teuthology.orchestra.run.smithi090.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-24T01:22:33.989 INFO:teuthology.orchestra.run.smithi090.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-24T01:22:33.989 INFO:teuthology.orchestra.run.smithi090.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-24T01:22:33.989 INFO:teuthology.orchestra.run.smithi090.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-24T01:22:33.989 INFO:teuthology.orchestra.run.smithi090.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-24T01:22:33.989 INFO:teuthology.orchestra.run.smithi090.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:33.989 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:33.989 INFO:teuthology.orchestra.run.smithi090.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-24T01:22:33.989 INFO:teuthology.orchestra.run.smithi090.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-24T01:22:33.989 INFO:teuthology.orchestra.run.smithi090.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-24T01:22:33.989 INFO:teuthology.orchestra.run.smithi090.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-24T01:22:33.989 INFO:teuthology.orchestra.run.smithi090.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-24T01:22:33.990 INFO:teuthology.orchestra.run.smithi090.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-24T01:22:33.990 INFO:teuthology.orchestra.run.smithi090.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-24T01:22:33.990 INFO:teuthology.orchestra.run.smithi090.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-24T01:22:33.990 INFO:teuthology.orchestra.run.smithi090.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-24T01:22:33.990 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-24T01:22:33.990 INFO:teuthology.orchestra.run.smithi090.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-24T01:22:33.990 INFO:teuthology.orchestra.run.smithi090.stdout: automake-1.16.2-8.el9.noarch 2024-03-24T01:22:33.990 INFO:teuthology.orchestra.run.smithi090.stdout: bison-3.7.4-5.el9.x86_64 2024-03-24T01:22:33.990 INFO:teuthology.orchestra.run.smithi090.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T01:22:33.990 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:33.990 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:33.990 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:33.990 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:33.991 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T01:22:33.991 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:33.991 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:33.991 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:33.991 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T01:22:33.991 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T01:22:33.991 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T01:22:33.991 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T01:22:33.991 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T01:22:33.991 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:33.991 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:33.991 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T01:22:33.991 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:33.991 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:33.991 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:33.992 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T01:22:33.992 INFO:teuthology.orchestra.run.smithi090.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T01:22:33.992 INFO:teuthology.orchestra.run.smithi090.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:33.992 INFO:teuthology.orchestra.run.smithi090.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:33.992 INFO:teuthology.orchestra.run.smithi090.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T01:22:33.992 INFO:teuthology.orchestra.run.smithi090.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-24T01:22:33.992 INFO:teuthology.orchestra.run.smithi090.stdout: flex-2.6.4-9.el9.x86_64 2024-03-24T01:22:33.992 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T01:22:33.992 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T01:22:33.992 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T01:22:33.992 INFO:teuthology.orchestra.run.smithi090.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-24T01:22:33.992 INFO:teuthology.orchestra.run.smithi090.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T01:22:33.992 INFO:teuthology.orchestra.run.smithi090.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-24T01:22:33.992 INFO:teuthology.orchestra.run.smithi090.stdout: jq-1.6-16.el9.x86_64 2024-03-24T01:22:33.992 INFO:teuthology.orchestra.run.smithi090.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-24T01:22:33.993 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T01:22:33.993 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T01:22:33.993 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:33.993 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:33.993 INFO:teuthology.orchestra.run.smithi090.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:33.993 INFO:teuthology.orchestra.run.smithi090.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T01:22:33.993 INFO:teuthology.orchestra.run.smithi090.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T01:22:33.993 INFO:teuthology.orchestra.run.smithi090.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T01:22:33.993 INFO:teuthology.orchestra.run.smithi090.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T01:22:33.993 INFO:teuthology.orchestra.run.smithi090.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T01:22:33.993 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:33.993 INFO:teuthology.orchestra.run.smithi090.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:33.993 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T01:22:33.993 INFO:teuthology.orchestra.run.smithi090.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:33.994 INFO:teuthology.orchestra.run.smithi090.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-24T01:22:33.994 INFO:teuthology.orchestra.run.smithi090.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T01:22:33.994 INFO:teuthology.orchestra.run.smithi090.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-24T01:22:33.994 INFO:teuthology.orchestra.run.smithi090.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-24T01:22:33.994 INFO:teuthology.orchestra.run.smithi090.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T01:22:33.994 INFO:teuthology.orchestra.run.smithi090.stdout: lua-5.4.4-4.el9.x86_64 2024-03-24T01:22:33.994 INFO:teuthology.orchestra.run.smithi090.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-24T01:22:33.994 INFO:teuthology.orchestra.run.smithi090.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-24T01:22:33.994 INFO:teuthology.orchestra.run.smithi090.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T01:22:33.994 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T01:22:33.994 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T01:22:33.994 INFO:teuthology.orchestra.run.smithi090.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-24T01:22:33.994 INFO:teuthology.orchestra.run.smithi090.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T01:22:33.994 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-13.11-1.el9.x86_64 2024-03-24T01:22:33.995 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-24T01:22:33.995 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-24T01:22:33.995 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-24T01:22:33.995 INFO:teuthology.orchestra.run.smithi090.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-24T01:22:33.995 INFO:teuthology.orchestra.run.smithi090.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-24T01:22:33.995 INFO:teuthology.orchestra.run.smithi090.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T01:22:33.995 INFO:teuthology.orchestra.run.smithi090.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-24T01:22:33.995 INFO:teuthology.orchestra.run.smithi090.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T01:22:33.995 INFO:teuthology.orchestra.run.smithi090.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T01:22:33.995 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T01:22:33.995 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:33.995 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:33.995 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:33.995 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T01:22:33.996 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T01:22:33.996 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T01:22:33.996 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T01:22:33.996 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-24T01:22:33.996 INFO:teuthology.orchestra.run.smithi090.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-24T01:22:33.996 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T01:22:33.996 INFO:teuthology.orchestra.run.smithi090.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T01:22:33.996 INFO:teuthology.orchestra.run.smithi090.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T01:22:33.996 INFO:teuthology.orchestra.run.smithi090.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-24T01:22:33.996 INFO:teuthology.orchestra.run.smithi090.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-24T01:22:33.996 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T01:22:33.996 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T01:22:33.996 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T01:22:33.996 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T01:22:33.996 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T01:22:33.997 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T01:22:33.997 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T01:22:33.997 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T01:22:33.997 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T01:22:33.997 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T01:22:33.997 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T01:22:33.997 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T01:22:33.997 INFO:teuthology.orchestra.run.smithi090.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T01:22:33.997 INFO:teuthology.orchestra.run.smithi090.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T01:22:33.997 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T01:22:33.997 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T01:22:33.997 INFO:teuthology.orchestra.run.smithi090.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T01:22:33.997 INFO:teuthology.orchestra.run.smithi090.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-24T01:22:33.997 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T01:22:33.997 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T01:22:33.998 INFO:teuthology.orchestra.run.smithi090.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T01:22:33.998 INFO:teuthology.orchestra.run.smithi090.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T01:22:33.998 INFO:teuthology.orchestra.run.smithi090.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-24T01:22:33.998 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T01:22:33.998 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T01:22:33.998 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T01:22:33.998 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T01:22:33.998 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-24T01:22:33.998 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T01:22:33.998 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:33.998 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:33.998 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T01:22:33.998 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T01:22:33.998 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T01:22:33.999 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:33.999 INFO:teuthology.orchestra.run.smithi090.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T01:22:33.999 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T01:22:33.999 INFO:teuthology.orchestra.run.smithi090.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T01:22:33.999 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T01:22:33.999 INFO:teuthology.orchestra.run.smithi090.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T01:22:33.999 INFO:teuthology.orchestra.run.smithi090.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T01:22:33.999 INFO:teuthology.orchestra.run.smithi090.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T01:22:33.999 INFO:teuthology.orchestra.run.smithi090.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-24T01:22:33.999 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T01:22:33.999 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T01:22:33.999 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T01:22:33.999 INFO:teuthology.orchestra.run.smithi090.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T01:22:33.999 INFO:teuthology.orchestra.run.smithi090.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-24T01:22:33.999 INFO:teuthology.orchestra.run.smithi090.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-24T01:22:33.999 INFO:teuthology.orchestra.run.smithi090.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-24T01:22:34.000 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:34.000 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:34.000 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:34.000 INFO:teuthology.orchestra.run.smithi090.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T01:22:34.000 INFO:teuthology.orchestra.run.smithi090.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T01:22:34.000 INFO:teuthology.orchestra.run.smithi090.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T01:22:34.000 INFO:teuthology.orchestra.run.smithi090.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-24T01:22:34.000 INFO:teuthology.orchestra.run.smithi090.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-24T01:22:34.000 INFO:teuthology.orchestra.run.smithi090.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T01:22:34.000 INFO:teuthology.orchestra.run.smithi090.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-24T01:22:34.001 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:34.001 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-24T01:22:34.821 DEBUG:teuthology.parallel:result is None 2024-03-24T01:22:38.490 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-24T01:22:38.546 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-24T01:22:38.580 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-24T01:22:38.619 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-24T01:22:38.911 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-24T01:22:39.035 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-24T01:22:39.082 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-24T01:22:39.139 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-24T01:22:39.199 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-24T01:22:39.244 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-24T01:22:39.279 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-24T01:22:39.316 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-24T01:22:39.369 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-24T01:22:39.462 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-24T01:22:39.597 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-24T01:22:39.634 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-24T01:22:40.497 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T01:22:40.540 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T01:22:40.596 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T01:22:41.898 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-24T01:22:41.997 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-24T01:22:42.044 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-24T01:22:42.180 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-24T01:22:42.216 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-24T01:22:42.258 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-24T01:22:42.292 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-24T01:22:42.333 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-24T01:22:42.333 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T01:22:42.463 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T01:22:42.505 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-24T01:22:42.505 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T01:22:42.635 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T01:22:42.681 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-24T01:22:42.726 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-24T01:22:42.784 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-24T01:22:42.816 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-24T01:22:42.884 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-24T01:22:42.956 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-24T01:22:42.996 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-24T01:22:43.030 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-24T01:22:43.063 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-24T01:22:43.097 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-24T01:22:43.130 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-24T01:22:43.165 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T01:22:43.165 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T01:22:43.464 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T01:22:43.464 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T01:22:43.604 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T01:22:43.634 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-24T01:22:43.687 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-24T01:22:43.720 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-24T01:22:43.760 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-24T01:22:43.880 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T01:22:44.133 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T01:22:44.133 INFO:teuthology.orchestra.run.smithi120.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-24T01:22:44.134 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:49.242 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-24T01:22:49.242 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-24T01:22:49.242 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-24T01:22:49.243 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-24T01:22:49.243 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-24T01:22:49.243 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-24T01:22:49.243 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-24T01:22:49.243 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-24T01:22:49.243 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-24T01:22:49.243 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T01:22:49.243 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T01:22:49.243 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-24T01:22:49.243 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-24T01:22:49.243 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-24T01:22:49.243 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-24T01:22:49.243 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T01:22:49.244 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-24T01:22:49.244 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-24T01:22:49.244 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-24T01:22:49.244 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-24T01:22:49.244 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-24T01:22:49.244 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-24T01:22:49.244 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-24T01:22:49.244 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-24T01:22:49.244 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-24T01:22:49.244 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-24T01:22:49.244 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-24T01:22:49.244 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-24T01:22:49.244 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-24T01:22:49.244 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-24T01:22:49.245 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-24T01:22:49.245 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-24T01:22:49.245 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-24T01:22:49.245 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-24T01:22:49.260 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-24T01:22:49.260 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-24T01:22:49.260 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-24T01:22:49.260 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-24T01:22:49.260 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-24T01:22:49.260 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-24T01:22:49.260 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-24T01:22:49.260 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-24T01:22:49.260 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-24T01:22:49.260 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-24T01:22:49.260 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-24T01:22:49.261 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-24T01:22:49.261 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-24T01:22:49.261 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-24T01:22:49.261 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-24T01:22:49.261 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-24T01:22:49.261 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-24T01:22:49.261 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-24T01:22:49.261 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-24T01:22:49.261 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-24T01:22:49.261 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-24T01:22:49.261 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-24T01:22:49.261 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-24T01:22:49.261 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-24T01:22:49.261 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-24T01:22:49.262 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-24T01:22:49.262 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-24T01:22:49.262 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-24T01:22:49.262 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-24T01:22:49.262 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-24T01:22:49.262 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-24T01:22:49.262 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-24T01:22:49.262 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-24T01:22:49.262 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-24T01:22:49.262 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-24T01:22:49.262 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-24T01:22:49.262 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-24T01:22:49.262 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-24T01:22:49.263 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-24T01:22:49.263 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-24T01:22:49.263 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-24T01:22:49.263 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-24T01:22:49.263 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-24T01:22:49.263 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-24T01:22:49.263 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-24T01:22:49.263 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-24T01:22:49.263 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-24T01:22:49.263 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-24T01:22:49.263 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-24T01:22:49.263 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-24T01:22:49.263 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-24T01:22:49.264 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-24T01:22:49.264 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-24T01:22:49.264 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-24T01:22:49.264 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-24T01:22:49.264 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-24T01:22:49.264 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-24T01:22:49.264 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-24T01:22:49.264 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-24T01:22:49.264 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-24T01:22:49.264 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-24T01:22:49.264 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-24T01:22:49.264 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-24T01:22:49.264 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-24T01:22:49.264 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-24T01:22:49.265 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-24T01:22:49.265 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-24T01:22:49.265 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-24T01:22:49.265 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-24T01:22:49.265 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-24T01:22:49.265 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-24T01:22:49.265 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-24T01:22:49.265 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-24T01:22:49.265 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-24T01:22:49.265 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-24T01:22:49.265 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-24T01:22:49.265 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-24T01:22:49.265 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-24T01:22:49.265 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-24T01:22:49.265 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-24T01:22:49.266 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-24T01:22:49.266 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-24T01:22:49.266 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-24T01:22:49.266 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-24T01:22:49.266 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-24T01:22:49.266 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-24T01:22:49.266 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-24T01:22:49.266 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-24T01:22:49.266 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-24T01:22:49.266 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-24T01:22:49.266 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-24T01:22:49.266 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-24T01:22:49.266 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-24T01:22:49.266 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-24T01:22:49.267 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-24T01:22:49.267 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-24T01:22:49.267 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-24T01:22:49.267 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-24T01:22:49.267 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-24T01:22:49.267 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-24T01:22:49.267 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-24T01:22:49.267 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-24T01:22:49.267 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-24T01:22:49.267 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-24T01:22:49.267 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-24T01:22:49.267 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-24T01:22:49.267 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-24T01:22:49.267 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-24T01:22:49.267 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-24T01:22:49.268 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-24T01:22:49.268 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-24T01:22:49.268 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-24T01:22:49.268 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-24T01:22:49.268 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-24T01:22:49.268 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-24T01:22:49.268 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-24T01:22:49.268 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T01:22:49.268 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-24T01:22:49.268 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-24T01:22:49.268 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-24T01:22:49.268 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-24T01:22:49.268 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-24T01:22:49.268 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-24T01:22:49.269 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-24T01:22:49.269 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-24T01:22:49.269 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-24T01:22:49.269 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-24T01:22:49.269 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-24T01:22:49.269 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-24T01:22:49.269 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-24T01:22:49.269 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-24T01:22:49.269 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-24T01:22:49.269 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-24T01:22:49.269 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-24T01:22:49.269 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-24T01:22:49.269 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-24T01:22:49.269 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-24T01:22:49.270 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-24T01:22:49.270 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-24T01:22:49.270 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-24T01:22:49.270 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-24T01:22:49.270 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-24T01:22:49.270 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-24T01:22:49.270 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-24T01:22:49.270 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-24T01:22:49.270 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-24T01:22:49.270 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-24T01:22:49.270 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-24T01:22:49.270 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-24T01:22:49.270 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-24T01:22:49.270 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-24T01:22:49.271 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-24T01:22:49.271 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-24T01:22:49.271 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-24T01:22:49.271 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-24T01:22:49.271 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-24T01:22:49.271 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-24T01:22:49.271 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-24T01:22:49.271 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-24T01:22:49.271 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-24T01:22:49.271 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-24T01:22:49.271 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-24T01:22:49.271 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-24T01:22:51.696 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-24T01:22:51.696 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:51.696 INFO:teuthology.orchestra.run.smithi120.stdout:Upgraded: 2024-03-24T01:22:51.696 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-24T01:22:51.696 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-24T01:22:51.697 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-24T01:22:51.697 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-24T01:22:51.697 INFO:teuthology.orchestra.run.smithi120.stdout: acl-2.3.1-4.el9.x86_64 2024-03-24T01:22:51.697 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-24T01:22:51.697 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-24T01:22:51.697 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-24T01:22:51.697 INFO:teuthology.orchestra.run.smithi120.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-24T01:22:51.697 INFO:teuthology.orchestra.run.smithi120.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-24T01:22:51.697 INFO:teuthology.orchestra.run.smithi120.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-24T01:22:51.697 INFO:teuthology.orchestra.run.smithi120.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-24T01:22:51.697 INFO:teuthology.orchestra.run.smithi120.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-24T01:22:51.697 INFO:teuthology.orchestra.run.smithi120.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:51.697 INFO:teuthology.orchestra.run.smithi120.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:51.698 INFO:teuthology.orchestra.run.smithi120.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-24T01:22:51.698 INFO:teuthology.orchestra.run.smithi120.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-24T01:22:51.698 INFO:teuthology.orchestra.run.smithi120.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-24T01:22:51.698 INFO:teuthology.orchestra.run.smithi120.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-24T01:22:51.698 INFO:teuthology.orchestra.run.smithi120.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-24T01:22:51.698 INFO:teuthology.orchestra.run.smithi120.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-24T01:22:51.698 INFO:teuthology.orchestra.run.smithi120.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-24T01:22:51.698 INFO:teuthology.orchestra.run.smithi120.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-24T01:22:51.698 INFO:teuthology.orchestra.run.smithi120.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-24T01:22:51.698 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-24T01:22:51.698 INFO:teuthology.orchestra.run.smithi120.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-24T01:22:51.698 INFO:teuthology.orchestra.run.smithi120.stdout: automake-1.16.2-8.el9.noarch 2024-03-24T01:22:51.698 INFO:teuthology.orchestra.run.smithi120.stdout: bison-3.7.4-5.el9.x86_64 2024-03-24T01:22:51.699 INFO:teuthology.orchestra.run.smithi120.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T01:22:51.699 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:51.699 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:51.699 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:51.699 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:51.699 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T01:22:51.699 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:51.699 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:51.699 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:51.699 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T01:22:51.699 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T01:22:51.699 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T01:22:51.699 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T01:22:51.700 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T01:22:51.700 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:51.700 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:51.700 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T01:22:51.700 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:51.700 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:51.700 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:51.700 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T01:22:51.700 INFO:teuthology.orchestra.run.smithi120.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T01:22:51.700 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:51.700 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:51.700 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T01:22:51.700 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-24T01:22:51.700 INFO:teuthology.orchestra.run.smithi120.stdout: flex-2.6.4-9.el9.x86_64 2024-03-24T01:22:51.701 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T01:22:51.701 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T01:22:51.701 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T01:22:51.701 INFO:teuthology.orchestra.run.smithi120.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-24T01:22:51.701 INFO:teuthology.orchestra.run.smithi120.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T01:22:51.701 INFO:teuthology.orchestra.run.smithi120.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-24T01:22:51.701 INFO:teuthology.orchestra.run.smithi120.stdout: jq-1.6-16.el9.x86_64 2024-03-24T01:22:51.701 INFO:teuthology.orchestra.run.smithi120.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-24T01:22:51.701 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T01:22:51.701 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T01:22:51.701 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:51.701 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:51.701 INFO:teuthology.orchestra.run.smithi120.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:51.701 INFO:teuthology.orchestra.run.smithi120.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T01:22:51.702 INFO:teuthology.orchestra.run.smithi120.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T01:22:51.702 INFO:teuthology.orchestra.run.smithi120.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T01:22:51.702 INFO:teuthology.orchestra.run.smithi120.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T01:22:51.702 INFO:teuthology.orchestra.run.smithi120.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T01:22:51.702 INFO:teuthology.orchestra.run.smithi120.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:51.702 INFO:teuthology.orchestra.run.smithi120.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:51.702 INFO:teuthology.orchestra.run.smithi120.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T01:22:51.702 INFO:teuthology.orchestra.run.smithi120.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:51.702 INFO:teuthology.orchestra.run.smithi120.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-24T01:22:51.702 INFO:teuthology.orchestra.run.smithi120.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T01:22:51.702 INFO:teuthology.orchestra.run.smithi120.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-24T01:22:51.702 INFO:teuthology.orchestra.run.smithi120.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-24T01:22:51.702 INFO:teuthology.orchestra.run.smithi120.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T01:22:51.702 INFO:teuthology.orchestra.run.smithi120.stdout: lua-5.4.4-4.el9.x86_64 2024-03-24T01:22:51.703 INFO:teuthology.orchestra.run.smithi120.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-24T01:22:51.703 INFO:teuthology.orchestra.run.smithi120.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-24T01:22:51.703 INFO:teuthology.orchestra.run.smithi120.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T01:22:51.703 INFO:teuthology.orchestra.run.smithi120.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T01:22:51.703 INFO:teuthology.orchestra.run.smithi120.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T01:22:51.703 INFO:teuthology.orchestra.run.smithi120.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-24T01:22:51.703 INFO:teuthology.orchestra.run.smithi120.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T01:22:51.703 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-13.11-1.el9.x86_64 2024-03-24T01:22:51.703 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-24T01:22:51.703 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-24T01:22:51.703 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-24T01:22:51.703 INFO:teuthology.orchestra.run.smithi120.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-24T01:22:51.703 INFO:teuthology.orchestra.run.smithi120.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-24T01:22:51.703 INFO:teuthology.orchestra.run.smithi120.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T01:22:51.704 INFO:teuthology.orchestra.run.smithi120.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-24T01:22:51.704 INFO:teuthology.orchestra.run.smithi120.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T01:22:51.704 INFO:teuthology.orchestra.run.smithi120.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T01:22:51.704 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T01:22:51.704 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:51.704 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:51.704 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:51.704 INFO:teuthology.orchestra.run.smithi120.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T01:22:51.704 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T01:22:51.704 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T01:22:51.704 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T01:22:51.704 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-24T01:22:51.704 INFO:teuthology.orchestra.run.smithi120.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-24T01:22:51.705 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T01:22:51.705 INFO:teuthology.orchestra.run.smithi120.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T01:22:51.705 INFO:teuthology.orchestra.run.smithi120.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T01:22:51.705 INFO:teuthology.orchestra.run.smithi120.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-24T01:22:51.705 INFO:teuthology.orchestra.run.smithi120.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-24T01:22:51.705 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T01:22:51.705 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T01:22:51.705 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T01:22:51.705 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T01:22:51.705 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T01:22:51.705 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T01:22:51.705 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T01:22:51.705 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T01:22:51.705 INFO:teuthology.orchestra.run.smithi120.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T01:22:51.706 INFO:teuthology.orchestra.run.smithi120.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T01:22:51.706 INFO:teuthology.orchestra.run.smithi120.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T01:22:51.706 INFO:teuthology.orchestra.run.smithi120.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T01:22:51.706 INFO:teuthology.orchestra.run.smithi120.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T01:22:51.706 INFO:teuthology.orchestra.run.smithi120.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T01:22:51.706 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T01:22:51.706 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T01:22:51.706 INFO:teuthology.orchestra.run.smithi120.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T01:22:51.706 INFO:teuthology.orchestra.run.smithi120.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-24T01:22:51.706 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T01:22:51.706 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T01:22:51.706 INFO:teuthology.orchestra.run.smithi120.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T01:22:51.706 INFO:teuthology.orchestra.run.smithi120.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T01:22:51.706 INFO:teuthology.orchestra.run.smithi120.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-24T01:22:51.707 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T01:22:51.707 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T01:22:51.707 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T01:22:51.707 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T01:22:51.707 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-24T01:22:51.707 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T01:22:51.707 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:51.707 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:51.707 INFO:teuthology.orchestra.run.smithi120.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T01:22:51.707 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T01:22:51.707 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T01:22:51.707 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:51.707 INFO:teuthology.orchestra.run.smithi120.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T01:22:51.708 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T01:22:51.708 INFO:teuthology.orchestra.run.smithi120.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T01:22:51.708 INFO:teuthology.orchestra.run.smithi120.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T01:22:51.708 INFO:teuthology.orchestra.run.smithi120.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T01:22:51.708 INFO:teuthology.orchestra.run.smithi120.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T01:22:51.708 INFO:teuthology.orchestra.run.smithi120.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T01:22:51.708 INFO:teuthology.orchestra.run.smithi120.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-24T01:22:51.708 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T01:22:51.708 INFO:teuthology.orchestra.run.smithi120.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T01:22:51.708 INFO:teuthology.orchestra.run.smithi120.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T01:22:51.708 INFO:teuthology.orchestra.run.smithi120.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T01:22:51.708 INFO:teuthology.orchestra.run.smithi120.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-24T01:22:51.708 INFO:teuthology.orchestra.run.smithi120.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-24T01:22:51.708 INFO:teuthology.orchestra.run.smithi120.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-24T01:22:51.708 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:51.709 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:51.709 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T01:22:51.709 INFO:teuthology.orchestra.run.smithi120.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T01:22:51.709 INFO:teuthology.orchestra.run.smithi120.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T01:22:51.709 INFO:teuthology.orchestra.run.smithi120.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T01:22:51.709 INFO:teuthology.orchestra.run.smithi120.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-24T01:22:51.709 INFO:teuthology.orchestra.run.smithi120.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-24T01:22:51.709 INFO:teuthology.orchestra.run.smithi120.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T01:22:51.709 INFO:teuthology.orchestra.run.smithi120.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-24T01:22:51.710 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:51.710 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-24T01:22:52.524 DEBUG:teuthology.parallel:result is None 2024-03-24T01:22:52.525 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T01:22:52.802 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T01:22:52.872 INFO:teuthology.orchestra.run.smithi090.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-24T01:22:52.873 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-24T01:22:52.873 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-24T01:22:52.875 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T01:22:53.116 DEBUG:teuthology.orchestra.run.smithi120:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T01:22:53.180 INFO:teuthology.orchestra.run.smithi120.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-24T01:22:53.180 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-24T01:22:53.181 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-24T01:22:53.182 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-24T01:22:53.183 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:22:53.183 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T01:22:53.285 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:22:53.286 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T01:22:53.379 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-24T01:22:53.380 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:22:53.380 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T01:22:53.478 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T01:22:53.577 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:22:53.578 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T01:22:53.660 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T01:22:53.771 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-24T01:22:53.771 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:22:53.771 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T01:22:53.863 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T01:22:53.997 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:22:53.998 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T01:22:54.072 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T01:22:54.180 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-24T01:22:54.180 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:22:54.181 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T01:22:54.259 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T01:22:54.383 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:22:54.384 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T01:22:54.453 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T01:22:54.565 INFO:teuthology.run_tasks:Running task ceph... 2024-03-24T01:22:54.655 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-24T01:22:54.655 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 777 /var/log/ceph 2024-03-24T01:22:54.661 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 777 /var/log/ceph 2024-03-24T01:22:54.744 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-24T01:22:54.744 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-24T01:22:54.749 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-24T01:22:54.831 INFO:tasks.ceph:Creating extra log directories... 2024-03-24T01:22:54.831 DEBUG:teuthology.orchestra.run.smithi090:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-24T01:22:54.835 DEBUG:teuthology.orchestra.run.smithi120:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-24T01:22:54.918 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-24T01:22:54.918 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-24T01:22:54.918 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-24T01:22:54.918 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:22:54.919 DEBUG:teuthology.orchestra.run.smithi090:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-24T01:22:54.964 DEBUG:teuthology.orchestra.run.smithi090:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-24T01:22:55.188 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-24T01:22:55.264 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:22:55.264 DEBUG:teuthology.orchestra.run.smithi120:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-24T01:22:55.303 DEBUG:teuthology.orchestra.run.smithi120:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-24T01:22:55.527 DEBUG:teuthology.orchestra.run.smithi120:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-24T01:22:55.604 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-24T01:22:55.604 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'bad backtrace on directory inode', 'error reading table object', 'Metadata damage detected', 'slow requests are blocked', 'Behind on trimming', 'error reading sessionmap'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-24T01:22:55.604 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616453', 'branch': 'wip-mchangir-testing-main-20240318.032620', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/journal-repair}', 'email': 'mchangir@redhat.com', 'first_in_suite': False, 'job_id': '7616453', 'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '5ec3e6e88b2e2783e09498588bb1f433ae9c8986'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-mchangir-testing-main-20240318.032620'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'bad backtrace on directory inode', 'error reading table object', 'Metadata damage detected', 'slow requests are blocked', 'Behind on trimming', 'error reading sessionmap'], 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'subvolume_version': 1, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-mchangir-testing-main-20240318.032620', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}}, 'owner': 'mchangir', 'priority': 100, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 3550, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'sleep_before_teardown': 0, 'subset': '613/1024', 'suite': 'fs', 'suite_branch': 'wip-mchangir-testing-main-20240318.032620', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi090.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDPxTpKXePPQNHpR4U7WKV34yOJfd7d/mNOkZ153u5OYNUU/BVD7SyJJJSic+YiPf10ceGPv0TJhVYnK5d2enao=', 'smithi120.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIsD5/kDR1BWRDWfmlrWw9avR8IC90oP1y5hOVOfAiJXGeLToZTQvDj9Ol/SdQ0iTE0wDvajJKkjTfhdzedsbTY='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '5ec3e6e88b2e2783e09498588bb1f433ae9c8986'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'bad backtrace on directory inode', 'error reading table object', 'Metadata damage detected', 'slow requests are blocked', 'Behind on trimming', 'error reading sessionmap'], 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_journal_repair']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-22_09:49:57', 'tube': 'smithi', 'user': 'mchangir', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-24T01:22:55.605 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-24T01:22:55.612 DEBUG:teuthology.orchestra.run.smithi120:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-24T01:22:55.662 DEBUG:teuthology.orchestra.run.smithi090:> sudo install -d -m0777 -- /var/run/ceph 2024-03-24T01:22:55.667 DEBUG:teuthology.orchestra.run.smithi120:> sudo install -d -m0777 -- /var/run/ceph 2024-03-24T01:22:55.761 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:22:55.761 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T01:22:55.816 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T01:22:55.816 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_1 2024-03-24T01:22:55.868 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T01:22:55.868 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T01:22:55.868 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 638 Links: 1 2024-03-24T01:22:55.868 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T01:22:55.868 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T01:22:55.868 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-03-24 01:22:25.091767230 +0000 2024-03-24T01:22:55.868 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-03-24 01:18:36.445148024 +0000 2024-03-24T01:22:55.868 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-03-24 01:18:36.445148024 +0000 2024-03-24T01:22:55.869 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-03-24 01:18:36.445148024 +0000 2024-03-24T01:22:55.869 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T01:22:55.950 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-03-24T01:22:55.950 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-03-24T01:22:55.950 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.00034914 s, 1.5 MB/s 2024-03-24T01:22:55.953 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T01:22:56.010 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_2 2024-03-24T01:22:56.060 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T01:22:56.060 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T01:22:56.060 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 659 Links: 1 2024-03-24T01:22:56.060 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T01:22:56.060 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T01:22:56.060 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-03-24 01:22:25.091767230 +0000 2024-03-24T01:22:56.060 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-03-24 01:18:37.027138617 +0000 2024-03-24T01:22:56.061 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-03-24 01:18:37.027138617 +0000 2024-03-24T01:22:56.061 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-03-24 01:18:37.027138617 +0000 2024-03-24T01:22:56.061 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T01:22:56.150 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-03-24T01:22:56.150 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-03-24T01:22:56.150 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000320394 s, 1.6 MB/s 2024-03-24T01:22:56.154 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T01:22:56.208 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_3 2024-03-24T01:22:56.254 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T01:22:56.254 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T01:22:56.255 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 677 Links: 1 2024-03-24T01:22:56.255 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T01:22:56.255 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T01:22:56.255 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-03-24 01:22:25.092767214 +0000 2024-03-24T01:22:56.255 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-03-24 01:18:37.590129518 +0000 2024-03-24T01:22:56.255 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-03-24 01:18:37.590129518 +0000 2024-03-24T01:22:56.255 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-03-24 01:18:37.590129518 +0000 2024-03-24T01:22:56.255 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T01:22:56.385 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-03-24T01:22:56.385 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-03-24T01:22:56.385 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000320552 s, 1.6 MB/s 2024-03-24T01:22:56.389 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T01:22:56.444 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_4 2024-03-24T01:22:56.492 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T01:22:56.492 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T01:22:56.492 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 695 Links: 1 2024-03-24T01:22:56.492 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T01:22:56.492 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T01:22:56.492 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-03-24 01:22:25.093767198 +0000 2024-03-24T01:22:56.492 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-03-24 01:18:38.159120322 +0000 2024-03-24T01:22:56.492 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-03-24 01:18:38.159120322 +0000 2024-03-24T01:22:56.492 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-03-24 01:18:38.159120322 +0000 2024-03-24T01:22:56.493 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T01:22:56.573 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-03-24T01:22:56.574 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-03-24T01:22:56.574 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000329407 s, 1.6 MB/s 2024-03-24T01:22:56.577 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T01:22:56.631 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-24T01:22:56.631 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:22:56.631 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T01:22:56.674 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T01:22:56.674 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_1 2024-03-24T01:22:56.763 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T01:22:56.763 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T01:22:56.763 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 634 Links: 1 2024-03-24T01:22:56.763 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T01:22:56.763 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T01:22:56.763 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-03-24 01:22:44.613401412 +0000 2024-03-24T01:22:56.763 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-03-24 01:19:47.724300810 +0000 2024-03-24T01:22:56.764 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-03-24 01:19:47.724300810 +0000 2024-03-24T01:22:56.764 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2024-03-24 01:19:47.724300810 +0000 2024-03-24T01:22:56.764 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T01:22:56.882 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-03-24T01:22:56.882 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-03-24T01:22:56.882 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000372903 s, 1.4 MB/s 2024-03-24T01:22:56.886 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T01:22:56.936 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_2 2024-03-24T01:22:56.978 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T01:22:56.978 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T01:22:56.978 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 652 Links: 1 2024-03-24T01:22:56.978 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T01:22:56.978 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T01:22:56.978 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-03-24 01:22:44.613401412 +0000 2024-03-24T01:22:56.979 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-03-24 01:19:48.282287280 +0000 2024-03-24T01:22:56.979 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-03-24 01:19:48.282287280 +0000 2024-03-24T01:22:56.979 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2024-03-24 01:19:48.282287280 +0000 2024-03-24T01:22:56.979 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T01:22:57.099 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-03-24T01:22:57.100 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-03-24T01:22:57.100 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000278016 s, 1.8 MB/s 2024-03-24T01:22:57.102 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T01:22:57.147 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_3 2024-03-24T01:22:57.241 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T01:22:57.241 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T01:22:57.241 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 671 Links: 1 2024-03-24T01:22:57.242 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T01:22:57.242 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T01:22:57.242 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-03-24 01:22:44.615401364 +0000 2024-03-24T01:22:57.242 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-03-24 01:19:48.854273411 +0000 2024-03-24T01:22:57.242 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-03-24 01:19:48.854273411 +0000 2024-03-24T01:22:57.242 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2024-03-24 01:19:48.854273411 +0000 2024-03-24T01:22:57.242 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T01:22:57.326 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-03-24T01:22:57.326 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-03-24T01:22:57.326 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000475459 s, 1.1 MB/s 2024-03-24T01:22:57.330 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T01:22:57.386 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_4 2024-03-24T01:22:57.436 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T01:22:57.436 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T01:22:57.436 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 688 Links: 1 2024-03-24T01:22:57.436 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T01:22:57.436 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T01:22:57.436 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-03-24 01:22:44.615401364 +0000 2024-03-24T01:22:57.436 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-03-24 01:19:49.410259929 +0000 2024-03-24T01:22:57.436 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-03-24 01:19:49.410259929 +0000 2024-03-24T01:22:57.436 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2024-03-24 01:19:49.410259929 +0000 2024-03-24T01:22:57.437 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T01:22:57.514 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-03-24T01:22:57.515 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-03-24T01:22:57.515 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000377268 s, 1.4 MB/s 2024-03-24T01:22:57.517 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T01:22:57.569 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-24T01:22:57.569 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi090.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi120.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-24T01:22:57.569 INFO:tasks.ceph:Generating config... 2024-03-24T01:22:57.570 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-24T01:22:57.571 INFO:tasks.ceph:[client] debug client = 20 2024-03-24T01:22:57.571 INFO:tasks.ceph:[client] debug ms = 1 2024-03-24T01:22:57.571 INFO:tasks.ceph:[client] pid file = 2024-03-24T01:22:57.571 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-24T01:22:57.571 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-24T01:22:57.571 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-24T01:22:57.571 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-24T01:22:57.571 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-24T01:22:57.571 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-24T01:22:57.571 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-24T01:22:57.571 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-24T01:22:57.571 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-24T01:22:57.571 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-24T01:22:57.572 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-24T01:22:57.572 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-24T01:22:57.572 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-24T01:22:57.572 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-24T01:22:57.572 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-24T01:22:57.572 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-24T01:22:57.572 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-24T01:22:57.572 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-24T01:22:57.572 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-24T01:22:57.572 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-24T01:22:57.572 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-24T01:22:57.572 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-24T01:22:57.572 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-24T01:22:57.573 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-24T01:22:57.573 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-24T01:22:57.573 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-24T01:22:57.573 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-24T01:22:57.573 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-24T01:22:57.573 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-24T01:22:57.573 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-24T01:22:57.573 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-24T01:22:57.573 INFO:tasks.ceph:Setting up mon.a... 2024-03-24T01:22:57.573 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-24T01:22:57.712 INFO:teuthology.orchestra.run.smithi090.stdout:creating /etc/ceph/ceph.keyring 2024-03-24T01:22:57.720 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-24T01:22:57.848 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-24T01:22:57.942 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.90'), ('mon.b', '172.21.15.120'), ('mon.c', '[v2:172.21.15.120:3301,v1:172.21.15.120:6790]')] 2024-03-24T01:22:57.942 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.90,172.21.15.120,[v2:172.21.15.120:3301,v1:172.21.15.120:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-24T01:22:57.943 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:22:57.943 DEBUG:teuthology.orchestra.run.smithi090:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-24T01:22:57.995 DEBUG:teuthology.orchestra.run.smithi090:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.90 --add b 172.21.15.120 --addv c '[v2:172.21.15.120:3301,v1:172.21.15.120:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-24T01:22:58.147 INFO:teuthology.orchestra.run.smithi090.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-24T01:22:58.147 INFO:teuthology.orchestra.run.smithi090.stdout:monmaptool: generated fsid 24712165-9839-4fcc-8033-988386efea74 2024-03-24T01:22:58.148 INFO:teuthology.orchestra.run.smithi090.stdout:setting min_mon_release = quincy 2024-03-24T01:22:58.148 INFO:teuthology.orchestra.run.smithi090.stdout:epoch 0 2024-03-24T01:22:58.148 INFO:teuthology.orchestra.run.smithi090.stdout:fsid 24712165-9839-4fcc-8033-988386efea74 2024-03-24T01:22:58.148 INFO:teuthology.orchestra.run.smithi090.stdout:last_changed 2024-03-24T01:22:58.148293+0000 2024-03-24T01:22:58.148 INFO:teuthology.orchestra.run.smithi090.stdout:created 2024-03-24T01:22:58.148293+0000 2024-03-24T01:22:58.148 INFO:teuthology.orchestra.run.smithi090.stdout:min_mon_release 17 (quincy) 2024-03-24T01:22:58.148 INFO:teuthology.orchestra.run.smithi090.stdout:election_strategy: 1 2024-03-24T01:22:58.148 INFO:teuthology.orchestra.run.smithi090.stdout:0: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.a 2024-03-24T01:22:58.148 INFO:teuthology.orchestra.run.smithi090.stdout:1: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.b 2024-03-24T01:22:58.148 INFO:teuthology.orchestra.run.smithi090.stdout:2: [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] mon.c 2024-03-24T01:22:58.148 INFO:teuthology.orchestra.run.smithi090.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-24T01:22:58.151 DEBUG:teuthology.orchestra.run.smithi090:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-24T01:22:58.199 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 24712165-9839-4fcc-8033-988386efea74... 2024-03-24T01:22:58.200 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-24T01:22:58.246 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-24T01:22:58.409 INFO:teuthology.orchestra.run.smithi090.stdout:[global] 2024-03-24T01:22:58.409 INFO:teuthology.orchestra.run.smithi090.stdout: chdir = "" 2024-03-24T01:22:58.409 INFO:teuthology.orchestra.run.smithi090.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-24T01:22:58.409 INFO:teuthology.orchestra.run.smithi090.stdout: auth supported = cephx 2024-03-24T01:22:58.409 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:58.409 INFO:teuthology.orchestra.run.smithi090.stdout: filestore xattr use omap = true 2024-03-24T01:22:58.410 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:58.410 INFO:teuthology.orchestra.run.smithi090.stdout: mon clock drift allowed = 1.000 2024-03-24T01:22:58.410 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:58.410 INFO:teuthology.orchestra.run.smithi090.stdout: osd crush chooseleaf type = 0 2024-03-24T01:22:58.410 INFO:teuthology.orchestra.run.smithi090.stdout: auth debug = true 2024-03-24T01:22:58.410 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:58.410 INFO:teuthology.orchestra.run.smithi090.stdout: ms die on old message = true 2024-03-24T01:22:58.410 INFO:teuthology.orchestra.run.smithi090.stdout: ms die on bug = true 2024-03-24T01:22:58.410 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:58.410 INFO:teuthology.orchestra.run.smithi090.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-24T01:22:58.410 INFO:teuthology.orchestra.run.smithi090.stdout: mon pg warn max object skew = 0 2024-03-24T01:22:58.410 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:58.410 INFO:teuthology.orchestra.run.smithi090.stdout: # disable pg_autoscaler by default for new pools 2024-03-24T01:22:58.410 INFO:teuthology.orchestra.run.smithi090.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-24T01:22:58.411 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:58.411 INFO:teuthology.orchestra.run.smithi090.stdout: osd pool default size = 2 2024-03-24T01:22:58.411 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:58.411 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd allow primary affinity = true 2024-03-24T01:22:58.411 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd allow pg remap = true 2024-03-24T01:22:58.411 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on legacy crush tunables = false 2024-03-24T01:22:58.411 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on crush straw calc version zero = false 2024-03-24T01:22:58.411 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on no sortbitwise = false 2024-03-24T01:22:58.411 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on osd down out interval zero = false 2024-03-24T01:22:58.411 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on too few osds = false 2024-03-24T01:22:58.411 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-24T01:22:58.411 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-24T01:22:58.411 INFO:teuthology.orchestra.run.smithi090.stdout: mon_allow_pool_size_one = true 2024-03-24T01:22:58.411 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:58.412 INFO:teuthology.orchestra.run.smithi090.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-24T01:22:58.412 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:58.412 INFO:teuthology.orchestra.run.smithi090.stdout: osd default data pool replay window = 5 2024-03-24T01:22:58.412 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:58.412 INFO:teuthology.orchestra.run.smithi090.stdout: mon allow pool delete = true 2024-03-24T01:22:58.412 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:58.412 INFO:teuthology.orchestra.run.smithi090.stdout: mon cluster log file level = debug 2024-03-24T01:22:58.412 INFO:teuthology.orchestra.run.smithi090.stdout: debug asserts on shutdown = true 2024-03-24T01:22:58.412 INFO:teuthology.orchestra.run.smithi090.stdout: mon health detail to clog = false 2024-03-24T01:22:58.412 INFO:teuthology.orchestra.run.smithi090.stdout: mon host = "172.21.15.90,172.21.15.120,[v2:172.21.15.120:3301,v1:172.21.15.120:6790]" 2024-03-24T01:22:58.412 INFO:teuthology.orchestra.run.smithi090.stdout: fsid = 24712165-9839-4fcc-8033-988386efea74 2024-03-24T01:22:58.412 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:58.412 INFO:teuthology.orchestra.run.smithi090.stdout:[osd] 2024-03-24T01:22:58.412 INFO:teuthology.orchestra.run.smithi090.stdout: osd journal size = 100 2024-03-24T01:22:58.412 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:58.413 INFO:teuthology.orchestra.run.smithi090.stdout: osd scrub load threshold = 5.0 2024-03-24T01:22:58.413 INFO:teuthology.orchestra.run.smithi090.stdout: osd scrub max interval = 600 2024-03-24T01:22:58.413 INFO:teuthology.orchestra.run.smithi090.stdout: osd mclock profile = high_recovery_ops 2024-03-24T01:22:58.413 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:58.413 INFO:teuthology.orchestra.run.smithi090.stdout: osd recover clone overlap = true 2024-03-24T01:22:58.413 INFO:teuthology.orchestra.run.smithi090.stdout: osd recovery max chunk = 1048576 2024-03-24T01:22:58.413 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:58.413 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug shutdown = true 2024-03-24T01:22:58.413 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug op order = true 2024-03-24T01:22:58.413 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug verify stray on activate = true 2024-03-24T01:22:58.413 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug trim objects = true 2024-03-24T01:22:58.413 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:58.413 INFO:teuthology.orchestra.run.smithi090.stdout: osd open classes on start = true 2024-03-24T01:22:58.413 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug pg log writeout = true 2024-03-24T01:22:58.413 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:58.414 INFO:teuthology.orchestra.run.smithi090.stdout: osd deep scrub update digest min age = 30 2024-03-24T01:22:58.414 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:58.414 INFO:teuthology.orchestra.run.smithi090.stdout: osd map max advance = 10 2024-03-24T01:22:58.414 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:58.414 INFO:teuthology.orchestra.run.smithi090.stdout: journal zero on create = true 2024-03-24T01:22:58.414 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:58.414 INFO:teuthology.orchestra.run.smithi090.stdout: filestore ondisk finisher threads = 3 2024-03-24T01:22:58.414 INFO:teuthology.orchestra.run.smithi090.stdout: filestore apply finisher threads = 3 2024-03-24T01:22:58.414 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:58.414 INFO:teuthology.orchestra.run.smithi090.stdout: bdev debug aio = true 2024-03-24T01:22:58.414 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug misdirected ops = true 2024-03-24T01:22:58.414 INFO:teuthology.orchestra.run.smithi090.stdout: bluestore block size = 96636764160 2024-03-24T01:22:58.414 INFO:teuthology.orchestra.run.smithi090.stdout: bluestore fsck on mount = True 2024-03-24T01:22:58.414 INFO:teuthology.orchestra.run.smithi090.stdout: debug bluefs = 20 2024-03-24T01:22:58.414 INFO:teuthology.orchestra.run.smithi090.stdout: debug bluestore = 20 2024-03-24T01:22:58.415 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2024-03-24T01:22:58.415 INFO:teuthology.orchestra.run.smithi090.stdout: debug osd = 20 2024-03-24T01:22:58.415 INFO:teuthology.orchestra.run.smithi090.stdout: debug rocksdb = 10 2024-03-24T01:22:58.415 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd backfillfull_ratio = 0.85 2024-03-24T01:22:58.415 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd full ratio = 0.9 2024-03-24T01:22:58.415 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd nearfull ratio = 0.8 2024-03-24T01:22:58.415 INFO:teuthology.orchestra.run.smithi090.stdout: osd failsafe full ratio = 0.95 2024-03-24T01:22:58.415 INFO:teuthology.orchestra.run.smithi090.stdout: osd objectstore = bluestore 2024-03-24T01:22:58.415 INFO:teuthology.orchestra.run.smithi090.stdout: osd op complaint time = 180 2024-03-24T01:22:58.415 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:58.415 INFO:teuthology.orchestra.run.smithi090.stdout:[mgr] 2024-03-24T01:22:58.415 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2024-03-24T01:22:58.415 INFO:teuthology.orchestra.run.smithi090.stdout: debug mgr = 20 2024-03-24T01:22:58.415 INFO:teuthology.orchestra.run.smithi090.stdout: debug mon = 20 2024-03-24T01:22:58.415 INFO:teuthology.orchestra.run.smithi090.stdout: debug auth = 20 2024-03-24T01:22:58.416 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min pgs per osd = 4 2024-03-24T01:22:58.416 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min bytes per osd = 10 2024-03-24T01:22:58.416 INFO:teuthology.orchestra.run.smithi090.stdout: mgr/telemetry/nag = false 2024-03-24T01:22:58.416 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:58.416 INFO:teuthology.orchestra.run.smithi090.stdout:[mon] 2024-03-24T01:22:58.416 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2024-03-24T01:22:58.416 INFO:teuthology.orchestra.run.smithi090.stdout: debug mon = 20 2024-03-24T01:22:58.416 INFO:teuthology.orchestra.run.smithi090.stdout: debug paxos = 20 2024-03-24T01:22:58.416 INFO:teuthology.orchestra.run.smithi090.stdout: debug auth = 20 2024-03-24T01:22:58.416 INFO:teuthology.orchestra.run.smithi090.stdout: mon data avail warn = 5 2024-03-24T01:22:58.416 INFO:teuthology.orchestra.run.smithi090.stdout: mon mgr mkfs grace = 240 2024-03-24T01:22:58.416 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min pgs per osd = 4 2024-03-24T01:22:58.416 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd reporter subtree level = osd 2024-03-24T01:22:58.417 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd prime pg temp = true 2024-03-24T01:22:58.417 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min bytes per osd = 10 2024-03-24T01:22:58.417 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:58.417 INFO:teuthology.orchestra.run.smithi090.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-24T01:22:58.418 INFO:teuthology.orchestra.run.smithi090.stdout: auth mon ticket ttl = 660 # 11m 2024-03-24T01:22:58.418 INFO:teuthology.orchestra.run.smithi090.stdout: auth service ticket ttl = 240 # 4m 2024-03-24T01:22:58.418 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:58.418 INFO:teuthology.orchestra.run.smithi090.stdout: # don't complain about insecure global_id in the test suite 2024-03-24T01:22:58.418 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-24T01:22:58.418 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-24T01:22:58.418 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:58.418 INFO:teuthology.orchestra.run.smithi090.stdout: # 1m isn't quite enough 2024-03-24T01:22:58.418 INFO:teuthology.orchestra.run.smithi090.stdout: mon_down_mkfs_grace = 2m 2024-03-24T01:22:58.418 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:58.418 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_filestore_osds = false 2024-03-24T01:22:58.418 INFO:teuthology.orchestra.run.smithi090.stdout: mon op complaint time = 120 2024-03-24T01:22:58.418 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:22:58.418 INFO:teuthology.orchestra.run.smithi090.stdout:[client] 2024-03-24T01:22:58.419 INFO:teuthology.orchestra.run.smithi090.stdout: rgw cache enabled = true 2024-03-24T01:22:58.419 INFO:teuthology.orchestra.run.smithi090.stdout: rgw enable ops log = true 2024-03-24T01:22:58.419 INFO:teuthology.orchestra.run.smithi090.stdout: rgw enable usage log = true 2024-03-24T01:22:58.419 INFO:teuthology.orchestra.run.smithi090.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-24T01:22:58.419 INFO:teuthology.orchestra.run.smithi090.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-24T01:22:58.419 INFO:teuthology.orchestra.run.smithi090.stdout: client mount timeout = 600 2024-03-24T01:22:58.419 INFO:teuthology.orchestra.run.smithi090.stdout: debug client = 20 2024-03-24T01:22:58.419 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2024-03-24T01:22:58.419 INFO:teuthology.orchestra.run.smithi090.stdout: pid file = "" 2024-03-24T01:22:58.419 INFO:teuthology.orchestra.run.smithi090.stdout: rados mon op timeout = 900 2024-03-24T01:22:58.419 INFO:teuthology.orchestra.run.smithi090.stdout: rados osd op timeout = 900 2024-03-24T01:22:58.419 INFO:teuthology.orchestra.run.smithi090.stdout:[mon.a] 2024-03-24T01:22:58.419 INFO:teuthology.orchestra.run.smithi090.stdout:[mon.b] 2024-03-24T01:22:58.419 INFO:teuthology.orchestra.run.smithi090.stdout:[mon.c] 2024-03-24T01:22:58.420 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.a] 2024-03-24T01:22:58.420 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.c] 2024-03-24T01:22:58.420 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.b] 2024-03-24T01:22:58.420 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.d] 2024-03-24T01:22:58.420 INFO:teuthology.orchestra.run.smithi090.stdout:[mds] 2024-03-24T01:22:58.420 INFO:teuthology.orchestra.run.smithi090.stdout: debug mds = 20 2024-03-24T01:22:58.420 INFO:teuthology.orchestra.run.smithi090.stdout: debug mds balancer = 20 2024-03-24T01:22:58.420 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2024-03-24T01:22:58.420 INFO:teuthology.orchestra.run.smithi090.stdout: mds debug frag = True 2024-03-24T01:22:58.420 INFO:teuthology.orchestra.run.smithi090.stdout: mds debug scatterstat = True 2024-03-24T01:22:58.420 INFO:teuthology.orchestra.run.smithi090.stdout: mds op complaint time = 180 2024-03-24T01:22:58.420 INFO:teuthology.orchestra.run.smithi090.stdout: mds verify scatter = True 2024-03-24T01:22:58.420 INFO:teuthology.orchestra.run.smithi090.stdout: osd op complaint time = 180 2024-03-24T01:22:58.420 INFO:teuthology.orchestra.run.smithi090.stdout: rados mon op timeout = 900 2024-03-24T01:22:58.420 INFO:teuthology.orchestra.run.smithi090.stdout: rados osd op timeout = 900 2024-03-24T01:22:58.421 INFO:teuthology.orchestra.run.smithi120.stdout:[global] 2024-03-24T01:22:58.421 INFO:teuthology.orchestra.run.smithi120.stdout: chdir = "" 2024-03-24T01:22:58.421 INFO:teuthology.orchestra.run.smithi120.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-24T01:22:58.421 INFO:teuthology.orchestra.run.smithi120.stdout: auth supported = cephx 2024-03-24T01:22:58.421 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:58.421 INFO:teuthology.orchestra.run.smithi120.stdout: filestore xattr use omap = true 2024-03-24T01:22:58.421 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:58.422 INFO:teuthology.orchestra.run.smithi120.stdout: mon clock drift allowed = 1.000 2024-03-24T01:22:58.422 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:58.422 INFO:teuthology.orchestra.run.smithi120.stdout: osd crush chooseleaf type = 0 2024-03-24T01:22:58.422 INFO:teuthology.orchestra.run.smithi120.stdout: auth debug = true 2024-03-24T01:22:58.422 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:58.422 INFO:teuthology.orchestra.run.smithi120.stdout: ms die on old message = true 2024-03-24T01:22:58.422 INFO:teuthology.orchestra.run.smithi120.stdout: ms die on bug = true 2024-03-24T01:22:58.422 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:58.422 INFO:teuthology.orchestra.run.smithi120.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-24T01:22:58.422 INFO:teuthology.orchestra.run.smithi120.stdout: mon pg warn max object skew = 0 2024-03-24T01:22:58.422 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:58.422 INFO:teuthology.orchestra.run.smithi120.stdout: # disable pg_autoscaler by default for new pools 2024-03-24T01:22:58.422 INFO:teuthology.orchestra.run.smithi120.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-24T01:22:58.423 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:58.423 INFO:teuthology.orchestra.run.smithi120.stdout: osd pool default size = 2 2024-03-24T01:22:58.423 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:58.423 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd allow primary affinity = true 2024-03-24T01:22:58.423 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd allow pg remap = true 2024-03-24T01:22:58.423 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on legacy crush tunables = false 2024-03-24T01:22:58.423 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on crush straw calc version zero = false 2024-03-24T01:22:58.423 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on no sortbitwise = false 2024-03-24T01:22:58.423 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on osd down out interval zero = false 2024-03-24T01:22:58.423 INFO:teuthology.orchestra.run.smithi120.stdout: mon warn on too few osds = false 2024-03-24T01:22:58.423 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-24T01:22:58.423 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-24T01:22:58.423 INFO:teuthology.orchestra.run.smithi120.stdout: mon_allow_pool_size_one = true 2024-03-24T01:22:58.423 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:58.423 INFO:teuthology.orchestra.run.smithi120.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-24T01:22:58.424 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:58.424 INFO:teuthology.orchestra.run.smithi120.stdout: osd default data pool replay window = 5 2024-03-24T01:22:58.424 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:58.424 INFO:teuthology.orchestra.run.smithi120.stdout: mon allow pool delete = true 2024-03-24T01:22:58.424 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:58.424 INFO:teuthology.orchestra.run.smithi120.stdout: mon cluster log file level = debug 2024-03-24T01:22:58.424 INFO:teuthology.orchestra.run.smithi120.stdout: debug asserts on shutdown = true 2024-03-24T01:22:58.424 INFO:teuthology.orchestra.run.smithi120.stdout: mon health detail to clog = false 2024-03-24T01:22:58.424 INFO:teuthology.orchestra.run.smithi120.stdout: mon host = "172.21.15.90,172.21.15.120,[v2:172.21.15.120:3301,v1:172.21.15.120:6790]" 2024-03-24T01:22:58.424 INFO:teuthology.orchestra.run.smithi120.stdout: fsid = 24712165-9839-4fcc-8033-988386efea74 2024-03-24T01:22:58.424 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:58.424 INFO:teuthology.orchestra.run.smithi120.stdout:[osd] 2024-03-24T01:22:58.424 INFO:teuthology.orchestra.run.smithi120.stdout: osd journal size = 100 2024-03-24T01:22:58.424 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:58.424 INFO:teuthology.orchestra.run.smithi120.stdout: osd scrub load threshold = 5.0 2024-03-24T01:22:58.425 INFO:teuthology.orchestra.run.smithi120.stdout: osd scrub max interval = 600 2024-03-24T01:22:58.425 INFO:teuthology.orchestra.run.smithi120.stdout: osd mclock profile = high_recovery_ops 2024-03-24T01:22:58.425 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:58.425 INFO:teuthology.orchestra.run.smithi120.stdout: osd recover clone overlap = true 2024-03-24T01:22:58.425 INFO:teuthology.orchestra.run.smithi120.stdout: osd recovery max chunk = 1048576 2024-03-24T01:22:58.425 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:58.425 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug shutdown = true 2024-03-24T01:22:58.425 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug op order = true 2024-03-24T01:22:58.425 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug verify stray on activate = true 2024-03-24T01:22:58.425 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug trim objects = true 2024-03-24T01:22:58.425 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:58.425 INFO:teuthology.orchestra.run.smithi120.stdout: osd open classes on start = true 2024-03-24T01:22:58.425 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug pg log writeout = true 2024-03-24T01:22:58.425 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:58.425 INFO:teuthology.orchestra.run.smithi120.stdout: osd deep scrub update digest min age = 30 2024-03-24T01:22:58.426 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:58.426 INFO:teuthology.orchestra.run.smithi120.stdout: osd map max advance = 10 2024-03-24T01:22:58.426 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:58.426 INFO:teuthology.orchestra.run.smithi120.stdout: journal zero on create = true 2024-03-24T01:22:58.426 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:58.426 INFO:teuthology.orchestra.run.smithi120.stdout: filestore ondisk finisher threads = 3 2024-03-24T01:22:58.426 INFO:teuthology.orchestra.run.smithi120.stdout: filestore apply finisher threads = 3 2024-03-24T01:22:58.426 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:58.426 INFO:teuthology.orchestra.run.smithi120.stdout: bdev debug aio = true 2024-03-24T01:22:58.426 INFO:teuthology.orchestra.run.smithi120.stdout: osd debug misdirected ops = true 2024-03-24T01:22:58.426 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore block size = 96636764160 2024-03-24T01:22:58.426 INFO:teuthology.orchestra.run.smithi120.stdout: bluestore fsck on mount = True 2024-03-24T01:22:58.426 INFO:teuthology.orchestra.run.smithi120.stdout: debug bluefs = 20 2024-03-24T01:22:58.426 INFO:teuthology.orchestra.run.smithi120.stdout: debug bluestore = 20 2024-03-24T01:22:58.427 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-03-24T01:22:58.427 INFO:teuthology.orchestra.run.smithi120.stdout: debug osd = 20 2024-03-24T01:22:58.427 INFO:teuthology.orchestra.run.smithi120.stdout: debug rocksdb = 10 2024-03-24T01:22:58.427 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd backfillfull_ratio = 0.85 2024-03-24T01:22:58.427 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd full ratio = 0.9 2024-03-24T01:22:58.427 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd nearfull ratio = 0.8 2024-03-24T01:22:58.427 INFO:teuthology.orchestra.run.smithi120.stdout: osd failsafe full ratio = 0.95 2024-03-24T01:22:58.427 INFO:teuthology.orchestra.run.smithi120.stdout: osd objectstore = bluestore 2024-03-24T01:22:58.427 INFO:teuthology.orchestra.run.smithi120.stdout: osd op complaint time = 180 2024-03-24T01:22:58.427 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:58.427 INFO:teuthology.orchestra.run.smithi120.stdout:[mgr] 2024-03-24T01:22:58.427 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-03-24T01:22:58.427 INFO:teuthology.orchestra.run.smithi120.stdout: debug mgr = 20 2024-03-24T01:22:58.427 INFO:teuthology.orchestra.run.smithi120.stdout: debug mon = 20 2024-03-24T01:22:58.427 INFO:teuthology.orchestra.run.smithi120.stdout: debug auth = 20 2024-03-24T01:22:58.428 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min pgs per osd = 4 2024-03-24T01:22:58.428 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min bytes per osd = 10 2024-03-24T01:22:58.428 INFO:teuthology.orchestra.run.smithi120.stdout: mgr/telemetry/nag = false 2024-03-24T01:22:58.428 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:58.428 INFO:teuthology.orchestra.run.smithi120.stdout:[mon] 2024-03-24T01:22:58.428 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-03-24T01:22:58.428 INFO:teuthology.orchestra.run.smithi120.stdout: debug mon = 20 2024-03-24T01:22:58.428 INFO:teuthology.orchestra.run.smithi120.stdout: debug paxos = 20 2024-03-24T01:22:58.428 INFO:teuthology.orchestra.run.smithi120.stdout: debug auth = 20 2024-03-24T01:22:58.428 INFO:teuthology.orchestra.run.smithi120.stdout: mon data avail warn = 5 2024-03-24T01:22:58.428 INFO:teuthology.orchestra.run.smithi120.stdout: mon mgr mkfs grace = 240 2024-03-24T01:22:58.428 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min pgs per osd = 4 2024-03-24T01:22:58.428 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd reporter subtree level = osd 2024-03-24T01:22:58.428 INFO:teuthology.orchestra.run.smithi120.stdout: mon osd prime pg temp = true 2024-03-24T01:22:58.428 INFO:teuthology.orchestra.run.smithi120.stdout: mon reweight min bytes per osd = 10 2024-03-24T01:22:58.429 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:58.429 INFO:teuthology.orchestra.run.smithi120.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-24T01:22:58.429 INFO:teuthology.orchestra.run.smithi120.stdout: auth mon ticket ttl = 660 # 11m 2024-03-24T01:22:58.429 INFO:teuthology.orchestra.run.smithi120.stdout: auth service ticket ttl = 240 # 4m 2024-03-24T01:22:58.429 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:58.429 INFO:teuthology.orchestra.run.smithi120.stdout: # don't complain about insecure global_id in the test suite 2024-03-24T01:22:58.429 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-24T01:22:58.429 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-24T01:22:58.429 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:58.429 INFO:teuthology.orchestra.run.smithi120.stdout: # 1m isn't quite enough 2024-03-24T01:22:58.429 INFO:teuthology.orchestra.run.smithi120.stdout: mon_down_mkfs_grace = 2m 2024-03-24T01:22:58.429 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:58.429 INFO:teuthology.orchestra.run.smithi120.stdout: mon_warn_on_filestore_osds = false 2024-03-24T01:22:58.429 INFO:teuthology.orchestra.run.smithi120.stdout: mon op complaint time = 120 2024-03-24T01:22:58.430 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-24T01:22:58.430 INFO:teuthology.orchestra.run.smithi120.stdout:[client] 2024-03-24T01:22:58.430 INFO:teuthology.orchestra.run.smithi120.stdout: rgw cache enabled = true 2024-03-24T01:22:58.430 INFO:teuthology.orchestra.run.smithi120.stdout: rgw enable ops log = true 2024-03-24T01:22:58.430 INFO:teuthology.orchestra.run.smithi120.stdout: rgw enable usage log = true 2024-03-24T01:22:58.430 INFO:teuthology.orchestra.run.smithi120.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-24T01:22:58.430 INFO:teuthology.orchestra.run.smithi120.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-24T01:22:58.430 INFO:teuthology.orchestra.run.smithi120.stdout: client mount timeout = 600 2024-03-24T01:22:58.430 INFO:teuthology.orchestra.run.smithi120.stdout: debug client = 20 2024-03-24T01:22:58.430 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-03-24T01:22:58.430 INFO:teuthology.orchestra.run.smithi120.stdout: pid file = "" 2024-03-24T01:22:58.430 INFO:teuthology.orchestra.run.smithi120.stdout: rados mon op timeout = 900 2024-03-24T01:22:58.430 INFO:teuthology.orchestra.run.smithi120.stdout: rados osd op timeout = 900 2024-03-24T01:22:58.430 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.a] 2024-03-24T01:22:58.430 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.b] 2024-03-24T01:22:58.431 INFO:teuthology.orchestra.run.smithi120.stdout:[mon.c] 2024-03-24T01:22:58.431 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.a] 2024-03-24T01:22:58.431 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.c] 2024-03-24T01:22:58.431 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.b] 2024-03-24T01:22:58.431 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.d] 2024-03-24T01:22:58.431 INFO:teuthology.orchestra.run.smithi120.stdout:[mds] 2024-03-24T01:22:58.431 INFO:teuthology.orchestra.run.smithi120.stdout: debug mds = 20 2024-03-24T01:22:58.431 INFO:teuthology.orchestra.run.smithi120.stdout: debug mds balancer = 20 2024-03-24T01:22:58.431 INFO:teuthology.orchestra.run.smithi120.stdout: debug ms = 1 2024-03-24T01:22:58.431 INFO:teuthology.orchestra.run.smithi120.stdout: mds debug frag = True 2024-03-24T01:22:58.431 INFO:teuthology.orchestra.run.smithi120.stdout: mds debug scatterstat = True 2024-03-24T01:22:58.431 INFO:teuthology.orchestra.run.smithi120.stdout: mds op complaint time = 180 2024-03-24T01:22:58.431 INFO:teuthology.orchestra.run.smithi120.stdout: mds verify scatter = True 2024-03-24T01:22:58.431 INFO:teuthology.orchestra.run.smithi120.stdout: osd op complaint time = 180 2024-03-24T01:22:58.431 INFO:teuthology.orchestra.run.smithi120.stdout: rados mon op timeout = 900 2024-03-24T01:22:58.432 INFO:teuthology.orchestra.run.smithi120.stdout: rados osd op timeout = 900 2024-03-24T01:22:58.457 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-24T01:22:58.458 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-24T01:22:58.601 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-24T01:22:58.602 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:22:58.602 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-24T01:22:58.651 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:22:58.652 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-24T01:22:58.747 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi090.front.sepia.ceph.com 2024-03-24T01:22:58.747 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:22:58.748 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-24T01:22:58.748 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-24T01:22:58.888 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:22:58.888 DEBUG:teuthology.orchestra.run.smithi090:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-24T01:22:58.986 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi120.front.sepia.ceph.com 2024-03-24T01:22:58.986 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:22:58.986 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-24T01:22:58.986 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-24T01:22:59.134 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:22:59.134 DEBUG:teuthology.orchestra.run.smithi120:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-24T01:22:59.230 INFO:tasks.ceph:Setting up mon nodes... 2024-03-24T01:22:59.230 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-24T01:22:59.230 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-24T01:22:59.417 INFO:teuthology.orchestra.run.smithi090.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-24T01:22:59.428 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-24T01:22:59.618 INFO:teuthology.orchestra.run.smithi120.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-24T01:22:59.630 INFO:tasks.ceph:Setting up mds nodes... 2024-03-24T01:22:59.630 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-24T01:22:59.815 INFO:teuthology.orchestra.run.smithi090.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-24T01:22:59.824 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-24T01:22:59.914 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-24T01:23:00.077 INFO:teuthology.orchestra.run.smithi090.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-24T01:23:00.085 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-24T01:23:00.173 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-24T01:23:00.359 INFO:teuthology.orchestra.run.smithi120.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-24T01:23:00.368 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-24T01:23:00.460 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-24T01:23:00.637 INFO:teuthology.orchestra.run.smithi120.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-24T01:23:00.646 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-24T01:23:00.737 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-24T01:23:00.738 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-24T01:23:00.864 INFO:teuthology.orchestra.run.smithi090.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-24T01:23:00.917 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-24T01:23:01.037 INFO:teuthology.orchestra.run.smithi090.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-03-24T01:23:01.091 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-24T01:23:01.220 INFO:teuthology.orchestra.run.smithi120.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-24T01:23:01.278 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-03-24T01:23:01.405 INFO:teuthology.orchestra.run.smithi120.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-03-24T01:23:01.454 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-24T01:23:01.454 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi090.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi120.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-24T01:23:01.454 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-24T01:23:01.549 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-24T01:23:01.550 INFO:tasks.ceph:role: osd.0 2024-03-24T01:23:01.550 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi090.front.sepia.ceph.com 2024-03-24T01:23:01.550 DEBUG:teuthology.orchestra.run.smithi090:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-24T01:23:01.643 INFO:teuthology.orchestra.run.smithi090.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T01:23:01.643 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T01:23:01.643 INFO:teuthology.orchestra.run.smithi090.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T01:23:01.644 INFO:teuthology.orchestra.run.smithi090.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-24T01:23:01.644 INFO:teuthology.orchestra.run.smithi090.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T01:23:01.644 INFO:teuthology.orchestra.run.smithi090.stdout: = sunit=0 swidth=0 blks 2024-03-24T01:23:01.644 INFO:teuthology.orchestra.run.smithi090.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T01:23:01.644 INFO:teuthology.orchestra.run.smithi090.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-24T01:23:01.644 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T01:23:01.644 INFO:teuthology.orchestra.run.smithi090.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T01:23:02.165 INFO:teuthology.orchestra.run.smithi090.stdout:Discarding blocks...Done. 2024-03-24T01:23:02.172 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi090.front.sepia.ceph.com -o noatime 2024-03-24T01:23:02.172 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-24T01:23:02.359 DEBUG:teuthology.orchestra.run.smithi090:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-24T01:23:02.461 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-24T01:23:02.551 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-24T01:23:02.551 INFO:tasks.ceph:role: osd.1 2024-03-24T01:23:02.551 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi090.front.sepia.ceph.com 2024-03-24T01:23:02.551 DEBUG:teuthology.orchestra.run.smithi090:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-24T01:23:02.650 INFO:teuthology.orchestra.run.smithi090.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T01:23:02.650 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T01:23:02.650 INFO:teuthology.orchestra.run.smithi090.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T01:23:02.650 INFO:teuthology.orchestra.run.smithi090.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-24T01:23:02.650 INFO:teuthology.orchestra.run.smithi090.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T01:23:02.650 INFO:teuthology.orchestra.run.smithi090.stdout: = sunit=0 swidth=0 blks 2024-03-24T01:23:02.650 INFO:teuthology.orchestra.run.smithi090.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T01:23:02.650 INFO:teuthology.orchestra.run.smithi090.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-24T01:23:02.651 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T01:23:02.651 INFO:teuthology.orchestra.run.smithi090.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T01:23:03.144 INFO:teuthology.orchestra.run.smithi090.stdout:Discarding blocks...Done. 2024-03-24T01:23:03.151 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi090.front.sepia.ceph.com -o noatime 2024-03-24T01:23:03.151 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-24T01:23:03.305 DEBUG:teuthology.orchestra.run.smithi090:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-24T01:23:03.401 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-24T01:23:03.488 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-24T01:23:03.489 INFO:tasks.ceph:role: osd.2 2024-03-24T01:23:03.489 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi090.front.sepia.ceph.com 2024-03-24T01:23:03.489 DEBUG:teuthology.orchestra.run.smithi090:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-24T01:23:03.590 INFO:teuthology.orchestra.run.smithi090.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T01:23:03.591 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T01:23:03.591 INFO:teuthology.orchestra.run.smithi090.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T01:23:03.591 INFO:teuthology.orchestra.run.smithi090.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-24T01:23:03.591 INFO:teuthology.orchestra.run.smithi090.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T01:23:03.591 INFO:teuthology.orchestra.run.smithi090.stdout: = sunit=0 swidth=0 blks 2024-03-24T01:23:03.591 INFO:teuthology.orchestra.run.smithi090.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T01:23:03.591 INFO:teuthology.orchestra.run.smithi090.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-24T01:23:03.591 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T01:23:03.591 INFO:teuthology.orchestra.run.smithi090.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T01:23:04.114 INFO:teuthology.orchestra.run.smithi090.stdout:Discarding blocks...Done. 2024-03-24T01:23:04.121 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi090.front.sepia.ceph.com -o noatime 2024-03-24T01:23:04.121 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-24T01:23:04.313 DEBUG:teuthology.orchestra.run.smithi090:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-24T01:23:04.417 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-24T01:23:04.509 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-24T01:23:04.509 INFO:tasks.ceph:role: osd.3 2024-03-24T01:23:04.509 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi090.front.sepia.ceph.com 2024-03-24T01:23:04.509 DEBUG:teuthology.orchestra.run.smithi090:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-24T01:23:04.605 INFO:teuthology.orchestra.run.smithi090.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T01:23:04.605 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T01:23:04.605 INFO:teuthology.orchestra.run.smithi090.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T01:23:04.605 INFO:teuthology.orchestra.run.smithi090.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-24T01:23:04.605 INFO:teuthology.orchestra.run.smithi090.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T01:23:04.605 INFO:teuthology.orchestra.run.smithi090.stdout: = sunit=0 swidth=0 blks 2024-03-24T01:23:04.605 INFO:teuthology.orchestra.run.smithi090.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T01:23:04.605 INFO:teuthology.orchestra.run.smithi090.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-24T01:23:04.605 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T01:23:04.605 INFO:teuthology.orchestra.run.smithi090.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T01:23:05.070 INFO:teuthology.orchestra.run.smithi090.stdout:Discarding blocks...Done. 2024-03-24T01:23:05.076 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi090.front.sepia.ceph.com -o noatime 2024-03-24T01:23:05.076 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-24T01:23:05.215 DEBUG:teuthology.orchestra.run.smithi090:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-24T01:23:05.299 DEBUG:teuthology.orchestra.run.smithi090:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T01:23:05.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:05.431+0000 7f5054173640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-24T01:23:05.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:05.432+0000 7f5054173640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-24T01:23:05.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:05.432+0000 7f5054173640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-24T01:23:05.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:05.433+0000 7f5054173640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-24T01:23:05.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:05.433+0000 7f5054173640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-24T01:23:05.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:05.442+0000 7f5054173640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-24T01:23:08.371 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-24T01:23:08.468 DEBUG:teuthology.orchestra.run.smithi090:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T01:23:08.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:08.601+0000 7f4c0acb1640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-24T01:23:08.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:08.602+0000 7f4c0acb1640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-24T01:23:08.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:08.602+0000 7f4c0acb1640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-24T01:23:08.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:08.602+0000 7f4c0acb1640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-24T01:23:08.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:08.602+0000 7f4c0acb1640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-24T01:23:08.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:08.604+0000 7f4c0acb1640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-24T01:23:11.529 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-24T01:23:11.625 DEBUG:teuthology.orchestra.run.smithi090:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T01:23:11.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:11.760+0000 7f8416df8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-24T01:23:11.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:11.761+0000 7f8416df8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-24T01:23:11.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:11.761+0000 7f8416df8640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-24T01:23:11.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:11.761+0000 7f8416df8640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-24T01:23:11.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:11.761+0000 7f8416df8640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-24T01:23:11.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:11.763+0000 7f8416df8640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-24T01:23:14.686 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-24T01:23:14.782 DEBUG:teuthology.orchestra.run.smithi090:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T01:23:14.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:14.922+0000 7f69f3911640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-24T01:23:14.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:14.922+0000 7f69f3911640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-24T01:23:14.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:14.923+0000 7f69f3911640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-24T01:23:14.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:14.923+0000 7f69f3911640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-24T01:23:14.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:14.923+0000 7f69f3911640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-24T01:23:14.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:14.925+0000 7f69f3911640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-24T01:23:18.103 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-24T01:23:18.200 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-24T01:23:18.297 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-24T01:23:18.297 INFO:tasks.ceph:role: osd.4 2024-03-24T01:23:18.297 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi120.front.sepia.ceph.com 2024-03-24T01:23:18.297 DEBUG:teuthology.orchestra.run.smithi120:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-24T01:23:18.393 INFO:teuthology.orchestra.run.smithi120.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T01:23:18.394 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T01:23:18.394 INFO:teuthology.orchestra.run.smithi120.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T01:23:18.394 INFO:teuthology.orchestra.run.smithi120.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-24T01:23:18.394 INFO:teuthology.orchestra.run.smithi120.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T01:23:18.394 INFO:teuthology.orchestra.run.smithi120.stdout: = sunit=0 swidth=0 blks 2024-03-24T01:23:18.394 INFO:teuthology.orchestra.run.smithi120.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T01:23:18.394 INFO:teuthology.orchestra.run.smithi120.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-24T01:23:18.394 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T01:23:18.394 INFO:teuthology.orchestra.run.smithi120.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T01:23:18.917 INFO:teuthology.orchestra.run.smithi120.stdout:Discarding blocks...Done. 2024-03-24T01:23:18.924 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi120.front.sepia.ceph.com -o noatime 2024-03-24T01:23:18.924 DEBUG:teuthology.orchestra.run.smithi120:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-24T01:23:19.118 DEBUG:teuthology.orchestra.run.smithi120:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-24T01:23:19.230 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-24T01:23:19.313 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-24T01:23:19.313 INFO:tasks.ceph:role: osd.5 2024-03-24T01:23:19.313 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi120.front.sepia.ceph.com 2024-03-24T01:23:19.313 DEBUG:teuthology.orchestra.run.smithi120:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-24T01:23:19.403 INFO:teuthology.orchestra.run.smithi120.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T01:23:19.403 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T01:23:19.403 INFO:teuthology.orchestra.run.smithi120.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T01:23:19.403 INFO:teuthology.orchestra.run.smithi120.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-24T01:23:19.403 INFO:teuthology.orchestra.run.smithi120.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T01:23:19.404 INFO:teuthology.orchestra.run.smithi120.stdout: = sunit=0 swidth=0 blks 2024-03-24T01:23:19.404 INFO:teuthology.orchestra.run.smithi120.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T01:23:19.404 INFO:teuthology.orchestra.run.smithi120.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-24T01:23:19.404 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T01:23:19.404 INFO:teuthology.orchestra.run.smithi120.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T01:23:19.884 INFO:teuthology.orchestra.run.smithi120.stdout:Discarding blocks...Done. 2024-03-24T01:23:19.892 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi120.front.sepia.ceph.com -o noatime 2024-03-24T01:23:19.892 DEBUG:teuthology.orchestra.run.smithi120:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-24T01:23:20.013 DEBUG:teuthology.orchestra.run.smithi120:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-24T01:23:20.100 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-24T01:23:20.184 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-24T01:23:20.184 INFO:tasks.ceph:role: osd.6 2024-03-24T01:23:20.184 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi120.front.sepia.ceph.com 2024-03-24T01:23:20.185 DEBUG:teuthology.orchestra.run.smithi120:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-24T01:23:20.263 INFO:teuthology.orchestra.run.smithi120.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T01:23:20.263 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T01:23:20.263 INFO:teuthology.orchestra.run.smithi120.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T01:23:20.264 INFO:teuthology.orchestra.run.smithi120.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-24T01:23:20.264 INFO:teuthology.orchestra.run.smithi120.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T01:23:20.264 INFO:teuthology.orchestra.run.smithi120.stdout: = sunit=0 swidth=0 blks 2024-03-24T01:23:20.264 INFO:teuthology.orchestra.run.smithi120.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T01:23:20.264 INFO:teuthology.orchestra.run.smithi120.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-24T01:23:20.264 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T01:23:20.264 INFO:teuthology.orchestra.run.smithi120.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T01:23:20.777 INFO:teuthology.orchestra.run.smithi120.stdout:Discarding blocks...Done. 2024-03-24T01:23:20.783 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi120.front.sepia.ceph.com -o noatime 2024-03-24T01:23:20.783 DEBUG:teuthology.orchestra.run.smithi120:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-24T01:23:20.948 DEBUG:teuthology.orchestra.run.smithi120:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-24T01:23:21.041 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-24T01:23:21.125 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-24T01:23:21.126 INFO:tasks.ceph:role: osd.7 2024-03-24T01:23:21.126 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi120.front.sepia.ceph.com 2024-03-24T01:23:21.126 DEBUG:teuthology.orchestra.run.smithi120:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-24T01:23:21.223 INFO:teuthology.orchestra.run.smithi120.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T01:23:21.223 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T01:23:21.223 INFO:teuthology.orchestra.run.smithi120.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T01:23:21.223 INFO:teuthology.orchestra.run.smithi120.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-24T01:23:21.223 INFO:teuthology.orchestra.run.smithi120.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T01:23:21.223 INFO:teuthology.orchestra.run.smithi120.stdout: = sunit=0 swidth=0 blks 2024-03-24T01:23:21.223 INFO:teuthology.orchestra.run.smithi120.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T01:23:21.223 INFO:teuthology.orchestra.run.smithi120.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-24T01:23:21.223 INFO:teuthology.orchestra.run.smithi120.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T01:23:21.224 INFO:teuthology.orchestra.run.smithi120.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T01:23:21.696 INFO:teuthology.orchestra.run.smithi120.stdout:Discarding blocks...Done. 2024-03-24T01:23:21.702 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi120.front.sepia.ceph.com -o noatime 2024-03-24T01:23:21.702 DEBUG:teuthology.orchestra.run.smithi120:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-24T01:23:21.847 DEBUG:teuthology.orchestra.run.smithi120:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-24T01:23:21.939 DEBUG:teuthology.orchestra.run.smithi120:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T01:23:22.069 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:22.069+0000 7fbe22f93640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-24T01:23:22.071 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:22.070+0000 7fbe22f93640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-24T01:23:22.071 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:22.070+0000 7fbe22f93640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-24T01:23:22.071 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:22.070+0000 7fbe22f93640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-24T01:23:22.071 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:22.070+0000 7fbe22f93640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-24T01:23:22.073 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:22.073+0000 7fbe22f93640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-24T01:23:25.241 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-24T01:23:25.341 DEBUG:teuthology.orchestra.run.smithi120:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T01:23:25.476 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:25.476+0000 7f0cbab85640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-24T01:23:25.477 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:25.476+0000 7f0cbab85640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-24T01:23:25.477 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:25.476+0000 7f0cbab85640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-24T01:23:25.477 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:25.476+0000 7f0cbab85640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-24T01:23:25.477 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:25.477+0000 7f0cbab85640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-24T01:23:25.479 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:25.479+0000 7f0cbab85640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-24T01:23:25.606 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:23:25.614 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:23:28.653 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-24T01:23:28.750 DEBUG:teuthology.orchestra.run.smithi120:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T01:23:28.888 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:28.887+0000 7fedc2414640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-24T01:23:28.888 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:28.888+0000 7fedc2414640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-24T01:23:28.889 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:28.888+0000 7fedc2414640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-24T01:23:28.889 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:28.888+0000 7fedc2414640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-24T01:23:28.889 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:28.888+0000 7fedc2414640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-24T01:23:28.890 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:28.890+0000 7fedc2414640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-24T01:23:31.837 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-24T01:23:31.932 DEBUG:teuthology.orchestra.run.smithi120:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T01:23:32.072 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:32.071+0000 7feb75211640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-24T01:23:32.072 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:32.072+0000 7feb75211640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-24T01:23:32.072 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:32.072+0000 7feb75211640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-24T01:23:32.072 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:32.072+0000 7feb75211640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-24T01:23:32.073 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:32.072+0000 7feb75211640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-24T01:23:32.074 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:32.074+0000 7feb75211640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-24T01:23:35.247 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-24T01:23:35.344 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-24T01:23:35.344 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:23:35.345 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-24T01:23:35.442 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:23:35.442 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-24T01:23:35.536 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:23:35.536 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-24T01:23:35.629 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:23:35.629 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-24T01:23:35.719 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:23:35.719 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-24T01:23:35.811 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:23:35.811 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-24T01:23:35.948 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:23:35.948 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-24T01:23:36.042 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:23:36.042 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-24T01:23:36.142 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:23:36.142 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-24T01:23:36.237 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:23:36.237 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-24T01:23:36.332 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:23:36.333 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-24T01:23:36.424 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:23:36.424 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-24T01:23:36.518 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:23:36.518 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-24T01:23:36.610 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:23:36.610 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-24T01:23:36.703 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:23:36.704 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-24T01:23:36.759 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:23:36.759 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-03-24T01:23:36.815 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:23:36.815 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-24T01:23:36.869 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:23:36.870 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-03-24T01:23:36.921 INFO:tasks.ceph:Adding keys to all mons... 2024-03-24T01:23:36.922 DEBUG:teuthology.orchestra.run.smithi090:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-24T01:23:36.929 DEBUG:teuthology.orchestra.run.smithi120:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-24T01:23:37.011 INFO:teuthology.orchestra.run.smithi120.stdout:[mgr.y] 2024-03-24T01:23:37.011 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQBzgP9l0KzyGBAA9zVfyPbngjHtBQyp+S37iQ== 2024-03-24T01:23:37.011 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.a] 2024-03-24T01:23:37.012 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQBzgP9lEcWmMBAAtUqishOFkZW8T4Hh1Cgj2A== 2024-03-24T01:23:37.012 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.c] 2024-03-24T01:23:37.012 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQB0gP9lRuOuBBAA3/3PMDba2Tlc6BqaY/499g== 2024-03-24T01:23:37.012 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.0] 2024-03-24T01:23:37.012 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQB5gP9lJ2jHGRAAPlZjS638Xdt6yVc+vT2ghw== 2024-03-24T01:23:37.012 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.1] 2024-03-24T01:23:37.012 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQB8gP9lEN/pIxAAvzVfJNpo7FP6c8KSbZdy9A== 2024-03-24T01:23:37.012 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.2] 2024-03-24T01:23:37.012 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQB/gP9lKv5mLRAAJXArmoRRVqO07jxWutfMbQ== 2024-03-24T01:23:37.012 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.3] 2024-03-24T01:23:37.012 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQCCgP9lTZ8YNxAA375ZLJHv6H2CJghmcqfe+A== 2024-03-24T01:23:37.012 INFO:teuthology.orchestra.run.smithi120.stdout:[mgr.x] 2024-03-24T01:23:37.012 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQBzgP9l8GPtJBAAgU+pArQqmyMDPwj0w+W/Dg== 2024-03-24T01:23:37.012 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.b] 2024-03-24T01:23:37.013 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQB0gP9lIPZ1FRAAmeGByOKmL/VQTpWm4Y2/Tg== 2024-03-24T01:23:37.013 INFO:teuthology.orchestra.run.smithi120.stdout:[mds.d] 2024-03-24T01:23:37.013 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQB0gP9lOtsLJhAAYUVPFa1eoDHF5MV42s8e6A== 2024-03-24T01:23:37.013 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.4] 2024-03-24T01:23:37.013 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQCKgP9ln1YuBBAAYCNFJGs3fApnhwnY/VYHhw== 2024-03-24T01:23:37.013 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.5] 2024-03-24T01:23:37.013 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQCNgP9l329zHBAA7VtZVd0XKEK5dzhyzcPFjQ== 2024-03-24T01:23:37.013 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.6] 2024-03-24T01:23:37.013 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQCQgP9lIv32NBAAZN6sDvBDKUZGB+TnYHO9jA== 2024-03-24T01:23:37.013 INFO:teuthology.orchestra.run.smithi120.stdout:[osd.7] 2024-03-24T01:23:37.013 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQCUgP9lFpBTBBAAbPF02et1cVIhSfkvJz3R3w== 2024-03-24T01:23:37.013 INFO:teuthology.orchestra.run.smithi120.stdout:[client.0] 2024-03-24T01:23:37.013 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQB0gP9lLKWXMxAAwrrRG8zLSylpIEuUgM7q8g== 2024-03-24T01:23:37.013 INFO:teuthology.orchestra.run.smithi120.stdout:[client.2] 2024-03-24T01:23:37.013 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQB1gP9lsGI+AhAAPLfzrMVwQpf8iGAfVJX1gA== 2024-03-24T01:23:37.014 INFO:teuthology.orchestra.run.smithi120.stdout:[client.1] 2024-03-24T01:23:37.014 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQB1gP9lFlgvDRAAJFhGpBldAqv7oKFr2lIXNQ== 2024-03-24T01:23:37.014 INFO:teuthology.orchestra.run.smithi120.stdout:[client.3] 2024-03-24T01:23:37.014 INFO:teuthology.orchestra.run.smithi120.stdout: key = AQB1gP9l1LA0GBAAp6dwwZf+25F64cERHm2/Ew== 2024-03-24T01:23:37.018 INFO:teuthology.orchestra.run.smithi090.stdout:[mgr.y] 2024-03-24T01:23:37.018 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQBzgP9l0KzyGBAA9zVfyPbngjHtBQyp+S37iQ== 2024-03-24T01:23:37.018 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.a] 2024-03-24T01:23:37.018 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQBzgP9lEcWmMBAAtUqishOFkZW8T4Hh1Cgj2A== 2024-03-24T01:23:37.018 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.c] 2024-03-24T01:23:37.018 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQB0gP9lRuOuBBAA3/3PMDba2Tlc6BqaY/499g== 2024-03-24T01:23:37.018 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.0] 2024-03-24T01:23:37.018 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQB5gP9lJ2jHGRAAPlZjS638Xdt6yVc+vT2ghw== 2024-03-24T01:23:37.018 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.1] 2024-03-24T01:23:37.018 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQB8gP9lEN/pIxAAvzVfJNpo7FP6c8KSbZdy9A== 2024-03-24T01:23:37.018 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.2] 2024-03-24T01:23:37.019 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQB/gP9lKv5mLRAAJXArmoRRVqO07jxWutfMbQ== 2024-03-24T01:23:37.019 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.3] 2024-03-24T01:23:37.019 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQCCgP9lTZ8YNxAA375ZLJHv6H2CJghmcqfe+A== 2024-03-24T01:23:37.019 INFO:teuthology.orchestra.run.smithi090.stdout:[mgr.x] 2024-03-24T01:23:37.019 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQBzgP9l8GPtJBAAgU+pArQqmyMDPwj0w+W/Dg== 2024-03-24T01:23:37.019 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.b] 2024-03-24T01:23:37.019 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQB0gP9lIPZ1FRAAmeGByOKmL/VQTpWm4Y2/Tg== 2024-03-24T01:23:37.019 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.d] 2024-03-24T01:23:37.019 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQB0gP9lOtsLJhAAYUVPFa1eoDHF5MV42s8e6A== 2024-03-24T01:23:37.019 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.4] 2024-03-24T01:23:37.019 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQCKgP9ln1YuBBAAYCNFJGs3fApnhwnY/VYHhw== 2024-03-24T01:23:37.019 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.5] 2024-03-24T01:23:37.019 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQCNgP9l329zHBAA7VtZVd0XKEK5dzhyzcPFjQ== 2024-03-24T01:23:37.019 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.6] 2024-03-24T01:23:37.020 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQCQgP9lIv32NBAAZN6sDvBDKUZGB+TnYHO9jA== 2024-03-24T01:23:37.020 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.7] 2024-03-24T01:23:37.020 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQCUgP9lFpBTBBAAbPF02et1cVIhSfkvJz3R3w== 2024-03-24T01:23:37.020 INFO:teuthology.orchestra.run.smithi090.stdout:[client.0] 2024-03-24T01:23:37.020 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQB0gP9lLKWXMxAAwrrRG8zLSylpIEuUgM7q8g== 2024-03-24T01:23:37.020 INFO:teuthology.orchestra.run.smithi090.stdout:[client.2] 2024-03-24T01:23:37.020 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQB1gP9lsGI+AhAAPLfzrMVwQpf8iGAfVJX1gA== 2024-03-24T01:23:37.020 INFO:teuthology.orchestra.run.smithi090.stdout:[client.1] 2024-03-24T01:23:37.020 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQB1gP9lFlgvDRAAJFhGpBldAqv7oKFr2lIXNQ== 2024-03-24T01:23:37.020 INFO:teuthology.orchestra.run.smithi090.stdout:[client.3] 2024-03-24T01:23:37.020 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQB1gP9l1LA0GBAAp6dwwZf+25F64cERHm2/Ew== 2024-03-24T01:23:37.022 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-24T01:23:37.070 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-24T01:23:37.215 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T01:23:37.222 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T01:23:37.367 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T01:23:37.374 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T01:23:37.520 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T01:23:37.526 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T01:23:37.670 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T01:23:37.677 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T01:23:37.821 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T01:23:37.828 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T01:23:37.969 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T01:23:37.974 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T01:23:38.110 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-24T01:23:38.115 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-24T01:23:38.254 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T01:23:38.261 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T01:23:38.406 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T01:23:38.411 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T01:23:38.550 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T01:23:38.557 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T01:23:38.698 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T01:23:38.705 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T01:23:38.850 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T01:23:38.857 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T01:23:38.995 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T01:23:39.000 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T01:23:39.136 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-24T01:23:39.144 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-24T01:23:39.282 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-24T01:23:39.289 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-24T01:23:39.431 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-24T01:23:39.437 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-24T01:23:39.581 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-24T01:23:39.588 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-24T01:23:39.727 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-24T01:23:39.728 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-24T01:23:39.821 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-24T01:23:39.967 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-24T01:23:40.049 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-24T01:23:40.147 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-24T01:23:40.295 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-24T01:23:40.386 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-24T01:23:40.473 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-24T01:23:40.628 DEBUG:teuthology.orchestra.run.smithi120:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-24T01:23:40.710 DEBUG:teuthology.orchestra.run.smithi090:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-24T01:23:40.718 DEBUG:teuthology.orchestra.run.smithi120:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-24T01:23:40.769 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-24T01:23:40.770 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-24T01:23:40.770 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-24T01:23:40.775 INFO:tasks.ceph.mon.a:Started 2024-03-24T01:23:40.775 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-24T01:23:40.775 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-24T01:23:40.782 INFO:tasks.ceph.mon.b:Started 2024-03-24T01:23:40.782 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-24T01:23:40.782 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-24T01:23:40.789 INFO:tasks.ceph.mon.c:Started 2024-03-24T01:23:40.789 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-24T01:23:40.789 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-24T01:23:40.789 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-24T01:23:40.793 INFO:tasks.ceph.mgr.y:Started 2024-03-24T01:23:40.794 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-24T01:23:40.794 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-24T01:23:40.798 INFO:tasks.ceph.mgr.x:Started 2024-03-24T01:23:40.798 INFO:tasks.ceph:Setting crush tunables to default 2024-03-24T01:23:40.798 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd crush tunables default 2024-03-24T01:23:40.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:40.980+0000 7f0d53a00640 1 Processor -- start 2024-03-24T01:23:40.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:40.980+0000 7f0d53a00640 1 -- start start 2024-03-24T01:23:40.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:40.981+0000 7f0d53a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d4c0d3e70 0x7f0d4c0d1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:40.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:40.981+0000 7f0d53a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0d4c0d4250 0x7f0d4c15d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:40.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:40.981+0000 7f0d53a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d4c0d2740 con 0x7f0d4c15dd00 2024-03-24T01:23:40.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:40.981+0000 7f0d53a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d4c0d2a20 con 0x7f0d4c0d4250 2024-03-24T01:23:40.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:40.981+0000 7f0d53a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4c0d2d00 con 0x7f0d4c0d3e70 2024-03-24T01:23:40.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:40.982+0000 7f0d4be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0d4c0d4250 0x7f0d4c15d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:40.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:40.982+0000 7f0d51000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d4c0d3e70 0x7f0d4c0d1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:40.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:40.982+0000 7f0d4be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0d4c0d4250 0x7f0d4c15d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58786/0 (socket says 172.21.15.90:58786) 2024-03-24T01:23:40.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:40.982+0000 7f0d4be00640 1 -- 172.21.15.90:0/4293018972 learned_addr learned my addr 172.21.15.90:0/4293018972 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:23:40.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:40.982+0000 7f0d4be00640 1 -- 172.21.15.90:0/4293018972 >> v2:172.21.15.90:3300/0 conn(0x7f0d4c0d4250 msgr2=0x7f0d4c15d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-24T01:23:40.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:40.982+0000 7f0d4be00640 1 -- 172.21.15.90:0/4293018972 >> v2:172.21.15.90:3300/0 conn(0x7f0d4c0d4250 msgr2=0x7f0d4c15d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T01:23:40.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:40.982+0000 7f0d4be00640 1 --2- 172.21.15.90:0/4293018972 >> v2:172.21.15.90:3300/0 conn(0x7f0d4c0d4250 0x7f0d4c15d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T01:23:40.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:40.982+0000 7f0d4be00640 1 --2- 172.21.15.90:0/4293018972 >> v2:172.21.15.90:3300/0 conn(0x7f0d4c0d4250 0x7f0d4c15d7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:23:40.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:40.982+0000 7f0d51000640 1 -- 172.21.15.90:0/4293018972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d4c0d3e70 msgr2=0x7f0d4c0d1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T01:23:40.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:40.982+0000 7f0d51000640 1 -- 172.21.15.90:0/4293018972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d4c0d3e70 msgr2=0x7f0d4c0d1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T01:23:40.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:40.982+0000 7f0d51000640 1 --2- 172.21.15.90:0/4293018972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d4c0d3e70 0x7f0d4c0d1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T01:23:40.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:40.982+0000 7f0d51000640 1 --2- 172.21.15.90:0/4293018972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d4c0d3e70 0x7f0d4c0d1dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:23:41.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:41.183+0000 7f0d4be00640 1 --2- 172.21.15.90:0/4293018972 >> v2:172.21.15.90:3300/0 conn(0x7f0d4c0d4250 0x7f0d4c15d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:41.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:41.183+0000 7f0d51000640 1 --2- 172.21.15.90:0/4293018972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d4c0d3e70 0x7f0d4c0d1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:41.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:41.184+0000 7f0d4be00640 1 -- 172.21.15.90:0/4293018972 >> v2:172.21.15.90:3300/0 conn(0x7f0d4c0d4250 msgr2=0x7f0d4c15d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T01:23:41.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:41.184+0000 7f0d4be00640 1 -- 172.21.15.90:0/4293018972 >> v2:172.21.15.90:3300/0 conn(0x7f0d4c0d4250 msgr2=0x7f0d4c15d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T01:23:41.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:41.184+0000 7f0d4be00640 1 --2- 172.21.15.90:0/4293018972 >> v2:172.21.15.90:3300/0 conn(0x7f0d4c0d4250 0x7f0d4c15d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T01:23:41.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:41.184+0000 7f0d4be00640 1 --2- 172.21.15.90:0/4293018972 >> v2:172.21.15.90:3300/0 conn(0x7f0d4c0d4250 0x7f0d4c15d7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:23:41.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:41.184+0000 7f0d51000640 1 -- 172.21.15.90:0/4293018972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d4c0d3e70 msgr2=0x7f0d4c0d1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-24T01:23:41.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:41.184+0000 7f0d51000640 1 -- 172.21.15.90:0/4293018972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d4c0d3e70 msgr2=0x7f0d4c0d1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T01:23:41.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:41.184+0000 7f0d51000640 1 --2- 172.21.15.90:0/4293018972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d4c0d3e70 0x7f0d4c0d1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T01:23:41.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:41.184+0000 7f0d51000640 1 --2- 172.21.15.90:0/4293018972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d4c0d3e70 0x7f0d4c0d1dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:23:41.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:41.585+0000 7f0d4be00640 1 --2- 172.21.15.90:0/4293018972 >> v2:172.21.15.90:3300/0 conn(0x7f0d4c0d4250 0x7f0d4c15d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:41.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:41.585+0000 7f0d51000640 1 --2- 172.21.15.90:0/4293018972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d4c0d3e70 0x7f0d4c0d1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:41.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:41.586+0000 7f0d4be00640 1 -- 172.21.15.90:0/4293018972 >> v2:172.21.15.90:3300/0 conn(0x7f0d4c0d4250 msgr2=0x7f0d4c15d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-24T01:23:41.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:41.586+0000 7f0d4be00640 1 -- 172.21.15.90:0/4293018972 >> v2:172.21.15.90:3300/0 conn(0x7f0d4c0d4250 msgr2=0x7f0d4c15d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T01:23:41.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:41.586+0000 7f0d4be00640 1 --2- 172.21.15.90:0/4293018972 >> v2:172.21.15.90:3300/0 conn(0x7f0d4c0d4250 0x7f0d4c15d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T01:23:41.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:41.586+0000 7f0d4be00640 1 --2- 172.21.15.90:0/4293018972 >> v2:172.21.15.90:3300/0 conn(0x7f0d4c0d4250 0x7f0d4c15d7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:23:41.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:41.586+0000 7f0d51000640 1 -- 172.21.15.90:0/4293018972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d4c0d3e70 msgr2=0x7f0d4c0d1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T01:23:41.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:41.586+0000 7f0d51000640 1 -- 172.21.15.90:0/4293018972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d4c0d3e70 msgr2=0x7f0d4c0d1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T01:23:41.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:41.586+0000 7f0d51000640 1 --2- 172.21.15.90:0/4293018972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d4c0d3e70 0x7f0d4c0d1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T01:23:41.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:41.586+0000 7f0d51000640 1 --2- 172.21.15.90:0/4293018972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d4c0d3e70 0x7f0d4c0d1dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:23:42.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:42.387+0000 7f0d4be00640 1 --2- 172.21.15.90:0/4293018972 >> v2:172.21.15.90:3300/0 conn(0x7f0d4c0d4250 0x7f0d4c15d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:42.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:42.388+0000 7f0d51000640 1 --2- 172.21.15.90:0/4293018972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d4c0d3e70 0x7f0d4c0d1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:42.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:42.388+0000 7f0d4be00640 1 -- 172.21.15.90:0/4293018972 >> v2:172.21.15.90:3300/0 conn(0x7f0d4c0d4250 msgr2=0x7f0d4c15d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T01:23:42.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:42.388+0000 7f0d4be00640 1 -- 172.21.15.90:0/4293018972 >> v2:172.21.15.90:3300/0 conn(0x7f0d4c0d4250 msgr2=0x7f0d4c15d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T01:23:42.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:42.388+0000 7f0d4be00640 1 --2- 172.21.15.90:0/4293018972 >> v2:172.21.15.90:3300/0 conn(0x7f0d4c0d4250 0x7f0d4c15d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T01:23:42.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:42.388+0000 7f0d4be00640 1 --2- 172.21.15.90:0/4293018972 >> v2:172.21.15.90:3300/0 conn(0x7f0d4c0d4250 0x7f0d4c15d7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:23:42.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:42.388+0000 7f0d51000640 1 -- 172.21.15.90:0/4293018972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d4c0d3e70 msgr2=0x7f0d4c0d1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-24T01:23:42.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:42.388+0000 7f0d51000640 1 -- 172.21.15.90:0/4293018972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d4c0d3e70 msgr2=0x7f0d4c0d1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T01:23:42.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:42.388+0000 7f0d51000640 1 --2- 172.21.15.90:0/4293018972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d4c0d3e70 0x7f0d4c0d1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T01:23:42.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:42.388+0000 7f0d51000640 1 --2- 172.21.15.90:0/4293018972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d4c0d3e70 0x7f0d4c0d1dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:23:43.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:43.981+0000 7f0d4a000640 1 -- 172.21.15.90:0/4293018972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d4c0d3e70 msgr2=0x7f0d4c0d1dd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:23:43.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:43.981+0000 7f0d4a000640 1 --2- 172.21.15.90:0/4293018972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d4c0d3e70 0x7f0d4c0d1dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:43.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:43.981+0000 7f0d4a000640 1 -- 172.21.15.90:0/4293018972 >> v2:172.21.15.90:3300/0 conn(0x7f0d4c0d4250 msgr2=0x7f0d4c15d7c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:23:43.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:43.981+0000 7f0d4a000640 1 --2- 172.21.15.90:0/4293018972 >> v2:172.21.15.90:3300/0 conn(0x7f0d4c0d4250 0x7f0d4c15d7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:43.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:43.981+0000 7f0d4a000640 1 -- 172.21.15.90:0/4293018972 >> v1:172.21.15.120:6789/0 conn(0x7f0d4c15dd00 legacy=0x7f0d4c0b56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:43.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:43.982+0000 7f0d4a000640 1 --2- 172.21.15.90:0/4293018972 >> v2:172.21.15.120:3300/0 conn(0x7f0d2c007350 0x7f0d2c009810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:43.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:43.982+0000 7f0d4a000640 1 -- 172.21.15.90:0/4293018972 --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d4c0d2fe0 con 0x7f0d2c0036c0 2024-03-24T01:23:43.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:43.982+0000 7f0d4a000640 1 -- 172.21.15.90:0/4293018972 --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d4c0d2a20 con 0x7f0d2c007350 2024-03-24T01:23:43.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:43.982+0000 7f0d4be00640 1 --2- 172.21.15.90:0/4293018972 >> v2:172.21.15.120:3300/0 conn(0x7f0d2c007350 0x7f0d2c009810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:43.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:43.983+0000 7f0d4be00640 1 -- 172.21.15.90:0/4293018972 >> v2:172.21.15.120:3300/0 conn(0x7f0d2c007350 msgr2=0x7f0d2c009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T01:23:43.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:43.983+0000 7f0d4be00640 1 -- 172.21.15.90:0/4293018972 >> v2:172.21.15.120:3300/0 conn(0x7f0d2c007350 msgr2=0x7f0d2c009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T01:23:43.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:43.983+0000 7f0d4be00640 1 --2- 172.21.15.90:0/4293018972 >> v2:172.21.15.120:3300/0 conn(0x7f0d2c007350 0x7f0d2c009810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T01:23:43.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:43.983+0000 7f0d4be00640 1 --2- 172.21.15.90:0/4293018972 >> v2:172.21.15.120:3300/0 conn(0x7f0d2c007350 0x7f0d2c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:23:44.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:44.184+0000 7f0d4be00640 1 --2- 172.21.15.90:0/4293018972 >> v2:172.21.15.120:3300/0 conn(0x7f0d2c007350 0x7f0d2c009810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:44.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:44.185+0000 7f0d4be00640 1 -- 172.21.15.90:0/4293018972 >> v2:172.21.15.120:3300/0 conn(0x7f0d2c007350 msgr2=0x7f0d2c009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T01:23:44.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:44.185+0000 7f0d4be00640 1 -- 172.21.15.90:0/4293018972 >> v2:172.21.15.120:3300/0 conn(0x7f0d2c007350 msgr2=0x7f0d2c009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T01:23:44.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:44.185+0000 7f0d4be00640 1 --2- 172.21.15.90:0/4293018972 >> v2:172.21.15.120:3300/0 conn(0x7f0d2c007350 0x7f0d2c009810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T01:23:44.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:44.185+0000 7f0d4be00640 1 --2- 172.21.15.90:0/4293018972 >> v2:172.21.15.120:3300/0 conn(0x7f0d2c007350 0x7f0d2c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:23:44.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:44.586+0000 7f0d4be00640 1 --2- 172.21.15.90:0/4293018972 >> v2:172.21.15.120:3300/0 conn(0x7f0d2c007350 0x7f0d2c009810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:44.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:44.587+0000 7f0d4be00640 1 -- 172.21.15.90:0/4293018972 >> v2:172.21.15.120:3300/0 conn(0x7f0d2c007350 msgr2=0x7f0d2c009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T01:23:44.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:44.587+0000 7f0d4be00640 1 -- 172.21.15.90:0/4293018972 >> v2:172.21.15.120:3300/0 conn(0x7f0d2c007350 msgr2=0x7f0d2c009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T01:23:44.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:44.587+0000 7f0d4be00640 1 --2- 172.21.15.90:0/4293018972 >> v2:172.21.15.120:3300/0 conn(0x7f0d2c007350 0x7f0d2c009810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T01:23:44.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:44.587+0000 7f0d4be00640 1 --2- 172.21.15.90:0/4293018972 >> v2:172.21.15.120:3300/0 conn(0x7f0d2c007350 0x7f0d2c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:23:45.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:45.388+0000 7f0d4be00640 1 --2- 172.21.15.90:0/4293018972 >> v2:172.21.15.120:3300/0 conn(0x7f0d2c007350 0x7f0d2c009810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:45.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:45.389+0000 7f0d4be00640 1 -- 172.21.15.90:0/4293018972 >> v2:172.21.15.120:3300/0 conn(0x7f0d2c007350 msgr2=0x7f0d2c009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T01:23:45.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:45.389+0000 7f0d4be00640 1 -- 172.21.15.90:0/4293018972 >> v2:172.21.15.120:3300/0 conn(0x7f0d2c007350 msgr2=0x7f0d2c009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T01:23:45.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:45.389+0000 7f0d4be00640 1 --2- 172.21.15.90:0/4293018972 >> v2:172.21.15.120:3300/0 conn(0x7f0d2c007350 0x7f0d2c009810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T01:23:45.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:45.389+0000 7f0d4be00640 1 --2- 172.21.15.90:0/4293018972 >> v2:172.21.15.120:3300/0 conn(0x7f0d2c007350 0x7f0d2c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:23:46.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.248+0000 7f0d4b400640 1 -- 172.21.15.90:0/4293018972 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 358721143 0 0) 0x7f0d4c0d2fe0 con 0x7f0d2c0036c0 2024-03-24T01:23:46.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.248+0000 7f0d4b400640 1 -- 172.21.15.90:0/4293018972 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d34003520 con 0x7f0d2c0036c0 2024-03-24T01:23:46.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.249+0000 7f0d4b400640 1 -- 172.21.15.90:0/4293018972 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2462210852 0 0) 0x7f0d34003520 con 0x7f0d2c0036c0 2024-03-24T01:23:46.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.249+0000 7f0d4b400640 1 -- 172.21.15.90:0/4293018972 >> v2:172.21.15.120:3300/0 conn(0x7f0d2c007350 msgr2=0x7f0d2c009810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:23:46.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.249+0000 7f0d4b400640 1 --2- 172.21.15.90:0/4293018972 >> v2:172.21.15.120:3300/0 conn(0x7f0d2c007350 0x7f0d2c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:46.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.249+0000 7f0d4b400640 1 -- 172.21.15.90:0/4293018972 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d2c00afd0 con 0x7f0d2c0036c0 2024-03-24T01:23:46.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.249+0000 7f0d4b400640 1 -- 172.21.15.90:0/4293018972 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f0d4000d580 con 0x7f0d2c0036c0 2024-03-24T01:23:46.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.249+0000 7f0d4b400640 1 -- 172.21.15.90:0/4293018972 >> v1:172.21.15.90:6789/0 conn(0x7f0d2c0036c0 legacy=0x7f0d2c005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:46.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.249+0000 7f0d51a00640 1 -- 172.21.15.90:0/4293018972 reap_dead start 2024-03-24T01:23:46.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.249+0000 7f0d4b400640 1 --2- 172.21.15.90:0/4293018972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d34003a00 0x7f0d34003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:46.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.249+0000 7f0d4b400640 1 --2- 172.21.15.90:0/4293018972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d34004800 0x7f0d34004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:46.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.249+0000 7f0d4b400640 1 --2- 172.21.15.90:0/4293018972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d340053c0 0x7f0d3400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:46.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.250+0000 7f0d4b400640 1 -- 172.21.15.90:0/4293018972 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4c0d2a20 con 0x7f0d340053c0 2024-03-24T01:23:46.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.250+0000 7f0d4b400640 1 -- 172.21.15.90:0/4293018972 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3400c280 con 0x7f0d34003a00 2024-03-24T01:23:46.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.250+0000 7f0d4b400640 1 -- 172.21.15.90:0/4293018972 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3400c480 con 0x7f0d34004800 2024-03-24T01:23:46.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.250+0000 7f0d4be00640 1 --2- 172.21.15.90:0/4293018972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d34004800 0x7f0d34004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:46.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.250+0000 7f0d51a00640 1 --2- 172.21.15.90:0/4293018972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d340053c0 0x7f0d3400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:46.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.250+0000 7f0d51000640 1 --2- 172.21.15.90:0/4293018972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d34003a00 0x7f0d34003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:46.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.251+0000 7f0d51a00640 1 -- 172.21.15.90:0/4293018972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d34004800 msgr2=0x7f0d34004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:23:46.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.251+0000 7f0d51a00640 1 --2- 172.21.15.90:0/4293018972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d34004800 0x7f0d34004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:46.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.251+0000 7f0d51a00640 1 -- 172.21.15.90:0/4293018972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d34003a00 msgr2=0x7f0d34003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:23:46.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.251+0000 7f0d51a00640 1 --2- 172.21.15.90:0/4293018972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d34003a00 0x7f0d34003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:46.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.251+0000 7f0d51a00640 1 -- 172.21.15.90:0/4293018972 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d34003520 con 0x7f0d340053c0 2024-03-24T01:23:46.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.253+0000 7f0d51a00640 1 --2- 172.21.15.90:0/4293018972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d340053c0 0x7f0d3400b9c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c15e0e0 tx=0x7f0d34005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a3e11cb590ca3e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:23:46.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.253+0000 7f0d4b400640 1 -- 172.21.15.90:0/4293018972 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d440018e0 con 0x7f0d340053c0 2024-03-24T01:23:46.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.254+0000 7f0d4b400640 1 -- 172.21.15.90:0/4293018972 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d44005d40 con 0x7f0d340053c0 2024-03-24T01:23:46.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.254+0000 7f0d4b400640 1 -- 172.21.15.90:0/4293018972 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d44007290 con 0x7f0d340053c0 2024-03-24T01:23:46.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.255+0000 7f0d53a00640 1 -- 172.21.15.90:0/4293018972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d340053c0 msgr2=0x7f0d3400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:46.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.255+0000 7f0d53a00640 1 --2- 172.21.15.90:0/4293018972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d340053c0 0x7f0d3400b9c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c15e0e0 tx=0x7f0d34005830 comp rx=0 tx=0).stop 2024-03-24T01:23:46.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.255+0000 7f0d53a00640 1 -- 172.21.15.90:0/4293018972 shutdown_connections 2024-03-24T01:23:46.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.255+0000 7f0d53a00640 1 --2- 172.21.15.90:0/4293018972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d340053c0 0x7f0d3400b9c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:46.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.255+0000 7f0d53a00640 1 --2- 172.21.15.90:0/4293018972 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d34004800 0x7f0d34004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:46.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.255+0000 7f0d53a00640 1 --2- 172.21.15.90:0/4293018972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d34003a00 0x7f0d34003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:46.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.255+0000 7f0d53a00640 1 -- 172.21.15.90:0/4293018972 >> 172.21.15.90:0/4293018972 conn(0x7f0d4c072c10 msgr2=0x7f0d4c073020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:23:46.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.255+0000 7f0d53a00640 1 -- 172.21.15.90:0/4293018972 shutdown_connections 2024-03-24T01:23:46.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.255+0000 7f0d53a00640 1 -- 172.21.15.90:0/4293018972 wait complete. 2024-03-24T01:23:46.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.261+0000 7f0d53a00640 1 Processor -- start 2024-03-24T01:23:46.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.261+0000 7f0d53a00640 1 -- start start 2024-03-24T01:23:46.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.264+0000 7f0d53a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d34003a00 0x7f0d4c150260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:46.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.264+0000 7f0d53a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d34004800 0x7f0d4c153130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:46.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.264+0000 7f0d53a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d340053c0 0x7f0d4c0cc270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:46.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.264+0000 7f0d53a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4c0d37c0 con 0x7f0d340053c0 2024-03-24T01:23:46.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.264+0000 7f0d53a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4c15e170 con 0x7f0d34004800 2024-03-24T01:23:46.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.264+0000 7f0d53a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4c0d3940 con 0x7f0d34003a00 2024-03-24T01:23:46.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.264+0000 7f0d51a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d340053c0 0x7f0d4c0cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:46.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.264+0000 7f0d4be00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d34004800 0x7f0d4c153130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:46.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.264+0000 7f0d51000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d34003a00 0x7f0d4c150260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:46.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.264+0000 7f0d51000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d34003a00 0x7f0d4c150260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.90:46220/0 (socket says 172.21.15.90:46220) 2024-03-24T01:23:46.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.264+0000 7f0d51000640 1 -- 172.21.15.90:0/743447792 learned_addr learned my addr 172.21.15.90:0/743447792 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:23:46.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.265+0000 7f0d51a00640 1 -- 172.21.15.90:0/743447792 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d34003a00 msgr2=0x7f0d4c150260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:46.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.265+0000 7f0d51a00640 1 --2- 172.21.15.90:0/743447792 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d34003a00 0x7f0d4c150260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:46.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.265+0000 7f0d51a00640 1 -- 172.21.15.90:0/743447792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d34004800 msgr2=0x7f0d4c153130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:46.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.265+0000 7f0d51a00640 1 --2- 172.21.15.90:0/743447792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d34004800 0x7f0d4c153130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:46.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.265+0000 7f0d51a00640 1 -- 172.21.15.90:0/743447792 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d4c0cb1d0 con 0x7f0d340053c0 2024-03-24T01:23:46.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.265+0000 7f0d51000640 1 --2- 172.21.15.90:0/743447792 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d34003a00 0x7f0d4c150260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T01:23:46.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.266+0000 7f0d51a00640 1 --2- 172.21.15.90:0/743447792 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d340053c0 0x7f0d4c0cc270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0d44007fd0 tx=0x7f0d440014d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:23:46.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.266+0000 7f0d48c00640 1 -- 172.21.15.90:0/743447792 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d44038040 con 0x7f0d340053c0 2024-03-24T01:23:46.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.266+0000 7f0d48c00640 1 -- 172.21.15.90:0/743447792 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d44005b40 con 0x7f0d340053c0 2024-03-24T01:23:46.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.266+0000 7f0d53a00640 1 -- 172.21.15.90:0/743447792 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d4c0cb4b0 con 0x7f0d340053c0 2024-03-24T01:23:46.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.266+0000 7f0d48c00640 1 -- 172.21.15.90:0/743447792 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d44003b90 con 0x7f0d340053c0 2024-03-24T01:23:46.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.266+0000 7f0d53a00640 1 -- 172.21.15.90:0/743447792 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d4c0cba70 con 0x7f0d340053c0 2024-03-24T01:23:46.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.267+0000 7f0d48c00640 1 -- 172.21.15.90:0/743447792 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f0d44045020 con 0x7f0d340053c0 2024-03-24T01:23:46.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.267+0000 7f0d48c00640 1 -- 172.21.15.90:0/743447792 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0d44033b50 con 0x7f0d340053c0 2024-03-24T01:23:46.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.268+0000 7f0d53a00640 1 -- 172.21.15.90:0/743447792 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d4c04f400 con 0x7f0d340053c0 2024-03-24T01:23:46.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.271+0000 7f0d48c00640 1 -- 172.21.15.90:0/743447792 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f0d44007990 con 0x7f0d340053c0 2024-03-24T01:23:46.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.345+0000 7f0d53a00640 1 -- 172.21.15.90:0/743447792 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f0d4c15a770 con 0x7f0d340053c0 2024-03-24T01:23:46.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.350+0000 7f0d48c00640 1 -- 172.21.15.90:0/743447792 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f0d44005d10 con 0x7f0d340053c0 2024-03-24T01:23:46.350 INFO:teuthology.orchestra.run.smithi090.stderr:adjusted tunables profile to default 2024-03-24T01:23:46.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.351+0000 7f0d53a00640 1 -- 172.21.15.90:0/743447792 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d340053c0 msgr2=0x7f0d4c0cc270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:46.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.351+0000 7f0d53a00640 1 --2- 172.21.15.90:0/743447792 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d340053c0 0x7f0d4c0cc270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0d44007fd0 tx=0x7f0d440014d0 comp rx=0 tx=0).stop 2024-03-24T01:23:46.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.351+0000 7f0d53a00640 1 -- 172.21.15.90:0/743447792 shutdown_connections 2024-03-24T01:23:46.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.351+0000 7f0d53a00640 1 --2- 172.21.15.90:0/743447792 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d340053c0 0x7f0d4c0cc270 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:46.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.351+0000 7f0d53a00640 1 --2- 172.21.15.90:0/743447792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d34004800 0x7f0d4c153130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:46.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.351+0000 7f0d53a00640 1 --2- 172.21.15.90:0/743447792 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d34003a00 0x7f0d4c150260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:46.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.351+0000 7f0d53a00640 1 -- 172.21.15.90:0/743447792 >> 172.21.15.90:0/743447792 conn(0x7f0d4c072c10 msgr2=0x7f0d4c0b6e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:23:46.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.351+0000 7f0d53a00640 1 -- 172.21.15.90:0/743447792 shutdown_connections 2024-03-24T01:23:46.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:46.351+0000 7f0d53a00640 1 -- 172.21.15.90:0/743447792 wait complete. 2024-03-24T01:23:46.370 INFO:tasks.ceph:check_enable_crimson: False 2024-03-24T01:23:46.370 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-24T01:23:46.370 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:23:46.370 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-24T01:23:46.461 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:23:46.461 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-24T01:23:46.552 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:23:46.552 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-24T01:23:46.636 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:23:46.636 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-24T01:23:46.722 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:23:46.722 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-24T01:23:46.812 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:23:46.813 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-24T01:23:46.901 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:23:46.901 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-24T01:23:46.930 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-24T01:23:46.929+0000 7f1993fdb040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T01:23:46.986 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:23:46.986 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-24T01:23:46.999 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-24T01:23:46.999+0000 7f1993fdb040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T01:23:47.090 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new aeced55e-300f-4d89-89ec-f59579c3f96c 0 2024-03-24T01:23:47.328 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.327+0000 7f1d74600640 1 Processor -- start 2024-03-24T01:23:47.328 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.328+0000 7f1d74600640 1 -- start start 2024-03-24T01:23:47.330 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.329+0000 7f1d74600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d6c0d3e70 0x7f1d6c0d1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:47.330 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.329+0000 7f1d74600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1d6c0d4250 0x7f1d6c15d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:47.330 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.329+0000 7f1d74600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1d6c15dd00 0x7f1d6c0b56f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:47.330 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.329+0000 7f1d74600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d6c0d2740 con 0x7f1d6c15dd00 2024-03-24T01:23:47.330 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.329+0000 7f1d74600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d6c0d2a20 con 0x7f1d6c0d4250 2024-03-24T01:23:47.330 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.329+0000 7f1d74600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6c0d2d00 con 0x7f1d6c0d3e70 2024-03-24T01:23:47.330 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.329+0000 7f1d71200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1d6c0d4250 0x7f1d6c15d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:47.330 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.329+0000 7f1d71c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d6c0d3e70 0x7f1d6c0d1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:47.330 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.329+0000 7f1d71200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1d6c0d4250 0x7f1d6c15d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:40022/0 (socket says 172.21.15.120:40022) 2024-03-24T01:23:47.330 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.329+0000 7f1d71200640 1 -- 172.21.15.120:0/3048008535 learned_addr learned my addr 172.21.15.120:0/3048008535 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-24T01:23:47.330 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.329+0000 7f1d71c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d6c0d3e70 0x7f1d6c0d1dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:55124/0 (socket says 172.21.15.120:55124) 2024-03-24T01:23:47.331 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.330+0000 7f1d72600640 1 --2- 172.21.15.120:0/3048008535 >> v2:172.21.15.90:3300/0 conn(0x7f1d6c15dd00 0x7f1d6c0b56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:47.331 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.330+0000 7f1d71c00640 1 -- 172.21.15.120:0/3048008535 >> v2:172.21.15.120:3300/0 conn(0x7f1d6c0d4250 msgr2=0x7f1d6c15d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:23:47.331 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.330+0000 7f1d71c00640 1 --2- 172.21.15.120:0/3048008535 >> v2:172.21.15.120:3300/0 conn(0x7f1d6c0d4250 0x7f1d6c15d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:47.332 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.330+0000 7f1d71c00640 1 -- 172.21.15.120:0/3048008535 >> v2:172.21.15.90:3300/0 conn(0x7f1d6c15dd00 msgr2=0x7f1d6c0b56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:23:47.332 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.330+0000 7f1d71c00640 1 --2- 172.21.15.120:0/3048008535 >> v2:172.21.15.90:3300/0 conn(0x7f1d6c15dd00 0x7f1d6c0b56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:47.332 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.330+0000 7f1d71c00640 1 -- 172.21.15.120:0/3048008535 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d6c0d2fe0 con 0x7f1d6c0d3e70 2024-03-24T01:23:47.332 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.330+0000 7f1d71200640 1 --2- 172.21.15.120:0/3048008535 >> v2:172.21.15.120:3300/0 conn(0x7f1d6c0d4250 0x7f1d6c15d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T01:23:47.332 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.331+0000 7f1d71c00640 1 --2- 172.21.15.120:0/3048008535 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d6c0d3e70 0x7f1d6c0d1dd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c0b6830 tx=0x7f1d6002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b04d00f0ddc05ab2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:23:47.333 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.331+0000 7f1d6be00640 1 -- 172.21.15.120:0/3048008535 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d600315a0 con 0x7f1d6c0d3e70 2024-03-24T01:23:47.333 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.332+0000 7f1d6be00640 1 -- 172.21.15.120:0/3048008535 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d60037070 con 0x7f1d6c0d3e70 2024-03-24T01:23:47.333 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.332+0000 7f1d6be00640 1 -- 172.21.15.120:0/3048008535 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d60031960 con 0x7f1d6c0d3e70 2024-03-24T01:23:47.333 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.332+0000 7f1d74600640 1 -- 172.21.15.120:0/3048008535 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d6c0d3e70 msgr2=0x7f1d6c0d1dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:47.333 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.332+0000 7f1d74600640 1 --2- 172.21.15.120:0/3048008535 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d6c0d3e70 0x7f1d6c0d1dd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c0b6830 tx=0x7f1d6002b840 comp rx=0 tx=0).stop 2024-03-24T01:23:47.333 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.332+0000 7f1d74600640 1 -- 172.21.15.120:0/3048008535 shutdown_connections 2024-03-24T01:23:47.333 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.332+0000 7f1d74600640 1 --2- 172.21.15.120:0/3048008535 >> v2:172.21.15.90:3300/0 conn(0x7f1d6c15dd00 0x7f1d6c0b56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:47.333 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.332+0000 7f1d74600640 1 --2- 172.21.15.120:0/3048008535 >> v2:172.21.15.120:3300/0 conn(0x7f1d6c0d4250 0x7f1d6c15d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:47.333 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.332+0000 7f1d74600640 1 --2- 172.21.15.120:0/3048008535 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d6c0d3e70 0x7f1d6c0d1dd0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:47.334 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.332+0000 7f1d74600640 1 -- 172.21.15.120:0/3048008535 >> 172.21.15.120:0/3048008535 conn(0x7f1d6c072c10 msgr2=0x7f1d6c073020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:23:47.334 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.332+0000 7f1d74600640 1 -- 172.21.15.120:0/3048008535 shutdown_connections 2024-03-24T01:23:47.334 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.333+0000 7f1d74600640 1 -- 172.21.15.120:0/3048008535 wait complete. 2024-03-24T01:23:47.335 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.334+0000 7f1d74600640 1 Processor -- start 2024-03-24T01:23:47.335 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.335+0000 7f1d74600640 1 -- start start 2024-03-24T01:23:47.338 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.336+0000 7f1d74600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d6c0d3e70 0x7f1d6c1f68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:47.338 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.336+0000 7f1d74600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d6c0d4250 0x7f1d6c160c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:47.338 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.336+0000 7f1d74600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d6c15dd00 0x7f1d6c158d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:47.338 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.336+0000 7f1d74600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6c0d37c0 con 0x7f1d6c0d3e70 2024-03-24T01:23:47.338 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.336+0000 7f1d74600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6c15e170 con 0x7f1d6c15dd00 2024-03-24T01:23:47.338 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.336+0000 7f1d74600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6c0d3940 con 0x7f1d6c0d4250 2024-03-24T01:23:47.338 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.336+0000 7f1d71200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d6c0d4250 0x7f1d6c160c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:47.339 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.336+0000 7f1d72600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d6c15dd00 0x7f1d6c158d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:47.339 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.336+0000 7f1d71200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d6c0d4250 0x7f1d6c160c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:55134/0 (socket says 172.21.15.120:55134) 2024-03-24T01:23:47.339 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.336+0000 7f1d71200640 1 -- 172.21.15.120:0/1737646137 learned_addr learned my addr 172.21.15.120:0/1737646137 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-24T01:23:47.339 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.336+0000 7f1d71c00640 1 --2- 172.21.15.120:0/1737646137 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d6c0d3e70 0x7f1d6c1f68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:47.339 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.336+0000 7f1d71200640 1 -- 172.21.15.120:0/1737646137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d6c15dd00 msgr2=0x7f1d6c158d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:47.339 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.336+0000 7f1d71200640 1 --2- 172.21.15.120:0/1737646137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d6c15dd00 0x7f1d6c158d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:47.339 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.336+0000 7f1d71200640 1 -- 172.21.15.120:0/1737646137 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d6c0d3e70 msgr2=0x7f1d6c1f68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:47.339 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.337+0000 7f1d71200640 1 --2- 172.21.15.120:0/1737646137 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d6c0d3e70 0x7f1d6c1f68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:47.339 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.337+0000 7f1d71200640 1 -- 172.21.15.120:0/1737646137 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d6c0c7d10 con 0x7f1d6c0d4250 2024-03-24T01:23:47.339 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.337+0000 7f1d71c00640 1 --2- 172.21.15.120:0/1737646137 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d6c0d3e70 0x7f1d6c1f68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:23:47.339 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.337+0000 7f1d71200640 1 --2- 172.21.15.120:0/1737646137 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d6c0d4250 0x7f1d6c160c30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1d5800e9d0 tx=0x7f1d5800ee90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:23:47.339 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.337+0000 7f1d69600640 1 -- 172.21.15.120:0/1737646137 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d58017070 con 0x7f1d6c0d4250 2024-03-24T01:23:47.339 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.337+0000 7f1d69600640 1 -- 172.21.15.120:0/1737646137 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d5800ca60 con 0x7f1d6c0d4250 2024-03-24T01:23:47.339 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.337+0000 7f1d74600640 1 -- 172.21.15.120:0/1737646137 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d6c0c7ff0 con 0x7f1d6c0d4250 2024-03-24T01:23:47.340 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.337+0000 7f1d69600640 1 -- 172.21.15.120:0/1737646137 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d5800ce10 con 0x7f1d6c0d4250 2024-03-24T01:23:47.340 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.337+0000 7f1d74600640 1 -- 172.21.15.120:0/1737646137 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d6c0c85b0 con 0x7f1d6c0d4250 2024-03-24T01:23:47.340 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.338+0000 7f1d69600640 1 -- 172.21.15.120:0/1737646137 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f1d5801a070 con 0x7f1d6c0d4250 2024-03-24T01:23:47.340 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.338+0000 7f1d69600640 1 -- 172.21.15.120:0/1737646137 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f1d5801c860 con 0x7f1d6c0d4250 2024-03-24T01:23:47.340 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.338+0000 7f1d74600640 1 -- 172.21.15.120:0/1737646137 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d6c04f400 con 0x7f1d6c0d4250 2024-03-24T01:23:47.341 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.341+0000 7f1d69600640 1 -- 172.21.15.120:0/1737646137 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f1d58013470 con 0x7f1d6c0d4250 2024-03-24T01:23:47.420 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.420+0000 7f1d74600640 1 -- 172.21.15.120:0/1737646137 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd new", "uuid": "aeced55e-300f-4d89-89ec-f59579c3f96c", "id": 0} v 0) v1 -- 0x7f1d6c1f6e00 con 0x7f1d6c0d4250 2024-03-24T01:23:47.445 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.427+0000 7f1d69600640 1 -- 172.21.15.120:0/1737646137 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "aeced55e-300f-4d89-89ec-f59579c3f96c", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f1d58005990 con 0x7f1d6c0d4250 2024-03-24T01:23:47.446 INFO:teuthology.orchestra.run.smithi120.stdout:0 2024-03-24T01:23:47.447 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.428+0000 7f1d74600640 1 -- 172.21.15.120:0/1737646137 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d6c0d4250 msgr2=0x7f1d6c160c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:47.447 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.428+0000 7f1d74600640 1 --2- 172.21.15.120:0/1737646137 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d6c0d4250 0x7f1d6c160c30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1d5800e9d0 tx=0x7f1d5800ee90 comp rx=0 tx=0).stop 2024-03-24T01:23:47.447 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.428+0000 7f1d74600640 1 -- 172.21.15.120:0/1737646137 shutdown_connections 2024-03-24T01:23:47.447 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.428+0000 7f1d74600640 1 --2- 172.21.15.120:0/1737646137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d6c15dd00 0x7f1d6c158d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:47.447 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.428+0000 7f1d74600640 1 --2- 172.21.15.120:0/1737646137 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1d6c0d4250 0x7f1d6c160c30 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:47.447 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.428+0000 7f1d74600640 1 --2- 172.21.15.120:0/1737646137 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d6c0d3e70 0x7f1d6c1f68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:47.447 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.428+0000 7f1d74600640 1 -- 172.21.15.120:0/1737646137 >> 172.21.15.120:0/1737646137 conn(0x7f1d6c072c10 msgr2=0x7f1d6c0d0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:23:47.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.428+0000 7f1d74600640 1 -- 172.21.15.120:0/1737646137 shutdown_connections 2024-03-24T01:23:47.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.428+0000 7f1d74600640 1 -- 172.21.15.120:0/1737646137 wait complete. 2024-03-24T01:23:47.448 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new f7214f99-b216-43b3-91b9-67ce92c611cc 1 2024-03-24T01:23:47.580 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-24T01:23:47.580+0000 7fd6d4a20040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T01:23:47.640 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.639+0000 7fcf28a00640 1 Processor -- start 2024-03-24T01:23:47.640 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.639+0000 7fcf28a00640 1 -- start start 2024-03-24T01:23:47.641 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.640+0000 7fcf28a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcf240b5570 0x7fcf240b5a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:47.641 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.640+0000 7fcf28a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf240b6f00 0x7fcf240b7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:47.641 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.640+0000 7fcf28a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf240b7870 con 0x7fcf2415dec0 2024-03-24T01:23:47.641 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.640+0000 7fcf28a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf2415e2a0 con 0x7fcf240b5570 2024-03-24T01:23:47.641 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.640+0000 7fcf28a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf240cf2f0 con 0x7fcf240b6f00 2024-03-24T01:23:47.642 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.641+0000 7fcf1be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcf240b5570 0x7fcf240b5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:47.642 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.641+0000 7fcf23200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf240b6f00 0x7fcf240b7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:47.642 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.641+0000 7fcf1be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcf240b5570 0x7fcf240b5a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:40042/0 (socket says 172.21.15.120:40042) 2024-03-24T01:23:47.642 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.641+0000 7fcf1be00640 1 -- 172.21.15.120:0/447169157 learned_addr learned my addr 172.21.15.120:0/447169157 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-24T01:23:47.642 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.641+0000 7fcf23200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf240b6f00 0x7fcf240b7330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:55144/0 (socket says 172.21.15.120:55144) 2024-03-24T01:23:47.642 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.641+0000 7fcf1be00640 1 -- 172.21.15.120:0/447169157 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf240b6f00 msgr2=0x7fcf240b7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:23:47.643 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.641+0000 7fcf1be00640 1 --2- 172.21.15.120:0/447169157 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf240b6f00 0x7fcf240b7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:47.643 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.641+0000 7fcf1be00640 1 -- 172.21.15.120:0/447169157 >> v1:172.21.15.120:6789/0 conn(0x7fcf2415dec0 legacy=0x7fcf240d28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:47.643 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.641+0000 7fcf1be00640 1 -- 172.21.15.120:0/447169157 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf240d1e00 con 0x7fcf240b5570 2024-03-24T01:23:47.643 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.641+0000 7fcf23200640 1 --2- 172.21.15.120:0/447169157 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf240b6f00 0x7fcf240b7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T01:23:47.643 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.642+0000 7fcf1be00640 1 --2- 172.21.15.120:0/447169157 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf240b5570 0x7fcf240b5a00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c009870 tx=0x7fcf0c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c52df33982c215de server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:23:47.644 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.643+0000 7fcf21e00640 1 -- 172.21.15.120:0/447169157 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c034070 con 0x7fcf240b5570 2024-03-24T01:23:47.644 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.643+0000 7fcf21e00640 1 -- 172.21.15.120:0/447169157 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf0c030840 con 0x7fcf240b5570 2024-03-24T01:23:47.644 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.643+0000 7fcf21e00640 1 -- 172.21.15.120:0/447169157 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c030c10 con 0x7fcf240b5570 2024-03-24T01:23:47.644 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.643+0000 7fcf28a00640 1 -- 172.21.15.120:0/447169157 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf240b5570 msgr2=0x7fcf240b5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:47.644 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.643+0000 7fcf28a00640 1 --2- 172.21.15.120:0/447169157 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf240b5570 0x7fcf240b5a00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c009870 tx=0x7fcf0c02b980 comp rx=0 tx=0).stop 2024-03-24T01:23:47.644 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.643+0000 7fcf28a00640 1 -- 172.21.15.120:0/447169157 shutdown_connections 2024-03-24T01:23:47.644 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.643+0000 7fcf28a00640 1 --2- 172.21.15.120:0/447169157 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf240b6f00 0x7fcf240b7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:47.644 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.643+0000 7fcf28a00640 1 --2- 172.21.15.120:0/447169157 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf240b5570 0x7fcf240b5a00 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:47.645 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.644+0000 7fcf28a00640 1 -- 172.21.15.120:0/447169157 >> 172.21.15.120:0/447169157 conn(0x7fcf240d0750 msgr2=0x7fcf240d0b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:23:47.645 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.644+0000 7fcf28a00640 1 -- 172.21.15.120:0/447169157 shutdown_connections 2024-03-24T01:23:47.645 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.644+0000 7fcf28a00640 1 -- 172.21.15.120:0/447169157 wait complete. 2024-03-24T01:23:47.645 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-24T01:23:47.644+0000 7f1993fdb040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T01:23:47.646 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.645+0000 7fcf28a00640 1 Processor -- start 2024-03-24T01:23:47.646 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.646+0000 7fcf28a00640 1 -- start start 2024-03-24T01:23:47.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.647+0000 7fcf28a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf240b5570 0x7fcf241f45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:47.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.647+0000 7fcf28a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf240b6f00 0x7fcf240c9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:47.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.647+0000 7fcf28a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf2415dec0 0x7fcf241f8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:47.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.647+0000 7fcf28a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf240d3170 con 0x7fcf240b5570 2024-03-24T01:23:47.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.647+0000 7fcf28a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf240d2ff0 con 0x7fcf2415dec0 2024-03-24T01:23:47.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.647+0000 7fcf28a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf240ce580 con 0x7fcf240b6f00 2024-03-24T01:23:47.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.647+0000 7fcf23200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf2415dec0 0x7fcf241f8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:47.648 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.647+0000 7fcf23200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf2415dec0 0x7fcf241f8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:40048/0 (socket says 172.21.15.120:40048) 2024-03-24T01:23:47.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.647+0000 7fcf23200640 1 -- 172.21.15.120:0/2946759648 learned_addr learned my addr 172.21.15.120:0/2946759648 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-24T01:23:47.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.647+0000 7fcf1be00640 1 --2- 172.21.15.120:0/2946759648 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf240b6f00 0x7fcf240c9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:47.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.647+0000 7fcf22800640 1 --2- 172.21.15.120:0/2946759648 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf240b5570 0x7fcf241f45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:47.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.647+0000 7fcf23200640 1 -- 172.21.15.120:0/2946759648 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf240b6f00 msgr2=0x7fcf240c9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:47.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.647+0000 7fcf23200640 1 --2- 172.21.15.120:0/2946759648 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf240b6f00 0x7fcf240c9bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:47.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.647+0000 7fcf23200640 1 -- 172.21.15.120:0/2946759648 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf240b5570 msgr2=0x7fcf241f45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:47.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.647+0000 7fcf23200640 1 --2- 172.21.15.120:0/2946759648 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf240b5570 0x7fcf241f45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:47.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.647+0000 7fcf23200640 1 -- 172.21.15.120:0/2946759648 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf240c78e0 con 0x7fcf2415dec0 2024-03-24T01:23:47.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.647+0000 7fcf22800640 1 --2- 172.21.15.120:0/2946759648 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf240b5570 0x7fcf241f45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:23:47.649 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.648+0000 7fcf23200640 1 --2- 172.21.15.120:0/2946759648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf2415dec0 0x7fcf241f8290 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c00efc0 tx=0x7fcf1c009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:23:47.650 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-24T01:23:47.648+0000 7fd6d4a20040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T01:23:47.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.648+0000 7fcf1aa00640 1 -- 172.21.15.120:0/2946759648 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c018030 con 0x7fcf2415dec0 2024-03-24T01:23:47.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.648+0000 7fcf1aa00640 1 -- 172.21.15.120:0/2946759648 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf1c00cde0 con 0x7fcf2415dec0 2024-03-24T01:23:47.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.648+0000 7fcf1aa00640 1 -- 172.21.15.120:0/2946759648 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c003bb0 con 0x7fcf2415dec0 2024-03-24T01:23:47.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.648+0000 7fcf28a00640 1 -- 172.21.15.120:0/2946759648 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf241f4d50 con 0x7fcf2415dec0 2024-03-24T01:23:47.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.648+0000 7fcf28a00640 1 -- 172.21.15.120:0/2946759648 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf241f5310 con 0x7fcf2415dec0 2024-03-24T01:23:47.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.648+0000 7fcf1aa00640 1 -- 172.21.15.120:0/2946759648 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fcf1c005d10 con 0x7fcf2415dec0 2024-03-24T01:23:47.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.649+0000 7fcf1aa00640 1 -- 172.21.15.120:0/2946759648 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7fcf1c01d980 con 0x7fcf2415dec0 2024-03-24T01:23:47.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.650+0000 7fcf28a00640 1 -- 172.21.15.120:0/2946759648 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf24067790 con 0x7fcf2415dec0 2024-03-24T01:23:47.653 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.653+0000 7fcf1aa00640 1 -- 172.21.15.120:0/2946759648 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fcf24067790 con 0x7fcf2415dec0 2024-03-24T01:23:47.712 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-24T01:23:47.712+0000 7f1993fdb040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T01:23:47.734 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.733+0000 7fcf28a00640 1 -- 172.21.15.120:0/2946759648 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f7214f99-b216-43b3-91b9-67ce92c611cc", "id": 1} v 0) v1 -- 0x7fcf2404f400 con 0x7fcf2415dec0 2024-03-24T01:23:47.741 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.741+0000 7fcf1aa00640 1 -- 172.21.15.120:0/2946759648 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f7214f99-b216-43b3-91b9-67ce92c611cc", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fcf1c022070 con 0x7fcf2415dec0 2024-03-24T01:23:47.741 INFO:teuthology.orchestra.run.smithi120.stdout:1 2024-03-24T01:23:47.743 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.742+0000 7fcf28a00640 1 -- 172.21.15.120:0/2946759648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf2415dec0 msgr2=0x7fcf241f8290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:47.743 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.742+0000 7fcf28a00640 1 --2- 172.21.15.120:0/2946759648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf2415dec0 0x7fcf241f8290 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c00efc0 tx=0x7fcf1c009f90 comp rx=0 tx=0).stop 2024-03-24T01:23:47.743 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.743+0000 7fcf28a00640 1 -- 172.21.15.120:0/2946759648 shutdown_connections 2024-03-24T01:23:47.743 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.743+0000 7fcf28a00640 1 --2- 172.21.15.120:0/2946759648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf2415dec0 0x7fcf241f8290 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:47.744 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.743+0000 7fcf28a00640 1 --2- 172.21.15.120:0/2946759648 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf240b6f00 0x7fcf240c9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:47.744 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.743+0000 7fcf28a00640 1 --2- 172.21.15.120:0/2946759648 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf240b5570 0x7fcf241f45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:47.744 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.743+0000 7fcf28a00640 1 -- 172.21.15.120:0/2946759648 >> 172.21.15.120:0/2946759648 conn(0x7fcf240d0750 msgr2=0x7fcf240b8280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:23:47.744 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.743+0000 7fcf28a00640 1 -- 172.21.15.120:0/2946759648 shutdown_connections 2024-03-24T01:23:47.744 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.743+0000 7fcf28a00640 1 -- 172.21.15.120:0/2946759648 wait complete. 2024-03-24T01:23:47.766 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new 9cb0e2f5-39b8-454e-87b5-4e3eb784ab24 2 2024-03-24T01:23:47.774 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-24T01:23:47.774+0000 7f1993fdb040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T01:23:47.844 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-24T01:23:47.843+0000 7f1993fdb040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T01:23:47.896 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-24T01:23:47.896+0000 7f1993fdb040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T01:23:47.957 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.956+0000 7fca56200640 1 Processor -- start 2024-03-24T01:23:47.957 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.956+0000 7fca56200640 1 -- start start 2024-03-24T01:23:47.959 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.957+0000 7fca56200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fca5006c4c0 0x7fca50099bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:47.959 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.957+0000 7fca56200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fca5009a100 0x7fca5015d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:47.959 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.957+0000 7fca56200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca500d0570 con 0x7fca5006d230 2024-03-24T01:23:47.959 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.957+0000 7fca56200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca500d0350 con 0x7fca5009a100 2024-03-24T01:23:47.959 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.957+0000 7fca56200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca500d1180 con 0x7fca5006c4c0 2024-03-24T01:23:47.959 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.958+0000 7fca54e00640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fca5006d230 0x7fca500cfc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:56988/0 (socket says 172.21.15.120:56988) 2024-03-24T01:23:47.959 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.958+0000 7fca54e00640 1 -- 172.21.15.120:0/3158360825 learned_addr learned my addr 172.21.15.120:0/3158360825 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-03-24T01:23:47.960 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.958+0000 7fca55800640 1 --2- 172.21.15.120:0/3158360825 >> v2:172.21.15.120:3300/0 conn(0x7fca5009a100 0x7fca5015d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:47.960 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.958+0000 7fca47e00640 1 --2- 172.21.15.120:0/3158360825 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fca5006c4c0 0x7fca50099bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:47.960 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.958+0000 7fca55800640 1 -- 172.21.15.120:0/3158360825 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fca5006c4c0 msgr2=0x7fca50099bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:23:47.960 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.958+0000 7fca55800640 1 --2- 172.21.15.120:0/3158360825 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fca5006c4c0 0x7fca50099bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:47.960 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.958+0000 7fca55800640 1 -- 172.21.15.120:0/3158360825 >> v1:172.21.15.120:6789/0 conn(0x7fca5006d230 legacy=0x7fca500cfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:47.960 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.958+0000 7fca55800640 1 -- 172.21.15.120:0/3158360825 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca500d0850 con 0x7fca5009a100 2024-03-24T01:23:47.960 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.958+0000 7fca47e00640 1 --2- 172.21.15.120:0/3158360825 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fca5006c4c0 0x7fca50099bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T01:23:47.961 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.959+0000 7fca55800640 1 --2- 172.21.15.120:0/3158360825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca5009a100 0x7fca5015d540 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fca48009d90 tx=0x7fca4802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b6a599cd89ba958 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:23:47.961 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.960+0000 7fca4fe00640 1 -- 172.21.15.120:0/3158360825 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca48039070 con 0x7fca5009a100 2024-03-24T01:23:47.962 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.960+0000 7fca4fe00640 1 -- 172.21.15.120:0/3158360825 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca4802cc50 con 0x7fca5009a100 2024-03-24T01:23:47.962 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.960+0000 7fca4fe00640 1 -- 172.21.15.120:0/3158360825 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca48034440 con 0x7fca5009a100 2024-03-24T01:23:47.962 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.960+0000 7fca56200640 1 -- 172.21.15.120:0/3158360825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca5009a100 msgr2=0x7fca5015d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:47.962 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.960+0000 7fca56200640 1 --2- 172.21.15.120:0/3158360825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca5009a100 0x7fca5015d540 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fca48009d90 tx=0x7fca4802bea0 comp rx=0 tx=0).stop 2024-03-24T01:23:47.962 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.960+0000 7fca56200640 1 -- 172.21.15.120:0/3158360825 shutdown_connections 2024-03-24T01:23:47.962 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.960+0000 7fca56200640 1 --2- 172.21.15.120:0/3158360825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca5009a100 0x7fca5015d540 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:47.962 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.960+0000 7fca56200640 1 --2- 172.21.15.120:0/3158360825 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fca5006c4c0 0x7fca50099bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:47.962 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.960+0000 7fca56200640 1 -- 172.21.15.120:0/3158360825 >> 172.21.15.120:0/3158360825 conn(0x7fca500b33c0 msgr2=0x7fca500b37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:23:47.962 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.960+0000 7fca56200640 1 -- 172.21.15.120:0/3158360825 shutdown_connections 2024-03-24T01:23:47.962 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.960+0000 7fca56200640 1 -- 172.21.15.120:0/3158360825 wait complete. 2024-03-24T01:23:47.965 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.962+0000 7fca56200640 1 Processor -- start 2024-03-24T01:23:47.965 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.962+0000 7fca56200640 1 -- start start 2024-03-24T01:23:47.965 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.963+0000 7fca56200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca5006c4c0 0x7fca50156ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:47.965 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.963+0000 7fca56200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca5006d230 0x7fca500b4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:47.966 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.963+0000 7fca56200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fca5009a100 0x7fca500b5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:47.966 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.963+0000 7fca56200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5006d610 con 0x7fca5006d230 2024-03-24T01:23:47.966 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.963+0000 7fca56200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca500d0e10 con 0x7fca5006c4c0 2024-03-24T01:23:47.966 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.963+0000 7fca56200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca5015da80 con 0x7fca5009a100 2024-03-24T01:23:47.966 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.963+0000 7fca54e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca5006c4c0 0x7fca50156ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:47.966 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.963+0000 7fca54e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca5006c4c0 0x7fca50156ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:40062/0 (socket says 172.21.15.120:40062) 2024-03-24T01:23:47.966 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.963+0000 7fca55800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fca5009a100 0x7fca500b5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:47.966 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.963+0000 7fca54e00640 1 -- 172.21.15.120:0/2585186872 learned_addr learned my addr 172.21.15.120:0/2585186872 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-24T01:23:47.966 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.964+0000 7fca55800640 1 -- 172.21.15.120:0/2585186872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca5006c4c0 msgr2=0x7fca50156ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:47.966 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.964+0000 7fca55800640 1 --2- 172.21.15.120:0/2585186872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca5006c4c0 0x7fca50156ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:47.966 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.964+0000 7fca55800640 1 -- 172.21.15.120:0/2585186872 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca5006d230 msgr2=0x7fca500b4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:47.966 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.964+0000 7fca47e00640 1 --2- 172.21.15.120:0/2585186872 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca5006d230 0x7fca500b4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:47.967 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.964+0000 7fca55800640 1 --2- 172.21.15.120:0/2585186872 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca5006d230 0x7fca500b4be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:47.967 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.964+0000 7fca55800640 1 -- 172.21.15.120:0/2585186872 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca50160c20 con 0x7fca5009a100 2024-03-24T01:23:47.967 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.964+0000 7fca54e00640 1 --2- 172.21.15.120:0/2585186872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca5006c4c0 0x7fca50156ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T01:23:47.967 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.964+0000 7fca55800640 1 --2- 172.21.15.120:0/2585186872 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fca5009a100 0x7fca500b5370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fca500736e0 tx=0x7fca48004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:23:47.967 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.964+0000 7fca4d600640 1 -- 172.21.15.120:0/2585186872 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca48044020 con 0x7fca5009a100 2024-03-24T01:23:47.967 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.964+0000 7fca4d600640 1 -- 172.21.15.120:0/2585186872 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca4802ca50 con 0x7fca5009a100 2024-03-24T01:23:47.967 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.964+0000 7fca4d600640 1 -- 172.21.15.120:0/2585186872 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca48039040 con 0x7fca5009a100 2024-03-24T01:23:47.967 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.964+0000 7fca56200640 1 -- 172.21.15.120:0/2585186872 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca500b5120 con 0x7fca5009a100 2024-03-24T01:23:47.967 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.964+0000 7fca56200640 1 -- 172.21.15.120:0/2585186872 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca500caaf0 con 0x7fca5009a100 2024-03-24T01:23:47.967 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.965+0000 7fca4d600640 1 -- 172.21.15.120:0/2585186872 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fca48047020 con 0x7fca5009a100 2024-03-24T01:23:47.967 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.965+0000 7fca4d600640 1 -- 172.21.15.120:0/2585186872 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fca48040ac0 con 0x7fca5009a100 2024-03-24T01:23:47.968 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.965+0000 7fca56200640 1 -- 172.21.15.120:0/2585186872 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca5004f400 con 0x7fca5009a100 2024-03-24T01:23:47.968 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:47.968+0000 7fca4d600640 1 -- 172.21.15.120:0/2585186872 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fca4804e030 con 0x7fca5009a100 2024-03-24T01:23:48.035 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-24T01:23:48.034+0000 7f1993fdb040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T01:23:48.046 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.046+0000 7fca56200640 1 -- 172.21.15.120:0/2585186872 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd new", "uuid": "9cb0e2f5-39b8-454e-87b5-4e3eb784ab24", "id": 2} v 0) v1 -- 0x7fca5006c9c0 con 0x7fca5009a100 2024-03-24T01:23:48.055 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.055+0000 7fca4d600640 1 -- 172.21.15.120:0/2585186872 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9cb0e2f5-39b8-454e-87b5-4e3eb784ab24", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fca50160c20 con 0x7fca5009a100 2024-03-24T01:23:48.055 INFO:teuthology.orchestra.run.smithi120.stdout:2 2024-03-24T01:23:48.056 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.056+0000 7fca56200640 1 -- 172.21.15.120:0/2585186872 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fca5009a100 msgr2=0x7fca500b5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:48.056 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.056+0000 7fca56200640 1 --2- 172.21.15.120:0/2585186872 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fca5009a100 0x7fca500b5370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fca500736e0 tx=0x7fca48004a40 comp rx=0 tx=0).stop 2024-03-24T01:23:48.056 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.056+0000 7fca56200640 1 -- 172.21.15.120:0/2585186872 shutdown_connections 2024-03-24T01:23:48.057 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.056+0000 7fca56200640 1 --2- 172.21.15.120:0/2585186872 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca5006d230 0x7fca500b4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:48.057 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.056+0000 7fca56200640 1 --2- 172.21.15.120:0/2585186872 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fca5009a100 0x7fca500b5370 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:48.057 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.056+0000 7fca56200640 1 --2- 172.21.15.120:0/2585186872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca5006c4c0 0x7fca50156ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:48.057 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.056+0000 7fca56200640 1 -- 172.21.15.120:0/2585186872 >> 172.21.15.120:0/2585186872 conn(0x7fca500b33c0 msgr2=0x7fca500cfc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:23:48.058 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.056+0000 7fca56200640 1 -- 172.21.15.120:0/2585186872 shutdown_connections 2024-03-24T01:23:48.058 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.056+0000 7fca56200640 1 -- 172.21.15.120:0/2585186872 wait complete. 2024-03-24T01:23:48.074 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new 4d8e413a-693a-4587-925f-2f5263a49617 3 2024-03-24T01:23:48.122 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-24T01:23:48.120+0000 7f1993fdb040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T01:23:48.177 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-24T01:23:48.177+0000 7f1993fdb040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T01:23:48.243 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-24T01:23:48.242+0000 7f1993fdb040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T01:23:48.276 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-24T01:23:48.276+0000 7fd6d4a20040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T01:23:48.301 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-24T01:23:48.301+0000 7f1993fdb040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T01:23:48.306 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.306+0000 7f7a0b400640 1 Processor -- start 2024-03-24T01:23:48.306 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.306+0000 7f7a0b400640 1 -- start start 2024-03-24T01:23:48.308 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.307+0000 7f7a0b400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7a0415de60 0x7f7a040d1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:48.308 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.307+0000 7f7a0b400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a040b97b0 0x7f7a040cead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:48.308 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.307+0000 7f7a0b400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a040d2790 con 0x7f7a040d3830 2024-03-24T01:23:48.308 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.307+0000 7f7a0b400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a040b7eb0 con 0x7f7a0415de60 2024-03-24T01:23:48.309 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.307+0000 7f7a0b400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a040d2a70 con 0x7f7a040b97b0 2024-03-24T01:23:48.309 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.307+0000 7f7a08a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7a0415de60 0x7f7a040d1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:48.309 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.307+0000 7f7a03e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a040b97b0 0x7f7a040cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:48.309 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.308+0000 7f7a08a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7a0415de60 0x7f7a040d1e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:40068/0 (socket says 172.21.15.120:40068) 2024-03-24T01:23:48.309 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.308+0000 7f7a03e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a040b97b0 0x7f7a040cead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:55172/0 (socket says 172.21.15.120:55172) 2024-03-24T01:23:48.309 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.308+0000 7f7a08a00640 1 -- 172.21.15.120:0/1894920466 learned_addr learned my addr 172.21.15.120:0/1894920466 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-24T01:23:48.309 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.308+0000 7f7a08a00640 1 -- 172.21.15.120:0/1894920466 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a040b97b0 msgr2=0x7f7a040cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:23:48.309 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.308+0000 7f7a08a00640 1 --2- 172.21.15.120:0/1894920466 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a040b97b0 0x7f7a040cead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:48.309 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.308+0000 7f7a08a00640 1 -- 172.21.15.120:0/1894920466 >> v1:172.21.15.90:6789/0 conn(0x7f7a040d3830 legacy=0x7f7a040b5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:23:48.309 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.308+0000 7f7a08a00640 1 -- 172.21.15.120:0/1894920466 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a040d2d50 con 0x7f7a0415de60 2024-03-24T01:23:48.310 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.310+0000 7f7a08a00640 1 --2- 172.21.15.120:0/1894920466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a0415de60 0x7f7a040d1e40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f79f8009870 tx=0x7f79f802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=497c0c6fb83d4e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:23:48.310 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.310+0000 7f7a03400640 1 -- 172.21.15.120:0/1894920466 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f8035020 con 0x7f7a0415de60 2024-03-24T01:23:48.311 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.310+0000 7f7a03400640 1 -- 172.21.15.120:0/1894920466 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79f8031640 con 0x7f7a0415de60 2024-03-24T01:23:48.311 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.310+0000 7f7a03400640 1 -- 172.21.15.120:0/1894920466 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f802d040 con 0x7f7a0415de60 2024-03-24T01:23:48.312 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.311+0000 7f7a0b400640 1 -- 172.21.15.120:0/1894920466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a0415de60 msgr2=0x7f7a040d1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:48.312 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.311+0000 7f7a0b400640 1 --2- 172.21.15.120:0/1894920466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a0415de60 0x7f7a040d1e40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f79f8009870 tx=0x7f79f802b8f0 comp rx=0 tx=0).stop 2024-03-24T01:23:48.312 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.311+0000 7f7a0b400640 1 -- 172.21.15.120:0/1894920466 shutdown_connections 2024-03-24T01:23:48.312 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.311+0000 7f7a0b400640 1 --2- 172.21.15.120:0/1894920466 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a040b97b0 0x7f7a040cead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:48.312 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.311+0000 7f7a0b400640 1 --2- 172.21.15.120:0/1894920466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a0415de60 0x7f7a040d1e40 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:48.312 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.311+0000 7f7a0b400640 1 -- 172.21.15.120:0/1894920466 >> 172.21.15.120:0/1894920466 conn(0x7f7a04072c10 msgr2=0x7f7a04073020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:23:48.312 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.311+0000 7f7a0b400640 1 -- 172.21.15.120:0/1894920466 shutdown_connections 2024-03-24T01:23:48.312 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.311+0000 7f7a0b400640 1 -- 172.21.15.120:0/1894920466 wait complete. 2024-03-24T01:23:48.313 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.313+0000 7f7a0b400640 1 Processor -- start 2024-03-24T01:23:48.313 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.313+0000 7f7a0b400640 1 -- start start 2024-03-24T01:23:48.315 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.314+0000 7f7a0b400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a040d3830 0x7f7a040cadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:48.316 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.314+0000 7f7a0b400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a0415de60 0x7f7a041f33c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:48.316 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.314+0000 7f7a0b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a041f3900 0x7f7a0415b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:48.316 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.314+0000 7f7a0b400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0415e240 con 0x7f7a041f3900 2024-03-24T01:23:48.316 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.314+0000 7f7a0b400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a040d3310 con 0x7f7a040d3830 2024-03-24T01:23:48.316 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.314+0000 7f7a0b400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0415e3c0 con 0x7f7a0415de60 2024-03-24T01:23:48.316 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.314+0000 7f7a08a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a040d3830 0x7f7a040cadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:48.316 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.314+0000 7f7a08a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a040d3830 0x7f7a040cadd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:40076/0 (socket says 172.21.15.120:40076) 2024-03-24T01:23:48.316 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.314+0000 7f7a08a00640 1 -- 172.21.15.120:0/3052262715 learned_addr learned my addr 172.21.15.120:0/3052262715 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-24T01:23:48.316 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.314+0000 7f7a03e00640 1 --2- 172.21.15.120:0/3052262715 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a0415de60 0x7f7a041f33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:48.316 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.314+0000 7f7a09400640 1 --2- 172.21.15.120:0/3052262715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a041f3900 0x7f7a0415b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:48.316 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.315+0000 7f7a08a00640 1 -- 172.21.15.120:0/3052262715 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a0415de60 msgr2=0x7f7a041f33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:48.316 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.315+0000 7f7a08a00640 1 --2- 172.21.15.120:0/3052262715 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a0415de60 0x7f7a041f33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:48.316 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.315+0000 7f7a08a00640 1 -- 172.21.15.120:0/3052262715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a041f3900 msgr2=0x7f7a0415b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:48.316 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.315+0000 7f7a08a00640 1 --2- 172.21.15.120:0/3052262715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a041f3900 0x7f7a0415b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:48.317 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.315+0000 7f7a08a00640 1 -- 172.21.15.120:0/3052262715 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a040cbbe0 con 0x7f7a040d3830 2024-03-24T01:23:48.317 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.315+0000 7f7a08a00640 1 --2- 172.21.15.120:0/3052262715 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a040d3830 0x7f7a040cadd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f79f8000c00 tx=0x7f79f80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:23:48.317 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.315+0000 7f7a00c00640 1 -- 172.21.15.120:0/3052262715 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f8040070 con 0x7f7a040d3830 2024-03-24T01:23:48.317 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.315+0000 7f7a00c00640 1 -- 172.21.15.120:0/3052262715 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79f80399f0 con 0x7f7a040d3830 2024-03-24T01:23:48.317 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.315+0000 7f7a00c00640 1 -- 172.21.15.120:0/3052262715 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f8035040 con 0x7f7a040d3830 2024-03-24T01:23:48.317 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.315+0000 7f7a0b400640 1 -- 172.21.15.120:0/3052262715 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a040cb620 con 0x7f7a040d3830 2024-03-24T01:23:48.317 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.315+0000 7f7a0b400640 1 -- 172.21.15.120:0/3052262715 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a041f3de0 con 0x7f7a040d3830 2024-03-24T01:23:48.318 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.316+0000 7f7a00c00640 1 -- 172.21.15.120:0/3052262715 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f79f804b060 con 0x7f7a040d3830 2024-03-24T01:23:48.318 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.316+0000 7f7a00c00640 1 -- 172.21.15.120:0/3052262715 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f79f803a8b0 con 0x7f7a040d3830 2024-03-24T01:23:48.318 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.316+0000 7f7a0b400640 1 -- 172.21.15.120:0/3052262715 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a0404f400 con 0x7f7a040d3830 2024-03-24T01:23:48.319 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.319+0000 7f7a00c00640 1 -- 172.21.15.120:0/3052262715 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f79f803abd0 con 0x7f7a040d3830 2024-03-24T01:23:48.340 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-24T01:23:48.340+0000 7fd6d4a20040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T01:23:48.391 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-24T01:23:48.390+0000 7fd6d4a20040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T01:23:48.398 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.398+0000 7f7a0b400640 1 -- 172.21.15.120:0/3052262715 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4d8e413a-693a-4587-925f-2f5263a49617", "id": 3} v 0) v1 -- 0x7f7a040b7900 con 0x7f7a040d3830 2024-03-24T01:23:48.406 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.406+0000 7f7a00c00640 1 -- 172.21.15.120:0/3052262715 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4d8e413a-693a-4587-925f-2f5263a49617", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f79f803a420 con 0x7f7a040d3830 2024-03-24T01:23:48.406 INFO:teuthology.orchestra.run.smithi120.stdout:3 2024-03-24T01:23:48.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.407+0000 7f7a0b400640 1 -- 172.21.15.120:0/3052262715 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a040d3830 msgr2=0x7f7a040cadd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:48.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.407+0000 7f7a0b400640 1 --2- 172.21.15.120:0/3052262715 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a040d3830 0x7f7a040cadd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f79f8000c00 tx=0x7f79f80046c0 comp rx=0 tx=0).stop 2024-03-24T01:23:48.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.407+0000 7f7a0b400640 1 -- 172.21.15.120:0/3052262715 shutdown_connections 2024-03-24T01:23:48.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.407+0000 7f7a0b400640 1 --2- 172.21.15.120:0/3052262715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a041f3900 0x7f7a0415b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:48.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.407+0000 7f7a0b400640 1 --2- 172.21.15.120:0/3052262715 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a0415de60 0x7f7a041f33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:48.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.407+0000 7f7a0b400640 1 --2- 172.21.15.120:0/3052262715 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a040d3830 0x7f7a040cadd0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:48.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.407+0000 7f7a0b400640 1 -- 172.21.15.120:0/3052262715 >> 172.21.15.120:0/3052262715 conn(0x7f7a04072c10 msgr2=0x7f7a040b9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:23:48.408 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.407+0000 7f7a0b400640 1 -- 172.21.15.120:0/3052262715 shutdown_connections 2024-03-24T01:23:48.409 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.407+0000 7f7a0b400640 1 -- 172.21.15.120:0/3052262715 wait complete. 2024-03-24T01:23:48.425 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new 898478a5-2b9d-4f13-8bdd-cb28aafe1f01 4 2024-03-24T01:23:48.454 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-24T01:23:48.454+0000 7fd6d4a20040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T01:23:48.503 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-24T01:23:48.503+0000 7fd6d4a20040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T01:23:48.616 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.615+0000 7f8623200640 1 Processor -- start 2024-03-24T01:23:48.616 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.615+0000 7f8623200640 1 -- start start 2024-03-24T01:23:48.618 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.617+0000 7f8623200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f86240b2a70 0x7f86240b2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:48.618 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.617+0000 7f8623200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f862406b1f0 con 0x7f86240cd700 2024-03-24T01:23:48.618 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.617+0000 7f8623200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f862406b4d0 con 0x7f8624070580 2024-03-24T01:23:48.618 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.617+0000 7f8623200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f862406b7b0 con 0x7f86240b2a70 2024-03-24T01:23:48.618 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.617+0000 7f8621e00640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f8624070580 0x7f8624161930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:56992/0 (socket says 172.21.15.120:56992) 2024-03-24T01:23:48.618 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.617+0000 7f8621e00640 1 -- 172.21.15.120:0/4076642356 learned_addr learned my addr 172.21.15.120:0/4076642356 (peer_addr_for_me v1:172.21.15.120:0/0) 2024-03-24T01:23:48.618 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.617+0000 7f861be00640 1 --2- 172.21.15.120:0/4076642356 >> v2:172.21.15.120:3300/0 conn(0x7f86240b2a70 0x7f86240b2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:48.619 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.618+0000 7f861be00640 1 -- 172.21.15.120:0/4076642356 >> v1:172.21.15.120:6789/0 conn(0x7f8624070580 legacy=0x7f8624161930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:48.619 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.618+0000 7f861be00640 1 -- 172.21.15.120:0/4076642356 >> v1:172.21.15.90:6789/0 conn(0x7f86240cd700 legacy=0x7f86240cfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:23:48.619 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.618+0000 7f861be00640 1 -- 172.21.15.120:0/4076642356 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f862406ba90 con 0x7f86240b2a70 2024-03-24T01:23:48.619 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.619+0000 7f861be00640 1 --2- 172.21.15.120:0/4076642356 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86240b2a70 0x7f86240b2ee0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f860c0097d0 tx=0x7f860c02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92fa86742f3a9bd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:23:48.620 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.619+0000 7f8621400640 1 -- 172.21.15.120:0/4076642356 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860c035020 con 0x7f86240b2a70 2024-03-24T01:23:48.620 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.619+0000 7f8621400640 1 -- 172.21.15.120:0/4076642356 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f860c031810 con 0x7f86240b2a70 2024-03-24T01:23:48.620 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.619+0000 7f8621400640 1 -- 172.21.15.120:0/4076642356 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860c031be0 con 0x7f86240b2a70 2024-03-24T01:23:48.621 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.620+0000 7f8623200640 1 -- 172.21.15.120:0/4076642356 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86240b2a70 msgr2=0x7f86240b2ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:48.621 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.620+0000 7f8623200640 1 --2- 172.21.15.120:0/4076642356 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86240b2a70 0x7f86240b2ee0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f860c0097d0 tx=0x7f860c02b8b0 comp rx=0 tx=0).stop 2024-03-24T01:23:48.621 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.620+0000 7f8623200640 1 -- 172.21.15.120:0/4076642356 shutdown_connections 2024-03-24T01:23:48.621 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.620+0000 7f8623200640 1 --2- 172.21.15.120:0/4076642356 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86240b2a70 0x7f86240b2ee0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:48.621 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.620+0000 7f8623200640 1 -- 172.21.15.120:0/4076642356 >> 172.21.15.120:0/4076642356 conn(0x7f8624099a70 msgr2=0x7f86240b2360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:23:48.621 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.620+0000 7f8623200640 1 -- 172.21.15.120:0/4076642356 shutdown_connections 2024-03-24T01:23:48.621 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.620+0000 7f8623200640 1 -- 172.21.15.120:0/4076642356 wait complete. 2024-03-24T01:23:48.622 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.622+0000 7f8623200640 1 Processor -- start 2024-03-24T01:23:48.622 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.622+0000 7f8623200640 1 -- start start 2024-03-24T01:23:48.624 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.623+0000 7f8623200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8624070580 0x7f86240b31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:48.624 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.623+0000 7f8623200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86240b2a70 0x7f86241600d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:48.624 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.623+0000 7f8623200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86240cd700 0x7f86240ca100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:48.624 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.623+0000 7f8623200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8624070960 con 0x7f86240b2a70 2024-03-24T01:23:48.624 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.623+0000 7f8623200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8624162040 con 0x7f86240cd700 2024-03-24T01:23:48.624 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.623+0000 7f8623200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86240ccb10 con 0x7f8624070580 2024-03-24T01:23:48.624 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.623+0000 7f8622800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86240cd700 0x7f86240ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:48.624 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.623+0000 7f8621e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8624070580 0x7f86240b31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:48.624 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.623+0000 7f8621e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8624070580 0x7f86240b31c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:55196/0 (socket says 172.21.15.120:55196) 2024-03-24T01:23:48.625 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.623+0000 7f8621e00640 1 -- 172.21.15.120:0/3843167215 learned_addr learned my addr 172.21.15.120:0/3843167215 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-24T01:23:48.625 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.623+0000 7f861be00640 1 --2- 172.21.15.120:0/3843167215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86240b2a70 0x7f86241600d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:48.625 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.623+0000 7f8621e00640 1 -- 172.21.15.120:0/3843167215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86240cd700 msgr2=0x7f86240ca100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:48.625 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.623+0000 7f8621e00640 1 --2- 172.21.15.120:0/3843167215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86240cd700 0x7f86240ca100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:48.625 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.623+0000 7f8621e00640 1 -- 172.21.15.120:0/3843167215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86240b2a70 msgr2=0x7f86241600d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:48.625 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.623+0000 7f8621e00640 1 --2- 172.21.15.120:0/3843167215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86240b2a70 0x7f86241600d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:48.625 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.623+0000 7f8621e00640 1 -- 172.21.15.120:0/3843167215 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86241e7490 con 0x7f8624070580 2024-03-24T01:23:48.625 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.624+0000 7f8622800640 1 --2- 172.21.15.120:0/3843167215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86240cd700 0x7f86240ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T01:23:48.625 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.624+0000 7f8621e00640 1 --2- 172.21.15.120:0/3843167215 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8624070580 0x7f86240b31c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f861400c400 tx=0x7f861400c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:23:48.626 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.624+0000 7f861a000640 1 -- 172.21.15.120:0/3843167215 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8614017070 con 0x7f8624070580 2024-03-24T01:23:48.626 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.624+0000 7f861a000640 1 -- 172.21.15.120:0/3843167215 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8614005ce0 con 0x7f8624070580 2024-03-24T01:23:48.626 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.624+0000 7f8623200640 1 -- 172.21.15.120:0/3843167215 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86241e7770 con 0x7f8624070580 2024-03-24T01:23:48.627 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.624+0000 7f861a000640 1 -- 172.21.15.120:0/3843167215 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86140072f0 con 0x7f8624070580 2024-03-24T01:23:48.627 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.624+0000 7f8623200640 1 -- 172.21.15.120:0/3843167215 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86241e8010 con 0x7f8624070580 2024-03-24T01:23:48.627 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.624+0000 7f861a000640 1 -- 172.21.15.120:0/3843167215 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f861401a070 con 0x7f8624070580 2024-03-24T01:23:48.627 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.624+0000 7f861a000640 1 -- 172.21.15.120:0/3843167215 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f8614007490 con 0x7f8624070580 2024-03-24T01:23:48.627 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.625+0000 7f8623200640 1 -- 172.21.15.120:0/3843167215 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f862404f400 con 0x7f8624070580 2024-03-24T01:23:48.628 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.628+0000 7f861a000640 1 -- 172.21.15.120:0/3843167215 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f8614021070 con 0x7f8624070580 2024-03-24T01:23:48.633 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-24T01:23:48.633+0000 7fd6d4a20040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T01:23:48.707 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.706+0000 7f8623200640 1 -- 172.21.15.120:0/3843167215 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd new", "uuid": "898478a5-2b9d-4f13-8bdd-cb28aafe1f01", "id": 4} v 0) v1 -- 0x7f86241e7b00 con 0x7f8624070580 2024-03-24T01:23:48.713 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-24T01:23:48.713+0000 7fd6d4a20040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T01:23:48.714 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.714+0000 7f861a000640 1 -- 172.21.15.120:0/3843167215 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "898478a5-2b9d-4f13-8bdd-cb28aafe1f01", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f861401c440 con 0x7f8624070580 2024-03-24T01:23:48.714 INFO:teuthology.orchestra.run.smithi120.stdout:4 2024-03-24T01:23:48.716 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.715+0000 7f8623200640 1 -- 172.21.15.120:0/3843167215 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8624070580 msgr2=0x7f86240b31c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:48.716 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.715+0000 7f8623200640 1 --2- 172.21.15.120:0/3843167215 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8624070580 0x7f86240b31c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f861400c400 tx=0x7f861400c8c0 comp rx=0 tx=0).stop 2024-03-24T01:23:48.716 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.715+0000 7f8623200640 1 -- 172.21.15.120:0/3843167215 shutdown_connections 2024-03-24T01:23:48.716 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.715+0000 7f8623200640 1 --2- 172.21.15.120:0/3843167215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86240b2a70 0x7f86241600d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:48.716 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.715+0000 7f8623200640 1 --2- 172.21.15.120:0/3843167215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86240cd700 0x7f86240ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:48.716 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.715+0000 7f8623200640 1 --2- 172.21.15.120:0/3843167215 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8624070580 0x7f86240b31c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:48.716 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.715+0000 7f8623200640 1 -- 172.21.15.120:0/3843167215 >> 172.21.15.120:0/3843167215 conn(0x7f8624099a70 msgr2=0x7f86240ad640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:23:48.716 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.715+0000 7f8623200640 1 -- 172.21.15.120:0/3843167215 shutdown_connections 2024-03-24T01:23:48.716 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.715+0000 7f8623200640 1 -- 172.21.15.120:0/3843167215 wait complete. 2024-03-24T01:23:48.732 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new 35fe45dc-a2aa-44de-a90b-a3fe9e274fed 5 2024-03-24T01:23:48.767 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-24T01:23:48.766+0000 7fd6d4a20040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T01:23:48.783 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-24T01:23:48.783+0000 7f1993fdb040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T01:23:48.827 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-24T01:23:48.826+0000 7fd6d4a20040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T01:23:48.881 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-24T01:23:48.881+0000 7fd6d4a20040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T01:23:48.920 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.920+0000 7f2e7e200640 1 Processor -- start 2024-03-24T01:23:48.921 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.920+0000 7f2e7e200640 1 -- start start 2024-03-24T01:23:48.923 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.921+0000 7f2e7e200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2e7815dec0 0x7f2e780d28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:48.923 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.921+0000 7f2e7e200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2e780d2e20 0x7f2e780b5570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:48.923 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.921+0000 7f2e7e200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2e780b5ab0 0x7f2e780b6f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:48.923 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.921+0000 7f2e7e200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e7815e2a0 con 0x7f2e780d2e20 2024-03-24T01:23:48.923 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.921+0000 7f2e7e200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e780b5ee0 con 0x7f2e7815dec0 2024-03-24T01:23:48.923 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.921+0000 7f2e7e200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e780cf2f0 con 0x7f2e780b5ab0 2024-03-24T01:23:48.923 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.922+0000 7f2e7ce00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2e7815dec0 0x7f2e780d28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:48.923 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.922+0000 7f2e7ce00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2e7815dec0 0x7f2e780d28e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:40112/0 (socket says 172.21.15.120:40112) 2024-03-24T01:23:48.923 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.922+0000 7f2e7ce00640 1 -- 172.21.15.120:0/3658280089 learned_addr learned my addr 172.21.15.120:0/3658280089 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-24T01:23:48.923 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.922+0000 7f2e7d800640 1 --2- 172.21.15.120:0/3658280089 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2e780b5ab0 0x7f2e780b6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:48.923 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.922+0000 7f2e77e00640 1 --2- 172.21.15.120:0/3658280089 >> v2:172.21.15.90:3300/0 conn(0x7f2e780d2e20 0x7f2e780b5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:48.923 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.922+0000 7f2e7ce00640 1 -- 172.21.15.120:0/3658280089 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2e780b5ab0 msgr2=0x7f2e780b6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:23:48.924 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.922+0000 7f2e7ce00640 1 --2- 172.21.15.120:0/3658280089 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2e780b5ab0 0x7f2e780b6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:48.924 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.922+0000 7f2e7ce00640 1 -- 172.21.15.120:0/3658280089 >> v2:172.21.15.90:3300/0 conn(0x7f2e780d2e20 msgr2=0x7f2e780b5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:23:48.924 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.922+0000 7f2e7ce00640 1 --2- 172.21.15.120:0/3658280089 >> v2:172.21.15.90:3300/0 conn(0x7f2e780d2e20 0x7f2e780b5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:48.924 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.922+0000 7f2e7ce00640 1 -- 172.21.15.120:0/3658280089 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e780d1e00 con 0x7f2e7815dec0 2024-03-24T01:23:48.925 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.923+0000 7f2e7ce00640 1 --2- 172.21.15.120:0/3658280089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e7815dec0 0x7f2e780d28e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2e7808c1e0 tx=0x7f2e6c02b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2d514fc3b5911b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:23:48.925 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.924+0000 7f2e77400640 1 -- 172.21.15.120:0/3658280089 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6c035020 con 0x7f2e7815dec0 2024-03-24T01:23:48.925 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.924+0000 7f2e77400640 1 -- 172.21.15.120:0/3658280089 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e6c031870 con 0x7f2e7815dec0 2024-03-24T01:23:48.925 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.924+0000 7f2e77400640 1 -- 172.21.15.120:0/3658280089 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6c031c40 con 0x7f2e7815dec0 2024-03-24T01:23:48.925 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.924+0000 7f2e7e200640 1 -- 172.21.15.120:0/3658280089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e7815dec0 msgr2=0x7f2e780d28e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:48.925 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.924+0000 7f2e7e200640 1 --2- 172.21.15.120:0/3658280089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e7815dec0 0x7f2e780d28e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2e7808c1e0 tx=0x7f2e6c02b910 comp rx=0 tx=0).stop 2024-03-24T01:23:48.925 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.924+0000 7f2e7e200640 1 -- 172.21.15.120:0/3658280089 shutdown_connections 2024-03-24T01:23:48.926 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.924+0000 7f2e7e200640 1 --2- 172.21.15.120:0/3658280089 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2e780b5ab0 0x7f2e780b6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:48.926 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.924+0000 7f2e7e200640 1 --2- 172.21.15.120:0/3658280089 >> v2:172.21.15.90:3300/0 conn(0x7f2e780d2e20 0x7f2e780b5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:48.926 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.924+0000 7f2e7e200640 1 --2- 172.21.15.120:0/3658280089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e7815dec0 0x7f2e780d28e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:48.926 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.924+0000 7f2e7e200640 1 -- 172.21.15.120:0/3658280089 >> 172.21.15.120:0/3658280089 conn(0x7f2e780d0750 msgr2=0x7f2e780d0b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:23:48.926 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.925+0000 7f2e7e200640 1 -- 172.21.15.120:0/3658280089 shutdown_connections 2024-03-24T01:23:48.926 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.925+0000 7f2e7e200640 1 -- 172.21.15.120:0/3658280089 wait complete. 2024-03-24T01:23:48.927 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.926+0000 7f2e7e200640 1 Processor -- start 2024-03-24T01:23:48.930 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.926+0000 7f2e7e200640 1 -- start start 2024-03-24T01:23:48.932 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.927+0000 7f2e7e200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2e780b5ab0 0x7f2e780ca010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:48.932 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.927+0000 7f2e7e200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e780d2e20 0x7f2e78154920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:48.932 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.927+0000 7f2e7e200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e7815dec0 0x7f2e7806c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:48.932 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.927+0000 7f2e7e200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e780b75c0 con 0x7f2e780d2e20 2024-03-24T01:23:48.932 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.927+0000 7f2e7e200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e780b7440 con 0x7f2e7815dec0 2024-03-24T01:23:48.932 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.927+0000 7f2e7e200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e780b7740 con 0x7f2e780b5ab0 2024-03-24T01:23:48.933 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.927+0000 7f2e7ce00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2e780b5ab0 0x7f2e780ca010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:48.933 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.928+0000 7f2e7ce00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2e780b5ab0 0x7f2e780ca010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:55216/0 (socket says 172.21.15.120:55216) 2024-03-24T01:23:48.933 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.928+0000 7f2e7ce00640 1 -- 172.21.15.120:0/2708526567 learned_addr learned my addr 172.21.15.120:0/2708526567 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-24T01:23:48.933 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.928+0000 7f2e7d800640 1 --2- 172.21.15.120:0/2708526567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e7815dec0 0x7f2e7806c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:48.933 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.928+0000 7f2e77e00640 1 --2- 172.21.15.120:0/2708526567 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e780d2e20 0x7f2e78154920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:48.933 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.928+0000 7f2e7ce00640 1 -- 172.21.15.120:0/2708526567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e7815dec0 msgr2=0x7f2e7806c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:48.933 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.928+0000 7f2e7ce00640 1 --2- 172.21.15.120:0/2708526567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e7815dec0 0x7f2e7806c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:48.933 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.928+0000 7f2e7ce00640 1 -- 172.21.15.120:0/2708526567 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e780d2e20 msgr2=0x7f2e78154920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:48.933 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.928+0000 7f2e7ce00640 1 --2- 172.21.15.120:0/2708526567 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e780d2e20 0x7f2e78154920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:48.933 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.928+0000 7f2e7ce00640 1 -- 172.21.15.120:0/2708526567 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e780c9680 con 0x7f2e780b5ab0 2024-03-24T01:23:48.933 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.928+0000 7f2e77e00640 1 --2- 172.21.15.120:0/2708526567 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e780d2e20 0x7f2e78154920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:23:48.933 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.928+0000 7f2e7ce00640 1 --2- 172.21.15.120:0/2708526567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2e780b5ab0 0x7f2e780ca010 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c000c00 tx=0x7f2e6c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:23:48.933 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.928+0000 7f2e74c00640 1 -- 172.21.15.120:0/2708526567 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6c035040 con 0x7f2e780b5ab0 2024-03-24T01:23:48.933 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.929+0000 7f2e74c00640 1 -- 172.21.15.120:0/2708526567 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e6c030af0 con 0x7f2e780b5ab0 2024-03-24T01:23:48.933 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.929+0000 7f2e74c00640 1 -- 172.21.15.120:0/2708526567 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6c043430 con 0x7f2e780b5ab0 2024-03-24T01:23:48.934 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.929+0000 7f2e7e200640 1 -- 172.21.15.120:0/2708526567 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e78158920 con 0x7f2e780b5ab0 2024-03-24T01:23:48.934 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.929+0000 7f2e7e200640 1 -- 172.21.15.120:0/2708526567 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e78158ee0 con 0x7f2e780b5ab0 2024-03-24T01:23:48.934 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.929+0000 7f2e74c00640 1 -- 172.21.15.120:0/2708526567 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f2e6c038020 con 0x7f2e780b5ab0 2024-03-24T01:23:48.934 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.929+0000 7f2e74c00640 1 -- 172.21.15.120:0/2708526567 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f2e6c03a420 con 0x7f2e780b5ab0 2024-03-24T01:23:48.934 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.930+0000 7f2e7e200640 1 -- 172.21.15.120:0/2708526567 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e78160b80 con 0x7f2e780b5ab0 2024-03-24T01:23:48.934 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:48.932+0000 7f2e74c00640 1 -- 172.21.15.120:0/2708526567 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f2e78160b80 con 0x7f2e780b5ab0 2024-03-24T01:23:48.952 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-24T01:23:48.952+0000 7f1993fdb040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T01:23:49.011 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.011+0000 7f2e7e200640 1 -- 172.21.15.120:0/2708526567 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd new", "uuid": "35fe45dc-a2aa-44de-a90b-a3fe9e274fed", "id": 5} v 0) v1 -- 0x7f2e78067790 con 0x7f2e780b5ab0 2024-03-24T01:23:49.017 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.017+0000 7f2e74c00640 1 -- 172.21.15.120:0/2708526567 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "35fe45dc-a2aa-44de-a90b-a3fe9e274fed", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2e78067790 con 0x7f2e780b5ab0 2024-03-24T01:23:49.017 INFO:teuthology.orchestra.run.smithi120.stdout:5 2024-03-24T01:23:49.019 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.018+0000 7f2e7e200640 1 -- 172.21.15.120:0/2708526567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2e780b5ab0 msgr2=0x7f2e780ca010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:49.019 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.018+0000 7f2e7e200640 1 --2- 172.21.15.120:0/2708526567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2e780b5ab0 0x7f2e780ca010 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c000c00 tx=0x7f2e6c0046c0 comp rx=0 tx=0).stop 2024-03-24T01:23:49.019 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.018+0000 7f2e7e200640 1 -- 172.21.15.120:0/2708526567 shutdown_connections 2024-03-24T01:23:49.019 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.018+0000 7f2e7e200640 1 --2- 172.21.15.120:0/2708526567 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2e780d2e20 0x7f2e78154920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:49.019 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.018+0000 7f2e7e200640 1 --2- 172.21.15.120:0/2708526567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e7815dec0 0x7f2e7806c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:49.019 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.018+0000 7f2e7e200640 1 --2- 172.21.15.120:0/2708526567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2e780b5ab0 0x7f2e780ca010 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:49.019 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.018+0000 7f2e7e200640 1 -- 172.21.15.120:0/2708526567 >> 172.21.15.120:0/2708526567 conn(0x7f2e780d0750 msgr2=0x7f2e780cd190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:23:49.019 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.018+0000 7f2e7e200640 1 -- 172.21.15.120:0/2708526567 shutdown_connections 2024-03-24T01:23:49.019 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.018+0000 7f2e7e200640 1 -- 172.21.15.120:0/2708526567 wait complete. 2024-03-24T01:23:49.034 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new 9af87141-0549-43a1-9350-abb0db6108f2 6 2024-03-24T01:23:49.105 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-24T01:23:49.104+0000 7f1993fdb040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T01:23:49.161 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-24T01:23:49.160+0000 7f1993fdb040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T01:23:49.216 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.215+0000 7f510ca00640 1 Processor -- start 2024-03-24T01:23:49.216 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.215+0000 7f510ca00640 1 -- start start 2024-03-24T01:23:49.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.216+0000 7f510ca00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f510815dc90 0x7f51080b6b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:49.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.217+0000 7f510ca00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f51080b70c0 0x7f51080d1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:49.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.217+0000 7f510ca00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51080b7560 con 0x7f51080d2090 2024-03-24T01:23:49.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.217+0000 7f510ca00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51080b7840 con 0x7f510815dc90 2024-03-24T01:23:49.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.217+0000 7f510ca00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51080b7b20 con 0x7f51080b70c0 2024-03-24T01:23:49.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.217+0000 7f5106a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f51080b70c0 0x7f51080d1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:49.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.217+0000 7f5106a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f51080b70c0 0x7f51080d1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:40134/0 (socket says 172.21.15.120:40134) 2024-03-24T01:23:49.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.217+0000 7f5106a00640 1 -- 172.21.15.120:0/1564328317 learned_addr learned my addr 172.21.15.120:0/1564328317 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-24T01:23:49.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.217+0000 7f5107400640 1 --2- 172.21.15.120:0/1564328317 >> v2:172.21.15.90:3300/0 conn(0x7f510815dc90 0x7f51080b6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:49.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.217+0000 7f5106a00640 1 -- 172.21.15.120:0/1564328317 >> v2:172.21.15.90:3300/0 conn(0x7f510815dc90 msgr2=0x7f51080b6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:23:49.218 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.217+0000 7f5106a00640 1 --2- 172.21.15.120:0/1564328317 >> v2:172.21.15.90:3300/0 conn(0x7f510815dc90 0x7f51080b6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:49.219 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.217+0000 7f5106a00640 1 -- 172.21.15.120:0/1564328317 >> v1:172.21.15.90:6789/0 conn(0x7f51080d2090 legacy=0x7f51080b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:23:49.219 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.217+0000 7f5106a00640 1 -- 172.21.15.120:0/1564328317 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51080b7e00 con 0x7f51080b70c0 2024-03-24T01:23:49.219 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.217+0000 7f5107400640 1 --2- 172.21.15.120:0/1564328317 >> v2:172.21.15.90:3300/0 conn(0x7f510815dc90 0x7f51080b6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:23:49.219 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.218+0000 7f5106a00640 1 --2- 172.21.15.120:0/1564328317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51080b70c0 0x7f51080d1b50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f51080d1410 tx=0x7f50f402b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba47c4e49282a3d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:23:49.219 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.219+0000 7f5106000640 1 -- 172.21.15.120:0/1564328317 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f4034070 con 0x7f51080b70c0 2024-03-24T01:23:49.219 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.219+0000 7f5106000640 1 -- 172.21.15.120:0/1564328317 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50f4030860 con 0x7f51080b70c0 2024-03-24T01:23:49.220 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.219+0000 7f5106000640 1 -- 172.21.15.120:0/1564328317 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f4030c30 con 0x7f51080b70c0 2024-03-24T01:23:49.220 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.219+0000 7f510ca00640 1 -- 172.21.15.120:0/1564328317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51080b70c0 msgr2=0x7f51080d1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:49.220 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.219+0000 7f510ca00640 1 --2- 172.21.15.120:0/1564328317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51080b70c0 0x7f51080d1b50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f51080d1410 tx=0x7f50f402b9a0 comp rx=0 tx=0).stop 2024-03-24T01:23:49.221 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.220+0000 7f510ca00640 1 -- 172.21.15.120:0/1564328317 shutdown_connections 2024-03-24T01:23:49.221 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.220+0000 7f510ca00640 1 --2- 172.21.15.120:0/1564328317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51080b70c0 0x7f51080d1b50 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:49.221 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.220+0000 7f510ca00640 1 --2- 172.21.15.120:0/1564328317 >> v2:172.21.15.90:3300/0 conn(0x7f510815dc90 0x7f51080b6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:49.221 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.220+0000 7f510ca00640 1 -- 172.21.15.120:0/1564328317 >> 172.21.15.120:0/1564328317 conn(0x7f5108072be0 msgr2=0x7f5108072ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:23:49.221 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.220+0000 7f510ca00640 1 -- 172.21.15.120:0/1564328317 shutdown_connections 2024-03-24T01:23:49.221 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.220+0000 7f510ca00640 1 -- 172.21.15.120:0/1564328317 wait complete. 2024-03-24T01:23:49.222 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.221+0000 7f510ca00640 1 Processor -- start 2024-03-24T01:23:49.222 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.221+0000 7f510ca00640 1 -- start start 2024-03-24T01:23:49.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.222+0000 7f510ca00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f51080b70c0 0x7f5108149850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:49.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.222+0000 7f510ca00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51080d2090 0x7f5108149d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:49.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.222+0000 7f510ca00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f510815dc90 0x7f510814c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:49.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.222+0000 7f510ca00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51080b8d10 con 0x7f51080b70c0 2024-03-24T01:23:49.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.222+0000 7f510ca00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f510815e070 con 0x7f510815dc90 2024-03-24T01:23:49.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.222+0000 7f510ca00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51080b8e90 con 0x7f51080d2090 2024-03-24T01:23:49.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.223+0000 7f5106a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51080d2090 0x7f5108149d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:49.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.223+0000 7f5106a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51080d2090 0x7f5108149d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.120:55228/0 (socket says 172.21.15.120:55228) 2024-03-24T01:23:49.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.223+0000 7f5106a00640 1 -- 172.21.15.120:0/874293355 learned_addr learned my addr 172.21.15.120:0/874293355 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-24T01:23:49.225 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.223+0000 7f5107e00640 1 --2- 172.21.15.120:0/874293355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f510815dc90 0x7f510814c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:49.226 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.223+0000 7f5107400640 1 --2- 172.21.15.120:0/874293355 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f51080b70c0 0x7f5108149850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:49.226 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.223+0000 7f5106a00640 1 -- 172.21.15.120:0/874293355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f510815dc90 msgr2=0x7f510814c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:49.226 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.223+0000 7f5106a00640 1 --2- 172.21.15.120:0/874293355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f510815dc90 0x7f510814c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:49.226 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.223+0000 7f5106a00640 1 -- 172.21.15.120:0/874293355 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f51080b70c0 msgr2=0x7f5108149850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:49.226 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.223+0000 7f5106a00640 1 --2- 172.21.15.120:0/874293355 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f51080b70c0 0x7f5108149850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:49.226 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.223+0000 7f5106a00640 1 -- 172.21.15.120:0/874293355 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f510814a640 con 0x7f51080d2090 2024-03-24T01:23:49.226 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.223+0000 7f5106a00640 1 --2- 172.21.15.120:0/874293355 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51080d2090 0x7f5108149d90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f50f4000c00 tx=0x7f50f4004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:23:49.226 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.224+0000 7f50ff400640 1 -- 172.21.15.120:0/874293355 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f4048070 con 0x7f51080d2090 2024-03-24T01:23:49.226 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.224+0000 7f50ff400640 1 -- 172.21.15.120:0/874293355 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50f4039d90 con 0x7f51080d2090 2024-03-24T01:23:49.226 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.224+0000 7f510ca00640 1 -- 172.21.15.120:0/874293355 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51080c9b30 con 0x7f51080d2090 2024-03-24T01:23:49.226 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.224+0000 7f50ff400640 1 -- 172.21.15.120:0/874293355 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f4034040 con 0x7f51080d2090 2024-03-24T01:23:49.226 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.224+0000 7f510ca00640 1 -- 172.21.15.120:0/874293355 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5108150760 con 0x7f51080d2090 2024-03-24T01:23:49.226 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.224+0000 7f50ff400640 1 -- 172.21.15.120:0/874293355 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f50f404b070 con 0x7f51080d2090 2024-03-24T01:23:49.226 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.224+0000 7f50ff400640 1 -- 172.21.15.120:0/874293355 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f50f404c9a0 con 0x7f51080d2090 2024-03-24T01:23:49.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.225+0000 7f510ca00640 1 -- 172.21.15.120:0/874293355 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f510804f400 con 0x7f51080d2090 2024-03-24T01:23:49.227 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.227+0000 7f50ff400640 1 -- 172.21.15.120:0/874293355 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f50f4030720 con 0x7f51080d2090 2024-03-24T01:23:49.296 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-24T01:23:49.295+0000 7f1993fdb040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T01:23:49.307 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.307+0000 7f510ca00640 1 -- 172.21.15.120:0/874293355 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd new", "uuid": "9af87141-0549-43a1-9350-abb0db6108f2", "id": 6} v 0) v1 -- 0x7f51080ca050 con 0x7f51080d2090 2024-03-24T01:23:49.313 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.312+0000 7f50ff400640 1 -- 172.21.15.120:0/874293355 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9af87141-0549-43a1-9350-abb0db6108f2", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f50f404cbf0 con 0x7f51080d2090 2024-03-24T01:23:49.313 INFO:teuthology.orchestra.run.smithi120.stdout:6 2024-03-24T01:23:49.314 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.313+0000 7f510ca00640 1 -- 172.21.15.120:0/874293355 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51080d2090 msgr2=0x7f5108149d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:49.314 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.313+0000 7f510ca00640 1 --2- 172.21.15.120:0/874293355 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51080d2090 0x7f5108149d90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f50f4000c00 tx=0x7f50f4004730 comp rx=0 tx=0).stop 2024-03-24T01:23:49.314 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.313+0000 7f510ca00640 1 -- 172.21.15.120:0/874293355 shutdown_connections 2024-03-24T01:23:49.314 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.313+0000 7f510ca00640 1 --2- 172.21.15.120:0/874293355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f510815dc90 0x7f510814c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:49.314 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.313+0000 7f510ca00640 1 --2- 172.21.15.120:0/874293355 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51080d2090 0x7f5108149d90 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:49.315 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.313+0000 7f510ca00640 1 --2- 172.21.15.120:0/874293355 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f51080b70c0 0x7f5108149850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:49.315 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.313+0000 7f510ca00640 1 -- 172.21.15.120:0/874293355 >> 172.21.15.120:0/874293355 conn(0x7f5108072be0 msgr2=0x7f51080d34d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:23:49.315 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.314+0000 7f510ca00640 1 -- 172.21.15.120:0/874293355 shutdown_connections 2024-03-24T01:23:49.315 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.314+0000 7f510ca00640 1 -- 172.21.15.120:0/874293355 wait complete. 2024-03-24T01:23:49.330 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-24T01:23:49.330+0000 7fd6d4a20040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T01:23:49.331 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --cluster ceph osd new 424703a0-ba08-4d01-9d48-b38eabdde4f1 7 2024-03-24T01:23:49.414 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-24T01:23:49.414+0000 7f1993fdb040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T01:23:49.488 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-24T01:23:49.487+0000 7fd6d4a20040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T01:23:49.516 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.515+0000 7fc89f000640 1 Processor -- start 2024-03-24T01:23:49.516 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.516+0000 7fc89f000640 1 -- start start 2024-03-24T01:23:49.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.517+0000 7fc89f000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc8980b97c0 0x7fc8980d1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:49.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.517+0000 7fc89f000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8980d2090 0x7fc8980b8600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:49.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.517+0000 7fc89f000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8980b7290 con 0x7fc89815dc90 2024-03-24T01:23:49.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.517+0000 7fc89f000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8980b7560 con 0x7fc8980b97c0 2024-03-24T01:23:49.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.517+0000 7fc89f000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8980b7840 con 0x7fc8980d2090 2024-03-24T01:23:49.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.517+0000 7fc89d200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc8980b97c0 0x7fc8980d1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:49.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.517+0000 7fc89e600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8980d2090 0x7fc8980b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:49.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.517+0000 7fc89d200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc8980b97c0 0x7fc8980d1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:40156/0 (socket says 172.21.15.120:40156) 2024-03-24T01:23:49.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.517+0000 7fc89d200640 1 -- 172.21.15.120:0/673990671 learned_addr learned my addr 172.21.15.120:0/673990671 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-24T01:23:49.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.517+0000 7fc89dc00640 1 --1- 172.21.15.120:0/673990671 >> v1:172.21.15.120:6789/0 conn(0x7fc89815dc90 0x7fc8980b6b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.120:56994/0 (socket says 172.21.15.120:56994) 2024-03-24T01:23:49.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.518+0000 7fc897e00640 1 -- 172.21.15.120:0/673990671 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989552078 0 0) 0x7fc8980b7290 con 0x7fc89815dc90 2024-03-24T01:23:49.519 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.518+0000 7fc897e00640 1 -- 172.21.15.120:0/673990671 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc880003520 con 0x7fc89815dc90 2024-03-24T01:23:49.520 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.518+0000 7fc89e600640 1 -- 172.21.15.120:0/673990671 >> v2:172.21.15.120:3300/0 conn(0x7fc8980b97c0 msgr2=0x7fc8980d1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:23:49.520 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.518+0000 7fc89e600640 1 --2- 172.21.15.120:0/673990671 >> v2:172.21.15.120:3300/0 conn(0x7fc8980b97c0 0x7fc8980d1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:49.520 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.518+0000 7fc89e600640 1 -- 172.21.15.120:0/673990671 >> v1:172.21.15.120:6789/0 conn(0x7fc89815dc90 legacy=0x7fc8980b6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:49.520 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.518+0000 7fc89e600640 1 -- 172.21.15.120:0/673990671 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8980b7b20 con 0x7fc8980d2090 2024-03-24T01:23:49.520 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.518+0000 7fc89d200640 1 --2- 172.21.15.120:0/673990671 >> v2:172.21.15.120:3300/0 conn(0x7fc8980b97c0 0x7fc8980d1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T01:23:49.520 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.520+0000 7fc89e600640 1 --2- 172.21.15.120:0/673990671 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8980d2090 0x7fc8980b8600 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc890009d90 tx=0x7fc89002bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f42fb92f01926132 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:23:49.520 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.520+0000 7fc897e00640 1 -- 172.21.15.120:0/673990671 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc890039070 con 0x7fc8980d2090 2024-03-24T01:23:49.521 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.520+0000 7fc897e00640 1 -- 172.21.15.120:0/673990671 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc89002cce0 con 0x7fc8980d2090 2024-03-24T01:23:49.521 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.520+0000 7fc897e00640 1 -- 172.21.15.120:0/673990671 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc890034660 con 0x7fc8980d2090 2024-03-24T01:23:49.521 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.520+0000 7fc89f000640 1 -- 172.21.15.120:0/673990671 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8980d2090 msgr2=0x7fc8980b8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:49.521 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.520+0000 7fc89f000640 1 --2- 172.21.15.120:0/673990671 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8980d2090 0x7fc8980b8600 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc890009d90 tx=0x7fc89002bf30 comp rx=0 tx=0).stop 2024-03-24T01:23:49.522 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.521+0000 7fc89f000640 1 -- 172.21.15.120:0/673990671 shutdown_connections 2024-03-24T01:23:49.522 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.521+0000 7fc89f000640 1 --2- 172.21.15.120:0/673990671 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8980d2090 0x7fc8980b8600 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:49.522 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.521+0000 7fc89f000640 1 --2- 172.21.15.120:0/673990671 >> v2:172.21.15.120:3300/0 conn(0x7fc8980b97c0 0x7fc8980d1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:49.522 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.521+0000 7fc89f000640 1 -- 172.21.15.120:0/673990671 >> 172.21.15.120:0/673990671 conn(0x7fc898072be0 msgr2=0x7fc898072ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:23:49.522 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.521+0000 7fc89f000640 1 -- 172.21.15.120:0/673990671 shutdown_connections 2024-03-24T01:23:49.522 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.521+0000 7fc89f000640 1 -- 172.21.15.120:0/673990671 wait complete. 2024-03-24T01:23:49.523 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.522+0000 7fc89f000640 1 Processor -- start 2024-03-24T01:23:49.523 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.523+0000 7fc89f000640 1 -- start start 2024-03-24T01:23:49.525 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.523+0000 7fc89f000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8980b97c0 0x7fc8981f4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:49.526 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.523+0000 7fc89f000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8980d2090 0x7fc8981516a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:49.526 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.523+0000 7fc89f000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc89815dc90 0x7fc898155630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:49.526 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.523+0000 7fc89f000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc89815e070 con 0x7fc8980d2090 2024-03-24T01:23:49.526 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.523+0000 7fc89f000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8980b80e0 con 0x7fc8980b97c0 2024-03-24T01:23:49.526 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.523+0000 7fc89f000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8980b8b40 con 0x7fc89815dc90 2024-03-24T01:23:49.526 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.524+0000 7fc89dc00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8980b97c0 0x7fc8981f4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:49.526 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.524+0000 7fc89dc00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8980b97c0 0x7fc8981f4250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:40160/0 (socket says 172.21.15.120:40160) 2024-03-24T01:23:49.526 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.524+0000 7fc89dc00640 1 -- 172.21.15.120:0/2876285970 learned_addr learned my addr 172.21.15.120:0/2876285970 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-24T01:23:49.526 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.524+0000 7fc89e600640 1 --2- 172.21.15.120:0/2876285970 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc89815dc90 0x7fc898155630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:49.526 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.524+0000 7fc89d200640 1 --2- 172.21.15.120:0/2876285970 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8980d2090 0x7fc8981516a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:49.526 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.524+0000 7fc89dc00640 1 -- 172.21.15.120:0/2876285970 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc89815dc90 msgr2=0x7fc898155630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:49.526 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.524+0000 7fc89dc00640 1 --2- 172.21.15.120:0/2876285970 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc89815dc90 0x7fc898155630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:49.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.524+0000 7fc89dc00640 1 -- 172.21.15.120:0/2876285970 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8980d2090 msgr2=0x7fc8981516a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:49.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.524+0000 7fc89dc00640 1 --2- 172.21.15.120:0/2876285970 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8980d2090 0x7fc8981516a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:49.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.524+0000 7fc89dc00640 1 -- 172.21.15.120:0/2876285970 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8981f4790 con 0x7fc8980b97c0 2024-03-24T01:23:49.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.524+0000 7fc89dc00640 1 --2- 172.21.15.120:0/2876285970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8980b97c0 0x7fc8981f4250 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc88c00c390 tx=0x7fc88c00c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:23:49.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.525+0000 7fc895600640 1 -- 172.21.15.120:0/2876285970 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc88c0074e0 con 0x7fc8980b97c0 2024-03-24T01:23:49.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.525+0000 7fc895600640 1 -- 172.21.15.120:0/2876285970 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc88c0047e0 con 0x7fc8980b97c0 2024-03-24T01:23:49.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.525+0000 7fc89f000640 1 -- 172.21.15.120:0/2876285970 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc898157c60 con 0x7fc8980b97c0 2024-03-24T01:23:49.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.525+0000 7fc895600640 1 -- 172.21.15.120:0/2876285970 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc88c004bd0 con 0x7fc8980b97c0 2024-03-24T01:23:49.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.525+0000 7fc89f000640 1 -- 172.21.15.120:0/2876285970 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc898158220 con 0x7fc8980b97c0 2024-03-24T01:23:49.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.525+0000 7fc895600640 1 -- 172.21.15.120:0/2876285970 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fc88c01a020 con 0x7fc8980b97c0 2024-03-24T01:23:49.527 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.525+0000 7fc895600640 1 -- 172.21.15.120:0/2876285970 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7fc88c013df0 con 0x7fc8980b97c0 2024-03-24T01:23:49.528 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.526+0000 7fc89f000640 1 -- 172.21.15.120:0/2876285970 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc89804f400 con 0x7fc8980b97c0 2024-03-24T01:23:49.528 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.528+0000 7fc895600640 1 -- 172.21.15.120:0/2876285970 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fc88c021070 con 0x7fc8980b97c0 2024-03-24T01:23:49.598 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-24T01:23:49.597+0000 7f1993fdb040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T01:23:49.606 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.606+0000 7fc89f000640 1 -- 172.21.15.120:0/2876285970 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd new", "uuid": "424703a0-ba08-4d01-9d48-b38eabdde4f1", "id": 7} v 0) v1 -- 0x7fc89806cde0 con 0x7fc8980b97c0 2024-03-24T01:23:49.613 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.612+0000 7fc895600640 1 -- 172.21.15.120:0/2876285970 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "424703a0-ba08-4d01-9d48-b38eabdde4f1", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc8981f4790 con 0x7fc8980b97c0 2024-03-24T01:23:49.613 INFO:teuthology.orchestra.run.smithi120.stdout:7 2024-03-24T01:23:49.614 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.613+0000 7fc89f000640 1 -- 172.21.15.120:0/2876285970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8980b97c0 msgr2=0x7fc8981f4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:49.614 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.613+0000 7fc89f000640 1 --2- 172.21.15.120:0/2876285970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8980b97c0 0x7fc8981f4250 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc88c00c390 tx=0x7fc88c00c850 comp rx=0 tx=0).stop 2024-03-24T01:23:49.614 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.613+0000 7fc89f000640 1 -- 172.21.15.120:0/2876285970 shutdown_connections 2024-03-24T01:23:49.614 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.613+0000 7fc89f000640 1 --2- 172.21.15.120:0/2876285970 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8980d2090 0x7fc8981516a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:49.614 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.613+0000 7fc89f000640 1 --2- 172.21.15.120:0/2876285970 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc89815dc90 0x7fc898155630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:49.615 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.613+0000 7fc89f000640 1 --2- 172.21.15.120:0/2876285970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8980b97c0 0x7fc8981f4250 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:49.615 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.613+0000 7fc89f000640 1 -- 172.21.15.120:0/2876285970 >> 172.21.15.120:0/2876285970 conn(0x7fc898072be0 msgr2=0x7fc898072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:23:49.615 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.613+0000 7fc89f000640 1 -- 172.21.15.120:0/2876285970 shutdown_connections 2024-03-24T01:23:49.615 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:23:49.614+0000 7fc89f000640 1 -- 172.21.15.120:0/2876285970 wait complete. 2024-03-24T01:23:49.624 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-24T01:23:49.624+0000 7fd6d4a20040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T01:23:49.630 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-24T01:23:49.631 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-24T01:23:49.638 INFO:tasks.ceph.osd.0:Started 2024-03-24T01:23:49.638 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-24T01:23:49.638 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-24T01:23:49.645 INFO:tasks.ceph.osd.1:Started 2024-03-24T01:23:49.645 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-24T01:23:49.645 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-24T01:23:49.651 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-24T01:23:49.650+0000 7f1993fdb040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T01:23:49.652 INFO:tasks.ceph.osd.2:Started 2024-03-24T01:23:49.652 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-24T01:23:49.653 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-24T01:23:49.659 INFO:tasks.ceph.osd.3:Started 2024-03-24T01:23:49.659 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-24T01:23:49.659 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-24T01:23:49.665 INFO:tasks.ceph.osd.4:Started 2024-03-24T01:23:49.665 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-24T01:23:49.665 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-24T01:23:49.670 INFO:tasks.ceph.osd.5:Started 2024-03-24T01:23:49.670 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-24T01:23:49.673 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-24T01:23:49.678 INFO:tasks.ceph.osd.6:Started 2024-03-24T01:23:49.679 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-24T01:23:49.679 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-24T01:23:49.684 INFO:tasks.ceph.osd.7:Started 2024-03-24T01:23:49.685 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:23:49.690 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-24T01:23:49.687+0000 7fd6d4a20040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T01:23:49.867 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-24T01:23:49.867+0000 7fd6d4a20040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T01:23:49.882 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-24T01:23:49.881+0000 7f1993fdb040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T01:23:49.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.912+0000 7f34d4a00640 1 Processor -- start 2024-03-24T01:23:49.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.913+0000 7f34d4a00640 1 -- start start 2024-03-24T01:23:49.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.914+0000 7f34d4a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f34d01603e0 0x7f34d006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:49.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.914+0000 7f34d4a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f34d0145430 0x7f34d006b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:49.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.914+0000 7f34d4a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d00cd490 con 0x7f34d00cc150 2024-03-24T01:23:49.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.914+0000 7f34d4a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34d00cd270 con 0x7f34d01603e0 2024-03-24T01:23:49.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.914+0000 7f34d4a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34d015f900 con 0x7f34d0145430 2024-03-24T01:23:49.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.914+0000 7f34cde00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f34d00cc150 0x7f34d00ccb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:58304/0 (socket says 172.21.15.90:58304) 2024-03-24T01:23:49.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.914+0000 7f34cde00640 1 -- 172.21.15.90:0/1240511910 learned_addr learned my addr 172.21.15.90:0/1240511910 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:23:49.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.914+0000 7f34cd400640 1 --2- 172.21.15.90:0/1240511910 >> v2:172.21.15.90:3300/0 conn(0x7f34d01603e0 0x7f34d006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:49.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.914+0000 7f34ce800640 1 --2- 172.21.15.90:0/1240511910 >> v2:172.21.15.120:3300/0 conn(0x7f34d0145430 0x7f34d006b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:49.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.914+0000 7f34cca00640 1 -- 172.21.15.90:0/1240511910 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 77367517 0 0) 0x7f34d00cd490 con 0x7f34d00cc150 2024-03-24T01:23:49.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.915+0000 7f34cca00640 1 -- 172.21.15.90:0/1240511910 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34b0003520 con 0x7f34d00cc150 2024-03-24T01:23:49.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.915+0000 7f34cd400640 1 -- 172.21.15.90:0/1240511910 >> v2:172.21.15.120:3300/0 conn(0x7f34d0145430 msgr2=0x7f34d006b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:23:49.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.915+0000 7f34cd400640 1 --2- 172.21.15.90:0/1240511910 >> v2:172.21.15.120:3300/0 conn(0x7f34d0145430 0x7f34d006b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:49.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.915+0000 7f34cd400640 1 -- 172.21.15.90:0/1240511910 >> v1:172.21.15.90:6789/0 conn(0x7f34d00cc150 legacy=0x7f34d00ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:49.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.915+0000 7f34cd400640 1 -- 172.21.15.90:0/1240511910 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d015fbe0 con 0x7f34d01603e0 2024-03-24T01:23:49.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.915+0000 7f34ce800640 1 --2- 172.21.15.90:0/1240511910 >> v2:172.21.15.120:3300/0 conn(0x7f34d0145430 0x7f34d006b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T01:23:49.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.916+0000 7f34cd400640 1 --2- 172.21.15.90:0/1240511910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34d01603e0 0x7f34d006b0c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f34d006fc10 tx=0x7f34b802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83937a8c005962fa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:23:49.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.916+0000 7f34cca00640 1 -- 172.21.15.90:0/1240511910 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b8035020 con 0x7f34d01603e0 2024-03-24T01:23:49.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.916+0000 7f34cca00640 1 -- 172.21.15.90:0/1240511910 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34b8031830 con 0x7f34d01603e0 2024-03-24T01:23:49.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.916+0000 7f34cca00640 1 -- 172.21.15.90:0/1240511910 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b8031be0 con 0x7f34d01603e0 2024-03-24T01:23:49.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.917+0000 7f34d4a00640 1 -- 172.21.15.90:0/1240511910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34d01603e0 msgr2=0x7f34d006b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:49.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.917+0000 7f34d4a00640 1 --2- 172.21.15.90:0/1240511910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34d01603e0 0x7f34d006b0c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f34d006fc10 tx=0x7f34b802b8d0 comp rx=0 tx=0).stop 2024-03-24T01:23:49.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.917+0000 7f34d4a00640 1 -- 172.21.15.90:0/1240511910 shutdown_connections 2024-03-24T01:23:49.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.917+0000 7f34d4a00640 1 --2- 172.21.15.90:0/1240511910 >> v2:172.21.15.120:3300/0 conn(0x7f34d0145430 0x7f34d006b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:49.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.917+0000 7f34d4a00640 1 --2- 172.21.15.90:0/1240511910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34d01603e0 0x7f34d006b0c0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:49.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.917+0000 7f34d4a00640 1 -- 172.21.15.90:0/1240511910 >> 172.21.15.90:0/1240511910 conn(0x7f34d00b1770 msgr2=0x7f34d00b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:23:49.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.917+0000 7f34d4a00640 1 -- 172.21.15.90:0/1240511910 shutdown_connections 2024-03-24T01:23:49.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.917+0000 7f34d4a00640 1 -- 172.21.15.90:0/1240511910 wait complete. 2024-03-24T01:23:49.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.918+0000 7f34d4a00640 1 Processor -- start 2024-03-24T01:23:49.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.919+0000 7f34d4a00640 1 -- start start 2024-03-24T01:23:49.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.919+0000 7f34d4a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34d00cc150 0x7f34d01f4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:49.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.919+0000 7f34d4a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34d0145430 0x7f34d006e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:49.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.919+0000 7f34d4a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34d01603e0 0x7f34d006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:49.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.919+0000 7f34d4a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d00b38d0 con 0x7f34d01603e0 2024-03-24T01:23:49.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.919+0000 7f34d4a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d00b3750 con 0x7f34d0145430 2024-03-24T01:23:49.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.919+0000 7f34d4a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34d00b3a50 con 0x7f34d00cc150 2024-03-24T01:23:49.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.920+0000 7f34ce800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34d01603e0 0x7f34d006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:49.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.920+0000 7f34ce800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34d01603e0 0x7f34d006e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35276/0 (socket says 172.21.15.90:35276) 2024-03-24T01:23:49.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.920+0000 7f34cde00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34d00cc150 0x7f34d01f4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:49.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.920+0000 7f34ce800640 1 -- 172.21.15.90:0/376905068 learned_addr learned my addr 172.21.15.90:0/376905068 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:23:49.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.920+0000 7f34cd400640 1 --2- 172.21.15.90:0/376905068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34d0145430 0x7f34d006e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:49.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.920+0000 7f34ce800640 1 -- 172.21.15.90:0/376905068 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34d00cc150 msgr2=0x7f34d01f4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:49.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.920+0000 7f34ce800640 1 --2- 172.21.15.90:0/376905068 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34d00cc150 0x7f34d01f4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:49.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.920+0000 7f34ce800640 1 -- 172.21.15.90:0/376905068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34d0145430 msgr2=0x7f34d006e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:49.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.920+0000 7f34ce800640 1 --2- 172.21.15.90:0/376905068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34d0145430 0x7f34d006e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:49.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.920+0000 7f34ce800640 1 -- 172.21.15.90:0/376905068 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d00ad5d0 con 0x7f34d01603e0 2024-03-24T01:23:49.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.920+0000 7f34ce800640 1 --2- 172.21.15.90:0/376905068 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34d01603e0 0x7f34d006e9a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f34c800efc0 tx=0x7f34c8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:23:49.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.921+0000 7f34c6000640 1 -- 172.21.15.90:0/376905068 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c8018030 con 0x7f34d01603e0 2024-03-24T01:23:49.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.921+0000 7f34c6000640 1 -- 172.21.15.90:0/376905068 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34c800cde0 con 0x7f34d01603e0 2024-03-24T01:23:49.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.921+0000 7f34c6000640 1 -- 172.21.15.90:0/376905068 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c8003bb0 con 0x7f34d01603e0 2024-03-24T01:23:49.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.921+0000 7f34d4a00640 1 -- 172.21.15.90:0/376905068 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d00ad8b0 con 0x7f34d01603e0 2024-03-24T01:23:49.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.921+0000 7f34d4a00640 1 -- 172.21.15.90:0/376905068 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d00ade70 con 0x7f34d01603e0 2024-03-24T01:23:49.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.921+0000 7f34c6000640 1 -- 172.21.15.90:0/376905068 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f34c801b070 con 0x7f34d01603e0 2024-03-24T01:23:49.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.921+0000 7f34c6000640 1 -- 172.21.15.90:0/376905068 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f34c8014b50 con 0x7f34d01603e0 2024-03-24T01:23:49.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.922+0000 7f34d4a00640 1 -- 172.21.15.90:0/376905068 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34d0068060 con 0x7f34d01603e0 2024-03-24T01:23:49.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.925+0000 7f34c6000640 1 -- 172.21.15.90:0/376905068 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f34c8010040 con 0x7f34d01603e0 2024-03-24T01:23:49.985 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-24T01:23:49.985+0000 7fd6d4a20040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T01:23:49.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.990+0000 7f34d4a00640 1 -- 172.21.15.90:0/376905068 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34d00b3220 con 0x7f34d01603e0 2024-03-24T01:23:49.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.990+0000 7f34c6000640 1 -- 172.21.15.90:0/376905068 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f34c8014480 con 0x7f34d01603e0 2024-03-24T01:23:49.991 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:23:49.991 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":10,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:23:49.609376+0000","last_up_change":"0.000000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:23:49.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.991+0000 7f34d4a00640 1 -- 172.21.15.90:0/376905068 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34d01603e0 msgr2=0x7f34d006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:49.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.991+0000 7f34d4a00640 1 --2- 172.21.15.90:0/376905068 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34d01603e0 0x7f34d006e9a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f34c800efc0 tx=0x7f34c8009f90 comp rx=0 tx=0).stop 2024-03-24T01:23:49.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.992+0000 7f34d4a00640 1 -- 172.21.15.90:0/376905068 shutdown_connections 2024-03-24T01:23:49.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.992+0000 7f34d4a00640 1 --2- 172.21.15.90:0/376905068 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34d01603e0 0x7f34d006e9a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:49.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.992+0000 7f34d4a00640 1 --2- 172.21.15.90:0/376905068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34d0145430 0x7f34d006e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:49.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.992+0000 7f34d4a00640 1 --2- 172.21.15.90:0/376905068 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f34d00cc150 0x7f34d01f4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:49.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.992+0000 7f34d4a00640 1 -- 172.21.15.90:0/376905068 >> 172.21.15.90:0/376905068 conn(0x7f34d00b1770 msgr2=0x7f34d00b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:23:49.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.992+0000 7f34d4a00640 1 -- 172.21.15.90:0/376905068 shutdown_connections 2024-03-24T01:23:49.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:23:49.992+0000 7f34d4a00640 1 -- 172.21.15.90:0/376905068 wait complete. 2024-03-24T01:23:50.009 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-24T01:23:50.009 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-24T01:23:50.166 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-24T01:23:50.165+0000 7fd6d4a20040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T01:23:50.215 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-24T01:23:50.215+0000 7fd6d4a20040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T01:23:50.321 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-03-24T01:23:50.321+0000 7f3212437640 -1 Falling back to public interface 2024-03-24T01:23:50.361 INFO:tasks.ceph.osd.4.smithi120.stderr:2024-03-24T01:23:50.358+0000 7f4d6faae640 -1 Falling back to public interface 2024-03-24T01:23:50.364 INFO:tasks.ceph.osd.7.smithi120.stderr:2024-03-24T01:23:50.363+0000 7fe72d21f640 -1 Falling back to public interface 2024-03-24T01:23:50.369 INFO:tasks.ceph.osd.3.smithi090.stderr:2024-03-24T01:23:50.369+0000 7fc7f9bfa640 -1 Falling back to public interface 2024-03-24T01:23:50.375 INFO:tasks.ceph.osd.6.smithi120.stderr:2024-03-24T01:23:50.375+0000 7fd681815640 -1 Falling back to public interface 2024-03-24T01:23:50.381 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-03-24T01:23:50.381+0000 7f9ad81bb640 -1 Falling back to public interface 2024-03-24T01:23:50.381 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-03-24T01:23:50.382+0000 7f1009bfd640 -1 Falling back to public interface 2024-03-24T01:23:50.389 INFO:tasks.ceph.osd.5.smithi120.stderr:2024-03-24T01:23:50.389+0000 7f7449b73640 -1 Falling back to public interface 2024-03-24T01:23:50.399 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-24T01:23:50.399+0000 7fd6d4a20040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T01:23:50.814 DEBUG:teuthology.orchestra.run.smithi090:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-24T01:23:50.945 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-24T01:23:50.945+0000 7f1993fdb040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T01:23:50.979 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.979+0000 7f37c5200640 1 Processor -- start 2024-03-24T01:23:50.979 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.979+0000 7f37c5200640 1 -- start start 2024-03-24T01:23:50.981 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.980+0000 7f37c5200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f37c006cc70 0x7f37c00cec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:50.981 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.980+0000 7f37c5200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37c00d3280 0x7f37c00cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:50.981 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.980+0000 7f37c5200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c00cf6f0 con 0x7f37c015e420 2024-03-24T01:23:50.981 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.980+0000 7f37c5200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37c00ce9e0 con 0x7f37c006cc70 2024-03-24T01:23:50.981 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.980+0000 7f37c5200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37c00d2220 con 0x7f37c00d3280 2024-03-24T01:23:50.981 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.980+0000 7f37bd400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f37c006cc70 0x7f37c00cec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:50.981 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.980+0000 7f37bd400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f37c006cc70 0x7f37c00cec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35288/0 (socket says 172.21.15.90:35288) 2024-03-24T01:23:50.981 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.980+0000 7f37bd400640 1 -- 172.21.15.90:0/1368981570 learned_addr learned my addr 172.21.15.90:0/1368981570 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:23:50.981 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.980+0000 7f37be800640 1 --2- 172.21.15.90:0/1368981570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37c00d3280 0x7f37c00cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:50.981 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.981+0000 7f37bd400640 1 -- 172.21.15.90:0/1368981570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37c00d3280 msgr2=0x7f37c00cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:23:50.982 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.981+0000 7f37bd400640 1 --2- 172.21.15.90:0/1368981570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37c00d3280 0x7f37c00cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:50.982 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.981+0000 7f37bd400640 1 -- 172.21.15.90:0/1368981570 >> v1:172.21.15.120:6789/0 conn(0x7f37c015e420 legacy=0x7f37c006c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:50.982 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.981+0000 7f37bd400640 1 -- 172.21.15.90:0/1368981570 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c00d24c0 con 0x7f37c006cc70 2024-03-24T01:23:50.982 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.981+0000 7f37be800640 1 --2- 172.21.15.90:0/1368981570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37c00d3280 0x7f37c00cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:23:50.983 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.982+0000 7f37bd400640 1 --2- 172.21.15.90:0/1368981570 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37c006cc70 0x7f37c00cec70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f37ac009870 tx=0x7f37ac02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c142e7c019f1943a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:23:50.983 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.982+0000 7f37bca00640 1 -- 172.21.15.90:0/1368981570 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ac02f840 con 0x7f37c006cc70 2024-03-24T01:23:50.983 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.982+0000 7f37bca00640 1 -- 172.21.15.90:0/1368981570 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37ac02f9e0 con 0x7f37c006cc70 2024-03-24T01:23:50.983 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.982+0000 7f37bca00640 1 -- 172.21.15.90:0/1368981570 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ac02fdb0 con 0x7f37c006cc70 2024-03-24T01:23:50.983 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.982+0000 7f37c5200640 1 -- 172.21.15.90:0/1368981570 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37c006cc70 msgr2=0x7f37c00cec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:50.983 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.982+0000 7f37c5200640 1 --2- 172.21.15.90:0/1368981570 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37c006cc70 0x7f37c00cec70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f37ac009870 tx=0x7f37ac02ba20 comp rx=0 tx=0).stop 2024-03-24T01:23:50.983 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.982+0000 7f37c5200640 1 -- 172.21.15.90:0/1368981570 shutdown_connections 2024-03-24T01:23:50.983 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.982+0000 7f37c5200640 1 --2- 172.21.15.90:0/1368981570 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37c00d3280 0x7f37c00cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:50.984 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.982+0000 7f37c5200640 1 --2- 172.21.15.90:0/1368981570 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37c006cc70 0x7f37c00cec70 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:50.984 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.982+0000 7f37c5200640 1 -- 172.21.15.90:0/1368981570 >> 172.21.15.90:0/1368981570 conn(0x7f37c00d06b0 msgr2=0x7f37c00d0ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:23:50.984 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.983+0000 7f37c5200640 1 -- 172.21.15.90:0/1368981570 shutdown_connections 2024-03-24T01:23:50.984 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.983+0000 7f37c5200640 1 -- 172.21.15.90:0/1368981570 wait complete. 2024-03-24T01:23:50.984 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.984+0000 7f37c5200640 1 Processor -- start 2024-03-24T01:23:50.984 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.984+0000 7f37c5200640 1 -- start start 2024-03-24T01:23:50.987 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.985+0000 7f37c5200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37c006cc70 0x7f37c015a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:50.987 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.985+0000 7f37c5200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37c00d3280 0x7f37c0160060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:50.987 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.985+0000 7f37c5200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37c015e420 0x7f37c00cd510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:50.987 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.985+0000 7f37c5200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c00d2d60 con 0x7f37c006cc70 2024-03-24T01:23:50.987 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.985+0000 7f37c5200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c00d2a80 con 0x7f37c015e420 2024-03-24T01:23:50.987 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.985+0000 7f37c5200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37c00b70d0 con 0x7f37c00d3280 2024-03-24T01:23:50.988 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.986+0000 7f37bde00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37c006cc70 0x7f37c015a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:50.988 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.986+0000 7f37bde00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37c006cc70 0x7f37c015a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35304/0 (socket says 172.21.15.90:35304) 2024-03-24T01:23:50.988 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.986+0000 7f37bde00640 1 -- 172.21.15.90:0/832586920 learned_addr learned my addr 172.21.15.90:0/832586920 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:23:50.988 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.986+0000 7f37be800640 1 --2- 172.21.15.90:0/832586920 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37c015e420 0x7f37c00cd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:50.988 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.986+0000 7f37bd400640 1 --2- 172.21.15.90:0/832586920 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37c00d3280 0x7f37c0160060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:50.988 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.986+0000 7f37bde00640 1 -- 172.21.15.90:0/832586920 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37c00d3280 msgr2=0x7f37c0160060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:50.988 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.986+0000 7f37bde00640 1 --2- 172.21.15.90:0/832586920 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37c00d3280 0x7f37c0160060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:50.988 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.986+0000 7f37bde00640 1 -- 172.21.15.90:0/832586920 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37c015e420 msgr2=0x7f37c00cd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:50.988 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.986+0000 7f37bde00640 1 --2- 172.21.15.90:0/832586920 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37c015e420 0x7f37c00cd510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:50.988 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.986+0000 7f37bde00640 1 -- 172.21.15.90:0/832586920 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c015afc0 con 0x7f37c006cc70 2024-03-24T01:23:50.988 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.986+0000 7f37bde00640 1 --2- 172.21.15.90:0/832586920 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37c006cc70 0x7f37c015a6c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f37b000c3f0 tx=0x7f37b000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:23:50.988 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.987+0000 7f37b6000640 1 -- 172.21.15.90:0/832586920 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b00052d0 con 0x7f37c006cc70 2024-03-24T01:23:50.988 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.987+0000 7f37b6000640 1 -- 172.21.15.90:0/832586920 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37b0005490 con 0x7f37c006cc70 2024-03-24T01:23:50.988 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.987+0000 7f37b6000640 1 -- 172.21.15.90:0/832586920 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b0007260 con 0x7f37c006cc70 2024-03-24T01:23:50.989 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.987+0000 7f37c5200640 1 -- 172.21.15.90:0/832586920 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37c015b2a0 con 0x7f37c006cc70 2024-03-24T01:23:50.989 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.987+0000 7f37c5200640 1 -- 172.21.15.90:0/832586920 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37c015b860 con 0x7f37c006cc70 2024-03-24T01:23:50.989 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.987+0000 7f37b6000640 1 -- 172.21.15.90:0/832586920 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f37b0007400 con 0x7f37c006cc70 2024-03-24T01:23:50.989 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.987+0000 7f37b6000640 1 -- 172.21.15.90:0/832586920 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f37b0003a00 con 0x7f37c006cc70 2024-03-24T01:23:50.989 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.988+0000 7f37c5200640 1 -- 172.21.15.90:0/832586920 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37c0074a70 con 0x7f37c006cc70 2024-03-24T01:23:50.990 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:50.990+0000 7f37b6000640 1 -- 172.21.15.90:0/832586920 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f37b00035f0 con 0x7f37c006cc70 2024-03-24T01:23:51.057 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:51.056+0000 7f37c5200640 1 -- 172.21.15.90:0/832586920 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37c00ce400 con 0x7f37c006cc70 2024-03-24T01:23:51.057 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:51.057+0000 7f37b6000640 1 -- 172.21.15.90:0/832586920 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f37b0003ce0 con 0x7f37c006cc70 2024-03-24T01:23:51.058 INFO:teuthology.misc.health.smithi090.stdout: 2024-03-24T01:23:51.058 INFO:teuthology.misc.health.smithi090.stdout:{"epoch":10,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:23:49.609376+0000","last_up_change":"0.000000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:23:51.059 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:51.058+0000 7f37c5200640 1 -- 172.21.15.90:0/832586920 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37c006cc70 msgr2=0x7f37c015a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:51.059 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:51.058+0000 7f37c5200640 1 --2- 172.21.15.90:0/832586920 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37c006cc70 0x7f37c015a6c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f37b000c3f0 tx=0x7f37b000c8b0 comp rx=0 tx=0).stop 2024-03-24T01:23:51.059 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:51.059+0000 7f37c5200640 1 -- 172.21.15.90:0/832586920 shutdown_connections 2024-03-24T01:23:51.059 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:51.059+0000 7f37c5200640 1 --2- 172.21.15.90:0/832586920 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37c015e420 0x7f37c00cd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:51.059 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:51.059+0000 7f37c5200640 1 --2- 172.21.15.90:0/832586920 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37c00d3280 0x7f37c0160060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:51.059 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:51.059+0000 7f37c5200640 1 --2- 172.21.15.90:0/832586920 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37c006cc70 0x7f37c015a6c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:51.059 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:51.059+0000 7f37c5200640 1 -- 172.21.15.90:0/832586920 >> 172.21.15.90:0/832586920 conn(0x7f37c00d06b0 msgr2=0x7f37c0162490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:23:51.060 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:51.059+0000 7f37c5200640 1 -- 172.21.15.90:0/832586920 shutdown_connections 2024-03-24T01:23:51.060 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:51.059+0000 7f37c5200640 1 -- 172.21.15.90:0/832586920 wait complete. 2024-03-24T01:23:51.072 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-24T01:23:51.261 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-24T01:23:51.260+0000 7f1993fdb040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T01:23:51.320 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-24T01:23:51.319+0000 7f1993fdb040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T01:23:51.429 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-24T01:23:51.428+0000 7fd6d4a20040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T01:23:51.450 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-24T01:23:51.449+0000 7f1993fdb040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T01:23:51.755 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-24T01:23:51.754+0000 7fd6d4a20040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T01:23:51.816 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-24T01:23:51.816+0000 7fd6d4a20040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T01:23:51.945 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-24T01:23:51.944+0000 7fd6d4a20040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T01:23:52.496 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-24T01:23:52.495+0000 7f1935000640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-24T01:23:52.496 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-24T01:23:52.496+0000 7f1935000640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-24T01:23:53.655 INFO:tasks.ceph.osd.4.smithi120.stderr:2024-03-24T01:23:53.654+0000 7f4d6faae640 -1 osd.4 0 log_to_monitors true 2024-03-24T01:23:53.655 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-03-24T01:23:53.655+0000 7f3212437640 -1 osd.0 0 log_to_monitors true 2024-03-24T01:23:53.680 INFO:tasks.ceph.osd.6.smithi120.stderr:2024-03-24T01:23:53.680+0000 7fd681815640 -1 osd.6 0 log_to_monitors true 2024-03-24T01:23:53.694 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-03-24T01:23:53.694+0000 7f9ad81bb640 -1 osd.1 0 log_to_monitors true 2024-03-24T01:23:53.717 INFO:tasks.ceph.osd.5.smithi120.stderr:2024-03-24T01:23:53.716+0000 7f7449b73640 -1 osd.5 0 log_to_monitors true 2024-03-24T01:23:53.929 INFO:tasks.ceph.osd.3.smithi090.stderr:2024-03-24T01:23:53.929+0000 7fc7f9bfa640 -1 osd.3 0 log_to_monitors true 2024-03-24T01:23:53.947 INFO:tasks.ceph.osd.7.smithi120.stderr:2024-03-24T01:23:53.947+0000 7fe72d21f640 -1 osd.7 0 log_to_monitors true 2024-03-24T01:23:53.978 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-03-24T01:23:53.978+0000 7f1009bfd640 -1 osd.2 0 log_to_monitors true 2024-03-24T01:23:55.808 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:23:55.813 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:23:57.832 INFO:tasks.ceph.osd.4.smithi120.stderr:2024-03-24T01:23:57.832+0000 7f4d6aa00640 -1 osd.4 0 waiting for initial osdmap 2024-03-24T01:23:57.841 INFO:tasks.ceph.osd.4.smithi120.stderr:2024-03-24T01:23:57.840+0000 7f4d60a00640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T01:23:57.879 DEBUG:teuthology.orchestra.run.smithi090:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-24T01:23:57.883 INFO:tasks.ceph.osd.6.smithi120.stderr:2024-03-24T01:23:57.883+0000 7fd67ca00640 -1 osd.6 0 waiting for initial osdmap 2024-03-24T01:23:57.883 INFO:tasks.ceph.osd.7.smithi120.stderr:2024-03-24T01:23:57.883+0000 7fe727c00640 -1 osd.7 0 waiting for initial osdmap 2024-03-24T01:23:57.888 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-03-24T01:23:57.882+0000 7f320d200640 -1 osd.0 0 waiting for initial osdmap 2024-03-24T01:23:57.889 INFO:tasks.ceph.osd.6.smithi120.stderr:2024-03-24T01:23:57.889+0000 7fd672000640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T01:23:57.891 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-03-24T01:23:57.891+0000 7f3203200640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T01:23:57.895 INFO:tasks.ceph.osd.7.smithi120.stderr:2024-03-24T01:23:57.894+0000 7fe71dc00640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T01:23:57.916 INFO:tasks.ceph.osd.5.smithi120.stderr:2024-03-24T01:23:57.916+0000 7f7445400640 -1 osd.5 0 waiting for initial osdmap 2024-03-24T01:23:57.921 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-03-24T01:23:57.921+0000 7f1005400640 -1 osd.2 0 waiting for initial osdmap 2024-03-24T01:23:57.924 INFO:tasks.ceph.osd.5.smithi120.stderr:2024-03-24T01:23:57.923+0000 7f743aa00640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T01:23:57.926 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-03-24T01:23:57.926+0000 7f0ffaa00640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T01:23:57.936 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-03-24T01:23:57.936+0000 7f9ad3000640 -1 osd.1 0 waiting for initial osdmap 2024-03-24T01:23:57.940 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-03-24T01:23:57.940+0000 7f9ac9000640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T01:23:57.983 INFO:tasks.ceph.osd.3.smithi090.stderr:2024-03-24T01:23:57.983+0000 7fc7f4a00640 -1 osd.3 0 waiting for initial osdmap 2024-03-24T01:23:57.987 INFO:tasks.ceph.osd.3.smithi090.stderr:2024-03-24T01:23:57.988+0000 7fc7eaa00640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T01:23:58.039 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.039+0000 7f654a400640 1 Processor -- start 2024-03-24T01:23:58.039 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.039+0000 7f654a400640 1 -- start start 2024-03-24T01:23:58.041 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.040+0000 7f654a400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f654415ab20 0x7f65440b4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:58.041 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.040+0000 7f654a400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65440b48d0 0x7f654406b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:58.041 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.040+0000 7f654a400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f654406ba40 con 0x7f654406b600 2024-03-24T01:23:58.041 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.040+0000 7f654a400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65440b4d10 con 0x7f654415ab20 2024-03-24T01:23:58.041 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.040+0000 7f654a400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f654415af00 con 0x7f65440b48d0 2024-03-24T01:23:58.041 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.040+0000 7f6543400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f654415ab20 0x7f65440b4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:58.041 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.040+0000 7f6543400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f654415ab20 0x7f65440b4390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35338/0 (socket says 172.21.15.90:35338) 2024-03-24T01:23:58.041 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.040+0000 7f6543e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f654406b600 0x7f654415e6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:58312/0 (socket says 172.21.15.90:58312) 2024-03-24T01:23:58.041 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.040+0000 7f6543400640 1 -- 172.21.15.90:0/2227454255 learned_addr learned my addr 172.21.15.90:0/2227454255 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:23:58.041 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.041+0000 7f653aa00640 1 --2- 172.21.15.90:0/2227454255 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65440b48d0 0x7f654406b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:58.042 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.041+0000 7f6543400640 1 -- 172.21.15.90:0/2227454255 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65440b48d0 msgr2=0x7f654406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:23:58.042 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.041+0000 7f6543400640 1 --2- 172.21.15.90:0/2227454255 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65440b48d0 0x7f654406b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:58.042 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.041+0000 7f6543400640 1 -- 172.21.15.90:0/2227454255 >> v1:172.21.15.90:6789/0 conn(0x7f654406b600 legacy=0x7f654415e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:58.042 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.041+0000 7f6543400640 1 -- 172.21.15.90:0/2227454255 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f654406da90 con 0x7f654415ab20 2024-03-24T01:23:58.043 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.042+0000 7f6543400640 1 --2- 172.21.15.90:0/2227454255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f654415ab20 0x7f65440b4390 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6534009870 tx=0x7f653402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec749356011b5d8a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:23:58.043 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.042+0000 7f6542a00640 1 -- 172.21.15.90:0/2227454255 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6534034070 con 0x7f654415ab20 2024-03-24T01:23:58.043 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.042+0000 7f6542a00640 1 -- 172.21.15.90:0/2227454255 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6534030840 con 0x7f654415ab20 2024-03-24T01:23:58.043 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.042+0000 7f6542a00640 1 -- 172.21.15.90:0/2227454255 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6534030c10 con 0x7f654415ab20 2024-03-24T01:23:58.043 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.042+0000 7f654a400640 1 -- 172.21.15.90:0/2227454255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f654415ab20 msgr2=0x7f65440b4390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:58.043 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.043+0000 7f654a400640 1 --2- 172.21.15.90:0/2227454255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f654415ab20 0x7f65440b4390 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6534009870 tx=0x7f653402b980 comp rx=0 tx=0).stop 2024-03-24T01:23:58.043 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.043+0000 7f654a400640 1 -- 172.21.15.90:0/2227454255 shutdown_connections 2024-03-24T01:23:58.043 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.043+0000 7f654a400640 1 --2- 172.21.15.90:0/2227454255 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f65440b48d0 0x7f654406b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:58.044 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.043+0000 7f654a400640 1 --2- 172.21.15.90:0/2227454255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f654415ab20 0x7f65440b4390 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:58.044 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.043+0000 7f654a400640 1 -- 172.21.15.90:0/2227454255 >> 172.21.15.90:0/2227454255 conn(0x7f65440b15c0 msgr2=0x7f65440b19a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:23:58.044 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.043+0000 7f654a400640 1 -- 172.21.15.90:0/2227454255 shutdown_connections 2024-03-24T01:23:58.044 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.043+0000 7f654a400640 1 -- 172.21.15.90:0/2227454255 wait complete. 2024-03-24T01:23:58.044 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.044+0000 7f654a400640 1 Processor -- start 2024-03-24T01:23:58.044 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.044+0000 7f654a400640 1 -- start start 2024-03-24T01:23:58.046 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.045+0000 7f654a400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f654406b600 0x7f65441f3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:58.047 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.045+0000 7f654a400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65440b48d0 0x7f65441419d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:58.047 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.045+0000 7f654a400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f654415ab20 0x7f65441f7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:58.047 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.045+0000 7f654a400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f654406e330 con 0x7f654406b600 2024-03-24T01:23:58.047 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.045+0000 7f654a400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f654406e050 con 0x7f65440b48d0 2024-03-24T01:23:58.047 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.045+0000 7f654a400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f654406e610 con 0x7f654415ab20 2024-03-24T01:23:58.047 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.045+0000 7f6543400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f654406b600 0x7f65441f3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:58.047 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.045+0000 7f6543400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f654406b600 0x7f65441f3bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35340/0 (socket says 172.21.15.90:35340) 2024-03-24T01:23:58.047 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.045+0000 7f6543400640 1 -- 172.21.15.90:0/3677166663 learned_addr learned my addr 172.21.15.90:0/3677166663 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:23:58.047 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.046+0000 7f653aa00640 1 --2- 172.21.15.90:0/3677166663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65440b48d0 0x7f65441419d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:58.047 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.046+0000 7f6543400640 1 -- 172.21.15.90:0/3677166663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f654415ab20 msgr2=0x7f65441f7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:58.047 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.046+0000 7f6543400640 1 --2- 172.21.15.90:0/3677166663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f654415ab20 0x7f65441f7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:58.047 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.046+0000 7f6543400640 1 -- 172.21.15.90:0/3677166663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65440b48d0 msgr2=0x7f65441419d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:58.047 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.046+0000 7f6543400640 1 --2- 172.21.15.90:0/3677166663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65440b48d0 0x7f65441419d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:58.048 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.046+0000 7f6543400640 1 -- 172.21.15.90:0/3677166663 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6544141f10 con 0x7f654406b600 2024-03-24T01:23:58.048 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.046+0000 7f6543400640 1 --2- 172.21.15.90:0/3677166663 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f654406b600 0x7f65441f3bf0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6534000c00 tx=0x7f65340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:23:58.048 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.046+0000 7f653be00640 1 -- 172.21.15.90:0/3677166663 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6534034040 con 0x7f654406b600 2024-03-24T01:23:58.048 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.046+0000 7f653be00640 1 -- 172.21.15.90:0/3677166663 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6534039d90 con 0x7f654406b600 2024-03-24T01:23:58.048 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.046+0000 7f653be00640 1 -- 172.21.15.90:0/3677166663 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6534043610 con 0x7f654406b600 2024-03-24T01:23:58.048 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.046+0000 7f654a400640 1 -- 172.21.15.90:0/3677166663 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65441f4200 con 0x7f654406b600 2024-03-24T01:23:58.048 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.047+0000 7f654a400640 1 -- 172.21.15.90:0/3677166663 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65441f47c0 con 0x7f654406b600 2024-03-24T01:23:58.048 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.047+0000 7f654a400640 1 -- 172.21.15.90:0/3677166663 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6544068530 con 0x7f654406b600 2024-03-24T01:23:58.049 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.048+0000 7f653be00640 1 -- 172.21.15.90:0/3677166663 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 4) v1 ==== 90080+0+0 (secure 0 0 0) 0x7f653404a060 con 0x7f654406b600 2024-03-24T01:23:58.049 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.049+0000 7f653be00640 1 --2- 172.21.15.90:0/3677166663 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f652006bb40 0x7f652006e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:23:58.049 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.049+0000 7f653be00640 1 -- 172.21.15.90:0/3677166663 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7f65340abd90 con 0x7f654406b600 2024-03-24T01:23:58.050 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.050+0000 7f653aa00640 1 --2- 172.21.15.90:0/3677166663 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f652006bb40 0x7f652006e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:23:58.051 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.051+0000 7f653aa00640 1 --2- 172.21.15.90:0/3677166663 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f652006bb40 0x7f652006e000 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f65441f4130 tx=0x7f652c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:23:58.054 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.054+0000 7f653be00640 1 -- 172.21.15.90:0/3677166663 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f653403e070 con 0x7f654406b600 2024-03-24T01:23:58.237 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.237+0000 7f654a400640 1 -- 172.21.15.90:0/3677166663 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f654415e430 con 0x7f654406b600 2024-03-24T01:23:58.239 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.238+0000 7f653be00640 1 -- 172.21.15.90:0/3677166663 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f653407eef0 con 0x7f654406b600 2024-03-24T01:23:58.239 INFO:teuthology.misc.health.smithi090.stdout: 2024-03-24T01:23:58.239 INFO:teuthology.misc.health.smithi090.stdout:{"epoch":12,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:23:55.497886+0000","last_up_change":"0.000000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:23:58.241 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.241+0000 7f654a400640 1 -- 172.21.15.90:0/3677166663 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f652006bb40 msgr2=0x7f652006e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:58.242 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.241+0000 7f654a400640 1 --2- 172.21.15.90:0/3677166663 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f652006bb40 0x7f652006e000 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f65441f4130 tx=0x7f652c008040 comp rx=0 tx=0).stop 2024-03-24T01:23:58.242 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.241+0000 7f654a400640 1 -- 172.21.15.90:0/3677166663 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f654406b600 msgr2=0x7f65441f3bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:23:58.242 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.241+0000 7f654a400640 1 --2- 172.21.15.90:0/3677166663 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f654406b600 0x7f65441f3bf0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6534000c00 tx=0x7f65340046c0 comp rx=0 tx=0).stop 2024-03-24T01:23:58.242 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.241+0000 7f654a400640 1 -- 172.21.15.90:0/3677166663 shutdown_connections 2024-03-24T01:23:58.242 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.241+0000 7f654a400640 1 --2- 172.21.15.90:0/3677166663 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f652006bb40 0x7f652006e000 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:58.242 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.241+0000 7f654a400640 1 --2- 172.21.15.90:0/3677166663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f654415ab20 0x7f65441f7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:58.242 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.241+0000 7f654a400640 1 --2- 172.21.15.90:0/3677166663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65440b48d0 0x7f65441419d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:58.242 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.241+0000 7f654a400640 1 --2- 172.21.15.90:0/3677166663 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f654406b600 0x7f65441f3bf0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:23:58.242 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.241+0000 7f654a400640 1 -- 172.21.15.90:0/3677166663 >> 172.21.15.90:0/3677166663 conn(0x7f65440b15c0 msgr2=0x7f65440ae8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:23:58.242 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.241+0000 7f654a400640 1 -- 172.21.15.90:0/3677166663 shutdown_connections 2024-03-24T01:23:58.242 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:23:58.241+0000 7f654a400640 1 -- 172.21.15.90:0/3677166663 wait complete. 2024-03-24T01:23:58.255 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-24T01:24:05.061 DEBUG:teuthology.orchestra.run.smithi090:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-24T01:24:05.247 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.246+0000 7f1dbfe00640 1 Processor -- start 2024-03-24T01:24:05.247 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.247+0000 7f1dbfe00640 1 -- start start 2024-03-24T01:24:05.249 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.248+0000 7f1dbfe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1dc015dbf0 0x7f1dc0162290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:05.249 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.248+0000 7f1dbfe00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1dc00d0ab0 0x7f1dc00d24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:05.249 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.248+0000 7f1dbfe00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dc01627d0 con 0x7f1dc00ce380 2024-03-24T01:24:05.249 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.248+0000 7f1dbfe00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dc00d1d80 con 0x7f1dc015dbf0 2024-03-24T01:24:05.249 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.248+0000 7f1dbfe00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dc00cec90 con 0x7f1dc00d0ab0 2024-03-24T01:24:05.249 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.248+0000 7f1dbea00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1dc015dbf0 0x7f1dc0162290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:05.249 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.248+0000 7f1dbea00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1dc015dbf0 0x7f1dc0162290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45794/0 (socket says 172.21.15.90:45794) 2024-03-24T01:24:05.250 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.248+0000 7f1dbea00640 1 -- 172.21.15.90:0/3998131841 learned_addr learned my addr 172.21.15.90:0/3998131841 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:05.250 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.248+0000 7f1dbf400640 1 --2- 172.21.15.90:0/3998131841 >> v2:172.21.15.120:3300/0 conn(0x7f1dc00d0ab0 0x7f1dc00d24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:05.250 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.249+0000 7f1dbea00640 1 -- 172.21.15.90:0/3998131841 >> v2:172.21.15.120:3300/0 conn(0x7f1dc00d0ab0 msgr2=0x7f1dc00d24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:05.250 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.249+0000 7f1dbea00640 1 --2- 172.21.15.90:0/3998131841 >> v2:172.21.15.120:3300/0 conn(0x7f1dc00d0ab0 0x7f1dc00d24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:05.250 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.249+0000 7f1dbea00640 1 -- 172.21.15.90:0/3998131841 >> v1:172.21.15.90:6789/0 conn(0x7f1dc00ce380 legacy=0x7f1dc00d03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:05.250 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.249+0000 7f1dbea00640 1 -- 172.21.15.90:0/3998131841 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dc00cef30 con 0x7f1dc015dbf0 2024-03-24T01:24:05.251 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.250+0000 7f1dbea00640 1 --2- 172.21.15.90:0/3998131841 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1dc015dbf0 0x7f1dc0162290 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1dc00ceb00 tx=0x7f1db002bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=738cf06085c09c2a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:05.251 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.250+0000 7f1dbd600640 1 -- 172.21.15.90:0/3998131841 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db002cb10 con 0x7f1dc015dbf0 2024-03-24T01:24:05.252 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.250+0000 7f1dbd600640 1 -- 172.21.15.90:0/3998131841 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1db002ccb0 con 0x7f1dc015dbf0 2024-03-24T01:24:05.252 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.250+0000 7f1dbd600640 1 -- 172.21.15.90:0/3998131841 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db002ce60 con 0x7f1dc015dbf0 2024-03-24T01:24:05.252 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.251+0000 7f1dbfe00640 1 -- 172.21.15.90:0/3998131841 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1dc015dbf0 msgr2=0x7f1dc0162290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:05.252 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.251+0000 7f1dbfe00640 1 --2- 172.21.15.90:0/3998131841 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1dc015dbf0 0x7f1dc0162290 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1dc00ceb00 tx=0x7f1db002bf70 comp rx=0 tx=0).stop 2024-03-24T01:24:05.252 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.251+0000 7f1dbfe00640 1 -- 172.21.15.90:0/3998131841 shutdown_connections 2024-03-24T01:24:05.252 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.251+0000 7f1dbfe00640 1 --2- 172.21.15.90:0/3998131841 >> v2:172.21.15.120:3300/0 conn(0x7f1dc00d0ab0 0x7f1dc00d24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:05.252 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.251+0000 7f1dbfe00640 1 --2- 172.21.15.90:0/3998131841 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1dc015dbf0 0x7f1dc0162290 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:05.252 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.251+0000 7f1dbfe00640 1 -- 172.21.15.90:0/3998131841 >> 172.21.15.90:0/3998131841 conn(0x7f1dc006cbb0 msgr2=0x7f1dc00727f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:05.252 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.251+0000 7f1dbfe00640 1 -- 172.21.15.90:0/3998131841 shutdown_connections 2024-03-24T01:24:05.252 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.251+0000 7f1dbfe00640 1 -- 172.21.15.90:0/3998131841 wait complete. 2024-03-24T01:24:05.253 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.253+0000 7f1dbfe00640 1 Processor -- start 2024-03-24T01:24:05.253 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.253+0000 7f1dbfe00640 1 -- start start 2024-03-24T01:24:05.255 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.254+0000 7f1dbfe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1dc00ce380 0x7f1dc014cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:05.255 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.254+0000 7f1dbfe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dc00d0ab0 0x7f1dc01f3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:05.256 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.254+0000 7f1dbfe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dc015dbf0 0x7f1dc015b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:05.256 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.254+0000 7f1dbfe00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc015dfd0 con 0x7f1dc00ce380 2024-03-24T01:24:05.256 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.254+0000 7f1dbfe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc00d1fd0 con 0x7f1dc015dbf0 2024-03-24T01:24:05.256 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.254+0000 7f1dbfe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc015e150 con 0x7f1dc00d0ab0 2024-03-24T01:24:05.256 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.254+0000 7f1dbea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1dc00ce380 0x7f1dc014cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:05.256 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.254+0000 7f1dbea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1dc00ce380 0x7f1dc014cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45800/0 (socket says 172.21.15.90:45800) 2024-03-24T01:24:05.256 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.254+0000 7f1dbea00640 1 -- 172.21.15.90:0/1257067810 learned_addr learned my addr 172.21.15.90:0/1257067810 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:05.256 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.254+0000 7f1dbea00640 1 -- 172.21.15.90:0/1257067810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dc00d0ab0 msgr2=0x7f1dc01f3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:05.256 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.254+0000 7f1dbe000640 1 --2- 172.21.15.90:0/1257067810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dc00d0ab0 0x7f1dc01f3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:05.256 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.254+0000 7f1dbea00640 1 --2- 172.21.15.90:0/1257067810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dc00d0ab0 0x7f1dc01f3c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:05.256 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.254+0000 7f1dbea00640 1 -- 172.21.15.90:0/1257067810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dc015dbf0 msgr2=0x7f1dc015b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:05.256 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.254+0000 7f1dbea00640 1 --2- 172.21.15.90:0/1257067810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dc015dbf0 0x7f1dc015b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:05.256 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.254+0000 7f1dbea00640 1 -- 172.21.15.90:0/1257067810 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dc0157640 con 0x7f1dc00ce380 2024-03-24T01:24:05.256 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.255+0000 7f1dbea00640 1 --2- 172.21.15.90:0/1257067810 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1dc00ce380 0x7f1dc014cfc0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1db0036040 tx=0x7f1db0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:05.257 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.255+0000 7f1db6a00640 1 -- 172.21.15.90:0/1257067810 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db0043020 con 0x7f1dc00ce380 2024-03-24T01:24:05.257 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.255+0000 7f1db6a00640 1 -- 172.21.15.90:0/1257067810 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1db003e450 con 0x7f1dc00ce380 2024-03-24T01:24:05.257 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.255+0000 7f1dbfe00640 1 -- 172.21.15.90:0/1257067810 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dc00c72a0 con 0x7f1dc00ce380 2024-03-24T01:24:05.257 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.255+0000 7f1db6a00640 1 -- 172.21.15.90:0/1257067810 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db002ce60 con 0x7f1dc00ce380 2024-03-24T01:24:05.257 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.255+0000 7f1dbfe00640 1 -- 172.21.15.90:0/1257067810 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dc00c7840 con 0x7f1dc00ce380 2024-03-24T01:24:05.257 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.256+0000 7f1dbfe00640 1 -- 172.21.15.90:0/1257067810 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dc0074a70 con 0x7f1dc00ce380 2024-03-24T01:24:05.257 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.257+0000 7f1db6a00640 1 -- 172.21.15.90:0/1257067810 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1db0046020 con 0x7f1dc00ce380 2024-03-24T01:24:05.258 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.258+0000 7f1db6a00640 1 --2- 172.21.15.90:0/1257067810 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1da006bbd0 0x7f1da006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:05.258 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.258+0000 7f1db6a00640 1 -- 172.21.15.90:0/1257067810 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1db00acdb0 con 0x7f1dc00ce380 2024-03-24T01:24:05.258 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.258+0000 7f1dbe000640 1 --2- 172.21.15.90:0/1257067810 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1da006bbd0 0x7f1da006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:05.259 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.259+0000 7f1dbe000640 1 --2- 172.21.15.90:0/1257067810 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1da006bbd0 0x7f1da006e090 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1dc00ceb40 tx=0x7f1dac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:05.263 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.263+0000 7f1db6a00640 1 -- 172.21.15.90:0/1257067810 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1db00b1070 con 0x7f1dc00ce380 2024-03-24T01:24:05.468 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.468+0000 7f1dbfe00640 1 -- 172.21.15.90:0/1257067810 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1dc00b5ca0 con 0x7f1dc00ce380 2024-03-24T01:24:05.470 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.470+0000 7f1db6a00640 1 -- 172.21.15.90:0/1257067810 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12860 (secure 0 0 0) 0x7f1db004d030 con 0x7f1dc00ce380 2024-03-24T01:24:05.470 INFO:teuthology.misc.health.smithi090.stdout: 2024-03-24T01:24:05.470 INFO:teuthology.misc.health.smithi090.stdout:{"epoch":16,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:24:01.886790+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:24:05.473 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.472+0000 7f1dbfe00640 1 -- 172.21.15.90:0/1257067810 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1da006bbd0 msgr2=0x7f1da006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:05.473 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.472+0000 7f1dbfe00640 1 --2- 172.21.15.90:0/1257067810 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1da006bbd0 0x7f1da006e090 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1dc00ceb40 tx=0x7f1dac009040 comp rx=0 tx=0).stop 2024-03-24T01:24:05.473 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.472+0000 7f1dbfe00640 1 -- 172.21.15.90:0/1257067810 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1dc00ce380 msgr2=0x7f1dc014cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:05.473 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.472+0000 7f1dbfe00640 1 --2- 172.21.15.90:0/1257067810 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1dc00ce380 0x7f1dc014cfc0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1db0036040 tx=0x7f1db0004a40 comp rx=0 tx=0).stop 2024-03-24T01:24:05.474 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.473+0000 7f1dbfe00640 1 -- 172.21.15.90:0/1257067810 shutdown_connections 2024-03-24T01:24:05.474 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.473+0000 7f1dbfe00640 1 --2- 172.21.15.90:0/1257067810 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1da006bbd0 0x7f1da006e090 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:05.474 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.473+0000 7f1dbfe00640 1 --2- 172.21.15.90:0/1257067810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dc015dbf0 0x7f1dc015b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:05.474 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.473+0000 7f1dbfe00640 1 --2- 172.21.15.90:0/1257067810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1dc00d0ab0 0x7f1dc01f3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:05.474 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.473+0000 7f1dbfe00640 1 --2- 172.21.15.90:0/1257067810 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1dc00ce380 0x7f1dc014cfc0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:05.474 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.473+0000 7f1dbfe00640 1 -- 172.21.15.90:0/1257067810 >> 172.21.15.90:0/1257067810 conn(0x7f1dc006cbb0 msgr2=0x7f1dc00b4d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:05.474 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.473+0000 7f1dbfe00640 1 -- 172.21.15.90:0/1257067810 shutdown_connections 2024-03-24T01:24:05.474 INFO:teuthology.misc.health.smithi090.stderr:2024-03-24T01:24:05.473+0000 7f1dbfe00640 1 -- 172.21.15.90:0/1257067810 wait complete. 2024-03-24T01:24:05.487 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-24T01:24:05.487 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-24T01:24:05.487 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-24T01:24:05.488 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-24T01:24:05.493 INFO:tasks.ceph.mds.a:Started 2024-03-24T01:24:05.493 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-24T01:24:05.493 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-24T01:24:05.497 INFO:tasks.ceph.mds.c:Started 2024-03-24T01:24:05.497 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-24T01:24:05.498 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-24T01:24:05.505 INFO:tasks.ceph.mds.b:Started 2024-03-24T01:24:05.505 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-24T01:24:05.505 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-24T01:24:05.512 INFO:tasks.ceph.mds.d:Started 2024-03-24T01:24:05.512 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-24T01:24:05.515 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:24:05.644 INFO:tasks.ceph.mds.a.smithi090.stdout:starting mds.a at 2024-03-24T01:24:05.644 INFO:tasks.ceph.mds.c.smithi090.stdout:starting mds.c at 2024-03-24T01:24:05.666 INFO:tasks.ceph.mds.d.smithi120.stdout:starting mds.d at 2024-03-24T01:24:05.667 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2024-03-24T01:24:05.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.697+0000 7face6800640 1 Processor -- start 2024-03-24T01:24:05.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.697+0000 7face6800640 1 -- start start 2024-03-24T01:24:05.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.698+0000 7face6800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7face015dba0 0x7face0162330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:05.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.698+0000 7face6800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face00ce4b0 con 0x7face00b5c40 2024-03-24T01:24:05.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.698+0000 7face6800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face00ce790 con 0x7face015df80 2024-03-24T01:24:05.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.698+0000 7face6800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7face0162870 con 0x7face015dba0 2024-03-24T01:24:05.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.699+0000 7face5400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7face015dba0 0x7face0162330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:05.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.699+0000 7face5400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7face015dba0 0x7face0162330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45840/0 (socket says 172.21.15.90:45840) 2024-03-24T01:24:05.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.699+0000 7face5e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7face00b5c40 0x7face00d1d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41700/0 (socket says 172.21.15.90:41700) 2024-03-24T01:24:05.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.699+0000 7face5400640 1 -- 172.21.15.90:0/241479949 learned_addr learned my addr 172.21.15.90:0/241479949 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:05.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.699+0000 7face5400640 1 -- 172.21.15.90:0/241479949 >> v1:172.21.15.120:6789/0 conn(0x7face015df80 legacy=0x7face00b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:05.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.699+0000 7face5400640 1 -- 172.21.15.90:0/241479949 >> v1:172.21.15.90:6789/0 conn(0x7face00b5c40 legacy=0x7face00d1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:05.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.699+0000 7face5400640 1 -- 172.21.15.90:0/241479949 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face00cea70 con 0x7face015dba0 2024-03-24T01:24:05.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.699+0000 7facdfe00640 1 -- 172.21.15.90:0/241479949 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1525083311 0 0) 0x7face00ce4b0 con 0x7face00b5c40 2024-03-24T01:24:05.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.700+0000 7face5400640 1 --2- 172.21.15.90:0/241479949 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7face015dba0 0x7face0162330 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7facd4009870 tx=0x7facd402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d781efda494f9235 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:05.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.700+0000 7facdfe00640 1 -- 172.21.15.90:0/241479949 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd4035020 con 0x7face015dba0 2024-03-24T01:24:05.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.701+0000 7facdfe00640 1 -- 172.21.15.90:0/241479949 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd4031870 con 0x7face015dba0 2024-03-24T01:24:05.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.701+0000 7facdfe00640 1 -- 172.21.15.90:0/241479949 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd4031c40 con 0x7face015dba0 2024-03-24T01:24:05.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.701+0000 7face6800640 1 -- 172.21.15.90:0/241479949 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7face015dba0 msgr2=0x7face0162330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:05.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.701+0000 7face6800640 1 --2- 172.21.15.90:0/241479949 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7face015dba0 0x7face0162330 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7facd4009870 tx=0x7facd402b910 comp rx=0 tx=0).stop 2024-03-24T01:24:05.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.701+0000 7face6800640 1 -- 172.21.15.90:0/241479949 shutdown_connections 2024-03-24T01:24:05.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.701+0000 7face6800640 1 --2- 172.21.15.90:0/241479949 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7face015dba0 0x7face0162330 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:05.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.701+0000 7face6800640 1 -- 172.21.15.90:0/241479949 >> 172.21.15.90:0/241479949 conn(0x7face00d0530 msgr2=0x7face00d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:05.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.701+0000 7face6800640 1 -- 172.21.15.90:0/241479949 shutdown_connections 2024-03-24T01:24:05.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.701+0000 7face6800640 1 -- 172.21.15.90:0/241479949 wait complete. 2024-03-24T01:24:05.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.703+0000 7face6800640 1 Processor -- start 2024-03-24T01:24:05.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.703+0000 7face6800640 1 -- start start 2024-03-24T01:24:05.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.704+0000 7face6800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7face00b5c40 0x7face014ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:05.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.704+0000 7face6800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7face015dba0 0x7face00d3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:05.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.704+0000 7face6800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7face015df80 0x7face00d3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:05.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.704+0000 7face6800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face00b2ae0 con 0x7face015df80 2024-03-24T01:24:05.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.704+0000 7face6800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face00b2960 con 0x7face00b5c40 2024-03-24T01:24:05.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.704+0000 7face6800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7face00b2c60 con 0x7face015dba0 2024-03-24T01:24:05.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.704+0000 7face5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7face015df80 0x7face00d3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:05.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.704+0000 7face5400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7face00b5c40 0x7face014ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:05.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.704+0000 7face5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7face015df80 0x7face00d3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45848/0 (socket says 172.21.15.90:45848) 2024-03-24T01:24:05.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.704+0000 7face5e00640 1 -- 172.21.15.90:0/4106375004 learned_addr learned my addr 172.21.15.90:0/4106375004 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:05.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.704+0000 7face4a00640 1 --2- 172.21.15.90:0/4106375004 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7face015dba0 0x7face00d3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:05.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.705+0000 7face5e00640 1 -- 172.21.15.90:0/4106375004 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7face015dba0 msgr2=0x7face00d3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:05.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.705+0000 7face5e00640 1 --2- 172.21.15.90:0/4106375004 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7face015dba0 0x7face00d3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:05.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.705+0000 7face5e00640 1 -- 172.21.15.90:0/4106375004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7face00b5c40 msgr2=0x7face014ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:05.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.705+0000 7face5e00640 1 --2- 172.21.15.90:0/4106375004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7face00b5c40 0x7face014ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:05.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.705+0000 7face5e00640 1 -- 172.21.15.90:0/4106375004 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face00c8540 con 0x7face015df80 2024-03-24T01:24:05.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.705+0000 7face5400640 1 --2- 172.21.15.90:0/4106375004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7face00b5c40 0x7face014ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T01:24:05.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.705+0000 7face5e00640 1 --2- 172.21.15.90:0/4106375004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7face015df80 0x7face00d3f80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7facd800c700 tx=0x7facd800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:05.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.705+0000 7facdd600640 1 -- 172.21.15.90:0/4106375004 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd8018070 con 0x7face015df80 2024-03-24T01:24:05.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.705+0000 7facdd600640 1 -- 172.21.15.90:0/4106375004 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd800ce10 con 0x7face015df80 2024-03-24T01:24:05.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.705+0000 7facdd600640 1 -- 172.21.15.90:0/4106375004 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd800a880 con 0x7face015df80 2024-03-24T01:24:05.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.705+0000 7face6800640 1 -- 172.21.15.90:0/4106375004 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face00c8820 con 0x7face015df80 2024-03-24T01:24:05.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.705+0000 7face6800640 1 -- 172.21.15.90:0/4106375004 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face00c8de0 con 0x7face015df80 2024-03-24T01:24:05.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.707+0000 7face6800640 1 -- 172.21.15.90:0/4106375004 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7face004f400 con 0x7face015df80 2024-03-24T01:24:05.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.707+0000 7facdd600640 1 -- 172.21.15.90:0/4106375004 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7facd800aa20 con 0x7face015df80 2024-03-24T01:24:05.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.708+0000 7facdd600640 1 --2- 172.21.15.90:0/4106375004 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7facc006bbd0 0x7facc006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:05.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.708+0000 7facdd600640 1 -- 172.21.15.90:0/4106375004 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7facd8089d10 con 0x7face015df80 2024-03-24T01:24:05.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.709+0000 7face5400640 1 --2- 172.21.15.90:0/4106375004 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7facc006bbd0 0x7facc006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:05.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.710+0000 7face5400640 1 --2- 172.21.15.90:0/4106375004 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7facc006bbd0 0x7facc006e090 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7facd4036000 tx=0x7facd4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:05.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.713+0000 7facdd600640 1 -- 172.21.15.90:0/4106375004 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7facd808f020 con 0x7face015df80 2024-03-24T01:24:05.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.908+0000 7face6800640 1 -- 172.21.15.90:0/4106375004 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7face00b64b0 con 0x7face015df80 2024-03-24T01:24:05.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.910+0000 7facdd600640 1 -- 172.21.15.90:0/4106375004 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12860 (secure 0 0 0) 0x7facd8058460 con 0x7face015df80 2024-03-24T01:24:05.911 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:24:05.911 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":16,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:24:01.886790+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:24:05.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.912+0000 7face6800640 1 -- 172.21.15.90:0/4106375004 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7facc006bbd0 msgr2=0x7facc006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:05.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.912+0000 7face6800640 1 --2- 172.21.15.90:0/4106375004 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7facc006bbd0 0x7facc006e090 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7facd4036000 tx=0x7facd4008030 comp rx=0 tx=0).stop 2024-03-24T01:24:05.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.912+0000 7face6800640 1 -- 172.21.15.90:0/4106375004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7face015df80 msgr2=0x7face00d3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:05.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.912+0000 7face6800640 1 --2- 172.21.15.90:0/4106375004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7face015df80 0x7face00d3f80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7facd800c700 tx=0x7facd800cbc0 comp rx=0 tx=0).stop 2024-03-24T01:24:05.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.912+0000 7face6800640 1 -- 172.21.15.90:0/4106375004 shutdown_connections 2024-03-24T01:24:05.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.912+0000 7face6800640 1 --2- 172.21.15.90:0/4106375004 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7facc006bbd0 0x7facc006e090 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:05.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.912+0000 7face6800640 1 --2- 172.21.15.90:0/4106375004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7face015df80 0x7face00d3f80 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:05.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.912+0000 7face6800640 1 --2- 172.21.15.90:0/4106375004 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7face015dba0 0x7face00d3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:05.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.912+0000 7face6800640 1 --2- 172.21.15.90:0/4106375004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7face00b5c40 0x7face014ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:05.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.912+0000 7face6800640 1 -- 172.21.15.90:0/4106375004 >> 172.21.15.90:0/4106375004 conn(0x7face00d0530 msgr2=0x7face00b7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:05.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.913+0000 7face6800640 1 -- 172.21.15.90:0/4106375004 shutdown_connections 2024-03-24T01:24:05.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:05.913+0000 7face6800640 1 -- 172.21.15.90:0/4106375004 wait complete. 2024-03-24T01:24:05.930 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T01:24:05.930 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:24:06.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.129+0000 7f20dc400640 1 Processor -- start 2024-03-24T01:24:06.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.130+0000 7f20dc400640 1 -- start start 2024-03-24T01:24:06.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.131+0000 7f20dc400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f20d4159620 0x7f20d406b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:06.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.131+0000 7f20dc400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20d406b600 0x7f20d40cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:06.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.131+0000 7f20dc400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20d406bca0 con 0x7f20d40cef80 2024-03-24T01:24:06.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.131+0000 7f20dc400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20d40b4470 con 0x7f20d4159620 2024-03-24T01:24:06.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.131+0000 7f20dc400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20d406ba60 con 0x7f20d406b600 2024-03-24T01:24:06.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.132+0000 7f20d9a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f20d4159620 0x7f20d406b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:06.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.132+0000 7f20d9a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f20d4159620 0x7f20d406b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45856/0 (socket says 172.21.15.90:45856) 2024-03-24T01:24:06.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.132+0000 7f20d9a00640 1 -- 172.21.15.90:0/520963453 learned_addr learned my addr 172.21.15.90:0/520963453 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:06.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.132+0000 7f20d9000640 1 --2- 172.21.15.90:0/520963453 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20d406b600 0x7f20d40cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:06.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.132+0000 7f20d9a00640 1 -- 172.21.15.90:0/520963453 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20d406b600 msgr2=0x7f20d40cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:06.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.132+0000 7f20d9a00640 1 --2- 172.21.15.90:0/520963453 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20d406b600 0x7f20d40cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:06.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.132+0000 7f20d9a00640 1 -- 172.21.15.90:0/520963453 >> v1:172.21.15.120:6789/0 conn(0x7f20d40cef80 legacy=0x7f20d40ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:06.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.132+0000 7f20d9a00640 1 -- 172.21.15.90:0/520963453 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d40b36f0 con 0x7f20d4159620 2024-03-24T01:24:06.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.133+0000 7f20d9a00640 1 --2- 172.21.15.90:0/520963453 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20d4159620 0x7f20d406b0c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f20c8009870 tx=0x7f20c802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eda51d4d2fbca528 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:06.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.134+0000 7f20d3e00640 1 -- 172.21.15.90:0/520963453 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c80317c0 con 0x7f20d4159620 2024-03-24T01:24:06.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.134+0000 7f20d3e00640 1 -- 172.21.15.90:0/520963453 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20c8037070 con 0x7f20d4159620 2024-03-24T01:24:06.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.134+0000 7f20d3e00640 1 -- 172.21.15.90:0/520963453 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c8031b80 con 0x7f20d4159620 2024-03-24T01:24:06.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.134+0000 7f20dc400640 1 -- 172.21.15.90:0/520963453 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20d4159620 msgr2=0x7f20d406b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:06.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.134+0000 7f20dc400640 1 --2- 172.21.15.90:0/520963453 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20d4159620 0x7f20d406b0c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f20c8009870 tx=0x7f20c802b860 comp rx=0 tx=0).stop 2024-03-24T01:24:06.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.134+0000 7f20dc400640 1 -- 172.21.15.90:0/520963453 shutdown_connections 2024-03-24T01:24:06.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.134+0000 7f20dc400640 1 --2- 172.21.15.90:0/520963453 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20d406b600 0x7f20d40cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:06.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.134+0000 7f20dc400640 1 --2- 172.21.15.90:0/520963453 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20d4159620 0x7f20d406b0c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:06.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.134+0000 7f20dc400640 1 -- 172.21.15.90:0/520963453 >> 172.21.15.90:0/520963453 conn(0x7f20d40b1770 msgr2=0x7f20d40b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:06.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.135+0000 7f20dc400640 1 -- 172.21.15.90:0/520963453 shutdown_connections 2024-03-24T01:24:06.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.135+0000 7f20dc400640 1 -- 172.21.15.90:0/520963453 wait complete. 2024-03-24T01:24:06.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.136+0000 7f20dc400640 1 Processor -- start 2024-03-24T01:24:06.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.137+0000 7f20dc400640 1 -- start start 2024-03-24T01:24:06.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.138+0000 7f20dc400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20d406b600 0x7f20d4143460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:06.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.138+0000 7f20dc400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20d40cef80 0x7f20d41f3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:06.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.138+0000 7f20dc400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20d4159620 0x7f20d41568e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:06.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.138+0000 7f20dc400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d40b3f50 con 0x7f20d406b600 2024-03-24T01:24:06.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.138+0000 7f20dc400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d40b3c70 con 0x7f20d40cef80 2024-03-24T01:24:06.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.138+0000 7f20dc400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20d4159a00 con 0x7f20d4159620 2024-03-24T01:24:06.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.138+0000 7f20d9a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20d406b600 0x7f20d4143460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:06.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.138+0000 7f20d9a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20d406b600 0x7f20d4143460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45860/0 (socket says 172.21.15.90:45860) 2024-03-24T01:24:06.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.138+0000 7f20d9a00640 1 -- 172.21.15.90:0/246417796 learned_addr learned my addr 172.21.15.90:0/246417796 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:06.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.138+0000 7f20d9000640 1 --2- 172.21.15.90:0/246417796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20d40cef80 0x7f20d41f3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:06.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.138+0000 7f20d9a00640 1 -- 172.21.15.90:0/246417796 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20d4159620 msgr2=0x7f20d41568e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:06.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.138+0000 7f20da400640 1 --2- 172.21.15.90:0/246417796 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20d4159620 0x7f20d41568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:06.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.139+0000 7f20d9a00640 1 --2- 172.21.15.90:0/246417796 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20d4159620 0x7f20d41568e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:06.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.139+0000 7f20d9a00640 1 -- 172.21.15.90:0/246417796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20d40cef80 msgr2=0x7f20d41f3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:06.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.139+0000 7f20d9a00640 1 --2- 172.21.15.90:0/246417796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20d40cef80 0x7f20d41f3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:06.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.139+0000 7f20d9a00640 1 -- 172.21.15.90:0/246417796 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d41439a0 con 0x7f20d406b600 2024-03-24T01:24:06.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.139+0000 7f20d9000640 1 --2- 172.21.15.90:0/246417796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20d40cef80 0x7f20d41f3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:06.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.139+0000 7f20d9a00640 1 --2- 172.21.15.90:0/246417796 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20d406b600 0x7f20d4143460 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f20c802b8a0 tx=0x7f20c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:06.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.139+0000 7f20d1600640 1 -- 172.21.15.90:0/246417796 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c8032650 con 0x7f20d406b600 2024-03-24T01:24:06.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.139+0000 7f20d1600640 1 -- 172.21.15.90:0/246417796 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20c80327f0 con 0x7f20d406b600 2024-03-24T01:24:06.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.139+0000 7f20dc400640 1 -- 172.21.15.90:0/246417796 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20d41f44a0 con 0x7f20d406b600 2024-03-24T01:24:06.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.139+0000 7f20d1600640 1 -- 172.21.15.90:0/246417796 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c8032be0 con 0x7f20d406b600 2024-03-24T01:24:06.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.139+0000 7f20dc400640 1 -- 172.21.15.90:0/246417796 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20d41f4a60 con 0x7f20d406b600 2024-03-24T01:24:06.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.140+0000 7f20dc400640 1 -- 172.21.15.90:0/246417796 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20d404f400 con 0x7f20d406b600 2024-03-24T01:24:06.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.141+0000 7f20d1600640 1 -- 172.21.15.90:0/246417796 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f20c804c060 con 0x7f20d406b600 2024-03-24T01:24:06.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.142+0000 7f20d1600640 1 --2- 172.21.15.90:0/246417796 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f20b406bbd0 0x7f20b406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:06.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.142+0000 7f20d1600640 1 -- 172.21.15.90:0/246417796 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f20c80ac280 con 0x7f20d406b600 2024-03-24T01:24:06.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.143+0000 7f20d9000640 1 --2- 172.21.15.90:0/246417796 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f20b406bbd0 0x7f20b406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:06.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.144+0000 7f20d9000640 1 --2- 172.21.15.90:0/246417796 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f20b406bbd0 0x7f20b406e090 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f20d406f850 tx=0x7f20c0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:06.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.147+0000 7f20d1600640 1 -- 172.21.15.90:0/246417796 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f20c8040070 con 0x7f20d406b600 2024-03-24T01:24:06.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.345+0000 7f20dc400640 1 -- 172.21.15.90:0/246417796 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f20d406ddf0 con 0x7f20d406b600 2024-03-24T01:24:06.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.346+0000 7f20d1600640 1 -- 172.21.15.90:0/246417796 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f20c807b1d0 con 0x7f20d406b600 2024-03-24T01:24:06.347 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:24:06.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.349+0000 7f20dc400640 1 -- 172.21.15.90:0/246417796 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f20b406bbd0 msgr2=0x7f20b406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:06.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.349+0000 7f20dc400640 1 --2- 172.21.15.90:0/246417796 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f20b406bbd0 0x7f20b406e090 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f20d406f850 tx=0x7f20c0008040 comp rx=0 tx=0).stop 2024-03-24T01:24:06.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.349+0000 7f20dc400640 1 -- 172.21.15.90:0/246417796 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20d406b600 msgr2=0x7f20d4143460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:06.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.349+0000 7f20dc400640 1 --2- 172.21.15.90:0/246417796 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20d406b600 0x7f20d4143460 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f20c802b8a0 tx=0x7f20c80046c0 comp rx=0 tx=0).stop 2024-03-24T01:24:06.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.349+0000 7f20dc400640 1 -- 172.21.15.90:0/246417796 shutdown_connections 2024-03-24T01:24:06.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.349+0000 7f20dc400640 1 --2- 172.21.15.90:0/246417796 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f20b406bbd0 0x7f20b406e090 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:06.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.349+0000 7f20dc400640 1 --2- 172.21.15.90:0/246417796 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20d4159620 0x7f20d41568e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:06.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.349+0000 7f20dc400640 1 --2- 172.21.15.90:0/246417796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20d40cef80 0x7f20d41f3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:06.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.349+0000 7f20dc400640 1 --2- 172.21.15.90:0/246417796 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20d406b600 0x7f20d4143460 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:06.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.349+0000 7f20dc400640 1 -- 172.21.15.90:0/246417796 >> 172.21.15.90:0/246417796 conn(0x7f20d40b1770 msgr2=0x7f20d40cc4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:06.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.350+0000 7f20dc400640 1 -- 172.21.15.90:0/246417796 shutdown_connections 2024-03-24T01:24:06.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.350+0000 7f20dc400640 1 -- 172.21.15.90:0/246417796 wait complete. 2024-03-24T01:24:06.369 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:24:06.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.569+0000 7fe85ba00640 1 Processor -- start 2024-03-24T01:24:06.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.569+0000 7fe85ba00640 1 -- start start 2024-03-24T01:24:06.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.570+0000 7fe85ba00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe85406cbc0 0x7fe8540b24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:06.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.570+0000 7fe85ba00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe8540b2ab0 0x7fe85406cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:06.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.570+0000 7fe85ba00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe85406d4e0 con 0x7fe854099cd0 2024-03-24T01:24:06.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.570+0000 7fe85ba00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8540d0e20 con 0x7fe8540b2ab0 2024-03-24T01:24:06.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.570+0000 7fe85ba00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8540cd050 con 0x7fe85406cbc0 2024-03-24T01:24:06.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.571+0000 7fe853e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe8540b2ab0 0x7fe85406cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:06.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.571+0000 7fe859a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe854099cd0 0x7fe85409a160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41704/0 (socket says 172.21.15.90:41704) 2024-03-24T01:24:06.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.571+0000 7fe859a00640 1 -- 172.21.15.90:0/2373616030 learned_addr learned my addr 172.21.15.90:0/2373616030 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:06.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.571+0000 7fe859000640 1 --2- 172.21.15.90:0/2373616030 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe85406cbc0 0x7fe8540b24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:06.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.571+0000 7fe853e00640 1 -- 172.21.15.90:0/2373616030 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe85406cbc0 msgr2=0x7fe8540b24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:06.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.571+0000 7fe853e00640 1 --2- 172.21.15.90:0/2373616030 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe85406cbc0 0x7fe8540b24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:06.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.571+0000 7fe853e00640 1 -- 172.21.15.90:0/2373616030 >> v1:172.21.15.90:6789/0 conn(0x7fe854099cd0 legacy=0x7fe85409a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:06.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.571+0000 7fe853e00640 1 -- 172.21.15.90:0/2373616030 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe85406c5b0 con 0x7fe8540b2ab0 2024-03-24T01:24:06.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.571+0000 7fe859000640 1 --2- 172.21.15.90:0/2373616030 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe85406cbc0 0x7fe8540b24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:06.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.571+0000 7fe853400640 1 -- 172.21.15.90:0/2373616030 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1856094407 0 0) 0x7fe85406d4e0 con 0x7fe854099cd0 2024-03-24T01:24:06.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.572+0000 7fe853e00640 1 --2- 172.21.15.90:0/2373616030 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8540b2ab0 0x7fe85406cfa0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe8400097d0 tx=0x7fe84002b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dedc332f19f3ebf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:06.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.573+0000 7fe853400640 1 -- 172.21.15.90:0/2373616030 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe840031770 con 0x7fe8540b2ab0 2024-03-24T01:24:06.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.573+0000 7fe853400640 1 -- 172.21.15.90:0/2373616030 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe840037070 con 0x7fe8540b2ab0 2024-03-24T01:24:06.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.573+0000 7fe853400640 1 -- 172.21.15.90:0/2373616030 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe840031b30 con 0x7fe8540b2ab0 2024-03-24T01:24:06.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.573+0000 7fe85ba00640 1 -- 172.21.15.90:0/2373616030 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8540b2ab0 msgr2=0x7fe85406cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:06.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.573+0000 7fe85ba00640 1 --2- 172.21.15.90:0/2373616030 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8540b2ab0 0x7fe85406cfa0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe8400097d0 tx=0x7fe84002b810 comp rx=0 tx=0).stop 2024-03-24T01:24:06.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.573+0000 7fe85ba00640 1 -- 172.21.15.90:0/2373616030 shutdown_connections 2024-03-24T01:24:06.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.573+0000 7fe85ba00640 1 --2- 172.21.15.90:0/2373616030 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8540b2ab0 0x7fe85406cfa0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:06.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.573+0000 7fe85ba00640 1 --2- 172.21.15.90:0/2373616030 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe85406cbc0 0x7fe8540b24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:06.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.573+0000 7fe85ba00640 1 -- 172.21.15.90:0/2373616030 >> 172.21.15.90:0/2373616030 conn(0x7fe8540b34e0 msgr2=0x7fe8540b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:06.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.573+0000 7fe85ba00640 1 -- 172.21.15.90:0/2373616030 shutdown_connections 2024-03-24T01:24:06.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.573+0000 7fe85ba00640 1 -- 172.21.15.90:0/2373616030 wait complete. 2024-03-24T01:24:06.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.575+0000 7fe85ba00640 1 Processor -- start 2024-03-24T01:24:06.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.575+0000 7fe85ba00640 1 -- start start 2024-03-24T01:24:06.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.576+0000 7fe85ba00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe85406cbc0 0x7fe85414ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:06.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.576+0000 7fe85ba00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe854099cd0 0x7fe8540b1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:06.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.576+0000 7fe85ba00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8540b2ab0 0x7fe854152720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:06.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.576+0000 7fe85ba00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8540d0ab0 con 0x7fe85406cbc0 2024-03-24T01:24:06.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.576+0000 7fe85ba00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8540d07d0 con 0x7fe8540b2ab0 2024-03-24T01:24:06.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.576+0000 7fe85ba00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8540d12b0 con 0x7fe854099cd0 2024-03-24T01:24:06.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.576+0000 7fe859000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe85406cbc0 0x7fe85414ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:06.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.576+0000 7fe859000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe85406cbc0 0x7fe85414ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45876/0 (socket says 172.21.15.90:45876) 2024-03-24T01:24:06.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.576+0000 7fe859000640 1 -- 172.21.15.90:0/3549077955 learned_addr learned my addr 172.21.15.90:0/3549077955 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:06.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.576+0000 7fe853e00640 1 --2- 172.21.15.90:0/3549077955 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe854099cd0 0x7fe8540b1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:06.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.577+0000 7fe859000640 1 -- 172.21.15.90:0/3549077955 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe854099cd0 msgr2=0x7fe8540b1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:06.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.577+0000 7fe859a00640 1 --2- 172.21.15.90:0/3549077955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8540b2ab0 0x7fe854152720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:06.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.577+0000 7fe859000640 1 --2- 172.21.15.90:0/3549077955 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe854099cd0 0x7fe8540b1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:06.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.577+0000 7fe859000640 1 -- 172.21.15.90:0/3549077955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8540b2ab0 msgr2=0x7fe854152720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:06.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.577+0000 7fe859000640 1 --2- 172.21.15.90:0/3549077955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8540b2ab0 0x7fe854152720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:06.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.577+0000 7fe859000640 1 -- 172.21.15.90:0/3549077955 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe854152e70 con 0x7fe85406cbc0 2024-03-24T01:24:06.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.577+0000 7fe853e00640 1 --2- 172.21.15.90:0/3549077955 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe854099cd0 0x7fe8540b1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:06.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.577+0000 7fe859000640 1 --2- 172.21.15.90:0/3549077955 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe85406cbc0 0x7fe85414ff90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe84800c7f0 tx=0x7fe84800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:06.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.577+0000 7fe850c00640 1 -- 172.21.15.90:0/3549077955 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe848015070 con 0x7fe85406cbc0 2024-03-24T01:24:06.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.577+0000 7fe850c00640 1 -- 172.21.15.90:0/3549077955 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe848007820 con 0x7fe85406cbc0 2024-03-24T01:24:06.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.577+0000 7fe850c00640 1 -- 172.21.15.90:0/3549077955 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe848007bd0 con 0x7fe85406cbc0 2024-03-24T01:24:06.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.577+0000 7fe85ba00640 1 -- 172.21.15.90:0/3549077955 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe854153150 con 0x7fe85406cbc0 2024-03-24T01:24:06.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.577+0000 7fe85ba00640 1 -- 172.21.15.90:0/3549077955 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8540c5be0 con 0x7fe85406cbc0 2024-03-24T01:24:06.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.579+0000 7fe85ba00640 1 -- 172.21.15.90:0/3549077955 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe854153670 con 0x7fe85406cbc0 2024-03-24T01:24:06.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.579+0000 7fe850c00640 1 -- 172.21.15.90:0/3549077955 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe848018070 con 0x7fe85406cbc0 2024-03-24T01:24:06.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.580+0000 7fe850c00640 1 --2- 172.21.15.90:0/3549077955 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe83406bbd0 0x7fe83406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:06.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.580+0000 7fe850c00640 1 -- 172.21.15.90:0/3549077955 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe84808a8a0 con 0x7fe85406cbc0 2024-03-24T01:24:06.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.581+0000 7fe853e00640 1 --2- 172.21.15.90:0/3549077955 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe83406bbd0 0x7fe83406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:06.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.582+0000 7fe853e00640 1 --2- 172.21.15.90:0/3549077955 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe83406bbd0 0x7fe83406e090 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe840036000 tx=0x7fe840008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:06.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.584+0000 7fe850c00640 1 -- 172.21.15.90:0/3549077955 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe854153670 con 0x7fe85406cbc0 2024-03-24T01:24:06.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.824+0000 7fe85ba00640 1 -- 172.21.15.90:0/3549077955 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe854068060 con 0x7fe85406cbc0 2024-03-24T01:24:06.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.824+0000 7fe850c00640 1 -- 172.21.15.90:0/3549077955 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3688 (secure 0 0 0) 0x7fe8480596e0 con 0x7fe85406cbc0 2024-03-24T01:24:06.825 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:24:06.825 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/1898972903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":1898972903},{"type":"v1","addr":"172.21.15.120:6837","nonce":1898972903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6834/775759501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":775759501},{"type":"v1","addr":"172.21.15.90:6834","nonce":775759501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/43630575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":43630575},{"type":"v1","addr":"172.21.15.90:6835","nonce":43630575}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4257,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/963928022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":963928022},{"type":"v1","addr":"172.21.15.120:6836","nonce":963928022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-24T01:24:06.825 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 3 2024-03-24T01:24:06.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.827+0000 7fe85ba00640 1 -- 172.21.15.90:0/3549077955 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe83406bbd0 msgr2=0x7fe83406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:06.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.827+0000 7fe85ba00640 1 --2- 172.21.15.90:0/3549077955 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe83406bbd0 0x7fe83406e090 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe840036000 tx=0x7fe840008030 comp rx=0 tx=0).stop 2024-03-24T01:24:06.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.827+0000 7fe85ba00640 1 -- 172.21.15.90:0/3549077955 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe85406cbc0 msgr2=0x7fe85414ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:06.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.827+0000 7fe85ba00640 1 --2- 172.21.15.90:0/3549077955 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe85406cbc0 0x7fe85414ff90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe84800c7f0 tx=0x7fe84800ccb0 comp rx=0 tx=0).stop 2024-03-24T01:24:06.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.827+0000 7fe85ba00640 1 -- 172.21.15.90:0/3549077955 shutdown_connections 2024-03-24T01:24:06.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.827+0000 7fe85ba00640 1 --2- 172.21.15.90:0/3549077955 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe83406bbd0 0x7fe83406e090 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:06.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.827+0000 7fe85ba00640 1 --2- 172.21.15.90:0/3549077955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8540b2ab0 0x7fe854152720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:06.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.827+0000 7fe85ba00640 1 --2- 172.21.15.90:0/3549077955 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe854099cd0 0x7fe8540b1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:06.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.827+0000 7fe85ba00640 1 --2- 172.21.15.90:0/3549077955 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe85406cbc0 0x7fe85414ff90 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:06.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.827+0000 7fe85ba00640 1 -- 172.21.15.90:0/3549077955 >> 172.21.15.90:0/3549077955 conn(0x7fe8540b34e0 msgr2=0x7fe854160f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:06.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.827+0000 7fe85ba00640 1 -- 172.21.15.90:0/3549077955 shutdown_connections 2024-03-24T01:24:06.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:06.827+0000 7fe85ba00640 1 -- 172.21.15.90:0/3549077955 wait complete. 2024-03-24T01:24:06.845 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:24:07.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.053+0000 7fe959a00640 1 Processor -- start 2024-03-24T01:24:07.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.053+0000 7fe959a00640 1 -- start start 2024-03-24T01:24:07.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.055+0000 7fe959a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe95415b8b0 0x7fe954099cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:07.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.055+0000 7fe959a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe95409a210 0x7fe9540cfd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:07.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.055+0000 7fe959a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9540d1340 con 0x7fe9540d0260 2024-03-24T01:24:07.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.055+0000 7fe959a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe95406c300 con 0x7fe95415b8b0 2024-03-24T01:24:07.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.055+0000 7fe959a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9540d0aa0 con 0x7fe95409a210 2024-03-24T01:24:07.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.055+0000 7fe952000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe95409a210 0x7fe9540cfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:07.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.056+0000 7fe952000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe95409a210 0x7fe9540cfd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.90:38912/0 (socket says 172.21.15.90:38912) 2024-03-24T01:24:07.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.056+0000 7fe952000640 1 -- 172.21.15.90:0/982552851 learned_addr learned my addr 172.21.15.90:0/982552851 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:07.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.056+0000 7fe952a00640 1 --2- 172.21.15.90:0/982552851 >> v2:172.21.15.120:3300/0 conn(0x7fe95415b8b0 0x7fe954099cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:07.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.056+0000 7fe952000640 1 -- 172.21.15.90:0/982552851 >> v2:172.21.15.120:3300/0 conn(0x7fe95415b8b0 msgr2=0x7fe954099cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:07.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.056+0000 7fe952000640 1 --2- 172.21.15.90:0/982552851 >> v2:172.21.15.120:3300/0 conn(0x7fe95415b8b0 0x7fe954099cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:07.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.056+0000 7fe952000640 1 -- 172.21.15.90:0/982552851 >> v1:172.21.15.120:6789/0 conn(0x7fe9540d0260 legacy=0x7fe9540cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:07.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.056+0000 7fe952000640 1 -- 172.21.15.90:0/982552851 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9540d0d80 con 0x7fe95409a210 2024-03-24T01:24:07.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.056+0000 7fe952a00640 1 --2- 172.21.15.90:0/982552851 >> v2:172.21.15.120:3300/0 conn(0x7fe95415b8b0 0x7fe954099cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:24:07.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.058+0000 7fe952000640 1 --2- 172.21.15.90:0/982552851 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe95409a210 0x7fe9540cfd20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe944009870 tx=0x7fe94402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f75c92d7dff23a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:07.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.058+0000 7fe951600640 1 -- 172.21.15.90:0/982552851 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe944034070 con 0x7fe95409a210 2024-03-24T01:24:07.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.058+0000 7fe951600640 1 -- 172.21.15.90:0/982552851 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe944030840 con 0x7fe95409a210 2024-03-24T01:24:07.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.058+0000 7fe951600640 1 -- 172.21.15.90:0/982552851 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe944030c10 con 0x7fe95409a210 2024-03-24T01:24:07.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.059+0000 7fe959a00640 1 -- 172.21.15.90:0/982552851 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe95409a210 msgr2=0x7fe9540cfd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:07.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.059+0000 7fe959a00640 1 --2- 172.21.15.90:0/982552851 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe95409a210 0x7fe9540cfd20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe944009870 tx=0x7fe94402b980 comp rx=0 tx=0).stop 2024-03-24T01:24:07.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.059+0000 7fe959a00640 1 -- 172.21.15.90:0/982552851 shutdown_connections 2024-03-24T01:24:07.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.059+0000 7fe959a00640 1 --2- 172.21.15.90:0/982552851 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe95409a210 0x7fe9540cfd20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:07.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.059+0000 7fe959a00640 1 --2- 172.21.15.90:0/982552851 >> v2:172.21.15.120:3300/0 conn(0x7fe95415b8b0 0x7fe954099cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:07.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.059+0000 7fe959a00640 1 -- 172.21.15.90:0/982552851 >> 172.21.15.90:0/982552851 conn(0x7fe9540b34e0 msgr2=0x7fe9540b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:07.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.059+0000 7fe959a00640 1 -- 172.21.15.90:0/982552851 shutdown_connections 2024-03-24T01:24:07.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.059+0000 7fe959a00640 1 -- 172.21.15.90:0/982552851 wait complete. 2024-03-24T01:24:07.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.061+0000 7fe959a00640 1 Processor -- start 2024-03-24T01:24:07.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.062+0000 7fe959a00640 1 -- start start 2024-03-24T01:24:07.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.063+0000 7fe959a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe95409a210 0x7fe954154d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:07.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.063+0000 7fe959a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9540d0260 0x7fe9540cc300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:07.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.063+0000 7fe959a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe95415b8b0 0x7fe95415a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:07.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.063+0000 7fe959a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9540d06f0 con 0x7fe9540d0260 2024-03-24T01:24:07.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.063+0000 7fe959a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9540cc040 con 0x7fe95409a210 2024-03-24T01:24:07.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.063+0000 7fe959a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9540ccc70 con 0x7fe95415b8b0 2024-03-24T01:24:07.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.063+0000 7fe952000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9540d0260 0x7fe9540cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:07.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.063+0000 7fe952000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9540d0260 0x7fe9540cc300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45882/0 (socket says 172.21.15.90:45882) 2024-03-24T01:24:07.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.063+0000 7fe952000640 1 -- 172.21.15.90:0/2389605231 learned_addr learned my addr 172.21.15.90:0/2389605231 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:07.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.063+0000 7fe952a00640 1 --2- 172.21.15.90:0/2389605231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe95409a210 0x7fe954154d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:07.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.063+0000 7fe953400640 1 --2- 172.21.15.90:0/2389605231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe95415b8b0 0x7fe95415a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:07.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.063+0000 7fe952000640 1 -- 172.21.15.90:0/2389605231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe95415b8b0 msgr2=0x7fe95415a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:07.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.063+0000 7fe952000640 1 --2- 172.21.15.90:0/2389605231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe95415b8b0 0x7fe95415a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:07.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.063+0000 7fe952000640 1 -- 172.21.15.90:0/2389605231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe95409a210 msgr2=0x7fe954154d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:07.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.063+0000 7fe952000640 1 --2- 172.21.15.90:0/2389605231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe95409a210 0x7fe954154d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:07.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.063+0000 7fe952000640 1 -- 172.21.15.90:0/2389605231 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9540ccf50 con 0x7fe9540d0260 2024-03-24T01:24:07.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.064+0000 7fe952000640 1 --2- 172.21.15.90:0/2389605231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9540d0260 0x7fe9540cc300 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe944000c00 tx=0x7fe944004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:07.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.064+0000 7fe94aa00640 1 -- 172.21.15.90:0/2389605231 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe944048070 con 0x7fe9540d0260 2024-03-24T01:24:07.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.064+0000 7fe94aa00640 1 -- 172.21.15.90:0/2389605231 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe944030640 con 0x7fe9540d0260 2024-03-24T01:24:07.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.064+0000 7fe94aa00640 1 -- 172.21.15.90:0/2389605231 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe944034040 con 0x7fe9540d0260 2024-03-24T01:24:07.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.064+0000 7fe959a00640 1 -- 172.21.15.90:0/2389605231 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9540cd230 con 0x7fe9540d0260 2024-03-24T01:24:07.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.064+0000 7fe959a00640 1 -- 172.21.15.90:0/2389605231 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9540cc840 con 0x7fe9540d0260 2024-03-24T01:24:07.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.066+0000 7fe959a00640 1 -- 172.21.15.90:0/2389605231 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe954068060 con 0x7fe9540d0260 2024-03-24T01:24:07.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.067+0000 7fe94aa00640 1 -- 172.21.15.90:0/2389605231 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe94404b070 con 0x7fe9540d0260 2024-03-24T01:24:07.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.067+0000 7fe94aa00640 1 --2- 172.21.15.90:0/2389605231 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe93006bbd0 0x7fe93006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:07.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.067+0000 7fe94aa00640 1 -- 172.21.15.90:0/2389605231 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe9440ac610 con 0x7fe9540d0260 2024-03-24T01:24:07.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.068+0000 7fe952a00640 1 --2- 172.21.15.90:0/2389605231 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe93006bbd0 0x7fe93006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:07.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.069+0000 7fe952a00640 1 --2- 172.21.15.90:0/2389605231 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe93006bbd0 0x7fe93006e090 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe93c006fd0 tx=0x7fe93c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:07.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.072+0000 7fe94aa00640 1 -- 172.21.15.90:0/2389605231 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe94407b570 con 0x7fe9540d0260 2024-03-24T01:24:07.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.271+0000 7fe959a00640 1 -- 172.21.15.90:0/2389605231 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9540cbaf0 con 0x7fe9540d0260 2024-03-24T01:24:07.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.273+0000 7fe94aa00640 1 -- 172.21.15.90:0/2389605231 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12860 (secure 0 0 0) 0x7fe944030810 con 0x7fe9540d0260 2024-03-24T01:24:07.273 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:24:07.274 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":16,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:24:01.886790+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:24:07.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.275+0000 7fe959a00640 1 -- 172.21.15.90:0/2389605231 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe93006bbd0 msgr2=0x7fe93006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:07.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.275+0000 7fe959a00640 1 --2- 172.21.15.90:0/2389605231 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe93006bbd0 0x7fe93006e090 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe93c006fd0 tx=0x7fe93c008040 comp rx=0 tx=0).stop 2024-03-24T01:24:07.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.275+0000 7fe959a00640 1 -- 172.21.15.90:0/2389605231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9540d0260 msgr2=0x7fe9540cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:07.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.275+0000 7fe959a00640 1 --2- 172.21.15.90:0/2389605231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9540d0260 0x7fe9540cc300 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe944000c00 tx=0x7fe944004730 comp rx=0 tx=0).stop 2024-03-24T01:24:07.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.275+0000 7fe959a00640 1 -- 172.21.15.90:0/2389605231 shutdown_connections 2024-03-24T01:24:07.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.275+0000 7fe959a00640 1 --2- 172.21.15.90:0/2389605231 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe93006bbd0 0x7fe93006e090 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:07.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.275+0000 7fe959a00640 1 --2- 172.21.15.90:0/2389605231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9540d0260 0x7fe9540cc300 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:07.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.275+0000 7fe959a00640 1 --2- 172.21.15.90:0/2389605231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe95415b8b0 0x7fe95415a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:07.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.276+0000 7fe959a00640 1 --2- 172.21.15.90:0/2389605231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe95409a210 0x7fe954154d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:07.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.276+0000 7fe959a00640 1 -- 172.21.15.90:0/2389605231 >> 172.21.15.90:0/2389605231 conn(0x7fe9540b34e0 msgr2=0x7fe95415cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:07.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.276+0000 7fe959a00640 1 -- 172.21.15.90:0/2389605231 shutdown_connections 2024-03-24T01:24:07.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.276+0000 7fe959a00640 1 -- 172.21.15.90:0/2389605231 wait complete. 2024-03-24T01:24:07.295 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T01:24:07.295 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:24:07.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.499+0000 7f6b71e00640 1 Processor -- start 2024-03-24T01:24:07.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.499+0000 7f6b71e00640 1 -- start start 2024-03-24T01:24:07.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.500+0000 7f6b71e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6b6c0cfa30 0x7f6b6c15c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:07.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.500+0000 7f6b71e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b6c0b2ed0 con 0x7f6b6c155100 2024-03-24T01:24:07.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.500+0000 7f6b71e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b6c0b31b0 con 0x7f6b6c154d20 2024-03-24T01:24:07.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.500+0000 7f6b71e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b6c0b3490 con 0x7f6b6c0cfa30 2024-03-24T01:24:07.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.500+0000 7f6b63e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f6b6c155100 0x7f6b6c0cf320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41708/0 (socket says 172.21.15.90:41708) 2024-03-24T01:24:07.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.501+0000 7f6b63e00640 1 -- 172.21.15.90:0/2181274043 learned_addr learned my addr 172.21.15.90:0/2181274043 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:07.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.501+0000 7f6b71400640 1 --2- 172.21.15.90:0/2181274043 >> v2:172.21.15.90:3300/0 conn(0x7f6b6c0cfa30 0x7f6b6c15c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:07.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.501+0000 7f6b71400640 1 -- 172.21.15.90:0/2181274043 >> v1:172.21.15.120:6789/0 conn(0x7f6b6c154d20 legacy=0x7f6b6c0b2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:07.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.501+0000 7f6b6be00640 1 -- 172.21.15.90:0/2181274043 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2945507959 0 0) 0x7f6b6c0b2ed0 con 0x7f6b6c155100 2024-03-24T01:24:07.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.501+0000 7f6b71400640 1 -- 172.21.15.90:0/2181274043 >> v1:172.21.15.90:6789/0 conn(0x7f6b6c155100 legacy=0x7f6b6c0cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:07.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.501+0000 7f6b71400640 1 -- 172.21.15.90:0/2181274043 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b6c0b3770 con 0x7f6b6c0cfa30 2024-03-24T01:24:07.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.502+0000 7f6b71400640 1 --2- 172.21.15.90:0/2181274043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b6c0cfa30 0x7f6b6c15c030 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6b64009d90 tx=0x7f6b6402e030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd0f8bcb5b0afe91 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:07.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.502+0000 7f6b6be00640 1 -- 172.21.15.90:0/2181274043 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6403b070 con 0x7f6b6c0cfa30 2024-03-24T01:24:07.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.502+0000 7f6b6be00640 1 -- 172.21.15.90:0/2181274043 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b6402ede0 con 0x7f6b6c0cfa30 2024-03-24T01:24:07.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.502+0000 7f6b6be00640 1 -- 172.21.15.90:0/2181274043 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b64036660 con 0x7f6b6c0cfa30 2024-03-24T01:24:07.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.503+0000 7f6b71e00640 1 -- 172.21.15.90:0/2181274043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b6c0cfa30 msgr2=0x7f6b6c15c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:07.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.503+0000 7f6b71e00640 1 --2- 172.21.15.90:0/2181274043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b6c0cfa30 0x7f6b6c15c030 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6b64009d90 tx=0x7f6b6402e030 comp rx=0 tx=0).stop 2024-03-24T01:24:07.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.503+0000 7f6b71e00640 1 -- 172.21.15.90:0/2181274043 shutdown_connections 2024-03-24T01:24:07.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.503+0000 7f6b71e00640 1 --2- 172.21.15.90:0/2181274043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b6c0cfa30 0x7f6b6c15c030 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:07.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.503+0000 7f6b71e00640 1 -- 172.21.15.90:0/2181274043 >> 172.21.15.90:0/2181274043 conn(0x7f6b6c06b0c0 msgr2=0x7f6b6c06b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:07.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.503+0000 7f6b71e00640 1 -- 172.21.15.90:0/2181274043 shutdown_connections 2024-03-24T01:24:07.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.503+0000 7f6b71e00640 1 -- 172.21.15.90:0/2181274043 wait complete. 2024-03-24T01:24:07.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.505+0000 7f6b71e00640 1 Processor -- start 2024-03-24T01:24:07.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.505+0000 7f6b71e00640 1 -- start start 2024-03-24T01:24:07.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.506+0000 7f6b71e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b6c0cfa30 0x7f6b6c152060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:07.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.506+0000 7f6b71e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b6c154d20 0x7f6b6c15ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:07.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.506+0000 7f6b71e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b6c155100 0x7f6b6c1511a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:07.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.506+0000 7f6b71e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6c15b200 con 0x7f6b6c155100 2024-03-24T01:24:07.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.506+0000 7f6b71e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6c15b080 con 0x7f6b6c0cfa30 2024-03-24T01:24:07.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.506+0000 7f6b71e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6c15d5c0 con 0x7f6b6c154d20 2024-03-24T01:24:07.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.506+0000 7f6b71400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b6c155100 0x7f6b6c1511a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:07.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.506+0000 7f6b71400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b6c155100 0x7f6b6c1511a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45894/0 (socket says 172.21.15.90:45894) 2024-03-24T01:24:07.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.506+0000 7f6b71400640 1 -- 172.21.15.90:0/3222566674 learned_addr learned my addr 172.21.15.90:0/3222566674 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:07.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.506+0000 7f6b70a00640 1 --2- 172.21.15.90:0/3222566674 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b6c0cfa30 0x7f6b6c152060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:07.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.506+0000 7f6b71400640 1 -- 172.21.15.90:0/3222566674 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b6c154d20 msgr2=0x7f6b6c15ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:07.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.506+0000 7f6b71400640 1 --2- 172.21.15.90:0/3222566674 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b6c154d20 0x7f6b6c15ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:07.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.506+0000 7f6b71400640 1 -- 172.21.15.90:0/3222566674 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b6c0cfa30 msgr2=0x7f6b6c152060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:07.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.506+0000 7f6b71400640 1 --2- 172.21.15.90:0/3222566674 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b6c0cfa30 0x7f6b6c152060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:07.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.506+0000 7f6b71400640 1 -- 172.21.15.90:0/3222566674 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b6c1581d0 con 0x7f6b6c155100 2024-03-24T01:24:07.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.507+0000 7f6b71400640 1 --2- 172.21.15.90:0/3222566674 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b6c155100 0x7f6b6c1511a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6b64009d90 tx=0x7f6b64004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:07.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.507+0000 7f6b69600640 1 -- 172.21.15.90:0/3222566674 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b64046070 con 0x7f6b6c155100 2024-03-24T01:24:07.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.507+0000 7f6b69600640 1 -- 172.21.15.90:0/3222566674 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b64036d90 con 0x7f6b6c155100 2024-03-24T01:24:07.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.507+0000 7f6b69600640 1 -- 172.21.15.90:0/3222566674 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6403b040 con 0x7f6b6c155100 2024-03-24T01:24:07.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.507+0000 7f6b71e00640 1 -- 172.21.15.90:0/3222566674 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b6c157ef0 con 0x7f6b6c155100 2024-03-24T01:24:07.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.507+0000 7f6b71e00640 1 -- 172.21.15.90:0/3222566674 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b6c0b0940 con 0x7f6b6c155100 2024-03-24T01:24:07.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.508+0000 7f6b71e00640 1 -- 172.21.15.90:0/3222566674 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b6c04f400 con 0x7f6b6c155100 2024-03-24T01:24:07.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.509+0000 7f6b69600640 1 -- 172.21.15.90:0/3222566674 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6b6404a430 con 0x7f6b6c155100 2024-03-24T01:24:07.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.510+0000 7f6b69600640 1 --2- 172.21.15.90:0/3222566674 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6b4806bc00 0x7f6b4806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:07.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.510+0000 7f6b69600640 1 -- 172.21.15.90:0/3222566674 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6b640ac1f0 con 0x7f6b6c155100 2024-03-24T01:24:07.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.511+0000 7f6b70a00640 1 --2- 172.21.15.90:0/3222566674 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6b4806bc00 0x7f6b4806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:07.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.512+0000 7f6b70a00640 1 --2- 172.21.15.90:0/3222566674 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6b4806bc00 0x7f6b4806e0c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c0cee90 tx=0x7f6b5c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:07.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.514+0000 7f6b69600640 1 -- 172.21.15.90:0/3222566674 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6b6407af90 con 0x7f6b6c155100 2024-03-24T01:24:07.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.714+0000 7f6b71e00640 1 -- 172.21.15.90:0/3222566674 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6b6c067a70 con 0x7f6b6c155100 2024-03-24T01:24:07.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.714+0000 7f6b69600640 1 -- 172.21.15.90:0/3222566674 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6b6407a890 con 0x7f6b6c155100 2024-03-24T01:24:07.715 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:24:07.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.717+0000 7f6b71e00640 1 -- 172.21.15.90:0/3222566674 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6b4806bc00 msgr2=0x7f6b4806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:07.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.717+0000 7f6b71e00640 1 --2- 172.21.15.90:0/3222566674 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6b4806bc00 0x7f6b4806e0c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c0cee90 tx=0x7f6b5c009040 comp rx=0 tx=0).stop 2024-03-24T01:24:07.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.717+0000 7f6b71e00640 1 -- 172.21.15.90:0/3222566674 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b6c155100 msgr2=0x7f6b6c1511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:07.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.717+0000 7f6b71e00640 1 --2- 172.21.15.90:0/3222566674 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b6c155100 0x7f6b6c1511a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6b64009d90 tx=0x7f6b64004ab0 comp rx=0 tx=0).stop 2024-03-24T01:24:07.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.717+0000 7f6b71e00640 1 -- 172.21.15.90:0/3222566674 shutdown_connections 2024-03-24T01:24:07.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.717+0000 7f6b71e00640 1 --2- 172.21.15.90:0/3222566674 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6b4806bc00 0x7f6b4806e0c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:07.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.717+0000 7f6b71e00640 1 --2- 172.21.15.90:0/3222566674 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b6c155100 0x7f6b6c1511a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:07.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.717+0000 7f6b71e00640 1 --2- 172.21.15.90:0/3222566674 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6b6c154d20 0x7f6b6c15ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:07.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.717+0000 7f6b71e00640 1 --2- 172.21.15.90:0/3222566674 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b6c0cfa30 0x7f6b6c152060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:07.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.717+0000 7f6b71e00640 1 -- 172.21.15.90:0/3222566674 >> 172.21.15.90:0/3222566674 conn(0x7f6b6c06b0c0 msgr2=0x7f6b6c0adf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:07.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.717+0000 7f6b71e00640 1 -- 172.21.15.90:0/3222566674 shutdown_connections 2024-03-24T01:24:07.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.717+0000 7f6b71e00640 1 -- 172.21.15.90:0/3222566674 wait complete. 2024-03-24T01:24:07.736 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-24T01:24:07.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.941+0000 7f6846c00640 1 Processor -- start 2024-03-24T01:24:07.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.941+0000 7f6846c00640 1 -- start start 2024-03-24T01:24:07.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.943+0000 7f6846c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6840155030 0x7f684015efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:07.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.943+0000 7f6846c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68400cdac0 0x7f684006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:07.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.943+0000 7f6846c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f684015f500 con 0x7f684006b600 2024-03-24T01:24:07.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.943+0000 7f6846c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f684006ba70 con 0x7f6840155030 2024-03-24T01:24:07.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.943+0000 7f6846c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f684015bf70 con 0x7f68400cdac0 2024-03-24T01:24:07.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.943+0000 7f683fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6840155030 0x7f684015efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:07.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.943+0000 7f683fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6840155030 0x7f684015efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45906/0 (socket says 172.21.15.90:45906) 2024-03-24T01:24:07.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.943+0000 7f683fe00640 1 -- 172.21.15.90:0/3235751715 learned_addr learned my addr 172.21.15.90:0/3235751715 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:07.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.943+0000 7f6844c00640 1 --1- 172.21.15.90:0/3235751715 >> v1:172.21.15.90:6789/0 conn(0x7f684006b600 0x7f68400ae530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41710/0 (socket says 172.21.15.90:41710) 2024-03-24T01:24:07.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.944+0000 7f683f400640 1 --2- 172.21.15.90:0/3235751715 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68400cdac0 0x7f684006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:07.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.944+0000 7f683fe00640 1 -- 172.21.15.90:0/3235751715 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68400cdac0 msgr2=0x7f684006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:07.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.944+0000 7f683fe00640 1 --2- 172.21.15.90:0/3235751715 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68400cdac0 0x7f684006b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:07.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.944+0000 7f683fe00640 1 -- 172.21.15.90:0/3235751715 >> v1:172.21.15.90:6789/0 conn(0x7f684006b600 legacy=0x7f68400ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:07.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.944+0000 7f683fe00640 1 -- 172.21.15.90:0/3235751715 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684015b360 con 0x7f6840155030 2024-03-24T01:24:07.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.944+0000 7f683ea00640 1 -- 172.21.15.90:0/3235751715 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1965690173 0 0) 0x7f6838001110 con 0x7f684006b600 2024-03-24T01:24:07.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.945+0000 7f683fe00640 1 --2- 172.21.15.90:0/3235751715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6840155030 0x7f684015efc0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6830009e50 tx=0x7f683002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5873f050e5b8405 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:07.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.945+0000 7f683ea00640 1 -- 172.21.15.90:0/3235751715 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683002cb00 con 0x7f6840155030 2024-03-24T01:24:07.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.945+0000 7f683ea00640 1 -- 172.21.15.90:0/3235751715 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f683002cca0 con 0x7f6840155030 2024-03-24T01:24:07.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.945+0000 7f683ea00640 1 -- 172.21.15.90:0/3235751715 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6830034610 con 0x7f6840155030 2024-03-24T01:24:07.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.946+0000 7f6846c00640 1 -- 172.21.15.90:0/3235751715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6840155030 msgr2=0x7f684015efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:07.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.946+0000 7f6846c00640 1 --2- 172.21.15.90:0/3235751715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6840155030 0x7f684015efc0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6830009e50 tx=0x7f683002bf60 comp rx=0 tx=0).stop 2024-03-24T01:24:07.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.946+0000 7f6846c00640 1 -- 172.21.15.90:0/3235751715 shutdown_connections 2024-03-24T01:24:07.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.946+0000 7f6846c00640 1 --2- 172.21.15.90:0/3235751715 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68400cdac0 0x7f684006b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:07.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.946+0000 7f6846c00640 1 --2- 172.21.15.90:0/3235751715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6840155030 0x7f684015efc0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:07.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.946+0000 7f6846c00640 1 -- 172.21.15.90:0/3235751715 >> 172.21.15.90:0/3235751715 conn(0x7f68400b1770 msgr2=0x7f68400b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:07.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.946+0000 7f6846c00640 1 -- 172.21.15.90:0/3235751715 shutdown_connections 2024-03-24T01:24:07.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.946+0000 7f6846c00640 1 -- 172.21.15.90:0/3235751715 wait complete. 2024-03-24T01:24:07.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.948+0000 7f6846c00640 1 Processor -- start 2024-03-24T01:24:07.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.948+0000 7f6846c00640 1 -- start start 2024-03-24T01:24:07.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.949+0000 7f6846c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f684006b600 0x7f684014f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:07.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.949+0000 7f6846c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68400cdac0 0x7f68401530c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:07.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.949+0000 7f6846c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6840155030 0x7f6840153600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:07.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.949+0000 7f6846c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f684015bc00 con 0x7f68400cdac0 2024-03-24T01:24:07.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.949+0000 7f6846c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f684015b920 con 0x7f6840155030 2024-03-24T01:24:07.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.949+0000 7f6846c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6840155410 con 0x7f684006b600 2024-03-24T01:24:07.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.949+0000 7f683f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68400cdac0 0x7f68401530c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:07.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.949+0000 7f683f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68400cdac0 0x7f68401530c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45912/0 (socket says 172.21.15.90:45912) 2024-03-24T01:24:07.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.949+0000 7f683f400640 1 -- 172.21.15.90:0/4008211821 learned_addr learned my addr 172.21.15.90:0/4008211821 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:07.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.949+0000 7f683fe00640 1 --2- 172.21.15.90:0/4008211821 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f684006b600 0x7f684014f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:07.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.949+0000 7f683f400640 1 -- 172.21.15.90:0/4008211821 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f684006b600 msgr2=0x7f684014f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:07.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.949+0000 7f683f400640 1 --2- 172.21.15.90:0/4008211821 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f684006b600 0x7f684014f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:07.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.949+0000 7f6844c00640 1 --2- 172.21.15.90:0/4008211821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6840155030 0x7f6840153600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:07.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.949+0000 7f683f400640 1 -- 172.21.15.90:0/4008211821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6840155030 msgr2=0x7f6840153600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:07.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.949+0000 7f683f400640 1 --2- 172.21.15.90:0/4008211821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6840155030 0x7f6840153600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:07.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.950+0000 7f683f400640 1 -- 172.21.15.90:0/4008211821 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68401f2360 con 0x7f68400cdac0 2024-03-24T01:24:07.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.950+0000 7f683f400640 1 --2- 172.21.15.90:0/4008211821 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68400cdac0 0x7f68401530c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f682800d800 tx=0x7f682800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:07.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.950+0000 7f6837e00640 1 -- 172.21.15.90:0/4008211821 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682800b660 con 0x7f68400cdac0 2024-03-24T01:24:07.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.950+0000 7f6837e00640 1 -- 172.21.15.90:0/4008211821 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6828019070 con 0x7f68400cdac0 2024-03-24T01:24:07.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.950+0000 7f6837e00640 1 -- 172.21.15.90:0/4008211821 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682800ba20 con 0x7f68400cdac0 2024-03-24T01:24:07.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.950+0000 7f6846c00640 1 -- 172.21.15.90:0/4008211821 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68400ce170 con 0x7f68400cdac0 2024-03-24T01:24:07.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.950+0000 7f6846c00640 1 -- 172.21.15.90:0/4008211821 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68400ce730 con 0x7f68400cdac0 2024-03-24T01:24:07.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.951+0000 7f6846c00640 1 -- 172.21.15.90:0/4008211821 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6840068060 con 0x7f68400cdac0 2024-03-24T01:24:07.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.952+0000 7f6837e00640 1 -- 172.21.15.90:0/4008211821 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f682800bbc0 con 0x7f68400cdac0 2024-03-24T01:24:07.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.953+0000 7f6837e00640 1 --2- 172.21.15.90:0/4008211821 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f681c06bbd0 0x7f681c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:07.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.953+0000 7f6837e00640 1 -- 172.21.15.90:0/4008211821 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f682808aaa0 con 0x7f68400cdac0 2024-03-24T01:24:07.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.954+0000 7f683fe00640 1 --2- 172.21.15.90:0/4008211821 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f681c06bbd0 0x7f681c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:07.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.955+0000 7f683fe00640 1 --2- 172.21.15.90:0/4008211821 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f681c06bbd0 0x7f681c06e090 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6830035000 tx=0x7f6830030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:07.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:07.958+0000 7f6837e00640 1 -- 172.21.15.90:0/4008211821 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6828059970 con 0x7f68400cdac0 2024-03-24T01:24:08.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.157+0000 7f6846c00640 1 -- 172.21.15.90:0/4008211821 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f68400d00d0 con 0x7f68400cdac0 2024-03-24T01:24:08.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.158+0000 7f6837e00640 1 -- 172.21.15.90:0/4008211821 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7f6828059270 con 0x7f68400cdac0 2024-03-24T01:24:08.158 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:24:08.158 INFO:teuthology.orchestra.run.smithi090.stdout:[ 2024-03-24T01:24:08.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:24:08.158 INFO:teuthology.orchestra.run.smithi090.stdout: "poolnum": 1, 2024-03-24T01:24:08.158 INFO:teuthology.orchestra.run.smithi090.stdout: "poolname": ".mgr" 2024-03-24T01:24:08.158 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:24:08.158 INFO:teuthology.orchestra.run.smithi090.stdout:] 2024-03-24T01:24:08.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.160+0000 7f6846c00640 1 -- 172.21.15.90:0/4008211821 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f681c06bbd0 msgr2=0x7f681c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:08.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.160+0000 7f6846c00640 1 --2- 172.21.15.90:0/4008211821 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f681c06bbd0 0x7f681c06e090 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6830035000 tx=0x7f6830030000 comp rx=0 tx=0).stop 2024-03-24T01:24:08.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.160+0000 7f6846c00640 1 -- 172.21.15.90:0/4008211821 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68400cdac0 msgr2=0x7f68401530c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:08.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.160+0000 7f6846c00640 1 --2- 172.21.15.90:0/4008211821 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68400cdac0 0x7f68401530c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f682800d800 tx=0x7f682800dcc0 comp rx=0 tx=0).stop 2024-03-24T01:24:08.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.160+0000 7f6846c00640 1 -- 172.21.15.90:0/4008211821 shutdown_connections 2024-03-24T01:24:08.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.160+0000 7f6846c00640 1 --2- 172.21.15.90:0/4008211821 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f681c06bbd0 0x7f681c06e090 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:08.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.160+0000 7f6846c00640 1 --2- 172.21.15.90:0/4008211821 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68400cdac0 0x7f68401530c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:08.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.160+0000 7f6846c00640 1 --2- 172.21.15.90:0/4008211821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6840155030 0x7f6840153600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:08.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.160+0000 7f6846c00640 1 --2- 172.21.15.90:0/4008211821 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f684006b600 0x7f684014f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:08.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.160+0000 7f6846c00640 1 -- 172.21.15.90:0/4008211821 >> 172.21.15.90:0/4008211821 conn(0x7f68400b1770 msgr2=0x7f68400b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:08.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.161+0000 7f6846c00640 1 -- 172.21.15.90:0/4008211821 shutdown_connections 2024-03-24T01:24:08.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.161+0000 7f6846c00640 1 -- 172.21.15.90:0/4008211821 wait complete. 2024-03-24T01:24:08.179 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-24T01:24:08.179 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-24T01:24:08.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.381+0000 7f9a83200640 1 Processor -- start 2024-03-24T01:24:08.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.382+0000 7f9a83200640 1 -- start start 2024-03-24T01:24:08.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.383+0000 7f9a83200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9a840b5550 0x7f9a840d1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:08.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.383+0000 7f9a83200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9a840d35b0 0x7f9a840b9540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:08.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.383+0000 7f9a83200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a840d2510 con 0x7f9a8415de50 2024-03-24T01:24:08.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.383+0000 7f9a83200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a840b9a80 con 0x7f9a840b5550 2024-03-24T01:24:08.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.383+0000 7f9a83200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a840b7c10 con 0x7f9a840d35b0 2024-03-24T01:24:08.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.383+0000 7f9a81e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9a840b5550 0x7f9a840d1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:08.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.383+0000 7f9a81e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9a840b5550 0x7f9a840d1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45924/0 (socket says 172.21.15.90:45924) 2024-03-24T01:24:08.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.383+0000 7f9a81e00640 1 -- 172.21.15.90:0/1017568826 learned_addr learned my addr 172.21.15.90:0/1017568826 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:08.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.383+0000 7f9a81e00640 1 -- 172.21.15.90:0/1017568826 >> v2:172.21.15.120:3300/0 conn(0x7f9a840d35b0 msgr2=0x7f9a840b9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:08.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.383+0000 7f9a82800640 1 --2- 172.21.15.90:0/1017568826 >> v2:172.21.15.120:3300/0 conn(0x7f9a840d35b0 0x7f9a840b9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:08.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.384+0000 7f9a81e00640 1 --2- 172.21.15.90:0/1017568826 >> v2:172.21.15.120:3300/0 conn(0x7f9a840d35b0 0x7f9a840b9540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:08.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.384+0000 7f9a81e00640 1 -- 172.21.15.90:0/1017568826 >> v1:172.21.15.90:6789/0 conn(0x7f9a8415de50 legacy=0x7f9a840b5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:08.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.384+0000 7f9a81e00640 1 -- 172.21.15.90:0/1017568826 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a840d27f0 con 0x7f9a840b5550 2024-03-24T01:24:08.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.384+0000 7f9a82800640 1 --2- 172.21.15.90:0/1017568826 >> v2:172.21.15.120:3300/0 conn(0x7f9a840d35b0 0x7f9a840b9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:08.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.385+0000 7f9a81e00640 1 --2- 172.21.15.90:0/1017568826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a840b5550 0x7f9a840d1ea0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9a840b69d0 tx=0x7f9a7402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4fbe69197d0d793 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:08.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.385+0000 7f9a80a00640 1 -- 172.21.15.90:0/1017568826 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a74035020 con 0x7f9a840b5550 2024-03-24T01:24:08.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.385+0000 7f9a80a00640 1 -- 172.21.15.90:0/1017568826 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a74031870 con 0x7f9a840b5550 2024-03-24T01:24:08.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.385+0000 7f9a80a00640 1 -- 172.21.15.90:0/1017568826 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a74031c40 con 0x7f9a840b5550 2024-03-24T01:24:08.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.385+0000 7f9a83200640 1 -- 172.21.15.90:0/1017568826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a840b5550 msgr2=0x7f9a840d1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:08.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.385+0000 7f9a83200640 1 --2- 172.21.15.90:0/1017568826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a840b5550 0x7f9a840d1ea0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9a840b69d0 tx=0x7f9a7402b910 comp rx=0 tx=0).stop 2024-03-24T01:24:08.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.386+0000 7f9a83200640 1 -- 172.21.15.90:0/1017568826 shutdown_connections 2024-03-24T01:24:08.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.386+0000 7f9a83200640 1 --2- 172.21.15.90:0/1017568826 >> v2:172.21.15.120:3300/0 conn(0x7f9a840d35b0 0x7f9a840b9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:08.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.386+0000 7f9a83200640 1 --2- 172.21.15.90:0/1017568826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a840b5550 0x7f9a840d1ea0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:08.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.386+0000 7f9a83200640 1 -- 172.21.15.90:0/1017568826 >> 172.21.15.90:0/1017568826 conn(0x7f9a840d0550 msgr2=0x7f9a840d0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:08.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.386+0000 7f9a83200640 1 -- 172.21.15.90:0/1017568826 shutdown_connections 2024-03-24T01:24:08.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.386+0000 7f9a83200640 1 -- 172.21.15.90:0/1017568826 wait complete. 2024-03-24T01:24:08.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.387+0000 7f9a83200640 1 Processor -- start 2024-03-24T01:24:08.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.387+0000 7f9a83200640 1 -- start start 2024-03-24T01:24:08.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.388+0000 7f9a83200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a840b5550 0x7f9a840cc000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:08.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.388+0000 7f9a83200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a840d35b0 0x7f9a84150610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:08.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.388+0000 7f9a83200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a8415de50 0x7f9a841e6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:08.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.388+0000 7f9a83200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a840d3090 con 0x7f9a8415de50 2024-03-24T01:24:08.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.388+0000 7f9a83200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a840d2db0 con 0x7f9a840d35b0 2024-03-24T01:24:08.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.388+0000 7f9a83200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a840b6c00 con 0x7f9a840b5550 2024-03-24T01:24:08.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.389+0000 7f9a82800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a8415de50 0x7f9a841e6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:08.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.389+0000 7f9a82800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a8415de50 0x7f9a841e6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45936/0 (socket says 172.21.15.90:45936) 2024-03-24T01:24:08.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.389+0000 7f9a82800640 1 -- 172.21.15.90:0/1867766138 learned_addr learned my addr 172.21.15.90:0/1867766138 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:08.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.389+0000 7f9a82800640 1 -- 172.21.15.90:0/1867766138 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a840b5550 msgr2=0x7f9a840cc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:08.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.389+0000 7f9a81e00640 1 --2- 172.21.15.90:0/1867766138 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a840b5550 0x7f9a840cc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:08.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.389+0000 7f9a82800640 1 --2- 172.21.15.90:0/1867766138 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a840b5550 0x7f9a840cc000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:08.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.389+0000 7f9a82800640 1 -- 172.21.15.90:0/1867766138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a840d35b0 msgr2=0x7f9a84150610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:08.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.389+0000 7f9a81400640 1 --2- 172.21.15.90:0/1867766138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a840d35b0 0x7f9a84150610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:08.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.389+0000 7f9a82800640 1 --2- 172.21.15.90:0/1867766138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a840d35b0 0x7f9a84150610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:08.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.389+0000 7f9a82800640 1 -- 172.21.15.90:0/1867766138 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a840cd550 con 0x7f9a8415de50 2024-03-24T01:24:08.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.389+0000 7f9a82800640 1 --2- 172.21.15.90:0/1867766138 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a8415de50 0x7f9a841e6180 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c00c7f0 tx=0x7f9a7c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:08.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.390+0000 7f9a7a000640 1 -- 172.21.15.90:0/1867766138 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c015070 con 0x7f9a8415de50 2024-03-24T01:24:08.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.390+0000 7f9a7a000640 1 -- 172.21.15.90:0/1867766138 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a7c007820 con 0x7f9a8415de50 2024-03-24T01:24:08.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.390+0000 7f9a7a000640 1 -- 172.21.15.90:0/1867766138 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c007c50 con 0x7f9a8415de50 2024-03-24T01:24:08.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.390+0000 7f9a83200640 1 -- 172.21.15.90:0/1867766138 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a840cd830 con 0x7f9a8415de50 2024-03-24T01:24:08.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.390+0000 7f9a83200640 1 -- 172.21.15.90:0/1867766138 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a840cc540 con 0x7f9a8415de50 2024-03-24T01:24:08.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.391+0000 7f9a83200640 1 -- 172.21.15.90:0/1867766138 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a8404f400 con 0x7f9a8415de50 2024-03-24T01:24:08.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.392+0000 7f9a7a000640 1 -- 172.21.15.90:0/1867766138 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9a7c018070 con 0x7f9a8415de50 2024-03-24T01:24:08.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.393+0000 7f9a7a000640 1 --2- 172.21.15.90:0/1867766138 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9a6006bbd0 0x7f9a6006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:08.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.393+0000 7f9a7a000640 1 -- 172.21.15.90:0/1867766138 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9a7c08a9c0 con 0x7f9a8415de50 2024-03-24T01:24:08.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.393+0000 7f9a81e00640 1 --2- 172.21.15.90:0/1867766138 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9a6006bbd0 0x7f9a6006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:08.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.394+0000 7f9a81e00640 1 --2- 172.21.15.90:0/1867766138 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9a6006bbd0 0x7f9a6006e090 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9a74036000 tx=0x7f9a74008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:08.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.397+0000 7f9a7a000640 1 -- 172.21.15.90:0/1867766138 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9a7c059760 con 0x7f9a8415de50 2024-03-24T01:24:08.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.594+0000 7f9a83200640 1 -- 172.21.15.90:0/1867766138 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f9a840d4570 con 0x7f9a8415de50 2024-03-24T01:24:08.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.682+0000 7f9a7a000640 1 -- 172.21.15.90:0/1867766138 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f9a7c059060 con 0x7f9a8415de50 2024-03-24T01:24:08.683 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_metadata' created 2024-03-24T01:24:08.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.686+0000 7f9a83200640 1 -- 172.21.15.90:0/1867766138 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9a6006bbd0 msgr2=0x7f9a6006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:08.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.686+0000 7f9a83200640 1 --2- 172.21.15.90:0/1867766138 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9a6006bbd0 0x7f9a6006e090 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9a74036000 tx=0x7f9a74008030 comp rx=0 tx=0).stop 2024-03-24T01:24:08.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.686+0000 7f9a83200640 1 -- 172.21.15.90:0/1867766138 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a8415de50 msgr2=0x7f9a841e6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:08.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.686+0000 7f9a83200640 1 --2- 172.21.15.90:0/1867766138 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a8415de50 0x7f9a841e6180 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c00c7f0 tx=0x7f9a7c00ccb0 comp rx=0 tx=0).stop 2024-03-24T01:24:08.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.686+0000 7f9a83200640 1 -- 172.21.15.90:0/1867766138 shutdown_connections 2024-03-24T01:24:08.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.686+0000 7f9a83200640 1 --2- 172.21.15.90:0/1867766138 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9a6006bbd0 0x7f9a6006e090 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:08.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.686+0000 7f9a83200640 1 --2- 172.21.15.90:0/1867766138 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a8415de50 0x7f9a841e6180 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:08.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.686+0000 7f9a83200640 1 --2- 172.21.15.90:0/1867766138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9a840d35b0 0x7f9a84150610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:08.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.686+0000 7f9a83200640 1 --2- 172.21.15.90:0/1867766138 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9a840b5550 0x7f9a840cc000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:08.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.686+0000 7f9a83200640 1 -- 172.21.15.90:0/1867766138 >> 172.21.15.90:0/1867766138 conn(0x7f9a840d0550 msgr2=0x7f9a840b47f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:08.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.686+0000 7f9a83200640 1 -- 172.21.15.90:0/1867766138 shutdown_connections 2024-03-24T01:24:08.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.686+0000 7f9a83200640 1 -- 172.21.15.90:0/1867766138 wait complete. 2024-03-24T01:24:08.714 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-24T01:24:08.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.922+0000 7f2964600640 1 Processor -- start 2024-03-24T01:24:08.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.922+0000 7f2964600640 1 -- start start 2024-03-24T01:24:08.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.924+0000 7f2964600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f295c0ae560 0x7f295c0b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:08.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.924+0000 7f2964600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f295c0b4010 con 0x7f295c06fdf0 2024-03-24T01:24:08.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.924+0000 7f2964600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f295c0b42f0 con 0x7f295c15df60 2024-03-24T01:24:08.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.924+0000 7f2964600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f295c0b45d0 con 0x7f295c0ae560 2024-03-24T01:24:08.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.924+0000 7f2961200640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f295c06fdf0 0x7f295c0b32a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41724/0 (socket says 172.21.15.90:41724) 2024-03-24T01:24:08.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.924+0000 7f2961200640 1 -- 172.21.15.90:0/949786587 learned_addr learned my addr 172.21.15.90:0/949786587 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:08.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.924+0000 7f295be00640 1 -- 172.21.15.90:0/949786587 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639494489 0 0) 0x7f295c0b4010 con 0x7f295c06fdf0 2024-03-24T01:24:08.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.924+0000 7f295be00640 1 -- 172.21.15.90:0/949786587 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2940003520 con 0x7f295c06fdf0 2024-03-24T01:24:08.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.924+0000 7f2962600640 1 --2- 172.21.15.90:0/949786587 >> v2:172.21.15.120:3300/0 conn(0x7f295c0ae560 0x7f295c0b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:08.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.925+0000 7f295be00640 1 -- 172.21.15.90:0/949786587 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 668015259 0 0) 0x7f2940003520 con 0x7f295c06fdf0 2024-03-24T01:24:08.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.925+0000 7f295be00640 1 -- 172.21.15.90:0/949786587 >> v2:172.21.15.120:3300/0 conn(0x7f295c0ae560 msgr2=0x7f295c0b39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:08.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.925+0000 7f295be00640 1 --2- 172.21.15.90:0/949786587 >> v2:172.21.15.120:3300/0 conn(0x7f295c0ae560 0x7f295c0b39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:08.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.925+0000 7f295be00640 1 -- 172.21.15.90:0/949786587 >> v1:172.21.15.120:6789/0 conn(0x7f295c15df60 legacy=0x7f295c06f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:08.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.925+0000 7f295be00640 1 -- 172.21.15.90:0/949786587 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f295c0b48b0 con 0x7f295c06fdf0 2024-03-24T01:24:08.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.925+0000 7f295be00640 1 -- 172.21.15.90:0/949786587 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f2948001670 con 0x7f295c06fdf0 2024-03-24T01:24:08.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.925+0000 7f295be00640 1 -- 172.21.15.90:0/949786587 >> v1:172.21.15.90:6789/0 conn(0x7f295c06fdf0 legacy=0x7f295c0b32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:08.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.925+0000 7f295be00640 1 --2- 172.21.15.90:0/949786587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29400039b0 0x7f2940003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:08.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.925+0000 7f295be00640 1 --2- 172.21.15.90:0/949786587 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2940004800 0x7f2940004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:08.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.925+0000 7f295be00640 1 --2- 172.21.15.90:0/949786587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29400051c0 0x7f294000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:08.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.925+0000 7f295be00640 1 -- 172.21.15.90:0/949786587 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f295c0b45d0 con 0x7f2940004800 2024-03-24T01:24:08.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.925+0000 7f295be00640 1 -- 172.21.15.90:0/949786587 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294000c110 con 0x7f29400051c0 2024-03-24T01:24:08.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.925+0000 7f295be00640 1 -- 172.21.15.90:0/949786587 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f294000c310 con 0x7f29400039b0 2024-03-24T01:24:08.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.925+0000 7f2961200640 1 --2- 172.21.15.90:0/949786587 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2940004800 0x7f2940004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:08.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.925+0000 7f2961c00640 1 --2- 172.21.15.90:0/949786587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29400039b0 0x7f2940003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:08.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.926+0000 7f2961200640 1 -- 172.21.15.90:0/949786587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29400039b0 msgr2=0x7f2940003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:08.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.926+0000 7f2961200640 1 --2- 172.21.15.90:0/949786587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29400039b0 0x7f2940003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:08.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.926+0000 7f2961200640 1 -- 172.21.15.90:0/949786587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29400051c0 msgr2=0x7f294000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:08.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.926+0000 7f2961200640 1 --2- 172.21.15.90:0/949786587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29400051c0 0x7f294000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:08.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.926+0000 7f2961200640 1 -- 172.21.15.90:0/949786587 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2940003520 con 0x7f2940004800 2024-03-24T01:24:08.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.926+0000 7f2962600640 1 -- 172.21.15.90:0/949786587 reap_dead start 2024-03-24T01:24:08.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.926+0000 7f2961c00640 1 --2- 172.21.15.90:0/949786587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29400039b0 0x7f2940003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:08.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.927+0000 7f2961200640 1 --2- 172.21.15.90:0/949786587 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2940004800 0x7f2940004c80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f294800c240 tx=0x7f2940005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0c5d27533e366ec server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:08.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.927+0000 7f295be00640 1 -- 172.21.15.90:0/949786587 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2948007cf0 con 0x7f2940004800 2024-03-24T01:24:08.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.927+0000 7f295be00640 1 -- 172.21.15.90:0/949786587 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2948030070 con 0x7f2940004800 2024-03-24T01:24:08.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.927+0000 7f295be00640 1 -- 172.21.15.90:0/949786587 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2948005d70 con 0x7f2940004800 2024-03-24T01:24:08.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.927+0000 7f2964600640 1 -- 172.21.15.90:0/949786587 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2940004800 msgr2=0x7f2940004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:08.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.927+0000 7f2964600640 1 --2- 172.21.15.90:0/949786587 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2940004800 0x7f2940004c80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f294800c240 tx=0x7f2940005830 comp rx=0 tx=0).stop 2024-03-24T01:24:08.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.928+0000 7f2964600640 1 -- 172.21.15.90:0/949786587 shutdown_connections 2024-03-24T01:24:08.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.928+0000 7f2964600640 1 --2- 172.21.15.90:0/949786587 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2940004800 0x7f2940004c80 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:08.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.928+0000 7f2964600640 1 -- 172.21.15.90:0/949786587 >> 172.21.15.90:0/949786587 conn(0x7f295c06e1a0 msgr2=0x7f295c06e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:08.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.928+0000 7f2964600640 1 -- 172.21.15.90:0/949786587 shutdown_connections 2024-03-24T01:24:08.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.928+0000 7f2964600640 1 -- 172.21.15.90:0/949786587 wait complete. 2024-03-24T01:24:08.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.929+0000 7f2964600640 1 Processor -- start 2024-03-24T01:24:08.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.930+0000 7f2964600640 1 -- start start 2024-03-24T01:24:08.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.931+0000 7f2964600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2940004800 0x7f295c14c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:08.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.931+0000 7f2964600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f295c14c120 0x7f295c0cf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:08.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.931+0000 7f2964600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f295c0ccdc0 0x7f295c0cd240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:08.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.931+0000 7f2964600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f295c06f860 con 0x7f2940004800 2024-03-24T01:24:08.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.931+0000 7f2964600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f295c06f6e0 con 0x7f295c14c120 2024-03-24T01:24:08.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.931+0000 7f2964600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f295c06f9e0 con 0x7f295c0ccdc0 2024-03-24T01:24:08.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.931+0000 7f2961c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2940004800 0x7f295c14c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:08.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.931+0000 7f2961c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2940004800 0x7f295c14c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45962/0 (socket says 172.21.15.90:45962) 2024-03-24T01:24:08.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.931+0000 7f2961c00640 1 -- 172.21.15.90:0/3335093702 learned_addr learned my addr 172.21.15.90:0/3335093702 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:08.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.931+0000 7f2962600640 1 --2- 172.21.15.90:0/3335093702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f295c0ccdc0 0x7f295c0cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:08.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.931+0000 7f2961200640 1 --2- 172.21.15.90:0/3335093702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f295c14c120 0x7f295c0cf1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:08.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.931+0000 7f2961c00640 1 -- 172.21.15.90:0/3335093702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f295c0ccdc0 msgr2=0x7f295c0cd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:08.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.931+0000 7f2961c00640 1 --2- 172.21.15.90:0/3335093702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f295c0ccdc0 0x7f295c0cd240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:08.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.931+0000 7f2961c00640 1 -- 172.21.15.90:0/3335093702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f295c14c120 msgr2=0x7f295c0cf1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:08.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.931+0000 7f2961c00640 1 --2- 172.21.15.90:0/3335093702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f295c14c120 0x7f295c0cf1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:08.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.931+0000 7f2961c00640 1 -- 172.21.15.90:0/3335093702 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f295c1506f0 con 0x7f2940004800 2024-03-24T01:24:08.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.931+0000 7f2962600640 1 --2- 172.21.15.90:0/3335093702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f295c0ccdc0 0x7f295c0cd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:08.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.932+0000 7f2961c00640 1 --2- 172.21.15.90:0/3335093702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2940004800 0x7f295c14c6f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f294c00d4b0 tx=0x7f294c00d970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:08.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.932+0000 7f2959600640 1 -- 172.21.15.90:0/3335093702 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f294c005480 con 0x7f2940004800 2024-03-24T01:24:08.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.932+0000 7f2959600640 1 -- 172.21.15.90:0/3335093702 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f294c017070 con 0x7f2940004800 2024-03-24T01:24:08.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.932+0000 7f2959600640 1 -- 172.21.15.90:0/3335093702 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f294c003850 con 0x7f2940004800 2024-03-24T01:24:08.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.932+0000 7f2964600640 1 -- 172.21.15.90:0/3335093702 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f295c1509d0 con 0x7f2940004800 2024-03-24T01:24:08.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.932+0000 7f2964600640 1 -- 172.21.15.90:0/3335093702 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f295c1536d0 con 0x7f2940004800 2024-03-24T01:24:08.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.933+0000 7f2964600640 1 -- 172.21.15.90:0/3335093702 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f295c150ba0 con 0x7f2940004800 2024-03-24T01:24:08.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.935+0000 7f2959600640 1 -- 172.21.15.90:0/3335093702 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f294c0039f0 con 0x7f2940004800 2024-03-24T01:24:08.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.936+0000 7f2959600640 1 --2- 172.21.15.90:0/3335093702 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f293c06bc00 0x7f293c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:08.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.936+0000 7f2959600640 1 -- 172.21.15.90:0/3335093702 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f294c08a5f0 con 0x7f2940004800 2024-03-24T01:24:08.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.936+0000 7f2961200640 1 --2- 172.21.15.90:0/3335093702 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f293c06bc00 0x7f293c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:08.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.937+0000 7f2961200640 1 --2- 172.21.15.90:0/3335093702 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f293c06bc00 0x7f293c06e0c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2948034000 tx=0x7f2948039040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:08.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:08.941+0000 7f2959600640 1 -- 172.21.15.90:0/3335093702 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f295c150ba0 con 0x7f2940004800 2024-03-24T01:24:09.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.139+0000 7f2964600640 1 -- 172.21.15.90:0/3335093702 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f295c04f400 con 0x7f2940004800 2024-03-24T01:24:09.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.689+0000 7f2959600640 1 -- 172.21.15.90:0/3335093702 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v18) v1 ==== 171+0+0 (secure 0 0 0) 0x7f294c059380 con 0x7f2940004800 2024-03-24T01:24:09.690 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_data' created 2024-03-24T01:24:09.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.692+0000 7f2964600640 1 -- 172.21.15.90:0/3335093702 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f293c06bc00 msgr2=0x7f293c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:09.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.692+0000 7f2964600640 1 --2- 172.21.15.90:0/3335093702 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f293c06bc00 0x7f293c06e0c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2948034000 tx=0x7f2948039040 comp rx=0 tx=0).stop 2024-03-24T01:24:09.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.692+0000 7f2964600640 1 -- 172.21.15.90:0/3335093702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2940004800 msgr2=0x7f295c14c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:09.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.692+0000 7f2964600640 1 --2- 172.21.15.90:0/3335093702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2940004800 0x7f295c14c6f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f294c00d4b0 tx=0x7f294c00d970 comp rx=0 tx=0).stop 2024-03-24T01:24:09.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.692+0000 7f2964600640 1 -- 172.21.15.90:0/3335093702 shutdown_connections 2024-03-24T01:24:09.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.692+0000 7f2964600640 1 --2- 172.21.15.90:0/3335093702 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f293c06bc00 0x7f293c06e0c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:09.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.692+0000 7f2964600640 1 --2- 172.21.15.90:0/3335093702 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f295c0ccdc0 0x7f295c0cd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:09.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.692+0000 7f2964600640 1 --2- 172.21.15.90:0/3335093702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f295c14c120 0x7f295c0cf1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:09.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.692+0000 7f2964600640 1 --2- 172.21.15.90:0/3335093702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2940004800 0x7f295c14c6f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:09.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.692+0000 7f2964600640 1 -- 172.21.15.90:0/3335093702 >> 172.21.15.90:0/3335093702 conn(0x7f295c06e1a0 msgr2=0x7f295c0b01a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:09.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.693+0000 7f2964600640 1 -- 172.21.15.90:0/3335093702 shutdown_connections 2024-03-24T01:24:09.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.693+0000 7f2964600640 1 -- 172.21.15.90:0/3335093702 wait complete. 2024-03-24T01:24:09.732 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-24T01:24:09.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.916+0000 7f0cb7200640 1 Processor -- start 2024-03-24T01:24:09.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.916+0000 7f0cb7200640 1 -- start start 2024-03-24T01:24:09.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.917+0000 7f0cb7200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0cb806f0d0 0x7f0cb815af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:09.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.917+0000 7f0cb7200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0cb815b4c0 0x7f0cb81549f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:09.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.917+0000 7f0cb7200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cb815b9d0 con 0x7f0cb806ecf0 2024-03-24T01:24:09.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.917+0000 7f0cb7200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cb815bcb0 con 0x7f0cb806f0d0 2024-03-24T01:24:09.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.917+0000 7f0cb7200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cb80b2930 con 0x7f0cb815b4c0 2024-03-24T01:24:09.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.918+0000 7f0cb5e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f0cb806ecf0 0x7f0cb80b3130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:37468/0 (socket says 172.21.15.90:37468) 2024-03-24T01:24:09.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.918+0000 7f0cb5e00640 1 -- 172.21.15.90:0/3923786226 learned_addr learned my addr 172.21.15.90:0/3923786226 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:09.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.918+0000 7f0cad400640 1 --2- 172.21.15.90:0/3923786226 >> v2:172.21.15.90:3300/0 conn(0x7f0cb806f0d0 0x7f0cb815af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:09.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.918+0000 7f0cb6800640 1 --2- 172.21.15.90:0/3923786226 >> v2:172.21.15.120:3300/0 conn(0x7f0cb815b4c0 0x7f0cb81549f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:09.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.918+0000 7f0cb5400640 1 -- 172.21.15.90:0/3923786226 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3781624753 0 0) 0x7f0cb815b9d0 con 0x7f0cb806ecf0 2024-03-24T01:24:09.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.918+0000 7f0cb5400640 1 -- 172.21.15.90:0/3923786226 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c9c005730 con 0x7f0cb806ecf0 2024-03-24T01:24:09.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.918+0000 7f0cad400640 1 -- 172.21.15.90:0/3923786226 >> v2:172.21.15.120:3300/0 conn(0x7f0cb815b4c0 msgr2=0x7f0cb81549f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:09.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.918+0000 7f0cad400640 1 --2- 172.21.15.90:0/3923786226 >> v2:172.21.15.120:3300/0 conn(0x7f0cb815b4c0 0x7f0cb81549f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:09.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.918+0000 7f0cad400640 1 -- 172.21.15.90:0/3923786226 >> v1:172.21.15.90:6789/0 conn(0x7f0cb806ecf0 legacy=0x7f0cb80b3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:09.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.918+0000 7f0cad400640 1 -- 172.21.15.90:0/3923786226 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb80b2c10 con 0x7f0cb806f0d0 2024-03-24T01:24:09.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.919+0000 7f0cad400640 1 --2- 172.21.15.90:0/3923786226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0cb806f0d0 0x7f0cb815af80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0cb80cd190 tx=0x7f0ca002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a26bbb6e1435190 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:09.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.920+0000 7f0cb5400640 1 -- 172.21.15.90:0/3923786226 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca0035020 con 0x7f0cb806f0d0 2024-03-24T01:24:09.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.920+0000 7f0cb5400640 1 -- 172.21.15.90:0/3923786226 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca0031830 con 0x7f0cb806f0d0 2024-03-24T01:24:09.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.920+0000 7f0cb5400640 1 -- 172.21.15.90:0/3923786226 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca0031c00 con 0x7f0cb806f0d0 2024-03-24T01:24:09.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.920+0000 7f0cb7200640 1 -- 172.21.15.90:0/3923786226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0cb806f0d0 msgr2=0x7f0cb815af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:09.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.920+0000 7f0cb7200640 1 --2- 172.21.15.90:0/3923786226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0cb806f0d0 0x7f0cb815af80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0cb80cd190 tx=0x7f0ca002b8d0 comp rx=0 tx=0).stop 2024-03-24T01:24:09.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.920+0000 7f0cb7200640 1 -- 172.21.15.90:0/3923786226 shutdown_connections 2024-03-24T01:24:09.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.920+0000 7f0cb7200640 1 --2- 172.21.15.90:0/3923786226 >> v2:172.21.15.120:3300/0 conn(0x7f0cb815b4c0 0x7f0cb81549f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:09.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.920+0000 7f0cb7200640 1 --2- 172.21.15.90:0/3923786226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0cb806f0d0 0x7f0cb815af80 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:09.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.920+0000 7f0cb7200640 1 -- 172.21.15.90:0/3923786226 >> 172.21.15.90:0/3923786226 conn(0x7f0cb806b0c0 msgr2=0x7f0cb806b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:09.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.920+0000 7f0cb7200640 1 -- 172.21.15.90:0/3923786226 shutdown_connections 2024-03-24T01:24:09.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.920+0000 7f0cb7200640 1 -- 172.21.15.90:0/3923786226 wait complete. 2024-03-24T01:24:09.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.922+0000 7f0cb7200640 1 Processor -- start 2024-03-24T01:24:09.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.922+0000 7f0cb7200640 1 -- start start 2024-03-24T01:24:09.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.923+0000 7f0cb7200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0cb806ecf0 0x7f0cb80afdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:09.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.923+0000 7f0cb7200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0cb806f0d0 0x7f0cb81ee990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:09.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.923+0000 7f0cb7200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cb815b4c0 0x7f0cb8153060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:09.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.923+0000 7f0cb7200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb8154f30 con 0x7f0cb806f0d0 2024-03-24T01:24:09.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.923+0000 7f0cb7200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb80b47b0 con 0x7f0cb815b4c0 2024-03-24T01:24:09.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.923+0000 7f0cb7200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb81550b0 con 0x7f0cb806ecf0 2024-03-24T01:24:09.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.923+0000 7f0cad400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0cb806f0d0 0x7f0cb81ee990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:09.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.923+0000 7f0cad400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0cb806f0d0 0x7f0cb81ee990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50608/0 (socket says 172.21.15.90:50608) 2024-03-24T01:24:09.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.923+0000 7f0cad400640 1 -- 172.21.15.90:0/926592610 learned_addr learned my addr 172.21.15.90:0/926592610 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:09.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.924+0000 7f0cb5e00640 1 --2- 172.21.15.90:0/926592610 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0cb806ecf0 0x7f0cb80afdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:09.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.924+0000 7f0cad400640 1 -- 172.21.15.90:0/926592610 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0cb806ecf0 msgr2=0x7f0cb80afdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:09.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.924+0000 7f0cb6800640 1 --2- 172.21.15.90:0/926592610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cb815b4c0 0x7f0cb8153060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:09.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.924+0000 7f0cad400640 1 --2- 172.21.15.90:0/926592610 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0cb806ecf0 0x7f0cb80afdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:09.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.924+0000 7f0cad400640 1 -- 172.21.15.90:0/926592610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cb815b4c0 msgr2=0x7f0cb8153060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:09.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.924+0000 7f0cad400640 1 --2- 172.21.15.90:0/926592610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cb815b4c0 0x7f0cb8153060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:09.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.924+0000 7f0cad400640 1 -- 172.21.15.90:0/926592610 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb8159af0 con 0x7f0cb806f0d0 2024-03-24T01:24:09.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.924+0000 7f0cad400640 1 --2- 172.21.15.90:0/926592610 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0cb806f0d0 0x7f0cb81ee990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0ca002bee0 tx=0x7f0ca00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:09.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.924+0000 7f0caea00640 1 -- 172.21.15.90:0/926592610 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca0041070 con 0x7f0cb806f0d0 2024-03-24T01:24:09.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.924+0000 7f0caea00640 1 -- 172.21.15.90:0/926592610 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca003f070 con 0x7f0cb806f0d0 2024-03-24T01:24:09.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.924+0000 7f0caea00640 1 -- 172.21.15.90:0/926592610 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca0035040 con 0x7f0cb806f0d0 2024-03-24T01:24:09.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.924+0000 7f0cb7200640 1 -- 172.21.15.90:0/926592610 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cb814fbc0 con 0x7f0cb806f0d0 2024-03-24T01:24:09.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.925+0000 7f0cb7200640 1 -- 172.21.15.90:0/926592610 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cb8150180 con 0x7f0cb806f0d0 2024-03-24T01:24:09.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.926+0000 7f0cb7200640 1 -- 172.21.15.90:0/926592610 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cb804f400 con 0x7f0cb806f0d0 2024-03-24T01:24:09.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.927+0000 7f0caea00640 1 -- 172.21.15.90:0/926592610 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0ca0038020 con 0x7f0cb806f0d0 2024-03-24T01:24:09.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.928+0000 7f0caea00640 1 --2- 172.21.15.90:0/926592610 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0c9406bbd0 0x7f0c9406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:09.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.928+0000 7f0caea00640 1 -- 172.21.15.90:0/926592610 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f0ca00abfa0 con 0x7f0cb806f0d0 2024-03-24T01:24:09.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.929+0000 7f0cb5e00640 1 --2- 172.21.15.90:0/926592610 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0c9406bbd0 0x7f0c9406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:09.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.930+0000 7f0cb5e00640 1 --2- 172.21.15.90:0/926592610 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0c9406bbd0 0x7f0c9406e090 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8000f60 tx=0x7f0ca8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:09.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:09.934+0000 7f0caea00640 1 -- 172.21.15.90:0/926592610 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0ca0031800 con 0x7f0cb806f0d0 2024-03-24T01:24:10.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.170+0000 7f0cb7200640 1 -- 172.21.15.90:0/926592610 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f0cb815f0f0 con 0x7f0cb806f0d0 2024-03-24T01:24:10.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.728+0000 7f0caea00640 1 -- 172.21.15.90:0/926592610 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f0ca007aaf0 con 0x7f0cb806f0d0 2024-03-24T01:24:10.729 INFO:teuthology.orchestra.run.smithi090.stderr:new fs with metadata pool 2 and data pool 3 2024-03-24T01:24:10.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.736+0000 7f0cb7200640 1 -- 172.21.15.90:0/926592610 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0c9406bbd0 msgr2=0x7f0c9406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:10.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.736+0000 7f0cb7200640 1 --2- 172.21.15.90:0/926592610 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0c9406bbd0 0x7f0c9406e090 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8000f60 tx=0x7f0ca8009040 comp rx=0 tx=0).stop 2024-03-24T01:24:10.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.736+0000 7f0cb7200640 1 -- 172.21.15.90:0/926592610 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0cb806f0d0 msgr2=0x7f0cb81ee990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:10.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.736+0000 7f0cb7200640 1 --2- 172.21.15.90:0/926592610 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0cb806f0d0 0x7f0cb81ee990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0ca002bee0 tx=0x7f0ca00046c0 comp rx=0 tx=0).stop 2024-03-24T01:24:10.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.736+0000 7f0cb7200640 1 -- 172.21.15.90:0/926592610 shutdown_connections 2024-03-24T01:24:10.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.736+0000 7f0cb7200640 1 --2- 172.21.15.90:0/926592610 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0c9406bbd0 0x7f0c9406e090 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:10.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.736+0000 7f0cb7200640 1 --2- 172.21.15.90:0/926592610 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0cb806f0d0 0x7f0cb81ee990 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:10.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.736+0000 7f0cb7200640 1 --2- 172.21.15.90:0/926592610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cb815b4c0 0x7f0cb8153060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:10.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.736+0000 7f0cb7200640 1 --2- 172.21.15.90:0/926592610 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0cb806ecf0 0x7f0cb80afdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:10.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.736+0000 7f0cb7200640 1 -- 172.21.15.90:0/926592610 >> 172.21.15.90:0/926592610 conn(0x7f0cb806b0c0 msgr2=0x7f0cb8073bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:10.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.737+0000 7f0cb7200640 1 -- 172.21.15.90:0/926592610 shutdown_connections 2024-03-24T01:24:10.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.737+0000 7f0cb7200640 1 -- 172.21.15.90:0/926592610 wait complete. 2024-03-24T01:24:10.775 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-24T01:24:10.776 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-24T01:24:10.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.958+0000 7f3a55c00640 1 Processor -- start 2024-03-24T01:24:10.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.959+0000 7f3a55c00640 1 -- start start 2024-03-24T01:24:10.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.960+0000 7f3a55c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3a50159620 0x7f3a5006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:10.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.960+0000 7f3a55c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3a500cf150 0x7f3a500ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:10.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.960+0000 7f3a55c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a5006bca0 con 0x7f3a5006b600 2024-03-24T01:24:10.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.960+0000 7f3a55c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a500b4470 con 0x7f3a500cf150 2024-03-24T01:24:10.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.960+0000 7f3a55c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a500b36f0 con 0x7f3a50159620 2024-03-24T01:24:10.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.960+0000 7f3a4f400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3a500cf150 0x7f3a500ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:10.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.960+0000 7f3a46000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f3a5006b600 0x7f3a500cea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:37482/0 (socket says 172.21.15.90:37482) 2024-03-24T01:24:10.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.960+0000 7f3a46000640 1 -- 172.21.15.90:0/3899747028 learned_addr learned my addr 172.21.15.90:0/3899747028 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:10.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.961+0000 7f3a4f400640 1 -- 172.21.15.90:0/3899747028 >> v2:172.21.15.120:3300/0 conn(0x7f3a50159620 msgr2=0x7f3a5006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:10.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.961+0000 7f3a4ea00640 1 --2- 172.21.15.90:0/3899747028 >> v2:172.21.15.120:3300/0 conn(0x7f3a50159620 0x7f3a5006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:10.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.961+0000 7f3a4f400640 1 --2- 172.21.15.90:0/3899747028 >> v2:172.21.15.120:3300/0 conn(0x7f3a50159620 0x7f3a5006b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:10.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.961+0000 7f3a4f400640 1 -- 172.21.15.90:0/3899747028 >> v1:172.21.15.90:6789/0 conn(0x7f3a5006b600 legacy=0x7f3a500cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:10.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.961+0000 7f3a4f400640 1 -- 172.21.15.90:0/3899747028 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a500b3990 con 0x7f3a500cf150 2024-03-24T01:24:10.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.961+0000 7f3a4ea00640 1 --2- 172.21.15.90:0/3899747028 >> v2:172.21.15.120:3300/0 conn(0x7f3a50159620 0x7f3a5006b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:10.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.962+0000 7f3a4f400640 1 --2- 172.21.15.90:0/3899747028 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a500cf150 0x7f3a500ae530 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3a5006c2d0 tx=0x7f3a4802be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1b348033bafb10b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:10.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.962+0000 7f3a4e000640 1 -- 172.21.15.90:0/3899747028 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a48039070 con 0x7f3a500cf150 2024-03-24T01:24:10.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.962+0000 7f3a4e000640 1 -- 172.21.15.90:0/3899747028 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a4802cbe0 con 0x7f3a500cf150 2024-03-24T01:24:10.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.962+0000 7f3a4e000640 1 -- 172.21.15.90:0/3899747028 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a48034440 con 0x7f3a500cf150 2024-03-24T01:24:10.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.963+0000 7f3a55c00640 1 -- 172.21.15.90:0/3899747028 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a500cf150 msgr2=0x7f3a500ae530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:10.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.963+0000 7f3a55c00640 1 --2- 172.21.15.90:0/3899747028 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a500cf150 0x7f3a500ae530 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3a5006c2d0 tx=0x7f3a4802be30 comp rx=0 tx=0).stop 2024-03-24T01:24:10.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.963+0000 7f3a55c00640 1 -- 172.21.15.90:0/3899747028 shutdown_connections 2024-03-24T01:24:10.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.963+0000 7f3a55c00640 1 --2- 172.21.15.90:0/3899747028 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a500cf150 0x7f3a500ae530 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:10.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.963+0000 7f3a55c00640 1 --2- 172.21.15.90:0/3899747028 >> v2:172.21.15.120:3300/0 conn(0x7f3a50159620 0x7f3a5006b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:10.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.963+0000 7f3a55c00640 1 -- 172.21.15.90:0/3899747028 >> 172.21.15.90:0/3899747028 conn(0x7f3a500b1770 msgr2=0x7f3a500b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:10.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.963+0000 7f3a55c00640 1 -- 172.21.15.90:0/3899747028 shutdown_connections 2024-03-24T01:24:10.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.963+0000 7f3a55c00640 1 -- 172.21.15.90:0/3899747028 wait complete. 2024-03-24T01:24:10.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.965+0000 7f3a55c00640 1 Processor -- start 2024-03-24T01:24:10.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.965+0000 7f3a55c00640 1 -- start start 2024-03-24T01:24:10.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.966+0000 7f3a55c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a5006b600 0x7f3a501e9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:10.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.966+0000 7f3a55c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a500cf150 0x7f3a501e69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:10.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.966+0000 7f3a55c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a50159620 0x7f3a500cae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:10.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.966+0000 7f3a55c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a50159a00 con 0x7f3a50159620 2024-03-24T01:24:10.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.966+0000 7f3a55c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a500b3f50 con 0x7f3a5006b600 2024-03-24T01:24:10.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.966+0000 7f3a55c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5015bfe0 con 0x7f3a500cf150 2024-03-24T01:24:10.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.966+0000 7f3a4f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a50159620 0x7f3a500cae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:10.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.966+0000 7f3a4f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a50159620 0x7f3a500cae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50628/0 (socket says 172.21.15.90:50628) 2024-03-24T01:24:10.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.966+0000 7f3a4f400640 1 -- 172.21.15.90:0/1130050326 learned_addr learned my addr 172.21.15.90:0/1130050326 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:10.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.966+0000 7f3a4ea00640 1 --2- 172.21.15.90:0/1130050326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a5006b600 0x7f3a501e9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:10.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.966+0000 7f3a46000640 1 --2- 172.21.15.90:0/1130050326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a500cf150 0x7f3a501e69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:10.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.966+0000 7f3a4f400640 1 -- 172.21.15.90:0/1130050326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a500cf150 msgr2=0x7f3a501e69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:10.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.966+0000 7f3a4f400640 1 --2- 172.21.15.90:0/1130050326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a500cf150 0x7f3a501e69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:10.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.966+0000 7f3a4f400640 1 -- 172.21.15.90:0/1130050326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a5006b600 msgr2=0x7f3a501e9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:10.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.966+0000 7f3a4f400640 1 --2- 172.21.15.90:0/1130050326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a5006b600 0x7f3a501e9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:10.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.966+0000 7f3a4f400640 1 -- 172.21.15.90:0/1130050326 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a501e7220 con 0x7f3a50159620 2024-03-24T01:24:10.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.966+0000 7f3a4ea00640 1 --2- 172.21.15.90:0/1130050326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a5006b600 0x7f3a501e9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:10.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.966+0000 7f3a46000640 1 --2- 172.21.15.90:0/1130050326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a500cf150 0x7f3a501e69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:10.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.967+0000 7f3a4f400640 1 --2- 172.21.15.90:0/1130050326 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a50159620 0x7f3a500cae90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3a4802c440 tx=0x7f3a48004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:10.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.967+0000 7f3a47400640 1 -- 172.21.15.90:0/1130050326 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a48039050 con 0x7f3a50159620 2024-03-24T01:24:10.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.967+0000 7f3a55c00640 1 -- 172.21.15.90:0/1130050326 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a501e7500 con 0x7f3a50159620 2024-03-24T01:24:10.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.967+0000 7f3a47400640 1 -- 172.21.15.90:0/1130050326 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a48045070 con 0x7f3a50159620 2024-03-24T01:24:10.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.967+0000 7f3a47400640 1 -- 172.21.15.90:0/1130050326 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4803e830 con 0x7f3a50159620 2024-03-24T01:24:10.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.967+0000 7f3a55c00640 1 -- 172.21.15.90:0/1130050326 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a501e7da0 con 0x7f3a50159620 2024-03-24T01:24:10.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.969+0000 7f3a55c00640 1 -- 172.21.15.90:0/1130050326 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a501e78f0 con 0x7f3a50159620 2024-03-24T01:24:10.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.969+0000 7f3a47400640 1 -- 172.21.15.90:0/1130050326 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3a4803e9d0 con 0x7f3a50159620 2024-03-24T01:24:10.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.970+0000 7f3a47400640 1 --2- 172.21.15.90:0/1130050326 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3a2c06bbd0 0x7f3a2c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:10.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.970+0000 7f3a47400640 1 -- 172.21.15.90:0/1130050326 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3a480ad9f0 con 0x7f3a50159620 2024-03-24T01:24:10.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.971+0000 7f3a4ea00640 1 --2- 172.21.15.90:0/1130050326 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3a2c06bbd0 0x7f3a2c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:10.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.971+0000 7f3a4ea00640 1 --2- 172.21.15.90:0/1130050326 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3a2c06bbd0 0x7f3a2c06e090 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3a40006fd0 tx=0x7f3a40008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:10.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:10.975+0000 7f3a47400640 1 -- 172.21.15.90:0/1130050326 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3a4807c3c0 con 0x7f3a50159620 2024-03-24T01:24:11.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.179+0000 7f3a55c00640 1 -- 172.21.15.90:0/1130050326 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f3a5004f400 con 0x7f3a50159620 2024-03-24T01:24:11.669 INFO:tasks.ceph.mds.d.smithi120.stderr:2024-03-24T01:24:11.669+0000 7ff5bbc00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T01:24:11.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.728+0000 7f3a47400640 1 -- 172.21.15.90:0/1130050326 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v20) v1 ==== 157+0+0 (secure 0 0 0) 0x7f3a4807bcc0 con 0x7f3a50159620 2024-03-24T01:24:11.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.732+0000 7f3a55c00640 1 -- 172.21.15.90:0/1130050326 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3a2c06bbd0 msgr2=0x7f3a2c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:11.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.732+0000 7f3a55c00640 1 --2- 172.21.15.90:0/1130050326 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3a2c06bbd0 0x7f3a2c06e090 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3a40006fd0 tx=0x7f3a40008040 comp rx=0 tx=0).stop 2024-03-24T01:24:11.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.732+0000 7f3a55c00640 1 -- 172.21.15.90:0/1130050326 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a50159620 msgr2=0x7f3a500cae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:11.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.732+0000 7f3a55c00640 1 --2- 172.21.15.90:0/1130050326 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a50159620 0x7f3a500cae90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3a4802c440 tx=0x7f3a48004a40 comp rx=0 tx=0).stop 2024-03-24T01:24:11.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.733+0000 7f3a55c00640 1 -- 172.21.15.90:0/1130050326 shutdown_connections 2024-03-24T01:24:11.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.733+0000 7f3a55c00640 1 --2- 172.21.15.90:0/1130050326 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3a2c06bbd0 0x7f3a2c06e090 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:11.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.733+0000 7f3a55c00640 1 --2- 172.21.15.90:0/1130050326 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a50159620 0x7f3a500cae90 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:11.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.733+0000 7f3a55c00640 1 --2- 172.21.15.90:0/1130050326 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a500cf150 0x7f3a501e69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:11.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.733+0000 7f3a55c00640 1 --2- 172.21.15.90:0/1130050326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a5006b600 0x7f3a501e9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:11.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.733+0000 7f3a55c00640 1 -- 172.21.15.90:0/1130050326 >> 172.21.15.90:0/1130050326 conn(0x7f3a500b1770 msgr2=0x7f3a5006ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:11.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.733+0000 7f3a55c00640 1 -- 172.21.15.90:0/1130050326 shutdown_connections 2024-03-24T01:24:11.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.733+0000 7f3a55c00640 1 -- 172.21.15.90:0/1130050326 wait complete. 2024-03-24T01:24:11.754 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-24T01:24:11.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.968+0000 7f222f200640 1 Processor -- start 2024-03-24T01:24:11.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.968+0000 7f222f200640 1 -- start start 2024-03-24T01:24:11.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.969+0000 7f222f200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2230161380 0x7f22300b9390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:11.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.969+0000 7f222f200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22300d0a10 0x7f22300d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:11.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.969+0000 7f222f200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f22300d1e90 0x7f22300d2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:11.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.969+0000 7f222f200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22300d1390 con 0x7f22300d1e90 2024-03-24T01:24:11.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.969+0000 7f222f200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22300b8150 con 0x7f2230161380 2024-03-24T01:24:11.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.969+0000 7f222f200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22300b8430 con 0x7f22300d0a10 2024-03-24T01:24:11.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.970+0000 7f222e800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f22300d1e90 0x7f22300d2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:11.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.970+0000 7f222de00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2230161380 0x7f22300b9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:11.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.970+0000 7f222e800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f22300d1e90 0x7f22300d2300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50640/0 (socket says 172.21.15.90:50640) 2024-03-24T01:24:11.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.970+0000 7f222e800640 1 -- 172.21.15.90:0/2701665527 learned_addr learned my addr 172.21.15.90:0/2701665527 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:11.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.970+0000 7f2225400640 1 --2- 172.21.15.90:0/2701665527 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22300d0a10 0x7f22300d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:11.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.970+0000 7f222e800640 1 -- 172.21.15.90:0/2701665527 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22300d0a10 msgr2=0x7f22300d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:11.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.970+0000 7f222e800640 1 --2- 172.21.15.90:0/2701665527 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22300d0a10 0x7f22300d0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:11.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.970+0000 7f222e800640 1 -- 172.21.15.90:0/2701665527 >> v2:172.21.15.120:3300/0 conn(0x7f2230161380 msgr2=0x7f22300b9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:11.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.970+0000 7f222e800640 1 --2- 172.21.15.90:0/2701665527 >> v2:172.21.15.120:3300/0 conn(0x7f2230161380 0x7f22300b9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:11.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.970+0000 7f222e800640 1 -- 172.21.15.90:0/2701665527 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22300b8710 con 0x7f22300d1e90 2024-03-24T01:24:11.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.971+0000 7f222e800640 1 --2- 172.21.15.90:0/2701665527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22300d1e90 0x7f22300d2300 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f22300b63e0 tx=0x7f222802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67269c0d130574cf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:11.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.972+0000 7f222d400640 1 -- 172.21.15.90:0/2701665527 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222802ca70 con 0x7f22300d1e90 2024-03-24T01:24:11.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.972+0000 7f222d400640 1 -- 172.21.15.90:0/2701665527 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f222802cc10 con 0x7f22300d1e90 2024-03-24T01:24:11.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.972+0000 7f222d400640 1 -- 172.21.15.90:0/2701665527 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22280343f0 con 0x7f22300d1e90 2024-03-24T01:24:11.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.972+0000 7f222f200640 1 -- 172.21.15.90:0/2701665527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22300d1e90 msgr2=0x7f22300d2300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:11.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.972+0000 7f222f200640 1 --2- 172.21.15.90:0/2701665527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22300d1e90 0x7f22300d2300 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f22300b63e0 tx=0x7f222802bed0 comp rx=0 tx=0).stop 2024-03-24T01:24:11.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.972+0000 7f222f200640 1 -- 172.21.15.90:0/2701665527 shutdown_connections 2024-03-24T01:24:11.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.972+0000 7f222f200640 1 --2- 172.21.15.90:0/2701665527 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22300d0a10 0x7f22300d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:11.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.972+0000 7f222f200640 1 --2- 172.21.15.90:0/2701665527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22300d1e90 0x7f22300d2300 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:11.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.972+0000 7f222f200640 1 --2- 172.21.15.90:0/2701665527 >> v2:172.21.15.120:3300/0 conn(0x7f2230161380 0x7f22300b9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:11.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.972+0000 7f222f200640 1 -- 172.21.15.90:0/2701665527 >> 172.21.15.90:0/2701665527 conn(0x7f2230072c10 msgr2=0x7f2230073020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:11.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.972+0000 7f222f200640 1 -- 172.21.15.90:0/2701665527 shutdown_connections 2024-03-24T01:24:11.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.972+0000 7f222f200640 1 -- 172.21.15.90:0/2701665527 wait complete. 2024-03-24T01:24:11.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.974+0000 7f222f200640 1 Processor -- start 2024-03-24T01:24:11.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.974+0000 7f222f200640 1 -- start start 2024-03-24T01:24:11.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.975+0000 7f222f200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22300d0a10 0x7f2230153e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:11.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.975+0000 7f222f200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22300d1e90 0x7f22300d3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:11.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.975+0000 7f222f200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2230161380 0x7f22300d3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:11.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.975+0000 7f222f200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22300b98d0 con 0x7f22300d1e90 2024-03-24T01:24:11.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.975+0000 7f222f200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22300b8cd0 con 0x7f2230161380 2024-03-24T01:24:11.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.975+0000 7f222f200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22300b9a50 con 0x7f22300d0a10 2024-03-24T01:24:11.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.976+0000 7f2225400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22300d1e90 0x7f22300d3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:11.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.976+0000 7f2225400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22300d1e90 0x7f22300d3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50644/0 (socket says 172.21.15.90:50644) 2024-03-24T01:24:11.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.976+0000 7f2225400640 1 -- 172.21.15.90:0/4181594481 learned_addr learned my addr 172.21.15.90:0/4181594481 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:11.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.976+0000 7f222e800640 1 --2- 172.21.15.90:0/4181594481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2230161380 0x7f22300d3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:11.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.976+0000 7f222de00640 1 --2- 172.21.15.90:0/4181594481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22300d0a10 0x7f2230153e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:11.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.976+0000 7f2225400640 1 -- 172.21.15.90:0/4181594481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22300d0a10 msgr2=0x7f2230153e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:11.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.976+0000 7f2225400640 1 --2- 172.21.15.90:0/4181594481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22300d0a10 0x7f2230153e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:11.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.976+0000 7f2225400640 1 -- 172.21.15.90:0/4181594481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2230161380 msgr2=0x7f22300d3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:11.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.976+0000 7f2225400640 1 --2- 172.21.15.90:0/4181594481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2230161380 0x7f22300d3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:11.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.976+0000 7f2225400640 1 -- 172.21.15.90:0/4181594481 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22301f3ac0 con 0x7f22300d1e90 2024-03-24T01:24:11.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.976+0000 7f2225400640 1 --2- 172.21.15.90:0/4181594481 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22300d1e90 0x7f22300d3970 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f221800d800 tx=0x7f221800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:11.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.977+0000 7f2226a00640 1 -- 172.21.15.90:0/4181594481 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2218018020 con 0x7f22300d1e90 2024-03-24T01:24:11.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.977+0000 7f2226a00640 1 -- 172.21.15.90:0/4181594481 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2218009640 con 0x7f22300d1e90 2024-03-24T01:24:11.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.977+0000 7f2226a00640 1 -- 172.21.15.90:0/4181594481 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2218009a10 con 0x7f22300d1e90 2024-03-24T01:24:11.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.977+0000 7f222f200640 1 -- 172.21.15.90:0/4181594481 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22301f3da0 con 0x7f22300d1e90 2024-03-24T01:24:11.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.977+0000 7f222f200640 1 -- 172.21.15.90:0/4181594481 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22301f4360 con 0x7f22300d1e90 2024-03-24T01:24:11.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.978+0000 7f222f200640 1 -- 172.21.15.90:0/4181594481 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f223004f400 con 0x7f22300d1e90 2024-03-24T01:24:11.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.979+0000 7f2226a00640 1 -- 172.21.15.90:0/4181594481 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f221801b020 con 0x7f22300d1e90 2024-03-24T01:24:11.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.980+0000 7f2226a00640 1 --2- 172.21.15.90:0/4181594481 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f220c06bc00 0x7f220c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:11.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.980+0000 7f2226a00640 1 -- 172.21.15.90:0/4181594481 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6007+0+0 (secure 0 0 0) 0x7f221808acd0 con 0x7f22300d1e90 2024-03-24T01:24:11.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.980+0000 7f222de00640 1 --2- 172.21.15.90:0/4181594481 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f220c06bc00 0x7f220c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:11.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.981+0000 7f222de00640 1 --2- 172.21.15.90:0/4181594481 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f220c06bc00 0x7f220c06e0c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2220009a30 tx=0x7f2220008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:11.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:11.984+0000 7f2226a00640 1 -- 172.21.15.90:0/4181594481 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2218022030 con 0x7f22300d1e90 2024-03-24T01:24:12.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:12.182+0000 7f222f200640 1 -- 172.21.15.90:0/4181594481 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f22300b6cb0 con 0x7f22300d1e90 2024-03-24T01:24:13.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:13.982+0000 7f2226a00640 1 -- 172.21.15.90:0/4181594481 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v22) v1 ==== 228+0+0 (secure 0 0 0) 0x7f2218059680 con 0x7f22300d1e90 2024-03-24T01:24:13.983 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_data_ec' created 2024-03-24T01:24:13.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:13.987+0000 7f222f200640 1 -- 172.21.15.90:0/4181594481 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f220c06bc00 msgr2=0x7f220c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:13.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:13.987+0000 7f222f200640 1 --2- 172.21.15.90:0/4181594481 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f220c06bc00 0x7f220c06e0c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2220009a30 tx=0x7f2220008040 comp rx=0 tx=0).stop 2024-03-24T01:24:13.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:13.987+0000 7f222f200640 1 -- 172.21.15.90:0/4181594481 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22300d1e90 msgr2=0x7f22300d3970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:13.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:13.987+0000 7f222f200640 1 --2- 172.21.15.90:0/4181594481 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22300d1e90 0x7f22300d3970 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f221800d800 tx=0x7f221800dcc0 comp rx=0 tx=0).stop 2024-03-24T01:24:13.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:13.987+0000 7f222f200640 1 -- 172.21.15.90:0/4181594481 shutdown_connections 2024-03-24T01:24:13.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:13.987+0000 7f222f200640 1 --2- 172.21.15.90:0/4181594481 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f220c06bc00 0x7f220c06e0c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:13.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:13.987+0000 7f222f200640 1 --2- 172.21.15.90:0/4181594481 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22300d1e90 0x7f22300d3970 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:13.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:13.987+0000 7f222f200640 1 --2- 172.21.15.90:0/4181594481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2230161380 0x7f22300d3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:13.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:13.987+0000 7f222f200640 1 --2- 172.21.15.90:0/4181594481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22300d0a10 0x7f2230153e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:13.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:13.987+0000 7f222f200640 1 -- 172.21.15.90:0/4181594481 >> 172.21.15.90:0/4181594481 conn(0x7f2230072c10 msgr2=0x7f2230160590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:13.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:13.988+0000 7f222f200640 1 -- 172.21.15.90:0/4181594481 shutdown_connections 2024-03-24T01:24:13.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:13.988+0000 7f222f200640 1 -- 172.21.15.90:0/4181594481 wait complete. 2024-03-24T01:24:14.019 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-24T01:24:14.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.204+0000 7f9fa3400640 1 Processor -- start 2024-03-24T01:24:14.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.204+0000 7f9fa3400640 1 -- start start 2024-03-24T01:24:14.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.205+0000 7f9fa3400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f9c1452b0 0x7f9f9c071990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:14.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.206+0000 7f9fa3400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f9c0b2890 con 0x7f9f9c071ed0 2024-03-24T01:24:14.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.206+0000 7f9fa3400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f9c0b2b70 con 0x7f9f9c0cf200 2024-03-24T01:24:14.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.206+0000 7f9fa3400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c072330 con 0x7f9f9c1452b0 2024-03-24T01:24:14.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.206+0000 7f9f93e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f9f9c071ed0 0x7f9f9c0ceaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:37484/0 (socket says 172.21.15.90:37484) 2024-03-24T01:24:14.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.206+0000 7f9f93e00640 1 -- 172.21.15.90:0/2862589276 learned_addr learned my addr 172.21.15.90:0/2862589276 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:14.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.206+0000 7f9fa0a00640 1 --2- 172.21.15.90:0/2862589276 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f9c1452b0 0x7f9f9c071990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:14.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.206+0000 7f9f9be00640 1 -- 172.21.15.90:0/2862589276 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2525430566 0 0) 0x7f9f9c0b2890 con 0x7f9f9c071ed0 2024-03-24T01:24:14.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.206+0000 7f9f9be00640 1 -- 172.21.15.90:0/2862589276 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f80003520 con 0x7f9f9c071ed0 2024-03-24T01:24:14.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.207+0000 7f9f9be00640 1 -- 172.21.15.90:0/2862589276 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2522509122 0 0) 0x7f9f9c0b2b70 con 0x7f9f9c0cf200 2024-03-24T01:24:14.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.207+0000 7f9f9be00640 1 -- 172.21.15.90:0/2862589276 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f9c0b2890 con 0x7f9f9c0cf200 2024-03-24T01:24:14.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.207+0000 7f9f9be00640 1 -- 172.21.15.90:0/2862589276 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3875865699 0 0) 0x7f9f80003520 con 0x7f9f9c071ed0 2024-03-24T01:24:14.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.207+0000 7f9f9be00640 1 -- 172.21.15.90:0/2862589276 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f9c1452b0 msgr2=0x7f9f9c071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:14.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.207+0000 7f9f9be00640 1 --2- 172.21.15.90:0/2862589276 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f9c1452b0 0x7f9f9c071990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:14.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.207+0000 7f9f9be00640 1 -- 172.21.15.90:0/2862589276 >> v1:172.21.15.120:6789/0 conn(0x7f9f9c0cf200 legacy=0x7f9f9c0b3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:14.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.207+0000 7f9f9be00640 1 -- 172.21.15.90:0/2862589276 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f9c0b2e50 con 0x7f9f9c071ed0 2024-03-24T01:24:14.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.207+0000 7f9f9be00640 1 -- 172.21.15.90:0/2862589276 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f9f840015f0 con 0x7f9f9c071ed0 2024-03-24T01:24:14.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.207+0000 7f9f9be00640 1 -- 172.21.15.90:0/2862589276 >> v1:172.21.15.90:6789/0 conn(0x7f9f9c071ed0 legacy=0x7f9f9c0ceaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:14.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.207+0000 7f9f9be00640 1 --2- 172.21.15.90:0/2862589276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f80003b70 0x7f9f80003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:14.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.207+0000 7f9f9be00640 1 --2- 172.21.15.90:0/2862589276 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f9c1452b0 0x7f9f80004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:14.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.207+0000 7f9f9be00640 1 --2- 172.21.15.90:0/2862589276 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f80004d40 0x7f9f800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:14.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.207+0000 7f9f9be00640 1 -- 172.21.15.90:0/2862589276 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8000ba70 con 0x7f9f80004d40 2024-03-24T01:24:14.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.207+0000 7f9f9be00640 1 -- 172.21.15.90:0/2862589276 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8000bbf0 con 0x7f9f80003b70 2024-03-24T01:24:14.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.207+0000 7f9f9be00640 1 -- 172.21.15.90:0/2862589276 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8000bdf0 con 0x7f9f9c1452b0 2024-03-24T01:24:14.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.207+0000 7f9fa1400640 1 --2- 172.21.15.90:0/2862589276 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f80004d40 0x7f9f800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:14.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.207+0000 7f9fa0a00640 1 --2- 172.21.15.90:0/2862589276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f80003b70 0x7f9f80003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:14.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.207+0000 7f9f93e00640 1 --2- 172.21.15.90:0/2862589276 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f9c1452b0 0x7f9f80004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:14.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.208+0000 7f9fa1400640 1 -- 172.21.15.90:0/2862589276 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f9c1452b0 msgr2=0x7f9f80004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:14.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.208+0000 7f9fa1400640 1 --2- 172.21.15.90:0/2862589276 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f9c1452b0 0x7f9f80004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:14.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.208+0000 7f9fa1400640 1 -- 172.21.15.90:0/2862589276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f80003b70 msgr2=0x7f9f80003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:14.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.208+0000 7f9fa1400640 1 --2- 172.21.15.90:0/2862589276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f80003b70 0x7f9f80003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:14.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.208+0000 7f9fa1400640 1 -- 172.21.15.90:0/2862589276 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f80003520 con 0x7f9f80004d40 2024-03-24T01:24:14.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.209+0000 7f9fa1400640 1 --2- 172.21.15.90:0/2862589276 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f80004d40 0x7f9f800051c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9f94001fd0 tx=0x7f9f80005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f17c3acb6002eb2a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:14.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.209+0000 7f9f9be00640 1 -- 172.21.15.90:0/2862589276 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f94001bf0 con 0x7f9f80004d40 2024-03-24T01:24:14.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.209+0000 7f9f9be00640 1 -- 172.21.15.90:0/2862589276 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f94003b70 con 0x7f9f80004d40 2024-03-24T01:24:14.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.209+0000 7f9f9be00640 1 -- 172.21.15.90:0/2862589276 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f940050d0 con 0x7f9f80004d40 2024-03-24T01:24:14.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.209+0000 7f9fa3400640 1 -- 172.21.15.90:0/2862589276 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f80004d40 msgr2=0x7f9f800051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:14.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.209+0000 7f9fa3400640 1 --2- 172.21.15.90:0/2862589276 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f80004d40 0x7f9f800051c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9f94001fd0 tx=0x7f9f80005830 comp rx=0 tx=0).stop 2024-03-24T01:24:14.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.209+0000 7f9fa1400640 1 -- 172.21.15.90:0/2862589276 reap_dead start 2024-03-24T01:24:14.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.209+0000 7f9fa3400640 1 -- 172.21.15.90:0/2862589276 shutdown_connections 2024-03-24T01:24:14.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.210+0000 7f9fa3400640 1 -- 172.21.15.90:0/2862589276 >> 172.21.15.90:0/2862589276 conn(0x7f9f9c0cf7e0 msgr2=0x7f9f9c0cfbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:14.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.210+0000 7f9fa3400640 1 -- 172.21.15.90:0/2862589276 shutdown_connections 2024-03-24T01:24:14.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.210+0000 7f9fa3400640 1 -- 172.21.15.90:0/2862589276 wait complete. 2024-03-24T01:24:14.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.211+0000 7f9fa3400640 1 Processor -- start 2024-03-24T01:24:14.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.211+0000 7f9fa3400640 1 -- start start 2024-03-24T01:24:14.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.212+0000 7f9fa3400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f80004d40 0x7f9f9c13baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:14.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.212+0000 7f9fa3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f9c14da20 0x7f9f9c0b0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:14.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.212+0000 7f9fa3400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f9c0b0d80 0x7f9f9c15d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:14.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.212+0000 7f9fa3400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c0b3410 con 0x7f9f9c14da20 2024-03-24T01:24:14.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.212+0000 7f9fa3400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c072990 con 0x7f9f9c0b0d80 2024-03-24T01:24:14.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.212+0000 7f9fa3400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c145690 con 0x7f9f80004d40 2024-03-24T01:24:14.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.213+0000 7f9f93e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f9c14da20 0x7f9f9c0b0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:14.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.213+0000 7f9f93e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f9c14da20 0x7f9f9c0b0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50664/0 (socket says 172.21.15.90:50664) 2024-03-24T01:24:14.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.213+0000 7f9f93e00640 1 -- 172.21.15.90:0/3635295235 learned_addr learned my addr 172.21.15.90:0/3635295235 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:14.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.213+0000 7f9fa0a00640 1 --2- 172.21.15.90:0/3635295235 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f80004d40 0x7f9f9c13baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:14.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.213+0000 7f9fa1400640 1 --2- 172.21.15.90:0/3635295235 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f9c0b0d80 0x7f9f9c15d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:14.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.213+0000 7f9f93e00640 1 -- 172.21.15.90:0/3635295235 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f80004d40 msgr2=0x7f9f9c13baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:14.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.213+0000 7f9f93e00640 1 --2- 172.21.15.90:0/3635295235 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f80004d40 0x7f9f9c13baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:14.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.213+0000 7f9f93e00640 1 -- 172.21.15.90:0/3635295235 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f9c0b0d80 msgr2=0x7f9f9c15d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:14.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.213+0000 7f9f93e00640 1 --2- 172.21.15.90:0/3635295235 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f9c0b0d80 0x7f9f9c15d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:14.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.213+0000 7f9f93e00640 1 -- 172.21.15.90:0/3635295235 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f9c15ddd0 con 0x7f9f9c14da20 2024-03-24T01:24:14.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.213+0000 7f9f93e00640 1 --2- 172.21.15.90:0/3635295235 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f9c14da20 0x7f9f9c0b0840 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9f8400e700 tx=0x7f9f8400ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:14.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.214+0000 7f9f99600640 1 -- 172.21.15.90:0/3635295235 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8400c660 con 0x7f9f9c14da20 2024-03-24T01:24:14.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.214+0000 7f9f99600640 1 -- 172.21.15.90:0/3635295235 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f84018070 con 0x7f9f9c14da20 2024-03-24T01:24:14.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.214+0000 7f9f99600640 1 -- 172.21.15.90:0/3635295235 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8400ca40 con 0x7f9f9c14da20 2024-03-24T01:24:14.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.214+0000 7f9fa3400640 1 -- 172.21.15.90:0/3635295235 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f9c14cf40 con 0x7f9f9c14da20 2024-03-24T01:24:14.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.214+0000 7f9fa3400640 1 -- 172.21.15.90:0/3635295235 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f9c14d500 con 0x7f9f9c14da20 2024-03-24T01:24:14.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.215+0000 7f9fa3400640 1 -- 172.21.15.90:0/3635295235 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f9c0b1190 con 0x7f9f9c14da20 2024-03-24T01:24:14.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.216+0000 7f9f99600640 1 -- 172.21.15.90:0/3635295235 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9f8400cbe0 con 0x7f9f9c14da20 2024-03-24T01:24:14.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.217+0000 7f9f99600640 1 --2- 172.21.15.90:0/3635295235 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9f7806bc00 0x7f9f7806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:14.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.217+0000 7f9f99600640 1 -- 172.21.15.90:0/3635295235 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9f8408abf0 con 0x7f9f9c14da20 2024-03-24T01:24:14.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.217+0000 7f9fa0a00640 1 --2- 172.21.15.90:0/3635295235 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9f7806bc00 0x7f9f7806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:14.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.218+0000 7f9fa0a00640 1 --2- 172.21.15.90:0/3635295235 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9f7806bc00 0x7f9f7806e0c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c009a10 tx=0x7f9f8c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:14.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.222+0000 7f9f99600640 1 -- 172.21.15.90:0/3635295235 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9f9c0b1190 con 0x7f9f9c14da20 2024-03-24T01:24:14.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.417+0000 7f9fa3400640 1 -- 172.21.15.90:0/3635295235 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f9f9c04f400 con 0x7f9f9c14da20 2024-03-24T01:24:14.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.986+0000 7f9f99600640 1 -- 172.21.15.90:0/3635295235 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v23) v1 ==== 169+0+0 (secure 0 0 0) 0x7f9f84059320 con 0x7f9f9c14da20 2024-03-24T01:24:14.988 INFO:teuthology.orchestra.run.smithi090.stderr:set pool 4 allow_ec_overwrites to true 2024-03-24T01:24:14.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.990+0000 7f9fa3400640 1 -- 172.21.15.90:0/3635295235 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9f7806bc00 msgr2=0x7f9f7806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:14.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.990+0000 7f9fa3400640 1 --2- 172.21.15.90:0/3635295235 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9f7806bc00 0x7f9f7806e0c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c009a10 tx=0x7f9f8c008040 comp rx=0 tx=0).stop 2024-03-24T01:24:14.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.990+0000 7f9fa3400640 1 -- 172.21.15.90:0/3635295235 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f9c14da20 msgr2=0x7f9f9c0b0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:14.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.990+0000 7f9fa3400640 1 --2- 172.21.15.90:0/3635295235 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f9c14da20 0x7f9f9c0b0840 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9f8400e700 tx=0x7f9f8400ebc0 comp rx=0 tx=0).stop 2024-03-24T01:24:14.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.991+0000 7f9fa3400640 1 -- 172.21.15.90:0/3635295235 shutdown_connections 2024-03-24T01:24:14.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.991+0000 7f9fa3400640 1 --2- 172.21.15.90:0/3635295235 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9f7806bc00 0x7f9f7806e0c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:14.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.991+0000 7f9fa3400640 1 --2- 172.21.15.90:0/3635295235 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f9c14da20 0x7f9f9c0b0840 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:14.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.991+0000 7f9fa3400640 1 --2- 172.21.15.90:0/3635295235 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f9c0b0d80 0x7f9f9c15d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:14.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.991+0000 7f9fa3400640 1 --2- 172.21.15.90:0/3635295235 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9f80004d40 0x7f9f9c13baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:14.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.991+0000 7f9fa3400640 1 -- 172.21.15.90:0/3635295235 >> 172.21.15.90:0/3635295235 conn(0x7f9f9c0cf7e0 msgr2=0x7f9f9c0af700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:14.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.991+0000 7f9fa3400640 1 -- 172.21.15.90:0/3635295235 shutdown_connections 2024-03-24T01:24:14.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:14.992+0000 7f9fa3400640 1 -- 172.21.15.90:0/3635295235 wait complete. 2024-03-24T01:24:15.026 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-24T01:24:15.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.208+0000 7fe0aec00640 1 Processor -- start 2024-03-24T01:24:15.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.208+0000 7fe0aec00640 1 -- start start 2024-03-24T01:24:15.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.209+0000 7fe0aec00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0a8145170 0x7fe0a80cf810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:15.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.210+0000 7fe0aec00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a806d990 con 0x7fe0a806ea30 2024-03-24T01:24:15.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.210+0000 7fe0aec00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a806dc70 con 0x7fe0a80cf190 2024-03-24T01:24:15.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.210+0000 7fe0aec00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a806df50 con 0x7fe0a8145170 2024-03-24T01:24:15.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.210+0000 7fe0ace00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe0a806ea30 0x7fe0a80cea80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:37490/0 (socket says 172.21.15.90:37490) 2024-03-24T01:24:15.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.210+0000 7fe0ace00640 1 -- 172.21.15.90:0/3760545237 learned_addr learned my addr 172.21.15.90:0/3760545237 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:15.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.210+0000 7fe0ad800640 1 --2- 172.21.15.90:0/3760545237 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0a8145170 0x7fe0a80cf810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:15.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.210+0000 7fe0a7e00640 1 -- 172.21.15.90:0/3760545237 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2587300713 0 0) 0x7fe0a806d990 con 0x7fe0a806ea30 2024-03-24T01:24:15.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.210+0000 7fe0a7e00640 1 -- 172.21.15.90:0/3760545237 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe090003520 con 0x7fe0a806ea30 2024-03-24T01:24:15.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.211+0000 7fe0a7e00640 1 -- 172.21.15.90:0/3760545237 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4281641865 0 0) 0x7fe090003520 con 0x7fe0a806ea30 2024-03-24T01:24:15.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.211+0000 7fe0a7e00640 1 -- 172.21.15.90:0/3760545237 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0a8145170 msgr2=0x7fe0a80cf810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:15.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.211+0000 7fe0a7e00640 1 --2- 172.21.15.90:0/3760545237 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0a8145170 0x7fe0a80cf810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:15.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.211+0000 7fe0a7e00640 1 -- 172.21.15.90:0/3760545237 >> v1:172.21.15.120:6789/0 conn(0x7fe0a80cf190 legacy=0x7fe0a815c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:15.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.211+0000 7fe0a7e00640 1 -- 172.21.15.90:0/3760545237 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a806e230 con 0x7fe0a806ea30 2024-03-24T01:24:15.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.211+0000 7fe0a7e00640 1 -- 172.21.15.90:0/3760545237 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7fe098001670 con 0x7fe0a806ea30 2024-03-24T01:24:15.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.211+0000 7fe0a7e00640 1 -- 172.21.15.90:0/3760545237 >> v1:172.21.15.90:6789/0 conn(0x7fe0a806ea30 legacy=0x7fe0a80cea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:15.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.211+0000 7fe0a7e00640 1 --2- 172.21.15.90:0/3760545237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0900039e0 0x7fe090003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:15.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.211+0000 7fe0a7e00640 1 --2- 172.21.15.90:0/3760545237 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe090004390 0x7fe090004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:15.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.211+0000 7fe0a7e00640 1 --2- 172.21.15.90:0/3760545237 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0a8145170 0x7fe0900050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:15.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.211+0000 7fe0a7e00640 1 -- 172.21.15.90:0/3760545237 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a806df50 con 0x7fe090004390 2024-03-24T01:24:15.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.211+0000 7fe0a7e00640 1 -- 172.21.15.90:0/3760545237 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe09000b9c0 con 0x7fe0900039e0 2024-03-24T01:24:15.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.211+0000 7fe0a7e00640 1 -- 172.21.15.90:0/3760545237 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe09000bbc0 con 0x7fe0a8145170 2024-03-24T01:24:15.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.211+0000 7fe0ace00640 1 --2- 172.21.15.90:0/3760545237 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe090004390 0x7fe090004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:15.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.211+0000 7fe0ad800640 1 --2- 172.21.15.90:0/3760545237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0900039e0 0x7fe090003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:15.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.211+0000 7fe0ae200640 1 --2- 172.21.15.90:0/3760545237 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0a8145170 0x7fe0900050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:15.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.212+0000 7fe0ace00640 1 -- 172.21.15.90:0/3760545237 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0a8145170 msgr2=0x7fe0900050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:15.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.212+0000 7fe0ace00640 1 --2- 172.21.15.90:0/3760545237 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0a8145170 0x7fe0900050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:15.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.212+0000 7fe0ace00640 1 -- 172.21.15.90:0/3760545237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0900039e0 msgr2=0x7fe090003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:15.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.212+0000 7fe0ace00640 1 --2- 172.21.15.90:0/3760545237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0900039e0 0x7fe090003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:15.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.212+0000 7fe0ace00640 1 -- 172.21.15.90:0/3760545237 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe090003520 con 0x7fe090004390 2024-03-24T01:24:15.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.212+0000 7fe0ad800640 1 --2- 172.21.15.90:0/3760545237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0900039e0 0x7fe090003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:24:15.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.212+0000 7fe0ae200640 1 --2- 172.21.15.90:0/3760545237 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0a8145170 0x7fe0900050c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:15.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.213+0000 7fe0ace00640 1 --2- 172.21.15.90:0/3760545237 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe090004390 0x7fe090004860 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe09800c240 tx=0x7fe090005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8569ae5ba4c810b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:15.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.213+0000 7fe0a7e00640 1 -- 172.21.15.90:0/3760545237 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe098007d10 con 0x7fe090004390 2024-03-24T01:24:15.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.213+0000 7fe0a7e00640 1 -- 172.21.15.90:0/3760545237 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe098030070 con 0x7fe090004390 2024-03-24T01:24:15.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.213+0000 7fe0a7e00640 1 -- 172.21.15.90:0/3760545237 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09802cd40 con 0x7fe090004390 2024-03-24T01:24:15.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.213+0000 7fe0aec00640 1 -- 172.21.15.90:0/3760545237 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe090004390 msgr2=0x7fe090004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:15.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.213+0000 7fe0aec00640 1 --2- 172.21.15.90:0/3760545237 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe090004390 0x7fe090004860 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe09800c240 tx=0x7fe090005830 comp rx=0 tx=0).stop 2024-03-24T01:24:15.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.213+0000 7fe0ae200640 1 -- 172.21.15.90:0/3760545237 reap_dead start 2024-03-24T01:24:15.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.213+0000 7fe0aec00640 1 -- 172.21.15.90:0/3760545237 shutdown_connections 2024-03-24T01:24:15.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.213+0000 7fe0aec00640 1 -- 172.21.15.90:0/3760545237 >> 172.21.15.90:0/3760545237 conn(0x7fe0a806b0c0 msgr2=0x7fe0a806b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:15.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.214+0000 7fe0aec00640 1 -- 172.21.15.90:0/3760545237 shutdown_connections 2024-03-24T01:24:15.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.214+0000 7fe0aec00640 1 -- 172.21.15.90:0/3760545237 wait complete. 2024-03-24T01:24:15.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.215+0000 7fe0aec00640 1 Processor -- start 2024-03-24T01:24:15.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.215+0000 7fe0aec00640 1 -- start start 2024-03-24T01:24:15.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.216+0000 7fe0aec00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe090004390 0x7fe0a8138660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:15.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.216+0000 7fe0aec00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0a80ac8e0 0x7fe0a815b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:15.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.216+0000 7fe0aec00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0a8137ac0 0x7fe0a8137f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:15.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.216+0000 7fe0aec00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a81456d0 con 0x7fe0a80ac8e0 2024-03-24T01:24:15.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.216+0000 7fe0aec00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a8145550 con 0x7fe090004390 2024-03-24T01:24:15.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.216+0000 7fe0aec00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a815c7b0 con 0x7fe0a8137ac0 2024-03-24T01:24:15.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.217+0000 7fe0ace00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0a80ac8e0 0x7fe0a815b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:15.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.217+0000 7fe0ace00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0a80ac8e0 0x7fe0a815b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50672/0 (socket says 172.21.15.90:50672) 2024-03-24T01:24:15.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.217+0000 7fe0ace00640 1 -- 172.21.15.90:0/735937142 learned_addr learned my addr 172.21.15.90:0/735937142 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:15.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.217+0000 7fe0ad800640 1 --2- 172.21.15.90:0/735937142 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe090004390 0x7fe0a8138660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:15.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.217+0000 7fe0ace00640 1 -- 172.21.15.90:0/735937142 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0a8137ac0 msgr2=0x7fe0a8137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:15.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.217+0000 7fe0ae200640 1 --2- 172.21.15.90:0/735937142 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0a8137ac0 0x7fe0a8137f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:15.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.217+0000 7fe0ace00640 1 --2- 172.21.15.90:0/735937142 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0a8137ac0 0x7fe0a8137f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:15.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.217+0000 7fe0ace00640 1 -- 172.21.15.90:0/735937142 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe090004390 msgr2=0x7fe0a8138660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:15.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.217+0000 7fe0ace00640 1 --2- 172.21.15.90:0/735937142 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe090004390 0x7fe0a8138660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:15.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.217+0000 7fe0ace00640 1 -- 172.21.15.90:0/735937142 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a80ae4e0 con 0x7fe0a80ac8e0 2024-03-24T01:24:15.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.217+0000 7fe0ad800640 1 --2- 172.21.15.90:0/735937142 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe090004390 0x7fe0a8138660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:15.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.217+0000 7fe0ace00640 1 --2- 172.21.15.90:0/735937142 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0a80ac8e0 0x7fe0a815b490 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe09802c550 tx=0x7fe0a806e400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:15.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.218+0000 7fe0a5600640 1 -- 172.21.15.90:0/735937142 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe098001c50 con 0x7fe0a80ac8e0 2024-03-24T01:24:15.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.218+0000 7fe0a5600640 1 -- 172.21.15.90:0/735937142 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe098004bf0 con 0x7fe0a80ac8e0 2024-03-24T01:24:15.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.218+0000 7fe0aec00640 1 -- 172.21.15.90:0/735937142 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0a80ae7c0 con 0x7fe0a80ac8e0 2024-03-24T01:24:15.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.218+0000 7fe0a5600640 1 -- 172.21.15.90:0/735937142 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe098007070 con 0x7fe0a80ac8e0 2024-03-24T01:24:15.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.218+0000 7fe0aec00640 1 -- 172.21.15.90:0/735937142 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0a80aed80 con 0x7fe0a80ac8e0 2024-03-24T01:24:15.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.219+0000 7fe0aec00640 1 -- 172.21.15.90:0/735937142 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0a804f400 con 0x7fe0a80ac8e0 2024-03-24T01:24:15.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.220+0000 7fe0a5600640 1 -- 172.21.15.90:0/735937142 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe098007210 con 0x7fe0a80ac8e0 2024-03-24T01:24:15.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.221+0000 7fe0a5600640 1 --2- 172.21.15.90:0/735937142 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe08c06bbd0 0x7fe08c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:15.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.221+0000 7fe0a5600640 1 -- 172.21.15.90:0/735937142 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe0980ac720 con 0x7fe0a80ac8e0 2024-03-24T01:24:15.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.221+0000 7fe0ad800640 1 --2- 172.21.15.90:0/735937142 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe08c06bbd0 0x7fe08c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:15.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.222+0000 7fe0ad800640 1 --2- 172.21.15.90:0/735937142 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe08c06bbd0 0x7fe08c06e090 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe0a80ae0c0 tx=0x7fe09c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:15.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.225+0000 7fe0a5600640 1 -- 172.21.15.90:0/735937142 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe09807aef0 con 0x7fe0a80ac8e0 2024-03-24T01:24:15.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:15.456+0000 7fe0aec00640 1 -- 172.21.15.90:0/735937142 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7fe0a80728e0 con 0x7fe0a80ac8e0 2024-03-24T01:24:16.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.018+0000 7fe0a5600640 1 -- 172.21.15.90:0/735937142 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v6) v1 ==== 137+0+0 (secure 0 0 0) 0x7fe09807a7f0 con 0x7fe0a80ac8e0 2024-03-24T01:24:16.020 INFO:teuthology.orchestra.run.smithi090.stderr:added data pool 4 to fsmap 2024-03-24T01:24:16.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.022+0000 7fe0aec00640 1 -- 172.21.15.90:0/735937142 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe08c06bbd0 msgr2=0x7fe08c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:16.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.022+0000 7fe0aec00640 1 --2- 172.21.15.90:0/735937142 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe08c06bbd0 0x7fe08c06e090 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe0a80ae0c0 tx=0x7fe09c008040 comp rx=0 tx=0).stop 2024-03-24T01:24:16.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.022+0000 7fe0aec00640 1 -- 172.21.15.90:0/735937142 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0a80ac8e0 msgr2=0x7fe0a815b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:16.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.022+0000 7fe0aec00640 1 --2- 172.21.15.90:0/735937142 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0a80ac8e0 0x7fe0a815b490 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe09802c550 tx=0x7fe0a806e400 comp rx=0 tx=0).stop 2024-03-24T01:24:16.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.022+0000 7fe0aec00640 1 -- 172.21.15.90:0/735937142 shutdown_connections 2024-03-24T01:24:16.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.022+0000 7fe0aec00640 1 --2- 172.21.15.90:0/735937142 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe08c06bbd0 0x7fe08c06e090 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:16.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.022+0000 7fe0aec00640 1 --2- 172.21.15.90:0/735937142 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0a80ac8e0 0x7fe0a815b490 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:16.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.022+0000 7fe0aec00640 1 --2- 172.21.15.90:0/735937142 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0a8137ac0 0x7fe0a8137f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:16.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.022+0000 7fe0aec00640 1 --2- 172.21.15.90:0/735937142 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe090004390 0x7fe0a8138660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:16.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.022+0000 7fe0aec00640 1 -- 172.21.15.90:0/735937142 >> 172.21.15.90:0/735937142 conn(0x7fe0a806b0c0 msgr2=0x7fe0a80cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:16.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.022+0000 7fe0aec00640 1 -- 172.21.15.90:0/735937142 shutdown_connections 2024-03-24T01:24:16.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.022+0000 7fe0aec00640 1 -- 172.21.15.90:0/735937142 wait complete. 2024-03-24T01:24:16.042 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:24:16.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.241+0000 7f9953a00640 1 Processor -- start 2024-03-24T01:24:16.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.241+0000 7f9953a00640 1 -- start start 2024-03-24T01:24:16.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.243+0000 7f9953a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f994c1454e0 0x7f994c0cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:16.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.243+0000 7f9953a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f994c06f000 0x7f994c06f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:16.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.243+0000 7f9953a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f994c0cdc50 0x7f994c0ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:16.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.243+0000 7f9953a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f994c0cc430 con 0x7f994c0cdc50 2024-03-24T01:24:16.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.243+0000 7f9953a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f994c0cc660 con 0x7f994c1454e0 2024-03-24T01:24:16.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.243+0000 7f9953a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994c0cc920 con 0x7f994c06f000 2024-03-24T01:24:16.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.243+0000 7f9951a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f994c0cdc50 0x7f994c0ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:16.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.243+0000 7f9951a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f994c0cdc50 0x7f994c0ce0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50674/0 (socket says 172.21.15.90:50674) 2024-03-24T01:24:16.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.243+0000 7f9951a00640 1 -- 172.21.15.90:0/637487994 learned_addr learned my addr 172.21.15.90:0/637487994 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:16.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.243+0000 7f994be00640 1 --2- 172.21.15.90:0/637487994 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f994c06f000 0x7f994c06f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:16.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.243+0000 7f9951000640 1 --2- 172.21.15.90:0/637487994 >> v2:172.21.15.120:3300/0 conn(0x7f994c1454e0 0x7f994c0cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:16.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.243+0000 7f9951a00640 1 -- 172.21.15.90:0/637487994 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f994c06f000 msgr2=0x7f994c06f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:16.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.243+0000 7f9951a00640 1 --2- 172.21.15.90:0/637487994 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f994c06f000 0x7f994c06f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:16.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.243+0000 7f9951a00640 1 -- 172.21.15.90:0/637487994 >> v2:172.21.15.120:3300/0 conn(0x7f994c1454e0 msgr2=0x7f994c0cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:16.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.243+0000 7f9951a00640 1 --2- 172.21.15.90:0/637487994 >> v2:172.21.15.120:3300/0 conn(0x7f994c1454e0 0x7f994c0cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:16.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.244+0000 7f9951a00640 1 -- 172.21.15.90:0/637487994 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994c0ccc00 con 0x7f994c0cdc50 2024-03-24T01:24:16.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.244+0000 7f994be00640 1 --2- 172.21.15.90:0/637487994 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f994c06f000 0x7f994c06f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:16.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.245+0000 7f9951a00640 1 --2- 172.21.15.90:0/637487994 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f994c0cdc50 0x7f994c0ce0e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f994c0b4df0 tx=0x7f994402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ec878b83bbb1920 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:16.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.245+0000 7f994b400640 1 -- 172.21.15.90:0/637487994 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994402ca80 con 0x7f994c0cdc50 2024-03-24T01:24:16.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.245+0000 7f994b400640 1 -- 172.21.15.90:0/637487994 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f994402cc20 con 0x7f994c0cdc50 2024-03-24T01:24:16.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.245+0000 7f994b400640 1 -- 172.21.15.90:0/637487994 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99440343f0 con 0x7f994c0cdc50 2024-03-24T01:24:16.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.245+0000 7f9953a00640 1 -- 172.21.15.90:0/637487994 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f994c0cdc50 msgr2=0x7f994c0ce0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:16.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.245+0000 7f9953a00640 1 --2- 172.21.15.90:0/637487994 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f994c0cdc50 0x7f994c0ce0e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f994c0b4df0 tx=0x7f994402bee0 comp rx=0 tx=0).stop 2024-03-24T01:24:16.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.245+0000 7f9953a00640 1 -- 172.21.15.90:0/637487994 shutdown_connections 2024-03-24T01:24:16.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.245+0000 7f9953a00640 1 --2- 172.21.15.90:0/637487994 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f994c06f000 0x7f994c06f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:16.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.245+0000 7f9953a00640 1 --2- 172.21.15.90:0/637487994 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f994c0cdc50 0x7f994c0ce0e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:16.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.245+0000 7f9953a00640 1 --2- 172.21.15.90:0/637487994 >> v2:172.21.15.120:3300/0 conn(0x7f994c1454e0 0x7f994c0cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:16.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.246+0000 7f9953a00640 1 -- 172.21.15.90:0/637487994 >> 172.21.15.90:0/637487994 conn(0x7f994c06b2a0 msgr2=0x7f994c06b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:16.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.246+0000 7f9953a00640 1 -- 172.21.15.90:0/637487994 shutdown_connections 2024-03-24T01:24:16.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.246+0000 7f9953a00640 1 -- 172.21.15.90:0/637487994 wait complete. 2024-03-24T01:24:16.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.247+0000 7f9953a00640 1 Processor -- start 2024-03-24T01:24:16.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.247+0000 7f9953a00640 1 -- start start 2024-03-24T01:24:16.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.248+0000 7f9953a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f994c06f000 0x7f994c1e6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:16.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.248+0000 7f9953a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f994c0cdc50 0x7f994c0cb320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:16.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.248+0000 7f9953a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f994c1454e0 0x7f994c1f4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:16.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.248+0000 7f9953a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994c06d820 con 0x7f994c06f000 2024-03-24T01:24:16.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.248+0000 7f9953a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994c0cd1c0 con 0x7f994c0cdc50 2024-03-24T01:24:16.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.248+0000 7f9953a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994c06d9a0 con 0x7f994c1454e0 2024-03-24T01:24:16.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.248+0000 7f9951000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f994c06f000 0x7f994c1e6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:16.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.249+0000 7f9951000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f994c06f000 0x7f994c1e6710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50682/0 (socket says 172.21.15.90:50682) 2024-03-24T01:24:16.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.249+0000 7f9951000640 1 -- 172.21.15.90:0/1172058335 learned_addr learned my addr 172.21.15.90:0/1172058335 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:16.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.249+0000 7f9951000640 1 -- 172.21.15.90:0/1172058335 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f994c1454e0 msgr2=0x7f994c1f4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:16.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.249+0000 7f9951000640 1 --2- 172.21.15.90:0/1172058335 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f994c1454e0 0x7f994c1f4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:16.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.249+0000 7f994be00640 1 --2- 172.21.15.90:0/1172058335 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f994c0cdc50 0x7f994c0cb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:16.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.249+0000 7f9951000640 1 -- 172.21.15.90:0/1172058335 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f994c0cdc50 msgr2=0x7f994c0cb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:16.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.249+0000 7f9951000640 1 --2- 172.21.15.90:0/1172058335 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f994c0cdc50 0x7f994c0cb320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:16.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.249+0000 7f9951000640 1 -- 172.21.15.90:0/1172058335 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994c0cb860 con 0x7f994c06f000 2024-03-24T01:24:16.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.249+0000 7f9951000640 1 --2- 172.21.15.90:0/1172058335 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f994c06f000 0x7f994c1e6710 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f994000d5f0 tx=0x7f994000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:16.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.249+0000 7f9948c00640 1 -- 172.21.15.90:0/1172058335 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99400125f0 con 0x7f994c06f000 2024-03-24T01:24:16.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.249+0000 7f9948c00640 1 -- 172.21.15.90:0/1172058335 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9940019070 con 0x7f994c06f000 2024-03-24T01:24:16.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.249+0000 7f9953a00640 1 -- 172.21.15.90:0/1172058335 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f994c1f4e90 con 0x7f994c06f000 2024-03-24T01:24:16.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.249+0000 7f9948c00640 1 -- 172.21.15.90:0/1172058335 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99400129d0 con 0x7f994c06f000 2024-03-24T01:24:16.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.250+0000 7f9953a00640 1 -- 172.21.15.90:0/1172058335 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f994c1e5950 con 0x7f994c06f000 2024-03-24T01:24:16.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.251+0000 7f9953a00640 1 -- 172.21.15.90:0/1172058335 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f994c073570 con 0x7f994c06f000 2024-03-24T01:24:16.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.251+0000 7f9948c00640 1 -- 172.21.15.90:0/1172058335 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9940012b70 con 0x7f994c06f000 2024-03-24T01:24:16.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.252+0000 7f9948c00640 1 --2- 172.21.15.90:0/1172058335 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f992c06bbd0 0x7f992c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:16.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.252+0000 7f9948c00640 1 -- 172.21.15.90:0/1172058335 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f994008b0f0 con 0x7f994c06f000 2024-03-24T01:24:16.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.253+0000 7f994be00640 1 --2- 172.21.15.90:0/1172058335 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f992c06bbd0 0x7f992c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:16.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.254+0000 7f994be00640 1 --2- 172.21.15.90:0/1172058335 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f992c06bbd0 0x7f992c06e090 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f994c1e78e0 tx=0x7f9938008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:16.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.257+0000 7f9948c00640 1 -- 172.21.15.90:0/1172058335 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9940059890 con 0x7f994c06f000 2024-03-24T01:24:16.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.492+0000 7f9953a00640 1 -- 172.21.15.90:0/1172058335 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f994c04f4e0 con 0x7f994c06f000 2024-03-24T01:24:16.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.493+0000 7f9948c00640 1 -- 172.21.15.90:0/1172058335 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4967 (secure 0 0 0) 0x7f9940059190 con 0x7f994c06f000 2024-03-24T01:24:16.495 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:24:16.495 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/1898972903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":1898972903},{"type":"v1","addr":"172.21.15.120:6837","nonce":1898972903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6834/775759501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":775759501},{"type":"v1","addr":"172.21.15.90:6834","nonce":775759501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/43630575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":43630575},{"type":"v1","addr":"172.21.15.90:6835","nonce":43630575}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T01:24:10.173532+0000","modified":"2024-03-24T01:24:15.459921+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6836/963928022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":963928022},{"type":"v1","addr":"172.21.15.120:6836","nonce":963928022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4257,"qdb_cluster":[4257]},"id":1}]} 2024-03-24T01:24:16.495 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 6 2024-03-24T01:24:16.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.495+0000 7f9953a00640 1 -- 172.21.15.90:0/1172058335 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f992c06bbd0 msgr2=0x7f992c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:16.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.495+0000 7f9953a00640 1 --2- 172.21.15.90:0/1172058335 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f992c06bbd0 0x7f992c06e090 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f994c1e78e0 tx=0x7f9938008040 comp rx=0 tx=0).stop 2024-03-24T01:24:16.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.495+0000 7f9953a00640 1 -- 172.21.15.90:0/1172058335 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f994c06f000 msgr2=0x7f994c1e6710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:16.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.495+0000 7f9953a00640 1 --2- 172.21.15.90:0/1172058335 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f994c06f000 0x7f994c1e6710 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f994000d5f0 tx=0x7f994000dab0 comp rx=0 tx=0).stop 2024-03-24T01:24:16.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.496+0000 7f9953a00640 1 -- 172.21.15.90:0/1172058335 shutdown_connections 2024-03-24T01:24:16.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.496+0000 7f9953a00640 1 --2- 172.21.15.90:0/1172058335 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f992c06bbd0 0x7f992c06e090 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:16.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.496+0000 7f9953a00640 1 --2- 172.21.15.90:0/1172058335 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f994c1454e0 0x7f994c1f4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:16.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.496+0000 7f9953a00640 1 --2- 172.21.15.90:0/1172058335 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f994c0cdc50 0x7f994c0cb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:16.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.496+0000 7f9953a00640 1 --2- 172.21.15.90:0/1172058335 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f994c06f000 0x7f994c1e6710 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:16.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.496+0000 7f9953a00640 1 -- 172.21.15.90:0/1172058335 >> 172.21.15.90:0/1172058335 conn(0x7f994c06b2a0 msgr2=0x7f994c0ceb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:16.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.496+0000 7f9953a00640 1 -- 172.21.15.90:0/1172058335 shutdown_connections 2024-03-24T01:24:16.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.496+0000 7f9953a00640 1 -- 172.21.15.90:0/1172058335 wait complete. 2024-03-24T01:24:16.514 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:24:16.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.719+0000 7ff34d000640 1 Processor -- start 2024-03-24T01:24:16.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.719+0000 7ff34d000640 1 -- start start 2024-03-24T01:24:16.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.720+0000 7ff34d000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff3481603e0 0x7ff34806b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:16.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.720+0000 7ff34d000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff348145430 0x7ff34806b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:16.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.720+0000 7ff34d000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3480cd490 con 0x7ff3480cc150 2024-03-24T01:24:16.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.720+0000 7ff34d000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3480cd270 con 0x7ff3481603e0 2024-03-24T01:24:16.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.720+0000 7ff34d000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff34815f900 con 0x7ff348145430 2024-03-24T01:24:16.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.721+0000 7ff33d400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff3481603e0 0x7ff34806b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:16.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.721+0000 7ff346800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff348145430 0x7ff34806b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:16.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.721+0000 7ff346800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff348145430 0x7ff34806b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.90:56694/0 (socket says 172.21.15.90:56694) 2024-03-24T01:24:16.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.721+0000 7ff346800640 1 -- 172.21.15.90:0/3768829323 learned_addr learned my addr 172.21.15.90:0/3768829323 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:16.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.721+0000 7ff33d400640 1 -- 172.21.15.90:0/3768829323 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff348145430 msgr2=0x7ff34806b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:16.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.721+0000 7ff33d400640 1 --2- 172.21.15.90:0/3768829323 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff348145430 0x7ff34806b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:16.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.721+0000 7ff33d400640 1 -- 172.21.15.90:0/3768829323 >> v1:172.21.15.120:6789/0 conn(0x7ff3480cc150 legacy=0x7ff3480ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:16.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.722+0000 7ff33d400640 1 -- 172.21.15.90:0/3768829323 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff34815fbe0 con 0x7ff3481603e0 2024-03-24T01:24:16.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.722+0000 7ff346800640 1 --2- 172.21.15.90:0/3768829323 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff348145430 0x7ff34806b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T01:24:16.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.722+0000 7ff345400640 1 -- 172.21.15.90:0/3768829323 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2424782775 0 0) 0x7ff3480cd490 con 0x7ff3480cc150 2024-03-24T01:24:16.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.723+0000 7ff33d400640 1 --2- 172.21.15.90:0/3768829323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3481603e0 0x7ff34806b0c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff3300097a0 tx=0x7ff33002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3d22f0d48d8ec2b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:16.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.723+0000 7ff345400640 1 -- 172.21.15.90:0/3768829323 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff330035020 con 0x7ff3481603e0 2024-03-24T01:24:16.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.723+0000 7ff345400640 1 -- 172.21.15.90:0/3768829323 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff330031810 con 0x7ff3481603e0 2024-03-24T01:24:16.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.724+0000 7ff345400640 1 -- 172.21.15.90:0/3768829323 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff330031be0 con 0x7ff3481603e0 2024-03-24T01:24:16.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.724+0000 7ff34d000640 1 -- 172.21.15.90:0/3768829323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3481603e0 msgr2=0x7ff34806b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:16.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.724+0000 7ff34d000640 1 --2- 172.21.15.90:0/3768829323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3481603e0 0x7ff34806b0c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff3300097a0 tx=0x7ff33002b8b0 comp rx=0 tx=0).stop 2024-03-24T01:24:16.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.724+0000 7ff34d000640 1 -- 172.21.15.90:0/3768829323 shutdown_connections 2024-03-24T01:24:16.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.724+0000 7ff34d000640 1 --2- 172.21.15.90:0/3768829323 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff348145430 0x7ff34806b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:16.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.724+0000 7ff34d000640 1 --2- 172.21.15.90:0/3768829323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3481603e0 0x7ff34806b0c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:16.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.724+0000 7ff34d000640 1 -- 172.21.15.90:0/3768829323 >> 172.21.15.90:0/3768829323 conn(0x7ff3480b1770 msgr2=0x7ff3480b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:16.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.724+0000 7ff34d000640 1 -- 172.21.15.90:0/3768829323 shutdown_connections 2024-03-24T01:24:16.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.724+0000 7ff34d000640 1 -- 172.21.15.90:0/3768829323 wait complete. 2024-03-24T01:24:16.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.726+0000 7ff34d000640 1 Processor -- start 2024-03-24T01:24:16.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.726+0000 7ff34d000640 1 -- start start 2024-03-24T01:24:16.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.727+0000 7ff34d000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3480cc150 0x7ff3480c73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:16.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.727+0000 7ff34d000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff348145430 0x7ff34815c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:16.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.727+0000 7ff34d000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3481603e0 0x7ff3481f7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:16.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.727+0000 7ff34d000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3480b38d0 con 0x7ff3481603e0 2024-03-24T01:24:16.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.727+0000 7ff34d000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3480b3750 con 0x7ff348145430 2024-03-24T01:24:16.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.727+0000 7ff34d000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3480b3a50 con 0x7ff3480cc150 2024-03-24T01:24:16.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.727+0000 7ff346800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3481603e0 0x7ff3481f7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:16.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.727+0000 7ff346800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3481603e0 0x7ff3481f7570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50684/0 (socket says 172.21.15.90:50684) 2024-03-24T01:24:16.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.727+0000 7ff346800640 1 -- 172.21.15.90:0/3370312640 learned_addr learned my addr 172.21.15.90:0/3370312640 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:16.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.728+0000 7ff33d400640 1 --2- 172.21.15.90:0/3370312640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff348145430 0x7ff34815c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:16.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.728+0000 7ff345e00640 1 --2- 172.21.15.90:0/3370312640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3480cc150 0x7ff3480c73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:16.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.728+0000 7ff346800640 1 -- 172.21.15.90:0/3370312640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3480cc150 msgr2=0x7ff3480c73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:16.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.728+0000 7ff346800640 1 --2- 172.21.15.90:0/3370312640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3480cc150 0x7ff3480c73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:16.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.728+0000 7ff346800640 1 -- 172.21.15.90:0/3370312640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff348145430 msgr2=0x7ff34815c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:16.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.728+0000 7ff346800640 1 --2- 172.21.15.90:0/3370312640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff348145430 0x7ff34815c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:16.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.728+0000 7ff346800640 1 -- 172.21.15.90:0/3370312640 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff34815cf00 con 0x7ff3481603e0 2024-03-24T01:24:16.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.728+0000 7ff33d400640 1 --2- 172.21.15.90:0/3370312640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff348145430 0x7ff34815c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:24:16.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.728+0000 7ff346800640 1 --2- 172.21.15.90:0/3370312640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3481603e0 0x7ff3481f7570 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff34000efc0 tx=0x7ff34000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:16.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.728+0000 7ff33ea00640 1 -- 172.21.15.90:0/3370312640 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34000cbe0 con 0x7ff3481603e0 2024-03-24T01:24:16.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.728+0000 7ff33ea00640 1 -- 172.21.15.90:0/3370312640 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff34000cd80 con 0x7ff3481603e0 2024-03-24T01:24:16.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.728+0000 7ff34d000640 1 -- 172.21.15.90:0/3370312640 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff34815d1e0 con 0x7ff3481603e0 2024-03-24T01:24:16.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.728+0000 7ff33ea00640 1 -- 172.21.15.90:0/3370312640 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff340003bb0 con 0x7ff3481603e0 2024-03-24T01:24:16.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.729+0000 7ff34d000640 1 -- 172.21.15.90:0/3370312640 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff34815d7a0 con 0x7ff3481603e0 2024-03-24T01:24:16.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.730+0000 7ff34d000640 1 -- 172.21.15.90:0/3370312640 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff348068060 con 0x7ff3481603e0 2024-03-24T01:24:16.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.731+0000 7ff33ea00640 1 -- 172.21.15.90:0/3370312640 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff340005d10 con 0x7ff3481603e0 2024-03-24T01:24:16.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.731+0000 7ff33ea00640 1 --2- 172.21.15.90:0/3370312640 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff32406bbd0 0x7ff32406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:16.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.732+0000 7ff33ea00640 1 -- 172.21.15.90:0/3370312640 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ff34008b0e0 con 0x7ff3481603e0 2024-03-24T01:24:16.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.732+0000 7ff345e00640 1 --2- 172.21.15.90:0/3370312640 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff32406bbd0 0x7ff32406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:16.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.733+0000 7ff345e00640 1 --2- 172.21.15.90:0/3370312640 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff32406bbd0 0x7ff32406e090 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff338000f60 tx=0x7ff338009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:16.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.736+0000 7ff33ea00640 1 -- 172.21.15.90:0/3370312640 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff340022020 con 0x7ff3481603e0 2024-03-24T01:24:16.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.932+0000 7ff34d000640 1 -- 172.21.15.90:0/3370312640 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3480b3220 con 0x7ff3481603e0 2024-03-24T01:24:16.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.935+0000 7ff33ea00640 1 -- 172.21.15.90:0/3370312640 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18503 (secure 0 0 0) 0x7ff340010040 con 0x7ff3481603e0 2024-03-24T01:24:16.937 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:24:16.937 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":24,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:24:15.459457+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:24:08.643966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T01:24:09.189904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T01:24:13.027323+0000","flags":32773,"flags_names":"hashpspool,ec_overwrites,creating","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:24:16.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.937+0000 7ff34d000640 1 -- 172.21.15.90:0/3370312640 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff32406bbd0 msgr2=0x7ff32406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:16.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.937+0000 7ff34d000640 1 --2- 172.21.15.90:0/3370312640 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff32406bbd0 0x7ff32406e090 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff338000f60 tx=0x7ff338009040 comp rx=0 tx=0).stop 2024-03-24T01:24:16.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.938+0000 7ff34d000640 1 -- 172.21.15.90:0/3370312640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3481603e0 msgr2=0x7ff3481f7570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:16.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.938+0000 7ff34d000640 1 --2- 172.21.15.90:0/3370312640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3481603e0 0x7ff3481f7570 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff34000efc0 tx=0x7ff34000c490 comp rx=0 tx=0).stop 2024-03-24T01:24:16.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.938+0000 7ff34d000640 1 -- 172.21.15.90:0/3370312640 shutdown_connections 2024-03-24T01:24:16.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.938+0000 7ff34d000640 1 --2- 172.21.15.90:0/3370312640 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff32406bbd0 0x7ff32406e090 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:16.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.938+0000 7ff34d000640 1 --2- 172.21.15.90:0/3370312640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3481603e0 0x7ff3481f7570 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:16.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.938+0000 7ff34d000640 1 --2- 172.21.15.90:0/3370312640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff348145430 0x7ff34815c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:16.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.938+0000 7ff34d000640 1 --2- 172.21.15.90:0/3370312640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3480cc150 0x7ff3480c73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:16.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.938+0000 7ff34d000640 1 -- 172.21.15.90:0/3370312640 >> 172.21.15.90:0/3370312640 conn(0x7ff3480b1770 msgr2=0x7ff3480ceb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:16.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.938+0000 7ff34d000640 1 -- 172.21.15.90:0/3370312640 shutdown_connections 2024-03-24T01:24:16.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:16.938+0000 7ff34d000640 1 -- 172.21.15.90:0/3370312640 wait complete. 2024-03-24T01:24:16.959 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:24:17.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.168+0000 7f7568800640 1 Processor -- start 2024-03-24T01:24:17.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.168+0000 7f7568800640 1 -- start start 2024-03-24T01:24:17.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.169+0000 7f7568800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f75601603e0 0x7f756006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:17.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.169+0000 7f7568800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7560145430 0x7f756006b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:17.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.169+0000 7f7568800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75600cd490 con 0x7f75600cc150 2024-03-24T01:24:17.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.169+0000 7f7568800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75600cd270 con 0x7f75601603e0 2024-03-24T01:24:17.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.169+0000 7f7568800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f756015f900 con 0x7f7560145430 2024-03-24T01:24:17.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.170+0000 7f7565400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f75601603e0 0x7f756006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:17.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.170+0000 7f7565400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f75601603e0 0x7f756006b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50688/0 (socket says 172.21.15.90:50688) 2024-03-24T01:24:17.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.170+0000 7f7565400640 1 -- 172.21.15.90:0/1118017789 learned_addr learned my addr 172.21.15.90:0/1118017789 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:17.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.170+0000 7f7566800640 1 --2- 172.21.15.90:0/1118017789 >> v2:172.21.15.120:3300/0 conn(0x7f7560145430 0x7f756006b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:17.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.170+0000 7f7565400640 1 -- 172.21.15.90:0/1118017789 >> v2:172.21.15.120:3300/0 conn(0x7f7560145430 msgr2=0x7f756006b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:17.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.170+0000 7f7565400640 1 --2- 172.21.15.90:0/1118017789 >> v2:172.21.15.120:3300/0 conn(0x7f7560145430 0x7f756006b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:17.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.170+0000 7f7565400640 1 -- 172.21.15.90:0/1118017789 >> v1:172.21.15.120:6789/0 conn(0x7f75600cc150 legacy=0x7f75600ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:17.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.170+0000 7f7565400640 1 -- 172.21.15.90:0/1118017789 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f756015fbe0 con 0x7f75601603e0 2024-03-24T01:24:17.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.171+0000 7f7565400640 1 --2- 172.21.15.90:0/1118017789 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75601603e0 0x7f756006b0c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f756006fc10 tx=0x7f755402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0a80fe990bbd864 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:17.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.172+0000 7f7564a00640 1 -- 172.21.15.90:0/1118017789 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7554035020 con 0x7f75601603e0 2024-03-24T01:24:17.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.172+0000 7f7564a00640 1 -- 172.21.15.90:0/1118017789 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7554031870 con 0x7f75601603e0 2024-03-24T01:24:17.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.172+0000 7f7564a00640 1 -- 172.21.15.90:0/1118017789 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7554031c40 con 0x7f75601603e0 2024-03-24T01:24:17.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.172+0000 7f7568800640 1 -- 172.21.15.90:0/1118017789 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75601603e0 msgr2=0x7f756006b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:17.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.172+0000 7f7568800640 1 --2- 172.21.15.90:0/1118017789 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75601603e0 0x7f756006b0c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f756006fc10 tx=0x7f755402b910 comp rx=0 tx=0).stop 2024-03-24T01:24:17.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.172+0000 7f7568800640 1 -- 172.21.15.90:0/1118017789 shutdown_connections 2024-03-24T01:24:17.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.172+0000 7f7568800640 1 --2- 172.21.15.90:0/1118017789 >> v2:172.21.15.120:3300/0 conn(0x7f7560145430 0x7f756006b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:17.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.172+0000 7f7568800640 1 --2- 172.21.15.90:0/1118017789 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75601603e0 0x7f756006b0c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:17.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.172+0000 7f7568800640 1 -- 172.21.15.90:0/1118017789 >> 172.21.15.90:0/1118017789 conn(0x7f75600b1770 msgr2=0x7f75600b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:17.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.172+0000 7f7568800640 1 -- 172.21.15.90:0/1118017789 shutdown_connections 2024-03-24T01:24:17.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.173+0000 7f7568800640 1 -- 172.21.15.90:0/1118017789 wait complete. 2024-03-24T01:24:17.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.174+0000 7f7568800640 1 Processor -- start 2024-03-24T01:24:17.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.174+0000 7f7568800640 1 -- start start 2024-03-24T01:24:17.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.175+0000 7f7568800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75600cc150 0x7f756015eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:17.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.175+0000 7f7568800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7560145430 0x7f756013e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:17.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.175+0000 7f7568800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75601603e0 0x7f756013ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:17.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.175+0000 7f7568800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75600b38d0 con 0x7f75600cc150 2024-03-24T01:24:17.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.175+0000 7f7568800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75600b3750 con 0x7f7560145430 2024-03-24T01:24:17.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.175+0000 7f7568800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75600b3a50 con 0x7f75601603e0 2024-03-24T01:24:17.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.175+0000 7f7565e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75600cc150 0x7f756015eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:17.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.176+0000 7f7565e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75600cc150 0x7f756015eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50700/0 (socket says 172.21.15.90:50700) 2024-03-24T01:24:17.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.176+0000 7f7565e00640 1 -- 172.21.15.90:0/1978945605 learned_addr learned my addr 172.21.15.90:0/1978945605 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:17.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.176+0000 7f7565400640 1 --2- 172.21.15.90:0/1978945605 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7560145430 0x7f756013e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:17.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.176+0000 7f7566800640 1 --2- 172.21.15.90:0/1978945605 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75601603e0 0x7f756013ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:17.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.176+0000 7f7565e00640 1 -- 172.21.15.90:0/1978945605 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75601603e0 msgr2=0x7f756013ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:17.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.176+0000 7f7565e00640 1 --2- 172.21.15.90:0/1978945605 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75601603e0 0x7f756013ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:17.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.176+0000 7f7565e00640 1 -- 172.21.15.90:0/1978945605 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7560145430 msgr2=0x7f756013e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:17.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.176+0000 7f7565e00640 1 --2- 172.21.15.90:0/1978945605 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7560145430 0x7f756013e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:17.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.176+0000 7f7565e00640 1 -- 172.21.15.90:0/1978945605 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75601e5b70 con 0x7f75600cc150 2024-03-24T01:24:17.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.176+0000 7f7565e00640 1 --2- 172.21.15.90:0/1978945605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75600cc150 0x7f756015eb30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f754c00c430 tx=0x7f754c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:17.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.176+0000 7f755a000640 1 -- 172.21.15.90:0/1978945605 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754c011790 con 0x7f75600cc150 2024-03-24T01:24:17.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.176+0000 7f755a000640 1 -- 172.21.15.90:0/1978945605 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f754c017070 con 0x7f75600cc150 2024-03-24T01:24:17.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.176+0000 7f755a000640 1 -- 172.21.15.90:0/1978945605 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754c011bb0 con 0x7f75600cc150 2024-03-24T01:24:17.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.176+0000 7f7568800640 1 -- 172.21.15.90:0/1978945605 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75601e5e50 con 0x7f75600cc150 2024-03-24T01:24:17.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.177+0000 7f7568800640 1 -- 172.21.15.90:0/1978945605 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75601e6410 con 0x7f75600cc150 2024-03-24T01:24:17.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.178+0000 7f7568800640 1 -- 172.21.15.90:0/1978945605 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7560144dd0 con 0x7f75600cc150 2024-03-24T01:24:17.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.178+0000 7f755a000640 1 -- 172.21.15.90:0/1978945605 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f754c01c460 con 0x7f75600cc150 2024-03-24T01:24:17.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.180+0000 7f755a000640 1 --2- 172.21.15.90:0/1978945605 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f754006bc00 0x7f754006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:17.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.180+0000 7f755a000640 1 -- 172.21.15.90:0/1978945605 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f754c08ac80 con 0x7f75600cc150 2024-03-24T01:24:17.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.180+0000 7f7565400640 1 --2- 172.21.15.90:0/1978945605 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f754006bc00 0x7f754006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:17.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.181+0000 7f7565400640 1 --2- 172.21.15.90:0/1978945605 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f754006bc00 0x7f754006e0c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7554036000 tx=0x7f755403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:17.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.184+0000 7f755a000640 1 -- 172.21.15.90:0/1978945605 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f754c059430 con 0x7f75600cc150 2024-03-24T01:24:17.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.381+0000 7f7568800640 1 -- 172.21.15.90:0/1978945605 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7560068060 con 0x7f75600cc150 2024-03-24T01:24:17.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.385+0000 7f755a000640 1 -- 172.21.15.90:0/1978945605 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18490 (secure 0 0 0) 0x7f754c058d30 con 0x7f75600cc150 2024-03-24T01:24:17.386 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:24:17.386 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:24:17.008224+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:24:08.643966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T01:24:09.189904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T01:24:13.027323+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:24:17.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.387+0000 7f7568800640 1 -- 172.21.15.90:0/1978945605 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f754006bc00 msgr2=0x7f754006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:17.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.387+0000 7f7568800640 1 --2- 172.21.15.90:0/1978945605 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f754006bc00 0x7f754006e0c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7554036000 tx=0x7f755403c040 comp rx=0 tx=0).stop 2024-03-24T01:24:17.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.387+0000 7f7568800640 1 -- 172.21.15.90:0/1978945605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75600cc150 msgr2=0x7f756015eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:17.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.387+0000 7f7568800640 1 --2- 172.21.15.90:0/1978945605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75600cc150 0x7f756015eb30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f754c00c430 tx=0x7f754c00c8f0 comp rx=0 tx=0).stop 2024-03-24T01:24:17.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.387+0000 7f7568800640 1 -- 172.21.15.90:0/1978945605 shutdown_connections 2024-03-24T01:24:17.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.387+0000 7f7568800640 1 --2- 172.21.15.90:0/1978945605 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f754006bc00 0x7f754006e0c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:17.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.387+0000 7f7568800640 1 --2- 172.21.15.90:0/1978945605 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f75601603e0 0x7f756013ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:17.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.387+0000 7f7568800640 1 --2- 172.21.15.90:0/1978945605 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7560145430 0x7f756013e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:17.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.387+0000 7f7568800640 1 --2- 172.21.15.90:0/1978945605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75600cc150 0x7f756015eb30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:17.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.387+0000 7f7568800640 1 -- 172.21.15.90:0/1978945605 >> 172.21.15.90:0/1978945605 conn(0x7f75600b1770 msgr2=0x7f75600ccb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:17.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.388+0000 7f7568800640 1 -- 172.21.15.90:0/1978945605 shutdown_connections 2024-03-24T01:24:17.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.388+0000 7f7568800640 1 -- 172.21.15.90:0/1978945605 wait complete. 2024-03-24T01:24:17.422 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:24:17.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.627+0000 7f5053600640 1 Processor -- start 2024-03-24T01:24:17.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.627+0000 7f5053600640 1 -- start start 2024-03-24T01:24:17.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.629+0000 7f5053600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f504c0cd6e0 0x7f504c0cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:17.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.629+0000 7f5053600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f504c0ce190 0x7f504c15c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:17.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.629+0000 7f5053600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f504c0cc660 con 0x7f504c1454e0 2024-03-24T01:24:17.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.629+0000 7f5053600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f504c0cc430 con 0x7f504c0ce190 2024-03-24T01:24:17.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.629+0000 7f5053600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f504c0cc920 con 0x7f504c0cd6e0 2024-03-24T01:24:17.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.629+0000 7f5043e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f504c0cd6e0 0x7f504c0cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:17.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.629+0000 7f5043e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f504c0cd6e0 0x7f504c0cdc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.90:56716/0 (socket says 172.21.15.90:56716) 2024-03-24T01:24:17.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.629+0000 7f5043e00640 1 -- 172.21.15.90:0/3748019413 learned_addr learned my addr 172.21.15.90:0/3748019413 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:17.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.629+0000 7f5051600640 1 --2- 172.21.15.90:0/3748019413 >> v2:172.21.15.120:3300/0 conn(0x7f504c0ce190 0x7f504c15c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:17.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.630+0000 7f5043e00640 1 -- 172.21.15.90:0/3748019413 >> v2:172.21.15.120:3300/0 conn(0x7f504c0ce190 msgr2=0x7f504c15c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:17.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.630+0000 7f5043e00640 1 --2- 172.21.15.90:0/3748019413 >> v2:172.21.15.120:3300/0 conn(0x7f504c0ce190 0x7f504c15c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:17.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.630+0000 7f5043e00640 1 -- 172.21.15.90:0/3748019413 >> v1:172.21.15.120:6789/0 conn(0x7f504c1454e0 legacy=0x7f504c06f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:17.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.630+0000 7f5043e00640 1 -- 172.21.15.90:0/3748019413 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f504c0ccc00 con 0x7f504c0cd6e0 2024-03-24T01:24:17.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.631+0000 7f5043e00640 1 --2- 172.21.15.90:0/3748019413 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f504c0cd6e0 0x7f504c0cdc50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5034009870 tx=0x7f503402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe6a9b921870c70 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:17.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.632+0000 7f504be00640 1 -- 172.21.15.90:0/3748019413 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5034034070 con 0x7f504c0cd6e0 2024-03-24T01:24:17.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.632+0000 7f504be00640 1 -- 172.21.15.90:0/3748019413 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5034030840 con 0x7f504c0cd6e0 2024-03-24T01:24:17.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.632+0000 7f504be00640 1 -- 172.21.15.90:0/3748019413 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5034030c10 con 0x7f504c0cd6e0 2024-03-24T01:24:17.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.632+0000 7f5053600640 1 -- 172.21.15.90:0/3748019413 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f504c0cd6e0 msgr2=0x7f504c0cdc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:17.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.632+0000 7f5053600640 1 --2- 172.21.15.90:0/3748019413 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f504c0cd6e0 0x7f504c0cdc50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5034009870 tx=0x7f503402b980 comp rx=0 tx=0).stop 2024-03-24T01:24:17.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.633+0000 7f5053600640 1 -- 172.21.15.90:0/3748019413 shutdown_connections 2024-03-24T01:24:17.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.633+0000 7f5053600640 1 --2- 172.21.15.90:0/3748019413 >> v2:172.21.15.120:3300/0 conn(0x7f504c0ce190 0x7f504c15c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:17.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.633+0000 7f5053600640 1 --2- 172.21.15.90:0/3748019413 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f504c0cd6e0 0x7f504c0cdc50 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:17.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.633+0000 7f5053600640 1 -- 172.21.15.90:0/3748019413 >> 172.21.15.90:0/3748019413 conn(0x7f504c06b2a0 msgr2=0x7f504c06b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:17.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.633+0000 7f5053600640 1 -- 172.21.15.90:0/3748019413 shutdown_connections 2024-03-24T01:24:17.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.633+0000 7f5053600640 1 -- 172.21.15.90:0/3748019413 wait complete. 2024-03-24T01:24:17.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.635+0000 7f5053600640 1 Processor -- start 2024-03-24T01:24:17.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.635+0000 7f5053600640 1 -- start start 2024-03-24T01:24:17.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.636+0000 7f5053600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f504c0cd6e0 0x7f504c13b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:17.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.636+0000 7f5053600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f504c0ce190 0x7f504c0acd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:17.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.636+0000 7f5053600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f504c1454e0 0x7f504c15c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:17.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.636+0000 7f5053600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f504c06f710 con 0x7f504c1454e0 2024-03-24T01:24:17.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.636+0000 7f5053600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f504c0cd1c0 con 0x7f504c0ce190 2024-03-24T01:24:17.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.636+0000 7f5053600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f504c06f890 con 0x7f504c0cd6e0 2024-03-24T01:24:17.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.636+0000 7f5051600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f504c1454e0 0x7f504c15c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:17.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.636+0000 7f5051600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f504c1454e0 0x7f504c15c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50702/0 (socket says 172.21.15.90:50702) 2024-03-24T01:24:17.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.636+0000 7f5051600640 1 -- 172.21.15.90:0/4002624022 learned_addr learned my addr 172.21.15.90:0/4002624022 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:17.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.637+0000 7f5050c00640 1 --2- 172.21.15.90:0/4002624022 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f504c0cd6e0 0x7f504c13b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:17.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.637+0000 7f5043e00640 1 --2- 172.21.15.90:0/4002624022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f504c0ce190 0x7f504c0acd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:17.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.637+0000 7f5051600640 1 -- 172.21.15.90:0/4002624022 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f504c0cd6e0 msgr2=0x7f504c13b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:17.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.637+0000 7f5051600640 1 --2- 172.21.15.90:0/4002624022 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f504c0cd6e0 0x7f504c13b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:17.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.637+0000 7f5051600640 1 -- 172.21.15.90:0/4002624022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f504c0ce190 msgr2=0x7f504c0acd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:17.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.637+0000 7f5051600640 1 --2- 172.21.15.90:0/4002624022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f504c0ce190 0x7f504c0acd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:17.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.637+0000 7f5051600640 1 -- 172.21.15.90:0/4002624022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f504c0ad800 con 0x7f504c1454e0 2024-03-24T01:24:17.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.637+0000 7f5051600640 1 --2- 172.21.15.90:0/4002624022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f504c1454e0 0x7f504c15c960 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f504400efc0 tx=0x7f504400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:17.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.637+0000 7f5049600640 1 -- 172.21.15.90:0/4002624022 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5044018020 con 0x7f504c1454e0 2024-03-24T01:24:17.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.637+0000 7f5049600640 1 -- 172.21.15.90:0/4002624022 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5044014480 con 0x7f504c1454e0 2024-03-24T01:24:17.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.637+0000 7f5049600640 1 -- 172.21.15.90:0/4002624022 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5044014870 con 0x7f504c1454e0 2024-03-24T01:24:17.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.637+0000 7f5053600640 1 -- 172.21.15.90:0/4002624022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f504c0adae0 con 0x7f504c1454e0 2024-03-24T01:24:17.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.638+0000 7f5053600640 1 -- 172.21.15.90:0/4002624022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f504c0ae0a0 con 0x7f504c1454e0 2024-03-24T01:24:17.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.639+0000 7f5053600640 1 -- 172.21.15.90:0/4002624022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f504c1438e0 con 0x7f504c1454e0 2024-03-24T01:24:17.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.640+0000 7f5049600640 1 -- 172.21.15.90:0/4002624022 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5044005d10 con 0x7f504c1454e0 2024-03-24T01:24:17.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.640+0000 7f5049600640 1 --2- 172.21.15.90:0/4002624022 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f502806bbd0 0x7f502806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:17.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.641+0000 7f5049600640 1 -- 172.21.15.90:0/4002624022 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f504408bca0 con 0x7f504c1454e0 2024-03-24T01:24:17.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.641+0000 7f5050c00640 1 --2- 172.21.15.90:0/4002624022 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f502806bbd0 0x7f502806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:17.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.642+0000 7f5050c00640 1 --2- 172.21.15.90:0/4002624022 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f502806bbd0 0x7f502806e090 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f503c007fd0 tx=0x7f503c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:17.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.645+0000 7f5049600640 1 -- 172.21.15.90:0/4002624022 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f504c1438e0 con 0x7f504c1454e0 2024-03-24T01:24:17.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.892+0000 7f5053600640 1 -- 172.21.15.90:0/4002624022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f504c073570 con 0x7f504c1454e0 2024-03-24T01:24:17.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.892+0000 7f5049600640 1 -- 172.21.15.90:0/4002624022 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4967 (secure 0 0 0) 0x7f5044022030 con 0x7f504c1454e0 2024-03-24T01:24:17.894 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:24:17.894 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/1898972903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":1898972903},{"type":"v1","addr":"172.21.15.120:6837","nonce":1898972903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6834/775759501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":775759501},{"type":"v1","addr":"172.21.15.90:6834","nonce":775759501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/43630575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":43630575},{"type":"v1","addr":"172.21.15.90:6835","nonce":43630575}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T01:24:10.173532+0000","modified":"2024-03-24T01:24:15.459921+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6836/963928022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":963928022},{"type":"v1","addr":"172.21.15.120:6836","nonce":963928022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4257,"qdb_cluster":[4257]},"id":1}]} 2024-03-24T01:24:17.894 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 6 2024-03-24T01:24:17.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.895+0000 7f5053600640 1 -- 172.21.15.90:0/4002624022 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f502806bbd0 msgr2=0x7f502806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:17.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.895+0000 7f5053600640 1 --2- 172.21.15.90:0/4002624022 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f502806bbd0 0x7f502806e090 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f503c007fd0 tx=0x7f503c009040 comp rx=0 tx=0).stop 2024-03-24T01:24:17.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.895+0000 7f5053600640 1 -- 172.21.15.90:0/4002624022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f504c1454e0 msgr2=0x7f504c15c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:17.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.895+0000 7f5053600640 1 --2- 172.21.15.90:0/4002624022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f504c1454e0 0x7f504c15c960 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f504400efc0 tx=0x7f504400c490 comp rx=0 tx=0).stop 2024-03-24T01:24:17.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.895+0000 7f5053600640 1 -- 172.21.15.90:0/4002624022 shutdown_connections 2024-03-24T01:24:17.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.895+0000 7f5053600640 1 --2- 172.21.15.90:0/4002624022 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f502806bbd0 0x7f502806e090 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:17.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.895+0000 7f5053600640 1 --2- 172.21.15.90:0/4002624022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f504c1454e0 0x7f504c15c960 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:17.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.895+0000 7f5053600640 1 --2- 172.21.15.90:0/4002624022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f504c0ce190 0x7f504c0acd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:17.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.895+0000 7f5053600640 1 --2- 172.21.15.90:0/4002624022 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f504c0cd6e0 0x7f504c13b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:17.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.895+0000 7f5053600640 1 -- 172.21.15.90:0/4002624022 >> 172.21.15.90:0/4002624022 conn(0x7f504c06b2a0 msgr2=0x7f504c0ce620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:17.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.896+0000 7f5053600640 1 -- 172.21.15.90:0/4002624022 shutdown_connections 2024-03-24T01:24:17.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:17.896+0000 7f5053600640 1 -- 172.21.15.90:0/4002624022 wait complete. 2024-03-24T01:24:17.915 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:24:18.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.122+0000 7fd968c00640 1 Processor -- start 2024-03-24T01:24:18.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.122+0000 7fd968c00640 1 -- start start 2024-03-24T01:24:18.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.123+0000 7fd968c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd9640cdac0 0x7fd96406b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:18.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.124+0000 7fd968c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd96415bf70 con 0x7fd964155030 2024-03-24T01:24:18.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.124+0000 7fd968c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd96406ba00 con 0x7fd96406b600 2024-03-24T01:24:18.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.124+0000 7fd968c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd96415b360 con 0x7fd9640cdac0 2024-03-24T01:24:18.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.124+0000 7fd961e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd964155030 0x7fd96415efc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:37500/0 (socket says 172.21.15.90:37500) 2024-03-24T01:24:18.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.124+0000 7fd961e00640 1 -- 172.21.15.90:0/3199213307 learned_addr learned my addr 172.21.15.90:0/3199213307 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:18.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.124+0000 7fd961400640 1 -- 172.21.15.90:0/3199213307 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 211129504 0 0) 0x7fd96415bf70 con 0x7fd964155030 2024-03-24T01:24:18.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.124+0000 7fd961400640 1 -- 172.21.15.90:0/3199213307 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd948005730 con 0x7fd964155030 2024-03-24T01:24:18.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.125+0000 7fd95be00640 1 --2- 172.21.15.90:0/3199213307 >> v2:172.21.15.120:3300/0 conn(0x7fd9640cdac0 0x7fd96406b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:18.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.125+0000 7fd961400640 1 -- 172.21.15.90:0/3199213307 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3023783011 0 0) 0x7fd948005730 con 0x7fd964155030 2024-03-24T01:24:18.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.125+0000 7fd961400640 1 -- 172.21.15.90:0/3199213307 >> v2:172.21.15.120:3300/0 conn(0x7fd9640cdac0 msgr2=0x7fd96406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:18.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.125+0000 7fd961400640 1 --2- 172.21.15.90:0/3199213307 >> v2:172.21.15.120:3300/0 conn(0x7fd9640cdac0 0x7fd96406b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:18.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.125+0000 7fd961400640 1 -- 172.21.15.90:0/3199213307 >> v1:172.21.15.120:6789/0 conn(0x7fd96406b600 legacy=0x7fd9640ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:18.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.125+0000 7fd961400640 1 -- 172.21.15.90:0/3199213307 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd96415b640 con 0x7fd964155030 2024-03-24T01:24:18.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.125+0000 7fd961400640 1 -- 172.21.15.90:0/3199213307 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7fd954001670 con 0x7fd964155030 2024-03-24T01:24:18.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.125+0000 7fd961400640 1 -- 172.21.15.90:0/3199213307 >> v1:172.21.15.90:6789/0 conn(0x7fd964155030 legacy=0x7fd96415efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:18.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.125+0000 7fd961400640 1 --2- 172.21.15.90:0/3199213307 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd948005c10 0x7fd948006030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:18.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.125+0000 7fd961400640 1 --2- 172.21.15.90:0/3199213307 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd948006a10 0x7fd948006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:18.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.125+0000 7fd961400640 1 --2- 172.21.15.90:0/3199213307 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd948007420 0x7fd94800dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:18.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.125+0000 7fd961400640 1 -- 172.21.15.90:0/3199213307 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96415b360 con 0x7fd948007420 2024-03-24T01:24:18.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.125+0000 7fd961400640 1 -- 172.21.15.90:0/3199213307 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd94800e370 con 0x7fd948005c10 2024-03-24T01:24:18.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.125+0000 7fd961400640 1 -- 172.21.15.90:0/3199213307 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd94800e570 con 0x7fd948006a10 2024-03-24T01:24:18.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.125+0000 7fd962800640 1 --2- 172.21.15.90:0/3199213307 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd948007420 0x7fd94800dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:18.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.125+0000 7fd95be00640 1 --2- 172.21.15.90:0/3199213307 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd948006a10 0x7fd948006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:18.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.126+0000 7fd961e00640 1 --2- 172.21.15.90:0/3199213307 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd948005c10 0x7fd948006030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:18.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.126+0000 7fd962800640 1 -- 172.21.15.90:0/3199213307 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd948006a10 msgr2=0x7fd948006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:18.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.126+0000 7fd962800640 1 --2- 172.21.15.90:0/3199213307 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd948006a10 0x7fd948006e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:18.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.126+0000 7fd962800640 1 -- 172.21.15.90:0/3199213307 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd948005c10 msgr2=0x7fd948006030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:18.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.126+0000 7fd962800640 1 --2- 172.21.15.90:0/3199213307 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd948005c10 0x7fd948006030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:18.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.126+0000 7fd962800640 1 -- 172.21.15.90:0/3199213307 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd948005730 con 0x7fd948007420 2024-03-24T01:24:18.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.126+0000 7fd95be00640 1 --2- 172.21.15.90:0/3199213307 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd948006a10 0x7fd948006e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:24:18.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.127+0000 7fd962800640 1 -- 172.21.15.90:0/3199213307 reap_dead start 2024-03-24T01:24:18.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.127+0000 7fd962800640 1 --2- 172.21.15.90:0/3199213307 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd948007420 0x7fd94800dbd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd95c001e50 tx=0x7fd948007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7346e54499feb0a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:18.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.127+0000 7fd961400640 1 -- 172.21.15.90:0/3199213307 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95c001a70 con 0x7fd948007420 2024-03-24T01:24:18.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.127+0000 7fd961400640 1 -- 172.21.15.90:0/3199213307 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd95c003bb0 con 0x7fd948007420 2024-03-24T01:24:18.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.127+0000 7fd961400640 1 -- 172.21.15.90:0/3199213307 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95c0050d0 con 0x7fd948007420 2024-03-24T01:24:18.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.128+0000 7fd968c00640 1 -- 172.21.15.90:0/3199213307 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd948007420 msgr2=0x7fd94800dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:18.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.128+0000 7fd968c00640 1 --2- 172.21.15.90:0/3199213307 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd948007420 0x7fd94800dbd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd95c001e50 tx=0x7fd948007a40 comp rx=0 tx=0).stop 2024-03-24T01:24:18.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.128+0000 7fd968c00640 1 -- 172.21.15.90:0/3199213307 shutdown_connections 2024-03-24T01:24:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.128+0000 7fd968c00640 1 --2- 172.21.15.90:0/3199213307 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd948007420 0x7fd94800dbd0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.128+0000 7fd968c00640 1 -- 172.21.15.90:0/3199213307 >> 172.21.15.90:0/3199213307 conn(0x7fd9640b1770 msgr2=0x7fd9640b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.128+0000 7fd968c00640 1 -- 172.21.15.90:0/3199213307 shutdown_connections 2024-03-24T01:24:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.128+0000 7fd968c00640 1 -- 172.21.15.90:0/3199213307 wait complete. 2024-03-24T01:24:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.130+0000 7fd968c00640 1 Processor -- start 2024-03-24T01:24:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.130+0000 7fd968c00640 1 -- start start 2024-03-24T01:24:18.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.131+0000 7fd968c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd948007420 0x7fd964144250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:18.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.131+0000 7fd968c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd9641e3250 0x7fd9641e36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:18.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.131+0000 7fd968c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd964156850 0x7fd964156d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:18.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.131+0000 7fd968c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd964155410 con 0x7fd9641e3250 2024-03-24T01:24:18.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.131+0000 7fd968c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96415bc00 con 0x7fd948007420 2024-03-24T01:24:18.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.131+0000 7fd968c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd96415af70 con 0x7fd964156850 2024-03-24T01:24:18.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.131+0000 7fd95be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd9641e3250 0x7fd9641e36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:18.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.131+0000 7fd95be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd9641e3250 0x7fd9641e36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50720/0 (socket says 172.21.15.90:50720) 2024-03-24T01:24:18.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.131+0000 7fd95be00640 1 -- 172.21.15.90:0/2780377595 learned_addr learned my addr 172.21.15.90:0/2780377595 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:18.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.131+0000 7fd961e00640 1 --2- 172.21.15.90:0/2780377595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd948007420 0x7fd964144250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:18.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.131+0000 7fd962800640 1 --2- 172.21.15.90:0/2780377595 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd964156850 0x7fd964156d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:18.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.131+0000 7fd95be00640 1 -- 172.21.15.90:0/2780377595 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd964156850 msgr2=0x7fd964156d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:18.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.131+0000 7fd95be00640 1 --2- 172.21.15.90:0/2780377595 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd964156850 0x7fd964156d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:18.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.131+0000 7fd95be00640 1 -- 172.21.15.90:0/2780377595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd948007420 msgr2=0x7fd964144250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:18.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.131+0000 7fd95be00640 1 --2- 172.21.15.90:0/2780377595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd948007420 0x7fd964144250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:18.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.131+0000 7fd95be00640 1 -- 172.21.15.90:0/2780377595 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9641575b0 con 0x7fd9641e3250 2024-03-24T01:24:18.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.131+0000 7fd961e00640 1 --2- 172.21.15.90:0/2780377595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd948007420 0x7fd964144250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:18.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.132+0000 7fd95be00640 1 --2- 172.21.15.90:0/2780377595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd9641e3250 0x7fd9641e36b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd94c00f4e0 tx=0x7fd94c00f9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:18.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.132+0000 7fd95a000640 1 -- 172.21.15.90:0/2780377595 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd94c004230 con 0x7fd9641e3250 2024-03-24T01:24:18.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.132+0000 7fd95a000640 1 -- 172.21.15.90:0/2780377595 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd94c0043d0 con 0x7fd9641e3250 2024-03-24T01:24:18.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.132+0000 7fd968c00640 1 -- 172.21.15.90:0/2780377595 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd964157890 con 0x7fd9641e3250 2024-03-24T01:24:18.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.132+0000 7fd95a000640 1 -- 172.21.15.90:0/2780377595 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd94c0148b0 con 0x7fd9641e3250 2024-03-24T01:24:18.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.132+0000 7fd968c00640 1 -- 172.21.15.90:0/2780377595 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd964157e50 con 0x7fd9641e3250 2024-03-24T01:24:18.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.133+0000 7fd968c00640 1 -- 172.21.15.90:0/2780377595 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd964068060 con 0x7fd9641e3250 2024-03-24T01:24:18.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.134+0000 7fd95a000640 1 -- 172.21.15.90:0/2780377595 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd94c012070 con 0x7fd9641e3250 2024-03-24T01:24:18.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.135+0000 7fd95a000640 1 --2- 172.21.15.90:0/2780377595 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd94006bbd0 0x7fd94006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:18.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.136+0000 7fd95a000640 1 -- 172.21.15.90:0/2780377595 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd94c08b440 con 0x7fd9641e3250 2024-03-24T01:24:18.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.136+0000 7fd961e00640 1 --2- 172.21.15.90:0/2780377595 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd94006bbd0 0x7fd94006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:18.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.137+0000 7fd961e00640 1 --2- 172.21.15.90:0/2780377595 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd94006bbd0 0x7fd94006e090 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd954001eb0 tx=0x7fd954000f70 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:18.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.140+0000 7fd95a000640 1 -- 172.21.15.90:0/2780377595 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd94c012360 con 0x7fd9641e3250 2024-03-24T01:24:18.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.339+0000 7fd968c00640 1 -- 172.21.15.90:0/2780377595 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9640b38c0 con 0x7fd9641e3250 2024-03-24T01:24:18.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.342+0000 7fd95a000640 1 -- 172.21.15.90:0/2780377595 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18490 (secure 0 0 0) 0x7fd94c059c80 con 0x7fd9641e3250 2024-03-24T01:24:18.343 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:24:18.343 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:24:17.008224+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:24:08.643966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T01:24:09.189904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T01:24:13.027323+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:24:18.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.344+0000 7fd968c00640 1 -- 172.21.15.90:0/2780377595 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd94006bbd0 msgr2=0x7fd94006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:18.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.344+0000 7fd968c00640 1 --2- 172.21.15.90:0/2780377595 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd94006bbd0 0x7fd94006e090 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd954001eb0 tx=0x7fd954000f70 comp rx=0 tx=0).stop 2024-03-24T01:24:18.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.344+0000 7fd968c00640 1 -- 172.21.15.90:0/2780377595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd9641e3250 msgr2=0x7fd9641e36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:18.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.344+0000 7fd968c00640 1 --2- 172.21.15.90:0/2780377595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd9641e3250 0x7fd9641e36b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd94c00f4e0 tx=0x7fd94c00f9a0 comp rx=0 tx=0).stop 2024-03-24T01:24:18.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.345+0000 7fd968c00640 1 -- 172.21.15.90:0/2780377595 shutdown_connections 2024-03-24T01:24:18.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.345+0000 7fd968c00640 1 --2- 172.21.15.90:0/2780377595 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd94006bbd0 0x7fd94006e090 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:18.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.345+0000 7fd968c00640 1 --2- 172.21.15.90:0/2780377595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd9641e3250 0x7fd9641e36b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:18.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.345+0000 7fd968c00640 1 --2- 172.21.15.90:0/2780377595 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd964156850 0x7fd964156d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:18.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.345+0000 7fd968c00640 1 --2- 172.21.15.90:0/2780377595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd948007420 0x7fd964144250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:18.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.345+0000 7fd968c00640 1 -- 172.21.15.90:0/2780377595 >> 172.21.15.90:0/2780377595 conn(0x7fd9640b1770 msgr2=0x7fd9640b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:18.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.345+0000 7fd968c00640 1 -- 172.21.15.90:0/2780377595 shutdown_connections 2024-03-24T01:24:18.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.345+0000 7fd968c00640 1 -- 172.21.15.90:0/2780377595 wait complete. 2024-03-24T01:24:18.766 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T01:24:10.173532+0000', 'modified': '2024-03-24T01:24:15.459921+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4257}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4257': {'gid': 4257, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.120:6836/963928022', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6834', 'nonce': 963928022}, {'type': 'v1', 'addr': '172.21.15.120:6836', 'nonce': 963928022}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 4257, 'qdb_cluster': [4257]} 2024-03-24T01:24:18.766 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T01:24:18.766 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-24T01:24:18.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.980+0000 7f463f000640 1 Processor -- start 2024-03-24T01:24:18.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.980+0000 7f463f000640 1 -- start start 2024-03-24T01:24:18.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.982+0000 7f463f000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f46380b4410 0x7f46380b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:18.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.982+0000 7f463f000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46380adfa0 0x7f46380ae430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:18.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.982+0000 7f463f000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f463806d7f0 con 0x7f4638071790 2024-03-24T01:24:18.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.982+0000 7f463f000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f463806dab0 con 0x7f46380b4410 2024-03-24T01:24:18.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.982+0000 7f463f000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f463806dd90 con 0x7f46380adfa0 2024-03-24T01:24:18.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.983+0000 7f463e600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46380adfa0 0x7f46380ae430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:18.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.983+0000 7f463e600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46380adfa0 0x7f46380ae430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.90:56746/0 (socket says 172.21.15.90:56746) 2024-03-24T01:24:18.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.983+0000 7f463e600640 1 -- 172.21.15.90:0/503024869 learned_addr learned my addr 172.21.15.90:0/503024869 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:18.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.983+0000 7f463d200640 1 --2- 172.21.15.90:0/503024869 >> v2:172.21.15.120:3300/0 conn(0x7f46380b4410 0x7f46380b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:18.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.983+0000 7f463e600640 1 -- 172.21.15.90:0/503024869 >> v2:172.21.15.120:3300/0 conn(0x7f46380b4410 msgr2=0x7f46380b48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:18.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.983+0000 7f463e600640 1 --2- 172.21.15.90:0/503024869 >> v2:172.21.15.120:3300/0 conn(0x7f46380b4410 0x7f46380b48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:18.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.983+0000 7f463e600640 1 -- 172.21.15.90:0/503024869 >> v1:172.21.15.120:6789/0 conn(0x7f4638071790 legacy=0x7f4638071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:18.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.983+0000 7f463e600640 1 -- 172.21.15.90:0/503024869 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f463806e070 con 0x7f46380adfa0 2024-03-24T01:24:18.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.985+0000 7f463e600640 1 --2- 172.21.15.90:0/503024869 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46380adfa0 0x7f46380ae430 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4630009e50 tx=0x7f463002bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a81144b16b28b1b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:18.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.985+0000 7f4637e00640 1 -- 172.21.15.90:0/503024869 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463002caf0 con 0x7f46380adfa0 2024-03-24T01:24:18.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.985+0000 7f4637e00640 1 -- 172.21.15.90:0/503024869 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f463002cc90 con 0x7f46380adfa0 2024-03-24T01:24:18.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.985+0000 7f4637e00640 1 -- 172.21.15.90:0/503024869 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4630034610 con 0x7f46380adfa0 2024-03-24T01:24:18.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.986+0000 7f463f000640 1 -- 172.21.15.90:0/503024869 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46380adfa0 msgr2=0x7f46380ae430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:18.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.986+0000 7f463f000640 1 --2- 172.21.15.90:0/503024869 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46380adfa0 0x7f46380ae430 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4630009e50 tx=0x7f463002bf50 comp rx=0 tx=0).stop 2024-03-24T01:24:18.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.986+0000 7f463f000640 1 -- 172.21.15.90:0/503024869 shutdown_connections 2024-03-24T01:24:18.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.986+0000 7f463f000640 1 --2- 172.21.15.90:0/503024869 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46380adfa0 0x7f46380ae430 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:18.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.986+0000 7f463f000640 1 --2- 172.21.15.90:0/503024869 >> v2:172.21.15.120:3300/0 conn(0x7f46380b4410 0x7f46380b48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:18.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.986+0000 7f463f000640 1 -- 172.21.15.90:0/503024869 >> 172.21.15.90:0/503024869 conn(0x7f463806b0c0 msgr2=0x7f463806b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:18.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.986+0000 7f463f000640 1 -- 172.21.15.90:0/503024869 shutdown_connections 2024-03-24T01:24:18.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.986+0000 7f463f000640 1 -- 172.21.15.90:0/503024869 wait complete. 2024-03-24T01:24:18.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.988+0000 7f463f000640 1 Processor -- start 2024-03-24T01:24:18.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.988+0000 7f463f000640 1 -- start start 2024-03-24T01:24:18.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.989+0000 7f463f000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4638071790 0x7f46380c4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:18.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.989+0000 7f463f000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46380adfa0 0x7f463815aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:18.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.989+0000 7f463f000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46380b4410 0x7f46381f6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:18.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.989+0000 7f463f000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463815f050 con 0x7f46380adfa0 2024-03-24T01:24:18.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.989+0000 7f463f000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46380cf460 con 0x7f4638071790 2024-03-24T01:24:18.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.989+0000 7f463f000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46380ae970 con 0x7f46380b4410 2024-03-24T01:24:18.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.989+0000 7f463d200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46380adfa0 0x7f463815aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:18.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.989+0000 7f463d200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46380adfa0 0x7f463815aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50732/0 (socket says 172.21.15.90:50732) 2024-03-24T01:24:18.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.989+0000 7f463d200640 1 -- 172.21.15.90:0/3176291411 learned_addr learned my addr 172.21.15.90:0/3176291411 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:18.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.989+0000 7f463e600640 1 --2- 172.21.15.90:0/3176291411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46380b4410 0x7f46381f6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:18.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.989+0000 7f463dc00640 1 --2- 172.21.15.90:0/3176291411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4638071790 0x7f46380c4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:18.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.990+0000 7f463d200640 1 -- 172.21.15.90:0/3176291411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46380b4410 msgr2=0x7f46381f6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:18.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.990+0000 7f463d200640 1 --2- 172.21.15.90:0/3176291411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46380b4410 0x7f46381f6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:18.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.990+0000 7f463d200640 1 -- 172.21.15.90:0/3176291411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4638071790 msgr2=0x7f46380c4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:18.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.990+0000 7f463d200640 1 --2- 172.21.15.90:0/3176291411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4638071790 0x7f46380c4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:18.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.990+0000 7f463d200640 1 -- 172.21.15.90:0/3176291411 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46381f5890 con 0x7f46380adfa0 2024-03-24T01:24:18.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.990+0000 7f463d200640 1 --2- 172.21.15.90:0/3176291411 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46380adfa0 0x7f463815aa30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f462400d5d0 tx=0x7f462400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:18.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.990+0000 7f4635600640 1 -- 172.21.15.90:0/3176291411 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4624004240 con 0x7f46380adfa0 2024-03-24T01:24:18.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.990+0000 7f4635600640 1 -- 172.21.15.90:0/3176291411 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46240043e0 con 0x7f46380adfa0 2024-03-24T01:24:18.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.990+0000 7f463f000640 1 -- 172.21.15.90:0/3176291411 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46381f5b70 con 0x7f46380adfa0 2024-03-24T01:24:18.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.990+0000 7f4635600640 1 -- 172.21.15.90:0/3176291411 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4624003830 con 0x7f46380adfa0 2024-03-24T01:24:18.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.990+0000 7f463f000640 1 -- 172.21.15.90:0/3176291411 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46381f6130 con 0x7f46380adfa0 2024-03-24T01:24:18.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.993+0000 7f4635600640 1 -- 172.21.15.90:0/3176291411 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f46240039d0 con 0x7f46380adfa0 2024-03-24T01:24:18.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.993+0000 7f4635600640 1 --2- 172.21.15.90:0/3176291411 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f461806bbd0 0x7f461806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:18.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.994+0000 7f4635600640 1 -- 172.21.15.90:0/3176291411 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f462408b060 con 0x7f46380adfa0 2024-03-24T01:24:18.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.994+0000 7f463dc00640 1 --2- 172.21.15.90:0/3176291411 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f461806bbd0 0x7f461806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:18.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.994+0000 7f4643235740 1 Processor -- start 2024-03-24T01:24:18.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.995+0000 7f4643235740 1 -- start start 2024-03-24T01:24:18.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.995+0000 7f463dc00640 1 --2- 172.21.15.90:0/3176291411 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f461806bbd0 0x7f461806e090 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f462c001640 tx=0x7f462c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:18.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.995+0000 7f4643235740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f95ecc2db0 0x55f95ecc3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:18.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.995+0000 7f4643235740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f95ec92dd0 0x55f95ecc0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:18.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.995+0000 7f4643235740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95ec96930 0x55f95ecc0560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:18.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.996+0000 7f4643235740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f95ebfcc90 con 0x55f95ec96930 2024-03-24T01:24:18.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.996+0000 7f4643235740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f95eba5980 con 0x55f95ecc2db0 2024-03-24T01:24:18.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.996+0000 7f4643235740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f95ebf8d30 con 0x55f95ec92dd0 2024-03-24T01:24:18.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.996+0000 7f463e600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95ec96930 0x55f95ecc0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:18.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.996+0000 7f463e600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95ec96930 0x55f95ecc0560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50746/0 (socket says 172.21.15.90:50746) 2024-03-24T01:24:18.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.996+0000 7f463e600640 1 -- 172.21.15.90:0/578535768 learned_addr learned my addr 172.21.15.90:0/578535768 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:18.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.996+0000 7f463d200640 1 --2- 172.21.15.90:0/578535768 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f95ec92dd0 0x55f95ecc0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:18.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.996+0000 7f463dc00640 1 --2- 172.21.15.90:0/578535768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f95ecc2db0 0x55f95ecc3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:18.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.996+0000 7f463e600640 1 -- 172.21.15.90:0/578535768 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f95ec92dd0 msgr2=0x55f95ecc0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:18.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.996+0000 7f463e600640 1 --2- 172.21.15.90:0/578535768 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f95ec92dd0 0x55f95ecc0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:18.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.996+0000 7f463e600640 1 -- 172.21.15.90:0/578535768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f95ecc2db0 msgr2=0x55f95ecc3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:18.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.996+0000 7f463e600640 1 --2- 172.21.15.90:0/578535768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f95ecc2db0 0x55f95ecc3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:18.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.996+0000 7f463e600640 1 -- 172.21.15.90:0/578535768 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f95ebbd510 con 0x55f95ec96930 2024-03-24T01:24:18.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.997+0000 7f463e600640 1 --2- 172.21.15.90:0/578535768 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95ec96930 0x55f95ecc0560 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4630009e50 tx=0x7f463002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a4f5ad1b66a1358 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:18.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.997+0000 7f462a000640 1 -- 172.21.15.90:0/578535768 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463002c9b0 con 0x55f95ec96930 2024-03-24T01:24:19.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.997+0000 7f462a000640 1 -- 172.21.15.90:0/578535768 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f463002cb50 con 0x55f95ec96930 2024-03-24T01:24:19.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.997+0000 7f462a000640 1 -- 172.21.15.90:0/578535768 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463003f660 con 0x55f95ec96930 2024-03-24T01:24:19.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.997+0000 7f4643235740 1 -- 172.21.15.90:0/578535768 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95ec96930 msgr2=0x55f95ecc0560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:19.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.997+0000 7f4643235740 1 --2- 172.21.15.90:0/578535768 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95ec96930 0x55f95ecc0560 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4630009e50 tx=0x7f463002e000 comp rx=0 tx=0).stop 2024-03-24T01:24:19.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.997+0000 7f4643235740 1 -- 172.21.15.90:0/578535768 shutdown_connections 2024-03-24T01:24:19.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.997+0000 7f4643235740 1 --2- 172.21.15.90:0/578535768 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95ec96930 0x55f95ecc0560 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:19.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.998+0000 7f4643235740 1 --2- 172.21.15.90:0/578535768 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f95ec92dd0 0x55f95ecc0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:19.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.998+0000 7f4643235740 1 --2- 172.21.15.90:0/578535768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f95ecc2db0 0x55f95ecc3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:19.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.998+0000 7f4643235740 1 -- 172.21.15.90:0/578535768 >> 172.21.15.90:0/578535768 conn(0x55f95ec95040 msgr2=0x55f95ecddbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:19.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.998+0000 7f4643235740 1 -- 172.21.15.90:0/578535768 shutdown_connections 2024-03-24T01:24:19.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.998+0000 7f4643235740 1 -- 172.21.15.90:0/578535768 wait complete. 2024-03-24T01:24:19.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.998+0000 7f4643235740 1 -- start start 2024-03-24T01:24:19.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.999+0000 7f4643235740 1 Processor -- start 2024-03-24T01:24:19.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:18.999+0000 7f4643235740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95ec96930 0x55f95ecb5f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:19.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.000+0000 7f4643235740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f95ecc2db0 0x55f95ecbdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:19.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.000+0000 7f4643235740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f95ecbe230 0x55f95ecc1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:19.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.000+0000 7f4643235740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f95ec97ae0 con 0x55f95ec96930 2024-03-24T01:24:19.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.000+0000 7f4643235740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f95ebb8dd0 con 0x55f95ecbe230 2024-03-24T01:24:19.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.000+0000 7f4643235740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f95ec96ed0 con 0x55f95ecc2db0 2024-03-24T01:24:19.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.000+0000 7f4643235740 10 client.0 _ll_register_callbacks cb 0x55f95e910ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T01:24:19.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.000+0000 7f463dc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95ec96930 0x55f95ecb5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:19.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.000+0000 7f463dc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95ec96930 0x55f95ecb5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50760/0 (socket says 172.21.15.90:50760) 2024-03-24T01:24:19.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.000+0000 7f463dc00640 1 -- 172.21.15.90:0/2344523303 learned_addr learned my addr 172.21.15.90:0/2344523303 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:19.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.000+0000 7f463d200640 1 --2- 172.21.15.90:0/2344523303 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f95ecc2db0 0x55f95ecbdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:19.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.000+0000 7f463e600640 1 --2- 172.21.15.90:0/2344523303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f95ecbe230 0x55f95ecc1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:19.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.000+0000 7f463dc00640 1 -- 172.21.15.90:0/2344523303 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f95ecc2db0 msgr2=0x55f95ecbdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:19.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.000+0000 7f463dc00640 1 --2- 172.21.15.90:0/2344523303 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f95ecc2db0 0x55f95ecbdcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:19.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.000+0000 7f463dc00640 1 -- 172.21.15.90:0/2344523303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f95ecbe230 msgr2=0x55f95ecc1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:19.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.000+0000 7f463dc00640 1 --2- 172.21.15.90:0/2344523303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f95ecbe230 0x55f95ecc1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:19.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.000+0000 7f463dc00640 1 -- 172.21.15.90:0/2344523303 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f95e9112d0 con 0x55f95ec96930 2024-03-24T01:24:19.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.001+0000 7f463dc00640 1 --2- 172.21.15.90:0/2344523303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95ec96930 0x55f95ecb5f80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f462c0038c0 tx=0x7f462c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51c7b3f954d40d1b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:19.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.001+0000 7f461ea00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-03-24T01:24:19.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.001+0000 7f461ea00640 1 -- 172.21.15.90:0/2344523303 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462c027060 con 0x55f95ec96930 2024-03-24T01:24:19.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.001+0000 7f461ea00640 1 -- 172.21.15.90:0/2344523303 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f462c010040 con 0x55f95ec96930 2024-03-24T01:24:19.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.001+0000 7f4643235740 1 -- 172.21.15.90:0/2344523303 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f463002cb50 con 0x55f95ec96930 2024-03-24T01:24:19.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.001+0000 7f461ea00640 1 -- 172.21.15.90:0/2344523303 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) v2 ==== 24+0+0 (secure 0 0 0) 0x7f462c003e40 con 0x55f95ec96930 2024-03-24T01:24:19.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.001+0000 7f4643235740 10 client.4386 fetch_fsmap learned FSMap version 6 2024-03-24T01:24:19.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.001+0000 7f4643235740 1 -- 172.21.15.90:0/2344523303 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) v3 -- 0x55f95e9db6b0 con 0x55f95ec96930 2024-03-24T01:24:19.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.001+0000 7f461ea00640 1 -- 172.21.15.90:0/2344523303 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f463002cb50 con 0x55f95ec96930 2024-03-24T01:24:19.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.001+0000 7f461ea00640 1 -- 172.21.15.90:0/2344523303 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 6) v1 ==== 3322+0+0 (secure 0 0 0) 0x7f462c00ecc0 con 0x55f95ec96930 2024-03-24T01:24:19.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.001+0000 7f4643235740 10 client.4386 fetch_fsmap finished waiting for FSMap version 6 2024-03-24T01:24:19.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.001+0000 7f4643235740 10 client.4386 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-24T01:24:19.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.001+0000 7f4643235740 20 client.4386 populate_metadata read hostname 'smithi090' 2024-03-24T01:24:19.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.001+0000 7f4643235740 1 --2- 172.21.15.90:0/2344523303 >> [v2:172.21.15.120:6834/963928022,v1:172.21.15.120:6836/963928022] conn(0x55f95ecb8dc0 0x55f95ecb91a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:19.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.001+0000 7f4643235740 4 client.4386 mds_command: new command op to 4257 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T01:24:19.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.002+0000 7f4643235740 1 -- 172.21.15.90:0/2344523303 --> [v2:172.21.15.120:6834/963928022,v1:172.21.15.120:6836/963928022] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f95eb9e520 con 0x55f95ecb8dc0 2024-03-24T01:24:19.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.002+0000 7f463d200640 1 --2- 172.21.15.90:0/2344523303 >> [v2:172.21.15.120:6834/963928022,v1:172.21.15.120:6836/963928022] conn(0x55f95ecb8dc0 0x55f95ecb91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:19.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.003+0000 7f463d200640 1 --2- 172.21.15.90:0/2344523303 >> [v2:172.21.15.120:6834/963928022,v1:172.21.15.120:6836/963928022] conn(0x55f95ecb8dc0 0x55f95ecb91a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9250d44ba86a80 server_cookie=f5f7a64e53b40ca1 in_seq=0 out_seq=0 2024-03-24T01:24:19.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.003+0000 7f461ea00640 10 client.4386 ms_handle_connect on v2:172.21.15.120:6834/963928022 2024-03-24T01:24:19.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.005+0000 7f461ea00640 1 -- 172.21.15.90:0/2344523303 <== mds.0 v2:172.21.15.120:6834/963928022 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f95eb9e520 con 0x55f95ecb8dc0 2024-03-24T01:24:19.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.005+0000 7f461ea00640 10 client.4386 handle_command_reply: tid=0 2024-03-24T01:24:19.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.005+0000 7f4643235740 1 client.4386 shutdown 2024-03-24T01:24:19.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.005+0000 7f4643235740 1 -- 172.21.15.90:0/2344523303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95ec96930 msgr2=0x55f95ecb5f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:19.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.005+0000 7f4643235740 1 --2- 172.21.15.90:0/2344523303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95ec96930 0x55f95ecb5f80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f462c0038c0 tx=0x7f462c000bc0 comp rx=0 tx=0).stop 2024-03-24T01:24:19.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.005+0000 7f4643235740 1 -- 172.21.15.90:0/2344523303 shutdown_connections 2024-03-24T01:24:19.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.005+0000 7f4643235740 1 --2- 172.21.15.90:0/2344523303 >> [v2:172.21.15.120:6834/963928022,v1:172.21.15.120:6836/963928022] conn(0x55f95ecb8dc0 0x55f95ecb91a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:19.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.005+0000 7f4643235740 1 --2- 172.21.15.90:0/2344523303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f95ecbe230 0x55f95ecc1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:19.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.005+0000 7f4643235740 1 --2- 172.21.15.90:0/2344523303 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f95ecc2db0 0x55f95ecbdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:19.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.005+0000 7f4643235740 1 --2- 172.21.15.90:0/2344523303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95ec96930 0x55f95ecb5f80 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:19.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.005+0000 7f461ea00640 0 client.4386 ms_handle_reset on v2:172.21.15.120:6834/963928022 2024-03-24T01:24:19.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.005+0000 7f4643235740 1 -- 172.21.15.90:0/2344523303 >> 172.21.15.90:0/2344523303 conn(0x55f95ec92dd0 msgr2=0x55f95ece10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:19.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.005+0000 7f4643235740 1 -- 172.21.15.90:0/2344523303 shutdown_connections 2024-03-24T01:24:19.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.005+0000 7f4643235740 1 -- 172.21.15.90:0/2344523303 wait complete. 2024-03-24T01:24:19.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.005+0000 7f4643235740 20 client.4386 trim_cache size 0 max 16384 2024-03-24T01:24:19.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.023+0000 7f4643235740 1 Processor -- start 2024-03-24T01:24:19.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.023+0000 7f4643235740 1 -- start start 2024-03-24T01:24:19.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.024+0000 7f4643235740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95ec96930 0x55f95ecbb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:19.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.024+0000 7f4643235740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f95ecbe230 0x55f95ece10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:19.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.024+0000 7f4643235740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f95ecc2db0 0x55f95ece1610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:19.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.024+0000 7f4643235740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46382005f0 con 0x55f95ec96930 2024-03-24T01:24:19.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.024+0000 7f4643235740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46381f50a0 con 0x55f95ecbe230 2024-03-24T01:24:19.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.024+0000 7f4643235740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4638200020 con 0x55f95ecc2db0 2024-03-24T01:24:19.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.024+0000 7f463dc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95ec96930 0x55f95ecbb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:19.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.025+0000 7f463dc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95ec96930 0x55f95ecbb4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50776/0 (socket says 172.21.15.90:50776) 2024-03-24T01:24:19.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.025+0000 7f463dc00640 1 -- 172.21.15.90:0/3413344614 learned_addr learned my addr 172.21.15.90:0/3413344614 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:19.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.025+0000 7f463dc00640 1 -- 172.21.15.90:0/3413344614 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f95ecc2db0 msgr2=0x55f95ece1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:19.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.025+0000 7f463dc00640 1 --2- 172.21.15.90:0/3413344614 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f95ecc2db0 0x55f95ece1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:19.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.025+0000 7f463d200640 1 --2- 172.21.15.90:0/3413344614 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f95ecbe230 0x55f95ece10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:19.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.025+0000 7f463dc00640 1 -- 172.21.15.90:0/3413344614 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f95ecbe230 msgr2=0x55f95ece10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:19.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.025+0000 7f463dc00640 1 --2- 172.21.15.90:0/3413344614 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f95ecbe230 0x55f95ece10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:19.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.025+0000 7f463dc00640 1 -- 172.21.15.90:0/3413344614 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f95ec99bd0 con 0x55f95ec96930 2024-03-24T01:24:19.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.025+0000 7f463dc00640 1 --2- 172.21.15.90:0/3413344614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95ec96930 0x55f95ecbb4f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f462c0012c0 tx=0x7f462c01f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddf66d99c354f0af server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:19.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.025+0000 7f4629600640 1 -- 172.21.15.90:0/3413344614 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462c00ecc0 con 0x55f95ec96930 2024-03-24T01:24:19.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.026+0000 7f4629600640 1 -- 172.21.15.90:0/3413344614 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f462c00ee60 con 0x55f95ec96930 2024-03-24T01:24:19.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.026+0000 7f4629600640 1 -- 172.21.15.90:0/3413344614 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462c013790 con 0x55f95ec96930 2024-03-24T01:24:19.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.026+0000 7f4643235740 1 -- 172.21.15.90:0/3413344614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95ec96930 msgr2=0x55f95ecbb4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:19.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.026+0000 7f4643235740 1 --2- 172.21.15.90:0/3413344614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95ec96930 0x55f95ecbb4f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f462c0012c0 tx=0x7f462c01f000 comp rx=0 tx=0).stop 2024-03-24T01:24:19.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.026+0000 7f4643235740 1 -- 172.21.15.90:0/3413344614 shutdown_connections 2024-03-24T01:24:19.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.026+0000 7f4643235740 1 --2- 172.21.15.90:0/3413344614 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f95ecc2db0 0x55f95ece1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:19.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.026+0000 7f4643235740 1 --2- 172.21.15.90:0/3413344614 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f95ecbe230 0x55f95ece10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:19.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.026+0000 7f4643235740 1 --2- 172.21.15.90:0/3413344614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95ec96930 0x55f95ecbb4f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:19.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.026+0000 7f4643235740 1 -- 172.21.15.90:0/3413344614 >> 172.21.15.90:0/3413344614 conn(0x55f95ec92dd0 msgr2=0x55f95ecb5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:19.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.026+0000 7f4643235740 1 -- 172.21.15.90:0/3413344614 shutdown_connections 2024-03-24T01:24:19.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.026+0000 7f4643235740 1 -- 172.21.15.90:0/3413344614 wait complete. 2024-03-24T01:24:19.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.027+0000 7f4643235740 1 -- start start 2024-03-24T01:24:19.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.027+0000 7f4643235740 1 Processor -- start 2024-03-24T01:24:19.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.028+0000 7f4643235740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f95ec96930 0x55f95ec97830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:19.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.028+0000 7f4643235740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f95ecbe230 0x55f95ec97d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:19.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.028+0000 7f4643235740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95ecc2db0 0x55f95ecf0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:19.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.028+0000 7f4643235740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f95ecb6790 con 0x55f95ecc2db0 2024-03-24T01:24:19.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.028+0000 7f4643235740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f95ecf1e20 con 0x55f95ec96930 2024-03-24T01:24:19.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.028+0000 7f4643235740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f95ecbc890 con 0x55f95ecbe230 2024-03-24T01:24:19.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.028+0000 7f4643235740 10 client.0 _ll_register_callbacks cb 0x7f462c0050d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T01:24:19.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.029+0000 7f463e600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95ecc2db0 0x55f95ecf0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:19.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.029+0000 7f463e600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95ecc2db0 0x55f95ecf0140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50778/0 (socket says 172.21.15.90:50778) 2024-03-24T01:24:19.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.029+0000 7f463e600640 1 -- 172.21.15.90:0/4032428366 learned_addr learned my addr 172.21.15.90:0/4032428366 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:19.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.029+0000 7f463dc00640 1 --2- 172.21.15.90:0/4032428366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f95ec96930 0x55f95ec97830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:19.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.029+0000 7f463d200640 1 --2- 172.21.15.90:0/4032428366 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f95ecbe230 0x55f95ec97d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:19.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.029+0000 7f463e600640 1 -- 172.21.15.90:0/4032428366 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f95ecbe230 msgr2=0x55f95ec97d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:19.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.029+0000 7f463e600640 1 --2- 172.21.15.90:0/4032428366 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f95ecbe230 0x55f95ec97d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:19.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.029+0000 7f463e600640 1 -- 172.21.15.90:0/4032428366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f95ec96930 msgr2=0x55f95ec97830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:19.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.029+0000 7f463e600640 1 --2- 172.21.15.90:0/4032428366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f95ec96930 0x55f95ec97830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:19.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.029+0000 7f463e600640 1 -- 172.21.15.90:0/4032428366 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f95ecbc030 con 0x55f95ecc2db0 2024-03-24T01:24:19.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.029+0000 7f463d200640 1 --2- 172.21.15.90:0/4032428366 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f95ecbe230 0x55f95ec97d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:19.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.029+0000 7f463e600640 1 --2- 172.21.15.90:0/4032428366 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95ecc2db0 0x55f95ecf0140 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4630009f40 tx=0x7f4630004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce07f9fc233d9aab server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:19.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.029+0000 7f461ea00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-03-24T01:24:19.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.030+0000 7f461ea00640 1 -- 172.21.15.90:0/4032428366 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463004d070 con 0x55f95ecc2db0 2024-03-24T01:24:19.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.030+0000 7f461ea00640 1 -- 172.21.15.90:0/4032428366 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f463003fbb0 con 0x55f95ecc2db0 2024-03-24T01:24:19.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.030+0000 7f4643235740 1 -- 172.21.15.90:0/4032428366 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f462c00ee60 con 0x55f95ecc2db0 2024-03-24T01:24:19.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.030+0000 7f461ea00640 1 -- 172.21.15.90:0/4032428366 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4630059050 con 0x55f95ecc2db0 2024-03-24T01:24:19.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.030+0000 7f4643235740 10 client.4392 fetch_fsmap learned FSMap version 6 2024-03-24T01:24:19.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.030+0000 7f4643235740 1 -- 172.21.15.90:0/4032428366 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) v3 -- 0x55f95ed55a30 con 0x55f95ecc2db0 2024-03-24T01:24:19.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.030+0000 7f461ea00640 1 -- 172.21.15.90:0/4032428366 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f462c00ee60 con 0x55f95ecc2db0 2024-03-24T01:24:19.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.031+0000 7f461ea00640 1 -- 172.21.15.90:0/4032428366 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 6) v1 ==== 3322+0+0 (secure 0 0 0) 0x7f463005c020 con 0x55f95ecc2db0 2024-03-24T01:24:19.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.031+0000 7f4643235740 10 client.4392 fetch_fsmap finished waiting for FSMap version 6 2024-03-24T01:24:19.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.031+0000 7f4643235740 10 client.4392 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-24T01:24:19.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.031+0000 7f4643235740 20 client.4392 populate_metadata read hostname 'smithi090' 2024-03-24T01:24:19.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.031+0000 7f4643235740 1 --2- 172.21.15.90:0/4032428366 >> [v2:172.21.15.120:6834/963928022,v1:172.21.15.120:6836/963928022] conn(0x55f95ecf06d0 0x55f95ecc5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:19.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.031+0000 7f4643235740 4 client.4392 mds_command: new command op to 4257 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T01:24:19.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.031+0000 7f4643235740 1 -- 172.21.15.90:0/4032428366 --> [v2:172.21.15.120:6834/963928022,v1:172.21.15.120:6836/963928022] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f95ecc3880 con 0x55f95ecf06d0 2024-03-24T01:24:19.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.031+0000 7f463dc00640 1 --2- 172.21.15.90:0/4032428366 >> [v2:172.21.15.120:6834/963928022,v1:172.21.15.120:6836/963928022] conn(0x55f95ecf06d0 0x55f95ecc5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:19.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.032+0000 7f463dc00640 1 --2- 172.21.15.90:0/4032428366 >> [v2:172.21.15.120:6834/963928022,v1:172.21.15.120:6836/963928022] conn(0x55f95ecf06d0 0x55f95ecc5ae0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d3074849ba3d82b server_cookie=42931e2786f638ce in_seq=0 out_seq=0 2024-03-24T01:24:19.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.032+0000 7f461ea00640 10 client.4392 ms_handle_connect on v2:172.21.15.120:6834/963928022 2024-03-24T01:24:19.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.033+0000 7f461ea00640 1 -- 172.21.15.90:0/4032428366 <== mds.0 v2:172.21.15.120:6834/963928022 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x55f95ecc3880 con 0x55f95ecf06d0 2024-03-24T01:24:19.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.033+0000 7f461ea00640 10 client.4392 handle_command_reply: tid=0 2024-03-24T01:24:19.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.033+0000 7f4643235740 1 client.4392 shutdown 2024-03-24T01:24:19.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.033+0000 7f4643235740 1 -- 172.21.15.90:0/4032428366 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95ecc2db0 msgr2=0x55f95ecf0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:19.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.033+0000 7f4643235740 1 --2- 172.21.15.90:0/4032428366 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95ecc2db0 0x55f95ecf0140 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4630009f40 tx=0x7f4630004a40 comp rx=0 tx=0).stop 2024-03-24T01:24:19.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.033+0000 7f4643235740 1 -- 172.21.15.90:0/4032428366 shutdown_connections 2024-03-24T01:24:19.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.033+0000 7f4643235740 1 --2- 172.21.15.90:0/4032428366 >> [v2:172.21.15.120:6834/963928022,v1:172.21.15.120:6836/963928022] conn(0x55f95ecf06d0 0x55f95ecc5ae0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:19.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.033+0000 7f4643235740 1 --2- 172.21.15.90:0/4032428366 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95ecc2db0 0x55f95ecf0140 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:19.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.033+0000 7f4643235740 1 --2- 172.21.15.90:0/4032428366 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55f95ecbe230 0x55f95ec97d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:19.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.033+0000 7f4643235740 1 --2- 172.21.15.90:0/4032428366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f95ec96930 0x55f95ec97830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:19.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.033+0000 7f461ea00640 0 client.4392 ms_handle_reset on v2:172.21.15.120:6834/963928022 2024-03-24T01:24:19.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.033+0000 7f4643235740 1 -- 172.21.15.90:0/4032428366 >> 172.21.15.90:0/4032428366 conn(0x55f95ec92dd0 msgr2=0x55f95ece10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:19.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.033+0000 7f4643235740 1 -- 172.21.15.90:0/4032428366 shutdown_connections 2024-03-24T01:24:19.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.033+0000 7f4643235740 1 -- 172.21.15.90:0/4032428366 wait complete. 2024-03-24T01:24:19.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.033+0000 7f4643235740 20 client.4392 trim_cache size 0 max 16384 2024-03-24T01:24:19.036 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-24T01:24:19.036 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "24712165-9839-4fcc-8033-988386efea74", 2024-03-24T01:24:19.037 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 0, 2024-03-24T01:24:19.037 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 4257, 2024-03-24T01:24:19.037 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-03-24T01:24:19.037 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-03-24T01:24:19.037 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-03-24T01:24:19.037 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 8.3066068519999998, 2024-03-24T01:24:19.037 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 6, 2024-03-24T01:24:19.037 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 25, 2024-03-24T01:24:19.037 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 24, 2024-03-24T01:24:19.037 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 13.363491467999999 2024-03-24T01:24:19.037 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-24T01:24:19.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.034+0000 7f463f000640 1 -- 172.21.15.90:0/3176291411 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f461806bbd0 msgr2=0x7f461806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:19.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.034+0000 7f463f000640 1 --2- 172.21.15.90:0/3176291411 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f461806bbd0 0x7f461806e090 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f462c001640 tx=0x7f462c009040 comp rx=0 tx=0).stop 2024-03-24T01:24:19.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.034+0000 7f463f000640 1 -- 172.21.15.90:0/3176291411 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46380adfa0 msgr2=0x7f463815aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:19.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.034+0000 7f463f000640 1 --2- 172.21.15.90:0/3176291411 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46380adfa0 0x7f463815aa30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f462400d5d0 tx=0x7f462400da90 comp rx=0 tx=0).stop 2024-03-24T01:24:19.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.034+0000 7f463f000640 1 -- 172.21.15.90:0/3176291411 shutdown_connections 2024-03-24T01:24:19.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.034+0000 7f463f000640 1 --2- 172.21.15.90:0/3176291411 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f461806bbd0 0x7f461806e090 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:19.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.034+0000 7f463f000640 1 --2- 172.21.15.90:0/3176291411 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46380adfa0 0x7f463815aa30 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:19.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.034+0000 7f463f000640 1 --2- 172.21.15.90:0/3176291411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f46380b4410 0x7f46381f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:19.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.034+0000 7f463f000640 1 --2- 172.21.15.90:0/3176291411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4638071790 0x7f46380c4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:19.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.034+0000 7f463f000640 1 -- 172.21.15.90:0/3176291411 >> 172.21.15.90:0/3176291411 conn(0x7f463806b0c0 msgr2=0x7f463815cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:19.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.034+0000 7f463f000640 1 -- 172.21.15.90:0/3176291411 shutdown_connections 2024-03-24T01:24:19.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.034+0000 7f463f000640 1 -- 172.21.15.90:0/3176291411 wait complete. 2024-03-24T01:24:19.054 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-24T01:24:19.054 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-24T01:24:19.055 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T01:24:19.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.258+0000 7f91dc400640 1 Processor -- start 2024-03-24T01:24:19.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.258+0000 7f91dc400640 1 -- start start 2024-03-24T01:24:19.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.260+0000 7f91dc400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91d40cefa0 0x7f91d4168c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:19.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.260+0000 7f91dc400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d41620a0 con 0x7f91d40b94e0 2024-03-24T01:24:19.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.260+0000 7f91dc400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d4162380 con 0x7f91d415df40 2024-03-24T01:24:19.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.260+0000 7f91dc400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91d40d1c70 con 0x7f91d40cefa0 2024-03-24T01:24:19.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.261+0000 7f91d9000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f91d40b94e0 0x7f91d40ce890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:37514/0 (socket says 172.21.15.90:37514) 2024-03-24T01:24:19.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.261+0000 7f91d9000640 1 -- 172.21.15.90:0/2376979276 learned_addr learned my addr 172.21.15.90:0/2376979276 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:19.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.261+0000 7f91da400640 1 --2- 172.21.15.90:0/2376979276 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91d40cefa0 0x7f91d4168c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:19.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.261+0000 7f91d3e00640 1 -- 172.21.15.90:0/2376979276 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4292527859 0 0) 0x7f91d41620a0 con 0x7f91d40b94e0 2024-03-24T01:24:19.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.261+0000 7f91d3e00640 1 -- 172.21.15.90:0/2376979276 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91bc003520 con 0x7f91d40b94e0 2024-03-24T01:24:19.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.261+0000 7f91d3e00640 1 -- 172.21.15.90:0/2376979276 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2135985017 0 0) 0x7f91bc003520 con 0x7f91d40b94e0 2024-03-24T01:24:19.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.261+0000 7f91d3e00640 1 -- 172.21.15.90:0/2376979276 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91d40cefa0 msgr2=0x7f91d4168c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:19.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.261+0000 7f91d3e00640 1 --2- 172.21.15.90:0/2376979276 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91d40cefa0 0x7f91d4168c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:19.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.261+0000 7f91d3e00640 1 -- 172.21.15.90:0/2376979276 >> v1:172.21.15.120:6789/0 conn(0x7f91d415df40 legacy=0x7f91d4161890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:19.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.261+0000 7f91d3e00640 1 -- 172.21.15.90:0/2376979276 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d4162660 con 0x7f91d40b94e0 2024-03-24T01:24:19.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.261+0000 7f91d3e00640 1 -- 172.21.15.90:0/2376979276 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f91c0001670 con 0x7f91d40b94e0 2024-03-24T01:24:19.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.261+0000 7f91d3e00640 1 -- 172.21.15.90:0/2376979276 >> v1:172.21.15.90:6789/0 conn(0x7f91d40b94e0 legacy=0x7f91d40ce890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:19.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.262+0000 7f91d3e00640 1 --2- 172.21.15.90:0/2376979276 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91bc0039e0 0x7f91bc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:19.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.262+0000 7f91d3e00640 1 --2- 172.21.15.90:0/2376979276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91bc004800 0x7f91bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:19.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.262+0000 7f91d3e00640 1 --2- 172.21.15.90:0/2376979276 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91d40cefa0 0x7f91bc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:19.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.262+0000 7f91d3e00640 1 -- 172.21.15.90:0/2376979276 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d40d1c70 con 0x7f91bc0039e0 2024-03-24T01:24:19.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.262+0000 7f91d3e00640 1 -- 172.21.15.90:0/2376979276 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91bc00baa0 con 0x7f91bc004800 2024-03-24T01:24:19.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.262+0000 7f91d3e00640 1 -- 172.21.15.90:0/2376979276 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91bc00bca0 con 0x7f91d40cefa0 2024-03-24T01:24:19.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.262+0000 7f91d9a00640 1 --2- 172.21.15.90:0/2376979276 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91bc0039e0 0x7f91bc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:19.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.262+0000 7f91d9000640 1 --2- 172.21.15.90:0/2376979276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91bc004800 0x7f91bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:19.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.262+0000 7f91da400640 1 --2- 172.21.15.90:0/2376979276 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91d40cefa0 0x7f91bc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:19.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.262+0000 7f91d9a00640 1 -- 172.21.15.90:0/2376979276 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91d40cefa0 msgr2=0x7f91bc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:19.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.262+0000 7f91d9a00640 1 --2- 172.21.15.90:0/2376979276 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91d40cefa0 0x7f91bc0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:19.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.262+0000 7f91d9a00640 1 -- 172.21.15.90:0/2376979276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91bc004800 msgr2=0x7f91bc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:19.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.262+0000 7f91d9a00640 1 --2- 172.21.15.90:0/2376979276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91bc004800 0x7f91bc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:19.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.262+0000 7f91d9a00640 1 -- 172.21.15.90:0/2376979276 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91bc003520 con 0x7f91bc0039e0 2024-03-24T01:24:19.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.263+0000 7f91d9a00640 1 --2- 172.21.15.90:0/2376979276 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91bc0039e0 0x7f91bc003e20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f91c8001720 tx=0x7f91bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29307a2cb688c3e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:19.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.264+0000 7f91d3e00640 1 -- 172.21.15.90:0/2376979276 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c8001370 con 0x7f91bc0039e0 2024-03-24T01:24:19.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.264+0000 7f91d3e00640 1 -- 172.21.15.90:0/2376979276 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91c80052d0 con 0x7f91bc0039e0 2024-03-24T01:24:19.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.264+0000 7f91d3e00640 1 -- 172.21.15.90:0/2376979276 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c8003830 con 0x7f91bc0039e0 2024-03-24T01:24:19.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.264+0000 7f91dc400640 1 -- 172.21.15.90:0/2376979276 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91bc0039e0 msgr2=0x7f91bc003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:19.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.264+0000 7f91dc400640 1 --2- 172.21.15.90:0/2376979276 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91bc0039e0 0x7f91bc003e20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f91c8001720 tx=0x7f91bc005830 comp rx=0 tx=0).stop 2024-03-24T01:24:19.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.264+0000 7f91da400640 1 -- 172.21.15.90:0/2376979276 reap_dead start 2024-03-24T01:24:19.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.264+0000 7f91dc400640 1 -- 172.21.15.90:0/2376979276 shutdown_connections 2024-03-24T01:24:19.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.264+0000 7f91dc400640 1 -- 172.21.15.90:0/2376979276 >> 172.21.15.90:0/2376979276 conn(0x7f91d406c9f0 msgr2=0x7f91d406ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:19.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.264+0000 7f91dc400640 1 -- 172.21.15.90:0/2376979276 shutdown_connections 2024-03-24T01:24:19.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.265+0000 7f91dc400640 1 -- 172.21.15.90:0/2376979276 wait complete. 2024-03-24T01:24:19.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.266+0000 7f91dc400640 1 Processor -- start 2024-03-24T01:24:19.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.266+0000 7f91dc400640 1 -- start start 2024-03-24T01:24:19.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.267+0000 7f91dc400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91bc0039e0 0x7f91d414f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:19.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.267+0000 7f91dc400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91d40ccf10 0x7f91d40d22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:19.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.267+0000 7f91dc400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91d40cd390 0x7f91d40d2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:19.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.267+0000 7f91dc400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d40d3570 con 0x7f91d40cd390 2024-03-24T01:24:19.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.267+0000 7f91dc400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d40d33f0 con 0x7f91d40ccf10 2024-03-24T01:24:19.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.267+0000 7f91dc400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91d40d36f0 con 0x7f91bc0039e0 2024-03-24T01:24:19.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.268+0000 7f91da400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91d40cd390 0x7f91d40d2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:19.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.268+0000 7f91da400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91d40cd390 0x7f91d40d2820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50792/0 (socket says 172.21.15.90:50792) 2024-03-24T01:24:19.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.268+0000 7f91da400640 1 -- 172.21.15.90:0/3236127719 learned_addr learned my addr 172.21.15.90:0/3236127719 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:19.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.268+0000 7f91da400640 1 -- 172.21.15.90:0/3236127719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91bc0039e0 msgr2=0x7f91d414f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:19.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.268+0000 7f91d9a00640 1 --2- 172.21.15.90:0/3236127719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91bc0039e0 0x7f91d414f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:19.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.268+0000 7f91d9000640 1 --2- 172.21.15.90:0/3236127719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91d40ccf10 0x7f91d40d22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:19.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.268+0000 7f91da400640 1 --2- 172.21.15.90:0/3236127719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91bc0039e0 0x7f91d414f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:19.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.268+0000 7f91da400640 1 -- 172.21.15.90:0/3236127719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91d40ccf10 msgr2=0x7f91d40d22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:19.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.268+0000 7f91da400640 1 --2- 172.21.15.90:0/3236127719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91d40ccf10 0x7f91d40d22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:19.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.268+0000 7f91da400640 1 -- 172.21.15.90:0/3236127719 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d4164e10 con 0x7f91d40cd390 2024-03-24T01:24:19.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.268+0000 7f91da400640 1 --2- 172.21.15.90:0/3236127719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91d40cd390 0x7f91d40d2820 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f91d40b6ae0 tx=0x7f91cc00f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:19.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.268+0000 7f91d1600640 1 -- 172.21.15.90:0/3236127719 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91cc01c070 con 0x7f91d40cd390 2024-03-24T01:24:19.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.268+0000 7f91d1600640 1 -- 172.21.15.90:0/3236127719 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91cc017400 con 0x7f91d40cd390 2024-03-24T01:24:19.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.268+0000 7f91d1600640 1 -- 172.21.15.90:0/3236127719 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91cc0177f0 con 0x7f91d40cd390 2024-03-24T01:24:19.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.268+0000 7f91dc400640 1 -- 172.21.15.90:0/3236127719 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91d4163d70 con 0x7f91d40cd390 2024-03-24T01:24:19.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.269+0000 7f91dc400640 1 -- 172.21.15.90:0/3236127719 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91d4164330 con 0x7f91d40cd390 2024-03-24T01:24:19.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.270+0000 7f91dc400640 1 -- 172.21.15.90:0/3236127719 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91d4068060 con 0x7f91d40cd390 2024-03-24T01:24:19.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.271+0000 7f91d1600640 1 -- 172.21.15.90:0/3236127719 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f91cc005d10 con 0x7f91d40cd390 2024-03-24T01:24:19.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.272+0000 7f91d1600640 1 --2- 172.21.15.90:0/3236127719 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f91b406bbd0 0x7f91b406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:19.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.272+0000 7f91d1600640 1 -- 172.21.15.90:0/3236127719 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f91cc08af00 con 0x7f91d40cd390 2024-03-24T01:24:19.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.273+0000 7f91d9a00640 1 --2- 172.21.15.90:0/3236127719 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f91b406bbd0 0x7f91b406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:19.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.274+0000 7f91d9a00640 1 --2- 172.21.15.90:0/3236127719 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f91b406bbd0 0x7f91b406e090 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f91c8033000 tx=0x7f91c8009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:19.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.277+0000 7f91d1600640 1 -- 172.21.15.90:0/3236127719 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f91cc012070 con 0x7f91d40cd390 2024-03-24T01:24:19.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.515+0000 7f91dc400640 1 -- 172.21.15.90:0/3236127719 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f91d40b3300 con 0x7f91d40cd390 2024-03-24T01:24:19.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.516+0000 7f91d1600640 1 -- 172.21.15.90:0/3236127719 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) v1 ==== 53+0+88 (secure 0 0 0) 0x7f91cc0596a0 con 0x7f91d40cd390 2024-03-24T01:24:19.517 INFO:teuthology.orchestra.run.smithi090.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-24T01:24:19.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.518+0000 7f91dc400640 1 -- 172.21.15.90:0/3236127719 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f91b406bbd0 msgr2=0x7f91b406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:19.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.518+0000 7f91dc400640 1 --2- 172.21.15.90:0/3236127719 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f91b406bbd0 0x7f91b406e090 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f91c8033000 tx=0x7f91c8009030 comp rx=0 tx=0).stop 2024-03-24T01:24:19.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.518+0000 7f91dc400640 1 -- 172.21.15.90:0/3236127719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91d40cd390 msgr2=0x7f91d40d2820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:19.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.518+0000 7f91dc400640 1 --2- 172.21.15.90:0/3236127719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91d40cd390 0x7f91d40d2820 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f91d40b6ae0 tx=0x7f91cc00f4a0 comp rx=0 tx=0).stop 2024-03-24T01:24:19.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.519+0000 7f91dc400640 1 -- 172.21.15.90:0/3236127719 shutdown_connections 2024-03-24T01:24:19.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.519+0000 7f91dc400640 1 --2- 172.21.15.90:0/3236127719 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f91b406bbd0 0x7f91b406e090 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:19.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.519+0000 7f91dc400640 1 --2- 172.21.15.90:0/3236127719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91d40cd390 0x7f91d40d2820 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:19.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.519+0000 7f91dc400640 1 --2- 172.21.15.90:0/3236127719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91d40ccf10 0x7f91d40d22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:19.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.519+0000 7f91dc400640 1 --2- 172.21.15.90:0/3236127719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91bc0039e0 0x7f91d414f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:19.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.519+0000 7f91dc400640 1 -- 172.21.15.90:0/3236127719 >> 172.21.15.90:0/3236127719 conn(0x7f91d406c9f0 msgr2=0x7f91d4073a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:19.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.519+0000 7f91dc400640 1 -- 172.21.15.90:0/3236127719 shutdown_connections 2024-03-24T01:24:19.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:19.519+0000 7f91dc400640 1 -- 172.21.15.90:0/3236127719 wait complete. 2024-03-24T01:24:19.539 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-24T01:24:19.539 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T01:24:19.539 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi090.front.sepia.ceph.com 2024-03-24T01:24:19.539 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-24T01:24:19.539 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-24T01:24:19.539 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-24T01:24:19.539 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T01:24:19.539 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T01:24:19.539 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-24T01:24:19.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:19.539 DEBUG:teuthology.orchestra.run.smithi090:> ip addr 2024-03-24T01:24:19.589 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T01:24:19.590 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T01:24:19.590 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T01:24:19.590 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:24:19.590 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-03-24T01:24:19.590 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:24:19.590 INFO:teuthology.orchestra.run.smithi090.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T01:24:19.590 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a8 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:24:19.590 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f0 2024-03-24T01:24:19.590 INFO:teuthology.orchestra.run.smithi090.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T01:24:19.590 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a9 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:24:19.590 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f1 2024-03-24T01:24:19.591 INFO:teuthology.orchestra.run.smithi090.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T01:24:19.591 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:66 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:24:19.591 INFO:teuthology.orchestra.run.smithi090.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T01:24:19.591 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:67 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:24:19.591 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.21.15.90/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T01:24:19.591 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft 42522sec preferred_lft 42522sec 2024-03-24T01:24:19.591 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::ec4:7aff:fe8f:d767/64 scope link 2024-03-24T01:24:19.591 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:24:19.592 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-24T01:24:19.593 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:19.593 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2024-03-24T01:24:19.593 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-24T01:24:19.593 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link add name ceph-brx type bridge 2024-03-24T01:24:19.593 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip addr flush dev ceph-brx 2024-03-24T01:24:19.593 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set ceph-brx up 2024-03-24T01:24:19.593 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-24T01:24:19.593 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-24T01:24:20.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:20.050 DEBUG:teuthology.orchestra.run.smithi090:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T01:24:20.132 INFO:teuthology.orchestra.run.smithi090.stdout:1 2024-03-24T01:24:20.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:20.136 DEBUG:teuthology.orchestra.run.smithi090:> route 2024-03-24T01:24:20.207 INFO:teuthology.orchestra.run.smithi090.stdout:Kernel IP routing table 2024-03-24T01:24:20.207 INFO:teuthology.orchestra.run.smithi090.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T01:24:20.207 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T01:24:20.207 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T01:24:20.207 INFO:teuthology.orchestra.run.smithi090.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T01:24:20.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:20.208 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2024-03-24T01:24:20.208 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-24T01:24:20.208 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T01:24:20.208 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T01:24:20.208 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T01:24:20.208 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-24T01:24:20.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:20.955 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-03-24T01:24:20.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:20.998 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list-id 2024-03-24T01:24:21.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:21.089 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2024-03-24T01:24:21.089 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-24T01:24:21.089 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-24T01:24:21.089 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-24T01:24:21.090 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-24T01:24:21.220 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-24T01:24:21.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:21.220 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2024-03-24T01:24:21.220 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-24T01:24:21.220 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-24T01:24:21.221 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-24T01:24:21.221 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-24T01:24:21.221 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-24T01:24:21.221 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-24T01:24:21.221 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-24T01:24:21.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:21.467 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2024-03-24T01:24:21.467 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-24T01:24:21.467 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set brx.0 up 2024-03-24T01:24:21.468 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set dev brx.0 master ceph-brx 2024-03-24T01:24:21.468 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-24T01:24:21.613 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-24T01:24:21.613 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T01:24:21.613 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-24T01:24:21.650 INFO:teuthology.orchestra.run.smithi090.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-24T01:24:21.651 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T01:24:21.651 DEBUG:teuthology.orchestra.run.smithi090:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-24T01:24:21.739 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-24T01:24:21.868 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T01:24:21.916 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-24T01:24:21.916 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-24T01:24:21.916 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-24T01:24:21.916 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-24T01:24:21.916 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-24T01:24:21.916 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-24T01:24:21.916 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-24T01:24:21.916 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-24T01:24:21.916 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-24T01:24:21.916 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-24T01:24:21.917 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-24T01:24:21.917 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/selinux 2024-03-24T01:24:21.917 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T01:24:21.917 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-24T01:24:21.917 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-24T01:24:21.917 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-24T01:24:21.917 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-24T01:24:21.917 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T01:24:21.917 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-24T01:24:21.917 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T01:24:21.917 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-24T01:24:21.917 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T01:24:21.917 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T01:24:21.917 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-24T01:24:21.918 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-24T01:24:21.918 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T01:24:21.918 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T01:24:21.918 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T01:24:21.918 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T01:24:21.918 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-24T01:24:21.918 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-24T01:24:21.918 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-24T01:24:21.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:21.919 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-24T01:24:21.960 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-24T01:24:21.961 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-24T01:24:22.008 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-24T01:24:22.083 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T01:24:22.122 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-24T01:24:22.122 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-24T01:24:22.122 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-24T01:24:22.122 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-24T01:24:22.122 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-24T01:24:22.123 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-24T01:24:22.123 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-24T01:24:22.123 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-24T01:24:22.123 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-24T01:24:22.123 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-24T01:24:22.123 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-24T01:24:22.123 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/selinux 2024-03-24T01:24:22.123 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T01:24:22.123 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-24T01:24:22.123 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-24T01:24:22.123 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-24T01:24:22.123 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-24T01:24:22.123 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T01:24:22.123 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-24T01:24:22.124 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T01:24:22.124 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-24T01:24:22.124 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T01:24:22.124 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T01:24:22.124 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-24T01:24:22.124 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-24T01:24:22.124 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T01:24:22.124 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T01:24:22.124 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T01:24:22.124 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T01:24:22.124 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-24T01:24:22.124 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-24T01:24:22.124 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-24T01:24:22.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:22.125 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-24T01:24:22.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi090.stderr:2024-03-24T01:24:22.195+0000 7fdc32ab5480 -1 init, newargv = 0x55a747b52430 newargc=15 2024-03-24T01:24:22.197 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi090.stderr:ceph-fuse[43766]: starting ceph client 2024-03-24T01:24:22.209 INFO:teuthology.orchestra.run.smithi090.stdout:40 2024-03-24T01:24:22.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi090.stderr:ceph-fuse[43766]: starting fuse 2024-03-24T01:24:22.210 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-03-24T01:24:22.210 DEBUG:teuthology.orchestra.run.smithi090:> sudo stdin-killer -- python3 -c ' 2024-03-24T01:24:22.210 DEBUG:teuthology.orchestra.run.smithi090:> import glob 2024-03-24T01:24:22.210 DEBUG:teuthology.orchestra.run.smithi090:> import re 2024-03-24T01:24:22.210 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-24T01:24:22.210 DEBUG:teuthology.orchestra.run.smithi090:> import subprocess 2024-03-24T01:24:22.210 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:24:22.211 DEBUG:teuthology.orchestra.run.smithi090:> def _find_admin_socket(client_name): 2024-03-24T01:24:22.211 DEBUG:teuthology.orchestra.run.smithi090:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-24T01:24:22.211 DEBUG:teuthology.orchestra.run.smithi090:> files = glob.glob(asok_path) 2024-03-24T01:24:22.211 DEBUG:teuthology.orchestra.run.smithi090:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-24T01:24:22.211 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:24:22.211 DEBUG:teuthology.orchestra.run.smithi090:> # Given a non-glob path, it better be there 2024-03-24T01:24:22.211 DEBUG:teuthology.orchestra.run.smithi090:> if "*" not in asok_path: 2024-03-24T01:24:22.211 DEBUG:teuthology.orchestra.run.smithi090:> assert(len(files) == 1) 2024-03-24T01:24:22.211 DEBUG:teuthology.orchestra.run.smithi090:> return files[0] 2024-03-24T01:24:22.211 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:24:22.211 DEBUG:teuthology.orchestra.run.smithi090:> for f in files: 2024-03-24T01:24:22.211 DEBUG:teuthology.orchestra.run.smithi090:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T01:24:22.211 DEBUG:teuthology.orchestra.run.smithi090:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T01:24:22.211 DEBUG:teuthology.orchestra.run.smithi090:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T01:24:22.211 DEBUG:teuthology.orchestra.run.smithi090:> contents = proc_f.read() 2024-03-24T01:24:22.211 DEBUG:teuthology.orchestra.run.smithi090:> if mountpoint in contents: 2024-03-24T01:24:22.212 DEBUG:teuthology.orchestra.run.smithi090:> return f 2024-03-24T01:24:22.212 DEBUG:teuthology.orchestra.run.smithi090:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T01:24:22.212 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:24:22.212 DEBUG:teuthology.orchestra.run.smithi090:> print(_find_admin_socket("client.admin")) 2024-03-24T01:24:22.212 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-24T01:24:22.390 INFO:teuthology.orchestra.run.smithi090.stdout:/var/run/ceph/ceph-client.admin.43766.asok 2024-03-24T01:24:22.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T01:24:22.405 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.43766.asok 2024-03-24T01:24:22.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:22.406 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.43766.asok status 2024-03-24T01:24:22.621 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-24T01:24:22.621 INFO:teuthology.orchestra.run.smithi090.stdout: "metadata": { 2024-03-24T01:24:22.621 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_sha1": "3be9a5c9bc793e11e2800a8c0c696e8b46742033", 2024-03-24T01:24:22.621 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-24T01:24:22.621 INFO:teuthology.orchestra.run.smithi090.stdout: "entity_id": "admin", 2024-03-24T01:24:22.622 INFO:teuthology.orchestra.run.smithi090.stdout: "hostname": "smithi090", 2024-03-24T01:24:22.622 INFO:teuthology.orchestra.run.smithi090.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-24T01:24:22.622 INFO:teuthology.orchestra.run.smithi090.stdout: "pid": "43766", 2024-03-24T01:24:22.622 INFO:teuthology.orchestra.run.smithi090.stdout: "root": "/" 2024-03-24T01:24:22.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:24:22.622 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_count": 0, 2024-03-24T01:24:22.622 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_pinned_count": 0, 2024-03-24T01:24:22.622 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 4404, 2024-03-24T01:24:22.622 INFO:teuthology.orchestra.run.smithi090.stdout: "inst": { 2024-03-24T01:24:22.622 INFO:teuthology.orchestra.run.smithi090.stdout: "name": { 2024-03-24T01:24:22.622 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "client", 2024-03-24T01:24:22.622 INFO:teuthology.orchestra.run.smithi090.stdout: "num": 4404 2024-03-24T01:24:22.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:24:22.622 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-24T01:24:22.623 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-24T01:24:22.623 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-03-24T01:24:22.623 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3993017403 2024-03-24T01:24:22.623 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:24:22.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:24:22.623 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-24T01:24:22.623 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-24T01:24:22.623 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-03-24T01:24:22.623 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3993017403 2024-03-24T01:24:22.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:24:22.623 INFO:teuthology.orchestra.run.smithi090.stdout: "inst_str": "client.4404 192.168.0.1:0/3993017403", 2024-03-24T01:24:22.623 INFO:teuthology.orchestra.run.smithi090.stdout: "addr_str": "192.168.0.1:0/3993017403", 2024-03-24T01:24:22.623 INFO:teuthology.orchestra.run.smithi090.stdout: "inode_count": 1, 2024-03-24T01:24:22.623 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_epoch": 6, 2024-03-24T01:24:22.623 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch": 25, 2024-03-24T01:24:22.623 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch_barrier": 0, 2024-03-24T01:24:22.624 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklisted": false, 2024-03-24T01:24:22.624 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs" 2024-03-24T01:24:22.624 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-24T01:24:22.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:22.635 DEBUG:teuthology.orchestra.run.smithi090:> stat --file-system '--printf=%T 2024-03-24T01:24:22.635 DEBUG:teuthology.orchestra.run.smithi090:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-24T01:24:22.684 INFO:teuthology.orchestra.run.smithi090.stdout:fuseblk 2024-03-24T01:24:22.684 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-24T01:24:22.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:22.685 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-24T01:24:22.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:22.775 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-24T01:24:22.870 INFO:teuthology.orchestra.run.smithi090.stdout:# file: . 2024-03-24T01:24:22.871 INFO:teuthology.orchestra.run.smithi090.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-24T01:24:22.871 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:24:22.872 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:24:22.872 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:24:22.925 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:24:23.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.133+0000 7f6613c00640 1 Processor -- start 2024-03-24T01:24:23.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.133+0000 7f6613c00640 1 -- start start 2024-03-24T01:24:23.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.134+0000 7f6613c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f660c0b3310 0x7f660c1658a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:23.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.134+0000 7f6613c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f660c1447b0 con 0x7f660c0cd5c0 2024-03-24T01:24:23.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.134+0000 7f6613c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f660c15b110 con 0x7f660c1443d0 2024-03-24T01:24:23.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.134+0000 7f6613c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f660c15aef0 con 0x7f660c0b3310 2024-03-24T01:24:23.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.135+0000 7f660be00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f660c0cd5c0 0x7f660c0cec90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:36274/0 (socket says 172.21.15.90:36274) 2024-03-24T01:24:23.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.135+0000 7f660be00640 1 -- 172.21.15.90:0/3623797815 learned_addr learned my addr 172.21.15.90:0/3623797815 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:23.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.135+0000 7f6611c00640 1 --2- 172.21.15.90:0/3623797815 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f660c0b3310 0x7f660c1658a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:23.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.135+0000 7f660b400640 1 -- 172.21.15.90:0/3623797815 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2643319850 0 0) 0x7f660c1447b0 con 0x7f660c0cd5c0 2024-03-24T01:24:23.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.135+0000 7f660b400640 1 -- 172.21.15.90:0/3623797815 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65f0003520 con 0x7f660c0cd5c0 2024-03-24T01:24:23.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.135+0000 7f660b400640 1 -- 172.21.15.90:0/3623797815 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1374247247 0 0) 0x7f65f0003520 con 0x7f660c0cd5c0 2024-03-24T01:24:23.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.135+0000 7f660b400640 1 -- 172.21.15.90:0/3623797815 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f660c0b3310 msgr2=0x7f660c1658a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:23.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.135+0000 7f660b400640 1 --2- 172.21.15.90:0/3623797815 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f660c0b3310 0x7f660c1658a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:23.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.136+0000 7f660b400640 1 -- 172.21.15.90:0/3623797815 >> v1:172.21.15.120:6789/0 conn(0x7f660c1443d0 legacy=0x7f660c0cceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:23.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.136+0000 7f660b400640 1 -- 172.21.15.90:0/3623797815 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f660c15b3f0 con 0x7f660c0cd5c0 2024-03-24T01:24:23.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.136+0000 7f660b400640 1 -- 172.21.15.90:0/3623797815 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f65f8001670 con 0x7f660c0cd5c0 2024-03-24T01:24:23.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.136+0000 7f660b400640 1 -- 172.21.15.90:0/3623797815 >> v1:172.21.15.90:6789/0 conn(0x7f660c0cd5c0 legacy=0x7f660c0cec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:23.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.136+0000 7f660b400640 1 --2- 172.21.15.90:0/3623797815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65f00039e0 0x7f65f0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:23.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.136+0000 7f660b400640 1 --2- 172.21.15.90:0/3623797815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65f0004800 0x7f65f0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:23.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.136+0000 7f660b400640 1 --2- 172.21.15.90:0/3623797815 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f660c0b3310 0x7f65f0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:23.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.136+0000 7f660b400640 1 -- 172.21.15.90:0/3623797815 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f660c15aef0 con 0x7f65f0004800 2024-03-24T01:24:23.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.136+0000 7f660b400640 1 -- 172.21.15.90:0/3623797815 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f000baf0 con 0x7f65f00039e0 2024-03-24T01:24:23.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.136+0000 7f660b400640 1 -- 172.21.15.90:0/3623797815 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65f000df00 con 0x7f660c0b3310 2024-03-24T01:24:23.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.136+0000 7f660be00640 1 --2- 172.21.15.90:0/3623797815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65f0004800 0x7f65f0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:23.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.136+0000 7f6611200640 1 --2- 172.21.15.90:0/3623797815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65f00039e0 0x7f65f0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:23.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.136+0000 7f660be00640 1 -- 172.21.15.90:0/3623797815 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f660c0b3310 msgr2=0x7f65f0005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:23.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.136+0000 7f660be00640 1 --2- 172.21.15.90:0/3623797815 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f660c0b3310 0x7f65f0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:23.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.136+0000 7f660be00640 1 -- 172.21.15.90:0/3623797815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65f00039e0 msgr2=0x7f65f0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:23.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.136+0000 7f660be00640 1 --2- 172.21.15.90:0/3623797815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65f00039e0 0x7f65f0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:23.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.136+0000 7f660be00640 1 -- 172.21.15.90:0/3623797815 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f0003520 con 0x7f65f0004800 2024-03-24T01:24:23.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.137+0000 7f660be00640 1 --2- 172.21.15.90:0/3623797815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65f0004800 0x7f65f0004c80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f660c072f10 tx=0x7f65f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a2d1440a2657a4b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:23.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.138+0000 7f660b400640 1 -- 172.21.15.90:0/3623797815 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f8007d30 con 0x7f65f0004800 2024-03-24T01:24:23.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.138+0000 7f660b400640 1 -- 172.21.15.90:0/3623797815 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65f8030070 con 0x7f65f0004800 2024-03-24T01:24:23.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.138+0000 7f660b400640 1 -- 172.21.15.90:0/3623797815 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f8005d30 con 0x7f65f0004800 2024-03-24T01:24:23.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.138+0000 7f6613c00640 1 -- 172.21.15.90:0/3623797815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65f0004800 msgr2=0x7f65f0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:23.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.138+0000 7f6613c00640 1 --2- 172.21.15.90:0/3623797815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65f0004800 0x7f65f0004c80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f660c072f10 tx=0x7f65f0005830 comp rx=0 tx=0).stop 2024-03-24T01:24:23.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.138+0000 7f6611c00640 1 -- 172.21.15.90:0/3623797815 reap_dead start 2024-03-24T01:24:23.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.138+0000 7f6613c00640 1 -- 172.21.15.90:0/3623797815 shutdown_connections 2024-03-24T01:24:23.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.138+0000 7f6613c00640 1 -- 172.21.15.90:0/3623797815 >> 172.21.15.90:0/3623797815 conn(0x7f660c06b2a0 msgr2=0x7f660c06b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:23.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.138+0000 7f6613c00640 1 -- 172.21.15.90:0/3623797815 shutdown_connections 2024-03-24T01:24:23.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.138+0000 7f6613c00640 1 -- 172.21.15.90:0/3623797815 wait complete. 2024-03-24T01:24:23.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.140+0000 7f6613c00640 1 Processor -- start 2024-03-24T01:24:23.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.140+0000 7f6613c00640 1 -- start start 2024-03-24T01:24:23.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.141+0000 7f6613c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65f0004800 0x7f660c1ef620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:23.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.141+0000 7f6613c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f660c1f2330 0x7f660c13e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:23.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.141+0000 7f6613c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f660c13e960 0x7f660c15d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:23.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.141+0000 7f6613c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f660c15bc90 con 0x7f660c13e960 2024-03-24T01:24:23.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.141+0000 7f6613c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f660c15b9b0 con 0x7f65f0004800 2024-03-24T01:24:23.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.141+0000 7f6613c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f660c0cf3a0 con 0x7f660c1f2330 2024-03-24T01:24:23.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.142+0000 7f6611c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f660c13e960 0x7f660c15d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:23.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.142+0000 7f6611c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f660c13e960 0x7f660c15d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40182/0 (socket says 172.21.15.90:40182) 2024-03-24T01:24:23.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.142+0000 7f6611c00640 1 -- 172.21.15.90:0/656950576 learned_addr learned my addr 172.21.15.90:0/656950576 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:23.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.142+0000 7f6611200640 1 --2- 172.21.15.90:0/656950576 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65f0004800 0x7f660c1ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:23.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.142+0000 7f660be00640 1 --2- 172.21.15.90:0/656950576 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f660c1f2330 0x7f660c13e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:23.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.142+0000 7f6611c00640 1 -- 172.21.15.90:0/656950576 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f660c1f2330 msgr2=0x7f660c13e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:23.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.142+0000 7f6611c00640 1 --2- 172.21.15.90:0/656950576 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f660c1f2330 0x7f660c13e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:23.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.142+0000 7f6611c00640 1 -- 172.21.15.90:0/656950576 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65f0004800 msgr2=0x7f660c1ef620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:23.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.142+0000 7f6611c00640 1 --2- 172.21.15.90:0/656950576 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65f0004800 0x7f660c1ef620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:23.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.142+0000 7f6611c00640 1 -- 172.21.15.90:0/656950576 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f660c13a0e0 con 0x7f660c13e960 2024-03-24T01:24:23.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.142+0000 7f6611c00640 1 --2- 172.21.15.90:0/656950576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f660c13e960 0x7f660c15d0e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f660c0b2de0 tx=0x7f660400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:23.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.143+0000 7f6608c00640 1 -- 172.21.15.90:0/656950576 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6604018020 con 0x7f660c13e960 2024-03-24T01:24:23.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.143+0000 7f6608c00640 1 -- 172.21.15.90:0/656950576 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f660400bce0 con 0x7f660c13e960 2024-03-24T01:24:23.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.143+0000 7f6613c00640 1 -- 172.21.15.90:0/656950576 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f660c13a380 con 0x7f660c13e960 2024-03-24T01:24:23.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.143+0000 7f6608c00640 1 -- 172.21.15.90:0/656950576 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6604014640 con 0x7f660c13e960 2024-03-24T01:24:23.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.143+0000 7f6613c00640 1 -- 172.21.15.90:0/656950576 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f660c13a940 con 0x7f660c13e960 2024-03-24T01:24:23.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.144+0000 7f6613c00640 1 -- 172.21.15.90:0/656950576 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f660c073570 con 0x7f660c13e960 2024-03-24T01:24:23.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.145+0000 7f6608c00640 1 -- 172.21.15.90:0/656950576 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6604005d10 con 0x7f660c13e960 2024-03-24T01:24:23.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.146+0000 7f6608c00640 1 --2- 172.21.15.90:0/656950576 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f65ec06bc00 0x7f65ec06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:23.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.146+0000 7f6608c00640 1 -- 172.21.15.90:0/656950576 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f660408b520 con 0x7f660c13e960 2024-03-24T01:24:23.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.147+0000 7f6611200640 1 --2- 172.21.15.90:0/656950576 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f65ec06bc00 0x7f65ec06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:23.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.148+0000 7f6611200640 1 --2- 172.21.15.90:0/656950576 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f65ec06bc00 0x7f65ec06e0c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f65fc005490 tx=0x7f65fc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:23.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.151+0000 7f6608c00640 1 -- 172.21.15.90:0/656950576 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6604022030 con 0x7f660c13e960 2024-03-24T01:24:23.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.351+0000 7f6613c00640 1 -- 172.21.15.90:0/656950576 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f660c04f4e0 con 0x7f660c13e960 2024-03-24T01:24:23.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.352+0000 7f6608c00640 1 -- 172.21.15.90:0/656950576 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18490 (secure 0 0 0) 0x7f6604059d60 con 0x7f660c13e960 2024-03-24T01:24:23.354 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:24:23.354 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:24:17.008224+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:24:08.643966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T01:24:09.189904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T01:24:13.027323+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:24:23.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.354+0000 7f6613c00640 1 -- 172.21.15.90:0/656950576 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f65ec06bc00 msgr2=0x7f65ec06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:23.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.355+0000 7f6613c00640 1 --2- 172.21.15.90:0/656950576 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f65ec06bc00 0x7f65ec06e0c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f65fc005490 tx=0x7f65fc009040 comp rx=0 tx=0).stop 2024-03-24T01:24:23.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.355+0000 7f6613c00640 1 -- 172.21.15.90:0/656950576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f660c13e960 msgr2=0x7f660c15d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:23.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.355+0000 7f6613c00640 1 --2- 172.21.15.90:0/656950576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f660c13e960 0x7f660c15d0e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f660c0b2de0 tx=0x7f660400b480 comp rx=0 tx=0).stop 2024-03-24T01:24:23.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.355+0000 7f6613c00640 1 -- 172.21.15.90:0/656950576 shutdown_connections 2024-03-24T01:24:23.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.355+0000 7f6613c00640 1 --2- 172.21.15.90:0/656950576 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f65ec06bc00 0x7f65ec06e0c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:23.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.355+0000 7f6613c00640 1 --2- 172.21.15.90:0/656950576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f660c13e960 0x7f660c15d0e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:23.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.355+0000 7f6613c00640 1 --2- 172.21.15.90:0/656950576 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f660c1f2330 0x7f660c13e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:23.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.355+0000 7f6613c00640 1 --2- 172.21.15.90:0/656950576 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65f0004800 0x7f660c1ef620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:23.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.355+0000 7f6613c00640 1 -- 172.21.15.90:0/656950576 >> 172.21.15.90:0/656950576 conn(0x7f660c06b2a0 msgr2=0x7f660c06b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:23.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.355+0000 7f6613c00640 1 -- 172.21.15.90:0/656950576 shutdown_connections 2024-03-24T01:24:23.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.355+0000 7f6613c00640 1 -- 172.21.15.90:0/656950576 wait complete. 2024-03-24T01:24:23.374 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T01:24:08.643966+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T01:24:09.189904+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T01:24:13.027323+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T01:24:23.375 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:24:23.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.581+0000 7f2868a00640 1 Processor -- start 2024-03-24T01:24:23.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.581+0000 7f2868a00640 1 -- start start 2024-03-24T01:24:23.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.583+0000 7f2868a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f286406b7d0 0x7f28640cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:23.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.583+0000 7f2868a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f28640cef80 0x7f28640ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:23.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.583+0000 7f2868a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f286406bca0 con 0x7f2864159620 2024-03-24T01:24:23.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.583+0000 7f2868a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28640b4470 con 0x7f28640cef80 2024-03-24T01:24:23.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.583+0000 7f2868a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28640b36f0 con 0x7f286406b7d0 2024-03-24T01:24:23.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.584+0000 7f2861400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f286406b7d0 0x7f28640cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:23.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.584+0000 7f2861400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f286406b7d0 0x7f28640cea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.90:51746/0 (socket says 172.21.15.90:51746) 2024-03-24T01:24:23.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.584+0000 7f2861400640 1 -- 172.21.15.90:0/1198822548 learned_addr learned my addr 172.21.15.90:0/1198822548 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:23.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.584+0000 7f2862800640 1 --2- 172.21.15.90:0/1198822548 >> v2:172.21.15.120:3300/0 conn(0x7f28640cef80 0x7f28640ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:23.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.584+0000 7f2861400640 1 -- 172.21.15.90:0/1198822548 >> v2:172.21.15.120:3300/0 conn(0x7f28640cef80 msgr2=0x7f28640ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:23.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.584+0000 7f2861400640 1 --2- 172.21.15.90:0/1198822548 >> v2:172.21.15.120:3300/0 conn(0x7f28640cef80 0x7f28640ae530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:23.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.584+0000 7f2861400640 1 -- 172.21.15.90:0/1198822548 >> v1:172.21.15.120:6789/0 conn(0x7f2864159620 legacy=0x7f286406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:23.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.584+0000 7f2861400640 1 -- 172.21.15.90:0/1198822548 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28640b3990 con 0x7f286406b7d0 2024-03-24T01:24:23.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.584+0000 7f2860a00640 1 -- 172.21.15.90:0/1198822548 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1673913760 0 0) 0x7f286406bca0 con 0x7f2864159620 2024-03-24T01:24:23.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.586+0000 7f2861400640 1 --2- 172.21.15.90:0/1198822548 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f286406b7d0 0x7f28640cea40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f284c009870 tx=0x7f284c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3079f70a6cd6cd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:23.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.587+0000 7f2860a00640 1 -- 172.21.15.90:0/1198822548 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284c035020 con 0x7f286406b7d0 2024-03-24T01:24:23.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.587+0000 7f2860a00640 1 -- 172.21.15.90:0/1198822548 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f284c031640 con 0x7f286406b7d0 2024-03-24T01:24:23.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.587+0000 7f2860a00640 1 -- 172.21.15.90:0/1198822548 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284c02d040 con 0x7f286406b7d0 2024-03-24T01:24:23.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.588+0000 7f2868a00640 1 -- 172.21.15.90:0/1198822548 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f286406b7d0 msgr2=0x7f28640cea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:23.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.588+0000 7f2868a00640 1 --2- 172.21.15.90:0/1198822548 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f286406b7d0 0x7f28640cea40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f284c009870 tx=0x7f284c02b8f0 comp rx=0 tx=0).stop 2024-03-24T01:24:23.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.588+0000 7f2868a00640 1 -- 172.21.15.90:0/1198822548 shutdown_connections 2024-03-24T01:24:23.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.588+0000 7f2868a00640 1 --2- 172.21.15.90:0/1198822548 >> v2:172.21.15.120:3300/0 conn(0x7f28640cef80 0x7f28640ae530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:23.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.588+0000 7f2868a00640 1 --2- 172.21.15.90:0/1198822548 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f286406b7d0 0x7f28640cea40 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:23.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.588+0000 7f2868a00640 1 -- 172.21.15.90:0/1198822548 >> 172.21.15.90:0/1198822548 conn(0x7f28640b1770 msgr2=0x7f28640b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:23.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.588+0000 7f2868a00640 1 -- 172.21.15.90:0/1198822548 shutdown_connections 2024-03-24T01:24:23.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.588+0000 7f2868a00640 1 -- 172.21.15.90:0/1198822548 wait complete. 2024-03-24T01:24:23.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.590+0000 7f2868a00640 1 Processor -- start 2024-03-24T01:24:23.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.590+0000 7f2868a00640 1 -- start start 2024-03-24T01:24:23.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.591+0000 7f2868a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f286406b7d0 0x7f286414b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:23.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.591+0000 7f2868a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28640cef80 0x7f28641411b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:23.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.591+0000 7f2868a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2864159620 0x7f286414eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:23.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.591+0000 7f2868a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2864159a00 con 0x7f286406b7d0 2024-03-24T01:24:23.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.591+0000 7f2868a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28640b3f50 con 0x7f28640cef80 2024-03-24T01:24:23.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.591+0000 7f2868a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28640cf3f0 con 0x7f2864159620 2024-03-24T01:24:23.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.591+0000 7f2861e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f286406b7d0 0x7f286414b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:23.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.591+0000 7f2861400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28640cef80 0x7f28641411b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:23.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.592+0000 7f2862800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2864159620 0x7f286414eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:23.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.591+0000 7f2861400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28640cef80 0x7f28641411b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.90:46178/0 (socket says 172.21.15.90:46178) 2024-03-24T01:24:23.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.592+0000 7f2861400640 1 -- 172.21.15.90:0/2796304724 learned_addr learned my addr 172.21.15.90:0/2796304724 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:23.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.592+0000 7f2861e00640 1 -- 172.21.15.90:0/2796304724 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2864159620 msgr2=0x7f286414eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:23.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.592+0000 7f2861e00640 1 --2- 172.21.15.90:0/2796304724 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2864159620 0x7f286414eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:23.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.592+0000 7f2861e00640 1 -- 172.21.15.90:0/2796304724 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28640cef80 msgr2=0x7f28641411b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:23.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.592+0000 7f2861e00640 1 --2- 172.21.15.90:0/2796304724 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28640cef80 0x7f28641411b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:23.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.592+0000 7f2861e00640 1 -- 172.21.15.90:0/2796304724 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2864152f60 con 0x7f286406b7d0 2024-03-24T01:24:23.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.592+0000 7f2861e00640 1 --2- 172.21.15.90:0/2796304724 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f286406b7d0 0x7f286414b3e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f285400c450 tx=0x7f285400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:23.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.593+0000 7f285a000640 1 -- 172.21.15.90:0/2796304724 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2854017070 con 0x7f286406b7d0 2024-03-24T01:24:23.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.593+0000 7f285a000640 1 -- 172.21.15.90:0/2796304724 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28540035f0 con 0x7f286406b7d0 2024-03-24T01:24:23.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.593+0000 7f285a000640 1 -- 172.21.15.90:0/2796304724 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28540039e0 con 0x7f286406b7d0 2024-03-24T01:24:23.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.593+0000 7f2868a00640 1 -- 172.21.15.90:0/2796304724 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2864153240 con 0x7f286406b7d0 2024-03-24T01:24:23.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.593+0000 7f2868a00640 1 -- 172.21.15.90:0/2796304724 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2864153800 con 0x7f286406b7d0 2024-03-24T01:24:23.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.594+0000 7f2868a00640 1 -- 172.21.15.90:0/2796304724 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f286404f400 con 0x7f286406b7d0 2024-03-24T01:24:23.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.595+0000 7f285a000640 1 -- 172.21.15.90:0/2796304724 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f285401a070 con 0x7f286406b7d0 2024-03-24T01:24:23.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.596+0000 7f285a000640 1 --2- 172.21.15.90:0/2796304724 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f284006bc00 0x7f284006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:23.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.596+0000 7f285a000640 1 -- 172.21.15.90:0/2796304724 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f285408a0f0 con 0x7f286406b7d0 2024-03-24T01:24:23.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.596+0000 7f2861400640 1 --2- 172.21.15.90:0/2796304724 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f284006bc00 0x7f284006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:23.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.597+0000 7f2861400640 1 --2- 172.21.15.90:0/2796304724 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f284006bc00 0x7f284006e0c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f284c02b930 tx=0x7f284c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:23.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.600+0000 7f285a000640 1 -- 172.21.15.90:0/2796304724 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2854058240 con 0x7f286406b7d0 2024-03-24T01:24:23.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.801+0000 7f2868a00640 1 -- 172.21.15.90:0/2796304724 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f28641539d0 con 0x7f286406b7d0 2024-03-24T01:24:23.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.802+0000 7f285a000640 1 -- 172.21.15.90:0/2796304724 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f285405c2e0 con 0x7f286406b7d0 2024-03-24T01:24:23.804 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:24:23.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.805+0000 7f2868a00640 1 -- 172.21.15.90:0/2796304724 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f284006bc00 msgr2=0x7f284006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:23.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.805+0000 7f2868a00640 1 --2- 172.21.15.90:0/2796304724 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f284006bc00 0x7f284006e0c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f284c02b930 tx=0x7f284c03c040 comp rx=0 tx=0).stop 2024-03-24T01:24:23.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.805+0000 7f2868a00640 1 -- 172.21.15.90:0/2796304724 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f286406b7d0 msgr2=0x7f286414b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:23.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.805+0000 7f2868a00640 1 --2- 172.21.15.90:0/2796304724 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f286406b7d0 0x7f286414b3e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f285400c450 tx=0x7f285400c910 comp rx=0 tx=0).stop 2024-03-24T01:24:23.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.805+0000 7f2868a00640 1 -- 172.21.15.90:0/2796304724 shutdown_connections 2024-03-24T01:24:23.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.805+0000 7f2868a00640 1 --2- 172.21.15.90:0/2796304724 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f284006bc00 0x7f284006e0c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:23.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.805+0000 7f2868a00640 1 --2- 172.21.15.90:0/2796304724 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2864159620 0x7f286414eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:23.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.805+0000 7f2868a00640 1 --2- 172.21.15.90:0/2796304724 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28640cef80 0x7f28641411b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:23.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.805+0000 7f2868a00640 1 --2- 172.21.15.90:0/2796304724 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f286406b7d0 0x7f286414b3e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:23.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.805+0000 7f2868a00640 1 -- 172.21.15.90:0/2796304724 >> 172.21.15.90:0/2796304724 conn(0x7f28640b1770 msgr2=0x7f28640cc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:23.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.805+0000 7f2868a00640 1 -- 172.21.15.90:0/2796304724 shutdown_connections 2024-03-24T01:24:23.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:23.805+0000 7f2868a00640 1 -- 172.21.15.90:0/2796304724 wait complete. 2024-03-24T01:24:23.825 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T01:24:24.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.033+0000 7f1c27000640 1 Processor -- start 2024-03-24T01:24:24.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.034+0000 7f1c27000640 1 -- start start 2024-03-24T01:24:24.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.035+0000 7f1c27000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1c20159620 0x7f1c2006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:24.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.035+0000 7f1c27000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c2006bca0 con 0x7f1c200cf150 2024-03-24T01:24:24.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.035+0000 7f1c27000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c200b36f0 con 0x7f1c2006b600 2024-03-24T01:24:24.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.035+0000 7f1c27000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c200b4470 con 0x7f1c20159620 2024-03-24T01:24:24.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.036+0000 7f1c25000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f1c200cf150 0x7f1c200ae530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:36282/0 (socket says 172.21.15.90:36282) 2024-03-24T01:24:24.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.036+0000 7f1c25000640 1 -- 172.21.15.90:0/2169206769 learned_addr learned my addr 172.21.15.90:0/2169206769 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:24.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.036+0000 7f1c1fe00640 1 --2- 172.21.15.90:0/2169206769 >> v2:172.21.15.120:3300/0 conn(0x7f1c20159620 0x7f1c2006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:24.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.036+0000 7f1c1ea00640 1 -- 172.21.15.90:0/2169206769 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2807345319 0 0) 0x7f1c2006bca0 con 0x7f1c200cf150 2024-03-24T01:24:24.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.036+0000 7f1c1ea00640 1 -- 172.21.15.90:0/2169206769 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c04003520 con 0x7f1c200cf150 2024-03-24T01:24:24.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.037+0000 7f1c1ea00640 1 -- 172.21.15.90:0/2169206769 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3479400604 0 0) 0x7f1c04003520 con 0x7f1c200cf150 2024-03-24T01:24:24.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.037+0000 7f1c1ea00640 1 -- 172.21.15.90:0/2169206769 >> v2:172.21.15.120:3300/0 conn(0x7f1c20159620 msgr2=0x7f1c2006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:24.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.037+0000 7f1c1ea00640 1 --2- 172.21.15.90:0/2169206769 >> v2:172.21.15.120:3300/0 conn(0x7f1c20159620 0x7f1c2006b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:24.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.037+0000 7f1c1ea00640 1 -- 172.21.15.90:0/2169206769 >> v1:172.21.15.120:6789/0 conn(0x7f1c2006b600 legacy=0x7f1c200cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:24.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.037+0000 7f1c1ea00640 1 -- 172.21.15.90:0/2169206769 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c200b3990 con 0x7f1c200cf150 2024-03-24T01:24:24.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.037+0000 7f1c1ea00640 1 -- 172.21.15.90:0/2169206769 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f1c18001670 con 0x7f1c200cf150 2024-03-24T01:24:24.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.037+0000 7f1c1ea00640 1 -- 172.21.15.90:0/2169206769 >> v1:172.21.15.90:6789/0 conn(0x7f1c200cf150 legacy=0x7f1c200ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:24.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.037+0000 7f1c1ea00640 1 --2- 172.21.15.90:0/2169206769 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c040039b0 0x7f1c04003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:24.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.037+0000 7f1c1ea00640 1 --2- 172.21.15.90:0/2169206769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c040043b0 0x7f1c04004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:24.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.037+0000 7f1c1ea00640 1 --2- 172.21.15.90:0/2169206769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c04005120 0x7f1c0400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:24.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.037+0000 7f1c1ea00640 1 -- 172.21.15.90:0/2169206769 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c200b4470 con 0x7f1c040039b0 2024-03-24T01:24:24.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.037+0000 7f1c1ea00640 1 -- 172.21.15.90:0/2169206769 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0400c060 con 0x7f1c040043b0 2024-03-24T01:24:24.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.037+0000 7f1c1ea00640 1 -- 172.21.15.90:0/2169206769 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0400c260 con 0x7f1c04005120 2024-03-24T01:24:24.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.037+0000 7f1c1fe00640 1 --2- 172.21.15.90:0/2169206769 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c040039b0 0x7f1c04003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:24.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.037+0000 7f1c1f400640 1 --2- 172.21.15.90:0/2169206769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c040043b0 0x7f1c04004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:24.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.037+0000 7f1c25000640 1 --2- 172.21.15.90:0/2169206769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c04005120 0x7f1c0400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:24.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.038+0000 7f1c1fe00640 1 -- 172.21.15.90:0/2169206769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c04005120 msgr2=0x7f1c0400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:24.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.038+0000 7f1c1fe00640 1 --2- 172.21.15.90:0/2169206769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c04005120 0x7f1c0400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:24.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.038+0000 7f1c1fe00640 1 -- 172.21.15.90:0/2169206769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c040043b0 msgr2=0x7f1c04004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:24.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.038+0000 7f1c1fe00640 1 --2- 172.21.15.90:0/2169206769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c040043b0 0x7f1c04004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:24.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.038+0000 7f1c1fe00640 1 -- 172.21.15.90:0/2169206769 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c04003520 con 0x7f1c040039b0 2024-03-24T01:24:24.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.038+0000 7f1c25000640 1 --2- 172.21.15.90:0/2169206769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c04005120 0x7f1c0400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:24.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.038+0000 7f1c25000640 1 -- 172.21.15.90:0/2169206769 reap_dead start 2024-03-24T01:24:24.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.039+0000 7f1c1fe00640 1 --2- 172.21.15.90:0/2169206769 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c040039b0 0x7f1c04003d90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1c08009ac0 tx=0x7f1c04005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5b25ac82a06538f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:24.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.039+0000 7f1c1ea00640 1 -- 172.21.15.90:0/2169206769 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c08039070 con 0x7f1c040039b0 2024-03-24T01:24:24.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.039+0000 7f1c1ea00640 1 -- 172.21.15.90:0/2169206769 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c0802c6b0 con 0x7f1c040039b0 2024-03-24T01:24:24.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.039+0000 7f1c1ea00640 1 -- 172.21.15.90:0/2169206769 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c08009690 con 0x7f1c040039b0 2024-03-24T01:24:24.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.040+0000 7f1c27000640 1 -- 172.21.15.90:0/2169206769 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c040039b0 msgr2=0x7f1c04003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:24.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.040+0000 7f1c27000640 1 --2- 172.21.15.90:0/2169206769 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c040039b0 0x7f1c04003d90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1c08009ac0 tx=0x7f1c04005830 comp rx=0 tx=0).stop 2024-03-24T01:24:24.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.040+0000 7f1c27000640 1 -- 172.21.15.90:0/2169206769 shutdown_connections 2024-03-24T01:24:24.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.040+0000 7f1c27000640 1 --2- 172.21.15.90:0/2169206769 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c040039b0 0x7f1c04003d90 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:24.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.040+0000 7f1c27000640 1 -- 172.21.15.90:0/2169206769 >> 172.21.15.90:0/2169206769 conn(0x7f1c200b1770 msgr2=0x7f1c200b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:24.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.040+0000 7f1c27000640 1 -- 172.21.15.90:0/2169206769 shutdown_connections 2024-03-24T01:24:24.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.040+0000 7f1c27000640 1 -- 172.21.15.90:0/2169206769 wait complete. 2024-03-24T01:24:24.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.041+0000 7f1c27000640 1 Processor -- start 2024-03-24T01:24:24.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.042+0000 7f1c27000640 1 -- start start 2024-03-24T01:24:24.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.042+0000 7f1c27000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c040039b0 0x7f1c2015de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:24.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.043+0000 7f1c27000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c20154d50 0x7f1c20157870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:24.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.043+0000 7f1c27000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c2014e080 0x7f1c2014e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:24.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.043+0000 7f1c27000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c20159a00 con 0x7f1c040039b0 2024-03-24T01:24:24.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.043+0000 7f1c27000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c200b3f50 con 0x7f1c20154d50 2024-03-24T01:24:24.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.043+0000 7f1c27000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2015bfe0 con 0x7f1c2014e080 2024-03-24T01:24:24.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.043+0000 7f1c1fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c040039b0 0x7f1c2015de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:24.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.043+0000 7f1c1fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c040039b0 0x7f1c2015de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40218/0 (socket says 172.21.15.90:40218) 2024-03-24T01:24:24.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.043+0000 7f1c1fe00640 1 -- 172.21.15.90:0/186146826 learned_addr learned my addr 172.21.15.90:0/186146826 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:24.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.043+0000 7f1c25000640 1 --2- 172.21.15.90:0/186146826 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c2014e080 0x7f1c2014e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:24.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.043+0000 7f1c1fe00640 1 -- 172.21.15.90:0/186146826 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c2014e080 msgr2=0x7f1c2014e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:24.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.043+0000 7f1c1fe00640 1 --2- 172.21.15.90:0/186146826 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c2014e080 0x7f1c2014e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:24.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.043+0000 7f1c1fe00640 1 -- 172.21.15.90:0/186146826 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c20154d50 msgr2=0x7f1c20157870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:24.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.043+0000 7f1c1fe00640 1 --2- 172.21.15.90:0/186146826 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c20154d50 0x7f1c20157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:24.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.043+0000 7f1c1fe00640 1 -- 172.21.15.90:0/186146826 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c201f7030 con 0x7f1c040039b0 2024-03-24T01:24:24.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.044+0000 7f1c1fe00640 1 --2- 172.21.15.90:0/186146826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c040039b0 0x7f1c2015de60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1c08000c00 tx=0x7f1c08004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:24.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.044+0000 7f1c17e00640 1 -- 172.21.15.90:0/186146826 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c08042020 con 0x7f1c040039b0 2024-03-24T01:24:24.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.044+0000 7f1c17e00640 1 -- 172.21.15.90:0/186146826 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c0803a040 con 0x7f1c040039b0 2024-03-24T01:24:24.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.044+0000 7f1c17e00640 1 -- 172.21.15.90:0/186146826 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c08039070 con 0x7f1c040039b0 2024-03-24T01:24:24.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.044+0000 7f1c27000640 1 -- 172.21.15.90:0/186146826 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c201f72d0 con 0x7f1c040039b0 2024-03-24T01:24:24.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.044+0000 7f1c27000640 1 -- 172.21.15.90:0/186146826 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c201f7890 con 0x7f1c040039b0 2024-03-24T01:24:24.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.046+0000 7f1c27000640 1 -- 172.21.15.90:0/186146826 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c2015f010 con 0x7f1c040039b0 2024-03-24T01:24:24.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.047+0000 7f1c17e00640 1 -- 172.21.15.90:0/186146826 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1c08045020 con 0x7f1c040039b0 2024-03-24T01:24:24.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.047+0000 7f1c17e00640 1 --2- 172.21.15.90:0/186146826 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1bfc06bbd0 0x7f1bfc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:24.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.048+0000 7f1c17e00640 1 -- 172.21.15.90:0/186146826 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f1c080ac240 con 0x7f1c040039b0 2024-03-24T01:24:24.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.048+0000 7f1c1f400640 1 --2- 172.21.15.90:0/186146826 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1bfc06bbd0 0x7f1bfc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:24.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.049+0000 7f1c1f400640 1 --2- 172.21.15.90:0/186146826 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1bfc06bbd0 0x7f1bfc06e090 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1c10007fd0 tx=0x7f1c10009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:24.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.054+0000 7f1c17e00640 1 -- 172.21.15.90:0/186146826 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1c2015f010 con 0x7f1c040039b0 2024-03-24T01:24:24.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.250+0000 7f1c27000640 1 -- 172.21.15.90:0/186146826 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1c2004f400 con 0x7f1c040039b0 2024-03-24T01:24:24.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.250+0000 7f1c17e00640 1 -- 172.21.15.90:0/186146826 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1c0804c070 con 0x7f1c040039b0 2024-03-24T01:24:24.252 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:24:24.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.253+0000 7f1c27000640 1 -- 172.21.15.90:0/186146826 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1bfc06bbd0 msgr2=0x7f1bfc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:24.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.253+0000 7f1c27000640 1 --2- 172.21.15.90:0/186146826 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1bfc06bbd0 0x7f1bfc06e090 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1c10007fd0 tx=0x7f1c10009040 comp rx=0 tx=0).stop 2024-03-24T01:24:24.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.253+0000 7f1c27000640 1 -- 172.21.15.90:0/186146826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c040039b0 msgr2=0x7f1c2015de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:24.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.253+0000 7f1c27000640 1 --2- 172.21.15.90:0/186146826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c040039b0 0x7f1c2015de60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1c08000c00 tx=0x7f1c08004730 comp rx=0 tx=0).stop 2024-03-24T01:24:24.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.253+0000 7f1c27000640 1 -- 172.21.15.90:0/186146826 shutdown_connections 2024-03-24T01:24:24.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.253+0000 7f1c27000640 1 --2- 172.21.15.90:0/186146826 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1bfc06bbd0 0x7f1bfc06e090 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:24.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.253+0000 7f1c27000640 1 --2- 172.21.15.90:0/186146826 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c2014e080 0x7f1c2014e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:24.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.253+0000 7f1c27000640 1 --2- 172.21.15.90:0/186146826 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c20154d50 0x7f1c20157870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:24.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.253+0000 7f1c27000640 1 --2- 172.21.15.90:0/186146826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c040039b0 0x7f1c2015de60 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:24.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.253+0000 7f1c27000640 1 -- 172.21.15.90:0/186146826 >> 172.21.15.90:0/186146826 conn(0x7f1c200b1770 msgr2=0x7f1c200ccca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:24.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.253+0000 7f1c27000640 1 -- 172.21.15.90:0/186146826 shutdown_connections 2024-03-24T01:24:24.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.253+0000 7f1c27000640 1 -- 172.21.15.90:0/186146826 wait complete. 2024-03-24T01:24:24.272 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T01:24:24.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.480+0000 7f37eba00640 1 Processor -- start 2024-03-24T01:24:24.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.480+0000 7f37eba00640 1 -- start start 2024-03-24T01:24:24.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.480+0000 7f37eba00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37e40cf150 0x7f37e40ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:24.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.480+0000 7f37eba00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37e406bca0 con 0x7f37e4159620 2024-03-24T01:24:24.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.480+0000 7f37eba00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37e40b36f0 con 0x7f37e406b7d0 2024-03-24T01:24:24.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.480+0000 7f37eba00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37e40b4470 con 0x7f37e40cf150 2024-03-24T01:24:24.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.481+0000 7f37e9000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f37e4159620 0x7f37e406b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:36290/0 (socket says 172.21.15.90:36290) 2024-03-24T01:24:24.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.481+0000 7f37e9000640 1 -- 172.21.15.90:0/719277092 learned_addr learned my addr 172.21.15.90:0/719277092 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:24.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.481+0000 7f37e9a00640 1 --2- 172.21.15.90:0/719277092 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37e40cf150 0x7f37e40ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:24.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.481+0000 7f37e3400640 1 -- 172.21.15.90:0/719277092 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4037289629 0 0) 0x7f37e406bca0 con 0x7f37e4159620 2024-03-24T01:24:24.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.481+0000 7f37e3400640 1 -- 172.21.15.90:0/719277092 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37cc003520 con 0x7f37e4159620 2024-03-24T01:24:24.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.481+0000 7f37e3400640 1 -- 172.21.15.90:0/719277092 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2141632584 0 0) 0x7f37cc003520 con 0x7f37e4159620 2024-03-24T01:24:24.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.481+0000 7f37e3400640 1 -- 172.21.15.90:0/719277092 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37e40cf150 msgr2=0x7f37e40ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:24.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.481+0000 7f37e3400640 1 --2- 172.21.15.90:0/719277092 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37e40cf150 0x7f37e40ae530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:24.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.481+0000 7f37e3400640 1 -- 172.21.15.90:0/719277092 >> v1:172.21.15.120:6789/0 conn(0x7f37e406b7d0 legacy=0x7f37e40cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:24.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.481+0000 7f37e3400640 1 -- 172.21.15.90:0/719277092 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e40b3990 con 0x7f37e4159620 2024-03-24T01:24:24.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.481+0000 7f37e3400640 1 -- 172.21.15.90:0/719277092 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f37d8001670 con 0x7f37e4159620 2024-03-24T01:24:24.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.482+0000 7f37e3400640 1 -- 172.21.15.90:0/719277092 >> v1:172.21.15.90:6789/0 conn(0x7f37e4159620 legacy=0x7f37e406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:24.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.482+0000 7f37e3400640 1 --2- 172.21.15.90:0/719277092 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37cc0039e0 0x7f37cc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:24.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.482+0000 7f37e3400640 1 --2- 172.21.15.90:0/719277092 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37cc004800 0x7f37cc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:24.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.482+0000 7f37e3400640 1 --2- 172.21.15.90:0/719277092 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37e40cf150 0x7f37cc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:24.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.482+0000 7f37e3400640 1 -- 172.21.15.90:0/719277092 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e40b4470 con 0x7f37cc0039e0 2024-03-24T01:24:24.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.482+0000 7f37e3400640 1 -- 172.21.15.90:0/719277092 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc00ff00 con 0x7f37cc004800 2024-03-24T01:24:24.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.482+0000 7f37e3400640 1 -- 172.21.15.90:0/719277092 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc010100 con 0x7f37e40cf150 2024-03-24T01:24:24.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.482+0000 7f37e9000640 1 --2- 172.21.15.90:0/719277092 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37cc0039e0 0x7f37cc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:24.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.482+0000 7f37e9000640 1 -- 172.21.15.90:0/719277092 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37e40cf150 msgr2=0x7f37cc005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:24.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.482+0000 7f37e9a00640 1 --2- 172.21.15.90:0/719277092 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37e40cf150 0x7f37cc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:24.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.482+0000 7f37e3e00640 1 --2- 172.21.15.90:0/719277092 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37cc004800 0x7f37cc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:24.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.482+0000 7f37e9000640 1 --2- 172.21.15.90:0/719277092 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37e40cf150 0x7f37cc005200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:24.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.482+0000 7f37e9000640 1 -- 172.21.15.90:0/719277092 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37cc004800 msgr2=0x7f37cc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:24.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.482+0000 7f37e9000640 1 --2- 172.21.15.90:0/719277092 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37cc004800 0x7f37cc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:24.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.482+0000 7f37e9000640 1 -- 172.21.15.90:0/719277092 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37cc003520 con 0x7f37cc0039e0 2024-03-24T01:24:24.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.482+0000 7f37e3e00640 1 --2- 172.21.15.90:0/719277092 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37cc004800 0x7f37cc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:24.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.483+0000 7f37e9000640 1 --2- 172.21.15.90:0/719277092 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37cc0039e0 0x7f37cc003e00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f37d800c240 tx=0x7f37cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b23f11bc4c515a04 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:24.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.484+0000 7f37e3400640 1 -- 172.21.15.90:0/719277092 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d8007d10 con 0x7f37cc0039e0 2024-03-24T01:24:24.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.484+0000 7f37e3400640 1 -- 172.21.15.90:0/719277092 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37d8030070 con 0x7f37cc0039e0 2024-03-24T01:24:24.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.484+0000 7f37e3400640 1 -- 172.21.15.90:0/719277092 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d802cd60 con 0x7f37cc0039e0 2024-03-24T01:24:24.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.484+0000 7f37eba00640 1 -- 172.21.15.90:0/719277092 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37cc0039e0 msgr2=0x7f37cc003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:24.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.484+0000 7f37eba00640 1 --2- 172.21.15.90:0/719277092 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37cc0039e0 0x7f37cc003e00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f37d800c240 tx=0x7f37cc005830 comp rx=0 tx=0).stop 2024-03-24T01:24:24.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.484+0000 7f37e9a00640 1 -- 172.21.15.90:0/719277092 reap_dead start 2024-03-24T01:24:24.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.484+0000 7f37eba00640 1 -- 172.21.15.90:0/719277092 shutdown_connections 2024-03-24T01:24:24.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.485+0000 7f37eba00640 1 -- 172.21.15.90:0/719277092 >> 172.21.15.90:0/719277092 conn(0x7f37e40b1770 msgr2=0x7f37e40b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:24.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.485+0000 7f37eba00640 1 -- 172.21.15.90:0/719277092 shutdown_connections 2024-03-24T01:24:24.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.485+0000 7f37eba00640 1 -- 172.21.15.90:0/719277092 wait complete. 2024-03-24T01:24:24.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.486+0000 7f37eba00640 1 Processor -- start 2024-03-24T01:24:24.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.486+0000 7f37eba00640 1 -- start start 2024-03-24T01:24:24.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.487+0000 7f37eba00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37cc0039e0 0x7f37e4154160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:24.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.487+0000 7f37eba00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37e41501f0 0x7f37e4152fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:24.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.487+0000 7f37eba00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37e4155890 0x7f37e4144e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:24.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.487+0000 7f37eba00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e4159a00 con 0x7f37e41501f0 2024-03-24T01:24:24.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.487+0000 7f37eba00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e40b3f50 con 0x7f37e4155890 2024-03-24T01:24:24.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.487+0000 7f37eba00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37e415bfe0 con 0x7f37cc0039e0 2024-03-24T01:24:24.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.488+0000 7f37e3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37e41501f0 0x7f37e4152fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:24.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.488+0000 7f37e3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37e41501f0 0x7f37e4152fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40232/0 (socket says 172.21.15.90:40232) 2024-03-24T01:24:24.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.488+0000 7f37e3e00640 1 -- 172.21.15.90:0/1200291544 learned_addr learned my addr 172.21.15.90:0/1200291544 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:24.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.488+0000 7f37e9000640 1 --2- 172.21.15.90:0/1200291544 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37cc0039e0 0x7f37e4154160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:24.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.488+0000 7f37e9a00640 1 --2- 172.21.15.90:0/1200291544 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37e4155890 0x7f37e4144e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:24.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.488+0000 7f37e3e00640 1 -- 172.21.15.90:0/1200291544 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37cc0039e0 msgr2=0x7f37e4154160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:24.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.488+0000 7f37e3e00640 1 --2- 172.21.15.90:0/1200291544 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37cc0039e0 0x7f37e4154160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:24.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.488+0000 7f37e3e00640 1 -- 172.21.15.90:0/1200291544 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37e4155890 msgr2=0x7f37e4144e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:24.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.488+0000 7f37e3e00640 1 --2- 172.21.15.90:0/1200291544 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37e4155890 0x7f37e4144e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:24.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.488+0000 7f37e3e00640 1 -- 172.21.15.90:0/1200291544 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e41547f0 con 0x7f37e41501f0 2024-03-24T01:24:24.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.488+0000 7f37e9a00640 1 --2- 172.21.15.90:0/1200291544 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37e4155890 0x7f37e4144e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:24.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.488+0000 7f37e3e00640 1 --2- 172.21.15.90:0/1200291544 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37e41501f0 0x7f37e4152fd0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f37d0000f60 tx=0x7f37d000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:24.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.489+0000 7f37e0c00640 1 -- 172.21.15.90:0/1200291544 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d0005270 con 0x7f37e41501f0 2024-03-24T01:24:24.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.489+0000 7f37e0c00640 1 -- 172.21.15.90:0/1200291544 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37d0017070 con 0x7f37e41501f0 2024-03-24T01:24:24.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.489+0000 7f37eba00640 1 -- 172.21.15.90:0/1200291544 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37e4154ad0 con 0x7f37e41501f0 2024-03-24T01:24:24.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.489+0000 7f37e0c00640 1 -- 172.21.15.90:0/1200291544 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d000b6b0 con 0x7f37e41501f0 2024-03-24T01:24:24.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.489+0000 7f37eba00640 1 -- 172.21.15.90:0/1200291544 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37e4155090 con 0x7f37e41501f0 2024-03-24T01:24:24.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.490+0000 7f37eba00640 1 -- 172.21.15.90:0/1200291544 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37e404f400 con 0x7f37e41501f0 2024-03-24T01:24:24.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.491+0000 7f37e0c00640 1 -- 172.21.15.90:0/1200291544 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f37d000b850 con 0x7f37e41501f0 2024-03-24T01:24:24.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.492+0000 7f37e0c00640 1 --2- 172.21.15.90:0/1200291544 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f37c406bc00 0x7f37c406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:24.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.492+0000 7f37e0c00640 1 -- 172.21.15.90:0/1200291544 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f37d008ac50 con 0x7f37e41501f0 2024-03-24T01:24:24.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.492+0000 7f37e9000640 1 --2- 172.21.15.90:0/1200291544 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f37c406bc00 0x7f37c406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:24.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.493+0000 7f37e9000640 1 --2- 172.21.15.90:0/1200291544 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f37c406bc00 0x7f37c406e0c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f37d8034000 tx=0x7f37d8009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:24.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.496+0000 7f37e0c00640 1 -- 172.21.15.90:0/1200291544 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f37d0019310 con 0x7f37e41501f0 2024-03-24T01:24:24.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.692+0000 7f37eba00640 1 -- 172.21.15.90:0/1200291544 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f37e406c970 con 0x7f37e41501f0 2024-03-24T01:24:24.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.693+0000 7f37e0c00640 1 -- 172.21.15.90:0/1200291544 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f37d0059360 con 0x7f37e41501f0 2024-03-24T01:24:24.695 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-24T01:24:24.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.697+0000 7f37eba00640 1 -- 172.21.15.90:0/1200291544 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f37c406bc00 msgr2=0x7f37c406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:24.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.697+0000 7f37eba00640 1 --2- 172.21.15.90:0/1200291544 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f37c406bc00 0x7f37c406e0c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f37d8034000 tx=0x7f37d8009030 comp rx=0 tx=0).stop 2024-03-24T01:24:24.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.697+0000 7f37eba00640 1 -- 172.21.15.90:0/1200291544 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37e41501f0 msgr2=0x7f37e4152fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:24.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.697+0000 7f37eba00640 1 --2- 172.21.15.90:0/1200291544 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37e41501f0 0x7f37e4152fd0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f37d0000f60 tx=0x7f37d000dc00 comp rx=0 tx=0).stop 2024-03-24T01:24:24.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.697+0000 7f37eba00640 1 -- 172.21.15.90:0/1200291544 shutdown_connections 2024-03-24T01:24:24.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.697+0000 7f37eba00640 1 --2- 172.21.15.90:0/1200291544 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f37c406bc00 0x7f37c406e0c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:24.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.697+0000 7f37eba00640 1 --2- 172.21.15.90:0/1200291544 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37e41501f0 0x7f37e4152fd0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:24.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.697+0000 7f37eba00640 1 --2- 172.21.15.90:0/1200291544 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37e4155890 0x7f37e4144e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:24.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.697+0000 7f37eba00640 1 --2- 172.21.15.90:0/1200291544 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37cc0039e0 0x7f37e4154160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:24.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.697+0000 7f37eba00640 1 -- 172.21.15.90:0/1200291544 >> 172.21.15.90:0/1200291544 conn(0x7f37e40b1770 msgr2=0x7f37e415e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:24.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.697+0000 7f37eba00640 1 -- 172.21.15.90:0/1200291544 shutdown_connections 2024-03-24T01:24:24.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.697+0000 7f37eba00640 1 -- 172.21.15.90:0/1200291544 wait complete. 2024-03-24T01:24:24.716 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T01:24:24.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.930+0000 7f5f3ac00640 1 Processor -- start 2024-03-24T01:24:24.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.930+0000 7f5f3ac00640 1 -- start start 2024-03-24T01:24:24.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.930+0000 7f5f3ac00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5f34159620 0x7f5f3406b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:24.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.931+0000 7f5f3ac00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f340cf150 0x7f5f340ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:24.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.931+0000 7f5f3ac00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f3406bca0 con 0x7f5f3406b600 2024-03-24T01:24:24.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.931+0000 7f5f3ac00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f340b4470 con 0x7f5f34159620 2024-03-24T01:24:24.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.931+0000 7f5f3ac00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f340b36f0 con 0x7f5f340cf150 2024-03-24T01:24:24.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.931+0000 7f5f33e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5f34159620 0x7f5f3406b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:24.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.931+0000 7f5f33400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f5f3406b600 0x7f5f340cea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:36300/0 (socket says 172.21.15.90:36300) 2024-03-24T01:24:24.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.931+0000 7f5f33400640 1 -- 172.21.15.90:0/3941240127 learned_addr learned my addr 172.21.15.90:0/3941240127 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:24.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.931+0000 7f5f38c00640 1 --2- 172.21.15.90:0/3941240127 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f340cf150 0x7f5f340ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:24.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.931+0000 7f5f33e00640 1 -- 172.21.15.90:0/3941240127 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f340cf150 msgr2=0x7f5f340ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:24.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.931+0000 7f5f33e00640 1 --2- 172.21.15.90:0/3941240127 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f340cf150 0x7f5f340ae530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:24.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.931+0000 7f5f33e00640 1 -- 172.21.15.90:0/3941240127 >> v1:172.21.15.90:6789/0 conn(0x7f5f3406b600 legacy=0x7f5f340cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:24.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.931+0000 7f5f33e00640 1 -- 172.21.15.90:0/3941240127 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f340b3990 con 0x7f5f34159620 2024-03-24T01:24:24.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.933+0000 7f5f33e00640 1 --2- 172.21.15.90:0/3941240127 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f34159620 0x7f5f3406b0c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5f240097a0 tx=0x7f5f2402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4ab35e2f61416d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:24.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.933+0000 7f5f32a00640 1 -- 172.21.15.90:0/3941240127 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f24031780 con 0x7f5f34159620 2024-03-24T01:24:24.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.933+0000 7f5f32a00640 1 -- 172.21.15.90:0/3941240127 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f24037070 con 0x7f5f34159620 2024-03-24T01:24:24.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.933+0000 7f5f32a00640 1 -- 172.21.15.90:0/3941240127 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f24031b40 con 0x7f5f34159620 2024-03-24T01:24:24.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.934+0000 7f5f3ac00640 1 -- 172.21.15.90:0/3941240127 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f34159620 msgr2=0x7f5f3406b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:24.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.934+0000 7f5f3ac00640 1 --2- 172.21.15.90:0/3941240127 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f34159620 0x7f5f3406b0c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5f240097a0 tx=0x7f5f2402b820 comp rx=0 tx=0).stop 2024-03-24T01:24:24.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.934+0000 7f5f3ac00640 1 -- 172.21.15.90:0/3941240127 shutdown_connections 2024-03-24T01:24:24.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.934+0000 7f5f3ac00640 1 --2- 172.21.15.90:0/3941240127 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f340cf150 0x7f5f340ae530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:24.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.934+0000 7f5f3ac00640 1 --2- 172.21.15.90:0/3941240127 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f34159620 0x7f5f3406b0c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:24.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.934+0000 7f5f3ac00640 1 -- 172.21.15.90:0/3941240127 >> 172.21.15.90:0/3941240127 conn(0x7f5f340b1770 msgr2=0x7f5f340b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:24.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.934+0000 7f5f3ac00640 1 -- 172.21.15.90:0/3941240127 shutdown_connections 2024-03-24T01:24:24.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.934+0000 7f5f3ac00640 1 -- 172.21.15.90:0/3941240127 wait complete. 2024-03-24T01:24:24.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.936+0000 7f5f3ac00640 1 Processor -- start 2024-03-24T01:24:24.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.936+0000 7f5f3ac00640 1 -- start start 2024-03-24T01:24:24.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.937+0000 7f5f3ac00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f3406b600 0x7f5f3413e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:24.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.937+0000 7f5f3ac00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f340cf150 0x7f5f3414ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:24.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.937+0000 7f5f3ac00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f34159620 0x7f5f34155430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:24.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.937+0000 7f5f3ac00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f34159a00 con 0x7f5f3406b600 2024-03-24T01:24:24.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.937+0000 7f5f3ac00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f340b3f50 con 0x7f5f34159620 2024-03-24T01:24:24.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.937+0000 7f5f3ac00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3415bfe0 con 0x7f5f340cf150 2024-03-24T01:24:24.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.937+0000 7f5f33e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f3406b600 0x7f5f3413e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:24.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.937+0000 7f5f33e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f3406b600 0x7f5f3413e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40248/0 (socket says 172.21.15.90:40248) 2024-03-24T01:24:24.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.937+0000 7f5f33e00640 1 -- 172.21.15.90:0/926326023 learned_addr learned my addr 172.21.15.90:0/926326023 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:24.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.938+0000 7f5f33400640 1 --2- 172.21.15.90:0/926326023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f340cf150 0x7f5f3414ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:24.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.938+0000 7f5f33e00640 1 -- 172.21.15.90:0/926326023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f340cf150 msgr2=0x7f5f3414ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:24.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.938+0000 7f5f33e00640 1 --2- 172.21.15.90:0/926326023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f340cf150 0x7f5f3414ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:24.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.938+0000 7f5f33e00640 1 -- 172.21.15.90:0/926326023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f34159620 msgr2=0x7f5f34155430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:24.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.938+0000 7f5f33e00640 1 --2- 172.21.15.90:0/926326023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f34159620 0x7f5f34155430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:24.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.938+0000 7f5f33e00640 1 -- 172.21.15.90:0/926326023 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f34152e30 con 0x7f5f3406b600 2024-03-24T01:24:24.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.938+0000 7f5f33e00640 1 --2- 172.21.15.90:0/926326023 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f3406b600 0x7f5f3413e0c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5f24034040 tx=0x7f5f240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:24.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.939+0000 7f5f2be00640 1 -- 172.21.15.90:0/926326023 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f24032820 con 0x7f5f3406b600 2024-03-24T01:24:24.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.939+0000 7f5f2be00640 1 -- 172.21.15.90:0/926326023 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f240329e0 con 0x7f5f3406b600 2024-03-24T01:24:24.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.939+0000 7f5f2be00640 1 -- 172.21.15.90:0/926326023 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f24032dd0 con 0x7f5f3406b600 2024-03-24T01:24:24.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.939+0000 7f5f3ac00640 1 -- 172.21.15.90:0/926326023 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f3414f750 con 0x7f5f3406b600 2024-03-24T01:24:24.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.939+0000 7f5f3ac00640 1 -- 172.21.15.90:0/926326023 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f341530d0 con 0x7f5f3406b600 2024-03-24T01:24:24.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.940+0000 7f5f3ac00640 1 -- 172.21.15.90:0/926326023 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f3404f400 con 0x7f5f3406b600 2024-03-24T01:24:24.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.941+0000 7f5f2be00640 1 -- 172.21.15.90:0/926326023 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5f2404e440 con 0x7f5f3406b600 2024-03-24T01:24:24.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.942+0000 7f5f2be00640 1 --2- 172.21.15.90:0/926326023 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5f1406bbd0 0x7f5f1406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:24.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.942+0000 7f5f2be00640 1 -- 172.21.15.90:0/926326023 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f5f240acbc0 con 0x7f5f3406b600 2024-03-24T01:24:24.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.943+0000 7f5f33400640 1 --2- 172.21.15.90:0/926326023 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5f1406bbd0 0x7f5f1406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:24.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.944+0000 7f5f33400640 1 --2- 172.21.15.90:0/926326023 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5f1406bbd0 0x7f5f1406e090 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5f3414ae40 tx=0x7f5f20009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:24.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:24.947+0000 7f5f2be00640 1 -- 172.21.15.90:0/926326023 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5f24040070 con 0x7f5f3406b600 2024-03-24T01:24:25.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.144+0000 7f5f3ac00640 1 -- 172.21.15.90:0/926326023 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f5f340ceb70 con 0x7f5f3406b600 2024-03-24T01:24:25.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.145+0000 7f5f2be00640 1 -- 172.21.15.90:0/926326023 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7f5f2402f050 con 0x7f5f3406b600 2024-03-24T01:24:25.147 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:24:25.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.148+0000 7f5f3ac00640 1 -- 172.21.15.90:0/926326023 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5f1406bbd0 msgr2=0x7f5f1406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:25.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.148+0000 7f5f3ac00640 1 --2- 172.21.15.90:0/926326023 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5f1406bbd0 0x7f5f1406e090 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5f3414ae40 tx=0x7f5f20009040 comp rx=0 tx=0).stop 2024-03-24T01:24:25.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.148+0000 7f5f3ac00640 1 -- 172.21.15.90:0/926326023 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f3406b600 msgr2=0x7f5f3413e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:25.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.148+0000 7f5f3ac00640 1 --2- 172.21.15.90:0/926326023 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f3406b600 0x7f5f3413e0c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5f24034040 tx=0x7f5f240046c0 comp rx=0 tx=0).stop 2024-03-24T01:24:25.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.148+0000 7f5f3ac00640 1 -- 172.21.15.90:0/926326023 shutdown_connections 2024-03-24T01:24:25.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.148+0000 7f5f3ac00640 1 --2- 172.21.15.90:0/926326023 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5f1406bbd0 0x7f5f1406e090 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:25.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.148+0000 7f5f3ac00640 1 --2- 172.21.15.90:0/926326023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f34159620 0x7f5f34155430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:25.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.148+0000 7f5f3ac00640 1 --2- 172.21.15.90:0/926326023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5f340cf150 0x7f5f3414ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:25.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.148+0000 7f5f3ac00640 1 --2- 172.21.15.90:0/926326023 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f3406b600 0x7f5f3413e0c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:25.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.148+0000 7f5f3ac00640 1 -- 172.21.15.90:0/926326023 >> 172.21.15.90:0/926326023 conn(0x7f5f340b1770 msgr2=0x7f5f340cf770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:25.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.148+0000 7f5f3ac00640 1 -- 172.21.15.90:0/926326023 shutdown_connections 2024-03-24T01:24:25.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.148+0000 7f5f3ac00640 1 -- 172.21.15.90:0/926326023 wait complete. 2024-03-24T01:24:25.168 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T01:24:25.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.375+0000 7f9e0d600640 1 Processor -- start 2024-03-24T01:24:25.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.375+0000 7f9e0d600640 1 -- start start 2024-03-24T01:24:25.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.376+0000 7f9e0d600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9e08145190 0x7f9e08071790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:25.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.376+0000 7f9e0d600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e08071cd0 0x7f9e080cc900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:25.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.376+0000 7f9e0d600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e0806e0f0 con 0x7f9e08144680 2024-03-24T01:24:25.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.376+0000 7f9e0d600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e08144a60 con 0x7f9e08145190 2024-03-24T01:24:25.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.376+0000 7f9e0d600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e080cc190 con 0x7f9e08071cd0 2024-03-24T01:24:25.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.376+0000 7f9e07400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9e08145190 0x7f9e08071790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:25.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.376+0000 7f9e07400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9e08145190 0x7f9e08071790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40256/0 (socket says 172.21.15.90:40256) 2024-03-24T01:24:25.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.376+0000 7f9e07400640 1 -- 172.21.15.90:0/3835481055 learned_addr learned my addr 172.21.15.90:0/3835481055 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:25.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.376+0000 7f9e0cc00640 1 --2- 172.21.15.90:0/3835481055 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e08071cd0 0x7f9e080cc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:25.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.376+0000 7f9e07400640 1 -- 172.21.15.90:0/3835481055 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e08071cd0 msgr2=0x7f9e080cc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:25.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.376+0000 7f9e07400640 1 --2- 172.21.15.90:0/3835481055 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e08071cd0 0x7f9e080cc900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:25.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.377+0000 7f9e07400640 1 -- 172.21.15.90:0/3835481055 >> v1:172.21.15.120:6789/0 conn(0x7f9e08144680 legacy=0x7f9e0806d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:25.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.377+0000 7f9e07400640 1 -- 172.21.15.90:0/3835481055 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e0806e3d0 con 0x7f9e08145190 2024-03-24T01:24:25.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.377+0000 7f9e0cc00640 1 --2- 172.21.15.90:0/3835481055 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e08071cd0 0x7f9e080cc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:24:25.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.378+0000 7f9e07400640 1 --2- 172.21.15.90:0/3835481055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e08145190 0x7f9e08071790 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9df4009870 tx=0x7f9df402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9c6f630e380f881 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:25.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.378+0000 7f9e06a00640 1 -- 172.21.15.90:0/3835481055 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df4034070 con 0x7f9e08145190 2024-03-24T01:24:25.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.378+0000 7f9e06a00640 1 -- 172.21.15.90:0/3835481055 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df4030840 con 0x7f9e08145190 2024-03-24T01:24:25.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.378+0000 7f9e06a00640 1 -- 172.21.15.90:0/3835481055 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df4030c10 con 0x7f9e08145190 2024-03-24T01:24:25.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.379+0000 7f9e0d600640 1 -- 172.21.15.90:0/3835481055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e08145190 msgr2=0x7f9e08071790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:25.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.379+0000 7f9e0d600640 1 --2- 172.21.15.90:0/3835481055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e08145190 0x7f9e08071790 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9df4009870 tx=0x7f9df402b980 comp rx=0 tx=0).stop 2024-03-24T01:24:25.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.379+0000 7f9e0d600640 1 -- 172.21.15.90:0/3835481055 shutdown_connections 2024-03-24T01:24:25.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.379+0000 7f9e0d600640 1 --2- 172.21.15.90:0/3835481055 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e08071cd0 0x7f9e080cc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:25.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.379+0000 7f9e0d600640 1 --2- 172.21.15.90:0/3835481055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e08145190 0x7f9e08071790 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:25.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.379+0000 7f9e0d600640 1 -- 172.21.15.90:0/3835481055 >> 172.21.15.90:0/3835481055 conn(0x7f9e0806b0c0 msgr2=0x7f9e0806b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:25.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.379+0000 7f9e0d600640 1 -- 172.21.15.90:0/3835481055 shutdown_connections 2024-03-24T01:24:25.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.379+0000 7f9e0d600640 1 -- 172.21.15.90:0/3835481055 wait complete. 2024-03-24T01:24:25.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.381+0000 7f9e0d600640 1 Processor -- start 2024-03-24T01:24:25.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.381+0000 7f9e0d600640 1 -- start start 2024-03-24T01:24:25.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.382+0000 7f9e0d600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e08071cd0 0x7f9e08143900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:25.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.382+0000 7f9e0d600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e08144680 0x7f9e0815d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:25.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.382+0000 7f9e0d600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e08145190 0x7f9e0815ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:25.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.382+0000 7f9e0d600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e080cce40 con 0x7f9e08145190 2024-03-24T01:24:25.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.382+0000 7f9e0d600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e080cc470 con 0x7f9e08071cd0 2024-03-24T01:24:25.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.382+0000 7f9e0d600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e080ccfc0 con 0x7f9e08144680 2024-03-24T01:24:25.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.382+0000 7f9e0cc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e08145190 0x7f9e0815ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:25.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.382+0000 7f9e0cc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e08145190 0x7f9e0815ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40266/0 (socket says 172.21.15.90:40266) 2024-03-24T01:24:25.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.382+0000 7f9e0cc00640 1 -- 172.21.15.90:0/2152081693 learned_addr learned my addr 172.21.15.90:0/2152081693 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:25.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.382+0000 7f9e07400640 1 --2- 172.21.15.90:0/2152081693 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e08144680 0x7f9e0815d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:25.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.382+0000 7f9e07e00640 1 --2- 172.21.15.90:0/2152081693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e08071cd0 0x7f9e08143900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:25.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.382+0000 7f9e0cc00640 1 -- 172.21.15.90:0/2152081693 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e08144680 msgr2=0x7f9e0815d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:25.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.382+0000 7f9e0cc00640 1 --2- 172.21.15.90:0/2152081693 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e08144680 0x7f9e0815d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:25.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.382+0000 7f9e0cc00640 1 -- 172.21.15.90:0/2152081693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e08071cd0 msgr2=0x7f9e08143900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:25.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.382+0000 7f9e0cc00640 1 --2- 172.21.15.90:0/2152081693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e08071cd0 0x7f9e08143900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:25.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.382+0000 7f9e0cc00640 1 -- 172.21.15.90:0/2152081693 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e08143e40 con 0x7f9e08145190 2024-03-24T01:24:25.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.382+0000 7f9e07400640 1 --2- 172.21.15.90:0/2152081693 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e08144680 0x7f9e0815d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:25.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.383+0000 7f9e0cc00640 1 --2- 172.21.15.90:0/2152081693 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e08145190 0x7f9e0815ac70 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9e00007f20 tx=0x7f9e0000d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:25.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.383+0000 7f9dffe00640 1 -- 172.21.15.90:0/2152081693 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e0001a070 con 0x7f9e08145190 2024-03-24T01:24:25.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.383+0000 7f9dffe00640 1 -- 172.21.15.90:0/2152081693 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e00003b90 con 0x7f9e08145190 2024-03-24T01:24:25.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.383+0000 7f9e0d600640 1 -- 172.21.15.90:0/2152081693 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e080ae3c0 con 0x7f9e08145190 2024-03-24T01:24:25.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.383+0000 7f9dffe00640 1 -- 172.21.15.90:0/2152081693 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e00004030 con 0x7f9e08145190 2024-03-24T01:24:25.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.383+0000 7f9e0d600640 1 -- 172.21.15.90:0/2152081693 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e08143100 con 0x7f9e08145190 2024-03-24T01:24:25.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.384+0000 7f9e0d600640 1 -- 172.21.15.90:0/2152081693 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e0804f400 con 0x7f9e08145190 2024-03-24T01:24:25.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.385+0000 7f9dffe00640 1 -- 172.21.15.90:0/2152081693 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9e0001d070 con 0x7f9e08145190 2024-03-24T01:24:25.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.386+0000 7f9dffe00640 1 --2- 172.21.15.90:0/2152081693 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9de806bc00 0x7f9de806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:25.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.387+0000 7f9dffe00640 1 -- 172.21.15.90:0/2152081693 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f9e0008b1b0 con 0x7f9e08145190 2024-03-24T01:24:25.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.387+0000 7f9e07e00640 1 --2- 172.21.15.90:0/2152081693 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9de806bc00 0x7f9de806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:25.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.388+0000 7f9e07e00640 1 --2- 172.21.15.90:0/2152081693 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9de806bc00 0x7f9de806e0c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9df8001640 tx=0x7f9df8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:25.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.391+0000 7f9dffe00640 1 -- 172.21.15.90:0/2152081693 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9e00010070 con 0x7f9e08145190 2024-03-24T01:24:25.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.598+0000 7f9e0d600640 1 -- 172.21.15.90:0/2152081693 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f9e080b1040 con 0x7f9e08145190 2024-03-24T01:24:25.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.600+0000 7f9dffe00640 1 -- 172.21.15.90:0/2152081693 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) v1 ==== 80+0+0 (secure 0 0 0) 0x7f9e00059830 con 0x7f9e08145190 2024-03-24T01:24:25.600 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-24T01:24:25.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.603+0000 7f9e0d600640 1 -- 172.21.15.90:0/2152081693 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9de806bc00 msgr2=0x7f9de806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:25.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.603+0000 7f9e0d600640 1 --2- 172.21.15.90:0/2152081693 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9de806bc00 0x7f9de806e0c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9df8001640 tx=0x7f9df8009040 comp rx=0 tx=0).stop 2024-03-24T01:24:25.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.603+0000 7f9e0d600640 1 -- 172.21.15.90:0/2152081693 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e08145190 msgr2=0x7f9e0815ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:25.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.603+0000 7f9e0d600640 1 --2- 172.21.15.90:0/2152081693 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e08145190 0x7f9e0815ac70 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9e00007f20 tx=0x7f9e0000d560 comp rx=0 tx=0).stop 2024-03-24T01:24:25.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.603+0000 7f9e0d600640 1 -- 172.21.15.90:0/2152081693 shutdown_connections 2024-03-24T01:24:25.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.603+0000 7f9e0d600640 1 --2- 172.21.15.90:0/2152081693 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9de806bc00 0x7f9de806e0c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:25.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.603+0000 7f9e0d600640 1 --2- 172.21.15.90:0/2152081693 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e08145190 0x7f9e0815ac70 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:25.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.603+0000 7f9e0d600640 1 --2- 172.21.15.90:0/2152081693 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9e08144680 0x7f9e0815d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:25.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.603+0000 7f9e0d600640 1 --2- 172.21.15.90:0/2152081693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e08071cd0 0x7f9e08143900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:25.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.603+0000 7f9e0d600640 1 -- 172.21.15.90:0/2152081693 >> 172.21.15.90:0/2152081693 conn(0x7f9e0806b0c0 msgr2=0x7f9e080739c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:25.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.603+0000 7f9e0d600640 1 -- 172.21.15.90:0/2152081693 shutdown_connections 2024-03-24T01:24:25.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.603+0000 7f9e0d600640 1 -- 172.21.15.90:0/2152081693 wait complete. 2024-03-24T01:24:25.619 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:24:25.620 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:24:25.669 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:24:25.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.918+0000 7f6489400640 1 Processor -- start 2024-03-24T01:24:25.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.918+0000 7f6489400640 1 -- start start 2024-03-24T01:24:25.919 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:24:25.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.920+0000 7f6489400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f648415dd50 0x7f648406e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:25.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.920+0000 7f6489400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f64840b3d60 0x7f64841618d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:25.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.920+0000 7f6489400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64840acf50 con 0x7f648406b0c0 2024-03-24T01:24:25.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.920+0000 7f6489400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64840ad280 con 0x7f64840b3d60 2024-03-24T01:24:25.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.920+0000 7f6489400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64840b41c0 con 0x7f648415dd50 2024-03-24T01:24:25.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.921+0000 7f6482800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f648415dd50 0x7f648406e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:25.924 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:24:25.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.921+0000 7f6481e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f64840b3d60 0x7f64841618d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:25.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.921+0000 7f6483200640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f648406b0c0 0x7f64840ad710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.90:58352/0 (socket says 172.21.15.90:58352) 2024-03-24T01:24:25.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.921+0000 7f6483200640 1 -- 172.21.15.90:0/2304818506 learned_addr learned my addr 172.21.15.90:0/2304818506 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:25.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.921+0000 7f6482800640 1 -- 172.21.15.90:0/2304818506 >> v2:172.21.15.120:3300/0 conn(0x7f64840b3d60 msgr2=0x7f64841618d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:25.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.921+0000 7f6482800640 1 --2- 172.21.15.90:0/2304818506 >> v2:172.21.15.120:3300/0 conn(0x7f64840b3d60 0x7f64841618d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:25.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.921+0000 7f6482800640 1 -- 172.21.15.90:0/2304818506 >> v1:172.21.15.120:6789/0 conn(0x7f648406b0c0 legacy=0x7f64840ad710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:25.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.921+0000 7f6482800640 1 -- 172.21.15.90:0/2304818506 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f648415e130 con 0x7f648415dd50 2024-03-24T01:24:25.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.923+0000 7f6482800640 1 --2- 172.21.15.90:0/2304818506 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f648415dd50 0x7f648406e020 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6474009d90 tx=0x7f647402bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1704bae6130211bf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:25.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.923+0000 7f6481400640 1 -- 172.21.15.90:0/2304818506 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6474039070 con 0x7f648415dd50 2024-03-24T01:24:25.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.923+0000 7f6481400640 1 -- 172.21.15.90:0/2304818506 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f647402cb30 con 0x7f648415dd50 2024-03-24T01:24:25.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.923+0000 7f6481400640 1 -- 172.21.15.90:0/2304818506 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6474034440 con 0x7f648415dd50 2024-03-24T01:24:25.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.924+0000 7f6489400640 1 -- 172.21.15.90:0/2304818506 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f648415dd50 msgr2=0x7f648406e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:25.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.924+0000 7f6489400640 1 --2- 172.21.15.90:0/2304818506 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f648415dd50 0x7f648406e020 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6474009d90 tx=0x7f647402bd80 comp rx=0 tx=0).stop 2024-03-24T01:24:25.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.924+0000 7f6489400640 1 -- 172.21.15.90:0/2304818506 shutdown_connections 2024-03-24T01:24:25.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.924+0000 7f6489400640 1 --2- 172.21.15.90:0/2304818506 >> v2:172.21.15.120:3300/0 conn(0x7f64840b3d60 0x7f64841618d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:25.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.924+0000 7f6489400640 1 --2- 172.21.15.90:0/2304818506 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f648415dd50 0x7f648406e020 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:25.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.924+0000 7f6489400640 1 -- 172.21.15.90:0/2304818506 >> 172.21.15.90:0/2304818506 conn(0x7f64840b1770 msgr2=0x7f64840b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:25.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.924+0000 7f6489400640 1 -- 172.21.15.90:0/2304818506 shutdown_connections 2024-03-24T01:24:25.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.924+0000 7f6489400640 1 -- 172.21.15.90:0/2304818506 wait complete. 2024-03-24T01:24:25.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.926+0000 7f6489400640 1 Processor -- start 2024-03-24T01:24:25.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.926+0000 7f6489400640 1 -- start start 2024-03-24T01:24:25.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.927+0000 7f6489400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f648406b0c0 0x7f64841607b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:25.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.927+0000 7f6489400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64840b3d60 0x7f648415ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:25.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.927+0000 7f6489400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f648415dd50 0x7f648415d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:25.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.927+0000 7f6489400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64841625c0 con 0x7f648406b0c0 2024-03-24T01:24:25.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.927+0000 7f6489400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64841622e0 con 0x7f64840b3d60 2024-03-24T01:24:25.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.927+0000 7f6489400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f648406b530 con 0x7f648415dd50 2024-03-24T01:24:25.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.927+0000 7f6482800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f648406b0c0 0x7f64841607b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:25.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.927+0000 7f6482800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f648406b0c0 0x7f64841607b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40274/0 (socket says 172.21.15.90:40274) 2024-03-24T01:24:25.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.927+0000 7f6482800640 1 -- 172.21.15.90:0/582294288 learned_addr learned my addr 172.21.15.90:0/582294288 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:25.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.927+0000 7f6481e00640 1 --2- 172.21.15.90:0/582294288 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64840b3d60 0x7f648415ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:25.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.927+0000 7f6483200640 1 --2- 172.21.15.90:0/582294288 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f648415dd50 0x7f648415d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:25.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.928+0000 7f6482800640 1 -- 172.21.15.90:0/582294288 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f648415dd50 msgr2=0x7f648415d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:25.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.928+0000 7f6482800640 1 --2- 172.21.15.90:0/582294288 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f648415dd50 0x7f648415d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:25.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.928+0000 7f6482800640 1 -- 172.21.15.90:0/582294288 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64840b3d60 msgr2=0x7f648415ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:25.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.928+0000 7f6482800640 1 --2- 172.21.15.90:0/582294288 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64840b3d60 0x7f648415ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:25.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.928+0000 7f6482800640 1 -- 172.21.15.90:0/582294288 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6484160cf0 con 0x7f648406b0c0 2024-03-24T01:24:25.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.928+0000 7f6482800640 1 --2- 172.21.15.90:0/582294288 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f648406b0c0 0x7f64841607b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6474009d90 tx=0x7f6474004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:25.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.928+0000 7f647aa00640 1 -- 172.21.15.90:0/582294288 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6474039050 con 0x7f648406b0c0 2024-03-24T01:24:25.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.928+0000 7f647aa00640 1 -- 172.21.15.90:0/582294288 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6474045070 con 0x7f648406b0c0 2024-03-24T01:24:25.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.928+0000 7f647aa00640 1 -- 172.21.15.90:0/582294288 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f647403e850 con 0x7f648406b0c0 2024-03-24T01:24:25.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.928+0000 7f6489400640 1 -- 172.21.15.90:0/582294288 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f648415f710 con 0x7f648406b0c0 2024-03-24T01:24:25.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.929+0000 7f6489400640 1 -- 172.21.15.90:0/582294288 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f648415fcd0 con 0x7f648406b0c0 2024-03-24T01:24:25.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.930+0000 7f6489400640 1 -- 172.21.15.90:0/582294288 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6484068060 con 0x7f648406b0c0 2024-03-24T01:24:25.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.931+0000 7f647aa00640 1 -- 172.21.15.90:0/582294288 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f647403e9f0 con 0x7f648406b0c0 2024-03-24T01:24:25.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.932+0000 7f647aa00640 1 --2- 172.21.15.90:0/582294288 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f646406bbd0 0x7f646406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:25.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.932+0000 7f647aa00640 1 -- 172.21.15.90:0/582294288 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f64740add90 con 0x7f648406b0c0 2024-03-24T01:24:25.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.932+0000 7f6481e00640 1 --2- 172.21.15.90:0/582294288 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f646406bbd0 0x7f646406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:25.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.933+0000 7f6481e00640 1 --2- 172.21.15.90:0/582294288 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f646406bbd0 0x7f646406e090 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f64840b0b60 tx=0x7f6470008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:25.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:25.936+0000 7f647aa00640 1 -- 172.21.15.90:0/582294288 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f647407c650 con 0x7f648406b0c0 2024-03-24T01:24:26.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.144+0000 7f6489400640 1 -- 172.21.15.90:0/582294288 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64840b1300 con 0x7f648406b0c0 2024-03-24T01:24:26.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.147+0000 7f647aa00640 1 -- 172.21.15.90:0/582294288 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18490 (secure 0 0 0) 0x7f647407bf50 con 0x7f648406b0c0 2024-03-24T01:24:26.148 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:24:26.149 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:24:17.008224+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:24:08.643966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T01:24:09.189904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T01:24:13.027323+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:24:26.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.150+0000 7f6489400640 1 -- 172.21.15.90:0/582294288 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f646406bbd0 msgr2=0x7f646406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:26.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.150+0000 7f6489400640 1 --2- 172.21.15.90:0/582294288 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f646406bbd0 0x7f646406e090 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f64840b0b60 tx=0x7f6470008040 comp rx=0 tx=0).stop 2024-03-24T01:24:26.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.150+0000 7f6489400640 1 -- 172.21.15.90:0/582294288 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f648406b0c0 msgr2=0x7f64841607b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:26.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.150+0000 7f6489400640 1 --2- 172.21.15.90:0/582294288 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f648406b0c0 0x7f64841607b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6474009d90 tx=0x7f6474004a40 comp rx=0 tx=0).stop 2024-03-24T01:24:26.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.150+0000 7f6489400640 1 -- 172.21.15.90:0/582294288 shutdown_connections 2024-03-24T01:24:26.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.150+0000 7f6489400640 1 --2- 172.21.15.90:0/582294288 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f646406bbd0 0x7f646406e090 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:26.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.150+0000 7f6489400640 1 --2- 172.21.15.90:0/582294288 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f648415dd50 0x7f648415d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:26.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.150+0000 7f6489400640 1 --2- 172.21.15.90:0/582294288 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64840b3d60 0x7f648415ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:26.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.150+0000 7f6489400640 1 --2- 172.21.15.90:0/582294288 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f648406b0c0 0x7f64841607b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:26.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.150+0000 7f6489400640 1 -- 172.21.15.90:0/582294288 >> 172.21.15.90:0/582294288 conn(0x7f64840b1770 msgr2=0x7f648406e2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:26.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.150+0000 7f6489400640 1 -- 172.21.15.90:0/582294288 shutdown_connections 2024-03-24T01:24:26.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.151+0000 7f6489400640 1 -- 172.21.15.90:0/582294288 wait complete. 2024-03-24T01:24:26.171 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T01:24:08.643966+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T01:24:09.189904+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T01:24:13.027323+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T01:24:26.172 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:24:26.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.375+0000 7ff437200640 1 Processor -- start 2024-03-24T01:24:26.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.375+0000 7ff437200640 1 -- start start 2024-03-24T01:24:26.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.376+0000 7ff437200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff4381549f0 0x7ff438154e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:26.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.376+0000 7ff437200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4381553a0 con 0x7ff43806f0d0 2024-03-24T01:24:26.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.376+0000 7ff437200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff43815b9d0 con 0x7ff43806ecf0 2024-03-24T01:24:26.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.376+0000 7ff437200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff43815bcb0 con 0x7ff4381549f0 2024-03-24T01:24:26.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.376+0000 7ff42d400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff43806f0d0 0x7ff43815af80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:36306/0 (socket says 172.21.15.90:36306) 2024-03-24T01:24:26.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.376+0000 7ff42d400640 1 -- 172.21.15.90:0/4007271215 learned_addr learned my addr 172.21.15.90:0/4007271215 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:26.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.377+0000 7ff436800640 1 --2- 172.21.15.90:0/4007271215 >> v2:172.21.15.90:3300/0 conn(0x7ff4381549f0 0x7ff438154e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:26.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.377+0000 7ff435400640 1 -- 172.21.15.90:0/4007271215 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729898499 0 0) 0x7ff4381553a0 con 0x7ff43806f0d0 2024-03-24T01:24:26.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.377+0000 7ff435400640 1 -- 172.21.15.90:0/4007271215 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff41c005730 con 0x7ff43806f0d0 2024-03-24T01:24:26.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.377+0000 7ff436800640 1 -- 172.21.15.90:0/4007271215 >> v1:172.21.15.120:6789/0 conn(0x7ff43806ecf0 legacy=0x7ff4380b3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:26.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.377+0000 7ff436800640 1 -- 172.21.15.90:0/4007271215 >> v1:172.21.15.90:6789/0 conn(0x7ff43806f0d0 legacy=0x7ff43815af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:26.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.377+0000 7ff436800640 1 -- 172.21.15.90:0/4007271215 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4380b2930 con 0x7ff4381549f0 2024-03-24T01:24:26.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.378+0000 7ff436800640 1 --2- 172.21.15.90:0/4007271215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff4381549f0 0x7ff438154e60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff430009d90 tx=0x7ff43002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1ce79ba8dfd6994 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:26.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.379+0000 7ff435400640 1 -- 172.21.15.90:0/4007271215 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff430039070 con 0x7ff4381549f0 2024-03-24T01:24:26.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.379+0000 7ff435400640 1 -- 172.21.15.90:0/4007271215 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff43002cd00 con 0x7ff4381549f0 2024-03-24T01:24:26.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.379+0000 7ff435400640 1 -- 172.21.15.90:0/4007271215 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff430034660 con 0x7ff4381549f0 2024-03-24T01:24:26.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.379+0000 7ff437200640 1 -- 172.21.15.90:0/4007271215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff4381549f0 msgr2=0x7ff438154e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:26.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.379+0000 7ff437200640 1 --2- 172.21.15.90:0/4007271215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff4381549f0 0x7ff438154e60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff430009d90 tx=0x7ff43002bf50 comp rx=0 tx=0).stop 2024-03-24T01:24:26.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.379+0000 7ff437200640 1 -- 172.21.15.90:0/4007271215 shutdown_connections 2024-03-24T01:24:26.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.379+0000 7ff437200640 1 --2- 172.21.15.90:0/4007271215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff4381549f0 0x7ff438154e60 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:26.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.379+0000 7ff437200640 1 -- 172.21.15.90:0/4007271215 >> 172.21.15.90:0/4007271215 conn(0x7ff43806b0c0 msgr2=0x7ff43806b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:26.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.379+0000 7ff437200640 1 -- 172.21.15.90:0/4007271215 shutdown_connections 2024-03-24T01:24:26.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.379+0000 7ff437200640 1 -- 172.21.15.90:0/4007271215 wait complete. 2024-03-24T01:24:26.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.381+0000 7ff437200640 1 Processor -- start 2024-03-24T01:24:26.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.381+0000 7ff437200640 1 -- start start 2024-03-24T01:24:26.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.382+0000 7ff437200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff43806ecf0 0x7ff4381462f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:26.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.382+0000 7ff437200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff43806f0d0 0x7ff438155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:26.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.382+0000 7ff437200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff4381549f0 0x7ff4381575b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:26.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.382+0000 7ff437200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4380b45a0 con 0x7ff4381549f0 2024-03-24T01:24:26.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.382+0000 7ff437200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff43815b690 con 0x7ff43806f0d0 2024-03-24T01:24:26.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.382+0000 7ff437200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4380b4720 con 0x7ff43806ecf0 2024-03-24T01:24:26.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.382+0000 7ff436800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff4381549f0 0x7ff4381575b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:26.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.382+0000 7ff436800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff4381549f0 0x7ff4381575b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40290/0 (socket says 172.21.15.90:40290) 2024-03-24T01:24:26.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.382+0000 7ff436800640 1 -- 172.21.15.90:0/571534104 learned_addr learned my addr 172.21.15.90:0/571534104 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:26.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.383+0000 7ff42d400640 1 --2- 172.21.15.90:0/571534104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff43806f0d0 0x7ff438155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:26.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.383+0000 7ff435e00640 1 --2- 172.21.15.90:0/571534104 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff43806ecf0 0x7ff4381462f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:26.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.383+0000 7ff436800640 1 -- 172.21.15.90:0/571534104 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff43806ecf0 msgr2=0x7ff4381462f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:26.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.383+0000 7ff436800640 1 --2- 172.21.15.90:0/571534104 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff43806ecf0 0x7ff4381462f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:26.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.383+0000 7ff436800640 1 -- 172.21.15.90:0/571534104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff43806f0d0 msgr2=0x7ff438155cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:26.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.383+0000 7ff436800640 1 --2- 172.21.15.90:0/571534104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff43806f0d0 0x7ff438155cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:26.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.383+0000 7ff436800640 1 -- 172.21.15.90:0/571534104 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff438156510 con 0x7ff4381549f0 2024-03-24T01:24:26.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.383+0000 7ff436800640 1 --2- 172.21.15.90:0/571534104 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff4381549f0 0x7ff4381575b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff430009eb0 tx=0x7ff430004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:26.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.383+0000 7ff42ea00640 1 -- 172.21.15.90:0/571534104 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff430039050 con 0x7ff4381549f0 2024-03-24T01:24:26.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.383+0000 7ff42ea00640 1 -- 172.21.15.90:0/571534104 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff430045070 con 0x7ff4381549f0 2024-03-24T01:24:26.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.383+0000 7ff42ea00640 1 -- 172.21.15.90:0/571534104 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff430009450 con 0x7ff4381549f0 2024-03-24T01:24:26.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.384+0000 7ff437200640 1 -- 172.21.15.90:0/571534104 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4381567f0 con 0x7ff4381549f0 2024-03-24T01:24:26.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.384+0000 7ff437200640 1 -- 172.21.15.90:0/571534104 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff438156db0 con 0x7ff4381549f0 2024-03-24T01:24:26.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.385+0000 7ff437200640 1 -- 172.21.15.90:0/571534104 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff43804f400 con 0x7ff4381549f0 2024-03-24T01:24:26.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.386+0000 7ff42ea00640 1 -- 172.21.15.90:0/571534104 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff43003e850 con 0x7ff4381549f0 2024-03-24T01:24:26.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.387+0000 7ff42ea00640 1 --2- 172.21.15.90:0/571534104 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff41406bbd0 0x7ff41406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:26.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.387+0000 7ff42ea00640 1 -- 172.21.15.90:0/571534104 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ff4300add70 con 0x7ff4381549f0 2024-03-24T01:24:26.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.387+0000 7ff435e00640 1 --2- 172.21.15.90:0/571534104 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff41406bbd0 0x7ff41406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:26.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.388+0000 7ff435e00640 1 --2- 172.21.15.90:0/571534104 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff41406bbd0 0x7ff41406e090 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff428000f60 tx=0x7ff428009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:26.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.392+0000 7ff42ea00640 1 -- 172.21.15.90:0/571534104 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff430032e20 con 0x7ff4381549f0 2024-03-24T01:24:26.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.586+0000 7ff437200640 1 -- 172.21.15.90:0/571534104 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff43815b470 con 0x7ff4381549f0 2024-03-24T01:24:26.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.587+0000 7ff42ea00640 1 -- 172.21.15.90:0/571534104 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff43007c640 con 0x7ff4381549f0 2024-03-24T01:24:26.587 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:24:26.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.590+0000 7ff437200640 1 -- 172.21.15.90:0/571534104 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff41406bbd0 msgr2=0x7ff41406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:26.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.590+0000 7ff437200640 1 --2- 172.21.15.90:0/571534104 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff41406bbd0 0x7ff41406e090 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff428000f60 tx=0x7ff428009040 comp rx=0 tx=0).stop 2024-03-24T01:24:26.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.590+0000 7ff437200640 1 -- 172.21.15.90:0/571534104 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff4381549f0 msgr2=0x7ff4381575b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:26.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.590+0000 7ff437200640 1 --2- 172.21.15.90:0/571534104 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff4381549f0 0x7ff4381575b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff430009eb0 tx=0x7ff430004a40 comp rx=0 tx=0).stop 2024-03-24T01:24:26.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.590+0000 7ff437200640 1 -- 172.21.15.90:0/571534104 shutdown_connections 2024-03-24T01:24:26.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.590+0000 7ff437200640 1 --2- 172.21.15.90:0/571534104 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff41406bbd0 0x7ff41406e090 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:26.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.590+0000 7ff437200640 1 --2- 172.21.15.90:0/571534104 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff4381549f0 0x7ff4381575b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:26.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.590+0000 7ff437200640 1 --2- 172.21.15.90:0/571534104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff43806f0d0 0x7ff438155cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:26.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.590+0000 7ff437200640 1 --2- 172.21.15.90:0/571534104 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff43806ecf0 0x7ff4381462f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:26.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.590+0000 7ff437200640 1 -- 172.21.15.90:0/571534104 >> 172.21.15.90:0/571534104 conn(0x7ff43806b0c0 msgr2=0x7ff4380b3130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:26.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.590+0000 7ff437200640 1 -- 172.21.15.90:0/571534104 shutdown_connections 2024-03-24T01:24:26.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.591+0000 7ff437200640 1 -- 172.21.15.90:0/571534104 wait complete. 2024-03-24T01:24:26.609 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T01:24:26.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.826+0000 7f45fa200640 1 Processor -- start 2024-03-24T01:24:26.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.826+0000 7f45fa200640 1 -- start start 2024-03-24T01:24:26.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.827+0000 7f45fa200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45f40cc8a0 0x7f45f40ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:26.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.827+0000 7f45fa200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f45f406b0c0 0x7f45f406b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:26.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.827+0000 7f45fa200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45f40b3eb0 con 0x7f45f40aee80 2024-03-24T01:24:26.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.827+0000 7f45fa200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45f406e680 con 0x7f45f406b0c0 2024-03-24T01:24:26.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.827+0000 7f45fa200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45f406e960 con 0x7f45f40cc8a0 2024-03-24T01:24:26.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.828+0000 7f45f3e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f45f406b0c0 0x7f45f406b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:26.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.828+0000 7f45f3e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f45f406b0c0 0x7f45f406b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40294/0 (socket says 172.21.15.90:40294) 2024-03-24T01:24:26.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.828+0000 7f45f3e00640 1 -- 172.21.15.90:0/3008538101 learned_addr learned my addr 172.21.15.90:0/3008538101 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:26.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.828+0000 7f45f2a00640 1 --2- 172.21.15.90:0/3008538101 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45f40cc8a0 0x7f45f40ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:26.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.828+0000 7f45f3e00640 1 -- 172.21.15.90:0/3008538101 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45f40cc8a0 msgr2=0x7f45f40ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:26.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.828+0000 7f45f3e00640 1 --2- 172.21.15.90:0/3008538101 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45f40cc8a0 0x7f45f40ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:26.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.828+0000 7f45f3e00640 1 -- 172.21.15.90:0/3008538101 >> v1:172.21.15.120:6789/0 conn(0x7f45f40aee80 legacy=0x7f45f40b33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:26.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.829+0000 7f45f3e00640 1 -- 172.21.15.90:0/3008538101 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f406ec40 con 0x7f45f406b0c0 2024-03-24T01:24:26.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.830+0000 7f45f3e00640 1 --2- 172.21.15.90:0/3008538101 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45f406b0c0 0x7f45f406b550 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f45ec009e50 tx=0x7f45ec02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3579a2cab3b221b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:26.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.830+0000 7f45f2000640 1 -- 172.21.15.90:0/3008538101 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ec02ca80 con 0x7f45f406b0c0 2024-03-24T01:24:26.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.830+0000 7f45f2000640 1 -- 172.21.15.90:0/3008538101 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45ec02cc20 con 0x7f45f406b0c0 2024-03-24T01:24:26.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.830+0000 7f45f2000640 1 -- 172.21.15.90:0/3008538101 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ec0343f0 con 0x7f45f406b0c0 2024-03-24T01:24:26.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.830+0000 7f45fa200640 1 -- 172.21.15.90:0/3008538101 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45f406b0c0 msgr2=0x7f45f406b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:26.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.831+0000 7f45fa200640 1 --2- 172.21.15.90:0/3008538101 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45f406b0c0 0x7f45f406b550 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f45ec009e50 tx=0x7f45ec02bee0 comp rx=0 tx=0).stop 2024-03-24T01:24:26.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.831+0000 7f45fa200640 1 -- 172.21.15.90:0/3008538101 shutdown_connections 2024-03-24T01:24:26.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.831+0000 7f45fa200640 1 --2- 172.21.15.90:0/3008538101 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45f406b0c0 0x7f45f406b550 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:26.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.831+0000 7f45fa200640 1 --2- 172.21.15.90:0/3008538101 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45f40cc8a0 0x7f45f40ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:26.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.831+0000 7f45fa200640 1 -- 172.21.15.90:0/3008538101 >> 172.21.15.90:0/3008538101 conn(0x7f45f40b1770 msgr2=0x7f45f40b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:26.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.831+0000 7f45fa200640 1 -- 172.21.15.90:0/3008538101 shutdown_connections 2024-03-24T01:24:26.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.831+0000 7f45fa200640 1 -- 172.21.15.90:0/3008538101 wait complete. 2024-03-24T01:24:26.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.832+0000 7f45fa200640 1 Processor -- start 2024-03-24T01:24:26.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.833+0000 7f45fa200640 1 -- start start 2024-03-24T01:24:26.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.834+0000 7f45fa200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45f406b0c0 0x7f45f4147580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:26.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.834+0000 7f45fa200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45f40aee80 0x7f45f4071990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:26.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.834+0000 7f45fa200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45f40cc8a0 0x7f45f4071ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:26.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.834+0000 7f45fa200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f40b3b00 con 0x7f45f406b0c0 2024-03-24T01:24:26.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.834+0000 7f45fa200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f4166ad0 con 0x7f45f40cc8a0 2024-03-24T01:24:26.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.834+0000 7f45fa200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45f41651c0 con 0x7f45f40aee80 2024-03-24T01:24:26.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.834+0000 7f45f3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45f406b0c0 0x7f45f4147580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:26.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.834+0000 7f45f3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45f406b0c0 0x7f45f4147580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40310/0 (socket says 172.21.15.90:40310) 2024-03-24T01:24:26.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.834+0000 7f45f3400640 1 -- 172.21.15.90:0/3406092022 learned_addr learned my addr 172.21.15.90:0/3406092022 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:26.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.834+0000 7f45f2a00640 1 --2- 172.21.15.90:0/3406092022 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45f40aee80 0x7f45f4071990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:26.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.834+0000 7f45f3400640 1 -- 172.21.15.90:0/3406092022 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45f40aee80 msgr2=0x7f45f4071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:26.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.834+0000 7f45f3400640 1 --2- 172.21.15.90:0/3406092022 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45f40aee80 0x7f45f4071990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:26.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.834+0000 7f45f3400640 1 -- 172.21.15.90:0/3406092022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45f40cc8a0 msgr2=0x7f45f4071ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:26.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.834+0000 7f45f3400640 1 --2- 172.21.15.90:0/3406092022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45f40cc8a0 0x7f45f4071ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:26.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.834+0000 7f45f3400640 1 -- 172.21.15.90:0/3406092022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f4165650 con 0x7f45f406b0c0 2024-03-24T01:24:26.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.835+0000 7f45f3400640 1 --2- 172.21.15.90:0/3406092022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45f406b0c0 0x7f45f4147580 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f45e400c400 tx=0x7f45e400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:26.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.835+0000 7f45eb400640 1 -- 172.21.15.90:0/3406092022 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e40052d0 con 0x7f45f406b0c0 2024-03-24T01:24:26.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.835+0000 7f45eb400640 1 -- 172.21.15.90:0/3406092022 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45e4005490 con 0x7f45f406b0c0 2024-03-24T01:24:26.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.835+0000 7f45fa200640 1 -- 172.21.15.90:0/3406092022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45f4147ac0 con 0x7f45f406b0c0 2024-03-24T01:24:26.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.835+0000 7f45eb400640 1 -- 172.21.15.90:0/3406092022 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e4007260 con 0x7f45f406b0c0 2024-03-24T01:24:26.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.835+0000 7f45fa200640 1 -- 172.21.15.90:0/3406092022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45f4164890 con 0x7f45f406b0c0 2024-03-24T01:24:26.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.836+0000 7f45fa200640 1 -- 172.21.15.90:0/3406092022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45f404f400 con 0x7f45f406b0c0 2024-03-24T01:24:26.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.837+0000 7f45eb400640 1 -- 172.21.15.90:0/3406092022 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f45e4007400 con 0x7f45f406b0c0 2024-03-24T01:24:26.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.838+0000 7f45eb400640 1 --2- 172.21.15.90:0/3406092022 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f45d406bbd0 0x7f45d406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:26.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.838+0000 7f45eb400640 1 -- 172.21.15.90:0/3406092022 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f45e408a2d0 con 0x7f45f406b0c0 2024-03-24T01:24:26.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.839+0000 7f45f2a00640 1 --2- 172.21.15.90:0/3406092022 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f45d406bbd0 0x7f45d406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:26.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.839+0000 7f45f2a00640 1 --2- 172.21.15.90:0/3406092022 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f45d406bbd0 0x7f45d406e090 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f45e0006fd0 tx=0x7f45e0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:26.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:26.842+0000 7f45eb400640 1 -- 172.21.15.90:0/3406092022 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f45e4058a80 con 0x7f45f406b0c0 2024-03-24T01:24:27.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.039+0000 7f45fa200640 1 -- 172.21.15.90:0/3406092022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f45f406e250 con 0x7f45f406b0c0 2024-03-24T01:24:27.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.040+0000 7f45eb400640 1 -- 172.21.15.90:0/3406092022 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f45e408a0f0 con 0x7f45f406b0c0 2024-03-24T01:24:27.041 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:24:27.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.043+0000 7f45fa200640 1 -- 172.21.15.90:0/3406092022 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f45d406bbd0 msgr2=0x7f45d406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:27.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.043+0000 7f45fa200640 1 --2- 172.21.15.90:0/3406092022 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f45d406bbd0 0x7f45d406e090 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f45e0006fd0 tx=0x7f45e0008040 comp rx=0 tx=0).stop 2024-03-24T01:24:27.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.043+0000 7f45fa200640 1 -- 172.21.15.90:0/3406092022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45f406b0c0 msgr2=0x7f45f4147580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:27.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.043+0000 7f45fa200640 1 --2- 172.21.15.90:0/3406092022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45f406b0c0 0x7f45f4147580 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f45e400c400 tx=0x7f45e400c8c0 comp rx=0 tx=0).stop 2024-03-24T01:24:27.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.043+0000 7f45fa200640 1 -- 172.21.15.90:0/3406092022 shutdown_connections 2024-03-24T01:24:27.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.043+0000 7f45fa200640 1 --2- 172.21.15.90:0/3406092022 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f45d406bbd0 0x7f45d406e090 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:27.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.043+0000 7f45fa200640 1 --2- 172.21.15.90:0/3406092022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45f40cc8a0 0x7f45f4071ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:27.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.043+0000 7f45fa200640 1 --2- 172.21.15.90:0/3406092022 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45f40aee80 0x7f45f4071990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:27.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.043+0000 7f45fa200640 1 --2- 172.21.15.90:0/3406092022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45f406b0c0 0x7f45f4147580 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:27.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.043+0000 7f45fa200640 1 -- 172.21.15.90:0/3406092022 >> 172.21.15.90:0/3406092022 conn(0x7f45f40b1770 msgr2=0x7f45f40b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:27.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.043+0000 7f45fa200640 1 -- 172.21.15.90:0/3406092022 shutdown_connections 2024-03-24T01:24:27.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.043+0000 7f45fa200640 1 -- 172.21.15.90:0/3406092022 wait complete. 2024-03-24T01:24:27.063 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T01:24:27.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.260+0000 7f0e21c00640 1 Processor -- start 2024-03-24T01:24:27.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.261+0000 7f0e21c00640 1 -- start start 2024-03-24T01:24:27.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.262+0000 7f0e21c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0e1c15ded0 0x7f0e1c0b9420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:27.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.262+0000 7f0e21c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0e1c0b2d90 0x7f0e1c0b6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:27.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.262+0000 7f0e21c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e1c0d1f70 con 0x7f0e1c0b7170 2024-03-24T01:24:27.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.262+0000 7f0e21c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e1c0d2250 con 0x7f0e1c0b2d90 2024-03-24T01:24:27.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.262+0000 7f0e21c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e1c0d2530 con 0x7f0e1c15ded0 2024-03-24T01:24:27.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.262+0000 7f0e1a000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0e1c0b2d90 0x7f0e1c0b6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:27.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.262+0000 7f0e1a000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0e1c0b2d90 0x7f0e1c0b6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40318/0 (socket says 172.21.15.90:40318) 2024-03-24T01:24:27.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.262+0000 7f0e1a000640 1 -- 172.21.15.90:0/3861542263 learned_addr learned my addr 172.21.15.90:0/3861542263 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:27.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.263+0000 7f0e1aa00640 1 --2- 172.21.15.90:0/3861542263 >> v2:172.21.15.120:3300/0 conn(0x7f0e1c15ded0 0x7f0e1c0b9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:27.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.263+0000 7f0e1a000640 1 -- 172.21.15.90:0/3861542263 >> v2:172.21.15.120:3300/0 conn(0x7f0e1c15ded0 msgr2=0x7f0e1c0b9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:27.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.263+0000 7f0e1a000640 1 --2- 172.21.15.90:0/3861542263 >> v2:172.21.15.120:3300/0 conn(0x7f0e1c15ded0 0x7f0e1c0b9420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:27.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.263+0000 7f0e1a000640 1 -- 172.21.15.90:0/3861542263 >> v1:172.21.15.120:6789/0 conn(0x7f0e1c0b7170 legacy=0x7f0e1c06c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:27.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.263+0000 7f0e1a000640 1 -- 172.21.15.90:0/3861542263 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e1c0d2810 con 0x7f0e1c0b2d90 2024-03-24T01:24:27.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.264+0000 7f0e1a000640 1 --2- 172.21.15.90:0/3861542263 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e1c0b2d90 0x7f0e1c0b6c30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c06d430 tx=0x7f0e0402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d106708b37771013 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:27.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.264+0000 7f0e19600640 1 -- 172.21.15.90:0/3861542263 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e04034070 con 0x7f0e1c0b2d90 2024-03-24T01:24:27.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.264+0000 7f0e19600640 1 -- 172.21.15.90:0/3861542263 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e04030860 con 0x7f0e1c0b2d90 2024-03-24T01:24:27.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.264+0000 7f0e19600640 1 -- 172.21.15.90:0/3861542263 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e04030c30 con 0x7f0e1c0b2d90 2024-03-24T01:24:27.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.265+0000 7f0e21c00640 1 -- 172.21.15.90:0/3861542263 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e1c0b2d90 msgr2=0x7f0e1c0b6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:27.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.265+0000 7f0e21c00640 1 --2- 172.21.15.90:0/3861542263 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e1c0b2d90 0x7f0e1c0b6c30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c06d430 tx=0x7f0e0402b9a0 comp rx=0 tx=0).stop 2024-03-24T01:24:27.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.265+0000 7f0e21c00640 1 -- 172.21.15.90:0/3861542263 shutdown_connections 2024-03-24T01:24:27.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.265+0000 7f0e21c00640 1 --2- 172.21.15.90:0/3861542263 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e1c0b2d90 0x7f0e1c0b6c30 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:27.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.265+0000 7f0e21c00640 1 --2- 172.21.15.90:0/3861542263 >> v2:172.21.15.120:3300/0 conn(0x7f0e1c15ded0 0x7f0e1c0b9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:27.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.265+0000 7f0e21c00640 1 -- 172.21.15.90:0/3861542263 >> 172.21.15.90:0/3861542263 conn(0x7f0e1c072e20 msgr2=0x7f0e1c0d3bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:27.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.265+0000 7f0e21c00640 1 -- 172.21.15.90:0/3861542263 shutdown_connections 2024-03-24T01:24:27.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.265+0000 7f0e21c00640 1 -- 172.21.15.90:0/3861542263 wait complete. 2024-03-24T01:24:27.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.267+0000 7f0e21c00640 1 Processor -- start 2024-03-24T01:24:27.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.267+0000 7f0e21c00640 1 -- start start 2024-03-24T01:24:27.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.268+0000 7f0e21c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e1c0b2d90 0x7f0e1c0b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:27.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.268+0000 7f0e21c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e1c0b7170 0x7f0e1c0b1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:27.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.268+0000 7f0e21c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e1c15ded0 0x7f0e1c159d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:27.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.268+0000 7f0e21c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1c0b9ae0 con 0x7f0e1c0b7170 2024-03-24T01:24:27.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.268+0000 7f0e21c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1c0b9960 con 0x7f0e1c15ded0 2024-03-24T01:24:27.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.268+0000 7f0e21c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1c06ce10 con 0x7f0e1c0b2d90 2024-03-24T01:24:27.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.268+0000 7f0e1a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e1c0b7170 0x7f0e1c0b1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:27.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.268+0000 7f0e1a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e1c0b7170 0x7f0e1c0b1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40332/0 (socket says 172.21.15.90:40332) 2024-03-24T01:24:27.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.268+0000 7f0e1a000640 1 -- 172.21.15.90:0/1629961908 learned_addr learned my addr 172.21.15.90:0/1629961908 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:27.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.268+0000 7f0e1a000640 1 -- 172.21.15.90:0/1629961908 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e1c0b2d90 msgr2=0x7f0e1c0b47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:27.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.268+0000 7f0e1aa00640 1 --2- 172.21.15.90:0/1629961908 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e1c0b2d90 0x7f0e1c0b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:27.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.268+0000 7f0e1a000640 1 --2- 172.21.15.90:0/1629961908 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e1c0b2d90 0x7f0e1c0b47b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:27.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.268+0000 7f0e1b400640 1 --2- 172.21.15.90:0/1629961908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e1c15ded0 0x7f0e1c159d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:27.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.268+0000 7f0e1a000640 1 -- 172.21.15.90:0/1629961908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e1c15ded0 msgr2=0x7f0e1c159d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:27.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.268+0000 7f0e1a000640 1 --2- 172.21.15.90:0/1629961908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e1c15ded0 0x7f0e1c159d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:27.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.268+0000 7f0e1a000640 1 -- 172.21.15.90:0/1629961908 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e1c0b38e0 con 0x7f0e1c0b7170 2024-03-24T01:24:27.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.268+0000 7f0e1b400640 1 --2- 172.21.15.90:0/1629961908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e1c15ded0 0x7f0e1c159d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:27.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.269+0000 7f0e1a000640 1 --2- 172.21.15.90:0/1629961908 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e1c0b7170 0x7f0e1c0b1960 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0e0402b9e0 tx=0x7f0e04004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:27.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.269+0000 7f0e12a00640 1 -- 172.21.15.90:0/1629961908 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e04048070 con 0x7f0e1c0b7170 2024-03-24T01:24:27.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.269+0000 7f0e12a00640 1 -- 172.21.15.90:0/1629961908 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e04030660 con 0x7f0e1c0b7170 2024-03-24T01:24:27.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.269+0000 7f0e21c00640 1 -- 172.21.15.90:0/1629961908 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e1c0b3bc0 con 0x7f0e1c0b7170 2024-03-24T01:24:27.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.269+0000 7f0e12a00640 1 -- 172.21.15.90:0/1629961908 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e04034040 con 0x7f0e1c0b7170 2024-03-24T01:24:27.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.269+0000 7f0e21c00640 1 -- 172.21.15.90:0/1629961908 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e1c0b4180 con 0x7f0e1c0b7170 2024-03-24T01:24:27.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.270+0000 7f0e21c00640 1 -- 172.21.15.90:0/1629961908 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e1c0cdf60 con 0x7f0e1c0b7170 2024-03-24T01:24:27.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.271+0000 7f0e12a00640 1 -- 172.21.15.90:0/1629961908 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0e0404b070 con 0x7f0e1c0b7170 2024-03-24T01:24:27.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.272+0000 7f0e12a00640 1 --2- 172.21.15.90:0/1629961908 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0df806bbd0 0x7f0df806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:27.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.272+0000 7f0e12a00640 1 -- 172.21.15.90:0/1629961908 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f0e040ace70 con 0x7f0e1c0b7170 2024-03-24T01:24:27.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.273+0000 7f0e1aa00640 1 --2- 172.21.15.90:0/1629961908 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0df806bbd0 0x7f0df806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:27.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.274+0000 7f0e1aa00640 1 --2- 172.21.15.90:0/1629961908 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0df806bbd0 0x7f0df806e090 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c156c10 tx=0x7f0e0c00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:27.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.276+0000 7f0e12a00640 1 -- 172.21.15.90:0/1629961908 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0e1c0cdf60 con 0x7f0e1c0b7170 2024-03-24T01:24:27.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.473+0000 7f0e21c00640 1 -- 172.21.15.90:0/1629961908 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0e1c161610 con 0x7f0e1c0b7170 2024-03-24T01:24:27.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.474+0000 7f0e12a00640 1 -- 172.21.15.90:0/1629961908 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0e0407b690 con 0x7f0e1c0b7170 2024-03-24T01:24:27.474 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-24T01:24:27.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.476+0000 7f0e21c00640 1 -- 172.21.15.90:0/1629961908 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0df806bbd0 msgr2=0x7f0df806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:27.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.476+0000 7f0e21c00640 1 --2- 172.21.15.90:0/1629961908 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0df806bbd0 0x7f0df806e090 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c156c10 tx=0x7f0e0c00b040 comp rx=0 tx=0).stop 2024-03-24T01:24:27.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.476+0000 7f0e21c00640 1 -- 172.21.15.90:0/1629961908 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e1c0b7170 msgr2=0x7f0e1c0b1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:27.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.476+0000 7f0e21c00640 1 --2- 172.21.15.90:0/1629961908 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e1c0b7170 0x7f0e1c0b1960 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0e0402b9e0 tx=0x7f0e04004730 comp rx=0 tx=0).stop 2024-03-24T01:24:27.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.476+0000 7f0e21c00640 1 -- 172.21.15.90:0/1629961908 shutdown_connections 2024-03-24T01:24:27.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.476+0000 7f0e21c00640 1 --2- 172.21.15.90:0/1629961908 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0df806bbd0 0x7f0df806e090 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:27.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.476+0000 7f0e21c00640 1 --2- 172.21.15.90:0/1629961908 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e1c0b7170 0x7f0e1c0b1960 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:27.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.476+0000 7f0e21c00640 1 --2- 172.21.15.90:0/1629961908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e1c15ded0 0x7f0e1c159d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:27.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.476+0000 7f0e21c00640 1 --2- 172.21.15.90:0/1629961908 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0e1c0b2d90 0x7f0e1c0b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:27.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.476+0000 7f0e21c00640 1 -- 172.21.15.90:0/1629961908 >> 172.21.15.90:0/1629961908 conn(0x7f0e1c072e20 msgr2=0x7f0e1c15eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:27.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.476+0000 7f0e21c00640 1 -- 172.21.15.90:0/1629961908 shutdown_connections 2024-03-24T01:24:27.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.477+0000 7f0e21c00640 1 -- 172.21.15.90:0/1629961908 wait complete. 2024-03-24T01:24:27.495 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T01:24:27.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.704+0000 7f85b3600640 1 Processor -- start 2024-03-24T01:24:27.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.704+0000 7f85b3600640 1 -- start start 2024-03-24T01:24:27.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.705+0000 7f85b3600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f85ac1452b0 0x7f85ac071990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.705+0000 7f85b3600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f85ac0cf200 0x7f85ac0b3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.706+0000 7f85b3600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85ac0b2890 con 0x7f85ac071ed0 2024-03-24T01:24:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.706+0000 7f85b3600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85ac072330 con 0x7f85ac1452b0 2024-03-24T01:24:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.706+0000 7f85b3600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85ac072990 con 0x7f85ac0cf200 2024-03-24T01:24:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.706+0000 7f85b0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f85ac1452b0 0x7f85ac071990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.706+0000 7f85b0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f85ac1452b0 0x7f85ac071990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40334/0 (socket says 172.21.15.90:40334) 2024-03-24T01:24:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.706+0000 7f85b0c00640 1 -- 172.21.15.90:0/2651605027 learned_addr learned my addr 172.21.15.90:0/2651605027 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.706+0000 7f85b1600640 1 --2- 172.21.15.90:0/2651605027 >> v2:172.21.15.120:3300/0 conn(0x7f85ac0cf200 0x7f85ac0b3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.706+0000 7f85b0c00640 1 -- 172.21.15.90:0/2651605027 >> v2:172.21.15.120:3300/0 conn(0x7f85ac0cf200 msgr2=0x7f85ac0b3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.706+0000 7f85b0c00640 1 --2- 172.21.15.90:0/2651605027 >> v2:172.21.15.120:3300/0 conn(0x7f85ac0cf200 0x7f85ac0b3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.706+0000 7f85b0c00640 1 -- 172.21.15.90:0/2651605027 >> v1:172.21.15.120:6789/0 conn(0x7f85ac071ed0 legacy=0x7f85ac0ceaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.706+0000 7f85b0c00640 1 -- 172.21.15.90:0/2651605027 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ac0b2b70 con 0x7f85ac1452b0 2024-03-24T01:24:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.706+0000 7f85b1600640 1 --2- 172.21.15.90:0/2651605027 >> v2:172.21.15.120:3300/0 conn(0x7f85ac0cf200 0x7f85ac0b3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:27.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.707+0000 7f85b0c00640 1 --2- 172.21.15.90:0/2651605027 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85ac1452b0 0x7f85ac071990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f85ac0af4b0 tx=0x7f85a002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12a74837fe4fdcf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:27.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.708+0000 7f85ab400640 1 -- 172.21.15.90:0/2651605027 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a0035020 con 0x7f85ac1452b0 2024-03-24T01:24:27.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.708+0000 7f85ab400640 1 -- 172.21.15.90:0/2651605027 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a0031870 con 0x7f85ac1452b0 2024-03-24T01:24:27.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.708+0000 7f85ab400640 1 -- 172.21.15.90:0/2651605027 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a0031c40 con 0x7f85ac1452b0 2024-03-24T01:24:27.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.708+0000 7f85b3600640 1 -- 172.21.15.90:0/2651605027 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85ac1452b0 msgr2=0x7f85ac071990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:27.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.708+0000 7f85b3600640 1 --2- 172.21.15.90:0/2651605027 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85ac1452b0 0x7f85ac071990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f85ac0af4b0 tx=0x7f85a002b910 comp rx=0 tx=0).stop 2024-03-24T01:24:27.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.708+0000 7f85b3600640 1 -- 172.21.15.90:0/2651605027 shutdown_connections 2024-03-24T01:24:27.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.708+0000 7f85b3600640 1 --2- 172.21.15.90:0/2651605027 >> v2:172.21.15.120:3300/0 conn(0x7f85ac0cf200 0x7f85ac0b3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:27.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.708+0000 7f85b3600640 1 --2- 172.21.15.90:0/2651605027 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85ac1452b0 0x7f85ac071990 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:27.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.708+0000 7f85b3600640 1 -- 172.21.15.90:0/2651605027 >> 172.21.15.90:0/2651605027 conn(0x7f85ac0cf7e0 msgr2=0x7f85ac0cfbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:27.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.709+0000 7f85b3600640 1 -- 172.21.15.90:0/2651605027 shutdown_connections 2024-03-24T01:24:27.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.709+0000 7f85b3600640 1 -- 172.21.15.90:0/2651605027 wait complete. 2024-03-24T01:24:27.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.710+0000 7f85b3600640 1 Processor -- start 2024-03-24T01:24:27.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.710+0000 7f85b3600640 1 -- start start 2024-03-24T01:24:27.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.711+0000 7f85b3600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85ac071ed0 0x7f85ac14e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:27.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.711+0000 7f85b3600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85ac0cf200 0x7f85ac15a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:27.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.711+0000 7f85b3600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85ac1452b0 0x7f85ac15a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:27.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.711+0000 7f85b3600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ac0b3410 con 0x7f85ac071ed0 2024-03-24T01:24:27.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.711+0000 7f85b3600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ac0b3130 con 0x7f85ac1452b0 2024-03-24T01:24:27.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.711+0000 7f85b3600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85ac145690 con 0x7f85ac0cf200 2024-03-24T01:24:27.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.712+0000 7f85b0c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85ac071ed0 0x7f85ac14e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:27.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.712+0000 7f85b0c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85ac071ed0 0x7f85ac14e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40342/0 (socket says 172.21.15.90:40342) 2024-03-24T01:24:27.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.712+0000 7f85b0c00640 1 -- 172.21.15.90:0/1584592744 learned_addr learned my addr 172.21.15.90:0/1584592744 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:27.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.712+0000 7f85b0c00640 1 -- 172.21.15.90:0/1584592744 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85ac0cf200 msgr2=0x7f85ac15a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:27.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.712+0000 7f85b0c00640 1 --2- 172.21.15.90:0/1584592744 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85ac0cf200 0x7f85ac15a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:27.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.712+0000 7f85b0c00640 1 -- 172.21.15.90:0/1584592744 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85ac1452b0 msgr2=0x7f85ac15a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:27.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.712+0000 7f85b0c00640 1 --2- 172.21.15.90:0/1584592744 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85ac1452b0 0x7f85ac15a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:27.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.712+0000 7f85b0c00640 1 -- 172.21.15.90:0/1584592744 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ac13d770 con 0x7f85ac071ed0 2024-03-24T01:24:27.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.712+0000 7f85b0c00640 1 --2- 172.21.15.90:0/1584592744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85ac071ed0 0x7f85ac14e100 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f85a002b950 tx=0x7f85a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:27.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.713+0000 7f85a8c00640 1 -- 172.21.15.90:0/1584592744 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a0040070 con 0x7f85ac071ed0 2024-03-24T01:24:27.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.713+0000 7f85a8c00640 1 -- 172.21.15.90:0/1584592744 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a003a420 con 0x7f85ac071ed0 2024-03-24T01:24:27.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.713+0000 7f85b3600640 1 -- 172.21.15.90:0/1584592744 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85ac1f4ea0 con 0x7f85ac071ed0 2024-03-24T01:24:27.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.713+0000 7f85a8c00640 1 -- 172.21.15.90:0/1584592744 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a0035040 con 0x7f85ac071ed0 2024-03-24T01:24:27.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.713+0000 7f85b3600640 1 -- 172.21.15.90:0/1584592744 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85ac1f5460 con 0x7f85ac071ed0 2024-03-24T01:24:27.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.714+0000 7f85b3600640 1 -- 172.21.15.90:0/1584592744 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85ac04f400 con 0x7f85ac071ed0 2024-03-24T01:24:27.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.715+0000 7f85a8c00640 1 -- 172.21.15.90:0/1584592744 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f85a004b060 con 0x7f85ac071ed0 2024-03-24T01:24:27.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.716+0000 7f85a8c00640 1 --2- 172.21.15.90:0/1584592744 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f858c06bbd0 0x7f858c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:27.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.716+0000 7f85a8c00640 1 -- 172.21.15.90:0/1584592744 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f85a00acdb0 con 0x7f85ac071ed0 2024-03-24T01:24:27.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.716+0000 7f85abe00640 1 --2- 172.21.15.90:0/1584592744 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f858c06bbd0 0x7f858c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:27.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.717+0000 7f85abe00640 1 --2- 172.21.15.90:0/1584592744 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f858c06bbd0 0x7f858c06e090 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f85ac072d40 tx=0x7f85980079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:27.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.720+0000 7f85a8c00640 1 -- 172.21.15.90:0/1584592744 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f85a007b4c0 con 0x7f85ac071ed0 2024-03-24T01:24:27.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.920+0000 7f85b3600640 1 -- 172.21.15.90:0/1584592744 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f85ac0b0d80 con 0x7f85ac071ed0 2024-03-24T01:24:27.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.920+0000 7f85a8c00640 1 -- 172.21.15.90:0/1584592744 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7f85a007adc0 con 0x7f85ac071ed0 2024-03-24T01:24:27.921 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:24:27.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.923+0000 7f85b3600640 1 -- 172.21.15.90:0/1584592744 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f858c06bbd0 msgr2=0x7f858c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:27.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.923+0000 7f85b3600640 1 --2- 172.21.15.90:0/1584592744 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f858c06bbd0 0x7f858c06e090 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f85ac072d40 tx=0x7f85980079b0 comp rx=0 tx=0).stop 2024-03-24T01:24:27.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.923+0000 7f85b3600640 1 -- 172.21.15.90:0/1584592744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85ac071ed0 msgr2=0x7f85ac14e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:27.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.923+0000 7f85b3600640 1 --2- 172.21.15.90:0/1584592744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85ac071ed0 0x7f85ac14e100 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f85a002b950 tx=0x7f85a00046c0 comp rx=0 tx=0).stop 2024-03-24T01:24:27.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.923+0000 7f85b3600640 1 -- 172.21.15.90:0/1584592744 shutdown_connections 2024-03-24T01:24:27.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.923+0000 7f85b3600640 1 --2- 172.21.15.90:0/1584592744 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f858c06bbd0 0x7f858c06e090 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:27.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.923+0000 7f85b3600640 1 --2- 172.21.15.90:0/1584592744 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85ac1452b0 0x7f85ac15a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:27.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.923+0000 7f85b3600640 1 --2- 172.21.15.90:0/1584592744 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85ac0cf200 0x7f85ac15a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:27.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.923+0000 7f85b3600640 1 --2- 172.21.15.90:0/1584592744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85ac071ed0 0x7f85ac14e100 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:27.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.923+0000 7f85b3600640 1 -- 172.21.15.90:0/1584592744 >> 172.21.15.90:0/1584592744 conn(0x7f85ac0cf7e0 msgr2=0x7f85ac0b0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:27.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.924+0000 7f85b3600640 1 -- 172.21.15.90:0/1584592744 shutdown_connections 2024-03-24T01:24:27.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:27.924+0000 7f85b3600640 1 -- 172.21.15.90:0/1584592744 wait complete. 2024-03-24T01:24:27.943 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T01:24:28.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.144+0000 7f22eee00640 1 Processor -- start 2024-03-24T01:24:28.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.144+0000 7f22eee00640 1 -- start start 2024-03-24T01:24:28.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.145+0000 7f22eee00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f22e806b7d0 0x7f22e81656f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:28.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.145+0000 7f22eee00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22e806dac0 con 0x7f22e815bc50 2024-03-24T01:24:28.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.145+0000 7f22eee00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22e806dda0 con 0x7f22e80aeeb0 2024-03-24T01:24:28.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.145+0000 7f22eee00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22e80b4da0 con 0x7f22e806b7d0 2024-03-24T01:24:28.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.146+0000 7f22e7400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f22e815bc50 0x7f22e806b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:36310/0 (socket says 172.21.15.90:36310) 2024-03-24T01:24:28.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.146+0000 7f22e7400640 1 -- 172.21.15.90:0/1509989771 learned_addr learned my addr 172.21.15.90:0/1509989771 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:28.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.146+0000 7f22ece00640 1 --2- 172.21.15.90:0/1509989771 >> v2:172.21.15.120:3300/0 conn(0x7f22e806b7d0 0x7f22e81656f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:28.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.146+0000 7f22e6a00640 1 -- 172.21.15.90:0/1509989771 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2490956324 0 0) 0x7f22e806dac0 con 0x7f22e815bc50 2024-03-24T01:24:28.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.146+0000 7f22e6a00640 1 -- 172.21.15.90:0/1509989771 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22cc003520 con 0x7f22e815bc50 2024-03-24T01:24:28.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.146+0000 7f22e6a00640 1 -- 172.21.15.90:0/1509989771 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 41811675 0 0) 0x7f22cc003520 con 0x7f22e815bc50 2024-03-24T01:24:28.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.146+0000 7f22e6a00640 1 -- 172.21.15.90:0/1509989771 >> v2:172.21.15.120:3300/0 conn(0x7f22e806b7d0 msgr2=0x7f22e81656f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:28.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.146+0000 7f22e6a00640 1 --2- 172.21.15.90:0/1509989771 >> v2:172.21.15.120:3300/0 conn(0x7f22e806b7d0 0x7f22e81656f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:28.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.146+0000 7f22e6a00640 1 -- 172.21.15.90:0/1509989771 >> v1:172.21.15.120:6789/0 conn(0x7f22e80aeeb0 legacy=0x7f22e80b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:28.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.146+0000 7f22e6a00640 1 -- 172.21.15.90:0/1509989771 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e806e080 con 0x7f22e815bc50 2024-03-24T01:24:28.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.146+0000 7f22e6a00640 1 -- 172.21.15.90:0/1509989771 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f22d4001670 con 0x7f22e815bc50 2024-03-24T01:24:28.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.147+0000 7f22e6a00640 1 -- 172.21.15.90:0/1509989771 >> v1:172.21.15.90:6789/0 conn(0x7f22e815bc50 legacy=0x7f22e806b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:28.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.147+0000 7f22e6a00640 1 --2- 172.21.15.90:0/1509989771 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22cc003a00 0x7f22cc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:28.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.147+0000 7f22e6a00640 1 --2- 172.21.15.90:0/1509989771 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22cc004800 0x7f22cc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:28.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.147+0000 7f22e6a00640 1 --2- 172.21.15.90:0/1509989771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22cc0052e0 0x7f22cc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:28.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.147+0000 7f22e6a00640 1 -- 172.21.15.90:0/1509989771 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e80b4da0 con 0x7f22cc004800 2024-03-24T01:24:28.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.147+0000 7f22e6a00640 1 -- 172.21.15.90:0/1509989771 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22cc00c1e0 con 0x7f22cc0052e0 2024-03-24T01:24:28.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.147+0000 7f22e6a00640 1 -- 172.21.15.90:0/1509989771 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22cc00c3e0 con 0x7f22cc003a00 2024-03-24T01:24:28.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.147+0000 7f22e7400640 1 --2- 172.21.15.90:0/1509989771 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22cc004800 0x7f22cc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:28.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.147+0000 7f22e7e00640 1 --2- 172.21.15.90:0/1509989771 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22cc003a00 0x7f22cc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:28.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.147+0000 7f22ece00640 1 --2- 172.21.15.90:0/1509989771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22cc0052e0 0x7f22cc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:28.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.147+0000 7f22e7400640 1 -- 172.21.15.90:0/1509989771 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22cc003a00 msgr2=0x7f22cc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:28.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.147+0000 7f22e7400640 1 --2- 172.21.15.90:0/1509989771 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22cc003a00 0x7f22cc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:28.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.147+0000 7f22e7400640 1 -- 172.21.15.90:0/1509989771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22cc0052e0 msgr2=0x7f22cc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:28.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.147+0000 7f22e7400640 1 --2- 172.21.15.90:0/1509989771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22cc0052e0 0x7f22cc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:28.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.147+0000 7f22e7400640 1 -- 172.21.15.90:0/1509989771 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22cc003520 con 0x7f22cc004800 2024-03-24T01:24:28.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.147+0000 7f22ece00640 1 --2- 172.21.15.90:0/1509989771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22cc0052e0 0x7f22cc00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:28.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.147+0000 7f22ece00640 1 -- 172.21.15.90:0/1509989771 reap_dead start 2024-03-24T01:24:28.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.148+0000 7f22e7400640 1 --2- 172.21.15.90:0/1509989771 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22cc004800 0x7f22cc004c80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f22d400c110 tx=0x7f22cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59235d89dd1809ec server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:28.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.149+0000 7f22e6a00640 1 -- 172.21.15.90:0/1509989771 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d4007c40 con 0x7f22cc004800 2024-03-24T01:24:28.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.149+0000 7f22e6a00640 1 -- 172.21.15.90:0/1509989771 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22d4030070 con 0x7f22cc004800 2024-03-24T01:24:28.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.149+0000 7f22e6a00640 1 -- 172.21.15.90:0/1509989771 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d4005d70 con 0x7f22cc004800 2024-03-24T01:24:28.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.149+0000 7f22eee00640 1 -- 172.21.15.90:0/1509989771 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22cc004800 msgr2=0x7f22cc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:28.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.149+0000 7f22eee00640 1 --2- 172.21.15.90:0/1509989771 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22cc004800 0x7f22cc004c80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f22d400c110 tx=0x7f22cc005830 comp rx=0 tx=0).stop 2024-03-24T01:24:28.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.149+0000 7f22eee00640 1 -- 172.21.15.90:0/1509989771 shutdown_connections 2024-03-24T01:24:28.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.149+0000 7f22eee00640 1 --2- 172.21.15.90:0/1509989771 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22cc004800 0x7f22cc004c80 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:28.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.149+0000 7f22eee00640 1 -- 172.21.15.90:0/1509989771 >> 172.21.15.90:0/1509989771 conn(0x7f22e80b1770 msgr2=0x7f22e80b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:28.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.149+0000 7f22eee00640 1 -- 172.21.15.90:0/1509989771 shutdown_connections 2024-03-24T01:24:28.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.149+0000 7f22eee00640 1 -- 172.21.15.90:0/1509989771 wait complete. 2024-03-24T01:24:28.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.151+0000 7f22eee00640 1 Processor -- start 2024-03-24T01:24:28.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.151+0000 7f22eee00640 1 -- start start 2024-03-24T01:24:28.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.152+0000 7f22eee00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22cc004800 0x7f22e815eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:28.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.152+0000 7f22eee00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22e81523d0 0x7f22e8153a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:28.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.152+0000 7f22eee00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22e814dae0 0x7f22e815cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:28.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.152+0000 7f22eee00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e806e880 con 0x7f22cc004800 2024-03-24T01:24:28.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.152+0000 7f22eee00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e80af290 con 0x7f22e814dae0 2024-03-24T01:24:28.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.152+0000 7f22eee00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22e806ea00 con 0x7f22e81523d0 2024-03-24T01:24:28.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.152+0000 7f22e7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22cc004800 0x7f22e815eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:28.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.152+0000 7f22e7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22cc004800 0x7f22e815eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40364/0 (socket says 172.21.15.90:40364) 2024-03-24T01:24:28.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.152+0000 7f22e7e00640 1 -- 172.21.15.90:0/1200068730 learned_addr learned my addr 172.21.15.90:0/1200068730 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:28.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.153+0000 7f22e7400640 1 --2- 172.21.15.90:0/1200068730 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22e81523d0 0x7f22e8153a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:28.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.153+0000 7f22e7e00640 1 -- 172.21.15.90:0/1200068730 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22e81523d0 msgr2=0x7f22e8153a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:28.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.153+0000 7f22e7e00640 1 --2- 172.21.15.90:0/1200068730 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22e81523d0 0x7f22e8153a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:28.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.153+0000 7f22e7e00640 1 -- 172.21.15.90:0/1200068730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22e814dae0 msgr2=0x7f22e815cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:28.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.153+0000 7f22e7e00640 1 --2- 172.21.15.90:0/1200068730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22e814dae0 0x7f22e815cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:28.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.153+0000 7f22e7e00640 1 -- 172.21.15.90:0/1200068730 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e815eff0 con 0x7f22cc004800 2024-03-24T01:24:28.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.153+0000 7f22e7e00640 1 --2- 172.21.15.90:0/1200068730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22cc004800 0x7f22e815eab0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f22d800e4c0 tx=0x7f22d800e980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:28.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.153+0000 7f22dfe00640 1 -- 172.21.15.90:0/1200068730 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d80052e0 con 0x7f22cc004800 2024-03-24T01:24:28.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.153+0000 7f22dfe00640 1 -- 172.21.15.90:0/1200068730 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22d8018070 con 0x7f22cc004800 2024-03-24T01:24:28.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.153+0000 7f22dfe00640 1 -- 172.21.15.90:0/1200068730 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d8003830 con 0x7f22cc004800 2024-03-24T01:24:28.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.153+0000 7f22eee00640 1 -- 172.21.15.90:0/1200068730 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22e8152970 con 0x7f22cc004800 2024-03-24T01:24:28.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.154+0000 7f22eee00640 1 -- 172.21.15.90:0/1200068730 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22e8152f30 con 0x7f22cc004800 2024-03-24T01:24:28.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.155+0000 7f22eee00640 1 -- 172.21.15.90:0/1200068730 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22e804f400 con 0x7f22cc004800 2024-03-24T01:24:28.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.156+0000 7f22dfe00640 1 -- 172.21.15.90:0/1200068730 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f22d80039d0 con 0x7f22cc004800 2024-03-24T01:24:28.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.156+0000 7f22dfe00640 1 --2- 172.21.15.90:0/1200068730 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f22c806bb70 0x7f22c806e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:28.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.157+0000 7f22dfe00640 1 -- 172.21.15.90:0/1200068730 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f22d808aa50 con 0x7f22cc004800 2024-03-24T01:24:28.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.157+0000 7f22e7400640 1 --2- 172.21.15.90:0/1200068730 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f22c806bb70 0x7f22c806e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:28.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.158+0000 7f22e7400640 1 --2- 172.21.15.90:0/1200068730 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f22c806bb70 0x7f22c806e030 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f22d4034000 tx=0x7f22d4039040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:28.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.161+0000 7f22dfe00640 1 -- 172.21.15.90:0/1200068730 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f22d80592c0 con 0x7f22cc004800 2024-03-24T01:24:28.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.375+0000 7f22eee00640 1 -- 172.21.15.90:0/1200068730 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f22e8068270 con 0x7f22cc004800 2024-03-24T01:24:28.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.375+0000 7f22dfe00640 1 -- 172.21.15.90:0/1200068730 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) v1 ==== 80+0+0 (secure 0 0 0) 0x7f22d8058bc0 con 0x7f22cc004800 2024-03-24T01:24:28.376 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-24T01:24:28.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.378+0000 7f22eee00640 1 -- 172.21.15.90:0/1200068730 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f22c806bb70 msgr2=0x7f22c806e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:28.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.378+0000 7f22eee00640 1 --2- 172.21.15.90:0/1200068730 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f22c806bb70 0x7f22c806e030 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f22d4034000 tx=0x7f22d4039040 comp rx=0 tx=0).stop 2024-03-24T01:24:28.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.378+0000 7f22eee00640 1 -- 172.21.15.90:0/1200068730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22cc004800 msgr2=0x7f22e815eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:28.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.378+0000 7f22eee00640 1 --2- 172.21.15.90:0/1200068730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22cc004800 0x7f22e815eab0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f22d800e4c0 tx=0x7f22d800e980 comp rx=0 tx=0).stop 2024-03-24T01:24:28.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.378+0000 7f22eee00640 1 -- 172.21.15.90:0/1200068730 shutdown_connections 2024-03-24T01:24:28.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.378+0000 7f22eee00640 1 --2- 172.21.15.90:0/1200068730 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f22c806bb70 0x7f22c806e030 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:28.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.378+0000 7f22eee00640 1 --2- 172.21.15.90:0/1200068730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22e814dae0 0x7f22e815cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:28.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.378+0000 7f22eee00640 1 --2- 172.21.15.90:0/1200068730 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f22e81523d0 0x7f22e8153a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:28.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.378+0000 7f22eee00640 1 --2- 172.21.15.90:0/1200068730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22cc004800 0x7f22e815eab0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:28.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.378+0000 7f22eee00640 1 -- 172.21.15.90:0/1200068730 >> 172.21.15.90:0/1200068730 conn(0x7f22e80b1770 msgr2=0x7f22e8158c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:28.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.379+0000 7f22eee00640 1 -- 172.21.15.90:0/1200068730 shutdown_connections 2024-03-24T01:24:28.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:28.379+0000 7f22eee00640 1 -- 172.21.15.90:0/1200068730 wait complete. 2024-03-24T01:24:28.396 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi090.front.sepia.ceph.com... 2024-03-24T01:24:28.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:28.397 DEBUG:teuthology.orchestra.run.smithi090:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-24T01:24:28.479 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi090.stderr:ceph-fuse[43766]: fuse finished with error 0 and tester_r 0 2024-03-24T01:24:28.486 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:24:34.587 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-03-24T01:24:34.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:34.588 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-24T01:24:34.656 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:24:34.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.872+0000 7fd187200640 1 Processor -- start 2024-03-24T01:24:34.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.872+0000 7fd187200640 1 -- start start 2024-03-24T01:24:34.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.873+0000 7fd187200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd18815ee50 0x7fd1880b2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:34.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.873+0000 7fd187200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1880b2bc0 0x7fd1880cf340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:34.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.873+0000 7fd187200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd18815f300 con 0x7fd1880cef60 2024-03-24T01:24:34.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.873+0000 7fd187200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd18815b980 con 0x7fd18815ee50 2024-03-24T01:24:34.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.873+0000 7fd187200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd18815bc60 con 0x7fd1880b2bc0 2024-03-24T01:24:34.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.874+0000 7fd185e00640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fd1880cef60 0x7fd18815ae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.90:50302/0 (socket says 172.21.15.90:50302) 2024-03-24T01:24:34.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.874+0000 7fd185e00640 1 -- 172.21.15.90:0/1339244860 learned_addr learned my addr 172.21.15.90:0/1339244860 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:34.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.874+0000 7fd17d400640 1 --2- 172.21.15.90:0/1339244860 >> v2:172.21.15.120:3300/0 conn(0x7fd18815ee50 0x7fd1880b2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:34.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.874+0000 7fd186800640 1 --2- 172.21.15.90:0/1339244860 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1880b2bc0 0x7fd1880cf340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:34.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.874+0000 7fd185400640 1 -- 172.21.15.90:0/1339244860 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3658702047 0 0) 0x7fd18815f300 con 0x7fd1880cef60 2024-03-24T01:24:34.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.874+0000 7fd185400640 1 -- 172.21.15.90:0/1339244860 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd16c003520 con 0x7fd1880cef60 2024-03-24T01:24:34.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.875+0000 7fd17d400640 1 -- 172.21.15.90:0/1339244860 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1880b2bc0 msgr2=0x7fd1880cf340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:34.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.875+0000 7fd17d400640 1 --2- 172.21.15.90:0/1339244860 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1880b2bc0 0x7fd1880cf340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:34.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.875+0000 7fd17d400640 1 -- 172.21.15.90:0/1339244860 >> v1:172.21.15.120:6789/0 conn(0x7fd1880cef60 legacy=0x7fd18815ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:34.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.875+0000 7fd17d400640 1 -- 172.21.15.90:0/1339244860 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd18815e650 con 0x7fd18815ee50 2024-03-24T01:24:34.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.876+0000 7fd17d400640 1 --2- 172.21.15.90:0/1339244860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd18815ee50 0x7fd1880b2680 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd1700097a0 tx=0x7fd17002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba95acdc86bb99d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:34.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.876+0000 7fd185400640 1 -- 172.21.15.90:0/1339244860 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd170035020 con 0x7fd18815ee50 2024-03-24T01:24:34.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.877+0000 7fd185400640 1 -- 172.21.15.90:0/1339244860 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd170031830 con 0x7fd18815ee50 2024-03-24T01:24:34.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.877+0000 7fd185400640 1 -- 172.21.15.90:0/1339244860 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd170031c00 con 0x7fd18815ee50 2024-03-24T01:24:34.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.877+0000 7fd187200640 1 -- 172.21.15.90:0/1339244860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd18815ee50 msgr2=0x7fd1880b2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:34.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.877+0000 7fd187200640 1 --2- 172.21.15.90:0/1339244860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd18815ee50 0x7fd1880b2680 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd1700097a0 tx=0x7fd17002b8d0 comp rx=0 tx=0).stop 2024-03-24T01:24:34.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.877+0000 7fd187200640 1 -- 172.21.15.90:0/1339244860 shutdown_connections 2024-03-24T01:24:34.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.877+0000 7fd187200640 1 --2- 172.21.15.90:0/1339244860 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1880b2bc0 0x7fd1880cf340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:34.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.877+0000 7fd187200640 1 --2- 172.21.15.90:0/1339244860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd18815ee50 0x7fd1880b2680 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:34.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.877+0000 7fd187200640 1 -- 172.21.15.90:0/1339244860 >> 172.21.15.90:0/1339244860 conn(0x7fd18806b0c0 msgr2=0x7fd18806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:34.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.878+0000 7fd187200640 1 -- 172.21.15.90:0/1339244860 shutdown_connections 2024-03-24T01:24:34.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.878+0000 7fd187200640 1 -- 172.21.15.90:0/1339244860 wait complete. 2024-03-24T01:24:34.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.879+0000 7fd187200640 1 Processor -- start 2024-03-24T01:24:34.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.879+0000 7fd187200640 1 -- start start 2024-03-24T01:24:34.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.880+0000 7fd187200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1880b2bc0 0x7fd188148ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:34.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.880+0000 7fd187200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1880cef60 0x7fd1880cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:34.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.880+0000 7fd187200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd18815ee50 0x7fd1880b3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:34.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.880+0000 7fd187200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd18815b560 con 0x7fd18815ee50 2024-03-24T01:24:34.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.880+0000 7fd187200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1880cc080 con 0x7fd1880cef60 2024-03-24T01:24:34.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.880+0000 7fd187200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd18815b6e0 con 0x7fd1880b2bc0 2024-03-24T01:24:34.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.881+0000 7fd186800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd18815ee50 0x7fd1880b3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:34.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.881+0000 7fd186800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd18815ee50 0x7fd1880b3010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43388/0 (socket says 172.21.15.90:43388) 2024-03-24T01:24:34.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.881+0000 7fd186800640 1 -- 172.21.15.90:0/2920293565 learned_addr learned my addr 172.21.15.90:0/2920293565 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:34.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.881+0000 7fd185e00640 1 --2- 172.21.15.90:0/2920293565 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1880b2bc0 0x7fd188148ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:34.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.881+0000 7fd17d400640 1 --2- 172.21.15.90:0/2920293565 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1880cef60 0x7fd1880cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:34.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.881+0000 7fd186800640 1 -- 172.21.15.90:0/2920293565 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1880b2bc0 msgr2=0x7fd188148ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:34.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.881+0000 7fd186800640 1 --2- 172.21.15.90:0/2920293565 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1880b2bc0 0x7fd188148ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:34.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.881+0000 7fd186800640 1 -- 172.21.15.90:0/2920293565 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1880cef60 msgr2=0x7fd1880cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:34.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.881+0000 7fd186800640 1 --2- 172.21.15.90:0/2920293565 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1880cef60 0x7fd1880cdcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:34.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.881+0000 7fd186800640 1 -- 172.21.15.90:0/2920293565 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1880ce230 con 0x7fd18815ee50 2024-03-24T01:24:34.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.881+0000 7fd17d400640 1 --2- 172.21.15.90:0/2920293565 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1880cef60 0x7fd1880cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:34.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.881+0000 7fd186800640 1 --2- 172.21.15.90:0/2920293565 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd18815ee50 0x7fd1880b3010 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd18000efc0 tx=0x7fd18000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:34.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.882+0000 7fd17ea00640 1 -- 172.21.15.90:0/2920293565 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd180018020 con 0x7fd18815ee50 2024-03-24T01:24:34.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.882+0000 7fd17ea00640 1 -- 172.21.15.90:0/2920293565 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd180014480 con 0x7fd18815ee50 2024-03-24T01:24:34.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.882+0000 7fd17ea00640 1 -- 172.21.15.90:0/2920293565 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd180014870 con 0x7fd18815ee50 2024-03-24T01:24:34.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.882+0000 7fd187200640 1 -- 172.21.15.90:0/2920293565 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1880b21d0 con 0x7fd18815ee50 2024-03-24T01:24:34.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.882+0000 7fd187200640 1 -- 172.21.15.90:0/2920293565 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1880b16f0 con 0x7fd18815ee50 2024-03-24T01:24:34.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.883+0000 7fd187200640 1 -- 172.21.15.90:0/2920293565 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd18804f400 con 0x7fd18815ee50 2024-03-24T01:24:34.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.884+0000 7fd17ea00640 1 -- 172.21.15.90:0/2920293565 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd180005d10 con 0x7fd18815ee50 2024-03-24T01:24:34.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.885+0000 7fd17ea00640 1 --2- 172.21.15.90:0/2920293565 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd16406bbd0 0x7fd16406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:34.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.885+0000 7fd17ea00640 1 -- 172.21.15.90:0/2920293565 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd18008b830 con 0x7fd18815ee50 2024-03-24T01:24:34.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.885+0000 7fd185e00640 1 --2- 172.21.15.90:0/2920293565 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd16406bbd0 0x7fd16406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:34.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.886+0000 7fd185e00640 1 --2- 172.21.15.90:0/2920293565 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd16406bbd0 0x7fd16406e090 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd1780011e0 tx=0x7fd178009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:34.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:34.890+0000 7fd17ea00640 1 -- 172.21.15.90:0/2920293565 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd180022070 con 0x7fd18815ee50 2024-03-24T01:24:35.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.087+0000 7fd187200640 1 -- 172.21.15.90:0/2920293565 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd18806f9f0 con 0x7fd18815ee50 2024-03-24T01:24:35.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.089+0000 7fd17ea00640 1 -- 172.21.15.90:0/2920293565 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18490 (secure 0 0 0) 0x7fd1880ce230 con 0x7fd18815ee50 2024-03-24T01:24:35.090 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:24:35.090 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:24:17.008224+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:24:08.643966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T01:24:09.189904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T01:24:13.027323+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:24:35.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.092+0000 7fd187200640 1 -- 172.21.15.90:0/2920293565 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd16406bbd0 msgr2=0x7fd16406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:35.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.092+0000 7fd187200640 1 --2- 172.21.15.90:0/2920293565 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd16406bbd0 0x7fd16406e090 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd1780011e0 tx=0x7fd178009040 comp rx=0 tx=0).stop 2024-03-24T01:24:35.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.092+0000 7fd187200640 1 -- 172.21.15.90:0/2920293565 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd18815ee50 msgr2=0x7fd1880b3010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:35.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.092+0000 7fd187200640 1 --2- 172.21.15.90:0/2920293565 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd18815ee50 0x7fd1880b3010 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd18000efc0 tx=0x7fd18000c490 comp rx=0 tx=0).stop 2024-03-24T01:24:35.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.092+0000 7fd187200640 1 -- 172.21.15.90:0/2920293565 shutdown_connections 2024-03-24T01:24:35.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.092+0000 7fd187200640 1 --2- 172.21.15.90:0/2920293565 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd16406bbd0 0x7fd16406e090 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:35.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.092+0000 7fd187200640 1 --2- 172.21.15.90:0/2920293565 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd18815ee50 0x7fd1880b3010 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:35.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.092+0000 7fd187200640 1 --2- 172.21.15.90:0/2920293565 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1880cef60 0x7fd1880cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:35.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.092+0000 7fd187200640 1 --2- 172.21.15.90:0/2920293565 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd1880b2bc0 0x7fd188148ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:35.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.092+0000 7fd187200640 1 -- 172.21.15.90:0/2920293565 >> 172.21.15.90:0/2920293565 conn(0x7fd18806b0c0 msgr2=0x7fd18815ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:35.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.093+0000 7fd187200640 1 -- 172.21.15.90:0/2920293565 shutdown_connections 2024-03-24T01:24:35.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.093+0000 7fd187200640 1 -- 172.21.15.90:0/2920293565 wait complete. 2024-03-24T01:24:35.111 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:24:35.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.320+0000 7fbda1a00640 1 Processor -- start 2024-03-24T01:24:35.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.320+0000 7fbda1a00640 1 -- start start 2024-03-24T01:24:35.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.321+0000 7fbda1a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fbd9c0b3310 0x7fbd9c1658a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:35.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.322+0000 7fbda1a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd9c1447b0 con 0x7fbd9c1443d0 2024-03-24T01:24:35.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.322+0000 7fbda1a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd9c15b110 con 0x7fbd9c0cd5c0 2024-03-24T01:24:35.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.322+0000 7fbda1a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd9c15aef0 con 0x7fbd9c0b3310 2024-03-24T01:24:35.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.322+0000 7fbd9aa00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fbd9c1443d0 0x7fbd9c0cceb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:32906/0 (socket says 172.21.15.90:32906) 2024-03-24T01:24:35.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.322+0000 7fbd9aa00640 1 -- 172.21.15.90:0/4048258156 learned_addr learned my addr 172.21.15.90:0/4048258156 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:35.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.322+0000 7fbd9b400640 1 --2- 172.21.15.90:0/4048258156 >> v2:172.21.15.90:3300/0 conn(0x7fbd9c0b3310 0x7fbd9c1658a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:35.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.322+0000 7fbd9a000640 1 -- 172.21.15.90:0/4048258156 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1571061686 0 0) 0x7fbd9c1447b0 con 0x7fbd9c1443d0 2024-03-24T01:24:35.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.322+0000 7fbd9a000640 1 -- 172.21.15.90:0/4048258156 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd80005730 con 0x7fbd9c1443d0 2024-03-24T01:24:35.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.322+0000 7fbd9b400640 1 -- 172.21.15.90:0/4048258156 >> v1:172.21.15.120:6789/0 conn(0x7fbd9c0cd5c0 legacy=0x7fbd9c0cec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:35.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.322+0000 7fbd9b400640 1 -- 172.21.15.90:0/4048258156 >> v1:172.21.15.90:6789/0 conn(0x7fbd9c1443d0 legacy=0x7fbd9c0cceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:35.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.322+0000 7fbd9b400640 1 -- 172.21.15.90:0/4048258156 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd9c15b3f0 con 0x7fbd9c0b3310 2024-03-24T01:24:35.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.323+0000 7fbd9b400640 1 --2- 172.21.15.90:0/4048258156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd9c0b3310 0x7fbd9c1658a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fbd94009d90 tx=0x7fbd9402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95785b77cb9b3d8d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:35.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.324+0000 7fbd9a000640 1 -- 172.21.15.90:0/4048258156 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd94039070 con 0x7fbd9c0b3310 2024-03-24T01:24:35.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.324+0000 7fbd9a000640 1 -- 172.21.15.90:0/4048258156 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd9402cc60 con 0x7fbd9c0b3310 2024-03-24T01:24:35.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.324+0000 7fbd9a000640 1 -- 172.21.15.90:0/4048258156 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd94034660 con 0x7fbd9c0b3310 2024-03-24T01:24:35.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.324+0000 7fbda1a00640 1 -- 172.21.15.90:0/4048258156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd9c0b3310 msgr2=0x7fbd9c1658a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:35.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.324+0000 7fbda1a00640 1 --2- 172.21.15.90:0/4048258156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd9c0b3310 0x7fbd9c1658a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fbd94009d90 tx=0x7fbd9402beb0 comp rx=0 tx=0).stop 2024-03-24T01:24:35.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.324+0000 7fbda1a00640 1 -- 172.21.15.90:0/4048258156 shutdown_connections 2024-03-24T01:24:35.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.324+0000 7fbda1a00640 1 --2- 172.21.15.90:0/4048258156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd9c0b3310 0x7fbd9c1658a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:35.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.324+0000 7fbda1a00640 1 -- 172.21.15.90:0/4048258156 >> 172.21.15.90:0/4048258156 conn(0x7fbd9c06b2a0 msgr2=0x7fbd9c06b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:35.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.324+0000 7fbda1a00640 1 -- 172.21.15.90:0/4048258156 shutdown_connections 2024-03-24T01:24:35.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.325+0000 7fbda1a00640 1 -- 172.21.15.90:0/4048258156 wait complete. 2024-03-24T01:24:35.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.326+0000 7fbda1a00640 1 Processor -- start 2024-03-24T01:24:35.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.326+0000 7fbda1a00640 1 -- start start 2024-03-24T01:24:35.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.327+0000 7fbda1a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd9c0b3310 0x7fbd9c1f3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:35.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.327+0000 7fbda1a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd9c0cd5c0 0x7fbd9c13f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:35.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.327+0000 7fbda1a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd9c1443d0 0x7fbd9c140f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:35.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.327+0000 7fbda1a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c15bc90 con 0x7fbd9c0b3310 2024-03-24T01:24:35.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.327+0000 7fbda1a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c15b9b0 con 0x7fbd9c0cd5c0 2024-03-24T01:24:35.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.327+0000 7fbda1a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c0cf3a0 con 0x7fbd9c1443d0 2024-03-24T01:24:35.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.327+0000 7fbd9aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd9c0b3310 0x7fbd9c1f3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:35.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.328+0000 7fbd9aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd9c0b3310 0x7fbd9c1f3b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43406/0 (socket says 172.21.15.90:43406) 2024-03-24T01:24:35.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.328+0000 7fbd9aa00640 1 -- 172.21.15.90:0/1630840210 learned_addr learned my addr 172.21.15.90:0/1630840210 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:35.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.328+0000 7fbd92000640 1 --2- 172.21.15.90:0/1630840210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd9c0cd5c0 0x7fbd9c13f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:35.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.328+0000 7fbd9aa00640 1 -- 172.21.15.90:0/1630840210 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd9c1443d0 msgr2=0x7fbd9c140f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:35.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.328+0000 7fbd9aa00640 1 --2- 172.21.15.90:0/1630840210 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd9c1443d0 0x7fbd9c140f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:35.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.328+0000 7fbd9aa00640 1 -- 172.21.15.90:0/1630840210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd9c0cd5c0 msgr2=0x7fbd9c13f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:35.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.328+0000 7fbd9aa00640 1 --2- 172.21.15.90:0/1630840210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd9c0cd5c0 0x7fbd9c13f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:35.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.328+0000 7fbd9aa00640 1 -- 172.21.15.90:0/1630840210 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd9c13d860 con 0x7fbd9c0b3310 2024-03-24T01:24:35.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.328+0000 7fbd92000640 1 --2- 172.21.15.90:0/1630840210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd9c0cd5c0 0x7fbd9c13f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:35.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.328+0000 7fbd9aa00640 1 --2- 172.21.15.90:0/1630840210 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd9c0b3310 0x7fbd9c1f3b80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c00c410 tx=0x7fbd8c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:35.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.328+0000 7fbd93400640 1 -- 172.21.15.90:0/1630840210 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c017070 con 0x7fbd9c0b3310 2024-03-24T01:24:35.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.329+0000 7fbd93400640 1 -- 172.21.15.90:0/1630840210 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd8c005ce0 con 0x7fbd9c0b3310 2024-03-24T01:24:35.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.329+0000 7fbd93400640 1 -- 172.21.15.90:0/1630840210 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c0072f0 con 0x7fbd9c0b3310 2024-03-24T01:24:35.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.329+0000 7fbda1a00640 1 -- 172.21.15.90:0/1630840210 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd9c13db40 con 0x7fbd9c0b3310 2024-03-24T01:24:35.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.329+0000 7fbda1a00640 1 -- 172.21.15.90:0/1630840210 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd9c13e100 con 0x7fbd9c0b3310 2024-03-24T01:24:35.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.330+0000 7fbda1a00640 1 -- 172.21.15.90:0/1630840210 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd9c073570 con 0x7fbd9c0b3310 2024-03-24T01:24:35.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.331+0000 7fbd93400640 1 -- 172.21.15.90:0/1630840210 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fbd8c01a070 con 0x7fbd9c0b3310 2024-03-24T01:24:35.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.332+0000 7fbd93400640 1 --2- 172.21.15.90:0/1630840210 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbd7806bbd0 0x7fbd7806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:35.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.332+0000 7fbd93400640 1 -- 172.21.15.90:0/1630840210 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fbd8c08a2b0 con 0x7fbd9c0b3310 2024-03-24T01:24:35.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.332+0000 7fbd92000640 1 --2- 172.21.15.90:0/1630840210 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbd7806bbd0 0x7fbd7806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:35.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.333+0000 7fbd92000640 1 --2- 172.21.15.90:0/1630840210 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbd7806bbd0 0x7fbd7806e090 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c1f7660 tx=0x7fbd84009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:35.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.336+0000 7fbd93400640 1 -- 172.21.15.90:0/1630840210 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fbd8c058a80 con 0x7fbd9c0b3310 2024-03-24T01:24:35.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.536+0000 7fbda1a00640 1 -- 172.21.15.90:0/1630840210 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd9c04f4e0 con 0x7fbd9c0b3310 2024-03-24T01:24:35.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.538+0000 7fbd93400640 1 -- 172.21.15.90:0/1630840210 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18490 (secure 0 0 0) 0x7fbd8c058380 con 0x7fbd9c0b3310 2024-03-24T01:24:35.539 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:24:35.539 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:24:17.008224+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:24:08.643966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T01:24:09.189904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T01:24:13.027323+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:24:35.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.541+0000 7fbda1a00640 1 -- 172.21.15.90:0/1630840210 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbd7806bbd0 msgr2=0x7fbd7806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:35.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.541+0000 7fbda1a00640 1 --2- 172.21.15.90:0/1630840210 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbd7806bbd0 0x7fbd7806e090 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c1f7660 tx=0x7fbd84009040 comp rx=0 tx=0).stop 2024-03-24T01:24:35.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.541+0000 7fbda1a00640 1 -- 172.21.15.90:0/1630840210 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd9c0b3310 msgr2=0x7fbd9c1f3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:35.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.541+0000 7fbda1a00640 1 --2- 172.21.15.90:0/1630840210 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd9c0b3310 0x7fbd9c1f3b80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c00c410 tx=0x7fbd8c00c8d0 comp rx=0 tx=0).stop 2024-03-24T01:24:35.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.541+0000 7fbda1a00640 1 -- 172.21.15.90:0/1630840210 shutdown_connections 2024-03-24T01:24:35.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.541+0000 7fbda1a00640 1 --2- 172.21.15.90:0/1630840210 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbd7806bbd0 0x7fbd7806e090 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:35.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.541+0000 7fbda1a00640 1 --2- 172.21.15.90:0/1630840210 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbd9c1443d0 0x7fbd9c140f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:35.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.541+0000 7fbda1a00640 1 --2- 172.21.15.90:0/1630840210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd9c0cd5c0 0x7fbd9c13f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:35.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.541+0000 7fbda1a00640 1 --2- 172.21.15.90:0/1630840210 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd9c0b3310 0x7fbd9c1f3b80 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:35.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.541+0000 7fbda1a00640 1 -- 172.21.15.90:0/1630840210 >> 172.21.15.90:0/1630840210 conn(0x7fbd9c06b2a0 msgr2=0x7fbd9c06b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:35.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.541+0000 7fbda1a00640 1 -- 172.21.15.90:0/1630840210 shutdown_connections 2024-03-24T01:24:35.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.541+0000 7fbda1a00640 1 -- 172.21.15.90:0/1630840210 wait complete. 2024-03-24T01:24:35.559 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-24T01:24:35.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.770+0000 7f1c5f200640 1 Processor -- start 2024-03-24T01:24:35.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.770+0000 7f1c5f200640 1 -- start start 2024-03-24T01:24:35.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.771+0000 7f1c5f200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1c5806ecf0 0x7f1c580b3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:35.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.772+0000 7f1c5f200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1c5806f0d0 0x7f1c5815af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:35.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.772+0000 7f1c5f200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c580b3670 con 0x7f1c5815b4c0 2024-03-24T01:24:35.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.772+0000 7f1c5f200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c5815b9d0 con 0x7f1c5806ecf0 2024-03-24T01:24:35.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.772+0000 7f1c5f200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c5815bcb0 con 0x7f1c5806f0d0 2024-03-24T01:24:35.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.772+0000 7f1c5de00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1c5806ecf0 0x7f1c580b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:35.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.772+0000 7f1c5e800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f1c5815b4c0 0x7f1c581549f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:32920/0 (socket says 172.21.15.90:32920) 2024-03-24T01:24:35.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.772+0000 7f1c5e800640 1 -- 172.21.15.90:0/581960640 learned_addr learned my addr 172.21.15.90:0/581960640 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:35.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.772+0000 7f1c5d400640 1 --2- 172.21.15.90:0/581960640 >> v2:172.21.15.120:3300/0 conn(0x7f1c5806f0d0 0x7f1c5815af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:35.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.772+0000 7f1c5ca00640 1 -- 172.21.15.90:0/581960640 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3399059417 0 0) 0x7f1c580b3670 con 0x7f1c5815b4c0 2024-03-24T01:24:35.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.773+0000 7f1c5ca00640 1 -- 172.21.15.90:0/581960640 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c40003520 con 0x7f1c5815b4c0 2024-03-24T01:24:35.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.773+0000 7f1c5de00640 1 -- 172.21.15.90:0/581960640 >> v2:172.21.15.120:3300/0 conn(0x7f1c5806f0d0 msgr2=0x7f1c5815af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:35.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.773+0000 7f1c5de00640 1 --2- 172.21.15.90:0/581960640 >> v2:172.21.15.120:3300/0 conn(0x7f1c5806f0d0 0x7f1c5815af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:35.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.773+0000 7f1c5de00640 1 -- 172.21.15.90:0/581960640 >> v1:172.21.15.90:6789/0 conn(0x7f1c5815b4c0 legacy=0x7f1c581549f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:35.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.773+0000 7f1c5de00640 1 -- 172.21.15.90:0/581960640 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c580b2930 con 0x7f1c5806ecf0 2024-03-24T01:24:35.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.774+0000 7f1c5de00640 1 --2- 172.21.15.90:0/581960640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c5806ecf0 0x7f1c580b3130 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1c580b4040 tx=0x7f1c4c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f05e33beafb93942 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:35.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.775+0000 7f1c5ca00640 1 -- 172.21.15.90:0/581960640 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c4c035020 con 0x7f1c5806ecf0 2024-03-24T01:24:35.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.775+0000 7f1c5ca00640 1 -- 172.21.15.90:0/581960640 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c4c031830 con 0x7f1c5806ecf0 2024-03-24T01:24:35.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.775+0000 7f1c5ca00640 1 -- 172.21.15.90:0/581960640 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c4c031be0 con 0x7f1c5806ecf0 2024-03-24T01:24:35.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.775+0000 7f1c5f200640 1 -- 172.21.15.90:0/581960640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c5806ecf0 msgr2=0x7f1c580b3130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:35.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.775+0000 7f1c5f200640 1 --2- 172.21.15.90:0/581960640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c5806ecf0 0x7f1c580b3130 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1c580b4040 tx=0x7f1c4c02b8d0 comp rx=0 tx=0).stop 2024-03-24T01:24:35.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.775+0000 7f1c5f200640 1 -- 172.21.15.90:0/581960640 shutdown_connections 2024-03-24T01:24:35.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.775+0000 7f1c5f200640 1 --2- 172.21.15.90:0/581960640 >> v2:172.21.15.120:3300/0 conn(0x7f1c5806f0d0 0x7f1c5815af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:35.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.775+0000 7f1c5f200640 1 --2- 172.21.15.90:0/581960640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c5806ecf0 0x7f1c580b3130 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:35.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.775+0000 7f1c5f200640 1 -- 172.21.15.90:0/581960640 >> 172.21.15.90:0/581960640 conn(0x7f1c5806b0c0 msgr2=0x7f1c5806b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:35.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.775+0000 7f1c5f200640 1 -- 172.21.15.90:0/581960640 shutdown_connections 2024-03-24T01:24:35.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.775+0000 7f1c5f200640 1 -- 172.21.15.90:0/581960640 wait complete. 2024-03-24T01:24:35.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.777+0000 7f1c5f200640 1 Processor -- start 2024-03-24T01:24:35.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.777+0000 7f1c5f200640 1 -- start start 2024-03-24T01:24:35.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.778+0000 7f1c5f200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c5806ecf0 0x7f1c580b19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:35.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.778+0000 7f1c5f200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c5806f0d0 0x7f1c5814dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:35.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.778+0000 7f1c5f200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c5815b4c0 0x7f1c58150440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:35.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.778+0000 7f1c5f200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c580b4720 con 0x7f1c5815b4c0 2024-03-24T01:24:35.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.778+0000 7f1c5f200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c580b45a0 con 0x7f1c5806f0d0 2024-03-24T01:24:35.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.778+0000 7f1c5f200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c58155100 con 0x7f1c5806ecf0 2024-03-24T01:24:35.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.778+0000 7f1c5e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c5815b4c0 0x7f1c58150440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:35.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.779+0000 7f1c5e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c5815b4c0 0x7f1c58150440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43430/0 (socket says 172.21.15.90:43430) 2024-03-24T01:24:35.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.779+0000 7f1c5e800640 1 -- 172.21.15.90:0/2917132799 learned_addr learned my addr 172.21.15.90:0/2917132799 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:35.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.779+0000 7f1c5d400640 1 --2- 172.21.15.90:0/2917132799 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c5806f0d0 0x7f1c5814dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:35.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.779+0000 7f1c5de00640 1 --2- 172.21.15.90:0/2917132799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c5806ecf0 0x7f1c580b19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:35.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.779+0000 7f1c5e800640 1 -- 172.21.15.90:0/2917132799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c5806ecf0 msgr2=0x7f1c580b19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:35.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.779+0000 7f1c5e800640 1 --2- 172.21.15.90:0/2917132799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c5806ecf0 0x7f1c580b19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:35.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.779+0000 7f1c5e800640 1 -- 172.21.15.90:0/2917132799 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c5806f0d0 msgr2=0x7f1c5814dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:35.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.779+0000 7f1c5e800640 1 --2- 172.21.15.90:0/2917132799 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c5806f0d0 0x7f1c5814dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:35.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.779+0000 7f1c5e800640 1 -- 172.21.15.90:0/2917132799 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c580ae0b0 con 0x7f1c5815b4c0 2024-03-24T01:24:35.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.779+0000 7f1c5de00640 1 --2- 172.21.15.90:0/2917132799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c5806ecf0 0x7f1c580b19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:35.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.779+0000 7f1c5d400640 1 --2- 172.21.15.90:0/2917132799 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c5806f0d0 0x7f1c5814dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:35.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.779+0000 7f1c5e800640 1 --2- 172.21.15.90:0/2917132799 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c5815b4c0 0x7f1c58150440 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1c5400cad0 tx=0x7f1c5400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:35.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.779+0000 7f1c52000640 1 -- 172.21.15.90:0/2917132799 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c54017070 con 0x7f1c5815b4c0 2024-03-24T01:24:35.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.780+0000 7f1c52000640 1 -- 172.21.15.90:0/2917132799 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c5400ac30 con 0x7f1c5815b4c0 2024-03-24T01:24:35.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.780+0000 7f1c52000640 1 -- 172.21.15.90:0/2917132799 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c54012400 con 0x7f1c5815b4c0 2024-03-24T01:24:35.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.780+0000 7f1c5f200640 1 -- 172.21.15.90:0/2917132799 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c580b0120 con 0x7f1c5815b4c0 2024-03-24T01:24:35.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.780+0000 7f1c5f200640 1 -- 172.21.15.90:0/2917132799 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c580b06e0 con 0x7f1c5815b4c0 2024-03-24T01:24:35.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.781+0000 7f1c5f200640 1 -- 172.21.15.90:0/2917132799 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c5804f400 con 0x7f1c5815b4c0 2024-03-24T01:24:35.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.782+0000 7f1c52000640 1 -- 172.21.15.90:0/2917132799 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1c5401a070 con 0x7f1c5815b4c0 2024-03-24T01:24:35.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.783+0000 7f1c52000640 1 --2- 172.21.15.90:0/2917132799 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1c3c06bbd0 0x7f1c3c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:35.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.783+0000 7f1c52000640 1 -- 172.21.15.90:0/2917132799 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f1c5408ac00 con 0x7f1c5815b4c0 2024-03-24T01:24:35.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.783+0000 7f1c5de00640 1 --2- 172.21.15.90:0/2917132799 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1c3c06bbd0 0x7f1c3c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:35.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.784+0000 7f1c5de00640 1 --2- 172.21.15.90:0/2917132799 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1c3c06bbd0 0x7f1c3c06e090 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c036000 tx=0x7f1c4c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:35.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:35.787+0000 7f1c52000640 1 -- 172.21.15.90:0/2917132799 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1c54059310 con 0x7f1c5815b4c0 2024-03-24T01:24:36.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.019+0000 7f1c5f200640 1 -- 172.21.15.90:0/2917132799 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f1c580d0670 con 0x7f1c5815b4c0 2024-03-24T01:24:36.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.111+0000 7f1c52000640 1 -- 172.21.15.90:0/2917132799 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1c54058c10 con 0x7f1c5815b4c0 2024-03-24T01:24:36.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.115+0000 7f1c5f200640 1 -- 172.21.15.90:0/2917132799 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1c3c06bbd0 msgr2=0x7f1c3c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:36.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.115+0000 7f1c5f200640 1 --2- 172.21.15.90:0/2917132799 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1c3c06bbd0 0x7f1c3c06e090 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c036000 tx=0x7f1c4c008030 comp rx=0 tx=0).stop 2024-03-24T01:24:36.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.115+0000 7f1c5f200640 1 -- 172.21.15.90:0/2917132799 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c5815b4c0 msgr2=0x7f1c58150440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:36.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.115+0000 7f1c5f200640 1 --2- 172.21.15.90:0/2917132799 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c5815b4c0 0x7f1c58150440 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1c5400cad0 tx=0x7f1c5400cf90 comp rx=0 tx=0).stop 2024-03-24T01:24:36.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.115+0000 7f1c5f200640 1 -- 172.21.15.90:0/2917132799 shutdown_connections 2024-03-24T01:24:36.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.115+0000 7f1c5f200640 1 --2- 172.21.15.90:0/2917132799 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1c3c06bbd0 0x7f1c3c06e090 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:36.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.115+0000 7f1c5f200640 1 --2- 172.21.15.90:0/2917132799 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c5815b4c0 0x7f1c58150440 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:36.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.115+0000 7f1c5f200640 1 --2- 172.21.15.90:0/2917132799 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c5806f0d0 0x7f1c5814dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:36.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.115+0000 7f1c5f200640 1 --2- 172.21.15.90:0/2917132799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c5806ecf0 0x7f1c580b19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:36.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.115+0000 7f1c5f200640 1 -- 172.21.15.90:0/2917132799 >> 172.21.15.90:0/2917132799 conn(0x7f1c5806b0c0 msgr2=0x7f1c5806b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:36.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.115+0000 7f1c5f200640 1 -- 172.21.15.90:0/2917132799 shutdown_connections 2024-03-24T01:24:36.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.115+0000 7f1c5f200640 1 -- 172.21.15.90:0/2917132799 wait complete. 2024-03-24T01:24:36.134 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2024-03-24T01:24:36.134 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-24T01:24:36.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.353+0000 7f574ae00640 1 Processor -- start 2024-03-24T01:24:36.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.353+0000 7f574ae00640 1 -- start start 2024-03-24T01:24:36.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.355+0000 7f574ae00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f57440b2e40 0x7f5744168c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:36.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.355+0000 7f574ae00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57440d20e0 con 0x7f57440d0b50 2024-03-24T01:24:36.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.355+0000 7f574ae00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57440ced70 con 0x7f574415dfd0 2024-03-24T01:24:36.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.355+0000 7f574ae00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57440cf050 con 0x7f57440b2e40 2024-03-24T01:24:36.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.355+0000 7f5743e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f57440d0b50 0x7f57440d0f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:32922/0 (socket says 172.21.15.90:32922) 2024-03-24T01:24:36.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.355+0000 7f5743e00640 1 -- 172.21.15.90:0/3809540694 learned_addr learned my addr 172.21.15.90:0/3809540694 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:36.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.355+0000 7f5742a00640 1 -- 172.21.15.90:0/3809540694 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3903136126 0 0) 0x7f57440d20e0 con 0x7f57440d0b50 2024-03-24T01:24:36.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.355+0000 7f5748e00640 1 --2- 172.21.15.90:0/3809540694 >> v2:172.21.15.120:3300/0 conn(0x7f57440b2e40 0x7f5744168c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:36.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.355+0000 7f5742a00640 1 -- 172.21.15.90:0/3809540694 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5728003520 con 0x7f57440d0b50 2024-03-24T01:24:36.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.356+0000 7f5742a00640 1 -- 172.21.15.90:0/3809540694 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1062770998 0 0) 0x7f5728003520 con 0x7f57440d0b50 2024-03-24T01:24:36.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.356+0000 7f5742a00640 1 -- 172.21.15.90:0/3809540694 >> v2:172.21.15.120:3300/0 conn(0x7f57440b2e40 msgr2=0x7f5744168c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:36.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.356+0000 7f5742a00640 1 --2- 172.21.15.90:0/3809540694 >> v2:172.21.15.120:3300/0 conn(0x7f57440b2e40 0x7f5744168c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:36.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.356+0000 7f5742a00640 1 -- 172.21.15.90:0/3809540694 >> v1:172.21.15.120:6789/0 conn(0x7f574415dfd0 legacy=0x7f57441622b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:36.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.356+0000 7f5742a00640 1 -- 172.21.15.90:0/3809540694 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57440d2380 con 0x7f57440d0b50 2024-03-24T01:24:36.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.356+0000 7f5742a00640 1 -- 172.21.15.90:0/3809540694 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f5734001670 con 0x7f57440d0b50 2024-03-24T01:24:36.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.356+0000 7f5742a00640 1 -- 172.21.15.90:0/3809540694 >> v1:172.21.15.90:6789/0 conn(0x7f57440d0b50 legacy=0x7f57440d0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:36.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.356+0000 7f5742a00640 1 --2- 172.21.15.90:0/3809540694 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5728003a00 0x7f5728003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:36.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.356+0000 7f5742a00640 1 --2- 172.21.15.90:0/3809540694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5728004800 0x7f5728004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:36.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.356+0000 7f5742a00640 1 --2- 172.21.15.90:0/3809540694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57280052e0 0x7f572800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:36.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.356+0000 7f5742a00640 1 -- 172.21.15.90:0/3809540694 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57440cf050 con 0x7f5728004800 2024-03-24T01:24:36.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.356+0000 7f5742a00640 1 -- 172.21.15.90:0/3809540694 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f572800c1e0 con 0x7f57280052e0 2024-03-24T01:24:36.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.356+0000 7f5742a00640 1 -- 172.21.15.90:0/3809540694 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f572800c3e0 con 0x7f5728003a00 2024-03-24T01:24:36.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.356+0000 7f5743400640 1 --2- 172.21.15.90:0/3809540694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5728004800 0x7f5728004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:36.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.357+0000 7f5743400640 1 -- 172.21.15.90:0/3809540694 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5728003a00 msgr2=0x7f5728003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:36.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.357+0000 7f5743e00640 1 --2- 172.21.15.90:0/3809540694 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5728003a00 0x7f5728003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:36.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.357+0000 7f5743400640 1 --2- 172.21.15.90:0/3809540694 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5728003a00 0x7f5728003e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:36.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.357+0000 7f5743400640 1 -- 172.21.15.90:0/3809540694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57280052e0 msgr2=0x7f572800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:36.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.357+0000 7f5743400640 1 --2- 172.21.15.90:0/3809540694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57280052e0 0x7f572800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:36.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.357+0000 7f5743400640 1 -- 172.21.15.90:0/3809540694 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5728003520 con 0x7f5728004800 2024-03-24T01:24:36.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.357+0000 7f5748e00640 1 -- 172.21.15.90:0/3809540694 reap_dead start 2024-03-24T01:24:36.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.358+0000 7f5743400640 1 --2- 172.21.15.90:0/3809540694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5728004800 0x7f5728004c80 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5730001750 tx=0x7f5728005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d76b843bc9c25acd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:36.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.358+0000 7f5742a00640 1 -- 172.21.15.90:0/3809540694 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5730001370 con 0x7f5728004800 2024-03-24T01:24:36.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.358+0000 7f5742a00640 1 -- 172.21.15.90:0/3809540694 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57300052d0 con 0x7f5728004800 2024-03-24T01:24:36.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.358+0000 7f5742a00640 1 -- 172.21.15.90:0/3809540694 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5730003830 con 0x7f5728004800 2024-03-24T01:24:36.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.358+0000 7f574ae00640 1 -- 172.21.15.90:0/3809540694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5728004800 msgr2=0x7f5728004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:36.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.358+0000 7f574ae00640 1 --2- 172.21.15.90:0/3809540694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5728004800 0x7f5728004c80 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5730001750 tx=0x7f5728005830 comp rx=0 tx=0).stop 2024-03-24T01:24:36.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.359+0000 7f574ae00640 1 -- 172.21.15.90:0/3809540694 shutdown_connections 2024-03-24T01:24:36.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.359+0000 7f574ae00640 1 --2- 172.21.15.90:0/3809540694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5728004800 0x7f5728004c80 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:36.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.359+0000 7f574ae00640 1 -- 172.21.15.90:0/3809540694 >> 172.21.15.90:0/3809540694 conn(0x7f5744072c10 msgr2=0x7f5744073020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:36.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.359+0000 7f574ae00640 1 -- 172.21.15.90:0/3809540694 shutdown_connections 2024-03-24T01:24:36.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.359+0000 7f574ae00640 1 -- 172.21.15.90:0/3809540694 wait complete. 2024-03-24T01:24:36.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.360+0000 7f574ae00640 1 Processor -- start 2024-03-24T01:24:36.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.360+0000 7f574ae00640 1 -- start start 2024-03-24T01:24:36.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.361+0000 7f574ae00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5728004800 0x7f5744157030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:36.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.361+0000 7f574ae00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57441f3c60 0x7f57441569a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:36.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.361+0000 7f574ae00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57440cb6c0 0x7f57440cd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:36.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.361+0000 7f574ae00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57440b6f00 con 0x7f57441f3c60 2024-03-24T01:24:36.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.361+0000 7f574ae00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57440b6d80 con 0x7f5728004800 2024-03-24T01:24:36.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.361+0000 7f574ae00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57440d0f30 con 0x7f57440cb6c0 2024-03-24T01:24:36.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.361+0000 7f5743400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57441f3c60 0x7f57441569a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:36.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.362+0000 7f5743e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5728004800 0x7f5744157030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:36.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.362+0000 7f5743e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5728004800 0x7f5744157030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.90:41922/0 (socket says 172.21.15.90:41922) 2024-03-24T01:24:36.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.362+0000 7f5743e00640 1 -- 172.21.15.90:0/415112756 learned_addr learned my addr 172.21.15.90:0/415112756 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:36.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.362+0000 7f5748e00640 1 --2- 172.21.15.90:0/415112756 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57440cb6c0 0x7f57440cd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:36.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.362+0000 7f5743400640 1 -- 172.21.15.90:0/415112756 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57440cb6c0 msgr2=0x7f57440cd2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:36.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.362+0000 7f5743400640 1 --2- 172.21.15.90:0/415112756 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57440cb6c0 0x7f57440cd2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:36.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.362+0000 7f5743400640 1 -- 172.21.15.90:0/415112756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5728004800 msgr2=0x7f5744157030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:36.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.362+0000 7f5743400640 1 --2- 172.21.15.90:0/415112756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5728004800 0x7f5744157030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:36.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.362+0000 7f5743400640 1 -- 172.21.15.90:0/415112756 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57441e86f0 con 0x7f57441f3c60 2024-03-24T01:24:36.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.363+0000 7f5743400640 1 --2- 172.21.15.90:0/415112756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57441f3c60 0x7f57441569a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f572800c1b0 tx=0x7f5730001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:36.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.363+0000 7f573be00640 1 -- 172.21.15.90:0/415112756 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5730001370 con 0x7f57441f3c60 2024-03-24T01:24:36.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.363+0000 7f573be00640 1 -- 172.21.15.90:0/415112756 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5730042070 con 0x7f57441f3c60 2024-03-24T01:24:36.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.363+0000 7f573be00640 1 -- 172.21.15.90:0/415112756 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5730007a30 con 0x7f57441f3c60 2024-03-24T01:24:36.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.363+0000 7f574ae00640 1 -- 172.21.15.90:0/415112756 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57441e89d0 con 0x7f57441f3c60 2024-03-24T01:24:36.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.363+0000 7f574ae00640 1 -- 172.21.15.90:0/415112756 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57441e8f90 con 0x7f57441f3c60 2024-03-24T01:24:36.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.364+0000 7f574ae00640 1 -- 172.21.15.90:0/415112756 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f574404f400 con 0x7f57441f3c60 2024-03-24T01:24:36.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.365+0000 7f573be00640 1 -- 172.21.15.90:0/415112756 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f573003b040 con 0x7f57441f3c60 2024-03-24T01:24:36.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.367+0000 7f573be00640 1 --2- 172.21.15.90:0/415112756 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f572406bc00 0x7f572406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:36.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.367+0000 7f573be00640 1 -- 172.21.15.90:0/415112756 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f57300ac2e0 con 0x7f57441f3c60 2024-03-24T01:24:36.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.368+0000 7f5743e00640 1 --2- 172.21.15.90:0/415112756 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f572406bc00 0x7f572406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:36.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.369+0000 7f5743e00640 1 --2- 172.21.15.90:0/415112756 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f572406bc00 0x7f572406e0c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5734000c00 tx=0x7f5734009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:36.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.372+0000 7f573be00640 1 -- 172.21.15.90:0/415112756 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f573007a440 con 0x7f57441f3c60 2024-03-24T01:24:36.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:36.606+0000 7f574ae00640 1 -- 172.21.15.90:0/415112756 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f57440b3320 con 0x7f57441f3c60 2024-03-24T01:24:37.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.114+0000 7f573be00640 1 -- 172.21.15.90:0/415112756 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f573007e4e0 con 0x7f57441f3c60 2024-03-24T01:24:37.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.117+0000 7f574ae00640 1 -- 172.21.15.90:0/415112756 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f572406bc00 msgr2=0x7f572406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:37.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.117+0000 7f574ae00640 1 --2- 172.21.15.90:0/415112756 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f572406bc00 0x7f572406e0c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5734000c00 tx=0x7f5734009040 comp rx=0 tx=0).stop 2024-03-24T01:24:37.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.118+0000 7f574ae00640 1 -- 172.21.15.90:0/415112756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57441f3c60 msgr2=0x7f57441569a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:37.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.118+0000 7f574ae00640 1 --2- 172.21.15.90:0/415112756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57441f3c60 0x7f57441569a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f572800c1b0 tx=0x7f5730001120 comp rx=0 tx=0).stop 2024-03-24T01:24:37.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.118+0000 7f574ae00640 1 -- 172.21.15.90:0/415112756 shutdown_connections 2024-03-24T01:24:37.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.118+0000 7f574ae00640 1 --2- 172.21.15.90:0/415112756 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f572406bc00 0x7f572406e0c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:37.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.118+0000 7f574ae00640 1 --2- 172.21.15.90:0/415112756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f57441f3c60 0x7f57441569a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:37.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.118+0000 7f574ae00640 1 --2- 172.21.15.90:0/415112756 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f57440cb6c0 0x7f57440cd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:37.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.118+0000 7f574ae00640 1 --2- 172.21.15.90:0/415112756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5728004800 0x7f5744157030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:37.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.118+0000 7f574ae00640 1 -- 172.21.15.90:0/415112756 >> 172.21.15.90:0/415112756 conn(0x7f5744072c10 msgr2=0x7f57440b50f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:37.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.118+0000 7f574ae00640 1 -- 172.21.15.90:0/415112756 shutdown_connections 2024-03-24T01:24:37.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.118+0000 7f574ae00640 1 -- 172.21.15.90:0/415112756 wait complete. 2024-03-24T01:24:37.138 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-24T01:24:37.139 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:24:37.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.342+0000 7f1e91800640 1 Processor -- start 2024-03-24T01:24:37.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.342+0000 7f1e91800640 1 -- start start 2024-03-24T01:24:37.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.344+0000 7f1e91800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e8c1443d0 0x7f1e8c0cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:37.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.344+0000 7f1e91800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1e8c0cf3a0 0x7f1e8c0cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:37.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.344+0000 7f1e91800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e8c1447b0 con 0x7f1e8c0b3310 2024-03-24T01:24:37.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.344+0000 7f1e91800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e8c15aef0 con 0x7f1e8c0cf3a0 2024-03-24T01:24:37.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.344+0000 7f1e91800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c15b110 con 0x7f1e8c1443d0 2024-03-24T01:24:37.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.344+0000 7f1e8b400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1e8c0cf3a0 0x7f1e8c0cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:37.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.344+0000 7f1e8b400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1e8c0cf3a0 0x7f1e8c0cd3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43466/0 (socket says 172.21.15.90:43466) 2024-03-24T01:24:37.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.344+0000 7f1e8b400640 1 -- 172.21.15.90:0/2284534573 learned_addr learned my addr 172.21.15.90:0/2284534573 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:37.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.344+0000 7f1e8a000640 1 --1- 172.21.15.90:0/2284534573 >> v1:172.21.15.90:6789/0 conn(0x7f1e8c0b3310 0x7f1e8c0cec90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:32938/0 (socket says 172.21.15.90:32938) 2024-03-24T01:24:37.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.344+0000 7f1e8aa00640 1 --2- 172.21.15.90:0/2284534573 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e8c1443d0 0x7f1e8c0cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:37.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.345+0000 7f1e8b400640 1 -- 172.21.15.90:0/2284534573 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e8c1443d0 msgr2=0x7f1e8c0cceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:37.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.345+0000 7f1e8b400640 1 --2- 172.21.15.90:0/2284534573 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e8c1443d0 0x7f1e8c0cceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:37.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.345+0000 7f1e8b400640 1 -- 172.21.15.90:0/2284534573 >> v1:172.21.15.90:6789/0 conn(0x7f1e8c0b3310 legacy=0x7f1e8c0cec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:37.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.345+0000 7f1e8b400640 1 -- 172.21.15.90:0/2284534573 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e8c15b3f0 con 0x7f1e8c0cf3a0 2024-03-24T01:24:37.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.345+0000 7f1e89600640 1 -- 172.21.15.90:0/2284534573 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2711516220 0 0) 0x7f1e8c1447b0 con 0x7f1e8c0b3310 2024-03-24T01:24:37.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.346+0000 7f1e8b400640 1 --2- 172.21.15.90:0/2284534573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e8c0cf3a0 0x7f1e8c0cd3f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1e84009e50 tx=0x7f1e8402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b9a7b2d07affff9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:37.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.346+0000 7f1e89600640 1 -- 172.21.15.90:0/2284534573 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8402ca70 con 0x7f1e8c0cf3a0 2024-03-24T01:24:37.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.346+0000 7f1e89600640 1 -- 172.21.15.90:0/2284534573 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e8402cc10 con 0x7f1e8c0cf3a0 2024-03-24T01:24:37.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.346+0000 7f1e89600640 1 -- 172.21.15.90:0/2284534573 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e840343f0 con 0x7f1e8c0cf3a0 2024-03-24T01:24:37.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.346+0000 7f1e91800640 1 -- 172.21.15.90:0/2284534573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e8c0cf3a0 msgr2=0x7f1e8c0cd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:37.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.346+0000 7f1e91800640 1 --2- 172.21.15.90:0/2284534573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e8c0cf3a0 0x7f1e8c0cd3f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1e84009e50 tx=0x7f1e8402bed0 comp rx=0 tx=0).stop 2024-03-24T01:24:37.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.347+0000 7f1e91800640 1 -- 172.21.15.90:0/2284534573 shutdown_connections 2024-03-24T01:24:37.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.347+0000 7f1e91800640 1 --2- 172.21.15.90:0/2284534573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e8c0cf3a0 0x7f1e8c0cd3f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:37.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.347+0000 7f1e91800640 1 --2- 172.21.15.90:0/2284534573 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e8c1443d0 0x7f1e8c0cceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:37.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.347+0000 7f1e91800640 1 -- 172.21.15.90:0/2284534573 >> 172.21.15.90:0/2284534573 conn(0x7f1e8c06b2a0 msgr2=0x7f1e8c06b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:37.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.347+0000 7f1e91800640 1 -- 172.21.15.90:0/2284534573 shutdown_connections 2024-03-24T01:24:37.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.347+0000 7f1e91800640 1 -- 172.21.15.90:0/2284534573 wait complete. 2024-03-24T01:24:37.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.348+0000 7f1e91800640 1 Processor -- start 2024-03-24T01:24:37.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.349+0000 7f1e91800640 1 -- start start 2024-03-24T01:24:37.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.349+0000 7f1e91800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e8c0b3310 0x7f1e8c159f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:37.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.349+0000 7f1e91800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e8c0cf3a0 0x7f1e8c0b37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:37.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.349+0000 7f1e91800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e8c1443d0 0x7f1e8c0b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:37.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.349+0000 7f1e91800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c15bc90 con 0x7f1e8c0b3310 2024-03-24T01:24:37.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.349+0000 7f1e91800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c15b9b0 con 0x7f1e8c1443d0 2024-03-24T01:24:37.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.349+0000 7f1e91800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c15e610 con 0x7f1e8c0cf3a0 2024-03-24T01:24:37.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.350+0000 7f1e8aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e8c0b3310 0x7f1e8c159f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:37.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.350+0000 7f1e8aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e8c0b3310 0x7f1e8c159f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43478/0 (socket says 172.21.15.90:43478) 2024-03-24T01:24:37.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.350+0000 7f1e8aa00640 1 -- 172.21.15.90:0/1356131469 learned_addr learned my addr 172.21.15.90:0/1356131469 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:37.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.350+0000 7f1e8a000640 1 --2- 172.21.15.90:0/1356131469 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e8c0cf3a0 0x7f1e8c0b37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:37.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.350+0000 7f1e8b400640 1 --2- 172.21.15.90:0/1356131469 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e8c1443d0 0x7f1e8c0b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:37.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.350+0000 7f1e8aa00640 1 -- 172.21.15.90:0/1356131469 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e8c0cf3a0 msgr2=0x7f1e8c0b37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:37.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.350+0000 7f1e8aa00640 1 --2- 172.21.15.90:0/1356131469 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e8c0cf3a0 0x7f1e8c0b37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:37.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.350+0000 7f1e8aa00640 1 -- 172.21.15.90:0/1356131469 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e8c1443d0 msgr2=0x7f1e8c0b3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:37.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.350+0000 7f1e8aa00640 1 --2- 172.21.15.90:0/1356131469 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e8c1443d0 0x7f1e8c0b3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:37.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.350+0000 7f1e8aa00640 1 -- 172.21.15.90:0/1356131469 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e8c0b0460 con 0x7f1e8c0b3310 2024-03-24T01:24:37.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.350+0000 7f1e8a000640 1 --2- 172.21.15.90:0/1356131469 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e8c0cf3a0 0x7f1e8c0b37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:37.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.350+0000 7f1e8b400640 1 --2- 172.21.15.90:0/1356131469 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e8c1443d0 0x7f1e8c0b3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:37.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.351+0000 7f1e8aa00640 1 --2- 172.21.15.90:0/1356131469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e8c0b3310 0x7f1e8c159f10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c00d620 tx=0x7f1e7c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:37.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.351+0000 7f1e82a00640 1 -- 172.21.15.90:0/1356131469 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7c00b620 con 0x7f1e8c0b3310 2024-03-24T01:24:37.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.351+0000 7f1e82a00640 1 -- 172.21.15.90:0/1356131469 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e7c019070 con 0x7f1e8c0b3310 2024-03-24T01:24:37.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.351+0000 7f1e91800640 1 -- 172.21.15.90:0/1356131469 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e8c0b0740 con 0x7f1e8c0b3310 2024-03-24T01:24:37.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.351+0000 7f1e82a00640 1 -- 172.21.15.90:0/1356131469 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7c00ba00 con 0x7f1e8c0b3310 2024-03-24T01:24:37.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.351+0000 7f1e91800640 1 -- 172.21.15.90:0/1356131469 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e8c0b0d00 con 0x7f1e8c0b3310 2024-03-24T01:24:37.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.352+0000 7f1e91800640 1 -- 172.21.15.90:0/1356131469 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e8c073570 con 0x7f1e8c0b3310 2024-03-24T01:24:37.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.353+0000 7f1e82a00640 1 -- 172.21.15.90:0/1356131469 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1e7c00bba0 con 0x7f1e8c0b3310 2024-03-24T01:24:37.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.354+0000 7f1e82a00640 1 --2- 172.21.15.90:0/1356131469 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1e6c06bc00 0x7f1e6c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:37.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.354+0000 7f1e82a00640 1 -- 172.21.15.90:0/1356131469 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f1e7c08b190 con 0x7f1e8c0b3310 2024-03-24T01:24:37.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.354+0000 7f1e8a000640 1 --2- 172.21.15.90:0/1356131469 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1e6c06bc00 0x7f1e6c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:37.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.355+0000 7f1e8a000640 1 --2- 172.21.15.90:0/1356131469 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1e6c06bc00 0x7f1e6c06e0c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c0b4830 tx=0x7f1e78009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:37.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.358+0000 7f1e82a00640 1 -- 172.21.15.90:0/1356131469 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1e7c059940 con 0x7f1e8c0b3310 2024-03-24T01:24:37.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.593+0000 7f1e91800640 1 -- 172.21.15.90:0/1356131469 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e8c04f4e0 con 0x7f1e8c0b3310 2024-03-24T01:24:37.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.594+0000 7f1e82a00640 1 -- 172.21.15.90:0/1356131469 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4967 (secure 0 0 0) 0x7f1e7c059240 con 0x7f1e8c0b3310 2024-03-24T01:24:37.595 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:24:37.595 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/1898972903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":1898972903},{"type":"v1","addr":"172.21.15.120:6837","nonce":1898972903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6834/775759501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":775759501},{"type":"v1","addr":"172.21.15.90:6834","nonce":775759501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/43630575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":43630575},{"type":"v1","addr":"172.21.15.90:6835","nonce":43630575}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T01:24:10.173532+0000","modified":"2024-03-24T01:24:36.608477+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6836/963928022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":963928022},{"type":"v1","addr":"172.21.15.120:6836","nonce":963928022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4257,"qdb_cluster":[4257]},"id":1}]} 2024-03-24T01:24:37.595 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 8 2024-03-24T01:24:37.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.597+0000 7f1e91800640 1 -- 172.21.15.90:0/1356131469 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1e6c06bc00 msgr2=0x7f1e6c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:37.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.597+0000 7f1e91800640 1 --2- 172.21.15.90:0/1356131469 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1e6c06bc00 0x7f1e6c06e0c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c0b4830 tx=0x7f1e78009040 comp rx=0 tx=0).stop 2024-03-24T01:24:37.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.597+0000 7f1e91800640 1 -- 172.21.15.90:0/1356131469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e8c0b3310 msgr2=0x7f1e8c159f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:37.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.597+0000 7f1e91800640 1 --2- 172.21.15.90:0/1356131469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e8c0b3310 0x7f1e8c159f10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c00d620 tx=0x7f1e7c00dae0 comp rx=0 tx=0).stop 2024-03-24T01:24:37.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.597+0000 7f1e91800640 1 -- 172.21.15.90:0/1356131469 shutdown_connections 2024-03-24T01:24:37.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.597+0000 7f1e91800640 1 --2- 172.21.15.90:0/1356131469 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1e6c06bc00 0x7f1e6c06e0c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:37.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.597+0000 7f1e91800640 1 --2- 172.21.15.90:0/1356131469 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e8c1443d0 0x7f1e8c0b3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:37.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.597+0000 7f1e91800640 1 --2- 172.21.15.90:0/1356131469 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1e8c0cf3a0 0x7f1e8c0b37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:37.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.597+0000 7f1e91800640 1 --2- 172.21.15.90:0/1356131469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e8c0b3310 0x7f1e8c159f10 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:37.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.597+0000 7f1e91800640 1 -- 172.21.15.90:0/1356131469 >> 172.21.15.90:0/1356131469 conn(0x7f1e8c06b2a0 msgr2=0x7f1e8c0af6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:37.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.597+0000 7f1e91800640 1 -- 172.21.15.90:0/1356131469 shutdown_connections 2024-03-24T01:24:37.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.597+0000 7f1e91800640 1 -- 172.21.15.90:0/1356131469 wait complete. 2024-03-24T01:24:37.615 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:24:37.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.819+0000 7fcf42600640 1 Processor -- start 2024-03-24T01:24:37.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.820+0000 7fcf42600640 1 -- start start 2024-03-24T01:24:37.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.821+0000 7fcf42600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcf3c0c5bb0 0x7fcf3c0b13c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:37.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.821+0000 7fcf42600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcf3c15ac60 0x7fcf3c0c5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:37.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.821+0000 7fcf42600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf3c0b31f0 con 0x7fcf3c0b1ab0 2024-03-24T01:24:37.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.821+0000 7fcf42600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf3c0b1900 con 0x7fcf3c0c5bb0 2024-03-24T01:24:37.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.821+0000 7fcf42600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf3c0c6830 con 0x7fcf3c15ac60 2024-03-24T01:24:37.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.821+0000 7fcf41200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcf3c0c5bb0 0x7fcf3c0b13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:37.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.822+0000 7fcf41200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcf3c0c5bb0 0x7fcf3c0b13c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43484/0 (socket says 172.21.15.90:43484) 2024-03-24T01:24:37.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.822+0000 7fcf41200640 1 -- 172.21.15.90:0/1682158406 learned_addr learned my addr 172.21.15.90:0/1682158406 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:37.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.822+0000 7fcf41c00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fcf3c0b1ab0 0x7fcf3c0ccfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:32946/0 (socket says 172.21.15.90:32946) 2024-03-24T01:24:37.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.822+0000 7fcf3be00640 1 --2- 172.21.15.90:0/1682158406 >> v2:172.21.15.120:3300/0 conn(0x7fcf3c15ac60 0x7fcf3c0c5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:37.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.822+0000 7fcf41200640 1 -- 172.21.15.90:0/1682158406 >> v2:172.21.15.120:3300/0 conn(0x7fcf3c15ac60 msgr2=0x7fcf3c0c5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:37.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.822+0000 7fcf41200640 1 --2- 172.21.15.90:0/1682158406 >> v2:172.21.15.120:3300/0 conn(0x7fcf3c15ac60 0x7fcf3c0c5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:37.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.822+0000 7fcf41200640 1 -- 172.21.15.90:0/1682158406 >> v1:172.21.15.90:6789/0 conn(0x7fcf3c0b1ab0 legacy=0x7fcf3c0ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:37.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.822+0000 7fcf41200640 1 -- 172.21.15.90:0/1682158406 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf3c0c64d0 con 0x7fcf3c0c5bb0 2024-03-24T01:24:37.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.823+0000 7fcf41200640 1 --2- 172.21.15.90:0/1682158406 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf3c0c5bb0 0x7fcf3c0b13c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c067810 tx=0x7fcf2c02e180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c067cdecbc641cd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:37.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.824+0000 7fcf3b400640 1 -- 172.21.15.90:0/1682158406 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c00b0d0 con 0x7fcf3c0c5bb0 2024-03-24T01:24:37.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.824+0000 7fcf3b400640 1 -- 172.21.15.90:0/1682158406 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf2c00b270 con 0x7fcf3c0c5bb0 2024-03-24T01:24:37.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.824+0000 7fcf3b400640 1 -- 172.21.15.90:0/1682158406 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c009600 con 0x7fcf3c0c5bb0 2024-03-24T01:24:37.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.824+0000 7fcf42600640 1 -- 172.21.15.90:0/1682158406 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf3c0c5bb0 msgr2=0x7fcf3c0b13c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:37.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.824+0000 7fcf42600640 1 --2- 172.21.15.90:0/1682158406 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf3c0c5bb0 0x7fcf3c0b13c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c067810 tx=0x7fcf2c02e180 comp rx=0 tx=0).stop 2024-03-24T01:24:37.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.824+0000 7fcf42600640 1 -- 172.21.15.90:0/1682158406 shutdown_connections 2024-03-24T01:24:37.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.824+0000 7fcf42600640 1 --2- 172.21.15.90:0/1682158406 >> v2:172.21.15.120:3300/0 conn(0x7fcf3c15ac60 0x7fcf3c0c5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:37.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.824+0000 7fcf42600640 1 --2- 172.21.15.90:0/1682158406 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf3c0c5bb0 0x7fcf3c0b13c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:37.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.824+0000 7fcf42600640 1 -- 172.21.15.90:0/1682158406 >> 172.21.15.90:0/1682158406 conn(0x7fcf3c0b2350 msgr2=0x7fcf3c0b2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:37.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.824+0000 7fcf42600640 1 -- 172.21.15.90:0/1682158406 shutdown_connections 2024-03-24T01:24:37.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.824+0000 7fcf42600640 1 -- 172.21.15.90:0/1682158406 wait complete. 2024-03-24T01:24:37.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.826+0000 7fcf42600640 1 Processor -- start 2024-03-24T01:24:37.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.826+0000 7fcf42600640 1 -- start start 2024-03-24T01:24:37.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.827+0000 7fcf42600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf3c0b1ab0 0x7fcf3c158e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:37.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.827+0000 7fcf42600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf3c0c5bb0 0x7fcf3c0b4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:37.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.827+0000 7fcf42600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf3c15ac60 0x7fcf3c1f4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:37.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.827+0000 7fcf42600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3c0cf4d0 con 0x7fcf3c0c5bb0 2024-03-24T01:24:37.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.827+0000 7fcf42600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3c0b2f10 con 0x7fcf3c15ac60 2024-03-24T01:24:37.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.827+0000 7fcf42600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3c0cf7b0 con 0x7fcf3c0b1ab0 2024-03-24T01:24:37.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.827+0000 7fcf3be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf3c0c5bb0 0x7fcf3c0b4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:37.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.828+0000 7fcf3be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf3c0c5bb0 0x7fcf3c0b4200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43488/0 (socket says 172.21.15.90:43488) 2024-03-24T01:24:37.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.828+0000 7fcf3be00640 1 -- 172.21.15.90:0/2227058596 learned_addr learned my addr 172.21.15.90:0/2227058596 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:37.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.828+0000 7fcf41200640 1 --2- 172.21.15.90:0/2227058596 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf3c0b1ab0 0x7fcf3c158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:37.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.828+0000 7fcf41c00640 1 --2- 172.21.15.90:0/2227058596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf3c15ac60 0x7fcf3c1f4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:37.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.828+0000 7fcf3be00640 1 -- 172.21.15.90:0/2227058596 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf3c0b1ab0 msgr2=0x7fcf3c158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:37.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.828+0000 7fcf3be00640 1 --2- 172.21.15.90:0/2227058596 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf3c0b1ab0 0x7fcf3c158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:37.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.828+0000 7fcf3be00640 1 -- 172.21.15.90:0/2227058596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf3c15ac60 msgr2=0x7fcf3c1f4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:37.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.828+0000 7fcf3be00640 1 --2- 172.21.15.90:0/2227058596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf3c15ac60 0x7fcf3c1f4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:37.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.828+0000 7fcf3be00640 1 -- 172.21.15.90:0/2227058596 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf3c0b64d0 con 0x7fcf3c0c5bb0 2024-03-24T01:24:37.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.828+0000 7fcf41c00640 1 --2- 172.21.15.90:0/2227058596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf3c15ac60 0x7fcf3c1f4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:37.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.828+0000 7fcf3be00640 1 --2- 172.21.15.90:0/2227058596 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf3c0c5bb0 0x7fcf3c0b4200 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcf2800d5f0 tx=0x7fcf2800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:37.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.828+0000 7fcf38c00640 1 -- 172.21.15.90:0/2227058596 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf280125f0 con 0x7fcf3c0c5bb0 2024-03-24T01:24:37.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.829+0000 7fcf38c00640 1 -- 172.21.15.90:0/2227058596 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf28019070 con 0x7fcf3c0c5bb0 2024-03-24T01:24:37.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.829+0000 7fcf42600640 1 -- 172.21.15.90:0/2227058596 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf3c0b3440 con 0x7fcf3c0c5bb0 2024-03-24T01:24:37.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.829+0000 7fcf38c00640 1 -- 172.21.15.90:0/2227058596 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf280129d0 con 0x7fcf3c0c5bb0 2024-03-24T01:24:37.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.829+0000 7fcf42600640 1 -- 172.21.15.90:0/2227058596 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf3c0b3a00 con 0x7fcf3c0c5bb0 2024-03-24T01:24:37.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.830+0000 7fcf42600640 1 -- 172.21.15.90:0/2227058596 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf3c04f400 con 0x7fcf3c0c5bb0 2024-03-24T01:24:37.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.831+0000 7fcf38c00640 1 -- 172.21.15.90:0/2227058596 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcf28012b70 con 0x7fcf3c0c5bb0 2024-03-24T01:24:37.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.831+0000 7fcf38c00640 1 --2- 172.21.15.90:0/2227058596 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcf1c06bbd0 0x7fcf1c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:37.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.832+0000 7fcf38c00640 1 -- 172.21.15.90:0/2227058596 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fcf2808b1f0 con 0x7fcf3c0c5bb0 2024-03-24T01:24:37.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.832+0000 7fcf41200640 1 --2- 172.21.15.90:0/2227058596 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcf1c06bbd0 0x7fcf1c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:37.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.833+0000 7fcf41200640 1 --2- 172.21.15.90:0/2227058596 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcf1c06bbd0 0x7fcf1c06e090 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c037000 tx=0x7fcf2c032000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:37.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:37.836+0000 7fcf38c00640 1 -- 172.21.15.90:0/2227058596 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcf28059990 con 0x7fcf3c0c5bb0 2024-03-24T01:24:38.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.033+0000 7fcf42600640 1 -- 172.21.15.90:0/2227058596 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf3c067910 con 0x7fcf3c0c5bb0 2024-03-24T01:24:38.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.035+0000 7fcf38c00640 1 -- 172.21.15.90:0/2227058596 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18490 (secure 0 0 0) 0x7fcf28059290 con 0x7fcf3c0c5bb0 2024-03-24T01:24:38.035 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:24:38.036 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:24:17.008224+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:24:08.643966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T01:24:09.189904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T01:24:13.027323+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:24:38.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.037+0000 7fcf42600640 1 -- 172.21.15.90:0/2227058596 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcf1c06bbd0 msgr2=0x7fcf1c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:38.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.037+0000 7fcf42600640 1 --2- 172.21.15.90:0/2227058596 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcf1c06bbd0 0x7fcf1c06e090 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c037000 tx=0x7fcf2c032000 comp rx=0 tx=0).stop 2024-03-24T01:24:38.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.037+0000 7fcf42600640 1 -- 172.21.15.90:0/2227058596 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf3c0c5bb0 msgr2=0x7fcf3c0b4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:38.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.037+0000 7fcf42600640 1 --2- 172.21.15.90:0/2227058596 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf3c0c5bb0 0x7fcf3c0b4200 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcf2800d5f0 tx=0x7fcf2800dab0 comp rx=0 tx=0).stop 2024-03-24T01:24:38.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.038+0000 7fcf42600640 1 -- 172.21.15.90:0/2227058596 shutdown_connections 2024-03-24T01:24:38.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.038+0000 7fcf42600640 1 --2- 172.21.15.90:0/2227058596 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcf1c06bbd0 0x7fcf1c06e090 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:38.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.038+0000 7fcf42600640 1 --2- 172.21.15.90:0/2227058596 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf3c0c5bb0 0x7fcf3c0b4200 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:38.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.038+0000 7fcf42600640 1 --2- 172.21.15.90:0/2227058596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf3c15ac60 0x7fcf3c1f4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:38.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.038+0000 7fcf42600640 1 --2- 172.21.15.90:0/2227058596 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf3c0b1ab0 0x7fcf3c158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:38.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.038+0000 7fcf42600640 1 -- 172.21.15.90:0/2227058596 >> 172.21.15.90:0/2227058596 conn(0x7fcf3c0b2350 msgr2=0x7fcf3c0ccfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:38.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.038+0000 7fcf42600640 1 -- 172.21.15.90:0/2227058596 shutdown_connections 2024-03-24T01:24:38.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.038+0000 7fcf42600640 1 -- 172.21.15.90:0/2227058596 wait complete. 2024-03-24T01:24:38.057 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-24T01:24:38.058 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T01:24:38.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.267+0000 7ff743400640 1 Processor -- start 2024-03-24T01:24:38.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.267+0000 7ff743400640 1 -- start start 2024-03-24T01:24:38.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.268+0000 7ff743400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff73c1443d0 0x7ff73c0cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:38.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.269+0000 7ff743400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff73c0b3310 0x7ff73c0cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:38.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.269+0000 7ff743400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff73c0cec90 0x7ff73c0cf0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:38.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.269+0000 7ff743400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff73c15aef0 con 0x7ff73c0b3310 2024-03-24T01:24:38.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.269+0000 7ff743400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff73c1447b0 con 0x7ff73c0cec90 2024-03-24T01:24:38.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.269+0000 7ff743400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff73c15b110 con 0x7ff73c1443d0 2024-03-24T01:24:38.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.269+0000 7ff73be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff73c0b3310 0x7ff73c0cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:38.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.269+0000 7ff73be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff73c0b3310 0x7ff73c0cd3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43498/0 (socket says 172.21.15.90:43498) 2024-03-24T01:24:38.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.269+0000 7ff73be00640 1 -- 172.21.15.90:0/3727195581 learned_addr learned my addr 172.21.15.90:0/3727195581 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:38.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.269+0000 7ff740a00640 1 --2- 172.21.15.90:0/3727195581 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff73c1443d0 0x7ff73c0cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:38.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.269+0000 7ff741400640 1 --2- 172.21.15.90:0/3727195581 >> v2:172.21.15.120:3300/0 conn(0x7ff73c0cec90 0x7ff73c0cf0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:38.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.269+0000 7ff73be00640 1 -- 172.21.15.90:0/3727195581 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff73c1443d0 msgr2=0x7ff73c0cceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:38.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.269+0000 7ff73be00640 1 --2- 172.21.15.90:0/3727195581 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff73c1443d0 0x7ff73c0cceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:38.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.269+0000 7ff73be00640 1 -- 172.21.15.90:0/3727195581 >> v2:172.21.15.120:3300/0 conn(0x7ff73c0cec90 msgr2=0x7ff73c0cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:38.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.269+0000 7ff73be00640 1 --2- 172.21.15.90:0/3727195581 >> v2:172.21.15.120:3300/0 conn(0x7ff73c0cec90 0x7ff73c0cf0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:38.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.269+0000 7ff73be00640 1 -- 172.21.15.90:0/3727195581 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff73c15b3f0 con 0x7ff73c0b3310 2024-03-24T01:24:38.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.270+0000 7ff73be00640 1 --2- 172.21.15.90:0/3727195581 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff73c0b3310 0x7ff73c0cd3f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff73c0b2de0 tx=0x7ff72802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c90e614c79f4df7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:38.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.271+0000 7ff73b400640 1 -- 172.21.15.90:0/3727195581 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff72802e070 con 0x7ff73c0b3310 2024-03-24T01:24:38.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.271+0000 7ff73b400640 1 -- 172.21.15.90:0/3727195581 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff728037070 con 0x7ff73c0b3310 2024-03-24T01:24:38.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.271+0000 7ff73b400640 1 -- 172.21.15.90:0/3727195581 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff728031a00 con 0x7ff73c0b3310 2024-03-24T01:24:38.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.271+0000 7ff743400640 1 -- 172.21.15.90:0/3727195581 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff73c0b3310 msgr2=0x7ff73c0cd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:38.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.271+0000 7ff743400640 1 --2- 172.21.15.90:0/3727195581 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff73c0b3310 0x7ff73c0cd3f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff73c0b2de0 tx=0x7ff72802b880 comp rx=0 tx=0).stop 2024-03-24T01:24:38.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.272+0000 7ff743400640 1 -- 172.21.15.90:0/3727195581 shutdown_connections 2024-03-24T01:24:38.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.272+0000 7ff743400640 1 --2- 172.21.15.90:0/3727195581 >> v2:172.21.15.120:3300/0 conn(0x7ff73c0cec90 0x7ff73c0cf0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:38.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.272+0000 7ff743400640 1 --2- 172.21.15.90:0/3727195581 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff73c0b3310 0x7ff73c0cd3f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:38.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.272+0000 7ff743400640 1 --2- 172.21.15.90:0/3727195581 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff73c1443d0 0x7ff73c0cceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:38.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.272+0000 7ff743400640 1 -- 172.21.15.90:0/3727195581 >> 172.21.15.90:0/3727195581 conn(0x7ff73c06b2a0 msgr2=0x7ff73c06b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:38.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.272+0000 7ff743400640 1 -- 172.21.15.90:0/3727195581 shutdown_connections 2024-03-24T01:24:38.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.272+0000 7ff743400640 1 -- 172.21.15.90:0/3727195581 wait complete. 2024-03-24T01:24:38.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.273+0000 7ff743400640 1 Processor -- start 2024-03-24T01:24:38.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.273+0000 7ff743400640 1 -- start start 2024-03-24T01:24:38.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.274+0000 7ff743400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff73c0b3310 0x7ff73c15a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:38.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.274+0000 7ff743400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff73c0cec90 0x7ff73c135a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:38.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.274+0000 7ff743400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff73c1443d0 0x7ff73c06d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:38.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.274+0000 7ff743400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73c15bc90 con 0x7ff73c0b3310 2024-03-24T01:24:38.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.274+0000 7ff743400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73c15b9b0 con 0x7ff73c1443d0 2024-03-24T01:24:38.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.274+0000 7ff743400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff73c0cf620 con 0x7ff73c0cec90 2024-03-24T01:24:38.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.275+0000 7ff740a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff73c0b3310 0x7ff73c15a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:38.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.275+0000 7ff740a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff73c0b3310 0x7ff73c15a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43514/0 (socket says 172.21.15.90:43514) 2024-03-24T01:24:38.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.275+0000 7ff740a00640 1 -- 172.21.15.90:0/3416663981 learned_addr learned my addr 172.21.15.90:0/3416663981 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:38.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.275+0000 7ff73be00640 1 --2- 172.21.15.90:0/3416663981 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff73c0cec90 0x7ff73c135a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:38.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.275+0000 7ff740a00640 1 -- 172.21.15.90:0/3416663981 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff73c0cec90 msgr2=0x7ff73c135a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:38.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.275+0000 7ff741400640 1 --2- 172.21.15.90:0/3416663981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff73c1443d0 0x7ff73c06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:38.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.275+0000 7ff740a00640 1 --2- 172.21.15.90:0/3416663981 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff73c0cec90 0x7ff73c135a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:38.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.275+0000 7ff740a00640 1 -- 172.21.15.90:0/3416663981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff73c1443d0 msgr2=0x7ff73c06d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:38.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.275+0000 7ff740a00640 1 --2- 172.21.15.90:0/3416663981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff73c1443d0 0x7ff73c06d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:38.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.275+0000 7ff740a00640 1 -- 172.21.15.90:0/3416663981 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff73c1364a0 con 0x7ff73c0b3310 2024-03-24T01:24:38.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.275+0000 7ff73be00640 1 --2- 172.21.15.90:0/3416663981 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff73c0cec90 0x7ff73c135a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:38.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.275+0000 7ff741400640 1 --2- 172.21.15.90:0/3416663981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff73c1443d0 0x7ff73c06d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:38.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.275+0000 7ff740a00640 1 --2- 172.21.15.90:0/3416663981 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff73c0b3310 0x7ff73c15a300 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff73000d800 tx=0x7ff73000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:38.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.276+0000 7ff738c00640 1 -- 172.21.15.90:0/3416663981 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff73000b6c0 con 0x7ff73c0b3310 2024-03-24T01:24:38.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.276+0000 7ff738c00640 1 -- 172.21.15.90:0/3416663981 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff730019070 con 0x7ff73c0b3310 2024-03-24T01:24:38.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.276+0000 7ff738c00640 1 -- 172.21.15.90:0/3416663981 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff73000ba80 con 0x7ff73c0b3310 2024-03-24T01:24:38.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.276+0000 7ff743400640 1 -- 172.21.15.90:0/3416663981 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff73c136780 con 0x7ff73c0b3310 2024-03-24T01:24:38.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.276+0000 7ff743400640 1 -- 172.21.15.90:0/3416663981 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff73c136d40 con 0x7ff73c0b3310 2024-03-24T01:24:38.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.277+0000 7ff743400640 1 -- 172.21.15.90:0/3416663981 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff73c073570 con 0x7ff73c0b3310 2024-03-24T01:24:38.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.278+0000 7ff738c00640 1 -- 172.21.15.90:0/3416663981 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff73000bc20 con 0x7ff73c0b3310 2024-03-24T01:24:38.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.278+0000 7ff738c00640 1 --2- 172.21.15.90:0/3416663981 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff71c06bbd0 0x7ff71c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:38.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.279+0000 7ff738c00640 1 -- 172.21.15.90:0/3416663981 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ff73008b230 con 0x7ff73c0b3310 2024-03-24T01:24:38.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.279+0000 7ff73be00640 1 --2- 172.21.15.90:0/3416663981 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff71c06bbd0 0x7ff71c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:38.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.280+0000 7ff73be00640 1 --2- 172.21.15.90:0/3416663981 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff71c06bbd0 0x7ff71c06e090 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff728036000 tx=0x7ff728008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:38.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.283+0000 7ff738c00640 1 -- 172.21.15.90:0/3416663981 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff7300599d0 con 0x7ff73c0b3310 2024-03-24T01:24:38.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.461+0000 7ff743400640 1 -- 172.21.15.90:0/3416663981 --> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff73c04f4e0 con 0x7ff71c06bbd0 2024-03-24T01:24:38.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.493+0000 7ff738c00640 1 -- 172.21.15.90:0/3416663981 <== mgr.4104 v2:172.21.15.120:6832/860392634 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332537 (secure 0 0 0) 0x7ff73c04f4e0 con 0x7ff71c06bbd0 2024-03-24T01:24:38.496 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:24:38.502 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-24T01:24:38.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.499+0000 7ff743400640 1 -- 172.21.15.90:0/3416663981 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff71c06bbd0 msgr2=0x7ff71c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:38.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.499+0000 7ff743400640 1 --2- 172.21.15.90:0/3416663981 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff71c06bbd0 0x7ff71c06e090 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff728036000 tx=0x7ff728008030 comp rx=0 tx=0).stop 2024-03-24T01:24:38.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.499+0000 7ff743400640 1 -- 172.21.15.90:0/3416663981 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff73c0b3310 msgr2=0x7ff73c15a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:38.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.499+0000 7ff743400640 1 --2- 172.21.15.90:0/3416663981 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff73c0b3310 0x7ff73c15a300 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff73000d800 tx=0x7ff73000dcc0 comp rx=0 tx=0).stop 2024-03-24T01:24:38.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.499+0000 7ff743400640 1 -- 172.21.15.90:0/3416663981 shutdown_connections 2024-03-24T01:24:38.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.499+0000 7ff743400640 1 --2- 172.21.15.90:0/3416663981 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff71c06bbd0 0x7ff71c06e090 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:38.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.499+0000 7ff743400640 1 --2- 172.21.15.90:0/3416663981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff73c1443d0 0x7ff73c06d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:38.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.499+0000 7ff743400640 1 --2- 172.21.15.90:0/3416663981 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff73c0cec90 0x7ff73c135a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:38.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.499+0000 7ff743400640 1 --2- 172.21.15.90:0/3416663981 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff73c0b3310 0x7ff73c15a300 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:38.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.499+0000 7ff743400640 1 -- 172.21.15.90:0/3416663981 >> 172.21.15.90:0/3416663981 conn(0x7ff73c06b2a0 msgr2=0x7ff73c0b0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:38.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.499+0000 7ff743400640 1 -- 172.21.15.90:0/3416663981 shutdown_connections 2024-03-24T01:24:38.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.499+0000 7ff743400640 1 -- 172.21.15.90:0/3416663981 wait complete. 2024-03-24T01:24:38.517 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-03-24T01:24:37.487823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219472,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755248,"statfs":{"total":773094113280,"available":772869373952,"internally_reserved":0,"allocated":3383296,"data_stored":2049160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12747,"internal_metadata":221236789},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005335"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":28,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811464+0000","last_change":"2024-03-24T01:24:09.722150+0000","last_active":"2024-03-24T01:24:17.811464+0000","last_peered":"2024-03-24T01:24:17.811464+0000","last_clean":"2024-03-24T01:24:17.811464+0000","last_became_active":"2024-03-24T01:24:09.721808+0000","last_became_peered":"2024-03-24T01:24:09.721808+0000","last_unstale":"2024-03-24T01:24:17.811464+0000","last_undegraded":"2024-03-24T01:24:17.811464+0000","last_fullsized":"2024-03-24T01:24:17.811464+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:31:46.512174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022533+0000","last_change":"2024-03-24T01:24:10.750478+0000","last_active":"2024-03-24T01:24:17.022533+0000","last_peered":"2024-03-24T01:24:17.022533+0000","last_clean":"2024-03-24T01:24:17.022533+0000","last_became_active":"2024-03-24T01:24:10.749299+0000","last_became_peered":"2024-03-24T01:24:10.749299+0000","last_unstale":"2024-03-24T01:24:17.022533+0000","last_undegraded":"2024-03-24T01:24:17.022533+0000","last_fullsized":"2024-03-24T01:24:17.022533+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:46:04.235199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021708+0000","last_change":"2024-03-24T01:24:15.662475+0000","last_active":"2024-03-24T01:24:17.021708+0000","last_peered":"2024-03-24T01:24:17.021708+0000","last_clean":"2024-03-24T01:24:17.021708+0000","last_became_active":"2024-03-24T01:24:15.661550+0000","last_became_peered":"2024-03-24T01:24:15.661550+0000","last_unstale":"2024-03-24T01:24:17.021708+0000","last_undegraded":"2024-03-24T01:24:17.021708+0000","last_fullsized":"2024-03-24T01:24:17.021708+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:14:59.259808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022711+0000","last_change":"2024-03-24T01:24:09.723434+0000","last_active":"2024-03-24T01:24:17.022711+0000","last_peered":"2024-03-24T01:24:17.022711+0000","last_clean":"2024-03-24T01:24:17.022711+0000","last_became_active":"2024-03-24T01:24:09.722439+0000","last_became_peered":"2024-03-24T01:24:09.722439+0000","last_unstale":"2024-03-24T01:24:17.022711+0000","last_undegraded":"2024-03-24T01:24:17.022711+0000","last_fullsized":"2024-03-24T01:24:17.022711+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:31:46.512174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811170+0000","last_change":"2024-03-24T01:24:10.745888+0000","last_active":"2024-03-24T01:24:17.811170+0000","last_peered":"2024-03-24T01:24:17.811170+0000","last_clean":"2024-03-24T01:24:17.811170+0000","last_became_active":"2024-03-24T01:24:10.742579+0000","last_became_peered":"2024-03-24T01:24:10.742579+0000","last_unstale":"2024-03-24T01:24:17.811170+0000","last_undegraded":"2024-03-24T01:24:17.811170+0000","last_fullsized":"2024-03-24T01:24:17.811170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:04:48.028091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.026871+0000","last_change":"2024-03-24T01:24:15.022407+0000","last_active":"2024-03-24T01:24:17.026871+0000","last_peered":"2024-03-24T01:24:17.026871+0000","last_clean":"2024-03-24T01:24:17.026871+0000","last_became_active":"2024-03-24T01:24:15.021962+0000","last_became_peered":"2024-03-24T01:24:15.021962+0000","last_unstale":"2024-03-24T01:24:17.026871+0000","last_undegraded":"2024-03-24T01:24:17.026871+0000","last_fullsized":"2024-03-24T01:24:17.026871+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:28.723031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.807761+0000","last_change":"2024-03-24T01:24:09.735868+0000","last_active":"2024-03-24T01:24:17.807761+0000","last_peered":"2024-03-24T01:24:17.807761+0000","last_clean":"2024-03-24T01:24:17.807761+0000","last_became_active":"2024-03-24T01:24:09.735573+0000","last_became_peered":"2024-03-24T01:24:09.735573+0000","last_unstale":"2024-03-24T01:24:17.807761+0000","last_undegraded":"2024-03-24T01:24:17.807761+0000","last_fullsized":"2024-03-24T01:24:17.807761+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:35:54.000044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811189+0000","last_change":"2024-03-24T01:24:10.752632+0000","last_active":"2024-03-24T01:24:17.811189+0000","last_peered":"2024-03-24T01:24:17.811189+0000","last_clean":"2024-03-24T01:24:17.811189+0000","last_became_active":"2024-03-24T01:24:10.752358+0000","last_became_peered":"2024-03-24T01:24:10.752358+0000","last_unstale":"2024-03-24T01:24:17.811189+0000","last_undegraded":"2024-03-24T01:24:17.811189+0000","last_fullsized":"2024-03-24T01:24:17.811189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:38:07.266851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.027523+0000","last_change":"2024-03-24T01:24:15.032062+0000","last_active":"2024-03-24T01:24:17.027523+0000","last_peered":"2024-03-24T01:24:17.027523+0000","last_clean":"2024-03-24T01:24:17.027523+0000","last_became_active":"2024-03-24T01:24:15.031556+0000","last_became_peered":"2024-03-24T01:24:15.031556+0000","last_unstale":"2024-03-24T01:24:17.027523+0000","last_undegraded":"2024-03-24T01:24:17.027523+0000","last_fullsized":"2024-03-24T01:24:17.027523+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:45:17.532730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.812001+0000","last_change":"2024-03-24T01:24:09.739225+0000","last_active":"2024-03-24T01:24:17.812001+0000","last_peered":"2024-03-24T01:24:17.812001+0000","last_clean":"2024-03-24T01:24:17.812001+0000","last_became_active":"2024-03-24T01:24:09.738488+0000","last_became_peered":"2024-03-24T01:24:09.738488+0000","last_unstale":"2024-03-24T01:24:17.812001+0000","last_undegraded":"2024-03-24T01:24:17.812001+0000","last_fullsized":"2024-03-24T01:24:17.812001+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:56:33.895912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811310+0000","last_change":"2024-03-24T01:24:10.753043+0000","last_active":"2024-03-24T01:24:17.811310+0000","last_peered":"2024-03-24T01:24:17.811310+0000","last_clean":"2024-03-24T01:24:17.811310+0000","last_became_active":"2024-03-24T01:24:10.751666+0000","last_became_peered":"2024-03-24T01:24:10.751666+0000","last_unstale":"2024-03-24T01:24:17.811310+0000","last_undegraded":"2024-03-24T01:24:17.811310+0000","last_fullsized":"2024-03-24T01:24:17.811310+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:02:18.904076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.027460+0000","last_change":"2024-03-24T01:24:15.021052+0000","last_active":"2024-03-24T01:24:17.027460+0000","last_peered":"2024-03-24T01:24:17.027460+0000","last_clean":"2024-03-24T01:24:17.027460+0000","last_became_active":"2024-03-24T01:24:15.020449+0000","last_became_peered":"2024-03-24T01:24:15.020449+0000","last_unstale":"2024-03-24T01:24:17.027460+0000","last_undegraded":"2024-03-24T01:24:17.027460+0000","last_fullsized":"2024-03-24T01:24:17.027460+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:26:07.104726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028245+0000","last_change":"2024-03-24T01:24:10.737914+0000","last_active":"2024-03-24T01:24:17.028245+0000","last_peered":"2024-03-24T01:24:17.028245+0000","last_clean":"2024-03-24T01:24:17.028245+0000","last_became_active":"2024-03-24T01:24:10.737652+0000","last_became_peered":"2024-03-24T01:24:10.737652+0000","last_unstale":"2024-03-24T01:24:17.028245+0000","last_undegraded":"2024-03-24T01:24:17.028245+0000","last_fullsized":"2024-03-24T01:24:17.028245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:09:31.705353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.807741+0000","last_change":"2024-03-24T01:24:09.726730+0000","last_active":"2024-03-24T01:24:17.807741+0000","last_peered":"2024-03-24T01:24:17.807741+0000","last_clean":"2024-03-24T01:24:17.807741+0000","last_became_active":"2024-03-24T01:24:09.725266+0000","last_became_peered":"2024-03-24T01:24:09.725266+0000","last_unstale":"2024-03-24T01:24:17.807741+0000","last_undegraded":"2024-03-24T01:24:17.807741+0000","last_fullsized":"2024-03-24T01:24:17.807741+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:27:50.329766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028585+0000","last_change":"2024-03-24T01:24:15.027082+0000","last_active":"2024-03-24T01:24:17.028585+0000","last_peered":"2024-03-24T01:24:17.028585+0000","last_clean":"2024-03-24T01:24:17.028585+0000","last_became_active":"2024-03-24T01:24:15.026296+0000","last_became_peered":"2024-03-24T01:24:15.026296+0000","last_unstale":"2024-03-24T01:24:17.028585+0000","last_undegraded":"2024-03-24T01:24:17.028585+0000","last_fullsized":"2024-03-24T01:24:17.028585+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:55.111215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022103+0000","last_change":"2024-03-24T01:24:10.740733+0000","last_active":"2024-03-24T01:24:17.022103+0000","last_peered":"2024-03-24T01:24:17.022103+0000","last_clean":"2024-03-24T01:24:17.022103+0000","last_became_active":"2024-03-24T01:24:10.740515+0000","last_became_peered":"2024-03-24T01:24:10.740515+0000","last_unstale":"2024-03-24T01:24:17.022103+0000","last_undegraded":"2024-03-24T01:24:17.022103+0000","last_fullsized":"2024-03-24T01:24:17.022103+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:04:48.028091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.738935+0000","last_change":"2024-03-24T01:24:09.725315+0000","last_active":"2024-03-24T01:24:17.738935+0000","last_peered":"2024-03-24T01:24:17.738935+0000","last_clean":"2024-03-24T01:24:17.738935+0000","last_became_active":"2024-03-24T01:24:09.723043+0000","last_became_peered":"2024-03-24T01:24:09.723043+0000","last_unstale":"2024-03-24T01:24:17.738935+0000","last_undegraded":"2024-03-24T01:24:17.738935+0000","last_fullsized":"2024-03-24T01:24:17.738935+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:36:52.007109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.027797+0000","last_change":"2024-03-24T01:24:15.023034+0000","last_active":"2024-03-24T01:24:17.027797+0000","last_peered":"2024-03-24T01:24:17.027797+0000","last_clean":"2024-03-24T01:24:17.027797+0000","last_became_active":"2024-03-24T01:24:15.021144+0000","last_became_peered":"2024-03-24T01:24:15.021144+0000","last_unstale":"2024-03-24T01:24:17.027797+0000","last_undegraded":"2024-03-24T01:24:17.027797+0000","last_fullsized":"2024-03-24T01:24:17.027797+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:00.868691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021560+0000","last_change":"2024-03-24T01:24:10.746085+0000","last_active":"2024-03-24T01:24:17.021560+0000","last_peered":"2024-03-24T01:24:17.021560+0000","last_clean":"2024-03-24T01:24:17.021560+0000","last_became_active":"2024-03-24T01:24:10.745493+0000","last_became_peered":"2024-03-24T01:24:10.745493+0000","last_unstale":"2024-03-24T01:24:17.021560+0000","last_undegraded":"2024-03-24T01:24:17.021560+0000","last_fullsized":"2024-03-24T01:24:17.021560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:19:51.907364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.739444+0000","last_change":"2024-03-24T01:24:09.723035+0000","last_active":"2024-03-24T01:24:17.739444+0000","last_peered":"2024-03-24T01:24:17.739444+0000","last_clean":"2024-03-24T01:24:17.739444+0000","last_became_active":"2024-03-24T01:24:09.722548+0000","last_became_peered":"2024-03-24T01:24:09.722548+0000","last_unstale":"2024-03-24T01:24:17.739444+0000","last_undegraded":"2024-03-24T01:24:17.739444+0000","last_fullsized":"2024-03-24T01:24:17.739444+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:58:17.247099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.741186+0000","last_change":"2024-03-24T01:24:15.658323+0000","last_active":"2024-03-24T01:24:17.741186+0000","last_peered":"2024-03-24T01:24:17.741186+0000","last_clean":"2024-03-24T01:24:17.741186+0000","last_became_active":"2024-03-24T01:24:15.656635+0000","last_became_peered":"2024-03-24T01:24:15.656635+0000","last_unstale":"2024-03-24T01:24:17.741186+0000","last_undegraded":"2024-03-24T01:24:17.741186+0000","last_fullsized":"2024-03-24T01:24:17.741186+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:52:30.401135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028531+0000","last_change":"2024-03-24T01:24:10.734344+0000","last_active":"2024-03-24T01:24:17.028531+0000","last_peered":"2024-03-24T01:24:17.028531+0000","last_clean":"2024-03-24T01:24:17.028531+0000","last_became_active":"2024-03-24T01:24:10.734010+0000","last_became_peered":"2024-03-24T01:24:10.734010+0000","last_unstale":"2024-03-24T01:24:17.028531+0000","last_undegraded":"2024-03-24T01:24:17.028531+0000","last_fullsized":"2024-03-24T01:24:17.028531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:36:53.019192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.808157+0000","last_change":"2024-03-24T01:24:09.729151+0000","last_active":"2024-03-24T01:24:17.808157+0000","last_peered":"2024-03-24T01:24:17.808157+0000","last_clean":"2024-03-24T01:24:17.808157+0000","last_became_active":"2024-03-24T01:24:09.728441+0000","last_became_peered":"2024-03-24T01:24:09.728441+0000","last_unstale":"2024-03-24T01:24:17.808157+0000","last_undegraded":"2024-03-24T01:24:17.808157+0000","last_fullsized":"2024-03-24T01:24:17.808157+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:33.381248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.027681+0000","last_change":"2024-03-24T01:24:15.024436+0000","last_active":"2024-03-24T01:24:17.027681+0000","last_peered":"2024-03-24T01:24:17.027681+0000","last_clean":"2024-03-24T01:24:17.027681+0000","last_became_active":"2024-03-24T01:24:15.022270+0000","last_became_peered":"2024-03-24T01:24:15.022270+0000","last_unstale":"2024-03-24T01:24:17.027681+0000","last_undegraded":"2024-03-24T01:24:17.027681+0000","last_fullsized":"2024-03-24T01:24:17.027681+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:54:28.205416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809625+0000","last_change":"2024-03-24T01:24:10.755973+0000","last_active":"2024-03-24T01:24:17.809625+0000","last_peered":"2024-03-24T01:24:17.809625+0000","last_clean":"2024-03-24T01:24:17.809625+0000","last_became_active":"2024-03-24T01:24:10.755558+0000","last_became_peered":"2024-03-24T01:24:10.755558+0000","last_unstale":"2024-03-24T01:24:17.809625+0000","last_undegraded":"2024-03-24T01:24:17.809625+0000","last_fullsized":"2024-03-24T01:24:17.809625+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:08:13.760239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.031629+0000","last_change":"2024-03-24T01:24:09.724815+0000","last_active":"2024-03-24T01:24:17.031629+0000","last_peered":"2024-03-24T01:24:17.031629+0000","last_clean":"2024-03-24T01:24:17.031629+0000","last_became_active":"2024-03-24T01:24:09.723550+0000","last_became_peered":"2024-03-24T01:24:09.723550+0000","last_unstale":"2024-03-24T01:24:17.031629+0000","last_undegraded":"2024-03-24T01:24:17.031629+0000","last_fullsized":"2024-03-24T01:24:17.031629+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:21:50.689532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028118+0000","last_change":"2024-03-24T01:24:15.651689+0000","last_active":"2024-03-24T01:24:17.028118+0000","last_peered":"2024-03-24T01:24:17.028118+0000","last_clean":"2024-03-24T01:24:17.028118+0000","last_became_active":"2024-03-24T01:24:15.651006+0000","last_became_peered":"2024-03-24T01:24:15.651006+0000","last_unstale":"2024-03-24T01:24:17.028118+0000","last_undegraded":"2024-03-24T01:24:17.028118+0000","last_fullsized":"2024-03-24T01:24:17.028118+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:52:10.223780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.023121+0000","last_change":"2024-03-24T01:24:10.767341+0000","last_active":"2024-03-24T01:24:17.023121+0000","last_peered":"2024-03-24T01:24:17.023121+0000","last_clean":"2024-03-24T01:24:17.023121+0000","last_became_active":"2024-03-24T01:24:10.767024+0000","last_became_peered":"2024-03-24T01:24:10.767024+0000","last_unstale":"2024-03-24T01:24:17.023121+0000","last_undegraded":"2024-03-24T01:24:17.023121+0000","last_fullsized":"2024-03-24T01:24:17.023121+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:35.441197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.738964+0000","last_change":"2024-03-24T01:24:09.716616+0000","last_active":"2024-03-24T01:24:17.738964+0000","last_peered":"2024-03-24T01:24:17.738964+0000","last_clean":"2024-03-24T01:24:17.738964+0000","last_became_active":"2024-03-24T01:24:09.716363+0000","last_became_peered":"2024-03-24T01:24:09.716363+0000","last_unstale":"2024-03-24T01:24:17.738964+0000","last_undegraded":"2024-03-24T01:24:17.738964+0000","last_fullsized":"2024-03-24T01:24:17.738964+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:33.381248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021124+0000","last_change":"2024-03-24T01:24:15.667639+0000","last_active":"2024-03-24T01:24:17.021124+0000","last_peered":"2024-03-24T01:24:17.021124+0000","last_clean":"2024-03-24T01:24:17.021124+0000","last_became_active":"2024-03-24T01:24:15.664579+0000","last_became_peered":"2024-03-24T01:24:15.664579+0000","last_unstale":"2024-03-24T01:24:17.021124+0000","last_undegraded":"2024-03-24T01:24:17.021124+0000","last_fullsized":"2024-03-24T01:24:17.021124+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:43:54.773518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.740372+0000","last_change":"2024-03-24T01:24:10.750277+0000","last_active":"2024-03-24T01:24:17.740372+0000","last_peered":"2024-03-24T01:24:17.740372+0000","last_clean":"2024-03-24T01:24:17.740372+0000","last_became_active":"2024-03-24T01:24:10.748912+0000","last_became_peered":"2024-03-24T01:24:10.748912+0000","last_unstale":"2024-03-24T01:24:17.740372+0000","last_undegraded":"2024-03-24T01:24:17.740372+0000","last_fullsized":"2024-03-24T01:24:17.740372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:19:51.907364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.812643+0000","last_change":"2024-03-24T01:24:09.723002+0000","last_active":"2024-03-24T01:24:17.812643+0000","last_peered":"2024-03-24T01:24:17.812643+0000","last_clean":"2024-03-24T01:24:17.812643+0000","last_became_active":"2024-03-24T01:24:09.722711+0000","last_became_peered":"2024-03-24T01:24:09.722711+0000","last_unstale":"2024-03-24T01:24:17.812643+0000","last_undegraded":"2024-03-24T01:24:17.812643+0000","last_fullsized":"2024-03-24T01:24:17.812643+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:44:56.790262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809628+0000","last_change":"2024-03-24T01:24:15.657482+0000","last_active":"2024-03-24T01:24:17.809628+0000","last_peered":"2024-03-24T01:24:17.809628+0000","last_clean":"2024-03-24T01:24:17.809628+0000","last_became_active":"2024-03-24T01:24:15.656331+0000","last_became_peered":"2024-03-24T01:24:15.656331+0000","last_unstale":"2024-03-24T01:24:17.809628+0000","last_undegraded":"2024-03-24T01:24:17.809628+0000","last_fullsized":"2024-03-24T01:24:17.809628+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:00.868691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811496+0000","last_change":"2024-03-24T01:24:10.753417+0000","last_active":"2024-03-24T01:24:17.811496+0000","last_peered":"2024-03-24T01:24:17.811496+0000","last_clean":"2024-03-24T01:24:17.811496+0000","last_became_active":"2024-03-24T01:24:10.750938+0000","last_became_peered":"2024-03-24T01:24:10.750938+0000","last_unstale":"2024-03-24T01:24:17.811496+0000","last_undegraded":"2024-03-24T01:24:17.811496+0000","last_fullsized":"2024-03-24T01:24:17.811496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:29:34.357876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.740584+0000","last_change":"2024-03-24T01:24:09.716278+0000","last_active":"2024-03-24T01:24:17.740584+0000","last_peered":"2024-03-24T01:24:17.740584+0000","last_clean":"2024-03-24T01:24:17.740584+0000","last_became_active":"2024-03-24T01:24:09.715984+0000","last_became_peered":"2024-03-24T01:24:09.715984+0000","last_unstale":"2024-03-24T01:24:17.740584+0000","last_undegraded":"2024-03-24T01:24:17.740584+0000","last_fullsized":"2024-03-24T01:24:17.740584+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:44:32.492325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.739876+0000","last_change":"2024-03-24T01:24:15.024902+0000","last_active":"2024-03-24T01:24:17.739876+0000","last_peered":"2024-03-24T01:24:17.739876+0000","last_clean":"2024-03-24T01:24:17.739876+0000","last_became_active":"2024-03-24T01:24:15.024379+0000","last_became_peered":"2024-03-24T01:24:15.024379+0000","last_unstale":"2024-03-24T01:24:17.739876+0000","last_undegraded":"2024-03-24T01:24:17.739876+0000","last_fullsized":"2024-03-24T01:24:17.739876+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:26:07.104726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028172+0000","last_change":"2024-03-24T01:24:10.730294+0000","last_active":"2024-03-24T01:24:17.028172+0000","last_peered":"2024-03-24T01:24:17.028172+0000","last_clean":"2024-03-24T01:24:17.028172+0000","last_became_active":"2024-03-24T01:24:10.727422+0000","last_became_peered":"2024-03-24T01:24:10.727422+0000","last_unstale":"2024-03-24T01:24:17.028172+0000","last_undegraded":"2024-03-24T01:24:17.028172+0000","last_fullsized":"2024-03-24T01:24:17.028172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:34.393331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811092+0000","last_change":"2024-03-24T01:24:09.736152+0000","last_active":"2024-03-24T01:24:17.811092+0000","last_peered":"2024-03-24T01:24:17.811092+0000","last_clean":"2024-03-24T01:24:17.811092+0000","last_became_active":"2024-03-24T01:24:09.735870+0000","last_became_peered":"2024-03-24T01:24:09.735870+0000","last_unstale":"2024-03-24T01:24:17.811092+0000","last_undegraded":"2024-03-24T01:24:17.811092+0000","last_fullsized":"2024-03-24T01:24:17.811092+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:50:31.967556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.740556+0000","last_change":"2024-03-24T01:24:15.027111+0000","last_active":"2024-03-24T01:24:17.740556+0000","last_peered":"2024-03-24T01:24:17.740556+0000","last_clean":"2024-03-24T01:24:17.740556+0000","last_became_active":"2024-03-24T01:24:15.025568+0000","last_became_peered":"2024-03-24T01:24:15.025568+0000","last_unstale":"2024-03-24T01:24:17.740556+0000","last_undegraded":"2024-03-24T01:24:17.740556+0000","last_fullsized":"2024-03-24T01:24:17.740556+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:54:28.205416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.739727+0000","last_change":"2024-03-24T01:24:10.744796+0000","last_active":"2024-03-24T01:24:17.739727+0000","last_peered":"2024-03-24T01:24:17.739727+0000","last_clean":"2024-03-24T01:24:17.739727+0000","last_became_active":"2024-03-24T01:24:10.744533+0000","last_became_peered":"2024-03-24T01:24:10.744533+0000","last_unstale":"2024-03-24T01:24:17.739727+0000","last_undegraded":"2024-03-24T01:24:17.739727+0000","last_fullsized":"2024-03-24T01:24:17.739727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:09:31.705353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.807544+0000","last_change":"2024-03-24T01:24:09.717922+0000","last_active":"2024-03-24T01:24:17.807544+0000","last_peered":"2024-03-24T01:24:17.807544+0000","last_clean":"2024-03-24T01:24:17.807544+0000","last_became_active":"2024-03-24T01:24:09.716468+0000","last_became_peered":"2024-03-24T01:24:09.716468+0000","last_unstale":"2024-03-24T01:24:17.807544+0000","last_undegraded":"2024-03-24T01:24:17.807544+0000","last_fullsized":"2024-03-24T01:24:17.807544+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:50:57.480847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.741366+0000","last_change":"2024-03-24T01:24:15.028389+0000","last_active":"2024-03-24T01:24:17.741366+0000","last_peered":"2024-03-24T01:24:17.741366+0000","last_clean":"2024-03-24T01:24:17.741366+0000","last_became_active":"2024-03-24T01:24:15.027052+0000","last_became_peered":"2024-03-24T01:24:15.027052+0000","last_unstale":"2024-03-24T01:24:17.741366+0000","last_undegraded":"2024-03-24T01:24:17.741366+0000","last_fullsized":"2024-03-24T01:24:17.741366+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:00.868691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.808527+0000","last_change":"2024-03-24T01:24:10.726234+0000","last_active":"2024-03-24T01:24:17.808527+0000","last_peered":"2024-03-24T01:24:17.808527+0000","last_clean":"2024-03-24T01:24:17.808527+0000","last_became_active":"2024-03-24T01:24:10.725471+0000","last_became_peered":"2024-03-24T01:24:10.725471+0000","last_unstale":"2024-03-24T01:24:17.808527+0000","last_undegraded":"2024-03-24T01:24:17.808527+0000","last_fullsized":"2024-03-24T01:24:17.808527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:55:14.169541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809601+0000","last_change":"2024-03-24T01:24:09.726977+0000","last_active":"2024-03-24T01:24:17.809601+0000","last_peered":"2024-03-24T01:24:17.809601+0000","last_clean":"2024-03-24T01:24:17.809601+0000","last_became_active":"2024-03-24T01:24:09.726515+0000","last_became_peered":"2024-03-24T01:24:09.726515+0000","last_unstale":"2024-03-24T01:24:17.809601+0000","last_undegraded":"2024-03-24T01:24:17.809601+0000","last_fullsized":"2024-03-24T01:24:17.809601+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:58:17.247099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809443+0000","last_change":"2024-03-24T01:24:15.662691+0000","last_active":"2024-03-24T01:24:17.809443+0000","last_peered":"2024-03-24T01:24:17.809443+0000","last_clean":"2024-03-24T01:24:17.809443+0000","last_became_active":"2024-03-24T01:24:15.657278+0000","last_became_peered":"2024-03-24T01:24:15.657278+0000","last_unstale":"2024-03-24T01:24:17.809443+0000","last_undegraded":"2024-03-24T01:24:17.809443+0000","last_fullsized":"2024-03-24T01:24:17.809443+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:54:28.205416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021915+0000","last_change":"2024-03-24T01:24:10.744706+0000","last_active":"2024-03-24T01:24:17.021915+0000","last_peered":"2024-03-24T01:24:17.021915+0000","last_clean":"2024-03-24T01:24:17.021915+0000","last_became_active":"2024-03-24T01:24:10.744433+0000","last_became_peered":"2024-03-24T01:24:10.744433+0000","last_unstale":"2024-03-24T01:24:17.021915+0000","last_undegraded":"2024-03-24T01:24:17.021915+0000","last_fullsized":"2024-03-24T01:24:17.021915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:09:31.705353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809513+0000","last_change":"2024-03-24T01:24:09.737525+0000","last_active":"2024-03-24T01:24:17.809513+0000","last_peered":"2024-03-24T01:24:17.809513+0000","last_clean":"2024-03-24T01:24:17.809513+0000","last_became_active":"2024-03-24T01:24:09.737206+0000","last_became_peered":"2024-03-24T01:24:09.737206+0000","last_unstale":"2024-03-24T01:24:17.809513+0000","last_undegraded":"2024-03-24T01:24:17.809513+0000","last_fullsized":"2024-03-24T01:24:17.809513+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:44:32.492325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":179,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.808453+0000","last_change":"2024-03-24T01:24:01.545855+0000","last_active":"2024-03-24T01:24:17.808453+0000","last_peered":"2024-03-24T01:24:17.808453+0000","last_clean":"2024-03-24T01:24:17.808453+0000","last_became_active":"2024-03-24T01:24:01.543811+0000","last_became_peered":"2024-03-24T01:24:01.543811+0000","last_unstale":"2024-03-24T01:24:17.808453+0000","last_undegraded":"2024-03-24T01:24:17.808453+0000","last_fullsized":"2024-03-24T01:24:17.808453+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_clean_scrub_stamp":"2024-03-24T01:23:59.861564+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:03.682631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.031327+0000","last_change":"2024-03-24T01:24:15.656774+0000","last_active":"2024-03-24T01:24:17.031327+0000","last_peered":"2024-03-24T01:24:17.031327+0000","last_clean":"2024-03-24T01:24:17.031327+0000","last_became_active":"2024-03-24T01:24:15.656377+0000","last_became_peered":"2024-03-24T01:24:15.656377+0000","last_unstale":"2024-03-24T01:24:17.031327+0000","last_undegraded":"2024-03-24T01:24:17.031327+0000","last_fullsized":"2024-03-24T01:24:17.031327+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:34:04.384239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.812176+0000","last_change":"2024-03-24T01:24:10.766537+0000","last_active":"2024-03-24T01:24:17.812176+0000","last_peered":"2024-03-24T01:24:17.812176+0000","last_clean":"2024-03-24T01:24:17.812176+0000","last_became_active":"2024-03-24T01:24:10.765713+0000","last_became_peered":"2024-03-24T01:24:10.765713+0000","last_unstale":"2024-03-24T01:24:17.812176+0000","last_undegraded":"2024-03-24T01:24:17.812176+0000","last_fullsized":"2024-03-24T01:24:17.812176+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:18:24.974821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.810980+0000","last_change":"2024-03-24T01:24:09.731795+0000","last_active":"2024-03-24T01:24:17.810980+0000","last_peered":"2024-03-24T01:24:17.810980+0000","last_clean":"2024-03-24T01:24:17.810980+0000","last_became_active":"2024-03-24T01:24:09.731547+0000","last_became_peered":"2024-03-24T01:24:09.731547+0000","last_unstale":"2024-03-24T01:24:17.810980+0000","last_undegraded":"2024-03-24T01:24:17.810980+0000","last_fullsized":"2024-03-24T01:24:17.810980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:31:46.512174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028049+0000","last_change":"2024-03-24T01:24:15.656206+0000","last_active":"2024-03-24T01:24:17.028049+0000","last_peered":"2024-03-24T01:24:17.028049+0000","last_clean":"2024-03-24T01:24:17.028049+0000","last_became_active":"2024-03-24T01:24:15.655557+0000","last_became_peered":"2024-03-24T01:24:15.655557+0000","last_unstale":"2024-03-24T01:24:17.028049+0000","last_undegraded":"2024-03-24T01:24:17.028049+0000","last_fullsized":"2024-03-24T01:24:17.028049+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:18:29.263700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811389+0000","last_change":"2024-03-24T01:24:10.750476+0000","last_active":"2024-03-24T01:24:17.811389+0000","last_peered":"2024-03-24T01:24:17.811389+0000","last_clean":"2024-03-24T01:24:17.811389+0000","last_became_active":"2024-03-24T01:24:10.750171+0000","last_became_peered":"2024-03-24T01:24:10.750171+0000","last_unstale":"2024-03-24T01:24:17.811389+0000","last_undegraded":"2024-03-24T01:24:17.811389+0000","last_fullsized":"2024-03-24T01:24:17.811389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:37:21.153754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.739005+0000","last_change":"2024-03-24T01:24:09.718128+0000","last_active":"2024-03-24T01:24:17.739005+0000","last_peered":"2024-03-24T01:24:17.739005+0000","last_clean":"2024-03-24T01:24:17.739005+0000","last_became_active":"2024-03-24T01:24:09.717669+0000","last_became_peered":"2024-03-24T01:24:09.717669+0000","last_unstale":"2024-03-24T01:24:17.739005+0000","last_undegraded":"2024-03-24T01:24:17.739005+0000","last_fullsized":"2024-03-24T01:24:17.739005+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:21:50.689532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.810011+0000","last_change":"2024-03-24T01:24:15.033291+0000","last_active":"2024-03-24T01:24:17.810011+0000","last_peered":"2024-03-24T01:24:17.810011+0000","last_clean":"2024-03-24T01:24:17.810011+0000","last_became_active":"2024-03-24T01:24:15.032790+0000","last_became_peered":"2024-03-24T01:24:15.032790+0000","last_unstale":"2024-03-24T01:24:17.810011+0000","last_undegraded":"2024-03-24T01:24:17.810011+0000","last_fullsized":"2024-03-24T01:24:17.810011+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:52:10.223780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.813313+0000","last_change":"2024-03-24T01:24:10.765266+0000","last_active":"2024-03-24T01:24:17.813313+0000","last_peered":"2024-03-24T01:24:17.813313+0000","last_clean":"2024-03-24T01:24:17.813313+0000","last_became_active":"2024-03-24T01:24:10.764758+0000","last_became_peered":"2024-03-24T01:24:10.764758+0000","last_unstale":"2024-03-24T01:24:17.813313+0000","last_undegraded":"2024-03-24T01:24:17.813313+0000","last_fullsized":"2024-03-24T01:24:17.813313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:34:00.095359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.807653+0000","last_change":"2024-03-24T01:24:09.714389+0000","last_active":"2024-03-24T01:24:17.807653+0000","last_peered":"2024-03-24T01:24:17.807653+0000","last_clean":"2024-03-24T01:24:17.807653+0000","last_became_active":"2024-03-24T01:24:09.713985+0000","last_became_peered":"2024-03-24T01:24:09.713985+0000","last_unstale":"2024-03-24T01:24:17.807653+0000","last_undegraded":"2024-03-24T01:24:17.807653+0000","last_fullsized":"2024-03-24T01:24:17.807653+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:50:31.967556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809004+0000","last_change":"2024-03-24T01:24:15.656465+0000","last_active":"2024-03-24T01:24:17.809004+0000","last_peered":"2024-03-24T01:24:17.809004+0000","last_clean":"2024-03-24T01:24:17.809004+0000","last_became_active":"2024-03-24T01:24:15.655849+0000","last_became_peered":"2024-03-24T01:24:15.655849+0000","last_unstale":"2024-03-24T01:24:17.809004+0000","last_undegraded":"2024-03-24T01:24:17.809004+0000","last_fullsized":"2024-03-24T01:24:17.809004+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:00.868691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.812175+0000","last_change":"2024-03-24T01:24:10.767829+0000","last_active":"2024-03-24T01:24:17.812175+0000","last_peered":"2024-03-24T01:24:17.812175+0000","last_clean":"2024-03-24T01:24:17.812175+0000","last_became_active":"2024-03-24T01:24:10.766346+0000","last_became_peered":"2024-03-24T01:24:10.766346+0000","last_unstale":"2024-03-24T01:24:17.812175+0000","last_undegraded":"2024-03-24T01:24:17.812175+0000","last_fullsized":"2024-03-24T01:24:17.812175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:24.434151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028899+0000","last_change":"2024-03-24T01:24:09.723282+0000","last_active":"2024-03-24T01:24:17.028899+0000","last_peered":"2024-03-24T01:24:17.028899+0000","last_clean":"2024-03-24T01:24:17.028899+0000","last_became_active":"2024-03-24T01:24:09.722727+0000","last_became_peered":"2024-03-24T01:24:09.722727+0000","last_unstale":"2024-03-24T01:24:17.028899+0000","last_undegraded":"2024-03-24T01:24:17.028899+0000","last_fullsized":"2024-03-24T01:24:17.028899+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:40:14.201223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.027169+0000","last_change":"2024-03-24T01:24:15.027468+0000","last_active":"2024-03-24T01:24:17.027169+0000","last_peered":"2024-03-24T01:24:17.027169+0000","last_clean":"2024-03-24T01:24:17.027169+0000","last_became_active":"2024-03-24T01:24:15.027086+0000","last_became_peered":"2024-03-24T01:24:15.027086+0000","last_unstale":"2024-03-24T01:24:17.027169+0000","last_undegraded":"2024-03-24T01:24:17.027169+0000","last_fullsized":"2024-03-24T01:24:17.027169+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:34:04.384239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809223+0000","last_change":"2024-03-24T01:24:10.756853+0000","last_active":"2024-03-24T01:24:17.809223+0000","last_peered":"2024-03-24T01:24:17.809223+0000","last_clean":"2024-03-24T01:24:17.809223+0000","last_became_active":"2024-03-24T01:24:10.756575+0000","last_became_peered":"2024-03-24T01:24:10.756575+0000","last_unstale":"2024-03-24T01:24:17.809223+0000","last_undegraded":"2024-03-24T01:24:17.809223+0000","last_fullsized":"2024-03-24T01:24:17.809223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:26:02.815847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.808921+0000","last_change":"2024-03-24T01:24:09.720936+0000","last_active":"2024-03-24T01:24:17.808921+0000","last_peered":"2024-03-24T01:24:17.808921+0000","last_clean":"2024-03-24T01:24:17.808921+0000","last_became_active":"2024-03-24T01:24:09.720598+0000","last_became_peered":"2024-03-24T01:24:09.720598+0000","last_unstale":"2024-03-24T01:24:17.808921+0000","last_undegraded":"2024-03-24T01:24:17.808921+0000","last_fullsized":"2024-03-24T01:24:17.808921+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:44:32.492325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809525+0000","last_change":"2024-03-24T01:24:15.027473+0000","last_active":"2024-03-24T01:24:17.809525+0000","last_peered":"2024-03-24T01:24:17.809525+0000","last_clean":"2024-03-24T01:24:17.809525+0000","last_became_active":"2024-03-24T01:24:15.025176+0000","last_became_peered":"2024-03-24T01:24:15.025176+0000","last_unstale":"2024-03-24T01:24:17.809525+0000","last_undegraded":"2024-03-24T01:24:17.809525+0000","last_fullsized":"2024-03-24T01:24:17.809525+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:52:30.401135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.030178+0000","last_change":"2024-03-24T01:24:10.737343+0000","last_active":"2024-03-24T01:24:17.030178+0000","last_peered":"2024-03-24T01:24:17.030178+0000","last_clean":"2024-03-24T01:24:17.030178+0000","last_became_active":"2024-03-24T01:24:10.737040+0000","last_became_peered":"2024-03-24T01:24:10.737040+0000","last_unstale":"2024-03-24T01:24:17.030178+0000","last_undegraded":"2024-03-24T01:24:17.030178+0000","last_fullsized":"2024-03-24T01:24:17.030178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:54:57.105205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.029522+0000","last_change":"2024-03-24T01:24:09.726911+0000","last_active":"2024-03-24T01:24:17.029522+0000","last_peered":"2024-03-24T01:24:17.029522+0000","last_clean":"2024-03-24T01:24:17.029522+0000","last_became_active":"2024-03-24T01:24:09.726578+0000","last_became_peered":"2024-03-24T01:24:09.726578+0000","last_unstale":"2024-03-24T01:24:17.029522+0000","last_undegraded":"2024-03-24T01:24:17.029522+0000","last_fullsized":"2024-03-24T01:24:17.029522+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:44:32.492325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.023074+0000","last_change":"2024-03-24T01:24:15.660631+0000","last_active":"2024-03-24T01:24:17.023074+0000","last_peered":"2024-03-24T01:24:17.023074+0000","last_clean":"2024-03-24T01:24:17.023074+0000","last_became_active":"2024-03-24T01:24:15.656242+0000","last_became_peered":"2024-03-24T01:24:15.656242+0000","last_unstale":"2024-03-24T01:24:17.023074+0000","last_undegraded":"2024-03-24T01:24:17.023074+0000","last_fullsized":"2024-03-24T01:24:17.023074+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:10:41.483921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811861+0000","last_change":"2024-03-24T01:24:10.762123+0000","last_active":"2024-03-24T01:24:17.811861+0000","last_peered":"2024-03-24T01:24:17.811861+0000","last_clean":"2024-03-24T01:24:17.811861+0000","last_became_active":"2024-03-24T01:24:10.761599+0000","last_became_peered":"2024-03-24T01:24:10.761599+0000","last_unstale":"2024-03-24T01:24:17.811861+0000","last_undegraded":"2024-03-24T01:24:17.811861+0000","last_fullsized":"2024-03-24T01:24:17.811861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:54:23.916536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809077+0000","last_change":"2024-03-24T01:24:09.735604+0000","last_active":"2024-03-24T01:24:17.809077+0000","last_peered":"2024-03-24T01:24:17.809077+0000","last_clean":"2024-03-24T01:24:17.809077+0000","last_became_active":"2024-03-24T01:24:09.735373+0000","last_became_peered":"2024-03-24T01:24:09.735373+0000","last_unstale":"2024-03-24T01:24:17.809077+0000","last_undegraded":"2024-03-24T01:24:17.809077+0000","last_fullsized":"2024-03-24T01:24:17.809077+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:36:52.007109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022699+0000","last_change":"2024-03-24T01:24:15.660466+0000","last_active":"2024-03-24T01:24:17.022699+0000","last_peered":"2024-03-24T01:24:17.022699+0000","last_clean":"2024-03-24T01:24:17.022699+0000","last_became_active":"2024-03-24T01:24:15.659265+0000","last_became_peered":"2024-03-24T01:24:15.659265+0000","last_unstale":"2024-03-24T01:24:17.022699+0000","last_undegraded":"2024-03-24T01:24:17.022699+0000","last_fullsized":"2024-03-24T01:24:17.022699+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:56.567936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809258+0000","last_change":"2024-03-24T01:24:10.762021+0000","last_active":"2024-03-24T01:24:17.809258+0000","last_peered":"2024-03-24T01:24:17.809258+0000","last_clean":"2024-03-24T01:24:17.809258+0000","last_became_active":"2024-03-24T01:24:10.761759+0000","last_became_peered":"2024-03-24T01:24:10.761759+0000","last_unstale":"2024-03-24T01:24:17.809258+0000","last_undegraded":"2024-03-24T01:24:17.809258+0000","last_fullsized":"2024-03-24T01:24:17.809258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:49:42.059380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.807512+0000","last_change":"2024-03-24T01:24:09.726149+0000","last_active":"2024-03-24T01:24:17.807512+0000","last_peered":"2024-03-24T01:24:17.807512+0000","last_clean":"2024-03-24T01:24:17.807512+0000","last_became_active":"2024-03-24T01:24:09.725594+0000","last_became_peered":"2024-03-24T01:24:09.725594+0000","last_unstale":"2024-03-24T01:24:17.807512+0000","last_undegraded":"2024-03-24T01:24:17.807512+0000","last_fullsized":"2024-03-24T01:24:17.807512+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:28:34.408651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811101+0000","last_change":"2024-03-24T01:24:15.655846+0000","last_active":"2024-03-24T01:24:17.811101+0000","last_peered":"2024-03-24T01:24:17.811101+0000","last_clean":"2024-03-24T01:24:17.811101+0000","last_became_active":"2024-03-24T01:24:15.655258+0000","last_became_peered":"2024-03-24T01:24:15.655258+0000","last_unstale":"2024-03-24T01:24:17.811101+0000","last_undegraded":"2024-03-24T01:24:17.811101+0000","last_fullsized":"2024-03-24T01:24:17.811101+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:52:10.223780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811833+0000","last_change":"2024-03-24T01:24:10.769350+0000","last_active":"2024-03-24T01:24:17.811833+0000","last_peered":"2024-03-24T01:24:17.811833+0000","last_clean":"2024-03-24T01:24:17.811833+0000","last_became_active":"2024-03-24T01:24:10.766162+0000","last_became_peered":"2024-03-24T01:24:10.766162+0000","last_unstale":"2024-03-24T01:24:17.811833+0000","last_undegraded":"2024-03-24T01:24:17.811833+0000","last_fullsized":"2024-03-24T01:24:17.811833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:38:07.266851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'10","reported_seq":34,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:28.484782+0000","last_change":"2024-03-24T01:24:09.716226+0000","last_active":"2024-03-24T01:24:28.484782+0000","last_peered":"2024-03-24T01:24:28.484782+0000","last_clean":"2024-03-24T01:24:28.484782+0000","last_became_active":"2024-03-24T01:24:09.715772+0000","last_became_peered":"2024-03-24T01:24:09.715772+0000","last_unstale":"2024-03-24T01:24:28.484782+0000","last_undegraded":"2024-03-24T01:24:28.484782+0000","last_fullsized":"2024-03-24T01:24:28.484782+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:50:31.967556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.740168+0000","last_change":"2024-03-24T01:24:15.024014+0000","last_active":"2024-03-24T01:24:17.740168+0000","last_peered":"2024-03-24T01:24:17.740168+0000","last_clean":"2024-03-24T01:24:17.740168+0000","last_became_active":"2024-03-24T01:24:15.023619+0000","last_became_peered":"2024-03-24T01:24:15.023619+0000","last_unstale":"2024-03-24T01:24:17.740168+0000","last_undegraded":"2024-03-24T01:24:17.740168+0000","last_fullsized":"2024-03-24T01:24:17.740168+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:45:17.532730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028200+0000","last_change":"2024-03-24T01:24:10.745876+0000","last_active":"2024-03-24T01:24:17.028200+0000","last_peered":"2024-03-24T01:24:17.028200+0000","last_clean":"2024-03-24T01:24:17.028200+0000","last_became_active":"2024-03-24T01:24:10.745539+0000","last_became_peered":"2024-03-24T01:24:10.745539+0000","last_unstale":"2024-03-24T01:24:17.028200+0000","last_undegraded":"2024-03-24T01:24:17.028200+0000","last_fullsized":"2024-03-24T01:24:17.028200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:56:34.907995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.740552+0000","last_change":"2024-03-24T01:24:09.736074+0000","last_active":"2024-03-24T01:24:17.740552+0000","last_peered":"2024-03-24T01:24:17.740552+0000","last_clean":"2024-03-24T01:24:17.740552+0000","last_became_active":"2024-03-24T01:24:09.735807+0000","last_became_peered":"2024-03-24T01:24:09.735807+0000","last_unstale":"2024-03-24T01:24:17.740552+0000","last_undegraded":"2024-03-24T01:24:17.740552+0000","last_fullsized":"2024-03-24T01:24:17.740552+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:50:31.967556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.027828+0000","last_change":"2024-03-24T01:24:15.031909+0000","last_active":"2024-03-24T01:24:17.027828+0000","last_peered":"2024-03-24T01:24:17.027828+0000","last_clean":"2024-03-24T01:24:17.027828+0000","last_became_active":"2024-03-24T01:24:15.031560+0000","last_became_peered":"2024-03-24T01:24:15.031560+0000","last_unstale":"2024-03-24T01:24:17.027828+0000","last_undegraded":"2024-03-24T01:24:17.027828+0000","last_fullsized":"2024-03-24T01:24:17.027828+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:52:30.401135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811414+0000","last_change":"2024-03-24T01:24:10.753950+0000","last_active":"2024-03-24T01:24:17.811414+0000","last_peered":"2024-03-24T01:24:17.811414+0000","last_clean":"2024-03-24T01:24:17.811414+0000","last_became_active":"2024-03-24T01:24:10.742209+0000","last_became_peered":"2024-03-24T01:24:10.742209+0000","last_unstale":"2024-03-24T01:24:17.811414+0000","last_undegraded":"2024-03-24T01:24:17.811414+0000","last_fullsized":"2024-03-24T01:24:17.811414+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:17.138453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809846+0000","last_change":"2024-03-24T01:24:09.735745+0000","last_active":"2024-03-24T01:24:17.809846+0000","last_peered":"2024-03-24T01:24:17.809846+0000","last_clean":"2024-03-24T01:24:17.809846+0000","last_became_active":"2024-03-24T01:24:09.735438+0000","last_became_peered":"2024-03-24T01:24:09.735438+0000","last_unstale":"2024-03-24T01:24:17.809846+0000","last_undegraded":"2024-03-24T01:24:17.809846+0000","last_fullsized":"2024-03-24T01:24:17.809846+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:58:17.247099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021471+0000","last_change":"2024-03-24T01:24:15.014619+0000","last_active":"2024-03-24T01:24:17.021471+0000","last_peered":"2024-03-24T01:24:17.021471+0000","last_clean":"2024-03-24T01:24:17.021471+0000","last_became_active":"2024-03-24T01:24:15.010859+0000","last_became_peered":"2024-03-24T01:24:15.010859+0000","last_unstale":"2024-03-24T01:24:17.021471+0000","last_undegraded":"2024-03-24T01:24:17.021471+0000","last_fullsized":"2024-03-24T01:24:17.021471+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:45:23.567638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811149+0000","last_change":"2024-03-24T01:24:10.767592+0000","last_active":"2024-03-24T01:24:17.811149+0000","last_peered":"2024-03-24T01:24:17.811149+0000","last_clean":"2024-03-24T01:24:17.811149+0000","last_became_active":"2024-03-24T01:24:10.767264+0000","last_became_peered":"2024-03-24T01:24:10.767264+0000","last_unstale":"2024-03-24T01:24:17.811149+0000","last_undegraded":"2024-03-24T01:24:17.811149+0000","last_fullsized":"2024-03-24T01:24:17.811149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:18:24.974821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.031354+0000","last_change":"2024-03-24T01:24:09.738634+0000","last_active":"2024-03-24T01:24:17.031354+0000","last_peered":"2024-03-24T01:24:17.031354+0000","last_clean":"2024-03-24T01:24:17.031354+0000","last_became_active":"2024-03-24T01:24:09.738308+0000","last_became_peered":"2024-03-24T01:24:09.738308+0000","last_unstale":"2024-03-24T01:24:17.031354+0000","last_undegraded":"2024-03-24T01:24:17.031354+0000","last_fullsized":"2024-03-24T01:24:17.031354+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:36:52.007109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021706+0000","last_change":"2024-03-24T01:24:15.666864+0000","last_active":"2024-03-24T01:24:17.021706+0000","last_peered":"2024-03-24T01:24:17.021706+0000","last_clean":"2024-03-24T01:24:17.021706+0000","last_became_active":"2024-03-24T01:24:15.659581+0000","last_became_peered":"2024-03-24T01:24:15.659581+0000","last_unstale":"2024-03-24T01:24:17.021706+0000","last_undegraded":"2024-03-24T01:24:17.021706+0000","last_fullsized":"2024-03-24T01:24:17.021706+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:40:09.145961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809258+0000","last_change":"2024-03-24T01:24:15.657385+0000","last_active":"2024-03-24T01:24:17.809258+0000","last_peered":"2024-03-24T01:24:17.809258+0000","last_clean":"2024-03-24T01:24:17.809258+0000","last_became_active":"2024-03-24T01:24:15.656275+0000","last_became_peered":"2024-03-24T01:24:15.656275+0000","last_unstale":"2024-03-24T01:24:17.809258+0000","last_undegraded":"2024-03-24T01:24:17.809258+0000","last_fullsized":"2024-03-24T01:24:17.809258+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:45:17.532730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.739356+0000","last_change":"2024-03-24T01:24:10.750827+0000","last_active":"2024-03-24T01:24:17.739356+0000","last_peered":"2024-03-24T01:24:17.739356+0000","last_clean":"2024-03-24T01:24:17.739356+0000","last_became_active":"2024-03-24T01:24:10.750522+0000","last_became_peered":"2024-03-24T01:24:10.750522+0000","last_unstale":"2024-03-24T01:24:17.739356+0000","last_undegraded":"2024-03-24T01:24:17.739356+0000","last_fullsized":"2024-03-24T01:24:17.739356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:04:48.028091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.808340+0000","last_change":"2024-03-24T01:24:09.730771+0000","last_active":"2024-03-24T01:24:17.808340+0000","last_peered":"2024-03-24T01:24:17.808340+0000","last_clean":"2024-03-24T01:24:17.808340+0000","last_became_active":"2024-03-24T01:24:09.730424+0000","last_became_peered":"2024-03-24T01:24:09.730424+0000","last_unstale":"2024-03-24T01:24:17.808340+0000","last_undegraded":"2024-03-24T01:24:17.808340+0000","last_fullsized":"2024-03-24T01:24:17.808340+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:50:57.480847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.031001+0000","last_change":"2024-03-24T01:24:15.029656+0000","last_active":"2024-03-24T01:24:17.031001+0000","last_peered":"2024-03-24T01:24:17.031001+0000","last_clean":"2024-03-24T01:24:17.031001+0000","last_became_active":"2024-03-24T01:24:15.029240+0000","last_became_peered":"2024-03-24T01:24:15.029240+0000","last_unstale":"2024-03-24T01:24:17.031001+0000","last_undegraded":"2024-03-24T01:24:17.031001+0000","last_fullsized":"2024-03-24T01:24:17.031001+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:28.723031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021468+0000","last_change":"2024-03-24T01:24:10.750421+0000","last_active":"2024-03-24T01:24:17.021468+0000","last_peered":"2024-03-24T01:24:17.021468+0000","last_clean":"2024-03-24T01:24:17.021468+0000","last_became_active":"2024-03-24T01:24:10.750160+0000","last_became_peered":"2024-03-24T01:24:10.750160+0000","last_unstale":"2024-03-24T01:24:17.021468+0000","last_undegraded":"2024-03-24T01:24:17.021468+0000","last_fullsized":"2024-03-24T01:24:17.021468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:54:57.105205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022107+0000","last_change":"2024-03-24T01:24:09.722993+0000","last_active":"2024-03-24T01:24:17.022107+0000","last_peered":"2024-03-24T01:24:17.022107+0000","last_clean":"2024-03-24T01:24:17.022107+0000","last_became_active":"2024-03-24T01:24:09.720133+0000","last_became_peered":"2024-03-24T01:24:09.720133+0000","last_unstale":"2024-03-24T01:24:17.022107+0000","last_undegraded":"2024-03-24T01:24:17.022107+0000","last_fullsized":"2024-03-24T01:24:17.022107+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:28:34.408651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.030554+0000","last_change":"2024-03-24T01:24:09.726157+0000","last_active":"2024-03-24T01:24:17.030554+0000","last_peered":"2024-03-24T01:24:17.030554+0000","last_clean":"2024-03-24T01:24:17.030554+0000","last_became_active":"2024-03-24T01:24:09.724571+0000","last_became_peered":"2024-03-24T01:24:09.724571+0000","last_unstale":"2024-03-24T01:24:17.030554+0000","last_undegraded":"2024-03-24T01:24:17.030554+0000","last_fullsized":"2024-03-24T01:24:17.030554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:33.381248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.810098+0000","last_change":"2024-03-24T01:24:10.726759+0000","last_active":"2024-03-24T01:24:17.810098+0000","last_peered":"2024-03-24T01:24:17.810098+0000","last_clean":"2024-03-24T01:24:17.810098+0000","last_became_active":"2024-03-24T01:24:10.726133+0000","last_became_peered":"2024-03-24T01:24:10.726133+0000","last_unstale":"2024-03-24T01:24:17.810098+0000","last_undegraded":"2024-03-24T01:24:17.810098+0000","last_fullsized":"2024-03-24T01:24:17.810098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:46:04.235199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.810763+0000","last_change":"2024-03-24T01:24:15.026408+0000","last_active":"2024-03-24T01:24:17.810763+0000","last_peered":"2024-03-24T01:24:17.810763+0000","last_clean":"2024-03-24T01:24:17.810763+0000","last_became_active":"2024-03-24T01:24:15.026038+0000","last_became_peered":"2024-03-24T01:24:15.026038+0000","last_unstale":"2024-03-24T01:24:17.810763+0000","last_undegraded":"2024-03-24T01:24:17.810763+0000","last_fullsized":"2024-03-24T01:24:17.810763+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:55:18.458421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022088+0000","last_change":"2024-03-24T01:24:09.722548+0000","last_active":"2024-03-24T01:24:17.022088+0000","last_peered":"2024-03-24T01:24:17.022088+0000","last_clean":"2024-03-24T01:24:17.022088+0000","last_became_active":"2024-03-24T01:24:09.721937+0000","last_became_peered":"2024-03-24T01:24:09.721937+0000","last_unstale":"2024-03-24T01:24:17.022088+0000","last_undegraded":"2024-03-24T01:24:17.022088+0000","last_fullsized":"2024-03-24T01:24:17.022088+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:50:57.480847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.027732+0000","last_change":"2024-03-24T01:24:10.743972+0000","last_active":"2024-03-24T01:24:17.027732+0000","last_peered":"2024-03-24T01:24:17.027732+0000","last_clean":"2024-03-24T01:24:17.027732+0000","last_became_active":"2024-03-24T01:24:10.743700+0000","last_became_peered":"2024-03-24T01:24:10.743700+0000","last_unstale":"2024-03-24T01:24:17.027732+0000","last_undegraded":"2024-03-24T01:24:17.027732+0000","last_fullsized":"2024-03-24T01:24:17.027732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:21:51.701615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022373+0000","last_change":"2024-03-24T01:24:15.023995+0000","last_active":"2024-03-24T01:24:17.022373+0000","last_peered":"2024-03-24T01:24:17.022373+0000","last_clean":"2024-03-24T01:24:17.022373+0000","last_became_active":"2024-03-24T01:24:15.023574+0000","last_became_peered":"2024-03-24T01:24:15.023574+0000","last_unstale":"2024-03-24T01:24:17.022373+0000","last_undegraded":"2024-03-24T01:24:17.022373+0000","last_fullsized":"2024-03-24T01:24:17.022373+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:26:01.912499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.029925+0000","last_change":"2024-03-24T01:24:10.740908+0000","last_active":"2024-03-24T01:24:17.029925+0000","last_peered":"2024-03-24T01:24:17.029925+0000","last_clean":"2024-03-24T01:24:17.029925+0000","last_became_active":"2024-03-24T01:24:10.740630+0000","last_became_peered":"2024-03-24T01:24:10.740630+0000","last_unstale":"2024-03-24T01:24:17.029925+0000","last_undegraded":"2024-03-24T01:24:17.029925+0000","last_fullsized":"2024-03-24T01:24:17.029925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:04:48.028091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809885+0000","last_change":"2024-03-24T01:24:10.743826+0000","last_active":"2024-03-24T01:24:17.809885+0000","last_peered":"2024-03-24T01:24:17.809885+0000","last_clean":"2024-03-24T01:24:17.809885+0000","last_became_active":"2024-03-24T01:24:10.743349+0000","last_became_peered":"2024-03-24T01:24:10.743349+0000","last_unstale":"2024-03-24T01:24:17.809885+0000","last_undegraded":"2024-03-24T01:24:17.809885+0000","last_fullsized":"2024-03-24T01:24:17.809885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:56:34.907995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.810496+0000","last_change":"2024-03-24T01:24:10.755545+0000","last_active":"2024-03-24T01:24:17.810496+0000","last_peered":"2024-03-24T01:24:17.810496+0000","last_clean":"2024-03-24T01:24:17.810496+0000","last_became_active":"2024-03-24T01:24:10.755147+0000","last_became_peered":"2024-03-24T01:24:10.755147+0000","last_unstale":"2024-03-24T01:24:17.810496+0000","last_undegraded":"2024-03-24T01:24:17.810496+0000","last_fullsized":"2024-03-24T01:24:17.810496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:35.441197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809878+0000","last_change":"2024-03-24T01:24:10.746109+0000","last_active":"2024-03-24T01:24:17.809878+0000","last_peered":"2024-03-24T01:24:17.809878+0000","last_clean":"2024-03-24T01:24:17.809878+0000","last_became_active":"2024-03-24T01:24:10.745809+0000","last_became_peered":"2024-03-24T01:24:10.745809+0000","last_unstale":"2024-03-24T01:24:17.809878+0000","last_undegraded":"2024-03-24T01:24:17.809878+0000","last_fullsized":"2024-03-24T01:24:17.809878+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:09:31.705353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022898+0000","last_change":"2024-03-24T01:24:10.752362+0000","last_active":"2024-03-24T01:24:17.022898+0000","last_peered":"2024-03-24T01:24:17.022898+0000","last_clean":"2024-03-24T01:24:17.022898+0000","last_became_active":"2024-03-24T01:24:10.752066+0000","last_became_peered":"2024-03-24T01:24:10.752066+0000","last_unstale":"2024-03-24T01:24:17.022898+0000","last_undegraded":"2024-03-24T01:24:17.022898+0000","last_fullsized":"2024-03-24T01:24:17.022898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:08:13.760239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.740402+0000","last_change":"2024-03-24T01:24:10.751536+0000","last_active":"2024-03-24T01:24:17.740402+0000","last_peered":"2024-03-24T01:24:17.740402+0000","last_clean":"2024-03-24T01:24:17.740402+0000","last_became_active":"2024-03-24T01:24:10.751260+0000","last_became_peered":"2024-03-24T01:24:10.751260+0000","last_unstale":"2024-03-24T01:24:17.740402+0000","last_undegraded":"2024-03-24T01:24:17.740402+0000","last_fullsized":"2024-03-24T01:24:17.740402+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:17.138453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.031836+0000","last_change":"2024-03-24T01:24:10.763037+0000","last_active":"2024-03-24T01:24:17.031836+0000","last_peered":"2024-03-24T01:24:17.031836+0000","last_clean":"2024-03-24T01:24:17.031836+0000","last_became_active":"2024-03-24T01:24:10.762718+0000","last_became_peered":"2024-03-24T01:24:10.762718+0000","last_unstale":"2024-03-24T01:24:17.031836+0000","last_undegraded":"2024-03-24T01:24:17.031836+0000","last_fullsized":"2024-03-24T01:24:17.031836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:46:04.235199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022992+0000","last_change":"2024-03-24T01:24:10.753332+0000","last_active":"2024-03-24T01:24:17.022992+0000","last_peered":"2024-03-24T01:24:17.022992+0000","last_clean":"2024-03-24T01:24:17.022992+0000","last_became_active":"2024-03-24T01:24:10.752905+0000","last_became_peered":"2024-03-24T01:24:10.752905+0000","last_unstale":"2024-03-24T01:24:17.022992+0000","last_undegraded":"2024-03-24T01:24:17.022992+0000","last_fullsized":"2024-03-24T01:24:17.022992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:57:11.968706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809861+0000","last_change":"2024-03-24T01:24:10.759635+0000","last_active":"2024-03-24T01:24:17.809861+0000","last_peered":"2024-03-24T01:24:17.809861+0000","last_clean":"2024-03-24T01:24:17.809861+0000","last_became_active":"2024-03-24T01:24:10.759310+0000","last_became_peered":"2024-03-24T01:24:10.759310+0000","last_unstale":"2024-03-24T01:24:17.809861+0000","last_undegraded":"2024-03-24T01:24:17.809861+0000","last_fullsized":"2024-03-24T01:24:17.809861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:08:13.760239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021975+0000","last_change":"2024-03-24T01:24:10.746635+0000","last_active":"2024-03-24T01:24:17.021975+0000","last_peered":"2024-03-24T01:24:17.021975+0000","last_clean":"2024-03-24T01:24:17.021975+0000","last_became_active":"2024-03-24T01:24:10.745597+0000","last_became_peered":"2024-03-24T01:24:10.745597+0000","last_unstale":"2024-03-24T01:24:17.021975+0000","last_undegraded":"2024-03-24T01:24:17.021975+0000","last_fullsized":"2024-03-24T01:24:17.021975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:42:22.671154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.810268+0000","last_change":"2024-03-24T01:24:10.743389+0000","last_active":"2024-03-24T01:24:17.810268+0000","last_peered":"2024-03-24T01:24:17.810268+0000","last_clean":"2024-03-24T01:24:17.810268+0000","last_became_active":"2024-03-24T01:24:10.743128+0000","last_became_peered":"2024-03-24T01:24:10.743128+0000","last_unstale":"2024-03-24T01:24:17.810268+0000","last_undegraded":"2024-03-24T01:24:17.810268+0000","last_fullsized":"2024-03-24T01:24:17.810268+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:42:22.671154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809576+0000","last_change":"2024-03-24T01:24:10.760940+0000","last_active":"2024-03-24T01:24:17.809576+0000","last_peered":"2024-03-24T01:24:17.809576+0000","last_clean":"2024-03-24T01:24:17.809576+0000","last_became_active":"2024-03-24T01:24:10.759991+0000","last_became_peered":"2024-03-24T01:24:10.759991+0000","last_unstale":"2024-03-24T01:24:17.809576+0000","last_undegraded":"2024-03-24T01:24:17.809576+0000","last_fullsized":"2024-03-24T01:24:17.809576+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:54:57.105205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.030845+0000","last_change":"2024-03-24T01:24:10.731391+0000","last_active":"2024-03-24T01:24:17.030845+0000","last_peered":"2024-03-24T01:24:17.030845+0000","last_clean":"2024-03-24T01:24:17.030845+0000","last_became_active":"2024-03-24T01:24:10.731094+0000","last_became_peered":"2024-03-24T01:24:10.731094+0000","last_unstale":"2024-03-24T01:24:17.030845+0000","last_undegraded":"2024-03-24T01:24:17.030845+0000","last_fullsized":"2024-03-24T01:24:17.030845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:17.138453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.812469+0000","last_change":"2024-03-24T01:24:10.766746+0000","last_active":"2024-03-24T01:24:17.812469+0000","last_peered":"2024-03-24T01:24:17.812469+0000","last_clean":"2024-03-24T01:24:17.812469+0000","last_became_active":"2024-03-24T01:24:10.766239+0000","last_became_peered":"2024-03-24T01:24:10.766239+0000","last_unstale":"2024-03-24T01:24:17.812469+0000","last_undegraded":"2024-03-24T01:24:17.812469+0000","last_fullsized":"2024-03-24T01:24:17.812469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:45:13.243851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811052+0000","last_change":"2024-03-24T01:24:10.741506+0000","last_active":"2024-03-24T01:24:17.811052+0000","last_peered":"2024-03-24T01:24:17.811052+0000","last_clean":"2024-03-24T01:24:17.811052+0000","last_became_active":"2024-03-24T01:24:10.741221+0000","last_became_peered":"2024-03-24T01:24:10.741221+0000","last_unstale":"2024-03-24T01:24:17.811052+0000","last_undegraded":"2024-03-24T01:24:17.811052+0000","last_fullsized":"2024-03-24T01:24:17.811052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:27:51.341849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.812926+0000","last_change":"2024-03-24T01:24:10.767451+0000","last_active":"2024-03-24T01:24:17.812926+0000","last_peered":"2024-03-24T01:24:17.812926+0000","last_clean":"2024-03-24T01:24:17.812926+0000","last_became_active":"2024-03-24T01:24:10.767159+0000","last_became_peered":"2024-03-24T01:24:10.767159+0000","last_unstale":"2024-03-24T01:24:17.812926+0000","last_undegraded":"2024-03-24T01:24:17.812926+0000","last_fullsized":"2024-03-24T01:24:17.812926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:24.434151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809619+0000","last_change":"2024-03-24T01:24:10.751373+0000","last_active":"2024-03-24T01:24:17.809619+0000","last_peered":"2024-03-24T01:24:17.809619+0000","last_clean":"2024-03-24T01:24:17.809619+0000","last_became_active":"2024-03-24T01:24:10.751084+0000","last_became_peered":"2024-03-24T01:24:10.751084+0000","last_unstale":"2024-03-24T01:24:17.809619+0000","last_undegraded":"2024-03-24T01:24:17.809619+0000","last_fullsized":"2024-03-24T01:24:17.809619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:17.138453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.739324+0000","last_change":"2024-03-24T01:24:10.752504+0000","last_active":"2024-03-24T01:24:17.739324+0000","last_peered":"2024-03-24T01:24:17.739324+0000","last_clean":"2024-03-24T01:24:17.739324+0000","last_became_active":"2024-03-24T01:24:10.752313+0000","last_became_peered":"2024-03-24T01:24:10.752313+0000","last_unstale":"2024-03-24T01:24:17.739324+0000","last_undegraded":"2024-03-24T01:24:17.739324+0000","last_fullsized":"2024-03-24T01:24:17.739324+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:54:57.105205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809195+0000","last_change":"2024-03-24T01:24:10.750967+0000","last_active":"2024-03-24T01:24:17.809195+0000","last_peered":"2024-03-24T01:24:17.809195+0000","last_clean":"2024-03-24T01:24:17.809195+0000","last_became_active":"2024-03-24T01:24:10.748587+0000","last_became_peered":"2024-03-24T01:24:10.748587+0000","last_unstale":"2024-03-24T01:24:17.809195+0000","last_undegraded":"2024-03-24T01:24:17.809195+0000","last_fullsized":"2024-03-24T01:24:17.809195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:44:57.802345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021966+0000","last_change":"2024-03-24T01:24:10.750792+0000","last_active":"2024-03-24T01:24:17.021966+0000","last_peered":"2024-03-24T01:24:17.021966+0000","last_clean":"2024-03-24T01:24:17.021966+0000","last_became_active":"2024-03-24T01:24:10.749384+0000","last_became_peered":"2024-03-24T01:24:10.749384+0000","last_unstale":"2024-03-24T01:24:17.021966+0000","last_undegraded":"2024-03-24T01:24:17.021966+0000","last_fullsized":"2024-03-24T01:24:17.021966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:17.138453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028133+0000","last_change":"2024-03-24T01:24:10.728222+0000","last_active":"2024-03-24T01:24:17.028133+0000","last_peered":"2024-03-24T01:24:17.028133+0000","last_clean":"2024-03-24T01:24:17.028133+0000","last_became_active":"2024-03-24T01:24:10.727874+0000","last_became_peered":"2024-03-24T01:24:10.727874+0000","last_unstale":"2024-03-24T01:24:17.028133+0000","last_undegraded":"2024-03-24T01:24:17.028133+0000","last_fullsized":"2024-03-24T01:24:17.028133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:58:18.259182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.031302+0000","last_change":"2024-03-24T01:24:10.741809+0000","last_active":"2024-03-24T01:24:17.031302+0000","last_peered":"2024-03-24T01:24:17.031302+0000","last_clean":"2024-03-24T01:24:17.031302+0000","last_became_active":"2024-03-24T01:24:10.741444+0000","last_became_peered":"2024-03-24T01:24:10.741444+0000","last_unstale":"2024-03-24T01:24:17.031302+0000","last_undegraded":"2024-03-24T01:24:17.031302+0000","last_fullsized":"2024-03-24T01:24:17.031302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:19:51.907364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.031726+0000","last_change":"2024-03-24T01:24:10.751752+0000","last_active":"2024-03-24T01:24:17.031726+0000","last_peered":"2024-03-24T01:24:17.031726+0000","last_clean":"2024-03-24T01:24:17.031726+0000","last_became_active":"2024-03-24T01:24:10.747073+0000","last_became_peered":"2024-03-24T01:24:10.747073+0000","last_unstale":"2024-03-24T01:24:17.031726+0000","last_undegraded":"2024-03-24T01:24:17.031726+0000","last_fullsized":"2024-03-24T01:24:17.031726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:08:13.760239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.813263+0000","last_change":"2024-03-24T01:24:10.765006+0000","last_active":"2024-03-24T01:24:17.813263+0000","last_peered":"2024-03-24T01:24:17.813263+0000","last_clean":"2024-03-24T01:24:17.813263+0000","last_became_active":"2024-03-24T01:24:10.764417+0000","last_became_peered":"2024-03-24T01:24:10.764417+0000","last_unstale":"2024-03-24T01:24:17.813263+0000","last_undegraded":"2024-03-24T01:24:17.813263+0000","last_fullsized":"2024-03-24T01:24:17.813263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:29:34.357876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021162+0000","last_change":"2024-03-24T01:24:10.757149+0000","last_active":"2024-03-24T01:24:17.021162+0000","last_peered":"2024-03-24T01:24:17.021162+0000","last_clean":"2024-03-24T01:24:17.021162+0000","last_became_active":"2024-03-24T01:24:10.756879+0000","last_became_peered":"2024-03-24T01:24:10.756879+0000","last_unstale":"2024-03-24T01:24:17.021162+0000","last_undegraded":"2024-03-24T01:24:17.021162+0000","last_fullsized":"2024-03-24T01:24:17.021162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:55:14.169541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809223+0000","last_change":"2024-03-24T01:24:10.766128+0000","last_active":"2024-03-24T01:24:17.809223+0000","last_peered":"2024-03-24T01:24:17.809223+0000","last_clean":"2024-03-24T01:24:17.809223+0000","last_became_active":"2024-03-24T01:24:10.763007+0000","last_became_peered":"2024-03-24T01:24:10.763007+0000","last_unstale":"2024-03-24T01:24:17.809223+0000","last_undegraded":"2024-03-24T01:24:17.809223+0000","last_fullsized":"2024-03-24T01:24:17.809223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:35.441197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809997+0000","last_change":"2024-03-24T01:24:10.741795+0000","last_active":"2024-03-24T01:24:17.809997+0000","last_peered":"2024-03-24T01:24:17.809997+0000","last_clean":"2024-03-24T01:24:17.809997+0000","last_became_active":"2024-03-24T01:24:10.741516+0000","last_became_peered":"2024-03-24T01:24:10.741516+0000","last_unstale":"2024-03-24T01:24:17.809997+0000","last_undegraded":"2024-03-24T01:24:17.809997+0000","last_fullsized":"2024-03-24T01:24:17.809997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:57:11.968706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811350+0000","last_change":"2024-03-24T01:24:10.761849+0000","last_active":"2024-03-24T01:24:17.811350+0000","last_peered":"2024-03-24T01:24:17.811350+0000","last_clean":"2024-03-24T01:24:17.811350+0000","last_became_active":"2024-03-24T01:24:10.761288+0000","last_became_peered":"2024-03-24T01:24:10.761288+0000","last_unstale":"2024-03-24T01:24:17.811350+0000","last_undegraded":"2024-03-24T01:24:17.811350+0000","last_fullsized":"2024-03-24T01:24:17.811350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:02:18.904076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.810985+0000","last_change":"2024-03-24T01:24:10.755426+0000","last_active":"2024-03-24T01:24:17.810985+0000","last_peered":"2024-03-24T01:24:17.810985+0000","last_clean":"2024-03-24T01:24:17.810985+0000","last_became_active":"2024-03-24T01:24:10.755073+0000","last_became_peered":"2024-03-24T01:24:10.755073+0000","last_unstale":"2024-03-24T01:24:17.810985+0000","last_undegraded":"2024-03-24T01:24:17.810985+0000","last_fullsized":"2024-03-24T01:24:17.810985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:46:04.235199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.810708+0000","last_change":"2024-03-24T01:24:10.732890+0000","last_active":"2024-03-24T01:24:17.810708+0000","last_peered":"2024-03-24T01:24:17.810708+0000","last_clean":"2024-03-24T01:24:17.810708+0000","last_became_active":"2024-03-24T01:24:10.731992+0000","last_became_peered":"2024-03-24T01:24:10.731992+0000","last_unstale":"2024-03-24T01:24:17.810708+0000","last_undegraded":"2024-03-24T01:24:17.810708+0000","last_fullsized":"2024-03-24T01:24:17.810708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:19:51.907364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.741104+0000","last_change":"2024-03-24T01:24:10.750945+0000","last_active":"2024-03-24T01:24:17.741104+0000","last_peered":"2024-03-24T01:24:17.741104+0000","last_clean":"2024-03-24T01:24:17.741104+0000","last_became_active":"2024-03-24T01:24:10.750099+0000","last_became_peered":"2024-03-24T01:24:10.750099+0000","last_unstale":"2024-03-24T01:24:17.741104+0000","last_undegraded":"2024-03-24T01:24:17.741104+0000","last_fullsized":"2024-03-24T01:24:17.741104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:08:13.760239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.813056+0000","last_change":"2024-03-24T01:24:10.762165+0000","last_active":"2024-03-24T01:24:17.813056+0000","last_peered":"2024-03-24T01:24:17.813056+0000","last_clean":"2024-03-24T01:24:17.813056+0000","last_became_active":"2024-03-24T01:24:10.758157+0000","last_became_peered":"2024-03-24T01:24:10.758157+0000","last_unstale":"2024-03-24T01:24:17.813056+0000","last_undegraded":"2024-03-24T01:24:17.813056+0000","last_fullsized":"2024-03-24T01:24:17.813056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:37:21.153754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27864,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343976,"statfs":{"total":96636764160,"available":96608231424,"internally_reserved":0,"allocated":864256,"data_stored":698343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":107483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":107519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":107393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":107553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":872448,"data_stored":698399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T01:24:38.522 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T01:24:38.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.765+0000 7f15ba000640 1 Processor -- start 2024-03-24T01:24:38.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.765+0000 7f15ba000640 1 -- start start 2024-03-24T01:24:38.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.766+0000 7f15ba000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f15b4155030 0x7f15b415efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:38.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.767+0000 7f15ba000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f15b40cdac0 0x7f15b406b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:38.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.767+0000 7f15ba000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b415f500 con 0x7f15b406b600 2024-03-24T01:24:38.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.767+0000 7f15ba000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15b415bf70 con 0x7f15b40cdac0 2024-03-24T01:24:38.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.767+0000 7f15ba000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15b406ba00 con 0x7f15b4155030 2024-03-24T01:24:38.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.767+0000 7f15b3e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f15b406b600 0x7f15b40ae530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:32960/0 (socket says 172.21.15.90:32960) 2024-03-24T01:24:38.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.767+0000 7f15b2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f15b40cdac0 0x7f15b406b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:38.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.767+0000 7f15b3e00640 1 -- 172.21.15.90:0/2218152403 learned_addr learned my addr 172.21.15.90:0/2218152403 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:38.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.767+0000 7f15b3400640 1 --2- 172.21.15.90:0/2218152403 >> v2:172.21.15.120:3300/0 conn(0x7f15b4155030 0x7f15b415efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:38.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.768+0000 7f15b2a00640 1 -- 172.21.15.90:0/2218152403 >> v2:172.21.15.120:3300/0 conn(0x7f15b4155030 msgr2=0x7f15b415efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:38.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.768+0000 7f15b2a00640 1 --2- 172.21.15.90:0/2218152403 >> v2:172.21.15.120:3300/0 conn(0x7f15b4155030 0x7f15b415efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:38.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.768+0000 7f15b2a00640 1 -- 172.21.15.90:0/2218152403 >> v1:172.21.15.90:6789/0 conn(0x7f15b406b600 legacy=0x7f15b40ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:38.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.768+0000 7f15b2a00640 1 -- 172.21.15.90:0/2218152403 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b415b360 con 0x7f15b40cdac0 2024-03-24T01:24:38.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.769+0000 7f15b2a00640 1 --2- 172.21.15.90:0/2218152403 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15b40cdac0 0x7f15b406b0c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f15b406ecc0 tx=0x7f15a002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c54a3181fed1db5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:38.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.769+0000 7f15b2000640 1 -- 172.21.15.90:0/2218152403 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a0035020 con 0x7f15b40cdac0 2024-03-24T01:24:38.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.769+0000 7f15b2000640 1 -- 172.21.15.90:0/2218152403 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a0031830 con 0x7f15b40cdac0 2024-03-24T01:24:38.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.769+0000 7f15b2000640 1 -- 172.21.15.90:0/2218152403 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a0031c00 con 0x7f15b40cdac0 2024-03-24T01:24:38.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.770+0000 7f15ba000640 1 -- 172.21.15.90:0/2218152403 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15b40cdac0 msgr2=0x7f15b406b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:38.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.770+0000 7f15ba000640 1 --2- 172.21.15.90:0/2218152403 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15b40cdac0 0x7f15b406b0c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f15b406ecc0 tx=0x7f15a002b8d0 comp rx=0 tx=0).stop 2024-03-24T01:24:38.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.770+0000 7f15ba000640 1 -- 172.21.15.90:0/2218152403 shutdown_connections 2024-03-24T01:24:38.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.770+0000 7f15ba000640 1 --2- 172.21.15.90:0/2218152403 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15b40cdac0 0x7f15b406b0c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:38.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.770+0000 7f15ba000640 1 --2- 172.21.15.90:0/2218152403 >> v2:172.21.15.120:3300/0 conn(0x7f15b4155030 0x7f15b415efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:38.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.770+0000 7f15ba000640 1 -- 172.21.15.90:0/2218152403 >> 172.21.15.90:0/2218152403 conn(0x7f15b40b1770 msgr2=0x7f15b40b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:38.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.770+0000 7f15ba000640 1 -- 172.21.15.90:0/2218152403 shutdown_connections 2024-03-24T01:24:38.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.770+0000 7f15ba000640 1 -- 172.21.15.90:0/2218152403 wait complete. 2024-03-24T01:24:38.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.771+0000 7f15ba000640 1 Processor -- start 2024-03-24T01:24:38.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.772+0000 7f15ba000640 1 -- start start 2024-03-24T01:24:38.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.772+0000 7f15ba000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f15b406b600 0x7f15b4158010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:38.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.772+0000 7f15ba000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15b40cdac0 0x7f15b4144250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:38.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.772+0000 7f15ba000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15b4155030 0x7f15b4156670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:38.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.772+0000 7f15ba000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b415bc00 con 0x7f15b40cdac0 2024-03-24T01:24:38.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.772+0000 7f15ba000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b415b920 con 0x7f15b4155030 2024-03-24T01:24:38.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.772+0000 7f15ba000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15b4155410 con 0x7f15b406b600 2024-03-24T01:24:38.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.773+0000 7f15b2a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15b40cdac0 0x7f15b4144250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:38.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.773+0000 7f15b2a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15b40cdac0 0x7f15b4144250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43532/0 (socket says 172.21.15.90:43532) 2024-03-24T01:24:38.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.773+0000 7f15b2a00640 1 -- 172.21.15.90:0/1219830156 learned_addr learned my addr 172.21.15.90:0/1219830156 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:38.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.773+0000 7f15b3e00640 1 --2- 172.21.15.90:0/1219830156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15b4155030 0x7f15b4156670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:38.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.773+0000 7f15b3400640 1 --2- 172.21.15.90:0/1219830156 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f15b406b600 0x7f15b4158010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:38.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.773+0000 7f15b2a00640 1 -- 172.21.15.90:0/1219830156 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f15b406b600 msgr2=0x7f15b4158010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:38.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.773+0000 7f15b2a00640 1 --2- 172.21.15.90:0/1219830156 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f15b406b600 0x7f15b4158010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:38.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.773+0000 7f15b2a00640 1 -- 172.21.15.90:0/1219830156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15b4155030 msgr2=0x7f15b4156670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:38.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.773+0000 7f15b2a00640 1 --2- 172.21.15.90:0/1219830156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15b4155030 0x7f15b4156670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:38.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.773+0000 7f15b2a00640 1 -- 172.21.15.90:0/1219830156 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b4156f70 con 0x7f15b40cdac0 2024-03-24T01:24:38.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.773+0000 7f15b3e00640 1 --2- 172.21.15.90:0/1219830156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15b4155030 0x7f15b4156670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:24:38.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.774+0000 7f15b2a00640 1 --2- 172.21.15.90:0/1219830156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15b40cdac0 0x7f15b4144250 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f15a0009890 tx=0x7f15a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:38.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.774+0000 7f15ab400640 1 -- 172.21.15.90:0/1219830156 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a0041070 con 0x7f15b40cdac0 2024-03-24T01:24:38.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.774+0000 7f15ab400640 1 -- 172.21.15.90:0/1219830156 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a003b920 con 0x7f15b40cdac0 2024-03-24T01:24:38.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.774+0000 7f15ab400640 1 -- 172.21.15.90:0/1219830156 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a0035040 con 0x7f15b40cdac0 2024-03-24T01:24:38.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.774+0000 7f15ba000640 1 -- 172.21.15.90:0/1219830156 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15b4157250 con 0x7f15b40cdac0 2024-03-24T01:24:38.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.774+0000 7f15ba000640 1 -- 172.21.15.90:0/1219830156 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15b4157810 con 0x7f15b40cdac0 2024-03-24T01:24:38.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.775+0000 7f15ba000640 1 -- 172.21.15.90:0/1219830156 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15b4068060 con 0x7f15b40cdac0 2024-03-24T01:24:38.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.776+0000 7f15ab400640 1 -- 172.21.15.90:0/1219830156 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f15a0038020 con 0x7f15b40cdac0 2024-03-24T01:24:38.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.777+0000 7f15ab400640 1 --2- 172.21.15.90:0/1219830156 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f159406bbd0 0x7f159406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:38.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.777+0000 7f15ab400640 1 -- 172.21.15.90:0/1219830156 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f15a00ac780 con 0x7f15b40cdac0 2024-03-24T01:24:38.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.778+0000 7f15b3400640 1 --2- 172.21.15.90:0/1219830156 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f159406bbd0 0x7f159406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:38.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.779+0000 7f15b3400640 1 --2- 172.21.15.90:0/1219830156 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f159406bbd0 0x7f159406e090 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f15a4009970 tx=0x7f15a4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:38.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.782+0000 7f15ab400640 1 -- 172.21.15.90:0/1219830156 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f15a002f2e0 con 0x7f15b40cdac0 2024-03-24T01:24:38.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:38.978+0000 7f15ba000640 1 -- 172.21.15.90:0/1219830156 --> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f15b406f150 con 0x7f159406bbd0 2024-03-24T01:24:39.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.002+0000 7f15ab400640 1 -- 172.21.15.90:0/1219830156 <== mgr.4104 v2:172.21.15.120:6832/860392634 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332537 (secure 0 0 0) 0x7f15b406f150 con 0x7f159406bbd0 2024-03-24T01:24:39.008 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:24:39.009 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-24T01:24:39.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.009+0000 7f15ba000640 1 -- 172.21.15.90:0/1219830156 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f159406bbd0 msgr2=0x7f159406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:39.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.009+0000 7f15ba000640 1 --2- 172.21.15.90:0/1219830156 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f159406bbd0 0x7f159406e090 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f15a4009970 tx=0x7f15a4008040 comp rx=0 tx=0).stop 2024-03-24T01:24:39.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.009+0000 7f15ba000640 1 -- 172.21.15.90:0/1219830156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15b40cdac0 msgr2=0x7f15b4144250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:39.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.009+0000 7f15ba000640 1 --2- 172.21.15.90:0/1219830156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15b40cdac0 0x7f15b4144250 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f15a0009890 tx=0x7f15a00046c0 comp rx=0 tx=0).stop 2024-03-24T01:24:39.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.009+0000 7f15ba000640 1 -- 172.21.15.90:0/1219830156 shutdown_connections 2024-03-24T01:24:39.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.009+0000 7f15ba000640 1 --2- 172.21.15.90:0/1219830156 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f159406bbd0 0x7f159406e090 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:39.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.009+0000 7f15ba000640 1 --2- 172.21.15.90:0/1219830156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15b40cdac0 0x7f15b4144250 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:39.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.009+0000 7f15ba000640 1 --2- 172.21.15.90:0/1219830156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15b4155030 0x7f15b4156670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:39.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.009+0000 7f15ba000640 1 --2- 172.21.15.90:0/1219830156 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f15b406b600 0x7f15b4158010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:39.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.009+0000 7f15ba000640 1 -- 172.21.15.90:0/1219830156 >> 172.21.15.90:0/1219830156 conn(0x7f15b40b1770 msgr2=0x7f15b40b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:39.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.010+0000 7f15ba000640 1 -- 172.21.15.90:0/1219830156 shutdown_connections 2024-03-24T01:24:39.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.010+0000 7f15ba000640 1 -- 172.21.15.90:0/1219830156 wait complete. 2024-03-24T01:24:39.030 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-03-24T01:24:37.487823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219472,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755248,"statfs":{"total":773094113280,"available":772869373952,"internally_reserved":0,"allocated":3383296,"data_stored":2049160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12747,"internal_metadata":221236789},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005335"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":28,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811464+0000","last_change":"2024-03-24T01:24:09.722150+0000","last_active":"2024-03-24T01:24:17.811464+0000","last_peered":"2024-03-24T01:24:17.811464+0000","last_clean":"2024-03-24T01:24:17.811464+0000","last_became_active":"2024-03-24T01:24:09.721808+0000","last_became_peered":"2024-03-24T01:24:09.721808+0000","last_unstale":"2024-03-24T01:24:17.811464+0000","last_undegraded":"2024-03-24T01:24:17.811464+0000","last_fullsized":"2024-03-24T01:24:17.811464+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:31:46.512174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022533+0000","last_change":"2024-03-24T01:24:10.750478+0000","last_active":"2024-03-24T01:24:17.022533+0000","last_peered":"2024-03-24T01:24:17.022533+0000","last_clean":"2024-03-24T01:24:17.022533+0000","last_became_active":"2024-03-24T01:24:10.749299+0000","last_became_peered":"2024-03-24T01:24:10.749299+0000","last_unstale":"2024-03-24T01:24:17.022533+0000","last_undegraded":"2024-03-24T01:24:17.022533+0000","last_fullsized":"2024-03-24T01:24:17.022533+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:46:04.235199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021708+0000","last_change":"2024-03-24T01:24:15.662475+0000","last_active":"2024-03-24T01:24:17.021708+0000","last_peered":"2024-03-24T01:24:17.021708+0000","last_clean":"2024-03-24T01:24:17.021708+0000","last_became_active":"2024-03-24T01:24:15.661550+0000","last_became_peered":"2024-03-24T01:24:15.661550+0000","last_unstale":"2024-03-24T01:24:17.021708+0000","last_undegraded":"2024-03-24T01:24:17.021708+0000","last_fullsized":"2024-03-24T01:24:17.021708+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:14:59.259808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022711+0000","last_change":"2024-03-24T01:24:09.723434+0000","last_active":"2024-03-24T01:24:17.022711+0000","last_peered":"2024-03-24T01:24:17.022711+0000","last_clean":"2024-03-24T01:24:17.022711+0000","last_became_active":"2024-03-24T01:24:09.722439+0000","last_became_peered":"2024-03-24T01:24:09.722439+0000","last_unstale":"2024-03-24T01:24:17.022711+0000","last_undegraded":"2024-03-24T01:24:17.022711+0000","last_fullsized":"2024-03-24T01:24:17.022711+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:31:46.512174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811170+0000","last_change":"2024-03-24T01:24:10.745888+0000","last_active":"2024-03-24T01:24:17.811170+0000","last_peered":"2024-03-24T01:24:17.811170+0000","last_clean":"2024-03-24T01:24:17.811170+0000","last_became_active":"2024-03-24T01:24:10.742579+0000","last_became_peered":"2024-03-24T01:24:10.742579+0000","last_unstale":"2024-03-24T01:24:17.811170+0000","last_undegraded":"2024-03-24T01:24:17.811170+0000","last_fullsized":"2024-03-24T01:24:17.811170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:04:48.028091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.026871+0000","last_change":"2024-03-24T01:24:15.022407+0000","last_active":"2024-03-24T01:24:17.026871+0000","last_peered":"2024-03-24T01:24:17.026871+0000","last_clean":"2024-03-24T01:24:17.026871+0000","last_became_active":"2024-03-24T01:24:15.021962+0000","last_became_peered":"2024-03-24T01:24:15.021962+0000","last_unstale":"2024-03-24T01:24:17.026871+0000","last_undegraded":"2024-03-24T01:24:17.026871+0000","last_fullsized":"2024-03-24T01:24:17.026871+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:28.723031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.807761+0000","last_change":"2024-03-24T01:24:09.735868+0000","last_active":"2024-03-24T01:24:17.807761+0000","last_peered":"2024-03-24T01:24:17.807761+0000","last_clean":"2024-03-24T01:24:17.807761+0000","last_became_active":"2024-03-24T01:24:09.735573+0000","last_became_peered":"2024-03-24T01:24:09.735573+0000","last_unstale":"2024-03-24T01:24:17.807761+0000","last_undegraded":"2024-03-24T01:24:17.807761+0000","last_fullsized":"2024-03-24T01:24:17.807761+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:35:54.000044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811189+0000","last_change":"2024-03-24T01:24:10.752632+0000","last_active":"2024-03-24T01:24:17.811189+0000","last_peered":"2024-03-24T01:24:17.811189+0000","last_clean":"2024-03-24T01:24:17.811189+0000","last_became_active":"2024-03-24T01:24:10.752358+0000","last_became_peered":"2024-03-24T01:24:10.752358+0000","last_unstale":"2024-03-24T01:24:17.811189+0000","last_undegraded":"2024-03-24T01:24:17.811189+0000","last_fullsized":"2024-03-24T01:24:17.811189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:38:07.266851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.027523+0000","last_change":"2024-03-24T01:24:15.032062+0000","last_active":"2024-03-24T01:24:17.027523+0000","last_peered":"2024-03-24T01:24:17.027523+0000","last_clean":"2024-03-24T01:24:17.027523+0000","last_became_active":"2024-03-24T01:24:15.031556+0000","last_became_peered":"2024-03-24T01:24:15.031556+0000","last_unstale":"2024-03-24T01:24:17.027523+0000","last_undegraded":"2024-03-24T01:24:17.027523+0000","last_fullsized":"2024-03-24T01:24:17.027523+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:45:17.532730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.812001+0000","last_change":"2024-03-24T01:24:09.739225+0000","last_active":"2024-03-24T01:24:17.812001+0000","last_peered":"2024-03-24T01:24:17.812001+0000","last_clean":"2024-03-24T01:24:17.812001+0000","last_became_active":"2024-03-24T01:24:09.738488+0000","last_became_peered":"2024-03-24T01:24:09.738488+0000","last_unstale":"2024-03-24T01:24:17.812001+0000","last_undegraded":"2024-03-24T01:24:17.812001+0000","last_fullsized":"2024-03-24T01:24:17.812001+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:56:33.895912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811310+0000","last_change":"2024-03-24T01:24:10.753043+0000","last_active":"2024-03-24T01:24:17.811310+0000","last_peered":"2024-03-24T01:24:17.811310+0000","last_clean":"2024-03-24T01:24:17.811310+0000","last_became_active":"2024-03-24T01:24:10.751666+0000","last_became_peered":"2024-03-24T01:24:10.751666+0000","last_unstale":"2024-03-24T01:24:17.811310+0000","last_undegraded":"2024-03-24T01:24:17.811310+0000","last_fullsized":"2024-03-24T01:24:17.811310+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:02:18.904076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.027460+0000","last_change":"2024-03-24T01:24:15.021052+0000","last_active":"2024-03-24T01:24:17.027460+0000","last_peered":"2024-03-24T01:24:17.027460+0000","last_clean":"2024-03-24T01:24:17.027460+0000","last_became_active":"2024-03-24T01:24:15.020449+0000","last_became_peered":"2024-03-24T01:24:15.020449+0000","last_unstale":"2024-03-24T01:24:17.027460+0000","last_undegraded":"2024-03-24T01:24:17.027460+0000","last_fullsized":"2024-03-24T01:24:17.027460+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:26:07.104726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028245+0000","last_change":"2024-03-24T01:24:10.737914+0000","last_active":"2024-03-24T01:24:17.028245+0000","last_peered":"2024-03-24T01:24:17.028245+0000","last_clean":"2024-03-24T01:24:17.028245+0000","last_became_active":"2024-03-24T01:24:10.737652+0000","last_became_peered":"2024-03-24T01:24:10.737652+0000","last_unstale":"2024-03-24T01:24:17.028245+0000","last_undegraded":"2024-03-24T01:24:17.028245+0000","last_fullsized":"2024-03-24T01:24:17.028245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:09:31.705353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.807741+0000","last_change":"2024-03-24T01:24:09.726730+0000","last_active":"2024-03-24T01:24:17.807741+0000","last_peered":"2024-03-24T01:24:17.807741+0000","last_clean":"2024-03-24T01:24:17.807741+0000","last_became_active":"2024-03-24T01:24:09.725266+0000","last_became_peered":"2024-03-24T01:24:09.725266+0000","last_unstale":"2024-03-24T01:24:17.807741+0000","last_undegraded":"2024-03-24T01:24:17.807741+0000","last_fullsized":"2024-03-24T01:24:17.807741+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:27:50.329766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028585+0000","last_change":"2024-03-24T01:24:15.027082+0000","last_active":"2024-03-24T01:24:17.028585+0000","last_peered":"2024-03-24T01:24:17.028585+0000","last_clean":"2024-03-24T01:24:17.028585+0000","last_became_active":"2024-03-24T01:24:15.026296+0000","last_became_peered":"2024-03-24T01:24:15.026296+0000","last_unstale":"2024-03-24T01:24:17.028585+0000","last_undegraded":"2024-03-24T01:24:17.028585+0000","last_fullsized":"2024-03-24T01:24:17.028585+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:55.111215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022103+0000","last_change":"2024-03-24T01:24:10.740733+0000","last_active":"2024-03-24T01:24:17.022103+0000","last_peered":"2024-03-24T01:24:17.022103+0000","last_clean":"2024-03-24T01:24:17.022103+0000","last_became_active":"2024-03-24T01:24:10.740515+0000","last_became_peered":"2024-03-24T01:24:10.740515+0000","last_unstale":"2024-03-24T01:24:17.022103+0000","last_undegraded":"2024-03-24T01:24:17.022103+0000","last_fullsized":"2024-03-24T01:24:17.022103+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:04:48.028091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.738935+0000","last_change":"2024-03-24T01:24:09.725315+0000","last_active":"2024-03-24T01:24:17.738935+0000","last_peered":"2024-03-24T01:24:17.738935+0000","last_clean":"2024-03-24T01:24:17.738935+0000","last_became_active":"2024-03-24T01:24:09.723043+0000","last_became_peered":"2024-03-24T01:24:09.723043+0000","last_unstale":"2024-03-24T01:24:17.738935+0000","last_undegraded":"2024-03-24T01:24:17.738935+0000","last_fullsized":"2024-03-24T01:24:17.738935+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:36:52.007109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.027797+0000","last_change":"2024-03-24T01:24:15.023034+0000","last_active":"2024-03-24T01:24:17.027797+0000","last_peered":"2024-03-24T01:24:17.027797+0000","last_clean":"2024-03-24T01:24:17.027797+0000","last_became_active":"2024-03-24T01:24:15.021144+0000","last_became_peered":"2024-03-24T01:24:15.021144+0000","last_unstale":"2024-03-24T01:24:17.027797+0000","last_undegraded":"2024-03-24T01:24:17.027797+0000","last_fullsized":"2024-03-24T01:24:17.027797+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:00.868691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021560+0000","last_change":"2024-03-24T01:24:10.746085+0000","last_active":"2024-03-24T01:24:17.021560+0000","last_peered":"2024-03-24T01:24:17.021560+0000","last_clean":"2024-03-24T01:24:17.021560+0000","last_became_active":"2024-03-24T01:24:10.745493+0000","last_became_peered":"2024-03-24T01:24:10.745493+0000","last_unstale":"2024-03-24T01:24:17.021560+0000","last_undegraded":"2024-03-24T01:24:17.021560+0000","last_fullsized":"2024-03-24T01:24:17.021560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:19:51.907364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.739444+0000","last_change":"2024-03-24T01:24:09.723035+0000","last_active":"2024-03-24T01:24:17.739444+0000","last_peered":"2024-03-24T01:24:17.739444+0000","last_clean":"2024-03-24T01:24:17.739444+0000","last_became_active":"2024-03-24T01:24:09.722548+0000","last_became_peered":"2024-03-24T01:24:09.722548+0000","last_unstale":"2024-03-24T01:24:17.739444+0000","last_undegraded":"2024-03-24T01:24:17.739444+0000","last_fullsized":"2024-03-24T01:24:17.739444+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:58:17.247099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.741186+0000","last_change":"2024-03-24T01:24:15.658323+0000","last_active":"2024-03-24T01:24:17.741186+0000","last_peered":"2024-03-24T01:24:17.741186+0000","last_clean":"2024-03-24T01:24:17.741186+0000","last_became_active":"2024-03-24T01:24:15.656635+0000","last_became_peered":"2024-03-24T01:24:15.656635+0000","last_unstale":"2024-03-24T01:24:17.741186+0000","last_undegraded":"2024-03-24T01:24:17.741186+0000","last_fullsized":"2024-03-24T01:24:17.741186+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:52:30.401135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028531+0000","last_change":"2024-03-24T01:24:10.734344+0000","last_active":"2024-03-24T01:24:17.028531+0000","last_peered":"2024-03-24T01:24:17.028531+0000","last_clean":"2024-03-24T01:24:17.028531+0000","last_became_active":"2024-03-24T01:24:10.734010+0000","last_became_peered":"2024-03-24T01:24:10.734010+0000","last_unstale":"2024-03-24T01:24:17.028531+0000","last_undegraded":"2024-03-24T01:24:17.028531+0000","last_fullsized":"2024-03-24T01:24:17.028531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:36:53.019192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.808157+0000","last_change":"2024-03-24T01:24:09.729151+0000","last_active":"2024-03-24T01:24:17.808157+0000","last_peered":"2024-03-24T01:24:17.808157+0000","last_clean":"2024-03-24T01:24:17.808157+0000","last_became_active":"2024-03-24T01:24:09.728441+0000","last_became_peered":"2024-03-24T01:24:09.728441+0000","last_unstale":"2024-03-24T01:24:17.808157+0000","last_undegraded":"2024-03-24T01:24:17.808157+0000","last_fullsized":"2024-03-24T01:24:17.808157+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:33.381248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.027681+0000","last_change":"2024-03-24T01:24:15.024436+0000","last_active":"2024-03-24T01:24:17.027681+0000","last_peered":"2024-03-24T01:24:17.027681+0000","last_clean":"2024-03-24T01:24:17.027681+0000","last_became_active":"2024-03-24T01:24:15.022270+0000","last_became_peered":"2024-03-24T01:24:15.022270+0000","last_unstale":"2024-03-24T01:24:17.027681+0000","last_undegraded":"2024-03-24T01:24:17.027681+0000","last_fullsized":"2024-03-24T01:24:17.027681+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:54:28.205416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809625+0000","last_change":"2024-03-24T01:24:10.755973+0000","last_active":"2024-03-24T01:24:17.809625+0000","last_peered":"2024-03-24T01:24:17.809625+0000","last_clean":"2024-03-24T01:24:17.809625+0000","last_became_active":"2024-03-24T01:24:10.755558+0000","last_became_peered":"2024-03-24T01:24:10.755558+0000","last_unstale":"2024-03-24T01:24:17.809625+0000","last_undegraded":"2024-03-24T01:24:17.809625+0000","last_fullsized":"2024-03-24T01:24:17.809625+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:08:13.760239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.031629+0000","last_change":"2024-03-24T01:24:09.724815+0000","last_active":"2024-03-24T01:24:17.031629+0000","last_peered":"2024-03-24T01:24:17.031629+0000","last_clean":"2024-03-24T01:24:17.031629+0000","last_became_active":"2024-03-24T01:24:09.723550+0000","last_became_peered":"2024-03-24T01:24:09.723550+0000","last_unstale":"2024-03-24T01:24:17.031629+0000","last_undegraded":"2024-03-24T01:24:17.031629+0000","last_fullsized":"2024-03-24T01:24:17.031629+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:21:50.689532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028118+0000","last_change":"2024-03-24T01:24:15.651689+0000","last_active":"2024-03-24T01:24:17.028118+0000","last_peered":"2024-03-24T01:24:17.028118+0000","last_clean":"2024-03-24T01:24:17.028118+0000","last_became_active":"2024-03-24T01:24:15.651006+0000","last_became_peered":"2024-03-24T01:24:15.651006+0000","last_unstale":"2024-03-24T01:24:17.028118+0000","last_undegraded":"2024-03-24T01:24:17.028118+0000","last_fullsized":"2024-03-24T01:24:17.028118+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:52:10.223780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.023121+0000","last_change":"2024-03-24T01:24:10.767341+0000","last_active":"2024-03-24T01:24:17.023121+0000","last_peered":"2024-03-24T01:24:17.023121+0000","last_clean":"2024-03-24T01:24:17.023121+0000","last_became_active":"2024-03-24T01:24:10.767024+0000","last_became_peered":"2024-03-24T01:24:10.767024+0000","last_unstale":"2024-03-24T01:24:17.023121+0000","last_undegraded":"2024-03-24T01:24:17.023121+0000","last_fullsized":"2024-03-24T01:24:17.023121+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:35.441197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.738964+0000","last_change":"2024-03-24T01:24:09.716616+0000","last_active":"2024-03-24T01:24:17.738964+0000","last_peered":"2024-03-24T01:24:17.738964+0000","last_clean":"2024-03-24T01:24:17.738964+0000","last_became_active":"2024-03-24T01:24:09.716363+0000","last_became_peered":"2024-03-24T01:24:09.716363+0000","last_unstale":"2024-03-24T01:24:17.738964+0000","last_undegraded":"2024-03-24T01:24:17.738964+0000","last_fullsized":"2024-03-24T01:24:17.738964+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:33.381248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021124+0000","last_change":"2024-03-24T01:24:15.667639+0000","last_active":"2024-03-24T01:24:17.021124+0000","last_peered":"2024-03-24T01:24:17.021124+0000","last_clean":"2024-03-24T01:24:17.021124+0000","last_became_active":"2024-03-24T01:24:15.664579+0000","last_became_peered":"2024-03-24T01:24:15.664579+0000","last_unstale":"2024-03-24T01:24:17.021124+0000","last_undegraded":"2024-03-24T01:24:17.021124+0000","last_fullsized":"2024-03-24T01:24:17.021124+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:43:54.773518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.740372+0000","last_change":"2024-03-24T01:24:10.750277+0000","last_active":"2024-03-24T01:24:17.740372+0000","last_peered":"2024-03-24T01:24:17.740372+0000","last_clean":"2024-03-24T01:24:17.740372+0000","last_became_active":"2024-03-24T01:24:10.748912+0000","last_became_peered":"2024-03-24T01:24:10.748912+0000","last_unstale":"2024-03-24T01:24:17.740372+0000","last_undegraded":"2024-03-24T01:24:17.740372+0000","last_fullsized":"2024-03-24T01:24:17.740372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:19:51.907364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.812643+0000","last_change":"2024-03-24T01:24:09.723002+0000","last_active":"2024-03-24T01:24:17.812643+0000","last_peered":"2024-03-24T01:24:17.812643+0000","last_clean":"2024-03-24T01:24:17.812643+0000","last_became_active":"2024-03-24T01:24:09.722711+0000","last_became_peered":"2024-03-24T01:24:09.722711+0000","last_unstale":"2024-03-24T01:24:17.812643+0000","last_undegraded":"2024-03-24T01:24:17.812643+0000","last_fullsized":"2024-03-24T01:24:17.812643+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:44:56.790262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809628+0000","last_change":"2024-03-24T01:24:15.657482+0000","last_active":"2024-03-24T01:24:17.809628+0000","last_peered":"2024-03-24T01:24:17.809628+0000","last_clean":"2024-03-24T01:24:17.809628+0000","last_became_active":"2024-03-24T01:24:15.656331+0000","last_became_peered":"2024-03-24T01:24:15.656331+0000","last_unstale":"2024-03-24T01:24:17.809628+0000","last_undegraded":"2024-03-24T01:24:17.809628+0000","last_fullsized":"2024-03-24T01:24:17.809628+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:00.868691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811496+0000","last_change":"2024-03-24T01:24:10.753417+0000","last_active":"2024-03-24T01:24:17.811496+0000","last_peered":"2024-03-24T01:24:17.811496+0000","last_clean":"2024-03-24T01:24:17.811496+0000","last_became_active":"2024-03-24T01:24:10.750938+0000","last_became_peered":"2024-03-24T01:24:10.750938+0000","last_unstale":"2024-03-24T01:24:17.811496+0000","last_undegraded":"2024-03-24T01:24:17.811496+0000","last_fullsized":"2024-03-24T01:24:17.811496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:29:34.357876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.740584+0000","last_change":"2024-03-24T01:24:09.716278+0000","last_active":"2024-03-24T01:24:17.740584+0000","last_peered":"2024-03-24T01:24:17.740584+0000","last_clean":"2024-03-24T01:24:17.740584+0000","last_became_active":"2024-03-24T01:24:09.715984+0000","last_became_peered":"2024-03-24T01:24:09.715984+0000","last_unstale":"2024-03-24T01:24:17.740584+0000","last_undegraded":"2024-03-24T01:24:17.740584+0000","last_fullsized":"2024-03-24T01:24:17.740584+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:44:32.492325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.739876+0000","last_change":"2024-03-24T01:24:15.024902+0000","last_active":"2024-03-24T01:24:17.739876+0000","last_peered":"2024-03-24T01:24:17.739876+0000","last_clean":"2024-03-24T01:24:17.739876+0000","last_became_active":"2024-03-24T01:24:15.024379+0000","last_became_peered":"2024-03-24T01:24:15.024379+0000","last_unstale":"2024-03-24T01:24:17.739876+0000","last_undegraded":"2024-03-24T01:24:17.739876+0000","last_fullsized":"2024-03-24T01:24:17.739876+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:26:07.104726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028172+0000","last_change":"2024-03-24T01:24:10.730294+0000","last_active":"2024-03-24T01:24:17.028172+0000","last_peered":"2024-03-24T01:24:17.028172+0000","last_clean":"2024-03-24T01:24:17.028172+0000","last_became_active":"2024-03-24T01:24:10.727422+0000","last_became_peered":"2024-03-24T01:24:10.727422+0000","last_unstale":"2024-03-24T01:24:17.028172+0000","last_undegraded":"2024-03-24T01:24:17.028172+0000","last_fullsized":"2024-03-24T01:24:17.028172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:34.393331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811092+0000","last_change":"2024-03-24T01:24:09.736152+0000","last_active":"2024-03-24T01:24:17.811092+0000","last_peered":"2024-03-24T01:24:17.811092+0000","last_clean":"2024-03-24T01:24:17.811092+0000","last_became_active":"2024-03-24T01:24:09.735870+0000","last_became_peered":"2024-03-24T01:24:09.735870+0000","last_unstale":"2024-03-24T01:24:17.811092+0000","last_undegraded":"2024-03-24T01:24:17.811092+0000","last_fullsized":"2024-03-24T01:24:17.811092+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:50:31.967556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.740556+0000","last_change":"2024-03-24T01:24:15.027111+0000","last_active":"2024-03-24T01:24:17.740556+0000","last_peered":"2024-03-24T01:24:17.740556+0000","last_clean":"2024-03-24T01:24:17.740556+0000","last_became_active":"2024-03-24T01:24:15.025568+0000","last_became_peered":"2024-03-24T01:24:15.025568+0000","last_unstale":"2024-03-24T01:24:17.740556+0000","last_undegraded":"2024-03-24T01:24:17.740556+0000","last_fullsized":"2024-03-24T01:24:17.740556+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:54:28.205416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.739727+0000","last_change":"2024-03-24T01:24:10.744796+0000","last_active":"2024-03-24T01:24:17.739727+0000","last_peered":"2024-03-24T01:24:17.739727+0000","last_clean":"2024-03-24T01:24:17.739727+0000","last_became_active":"2024-03-24T01:24:10.744533+0000","last_became_peered":"2024-03-24T01:24:10.744533+0000","last_unstale":"2024-03-24T01:24:17.739727+0000","last_undegraded":"2024-03-24T01:24:17.739727+0000","last_fullsized":"2024-03-24T01:24:17.739727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:09:31.705353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.807544+0000","last_change":"2024-03-24T01:24:09.717922+0000","last_active":"2024-03-24T01:24:17.807544+0000","last_peered":"2024-03-24T01:24:17.807544+0000","last_clean":"2024-03-24T01:24:17.807544+0000","last_became_active":"2024-03-24T01:24:09.716468+0000","last_became_peered":"2024-03-24T01:24:09.716468+0000","last_unstale":"2024-03-24T01:24:17.807544+0000","last_undegraded":"2024-03-24T01:24:17.807544+0000","last_fullsized":"2024-03-24T01:24:17.807544+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:50:57.480847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.741366+0000","last_change":"2024-03-24T01:24:15.028389+0000","last_active":"2024-03-24T01:24:17.741366+0000","last_peered":"2024-03-24T01:24:17.741366+0000","last_clean":"2024-03-24T01:24:17.741366+0000","last_became_active":"2024-03-24T01:24:15.027052+0000","last_became_peered":"2024-03-24T01:24:15.027052+0000","last_unstale":"2024-03-24T01:24:17.741366+0000","last_undegraded":"2024-03-24T01:24:17.741366+0000","last_fullsized":"2024-03-24T01:24:17.741366+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:00.868691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.808527+0000","last_change":"2024-03-24T01:24:10.726234+0000","last_active":"2024-03-24T01:24:17.808527+0000","last_peered":"2024-03-24T01:24:17.808527+0000","last_clean":"2024-03-24T01:24:17.808527+0000","last_became_active":"2024-03-24T01:24:10.725471+0000","last_became_peered":"2024-03-24T01:24:10.725471+0000","last_unstale":"2024-03-24T01:24:17.808527+0000","last_undegraded":"2024-03-24T01:24:17.808527+0000","last_fullsized":"2024-03-24T01:24:17.808527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:55:14.169541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809601+0000","last_change":"2024-03-24T01:24:09.726977+0000","last_active":"2024-03-24T01:24:17.809601+0000","last_peered":"2024-03-24T01:24:17.809601+0000","last_clean":"2024-03-24T01:24:17.809601+0000","last_became_active":"2024-03-24T01:24:09.726515+0000","last_became_peered":"2024-03-24T01:24:09.726515+0000","last_unstale":"2024-03-24T01:24:17.809601+0000","last_undegraded":"2024-03-24T01:24:17.809601+0000","last_fullsized":"2024-03-24T01:24:17.809601+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:58:17.247099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809443+0000","last_change":"2024-03-24T01:24:15.662691+0000","last_active":"2024-03-24T01:24:17.809443+0000","last_peered":"2024-03-24T01:24:17.809443+0000","last_clean":"2024-03-24T01:24:17.809443+0000","last_became_active":"2024-03-24T01:24:15.657278+0000","last_became_peered":"2024-03-24T01:24:15.657278+0000","last_unstale":"2024-03-24T01:24:17.809443+0000","last_undegraded":"2024-03-24T01:24:17.809443+0000","last_fullsized":"2024-03-24T01:24:17.809443+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:54:28.205416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021915+0000","last_change":"2024-03-24T01:24:10.744706+0000","last_active":"2024-03-24T01:24:17.021915+0000","last_peered":"2024-03-24T01:24:17.021915+0000","last_clean":"2024-03-24T01:24:17.021915+0000","last_became_active":"2024-03-24T01:24:10.744433+0000","last_became_peered":"2024-03-24T01:24:10.744433+0000","last_unstale":"2024-03-24T01:24:17.021915+0000","last_undegraded":"2024-03-24T01:24:17.021915+0000","last_fullsized":"2024-03-24T01:24:17.021915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:09:31.705353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809513+0000","last_change":"2024-03-24T01:24:09.737525+0000","last_active":"2024-03-24T01:24:17.809513+0000","last_peered":"2024-03-24T01:24:17.809513+0000","last_clean":"2024-03-24T01:24:17.809513+0000","last_became_active":"2024-03-24T01:24:09.737206+0000","last_became_peered":"2024-03-24T01:24:09.737206+0000","last_unstale":"2024-03-24T01:24:17.809513+0000","last_undegraded":"2024-03-24T01:24:17.809513+0000","last_fullsized":"2024-03-24T01:24:17.809513+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:44:32.492325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":179,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.808453+0000","last_change":"2024-03-24T01:24:01.545855+0000","last_active":"2024-03-24T01:24:17.808453+0000","last_peered":"2024-03-24T01:24:17.808453+0000","last_clean":"2024-03-24T01:24:17.808453+0000","last_became_active":"2024-03-24T01:24:01.543811+0000","last_became_peered":"2024-03-24T01:24:01.543811+0000","last_unstale":"2024-03-24T01:24:17.808453+0000","last_undegraded":"2024-03-24T01:24:17.808453+0000","last_fullsized":"2024-03-24T01:24:17.808453+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_clean_scrub_stamp":"2024-03-24T01:23:59.861564+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:03.682631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.031327+0000","last_change":"2024-03-24T01:24:15.656774+0000","last_active":"2024-03-24T01:24:17.031327+0000","last_peered":"2024-03-24T01:24:17.031327+0000","last_clean":"2024-03-24T01:24:17.031327+0000","last_became_active":"2024-03-24T01:24:15.656377+0000","last_became_peered":"2024-03-24T01:24:15.656377+0000","last_unstale":"2024-03-24T01:24:17.031327+0000","last_undegraded":"2024-03-24T01:24:17.031327+0000","last_fullsized":"2024-03-24T01:24:17.031327+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:34:04.384239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.812176+0000","last_change":"2024-03-24T01:24:10.766537+0000","last_active":"2024-03-24T01:24:17.812176+0000","last_peered":"2024-03-24T01:24:17.812176+0000","last_clean":"2024-03-24T01:24:17.812176+0000","last_became_active":"2024-03-24T01:24:10.765713+0000","last_became_peered":"2024-03-24T01:24:10.765713+0000","last_unstale":"2024-03-24T01:24:17.812176+0000","last_undegraded":"2024-03-24T01:24:17.812176+0000","last_fullsized":"2024-03-24T01:24:17.812176+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:18:24.974821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.810980+0000","last_change":"2024-03-24T01:24:09.731795+0000","last_active":"2024-03-24T01:24:17.810980+0000","last_peered":"2024-03-24T01:24:17.810980+0000","last_clean":"2024-03-24T01:24:17.810980+0000","last_became_active":"2024-03-24T01:24:09.731547+0000","last_became_peered":"2024-03-24T01:24:09.731547+0000","last_unstale":"2024-03-24T01:24:17.810980+0000","last_undegraded":"2024-03-24T01:24:17.810980+0000","last_fullsized":"2024-03-24T01:24:17.810980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:31:46.512174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028049+0000","last_change":"2024-03-24T01:24:15.656206+0000","last_active":"2024-03-24T01:24:17.028049+0000","last_peered":"2024-03-24T01:24:17.028049+0000","last_clean":"2024-03-24T01:24:17.028049+0000","last_became_active":"2024-03-24T01:24:15.655557+0000","last_became_peered":"2024-03-24T01:24:15.655557+0000","last_unstale":"2024-03-24T01:24:17.028049+0000","last_undegraded":"2024-03-24T01:24:17.028049+0000","last_fullsized":"2024-03-24T01:24:17.028049+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:18:29.263700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811389+0000","last_change":"2024-03-24T01:24:10.750476+0000","last_active":"2024-03-24T01:24:17.811389+0000","last_peered":"2024-03-24T01:24:17.811389+0000","last_clean":"2024-03-24T01:24:17.811389+0000","last_became_active":"2024-03-24T01:24:10.750171+0000","last_became_peered":"2024-03-24T01:24:10.750171+0000","last_unstale":"2024-03-24T01:24:17.811389+0000","last_undegraded":"2024-03-24T01:24:17.811389+0000","last_fullsized":"2024-03-24T01:24:17.811389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:37:21.153754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.739005+0000","last_change":"2024-03-24T01:24:09.718128+0000","last_active":"2024-03-24T01:24:17.739005+0000","last_peered":"2024-03-24T01:24:17.739005+0000","last_clean":"2024-03-24T01:24:17.739005+0000","last_became_active":"2024-03-24T01:24:09.717669+0000","last_became_peered":"2024-03-24T01:24:09.717669+0000","last_unstale":"2024-03-24T01:24:17.739005+0000","last_undegraded":"2024-03-24T01:24:17.739005+0000","last_fullsized":"2024-03-24T01:24:17.739005+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:21:50.689532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.810011+0000","last_change":"2024-03-24T01:24:15.033291+0000","last_active":"2024-03-24T01:24:17.810011+0000","last_peered":"2024-03-24T01:24:17.810011+0000","last_clean":"2024-03-24T01:24:17.810011+0000","last_became_active":"2024-03-24T01:24:15.032790+0000","last_became_peered":"2024-03-24T01:24:15.032790+0000","last_unstale":"2024-03-24T01:24:17.810011+0000","last_undegraded":"2024-03-24T01:24:17.810011+0000","last_fullsized":"2024-03-24T01:24:17.810011+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:52:10.223780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.813313+0000","last_change":"2024-03-24T01:24:10.765266+0000","last_active":"2024-03-24T01:24:17.813313+0000","last_peered":"2024-03-24T01:24:17.813313+0000","last_clean":"2024-03-24T01:24:17.813313+0000","last_became_active":"2024-03-24T01:24:10.764758+0000","last_became_peered":"2024-03-24T01:24:10.764758+0000","last_unstale":"2024-03-24T01:24:17.813313+0000","last_undegraded":"2024-03-24T01:24:17.813313+0000","last_fullsized":"2024-03-24T01:24:17.813313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:34:00.095359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.807653+0000","last_change":"2024-03-24T01:24:09.714389+0000","last_active":"2024-03-24T01:24:17.807653+0000","last_peered":"2024-03-24T01:24:17.807653+0000","last_clean":"2024-03-24T01:24:17.807653+0000","last_became_active":"2024-03-24T01:24:09.713985+0000","last_became_peered":"2024-03-24T01:24:09.713985+0000","last_unstale":"2024-03-24T01:24:17.807653+0000","last_undegraded":"2024-03-24T01:24:17.807653+0000","last_fullsized":"2024-03-24T01:24:17.807653+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:50:31.967556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809004+0000","last_change":"2024-03-24T01:24:15.656465+0000","last_active":"2024-03-24T01:24:17.809004+0000","last_peered":"2024-03-24T01:24:17.809004+0000","last_clean":"2024-03-24T01:24:17.809004+0000","last_became_active":"2024-03-24T01:24:15.655849+0000","last_became_peered":"2024-03-24T01:24:15.655849+0000","last_unstale":"2024-03-24T01:24:17.809004+0000","last_undegraded":"2024-03-24T01:24:17.809004+0000","last_fullsized":"2024-03-24T01:24:17.809004+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:00.868691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.812175+0000","last_change":"2024-03-24T01:24:10.767829+0000","last_active":"2024-03-24T01:24:17.812175+0000","last_peered":"2024-03-24T01:24:17.812175+0000","last_clean":"2024-03-24T01:24:17.812175+0000","last_became_active":"2024-03-24T01:24:10.766346+0000","last_became_peered":"2024-03-24T01:24:10.766346+0000","last_unstale":"2024-03-24T01:24:17.812175+0000","last_undegraded":"2024-03-24T01:24:17.812175+0000","last_fullsized":"2024-03-24T01:24:17.812175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:24.434151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028899+0000","last_change":"2024-03-24T01:24:09.723282+0000","last_active":"2024-03-24T01:24:17.028899+0000","last_peered":"2024-03-24T01:24:17.028899+0000","last_clean":"2024-03-24T01:24:17.028899+0000","last_became_active":"2024-03-24T01:24:09.722727+0000","last_became_peered":"2024-03-24T01:24:09.722727+0000","last_unstale":"2024-03-24T01:24:17.028899+0000","last_undegraded":"2024-03-24T01:24:17.028899+0000","last_fullsized":"2024-03-24T01:24:17.028899+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:40:14.201223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.027169+0000","last_change":"2024-03-24T01:24:15.027468+0000","last_active":"2024-03-24T01:24:17.027169+0000","last_peered":"2024-03-24T01:24:17.027169+0000","last_clean":"2024-03-24T01:24:17.027169+0000","last_became_active":"2024-03-24T01:24:15.027086+0000","last_became_peered":"2024-03-24T01:24:15.027086+0000","last_unstale":"2024-03-24T01:24:17.027169+0000","last_undegraded":"2024-03-24T01:24:17.027169+0000","last_fullsized":"2024-03-24T01:24:17.027169+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:34:04.384239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809223+0000","last_change":"2024-03-24T01:24:10.756853+0000","last_active":"2024-03-24T01:24:17.809223+0000","last_peered":"2024-03-24T01:24:17.809223+0000","last_clean":"2024-03-24T01:24:17.809223+0000","last_became_active":"2024-03-24T01:24:10.756575+0000","last_became_peered":"2024-03-24T01:24:10.756575+0000","last_unstale":"2024-03-24T01:24:17.809223+0000","last_undegraded":"2024-03-24T01:24:17.809223+0000","last_fullsized":"2024-03-24T01:24:17.809223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:26:02.815847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.808921+0000","last_change":"2024-03-24T01:24:09.720936+0000","last_active":"2024-03-24T01:24:17.808921+0000","last_peered":"2024-03-24T01:24:17.808921+0000","last_clean":"2024-03-24T01:24:17.808921+0000","last_became_active":"2024-03-24T01:24:09.720598+0000","last_became_peered":"2024-03-24T01:24:09.720598+0000","last_unstale":"2024-03-24T01:24:17.808921+0000","last_undegraded":"2024-03-24T01:24:17.808921+0000","last_fullsized":"2024-03-24T01:24:17.808921+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:44:32.492325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809525+0000","last_change":"2024-03-24T01:24:15.027473+0000","last_active":"2024-03-24T01:24:17.809525+0000","last_peered":"2024-03-24T01:24:17.809525+0000","last_clean":"2024-03-24T01:24:17.809525+0000","last_became_active":"2024-03-24T01:24:15.025176+0000","last_became_peered":"2024-03-24T01:24:15.025176+0000","last_unstale":"2024-03-24T01:24:17.809525+0000","last_undegraded":"2024-03-24T01:24:17.809525+0000","last_fullsized":"2024-03-24T01:24:17.809525+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:52:30.401135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.030178+0000","last_change":"2024-03-24T01:24:10.737343+0000","last_active":"2024-03-24T01:24:17.030178+0000","last_peered":"2024-03-24T01:24:17.030178+0000","last_clean":"2024-03-24T01:24:17.030178+0000","last_became_active":"2024-03-24T01:24:10.737040+0000","last_became_peered":"2024-03-24T01:24:10.737040+0000","last_unstale":"2024-03-24T01:24:17.030178+0000","last_undegraded":"2024-03-24T01:24:17.030178+0000","last_fullsized":"2024-03-24T01:24:17.030178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:54:57.105205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.029522+0000","last_change":"2024-03-24T01:24:09.726911+0000","last_active":"2024-03-24T01:24:17.029522+0000","last_peered":"2024-03-24T01:24:17.029522+0000","last_clean":"2024-03-24T01:24:17.029522+0000","last_became_active":"2024-03-24T01:24:09.726578+0000","last_became_peered":"2024-03-24T01:24:09.726578+0000","last_unstale":"2024-03-24T01:24:17.029522+0000","last_undegraded":"2024-03-24T01:24:17.029522+0000","last_fullsized":"2024-03-24T01:24:17.029522+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:44:32.492325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.023074+0000","last_change":"2024-03-24T01:24:15.660631+0000","last_active":"2024-03-24T01:24:17.023074+0000","last_peered":"2024-03-24T01:24:17.023074+0000","last_clean":"2024-03-24T01:24:17.023074+0000","last_became_active":"2024-03-24T01:24:15.656242+0000","last_became_peered":"2024-03-24T01:24:15.656242+0000","last_unstale":"2024-03-24T01:24:17.023074+0000","last_undegraded":"2024-03-24T01:24:17.023074+0000","last_fullsized":"2024-03-24T01:24:17.023074+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:10:41.483921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811861+0000","last_change":"2024-03-24T01:24:10.762123+0000","last_active":"2024-03-24T01:24:17.811861+0000","last_peered":"2024-03-24T01:24:17.811861+0000","last_clean":"2024-03-24T01:24:17.811861+0000","last_became_active":"2024-03-24T01:24:10.761599+0000","last_became_peered":"2024-03-24T01:24:10.761599+0000","last_unstale":"2024-03-24T01:24:17.811861+0000","last_undegraded":"2024-03-24T01:24:17.811861+0000","last_fullsized":"2024-03-24T01:24:17.811861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:54:23.916536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809077+0000","last_change":"2024-03-24T01:24:09.735604+0000","last_active":"2024-03-24T01:24:17.809077+0000","last_peered":"2024-03-24T01:24:17.809077+0000","last_clean":"2024-03-24T01:24:17.809077+0000","last_became_active":"2024-03-24T01:24:09.735373+0000","last_became_peered":"2024-03-24T01:24:09.735373+0000","last_unstale":"2024-03-24T01:24:17.809077+0000","last_undegraded":"2024-03-24T01:24:17.809077+0000","last_fullsized":"2024-03-24T01:24:17.809077+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:36:52.007109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022699+0000","last_change":"2024-03-24T01:24:15.660466+0000","last_active":"2024-03-24T01:24:17.022699+0000","last_peered":"2024-03-24T01:24:17.022699+0000","last_clean":"2024-03-24T01:24:17.022699+0000","last_became_active":"2024-03-24T01:24:15.659265+0000","last_became_peered":"2024-03-24T01:24:15.659265+0000","last_unstale":"2024-03-24T01:24:17.022699+0000","last_undegraded":"2024-03-24T01:24:17.022699+0000","last_fullsized":"2024-03-24T01:24:17.022699+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:56.567936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809258+0000","last_change":"2024-03-24T01:24:10.762021+0000","last_active":"2024-03-24T01:24:17.809258+0000","last_peered":"2024-03-24T01:24:17.809258+0000","last_clean":"2024-03-24T01:24:17.809258+0000","last_became_active":"2024-03-24T01:24:10.761759+0000","last_became_peered":"2024-03-24T01:24:10.761759+0000","last_unstale":"2024-03-24T01:24:17.809258+0000","last_undegraded":"2024-03-24T01:24:17.809258+0000","last_fullsized":"2024-03-24T01:24:17.809258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:49:42.059380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.807512+0000","last_change":"2024-03-24T01:24:09.726149+0000","last_active":"2024-03-24T01:24:17.807512+0000","last_peered":"2024-03-24T01:24:17.807512+0000","last_clean":"2024-03-24T01:24:17.807512+0000","last_became_active":"2024-03-24T01:24:09.725594+0000","last_became_peered":"2024-03-24T01:24:09.725594+0000","last_unstale":"2024-03-24T01:24:17.807512+0000","last_undegraded":"2024-03-24T01:24:17.807512+0000","last_fullsized":"2024-03-24T01:24:17.807512+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:28:34.408651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811101+0000","last_change":"2024-03-24T01:24:15.655846+0000","last_active":"2024-03-24T01:24:17.811101+0000","last_peered":"2024-03-24T01:24:17.811101+0000","last_clean":"2024-03-24T01:24:17.811101+0000","last_became_active":"2024-03-24T01:24:15.655258+0000","last_became_peered":"2024-03-24T01:24:15.655258+0000","last_unstale":"2024-03-24T01:24:17.811101+0000","last_undegraded":"2024-03-24T01:24:17.811101+0000","last_fullsized":"2024-03-24T01:24:17.811101+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:52:10.223780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811833+0000","last_change":"2024-03-24T01:24:10.769350+0000","last_active":"2024-03-24T01:24:17.811833+0000","last_peered":"2024-03-24T01:24:17.811833+0000","last_clean":"2024-03-24T01:24:17.811833+0000","last_became_active":"2024-03-24T01:24:10.766162+0000","last_became_peered":"2024-03-24T01:24:10.766162+0000","last_unstale":"2024-03-24T01:24:17.811833+0000","last_undegraded":"2024-03-24T01:24:17.811833+0000","last_fullsized":"2024-03-24T01:24:17.811833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:38:07.266851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'10","reported_seq":34,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:28.484782+0000","last_change":"2024-03-24T01:24:09.716226+0000","last_active":"2024-03-24T01:24:28.484782+0000","last_peered":"2024-03-24T01:24:28.484782+0000","last_clean":"2024-03-24T01:24:28.484782+0000","last_became_active":"2024-03-24T01:24:09.715772+0000","last_became_peered":"2024-03-24T01:24:09.715772+0000","last_unstale":"2024-03-24T01:24:28.484782+0000","last_undegraded":"2024-03-24T01:24:28.484782+0000","last_fullsized":"2024-03-24T01:24:28.484782+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:50:31.967556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.740168+0000","last_change":"2024-03-24T01:24:15.024014+0000","last_active":"2024-03-24T01:24:17.740168+0000","last_peered":"2024-03-24T01:24:17.740168+0000","last_clean":"2024-03-24T01:24:17.740168+0000","last_became_active":"2024-03-24T01:24:15.023619+0000","last_became_peered":"2024-03-24T01:24:15.023619+0000","last_unstale":"2024-03-24T01:24:17.740168+0000","last_undegraded":"2024-03-24T01:24:17.740168+0000","last_fullsized":"2024-03-24T01:24:17.740168+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:45:17.532730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028200+0000","last_change":"2024-03-24T01:24:10.745876+0000","last_active":"2024-03-24T01:24:17.028200+0000","last_peered":"2024-03-24T01:24:17.028200+0000","last_clean":"2024-03-24T01:24:17.028200+0000","last_became_active":"2024-03-24T01:24:10.745539+0000","last_became_peered":"2024-03-24T01:24:10.745539+0000","last_unstale":"2024-03-24T01:24:17.028200+0000","last_undegraded":"2024-03-24T01:24:17.028200+0000","last_fullsized":"2024-03-24T01:24:17.028200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:56:34.907995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.740552+0000","last_change":"2024-03-24T01:24:09.736074+0000","last_active":"2024-03-24T01:24:17.740552+0000","last_peered":"2024-03-24T01:24:17.740552+0000","last_clean":"2024-03-24T01:24:17.740552+0000","last_became_active":"2024-03-24T01:24:09.735807+0000","last_became_peered":"2024-03-24T01:24:09.735807+0000","last_unstale":"2024-03-24T01:24:17.740552+0000","last_undegraded":"2024-03-24T01:24:17.740552+0000","last_fullsized":"2024-03-24T01:24:17.740552+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:50:31.967556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.027828+0000","last_change":"2024-03-24T01:24:15.031909+0000","last_active":"2024-03-24T01:24:17.027828+0000","last_peered":"2024-03-24T01:24:17.027828+0000","last_clean":"2024-03-24T01:24:17.027828+0000","last_became_active":"2024-03-24T01:24:15.031560+0000","last_became_peered":"2024-03-24T01:24:15.031560+0000","last_unstale":"2024-03-24T01:24:17.027828+0000","last_undegraded":"2024-03-24T01:24:17.027828+0000","last_fullsized":"2024-03-24T01:24:17.027828+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:52:30.401135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811414+0000","last_change":"2024-03-24T01:24:10.753950+0000","last_active":"2024-03-24T01:24:17.811414+0000","last_peered":"2024-03-24T01:24:17.811414+0000","last_clean":"2024-03-24T01:24:17.811414+0000","last_became_active":"2024-03-24T01:24:10.742209+0000","last_became_peered":"2024-03-24T01:24:10.742209+0000","last_unstale":"2024-03-24T01:24:17.811414+0000","last_undegraded":"2024-03-24T01:24:17.811414+0000","last_fullsized":"2024-03-24T01:24:17.811414+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:17.138453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809846+0000","last_change":"2024-03-24T01:24:09.735745+0000","last_active":"2024-03-24T01:24:17.809846+0000","last_peered":"2024-03-24T01:24:17.809846+0000","last_clean":"2024-03-24T01:24:17.809846+0000","last_became_active":"2024-03-24T01:24:09.735438+0000","last_became_peered":"2024-03-24T01:24:09.735438+0000","last_unstale":"2024-03-24T01:24:17.809846+0000","last_undegraded":"2024-03-24T01:24:17.809846+0000","last_fullsized":"2024-03-24T01:24:17.809846+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:58:17.247099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021471+0000","last_change":"2024-03-24T01:24:15.014619+0000","last_active":"2024-03-24T01:24:17.021471+0000","last_peered":"2024-03-24T01:24:17.021471+0000","last_clean":"2024-03-24T01:24:17.021471+0000","last_became_active":"2024-03-24T01:24:15.010859+0000","last_became_peered":"2024-03-24T01:24:15.010859+0000","last_unstale":"2024-03-24T01:24:17.021471+0000","last_undegraded":"2024-03-24T01:24:17.021471+0000","last_fullsized":"2024-03-24T01:24:17.021471+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:45:23.567638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811149+0000","last_change":"2024-03-24T01:24:10.767592+0000","last_active":"2024-03-24T01:24:17.811149+0000","last_peered":"2024-03-24T01:24:17.811149+0000","last_clean":"2024-03-24T01:24:17.811149+0000","last_became_active":"2024-03-24T01:24:10.767264+0000","last_became_peered":"2024-03-24T01:24:10.767264+0000","last_unstale":"2024-03-24T01:24:17.811149+0000","last_undegraded":"2024-03-24T01:24:17.811149+0000","last_fullsized":"2024-03-24T01:24:17.811149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:18:24.974821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.031354+0000","last_change":"2024-03-24T01:24:09.738634+0000","last_active":"2024-03-24T01:24:17.031354+0000","last_peered":"2024-03-24T01:24:17.031354+0000","last_clean":"2024-03-24T01:24:17.031354+0000","last_became_active":"2024-03-24T01:24:09.738308+0000","last_became_peered":"2024-03-24T01:24:09.738308+0000","last_unstale":"2024-03-24T01:24:17.031354+0000","last_undegraded":"2024-03-24T01:24:17.031354+0000","last_fullsized":"2024-03-24T01:24:17.031354+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:36:52.007109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021706+0000","last_change":"2024-03-24T01:24:15.666864+0000","last_active":"2024-03-24T01:24:17.021706+0000","last_peered":"2024-03-24T01:24:17.021706+0000","last_clean":"2024-03-24T01:24:17.021706+0000","last_became_active":"2024-03-24T01:24:15.659581+0000","last_became_peered":"2024-03-24T01:24:15.659581+0000","last_unstale":"2024-03-24T01:24:17.021706+0000","last_undegraded":"2024-03-24T01:24:17.021706+0000","last_fullsized":"2024-03-24T01:24:17.021706+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:40:09.145961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809258+0000","last_change":"2024-03-24T01:24:15.657385+0000","last_active":"2024-03-24T01:24:17.809258+0000","last_peered":"2024-03-24T01:24:17.809258+0000","last_clean":"2024-03-24T01:24:17.809258+0000","last_became_active":"2024-03-24T01:24:15.656275+0000","last_became_peered":"2024-03-24T01:24:15.656275+0000","last_unstale":"2024-03-24T01:24:17.809258+0000","last_undegraded":"2024-03-24T01:24:17.809258+0000","last_fullsized":"2024-03-24T01:24:17.809258+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:45:17.532730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.739356+0000","last_change":"2024-03-24T01:24:10.750827+0000","last_active":"2024-03-24T01:24:17.739356+0000","last_peered":"2024-03-24T01:24:17.739356+0000","last_clean":"2024-03-24T01:24:17.739356+0000","last_became_active":"2024-03-24T01:24:10.750522+0000","last_became_peered":"2024-03-24T01:24:10.750522+0000","last_unstale":"2024-03-24T01:24:17.739356+0000","last_undegraded":"2024-03-24T01:24:17.739356+0000","last_fullsized":"2024-03-24T01:24:17.739356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:04:48.028091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.808340+0000","last_change":"2024-03-24T01:24:09.730771+0000","last_active":"2024-03-24T01:24:17.808340+0000","last_peered":"2024-03-24T01:24:17.808340+0000","last_clean":"2024-03-24T01:24:17.808340+0000","last_became_active":"2024-03-24T01:24:09.730424+0000","last_became_peered":"2024-03-24T01:24:09.730424+0000","last_unstale":"2024-03-24T01:24:17.808340+0000","last_undegraded":"2024-03-24T01:24:17.808340+0000","last_fullsized":"2024-03-24T01:24:17.808340+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:50:57.480847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.031001+0000","last_change":"2024-03-24T01:24:15.029656+0000","last_active":"2024-03-24T01:24:17.031001+0000","last_peered":"2024-03-24T01:24:17.031001+0000","last_clean":"2024-03-24T01:24:17.031001+0000","last_became_active":"2024-03-24T01:24:15.029240+0000","last_became_peered":"2024-03-24T01:24:15.029240+0000","last_unstale":"2024-03-24T01:24:17.031001+0000","last_undegraded":"2024-03-24T01:24:17.031001+0000","last_fullsized":"2024-03-24T01:24:17.031001+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:28.723031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021468+0000","last_change":"2024-03-24T01:24:10.750421+0000","last_active":"2024-03-24T01:24:17.021468+0000","last_peered":"2024-03-24T01:24:17.021468+0000","last_clean":"2024-03-24T01:24:17.021468+0000","last_became_active":"2024-03-24T01:24:10.750160+0000","last_became_peered":"2024-03-24T01:24:10.750160+0000","last_unstale":"2024-03-24T01:24:17.021468+0000","last_undegraded":"2024-03-24T01:24:17.021468+0000","last_fullsized":"2024-03-24T01:24:17.021468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:54:57.105205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022107+0000","last_change":"2024-03-24T01:24:09.722993+0000","last_active":"2024-03-24T01:24:17.022107+0000","last_peered":"2024-03-24T01:24:17.022107+0000","last_clean":"2024-03-24T01:24:17.022107+0000","last_became_active":"2024-03-24T01:24:09.720133+0000","last_became_peered":"2024-03-24T01:24:09.720133+0000","last_unstale":"2024-03-24T01:24:17.022107+0000","last_undegraded":"2024-03-24T01:24:17.022107+0000","last_fullsized":"2024-03-24T01:24:17.022107+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:28:34.408651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.030554+0000","last_change":"2024-03-24T01:24:09.726157+0000","last_active":"2024-03-24T01:24:17.030554+0000","last_peered":"2024-03-24T01:24:17.030554+0000","last_clean":"2024-03-24T01:24:17.030554+0000","last_became_active":"2024-03-24T01:24:09.724571+0000","last_became_peered":"2024-03-24T01:24:09.724571+0000","last_unstale":"2024-03-24T01:24:17.030554+0000","last_undegraded":"2024-03-24T01:24:17.030554+0000","last_fullsized":"2024-03-24T01:24:17.030554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:33.381248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.810098+0000","last_change":"2024-03-24T01:24:10.726759+0000","last_active":"2024-03-24T01:24:17.810098+0000","last_peered":"2024-03-24T01:24:17.810098+0000","last_clean":"2024-03-24T01:24:17.810098+0000","last_became_active":"2024-03-24T01:24:10.726133+0000","last_became_peered":"2024-03-24T01:24:10.726133+0000","last_unstale":"2024-03-24T01:24:17.810098+0000","last_undegraded":"2024-03-24T01:24:17.810098+0000","last_fullsized":"2024-03-24T01:24:17.810098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:46:04.235199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.810763+0000","last_change":"2024-03-24T01:24:15.026408+0000","last_active":"2024-03-24T01:24:17.810763+0000","last_peered":"2024-03-24T01:24:17.810763+0000","last_clean":"2024-03-24T01:24:17.810763+0000","last_became_active":"2024-03-24T01:24:15.026038+0000","last_became_peered":"2024-03-24T01:24:15.026038+0000","last_unstale":"2024-03-24T01:24:17.810763+0000","last_undegraded":"2024-03-24T01:24:17.810763+0000","last_fullsized":"2024-03-24T01:24:17.810763+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:55:18.458421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022088+0000","last_change":"2024-03-24T01:24:09.722548+0000","last_active":"2024-03-24T01:24:17.022088+0000","last_peered":"2024-03-24T01:24:17.022088+0000","last_clean":"2024-03-24T01:24:17.022088+0000","last_became_active":"2024-03-24T01:24:09.721937+0000","last_became_peered":"2024-03-24T01:24:09.721937+0000","last_unstale":"2024-03-24T01:24:17.022088+0000","last_undegraded":"2024-03-24T01:24:17.022088+0000","last_fullsized":"2024-03-24T01:24:17.022088+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:50:57.480847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.027732+0000","last_change":"2024-03-24T01:24:10.743972+0000","last_active":"2024-03-24T01:24:17.027732+0000","last_peered":"2024-03-24T01:24:17.027732+0000","last_clean":"2024-03-24T01:24:17.027732+0000","last_became_active":"2024-03-24T01:24:10.743700+0000","last_became_peered":"2024-03-24T01:24:10.743700+0000","last_unstale":"2024-03-24T01:24:17.027732+0000","last_undegraded":"2024-03-24T01:24:17.027732+0000","last_fullsized":"2024-03-24T01:24:17.027732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:21:51.701615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022373+0000","last_change":"2024-03-24T01:24:15.023995+0000","last_active":"2024-03-24T01:24:17.022373+0000","last_peered":"2024-03-24T01:24:17.022373+0000","last_clean":"2024-03-24T01:24:17.022373+0000","last_became_active":"2024-03-24T01:24:15.023574+0000","last_became_peered":"2024-03-24T01:24:15.023574+0000","last_unstale":"2024-03-24T01:24:17.022373+0000","last_undegraded":"2024-03-24T01:24:17.022373+0000","last_fullsized":"2024-03-24T01:24:17.022373+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:26:01.912499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.029925+0000","last_change":"2024-03-24T01:24:10.740908+0000","last_active":"2024-03-24T01:24:17.029925+0000","last_peered":"2024-03-24T01:24:17.029925+0000","last_clean":"2024-03-24T01:24:17.029925+0000","last_became_active":"2024-03-24T01:24:10.740630+0000","last_became_peered":"2024-03-24T01:24:10.740630+0000","last_unstale":"2024-03-24T01:24:17.029925+0000","last_undegraded":"2024-03-24T01:24:17.029925+0000","last_fullsized":"2024-03-24T01:24:17.029925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:04:48.028091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809885+0000","last_change":"2024-03-24T01:24:10.743826+0000","last_active":"2024-03-24T01:24:17.809885+0000","last_peered":"2024-03-24T01:24:17.809885+0000","last_clean":"2024-03-24T01:24:17.809885+0000","last_became_active":"2024-03-24T01:24:10.743349+0000","last_became_peered":"2024-03-24T01:24:10.743349+0000","last_unstale":"2024-03-24T01:24:17.809885+0000","last_undegraded":"2024-03-24T01:24:17.809885+0000","last_fullsized":"2024-03-24T01:24:17.809885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:56:34.907995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.810496+0000","last_change":"2024-03-24T01:24:10.755545+0000","last_active":"2024-03-24T01:24:17.810496+0000","last_peered":"2024-03-24T01:24:17.810496+0000","last_clean":"2024-03-24T01:24:17.810496+0000","last_became_active":"2024-03-24T01:24:10.755147+0000","last_became_peered":"2024-03-24T01:24:10.755147+0000","last_unstale":"2024-03-24T01:24:17.810496+0000","last_undegraded":"2024-03-24T01:24:17.810496+0000","last_fullsized":"2024-03-24T01:24:17.810496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:35.441197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809878+0000","last_change":"2024-03-24T01:24:10.746109+0000","last_active":"2024-03-24T01:24:17.809878+0000","last_peered":"2024-03-24T01:24:17.809878+0000","last_clean":"2024-03-24T01:24:17.809878+0000","last_became_active":"2024-03-24T01:24:10.745809+0000","last_became_peered":"2024-03-24T01:24:10.745809+0000","last_unstale":"2024-03-24T01:24:17.809878+0000","last_undegraded":"2024-03-24T01:24:17.809878+0000","last_fullsized":"2024-03-24T01:24:17.809878+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:09:31.705353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022898+0000","last_change":"2024-03-24T01:24:10.752362+0000","last_active":"2024-03-24T01:24:17.022898+0000","last_peered":"2024-03-24T01:24:17.022898+0000","last_clean":"2024-03-24T01:24:17.022898+0000","last_became_active":"2024-03-24T01:24:10.752066+0000","last_became_peered":"2024-03-24T01:24:10.752066+0000","last_unstale":"2024-03-24T01:24:17.022898+0000","last_undegraded":"2024-03-24T01:24:17.022898+0000","last_fullsized":"2024-03-24T01:24:17.022898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:08:13.760239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.740402+0000","last_change":"2024-03-24T01:24:10.751536+0000","last_active":"2024-03-24T01:24:17.740402+0000","last_peered":"2024-03-24T01:24:17.740402+0000","last_clean":"2024-03-24T01:24:17.740402+0000","last_became_active":"2024-03-24T01:24:10.751260+0000","last_became_peered":"2024-03-24T01:24:10.751260+0000","last_unstale":"2024-03-24T01:24:17.740402+0000","last_undegraded":"2024-03-24T01:24:17.740402+0000","last_fullsized":"2024-03-24T01:24:17.740402+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:17.138453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.031836+0000","last_change":"2024-03-24T01:24:10.763037+0000","last_active":"2024-03-24T01:24:17.031836+0000","last_peered":"2024-03-24T01:24:17.031836+0000","last_clean":"2024-03-24T01:24:17.031836+0000","last_became_active":"2024-03-24T01:24:10.762718+0000","last_became_peered":"2024-03-24T01:24:10.762718+0000","last_unstale":"2024-03-24T01:24:17.031836+0000","last_undegraded":"2024-03-24T01:24:17.031836+0000","last_fullsized":"2024-03-24T01:24:17.031836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:46:04.235199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022992+0000","last_change":"2024-03-24T01:24:10.753332+0000","last_active":"2024-03-24T01:24:17.022992+0000","last_peered":"2024-03-24T01:24:17.022992+0000","last_clean":"2024-03-24T01:24:17.022992+0000","last_became_active":"2024-03-24T01:24:10.752905+0000","last_became_peered":"2024-03-24T01:24:10.752905+0000","last_unstale":"2024-03-24T01:24:17.022992+0000","last_undegraded":"2024-03-24T01:24:17.022992+0000","last_fullsized":"2024-03-24T01:24:17.022992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:57:11.968706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809861+0000","last_change":"2024-03-24T01:24:10.759635+0000","last_active":"2024-03-24T01:24:17.809861+0000","last_peered":"2024-03-24T01:24:17.809861+0000","last_clean":"2024-03-24T01:24:17.809861+0000","last_became_active":"2024-03-24T01:24:10.759310+0000","last_became_peered":"2024-03-24T01:24:10.759310+0000","last_unstale":"2024-03-24T01:24:17.809861+0000","last_undegraded":"2024-03-24T01:24:17.809861+0000","last_fullsized":"2024-03-24T01:24:17.809861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:08:13.760239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021975+0000","last_change":"2024-03-24T01:24:10.746635+0000","last_active":"2024-03-24T01:24:17.021975+0000","last_peered":"2024-03-24T01:24:17.021975+0000","last_clean":"2024-03-24T01:24:17.021975+0000","last_became_active":"2024-03-24T01:24:10.745597+0000","last_became_peered":"2024-03-24T01:24:10.745597+0000","last_unstale":"2024-03-24T01:24:17.021975+0000","last_undegraded":"2024-03-24T01:24:17.021975+0000","last_fullsized":"2024-03-24T01:24:17.021975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:42:22.671154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.810268+0000","last_change":"2024-03-24T01:24:10.743389+0000","last_active":"2024-03-24T01:24:17.810268+0000","last_peered":"2024-03-24T01:24:17.810268+0000","last_clean":"2024-03-24T01:24:17.810268+0000","last_became_active":"2024-03-24T01:24:10.743128+0000","last_became_peered":"2024-03-24T01:24:10.743128+0000","last_unstale":"2024-03-24T01:24:17.810268+0000","last_undegraded":"2024-03-24T01:24:17.810268+0000","last_fullsized":"2024-03-24T01:24:17.810268+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:42:22.671154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809576+0000","last_change":"2024-03-24T01:24:10.760940+0000","last_active":"2024-03-24T01:24:17.809576+0000","last_peered":"2024-03-24T01:24:17.809576+0000","last_clean":"2024-03-24T01:24:17.809576+0000","last_became_active":"2024-03-24T01:24:10.759991+0000","last_became_peered":"2024-03-24T01:24:10.759991+0000","last_unstale":"2024-03-24T01:24:17.809576+0000","last_undegraded":"2024-03-24T01:24:17.809576+0000","last_fullsized":"2024-03-24T01:24:17.809576+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:54:57.105205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.030845+0000","last_change":"2024-03-24T01:24:10.731391+0000","last_active":"2024-03-24T01:24:17.030845+0000","last_peered":"2024-03-24T01:24:17.030845+0000","last_clean":"2024-03-24T01:24:17.030845+0000","last_became_active":"2024-03-24T01:24:10.731094+0000","last_became_peered":"2024-03-24T01:24:10.731094+0000","last_unstale":"2024-03-24T01:24:17.030845+0000","last_undegraded":"2024-03-24T01:24:17.030845+0000","last_fullsized":"2024-03-24T01:24:17.030845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:17.138453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.812469+0000","last_change":"2024-03-24T01:24:10.766746+0000","last_active":"2024-03-24T01:24:17.812469+0000","last_peered":"2024-03-24T01:24:17.812469+0000","last_clean":"2024-03-24T01:24:17.812469+0000","last_became_active":"2024-03-24T01:24:10.766239+0000","last_became_peered":"2024-03-24T01:24:10.766239+0000","last_unstale":"2024-03-24T01:24:17.812469+0000","last_undegraded":"2024-03-24T01:24:17.812469+0000","last_fullsized":"2024-03-24T01:24:17.812469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:45:13.243851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811052+0000","last_change":"2024-03-24T01:24:10.741506+0000","last_active":"2024-03-24T01:24:17.811052+0000","last_peered":"2024-03-24T01:24:17.811052+0000","last_clean":"2024-03-24T01:24:17.811052+0000","last_became_active":"2024-03-24T01:24:10.741221+0000","last_became_peered":"2024-03-24T01:24:10.741221+0000","last_unstale":"2024-03-24T01:24:17.811052+0000","last_undegraded":"2024-03-24T01:24:17.811052+0000","last_fullsized":"2024-03-24T01:24:17.811052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:27:51.341849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.812926+0000","last_change":"2024-03-24T01:24:10.767451+0000","last_active":"2024-03-24T01:24:17.812926+0000","last_peered":"2024-03-24T01:24:17.812926+0000","last_clean":"2024-03-24T01:24:17.812926+0000","last_became_active":"2024-03-24T01:24:10.767159+0000","last_became_peered":"2024-03-24T01:24:10.767159+0000","last_unstale":"2024-03-24T01:24:17.812926+0000","last_undegraded":"2024-03-24T01:24:17.812926+0000","last_fullsized":"2024-03-24T01:24:17.812926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:24.434151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809619+0000","last_change":"2024-03-24T01:24:10.751373+0000","last_active":"2024-03-24T01:24:17.809619+0000","last_peered":"2024-03-24T01:24:17.809619+0000","last_clean":"2024-03-24T01:24:17.809619+0000","last_became_active":"2024-03-24T01:24:10.751084+0000","last_became_peered":"2024-03-24T01:24:10.751084+0000","last_unstale":"2024-03-24T01:24:17.809619+0000","last_undegraded":"2024-03-24T01:24:17.809619+0000","last_fullsized":"2024-03-24T01:24:17.809619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:17.138453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.739324+0000","last_change":"2024-03-24T01:24:10.752504+0000","last_active":"2024-03-24T01:24:17.739324+0000","last_peered":"2024-03-24T01:24:17.739324+0000","last_clean":"2024-03-24T01:24:17.739324+0000","last_became_active":"2024-03-24T01:24:10.752313+0000","last_became_peered":"2024-03-24T01:24:10.752313+0000","last_unstale":"2024-03-24T01:24:17.739324+0000","last_undegraded":"2024-03-24T01:24:17.739324+0000","last_fullsized":"2024-03-24T01:24:17.739324+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:54:57.105205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809195+0000","last_change":"2024-03-24T01:24:10.750967+0000","last_active":"2024-03-24T01:24:17.809195+0000","last_peered":"2024-03-24T01:24:17.809195+0000","last_clean":"2024-03-24T01:24:17.809195+0000","last_became_active":"2024-03-24T01:24:10.748587+0000","last_became_peered":"2024-03-24T01:24:10.748587+0000","last_unstale":"2024-03-24T01:24:17.809195+0000","last_undegraded":"2024-03-24T01:24:17.809195+0000","last_fullsized":"2024-03-24T01:24:17.809195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:44:57.802345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021966+0000","last_change":"2024-03-24T01:24:10.750792+0000","last_active":"2024-03-24T01:24:17.021966+0000","last_peered":"2024-03-24T01:24:17.021966+0000","last_clean":"2024-03-24T01:24:17.021966+0000","last_became_active":"2024-03-24T01:24:10.749384+0000","last_became_peered":"2024-03-24T01:24:10.749384+0000","last_unstale":"2024-03-24T01:24:17.021966+0000","last_undegraded":"2024-03-24T01:24:17.021966+0000","last_fullsized":"2024-03-24T01:24:17.021966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:17.138453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028133+0000","last_change":"2024-03-24T01:24:10.728222+0000","last_active":"2024-03-24T01:24:17.028133+0000","last_peered":"2024-03-24T01:24:17.028133+0000","last_clean":"2024-03-24T01:24:17.028133+0000","last_became_active":"2024-03-24T01:24:10.727874+0000","last_became_peered":"2024-03-24T01:24:10.727874+0000","last_unstale":"2024-03-24T01:24:17.028133+0000","last_undegraded":"2024-03-24T01:24:17.028133+0000","last_fullsized":"2024-03-24T01:24:17.028133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:58:18.259182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.031302+0000","last_change":"2024-03-24T01:24:10.741809+0000","last_active":"2024-03-24T01:24:17.031302+0000","last_peered":"2024-03-24T01:24:17.031302+0000","last_clean":"2024-03-24T01:24:17.031302+0000","last_became_active":"2024-03-24T01:24:10.741444+0000","last_became_peered":"2024-03-24T01:24:10.741444+0000","last_unstale":"2024-03-24T01:24:17.031302+0000","last_undegraded":"2024-03-24T01:24:17.031302+0000","last_fullsized":"2024-03-24T01:24:17.031302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:19:51.907364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.031726+0000","last_change":"2024-03-24T01:24:10.751752+0000","last_active":"2024-03-24T01:24:17.031726+0000","last_peered":"2024-03-24T01:24:17.031726+0000","last_clean":"2024-03-24T01:24:17.031726+0000","last_became_active":"2024-03-24T01:24:10.747073+0000","last_became_peered":"2024-03-24T01:24:10.747073+0000","last_unstale":"2024-03-24T01:24:17.031726+0000","last_undegraded":"2024-03-24T01:24:17.031726+0000","last_fullsized":"2024-03-24T01:24:17.031726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:08:13.760239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.813263+0000","last_change":"2024-03-24T01:24:10.765006+0000","last_active":"2024-03-24T01:24:17.813263+0000","last_peered":"2024-03-24T01:24:17.813263+0000","last_clean":"2024-03-24T01:24:17.813263+0000","last_became_active":"2024-03-24T01:24:10.764417+0000","last_became_peered":"2024-03-24T01:24:10.764417+0000","last_unstale":"2024-03-24T01:24:17.813263+0000","last_undegraded":"2024-03-24T01:24:17.813263+0000","last_fullsized":"2024-03-24T01:24:17.813263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:29:34.357876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021162+0000","last_change":"2024-03-24T01:24:10.757149+0000","last_active":"2024-03-24T01:24:17.021162+0000","last_peered":"2024-03-24T01:24:17.021162+0000","last_clean":"2024-03-24T01:24:17.021162+0000","last_became_active":"2024-03-24T01:24:10.756879+0000","last_became_peered":"2024-03-24T01:24:10.756879+0000","last_unstale":"2024-03-24T01:24:17.021162+0000","last_undegraded":"2024-03-24T01:24:17.021162+0000","last_fullsized":"2024-03-24T01:24:17.021162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:55:14.169541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809223+0000","last_change":"2024-03-24T01:24:10.766128+0000","last_active":"2024-03-24T01:24:17.809223+0000","last_peered":"2024-03-24T01:24:17.809223+0000","last_clean":"2024-03-24T01:24:17.809223+0000","last_became_active":"2024-03-24T01:24:10.763007+0000","last_became_peered":"2024-03-24T01:24:10.763007+0000","last_unstale":"2024-03-24T01:24:17.809223+0000","last_undegraded":"2024-03-24T01:24:17.809223+0000","last_fullsized":"2024-03-24T01:24:17.809223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:35.441197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809997+0000","last_change":"2024-03-24T01:24:10.741795+0000","last_active":"2024-03-24T01:24:17.809997+0000","last_peered":"2024-03-24T01:24:17.809997+0000","last_clean":"2024-03-24T01:24:17.809997+0000","last_became_active":"2024-03-24T01:24:10.741516+0000","last_became_peered":"2024-03-24T01:24:10.741516+0000","last_unstale":"2024-03-24T01:24:17.809997+0000","last_undegraded":"2024-03-24T01:24:17.809997+0000","last_fullsized":"2024-03-24T01:24:17.809997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:57:11.968706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811350+0000","last_change":"2024-03-24T01:24:10.761849+0000","last_active":"2024-03-24T01:24:17.811350+0000","last_peered":"2024-03-24T01:24:17.811350+0000","last_clean":"2024-03-24T01:24:17.811350+0000","last_became_active":"2024-03-24T01:24:10.761288+0000","last_became_peered":"2024-03-24T01:24:10.761288+0000","last_unstale":"2024-03-24T01:24:17.811350+0000","last_undegraded":"2024-03-24T01:24:17.811350+0000","last_fullsized":"2024-03-24T01:24:17.811350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:02:18.904076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.810985+0000","last_change":"2024-03-24T01:24:10.755426+0000","last_active":"2024-03-24T01:24:17.810985+0000","last_peered":"2024-03-24T01:24:17.810985+0000","last_clean":"2024-03-24T01:24:17.810985+0000","last_became_active":"2024-03-24T01:24:10.755073+0000","last_became_peered":"2024-03-24T01:24:10.755073+0000","last_unstale":"2024-03-24T01:24:17.810985+0000","last_undegraded":"2024-03-24T01:24:17.810985+0000","last_fullsized":"2024-03-24T01:24:17.810985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:46:04.235199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.810708+0000","last_change":"2024-03-24T01:24:10.732890+0000","last_active":"2024-03-24T01:24:17.810708+0000","last_peered":"2024-03-24T01:24:17.810708+0000","last_clean":"2024-03-24T01:24:17.810708+0000","last_became_active":"2024-03-24T01:24:10.731992+0000","last_became_peered":"2024-03-24T01:24:10.731992+0000","last_unstale":"2024-03-24T01:24:17.810708+0000","last_undegraded":"2024-03-24T01:24:17.810708+0000","last_fullsized":"2024-03-24T01:24:17.810708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:19:51.907364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.741104+0000","last_change":"2024-03-24T01:24:10.750945+0000","last_active":"2024-03-24T01:24:17.741104+0000","last_peered":"2024-03-24T01:24:17.741104+0000","last_clean":"2024-03-24T01:24:17.741104+0000","last_became_active":"2024-03-24T01:24:10.750099+0000","last_became_peered":"2024-03-24T01:24:10.750099+0000","last_unstale":"2024-03-24T01:24:17.741104+0000","last_undegraded":"2024-03-24T01:24:17.741104+0000","last_fullsized":"2024-03-24T01:24:17.741104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:08:13.760239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.813056+0000","last_change":"2024-03-24T01:24:10.762165+0000","last_active":"2024-03-24T01:24:17.813056+0000","last_peered":"2024-03-24T01:24:17.813056+0000","last_clean":"2024-03-24T01:24:17.813056+0000","last_became_active":"2024-03-24T01:24:10.758157+0000","last_became_peered":"2024-03-24T01:24:10.758157+0000","last_unstale":"2024-03-24T01:24:17.813056+0000","last_undegraded":"2024-03-24T01:24:17.813056+0000","last_fullsized":"2024-03-24T01:24:17.813056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:37:21.153754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27864,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343976,"statfs":{"total":96636764160,"available":96608231424,"internally_reserved":0,"allocated":864256,"data_stored":698343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":107483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":107519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":107393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":107553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":872448,"data_stored":698399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T01:24:39.035 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-24T01:24:39.035 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-24T01:24:39.035 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-24T01:24:39.036 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-24T01:24:39.036 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-24T01:24:39.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.282+0000 7f4957200640 1 Processor -- start 2024-03-24T01:24:39.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.282+0000 7f4957200640 1 -- start start 2024-03-24T01:24:39.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.283+0000 7f4957200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f495815ee50 0x7f49580b2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:39.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.283+0000 7f4957200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f49580b2bc0 0x7f49580cf340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:39.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.283+0000 7f4957200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f495815b980 con 0x7f49580cef60 2024-03-24T01:24:39.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.283+0000 7f4957200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f495815bc60 con 0x7f49580b2bc0 2024-03-24T01:24:39.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.283+0000 7f4957200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f495815e650 con 0x7f495815ee50 2024-03-24T01:24:39.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.284+0000 7f4956800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f49580b2bc0 0x7f49580cf340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:39.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.284+0000 7f4956800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f49580b2bc0 0x7f49580cf340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43534/0 (socket says 172.21.15.90:43534) 2024-03-24T01:24:39.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.284+0000 7f4956800640 1 -- 172.21.15.90:0/2571107637 learned_addr learned my addr 172.21.15.90:0/2571107637 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:39.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.284+0000 7f4955400640 1 --2- 172.21.15.90:0/2571107637 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f495815ee50 0x7f49580b2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:39.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.284+0000 7f4956800640 1 -- 172.21.15.90:0/2571107637 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f495815ee50 msgr2=0x7f49580b2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:39.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.284+0000 7f4956800640 1 --2- 172.21.15.90:0/2571107637 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f495815ee50 0x7f49580b2680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:39.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.284+0000 7f4956800640 1 -- 172.21.15.90:0/2571107637 >> v1:172.21.15.120:6789/0 conn(0x7f49580cef60 legacy=0x7f495815ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:39.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.284+0000 7f4956800640 1 -- 172.21.15.90:0/2571107637 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f495815e930 con 0x7f49580b2bc0 2024-03-24T01:24:39.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.285+0000 7f4956800640 1 --2- 172.21.15.90:0/2571107637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49580b2bc0 0x7f49580cf340 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4950009870 tx=0x7f495002db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d78c0c5bf8d2142e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:39.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.286+0000 7f4954a00640 1 -- 172.21.15.90:0/2571107637 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f495003b070 con 0x7f49580b2bc0 2024-03-24T01:24:39.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.286+0000 7f4954a00640 1 -- 172.21.15.90:0/2571107637 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f495002e940 con 0x7f49580b2bc0 2024-03-24T01:24:39.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.286+0000 7f4954a00640 1 -- 172.21.15.90:0/2571107637 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f495002ed10 con 0x7f49580b2bc0 2024-03-24T01:24:39.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.286+0000 7f4957200640 1 -- 172.21.15.90:0/2571107637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49580b2bc0 msgr2=0x7f49580cf340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:39.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.286+0000 7f4957200640 1 --2- 172.21.15.90:0/2571107637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49580b2bc0 0x7f49580cf340 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4950009870 tx=0x7f495002db90 comp rx=0 tx=0).stop 2024-03-24T01:24:39.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.286+0000 7f4957200640 1 -- 172.21.15.90:0/2571107637 shutdown_connections 2024-03-24T01:24:39.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.286+0000 7f4957200640 1 --2- 172.21.15.90:0/2571107637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49580b2bc0 0x7f49580cf340 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:39.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.286+0000 7f4957200640 1 --2- 172.21.15.90:0/2571107637 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f495815ee50 0x7f49580b2680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:39.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.286+0000 7f4957200640 1 -- 172.21.15.90:0/2571107637 >> 172.21.15.90:0/2571107637 conn(0x7f495806b0c0 msgr2=0x7f495806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:39.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.286+0000 7f4957200640 1 -- 172.21.15.90:0/2571107637 shutdown_connections 2024-03-24T01:24:39.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.286+0000 7f4957200640 1 -- 172.21.15.90:0/2571107637 wait complete. 2024-03-24T01:24:39.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.288+0000 7f4957200640 1 Processor -- start 2024-03-24T01:24:39.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.288+0000 7f4957200640 1 -- start start 2024-03-24T01:24:39.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.289+0000 7f4957200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49580b2bc0 0x7f49581451a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:39.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.289+0000 7f4957200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49580cef60 0x7f49580aefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:39.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.289+0000 7f4957200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f495815ee50 0x7f49580b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:39.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.289+0000 7f4957200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f495815b560 con 0x7f495815ee50 2024-03-24T01:24:39.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.289+0000 7f4957200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49580cc080 con 0x7f49580cef60 2024-03-24T01:24:39.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.289+0000 7f4957200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f495815b6e0 con 0x7f49580b2bc0 2024-03-24T01:24:39.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.289+0000 7f4956800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f495815ee50 0x7f49580b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:39.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.290+0000 7f4956800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f495815ee50 0x7f49580b1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43546/0 (socket says 172.21.15.90:43546) 2024-03-24T01:24:39.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.290+0000 7f4956800640 1 -- 172.21.15.90:0/1975392043 learned_addr learned my addr 172.21.15.90:0/1975392043 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:39.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.290+0000 7f4955e00640 1 --2- 172.21.15.90:0/1975392043 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49580b2bc0 0x7f49581451a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:39.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.290+0000 7f4955400640 1 --2- 172.21.15.90:0/1975392043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49580cef60 0x7f49580aefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:39.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.290+0000 7f4956800640 1 -- 172.21.15.90:0/1975392043 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49580b2bc0 msgr2=0x7f49581451a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:39.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.290+0000 7f4956800640 1 --2- 172.21.15.90:0/1975392043 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49580b2bc0 0x7f49581451a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:39.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.290+0000 7f4956800640 1 -- 172.21.15.90:0/1975392043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49580cef60 msgr2=0x7f49580aefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:39.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.290+0000 7f4956800640 1 --2- 172.21.15.90:0/1975392043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49580cef60 0x7f49580aefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:39.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.290+0000 7f4956800640 1 -- 172.21.15.90:0/1975392043 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49580b2780 con 0x7f495815ee50 2024-03-24T01:24:39.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.290+0000 7f4956800640 1 --2- 172.21.15.90:0/1975392043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f495815ee50 0x7f49580b1a00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4950009870 tx=0x7f49500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:39.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.290+0000 7f494e000640 1 -- 172.21.15.90:0/1975392043 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4950046070 con 0x7f495815ee50 2024-03-24T01:24:39.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.291+0000 7f494e000640 1 -- 172.21.15.90:0/1975392043 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4950036d80 con 0x7f495815ee50 2024-03-24T01:24:39.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.291+0000 7f494e000640 1 -- 172.21.15.90:0/1975392043 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f495003b070 con 0x7f495815ee50 2024-03-24T01:24:39.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.291+0000 7f4957200640 1 -- 172.21.15.90:0/1975392043 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49581e4440 con 0x7f495815ee50 2024-03-24T01:24:39.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.291+0000 7f4957200640 1 -- 172.21.15.90:0/1975392043 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49581e4a00 con 0x7f495815ee50 2024-03-24T01:24:39.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.292+0000 7f4957200640 1 -- 172.21.15.90:0/1975392043 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f495804f400 con 0x7f495815ee50 2024-03-24T01:24:39.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.293+0000 7f494e000640 1 -- 172.21.15.90:0/1975392043 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4950049070 con 0x7f495815ee50 2024-03-24T01:24:39.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.294+0000 7f494e000640 1 --2- 172.21.15.90:0/1975392043 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f493406bbd0 0x7f493406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:39.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.294+0000 7f494e000640 1 -- 172.21.15.90:0/1975392043 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f49500ac600 con 0x7f495815ee50 2024-03-24T01:24:39.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.294+0000 7f4955e00640 1 --2- 172.21.15.90:0/1975392043 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f493406bbd0 0x7f493406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:39.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.295+0000 7f4955e00640 1 --2- 172.21.15.90:0/1975392043 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f493406bbd0 0x7f493406e090 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4940007fd0 tx=0x7f4940009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:39.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.298+0000 7f494e000640 1 -- 172.21.15.90:0/1975392043 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f495007ad10 con 0x7f495815ee50 2024-03-24T01:24:39.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.551+0000 7f4957200640 1 -- 172.21.15.90:0/1975392043 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f495806f9f0 con 0x7f495815ee50 2024-03-24T01:24:39.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.560+0000 7f494e000640 1 -- 172.21.15.90:0/1975392043 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+172581 (secure 0 0 0) 0x7f495007a610 con 0x7f495815ee50 2024-03-24T01:24:39.562 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:24:39.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.565+0000 7f4957200640 1 -- 172.21.15.90:0/1975392043 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f493406bbd0 msgr2=0x7f493406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:39.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.565+0000 7f4957200640 1 --2- 172.21.15.90:0/1975392043 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f493406bbd0 0x7f493406e090 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4940007fd0 tx=0x7f4940009040 comp rx=0 tx=0).stop 2024-03-24T01:24:39.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.565+0000 7f4957200640 1 -- 172.21.15.90:0/1975392043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f495815ee50 msgr2=0x7f49580b1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:39.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.565+0000 7f4957200640 1 --2- 172.21.15.90:0/1975392043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f495815ee50 0x7f49580b1a00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4950009870 tx=0x7f49500046c0 comp rx=0 tx=0).stop 2024-03-24T01:24:39.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.565+0000 7f4957200640 1 -- 172.21.15.90:0/1975392043 shutdown_connections 2024-03-24T01:24:39.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.565+0000 7f4957200640 1 --2- 172.21.15.90:0/1975392043 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f493406bbd0 0x7f493406e090 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:39.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.565+0000 7f4957200640 1 --2- 172.21.15.90:0/1975392043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f495815ee50 0x7f49580b1a00 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:39.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.565+0000 7f4957200640 1 --2- 172.21.15.90:0/1975392043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49580cef60 0x7f49580aefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:39.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.565+0000 7f4957200640 1 --2- 172.21.15.90:0/1975392043 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49580b2bc0 0x7f49581451a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:39.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.565+0000 7f4957200640 1 -- 172.21.15.90:0/1975392043 >> 172.21.15.90:0/1975392043 conn(0x7f495806b0c0 msgr2=0x7f49580ce010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:39.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.565+0000 7f4957200640 1 -- 172.21.15.90:0/1975392043 shutdown_connections 2024-03-24T01:24:39.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.565+0000 7f4957200640 1 -- 172.21.15.90:0/1975392043 wait complete. 2024-03-24T01:24:39.583 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":5,"flags":0,"active_gid":4104,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":860392634},{"type":"v1","addr":"172.21.15.120:6833","nonce":860392634}]},"active_addr":"172.21.15.120:6833/860392634","active_change":"2024-03-24T01:23:51.452653+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4110,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.120:0","nonce":661862998}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.120:0","nonce":465688319}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.120:0","nonce":1696993661}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.120:0","nonce":4149151857}]}]} 2024-03-24T01:24:39.586 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-24T01:24:39.587 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-24T01:24:39.587 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:24:39.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.792+0000 7fe46b800640 1 Processor -- start 2024-03-24T01:24:39.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.793+0000 7fe46b800640 1 -- start start 2024-03-24T01:24:39.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.794+0000 7fe46b800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe46415dd50 0x7fe46406e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:39.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.794+0000 7fe46b800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe4640b3d60 0x7fe4641618d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:39.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.794+0000 7fe46b800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe46406b0c0 0x7fe46406b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:39.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.794+0000 7fe46b800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4640b41f0 con 0x7fe46415dd50 2024-03-24T01:24:39.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.794+0000 7fe46b800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe46406ba70 con 0x7fe4640b3d60 2024-03-24T01:24:39.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.794+0000 7fe46b800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4640ad280 con 0x7fe46406b0c0 2024-03-24T01:24:39.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.795+0000 7fe468e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe46415dd50 0x7fe46406e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:39.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.795+0000 7fe468e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe46415dd50 0x7fe46406e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43550/0 (socket says 172.21.15.90:43550) 2024-03-24T01:24:39.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.795+0000 7fe468e00640 1 -- 172.21.15.90:0/168048093 learned_addr learned my addr 172.21.15.90:0/168048093 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:39.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.795+0000 7fe45be00640 1 --2- 172.21.15.90:0/168048093 >> v2:172.21.15.120:3300/0 conn(0x7fe4640b3d60 0x7fe4641618d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:39.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.795+0000 7fe469800640 1 --2- 172.21.15.90:0/168048093 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe46406b0c0 0x7fe46406b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:39.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.795+0000 7fe468e00640 1 -- 172.21.15.90:0/168048093 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe46406b0c0 msgr2=0x7fe46406b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:39.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.795+0000 7fe468e00640 1 --2- 172.21.15.90:0/168048093 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe46406b0c0 0x7fe46406b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:39.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.795+0000 7fe468e00640 1 -- 172.21.15.90:0/168048093 >> v2:172.21.15.120:3300/0 conn(0x7fe4640b3d60 msgr2=0x7fe4641618d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:39.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.795+0000 7fe468e00640 1 --2- 172.21.15.90:0/168048093 >> v2:172.21.15.120:3300/0 conn(0x7fe4640b3d60 0x7fe4641618d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:39.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.795+0000 7fe468e00640 1 -- 172.21.15.90:0/168048093 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4640acf50 con 0x7fe46415dd50 2024-03-24T01:24:39.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.796+0000 7fe468e00640 1 --2- 172.21.15.90:0/168048093 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe46415dd50 0x7fe46406e020 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe4640ae610 tx=0x7fe45402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=145a13d1fa2e873b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:39.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.796+0000 7fe463e00640 1 -- 172.21.15.90:0/168048093 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe454038070 con 0x7fe46415dd50 2024-03-24T01:24:39.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.796+0000 7fe463e00640 1 -- 172.21.15.90:0/168048093 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe45402f050 con 0x7fe46415dd50 2024-03-24T01:24:39.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.796+0000 7fe463e00640 1 -- 172.21.15.90:0/168048093 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe454031970 con 0x7fe46415dd50 2024-03-24T01:24:39.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.797+0000 7fe46b800640 1 -- 172.21.15.90:0/168048093 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe46415dd50 msgr2=0x7fe46406e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:39.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.797+0000 7fe46b800640 1 --2- 172.21.15.90:0/168048093 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe46415dd50 0x7fe46406e020 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe4640ae610 tx=0x7fe45402b7f0 comp rx=0 tx=0).stop 2024-03-24T01:24:39.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.797+0000 7fe46b800640 1 -- 172.21.15.90:0/168048093 shutdown_connections 2024-03-24T01:24:39.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.797+0000 7fe46b800640 1 --2- 172.21.15.90:0/168048093 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe46406b0c0 0x7fe46406b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:39.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.797+0000 7fe46b800640 1 --2- 172.21.15.90:0/168048093 >> v2:172.21.15.120:3300/0 conn(0x7fe4640b3d60 0x7fe4641618d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:39.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.797+0000 7fe46b800640 1 --2- 172.21.15.90:0/168048093 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe46415dd50 0x7fe46406e020 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:39.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.797+0000 7fe46b800640 1 -- 172.21.15.90:0/168048093 >> 172.21.15.90:0/168048093 conn(0x7fe4640b1770 msgr2=0x7fe4640b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:39.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.797+0000 7fe46b800640 1 -- 172.21.15.90:0/168048093 shutdown_connections 2024-03-24T01:24:39.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.797+0000 7fe46b800640 1 -- 172.21.15.90:0/168048093 wait complete. 2024-03-24T01:24:39.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.799+0000 7fe46b800640 1 Processor -- start 2024-03-24T01:24:39.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.799+0000 7fe46b800640 1 -- start start 2024-03-24T01:24:39.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.800+0000 7fe46b800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe46406b0c0 0x7fe46414e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:39.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.800+0000 7fe46b800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4640b3d60 0x7fe464150b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:39.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.800+0000 7fe46b800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe46415dd50 0x7fe464154310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:39.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.800+0000 7fe46b800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4641622e0 con 0x7fe46406b0c0 2024-03-24T01:24:39.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.800+0000 7fe46b800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe464162000 con 0x7fe46415dd50 2024-03-24T01:24:39.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.800+0000 7fe46b800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4641625c0 con 0x7fe4640b3d60 2024-03-24T01:24:39.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.801+0000 7fe468e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe46406b0c0 0x7fe46414e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:39.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.801+0000 7fe468e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe46406b0c0 0x7fe46414e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43552/0 (socket says 172.21.15.90:43552) 2024-03-24T01:24:39.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.801+0000 7fe468e00640 1 -- 172.21.15.90:0/3554252960 learned_addr learned my addr 172.21.15.90:0/3554252960 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:39.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.801+0000 7fe45be00640 1 --2- 172.21.15.90:0/3554252960 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4640b3d60 0x7fe464150b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:39.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.801+0000 7fe469800640 1 --2- 172.21.15.90:0/3554252960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe46415dd50 0x7fe464154310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:39.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.801+0000 7fe468e00640 1 -- 172.21.15.90:0/3554252960 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4640b3d60 msgr2=0x7fe464150b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:39.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.801+0000 7fe468e00640 1 --2- 172.21.15.90:0/3554252960 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4640b3d60 0x7fe464150b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:39.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.801+0000 7fe468e00640 1 -- 172.21.15.90:0/3554252960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe46415dd50 msgr2=0x7fe464154310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:39.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.801+0000 7fe468e00640 1 --2- 172.21.15.90:0/3554252960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe46415dd50 0x7fe464154310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:39.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.801+0000 7fe468e00640 1 -- 172.21.15.90:0/3554252960 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe464160f10 con 0x7fe46406b0c0 2024-03-24T01:24:39.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.801+0000 7fe468e00640 1 --2- 172.21.15.90:0/3554252960 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe46406b0c0 0x7fe46414e120 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe45402b830 tx=0x7fe4540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:39.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.802+0000 7fe461600640 1 -- 172.21.15.90:0/3554252960 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe454042070 con 0x7fe46406b0c0 2024-03-24T01:24:39.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.802+0000 7fe461600640 1 -- 172.21.15.90:0/3554252960 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe45403d890 con 0x7fe46406b0c0 2024-03-24T01:24:39.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.802+0000 7fe461600640 1 -- 172.21.15.90:0/3554252960 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe454038040 con 0x7fe46406b0c0 2024-03-24T01:24:39.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.802+0000 7fe46b800640 1 -- 172.21.15.90:0/3554252960 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe464150320 con 0x7fe46406b0c0 2024-03-24T01:24:39.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.802+0000 7fe46b800640 1 -- 172.21.15.90:0/3554252960 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4641611f0 con 0x7fe46406b0c0 2024-03-24T01:24:39.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.803+0000 7fe46b800640 1 -- 172.21.15.90:0/3554252960 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe464068060 con 0x7fe46406b0c0 2024-03-24T01:24:39.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.804+0000 7fe461600640 1 -- 172.21.15.90:0/3554252960 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe454045070 con 0x7fe46406b0c0 2024-03-24T01:24:39.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.805+0000 7fe461600640 1 --2- 172.21.15.90:0/3554252960 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe44006bc20 0x7fe44006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:39.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.805+0000 7fe461600640 1 -- 172.21.15.90:0/3554252960 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fe4540ac7c0 con 0x7fe46406b0c0 2024-03-24T01:24:39.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.805+0000 7fe45be00640 1 --2- 172.21.15.90:0/3554252960 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe44006bc20 0x7fe44006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:39.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.806+0000 7fe45be00640 1 --2- 172.21.15.90:0/3554252960 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe44006bc20 0x7fe44006e0e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe44c006fd0 tx=0x7fe44c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:39.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:39.810+0000 7fe461600640 1 -- 172.21.15.90:0/3554252960 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe45407b050 con 0x7fe46406b0c0 2024-03-24T01:24:40.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.010+0000 7fe46b800640 1 -- 172.21.15.90:0/3554252960 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4640cf4b0 con 0x7fe46406b0c0 2024-03-24T01:24:40.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.013+0000 7fe461600640 1 -- 172.21.15.90:0/3554252960 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18490 (secure 0 0 0) 0x7fe45407a950 con 0x7fe46406b0c0 2024-03-24T01:24:40.013 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:24:40.013 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:24:17.008224+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:24:08.643966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T01:24:09.189904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T01:24:13.027323+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:24:40.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.015+0000 7fe46b800640 1 -- 172.21.15.90:0/3554252960 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe44006bc20 msgr2=0x7fe44006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.015+0000 7fe46b800640 1 --2- 172.21.15.90:0/3554252960 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe44006bc20 0x7fe44006e0e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe44c006fd0 tx=0x7fe44c008040 comp rx=0 tx=0).stop 2024-03-24T01:24:40.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.015+0000 7fe46b800640 1 -- 172.21.15.90:0/3554252960 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe46406b0c0 msgr2=0x7fe46414e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.016+0000 7fe46b800640 1 --2- 172.21.15.90:0/3554252960 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe46406b0c0 0x7fe46414e120 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe45402b830 tx=0x7fe4540046c0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.016+0000 7fe46b800640 1 -- 172.21.15.90:0/3554252960 shutdown_connections 2024-03-24T01:24:40.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.016+0000 7fe46b800640 1 --2- 172.21.15.90:0/3554252960 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe44006bc20 0x7fe44006e0e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.016+0000 7fe46b800640 1 --2- 172.21.15.90:0/3554252960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe46415dd50 0x7fe464154310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.016+0000 7fe46b800640 1 --2- 172.21.15.90:0/3554252960 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe4640b3d60 0x7fe464150b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.016+0000 7fe46b800640 1 --2- 172.21.15.90:0/3554252960 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe46406b0c0 0x7fe46414e120 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.016+0000 7fe46b800640 1 -- 172.21.15.90:0/3554252960 >> 172.21.15.90:0/3554252960 conn(0x7fe4640b1770 msgr2=0x7fe46406f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:40.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.016+0000 7fe46b800640 1 -- 172.21.15.90:0/3554252960 shutdown_connections 2024-03-24T01:24:40.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.016+0000 7fe46b800640 1 -- 172.21.15.90:0/3554252960 wait complete. 2024-03-24T01:24:40.037 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-24T01:24:40.037 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:24:40.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.243+0000 7f4c43800640 1 Processor -- start 2024-03-24T01:24:40.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.243+0000 7f4c43800640 1 -- start start 2024-03-24T01:24:40.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.244+0000 7f4c43800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c3c1454e0 0x7f4c3c0cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.244+0000 7f4c43800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4c3c06f000 0x7f4c3c06f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.244+0000 7f4c43800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c3c0cc660 con 0x7f4c3c0cdc50 2024-03-24T01:24:40.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.244+0000 7f4c43800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c3c0cc430 con 0x7f4c3c06f000 2024-03-24T01:24:40.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.244+0000 7f4c43800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3c0cc920 con 0x7f4c3c1454e0 2024-03-24T01:24:40.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.244+0000 7f4c3be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4c3c06f000 0x7f4c3c06f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.244+0000 7f4c41800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4c3c0cdc50 0x7f4c3c06d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51142/0 (socket says 172.21.15.90:51142) 2024-03-24T01:24:40.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.244+0000 7f4c41800640 1 -- 172.21.15.90:0/1726177973 learned_addr learned my addr 172.21.15.90:0/1726177973 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:40.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.245+0000 7f4c40e00640 1 --2- 172.21.15.90:0/1726177973 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c3c1454e0 0x7f4c3c0cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.245+0000 7f4c3be00640 1 -- 172.21.15.90:0/1726177973 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c3c1454e0 msgr2=0x7f4c3c0cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:40.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.245+0000 7f4c3be00640 1 --2- 172.21.15.90:0/1726177973 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c3c1454e0 0x7f4c3c0cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.245+0000 7f4c3be00640 1 -- 172.21.15.90:0/1726177973 >> v1:172.21.15.90:6789/0 conn(0x7f4c3c0cdc50 legacy=0x7f4c3c06d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.245+0000 7f4c3be00640 1 -- 172.21.15.90:0/1726177973 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c3c0ccc00 con 0x7f4c3c06f000 2024-03-24T01:24:40.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.245+0000 7f4c3b400640 1 -- 172.21.15.90:0/1726177973 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3231891006 0 0) 0x7f4c3c0cc660 con 0x7f4c3c0cdc50 2024-03-24T01:24:40.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.246+0000 7f4c3be00640 1 --2- 172.21.15.90:0/1726177973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c3c06f000 0x7f4c3c06f440 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4c280097a0 tx=0x7f4c2802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=718606be66332b42 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:40.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.246+0000 7f4c3b400640 1 -- 172.21.15.90:0/1726177973 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c28035020 con 0x7f4c3c06f000 2024-03-24T01:24:40.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.246+0000 7f4c3b400640 1 -- 172.21.15.90:0/1726177973 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c28031830 con 0x7f4c3c06f000 2024-03-24T01:24:40.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.246+0000 7f4c3b400640 1 -- 172.21.15.90:0/1726177973 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c28031c00 con 0x7f4c3c06f000 2024-03-24T01:24:40.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.247+0000 7f4c43800640 1 -- 172.21.15.90:0/1726177973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c3c06f000 msgr2=0x7f4c3c06f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.247+0000 7f4c43800640 1 --2- 172.21.15.90:0/1726177973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c3c06f000 0x7f4c3c06f440 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4c280097a0 tx=0x7f4c2802b8d0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.247+0000 7f4c43800640 1 -- 172.21.15.90:0/1726177973 shutdown_connections 2024-03-24T01:24:40.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.247+0000 7f4c43800640 1 --2- 172.21.15.90:0/1726177973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c3c06f000 0x7f4c3c06f440 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.247+0000 7f4c43800640 1 --2- 172.21.15.90:0/1726177973 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c3c1454e0 0x7f4c3c0cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.247+0000 7f4c43800640 1 -- 172.21.15.90:0/1726177973 >> 172.21.15.90:0/1726177973 conn(0x7f4c3c06b2a0 msgr2=0x7f4c3c06b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:40.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.247+0000 7f4c43800640 1 -- 172.21.15.90:0/1726177973 shutdown_connections 2024-03-24T01:24:40.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.247+0000 7f4c43800640 1 -- 172.21.15.90:0/1726177973 wait complete. 2024-03-24T01:24:40.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.249+0000 7f4c43800640 1 Processor -- start 2024-03-24T01:24:40.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.249+0000 7f4c43800640 1 -- start start 2024-03-24T01:24:40.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.250+0000 7f4c43800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c3c06f000 0x7f4c3c138ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.250+0000 7f4c43800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c3c0cdc50 0x7f4c3c15a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.250+0000 7f4c43800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c3c1454e0 0x7f4c3c1415b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.250+0000 7f4c43800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3c0ce0e0 con 0x7f4c3c06f000 2024-03-24T01:24:40.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.250+0000 7f4c43800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3c0cd1c0 con 0x7f4c3c1454e0 2024-03-24T01:24:40.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.250+0000 7f4c43800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3c0ce260 con 0x7f4c3c0cdc50 2024-03-24T01:24:40.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.250+0000 7f4c40e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c3c06f000 0x7f4c3c138ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.250+0000 7f4c40e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c3c06f000 0x7f4c3c138ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59968/0 (socket says 172.21.15.90:59968) 2024-03-24T01:24:40.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.250+0000 7f4c40e00640 1 -- 172.21.15.90:0/4029022622 learned_addr learned my addr 172.21.15.90:0/4029022622 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:40.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.250+0000 7f4c3be00640 1 --2- 172.21.15.90:0/4029022622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c3c0cdc50 0x7f4c3c15a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.250+0000 7f4c41800640 1 --2- 172.21.15.90:0/4029022622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c3c1454e0 0x7f4c3c1415b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.250+0000 7f4c40e00640 1 -- 172.21.15.90:0/4029022622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c3c0cdc50 msgr2=0x7f4c3c15a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.250+0000 7f4c40e00640 1 --2- 172.21.15.90:0/4029022622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c3c0cdc50 0x7f4c3c15a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.250+0000 7f4c40e00640 1 -- 172.21.15.90:0/4029022622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c3c1454e0 msgr2=0x7f4c3c1415b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.250+0000 7f4c40e00640 1 --2- 172.21.15.90:0/4029022622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c3c1454e0 0x7f4c3c1415b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.250+0000 7f4c40e00640 1 -- 172.21.15.90:0/4029022622 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c3c1f1ba0 con 0x7f4c3c06f000 2024-03-24T01:24:40.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.250+0000 7f4c3be00640 1 --2- 172.21.15.90:0/4029022622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c3c0cdc50 0x7f4c3c15a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:40.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.250+0000 7f4c41800640 1 --2- 172.21.15.90:0/4029022622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c3c1454e0 0x7f4c3c1415b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:40.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.251+0000 7f4c40e00640 1 --2- 172.21.15.90:0/4029022622 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c3c06f000 0x7f4c3c138ba0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4c3000d5f0 tx=0x7f4c3000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:40.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.251+0000 7f4c38c00640 1 -- 172.21.15.90:0/4029022622 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c300125f0 con 0x7f4c3c06f000 2024-03-24T01:24:40.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.251+0000 7f4c38c00640 1 -- 172.21.15.90:0/4029022622 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c30019070 con 0x7f4c3c06f000 2024-03-24T01:24:40.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.251+0000 7f4c38c00640 1 -- 172.21.15.90:0/4029022622 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c300129d0 con 0x7f4c3c06f000 2024-03-24T01:24:40.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.251+0000 7f4c43800640 1 -- 172.21.15.90:0/4029022622 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c3c1f1e60 con 0x7f4c3c06f000 2024-03-24T01:24:40.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.251+0000 7f4c43800640 1 -- 172.21.15.90:0/4029022622 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c3c1f2420 con 0x7f4c3c06f000 2024-03-24T01:24:40.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.252+0000 7f4c43800640 1 -- 172.21.15.90:0/4029022622 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c3c073570 con 0x7f4c3c06f000 2024-03-24T01:24:40.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.253+0000 7f4c38c00640 1 -- 172.21.15.90:0/4029022622 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4c30012b70 con 0x7f4c3c06f000 2024-03-24T01:24:40.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.254+0000 7f4c38c00640 1 --2- 172.21.15.90:0/4029022622 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4c1c06bbd0 0x7f4c1c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.254+0000 7f4c38c00640 1 -- 172.21.15.90:0/4029022622 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f4c3008b1c0 con 0x7f4c3c06f000 2024-03-24T01:24:40.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.255+0000 7f4c3be00640 1 --2- 172.21.15.90:0/4029022622 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4c1c06bbd0 0x7f4c1c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.256+0000 7f4c3be00640 1 --2- 172.21.15.90:0/4029022622 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4c1c06bbd0 0x7f4c1c06e090 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4c28036000 tx=0x7f4c28008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:40.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.259+0000 7f4c38c00640 1 -- 172.21.15.90:0/4029022622 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4c30059960 con 0x7f4c3c06f000 2024-03-24T01:24:40.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.458+0000 7f4c43800640 1 -- 172.21.15.90:0/4029022622 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c3c04f4e0 con 0x7f4c3c06f000 2024-03-24T01:24:40.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.460+0000 7f4c38c00640 1 -- 172.21.15.90:0/4029022622 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18490 (secure 0 0 0) 0x7f4c30059260 con 0x7f4c3c06f000 2024-03-24T01:24:40.461 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:24:40.461 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:24:17.008224+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:24:08.643966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T01:24:09.189904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T01:24:13.027323+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:24:40.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.463+0000 7f4c43800640 1 -- 172.21.15.90:0/4029022622 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4c1c06bbd0 msgr2=0x7f4c1c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.463+0000 7f4c43800640 1 --2- 172.21.15.90:0/4029022622 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4c1c06bbd0 0x7f4c1c06e090 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4c28036000 tx=0x7f4c28008030 comp rx=0 tx=0).stop 2024-03-24T01:24:40.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.463+0000 7f4c43800640 1 -- 172.21.15.90:0/4029022622 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c3c06f000 msgr2=0x7f4c3c138ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.463+0000 7f4c43800640 1 --2- 172.21.15.90:0/4029022622 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c3c06f000 0x7f4c3c138ba0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4c3000d5f0 tx=0x7f4c3000dab0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.463+0000 7f4c43800640 1 -- 172.21.15.90:0/4029022622 shutdown_connections 2024-03-24T01:24:40.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.463+0000 7f4c43800640 1 --2- 172.21.15.90:0/4029022622 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4c1c06bbd0 0x7f4c1c06e090 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.463+0000 7f4c43800640 1 --2- 172.21.15.90:0/4029022622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c3c1454e0 0x7f4c3c1415b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.463+0000 7f4c43800640 1 --2- 172.21.15.90:0/4029022622 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c3c0cdc50 0x7f4c3c15a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.463+0000 7f4c43800640 1 --2- 172.21.15.90:0/4029022622 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c3c06f000 0x7f4c3c138ba0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.463+0000 7f4c43800640 1 -- 172.21.15.90:0/4029022622 >> 172.21.15.90:0/4029022622 conn(0x7f4c3c06b2a0 msgr2=0x7f4c3c15c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:40.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.463+0000 7f4c43800640 1 -- 172.21.15.90:0/4029022622 shutdown_connections 2024-03-24T01:24:40.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.463+0000 7f4c43800640 1 -- 172.21.15.90:0/4029022622 wait complete. 2024-03-24T01:24:40.482 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-24T01:24:40.482 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-24T01:24:40.483 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-24T01:24:40.483 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-24T01:24:40.484 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-24T01:24:40.484 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-24T01:24:40.484 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-24T01:24:40.485 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-24T01:24:40.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.783+0000 7f069ea00640 1 Processor -- start 2024-03-24T01:24:40.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.783+0000 7f069ea00640 1 -- start start 2024-03-24T01:24:40.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.785+0000 7f069ea00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06980cbbe0 0x7f069815e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.785+0000 7f069ea00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f06980cbfc0 0x7f0698071990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.785+0000 7f069ea00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069815a7e0 con 0x7f069815a320 2024-03-24T01:24:40.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.785+0000 7f069ea00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f069815aac0 con 0x7f06980cbfc0 2024-03-24T01:24:40.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.785+0000 7f069ea00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f069815ada0 con 0x7f06980cbbe0 2024-03-24T01:24:40.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.787+0000 7f0697e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06980cbbe0 0x7f069815e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.787+0000 7f0697400640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f069815a320 0x7f06980b2890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.90:48694/0 (socket says 172.21.15.90:48694) 2024-03-24T01:24:40.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.787+0000 7f0697e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06980cbbe0 0x7f069815e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.90:50014/0 (socket says 172.21.15.90:50014) 2024-03-24T01:24:40.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.787+0000 7f0697e00640 1 -- 172.21.15.90:0/1585396817 learned_addr learned my addr 172.21.15.90:0/1585396817 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:40.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.787+0000 7f069ca00640 1 --2- 172.21.15.90:0/1585396817 >> v2:172.21.15.120:3300/0 conn(0x7f06980cbfc0 0x7f0698071990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.787+0000 7f260ac00640 1 Processor -- start 2024-03-24T01:24:40.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.788+0000 7f0697e00640 1 -- 172.21.15.90:0/1585396817 >> v2:172.21.15.120:3300/0 conn(0x7f06980cbfc0 msgr2=0x7f0698071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:40.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.787+0000 7f260ac00640 1 -- start start 2024-03-24T01:24:40.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.789+0000 7f260ac00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25fc0e30c0 0x7f25fc057d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.788+0000 7f0697e00640 1 --2- 172.21.15.90:0/1585396817 >> v2:172.21.15.120:3300/0 conn(0x7f06980cbfc0 0x7f0698071990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.789+0000 7f260ac00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f25fc054300 0x7f25fc0068d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.789+0000 7f260ac00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25fc0582d0 con 0x7f25fc03e840 2024-03-24T01:24:40.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.789+0000 7f260ac00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25fc058570 con 0x7f25fc054300 2024-03-24T01:24:40.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.789+0000 7f260ac00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25fc058850 con 0x7f25fc0e30c0 2024-03-24T01:24:40.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.788+0000 7f0697e00640 1 -- 172.21.15.90:0/1585396817 >> v1:172.21.15.120:6789/0 conn(0x7f069815a320 legacy=0x7f06980b2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.788+0000 7f0697e00640 1 -- 172.21.15.90:0/1585396817 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f069815b080 con 0x7f06980cbbe0 2024-03-24T01:24:40.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.790+0000 7f7106e00640 1 Processor -- start 2024-03-24T01:24:40.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.790+0000 7f260a200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f25fc054300 0x7f25fc0068d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.790+0000 7f260a200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f25fc054300 0x7f25fc0068d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59970/0 (socket says 172.21.15.90:59970) 2024-03-24T01:24:40.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.790+0000 7f260a200640 1 -- 172.21.15.90:0/4270135878 learned_addr learned my addr 172.21.15.90:0/4270135878 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:40.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.790+0000 7f0697e00640 1 --2- 172.21.15.90:0/1585396817 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06980cbbe0 0x7f069815e320 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0688008870 tx=0x7f068802dfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=769e89c5ea55e727 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:40.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.790+0000 7f7106e00640 1 -- start start 2024-03-24T01:24:40.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.790+0000 7f2609800640 1 --2- 172.21.15.90:0/4270135878 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25fc0e30c0 0x7f25fc057d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.791+0000 7f260a200640 1 -- 172.21.15.90:0/4270135878 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25fc0e30c0 msgr2=0x7f25fc057d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:40.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.791+0000 7f0696a00640 1 -- 172.21.15.90:0/1585396817 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0688034070 con 0x7f06980cbbe0 2024-03-24T01:24:40.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.791+0000 7f0696a00640 1 -- 172.21.15.90:0/1585396817 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f068802ed80 con 0x7f06980cbbe0 2024-03-24T01:24:40.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.791+0000 7f0696a00640 1 -- 172.21.15.90:0/1585396817 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0688034070 con 0x7f06980cbbe0 2024-03-24T01:24:40.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.791+0000 7f260a200640 1 --2- 172.21.15.90:0/4270135878 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25fc0e30c0 0x7f25fc057d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.791+0000 7f260a200640 1 -- 172.21.15.90:0/4270135878 >> v1:172.21.15.90:6789/0 conn(0x7f25fc03e840 legacy=0x7f25fc053bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.791+0000 7f2603e00640 1 -- 172.21.15.90:0/4270135878 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3747660773 0 0) 0x7f25fc0582d0 con 0x7f25fc03e840 2024-03-24T01:24:40.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.792+0000 7f260a200640 1 -- 172.21.15.90:0/4270135878 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25fc058b30 con 0x7f25fc054300 2024-03-24T01:24:40.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.792+0000 7f069ea00640 1 -- 172.21.15.90:0/1585396817 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06980cbbe0 msgr2=0x7f069815e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.792+0000 7f069ea00640 1 --2- 172.21.15.90:0/1585396817 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06980cbbe0 0x7f069815e320 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0688008870 tx=0x7f068802dfd0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.792+0000 7f069ea00640 1 -- 172.21.15.90:0/1585396817 shutdown_connections 2024-03-24T01:24:40.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.792+0000 7f069ea00640 1 --2- 172.21.15.90:0/1585396817 >> v2:172.21.15.120:3300/0 conn(0x7f06980cbfc0 0x7f0698071990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.792+0000 7f069ea00640 1 --2- 172.21.15.90:0/1585396817 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f06980cbbe0 0x7f069815e320 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.792+0000 7f069ea00640 1 -- 172.21.15.90:0/1585396817 >> 172.21.15.90:0/1585396817 conn(0x7f06980b1c80 msgr2=0x7f06980cf7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:40.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.792+0000 7f7106e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f710006b7d0 0x7f71001629e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.792+0000 7f7106e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7100162f20 0x7f710015f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.792+0000 7f7106e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7100165ed0 con 0x7f71000af040 2024-03-24T01:24:40.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.792+0000 7f7106e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71001661b0 con 0x7f710006b7d0 2024-03-24T01:24:40.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.792+0000 7f7106e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7100166490 con 0x7f7100162f20 2024-03-24T01:24:40.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.793+0000 7f069ea00640 1 -- 172.21.15.90:0/1585396817 shutdown_connections 2024-03-24T01:24:40.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.793+0000 7f069ea00640 1 -- 172.21.15.90:0/1585396817 wait complete. 2024-03-24T01:24:40.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.793+0000 7f70ffe00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f71000af040 0x7f710006b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51160/0 (socket says 172.21.15.90:51160) 2024-03-24T01:24:40.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.793+0000 7f70ffe00640 1 -- 172.21.15.90:0/4275391063 learned_addr learned my addr 172.21.15.90:0/4275391063 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:40.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.793+0000 7f70ff400640 1 --2- 172.21.15.90:0/4275391063 >> v2:172.21.15.90:3300/0 conn(0x7f710006b7d0 0x7f71001629e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.793+0000 7f2609800640 1 --2- 172.21.15.90:0/4270135878 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25fc0e30c0 0x7f25fc057d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:40.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.794+0000 7f7104e00640 1 --2- 172.21.15.90:0/4275391063 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7100162f20 0x7f710015f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.794+0000 7f260a200640 1 --2- 172.21.15.90:0/4270135878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25fc054300 0x7f25fc0068d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f260406a9a0 tx=0x7f26040a7200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6eee859c1f852ea server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:40.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.795+0000 7f7104e00640 1 -- 172.21.15.90:0/4275391063 >> v2:172.21.15.90:3300/0 conn(0x7f710006b7d0 msgr2=0x7f71001629e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:40.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.795+0000 7f2603e00640 1 -- 172.21.15.90:0/4270135878 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26040b4070 con 0x7f25fc054300 2024-03-24T01:24:40.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.795+0000 7f7104e00640 1 --2- 172.21.15.90:0/4275391063 >> v2:172.21.15.90:3300/0 conn(0x7f710006b7d0 0x7f71001629e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.795+0000 7f7104e00640 1 -- 172.21.15.90:0/4275391063 >> v1:172.21.15.90:6789/0 conn(0x7f71000af040 legacy=0x7f710006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.795+0000 7fda4da00640 1 Processor -- start 2024-03-24T01:24:40.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.795+0000 7f2603e00640 1 -- 172.21.15.90:0/4270135878 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26040af640 con 0x7f25fc054300 2024-03-24T01:24:40.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.795+0000 7f2603e00640 1 -- 172.21.15.90:0/4270135878 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26040afa70 con 0x7f25fc054300 2024-03-24T01:24:40.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.795+0000 7fda4da00640 1 -- start start 2024-03-24T01:24:40.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.796+0000 7f7104e00640 1 -- 172.21.15.90:0/4275391063 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7100166770 con 0x7f7100162f20 2024-03-24T01:24:40.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.796+0000 7f069ea00640 1 Processor -- start 2024-03-24T01:24:40.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.796+0000 7f260ac00640 1 -- 172.21.15.90:0/4270135878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25fc054300 msgr2=0x7f25fc0068d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.796+0000 7f260ac00640 1 --2- 172.21.15.90:0/4270135878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25fc054300 0x7f25fc0068d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f260406a9a0 tx=0x7f26040a7200 comp rx=0 tx=0).stop 2024-03-24T01:24:40.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.797+0000 7f70ff400640 1 --2- 172.21.15.90:0/4275391063 >> v2:172.21.15.90:3300/0 conn(0x7f710006b7d0 0x7f71001629e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:24:40.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.796+0000 7f069ea00640 1 -- start start 2024-03-24T01:24:40.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.797+0000 7f260ac00640 1 -- 172.21.15.90:0/4270135878 shutdown_connections 2024-03-24T01:24:40.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.797+0000 7f7104e00640 1 --2- 172.21.15.90:0/4275391063 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7100162f20 0x7f710015f2a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f70ec0097a0 tx=0x7f70ec02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa102be1ef2410f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:40.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.797+0000 7f069ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f06980cbbe0 0x7f06981543f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.797+0000 7f069ea00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06980cbfc0 0x7f0698155520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.797+0000 7f260ac00640 1 --2- 172.21.15.90:0/4270135878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25fc054300 0x7f25fc0068d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.798+0000 7f260ac00640 1 --2- 172.21.15.90:0/4270135878 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25fc0e30c0 0x7f25fc057d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.798+0000 7f70fea00640 1 -- 172.21.15.90:0/4275391063 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ec035020 con 0x7f7100162f20 2024-03-24T01:24:40.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.798+0000 7f70fea00640 1 -- 172.21.15.90:0/4275391063 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70ec031830 con 0x7f7100162f20 2024-03-24T01:24:40.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.798+0000 7f70fea00640 1 -- 172.21.15.90:0/4275391063 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ec031c40 con 0x7f7100162f20 2024-03-24T01:24:40.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.798+0000 7f069ea00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f069815a320 0x7f0698152780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.798+0000 7f069ea00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06980b22b0 con 0x7f06980cbbe0 2024-03-24T01:24:40.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.798+0000 7f069ea00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06980b2130 con 0x7f06980cbfc0 2024-03-24T01:24:40.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.798+0000 7f069ea00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0698071ed0 con 0x7f069815a320 2024-03-24T01:24:40.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.798+0000 7fda4da00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fda480d10c0 0x7fda480ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.798+0000 7f260ac00640 1 -- 172.21.15.90:0/4270135878 >> 172.21.15.90:0/4270135878 conn(0x7f25fc0559e0 msgr2=0x7f25fc055df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:40.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.798+0000 7fda4da00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda480ce960 0x7fda480d28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.798+0000 7f0697e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f06980cbbe0 0x7f06981543f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.798+0000 7f0697e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f06980cbbe0 0x7f06981543f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59990/0 (socket says 172.21.15.90:59990) 2024-03-24T01:24:40.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.798+0000 7f0697e00640 1 -- 172.21.15.90:0/2968220333 learned_addr learned my addr 172.21.15.90:0/2968220333 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:40.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.798+0000 7f0697400640 1 --2- 172.21.15.90:0/2968220333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06980cbfc0 0x7f0698155520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.799+0000 7f069ca00640 1 --2- 172.21.15.90:0/2968220333 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f069815a320 0x7f0698152780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.798+0000 7fda4da00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda48161dd0 con 0x7fda4815ded0 2024-03-24T01:24:40.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.799+0000 7fda3fe00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fda480d10c0 0x7fda480ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.799+0000 7fda4da00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda480d2410 con 0x7fda480d10c0 2024-03-24T01:24:40.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.799+0000 7fda3fe00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fda480d10c0 0x7fda480ce420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.90:39012/0 (socket says 172.21.15.90:39012) 2024-03-24T01:24:40.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.799+0000 7fda3fe00640 1 -- 172.21.15.90:0/2948711698 learned_addr learned my addr 172.21.15.90:0/2948711698 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:40.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.799+0000 7fda47400640 1 --2- 172.21.15.90:0/2948711698 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda480ce960 0x7fda480d28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.799+0000 7f260ac00640 1 -- 172.21.15.90:0/4270135878 shutdown_connections 2024-03-24T01:24:40.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.799+0000 7f260ac00640 1 -- 172.21.15.90:0/4270135878 wait complete. 2024-03-24T01:24:40.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.799+0000 7fda4da00640 1 -- 172.21.15.90:0/2948711698 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda480cf000 con 0x7fda480ce960 2024-03-24T01:24:40.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.800+0000 7f7106e00640 1 -- 172.21.15.90:0/4275391063 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7100162f20 msgr2=0x7f710015f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.800+0000 7f7106e00640 1 --2- 172.21.15.90:0/4275391063 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7100162f20 0x7f710015f2a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f70ec0097a0 tx=0x7f70ec02b8d0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.800+0000 7f7106e00640 1 -- 172.21.15.90:0/4275391063 shutdown_connections 2024-03-24T01:24:40.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.800+0000 7f7106e00640 1 --2- 172.21.15.90:0/4275391063 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7100162f20 0x7f710015f2a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.800+0000 7f7106e00640 1 --2- 172.21.15.90:0/4275391063 >> v2:172.21.15.90:3300/0 conn(0x7f710006b7d0 0x7f71001629e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.800+0000 7f7106e00640 1 -- 172.21.15.90:0/4275391063 >> 172.21.15.90:0/4275391063 conn(0x7f71000b1770 msgr2=0x7f71000b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:40.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.800+0000 7fda46000640 1 -- 172.21.15.90:0/2948711698 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 52578312 0 0) 0x7fda38001110 con 0x7fda4815ded0 2024-03-24T01:24:40.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.800+0000 7fda46000640 1 -- 172.21.15.90:0/2948711698 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda2c003520 con 0x7fda4815ded0 2024-03-24T01:24:40.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.800+0000 7f0697400640 1 -- 172.21.15.90:0/2968220333 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f069815a320 msgr2=0x7f0698152780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.800+0000 7f0697400640 1 --2- 172.21.15.90:0/2968220333 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f069815a320 0x7f0698152780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.800+0000 7f0697400640 1 -- 172.21.15.90:0/2968220333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f06980cbbe0 msgr2=0x7f06981543f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.800+0000 7f0697400640 1 --2- 172.21.15.90:0/2968220333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f06980cbbe0 0x7f06981543f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.800+0000 7f0697400640 1 -- 172.21.15.90:0/2968220333 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0698155d50 con 0x7f06980cbfc0 2024-03-24T01:24:40.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.800+0000 7f0697e00640 1 --2- 172.21.15.90:0/2968220333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f06980cbbe0 0x7f06981543f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T01:24:40.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.801+0000 7f7106e00640 1 -- 172.21.15.90:0/4275391063 shutdown_connections 2024-03-24T01:24:40.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.801+0000 7f7106e00640 1 -- 172.21.15.90:0/4275391063 wait complete. 2024-03-24T01:24:40.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.801+0000 7face5200640 1 Processor -- start 2024-03-24T01:24:40.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.801+0000 7face5200640 1 -- start start 2024-03-24T01:24:40.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.801+0000 7f0697400640 1 --2- 172.21.15.90:0/2968220333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06980cbfc0 0x7f0698155520 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f068400c3c0 tx=0x7f068400c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:40.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.802+0000 7f3ff3800640 1 Processor -- start 2024-03-24T01:24:40.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.801+0000 7f068fe00640 1 -- 172.21.15.90:0/2968220333 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06840074e0 con 0x7f06980cbfc0 2024-03-24T01:24:40.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.802+0000 7f069ea00640 1 -- 172.21.15.90:0/2968220333 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0698156030 con 0x7f06980cbfc0 2024-03-24T01:24:40.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.802+0000 7f069ea00640 1 -- 172.21.15.90:0/2968220333 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06981565f0 con 0x7f06980cbfc0 2024-03-24T01:24:40.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.802+0000 7fbf3e200640 1 Processor -- start 2024-03-24T01:24:40.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.802+0000 7f3ff3800640 1 -- start start 2024-03-24T01:24:40.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.802+0000 7f068fe00640 1 -- 172.21.15.90:0/2968220333 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0684007680 con 0x7f06980cbfc0 2024-03-24T01:24:40.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.802+0000 7f260ac00640 1 Processor -- start 2024-03-24T01:24:40.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.803+0000 7f260ac00640 1 -- start start 2024-03-24T01:24:40.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.803+0000 7f069ea00640 1 -- 172.21.15.90:0/2968220333 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0670001030 con 0x7f06980cbfc0 2024-03-24T01:24:40.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.804+0000 7f068fe00640 1 -- 172.21.15.90:0/2968220333 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06840074e0 con 0x7f06980cbfc0 2024-03-24T01:24:40.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.803+0000 7f7106e00640 1 Processor -- start 2024-03-24T01:24:40.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.803+0000 7f7106e00640 1 -- start start 2024-03-24T01:24:40.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.802+0000 7fda47400640 1 -- 172.21.15.90:0/2948711698 >> v2:172.21.15.120:3300/0 conn(0x7fda480d10c0 msgr2=0x7fda480ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:40.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.802+0000 7fda47400640 1 --2- 172.21.15.90:0/2948711698 >> v2:172.21.15.120:3300/0 conn(0x7fda480d10c0 0x7fda480ce420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.802+0000 7fda47400640 1 -- 172.21.15.90:0/2948711698 >> v1:172.21.15.120:6789/0 conn(0x7fda4815ded0 legacy=0x7fda480d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.802+0000 7fda47400640 1 -- 172.21.15.90:0/2948711698 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda480cf2e0 con 0x7fda480ce960 2024-03-24T01:24:40.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.803+0000 7f2bf8000640 1 Processor -- start 2024-03-24T01:24:40.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.802+0000 7fbf3e200640 1 -- start start 2024-03-24T01:24:40.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.804+0000 7f2bf8000640 1 -- start start 2024-03-24T01:24:40.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.804+0000 7f3ff3800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3fec15e1d0 0x7f3fec0b5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.804+0000 7f3ff3800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3fec0b5ba0 0x7f3fec0d2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.804+0000 7f3ff3800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fec0b6d10 con 0x7f3fec15ddf0 2024-03-24T01:24:40.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.804+0000 7f3ff3800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fec0b5ff0 con 0x7f3fec0b5ba0 2024-03-24T01:24:40.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.804+0000 7f3ff3800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fec0d1cf0 con 0x7f3fec15e1d0 2024-03-24T01:24:40.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.804+0000 7f3ff0e00640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f3fec15ddf0 0x7f3fec0d0380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.90:48710/0 (socket says 172.21.15.90:48710) 2024-03-24T01:24:40.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.804+0000 7f3ff0e00640 1 -- 172.21.15.90:0/2066522336 learned_addr learned my addr 172.21.15.90:0/2066522336 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:40.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.804+0000 7f3ff1800640 1 --2- 172.21.15.90:0/2066522336 >> v2:172.21.15.120:3300/0 conn(0x7f3fec0b5ba0 0x7f3fec0d2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.805+0000 7f3febe00640 1 --2- 172.21.15.90:0/2066522336 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3fec15e1d0 0x7f3fec0b5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.805+0000 7face5200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7face015ded0 0x7face00d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.805+0000 7face5200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7face00d0ef0 0x7face00ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.805+0000 7fda3fe00640 1 --2- 172.21.15.90:0/2948711698 >> v2:172.21.15.120:3300/0 conn(0x7fda480d10c0 0x7fda480ce420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T01:24:40.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.802+0000 7fda46000640 1 -- 172.21.15.90:0/2948711698 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1675827217 0 0) 0x7fda2c003520 con 0x7fda4815ded0 2024-03-24T01:24:40.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.804+0000 7f260ac00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25fc03e840 0x7f25fc04f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.804+0000 7f260ac00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25fc054300 0x7f25fc0d2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.804+0000 7f260ac00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25fc0e30c0 0x7f25fc04c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.805+0000 7f260ac00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25fc0e34a0 con 0x7f25fc0e30c0 2024-03-24T01:24:40.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.805+0000 7f260ac00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25fc03bfc0 con 0x7f25fc054300 2024-03-24T01:24:40.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.805+0000 7f260ac00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25fc006e10 con 0x7f25fc03e840 2024-03-24T01:24:40.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.805+0000 7face5200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face0161dd0 con 0x7face00ce960 2024-03-24T01:24:40.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.805+0000 7face5200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7face00d1350 con 0x7face00d0ef0 2024-03-24T01:24:40.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.805+0000 7face5200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7face00d2410 con 0x7face015ded0 2024-03-24T01:24:40.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.805+0000 7fda47400640 1 --2- 172.21.15.90:0/2948711698 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda480ce960 0x7fda480d28a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fda40009d90 tx=0x7fda4002bd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29fa74b24c1d7fcf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:40.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.805+0000 7fbf3e200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbf380b2ac0 0x7fbf380b4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.805+0000 7f3febe00640 1 -- 172.21.15.90:0/2066522336 >> v2:172.21.15.120:3300/0 conn(0x7f3fec0b5ba0 msgr2=0x7f3fec0d2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:40.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.805+0000 7f3febe00640 1 --2- 172.21.15.90:0/2066522336 >> v2:172.21.15.120:3300/0 conn(0x7f3fec0b5ba0 0x7f3fec0d2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.806+0000 7f3febe00640 1 -- 172.21.15.90:0/2066522336 >> v1:172.21.15.120:6789/0 conn(0x7f3fec15ddf0 legacy=0x7f3fec0d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.806+0000 7f3feb400640 1 -- 172.21.15.90:0/2066522336 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3411799478 0 0) 0x7f3fe0001110 con 0x7f3fec15ddf0 2024-03-24T01:24:40.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.806+0000 7fbf3e200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf381625b0 con 0x7fbf3806ed40 2024-03-24T01:24:40.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.806+0000 7fbf3e200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf380ceb30 con 0x7fbf380b4d50 2024-03-24T01:24:40.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.805+0000 7f260a200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25fc0e30c0 0x7f25fc04c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.806+0000 7fbf3e200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf380cee10 con 0x7fbf380b2ac0 2024-03-24T01:24:40.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.806+0000 7f3febe00640 1 -- 172.21.15.90:0/2066522336 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fec0d0e70 con 0x7f3fec15e1d0 2024-03-24T01:24:40.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.806+0000 7f260a200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25fc0e30c0 0x7f25fc04c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59992/0 (socket says 172.21.15.90:59992) 2024-03-24T01:24:40.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.806+0000 7f260a200640 1 -- 172.21.15.90:0/1290857908 learned_addr learned my addr 172.21.15.90:0/1290857908 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:40.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.805+0000 7f2609800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25fc03e840 0x7f25fc04f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.805+0000 7f2608e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25fc054300 0x7f25fc0d2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.806+0000 7f7106e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f710006b7d0 0x7f71002000c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.806+0000 7f7106e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71000af040 0x7f7100164780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.806+0000 7f7106e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7100162f20 0x7f7100164cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.806+0000 7fbf3cc00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fbf3806ed40 0x7fbf3816bce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51176/0 (socket says 172.21.15.90:51176) 2024-03-24T01:24:40.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.806+0000 7fbf3cc00640 1 -- 172.21.15.90:0/2373654596 learned_addr learned my addr 172.21.15.90:0/2373654596 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:40.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.807+0000 7f7106e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f710015f960 con 0x7f71000af040 2024-03-24T01:24:40.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.807+0000 7f7106e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f710015f7e0 con 0x7f7100162f20 2024-03-24T01:24:40.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.807+0000 7f7106e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f710015fae0 con 0x7f710006b7d0 2024-03-24T01:24:40.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.806+0000 7facd7e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7face00d0ef0 0x7face00ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.806+0000 7facdde00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7face015ded0 0x7face00d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.807+0000 7facdde00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7face015ded0 0x7face00d09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.90:50072/0 (socket says 172.21.15.90:50072) 2024-03-24T01:24:40.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.807+0000 7facdde00640 1 -- 172.21.15.90:0/362221032 learned_addr learned my addr 172.21.15.90:0/362221032 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:40.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.807+0000 7f260a200640 1 -- 172.21.15.90:0/1290857908 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25fc03e840 msgr2=0x7f25fc04f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.807+0000 7f260a200640 1 --2- 172.21.15.90:0/1290857908 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f25fc03e840 0x7f25fc04f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.807+0000 7f260a200640 1 -- 172.21.15.90:0/1290857908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25fc054300 msgr2=0x7f25fc0d2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.807+0000 7f260a200640 1 --2- 172.21.15.90:0/1290857908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25fc054300 0x7f25fc0d2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.807+0000 7f260a200640 1 -- 172.21.15.90:0/1290857908 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25fc04e770 con 0x7f25fc0e30c0 2024-03-24T01:24:40.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.807+0000 7f2bf8000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2bf00b2130 0x7f2bf015e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.807+0000 7f70ffe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f710006b7d0 0x7f71002000c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.807+0000 7f70ffe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f710006b7d0 0x7f71002000c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.90:50080/0 (socket says 172.21.15.90:50080) 2024-03-24T01:24:40.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.807+0000 7f70ffe00640 1 -- 172.21.15.90:0/3485447629 learned_addr learned my addr 172.21.15.90:0/3485447629 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:40.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.807+0000 7fda46000640 1 -- 172.21.15.90:0/2948711698 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda40039070 con 0x7fda480ce960 2024-03-24T01:24:40.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.807+0000 7fbf36000640 1 -- 172.21.15.90:0/2373654596 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2176546866 0 0) 0x7fbf381625b0 con 0x7fbf3806ed40 2024-03-24T01:24:40.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.807+0000 7fbf36000640 1 -- 172.21.15.90:0/2373654596 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf20003520 con 0x7fbf3806ed40 2024-03-24T01:24:40.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.807+0000 7f70ff400640 1 --2- 172.21.15.90:0/3485447629 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71000af040 0x7f7100164780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.807+0000 7facd7e00640 1 -- 172.21.15.90:0/362221032 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7face015ded0 msgr2=0x7face00d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:40.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.807+0000 7facd7e00640 1 --2- 172.21.15.90:0/362221032 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7face015ded0 0x7face00d09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.807+0000 7facd7e00640 1 -- 172.21.15.90:0/362221032 >> v1:172.21.15.120:6789/0 conn(0x7face00ce960 legacy=0x7face00b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:40.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.807+0000 7facd7e00640 1 -- 172.21.15.90:0/362221032 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face00d38b0 con 0x7face00d0ef0 2024-03-24T01:24:40.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.807+0000 7fda46000640 1 -- 172.21.15.90:0/2948711698 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda4002c8c0 con 0x7fda480ce960 2024-03-24T01:24:40.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.808+0000 7fda46000640 1 -- 172.21.15.90:0/2948711698 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda38001840 con 0x7fda480ce960 2024-03-24T01:24:40.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.808+0000 7f260a200640 1 --2- 172.21.15.90:0/1290857908 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25fc0e30c0 0x7f25fc04c8a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2604000b60 tx=0x7f2604074830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:40.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.808+0000 7f70ff400640 1 -- 172.21.15.90:0/3485447629 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f710006b7d0 msgr2=0x7f71002000c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.808+0000 7f70ff400640 1 --2- 172.21.15.90:0/3485447629 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f710006b7d0 0x7f71002000c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.808+0000 7f70ff400640 1 -- 172.21.15.90:0/3485447629 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7100162f20 msgr2=0x7f7100164cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.808+0000 7f70ff400640 1 --2- 172.21.15.90:0/3485447629 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7100162f20 0x7f7100164cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.808+0000 7f70ff400640 1 -- 172.21.15.90:0/3485447629 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7100200a00 con 0x7f71000af040 2024-03-24T01:24:40.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.808+0000 7f3febe00640 1 --2- 172.21.15.90:0/2066522336 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3fec15e1d0 0x7f3fec0b5660 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3fd80097a0 tx=0x7f3fd802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c99809d112fc28b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:40.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.807+0000 7f2bf6000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f2bf0071990 0x7f2bf006f090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51182/0 (socket says 172.21.15.90:51182) 2024-03-24T01:24:40.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.808+0000 7f2bf8000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bf015a7e0 con 0x7f2bf0071990 2024-03-24T01:24:40.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.808+0000 7f2bf8000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bf015aac0 con 0x7f2bf015a320 2024-03-24T01:24:40.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.808+0000 7f2bf8000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bf015ada0 con 0x7f2bf00b2130 2024-03-24T01:24:40.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.808+0000 7f2bf6000640 1 -- 172.21.15.90:0/3580436805 learned_addr learned my addr 172.21.15.90:0/3580436805 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:40.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.809+0000 7f2601600640 1 -- 172.21.15.90:0/1290857908 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26040be020 con 0x7f25fc0e30c0 2024-03-24T01:24:40.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.808+0000 7f70ff400640 1 --2- 172.21.15.90:0/3485447629 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71000af040 0x7f7100164780 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f70f000e7e0 tx=0x7f70f000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:40.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.808+0000 7fbf37400640 1 --2- 172.21.15.90:0/2373654596 >> v2:172.21.15.120:3300/0 conn(0x7fbf380b2ac0 0x7fbf380b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.809+0000 7fbf36000640 1 -- 172.21.15.90:0/2373654596 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2524476738 0 0) 0x7fbf20003520 con 0x7fbf3806ed40 2024-03-24T01:24:40.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.809+0000 7fbf36000640 1 -- 172.21.15.90:0/2373654596 >> v2:172.21.15.120:3300/0 conn(0x7fbf380b2ac0 msgr2=0x7fbf380b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:40.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.809+0000 7fbf36000640 1 --2- 172.21.15.90:0/2373654596 >> v2:172.21.15.120:3300/0 conn(0x7fbf380b2ac0 0x7fbf380b4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.809+0000 7fbf36000640 1 -- 172.21.15.90:0/2373654596 >> v1:172.21.15.120:6789/0 conn(0x7fbf380b4d50 legacy=0x7fbf380b2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:40.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.809+0000 7fbf36000640 1 -- 172.21.15.90:0/2373654596 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf380cf0f0 con 0x7fbf3806ed40 2024-03-24T01:24:40.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.809+0000 7fbf36000640 1 -- 172.21.15.90:0/2373654596 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7fbf30001a20 con 0x7fbf3806ed40 2024-03-24T01:24:40.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.809+0000 7fbf36000640 1 -- 172.21.15.90:0/2373654596 >> v1:172.21.15.90:6789/0 conn(0x7fbf3806ed40 legacy=0x7fbf3816bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.809+0000 7f2befe00640 1 -- 172.21.15.90:0/3580436805 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1083239688 0 0) 0x7f2bf015a7e0 con 0x7f2bf0071990 2024-03-24T01:24:40.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.809+0000 7f2befe00640 1 -- 172.21.15.90:0/3580436805 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bd8003440 con 0x7f2bf0071990 2024-03-24T01:24:40.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.808+0000 7f3feb400640 1 -- 172.21.15.90:0/2066522336 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd8035020 con 0x7f3fec15e1d0 2024-03-24T01:24:40.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.809+0000 7f3feb400640 1 -- 172.21.15.90:0/2066522336 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd8031830 con 0x7f3fec15e1d0 2024-03-24T01:24:40.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.809+0000 7f3feb400640 1 -- 172.21.15.90:0/2066522336 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd8031c00 con 0x7f3fec15e1d0 2024-03-24T01:24:40.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.809+0000 7fbf36000640 1 --2- 172.21.15.90:0/2373654596 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf200039b0 0x7fbf20003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.809+0000 7fbf36000640 1 --2- 172.21.15.90:0/2373654596 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf20004800 0x7fbf20004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.809+0000 7fbf36000640 1 --2- 172.21.15.90:0/2373654596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf200051c0 0x7fbf2000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.809+0000 7fbf36000640 1 -- 172.21.15.90:0/2373654596 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf380cee10 con 0x7fbf200039b0 2024-03-24T01:24:40.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.809+0000 7fbf36000640 1 -- 172.21.15.90:0/2373654596 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2000c040 con 0x7fbf200051c0 2024-03-24T01:24:40.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.809+0000 7fbf36000640 1 -- 172.21.15.90:0/2373654596 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2000c240 con 0x7fbf20004800 2024-03-24T01:24:40.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.809+0000 7f260ac00640 1 -- 172.21.15.90:0/1290857908 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25fc04f720 con 0x7f25fc0e30c0 2024-03-24T01:24:40.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.809+0000 7f2601600640 1 -- 172.21.15.90:0/1290857908 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2604062240 con 0x7f25fc0e30c0 2024-03-24T01:24:40.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.809+0000 7f2601600640 1 -- 172.21.15.90:0/1290857908 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26040be020 con 0x7f25fc0e30c0 2024-03-24T01:24:40.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.809+0000 7f70f7e00640 1 -- 172.21.15.90:0/3485447629 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f000f040 con 0x7f71000af040 2024-03-24T01:24:40.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.809+0000 7f70f7e00640 1 -- 172.21.15.90:0/3485447629 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70f0004030 con 0x7f71000af040 2024-03-24T01:24:40.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.809+0000 7f70f7e00640 1 -- 172.21.15.90:0/3485447629 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f00043e0 con 0x7f71000af040 2024-03-24T01:24:40.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.809+0000 7f260ac00640 1 -- 172.21.15.90:0/1290857908 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25fc04e9e0 con 0x7f25fc0e30c0 2024-03-24T01:24:40.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.809+0000 7f7106e00640 1 -- 172.21.15.90:0/3485447629 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7100200ca0 con 0x7f71000af040 2024-03-24T01:24:40.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.810+0000 7f7106e00640 1 -- 172.21.15.90:0/3485447629 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7100201260 con 0x7f71000af040 2024-03-24T01:24:40.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.809+0000 7fda4da00640 1 -- 172.21.15.90:0/2948711698 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda480ce960 msgr2=0x7fda480d28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.809+0000 7fda4da00640 1 --2- 172.21.15.90:0/2948711698 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda480ce960 0x7fda480d28a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fda40009d90 tx=0x7fda4002bd90 comp rx=0 tx=0).stop 2024-03-24T01:24:40.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.809+0000 7fda4da00640 1 -- 172.21.15.90:0/2948711698 shutdown_connections 2024-03-24T01:24:40.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.809+0000 7fda4da00640 1 --2- 172.21.15.90:0/2948711698 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda480ce960 0x7fda480d28a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.809+0000 7fda4da00640 1 --2- 172.21.15.90:0/2948711698 >> v2:172.21.15.120:3300/0 conn(0x7fda480d10c0 0x7fda480ce420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.809+0000 7fda4da00640 1 -- 172.21.15.90:0/2948711698 >> 172.21.15.90:0/2948711698 conn(0x7fda4806c9c0 msgr2=0x7fda4806cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:40.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.809+0000 7f2bf4c00640 1 --2- 172.21.15.90:0/3580436805 >> v2:172.21.15.120:3300/0 conn(0x7f2bf00b2130 0x7f2bf015e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.810+0000 7f2befe00640 1 -- 172.21.15.90:0/3580436805 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2355021024 0 0) 0x7f2bf015aac0 con 0x7f2bf015a320 2024-03-24T01:24:40.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.810+0000 7f2befe00640 1 -- 172.21.15.90:0/3580436805 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bf015a7e0 con 0x7f2bf015a320 2024-03-24T01:24:40.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.810+0000 7fbf36a00640 1 --2- 172.21.15.90:0/2373654596 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf20004800 0x7fbf20004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.810+0000 7fbf37400640 1 --2- 172.21.15.90:0/2373654596 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf200039b0 0x7fbf20003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.810+0000 7f3ff3800640 1 -- 172.21.15.90:0/2066522336 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3fec15e1d0 msgr2=0x7f3fec0b5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.810+0000 7f3ff3800640 1 --2- 172.21.15.90:0/2066522336 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3fec15e1d0 0x7f3fec0b5660 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3fd80097a0 tx=0x7f3fd802b8d0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.810+0000 7f3ff3800640 1 -- 172.21.15.90:0/2066522336 shutdown_connections 2024-03-24T01:24:40.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.810+0000 7f3ff3800640 1 --2- 172.21.15.90:0/2066522336 >> v2:172.21.15.120:3300/0 conn(0x7f3fec0b5ba0 0x7f3fec0d2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.810+0000 7f3ff3800640 1 --2- 172.21.15.90:0/2066522336 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3fec15e1d0 0x7f3fec0b5660 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.810+0000 7f3ff3800640 1 -- 172.21.15.90:0/2066522336 >> 172.21.15.90:0/2066522336 conn(0x7f3fec06c930 msgr2=0x7f3fec06cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:40.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.810+0000 7f068fe00640 1 -- 172.21.15.90:0/2968220333 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f06840047e0 con 0x7f06980cbfc0 2024-03-24T01:24:40.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.810+0000 7fda4da00640 1 -- 172.21.15.90:0/2948711698 shutdown_connections 2024-03-24T01:24:40.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.810+0000 7fda4da00640 1 -- 172.21.15.90:0/2948711698 wait complete. 2024-03-24T01:24:40.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.810+0000 7fbf3cc00640 1 --2- 172.21.15.90:0/2373654596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf200051c0 0x7fbf2000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.810+0000 7f3ff3800640 1 -- 172.21.15.90:0/2066522336 shutdown_connections 2024-03-24T01:24:40.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.810+0000 7f3ff3800640 1 -- 172.21.15.90:0/2066522336 wait complete. 2024-03-24T01:24:40.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.811+0000 7fbf37400640 1 -- 172.21.15.90:0/2373654596 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf20004800 msgr2=0x7fbf20004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:40.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.811+0000 7fbf37400640 1 --2- 172.21.15.90:0/2373654596 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf20004800 0x7fbf20004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.811+0000 7fbf37400640 1 -- 172.21.15.90:0/2373654596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf200051c0 msgr2=0x7fbf2000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:40.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.811+0000 7fbf37400640 1 --2- 172.21.15.90:0/2373654596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf200051c0 0x7fbf2000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.811+0000 7fbf37400640 1 -- 172.21.15.90:0/2373654596 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf380cf0f0 con 0x7fbf200039b0 2024-03-24T01:24:40.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.811+0000 7f260ac00640 1 -- 172.21.15.90:0/1290857908 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f25fc0057e0 con 0x7f25fc0e30c0 2024-03-24T01:24:40.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.811+0000 7f2befe00640 1 -- 172.21.15.90:0/3580436805 <== mon.1 v1:172.21.15.120:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1302259135 0 0) 0x7f2bf015a7e0 con 0x7f2bf015a320 2024-03-24T01:24:40.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.811+0000 7f2befe00640 1 -- 172.21.15.90:0/3580436805 >> v2:172.21.15.120:3300/0 conn(0x7f2bf00b2130 msgr2=0x7f2bf015e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:40.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.811+0000 7f2befe00640 1 --2- 172.21.15.90:0/3580436805 >> v2:172.21.15.120:3300/0 conn(0x7f2bf00b2130 0x7f2bf015e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.811+0000 7f2befe00640 1 -- 172.21.15.90:0/3580436805 >> v1:172.21.15.90:6789/0 conn(0x7f2bf0071990 legacy=0x7f2bf006f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.811+0000 7f2befe00640 1 -- 172.21.15.90:0/3580436805 --> v1:172.21.15.120:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf015b080 con 0x7f2bf015a320 2024-03-24T01:24:40.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.811+0000 7f2befe00640 1 -- 172.21.15.90:0/3580436805 <== mon.1 v1:172.21.15.120:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f2be4001670 con 0x7f2bf015a320 2024-03-24T01:24:40.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.811+0000 7fbf3cc00640 1 -- 172.21.15.90:0/2373654596 reap_dead start 2024-03-24T01:24:40.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.811+0000 7f2befe00640 1 -- 172.21.15.90:0/3580436805 >> v1:172.21.15.120:6789/0 conn(0x7f2bf015a320 legacy=0x7f2bf00b2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.811+0000 7f2befe00640 1 --2- 172.21.15.90:0/3580436805 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bd8003c80 0x7f2bd800ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.812+0000 7f2befe00640 1 --2- 172.21.15.90:0/3580436805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2bd8004080 0x7f2bd800d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.812+0000 7f2befe00640 1 --2- 172.21.15.90:0/3580436805 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2bd800d7a0 0x7f2bd800fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.812+0000 7f2befe00640 1 -- 172.21.15.90:0/3580436805 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf015ada0 con 0x7f2bd8004080 2024-03-24T01:24:40.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.812+0000 7f2befe00640 1 -- 172.21.15.90:0/3580436805 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd8010540 con 0x7f2bd8003c80 2024-03-24T01:24:40.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.812+0000 7f2befe00640 1 -- 172.21.15.90:0/3580436805 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd8010740 con 0x7f2bd800d7a0 2024-03-24T01:24:40.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.812+0000 7f068fe00640 1 --2- 172.21.15.90:0/2968220333 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f067806bfd0 0x7f067806e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.812+0000 7f068fe00640 1 -- 172.21.15.90:0/2968220333 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f06840072e0 con 0x7f06980cbfc0 2024-03-24T01:24:40.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.812+0000 7f068fe00640 1 --2- 172.21.15.90:0/2968220333 >> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] conn(0x7f0678075650 0x7f0678077a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.812+0000 7f068fe00640 1 -- 172.21.15.90:0/2968220333 --> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0678078160 con 0x7f0678075650 2024-03-24T01:24:40.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.812+0000 7f068fe00640 1 -- 172.21.15.90:0/2968220333 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f068408a920 con 0x7f06980cbfc0 2024-03-24T01:24:40.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.812+0000 7f3ff3800640 1 Processor -- start 2024-03-24T01:24:40.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.812+0000 7f3ff3800640 1 -- start start 2024-03-24T01:24:40.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.813+0000 7fda4da00640 1 Processor -- start 2024-03-24T01:24:40.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.813+0000 7f2bf5600640 1 --2- 172.21.15.90:0/3580436805 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bd8003c80 0x7f2bd800ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.813+0000 7f2bf6000640 1 --2- 172.21.15.90:0/3580436805 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2bd800d7a0 0x7f2bd800fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.812+0000 7f0697e00640 1 --2- 172.21.15.90:0/2968220333 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f067806bfd0 0x7f067806e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.813+0000 7fda4da00640 1 -- start start 2024-03-24T01:24:40.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.813+0000 7fbf37400640 1 --2- 172.21.15.90:0/2373654596 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf200039b0 0x7fbf20003db0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c009ac0 tx=0x7fbf20005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bc5f2231221eff8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:40.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.813+0000 7f3ff3800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fec0b5ba0 0x7f3fec0ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.813+0000 7f3ff3800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3fec15ddf0 0x7f3fec0cd530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.813+0000 7f3ff3800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3fec0cb030 0x7f3fec0cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.813+0000 7fbf36000640 1 -- 172.21.15.90:0/2373654596 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c039070 con 0x7fbf200039b0 2024-03-24T01:24:40.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.813+0000 7facd7e00640 1 --2- 172.21.15.90:0/362221032 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7face00d0ef0 0x7face00ce420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7facd00097a0 tx=0x7facd002b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5332fb9323ad13e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:40.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.813+0000 7f3ff3800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fec0d2220 con 0x7f3fec0b5ba0 2024-03-24T01:24:40.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.813+0000 7f3ff3800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fec0d1f40 con 0x7f3fec0cb030 2024-03-24T01:24:40.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.813+0000 7f3ff3800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fec0b6340 con 0x7f3fec15ddf0 2024-03-24T01:24:40.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.814+0000 7f2bf4c00640 1 --2- 172.21.15.90:0/3580436805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2bd8004080 0x7f2bd800d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.814+0000 7fbf36000640 1 -- 172.21.15.90:0/2373654596 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf2c02c6b0 con 0x7fbf200039b0 2024-03-24T01:24:40.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.813+0000 7f7106e00640 1 -- 172.21.15.90:0/3485447629 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f71000b4930 con 0x7f71000af040 2024-03-24T01:24:40.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.814+0000 7f2bf6000640 1 -- 172.21.15.90:0/3580436805 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bd8003c80 msgr2=0x7f2bd800ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:40.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.814+0000 7f3ff1800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3fec0cb030 0x7f3fec0cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.814+0000 7f3ff1800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3fec0cb030 0x7f3fec0cb4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.90:39082/0 (socket says 172.21.15.90:39082) 2024-03-24T01:24:40.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.814+0000 7f3ff1800640 1 -- 172.21.15.90:0/536219688 learned_addr learned my addr 172.21.15.90:0/536219688 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:40.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.814+0000 7f3ff0e00640 1 --2- 172.21.15.90:0/536219688 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fec0b5ba0 0x7f3fec0ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.814+0000 7fbf36000640 1 -- 172.21.15.90:0/2373654596 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c009690 con 0x7fbf200039b0 2024-03-24T01:24:40.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.814+0000 7facdd400640 1 -- 172.21.15.90:0/362221032 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd0031780 con 0x7face00d0ef0 2024-03-24T01:24:40.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.814+0000 7facdd400640 1 -- 172.21.15.90:0/362221032 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd0037070 con 0x7face00d0ef0 2024-03-24T01:24:40.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.814+0000 7facdd400640 1 -- 172.21.15.90:0/362221032 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd0031b40 con 0x7face00d0ef0 2024-03-24T01:24:40.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.814+0000 7f3febe00640 1 --2- 172.21.15.90:0/536219688 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3fec15ddf0 0x7f3fec0cd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.814+0000 7f069ca00640 1 --2- 172.21.15.90:0/2968220333 >> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] conn(0x7f0678075650 0x7f0678077a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.815+0000 7f3ff0e00640 1 -- 172.21.15.90:0/536219688 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3fec15ddf0 msgr2=0x7f3fec0cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.815+0000 7f3ff0e00640 1 --2- 172.21.15.90:0/536219688 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3fec15ddf0 0x7f3fec0cd530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.815+0000 7f3ff0e00640 1 -- 172.21.15.90:0/536219688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3fec0cb030 msgr2=0x7f3fec0cb4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.815+0000 7f3ff0e00640 1 --2- 172.21.15.90:0/536219688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3fec0cb030 0x7f3fec0cb4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.815+0000 7f3ff0e00640 1 -- 172.21.15.90:0/536219688 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fec13b740 con 0x7f3fec0b5ba0 2024-03-24T01:24:40.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.815+0000 7fda4da00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda480ce960 0x7fda4815e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.815+0000 7fda4da00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda480d10c0 0x7fda481e88f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.815+0000 7fbf3e200640 1 -- 172.21.15.90:0/2373654596 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf200039b0 msgr2=0x7fbf20003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.815+0000 7fbf3e200640 1 --2- 172.21.15.90:0/2373654596 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf200039b0 0x7fbf20003db0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c009ac0 tx=0x7fbf20005830 comp rx=0 tx=0).stop 2024-03-24T01:24:40.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.815+0000 7fda4da00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda4815ded0 0x7fda480c6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.815+0000 7fda4da00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda4815e2b0 con 0x7fda480d10c0 2024-03-24T01:24:40.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.815+0000 7fda4da00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda480d20a0 con 0x7fda480ce960 2024-03-24T01:24:40.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.815+0000 7fda4da00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda480b6df0 con 0x7fda4815ded0 2024-03-24T01:24:40.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.814+0000 7f2bf6000640 1 --2- 172.21.15.90:0/3580436805 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bd8003c80 0x7f2bd800ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.814+0000 7f2bf6000640 1 -- 172.21.15.90:0/3580436805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2bd8004080 msgr2=0x7f2bd800d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:40.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.814+0000 7f2bf6000640 1 --2- 172.21.15.90:0/3580436805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2bd8004080 0x7f2bd800d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.814+0000 7f2bf6000640 1 -- 172.21.15.90:0/3580436805 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf015a7e0 con 0x7f2bd800d7a0 2024-03-24T01:24:40.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.815+0000 7f2bf6000640 1 -- 172.21.15.90:0/3580436805 reap_dead start 2024-03-24T01:24:40.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.815+0000 7f3ff0e00640 1 --2- 172.21.15.90:0/536219688 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fec0b5ba0 0x7f3fec0ccff0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3fe000c750 tx=0x7f3fe000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:40.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.816+0000 7fda3fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda480d10c0 0x7fda481e88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.816+0000 7fda3fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda480d10c0 0x7fda481e88f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60044/0 (socket says 172.21.15.90:60044) 2024-03-24T01:24:40.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.816+0000 7fda3fe00640 1 -- 172.21.15.90:0/3398845861 learned_addr learned my addr 172.21.15.90:0/3398845861 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:40.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.816+0000 7fda46a00640 1 --2- 172.21.15.90:0/3398845861 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda480ce960 0x7fda4815e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.816+0000 7f3fe8c00640 1 -- 172.21.15.90:0/536219688 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe0018070 con 0x7f3fec0b5ba0 2024-03-24T01:24:40.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.816+0000 7f3fe8c00640 1 -- 172.21.15.90:0/536219688 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fe00070d0 con 0x7f3fec0b5ba0 2024-03-24T01:24:40.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.816+0000 7f3fe8c00640 1 -- 172.21.15.90:0/536219688 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe0007500 con 0x7f3fec0b5ba0 2024-03-24T01:24:40.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.816+0000 7face5200640 1 -- 172.21.15.90:0/362221032 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7face00d0ef0 msgr2=0x7face00ce420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.816+0000 7face5200640 1 --2- 172.21.15.90:0/362221032 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7face00d0ef0 0x7face00ce420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7facd00097a0 tx=0x7facd002b820 comp rx=0 tx=0).stop 2024-03-24T01:24:40.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.816+0000 7fbf3e200640 1 -- 172.21.15.90:0/2373654596 shutdown_connections 2024-03-24T01:24:40.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.816+0000 7fbf3e200640 1 --2- 172.21.15.90:0/2373654596 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf200039b0 0x7fbf20003db0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.816+0000 7fbf3e200640 1 -- 172.21.15.90:0/2373654596 >> 172.21.15.90:0/2373654596 conn(0x7fbf380b17c0 msgr2=0x7fbf380b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:40.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.816+0000 7f3ff3800640 1 -- 172.21.15.90:0/536219688 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fec13ba20 con 0x7f3fec0b5ba0 2024-03-24T01:24:40.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.816+0000 7f0697e00640 1 --2- 172.21.15.90:0/2968220333 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f067806bfd0 0x7f067806e490 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f068803a040 tx=0x7f068800b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:40.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.816+0000 7fda47400640 1 --2- 172.21.15.90:0/3398845861 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda4815ded0 0x7fda480c6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.816+0000 7fda3fe00640 1 -- 172.21.15.90:0/3398845861 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda4815ded0 msgr2=0x7fda480c6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.816+0000 7fda3fe00640 1 --2- 172.21.15.90:0/3398845861 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda4815ded0 0x7fda480c6220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.816+0000 7fda3fe00640 1 -- 172.21.15.90:0/3398845861 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda480ce960 msgr2=0x7fda4815e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.816+0000 7fda3fe00640 1 --2- 172.21.15.90:0/3398845861 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda480ce960 0x7fda4815e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.816+0000 7fda3fe00640 1 -- 172.21.15.90:0/3398845861 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda481e9260 con 0x7fda480d10c0 2024-03-24T01:24:40.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.817+0000 7f069ca00640 1 --2- 172.21.15.90:0/2968220333 >> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] conn(0x7f0678075650 0x7f0678077a90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:40.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.816+0000 7fbf3e200640 1 -- 172.21.15.90:0/2373654596 shutdown_connections 2024-03-24T01:24:40.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.817+0000 7fbf3e200640 1 -- 172.21.15.90:0/2373654596 wait complete. 2024-03-24T01:24:40.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.816+0000 7f3ff3800640 1 -- 172.21.15.90:0/536219688 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fec13bfe0 con 0x7f3fec0b5ba0 2024-03-24T01:24:40.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.817+0000 7fda3fe00640 1 --2- 172.21.15.90:0/3398845861 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda480d10c0 0x7fda481e88f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fda3000e9d0 tx=0x7fda3000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:40.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.816+0000 7f2bf6000640 1 --2- 172.21.15.90:0/3580436805 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2bd800d7a0 0x7f2bd800fc60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2be800c2f0 tx=0x7f2bd8003730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad18890d973f9aa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:40.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.817+0000 7fda3ea00640 1 -- 172.21.15.90:0/3398845861 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda30017070 con 0x7fda480d10c0 2024-03-24T01:24:40.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.818+0000 7fda3ea00640 1 -- 172.21.15.90:0/3398845861 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda3000ca00 con 0x7fda480d10c0 2024-03-24T01:24:40.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.818+0000 7fda3ea00640 1 -- 172.21.15.90:0/3398845861 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda3000cdb0 con 0x7fda480d10c0 2024-03-24T01:24:40.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.818+0000 7fda4da00640 1 -- 172.21.15.90:0/3398845861 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda481e9540 con 0x7fda480d10c0 2024-03-24T01:24:40.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.818+0000 7fda4da00640 1 -- 172.21.15.90:0/3398845861 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda481e9b00 con 0x7fda480d10c0 2024-03-24T01:24:40.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.819+0000 7fda4da00640 1 -- 172.21.15.90:0/3398845861 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fda48068060 con 0x7fda480d10c0 2024-03-24T01:24:40.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.817+0000 7f2befe00640 1 -- 172.21.15.90:0/3580436805 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be8002090 con 0x7f2bd800d7a0 2024-03-24T01:24:40.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.818+0000 7f2befe00640 1 -- 172.21.15.90:0/3580436805 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2be8030070 con 0x7f2bd800d7a0 2024-03-24T01:24:40.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.819+0000 7fbf3e200640 1 Processor -- start 2024-03-24T01:24:40.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.819+0000 7fbf3e200640 1 -- start start 2024-03-24T01:24:40.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.817+0000 7face5200640 1 -- 172.21.15.90:0/362221032 shutdown_connections 2024-03-24T01:24:40.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.817+0000 7face5200640 1 --2- 172.21.15.90:0/362221032 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7face00d0ef0 0x7face00ce420 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.817+0000 7face5200640 1 --2- 172.21.15.90:0/362221032 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7face015ded0 0x7face00d09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.817+0000 7face5200640 1 -- 172.21.15.90:0/362221032 >> 172.21.15.90:0/362221032 conn(0x7face006c9c0 msgr2=0x7face006cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.818+0000 7face5200640 1 -- 172.21.15.90:0/362221032 shutdown_connections 2024-03-24T01:24:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.817+0000 7f3ff3800640 1 -- 172.21.15.90:0/536219688 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3fec0cda70 con 0x7f3fec0b5ba0 2024-03-24T01:24:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.818+0000 7face5200640 1 -- 172.21.15.90:0/362221032 wait complete. 2024-03-24T01:24:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.818+0000 7f2601600640 1 -- 172.21.15.90:0/1290857908 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f26040c1020 con 0x7f25fc0e30c0 2024-03-24T01:24:40.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.819+0000 7f068fe00640 1 -- 172.21.15.90:0/2968220333 <== osd.1 v2:172.21.15.90:6816/133441085 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0678078160 con 0x7f0678075650 2024-03-24T01:24:40.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.818+0000 7f70f7e00640 1 -- 172.21.15.90:0/3485447629 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f70f000c650 con 0x7f71000af040 2024-03-24T01:24:40.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.819+0000 7f70f7e00640 1 --2- 172.21.15.90:0/3485447629 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f70e006bca0 0x7f70e006e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.818+0000 7f2bf8000640 1 -- 172.21.15.90:0/3580436805 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2bd800d7a0 msgr2=0x7f2bd800fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.818+0000 7f2bf8000640 1 --2- 172.21.15.90:0/3580436805 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2bd800d7a0 0x7f2bd800fc60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2be800c2f0 tx=0x7f2bd8003730 comp rx=0 tx=0).stop 2024-03-24T01:24:40.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.820+0000 7f2bf8000640 1 -- 172.21.15.90:0/3580436805 shutdown_connections 2024-03-24T01:24:40.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.820+0000 7fbf3e200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf200039b0 0x7fbf38162d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.820+0000 7fbf3e200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf381fb990 0x7fbf381ec950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.820+0000 7fbf3e200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf381fad80 0x7fbf381fb230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.820+0000 7fbf3e200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf380b4810 con 0x7fbf381fb990 2024-03-24T01:24:40.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.820+0000 7fbf3e200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf380ce800 con 0x7fbf200039b0 2024-03-24T01:24:40.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.820+0000 7f2bf8000640 1 --2- 172.21.15.90:0/3580436805 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2bd800d7a0 0x7f2bd800fc60 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.820+0000 7f2bf8000640 1 -- 172.21.15.90:0/3580436805 >> 172.21.15.90:0/3580436805 conn(0x7f2bf00b1c80 msgr2=0x7f2bf00cf7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:40.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.820+0000 7fbf3e200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf380b4990 con 0x7fbf381fad80 2024-03-24T01:24:40.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.820+0000 7f70f7e00640 1 -- 172.21.15.90:0/3485447629 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f70f008a8b0 con 0x7f71000af040 2024-03-24T01:24:40.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.820+0000 7f70ffe00640 1 --2- 172.21.15.90:0/3485447629 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f70e006bca0 0x7f70e006e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.820+0000 7f2601600640 1 --2- 172.21.15.90:0/1290857908 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f25e406be00 0x7f25e406e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.820+0000 7f2601600640 1 -- 172.21.15.90:0/1290857908 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f2604128650 con 0x7f25fc0e30c0 2024-03-24T01:24:40.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.821+0000 7f2601600640 1 --2- 172.21.15.90:0/1290857908 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x7f25e4075500 0x7f25e4077940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.821+0000 7f70f7e00640 1 --2- 172.21.15.90:0/3485447629 >> [v2:172.21.15.90:6817/865053620,v1:172.21.15.90:6820/865053620] conn(0x7f70e0075420 0x7f70e0077860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.821+0000 7fbf36a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf381fb990 0x7fbf381ec950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.821+0000 7fbf37400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf200039b0 0x7fbf38162d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.821+0000 7fbf37400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf200039b0 0x7fbf38162d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.90:39094/0 (socket says 172.21.15.90:39094) 2024-03-24T01:24:40.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.821+0000 7fbf37400640 1 -- 172.21.15.90:0/454894854 learned_addr learned my addr 172.21.15.90:0/454894854 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:40.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.821+0000 7f2601600640 1 -- 172.21.15.90:0/1290857908 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f26040b9e50 con 0x7f25e4075500 2024-03-24T01:24:40.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.821+0000 7f2601600640 1 -- 172.21.15.90:0/1290857908 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2604128a40 con 0x7f25fc0e30c0 2024-03-24T01:24:40.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.821+0000 7f2bf8000640 1 -- 172.21.15.90:0/3580436805 shutdown_connections 2024-03-24T01:24:40.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.821+0000 7f2bf8000640 1 -- 172.21.15.90:0/3580436805 wait complete. 2024-03-24T01:24:40.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.821+0000 7f2609800640 1 --2- 172.21.15.90:0/1290857908 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f25e406be00 0x7f25e406e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.821+0000 7f70f7e00640 1 -- 172.21.15.90:0/3485447629 --> [v2:172.21.15.90:6817/865053620,v1:172.21.15.90:6820/865053620] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f70f0001d30 con 0x7f70e0075420 2024-03-24T01:24:40.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.821+0000 7f70f7e00640 1 -- 172.21.15.90:0/3485447629 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f70f008f2a0 con 0x7f71000af040 2024-03-24T01:24:40.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.821+0000 7face5200640 1 Processor -- start 2024-03-24T01:24:40.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.821+0000 7f70ffe00640 1 --2- 172.21.15.90:0/3485447629 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f70e006bca0 0x7f70e006e160 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f70f8005c00 tx=0x7f70f8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:40.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.822+0000 7f7104e00640 1 --2- 172.21.15.90:0/3485447629 >> [v2:172.21.15.90:6817/865053620,v1:172.21.15.90:6820/865053620] conn(0x7f70e0075420 0x7f70e0077860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.821+0000 7f2608e00640 1 --2- 172.21.15.90:0/1290857908 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x7f25e4075500 0x7f25e4077940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.822+0000 7fbf36a00640 1 -- 172.21.15.90:0/454894854 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf381fad80 msgr2=0x7fbf381fb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.822+0000 7fbf36a00640 1 --2- 172.21.15.90:0/454894854 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf381fad80 0x7fbf381fb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.822+0000 7fbf36a00640 1 -- 172.21.15.90:0/454894854 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf200039b0 msgr2=0x7fbf38162d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.822+0000 7fbf36a00640 1 --2- 172.21.15.90:0/454894854 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf200039b0 0x7fbf38162d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.822+0000 7fbf36a00640 1 -- 172.21.15.90:0/454894854 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf38144250 con 0x7fbf381fb990 2024-03-24T01:24:40.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.822+0000 7fbf36a00640 1 --2- 172.21.15.90:0/454894854 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf381fb990 0x7fbf381ec950 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbf38062c60 tx=0x7fbf2400f900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:40.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.823+0000 7f7104e00640 1 --2- 172.21.15.90:0/3485447629 >> [v2:172.21.15.90:6817/865053620,v1:172.21.15.90:6820/865053620] conn(0x7f70e0075420 0x7f70e0077860 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:40.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.822+0000 7f2608e00640 1 --2- 172.21.15.90:0/1290857908 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x7f25e4075500 0x7f25e4077940 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:40.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.823+0000 7f2609800640 1 --2- 172.21.15.90:0/1290857908 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f25e406be00 0x7f25e406e2c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f25fc03e5c0 tx=0x7f25f8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:40.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.823+0000 7fbf2b400640 1 -- 172.21.15.90:0/454894854 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2401a070 con 0x7fbf381fb990 2024-03-24T01:24:40.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.823+0000 7fbf2b400640 1 -- 172.21.15.90:0/454894854 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf24005320 con 0x7fbf381fb990 2024-03-24T01:24:40.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.823+0000 7fbf2b400640 1 -- 172.21.15.90:0/454894854 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf24007290 con 0x7fbf381fb990 2024-03-24T01:24:40.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.824+0000 7f2bf8000640 1 Processor -- start 2024-03-24T01:24:40.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.823+0000 7face5200640 1 -- start start 2024-03-24T01:24:40.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.824+0000 7f2bf8000640 1 -- start start 2024-03-24T01:24:40.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.824+0000 7f2601600640 1 -- 172.21.15.90:0/1290857908 <== osd.0 v2:172.21.15.90:6800/307045587 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f26040b9e50 con 0x7f25e4075500 2024-03-24T01:24:40.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.823+0000 7f3fe8c00640 1 -- 172.21.15.90:0/536219688 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3fe001b070 con 0x7f3fec0b5ba0 2024-03-24T01:24:40.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.825+0000 7f3fe8c00640 1 --2- 172.21.15.90:0/536219688 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3fcc06bca0 0x7f3fcc06e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.825+0000 7face5200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7face00ce960 0x7face00cbff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.825+0000 7face5200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7face00d0ef0 0x7face00b71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.825+0000 7f70f7e00640 1 -- 172.21.15.90:0/3485447629 <== osd.2 v2:172.21.15.90:6817/865053620 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f70f0001d30 con 0x7f70e0075420 2024-03-24T01:24:40.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.825+0000 7face5200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7face015ded0 0x7face0155ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.825+0000 7face5200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face00d1dc0 con 0x7face015ded0 2024-03-24T01:24:40.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.825+0000 7face5200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face00cf2e0 con 0x7face00d0ef0 2024-03-24T01:24:40.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.825+0000 7face5200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7face00d20a0 con 0x7face00ce960 2024-03-24T01:24:40.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.825+0000 7fbf3e200640 1 -- 172.21.15.90:0/454894854 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf381ebd40 con 0x7fbf381fb990 2024-03-24T01:24:40.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.825+0000 7facd7e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7face00d0ef0 0x7face00b71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.825+0000 7f3febe00640 1 --2- 172.21.15.90:0/536219688 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3fcc06bca0 0x7f3fcc06e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.825+0000 7facdde00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7face00ce960 0x7face00cbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.825+0000 7facdde00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7face00ce960 0x7face00cbff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.90:50136/0 (socket says 172.21.15.90:50136) 2024-03-24T01:24:40.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.825+0000 7facdde00640 1 -- 172.21.15.90:0/2172615501 learned_addr learned my addr 172.21.15.90:0/2172615501 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:40.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.826+0000 7facde800640 1 --2- 172.21.15.90:0/2172615501 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7face015ded0 0x7face0155ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.826+0000 7fbf3e200640 1 -- 172.21.15.90:0/454894854 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf381447d0 con 0x7fbf381fb990 2024-03-24T01:24:40.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.823+0000 7fda3ea00640 1 -- 172.21.15.90:0/3398845861 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fda3001a070 con 0x7fda480d10c0 2024-03-24T01:24:40.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.826+0000 7facde800640 1 -- 172.21.15.90:0/2172615501 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7face00ce960 msgr2=0x7face00cbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.826+0000 7facde800640 1 --2- 172.21.15.90:0/2172615501 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7face00ce960 0x7face00cbff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.826+0000 7facde800640 1 -- 172.21.15.90:0/2172615501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7face00d0ef0 msgr2=0x7face00b71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.826+0000 7facde800640 1 --2- 172.21.15.90:0/2172615501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7face00d0ef0 0x7face00b71f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.826+0000 7facde800640 1 -- 172.21.15.90:0/2172615501 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face00b7960 con 0x7face015ded0 2024-03-24T01:24:40.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.825+0000 7f3fe8c00640 1 -- 172.21.15.90:0/536219688 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f3fe008a7c0 con 0x7f3fec0b5ba0 2024-03-24T01:24:40.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.826+0000 7f3fe8c00640 1 --2- 172.21.15.90:0/536219688 >> [v2:172.21.15.120:6808/1045862942,v1:172.21.15.120:6809/1045862942] conn(0x7f3fcc075420 0x7f3fcc077860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.826+0000 7f3febe00640 1 --2- 172.21.15.90:0/536219688 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3fcc06bca0 0x7f3fcc06e160 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3fd8036000 tx=0x7f3fd803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:40.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.827+0000 7f3fe8c00640 1 -- 172.21.15.90:0/536219688 --> [v2:172.21.15.120:6808/1045862942,v1:172.21.15.120:6809/1045862942] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3fcc077f30 con 0x7f3fcc075420 2024-03-24T01:24:40.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.827+0000 7f3fe8c00640 1 -- 172.21.15.90:0/536219688 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3fe008abb0 con 0x7f3fec0b5ba0 2024-03-24T01:24:40.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.827+0000 7fbf3e200640 1 -- 172.21.15.90:0/454894854 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbf38068060 con 0x7fbf381fb990 2024-03-24T01:24:40.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.827+0000 7facde800640 1 --2- 172.21.15.90:0/2172615501 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7face015ded0 0x7face0155ff0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7facd800c7b0 tx=0x7facd800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:40.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.827+0000 7f2bf8000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2bd800d7a0 0x7f2bf013e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.827+0000 7f3ff1800640 1 --2- 172.21.15.90:0/536219688 >> [v2:172.21.15.120:6808/1045862942,v1:172.21.15.120:6809/1045862942] conn(0x7f3fcc075420 0x7f3fcc077860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.827+0000 7f2bf8000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2bf0152d30 0x7f2bf01eed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.827+0000 7f2bf8000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bf014e160 0x7f2bf0156e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.828+0000 7facd6000640 1 -- 172.21.15.90:0/2172615501 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd8017070 con 0x7face015ded0 2024-03-24T01:24:40.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.828+0000 7f2bf5600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2bd800d7a0 0x7f2bf013e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.828+0000 7f2bf5600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2bd800d7a0 0x7f2bf013e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.90:50146/0 (socket says 172.21.15.90:50146) 2024-03-24T01:24:40.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.828+0000 7f2bf5600640 1 -- 172.21.15.90:0/3153072976 learned_addr learned my addr 172.21.15.90:0/3153072976 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:40.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.828+0000 7face5200640 1 -- 172.21.15.90:0/2172615501 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face00b7c40 con 0x7face015ded0 2024-03-24T01:24:40.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.828+0000 7facd6000640 1 -- 172.21.15.90:0/2172615501 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd8003b70 con 0x7face015ded0 2024-03-24T01:24:40.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.828+0000 7face5200640 1 -- 172.21.15.90:0/2172615501 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face00b8200 con 0x7face015ded0 2024-03-24T01:24:40.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.828+0000 7facd6000640 1 -- 172.21.15.90:0/2172615501 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd8017070 con 0x7face015ded0 2024-03-24T01:24:40.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.828+0000 7f2bf8000640 1 -- 172.21.15.90:0/3153072976 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf015e4a0 con 0x7f2bf0152d30 2024-03-24T01:24:40.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.828+0000 7f2bf8000640 1 -- 172.21.15.90:0/3153072976 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf015e320 con 0x7f2bf014e160 2024-03-24T01:24:40.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.828+0000 7f3ff1800640 1 --2- 172.21.15.90:0/536219688 >> [v2:172.21.15.120:6808/1045862942,v1:172.21.15.120:6809/1045862942] conn(0x7f3fcc075420 0x7f3fcc077860 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:40.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.828+0000 7f2bf8000640 1 -- 172.21.15.90:0/3153072976 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf015e620 con 0x7f2bd800d7a0 2024-03-24T01:24:40.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.828+0000 7f2bf4c00640 1 --2- 172.21.15.90:0/3153072976 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2bf0152d30 0x7f2bf01eed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.829+0000 7face5200640 1 -- 172.21.15.90:0/2172615501 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7facb8001030 con 0x7face015ded0 2024-03-24T01:24:40.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.829+0000 7fda3ea00640 1 --2- 172.21.15.90:0/3398845861 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fda2406bbd0 0x7fda2406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.829+0000 7fda3ea00640 1 -- 172.21.15.90:0/3398845861 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fda3008af30 con 0x7fda480d10c0 2024-03-24T01:24:40.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.830+0000 7fda3ea00640 1 --2- 172.21.15.90:0/3398845861 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x7fda24075250 0x7fda24077690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.830+0000 7fda3ea00640 1 -- 172.21.15.90:0/3398845861 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda24077d60 con 0x7fda24075250 2024-03-24T01:24:40.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.830+0000 7fda3ea00640 1 -- 172.21.15.90:0/3398845861 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda3008b3c0 con 0x7fda480d10c0 2024-03-24T01:24:40.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.829+0000 7f2bf4c00640 1 -- 172.21.15.90:0/3153072976 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2bd800d7a0 msgr2=0x7f2bf013e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.829+0000 7f2bf4c00640 1 --2- 172.21.15.90:0/3153072976 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2bd800d7a0 0x7f2bf013e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.829+0000 7f2bf4c00640 1 -- 172.21.15.90:0/3153072976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bf014e160 msgr2=0x7f2bf0156e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.829+0000 7f2bf4c00640 1 --2- 172.21.15.90:0/3153072976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bf014e160 0x7f2bf0156e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.829+0000 7f2bf4c00640 1 -- 172.21.15.90:0/3153072976 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf013ed90 con 0x7f2bf0152d30 2024-03-24T01:24:40.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.829+0000 7f2bf4c00640 1 --2- 172.21.15.90:0/3153072976 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2bf0152d30 0x7f2bf01eed40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc011590 tx=0x7f2bdc011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:40.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.830+0000 7f2bed600640 1 -- 172.21.15.90:0/3153072976 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc015930 con 0x7f2bf0152d30 2024-03-24T01:24:40.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.830+0000 7f2bed600640 1 -- 172.21.15.90:0/3153072976 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bdc015ad0 con 0x7f2bf0152d30 2024-03-24T01:24:40.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.830+0000 7f3fe8c00640 1 -- 172.21.15.90:0/536219688 <== osd.7 v2:172.21.15.120:6808/1045862942 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3fcc077f30 con 0x7f3fcc075420 2024-03-24T01:24:40.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.830+0000 7f2bf8000640 1 -- 172.21.15.90:0/3153072976 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bf013c030 con 0x7f2bf0152d30 2024-03-24T01:24:40.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.830+0000 7fda46a00640 1 --2- 172.21.15.90:0/3398845861 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fda2406bbd0 0x7fda2406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.830+0000 7f2bf8000640 1 -- 172.21.15.90:0/3153072976 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bf013c5f0 con 0x7f2bf0152d30 2024-03-24T01:24:40.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.831+0000 7fda47400640 1 --2- 172.21.15.90:0/3398845861 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x7fda24075250 0x7fda24077690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.832+0000 7fda46a00640 1 --2- 172.21.15.90:0/3398845861 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fda2406bbd0 0x7fda2406e090 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fda380013b0 tx=0x7fda380011c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:40.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.831+0000 7f2bed600640 1 -- 172.21.15.90:0/3153072976 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc015930 con 0x7f2bf0152d30 2024-03-24T01:24:40.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.833+0000 7facd6000640 1 -- 172.21.15.90:0/2172615501 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7facd8005070 con 0x7face015ded0 2024-03-24T01:24:40.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.832+0000 7f2bf8000640 1 -- 172.21.15.90:0/3153072976 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2bf013cca0 con 0x7f2bf0152d30 2024-03-24T01:24:40.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.833+0000 7fda47400640 1 --2- 172.21.15.90:0/3398845861 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x7fda24075250 0x7fda24077690 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:40.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.832+0000 7fbf2b400640 1 -- 172.21.15.90:0/454894854 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fbf2401d070 con 0x7fbf381fb990 2024-03-24T01:24:40.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.833+0000 7fbf2b400640 1 --2- 172.21.15.90:0/454894854 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbf1806bc50 0x7fbf1806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.834+0000 7fbf2b400640 1 -- 172.21.15.90:0/454894854 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fbf24056430 con 0x7fbf381fb990 2024-03-24T01:24:40.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.834+0000 7fbf2b400640 1 --2- 172.21.15.90:0/454894854 >> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] conn(0x7fbf180760f0 0x7fbf18078590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.834+0000 7fbf2b400640 1 -- 172.21.15.90:0/454894854 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf18078c60 con 0x7fbf180760f0 2024-03-24T01:24:40.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.834+0000 7fbf2b400640 1 -- 172.21.15.90:0/454894854 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbf24054dc0 con 0x7fbf381fb990 2024-03-24T01:24:40.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.835+0000 7fda3ea00640 1 -- 172.21.15.90:0/3398845861 <== osd.3 v2:172.21.15.90:6808/2999931983 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fda24077d60 con 0x7fda24075250 2024-03-24T01:24:40.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.834+0000 7facd6000640 1 --2- 172.21.15.90:0/2172615501 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7facbc06be00 0x7facbc06e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.834+0000 7facd6000640 1 -- 172.21.15.90:0/2172615501 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7facd808a310 con 0x7face015ded0 2024-03-24T01:24:40.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.835+0000 7facd6000640 1 --2- 172.21.15.90:0/2172615501 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x7facbc0754f0 0x7facbc077930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.835+0000 7facd6000640 1 -- 172.21.15.90:0/2172615501 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7facbc078000 con 0x7facbc0754f0 2024-03-24T01:24:40.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.835+0000 7facd6000640 1 -- 172.21.15.90:0/2172615501 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7facd808a700 con 0x7face015ded0 2024-03-24T01:24:40.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.835+0000 7f2bed600640 1 -- 172.21.15.90:0/3153072976 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2bdc015c80 con 0x7f2bf0152d30 2024-03-24T01:24:40.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.835+0000 7fbf3cc00640 1 --2- 172.21.15.90:0/454894854 >> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] conn(0x7fbf180760f0 0x7fbf18078590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.835+0000 7fbf37400640 1 --2- 172.21.15.90:0/454894854 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbf1806bc50 0x7fbf1806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.835+0000 7facdde00640 1 --2- 172.21.15.90:0/2172615501 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7facbc06be00 0x7facbc06e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.836+0000 7facd7e00640 1 --2- 172.21.15.90:0/2172615501 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x7facbc0754f0 0x7facbc077930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.836+0000 7f2bed600640 1 --2- 172.21.15.90:0/3153072976 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f2bd006bdd0 0x7f2bd006e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.836+0000 7f2bed600640 1 -- 172.21.15.90:0/3153072976 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f2bdc08b210 con 0x7f2bf0152d30 2024-03-24T01:24:40.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.836+0000 7fbf3cc00640 1 --2- 172.21.15.90:0/454894854 >> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] conn(0x7fbf180760f0 0x7fbf18078590 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:40.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.837+0000 7f2bed600640 1 --2- 172.21.15.90:0/3153072976 >> [v2:172.21.15.120:6800/2150486081,v1:172.21.15.120:6801/2150486081] conn(0x7f2bd0075450 0x7f2bd0077890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:40.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.837+0000 7f2bed600640 1 -- 172.21.15.90:0/3153072976 --> [v2:172.21.15.120:6800/2150486081,v1:172.21.15.120:6801/2150486081] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2bdc001d30 con 0x7f2bd0075450 2024-03-24T01:24:40.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.837+0000 7f2bed600640 1 -- 172.21.15.90:0/3153072976 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2bdc092050 con 0x7f2bf0152d30 2024-03-24T01:24:40.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.837+0000 7fbf37400640 1 --2- 172.21.15.90:0/454894854 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbf1806bc50 0x7fbf1806e110 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c009ac0 tx=0x7fbf2c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:40.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.837+0000 7facdde00640 1 --2- 172.21.15.90:0/2172615501 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7facbc06be00 0x7facbc06e2c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7facc8006a90 tx=0x7facc8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:40.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.837+0000 7facd7e00640 1 --2- 172.21.15.90:0/2172615501 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x7facbc0754f0 0x7facbc077930 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:40.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.837+0000 7f2bf5600640 1 --2- 172.21.15.90:0/3153072976 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f2bd006bdd0 0x7f2bd006e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.837+0000 7f2bf6000640 1 --2- 172.21.15.90:0/3153072976 >> [v2:172.21.15.120:6800/2150486081,v1:172.21.15.120:6801/2150486081] conn(0x7f2bd0075450 0x7f2bd0077890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:40.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.838+0000 7f2bf5600640 1 --2- 172.21.15.90:0/3153072976 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f2bd006bdd0 0x7f2bd006e290 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2be4001030 tx=0x7f2be4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:40.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.838+0000 7f2bf6000640 1 --2- 172.21.15.90:0/3153072976 >> [v2:172.21.15.120:6800/2150486081,v1:172.21.15.120:6801/2150486081] conn(0x7f2bd0075450 0x7f2bd0077890 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:40.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.839+0000 7facd6000640 1 -- 172.21.15.90:0/2172615501 <== osd.6 v2:172.21.15.120:6816/1454745630 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7facbc078000 con 0x7facbc0754f0 2024-03-24T01:24:40.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.840+0000 7fbf2b400640 1 -- 172.21.15.90:0/454894854 <== osd.5 v2:172.21.15.120:6824/1739969887 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fbf18078c60 con 0x7fbf180760f0 2024-03-24T01:24:40.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.841+0000 7f2bed600640 1 -- 172.21.15.90:0/3153072976 <== osd.4 v2:172.21.15.120:6800/2150486081 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2bdc001d30 con 0x7f2bd0075450 2024-03-24T01:24:40.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.846+0000 7f069ea00640 1 -- 172.21.15.90:0/2968220333 --> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0670002dd0 con 0x7f0678075650 2024-03-24T01:24:40.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.847+0000 7f068fe00640 1 -- 172.21.15.90:0/2968220333 <== osd.1 v2:172.21.15.90:6816/133441085 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0670002dd0 con 0x7f0678075650 2024-03-24T01:24:40.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.848+0000 7f069ea00640 1 -- 172.21.15.90:0/2968220333 >> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] conn(0x7f0678075650 msgr2=0x7f0678077a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.848+0000 7f069ea00640 1 --2- 172.21.15.90:0/2968220333 >> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] conn(0x7f0678075650 0x7f0678077a90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.849+0000 7f260ac00640 1 -- 172.21.15.90:0/1290857908 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f25fc04eec0 con 0x7f25e4075500 2024-03-24T01:24:40.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.850+0000 7f2601600640 1 -- 172.21.15.90:0/1290857908 <== osd.0 v2:172.21.15.90:6800/307045587 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f25fc04eec0 con 0x7f25e4075500 2024-03-24T01:24:40.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.850+0000 7f260ac00640 1 -- 172.21.15.90:0/1290857908 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x7f25e4075500 msgr2=0x7f25e4077940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.850+0000 7f260ac00640 1 --2- 172.21.15.90:0/1290857908 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x7f25e4075500 0x7f25e4077940 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.851+0000 7f260ac00640 1 -- 172.21.15.90:0/1290857908 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f25e406be00 msgr2=0x7f25e406e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.851+0000 7f260ac00640 1 --2- 172.21.15.90:0/1290857908 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f25e406be00 0x7f25e406e2c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f25fc03e5c0 tx=0x7f25f8008040 comp rx=0 tx=0).stop 2024-03-24T01:24:40.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.851+0000 7f260ac00640 1 -- 172.21.15.90:0/1290857908 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25fc0e30c0 msgr2=0x7f25fc04c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.851+0000 7f260ac00640 1 --2- 172.21.15.90:0/1290857908 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25fc0e30c0 0x7f25fc04c8a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2604000b60 tx=0x7f2604074830 comp rx=0 tx=0).stop 2024-03-24T01:24:40.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.851+0000 7f260a200640 1 -- 172.21.15.90:0/1290857908 reap_dead start 2024-03-24T01:24:40.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.848+0000 7f069ea00640 1 -- 172.21.15.90:0/2968220333 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f067806bfd0 msgr2=0x7f067806e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.848+0000 7f069ea00640 1 --2- 172.21.15.90:0/2968220333 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f067806bfd0 0x7f067806e490 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f068803a040 tx=0x7f068800b040 comp rx=0 tx=0).stop 2024-03-24T01:24:40.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.849+0000 7f069ea00640 1 -- 172.21.15.90:0/2968220333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06980cbfc0 msgr2=0x7f0698155520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.851+0000 7f260ac00640 1 -- 172.21.15.90:0/1290857908 shutdown_connections 2024-03-24T01:24:40.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.851+0000 7f260ac00640 1 -- 172.21.15.90:0/1290857908 >> 172.21.15.90:0/1290857908 conn(0x7f25fc0559e0 msgr2=0x7f25fc0076e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:40.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.849+0000 7f069ea00640 1 --2- 172.21.15.90:0/2968220333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06980cbfc0 0x7f0698155520 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f068400c3c0 tx=0x7f068400c880 comp rx=0 tx=0).stop 2024-03-24T01:24:40.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.849+0000 7f069ca00640 1 -- 172.21.15.90:0/2968220333 reap_dead start 2024-03-24T01:24:40.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.849+0000 7f069ea00640 1 -- 172.21.15.90:0/2968220333 shutdown_connections 2024-03-24T01:24:40.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.849+0000 7f069ea00640 1 -- 172.21.15.90:0/2968220333 >> 172.21.15.90:0/2968220333 conn(0x7f06980b1c80 msgr2=0x7f069806b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:40.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.850+0000 7f069ea00640 1 -- 172.21.15.90:0/2968220333 shutdown_connections 2024-03-24T01:24:40.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.850+0000 7f069ea00640 1 -- 172.21.15.90:0/2968220333 wait complete. 2024-03-24T01:24:40.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.851+0000 7f260ac00640 1 -- 172.21.15.90:0/1290857908 shutdown_connections 2024-03-24T01:24:40.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.851+0000 7f260ac00640 1 -- 172.21.15.90:0/1290857908 wait complete. 2024-03-24T01:24:40.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.859+0000 7f7106e00640 1 -- 172.21.15.90:0/3485447629 --> [v2:172.21.15.90:6817/865053620,v1:172.21.15.90:6820/865053620] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f71002016a0 con 0x7f70e0075420 2024-03-24T01:24:40.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.862+0000 7f3ff3800640 1 -- 172.21.15.90:0/536219688 --> [v2:172.21.15.120:6808/1045862942,v1:172.21.15.120:6809/1045862942] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3fec08b000 con 0x7f3fcc075420 2024-03-24T01:24:40.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.862+0000 7f70f7e00640 1 -- 172.21.15.90:0/3485447629 <== osd.2 v2:172.21.15.90:6817/865053620 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f71002016a0 con 0x7f70e0075420 2024-03-24T01:24:40.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.863+0000 7f7106e00640 1 -- 172.21.15.90:0/3485447629 >> [v2:172.21.15.90:6817/865053620,v1:172.21.15.90:6820/865053620] conn(0x7f70e0075420 msgr2=0x7f70e0077860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.863+0000 7f7106e00640 1 --2- 172.21.15.90:0/3485447629 >> [v2:172.21.15.90:6817/865053620,v1:172.21.15.90:6820/865053620] conn(0x7f70e0075420 0x7f70e0077860 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.863+0000 7f7106e00640 1 -- 172.21.15.90:0/3485447629 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f70e006bca0 msgr2=0x7f70e006e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.863+0000 7f3fe8c00640 1 -- 172.21.15.90:0/536219688 <== osd.7 v2:172.21.15.120:6808/1045862942 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3fec08b000 con 0x7f3fcc075420 2024-03-24T01:24:40.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.863+0000 7f7106e00640 1 --2- 172.21.15.90:0/3485447629 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f70e006bca0 0x7f70e006e160 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f70f8005c00 tx=0x7f70f8009040 comp rx=0 tx=0).stop 2024-03-24T01:24:40.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.863+0000 7f7106e00640 1 -- 172.21.15.90:0/3485447629 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71000af040 msgr2=0x7f7100164780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.863+0000 7f7106e00640 1 --2- 172.21.15.90:0/3485447629 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71000af040 0x7f7100164780 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f70f000e7e0 tx=0x7f70f000eca0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.864+0000 7f7104e00640 1 -- 172.21.15.90:0/3485447629 reap_dead start 2024-03-24T01:24:40.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.864+0000 7f7106e00640 1 -- 172.21.15.90:0/3485447629 shutdown_connections 2024-03-24T01:24:40.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.864+0000 7f7106e00640 1 -- 172.21.15.90:0/3485447629 >> 172.21.15.90:0/3485447629 conn(0x7f71000b1770 msgr2=0x7f710006b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:40.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.864+0000 7f7106e00640 1 -- 172.21.15.90:0/3485447629 shutdown_connections 2024-03-24T01:24:40.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.864+0000 7f7106e00640 1 -- 172.21.15.90:0/3485447629 wait complete. 2024-03-24T01:24:40.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.864+0000 7f3ff3800640 1 -- 172.21.15.90:0/536219688 >> [v2:172.21.15.120:6808/1045862942,v1:172.21.15.120:6809/1045862942] conn(0x7f3fcc075420 msgr2=0x7f3fcc077860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.864+0000 7f3ff3800640 1 --2- 172.21.15.90:0/536219688 >> [v2:172.21.15.120:6808/1045862942,v1:172.21.15.120:6809/1045862942] conn(0x7f3fcc075420 0x7f3fcc077860 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.866+0000 7f3ff3800640 1 -- 172.21.15.90:0/536219688 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3fcc06bca0 msgr2=0x7f3fcc06e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.866+0000 7f3ff3800640 1 --2- 172.21.15.90:0/536219688 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3fcc06bca0 0x7f3fcc06e160 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3fd8036000 tx=0x7f3fd803c040 comp rx=0 tx=0).stop 2024-03-24T01:24:40.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.866+0000 7f3ff3800640 1 -- 172.21.15.90:0/536219688 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fec0b5ba0 msgr2=0x7f3fec0ccff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.866+0000 7f3ff3800640 1 --2- 172.21.15.90:0/536219688 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fec0b5ba0 0x7f3fec0ccff0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3fe000c750 tx=0x7f3fe000cc10 comp rx=0 tx=0).stop 2024-03-24T01:24:40.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.866+0000 7f3ff1800640 1 -- 172.21.15.90:0/536219688 reap_dead start 2024-03-24T01:24:40.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.866+0000 7f3ff3800640 1 -- 172.21.15.90:0/536219688 shutdown_connections 2024-03-24T01:24:40.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.866+0000 7f3ff3800640 1 -- 172.21.15.90:0/536219688 >> 172.21.15.90:0/536219688 conn(0x7f3fec06c930 msgr2=0x7f3fec0b92f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:40.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.866+0000 7f3ff3800640 1 -- 172.21.15.90:0/536219688 shutdown_connections 2024-03-24T01:24:40.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.867+0000 7f3ff3800640 1 -- 172.21.15.90:0/536219688 wait complete. 2024-03-24T01:24:40.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.870+0000 7fda4da00640 1 -- 172.21.15.90:0/3398845861 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fda480d18d0 con 0x7fda24075250 2024-03-24T01:24:40.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.871+0000 7fda3ea00640 1 -- 172.21.15.90:0/3398845861 <== osd.3 v2:172.21.15.90:6808/2999931983 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fda480d18d0 con 0x7fda24075250 2024-03-24T01:24:40.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.873+0000 7fda4da00640 1 -- 172.21.15.90:0/3398845861 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x7fda24075250 msgr2=0x7fda24077690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.873+0000 7fda4da00640 1 --2- 172.21.15.90:0/3398845861 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x7fda24075250 0x7fda24077690 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.873+0000 7fda4da00640 1 -- 172.21.15.90:0/3398845861 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fda2406bbd0 msgr2=0x7fda2406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.873+0000 7fda4da00640 1 --2- 172.21.15.90:0/3398845861 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fda2406bbd0 0x7fda2406e090 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fda380013b0 tx=0x7fda380011c0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.873+0000 7fda4da00640 1 -- 172.21.15.90:0/3398845861 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda480d10c0 msgr2=0x7fda481e88f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.873+0000 7fda4da00640 1 --2- 172.21.15.90:0/3398845861 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda480d10c0 0x7fda481e88f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fda3000e9d0 tx=0x7fda3000ee90 comp rx=0 tx=0).stop 2024-03-24T01:24:40.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.873+0000 7face5200640 1 -- 172.21.15.90:0/2172615501 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7facb8002dd0 con 0x7facbc0754f0 2024-03-24T01:24:40.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.873+0000 7fda47400640 1 -- 172.21.15.90:0/3398845861 reap_dead start 2024-03-24T01:24:40.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.873+0000 7fda4da00640 1 -- 172.21.15.90:0/3398845861 shutdown_connections 2024-03-24T01:24:40.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.873+0000 7fda4da00640 1 -- 172.21.15.90:0/3398845861 >> 172.21.15.90:0/3398845861 conn(0x7fda4806c9c0 msgr2=0x7fda480cdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:40.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.873+0000 7fda4da00640 1 -- 172.21.15.90:0/3398845861 shutdown_connections 2024-03-24T01:24:40.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.874+0000 7fda4da00640 1 -- 172.21.15.90:0/3398845861 wait complete. 2024-03-24T01:24:40.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.874+0000 7facd6000640 1 -- 172.21.15.90:0/2172615501 <== osd.6 v2:172.21.15.120:6816/1454745630 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7facb8002dd0 con 0x7facbc0754f0 2024-03-24T01:24:40.900 INFO:teuthology.orchestra.run.smithi090.stdout:55834574860 2024-03-24T01:24:40.900 INFO:teuthology.orchestra.run.smithi090.stdout:55834574859 2024-03-24T01:24:40.901 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-24T01:24:40.902 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-24T01:24:40.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.875+0000 7face5200640 1 -- 172.21.15.90:0/2172615501 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x7facbc0754f0 msgr2=0x7facbc077930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.875+0000 7face5200640 1 --2- 172.21.15.90:0/2172615501 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x7facbc0754f0 0x7facbc077930 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.875+0000 7face5200640 1 -- 172.21.15.90:0/2172615501 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7facbc06be00 msgr2=0x7facbc06e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.875+0000 7face5200640 1 --2- 172.21.15.90:0/2172615501 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7facbc06be00 0x7facbc06e2c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7facc8006a90 tx=0x7facc8008040 comp rx=0 tx=0).stop 2024-03-24T01:24:40.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.875+0000 7face5200640 1 -- 172.21.15.90:0/2172615501 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7face015ded0 msgr2=0x7face0155ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.875+0000 7face5200640 1 --2- 172.21.15.90:0/2172615501 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7face015ded0 0x7face0155ff0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7facd800c7b0 tx=0x7facd800cc70 comp rx=0 tx=0).stop 2024-03-24T01:24:40.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.875+0000 7facde800640 1 -- 172.21.15.90:0/2172615501 reap_dead start 2024-03-24T01:24:40.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.875+0000 7face5200640 1 -- 172.21.15.90:0/2172615501 shutdown_connections 2024-03-24T01:24:40.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.875+0000 7face5200640 1 -- 172.21.15.90:0/2172615501 >> 172.21.15.90:0/2172615501 conn(0x7face006c9c0 msgr2=0x7face00d3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:40.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.876+0000 7fbf3e200640 1 -- 172.21.15.90:0/454894854 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbf38073c60 con 0x7fbf180760f0 2024-03-24T01:24:40.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.876+0000 7fbf2b400640 1 -- 172.21.15.90:0/454894854 <== osd.5 v2:172.21.15.120:6824/1739969887 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbf38073c60 con 0x7fbf180760f0 2024-03-24T01:24:40.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.877+0000 7face5200640 1 -- 172.21.15.90:0/2172615501 shutdown_connections 2024-03-24T01:24:40.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.877+0000 7face5200640 1 -- 172.21.15.90:0/2172615501 wait complete. 2024-03-24T01:24:40.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.877+0000 7fbf3e200640 1 -- 172.21.15.90:0/454894854 >> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] conn(0x7fbf180760f0 msgr2=0x7fbf18078590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.877+0000 7fbf3e200640 1 --2- 172.21.15.90:0/454894854 >> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] conn(0x7fbf180760f0 0x7fbf18078590 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.877+0000 7fbf3e200640 1 -- 172.21.15.90:0/454894854 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbf1806bc50 msgr2=0x7fbf1806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.877+0000 7fbf3e200640 1 --2- 172.21.15.90:0/454894854 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbf1806bc50 0x7fbf1806e110 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c009ac0 tx=0x7fbf2c030000 comp rx=0 tx=0).stop 2024-03-24T01:24:40.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.877+0000 7fbf3e200640 1 -- 172.21.15.90:0/454894854 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf381fb990 msgr2=0x7fbf381ec950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.877+0000 7fbf3e200640 1 --2- 172.21.15.90:0/454894854 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf381fb990 0x7fbf381ec950 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbf38062c60 tx=0x7fbf2400f900 comp rx=0 tx=0).stop 2024-03-24T01:24:40.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.878+0000 7fbf3cc00640 1 -- 172.21.15.90:0/454894854 reap_dead start 2024-03-24T01:24:40.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.878+0000 7fbf3e200640 1 -- 172.21.15.90:0/454894854 shutdown_connections 2024-03-24T01:24:40.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.878+0000 7fbf3e200640 1 -- 172.21.15.90:0/454894854 >> 172.21.15.90:0/454894854 conn(0x7fbf380b17c0 msgr2=0x7fbf3814ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:40.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.878+0000 7f2bf8000640 1 -- 172.21.15.90:0/3153072976 --> [v2:172.21.15.120:6800/2150486081,v1:172.21.15.120:6801/2150486081] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2bf0068060 con 0x7f2bd0075450 2024-03-24T01:24:40.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.878+0000 7fbf3e200640 1 -- 172.21.15.90:0/454894854 shutdown_connections 2024-03-24T01:24:40.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.878+0000 7fbf3e200640 1 -- 172.21.15.90:0/454894854 wait complete. 2024-03-24T01:24:40.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.879+0000 7f2bed600640 1 -- 172.21.15.90:0/3153072976 <== osd.4 v2:172.21.15.120:6800/2150486081 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2bf0068060 con 0x7f2bd0075450 2024-03-24T01:24:40.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.880+0000 7f2bf8000640 1 -- 172.21.15.90:0/3153072976 >> [v2:172.21.15.120:6800/2150486081,v1:172.21.15.120:6801/2150486081] conn(0x7f2bd0075450 msgr2=0x7f2bd0077890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.880+0000 7f2bf8000640 1 --2- 172.21.15.90:0/3153072976 >> [v2:172.21.15.120:6800/2150486081,v1:172.21.15.120:6801/2150486081] conn(0x7f2bd0075450 0x7f2bd0077890 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:40.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.880+0000 7f2bf8000640 1 -- 172.21.15.90:0/3153072976 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f2bd006bdd0 msgr2=0x7f2bd006e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.880+0000 7f2bf8000640 1 --2- 172.21.15.90:0/3153072976 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f2bd006bdd0 0x7f2bd006e290 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2be4001030 tx=0x7f2be4009040 comp rx=0 tx=0).stop 2024-03-24T01:24:40.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.880+0000 7f2bf8000640 1 -- 172.21.15.90:0/3153072976 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2bf0152d30 msgr2=0x7f2bf01eed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:40.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.880+0000 7f2bf8000640 1 --2- 172.21.15.90:0/3153072976 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2bf0152d30 0x7f2bf01eed40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc011590 tx=0x7f2bdc011a50 comp rx=0 tx=0).stop 2024-03-24T01:24:40.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.880+0000 7f2bf6000640 1 -- 172.21.15.90:0/3153072976 reap_dead start 2024-03-24T01:24:40.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.881+0000 7f2bf8000640 1 -- 172.21.15.90:0/3153072976 shutdown_connections 2024-03-24T01:24:40.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.881+0000 7f2bf8000640 1 -- 172.21.15.90:0/3153072976 >> 172.21.15.90:0/3153072976 conn(0x7f2bf00b1c80 msgr2=0x7f2bf0156470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:40.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.881+0000 7f2bf8000640 1 -- 172.21.15.90:0/3153072976 shutdown_connections 2024-03-24T01:24:40.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:40.881+0000 7f2bf8000640 1 -- 172.21.15.90:0/3153072976 wait complete. 2024-03-24T01:24:40.909 INFO:teuthology.orchestra.run.smithi090.stdout:55834574860 2024-03-24T01:24:40.909 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-24T01:24:40.910 INFO:teuthology.orchestra.run.smithi090.stdout:55834574859 2024-03-24T01:24:40.910 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-24T01:24:40.911 INFO:teuthology.orchestra.run.smithi090.stdout:55834574859 2024-03-24T01:24:40.911 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-24T01:24:40.912 INFO:teuthology.orchestra.run.smithi090.stdout:55834574859 2024-03-24T01:24:40.912 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-24T01:24:40.913 INFO:teuthology.orchestra.run.smithi090.stdout:55834574859 2024-03-24T01:24:40.914 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-24T01:24:40.914 INFO:teuthology.orchestra.run.smithi090.stdout:55834574860 2024-03-24T01:24:40.915 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-24T01:24:41.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.236+0000 7f7895400640 1 Processor -- start 2024-03-24T01:24:41.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.237+0000 7f7895400640 1 -- start start 2024-03-24T01:24:41.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.242+0000 7f7895400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78880e3160 0x7f78880e7780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.243+0000 7f7895400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f788803c0a0 0x7f7888058050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.243+0000 7f788fe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78880e3160 0x7f78880e7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.243+0000 7f788fe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78880e3160 0x7f78880e7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.90:50150/0 (socket says 172.21.15.90:50150) 2024-03-24T01:24:41.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.244+0000 7f7894a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f7888053cb0 0x7f7888054120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51188/0 (socket says 172.21.15.90:51188) 2024-03-24T01:24:41.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.244+0000 7f788f400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f788803c0a0 0x7f7888058050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.244+0000 7f788f400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f788803c0a0 0x7f7888058050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60084/0 (socket says 172.21.15.90:60084) 2024-03-24T01:24:41.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.244+0000 7f788fe00640 1 -- 172.21.15.90:0/3962909663 learned_addr learned my addr 172.21.15.90:0/3962909663 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:41.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.245+0000 7f7895400640 1 -- 172.21.15.90:0/3962909663 --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7888058590 con 0x7f7888053cb0 2024-03-24T01:24:41.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.245+0000 7f7895400640 1 -- 172.21.15.90:0/3962909663 --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7888007b70 con 0x7f788803c0a0 2024-03-24T01:24:41.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.245+0000 7f7895400640 1 -- 172.21.15.90:0/3962909663 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7888058810 con 0x7f78880e3160 2024-03-24T01:24:41.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.246+0000 7f788ea00640 1 -- 172.21.15.90:0/3962909663 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3064229270 0 0) 0x7f7884001110 con 0x7f7888053cb0 2024-03-24T01:24:41.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.246+0000 7f788ea00640 1 -- 172.21.15.90:0/3962909663 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7878003520 con 0x7f7888053cb0 2024-03-24T01:24:41.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.246+0000 7f788ea00640 1 -- 172.21.15.90:0/3962909663 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2702767611 0 0) 0x7f7878003520 con 0x7f7888053cb0 2024-03-24T01:24:41.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.246+0000 7f788ea00640 1 -- 172.21.15.90:0/3962909663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78880e3160 msgr2=0x7f78880e7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:41.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.246+0000 7f788ea00640 1 --2- 172.21.15.90:0/3962909663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78880e3160 0x7f78880e7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.246+0000 7f788ea00640 1 -- 172.21.15.90:0/3962909663 >> v2:172.21.15.90:3300/0 conn(0x7f788803c0a0 msgr2=0x7f7888058050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:41.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.246+0000 7f788ea00640 1 --2- 172.21.15.90:0/3962909663 >> v2:172.21.15.90:3300/0 conn(0x7f788803c0a0 0x7f7888058050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.246+0000 7f788ea00640 1 -- 172.21.15.90:0/3962909663 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7888058af0 con 0x7f7888053cb0 2024-03-24T01:24:41.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.246+0000 7f788ea00640 1 -- 172.21.15.90:0/3962909663 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f7884001840 con 0x7f7888053cb0 2024-03-24T01:24:41.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.246+0000 7f788ea00640 1 -- 172.21.15.90:0/3962909663 >> v1:172.21.15.90:6789/0 conn(0x7f7888053cb0 legacy=0x7f7888054120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.246+0000 7f788ea00640 1 --2- 172.21.15.90:0/3962909663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78780039b0 0x7f7878003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.246+0000 7f788ea00640 1 --2- 172.21.15.90:0/3962909663 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7878004800 0x7f7878004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.246+0000 7f788ea00640 1 --2- 172.21.15.90:0/3962909663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78780051c0 0x7f787800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.246+0000 7f788ea00640 1 -- 172.21.15.90:0/3962909663 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7888058810 con 0x7f7878004800 2024-03-24T01:24:41.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.246+0000 7f788ea00640 1 -- 172.21.15.90:0/3962909663 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f787800c030 con 0x7f78780051c0 2024-03-24T01:24:41.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.246+0000 7f788ea00640 1 -- 172.21.15.90:0/3962909663 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f787800c230 con 0x7f78780039b0 2024-03-24T01:24:41.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.247+0000 7f788f400640 1 --2- 172.21.15.90:0/3962909663 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7878004800 0x7f7878004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.247+0000 7f788fe00640 1 --2- 172.21.15.90:0/3962909663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78780039b0 0x7f7878003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.247+0000 7f7894a00640 1 --2- 172.21.15.90:0/3962909663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78780051c0 0x7f787800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.248+0000 7f788fe00640 1 -- 172.21.15.90:0/3962909663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78780051c0 msgr2=0x7f787800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:41.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.248+0000 7f788fe00640 1 --2- 172.21.15.90:0/3962909663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78780051c0 0x7f787800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.248+0000 7f788fe00640 1 -- 172.21.15.90:0/3962909663 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7878004800 msgr2=0x7f7878004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:41.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.248+0000 7f788fe00640 1 --2- 172.21.15.90:0/3962909663 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7878004800 0x7f7878004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.248+0000 7f788fe00640 1 -- 172.21.15.90:0/3962909663 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7888058af0 con 0x7f78780039b0 2024-03-24T01:24:41.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.251+0000 7fcc33800640 1 Processor -- start 2024-03-24T01:24:41.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.251+0000 7fcc33800640 1 -- start start 2024-03-24T01:24:41.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.252+0000 7f788fe00640 1 --2- 172.21.15.90:0/3962909663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78780039b0 0x7f7878003dd0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f788800e300 tx=0x7f7878005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed90c50940b0556b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:41.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.252+0000 7fcff8a00640 1 Processor -- start 2024-03-24T01:24:41.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.253+0000 7f788ea00640 1 -- 172.21.15.90:0/3962909663 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787c035020 con 0x7f78780039b0 2024-03-24T01:24:41.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.253+0000 7f788ea00640 1 -- 172.21.15.90:0/3962909663 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f787c031600 con 0x7f78780039b0 2024-03-24T01:24:41.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.253+0000 7f788ea00640 1 -- 172.21.15.90:0/3962909663 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787c00f040 con 0x7f78780039b0 2024-03-24T01:24:41.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.252+0000 7f6eea000640 1 Processor -- start 2024-03-24T01:24:41.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.254+0000 7f7895400640 1 -- 172.21.15.90:0/3962909663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78780039b0 msgr2=0x7f7878003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.254+0000 7f7895400640 1 --2- 172.21.15.90:0/3962909663 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78780039b0 0x7f7878003dd0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f788800e300 tx=0x7f7878005830 comp rx=0 tx=0).stop 2024-03-24T01:24:41.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.254+0000 7f7894a00640 1 -- 172.21.15.90:0/3962909663 reap_dead start 2024-03-24T01:24:41.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.254+0000 7f7895400640 1 -- 172.21.15.90:0/3962909663 shutdown_connections 2024-03-24T01:24:41.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.254+0000 7f7895400640 1 -- 172.21.15.90:0/3962909663 >> 172.21.15.90:0/3962909663 conn(0x7f7888058ee0 msgr2=0x7f78880592c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:41.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.255+0000 7f7895400640 1 -- 172.21.15.90:0/3962909663 shutdown_connections 2024-03-24T01:24:41.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.253+0000 7f6eea000640 1 -- start start 2024-03-24T01:24:41.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.255+0000 7f6eea000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ee40d0b50 0x7f6ee40d0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.255+0000 7f6eea000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6ee40b2e40 0x7f6ee40d28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.255+0000 7f6eea000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ee40d20e0 con 0x7f6ee415dfd0 2024-03-24T01:24:41.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.255+0000 7f6eea000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ee40d1470 con 0x7f6ee40b2e40 2024-03-24T01:24:41.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.255+0000 7f6eea000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee40ced70 con 0x7f6ee40d0b50 2024-03-24T01:24:41.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.253+0000 7fcc33800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc24033990 0x7fcc240543c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.253+0000 7fcc33800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcc24007190 0x7fcc24037c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.253+0000 7fcc33800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcc240075b0 0x7fcc24038180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.253+0000 7fcc33800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc24108720 con 0x7fcc24007190 2024-03-24T01:24:41.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.253+0000 7fcc33800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc24053e00 con 0x7fcc240075b0 2024-03-24T01:24:41.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.253+0000 7fcc33800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc240540e0 con 0x7fcc24033990 2024-03-24T01:24:41.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.253+0000 7fcff8a00640 1 -- start start 2024-03-24T01:24:41.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.255+0000 7fcff8a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcff415de60 0x7fcff40d1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.255+0000 7fcff8a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcff40b97b0 0x7fcff40cead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.255+0000 7fcff8a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcff40d2790 con 0x7fcff40d3830 2024-03-24T01:24:41.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.255+0000 7fcff8a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcff40b7eb0 con 0x7fcff40b97b0 2024-03-24T01:24:41.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.255+0000 7fcff8a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcff40d2a70 con 0x7fcff415de60 2024-03-24T01:24:41.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.255+0000 7fcff2800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fcff40d3830 0x7fcff40b5780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51202/0 (socket says 172.21.15.90:51202) 2024-03-24T01:24:41.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.255+0000 7fcff2800640 1 -- 172.21.15.90:0/1304873799 learned_addr learned my addr 172.21.15.90:0/1304873799 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:41.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.254+0000 7fcc2be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcc24007190 0x7fcc24037c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.256+0000 7fcc2be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcc24007190 0x7fcc24037c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60094/0 (socket says 172.21.15.90:60094) 2024-03-24T01:24:41.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.256+0000 7fcc2be00640 1 -- 172.21.15.90:0/811586692 learned_addr learned my addr 172.21.15.90:0/811586692 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:41.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.255+0000 7fcc31800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcc240075b0 0x7fcc24038180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.256+0000 7f6ee3400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ee40d0b50 0x7f6ee40d0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.256+0000 7fcc2be00640 1 -- 172.21.15.90:0/811586692 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc24033990 msgr2=0x7fcc240543c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:41.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.256+0000 7f6ee3e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6ee40b2e40 0x7f6ee40d28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.256+0000 7f6ee3e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6ee40b2e40 0x7f6ee40d28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.90:39144/0 (socket says 172.21.15.90:39144) 2024-03-24T01:24:41.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.256+0000 7f6ee3e00640 1 -- 172.21.15.90:0/823029301 learned_addr learned my addr 172.21.15.90:0/823029301 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:41.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.257+0000 7fcff1e00640 1 --2- 172.21.15.90:0/1304873799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcff415de60 0x7fcff40d1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.256+0000 7fcff0a00640 1 -- 172.21.15.90:0/1304873799 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4247168603 0 0) 0x7fcff40d2790 con 0x7fcff40d3830 2024-03-24T01:24:41.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.257+0000 7fcff1400640 1 --2- 172.21.15.90:0/1304873799 >> v2:172.21.15.120:3300/0 conn(0x7fcff40b97b0 0x7fcff40cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.257+0000 7fcff0a00640 1 -- 172.21.15.90:0/1304873799 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfd8003520 con 0x7fcff40d3830 2024-03-24T01:24:41.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.255+0000 7fcc30e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc24033990 0x7fcc240543c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.257+0000 7f440dc00640 1 Processor -- start 2024-03-24T01:24:41.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.255+0000 7f7895400640 1 -- 172.21.15.90:0/3962909663 wait complete. 2024-03-24T01:24:41.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.257+0000 7f440dc00640 1 -- start start 2024-03-24T01:24:41.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.257+0000 7fcc2be00640 1 --2- 172.21.15.90:0/811586692 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc24033990 0x7fcc240543c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.257+0000 7fcc2be00640 1 -- 172.21.15.90:0/811586692 >> v2:172.21.15.120:3300/0 conn(0x7fcc240075b0 msgr2=0x7fcc24038180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:41.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.257+0000 7fcc2be00640 1 --2- 172.21.15.90:0/811586692 >> v2:172.21.15.120:3300/0 conn(0x7fcc240075b0 0x7fcc24038180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.257+0000 7fcc2be00640 1 -- 172.21.15.90:0/811586692 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc24006c70 con 0x7fcc24007190 2024-03-24T01:24:41.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.257+0000 7f6ee3e00640 1 -- 172.21.15.90:0/823029301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ee40d0b50 msgr2=0x7f6ee40d0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:41.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.257+0000 7fcc31800640 1 --2- 172.21.15.90:0/811586692 >> v2:172.21.15.120:3300/0 conn(0x7fcc240075b0 0x7fcc24038180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T01:24:41.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.257+0000 7f6ee3e00640 1 --2- 172.21.15.90:0/823029301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ee40d0b50 0x7f6ee40d0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.257+0000 7f6ee3e00640 1 -- 172.21.15.90:0/823029301 >> v1:172.21.15.90:6789/0 conn(0x7f6ee415dfd0 legacy=0x7f6ee41622b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:41.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.257+0000 7fcc30e00640 1 --2- 172.21.15.90:0/811586692 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc24033990 0x7fcc240543c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:41.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.257+0000 7fcff0a00640 1 -- 172.21.15.90:0/1304873799 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 833228750 0 0) 0x7fcfd8003520 con 0x7fcff40d3830 2024-03-24T01:24:41.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.257+0000 7fcff0a00640 1 -- 172.21.15.90:0/1304873799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcff415de60 msgr2=0x7fcff40d1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:41.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.257+0000 7fcff0a00640 1 --2- 172.21.15.90:0/1304873799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcff415de60 0x7fcff40d1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.257+0000 7fcff0a00640 1 -- 172.21.15.90:0/1304873799 >> v2:172.21.15.120:3300/0 conn(0x7fcff40b97b0 msgr2=0x7fcff40cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:41.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.257+0000 7fcff0a00640 1 --2- 172.21.15.90:0/1304873799 >> v2:172.21.15.120:3300/0 conn(0x7fcff40b97b0 0x7fcff40cead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.257+0000 7fcff0a00640 1 -- 172.21.15.90:0/1304873799 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff40d2d50 con 0x7fcff40d3830 2024-03-24T01:24:41.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.257+0000 7fcff0a00640 1 -- 172.21.15.90:0/1304873799 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7fcfec0019f0 con 0x7fcff40d3830 2024-03-24T01:24:41.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.257+0000 7fcff0a00640 1 -- 172.21.15.90:0/1304873799 >> v1:172.21.15.90:6789/0 conn(0x7fcff40d3830 legacy=0x7fcff40b5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.257+0000 7fcff0a00640 1 --2- 172.21.15.90:0/1304873799 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfd80039b0 0x7fcfd8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.257+0000 7fcff0a00640 1 --2- 172.21.15.90:0/1304873799 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfd8004800 0x7fcfd8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.257+0000 7fcff0a00640 1 --2- 172.21.15.90:0/1304873799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfd80051c0 0x7fcfd800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.257+0000 7fcff0a00640 1 -- 172.21.15.90:0/1304873799 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff40b7eb0 con 0x7fcfd8004800 2024-03-24T01:24:41.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.257+0000 7fcff0a00640 1 -- 172.21.15.90:0/1304873799 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff40d2a70 con 0x7fcfd80039b0 2024-03-24T01:24:41.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.257+0000 7fcff0a00640 1 -- 172.21.15.90:0/1304873799 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd800bfe0 con 0x7fcfd80051c0 2024-03-24T01:24:41.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.258+0000 7f6ee3e00640 1 -- 172.21.15.90:0/823029301 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee40cf050 con 0x7f6ee40b2e40 2024-03-24T01:24:41.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.258+0000 7fcff1e00640 1 --2- 172.21.15.90:0/1304873799 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfd80039b0 0x7fcfd8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.259+0000 7fcff1400640 1 --2- 172.21.15.90:0/1304873799 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfd8004800 0x7fcfd8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.259+0000 7fcff2800640 1 --2- 172.21.15.90:0/1304873799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfd80051c0 0x7fcfd800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.259+0000 7f440dc00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4408071990 0x7f4408071e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.259+0000 7f440dc00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f44080b2890 0x7f44080b2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.259+0000 7fcc2be00640 1 --2- 172.21.15.90:0/811586692 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc24007190 0x7fcc24037c40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fcc24051470 tx=0x7fcc1802b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5362cb2e2888a16d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:41.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.259+0000 7fcff1e00640 1 -- 172.21.15.90:0/1304873799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfd80051c0 msgr2=0x7fcfd800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:41.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.260+0000 7f6ee3e00640 1 --2- 172.21.15.90:0/823029301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ee40b2e40 0x7f6ee40d28a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6ee40d22b0 tx=0x7f6ed402b7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8000f8abff89f2ea server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:41.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.259+0000 7f440dc00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44080b3530 con 0x7f4408161240 2024-03-24T01:24:41.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.259+0000 7f440dc00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4408072340 con 0x7f4408071990 2024-03-24T01:24:41.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.259+0000 7f440dc00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44080acff0 con 0x7f44080b2890 2024-03-24T01:24:41.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.260+0000 7f6ee2000640 1 -- 172.21.15.90:0/823029301 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed4038070 con 0x7f6ee40b2e40 2024-03-24T01:24:41.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.260+0000 7f4406000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4408071990 0x7f4408071e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.260+0000 7f4406000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4408071990 0x7f4408071e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60116/0 (socket says 172.21.15.90:60116) 2024-03-24T01:24:41.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.260+0000 7f4406000640 1 -- 172.21.15.90:0/4100261528 learned_addr learned my addr 172.21.15.90:0/4100261528 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:41.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.260+0000 7f4407400640 1 --2- 172.21.15.90:0/4100261528 >> v2:172.21.15.120:3300/0 conn(0x7f44080b2890 0x7f44080b2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.260+0000 7f6ee2000640 1 -- 172.21.15.90:0/823029301 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed4031520 con 0x7f6ee40b2e40 2024-03-24T01:24:41.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.260+0000 7f6ee2000640 1 -- 172.21.15.90:0/823029301 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed40318f0 con 0x7f6ee40b2e40 2024-03-24T01:24:41.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.261+0000 7f7895400640 1 Processor -- start 2024-03-24T01:24:41.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.260+0000 7fcff1e00640 1 --2- 172.21.15.90:0/1304873799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfd80051c0 0x7fcfd800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.260+0000 7fcff1e00640 1 -- 172.21.15.90:0/1304873799 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfd8004800 msgr2=0x7fcfd8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:41.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.260+0000 7fcff1e00640 1 --2- 172.21.15.90:0/1304873799 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfd8004800 0x7fcfd8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.260+0000 7fcff1e00640 1 -- 172.21.15.90:0/1304873799 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff40d2d50 con 0x7fcfd80039b0 2024-03-24T01:24:41.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.260+0000 7fcff1400640 1 --2- 172.21.15.90:0/1304873799 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfd8004800 0x7fcfd8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:41.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.260+0000 7fcff2800640 1 --2- 172.21.15.90:0/1304873799 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcfd80051c0 0x7fcfd800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T01:24:41.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.261+0000 7f7895400640 1 -- start start 2024-03-24T01:24:41.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.261+0000 7f6eea000640 1 -- 172.21.15.90:0/823029301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ee40b2e40 msgr2=0x7f6ee40d28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.261+0000 7f4406000640 1 -- 172.21.15.90:0/4100261528 >> v2:172.21.15.120:3300/0 conn(0x7f44080b2890 msgr2=0x7f44080b2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:41.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.261+0000 7f4406000640 1 --2- 172.21.15.90:0/4100261528 >> v2:172.21.15.120:3300/0 conn(0x7f44080b2890 0x7f44080b2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.261+0000 7f4406000640 1 -- 172.21.15.90:0/4100261528 >> v1:172.21.15.90:6789/0 conn(0x7f4408161240 legacy=0x7f44080ac8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:41.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.261+0000 7f4406000640 1 -- 172.21.15.90:0/4100261528 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44080b3810 con 0x7f4408071990 2024-03-24T01:24:41.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.261+0000 7f4407400640 1 --2- 172.21.15.90:0/4100261528 >> v2:172.21.15.120:3300/0 conn(0x7f44080b2890 0x7f44080b2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:24:41.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.261+0000 7f6eea000640 1 --2- 172.21.15.90:0/823029301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ee40b2e40 0x7f6ee40d28a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6ee40d22b0 tx=0x7f6ed402b7c0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.261+0000 7f6eea000640 1 -- 172.21.15.90:0/823029301 shutdown_connections 2024-03-24T01:24:41.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.261+0000 7f6eea000640 1 --2- 172.21.15.90:0/823029301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ee40b2e40 0x7f6ee40d28a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.262+0000 7f6eea000640 1 --2- 172.21.15.90:0/823029301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ee40d0b50 0x7f6ee40d0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.262+0000 7f6eea000640 1 -- 172.21.15.90:0/823029301 >> 172.21.15.90:0/823029301 conn(0x7f6ee4072c10 msgr2=0x7f6ee4073020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:41.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.262+0000 7f6eea000640 1 -- 172.21.15.90:0/823029301 shutdown_connections 2024-03-24T01:24:41.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.261+0000 7fcc2b400640 1 -- 172.21.15.90:0/811586692 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc18034070 con 0x7fcc24007190 2024-03-24T01:24:41.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.261+0000 7fcc2b400640 1 -- 172.21.15.90:0/811586692 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc18030850 con 0x7fcc24007190 2024-03-24T01:24:41.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.261+0000 7fcc2b400640 1 -- 172.21.15.90:0/811586692 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc18030c20 con 0x7fcc24007190 2024-03-24T01:24:41.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.262+0000 7fcc33800640 1 -- 172.21.15.90:0/811586692 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc24007190 msgr2=0x7fcc24037c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.262+0000 7fcc33800640 1 --2- 172.21.15.90:0/811586692 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc24007190 0x7fcc24037c40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fcc24051470 tx=0x7fcc1802b990 comp rx=0 tx=0).stop 2024-03-24T01:24:41.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.262+0000 7fcc33800640 1 -- 172.21.15.90:0/811586692 shutdown_connections 2024-03-24T01:24:41.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.262+0000 7fcc33800640 1 --2- 172.21.15.90:0/811586692 >> v2:172.21.15.120:3300/0 conn(0x7fcc240075b0 0x7fcc24038180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.262+0000 7fcc33800640 1 --2- 172.21.15.90:0/811586692 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc24007190 0x7fcc24037c40 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.262+0000 7fcc33800640 1 --2- 172.21.15.90:0/811586692 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc24033990 0x7fcc240543c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.262+0000 7f7895400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78780039b0 0x7f78880d4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.262+0000 7f7895400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f788804c320 0x7f78880d79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.262+0000 7f7895400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78880d4ac0 0x7f78880d4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.262+0000 7f7895400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78880e3540 con 0x7f788804c320 2024-03-24T01:24:41.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.262+0000 7f7895400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7888006640 con 0x7f78880d4ac0 2024-03-24T01:24:41.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.262+0000 7f7895400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78880e7780 con 0x7f78780039b0 2024-03-24T01:24:41.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.263+0000 7f788f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f788804c320 0x7f78880d79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.263+0000 7fcc33800640 1 -- 172.21.15.90:0/811586692 >> 172.21.15.90:0/811586692 conn(0x7fcc24009fb0 msgr2=0x7fcc2400a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:41.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.263+0000 7fcc33800640 1 -- 172.21.15.90:0/811586692 shutdown_connections 2024-03-24T01:24:41.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.263+0000 7fcc33800640 1 -- 172.21.15.90:0/811586692 wait complete. 2024-03-24T01:24:41.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.263+0000 7f6eea000640 1 -- 172.21.15.90:0/823029301 wait complete. 2024-03-24T01:24:41.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.263+0000 7fcff1e00640 1 --2- 172.21.15.90:0/1304873799 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfd80039b0 0x7fcfd8003dd0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcff40d0ce0 tx=0x7fcfd8005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aae4822e54ae296d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:41.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.263+0000 7f4406000640 1 --2- 172.21.15.90:0/4100261528 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4408071990 0x7f4408071e00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f440806e710 tx=0x7f43f802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef8304614720f080 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:41.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.263+0000 7f788fe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78780039b0 0x7f78880d4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.264+0000 7f7894a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78880d4ac0 0x7f78880d4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.264+0000 7f788f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f788804c320 0x7f78880d79f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60124/0 (socket says 172.21.15.90:60124) 2024-03-24T01:24:41.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.264+0000 7f788f400640 1 -- 172.21.15.90:0/17958593 learned_addr learned my addr 172.21.15.90:0/17958593 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:41.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.264+0000 7fcff0a00640 1 -- 172.21.15.90:0/1304873799 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe400b550 con 0x7fcfd80039b0 2024-03-24T01:24:41.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.264+0000 7fcff0a00640 1 -- 172.21.15.90:0/1304873799 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfe4034630 con 0x7fcfd80039b0 2024-03-24T01:24:41.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.264+0000 7f4405600640 1 -- 172.21.15.90:0/4100261528 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f802e070 con 0x7f4408071990 2024-03-24T01:24:41.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.264+0000 7f4405600640 1 -- 172.21.15.90:0/4100261528 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f8037070 con 0x7f4408071990 2024-03-24T01:24:41.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.264+0000 7f4405600640 1 -- 172.21.15.90:0/4100261528 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f8031a00 con 0x7f4408071990 2024-03-24T01:24:41.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.264+0000 7fcff0a00640 1 -- 172.21.15.90:0/1304873799 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe4034a00 con 0x7fcfd80039b0 2024-03-24T01:24:41.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.264+0000 7f788f400640 1 -- 172.21.15.90:0/17958593 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78780039b0 msgr2=0x7f78880d4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.264+0000 7f788f400640 1 --2- 172.21.15.90:0/17958593 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78780039b0 0x7f78880d4500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.264+0000 7f788f400640 1 -- 172.21.15.90:0/17958593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78880d4ac0 msgr2=0x7f78880d4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.264+0000 7f788f400640 1 --2- 172.21.15.90:0/17958593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78880d4ac0 0x7f78880d4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.264+0000 7f788f400640 1 -- 172.21.15.90:0/17958593 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f788804c7a0 con 0x7f788804c320 2024-03-24T01:24:41.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.265+0000 7f7894a00640 1 --2- 172.21.15.90:0/17958593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78880d4ac0 0x7f78880d4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:41.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.265+0000 7f6eea000640 1 Processor -- start 2024-03-24T01:24:41.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.265+0000 7f440dc00640 1 -- 172.21.15.90:0/4100261528 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4408071990 msgr2=0x7f4408071e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.264+0000 7f176c200640 1 Processor -- start 2024-03-24T01:24:41.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.265+0000 7f176c200640 1 -- start start 2024-03-24T01:24:41.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.265+0000 7f788f400640 1 --2- 172.21.15.90:0/17958593 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f788804c320 0x7f78880d79f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7890050670 tx=0x7f7890068b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:41.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.265+0000 7fcc33800640 1 Processor -- start 2024-03-24T01:24:41.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.265+0000 7fcff8a00640 1 -- 172.21.15.90:0/1304873799 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfd80039b0 msgr2=0x7fcfd8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.265+0000 7fcff8a00640 1 --2- 172.21.15.90:0/1304873799 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfd80039b0 0x7fcfd8003dd0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcff40d0ce0 tx=0x7fcfd8005830 comp rx=0 tx=0).stop 2024-03-24T01:24:41.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.265+0000 7f7883e00640 1 -- 172.21.15.90:0/17958593 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789009a070 con 0x7f788804c320 2024-03-24T01:24:41.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.266+0000 7fcff2800640 1 -- 172.21.15.90:0/1304873799 reap_dead start 2024-03-24T01:24:41.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.266+0000 7f6eea000640 1 -- start start 2024-03-24T01:24:41.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.266+0000 7f7883e00640 1 -- 172.21.15.90:0/17958593 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7890074110 con 0x7f788804c320 2024-03-24T01:24:41.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.266+0000 7f7883e00640 1 -- 172.21.15.90:0/17958593 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789009a070 con 0x7f788804c320 2024-03-24T01:24:41.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.266+0000 7fcc33800640 1 -- start start 2024-03-24T01:24:41.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.266+0000 7fcff8a00640 1 -- 172.21.15.90:0/1304873799 shutdown_connections 2024-03-24T01:24:41.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.266+0000 7fcff8a00640 1 -- 172.21.15.90:0/1304873799 >> 172.21.15.90:0/1304873799 conn(0x7fcff4072c10 msgr2=0x7fcff4073020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:41.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.265+0000 7f440dc00640 1 --2- 172.21.15.90:0/4100261528 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4408071990 0x7f4408071e00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f440806e710 tx=0x7f43f802b880 comp rx=0 tx=0).stop 2024-03-24T01:24:41.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.266+0000 7f7895400640 1 -- 172.21.15.90:0/17958593 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78880dff70 con 0x7f788804c320 2024-03-24T01:24:41.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.267+0000 7f176c200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f176406fa80 0x7f176406ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.267+0000 7f176c200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1764159d10 0x7f17640b4120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.267+0000 7f176c200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f176406e470 con 0x7f17640aefb0 2024-03-24T01:24:41.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.267+0000 7f176c200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f176415a160 con 0x7f1764159d10 2024-03-24T01:24:41.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.267+0000 7f176c200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17640b2ba0 con 0x7f176406fa80 2024-03-24T01:24:41.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.267+0000 7fcff8a00640 1 -- 172.21.15.90:0/1304873799 shutdown_connections 2024-03-24T01:24:41.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.267+0000 7f440dc00640 1 -- 172.21.15.90:0/4100261528 shutdown_connections 2024-03-24T01:24:41.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.267+0000 7f440dc00640 1 --2- 172.21.15.90:0/4100261528 >> v2:172.21.15.120:3300/0 conn(0x7f44080b2890 0x7f44080b2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.267+0000 7f440dc00640 1 --2- 172.21.15.90:0/4100261528 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4408071990 0x7f4408071e00 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.267+0000 7f440dc00640 1 -- 172.21.15.90:0/4100261528 >> 172.21.15.90:0/4100261528 conn(0x7f44080b1cd0 msgr2=0x7f44080cf7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:41.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.267+0000 7f440dc00640 1 -- 172.21.15.90:0/4100261528 shutdown_connections 2024-03-24T01:24:41.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.267+0000 7fcc33800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc24007190 0x7fcc240f4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.267+0000 7f440dc00640 1 -- 172.21.15.90:0/4100261528 wait complete. 2024-03-24T01:24:41.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.268+0000 7f6eea000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ee40b2e40 0x7f6ee4156d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.268+0000 7eff15600640 1 Processor -- start 2024-03-24T01:24:41.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.268+0000 7fcc33800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc24033990 0x7fcc24107cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.268+0000 7f6eea000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ee40d0b50 0x7f6ee40c92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.268+0000 7fcc33800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc240f5a60 0x7fcc24047000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.268+0000 7f6eea000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ee415dfd0 0x7f6ee40c9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.268+0000 7f6eea000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee40b6d80 con 0x7f6ee40d0b50 2024-03-24T01:24:41.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.268+0000 7f6eea000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee40cf330 con 0x7f6ee415dfd0 2024-03-24T01:24:41.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.268+0000 7f6eea000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee40b6f00 con 0x7f6ee40b2e40 2024-03-24T01:24:41.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.268+0000 7f6ee3400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ee40b2e40 0x7f6ee4156d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.268+0000 7f7895400640 1 -- 172.21.15.90:0/17958593 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f788804ef80 con 0x7f788804c320 2024-03-24T01:24:41.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.268+0000 7eff15600640 1 -- start start 2024-03-24T01:24:41.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.268+0000 7fcc30e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc24007190 0x7fcc240f4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.269+0000 7fcc30e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc24007190 0x7fcc240f4060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.90:39194/0 (socket says 172.21.15.90:39194) 2024-03-24T01:24:41.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.269+0000 7fcc33800640 1 -- 172.21.15.90:0/48137839 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc24038840 con 0x7fcc24033990 2024-03-24T01:24:41.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.269+0000 7fcc33800640 1 -- 172.21.15.90:0/48137839 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc240386c0 con 0x7fcc24007190 2024-03-24T01:24:41.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.269+0000 7fcc33800640 1 -- 172.21.15.90:0/48137839 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc241054f0 con 0x7fcc240f5a60 2024-03-24T01:24:41.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.269+0000 7fcc30e00640 1 -- 172.21.15.90:0/48137839 learned_addr learned my addr 172.21.15.90:0/48137839 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:41.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.270+0000 7f7895400640 1 -- 172.21.15.90:0/17958593 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7888005390 con 0x7f788804c320 2024-03-24T01:24:41.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.270+0000 7f6ee3400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ee40b2e40 0x7f6ee4156d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.90:50230/0 (socket says 172.21.15.90:50230) 2024-03-24T01:24:41.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.270+0000 7f6ee3400640 1 -- 172.21.15.90:0/4160758613 learned_addr learned my addr 172.21.15.90:0/4160758613 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:41.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.270+0000 7f6ee2a00640 1 --2- 172.21.15.90:0/4160758613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ee40d0b50 0x7f6ee40c92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.270+0000 7f6ee3e00640 1 --2- 172.21.15.90:0/4160758613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ee415dfd0 0x7f6ee40c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.267+0000 7fcff8a00640 1 -- 172.21.15.90:0/1304873799 wait complete. 2024-03-24T01:24:41.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.270+0000 7f440dc00640 1 Processor -- start 2024-03-24T01:24:41.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.270+0000 7f440dc00640 1 -- start start 2024-03-24T01:24:41.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.267+0000 7f1769800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f176406fa80 0x7f176406ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.267+0000 7f1769800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f176406fa80 0x7f176406ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.90:39178/0 (socket says 172.21.15.90:39178) 2024-03-24T01:24:41.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.267+0000 7f1769800640 1 -- 172.21.15.90:0/1566221147 learned_addr learned my addr 172.21.15.90:0/1566221147 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:41.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.268+0000 7f1769800640 1 -- 172.21.15.90:0/1566221147 >> v2:172.21.15.90:3300/0 conn(0x7f1764159d10 msgr2=0x7f17640b4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:41.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.268+0000 7f1769800640 1 --2- 172.21.15.90:0/1566221147 >> v2:172.21.15.90:3300/0 conn(0x7f1764159d10 0x7f17640b4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.268+0000 7f1769800640 1 -- 172.21.15.90:0/1566221147 >> v1:172.21.15.120:6789/0 conn(0x7f17640aefb0 legacy=0x7f176415d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.268+0000 7f1769800640 1 -- 172.21.15.90:0/1566221147 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17640b2e80 con 0x7f176406fa80 2024-03-24T01:24:41.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.270+0000 7f1769800640 1 --2- 172.21.15.90:0/1566221147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f176406fa80 0x7f176406ef10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1764072ba0 tx=0x7f1758034410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e06f603bde996cc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:41.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.271+0000 7f6ee3400640 1 -- 172.21.15.90:0/4160758613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ee415dfd0 msgr2=0x7f6ee40c9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.271+0000 7f6ee3400640 1 --2- 172.21.15.90:0/4160758613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ee415dfd0 0x7f6ee40c9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.271+0000 7f6ee3400640 1 -- 172.21.15.90:0/4160758613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ee40d0b50 msgr2=0x7f6ee40c92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.271+0000 7f6ee3400640 1 --2- 172.21.15.90:0/4160758613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ee40d0b50 0x7f6ee40c92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.269+0000 7fcc31800640 1 --2- 172.21.15.90:0/48137839 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc240f5a60 0x7fcc24047000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.270+0000 7fcc2be00640 1 --2- 172.21.15.90:0/48137839 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc24033990 0x7fcc24107cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.270+0000 7fcc2be00640 1 -- 172.21.15.90:0/48137839 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc240f5a60 msgr2=0x7fcc24047000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.271+0000 7f6ee3400640 1 -- 172.21.15.90:0/4160758613 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee4157740 con 0x7f6ee40b2e40 2024-03-24T01:24:41.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.270+0000 7fcc2be00640 1 --2- 172.21.15.90:0/48137839 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc240f5a60 0x7fcc24047000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.270+0000 7fcc2be00640 1 -- 172.21.15.90:0/48137839 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc24007190 msgr2=0x7fcc240f4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.270+0000 7fcc2be00640 1 --2- 172.21.15.90:0/48137839 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc24007190 0x7fcc240f4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.270+0000 7fcc2be00640 1 -- 172.21.15.90:0/48137839 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc2404cdf0 con 0x7fcc24033990 2024-03-24T01:24:41.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.271+0000 7eff15600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7eff1006ee00 0x7eff1006f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.271+0000 7f1763e00640 1 -- 172.21.15.90:0/1566221147 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1758039710 con 0x7f176406fa80 2024-03-24T01:24:41.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.271+0000 7f1763e00640 1 -- 172.21.15.90:0/1566221147 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17580398b0 con 0x7f176406fa80 2024-03-24T01:24:41.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.271+0000 7f1763e00640 1 -- 172.21.15.90:0/1566221147 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1758039c60 con 0x7f176406fa80 2024-03-24T01:24:41.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.271+0000 7fcff8a00640 1 Processor -- start 2024-03-24T01:24:41.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.271+0000 7eff15600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff10071b20 0x7eff10071f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.271+0000 7fcc2be00640 1 --2- 172.21.15.90:0/48137839 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc24033990 0x7fcc24107cc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcc24008680 tx=0x7fcc18004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:41.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.272+0000 7f6ee3400640 1 --2- 172.21.15.90:0/4160758613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ee40b2e40 0x7f6ee4156d60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6ed000d800 tx=0x7f6ed000dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:41.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.272+0000 7eff0ea00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7eff1006ee00 0x7eff1006f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.272+0000 7eff0ea00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7eff1006ee00 0x7eff1006f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60162/0 (socket says 172.21.15.90:60162) 2024-03-24T01:24:41.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.272+0000 7eff0ea00640 1 -- 172.21.15.90:0/3935410561 learned_addr learned my addr 172.21.15.90:0/3935410561 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:41.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.272+0000 7eff15600640 1 -- 172.21.15.90:0/3935410561 --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff1006f720 con 0x7eff1006d990 2024-03-24T01:24:41.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.272+0000 7eff0e000640 1 --2- 172.21.15.90:0/3935410561 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff10071b20 0x7eff10071f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.272+0000 7f6edb400640 1 -- 172.21.15.90:0/4160758613 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed000b660 con 0x7f6ee40b2e40 2024-03-24T01:24:41.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.272+0000 7f6edb400640 1 -- 172.21.15.90:0/4160758613 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed0019070 con 0x7f6ee40b2e40 2024-03-24T01:24:41.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.272+0000 7f6edb400640 1 -- 172.21.15.90:0/4160758613 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed000ba20 con 0x7f6ee40b2e40 2024-03-24T01:24:41.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.272+0000 7f6eea000640 1 -- 172.21.15.90:0/4160758613 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ee40c9e20 con 0x7f6ee40b2e40 2024-03-24T01:24:41.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.272+0000 7eff15600640 1 -- 172.21.15.90:0/3935410561 --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff1015a430 con 0x7eff1006ee00 2024-03-24T01:24:41.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.272+0000 7fcc28c00640 1 -- 172.21.15.90:0/48137839 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc18048070 con 0x7fcc24033990 2024-03-24T01:24:41.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.272+0000 7fcc33800640 1 -- 172.21.15.90:0/48137839 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc2404d0d0 con 0x7fcc24033990 2024-03-24T01:24:41.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.272+0000 7fcc33800640 1 -- 172.21.15.90:0/48137839 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc2404d690 con 0x7fcc24033990 2024-03-24T01:24:41.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.272+0000 7eff15600640 1 -- 172.21.15.90:0/3935410561 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff1015a710 con 0x7eff10071b20 2024-03-24T01:24:41.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.272+0000 7fcff8a00640 1 -- start start 2024-03-24T01:24:41.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.272+0000 7f440dc00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4408071990 0x7f4408157800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.272+0000 7f440dc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44080b2890 0x7f44081554f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.272+0000 7f440dc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4408161240 0x7f440806f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.272+0000 7f440dc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4408161620 con 0x7f44080b2890 2024-03-24T01:24:41.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.272+0000 7f440dc00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44080b3240 con 0x7f4408071990 2024-03-24T01:24:41.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.272+0000 7f440dc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f440806e050 con 0x7f4408161240 2024-03-24T01:24:41.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.272+0000 7f4069c00640 1 Processor -- start 2024-03-24T01:24:41.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.272+0000 7f4069c00640 1 -- start start 2024-03-24T01:24:41.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.275+0000 7f7883e00640 1 -- 172.21.15.90:0/17958593 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f789006f040 con 0x7f788804c320 2024-03-24T01:24:41.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.274+0000 7fcc1e000640 1 -- 172.21.15.90:0/48137839 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc08005290 con 0x7fcc24033990 2024-03-24T01:24:41.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.274+0000 7fcc28c00640 1 -- 172.21.15.90:0/48137839 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc18030650 con 0x7fcc24033990 2024-03-24T01:24:41.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.274+0000 7fcc28c00640 1 -- 172.21.15.90:0/48137839 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc18048070 con 0x7fcc24033990 2024-03-24T01:24:41.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.273+0000 7f6eea000640 1 -- 172.21.15.90:0/4160758613 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ee415f560 con 0x7f6ee40b2e40 2024-03-24T01:24:41.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.274+0000 7f4406a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4408071990 0x7f4408157800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.274+0000 7f4406000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44080b2890 0x7f44081554f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.274+0000 7f4407400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4408161240 0x7f440806f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.274+0000 7f4407400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4408161240 0x7f440806f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.90:50260/0 (socket says 172.21.15.90:50260) 2024-03-24T01:24:41.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.274+0000 7f4407400640 1 -- 172.21.15.90:0/4135877352 learned_addr learned my addr 172.21.15.90:0/4135877352 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:41.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.275+0000 7f4407400640 1 -- 172.21.15.90:0/4135877352 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4408071990 msgr2=0x7f4408157800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.275+0000 7f4407400640 1 --2- 172.21.15.90:0/4135877352 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4408071990 0x7f4408157800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.275+0000 7f4407400640 1 -- 172.21.15.90:0/4135877352 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44080b2890 msgr2=0x7f44081554f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.276+0000 7f4407400640 1 --2- 172.21.15.90:0/4135877352 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44080b2890 0x7f44081554f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.273+0000 7f176c200640 1 -- 172.21.15.90:0/1566221147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f176406fa80 msgr2=0x7f176406ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.273+0000 7f176c200640 1 --2- 172.21.15.90:0/1566221147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f176406fa80 0x7f176406ef10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1764072ba0 tx=0x7f1758034410 comp rx=0 tx=0).stop 2024-03-24T01:24:41.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.273+0000 7f176c200640 1 -- 172.21.15.90:0/1566221147 shutdown_connections 2024-03-24T01:24:41.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.273+0000 7f176c200640 1 --2- 172.21.15.90:0/1566221147 >> v2:172.21.15.90:3300/0 conn(0x7f1764159d10 0x7f17640b4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.273+0000 7f176c200640 1 --2- 172.21.15.90:0/1566221147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f176406fa80 0x7f176406ef10 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.273+0000 7f176c200640 1 -- 172.21.15.90:0/1566221147 >> 172.21.15.90:0/1566221147 conn(0x7f17640b1770 msgr2=0x7f17640b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:41.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.274+0000 7f176c200640 1 -- 172.21.15.90:0/1566221147 shutdown_connections 2024-03-24T01:24:41.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.274+0000 7f176c200640 1 -- 172.21.15.90:0/1566221147 wait complete. 2024-03-24T01:24:41.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.273+0000 7eff0ea00640 1 -- 172.21.15.90:0/3935410561 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff10071b20 msgr2=0x7eff10071f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:41.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.273+0000 7eff0ea00640 1 --2- 172.21.15.90:0/3935410561 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff10071b20 0x7eff10071f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.273+0000 7eff0d600640 1 -- 172.21.15.90:0/3935410561 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 580106285 0 0) 0x7eff1006f720 con 0x7eff1006d990 2024-03-24T01:24:41.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.273+0000 7eff0ea00640 1 -- 172.21.15.90:0/3935410561 >> v1:172.21.15.120:6789/0 conn(0x7eff1006d990 legacy=0x7eff1006de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.273+0000 7eff0ea00640 1 -- 172.21.15.90:0/3935410561 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff1015a9f0 con 0x7eff1006ee00 2024-03-24T01:24:41.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.274+0000 7f4069c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40641620d0 0x7f40640b6a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.275+0000 7f4069c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f40641624b0 0x7f4064165e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.275+0000 7f4069c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40641668d0 con 0x7f4064166370 2024-03-24T01:24:41.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.275+0000 7f4069c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40640b6f70 con 0x7f40641624b0 2024-03-24T01:24:41.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.275+0000 7f4069c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40640d17b0 con 0x7f40641620d0 2024-03-24T01:24:41.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.275+0000 7f4063400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4064166370 0x7f40640d0a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51222/0 (socket says 172.21.15.90:51222) 2024-03-24T01:24:41.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.275+0000 7f4063400640 1 -- 172.21.15.90:0/2283744359 learned_addr learned my addr 172.21.15.90:0/2283744359 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:41.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.275+0000 7f4062a00640 1 --2- 172.21.15.90:0/2283744359 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40641620d0 0x7f40640b6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.276+0000 7f4407400640 1 -- 172.21.15.90:0/4135877352 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44081580e0 con 0x7f4408161240 2024-03-24T01:24:41.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.276+0000 7eff0ea00640 1 --2- 172.21.15.90:0/3935410561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff1006ee00 0x7eff1006f1e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7eff000098a0 tx=0x7eff0002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48f58bebbd954602 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:41.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.276+0000 7f6eea000640 1 -- 172.21.15.90:0/4160758613 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ee415f380 con 0x7f6ee40b2e40 2024-03-24T01:24:41.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.276+0000 7f4406a00640 1 --2- 172.21.15.90:0/4135877352 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4408071990 0x7f4408157800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T01:24:41.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.276+0000 7f4062000640 1 --2- 172.21.15.90:0/2283744359 >> v2:172.21.15.120:3300/0 conn(0x7f40641624b0 0x7f4064165e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.276+0000 7f4407400640 1 --2- 172.21.15.90:0/4135877352 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4408161240 0x7f440806f4b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4400007f20 tx=0x7f440000d560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:41.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.276+0000 7f4062a00640 1 -- 172.21.15.90:0/2283744359 >> v2:172.21.15.120:3300/0 conn(0x7f40641624b0 msgr2=0x7f4064165e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:41.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.276+0000 7f4062a00640 1 --2- 172.21.15.90:0/2283744359 >> v2:172.21.15.120:3300/0 conn(0x7f40641624b0 0x7f4064165e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.276+0000 7f4062a00640 1 -- 172.21.15.90:0/2283744359 >> v1:172.21.15.90:6789/0 conn(0x7f4064166370 legacy=0x7f40640d0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.276+0000 7f4062a00640 1 -- 172.21.15.90:0/2283744359 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4064166bb0 con 0x7f40641620d0 2024-03-24T01:24:41.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.277+0000 7f43fea00640 1 -- 172.21.15.90:0/4135877352 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440001a070 con 0x7f4408161240 2024-03-24T01:24:41.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.277+0000 7f43fea00640 1 -- 172.21.15.90:0/4135877352 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4400003b90 con 0x7f4408161240 2024-03-24T01:24:41.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.277+0000 7f43fea00640 1 -- 172.21.15.90:0/4135877352 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4400004030 con 0x7f4408161240 2024-03-24T01:24:41.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.277+0000 7f176c200640 1 Processor -- start 2024-03-24T01:24:41.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.277+0000 7f176c200640 1 -- start start 2024-03-24T01:24:41.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.277+0000 7f440dc00640 1 -- 172.21.15.90:0/4135877352 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44081583c0 con 0x7f4408161240 2024-03-24T01:24:41.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.277+0000 7f440dc00640 1 -- 172.21.15.90:0/4135877352 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4408158980 con 0x7f4408161240 2024-03-24T01:24:41.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.278+0000 7f7883e00640 1 --2- 172.21.15.90:0/17958593 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f787006be00 0x7f787006e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.278+0000 7fcc28c00640 1 -- 172.21.15.90:0/48137839 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcc1804b070 con 0x7fcc24033990 2024-03-24T01:24:41.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.278+0000 7f6edb400640 1 -- 172.21.15.90:0/4160758613 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6ed000bbc0 con 0x7f6ee40b2e40 2024-03-24T01:24:41.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.279+0000 7f176c200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f176406fa80 0x7f17641442e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.277+0000 7fcff8a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfd80039b0 0x7fcff40cb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.277+0000 7fcff8a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcff40cbe30 0x7fcff4200680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.277+0000 7fcff8a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcff4200bc0 0x7fcff4201070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.277+0000 7fcff8a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff415e240 con 0x7fcfd80039b0 2024-03-24T01:24:41.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.277+0000 7fcff8a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff40d3310 con 0x7fcff40cbe30 2024-03-24T01:24:41.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.277+0000 7fcff8a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcff415e3c0 con 0x7fcff4200bc0 2024-03-24T01:24:41.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.278+0000 7fcff2800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcff4200bc0 0x7fcff4201070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.278+0000 7fcff2800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcff4200bc0 0x7fcff4201070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.90:50280/0 (socket says 172.21.15.90:50280) 2024-03-24T01:24:41.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.278+0000 7fcff2800640 1 -- 172.21.15.90:0/3257312065 learned_addr learned my addr 172.21.15.90:0/3257312065 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:41.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.279+0000 7fcff1400640 1 --2- 172.21.15.90:0/3257312065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcff40cbe30 0x7fcff4200680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.279+0000 7f176c200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17640aefb0 0x7f176415ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.279+0000 7f176c200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1764159d10 0x7f176415b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.279+0000 7fcff2800640 1 -- 172.21.15.90:0/3257312065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcff40cbe30 msgr2=0x7fcff4200680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.279+0000 7fcff2800640 1 --2- 172.21.15.90:0/3257312065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcff40cbe30 0x7fcff4200680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.279+0000 7fcff2800640 1 -- 172.21.15.90:0/3257312065 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfd80039b0 msgr2=0x7fcff40cb8f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T01:24:41.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.279+0000 7fcff2800640 1 --2- 172.21.15.90:0/3257312065 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfd80039b0 0x7fcff40cb8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.279+0000 7fcff2800640 1 -- 172.21.15.90:0/3257312065 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff4156b90 con 0x7fcff4200bc0 2024-03-24T01:24:41.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.279+0000 7f176c200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17640b2890 con 0x7f176406fa80 2024-03-24T01:24:41.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.279+0000 7f176c200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f176406e9f0 con 0x7f17640aefb0 2024-03-24T01:24:41.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.279+0000 7f176c200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f176415df90 con 0x7f1764159d10 2024-03-24T01:24:41.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.279+0000 7f440dc00640 1 -- 172.21.15.90:0/4135877352 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4408158bc0 con 0x7f4408161240 2024-03-24T01:24:41.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.279+0000 7fcc28c00640 1 --2- 172.21.15.90:0/48137839 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcc0c06be00 0x7fcc0c06e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.279+0000 7f788fe00640 1 --2- 172.21.15.90:0/17958593 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f787006be00 0x7f787006e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.280+0000 7f1769800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f176406fa80 0x7f17641442e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.280+0000 7f1769800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f176406fa80 0x7f17641442e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60194/0 (socket says 172.21.15.90:60194) 2024-03-24T01:24:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.280+0000 7f1769800640 1 -- 172.21.15.90:0/302351809 learned_addr learned my addr 172.21.15.90:0/302351809 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.280+0000 7f4062a00640 1 --2- 172.21.15.90:0/2283744359 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40641620d0 0x7f40640b6a30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f40540097a0 tx=0x7f405402b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34c2ea327e304384 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.280+0000 7fcc28c00640 1 -- 172.21.15.90:0/48137839 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fcc180aca20 con 0x7fcc24033990 2024-03-24T01:24:41.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.280+0000 7f6edb400640 1 --2- 172.21.15.90:0/4160758613 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6ec406bca0 0x7f6ec406e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.280+0000 7f6edb400640 1 -- 172.21.15.90:0/4160758613 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f6ed008b0e0 con 0x7f6ee40b2e40 2024-03-24T01:24:41.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.280+0000 7fcff2800640 1 --2- 172.21.15.90:0/3257312065 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcff4200bc0 0x7fcff4201070 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcff40b6c30 tx=0x7fcfec007430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:41.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.280+0000 7f1768e00640 1 --2- 172.21.15.90:0/302351809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17640aefb0 0x7f176415ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.280+0000 7f1769800640 1 -- 172.21.15.90:0/302351809 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1764159d10 msgr2=0x7f176415b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.280+0000 7f1769800640 1 --2- 172.21.15.90:0/302351809 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1764159d10 0x7f176415b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.280+0000 7f1769800640 1 -- 172.21.15.90:0/302351809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17640aefb0 msgr2=0x7f176415ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.280+0000 7f1769800640 1 --2- 172.21.15.90:0/302351809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17640aefb0 0x7f176415ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.280+0000 7fcc30e00640 1 --2- 172.21.15.90:0/48137839 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcc0c06be00 0x7fcc0c06e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.281+0000 7f6ee2a00640 1 --2- 172.21.15.90:0/4160758613 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6ec406bca0 0x7f6ec406e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.280+0000 7f1769800640 1 -- 172.21.15.90:0/302351809 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17640ccec0 con 0x7f176406fa80 2024-03-24T01:24:41.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.281+0000 7f1768e00640 1 --2- 172.21.15.90:0/302351809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17640aefb0 0x7f176415ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:41.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.281+0000 7f1769800640 1 --2- 172.21.15.90:0/302351809 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f176406fa80 0x7f17641442e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1758000c00 tx=0x7f17580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:41.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.282+0000 7f788fe00640 1 --2- 172.21.15.90:0/17958593 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f787006be00 0x7f787006e2c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f787c036000 tx=0x7f787c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:41.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.282+0000 7f6ee2a00640 1 --2- 172.21.15.90:0/4160758613 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6ec406bca0 0x7f6ec406e160 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6ee40ca6f0 tx=0x7f6edc00d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:41.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.282+0000 7fcc30e00640 1 --2- 172.21.15.90:0/48137839 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcc0c06be00 0x7fcc0c06e2c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcc2000a9b0 tx=0x7fcc20008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:41.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.282+0000 7f43fea00640 1 -- 172.21.15.90:0/4135877352 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f440001d070 con 0x7f4408161240 2024-03-24T01:24:41.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.283+0000 7f43fea00640 1 --2- 172.21.15.90:0/4135877352 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f43e806bbd0 0x7f43e806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.284+0000 7f43fea00640 1 -- 172.21.15.90:0/4135877352 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f440008b3c0 con 0x7f4408161240 2024-03-24T01:24:41.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.284+0000 7f4406a00640 1 --2- 172.21.15.90:0/4135877352 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f43e806bbd0 0x7f43e806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.284+0000 7f7883e00640 1 -- 172.21.15.90:0/17958593 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f789010a060 con 0x7f788804c320 2024-03-24T01:24:41.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.284+0000 7eff0d600640 1 -- 172.21.15.90:0/3935410561 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff00035020 con 0x7eff1006ee00 2024-03-24T01:24:41.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.284+0000 7f1761600640 1 -- 172.21.15.90:0/302351809 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1758049070 con 0x7f176406fa80 2024-03-24T01:24:41.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.284+0000 7f176c200640 1 -- 172.21.15.90:0/302351809 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17640cd1a0 con 0x7f176406fa80 2024-03-24T01:24:41.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.284+0000 7fcc28c00640 1 -- 172.21.15.90:0/48137839 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcc1807acf0 con 0x7fcc24033990 2024-03-24T01:24:41.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.285+0000 7f176c200640 1 -- 172.21.15.90:0/302351809 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17640cd760 con 0x7f176406fa80 2024-03-24T01:24:41.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.285+0000 7f4406a00640 1 --2- 172.21.15.90:0/4135877352 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f43e806bbd0 0x7f43e806e090 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f43f4006fd0 tx=0x7f43f4009060 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:41.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.286+0000 7f176c200640 1 -- 172.21.15.90:0/302351809 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f176404f400 con 0x7f176406fa80 2024-03-24T01:24:41.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.292+0000 7f4061600640 1 -- 172.21.15.90:0/2283744359 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4054031780 con 0x7f40641620d0 2024-03-24T01:24:41.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.292+0000 7fcfea000640 1 -- 172.21.15.90:0/3257312065 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfec00a4e0 con 0x7fcff4200bc0 2024-03-24T01:24:41.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.292+0000 7fcff8a00640 1 -- 172.21.15.90:0/3257312065 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcff4154e20 con 0x7fcff4200bc0 2024-03-24T01:24:41.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.292+0000 7f6edb400640 1 -- 172.21.15.90:0/4160758613 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6ee415f380 con 0x7f6ee40b2e40 2024-03-24T01:24:41.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.293+0000 7fcff8a00640 1 -- 172.21.15.90:0/3257312065 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcff41550e0 con 0x7fcff4200bc0 2024-03-24T01:24:41.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.293+0000 7f1761600640 1 -- 172.21.15.90:0/302351809 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1758039810 con 0x7f176406fa80 2024-03-24T01:24:41.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.293+0000 7f1761600640 1 -- 172.21.15.90:0/302351809 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1758003a10 con 0x7f176406fa80 2024-03-24T01:24:41.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.293+0000 7eff0d600640 1 -- 172.21.15.90:0/3935410561 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff00031640 con 0x7eff1006ee00 2024-03-24T01:24:41.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.293+0000 7eff0d600640 1 -- 172.21.15.90:0/3935410561 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff0002d040 con 0x7eff1006ee00 2024-03-24T01:24:41.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.293+0000 7f7883e00640 1 -- 172.21.15.90:0/17958593 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f78900d3e10 con 0x7f788804c320 2024-03-24T01:24:41.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.294+0000 7eff15600640 1 -- 172.21.15.90:0/3935410561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff1006ee00 msgr2=0x7eff1006f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.294+0000 7eff15600640 1 --2- 172.21.15.90:0/3935410561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff1006ee00 0x7eff1006f1e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7eff000098a0 tx=0x7eff0002b8f0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.294+0000 7eff15600640 1 -- 172.21.15.90:0/3935410561 shutdown_connections 2024-03-24T01:24:41.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.294+0000 7eff15600640 1 --2- 172.21.15.90:0/3935410561 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff10071b20 0x7eff10071f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.294+0000 7eff15600640 1 --2- 172.21.15.90:0/3935410561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff1006ee00 0x7eff1006f1e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.294+0000 7eff15600640 1 -- 172.21.15.90:0/3935410561 >> 172.21.15.90:0/3935410561 conn(0x7eff100b1cd0 msgr2=0x7eff100cf7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:41.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.294+0000 7eff15600640 1 -- 172.21.15.90:0/3935410561 shutdown_connections 2024-03-24T01:24:41.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.294+0000 7fcff8a00640 1 -- 172.21.15.90:0/3257312065 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcff404f400 con 0x7fcff4200bc0 2024-03-24T01:24:41.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.294+0000 7eff15600640 1 -- 172.21.15.90:0/3935410561 wait complete. 2024-03-24T01:24:41.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.296+0000 7f1761600640 1 -- 172.21.15.90:0/302351809 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1758003bb0 con 0x7f176406fa80 2024-03-24T01:24:41.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.297+0000 7f1761600640 1 --2- 172.21.15.90:0/302351809 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f174406bbd0 0x7f174406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.297+0000 7f1761600640 1 -- 172.21.15.90:0/302351809 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f17580b63a0 con 0x7f176406fa80 2024-03-24T01:24:41.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.297+0000 7f1768e00640 1 --2- 172.21.15.90:0/302351809 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f174406bbd0 0x7f174406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.297+0000 7eff15600640 1 Processor -- start 2024-03-24T01:24:41.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.298+0000 7eff15600640 1 -- start start 2024-03-24T01:24:41.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.298+0000 7f1768e00640 1 --2- 172.21.15.90:0/302351809 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f174406bbd0 0x7f174406e090 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f176415b770 tx=0x7f1750009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:41.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.299+0000 7eff15600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff1006d990 0x7eff10138de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.299+0000 7eff15600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff1006ee00 0x7eff101e3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.300+0000 7eff15600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff10071b20 0x7eff101efae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.300+0000 7eff15600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff10072640 con 0x7eff1006ee00 2024-03-24T01:24:41.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.300+0000 7eff15600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff100724c0 con 0x7eff1006d990 2024-03-24T01:24:41.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.300+0000 7eff15600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff101405c0 con 0x7eff10071b20 2024-03-24T01:24:41.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.300+0000 7eff0ea00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff1006d990 0x7eff10138de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.300+0000 7eff0e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff1006ee00 0x7eff101e3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.300+0000 7eff0e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff1006ee00 0x7eff101e3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60208/0 (socket says 172.21.15.90:60208) 2024-03-24T01:24:41.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.300+0000 7eff0e000640 1 -- 172.21.15.90:0/4014288804 learned_addr learned my addr 172.21.15.90:0/4014288804 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:41.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.300+0000 7eff0f400640 1 --2- 172.21.15.90:0/4014288804 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff10071b20 0x7eff101efae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.301+0000 7eff0e000640 1 -- 172.21.15.90:0/4014288804 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff10071b20 msgr2=0x7eff101efae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.301+0000 7eff0e000640 1 --2- 172.21.15.90:0/4014288804 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff10071b20 0x7eff101efae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.301+0000 7eff0e000640 1 -- 172.21.15.90:0/4014288804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff1006d990 msgr2=0x7eff10138de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.301+0000 7eff0e000640 1 --2- 172.21.15.90:0/4014288804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff1006d990 0x7eff10138de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.301+0000 7eff0e000640 1 -- 172.21.15.90:0/4014288804 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff101377e0 con 0x7eff1006ee00 2024-03-24T01:24:41.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.301+0000 7eff0e000640 1 --2- 172.21.15.90:0/4014288804 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff1006ee00 0x7eff101e3870 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7efefc00e870 tx=0x7efefc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:41.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.303+0000 7eff06a00640 1 -- 172.21.15.90:0/4014288804 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efefc017070 con 0x7eff1006ee00 2024-03-24T01:24:41.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.304+0000 7eff06a00640 1 -- 172.21.15.90:0/4014288804 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efefc00c8e0 con 0x7eff1006ee00 2024-03-24T01:24:41.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.304+0000 7eff06a00640 1 -- 172.21.15.90:0/4014288804 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efefc00ccd0 con 0x7eff1006ee00 2024-03-24T01:24:41.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.304+0000 7eff15600640 1 -- 172.21.15.90:0/4014288804 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff10142360 con 0x7eff1006ee00 2024-03-24T01:24:41.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.304+0000 7f4061600640 1 -- 172.21.15.90:0/2283744359 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4054037070 con 0x7f40641620d0 2024-03-24T01:24:41.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.304+0000 7f4061600640 1 -- 172.21.15.90:0/2283744359 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4054031b40 con 0x7f40641620d0 2024-03-24T01:24:41.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.304+0000 7f1761600640 1 -- 172.21.15.90:0/302351809 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1758084ba0 con 0x7f176406fa80 2024-03-24T01:24:41.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.304+0000 7fcfea000640 1 -- 172.21.15.90:0/3257312065 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfec019070 con 0x7fcff4200bc0 2024-03-24T01:24:41.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.304+0000 7fcfea000640 1 -- 172.21.15.90:0/3257312065 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfec01e660 con 0x7fcff4200bc0 2024-03-24T01:24:41.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.304+0000 7eff15600640 1 -- 172.21.15.90:0/4014288804 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff10142920 con 0x7eff1006ee00 2024-03-24T01:24:41.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.306+0000 7f4069c00640 1 -- 172.21.15.90:0/2283744359 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40641620d0 msgr2=0x7f40640b6a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.306+0000 7f4069c00640 1 --2- 172.21.15.90:0/2283744359 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40641620d0 0x7f40640b6a30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f40540097a0 tx=0x7f405402b820 comp rx=0 tx=0).stop 2024-03-24T01:24:41.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.306+0000 7f4069c00640 1 -- 172.21.15.90:0/2283744359 shutdown_connections 2024-03-24T01:24:41.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.306+0000 7f4069c00640 1 --2- 172.21.15.90:0/2283744359 >> v2:172.21.15.120:3300/0 conn(0x7f40641624b0 0x7f4064165e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.306+0000 7f4069c00640 1 --2- 172.21.15.90:0/2283744359 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40641620d0 0x7f40640b6a30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.306+0000 7f4069c00640 1 -- 172.21.15.90:0/2283744359 >> 172.21.15.90:0/2283744359 conn(0x7f4064072c10 msgr2=0x7f4064073020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:41.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.306+0000 7f43fea00640 1 -- 172.21.15.90:0/4135877352 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4400024020 con 0x7f4408161240 2024-03-24T01:24:41.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.306+0000 7f4069c00640 1 -- 172.21.15.90:0/2283744359 shutdown_connections 2024-03-24T01:24:41.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.307+0000 7f4069c00640 1 -- 172.21.15.90:0/2283744359 wait complete. 2024-03-24T01:24:41.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.306+0000 7eff15600640 1 -- 172.21.15.90:0/4014288804 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff10142b20 con 0x7eff1006ee00 2024-03-24T01:24:41.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.309+0000 7f4069c00640 1 Processor -- start 2024-03-24T01:24:41.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.309+0000 7fcfea000640 1 -- 172.21.15.90:0/3257312065 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcfec01e800 con 0x7fcff4200bc0 2024-03-24T01:24:41.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.309+0000 7f4069c00640 1 -- start start 2024-03-24T01:24:41.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.309+0000 7eff06a00640 1 -- 172.21.15.90:0/4014288804 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7efefc01a070 con 0x7eff1006ee00 2024-03-24T01:24:41.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.311+0000 7fcfea000640 1 --2- 172.21.15.90:0/3257312065 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcfd406bc00 0x7fcfd406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.311+0000 7eff06a00640 1 --2- 172.21.15.90:0/4014288804 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7efef00700b0 0x7efef0072570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.311+0000 7eff06a00640 1 -- 172.21.15.90:0/4014288804 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7efefc08af00 con 0x7eff1006ee00 2024-03-24T01:24:41.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.311+0000 7eff0ea00640 1 --2- 172.21.15.90:0/4014288804 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7efef00700b0 0x7efef0072570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.312+0000 7fcff1e00640 1 --2- 172.21.15.90:0/3257312065 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcfd406bc00 0x7fcfd406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.312+0000 7fcfea000640 1 -- 172.21.15.90:0/3257312065 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fcfec08bf00 con 0x7fcff4200bc0 2024-03-24T01:24:41.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.311+0000 7f4069c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40641620d0 0x7f4064163c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.311+0000 7f4069c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40641624b0 0x7f406415ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.311+0000 7f4069c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4064166370 0x7f40640c7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.311+0000 7f4069c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40640d0380 con 0x7f40641620d0 2024-03-24T01:24:41.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.311+0000 7f4069c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40640d1ce0 con 0x7f4064166370 2024-03-24T01:24:41.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.311+0000 7f4069c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40640d1190 con 0x7f40641624b0 2024-03-24T01:24:41.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.311+0000 7f4062a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40641620d0 0x7f4064163c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.311+0000 7f4062a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40641620d0 0x7f4064163c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60224/0 (socket says 172.21.15.90:60224) 2024-03-24T01:24:41.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.311+0000 7f4062a00640 1 -- 172.21.15.90:0/4088910608 learned_addr learned my addr 172.21.15.90:0/4088910608 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:41.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.312+0000 7f4062000640 1 --2- 172.21.15.90:0/4088910608 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40641624b0 0x7f406415ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.312+0000 7f4062a00640 1 -- 172.21.15.90:0/4088910608 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40641624b0 msgr2=0x7f406415ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.312+0000 7f4062a00640 1 --2- 172.21.15.90:0/4088910608 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40641624b0 0x7f406415ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.312+0000 7f4062a00640 1 -- 172.21.15.90:0/4088910608 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4064166370 msgr2=0x7f40640c7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.312+0000 7f4062a00640 1 --2- 172.21.15.90:0/4088910608 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4064166370 0x7f40640c7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.312+0000 7f4062a00640 1 -- 172.21.15.90:0/4088910608 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40640ccf10 con 0x7f40641620d0 2024-03-24T01:24:41.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.313+0000 7f4062a00640 1 --2- 172.21.15.90:0/4088910608 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40641620d0 0x7f4064163c60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f40540098c0 tx=0x7f40540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:41.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.314+0000 7eff0ea00640 1 --2- 172.21.15.90:0/4014288804 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7efef00700b0 0x7efef0072570 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7eff00036000 tx=0x7eff0003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:41.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.314+0000 7fcff1e00640 1 --2- 172.21.15.90:0/3257312065 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcfd406bc00 0x7fcfd406e0c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcff40d3c60 tx=0x7fcfe4030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:41.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.316+0000 7fcfea000640 1 -- 172.21.15.90:0/3257312065 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcfec05a690 con 0x7fcff4200bc0 2024-03-24T01:24:41.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.318+0000 7f405aa00640 1 -- 172.21.15.90:0/4088910608 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4054032610 con 0x7f40641620d0 2024-03-24T01:24:41.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.318+0000 7f405aa00640 1 -- 172.21.15.90:0/4088910608 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40540327b0 con 0x7f40641620d0 2024-03-24T01:24:41.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.318+0000 7f405aa00640 1 -- 172.21.15.90:0/4088910608 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4054032b80 con 0x7f40641620d0 2024-03-24T01:24:41.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.318+0000 7f4069c00640 1 -- 172.21.15.90:0/4088910608 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40640cd1f0 con 0x7f40641620d0 2024-03-24T01:24:41.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.318+0000 7f4069c00640 1 -- 172.21.15.90:0/4088910608 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40640cd7b0 con 0x7f40641620d0 2024-03-24T01:24:41.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.320+0000 7eff06a00640 1 -- 172.21.15.90:0/4014288804 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7eff10142b20 con 0x7eff1006ee00 2024-03-24T01:24:41.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.320+0000 7f4069c00640 1 -- 172.21.15.90:0/4088910608 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f406404f400 con 0x7f40641620d0 2024-03-24T01:24:41.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.329+0000 7f405aa00640 1 -- 172.21.15.90:0/4088910608 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f405404e430 con 0x7f40641620d0 2024-03-24T01:24:41.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.330+0000 7f405aa00640 1 --2- 172.21.15.90:0/4088910608 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f404406bca0 0x7f404406e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:41.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.330+0000 7f405aa00640 1 -- 172.21.15.90:0/4088910608 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f40540acd10 con 0x7f40641620d0 2024-03-24T01:24:41.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.334+0000 7f405aa00640 1 -- 172.21.15.90:0/4088910608 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4054040070 con 0x7f40641620d0 2024-03-24T01:24:41.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.334+0000 7f4062000640 1 --2- 172.21.15.90:0/4088910608 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f404406bca0 0x7f404406e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:41.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.338+0000 7f4062000640 1 --2- 172.21.15.90:0/4088910608 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f404406bca0 0x7f404406e160 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4050006fd0 tx=0x7f4050008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:41.712 INFO:teuthology.orchestra.run.smithi090.stdout:55834574858 2024-03-24T01:24:41.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.709+0000 7fcc1e000640 1 -- 172.21.15.90:0/48137839 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fcc08005850 con 0x7fcc24033990 2024-03-24T01:24:41.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.709+0000 7fcc28c00640 1 -- 172.21.15.90:0/48137839 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcc1807ed90 con 0x7fcc24033990 2024-03-24T01:24:41.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.714+0000 7fcc1e000640 1 -- 172.21.15.90:0/48137839 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcc0c06be00 msgr2=0x7fcc0c06e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.714+0000 7fcc1e000640 1 --2- 172.21.15.90:0/48137839 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcc0c06be00 0x7fcc0c06e2c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcc2000a9b0 tx=0x7fcc20008040 comp rx=0 tx=0).stop 2024-03-24T01:24:41.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.714+0000 7fcc1e000640 1 -- 172.21.15.90:0/48137839 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc24033990 msgr2=0x7fcc24107cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.714+0000 7fcc1e000640 1 --2- 172.21.15.90:0/48137839 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc24033990 0x7fcc24107cc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcc24008680 tx=0x7fcc18004730 comp rx=0 tx=0).stop 2024-03-24T01:24:41.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.714+0000 7fcc1e000640 1 -- 172.21.15.90:0/48137839 shutdown_connections 2024-03-24T01:24:41.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.714+0000 7fcc1e000640 1 --2- 172.21.15.90:0/48137839 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcc0c06be00 0x7fcc0c06e2c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.714+0000 7fcc1e000640 1 --2- 172.21.15.90:0/48137839 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc24033990 0x7fcc24107cc0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.714+0000 7fcc1e000640 1 --2- 172.21.15.90:0/48137839 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc240f5a60 0x7fcc24047000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.714+0000 7fcc1e000640 1 --2- 172.21.15.90:0/48137839 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc24007190 0x7fcc240f4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.714+0000 7fcc1e000640 1 -- 172.21.15.90:0/48137839 >> 172.21.15.90:0/48137839 conn(0x7fcc24009fb0 msgr2=0x7fcc240f6420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:41.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.715+0000 7fcc1e000640 1 -- 172.21.15.90:0/48137839 shutdown_connections 2024-03-24T01:24:41.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.715+0000 7fcc1e000640 1 -- 172.21.15.90:0/48137839 wait complete. 2024-03-24T01:24:41.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.730+0000 7f7895400640 1 -- 172.21.15.90:0/17958593 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f7888056bf0 con 0x7f788804c320 2024-03-24T01:24:41.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.732+0000 7f7883e00640 1 -- 172.21.15.90:0/17958593 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f78900d3710 con 0x7f788804c320 2024-03-24T01:24:41.735 INFO:teuthology.orchestra.run.smithi090.stdout:55834574857 2024-03-24T01:24:41.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.736+0000 7f7895400640 1 -- 172.21.15.90:0/17958593 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f787006be00 msgr2=0x7f787006e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.736+0000 7f7895400640 1 --2- 172.21.15.90:0/17958593 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f787006be00 0x7f787006e2c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f787c036000 tx=0x7f787c03c040 comp rx=0 tx=0).stop 2024-03-24T01:24:41.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.736+0000 7f7895400640 1 -- 172.21.15.90:0/17958593 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f788804c320 msgr2=0x7f78880d79f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.736+0000 7f7895400640 1 --2- 172.21.15.90:0/17958593 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f788804c320 0x7f78880d79f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7890050670 tx=0x7f7890068b40 comp rx=0 tx=0).stop 2024-03-24T01:24:41.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.737+0000 7f7895400640 1 -- 172.21.15.90:0/17958593 shutdown_connections 2024-03-24T01:24:41.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.737+0000 7f7895400640 1 --2- 172.21.15.90:0/17958593 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f787006be00 0x7f787006e2c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.737+0000 7f7895400640 1 --2- 172.21.15.90:0/17958593 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f788804c320 0x7f78880d79f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.737+0000 7f7895400640 1 --2- 172.21.15.90:0/17958593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78880d4ac0 0x7f78880d4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.737+0000 7f7895400640 1 --2- 172.21.15.90:0/17958593 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f78780039b0 0x7f78880d4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.737+0000 7f7895400640 1 -- 172.21.15.90:0/17958593 >> 172.21.15.90:0/17958593 conn(0x7f7888058ee0 msgr2=0x7f7888054120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:41.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.738+0000 7f7895400640 1 -- 172.21.15.90:0/17958593 shutdown_connections 2024-03-24T01:24:41.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.738+0000 7f7895400640 1 -- 172.21.15.90:0/17958593 wait complete. 2024-03-24T01:24:41.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.741+0000 7f176c200640 1 -- 172.21.15.90:0/302351809 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f176406e250 con 0x7f176406fa80 2024-03-24T01:24:41.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.742+0000 7f1761600640 1 -- 172.21.15.90:0/302351809 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f17580844a0 con 0x7f176406fa80 2024-03-24T01:24:41.742 INFO:teuthology.orchestra.run.smithi090.stdout:55834574857 2024-03-24T01:24:41.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.742+0000 7f440dc00640 1 -- 172.21.15.90:0/4135877352 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f44080afcb0 con 0x7f4408161240 2024-03-24T01:24:41.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.743+0000 7f43fea00640 1 -- 172.21.15.90:0/4135877352 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4400010070 con 0x7f4408161240 2024-03-24T01:24:41.743 INFO:teuthology.orchestra.run.smithi090.stdout:55834574857 2024-03-24T01:24:41.745 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574858 for osd.0 2024-03-24T01:24:41.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.745+0000 7f176c200640 1 -- 172.21.15.90:0/302351809 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f174406bbd0 msgr2=0x7f174406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.745+0000 7f176c200640 1 --2- 172.21.15.90:0/302351809 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f174406bbd0 0x7f174406e090 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f176415b770 tx=0x7f1750009040 comp rx=0 tx=0).stop 2024-03-24T01:24:41.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.745+0000 7f176c200640 1 -- 172.21.15.90:0/302351809 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f176406fa80 msgr2=0x7f17641442e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.745+0000 7f176c200640 1 --2- 172.21.15.90:0/302351809 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f176406fa80 0x7f17641442e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1758000c00 tx=0x7f17580046c0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.746+0000 7eff15600640 1 -- 172.21.15.90:0/4014288804 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7eff1004f400 con 0x7eff1006ee00 2024-03-24T01:24:41.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.747+0000 7f176c200640 1 -- 172.21.15.90:0/302351809 shutdown_connections 2024-03-24T01:24:41.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.747+0000 7f176c200640 1 --2- 172.21.15.90:0/302351809 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f174406bbd0 0x7f174406e090 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.747+0000 7f176c200640 1 --2- 172.21.15.90:0/302351809 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1764159d10 0x7f176415b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.747+0000 7f176c200640 1 --2- 172.21.15.90:0/302351809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17640aefb0 0x7f176415ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.747+0000 7f176c200640 1 --2- 172.21.15.90:0/302351809 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f176406fa80 0x7f17641442e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.747+0000 7f176c200640 1 -- 172.21.15.90:0/302351809 >> 172.21.15.90:0/302351809 conn(0x7f17640b1770 msgr2=0x7f176406b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:41.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.747+0000 7f176c200640 1 -- 172.21.15.90:0/302351809 shutdown_connections 2024-03-24T01:24:41.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.747+0000 7f176c200640 1 -- 172.21.15.90:0/302351809 wait complete. 2024-03-24T01:24:41.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.747+0000 7eff06a00640 1 -- 172.21.15.90:0/4014288804 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efefc059040 con 0x7eff1006ee00 2024-03-24T01:24:41.748 INFO:teuthology.orchestra.run.smithi090.stdout:55834574857 2024-03-24T01:24:41.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.748+0000 7f440dc00640 1 -- 172.21.15.90:0/4135877352 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f43e806bbd0 msgr2=0x7f43e806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.748+0000 7f440dc00640 1 --2- 172.21.15.90:0/4135877352 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f43e806bbd0 0x7f43e806e090 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f43f4006fd0 tx=0x7f43f4009060 comp rx=0 tx=0).stop 2024-03-24T01:24:41.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.748+0000 7f440dc00640 1 -- 172.21.15.90:0/4135877352 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4408161240 msgr2=0x7f440806f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.748+0000 7f440dc00640 1 --2- 172.21.15.90:0/4135877352 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4408161240 0x7f440806f4b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4400007f20 tx=0x7f440000d560 comp rx=0 tx=0).stop 2024-03-24T01:24:41.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.748+0000 7fcff8a00640 1 -- 172.21.15.90:0/3257312065 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fcff415b620 con 0x7fcff4200bc0 2024-03-24T01:24:41.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.748+0000 7f440dc00640 1 -- 172.21.15.90:0/4135877352 shutdown_connections 2024-03-24T01:24:41.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.748+0000 7f440dc00640 1 --2- 172.21.15.90:0/4135877352 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f43e806bbd0 0x7f43e806e090 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.748+0000 7f440dc00640 1 --2- 172.21.15.90:0/4135877352 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44080b2890 0x7f44081554f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.748+0000 7f440dc00640 1 --2- 172.21.15.90:0/4135877352 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4408161240 0x7f440806f4b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.748+0000 7f440dc00640 1 --2- 172.21.15.90:0/4135877352 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4408071990 0x7f4408157800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.748+0000 7f440dc00640 1 -- 172.21.15.90:0/4135877352 >> 172.21.15.90:0/4135877352 conn(0x7f44080b1cd0 msgr2=0x7f440815f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:41.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.748+0000 7f440dc00640 1 -- 172.21.15.90:0/4135877352 shutdown_connections 2024-03-24T01:24:41.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.748+0000 7f440dc00640 1 -- 172.21.15.90:0/4135877352 wait complete. 2024-03-24T01:24:41.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.748+0000 7fcfea000640 1 -- 172.21.15.90:0/3257312065 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcfec05a690 con 0x7fcff4200bc0 2024-03-24T01:24:41.751 INFO:teuthology.orchestra.run.smithi090.stdout:55834574857 2024-03-24T01:24:41.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.751+0000 7eff15600640 1 -- 172.21.15.90:0/4014288804 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7efef00700b0 msgr2=0x7efef0072570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.751+0000 7eff15600640 1 --2- 172.21.15.90:0/4014288804 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7efef00700b0 0x7efef0072570 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7eff00036000 tx=0x7eff0003c040 comp rx=0 tx=0).stop 2024-03-24T01:24:41.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.751+0000 7eff15600640 1 -- 172.21.15.90:0/4014288804 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff1006ee00 msgr2=0x7eff101e3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.751+0000 7eff15600640 1 --2- 172.21.15.90:0/4014288804 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff1006ee00 0x7eff101e3870 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7efefc00e870 tx=0x7efefc00ed30 comp rx=0 tx=0).stop 2024-03-24T01:24:41.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.751+0000 7eff15600640 1 -- 172.21.15.90:0/4014288804 shutdown_connections 2024-03-24T01:24:41.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.751+0000 7eff15600640 1 --2- 172.21.15.90:0/4014288804 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7efef00700b0 0x7efef0072570 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.751+0000 7eff15600640 1 --2- 172.21.15.90:0/4014288804 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff1006ee00 0x7eff101e3870 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.751+0000 7eff15600640 1 --2- 172.21.15.90:0/4014288804 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7eff10071b20 0x7eff101efae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.751+0000 7eff15600640 1 --2- 172.21.15.90:0/4014288804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff1006d990 0x7eff10138de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.751+0000 7eff15600640 1 -- 172.21.15.90:0/4014288804 >> 172.21.15.90:0/4014288804 conn(0x7eff100b1cd0 msgr2=0x7eff100b4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:41.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.752+0000 7fcff8a00640 1 -- 172.21.15.90:0/3257312065 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcfd406bc00 msgr2=0x7fcfd406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.752+0000 7fcff8a00640 1 --2- 172.21.15.90:0/3257312065 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcfd406bc00 0x7fcfd406e0c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcff40d3c60 tx=0x7fcfe4030000 comp rx=0 tx=0).stop 2024-03-24T01:24:41.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.752+0000 7eff15600640 1 -- 172.21.15.90:0/4014288804 shutdown_connections 2024-03-24T01:24:41.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.752+0000 7fcff8a00640 1 -- 172.21.15.90:0/3257312065 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcff4200bc0 msgr2=0x7fcff4201070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.752+0000 7fcff8a00640 1 --2- 172.21.15.90:0/3257312065 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcff4200bc0 0x7fcff4201070 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcff40b6c30 tx=0x7fcfec007430 comp rx=0 tx=0).stop 2024-03-24T01:24:41.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.752+0000 7fcff8a00640 1 -- 172.21.15.90:0/3257312065 shutdown_connections 2024-03-24T01:24:41.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.752+0000 7fcff8a00640 1 --2- 172.21.15.90:0/3257312065 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcfd406bc00 0x7fcfd406e0c0 secure :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcff40d3c60 tx=0x7fcfe4030000 comp rx=0 tx=0).stop 2024-03-24T01:24:41.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.752+0000 7fcff8a00640 1 --2- 172.21.15.90:0/3257312065 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcff4200bc0 0x7fcff4201070 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.752+0000 7fcff8a00640 1 --2- 172.21.15.90:0/3257312065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcff40cbe30 0x7fcff4200680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.752+0000 7fcff8a00640 1 --2- 172.21.15.90:0/3257312065 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcfd80039b0 0x7fcff40cb8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.752+0000 7fcff8a00640 1 -- 172.21.15.90:0/3257312065 >> 172.21.15.90:0/3257312065 conn(0x7fcff4072c10 msgr2=0x7fcff40d19f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:41.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.753+0000 7fcff8a00640 1 -- 172.21.15.90:0/3257312065 shutdown_connections 2024-03-24T01:24:41.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.753+0000 7eff15600640 1 -- 172.21.15.90:0/4014288804 wait complete. 2024-03-24T01:24:41.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.753+0000 7fcff8a00640 1 -- 172.21.15.90:0/3257312065 wait complete. 2024-03-24T01:24:41.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.753+0000 7f6eea000640 1 -- 172.21.15.90:0/4160758613 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f6ee404f400 con 0x7f6ee40b2e40 2024-03-24T01:24:41.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.754+0000 7f6edb400640 1 -- 172.21.15.90:0/4160758613 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6ed0059220 con 0x7f6ee40b2e40 2024-03-24T01:24:41.759 INFO:teuthology.orchestra.run.smithi090.stdout:55834574858 2024-03-24T01:24:41.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.758+0000 7f6eea000640 1 -- 172.21.15.90:0/4160758613 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6ec406bca0 msgr2=0x7f6ec406e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.758+0000 7f6eea000640 1 --2- 172.21.15.90:0/4160758613 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6ec406bca0 0x7f6ec406e160 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6ee40ca6f0 tx=0x7f6edc00d040 comp rx=0 tx=0).stop 2024-03-24T01:24:41.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.758+0000 7f6eea000640 1 -- 172.21.15.90:0/4160758613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ee40b2e40 msgr2=0x7f6ee4156d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.758+0000 7f6eea000640 1 --2- 172.21.15.90:0/4160758613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ee40b2e40 0x7f6ee4156d60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6ed000d800 tx=0x7f6ed000dcc0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.759+0000 7f6eea000640 1 -- 172.21.15.90:0/4160758613 shutdown_connections 2024-03-24T01:24:41.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.759+0000 7f6eea000640 1 --2- 172.21.15.90:0/4160758613 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6ec406bca0 0x7f6ec406e160 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.759+0000 7f6eea000640 1 --2- 172.21.15.90:0/4160758613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ee40d0b50 0x7f6ee40c92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.759+0000 7f6eea000640 1 --2- 172.21.15.90:0/4160758613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ee415dfd0 0x7f6ee40c9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.759+0000 7f6eea000640 1 --2- 172.21.15.90:0/4160758613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ee40b2e40 0x7f6ee4156d60 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.759+0000 7f6eea000640 1 -- 172.21.15.90:0/4160758613 >> 172.21.15.90:0/4160758613 conn(0x7f6ee4072c10 msgr2=0x7f6ee4072ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:41.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.761+0000 7f6eea000640 1 -- 172.21.15.90:0/4160758613 shutdown_connections 2024-03-24T01:24:41.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.761+0000 7f6eea000640 1 -- 172.21.15.90:0/4160758613 wait complete. 2024-03-24T01:24:41.764 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574857 for osd.1 2024-03-24T01:24:41.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.767+0000 7f4069c00640 1 -- 172.21.15.90:0/4088910608 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f40640b6120 con 0x7f40641620d0 2024-03-24T01:24:41.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.768+0000 7f405aa00640 1 -- 172.21.15.90:0/4088910608 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f405402f050 con 0x7f40641620d0 2024-03-24T01:24:41.768 INFO:teuthology.orchestra.run.smithi090.stdout:55834574858 2024-03-24T01:24:41.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.771+0000 7f4069c00640 1 -- 172.21.15.90:0/4088910608 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f404406bca0 msgr2=0x7f404406e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.771+0000 7f4069c00640 1 --2- 172.21.15.90:0/4088910608 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f404406bca0 0x7f404406e160 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4050006fd0 tx=0x7f4050008040 comp rx=0 tx=0).stop 2024-03-24T01:24:41.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.771+0000 7f4069c00640 1 -- 172.21.15.90:0/4088910608 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40641620d0 msgr2=0x7f4064163c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:41.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.771+0000 7f4069c00640 1 --2- 172.21.15.90:0/4088910608 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40641620d0 0x7f4064163c60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f40540098c0 tx=0x7f40540046c0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.771+0000 7f4069c00640 1 -- 172.21.15.90:0/4088910608 shutdown_connections 2024-03-24T01:24:41.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.771+0000 7f4069c00640 1 --2- 172.21.15.90:0/4088910608 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f404406bca0 0x7f404406e160 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.771+0000 7f4069c00640 1 --2- 172.21.15.90:0/4088910608 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4064166370 0x7f40640c7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.771+0000 7f4069c00640 1 --2- 172.21.15.90:0/4088910608 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40641624b0 0x7f406415ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.771+0000 7f4069c00640 1 --2- 172.21.15.90:0/4088910608 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40641620d0 0x7f4064163c60 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:41.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.771+0000 7f4069c00640 1 -- 172.21.15.90:0/4088910608 >> 172.21.15.90:0/4088910608 conn(0x7f4064072c10 msgr2=0x7f40640d2af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:41.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.771+0000 7f4069c00640 1 -- 172.21.15.90:0/4088910608 shutdown_connections 2024-03-24T01:24:41.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:41.771+0000 7f4069c00640 1 -- 172.21.15.90:0/4088910608 wait complete. 2024-03-24T01:24:41.773 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574857 for osd.4 2024-03-24T01:24:41.777 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574857 for osd.3 2024-03-24T01:24:41.779 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574857 for osd.7 2024-03-24T01:24:41.780 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574857 for osd.6 2024-03-24T01:24:41.783 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574858 for osd.2 2024-03-24T01:24:41.791 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574858 for osd.5 2024-03-24T01:24:42.746 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-24T01:24:42.765 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-24T01:24:42.774 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-24T01:24:42.778 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-24T01:24:42.780 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-24T01:24:42.780 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-24T01:24:42.783 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-24T01:24:42.792 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-24T01:24:43.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.034+0000 7f8a50c00640 1 Processor -- start 2024-03-24T01:24:43.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.034+0000 7f8a50c00640 1 -- start start 2024-03-24T01:24:43.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.037+0000 7f8a50c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8a4c0719d0 0x7f8a4c06e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.037+0000 7f8a50c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8a4c071db0 0x7f8a4c162d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.038+0000 7f8a50c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a4c06e780 con 0x7f8a4c165ba0 2024-03-24T01:24:43.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.038+0000 7f8a50c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a4c06ea60 con 0x7f8a4c071db0 2024-03-24T01:24:43.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.038+0000 7f8a50c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a4c06ed40 con 0x7f8a4c0719d0 2024-03-24T01:24:43.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.039+0000 7f8a49400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f8a4c165ba0 0x7f8a4c0d2610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51234/0 (socket says 172.21.15.90:51234) 2024-03-24T01:24:43.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.039+0000 7f8a49400640 1 -- 172.21.15.90:0/2576500211 learned_addr learned my addr 172.21.15.90:0/2576500211 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:43.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.040+0000 7f8a49e00640 1 --2- 172.21.15.90:0/2576500211 >> v2:172.21.15.120:3300/0 conn(0x7f8a4c0719d0 0x7f8a4c06e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.040+0000 7f8a4a800640 1 --2- 172.21.15.90:0/2576500211 >> v2:172.21.15.90:3300/0 conn(0x7f8a4c071db0 0x7f8a4c162d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.042+0000 7f8a4a800640 1 -- 172.21.15.90:0/2576500211 >> v2:172.21.15.120:3300/0 conn(0x7f8a4c0719d0 msgr2=0x7f8a4c06e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:43.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.042+0000 7f8a4a800640 1 --2- 172.21.15.90:0/2576500211 >> v2:172.21.15.120:3300/0 conn(0x7f8a4c0719d0 0x7f8a4c06e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.042+0000 7f8a4a800640 1 -- 172.21.15.90:0/2576500211 >> v1:172.21.15.90:6789/0 conn(0x7f8a4c165ba0 legacy=0x7f8a4c0d2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.042+0000 7f8a4a800640 1 -- 172.21.15.90:0/2576500211 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a4c06f020 con 0x7f8a4c071db0 2024-03-24T01:24:43.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.044+0000 7f8a4a800640 1 --2- 172.21.15.90:0/2576500211 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a4c071db0 0x7f8a4c162d50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c06cf70 tx=0x7f8a4402bc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d54800f77f6bbc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:43.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.045+0000 7f8a48a00640 1 -- 172.21.15.90:0/2576500211 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a44039070 con 0x7f8a4c071db0 2024-03-24T01:24:43.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.046+0000 7f8a48a00640 1 -- 172.21.15.90:0/2576500211 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a4402c9a0 con 0x7f8a4c071db0 2024-03-24T01:24:43.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.047+0000 7f8a50c00640 1 -- 172.21.15.90:0/2576500211 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a4c071db0 msgr2=0x7f8a4c162d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.047+0000 7f8a50c00640 1 --2- 172.21.15.90:0/2576500211 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a4c071db0 0x7f8a4c162d50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c06cf70 tx=0x7f8a4402bc70 comp rx=0 tx=0).stop 2024-03-24T01:24:43.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.047+0000 7f8a50c00640 1 -- 172.21.15.90:0/2576500211 shutdown_connections 2024-03-24T01:24:43.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.048+0000 7f8a50c00640 1 --2- 172.21.15.90:0/2576500211 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a4c071db0 0x7f8a4c162d50 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.048+0000 7f8a50c00640 1 --2- 172.21.15.90:0/2576500211 >> v2:172.21.15.120:3300/0 conn(0x7f8a4c0719d0 0x7f8a4c06e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.048+0000 7f8a50c00640 1 -- 172.21.15.90:0/2576500211 >> 172.21.15.90:0/2576500211 conn(0x7f8a4c0d7ec0 msgr2=0x7f8a4c0d82a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:43.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.048+0000 7f8a50c00640 1 -- 172.21.15.90:0/2576500211 shutdown_connections 2024-03-24T01:24:43.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.048+0000 7f8a50c00640 1 -- 172.21.15.90:0/2576500211 wait complete. 2024-03-24T01:24:43.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.052+0000 7f8a50c00640 1 Processor -- start 2024-03-24T01:24:43.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.052+0000 7f8a50c00640 1 -- start start 2024-03-24T01:24:43.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.054+0000 7f8a50c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a4c0719d0 0x7f8a4c0bbf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.054+0000 7f8a50c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a4c071db0 0x7f8a4c0bd590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.055+0000 7f8a49400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a4c071db0 0x7f8a4c0bd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.055+0000 7f8a49400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a4c071db0 0x7f8a4c0bd590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60246/0 (socket says 172.21.15.90:60246) 2024-03-24T01:24:43.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.055+0000 7f8a49e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a4c0719d0 0x7f8a4c0bbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.056+0000 7f8a49400640 1 -- 172.21.15.90:0/1196181580 learned_addr learned my addr 172.21.15.90:0/1196181580 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:43.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.056+0000 7f8a50c00640 1 --2- 172.21.15.90:0/1196181580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a4c165ba0 0x7f8a4c203e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.056+0000 7f8a50c00640 1 -- 172.21.15.90:0/1196181580 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c072a10 con 0x7f8a4c071db0 2024-03-24T01:24:43.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.056+0000 7f8a50c00640 1 -- 172.21.15.90:0/1196181580 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c072890 con 0x7f8a4c165ba0 2024-03-24T01:24:43.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.056+0000 7f8a50c00640 1 -- 172.21.15.90:0/1196181580 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c163290 con 0x7f8a4c0719d0 2024-03-24T01:24:43.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.057+0000 7f8a4a800640 1 --2- 172.21.15.90:0/1196181580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a4c165ba0 0x7f8a4c203e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.058+0000 7f8a49400640 1 -- 172.21.15.90:0/1196181580 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a4c0719d0 msgr2=0x7f8a4c0bbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.058+0000 7f8a49400640 1 --2- 172.21.15.90:0/1196181580 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a4c0719d0 0x7f8a4c0bbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.058+0000 7f8a49400640 1 -- 172.21.15.90:0/1196181580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a4c165ba0 msgr2=0x7f8a4c203e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.058+0000 7f8a49400640 1 --2- 172.21.15.90:0/1196181580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a4c165ba0 0x7f8a4c203e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.058+0000 7f8a49400640 1 -- 172.21.15.90:0/1196181580 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a4c0bc4f0 con 0x7f8a4c071db0 2024-03-24T01:24:43.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.059+0000 7f8a4a800640 1 --2- 172.21.15.90:0/1196181580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a4c165ba0 0x7f8a4c203e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:43.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.059+0000 7f8a49400640 1 --2- 172.21.15.90:0/1196181580 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a4c071db0 0x7f8a4c0bd590 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8a3800c810 tx=0x7f8a3800ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:43.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.060+0000 7f8a42000640 1 -- 172.21.15.90:0/1196181580 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a38018070 con 0x7f8a4c071db0 2024-03-24T01:24:43.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.060+0000 7f8a50c00640 1 -- 172.21.15.90:0/1196181580 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a4c0bc7d0 con 0x7f8a4c071db0 2024-03-24T01:24:43.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.061+0000 7f8a42000640 1 -- 172.21.15.90:0/1196181580 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a380072d0 con 0x7f8a4c071db0 2024-03-24T01:24:43.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.061+0000 7f8a42000640 1 -- 172.21.15.90:0/1196181580 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a38018070 con 0x7f8a4c071db0 2024-03-24T01:24:43.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.064+0000 7f8a50c00640 1 -- 172.21.15.90:0/1196181580 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a4c0bcd90 con 0x7f8a4c071db0 2024-03-24T01:24:43.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.067+0000 7f8a37400640 1 -- 172.21.15.90:0/1196181580 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a4c04f400 con 0x7f8a4c071db0 2024-03-24T01:24:43.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.068+0000 7f8a42000640 1 -- 172.21.15.90:0/1196181580 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8a3801b070 con 0x7f8a4c071db0 2024-03-24T01:24:43.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.070+0000 7f8a42000640 1 --2- 172.21.15.90:0/1196181580 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8a2c06bdd0 0x7f8a2c06e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.070+0000 7f8a42000640 1 -- 172.21.15.90:0/1196181580 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f8a38089cc0 con 0x7f8a4c071db0 2024-03-24T01:24:43.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.073+0000 7f8a49e00640 1 --2- 172.21.15.90:0/1196181580 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8a2c06bdd0 0x7f8a2c06e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.081+0000 7f8a49e00640 1 --2- 172.21.15.90:0/1196181580 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8a2c06bdd0 0x7f8a2c06e290 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c00a970 tx=0x7f8a3c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:43.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.081+0000 7f8a42000640 1 -- 172.21.15.90:0/1196181580 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8a38058470 con 0x7f8a4c071db0 2024-03-24T01:24:43.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.089+0000 7fd617e00640 1 Processor -- start 2024-03-24T01:24:43.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.089+0000 7fd617e00640 1 -- start start 2024-03-24T01:24:43.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.090+0000 7f387f200640 1 Processor -- start 2024-03-24T01:24:43.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.091+0000 7f387f200640 1 -- start start 2024-03-24T01:24:43.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.091+0000 7fd617e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd61003c0a0 0x7fd6100e7780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.091+0000 7fd617e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd610058810 con 0x7fd610053cb0 2024-03-24T01:24:43.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.091+0000 7fd617e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd610058af0 con 0x7fd6100e3160 2024-03-24T01:24:43.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.091+0000 7fd617e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd610007b70 con 0x7fd61003c0a0 2024-03-24T01:24:43.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.092+0000 7fd616a00640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fd6100e3160 0x7fd610058050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.90:48756/0 (socket says 172.21.15.90:48756) 2024-03-24T01:24:43.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.092+0000 7fd616a00640 1 -- 172.21.15.90:0/2255658753 learned_addr learned my addr 172.21.15.90:0/2255658753 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:43.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.092+0000 7fd616000640 1 --2- 172.21.15.90:0/2255658753 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd61003c0a0 0x7fd6100e7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.093+0000 7fd616000640 1 -- 172.21.15.90:0/2255658753 >> v1:172.21.15.120:6789/0 conn(0x7fd6100e3160 legacy=0x7fd610058050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.093+0000 7fd616000640 1 -- 172.21.15.90:0/2255658753 >> v1:172.21.15.90:6789/0 conn(0x7fd610053cb0 legacy=0x7fd610054120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.093+0000 7fd615600640 1 -- 172.21.15.90:0/2255658753 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3574786594 0 0) 0x7fd610058810 con 0x7fd610053cb0 2024-03-24T01:24:43.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.093+0000 7fd616000640 1 -- 172.21.15.90:0/2255658753 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd610006360 con 0x7fd61003c0a0 2024-03-24T01:24:43.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.093+0000 7f387f200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3880161f20 0x7f38800cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.094+0000 7f387f200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3880166050 con 0x7f3880165bc0 2024-03-24T01:24:43.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.094+0000 7f387f200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3880166310 con 0x7f38800d1d60 2024-03-24T01:24:43.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.094+0000 7f387f200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38800d3930 con 0x7f3880161f20 2024-03-24T01:24:43.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.095+0000 7f387e800640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f38800d1d60 0x7f38800b5550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.90:48772/0 (socket says 172.21.15.90:48772) 2024-03-24T01:24:43.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.095+0000 7f387e800640 1 -- 172.21.15.90:0/948766753 learned_addr learned my addr 172.21.15.90:0/948766753 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:43.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.095+0000 7f387de00640 1 --2- 172.21.15.90:0/948766753 >> v2:172.21.15.90:3300/0 conn(0x7f3880161f20 0x7f38800cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.095+0000 7fd616000640 1 --2- 172.21.15.90:0/2255658753 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd61003c0a0 0x7fd6100e7780 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd618067e50 tx=0x7fd6180a70e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c198ad7d16f5af9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:43.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.096+0000 7f387ca00640 1 -- 172.21.15.90:0/948766753 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 512954082 0 0) 0x7f3880166310 con 0x7f38800d1d60 2024-03-24T01:24:43.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.096+0000 7f387ca00640 1 -- 172.21.15.90:0/948766753 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3864003520 con 0x7f38800d1d60 2024-03-24T01:24:43.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.096+0000 7f387de00640 1 -- 172.21.15.90:0/948766753 >> v1:172.21.15.120:6789/0 conn(0x7f38800d1d60 legacy=0x7f38800b5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.096+0000 7f387de00640 1 -- 172.21.15.90:0/948766753 >> v1:172.21.15.90:6789/0 conn(0x7f3880165bc0 legacy=0x7f38800b6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:43.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.096+0000 7f387de00640 1 -- 172.21.15.90:0/948766753 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38801665f0 con 0x7f3880161f20 2024-03-24T01:24:43.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.096+0000 7fc94ec00640 1 Processor -- start 2024-03-24T01:24:43.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.096+0000 7fd615600640 1 -- 172.21.15.90:0/2255658753 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6180b4070 con 0x7fd61003c0a0 2024-03-24T01:24:43.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.096+0000 7fd615600640 1 -- 172.21.15.90:0/2255658753 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6180af440 con 0x7fd61003c0a0 2024-03-24T01:24:43.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.097+0000 7fd617e00640 1 -- 172.21.15.90:0/2255658753 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd61003c0a0 msgr2=0x7fd6100e7780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.097+0000 7fd617e00640 1 --2- 172.21.15.90:0/2255658753 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd61003c0a0 0x7fd6100e7780 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd618067e50 tx=0x7fd6180a70e0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.098+0000 7fd617e00640 1 -- 172.21.15.90:0/2255658753 shutdown_connections 2024-03-24T01:24:43.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.098+0000 7fd617e00640 1 --2- 172.21.15.90:0/2255658753 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd61003c0a0 0x7fd6100e7780 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd618067e50 tx=0x7fd6180a70e0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.098+0000 7fd617e00640 1 -- 172.21.15.90:0/2255658753 >> 172.21.15.90:0/2255658753 conn(0x7fd610058ee0 msgr2=0x7fd6100592c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:43.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.098+0000 7fc94ec00640 1 -- start start 2024-03-24T01:24:43.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.098+0000 7f387de00640 1 --2- 172.21.15.90:0/948766753 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3880161f20 0x7f38800cfe70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f38700097a0 tx=0x7f387002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25ff2edb10be4694 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:43.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.098+0000 7fd617e00640 1 -- 172.21.15.90:0/2255658753 shutdown_connections 2024-03-24T01:24:43.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.098+0000 7f387ca00640 1 -- 172.21.15.90:0/948766753 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3870035020 con 0x7f3880161f20 2024-03-24T01:24:43.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.098+0000 7f387ca00640 1 -- 172.21.15.90:0/948766753 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3870031830 con 0x7f3880161f20 2024-03-24T01:24:43.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.098+0000 7f387ca00640 1 -- 172.21.15.90:0/948766753 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3870031c00 con 0x7f3880161f20 2024-03-24T01:24:43.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.099+0000 7f387f200640 1 -- 172.21.15.90:0/948766753 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3880161f20 msgr2=0x7f38800cfe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.099+0000 7f387f200640 1 --2- 172.21.15.90:0/948766753 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3880161f20 0x7f38800cfe70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f38700097a0 tx=0x7f387002b8d0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.100+0000 7fc94ec00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9400392d0 0x7fc9400df1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.100+0000 7fc94ec00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc940006dd0 con 0x7fc9400501a0 2024-03-24T01:24:43.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.100+0000 7fc94ec00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9400070b0 con 0x7fc94000aaa0 2024-03-24T01:24:43.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.100+0000 7fc94ec00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9400068a0 con 0x7fc9400392d0 2024-03-24T01:24:43.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.100+0000 7f387f200640 1 -- 172.21.15.90:0/948766753 shutdown_connections 2024-03-24T01:24:43.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.100+0000 7f387f200640 1 --2- 172.21.15.90:0/948766753 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3880161f20 0x7f38800cfe70 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.100+0000 7f387f200640 1 -- 172.21.15.90:0/948766753 >> 172.21.15.90:0/948766753 conn(0x7f38800d0550 msgr2=0x7f38800d0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:43.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.100+0000 7fc94e200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9400392d0 0x7fc9400df1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.098+0000 7fd617e00640 1 -- 172.21.15.90:0/2255658753 wait complete. 2024-03-24T01:24:43.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.101+0000 7fc94d800640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fc94000aaa0 0x7fc94004fa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.90:48782/0 (socket says 172.21.15.90:48782) 2024-03-24T01:24:43.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.101+0000 7fd617e00640 1 Processor -- start 2024-03-24T01:24:43.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.101+0000 7fc94d800640 1 -- 172.21.15.90:0/4142555391 learned_addr learned my addr 172.21.15.90:0/4142555391 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:43.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.101+0000 7fd617e00640 1 -- start start 2024-03-24T01:24:43.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.101+0000 7f387f200640 1 -- 172.21.15.90:0/948766753 shutdown_connections 2024-03-24T01:24:43.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.101+0000 7f387f200640 1 -- 172.21.15.90:0/948766753 wait complete. 2024-03-24T01:24:43.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.102+0000 7fc94e200640 1 -- 172.21.15.90:0/4142555391 >> v1:172.21.15.120:6789/0 conn(0x7fc94000aaa0 legacy=0x7fc94004fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.102+0000 7fc94e200640 1 -- 172.21.15.90:0/4142555391 >> v1:172.21.15.90:6789/0 conn(0x7fc9400501a0 legacy=0x7fc940053260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.102+0000 7fc94e200640 1 -- 172.21.15.90:0/4142555391 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9400e5d30 con 0x7fc9400392d0 2024-03-24T01:24:43.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.103+0000 7fd617e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd610053cb0 0x7fd6100d4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.103+0000 7fd617e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6100e3160 0x7fd6100dd710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.103+0000 7fd617e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6100dfa30 0x7fd61003d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.104+0000 7fd617e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6100ee050 con 0x7fd610053cb0 2024-03-24T01:24:43.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.104+0000 7fd616a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd610053cb0 0x7fd6100d4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.104+0000 7fd616a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd610053cb0 0x7fd6100d4750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60268/0 (socket says 172.21.15.90:60268) 2024-03-24T01:24:43.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.104+0000 7fd616a00640 1 -- 172.21.15.90:0/2719914000 learned_addr learned my addr 172.21.15.90:0/2719914000 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:43.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.104+0000 7fd617e00640 1 -- 172.21.15.90:0/2719914000 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6100e3540 con 0x7fd6100e3160 2024-03-24T01:24:43.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.104+0000 7fd617e00640 1 -- 172.21.15.90:0/2719914000 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6100ee1d0 con 0x7fd6100dfa30 2024-03-24T01:24:43.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.104+0000 7fd617400640 1 --2- 172.21.15.90:0/2719914000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6100dfa30 0x7fd61003d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.105+0000 7fd616a00640 1 -- 172.21.15.90:0/2719914000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6100dfa30 msgr2=0x7fd61003d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.104+0000 7f387f200640 1 Processor -- start 2024-03-24T01:24:43.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.104+0000 7f387f200640 1 -- start start 2024-03-24T01:24:43.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.104+0000 7fc94e200640 1 --2- 172.21.15.90:0/4142555391 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9400392d0 0x7fc9400df1b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc93c0097d0 tx=0x7fc93c02b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8024dc5ae76c50a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:43.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.105+0000 7f59ed000640 1 Processor -- start 2024-03-24T01:24:43.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.106+0000 7f59ed000640 1 -- start start 2024-03-24T01:24:43.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.105+0000 7fd616a00640 1 --2- 172.21.15.90:0/2719914000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6100dfa30 0x7fd61003d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.105+0000 7fd616a00640 1 -- 172.21.15.90:0/2719914000 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6100e3160 msgr2=0x7fd6100dd710 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-24T01:24:43.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.106+0000 7fd616a00640 1 --2- 172.21.15.90:0/2719914000 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6100e3160 0x7fd6100dd710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.106+0000 7fd616a00640 1 -- 172.21.15.90:0/2719914000 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6100dff30 con 0x7fd610053cb0 2024-03-24T01:24:43.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.106+0000 7fc947e00640 1 -- 172.21.15.90:0/4142555391 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93c037070 con 0x7fc9400392d0 2024-03-24T01:24:43.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.106+0000 7f387f200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38800d1d60 0x7f38801649a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.106+0000 7fd616a00640 1 --2- 172.21.15.90:0/2719914000 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd610053cb0 0x7fd6100d4750 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd60800c730 tx=0x7fd60800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:43.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.106+0000 7f387f200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3880161f20 0x7f38800cd110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.106+0000 7f387f200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3880165bc0 0x7f38800cd650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.106+0000 7f387f200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38800b7030 con 0x7f3880165bc0 2024-03-24T01:24:43.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.106+0000 7f387f200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38800d1840 con 0x7f38800d1d60 2024-03-24T01:24:43.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.106+0000 7f387f200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3880162300 con 0x7f3880161f20 2024-03-24T01:24:43.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.106+0000 7fc947e00640 1 -- 172.21.15.90:0/4142555391 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc93c030710 con 0x7fc9400392d0 2024-03-24T01:24:43.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.106+0000 7fc947e00640 1 -- 172.21.15.90:0/4142555391 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93c030ae0 con 0x7fc9400392d0 2024-03-24T01:24:43.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.106+0000 7f387de00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38800d1d60 0x7f38801649a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.106+0000 7f387de00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38800d1d60 0x7f38801649a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.90:39294/0 (socket says 172.21.15.90:39294) 2024-03-24T01:24:43.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.106+0000 7f387de00640 1 -- 172.21.15.90:0/3387204198 learned_addr learned my addr 172.21.15.90:0/3387204198 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:43.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.107+0000 7f387d400640 1 --2- 172.21.15.90:0/3387204198 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3880161f20 0x7f38800cd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.107+0000 7f387e800640 1 --2- 172.21.15.90:0/3387204198 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3880165bc0 0x7f38800cd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.107+0000 7fd60ea00640 1 -- 172.21.15.90:0/2719914000 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd608017070 con 0x7fd610053cb0 2024-03-24T01:24:43.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.107+0000 7fd60ea00640 1 -- 172.21.15.90:0/2719914000 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6080058b0 con 0x7fd610053cb0 2024-03-24T01:24:43.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.107+0000 7fc94ec00640 1 -- 172.21.15.90:0/4142555391 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9400392d0 msgr2=0x7fc9400df1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.107+0000 7fc94ec00640 1 --2- 172.21.15.90:0/4142555391 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9400392d0 0x7fc9400df1b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc93c0097d0 tx=0x7fc93c02b800 comp rx=0 tx=0).stop 2024-03-24T01:24:43.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.107+0000 7f59ed000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f59e809a2d0 0x7f59e806cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.107+0000 7fd60ea00640 1 -- 172.21.15.90:0/2719914000 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd608005c60 con 0x7fd610053cb0 2024-03-24T01:24:43.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.107+0000 7f59ed000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59e806c3e0 0x7f59e80cce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.107+0000 7f59ed000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e80b43f0 con 0x7f59e81615c0 2024-03-24T01:24:43.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.107+0000 7f59ed000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59e80cca00 con 0x7f59e809a2d0 2024-03-24T01:24:43.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.107+0000 7f59ed000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59e80cc0d0 con 0x7f59e806c3e0 2024-03-24T01:24:43.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.107+0000 7fd617e00640 1 -- 172.21.15.90:0/2719914000 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6100e0210 con 0x7fd610053cb0 2024-03-24T01:24:43.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.108+0000 7fd617e00640 1 -- 172.21.15.90:0/2719914000 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6100e07d0 con 0x7fd610053cb0 2024-03-24T01:24:43.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.108+0000 7f387e800640 1 -- 172.21.15.90:0/3387204198 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3880161f20 msgr2=0x7f38800cd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.108+0000 7f387e800640 1 --2- 172.21.15.90:0/3387204198 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3880161f20 0x7f38800cd110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.108+0000 7f387e800640 1 -- 172.21.15.90:0/3387204198 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38800d1d60 msgr2=0x7f38801649a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.108+0000 7f387e800640 1 --2- 172.21.15.90:0/3387204198 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38800d1d60 0x7f38801649a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.108+0000 7f387e800640 1 -- 172.21.15.90:0/3387204198 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3880163620 con 0x7f3880165bc0 2024-03-24T01:24:43.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.108+0000 7fc94ec00640 1 -- 172.21.15.90:0/4142555391 shutdown_connections 2024-03-24T01:24:43.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.108+0000 7fc94ec00640 1 --2- 172.21.15.90:0/4142555391 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9400392d0 0x7fc9400df1b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.108+0000 7fc94ec00640 1 -- 172.21.15.90:0/4142555391 >> 172.21.15.90:0/4142555391 conn(0x7fc940009fb0 msgr2=0x7fc94000a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:43.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.108+0000 7fc94ec00640 1 -- 172.21.15.90:0/4142555391 shutdown_connections 2024-03-24T01:24:43.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.108+0000 7fc94ec00640 1 -- 172.21.15.90:0/4142555391 wait complete. 2024-03-24T01:24:43.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.108+0000 7f59e6800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59e806c3e0 0x7f59e80cce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.108+0000 7f59e6800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59e806c3e0 0x7f59e80cce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.90:50344/0 (socket says 172.21.15.90:50344) 2024-03-24T01:24:43.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.108+0000 7f59e6800640 1 -- 172.21.15.90:0/1588407053 learned_addr learned my addr 172.21.15.90:0/1588407053 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:43.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.108+0000 7f59e5400640 1 --2- 172.21.15.90:0/1588407053 >> v2:172.21.15.120:3300/0 conn(0x7f59e809a2d0 0x7f59e806cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.109+0000 7feb46e00640 1 Processor -- start 2024-03-24T01:24:43.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.109+0000 7feb46e00640 1 -- start start 2024-03-24T01:24:43.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.109+0000 7f59e5400640 1 -- 172.21.15.90:0/1588407053 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59e806c3e0 msgr2=0x7f59e80cce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:43.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.109+0000 7f59e5400640 1 --2- 172.21.15.90:0/1588407053 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59e806c3e0 0x7f59e80cce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.109+0000 7f59e5400640 1 -- 172.21.15.90:0/1588407053 >> v1:172.21.15.90:6789/0 conn(0x7f59e81615c0 legacy=0x7f59e8099bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:43.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.111+0000 7fd617e00640 1 -- 172.21.15.90:0/2719914000 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5f0005290 con 0x7fd610053cb0 2024-03-24T01:24:43.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.110+0000 7f387e800640 1 --2- 172.21.15.90:0/3387204198 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3880165bc0 0x7f38800cd650 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f38780012e0 tx=0x7f387800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:43.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.111+0000 7f3876000640 1 -- 172.21.15.90:0/3387204198 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3878018070 con 0x7f3880165bc0 2024-03-24T01:24:43.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.111+0000 7f387f200640 1 -- 172.21.15.90:0/3387204198 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3880163900 con 0x7f3880165bc0 2024-03-24T01:24:43.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.111+0000 7f387f200640 1 -- 172.21.15.90:0/3387204198 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3880163ec0 con 0x7f3880165bc0 2024-03-24T01:24:43.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.111+0000 7f3876000640 1 -- 172.21.15.90:0/3387204198 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3878004d80 con 0x7f3880165bc0 2024-03-24T01:24:43.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.111+0000 7f3876000640 1 -- 172.21.15.90:0/3387204198 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387800a880 con 0x7f3880165bc0 2024-03-24T01:24:43.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.111+0000 7fc94ec00640 1 Processor -- start 2024-03-24T01:24:43.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.111+0000 7fc94ec00640 1 -- start start 2024-03-24T01:24:43.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.110+0000 7f68bc600640 1 Processor -- start 2024-03-24T01:24:43.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.108+0000 7f0762200640 1 Processor -- start 2024-03-24T01:24:43.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.110+0000 7f0762200640 1 -- start start 2024-03-24T01:24:43.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.110+0000 7f59e5400640 1 -- 172.21.15.90:0/1588407053 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e80cc3b0 con 0x7f59e809a2d0 2024-03-24T01:24:43.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.111+0000 7feb46e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7feb380e7a50 0x7feb38049b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.111+0000 7feb46e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb3804a0a0 0x7feb380548a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.111+0000 7feb46e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7feb38054de0 0x7feb3803c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.111+0000 7feb46e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb380e7e30 con 0x7feb380e7a50 2024-03-24T01:24:43.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.111+0000 7feb46e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb3803b5a0 con 0x7feb38054de0 2024-03-24T01:24:43.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.111+0000 7feb46e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb3803b880 con 0x7feb3804a0a0 2024-03-24T01:24:43.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.110+0000 7f68bc600640 1 -- start start 2024-03-24T01:24:43.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.112+0000 7feb45000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb3804a0a0 0x7feb380548a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.112+0000 7feb46400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7feb38054de0 0x7feb3803c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.112+0000 7feb45000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb3804a0a0 0x7feb380548a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.90:50360/0 (socket says 172.21.15.90:50360) 2024-03-24T01:24:43.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.112+0000 7f68bc600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f68b4071b20 0x7f68b4071f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.112+0000 7feb45000640 1 -- 172.21.15.90:0/395669299 learned_addr learned my addr 172.21.15.90:0/395669299 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:43.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.112+0000 7fd60ea00640 1 -- 172.21.15.90:0/2719914000 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd60801a070 con 0x7fd610053cb0 2024-03-24T01:24:43.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.112+0000 7f68bc600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68b415d910 0x7f68b415dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.112+0000 7f68bc600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b415a430 con 0x7f68b406ee00 2024-03-24T01:24:43.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.112+0000 7f68bc600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68b415a710 con 0x7f68b4071b20 2024-03-24T01:24:43.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.112+0000 7f68bc600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68b415a9f0 con 0x7f68b415d910 2024-03-24T01:24:43.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.112+0000 7f59e5400640 1 --2- 172.21.15.90:0/1588407053 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59e809a2d0 0x7f59e806cbe0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f59d40097a0 tx=0x7f59d402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d9591d0b6da89d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:43.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.113+0000 7f68b9200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f68b4071b20 0x7f68b4071f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.113+0000 7f387f200640 1 -- 172.21.15.90:0/3387204198 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3858005290 con 0x7f3880165bc0 2024-03-24T01:24:43.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.113+0000 7f68b9200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f68b4071b20 0x7f68b4071f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60290/0 (socket says 172.21.15.90:60290) 2024-03-24T01:24:43.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.113+0000 7f0762200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f075c0d3e70 0x7f075c0d1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.113+0000 7f68b9200640 1 -- 172.21.15.90:0/3569601014 learned_addr learned my addr 172.21.15.90:0/3569601014 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:43.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.114+0000 7f0762200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f075c0d4250 0x7f075c15d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.114+0000 7f0762200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f075c15dd00 0x7f075c0b56f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.114+0000 7fd60ea00640 1 --2- 172.21.15.90:0/2719914000 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd5f806bc20 0x7fd5f806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.114+0000 7f68b9200640 1 -- 172.21.15.90:0/3569601014 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68b415d910 msgr2=0x7f68b415dda0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:24:43.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.114+0000 7f0762200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f075c0d2740 con 0x7f075c0d4250 2024-03-24T01:24:43.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.114+0000 7f0762200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f075c0d2a20 con 0x7f075c15dd00 2024-03-24T01:24:43.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.114+0000 7f0762200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f075c0d2d00 con 0x7f075c0d3e70 2024-03-24T01:24:43.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.114+0000 7f68b9200640 1 --2- 172.21.15.90:0/3569601014 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68b415d910 0x7f68b415dda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.114+0000 7f68b9200640 1 -- 172.21.15.90:0/3569601014 >> v1:172.21.15.120:6789/0 conn(0x7f68b406ee00 legacy=0x7f68b406f1e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:24:43.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.114+0000 7f68b9200640 1 -- 172.21.15.90:0/3569601014 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b415acd0 con 0x7f68b4071b20 2024-03-24T01:24:43.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.115+0000 7f59e4a00640 1 -- 172.21.15.90:0/1588407053 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d4035020 con 0x7f59e809a2d0 2024-03-24T01:24:43.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.115+0000 7f59e4a00640 1 -- 172.21.15.90:0/1588407053 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d4031810 con 0x7f59e809a2d0 2024-03-24T01:24:43.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.115+0000 7f59e4a00640 1 -- 172.21.15.90:0/1588407053 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d4031be0 con 0x7f59e809a2d0 2024-03-24T01:24:43.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.115+0000 7f075b400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f075c0d3e70 0x7f075c0d1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.115+0000 7fd60ea00640 1 -- 172.21.15.90:0/2719914000 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd60808a060 con 0x7fd610053cb0 2024-03-24T01:24:43.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.115+0000 7fd616000640 1 --2- 172.21.15.90:0/2719914000 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd5f806bc20 0x7fd5f806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.115+0000 7f075b400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f075c0d3e70 0x7f075c0d1dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.90:50370/0 (socket says 172.21.15.90:50370) 2024-03-24T01:24:43.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.115+0000 7f075b400640 1 -- 172.21.15.90:0/170817198 learned_addr learned my addr 172.21.15.90:0/170817198 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:43.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.113+0000 7feb45000640 1 -- 172.21.15.90:0/395669299 >> v2:172.21.15.120:3300/0 conn(0x7feb38054de0 msgr2=0x7feb3803c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:43.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.113+0000 7feb45000640 1 --2- 172.21.15.90:0/395669299 >> v2:172.21.15.120:3300/0 conn(0x7feb38054de0 0x7feb3803c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.113+0000 7feb45000640 1 -- 172.21.15.90:0/395669299 >> v2:172.21.15.90:3300/0 conn(0x7feb380e7a50 msgr2=0x7feb38049b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:43.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.113+0000 7feb45000640 1 --2- 172.21.15.90:0/395669299 >> v2:172.21.15.90:3300/0 conn(0x7feb380e7a50 0x7feb38049b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.113+0000 7feb45000640 1 -- 172.21.15.90:0/395669299 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb3803bb60 con 0x7feb3804a0a0 2024-03-24T01:24:43.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.114+0000 7fc94ec00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc94000aaa0 0x7fc940052e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.114+0000 7fc94ec00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9400392d0 0x7fc9400533d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.114+0000 7fc94ec00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9400501a0 0x7fc9400db830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.114+0000 7fc94ec00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc940052cc0 con 0x7fc9400501a0 2024-03-24T01:24:43.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.114+0000 7fc94ec00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc940052b40 con 0x7fc94000aaa0 2024-03-24T01:24:43.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.114+0000 7fc94ec00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9400e3d10 con 0x7fc9400392d0 2024-03-24T01:24:43.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.114+0000 7fc94d800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc94000aaa0 0x7fc940052e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.114+0000 7fc94ce00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9400392d0 0x7fc9400533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.115+0000 7fc94d800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc94000aaa0 0x7fc940052e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.90:39330/0 (socket says 172.21.15.90:39330) 2024-03-24T01:24:43.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.115+0000 7fc94d800640 1 -- 172.21.15.90:0/3469621265 learned_addr learned my addr 172.21.15.90:0/3469621265 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:43.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.115+0000 7fc94e200640 1 --2- 172.21.15.90:0/3469621265 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9400501a0 0x7fc9400db830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.116+0000 7fc94ce00640 1 -- 172.21.15.90:0/3469621265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc94000aaa0 msgr2=0x7fc940052e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.116+0000 7fc94ce00640 1 --2- 172.21.15.90:0/3469621265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc94000aaa0 0x7fc940052e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.116+0000 7fc94ce00640 1 -- 172.21.15.90:0/3469621265 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9400501a0 msgr2=0x7fc9400db830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.116+0000 7fc94ce00640 1 --2- 172.21.15.90:0/3469621265 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9400501a0 0x7fc9400db830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.116+0000 7f075b400640 1 -- 172.21.15.90:0/170817198 >> v2:172.21.15.120:3300/0 conn(0x7f075c15dd00 msgr2=0x7f075c0b56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:43.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.116+0000 7f075be00640 1 --2- 172.21.15.90:0/170817198 >> v2:172.21.15.120:3300/0 conn(0x7f075c15dd00 0x7f075c0b56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.116+0000 7f59ed000640 1 -- 172.21.15.90:0/1588407053 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59e809a2d0 msgr2=0x7f59e806cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.116+0000 7fc94ce00640 1 -- 172.21.15.90:0/3469621265 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9400dc680 con 0x7fc9400392d0 2024-03-24T01:24:43.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.116+0000 7f59ed000640 1 --2- 172.21.15.90:0/1588407053 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59e809a2d0 0x7f59e806cbe0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f59d40097a0 tx=0x7f59d402b8b0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.116+0000 7f075b400640 1 --2- 172.21.15.90:0/170817198 >> v2:172.21.15.120:3300/0 conn(0x7f075c15dd00 0x7f075c0b56f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.116+0000 7f075b400640 1 -- 172.21.15.90:0/170817198 >> v2:172.21.15.90:3300/0 conn(0x7f075c0d4250 msgr2=0x7f075c15d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:43.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.116+0000 7f075aa00640 1 --2- 172.21.15.90:0/170817198 >> v2:172.21.15.90:3300/0 conn(0x7f075c0d4250 0x7f075c15d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.116+0000 7f59ed000640 1 -- 172.21.15.90:0/1588407053 shutdown_connections 2024-03-24T01:24:43.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.116+0000 7fc94ce00640 1 --2- 172.21.15.90:0/3469621265 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9400392d0 0x7fc9400533d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc93800c500 tx=0x7fc93800c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:43.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.116+0000 7f59ed000640 1 --2- 172.21.15.90:0/1588407053 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59e806c3e0 0x7f59e80cce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.116+0000 7f59ed000640 1 --2- 172.21.15.90:0/1588407053 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59e809a2d0 0x7f59e806cbe0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.116+0000 7f59ed000640 1 -- 172.21.15.90:0/1588407053 >> 172.21.15.90:0/1588407053 conn(0x7f59e80b34d0 msgr2=0x7f59e80b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:43.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.117+0000 7f075b400640 1 --2- 172.21.15.90:0/170817198 >> v2:172.21.15.90:3300/0 conn(0x7f075c0d4250 0x7f075c15d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.117+0000 7f075b400640 1 -- 172.21.15.90:0/170817198 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f075c0d2fe0 con 0x7f075c0d3e70 2024-03-24T01:24:43.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.117+0000 7f075aa00640 1 --2- 172.21.15.90:0/170817198 >> v2:172.21.15.90:3300/0 conn(0x7f075c0d4250 0x7f075c15d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:43.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.117+0000 7fd616000640 1 --2- 172.21.15.90:0/2719914000 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd5f806bc20 0x7fd5f806e0e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd61004ff60 tx=0x7fd6180ab000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:43.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.117+0000 7f59ed000640 1 -- 172.21.15.90:0/1588407053 shutdown_connections 2024-03-24T01:24:43.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.117+0000 7f68b9200640 1 --2- 172.21.15.90:0/3569601014 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68b4071b20 0x7f68b4071f80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f68a00091e0 tx=0x7f68a002cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a601cf6369a894bc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:43.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.117+0000 7fc945600640 1 -- 172.21.15.90:0/3469621265 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc938017070 con 0x7fc9400392d0 2024-03-24T01:24:43.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.117+0000 7fc945600640 1 -- 172.21.15.90:0/3469621265 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9380035f0 con 0x7fc9400392d0 2024-03-24T01:24:43.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.117+0000 7f3876000640 1 -- 172.21.15.90:0/3387204198 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f387800aa20 con 0x7f3880165bc0 2024-03-24T01:24:43.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.117+0000 7fc945600640 1 -- 172.21.15.90:0/3469621265 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc938003a20 con 0x7fc9400392d0 2024-03-24T01:24:43.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.117+0000 7fc94ec00640 1 -- 172.21.15.90:0/3469621265 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9400dc960 con 0x7fc9400392d0 2024-03-24T01:24:43.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.117+0000 7fc94ec00640 1 -- 172.21.15.90:0/3469621265 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9400dcf20 con 0x7fc9400392d0 2024-03-24T01:24:43.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.117+0000 7feb45000640 1 --2- 172.21.15.90:0/395669299 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb3804a0a0 0x7feb380548a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7feb34009870 tx=0x7feb3402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f0bd2e0e6556fb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:43.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.118+0000 7feb3fe00640 1 -- 172.21.15.90:0/395669299 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb34035020 con 0x7feb3804a0a0 2024-03-24T01:24:43.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.118+0000 7feb3fe00640 1 -- 172.21.15.90:0/395669299 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb34031890 con 0x7feb3804a0a0 2024-03-24T01:24:43.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.118+0000 7feb3fe00640 1 -- 172.21.15.90:0/395669299 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb34031ca0 con 0x7feb3804a0a0 2024-03-24T01:24:43.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.117+0000 7f59ed000640 1 -- 172.21.15.90:0/1588407053 wait complete. 2024-03-24T01:24:43.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.118+0000 7f075b400640 1 --2- 172.21.15.90:0/170817198 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f075c0d3e70 0x7f075c0d1dd0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f075c0cf5b0 tx=0x7f074c02b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7b47175303b5a37 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:43.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.119+0000 7feb46e00640 1 -- 172.21.15.90:0/395669299 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb3804a0a0 msgr2=0x7feb380548a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.119+0000 7feb46e00640 1 --2- 172.21.15.90:0/395669299 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb3804a0a0 0x7feb380548a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7feb34009870 tx=0x7feb3402b930 comp rx=0 tx=0).stop 2024-03-24T01:24:43.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.119+0000 7feb46e00640 1 -- 172.21.15.90:0/395669299 shutdown_connections 2024-03-24T01:24:43.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.119+0000 7feb46e00640 1 --2- 172.21.15.90:0/395669299 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb3804a0a0 0x7feb380548a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.119+0000 7feb46e00640 1 --2- 172.21.15.90:0/395669299 >> v2:172.21.15.120:3300/0 conn(0x7feb38054de0 0x7feb3803c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.119+0000 7feb46e00640 1 --2- 172.21.15.90:0/395669299 >> v2:172.21.15.90:3300/0 conn(0x7feb380e7a50 0x7feb38049b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.119+0000 7f075a000640 1 -- 172.21.15.90:0/170817198 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f074c035020 con 0x7f075c0d3e70 2024-03-24T01:24:43.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.119+0000 7f075a000640 1 -- 172.21.15.90:0/170817198 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f074c02d040 con 0x7f075c0d3e70 2024-03-24T01:24:43.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.119+0000 7f075a000640 1 -- 172.21.15.90:0/170817198 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f074c031a80 con 0x7f075c0d3e70 2024-03-24T01:24:43.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.119+0000 7f3876000640 1 --2- 172.21.15.90:0/3387204198 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f386006bbd0 0x7f386006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.120+0000 7f3876000640 1 -- 172.21.15.90:0/3387204198 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f387808a1a0 con 0x7f3880165bc0 2024-03-24T01:24:43.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.120+0000 7f387de00640 1 --2- 172.21.15.90:0/3387204198 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f386006bbd0 0x7f386006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.119+0000 7feb46e00640 1 -- 172.21.15.90:0/395669299 >> 172.21.15.90:0/395669299 conn(0x7feb38056cd0 msgr2=0x7feb380570e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:43.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.120+0000 7f59ed000640 1 Processor -- start 2024-03-24T01:24:43.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.120+0000 7f59ed000640 1 -- start start 2024-03-24T01:24:43.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.121+0000 7f0762200640 1 -- 172.21.15.90:0/170817198 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f075c0d3e70 msgr2=0x7f075c0d1dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.121+0000 7feb46e00640 1 -- 172.21.15.90:0/395669299 shutdown_connections 2024-03-24T01:24:43.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.121+0000 7feb46e00640 1 -- 172.21.15.90:0/395669299 wait complete. 2024-03-24T01:24:43.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.121+0000 7fd60ea00640 1 -- 172.21.15.90:0/2719914000 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd6080587f0 con 0x7fd610053cb0 2024-03-24T01:24:43.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.121+0000 7f0762200640 1 --2- 172.21.15.90:0/170817198 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f075c0d3e70 0x7f075c0d1dd0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f075c0cf5b0 tx=0x7f074c02b900 comp rx=0 tx=0).stop 2024-03-24T01:24:43.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.121+0000 7f387de00640 1 --2- 172.21.15.90:0/3387204198 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f386006bbd0 0x7f386006e090 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3870009560 tx=0x7f3870008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:43.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.121+0000 7f0762200640 1 -- 172.21.15.90:0/170817198 shutdown_connections 2024-03-24T01:24:43.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.121+0000 7f0762200640 1 --2- 172.21.15.90:0/170817198 >> v2:172.21.15.120:3300/0 conn(0x7f075c15dd00 0x7f075c0b56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.121+0000 7f0762200640 1 --2- 172.21.15.90:0/170817198 >> v2:172.21.15.90:3300/0 conn(0x7f075c0d4250 0x7f075c15d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.121+0000 7f0762200640 1 --2- 172.21.15.90:0/170817198 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f075c0d3e70 0x7f075c0d1dd0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.121+0000 7f0762200640 1 -- 172.21.15.90:0/170817198 >> 172.21.15.90:0/170817198 conn(0x7f075c072c10 msgr2=0x7f075c073020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:43.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.121+0000 7fc94ec00640 1 -- 172.21.15.90:0/3469621265 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9400089d0 con 0x7fc9400392d0 2024-03-24T01:24:43.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.121+0000 7fc945600640 1 -- 172.21.15.90:0/3469621265 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc93801a070 con 0x7fc9400392d0 2024-03-24T01:24:43.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.122+0000 7f59ed000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59e806c3e0 0x7f59e814fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.122+0000 7f59ed000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59e809a2d0 0x7f59e80b59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.122+0000 7f59ed000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59e81615c0 0x7f59e815f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.122+0000 7f0762200640 1 -- 172.21.15.90:0/170817198 shutdown_connections 2024-03-24T01:24:43.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.123+0000 7f0762200640 1 -- 172.21.15.90:0/170817198 wait complete. 2024-03-24T01:24:43.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.123+0000 7f59ed000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e806d3d0 con 0x7f59e806c3e0 2024-03-24T01:24:43.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.123+0000 7f59ed000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e806d250 con 0x7f59e809a2d0 2024-03-24T01:24:43.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.123+0000 7f59ed000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59e80cd3d0 con 0x7f59e81615c0 2024-03-24T01:24:43.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.123+0000 7f59e5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59e806c3e0 0x7f59e814fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.123+0000 7f59e5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59e806c3e0 0x7f59e814fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60308/0 (socket says 172.21.15.90:60308) 2024-03-24T01:24:43.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.123+0000 7feb46e00640 1 Processor -- start 2024-03-24T01:24:43.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.123+0000 7feb46e00640 1 -- start start 2024-03-24T01:24:43.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.123+0000 7f59e5e00640 1 -- 172.21.15.90:0/1556600059 learned_addr learned my addr 172.21.15.90:0/1556600059 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:43.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.123+0000 7fc945600640 1 --2- 172.21.15.90:0/3469621265 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc92c06bc20 0x7fc92c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.123+0000 7fc945600640 1 -- 172.21.15.90:0/3469621265 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fc93808a1b0 con 0x7fc9400392d0 2024-03-24T01:24:43.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.124+0000 7f59e5400640 1 --2- 172.21.15.90:0/1556600059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59e809a2d0 0x7f59e80b59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.124+0000 7f59e5e00640 1 -- 172.21.15.90:0/1556600059 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59e81615c0 msgr2=0x7f59e815f680 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-24T01:24:43.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.124+0000 7f59e5e00640 1 --2- 172.21.15.90:0/1556600059 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59e81615c0 0x7f59e815f680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.124+0000 7f59e5e00640 1 -- 172.21.15.90:0/1556600059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59e809a2d0 msgr2=0x7f59e80b59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.124+0000 7f59e5e00640 1 --2- 172.21.15.90:0/1556600059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59e809a2d0 0x7f59e80b59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.124+0000 7f59e5e00640 1 -- 172.21.15.90:0/1556600059 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e80b5ee0 con 0x7f59e806c3e0 2024-03-24T01:24:43.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.124+0000 7f59e5400640 1 --2- 172.21.15.90:0/1556600059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59e809a2d0 0x7f59e80b59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:43.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.124+0000 7feb46e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb3804a0a0 0x7feb3817d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.124+0000 7feb46e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb38054de0 0x7feb380e1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.124+0000 7feb46e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb380e7a50 0x7feb380e2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.124+0000 7feb46e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb38039cb0 con 0x7feb3804a0a0 2024-03-24T01:24:43.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.124+0000 7feb46e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb3804a5b0 con 0x7feb38054de0 2024-03-24T01:24:43.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.124+0000 7feb46e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb38039e30 con 0x7feb380e7a50 2024-03-24T01:24:43.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.124+0000 7fc94d800640 1 --2- 172.21.15.90:0/3469621265 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc92c06bc20 0x7fc92c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.125+0000 7f0762200640 1 Processor -- start 2024-03-24T01:24:43.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.125+0000 7feb45a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb3804a0a0 0x7feb3817d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.125+0000 7f0762200640 1 -- start start 2024-03-24T01:24:43.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.125+0000 7feb45a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb3804a0a0 0x7feb3817d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60314/0 (socket says 172.21.15.90:60314) 2024-03-24T01:24:43.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.125+0000 7feb45a00640 1 -- 172.21.15.90:0/1331323913 learned_addr learned my addr 172.21.15.90:0/1331323913 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:43.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.125+0000 7f59e5e00640 1 --2- 172.21.15.90:0/1556600059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59e806c3e0 0x7f59e814fed0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f59e000c730 tx=0x7f59e000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:43.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.125+0000 7fc94d800640 1 --2- 172.21.15.90:0/3469621265 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc92c06bc20 0x7fc92c06e0e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc948067e50 tx=0x7fc948068de0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:43.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.126+0000 7f0762200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f075c0d3e70 0x7f075c1f6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.126+0000 7f0762200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f075c0d4250 0x7f075c153d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.126+0000 7f0762200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f075c15dd00 0x7f075c14fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.126+0000 7f0762200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075c0d37c0 con 0x7f075c0d4250 2024-03-24T01:24:43.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.126+0000 7f0762200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075c15e170 con 0x7f075c0d3e70 2024-03-24T01:24:43.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.126+0000 7f0762200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f075c0d3940 con 0x7f075c15dd00 2024-03-24T01:24:43.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.126+0000 7feb46400640 1 --2- 172.21.15.90:0/1331323913 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb380e7a50 0x7feb380e2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.126+0000 7feb45a00640 1 -- 172.21.15.90:0/1331323913 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb380e7a50 msgr2=0x7feb380e2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.126+0000 7feb45a00640 1 --2- 172.21.15.90:0/1331323913 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb380e7a50 0x7feb380e2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.126+0000 7feb45a00640 1 -- 172.21.15.90:0/1331323913 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb38054de0 msgr2=0x7feb380e1ac0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-24T01:24:43.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.126+0000 7feb45a00640 1 --2- 172.21.15.90:0/1331323913 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb38054de0 0x7feb380e1ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.126+0000 7feb45a00640 1 -- 172.21.15.90:0/1331323913 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb380d68c0 con 0x7feb3804a0a0 2024-03-24T01:24:43.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.126+0000 7feb45a00640 1 --2- 172.21.15.90:0/1331323913 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb3804a0a0 0x7feb3817d1b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7feb2c00b4a0 tx=0x7feb2c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:43.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.127+0000 7f075be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f075c15dd00 0x7f075c14fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.127+0000 7f075aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f075c0d4250 0x7f075c153d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.127+0000 7f075aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f075c0d4250 0x7f075c153d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60326/0 (socket says 172.21.15.90:60326) 2024-03-24T01:24:43.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.127+0000 7f075aa00640 1 -- 172.21.15.90:0/232730540 learned_addr learned my addr 172.21.15.90:0/232730540 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:43.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.127+0000 7f075b400640 1 --2- 172.21.15.90:0/232730540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f075c0d3e70 0x7f075c1f6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.128+0000 7f075aa00640 1 -- 172.21.15.90:0/232730540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f075c15dd00 msgr2=0x7f075c14fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.128+0000 7f075aa00640 1 --2- 172.21.15.90:0/232730540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f075c15dd00 0x7f075c14fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.128+0000 7f075aa00640 1 -- 172.21.15.90:0/232730540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f075c0d3e70 msgr2=0x7f075c1f6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.128+0000 7f075aa00640 1 --2- 172.21.15.90:0/232730540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f075c0d3e70 0x7f075c1f6fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.128+0000 7f075aa00640 1 -- 172.21.15.90:0/232730540 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f075c154280 con 0x7f075c0d4250 2024-03-24T01:24:43.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.128+0000 7f075b400640 1 --2- 172.21.15.90:0/232730540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f075c0d3e70 0x7f075c1f6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:24:43.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.128+0000 7f075aa00640 1 --2- 172.21.15.90:0/232730540 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f075c0d4250 0x7f075c153d40 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f074800c820 tx=0x7f074800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:43.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.130+0000 7f0753400640 1 -- 172.21.15.90:0/232730540 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0748015070 con 0x7f075c0d4250 2024-03-24T01:24:43.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.130+0000 7f3876000640 1 -- 172.21.15.90:0/3387204198 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f38780582f0 con 0x7f3880165bc0 2024-03-24T01:24:43.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.130+0000 7f0753400640 1 -- 172.21.15.90:0/232730540 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0748007a30 con 0x7f075c0d4250 2024-03-24T01:24:43.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.130+0000 7f0753400640 1 -- 172.21.15.90:0/232730540 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0748007e00 con 0x7f075c0d4250 2024-03-24T01:24:43.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.130+0000 7f0762200640 1 -- 172.21.15.90:0/232730540 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f075c1f75c0 con 0x7f075c0d4250 2024-03-24T01:24:43.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.130+0000 7f0762200640 1 -- 172.21.15.90:0/232730540 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f075c1f7b80 con 0x7f075c0d4250 2024-03-24T01:24:43.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.130+0000 7feb3d600640 1 -- 172.21.15.90:0/1331323913 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c014070 con 0x7feb3804a0a0 2024-03-24T01:24:43.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.131+0000 7feb3d600640 1 -- 172.21.15.90:0/1331323913 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb2c010610 con 0x7feb3804a0a0 2024-03-24T01:24:43.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.131+0000 7feb3d600640 1 -- 172.21.15.90:0/1331323913 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c0109e0 con 0x7feb3804a0a0 2024-03-24T01:24:43.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.131+0000 7feb46e00640 1 -- 172.21.15.90:0/1331323913 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb380e2630 con 0x7feb3804a0a0 2024-03-24T01:24:43.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.130+0000 7f68b3e00640 1 -- 172.21.15.90:0/3569601014 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a003a070 con 0x7f68b4071b20 2024-03-24T01:24:43.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.130+0000 7f68b3e00640 1 -- 172.21.15.90:0/3569601014 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68a002d970 con 0x7f68b4071b20 2024-03-24T01:24:43.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.130+0000 7f68b3e00640 1 -- 172.21.15.90:0/3569601014 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a002dd40 con 0x7f68b4071b20 2024-03-24T01:24:43.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.131+0000 7feb46e00640 1 -- 172.21.15.90:0/1331323913 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb380e2bf0 con 0x7feb3804a0a0 2024-03-24T01:24:43.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.131+0000 7f59de000640 1 -- 172.21.15.90:0/1556600059 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e0003b70 con 0x7f59e806c3e0 2024-03-24T01:24:43.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.132+0000 7f59de000640 1 -- 172.21.15.90:0/1556600059 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59e0003d30 con 0x7f59e806c3e0 2024-03-24T01:24:43.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.132+0000 7f59de000640 1 -- 172.21.15.90:0/1556600059 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e000aa90 con 0x7f59e806c3e0 2024-03-24T01:24:43.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.132+0000 7f59ed000640 1 -- 172.21.15.90:0/1556600059 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59e80b6580 con 0x7f59e806c3e0 2024-03-24T01:24:43.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.132+0000 7f59ed000640 1 -- 172.21.15.90:0/1556600059 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59e80b6b40 con 0x7f59e806c3e0 2024-03-24T01:24:43.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.132+0000 7feb46e00640 1 -- 172.21.15.90:0/1331323913 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb380057e0 con 0x7feb3804a0a0 2024-03-24T01:24:43.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.134+0000 7f0762200640 1 -- 172.21.15.90:0/232730540 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f075c04f400 con 0x7f075c0d4250 2024-03-24T01:24:43.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.134+0000 7f59ed000640 1 -- 172.21.15.90:0/1556600059 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59e804f400 con 0x7f59e806c3e0 2024-03-24T01:24:43.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.134+0000 7f0753400640 1 -- 172.21.15.90:0/232730540 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0748018070 con 0x7f075c0d4250 2024-03-24T01:24:43.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.135+0000 7feb3d600640 1 -- 172.21.15.90:0/1331323913 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7feb2c017070 con 0x7feb3804a0a0 2024-03-24T01:24:43.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.136+0000 7f0753400640 1 --2- 172.21.15.90:0/232730540 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f073c06bc20 0x7f073c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.136+0000 7f0753400640 1 -- 172.21.15.90:0/232730540 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f074808ac50 con 0x7f075c0d4250 2024-03-24T01:24:43.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.136+0000 7f075b400640 1 --2- 172.21.15.90:0/232730540 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f073c06bc20 0x7f073c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.136+0000 7f59de000640 1 -- 172.21.15.90:0/1556600059 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f59e000ac30 con 0x7f59e806c3e0 2024-03-24T01:24:43.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.137+0000 7feb3d600640 1 --2- 172.21.15.90:0/1331323913 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7feb2006bc20 0x7feb2006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.138+0000 7fc945600640 1 -- 172.21.15.90:0/3469621265 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc938058380 con 0x7fc9400392d0 2024-03-24T01:24:43.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.137+0000 7feb3d600640 1 -- 172.21.15.90:0/1331323913 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7feb2c08ab90 con 0x7feb3804a0a0 2024-03-24T01:24:43.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.138+0000 7feb45000640 1 --2- 172.21.15.90:0/1331323913 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7feb2006bc20 0x7feb2006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.136+0000 7f68bc600640 1 -- 172.21.15.90:0/3569601014 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68b4071b20 msgr2=0x7f68b4071f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.136+0000 7f68bc600640 1 --2- 172.21.15.90:0/3569601014 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68b4071b20 0x7f68b4071f80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f68a00091e0 tx=0x7f68a002cbc0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.137+0000 7f68bc600640 1 -- 172.21.15.90:0/3569601014 shutdown_connections 2024-03-24T01:24:43.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.137+0000 7f68bc600640 1 --2- 172.21.15.90:0/3569601014 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68b415d910 0x7f68b415dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.137+0000 7f68bc600640 1 --2- 172.21.15.90:0/3569601014 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68b4071b20 0x7f68b4071f80 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.137+0000 7f68bc600640 1 -- 172.21.15.90:0/3569601014 >> 172.21.15.90:0/3569601014 conn(0x7f68b40b1cd0 msgr2=0x7f68b40cf7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:43.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.137+0000 7f68bc600640 1 -- 172.21.15.90:0/3569601014 shutdown_connections 2024-03-24T01:24:43.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.137+0000 7f68bc600640 1 -- 172.21.15.90:0/3569601014 wait complete. 2024-03-24T01:24:43.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.138+0000 7f075b400640 1 --2- 172.21.15.90:0/232730540 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f073c06bc20 0x7f073c06e0e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f074c036000 tx=0x7f074c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:43.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.138+0000 7f59de000640 1 --2- 172.21.15.90:0/1556600059 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f59c806bca0 0x7f59c806e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.138+0000 7f59de000640 1 -- 172.21.15.90:0/1556600059 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f59e008b7c0 con 0x7f59e806c3e0 2024-03-24T01:24:43.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.139+0000 7f59e5400640 1 --2- 172.21.15.90:0/1556600059 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f59c806bca0 0x7f59c806e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.139+0000 7feb45000640 1 --2- 172.21.15.90:0/1331323913 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7feb2006bc20 0x7feb2006e0e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7feb34036000 tx=0x7feb34008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:43.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.140+0000 7f68bc600640 1 Processor -- start 2024-03-24T01:24:43.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.140+0000 7f59e5400640 1 --2- 172.21.15.90:0/1556600059 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f59c806bca0 0x7f59c806e160 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f59d4036000 tx=0x7f59d4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:43.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.140+0000 7f68bc600640 1 -- start start 2024-03-24T01:24:43.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.142+0000 7f68bc600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68b406ee00 0x7f68b40c5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.142+0000 7f68bc600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68b4071b20 0x7f68b40a8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.142+0000 7f68bc600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68b415d910 0x7f68b40a90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.142+0000 7f68bc600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b4140740 con 0x7f68b415d910 2024-03-24T01:24:43.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.142+0000 7f68bc600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b41405c0 con 0x7f68b4071b20 2024-03-24T01:24:43.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.142+0000 7f68bc600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68b40b2890 con 0x7f68b406ee00 2024-03-24T01:24:43.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.142+0000 7f68b9c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68b406ee00 0x7f68b40c5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.142+0000 7f68b9c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68b406ee00 0x7f68b40c5180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.90:50418/0 (socket says 172.21.15.90:50418) 2024-03-24T01:24:43.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.142+0000 7f68b9c00640 1 -- 172.21.15.90:0/577170650 learned_addr learned my addr 172.21.15.90:0/577170650 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:43.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.142+0000 7f68ba600640 1 --2- 172.21.15.90:0/577170650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68b415d910 0x7f68b40a90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.143+0000 7f68b9200640 1 --2- 172.21.15.90:0/577170650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68b4071b20 0x7f68b40a8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.143+0000 7f68b9c00640 1 -- 172.21.15.90:0/577170650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68b4071b20 msgr2=0x7f68b40a8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.143+0000 7f68b9c00640 1 --2- 172.21.15.90:0/577170650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68b4071b20 0x7f68b40a8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.143+0000 7f68b9c00640 1 -- 172.21.15.90:0/577170650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68b415d910 msgr2=0x7f68b40a90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.144+0000 7f68b9c00640 1 --2- 172.21.15.90:0/577170650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68b415d910 0x7f68b40a90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.144+0000 7f68b9c00640 1 -- 172.21.15.90:0/577170650 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b40a9ac0 con 0x7f68b406ee00 2024-03-24T01:24:43.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.144+0000 7f68ba600640 1 --2- 172.21.15.90:0/577170650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68b415d910 0x7f68b40a90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T01:24:43.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.144+0000 7f68b9c00640 1 --2- 172.21.15.90:0/577170650 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68b406ee00 0x7f68b40c5180 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f68a800b450 tx=0x7f68a800b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:43.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.145+0000 7f68b1600640 1 -- 172.21.15.90:0/577170650 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a8015020 con 0x7f68b406ee00 2024-03-24T01:24:43.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.145+0000 7f68bc600640 1 -- 172.21.15.90:0/577170650 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68b40a9da0 con 0x7f68b406ee00 2024-03-24T01:24:43.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.145+0000 7f68b1600640 1 -- 172.21.15.90:0/577170650 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68a80118b0 con 0x7f68b406ee00 2024-03-24T01:24:43.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.145+0000 7feb3d600640 1 -- 172.21.15.90:0/1331323913 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7feb2c059420 con 0x7feb3804a0a0 2024-03-24T01:24:43.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.145+0000 7f68bc600640 1 -- 172.21.15.90:0/577170650 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68b40cbe90 con 0x7f68b406ee00 2024-03-24T01:24:43.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.145+0000 7f68b1600640 1 -- 172.21.15.90:0/577170650 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a8011ca0 con 0x7f68b406ee00 2024-03-24T01:24:43.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.147+0000 7f68bc600640 1 -- 172.21.15.90:0/577170650 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6890005290 con 0x7f68b406ee00 2024-03-24T01:24:43.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.149+0000 7f68b1600640 1 -- 172.21.15.90:0/577170650 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f68a8018020 con 0x7f68b406ee00 2024-03-24T01:24:43.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.151+0000 7f68b1600640 1 --2- 172.21.15.90:0/577170650 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f689406bbd0 0x7f689406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.151+0000 7f68b1600640 1 -- 172.21.15.90:0/577170650 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f68a808b410 con 0x7f68b406ee00 2024-03-24T01:24:43.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.152+0000 7f68b9200640 1 --2- 172.21.15.90:0/577170650 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f689406bbd0 0x7f689406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.153+0000 7f68b9200640 1 --2- 172.21.15.90:0/577170650 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f689406bbd0 0x7f689406e090 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f68a0037000 tx=0x7f68a0031000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:43.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.155+0000 7f0753400640 1 -- 172.21.15.90:0/232730540 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0748058d90 con 0x7f075c0d4250 2024-03-24T01:24:43.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.163+0000 7f68b1600640 1 -- 172.21.15.90:0/577170650 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f68a801f020 con 0x7f68b406ee00 2024-03-24T01:24:43.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.168+0000 7f59de000640 1 -- 172.21.15.90:0/1556600059 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f59e80b5ee0 con 0x7f59e806c3e0 2024-03-24T01:24:43.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.501+0000 7f8a37400640 1 -- 172.21.15.90:0/1196181580 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f8a4c0d3bf0 con 0x7f8a4c071db0 2024-03-24T01:24:43.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.502+0000 7f8a42000640 1 -- 172.21.15.90:0/1196181580 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8a38057d70 con 0x7f8a4c071db0 2024-03-24T01:24:43.503 INFO:teuthology.orchestra.run.smithi090.stdout:55834574860 2024-03-24T01:24:43.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.505+0000 7f8a37400640 1 -- 172.21.15.90:0/1196181580 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8a2c06bdd0 msgr2=0x7f8a2c06e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.506+0000 7f8a37400640 1 --2- 172.21.15.90:0/1196181580 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8a2c06bdd0 0x7f8a2c06e290 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c00a970 tx=0x7f8a3c008040 comp rx=0 tx=0).stop 2024-03-24T01:24:43.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.506+0000 7f8a37400640 1 -- 172.21.15.90:0/1196181580 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a4c071db0 msgr2=0x7f8a4c0bd590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.506+0000 7f8a37400640 1 --2- 172.21.15.90:0/1196181580 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a4c071db0 0x7f8a4c0bd590 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8a3800c810 tx=0x7f8a3800ccd0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.506+0000 7f8a37400640 1 -- 172.21.15.90:0/1196181580 shutdown_connections 2024-03-24T01:24:43.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.507+0000 7f8a37400640 1 --2- 172.21.15.90:0/1196181580 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8a2c06bdd0 0x7f8a2c06e290 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.507+0000 7f8a37400640 1 --2- 172.21.15.90:0/1196181580 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a4c071db0 0x7f8a4c0bd590 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.507+0000 7f8a37400640 1 --2- 172.21.15.90:0/1196181580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a4c165ba0 0x7f8a4c203e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.507+0000 7f8a37400640 1 --2- 172.21.15.90:0/1196181580 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a4c0719d0 0x7f8a4c0bbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.507+0000 7f8a37400640 1 -- 172.21.15.90:0/1196181580 >> 172.21.15.90:0/1196181580 conn(0x7f8a4c0d7ec0 msgr2=0x7f8a4c163e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:43.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.507+0000 7f8a37400640 1 -- 172.21.15.90:0/1196181580 shutdown_connections 2024-03-24T01:24:43.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.507+0000 7f8a37400640 1 -- 172.21.15.90:0/1196181580 wait complete. 2024-03-24T01:24:43.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.524+0000 7fd617e00640 1 -- 172.21.15.90:0/2719914000 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fd5f0005580 con 0x7fd610053cb0 2024-03-24T01:24:43.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.525+0000 7fd60ea00640 1 -- 172.21.15.90:0/2719914000 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd6080580f0 con 0x7fd610053cb0 2024-03-24T01:24:43.525 INFO:teuthology.orchestra.run.smithi090.stdout:55834574859 2024-03-24T01:24:43.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.529+0000 7fd617e00640 1 -- 172.21.15.90:0/2719914000 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd5f806bc20 msgr2=0x7fd5f806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.529+0000 7fd617e00640 1 --2- 172.21.15.90:0/2719914000 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd5f806bc20 0x7fd5f806e0e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd61004ff60 tx=0x7fd6180ab000 comp rx=0 tx=0).stop 2024-03-24T01:24:43.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.529+0000 7fd617e00640 1 -- 172.21.15.90:0/2719914000 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd610053cb0 msgr2=0x7fd6100d4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.529+0000 7fd617e00640 1 --2- 172.21.15.90:0/2719914000 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd610053cb0 0x7fd6100d4750 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd60800c730 tx=0x7fd60800cbf0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.529+0000 7fd617e00640 1 -- 172.21.15.90:0/2719914000 shutdown_connections 2024-03-24T01:24:43.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.529+0000 7fd617e00640 1 --2- 172.21.15.90:0/2719914000 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd5f806bc20 0x7fd5f806e0e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.530+0000 7fd617e00640 1 --2- 172.21.15.90:0/2719914000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd6100dfa30 0x7fd61003d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.530+0000 7fd617e00640 1 --2- 172.21.15.90:0/2719914000 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6100e3160 0x7fd6100dd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.530+0000 7fd617e00640 1 --2- 172.21.15.90:0/2719914000 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd610053cb0 0x7fd6100d4750 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.530+0000 7fd617e00640 1 -- 172.21.15.90:0/2719914000 >> 172.21.15.90:0/2719914000 conn(0x7fd610058ee0 msgr2=0x7fd6100e6c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:43.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.530+0000 7fd617e00640 1 -- 172.21.15.90:0/2719914000 shutdown_connections 2024-03-24T01:24:43.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.530+0000 7fd617e00640 1 -- 172.21.15.90:0/2719914000 wait complete. 2024-03-24T01:24:43.538 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574860 for osd.0 2024-03-24T01:24:43.538 DEBUG:teuthology.parallel:result is None 2024-03-24T01:24:43.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.550+0000 7feb46e00640 1 -- 172.21.15.90:0/1331323913 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7feb3803c2d0 con 0x7feb3804a0a0 2024-03-24T01:24:43.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.550+0000 7feb3d600640 1 -- 172.21.15.90:0/1331323913 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7feb2c058d20 con 0x7feb3804a0a0 2024-03-24T01:24:43.551 INFO:teuthology.orchestra.run.smithi090.stdout:55834574859 2024-03-24T01:24:43.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.552+0000 7feb46e00640 1 -- 172.21.15.90:0/1331323913 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7feb2006bc20 msgr2=0x7feb2006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.552+0000 7feb46e00640 1 --2- 172.21.15.90:0/1331323913 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7feb2006bc20 0x7feb2006e0e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7feb34036000 tx=0x7feb34008030 comp rx=0 tx=0).stop 2024-03-24T01:24:43.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.553+0000 7feb46e00640 1 -- 172.21.15.90:0/1331323913 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb3804a0a0 msgr2=0x7feb3817d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.553+0000 7feb46e00640 1 --2- 172.21.15.90:0/1331323913 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb3804a0a0 0x7feb3817d1b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7feb2c00b4a0 tx=0x7feb2c00b960 comp rx=0 tx=0).stop 2024-03-24T01:24:43.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.553+0000 7feb46e00640 1 -- 172.21.15.90:0/1331323913 shutdown_connections 2024-03-24T01:24:43.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.553+0000 7feb46e00640 1 --2- 172.21.15.90:0/1331323913 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7feb2006bc20 0x7feb2006e0e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.553+0000 7feb46e00640 1 --2- 172.21.15.90:0/1331323913 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb380e7a50 0x7feb380e2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.553+0000 7feb46e00640 1 --2- 172.21.15.90:0/1331323913 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb38054de0 0x7feb380e1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.553+0000 7feb46e00640 1 --2- 172.21.15.90:0/1331323913 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb3804a0a0 0x7feb3817d1b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.553+0000 7feb46e00640 1 -- 172.21.15.90:0/1331323913 >> 172.21.15.90:0/1331323913 conn(0x7feb38056cd0 msgr2=0x7feb3803c740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:43.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.553+0000 7feb46e00640 1 -- 172.21.15.90:0/1331323913 shutdown_connections 2024-03-24T01:24:43.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.553+0000 7feb46e00640 1 -- 172.21.15.90:0/1331323913 wait complete. 2024-03-24T01:24:43.557 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.1 2024-03-24T01:24:43.557 DEBUG:teuthology.parallel:result is None 2024-03-24T01:24:43.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.558+0000 7f387f200640 1 -- 172.21.15.90:0/3387204198 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f3858005580 con 0x7f3880165bc0 2024-03-24T01:24:43.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.558+0000 7f3876000640 1 -- 172.21.15.90:0/3387204198 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f387805c390 con 0x7f3880165bc0 2024-03-24T01:24:43.559 INFO:teuthology.orchestra.run.smithi090.stdout:55834574859 2024-03-24T01:24:43.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.562+0000 7f387f200640 1 -- 172.21.15.90:0/3387204198 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f386006bbd0 msgr2=0x7f386006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.562+0000 7f387f200640 1 --2- 172.21.15.90:0/3387204198 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f386006bbd0 0x7f386006e090 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3870009560 tx=0x7f3870008030 comp rx=0 tx=0).stop 2024-03-24T01:24:43.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.562+0000 7f387f200640 1 -- 172.21.15.90:0/3387204198 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3880165bc0 msgr2=0x7f38800cd650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.562+0000 7f387f200640 1 --2- 172.21.15.90:0/3387204198 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3880165bc0 0x7f38800cd650 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f38780012e0 tx=0x7f387800cb90 comp rx=0 tx=0).stop 2024-03-24T01:24:43.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.562+0000 7f387f200640 1 -- 172.21.15.90:0/3387204198 shutdown_connections 2024-03-24T01:24:43.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.562+0000 7f387f200640 1 --2- 172.21.15.90:0/3387204198 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f386006bbd0 0x7f386006e090 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.562+0000 7f387f200640 1 --2- 172.21.15.90:0/3387204198 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3880165bc0 0x7f38800cd650 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.562+0000 7f387f200640 1 --2- 172.21.15.90:0/3387204198 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3880161f20 0x7f38800cd110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.562+0000 7f387f200640 1 --2- 172.21.15.90:0/3387204198 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38800d1d60 0x7f38801649a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.562+0000 7f387f200640 1 -- 172.21.15.90:0/3387204198 >> 172.21.15.90:0/3387204198 conn(0x7f38800d0550 msgr2=0x7f38800b8420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:43.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.562+0000 7fc94ec00640 1 -- 172.21.15.90:0/3469621265 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fc940005390 con 0x7fc9400392d0 2024-03-24T01:24:43.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.562+0000 7f387f200640 1 -- 172.21.15.90:0/3387204198 shutdown_connections 2024-03-24T01:24:43.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.563+0000 7f387f200640 1 -- 172.21.15.90:0/3387204198 wait complete. 2024-03-24T01:24:43.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.563+0000 7fc945600640 1 -- 172.21.15.90:0/3469621265 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc93805c420 con 0x7fc9400392d0 2024-03-24T01:24:43.564 INFO:teuthology.orchestra.run.smithi090.stdout:55834574859 2024-03-24T01:24:43.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.566+0000 7fc94ec00640 1 -- 172.21.15.90:0/3469621265 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc92c06bc20 msgr2=0x7fc92c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.566+0000 7fc94ec00640 1 --2- 172.21.15.90:0/3469621265 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc92c06bc20 0x7fc92c06e0e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc948067e50 tx=0x7fc948068de0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.566+0000 7fc94ec00640 1 -- 172.21.15.90:0/3469621265 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9400392d0 msgr2=0x7fc9400533d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.566+0000 7fc94ec00640 1 --2- 172.21.15.90:0/3469621265 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9400392d0 0x7fc9400533d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc93800c500 tx=0x7fc93800c9c0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.566+0000 7fc94ec00640 1 -- 172.21.15.90:0/3469621265 shutdown_connections 2024-03-24T01:24:43.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.566+0000 7fc94ec00640 1 --2- 172.21.15.90:0/3469621265 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc92c06bc20 0x7fc92c06e0e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.566+0000 7fc94ec00640 1 --2- 172.21.15.90:0/3469621265 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9400501a0 0x7fc9400db830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.566+0000 7fc94ec00640 1 --2- 172.21.15.90:0/3469621265 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9400392d0 0x7fc9400533d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.566+0000 7fc94ec00640 1 --2- 172.21.15.90:0/3469621265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc94000aaa0 0x7fc940052e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.567+0000 7fc94ec00640 1 -- 172.21.15.90:0/3469621265 >> 172.21.15.90:0/3469621265 conn(0x7fc940009fb0 msgr2=0x7fc94000a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:43.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.567+0000 7fc94ec00640 1 -- 172.21.15.90:0/3469621265 shutdown_connections 2024-03-24T01:24:43.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.567+0000 7fc94ec00640 1 -- 172.21.15.90:0/3469621265 wait complete. 2024-03-24T01:24:43.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.570+0000 7f0762200640 1 -- 172.21.15.90:0/232730540 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f075c0b9720 con 0x7f075c0d4250 2024-03-24T01:24:43.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.571+0000 7f0753400640 1 -- 172.21.15.90:0/232730540 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f074805ce30 con 0x7f075c0d4250 2024-03-24T01:24:43.571 INFO:teuthology.orchestra.run.smithi090.stdout:55834574860 2024-03-24T01:24:43.574 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.7 2024-03-24T01:24:43.574 DEBUG:teuthology.parallel:result is None 2024-03-24T01:24:43.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.573+0000 7f0762200640 1 -- 172.21.15.90:0/232730540 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f073c06bc20 msgr2=0x7f073c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.573+0000 7f0762200640 1 --2- 172.21.15.90:0/232730540 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f073c06bc20 0x7f073c06e0e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f074c036000 tx=0x7f074c03c040 comp rx=0 tx=0).stop 2024-03-24T01:24:43.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.573+0000 7f0762200640 1 -- 172.21.15.90:0/232730540 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f075c0d4250 msgr2=0x7f075c153d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.573+0000 7f0762200640 1 --2- 172.21.15.90:0/232730540 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f075c0d4250 0x7f075c153d40 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f074800c820 tx=0x7f074800cc20 comp rx=0 tx=0).stop 2024-03-24T01:24:43.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.573+0000 7f0762200640 1 -- 172.21.15.90:0/232730540 shutdown_connections 2024-03-24T01:24:43.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.574+0000 7f0762200640 1 --2- 172.21.15.90:0/232730540 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f073c06bc20 0x7f073c06e0e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.574+0000 7f0762200640 1 --2- 172.21.15.90:0/232730540 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f075c0d4250 0x7f075c153d40 secure :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f074800c820 tx=0x7f074800cc20 comp rx=0 tx=0).stop 2024-03-24T01:24:43.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.574+0000 7f0762200640 1 --2- 172.21.15.90:0/232730540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f075c15dd00 0x7f075c14fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.574+0000 7f0762200640 1 --2- 172.21.15.90:0/232730540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f075c0d3e70 0x7f075c1f6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.574+0000 7f0762200640 1 -- 172.21.15.90:0/232730540 >> 172.21.15.90:0/232730540 conn(0x7f075c072c10 msgr2=0x7f075c0ce540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:43.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.574+0000 7f0762200640 1 -- 172.21.15.90:0/232730540 shutdown_connections 2024-03-24T01:24:43.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.574+0000 7f0762200640 1 -- 172.21.15.90:0/232730540 wait complete. 2024-03-24T01:24:43.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.584+0000 7f68bc600640 1 -- 172.21.15.90:0/577170650 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f6890005850 con 0x7f68b406ee00 2024-03-24T01:24:43.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.584+0000 7f68b1600640 1 -- 172.21.15.90:0/577170650 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f68a800d040 con 0x7f68b406ee00 2024-03-24T01:24:43.585 INFO:teuthology.orchestra.run.smithi090.stdout:55834574859 2024-03-24T01:24:43.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.585+0000 7f59ed000640 1 -- 172.21.15.90:0/1556600059 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f59e80b2f90 con 0x7f59e806c3e0 2024-03-24T01:24:43.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.585+0000 7f59de000640 1 -- 172.21.15.90:0/1556600059 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f59e0059f50 con 0x7f59e806c3e0 2024-03-24T01:24:43.586 INFO:teuthology.orchestra.run.smithi090.stdout:55834574860 2024-03-24T01:24:43.587 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.3 2024-03-24T01:24:43.587 DEBUG:teuthology.parallel:result is None 2024-03-24T01:24:43.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.588+0000 7f68bc600640 1 -- 172.21.15.90:0/577170650 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f689406bbd0 msgr2=0x7f689406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.588+0000 7f68bc600640 1 --2- 172.21.15.90:0/577170650 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f689406bbd0 0x7f689406e090 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f68a0037000 tx=0x7f68a0031000 comp rx=0 tx=0).stop 2024-03-24T01:24:43.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.588+0000 7f68bc600640 1 -- 172.21.15.90:0/577170650 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68b406ee00 msgr2=0x7f68b40c5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.588+0000 7f68bc600640 1 --2- 172.21.15.90:0/577170650 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68b406ee00 0x7f68b40c5180 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f68a800b450 tx=0x7f68a800b910 comp rx=0 tx=0).stop 2024-03-24T01:24:43.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.588+0000 7f59ed000640 1 -- 172.21.15.90:0/1556600059 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f59c806bca0 msgr2=0x7f59c806e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.588+0000 7f59ed000640 1 --2- 172.21.15.90:0/1556600059 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f59c806bca0 0x7f59c806e160 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f59d4036000 tx=0x7f59d4008030 comp rx=0 tx=0).stop 2024-03-24T01:24:43.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.588+0000 7f59ed000640 1 -- 172.21.15.90:0/1556600059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59e806c3e0 msgr2=0x7f59e814fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.588+0000 7f59ed000640 1 --2- 172.21.15.90:0/1556600059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59e806c3e0 0x7f59e814fed0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f59e000c730 tx=0x7f59e000cbf0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.588+0000 7f68bc600640 1 -- 172.21.15.90:0/577170650 shutdown_connections 2024-03-24T01:24:43.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.588+0000 7f68bc600640 1 --2- 172.21.15.90:0/577170650 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f689406bbd0 0x7f689406e090 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.588+0000 7f68bc600640 1 --2- 172.21.15.90:0/577170650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f68b415d910 0x7f68b40a90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.588+0000 7f68bc600640 1 --2- 172.21.15.90:0/577170650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68b4071b20 0x7f68b40a8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.588+0000 7f68bc600640 1 --2- 172.21.15.90:0/577170650 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f68b406ee00 0x7f68b40c5180 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.588+0000 7f68bc600640 1 -- 172.21.15.90:0/577170650 >> 172.21.15.90:0/577170650 conn(0x7f68b40b1cd0 msgr2=0x7f68b40b2bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:43.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.588+0000 7f59ed000640 1 -- 172.21.15.90:0/1556600059 shutdown_connections 2024-03-24T01:24:43.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.588+0000 7f59ed000640 1 --2- 172.21.15.90:0/1556600059 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f59c806bca0 0x7f59c806e160 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.588+0000 7f59ed000640 1 --2- 172.21.15.90:0/1556600059 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59e81615c0 0x7f59e815f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.588+0000 7f59ed000640 1 --2- 172.21.15.90:0/1556600059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59e809a2d0 0x7f59e80b59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.588+0000 7f59ed000640 1 --2- 172.21.15.90:0/1556600059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59e806c3e0 0x7f59e814fed0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.588+0000 7f59ed000640 1 -- 172.21.15.90:0/1556600059 >> 172.21.15.90:0/1556600059 conn(0x7f59e80b34d0 msgr2=0x7f59e80c7520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:43.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.588+0000 7f59ed000640 1 -- 172.21.15.90:0/1556600059 shutdown_connections 2024-03-24T01:24:43.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.588+0000 7f68bc600640 1 -- 172.21.15.90:0/577170650 shutdown_connections 2024-03-24T01:24:43.591 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.4 2024-03-24T01:24:43.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.588+0000 7f68bc600640 1 -- 172.21.15.90:0/577170650 wait complete. 2024-03-24T01:24:43.593 DEBUG:teuthology.parallel:result is None 2024-03-24T01:24:43.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.588+0000 7f59ed000640 1 -- 172.21.15.90:0/1556600059 wait complete. 2024-03-24T01:24:43.594 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574860 for osd.2 2024-03-24T01:24:43.594 DEBUG:teuthology.parallel:result is None 2024-03-24T01:24:43.605 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574860 for osd.5 2024-03-24T01:24:43.605 DEBUG:teuthology.parallel:result is None 2024-03-24T01:24:43.608 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.6 2024-03-24T01:24:43.609 DEBUG:teuthology.parallel:result is None 2024-03-24T01:24:43.609 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-24T01:24:43.609 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T01:24:43.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.851+0000 7ff3c8400640 1 Processor -- start 2024-03-24T01:24:43.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.851+0000 7ff3c8400640 1 -- start start 2024-03-24T01:24:43.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.853+0000 7ff3c8400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff3c00cc150 0x7ff3c00ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.853+0000 7ff3c8400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c006b7d0 0x7ff3c00b3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.853+0000 7ff3c8400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c00cd490 con 0x7ff3c01603e0 2024-03-24T01:24:43.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.853+0000 7ff3c8400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3c015f900 con 0x7ff3c00cc150 2024-03-24T01:24:43.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.853+0000 7ff3c8400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c015fbe0 con 0x7ff3c006b7d0 2024-03-24T01:24:43.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.853+0000 7ff3c6400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c006b7d0 0x7ff3c00b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.853+0000 7ff3c6400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c006b7d0 0x7ff3c00b3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.90:50434/0 (socket says 172.21.15.90:50434) 2024-03-24T01:24:43.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.853+0000 7ff3c6400640 1 -- 172.21.15.90:0/1161186720 learned_addr learned my addr 172.21.15.90:0/1161186720 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:43.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.853+0000 7ff3c5a00640 1 --2- 172.21.15.90:0/1161186720 >> v2:172.21.15.120:3300/0 conn(0x7ff3c00cc150 0x7ff3c00ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.854+0000 7ff3bfe00640 1 -- 172.21.15.90:0/1161186720 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4134127090 0 0) 0x7ff3c00cd490 con 0x7ff3c01603e0 2024-03-24T01:24:43.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.854+0000 7ff3bfe00640 1 -- 172.21.15.90:0/1161186720 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3a8003520 con 0x7ff3c01603e0 2024-03-24T01:24:43.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.854+0000 7ff3c6400640 1 -- 172.21.15.90:0/1161186720 >> v2:172.21.15.120:3300/0 conn(0x7ff3c00cc150 msgr2=0x7ff3c00ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:43.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.854+0000 7ff3c6400640 1 --2- 172.21.15.90:0/1161186720 >> v2:172.21.15.120:3300/0 conn(0x7ff3c00cc150 0x7ff3c00ccb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.854+0000 7ff3c6400640 1 -- 172.21.15.90:0/1161186720 >> v1:172.21.15.120:6789/0 conn(0x7ff3c01603e0 legacy=0x7ff3c006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.854+0000 7ff3c6400640 1 -- 172.21.15.90:0/1161186720 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c015fec0 con 0x7ff3c006b7d0 2024-03-24T01:24:43.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.855+0000 7ff3c6400640 1 --2- 172.21.15.90:0/1161186720 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c006b7d0 0x7ff3c00b3750 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8009e50 tx=0x7ff3b802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d7cabd6f1f9631b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:43.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.856+0000 7ff3bfe00640 1 -- 172.21.15.90:0/1161186720 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b802cb00 con 0x7ff3c006b7d0 2024-03-24T01:24:43.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.856+0000 7ff3bfe00640 1 -- 172.21.15.90:0/1161186720 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3b802cca0 con 0x7ff3c006b7d0 2024-03-24T01:24:43.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.856+0000 7ff3bfe00640 1 -- 172.21.15.90:0/1161186720 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b8034610 con 0x7ff3c006b7d0 2024-03-24T01:24:43.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.856+0000 7ff3c8400640 1 -- 172.21.15.90:0/1161186720 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c006b7d0 msgr2=0x7ff3c00b3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.856+0000 7ff3c8400640 1 --2- 172.21.15.90:0/1161186720 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c006b7d0 0x7ff3c00b3750 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8009e50 tx=0x7ff3b802bf60 comp rx=0 tx=0).stop 2024-03-24T01:24:43.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.856+0000 7ff3c8400640 1 -- 172.21.15.90:0/1161186720 shutdown_connections 2024-03-24T01:24:43.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.856+0000 7ff3c8400640 1 --2- 172.21.15.90:0/1161186720 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c006b7d0 0x7ff3c00b3750 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.856+0000 7ff3c8400640 1 --2- 172.21.15.90:0/1161186720 >> v2:172.21.15.120:3300/0 conn(0x7ff3c00cc150 0x7ff3c00ccb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.856+0000 7ff3c8400640 1 -- 172.21.15.90:0/1161186720 >> 172.21.15.90:0/1161186720 conn(0x7ff3c00b1770 msgr2=0x7ff3c00b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:43.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.856+0000 7ff3c8400640 1 -- 172.21.15.90:0/1161186720 shutdown_connections 2024-03-24T01:24:43.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.857+0000 7ff3c8400640 1 -- 172.21.15.90:0/1161186720 wait complete. 2024-03-24T01:24:43.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.858+0000 7ff3c8400640 1 Processor -- start 2024-03-24T01:24:43.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.858+0000 7ff3c8400640 1 -- start start 2024-03-24T01:24:43.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.859+0000 7ff3c8400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c006b7d0 0x7ff3c01f3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.859+0000 7ff3c8400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3c00cc150 0x7ff3c013eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.859+0000 7ff3c8400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3c01603e0 0x7ff3c006dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.859+0000 7ff3c8400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c01455b0 con 0x7ff3c01603e0 2024-03-24T01:24:43.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.859+0000 7ff3c8400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c0145430 con 0x7ff3c00cc150 2024-03-24T01:24:43.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.859+0000 7ff3c8400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c00b3c90 con 0x7ff3c006b7d0 2024-03-24T01:24:43.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.859+0000 7ff3c6400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3c01603e0 0x7ff3c006dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.859+0000 7ff3c6400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3c01603e0 0x7ff3c006dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60342/0 (socket says 172.21.15.90:60342) 2024-03-24T01:24:43.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.859+0000 7ff3c6400640 1 -- 172.21.15.90:0/4114937778 learned_addr learned my addr 172.21.15.90:0/4114937778 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:43.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.860+0000 7ff3c5000640 1 --2- 172.21.15.90:0/4114937778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3c00cc150 0x7ff3c013eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.860+0000 7ff3c6400640 1 -- 172.21.15.90:0/4114937778 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c006b7d0 msgr2=0x7ff3c01f3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.860+0000 7ff3c6400640 1 --2- 172.21.15.90:0/4114937778 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c006b7d0 0x7ff3c01f3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.860+0000 7ff3c6400640 1 -- 172.21.15.90:0/4114937778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3c00cc150 msgr2=0x7ff3c013eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:43.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.860+0000 7ff3c6400640 1 --2- 172.21.15.90:0/4114937778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3c00cc150 0x7ff3c013eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:43.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.860+0000 7ff3c6400640 1 -- 172.21.15.90:0/4114937778 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c0142000 con 0x7ff3c01603e0 2024-03-24T01:24:43.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.860+0000 7ff3c6400640 1 --2- 172.21.15.90:0/4114937778 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3c01603e0 0x7ff3c006dc00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8009e50 tx=0x7ff3b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:43.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.860+0000 7ff3bd600640 1 -- 172.21.15.90:0/4114937778 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b8043020 con 0x7ff3c01603e0 2024-03-24T01:24:43.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.860+0000 7ff3bd600640 1 -- 172.21.15.90:0/4114937778 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3b802ce50 con 0x7ff3c01603e0 2024-03-24T01:24:43.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.860+0000 7ff3bd600640 1 -- 172.21.15.90:0/4114937778 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b8009510 con 0x7ff3c01603e0 2024-03-24T01:24:43.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.861+0000 7ff3c8400640 1 -- 172.21.15.90:0/4114937778 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3c015d7c0 con 0x7ff3c01603e0 2024-03-24T01:24:43.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.861+0000 7ff3c8400640 1 -- 172.21.15.90:0/4114937778 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c015dd80 con 0x7ff3c01603e0 2024-03-24T01:24:43.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.862+0000 7ff3c8400640 1 -- 172.21.15.90:0/4114937778 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3c0068060 con 0x7ff3c01603e0 2024-03-24T01:24:43.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.862+0000 7ff3bd600640 1 -- 172.21.15.90:0/4114937778 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff3b8046020 con 0x7ff3c01603e0 2024-03-24T01:24:43.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.863+0000 7ff3bd600640 1 --2- 172.21.15.90:0/4114937778 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff3a006bbd0 0x7ff3a006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:43.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.864+0000 7ff3bd600640 1 -- 172.21.15.90:0/4114937778 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ff3b80ad4f0 con 0x7ff3c01603e0 2024-03-24T01:24:43.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.864+0000 7ff3c5a00640 1 --2- 172.21.15.90:0/4114937778 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff3a006bbd0 0x7ff3a006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:43.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.865+0000 7ff3c5a00640 1 --2- 172.21.15.90:0/4114937778 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff3a006bbd0 0x7ff3a006e090 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4006a90 tx=0x7ff3b4006a20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:43.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:43.868+0000 7ff3bd600640 1 -- 172.21.15.90:0/4114937778 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff3b804d030 con 0x7ff3c01603e0 2024-03-24T01:24:44.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.046+0000 7ff3c8400640 1 -- 172.21.15.90:0/4114937778 --> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff3c015df50 con 0x7ff3a006bbd0 2024-03-24T01:24:44.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.071+0000 7ff3bd600640 1 -- 172.21.15.90:0/4114937778 <== mgr.4104 v2:172.21.15.120:6832/860392634 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332535 (secure 0 0 0) 0x7ff3c015df50 con 0x7ff3a006bbd0 2024-03-24T01:24:44.076 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:24:44.078 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-24T01:24:44.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.077+0000 7ff3c8400640 1 -- 172.21.15.90:0/4114937778 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff3a006bbd0 msgr2=0x7ff3a006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:44.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.077+0000 7ff3c8400640 1 --2- 172.21.15.90:0/4114937778 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff3a006bbd0 0x7ff3a006e090 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4006a90 tx=0x7ff3b4006a20 comp rx=0 tx=0).stop 2024-03-24T01:24:44.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.077+0000 7ff3c8400640 1 -- 172.21.15.90:0/4114937778 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3c01603e0 msgr2=0x7ff3c006dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:44.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.077+0000 7ff3c8400640 1 --2- 172.21.15.90:0/4114937778 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3c01603e0 0x7ff3c006dc00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8009e50 tx=0x7ff3b8004a40 comp rx=0 tx=0).stop 2024-03-24T01:24:44.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.077+0000 7ff3c8400640 1 -- 172.21.15.90:0/4114937778 shutdown_connections 2024-03-24T01:24:44.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.077+0000 7ff3c8400640 1 --2- 172.21.15.90:0/4114937778 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff3a006bbd0 0x7ff3a006e090 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:44.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.077+0000 7ff3c8400640 1 --2- 172.21.15.90:0/4114937778 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3c01603e0 0x7ff3c006dc00 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:44.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.077+0000 7ff3c8400640 1 --2- 172.21.15.90:0/4114937778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3c00cc150 0x7ff3c013eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:44.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.077+0000 7ff3c8400640 1 --2- 172.21.15.90:0/4114937778 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3c006b7d0 0x7ff3c01f3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:44.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.077+0000 7ff3c8400640 1 -- 172.21.15.90:0/4114937778 >> 172.21.15.90:0/4114937778 conn(0x7ff3c00b1770 msgr2=0x7ff3c00ced50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:44.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.077+0000 7ff3c8400640 1 -- 172.21.15.90:0/4114937778 shutdown_connections 2024-03-24T01:24:44.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.077+0000 7ff3c8400640 1 -- 172.21.15.90:0/4114937778 wait complete. 2024-03-24T01:24:44.096 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-03-24T01:24:43.490304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219472,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755248,"statfs":{"total":773094113280,"available":772869373952,"internally_reserved":0,"allocated":3383296,"data_stored":2049160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12747,"internal_metadata":221236789},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005471"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":28,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811464+0000","last_change":"2024-03-24T01:24:09.722150+0000","last_active":"2024-03-24T01:24:17.811464+0000","last_peered":"2024-03-24T01:24:17.811464+0000","last_clean":"2024-03-24T01:24:17.811464+0000","last_became_active":"2024-03-24T01:24:09.721808+0000","last_became_peered":"2024-03-24T01:24:09.721808+0000","last_unstale":"2024-03-24T01:24:17.811464+0000","last_undegraded":"2024-03-24T01:24:17.811464+0000","last_fullsized":"2024-03-24T01:24:17.811464+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:31:46.512174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022533+0000","last_change":"2024-03-24T01:24:10.750478+0000","last_active":"2024-03-24T01:24:17.022533+0000","last_peered":"2024-03-24T01:24:17.022533+0000","last_clean":"2024-03-24T01:24:17.022533+0000","last_became_active":"2024-03-24T01:24:10.749299+0000","last_became_peered":"2024-03-24T01:24:10.749299+0000","last_unstale":"2024-03-24T01:24:17.022533+0000","last_undegraded":"2024-03-24T01:24:17.022533+0000","last_fullsized":"2024-03-24T01:24:17.022533+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:46:04.235199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021708+0000","last_change":"2024-03-24T01:24:15.662475+0000","last_active":"2024-03-24T01:24:17.021708+0000","last_peered":"2024-03-24T01:24:17.021708+0000","last_clean":"2024-03-24T01:24:17.021708+0000","last_became_active":"2024-03-24T01:24:15.661550+0000","last_became_peered":"2024-03-24T01:24:15.661550+0000","last_unstale":"2024-03-24T01:24:17.021708+0000","last_undegraded":"2024-03-24T01:24:17.021708+0000","last_fullsized":"2024-03-24T01:24:17.021708+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:14:59.259808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022711+0000","last_change":"2024-03-24T01:24:09.723434+0000","last_active":"2024-03-24T01:24:17.022711+0000","last_peered":"2024-03-24T01:24:17.022711+0000","last_clean":"2024-03-24T01:24:17.022711+0000","last_became_active":"2024-03-24T01:24:09.722439+0000","last_became_peered":"2024-03-24T01:24:09.722439+0000","last_unstale":"2024-03-24T01:24:17.022711+0000","last_undegraded":"2024-03-24T01:24:17.022711+0000","last_fullsized":"2024-03-24T01:24:17.022711+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:31:46.512174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811170+0000","last_change":"2024-03-24T01:24:10.745888+0000","last_active":"2024-03-24T01:24:17.811170+0000","last_peered":"2024-03-24T01:24:17.811170+0000","last_clean":"2024-03-24T01:24:17.811170+0000","last_became_active":"2024-03-24T01:24:10.742579+0000","last_became_peered":"2024-03-24T01:24:10.742579+0000","last_unstale":"2024-03-24T01:24:17.811170+0000","last_undegraded":"2024-03-24T01:24:17.811170+0000","last_fullsized":"2024-03-24T01:24:17.811170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:04:48.028091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.026871+0000","last_change":"2024-03-24T01:24:15.022407+0000","last_active":"2024-03-24T01:24:17.026871+0000","last_peered":"2024-03-24T01:24:17.026871+0000","last_clean":"2024-03-24T01:24:17.026871+0000","last_became_active":"2024-03-24T01:24:15.021962+0000","last_became_peered":"2024-03-24T01:24:15.021962+0000","last_unstale":"2024-03-24T01:24:17.026871+0000","last_undegraded":"2024-03-24T01:24:17.026871+0000","last_fullsized":"2024-03-24T01:24:17.026871+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:28.723031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.807761+0000","last_change":"2024-03-24T01:24:09.735868+0000","last_active":"2024-03-24T01:24:17.807761+0000","last_peered":"2024-03-24T01:24:17.807761+0000","last_clean":"2024-03-24T01:24:17.807761+0000","last_became_active":"2024-03-24T01:24:09.735573+0000","last_became_peered":"2024-03-24T01:24:09.735573+0000","last_unstale":"2024-03-24T01:24:17.807761+0000","last_undegraded":"2024-03-24T01:24:17.807761+0000","last_fullsized":"2024-03-24T01:24:17.807761+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:35:54.000044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811189+0000","last_change":"2024-03-24T01:24:10.752632+0000","last_active":"2024-03-24T01:24:17.811189+0000","last_peered":"2024-03-24T01:24:17.811189+0000","last_clean":"2024-03-24T01:24:17.811189+0000","last_became_active":"2024-03-24T01:24:10.752358+0000","last_became_peered":"2024-03-24T01:24:10.752358+0000","last_unstale":"2024-03-24T01:24:17.811189+0000","last_undegraded":"2024-03-24T01:24:17.811189+0000","last_fullsized":"2024-03-24T01:24:17.811189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:38:07.266851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.027523+0000","last_change":"2024-03-24T01:24:15.032062+0000","last_active":"2024-03-24T01:24:17.027523+0000","last_peered":"2024-03-24T01:24:17.027523+0000","last_clean":"2024-03-24T01:24:17.027523+0000","last_became_active":"2024-03-24T01:24:15.031556+0000","last_became_peered":"2024-03-24T01:24:15.031556+0000","last_unstale":"2024-03-24T01:24:17.027523+0000","last_undegraded":"2024-03-24T01:24:17.027523+0000","last_fullsized":"2024-03-24T01:24:17.027523+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:45:17.532730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.812001+0000","last_change":"2024-03-24T01:24:09.739225+0000","last_active":"2024-03-24T01:24:17.812001+0000","last_peered":"2024-03-24T01:24:17.812001+0000","last_clean":"2024-03-24T01:24:17.812001+0000","last_became_active":"2024-03-24T01:24:09.738488+0000","last_became_peered":"2024-03-24T01:24:09.738488+0000","last_unstale":"2024-03-24T01:24:17.812001+0000","last_undegraded":"2024-03-24T01:24:17.812001+0000","last_fullsized":"2024-03-24T01:24:17.812001+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:56:33.895912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811310+0000","last_change":"2024-03-24T01:24:10.753043+0000","last_active":"2024-03-24T01:24:17.811310+0000","last_peered":"2024-03-24T01:24:17.811310+0000","last_clean":"2024-03-24T01:24:17.811310+0000","last_became_active":"2024-03-24T01:24:10.751666+0000","last_became_peered":"2024-03-24T01:24:10.751666+0000","last_unstale":"2024-03-24T01:24:17.811310+0000","last_undegraded":"2024-03-24T01:24:17.811310+0000","last_fullsized":"2024-03-24T01:24:17.811310+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:02:18.904076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.027460+0000","last_change":"2024-03-24T01:24:15.021052+0000","last_active":"2024-03-24T01:24:17.027460+0000","last_peered":"2024-03-24T01:24:17.027460+0000","last_clean":"2024-03-24T01:24:17.027460+0000","last_became_active":"2024-03-24T01:24:15.020449+0000","last_became_peered":"2024-03-24T01:24:15.020449+0000","last_unstale":"2024-03-24T01:24:17.027460+0000","last_undegraded":"2024-03-24T01:24:17.027460+0000","last_fullsized":"2024-03-24T01:24:17.027460+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:26:07.104726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028245+0000","last_change":"2024-03-24T01:24:10.737914+0000","last_active":"2024-03-24T01:24:17.028245+0000","last_peered":"2024-03-24T01:24:17.028245+0000","last_clean":"2024-03-24T01:24:17.028245+0000","last_became_active":"2024-03-24T01:24:10.737652+0000","last_became_peered":"2024-03-24T01:24:10.737652+0000","last_unstale":"2024-03-24T01:24:17.028245+0000","last_undegraded":"2024-03-24T01:24:17.028245+0000","last_fullsized":"2024-03-24T01:24:17.028245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:09:31.705353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.807741+0000","last_change":"2024-03-24T01:24:09.726730+0000","last_active":"2024-03-24T01:24:17.807741+0000","last_peered":"2024-03-24T01:24:17.807741+0000","last_clean":"2024-03-24T01:24:17.807741+0000","last_became_active":"2024-03-24T01:24:09.725266+0000","last_became_peered":"2024-03-24T01:24:09.725266+0000","last_unstale":"2024-03-24T01:24:17.807741+0000","last_undegraded":"2024-03-24T01:24:17.807741+0000","last_fullsized":"2024-03-24T01:24:17.807741+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:27:50.329766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028585+0000","last_change":"2024-03-24T01:24:15.027082+0000","last_active":"2024-03-24T01:24:17.028585+0000","last_peered":"2024-03-24T01:24:17.028585+0000","last_clean":"2024-03-24T01:24:17.028585+0000","last_became_active":"2024-03-24T01:24:15.026296+0000","last_became_peered":"2024-03-24T01:24:15.026296+0000","last_unstale":"2024-03-24T01:24:17.028585+0000","last_undegraded":"2024-03-24T01:24:17.028585+0000","last_fullsized":"2024-03-24T01:24:17.028585+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:55.111215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022103+0000","last_change":"2024-03-24T01:24:10.740733+0000","last_active":"2024-03-24T01:24:17.022103+0000","last_peered":"2024-03-24T01:24:17.022103+0000","last_clean":"2024-03-24T01:24:17.022103+0000","last_became_active":"2024-03-24T01:24:10.740515+0000","last_became_peered":"2024-03-24T01:24:10.740515+0000","last_unstale":"2024-03-24T01:24:17.022103+0000","last_undegraded":"2024-03-24T01:24:17.022103+0000","last_fullsized":"2024-03-24T01:24:17.022103+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:04:48.028091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.738935+0000","last_change":"2024-03-24T01:24:09.725315+0000","last_active":"2024-03-24T01:24:17.738935+0000","last_peered":"2024-03-24T01:24:17.738935+0000","last_clean":"2024-03-24T01:24:17.738935+0000","last_became_active":"2024-03-24T01:24:09.723043+0000","last_became_peered":"2024-03-24T01:24:09.723043+0000","last_unstale":"2024-03-24T01:24:17.738935+0000","last_undegraded":"2024-03-24T01:24:17.738935+0000","last_fullsized":"2024-03-24T01:24:17.738935+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:36:52.007109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.027797+0000","last_change":"2024-03-24T01:24:15.023034+0000","last_active":"2024-03-24T01:24:17.027797+0000","last_peered":"2024-03-24T01:24:17.027797+0000","last_clean":"2024-03-24T01:24:17.027797+0000","last_became_active":"2024-03-24T01:24:15.021144+0000","last_became_peered":"2024-03-24T01:24:15.021144+0000","last_unstale":"2024-03-24T01:24:17.027797+0000","last_undegraded":"2024-03-24T01:24:17.027797+0000","last_fullsized":"2024-03-24T01:24:17.027797+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:00.868691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021560+0000","last_change":"2024-03-24T01:24:10.746085+0000","last_active":"2024-03-24T01:24:17.021560+0000","last_peered":"2024-03-24T01:24:17.021560+0000","last_clean":"2024-03-24T01:24:17.021560+0000","last_became_active":"2024-03-24T01:24:10.745493+0000","last_became_peered":"2024-03-24T01:24:10.745493+0000","last_unstale":"2024-03-24T01:24:17.021560+0000","last_undegraded":"2024-03-24T01:24:17.021560+0000","last_fullsized":"2024-03-24T01:24:17.021560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:19:51.907364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.739444+0000","last_change":"2024-03-24T01:24:09.723035+0000","last_active":"2024-03-24T01:24:17.739444+0000","last_peered":"2024-03-24T01:24:17.739444+0000","last_clean":"2024-03-24T01:24:17.739444+0000","last_became_active":"2024-03-24T01:24:09.722548+0000","last_became_peered":"2024-03-24T01:24:09.722548+0000","last_unstale":"2024-03-24T01:24:17.739444+0000","last_undegraded":"2024-03-24T01:24:17.739444+0000","last_fullsized":"2024-03-24T01:24:17.739444+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:58:17.247099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.741186+0000","last_change":"2024-03-24T01:24:15.658323+0000","last_active":"2024-03-24T01:24:17.741186+0000","last_peered":"2024-03-24T01:24:17.741186+0000","last_clean":"2024-03-24T01:24:17.741186+0000","last_became_active":"2024-03-24T01:24:15.656635+0000","last_became_peered":"2024-03-24T01:24:15.656635+0000","last_unstale":"2024-03-24T01:24:17.741186+0000","last_undegraded":"2024-03-24T01:24:17.741186+0000","last_fullsized":"2024-03-24T01:24:17.741186+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:52:30.401135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028531+0000","last_change":"2024-03-24T01:24:10.734344+0000","last_active":"2024-03-24T01:24:17.028531+0000","last_peered":"2024-03-24T01:24:17.028531+0000","last_clean":"2024-03-24T01:24:17.028531+0000","last_became_active":"2024-03-24T01:24:10.734010+0000","last_became_peered":"2024-03-24T01:24:10.734010+0000","last_unstale":"2024-03-24T01:24:17.028531+0000","last_undegraded":"2024-03-24T01:24:17.028531+0000","last_fullsized":"2024-03-24T01:24:17.028531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:36:53.019192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.808157+0000","last_change":"2024-03-24T01:24:09.729151+0000","last_active":"2024-03-24T01:24:17.808157+0000","last_peered":"2024-03-24T01:24:17.808157+0000","last_clean":"2024-03-24T01:24:17.808157+0000","last_became_active":"2024-03-24T01:24:09.728441+0000","last_became_peered":"2024-03-24T01:24:09.728441+0000","last_unstale":"2024-03-24T01:24:17.808157+0000","last_undegraded":"2024-03-24T01:24:17.808157+0000","last_fullsized":"2024-03-24T01:24:17.808157+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:33.381248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.027681+0000","last_change":"2024-03-24T01:24:15.024436+0000","last_active":"2024-03-24T01:24:17.027681+0000","last_peered":"2024-03-24T01:24:17.027681+0000","last_clean":"2024-03-24T01:24:17.027681+0000","last_became_active":"2024-03-24T01:24:15.022270+0000","last_became_peered":"2024-03-24T01:24:15.022270+0000","last_unstale":"2024-03-24T01:24:17.027681+0000","last_undegraded":"2024-03-24T01:24:17.027681+0000","last_fullsized":"2024-03-24T01:24:17.027681+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:54:28.205416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809625+0000","last_change":"2024-03-24T01:24:10.755973+0000","last_active":"2024-03-24T01:24:17.809625+0000","last_peered":"2024-03-24T01:24:17.809625+0000","last_clean":"2024-03-24T01:24:17.809625+0000","last_became_active":"2024-03-24T01:24:10.755558+0000","last_became_peered":"2024-03-24T01:24:10.755558+0000","last_unstale":"2024-03-24T01:24:17.809625+0000","last_undegraded":"2024-03-24T01:24:17.809625+0000","last_fullsized":"2024-03-24T01:24:17.809625+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:08:13.760239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.031629+0000","last_change":"2024-03-24T01:24:09.724815+0000","last_active":"2024-03-24T01:24:17.031629+0000","last_peered":"2024-03-24T01:24:17.031629+0000","last_clean":"2024-03-24T01:24:17.031629+0000","last_became_active":"2024-03-24T01:24:09.723550+0000","last_became_peered":"2024-03-24T01:24:09.723550+0000","last_unstale":"2024-03-24T01:24:17.031629+0000","last_undegraded":"2024-03-24T01:24:17.031629+0000","last_fullsized":"2024-03-24T01:24:17.031629+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:21:50.689532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028118+0000","last_change":"2024-03-24T01:24:15.651689+0000","last_active":"2024-03-24T01:24:17.028118+0000","last_peered":"2024-03-24T01:24:17.028118+0000","last_clean":"2024-03-24T01:24:17.028118+0000","last_became_active":"2024-03-24T01:24:15.651006+0000","last_became_peered":"2024-03-24T01:24:15.651006+0000","last_unstale":"2024-03-24T01:24:17.028118+0000","last_undegraded":"2024-03-24T01:24:17.028118+0000","last_fullsized":"2024-03-24T01:24:17.028118+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:52:10.223780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.023121+0000","last_change":"2024-03-24T01:24:10.767341+0000","last_active":"2024-03-24T01:24:17.023121+0000","last_peered":"2024-03-24T01:24:17.023121+0000","last_clean":"2024-03-24T01:24:17.023121+0000","last_became_active":"2024-03-24T01:24:10.767024+0000","last_became_peered":"2024-03-24T01:24:10.767024+0000","last_unstale":"2024-03-24T01:24:17.023121+0000","last_undegraded":"2024-03-24T01:24:17.023121+0000","last_fullsized":"2024-03-24T01:24:17.023121+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:35.441197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.738964+0000","last_change":"2024-03-24T01:24:09.716616+0000","last_active":"2024-03-24T01:24:17.738964+0000","last_peered":"2024-03-24T01:24:17.738964+0000","last_clean":"2024-03-24T01:24:17.738964+0000","last_became_active":"2024-03-24T01:24:09.716363+0000","last_became_peered":"2024-03-24T01:24:09.716363+0000","last_unstale":"2024-03-24T01:24:17.738964+0000","last_undegraded":"2024-03-24T01:24:17.738964+0000","last_fullsized":"2024-03-24T01:24:17.738964+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:33.381248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021124+0000","last_change":"2024-03-24T01:24:15.667639+0000","last_active":"2024-03-24T01:24:17.021124+0000","last_peered":"2024-03-24T01:24:17.021124+0000","last_clean":"2024-03-24T01:24:17.021124+0000","last_became_active":"2024-03-24T01:24:15.664579+0000","last_became_peered":"2024-03-24T01:24:15.664579+0000","last_unstale":"2024-03-24T01:24:17.021124+0000","last_undegraded":"2024-03-24T01:24:17.021124+0000","last_fullsized":"2024-03-24T01:24:17.021124+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:43:54.773518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.740372+0000","last_change":"2024-03-24T01:24:10.750277+0000","last_active":"2024-03-24T01:24:17.740372+0000","last_peered":"2024-03-24T01:24:17.740372+0000","last_clean":"2024-03-24T01:24:17.740372+0000","last_became_active":"2024-03-24T01:24:10.748912+0000","last_became_peered":"2024-03-24T01:24:10.748912+0000","last_unstale":"2024-03-24T01:24:17.740372+0000","last_undegraded":"2024-03-24T01:24:17.740372+0000","last_fullsized":"2024-03-24T01:24:17.740372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:19:51.907364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.812643+0000","last_change":"2024-03-24T01:24:09.723002+0000","last_active":"2024-03-24T01:24:17.812643+0000","last_peered":"2024-03-24T01:24:17.812643+0000","last_clean":"2024-03-24T01:24:17.812643+0000","last_became_active":"2024-03-24T01:24:09.722711+0000","last_became_peered":"2024-03-24T01:24:09.722711+0000","last_unstale":"2024-03-24T01:24:17.812643+0000","last_undegraded":"2024-03-24T01:24:17.812643+0000","last_fullsized":"2024-03-24T01:24:17.812643+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:44:56.790262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809628+0000","last_change":"2024-03-24T01:24:15.657482+0000","last_active":"2024-03-24T01:24:17.809628+0000","last_peered":"2024-03-24T01:24:17.809628+0000","last_clean":"2024-03-24T01:24:17.809628+0000","last_became_active":"2024-03-24T01:24:15.656331+0000","last_became_peered":"2024-03-24T01:24:15.656331+0000","last_unstale":"2024-03-24T01:24:17.809628+0000","last_undegraded":"2024-03-24T01:24:17.809628+0000","last_fullsized":"2024-03-24T01:24:17.809628+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:00.868691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811496+0000","last_change":"2024-03-24T01:24:10.753417+0000","last_active":"2024-03-24T01:24:17.811496+0000","last_peered":"2024-03-24T01:24:17.811496+0000","last_clean":"2024-03-24T01:24:17.811496+0000","last_became_active":"2024-03-24T01:24:10.750938+0000","last_became_peered":"2024-03-24T01:24:10.750938+0000","last_unstale":"2024-03-24T01:24:17.811496+0000","last_undegraded":"2024-03-24T01:24:17.811496+0000","last_fullsized":"2024-03-24T01:24:17.811496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:29:34.357876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.740584+0000","last_change":"2024-03-24T01:24:09.716278+0000","last_active":"2024-03-24T01:24:17.740584+0000","last_peered":"2024-03-24T01:24:17.740584+0000","last_clean":"2024-03-24T01:24:17.740584+0000","last_became_active":"2024-03-24T01:24:09.715984+0000","last_became_peered":"2024-03-24T01:24:09.715984+0000","last_unstale":"2024-03-24T01:24:17.740584+0000","last_undegraded":"2024-03-24T01:24:17.740584+0000","last_fullsized":"2024-03-24T01:24:17.740584+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:44:32.492325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.739876+0000","last_change":"2024-03-24T01:24:15.024902+0000","last_active":"2024-03-24T01:24:17.739876+0000","last_peered":"2024-03-24T01:24:17.739876+0000","last_clean":"2024-03-24T01:24:17.739876+0000","last_became_active":"2024-03-24T01:24:15.024379+0000","last_became_peered":"2024-03-24T01:24:15.024379+0000","last_unstale":"2024-03-24T01:24:17.739876+0000","last_undegraded":"2024-03-24T01:24:17.739876+0000","last_fullsized":"2024-03-24T01:24:17.739876+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:26:07.104726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028172+0000","last_change":"2024-03-24T01:24:10.730294+0000","last_active":"2024-03-24T01:24:17.028172+0000","last_peered":"2024-03-24T01:24:17.028172+0000","last_clean":"2024-03-24T01:24:17.028172+0000","last_became_active":"2024-03-24T01:24:10.727422+0000","last_became_peered":"2024-03-24T01:24:10.727422+0000","last_unstale":"2024-03-24T01:24:17.028172+0000","last_undegraded":"2024-03-24T01:24:17.028172+0000","last_fullsized":"2024-03-24T01:24:17.028172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:34.393331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811092+0000","last_change":"2024-03-24T01:24:09.736152+0000","last_active":"2024-03-24T01:24:17.811092+0000","last_peered":"2024-03-24T01:24:17.811092+0000","last_clean":"2024-03-24T01:24:17.811092+0000","last_became_active":"2024-03-24T01:24:09.735870+0000","last_became_peered":"2024-03-24T01:24:09.735870+0000","last_unstale":"2024-03-24T01:24:17.811092+0000","last_undegraded":"2024-03-24T01:24:17.811092+0000","last_fullsized":"2024-03-24T01:24:17.811092+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:50:31.967556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.740556+0000","last_change":"2024-03-24T01:24:15.027111+0000","last_active":"2024-03-24T01:24:17.740556+0000","last_peered":"2024-03-24T01:24:17.740556+0000","last_clean":"2024-03-24T01:24:17.740556+0000","last_became_active":"2024-03-24T01:24:15.025568+0000","last_became_peered":"2024-03-24T01:24:15.025568+0000","last_unstale":"2024-03-24T01:24:17.740556+0000","last_undegraded":"2024-03-24T01:24:17.740556+0000","last_fullsized":"2024-03-24T01:24:17.740556+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:54:28.205416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.739727+0000","last_change":"2024-03-24T01:24:10.744796+0000","last_active":"2024-03-24T01:24:17.739727+0000","last_peered":"2024-03-24T01:24:17.739727+0000","last_clean":"2024-03-24T01:24:17.739727+0000","last_became_active":"2024-03-24T01:24:10.744533+0000","last_became_peered":"2024-03-24T01:24:10.744533+0000","last_unstale":"2024-03-24T01:24:17.739727+0000","last_undegraded":"2024-03-24T01:24:17.739727+0000","last_fullsized":"2024-03-24T01:24:17.739727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:09:31.705353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.807544+0000","last_change":"2024-03-24T01:24:09.717922+0000","last_active":"2024-03-24T01:24:17.807544+0000","last_peered":"2024-03-24T01:24:17.807544+0000","last_clean":"2024-03-24T01:24:17.807544+0000","last_became_active":"2024-03-24T01:24:09.716468+0000","last_became_peered":"2024-03-24T01:24:09.716468+0000","last_unstale":"2024-03-24T01:24:17.807544+0000","last_undegraded":"2024-03-24T01:24:17.807544+0000","last_fullsized":"2024-03-24T01:24:17.807544+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:50:57.480847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.741366+0000","last_change":"2024-03-24T01:24:15.028389+0000","last_active":"2024-03-24T01:24:17.741366+0000","last_peered":"2024-03-24T01:24:17.741366+0000","last_clean":"2024-03-24T01:24:17.741366+0000","last_became_active":"2024-03-24T01:24:15.027052+0000","last_became_peered":"2024-03-24T01:24:15.027052+0000","last_unstale":"2024-03-24T01:24:17.741366+0000","last_undegraded":"2024-03-24T01:24:17.741366+0000","last_fullsized":"2024-03-24T01:24:17.741366+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:00.868691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.808527+0000","last_change":"2024-03-24T01:24:10.726234+0000","last_active":"2024-03-24T01:24:17.808527+0000","last_peered":"2024-03-24T01:24:17.808527+0000","last_clean":"2024-03-24T01:24:17.808527+0000","last_became_active":"2024-03-24T01:24:10.725471+0000","last_became_peered":"2024-03-24T01:24:10.725471+0000","last_unstale":"2024-03-24T01:24:17.808527+0000","last_undegraded":"2024-03-24T01:24:17.808527+0000","last_fullsized":"2024-03-24T01:24:17.808527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:55:14.169541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809601+0000","last_change":"2024-03-24T01:24:09.726977+0000","last_active":"2024-03-24T01:24:17.809601+0000","last_peered":"2024-03-24T01:24:17.809601+0000","last_clean":"2024-03-24T01:24:17.809601+0000","last_became_active":"2024-03-24T01:24:09.726515+0000","last_became_peered":"2024-03-24T01:24:09.726515+0000","last_unstale":"2024-03-24T01:24:17.809601+0000","last_undegraded":"2024-03-24T01:24:17.809601+0000","last_fullsized":"2024-03-24T01:24:17.809601+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:58:17.247099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809443+0000","last_change":"2024-03-24T01:24:15.662691+0000","last_active":"2024-03-24T01:24:17.809443+0000","last_peered":"2024-03-24T01:24:17.809443+0000","last_clean":"2024-03-24T01:24:17.809443+0000","last_became_active":"2024-03-24T01:24:15.657278+0000","last_became_peered":"2024-03-24T01:24:15.657278+0000","last_unstale":"2024-03-24T01:24:17.809443+0000","last_undegraded":"2024-03-24T01:24:17.809443+0000","last_fullsized":"2024-03-24T01:24:17.809443+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:54:28.205416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021915+0000","last_change":"2024-03-24T01:24:10.744706+0000","last_active":"2024-03-24T01:24:17.021915+0000","last_peered":"2024-03-24T01:24:17.021915+0000","last_clean":"2024-03-24T01:24:17.021915+0000","last_became_active":"2024-03-24T01:24:10.744433+0000","last_became_peered":"2024-03-24T01:24:10.744433+0000","last_unstale":"2024-03-24T01:24:17.021915+0000","last_undegraded":"2024-03-24T01:24:17.021915+0000","last_fullsized":"2024-03-24T01:24:17.021915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:09:31.705353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809513+0000","last_change":"2024-03-24T01:24:09.737525+0000","last_active":"2024-03-24T01:24:17.809513+0000","last_peered":"2024-03-24T01:24:17.809513+0000","last_clean":"2024-03-24T01:24:17.809513+0000","last_became_active":"2024-03-24T01:24:09.737206+0000","last_became_peered":"2024-03-24T01:24:09.737206+0000","last_unstale":"2024-03-24T01:24:17.809513+0000","last_undegraded":"2024-03-24T01:24:17.809513+0000","last_fullsized":"2024-03-24T01:24:17.809513+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:44:32.492325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":179,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.808453+0000","last_change":"2024-03-24T01:24:01.545855+0000","last_active":"2024-03-24T01:24:17.808453+0000","last_peered":"2024-03-24T01:24:17.808453+0000","last_clean":"2024-03-24T01:24:17.808453+0000","last_became_active":"2024-03-24T01:24:01.543811+0000","last_became_peered":"2024-03-24T01:24:01.543811+0000","last_unstale":"2024-03-24T01:24:17.808453+0000","last_undegraded":"2024-03-24T01:24:17.808453+0000","last_fullsized":"2024-03-24T01:24:17.808453+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_clean_scrub_stamp":"2024-03-24T01:23:59.861564+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:03.682631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.031327+0000","last_change":"2024-03-24T01:24:15.656774+0000","last_active":"2024-03-24T01:24:17.031327+0000","last_peered":"2024-03-24T01:24:17.031327+0000","last_clean":"2024-03-24T01:24:17.031327+0000","last_became_active":"2024-03-24T01:24:15.656377+0000","last_became_peered":"2024-03-24T01:24:15.656377+0000","last_unstale":"2024-03-24T01:24:17.031327+0000","last_undegraded":"2024-03-24T01:24:17.031327+0000","last_fullsized":"2024-03-24T01:24:17.031327+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:34:04.384239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.812176+0000","last_change":"2024-03-24T01:24:10.766537+0000","last_active":"2024-03-24T01:24:17.812176+0000","last_peered":"2024-03-24T01:24:17.812176+0000","last_clean":"2024-03-24T01:24:17.812176+0000","last_became_active":"2024-03-24T01:24:10.765713+0000","last_became_peered":"2024-03-24T01:24:10.765713+0000","last_unstale":"2024-03-24T01:24:17.812176+0000","last_undegraded":"2024-03-24T01:24:17.812176+0000","last_fullsized":"2024-03-24T01:24:17.812176+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:18:24.974821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.810980+0000","last_change":"2024-03-24T01:24:09.731795+0000","last_active":"2024-03-24T01:24:17.810980+0000","last_peered":"2024-03-24T01:24:17.810980+0000","last_clean":"2024-03-24T01:24:17.810980+0000","last_became_active":"2024-03-24T01:24:09.731547+0000","last_became_peered":"2024-03-24T01:24:09.731547+0000","last_unstale":"2024-03-24T01:24:17.810980+0000","last_undegraded":"2024-03-24T01:24:17.810980+0000","last_fullsized":"2024-03-24T01:24:17.810980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:31:46.512174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028049+0000","last_change":"2024-03-24T01:24:15.656206+0000","last_active":"2024-03-24T01:24:17.028049+0000","last_peered":"2024-03-24T01:24:17.028049+0000","last_clean":"2024-03-24T01:24:17.028049+0000","last_became_active":"2024-03-24T01:24:15.655557+0000","last_became_peered":"2024-03-24T01:24:15.655557+0000","last_unstale":"2024-03-24T01:24:17.028049+0000","last_undegraded":"2024-03-24T01:24:17.028049+0000","last_fullsized":"2024-03-24T01:24:17.028049+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:18:29.263700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811389+0000","last_change":"2024-03-24T01:24:10.750476+0000","last_active":"2024-03-24T01:24:17.811389+0000","last_peered":"2024-03-24T01:24:17.811389+0000","last_clean":"2024-03-24T01:24:17.811389+0000","last_became_active":"2024-03-24T01:24:10.750171+0000","last_became_peered":"2024-03-24T01:24:10.750171+0000","last_unstale":"2024-03-24T01:24:17.811389+0000","last_undegraded":"2024-03-24T01:24:17.811389+0000","last_fullsized":"2024-03-24T01:24:17.811389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:37:21.153754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.739005+0000","last_change":"2024-03-24T01:24:09.718128+0000","last_active":"2024-03-24T01:24:17.739005+0000","last_peered":"2024-03-24T01:24:17.739005+0000","last_clean":"2024-03-24T01:24:17.739005+0000","last_became_active":"2024-03-24T01:24:09.717669+0000","last_became_peered":"2024-03-24T01:24:09.717669+0000","last_unstale":"2024-03-24T01:24:17.739005+0000","last_undegraded":"2024-03-24T01:24:17.739005+0000","last_fullsized":"2024-03-24T01:24:17.739005+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:21:50.689532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.810011+0000","last_change":"2024-03-24T01:24:15.033291+0000","last_active":"2024-03-24T01:24:17.810011+0000","last_peered":"2024-03-24T01:24:17.810011+0000","last_clean":"2024-03-24T01:24:17.810011+0000","last_became_active":"2024-03-24T01:24:15.032790+0000","last_became_peered":"2024-03-24T01:24:15.032790+0000","last_unstale":"2024-03-24T01:24:17.810011+0000","last_undegraded":"2024-03-24T01:24:17.810011+0000","last_fullsized":"2024-03-24T01:24:17.810011+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:52:10.223780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.813313+0000","last_change":"2024-03-24T01:24:10.765266+0000","last_active":"2024-03-24T01:24:17.813313+0000","last_peered":"2024-03-24T01:24:17.813313+0000","last_clean":"2024-03-24T01:24:17.813313+0000","last_became_active":"2024-03-24T01:24:10.764758+0000","last_became_peered":"2024-03-24T01:24:10.764758+0000","last_unstale":"2024-03-24T01:24:17.813313+0000","last_undegraded":"2024-03-24T01:24:17.813313+0000","last_fullsized":"2024-03-24T01:24:17.813313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:34:00.095359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.807653+0000","last_change":"2024-03-24T01:24:09.714389+0000","last_active":"2024-03-24T01:24:17.807653+0000","last_peered":"2024-03-24T01:24:17.807653+0000","last_clean":"2024-03-24T01:24:17.807653+0000","last_became_active":"2024-03-24T01:24:09.713985+0000","last_became_peered":"2024-03-24T01:24:09.713985+0000","last_unstale":"2024-03-24T01:24:17.807653+0000","last_undegraded":"2024-03-24T01:24:17.807653+0000","last_fullsized":"2024-03-24T01:24:17.807653+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:50:31.967556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809004+0000","last_change":"2024-03-24T01:24:15.656465+0000","last_active":"2024-03-24T01:24:17.809004+0000","last_peered":"2024-03-24T01:24:17.809004+0000","last_clean":"2024-03-24T01:24:17.809004+0000","last_became_active":"2024-03-24T01:24:15.655849+0000","last_became_peered":"2024-03-24T01:24:15.655849+0000","last_unstale":"2024-03-24T01:24:17.809004+0000","last_undegraded":"2024-03-24T01:24:17.809004+0000","last_fullsized":"2024-03-24T01:24:17.809004+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:00.868691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.812175+0000","last_change":"2024-03-24T01:24:10.767829+0000","last_active":"2024-03-24T01:24:17.812175+0000","last_peered":"2024-03-24T01:24:17.812175+0000","last_clean":"2024-03-24T01:24:17.812175+0000","last_became_active":"2024-03-24T01:24:10.766346+0000","last_became_peered":"2024-03-24T01:24:10.766346+0000","last_unstale":"2024-03-24T01:24:17.812175+0000","last_undegraded":"2024-03-24T01:24:17.812175+0000","last_fullsized":"2024-03-24T01:24:17.812175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:24.434151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028899+0000","last_change":"2024-03-24T01:24:09.723282+0000","last_active":"2024-03-24T01:24:17.028899+0000","last_peered":"2024-03-24T01:24:17.028899+0000","last_clean":"2024-03-24T01:24:17.028899+0000","last_became_active":"2024-03-24T01:24:09.722727+0000","last_became_peered":"2024-03-24T01:24:09.722727+0000","last_unstale":"2024-03-24T01:24:17.028899+0000","last_undegraded":"2024-03-24T01:24:17.028899+0000","last_fullsized":"2024-03-24T01:24:17.028899+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:40:14.201223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.027169+0000","last_change":"2024-03-24T01:24:15.027468+0000","last_active":"2024-03-24T01:24:17.027169+0000","last_peered":"2024-03-24T01:24:17.027169+0000","last_clean":"2024-03-24T01:24:17.027169+0000","last_became_active":"2024-03-24T01:24:15.027086+0000","last_became_peered":"2024-03-24T01:24:15.027086+0000","last_unstale":"2024-03-24T01:24:17.027169+0000","last_undegraded":"2024-03-24T01:24:17.027169+0000","last_fullsized":"2024-03-24T01:24:17.027169+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:34:04.384239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809223+0000","last_change":"2024-03-24T01:24:10.756853+0000","last_active":"2024-03-24T01:24:17.809223+0000","last_peered":"2024-03-24T01:24:17.809223+0000","last_clean":"2024-03-24T01:24:17.809223+0000","last_became_active":"2024-03-24T01:24:10.756575+0000","last_became_peered":"2024-03-24T01:24:10.756575+0000","last_unstale":"2024-03-24T01:24:17.809223+0000","last_undegraded":"2024-03-24T01:24:17.809223+0000","last_fullsized":"2024-03-24T01:24:17.809223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:26:02.815847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.808921+0000","last_change":"2024-03-24T01:24:09.720936+0000","last_active":"2024-03-24T01:24:17.808921+0000","last_peered":"2024-03-24T01:24:17.808921+0000","last_clean":"2024-03-24T01:24:17.808921+0000","last_became_active":"2024-03-24T01:24:09.720598+0000","last_became_peered":"2024-03-24T01:24:09.720598+0000","last_unstale":"2024-03-24T01:24:17.808921+0000","last_undegraded":"2024-03-24T01:24:17.808921+0000","last_fullsized":"2024-03-24T01:24:17.808921+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:44:32.492325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809525+0000","last_change":"2024-03-24T01:24:15.027473+0000","last_active":"2024-03-24T01:24:17.809525+0000","last_peered":"2024-03-24T01:24:17.809525+0000","last_clean":"2024-03-24T01:24:17.809525+0000","last_became_active":"2024-03-24T01:24:15.025176+0000","last_became_peered":"2024-03-24T01:24:15.025176+0000","last_unstale":"2024-03-24T01:24:17.809525+0000","last_undegraded":"2024-03-24T01:24:17.809525+0000","last_fullsized":"2024-03-24T01:24:17.809525+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:52:30.401135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.030178+0000","last_change":"2024-03-24T01:24:10.737343+0000","last_active":"2024-03-24T01:24:17.030178+0000","last_peered":"2024-03-24T01:24:17.030178+0000","last_clean":"2024-03-24T01:24:17.030178+0000","last_became_active":"2024-03-24T01:24:10.737040+0000","last_became_peered":"2024-03-24T01:24:10.737040+0000","last_unstale":"2024-03-24T01:24:17.030178+0000","last_undegraded":"2024-03-24T01:24:17.030178+0000","last_fullsized":"2024-03-24T01:24:17.030178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:54:57.105205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.029522+0000","last_change":"2024-03-24T01:24:09.726911+0000","last_active":"2024-03-24T01:24:17.029522+0000","last_peered":"2024-03-24T01:24:17.029522+0000","last_clean":"2024-03-24T01:24:17.029522+0000","last_became_active":"2024-03-24T01:24:09.726578+0000","last_became_peered":"2024-03-24T01:24:09.726578+0000","last_unstale":"2024-03-24T01:24:17.029522+0000","last_undegraded":"2024-03-24T01:24:17.029522+0000","last_fullsized":"2024-03-24T01:24:17.029522+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:44:32.492325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.023074+0000","last_change":"2024-03-24T01:24:15.660631+0000","last_active":"2024-03-24T01:24:17.023074+0000","last_peered":"2024-03-24T01:24:17.023074+0000","last_clean":"2024-03-24T01:24:17.023074+0000","last_became_active":"2024-03-24T01:24:15.656242+0000","last_became_peered":"2024-03-24T01:24:15.656242+0000","last_unstale":"2024-03-24T01:24:17.023074+0000","last_undegraded":"2024-03-24T01:24:17.023074+0000","last_fullsized":"2024-03-24T01:24:17.023074+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:10:41.483921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811861+0000","last_change":"2024-03-24T01:24:10.762123+0000","last_active":"2024-03-24T01:24:17.811861+0000","last_peered":"2024-03-24T01:24:17.811861+0000","last_clean":"2024-03-24T01:24:17.811861+0000","last_became_active":"2024-03-24T01:24:10.761599+0000","last_became_peered":"2024-03-24T01:24:10.761599+0000","last_unstale":"2024-03-24T01:24:17.811861+0000","last_undegraded":"2024-03-24T01:24:17.811861+0000","last_fullsized":"2024-03-24T01:24:17.811861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:54:23.916536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809077+0000","last_change":"2024-03-24T01:24:09.735604+0000","last_active":"2024-03-24T01:24:17.809077+0000","last_peered":"2024-03-24T01:24:17.809077+0000","last_clean":"2024-03-24T01:24:17.809077+0000","last_became_active":"2024-03-24T01:24:09.735373+0000","last_became_peered":"2024-03-24T01:24:09.735373+0000","last_unstale":"2024-03-24T01:24:17.809077+0000","last_undegraded":"2024-03-24T01:24:17.809077+0000","last_fullsized":"2024-03-24T01:24:17.809077+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:36:52.007109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022699+0000","last_change":"2024-03-24T01:24:15.660466+0000","last_active":"2024-03-24T01:24:17.022699+0000","last_peered":"2024-03-24T01:24:17.022699+0000","last_clean":"2024-03-24T01:24:17.022699+0000","last_became_active":"2024-03-24T01:24:15.659265+0000","last_became_peered":"2024-03-24T01:24:15.659265+0000","last_unstale":"2024-03-24T01:24:17.022699+0000","last_undegraded":"2024-03-24T01:24:17.022699+0000","last_fullsized":"2024-03-24T01:24:17.022699+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:56.567936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809258+0000","last_change":"2024-03-24T01:24:10.762021+0000","last_active":"2024-03-24T01:24:17.809258+0000","last_peered":"2024-03-24T01:24:17.809258+0000","last_clean":"2024-03-24T01:24:17.809258+0000","last_became_active":"2024-03-24T01:24:10.761759+0000","last_became_peered":"2024-03-24T01:24:10.761759+0000","last_unstale":"2024-03-24T01:24:17.809258+0000","last_undegraded":"2024-03-24T01:24:17.809258+0000","last_fullsized":"2024-03-24T01:24:17.809258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:49:42.059380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.807512+0000","last_change":"2024-03-24T01:24:09.726149+0000","last_active":"2024-03-24T01:24:17.807512+0000","last_peered":"2024-03-24T01:24:17.807512+0000","last_clean":"2024-03-24T01:24:17.807512+0000","last_became_active":"2024-03-24T01:24:09.725594+0000","last_became_peered":"2024-03-24T01:24:09.725594+0000","last_unstale":"2024-03-24T01:24:17.807512+0000","last_undegraded":"2024-03-24T01:24:17.807512+0000","last_fullsized":"2024-03-24T01:24:17.807512+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:28:34.408651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811101+0000","last_change":"2024-03-24T01:24:15.655846+0000","last_active":"2024-03-24T01:24:17.811101+0000","last_peered":"2024-03-24T01:24:17.811101+0000","last_clean":"2024-03-24T01:24:17.811101+0000","last_became_active":"2024-03-24T01:24:15.655258+0000","last_became_peered":"2024-03-24T01:24:15.655258+0000","last_unstale":"2024-03-24T01:24:17.811101+0000","last_undegraded":"2024-03-24T01:24:17.811101+0000","last_fullsized":"2024-03-24T01:24:17.811101+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:52:10.223780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811833+0000","last_change":"2024-03-24T01:24:10.769350+0000","last_active":"2024-03-24T01:24:17.811833+0000","last_peered":"2024-03-24T01:24:17.811833+0000","last_clean":"2024-03-24T01:24:17.811833+0000","last_became_active":"2024-03-24T01:24:10.766162+0000","last_became_peered":"2024-03-24T01:24:10.766162+0000","last_unstale":"2024-03-24T01:24:17.811833+0000","last_undegraded":"2024-03-24T01:24:17.811833+0000","last_fullsized":"2024-03-24T01:24:17.811833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:38:07.266851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'10","reported_seq":34,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:28.484782+0000","last_change":"2024-03-24T01:24:09.716226+0000","last_active":"2024-03-24T01:24:28.484782+0000","last_peered":"2024-03-24T01:24:28.484782+0000","last_clean":"2024-03-24T01:24:28.484782+0000","last_became_active":"2024-03-24T01:24:09.715772+0000","last_became_peered":"2024-03-24T01:24:09.715772+0000","last_unstale":"2024-03-24T01:24:28.484782+0000","last_undegraded":"2024-03-24T01:24:28.484782+0000","last_fullsized":"2024-03-24T01:24:28.484782+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:50:31.967556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.740168+0000","last_change":"2024-03-24T01:24:15.024014+0000","last_active":"2024-03-24T01:24:17.740168+0000","last_peered":"2024-03-24T01:24:17.740168+0000","last_clean":"2024-03-24T01:24:17.740168+0000","last_became_active":"2024-03-24T01:24:15.023619+0000","last_became_peered":"2024-03-24T01:24:15.023619+0000","last_unstale":"2024-03-24T01:24:17.740168+0000","last_undegraded":"2024-03-24T01:24:17.740168+0000","last_fullsized":"2024-03-24T01:24:17.740168+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:45:17.532730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028200+0000","last_change":"2024-03-24T01:24:10.745876+0000","last_active":"2024-03-24T01:24:17.028200+0000","last_peered":"2024-03-24T01:24:17.028200+0000","last_clean":"2024-03-24T01:24:17.028200+0000","last_became_active":"2024-03-24T01:24:10.745539+0000","last_became_peered":"2024-03-24T01:24:10.745539+0000","last_unstale":"2024-03-24T01:24:17.028200+0000","last_undegraded":"2024-03-24T01:24:17.028200+0000","last_fullsized":"2024-03-24T01:24:17.028200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:56:34.907995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.740552+0000","last_change":"2024-03-24T01:24:09.736074+0000","last_active":"2024-03-24T01:24:17.740552+0000","last_peered":"2024-03-24T01:24:17.740552+0000","last_clean":"2024-03-24T01:24:17.740552+0000","last_became_active":"2024-03-24T01:24:09.735807+0000","last_became_peered":"2024-03-24T01:24:09.735807+0000","last_unstale":"2024-03-24T01:24:17.740552+0000","last_undegraded":"2024-03-24T01:24:17.740552+0000","last_fullsized":"2024-03-24T01:24:17.740552+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:50:31.967556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.027828+0000","last_change":"2024-03-24T01:24:15.031909+0000","last_active":"2024-03-24T01:24:17.027828+0000","last_peered":"2024-03-24T01:24:17.027828+0000","last_clean":"2024-03-24T01:24:17.027828+0000","last_became_active":"2024-03-24T01:24:15.031560+0000","last_became_peered":"2024-03-24T01:24:15.031560+0000","last_unstale":"2024-03-24T01:24:17.027828+0000","last_undegraded":"2024-03-24T01:24:17.027828+0000","last_fullsized":"2024-03-24T01:24:17.027828+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:52:30.401135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811414+0000","last_change":"2024-03-24T01:24:10.753950+0000","last_active":"2024-03-24T01:24:17.811414+0000","last_peered":"2024-03-24T01:24:17.811414+0000","last_clean":"2024-03-24T01:24:17.811414+0000","last_became_active":"2024-03-24T01:24:10.742209+0000","last_became_peered":"2024-03-24T01:24:10.742209+0000","last_unstale":"2024-03-24T01:24:17.811414+0000","last_undegraded":"2024-03-24T01:24:17.811414+0000","last_fullsized":"2024-03-24T01:24:17.811414+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:17.138453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809846+0000","last_change":"2024-03-24T01:24:09.735745+0000","last_active":"2024-03-24T01:24:17.809846+0000","last_peered":"2024-03-24T01:24:17.809846+0000","last_clean":"2024-03-24T01:24:17.809846+0000","last_became_active":"2024-03-24T01:24:09.735438+0000","last_became_peered":"2024-03-24T01:24:09.735438+0000","last_unstale":"2024-03-24T01:24:17.809846+0000","last_undegraded":"2024-03-24T01:24:17.809846+0000","last_fullsized":"2024-03-24T01:24:17.809846+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:58:17.247099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021471+0000","last_change":"2024-03-24T01:24:15.014619+0000","last_active":"2024-03-24T01:24:17.021471+0000","last_peered":"2024-03-24T01:24:17.021471+0000","last_clean":"2024-03-24T01:24:17.021471+0000","last_became_active":"2024-03-24T01:24:15.010859+0000","last_became_peered":"2024-03-24T01:24:15.010859+0000","last_unstale":"2024-03-24T01:24:17.021471+0000","last_undegraded":"2024-03-24T01:24:17.021471+0000","last_fullsized":"2024-03-24T01:24:17.021471+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:45:23.567638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811149+0000","last_change":"2024-03-24T01:24:10.767592+0000","last_active":"2024-03-24T01:24:17.811149+0000","last_peered":"2024-03-24T01:24:17.811149+0000","last_clean":"2024-03-24T01:24:17.811149+0000","last_became_active":"2024-03-24T01:24:10.767264+0000","last_became_peered":"2024-03-24T01:24:10.767264+0000","last_unstale":"2024-03-24T01:24:17.811149+0000","last_undegraded":"2024-03-24T01:24:17.811149+0000","last_fullsized":"2024-03-24T01:24:17.811149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:18:24.974821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.031354+0000","last_change":"2024-03-24T01:24:09.738634+0000","last_active":"2024-03-24T01:24:17.031354+0000","last_peered":"2024-03-24T01:24:17.031354+0000","last_clean":"2024-03-24T01:24:17.031354+0000","last_became_active":"2024-03-24T01:24:09.738308+0000","last_became_peered":"2024-03-24T01:24:09.738308+0000","last_unstale":"2024-03-24T01:24:17.031354+0000","last_undegraded":"2024-03-24T01:24:17.031354+0000","last_fullsized":"2024-03-24T01:24:17.031354+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:36:52.007109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021706+0000","last_change":"2024-03-24T01:24:15.666864+0000","last_active":"2024-03-24T01:24:17.021706+0000","last_peered":"2024-03-24T01:24:17.021706+0000","last_clean":"2024-03-24T01:24:17.021706+0000","last_became_active":"2024-03-24T01:24:15.659581+0000","last_became_peered":"2024-03-24T01:24:15.659581+0000","last_unstale":"2024-03-24T01:24:17.021706+0000","last_undegraded":"2024-03-24T01:24:17.021706+0000","last_fullsized":"2024-03-24T01:24:17.021706+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:40:09.145961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809258+0000","last_change":"2024-03-24T01:24:15.657385+0000","last_active":"2024-03-24T01:24:17.809258+0000","last_peered":"2024-03-24T01:24:17.809258+0000","last_clean":"2024-03-24T01:24:17.809258+0000","last_became_active":"2024-03-24T01:24:15.656275+0000","last_became_peered":"2024-03-24T01:24:15.656275+0000","last_unstale":"2024-03-24T01:24:17.809258+0000","last_undegraded":"2024-03-24T01:24:17.809258+0000","last_fullsized":"2024-03-24T01:24:17.809258+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:45:17.532730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.739356+0000","last_change":"2024-03-24T01:24:10.750827+0000","last_active":"2024-03-24T01:24:17.739356+0000","last_peered":"2024-03-24T01:24:17.739356+0000","last_clean":"2024-03-24T01:24:17.739356+0000","last_became_active":"2024-03-24T01:24:10.750522+0000","last_became_peered":"2024-03-24T01:24:10.750522+0000","last_unstale":"2024-03-24T01:24:17.739356+0000","last_undegraded":"2024-03-24T01:24:17.739356+0000","last_fullsized":"2024-03-24T01:24:17.739356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:04:48.028091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.808340+0000","last_change":"2024-03-24T01:24:09.730771+0000","last_active":"2024-03-24T01:24:17.808340+0000","last_peered":"2024-03-24T01:24:17.808340+0000","last_clean":"2024-03-24T01:24:17.808340+0000","last_became_active":"2024-03-24T01:24:09.730424+0000","last_became_peered":"2024-03-24T01:24:09.730424+0000","last_unstale":"2024-03-24T01:24:17.808340+0000","last_undegraded":"2024-03-24T01:24:17.808340+0000","last_fullsized":"2024-03-24T01:24:17.808340+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:50:57.480847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.031001+0000","last_change":"2024-03-24T01:24:15.029656+0000","last_active":"2024-03-24T01:24:17.031001+0000","last_peered":"2024-03-24T01:24:17.031001+0000","last_clean":"2024-03-24T01:24:17.031001+0000","last_became_active":"2024-03-24T01:24:15.029240+0000","last_became_peered":"2024-03-24T01:24:15.029240+0000","last_unstale":"2024-03-24T01:24:17.031001+0000","last_undegraded":"2024-03-24T01:24:17.031001+0000","last_fullsized":"2024-03-24T01:24:17.031001+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:28.723031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021468+0000","last_change":"2024-03-24T01:24:10.750421+0000","last_active":"2024-03-24T01:24:17.021468+0000","last_peered":"2024-03-24T01:24:17.021468+0000","last_clean":"2024-03-24T01:24:17.021468+0000","last_became_active":"2024-03-24T01:24:10.750160+0000","last_became_peered":"2024-03-24T01:24:10.750160+0000","last_unstale":"2024-03-24T01:24:17.021468+0000","last_undegraded":"2024-03-24T01:24:17.021468+0000","last_fullsized":"2024-03-24T01:24:17.021468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:54:57.105205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022107+0000","last_change":"2024-03-24T01:24:09.722993+0000","last_active":"2024-03-24T01:24:17.022107+0000","last_peered":"2024-03-24T01:24:17.022107+0000","last_clean":"2024-03-24T01:24:17.022107+0000","last_became_active":"2024-03-24T01:24:09.720133+0000","last_became_peered":"2024-03-24T01:24:09.720133+0000","last_unstale":"2024-03-24T01:24:17.022107+0000","last_undegraded":"2024-03-24T01:24:17.022107+0000","last_fullsized":"2024-03-24T01:24:17.022107+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:28:34.408651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.030554+0000","last_change":"2024-03-24T01:24:09.726157+0000","last_active":"2024-03-24T01:24:17.030554+0000","last_peered":"2024-03-24T01:24:17.030554+0000","last_clean":"2024-03-24T01:24:17.030554+0000","last_became_active":"2024-03-24T01:24:09.724571+0000","last_became_peered":"2024-03-24T01:24:09.724571+0000","last_unstale":"2024-03-24T01:24:17.030554+0000","last_undegraded":"2024-03-24T01:24:17.030554+0000","last_fullsized":"2024-03-24T01:24:17.030554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:33.381248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.810098+0000","last_change":"2024-03-24T01:24:10.726759+0000","last_active":"2024-03-24T01:24:17.810098+0000","last_peered":"2024-03-24T01:24:17.810098+0000","last_clean":"2024-03-24T01:24:17.810098+0000","last_became_active":"2024-03-24T01:24:10.726133+0000","last_became_peered":"2024-03-24T01:24:10.726133+0000","last_unstale":"2024-03-24T01:24:17.810098+0000","last_undegraded":"2024-03-24T01:24:17.810098+0000","last_fullsized":"2024-03-24T01:24:17.810098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:46:04.235199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.810763+0000","last_change":"2024-03-24T01:24:15.026408+0000","last_active":"2024-03-24T01:24:17.810763+0000","last_peered":"2024-03-24T01:24:17.810763+0000","last_clean":"2024-03-24T01:24:17.810763+0000","last_became_active":"2024-03-24T01:24:15.026038+0000","last_became_peered":"2024-03-24T01:24:15.026038+0000","last_unstale":"2024-03-24T01:24:17.810763+0000","last_undegraded":"2024-03-24T01:24:17.810763+0000","last_fullsized":"2024-03-24T01:24:17.810763+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:55:18.458421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022088+0000","last_change":"2024-03-24T01:24:09.722548+0000","last_active":"2024-03-24T01:24:17.022088+0000","last_peered":"2024-03-24T01:24:17.022088+0000","last_clean":"2024-03-24T01:24:17.022088+0000","last_became_active":"2024-03-24T01:24:09.721937+0000","last_became_peered":"2024-03-24T01:24:09.721937+0000","last_unstale":"2024-03-24T01:24:17.022088+0000","last_undegraded":"2024-03-24T01:24:17.022088+0000","last_fullsized":"2024-03-24T01:24:17.022088+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:50:57.480847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.027732+0000","last_change":"2024-03-24T01:24:10.743972+0000","last_active":"2024-03-24T01:24:17.027732+0000","last_peered":"2024-03-24T01:24:17.027732+0000","last_clean":"2024-03-24T01:24:17.027732+0000","last_became_active":"2024-03-24T01:24:10.743700+0000","last_became_peered":"2024-03-24T01:24:10.743700+0000","last_unstale":"2024-03-24T01:24:17.027732+0000","last_undegraded":"2024-03-24T01:24:17.027732+0000","last_fullsized":"2024-03-24T01:24:17.027732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:21:51.701615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022373+0000","last_change":"2024-03-24T01:24:15.023995+0000","last_active":"2024-03-24T01:24:17.022373+0000","last_peered":"2024-03-24T01:24:17.022373+0000","last_clean":"2024-03-24T01:24:17.022373+0000","last_became_active":"2024-03-24T01:24:15.023574+0000","last_became_peered":"2024-03-24T01:24:15.023574+0000","last_unstale":"2024-03-24T01:24:17.022373+0000","last_undegraded":"2024-03-24T01:24:17.022373+0000","last_fullsized":"2024-03-24T01:24:17.022373+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:26:01.912499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.029925+0000","last_change":"2024-03-24T01:24:10.740908+0000","last_active":"2024-03-24T01:24:17.029925+0000","last_peered":"2024-03-24T01:24:17.029925+0000","last_clean":"2024-03-24T01:24:17.029925+0000","last_became_active":"2024-03-24T01:24:10.740630+0000","last_became_peered":"2024-03-24T01:24:10.740630+0000","last_unstale":"2024-03-24T01:24:17.029925+0000","last_undegraded":"2024-03-24T01:24:17.029925+0000","last_fullsized":"2024-03-24T01:24:17.029925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:04:48.028091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809885+0000","last_change":"2024-03-24T01:24:10.743826+0000","last_active":"2024-03-24T01:24:17.809885+0000","last_peered":"2024-03-24T01:24:17.809885+0000","last_clean":"2024-03-24T01:24:17.809885+0000","last_became_active":"2024-03-24T01:24:10.743349+0000","last_became_peered":"2024-03-24T01:24:10.743349+0000","last_unstale":"2024-03-24T01:24:17.809885+0000","last_undegraded":"2024-03-24T01:24:17.809885+0000","last_fullsized":"2024-03-24T01:24:17.809885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:56:34.907995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.810496+0000","last_change":"2024-03-24T01:24:10.755545+0000","last_active":"2024-03-24T01:24:17.810496+0000","last_peered":"2024-03-24T01:24:17.810496+0000","last_clean":"2024-03-24T01:24:17.810496+0000","last_became_active":"2024-03-24T01:24:10.755147+0000","last_became_peered":"2024-03-24T01:24:10.755147+0000","last_unstale":"2024-03-24T01:24:17.810496+0000","last_undegraded":"2024-03-24T01:24:17.810496+0000","last_fullsized":"2024-03-24T01:24:17.810496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:35.441197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809878+0000","last_change":"2024-03-24T01:24:10.746109+0000","last_active":"2024-03-24T01:24:17.809878+0000","last_peered":"2024-03-24T01:24:17.809878+0000","last_clean":"2024-03-24T01:24:17.809878+0000","last_became_active":"2024-03-24T01:24:10.745809+0000","last_became_peered":"2024-03-24T01:24:10.745809+0000","last_unstale":"2024-03-24T01:24:17.809878+0000","last_undegraded":"2024-03-24T01:24:17.809878+0000","last_fullsized":"2024-03-24T01:24:17.809878+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:09:31.705353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022898+0000","last_change":"2024-03-24T01:24:10.752362+0000","last_active":"2024-03-24T01:24:17.022898+0000","last_peered":"2024-03-24T01:24:17.022898+0000","last_clean":"2024-03-24T01:24:17.022898+0000","last_became_active":"2024-03-24T01:24:10.752066+0000","last_became_peered":"2024-03-24T01:24:10.752066+0000","last_unstale":"2024-03-24T01:24:17.022898+0000","last_undegraded":"2024-03-24T01:24:17.022898+0000","last_fullsized":"2024-03-24T01:24:17.022898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:08:13.760239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.740402+0000","last_change":"2024-03-24T01:24:10.751536+0000","last_active":"2024-03-24T01:24:17.740402+0000","last_peered":"2024-03-24T01:24:17.740402+0000","last_clean":"2024-03-24T01:24:17.740402+0000","last_became_active":"2024-03-24T01:24:10.751260+0000","last_became_peered":"2024-03-24T01:24:10.751260+0000","last_unstale":"2024-03-24T01:24:17.740402+0000","last_undegraded":"2024-03-24T01:24:17.740402+0000","last_fullsized":"2024-03-24T01:24:17.740402+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:17.138453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.031836+0000","last_change":"2024-03-24T01:24:10.763037+0000","last_active":"2024-03-24T01:24:17.031836+0000","last_peered":"2024-03-24T01:24:17.031836+0000","last_clean":"2024-03-24T01:24:17.031836+0000","last_became_active":"2024-03-24T01:24:10.762718+0000","last_became_peered":"2024-03-24T01:24:10.762718+0000","last_unstale":"2024-03-24T01:24:17.031836+0000","last_undegraded":"2024-03-24T01:24:17.031836+0000","last_fullsized":"2024-03-24T01:24:17.031836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:46:04.235199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022992+0000","last_change":"2024-03-24T01:24:10.753332+0000","last_active":"2024-03-24T01:24:17.022992+0000","last_peered":"2024-03-24T01:24:17.022992+0000","last_clean":"2024-03-24T01:24:17.022992+0000","last_became_active":"2024-03-24T01:24:10.752905+0000","last_became_peered":"2024-03-24T01:24:10.752905+0000","last_unstale":"2024-03-24T01:24:17.022992+0000","last_undegraded":"2024-03-24T01:24:17.022992+0000","last_fullsized":"2024-03-24T01:24:17.022992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:57:11.968706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809861+0000","last_change":"2024-03-24T01:24:10.759635+0000","last_active":"2024-03-24T01:24:17.809861+0000","last_peered":"2024-03-24T01:24:17.809861+0000","last_clean":"2024-03-24T01:24:17.809861+0000","last_became_active":"2024-03-24T01:24:10.759310+0000","last_became_peered":"2024-03-24T01:24:10.759310+0000","last_unstale":"2024-03-24T01:24:17.809861+0000","last_undegraded":"2024-03-24T01:24:17.809861+0000","last_fullsized":"2024-03-24T01:24:17.809861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:08:13.760239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021975+0000","last_change":"2024-03-24T01:24:10.746635+0000","last_active":"2024-03-24T01:24:17.021975+0000","last_peered":"2024-03-24T01:24:17.021975+0000","last_clean":"2024-03-24T01:24:17.021975+0000","last_became_active":"2024-03-24T01:24:10.745597+0000","last_became_peered":"2024-03-24T01:24:10.745597+0000","last_unstale":"2024-03-24T01:24:17.021975+0000","last_undegraded":"2024-03-24T01:24:17.021975+0000","last_fullsized":"2024-03-24T01:24:17.021975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:42:22.671154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.810268+0000","last_change":"2024-03-24T01:24:10.743389+0000","last_active":"2024-03-24T01:24:17.810268+0000","last_peered":"2024-03-24T01:24:17.810268+0000","last_clean":"2024-03-24T01:24:17.810268+0000","last_became_active":"2024-03-24T01:24:10.743128+0000","last_became_peered":"2024-03-24T01:24:10.743128+0000","last_unstale":"2024-03-24T01:24:17.810268+0000","last_undegraded":"2024-03-24T01:24:17.810268+0000","last_fullsized":"2024-03-24T01:24:17.810268+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:42:22.671154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809576+0000","last_change":"2024-03-24T01:24:10.760940+0000","last_active":"2024-03-24T01:24:17.809576+0000","last_peered":"2024-03-24T01:24:17.809576+0000","last_clean":"2024-03-24T01:24:17.809576+0000","last_became_active":"2024-03-24T01:24:10.759991+0000","last_became_peered":"2024-03-24T01:24:10.759991+0000","last_unstale":"2024-03-24T01:24:17.809576+0000","last_undegraded":"2024-03-24T01:24:17.809576+0000","last_fullsized":"2024-03-24T01:24:17.809576+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:54:57.105205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.030845+0000","last_change":"2024-03-24T01:24:10.731391+0000","last_active":"2024-03-24T01:24:17.030845+0000","last_peered":"2024-03-24T01:24:17.030845+0000","last_clean":"2024-03-24T01:24:17.030845+0000","last_became_active":"2024-03-24T01:24:10.731094+0000","last_became_peered":"2024-03-24T01:24:10.731094+0000","last_unstale":"2024-03-24T01:24:17.030845+0000","last_undegraded":"2024-03-24T01:24:17.030845+0000","last_fullsized":"2024-03-24T01:24:17.030845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:17.138453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.812469+0000","last_change":"2024-03-24T01:24:10.766746+0000","last_active":"2024-03-24T01:24:17.812469+0000","last_peered":"2024-03-24T01:24:17.812469+0000","last_clean":"2024-03-24T01:24:17.812469+0000","last_became_active":"2024-03-24T01:24:10.766239+0000","last_became_peered":"2024-03-24T01:24:10.766239+0000","last_unstale":"2024-03-24T01:24:17.812469+0000","last_undegraded":"2024-03-24T01:24:17.812469+0000","last_fullsized":"2024-03-24T01:24:17.812469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:45:13.243851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811052+0000","last_change":"2024-03-24T01:24:10.741506+0000","last_active":"2024-03-24T01:24:17.811052+0000","last_peered":"2024-03-24T01:24:17.811052+0000","last_clean":"2024-03-24T01:24:17.811052+0000","last_became_active":"2024-03-24T01:24:10.741221+0000","last_became_peered":"2024-03-24T01:24:10.741221+0000","last_unstale":"2024-03-24T01:24:17.811052+0000","last_undegraded":"2024-03-24T01:24:17.811052+0000","last_fullsized":"2024-03-24T01:24:17.811052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:27:51.341849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.812926+0000","last_change":"2024-03-24T01:24:10.767451+0000","last_active":"2024-03-24T01:24:17.812926+0000","last_peered":"2024-03-24T01:24:17.812926+0000","last_clean":"2024-03-24T01:24:17.812926+0000","last_became_active":"2024-03-24T01:24:10.767159+0000","last_became_peered":"2024-03-24T01:24:10.767159+0000","last_unstale":"2024-03-24T01:24:17.812926+0000","last_undegraded":"2024-03-24T01:24:17.812926+0000","last_fullsized":"2024-03-24T01:24:17.812926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:24.434151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809619+0000","last_change":"2024-03-24T01:24:10.751373+0000","last_active":"2024-03-24T01:24:17.809619+0000","last_peered":"2024-03-24T01:24:17.809619+0000","last_clean":"2024-03-24T01:24:17.809619+0000","last_became_active":"2024-03-24T01:24:10.751084+0000","last_became_peered":"2024-03-24T01:24:10.751084+0000","last_unstale":"2024-03-24T01:24:17.809619+0000","last_undegraded":"2024-03-24T01:24:17.809619+0000","last_fullsized":"2024-03-24T01:24:17.809619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:17.138453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.739324+0000","last_change":"2024-03-24T01:24:10.752504+0000","last_active":"2024-03-24T01:24:17.739324+0000","last_peered":"2024-03-24T01:24:17.739324+0000","last_clean":"2024-03-24T01:24:17.739324+0000","last_became_active":"2024-03-24T01:24:10.752313+0000","last_became_peered":"2024-03-24T01:24:10.752313+0000","last_unstale":"2024-03-24T01:24:17.739324+0000","last_undegraded":"2024-03-24T01:24:17.739324+0000","last_fullsized":"2024-03-24T01:24:17.739324+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:54:57.105205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809195+0000","last_change":"2024-03-24T01:24:10.750967+0000","last_active":"2024-03-24T01:24:17.809195+0000","last_peered":"2024-03-24T01:24:17.809195+0000","last_clean":"2024-03-24T01:24:17.809195+0000","last_became_active":"2024-03-24T01:24:10.748587+0000","last_became_peered":"2024-03-24T01:24:10.748587+0000","last_unstale":"2024-03-24T01:24:17.809195+0000","last_undegraded":"2024-03-24T01:24:17.809195+0000","last_fullsized":"2024-03-24T01:24:17.809195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:44:57.802345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021966+0000","last_change":"2024-03-24T01:24:10.750792+0000","last_active":"2024-03-24T01:24:17.021966+0000","last_peered":"2024-03-24T01:24:17.021966+0000","last_clean":"2024-03-24T01:24:17.021966+0000","last_became_active":"2024-03-24T01:24:10.749384+0000","last_became_peered":"2024-03-24T01:24:10.749384+0000","last_unstale":"2024-03-24T01:24:17.021966+0000","last_undegraded":"2024-03-24T01:24:17.021966+0000","last_fullsized":"2024-03-24T01:24:17.021966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:17.138453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028133+0000","last_change":"2024-03-24T01:24:10.728222+0000","last_active":"2024-03-24T01:24:17.028133+0000","last_peered":"2024-03-24T01:24:17.028133+0000","last_clean":"2024-03-24T01:24:17.028133+0000","last_became_active":"2024-03-24T01:24:10.727874+0000","last_became_peered":"2024-03-24T01:24:10.727874+0000","last_unstale":"2024-03-24T01:24:17.028133+0000","last_undegraded":"2024-03-24T01:24:17.028133+0000","last_fullsized":"2024-03-24T01:24:17.028133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:58:18.259182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.031302+0000","last_change":"2024-03-24T01:24:10.741809+0000","last_active":"2024-03-24T01:24:17.031302+0000","last_peered":"2024-03-24T01:24:17.031302+0000","last_clean":"2024-03-24T01:24:17.031302+0000","last_became_active":"2024-03-24T01:24:10.741444+0000","last_became_peered":"2024-03-24T01:24:10.741444+0000","last_unstale":"2024-03-24T01:24:17.031302+0000","last_undegraded":"2024-03-24T01:24:17.031302+0000","last_fullsized":"2024-03-24T01:24:17.031302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:19:51.907364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.031726+0000","last_change":"2024-03-24T01:24:10.751752+0000","last_active":"2024-03-24T01:24:17.031726+0000","last_peered":"2024-03-24T01:24:17.031726+0000","last_clean":"2024-03-24T01:24:17.031726+0000","last_became_active":"2024-03-24T01:24:10.747073+0000","last_became_peered":"2024-03-24T01:24:10.747073+0000","last_unstale":"2024-03-24T01:24:17.031726+0000","last_undegraded":"2024-03-24T01:24:17.031726+0000","last_fullsized":"2024-03-24T01:24:17.031726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:08:13.760239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.813263+0000","last_change":"2024-03-24T01:24:10.765006+0000","last_active":"2024-03-24T01:24:17.813263+0000","last_peered":"2024-03-24T01:24:17.813263+0000","last_clean":"2024-03-24T01:24:17.813263+0000","last_became_active":"2024-03-24T01:24:10.764417+0000","last_became_peered":"2024-03-24T01:24:10.764417+0000","last_unstale":"2024-03-24T01:24:17.813263+0000","last_undegraded":"2024-03-24T01:24:17.813263+0000","last_fullsized":"2024-03-24T01:24:17.813263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:29:34.357876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021162+0000","last_change":"2024-03-24T01:24:10.757149+0000","last_active":"2024-03-24T01:24:17.021162+0000","last_peered":"2024-03-24T01:24:17.021162+0000","last_clean":"2024-03-24T01:24:17.021162+0000","last_became_active":"2024-03-24T01:24:10.756879+0000","last_became_peered":"2024-03-24T01:24:10.756879+0000","last_unstale":"2024-03-24T01:24:17.021162+0000","last_undegraded":"2024-03-24T01:24:17.021162+0000","last_fullsized":"2024-03-24T01:24:17.021162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:55:14.169541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809223+0000","last_change":"2024-03-24T01:24:10.766128+0000","last_active":"2024-03-24T01:24:17.809223+0000","last_peered":"2024-03-24T01:24:17.809223+0000","last_clean":"2024-03-24T01:24:17.809223+0000","last_became_active":"2024-03-24T01:24:10.763007+0000","last_became_peered":"2024-03-24T01:24:10.763007+0000","last_unstale":"2024-03-24T01:24:17.809223+0000","last_undegraded":"2024-03-24T01:24:17.809223+0000","last_fullsized":"2024-03-24T01:24:17.809223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:35.441197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809997+0000","last_change":"2024-03-24T01:24:10.741795+0000","last_active":"2024-03-24T01:24:17.809997+0000","last_peered":"2024-03-24T01:24:17.809997+0000","last_clean":"2024-03-24T01:24:17.809997+0000","last_became_active":"2024-03-24T01:24:10.741516+0000","last_became_peered":"2024-03-24T01:24:10.741516+0000","last_unstale":"2024-03-24T01:24:17.809997+0000","last_undegraded":"2024-03-24T01:24:17.809997+0000","last_fullsized":"2024-03-24T01:24:17.809997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:57:11.968706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811350+0000","last_change":"2024-03-24T01:24:10.761849+0000","last_active":"2024-03-24T01:24:17.811350+0000","last_peered":"2024-03-24T01:24:17.811350+0000","last_clean":"2024-03-24T01:24:17.811350+0000","last_became_active":"2024-03-24T01:24:10.761288+0000","last_became_peered":"2024-03-24T01:24:10.761288+0000","last_unstale":"2024-03-24T01:24:17.811350+0000","last_undegraded":"2024-03-24T01:24:17.811350+0000","last_fullsized":"2024-03-24T01:24:17.811350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:02:18.904076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.810985+0000","last_change":"2024-03-24T01:24:10.755426+0000","last_active":"2024-03-24T01:24:17.810985+0000","last_peered":"2024-03-24T01:24:17.810985+0000","last_clean":"2024-03-24T01:24:17.810985+0000","last_became_active":"2024-03-24T01:24:10.755073+0000","last_became_peered":"2024-03-24T01:24:10.755073+0000","last_unstale":"2024-03-24T01:24:17.810985+0000","last_undegraded":"2024-03-24T01:24:17.810985+0000","last_fullsized":"2024-03-24T01:24:17.810985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:46:04.235199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.810708+0000","last_change":"2024-03-24T01:24:10.732890+0000","last_active":"2024-03-24T01:24:17.810708+0000","last_peered":"2024-03-24T01:24:17.810708+0000","last_clean":"2024-03-24T01:24:17.810708+0000","last_became_active":"2024-03-24T01:24:10.731992+0000","last_became_peered":"2024-03-24T01:24:10.731992+0000","last_unstale":"2024-03-24T01:24:17.810708+0000","last_undegraded":"2024-03-24T01:24:17.810708+0000","last_fullsized":"2024-03-24T01:24:17.810708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:19:51.907364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.741104+0000","last_change":"2024-03-24T01:24:10.750945+0000","last_active":"2024-03-24T01:24:17.741104+0000","last_peered":"2024-03-24T01:24:17.741104+0000","last_clean":"2024-03-24T01:24:17.741104+0000","last_became_active":"2024-03-24T01:24:10.750099+0000","last_became_peered":"2024-03-24T01:24:10.750099+0000","last_unstale":"2024-03-24T01:24:17.741104+0000","last_undegraded":"2024-03-24T01:24:17.741104+0000","last_fullsized":"2024-03-24T01:24:17.741104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:08:13.760239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.813056+0000","last_change":"2024-03-24T01:24:10.762165+0000","last_active":"2024-03-24T01:24:17.813056+0000","last_peered":"2024-03-24T01:24:17.813056+0000","last_clean":"2024-03-24T01:24:17.813056+0000","last_became_active":"2024-03-24T01:24:10.758157+0000","last_became_peered":"2024-03-24T01:24:10.758157+0000","last_unstale":"2024-03-24T01:24:17.813056+0000","last_undegraded":"2024-03-24T01:24:17.813056+0000","last_fullsized":"2024-03-24T01:24:17.813056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:37:21.153754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574859,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27864,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343976,"statfs":{"total":96636764160,"available":96608231424,"internally_reserved":0,"allocated":864256,"data_stored":698343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574860,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":107483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":107519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574859,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":107393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":107553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574860,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":872448,"data_stored":698399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T01:24:44.101 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T01:24:44.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.344+0000 7f59b8e00640 1 Processor -- start 2024-03-24T01:24:44.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.345+0000 7f59b8e00640 1 -- start start 2024-03-24T01:24:44.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.346+0000 7f59b8e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f59b4144420 0x7f59b406d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:44.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.346+0000 7f59b8e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f59b40cc8f0 0x7f59b40cfa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:44.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.346+0000 7f59b8e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59b40cceb0 con 0x7f59b4144800 2024-03-24T01:24:44.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.346+0000 7f59b8e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59b40cd190 con 0x7f59b40cc8f0 2024-03-24T01:24:44.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.346+0000 7f59b8e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59b40ceb70 con 0x7f59b4144420 2024-03-24T01:24:44.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.346+0000 7f59b3e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f59b40cc8f0 0x7f59b40cfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:44.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.346+0000 7f59b3e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f59b40cc8f0 0x7f59b40cfa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60346/0 (socket says 172.21.15.90:60346) 2024-03-24T01:24:44.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.346+0000 7f59b3e00640 1 -- 172.21.15.90:0/2659733107 learned_addr learned my addr 172.21.15.90:0/2659733107 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:44.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.347+0000 7f59b3400640 1 --2- 172.21.15.90:0/2659733107 >> v2:172.21.15.120:3300/0 conn(0x7f59b4144420 0x7f59b406d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:44.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.347+0000 7f59b3e00640 1 -- 172.21.15.90:0/2659733107 >> v2:172.21.15.120:3300/0 conn(0x7f59b4144420 msgr2=0x7f59b406d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:44.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.347+0000 7f59b3e00640 1 --2- 172.21.15.90:0/2659733107 >> v2:172.21.15.120:3300/0 conn(0x7f59b4144420 0x7f59b406d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:44.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.347+0000 7f59b3e00640 1 -- 172.21.15.90:0/2659733107 >> v1:172.21.15.90:6789/0 conn(0x7f59b4144800 legacy=0x7f59b40cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:44.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.347+0000 7f59b3e00640 1 -- 172.21.15.90:0/2659733107 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b40cee50 con 0x7f59b40cc8f0 2024-03-24T01:24:44.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.348+0000 7f59b3e00640 1 --2- 172.21.15.90:0/2659733107 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b40cc8f0 0x7f59b40cfa80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f59b406f760 tx=0x7f59ac02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=144b73f57a267bef server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:44.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.348+0000 7f59b2a00640 1 -- 172.21.15.90:0/2659733107 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59ac02ca80 con 0x7f59b40cc8f0 2024-03-24T01:24:44.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.348+0000 7f59b2a00640 1 -- 172.21.15.90:0/2659733107 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59ac02cc20 con 0x7f59b40cc8f0 2024-03-24T01:24:44.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.348+0000 7f59b2a00640 1 -- 172.21.15.90:0/2659733107 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59ac0343f0 con 0x7f59b40cc8f0 2024-03-24T01:24:44.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.349+0000 7f59b8e00640 1 -- 172.21.15.90:0/2659733107 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b40cc8f0 msgr2=0x7f59b40cfa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:44.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.349+0000 7f59b8e00640 1 --2- 172.21.15.90:0/2659733107 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b40cc8f0 0x7f59b40cfa80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f59b406f760 tx=0x7f59ac02bee0 comp rx=0 tx=0).stop 2024-03-24T01:24:44.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.349+0000 7f59b8e00640 1 -- 172.21.15.90:0/2659733107 shutdown_connections 2024-03-24T01:24:44.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.349+0000 7f59b8e00640 1 --2- 172.21.15.90:0/2659733107 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b40cc8f0 0x7f59b40cfa80 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:44.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.349+0000 7f59b8e00640 1 --2- 172.21.15.90:0/2659733107 >> v2:172.21.15.120:3300/0 conn(0x7f59b4144420 0x7f59b406d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:44.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.349+0000 7f59b8e00640 1 -- 172.21.15.90:0/2659733107 >> 172.21.15.90:0/2659733107 conn(0x7f59b406b0c0 msgr2=0x7f59b406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:44.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.349+0000 7f59b8e00640 1 -- 172.21.15.90:0/2659733107 shutdown_connections 2024-03-24T01:24:44.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.349+0000 7f59b8e00640 1 -- 172.21.15.90:0/2659733107 wait complete. 2024-03-24T01:24:44.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.351+0000 7f59b8e00640 1 Processor -- start 2024-03-24T01:24:44.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.351+0000 7f59b8e00640 1 -- start start 2024-03-24T01:24:44.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.352+0000 7f59b8e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59b40cc8f0 0x7f59b40b29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:44.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.352+0000 7f59b8e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59b4144420 0x7f59b415d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:44.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.352+0000 7f59b8e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b4144800 0x7f59b415dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:44.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.352+0000 7f59b8e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b40ce910 con 0x7f59b4144800 2024-03-24T01:24:44.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.352+0000 7f59b8e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b40ce790 con 0x7f59b4144420 2024-03-24T01:24:44.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.352+0000 7f59b8e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59b415f540 con 0x7f59b40cc8f0 2024-03-24T01:24:44.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.352+0000 7f59b3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b4144800 0x7f59b415dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:44.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.352+0000 7f59b3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b4144800 0x7f59b415dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60362/0 (socket says 172.21.15.90:60362) 2024-03-24T01:24:44.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.352+0000 7f59b3e00640 1 -- 172.21.15.90:0/478221442 learned_addr learned my addr 172.21.15.90:0/478221442 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:44.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.352+0000 7f59b3400640 1 --2- 172.21.15.90:0/478221442 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59b40cc8f0 0x7f59b40b29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:44.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.352+0000 7f59aaa00640 1 --2- 172.21.15.90:0/478221442 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59b4144420 0x7f59b415d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:44.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.352+0000 7f59b3e00640 1 -- 172.21.15.90:0/478221442 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59b40cc8f0 msgr2=0x7f59b40b29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:44.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.352+0000 7f59b3e00640 1 --2- 172.21.15.90:0/478221442 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59b40cc8f0 0x7f59b40b29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:44.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.352+0000 7f59b3e00640 1 -- 172.21.15.90:0/478221442 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59b4144420 msgr2=0x7f59b415d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:44.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.352+0000 7f59b3e00640 1 --2- 172.21.15.90:0/478221442 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59b4144420 0x7f59b415d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:44.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.352+0000 7f59b3e00640 1 -- 172.21.15.90:0/478221442 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b40ac980 con 0x7f59b4144800 2024-03-24T01:24:44.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.353+0000 7f59b3e00640 1 --2- 172.21.15.90:0/478221442 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b4144800 0x7f59b415dec0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f59ac02c4f0 tx=0x7f59ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:44.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.353+0000 7f59abe00640 1 -- 172.21.15.90:0/478221442 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59ac043020 con 0x7f59b4144800 2024-03-24T01:24:44.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.353+0000 7f59abe00640 1 -- 172.21.15.90:0/478221442 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59ac03e450 con 0x7f59b4144800 2024-03-24T01:24:44.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.353+0000 7f59abe00640 1 -- 172.21.15.90:0/478221442 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59ac009510 con 0x7f59b4144800 2024-03-24T01:24:44.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.353+0000 7f59b8e00640 1 -- 172.21.15.90:0/478221442 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59b40b2ef0 con 0x7f59b4144800 2024-03-24T01:24:44.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.353+0000 7f59b8e00640 1 -- 172.21.15.90:0/478221442 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59b40b4120 con 0x7f59b4144800 2024-03-24T01:24:44.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.354+0000 7f59b8e00640 1 -- 172.21.15.90:0/478221442 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59b404f400 con 0x7f59b4144800 2024-03-24T01:24:44.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.355+0000 7f59abe00640 1 -- 172.21.15.90:0/478221442 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f59ac046020 con 0x7f59b4144800 2024-03-24T01:24:44.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.356+0000 7f59abe00640 1 --2- 172.21.15.90:0/478221442 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f599006bc00 0x7f599006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:44.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.356+0000 7f59abe00640 1 -- 172.21.15.90:0/478221442 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f59ac0ad5c0 con 0x7f59b4144800 2024-03-24T01:24:44.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.356+0000 7f59b3400640 1 --2- 172.21.15.90:0/478221442 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f599006bc00 0x7f599006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:44.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.357+0000 7f59b3400640 1 --2- 172.21.15.90:0/478221442 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f599006bc00 0x7f599006e0c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f59a4006fd0 tx=0x7f59a4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:44.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.361+0000 7f59abe00640 1 -- 172.21.15.90:0/478221442 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f59ac04d030 con 0x7f59b4144800 2024-03-24T01:24:44.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.538+0000 7f59b8e00640 1 -- 172.21.15.90:0/478221442 --> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f59b40b30c0 con 0x7f599006bc00 2024-03-24T01:24:44.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.563+0000 7f59abe00640 1 -- 172.21.15.90:0/478221442 <== mgr.4104 v2:172.21.15.120:6832/860392634 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332535 (secure 0 0 0) 0x7f59b40b30c0 con 0x7f599006bc00 2024-03-24T01:24:44.568 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:24:44.570 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-24T01:24:44.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.568+0000 7f59b8e00640 1 -- 172.21.15.90:0/478221442 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f599006bc00 msgr2=0x7f599006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:44.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.568+0000 7f59b8e00640 1 --2- 172.21.15.90:0/478221442 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f599006bc00 0x7f599006e0c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f59a4006fd0 tx=0x7f59a4008040 comp rx=0 tx=0).stop 2024-03-24T01:24:44.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.568+0000 7f59b8e00640 1 -- 172.21.15.90:0/478221442 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b4144800 msgr2=0x7f59b415dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:44.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.568+0000 7f59b8e00640 1 --2- 172.21.15.90:0/478221442 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b4144800 0x7f59b415dec0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f59ac02c4f0 tx=0x7f59ac004a40 comp rx=0 tx=0).stop 2024-03-24T01:24:44.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.569+0000 7f59b8e00640 1 -- 172.21.15.90:0/478221442 shutdown_connections 2024-03-24T01:24:44.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.569+0000 7f59b8e00640 1 --2- 172.21.15.90:0/478221442 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f599006bc00 0x7f599006e0c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:44.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.569+0000 7f59b8e00640 1 --2- 172.21.15.90:0/478221442 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b4144800 0x7f59b415dec0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:44.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.569+0000 7f59b8e00640 1 --2- 172.21.15.90:0/478221442 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59b4144420 0x7f59b415d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:44.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.569+0000 7f59b8e00640 1 --2- 172.21.15.90:0/478221442 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59b40cc8f0 0x7f59b40b29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:44.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.569+0000 7f59b8e00640 1 -- 172.21.15.90:0/478221442 >> 172.21.15.90:0/478221442 conn(0x7f59b406b0c0 msgr2=0x7f59b4142630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:44.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.569+0000 7f59b8e00640 1 -- 172.21.15.90:0/478221442 shutdown_connections 2024-03-24T01:24:44.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.569+0000 7f59b8e00640 1 -- 172.21.15.90:0/478221442 wait complete. 2024-03-24T01:24:44.587 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-03-24T01:24:43.490304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219472,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755248,"statfs":{"total":773094113280,"available":772869373952,"internally_reserved":0,"allocated":3383296,"data_stored":2049160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12747,"internal_metadata":221236789},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005471"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":28,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811464+0000","last_change":"2024-03-24T01:24:09.722150+0000","last_active":"2024-03-24T01:24:17.811464+0000","last_peered":"2024-03-24T01:24:17.811464+0000","last_clean":"2024-03-24T01:24:17.811464+0000","last_became_active":"2024-03-24T01:24:09.721808+0000","last_became_peered":"2024-03-24T01:24:09.721808+0000","last_unstale":"2024-03-24T01:24:17.811464+0000","last_undegraded":"2024-03-24T01:24:17.811464+0000","last_fullsized":"2024-03-24T01:24:17.811464+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:31:46.512174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022533+0000","last_change":"2024-03-24T01:24:10.750478+0000","last_active":"2024-03-24T01:24:17.022533+0000","last_peered":"2024-03-24T01:24:17.022533+0000","last_clean":"2024-03-24T01:24:17.022533+0000","last_became_active":"2024-03-24T01:24:10.749299+0000","last_became_peered":"2024-03-24T01:24:10.749299+0000","last_unstale":"2024-03-24T01:24:17.022533+0000","last_undegraded":"2024-03-24T01:24:17.022533+0000","last_fullsized":"2024-03-24T01:24:17.022533+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:46:04.235199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021708+0000","last_change":"2024-03-24T01:24:15.662475+0000","last_active":"2024-03-24T01:24:17.021708+0000","last_peered":"2024-03-24T01:24:17.021708+0000","last_clean":"2024-03-24T01:24:17.021708+0000","last_became_active":"2024-03-24T01:24:15.661550+0000","last_became_peered":"2024-03-24T01:24:15.661550+0000","last_unstale":"2024-03-24T01:24:17.021708+0000","last_undegraded":"2024-03-24T01:24:17.021708+0000","last_fullsized":"2024-03-24T01:24:17.021708+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:14:59.259808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022711+0000","last_change":"2024-03-24T01:24:09.723434+0000","last_active":"2024-03-24T01:24:17.022711+0000","last_peered":"2024-03-24T01:24:17.022711+0000","last_clean":"2024-03-24T01:24:17.022711+0000","last_became_active":"2024-03-24T01:24:09.722439+0000","last_became_peered":"2024-03-24T01:24:09.722439+0000","last_unstale":"2024-03-24T01:24:17.022711+0000","last_undegraded":"2024-03-24T01:24:17.022711+0000","last_fullsized":"2024-03-24T01:24:17.022711+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:31:46.512174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811170+0000","last_change":"2024-03-24T01:24:10.745888+0000","last_active":"2024-03-24T01:24:17.811170+0000","last_peered":"2024-03-24T01:24:17.811170+0000","last_clean":"2024-03-24T01:24:17.811170+0000","last_became_active":"2024-03-24T01:24:10.742579+0000","last_became_peered":"2024-03-24T01:24:10.742579+0000","last_unstale":"2024-03-24T01:24:17.811170+0000","last_undegraded":"2024-03-24T01:24:17.811170+0000","last_fullsized":"2024-03-24T01:24:17.811170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:04:48.028091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.026871+0000","last_change":"2024-03-24T01:24:15.022407+0000","last_active":"2024-03-24T01:24:17.026871+0000","last_peered":"2024-03-24T01:24:17.026871+0000","last_clean":"2024-03-24T01:24:17.026871+0000","last_became_active":"2024-03-24T01:24:15.021962+0000","last_became_peered":"2024-03-24T01:24:15.021962+0000","last_unstale":"2024-03-24T01:24:17.026871+0000","last_undegraded":"2024-03-24T01:24:17.026871+0000","last_fullsized":"2024-03-24T01:24:17.026871+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:28.723031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.807761+0000","last_change":"2024-03-24T01:24:09.735868+0000","last_active":"2024-03-24T01:24:17.807761+0000","last_peered":"2024-03-24T01:24:17.807761+0000","last_clean":"2024-03-24T01:24:17.807761+0000","last_became_active":"2024-03-24T01:24:09.735573+0000","last_became_peered":"2024-03-24T01:24:09.735573+0000","last_unstale":"2024-03-24T01:24:17.807761+0000","last_undegraded":"2024-03-24T01:24:17.807761+0000","last_fullsized":"2024-03-24T01:24:17.807761+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:35:54.000044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811189+0000","last_change":"2024-03-24T01:24:10.752632+0000","last_active":"2024-03-24T01:24:17.811189+0000","last_peered":"2024-03-24T01:24:17.811189+0000","last_clean":"2024-03-24T01:24:17.811189+0000","last_became_active":"2024-03-24T01:24:10.752358+0000","last_became_peered":"2024-03-24T01:24:10.752358+0000","last_unstale":"2024-03-24T01:24:17.811189+0000","last_undegraded":"2024-03-24T01:24:17.811189+0000","last_fullsized":"2024-03-24T01:24:17.811189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:38:07.266851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.027523+0000","last_change":"2024-03-24T01:24:15.032062+0000","last_active":"2024-03-24T01:24:17.027523+0000","last_peered":"2024-03-24T01:24:17.027523+0000","last_clean":"2024-03-24T01:24:17.027523+0000","last_became_active":"2024-03-24T01:24:15.031556+0000","last_became_peered":"2024-03-24T01:24:15.031556+0000","last_unstale":"2024-03-24T01:24:17.027523+0000","last_undegraded":"2024-03-24T01:24:17.027523+0000","last_fullsized":"2024-03-24T01:24:17.027523+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:45:17.532730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.812001+0000","last_change":"2024-03-24T01:24:09.739225+0000","last_active":"2024-03-24T01:24:17.812001+0000","last_peered":"2024-03-24T01:24:17.812001+0000","last_clean":"2024-03-24T01:24:17.812001+0000","last_became_active":"2024-03-24T01:24:09.738488+0000","last_became_peered":"2024-03-24T01:24:09.738488+0000","last_unstale":"2024-03-24T01:24:17.812001+0000","last_undegraded":"2024-03-24T01:24:17.812001+0000","last_fullsized":"2024-03-24T01:24:17.812001+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:56:33.895912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811310+0000","last_change":"2024-03-24T01:24:10.753043+0000","last_active":"2024-03-24T01:24:17.811310+0000","last_peered":"2024-03-24T01:24:17.811310+0000","last_clean":"2024-03-24T01:24:17.811310+0000","last_became_active":"2024-03-24T01:24:10.751666+0000","last_became_peered":"2024-03-24T01:24:10.751666+0000","last_unstale":"2024-03-24T01:24:17.811310+0000","last_undegraded":"2024-03-24T01:24:17.811310+0000","last_fullsized":"2024-03-24T01:24:17.811310+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:02:18.904076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.027460+0000","last_change":"2024-03-24T01:24:15.021052+0000","last_active":"2024-03-24T01:24:17.027460+0000","last_peered":"2024-03-24T01:24:17.027460+0000","last_clean":"2024-03-24T01:24:17.027460+0000","last_became_active":"2024-03-24T01:24:15.020449+0000","last_became_peered":"2024-03-24T01:24:15.020449+0000","last_unstale":"2024-03-24T01:24:17.027460+0000","last_undegraded":"2024-03-24T01:24:17.027460+0000","last_fullsized":"2024-03-24T01:24:17.027460+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:26:07.104726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028245+0000","last_change":"2024-03-24T01:24:10.737914+0000","last_active":"2024-03-24T01:24:17.028245+0000","last_peered":"2024-03-24T01:24:17.028245+0000","last_clean":"2024-03-24T01:24:17.028245+0000","last_became_active":"2024-03-24T01:24:10.737652+0000","last_became_peered":"2024-03-24T01:24:10.737652+0000","last_unstale":"2024-03-24T01:24:17.028245+0000","last_undegraded":"2024-03-24T01:24:17.028245+0000","last_fullsized":"2024-03-24T01:24:17.028245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:09:31.705353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.807741+0000","last_change":"2024-03-24T01:24:09.726730+0000","last_active":"2024-03-24T01:24:17.807741+0000","last_peered":"2024-03-24T01:24:17.807741+0000","last_clean":"2024-03-24T01:24:17.807741+0000","last_became_active":"2024-03-24T01:24:09.725266+0000","last_became_peered":"2024-03-24T01:24:09.725266+0000","last_unstale":"2024-03-24T01:24:17.807741+0000","last_undegraded":"2024-03-24T01:24:17.807741+0000","last_fullsized":"2024-03-24T01:24:17.807741+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:27:50.329766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028585+0000","last_change":"2024-03-24T01:24:15.027082+0000","last_active":"2024-03-24T01:24:17.028585+0000","last_peered":"2024-03-24T01:24:17.028585+0000","last_clean":"2024-03-24T01:24:17.028585+0000","last_became_active":"2024-03-24T01:24:15.026296+0000","last_became_peered":"2024-03-24T01:24:15.026296+0000","last_unstale":"2024-03-24T01:24:17.028585+0000","last_undegraded":"2024-03-24T01:24:17.028585+0000","last_fullsized":"2024-03-24T01:24:17.028585+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:55.111215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022103+0000","last_change":"2024-03-24T01:24:10.740733+0000","last_active":"2024-03-24T01:24:17.022103+0000","last_peered":"2024-03-24T01:24:17.022103+0000","last_clean":"2024-03-24T01:24:17.022103+0000","last_became_active":"2024-03-24T01:24:10.740515+0000","last_became_peered":"2024-03-24T01:24:10.740515+0000","last_unstale":"2024-03-24T01:24:17.022103+0000","last_undegraded":"2024-03-24T01:24:17.022103+0000","last_fullsized":"2024-03-24T01:24:17.022103+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:04:48.028091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.738935+0000","last_change":"2024-03-24T01:24:09.725315+0000","last_active":"2024-03-24T01:24:17.738935+0000","last_peered":"2024-03-24T01:24:17.738935+0000","last_clean":"2024-03-24T01:24:17.738935+0000","last_became_active":"2024-03-24T01:24:09.723043+0000","last_became_peered":"2024-03-24T01:24:09.723043+0000","last_unstale":"2024-03-24T01:24:17.738935+0000","last_undegraded":"2024-03-24T01:24:17.738935+0000","last_fullsized":"2024-03-24T01:24:17.738935+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:36:52.007109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.027797+0000","last_change":"2024-03-24T01:24:15.023034+0000","last_active":"2024-03-24T01:24:17.027797+0000","last_peered":"2024-03-24T01:24:17.027797+0000","last_clean":"2024-03-24T01:24:17.027797+0000","last_became_active":"2024-03-24T01:24:15.021144+0000","last_became_peered":"2024-03-24T01:24:15.021144+0000","last_unstale":"2024-03-24T01:24:17.027797+0000","last_undegraded":"2024-03-24T01:24:17.027797+0000","last_fullsized":"2024-03-24T01:24:17.027797+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:00.868691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021560+0000","last_change":"2024-03-24T01:24:10.746085+0000","last_active":"2024-03-24T01:24:17.021560+0000","last_peered":"2024-03-24T01:24:17.021560+0000","last_clean":"2024-03-24T01:24:17.021560+0000","last_became_active":"2024-03-24T01:24:10.745493+0000","last_became_peered":"2024-03-24T01:24:10.745493+0000","last_unstale":"2024-03-24T01:24:17.021560+0000","last_undegraded":"2024-03-24T01:24:17.021560+0000","last_fullsized":"2024-03-24T01:24:17.021560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:19:51.907364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.739444+0000","last_change":"2024-03-24T01:24:09.723035+0000","last_active":"2024-03-24T01:24:17.739444+0000","last_peered":"2024-03-24T01:24:17.739444+0000","last_clean":"2024-03-24T01:24:17.739444+0000","last_became_active":"2024-03-24T01:24:09.722548+0000","last_became_peered":"2024-03-24T01:24:09.722548+0000","last_unstale":"2024-03-24T01:24:17.739444+0000","last_undegraded":"2024-03-24T01:24:17.739444+0000","last_fullsized":"2024-03-24T01:24:17.739444+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:58:17.247099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.741186+0000","last_change":"2024-03-24T01:24:15.658323+0000","last_active":"2024-03-24T01:24:17.741186+0000","last_peered":"2024-03-24T01:24:17.741186+0000","last_clean":"2024-03-24T01:24:17.741186+0000","last_became_active":"2024-03-24T01:24:15.656635+0000","last_became_peered":"2024-03-24T01:24:15.656635+0000","last_unstale":"2024-03-24T01:24:17.741186+0000","last_undegraded":"2024-03-24T01:24:17.741186+0000","last_fullsized":"2024-03-24T01:24:17.741186+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:52:30.401135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028531+0000","last_change":"2024-03-24T01:24:10.734344+0000","last_active":"2024-03-24T01:24:17.028531+0000","last_peered":"2024-03-24T01:24:17.028531+0000","last_clean":"2024-03-24T01:24:17.028531+0000","last_became_active":"2024-03-24T01:24:10.734010+0000","last_became_peered":"2024-03-24T01:24:10.734010+0000","last_unstale":"2024-03-24T01:24:17.028531+0000","last_undegraded":"2024-03-24T01:24:17.028531+0000","last_fullsized":"2024-03-24T01:24:17.028531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:36:53.019192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.808157+0000","last_change":"2024-03-24T01:24:09.729151+0000","last_active":"2024-03-24T01:24:17.808157+0000","last_peered":"2024-03-24T01:24:17.808157+0000","last_clean":"2024-03-24T01:24:17.808157+0000","last_became_active":"2024-03-24T01:24:09.728441+0000","last_became_peered":"2024-03-24T01:24:09.728441+0000","last_unstale":"2024-03-24T01:24:17.808157+0000","last_undegraded":"2024-03-24T01:24:17.808157+0000","last_fullsized":"2024-03-24T01:24:17.808157+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:33.381248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.027681+0000","last_change":"2024-03-24T01:24:15.024436+0000","last_active":"2024-03-24T01:24:17.027681+0000","last_peered":"2024-03-24T01:24:17.027681+0000","last_clean":"2024-03-24T01:24:17.027681+0000","last_became_active":"2024-03-24T01:24:15.022270+0000","last_became_peered":"2024-03-24T01:24:15.022270+0000","last_unstale":"2024-03-24T01:24:17.027681+0000","last_undegraded":"2024-03-24T01:24:17.027681+0000","last_fullsized":"2024-03-24T01:24:17.027681+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:54:28.205416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809625+0000","last_change":"2024-03-24T01:24:10.755973+0000","last_active":"2024-03-24T01:24:17.809625+0000","last_peered":"2024-03-24T01:24:17.809625+0000","last_clean":"2024-03-24T01:24:17.809625+0000","last_became_active":"2024-03-24T01:24:10.755558+0000","last_became_peered":"2024-03-24T01:24:10.755558+0000","last_unstale":"2024-03-24T01:24:17.809625+0000","last_undegraded":"2024-03-24T01:24:17.809625+0000","last_fullsized":"2024-03-24T01:24:17.809625+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:08:13.760239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.031629+0000","last_change":"2024-03-24T01:24:09.724815+0000","last_active":"2024-03-24T01:24:17.031629+0000","last_peered":"2024-03-24T01:24:17.031629+0000","last_clean":"2024-03-24T01:24:17.031629+0000","last_became_active":"2024-03-24T01:24:09.723550+0000","last_became_peered":"2024-03-24T01:24:09.723550+0000","last_unstale":"2024-03-24T01:24:17.031629+0000","last_undegraded":"2024-03-24T01:24:17.031629+0000","last_fullsized":"2024-03-24T01:24:17.031629+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:21:50.689532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028118+0000","last_change":"2024-03-24T01:24:15.651689+0000","last_active":"2024-03-24T01:24:17.028118+0000","last_peered":"2024-03-24T01:24:17.028118+0000","last_clean":"2024-03-24T01:24:17.028118+0000","last_became_active":"2024-03-24T01:24:15.651006+0000","last_became_peered":"2024-03-24T01:24:15.651006+0000","last_unstale":"2024-03-24T01:24:17.028118+0000","last_undegraded":"2024-03-24T01:24:17.028118+0000","last_fullsized":"2024-03-24T01:24:17.028118+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:52:10.223780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.023121+0000","last_change":"2024-03-24T01:24:10.767341+0000","last_active":"2024-03-24T01:24:17.023121+0000","last_peered":"2024-03-24T01:24:17.023121+0000","last_clean":"2024-03-24T01:24:17.023121+0000","last_became_active":"2024-03-24T01:24:10.767024+0000","last_became_peered":"2024-03-24T01:24:10.767024+0000","last_unstale":"2024-03-24T01:24:17.023121+0000","last_undegraded":"2024-03-24T01:24:17.023121+0000","last_fullsized":"2024-03-24T01:24:17.023121+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:35.441197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.738964+0000","last_change":"2024-03-24T01:24:09.716616+0000","last_active":"2024-03-24T01:24:17.738964+0000","last_peered":"2024-03-24T01:24:17.738964+0000","last_clean":"2024-03-24T01:24:17.738964+0000","last_became_active":"2024-03-24T01:24:09.716363+0000","last_became_peered":"2024-03-24T01:24:09.716363+0000","last_unstale":"2024-03-24T01:24:17.738964+0000","last_undegraded":"2024-03-24T01:24:17.738964+0000","last_fullsized":"2024-03-24T01:24:17.738964+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:33.381248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021124+0000","last_change":"2024-03-24T01:24:15.667639+0000","last_active":"2024-03-24T01:24:17.021124+0000","last_peered":"2024-03-24T01:24:17.021124+0000","last_clean":"2024-03-24T01:24:17.021124+0000","last_became_active":"2024-03-24T01:24:15.664579+0000","last_became_peered":"2024-03-24T01:24:15.664579+0000","last_unstale":"2024-03-24T01:24:17.021124+0000","last_undegraded":"2024-03-24T01:24:17.021124+0000","last_fullsized":"2024-03-24T01:24:17.021124+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:43:54.773518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.740372+0000","last_change":"2024-03-24T01:24:10.750277+0000","last_active":"2024-03-24T01:24:17.740372+0000","last_peered":"2024-03-24T01:24:17.740372+0000","last_clean":"2024-03-24T01:24:17.740372+0000","last_became_active":"2024-03-24T01:24:10.748912+0000","last_became_peered":"2024-03-24T01:24:10.748912+0000","last_unstale":"2024-03-24T01:24:17.740372+0000","last_undegraded":"2024-03-24T01:24:17.740372+0000","last_fullsized":"2024-03-24T01:24:17.740372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:19:51.907364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.812643+0000","last_change":"2024-03-24T01:24:09.723002+0000","last_active":"2024-03-24T01:24:17.812643+0000","last_peered":"2024-03-24T01:24:17.812643+0000","last_clean":"2024-03-24T01:24:17.812643+0000","last_became_active":"2024-03-24T01:24:09.722711+0000","last_became_peered":"2024-03-24T01:24:09.722711+0000","last_unstale":"2024-03-24T01:24:17.812643+0000","last_undegraded":"2024-03-24T01:24:17.812643+0000","last_fullsized":"2024-03-24T01:24:17.812643+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:44:56.790262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809628+0000","last_change":"2024-03-24T01:24:15.657482+0000","last_active":"2024-03-24T01:24:17.809628+0000","last_peered":"2024-03-24T01:24:17.809628+0000","last_clean":"2024-03-24T01:24:17.809628+0000","last_became_active":"2024-03-24T01:24:15.656331+0000","last_became_peered":"2024-03-24T01:24:15.656331+0000","last_unstale":"2024-03-24T01:24:17.809628+0000","last_undegraded":"2024-03-24T01:24:17.809628+0000","last_fullsized":"2024-03-24T01:24:17.809628+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:00.868691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811496+0000","last_change":"2024-03-24T01:24:10.753417+0000","last_active":"2024-03-24T01:24:17.811496+0000","last_peered":"2024-03-24T01:24:17.811496+0000","last_clean":"2024-03-24T01:24:17.811496+0000","last_became_active":"2024-03-24T01:24:10.750938+0000","last_became_peered":"2024-03-24T01:24:10.750938+0000","last_unstale":"2024-03-24T01:24:17.811496+0000","last_undegraded":"2024-03-24T01:24:17.811496+0000","last_fullsized":"2024-03-24T01:24:17.811496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:29:34.357876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.740584+0000","last_change":"2024-03-24T01:24:09.716278+0000","last_active":"2024-03-24T01:24:17.740584+0000","last_peered":"2024-03-24T01:24:17.740584+0000","last_clean":"2024-03-24T01:24:17.740584+0000","last_became_active":"2024-03-24T01:24:09.715984+0000","last_became_peered":"2024-03-24T01:24:09.715984+0000","last_unstale":"2024-03-24T01:24:17.740584+0000","last_undegraded":"2024-03-24T01:24:17.740584+0000","last_fullsized":"2024-03-24T01:24:17.740584+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:44:32.492325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.739876+0000","last_change":"2024-03-24T01:24:15.024902+0000","last_active":"2024-03-24T01:24:17.739876+0000","last_peered":"2024-03-24T01:24:17.739876+0000","last_clean":"2024-03-24T01:24:17.739876+0000","last_became_active":"2024-03-24T01:24:15.024379+0000","last_became_peered":"2024-03-24T01:24:15.024379+0000","last_unstale":"2024-03-24T01:24:17.739876+0000","last_undegraded":"2024-03-24T01:24:17.739876+0000","last_fullsized":"2024-03-24T01:24:17.739876+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:26:07.104726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028172+0000","last_change":"2024-03-24T01:24:10.730294+0000","last_active":"2024-03-24T01:24:17.028172+0000","last_peered":"2024-03-24T01:24:17.028172+0000","last_clean":"2024-03-24T01:24:17.028172+0000","last_became_active":"2024-03-24T01:24:10.727422+0000","last_became_peered":"2024-03-24T01:24:10.727422+0000","last_unstale":"2024-03-24T01:24:17.028172+0000","last_undegraded":"2024-03-24T01:24:17.028172+0000","last_fullsized":"2024-03-24T01:24:17.028172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:34.393331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811092+0000","last_change":"2024-03-24T01:24:09.736152+0000","last_active":"2024-03-24T01:24:17.811092+0000","last_peered":"2024-03-24T01:24:17.811092+0000","last_clean":"2024-03-24T01:24:17.811092+0000","last_became_active":"2024-03-24T01:24:09.735870+0000","last_became_peered":"2024-03-24T01:24:09.735870+0000","last_unstale":"2024-03-24T01:24:17.811092+0000","last_undegraded":"2024-03-24T01:24:17.811092+0000","last_fullsized":"2024-03-24T01:24:17.811092+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:50:31.967556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.740556+0000","last_change":"2024-03-24T01:24:15.027111+0000","last_active":"2024-03-24T01:24:17.740556+0000","last_peered":"2024-03-24T01:24:17.740556+0000","last_clean":"2024-03-24T01:24:17.740556+0000","last_became_active":"2024-03-24T01:24:15.025568+0000","last_became_peered":"2024-03-24T01:24:15.025568+0000","last_unstale":"2024-03-24T01:24:17.740556+0000","last_undegraded":"2024-03-24T01:24:17.740556+0000","last_fullsized":"2024-03-24T01:24:17.740556+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:54:28.205416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.739727+0000","last_change":"2024-03-24T01:24:10.744796+0000","last_active":"2024-03-24T01:24:17.739727+0000","last_peered":"2024-03-24T01:24:17.739727+0000","last_clean":"2024-03-24T01:24:17.739727+0000","last_became_active":"2024-03-24T01:24:10.744533+0000","last_became_peered":"2024-03-24T01:24:10.744533+0000","last_unstale":"2024-03-24T01:24:17.739727+0000","last_undegraded":"2024-03-24T01:24:17.739727+0000","last_fullsized":"2024-03-24T01:24:17.739727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:09:31.705353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.807544+0000","last_change":"2024-03-24T01:24:09.717922+0000","last_active":"2024-03-24T01:24:17.807544+0000","last_peered":"2024-03-24T01:24:17.807544+0000","last_clean":"2024-03-24T01:24:17.807544+0000","last_became_active":"2024-03-24T01:24:09.716468+0000","last_became_peered":"2024-03-24T01:24:09.716468+0000","last_unstale":"2024-03-24T01:24:17.807544+0000","last_undegraded":"2024-03-24T01:24:17.807544+0000","last_fullsized":"2024-03-24T01:24:17.807544+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:50:57.480847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.741366+0000","last_change":"2024-03-24T01:24:15.028389+0000","last_active":"2024-03-24T01:24:17.741366+0000","last_peered":"2024-03-24T01:24:17.741366+0000","last_clean":"2024-03-24T01:24:17.741366+0000","last_became_active":"2024-03-24T01:24:15.027052+0000","last_became_peered":"2024-03-24T01:24:15.027052+0000","last_unstale":"2024-03-24T01:24:17.741366+0000","last_undegraded":"2024-03-24T01:24:17.741366+0000","last_fullsized":"2024-03-24T01:24:17.741366+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:00.868691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.808527+0000","last_change":"2024-03-24T01:24:10.726234+0000","last_active":"2024-03-24T01:24:17.808527+0000","last_peered":"2024-03-24T01:24:17.808527+0000","last_clean":"2024-03-24T01:24:17.808527+0000","last_became_active":"2024-03-24T01:24:10.725471+0000","last_became_peered":"2024-03-24T01:24:10.725471+0000","last_unstale":"2024-03-24T01:24:17.808527+0000","last_undegraded":"2024-03-24T01:24:17.808527+0000","last_fullsized":"2024-03-24T01:24:17.808527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:55:14.169541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809601+0000","last_change":"2024-03-24T01:24:09.726977+0000","last_active":"2024-03-24T01:24:17.809601+0000","last_peered":"2024-03-24T01:24:17.809601+0000","last_clean":"2024-03-24T01:24:17.809601+0000","last_became_active":"2024-03-24T01:24:09.726515+0000","last_became_peered":"2024-03-24T01:24:09.726515+0000","last_unstale":"2024-03-24T01:24:17.809601+0000","last_undegraded":"2024-03-24T01:24:17.809601+0000","last_fullsized":"2024-03-24T01:24:17.809601+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:58:17.247099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809443+0000","last_change":"2024-03-24T01:24:15.662691+0000","last_active":"2024-03-24T01:24:17.809443+0000","last_peered":"2024-03-24T01:24:17.809443+0000","last_clean":"2024-03-24T01:24:17.809443+0000","last_became_active":"2024-03-24T01:24:15.657278+0000","last_became_peered":"2024-03-24T01:24:15.657278+0000","last_unstale":"2024-03-24T01:24:17.809443+0000","last_undegraded":"2024-03-24T01:24:17.809443+0000","last_fullsized":"2024-03-24T01:24:17.809443+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:54:28.205416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021915+0000","last_change":"2024-03-24T01:24:10.744706+0000","last_active":"2024-03-24T01:24:17.021915+0000","last_peered":"2024-03-24T01:24:17.021915+0000","last_clean":"2024-03-24T01:24:17.021915+0000","last_became_active":"2024-03-24T01:24:10.744433+0000","last_became_peered":"2024-03-24T01:24:10.744433+0000","last_unstale":"2024-03-24T01:24:17.021915+0000","last_undegraded":"2024-03-24T01:24:17.021915+0000","last_fullsized":"2024-03-24T01:24:17.021915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:09:31.705353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809513+0000","last_change":"2024-03-24T01:24:09.737525+0000","last_active":"2024-03-24T01:24:17.809513+0000","last_peered":"2024-03-24T01:24:17.809513+0000","last_clean":"2024-03-24T01:24:17.809513+0000","last_became_active":"2024-03-24T01:24:09.737206+0000","last_became_peered":"2024-03-24T01:24:09.737206+0000","last_unstale":"2024-03-24T01:24:17.809513+0000","last_undegraded":"2024-03-24T01:24:17.809513+0000","last_fullsized":"2024-03-24T01:24:17.809513+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:44:32.492325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":179,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.808453+0000","last_change":"2024-03-24T01:24:01.545855+0000","last_active":"2024-03-24T01:24:17.808453+0000","last_peered":"2024-03-24T01:24:17.808453+0000","last_clean":"2024-03-24T01:24:17.808453+0000","last_became_active":"2024-03-24T01:24:01.543811+0000","last_became_peered":"2024-03-24T01:24:01.543811+0000","last_unstale":"2024-03-24T01:24:17.808453+0000","last_undegraded":"2024-03-24T01:24:17.808453+0000","last_fullsized":"2024-03-24T01:24:17.808453+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_clean_scrub_stamp":"2024-03-24T01:23:59.861564+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:03.682631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.031327+0000","last_change":"2024-03-24T01:24:15.656774+0000","last_active":"2024-03-24T01:24:17.031327+0000","last_peered":"2024-03-24T01:24:17.031327+0000","last_clean":"2024-03-24T01:24:17.031327+0000","last_became_active":"2024-03-24T01:24:15.656377+0000","last_became_peered":"2024-03-24T01:24:15.656377+0000","last_unstale":"2024-03-24T01:24:17.031327+0000","last_undegraded":"2024-03-24T01:24:17.031327+0000","last_fullsized":"2024-03-24T01:24:17.031327+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:34:04.384239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.812176+0000","last_change":"2024-03-24T01:24:10.766537+0000","last_active":"2024-03-24T01:24:17.812176+0000","last_peered":"2024-03-24T01:24:17.812176+0000","last_clean":"2024-03-24T01:24:17.812176+0000","last_became_active":"2024-03-24T01:24:10.765713+0000","last_became_peered":"2024-03-24T01:24:10.765713+0000","last_unstale":"2024-03-24T01:24:17.812176+0000","last_undegraded":"2024-03-24T01:24:17.812176+0000","last_fullsized":"2024-03-24T01:24:17.812176+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:18:24.974821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.810980+0000","last_change":"2024-03-24T01:24:09.731795+0000","last_active":"2024-03-24T01:24:17.810980+0000","last_peered":"2024-03-24T01:24:17.810980+0000","last_clean":"2024-03-24T01:24:17.810980+0000","last_became_active":"2024-03-24T01:24:09.731547+0000","last_became_peered":"2024-03-24T01:24:09.731547+0000","last_unstale":"2024-03-24T01:24:17.810980+0000","last_undegraded":"2024-03-24T01:24:17.810980+0000","last_fullsized":"2024-03-24T01:24:17.810980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:31:46.512174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028049+0000","last_change":"2024-03-24T01:24:15.656206+0000","last_active":"2024-03-24T01:24:17.028049+0000","last_peered":"2024-03-24T01:24:17.028049+0000","last_clean":"2024-03-24T01:24:17.028049+0000","last_became_active":"2024-03-24T01:24:15.655557+0000","last_became_peered":"2024-03-24T01:24:15.655557+0000","last_unstale":"2024-03-24T01:24:17.028049+0000","last_undegraded":"2024-03-24T01:24:17.028049+0000","last_fullsized":"2024-03-24T01:24:17.028049+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:18:29.263700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811389+0000","last_change":"2024-03-24T01:24:10.750476+0000","last_active":"2024-03-24T01:24:17.811389+0000","last_peered":"2024-03-24T01:24:17.811389+0000","last_clean":"2024-03-24T01:24:17.811389+0000","last_became_active":"2024-03-24T01:24:10.750171+0000","last_became_peered":"2024-03-24T01:24:10.750171+0000","last_unstale":"2024-03-24T01:24:17.811389+0000","last_undegraded":"2024-03-24T01:24:17.811389+0000","last_fullsized":"2024-03-24T01:24:17.811389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:37:21.153754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.739005+0000","last_change":"2024-03-24T01:24:09.718128+0000","last_active":"2024-03-24T01:24:17.739005+0000","last_peered":"2024-03-24T01:24:17.739005+0000","last_clean":"2024-03-24T01:24:17.739005+0000","last_became_active":"2024-03-24T01:24:09.717669+0000","last_became_peered":"2024-03-24T01:24:09.717669+0000","last_unstale":"2024-03-24T01:24:17.739005+0000","last_undegraded":"2024-03-24T01:24:17.739005+0000","last_fullsized":"2024-03-24T01:24:17.739005+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:21:50.689532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.810011+0000","last_change":"2024-03-24T01:24:15.033291+0000","last_active":"2024-03-24T01:24:17.810011+0000","last_peered":"2024-03-24T01:24:17.810011+0000","last_clean":"2024-03-24T01:24:17.810011+0000","last_became_active":"2024-03-24T01:24:15.032790+0000","last_became_peered":"2024-03-24T01:24:15.032790+0000","last_unstale":"2024-03-24T01:24:17.810011+0000","last_undegraded":"2024-03-24T01:24:17.810011+0000","last_fullsized":"2024-03-24T01:24:17.810011+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:52:10.223780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.813313+0000","last_change":"2024-03-24T01:24:10.765266+0000","last_active":"2024-03-24T01:24:17.813313+0000","last_peered":"2024-03-24T01:24:17.813313+0000","last_clean":"2024-03-24T01:24:17.813313+0000","last_became_active":"2024-03-24T01:24:10.764758+0000","last_became_peered":"2024-03-24T01:24:10.764758+0000","last_unstale":"2024-03-24T01:24:17.813313+0000","last_undegraded":"2024-03-24T01:24:17.813313+0000","last_fullsized":"2024-03-24T01:24:17.813313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:34:00.095359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.807653+0000","last_change":"2024-03-24T01:24:09.714389+0000","last_active":"2024-03-24T01:24:17.807653+0000","last_peered":"2024-03-24T01:24:17.807653+0000","last_clean":"2024-03-24T01:24:17.807653+0000","last_became_active":"2024-03-24T01:24:09.713985+0000","last_became_peered":"2024-03-24T01:24:09.713985+0000","last_unstale":"2024-03-24T01:24:17.807653+0000","last_undegraded":"2024-03-24T01:24:17.807653+0000","last_fullsized":"2024-03-24T01:24:17.807653+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:50:31.967556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809004+0000","last_change":"2024-03-24T01:24:15.656465+0000","last_active":"2024-03-24T01:24:17.809004+0000","last_peered":"2024-03-24T01:24:17.809004+0000","last_clean":"2024-03-24T01:24:17.809004+0000","last_became_active":"2024-03-24T01:24:15.655849+0000","last_became_peered":"2024-03-24T01:24:15.655849+0000","last_unstale":"2024-03-24T01:24:17.809004+0000","last_undegraded":"2024-03-24T01:24:17.809004+0000","last_fullsized":"2024-03-24T01:24:17.809004+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:00.868691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.812175+0000","last_change":"2024-03-24T01:24:10.767829+0000","last_active":"2024-03-24T01:24:17.812175+0000","last_peered":"2024-03-24T01:24:17.812175+0000","last_clean":"2024-03-24T01:24:17.812175+0000","last_became_active":"2024-03-24T01:24:10.766346+0000","last_became_peered":"2024-03-24T01:24:10.766346+0000","last_unstale":"2024-03-24T01:24:17.812175+0000","last_undegraded":"2024-03-24T01:24:17.812175+0000","last_fullsized":"2024-03-24T01:24:17.812175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:24.434151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028899+0000","last_change":"2024-03-24T01:24:09.723282+0000","last_active":"2024-03-24T01:24:17.028899+0000","last_peered":"2024-03-24T01:24:17.028899+0000","last_clean":"2024-03-24T01:24:17.028899+0000","last_became_active":"2024-03-24T01:24:09.722727+0000","last_became_peered":"2024-03-24T01:24:09.722727+0000","last_unstale":"2024-03-24T01:24:17.028899+0000","last_undegraded":"2024-03-24T01:24:17.028899+0000","last_fullsized":"2024-03-24T01:24:17.028899+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:40:14.201223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.027169+0000","last_change":"2024-03-24T01:24:15.027468+0000","last_active":"2024-03-24T01:24:17.027169+0000","last_peered":"2024-03-24T01:24:17.027169+0000","last_clean":"2024-03-24T01:24:17.027169+0000","last_became_active":"2024-03-24T01:24:15.027086+0000","last_became_peered":"2024-03-24T01:24:15.027086+0000","last_unstale":"2024-03-24T01:24:17.027169+0000","last_undegraded":"2024-03-24T01:24:17.027169+0000","last_fullsized":"2024-03-24T01:24:17.027169+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:34:04.384239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809223+0000","last_change":"2024-03-24T01:24:10.756853+0000","last_active":"2024-03-24T01:24:17.809223+0000","last_peered":"2024-03-24T01:24:17.809223+0000","last_clean":"2024-03-24T01:24:17.809223+0000","last_became_active":"2024-03-24T01:24:10.756575+0000","last_became_peered":"2024-03-24T01:24:10.756575+0000","last_unstale":"2024-03-24T01:24:17.809223+0000","last_undegraded":"2024-03-24T01:24:17.809223+0000","last_fullsized":"2024-03-24T01:24:17.809223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:26:02.815847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.808921+0000","last_change":"2024-03-24T01:24:09.720936+0000","last_active":"2024-03-24T01:24:17.808921+0000","last_peered":"2024-03-24T01:24:17.808921+0000","last_clean":"2024-03-24T01:24:17.808921+0000","last_became_active":"2024-03-24T01:24:09.720598+0000","last_became_peered":"2024-03-24T01:24:09.720598+0000","last_unstale":"2024-03-24T01:24:17.808921+0000","last_undegraded":"2024-03-24T01:24:17.808921+0000","last_fullsized":"2024-03-24T01:24:17.808921+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:44:32.492325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809525+0000","last_change":"2024-03-24T01:24:15.027473+0000","last_active":"2024-03-24T01:24:17.809525+0000","last_peered":"2024-03-24T01:24:17.809525+0000","last_clean":"2024-03-24T01:24:17.809525+0000","last_became_active":"2024-03-24T01:24:15.025176+0000","last_became_peered":"2024-03-24T01:24:15.025176+0000","last_unstale":"2024-03-24T01:24:17.809525+0000","last_undegraded":"2024-03-24T01:24:17.809525+0000","last_fullsized":"2024-03-24T01:24:17.809525+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:52:30.401135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.030178+0000","last_change":"2024-03-24T01:24:10.737343+0000","last_active":"2024-03-24T01:24:17.030178+0000","last_peered":"2024-03-24T01:24:17.030178+0000","last_clean":"2024-03-24T01:24:17.030178+0000","last_became_active":"2024-03-24T01:24:10.737040+0000","last_became_peered":"2024-03-24T01:24:10.737040+0000","last_unstale":"2024-03-24T01:24:17.030178+0000","last_undegraded":"2024-03-24T01:24:17.030178+0000","last_fullsized":"2024-03-24T01:24:17.030178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:54:57.105205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.029522+0000","last_change":"2024-03-24T01:24:09.726911+0000","last_active":"2024-03-24T01:24:17.029522+0000","last_peered":"2024-03-24T01:24:17.029522+0000","last_clean":"2024-03-24T01:24:17.029522+0000","last_became_active":"2024-03-24T01:24:09.726578+0000","last_became_peered":"2024-03-24T01:24:09.726578+0000","last_unstale":"2024-03-24T01:24:17.029522+0000","last_undegraded":"2024-03-24T01:24:17.029522+0000","last_fullsized":"2024-03-24T01:24:17.029522+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:44:32.492325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.023074+0000","last_change":"2024-03-24T01:24:15.660631+0000","last_active":"2024-03-24T01:24:17.023074+0000","last_peered":"2024-03-24T01:24:17.023074+0000","last_clean":"2024-03-24T01:24:17.023074+0000","last_became_active":"2024-03-24T01:24:15.656242+0000","last_became_peered":"2024-03-24T01:24:15.656242+0000","last_unstale":"2024-03-24T01:24:17.023074+0000","last_undegraded":"2024-03-24T01:24:17.023074+0000","last_fullsized":"2024-03-24T01:24:17.023074+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:10:41.483921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811861+0000","last_change":"2024-03-24T01:24:10.762123+0000","last_active":"2024-03-24T01:24:17.811861+0000","last_peered":"2024-03-24T01:24:17.811861+0000","last_clean":"2024-03-24T01:24:17.811861+0000","last_became_active":"2024-03-24T01:24:10.761599+0000","last_became_peered":"2024-03-24T01:24:10.761599+0000","last_unstale":"2024-03-24T01:24:17.811861+0000","last_undegraded":"2024-03-24T01:24:17.811861+0000","last_fullsized":"2024-03-24T01:24:17.811861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:54:23.916536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809077+0000","last_change":"2024-03-24T01:24:09.735604+0000","last_active":"2024-03-24T01:24:17.809077+0000","last_peered":"2024-03-24T01:24:17.809077+0000","last_clean":"2024-03-24T01:24:17.809077+0000","last_became_active":"2024-03-24T01:24:09.735373+0000","last_became_peered":"2024-03-24T01:24:09.735373+0000","last_unstale":"2024-03-24T01:24:17.809077+0000","last_undegraded":"2024-03-24T01:24:17.809077+0000","last_fullsized":"2024-03-24T01:24:17.809077+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:36:52.007109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022699+0000","last_change":"2024-03-24T01:24:15.660466+0000","last_active":"2024-03-24T01:24:17.022699+0000","last_peered":"2024-03-24T01:24:17.022699+0000","last_clean":"2024-03-24T01:24:17.022699+0000","last_became_active":"2024-03-24T01:24:15.659265+0000","last_became_peered":"2024-03-24T01:24:15.659265+0000","last_unstale":"2024-03-24T01:24:17.022699+0000","last_undegraded":"2024-03-24T01:24:17.022699+0000","last_fullsized":"2024-03-24T01:24:17.022699+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:56.567936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809258+0000","last_change":"2024-03-24T01:24:10.762021+0000","last_active":"2024-03-24T01:24:17.809258+0000","last_peered":"2024-03-24T01:24:17.809258+0000","last_clean":"2024-03-24T01:24:17.809258+0000","last_became_active":"2024-03-24T01:24:10.761759+0000","last_became_peered":"2024-03-24T01:24:10.761759+0000","last_unstale":"2024-03-24T01:24:17.809258+0000","last_undegraded":"2024-03-24T01:24:17.809258+0000","last_fullsized":"2024-03-24T01:24:17.809258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:49:42.059380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.807512+0000","last_change":"2024-03-24T01:24:09.726149+0000","last_active":"2024-03-24T01:24:17.807512+0000","last_peered":"2024-03-24T01:24:17.807512+0000","last_clean":"2024-03-24T01:24:17.807512+0000","last_became_active":"2024-03-24T01:24:09.725594+0000","last_became_peered":"2024-03-24T01:24:09.725594+0000","last_unstale":"2024-03-24T01:24:17.807512+0000","last_undegraded":"2024-03-24T01:24:17.807512+0000","last_fullsized":"2024-03-24T01:24:17.807512+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:28:34.408651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811101+0000","last_change":"2024-03-24T01:24:15.655846+0000","last_active":"2024-03-24T01:24:17.811101+0000","last_peered":"2024-03-24T01:24:17.811101+0000","last_clean":"2024-03-24T01:24:17.811101+0000","last_became_active":"2024-03-24T01:24:15.655258+0000","last_became_peered":"2024-03-24T01:24:15.655258+0000","last_unstale":"2024-03-24T01:24:17.811101+0000","last_undegraded":"2024-03-24T01:24:17.811101+0000","last_fullsized":"2024-03-24T01:24:17.811101+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:52:10.223780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811833+0000","last_change":"2024-03-24T01:24:10.769350+0000","last_active":"2024-03-24T01:24:17.811833+0000","last_peered":"2024-03-24T01:24:17.811833+0000","last_clean":"2024-03-24T01:24:17.811833+0000","last_became_active":"2024-03-24T01:24:10.766162+0000","last_became_peered":"2024-03-24T01:24:10.766162+0000","last_unstale":"2024-03-24T01:24:17.811833+0000","last_undegraded":"2024-03-24T01:24:17.811833+0000","last_fullsized":"2024-03-24T01:24:17.811833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:38:07.266851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'10","reported_seq":34,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:28.484782+0000","last_change":"2024-03-24T01:24:09.716226+0000","last_active":"2024-03-24T01:24:28.484782+0000","last_peered":"2024-03-24T01:24:28.484782+0000","last_clean":"2024-03-24T01:24:28.484782+0000","last_became_active":"2024-03-24T01:24:09.715772+0000","last_became_peered":"2024-03-24T01:24:09.715772+0000","last_unstale":"2024-03-24T01:24:28.484782+0000","last_undegraded":"2024-03-24T01:24:28.484782+0000","last_fullsized":"2024-03-24T01:24:28.484782+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:50:31.967556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.740168+0000","last_change":"2024-03-24T01:24:15.024014+0000","last_active":"2024-03-24T01:24:17.740168+0000","last_peered":"2024-03-24T01:24:17.740168+0000","last_clean":"2024-03-24T01:24:17.740168+0000","last_became_active":"2024-03-24T01:24:15.023619+0000","last_became_peered":"2024-03-24T01:24:15.023619+0000","last_unstale":"2024-03-24T01:24:17.740168+0000","last_undegraded":"2024-03-24T01:24:17.740168+0000","last_fullsized":"2024-03-24T01:24:17.740168+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:45:17.532730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028200+0000","last_change":"2024-03-24T01:24:10.745876+0000","last_active":"2024-03-24T01:24:17.028200+0000","last_peered":"2024-03-24T01:24:17.028200+0000","last_clean":"2024-03-24T01:24:17.028200+0000","last_became_active":"2024-03-24T01:24:10.745539+0000","last_became_peered":"2024-03-24T01:24:10.745539+0000","last_unstale":"2024-03-24T01:24:17.028200+0000","last_undegraded":"2024-03-24T01:24:17.028200+0000","last_fullsized":"2024-03-24T01:24:17.028200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:56:34.907995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.740552+0000","last_change":"2024-03-24T01:24:09.736074+0000","last_active":"2024-03-24T01:24:17.740552+0000","last_peered":"2024-03-24T01:24:17.740552+0000","last_clean":"2024-03-24T01:24:17.740552+0000","last_became_active":"2024-03-24T01:24:09.735807+0000","last_became_peered":"2024-03-24T01:24:09.735807+0000","last_unstale":"2024-03-24T01:24:17.740552+0000","last_undegraded":"2024-03-24T01:24:17.740552+0000","last_fullsized":"2024-03-24T01:24:17.740552+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:50:31.967556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.027828+0000","last_change":"2024-03-24T01:24:15.031909+0000","last_active":"2024-03-24T01:24:17.027828+0000","last_peered":"2024-03-24T01:24:17.027828+0000","last_clean":"2024-03-24T01:24:17.027828+0000","last_became_active":"2024-03-24T01:24:15.031560+0000","last_became_peered":"2024-03-24T01:24:15.031560+0000","last_unstale":"2024-03-24T01:24:17.027828+0000","last_undegraded":"2024-03-24T01:24:17.027828+0000","last_fullsized":"2024-03-24T01:24:17.027828+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:52:30.401135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811414+0000","last_change":"2024-03-24T01:24:10.753950+0000","last_active":"2024-03-24T01:24:17.811414+0000","last_peered":"2024-03-24T01:24:17.811414+0000","last_clean":"2024-03-24T01:24:17.811414+0000","last_became_active":"2024-03-24T01:24:10.742209+0000","last_became_peered":"2024-03-24T01:24:10.742209+0000","last_unstale":"2024-03-24T01:24:17.811414+0000","last_undegraded":"2024-03-24T01:24:17.811414+0000","last_fullsized":"2024-03-24T01:24:17.811414+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:17.138453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809846+0000","last_change":"2024-03-24T01:24:09.735745+0000","last_active":"2024-03-24T01:24:17.809846+0000","last_peered":"2024-03-24T01:24:17.809846+0000","last_clean":"2024-03-24T01:24:17.809846+0000","last_became_active":"2024-03-24T01:24:09.735438+0000","last_became_peered":"2024-03-24T01:24:09.735438+0000","last_unstale":"2024-03-24T01:24:17.809846+0000","last_undegraded":"2024-03-24T01:24:17.809846+0000","last_fullsized":"2024-03-24T01:24:17.809846+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:58:17.247099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021471+0000","last_change":"2024-03-24T01:24:15.014619+0000","last_active":"2024-03-24T01:24:17.021471+0000","last_peered":"2024-03-24T01:24:17.021471+0000","last_clean":"2024-03-24T01:24:17.021471+0000","last_became_active":"2024-03-24T01:24:15.010859+0000","last_became_peered":"2024-03-24T01:24:15.010859+0000","last_unstale":"2024-03-24T01:24:17.021471+0000","last_undegraded":"2024-03-24T01:24:17.021471+0000","last_fullsized":"2024-03-24T01:24:17.021471+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:45:23.567638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811149+0000","last_change":"2024-03-24T01:24:10.767592+0000","last_active":"2024-03-24T01:24:17.811149+0000","last_peered":"2024-03-24T01:24:17.811149+0000","last_clean":"2024-03-24T01:24:17.811149+0000","last_became_active":"2024-03-24T01:24:10.767264+0000","last_became_peered":"2024-03-24T01:24:10.767264+0000","last_unstale":"2024-03-24T01:24:17.811149+0000","last_undegraded":"2024-03-24T01:24:17.811149+0000","last_fullsized":"2024-03-24T01:24:17.811149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:18:24.974821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.031354+0000","last_change":"2024-03-24T01:24:09.738634+0000","last_active":"2024-03-24T01:24:17.031354+0000","last_peered":"2024-03-24T01:24:17.031354+0000","last_clean":"2024-03-24T01:24:17.031354+0000","last_became_active":"2024-03-24T01:24:09.738308+0000","last_became_peered":"2024-03-24T01:24:09.738308+0000","last_unstale":"2024-03-24T01:24:17.031354+0000","last_undegraded":"2024-03-24T01:24:17.031354+0000","last_fullsized":"2024-03-24T01:24:17.031354+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:36:52.007109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021706+0000","last_change":"2024-03-24T01:24:15.666864+0000","last_active":"2024-03-24T01:24:17.021706+0000","last_peered":"2024-03-24T01:24:17.021706+0000","last_clean":"2024-03-24T01:24:17.021706+0000","last_became_active":"2024-03-24T01:24:15.659581+0000","last_became_peered":"2024-03-24T01:24:15.659581+0000","last_unstale":"2024-03-24T01:24:17.021706+0000","last_undegraded":"2024-03-24T01:24:17.021706+0000","last_fullsized":"2024-03-24T01:24:17.021706+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:40:09.145961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809258+0000","last_change":"2024-03-24T01:24:15.657385+0000","last_active":"2024-03-24T01:24:17.809258+0000","last_peered":"2024-03-24T01:24:17.809258+0000","last_clean":"2024-03-24T01:24:17.809258+0000","last_became_active":"2024-03-24T01:24:15.656275+0000","last_became_peered":"2024-03-24T01:24:15.656275+0000","last_unstale":"2024-03-24T01:24:17.809258+0000","last_undegraded":"2024-03-24T01:24:17.809258+0000","last_fullsized":"2024-03-24T01:24:17.809258+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:45:17.532730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.739356+0000","last_change":"2024-03-24T01:24:10.750827+0000","last_active":"2024-03-24T01:24:17.739356+0000","last_peered":"2024-03-24T01:24:17.739356+0000","last_clean":"2024-03-24T01:24:17.739356+0000","last_became_active":"2024-03-24T01:24:10.750522+0000","last_became_peered":"2024-03-24T01:24:10.750522+0000","last_unstale":"2024-03-24T01:24:17.739356+0000","last_undegraded":"2024-03-24T01:24:17.739356+0000","last_fullsized":"2024-03-24T01:24:17.739356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:04:48.028091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.808340+0000","last_change":"2024-03-24T01:24:09.730771+0000","last_active":"2024-03-24T01:24:17.808340+0000","last_peered":"2024-03-24T01:24:17.808340+0000","last_clean":"2024-03-24T01:24:17.808340+0000","last_became_active":"2024-03-24T01:24:09.730424+0000","last_became_peered":"2024-03-24T01:24:09.730424+0000","last_unstale":"2024-03-24T01:24:17.808340+0000","last_undegraded":"2024-03-24T01:24:17.808340+0000","last_fullsized":"2024-03-24T01:24:17.808340+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:50:57.480847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.031001+0000","last_change":"2024-03-24T01:24:15.029656+0000","last_active":"2024-03-24T01:24:17.031001+0000","last_peered":"2024-03-24T01:24:17.031001+0000","last_clean":"2024-03-24T01:24:17.031001+0000","last_became_active":"2024-03-24T01:24:15.029240+0000","last_became_peered":"2024-03-24T01:24:15.029240+0000","last_unstale":"2024-03-24T01:24:17.031001+0000","last_undegraded":"2024-03-24T01:24:17.031001+0000","last_fullsized":"2024-03-24T01:24:17.031001+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:28.723031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021468+0000","last_change":"2024-03-24T01:24:10.750421+0000","last_active":"2024-03-24T01:24:17.021468+0000","last_peered":"2024-03-24T01:24:17.021468+0000","last_clean":"2024-03-24T01:24:17.021468+0000","last_became_active":"2024-03-24T01:24:10.750160+0000","last_became_peered":"2024-03-24T01:24:10.750160+0000","last_unstale":"2024-03-24T01:24:17.021468+0000","last_undegraded":"2024-03-24T01:24:17.021468+0000","last_fullsized":"2024-03-24T01:24:17.021468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:54:57.105205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022107+0000","last_change":"2024-03-24T01:24:09.722993+0000","last_active":"2024-03-24T01:24:17.022107+0000","last_peered":"2024-03-24T01:24:17.022107+0000","last_clean":"2024-03-24T01:24:17.022107+0000","last_became_active":"2024-03-24T01:24:09.720133+0000","last_became_peered":"2024-03-24T01:24:09.720133+0000","last_unstale":"2024-03-24T01:24:17.022107+0000","last_undegraded":"2024-03-24T01:24:17.022107+0000","last_fullsized":"2024-03-24T01:24:17.022107+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:28:34.408651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.030554+0000","last_change":"2024-03-24T01:24:09.726157+0000","last_active":"2024-03-24T01:24:17.030554+0000","last_peered":"2024-03-24T01:24:17.030554+0000","last_clean":"2024-03-24T01:24:17.030554+0000","last_became_active":"2024-03-24T01:24:09.724571+0000","last_became_peered":"2024-03-24T01:24:09.724571+0000","last_unstale":"2024-03-24T01:24:17.030554+0000","last_undegraded":"2024-03-24T01:24:17.030554+0000","last_fullsized":"2024-03-24T01:24:17.030554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:33.381248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.810098+0000","last_change":"2024-03-24T01:24:10.726759+0000","last_active":"2024-03-24T01:24:17.810098+0000","last_peered":"2024-03-24T01:24:17.810098+0000","last_clean":"2024-03-24T01:24:17.810098+0000","last_became_active":"2024-03-24T01:24:10.726133+0000","last_became_peered":"2024-03-24T01:24:10.726133+0000","last_unstale":"2024-03-24T01:24:17.810098+0000","last_undegraded":"2024-03-24T01:24:17.810098+0000","last_fullsized":"2024-03-24T01:24:17.810098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:46:04.235199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.810763+0000","last_change":"2024-03-24T01:24:15.026408+0000","last_active":"2024-03-24T01:24:17.810763+0000","last_peered":"2024-03-24T01:24:17.810763+0000","last_clean":"2024-03-24T01:24:17.810763+0000","last_became_active":"2024-03-24T01:24:15.026038+0000","last_became_peered":"2024-03-24T01:24:15.026038+0000","last_unstale":"2024-03-24T01:24:17.810763+0000","last_undegraded":"2024-03-24T01:24:17.810763+0000","last_fullsized":"2024-03-24T01:24:17.810763+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:55:18.458421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022088+0000","last_change":"2024-03-24T01:24:09.722548+0000","last_active":"2024-03-24T01:24:17.022088+0000","last_peered":"2024-03-24T01:24:17.022088+0000","last_clean":"2024-03-24T01:24:17.022088+0000","last_became_active":"2024-03-24T01:24:09.721937+0000","last_became_peered":"2024-03-24T01:24:09.721937+0000","last_unstale":"2024-03-24T01:24:17.022088+0000","last_undegraded":"2024-03-24T01:24:17.022088+0000","last_fullsized":"2024-03-24T01:24:17.022088+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:08.671148+0000","last_clean_scrub_stamp":"2024-03-24T01:24:08.671148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:50:57.480847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.027732+0000","last_change":"2024-03-24T01:24:10.743972+0000","last_active":"2024-03-24T01:24:17.027732+0000","last_peered":"2024-03-24T01:24:17.027732+0000","last_clean":"2024-03-24T01:24:17.027732+0000","last_became_active":"2024-03-24T01:24:10.743700+0000","last_became_peered":"2024-03-24T01:24:10.743700+0000","last_unstale":"2024-03-24T01:24:17.027732+0000","last_undegraded":"2024-03-24T01:24:17.027732+0000","last_fullsized":"2024-03-24T01:24:17.027732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:21:51.701615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022373+0000","last_change":"2024-03-24T01:24:15.023995+0000","last_active":"2024-03-24T01:24:17.022373+0000","last_peered":"2024-03-24T01:24:17.022373+0000","last_clean":"2024-03-24T01:24:17.022373+0000","last_became_active":"2024-03-24T01:24:15.023574+0000","last_became_peered":"2024-03-24T01:24:15.023574+0000","last_unstale":"2024-03-24T01:24:17.022373+0000","last_undegraded":"2024-03-24T01:24:17.022373+0000","last_fullsized":"2024-03-24T01:24:17.022373+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:13.972110+0000","last_clean_scrub_stamp":"2024-03-24T01:24:13.972110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:26:01.912499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.029925+0000","last_change":"2024-03-24T01:24:10.740908+0000","last_active":"2024-03-24T01:24:17.029925+0000","last_peered":"2024-03-24T01:24:17.029925+0000","last_clean":"2024-03-24T01:24:17.029925+0000","last_became_active":"2024-03-24T01:24:10.740630+0000","last_became_peered":"2024-03-24T01:24:10.740630+0000","last_unstale":"2024-03-24T01:24:17.029925+0000","last_undegraded":"2024-03-24T01:24:17.029925+0000","last_fullsized":"2024-03-24T01:24:17.029925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:04:48.028091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809885+0000","last_change":"2024-03-24T01:24:10.743826+0000","last_active":"2024-03-24T01:24:17.809885+0000","last_peered":"2024-03-24T01:24:17.809885+0000","last_clean":"2024-03-24T01:24:17.809885+0000","last_became_active":"2024-03-24T01:24:10.743349+0000","last_became_peered":"2024-03-24T01:24:10.743349+0000","last_unstale":"2024-03-24T01:24:17.809885+0000","last_undegraded":"2024-03-24T01:24:17.809885+0000","last_fullsized":"2024-03-24T01:24:17.809885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:56:34.907995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.810496+0000","last_change":"2024-03-24T01:24:10.755545+0000","last_active":"2024-03-24T01:24:17.810496+0000","last_peered":"2024-03-24T01:24:17.810496+0000","last_clean":"2024-03-24T01:24:17.810496+0000","last_became_active":"2024-03-24T01:24:10.755147+0000","last_became_peered":"2024-03-24T01:24:10.755147+0000","last_unstale":"2024-03-24T01:24:17.810496+0000","last_undegraded":"2024-03-24T01:24:17.810496+0000","last_fullsized":"2024-03-24T01:24:17.810496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:35.441197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809878+0000","last_change":"2024-03-24T01:24:10.746109+0000","last_active":"2024-03-24T01:24:17.809878+0000","last_peered":"2024-03-24T01:24:17.809878+0000","last_clean":"2024-03-24T01:24:17.809878+0000","last_became_active":"2024-03-24T01:24:10.745809+0000","last_became_peered":"2024-03-24T01:24:10.745809+0000","last_unstale":"2024-03-24T01:24:17.809878+0000","last_undegraded":"2024-03-24T01:24:17.809878+0000","last_fullsized":"2024-03-24T01:24:17.809878+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:09:31.705353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022898+0000","last_change":"2024-03-24T01:24:10.752362+0000","last_active":"2024-03-24T01:24:17.022898+0000","last_peered":"2024-03-24T01:24:17.022898+0000","last_clean":"2024-03-24T01:24:17.022898+0000","last_became_active":"2024-03-24T01:24:10.752066+0000","last_became_peered":"2024-03-24T01:24:10.752066+0000","last_unstale":"2024-03-24T01:24:17.022898+0000","last_undegraded":"2024-03-24T01:24:17.022898+0000","last_fullsized":"2024-03-24T01:24:17.022898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:08:13.760239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.740402+0000","last_change":"2024-03-24T01:24:10.751536+0000","last_active":"2024-03-24T01:24:17.740402+0000","last_peered":"2024-03-24T01:24:17.740402+0000","last_clean":"2024-03-24T01:24:17.740402+0000","last_became_active":"2024-03-24T01:24:10.751260+0000","last_became_peered":"2024-03-24T01:24:10.751260+0000","last_unstale":"2024-03-24T01:24:17.740402+0000","last_undegraded":"2024-03-24T01:24:17.740402+0000","last_fullsized":"2024-03-24T01:24:17.740402+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:17.138453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.031836+0000","last_change":"2024-03-24T01:24:10.763037+0000","last_active":"2024-03-24T01:24:17.031836+0000","last_peered":"2024-03-24T01:24:17.031836+0000","last_clean":"2024-03-24T01:24:17.031836+0000","last_became_active":"2024-03-24T01:24:10.762718+0000","last_became_peered":"2024-03-24T01:24:10.762718+0000","last_unstale":"2024-03-24T01:24:17.031836+0000","last_undegraded":"2024-03-24T01:24:17.031836+0000","last_fullsized":"2024-03-24T01:24:17.031836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:46:04.235199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.022992+0000","last_change":"2024-03-24T01:24:10.753332+0000","last_active":"2024-03-24T01:24:17.022992+0000","last_peered":"2024-03-24T01:24:17.022992+0000","last_clean":"2024-03-24T01:24:17.022992+0000","last_became_active":"2024-03-24T01:24:10.752905+0000","last_became_peered":"2024-03-24T01:24:10.752905+0000","last_unstale":"2024-03-24T01:24:17.022992+0000","last_undegraded":"2024-03-24T01:24:17.022992+0000","last_fullsized":"2024-03-24T01:24:17.022992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:57:11.968706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809861+0000","last_change":"2024-03-24T01:24:10.759635+0000","last_active":"2024-03-24T01:24:17.809861+0000","last_peered":"2024-03-24T01:24:17.809861+0000","last_clean":"2024-03-24T01:24:17.809861+0000","last_became_active":"2024-03-24T01:24:10.759310+0000","last_became_peered":"2024-03-24T01:24:10.759310+0000","last_unstale":"2024-03-24T01:24:17.809861+0000","last_undegraded":"2024-03-24T01:24:17.809861+0000","last_fullsized":"2024-03-24T01:24:17.809861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:08:13.760239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021975+0000","last_change":"2024-03-24T01:24:10.746635+0000","last_active":"2024-03-24T01:24:17.021975+0000","last_peered":"2024-03-24T01:24:17.021975+0000","last_clean":"2024-03-24T01:24:17.021975+0000","last_became_active":"2024-03-24T01:24:10.745597+0000","last_became_peered":"2024-03-24T01:24:10.745597+0000","last_unstale":"2024-03-24T01:24:17.021975+0000","last_undegraded":"2024-03-24T01:24:17.021975+0000","last_fullsized":"2024-03-24T01:24:17.021975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:42:22.671154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.810268+0000","last_change":"2024-03-24T01:24:10.743389+0000","last_active":"2024-03-24T01:24:17.810268+0000","last_peered":"2024-03-24T01:24:17.810268+0000","last_clean":"2024-03-24T01:24:17.810268+0000","last_became_active":"2024-03-24T01:24:10.743128+0000","last_became_peered":"2024-03-24T01:24:10.743128+0000","last_unstale":"2024-03-24T01:24:17.810268+0000","last_undegraded":"2024-03-24T01:24:17.810268+0000","last_fullsized":"2024-03-24T01:24:17.810268+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:42:22.671154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809576+0000","last_change":"2024-03-24T01:24:10.760940+0000","last_active":"2024-03-24T01:24:17.809576+0000","last_peered":"2024-03-24T01:24:17.809576+0000","last_clean":"2024-03-24T01:24:17.809576+0000","last_became_active":"2024-03-24T01:24:10.759991+0000","last_became_peered":"2024-03-24T01:24:10.759991+0000","last_unstale":"2024-03-24T01:24:17.809576+0000","last_undegraded":"2024-03-24T01:24:17.809576+0000","last_fullsized":"2024-03-24T01:24:17.809576+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:54:57.105205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.030845+0000","last_change":"2024-03-24T01:24:10.731391+0000","last_active":"2024-03-24T01:24:17.030845+0000","last_peered":"2024-03-24T01:24:17.030845+0000","last_clean":"2024-03-24T01:24:17.030845+0000","last_became_active":"2024-03-24T01:24:10.731094+0000","last_became_peered":"2024-03-24T01:24:10.731094+0000","last_unstale":"2024-03-24T01:24:17.030845+0000","last_undegraded":"2024-03-24T01:24:17.030845+0000","last_fullsized":"2024-03-24T01:24:17.030845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:17.138453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.812469+0000","last_change":"2024-03-24T01:24:10.766746+0000","last_active":"2024-03-24T01:24:17.812469+0000","last_peered":"2024-03-24T01:24:17.812469+0000","last_clean":"2024-03-24T01:24:17.812469+0000","last_became_active":"2024-03-24T01:24:10.766239+0000","last_became_peered":"2024-03-24T01:24:10.766239+0000","last_unstale":"2024-03-24T01:24:17.812469+0000","last_undegraded":"2024-03-24T01:24:17.812469+0000","last_fullsized":"2024-03-24T01:24:17.812469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:45:13.243851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811052+0000","last_change":"2024-03-24T01:24:10.741506+0000","last_active":"2024-03-24T01:24:17.811052+0000","last_peered":"2024-03-24T01:24:17.811052+0000","last_clean":"2024-03-24T01:24:17.811052+0000","last_became_active":"2024-03-24T01:24:10.741221+0000","last_became_peered":"2024-03-24T01:24:10.741221+0000","last_unstale":"2024-03-24T01:24:17.811052+0000","last_undegraded":"2024-03-24T01:24:17.811052+0000","last_fullsized":"2024-03-24T01:24:17.811052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:27:51.341849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.812926+0000","last_change":"2024-03-24T01:24:10.767451+0000","last_active":"2024-03-24T01:24:17.812926+0000","last_peered":"2024-03-24T01:24:17.812926+0000","last_clean":"2024-03-24T01:24:17.812926+0000","last_became_active":"2024-03-24T01:24:10.767159+0000","last_became_peered":"2024-03-24T01:24:10.767159+0000","last_unstale":"2024-03-24T01:24:17.812926+0000","last_undegraded":"2024-03-24T01:24:17.812926+0000","last_fullsized":"2024-03-24T01:24:17.812926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:24.434151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809619+0000","last_change":"2024-03-24T01:24:10.751373+0000","last_active":"2024-03-24T01:24:17.809619+0000","last_peered":"2024-03-24T01:24:17.809619+0000","last_clean":"2024-03-24T01:24:17.809619+0000","last_became_active":"2024-03-24T01:24:10.751084+0000","last_became_peered":"2024-03-24T01:24:10.751084+0000","last_unstale":"2024-03-24T01:24:17.809619+0000","last_undegraded":"2024-03-24T01:24:17.809619+0000","last_fullsized":"2024-03-24T01:24:17.809619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:17.138453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.739324+0000","last_change":"2024-03-24T01:24:10.752504+0000","last_active":"2024-03-24T01:24:17.739324+0000","last_peered":"2024-03-24T01:24:17.739324+0000","last_clean":"2024-03-24T01:24:17.739324+0000","last_became_active":"2024-03-24T01:24:10.752313+0000","last_became_peered":"2024-03-24T01:24:10.752313+0000","last_unstale":"2024-03-24T01:24:17.739324+0000","last_undegraded":"2024-03-24T01:24:17.739324+0000","last_fullsized":"2024-03-24T01:24:17.739324+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:54:57.105205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809195+0000","last_change":"2024-03-24T01:24:10.750967+0000","last_active":"2024-03-24T01:24:17.809195+0000","last_peered":"2024-03-24T01:24:17.809195+0000","last_clean":"2024-03-24T01:24:17.809195+0000","last_became_active":"2024-03-24T01:24:10.748587+0000","last_became_peered":"2024-03-24T01:24:10.748587+0000","last_unstale":"2024-03-24T01:24:17.809195+0000","last_undegraded":"2024-03-24T01:24:17.809195+0000","last_fullsized":"2024-03-24T01:24:17.809195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:44:57.802345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021966+0000","last_change":"2024-03-24T01:24:10.750792+0000","last_active":"2024-03-24T01:24:17.021966+0000","last_peered":"2024-03-24T01:24:17.021966+0000","last_clean":"2024-03-24T01:24:17.021966+0000","last_became_active":"2024-03-24T01:24:10.749384+0000","last_became_peered":"2024-03-24T01:24:10.749384+0000","last_unstale":"2024-03-24T01:24:17.021966+0000","last_undegraded":"2024-03-24T01:24:17.021966+0000","last_fullsized":"2024-03-24T01:24:17.021966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:17.138453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.028133+0000","last_change":"2024-03-24T01:24:10.728222+0000","last_active":"2024-03-24T01:24:17.028133+0000","last_peered":"2024-03-24T01:24:17.028133+0000","last_clean":"2024-03-24T01:24:17.028133+0000","last_became_active":"2024-03-24T01:24:10.727874+0000","last_became_peered":"2024-03-24T01:24:10.727874+0000","last_unstale":"2024-03-24T01:24:17.028133+0000","last_undegraded":"2024-03-24T01:24:17.028133+0000","last_fullsized":"2024-03-24T01:24:17.028133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:58:18.259182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.031302+0000","last_change":"2024-03-24T01:24:10.741809+0000","last_active":"2024-03-24T01:24:17.031302+0000","last_peered":"2024-03-24T01:24:17.031302+0000","last_clean":"2024-03-24T01:24:17.031302+0000","last_became_active":"2024-03-24T01:24:10.741444+0000","last_became_peered":"2024-03-24T01:24:10.741444+0000","last_unstale":"2024-03-24T01:24:17.031302+0000","last_undegraded":"2024-03-24T01:24:17.031302+0000","last_fullsized":"2024-03-24T01:24:17.031302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:19:51.907364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.031726+0000","last_change":"2024-03-24T01:24:10.751752+0000","last_active":"2024-03-24T01:24:17.031726+0000","last_peered":"2024-03-24T01:24:17.031726+0000","last_clean":"2024-03-24T01:24:17.031726+0000","last_became_active":"2024-03-24T01:24:10.747073+0000","last_became_peered":"2024-03-24T01:24:10.747073+0000","last_unstale":"2024-03-24T01:24:17.031726+0000","last_undegraded":"2024-03-24T01:24:17.031726+0000","last_fullsized":"2024-03-24T01:24:17.031726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:08:13.760239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.813263+0000","last_change":"2024-03-24T01:24:10.765006+0000","last_active":"2024-03-24T01:24:17.813263+0000","last_peered":"2024-03-24T01:24:17.813263+0000","last_clean":"2024-03-24T01:24:17.813263+0000","last_became_active":"2024-03-24T01:24:10.764417+0000","last_became_peered":"2024-03-24T01:24:10.764417+0000","last_unstale":"2024-03-24T01:24:17.813263+0000","last_undegraded":"2024-03-24T01:24:17.813263+0000","last_fullsized":"2024-03-24T01:24:17.813263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:29:34.357876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.021162+0000","last_change":"2024-03-24T01:24:10.757149+0000","last_active":"2024-03-24T01:24:17.021162+0000","last_peered":"2024-03-24T01:24:17.021162+0000","last_clean":"2024-03-24T01:24:17.021162+0000","last_became_active":"2024-03-24T01:24:10.756879+0000","last_became_peered":"2024-03-24T01:24:10.756879+0000","last_unstale":"2024-03-24T01:24:17.021162+0000","last_undegraded":"2024-03-24T01:24:17.021162+0000","last_fullsized":"2024-03-24T01:24:17.021162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:55:14.169541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809223+0000","last_change":"2024-03-24T01:24:10.766128+0000","last_active":"2024-03-24T01:24:17.809223+0000","last_peered":"2024-03-24T01:24:17.809223+0000","last_clean":"2024-03-24T01:24:17.809223+0000","last_became_active":"2024-03-24T01:24:10.763007+0000","last_became_peered":"2024-03-24T01:24:10.763007+0000","last_unstale":"2024-03-24T01:24:17.809223+0000","last_undegraded":"2024-03-24T01:24:17.809223+0000","last_fullsized":"2024-03-24T01:24:17.809223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:35.441197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.809997+0000","last_change":"2024-03-24T01:24:10.741795+0000","last_active":"2024-03-24T01:24:17.809997+0000","last_peered":"2024-03-24T01:24:17.809997+0000","last_clean":"2024-03-24T01:24:17.809997+0000","last_became_active":"2024-03-24T01:24:10.741516+0000","last_became_peered":"2024-03-24T01:24:10.741516+0000","last_unstale":"2024-03-24T01:24:17.809997+0000","last_undegraded":"2024-03-24T01:24:17.809997+0000","last_fullsized":"2024-03-24T01:24:17.809997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:57:11.968706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.811350+0000","last_change":"2024-03-24T01:24:10.761849+0000","last_active":"2024-03-24T01:24:17.811350+0000","last_peered":"2024-03-24T01:24:17.811350+0000","last_clean":"2024-03-24T01:24:17.811350+0000","last_became_active":"2024-03-24T01:24:10.761288+0000","last_became_peered":"2024-03-24T01:24:10.761288+0000","last_unstale":"2024-03-24T01:24:17.811350+0000","last_undegraded":"2024-03-24T01:24:17.811350+0000","last_fullsized":"2024-03-24T01:24:17.811350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:02:18.904076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.810985+0000","last_change":"2024-03-24T01:24:10.755426+0000","last_active":"2024-03-24T01:24:17.810985+0000","last_peered":"2024-03-24T01:24:17.810985+0000","last_clean":"2024-03-24T01:24:17.810985+0000","last_became_active":"2024-03-24T01:24:10.755073+0000","last_became_peered":"2024-03-24T01:24:10.755073+0000","last_unstale":"2024-03-24T01:24:17.810985+0000","last_undegraded":"2024-03-24T01:24:17.810985+0000","last_fullsized":"2024-03-24T01:24:17.810985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:46:04.235199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.810708+0000","last_change":"2024-03-24T01:24:10.732890+0000","last_active":"2024-03-24T01:24:17.810708+0000","last_peered":"2024-03-24T01:24:17.810708+0000","last_clean":"2024-03-24T01:24:17.810708+0000","last_became_active":"2024-03-24T01:24:10.731992+0000","last_became_peered":"2024-03-24T01:24:10.731992+0000","last_unstale":"2024-03-24T01:24:17.810708+0000","last_undegraded":"2024-03-24T01:24:17.810708+0000","last_fullsized":"2024-03-24T01:24:17.810708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:19:51.907364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.741104+0000","last_change":"2024-03-24T01:24:10.750945+0000","last_active":"2024-03-24T01:24:17.741104+0000","last_peered":"2024-03-24T01:24:17.741104+0000","last_clean":"2024-03-24T01:24:17.741104+0000","last_became_active":"2024-03-24T01:24:10.750099+0000","last_became_peered":"2024-03-24T01:24:10.750099+0000","last_unstale":"2024-03-24T01:24:17.741104+0000","last_undegraded":"2024-03-24T01:24:17.741104+0000","last_fullsized":"2024-03-24T01:24:17.741104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:08:13.760239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-24T01:24:17.813056+0000","last_change":"2024-03-24T01:24:10.762165+0000","last_active":"2024-03-24T01:24:17.813056+0000","last_peered":"2024-03-24T01:24:17.813056+0000","last_clean":"2024-03-24T01:24:17.813056+0000","last_became_active":"2024-03-24T01:24:10.758157+0000","last_became_peered":"2024-03-24T01:24:10.758157+0000","last_unstale":"2024-03-24T01:24:17.813056+0000","last_undegraded":"2024-03-24T01:24:17.813056+0000","last_fullsized":"2024-03-24T01:24:17.813056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:24:09.683231+0000","last_clean_scrub_stamp":"2024-03-24T01:24:09.683231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:37:21.153754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574859,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27864,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343976,"statfs":{"total":96636764160,"available":96608231424,"internally_reserved":0,"allocated":864256,"data_stored":698343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574860,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":107483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":107519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574859,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":107393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":107553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574860,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":872448,"data_stored":698399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T01:24:44.592 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-24T01:24:44.592 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-24T01:24:44.592 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-24T01:24:44.592 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-24T01:24:44.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.839+0000 7f2886a00640 1 Processor -- start 2024-03-24T01:24:44.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.839+0000 7f2886a00640 1 -- start start 2024-03-24T01:24:44.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.841+0000 7f2886a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2880155030 0x7f288015efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:44.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.841+0000 7f2886a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28800cdac0 0x7f288006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:44.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.841+0000 7f2886a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f288006b600 0x7f28800ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:44.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.841+0000 7f2886a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f288006ba70 con 0x7f288006b600 2024-03-24T01:24:44.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.841+0000 7f2886a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f288015f500 con 0x7f2880155030 2024-03-24T01:24:44.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.841+0000 7f2886a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f288015bf70 con 0x7f28800cdac0 2024-03-24T01:24:44.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.842+0000 7f2884a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f288006b600 0x7f28800ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:44.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.842+0000 7f2884a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f288006b600 0x7f28800ae530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60376/0 (socket says 172.21.15.90:60376) 2024-03-24T01:24:44.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.842+0000 7f2884a00640 1 -- 172.21.15.90:0/482023561 learned_addr learned my addr 172.21.15.90:0/482023561 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:44.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.842+0000 7f287f400640 1 --2- 172.21.15.90:0/482023561 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28800cdac0 0x7f288006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:44.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.842+0000 7f287fe00640 1 --2- 172.21.15.90:0/482023561 >> v2:172.21.15.120:3300/0 conn(0x7f2880155030 0x7f288015efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:44.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.842+0000 7f2884a00640 1 -- 172.21.15.90:0/482023561 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28800cdac0 msgr2=0x7f288006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:44.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.842+0000 7f2884a00640 1 --2- 172.21.15.90:0/482023561 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28800cdac0 0x7f288006b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:44.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.842+0000 7f2884a00640 1 -- 172.21.15.90:0/482023561 >> v2:172.21.15.120:3300/0 conn(0x7f2880155030 msgr2=0x7f288015efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:44.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.842+0000 7f2884a00640 1 --2- 172.21.15.90:0/482023561 >> v2:172.21.15.120:3300/0 conn(0x7f2880155030 0x7f288015efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:44.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.842+0000 7f2884a00640 1 -- 172.21.15.90:0/482023561 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f288015b360 con 0x7f288006b600 2024-03-24T01:24:44.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.842+0000 7f287fe00640 1 --2- 172.21.15.90:0/482023561 >> v2:172.21.15.120:3300/0 conn(0x7f2880155030 0x7f288015efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:24:44.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.843+0000 7f2884a00640 1 --2- 172.21.15.90:0/482023561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f288006b600 0x7f28800ae530 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f288006ecc0 tx=0x7f287802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a943efc0b1585c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:44.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.843+0000 7f287ea00640 1 -- 172.21.15.90:0/482023561 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287802ca80 con 0x7f288006b600 2024-03-24T01:24:44.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.843+0000 7f287ea00640 1 -- 172.21.15.90:0/482023561 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f287802cc20 con 0x7f288006b600 2024-03-24T01:24:44.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.843+0000 7f287ea00640 1 -- 172.21.15.90:0/482023561 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28780343f0 con 0x7f288006b600 2024-03-24T01:24:44.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.844+0000 7f2886a00640 1 -- 172.21.15.90:0/482023561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f288006b600 msgr2=0x7f28800ae530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:44.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.844+0000 7f2886a00640 1 --2- 172.21.15.90:0/482023561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f288006b600 0x7f28800ae530 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f288006ecc0 tx=0x7f287802bee0 comp rx=0 tx=0).stop 2024-03-24T01:24:44.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.844+0000 7f2886a00640 1 -- 172.21.15.90:0/482023561 shutdown_connections 2024-03-24T01:24:44.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.844+0000 7f2886a00640 1 --2- 172.21.15.90:0/482023561 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f28800cdac0 0x7f288006b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:44.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.844+0000 7f2886a00640 1 --2- 172.21.15.90:0/482023561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f288006b600 0x7f28800ae530 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:44.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.844+0000 7f2886a00640 1 --2- 172.21.15.90:0/482023561 >> v2:172.21.15.120:3300/0 conn(0x7f2880155030 0x7f288015efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:44.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.844+0000 7f2886a00640 1 -- 172.21.15.90:0/482023561 >> 172.21.15.90:0/482023561 conn(0x7f28800b1770 msgr2=0x7f28800b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:44.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.844+0000 7f2886a00640 1 -- 172.21.15.90:0/482023561 shutdown_connections 2024-03-24T01:24:44.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.844+0000 7f2886a00640 1 -- 172.21.15.90:0/482023561 wait complete. 2024-03-24T01:24:44.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.846+0000 7f2886a00640 1 Processor -- start 2024-03-24T01:24:44.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.846+0000 7f2886a00640 1 -- start start 2024-03-24T01:24:44.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.847+0000 7f2886a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f288006b600 0x7f28801544c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:44.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.847+0000 7f2886a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28800cdac0 0x7f2880157800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:44.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.847+0000 7f2886a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2880155030 0x7f2880157d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:44.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.847+0000 7f2886a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f288015bc00 con 0x7f28800cdac0 2024-03-24T01:24:44.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.847+0000 7f2886a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f288015b920 con 0x7f288006b600 2024-03-24T01:24:44.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.847+0000 7f2886a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2880155410 con 0x7f2880155030 2024-03-24T01:24:44.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.847+0000 7f287f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28800cdac0 0x7f2880157800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:44.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.847+0000 7f287f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28800cdac0 0x7f2880157800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60380/0 (socket says 172.21.15.90:60380) 2024-03-24T01:24:44.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.847+0000 7f287f400640 1 -- 172.21.15.90:0/3347817564 learned_addr learned my addr 172.21.15.90:0/3347817564 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:44.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.847+0000 7f2884a00640 1 --2- 172.21.15.90:0/3347817564 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2880155030 0x7f2880157d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:44.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.847+0000 7f287f400640 1 -- 172.21.15.90:0/3347817564 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2880155030 msgr2=0x7f2880157d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:44.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.847+0000 7f287fe00640 1 --2- 172.21.15.90:0/3347817564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f288006b600 0x7f28801544c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:44.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.847+0000 7f287f400640 1 --2- 172.21.15.90:0/3347817564 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2880155030 0x7f2880157d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:44.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.847+0000 7f287f400640 1 -- 172.21.15.90:0/3347817564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f288006b600 msgr2=0x7f28801544c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:44.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.847+0000 7f287f400640 1 --2- 172.21.15.90:0/3347817564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f288006b600 0x7f28801544c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:44.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.847+0000 7f287f400640 1 -- 172.21.15.90:0/3347817564 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2880156500 con 0x7f28800cdac0 2024-03-24T01:24:44.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.848+0000 7f287f400640 1 --2- 172.21.15.90:0/3347817564 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28800cdac0 0x7f2880157800 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f286800d800 tx=0x7f286800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:44.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.848+0000 7f2877e00640 1 -- 172.21.15.90:0/3347817564 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2868018020 con 0x7f28800cdac0 2024-03-24T01:24:44.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.848+0000 7f2877e00640 1 -- 172.21.15.90:0/3347817564 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28680096a0 con 0x7f28800cdac0 2024-03-24T01:24:44.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.848+0000 7f2886a00640 1 -- 172.21.15.90:0/3347817564 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28801567e0 con 0x7f28800cdac0 2024-03-24T01:24:44.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.848+0000 7f2877e00640 1 -- 172.21.15.90:0/3347817564 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2868009a70 con 0x7f28800cdac0 2024-03-24T01:24:44.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.848+0000 7f2886a00640 1 -- 172.21.15.90:0/3347817564 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2880156da0 con 0x7f28800cdac0 2024-03-24T01:24:44.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.849+0000 7f2886a00640 1 -- 172.21.15.90:0/3347817564 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2880068060 con 0x7f28800cdac0 2024-03-24T01:24:44.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.850+0000 7f2877e00640 1 -- 172.21.15.90:0/3347817564 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f286801b020 con 0x7f28800cdac0 2024-03-24T01:24:44.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.851+0000 7f2877e00640 1 --2- 172.21.15.90:0/3347817564 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f285c06bbd0 0x7f285c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:44.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.851+0000 7f2877e00640 1 -- 172.21.15.90:0/3347817564 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f286808b380 con 0x7f28800cdac0 2024-03-24T01:24:44.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.851+0000 7f287fe00640 1 --2- 172.21.15.90:0/3347817564 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f285c06bbd0 0x7f285c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:44.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.852+0000 7f287fe00640 1 --2- 172.21.15.90:0/3347817564 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f285c06bbd0 0x7f285c06e090 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2870006fd0 tx=0x7f2870008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:44.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:44.855+0000 7f2877e00640 1 -- 172.21.15.90:0/3347817564 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2868022020 con 0x7f28800cdac0 2024-03-24T01:24:45.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.092+0000 7f2886a00640 1 -- 172.21.15.90:0/3347817564 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f28800d03a0 con 0x7f28800cdac0 2024-03-24T01:24:45.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.093+0000 7f2877e00640 1 -- 172.21.15.90:0/3347817564 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f2868059450 con 0x7f28800cdac0 2024-03-24T01:24:45.093 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:24:45.093 INFO:teuthology.orchestra.run.smithi090.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-24T01:24:45.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.095+0000 7f2886a00640 1 -- 172.21.15.90:0/3347817564 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f285c06bbd0 msgr2=0x7f285c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:45.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.095+0000 7f2886a00640 1 --2- 172.21.15.90:0/3347817564 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f285c06bbd0 0x7f285c06e090 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2870006fd0 tx=0x7f2870008040 comp rx=0 tx=0).stop 2024-03-24T01:24:45.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.095+0000 7f2886a00640 1 -- 172.21.15.90:0/3347817564 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28800cdac0 msgr2=0x7f2880157800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:45.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.095+0000 7f2886a00640 1 --2- 172.21.15.90:0/3347817564 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28800cdac0 0x7f2880157800 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f286800d800 tx=0x7f286800dcc0 comp rx=0 tx=0).stop 2024-03-24T01:24:45.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.095+0000 7f2886a00640 1 -- 172.21.15.90:0/3347817564 shutdown_connections 2024-03-24T01:24:45.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.095+0000 7f2886a00640 1 --2- 172.21.15.90:0/3347817564 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f285c06bbd0 0x7f285c06e090 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:45.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.095+0000 7f2886a00640 1 --2- 172.21.15.90:0/3347817564 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28800cdac0 0x7f2880157800 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:45.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.095+0000 7f2886a00640 1 --2- 172.21.15.90:0/3347817564 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2880155030 0x7f2880157d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:45.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.095+0000 7f2886a00640 1 --2- 172.21.15.90:0/3347817564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f288006b600 0x7f28801544c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:45.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.095+0000 7f2886a00640 1 -- 172.21.15.90:0/3347817564 >> 172.21.15.90:0/3347817564 conn(0x7f28800b1770 msgr2=0x7f28800b0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:45.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.096+0000 7f2886a00640 1 -- 172.21.15.90:0/3347817564 shutdown_connections 2024-03-24T01:24:45.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.096+0000 7f2886a00640 1 -- 172.21.15.90:0/3347817564 wait complete. 2024-03-24T01:24:45.113 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-24T01:24:45.113 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:24:45.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.310+0000 7fcb80000640 1 Processor -- start 2024-03-24T01:24:45.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.310+0000 7fcb80000640 1 -- start start 2024-03-24T01:24:45.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.311+0000 7fcb80000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb780cdac0 0x7fcb7806b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:45.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.311+0000 7fcb80000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcb7806b600 0x7fcb780ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:45.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.312+0000 7fcb80000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb7815bf70 con 0x7fcb78155030 2024-03-24T01:24:45.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.312+0000 7fcb80000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb7806ba70 con 0x7fcb7806b600 2024-03-24T01:24:45.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.312+0000 7fcb80000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7815b360 con 0x7fcb780cdac0 2024-03-24T01:24:45.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.312+0000 7fcb7cc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb780cdac0 0x7fcb7806b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:45.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.312+0000 7fcb7cc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb780cdac0 0x7fcb7806b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.90:50482/0 (socket says 172.21.15.90:50482) 2024-03-24T01:24:45.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.312+0000 7fcb7cc00640 1 -- 172.21.15.90:0/1911131145 learned_addr learned my addr 172.21.15.90:0/1911131145 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:45.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.312+0000 7fcb7e000640 1 --2- 172.21.15.90:0/1911131145 >> v2:172.21.15.120:3300/0 conn(0x7fcb7806b600 0x7fcb780ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:45.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.313+0000 7fcb7cc00640 1 -- 172.21.15.90:0/1911131145 >> v2:172.21.15.120:3300/0 conn(0x7fcb7806b600 msgr2=0x7fcb780ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:45.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.313+0000 7fcb7cc00640 1 --2- 172.21.15.90:0/1911131145 >> v2:172.21.15.120:3300/0 conn(0x7fcb7806b600 0x7fcb780ae530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:45.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.313+0000 7fcb7cc00640 1 -- 172.21.15.90:0/1911131145 >> v1:172.21.15.120:6789/0 conn(0x7fcb78155030 legacy=0x7fcb7815efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:45.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.313+0000 7fcb7cc00640 1 -- 172.21.15.90:0/1911131145 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb7815b640 con 0x7fcb780cdac0 2024-03-24T01:24:45.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.314+0000 7fcb7cc00640 1 --2- 172.21.15.90:0/1911131145 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb780cdac0 0x7fcb7806b0c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcb64009870 tx=0x7fcb6402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b896705d56ff1e0f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:45.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.314+0000 7fcb77e00640 1 -- 172.21.15.90:0/1911131145 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb64034070 con 0x7fcb780cdac0 2024-03-24T01:24:45.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.314+0000 7fcb77e00640 1 -- 172.21.15.90:0/1911131145 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb64030840 con 0x7fcb780cdac0 2024-03-24T01:24:45.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.314+0000 7fcb77e00640 1 -- 172.21.15.90:0/1911131145 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb64030c10 con 0x7fcb780cdac0 2024-03-24T01:24:45.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.315+0000 7fcb80000640 1 -- 172.21.15.90:0/1911131145 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb780cdac0 msgr2=0x7fcb7806b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:45.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.315+0000 7fcb80000640 1 --2- 172.21.15.90:0/1911131145 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb780cdac0 0x7fcb7806b0c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcb64009870 tx=0x7fcb6402b980 comp rx=0 tx=0).stop 2024-03-24T01:24:45.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.315+0000 7fcb80000640 1 -- 172.21.15.90:0/1911131145 shutdown_connections 2024-03-24T01:24:45.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.315+0000 7fcb80000640 1 --2- 172.21.15.90:0/1911131145 >> v2:172.21.15.120:3300/0 conn(0x7fcb7806b600 0x7fcb780ae530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:45.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.315+0000 7fcb80000640 1 --2- 172.21.15.90:0/1911131145 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb780cdac0 0x7fcb7806b0c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:45.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.315+0000 7fcb80000640 1 -- 172.21.15.90:0/1911131145 >> 172.21.15.90:0/1911131145 conn(0x7fcb780b1770 msgr2=0x7fcb780b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:45.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.315+0000 7fcb80000640 1 -- 172.21.15.90:0/1911131145 shutdown_connections 2024-03-24T01:24:45.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.315+0000 7fcb80000640 1 -- 172.21.15.90:0/1911131145 wait complete. 2024-03-24T01:24:45.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.317+0000 7fcb80000640 1 Processor -- start 2024-03-24T01:24:45.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.317+0000 7fcb80000640 1 -- start start 2024-03-24T01:24:45.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.319+0000 7fcb80000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb7806b600 0x7fcb781501c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:45.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.319+0000 7fcb80000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb780cdac0 0x7fcb78150be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:45.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.319+0000 7fcb80000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb78155030 0x7fcb78165b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:45.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.319+0000 7fcb80000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb78155410 con 0x7fcb78155030 2024-03-24T01:24:45.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.319+0000 7fcb80000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7815bc00 con 0x7fcb7806b600 2024-03-24T01:24:45.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.319+0000 7fcb80000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7815af70 con 0x7fcb780cdac0 2024-03-24T01:24:45.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.319+0000 7fcb7e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb78155030 0x7fcb78165b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:45.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.319+0000 7fcb7e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb78155030 0x7fcb78165b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60392/0 (socket says 172.21.15.90:60392) 2024-03-24T01:24:45.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.319+0000 7fcb7e000640 1 -- 172.21.15.90:0/834235828 learned_addr learned my addr 172.21.15.90:0/834235828 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:45.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.319+0000 7fcb7cc00640 1 --2- 172.21.15.90:0/834235828 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb780cdac0 0x7fcb78150be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:45.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.319+0000 7fcb7d600640 1 --2- 172.21.15.90:0/834235828 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb7806b600 0x7fcb781501c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:45.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.319+0000 7fcb7e000640 1 -- 172.21.15.90:0/834235828 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb780cdac0 msgr2=0x7fcb78150be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:45.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.319+0000 7fcb7e000640 1 --2- 172.21.15.90:0/834235828 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb780cdac0 0x7fcb78150be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:45.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.319+0000 7fcb7e000640 1 -- 172.21.15.90:0/834235828 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb7806b600 msgr2=0x7fcb781501c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:45.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.319+0000 7fcb7e000640 1 --2- 172.21.15.90:0/834235828 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb7806b600 0x7fcb781501c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:45.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.319+0000 7fcb7e000640 1 -- 172.21.15.90:0/834235828 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb781f1ba0 con 0x7fcb78155030 2024-03-24T01:24:45.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.320+0000 7fcb7e000640 1 --2- 172.21.15.90:0/834235828 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb78155030 0x7fcb78165b90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcb6800e960 tx=0x7fcb6800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:45.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.320+0000 7fcb75600640 1 -- 172.21.15.90:0/834235828 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb68017070 con 0x7fcb78155030 2024-03-24T01:24:45.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.320+0000 7fcb75600640 1 -- 172.21.15.90:0/834235828 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb6800c980 con 0x7fcb78155030 2024-03-24T01:24:45.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.320+0000 7fcb75600640 1 -- 172.21.15.90:0/834235828 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb6800cd70 con 0x7fcb78155030 2024-03-24T01:24:45.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.320+0000 7fcb80000640 1 -- 172.21.15.90:0/834235828 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb78151610 con 0x7fcb78155030 2024-03-24T01:24:45.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.320+0000 7fcb80000640 1 -- 172.21.15.90:0/834235828 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb78151bd0 con 0x7fcb78155030 2024-03-24T01:24:45.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.321+0000 7fcb80000640 1 -- 172.21.15.90:0/834235828 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb781f1d70 con 0x7fcb78155030 2024-03-24T01:24:45.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.322+0000 7fcb75600640 1 -- 172.21.15.90:0/834235828 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcb6801a070 con 0x7fcb78155030 2024-03-24T01:24:45.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.323+0000 7fcb75600640 1 --2- 172.21.15.90:0/834235828 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcb5806bbd0 0x7fcb5806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:45.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.323+0000 7fcb75600640 1 -- 172.21.15.90:0/834235828 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fcb6808adb0 con 0x7fcb78155030 2024-03-24T01:24:45.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.324+0000 7fcb7d600640 1 --2- 172.21.15.90:0/834235828 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcb5806bbd0 0x7fcb5806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:45.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.325+0000 7fcb7d600640 1 --2- 172.21.15.90:0/834235828 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcb5806bbd0 0x7fcb5806e090 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fcb700012e0 tx=0x7fcb70009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:45.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.327+0000 7fcb75600640 1 -- 172.21.15.90:0/834235828 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcb68059560 con 0x7fcb78155030 2024-03-24T01:24:45.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.509+0000 7fcb80000640 1 -- 172.21.15.90:0/834235828 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb78068060 con 0x7fcb78155030 2024-03-24T01:24:45.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.512+0000 7fcb75600640 1 -- 172.21.15.90:0/834235828 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18490 (secure 0 0 0) 0x7fcb68058e60 con 0x7fcb78155030 2024-03-24T01:24:45.512 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:24:45.513 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:24:17.008224+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:24:08.643966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T01:24:09.189904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T01:24:13.027323+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:24:45.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.514+0000 7fcb80000640 1 -- 172.21.15.90:0/834235828 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcb5806bbd0 msgr2=0x7fcb5806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:45.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.514+0000 7fcb80000640 1 --2- 172.21.15.90:0/834235828 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcb5806bbd0 0x7fcb5806e090 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fcb700012e0 tx=0x7fcb70009040 comp rx=0 tx=0).stop 2024-03-24T01:24:45.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.514+0000 7fcb80000640 1 -- 172.21.15.90:0/834235828 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb78155030 msgr2=0x7fcb78165b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:45.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.514+0000 7fcb80000640 1 --2- 172.21.15.90:0/834235828 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb78155030 0x7fcb78165b90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcb6800e960 tx=0x7fcb6800ee20 comp rx=0 tx=0).stop 2024-03-24T01:24:45.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.514+0000 7fcb80000640 1 -- 172.21.15.90:0/834235828 shutdown_connections 2024-03-24T01:24:45.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.514+0000 7fcb80000640 1 --2- 172.21.15.90:0/834235828 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcb5806bbd0 0x7fcb5806e090 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:45.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.514+0000 7fcb80000640 1 --2- 172.21.15.90:0/834235828 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb78155030 0x7fcb78165b90 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:45.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.514+0000 7fcb80000640 1 --2- 172.21.15.90:0/834235828 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb780cdac0 0x7fcb78150be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:45.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.514+0000 7fcb80000640 1 --2- 172.21.15.90:0/834235828 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb7806b600 0x7fcb781501c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:45.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.514+0000 7fcb80000640 1 -- 172.21.15.90:0/834235828 >> 172.21.15.90:0/834235828 conn(0x7fcb780b1770 msgr2=0x7fcb7815c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:45.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.515+0000 7fcb80000640 1 -- 172.21.15.90:0/834235828 shutdown_connections 2024-03-24T01:24:45.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.515+0000 7fcb80000640 1 -- 172.21.15.90:0/834235828 wait complete. 2024-03-24T01:24:45.532 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T01:24:08.643966+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T01:24:09.189904+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T01:24:13.027323+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T01:24:45.533 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:24:45.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.740+0000 7f5994a00640 1 Processor -- start 2024-03-24T01:24:45.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.740+0000 7f5994a00640 1 -- start start 2024-03-24T01:24:45.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.741+0000 7f5994a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59900cf150 0x7f59900ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:45.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.741+0000 7f5994a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f599006bca0 con 0x7f5990159620 2024-03-24T01:24:45.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.742+0000 7f5994a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59900b36f0 con 0x7f599006b7d0 2024-03-24T01:24:45.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.742+0000 7f5994a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59900b4470 con 0x7f59900cf150 2024-03-24T01:24:45.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.742+0000 7f598de00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f5990159620 0x7f599006b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51312/0 (socket says 172.21.15.90:51312) 2024-03-24T01:24:45.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.742+0000 7f598de00640 1 -- 172.21.15.90:0/1394595688 learned_addr learned my addr 172.21.15.90:0/1394595688 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:45.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.742+0000 7f598e800640 1 --2- 172.21.15.90:0/1394595688 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59900cf150 0x7f59900ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:45.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.742+0000 7f598d400640 1 -- 172.21.15.90:0/1394595688 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 683165225 0 0) 0x7f599006bca0 con 0x7f5990159620 2024-03-24T01:24:45.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.742+0000 7f598d400640 1 -- 172.21.15.90:0/1394595688 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5974003520 con 0x7f5990159620 2024-03-24T01:24:45.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.743+0000 7f598e800640 1 -- 172.21.15.90:0/1394595688 >> v1:172.21.15.120:6789/0 conn(0x7f599006b7d0 legacy=0x7f59900cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:45.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.743+0000 7f598d400640 1 -- 172.21.15.90:0/1394595688 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2297358405 0 0) 0x7f5974003520 con 0x7f5990159620 2024-03-24T01:24:45.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.743+0000 7f598e800640 1 -- 172.21.15.90:0/1394595688 >> v1:172.21.15.90:6789/0 conn(0x7f5990159620 legacy=0x7f599006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:45.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.743+0000 7f598e800640 1 -- 172.21.15.90:0/1394595688 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59900b3990 con 0x7f59900cf150 2024-03-24T01:24:45.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.744+0000 7f598e800640 1 --2- 172.21.15.90:0/1394595688 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59900cf150 0x7f59900ae530 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5988009dc0 tx=0x7f598802bbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8801f321cffcc33c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:45.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.744+0000 7f598d400640 1 -- 172.21.15.90:0/1394595688 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5980001670 con 0x7f59900cf150 2024-03-24T01:24:45.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.744+0000 7f598d400640 1 -- 172.21.15.90:0/1394595688 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f598802d6f0 con 0x7f59900cf150 2024-03-24T01:24:45.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.744+0000 7f598d400640 1 -- 172.21.15.90:0/1394595688 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598802dac0 con 0x7f59900cf150 2024-03-24T01:24:45.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.745+0000 7f5994a00640 1 -- 172.21.15.90:0/1394595688 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59900cf150 msgr2=0x7f59900ae530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:45.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.745+0000 7f5994a00640 1 --2- 172.21.15.90:0/1394595688 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59900cf150 0x7f59900ae530 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5988009dc0 tx=0x7f598802bbb0 comp rx=0 tx=0).stop 2024-03-24T01:24:45.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.745+0000 7f5994a00640 1 -- 172.21.15.90:0/1394595688 shutdown_connections 2024-03-24T01:24:45.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.745+0000 7f5994a00640 1 --2- 172.21.15.90:0/1394595688 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f59900cf150 0x7f59900ae530 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:45.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.745+0000 7f5994a00640 1 -- 172.21.15.90:0/1394595688 >> 172.21.15.90:0/1394595688 conn(0x7f59900b1770 msgr2=0x7f59900b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:45.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.745+0000 7f5994a00640 1 -- 172.21.15.90:0/1394595688 shutdown_connections 2024-03-24T01:24:45.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.745+0000 7f5994a00640 1 -- 172.21.15.90:0/1394595688 wait complete. 2024-03-24T01:24:45.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.747+0000 7f5994a00640 1 Processor -- start 2024-03-24T01:24:45.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.747+0000 7f5994a00640 1 -- start start 2024-03-24T01:24:45.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.748+0000 7f5994a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f599006b7d0 0x7f599006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:45.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.748+0000 7f5994a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59900cf150 0x7f59900c5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:45.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.748+0000 7f5994a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5990159620 0x7f59900c5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:45.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.748+0000 7f5994a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5990159a00 con 0x7f59900cf150 2024-03-24T01:24:45.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.748+0000 7f5994a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59900b3f50 con 0x7f5990159620 2024-03-24T01:24:45.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.748+0000 7f5994a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f599015bfe0 con 0x7f599006b7d0 2024-03-24T01:24:45.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.748+0000 7f5985400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59900cf150 0x7f59900c5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:45.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.748+0000 7f5985400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59900cf150 0x7f59900c5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60396/0 (socket says 172.21.15.90:60396) 2024-03-24T01:24:45.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.748+0000 7f5985400640 1 -- 172.21.15.90:0/1449575301 learned_addr learned my addr 172.21.15.90:0/1449575301 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:45.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.748+0000 7f598de00640 1 --2- 172.21.15.90:0/1449575301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f599006b7d0 0x7f599006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:45.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.748+0000 7f598e800640 1 --2- 172.21.15.90:0/1449575301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5990159620 0x7f59900c5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:45.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.748+0000 7f5985400640 1 -- 172.21.15.90:0/1449575301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f599006b7d0 msgr2=0x7f599006d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:45.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.748+0000 7f5985400640 1 --2- 172.21.15.90:0/1449575301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f599006b7d0 0x7f599006d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:45.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.748+0000 7f5985400640 1 -- 172.21.15.90:0/1449575301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5990159620 msgr2=0x7f59900c5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:45.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.748+0000 7f5985400640 1 --2- 172.21.15.90:0/1449575301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5990159620 0x7f59900c5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:45.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.748+0000 7f5985400640 1 -- 172.21.15.90:0/1449575301 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f599006e100 con 0x7f59900cf150 2024-03-24T01:24:45.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.748+0000 7f598e800640 1 --2- 172.21.15.90:0/1449575301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5990159620 0x7f59900c5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:45.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.749+0000 7f5985400640 1 --2- 172.21.15.90:0/1449575301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59900cf150 0x7f59900c5510 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f597800c4b0 tx=0x7f597800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:45.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.749+0000 7f5986a00640 1 -- 172.21.15.90:0/1449575301 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5978017070 con 0x7f59900cf150 2024-03-24T01:24:45.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.749+0000 7f5986a00640 1 -- 172.21.15.90:0/1449575301 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5978005480 con 0x7f59900cf150 2024-03-24T01:24:45.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.749+0000 7f5994a00640 1 -- 172.21.15.90:0/1449575301 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f599006e3e0 con 0x7f59900cf150 2024-03-24T01:24:45.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.749+0000 7f5986a00640 1 -- 172.21.15.90:0/1449575301 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59780037f0 con 0x7f59900cf150 2024-03-24T01:24:45.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.749+0000 7f5994a00640 1 -- 172.21.15.90:0/1449575301 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f599006e9a0 con 0x7f59900cf150 2024-03-24T01:24:45.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.750+0000 7f5994a00640 1 -- 172.21.15.90:0/1449575301 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f599004f400 con 0x7f59900cf150 2024-03-24T01:24:45.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.751+0000 7f5986a00640 1 -- 172.21.15.90:0/1449575301 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f597801a070 con 0x7f59900cf150 2024-03-24T01:24:45.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.752+0000 7f5986a00640 1 --2- 172.21.15.90:0/1449575301 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f596c06bbd0 0x7f596c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:45.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.752+0000 7f5986a00640 1 -- 172.21.15.90:0/1449575301 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f597808a080 con 0x7f59900cf150 2024-03-24T01:24:45.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.752+0000 7f598de00640 1 --2- 172.21.15.90:0/1449575301 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f596c06bbd0 0x7f596c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:45.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.753+0000 7f598de00640 1 --2- 172.21.15.90:0/1449575301 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f596c06bbd0 0x7f596c06e090 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f59800017e0 tx=0x7f5980001630 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:45.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.756+0000 7f5986a00640 1 -- 172.21.15.90:0/1449575301 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f59780581d0 con 0x7f59900cf150 2024-03-24T01:24:45.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.951+0000 7f5994a00640 1 -- 172.21.15.90:0/1449575301 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f599006c970 con 0x7f59900cf150 2024-03-24T01:24:45.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.952+0000 7f5986a00640 1 -- 172.21.15.90:0/1449575301 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f597805c270 con 0x7f59900cf150 2024-03-24T01:24:45.952 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:24:45.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.954+0000 7f5994a00640 1 -- 172.21.15.90:0/1449575301 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f596c06bbd0 msgr2=0x7f596c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:45.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.954+0000 7f5994a00640 1 --2- 172.21.15.90:0/1449575301 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f596c06bbd0 0x7f596c06e090 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f59800017e0 tx=0x7f5980001630 comp rx=0 tx=0).stop 2024-03-24T01:24:45.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.954+0000 7f5994a00640 1 -- 172.21.15.90:0/1449575301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59900cf150 msgr2=0x7f59900c5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:45.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.954+0000 7f5994a00640 1 --2- 172.21.15.90:0/1449575301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59900cf150 0x7f59900c5510 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f597800c4b0 tx=0x7f597800c970 comp rx=0 tx=0).stop 2024-03-24T01:24:45.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.954+0000 7f5994a00640 1 -- 172.21.15.90:0/1449575301 shutdown_connections 2024-03-24T01:24:45.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.954+0000 7f5994a00640 1 --2- 172.21.15.90:0/1449575301 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f596c06bbd0 0x7f596c06e090 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:45.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.954+0000 7f5994a00640 1 --2- 172.21.15.90:0/1449575301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59900cf150 0x7f59900c5510 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:45.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.954+0000 7f5994a00640 1 --2- 172.21.15.90:0/1449575301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5990159620 0x7f59900c5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:45.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.954+0000 7f5994a00640 1 --2- 172.21.15.90:0/1449575301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f599006b7d0 0x7f599006d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:45.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.954+0000 7f5994a00640 1 -- 172.21.15.90:0/1449575301 >> 172.21.15.90:0/1449575301 conn(0x7f59900b1770 msgr2=0x7f599015e250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:45.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.955+0000 7f5994a00640 1 -- 172.21.15.90:0/1449575301 shutdown_connections 2024-03-24T01:24:45.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:45.955+0000 7f5994a00640 1 -- 172.21.15.90:0/1449575301 wait complete. 2024-03-24T01:24:45.974 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T01:24:46.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.183+0000 7efe8a400640 1 Processor -- start 2024-03-24T01:24:46.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.183+0000 7efe8a400640 1 -- start start 2024-03-24T01:24:46.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.184+0000 7efe8a400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7efe8406ee90 0x7efe84162470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:46.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.184+0000 7efe8a400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe841629b0 con 0x7efe8406dfc0 2024-03-24T01:24:46.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.184+0000 7efe8a400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe84162c60 con 0x7efe840b38b0 2024-03-24T01:24:46.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.184+0000 7efe8a400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe84162f40 con 0x7efe8406ee90 2024-03-24T01:24:46.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.185+0000 7efe89000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7efe8406ee90 0x7efe84162470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:46.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.185+0000 7efe89000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7efe8406ee90 0x7efe84162470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60402/0 (socket says 172.21.15.90:60402) 2024-03-24T01:24:46.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.185+0000 7efe89000640 1 -- 172.21.15.90:0/2049867297 learned_addr learned my addr 172.21.15.90:0/2049867297 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:46.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.185+0000 7efe83e00640 1 --1- 172.21.15.90:0/2049867297 >> v1:172.21.15.90:6789/0 conn(0x7efe8406dfc0 0x7efe840b31a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51324/0 (socket says 172.21.15.90:51324) 2024-03-24T01:24:46.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.185+0000 7efe89000640 1 -- 172.21.15.90:0/2049867297 >> v1:172.21.15.120:6789/0 conn(0x7efe840b38b0 legacy=0x7efe84163d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:46.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.185+0000 7efe89000640 1 -- 172.21.15.90:0/2049867297 >> v1:172.21.15.90:6789/0 conn(0x7efe8406dfc0 legacy=0x7efe840b31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:46.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.185+0000 7efe89000640 1 -- 172.21.15.90:0/2049867297 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe84163220 con 0x7efe8406ee90 2024-03-24T01:24:46.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.186+0000 7efe89000640 1 --2- 172.21.15.90:0/2049867297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe8406ee90 0x7efe84162470 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7efe74009e50 tx=0x7efe7402bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0aab4082b7b88d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:46.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.187+0000 7efe83400640 1 -- 172.21.15.90:0/2049867297 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe7402cb90 con 0x7efe8406ee90 2024-03-24T01:24:46.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.187+0000 7efe83400640 1 -- 172.21.15.90:0/2049867297 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe7402cd30 con 0x7efe8406ee90 2024-03-24T01:24:46.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.187+0000 7efe83400640 1 -- 172.21.15.90:0/2049867297 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe74034610 con 0x7efe8406ee90 2024-03-24T01:24:46.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.187+0000 7efe8a400640 1 -- 172.21.15.90:0/2049867297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe8406ee90 msgr2=0x7efe84162470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:46.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.187+0000 7efe8a400640 1 --2- 172.21.15.90:0/2049867297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe8406ee90 0x7efe84162470 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7efe74009e50 tx=0x7efe7402bff0 comp rx=0 tx=0).stop 2024-03-24T01:24:46.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.188+0000 7efe8a400640 1 -- 172.21.15.90:0/2049867297 shutdown_connections 2024-03-24T01:24:46.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.188+0000 7efe8a400640 1 --2- 172.21.15.90:0/2049867297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe8406ee90 0x7efe84162470 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:46.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.188+0000 7efe8a400640 1 -- 172.21.15.90:0/2049867297 >> 172.21.15.90:0/2049867297 conn(0x7efe8406b0c0 msgr2=0x7efe8406b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:46.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.188+0000 7efe8a400640 1 -- 172.21.15.90:0/2049867297 shutdown_connections 2024-03-24T01:24:46.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.188+0000 7efe8a400640 1 -- 172.21.15.90:0/2049867297 wait complete. 2024-03-24T01:24:46.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.189+0000 7efe8a400640 1 Processor -- start 2024-03-24T01:24:46.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.190+0000 7efe8a400640 1 -- start start 2024-03-24T01:24:46.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.191+0000 7efe8a400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe8406dfc0 0x7efe841fb250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:46.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.191+0000 7efe8a400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe8406ee90 0x7efe8415aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:46.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.191+0000 7efe8a400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe840b38b0 0x7efe841543a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:46.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.191+0000 7efe8a400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe840b3e60 con 0x7efe8406ee90 2024-03-24T01:24:46.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.191+0000 7efe8a400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe841637e0 con 0x7efe8406dfc0 2024-03-24T01:24:46.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.191+0000 7efe8a400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe840b3fe0 con 0x7efe840b38b0 2024-03-24T01:24:46.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.191+0000 7efe83e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe8406ee90 0x7efe8415aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:46.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.191+0000 7efe83e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe8406ee90 0x7efe8415aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60406/0 (socket says 172.21.15.90:60406) 2024-03-24T01:24:46.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.191+0000 7efe83e00640 1 -- 172.21.15.90:0/2303534919 learned_addr learned my addr 172.21.15.90:0/2303534919 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:46.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.191+0000 7efe89000640 1 --2- 172.21.15.90:0/2303534919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe8406dfc0 0x7efe841fb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:46.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.191+0000 7efe83e00640 1 -- 172.21.15.90:0/2303534919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe840b38b0 msgr2=0x7efe841543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:46.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.191+0000 7efe83e00640 1 --2- 172.21.15.90:0/2303534919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe840b38b0 0x7efe841543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:46.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.191+0000 7efe83e00640 1 -- 172.21.15.90:0/2303534919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe8406dfc0 msgr2=0x7efe841fb250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:46.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.191+0000 7efe83e00640 1 --2- 172.21.15.90:0/2303534919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe8406dfc0 0x7efe841fb250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:46.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.191+0000 7efe83e00640 1 -- 172.21.15.90:0/2303534919 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe841fb790 con 0x7efe8406ee90 2024-03-24T01:24:46.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.192+0000 7efe83e00640 1 --2- 172.21.15.90:0/2303534919 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe8406ee90 0x7efe8415aa80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7efe6c00c500 tx=0x7efe6c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:46.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.192+0000 7efe80c00640 1 -- 172.21.15.90:0/2303534919 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c018070 con 0x7efe8406ee90 2024-03-24T01:24:46.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.192+0000 7efe80c00640 1 -- 172.21.15.90:0/2303534919 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe6c007500 con 0x7efe8406ee90 2024-03-24T01:24:46.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.192+0000 7efe80c00640 1 -- 172.21.15.90:0/2303534919 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c0049e0 con 0x7efe8406ee90 2024-03-24T01:24:46.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.192+0000 7efe8a400640 1 -- 172.21.15.90:0/2303534919 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe841fba10 con 0x7efe8406ee90 2024-03-24T01:24:46.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.192+0000 7efe8a400640 1 -- 172.21.15.90:0/2303534919 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe841fbfd0 con 0x7efe8406ee90 2024-03-24T01:24:46.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.193+0000 7efe8a400640 1 -- 172.21.15.90:0/2303534919 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe8404f400 con 0x7efe8406ee90 2024-03-24T01:24:46.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.194+0000 7efe80c00640 1 -- 172.21.15.90:0/2303534919 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7efe6c01b070 con 0x7efe8406ee90 2024-03-24T01:24:46.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.195+0000 7efe80c00640 1 --2- 172.21.15.90:0/2303534919 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7efe6006bbd0 0x7efe6006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:46.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.195+0000 7efe80c00640 1 -- 172.21.15.90:0/2303534919 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7efe6c08a140 con 0x7efe8406ee90 2024-03-24T01:24:46.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.196+0000 7efe89000640 1 --2- 172.21.15.90:0/2303534919 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7efe6006bbd0 0x7efe6006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:46.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.197+0000 7efe89000640 1 --2- 172.21.15.90:0/2303534919 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7efe6006bbd0 0x7efe6006e090 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7efe74035000 tx=0x7efe74030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:46.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.201+0000 7efe80c00640 1 -- 172.21.15.90:0/2303534919 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7efe841fb790 con 0x7efe8406ee90 2024-03-24T01:24:46.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.395+0000 7efe8a400640 1 -- 172.21.15.90:0/2303534919 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efe84161f80 con 0x7efe8406ee90 2024-03-24T01:24:46.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.396+0000 7efe80c00640 1 -- 172.21.15.90:0/2303534919 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7efe6c058980 con 0x7efe8406ee90 2024-03-24T01:24:46.397 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:24:46.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.399+0000 7efe8a400640 1 -- 172.21.15.90:0/2303534919 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7efe6006bbd0 msgr2=0x7efe6006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:46.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.399+0000 7efe8a400640 1 --2- 172.21.15.90:0/2303534919 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7efe6006bbd0 0x7efe6006e090 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7efe74035000 tx=0x7efe74030000 comp rx=0 tx=0).stop 2024-03-24T01:24:46.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.399+0000 7efe8a400640 1 -- 172.21.15.90:0/2303534919 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe8406ee90 msgr2=0x7efe8415aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:46.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.399+0000 7efe8a400640 1 --2- 172.21.15.90:0/2303534919 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe8406ee90 0x7efe8415aa80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7efe6c00c500 tx=0x7efe6c00c9c0 comp rx=0 tx=0).stop 2024-03-24T01:24:46.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.399+0000 7efe8a400640 1 -- 172.21.15.90:0/2303534919 shutdown_connections 2024-03-24T01:24:46.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.399+0000 7efe8a400640 1 --2- 172.21.15.90:0/2303534919 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7efe6006bbd0 0x7efe6006e090 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:46.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.399+0000 7efe8a400640 1 --2- 172.21.15.90:0/2303534919 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe8406ee90 0x7efe8415aa80 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:46.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.399+0000 7efe8a400640 1 --2- 172.21.15.90:0/2303534919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efe840b38b0 0x7efe841543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:46.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.399+0000 7efe8a400640 1 --2- 172.21.15.90:0/2303534919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efe8406dfc0 0x7efe841fb250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:46.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.399+0000 7efe8a400640 1 -- 172.21.15.90:0/2303534919 >> 172.21.15.90:0/2303534919 conn(0x7efe8406b0c0 msgr2=0x7efe84163b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:46.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.399+0000 7efe8a400640 1 -- 172.21.15.90:0/2303534919 shutdown_connections 2024-03-24T01:24:46.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.399+0000 7efe8a400640 1 -- 172.21.15.90:0/2303534919 wait complete. 2024-03-24T01:24:46.417 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T01:24:46.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.620+0000 7f88eaa00640 1 Processor -- start 2024-03-24T01:24:46.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.620+0000 7f88eaa00640 1 -- start start 2024-03-24T01:24:46.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.622+0000 7f88eaa00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f88e40d0b50 0x7f88e40d0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:46.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.622+0000 7f88eaa00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88e40d20e0 con 0x7f88e415dfd0 2024-03-24T01:24:46.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.622+0000 7f88eaa00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88e40d1470 con 0x7f88e40b2e40 2024-03-24T01:24:46.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.622+0000 7f88eaa00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88e40ced70 con 0x7f88e40d0b50 2024-03-24T01:24:46.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.622+0000 7f88e3400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f88e415dfd0 0x7f88e41622b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51330/0 (socket says 172.21.15.90:51330) 2024-03-24T01:24:46.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.622+0000 7f88e3400640 1 -- 172.21.15.90:0/3768110531 learned_addr learned my addr 172.21.15.90:0/3768110531 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:46.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.623+0000 7f88e2a00640 1 -- 172.21.15.90:0/3768110531 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3738501044 0 0) 0x7f88e40d20e0 con 0x7f88e415dfd0 2024-03-24T01:24:46.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.623+0000 7f88e3e00640 1 --2- 172.21.15.90:0/3768110531 >> v2:172.21.15.120:3300/0 conn(0x7f88e40d0b50 0x7f88e40d0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:46.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.623+0000 7f88e2a00640 1 -- 172.21.15.90:0/3768110531 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88c8003520 con 0x7f88e415dfd0 2024-03-24T01:24:46.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.623+0000 7f88e2a00640 1 -- 172.21.15.90:0/3768110531 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4228491738 0 0) 0x7f88c8003520 con 0x7f88e415dfd0 2024-03-24T01:24:46.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.623+0000 7f88e2a00640 1 -- 172.21.15.90:0/3768110531 >> v2:172.21.15.120:3300/0 conn(0x7f88e40d0b50 msgr2=0x7f88e40d0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:46.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.623+0000 7f88e2a00640 1 --2- 172.21.15.90:0/3768110531 >> v2:172.21.15.120:3300/0 conn(0x7f88e40d0b50 0x7f88e40d0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:46.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.623+0000 7f88e2a00640 1 -- 172.21.15.90:0/3768110531 >> v1:172.21.15.120:6789/0 conn(0x7f88e40b2e40 legacy=0x7f88e40d28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:46.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.623+0000 7f88e2a00640 1 -- 172.21.15.90:0/3768110531 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e40cf050 con 0x7f88e415dfd0 2024-03-24T01:24:46.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.623+0000 7f88e2a00640 1 -- 172.21.15.90:0/3768110531 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f88d0001690 con 0x7f88e415dfd0 2024-03-24T01:24:46.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.623+0000 7f88e2a00640 1 -- 172.21.15.90:0/3768110531 >> v1:172.21.15.90:6789/0 conn(0x7f88e415dfd0 legacy=0x7f88e41622b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:46.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.623+0000 7f88e2a00640 1 --2- 172.21.15.90:0/3768110531 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88c8003a00 0x7f88c8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:46.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.623+0000 7f88e2a00640 1 --2- 172.21.15.90:0/3768110531 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88c8004800 0x7f88c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:46.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.623+0000 7f88e2a00640 1 --2- 172.21.15.90:0/3768110531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88c80053c0 0x7f88c800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:46.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.623+0000 7f88e2a00640 1 -- 172.21.15.90:0/3768110531 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e40ced70 con 0x7f88c8004800 2024-03-24T01:24:46.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.623+0000 7f88e2a00640 1 -- 172.21.15.90:0/3768110531 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c800c2a0 con 0x7f88c80053c0 2024-03-24T01:24:46.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.623+0000 7f88e2a00640 1 -- 172.21.15.90:0/3768110531 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c800c4a0 con 0x7f88c8003a00 2024-03-24T01:24:46.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.623+0000 7f88e3400640 1 --2- 172.21.15.90:0/3768110531 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88c8004800 0x7f88c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:46.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.624+0000 7f88e3e00640 1 --2- 172.21.15.90:0/3768110531 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88c8003a00 0x7f88c8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:46.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.624+0000 7f88e8a00640 1 --2- 172.21.15.90:0/3768110531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88c80053c0 0x7f88c800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:46.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.624+0000 7f88e3400640 1 -- 172.21.15.90:0/3768110531 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88c8003a00 msgr2=0x7f88c8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:46.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.624+0000 7f88e3400640 1 --2- 172.21.15.90:0/3768110531 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88c8003a00 0x7f88c8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:46.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.624+0000 7f88e3400640 1 -- 172.21.15.90:0/3768110531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88c80053c0 msgr2=0x7f88c800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:46.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.624+0000 7f88e3400640 1 --2- 172.21.15.90:0/3768110531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88c80053c0 0x7f88c800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:46.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.624+0000 7f88e3400640 1 -- 172.21.15.90:0/3768110531 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c8003520 con 0x7f88c8004800 2024-03-24T01:24:46.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.624+0000 7f88e8a00640 1 -- 172.21.15.90:0/3768110531 reap_dead start 2024-03-24T01:24:46.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.624+0000 7f88e3e00640 1 --2- 172.21.15.90:0/3768110531 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88c8003a00 0x7f88c8003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:46.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.625+0000 7f88e3400640 1 --2- 172.21.15.90:0/3768110531 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88c8004800 0x7f88c8004c80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f88d000c160 tx=0x7f88c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c0cb98a3ea0f6f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:46.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.625+0000 7f88e2a00640 1 -- 172.21.15.90:0/3768110531 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88d0007c60 con 0x7f88c8004800 2024-03-24T01:24:46.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.625+0000 7f88e2a00640 1 -- 172.21.15.90:0/3768110531 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88d0030070 con 0x7f88c8004800 2024-03-24T01:24:46.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.625+0000 7f88e2a00640 1 -- 172.21.15.90:0/3768110531 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88d0005d10 con 0x7f88c8004800 2024-03-24T01:24:46.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.625+0000 7f88eaa00640 1 -- 172.21.15.90:0/3768110531 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88c8004800 msgr2=0x7f88c8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:46.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.625+0000 7f88eaa00640 1 --2- 172.21.15.90:0/3768110531 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88c8004800 0x7f88c8004c80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f88d000c160 tx=0x7f88c8005830 comp rx=0 tx=0).stop 2024-03-24T01:24:46.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.626+0000 7f88eaa00640 1 -- 172.21.15.90:0/3768110531 shutdown_connections 2024-03-24T01:24:46.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.626+0000 7f88eaa00640 1 --2- 172.21.15.90:0/3768110531 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88c8004800 0x7f88c8004c80 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:46.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.626+0000 7f88eaa00640 1 -- 172.21.15.90:0/3768110531 >> 172.21.15.90:0/3768110531 conn(0x7f88e4072c10 msgr2=0x7f88e4073020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:46.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.626+0000 7f88eaa00640 1 -- 172.21.15.90:0/3768110531 shutdown_connections 2024-03-24T01:24:46.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.626+0000 7f88eaa00640 1 -- 172.21.15.90:0/3768110531 wait complete. 2024-03-24T01:24:46.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.627+0000 7f88eaa00640 1 Processor -- start 2024-03-24T01:24:46.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.628+0000 7f88eaa00640 1 -- start start 2024-03-24T01:24:46.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.628+0000 7f88eaa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88c8004800 0x7f88e414bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:46.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.628+0000 7f88eaa00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88e40c93e0 0x7f88e40b8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:46.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.628+0000 7f88eaa00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88e40b8a80 0x7f88e41f4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:46.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.628+0000 7f88eaa00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e40b6d80 con 0x7f88c8004800 2024-03-24T01:24:46.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.628+0000 7f88eaa00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e40cf330 con 0x7f88e40c93e0 2024-03-24T01:24:46.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.628+0000 7f88eaa00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88e40b6f00 con 0x7f88e40b8a80 2024-03-24T01:24:46.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.629+0000 7f88e3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88c8004800 0x7f88e414bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:46.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.629+0000 7f88e3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88c8004800 0x7f88e414bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60434/0 (socket says 172.21.15.90:60434) 2024-03-24T01:24:46.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.629+0000 7f88e3e00640 1 -- 172.21.15.90:0/3287999749 learned_addr learned my addr 172.21.15.90:0/3287999749 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:46.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.629+0000 7f88e8a00640 1 --2- 172.21.15.90:0/3287999749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88e40b8a80 0x7f88e41f4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:46.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.629+0000 7f88e3e00640 1 -- 172.21.15.90:0/3287999749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88e40b8a80 msgr2=0x7f88e41f4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:46.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.629+0000 7f88e3e00640 1 --2- 172.21.15.90:0/3287999749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88e40b8a80 0x7f88e41f4320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:46.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.629+0000 7f88e3e00640 1 -- 172.21.15.90:0/3287999749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88e40c93e0 msgr2=0x7f88e40b8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:46.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.629+0000 7f88e3e00640 1 --2- 172.21.15.90:0/3287999749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88e40c93e0 0x7f88e40b8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:46.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.629+0000 7f88e3e00640 1 -- 172.21.15.90:0/3287999749 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e41f5050 con 0x7f88c8004800 2024-03-24T01:24:46.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.629+0000 7f88e3e00640 1 --2- 172.21.15.90:0/3287999749 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88c8004800 0x7f88e414bf70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f88d400e560 tx=0x7f88d400ea20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:46.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.629+0000 7f88dbe00640 1 -- 172.21.15.90:0/3287999749 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88d4017070 con 0x7f88c8004800 2024-03-24T01:24:46.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.630+0000 7f88dbe00640 1 -- 172.21.15.90:0/3287999749 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88d4013610 con 0x7f88c8004800 2024-03-24T01:24:46.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.630+0000 7f88dbe00640 1 -- 172.21.15.90:0/3287999749 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88d400f040 con 0x7f88c8004800 2024-03-24T01:24:46.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.630+0000 7f88eaa00640 1 -- 172.21.15.90:0/3287999749 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88e40c8900 con 0x7f88c8004800 2024-03-24T01:24:46.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.630+0000 7f88eaa00640 1 -- 172.21.15.90:0/3287999749 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88e40c8ec0 con 0x7f88c8004800 2024-03-24T01:24:46.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.631+0000 7f88eaa00640 1 -- 172.21.15.90:0/3287999749 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88e40682f0 con 0x7f88c8004800 2024-03-24T01:24:46.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.632+0000 7f88dbe00640 1 -- 172.21.15.90:0/3287999749 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f88d401a070 con 0x7f88c8004800 2024-03-24T01:24:46.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.632+0000 7f88dbe00640 1 --2- 172.21.15.90:0/3287999749 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f88c406bbd0 0x7f88c406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:46.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.632+0000 7f88dbe00640 1 -- 172.21.15.90:0/3287999749 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f88d408a8c0 con 0x7f88c8004800 2024-03-24T01:24:46.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.633+0000 7f88e3400640 1 --2- 172.21.15.90:0/3287999749 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f88c406bbd0 0x7f88c406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:46.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.634+0000 7f88e3400640 1 --2- 172.21.15.90:0/3287999749 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f88c406bbd0 0x7f88c406e090 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f88d0034000 tx=0x7f88d0007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:46.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.637+0000 7f88dbe00640 1 -- 172.21.15.90:0/3287999749 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f88e40682f0 con 0x7f88c8004800 2024-03-24T01:24:46.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.818+0000 7f88eaa00640 1 -- 172.21.15.90:0/3287999749 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f88e404f400 con 0x7f88c8004800 2024-03-24T01:24:46.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.819+0000 7f88dbe00640 1 -- 172.21.15.90:0/3287999749 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f88d4059070 con 0x7f88c8004800 2024-03-24T01:24:46.820 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-24T01:24:46.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.822+0000 7f88eaa00640 1 -- 172.21.15.90:0/3287999749 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f88c406bbd0 msgr2=0x7f88c406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:46.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.822+0000 7f88eaa00640 1 --2- 172.21.15.90:0/3287999749 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f88c406bbd0 0x7f88c406e090 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f88d0034000 tx=0x7f88d0007000 comp rx=0 tx=0).stop 2024-03-24T01:24:46.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.822+0000 7f88eaa00640 1 -- 172.21.15.90:0/3287999749 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88c8004800 msgr2=0x7f88e414bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:46.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.822+0000 7f88eaa00640 1 --2- 172.21.15.90:0/3287999749 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88c8004800 0x7f88e414bf70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f88d400e560 tx=0x7f88d400ea20 comp rx=0 tx=0).stop 2024-03-24T01:24:46.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.822+0000 7f88eaa00640 1 -- 172.21.15.90:0/3287999749 shutdown_connections 2024-03-24T01:24:46.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.822+0000 7f88eaa00640 1 --2- 172.21.15.90:0/3287999749 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f88c406bbd0 0x7f88c406e090 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:46.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.822+0000 7f88eaa00640 1 --2- 172.21.15.90:0/3287999749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f88e40b8a80 0x7f88e41f4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:46.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.822+0000 7f88eaa00640 1 --2- 172.21.15.90:0/3287999749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f88e40c93e0 0x7f88e40b8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:46.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.822+0000 7f88eaa00640 1 --2- 172.21.15.90:0/3287999749 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88c8004800 0x7f88e414bf70 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:46.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.822+0000 7f88eaa00640 1 -- 172.21.15.90:0/3287999749 >> 172.21.15.90:0/3287999749 conn(0x7f88e4072c10 msgr2=0x7f88e40d26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:46.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.822+0000 7f88eaa00640 1 -- 172.21.15.90:0/3287999749 shutdown_connections 2024-03-24T01:24:46.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:46.822+0000 7f88eaa00640 1 -- 172.21.15.90:0/3287999749 wait complete. 2024-03-24T01:24:46.840 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T01:24:47.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.049+0000 7f3602800640 1 Processor -- start 2024-03-24T01:24:47.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.049+0000 7f3602800640 1 -- start start 2024-03-24T01:24:47.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.050+0000 7f3602800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f35fc1550a0 0x7f35fc15bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:47.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.050+0000 7f3602800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f35fc06f5c0 0x7f35fc0b3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:47.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.050+0000 7f3602800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35fc0b3fe0 0x7f35fc0ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:47.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.050+0000 7f3602800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35fc0b4430 con 0x7f35fc06f5c0 2024-03-24T01:24:47.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.050+0000 7f3602800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35fc06fa70 con 0x7f35fc1550a0 2024-03-24T01:24:47.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.050+0000 7f3602800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35fc155480 con 0x7f35fc0b3fe0 2024-03-24T01:24:47.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.050+0000 7f35f2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f35fc06f5c0 0x7f35fc0b3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:47.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.051+0000 7f35f2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f35fc06f5c0 0x7f35fc0b3aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60446/0 (socket says 172.21.15.90:60446) 2024-03-24T01:24:47.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.051+0000 7f35f2a00640 1 -- 172.21.15.90:0/1086902713 learned_addr learned my addr 172.21.15.90:0/1086902713 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:47.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.051+0000 7f35f2a00640 1 -- 172.21.15.90:0/1086902713 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35fc0b3fe0 msgr2=0x7f35fc0ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:47.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.051+0000 7f35f2a00640 1 --2- 172.21.15.90:0/1086902713 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35fc0b3fe0 0x7f35fc0ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:47.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.051+0000 7f35f2a00640 1 -- 172.21.15.90:0/1086902713 >> v2:172.21.15.120:3300/0 conn(0x7f35fc1550a0 msgr2=0x7f35fc15bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:47.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.051+0000 7f35fb400640 1 --2- 172.21.15.90:0/1086902713 >> v2:172.21.15.120:3300/0 conn(0x7f35fc1550a0 0x7f35fc15bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:47.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.051+0000 7f35f2a00640 1 --2- 172.21.15.90:0/1086902713 >> v2:172.21.15.120:3300/0 conn(0x7f35fc1550a0 0x7f35fc15bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:47.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.051+0000 7f35f2a00640 1 -- 172.21.15.90:0/1086902713 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35fc06fcc0 con 0x7f35fc06f5c0 2024-03-24T01:24:47.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.051+0000 7f35fb400640 1 --2- 172.21.15.90:0/1086902713 >> v2:172.21.15.120:3300/0 conn(0x7f35fc1550a0 0x7f35fc15bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:47.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.052+0000 7f35f2a00640 1 --2- 172.21.15.90:0/1086902713 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35fc06f5c0 0x7f35fc0b3aa0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f35fc072d70 tx=0x7f35e402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b841cf5ef03952b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:47.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.052+0000 7f35faa00640 1 -- 172.21.15.90:0/1086902713 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e402e070 con 0x7f35fc06f5c0 2024-03-24T01:24:47.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.052+0000 7f35faa00640 1 -- 172.21.15.90:0/1086902713 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e4037070 con 0x7f35fc06f5c0 2024-03-24T01:24:47.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.052+0000 7f35faa00640 1 -- 172.21.15.90:0/1086902713 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e4031a00 con 0x7f35fc06f5c0 2024-03-24T01:24:47.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.053+0000 7f3602800640 1 -- 172.21.15.90:0/1086902713 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35fc06f5c0 msgr2=0x7f35fc0b3aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:47.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.053+0000 7f3602800640 1 --2- 172.21.15.90:0/1086902713 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35fc06f5c0 0x7f35fc0b3aa0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f35fc072d70 tx=0x7f35e402b880 comp rx=0 tx=0).stop 2024-03-24T01:24:47.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.053+0000 7f3602800640 1 -- 172.21.15.90:0/1086902713 shutdown_connections 2024-03-24T01:24:47.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.053+0000 7f3602800640 1 --2- 172.21.15.90:0/1086902713 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35fc0b3fe0 0x7f35fc0ae530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:47.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.053+0000 7f3602800640 1 --2- 172.21.15.90:0/1086902713 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35fc06f5c0 0x7f35fc0b3aa0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:47.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.053+0000 7f3602800640 1 --2- 172.21.15.90:0/1086902713 >> v2:172.21.15.120:3300/0 conn(0x7f35fc1550a0 0x7f35fc15bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:47.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.053+0000 7f3602800640 1 -- 172.21.15.90:0/1086902713 >> 172.21.15.90:0/1086902713 conn(0x7f35fc0cf7e0 msgr2=0x7f35fc0cfbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:47.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.053+0000 7f3602800640 1 -- 172.21.15.90:0/1086902713 shutdown_connections 2024-03-24T01:24:47.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.053+0000 7f3602800640 1 -- 172.21.15.90:0/1086902713 wait complete. 2024-03-24T01:24:47.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.055+0000 7f3602800640 1 Processor -- start 2024-03-24T01:24:47.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.055+0000 7f3602800640 1 -- start start 2024-03-24T01:24:47.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.056+0000 7f3602800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35fc06f5c0 0x7f35fc15b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:47.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.056+0000 7f3602800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35fc0b3fe0 0x7f35fc14efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:47.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.056+0000 7f3602800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35fc1550a0 0x7f35fc1e87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:47.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.056+0000 7f3602800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35fc15e9c0 con 0x7f35fc0b3fe0 2024-03-24T01:24:47.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.056+0000 7f3602800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35fc15e6e0 con 0x7f35fc1550a0 2024-03-24T01:24:47.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.056+0000 7f3602800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35fc15eca0 con 0x7f35fc06f5c0 2024-03-24T01:24:47.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.056+0000 7f35f2a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35fc0b3fe0 0x7f35fc14efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:47.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.056+0000 7f35f2a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35fc0b3fe0 0x7f35fc14efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60448/0 (socket says 172.21.15.90:60448) 2024-03-24T01:24:47.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.056+0000 7f35f2a00640 1 -- 172.21.15.90:0/2753203976 learned_addr learned my addr 172.21.15.90:0/2753203976 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:47.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.057+0000 7f35fb400640 1 --2- 172.21.15.90:0/2753203976 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35fc06f5c0 0x7f35fc15b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:47.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.057+0000 7f35fbe00640 1 --2- 172.21.15.90:0/2753203976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35fc1550a0 0x7f35fc1e87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:47.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.057+0000 7f35f2a00640 1 -- 172.21.15.90:0/2753203976 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35fc06f5c0 msgr2=0x7f35fc15b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:47.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.057+0000 7f35f2a00640 1 --2- 172.21.15.90:0/2753203976 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35fc06f5c0 0x7f35fc15b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:47.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.057+0000 7f35f2a00640 1 -- 172.21.15.90:0/2753203976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35fc1550a0 msgr2=0x7f35fc1e87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:47.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.057+0000 7f35f2a00640 1 --2- 172.21.15.90:0/2753203976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35fc1550a0 0x7f35fc1e87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:47.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.057+0000 7f35f2a00640 1 -- 172.21.15.90:0/2753203976 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35fc14f4f0 con 0x7f35fc0b3fe0 2024-03-24T01:24:47.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.057+0000 7f35fbe00640 1 --2- 172.21.15.90:0/2753203976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35fc1550a0 0x7f35fc1e87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:47.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.057+0000 7f35f2a00640 1 --2- 172.21.15.90:0/2753203976 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35fc0b3fe0 0x7f35fc14efb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f35e402b8c0 tx=0x7f35e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:47.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.057+0000 7f35f3e00640 1 -- 172.21.15.90:0/2753203976 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e4032880 con 0x7f35fc0b3fe0 2024-03-24T01:24:47.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.057+0000 7f35f3e00640 1 -- 172.21.15.90:0/2753203976 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e4032a20 con 0x7f35fc0b3fe0 2024-03-24T01:24:47.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.057+0000 7f3602800640 1 -- 172.21.15.90:0/2753203976 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35fc15bf40 con 0x7f35fc0b3fe0 2024-03-24T01:24:47.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.057+0000 7f35f3e00640 1 -- 172.21.15.90:0/2753203976 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e4032e10 con 0x7f35fc0b3fe0 2024-03-24T01:24:47.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.057+0000 7f3602800640 1 -- 172.21.15.90:0/2753203976 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35fc0cb920 con 0x7f35fc0b3fe0 2024-03-24T01:24:47.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.058+0000 7f3602800640 1 -- 172.21.15.90:0/2753203976 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35fc04f400 con 0x7f35fc0b3fe0 2024-03-24T01:24:47.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.059+0000 7f35f3e00640 1 -- 172.21.15.90:0/2753203976 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f35e404c060 con 0x7f35fc0b3fe0 2024-03-24T01:24:47.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.060+0000 7f35f3e00640 1 --2- 172.21.15.90:0/2753203976 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f35d806bbd0 0x7f35d806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:47.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.060+0000 7f35f3e00640 1 -- 172.21.15.90:0/2753203976 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f35e40ac8b0 con 0x7f35fc0b3fe0 2024-03-24T01:24:47.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.061+0000 7f35fb400640 1 --2- 172.21.15.90:0/2753203976 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f35d806bbd0 0x7f35d806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:47.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.062+0000 7f35fb400640 1 --2- 172.21.15.90:0/2753203976 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f35d806bbd0 0x7f35d806e090 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f35ec00a990 tx=0x7f35ec008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:47.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.065+0000 7f35f3e00640 1 -- 172.21.15.90:0/2753203976 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f35e4040070 con 0x7f35fc0b3fe0 2024-03-24T01:24:47.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.263+0000 7f3602800640 1 -- 172.21.15.90:0/2753203976 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f35fc0d04b0 con 0x7f35fc0b3fe0 2024-03-24T01:24:47.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.264+0000 7f35f3e00640 1 -- 172.21.15.90:0/2753203976 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7f35e40aae20 con 0x7f35fc0b3fe0 2024-03-24T01:24:47.264 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:24:47.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.267+0000 7f3602800640 1 -- 172.21.15.90:0/2753203976 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f35d806bbd0 msgr2=0x7f35d806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:47.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.267+0000 7f3602800640 1 --2- 172.21.15.90:0/2753203976 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f35d806bbd0 0x7f35d806e090 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f35ec00a990 tx=0x7f35ec008040 comp rx=0 tx=0).stop 2024-03-24T01:24:47.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.267+0000 7f3602800640 1 -- 172.21.15.90:0/2753203976 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35fc0b3fe0 msgr2=0x7f35fc14efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:47.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.267+0000 7f3602800640 1 --2- 172.21.15.90:0/2753203976 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35fc0b3fe0 0x7f35fc14efb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f35e402b8c0 tx=0x7f35e40046c0 comp rx=0 tx=0).stop 2024-03-24T01:24:47.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.267+0000 7f3602800640 1 -- 172.21.15.90:0/2753203976 shutdown_connections 2024-03-24T01:24:47.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.267+0000 7f3602800640 1 --2- 172.21.15.90:0/2753203976 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f35d806bbd0 0x7f35d806e090 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:47.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.267+0000 7f3602800640 1 --2- 172.21.15.90:0/2753203976 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35fc0b3fe0 0x7f35fc14efb0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:47.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.267+0000 7f3602800640 1 --2- 172.21.15.90:0/2753203976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35fc1550a0 0x7f35fc1e87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:47.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.267+0000 7f3602800640 1 --2- 172.21.15.90:0/2753203976 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35fc06f5c0 0x7f35fc15b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:47.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.267+0000 7f3602800640 1 -- 172.21.15.90:0/2753203976 >> 172.21.15.90:0/2753203976 conn(0x7f35fc0cf7e0 msgr2=0x7f35fc157890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:47.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.267+0000 7f3602800640 1 -- 172.21.15.90:0/2753203976 shutdown_connections 2024-03-24T01:24:47.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.267+0000 7f3602800640 1 -- 172.21.15.90:0/2753203976 wait complete. 2024-03-24T01:24:47.285 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:24:47.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.487+0000 7f8069a00640 1 Processor -- start 2024-03-24T01:24:47.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.487+0000 7f8069a00640 1 -- start start 2024-03-24T01:24:47.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.489+0000 7f8069a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80640cfb80 0x7f80640acda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:47.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.489+0000 7f8069a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f80640716d0 0x7f80640b2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:47.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.489+0000 7f8069a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80640ad380 con 0x7f806415da90 2024-03-24T01:24:47.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.489+0000 7f8069a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80640adf90 con 0x7f80640716d0 2024-03-24T01:24:47.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.489+0000 7f8069a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80640ad660 con 0x7f80640cfb80 2024-03-24T01:24:47.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.489+0000 7f8063e00640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f806415da90 0x7f80640cf470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.90:48836/0 (socket says 172.21.15.90:48836) 2024-03-24T01:24:47.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.489+0000 7f8063e00640 1 -- 172.21.15.90:0/2710685475 learned_addr learned my addr 172.21.15.90:0/2710685475 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:47.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.489+0000 7f8069000640 1 --2- 172.21.15.90:0/2710685475 >> v2:172.21.15.120:3300/0 conn(0x7f80640716d0 0x7f80640b2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:47.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.489+0000 7f805b400640 1 --2- 172.21.15.90:0/2710685475 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80640cfb80 0x7f80640acda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:47.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.489+0000 7f8063400640 1 -- 172.21.15.90:0/2710685475 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1455132941 0 0) 0x7f80640ad380 con 0x7f806415da90 2024-03-24T01:24:47.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.490+0000 7f8063400640 1 -- 172.21.15.90:0/2710685475 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8048003520 con 0x7f806415da90 2024-03-24T01:24:47.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.490+0000 7f805b400640 1 -- 172.21.15.90:0/2710685475 >> v2:172.21.15.120:3300/0 conn(0x7f80640716d0 msgr2=0x7f80640b2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:47.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.490+0000 7f805b400640 1 --2- 172.21.15.90:0/2710685475 >> v2:172.21.15.120:3300/0 conn(0x7f80640716d0 0x7f80640b2680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:47.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.490+0000 7f805b400640 1 -- 172.21.15.90:0/2710685475 >> v1:172.21.15.120:6789/0 conn(0x7f806415da90 legacy=0x7f80640cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:47.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.490+0000 7f805b400640 1 -- 172.21.15.90:0/2710685475 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80640ad940 con 0x7f80640cfb80 2024-03-24T01:24:47.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.491+0000 7f805b400640 1 --2- 172.21.15.90:0/2710685475 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80640cfb80 0x7f80640acda0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f804c0097a0 tx=0x7f804c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b512afd9e76629f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:47.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.491+0000 7f8063400640 1 -- 172.21.15.90:0/2710685475 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804c035020 con 0x7f80640cfb80 2024-03-24T01:24:47.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.491+0000 7f8063400640 1 -- 172.21.15.90:0/2710685475 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f804c031810 con 0x7f80640cfb80 2024-03-24T01:24:47.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.491+0000 7f8063400640 1 -- 172.21.15.90:0/2710685475 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804c031bc0 con 0x7f80640cfb80 2024-03-24T01:24:47.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.492+0000 7f8069a00640 1 -- 172.21.15.90:0/2710685475 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80640cfb80 msgr2=0x7f80640acda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:47.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.492+0000 7f8069a00640 1 --2- 172.21.15.90:0/2710685475 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80640cfb80 0x7f80640acda0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f804c0097a0 tx=0x7f804c02b8b0 comp rx=0 tx=0).stop 2024-03-24T01:24:47.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.492+0000 7f8069a00640 1 -- 172.21.15.90:0/2710685475 shutdown_connections 2024-03-24T01:24:47.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.492+0000 7f8069a00640 1 --2- 172.21.15.90:0/2710685475 >> v2:172.21.15.120:3300/0 conn(0x7f80640716d0 0x7f80640b2680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:47.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.492+0000 7f8069a00640 1 --2- 172.21.15.90:0/2710685475 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80640cfb80 0x7f80640acda0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:47.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.492+0000 7f8069a00640 1 -- 172.21.15.90:0/2710685475 >> 172.21.15.90:0/2710685475 conn(0x7f806406b0c0 msgr2=0x7f806406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:47.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.492+0000 7f8069a00640 1 -- 172.21.15.90:0/2710685475 shutdown_connections 2024-03-24T01:24:47.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.492+0000 7f8069a00640 1 -- 172.21.15.90:0/2710685475 wait complete. 2024-03-24T01:24:47.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.494+0000 7f8069a00640 1 Processor -- start 2024-03-24T01:24:47.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.494+0000 7f8069a00640 1 -- start start 2024-03-24T01:24:47.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.495+0000 7f8069a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80640716d0 0x7f806415f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:47.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.495+0000 7f8069a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80640cfb80 0x7f8064152810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:47.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.495+0000 7f8069a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f806415da90 0x7f80640b1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:47.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.495+0000 7f8069a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806415de70 con 0x7f806415da90 2024-03-24T01:24:47.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.495+0000 7f8069a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80640aee40 con 0x7f80640716d0 2024-03-24T01:24:47.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.495+0000 7f8069a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8064152d50 con 0x7f80640cfb80 2024-03-24T01:24:47.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.495+0000 7f8069000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f806415da90 0x7f80640b1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:47.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.495+0000 7f8069000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f806415da90 0x7f80640b1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60450/0 (socket says 172.21.15.90:60450) 2024-03-24T01:24:47.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.495+0000 7f8069000640 1 -- 172.21.15.90:0/319124851 learned_addr learned my addr 172.21.15.90:0/319124851 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:47.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.495+0000 7f8063e00640 1 --2- 172.21.15.90:0/319124851 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80640716d0 0x7f806415f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:47.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.496+0000 7f805b400640 1 --2- 172.21.15.90:0/319124851 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80640cfb80 0x7f8064152810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:47.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.496+0000 7f8069000640 1 -- 172.21.15.90:0/319124851 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80640cfb80 msgr2=0x7f8064152810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:47.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.496+0000 7f8069000640 1 --2- 172.21.15.90:0/319124851 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80640cfb80 0x7f8064152810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:47.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.496+0000 7f8069000640 1 -- 172.21.15.90:0/319124851 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80640716d0 msgr2=0x7f806415f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:47.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.496+0000 7f8069000640 1 --2- 172.21.15.90:0/319124851 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80640716d0 0x7f806415f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:47.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.496+0000 7f8069000640 1 -- 172.21.15.90:0/319124851 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8064152f80 con 0x7f806415da90 2024-03-24T01:24:47.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.496+0000 7f8069000640 1 --2- 172.21.15.90:0/319124851 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f806415da90 0x7f80640b1760 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f805c00efc0 tx=0x7f805c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:47.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.496+0000 7f8060c00640 1 -- 172.21.15.90:0/319124851 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805c018020 con 0x7f806415da90 2024-03-24T01:24:47.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.496+0000 7f8060c00640 1 -- 172.21.15.90:0/319124851 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f805c014480 con 0x7f806415da90 2024-03-24T01:24:47.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.496+0000 7f8060c00640 1 -- 172.21.15.90:0/319124851 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805c014870 con 0x7f806415da90 2024-03-24T01:24:47.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.497+0000 7f8069a00640 1 -- 172.21.15.90:0/319124851 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f806415c160 con 0x7f806415da90 2024-03-24T01:24:47.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.497+0000 7f8069a00640 1 -- 172.21.15.90:0/319124851 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f806415c720 con 0x7f806415da90 2024-03-24T01:24:47.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.498+0000 7f8069a00640 1 -- 172.21.15.90:0/319124851 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f806415dff0 con 0x7f806415da90 2024-03-24T01:24:47.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.499+0000 7f8060c00640 1 -- 172.21.15.90:0/319124851 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f805c005d10 con 0x7f806415da90 2024-03-24T01:24:47.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.499+0000 7f8060c00640 1 --2- 172.21.15.90:0/319124851 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f804006bbd0 0x7f804006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:47.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.499+0000 7f8060c00640 1 -- 172.21.15.90:0/319124851 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f805c08b750 con 0x7f806415da90 2024-03-24T01:24:47.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.500+0000 7f8063e00640 1 --2- 172.21.15.90:0/319124851 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f804006bbd0 0x7f804006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:47.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.501+0000 7f8063e00640 1 --2- 172.21.15.90:0/319124851 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f804006bbd0 0x7f804006e090 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f80540011e0 tx=0x7f8054007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:47.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.504+0000 7f8060c00640 1 -- 172.21.15.90:0/319124851 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f805c022070 con 0x7f806415da90 2024-03-24T01:24:47.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.743+0000 7f8069a00640 1 -- 172.21.15.90:0/319124851 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8064161470 con 0x7f806415da90 2024-03-24T01:24:47.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.744+0000 7f8060c00640 1 -- 172.21.15.90:0/319124851 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4967 (secure 0 0 0) 0x7f805c010040 con 0x7f806415da90 2024-03-24T01:24:47.745 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:24:47.745 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/1898972903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":1898972903},{"type":"v1","addr":"172.21.15.120:6837","nonce":1898972903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6834/775759501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":775759501},{"type":"v1","addr":"172.21.15.90:6834","nonce":775759501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/43630575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":43630575},{"type":"v1","addr":"172.21.15.90:6835","nonce":43630575}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T01:24:10.173532+0000","modified":"2024-03-24T01:24:36.608477+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6836/963928022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":963928022},{"type":"v1","addr":"172.21.15.120:6836","nonce":963928022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4257,"qdb_cluster":[4257]},"id":1}]} 2024-03-24T01:24:47.745 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 8 2024-03-24T01:24:47.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.747+0000 7f8069a00640 1 -- 172.21.15.90:0/319124851 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f804006bbd0 msgr2=0x7f804006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:47.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.747+0000 7f8069a00640 1 --2- 172.21.15.90:0/319124851 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f804006bbd0 0x7f804006e090 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f80540011e0 tx=0x7f8054007d30 comp rx=0 tx=0).stop 2024-03-24T01:24:47.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.747+0000 7f8069a00640 1 -- 172.21.15.90:0/319124851 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f806415da90 msgr2=0x7f80640b1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:47.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.747+0000 7f8069a00640 1 --2- 172.21.15.90:0/319124851 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f806415da90 0x7f80640b1760 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f805c00efc0 tx=0x7f805c00c490 comp rx=0 tx=0).stop 2024-03-24T01:24:47.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.747+0000 7f8069a00640 1 -- 172.21.15.90:0/319124851 shutdown_connections 2024-03-24T01:24:47.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.747+0000 7f8069a00640 1 --2- 172.21.15.90:0/319124851 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f804006bbd0 0x7f804006e090 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:47.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.747+0000 7f8069a00640 1 --2- 172.21.15.90:0/319124851 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f806415da90 0x7f80640b1760 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:47.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.747+0000 7f8069a00640 1 --2- 172.21.15.90:0/319124851 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80640cfb80 0x7f8064152810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:47.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.747+0000 7f8069a00640 1 --2- 172.21.15.90:0/319124851 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80640716d0 0x7f806415f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:47.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.747+0000 7f8069a00640 1 -- 172.21.15.90:0/319124851 >> 172.21.15.90:0/319124851 conn(0x7f806406b0c0 msgr2=0x7f80640b2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:47.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.747+0000 7f8069a00640 1 -- 172.21.15.90:0/319124851 shutdown_connections 2024-03-24T01:24:47.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.747+0000 7f8069a00640 1 -- 172.21.15.90:0/319124851 wait complete. 2024-03-24T01:24:47.766 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:24:47.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.977+0000 7f91c0000640 1 Processor -- start 2024-03-24T01:24:47.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.977+0000 7f91c0000640 1 -- start start 2024-03-24T01:24:47.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.979+0000 7f91c0000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f91b80cdac0 0x7f91b806b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:47.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.979+0000 7f91c0000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b815bf70 con 0x7f91b8155030 2024-03-24T01:24:47.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.979+0000 7f91c0000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b806ba00 con 0x7f91b806b600 2024-03-24T01:24:47.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.979+0000 7f91c0000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91b815b360 con 0x7f91b80cdac0 2024-03-24T01:24:47.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.979+0000 7f91bd600640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f91b8155030 0x7f91b815efc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51346/0 (socket says 172.21.15.90:51346) 2024-03-24T01:24:47.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.979+0000 7f91bd600640 1 -- 172.21.15.90:0/211011944 learned_addr learned my addr 172.21.15.90:0/211011944 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:47.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.979+0000 7f91bcc00640 1 --2- 172.21.15.90:0/211011944 >> v2:172.21.15.90:3300/0 conn(0x7f91b80cdac0 0x7f91b806b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:47.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.979+0000 7f91bcc00640 1 -- 172.21.15.90:0/211011944 >> v1:172.21.15.120:6789/0 conn(0x7f91b806b600 legacy=0x7f91b80ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:47.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.979+0000 7f91bcc00640 1 -- 172.21.15.90:0/211011944 >> v1:172.21.15.90:6789/0 conn(0x7f91b8155030 legacy=0x7f91b815efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:47.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.979+0000 7f91bcc00640 1 -- 172.21.15.90:0/211011944 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b815b640 con 0x7f91b80cdac0 2024-03-24T01:24:47.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.980+0000 7f91bcc00640 1 --2- 172.21.15.90:0/211011944 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91b80cdac0 0x7f91b806b0c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f91a80097a0 tx=0x7f91a802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a33e81422c3eff5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:47.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.981+0000 7f91b7e00640 1 -- 172.21.15.90:0/211011944 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a8035020 con 0x7f91b80cdac0 2024-03-24T01:24:47.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.981+0000 7f91b7e00640 1 -- 172.21.15.90:0/211011944 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91a8031830 con 0x7f91b80cdac0 2024-03-24T01:24:47.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.981+0000 7f91b7e00640 1 -- 172.21.15.90:0/211011944 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a8031c00 con 0x7f91b80cdac0 2024-03-24T01:24:47.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.981+0000 7f91c0000640 1 -- 172.21.15.90:0/211011944 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91b80cdac0 msgr2=0x7f91b806b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:47.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.981+0000 7f91c0000640 1 --2- 172.21.15.90:0/211011944 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91b80cdac0 0x7f91b806b0c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f91a80097a0 tx=0x7f91a802b8d0 comp rx=0 tx=0).stop 2024-03-24T01:24:47.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.981+0000 7f91c0000640 1 -- 172.21.15.90:0/211011944 shutdown_connections 2024-03-24T01:24:47.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.981+0000 7f91c0000640 1 --2- 172.21.15.90:0/211011944 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91b80cdac0 0x7f91b806b0c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:47.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.981+0000 7f91c0000640 1 -- 172.21.15.90:0/211011944 >> 172.21.15.90:0/211011944 conn(0x7f91b80b1770 msgr2=0x7f91b80b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:47.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.981+0000 7f91c0000640 1 -- 172.21.15.90:0/211011944 shutdown_connections 2024-03-24T01:24:47.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.981+0000 7f91c0000640 1 -- 172.21.15.90:0/211011944 wait complete. 2024-03-24T01:24:47.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.983+0000 7f91c0000640 1 Processor -- start 2024-03-24T01:24:47.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.983+0000 7f91c0000640 1 -- start start 2024-03-24T01:24:47.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.984+0000 7f91c0000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91b806b600 0x7f91b815dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:47.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.984+0000 7f91c0000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91b80cdac0 0x7f91b8153440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:47.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.984+0000 7f91c0000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91b8155030 0x7f91b81e5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:47.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.984+0000 7f91c0000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b8155410 con 0x7f91b8155030 2024-03-24T01:24:47.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.984+0000 7f91c0000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b815bc00 con 0x7f91b80cdac0 2024-03-24T01:24:47.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.984+0000 7f91c0000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91b815af70 con 0x7f91b806b600 2024-03-24T01:24:47.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.984+0000 7f91be000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91b8155030 0x7f91b81e5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:47.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.984+0000 7f91be000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91b8155030 0x7f91b81e5570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60458/0 (socket says 172.21.15.90:60458) 2024-03-24T01:24:47.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.984+0000 7f91be000640 1 -- 172.21.15.90:0/381210531 learned_addr learned my addr 172.21.15.90:0/381210531 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:47.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.985+0000 7f91bcc00640 1 --2- 172.21.15.90:0/381210531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91b80cdac0 0x7f91b8153440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:47.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.985+0000 7f91bd600640 1 --2- 172.21.15.90:0/381210531 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91b806b600 0x7f91b815dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:47.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.985+0000 7f91be000640 1 -- 172.21.15.90:0/381210531 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91b806b600 msgr2=0x7f91b815dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:47.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.985+0000 7f91be000640 1 --2- 172.21.15.90:0/381210531 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91b806b600 0x7f91b815dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:47.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.985+0000 7f91be000640 1 -- 172.21.15.90:0/381210531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91b80cdac0 msgr2=0x7f91b8153440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:47.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.985+0000 7f91be000640 1 --2- 172.21.15.90:0/381210531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91b80cdac0 0x7f91b8153440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:47.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.985+0000 7f91be000640 1 -- 172.21.15.90:0/381210531 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b8153980 con 0x7f91b8155030 2024-03-24T01:24:47.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.985+0000 7f91be000640 1 --2- 172.21.15.90:0/381210531 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91b8155030 0x7f91b81e5570 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f91b000b970 tx=0x7f91b000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:47.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.985+0000 7f91b5600640 1 -- 172.21.15.90:0/381210531 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b0019070 con 0x7f91b8155030 2024-03-24T01:24:47.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.985+0000 7f91b5600640 1 -- 172.21.15.90:0/381210531 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91b000cb40 con 0x7f91b8155030 2024-03-24T01:24:47.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.985+0000 7f91b5600640 1 -- 172.21.15.90:0/381210531 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b0014440 con 0x7f91b8155030 2024-03-24T01:24:47.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.985+0000 7f91c0000640 1 -- 172.21.15.90:0/381210531 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91b815c5b0 con 0x7f91b8155030 2024-03-24T01:24:47.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.986+0000 7f91c0000640 1 -- 172.21.15.90:0/381210531 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91b815cb70 con 0x7f91b8155030 2024-03-24T01:24:47.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.987+0000 7f91c0000640 1 -- 172.21.15.90:0/381210531 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91b8068060 con 0x7f91b8155030 2024-03-24T01:24:47.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.988+0000 7f91b5600640 1 -- 172.21.15.90:0/381210531 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f91b001c070 con 0x7f91b8155030 2024-03-24T01:24:47.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.989+0000 7f91b5600640 1 --2- 172.21.15.90:0/381210531 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f919c06bbd0 0x7f919c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:47.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.989+0000 7f91b5600640 1 -- 172.21.15.90:0/381210531 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f91b008a960 con 0x7f91b8155030 2024-03-24T01:24:47.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.989+0000 7f91bd600640 1 --2- 172.21.15.90:0/381210531 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f919c06bbd0 0x7f919c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:47.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.990+0000 7f91bd600640 1 --2- 172.21.15.90:0/381210531 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f919c06bbd0 0x7f919c06e090 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f91ac000f60 tx=0x7f91ac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:47.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:47.993+0000 7f91b5600640 1 -- 172.21.15.90:0/381210531 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f91b8153980 con 0x7f91b8155030 2024-03-24T01:24:48.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.180+0000 7f91c0000640 1 -- 172.21.15.90:0/381210531 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91b81595e0 con 0x7f91b8155030 2024-03-24T01:24:48.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.182+0000 7f91b5600640 1 -- 172.21.15.90:0/381210531 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18490 (secure 0 0 0) 0x7f91b0059110 con 0x7f91b8155030 2024-03-24T01:24:48.183 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:24:48.183 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:24:17.008224+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:24:08.643966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T01:24:09.189904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T01:24:13.027323+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:24:48.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.185+0000 7f91c0000640 1 -- 172.21.15.90:0/381210531 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f919c06bbd0 msgr2=0x7f919c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:48.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.185+0000 7f91c0000640 1 --2- 172.21.15.90:0/381210531 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f919c06bbd0 0x7f919c06e090 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f91ac000f60 tx=0x7f91ac009040 comp rx=0 tx=0).stop 2024-03-24T01:24:48.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.185+0000 7f91c0000640 1 -- 172.21.15.90:0/381210531 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91b8155030 msgr2=0x7f91b81e5570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:48.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.185+0000 7f91c0000640 1 --2- 172.21.15.90:0/381210531 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91b8155030 0x7f91b81e5570 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f91b000b970 tx=0x7f91b000be30 comp rx=0 tx=0).stop 2024-03-24T01:24:48.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.185+0000 7f91c0000640 1 -- 172.21.15.90:0/381210531 shutdown_connections 2024-03-24T01:24:48.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.185+0000 7f91c0000640 1 --2- 172.21.15.90:0/381210531 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f919c06bbd0 0x7f919c06e090 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:48.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.185+0000 7f91c0000640 1 --2- 172.21.15.90:0/381210531 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91b8155030 0x7f91b81e5570 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:48.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.185+0000 7f91c0000640 1 --2- 172.21.15.90:0/381210531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91b80cdac0 0x7f91b8153440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:48.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.185+0000 7f91c0000640 1 --2- 172.21.15.90:0/381210531 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f91b806b600 0x7f91b815dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:48.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.185+0000 7f91c0000640 1 -- 172.21.15.90:0/381210531 >> 172.21.15.90:0/381210531 conn(0x7f91b80b1770 msgr2=0x7f91b8154300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:48.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.185+0000 7f91c0000640 1 -- 172.21.15.90:0/381210531 shutdown_connections 2024-03-24T01:24:48.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.185+0000 7f91c0000640 1 -- 172.21.15.90:0/381210531 wait complete. 2024-03-24T01:24:48.203 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T01:24:08.643966+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T01:24:09.189904+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T01:24:13.027323+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T01:24:48.203 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:24:48.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.414+0000 7f6fc1e00640 1 Processor -- start 2024-03-24T01:24:48.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.414+0000 7f6fc1e00640 1 -- start start 2024-03-24T01:24:48.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.416+0000 7f6fc1e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fbc154d20 0x7f6fbc15c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:48.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.416+0000 7f6fc1e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6fbc155100 0x7f6fbc15adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:48.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.416+0000 7f6fc1e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6fbc0b2680 0x7f6fbc0cf320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:48.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.416+0000 7f6fc1e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fbc0b2ed0 con 0x7f6fbc0b2680 2024-03-24T01:24:48.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.416+0000 7f6fc1e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fbc0b31b0 con 0x7f6fbc155100 2024-03-24T01:24:48.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.416+0000 7f6fc1e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fbc0b3490 con 0x7f6fbc154d20 2024-03-24T01:24:48.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.416+0000 7f6fc1400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6fbc0b2680 0x7f6fbc0cf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:48.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.416+0000 7f6fc1400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6fbc0b2680 0x7f6fbc0cf320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60466/0 (socket says 172.21.15.90:60466) 2024-03-24T01:24:48.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.416+0000 7f6fc1400640 1 -- 172.21.15.90:0/72579232 learned_addr learned my addr 172.21.15.90:0/72579232 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:48.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.416+0000 7f6fbbe00640 1 --2- 172.21.15.90:0/72579232 >> v2:172.21.15.120:3300/0 conn(0x7f6fbc155100 0x7f6fbc15adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:48.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.416+0000 7f6fc0a00640 1 --2- 172.21.15.90:0/72579232 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fbc154d20 0x7f6fbc15c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:48.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.416+0000 7f6fc1400640 1 -- 172.21.15.90:0/72579232 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fbc154d20 msgr2=0x7f6fbc15c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:48.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.416+0000 7f6fc1400640 1 --2- 172.21.15.90:0/72579232 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fbc154d20 0x7f6fbc15c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:48.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.416+0000 7f6fc1400640 1 -- 172.21.15.90:0/72579232 >> v2:172.21.15.120:3300/0 conn(0x7f6fbc155100 msgr2=0x7f6fbc15adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:48.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.416+0000 7f6fc1400640 1 --2- 172.21.15.90:0/72579232 >> v2:172.21.15.120:3300/0 conn(0x7f6fbc155100 0x7f6fbc15adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:48.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.416+0000 7f6fc1400640 1 -- 172.21.15.90:0/72579232 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fbc0b3770 con 0x7f6fbc0b2680 2024-03-24T01:24:48.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.418+0000 7f6fc1400640 1 --2- 172.21.15.90:0/72579232 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fbc0b2680 0x7f6fbc0cf320 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc067e10 tx=0x7f6fb402bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee98e16e33903aac server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:48.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.418+0000 7f6fbb400640 1 -- 172.21.15.90:0/72579232 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb402c950 con 0x7f6fbc0b2680 2024-03-24T01:24:48.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.418+0000 7f6fbb400640 1 -- 172.21.15.90:0/72579232 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb402caf0 con 0x7f6fbc0b2680 2024-03-24T01:24:48.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.418+0000 7f6fbb400640 1 -- 172.21.15.90:0/72579232 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb40343f0 con 0x7f6fbc0b2680 2024-03-24T01:24:48.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.418+0000 7f6fc1e00640 1 -- 172.21.15.90:0/72579232 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fbc0b2680 msgr2=0x7f6fbc0cf320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:48.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.418+0000 7f6fc1e00640 1 --2- 172.21.15.90:0/72579232 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fbc0b2680 0x7f6fbc0cf320 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc067e10 tx=0x7f6fb402bdb0 comp rx=0 tx=0).stop 2024-03-24T01:24:48.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.419+0000 7f6fc1e00640 1 -- 172.21.15.90:0/72579232 shutdown_connections 2024-03-24T01:24:48.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.419+0000 7f6fc1e00640 1 --2- 172.21.15.90:0/72579232 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fbc0b2680 0x7f6fbc0cf320 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:48.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.419+0000 7f6fc1e00640 1 --2- 172.21.15.90:0/72579232 >> v2:172.21.15.120:3300/0 conn(0x7f6fbc155100 0x7f6fbc15adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:48.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.419+0000 7f6fc1e00640 1 --2- 172.21.15.90:0/72579232 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fbc154d20 0x7f6fbc15c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:48.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.419+0000 7f6fc1e00640 1 -- 172.21.15.90:0/72579232 >> 172.21.15.90:0/72579232 conn(0x7f6fbc06b0c0 msgr2=0x7f6fbc06b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:48.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.419+0000 7f6fc1e00640 1 -- 172.21.15.90:0/72579232 shutdown_connections 2024-03-24T01:24:48.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.419+0000 7f6fc1e00640 1 -- 172.21.15.90:0/72579232 wait complete. 2024-03-24T01:24:48.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.420+0000 7f6fc1e00640 1 Processor -- start 2024-03-24T01:24:48.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.421+0000 7f6fc1e00640 1 -- start start 2024-03-24T01:24:48.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.421+0000 7f6fc1e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fbc0b2680 0x7f6fbc14ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:48.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.421+0000 7f6fc1e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fbc154d20 0x7f6fbc0afe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:48.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.422+0000 7f6fc1e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fbc155100 0x7f6fbc157300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:48.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.422+0000 7f6fc1e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fbc0cf9e0 con 0x7f6fbc155100 2024-03-24T01:24:48.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.422+0000 7f6fc1e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fbc0cf860 con 0x7f6fbc154d20 2024-03-24T01:24:48.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.422+0000 7f6fc1e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fbc0cfb60 con 0x7f6fbc0b2680 2024-03-24T01:24:48.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.422+0000 7f6fc1400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fbc155100 0x7f6fbc157300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:48.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.422+0000 7f6fc1400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fbc155100 0x7f6fbc157300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60470/0 (socket says 172.21.15.90:60470) 2024-03-24T01:24:48.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.422+0000 7f6fc1400640 1 -- 172.21.15.90:0/958628406 learned_addr learned my addr 172.21.15.90:0/958628406 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:48.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.422+0000 7f6fbbe00640 1 --2- 172.21.15.90:0/958628406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fbc154d20 0x7f6fbc0afe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:48.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.422+0000 7f6fc0a00640 1 --2- 172.21.15.90:0/958628406 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fbc0b2680 0x7f6fbc14ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:48.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.422+0000 7f6fc1400640 1 -- 172.21.15.90:0/958628406 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fbc0b2680 msgr2=0x7f6fbc14ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:48.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.422+0000 7f6fc1400640 1 --2- 172.21.15.90:0/958628406 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fbc0b2680 0x7f6fbc14ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:48.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.422+0000 7f6fc1400640 1 -- 172.21.15.90:0/958628406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fbc154d20 msgr2=0x7f6fbc0afe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:48.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.422+0000 7f6fc1400640 1 --2- 172.21.15.90:0/958628406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fbc154d20 0x7f6fbc0afe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:48.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.422+0000 7f6fc1400640 1 -- 172.21.15.90:0/958628406 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fbc159650 con 0x7f6fbc155100 2024-03-24T01:24:48.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.422+0000 7f6fc0a00640 1 --2- 172.21.15.90:0/958628406 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fbc0b2680 0x7f6fbc14ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:48.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.423+0000 7f6fc1400640 1 --2- 172.21.15.90:0/958628406 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fbc155100 0x7f6fbc157300 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4009f40 tx=0x7f6fb4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:48.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.423+0000 7f6fb8c00640 1 -- 172.21.15.90:0/958628406 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb4043020 con 0x7f6fbc155100 2024-03-24T01:24:48.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.423+0000 7f6fb8c00640 1 -- 172.21.15.90:0/958628406 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb403e450 con 0x7f6fbc155100 2024-03-24T01:24:48.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.423+0000 7f6fb8c00640 1 -- 172.21.15.90:0/958628406 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb4009510 con 0x7f6fbc155100 2024-03-24T01:24:48.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.423+0000 7f6fc1e00640 1 -- 172.21.15.90:0/958628406 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fbc159930 con 0x7f6fbc155100 2024-03-24T01:24:48.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.423+0000 7f6fc1e00640 1 -- 172.21.15.90:0/958628406 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fbc159ef0 con 0x7f6fbc155100 2024-03-24T01:24:48.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.424+0000 7f6fc1e00640 1 -- 172.21.15.90:0/958628406 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fbc04f400 con 0x7f6fbc155100 2024-03-24T01:24:48.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.425+0000 7f6fb8c00640 1 -- 172.21.15.90:0/958628406 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6fb4046020 con 0x7f6fbc155100 2024-03-24T01:24:48.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.426+0000 7f6fb8c00640 1 --2- 172.21.15.90:0/958628406 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6f9c06bc00 0x7f6f9c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:48.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.426+0000 7f6fb8c00640 1 -- 172.21.15.90:0/958628406 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f6fb40ad670 con 0x7f6fbc155100 2024-03-24T01:24:48.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.427+0000 7f6fc0a00640 1 --2- 172.21.15.90:0/958628406 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6f9c06bc00 0x7f6f9c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:48.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.428+0000 7f6fc0a00640 1 --2- 172.21.15.90:0/958628406 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6f9c06bc00 0x7f6f9c06e0c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6fb0006fd0 tx=0x7f6fb0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:48.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.431+0000 7f6fb8c00640 1 -- 172.21.15.90:0/958628406 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6fb404d030 con 0x7f6fbc155100 2024-03-24T01:24:48.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.612+0000 7f6fc1e00640 1 -- 172.21.15.90:0/958628406 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6fbc0b3ca0 con 0x7f6fbc155100 2024-03-24T01:24:48.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.613+0000 7f6fb8c00640 1 -- 172.21.15.90:0/958628406 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6fb407bd80 con 0x7f6fbc155100 2024-03-24T01:24:48.613 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:24:48.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.615+0000 7f6fc1e00640 1 -- 172.21.15.90:0/958628406 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6f9c06bc00 msgr2=0x7f6f9c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:48.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.615+0000 7f6fc1e00640 1 --2- 172.21.15.90:0/958628406 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6f9c06bc00 0x7f6f9c06e0c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6fb0006fd0 tx=0x7f6fb0008040 comp rx=0 tx=0).stop 2024-03-24T01:24:48.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.615+0000 7f6fc1e00640 1 -- 172.21.15.90:0/958628406 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fbc155100 msgr2=0x7f6fbc157300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:48.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.615+0000 7f6fc1e00640 1 --2- 172.21.15.90:0/958628406 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fbc155100 0x7f6fbc157300 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4009f40 tx=0x7f6fb4004a40 comp rx=0 tx=0).stop 2024-03-24T01:24:48.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.615+0000 7f6fc1e00640 1 -- 172.21.15.90:0/958628406 shutdown_connections 2024-03-24T01:24:48.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.615+0000 7f6fc1e00640 1 --2- 172.21.15.90:0/958628406 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6f9c06bc00 0x7f6f9c06e0c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:48.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.615+0000 7f6fc1e00640 1 --2- 172.21.15.90:0/958628406 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fbc155100 0x7f6fbc157300 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:48.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.615+0000 7f6fc1e00640 1 --2- 172.21.15.90:0/958628406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fbc154d20 0x7f6fbc0afe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:48.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.615+0000 7f6fc1e00640 1 --2- 172.21.15.90:0/958628406 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6fbc0b2680 0x7f6fbc14ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:48.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.615+0000 7f6fc1e00640 1 -- 172.21.15.90:0/958628406 >> 172.21.15.90:0/958628406 conn(0x7f6fbc06b0c0 msgr2=0x7f6fbc0ce3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:48.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.615+0000 7f6fc1e00640 1 -- 172.21.15.90:0/958628406 shutdown_connections 2024-03-24T01:24:48.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.615+0000 7f6fc1e00640 1 -- 172.21.15.90:0/958628406 wait complete. 2024-03-24T01:24:48.633 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T01:24:48.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.839+0000 7f4bc1c00640 1 Processor -- start 2024-03-24T01:24:48.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.839+0000 7f4bc1c00640 1 -- start start 2024-03-24T01:24:48.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.841+0000 7f4bc1c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4bbc0706f0 0x7f4bbc0cda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:48.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.841+0000 7f4bc1c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4bbc0ccb10 0x7f4bbc06b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:48.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.841+0000 7f4bc1c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4bbc06b600 0x7f4bbc15d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:48.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.841+0000 7f4bc1c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bbc1629a0 con 0x7f4bbc0706f0 2024-03-24T01:24:48.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.841+0000 7f4bc1c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bbc162510 con 0x7f4bbc0ccb10 2024-03-24T01:24:48.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.841+0000 7f4bc1c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bbc161be0 con 0x7f4bbc06b600 2024-03-24T01:24:48.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.841+0000 7f4bbaa00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4bbc0706f0 0x7f4bbc0cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:48.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.841+0000 7f4bbaa00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4bbc0706f0 0x7f4bbc0cda80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60478/0 (socket says 172.21.15.90:60478) 2024-03-24T01:24:48.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.841+0000 7f4bbaa00640 1 -- 172.21.15.90:0/1526312261 learned_addr learned my addr 172.21.15.90:0/1526312261 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:48.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.841+0000 7f4bba000640 1 --2- 172.21.15.90:0/1526312261 >> v2:172.21.15.120:3300/0 conn(0x7f4bbc0ccb10 0x7f4bbc06b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:48.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.842+0000 7f4bbb400640 1 --2- 172.21.15.90:0/1526312261 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4bbc06b600 0x7f4bbc15d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:48.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.842+0000 7f4bbaa00640 1 -- 172.21.15.90:0/1526312261 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4bbc06b600 msgr2=0x7f4bbc15d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:48.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.842+0000 7f4bbaa00640 1 --2- 172.21.15.90:0/1526312261 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4bbc06b600 0x7f4bbc15d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:48.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.842+0000 7f4bbaa00640 1 -- 172.21.15.90:0/1526312261 >> v2:172.21.15.120:3300/0 conn(0x7f4bbc0ccb10 msgr2=0x7f4bbc06b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:48.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.842+0000 7f4bbaa00640 1 --2- 172.21.15.90:0/1526312261 >> v2:172.21.15.120:3300/0 conn(0x7f4bbc0ccb10 0x7f4bbc06b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:48.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.842+0000 7f4bbaa00640 1 -- 172.21.15.90:0/1526312261 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bbc161ec0 con 0x7f4bbc0706f0 2024-03-24T01:24:48.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.842+0000 7f4bbb400640 1 --2- 172.21.15.90:0/1526312261 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4bbc06b600 0x7f4bbc15d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:24:48.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.844+0000 7f4bbaa00640 1 --2- 172.21.15.90:0/1526312261 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4bbc0706f0 0x7f4bbc0cda80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc0cc860 tx=0x7f4bac02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee3b077611e5509c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:48.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.844+0000 7f4bb9600640 1 -- 172.21.15.90:0/1526312261 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bac02ca70 con 0x7f4bbc0706f0 2024-03-24T01:24:48.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.844+0000 7f4bb9600640 1 -- 172.21.15.90:0/1526312261 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bac02cc10 con 0x7f4bbc0706f0 2024-03-24T01:24:48.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.844+0000 7f4bb9600640 1 -- 172.21.15.90:0/1526312261 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bac0343f0 con 0x7f4bbc0706f0 2024-03-24T01:24:48.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.845+0000 7f4bc1c00640 1 -- 172.21.15.90:0/1526312261 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4bbc0706f0 msgr2=0x7f4bbc0cda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:48.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.845+0000 7f4bc1c00640 1 --2- 172.21.15.90:0/1526312261 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4bbc0706f0 0x7f4bbc0cda80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc0cc860 tx=0x7f4bac02bed0 comp rx=0 tx=0).stop 2024-03-24T01:24:48.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.845+0000 7f4bc1c00640 1 -- 172.21.15.90:0/1526312261 shutdown_connections 2024-03-24T01:24:48.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.845+0000 7f4bc1c00640 1 --2- 172.21.15.90:0/1526312261 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4bbc06b600 0x7f4bbc15d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:48.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.845+0000 7f4bc1c00640 1 --2- 172.21.15.90:0/1526312261 >> v2:172.21.15.120:3300/0 conn(0x7f4bbc0ccb10 0x7f4bbc06b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:48.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.845+0000 7f4bc1c00640 1 --2- 172.21.15.90:0/1526312261 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4bbc0706f0 0x7f4bbc0cda80 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:48.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.845+0000 7f4bc1c00640 1 -- 172.21.15.90:0/1526312261 >> 172.21.15.90:0/1526312261 conn(0x7f4bbc099a70 msgr2=0x7f4bbc099e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:48.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.845+0000 7f4bc1c00640 1 -- 172.21.15.90:0/1526312261 shutdown_connections 2024-03-24T01:24:48.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.845+0000 7f4bc1c00640 1 -- 172.21.15.90:0/1526312261 wait complete. 2024-03-24T01:24:48.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.847+0000 7f4bc1c00640 1 Processor -- start 2024-03-24T01:24:48.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.847+0000 7f4bc1c00640 1 -- start start 2024-03-24T01:24:48.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.848+0000 7f4bc1c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4bbc06b600 0x7f4bbc15de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:48.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.848+0000 7f4bc1c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4bbc0706f0 0x7f4bbc156060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:48.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.848+0000 7f4bc1c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4bbc0ccb10 0x7f4bbc0c8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:48.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.848+0000 7f4bc1c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bbc0cdfc0 con 0x7f4bbc0ccb10 2024-03-24T01:24:48.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.848+0000 7f4bc1c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bbc070ad0 con 0x7f4bbc0706f0 2024-03-24T01:24:48.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.848+0000 7f4bc1c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bbc15daa0 con 0x7f4bbc06b600 2024-03-24T01:24:48.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.849+0000 7f4bbb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4bbc0ccb10 0x7f4bbc0c8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:48.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.849+0000 7f4bbb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4bbc0ccb10 0x7f4bbc0c8e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60484/0 (socket says 172.21.15.90:60484) 2024-03-24T01:24:48.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.849+0000 7f4bbb400640 1 -- 172.21.15.90:0/3170370241 learned_addr learned my addr 172.21.15.90:0/3170370241 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:48.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.849+0000 7f4bbaa00640 1 --2- 172.21.15.90:0/3170370241 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4bbc06b600 0x7f4bbc15de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:48.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.849+0000 7f4bba000640 1 --2- 172.21.15.90:0/3170370241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4bbc0706f0 0x7f4bbc156060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:48.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.849+0000 7f4bbb400640 1 -- 172.21.15.90:0/3170370241 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4bbc06b600 msgr2=0x7f4bbc15de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:48.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.849+0000 7f4bbb400640 1 --2- 172.21.15.90:0/3170370241 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4bbc06b600 0x7f4bbc15de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:48.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.849+0000 7f4bbb400640 1 -- 172.21.15.90:0/3170370241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4bbc0706f0 msgr2=0x7f4bbc156060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:48.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.849+0000 7f4bbb400640 1 --2- 172.21.15.90:0/3170370241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4bbc0706f0 0x7f4bbc156060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:48.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.849+0000 7f4bbb400640 1 -- 172.21.15.90:0/3170370241 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bbc15edb0 con 0x7f4bbc0ccb10 2024-03-24T01:24:48.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.850+0000 7f4bbb400640 1 --2- 172.21.15.90:0/3170370241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4bbc0ccb10 0x7f4bbc0c8e80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4bb400e7e0 tx=0x7f4bb400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:48.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.850+0000 7f4bb2a00640 1 -- 172.21.15.90:0/3170370241 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb400f040 con 0x7f4bbc0ccb10 2024-03-24T01:24:48.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.850+0000 7f4bb2a00640 1 -- 172.21.15.90:0/3170370241 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb4004030 con 0x7f4bbc0ccb10 2024-03-24T01:24:48.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.850+0000 7f4bb2a00640 1 -- 172.21.15.90:0/3170370241 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb40043e0 con 0x7f4bbc0ccb10 2024-03-24T01:24:48.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.850+0000 7f4bc1c00640 1 -- 172.21.15.90:0/3170370241 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bbc15e510 con 0x7f4bbc0ccb10 2024-03-24T01:24:48.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.850+0000 7f4bc1c00640 1 -- 172.21.15.90:0/3170370241 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bbc15ead0 con 0x7f4bbc0ccb10 2024-03-24T01:24:48.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.851+0000 7f4bc1c00640 1 -- 172.21.15.90:0/3170370241 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bbc04f400 con 0x7f4bbc0ccb10 2024-03-24T01:24:48.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.852+0000 7f4bb2a00640 1 -- 172.21.15.90:0/3170370241 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4bb400c6d0 con 0x7f4bbc0ccb10 2024-03-24T01:24:48.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.853+0000 7f4bb2a00640 1 --2- 172.21.15.90:0/3170370241 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4b9c06bbd0 0x7f4b9c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:48.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.853+0000 7f4bb2a00640 1 -- 172.21.15.90:0/3170370241 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f4bb408ab30 con 0x7f4bbc0ccb10 2024-03-24T01:24:48.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.853+0000 7f4bbaa00640 1 --2- 172.21.15.90:0/3170370241 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4b9c06bbd0 0x7f4b9c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:48.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.854+0000 7f4bbaa00640 1 --2- 172.21.15.90:0/3170370241 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4b9c06bbd0 0x7f4b9c06e090 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4bac035000 tx=0x7f4bac030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:48.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:48.857+0000 7f4bb2a00640 1 -- 172.21.15.90:0/3170370241 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4bb4017020 con 0x7f4bbc0ccb10 2024-03-24T01:24:49.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.057+0000 7f4bc1c00640 1 -- 172.21.15.90:0/3170370241 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4bbc0cd5a0 con 0x7f4bbc0ccb10 2024-03-24T01:24:49.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.058+0000 7f4bb2a00640 1 -- 172.21.15.90:0/3170370241 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4bb4059240 con 0x7f4bbc0ccb10 2024-03-24T01:24:49.058 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:24:49.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.061+0000 7f4bc1c00640 1 -- 172.21.15.90:0/3170370241 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4b9c06bbd0 msgr2=0x7f4b9c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:49.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.061+0000 7f4bc1c00640 1 --2- 172.21.15.90:0/3170370241 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4b9c06bbd0 0x7f4b9c06e090 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4bac035000 tx=0x7f4bac030000 comp rx=0 tx=0).stop 2024-03-24T01:24:49.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.061+0000 7f4bc1c00640 1 -- 172.21.15.90:0/3170370241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4bbc0ccb10 msgr2=0x7f4bbc0c8e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:49.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.061+0000 7f4bc1c00640 1 --2- 172.21.15.90:0/3170370241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4bbc0ccb10 0x7f4bbc0c8e80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4bb400e7e0 tx=0x7f4bb400eca0 comp rx=0 tx=0).stop 2024-03-24T01:24:49.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.061+0000 7f4bc1c00640 1 -- 172.21.15.90:0/3170370241 shutdown_connections 2024-03-24T01:24:49.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.061+0000 7f4bc1c00640 1 --2- 172.21.15.90:0/3170370241 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4b9c06bbd0 0x7f4b9c06e090 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:49.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.061+0000 7f4bc1c00640 1 --2- 172.21.15.90:0/3170370241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4bbc0ccb10 0x7f4bbc0c8e80 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:49.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.061+0000 7f4bc1c00640 1 --2- 172.21.15.90:0/3170370241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4bbc0706f0 0x7f4bbc156060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:49.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.061+0000 7f4bc1c00640 1 --2- 172.21.15.90:0/3170370241 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4bbc06b600 0x7f4bbc15de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:49.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.061+0000 7f4bc1c00640 1 -- 172.21.15.90:0/3170370241 >> 172.21.15.90:0/3170370241 conn(0x7f4bbc099a70 msgr2=0x7f4bbc099e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:49.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.061+0000 7f4bc1c00640 1 -- 172.21.15.90:0/3170370241 shutdown_connections 2024-03-24T01:24:49.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.061+0000 7f4bc1c00640 1 -- 172.21.15.90:0/3170370241 wait complete. 2024-03-24T01:24:49.081 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T01:24:49.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.291+0000 7f1762000640 1 Processor -- start 2024-03-24T01:24:49.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.292+0000 7f1762000640 1 -- start start 2024-03-24T01:24:49.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.293+0000 7f1762000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f175c09a3e0 0x7f175c06c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:49.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.293+0000 7f1762000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f175c0d0870 0x7f175c06c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:49.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.293+0000 7f1762000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f175c0d1600 con 0x7f175c15e0e0 2024-03-24T01:24:49.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.293+0000 7f1762000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f175c0cc200 con 0x7f175c09a3e0 2024-03-24T01:24:49.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.293+0000 7f1762000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f175c0d0d60 con 0x7f175c0d0870 2024-03-24T01:24:49.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.293+0000 7f175b400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f175c15e0e0 0x7f175c099cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51360/0 (socket says 172.21.15.90:51360) 2024-03-24T01:24:49.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.293+0000 7f175b400640 1 -- 172.21.15.90:0/3092112854 learned_addr learned my addr 172.21.15.90:0/3092112854 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:49.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.294+0000 7f175aa00640 1 --2- 172.21.15.90:0/3092112854 >> v2:172.21.15.120:3300/0 conn(0x7f175c09a3e0 0x7f175c06c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:49.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.294+0000 7f175be00640 1 --2- 172.21.15.90:0/3092112854 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f175c0d0870 0x7f175c06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:49.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.294+0000 7f175a000640 1 -- 172.21.15.90:0/3092112854 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1090288469 0 0) 0x7f175c0d1600 con 0x7f175c15e0e0 2024-03-24T01:24:49.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.294+0000 7f175a000640 1 -- 172.21.15.90:0/3092112854 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1740003520 con 0x7f175c15e0e0 2024-03-24T01:24:49.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.294+0000 7f175a000640 1 -- 172.21.15.90:0/3092112854 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2272103622 0 0) 0x7f1740003520 con 0x7f175c15e0e0 2024-03-24T01:24:49.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.294+0000 7f175a000640 1 -- 172.21.15.90:0/3092112854 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f175c0d0870 msgr2=0x7f175c06c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:49.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.294+0000 7f175a000640 1 --2- 172.21.15.90:0/3092112854 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f175c0d0870 0x7f175c06c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:49.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.294+0000 7f175a000640 1 -- 172.21.15.90:0/3092112854 >> v2:172.21.15.120:3300/0 conn(0x7f175c09a3e0 msgr2=0x7f175c06c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:49.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.294+0000 7f175a000640 1 --2- 172.21.15.90:0/3092112854 >> v2:172.21.15.120:3300/0 conn(0x7f175c09a3e0 0x7f175c06c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:49.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.294+0000 7f175a000640 1 -- 172.21.15.90:0/3092112854 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f175c0d1040 con 0x7f175c15e0e0 2024-03-24T01:24:49.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.294+0000 7f175a000640 1 -- 172.21.15.90:0/3092112854 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f174c001670 con 0x7f175c15e0e0 2024-03-24T01:24:49.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.294+0000 7f175a000640 1 -- 172.21.15.90:0/3092112854 >> v1:172.21.15.90:6789/0 conn(0x7f175c15e0e0 legacy=0x7f175c099cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:49.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.294+0000 7f175a000640 1 --2- 172.21.15.90:0/3092112854 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17400039b0 0x7f1740003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:49.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.294+0000 7f175a000640 1 --2- 172.21.15.90:0/3092112854 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1740004800 0x7f1740004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:49.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.294+0000 7f175a000640 1 --2- 172.21.15.90:0/3092112854 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f175c0d0870 0x7f17400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:49.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.294+0000 7f175a000640 1 -- 172.21.15.90:0/3092112854 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f175c0cc200 con 0x7f17400039b0 2024-03-24T01:24:49.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.294+0000 7f175a000640 1 -- 172.21.15.90:0/3092112854 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f175c0d0d60 con 0x7f1740004800 2024-03-24T01:24:49.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.294+0000 7f175a000640 1 -- 172.21.15.90:0/3092112854 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f174000ba70 con 0x7f175c0d0870 2024-03-24T01:24:49.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.295+0000 7f175b400640 1 --2- 172.21.15.90:0/3092112854 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17400039b0 0x7f1740003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:49.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.295+0000 7f175be00640 1 --2- 172.21.15.90:0/3092112854 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f175c0d0870 0x7f17400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:49.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.295+0000 7f175aa00640 1 --2- 172.21.15.90:0/3092112854 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1740004800 0x7f1740004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:49.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.295+0000 7f175b400640 1 -- 172.21.15.90:0/3092112854 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f175c0d0870 msgr2=0x7f17400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:49.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.295+0000 7f175b400640 1 --2- 172.21.15.90:0/3092112854 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f175c0d0870 0x7f17400051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:49.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.295+0000 7f175b400640 1 -- 172.21.15.90:0/3092112854 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1740004800 msgr2=0x7f1740004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:49.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.295+0000 7f175b400640 1 --2- 172.21.15.90:0/3092112854 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1740004800 0x7f1740004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:49.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.295+0000 7f175b400640 1 -- 172.21.15.90:0/3092112854 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1740003520 con 0x7f17400039b0 2024-03-24T01:24:49.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.296+0000 7f175b400640 1 --2- 172.21.15.90:0/3092112854 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17400039b0 0x7f1740003dd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f174c001eb0 tx=0x7f1740005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a04bc6791c92169 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:49.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.296+0000 7f175a000640 1 -- 172.21.15.90:0/3092112854 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174c007d70 con 0x7f17400039b0 2024-03-24T01:24:49.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.296+0000 7f175a000640 1 -- 172.21.15.90:0/3092112854 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f174c030070 con 0x7f17400039b0 2024-03-24T01:24:49.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.296+0000 7f175a000640 1 -- 172.21.15.90:0/3092112854 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174c005d30 con 0x7f17400039b0 2024-03-24T01:24:49.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.297+0000 7f1762000640 1 -- 172.21.15.90:0/3092112854 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17400039b0 msgr2=0x7f1740003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:49.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.297+0000 7f1762000640 1 --2- 172.21.15.90:0/3092112854 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17400039b0 0x7f1740003dd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f174c001eb0 tx=0x7f1740005830 comp rx=0 tx=0).stop 2024-03-24T01:24:49.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.297+0000 7f175be00640 1 -- 172.21.15.90:0/3092112854 reap_dead start 2024-03-24T01:24:49.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.297+0000 7f1762000640 1 -- 172.21.15.90:0/3092112854 shutdown_connections 2024-03-24T01:24:49.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.297+0000 7f1762000640 1 -- 172.21.15.90:0/3092112854 >> 172.21.15.90:0/3092112854 conn(0x7f175c0b34e0 msgr2=0x7f175c0b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:49.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.297+0000 7f1762000640 1 -- 172.21.15.90:0/3092112854 shutdown_connections 2024-03-24T01:24:49.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.297+0000 7f1762000640 1 -- 172.21.15.90:0/3092112854 wait complete. 2024-03-24T01:24:49.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.299+0000 7f1762000640 1 Processor -- start 2024-03-24T01:24:49.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.299+0000 7f1762000640 1 -- start start 2024-03-24T01:24:49.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.300+0000 7f1762000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17400039b0 0x7f175c09b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.300+0000 7f1762000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f175c15cc10 0x7f175c0ca500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.300+0000 7f1762000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f175c15d090 0x7f175c070a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.300+0000 7f1762000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f175c15e4c0 con 0x7f17400039b0 2024-03-24T01:24:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.300+0000 7f1762000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f175c0b2f70 con 0x7f175c15d090 2024-03-24T01:24:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.300+0000 7f1762000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f175c15e640 con 0x7f175c15cc10 2024-03-24T01:24:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.300+0000 7f175b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17400039b0 0x7f175c09b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.300+0000 7f175b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17400039b0 0x7f175c09b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60502/0 (socket says 172.21.15.90:60502) 2024-03-24T01:24:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.300+0000 7f175b400640 1 -- 172.21.15.90:0/4137564632 learned_addr learned my addr 172.21.15.90:0/4137564632 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.300+0000 7f175be00640 1 --2- 172.21.15.90:0/4137564632 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f175c15d090 0x7f175c070a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.300+0000 7f175aa00640 1 --2- 172.21.15.90:0/4137564632 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f175c15cc10 0x7f175c0ca500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.300+0000 7f175b400640 1 -- 172.21.15.90:0/4137564632 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f175c15cc10 msgr2=0x7f175c0ca500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.300+0000 7f175b400640 1 --2- 172.21.15.90:0/4137564632 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f175c15cc10 0x7f175c0ca500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.300+0000 7f175b400640 1 -- 172.21.15.90:0/4137564632 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f175c15d090 msgr2=0x7f175c070a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.300+0000 7f175b400640 1 --2- 172.21.15.90:0/4137564632 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f175c15d090 0x7f175c070a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.300+0000 7f175b400640 1 -- 172.21.15.90:0/4137564632 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f175c1f74a0 con 0x7f17400039b0 2024-03-24T01:24:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.301+0000 7f175b400640 1 --2- 172.21.15.90:0/4137564632 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17400039b0 0x7f175c09b830 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f174c00c210 tx=0x7f174c000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.301+0000 7f1753400640 1 -- 172.21.15.90:0/4137564632 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174c004bf0 con 0x7f17400039b0 2024-03-24T01:24:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.301+0000 7f1753400640 1 -- 172.21.15.90:0/4137564632 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f174c004d90 con 0x7f17400039b0 2024-03-24T01:24:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.301+0000 7f1762000640 1 -- 172.21.15.90:0/4137564632 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f175c1f7780 con 0x7f17400039b0 2024-03-24T01:24:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.301+0000 7f1753400640 1 -- 172.21.15.90:0/4137564632 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174c007270 con 0x7f17400039b0 2024-03-24T01:24:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.301+0000 7f1762000640 1 -- 172.21.15.90:0/4137564632 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f175c1f7d40 con 0x7f17400039b0 2024-03-24T01:24:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.302+0000 7f1762000640 1 -- 172.21.15.90:0/4137564632 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f175c1f3ad0 con 0x7f17400039b0 2024-03-24T01:24:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.303+0000 7f1753400640 1 -- 172.21.15.90:0/4137564632 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f174c047020 con 0x7f17400039b0 2024-03-24T01:24:49.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.304+0000 7f1753400640 1 --2- 172.21.15.90:0/4137564632 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f173c06bbd0 0x7f173c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:49.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.304+0000 7f1753400640 1 -- 172.21.15.90:0/4137564632 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f174c0ac0a0 con 0x7f17400039b0 2024-03-24T01:24:49.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.304+0000 7f175aa00640 1 --2- 172.21.15.90:0/4137564632 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f173c06bbd0 0x7f173c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:49.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.305+0000 7f175aa00640 1 --2- 172.21.15.90:0/4137564632 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f173c06bbd0 0x7f173c06e090 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f175c0c9d50 tx=0x7f1748008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:49.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.308+0000 7f1753400640 1 -- 172.21.15.90:0/4137564632 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f174c07a7b0 con 0x7f17400039b0 2024-03-24T01:24:49.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.505+0000 7f1762000640 1 -- 172.21.15.90:0/4137564632 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f175c04f400 con 0x7f17400039b0 2024-03-24T01:24:49.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.506+0000 7f1753400640 1 -- 172.21.15.90:0/4137564632 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f174c07a0b0 con 0x7f17400039b0 2024-03-24T01:24:49.506 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-24T01:24:49.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.509+0000 7f1762000640 1 -- 172.21.15.90:0/4137564632 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f173c06bbd0 msgr2=0x7f173c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:49.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.509+0000 7f1762000640 1 --2- 172.21.15.90:0/4137564632 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f173c06bbd0 0x7f173c06e090 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f175c0c9d50 tx=0x7f1748008040 comp rx=0 tx=0).stop 2024-03-24T01:24:49.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.509+0000 7f1762000640 1 -- 172.21.15.90:0/4137564632 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17400039b0 msgr2=0x7f175c09b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:49.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.509+0000 7f1762000640 1 --2- 172.21.15.90:0/4137564632 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17400039b0 0x7f175c09b830 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f174c00c210 tx=0x7f174c000fe0 comp rx=0 tx=0).stop 2024-03-24T01:24:49.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.509+0000 7f1762000640 1 -- 172.21.15.90:0/4137564632 shutdown_connections 2024-03-24T01:24:49.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.509+0000 7f1762000640 1 --2- 172.21.15.90:0/4137564632 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f173c06bbd0 0x7f173c06e090 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:49.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.509+0000 7f1762000640 1 --2- 172.21.15.90:0/4137564632 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f175c15d090 0x7f175c070a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:49.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.509+0000 7f1762000640 1 --2- 172.21.15.90:0/4137564632 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f175c15cc10 0x7f175c0ca500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:49.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.509+0000 7f1762000640 1 --2- 172.21.15.90:0/4137564632 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17400039b0 0x7f175c09b830 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:49.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.509+0000 7f1762000640 1 -- 172.21.15.90:0/4137564632 >> 172.21.15.90:0/4137564632 conn(0x7f175c0b34e0 msgr2=0x7f175c0b64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:49.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.509+0000 7f1762000640 1 -- 172.21.15.90:0/4137564632 shutdown_connections 2024-03-24T01:24:49.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.509+0000 7f1762000640 1 -- 172.21.15.90:0/4137564632 wait complete. 2024-03-24T01:24:49.526 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T01:24:49.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.738+0000 7f8fd2a00640 1 Processor -- start 2024-03-24T01:24:49.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.738+0000 7f8fd2a00640 1 -- start start 2024-03-24T01:24:49.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.740+0000 7f8fd2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8fcc06b7d0 0x7f8fcc1664d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:49.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.740+0000 7f8fd2a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fcc0ad8d0 con 0x7f8fcc072e90 2024-03-24T01:24:49.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.740+0000 7f8fd2a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fcc0b16d0 con 0x7f8fcc1454c0 2024-03-24T01:24:49.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.740+0000 7f8fd2a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fcc0b0230 con 0x7f8fcc06b7d0 2024-03-24T01:24:49.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.740+0000 7f8fd0c00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f8fcc072e90 0x7f8fcc06b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51376/0 (socket says 172.21.15.90:51376) 2024-03-24T01:24:49.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.740+0000 7f8fd0c00640 1 -- 172.21.15.90:0/4021563338 learned_addr learned my addr 172.21.15.90:0/4021563338 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:49.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.741+0000 7f8fd2000640 1 --2- 172.21.15.90:0/4021563338 >> v2:172.21.15.120:3300/0 conn(0x7f8fcc06b7d0 0x7f8fcc1664d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:49.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.741+0000 7f8fcbe00640 1 -- 172.21.15.90:0/4021563338 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3715875581 0 0) 0x7f8fb8001110 con 0x7f8fcc072e90 2024-03-24T01:24:49.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.741+0000 7f8fcbe00640 1 -- 172.21.15.90:0/4021563338 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fb4003520 con 0x7f8fcc072e90 2024-03-24T01:24:49.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.741+0000 7f8fcbe00640 1 -- 172.21.15.90:0/4021563338 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 522908631 0 0) 0x7f8fb4003520 con 0x7f8fcc072e90 2024-03-24T01:24:49.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.741+0000 7f8fcbe00640 1 -- 172.21.15.90:0/4021563338 >> v2:172.21.15.120:3300/0 conn(0x7f8fcc06b7d0 msgr2=0x7f8fcc1664d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:49.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.741+0000 7f8fcbe00640 1 --2- 172.21.15.90:0/4021563338 >> v2:172.21.15.120:3300/0 conn(0x7f8fcc06b7d0 0x7f8fcc1664d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:49.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.741+0000 7f8fcbe00640 1 -- 172.21.15.90:0/4021563338 >> v1:172.21.15.120:6789/0 conn(0x7f8fcc1454c0 legacy=0x7f8fcc0aee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:49.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.741+0000 7f8fcbe00640 1 -- 172.21.15.90:0/4021563338 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fcc0af620 con 0x7f8fcc072e90 2024-03-24T01:24:49.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.741+0000 7f8fcbe00640 1 -- 172.21.15.90:0/4021563338 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f8fb8001840 con 0x7f8fcc072e90 2024-03-24T01:24:49.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.741+0000 7f8fcbe00640 1 -- 172.21.15.90:0/4021563338 >> v1:172.21.15.90:6789/0 conn(0x7f8fcc072e90 legacy=0x7f8fcc06b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:49.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.741+0000 7f8fcbe00640 1 --2- 172.21.15.90:0/4021563338 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fb40039b0 0x7f8fb4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:49.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.741+0000 7f8fcbe00640 1 --2- 172.21.15.90:0/4021563338 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fb4004800 0x7f8fb4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:49.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.741+0000 7f8fcbe00640 1 --2- 172.21.15.90:0/4021563338 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fb40051c0 0x7f8fb400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:49.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.741+0000 7f8fcbe00640 1 -- 172.21.15.90:0/4021563338 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc0b0230 con 0x7f8fb40051c0 2024-03-24T01:24:49.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.741+0000 7f8fcbe00640 1 -- 172.21.15.90:0/4021563338 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb400c060 con 0x7f8fb40039b0 2024-03-24T01:24:49.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.741+0000 7f8fcbe00640 1 -- 172.21.15.90:0/4021563338 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb400c260 con 0x7f8fb4004800 2024-03-24T01:24:49.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.742+0000 7f8fd2000640 1 --2- 172.21.15.90:0/4021563338 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fb40051c0 0x7f8fb400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:49.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.742+0000 7f8fd1600640 1 --2- 172.21.15.90:0/4021563338 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fb40039b0 0x7f8fb4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:49.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.742+0000 7f8fd0c00640 1 --2- 172.21.15.90:0/4021563338 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fb4004800 0x7f8fb4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:49.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.742+0000 7f8fd2000640 1 -- 172.21.15.90:0/4021563338 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fb4004800 msgr2=0x7f8fb4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:49.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.742+0000 7f8fd2000640 1 --2- 172.21.15.90:0/4021563338 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fb4004800 0x7f8fb4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:49.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.742+0000 7f8fd2000640 1 -- 172.21.15.90:0/4021563338 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fb40039b0 msgr2=0x7f8fb4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:49.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.742+0000 7f8fd2000640 1 --2- 172.21.15.90:0/4021563338 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fb40039b0 0x7f8fb4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:49.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.742+0000 7f8fd2000640 1 -- 172.21.15.90:0/4021563338 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fb4003520 con 0x7f8fb40051c0 2024-03-24T01:24:49.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.743+0000 7f8fd2000640 1 -- 172.21.15.90:0/4021563338 reap_dead start 2024-03-24T01:24:49.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.743+0000 7f8fd2000640 1 --2- 172.21.15.90:0/4021563338 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fb40051c0 0x7f8fb400b9c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4009f90 tx=0x7f8fb4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0a2ad742dcb733 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:49.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.743+0000 7f8fcbe00640 1 -- 172.21.15.90:0/4021563338 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4009c10 con 0x7f8fb40051c0 2024-03-24T01:24:49.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.744+0000 7f8fcbe00640 1 -- 172.21.15.90:0/4021563338 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc402cc20 con 0x7f8fb40051c0 2024-03-24T01:24:49.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.744+0000 7f8fcbe00640 1 -- 172.21.15.90:0/4021563338 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4003990 con 0x7f8fb40051c0 2024-03-24T01:24:49.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.744+0000 7f8fd2a00640 1 -- 172.21.15.90:0/4021563338 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fb40051c0 msgr2=0x7f8fb400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:49.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.744+0000 7f8fd2a00640 1 --2- 172.21.15.90:0/4021563338 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fb40051c0 0x7f8fb400b9c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4009f90 tx=0x7f8fb4005830 comp rx=0 tx=0).stop 2024-03-24T01:24:49.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.744+0000 7f8fd2a00640 1 -- 172.21.15.90:0/4021563338 shutdown_connections 2024-03-24T01:24:49.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.744+0000 7f8fd2a00640 1 --2- 172.21.15.90:0/4021563338 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fb40051c0 0x7f8fb400b9c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:49.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.744+0000 7f8fd2a00640 1 -- 172.21.15.90:0/4021563338 >> 172.21.15.90:0/4021563338 conn(0x7f8fcc0b5ea0 msgr2=0x7f8fcc099a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:49.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.744+0000 7f8fd2a00640 1 -- 172.21.15.90:0/4021563338 shutdown_connections 2024-03-24T01:24:49.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.744+0000 7f8fd2a00640 1 -- 172.21.15.90:0/4021563338 wait complete. 2024-03-24T01:24:49.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.746+0000 7f8fd2a00640 1 Processor -- start 2024-03-24T01:24:49.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.746+0000 7f8fd2a00640 1 -- start start 2024-03-24T01:24:49.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.747+0000 7f8fd2a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fb40051c0 0x7f8fcc0cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:49.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.747+0000 7f8fd2a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fcc13e710 0x7f8fcc140730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:49.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.747+0000 7f8fd2a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fcc13c6f0 0x7f8fcc140c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:49.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.747+0000 7f8fd2a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc0afec0 con 0x7f8fb40051c0 2024-03-24T01:24:49.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.747+0000 7f8fd2a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc0afbe0 con 0x7f8fcc13c6f0 2024-03-24T01:24:49.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.747+0000 7f8fd2a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc15be50 con 0x7f8fcc13e710 2024-03-24T01:24:49.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.747+0000 7f8fd1600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fb40051c0 0x7f8fcc0cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:49.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.747+0000 7f8fd1600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fb40051c0 0x7f8fcc0cfb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60510/0 (socket says 172.21.15.90:60510) 2024-03-24T01:24:49.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.747+0000 7f8fd1600640 1 -- 172.21.15.90:0/3061377039 learned_addr learned my addr 172.21.15.90:0/3061377039 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:49.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.747+0000 7f8fd0c00640 1 --2- 172.21.15.90:0/3061377039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fcc13e710 0x7f8fcc140730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:49.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.748+0000 7f8fd2000640 1 --2- 172.21.15.90:0/3061377039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fcc13c6f0 0x7f8fcc140c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:49.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.748+0000 7f8fd1600640 1 -- 172.21.15.90:0/3061377039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fcc13e710 msgr2=0x7f8fcc140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:49.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.748+0000 7f8fd1600640 1 --2- 172.21.15.90:0/3061377039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fcc13e710 0x7f8fcc140730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:49.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.748+0000 7f8fd1600640 1 -- 172.21.15.90:0/3061377039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fcc13c6f0 msgr2=0x7f8fcc140c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:49.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.748+0000 7f8fd1600640 1 --2- 172.21.15.90:0/3061377039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fcc13c6f0 0x7f8fcc140c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:49.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.748+0000 7f8fd1600640 1 -- 172.21.15.90:0/3061377039 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fcc13ec10 con 0x7f8fb40051c0 2024-03-24T01:24:49.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.748+0000 7f8fd0c00640 1 --2- 172.21.15.90:0/3061377039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fcc13e710 0x7f8fcc140730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:49.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.748+0000 7f8fd2000640 1 --2- 172.21.15.90:0/3061377039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fcc13c6f0 0x7f8fcc140c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:49.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.748+0000 7f8fd1600640 1 --2- 172.21.15.90:0/3061377039 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fb40051c0 0x7f8fcc0cfb10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8fc000e480 tx=0x7f8fc000e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:49.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.748+0000 7f8fc9600640 1 -- 172.21.15.90:0/3061377039 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc00052e0 con 0x7f8fb40051c0 2024-03-24T01:24:49.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.748+0000 7f8fc9600640 1 -- 172.21.15.90:0/3061377039 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc0018070 con 0x7f8fb40051c0 2024-03-24T01:24:49.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.748+0000 7f8fd2a00640 1 -- 172.21.15.90:0/3061377039 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fcc13eef0 con 0x7f8fb40051c0 2024-03-24T01:24:49.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.748+0000 7f8fc9600640 1 -- 172.21.15.90:0/3061377039 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc0003830 con 0x7f8fb40051c0 2024-03-24T01:24:49.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.748+0000 7f8fd2a00640 1 -- 172.21.15.90:0/3061377039 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fcc13f4b0 con 0x7f8fb40051c0 2024-03-24T01:24:49.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.749+0000 7f8fd2a00640 1 -- 172.21.15.90:0/3061377039 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fcc04f400 con 0x7f8fb40051c0 2024-03-24T01:24:49.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.751+0000 7f8fc9600640 1 -- 172.21.15.90:0/3061377039 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8fc00039d0 con 0x7f8fb40051c0 2024-03-24T01:24:49.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.751+0000 7f8fc9600640 1 --2- 172.21.15.90:0/3061377039 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8fac06bbd0 0x7f8fac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:49.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.752+0000 7f8fc9600640 1 -- 172.21.15.90:0/3061377039 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f8fc008ab00 con 0x7f8fb40051c0 2024-03-24T01:24:49.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.752+0000 7f8fd0c00640 1 --2- 172.21.15.90:0/3061377039 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8fac06bbd0 0x7f8fac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:49.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.753+0000 7f8fd0c00640 1 --2- 172.21.15.90:0/3061377039 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8fac06bbd0 0x7f8fac06e090 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8fb80012b0 tx=0x7f8fb8000ed0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:49.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.756+0000 7f8fc9600640 1 -- 172.21.15.90:0/3061377039 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8fc00592a0 con 0x7f8fb40051c0 2024-03-24T01:24:49.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.951+0000 7f8fd2a00640 1 -- 172.21.15.90:0/3061377039 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f8fcc0d0f80 con 0x7f8fb40051c0 2024-03-24T01:24:49.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.952+0000 7f8fc9600640 1 -- 172.21.15.90:0/3061377039 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7f8fc008ae20 con 0x7f8fb40051c0 2024-03-24T01:24:49.953 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:24:49.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.955+0000 7f8fd2a00640 1 -- 172.21.15.90:0/3061377039 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8fac06bbd0 msgr2=0x7f8fac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:49.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.955+0000 7f8fd2a00640 1 --2- 172.21.15.90:0/3061377039 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8fac06bbd0 0x7f8fac06e090 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8fb80012b0 tx=0x7f8fb8000ed0 comp rx=0 tx=0).stop 2024-03-24T01:24:49.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.955+0000 7f8fd2a00640 1 -- 172.21.15.90:0/3061377039 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fb40051c0 msgr2=0x7f8fcc0cfb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:49.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.955+0000 7f8fd2a00640 1 --2- 172.21.15.90:0/3061377039 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fb40051c0 0x7f8fcc0cfb10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8fc000e480 tx=0x7f8fc000e940 comp rx=0 tx=0).stop 2024-03-24T01:24:49.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.955+0000 7f8fd2a00640 1 -- 172.21.15.90:0/3061377039 shutdown_connections 2024-03-24T01:24:49.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.955+0000 7f8fd2a00640 1 --2- 172.21.15.90:0/3061377039 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8fac06bbd0 0x7f8fac06e090 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:49.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.955+0000 7f8fd2a00640 1 --2- 172.21.15.90:0/3061377039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fcc13c6f0 0x7f8fcc140c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:49.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.955+0000 7f8fd2a00640 1 --2- 172.21.15.90:0/3061377039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8fcc13e710 0x7f8fcc140730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:49.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.955+0000 7f8fd2a00640 1 --2- 172.21.15.90:0/3061377039 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fb40051c0 0x7f8fcc0cfb10 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:49.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.955+0000 7f8fd2a00640 1 -- 172.21.15.90:0/3061377039 >> 172.21.15.90:0/3061377039 conn(0x7f8fcc0b5ea0 msgr2=0x7f8fcc06ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:49.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.955+0000 7f8fd2a00640 1 -- 172.21.15.90:0/3061377039 shutdown_connections 2024-03-24T01:24:49.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:49.955+0000 7f8fd2a00640 1 -- 172.21.15.90:0/3061377039 wait complete. 2024-03-24T01:24:49.974 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:24:50.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.184+0000 7f7efa000640 1 Processor -- start 2024-03-24T01:24:50.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.184+0000 7f7efa000640 1 -- start start 2024-03-24T01:24:50.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.186+0000 7f7efa000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7ef406b7d0 0x7f7ef406e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:50.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.186+0000 7f7efa000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7ef40b3d60 0x7f7ef41618d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:50.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.186+0000 7f7efa000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ef40acf50 con 0x7f7ef415dd50 2024-03-24T01:24:50.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.186+0000 7f7efa000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ef40ad280 con 0x7f7ef406b7d0 2024-03-24T01:24:50.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.186+0000 7f7efa000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ef40b4170 con 0x7f7ef40b3d60 2024-03-24T01:24:50.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.186+0000 7f7eebe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7ef406b7d0 0x7f7ef406e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:50.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.186+0000 7f7eebe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7ef406b7d0 0x7f7ef406e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37824/0 (socket says 172.21.15.90:37824) 2024-03-24T01:24:50.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.186+0000 7f7eebe00640 1 -- 172.21.15.90:0/329205023 learned_addr learned my addr 172.21.15.90:0/329205023 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:50.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.186+0000 7f7ef3400640 1 --1- 172.21.15.90:0/329205023 >> v1:172.21.15.90:6789/0 conn(0x7f7ef415dd50 0x7f7ef406b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:36556/0 (socket says 172.21.15.90:36556) 2024-03-24T01:24:50.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.186+0000 7f7ef3e00640 1 --2- 172.21.15.90:0/329205023 >> v2:172.21.15.120:3300/0 conn(0x7f7ef40b3d60 0x7f7ef41618d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:50.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.186+0000 7f7eebe00640 1 -- 172.21.15.90:0/329205023 >> v2:172.21.15.120:3300/0 conn(0x7f7ef40b3d60 msgr2=0x7f7ef41618d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:50.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.187+0000 7f7eebe00640 1 --2- 172.21.15.90:0/329205023 >> v2:172.21.15.120:3300/0 conn(0x7f7ef40b3d60 0x7f7ef41618d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:50.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.187+0000 7f7eebe00640 1 -- 172.21.15.90:0/329205023 >> v1:172.21.15.90:6789/0 conn(0x7f7ef415dd50 legacy=0x7f7ef406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:50.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.187+0000 7f7eebe00640 1 -- 172.21.15.90:0/329205023 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef415e130 con 0x7f7ef406b7d0 2024-03-24T01:24:50.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.188+0000 7f7eebe00640 1 --2- 172.21.15.90:0/329205023 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ef406b7d0 0x7f7ef406e020 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7ef40b2f20 tx=0x7f7edc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c40b85ce10d07121 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:50.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.188+0000 7f7ef2a00640 1 -- 172.21.15.90:0/329205023 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7edc02e070 con 0x7f7ef406b7d0 2024-03-24T01:24:50.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.188+0000 7f7ef2a00640 1 -- 172.21.15.90:0/329205023 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7edc037070 con 0x7f7ef406b7d0 2024-03-24T01:24:50.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.188+0000 7f7ef2a00640 1 -- 172.21.15.90:0/329205023 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7edc031a00 con 0x7f7ef406b7d0 2024-03-24T01:24:50.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.188+0000 7f7efa000640 1 -- 172.21.15.90:0/329205023 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ef406b7d0 msgr2=0x7f7ef406e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:50.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.188+0000 7f7efa000640 1 --2- 172.21.15.90:0/329205023 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ef406b7d0 0x7f7ef406e020 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7ef40b2f20 tx=0x7f7edc02b880 comp rx=0 tx=0).stop 2024-03-24T01:24:50.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.189+0000 7f7efa000640 1 -- 172.21.15.90:0/329205023 shutdown_connections 2024-03-24T01:24:50.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.189+0000 7f7efa000640 1 --2- 172.21.15.90:0/329205023 >> v2:172.21.15.120:3300/0 conn(0x7f7ef40b3d60 0x7f7ef41618d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:50.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.189+0000 7f7efa000640 1 --2- 172.21.15.90:0/329205023 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ef406b7d0 0x7f7ef406e020 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:50.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.189+0000 7f7efa000640 1 -- 172.21.15.90:0/329205023 >> 172.21.15.90:0/329205023 conn(0x7f7ef40b1770 msgr2=0x7f7ef40b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:50.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.189+0000 7f7efa000640 1 -- 172.21.15.90:0/329205023 shutdown_connections 2024-03-24T01:24:50.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.189+0000 7f7efa000640 1 -- 172.21.15.90:0/329205023 wait complete. 2024-03-24T01:24:50.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.190+0000 7f7efa000640 1 Processor -- start 2024-03-24T01:24:50.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.191+0000 7f7efa000640 1 -- start start 2024-03-24T01:24:50.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.191+0000 7f7efa000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ef406b7d0 0x7f7ef4145260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:50.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.191+0000 7f7efa000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ef40b3d60 0x7f7ef406f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:50.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.191+0000 7f7efa000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ef415dd50 0x7f7ef406f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:50.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.191+0000 7f7efa000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef41625c0 con 0x7f7ef406b7d0 2024-03-24T01:24:50.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.192+0000 7f7efa000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef41622e0 con 0x7f7ef415dd50 2024-03-24T01:24:50.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.192+0000 7f7efa000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef40ad710 con 0x7f7ef40b3d60 2024-03-24T01:24:50.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.192+0000 7f7ef3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ef406b7d0 0x7f7ef4145260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:50.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.192+0000 7f7ef3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ef406b7d0 0x7f7ef4145260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37838/0 (socket says 172.21.15.90:37838) 2024-03-24T01:24:50.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.192+0000 7f7ef3400640 1 -- 172.21.15.90:0/3638077202 learned_addr learned my addr 172.21.15.90:0/3638077202 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:50.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.192+0000 7f7eebe00640 1 --2- 172.21.15.90:0/3638077202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ef40b3d60 0x7f7ef406f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:50.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.192+0000 7f7ef3e00640 1 --2- 172.21.15.90:0/3638077202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ef415dd50 0x7f7ef406f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:50.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.192+0000 7f7ef3400640 1 -- 172.21.15.90:0/3638077202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ef40b3d60 msgr2=0x7f7ef406f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:50.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.192+0000 7f7ef3400640 1 --2- 172.21.15.90:0/3638077202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ef40b3d60 0x7f7ef406f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:50.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.192+0000 7f7ef3400640 1 -- 172.21.15.90:0/3638077202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ef415dd50 msgr2=0x7f7ef406f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:50.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.192+0000 7f7ef3400640 1 --2- 172.21.15.90:0/3638077202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ef415dd50 0x7f7ef406f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:50.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.192+0000 7f7ef3400640 1 -- 172.21.15.90:0/3638077202 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef4160230 con 0x7f7ef406b7d0 2024-03-24T01:24:50.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.192+0000 7f7ef3e00640 1 --2- 172.21.15.90:0/3638077202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ef415dd50 0x7f7ef406f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:50.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.192+0000 7f7ef3400640 1 --2- 172.21.15.90:0/3638077202 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ef406b7d0 0x7f7ef4145260 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7ee400c3c0 tx=0x7f7ee400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:50.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.193+0000 7f7eeb400640 1 -- 172.21.15.90:0/3638077202 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee40074e0 con 0x7f7ef406b7d0 2024-03-24T01:24:50.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.193+0000 7f7eeb400640 1 -- 172.21.15.90:0/3638077202 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ee4007680 con 0x7f7ef406b7d0 2024-03-24T01:24:50.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.193+0000 7f7eeb400640 1 -- 172.21.15.90:0/3638077202 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee4004a00 con 0x7f7ef406b7d0 2024-03-24T01:24:50.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.193+0000 7f7efa000640 1 -- 172.21.15.90:0/3638077202 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ef4160510 con 0x7f7ef406b7d0 2024-03-24T01:24:50.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.193+0000 7f7efa000640 1 -- 172.21.15.90:0/3638077202 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ef4160ad0 con 0x7f7ef406b7d0 2024-03-24T01:24:50.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.194+0000 7f7efa000640 1 -- 172.21.15.90:0/3638077202 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ef4068060 con 0x7f7ef406b7d0 2024-03-24T01:24:50.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.195+0000 7f7eeb400640 1 -- 172.21.15.90:0/3638077202 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7ee401a020 con 0x7f7ef406b7d0 2024-03-24T01:24:50.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.196+0000 7f7eeb400640 1 --2- 172.21.15.90:0/3638077202 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f7ed006bbd0 0x7f7ed006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:50.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.196+0000 7f7eeb400640 1 -- 172.21.15.90:0/3638077202 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f7ee408a4a0 con 0x7f7ef406b7d0 2024-03-24T01:24:50.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.196+0000 7f7eebe00640 1 --2- 172.21.15.90:0/3638077202 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f7ed006bbd0 0x7f7ed006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:50.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.197+0000 7f7eebe00640 1 --2- 172.21.15.90:0/3638077202 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f7ed006bbd0 0x7f7ed006e090 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7edc036000 tx=0x7f7edc008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:50.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.200+0000 7f7eeb400640 1 -- 172.21.15.90:0/3638077202 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7ee4058ce0 con 0x7f7ef406b7d0 2024-03-24T01:24:50.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.441+0000 7f7efa000640 1 -- 172.21.15.90:0/3638077202 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ef40af6a0 con 0x7f7ef406b7d0 2024-03-24T01:24:50.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.442+0000 7f7eeb400640 1 -- 172.21.15.90:0/3638077202 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4967 (secure 0 0 0) 0x7f7ee40585e0 con 0x7f7ef406b7d0 2024-03-24T01:24:50.443 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:24:50.443 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/1898972903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":1898972903},{"type":"v1","addr":"172.21.15.120:6837","nonce":1898972903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6834/775759501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":775759501},{"type":"v1","addr":"172.21.15.90:6834","nonce":775759501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/43630575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":43630575},{"type":"v1","addr":"172.21.15.90:6835","nonce":43630575}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T01:24:10.173532+0000","modified":"2024-03-24T01:24:36.608477+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6836/963928022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":963928022},{"type":"v1","addr":"172.21.15.120:6836","nonce":963928022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4257,"qdb_cluster":[4257]},"id":1}]} 2024-03-24T01:24:50.443 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 8 2024-03-24T01:24:50.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.445+0000 7f7efa000640 1 -- 172.21.15.90:0/3638077202 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f7ed006bbd0 msgr2=0x7f7ed006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:50.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.445+0000 7f7efa000640 1 --2- 172.21.15.90:0/3638077202 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f7ed006bbd0 0x7f7ed006e090 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7edc036000 tx=0x7f7edc008030 comp rx=0 tx=0).stop 2024-03-24T01:24:50.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.445+0000 7f7efa000640 1 -- 172.21.15.90:0/3638077202 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ef406b7d0 msgr2=0x7f7ef4145260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:50.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.445+0000 7f7efa000640 1 --2- 172.21.15.90:0/3638077202 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ef406b7d0 0x7f7ef4145260 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7ee400c3c0 tx=0x7f7ee400c880 comp rx=0 tx=0).stop 2024-03-24T01:24:50.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.445+0000 7f7efa000640 1 -- 172.21.15.90:0/3638077202 shutdown_connections 2024-03-24T01:24:50.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.445+0000 7f7efa000640 1 --2- 172.21.15.90:0/3638077202 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f7ed006bbd0 0x7f7ed006e090 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:50.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.445+0000 7f7efa000640 1 --2- 172.21.15.90:0/3638077202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ef415dd50 0x7f7ef406f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:50.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.445+0000 7f7efa000640 1 --2- 172.21.15.90:0/3638077202 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7ef40b3d60 0x7f7ef406f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:50.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.445+0000 7f7efa000640 1 --2- 172.21.15.90:0/3638077202 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ef406b7d0 0x7f7ef4145260 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:50.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.445+0000 7f7efa000640 1 -- 172.21.15.90:0/3638077202 >> 172.21.15.90:0/3638077202 conn(0x7f7ef40b1770 msgr2=0x7f7ef40b3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:50.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.445+0000 7f7efa000640 1 -- 172.21.15.90:0/3638077202 shutdown_connections 2024-03-24T01:24:50.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.445+0000 7f7efa000640 1 -- 172.21.15.90:0/3638077202 wait complete. 2024-03-24T01:24:50.465 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:24:50.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.673+0000 7fcc38800640 1 Processor -- start 2024-03-24T01:24:50.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.673+0000 7fcc38800640 1 -- start start 2024-03-24T01:24:50.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.674+0000 7fcc38800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcc30155030 0x7fcc3015efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:50.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.674+0000 7fcc38800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcc3006b7d0 0x7fcc300ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:50.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.674+0000 7fcc38800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc3015f500 con 0x7fcc300cdac0 2024-03-24T01:24:50.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.674+0000 7fcc38800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc3015bf70 con 0x7fcc3006b7d0 2024-03-24T01:24:50.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.674+0000 7fcc38800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc3015b360 con 0x7fcc30155030 2024-03-24T01:24:50.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.675+0000 7fcc36800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcc3006b7d0 0x7fcc300ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:50.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.675+0000 7fcc36800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcc3006b7d0 0x7fcc300ae530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37852/0 (socket says 172.21.15.90:37852) 2024-03-24T01:24:50.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.675+0000 7fcc36800640 1 -- 172.21.15.90:0/3661110948 learned_addr learned my addr 172.21.15.90:0/3661110948 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:50.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.675+0000 7fcc36800640 1 -- 172.21.15.90:0/3661110948 >> v2:172.21.15.120:3300/0 conn(0x7fcc30155030 msgr2=0x7fcc3015efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:50.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.675+0000 7fcc35e00640 1 --2- 172.21.15.90:0/3661110948 >> v2:172.21.15.120:3300/0 conn(0x7fcc30155030 0x7fcc3015efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:50.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.675+0000 7fcc36800640 1 --2- 172.21.15.90:0/3661110948 >> v2:172.21.15.120:3300/0 conn(0x7fcc30155030 0x7fcc3015efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:50.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.675+0000 7fcc36800640 1 -- 172.21.15.90:0/3661110948 >> v1:172.21.15.120:6789/0 conn(0x7fcc300cdac0 legacy=0x7fcc3006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:50.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.675+0000 7fcc36800640 1 -- 172.21.15.90:0/3661110948 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc3015b640 con 0x7fcc3006b7d0 2024-03-24T01:24:50.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.675+0000 7fcc35e00640 1 --2- 172.21.15.90:0/3661110948 >> v2:172.21.15.120:3300/0 conn(0x7fcc30155030 0x7fcc3015efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:50.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.676+0000 7fcc36800640 1 --2- 172.21.15.90:0/3661110948 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc3006b7d0 0x7fcc300ae530 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fcc3006ecc0 tx=0x7fcc2c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35bf5eaa76c35737 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:50.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.677+0000 7fcc34a00640 1 -- 172.21.15.90:0/3661110948 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2c02ca80 con 0x7fcc3006b7d0 2024-03-24T01:24:50.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.677+0000 7fcc34a00640 1 -- 172.21.15.90:0/3661110948 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc2c02cc20 con 0x7fcc3006b7d0 2024-03-24T01:24:50.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.677+0000 7fcc34a00640 1 -- 172.21.15.90:0/3661110948 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2c0343f0 con 0x7fcc3006b7d0 2024-03-24T01:24:50.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.677+0000 7fcc38800640 1 -- 172.21.15.90:0/3661110948 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc3006b7d0 msgr2=0x7fcc300ae530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:50.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.677+0000 7fcc38800640 1 --2- 172.21.15.90:0/3661110948 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc3006b7d0 0x7fcc300ae530 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fcc3006ecc0 tx=0x7fcc2c02bee0 comp rx=0 tx=0).stop 2024-03-24T01:24:50.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.678+0000 7fcc38800640 1 -- 172.21.15.90:0/3661110948 shutdown_connections 2024-03-24T01:24:50.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.678+0000 7fcc38800640 1 --2- 172.21.15.90:0/3661110948 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc3006b7d0 0x7fcc300ae530 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:50.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.678+0000 7fcc38800640 1 --2- 172.21.15.90:0/3661110948 >> v2:172.21.15.120:3300/0 conn(0x7fcc30155030 0x7fcc3015efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:50.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.678+0000 7fcc38800640 1 -- 172.21.15.90:0/3661110948 >> 172.21.15.90:0/3661110948 conn(0x7fcc300b1770 msgr2=0x7fcc300b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:50.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.678+0000 7fcc38800640 1 -- 172.21.15.90:0/3661110948 shutdown_connections 2024-03-24T01:24:50.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.678+0000 7fcc38800640 1 -- 172.21.15.90:0/3661110948 wait complete. 2024-03-24T01:24:50.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.679+0000 7fcc38800640 1 Processor -- start 2024-03-24T01:24:50.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.680+0000 7fcc38800640 1 -- start start 2024-03-24T01:24:50.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.680+0000 7fcc38800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc3006b7d0 0x7fcc30148e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:50.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.680+0000 7fcc38800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc300cdac0 0x7fcc30148390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:50.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.680+0000 7fcc38800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc30155030 0x7fcc30152fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:50.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.681+0000 7fcc38800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc30155410 con 0x7fcc300cdac0 2024-03-24T01:24:50.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.681+0000 7fcc38800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3015bc00 con 0x7fcc3006b7d0 2024-03-24T01:24:50.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.681+0000 7fcc38800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3015af70 con 0x7fcc30155030 2024-03-24T01:24:50.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.681+0000 7fcc35400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc300cdac0 0x7fcc30148390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:50.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.681+0000 7fcc35400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc300cdac0 0x7fcc30148390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37858/0 (socket says 172.21.15.90:37858) 2024-03-24T01:24:50.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.681+0000 7fcc35400640 1 -- 172.21.15.90:0/1642756939 learned_addr learned my addr 172.21.15.90:0/1642756939 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:50.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.681+0000 7fcc35e00640 1 --2- 172.21.15.90:0/1642756939 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc3006b7d0 0x7fcc30148e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:50.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.681+0000 7fcc36800640 1 --2- 172.21.15.90:0/1642756939 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc30155030 0x7fcc30152fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:50.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.681+0000 7fcc35400640 1 -- 172.21.15.90:0/1642756939 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc30155030 msgr2=0x7fcc30152fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:50.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.681+0000 7fcc35400640 1 --2- 172.21.15.90:0/1642756939 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc30155030 0x7fcc30152fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:50.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.681+0000 7fcc35400640 1 -- 172.21.15.90:0/1642756939 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc3006b7d0 msgr2=0x7fcc30148e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:50.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.681+0000 7fcc35400640 1 --2- 172.21.15.90:0/1642756939 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc3006b7d0 0x7fcc30148e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:50.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.681+0000 7fcc35400640 1 -- 172.21.15.90:0/1642756939 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc30149340 con 0x7fcc300cdac0 2024-03-24T01:24:50.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.682+0000 7fcc35400640 1 --2- 172.21.15.90:0/1642756939 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc300cdac0 0x7fcc30148390 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fcc2000c400 tx=0x7fcc2000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:50.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.682+0000 7fcc2a000640 1 -- 172.21.15.90:0/1642756939 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc20011790 con 0x7fcc300cdac0 2024-03-24T01:24:50.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.682+0000 7fcc2a000640 1 -- 172.21.15.90:0/1642756939 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc20017070 con 0x7fcc300cdac0 2024-03-24T01:24:50.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.682+0000 7fcc2a000640 1 -- 172.21.15.90:0/1642756939 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc20011bb0 con 0x7fcc300cdac0 2024-03-24T01:24:50.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.682+0000 7fcc38800640 1 -- 172.21.15.90:0/1642756939 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc301488d0 con 0x7fcc300cdac0 2024-03-24T01:24:50.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.682+0000 7fcc38800640 1 -- 172.21.15.90:0/1642756939 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc3015c8a0 con 0x7fcc300cdac0 2024-03-24T01:24:50.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.683+0000 7fcc38800640 1 -- 172.21.15.90:0/1642756939 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc3006f300 con 0x7fcc300cdac0 2024-03-24T01:24:50.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.684+0000 7fcc2a000640 1 -- 172.21.15.90:0/1642756939 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcc2001c460 con 0x7fcc300cdac0 2024-03-24T01:24:50.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.685+0000 7fcc2a000640 1 --2- 172.21.15.90:0/1642756939 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcc1406bbd0 0x7fcc1406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:50.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.685+0000 7fcc2a000640 1 -- 172.21.15.90:0/1642756939 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fcc2008ac70 con 0x7fcc300cdac0 2024-03-24T01:24:50.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.685+0000 7fcc35e00640 1 --2- 172.21.15.90:0/1642756939 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcc1406bbd0 0x7fcc1406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:50.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.686+0000 7fcc35e00640 1 --2- 172.21.15.90:0/1642756939 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcc1406bbd0 0x7fcc1406e090 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400a950 tx=0x7fcc24008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:50.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.689+0000 7fcc2a000640 1 -- 172.21.15.90:0/1642756939 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcc3006f300 con 0x7fcc300cdac0 2024-03-24T01:24:50.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.890+0000 7fcc38800640 1 -- 172.21.15.90:0/1642756939 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc30068060 con 0x7fcc300cdac0 2024-03-24T01:24:50.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.894+0000 7fcc2a000640 1 -- 172.21.15.90:0/1642756939 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18490 (secure 0 0 0) 0x7fcc20059410 con 0x7fcc300cdac0 2024-03-24T01:24:50.895 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:24:50.895 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:24:17.008224+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:24:08.643966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T01:24:09.189904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T01:24:13.027323+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:24:50.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.897+0000 7fcc38800640 1 -- 172.21.15.90:0/1642756939 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcc1406bbd0 msgr2=0x7fcc1406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:50.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.897+0000 7fcc38800640 1 --2- 172.21.15.90:0/1642756939 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcc1406bbd0 0x7fcc1406e090 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400a950 tx=0x7fcc24008040 comp rx=0 tx=0).stop 2024-03-24T01:24:50.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.897+0000 7fcc38800640 1 -- 172.21.15.90:0/1642756939 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc300cdac0 msgr2=0x7fcc30148390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:50.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.897+0000 7fcc38800640 1 --2- 172.21.15.90:0/1642756939 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc300cdac0 0x7fcc30148390 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fcc2000c400 tx=0x7fcc2000c8c0 comp rx=0 tx=0).stop 2024-03-24T01:24:50.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.897+0000 7fcc38800640 1 -- 172.21.15.90:0/1642756939 shutdown_connections 2024-03-24T01:24:50.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.897+0000 7fcc38800640 1 --2- 172.21.15.90:0/1642756939 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcc1406bbd0 0x7fcc1406e090 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:50.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.897+0000 7fcc38800640 1 --2- 172.21.15.90:0/1642756939 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc300cdac0 0x7fcc30148390 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:50.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.897+0000 7fcc38800640 1 --2- 172.21.15.90:0/1642756939 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc30155030 0x7fcc30152fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:50.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.897+0000 7fcc38800640 1 --2- 172.21.15.90:0/1642756939 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc3006b7d0 0x7fcc30148e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:50.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.897+0000 7fcc38800640 1 -- 172.21.15.90:0/1642756939 >> 172.21.15.90:0/1642756939 conn(0x7fcc300b1770 msgr2=0x7fcc300b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:50.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.897+0000 7fcc38800640 1 -- 172.21.15.90:0/1642756939 shutdown_connections 2024-03-24T01:24:50.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:50.897+0000 7fcc38800640 1 -- 172.21.15.90:0/1642756939 wait complete. 2024-03-24T01:24:50.917 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:24:51.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.127+0000 7fc562a00640 1 Processor -- start 2024-03-24T01:24:51.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.127+0000 7fc562a00640 1 -- start start 2024-03-24T01:24:51.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.129+0000 7fc562a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc55c1454e0 0x7fc55c0cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:51.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.129+0000 7fc562a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc55c0cc660 con 0x7fc55c06f000 2024-03-24T01:24:51.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.129+0000 7fc562a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc55c0cc920 con 0x7fc55c0ce360 2024-03-24T01:24:51.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.129+0000 7fc562a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c0cc430 con 0x7fc55c1454e0 2024-03-24T01:24:51.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.129+0000 7fc553400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc55c06f000 0x7fc55c0cdc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:36568/0 (socket says 172.21.15.90:36568) 2024-03-24T01:24:51.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.129+0000 7fc553400640 1 -- 172.21.15.90:0/1706425197 learned_addr learned my addr 172.21.15.90:0/1706425197 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:51.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.129+0000 7fc55be00640 1 --2- 172.21.15.90:0/1706425197 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc55c1454e0 0x7fc55c0cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:51.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.130+0000 7fc55b400640 1 -- 172.21.15.90:0/1706425197 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1975307332 0 0) 0x7fc55c0cc660 con 0x7fc55c06f000 2024-03-24T01:24:51.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.130+0000 7fc55b400640 1 -- 172.21.15.90:0/1706425197 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc540003520 con 0x7fc55c06f000 2024-03-24T01:24:51.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.130+0000 7fc55b400640 1 -- 172.21.15.90:0/1706425197 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3136919162 0 0) 0x7fc540003520 con 0x7fc55c06f000 2024-03-24T01:24:51.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.130+0000 7fc55b400640 1 -- 172.21.15.90:0/1706425197 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc55c1454e0 msgr2=0x7fc55c0cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:51.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.130+0000 7fc55b400640 1 --2- 172.21.15.90:0/1706425197 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc55c1454e0 0x7fc55c0cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:51.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.130+0000 7fc55b400640 1 -- 172.21.15.90:0/1706425197 >> v1:172.21.15.120:6789/0 conn(0x7fc55c0ce360 legacy=0x7fc55c15c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:51.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.130+0000 7fc55b400640 1 -- 172.21.15.90:0/1706425197 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc55c0ccc00 con 0x7fc55c06f000 2024-03-24T01:24:51.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.130+0000 7fc55b400640 1 -- 172.21.15.90:0/1706425197 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7fc544001670 con 0x7fc55c06f000 2024-03-24T01:24:51.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.130+0000 7fc55b400640 1 -- 172.21.15.90:0/1706425197 >> v1:172.21.15.90:6789/0 conn(0x7fc55c06f000 legacy=0x7fc55c0cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:51.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.130+0000 7fc55b400640 1 --2- 172.21.15.90:0/1706425197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5400039e0 0x7fc540003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:51.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.130+0000 7fc55b400640 1 --2- 172.21.15.90:0/1706425197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc540004800 0x7fc540004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:51.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.130+0000 7fc55b400640 1 --2- 172.21.15.90:0/1706425197 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc55c1454e0 0x7fc5400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:51.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.130+0000 7fc55b400640 1 -- 172.21.15.90:0/1706425197 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc54000ba70 con 0x7fc5400039e0 2024-03-24T01:24:51.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.130+0000 7fc55b400640 1 -- 172.21.15.90:0/1706425197 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc54000bbf0 con 0x7fc540004800 2024-03-24T01:24:51.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.130+0000 7fc55b400640 1 -- 172.21.15.90:0/1706425197 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc54000bda0 con 0x7fc55c1454e0 2024-03-24T01:24:51.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.130+0000 7fc55be00640 1 --2- 172.21.15.90:0/1706425197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5400039e0 0x7fc540003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:51.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.131+0000 7fc560a00640 1 --2- 172.21.15.90:0/1706425197 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc55c1454e0 0x7fc5400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:51.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.131+0000 7fc553400640 1 --2- 172.21.15.90:0/1706425197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc540004800 0x7fc540004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:51.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.131+0000 7fc55be00640 1 -- 172.21.15.90:0/1706425197 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc55c1454e0 msgr2=0x7fc5400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:51.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.131+0000 7fc55be00640 1 --2- 172.21.15.90:0/1706425197 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc55c1454e0 0x7fc5400051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:51.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.131+0000 7fc55be00640 1 -- 172.21.15.90:0/1706425197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc540004800 msgr2=0x7fc540004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:51.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.131+0000 7fc55be00640 1 --2- 172.21.15.90:0/1706425197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc540004800 0x7fc540004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:51.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.131+0000 7fc55be00640 1 -- 172.21.15.90:0/1706425197 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc540003520 con 0x7fc5400039e0 2024-03-24T01:24:51.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.132+0000 7fc55be00640 1 --2- 172.21.15.90:0/1706425197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5400039e0 0x7fc540003dc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc54c009a90 tx=0x7fc540005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77af0f6f2d8247be server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:51.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.132+0000 7fc55b400640 1 -- 172.21.15.90:0/1706425197 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54c039070 con 0x7fc5400039e0 2024-03-24T01:24:51.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.132+0000 7fc55b400640 1 -- 172.21.15.90:0/1706425197 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc54c02c650 con 0x7fc5400039e0 2024-03-24T01:24:51.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.132+0000 7fc55b400640 1 -- 172.21.15.90:0/1706425197 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54c009690 con 0x7fc5400039e0 2024-03-24T01:24:51.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.133+0000 7fc562a00640 1 -- 172.21.15.90:0/1706425197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5400039e0 msgr2=0x7fc540003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:51.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.133+0000 7fc562a00640 1 --2- 172.21.15.90:0/1706425197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5400039e0 0x7fc540003dc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc54c009a90 tx=0x7fc540005830 comp rx=0 tx=0).stop 2024-03-24T01:24:51.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.133+0000 7fc560a00640 1 -- 172.21.15.90:0/1706425197 reap_dead start 2024-03-24T01:24:51.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.133+0000 7fc562a00640 1 -- 172.21.15.90:0/1706425197 shutdown_connections 2024-03-24T01:24:51.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.133+0000 7fc562a00640 1 -- 172.21.15.90:0/1706425197 >> 172.21.15.90:0/1706425197 conn(0x7fc55c06b2a0 msgr2=0x7fc55c06b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:51.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.133+0000 7fc562a00640 1 -- 172.21.15.90:0/1706425197 shutdown_connections 2024-03-24T01:24:51.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.133+0000 7fc562a00640 1 -- 172.21.15.90:0/1706425197 wait complete. 2024-03-24T01:24:51.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.135+0000 7fc562a00640 1 Processor -- start 2024-03-24T01:24:51.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.135+0000 7fc562a00640 1 -- start start 2024-03-24T01:24:51.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.136+0000 7fc562a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5400039e0 0x7fc55c138430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:51.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.136+0000 7fc562a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc55c139dd0 0x7fc55c15dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:51.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.136+0000 7fc562a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc55c13c5c0 0x7fc55c13ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:51.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.136+0000 7fc562a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c0cd6e0 con 0x7fc5400039e0 2024-03-24T01:24:51.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.136+0000 7fc562a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c0cd1c0 con 0x7fc55c139dd0 2024-03-24T01:24:51.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.136+0000 7fc562a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c0cd860 con 0x7fc55c13c5c0 2024-03-24T01:24:51.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.136+0000 7fc55be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5400039e0 0x7fc55c138430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:51.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.136+0000 7fc55be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5400039e0 0x7fc55c138430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37876/0 (socket says 172.21.15.90:37876) 2024-03-24T01:24:51.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.136+0000 7fc55be00640 1 -- 172.21.15.90:0/434722710 learned_addr learned my addr 172.21.15.90:0/434722710 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:51.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.136+0000 7fc553400640 1 --2- 172.21.15.90:0/434722710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc55c139dd0 0x7fc55c15dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:51.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.136+0000 7fc560a00640 1 --2- 172.21.15.90:0/434722710 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc55c13c5c0 0x7fc55c13ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:51.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.136+0000 7fc55be00640 1 -- 172.21.15.90:0/434722710 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc55c13c5c0 msgr2=0x7fc55c13ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:51.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.136+0000 7fc55be00640 1 --2- 172.21.15.90:0/434722710 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc55c13c5c0 0x7fc55c13ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:51.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.136+0000 7fc55be00640 1 -- 172.21.15.90:0/434722710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc55c139dd0 msgr2=0x7fc55c15dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:51.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.136+0000 7fc55be00640 1 --2- 172.21.15.90:0/434722710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc55c139dd0 0x7fc55c15dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:51.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.136+0000 7fc55be00640 1 -- 172.21.15.90:0/434722710 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc55c15e3e0 con 0x7fc5400039e0 2024-03-24T01:24:51.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.136+0000 7fc553400640 1 --2- 172.21.15.90:0/434722710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc55c139dd0 0x7fc55c15dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:51.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.137+0000 7fc55be00640 1 --2- 172.21.15.90:0/434722710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5400039e0 0x7fc55c138430 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fc54c009bb0 tx=0x7fc54c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:51.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.137+0000 7fc558c00640 1 -- 172.21.15.90:0/434722710 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54c039050 con 0x7fc5400039e0 2024-03-24T01:24:51.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.137+0000 7fc558c00640 1 -- 172.21.15.90:0/434722710 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc54c045070 con 0x7fc5400039e0 2024-03-24T01:24:51.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.137+0000 7fc562a00640 1 -- 172.21.15.90:0/434722710 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc55c15e6c0 con 0x7fc5400039e0 2024-03-24T01:24:51.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.137+0000 7fc558c00640 1 -- 172.21.15.90:0/434722710 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54c003850 con 0x7fc5400039e0 2024-03-24T01:24:51.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.137+0000 7fc562a00640 1 -- 172.21.15.90:0/434722710 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc55c13e5b0 con 0x7fc5400039e0 2024-03-24T01:24:51.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.138+0000 7fc562a00640 1 -- 172.21.15.90:0/434722710 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc55c073570 con 0x7fc5400039e0 2024-03-24T01:24:51.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.139+0000 7fc558c00640 1 -- 172.21.15.90:0/434722710 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc54c0039f0 con 0x7fc5400039e0 2024-03-24T01:24:51.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.140+0000 7fc558c00640 1 --2- 172.21.15.90:0/434722710 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc53806bbd0 0x7fc53806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:51.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.140+0000 7fc558c00640 1 -- 172.21.15.90:0/434722710 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fc54c0ac8d0 con 0x7fc5400039e0 2024-03-24T01:24:51.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.141+0000 7fc553400640 1 --2- 172.21.15.90:0/434722710 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc53806bbd0 0x7fc53806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:51.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.142+0000 7fc553400640 1 --2- 172.21.15.90:0/434722710 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc53806bbd0 0x7fc53806e090 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc544001eb0 tx=0x7fc544009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:51.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.145+0000 7fc558c00640 1 -- 172.21.15.90:0/434722710 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc54c04e020 con 0x7fc5400039e0 2024-03-24T01:24:51.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.382+0000 7fc562a00640 1 -- 172.21.15.90:0/434722710 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc55c04f4e0 con 0x7fc5400039e0 2024-03-24T01:24:51.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.383+0000 7fc558c00640 1 -- 172.21.15.90:0/434722710 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4967 (secure 0 0 0) 0x7fc54c07b0a0 con 0x7fc5400039e0 2024-03-24T01:24:51.384 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:24:51.384 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/1898972903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":1898972903},{"type":"v1","addr":"172.21.15.120:6837","nonce":1898972903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6834/775759501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":775759501},{"type":"v1","addr":"172.21.15.90:6834","nonce":775759501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/43630575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":43630575},{"type":"v1","addr":"172.21.15.90:6835","nonce":43630575}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T01:24:10.173532+0000","modified":"2024-03-24T01:24:36.608477+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6836/963928022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":963928022},{"type":"v1","addr":"172.21.15.120:6836","nonce":963928022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4257,"qdb_cluster":[4257]},"id":1}]} 2024-03-24T01:24:51.384 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 8 2024-03-24T01:24:51.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.386+0000 7fc562a00640 1 -- 172.21.15.90:0/434722710 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc53806bbd0 msgr2=0x7fc53806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:51.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.386+0000 7fc562a00640 1 --2- 172.21.15.90:0/434722710 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc53806bbd0 0x7fc53806e090 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc544001eb0 tx=0x7fc544009040 comp rx=0 tx=0).stop 2024-03-24T01:24:51.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.386+0000 7fc562a00640 1 -- 172.21.15.90:0/434722710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5400039e0 msgr2=0x7fc55c138430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:51.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.386+0000 7fc562a00640 1 --2- 172.21.15.90:0/434722710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5400039e0 0x7fc55c138430 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fc54c009bb0 tx=0x7fc54c004730 comp rx=0 tx=0).stop 2024-03-24T01:24:51.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.386+0000 7fc562a00640 1 -- 172.21.15.90:0/434722710 shutdown_connections 2024-03-24T01:24:51.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.386+0000 7fc562a00640 1 --2- 172.21.15.90:0/434722710 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc53806bbd0 0x7fc53806e090 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:51.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.386+0000 7fc562a00640 1 --2- 172.21.15.90:0/434722710 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc55c13c5c0 0x7fc55c13ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:51.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.386+0000 7fc562a00640 1 --2- 172.21.15.90:0/434722710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc55c139dd0 0x7fc55c15dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:51.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.386+0000 7fc562a00640 1 --2- 172.21.15.90:0/434722710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5400039e0 0x7fc55c138430 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:51.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.386+0000 7fc562a00640 1 -- 172.21.15.90:0/434722710 >> 172.21.15.90:0/434722710 conn(0x7fc55c06b2a0 msgr2=0x7fc55c06d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:51.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.386+0000 7fc562a00640 1 -- 172.21.15.90:0/434722710 shutdown_connections 2024-03-24T01:24:51.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:51.386+0000 7fc562a00640 1 -- 172.21.15.90:0/434722710 wait complete. 2024-03-24T01:24:51.808 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T01:24:10.173532+0000', 'modified': '2024-03-24T01:24:36.608477+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4257}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4257': {'gid': 4257, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.120:6836/963928022', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6834', 'nonce': 963928022}, {'type': 'v1', 'addr': '172.21.15.120:6836', 'nonce': 963928022}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4257, 'qdb_cluster': [4257]} 2024-03-24T01:24:51.809 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T01:24:51.809 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-24T01:24:52.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.022+0000 7f0dbbe00640 1 Processor -- start 2024-03-24T01:24:52.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.022+0000 7f0dbbe00640 1 -- start start 2024-03-24T01:24:52.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.023+0000 7f0dbbe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0dbc144710 0x7f0dbc06d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:52.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.023+0000 7f0dbbe00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0dbc06dd30 0x7f0dbc071790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:52.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.023+0000 7f0dbbe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0dbc071cd0 0x7f0dbc15f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:52.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.023+0000 7f0dbbe00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dbc072140 con 0x7f0dbc144710 2024-03-24T01:24:52.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.023+0000 7f0dbbe00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dbc0cc230 con 0x7f0dbc06dd30 2024-03-24T01:24:52.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.023+0000 7f0dbbe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dbc06e800 con 0x7f0dbc071cd0 2024-03-24T01:24:52.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.024+0000 7f0dbaa00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0dbc144710 0x7f0dbc06d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:52.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.024+0000 7f0dbaa00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0dbc144710 0x7f0dbc06d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37882/0 (socket says 172.21.15.90:37882) 2024-03-24T01:24:52.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.024+0000 7f0dbaa00640 1 -- 172.21.15.90:0/3285144519 learned_addr learned my addr 172.21.15.90:0/3285144519 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:52.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.024+0000 7f0dbb400640 1 --2- 172.21.15.90:0/3285144519 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0dbc071cd0 0x7f0dbc15f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:52.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.024+0000 7f0db3e00640 1 --2- 172.21.15.90:0/3285144519 >> v2:172.21.15.120:3300/0 conn(0x7f0dbc06dd30 0x7f0dbc071790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:52.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.024+0000 7f0dbaa00640 1 -- 172.21.15.90:0/3285144519 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0dbc071cd0 msgr2=0x7f0dbc15f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:52.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.024+0000 7f0dbaa00640 1 --2- 172.21.15.90:0/3285144519 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0dbc071cd0 0x7f0dbc15f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.024+0000 7f0dbaa00640 1 -- 172.21.15.90:0/3285144519 >> v2:172.21.15.120:3300/0 conn(0x7f0dbc06dd30 msgr2=0x7f0dbc071790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:52.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.024+0000 7f0dbaa00640 1 --2- 172.21.15.90:0/3285144519 >> v2:172.21.15.120:3300/0 conn(0x7f0dbc06dd30 0x7f0dbc071790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.024+0000 7f0dbaa00640 1 -- 172.21.15.90:0/3285144519 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dbc06e240 con 0x7f0dbc144710 2024-03-24T01:24:52.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.025+0000 7f0dbaa00640 1 --2- 172.21.15.90:0/3285144519 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dbc144710 0x7f0dbc06d7f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc160670 tx=0x7f0dac02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43416d96ebfefc96 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:52.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.026+0000 7f0dba000640 1 -- 172.21.15.90:0/3285144519 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dac02e070 con 0x7f0dbc144710 2024-03-24T01:24:52.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.026+0000 7f0dba000640 1 -- 172.21.15.90:0/3285144519 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dac037070 con 0x7f0dbc144710 2024-03-24T01:24:52.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.026+0000 7f0dba000640 1 -- 172.21.15.90:0/3285144519 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dac031a00 con 0x7f0dbc144710 2024-03-24T01:24:52.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.027+0000 7f0dbbe00640 1 -- 172.21.15.90:0/3285144519 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dbc144710 msgr2=0x7f0dbc06d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:52.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.027+0000 7f0dbbe00640 1 --2- 172.21.15.90:0/3285144519 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dbc144710 0x7f0dbc06d7f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc160670 tx=0x7f0dac02b880 comp rx=0 tx=0).stop 2024-03-24T01:24:52.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.027+0000 7f0dbbe00640 1 -- 172.21.15.90:0/3285144519 shutdown_connections 2024-03-24T01:24:52.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.027+0000 7f0dbbe00640 1 --2- 172.21.15.90:0/3285144519 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0dbc071cd0 0x7f0dbc15f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.027+0000 7f0dbbe00640 1 --2- 172.21.15.90:0/3285144519 >> v2:172.21.15.120:3300/0 conn(0x7f0dbc06dd30 0x7f0dbc071790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.027+0000 7f0dbbe00640 1 --2- 172.21.15.90:0/3285144519 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dbc144710 0x7f0dbc06d7f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.027+0000 7f0dbbe00640 1 -- 172.21.15.90:0/3285144519 >> 172.21.15.90:0/3285144519 conn(0x7f0dbc06b0c0 msgr2=0x7f0dbc06b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:52.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.027+0000 7f0dbbe00640 1 -- 172.21.15.90:0/3285144519 shutdown_connections 2024-03-24T01:24:52.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.027+0000 7f0dbbe00640 1 -- 172.21.15.90:0/3285144519 wait complete. 2024-03-24T01:24:52.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.028+0000 7f0dbbe00640 1 Processor -- start 2024-03-24T01:24:52.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.029+0000 7f0dbbe00640 1 -- start start 2024-03-24T01:24:52.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.029+0000 7f0dbbe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0dbc06dd30 0x7f0dbc15b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:52.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.030+0000 7f0dbbe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0dbc071cd0 0x7f0dbc13bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:52.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.030+0000 7f0dbbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dbc144710 0x7f0dbc1f4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:52.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.030+0000 7f0dbbe00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dbc06f670 con 0x7f0dbc144710 2024-03-24T01:24:52.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.030+0000 7f0dbbe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dbc06f4f0 con 0x7f0dbc071cd0 2024-03-24T01:24:52.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.030+0000 7f0dbbe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dbc15fb10 con 0x7f0dbc06dd30 2024-03-24T01:24:52.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.030+0000 7f0dbb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dbc144710 0x7f0dbc1f4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:52.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.030+0000 7f0dbb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dbc144710 0x7f0dbc1f4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37888/0 (socket says 172.21.15.90:37888) 2024-03-24T01:24:52.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.030+0000 7f0dbb400640 1 -- 172.21.15.90:0/863637661 learned_addr learned my addr 172.21.15.90:0/863637661 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:52.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.030+0000 7f0db3e00640 1 --2- 172.21.15.90:0/863637661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0dbc071cd0 0x7f0dbc13bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:52.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.030+0000 7f0dbaa00640 1 --2- 172.21.15.90:0/863637661 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0dbc06dd30 0x7f0dbc15b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:52.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.030+0000 7f0dbb400640 1 -- 172.21.15.90:0/863637661 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0dbc06dd30 msgr2=0x7f0dbc15b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:52.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.030+0000 7f0dbb400640 1 --2- 172.21.15.90:0/863637661 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0dbc06dd30 0x7f0dbc15b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.030+0000 7f0dbb400640 1 -- 172.21.15.90:0/863637661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0dbc071cd0 msgr2=0x7f0dbc13bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:52.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.030+0000 7f0dbb400640 1 --2- 172.21.15.90:0/863637661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0dbc071cd0 0x7f0dbc13bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.030+0000 7f0dbb400640 1 -- 172.21.15.90:0/863637661 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dbc1f2350 con 0x7f0dbc144710 2024-03-24T01:24:52.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.030+0000 7f0db3e00640 1 --2- 172.21.15.90:0/863637661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0dbc071cd0 0x7f0dbc13bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:52.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.030+0000 7f0dbaa00640 1 --2- 172.21.15.90:0/863637661 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0dbc06dd30 0x7f0dbc15b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:52.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.031+0000 7f0dbb400640 1 --2- 172.21.15.90:0/863637661 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dbc144710 0x7f0dbc1f4830 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0db4007e40 tx=0x7f0db400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:52.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.031+0000 7f0db2a00640 1 -- 172.21.15.90:0/863637661 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db4014620 con 0x7f0dbc144710 2024-03-24T01:24:52.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.031+0000 7f0db2a00640 1 -- 172.21.15.90:0/863637661 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db4019070 con 0x7f0dbc144710 2024-03-24T01:24:52.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.031+0000 7f0db2a00640 1 -- 172.21.15.90:0/863637661 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db4014a20 con 0x7f0dbc144710 2024-03-24T01:24:52.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.031+0000 7f0dbbe00640 1 -- 172.21.15.90:0/863637661 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dbc1f2610 con 0x7f0dbc144710 2024-03-24T01:24:52.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.031+0000 7f0dbbe00640 1 -- 172.21.15.90:0/863637661 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dbc1f2bd0 con 0x7f0dbc144710 2024-03-24T01:24:52.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.033+0000 7f0db2a00640 1 -- 172.21.15.90:0/863637661 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0db4014bc0 con 0x7f0dbc144710 2024-03-24T01:24:52.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.035+0000 7f0db2a00640 1 --2- 172.21.15.90:0/863637661 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0d9806bc00 0x7f0d9806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:52.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.035+0000 7f0db2a00640 1 -- 172.21.15.90:0/863637661 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f0db408b6b0 con 0x7f0dbc144710 2024-03-24T01:24:52.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.035+0000 7f0dc4977740 1 Processor -- start 2024-03-24T01:24:52.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.035+0000 7f0dc4977740 1 -- start start 2024-03-24T01:24:52.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.036+0000 7f0dbaa00640 1 --2- 172.21.15.90:0/863637661 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0d9806bc00 0x7f0d9806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:52.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.036+0000 7f0dc4977740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557bbcbbddb0 0x557bbcbbe190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:52.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.036+0000 7f0dc4977740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557bbcb8ddd0 0x557bbcbbb020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:52.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.036+0000 7f0dc4977740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557bbcb91930 0x557bbcbbb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:52.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.036+0000 7f0dc4977740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bbcaf7c90 con 0x557bbcb91930 2024-03-24T01:24:52.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.036+0000 7f0dc4977740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bbcaa0980 con 0x557bbcb8ddd0 2024-03-24T01:24:52.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.036+0000 7f0dc4977740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x557bbcaf3d30 con 0x557bbcbbddb0 2024-03-24T01:24:52.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.037+0000 7f0dbb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557bbcb91930 0x557bbcbbb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:52.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.037+0000 7f0dbb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557bbcb91930 0x557bbcbbb560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37904/0 (socket says 172.21.15.90:37904) 2024-03-24T01:24:52.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.037+0000 7f0dbb400640 1 -- 172.21.15.90:0/231384841 learned_addr learned my addr 172.21.15.90:0/231384841 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:52.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.037+0000 7f0dbaa00640 1 --2- 172.21.15.90:0/863637661 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0d9806bc00 0x7f0d9806e0c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0dac036000 tx=0x7f0dac008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:52.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.037+0000 7f0dbb400640 1 -- 172.21.15.90:0/231384841 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557bbcbbddb0 msgr2=0x557bbcbbe190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:52.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.037+0000 7f0dbb400640 1 --2- 172.21.15.90:0/231384841 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557bbcbbddb0 0x557bbcbbe190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.037+0000 7f0dbb400640 1 -- 172.21.15.90:0/231384841 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557bbcb8ddd0 msgr2=0x557bbcbbb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:52.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.037+0000 7f0dbb400640 1 --2- 172.21.15.90:0/231384841 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557bbcb8ddd0 0x557bbcbbb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.037+0000 7f0dbb400640 1 -- 172.21.15.90:0/231384841 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557bbcab8510 con 0x557bbcb91930 2024-03-24T01:24:52.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.037+0000 7f0dbb400640 1 --2- 172.21.15.90:0/231384841 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557bbcb91930 0x557bbcbbb560 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0db400c040 tx=0x7f0db4059e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb0bcea69c2d04ec server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:52.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.037+0000 7f0dab400640 1 -- 172.21.15.90:0/231384841 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db405d8a0 con 0x557bbcb91930 2024-03-24T01:24:52.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.037+0000 7f0dab400640 1 -- 172.21.15.90:0/231384841 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db4061e60 con 0x557bbcb91930 2024-03-24T01:24:52.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.037+0000 7f0dab400640 1 -- 172.21.15.90:0/231384841 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db4055dc0 con 0x557bbcb91930 2024-03-24T01:24:52.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.038+0000 7f0dc4977740 1 -- 172.21.15.90:0/231384841 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557bbcb91930 msgr2=0x557bbcbbb560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:52.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.038+0000 7f0dc4977740 1 --2- 172.21.15.90:0/231384841 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557bbcb91930 0x557bbcbbb560 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0db400c040 tx=0x7f0db4059e50 comp rx=0 tx=0).stop 2024-03-24T01:24:52.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.038+0000 7f0dc4977740 1 -- 172.21.15.90:0/231384841 shutdown_connections 2024-03-24T01:24:52.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.038+0000 7f0dc4977740 1 --2- 172.21.15.90:0/231384841 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557bbcb91930 0x557bbcbbb560 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.038+0000 7f0dc4977740 1 --2- 172.21.15.90:0/231384841 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557bbcb8ddd0 0x557bbcbbb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.038+0000 7f0dc4977740 1 --2- 172.21.15.90:0/231384841 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557bbcbbddb0 0x557bbcbbe190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.038+0000 7f0dc4977740 1 -- 172.21.15.90:0/231384841 >> 172.21.15.90:0/231384841 conn(0x557bbcb90040 msgr2=0x557bbcbd8bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:52.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.038+0000 7f0dc4977740 1 -- 172.21.15.90:0/231384841 shutdown_connections 2024-03-24T01:24:52.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.038+0000 7f0dc4977740 1 -- 172.21.15.90:0/231384841 wait complete. 2024-03-24T01:24:52.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.039+0000 7f0dc4977740 1 -- start start 2024-03-24T01:24:52.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.039+0000 7f0dc4977740 1 Processor -- start 2024-03-24T01:24:52.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.040+0000 7f0dc4977740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557bbcb91930 0x557bbcbac2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:52.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.040+0000 7f0dc4977740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557bbcbbddb0 0x557bbcbc0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:52.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.040+0000 7f0dc4977740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557bbcbc0660 0x557bbcbb2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:52.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.040+0000 7f0dc4977740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bbcb92ae0 con 0x557bbcb91930 2024-03-24T01:24:52.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.040+0000 7f0dc4977740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bbcab3dd0 con 0x557bbcbc0660 2024-03-24T01:24:52.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.040+0000 7f0dc4977740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x557bbcb91ed0 con 0x557bbcbbddb0 2024-03-24T01:24:52.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.040+0000 7f0dc4977740 10 client.0 _ll_register_callbacks cb 0x557bbc80bac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T01:24:52.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.040+0000 7f0dbaa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557bbcb91930 0x557bbcbac2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:52.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.040+0000 7f0dbaa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557bbcb91930 0x557bbcbac2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37920/0 (socket says 172.21.15.90:37920) 2024-03-24T01:24:52.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.040+0000 7f0dbaa00640 1 -- 172.21.15.90:0/3916238996 learned_addr learned my addr 172.21.15.90:0/3916238996 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:52.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.040+0000 7f0db3e00640 1 --2- 172.21.15.90:0/3916238996 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557bbcbbddb0 0x557bbcbc0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:52.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.041+0000 7f0dbb400640 1 --2- 172.21.15.90:0/3916238996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557bbcbc0660 0x557bbcbb2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:52.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.041+0000 7f0dbaa00640 1 -- 172.21.15.90:0/3916238996 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557bbcbbddb0 msgr2=0x557bbcbc0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:52.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.041+0000 7f0dbaa00640 1 --2- 172.21.15.90:0/3916238996 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557bbcbbddb0 0x557bbcbc0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.041+0000 7f0dbaa00640 1 -- 172.21.15.90:0/3916238996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557bbcbc0660 msgr2=0x557bbcbb2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:52.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.041+0000 7f0dbaa00640 1 --2- 172.21.15.90:0/3916238996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557bbcbc0660 0x557bbcbb2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.041+0000 7f0dbaa00640 1 -- 172.21.15.90:0/3916238996 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557bbc80c2d0 con 0x557bbcb91930 2024-03-24T01:24:52.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.041+0000 7f0dbb400640 1 --2- 172.21.15.90:0/3916238996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557bbcbc0660 0x557bbcbb2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:52.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.041+0000 7f0dbaa00640 1 --2- 172.21.15.90:0/3916238996 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557bbcb91930 0x557bbcbac2d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0dac02be90 tx=0x7f0dac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10be6622454d804d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:52.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.041+0000 7f0d9fe00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-03-24T01:24:52.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.041+0000 7f0d9fe00640 1 -- 172.21.15.90:0/3916238996 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dac04b070 con 0x557bbcb91930 2024-03-24T01:24:52.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.041+0000 7f0dc4977740 1 -- 172.21.15.90:0/3916238996 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0db4061e60 con 0x557bbcb91930 2024-03-24T01:24:52.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.041+0000 7f0d9fe00640 1 -- 172.21.15.90:0/3916238996 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f0dac045800 con 0x557bbcb91930 2024-03-24T01:24:52.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.042+0000 7f0d9fe00640 1 -- 172.21.15.90:0/3916238996 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0dac045a00 con 0x557bbcb91930 2024-03-24T01:24:52.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.042+0000 7f0dc4977740 10 client.4797 fetch_fsmap learned FSMap version 8 2024-03-24T01:24:52.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.042+0000 7f0dc4977740 1 -- 172.21.15.90:0/3916238996 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x557bbc8d66b0 con 0x557bbcb91930 2024-03-24T01:24:52.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.042+0000 7f0d9fe00640 1 -- 172.21.15.90:0/3916238996 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db4061e60 con 0x557bbcb91930 2024-03-24T01:24:52.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.042+0000 7f0d9fe00640 1 -- 172.21.15.90:0/3916238996 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 8) v1 ==== 3322+0+0 (secure 0 0 0) 0x7f0dac045ba0 con 0x557bbcb91930 2024-03-24T01:24:52.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.042+0000 7f0dc4977740 10 client.4797 fetch_fsmap finished waiting for FSMap version 8 2024-03-24T01:24:52.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.042+0000 7f0dc4977740 10 client.4797 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-24T01:24:52.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.042+0000 7f0dc4977740 20 client.4797 populate_metadata read hostname 'smithi090' 2024-03-24T01:24:52.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.042+0000 7f0dc4977740 1 --2- 172.21.15.90:0/3916238996 >> [v2:172.21.15.120:6834/963928022,v1:172.21.15.120:6836/963928022] conn(0x557bbcbf0fe0 0x557bbcbfb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:52.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.042+0000 7f0dc4977740 4 client.4797 mds_command: new command op to 4257 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T01:24:52.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.042+0000 7f0dc4977740 1 -- 172.21.15.90:0/3916238996 --> [v2:172.21.15.120:6834/963928022,v1:172.21.15.120:6836/963928022] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557bbca99520 con 0x557bbcbf0fe0 2024-03-24T01:24:52.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.043+0000 7f0db3e00640 1 --2- 172.21.15.90:0/3916238996 >> [v2:172.21.15.120:6834/963928022,v1:172.21.15.120:6836/963928022] conn(0x557bbcbf0fe0 0x557bbcbfb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:52.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.044+0000 7f0db3e00640 1 --2- 172.21.15.90:0/3916238996 >> [v2:172.21.15.120:6834/963928022,v1:172.21.15.120:6836/963928022] conn(0x557bbcbf0fe0 0x557bbcbfb170 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c376a3a3360b0f1a server_cookie=d82ba6ae52ece82b in_seq=0 out_seq=0 2024-03-24T01:24:52.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.044+0000 7f0d9fe00640 10 client.4797 ms_handle_connect on v2:172.21.15.120:6834/963928022 2024-03-24T01:24:52.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.046+0000 7f0d9fe00640 1 -- 172.21.15.90:0/3916238996 <== mds.0 v2:172.21.15.120:6834/963928022 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557bbca99520 con 0x557bbcbf0fe0 2024-03-24T01:24:52.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.046+0000 7f0d9fe00640 10 client.4797 handle_command_reply: tid=0 2024-03-24T01:24:52.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.046+0000 7f0dc4977740 1 client.4797 shutdown 2024-03-24T01:24:52.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.046+0000 7f0dc4977740 1 -- 172.21.15.90:0/3916238996 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557bbcb91930 msgr2=0x557bbcbac2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:52.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.046+0000 7f0dc4977740 1 --2- 172.21.15.90:0/3916238996 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557bbcb91930 0x557bbcbac2d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0dac02be90 tx=0x7f0dac0046c0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.046+0000 7f0dc4977740 1 -- 172.21.15.90:0/3916238996 shutdown_connections 2024-03-24T01:24:52.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.046+0000 7f0dc4977740 1 --2- 172.21.15.90:0/3916238996 >> [v2:172.21.15.120:6834/963928022,v1:172.21.15.120:6836/963928022] conn(0x557bbcbf0fe0 0x557bbcbfb170 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.046+0000 7f0dc4977740 1 --2- 172.21.15.90:0/3916238996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557bbcbc0660 0x557bbcbb2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.046+0000 7f0d9fe00640 0 client.4797 ms_handle_reset on v2:172.21.15.120:6834/963928022 2024-03-24T01:24:52.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.046+0000 7f0dc4977740 1 --2- 172.21.15.90:0/3916238996 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557bbcbbddb0 0x557bbcbc0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.046+0000 7f0dc4977740 1 --2- 172.21.15.90:0/3916238996 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557bbcb91930 0x557bbcbac2d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.046+0000 7f0dc4977740 1 -- 172.21.15.90:0/3916238996 >> 172.21.15.90:0/3916238996 conn(0x557bbcb8ddd0 msgr2=0x557bbcb92c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:52.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.046+0000 7f0dc4977740 1 -- 172.21.15.90:0/3916238996 shutdown_connections 2024-03-24T01:24:52.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.046+0000 7f0dc4977740 1 -- 172.21.15.90:0/3916238996 wait complete. 2024-03-24T01:24:52.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.046+0000 7f0dc4977740 20 client.4797 trim_cache size 0 max 16384 2024-03-24T01:24:52.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.064+0000 7f0dc4977740 1 Processor -- start 2024-03-24T01:24:52.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.064+0000 7f0dc4977740 1 -- start start 2024-03-24T01:24:52.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.065+0000 7f0dc4977740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557bbcb91930 0x557bbcb92a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:52.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.065+0000 7f0dc4977740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557bbcbbddb0 0x557bbcbd6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:52.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.065+0000 7f0dc4977740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557bbcbc0660 0x557bbcbd88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:52.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.065+0000 7f0dc4977740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dbc1fc150 con 0x557bbcbc0660 2024-03-24T01:24:52.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.065+0000 7f0dc4977740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dbc0b4660 con 0x557bbcbbddb0 2024-03-24T01:24:52.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.065+0000 7f0dc4977740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dbc1fbb50 con 0x557bbcb91930 2024-03-24T01:24:52.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.065+0000 7f0dbb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557bbcbc0660 0x557bbcbd88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:52.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.065+0000 7f0dbb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557bbcbc0660 0x557bbcbd88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37928/0 (socket says 172.21.15.90:37928) 2024-03-24T01:24:52.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.065+0000 7f0dbb400640 1 -- 172.21.15.90:0/2609958355 learned_addr learned my addr 172.21.15.90:0/2609958355 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:52.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.065+0000 7f0db3e00640 1 --2- 172.21.15.90:0/2609958355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557bbcbbddb0 0x557bbcbd6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:52.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.065+0000 7f0dbb400640 1 -- 172.21.15.90:0/2609958355 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557bbcb91930 msgr2=0x557bbcb92a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:52.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.065+0000 7f0dbb400640 1 --2- 172.21.15.90:0/2609958355 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557bbcb91930 0x557bbcb92a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.065+0000 7f0dbb400640 1 -- 172.21.15.90:0/2609958355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557bbcbbddb0 msgr2=0x557bbcbd6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:52.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.065+0000 7f0dbb400640 1 --2- 172.21.15.90:0/2609958355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557bbcbbddb0 0x557bbcbd6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.066+0000 7f0dbb400640 1 -- 172.21.15.90:0/2609958355 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557bbcbc0450 con 0x557bbcbc0660 2024-03-24T01:24:52.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.066+0000 7f0db3e00640 1 --2- 172.21.15.90:0/2609958355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557bbcbbddb0 0x557bbcbd6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:52.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.066+0000 7f0dbb400640 1 --2- 172.21.15.90:0/2609958355 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557bbcbc0660 0x557bbcbd88e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0db4059450 tx=0x7f0db4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c3b629d8ca27f3b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:52.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.066+0000 7f0daaa00640 1 -- 172.21.15.90:0/2609958355 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db4076070 con 0x557bbcbc0660 2024-03-24T01:24:52.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.066+0000 7f0daaa00640 1 -- 172.21.15.90:0/2609958355 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db4061e60 con 0x557bbcbc0660 2024-03-24T01:24:52.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.066+0000 7f0daaa00640 1 -- 172.21.15.90:0/2609958355 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db405d8a0 con 0x557bbcbc0660 2024-03-24T01:24:52.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.067+0000 7f0dc4977740 1 -- 172.21.15.90:0/2609958355 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557bbcbc0660 msgr2=0x557bbcbd88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:52.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.067+0000 7f0dc4977740 1 --2- 172.21.15.90:0/2609958355 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557bbcbc0660 0x557bbcbd88e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0db4059450 tx=0x7f0db4016000 comp rx=0 tx=0).stop 2024-03-24T01:24:52.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.067+0000 7f0dc4977740 1 -- 172.21.15.90:0/2609958355 shutdown_connections 2024-03-24T01:24:52.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.067+0000 7f0dc4977740 1 --2- 172.21.15.90:0/2609958355 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557bbcbc0660 0x557bbcbd88e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.067+0000 7f0dc4977740 1 --2- 172.21.15.90:0/2609958355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557bbcbbddb0 0x557bbcbd6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.067+0000 7f0dc4977740 1 --2- 172.21.15.90:0/2609958355 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557bbcb91930 0x557bbcb92a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.067+0000 7f0dc4977740 1 -- 172.21.15.90:0/2609958355 >> 172.21.15.90:0/2609958355 conn(0x557bbcb8ddd0 msgr2=0x557bbcb8e260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:52.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.067+0000 7f0dc4977740 1 -- 172.21.15.90:0/2609958355 shutdown_connections 2024-03-24T01:24:52.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.067+0000 7f0dc4977740 1 -- 172.21.15.90:0/2609958355 wait complete. 2024-03-24T01:24:52.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.067+0000 7f0dc4977740 1 -- start start 2024-03-24T01:24:52.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.068+0000 7f0dc4977740 1 Processor -- start 2024-03-24T01:24:52.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.068+0000 7f0dc4977740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557bbcb91930 0x557bbcc4da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:52.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.068+0000 7f0dc4977740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557bbcbbddb0 0x557bbcc50660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:52.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.068+0000 7f0dc4977740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557bbcbc0660 0x557bbcbce550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:52.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.068+0000 7f0dc4977740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bbcbf0e60 con 0x557bbcbc0660 2024-03-24T01:24:52.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.068+0000 7f0dc4977740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bbcbc1730 con 0x557bbcb91930 2024-03-24T01:24:52.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.068+0000 7f0dc4977740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x557bbcbf13d0 con 0x557bbcbbddb0 2024-03-24T01:24:52.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.069+0000 7f0dc4977740 10 client.0 _ll_register_callbacks cb 0x7f0dac0096f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T01:24:52.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.069+0000 7f0dbb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557bbcbc0660 0x557bbcbce550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:52.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.069+0000 7f0dbb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557bbcbc0660 0x557bbcbce550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37930/0 (socket says 172.21.15.90:37930) 2024-03-24T01:24:52.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.069+0000 7f0dbb400640 1 -- 172.21.15.90:0/979588967 learned_addr learned my addr 172.21.15.90:0/979588967 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:52.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.069+0000 7f0dbaa00640 1 --2- 172.21.15.90:0/979588967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557bbcb91930 0x557bbcc4da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:52.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.069+0000 7f0dbb400640 1 -- 172.21.15.90:0/979588967 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557bbcbbddb0 msgr2=0x557bbcc50660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:52.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.069+0000 7f0db3e00640 1 --2- 172.21.15.90:0/979588967 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557bbcbbddb0 0x557bbcc50660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:52.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.069+0000 7f0dbb400640 1 --2- 172.21.15.90:0/979588967 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557bbcbbddb0 0x557bbcc50660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.069+0000 7f0dbb400640 1 -- 172.21.15.90:0/979588967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557bbcb91930 msgr2=0x557bbcc4da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:52.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.069+0000 7f0dbb400640 1 --2- 172.21.15.90:0/979588967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557bbcb91930 0x557bbcc4da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.069+0000 7f0dbb400640 1 -- 172.21.15.90:0/979588967 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557bbcc43ec0 con 0x557bbcbc0660 2024-03-24T01:24:52.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.069+0000 7f0dbaa00640 1 --2- 172.21.15.90:0/979588967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557bbcb91930 0x557bbcc4da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:52.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.069+0000 7f0dbb400640 1 --2- 172.21.15.90:0/979588967 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557bbcbc0660 0x557bbcbce550 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0db400e9b0 tx=0x7f0db408b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5ade56f60afdcd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:52.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.069+0000 7f0d9fe00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-03-24T01:24:52.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.070+0000 7f0d9fe00640 1 -- 172.21.15.90:0/979588967 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db4076030 con 0x557bbcbc0660 2024-03-24T01:24:52.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.070+0000 7f0d9fe00640 1 -- 172.21.15.90:0/979588967 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f0db4089430 con 0x557bbcbc0660 2024-03-24T01:24:52.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.070+0000 7f0dc4977740 1 -- 172.21.15.90:0/979588967 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0db4061e60 con 0x557bbcbc0660 2024-03-24T01:24:52.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.070+0000 7f0d9fe00640 1 -- 172.21.15.90:0/979588967 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0db4055dc0 con 0x557bbcbc0660 2024-03-24T01:24:52.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.070+0000 7f0dc4977740 10 client.4803 fetch_fsmap learned FSMap version 8 2024-03-24T01:24:52.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.070+0000 7f0dc4977740 1 -- 172.21.15.90:0/979588967 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x557bbcc4d560 con 0x557bbcbc0660 2024-03-24T01:24:52.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.070+0000 7f0d9fe00640 1 -- 172.21.15.90:0/979588967 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db4061e60 con 0x557bbcbc0660 2024-03-24T01:24:52.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.070+0000 7f0d9fe00640 1 -- 172.21.15.90:0/979588967 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 8) v1 ==== 3322+0+0 (secure 0 0 0) 0x7f0db40599b0 con 0x557bbcbc0660 2024-03-24T01:24:52.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.070+0000 7f0dc4977740 10 client.4803 fetch_fsmap finished waiting for FSMap version 8 2024-03-24T01:24:52.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.070+0000 7f0dc4977740 10 client.4803 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-24T01:24:52.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.070+0000 7f0dc4977740 20 client.4803 populate_metadata read hostname 'smithi090' 2024-03-24T01:24:52.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.070+0000 7f0dc4977740 1 --2- 172.21.15.90:0/979588967 >> [v2:172.21.15.120:6834/963928022,v1:172.21.15.120:6836/963928022] conn(0x557bbcbcf1e0 0x557bbcbcea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:52.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.070+0000 7f0dc4977740 4 client.4803 mds_command: new command op to 4257 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T01:24:52.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.070+0000 7f0dc4977740 1 -- 172.21.15.90:0/979588967 --> [v2:172.21.15.120:6834/963928022,v1:172.21.15.120:6836/963928022] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557bbcb051d0 con 0x557bbcbcf1e0 2024-03-24T01:24:52.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.071+0000 7f0dbaa00640 1 --2- 172.21.15.90:0/979588967 >> [v2:172.21.15.120:6834/963928022,v1:172.21.15.120:6836/963928022] conn(0x557bbcbcf1e0 0x557bbcbcea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:52.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.072+0000 7f0dbaa00640 1 --2- 172.21.15.90:0/979588967 >> [v2:172.21.15.120:6834/963928022,v1:172.21.15.120:6836/963928022] conn(0x557bbcbcf1e0 0x557bbcbcea90 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d5189e9732622c9 server_cookie=6ea8577da1703c87 in_seq=0 out_seq=0 2024-03-24T01:24:52.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.072+0000 7f0d9fe00640 10 client.4803 ms_handle_connect on v2:172.21.15.120:6834/963928022 2024-03-24T01:24:52.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.072+0000 7f0d9fe00640 1 -- 172.21.15.90:0/979588967 <== mds.0 v2:172.21.15.120:6834/963928022 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x557bbcb051d0 con 0x557bbcbcf1e0 2024-03-24T01:24:52.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.072+0000 7f0d9fe00640 10 client.4803 handle_command_reply: tid=0 2024-03-24T01:24:52.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.072+0000 7f0dc4977740 1 client.4803 shutdown 2024-03-24T01:24:52.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.072+0000 7f0dc4977740 1 -- 172.21.15.90:0/979588967 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557bbcbc0660 msgr2=0x557bbcbce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:52.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.072+0000 7f0dc4977740 1 --2- 172.21.15.90:0/979588967 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557bbcbc0660 0x557bbcbce550 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0db400e9b0 tx=0x7f0db408b040 comp rx=0 tx=0).stop 2024-03-24T01:24:52.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.073+0000 7f0dc4977740 1 -- 172.21.15.90:0/979588967 shutdown_connections 2024-03-24T01:24:52.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.073+0000 7f0dc4977740 1 --2- 172.21.15.90:0/979588967 >> [v2:172.21.15.120:6834/963928022,v1:172.21.15.120:6836/963928022] conn(0x557bbcbcf1e0 0x557bbcbcea90 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.073+0000 7f0dc4977740 1 --2- 172.21.15.90:0/979588967 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557bbcbc0660 0x557bbcbce550 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.073+0000 7f0dc4977740 1 --2- 172.21.15.90:0/979588967 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557bbcbbddb0 0x557bbcc50660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.073+0000 7f0dc4977740 1 --2- 172.21.15.90:0/979588967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557bbcb91930 0x557bbcc4da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.073+0000 7f0d9fe00640 0 client.4803 ms_handle_reset on v2:172.21.15.120:6834/963928022 2024-03-24T01:24:52.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.073+0000 7f0dc4977740 1 -- 172.21.15.90:0/979588967 >> 172.21.15.90:0/979588967 conn(0x557bbcb8ddd0 msgr2=0x557bbcb92a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:52.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.073+0000 7f0dc4977740 1 -- 172.21.15.90:0/979588967 shutdown_connections 2024-03-24T01:24:52.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.073+0000 7f0dc4977740 1 -- 172.21.15.90:0/979588967 wait complete. 2024-03-24T01:24:52.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.073+0000 7f0dc4977740 20 client.4803 trim_cache size 0 max 16384 2024-03-24T01:24:52.077 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-24T01:24:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "24712165-9839-4fcc-8033-988386efea74", 2024-03-24T01:24:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 0, 2024-03-24T01:24:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 4257, 2024-03-24T01:24:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-03-24T01:24:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-03-24T01:24:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-03-24T01:24:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 41.345298550000003, 2024-03-24T01:24:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 8, 2024-03-24T01:24:52.078 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 25, 2024-03-24T01:24:52.078 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 25, 2024-03-24T01:24:52.078 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 46.402184138000003 2024-03-24T01:24:52.078 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-24T01:24:52.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.073+0000 7f0dbbe00640 1 -- 172.21.15.90:0/863637661 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0d9806bc00 msgr2=0x7f0d9806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:52.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.073+0000 7f0dbbe00640 1 --2- 172.21.15.90:0/863637661 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0d9806bc00 0x7f0d9806e0c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0dac036000 tx=0x7f0dac008030 comp rx=0 tx=0).stop 2024-03-24T01:24:52.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.073+0000 7f0dbbe00640 1 -- 172.21.15.90:0/863637661 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dbc144710 msgr2=0x7f0dbc1f4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:52.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.073+0000 7f0dbbe00640 1 --2- 172.21.15.90:0/863637661 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dbc144710 0x7f0dbc1f4830 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0db4007e40 tx=0x7f0db400e4b0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.073+0000 7f0dbbe00640 1 -- 172.21.15.90:0/863637661 shutdown_connections 2024-03-24T01:24:52.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.073+0000 7f0dbbe00640 1 --2- 172.21.15.90:0/863637661 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0d9806bc00 0x7f0d9806e0c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.073+0000 7f0dbbe00640 1 --2- 172.21.15.90:0/863637661 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dbc144710 0x7f0dbc1f4830 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.073+0000 7f0dbbe00640 1 --2- 172.21.15.90:0/863637661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0dbc071cd0 0x7f0dbc13bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.073+0000 7f0dbbe00640 1 --2- 172.21.15.90:0/863637661 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0dbc06dd30 0x7f0dbc15b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.073+0000 7f0dbbe00640 1 -- 172.21.15.90:0/863637661 >> 172.21.15.90:0/863637661 conn(0x7f0dbc06b0c0 msgr2=0x7f0dbc144af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:52.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.074+0000 7f0dbbe00640 1 -- 172.21.15.90:0/863637661 shutdown_connections 2024-03-24T01:24:52.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.074+0000 7f0dbbe00640 1 -- 172.21.15.90:0/863637661 wait complete. 2024-03-24T01:24:52.089 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-24T01:24:52.100 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-24T01:24:52.100 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-03-24T01:24:52.100 INFO:tasks.ceph_fuse:client.0 config is {} 2024-03-24T01:24:52.100 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-24T01:24:52.101 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T01:24:52.101 INFO:tasks.ceph_fuse:client.2 config is {} 2024-03-24T01:24:52.101 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-24T01:24:52.101 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-24T01:24:52.101 INFO:tasks.ceph_fuse:client.1 config is {} 2024-03-24T01:24:52.101 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-24T01:24:52.101 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-24T01:24:52.101 INFO:tasks.ceph_fuse:client.3 config is {} 2024-03-24T01:24:52.101 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-24T01:24:52.101 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-24T01:24:52.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:52.101 DEBUG:teuthology.orchestra.run.smithi120:> ip netns list 2024-03-24T01:24:52.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:52.158 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link delete ceph-brx 2024-03-24T01:24:52.256 INFO:teuthology.orchestra.run.smithi120.stderr:Cannot find device "ceph-brx" 2024-03-24T01:24:52.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T01:24:52.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:52.261 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-03-24T01:24:52.303 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-24T01:24:52.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:52.304 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-24T01:24:52.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:52.439 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link delete ceph-brx 2024-03-24T01:24:52.552 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-24T01:24:52.552 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T01:24:52.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.742+0000 7efca1000640 1 Processor -- start 2024-03-24T01:24:52.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.742+0000 7efca1000640 1 -- start start 2024-03-24T01:24:52.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.744+0000 7efca1000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7efc9c15ea20 0x7efc9c06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:52.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.744+0000 7efca1000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7efc9c0cc430 0x7efc9c0cc860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:52.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.744+0000 7efca1000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc9c15b0b0 con 0x7efc9c0aec90 2024-03-24T01:24:52.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.744+0000 7efca1000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc9c0ccda0 con 0x7efc9c0cc430 2024-03-24T01:24:52.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.744+0000 7efca1000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc9c15b390 con 0x7efc9c15ea20 2024-03-24T01:24:52.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.744+0000 7efc9a800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7efc9c0cc430 0x7efc9c0cc860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:52.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.744+0000 7efc9a800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7efc9c0cc430 0x7efc9c0cc860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37934/0 (socket says 172.21.15.90:37934) 2024-03-24T01:24:52.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.744+0000 7efc99e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7efc9c0aec90 0x7efc9c15a830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:36582/0 (socket says 172.21.15.90:36582) 2024-03-24T01:24:52.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.744+0000 7efc9a800640 1 -- 172.21.15.90:0/2923084375 learned_addr learned my addr 172.21.15.90:0/2923084375 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:52.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.745+0000 7efc91400640 1 --2- 172.21.15.90:0/2923084375 >> v2:172.21.15.120:3300/0 conn(0x7efc9c15ea20 0x7efc9c06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:52.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.745+0000 7efc9a800640 1 -- 172.21.15.90:0/2923084375 >> v2:172.21.15.120:3300/0 conn(0x7efc9c15ea20 msgr2=0x7efc9c06d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:52.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.745+0000 7efc9a800640 1 --2- 172.21.15.90:0/2923084375 >> v2:172.21.15.120:3300/0 conn(0x7efc9c15ea20 0x7efc9c06d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.745+0000 7efc9a800640 1 -- 172.21.15.90:0/2923084375 >> v1:172.21.15.90:6789/0 conn(0x7efc9c0aec90 legacy=0x7efc9c15a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:52.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.745+0000 7efc9a800640 1 -- 172.21.15.90:0/2923084375 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc9c15b670 con 0x7efc9c0cc430 2024-03-24T01:24:52.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.746+0000 7efc9a800640 1 --2- 172.21.15.90:0/2923084375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc9c0cc430 0x7efc9c0cc860 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7efc9c15b950 tx=0x7efc9402d080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ceb9178078a2ca8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:52.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.746+0000 7efc99400640 1 -- 172.21.15.90:0/2923084375 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9402da20 con 0x7efc9c0cc430 2024-03-24T01:24:52.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.746+0000 7efc99400640 1 -- 172.21.15.90:0/2923084375 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc9402dbc0 con 0x7efc9c0cc430 2024-03-24T01:24:52.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.746+0000 7efc99400640 1 -- 172.21.15.90:0/2923084375 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc94008500 con 0x7efc9c0cc430 2024-03-24T01:24:52.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.747+0000 7efca1000640 1 -- 172.21.15.90:0/2923084375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc9c0cc430 msgr2=0x7efc9c0cc860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:52.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.747+0000 7efca1000640 1 --2- 172.21.15.90:0/2923084375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc9c0cc430 0x7efc9c0cc860 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7efc9c15b950 tx=0x7efc9402d080 comp rx=0 tx=0).stop 2024-03-24T01:24:52.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.747+0000 7efca1000640 1 -- 172.21.15.90:0/2923084375 shutdown_connections 2024-03-24T01:24:52.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.747+0000 7efca1000640 1 --2- 172.21.15.90:0/2923084375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc9c0cc430 0x7efc9c0cc860 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.747+0000 7efca1000640 1 --2- 172.21.15.90:0/2923084375 >> v2:172.21.15.120:3300/0 conn(0x7efc9c15ea20 0x7efc9c06d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.747+0000 7efca1000640 1 -- 172.21.15.90:0/2923084375 >> 172.21.15.90:0/2923084375 conn(0x7efc9c06b2a0 msgr2=0x7efc9c06b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:52.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.747+0000 7efca1000640 1 -- 172.21.15.90:0/2923084375 shutdown_connections 2024-03-24T01:24:52.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.747+0000 7efca1000640 1 -- 172.21.15.90:0/2923084375 wait complete. 2024-03-24T01:24:52.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.749+0000 7efca1000640 1 Processor -- start 2024-03-24T01:24:52.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.749+0000 7efca1000640 1 -- start start 2024-03-24T01:24:52.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.750+0000 7efca1000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc9c0aec90 0x7efc9c1e4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:52.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.750+0000 7efca1000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efc9c0cc430 0x7efc9c1e5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:52.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.750+0000 7efca1000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efc9c15ea20 0x7efc9c14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:52.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.750+0000 7efca1000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c06dd60 con 0x7efc9c0aec90 2024-03-24T01:24:52.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.750+0000 7efca1000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c0ce450 con 0x7efc9c0cc430 2024-03-24T01:24:52.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.750+0000 7efca1000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c0af070 con 0x7efc9c15ea20 2024-03-24T01:24:52.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.750+0000 7efc99e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc9c0aec90 0x7efc9c1e4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:52.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.750+0000 7efc99e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc9c0aec90 0x7efc9c1e4560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37942/0 (socket says 172.21.15.90:37942) 2024-03-24T01:24:52.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.750+0000 7efc99e00640 1 -- 172.21.15.90:0/1376671241 learned_addr learned my addr 172.21.15.90:0/1376671241 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:52.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.750+0000 7efc91400640 1 --2- 172.21.15.90:0/1376671241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efc9c0cc430 0x7efc9c1e5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:52.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.751+0000 7efc9a800640 1 --2- 172.21.15.90:0/1376671241 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efc9c15ea20 0x7efc9c14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:52.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.751+0000 7efc99e00640 1 -- 172.21.15.90:0/1376671241 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efc9c15ea20 msgr2=0x7efc9c14fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:52.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.751+0000 7efc99e00640 1 --2- 172.21.15.90:0/1376671241 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efc9c15ea20 0x7efc9c14fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.751+0000 7efc99e00640 1 -- 172.21.15.90:0/1376671241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efc9c0cc430 msgr2=0x7efc9c1e5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:52.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.751+0000 7efc99e00640 1 --2- 172.21.15.90:0/1376671241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efc9c0cc430 0x7efc9c1e5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:52.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.751+0000 7efc99e00640 1 -- 172.21.15.90:0/1376671241 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc9c0ae530 con 0x7efc9c0aec90 2024-03-24T01:24:52.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.751+0000 7efc91400640 1 --2- 172.21.15.90:0/1376671241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efc9c0cc430 0x7efc9c1e5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:24:52.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.751+0000 7efc99e00640 1 --2- 172.21.15.90:0/1376671241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc9c0aec90 0x7efc9c1e4560 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7efc8c00c3c0 tx=0x7efc8c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:52.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.751+0000 7efc92a00640 1 -- 172.21.15.90:0/1376671241 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc8c0074e0 con 0x7efc9c0aec90 2024-03-24T01:24:52.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.751+0000 7efc92a00640 1 -- 172.21.15.90:0/1376671241 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc8c007680 con 0x7efc9c0aec90 2024-03-24T01:24:52.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.751+0000 7efc92a00640 1 -- 172.21.15.90:0/1376671241 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc8c004a00 con 0x7efc9c0aec90 2024-03-24T01:24:52.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.751+0000 7efca1000640 1 -- 172.21.15.90:0/1376671241 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc9c1e4c40 con 0x7efc9c0aec90 2024-03-24T01:24:52.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.752+0000 7efca1000640 1 -- 172.21.15.90:0/1376671241 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc9c1e5200 con 0x7efc9c0aec90 2024-03-24T01:24:52.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.753+0000 7efca1000640 1 -- 172.21.15.90:0/1376671241 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc9c073570 con 0x7efc9c0aec90 2024-03-24T01:24:52.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.754+0000 7efc92a00640 1 -- 172.21.15.90:0/1376671241 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7efc8c01a020 con 0x7efc9c0aec90 2024-03-24T01:24:52.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.754+0000 7efc92a00640 1 --2- 172.21.15.90:0/1376671241 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7efc7806bbd0 0x7efc7806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:52.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.754+0000 7efc92a00640 1 -- 172.21.15.90:0/1376671241 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6764+0+0 (secure 0 0 0) 0x7efc8c08a610 con 0x7efc9c0aec90 2024-03-24T01:24:52.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.755+0000 7efc91400640 1 --2- 172.21.15.90:0/1376671241 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7efc7806bbd0 0x7efc7806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:52.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.756+0000 7efc91400640 1 --2- 172.21.15.90:0/1376671241 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7efc7806bbd0 0x7efc7806e090 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7efc9c1f73a0 tx=0x7efc84008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:52.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.759+0000 7efc92a00640 1 -- 172.21.15.90:0/1376671241 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7efc8c058ce0 con 0x7efc9c0aec90 2024-03-24T01:24:52.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.998+0000 7efca1000640 1 -- 172.21.15.90:0/1376671241 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7efc9c15bab0 con 0x7efc9c0aec90 2024-03-24T01:24:52.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:52.999+0000 7efc92a00640 1 -- 172.21.15.90:0/1376671241 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+88 (secure 0 0 0) 0x7efc8c0585e0 con 0x7efc9c0aec90 2024-03-24T01:24:52.999 INFO:teuthology.orchestra.run.smithi090.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-24T01:24:53.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:53.002+0000 7efca1000640 1 -- 172.21.15.90:0/1376671241 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7efc7806bbd0 msgr2=0x7efc7806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:53.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:53.002+0000 7efca1000640 1 --2- 172.21.15.90:0/1376671241 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7efc7806bbd0 0x7efc7806e090 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7efc9c1f73a0 tx=0x7efc84008040 comp rx=0 tx=0).stop 2024-03-24T01:24:53.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:53.002+0000 7efca1000640 1 -- 172.21.15.90:0/1376671241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc9c0aec90 msgr2=0x7efc9c1e4560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:53.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:53.002+0000 7efca1000640 1 --2- 172.21.15.90:0/1376671241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc9c0aec90 0x7efc9c1e4560 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7efc8c00c3c0 tx=0x7efc8c00c880 comp rx=0 tx=0).stop 2024-03-24T01:24:53.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:53.002+0000 7efca1000640 1 -- 172.21.15.90:0/1376671241 shutdown_connections 2024-03-24T01:24:53.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:53.002+0000 7efca1000640 1 --2- 172.21.15.90:0/1376671241 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7efc7806bbd0 0x7efc7806e090 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:53.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:53.002+0000 7efca1000640 1 --2- 172.21.15.90:0/1376671241 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7efc9c15ea20 0x7efc9c14fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:53.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:53.002+0000 7efca1000640 1 --2- 172.21.15.90:0/1376671241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efc9c0cc430 0x7efc9c1e5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:53.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:53.002+0000 7efca1000640 1 --2- 172.21.15.90:0/1376671241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc9c0aec90 0x7efc9c1e4560 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:53.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:53.002+0000 7efca1000640 1 -- 172.21.15.90:0/1376671241 >> 172.21.15.90:0/1376671241 conn(0x7efc9c06b2a0 msgr2=0x7efc9c158ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:53.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:53.002+0000 7efca1000640 1 -- 172.21.15.90:0/1376671241 shutdown_connections 2024-03-24T01:24:53.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:53.002+0000 7efca1000640 1 -- 172.21.15.90:0/1376671241 wait complete. 2024-03-24T01:24:53.019 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T01:24:53.019 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T01:24:53.019 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi090.front.sepia.ceph.com 2024-03-24T01:24:53.020 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T01:24:53.020 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T01:24:53.020 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T01:24:53.020 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T01:24:53.020 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T01:24:53.020 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T01:24:53.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:53.020 DEBUG:teuthology.orchestra.run.smithi090:> ip addr 2024-03-24T01:24:53.067 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T01:24:53.068 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T01:24:53.068 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T01:24:53.068 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:24:53.068 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-03-24T01:24:53.068 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:24:53.068 INFO:teuthology.orchestra.run.smithi090.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T01:24:53.068 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a8 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:24:53.068 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f0 2024-03-24T01:24:53.068 INFO:teuthology.orchestra.run.smithi090.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T01:24:53.068 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a9 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:24:53.068 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f1 2024-03-24T01:24:53.068 INFO:teuthology.orchestra.run.smithi090.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T01:24:53.068 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:66 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:24:53.069 INFO:teuthology.orchestra.run.smithi090.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T01:24:53.069 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:67 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:24:53.069 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.21.15.90/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T01:24:53.069 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft 42489sec preferred_lft 42489sec 2024-03-24T01:24:53.069 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::ec4:7aff:fe8f:d767/64 scope link 2024-03-24T01:24:53.069 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:24:53.070 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-24T01:24:53.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:53.070 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2024-03-24T01:24:53.070 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-24T01:24:53.070 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link add name ceph-brx type bridge 2024-03-24T01:24:53.070 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip addr flush dev ceph-brx 2024-03-24T01:24:53.070 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set ceph-brx up 2024-03-24T01:24:53.070 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-24T01:24:53.071 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-24T01:24:53.286 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:53.286 DEBUG:teuthology.orchestra.run.smithi090:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T01:24:53.361 INFO:teuthology.orchestra.run.smithi090.stdout:1 2024-03-24T01:24:53.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:53.365 DEBUG:teuthology.orchestra.run.smithi090:> route 2024-03-24T01:24:53.417 INFO:teuthology.orchestra.run.smithi090.stdout:Kernel IP routing table 2024-03-24T01:24:53.418 INFO:teuthology.orchestra.run.smithi090.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T01:24:53.418 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T01:24:53.418 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T01:24:53.418 INFO:teuthology.orchestra.run.smithi090.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T01:24:53.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:53.419 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2024-03-24T01:24:53.419 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-24T01:24:53.419 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T01:24:53.419 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T01:24:53.419 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T01:24:53.419 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-24T01:24:53.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:53.598 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-03-24T01:24:53.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:53.647 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list-id 2024-03-24T01:24:53.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:53.742 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2024-03-24T01:24:53.743 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-24T01:24:53.743 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:24:53.743 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-24T01:24:53.743 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-24T01:24:53.883 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-24T01:24:53.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:53.883 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2024-03-24T01:24:53.883 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-24T01:24:53.883 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-24T01:24:53.883 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-24T01:24:53.883 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-24T01:24:53.883 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-24T01:24:53.883 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-24T01:24:53.884 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-24T01:24:54.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:54.128 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2024-03-24T01:24:54.128 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-24T01:24:54.128 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set brx.0 up 2024-03-24T01:24:54.128 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set dev brx.0 master ceph-brx 2024-03-24T01:24:54.128 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-24T01:24:54.288 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-24T01:24:54.288 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T01:24:54.288 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T01:24:54.335 INFO:teuthology.orchestra.run.smithi090.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T01:24:54.335 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T01:24:54.336 DEBUG:teuthology.orchestra.run.smithi090:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T01:24:54.429 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-24T01:24:54.525 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T01:24:54.580 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-24T01:24:54.580 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-24T01:24:54.580 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-24T01:24:54.580 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-24T01:24:54.580 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-24T01:24:54.581 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-24T01:24:54.581 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-24T01:24:54.581 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-24T01:24:54.581 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-24T01:24:54.581 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-24T01:24:54.581 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-24T01:24:54.581 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/selinux 2024-03-24T01:24:54.581 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T01:24:54.581 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-24T01:24:54.581 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-24T01:24:54.581 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-24T01:24:54.582 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-24T01:24:54.582 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T01:24:54.582 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-24T01:24:54.582 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T01:24:54.582 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-24T01:24:54.582 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T01:24:54.582 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T01:24:54.582 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-24T01:24:54.582 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-24T01:24:54.582 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T01:24:54.582 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T01:24:54.582 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T01:24:54.582 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T01:24:54.583 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-24T01:24:54.583 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:24:54.583 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:24:54.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:54.584 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-24T01:24:54.633 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-24T01:24:54.633 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-24T01:24:54.640 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-24T01:24:54.716 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T01:24:54.761 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-24T01:24:54.761 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-24T01:24:54.761 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-24T01:24:54.761 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-24T01:24:54.761 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-24T01:24:54.762 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-24T01:24:54.762 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-24T01:24:54.762 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-24T01:24:54.762 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-24T01:24:54.762 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-24T01:24:54.763 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-24T01:24:54.763 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/selinux 2024-03-24T01:24:54.763 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T01:24:54.763 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-24T01:24:54.763 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-24T01:24:54.763 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-24T01:24:54.764 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-24T01:24:54.764 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T01:24:54.764 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-24T01:24:54.764 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T01:24:54.764 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-24T01:24:54.764 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T01:24:54.765 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T01:24:54.765 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-24T01:24:54.765 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-24T01:24:54.765 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T01:24:54.765 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T01:24:54.765 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T01:24:54.766 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T01:24:54.766 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-24T01:24:54.766 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:24:54.766 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:24:54.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:54.768 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-24T01:24:54.832 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:2024-03-24T01:24:54.832+0000 7f30b3a3f480 -1 init, newargv = 0x56391daa6ff0 newargc=15 2024-03-24T01:24:54.832 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[47228]: starting ceph client 2024-03-24T01:24:54.842 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[47228]: starting fuse 2024-03-24T01:24:54.853 INFO:teuthology.orchestra.run.smithi090.stdout:40 2024-03-24T01:24:54.853 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-03-24T01:24:54.853 DEBUG:teuthology.orchestra.run.smithi090:> sudo stdin-killer -- python3 -c ' 2024-03-24T01:24:54.853 DEBUG:teuthology.orchestra.run.smithi090:> import glob 2024-03-24T01:24:54.853 DEBUG:teuthology.orchestra.run.smithi090:> import re 2024-03-24T01:24:54.853 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-24T01:24:54.854 DEBUG:teuthology.orchestra.run.smithi090:> import subprocess 2024-03-24T01:24:54.854 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:24:54.854 DEBUG:teuthology.orchestra.run.smithi090:> def _find_admin_socket(client_name): 2024-03-24T01:24:54.854 DEBUG:teuthology.orchestra.run.smithi090:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-24T01:24:54.854 DEBUG:teuthology.orchestra.run.smithi090:> files = glob.glob(asok_path) 2024-03-24T01:24:54.854 DEBUG:teuthology.orchestra.run.smithi090:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-24T01:24:54.854 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:24:54.854 DEBUG:teuthology.orchestra.run.smithi090:> # Given a non-glob path, it better be there 2024-03-24T01:24:54.854 DEBUG:teuthology.orchestra.run.smithi090:> if "*" not in asok_path: 2024-03-24T01:24:54.854 DEBUG:teuthology.orchestra.run.smithi090:> assert(len(files) == 1) 2024-03-24T01:24:54.854 DEBUG:teuthology.orchestra.run.smithi090:> return files[0] 2024-03-24T01:24:54.854 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:24:54.854 DEBUG:teuthology.orchestra.run.smithi090:> for f in files: 2024-03-24T01:24:54.854 DEBUG:teuthology.orchestra.run.smithi090:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T01:24:54.854 DEBUG:teuthology.orchestra.run.smithi090:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T01:24:54.855 DEBUG:teuthology.orchestra.run.smithi090:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T01:24:54.855 DEBUG:teuthology.orchestra.run.smithi090:> contents = proc_f.read() 2024-03-24T01:24:54.855 DEBUG:teuthology.orchestra.run.smithi090:> if mountpoint in contents: 2024-03-24T01:24:54.855 DEBUG:teuthology.orchestra.run.smithi090:> return f 2024-03-24T01:24:54.855 DEBUG:teuthology.orchestra.run.smithi090:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T01:24:54.855 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:24:54.855 DEBUG:teuthology.orchestra.run.smithi090:> print(_find_admin_socket("client.0")) 2024-03-24T01:24:54.855 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-24T01:24:55.049 INFO:teuthology.orchestra.run.smithi090.stdout:/var/run/ceph/ceph-client.0.47228.asok 2024-03-24T01:24:55.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T01:24:55.067 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.47228.asok 2024-03-24T01:24:55.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:55.067 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.47228.asok status 2024-03-24T01:24:55.306 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-24T01:24:55.307 INFO:teuthology.orchestra.run.smithi090.stdout: "metadata": { 2024-03-24T01:24:55.307 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_sha1": "3be9a5c9bc793e11e2800a8c0c696e8b46742033", 2024-03-24T01:24:55.307 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-24T01:24:55.307 INFO:teuthology.orchestra.run.smithi090.stdout: "entity_id": "0", 2024-03-24T01:24:55.307 INFO:teuthology.orchestra.run.smithi090.stdout: "hostname": "smithi090", 2024-03-24T01:24:55.307 INFO:teuthology.orchestra.run.smithi090.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-24T01:24:55.307 INFO:teuthology.orchestra.run.smithi090.stdout: "pid": "47228", 2024-03-24T01:24:55.307 INFO:teuthology.orchestra.run.smithi090.stdout: "root": "/" 2024-03-24T01:24:55.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:24:55.307 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_count": 0, 2024-03-24T01:24:55.307 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_pinned_count": 0, 2024-03-24T01:24:55.307 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 4818, 2024-03-24T01:24:55.307 INFO:teuthology.orchestra.run.smithi090.stdout: "inst": { 2024-03-24T01:24:55.307 INFO:teuthology.orchestra.run.smithi090.stdout: "name": { 2024-03-24T01:24:55.308 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "client", 2024-03-24T01:24:55.308 INFO:teuthology.orchestra.run.smithi090.stdout: "num": 4818 2024-03-24T01:24:55.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:24:55.308 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-24T01:24:55.308 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-24T01:24:55.308 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-03-24T01:24:55.308 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1022127140 2024-03-24T01:24:55.308 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:24:55.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:24:55.308 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-24T01:24:55.308 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-24T01:24:55.308 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-03-24T01:24:55.308 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1022127140 2024-03-24T01:24:55.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:24:55.308 INFO:teuthology.orchestra.run.smithi090.stdout: "inst_str": "client.4818 192.168.0.1:0/1022127140", 2024-03-24T01:24:55.309 INFO:teuthology.orchestra.run.smithi090.stdout: "addr_str": "192.168.0.1:0/1022127140", 2024-03-24T01:24:55.309 INFO:teuthology.orchestra.run.smithi090.stdout: "inode_count": 1, 2024-03-24T01:24:55.309 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_epoch": 8, 2024-03-24T01:24:55.309 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch": 27, 2024-03-24T01:24:55.309 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch_barrier": 0, 2024-03-24T01:24:55.309 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklisted": false, 2024-03-24T01:24:55.309 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs" 2024-03-24T01:24:55.309 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-24T01:24:55.321 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T01:24:55.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.529+0000 7f4a67200640 1 Processor -- start 2024-03-24T01:24:55.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.529+0000 7f4a67200640 1 -- start start 2024-03-24T01:24:55.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.530+0000 7f4a67200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4a680cef60 0x7f4a6815ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:55.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.530+0000 7f4a67200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a6815b300 0x7f4a680b2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:55.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.530+0000 7f4a67200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a6815b980 con 0x7f4a6815ae50 2024-03-24T01:24:55.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.530+0000 7f4a67200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a6815f390 con 0x7f4a680cef60 2024-03-24T01:24:55.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.530+0000 7f4a67200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6815b750 con 0x7f4a6815b300 2024-03-24T01:24:55.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.530+0000 7f4a5d400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4a6815ae50 0x7f4a680b2680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:36598/0 (socket says 172.21.15.90:36598) 2024-03-24T01:24:55.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.531+0000 7f4a5d400640 1 -- 172.21.15.90:0/2645318132 learned_addr learned my addr 172.21.15.90:0/2645318132 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:55.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.531+0000 7f4a65e00640 1 --2- 172.21.15.90:0/2645318132 >> v2:172.21.15.120:3300/0 conn(0x7f4a680cef60 0x7f4a6815ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:55.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.531+0000 7f4a66800640 1 --2- 172.21.15.90:0/2645318132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a6815b300 0x7f4a680b2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:55.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.531+0000 7f4a65400640 1 -- 172.21.15.90:0/2645318132 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 170033084 0 0) 0x7f4a6815b980 con 0x7f4a6815ae50 2024-03-24T01:24:55.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.531+0000 7f4a65400640 1 -- 172.21.15.90:0/2645318132 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a4c003520 con 0x7f4a6815ae50 2024-03-24T01:24:55.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.531+0000 7f4a65400640 1 -- 172.21.15.90:0/2645318132 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1940156556 0 0) 0x7f4a4c003520 con 0x7f4a6815ae50 2024-03-24T01:24:55.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.531+0000 7f4a65400640 1 -- 172.21.15.90:0/2645318132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a6815b300 msgr2=0x7f4a680b2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:55.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.531+0000 7f4a65400640 1 --2- 172.21.15.90:0/2645318132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a6815b300 0x7f4a680b2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:55.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.531+0000 7f4a65400640 1 -- 172.21.15.90:0/2645318132 >> v2:172.21.15.120:3300/0 conn(0x7f4a680cef60 msgr2=0x7f4a6815ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:55.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.531+0000 7f4a65400640 1 --2- 172.21.15.90:0/2645318132 >> v2:172.21.15.120:3300/0 conn(0x7f4a680cef60 0x7f4a6815ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:55.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.531+0000 7f4a65400640 1 -- 172.21.15.90:0/2645318132 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a6815bc60 con 0x7f4a6815ae50 2024-03-24T01:24:55.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.531+0000 7f4a65400640 1 -- 172.21.15.90:0/2645318132 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f4a50001670 con 0x7f4a6815ae50 2024-03-24T01:24:55.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.531+0000 7f4a65400640 1 -- 172.21.15.90:0/2645318132 >> v1:172.21.15.90:6789/0 conn(0x7f4a6815ae50 legacy=0x7f4a680b2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:55.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.531+0000 7f4a65400640 1 --2- 172.21.15.90:0/2645318132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a4c0039b0 0x7f4a4c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:55.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.532+0000 7f4a65400640 1 --2- 172.21.15.90:0/2645318132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a6815b300 0x7f4a4c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:55.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.532+0000 7f4a65400640 1 --2- 172.21.15.90:0/2645318132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a4c004d40 0x7f4a4c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:55.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.532+0000 7f4a65400640 1 -- 172.21.15.90:0/2645318132 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6815f390 con 0x7f4a4c0039b0 2024-03-24T01:24:55.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.532+0000 7f4a65400640 1 -- 172.21.15.90:0/2645318132 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4c00dbd0 con 0x7f4a4c004d40 2024-03-24T01:24:55.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.532+0000 7f4a65400640 1 -- 172.21.15.90:0/2645318132 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4c00dd50 con 0x7f4a6815b300 2024-03-24T01:24:55.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.532+0000 7f4a65e00640 1 --2- 172.21.15.90:0/2645318132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a4c0039b0 0x7f4a4c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:55.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.532+0000 7f4a5d400640 1 --2- 172.21.15.90:0/2645318132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a6815b300 0x7f4a4c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:55.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.532+0000 7f4a66800640 1 --2- 172.21.15.90:0/2645318132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a4c004d40 0x7f4a4c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:55.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.532+0000 7f4a65e00640 1 -- 172.21.15.90:0/2645318132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a6815b300 msgr2=0x7f4a4c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:55.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.532+0000 7f4a65e00640 1 --2- 172.21.15.90:0/2645318132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a6815b300 0x7f4a4c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:55.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.532+0000 7f4a65e00640 1 -- 172.21.15.90:0/2645318132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a4c004d40 msgr2=0x7f4a4c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:55.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.532+0000 7f4a65e00640 1 --2- 172.21.15.90:0/2645318132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a4c004d40 0x7f4a4c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:55.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.532+0000 7f4a65e00640 1 -- 172.21.15.90:0/2645318132 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a4c003520 con 0x7f4a4c0039b0 2024-03-24T01:24:55.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.533+0000 7f4a65e00640 1 --2- 172.21.15.90:0/2645318132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a4c0039b0 0x7f4a4c003dd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4a58009ac0 tx=0x7f4a4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96b5ec7ca89cb945 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:55.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.534+0000 7f4a65400640 1 -- 172.21.15.90:0/2645318132 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a58039070 con 0x7f4a4c0039b0 2024-03-24T01:24:55.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.534+0000 7f4a65400640 1 -- 172.21.15.90:0/2645318132 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a5802c710 con 0x7f4a4c0039b0 2024-03-24T01:24:55.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.534+0000 7f4a65400640 1 -- 172.21.15.90:0/2645318132 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a58009690 con 0x7f4a4c0039b0 2024-03-24T01:24:55.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.534+0000 7f4a67200640 1 -- 172.21.15.90:0/2645318132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a4c0039b0 msgr2=0x7f4a4c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:55.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.534+0000 7f4a67200640 1 --2- 172.21.15.90:0/2645318132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a4c0039b0 0x7f4a4c003dd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4a58009ac0 tx=0x7f4a4c005830 comp rx=0 tx=0).stop 2024-03-24T01:24:55.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.534+0000 7f4a66800640 1 -- 172.21.15.90:0/2645318132 reap_dead start 2024-03-24T01:24:55.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.534+0000 7f4a67200640 1 -- 172.21.15.90:0/2645318132 shutdown_connections 2024-03-24T01:24:55.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.534+0000 7f4a67200640 1 -- 172.21.15.90:0/2645318132 >> 172.21.15.90:0/2645318132 conn(0x7f4a6806b0c0 msgr2=0x7f4a6806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:55.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.534+0000 7f4a67200640 1 -- 172.21.15.90:0/2645318132 shutdown_connections 2024-03-24T01:24:55.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.534+0000 7f4a67200640 1 -- 172.21.15.90:0/2645318132 wait complete. 2024-03-24T01:24:55.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.536+0000 7f4a67200640 1 Processor -- start 2024-03-24T01:24:55.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.536+0000 7f4a67200640 1 -- start start 2024-03-24T01:24:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.537+0000 7f4a67200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a4c0039b0 0x7f4a680b0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.537+0000 7f4a67200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a6814ba80 0x7f4a6814b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.537+0000 7f4a67200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a680af3e0 0x7f4a680af860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.537+0000 7f4a67200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a680cc080 con 0x7f4a4c0039b0 2024-03-24T01:24:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.537+0000 7f4a67200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6815e930 con 0x7f4a6814ba80 2024-03-24T01:24:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.537+0000 7f4a67200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a680b2d90 con 0x7f4a680af3e0 2024-03-24T01:24:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.537+0000 7f4a65e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a4c0039b0 0x7f4a680b0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.537+0000 7f4a65e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a4c0039b0 0x7f4a680b0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37964/0 (socket says 172.21.15.90:37964) 2024-03-24T01:24:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.537+0000 7f4a65e00640 1 -- 172.21.15.90:0/4090079110 learned_addr learned my addr 172.21.15.90:0/4090079110 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.538+0000 7f4a66800640 1 --2- 172.21.15.90:0/4090079110 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a680af3e0 0x7f4a680af860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.538+0000 7f4a5d400640 1 --2- 172.21.15.90:0/4090079110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a6814ba80 0x7f4a6814b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.538+0000 7f4a65e00640 1 -- 172.21.15.90:0/4090079110 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a680af3e0 msgr2=0x7f4a680af860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.538+0000 7f4a65e00640 1 --2- 172.21.15.90:0/4090079110 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a680af3e0 0x7f4a680af860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.538+0000 7f4a65e00640 1 -- 172.21.15.90:0/4090079110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a6814ba80 msgr2=0x7f4a6814b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.538+0000 7f4a65e00640 1 --2- 172.21.15.90:0/4090079110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a6814ba80 0x7f4a6814b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:55.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.538+0000 7f4a65e00640 1 -- 172.21.15.90:0/4090079110 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a680b0ec0 con 0x7f4a4c0039b0 2024-03-24T01:24:55.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.538+0000 7f4a5d400640 1 --2- 172.21.15.90:0/4090079110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a6814ba80 0x7f4a6814b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:55.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.538+0000 7f4a65e00640 1 --2- 172.21.15.90:0/4090079110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a4c0039b0 0x7f4a680b0830 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4a58000c00 tx=0x7f4a58004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:55.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.538+0000 7f4a5ea00640 1 -- 172.21.15.90:0/4090079110 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a58039050 con 0x7f4a4c0039b0 2024-03-24T01:24:55.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.539+0000 7f4a5ea00640 1 -- 172.21.15.90:0/4090079110 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a58045070 con 0x7f4a4c0039b0 2024-03-24T01:24:55.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.539+0000 7f4a5ea00640 1 -- 172.21.15.90:0/4090079110 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a58003a50 con 0x7f4a4c0039b0 2024-03-24T01:24:55.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.539+0000 7f4a67200640 1 -- 172.21.15.90:0/4090079110 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a680b11a0 con 0x7f4a4c0039b0 2024-03-24T01:24:55.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.539+0000 7f4a67200640 1 -- 172.21.15.90:0/4090079110 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a680aeec0 con 0x7f4a4c0039b0 2024-03-24T01:24:55.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.540+0000 7f4a67200640 1 -- 172.21.15.90:0/4090079110 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a680ae9d0 con 0x7f4a4c0039b0 2024-03-24T01:24:55.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.541+0000 7f4a5ea00640 1 -- 172.21.15.90:0/4090079110 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4a58003bf0 con 0x7f4a4c0039b0 2024-03-24T01:24:55.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.541+0000 7f4a5ea00640 1 --2- 172.21.15.90:0/4090079110 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4a4406bbd0 0x7f4a4406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:55.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.542+0000 7f4a5ea00640 1 -- 172.21.15.90:0/4090079110 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f4a580ac940 con 0x7f4a4c0039b0 2024-03-24T01:24:55.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.542+0000 7f4a5d400640 1 --2- 172.21.15.90:0/4090079110 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4a4406bbd0 0x7f4a4406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:55.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.543+0000 7f4a5d400640 1 --2- 172.21.15.90:0/4090079110 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4a4406bbd0 0x7f4a4406e090 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4a50001d30 tx=0x7f4a50001cc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:55.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.546+0000 7f4a5ea00640 1 -- 172.21.15.90:0/4090079110 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4a5804e020 con 0x7f4a4c0039b0 2024-03-24T01:24:55.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.789+0000 7f4a67200640 1 -- 172.21.15.90:0/4090079110 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f4a6804f400 con 0x7f4a4c0039b0 2024-03-24T01:24:55.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.790+0000 7f4a5ea00640 1 -- 172.21.15.90:0/4090079110 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+88 (secure 0 0 0) 0x7f4a5807b120 con 0x7f4a4c0039b0 2024-03-24T01:24:55.790 INFO:teuthology.orchestra.run.smithi090.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-24T01:24:55.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.792+0000 7f4a67200640 1 -- 172.21.15.90:0/4090079110 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4a4406bbd0 msgr2=0x7f4a4406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:55.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.792+0000 7f4a67200640 1 --2- 172.21.15.90:0/4090079110 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4a4406bbd0 0x7f4a4406e090 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4a50001d30 tx=0x7f4a50001cc0 comp rx=0 tx=0).stop 2024-03-24T01:24:55.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.792+0000 7f4a67200640 1 -- 172.21.15.90:0/4090079110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a4c0039b0 msgr2=0x7f4a680b0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:55.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.792+0000 7f4a67200640 1 --2- 172.21.15.90:0/4090079110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a4c0039b0 0x7f4a680b0830 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4a58000c00 tx=0x7f4a58004730 comp rx=0 tx=0).stop 2024-03-24T01:24:55.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.793+0000 7f4a67200640 1 -- 172.21.15.90:0/4090079110 shutdown_connections 2024-03-24T01:24:55.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.793+0000 7f4a67200640 1 --2- 172.21.15.90:0/4090079110 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4a4406bbd0 0x7f4a4406e090 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:55.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.793+0000 7f4a67200640 1 --2- 172.21.15.90:0/4090079110 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a680af3e0 0x7f4a680af860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:55.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.793+0000 7f4a67200640 1 --2- 172.21.15.90:0/4090079110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a6814ba80 0x7f4a6814b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:55.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.793+0000 7f4a67200640 1 --2- 172.21.15.90:0/4090079110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a4c0039b0 0x7f4a680b0830 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:55.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.793+0000 7f4a67200640 1 -- 172.21.15.90:0/4090079110 >> 172.21.15.90:0/4090079110 conn(0x7f4a6806b0c0 msgr2=0x7f4a68158160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:55.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.793+0000 7f4a67200640 1 -- 172.21.15.90:0/4090079110 shutdown_connections 2024-03-24T01:24:55.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:55.793+0000 7f4a67200640 1 -- 172.21.15.90:0/4090079110 wait complete. 2024-03-24T01:24:55.812 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T01:24:55.812 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T01:24:55.812 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi090.front.sepia.ceph.com 2024-03-24T01:24:55.812 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-03-24T01:24:55.812 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-24T01:24:55.812 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T01:24:55.812 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T01:24:55.812 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T01:24:55.812 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-03-24T01:24:55.813 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:55.813 DEBUG:teuthology.orchestra.run.smithi090:> ip addr 2024-03-24T01:24:55.863 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T01:24:55.863 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T01:24:55.863 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T01:24:55.863 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:24:55.863 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-03-24T01:24:55.863 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:24:55.863 INFO:teuthology.orchestra.run.smithi090.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T01:24:55.863 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a8 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:24:55.863 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f0 2024-03-24T01:24:55.863 INFO:teuthology.orchestra.run.smithi090.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T01:24:55.864 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a9 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:24:55.864 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f1 2024-03-24T01:24:55.864 INFO:teuthology.orchestra.run.smithi090.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T01:24:55.864 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:66 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:24:55.864 INFO:teuthology.orchestra.run.smithi090.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T01:24:55.864 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:67 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:24:55.864 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.21.15.90/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T01:24:55.864 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft 42486sec preferred_lft 42486sec 2024-03-24T01:24:55.864 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::ec4:7aff:fe8f:d767/64 scope link 2024-03-24T01:24:55.864 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:24:55.864 INFO:teuthology.orchestra.run.smithi090.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T01:24:55.864 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether de:94:0b:7e:c6:41 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:24:55.864 INFO:teuthology.orchestra.run.smithi090.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T01:24:55.865 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:24:55.865 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::a05b:d3ff:fea5:6fd0/64 scope link 2024-03-24T01:24:55.865 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:24:55.865 INFO:teuthology.orchestra.run.smithi090.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T01:24:55.865 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether de:94:0b:7e:c6:41 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:24:55.865 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::dc94:bff:fe7e:c641/64 scope link 2024-03-24T01:24:55.865 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:24:55.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:55.865 DEBUG:teuthology.orchestra.run.smithi090:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T01:24:55.996 INFO:teuthology.orchestra.run.smithi090.stdout:1 2024-03-24T01:24:56.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:56.001 DEBUG:teuthology.orchestra.run.smithi090:> route 2024-03-24T01:24:56.053 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:24:56.054 INFO:teuthology.orchestra.run.smithi090.stdout:Kernel IP routing table 2024-03-24T01:24:56.054 INFO:teuthology.orchestra.run.smithi090.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T01:24:56.055 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T01:24:56.055 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T01:24:56.055 INFO:teuthology.orchestra.run.smithi090.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T01:24:56.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:56.055 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2024-03-24T01:24:56.055 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-24T01:24:56.055 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T01:24:56.055 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T01:24:56.055 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T01:24:56.055 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-24T01:24:56.102 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:24:56.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:56.272 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-03-24T01:24:56.323 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T01:24:56.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:56.323 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list-id 2024-03-24T01:24:56.373 INFO:teuthology.orchestra.run.smithi090.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-03-24T01:24:56.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:56.375 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2024-03-24T01:24:56.375 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-24T01:24:56.375 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:24:56.375 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-03-24T01:24:56.375 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-24T01:24:56.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:56.512 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-24T01:24:56.599 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T01:24:56.599 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T01:24:56.600 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T01:24:56.600 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:24:56.600 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-03-24T01:24:56.600 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:24:56.600 INFO:teuthology.orchestra.run.smithi090.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T01:24:56.600 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 16:0b:7d:d3:a8:58 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-24T01:24:56.601 INFO:teuthology.orchestra.run.smithi090.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-24T01:24:56.601 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:24:56.601 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::140b:7dff:fed3:a858/64 scope link 2024-03-24T01:24:56.601 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:24:56.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:56.604 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-24T01:24:56.693 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T01:24:56.694 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T01:24:56.694 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T01:24:56.694 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:24:56.694 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-03-24T01:24:56.694 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:24:56.694 INFO:teuthology.orchestra.run.smithi090.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T01:24:56.694 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 16:0b:7d:d3:a8:58 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-24T01:24:56.694 INFO:teuthology.orchestra.run.smithi090.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-24T01:24:56.694 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:24:56.694 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::140b:7dff:fed3:a858/64 scope link 2024-03-24T01:24:56.694 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:24:56.698 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-03-24T01:24:56.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:56.698 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2024-03-24T01:24:56.698 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-24T01:24:56.698 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-03-24T01:24:56.698 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-24T01:24:56.698 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-03-24T01:24:56.698 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-03-24T01:24:56.698 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-03-24T01:24:56.698 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-24T01:24:56.961 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:56.961 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2024-03-24T01:24:56.961 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-24T01:24:56.961 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set brx.1 up 2024-03-24T01:24:56.961 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set dev brx.1 master ceph-brx 2024-03-24T01:24:56.961 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-24T01:24:57.131 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-03-24T01:24:57.131 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T01:24:57.131 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-03-24T01:24:57.178 INFO:teuthology.orchestra.run.smithi090.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-03-24T01:24:57.178 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T01:24:57.178 DEBUG:teuthology.orchestra.run.smithi090:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-03-24T01:24:57.273 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-24T01:24:57.361 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T01:24:57.408 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-24T01:24:57.408 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-24T01:24:57.408 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-24T01:24:57.408 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-24T01:24:57.409 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-24T01:24:57.409 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-24T01:24:57.409 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-24T01:24:57.409 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-24T01:24:57.409 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-24T01:24:57.409 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-24T01:24:57.409 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-24T01:24:57.409 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/selinux 2024-03-24T01:24:57.409 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T01:24:57.409 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-24T01:24:57.409 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-24T01:24:57.409 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-24T01:24:57.409 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-24T01:24:57.409 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T01:24:57.410 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-24T01:24:57.410 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T01:24:57.410 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-24T01:24:57.410 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T01:24:57.410 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T01:24:57.410 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-24T01:24:57.410 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-24T01:24:57.410 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T01:24:57.410 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T01:24:57.410 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T01:24:57.410 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T01:24:57.410 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-24T01:24:57.410 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:24:57.410 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:24:57.410 INFO:teuthology.orchestra.run.smithi090.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-24T01:24:57.411 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:24:57.411 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:24:57.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:57.412 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-24T01:24:57.458 INFO:teuthology.orchestra.run.smithi090.stdout:40 2024-03-24T01:24:57.458 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [40] 2024-03-24T01:24:57.459 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-03-24T01:24:57.464 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-24T01:24:57.537 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T01:24:57.580 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-24T01:24:57.580 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-24T01:24:57.580 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-24T01:24:57.580 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-24T01:24:57.580 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-24T01:24:57.580 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-24T01:24:57.580 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-24T01:24:57.580 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-24T01:24:57.581 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-24T01:24:57.581 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-24T01:24:57.581 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-24T01:24:57.581 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/selinux 2024-03-24T01:24:57.581 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T01:24:57.581 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-24T01:24:57.581 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-24T01:24:57.581 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-24T01:24:57.581 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-24T01:24:57.581 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T01:24:57.581 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-24T01:24:57.581 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T01:24:57.581 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-24T01:24:57.581 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T01:24:57.582 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T01:24:57.582 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-24T01:24:57.582 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-24T01:24:57.582 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T01:24:57.582 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T01:24:57.582 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T01:24:57.582 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T01:24:57.582 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-24T01:24:57.582 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:24:57.582 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:24:57.582 INFO:teuthology.orchestra.run.smithi090.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-24T01:24:57.582 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:24:57.582 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:24:57.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:57.583 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-24T01:24:57.646 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi090.stderr:2024-03-24T01:24:57.645+0000 7f9416835480 -1 init, newargv = 0x55f938db2c60 newargc=15 2024-03-24T01:24:57.646 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi090.stderr:ceph-fuse[47740]: starting ceph client 2024-03-24T01:24:57.656 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi090.stderr:ceph-fuse[47740]: starting fuse 2024-03-24T01:24:57.669 INFO:teuthology.orchestra.run.smithi090.stdout:40 2024-03-24T01:24:57.669 INFO:teuthology.orchestra.run.smithi090.stdout:43 2024-03-24T01:24:57.669 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40, 43] 2024-03-24T01:24:57.670 DEBUG:teuthology.orchestra.run.smithi090:> sudo stdin-killer -- python3 -c ' 2024-03-24T01:24:57.670 DEBUG:teuthology.orchestra.run.smithi090:> import glob 2024-03-24T01:24:57.670 DEBUG:teuthology.orchestra.run.smithi090:> import re 2024-03-24T01:24:57.670 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-24T01:24:57.670 DEBUG:teuthology.orchestra.run.smithi090:> import subprocess 2024-03-24T01:24:57.670 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:24:57.670 DEBUG:teuthology.orchestra.run.smithi090:> def _find_admin_socket(client_name): 2024-03-24T01:24:57.670 DEBUG:teuthology.orchestra.run.smithi090:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-03-24T01:24:57.670 DEBUG:teuthology.orchestra.run.smithi090:> files = glob.glob(asok_path) 2024-03-24T01:24:57.670 DEBUG:teuthology.orchestra.run.smithi090:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-03-24T01:24:57.670 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:24:57.670 DEBUG:teuthology.orchestra.run.smithi090:> # Given a non-glob path, it better be there 2024-03-24T01:24:57.671 DEBUG:teuthology.orchestra.run.smithi090:> if "*" not in asok_path: 2024-03-24T01:24:57.671 DEBUG:teuthology.orchestra.run.smithi090:> assert(len(files) == 1) 2024-03-24T01:24:57.671 DEBUG:teuthology.orchestra.run.smithi090:> return files[0] 2024-03-24T01:24:57.671 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:24:57.671 DEBUG:teuthology.orchestra.run.smithi090:> for f in files: 2024-03-24T01:24:57.671 DEBUG:teuthology.orchestra.run.smithi090:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T01:24:57.671 DEBUG:teuthology.orchestra.run.smithi090:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T01:24:57.671 DEBUG:teuthology.orchestra.run.smithi090:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T01:24:57.671 DEBUG:teuthology.orchestra.run.smithi090:> contents = proc_f.read() 2024-03-24T01:24:57.671 DEBUG:teuthology.orchestra.run.smithi090:> if mountpoint in contents: 2024-03-24T01:24:57.671 DEBUG:teuthology.orchestra.run.smithi090:> return f 2024-03-24T01:24:57.671 DEBUG:teuthology.orchestra.run.smithi090:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T01:24:57.671 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:24:57.671 DEBUG:teuthology.orchestra.run.smithi090:> print(_find_admin_socket("client.2")) 2024-03-24T01:24:57.671 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-24T01:24:57.867 INFO:teuthology.orchestra.run.smithi090.stdout:/var/run/ceph/ceph-client.2.47740.asok 2024-03-24T01:24:57.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T01:24:57.884 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.47740.asok 2024-03-24T01:24:57.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:57.884 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.47740.asok status 2024-03-24T01:24:58.118 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-24T01:24:58.119 INFO:teuthology.orchestra.run.smithi090.stdout: "metadata": { 2024-03-24T01:24:58.119 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_sha1": "3be9a5c9bc793e11e2800a8c0c696e8b46742033", 2024-03-24T01:24:58.119 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-24T01:24:58.119 INFO:teuthology.orchestra.run.smithi090.stdout: "entity_id": "2", 2024-03-24T01:24:58.119 INFO:teuthology.orchestra.run.smithi090.stdout: "hostname": "smithi090", 2024-03-24T01:24:58.119 INFO:teuthology.orchestra.run.smithi090.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-03-24T01:24:58.119 INFO:teuthology.orchestra.run.smithi090.stdout: "pid": "47740", 2024-03-24T01:24:58.119 INFO:teuthology.orchestra.run.smithi090.stdout: "root": "/" 2024-03-24T01:24:58.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:24:58.119 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_count": 0, 2024-03-24T01:24:58.119 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_pinned_count": 0, 2024-03-24T01:24:58.120 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 4830, 2024-03-24T01:24:58.120 INFO:teuthology.orchestra.run.smithi090.stdout: "inst": { 2024-03-24T01:24:58.120 INFO:teuthology.orchestra.run.smithi090.stdout: "name": { 2024-03-24T01:24:58.120 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "client", 2024-03-24T01:24:58.120 INFO:teuthology.orchestra.run.smithi090.stdout: "num": 4830 2024-03-24T01:24:58.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:24:58.120 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-24T01:24:58.120 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-24T01:24:58.120 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.2:0", 2024-03-24T01:24:58.120 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1651996549 2024-03-24T01:24:58.120 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:24:58.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:24:58.120 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-24T01:24:58.120 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-24T01:24:58.120 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.2:0", 2024-03-24T01:24:58.120 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1651996549 2024-03-24T01:24:58.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:24:58.121 INFO:teuthology.orchestra.run.smithi090.stdout: "inst_str": "client.4830 192.168.0.2:0/1651996549", 2024-03-24T01:24:58.121 INFO:teuthology.orchestra.run.smithi090.stdout: "addr_str": "192.168.0.2:0/1651996549", 2024-03-24T01:24:58.121 INFO:teuthology.orchestra.run.smithi090.stdout: "inode_count": 1, 2024-03-24T01:24:58.121 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_epoch": 8, 2024-03-24T01:24:58.121 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch": 27, 2024-03-24T01:24:58.121 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch_barrier": 0, 2024-03-24T01:24:58.121 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklisted": false, 2024-03-24T01:24:58.121 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs" 2024-03-24T01:24:58.121 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-24T01:24:58.133 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T01:24:58.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.343+0000 7f4796600640 1 Processor -- start 2024-03-24T01:24:58.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.343+0000 7f4796600640 1 -- start start 2024-03-24T01:24:58.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.345+0000 7f4796600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f47900cc150 0x7f47900ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:58.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.345+0000 7f4796600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47901603e0 0x7f479006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:58.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.345+0000 7f4796600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47900cd490 con 0x7f4790145430 2024-03-24T01:24:58.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.345+0000 7f4796600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f479015f900 con 0x7f47900cc150 2024-03-24T01:24:58.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.345+0000 7f4796600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f479015fbe0 con 0x7f47901603e0 2024-03-24T01:24:58.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.345+0000 7f478fe00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4790145430 0x7f4790071990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:36602/0 (socket says 172.21.15.90:36602) 2024-03-24T01:24:58.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.345+0000 7f478fe00640 1 -- 172.21.15.90:0/3465760498 learned_addr learned my addr 172.21.15.90:0/3465760498 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:24:58.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.346+0000 7f4787e00640 1 --2- 172.21.15.90:0/3465760498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47901603e0 0x7f479006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:58.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.346+0000 7f478f400640 1 --2- 172.21.15.90:0/3465760498 >> v2:172.21.15.120:3300/0 conn(0x7f47900cc150 0x7f47900ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:58.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.346+0000 7f478ea00640 1 -- 172.21.15.90:0/3465760498 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 849704205 0 0) 0x7f47900cd490 con 0x7f4790145430 2024-03-24T01:24:58.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.346+0000 7f478ea00640 1 -- 172.21.15.90:0/3465760498 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4774003520 con 0x7f4790145430 2024-03-24T01:24:58.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.346+0000 7f478ea00640 1 -- 172.21.15.90:0/3465760498 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3702072832 0 0) 0x7f4774003520 con 0x7f4790145430 2024-03-24T01:24:58.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.346+0000 7f478ea00640 1 -- 172.21.15.90:0/3465760498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47901603e0 msgr2=0x7f479006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:58.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.346+0000 7f478ea00640 1 --2- 172.21.15.90:0/3465760498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47901603e0 0x7f479006b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:58.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.346+0000 7f478ea00640 1 -- 172.21.15.90:0/3465760498 >> v2:172.21.15.120:3300/0 conn(0x7f47900cc150 msgr2=0x7f47900ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:58.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.346+0000 7f478ea00640 1 --2- 172.21.15.90:0/3465760498 >> v2:172.21.15.120:3300/0 conn(0x7f47900cc150 0x7f47900ccb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:58.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.346+0000 7f478ea00640 1 -- 172.21.15.90:0/3465760498 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f479015fec0 con 0x7f4790145430 2024-03-24T01:24:58.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.346+0000 7f478ea00640 1 -- 172.21.15.90:0/3465760498 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f47880019f0 con 0x7f4790145430 2024-03-24T01:24:58.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.346+0000 7f478ea00640 1 -- 172.21.15.90:0/3465760498 >> v1:172.21.15.90:6789/0 conn(0x7f4790145430 legacy=0x7f4790071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:58.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.346+0000 7f478ea00640 1 --2- 172.21.15.90:0/3465760498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47901603e0 0x7f47740039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:58.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.346+0000 7f478ea00640 1 --2- 172.21.15.90:0/3465760498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4774003f30 0x7f4774004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:58.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.346+0000 7f478ea00640 1 --2- 172.21.15.90:0/3465760498 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4774004d40 0x7f47740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:58.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.346+0000 7f478ea00640 1 -- 172.21.15.90:0/3465760498 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479015f900 con 0x7f4774004d40 2024-03-24T01:24:58.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.346+0000 7f478ea00640 1 -- 172.21.15.90:0/3465760498 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479015fbe0 con 0x7f4774003f30 2024-03-24T01:24:58.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.347+0000 7f478ea00640 1 -- 172.21.15.90:0/3465760498 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f477400b9c0 con 0x7f47901603e0 2024-03-24T01:24:58.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.347+0000 7f478fe00640 1 --2- 172.21.15.90:0/3465760498 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4774004d40 0x7f47740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:58.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.347+0000 7f478f400640 1 --2- 172.21.15.90:0/3465760498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47901603e0 0x7f47740039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:58.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.347+0000 7f4787e00640 1 --2- 172.21.15.90:0/3465760498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4774003f30 0x7f4774004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:58.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.347+0000 7f478fe00640 1 -- 172.21.15.90:0/3465760498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47901603e0 msgr2=0x7f47740039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:58.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.347+0000 7f478fe00640 1 --2- 172.21.15.90:0/3465760498 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f47901603e0 0x7f47740039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:58.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.347+0000 7f478fe00640 1 -- 172.21.15.90:0/3465760498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4774003f30 msgr2=0x7f4774004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:24:58.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.347+0000 7f478fe00640 1 --2- 172.21.15.90:0/3465760498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4774003f30 0x7f4774004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:58.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.347+0000 7f478fe00640 1 -- 172.21.15.90:0/3465760498 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4774003520 con 0x7f4774004d40 2024-03-24T01:24:58.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.347+0000 7f4787e00640 1 --2- 172.21.15.90:0/3465760498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4774003f30 0x7f4774004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:58.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.348+0000 7f478fe00640 1 --2- 172.21.15.90:0/3465760498 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4774004d40 0x7f47740051c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f478800a8a0 tx=0x7f4774005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd0df2a0e933e85d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:58.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.348+0000 7f478ea00640 1 -- 172.21.15.90:0/3465760498 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478800a4c0 con 0x7f4774004d40 2024-03-24T01:24:58.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.349+0000 7f478ea00640 1 -- 172.21.15.90:0/3465760498 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4788034650 con 0x7f4774004d40 2024-03-24T01:24:58.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.349+0000 7f478ea00640 1 -- 172.21.15.90:0/3465760498 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4788034a80 con 0x7f4774004d40 2024-03-24T01:24:58.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.349+0000 7f4796600640 1 -- 172.21.15.90:0/3465760498 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4774004d40 msgr2=0x7f47740051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:58.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.349+0000 7f4796600640 1 --2- 172.21.15.90:0/3465760498 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4774004d40 0x7f47740051c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f478800a8a0 tx=0x7f4774005830 comp rx=0 tx=0).stop 2024-03-24T01:24:58.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.349+0000 7f478fe00640 1 -- 172.21.15.90:0/3465760498 reap_dead start 2024-03-24T01:24:58.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.349+0000 7f4796600640 1 -- 172.21.15.90:0/3465760498 shutdown_connections 2024-03-24T01:24:58.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.349+0000 7f4796600640 1 -- 172.21.15.90:0/3465760498 >> 172.21.15.90:0/3465760498 conn(0x7f47900b1770 msgr2=0x7f47900b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:58.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.349+0000 7f4796600640 1 -- 172.21.15.90:0/3465760498 shutdown_connections 2024-03-24T01:24:58.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.349+0000 7f4796600640 1 -- 172.21.15.90:0/3465760498 wait complete. 2024-03-24T01:24:58.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.351+0000 7f4796600640 1 Processor -- start 2024-03-24T01:24:58.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.351+0000 7f4796600640 1 -- start start 2024-03-24T01:24:58.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.352+0000 7f4796600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4774004d40 0x7f479015e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:58.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.352+0000 7f4796600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f479015d3e0 0x7f47900abf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:58.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.352+0000 7f4796600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f47901378a0 0x7f4790137d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:58.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.352+0000 7f4796600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4790072220 con 0x7f4774004d40 2024-03-24T01:24:58.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.352+0000 7f4796600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47900720a0 con 0x7f47901378a0 2024-03-24T01:24:58.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.352+0000 7f4796600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47900723a0 con 0x7f479015d3e0 2024-03-24T01:24:58.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.352+0000 7f478f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4774004d40 0x7f479015e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:58.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.352+0000 7f478f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4774004d40 0x7f479015e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37982/0 (socket says 172.21.15.90:37982) 2024-03-24T01:24:58.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.352+0000 7f478f400640 1 -- 172.21.15.90:0/2498885645 learned_addr learned my addr 172.21.15.90:0/2498885645 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:24:58.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.353+0000 7f4787e00640 1 --2- 172.21.15.90:0/2498885645 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f479015d3e0 0x7f47900abf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:58.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.353+0000 7f478fe00640 1 --2- 172.21.15.90:0/2498885645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f47901378a0 0x7f4790137d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:58.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.353+0000 7f478f400640 1 -- 172.21.15.90:0/2498885645 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f479015d3e0 msgr2=0x7f47900abf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:58.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.353+0000 7f478f400640 1 --2- 172.21.15.90:0/2498885645 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f479015d3e0 0x7f47900abf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:58.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.353+0000 7f478f400640 1 -- 172.21.15.90:0/2498885645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f47901378a0 msgr2=0x7f4790137d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:58.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.353+0000 7f478f400640 1 --2- 172.21.15.90:0/2498885645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f47901378a0 0x7f4790137d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:58.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.353+0000 7f478f400640 1 -- 172.21.15.90:0/2498885645 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4790138270 con 0x7f4774004d40 2024-03-24T01:24:58.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.353+0000 7f478fe00640 1 --2- 172.21.15.90:0/2498885645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f47901378a0 0x7f4790137d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:24:58.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.353+0000 7f478f400640 1 --2- 172.21.15.90:0/2498885645 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4774004d40 0x7f479015e940 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f47800116d0 tx=0x7f4780011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:58.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.353+0000 7f4787400640 1 -- 172.21.15.90:0/2498885645 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478000f960 con 0x7f4774004d40 2024-03-24T01:24:58.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.353+0000 7f4787400640 1 -- 172.21.15.90:0/2498885645 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f478000fb00 con 0x7f4774004d40 2024-03-24T01:24:58.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.353+0000 7f4787400640 1 -- 172.21.15.90:0/2498885645 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4780015460 con 0x7f4774004d40 2024-03-24T01:24:58.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.354+0000 7f4796600640 1 -- 172.21.15.90:0/2498885645 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47901f7d80 con 0x7f4774004d40 2024-03-24T01:24:58.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.354+0000 7f4796600640 1 -- 172.21.15.90:0/2498885645 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f479015db80 con 0x7f4774004d40 2024-03-24T01:24:58.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.355+0000 7f4796600640 1 -- 172.21.15.90:0/2498885645 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4790068060 con 0x7f4774004d40 2024-03-24T01:24:58.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.356+0000 7f4787400640 1 -- 172.21.15.90:0/2498885645 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4780015600 con 0x7f4774004d40 2024-03-24T01:24:58.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.356+0000 7f4787400640 1 --2- 172.21.15.90:0/2498885645 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f476c06bc00 0x7f476c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:24:58.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.357+0000 7f4787400640 1 -- 172.21.15.90:0/2498885645 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f478008b4d0 con 0x7f4774004d40 2024-03-24T01:24:58.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.357+0000 7f4787e00640 1 --2- 172.21.15.90:0/2498885645 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f476c06bc00 0x7f476c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:24:58.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.358+0000 7f4787e00640 1 --2- 172.21.15.90:0/2498885645 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f476c06bc00 0x7f476c06e0c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f479013e180 tx=0x7f4778008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:24:58.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.362+0000 7f4787400640 1 -- 172.21.15.90:0/2498885645 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4780019030 con 0x7f4774004d40 2024-03-24T01:24:58.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.597+0000 7f4796600640 1 -- 172.21.15.90:0/2498885645 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f47900b3220 con 0x7f4774004d40 2024-03-24T01:24:58.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.598+0000 7f4787400640 1 -- 172.21.15.90:0/2498885645 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+88 (secure 0 0 0) 0x7f4780059b00 con 0x7f4774004d40 2024-03-24T01:24:58.599 INFO:teuthology.orchestra.run.smithi090.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-24T01:24:58.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.600+0000 7f4796600640 1 -- 172.21.15.90:0/2498885645 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f476c06bc00 msgr2=0x7f476c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:58.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.600+0000 7f4796600640 1 --2- 172.21.15.90:0/2498885645 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f476c06bc00 0x7f476c06e0c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f479013e180 tx=0x7f4778008040 comp rx=0 tx=0).stop 2024-03-24T01:24:58.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.601+0000 7f4796600640 1 -- 172.21.15.90:0/2498885645 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4774004d40 msgr2=0x7f479015e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:24:58.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.601+0000 7f4796600640 1 --2- 172.21.15.90:0/2498885645 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4774004d40 0x7f479015e940 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f47800116d0 tx=0x7f4780011b90 comp rx=0 tx=0).stop 2024-03-24T01:24:58.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.601+0000 7f4796600640 1 -- 172.21.15.90:0/2498885645 shutdown_connections 2024-03-24T01:24:58.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.601+0000 7f4796600640 1 --2- 172.21.15.90:0/2498885645 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f476c06bc00 0x7f476c06e0c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:58.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.601+0000 7f4796600640 1 --2- 172.21.15.90:0/2498885645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f47901378a0 0x7f4790137d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:58.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.601+0000 7f4796600640 1 --2- 172.21.15.90:0/2498885645 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f479015d3e0 0x7f47900abf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:58.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.601+0000 7f4796600640 1 --2- 172.21.15.90:0/2498885645 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4774004d40 0x7f479015e940 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:24:58.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.601+0000 7f4796600640 1 -- 172.21.15.90:0/2498885645 >> 172.21.15.90:0/2498885645 conn(0x7f47900b1770 msgr2=0x7f47900b0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:24:58.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.601+0000 7f4796600640 1 -- 172.21.15.90:0/2498885645 shutdown_connections 2024-03-24T01:24:58.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:24:58.601+0000 7f4796600640 1 -- 172.21.15.90:0/2498885645 wait complete. 2024-03-24T01:24:58.620 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T01:24:58.620 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T01:24:58.620 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi120.front.sepia.ceph.com 2024-03-24T01:24:58.620 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-24T01:24:58.620 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-24T01:24:58.620 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T01:24:58.620 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T01:24:58.620 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T01:24:58.620 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-24T01:24:58.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:58.621 DEBUG:teuthology.orchestra.run.smithi120:> ip addr 2024-03-24T01:24:58.677 INFO:teuthology.orchestra.run.smithi120.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T01:24:58.677 INFO:teuthology.orchestra.run.smithi120.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T01:24:58.677 INFO:teuthology.orchestra.run.smithi120.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T01:24:58.677 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:24:58.677 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 ::1/128 scope host 2024-03-24T01:24:58.677 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:24:58.677 INFO:teuthology.orchestra.run.smithi120.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T01:24:58.677 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:26 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:24:58.677 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f0 2024-03-24T01:24:58.677 INFO:teuthology.orchestra.run.smithi120.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T01:24:58.677 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:27 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:24:58.678 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f1 2024-03-24T01:24:58.678 INFO:teuthology.orchestra.run.smithi120.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T01:24:58.678 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8e brd ff:ff:ff:ff:ff:ff 2024-03-24T01:24:58.678 INFO:teuthology.orchestra.run.smithi120.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T01:24:58.678 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8f brd ff:ff:ff:ff:ff:ff 2024-03-24T01:24:58.678 INFO:teuthology.orchestra.run.smithi120.stdout: inet 172.21.15.120/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T01:24:58.678 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft 42440sec preferred_lft 42440sec 2024-03-24T01:24:58.678 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::ec4:7aff:fe8f:478f/64 scope link 2024-03-24T01:24:58.678 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:24:58.679 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-24T01:24:58.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:58.679 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2024-03-24T01:24:58.679 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2024-03-24T01:24:58.680 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link add name ceph-brx type bridge 2024-03-24T01:24:58.680 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip addr flush dev ceph-brx 2024-03-24T01:24:58.680 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link set ceph-brx up 2024-03-24T01:24:58.680 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-24T01:24:58.680 DEBUG:teuthology.orchestra.run.smithi120:> ') 2024-03-24T01:24:59.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:59.115 DEBUG:teuthology.orchestra.run.smithi120:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T01:24:59.189 INFO:teuthology.orchestra.run.smithi120.stdout:1 2024-03-24T01:24:59.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:59.194 DEBUG:teuthology.orchestra.run.smithi120:> route 2024-03-24T01:24:59.255 INFO:teuthology.orchestra.run.smithi120.stdout:Kernel IP routing table 2024-03-24T01:24:59.255 INFO:teuthology.orchestra.run.smithi120.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T01:24:59.255 INFO:teuthology.orchestra.run.smithi120.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T01:24:59.255 INFO:teuthology.orchestra.run.smithi120.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T01:24:59.255 INFO:teuthology.orchestra.run.smithi120.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T01:24:59.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:59.256 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2024-03-24T01:24:59.256 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2024-03-24T01:24:59.256 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T01:24:59.256 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T01:24:59.256 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T01:24:59.256 DEBUG:teuthology.orchestra.run.smithi120:> ') 2024-03-24T01:24:59.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:24:59.955 DEBUG:teuthology.orchestra.run.smithi120:> ip netns list 2024-03-24T01:25:00.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:00.008 DEBUG:teuthology.orchestra.run.smithi120:> ip netns list-id 2024-03-24T01:25:00.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:00.065 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2024-03-24T01:25:00.065 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2024-03-24T01:25:00.065 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T01:25:00.065 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-24T01:25:00.065 DEBUG:teuthology.orchestra.run.smithi120:> ') 2024-03-24T01:25:00.190 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-24T01:25:00.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:00.190 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2024-03-24T01:25:00.190 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2024-03-24T01:25:00.190 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-24T01:25:00.190 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-24T01:25:00.190 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-24T01:25:00.190 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-24T01:25:00.191 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-24T01:25:00.191 DEBUG:teuthology.orchestra.run.smithi120:> ') 2024-03-24T01:25:00.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:00.465 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2024-03-24T01:25:00.465 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2024-03-24T01:25:00.465 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link set brx.0 up 2024-03-24T01:25:00.465 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link set dev brx.0 master ceph-brx 2024-03-24T01:25:00.465 DEBUG:teuthology.orchestra.run.smithi120:> ') 2024-03-24T01:25:00.620 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-03-24T01:25:00.621 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T01:25:00.621 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-24T01:25:00.662 INFO:teuthology.orchestra.run.smithi120.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-24T01:25:00.662 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T01:25:00.662 DEBUG:teuthology.orchestra.run.smithi120:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-24T01:25:00.756 DEBUG:teuthology.orchestra.run.smithi120:> sudo modprobe fuse 2024-03-24T01:25:00.845 DEBUG:teuthology.orchestra.run.smithi120:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T01:25:00.898 INFO:teuthology.orchestra.run.smithi120.stdout:/proc 2024-03-24T01:25:00.898 INFO:teuthology.orchestra.run.smithi120.stdout:/sys 2024-03-24T01:25:00.899 INFO:teuthology.orchestra.run.smithi120.stdout:/dev 2024-03-24T01:25:00.899 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/kernel/security 2024-03-24T01:25:00.899 INFO:teuthology.orchestra.run.smithi120.stdout:/dev/shm 2024-03-24T01:25:00.899 INFO:teuthology.orchestra.run.smithi120.stdout:/dev/pts 2024-03-24T01:25:00.899 INFO:teuthology.orchestra.run.smithi120.stdout:/run 2024-03-24T01:25:00.899 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/fs/cgroup 2024-03-24T01:25:00.899 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/fs/pstore 2024-03-24T01:25:00.899 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/fs/bpf 2024-03-24T01:25:00.899 INFO:teuthology.orchestra.run.smithi120.stdout:/ 2024-03-24T01:25:00.899 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/fs/selinux 2024-03-24T01:25:00.899 INFO:teuthology.orchestra.run.smithi120.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T01:25:00.899 INFO:teuthology.orchestra.run.smithi120.stdout:/dev/hugepages 2024-03-24T01:25:00.899 INFO:teuthology.orchestra.run.smithi120.stdout:/dev/mqueue 2024-03-24T01:25:00.899 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/kernel/debug 2024-03-24T01:25:00.900 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/kernel/tracing 2024-03-24T01:25:00.900 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/fs/fuse/connections 2024-03-24T01:25:00.900 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/kernel/config 2024-03-24T01:25:00.900 INFO:teuthology.orchestra.run.smithi120.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T01:25:00.900 INFO:teuthology.orchestra.run.smithi120.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T01:25:00.900 INFO:teuthology.orchestra.run.smithi120.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T01:25:00.900 INFO:teuthology.orchestra.run.smithi120.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T01:25:00.900 INFO:teuthology.orchestra.run.smithi120.stdout:/run/user/1000 2024-03-24T01:25:00.900 INFO:teuthology.orchestra.run.smithi120.stdout:/var/lib/ceph 2024-03-24T01:25:00.900 INFO:teuthology.orchestra.run.smithi120.stdout:/var/lib/ceph/osd/ceph-4 2024-03-24T01:25:00.900 INFO:teuthology.orchestra.run.smithi120.stdout:/var/lib/ceph/osd/ceph-5 2024-03-24T01:25:00.900 INFO:teuthology.orchestra.run.smithi120.stdout:/var/lib/ceph/osd/ceph-6 2024-03-24T01:25:00.900 INFO:teuthology.orchestra.run.smithi120.stdout:/var/lib/ceph/osd/ceph-7 2024-03-24T01:25:00.900 INFO:teuthology.orchestra.run.smithi120.stdout:/run/netns 2024-03-24T01:25:00.901 INFO:teuthology.orchestra.run.smithi120.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T01:25:00.901 INFO:teuthology.orchestra.run.smithi120.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T01:25:00.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:00.902 DEBUG:teuthology.orchestra.run.smithi120:> ls /sys/fs/fuse/connections 2024-03-24T01:25:00.950 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-24T01:25:00.950 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-03-24T01:25:00.997 DEBUG:teuthology.orchestra.run.smithi120:> sudo modprobe fuse 2024-03-24T01:25:01.074 DEBUG:teuthology.orchestra.run.smithi120:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T01:25:01.117 INFO:teuthology.orchestra.run.smithi120.stdout:/proc 2024-03-24T01:25:01.117 INFO:teuthology.orchestra.run.smithi120.stdout:/sys 2024-03-24T01:25:01.117 INFO:teuthology.orchestra.run.smithi120.stdout:/dev 2024-03-24T01:25:01.118 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/kernel/security 2024-03-24T01:25:01.118 INFO:teuthology.orchestra.run.smithi120.stdout:/dev/shm 2024-03-24T01:25:01.118 INFO:teuthology.orchestra.run.smithi120.stdout:/dev/pts 2024-03-24T01:25:01.118 INFO:teuthology.orchestra.run.smithi120.stdout:/run 2024-03-24T01:25:01.118 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/fs/cgroup 2024-03-24T01:25:01.118 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/fs/pstore 2024-03-24T01:25:01.118 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/fs/bpf 2024-03-24T01:25:01.118 INFO:teuthology.orchestra.run.smithi120.stdout:/ 2024-03-24T01:25:01.118 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/fs/selinux 2024-03-24T01:25:01.118 INFO:teuthology.orchestra.run.smithi120.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T01:25:01.118 INFO:teuthology.orchestra.run.smithi120.stdout:/dev/hugepages 2024-03-24T01:25:01.118 INFO:teuthology.orchestra.run.smithi120.stdout:/dev/mqueue 2024-03-24T01:25:01.118 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/kernel/debug 2024-03-24T01:25:01.119 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/kernel/tracing 2024-03-24T01:25:01.119 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/fs/fuse/connections 2024-03-24T01:25:01.119 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/kernel/config 2024-03-24T01:25:01.119 INFO:teuthology.orchestra.run.smithi120.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T01:25:01.119 INFO:teuthology.orchestra.run.smithi120.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T01:25:01.119 INFO:teuthology.orchestra.run.smithi120.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T01:25:01.119 INFO:teuthology.orchestra.run.smithi120.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T01:25:01.119 INFO:teuthology.orchestra.run.smithi120.stdout:/run/user/1000 2024-03-24T01:25:01.119 INFO:teuthology.orchestra.run.smithi120.stdout:/var/lib/ceph 2024-03-24T01:25:01.119 INFO:teuthology.orchestra.run.smithi120.stdout:/var/lib/ceph/osd/ceph-4 2024-03-24T01:25:01.119 INFO:teuthology.orchestra.run.smithi120.stdout:/var/lib/ceph/osd/ceph-5 2024-03-24T01:25:01.119 INFO:teuthology.orchestra.run.smithi120.stdout:/var/lib/ceph/osd/ceph-6 2024-03-24T01:25:01.119 INFO:teuthology.orchestra.run.smithi120.stdout:/var/lib/ceph/osd/ceph-7 2024-03-24T01:25:01.119 INFO:teuthology.orchestra.run.smithi120.stdout:/run/netns 2024-03-24T01:25:01.119 INFO:teuthology.orchestra.run.smithi120.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T01:25:01.120 INFO:teuthology.orchestra.run.smithi120.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T01:25:01.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:01.121 DEBUG:teuthology.orchestra.run.smithi120:> ls /sys/fs/fuse/connections 2024-03-24T01:25:01.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi120.stderr:2024-03-24T01:25:01.178+0000 7f795c17a480 -1 init, newargv = 0x5613431a7c10 newargc=15 2024-03-24T01:25:01.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi120.stderr:ceph-fuse[43083]: starting ceph client 2024-03-24T01:25:01.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi120.stderr:ceph-fuse[43083]: starting fuse 2024-03-24T01:25:01.207 INFO:teuthology.orchestra.run.smithi120.stdout:41 2024-03-24T01:25:01.207 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-24T01:25:01.207 DEBUG:teuthology.orchestra.run.smithi120:> sudo stdin-killer -- python3 -c ' 2024-03-24T01:25:01.208 DEBUG:teuthology.orchestra.run.smithi120:> import glob 2024-03-24T01:25:01.208 DEBUG:teuthology.orchestra.run.smithi120:> import re 2024-03-24T01:25:01.208 DEBUG:teuthology.orchestra.run.smithi120:> import os 2024-03-24T01:25:01.208 DEBUG:teuthology.orchestra.run.smithi120:> import subprocess 2024-03-24T01:25:01.208 DEBUG:teuthology.orchestra.run.smithi120:> 2024-03-24T01:25:01.208 DEBUG:teuthology.orchestra.run.smithi120:> def _find_admin_socket(client_name): 2024-03-24T01:25:01.208 DEBUG:teuthology.orchestra.run.smithi120:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-03-24T01:25:01.208 DEBUG:teuthology.orchestra.run.smithi120:> files = glob.glob(asok_path) 2024-03-24T01:25:01.208 DEBUG:teuthology.orchestra.run.smithi120:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-03-24T01:25:01.208 DEBUG:teuthology.orchestra.run.smithi120:> 2024-03-24T01:25:01.208 DEBUG:teuthology.orchestra.run.smithi120:> # Given a non-glob path, it better be there 2024-03-24T01:25:01.208 DEBUG:teuthology.orchestra.run.smithi120:> if "*" not in asok_path: 2024-03-24T01:25:01.208 DEBUG:teuthology.orchestra.run.smithi120:> assert(len(files) == 1) 2024-03-24T01:25:01.209 DEBUG:teuthology.orchestra.run.smithi120:> return files[0] 2024-03-24T01:25:01.209 DEBUG:teuthology.orchestra.run.smithi120:> 2024-03-24T01:25:01.209 DEBUG:teuthology.orchestra.run.smithi120:> for f in files: 2024-03-24T01:25:01.209 DEBUG:teuthology.orchestra.run.smithi120:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T01:25:01.209 DEBUG:teuthology.orchestra.run.smithi120:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T01:25:01.209 DEBUG:teuthology.orchestra.run.smithi120:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T01:25:01.209 DEBUG:teuthology.orchestra.run.smithi120:> contents = proc_f.read() 2024-03-24T01:25:01.209 DEBUG:teuthology.orchestra.run.smithi120:> if mountpoint in contents: 2024-03-24T01:25:01.209 DEBUG:teuthology.orchestra.run.smithi120:> return f 2024-03-24T01:25:01.209 DEBUG:teuthology.orchestra.run.smithi120:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T01:25:01.209 DEBUG:teuthology.orchestra.run.smithi120:> 2024-03-24T01:25:01.209 DEBUG:teuthology.orchestra.run.smithi120:> print(_find_admin_socket("client.1")) 2024-03-24T01:25:01.209 DEBUG:teuthology.orchestra.run.smithi120:> ' 2024-03-24T01:25:01.395 INFO:teuthology.orchestra.run.smithi120.stdout:/var/run/ceph/ceph-client.1.43083.asok 2024-03-24T01:25:01.399 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:25:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T01:25:01.416 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.43083.asok 2024-03-24T01:25:01.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:01.416 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.43083.asok status 2024-03-24T01:25:01.646 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-03-24T01:25:01.646 INFO:teuthology.orchestra.run.smithi120.stdout: "metadata": { 2024-03-24T01:25:01.646 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_sha1": "3be9a5c9bc793e11e2800a8c0c696e8b46742033", 2024-03-24T01:25:01.646 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-24T01:25:01.647 INFO:teuthology.orchestra.run.smithi120.stdout: "entity_id": "1", 2024-03-24T01:25:01.647 INFO:teuthology.orchestra.run.smithi120.stdout: "hostname": "smithi120", 2024-03-24T01:25:01.647 INFO:teuthology.orchestra.run.smithi120.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-03-24T01:25:01.647 INFO:teuthology.orchestra.run.smithi120.stdout: "pid": "43083", 2024-03-24T01:25:01.647 INFO:teuthology.orchestra.run.smithi120.stdout: "root": "/" 2024-03-24T01:25:01.647 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-24T01:25:01.647 INFO:teuthology.orchestra.run.smithi120.stdout: "dentry_count": 0, 2024-03-24T01:25:01.647 INFO:teuthology.orchestra.run.smithi120.stdout: "dentry_pinned_count": 0, 2024-03-24T01:25:01.647 INFO:teuthology.orchestra.run.smithi120.stdout: "id": 4702, 2024-03-24T01:25:01.647 INFO:teuthology.orchestra.run.smithi120.stdout: "inst": { 2024-03-24T01:25:01.647 INFO:teuthology.orchestra.run.smithi120.stdout: "name": { 2024-03-24T01:25:01.647 INFO:teuthology.orchestra.run.smithi120.stdout: "type": "client", 2024-03-24T01:25:01.647 INFO:teuthology.orchestra.run.smithi120.stdout: "num": 4702 2024-03-24T01:25:01.647 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-24T01:25:01.648 INFO:teuthology.orchestra.run.smithi120.stdout: "addr": { 2024-03-24T01:25:01.648 INFO:teuthology.orchestra.run.smithi120.stdout: "type": "v1", 2024-03-24T01:25:01.648 INFO:teuthology.orchestra.run.smithi120.stdout: "addr": "192.168.0.1:0", 2024-03-24T01:25:01.648 INFO:teuthology.orchestra.run.smithi120.stdout: "nonce": 3249212776 2024-03-24T01:25:01.648 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-24T01:25:01.648 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-24T01:25:01.648 INFO:teuthology.orchestra.run.smithi120.stdout: "addr": { 2024-03-24T01:25:01.648 INFO:teuthology.orchestra.run.smithi120.stdout: "type": "v1", 2024-03-24T01:25:01.648 INFO:teuthology.orchestra.run.smithi120.stdout: "addr": "192.168.0.1:0", 2024-03-24T01:25:01.648 INFO:teuthology.orchestra.run.smithi120.stdout: "nonce": 3249212776 2024-03-24T01:25:01.648 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-24T01:25:01.648 INFO:teuthology.orchestra.run.smithi120.stdout: "inst_str": "client.4702 192.168.0.1:0/3249212776", 2024-03-24T01:25:01.648 INFO:teuthology.orchestra.run.smithi120.stdout: "addr_str": "192.168.0.1:0/3249212776", 2024-03-24T01:25:01.648 INFO:teuthology.orchestra.run.smithi120.stdout: "inode_count": 1, 2024-03-24T01:25:01.648 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_epoch": 8, 2024-03-24T01:25:01.649 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_epoch": 27, 2024-03-24T01:25:01.649 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_epoch_barrier": 0, 2024-03-24T01:25:01.649 INFO:teuthology.orchestra.run.smithi120.stdout: "blocklisted": false, 2024-03-24T01:25:01.649 INFO:teuthology.orchestra.run.smithi120.stdout: "fs_name": "cephfs" 2024-03-24T01:25:01.649 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-03-24T01:25:01.662 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T01:25:01.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.880+0000 7fb23b200640 1 Processor -- start 2024-03-24T01:25:01.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.880+0000 7fb23b200640 1 -- start start 2024-03-24T01:25:01.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.882+0000 7fb23b200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb23c15da70 0x7fb23c0b54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:01.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.882+0000 7fb23b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb23c15de50 0x7fb23c06b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:01.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.882+0000 7fb23b200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb23c0d0470 0x7fb23c06b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:01.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.882+0000 7fb23b200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb23c0d0950 con 0x7fb23c0d0470 2024-03-24T01:25:01.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.882+0000 7fb23b200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb23c0d0c30 con 0x7fb23c15da70 2024-03-24T01:25:01.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.882+0000 7fb23b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb23c0d0f10 con 0x7fb23c15de50 2024-03-24T01:25:01.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.883+0000 7fb23a800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb23c0d0470 0x7fb23c06b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:01.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.883+0000 7fb23a800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb23c0d0470 0x7fb23c06b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37418/0 (socket says 172.21.15.90:37418) 2024-03-24T01:25:01.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.883+0000 7fb23a800640 1 -- 172.21.15.90:0/3991830115 learned_addr learned my addr 172.21.15.90:0/3991830115 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:01.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.883+0000 7fb239400640 1 --2- 172.21.15.90:0/3991830115 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb23c15de50 0x7fb23c06b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:01.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.883+0000 7fb239e00640 1 --2- 172.21.15.90:0/3991830115 >> v2:172.21.15.120:3300/0 conn(0x7fb23c15da70 0x7fb23c0b54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:01.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.883+0000 7fb23a800640 1 -- 172.21.15.90:0/3991830115 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb23c15de50 msgr2=0x7fb23c06b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:01.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.883+0000 7fb23a800640 1 --2- 172.21.15.90:0/3991830115 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb23c15de50 0x7fb23c06b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:01.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.883+0000 7fb23a800640 1 -- 172.21.15.90:0/3991830115 >> v2:172.21.15.120:3300/0 conn(0x7fb23c15da70 msgr2=0x7fb23c0b54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:01.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.883+0000 7fb23a800640 1 --2- 172.21.15.90:0/3991830115 >> v2:172.21.15.120:3300/0 conn(0x7fb23c15da70 0x7fb23c0b54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:01.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.883+0000 7fb23a800640 1 -- 172.21.15.90:0/3991830115 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb23c0d11f0 con 0x7fb23c0d0470 2024-03-24T01:25:01.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.883+0000 7fb239e00640 1 --2- 172.21.15.90:0/3991830115 >> v2:172.21.15.120:3300/0 conn(0x7fb23c15da70 0x7fb23c0b54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:01.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.884+0000 7fb23a800640 1 --2- 172.21.15.90:0/3991830115 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb23c0d0470 0x7fb23c06b600 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb23c0d1850 tx=0x7fb23402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4921854f4b14dfb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:01.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.885+0000 7fb238a00640 1 -- 172.21.15.90:0/3991830115 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb234035020 con 0x7fb23c0d0470 2024-03-24T01:25:01.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.885+0000 7fb238a00640 1 -- 172.21.15.90:0/3991830115 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb234031870 con 0x7fb23c0d0470 2024-03-24T01:25:01.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.885+0000 7fb238a00640 1 -- 172.21.15.90:0/3991830115 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb234031c40 con 0x7fb23c0d0470 2024-03-24T01:25:01.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.885+0000 7fb23b200640 1 -- 172.21.15.90:0/3991830115 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb23c0d0470 msgr2=0x7fb23c06b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:01.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.885+0000 7fb23b200640 1 --2- 172.21.15.90:0/3991830115 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb23c0d0470 0x7fb23c06b600 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb23c0d1850 tx=0x7fb23402b910 comp rx=0 tx=0).stop 2024-03-24T01:25:01.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.885+0000 7fb23b200640 1 -- 172.21.15.90:0/3991830115 shutdown_connections 2024-03-24T01:25:01.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.885+0000 7fb23b200640 1 --2- 172.21.15.90:0/3991830115 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb23c15de50 0x7fb23c06b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:01.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.885+0000 7fb23b200640 1 --2- 172.21.15.90:0/3991830115 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb23c0d0470 0x7fb23c06b600 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:01.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.885+0000 7fb23b200640 1 --2- 172.21.15.90:0/3991830115 >> v2:172.21.15.120:3300/0 conn(0x7fb23c15da70 0x7fb23c0b54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:01.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.885+0000 7fb23b200640 1 -- 172.21.15.90:0/3991830115 >> 172.21.15.90:0/3991830115 conn(0x7fb23c0b5e00 msgr2=0x7fb23c099a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:01.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.885+0000 7fb23b200640 1 -- 172.21.15.90:0/3991830115 shutdown_connections 2024-03-24T01:25:01.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.886+0000 7fb23b200640 1 -- 172.21.15.90:0/3991830115 wait complete. 2024-03-24T01:25:01.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.887+0000 7fb23b200640 1 Processor -- start 2024-03-24T01:25:01.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.887+0000 7fb23b200640 1 -- start start 2024-03-24T01:25:01.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.888+0000 7fb23b200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb23c0d0470 0x7fb23c15ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:01.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.888+0000 7fb23b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb23c15da70 0x7fb23c154c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:01.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.888+0000 7fb23b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb23c15de50 0x7fb23c0aed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:01.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.888+0000 7fb23b200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23c0b5b60 con 0x7fb23c15da70 2024-03-24T01:25:01.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.888+0000 7fb23b200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23c0b59e0 con 0x7fb23c0d0470 2024-03-24T01:25:01.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.888+0000 7fb23b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb23c1617e0 con 0x7fb23c15de50 2024-03-24T01:25:01.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.889+0000 7fb239400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb23c15da70 0x7fb23c154c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:01.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.889+0000 7fb239e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb23c0d0470 0x7fb23c15ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:01.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.889+0000 7fb239400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb23c15da70 0x7fb23c154c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37428/0 (socket says 172.21.15.90:37428) 2024-03-24T01:25:01.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.889+0000 7fb239400640 1 -- 172.21.15.90:0/1318419737 learned_addr learned my addr 172.21.15.90:0/1318419737 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:01.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.889+0000 7fb23a800640 1 --2- 172.21.15.90:0/1318419737 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb23c15de50 0x7fb23c0aed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:01.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.889+0000 7fb239400640 1 -- 172.21.15.90:0/1318419737 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb23c15de50 msgr2=0x7fb23c0aed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:01.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.889+0000 7fb239400640 1 --2- 172.21.15.90:0/1318419737 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb23c15de50 0x7fb23c0aed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:01.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.889+0000 7fb239400640 1 -- 172.21.15.90:0/1318419737 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb23c0d0470 msgr2=0x7fb23c15ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:01.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.889+0000 7fb239400640 1 --2- 172.21.15.90:0/1318419737 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb23c0d0470 0x7fb23c15ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:01.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.889+0000 7fb239400640 1 -- 172.21.15.90:0/1318419737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb23c0af290 con 0x7fb23c15da70 2024-03-24T01:25:01.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.890+0000 7fb239400640 1 --2- 172.21.15.90:0/1318419737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb23c15da70 0x7fb23c154c90 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb224007e30 tx=0x7fb22400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:01.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.890+0000 7fb232000640 1 -- 172.21.15.90:0/1318419737 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb224014620 con 0x7fb23c15da70 2024-03-24T01:25:01.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.890+0000 7fb232000640 1 -- 172.21.15.90:0/1318419737 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb224019070 con 0x7fb23c15da70 2024-03-24T01:25:01.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.890+0000 7fb232000640 1 -- 172.21.15.90:0/1318419737 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb224014a00 con 0x7fb23c15da70 2024-03-24T01:25:01.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.890+0000 7fb23b200640 1 -- 172.21.15.90:0/1318419737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb23c0af560 con 0x7fb23c15da70 2024-03-24T01:25:01.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.890+0000 7fb23b200640 1 -- 172.21.15.90:0/1318419737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb23c0afb20 con 0x7fb23c15da70 2024-03-24T01:25:01.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.891+0000 7fb23b200640 1 -- 172.21.15.90:0/1318419737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb23c04f400 con 0x7fb23c15da70 2024-03-24T01:25:01.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.892+0000 7fb232000640 1 -- 172.21.15.90:0/1318419737 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb224014ba0 con 0x7fb23c15da70 2024-03-24T01:25:01.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.893+0000 7fb232000640 1 --2- 172.21.15.90:0/1318419737 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb21806bbd0 0x7fb21806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:01.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.893+0000 7fb232000640 1 -- 172.21.15.90:0/1318419737 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7fb22408b560 con 0x7fb23c15da70 2024-03-24T01:25:01.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.894+0000 7fb239e00640 1 --2- 172.21.15.90:0/1318419737 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb21806bbd0 0x7fb21806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:01.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.895+0000 7fb239e00640 1 --2- 172.21.15.90:0/1318419737 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb21806bbd0 0x7fb21806e090 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb22c00ab60 tx=0x7fb22c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:01.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:01.898+0000 7fb232000640 1 -- 172.21.15.90:0/1318419737 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb2240595e0 con 0x7fb23c15da70 2024-03-24T01:25:02.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:02.138+0000 7fb23b200640 1 -- 172.21.15.90:0/1318419737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb23c06bbf0 con 0x7fb23c15da70 2024-03-24T01:25:02.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:02.139+0000 7fb232000640 1 -- 172.21.15.90:0/1318419737 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+88 (secure 0 0 0) 0x7fb22405d680 con 0x7fb23c15da70 2024-03-24T01:25:02.139 INFO:teuthology.orchestra.run.smithi090.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-24T01:25:02.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:02.141+0000 7fb23b200640 1 -- 172.21.15.90:0/1318419737 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb21806bbd0 msgr2=0x7fb21806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:02.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:02.141+0000 7fb23b200640 1 --2- 172.21.15.90:0/1318419737 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb21806bbd0 0x7fb21806e090 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb22c00ab60 tx=0x7fb22c008040 comp rx=0 tx=0).stop 2024-03-24T01:25:02.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:02.141+0000 7fb23b200640 1 -- 172.21.15.90:0/1318419737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb23c15da70 msgr2=0x7fb23c154c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:02.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:02.141+0000 7fb23b200640 1 --2- 172.21.15.90:0/1318419737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb23c15da70 0x7fb23c154c90 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb224007e30 tx=0x7fb22400e4b0 comp rx=0 tx=0).stop 2024-03-24T01:25:02.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:02.142+0000 7fb23b200640 1 -- 172.21.15.90:0/1318419737 shutdown_connections 2024-03-24T01:25:02.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:02.142+0000 7fb23b200640 1 --2- 172.21.15.90:0/1318419737 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb21806bbd0 0x7fb21806e090 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:02.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:02.142+0000 7fb23b200640 1 --2- 172.21.15.90:0/1318419737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb23c15da70 0x7fb23c154c90 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:02.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:02.142+0000 7fb23b200640 1 --2- 172.21.15.90:0/1318419737 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb23c15de50 0x7fb23c0aed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:02.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:02.142+0000 7fb23b200640 1 --2- 172.21.15.90:0/1318419737 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb23c0d0470 0x7fb23c15ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:02.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:02.142+0000 7fb23b200640 1 -- 172.21.15.90:0/1318419737 >> 172.21.15.90:0/1318419737 conn(0x7fb23c0b5e00 msgr2=0x7fb23c161bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:02.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:02.142+0000 7fb23b200640 1 -- 172.21.15.90:0/1318419737 shutdown_connections 2024-03-24T01:25:02.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:02.142+0000 7fb23b200640 1 -- 172.21.15.90:0/1318419737 wait complete. 2024-03-24T01:25:02.160 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T01:25:02.160 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T01:25:02.160 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi120.front.sepia.ceph.com 2024-03-24T01:25:02.161 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-03-24T01:25:02.161 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-24T01:25:02.161 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T01:25:02.161 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T01:25:02.161 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T01:25:02.161 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-03-24T01:25:02.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:02.161 DEBUG:teuthology.orchestra.run.smithi120:> ip addr 2024-03-24T01:25:02.218 INFO:teuthology.orchestra.run.smithi120.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T01:25:02.218 INFO:teuthology.orchestra.run.smithi120.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T01:25:02.218 INFO:teuthology.orchestra.run.smithi120.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T01:25:02.218 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:25:02.218 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 ::1/128 scope host 2024-03-24T01:25:02.218 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:25:02.218 INFO:teuthology.orchestra.run.smithi120.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T01:25:02.218 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:26 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:25:02.219 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f0 2024-03-24T01:25:02.219 INFO:teuthology.orchestra.run.smithi120.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T01:25:02.219 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:d8:a1:27 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:25:02.219 INFO:teuthology.orchestra.run.smithi120.stdout: altname enp7s0f1 2024-03-24T01:25:02.219 INFO:teuthology.orchestra.run.smithi120.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T01:25:02.219 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8e brd ff:ff:ff:ff:ff:ff 2024-03-24T01:25:02.219 INFO:teuthology.orchestra.run.smithi120.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T01:25:02.219 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 0c:c4:7a:8f:47:8f brd ff:ff:ff:ff:ff:ff 2024-03-24T01:25:02.219 INFO:teuthology.orchestra.run.smithi120.stdout: inet 172.21.15.120/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T01:25:02.219 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft 42437sec preferred_lft 42437sec 2024-03-24T01:25:02.219 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::ec4:7aff:fe8f:478f/64 scope link 2024-03-24T01:25:02.219 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:25:02.219 INFO:teuthology.orchestra.run.smithi120.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T01:25:02.219 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether d6:17:f9:cf:ed:3a brd ff:ff:ff:ff:ff:ff 2024-03-24T01:25:02.219 INFO:teuthology.orchestra.run.smithi120.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T01:25:02.220 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:25:02.220 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::8cac:d3ff:fe8b:8f0d/64 scope link 2024-03-24T01:25:02.220 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:25:02.220 INFO:teuthology.orchestra.run.smithi120.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T01:25:02.220 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether d6:17:f9:cf:ed:3a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T01:25:02.220 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::d417:f9ff:fecf:ed3a/64 scope link 2024-03-24T01:25:02.220 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:25:02.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:02.220 DEBUG:teuthology.orchestra.run.smithi120:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T01:25:02.301 INFO:teuthology.orchestra.run.smithi120.stdout:1 2024-03-24T01:25:02.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:02.305 DEBUG:teuthology.orchestra.run.smithi120:> route 2024-03-24T01:25:02.351 INFO:teuthology.orchestra.run.smithi120.stdout:Kernel IP routing table 2024-03-24T01:25:02.351 INFO:teuthology.orchestra.run.smithi120.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T01:25:02.351 INFO:teuthology.orchestra.run.smithi120.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T01:25:02.351 INFO:teuthology.orchestra.run.smithi120.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T01:25:02.351 INFO:teuthology.orchestra.run.smithi120.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T01:25:02.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:02.351 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2024-03-24T01:25:02.351 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2024-03-24T01:25:02.352 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T01:25:02.352 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T01:25:02.352 DEBUG:teuthology.orchestra.run.smithi120:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T01:25:02.352 DEBUG:teuthology.orchestra.run.smithi120:> ') 2024-03-24T01:25:02.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:02.576 DEBUG:teuthology.orchestra.run.smithi120:> ip netns list 2024-03-24T01:25:02.622 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-24T01:25:02.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:02.623 DEBUG:teuthology.orchestra.run.smithi120:> ip netns list-id 2024-03-24T01:25:02.712 INFO:teuthology.orchestra.run.smithi120.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-03-24T01:25:02.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:02.713 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2024-03-24T01:25:02.713 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2024-03-24T01:25:02.713 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-24T01:25:02.713 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-03-24T01:25:02.714 DEBUG:teuthology.orchestra.run.smithi120:> ') 2024-03-24T01:25:02.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:02.830 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-24T01:25:02.912 INFO:teuthology.orchestra.run.smithi120.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T01:25:02.912 INFO:teuthology.orchestra.run.smithi120.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T01:25:02.912 INFO:teuthology.orchestra.run.smithi120.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T01:25:02.912 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:25:02.912 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 ::1/128 scope host 2024-03-24T01:25:02.912 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:25:02.912 INFO:teuthology.orchestra.run.smithi120.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T01:25:02.912 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 46:92:9a:5e:8e:28 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-24T01:25:02.912 INFO:teuthology.orchestra.run.smithi120.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-24T01:25:02.912 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:25:02.912 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::4492:9aff:fe5e:8e28/64 scope link 2024-03-24T01:25:02.913 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:25:02.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:02.916 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-24T01:25:02.994 INFO:teuthology.orchestra.run.smithi120.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T01:25:02.994 INFO:teuthology.orchestra.run.smithi120.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T01:25:02.994 INFO:teuthology.orchestra.run.smithi120.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T01:25:02.995 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:25:02.995 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 ::1/128 scope host 2024-03-24T01:25:02.995 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:25:02.995 INFO:teuthology.orchestra.run.smithi120.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T01:25:02.995 INFO:teuthology.orchestra.run.smithi120.stdout: link/ether 46:92:9a:5e:8e:28 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-24T01:25:02.995 INFO:teuthology.orchestra.run.smithi120.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-24T01:25:02.995 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:25:02.995 INFO:teuthology.orchestra.run.smithi120.stdout: inet6 fe80::4492:9aff:fe5e:8e28/64 scope link 2024-03-24T01:25:02.995 INFO:teuthology.orchestra.run.smithi120.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:25:02.998 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-03-24T01:25:02.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:02.999 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2024-03-24T01:25:02.999 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2024-03-24T01:25:02.999 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-03-24T01:25:02.999 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-24T01:25:02.999 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-03-24T01:25:02.999 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-03-24T01:25:02.999 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-03-24T01:25:02.999 DEBUG:teuthology.orchestra.run.smithi120:> ') 2024-03-24T01:25:03.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:03.248 DEBUG:teuthology.orchestra.run.smithi120:> (cd / && exec bash -c ' 2024-03-24T01:25:03.248 DEBUG:teuthology.orchestra.run.smithi120:> set -e 2024-03-24T01:25:03.248 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link set brx.1 up 2024-03-24T01:25:03.248 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link set dev brx.1 master ceph-brx 2024-03-24T01:25:03.248 DEBUG:teuthology.orchestra.run.smithi120:> ') 2024-03-24T01:25:03.398 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-03-24T01:25:03.398 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T01:25:03.398 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-03-24T01:25:03.443 INFO:teuthology.orchestra.run.smithi120.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-03-24T01:25:03.443 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T01:25:03.443 DEBUG:teuthology.orchestra.run.smithi120:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-03-24T01:25:03.536 DEBUG:teuthology.orchestra.run.smithi120:> sudo modprobe fuse 2024-03-24T01:25:03.623 DEBUG:teuthology.orchestra.run.smithi120:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T01:25:03.672 INFO:teuthology.orchestra.run.smithi120.stdout:/proc 2024-03-24T01:25:03.672 INFO:teuthology.orchestra.run.smithi120.stdout:/sys 2024-03-24T01:25:03.673 INFO:teuthology.orchestra.run.smithi120.stdout:/dev 2024-03-24T01:25:03.673 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/kernel/security 2024-03-24T01:25:03.673 INFO:teuthology.orchestra.run.smithi120.stdout:/dev/shm 2024-03-24T01:25:03.673 INFO:teuthology.orchestra.run.smithi120.stdout:/dev/pts 2024-03-24T01:25:03.673 INFO:teuthology.orchestra.run.smithi120.stdout:/run 2024-03-24T01:25:03.673 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/fs/cgroup 2024-03-24T01:25:03.673 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/fs/pstore 2024-03-24T01:25:03.673 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/fs/bpf 2024-03-24T01:25:03.673 INFO:teuthology.orchestra.run.smithi120.stdout:/ 2024-03-24T01:25:03.673 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/fs/selinux 2024-03-24T01:25:03.673 INFO:teuthology.orchestra.run.smithi120.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T01:25:03.673 INFO:teuthology.orchestra.run.smithi120.stdout:/dev/hugepages 2024-03-24T01:25:03.673 INFO:teuthology.orchestra.run.smithi120.stdout:/dev/mqueue 2024-03-24T01:25:03.673 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/kernel/debug 2024-03-24T01:25:03.673 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/kernel/tracing 2024-03-24T01:25:03.674 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/fs/fuse/connections 2024-03-24T01:25:03.674 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/kernel/config 2024-03-24T01:25:03.674 INFO:teuthology.orchestra.run.smithi120.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T01:25:03.674 INFO:teuthology.orchestra.run.smithi120.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T01:25:03.674 INFO:teuthology.orchestra.run.smithi120.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T01:25:03.674 INFO:teuthology.orchestra.run.smithi120.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T01:25:03.674 INFO:teuthology.orchestra.run.smithi120.stdout:/run/user/1000 2024-03-24T01:25:03.674 INFO:teuthology.orchestra.run.smithi120.stdout:/var/lib/ceph 2024-03-24T01:25:03.674 INFO:teuthology.orchestra.run.smithi120.stdout:/var/lib/ceph/osd/ceph-4 2024-03-24T01:25:03.674 INFO:teuthology.orchestra.run.smithi120.stdout:/var/lib/ceph/osd/ceph-5 2024-03-24T01:25:03.674 INFO:teuthology.orchestra.run.smithi120.stdout:/var/lib/ceph/osd/ceph-6 2024-03-24T01:25:03.674 INFO:teuthology.orchestra.run.smithi120.stdout:/var/lib/ceph/osd/ceph-7 2024-03-24T01:25:03.674 INFO:teuthology.orchestra.run.smithi120.stdout:/run/netns 2024-03-24T01:25:03.674 INFO:teuthology.orchestra.run.smithi120.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T01:25:03.675 INFO:teuthology.orchestra.run.smithi120.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T01:25:03.675 INFO:teuthology.orchestra.run.smithi120.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-24T01:25:03.675 INFO:teuthology.orchestra.run.smithi120.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-24T01:25:03.675 INFO:teuthology.orchestra.run.smithi120.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-24T01:25:03.676 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:03.676 DEBUG:teuthology.orchestra.run.smithi120:> ls /sys/fs/fuse/connections 2024-03-24T01:25:03.720 INFO:teuthology.orchestra.run.smithi120.stdout:41 2024-03-24T01:25:03.720 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [41] 2024-03-24T01:25:03.720 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-03-24T01:25:03.767 DEBUG:teuthology.orchestra.run.smithi120:> sudo modprobe fuse 2024-03-24T01:25:03.850 DEBUG:teuthology.orchestra.run.smithi120:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T01:25:03.892 INFO:teuthology.orchestra.run.smithi120.stdout:/proc 2024-03-24T01:25:03.893 INFO:teuthology.orchestra.run.smithi120.stdout:/sys 2024-03-24T01:25:03.893 INFO:teuthology.orchestra.run.smithi120.stdout:/dev 2024-03-24T01:25:03.893 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/kernel/security 2024-03-24T01:25:03.893 INFO:teuthology.orchestra.run.smithi120.stdout:/dev/shm 2024-03-24T01:25:03.893 INFO:teuthology.orchestra.run.smithi120.stdout:/dev/pts 2024-03-24T01:25:03.893 INFO:teuthology.orchestra.run.smithi120.stdout:/run 2024-03-24T01:25:03.893 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/fs/cgroup 2024-03-24T01:25:03.893 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/fs/pstore 2024-03-24T01:25:03.893 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/fs/bpf 2024-03-24T01:25:03.893 INFO:teuthology.orchestra.run.smithi120.stdout:/ 2024-03-24T01:25:03.893 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/fs/selinux 2024-03-24T01:25:03.893 INFO:teuthology.orchestra.run.smithi120.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T01:25:03.893 INFO:teuthology.orchestra.run.smithi120.stdout:/dev/hugepages 2024-03-24T01:25:03.893 INFO:teuthology.orchestra.run.smithi120.stdout:/dev/mqueue 2024-03-24T01:25:03.894 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/kernel/debug 2024-03-24T01:25:03.894 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/kernel/tracing 2024-03-24T01:25:03.894 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/fs/fuse/connections 2024-03-24T01:25:03.894 INFO:teuthology.orchestra.run.smithi120.stdout:/sys/kernel/config 2024-03-24T01:25:03.894 INFO:teuthology.orchestra.run.smithi120.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T01:25:03.894 INFO:teuthology.orchestra.run.smithi120.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T01:25:03.894 INFO:teuthology.orchestra.run.smithi120.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T01:25:03.894 INFO:teuthology.orchestra.run.smithi120.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T01:25:03.894 INFO:teuthology.orchestra.run.smithi120.stdout:/run/user/1000 2024-03-24T01:25:03.894 INFO:teuthology.orchestra.run.smithi120.stdout:/var/lib/ceph 2024-03-24T01:25:03.894 INFO:teuthology.orchestra.run.smithi120.stdout:/var/lib/ceph/osd/ceph-4 2024-03-24T01:25:03.894 INFO:teuthology.orchestra.run.smithi120.stdout:/var/lib/ceph/osd/ceph-5 2024-03-24T01:25:03.894 INFO:teuthology.orchestra.run.smithi120.stdout:/var/lib/ceph/osd/ceph-6 2024-03-24T01:25:03.894 INFO:teuthology.orchestra.run.smithi120.stdout:/var/lib/ceph/osd/ceph-7 2024-03-24T01:25:03.894 INFO:teuthology.orchestra.run.smithi120.stdout:/run/netns 2024-03-24T01:25:03.895 INFO:teuthology.orchestra.run.smithi120.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T01:25:03.895 INFO:teuthology.orchestra.run.smithi120.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T01:25:03.895 INFO:teuthology.orchestra.run.smithi120.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-24T01:25:03.895 INFO:teuthology.orchestra.run.smithi120.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-24T01:25:03.895 INFO:teuthology.orchestra.run.smithi120.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-24T01:25:03.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:03.896 DEBUG:teuthology.orchestra.run.smithi120:> ls /sys/fs/fuse/connections 2024-03-24T01:25:03.960 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi120.stderr:2024-03-24T01:25:03.959+0000 7f3ea164c480 -1 init, newargv = 0x557314f509f0 newargc=15 2024-03-24T01:25:03.960 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi120.stderr:ceph-fuse[43535]: starting ceph client 2024-03-24T01:25:03.970 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi120.stderr:ceph-fuse[43535]: starting fuse 2024-03-24T01:25:03.982 INFO:teuthology.orchestra.run.smithi120.stdout:41 2024-03-24T01:25:03.982 INFO:teuthology.orchestra.run.smithi120.stdout:43 2024-03-24T01:25:03.983 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41, 43] 2024-03-24T01:25:03.983 DEBUG:teuthology.orchestra.run.smithi120:> sudo stdin-killer -- python3 -c ' 2024-03-24T01:25:03.983 DEBUG:teuthology.orchestra.run.smithi120:> import glob 2024-03-24T01:25:03.983 DEBUG:teuthology.orchestra.run.smithi120:> import re 2024-03-24T01:25:03.983 DEBUG:teuthology.orchestra.run.smithi120:> import os 2024-03-24T01:25:03.983 DEBUG:teuthology.orchestra.run.smithi120:> import subprocess 2024-03-24T01:25:03.983 DEBUG:teuthology.orchestra.run.smithi120:> 2024-03-24T01:25:03.983 DEBUG:teuthology.orchestra.run.smithi120:> def _find_admin_socket(client_name): 2024-03-24T01:25:03.983 DEBUG:teuthology.orchestra.run.smithi120:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-03-24T01:25:03.983 DEBUG:teuthology.orchestra.run.smithi120:> files = glob.glob(asok_path) 2024-03-24T01:25:03.983 DEBUG:teuthology.orchestra.run.smithi120:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-03-24T01:25:03.983 DEBUG:teuthology.orchestra.run.smithi120:> 2024-03-24T01:25:03.984 DEBUG:teuthology.orchestra.run.smithi120:> # Given a non-glob path, it better be there 2024-03-24T01:25:03.984 DEBUG:teuthology.orchestra.run.smithi120:> if "*" not in asok_path: 2024-03-24T01:25:03.984 DEBUG:teuthology.orchestra.run.smithi120:> assert(len(files) == 1) 2024-03-24T01:25:03.984 DEBUG:teuthology.orchestra.run.smithi120:> return files[0] 2024-03-24T01:25:03.984 DEBUG:teuthology.orchestra.run.smithi120:> 2024-03-24T01:25:03.984 DEBUG:teuthology.orchestra.run.smithi120:> for f in files: 2024-03-24T01:25:03.984 DEBUG:teuthology.orchestra.run.smithi120:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T01:25:03.984 DEBUG:teuthology.orchestra.run.smithi120:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T01:25:03.984 DEBUG:teuthology.orchestra.run.smithi120:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T01:25:03.984 DEBUG:teuthology.orchestra.run.smithi120:> contents = proc_f.read() 2024-03-24T01:25:03.984 DEBUG:teuthology.orchestra.run.smithi120:> if mountpoint in contents: 2024-03-24T01:25:03.984 DEBUG:teuthology.orchestra.run.smithi120:> return f 2024-03-24T01:25:03.984 DEBUG:teuthology.orchestra.run.smithi120:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T01:25:03.984 DEBUG:teuthology.orchestra.run.smithi120:> 2024-03-24T01:25:03.984 DEBUG:teuthology.orchestra.run.smithi120:> print(_find_admin_socket("client.3")) 2024-03-24T01:25:03.985 DEBUG:teuthology.orchestra.run.smithi120:> ' 2024-03-24T01:25:04.170 INFO:teuthology.orchestra.run.smithi120.stdout:/var/run/ceph/ceph-client.3.43535.asok 2024-03-24T01:25:04.174 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-24T01:25:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T01:25:04.186 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.43535.asok 2024-03-24T01:25:04.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:04.187 DEBUG:teuthology.orchestra.run.smithi120:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.43535.asok status 2024-03-24T01:25:04.412 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-03-24T01:25:04.413 INFO:teuthology.orchestra.run.smithi120.stdout: "metadata": { 2024-03-24T01:25:04.413 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_sha1": "3be9a5c9bc793e11e2800a8c0c696e8b46742033", 2024-03-24T01:25:04.413 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-24T01:25:04.413 INFO:teuthology.orchestra.run.smithi120.stdout: "entity_id": "3", 2024-03-24T01:25:04.413 INFO:teuthology.orchestra.run.smithi120.stdout: "hostname": "smithi120", 2024-03-24T01:25:04.413 INFO:teuthology.orchestra.run.smithi120.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-03-24T01:25:04.413 INFO:teuthology.orchestra.run.smithi120.stdout: "pid": "43535", 2024-03-24T01:25:04.413 INFO:teuthology.orchestra.run.smithi120.stdout: "root": "/" 2024-03-24T01:25:04.413 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-24T01:25:04.413 INFO:teuthology.orchestra.run.smithi120.stdout: "dentry_count": 0, 2024-03-24T01:25:04.413 INFO:teuthology.orchestra.run.smithi120.stdout: "dentry_pinned_count": 0, 2024-03-24T01:25:04.414 INFO:teuthology.orchestra.run.smithi120.stdout: "id": 4553, 2024-03-24T01:25:04.414 INFO:teuthology.orchestra.run.smithi120.stdout: "inst": { 2024-03-24T01:25:04.414 INFO:teuthology.orchestra.run.smithi120.stdout: "name": { 2024-03-24T01:25:04.414 INFO:teuthology.orchestra.run.smithi120.stdout: "type": "client", 2024-03-24T01:25:04.414 INFO:teuthology.orchestra.run.smithi120.stdout: "num": 4553 2024-03-24T01:25:04.414 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-24T01:25:04.414 INFO:teuthology.orchestra.run.smithi120.stdout: "addr": { 2024-03-24T01:25:04.414 INFO:teuthology.orchestra.run.smithi120.stdout: "type": "v1", 2024-03-24T01:25:04.414 INFO:teuthology.orchestra.run.smithi120.stdout: "addr": "192.168.0.2:0", 2024-03-24T01:25:04.414 INFO:teuthology.orchestra.run.smithi120.stdout: "nonce": 2916880519 2024-03-24T01:25:04.414 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-24T01:25:04.414 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-24T01:25:04.414 INFO:teuthology.orchestra.run.smithi120.stdout: "addr": { 2024-03-24T01:25:04.414 INFO:teuthology.orchestra.run.smithi120.stdout: "type": "v1", 2024-03-24T01:25:04.415 INFO:teuthology.orchestra.run.smithi120.stdout: "addr": "192.168.0.2:0", 2024-03-24T01:25:04.415 INFO:teuthology.orchestra.run.smithi120.stdout: "nonce": 2916880519 2024-03-24T01:25:04.415 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-24T01:25:04.415 INFO:teuthology.orchestra.run.smithi120.stdout: "inst_str": "client.4553 192.168.0.2:0/2916880519", 2024-03-24T01:25:04.415 INFO:teuthology.orchestra.run.smithi120.stdout: "addr_str": "192.168.0.2:0/2916880519", 2024-03-24T01:25:04.415 INFO:teuthology.orchestra.run.smithi120.stdout: "inode_count": 1, 2024-03-24T01:25:04.415 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_epoch": 8, 2024-03-24T01:25:04.415 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_epoch": 27, 2024-03-24T01:25:04.415 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_epoch_barrier": 0, 2024-03-24T01:25:04.415 INFO:teuthology.orchestra.run.smithi120.stdout: "blocklisted": false, 2024-03-24T01:25:04.415 INFO:teuthology.orchestra.run.smithi120.stdout: "fs_name": "cephfs" 2024-03-24T01:25:04.415 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-03-24T01:25:04.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:04.427 DEBUG:teuthology.orchestra.run.smithi090:> stat --file-system '--printf=%T 2024-03-24T01:25:04.427 DEBUG:teuthology.orchestra.run.smithi090:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-24T01:25:04.485 INFO:teuthology.orchestra.run.smithi090.stdout:fuseblk 2024-03-24T01:25:04.485 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-24T01:25:04.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:04.486 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T01:25:04.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:04.583 DEBUG:teuthology.orchestra.run.smithi090:> stat --file-system '--printf=%T 2024-03-24T01:25:04.583 DEBUG:teuthology.orchestra.run.smithi090:> ' -- /home/ubuntu/cephtest/mnt.2 2024-03-24T01:25:04.636 INFO:teuthology.orchestra.run.smithi090.stdout:fuseblk 2024-03-24T01:25:04.637 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-03-24T01:25:04.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:04.637 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-03-24T01:25:04.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:04.727 DEBUG:teuthology.orchestra.run.smithi120:> stat --file-system '--printf=%T 2024-03-24T01:25:04.727 DEBUG:teuthology.orchestra.run.smithi120:> ' -- /home/ubuntu/cephtest/mnt.1 2024-03-24T01:25:04.785 INFO:teuthology.orchestra.run.smithi120.stdout:fuseblk 2024-03-24T01:25:04.785 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-03-24T01:25:04.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:04.785 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-24T01:25:04.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:04.873 DEBUG:teuthology.orchestra.run.smithi120:> stat --file-system '--printf=%T 2024-03-24T01:25:04.873 DEBUG:teuthology.orchestra.run.smithi120:> ' -- /home/ubuntu/cephtest/mnt.3 2024-03-24T01:25:04.928 INFO:teuthology.orchestra.run.smithi120.stdout:fuseblk 2024-03-24T01:25:04.928 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-03-24T01:25:04.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:04.928 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-03-24T01:25:05.012 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-03-24T01:25:05.031 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:25:05.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.247+0000 7fd55dc00640 1 Processor -- start 2024-03-24T01:25:05.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.247+0000 7fd55dc00640 1 -- start start 2024-03-24T01:25:05.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.248+0000 7fd55dc00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd5580cc150 0x7fd5580ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:05.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.248+0000 7fd55dc00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd55806b7d0 0x7fd5580b3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:05.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.249+0000 7fd55dc00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5580cd490 con 0x7fd5581603e0 2024-03-24T01:25:05.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.249+0000 7fd55dc00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd55815f900 con 0x7fd5580cc150 2024-03-24T01:25:05.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.249+0000 7fd55dc00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd55815fbe0 con 0x7fd55806b7d0 2024-03-24T01:25:05.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.249+0000 7fd556a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd5580cc150 0x7fd5580ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:05.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.249+0000 7fd556a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd5580cc150 0x7fd5580ccb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37442/0 (socket says 172.21.15.90:37442) 2024-03-24T01:25:05.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.249+0000 7fd556a00640 1 -- 172.21.15.90:0/2571443845 learned_addr learned my addr 172.21.15.90:0/2571443845 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:05.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.249+0000 7fd557400640 1 --2- 172.21.15.90:0/2571443845 >> v2:172.21.15.120:3300/0 conn(0x7fd55806b7d0 0x7fd5580b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:05.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.249+0000 7fd556a00640 1 -- 172.21.15.90:0/2571443845 >> v2:172.21.15.120:3300/0 conn(0x7fd55806b7d0 msgr2=0x7fd5580b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:05.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.249+0000 7fd556a00640 1 --2- 172.21.15.90:0/2571443845 >> v2:172.21.15.120:3300/0 conn(0x7fd55806b7d0 0x7fd5580b3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:05.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.249+0000 7fd556a00640 1 -- 172.21.15.90:0/2571443845 >> v1:172.21.15.120:6789/0 conn(0x7fd5581603e0 legacy=0x7fd55806b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:05.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.249+0000 7fd556a00640 1 -- 172.21.15.90:0/2571443845 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd55815fec0 con 0x7fd5580cc150 2024-03-24T01:25:05.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.250+0000 7fd556a00640 1 --2- 172.21.15.90:0/2571443845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5580cc150 0x7fd5580ccb60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd55806fc10 tx=0x7fd54802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=791e23206eb18d60 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:05.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.251+0000 7fd555600640 1 -- 172.21.15.90:0/2571443845 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd548035020 con 0x7fd5580cc150 2024-03-24T01:25:05.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.251+0000 7fd555600640 1 -- 172.21.15.90:0/2571443845 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd548031870 con 0x7fd5580cc150 2024-03-24T01:25:05.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.251+0000 7fd555600640 1 -- 172.21.15.90:0/2571443845 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd548031c40 con 0x7fd5580cc150 2024-03-24T01:25:05.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.251+0000 7fd55dc00640 1 -- 172.21.15.90:0/2571443845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5580cc150 msgr2=0x7fd5580ccb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:05.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.251+0000 7fd55dc00640 1 --2- 172.21.15.90:0/2571443845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5580cc150 0x7fd5580ccb60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd55806fc10 tx=0x7fd54802b910 comp rx=0 tx=0).stop 2024-03-24T01:25:05.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.251+0000 7fd55dc00640 1 -- 172.21.15.90:0/2571443845 shutdown_connections 2024-03-24T01:25:05.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.251+0000 7fd55dc00640 1 --2- 172.21.15.90:0/2571443845 >> v2:172.21.15.120:3300/0 conn(0x7fd55806b7d0 0x7fd5580b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:05.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.251+0000 7fd55dc00640 1 --2- 172.21.15.90:0/2571443845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5580cc150 0x7fd5580ccb60 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:05.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.252+0000 7fd55dc00640 1 -- 172.21.15.90:0/2571443845 >> 172.21.15.90:0/2571443845 conn(0x7fd5580b1770 msgr2=0x7fd5580b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:05.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.252+0000 7fd55dc00640 1 -- 172.21.15.90:0/2571443845 shutdown_connections 2024-03-24T01:25:05.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.252+0000 7fd55dc00640 1 -- 172.21.15.90:0/2571443845 wait complete. 2024-03-24T01:25:05.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.253+0000 7fd55dc00640 1 Processor -- start 2024-03-24T01:25:05.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.253+0000 7fd55dc00640 1 -- start start 2024-03-24T01:25:05.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.254+0000 7fd55dc00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd55806b7d0 0x7fd558135220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:05.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.254+0000 7fd55dc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5580cc150 0x7fd5580adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:05.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.254+0000 7fd55dc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd5581603e0 0x7fd5580ae060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:05.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.254+0000 7fd55dc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5581455b0 con 0x7fd5580cc150 2024-03-24T01:25:05.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.254+0000 7fd55dc00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd558145430 con 0x7fd55806b7d0 2024-03-24T01:25:05.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.254+0000 7fd55dc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5580b3c90 con 0x7fd5581603e0 2024-03-24T01:25:05.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.255+0000 7fd556000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5580cc150 0x7fd5580adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:05.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.255+0000 7fd556000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5580cc150 0x7fd5580adb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37450/0 (socket says 172.21.15.90:37450) 2024-03-24T01:25:05.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.255+0000 7fd556000640 1 -- 172.21.15.90:0/3156087384 learned_addr learned my addr 172.21.15.90:0/3156087384 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:05.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.255+0000 7fd556a00640 1 --2- 172.21.15.90:0/3156087384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd55806b7d0 0x7fd558135220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:05.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.255+0000 7fd556000640 1 -- 172.21.15.90:0/3156087384 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd5581603e0 msgr2=0x7fd5580ae060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:05.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.255+0000 7fd556000640 1 --2- 172.21.15.90:0/3156087384 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd5581603e0 0x7fd5580ae060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:05.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.255+0000 7fd556000640 1 -- 172.21.15.90:0/3156087384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd55806b7d0 msgr2=0x7fd558135220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:05.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.255+0000 7fd556000640 1 --2- 172.21.15.90:0/3156087384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd55806b7d0 0x7fd558135220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:05.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.255+0000 7fd556000640 1 -- 172.21.15.90:0/3156087384 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd55815de20 con 0x7fd5580cc150 2024-03-24T01:25:05.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.255+0000 7fd556a00640 1 --2- 172.21.15.90:0/3156087384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd55806b7d0 0x7fd558135220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:05.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.255+0000 7fd556000640 1 --2- 172.21.15.90:0/3156087384 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5580cc150 0x7fd5580adb20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd54400b3c0 tx=0x7fd54400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:05.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.256+0000 7fd54ea00640 1 -- 172.21.15.90:0/3156087384 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd544011830 con 0x7fd5580cc150 2024-03-24T01:25:05.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.256+0000 7fd54ea00640 1 -- 172.21.15.90:0/3156087384 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd544016070 con 0x7fd5580cc150 2024-03-24T01:25:05.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.256+0000 7fd54ea00640 1 -- 172.21.15.90:0/3156087384 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd544011c50 con 0x7fd5580cc150 2024-03-24T01:25:05.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.256+0000 7fd55dc00640 1 -- 172.21.15.90:0/3156087384 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd55815e100 con 0x7fd5580cc150 2024-03-24T01:25:05.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.256+0000 7fd55dc00640 1 -- 172.21.15.90:0/3156087384 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd55815e6c0 con 0x7fd5580cc150 2024-03-24T01:25:05.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.257+0000 7fd55dc00640 1 -- 172.21.15.90:0/3156087384 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd558068060 con 0x7fd5580cc150 2024-03-24T01:25:05.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.258+0000 7fd54ea00640 1 -- 172.21.15.90:0/3156087384 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd54401b460 con 0x7fd5580cc150 2024-03-24T01:25:05.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.259+0000 7fd54ea00640 1 --2- 172.21.15.90:0/3156087384 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd53806bbd0 0x7fd53806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:05.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.259+0000 7fd54ea00640 1 -- 172.21.15.90:0/3156087384 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7fd54408aa80 con 0x7fd5580cc150 2024-03-24T01:25:05.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.260+0000 7fd556a00640 1 --2- 172.21.15.90:0/3156087384 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd53806bbd0 0x7fd53806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:05.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.261+0000 7fd556a00640 1 --2- 172.21.15.90:0/3156087384 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd53806bbd0 0x7fd53806e090 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd548036000 tx=0x7fd548008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:05.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.264+0000 7fd54ea00640 1 -- 172.21.15.90:0/3156087384 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd54408ce20 con 0x7fd5580cc150 2024-03-24T01:25:05.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.460+0000 7fd55dc00640 1 -- 172.21.15.90:0/3156087384 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5580b3220 con 0x7fd5580cc150 2024-03-24T01:25:05.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.462+0000 7fd54ea00640 1 -- 172.21.15.90:0/3156087384 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18925 (secure 0 0 0) 0x7fd544020020 con 0x7fd5580cc150 2024-03-24T01:25:05.463 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:25:05.463 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:24:53.236401+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:24:08.643966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T01:24:09.189904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T01:24:13.027323+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:25:05.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.464+0000 7fd55dc00640 1 -- 172.21.15.90:0/3156087384 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd53806bbd0 msgr2=0x7fd53806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:05.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.464+0000 7fd55dc00640 1 --2- 172.21.15.90:0/3156087384 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd53806bbd0 0x7fd53806e090 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd548036000 tx=0x7fd548008030 comp rx=0 tx=0).stop 2024-03-24T01:25:05.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.464+0000 7fd55dc00640 1 -- 172.21.15.90:0/3156087384 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5580cc150 msgr2=0x7fd5580adb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:05.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.464+0000 7fd55dc00640 1 --2- 172.21.15.90:0/3156087384 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5580cc150 0x7fd5580adb20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd54400b3c0 tx=0x7fd54400b880 comp rx=0 tx=0).stop 2024-03-24T01:25:05.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.464+0000 7fd55dc00640 1 -- 172.21.15.90:0/3156087384 shutdown_connections 2024-03-24T01:25:05.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.464+0000 7fd55dc00640 1 --2- 172.21.15.90:0/3156087384 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd53806bbd0 0x7fd53806e090 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:05.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.464+0000 7fd55dc00640 1 --2- 172.21.15.90:0/3156087384 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5580cc150 0x7fd5580adb20 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:05.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.464+0000 7fd55dc00640 1 --2- 172.21.15.90:0/3156087384 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd5581603e0 0x7fd5580ae060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:05.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.464+0000 7fd55dc00640 1 --2- 172.21.15.90:0/3156087384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd55806b7d0 0x7fd558135220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:05.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.464+0000 7fd55dc00640 1 -- 172.21.15.90:0/3156087384 >> 172.21.15.90:0/3156087384 conn(0x7fd5580b1770 msgr2=0x7fd558143500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:05.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.464+0000 7fd55dc00640 1 -- 172.21.15.90:0/3156087384 shutdown_connections 2024-03-24T01:25:05.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.464+0000 7fd55dc00640 1 -- 172.21.15.90:0/3156087384 wait complete. 2024-03-24T01:25:05.483 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T01:24:08.643966+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T01:24:09.189904+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T01:24:13.027323+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T01:25:05.483 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:25:05.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.698+0000 7fe652a00640 1 Processor -- start 2024-03-24T01:25:05.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.698+0000 7fe652a00640 1 -- start start 2024-03-24T01:25:05.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.700+0000 7fe652a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe64c15dd90 0x7fe64c0b6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:05.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.700+0000 7fe652a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe64c0b2720 0x7fe64c0d33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:05.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.700+0000 7fe652a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe64c06c6c0 0x7fe64c06cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:05.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.700+0000 7fe652a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe64c06d050 con 0x7fe64c06c6c0 2024-03-24T01:25:05.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.700+0000 7fe652a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe64c0b9a20 con 0x7fe64c0b2720 2024-03-24T01:25:05.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.700+0000 7fe652a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe64c0b9460 con 0x7fe64c15dd90 2024-03-24T01:25:05.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.700+0000 7fe652000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe64c06c6c0 0x7fe64c06cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:05.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.700+0000 7fe652000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe64c06c6c0 0x7fe64c06cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37456/0 (socket says 172.21.15.90:37456) 2024-03-24T01:25:05.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.700+0000 7fe652000640 1 -- 172.21.15.90:0/1410578375 learned_addr learned my addr 172.21.15.90:0/1410578375 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:05.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.700+0000 7fe650c00640 1 --2- 172.21.15.90:0/1410578375 >> v2:172.21.15.120:3300/0 conn(0x7fe64c0b2720 0x7fe64c0d33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:05.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.700+0000 7fe651600640 1 --2- 172.21.15.90:0/1410578375 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe64c15dd90 0x7fe64c0b6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:05.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.700+0000 7fe652000640 1 -- 172.21.15.90:0/1410578375 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe64c15dd90 msgr2=0x7fe64c0b6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:05.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.701+0000 7fe652000640 1 --2- 172.21.15.90:0/1410578375 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe64c15dd90 0x7fe64c0b6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:05.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.701+0000 7fe652000640 1 -- 172.21.15.90:0/1410578375 >> v2:172.21.15.120:3300/0 conn(0x7fe64c0b2720 msgr2=0x7fe64c0d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:05.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.701+0000 7fe652000640 1 --2- 172.21.15.90:0/1410578375 >> v2:172.21.15.120:3300/0 conn(0x7fe64c0b2720 0x7fe64c0d33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:05.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.701+0000 7fe652000640 1 -- 172.21.15.90:0/1410578375 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe64c0b9740 con 0x7fe64c06c6c0 2024-03-24T01:25:05.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.701+0000 7fe650c00640 1 --2- 172.21.15.90:0/1410578375 >> v2:172.21.15.120:3300/0 conn(0x7fe64c0b2720 0x7fe64c0d33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:05.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.701+0000 7fe651600640 1 --2- 172.21.15.90:0/1410578375 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe64c15dd90 0x7fe64c0b6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:05.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.702+0000 7fe652000640 1 --2- 172.21.15.90:0/1410578375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe64c06c6c0 0x7fe64c06cb10 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe64c0b65d0 tx=0x7fe64402b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=636b46722210612d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:05.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.702+0000 7fe64be00640 1 -- 172.21.15.90:0/1410578375 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe644035020 con 0x7fe64c06c6c0 2024-03-24T01:25:05.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.702+0000 7fe64be00640 1 -- 172.21.15.90:0/1410578375 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe644031880 con 0x7fe64c06c6c0 2024-03-24T01:25:05.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.702+0000 7fe64be00640 1 -- 172.21.15.90:0/1410578375 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe644031c50 con 0x7fe64c06c6c0 2024-03-24T01:25:05.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.703+0000 7fe652a00640 1 -- 172.21.15.90:0/1410578375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe64c06c6c0 msgr2=0x7fe64c06cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:05.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.703+0000 7fe652a00640 1 --2- 172.21.15.90:0/1410578375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe64c06c6c0 0x7fe64c06cb10 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe64c0b65d0 tx=0x7fe64402b920 comp rx=0 tx=0).stop 2024-03-24T01:25:05.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.703+0000 7fe652a00640 1 -- 172.21.15.90:0/1410578375 shutdown_connections 2024-03-24T01:25:05.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.703+0000 7fe652a00640 1 --2- 172.21.15.90:0/1410578375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe64c06c6c0 0x7fe64c06cb10 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:05.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.703+0000 7fe652a00640 1 --2- 172.21.15.90:0/1410578375 >> v2:172.21.15.120:3300/0 conn(0x7fe64c0b2720 0x7fe64c0d33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:05.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.703+0000 7fe652a00640 1 --2- 172.21.15.90:0/1410578375 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe64c15dd90 0x7fe64c0b6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:05.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.703+0000 7fe652a00640 1 -- 172.21.15.90:0/1410578375 >> 172.21.15.90:0/1410578375 conn(0x7fe64c072e20 msgr2=0x7fe64c0d3a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:05.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.703+0000 7fe652a00640 1 -- 172.21.15.90:0/1410578375 shutdown_connections 2024-03-24T01:25:05.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.703+0000 7fe652a00640 1 -- 172.21.15.90:0/1410578375 wait complete. 2024-03-24T01:25:05.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.704+0000 7fe652a00640 1 Processor -- start 2024-03-24T01:25:05.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.705+0000 7fe652a00640 1 -- start start 2024-03-24T01:25:05.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.705+0000 7fe652a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe64c06c6c0 0x7fe64c0cb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:05.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.705+0000 7fe652a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe64c0b2720 0x7fe64c0cd990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:05.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.706+0000 7fe652a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe64c15dd90 0x7fe64c159de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:05.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.706+0000 7fe652a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64c0d2410 con 0x7fe64c15dd90 2024-03-24T01:25:05.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.706+0000 7fe652a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64c0d1ef0 con 0x7fe64c0b2720 2024-03-24T01:25:05.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.706+0000 7fe652a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe64c0d2590 con 0x7fe64c06c6c0 2024-03-24T01:25:05.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.706+0000 7fe652000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe64c15dd90 0x7fe64c159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:05.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.706+0000 7fe652000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe64c15dd90 0x7fe64c159de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37468/0 (socket says 172.21.15.90:37468) 2024-03-24T01:25:05.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.706+0000 7fe652000640 1 -- 172.21.15.90:0/2451690292 learned_addr learned my addr 172.21.15.90:0/2451690292 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:05.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.706+0000 7fe650c00640 1 --2- 172.21.15.90:0/2451690292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe64c0b2720 0x7fe64c0cd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:05.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.706+0000 7fe652000640 1 -- 172.21.15.90:0/2451690292 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe64c06c6c0 msgr2=0x7fe64c0cb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:05.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.706+0000 7fe652000640 1 --2- 172.21.15.90:0/2451690292 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe64c06c6c0 0x7fe64c0cb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:05.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.706+0000 7fe652000640 1 -- 172.21.15.90:0/2451690292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe64c0b2720 msgr2=0x7fe64c0cd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:05.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.706+0000 7fe652000640 1 --2- 172.21.15.90:0/2451690292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe64c0b2720 0x7fe64c0cd990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:05.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.706+0000 7fe652000640 1 -- 172.21.15.90:0/2451690292 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe64c15a320 con 0x7fe64c15dd90 2024-03-24T01:25:05.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.706+0000 7fe650c00640 1 --2- 172.21.15.90:0/2451690292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe64c0b2720 0x7fe64c0cd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:05.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.707+0000 7fe652000640 1 --2- 172.21.15.90:0/2451690292 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe64c15dd90 0x7fe64c159de0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe64402b960 tx=0x7fe6440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:05.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.707+0000 7fe649600640 1 -- 172.21.15.90:0/2451690292 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe644040070 con 0x7fe64c15dd90 2024-03-24T01:25:05.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.707+0000 7fe649600640 1 -- 172.21.15.90:0/2451690292 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe64403a420 con 0x7fe64c15dd90 2024-03-24T01:25:05.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.707+0000 7fe652a00640 1 -- 172.21.15.90:0/2451690292 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe64c0cac60 con 0x7fe64c15dd90 2024-03-24T01:25:05.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.707+0000 7fe649600640 1 -- 172.21.15.90:0/2451690292 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe644035040 con 0x7fe64c15dd90 2024-03-24T01:25:05.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.707+0000 7fe652a00640 1 -- 172.21.15.90:0/2451690292 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe64c15f360 con 0x7fe64c15dd90 2024-03-24T01:25:05.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.708+0000 7fe652a00640 1 -- 172.21.15.90:0/2451690292 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe64c04f400 con 0x7fe64c15dd90 2024-03-24T01:25:05.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.709+0000 7fe649600640 1 -- 172.21.15.90:0/2451690292 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe644038020 con 0x7fe64c15dd90 2024-03-24T01:25:05.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.710+0000 7fe649600640 1 --2- 172.21.15.90:0/2451690292 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe62c06bc00 0x7fe62c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:05.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.710+0000 7fe649600640 1 -- 172.21.15.90:0/2451690292 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7fe6440acec0 con 0x7fe64c15dd90 2024-03-24T01:25:05.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.710+0000 7fe651600640 1 --2- 172.21.15.90:0/2451690292 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe62c06bc00 0x7fe62c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:05.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.712+0000 7fe651600640 1 --2- 172.21.15.90:0/2451690292 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe62c06bc00 0x7fe62c06e0c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe63c007130 tx=0x7fe63c007300 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:05.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.714+0000 7fe649600640 1 -- 172.21.15.90:0/2451690292 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe64407b580 con 0x7fe64c15dd90 2024-03-24T01:25:05.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.914+0000 7fe652a00640 1 -- 172.21.15.90:0/2451690292 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe64c0b31c0 con 0x7fe64c15dd90 2024-03-24T01:25:05.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.915+0000 7fe649600640 1 -- 172.21.15.90:0/2451690292 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe64407ae80 con 0x7fe64c15dd90 2024-03-24T01:25:05.916 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:25:05.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.919+0000 7fe652a00640 1 -- 172.21.15.90:0/2451690292 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe62c06bc00 msgr2=0x7fe62c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:05.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.919+0000 7fe652a00640 1 --2- 172.21.15.90:0/2451690292 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe62c06bc00 0x7fe62c06e0c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe63c007130 tx=0x7fe63c007300 comp rx=0 tx=0).stop 2024-03-24T01:25:05.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.919+0000 7fe652a00640 1 -- 172.21.15.90:0/2451690292 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe64c15dd90 msgr2=0x7fe64c159de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:05.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.919+0000 7fe652a00640 1 --2- 172.21.15.90:0/2451690292 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe64c15dd90 0x7fe64c159de0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe64402b960 tx=0x7fe6440046c0 comp rx=0 tx=0).stop 2024-03-24T01:25:05.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.919+0000 7fe652a00640 1 -- 172.21.15.90:0/2451690292 shutdown_connections 2024-03-24T01:25:05.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.919+0000 7fe652a00640 1 --2- 172.21.15.90:0/2451690292 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe62c06bc00 0x7fe62c06e0c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:05.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.919+0000 7fe652a00640 1 --2- 172.21.15.90:0/2451690292 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe64c15dd90 0x7fe64c159de0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:05.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.919+0000 7fe652a00640 1 --2- 172.21.15.90:0/2451690292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe64c0b2720 0x7fe64c0cd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:05.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.919+0000 7fe652a00640 1 --2- 172.21.15.90:0/2451690292 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe64c06c6c0 0x7fe64c0cb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:05.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.919+0000 7fe652a00640 1 -- 172.21.15.90:0/2451690292 >> 172.21.15.90:0/2451690292 conn(0x7fe64c072e20 msgr2=0x7fe64c0c83e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:05.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.919+0000 7fe652a00640 1 -- 172.21.15.90:0/2451690292 shutdown_connections 2024-03-24T01:25:05.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:05.919+0000 7fe652a00640 1 -- 172.21.15.90:0/2451690292 wait complete. 2024-03-24T01:25:05.939 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T01:25:06.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.137+0000 7f8c2de00640 1 Processor -- start 2024-03-24T01:25:06.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.137+0000 7f8c2de00640 1 -- start start 2024-03-24T01:25:06.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.139+0000 7f8c2de00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8c28159620 0x7f8c2806b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:06.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.139+0000 7f8c2de00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c2806bca0 con 0x7f8c280cf150 2024-03-24T01:25:06.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.139+0000 7f8c2de00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c280b36f0 con 0x7f8c2806b600 2024-03-24T01:25:06.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.139+0000 7f8c2de00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c280b4470 con 0x7f8c28159620 2024-03-24T01:25:06.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.139+0000 7f8c26a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8c28159620 0x7f8c2806b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:06.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.139+0000 7f8c26a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8c28159620 0x7f8c2806b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37474/0 (socket says 172.21.15.90:37474) 2024-03-24T01:25:06.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.139+0000 7f8c26a00640 1 -- 172.21.15.90:0/519808514 learned_addr learned my addr 172.21.15.90:0/519808514 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:06.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.140+0000 7f8c26a00640 1 -- 172.21.15.90:0/519808514 >> v1:172.21.15.120:6789/0 conn(0x7f8c2806b600 legacy=0x7f8c280cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:06.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.140+0000 7f8c26a00640 1 -- 172.21.15.90:0/519808514 >> v1:172.21.15.90:6789/0 conn(0x7f8c280cf150 legacy=0x7f8c280ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:06.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.140+0000 7f8c26a00640 1 -- 172.21.15.90:0/519808514 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c280b3990 con 0x7f8c28159620 2024-03-24T01:25:06.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.141+0000 7f8c26a00640 1 --2- 172.21.15.90:0/519808514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c28159620 0x7f8c2806b0c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8c18009870 tx=0x7f8c1802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0e14a8b4a475f64 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:06.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.142+0000 7f8c25600640 1 -- 172.21.15.90:0/519808514 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c18034070 con 0x7f8c28159620 2024-03-24T01:25:06.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.142+0000 7f8c25600640 1 -- 172.21.15.90:0/519808514 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c18030860 con 0x7f8c28159620 2024-03-24T01:25:06.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.142+0000 7f8c25600640 1 -- 172.21.15.90:0/519808514 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c18030c30 con 0x7f8c28159620 2024-03-24T01:25:06.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.142+0000 7f8c2de00640 1 -- 172.21.15.90:0/519808514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c28159620 msgr2=0x7f8c2806b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:06.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.142+0000 7f8c2de00640 1 --2- 172.21.15.90:0/519808514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c28159620 0x7f8c2806b0c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8c18009870 tx=0x7f8c1802b9a0 comp rx=0 tx=0).stop 2024-03-24T01:25:06.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.142+0000 7f8c2de00640 1 -- 172.21.15.90:0/519808514 shutdown_connections 2024-03-24T01:25:06.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.142+0000 7f8c2de00640 1 --2- 172.21.15.90:0/519808514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c28159620 0x7f8c2806b0c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:06.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.142+0000 7f8c2de00640 1 -- 172.21.15.90:0/519808514 >> 172.21.15.90:0/519808514 conn(0x7f8c280b1770 msgr2=0x7f8c280b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:06.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.142+0000 7f8c2de00640 1 -- 172.21.15.90:0/519808514 shutdown_connections 2024-03-24T01:25:06.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.142+0000 7f8c2de00640 1 -- 172.21.15.90:0/519808514 wait complete. 2024-03-24T01:25:06.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.144+0000 7f8c2de00640 1 Processor -- start 2024-03-24T01:25:06.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.144+0000 7f8c2de00640 1 -- start start 2024-03-24T01:25:06.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.145+0000 7f8c2de00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c2806b600 0x7f8c281e5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:06.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.145+0000 7f8c2de00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c280cf150 0x7f8c281e6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:06.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.145+0000 7f8c2de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c28159620 0x7f8c281e6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:06.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.145+0000 7f8c2de00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c28159a00 con 0x7f8c28159620 2024-03-24T01:25:06.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.145+0000 7f8c2de00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c280b3f50 con 0x7f8c280cf150 2024-03-24T01:25:06.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.145+0000 7f8c2de00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c2815bfe0 con 0x7f8c2806b600 2024-03-24T01:25:06.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.145+0000 7f8c27400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c28159620 0x7f8c281e6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:06.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.146+0000 7f8c27400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c28159620 0x7f8c281e6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37490/0 (socket says 172.21.15.90:37490) 2024-03-24T01:25:06.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.146+0000 7f8c27400640 1 -- 172.21.15.90:0/1885108933 learned_addr learned my addr 172.21.15.90:0/1885108933 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:06.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.146+0000 7f8c26a00640 1 --2- 172.21.15.90:0/1885108933 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c2806b600 0x7f8c281e5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:06.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.146+0000 7f8c26000640 1 --2- 172.21.15.90:0/1885108933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c280cf150 0x7f8c281e6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:06.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.146+0000 7f8c27400640 1 -- 172.21.15.90:0/1885108933 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c2806b600 msgr2=0x7f8c281e5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:06.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.146+0000 7f8c27400640 1 --2- 172.21.15.90:0/1885108933 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c2806b600 0x7f8c281e5870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:06.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.146+0000 7f8c27400640 1 -- 172.21.15.90:0/1885108933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c280cf150 msgr2=0x7f8c281e6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:06.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.146+0000 7f8c27400640 1 --2- 172.21.15.90:0/1885108933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c280cf150 0x7f8c281e6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:06.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.146+0000 7f8c27400640 1 -- 172.21.15.90:0/1885108933 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c281e47d0 con 0x7f8c28159620 2024-03-24T01:25:06.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.146+0000 7f8c27400640 1 --2- 172.21.15.90:0/1885108933 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c28159620 0x7f8c281e6a80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8c2000c7d0 tx=0x7f8c2000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:06.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.147+0000 7f8c1f400640 1 -- 172.21.15.90:0/1885108933 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c20003b70 con 0x7f8c28159620 2024-03-24T01:25:06.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.147+0000 7f8c1f400640 1 -- 172.21.15.90:0/1885108933 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c20003d30 con 0x7f8c28159620 2024-03-24T01:25:06.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.147+0000 7f8c1f400640 1 -- 172.21.15.90:0/1885108933 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2000aa90 con 0x7f8c28159620 2024-03-24T01:25:06.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.147+0000 7f8c2de00640 1 -- 172.21.15.90:0/1885108933 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c281e4ab0 con 0x7f8c28159620 2024-03-24T01:25:06.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.147+0000 7f8c2de00640 1 -- 172.21.15.90:0/1885108933 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c281e5070 con 0x7f8c28159620 2024-03-24T01:25:06.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.148+0000 7f8c2de00640 1 -- 172.21.15.90:0/1885108933 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c2804f400 con 0x7f8c28159620 2024-03-24T01:25:06.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.149+0000 7f8c1f400640 1 -- 172.21.15.90:0/1885108933 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8c2000ac30 con 0x7f8c28159620 2024-03-24T01:25:06.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.150+0000 7f8c1f400640 1 --2- 172.21.15.90:0/1885108933 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8c0806bbd0 0x7f8c0806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:06.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.150+0000 7f8c1f400640 1 -- 172.21.15.90:0/1885108933 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f8c2008ad00 con 0x7f8c28159620 2024-03-24T01:25:06.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.151+0000 7f8c26a00640 1 --2- 172.21.15.90:0/1885108933 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8c0806bbd0 0x7f8c0806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:06.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.152+0000 7f8c26a00640 1 --2- 172.21.15.90:0/1885108933 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8c0806bbd0 0x7f8c0806e090 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8c18036000 tx=0x7f8c18008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:06.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.155+0000 7f8c1f400640 1 -- 172.21.15.90:0/1885108933 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8c20058ce0 con 0x7f8c28159620 2024-03-24T01:25:06.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.351+0000 7f8c2de00640 1 -- 172.21.15.90:0/1885108933 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8c280b2e90 con 0x7f8c28159620 2024-03-24T01:25:06.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.352+0000 7f8c1f400640 1 -- 172.21.15.90:0/1885108933 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8c2005cd80 con 0x7f8c28159620 2024-03-24T01:25:06.352 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:25:06.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.355+0000 7f8c2de00640 1 -- 172.21.15.90:0/1885108933 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8c0806bbd0 msgr2=0x7f8c0806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:06.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.355+0000 7f8c2de00640 1 --2- 172.21.15.90:0/1885108933 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8c0806bbd0 0x7f8c0806e090 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8c18036000 tx=0x7f8c18008030 comp rx=0 tx=0).stop 2024-03-24T01:25:06.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.355+0000 7f8c2de00640 1 -- 172.21.15.90:0/1885108933 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c28159620 msgr2=0x7f8c281e6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:06.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.355+0000 7f8c2de00640 1 --2- 172.21.15.90:0/1885108933 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c28159620 0x7f8c281e6a80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8c2000c7d0 tx=0x7f8c2000cc90 comp rx=0 tx=0).stop 2024-03-24T01:25:06.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.355+0000 7f8c2de00640 1 -- 172.21.15.90:0/1885108933 shutdown_connections 2024-03-24T01:25:06.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.355+0000 7f8c2de00640 1 --2- 172.21.15.90:0/1885108933 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8c0806bbd0 0x7f8c0806e090 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:06.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.355+0000 7f8c2de00640 1 --2- 172.21.15.90:0/1885108933 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c28159620 0x7f8c281e6a80 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:06.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.355+0000 7f8c2de00640 1 --2- 172.21.15.90:0/1885108933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c280cf150 0x7f8c281e6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:06.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.355+0000 7f8c2de00640 1 --2- 172.21.15.90:0/1885108933 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8c2806b600 0x7f8c281e5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:06.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.355+0000 7f8c2de00640 1 -- 172.21.15.90:0/1885108933 >> 172.21.15.90:0/1885108933 conn(0x7f8c280b1770 msgr2=0x7f8c2815d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:06.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.355+0000 7f8c2de00640 1 -- 172.21.15.90:0/1885108933 shutdown_connections 2024-03-24T01:25:06.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.355+0000 7f8c2de00640 1 -- 172.21.15.90:0/1885108933 wait complete. 2024-03-24T01:25:06.373 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T01:25:06.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.582+0000 7f43cd600640 1 Processor -- start 2024-03-24T01:25:06.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.582+0000 7f43cd600640 1 -- start start 2024-03-24T01:25:06.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.584+0000 7f43cd600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f43c806ecf0 0x7f43c80b3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:06.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.584+0000 7f43cd600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43c80b3670 con 0x7f43c806f0d0 2024-03-24T01:25:06.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.584+0000 7f43cd600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43c815b9d0 con 0x7f43c81549f0 2024-03-24T01:25:06.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.584+0000 7f43cd600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43c815bcb0 con 0x7f43c806ecf0 2024-03-24T01:25:06.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.584+0000 7f43c7400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f43c806f0d0 0x7f43c815af80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51498/0 (socket says 172.21.15.90:51498) 2024-03-24T01:25:06.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.584+0000 7f43c7400640 1 -- 172.21.15.90:0/209363949 learned_addr learned my addr 172.21.15.90:0/209363949 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:06.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.584+0000 7f43c7e00640 1 --2- 172.21.15.90:0/209363949 >> v2:172.21.15.120:3300/0 conn(0x7f43c806ecf0 0x7f43c80b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:06.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.584+0000 7f43c6a00640 1 -- 172.21.15.90:0/209363949 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3190606720 0 0) 0x7f43c80b3670 con 0x7f43c806f0d0 2024-03-24T01:25:06.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.585+0000 7f43c6a00640 1 -- 172.21.15.90:0/209363949 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43ac003520 con 0x7f43c806f0d0 2024-03-24T01:25:06.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.585+0000 7f43c6a00640 1 -- 172.21.15.90:0/209363949 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4055695371 0 0) 0x7f43ac003520 con 0x7f43c806f0d0 2024-03-24T01:25:06.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.585+0000 7f43c6a00640 1 -- 172.21.15.90:0/209363949 >> v2:172.21.15.120:3300/0 conn(0x7f43c806ecf0 msgr2=0x7f43c80b3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:06.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.585+0000 7f43c6a00640 1 --2- 172.21.15.90:0/209363949 >> v2:172.21.15.120:3300/0 conn(0x7f43c806ecf0 0x7f43c80b3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:06.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.585+0000 7f43c6a00640 1 -- 172.21.15.90:0/209363949 >> v1:172.21.15.120:6789/0 conn(0x7f43c81549f0 legacy=0x7f43c8154ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:06.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.585+0000 7f43c6a00640 1 -- 172.21.15.90:0/209363949 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c80b2930 con 0x7f43c806f0d0 2024-03-24T01:25:06.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.585+0000 7f43c6a00640 1 -- 172.21.15.90:0/209363949 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f43b4001670 con 0x7f43c806f0d0 2024-03-24T01:25:06.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.585+0000 7f43c6a00640 1 -- 172.21.15.90:0/209363949 >> v1:172.21.15.90:6789/0 conn(0x7f43c806f0d0 legacy=0x7f43c815af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:06.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.585+0000 7f43c6a00640 1 --2- 172.21.15.90:0/209363949 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43ac0039b0 0x7f43ac003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:06.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.585+0000 7f43c6a00640 1 --2- 172.21.15.90:0/209363949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43ac004800 0x7f43ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:06.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.585+0000 7f43c6a00640 1 --2- 172.21.15.90:0/209363949 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43ac0051c0 0x7f43ac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:06.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.585+0000 7f43c6a00640 1 -- 172.21.15.90:0/209363949 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c815bcb0 con 0x7f43ac0039b0 2024-03-24T01:25:06.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.585+0000 7f43c6a00640 1 -- 172.21.15.90:0/209363949 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac00c060 con 0x7f43ac004800 2024-03-24T01:25:06.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.585+0000 7f43c6a00640 1 -- 172.21.15.90:0/209363949 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac00c260 con 0x7f43ac0051c0 2024-03-24T01:25:06.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.585+0000 7f43c7e00640 1 --2- 172.21.15.90:0/209363949 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43ac0039b0 0x7f43ac003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:06.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.586+0000 7f43c7e00640 1 -- 172.21.15.90:0/209363949 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43ac0051c0 msgr2=0x7f43ac00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:06.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.586+0000 7f43ccc00640 1 --2- 172.21.15.90:0/209363949 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43ac0051c0 0x7f43ac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:06.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.586+0000 7f43c7400640 1 --2- 172.21.15.90:0/209363949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43ac004800 0x7f43ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:06.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.586+0000 7f43c7e00640 1 --2- 172.21.15.90:0/209363949 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43ac0051c0 0x7f43ac00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:06.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.586+0000 7f43c7e00640 1 -- 172.21.15.90:0/209363949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43ac004800 msgr2=0x7f43ac004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:06.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.586+0000 7f43c7e00640 1 --2- 172.21.15.90:0/209363949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43ac004800 0x7f43ac004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:06.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.586+0000 7f43c7e00640 1 -- 172.21.15.90:0/209363949 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43ac003520 con 0x7f43ac0039b0 2024-03-24T01:25:06.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.586+0000 7f43ccc00640 1 --2- 172.21.15.90:0/209363949 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43ac0051c0 0x7f43ac00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:06.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.586+0000 7f43ccc00640 1 -- 172.21.15.90:0/209363949 reap_dead start 2024-03-24T01:25:06.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.587+0000 7f43c7e00640 1 --2- 172.21.15.90:0/209363949 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43ac0039b0 0x7f43ac003dd0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f43b8009ac0 tx=0x7f43ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d33fbbc6ada965f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:06.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.587+0000 7f43c6a00640 1 -- 172.21.15.90:0/209363949 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b8039070 con 0x7f43ac0039b0 2024-03-24T01:25:06.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.587+0000 7f43c6a00640 1 -- 172.21.15.90:0/209363949 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43b802c6b0 con 0x7f43ac0039b0 2024-03-24T01:25:06.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.587+0000 7f43c6a00640 1 -- 172.21.15.90:0/209363949 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b8009690 con 0x7f43ac0039b0 2024-03-24T01:25:06.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.588+0000 7f43cd600640 1 -- 172.21.15.90:0/209363949 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43ac0039b0 msgr2=0x7f43ac003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:06.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.588+0000 7f43cd600640 1 --2- 172.21.15.90:0/209363949 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43ac0039b0 0x7f43ac003dd0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f43b8009ac0 tx=0x7f43ac005830 comp rx=0 tx=0).stop 2024-03-24T01:25:06.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.588+0000 7f43cd600640 1 -- 172.21.15.90:0/209363949 shutdown_connections 2024-03-24T01:25:06.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.588+0000 7f43cd600640 1 --2- 172.21.15.90:0/209363949 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43ac0039b0 0x7f43ac003dd0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:06.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.588+0000 7f43cd600640 1 -- 172.21.15.90:0/209363949 >> 172.21.15.90:0/209363949 conn(0x7f43c806b0c0 msgr2=0x7f43c806b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:06.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.588+0000 7f43cd600640 1 -- 172.21.15.90:0/209363949 shutdown_connections 2024-03-24T01:25:06.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.588+0000 7f43cd600640 1 -- 172.21.15.90:0/209363949 wait complete. 2024-03-24T01:25:06.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.590+0000 7f43cd600640 1 Processor -- start 2024-03-24T01:25:06.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.590+0000 7f43cd600640 1 -- start start 2024-03-24T01:25:06.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.591+0000 7f43cd600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43ac0039b0 0x7f43c813acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:06.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.591+0000 7f43cd600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43c815e370 0x7f43c815e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:06.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.591+0000 7f43cd600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43c8141580 0x7f43c8141a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:06.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.591+0000 7f43cd600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c80b4720 con 0x7f43c815e370 2024-03-24T01:25:06.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.591+0000 7f43cd600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c80b45a0 con 0x7f43ac0039b0 2024-03-24T01:25:06.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.591+0000 7f43cd600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43c80b3130 con 0x7f43c8141580 2024-03-24T01:25:06.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.591+0000 7f43c7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43c815e370 0x7f43c815e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:06.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.591+0000 7f43c7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43c815e370 0x7f43c815e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37506/0 (socket says 172.21.15.90:37506) 2024-03-24T01:25:06.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.591+0000 7f43c7400640 1 -- 172.21.15.90:0/1302529527 learned_addr learned my addr 172.21.15.90:0/1302529527 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:06.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.591+0000 7f43c7e00640 1 --2- 172.21.15.90:0/1302529527 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43ac0039b0 0x7f43c813acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:06.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.591+0000 7f43ccc00640 1 --2- 172.21.15.90:0/1302529527 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43c8141580 0x7f43c8141a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:06.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.592+0000 7f43c7400640 1 -- 172.21.15.90:0/1302529527 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43c8141580 msgr2=0x7f43c8141a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:06.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.592+0000 7f43c7400640 1 --2- 172.21.15.90:0/1302529527 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43c8141580 0x7f43c8141a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:06.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.592+0000 7f43c7400640 1 -- 172.21.15.90:0/1302529527 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43ac0039b0 msgr2=0x7f43c813acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:06.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.592+0000 7f43c7400640 1 --2- 172.21.15.90:0/1302529527 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43ac0039b0 0x7f43c813acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:06.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.592+0000 7f43c7400640 1 -- 172.21.15.90:0/1302529527 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c81422b0 con 0x7f43c815e370 2024-03-24T01:25:06.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.592+0000 7f43c7400640 1 --2- 172.21.15.90:0/1302529527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43c815e370 0x7f43c815e7f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f43b400d810 tx=0x7f43b400dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:06.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.592+0000 7f43bfe00640 1 -- 172.21.15.90:0/1302529527 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b4019070 con 0x7f43c815e370 2024-03-24T01:25:06.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.592+0000 7f43bfe00640 1 -- 172.21.15.90:0/1302529527 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43b40074e0 con 0x7f43c815e370 2024-03-24T01:25:06.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.592+0000 7f43cd600640 1 -- 172.21.15.90:0/1302529527 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43c8142590 con 0x7f43c815e370 2024-03-24T01:25:06.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.592+0000 7f43bfe00640 1 -- 172.21.15.90:0/1302529527 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b4004a20 con 0x7f43c815e370 2024-03-24T01:25:06.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.592+0000 7f43cd600640 1 -- 172.21.15.90:0/1302529527 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43c80af390 con 0x7f43c815e370 2024-03-24T01:25:06.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.593+0000 7f43cd600640 1 -- 172.21.15.90:0/1302529527 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43c804f400 con 0x7f43c815e370 2024-03-24T01:25:06.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.594+0000 7f43bfe00640 1 -- 172.21.15.90:0/1302529527 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f43b401c070 con 0x7f43c815e370 2024-03-24T01:25:06.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.595+0000 7f43bfe00640 1 --2- 172.21.15.90:0/1302529527 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f43a806bc00 0x7f43a806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:06.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.595+0000 7f43bfe00640 1 -- 172.21.15.90:0/1302529527 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f43b408a8f0 con 0x7f43c815e370 2024-03-24T01:25:06.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.596+0000 7f43c7e00640 1 --2- 172.21.15.90:0/1302529527 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f43a806bc00 0x7f43a806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:06.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.597+0000 7f43c7e00640 1 --2- 172.21.15.90:0/1302529527 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f43a806bc00 0x7f43a806e0c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f43b8036000 tx=0x7f43b8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:06.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.600+0000 7f43bfe00640 1 -- 172.21.15.90:0/1302529527 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f43b4058fc0 con 0x7f43c815e370 2024-03-24T01:25:06.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.802+0000 7f43cd600640 1 -- 172.21.15.90:0/1302529527 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f43c80d0670 con 0x7f43c815e370 2024-03-24T01:25:06.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.803+0000 7f43bfe00640 1 -- 172.21.15.90:0/1302529527 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f43b4088e20 con 0x7f43c815e370 2024-03-24T01:25:06.803 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-24T01:25:06.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.805+0000 7f43cd600640 1 -- 172.21.15.90:0/1302529527 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f43a806bc00 msgr2=0x7f43a806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:06.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.805+0000 7f43cd600640 1 --2- 172.21.15.90:0/1302529527 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f43a806bc00 0x7f43a806e0c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f43b8036000 tx=0x7f43b8030000 comp rx=0 tx=0).stop 2024-03-24T01:25:06.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.805+0000 7f43cd600640 1 -- 172.21.15.90:0/1302529527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43c815e370 msgr2=0x7f43c815e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:06.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.805+0000 7f43cd600640 1 --2- 172.21.15.90:0/1302529527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43c815e370 0x7f43c815e7f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f43b400d810 tx=0x7f43b400dcd0 comp rx=0 tx=0).stop 2024-03-24T01:25:06.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.805+0000 7f43cd600640 1 -- 172.21.15.90:0/1302529527 shutdown_connections 2024-03-24T01:25:06.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.805+0000 7f43cd600640 1 --2- 172.21.15.90:0/1302529527 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f43a806bc00 0x7f43a806e0c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:06.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.805+0000 7f43cd600640 1 --2- 172.21.15.90:0/1302529527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43c815e370 0x7f43c815e7f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:06.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.805+0000 7f43cd600640 1 --2- 172.21.15.90:0/1302529527 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43c8141580 0x7f43c8141a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:06.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.805+0000 7f43cd600640 1 --2- 172.21.15.90:0/1302529527 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43ac0039b0 0x7f43c813acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:06.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.805+0000 7f43cd600640 1 -- 172.21.15.90:0/1302529527 >> 172.21.15.90:0/1302529527 conn(0x7f43c806b0c0 msgr2=0x7f43c806b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:06.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.806+0000 7f43cd600640 1 -- 172.21.15.90:0/1302529527 shutdown_connections 2024-03-24T01:25:06.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:06.806+0000 7f43cd600640 1 -- 172.21.15.90:0/1302529527 wait complete. 2024-03-24T01:25:06.824 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T01:25:07.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.042+0000 7f055c800640 1 Processor -- start 2024-03-24T01:25:07.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.042+0000 7f055c800640 1 -- start start 2024-03-24T01:25:07.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.044+0000 7f055c800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f055415da30 0x7f055415de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:07.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.044+0000 7f055c800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05540af140 con 0x7f055406b0c0 2024-03-24T01:25:07.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.044+0000 7f055c800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f055415e550 con 0x7f05540acea0 2024-03-24T01:25:07.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.044+0000 7f055c800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05540b2c70 con 0x7f055415da30 2024-03-24T01:25:07.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.044+0000 7f0559e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f055415da30 0x7f055415de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:07.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.044+0000 7f055a800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f055406b0c0 0x7f0554161a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51510/0 (socket says 172.21.15.90:51510) 2024-03-24T01:25:07.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.044+0000 7f055a800640 1 -- 172.21.15.90:0/2831391103 learned_addr learned my addr 172.21.15.90:0/2831391103 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:07.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.044+0000 7f0559e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f055415da30 0x7f055415de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37518/0 (socket says 172.21.15.90:37518) 2024-03-24T01:25:07.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.045+0000 7f0559e00640 1 -- 172.21.15.90:0/2831391103 >> v1:172.21.15.120:6789/0 conn(0x7f05540acea0 legacy=0x7f05540aea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:07.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.045+0000 7f0559e00640 1 -- 172.21.15.90:0/2831391103 >> v1:172.21.15.90:6789/0 conn(0x7f055406b0c0 legacy=0x7f0554161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:07.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.045+0000 7f0559e00640 1 -- 172.21.15.90:0/2831391103 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f055415e830 con 0x7f055415da30 2024-03-24T01:25:07.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.045+0000 7f0558a00640 1 -- 172.21.15.90:0/2831391103 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3422274966 0 0) 0x7f05540af140 con 0x7f055406b0c0 2024-03-24T01:25:07.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.046+0000 7f0559e00640 1 --2- 172.21.15.90:0/2831391103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f055415da30 0x7f055415de10 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f05480097a0 tx=0x7f054802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab84ca9d0813a582 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:07.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.046+0000 7f0558a00640 1 -- 172.21.15.90:0/2831391103 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0548035020 con 0x7f055415da30 2024-03-24T01:25:07.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.047+0000 7f0558a00640 1 -- 172.21.15.90:0/2831391103 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0548031830 con 0x7f055415da30 2024-03-24T01:25:07.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.047+0000 7f0558a00640 1 -- 172.21.15.90:0/2831391103 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0548031c00 con 0x7f055415da30 2024-03-24T01:25:07.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.047+0000 7f055c800640 1 -- 172.21.15.90:0/2831391103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f055415da30 msgr2=0x7f055415de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:07.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.047+0000 7f055c800640 1 --2- 172.21.15.90:0/2831391103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f055415da30 0x7f055415de10 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f05480097a0 tx=0x7f054802b8d0 comp rx=0 tx=0).stop 2024-03-24T01:25:07.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.047+0000 7f055c800640 1 -- 172.21.15.90:0/2831391103 shutdown_connections 2024-03-24T01:25:07.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.047+0000 7f055c800640 1 --2- 172.21.15.90:0/2831391103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f055415da30 0x7f055415de10 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:07.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.047+0000 7f055c800640 1 -- 172.21.15.90:0/2831391103 >> 172.21.15.90:0/2831391103 conn(0x7f05540b1770 msgr2=0x7f05540b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:07.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.047+0000 7f055c800640 1 -- 172.21.15.90:0/2831391103 shutdown_connections 2024-03-24T01:25:07.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.047+0000 7f055c800640 1 -- 172.21.15.90:0/2831391103 wait complete. 2024-03-24T01:25:07.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.049+0000 7f055c800640 1 Processor -- start 2024-03-24T01:25:07.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.049+0000 7f055c800640 1 -- start start 2024-03-24T01:25:07.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.050+0000 7f055c800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f055406b0c0 0x7f055415ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:07.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.050+0000 7f055c800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f05540acea0 0x7f0554151670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:07.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.050+0000 7f055c800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f055415da30 0x7f0554151bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:07.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.050+0000 7f055c800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f055415edf0 con 0x7f055406b0c0 2024-03-24T01:25:07.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.050+0000 7f055c800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f055415f160 con 0x7f055415da30 2024-03-24T01:25:07.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.050+0000 7f055c800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05540ad300 con 0x7f05540acea0 2024-03-24T01:25:07.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.050+0000 7f0559e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f055406b0c0 0x7f055415ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:07.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.050+0000 7f0559e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f055406b0c0 0x7f055415ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37534/0 (socket says 172.21.15.90:37534) 2024-03-24T01:25:07.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.050+0000 7f0559e00640 1 -- 172.21.15.90:0/4108501734 learned_addr learned my addr 172.21.15.90:0/4108501734 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:07.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.051+0000 7f0559e00640 1 -- 172.21.15.90:0/4108501734 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f05540acea0 msgr2=0x7f0554151670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:07.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.051+0000 7f0559e00640 1 --2- 172.21.15.90:0/4108501734 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f05540acea0 0x7f0554151670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:07.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.051+0000 7f0559e00640 1 -- 172.21.15.90:0/4108501734 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f055415da30 msgr2=0x7f0554151bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:07.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.051+0000 7f0559e00640 1 --2- 172.21.15.90:0/4108501734 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f055415da30 0x7f0554151bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:07.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.051+0000 7f0559e00640 1 -- 172.21.15.90:0/4108501734 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05541509e0 con 0x7f055406b0c0 2024-03-24T01:25:07.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.051+0000 7f0559e00640 1 --2- 172.21.15.90:0/4108501734 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f055406b0c0 0x7f055415ced0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f05480097a0 tx=0x7f05480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:07.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.051+0000 7f054e000640 1 -- 172.21.15.90:0/4108501734 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054803aa70 con 0x7f055406b0c0 2024-03-24T01:25:07.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.051+0000 7f054e000640 1 -- 172.21.15.90:0/4108501734 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f054803ac10 con 0x7f055406b0c0 2024-03-24T01:25:07.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.051+0000 7f054e000640 1 -- 172.21.15.90:0/4108501734 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05480443f0 con 0x7f055406b0c0 2024-03-24T01:25:07.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.051+0000 7f055c800640 1 -- 172.21.15.90:0/4108501734 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0554150c60 con 0x7f055406b0c0 2024-03-24T01:25:07.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.052+0000 7f055c800640 1 -- 172.21.15.90:0/4108501734 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0554151220 con 0x7f055406b0c0 2024-03-24T01:25:07.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.053+0000 7f055c800640 1 -- 172.21.15.90:0/4108501734 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f055404f400 con 0x7f055406b0c0 2024-03-24T01:25:07.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.053+0000 7f054e000640 1 -- 172.21.15.90:0/4108501734 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f054804b060 con 0x7f055406b0c0 2024-03-24T01:25:07.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.054+0000 7f054e000640 1 --2- 172.21.15.90:0/4108501734 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f053806bbd0 0x7f053806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:07.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.054+0000 7f054e000640 1 -- 172.21.15.90:0/4108501734 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f05480acbc0 con 0x7f055406b0c0 2024-03-24T01:25:07.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.055+0000 7f0559400640 1 --2- 172.21.15.90:0/4108501734 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f053806bbd0 0x7f053806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:07.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.056+0000 7f0559400640 1 --2- 172.21.15.90:0/4108501734 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f053806bbd0 0x7f053806e090 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f055415d410 tx=0x7f05440079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:07.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.059+0000 7f054e000640 1 -- 172.21.15.90:0/4108501734 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f05541509e0 con 0x7f055406b0c0 2024-03-24T01:25:07.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.256+0000 7f055c800640 1 -- 172.21.15.90:0/4108501734 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f055406e250 con 0x7f055406b0c0 2024-03-24T01:25:07.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.257+0000 7f054e000640 1 -- 172.21.15.90:0/4108501734 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f054803f070 con 0x7f055406b0c0 2024-03-24T01:25:07.257 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:25:07.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.260+0000 7f055c800640 1 -- 172.21.15.90:0/4108501734 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f053806bbd0 msgr2=0x7f053806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:07.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.260+0000 7f055c800640 1 --2- 172.21.15.90:0/4108501734 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f053806bbd0 0x7f053806e090 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f055415d410 tx=0x7f05440079b0 comp rx=0 tx=0).stop 2024-03-24T01:25:07.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.260+0000 7f055c800640 1 -- 172.21.15.90:0/4108501734 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f055406b0c0 msgr2=0x7f055415ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:07.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.260+0000 7f055c800640 1 --2- 172.21.15.90:0/4108501734 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f055406b0c0 0x7f055415ced0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f05480097a0 tx=0x7f05480046c0 comp rx=0 tx=0).stop 2024-03-24T01:25:07.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.260+0000 7f055c800640 1 -- 172.21.15.90:0/4108501734 shutdown_connections 2024-03-24T01:25:07.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.260+0000 7f055c800640 1 --2- 172.21.15.90:0/4108501734 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f053806bbd0 0x7f053806e090 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:07.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.260+0000 7f055c800640 1 --2- 172.21.15.90:0/4108501734 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f055415da30 0x7f0554151bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:07.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.260+0000 7f055c800640 1 --2- 172.21.15.90:0/4108501734 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f05540acea0 0x7f0554151670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:07.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.260+0000 7f055c800640 1 --2- 172.21.15.90:0/4108501734 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f055406b0c0 0x7f055415ced0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:07.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.260+0000 7f055c800640 1 -- 172.21.15.90:0/4108501734 >> 172.21.15.90:0/4108501734 conn(0x7f05540b1770 msgr2=0x7f05540b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:07.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.260+0000 7f055c800640 1 -- 172.21.15.90:0/4108501734 shutdown_connections 2024-03-24T01:25:07.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.260+0000 7f055c800640 1 -- 172.21.15.90:0/4108501734 wait complete. 2024-03-24T01:25:07.279 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:25:07.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.482+0000 7f5409000640 1 Processor -- start 2024-03-24T01:25:07.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.482+0000 7f5409000640 1 -- start start 2024-03-24T01:25:07.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.484+0000 7f5409000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54040cdac0 0x7f540406b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:07.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.484+0000 7f5409000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f540415bf70 con 0x7f540406b600 2024-03-24T01:25:07.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.484+0000 7f5409000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f540415b360 con 0x7f5404155030 2024-03-24T01:25:07.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.484+0000 7f5409000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f540406ba70 con 0x7f54040cdac0 2024-03-24T01:25:07.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.484+0000 7f5402800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f540406b600 0x7f54040ae530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51526/0 (socket says 172.21.15.90:51526) 2024-03-24T01:25:07.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.484+0000 7f5402800640 1 -- 172.21.15.90:0/3978920758 learned_addr learned my addr 172.21.15.90:0/3978920758 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:07.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.484+0000 7f5401400640 1 --2- 172.21.15.90:0/3978920758 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54040cdac0 0x7f540406b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:07.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.485+0000 7f5400a00640 1 -- 172.21.15.90:0/3978920758 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 980990779 0 0) 0x7f540415bf70 con 0x7f540406b600 2024-03-24T01:25:07.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.485+0000 7f5400a00640 1 -- 172.21.15.90:0/3978920758 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53e8003520 con 0x7f540406b600 2024-03-24T01:25:07.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.485+0000 7f5400a00640 1 -- 172.21.15.90:0/3978920758 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4154556958 0 0) 0x7f540415b360 con 0x7f5404155030 2024-03-24T01:25:07.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.485+0000 7f5400a00640 1 -- 172.21.15.90:0/3978920758 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f540415bf70 con 0x7f5404155030 2024-03-24T01:25:07.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.485+0000 7f5400a00640 1 -- 172.21.15.90:0/3978920758 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1369891260 0 0) 0x7f53e8003520 con 0x7f540406b600 2024-03-24T01:25:07.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.485+0000 7f5400a00640 1 -- 172.21.15.90:0/3978920758 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54040cdac0 msgr2=0x7f540406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:07.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.485+0000 7f5400a00640 1 --2- 172.21.15.90:0/3978920758 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54040cdac0 0x7f540406b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:07.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.485+0000 7f5400a00640 1 -- 172.21.15.90:0/3978920758 >> v1:172.21.15.120:6789/0 conn(0x7f5404155030 legacy=0x7f540415efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:07.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.485+0000 7f5400a00640 1 -- 172.21.15.90:0/3978920758 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f540415b640 con 0x7f540406b600 2024-03-24T01:25:07.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.485+0000 7f5400a00640 1 -- 172.21.15.90:0/3978920758 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f53fc0019f0 con 0x7f540406b600 2024-03-24T01:25:07.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.485+0000 7f5400a00640 1 -- 172.21.15.90:0/3978920758 >> v1:172.21.15.90:6789/0 conn(0x7f540406b600 legacy=0x7f54040ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:07.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.485+0000 7f5400a00640 1 --2- 172.21.15.90:0/3978920758 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53e8003b70 0x7f53e8003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:07.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.485+0000 7f5400a00640 1 --2- 172.21.15.90:0/3978920758 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54040cdac0 0x7f53e8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:07.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.485+0000 7f5400a00640 1 --2- 172.21.15.90:0/3978920758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53e8004d40 0x7f53e80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:07.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.485+0000 7f5400a00640 1 -- 172.21.15.90:0/3978920758 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540406ba70 con 0x7f53e8003b70 2024-03-24T01:25:07.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.485+0000 7f5400a00640 1 -- 172.21.15.90:0/3978920758 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e800dc80 con 0x7f53e8004d40 2024-03-24T01:25:07.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.485+0000 7f5400a00640 1 -- 172.21.15.90:0/3978920758 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53e8005990 con 0x7f54040cdac0 2024-03-24T01:25:07.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.485+0000 7f5401e00640 1 --2- 172.21.15.90:0/3978920758 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53e8003b70 0x7f53e8003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:07.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.486+0000 7f5401400640 1 --2- 172.21.15.90:0/3978920758 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54040cdac0 0x7f53e8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:07.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.486+0000 7f5402800640 1 --2- 172.21.15.90:0/3978920758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53e8004d40 0x7f53e80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:07.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.486+0000 7f5401e00640 1 -- 172.21.15.90:0/3978920758 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54040cdac0 msgr2=0x7f53e8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:07.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.486+0000 7f5401e00640 1 --2- 172.21.15.90:0/3978920758 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54040cdac0 0x7f53e8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:07.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.486+0000 7f5401e00640 1 -- 172.21.15.90:0/3978920758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53e8004d40 msgr2=0x7f53e80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:07.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.486+0000 7f5401e00640 1 --2- 172.21.15.90:0/3978920758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53e8004d40 0x7f53e80051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:07.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.486+0000 7f5401e00640 1 -- 172.21.15.90:0/3978920758 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e8003520 con 0x7f53e8003b70 2024-03-24T01:25:07.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.486+0000 7f5401400640 1 --2- 172.21.15.90:0/3978920758 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54040cdac0 0x7f53e8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:07.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.487+0000 7f5401e00640 1 --2- 172.21.15.90:0/3978920758 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53e8003b70 0x7f53e8003f70 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f53f40018d0 tx=0x7f53e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fe756831a25b57c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:07.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.487+0000 7f5400a00640 1 -- 172.21.15.90:0/3978920758 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f4037070 con 0x7f53e8003b70 2024-03-24T01:25:07.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.487+0000 7f5400a00640 1 -- 172.21.15.90:0/3978920758 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53f40052d0 con 0x7f53e8003b70 2024-03-24T01:25:07.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.487+0000 7f5400a00640 1 -- 172.21.15.90:0/3978920758 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f40014f0 con 0x7f53e8003b70 2024-03-24T01:25:07.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.488+0000 7f5409000640 1 -- 172.21.15.90:0/3978920758 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53e8003b70 msgr2=0x7f53e8003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:07.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.488+0000 7f5409000640 1 --2- 172.21.15.90:0/3978920758 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53e8003b70 0x7f53e8003f70 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f53f40018d0 tx=0x7f53e8005830 comp rx=0 tx=0).stop 2024-03-24T01:25:07.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.488+0000 7f5402800640 1 -- 172.21.15.90:0/3978920758 reap_dead start 2024-03-24T01:25:07.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.488+0000 7f5409000640 1 -- 172.21.15.90:0/3978920758 shutdown_connections 2024-03-24T01:25:07.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.488+0000 7f5409000640 1 -- 172.21.15.90:0/3978920758 >> 172.21.15.90:0/3978920758 conn(0x7f54040b1770 msgr2=0x7f54040b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:07.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.488+0000 7f5409000640 1 -- 172.21.15.90:0/3978920758 shutdown_connections 2024-03-24T01:25:07.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.488+0000 7f5409000640 1 -- 172.21.15.90:0/3978920758 wait complete. 2024-03-24T01:25:07.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.490+0000 7f5409000640 1 Processor -- start 2024-03-24T01:25:07.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.490+0000 7f5409000640 1 -- start start 2024-03-24T01:25:07.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.491+0000 7f5409000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5404141fe0 0x7f54040c4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:07.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.491+0000 7f5409000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54041423c0 0x7f5404142840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:07.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.491+0000 7f5409000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f540413f070 0x7f540413f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:07.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.491+0000 7f5409000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5404155410 con 0x7f540413f070 2024-03-24T01:25:07.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.491+0000 7f5409000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540415bc00 con 0x7f5404141fe0 2024-03-24T01:25:07.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.491+0000 7f5409000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f540415af70 con 0x7f54041423c0 2024-03-24T01:25:07.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.491+0000 7f5402800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f540413f070 0x7f540413f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:07.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.491+0000 7f5402800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f540413f070 0x7f540413f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37564/0 (socket says 172.21.15.90:37564) 2024-03-24T01:25:07.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.491+0000 7f5402800640 1 -- 172.21.15.90:0/842293885 learned_addr learned my addr 172.21.15.90:0/842293885 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:07.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.491+0000 7f5401400640 1 --2- 172.21.15.90:0/842293885 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54041423c0 0x7f5404142840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:07.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.492+0000 7f5401e00640 1 --2- 172.21.15.90:0/842293885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5404141fe0 0x7f54040c4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:07.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.492+0000 7f5402800640 1 -- 172.21.15.90:0/842293885 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54041423c0 msgr2=0x7f5404142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:07.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.492+0000 7f5402800640 1 --2- 172.21.15.90:0/842293885 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54041423c0 0x7f5404142840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:07.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.492+0000 7f5402800640 1 -- 172.21.15.90:0/842293885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5404141fe0 msgr2=0x7f54040c4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:07.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.492+0000 7f5402800640 1 --2- 172.21.15.90:0/842293885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5404141fe0 0x7f54040c4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:07.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.492+0000 7f5402800640 1 -- 172.21.15.90:0/842293885 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5404150080 con 0x7f540413f070 2024-03-24T01:25:07.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.492+0000 7f5401400640 1 --2- 172.21.15.90:0/842293885 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54041423c0 0x7f5404142840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:07.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.492+0000 7f5402800640 1 --2- 172.21.15.90:0/842293885 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f540413f070 0x7f540413f4f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f53fc004bc0 tx=0x7f53fc00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:07.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.492+0000 7f53fa000640 1 -- 172.21.15.90:0/842293885 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc019070 con 0x7f540413f070 2024-03-24T01:25:07.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.492+0000 7f53fa000640 1 -- 172.21.15.90:0/842293885 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53fc00de80 con 0x7f540413f070 2024-03-24T01:25:07.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.492+0000 7f53fa000640 1 -- 172.21.15.90:0/842293885 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc0125f0 con 0x7f540413f070 2024-03-24T01:25:07.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.492+0000 7f5409000640 1 -- 172.21.15.90:0/842293885 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f540413fd80 con 0x7f540413f070 2024-03-24T01:25:07.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.492+0000 7f5409000640 1 -- 172.21.15.90:0/842293885 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5404140340 con 0x7f540413f070 2024-03-24T01:25:07.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.493+0000 7f5409000640 1 -- 172.21.15.90:0/842293885 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5404068060 con 0x7f540413f070 2024-03-24T01:25:07.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.494+0000 7f53fa000640 1 -- 172.21.15.90:0/842293885 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f53fc012790 con 0x7f540413f070 2024-03-24T01:25:07.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.495+0000 7f53fa000640 1 --2- 172.21.15.90:0/842293885 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f53e406bbd0 0x7f53e406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:07.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.495+0000 7f53fa000640 1 -- 172.21.15.90:0/842293885 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f53fc08af70 con 0x7f540413f070 2024-03-24T01:25:07.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.496+0000 7f5401e00640 1 --2- 172.21.15.90:0/842293885 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f53e406bbd0 0x7f53e406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:07.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.497+0000 7f5401e00640 1 --2- 172.21.15.90:0/842293885 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f53e406bbd0 0x7f53e406e090 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f53f4034000 tx=0x7f53f4031000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:07.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.500+0000 7f53fa000640 1 -- 172.21.15.90:0/842293885 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f53fc0595a0 con 0x7f540413f070 2024-03-24T01:25:07.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.702+0000 7f5409000640 1 -- 172.21.15.90:0/842293885 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5404154930 con 0x7f540413f070 2024-03-24T01:25:07.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.705+0000 7f53fa000640 1 -- 172.21.15.90:0/842293885 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18925 (secure 0 0 0) 0x7f53fc058ea0 con 0x7f540413f070 2024-03-24T01:25:07.706 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:25:07.706 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:24:53.236401+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:24:08.643966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T01:24:09.189904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T01:24:13.027323+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:25:07.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.708+0000 7f5409000640 1 -- 172.21.15.90:0/842293885 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f53e406bbd0 msgr2=0x7f53e406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:07.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.708+0000 7f5409000640 1 --2- 172.21.15.90:0/842293885 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f53e406bbd0 0x7f53e406e090 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f53f4034000 tx=0x7f53f4031000 comp rx=0 tx=0).stop 2024-03-24T01:25:07.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.708+0000 7f5409000640 1 -- 172.21.15.90:0/842293885 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f540413f070 msgr2=0x7f540413f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:07.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.708+0000 7f5409000640 1 --2- 172.21.15.90:0/842293885 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f540413f070 0x7f540413f4f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f53fc004bc0 tx=0x7f53fc00b4a0 comp rx=0 tx=0).stop 2024-03-24T01:25:07.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.708+0000 7f5409000640 1 -- 172.21.15.90:0/842293885 shutdown_connections 2024-03-24T01:25:07.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.708+0000 7f5409000640 1 --2- 172.21.15.90:0/842293885 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f53e406bbd0 0x7f53e406e090 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:07.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.708+0000 7f5409000640 1 --2- 172.21.15.90:0/842293885 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f540413f070 0x7f540413f4f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:07.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.708+0000 7f5409000640 1 --2- 172.21.15.90:0/842293885 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f54041423c0 0x7f5404142840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:07.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.708+0000 7f5409000640 1 --2- 172.21.15.90:0/842293885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5404141fe0 0x7f54040c4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:07.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.708+0000 7f5409000640 1 -- 172.21.15.90:0/842293885 >> 172.21.15.90:0/842293885 conn(0x7f54040b1770 msgr2=0x7f54040cec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:07.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.708+0000 7f5409000640 1 -- 172.21.15.90:0/842293885 shutdown_connections 2024-03-24T01:25:07.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.708+0000 7f5409000640 1 -- 172.21.15.90:0/842293885 wait complete. 2024-03-24T01:25:07.726 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T01:24:08.643966+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T01:24:09.189904+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T01:24:13.027323+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T01:25:07.727 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:25:07.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.941+0000 7fd4b4800640 1 Processor -- start 2024-03-24T01:25:07.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.941+0000 7fd4b4800640 1 -- start start 2024-03-24T01:25:07.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.942+0000 7fd4b4800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4ac0720a0 0x7fd4ac0ceaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:07.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.942+0000 7fd4b4800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd4ac0cf030 0x7fd4ac0b3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:07.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.942+0000 7fd4b4800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4ac0b2890 con 0x7fd4ac1452b0 2024-03-24T01:25:07.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.942+0000 7fd4b4800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4ac072990 con 0x7fd4ac0cf030 2024-03-24T01:25:07.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.942+0000 7fd4b4800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac0b2b70 con 0x7fd4ac0720a0 2024-03-24T01:25:07.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.943+0000 7fd4b1e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd4ac1452b0 0x7fd4ac071990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51536/0 (socket says 172.21.15.90:51536) 2024-03-24T01:25:07.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.943+0000 7fd4b1e00640 1 -- 172.21.15.90:0/3874248587 learned_addr learned my addr 172.21.15.90:0/3874248587 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:07.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.943+0000 7fd4b1400640 1 --2- 172.21.15.90:0/3874248587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4ac0720a0 0x7fd4ac0ceaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:07.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.943+0000 7fd4b2800640 1 --2- 172.21.15.90:0/3874248587 >> v2:172.21.15.120:3300/0 conn(0x7fd4ac0cf030 0x7fd4ac0b3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:07.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.943+0000 7fd4b0a00640 1 -- 172.21.15.90:0/3874248587 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 295829941 0 0) 0x7fd4ac0b2890 con 0x7fd4ac1452b0 2024-03-24T01:25:07.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.943+0000 7fd4b0a00640 1 -- 172.21.15.90:0/3874248587 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd498003520 con 0x7fd4ac1452b0 2024-03-24T01:25:07.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.943+0000 7fd4b0a00640 1 -- 172.21.15.90:0/3874248587 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 883132142 0 0) 0x7fd498003520 con 0x7fd4ac1452b0 2024-03-24T01:25:07.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.943+0000 7fd4b0a00640 1 -- 172.21.15.90:0/3874248587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4ac0720a0 msgr2=0x7fd4ac0ceaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:07.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.943+0000 7fd4b0a00640 1 --2- 172.21.15.90:0/3874248587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4ac0720a0 0x7fd4ac0ceaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:07.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.943+0000 7fd4b0a00640 1 -- 172.21.15.90:0/3874248587 >> v2:172.21.15.120:3300/0 conn(0x7fd4ac0cf030 msgr2=0x7fd4ac0b3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:07.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.943+0000 7fd4b0a00640 1 --2- 172.21.15.90:0/3874248587 >> v2:172.21.15.120:3300/0 conn(0x7fd4ac0cf030 0x7fd4ac0b3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:07.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.943+0000 7fd4b0a00640 1 -- 172.21.15.90:0/3874248587 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ac0b2e50 con 0x7fd4ac1452b0 2024-03-24T01:25:07.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.943+0000 7fd4b0a00640 1 -- 172.21.15.90:0/3874248587 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7fd4a40019f0 con 0x7fd4ac1452b0 2024-03-24T01:25:07.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.943+0000 7fd4b2800640 1 --2- 172.21.15.90:0/3874248587 >> v2:172.21.15.120:3300/0 conn(0x7fd4ac0cf030 0x7fd4ac0b3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:25:07.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.943+0000 7fd4b0a00640 1 -- 172.21.15.90:0/3874248587 >> v1:172.21.15.90:6789/0 conn(0x7fd4ac1452b0 legacy=0x7fd4ac071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:07.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.943+0000 7fd4b0a00640 1 --2- 172.21.15.90:0/3874248587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4ac0720a0 0x7fd4980039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:07.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.944+0000 7fd4b0a00640 1 --2- 172.21.15.90:0/3874248587 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd498003f80 0x7fd498004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:07.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.944+0000 7fd4b0a00640 1 --2- 172.21.15.90:0/3874248587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd498004d40 0x7fd4980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:07.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.944+0000 7fd4b0a00640 1 -- 172.21.15.90:0/3874248587 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac072990 con 0x7fd498003f80 2024-03-24T01:25:07.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.944+0000 7fd4b0a00640 1 -- 172.21.15.90:0/3874248587 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac0b2b70 con 0x7fd498004d40 2024-03-24T01:25:07.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.944+0000 7fd4b0a00640 1 -- 172.21.15.90:0/3874248587 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd49800ba70 con 0x7fd4ac0720a0 2024-03-24T01:25:07.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.944+0000 7fd4b1400640 1 --2- 172.21.15.90:0/3874248587 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd498003f80 0x7fd498004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:07.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.944+0000 7fd4b1e00640 1 --2- 172.21.15.90:0/3874248587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4ac0720a0 0x7fd4980039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:07.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.944+0000 7fd4b2800640 1 --2- 172.21.15.90:0/3874248587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd498004d40 0x7fd4980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:07.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.944+0000 7fd4b1400640 1 -- 172.21.15.90:0/3874248587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4ac0720a0 msgr2=0x7fd4980039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:07.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.944+0000 7fd4b1400640 1 --2- 172.21.15.90:0/3874248587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4ac0720a0 0x7fd4980039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:07.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.944+0000 7fd4b1400640 1 -- 172.21.15.90:0/3874248587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd498004d40 msgr2=0x7fd4980051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:07.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.944+0000 7fd4b1400640 1 --2- 172.21.15.90:0/3874248587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd498004d40 0x7fd4980051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:07.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.944+0000 7fd4b1400640 1 -- 172.21.15.90:0/3874248587 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd498003520 con 0x7fd498003f80 2024-03-24T01:25:07.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.945+0000 7fd4b1400640 1 --2- 172.21.15.90:0/3874248587 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd498003f80 0x7fd498004800 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd49c009ac0 tx=0x7fd498005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ed28527e302507 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:07.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.946+0000 7fd4b0a00640 1 -- 172.21.15.90:0/3874248587 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49c039070 con 0x7fd498003f80 2024-03-24T01:25:07.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.946+0000 7fd4b0a00640 1 -- 172.21.15.90:0/3874248587 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd49c02c6b0 con 0x7fd498003f80 2024-03-24T01:25:07.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.946+0000 7fd4b0a00640 1 -- 172.21.15.90:0/3874248587 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49c009690 con 0x7fd498003f80 2024-03-24T01:25:07.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.946+0000 7fd4b4800640 1 -- 172.21.15.90:0/3874248587 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd498003f80 msgr2=0x7fd498004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:07.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.946+0000 7fd4b4800640 1 --2- 172.21.15.90:0/3874248587 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd498003f80 0x7fd498004800 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd49c009ac0 tx=0x7fd498005830 comp rx=0 tx=0).stop 2024-03-24T01:25:07.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.946+0000 7fd4b2800640 1 -- 172.21.15.90:0/3874248587 reap_dead start 2024-03-24T01:25:07.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.946+0000 7fd4b4800640 1 -- 172.21.15.90:0/3874248587 shutdown_connections 2024-03-24T01:25:07.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.946+0000 7fd4b4800640 1 -- 172.21.15.90:0/3874248587 >> 172.21.15.90:0/3874248587 conn(0x7fd4ac0cf7e0 msgr2=0x7fd4ac0cfbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:07.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.946+0000 7fd4b4800640 1 -- 172.21.15.90:0/3874248587 shutdown_connections 2024-03-24T01:25:07.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.946+0000 7fd4b4800640 1 -- 172.21.15.90:0/3874248587 wait complete. 2024-03-24T01:25:07.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.948+0000 7fd4b4800640 1 Processor -- start 2024-03-24T01:25:07.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.948+0000 7fd4b4800640 1 -- start start 2024-03-24T01:25:07.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.949+0000 7fd4b4800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd498003f80 0x7fd4ac1f2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:07.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.949+0000 7fd4b4800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4ac14cd00 0x7fd4ac06da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:07.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.949+0000 7fd4b4800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4ac06df60 0x7fd4ac1428a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:07.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.949+0000 7fd4b4800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac0cf4c0 con 0x7fd498003f80 2024-03-24T01:25:07.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.949+0000 7fd4b4800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac0b3410 con 0x7fd4ac14cd00 2024-03-24T01:25:07.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.949+0000 7fd4b4800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac145690 con 0x7fd4ac06df60 2024-03-24T01:25:07.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.949+0000 7fd4b1e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd498003f80 0x7fd4ac1f2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:07.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.949+0000 7fd4b1e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd498003f80 0x7fd4ac1f2da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37584/0 (socket says 172.21.15.90:37584) 2024-03-24T01:25:07.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.949+0000 7fd4b1e00640 1 -- 172.21.15.90:0/471832960 learned_addr learned my addr 172.21.15.90:0/471832960 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:07.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.950+0000 7fd4b1400640 1 --2- 172.21.15.90:0/471832960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4ac14cd00 0x7fd4ac06da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:07.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.950+0000 7fd4b2800640 1 --2- 172.21.15.90:0/471832960 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4ac06df60 0x7fd4ac1428a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:07.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.950+0000 7fd4b1e00640 1 -- 172.21.15.90:0/471832960 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4ac06df60 msgr2=0x7fd4ac1428a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:07.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.950+0000 7fd4b1e00640 1 --2- 172.21.15.90:0/471832960 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4ac06df60 0x7fd4ac1428a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:07.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.950+0000 7fd4b1e00640 1 -- 172.21.15.90:0/471832960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4ac14cd00 msgr2=0x7fd4ac06da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:07.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.950+0000 7fd4b1e00640 1 --2- 172.21.15.90:0/471832960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4ac14cd00 0x7fd4ac06da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:07.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.950+0000 7fd4b1e00640 1 -- 172.21.15.90:0/471832960 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ac15c0f0 con 0x7fd498003f80 2024-03-24T01:25:07.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.950+0000 7fd4b1400640 1 --2- 172.21.15.90:0/471832960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4ac14cd00 0x7fd4ac06da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:07.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.950+0000 7fd4b1e00640 1 --2- 172.21.15.90:0/471832960 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd498003f80 0x7fd4ac1f2da0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4001ea0 tx=0x7fd4a400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:07.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.950+0000 7fd4a2000640 1 -- 172.21.15.90:0/471832960 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a4013420 con 0x7fd498003f80 2024-03-24T01:25:07.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.950+0000 7fd4a2000640 1 -- 172.21.15.90:0/471832960 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4a4018070 con 0x7fd498003f80 2024-03-24T01:25:07.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.951+0000 7fd4a2000640 1 -- 172.21.15.90:0/471832960 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a4013800 con 0x7fd498003f80 2024-03-24T01:25:07.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.951+0000 7fd4b4800640 1 -- 172.21.15.90:0/471832960 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4ac15c3d0 con 0x7fd498003f80 2024-03-24T01:25:07.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.951+0000 7fd4b4800640 1 -- 172.21.15.90:0/471832960 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4ac15c990 con 0x7fd498003f80 2024-03-24T01:25:07.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.952+0000 7fd4b4800640 1 -- 172.21.15.90:0/471832960 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4ac04f400 con 0x7fd498003f80 2024-03-24T01:25:07.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.953+0000 7fd4a2000640 1 -- 172.21.15.90:0/471832960 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd4a40139a0 con 0x7fd498003f80 2024-03-24T01:25:07.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.954+0000 7fd4a2000640 1 --2- 172.21.15.90:0/471832960 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd49006bba0 0x7fd49006e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:07.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.954+0000 7fd4a2000640 1 -- 172.21.15.90:0/471832960 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7fd4a408b610 con 0x7fd498003f80 2024-03-24T01:25:07.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.954+0000 7fd4b1400640 1 --2- 172.21.15.90:0/471832960 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd49006bba0 0x7fd49006e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:07.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.955+0000 7fd4b1400640 1 --2- 172.21.15.90:0/471832960 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd49006bba0 0x7fd49006e060 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd49c036000 tx=0x7fd49c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:07.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:07.958+0000 7fd4a2000640 1 -- 172.21.15.90:0/471832960 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd4a4059d50 con 0x7fd498003f80 2024-03-24T01:25:08.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.154+0000 7fd4b4800640 1 -- 172.21.15.90:0/471832960 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd4ac0af070 con 0x7fd498003f80 2024-03-24T01:25:08.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.155+0000 7fd4a2000640 1 -- 172.21.15.90:0/471832960 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd4a4059650 con 0x7fd498003f80 2024-03-24T01:25:08.155 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:25:08.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.158+0000 7fd4b4800640 1 -- 172.21.15.90:0/471832960 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd49006bba0 msgr2=0x7fd49006e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:08.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.158+0000 7fd4b4800640 1 --2- 172.21.15.90:0/471832960 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd49006bba0 0x7fd49006e060 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd49c036000 tx=0x7fd49c030000 comp rx=0 tx=0).stop 2024-03-24T01:25:08.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.158+0000 7fd4b4800640 1 -- 172.21.15.90:0/471832960 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd498003f80 msgr2=0x7fd4ac1f2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:08.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.158+0000 7fd4b4800640 1 --2- 172.21.15.90:0/471832960 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd498003f80 0x7fd4ac1f2da0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4001ea0 tx=0x7fd4a400e530 comp rx=0 tx=0).stop 2024-03-24T01:25:08.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.158+0000 7fd4b4800640 1 -- 172.21.15.90:0/471832960 shutdown_connections 2024-03-24T01:25:08.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.158+0000 7fd4b4800640 1 --2- 172.21.15.90:0/471832960 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd49006bba0 0x7fd49006e060 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:08.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.158+0000 7fd4b4800640 1 --2- 172.21.15.90:0/471832960 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd4ac06df60 0x7fd4ac1428a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:08.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.158+0000 7fd4b4800640 1 --2- 172.21.15.90:0/471832960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4ac14cd00 0x7fd4ac06da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:08.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.158+0000 7fd4b4800640 1 --2- 172.21.15.90:0/471832960 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd498003f80 0x7fd4ac1f2da0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:08.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.158+0000 7fd4b4800640 1 -- 172.21.15.90:0/471832960 >> 172.21.15.90:0/471832960 conn(0x7fd4ac0cf7e0 msgr2=0x7fd4ac0af290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:08.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.158+0000 7fd4b4800640 1 -- 172.21.15.90:0/471832960 shutdown_connections 2024-03-24T01:25:08.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.158+0000 7fd4b4800640 1 -- 172.21.15.90:0/471832960 wait complete. 2024-03-24T01:25:08.178 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T01:25:08.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.389+0000 7f7165000640 1 Processor -- start 2024-03-24T01:25:08.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.389+0000 7f7165000640 1 -- start start 2024-03-24T01:25:08.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.391+0000 7f7165000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7160071990 0x7f7160071dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:08.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.391+0000 7f7165000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f71600b2890 0x7f71600b2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:08.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.391+0000 7f7165000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71600b3530 con 0x7f7160161240 2024-03-24T01:25:08.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.391+0000 7f7165000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71600acff0 con 0x7f71600b2890 2024-03-24T01:25:08.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.391+0000 7f7165000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7160072310 con 0x7f7160071990 2024-03-24T01:25:08.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.391+0000 7f715d400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7160071990 0x7f7160071dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:08.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.391+0000 7f715e800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f71600b2890 0x7f71600b2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:08.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.391+0000 7f715d400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7160071990 0x7f7160071dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.90:51188/0 (socket says 172.21.15.90:51188) 2024-03-24T01:25:08.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.391+0000 7f715d400640 1 -- 172.21.15.90:0/2222590213 learned_addr learned my addr 172.21.15.90:0/2222590213 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:08.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.392+0000 7f715e800640 1 -- 172.21.15.90:0/2222590213 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7160071990 msgr2=0x7f7160071dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:08.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.392+0000 7f715e800640 1 --2- 172.21.15.90:0/2222590213 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7160071990 0x7f7160071dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:08.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.392+0000 7f715e800640 1 -- 172.21.15.90:0/2222590213 >> v1:172.21.15.120:6789/0 conn(0x7f7160161240 legacy=0x7f71600ac8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:08.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.392+0000 7f715e800640 1 -- 172.21.15.90:0/2222590213 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71600b3810 con 0x7f71600b2890 2024-03-24T01:25:08.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.393+0000 7f715e800640 1 --2- 172.21.15.90:0/2222590213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71600b2890 0x7f71600b2d00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7158009d90 tx=0x7f715802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d46a4f57cd7abec server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:08.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.394+0000 7f715ca00640 1 -- 172.21.15.90:0/2222590213 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7158039070 con 0x7f71600b2890 2024-03-24T01:25:08.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.394+0000 7f715ca00640 1 -- 172.21.15.90:0/2222590213 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f715802cc50 con 0x7f71600b2890 2024-03-24T01:25:08.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.394+0000 7f715ca00640 1 -- 172.21.15.90:0/2222590213 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7158034440 con 0x7f71600b2890 2024-03-24T01:25:08.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.395+0000 7f7165000640 1 -- 172.21.15.90:0/2222590213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71600b2890 msgr2=0x7f71600b2d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:08.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.395+0000 7f7165000640 1 --2- 172.21.15.90:0/2222590213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71600b2890 0x7f71600b2d00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7158009d90 tx=0x7f715802bea0 comp rx=0 tx=0).stop 2024-03-24T01:25:08.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.395+0000 7f7165000640 1 -- 172.21.15.90:0/2222590213 shutdown_connections 2024-03-24T01:25:08.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.395+0000 7f7165000640 1 --2- 172.21.15.90:0/2222590213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71600b2890 0x7f71600b2d00 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:08.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.395+0000 7f7165000640 1 --2- 172.21.15.90:0/2222590213 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7160071990 0x7f7160071dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:08.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.395+0000 7f7165000640 1 -- 172.21.15.90:0/2222590213 >> 172.21.15.90:0/2222590213 conn(0x7f71600b1cd0 msgr2=0x7f71600cf7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:08.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.395+0000 7f7165000640 1 -- 172.21.15.90:0/2222590213 shutdown_connections 2024-03-24T01:25:08.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.395+0000 7f7165000640 1 -- 172.21.15.90:0/2222590213 wait complete. 2024-03-24T01:25:08.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.397+0000 7f7165000640 1 Processor -- start 2024-03-24T01:25:08.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.397+0000 7f7165000640 1 -- start start 2024-03-24T01:25:08.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.399+0000 7f7165000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7160071990 0x7f716015a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:08.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.399+0000 7f7165000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71600b2890 0x7f716015bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:08.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.399+0000 7f7165000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7160161240 0x7f716015d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:08.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.399+0000 7f7165000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7160161620 con 0x7f71600b2890 2024-03-24T01:25:08.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.399+0000 7f7165000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71600b3240 con 0x7f7160161240 2024-03-24T01:25:08.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.399+0000 7f7165000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f716006e050 con 0x7f7160071990 2024-03-24T01:25:08.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.399+0000 7f715d400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71600b2890 0x7f716015bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:08.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.399+0000 7f715de00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7160071990 0x7f716015a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:08.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.399+0000 7f715d400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71600b2890 0x7f716015bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37588/0 (socket says 172.21.15.90:37588) 2024-03-24T01:25:08.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.399+0000 7f715d400640 1 -- 172.21.15.90:0/3184141295 learned_addr learned my addr 172.21.15.90:0/3184141295 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:08.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.399+0000 7f715e800640 1 --2- 172.21.15.90:0/3184141295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7160161240 0x7f716015d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:08.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.400+0000 7f715d400640 1 -- 172.21.15.90:0/3184141295 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7160071990 msgr2=0x7f716015a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:08.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.400+0000 7f715d400640 1 --2- 172.21.15.90:0/3184141295 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7160071990 0x7f716015a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:08.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.400+0000 7f715d400640 1 -- 172.21.15.90:0/3184141295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7160161240 msgr2=0x7f716015d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:08.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.400+0000 7f715d400640 1 --2- 172.21.15.90:0/3184141295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7160161240 0x7f716015d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:08.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.400+0000 7f715d400640 1 -- 172.21.15.90:0/3184141295 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f716015ac60 con 0x7f71600b2890 2024-03-24T01:25:08.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.400+0000 7f715d400640 1 --2- 172.21.15.90:0/3184141295 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71600b2890 0x7f716015bd00 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f714c00e840 tx=0x7f714c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:08.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.401+0000 7f7156000640 1 -- 172.21.15.90:0/3184141295 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c017070 con 0x7f71600b2890 2024-03-24T01:25:08.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.401+0000 7f7156000640 1 -- 172.21.15.90:0/3184141295 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f714c00c8e0 con 0x7f71600b2890 2024-03-24T01:25:08.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.401+0000 7f7156000640 1 -- 172.21.15.90:0/3184141295 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c00ccd0 con 0x7f71600b2890 2024-03-24T01:25:08.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.401+0000 7f7165000640 1 -- 172.21.15.90:0/3184141295 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f716015af40 con 0x7f71600b2890 2024-03-24T01:25:08.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.401+0000 7f7165000640 1 -- 172.21.15.90:0/3184141295 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f716015b500 con 0x7f71600b2890 2024-03-24T01:25:08.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.402+0000 7f7165000640 1 -- 172.21.15.90:0/3184141295 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7160072b20 con 0x7f71600b2890 2024-03-24T01:25:08.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.403+0000 7f7156000640 1 -- 172.21.15.90:0/3184141295 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f714c01a070 con 0x7f71600b2890 2024-03-24T01:25:08.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.404+0000 7f7156000640 1 --2- 172.21.15.90:0/3184141295 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f714006bbd0 0x7f714006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:08.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.404+0000 7f7156000640 1 -- 172.21.15.90:0/3184141295 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f714c08b130 con 0x7f71600b2890 2024-03-24T01:25:08.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.405+0000 7f715de00640 1 --2- 172.21.15.90:0/3184141295 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f714006bbd0 0x7f714006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:08.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.406+0000 7f715de00640 1 --2- 172.21.15.90:0/3184141295 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f714006bbd0 0x7f714006e090 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7150001780 tx=0x7f7150009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:08.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.409+0000 7f7156000640 1 -- 172.21.15.90:0/3184141295 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7160072b20 con 0x7f71600b2890 2024-03-24T01:25:08.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.606+0000 7f7165000640 1 -- 172.21.15.90:0/3184141295 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f716004f400 con 0x7f71600b2890 2024-03-24T01:25:08.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.606+0000 7f7156000640 1 -- 172.21.15.90:0/3184141295 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f714c059110 con 0x7f71600b2890 2024-03-24T01:25:08.607 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:25:08.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.609+0000 7f7165000640 1 -- 172.21.15.90:0/3184141295 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f714006bbd0 msgr2=0x7f714006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:08.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.609+0000 7f7165000640 1 --2- 172.21.15.90:0/3184141295 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f714006bbd0 0x7f714006e090 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7150001780 tx=0x7f7150009040 comp rx=0 tx=0).stop 2024-03-24T01:25:08.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.609+0000 7f7165000640 1 -- 172.21.15.90:0/3184141295 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71600b2890 msgr2=0x7f716015bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:08.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.609+0000 7f7165000640 1 --2- 172.21.15.90:0/3184141295 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71600b2890 0x7f716015bd00 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f714c00e840 tx=0x7f714c00ed00 comp rx=0 tx=0).stop 2024-03-24T01:25:08.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.609+0000 7f7165000640 1 -- 172.21.15.90:0/3184141295 shutdown_connections 2024-03-24T01:25:08.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.609+0000 7f7165000640 1 --2- 172.21.15.90:0/3184141295 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f714006bbd0 0x7f714006e090 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:08.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.609+0000 7f7165000640 1 --2- 172.21.15.90:0/3184141295 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71600b2890 0x7f716015bd00 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:08.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.609+0000 7f7165000640 1 --2- 172.21.15.90:0/3184141295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7160161240 0x7f716015d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:08.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.609+0000 7f7165000640 1 --2- 172.21.15.90:0/3184141295 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7160071990 0x7f716015a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:08.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.609+0000 7f7165000640 1 -- 172.21.15.90:0/3184141295 >> 172.21.15.90:0/3184141295 conn(0x7f71600b1cd0 msgr2=0x7f716006eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:08.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.609+0000 7f7165000640 1 -- 172.21.15.90:0/3184141295 shutdown_connections 2024-03-24T01:25:08.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.609+0000 7f7165000640 1 -- 172.21.15.90:0/3184141295 wait complete. 2024-03-24T01:25:08.628 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T01:25:08.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.833+0000 7fe725600640 1 Processor -- start 2024-03-24T01:25:08.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.833+0000 7fe725600640 1 -- start start 2024-03-24T01:25:08.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.834+0000 7fe725600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe720159620 0x7fe72006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:08.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.834+0000 7fe725600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe7200cf150 0x7fe7200ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:08.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.834+0000 7fe725600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe72006bca0 con 0x7fe72006b600 2024-03-24T01:25:08.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.834+0000 7fe725600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7200b4470 con 0x7fe7200cf150 2024-03-24T01:25:08.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.834+0000 7fe725600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe72006ba60 con 0x7fe720159620 2024-03-24T01:25:08.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.835+0000 7fe71f400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe7200cf150 0x7fe7200ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:08.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.835+0000 7fe71f400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe7200cf150 0x7fe7200ae530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37590/0 (socket says 172.21.15.90:37590) 2024-03-24T01:25:08.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.835+0000 7fe71f400640 1 -- 172.21.15.90:0/1668188294 learned_addr learned my addr 172.21.15.90:0/1668188294 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:08.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.835+0000 7fe71ea00640 1 --2- 172.21.15.90:0/1668188294 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe720159620 0x7fe72006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:08.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.835+0000 7fe71f400640 1 -- 172.21.15.90:0/1668188294 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe720159620 msgr2=0x7fe72006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:08.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.835+0000 7fe71f400640 1 --2- 172.21.15.90:0/1668188294 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe720159620 0x7fe72006b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:08.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.835+0000 7fe71f400640 1 -- 172.21.15.90:0/1668188294 >> v1:172.21.15.120:6789/0 conn(0x7fe72006b600 legacy=0x7fe7200cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:08.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.835+0000 7fe71f400640 1 -- 172.21.15.90:0/1668188294 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7200b36f0 con 0x7fe7200cf150 2024-03-24T01:25:08.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.835+0000 7fe71ea00640 1 --2- 172.21.15.90:0/1668188294 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe720159620 0x7fe72006b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:08.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.836+0000 7fe71f400640 1 --2- 172.21.15.90:0/1668188294 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7200cf150 0x7fe7200ae530 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe718009e50 tx=0x7fe71802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80683f9776aef5e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:08.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.837+0000 7fe71e000640 1 -- 172.21.15.90:0/1668188294 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe71802c9f0 con 0x7fe7200cf150 2024-03-24T01:25:08.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.837+0000 7fe71e000640 1 -- 172.21.15.90:0/1668188294 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe71802cb90 con 0x7fe7200cf150 2024-03-24T01:25:08.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.837+0000 7fe71e000640 1 -- 172.21.15.90:0/1668188294 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7180343f0 con 0x7fe7200cf150 2024-03-24T01:25:08.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.837+0000 7fe725600640 1 -- 172.21.15.90:0/1668188294 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7200cf150 msgr2=0x7fe7200ae530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:08.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.837+0000 7fe725600640 1 --2- 172.21.15.90:0/1668188294 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7200cf150 0x7fe7200ae530 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe718009e50 tx=0x7fe71802be50 comp rx=0 tx=0).stop 2024-03-24T01:25:08.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.837+0000 7fe725600640 1 -- 172.21.15.90:0/1668188294 shutdown_connections 2024-03-24T01:25:08.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.837+0000 7fe725600640 1 --2- 172.21.15.90:0/1668188294 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7200cf150 0x7fe7200ae530 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:08.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.837+0000 7fe725600640 1 --2- 172.21.15.90:0/1668188294 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe720159620 0x7fe72006b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:08.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.837+0000 7fe725600640 1 -- 172.21.15.90:0/1668188294 >> 172.21.15.90:0/1668188294 conn(0x7fe7200b1770 msgr2=0x7fe7200b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:08.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.837+0000 7fe725600640 1 -- 172.21.15.90:0/1668188294 shutdown_connections 2024-03-24T01:25:08.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.837+0000 7fe725600640 1 -- 172.21.15.90:0/1668188294 wait complete. 2024-03-24T01:25:08.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.839+0000 7fe725600640 1 Processor -- start 2024-03-24T01:25:08.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.839+0000 7fe725600640 1 -- start start 2024-03-24T01:25:08.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.840+0000 7fe725600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe72006b600 0x7fe720154450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:08.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.840+0000 7fe725600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7200cf150 0x7fe72014f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:08.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.840+0000 7fe725600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe720159620 0x7fe720155400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:08.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.840+0000 7fe725600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7200b3f50 con 0x7fe72006b600 2024-03-24T01:25:08.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.840+0000 7fe725600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7200b3c70 con 0x7fe720159620 2024-03-24T01:25:08.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.840+0000 7fe725600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe720159a00 con 0x7fe7200cf150 2024-03-24T01:25:08.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.840+0000 7fe71ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe72006b600 0x7fe720154450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:08.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.840+0000 7fe71ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe72006b600 0x7fe720154450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37598/0 (socket says 172.21.15.90:37598) 2024-03-24T01:25:08.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.840+0000 7fe71ea00640 1 -- 172.21.15.90:0/2224305550 learned_addr learned my addr 172.21.15.90:0/2224305550 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:08.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.841+0000 7fe717e00640 1 --2- 172.21.15.90:0/2224305550 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7200cf150 0x7fe72014f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:08.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.841+0000 7fe71f400640 1 --2- 172.21.15.90:0/2224305550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe720159620 0x7fe720155400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:08.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.841+0000 7fe71ea00640 1 -- 172.21.15.90:0/2224305550 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7200cf150 msgr2=0x7fe72014f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:08.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.841+0000 7fe71ea00640 1 --2- 172.21.15.90:0/2224305550 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7200cf150 0x7fe72014f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:08.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.841+0000 7fe71ea00640 1 -- 172.21.15.90:0/2224305550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe720159620 msgr2=0x7fe720155400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:08.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.841+0000 7fe71ea00640 1 --2- 172.21.15.90:0/2224305550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe720159620 0x7fe720155400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:08.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.841+0000 7fe71ea00640 1 -- 172.21.15.90:0/2224305550 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7201533b0 con 0x7fe72006b600 2024-03-24T01:25:08.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.841+0000 7fe71ea00640 1 --2- 172.21.15.90:0/2224305550 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe72006b600 0x7fe720154450 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe71000c820 tx=0x7fe71000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:08.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.841+0000 7fe716a00640 1 -- 172.21.15.90:0/2224305550 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe710015070 con 0x7fe72006b600 2024-03-24T01:25:08.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.841+0000 7fe716a00640 1 -- 172.21.15.90:0/2224305550 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe710007a30 con 0x7fe72006b600 2024-03-24T01:25:08.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.841+0000 7fe716a00640 1 -- 172.21.15.90:0/2224305550 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe710007de0 con 0x7fe72006b600 2024-03-24T01:25:08.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.841+0000 7fe725600640 1 -- 172.21.15.90:0/2224305550 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe720153690 con 0x7fe72006b600 2024-03-24T01:25:08.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.842+0000 7fe725600640 1 -- 172.21.15.90:0/2224305550 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe720153c50 con 0x7fe72006b600 2024-03-24T01:25:08.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.843+0000 7fe725600640 1 -- 172.21.15.90:0/2224305550 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe72004f400 con 0x7fe72006b600 2024-03-24T01:25:08.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.843+0000 7fe716a00640 1 -- 172.21.15.90:0/2224305550 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe710018070 con 0x7fe72006b600 2024-03-24T01:25:08.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.844+0000 7fe716a00640 1 --2- 172.21.15.90:0/2224305550 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe6fc06bbd0 0x7fe6fc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:08.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.844+0000 7fe716a00640 1 -- 172.21.15.90:0/2224305550 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7fe71008ae70 con 0x7fe72006b600 2024-03-24T01:25:08.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.845+0000 7fe717e00640 1 --2- 172.21.15.90:0/2224305550 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe6fc06bbd0 0x7fe6fc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:08.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.846+0000 7fe717e00640 1 --2- 172.21.15.90:0/2224305550 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe6fc06bbd0 0x7fe6fc06e090 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe7200ce7d0 tx=0x7fe708009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:08.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:08.849+0000 7fe716a00640 1 -- 172.21.15.90:0/2224305550 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe7100594b0 con 0x7fe72006b600 2024-03-24T01:25:09.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.030+0000 7fe725600640 1 -- 172.21.15.90:0/2224305550 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe7200b2e90 con 0x7fe72006b600 2024-03-24T01:25:09.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.031+0000 7fe716a00640 1 -- 172.21.15.90:0/2224305550 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe710058db0 con 0x7fe72006b600 2024-03-24T01:25:09.032 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-24T01:25:09.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.034+0000 7fe725600640 1 -- 172.21.15.90:0/2224305550 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe6fc06bbd0 msgr2=0x7fe6fc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:09.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.034+0000 7fe725600640 1 --2- 172.21.15.90:0/2224305550 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe6fc06bbd0 0x7fe6fc06e090 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe7200ce7d0 tx=0x7fe708009040 comp rx=0 tx=0).stop 2024-03-24T01:25:09.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.034+0000 7fe725600640 1 -- 172.21.15.90:0/2224305550 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe72006b600 msgr2=0x7fe720154450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:09.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.034+0000 7fe725600640 1 --2- 172.21.15.90:0/2224305550 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe72006b600 0x7fe720154450 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe71000c820 tx=0x7fe71000cce0 comp rx=0 tx=0).stop 2024-03-24T01:25:09.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.034+0000 7fe725600640 1 -- 172.21.15.90:0/2224305550 shutdown_connections 2024-03-24T01:25:09.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.034+0000 7fe725600640 1 --2- 172.21.15.90:0/2224305550 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe6fc06bbd0 0x7fe6fc06e090 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:09.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.034+0000 7fe725600640 1 --2- 172.21.15.90:0/2224305550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe720159620 0x7fe720155400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:09.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.034+0000 7fe725600640 1 --2- 172.21.15.90:0/2224305550 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7200cf150 0x7fe72014f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:09.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.034+0000 7fe725600640 1 --2- 172.21.15.90:0/2224305550 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe72006b600 0x7fe720154450 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:09.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.034+0000 7fe725600640 1 -- 172.21.15.90:0/2224305550 >> 172.21.15.90:0/2224305550 conn(0x7fe7200b1770 msgr2=0x7fe7200cf770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:09.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.034+0000 7fe725600640 1 -- 172.21.15.90:0/2224305550 shutdown_connections 2024-03-24T01:25:09.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.034+0000 7fe725600640 1 -- 172.21.15.90:0/2224305550 wait complete. 2024-03-24T01:25:09.052 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T01:25:09.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.262+0000 7f2643e00640 1 Processor -- start 2024-03-24T01:25:09.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.263+0000 7f2643e00640 1 -- start start 2024-03-24T01:25:09.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.264+0000 7f2643e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f263c1452b0 0x7f263c071990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:09.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.264+0000 7f2643e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f263c0cf200 0x7f263c0b3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:09.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.264+0000 7f2643e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f263c0b2890 con 0x7f263c071ed0 2024-03-24T01:25:09.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.264+0000 7f2643e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f263c072330 con 0x7f263c1452b0 2024-03-24T01:25:09.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.264+0000 7f2643e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f263c072990 con 0x7f263c0cf200 2024-03-24T01:25:09.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.264+0000 7f2641400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f263c1452b0 0x7f263c071990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:09.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.264+0000 7f2640a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f263c071ed0 0x7f263c0ceaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51548/0 (socket says 172.21.15.90:51548) 2024-03-24T01:25:09.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.264+0000 7f2640a00640 1 -- 172.21.15.90:0/369753414 learned_addr learned my addr 172.21.15.90:0/369753414 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:09.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.265+0000 7f2641e00640 1 --2- 172.21.15.90:0/369753414 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f263c0cf200 0x7f263c0b3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:09.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.265+0000 7f2641400640 1 -- 172.21.15.90:0/369753414 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f263c0cf200 msgr2=0x7f263c0b3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:09.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.265+0000 7f2641400640 1 --2- 172.21.15.90:0/369753414 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f263c0cf200 0x7f263c0b3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:09.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.265+0000 7f2641400640 1 -- 172.21.15.90:0/369753414 >> v1:172.21.15.90:6789/0 conn(0x7f263c071ed0 legacy=0x7f263c0ceaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:09.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.265+0000 7f2641400640 1 -- 172.21.15.90:0/369753414 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f263c0b2b70 con 0x7f263c1452b0 2024-03-24T01:25:09.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.265+0000 7f2641e00640 1 --2- 172.21.15.90:0/369753414 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f263c0cf200 0x7f263c0b3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:09.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.265+0000 7f263be00640 1 -- 172.21.15.90:0/369753414 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2358692285 0 0) 0x7f263c0b2890 con 0x7f263c071ed0 2024-03-24T01:25:09.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.266+0000 7f2641400640 1 --2- 172.21.15.90:0/369753414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f263c1452b0 0x7f263c071990 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f26300097a0 tx=0x7f263002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8c5ff559ae8501e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:09.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.266+0000 7f263be00640 1 -- 172.21.15.90:0/369753414 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2630035020 con 0x7f263c1452b0 2024-03-24T01:25:09.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.266+0000 7f263be00640 1 -- 172.21.15.90:0/369753414 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2630031810 con 0x7f263c1452b0 2024-03-24T01:25:09.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.266+0000 7f263be00640 1 -- 172.21.15.90:0/369753414 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2630031be0 con 0x7f263c1452b0 2024-03-24T01:25:09.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.267+0000 7f2643e00640 1 -- 172.21.15.90:0/369753414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f263c1452b0 msgr2=0x7f263c071990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:09.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.267+0000 7f2643e00640 1 --2- 172.21.15.90:0/369753414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f263c1452b0 0x7f263c071990 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f26300097a0 tx=0x7f263002b8b0 comp rx=0 tx=0).stop 2024-03-24T01:25:09.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.267+0000 7f2643e00640 1 -- 172.21.15.90:0/369753414 shutdown_connections 2024-03-24T01:25:09.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.267+0000 7f2643e00640 1 --2- 172.21.15.90:0/369753414 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f263c0cf200 0x7f263c0b3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:09.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.267+0000 7f2643e00640 1 --2- 172.21.15.90:0/369753414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f263c1452b0 0x7f263c071990 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:09.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.267+0000 7f2643e00640 1 -- 172.21.15.90:0/369753414 >> 172.21.15.90:0/369753414 conn(0x7f263c0cf7e0 msgr2=0x7f263c0cfbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:09.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.267+0000 7f2643e00640 1 -- 172.21.15.90:0/369753414 shutdown_connections 2024-03-24T01:25:09.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.267+0000 7f2643e00640 1 -- 172.21.15.90:0/369753414 wait complete. 2024-03-24T01:25:09.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.269+0000 7f2643e00640 1 Processor -- start 2024-03-24T01:25:09.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.269+0000 7f2643e00640 1 -- start start 2024-03-24T01:25:09.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.270+0000 7f2643e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f263c071ed0 0x7f263c1390f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:09.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.270+0000 7f2643e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f263c0cf200 0x7f263c15a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:09.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.270+0000 7f2643e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f263c1452b0 0x7f263c15a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:09.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.270+0000 7f2643e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f263c0b3410 con 0x7f263c071ed0 2024-03-24T01:25:09.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.270+0000 7f2643e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f263c0b3130 con 0x7f263c0cf200 2024-03-24T01:25:09.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.270+0000 7f2643e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f263c145690 con 0x7f263c1452b0 2024-03-24T01:25:09.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.270+0000 7f2641400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f263c071ed0 0x7f263c1390f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:09.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.270+0000 7f2641400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f263c071ed0 0x7f263c1390f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37618/0 (socket says 172.21.15.90:37618) 2024-03-24T01:25:09.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.270+0000 7f2641400640 1 -- 172.21.15.90:0/14974124 learned_addr learned my addr 172.21.15.90:0/14974124 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:09.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.270+0000 7f2640a00640 1 --2- 172.21.15.90:0/14974124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f263c0cf200 0x7f263c15a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:09.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.270+0000 7f2641e00640 1 --2- 172.21.15.90:0/14974124 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f263c1452b0 0x7f263c15a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:09.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.270+0000 7f2641400640 1 -- 172.21.15.90:0/14974124 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f263c1452b0 msgr2=0x7f263c15a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:09.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.270+0000 7f2641400640 1 --2- 172.21.15.90:0/14974124 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f263c1452b0 0x7f263c15a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:09.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.270+0000 7f2641400640 1 -- 172.21.15.90:0/14974124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f263c0cf200 msgr2=0x7f263c15a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:09.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.270+0000 7f2641400640 1 --2- 172.21.15.90:0/14974124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f263c0cf200 0x7f263c15a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:09.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.270+0000 7f2641400640 1 -- 172.21.15.90:0/14974124 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f263c06da90 con 0x7f263c071ed0 2024-03-24T01:25:09.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.270+0000 7f2640a00640 1 --2- 172.21.15.90:0/14974124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f263c0cf200 0x7f263c15a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:09.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.271+0000 7f2641400640 1 --2- 172.21.15.90:0/14974124 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f263c071ed0 0x7f263c1390f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2630034040 tx=0x7f26300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:09.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.271+0000 7f2639600640 1 -- 172.21.15.90:0/14974124 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2630040070 con 0x7f263c071ed0 2024-03-24T01:25:09.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.271+0000 7f2639600640 1 -- 172.21.15.90:0/14974124 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2630039cf0 con 0x7f263c071ed0 2024-03-24T01:25:09.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.271+0000 7f2639600640 1 -- 172.21.15.90:0/14974124 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2630035040 con 0x7f263c071ed0 2024-03-24T01:25:09.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.271+0000 7f2643e00640 1 -- 172.21.15.90:0/14974124 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f263c06dd70 con 0x7f263c071ed0 2024-03-24T01:25:09.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.271+0000 7f2643e00640 1 -- 172.21.15.90:0/14974124 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f263c139630 con 0x7f263c071ed0 2024-03-24T01:25:09.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.272+0000 7f2643e00640 1 -- 172.21.15.90:0/14974124 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f263c068ad0 con 0x7f263c071ed0 2024-03-24T01:25:09.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.273+0000 7f2639600640 1 -- 172.21.15.90:0/14974124 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2630038020 con 0x7f263c071ed0 2024-03-24T01:25:09.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.274+0000 7f2639600640 1 --2- 172.21.15.90:0/14974124 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f261c06bbd0 0x7f261c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:09.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.274+0000 7f2639600640 1 -- 172.21.15.90:0/14974124 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f26300acbc0 con 0x7f263c071ed0 2024-03-24T01:25:09.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.275+0000 7f2640a00640 1 --2- 172.21.15.90:0/14974124 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f261c06bbd0 0x7f261c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:09.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.276+0000 7f2640a00640 1 --2- 172.21.15.90:0/14974124 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f261c06bbd0 0x7f261c06e090 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2628000f60 tx=0x7f2628009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:09.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.278+0000 7f2639600640 1 -- 172.21.15.90:0/14974124 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f263c068ad0 con 0x7f263c071ed0 2024-03-24T01:25:09.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.462+0000 7f2643e00640 1 -- 172.21.15.90:0/14974124 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f263c074190 con 0x7f263c071ed0 2024-03-24T01:25:09.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.463+0000 7f2639600640 1 -- 172.21.15.90:0/14974124 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f263007b280 con 0x7f263c071ed0 2024-03-24T01:25:09.463 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:25:09.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.465+0000 7f2643e00640 1 -- 172.21.15.90:0/14974124 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f261c06bbd0 msgr2=0x7f261c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:09.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.465+0000 7f2643e00640 1 --2- 172.21.15.90:0/14974124 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f261c06bbd0 0x7f261c06e090 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2628000f60 tx=0x7f2628009040 comp rx=0 tx=0).stop 2024-03-24T01:25:09.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.465+0000 7f2643e00640 1 -- 172.21.15.90:0/14974124 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f263c071ed0 msgr2=0x7f263c1390f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:09.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.465+0000 7f2643e00640 1 --2- 172.21.15.90:0/14974124 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f263c071ed0 0x7f263c1390f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2630034040 tx=0x7f26300046c0 comp rx=0 tx=0).stop 2024-03-24T01:25:09.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.466+0000 7f2643e00640 1 -- 172.21.15.90:0/14974124 shutdown_connections 2024-03-24T01:25:09.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.466+0000 7f2643e00640 1 --2- 172.21.15.90:0/14974124 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f261c06bbd0 0x7f261c06e090 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:09.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.466+0000 7f2643e00640 1 --2- 172.21.15.90:0/14974124 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f263c1452b0 0x7f263c15a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:09.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.466+0000 7f2643e00640 1 --2- 172.21.15.90:0/14974124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f263c0cf200 0x7f263c15a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:09.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.466+0000 7f2643e00640 1 --2- 172.21.15.90:0/14974124 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f263c071ed0 0x7f263c1390f0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:09.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.466+0000 7f2643e00640 1 -- 172.21.15.90:0/14974124 >> 172.21.15.90:0/14974124 conn(0x7f263c0cf7e0 msgr2=0x7f263c0b04d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:09.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.466+0000 7f2643e00640 1 -- 172.21.15.90:0/14974124 shutdown_connections 2024-03-24T01:25:09.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.466+0000 7f2643e00640 1 -- 172.21.15.90:0/14974124 wait complete. 2024-03-24T01:25:09.484 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:25:09.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.689+0000 7f3e3fe00640 1 Processor -- start 2024-03-24T01:25:09.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.689+0000 7f3e3fe00640 1 -- start start 2024-03-24T01:25:09.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.690+0000 7f3e3fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3e40144420 0x7f3e4006d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:09.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.690+0000 7f3e3fe00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3e40144800 0x7f3e400cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:09.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.691+0000 7f3e3fe00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e400cceb0 con 0x7f3e400cc8f0 2024-03-24T01:25:09.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.691+0000 7f3e3fe00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e400cd190 con 0x7f3e40144420 2024-03-24T01:25:09.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.691+0000 7f3e3fe00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e400ceb70 con 0x7f3e40144800 2024-03-24T01:25:09.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.691+0000 7f3e3ea00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3e40144420 0x7f3e4006d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:09.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.691+0000 7f3e3ea00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3e40144420 0x7f3e4006d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37632/0 (socket says 172.21.15.90:37632) 2024-03-24T01:25:09.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.691+0000 7f3e3ea00640 1 -- 172.21.15.90:0/3452906443 learned_addr learned my addr 172.21.15.90:0/3452906443 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:09.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.691+0000 7f3e3ea00640 1 -- 172.21.15.90:0/3452906443 >> v2:172.21.15.120:3300/0 conn(0x7f3e40144800 msgr2=0x7f3e400cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:09.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.691+0000 7f3e3ea00640 1 --2- 172.21.15.90:0/3452906443 >> v2:172.21.15.120:3300/0 conn(0x7f3e40144800 0x7f3e400cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:09.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.691+0000 7f3e3ea00640 1 -- 172.21.15.90:0/3452906443 >> v1:172.21.15.120:6789/0 conn(0x7f3e400cc8f0 legacy=0x7f3e400cf8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:09.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.691+0000 7f3e3ea00640 1 -- 172.21.15.90:0/3452906443 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e400cee50 con 0x7f3e40144420 2024-03-24T01:25:09.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.692+0000 7f3e3ea00640 1 --2- 172.21.15.90:0/3452906443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e40144420 0x7f3e4006d900 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3e4006bf50 tx=0x7f3e3002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16b8a2e80cbc2805 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:09.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.693+0000 7f3e3e000640 1 -- 172.21.15.90:0/3452906443 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3002e070 con 0x7f3e40144420 2024-03-24T01:25:09.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.693+0000 7f3e3e000640 1 -- 172.21.15.90:0/3452906443 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e30037070 con 0x7f3e40144420 2024-03-24T01:25:09.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.693+0000 7f3e3e000640 1 -- 172.21.15.90:0/3452906443 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e30031a00 con 0x7f3e40144420 2024-03-24T01:25:09.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.693+0000 7f3e3fe00640 1 -- 172.21.15.90:0/3452906443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e40144420 msgr2=0x7f3e4006d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:09.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.693+0000 7f3e3fe00640 1 --2- 172.21.15.90:0/3452906443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e40144420 0x7f3e4006d900 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3e4006bf50 tx=0x7f3e3002b880 comp rx=0 tx=0).stop 2024-03-24T01:25:09.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.693+0000 7f3e3fe00640 1 -- 172.21.15.90:0/3452906443 shutdown_connections 2024-03-24T01:25:09.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.693+0000 7f3e3fe00640 1 --2- 172.21.15.90:0/3452906443 >> v2:172.21.15.120:3300/0 conn(0x7f3e40144800 0x7f3e400cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:09.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.693+0000 7f3e3fe00640 1 --2- 172.21.15.90:0/3452906443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e40144420 0x7f3e4006d900 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:09.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.694+0000 7f3e3fe00640 1 -- 172.21.15.90:0/3452906443 >> 172.21.15.90:0/3452906443 conn(0x7f3e4006b0c0 msgr2=0x7f3e4006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:09.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.694+0000 7f3e3fe00640 1 -- 172.21.15.90:0/3452906443 shutdown_connections 2024-03-24T01:25:09.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.694+0000 7f3e3fe00640 1 -- 172.21.15.90:0/3452906443 wait complete. 2024-03-24T01:25:09.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.695+0000 7f3e3fe00640 1 Processor -- start 2024-03-24T01:25:09.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.695+0000 7f3e3fe00640 1 -- start start 2024-03-24T01:25:09.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.696+0000 7f3e3fe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e400cc8f0 0x7f3e4015b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:09.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.696+0000 7f3e3fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e40144420 0x7f3e401e7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:09.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.696+0000 7f3e3fe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e40144800 0x7f3e4015dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:09.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.696+0000 7f3e3fe00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e400ce910 con 0x7f3e40144420 2024-03-24T01:25:09.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.696+0000 7f3e3fe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e400ce790 con 0x7f3e40144800 2024-03-24T01:25:09.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.696+0000 7f3e3fe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4015f540 con 0x7f3e400cc8f0 2024-03-24T01:25:09.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.697+0000 7f3e36000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e40144420 0x7f3e401e7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:09.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.697+0000 7f3e36000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e40144420 0x7f3e401e7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37648/0 (socket says 172.21.15.90:37648) 2024-03-24T01:25:09.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.697+0000 7f3e36000640 1 -- 172.21.15.90:0/2277755571 learned_addr learned my addr 172.21.15.90:0/2277755571 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:09.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.697+0000 7f3e3f400640 1 --2- 172.21.15.90:0/2277755571 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e40144800 0x7f3e4015dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:09.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.697+0000 7f3e3ea00640 1 --2- 172.21.15.90:0/2277755571 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e400cc8f0 0x7f3e4015b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:09.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.697+0000 7f3e36000640 1 -- 172.21.15.90:0/2277755571 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e400cc8f0 msgr2=0x7f3e4015b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:09.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.697+0000 7f3e36000640 1 --2- 172.21.15.90:0/2277755571 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e400cc8f0 0x7f3e4015b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:09.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.697+0000 7f3e36000640 1 -- 172.21.15.90:0/2277755571 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e40144800 msgr2=0x7f3e4015dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:09.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.697+0000 7f3e36000640 1 --2- 172.21.15.90:0/2277755571 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e40144800 0x7f3e4015dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:09.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.697+0000 7f3e36000640 1 -- 172.21.15.90:0/2277755571 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e401e48a0 con 0x7f3e40144420 2024-03-24T01:25:09.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.697+0000 7f3e3f400640 1 --2- 172.21.15.90:0/2277755571 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e40144800 0x7f3e4015dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:09.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.697+0000 7f3e3ea00640 1 --2- 172.21.15.90:0/2277755571 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e400cc8f0 0x7f3e4015b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:09.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.698+0000 7f3e36000640 1 --2- 172.21.15.90:0/2277755571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e40144420 0x7f3e401e7df0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3e2800b410 tx=0x7f3e2800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:09.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.698+0000 7f3e37400640 1 -- 172.21.15.90:0/2277755571 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e28015020 con 0x7f3e40144420 2024-03-24T01:25:09.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.698+0000 7f3e37400640 1 -- 172.21.15.90:0/2277755571 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e28011850 con 0x7f3e40144420 2024-03-24T01:25:09.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.698+0000 7f3e37400640 1 -- 172.21.15.90:0/2277755571 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e28011c40 con 0x7f3e40144420 2024-03-24T01:25:09.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.698+0000 7f3e3fe00640 1 -- 172.21.15.90:0/2277755571 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e401e4b80 con 0x7f3e40144420 2024-03-24T01:25:09.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.698+0000 7f3e3fe00640 1 -- 172.21.15.90:0/2277755571 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e401e5140 con 0x7f3e40144420 2024-03-24T01:25:09.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.699+0000 7f3e3fe00640 1 -- 172.21.15.90:0/2277755571 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e4004f400 con 0x7f3e40144420 2024-03-24T01:25:09.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.700+0000 7f3e37400640 1 -- 172.21.15.90:0/2277755571 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3e28018020 con 0x7f3e40144420 2024-03-24T01:25:09.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.701+0000 7f3e37400640 1 --2- 172.21.15.90:0/2277755571 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3e1c06bbd0 0x7f3e1c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:09.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.701+0000 7f3e37400640 1 -- 172.21.15.90:0/2277755571 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f3e2808ae00 con 0x7f3e40144420 2024-03-24T01:25:09.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.702+0000 7f3e3ea00640 1 --2- 172.21.15.90:0/2277755571 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3e1c06bbd0 0x7f3e1c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:09.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.703+0000 7f3e3ea00640 1 --2- 172.21.15.90:0/2277755571 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3e1c06bbd0 0x7f3e1c06e090 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3e30036000 tx=0x7f3e30008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:09.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.706+0000 7f3e37400640 1 -- 172.21.15.90:0/2277755571 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3e2801f020 con 0x7f3e40144420 2024-03-24T01:25:09.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.905+0000 7f3e3fe00640 1 -- 172.21.15.90:0/2277755571 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e400adf60 con 0x7f3e40144420 2024-03-24T01:25:09.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.907+0000 7f3e37400640 1 -- 172.21.15.90:0/2277755571 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18925 (secure 0 0 0) 0x7f3e280172c0 con 0x7f3e40144420 2024-03-24T01:25:09.908 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:25:09.908 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:24:53.236401+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:24:08.643966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T01:24:09.189904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T01:24:13.027323+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:25:09.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.910+0000 7f3e3fe00640 1 -- 172.21.15.90:0/2277755571 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3e1c06bbd0 msgr2=0x7f3e1c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:09.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.910+0000 7f3e3fe00640 1 --2- 172.21.15.90:0/2277755571 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3e1c06bbd0 0x7f3e1c06e090 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3e30036000 tx=0x7f3e30008030 comp rx=0 tx=0).stop 2024-03-24T01:25:09.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.910+0000 7f3e3fe00640 1 -- 172.21.15.90:0/2277755571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e40144420 msgr2=0x7f3e401e7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:09.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.910+0000 7f3e3fe00640 1 --2- 172.21.15.90:0/2277755571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e40144420 0x7f3e401e7df0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3e2800b410 tx=0x7f3e2800b8d0 comp rx=0 tx=0).stop 2024-03-24T01:25:09.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.910+0000 7f3e3fe00640 1 -- 172.21.15.90:0/2277755571 shutdown_connections 2024-03-24T01:25:09.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.910+0000 7f3e3fe00640 1 --2- 172.21.15.90:0/2277755571 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3e1c06bbd0 0x7f3e1c06e090 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:09.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.910+0000 7f3e3fe00640 1 --2- 172.21.15.90:0/2277755571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e40144420 0x7f3e401e7df0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:09.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.910+0000 7f3e3fe00640 1 --2- 172.21.15.90:0/2277755571 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e40144800 0x7f3e4015dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:09.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.910+0000 7f3e3fe00640 1 --2- 172.21.15.90:0/2277755571 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e400cc8f0 0x7f3e4015b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:09.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.910+0000 7f3e3fe00640 1 -- 172.21.15.90:0/2277755571 >> 172.21.15.90:0/2277755571 conn(0x7f3e4006b0c0 msgr2=0x7f3e400cf370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:09.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.910+0000 7f3e3fe00640 1 -- 172.21.15.90:0/2277755571 shutdown_connections 2024-03-24T01:25:09.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:09.911+0000 7f3e3fe00640 1 -- 172.21.15.90:0/2277755571 wait complete. 2024-03-24T01:25:09.930 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T01:24:08.643966+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T01:24:09.189904+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T01:24:13.027323+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T01:25:09.930 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:25:10.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.141+0000 7fd36f000640 1 Processor -- start 2024-03-24T01:25:10.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.142+0000 7fd36f000640 1 -- start start 2024-03-24T01:25:10.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.143+0000 7fd36f000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3680af970 0x7fd3680674a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:10.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.143+0000 7fd36f000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd368099b50 0x7fd3680afd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:10.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.143+0000 7fd36f000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3680b0290 con 0x7fd36806b0c0 2024-03-24T01:25:10.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.143+0000 7fd36f000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd36815e3e0 con 0x7fd368099b50 2024-03-24T01:25:10.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.143+0000 7fd36f000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3680cfff0 con 0x7fd3680af970 2024-03-24T01:25:10.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.144+0000 7fd367e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3680af970 0x7fd3680674a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:10.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.144+0000 7fd35fe00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd368099b50 0x7fd3680afd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:10.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.144+0000 7fd367e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3680af970 0x7fd3680674a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.90:39528/0 (socket says 172.21.15.90:39528) 2024-03-24T01:25:10.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.144+0000 7fd367e00640 1 -- 172.21.15.90:0/2703839689 learned_addr learned my addr 172.21.15.90:0/2703839689 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:10.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.145+0000 7fd367e00640 1 -- 172.21.15.90:0/2703839689 >> v2:172.21.15.120:3300/0 conn(0x7fd368099b50 msgr2=0x7fd3680afd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:10.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.145+0000 7fd367e00640 1 --2- 172.21.15.90:0/2703839689 >> v2:172.21.15.120:3300/0 conn(0x7fd368099b50 0x7fd3680afd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:10.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.145+0000 7fd367e00640 1 -- 172.21.15.90:0/2703839689 >> v1:172.21.15.120:6789/0 conn(0x7fd36806b0c0 legacy=0x7fd36806b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:10.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.145+0000 7fd367e00640 1 -- 172.21.15.90:0/2703839689 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3680d02d0 con 0x7fd3680af970 2024-03-24T01:25:10.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.146+0000 7fd367e00640 1 --2- 172.21.15.90:0/2703839689 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3680af970 0x7fd3680674a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd358009870 tx=0x7fd35802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b0d8394fc3f0e89 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:10.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.146+0000 7fd367400640 1 -- 172.21.15.90:0/2703839689 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd358034070 con 0x7fd3680af970 2024-03-24T01:25:10.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.146+0000 7fd367400640 1 -- 172.21.15.90:0/2703839689 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd358030840 con 0x7fd3680af970 2024-03-24T01:25:10.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.146+0000 7fd367400640 1 -- 172.21.15.90:0/2703839689 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd358030c10 con 0x7fd3680af970 2024-03-24T01:25:10.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.147+0000 7fd36f000640 1 -- 172.21.15.90:0/2703839689 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3680af970 msgr2=0x7fd3680674a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:10.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.147+0000 7fd36f000640 1 --2- 172.21.15.90:0/2703839689 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3680af970 0x7fd3680674a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd358009870 tx=0x7fd35802b980 comp rx=0 tx=0).stop 2024-03-24T01:25:10.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.147+0000 7fd36f000640 1 -- 172.21.15.90:0/2703839689 shutdown_connections 2024-03-24T01:25:10.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.147+0000 7fd36f000640 1 --2- 172.21.15.90:0/2703839689 >> v2:172.21.15.120:3300/0 conn(0x7fd368099b50 0x7fd3680afd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:10.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.147+0000 7fd36f000640 1 --2- 172.21.15.90:0/2703839689 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3680af970 0x7fd3680674a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:10.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.147+0000 7fd36f000640 1 -- 172.21.15.90:0/2703839689 >> 172.21.15.90:0/2703839689 conn(0x7fd3680b3c90 msgr2=0x7fd3680b4ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:10.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.147+0000 7fd36f000640 1 -- 172.21.15.90:0/2703839689 shutdown_connections 2024-03-24T01:25:10.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.147+0000 7fd36f000640 1 -- 172.21.15.90:0/2703839689 wait complete. 2024-03-24T01:25:10.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.149+0000 7fd36f000640 1 Processor -- start 2024-03-24T01:25:10.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.149+0000 7fd36f000640 1 -- start start 2024-03-24T01:25:10.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.150+0000 7fd36f000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd36806b0c0 0x7fd36814ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:10.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.150+0000 7fd36f000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd368099b50 0x7fd36815a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:10.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.150+0000 7fd36f000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3680af970 0x7fd368152590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:10.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.150+0000 7fd36f000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3680d0b70 con 0x7fd3680af970 2024-03-24T01:25:10.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.150+0000 7fd36f000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3680d0890 con 0x7fd36806b0c0 2024-03-24T01:25:10.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.150+0000 7fd36f000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd36806bc60 con 0x7fd368099b50 2024-03-24T01:25:10.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.150+0000 7fd36d000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3680af970 0x7fd368152590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:10.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.151+0000 7fd36d000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3680af970 0x7fd368152590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46152/0 (socket says 172.21.15.90:46152) 2024-03-24T01:25:10.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.151+0000 7fd36d000640 1 -- 172.21.15.90:0/78347955 learned_addr learned my addr 172.21.15.90:0/78347955 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:10.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.151+0000 7fd367e00640 1 --2- 172.21.15.90:0/78347955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd36806b0c0 0x7fd36814ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:10.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.151+0000 7fd35fe00640 1 --2- 172.21.15.90:0/78347955 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd368099b50 0x7fd36815a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:10.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.151+0000 7fd36d000640 1 -- 172.21.15.90:0/78347955 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd368099b50 msgr2=0x7fd36815a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:10.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.151+0000 7fd36d000640 1 --2- 172.21.15.90:0/78347955 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd368099b50 0x7fd36815a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:10.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.151+0000 7fd36d000640 1 -- 172.21.15.90:0/78347955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd36806b0c0 msgr2=0x7fd36814ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:10.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.151+0000 7fd36d000640 1 --2- 172.21.15.90:0/78347955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd36806b0c0 0x7fd36814ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:10.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.151+0000 7fd36d000640 1 -- 172.21.15.90:0/78347955 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3681516b0 con 0x7fd3680af970 2024-03-24T01:25:10.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.151+0000 7fd36d000640 1 --2- 172.21.15.90:0/78347955 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3680af970 0x7fd368152590 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd36000c7d0 tx=0x7fd36000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:10.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.152+0000 7fd364c00640 1 -- 172.21.15.90:0/78347955 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd360003b70 con 0x7fd3680af970 2024-03-24T01:25:10.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.152+0000 7fd364c00640 1 -- 172.21.15.90:0/78347955 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd360003d30 con 0x7fd3680af970 2024-03-24T01:25:10.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.152+0000 7fd364c00640 1 -- 172.21.15.90:0/78347955 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36000aa90 con 0x7fd3680af970 2024-03-24T01:25:10.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.152+0000 7fd36f000640 1 -- 172.21.15.90:0/78347955 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3681513d0 con 0x7fd3680af970 2024-03-24T01:25:10.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.152+0000 7fd36f000640 1 -- 172.21.15.90:0/78347955 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd368151d90 con 0x7fd3680af970 2024-03-24T01:25:10.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.153+0000 7fd36f000640 1 -- 172.21.15.90:0/78347955 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd36804f400 con 0x7fd3680af970 2024-03-24T01:25:10.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.154+0000 7fd364c00640 1 -- 172.21.15.90:0/78347955 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd36000ac30 con 0x7fd3680af970 2024-03-24T01:25:10.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.155+0000 7fd364c00640 1 --2- 172.21.15.90:0/78347955 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd34406bbd0 0x7fd34406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:10.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.155+0000 7fd364c00640 1 -- 172.21.15.90:0/78347955 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7fd36008add0 con 0x7fd3680af970 2024-03-24T01:25:10.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.156+0000 7fd367e00640 1 --2- 172.21.15.90:0/78347955 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd34406bbd0 0x7fd34406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:10.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.157+0000 7fd367e00640 1 --2- 172.21.15.90:0/78347955 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd34406bbd0 0x7fd34406e090 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd358036000 tx=0x7fd35803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:10.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.159+0000 7fd364c00640 1 -- 172.21.15.90:0/78347955 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd3600594a0 con 0x7fd3680af970 2024-03-24T01:25:10.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.358+0000 7fd36f000640 1 -- 172.21.15.90:0/78347955 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd36806f300 con 0x7fd3680af970 2024-03-24T01:25:10.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.358+0000 7fd364c00640 1 -- 172.21.15.90:0/78347955 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd36008c020 con 0x7fd3680af970 2024-03-24T01:25:10.359 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:25:10.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.361+0000 7fd36f000640 1 -- 172.21.15.90:0/78347955 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd34406bbd0 msgr2=0x7fd34406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:10.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.361+0000 7fd36f000640 1 --2- 172.21.15.90:0/78347955 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd34406bbd0 0x7fd34406e090 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd358036000 tx=0x7fd35803b040 comp rx=0 tx=0).stop 2024-03-24T01:25:10.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.361+0000 7fd36f000640 1 -- 172.21.15.90:0/78347955 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3680af970 msgr2=0x7fd368152590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:10.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.361+0000 7fd36f000640 1 --2- 172.21.15.90:0/78347955 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3680af970 0x7fd368152590 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd36000c7d0 tx=0x7fd36000cc90 comp rx=0 tx=0).stop 2024-03-24T01:25:10.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.361+0000 7fd36f000640 1 -- 172.21.15.90:0/78347955 shutdown_connections 2024-03-24T01:25:10.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.361+0000 7fd36f000640 1 --2- 172.21.15.90:0/78347955 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd34406bbd0 0x7fd34406e090 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:10.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.361+0000 7fd36f000640 1 --2- 172.21.15.90:0/78347955 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3680af970 0x7fd368152590 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:10.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.361+0000 7fd36f000640 1 --2- 172.21.15.90:0/78347955 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd368099b50 0x7fd36815a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:10.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.361+0000 7fd36f000640 1 --2- 172.21.15.90:0/78347955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd36806b0c0 0x7fd36814ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:10.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.361+0000 7fd36f000640 1 -- 172.21.15.90:0/78347955 >> 172.21.15.90:0/78347955 conn(0x7fd3680b3c90 msgr2=0x7fd36806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:10.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.361+0000 7fd36f000640 1 -- 172.21.15.90:0/78347955 shutdown_connections 2024-03-24T01:25:10.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.361+0000 7fd36f000640 1 -- 172.21.15.90:0/78347955 wait complete. 2024-03-24T01:25:10.380 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T01:25:10.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.592+0000 7fbdf6000640 1 Processor -- start 2024-03-24T01:25:10.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.592+0000 7fbdf6000640 1 -- start start 2024-03-24T01:25:10.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.593+0000 7fbdf6000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fbdf015da30 0x7fbdf015de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:10.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.593+0000 7fbdf6000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbdf00acea0 0x7fbdf00aea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:10.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.593+0000 7fbdf6000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf015e550 con 0x7fbdf006b0c0 2024-03-24T01:25:10.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.593+0000 7fbdf6000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdf00b2c70 con 0x7fbdf015da30 2024-03-24T01:25:10.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.593+0000 7fbdf6000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdf015f160 con 0x7fbdf00acea0 2024-03-24T01:25:10.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.594+0000 7fbdef400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fbdf015da30 0x7fbdf015de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:10.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.594+0000 7fbdefe00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fbdf006b0c0 0x7fbdf0161a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:46652/0 (socket says 172.21.15.90:46652) 2024-03-24T01:25:10.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.594+0000 7fbdefe00640 1 -- 172.21.15.90:0/3207328081 learned_addr learned my addr 172.21.15.90:0/3207328081 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:10.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.594+0000 7fbde7e00640 1 --2- 172.21.15.90:0/3207328081 >> v2:172.21.15.120:3300/0 conn(0x7fbdf00acea0 0x7fbdf00aea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:10.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.594+0000 7fbdef400640 1 -- 172.21.15.90:0/3207328081 >> v2:172.21.15.120:3300/0 conn(0x7fbdf00acea0 msgr2=0x7fbdf00aea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:10.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.594+0000 7fbdef400640 1 --2- 172.21.15.90:0/3207328081 >> v2:172.21.15.120:3300/0 conn(0x7fbdf00acea0 0x7fbdf00aea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:10.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.594+0000 7fbdef400640 1 -- 172.21.15.90:0/3207328081 >> v1:172.21.15.90:6789/0 conn(0x7fbdf006b0c0 legacy=0x7fbdf0161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:10.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.594+0000 7fbdef400640 1 -- 172.21.15.90:0/3207328081 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf015e830 con 0x7fbdf015da30 2024-03-24T01:25:10.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.594+0000 7fbdeea00640 1 -- 172.21.15.90:0/3207328081 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3235364162 0 0) 0x7fbdf015e550 con 0x7fbdf006b0c0 2024-03-24T01:25:10.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.595+0000 7fbdef400640 1 --2- 172.21.15.90:0/3207328081 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdf015da30 0x7fbdf015de10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fbdf006e220 tx=0x7fbde002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3689cf00c8c41488 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:10.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.596+0000 7fbdeea00640 1 -- 172.21.15.90:0/3207328081 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde0035020 con 0x7fbdf015da30 2024-03-24T01:25:10.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.596+0000 7fbdeea00640 1 -- 172.21.15.90:0/3207328081 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde0031830 con 0x7fbdf015da30 2024-03-24T01:25:10.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.596+0000 7fbdeea00640 1 -- 172.21.15.90:0/3207328081 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde0031c00 con 0x7fbdf015da30 2024-03-24T01:25:10.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.596+0000 7fbdf6000640 1 -- 172.21.15.90:0/3207328081 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdf015da30 msgr2=0x7fbdf015de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:10.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.596+0000 7fbdf6000640 1 --2- 172.21.15.90:0/3207328081 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdf015da30 0x7fbdf015de10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fbdf006e220 tx=0x7fbde002b8d0 comp rx=0 tx=0).stop 2024-03-24T01:25:10.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.596+0000 7fbdf6000640 1 -- 172.21.15.90:0/3207328081 shutdown_connections 2024-03-24T01:25:10.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.596+0000 7fbdf6000640 1 --2- 172.21.15.90:0/3207328081 >> v2:172.21.15.120:3300/0 conn(0x7fbdf00acea0 0x7fbdf00aea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:10.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.596+0000 7fbdf6000640 1 --2- 172.21.15.90:0/3207328081 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdf015da30 0x7fbdf015de10 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:10.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.596+0000 7fbdf6000640 1 -- 172.21.15.90:0/3207328081 >> 172.21.15.90:0/3207328081 conn(0x7fbdf00b1770 msgr2=0x7fbdf00b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:10.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.597+0000 7fbdf6000640 1 -- 172.21.15.90:0/3207328081 shutdown_connections 2024-03-24T01:25:10.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.597+0000 7fbdf6000640 1 -- 172.21.15.90:0/3207328081 wait complete. 2024-03-24T01:25:10.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.598+0000 7fbdf6000640 1 Processor -- start 2024-03-24T01:25:10.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.599+0000 7fbdf6000640 1 -- start start 2024-03-24T01:25:10.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.600+0000 7fbdf6000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbdf006b0c0 0x7fbdf014f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:10.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.600+0000 7fbdf6000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdf00acea0 0x7fbdf015c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:10.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.600+0000 7fbdf6000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbdf015da30 0x7fbdf01462c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:10.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.600+0000 7fbdf6000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf00ad300 con 0x7fbdf00acea0 2024-03-24T01:25:10.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.600+0000 7fbdf6000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf015edf0 con 0x7fbdf015da30 2024-03-24T01:25:10.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.600+0000 7fbdf6000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf00aef70 con 0x7fbdf006b0c0 2024-03-24T01:25:10.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.600+0000 7fbde7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdf00acea0 0x7fbdf015c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:10.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.600+0000 7fbde7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdf00acea0 0x7fbdf015c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46172/0 (socket says 172.21.15.90:46172) 2024-03-24T01:25:10.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.600+0000 7fbde7e00640 1 -- 172.21.15.90:0/1492721892 learned_addr learned my addr 172.21.15.90:0/1492721892 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:10.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.600+0000 7fbdef400640 1 --2- 172.21.15.90:0/1492721892 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbdf006b0c0 0x7fbdf014f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:10.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.600+0000 7fbde7e00640 1 -- 172.21.15.90:0/1492721892 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbdf006b0c0 msgr2=0x7fbdf014f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:10.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.600+0000 7fbde7e00640 1 --2- 172.21.15.90:0/1492721892 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbdf006b0c0 0x7fbdf014f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:10.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.600+0000 7fbde7e00640 1 -- 172.21.15.90:0/1492721892 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbdf015da30 msgr2=0x7fbdf01462c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:10.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.600+0000 7fbde7e00640 1 --2- 172.21.15.90:0/1492721892 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbdf015da30 0x7fbdf01462c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:10.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.600+0000 7fbde7e00640 1 -- 172.21.15.90:0/1492721892 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf01501c0 con 0x7fbdf00acea0 2024-03-24T01:25:10.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.601+0000 7fbde7e00640 1 --2- 172.21.15.90:0/1492721892 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdf00acea0 0x7fbdf015c850 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fbdd800d800 tx=0x7fbdd800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:10.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.601+0000 7fbde7400640 1 -- 172.21.15.90:0/1492721892 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd800b6c0 con 0x7fbdf00acea0 2024-03-24T01:25:10.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.601+0000 7fbde7400640 1 -- 172.21.15.90:0/1492721892 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdd8019070 con 0x7fbdf00acea0 2024-03-24T01:25:10.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.601+0000 7fbdf6000640 1 -- 172.21.15.90:0/1492721892 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdf0150460 con 0x7fbdf00acea0 2024-03-24T01:25:10.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.601+0000 7fbde7400640 1 -- 172.21.15.90:0/1492721892 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd800ba80 con 0x7fbdf00acea0 2024-03-24T01:25:10.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.601+0000 7fbdf6000640 1 -- 172.21.15.90:0/1492721892 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdf015cee0 con 0x7fbdf00acea0 2024-03-24T01:25:10.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.602+0000 7fbdf6000640 1 -- 172.21.15.90:0/1492721892 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdf004f400 con 0x7fbdf00acea0 2024-03-24T01:25:10.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.603+0000 7fbde7400640 1 -- 172.21.15.90:0/1492721892 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fbdd800bc20 con 0x7fbdf00acea0 2024-03-24T01:25:10.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.604+0000 7fbde7400640 1 --2- 172.21.15.90:0/1492721892 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbdcc06bbd0 0x7fbdcc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:10.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.604+0000 7fbde7400640 1 -- 172.21.15.90:0/1492721892 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7fbdd808b4b0 con 0x7fbdf00acea0 2024-03-24T01:25:10.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.605+0000 7fbdef400640 1 --2- 172.21.15.90:0/1492721892 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbdcc06bbd0 0x7fbdcc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:10.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.606+0000 7fbdef400640 1 --2- 172.21.15.90:0/1492721892 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbdcc06bbd0 0x7fbdcc06e090 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbde0036000 tx=0x7fbde0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:10.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.609+0000 7fbde7400640 1 -- 172.21.15.90:0/1492721892 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fbdd8059a50 con 0x7fbdf00acea0 2024-03-24T01:25:10.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.805+0000 7fbdf6000640 1 -- 172.21.15.90:0/1492721892 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbdf006e250 con 0x7fbdf00acea0 2024-03-24T01:25:10.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.806+0000 7fbde7400640 1 -- 172.21.15.90:0/1492721892 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbdd8059350 con 0x7fbdf00acea0 2024-03-24T01:25:10.807 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:25:10.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.809+0000 7fbdf6000640 1 -- 172.21.15.90:0/1492721892 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbdcc06bbd0 msgr2=0x7fbdcc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:10.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.809+0000 7fbdf6000640 1 --2- 172.21.15.90:0/1492721892 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbdcc06bbd0 0x7fbdcc06e090 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbde0036000 tx=0x7fbde0008030 comp rx=0 tx=0).stop 2024-03-24T01:25:10.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.809+0000 7fbdf6000640 1 -- 172.21.15.90:0/1492721892 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdf00acea0 msgr2=0x7fbdf015c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:10.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.809+0000 7fbdf6000640 1 --2- 172.21.15.90:0/1492721892 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdf00acea0 0x7fbdf015c850 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fbdd800d800 tx=0x7fbdd800dcc0 comp rx=0 tx=0).stop 2024-03-24T01:25:10.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.809+0000 7fbdf6000640 1 -- 172.21.15.90:0/1492721892 shutdown_connections 2024-03-24T01:25:10.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.809+0000 7fbdf6000640 1 --2- 172.21.15.90:0/1492721892 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbdcc06bbd0 0x7fbdcc06e090 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:10.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.809+0000 7fbdf6000640 1 --2- 172.21.15.90:0/1492721892 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdf00acea0 0x7fbdf015c850 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:10.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.809+0000 7fbdf6000640 1 --2- 172.21.15.90:0/1492721892 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbdf015da30 0x7fbdf01462c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:10.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.809+0000 7fbdf6000640 1 --2- 172.21.15.90:0/1492721892 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbdf006b0c0 0x7fbdf014f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:10.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.809+0000 7fbdf6000640 1 -- 172.21.15.90:0/1492721892 >> 172.21.15.90:0/1492721892 conn(0x7fbdf00b1770 msgr2=0x7fbdf006ef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:10.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.809+0000 7fbdf6000640 1 -- 172.21.15.90:0/1492721892 shutdown_connections 2024-03-24T01:25:10.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:10.809+0000 7fbdf6000640 1 -- 172.21.15.90:0/1492721892 wait complete. 2024-03-24T01:25:10.829 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T01:25:11.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.033+0000 7f023ca00640 1 Processor -- start 2024-03-24T01:25:11.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.033+0000 7f023ca00640 1 -- start start 2024-03-24T01:25:11.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.035+0000 7f023ca00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f02380b2a70 0x7f02380b2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:11.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.035+0000 7f023ca00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f023806b1f0 con 0x7f02380cd700 2024-03-24T01:25:11.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.035+0000 7f023ca00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f023806b4d0 con 0x7f0238070580 2024-03-24T01:25:11.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.035+0000 7f023ca00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f023806b7b0 con 0x7f02380b2a70 2024-03-24T01:25:11.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.035+0000 7f0237200640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f02380cd700 0x7f02380cfa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:46666/0 (socket says 172.21.15.90:46666) 2024-03-24T01:25:11.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.035+0000 7f0237200640 1 -- 172.21.15.90:0/2077632943 learned_addr learned my addr 172.21.15.90:0/2077632943 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:11.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.035+0000 7f0235e00640 1 --2- 172.21.15.90:0/2077632943 >> v2:172.21.15.120:3300/0 conn(0x7f02380b2a70 0x7f02380b2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:11.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.036+0000 7f0235400640 1 -- 172.21.15.90:0/2077632943 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2150975137 0 0) 0x7f023806b1f0 con 0x7f02380cd700 2024-03-24T01:25:11.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.036+0000 7f0235400640 1 -- 172.21.15.90:0/2077632943 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f021c003520 con 0x7f02380cd700 2024-03-24T01:25:11.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.036+0000 7f0235400640 1 -- 172.21.15.90:0/2077632943 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 168311122 0 0) 0x7f021c003520 con 0x7f02380cd700 2024-03-24T01:25:11.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.036+0000 7f0235400640 1 -- 172.21.15.90:0/2077632943 >> v2:172.21.15.120:3300/0 conn(0x7f02380b2a70 msgr2=0x7f02380b2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:11.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.036+0000 7f0235400640 1 --2- 172.21.15.90:0/2077632943 >> v2:172.21.15.120:3300/0 conn(0x7f02380b2a70 0x7f02380b2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:11.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.036+0000 7f0235400640 1 -- 172.21.15.90:0/2077632943 >> v1:172.21.15.120:6789/0 conn(0x7f0238070580 legacy=0x7f0238161930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:11.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.036+0000 7f0235400640 1 -- 172.21.15.90:0/2077632943 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f023806ba90 con 0x7f02380cd700 2024-03-24T01:25:11.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.036+0000 7f0235400640 1 -- 172.21.15.90:0/2077632943 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f02300019f0 con 0x7f02380cd700 2024-03-24T01:25:11.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.036+0000 7f0235400640 1 -- 172.21.15.90:0/2077632943 >> v1:172.21.15.90:6789/0 conn(0x7f02380cd700 legacy=0x7f02380cfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:11.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.036+0000 7f0235400640 1 --2- 172.21.15.90:0/2077632943 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f021c0039e0 0x7f021c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:11.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.036+0000 7f0235400640 1 --2- 172.21.15.90:0/2077632943 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f021c004800 0x7f021c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:11.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.036+0000 7f0235400640 1 --2- 172.21.15.90:0/2077632943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f021c0051c0 0x7f021c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:11.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.036+0000 7f0235400640 1 -- 172.21.15.90:0/2077632943 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f023806b7b0 con 0x7f021c0039e0 2024-03-24T01:25:11.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.036+0000 7f0235400640 1 -- 172.21.15.90:0/2077632943 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021c00c090 con 0x7f021c0051c0 2024-03-24T01:25:11.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.036+0000 7f0235400640 1 -- 172.21.15.90:0/2077632943 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f021c00c290 con 0x7f021c004800 2024-03-24T01:25:11.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.037+0000 7f0236800640 1 --2- 172.21.15.90:0/2077632943 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f021c0039e0 0x7f021c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:11.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.037+0000 7f0235e00640 1 --2- 172.21.15.90:0/2077632943 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f021c004800 0x7f021c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:11.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.037+0000 7f0236800640 1 -- 172.21.15.90:0/2077632943 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f021c004800 msgr2=0x7f021c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:11.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.037+0000 7f0236800640 1 --2- 172.21.15.90:0/2077632943 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f021c004800 0x7f021c004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:11.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.037+0000 7f0237200640 1 --2- 172.21.15.90:0/2077632943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f021c0051c0 0x7f021c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:11.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.037+0000 7f0236800640 1 -- 172.21.15.90:0/2077632943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f021c0051c0 msgr2=0x7f021c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:11.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.037+0000 7f0236800640 1 --2- 172.21.15.90:0/2077632943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f021c0051c0 0x7f021c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:11.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.037+0000 7f0236800640 1 -- 172.21.15.90:0/2077632943 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f021c003520 con 0x7f021c0039e0 2024-03-24T01:25:11.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.037+0000 7f0237200640 1 --2- 172.21.15.90:0/2077632943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f021c0051c0 0x7f021c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:11.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.037+0000 7f0237200640 1 -- 172.21.15.90:0/2077632943 reap_dead start 2024-03-24T01:25:11.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.038+0000 7f0236800640 1 --2- 172.21.15.90:0/2077632943 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f021c0039e0 0x7f021c003e00 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0228001750 tx=0x7f021c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=241478bb67b33f5c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:11.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.038+0000 7f0235400640 1 -- 172.21.15.90:0/2077632943 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0228001370 con 0x7f021c0039e0 2024-03-24T01:25:11.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.039+0000 7f0235400640 1 -- 172.21.15.90:0/2077632943 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02280052d0 con 0x7f021c0039e0 2024-03-24T01:25:11.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.039+0000 7f0235400640 1 -- 172.21.15.90:0/2077632943 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0228003830 con 0x7f021c0039e0 2024-03-24T01:25:11.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.039+0000 7f023ca00640 1 -- 172.21.15.90:0/2077632943 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f021c0039e0 msgr2=0x7f021c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:11.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.039+0000 7f023ca00640 1 --2- 172.21.15.90:0/2077632943 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f021c0039e0 0x7f021c003e00 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0228001750 tx=0x7f021c005830 comp rx=0 tx=0).stop 2024-03-24T01:25:11.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.039+0000 7f023ca00640 1 -- 172.21.15.90:0/2077632943 shutdown_connections 2024-03-24T01:25:11.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.039+0000 7f023ca00640 1 --2- 172.21.15.90:0/2077632943 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f021c0039e0 0x7f021c003e00 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:11.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.039+0000 7f023ca00640 1 -- 172.21.15.90:0/2077632943 >> 172.21.15.90:0/2077632943 conn(0x7f0238099a70 msgr2=0x7f02380b2360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:11.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.039+0000 7f023ca00640 1 -- 172.21.15.90:0/2077632943 shutdown_connections 2024-03-24T01:25:11.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.039+0000 7f023ca00640 1 -- 172.21.15.90:0/2077632943 wait complete. 2024-03-24T01:25:11.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.041+0000 7f023ca00640 1 Processor -- start 2024-03-24T01:25:11.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.041+0000 7f023ca00640 1 -- start start 2024-03-24T01:25:11.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.042+0000 7f023ca00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f021c0039e0 0x7f02381e4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:11.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.042+0000 7f023ca00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02381ddb00 0x7f02380b0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:11.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.042+0000 7f023ca00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02380b07d0 0x7f02381f7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:11.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.042+0000 7f023ca00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02380ccb10 con 0x7f02380b07d0 2024-03-24T01:25:11.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.042+0000 7f023ca00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0238070960 con 0x7f02381ddb00 2024-03-24T01:25:11.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.042+0000 7f023ca00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02380b2ee0 con 0x7f021c0039e0 2024-03-24T01:25:11.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.042+0000 7f0237200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02380b07d0 0x7f02381f7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:11.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.042+0000 7f0237200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02380b07d0 0x7f02381f7200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46188/0 (socket says 172.21.15.90:46188) 2024-03-24T01:25:11.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.042+0000 7f0237200640 1 -- 172.21.15.90:0/2119290551 learned_addr learned my addr 172.21.15.90:0/2119290551 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:11.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.043+0000 7f0236800640 1 --2- 172.21.15.90:0/2119290551 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f021c0039e0 0x7f02381e4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:11.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.043+0000 7f0235e00640 1 --2- 172.21.15.90:0/2119290551 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02381ddb00 0x7f02380b0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:11.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.043+0000 7f0237200640 1 -- 172.21.15.90:0/2119290551 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f021c0039e0 msgr2=0x7f02381e4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:11.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.043+0000 7f0237200640 1 --2- 172.21.15.90:0/2119290551 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f021c0039e0 0x7f02381e4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:11.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.043+0000 7f0237200640 1 -- 172.21.15.90:0/2119290551 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02381ddb00 msgr2=0x7f02380b0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:11.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.043+0000 7f0237200640 1 --2- 172.21.15.90:0/2119290551 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02381ddb00 0x7f02380b0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:11.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.043+0000 7f0237200640 1 -- 172.21.15.90:0/2119290551 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02381e5500 con 0x7f02380b07d0 2024-03-24T01:25:11.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.043+0000 7f0237200640 1 --2- 172.21.15.90:0/2119290551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02380b07d0 0x7f02381f7200 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0230001580 tx=0x7f023000b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:11.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.043+0000 7f022ea00640 1 -- 172.21.15.90:0/2119290551 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0230004d60 con 0x7f02380b07d0 2024-03-24T01:25:11.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.044+0000 7f022ea00640 1 -- 172.21.15.90:0/2119290551 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f023000bb80 con 0x7f02380b07d0 2024-03-24T01:25:11.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.044+0000 7f023ca00640 1 -- 172.21.15.90:0/2119290551 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02381f3ed0 con 0x7f02380b07d0 2024-03-24T01:25:11.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.044+0000 7f022ea00640 1 -- 172.21.15.90:0/2119290551 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0230007070 con 0x7f02380b07d0 2024-03-24T01:25:11.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.044+0000 7f023ca00640 1 -- 172.21.15.90:0/2119290551 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02381e5120 con 0x7f02380b07d0 2024-03-24T01:25:11.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.045+0000 7f023ca00640 1 -- 172.21.15.90:0/2119290551 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f023804f400 con 0x7f02380b07d0 2024-03-24T01:25:11.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.046+0000 7f022ea00640 1 -- 172.21.15.90:0/2119290551 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f023001b020 con 0x7f02380b07d0 2024-03-24T01:25:11.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.047+0000 7f022ea00640 1 --2- 172.21.15.90:0/2119290551 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f021806bbd0 0x7f021806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:11.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.047+0000 7f022ea00640 1 -- 172.21.15.90:0/2119290551 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f023008b080 con 0x7f02380b07d0 2024-03-24T01:25:11.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.047+0000 7f0236800640 1 --2- 172.21.15.90:0/2119290551 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f021806bbd0 0x7f021806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:11.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.048+0000 7f0236800640 1 --2- 172.21.15.90:0/2119290551 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f021806bbd0 0x7f021806e090 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0228033000 tx=0x7f0228009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:11.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.051+0000 7f022ea00640 1 -- 172.21.15.90:0/2119290551 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0230059740 con 0x7f02380b07d0 2024-03-24T01:25:11.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.251+0000 7f023ca00640 1 -- 172.21.15.90:0/2119290551 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f02380cbfd0 con 0x7f02380b07d0 2024-03-24T01:25:11.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.252+0000 7f022ea00640 1 -- 172.21.15.90:0/2119290551 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0230059040 con 0x7f02380b07d0 2024-03-24T01:25:11.252 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-24T01:25:11.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.254+0000 7f023ca00640 1 -- 172.21.15.90:0/2119290551 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f021806bbd0 msgr2=0x7f021806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:11.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.255+0000 7f023ca00640 1 --2- 172.21.15.90:0/2119290551 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f021806bbd0 0x7f021806e090 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0228033000 tx=0x7f0228009030 comp rx=0 tx=0).stop 2024-03-24T01:25:11.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.255+0000 7f023ca00640 1 -- 172.21.15.90:0/2119290551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02380b07d0 msgr2=0x7f02381f7200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:11.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.255+0000 7f023ca00640 1 --2- 172.21.15.90:0/2119290551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02380b07d0 0x7f02381f7200 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0230001580 tx=0x7f023000b4a0 comp rx=0 tx=0).stop 2024-03-24T01:25:11.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.255+0000 7f023ca00640 1 -- 172.21.15.90:0/2119290551 shutdown_connections 2024-03-24T01:25:11.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.255+0000 7f023ca00640 1 --2- 172.21.15.90:0/2119290551 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f021806bbd0 0x7f021806e090 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:11.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.255+0000 7f023ca00640 1 --2- 172.21.15.90:0/2119290551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02380b07d0 0x7f02381f7200 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:11.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.255+0000 7f023ca00640 1 --2- 172.21.15.90:0/2119290551 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02381ddb00 0x7f02380b0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:11.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.255+0000 7f023ca00640 1 --2- 172.21.15.90:0/2119290551 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f021c0039e0 0x7f02381e4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:11.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.255+0000 7f023ca00640 1 -- 172.21.15.90:0/2119290551 >> 172.21.15.90:0/2119290551 conn(0x7f0238099a70 msgr2=0x7f02381e6210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:11.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.255+0000 7f023ca00640 1 -- 172.21.15.90:0/2119290551 shutdown_connections 2024-03-24T01:25:11.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.255+0000 7f023ca00640 1 -- 172.21.15.90:0/2119290551 wait complete. 2024-03-24T01:25:11.274 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T01:25:11.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.480+0000 7f3d36c00640 1 Processor -- start 2024-03-24T01:25:11.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.480+0000 7f3d36c00640 1 -- start start 2024-03-24T01:25:11.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.482+0000 7f3d36c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3d3006b7d0 0x7f3d300cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:11.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.482+0000 7f3d36c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d300cef80 0x7f3d300ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:11.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.482+0000 7f3d36c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d3006bca0 con 0x7f3d30159620 2024-03-24T01:25:11.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.482+0000 7f3d36c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d300b4470 con 0x7f3d3006b7d0 2024-03-24T01:25:11.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.482+0000 7f3d36c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d300b36f0 con 0x7f3d300cef80 2024-03-24T01:25:11.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.482+0000 7f3d27e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3d3006b7d0 0x7f3d300cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:11.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.482+0000 7f3d34c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d300cef80 0x7f3d300ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:11.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.482+0000 7f3d34c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d300cef80 0x7f3d300ae530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.90:39566/0 (socket says 172.21.15.90:39566) 2024-03-24T01:25:11.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.482+0000 7f3d27e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3d3006b7d0 0x7f3d300cea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.90:57104/0 (socket says 172.21.15.90:57104) 2024-03-24T01:25:11.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.483+0000 7f3d34c00640 1 -- 172.21.15.90:0/2234881148 learned_addr learned my addr 172.21.15.90:0/2234881148 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:11.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.483+0000 7f3d34c00640 1 -- 172.21.15.90:0/2234881148 >> v2:172.21.15.120:3300/0 conn(0x7f3d3006b7d0 msgr2=0x7f3d300cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:11.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.483+0000 7f3d34c00640 1 --2- 172.21.15.90:0/2234881148 >> v2:172.21.15.120:3300/0 conn(0x7f3d3006b7d0 0x7f3d300cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:11.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.483+0000 7f3d34c00640 1 -- 172.21.15.90:0/2234881148 >> v1:172.21.15.120:6789/0 conn(0x7f3d30159620 legacy=0x7f3d3006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:11.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.483+0000 7f3d34c00640 1 -- 172.21.15.90:0/2234881148 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d300b3990 con 0x7f3d300cef80 2024-03-24T01:25:11.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.485+0000 7f3d34c00640 1 --2- 172.21.15.90:0/2234881148 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d300cef80 0x7f3d300ae530 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3d28009e50 tx=0x7f3d2802bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91fe96b1fab706c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:11.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.485+0000 7f3d2f400640 1 -- 172.21.15.90:0/2234881148 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2802ca80 con 0x7f3d300cef80 2024-03-24T01:25:11.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.485+0000 7f3d2f400640 1 -- 172.21.15.90:0/2234881148 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d2802cc20 con 0x7f3d300cef80 2024-03-24T01:25:11.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.485+0000 7f3d2f400640 1 -- 172.21.15.90:0/2234881148 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d280343f0 con 0x7f3d300cef80 2024-03-24T01:25:11.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.486+0000 7f3d36c00640 1 -- 172.21.15.90:0/2234881148 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d300cef80 msgr2=0x7f3d300ae530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:11.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.486+0000 7f3d36c00640 1 --2- 172.21.15.90:0/2234881148 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d300cef80 0x7f3d300ae530 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3d28009e50 tx=0x7f3d2802bee0 comp rx=0 tx=0).stop 2024-03-24T01:25:11.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.486+0000 7f3d36c00640 1 -- 172.21.15.90:0/2234881148 shutdown_connections 2024-03-24T01:25:11.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.486+0000 7f3d36c00640 1 --2- 172.21.15.90:0/2234881148 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d300cef80 0x7f3d300ae530 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:11.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.486+0000 7f3d36c00640 1 --2- 172.21.15.90:0/2234881148 >> v2:172.21.15.120:3300/0 conn(0x7f3d3006b7d0 0x7f3d300cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:11.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.486+0000 7f3d36c00640 1 -- 172.21.15.90:0/2234881148 >> 172.21.15.90:0/2234881148 conn(0x7f3d300b1770 msgr2=0x7f3d300b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:11.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.486+0000 7f3d36c00640 1 -- 172.21.15.90:0/2234881148 shutdown_connections 2024-03-24T01:25:11.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.486+0000 7f3d36c00640 1 -- 172.21.15.90:0/2234881148 wait complete. 2024-03-24T01:25:11.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.488+0000 7f3d36c00640 1 Processor -- start 2024-03-24T01:25:11.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.488+0000 7f3d36c00640 1 -- start start 2024-03-24T01:25:11.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.489+0000 7f3d36c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d3006b7d0 0x7f3d3006e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:11.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.489+0000 7f3d36c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d300cef80 0x7f3d300c9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:11.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.489+0000 7f3d36c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d30159620 0x7f3d301460b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:11.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.489+0000 7f3d36c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d30159a00 con 0x7f3d300cef80 2024-03-24T01:25:11.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.489+0000 7f3d36c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d300b3f50 con 0x7f3d3006b7d0 2024-03-24T01:25:11.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.489+0000 7f3d36c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d300cf400 con 0x7f3d30159620 2024-03-24T01:25:11.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.489+0000 7f3d27e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d300cef80 0x7f3d300c9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:11.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.489+0000 7f3d27e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d300cef80 0x7f3d300c9de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46202/0 (socket says 172.21.15.90:46202) 2024-03-24T01:25:11.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.489+0000 7f3d27e00640 1 -- 172.21.15.90:0/3081910991 learned_addr learned my addr 172.21.15.90:0/3081910991 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:11.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.489+0000 7f3d2fe00640 1 --2- 172.21.15.90:0/3081910991 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d3006b7d0 0x7f3d3006e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:11.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.489+0000 7f3d34c00640 1 --2- 172.21.15.90:0/3081910991 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d30159620 0x7f3d301460b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:11.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.489+0000 7f3d27e00640 1 -- 172.21.15.90:0/3081910991 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d30159620 msgr2=0x7f3d301460b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:11.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.489+0000 7f3d27e00640 1 --2- 172.21.15.90:0/3081910991 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d30159620 0x7f3d301460b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:11.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.489+0000 7f3d27e00640 1 -- 172.21.15.90:0/3081910991 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d3006b7d0 msgr2=0x7f3d3006e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:11.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.489+0000 7f3d27e00640 1 --2- 172.21.15.90:0/3081910991 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d3006b7d0 0x7f3d3006e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:11.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.489+0000 7f3d27e00640 1 -- 172.21.15.90:0/3081910991 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d30153f60 con 0x7f3d300cef80 2024-03-24T01:25:11.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.489+0000 7f3d2fe00640 1 --2- 172.21.15.90:0/3081910991 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d3006b7d0 0x7f3d3006e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:11.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.490+0000 7f3d27e00640 1 --2- 172.21.15.90:0/3081910991 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d300cef80 0x7f3d300c9de0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3d1800e840 tx=0x7f3d1800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:11.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.490+0000 7f3d2cc00640 1 -- 172.21.15.90:0/3081910991 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d18017070 con 0x7f3d300cef80 2024-03-24T01:25:11.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.490+0000 7f3d2cc00640 1 -- 172.21.15.90:0/3081910991 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d1800c940 con 0x7f3d300cef80 2024-03-24T01:25:11.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.490+0000 7f3d36c00640 1 -- 172.21.15.90:0/3081910991 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d3006e980 con 0x7f3d300cef80 2024-03-24T01:25:11.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.490+0000 7f3d2cc00640 1 -- 172.21.15.90:0/3081910991 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1800cd10 con 0x7f3d300cef80 2024-03-24T01:25:11.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.490+0000 7f3d36c00640 1 -- 172.21.15.90:0/3081910991 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d3006ef40 con 0x7f3d300cef80 2024-03-24T01:25:11.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.491+0000 7f3d36c00640 1 -- 172.21.15.90:0/3081910991 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d3004f400 con 0x7f3d300cef80 2024-03-24T01:25:11.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.492+0000 7f3d2cc00640 1 -- 172.21.15.90:0/3081910991 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3d1801a070 con 0x7f3d300cef80 2024-03-24T01:25:11.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.493+0000 7f3d2cc00640 1 --2- 172.21.15.90:0/3081910991 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3d0c06bbd0 0x7f3d0c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:11.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.493+0000 7f3d2cc00640 1 -- 172.21.15.90:0/3081910991 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f3d1808ae50 con 0x7f3d300cef80 2024-03-24T01:25:11.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.494+0000 7f3d2fe00640 1 --2- 172.21.15.90:0/3081910991 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3d0c06bbd0 0x7f3d0c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:11.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.495+0000 7f3d2fe00640 1 --2- 172.21.15.90:0/3081910991 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3d0c06bbd0 0x7f3d0c06e090 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3d20001680 tx=0x7f3d20009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:11.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.498+0000 7f3d2cc00640 1 -- 172.21.15.90:0/3081910991 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3d180593f0 con 0x7f3d300cef80 2024-03-24T01:25:11.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.700+0000 7f3d36c00640 1 -- 172.21.15.90:0/3081910991 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f3d300b2e90 con 0x7f3d300cef80 2024-03-24T01:25:11.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.700+0000 7f3d2cc00640 1 -- 172.21.15.90:0/3081910991 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f3d18058cf0 con 0x7f3d300cef80 2024-03-24T01:25:11.701 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:25:11.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.703+0000 7f3d36c00640 1 -- 172.21.15.90:0/3081910991 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3d0c06bbd0 msgr2=0x7f3d0c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:11.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.703+0000 7f3d36c00640 1 --2- 172.21.15.90:0/3081910991 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3d0c06bbd0 0x7f3d0c06e090 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3d20001680 tx=0x7f3d20009040 comp rx=0 tx=0).stop 2024-03-24T01:25:11.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.703+0000 7f3d36c00640 1 -- 172.21.15.90:0/3081910991 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d300cef80 msgr2=0x7f3d300c9de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:11.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.703+0000 7f3d36c00640 1 --2- 172.21.15.90:0/3081910991 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d300cef80 0x7f3d300c9de0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3d1800e840 tx=0x7f3d1800ed00 comp rx=0 tx=0).stop 2024-03-24T01:25:11.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.703+0000 7f3d36c00640 1 -- 172.21.15.90:0/3081910991 shutdown_connections 2024-03-24T01:25:11.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.703+0000 7f3d36c00640 1 --2- 172.21.15.90:0/3081910991 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3d0c06bbd0 0x7f3d0c06e090 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:11.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.703+0000 7f3d36c00640 1 --2- 172.21.15.90:0/3081910991 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d300cef80 0x7f3d300c9de0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:11.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.703+0000 7f3d36c00640 1 --2- 172.21.15.90:0/3081910991 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d30159620 0x7f3d301460b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:11.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.703+0000 7f3d36c00640 1 --2- 172.21.15.90:0/3081910991 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d3006b7d0 0x7f3d3006e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:11.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.703+0000 7f3d36c00640 1 -- 172.21.15.90:0/3081910991 >> 172.21.15.90:0/3081910991 conn(0x7f3d300b1770 msgr2=0x7f3d3015e620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:11.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.703+0000 7f3d36c00640 1 -- 172.21.15.90:0/3081910991 shutdown_connections 2024-03-24T01:25:11.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.703+0000 7f3d36c00640 1 -- 172.21.15.90:0/3081910991 wait complete. 2024-03-24T01:25:11.721 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:25:11.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.930+0000 7f299aa00640 1 Processor -- start 2024-03-24T01:25:11.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.930+0000 7f299aa00640 1 -- start start 2024-03-24T01:25:11.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.931+0000 7f299aa00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f29940cc150 0x7f29940ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:11.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.931+0000 7f299aa00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f299406b7d0 0x7f29940b3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:11.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.931+0000 7f299aa00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29940cd490 con 0x7f29941603e0 2024-03-24T01:25:11.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.931+0000 7f299aa00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f299415f900 con 0x7f29940cc150 2024-03-24T01:25:11.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.931+0000 7f299aa00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f299415fbe0 con 0x7f299406b7d0 2024-03-24T01:25:11.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.931+0000 7f2993e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f29940cc150 0x7f29940ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:11.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.932+0000 7f2993e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f29940cc150 0x7f29940ccb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46214/0 (socket says 172.21.15.90:46214) 2024-03-24T01:25:11.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.932+0000 7f2993e00640 1 -- 172.21.15.90:0/581876632 learned_addr learned my addr 172.21.15.90:0/581876632 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:11.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.932+0000 7f2998a00640 1 --2- 172.21.15.90:0/581876632 >> v2:172.21.15.120:3300/0 conn(0x7f299406b7d0 0x7f29940b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:11.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.932+0000 7f2993e00640 1 -- 172.21.15.90:0/581876632 >> v2:172.21.15.120:3300/0 conn(0x7f299406b7d0 msgr2=0x7f29940b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:11.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.932+0000 7f2993e00640 1 --2- 172.21.15.90:0/581876632 >> v2:172.21.15.120:3300/0 conn(0x7f299406b7d0 0x7f29940b3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:11.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.932+0000 7f2993e00640 1 -- 172.21.15.90:0/581876632 >> v1:172.21.15.120:6789/0 conn(0x7f29941603e0 legacy=0x7f299406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:11.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.932+0000 7f2993e00640 1 -- 172.21.15.90:0/581876632 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f299415fec0 con 0x7f29940cc150 2024-03-24T01:25:11.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.933+0000 7f2993e00640 1 --2- 172.21.15.90:0/581876632 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29940cc150 0x7f29940ccb60 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f299406fc10 tx=0x7f298402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaf8084657f406a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:11.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.933+0000 7f2993400640 1 -- 172.21.15.90:0/581876632 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2984035020 con 0x7f29940cc150 2024-03-24T01:25:11.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.933+0000 7f2993400640 1 -- 172.21.15.90:0/581876632 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2984031870 con 0x7f29940cc150 2024-03-24T01:25:11.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.933+0000 7f2993400640 1 -- 172.21.15.90:0/581876632 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2984031c40 con 0x7f29940cc150 2024-03-24T01:25:11.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.934+0000 7f299aa00640 1 -- 172.21.15.90:0/581876632 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29940cc150 msgr2=0x7f29940ccb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:11.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.934+0000 7f299aa00640 1 --2- 172.21.15.90:0/581876632 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29940cc150 0x7f29940ccb60 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f299406fc10 tx=0x7f298402b910 comp rx=0 tx=0).stop 2024-03-24T01:25:11.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.934+0000 7f299aa00640 1 -- 172.21.15.90:0/581876632 shutdown_connections 2024-03-24T01:25:11.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.934+0000 7f299aa00640 1 --2- 172.21.15.90:0/581876632 >> v2:172.21.15.120:3300/0 conn(0x7f299406b7d0 0x7f29940b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:11.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.934+0000 7f299aa00640 1 --2- 172.21.15.90:0/581876632 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29940cc150 0x7f29940ccb60 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:11.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.934+0000 7f299aa00640 1 -- 172.21.15.90:0/581876632 >> 172.21.15.90:0/581876632 conn(0x7f29940b1770 msgr2=0x7f29940b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:11.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.934+0000 7f299aa00640 1 -- 172.21.15.90:0/581876632 shutdown_connections 2024-03-24T01:25:11.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.934+0000 7f299aa00640 1 -- 172.21.15.90:0/581876632 wait complete. 2024-03-24T01:25:11.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.936+0000 7f299aa00640 1 Processor -- start 2024-03-24T01:25:11.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.936+0000 7f299aa00640 1 -- start start 2024-03-24T01:25:11.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.937+0000 7f299aa00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f299406b7d0 0x7f2994135220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:11.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.937+0000 7f299aa00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29940cc150 0x7f29940adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:11.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.937+0000 7f299aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29941603e0 0x7f29940ae060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:11.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.937+0000 7f299aa00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29941455b0 con 0x7f29941603e0 2024-03-24T01:25:11.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.937+0000 7f299aa00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2994145430 con 0x7f29940cc150 2024-03-24T01:25:11.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.937+0000 7f299aa00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29940b3c90 con 0x7f299406b7d0 2024-03-24T01:25:11.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.937+0000 7f2998a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29941603e0 0x7f29940ae060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:11.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.937+0000 7f2998a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29941603e0 0x7f29940ae060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46226/0 (socket says 172.21.15.90:46226) 2024-03-24T01:25:11.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.937+0000 7f2998a00640 1 -- 172.21.15.90:0/905478478 learned_addr learned my addr 172.21.15.90:0/905478478 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:11.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.938+0000 7f2993e00640 1 --2- 172.21.15.90:0/905478478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f299406b7d0 0x7f2994135220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:11.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.938+0000 7f298b400640 1 --2- 172.21.15.90:0/905478478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29940cc150 0x7f29940adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:11.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.938+0000 7f2998a00640 1 -- 172.21.15.90:0/905478478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f299406b7d0 msgr2=0x7f2994135220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:11.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.938+0000 7f2998a00640 1 --2- 172.21.15.90:0/905478478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f299406b7d0 0x7f2994135220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:11.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.938+0000 7f2998a00640 1 -- 172.21.15.90:0/905478478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29940cc150 msgr2=0x7f29940adb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:11.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.938+0000 7f2998a00640 1 --2- 172.21.15.90:0/905478478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29940cc150 0x7f29940adb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:11.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.938+0000 7f2998a00640 1 -- 172.21.15.90:0/905478478 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f299415de20 con 0x7f29941603e0 2024-03-24T01:25:11.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.938+0000 7f2993e00640 1 --2- 172.21.15.90:0/905478478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f299406b7d0 0x7f2994135220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:11.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.938+0000 7f2998a00640 1 --2- 172.21.15.90:0/905478478 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29941603e0 0x7f29940ae060 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f298c007de0 tx=0x7f298c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:11.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.938+0000 7f2990c00640 1 -- 172.21.15.90:0/905478478 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f298c014620 con 0x7f29941603e0 2024-03-24T01:25:11.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.938+0000 7f2990c00640 1 -- 172.21.15.90:0/905478478 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f298c019070 con 0x7f29941603e0 2024-03-24T01:25:11.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.938+0000 7f299aa00640 1 -- 172.21.15.90:0/905478478 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f299415e100 con 0x7f29941603e0 2024-03-24T01:25:11.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.938+0000 7f2990c00640 1 -- 172.21.15.90:0/905478478 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f298c014a20 con 0x7f29941603e0 2024-03-24T01:25:11.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.938+0000 7f299aa00640 1 -- 172.21.15.90:0/905478478 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f299415e6c0 con 0x7f29941603e0 2024-03-24T01:25:11.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.940+0000 7f299aa00640 1 -- 172.21.15.90:0/905478478 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2994068060 con 0x7f29941603e0 2024-03-24T01:25:11.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.941+0000 7f2990c00640 1 -- 172.21.15.90:0/905478478 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f298c014bc0 con 0x7f29941603e0 2024-03-24T01:25:11.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.941+0000 7f2990c00640 1 --2- 172.21.15.90:0/905478478 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f297006bbd0 0x7f297006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:11.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.942+0000 7f2990c00640 1 -- 172.21.15.90:0/905478478 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f298c08b840 con 0x7f29941603e0 2024-03-24T01:25:11.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.942+0000 7f2993e00640 1 --2- 172.21.15.90:0/905478478 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f297006bbd0 0x7f297006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:11.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.943+0000 7f2993e00640 1 --2- 172.21.15.90:0/905478478 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f297006bbd0 0x7f297006e090 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2984036000 tx=0x7f2984008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:11.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:11.946+0000 7f2990c00640 1 -- 172.21.15.90:0/905478478 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f298c059f10 con 0x7f29941603e0 2024-03-24T01:25:12.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.145+0000 7f299aa00640 1 -- 172.21.15.90:0/905478478 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29940b3220 con 0x7f29941603e0 2024-03-24T01:25:12.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.148+0000 7f2990c00640 1 -- 172.21.15.90:0/905478478 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18925 (secure 0 0 0) 0x7f298c059810 con 0x7f29941603e0 2024-03-24T01:25:12.149 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:25:12.149 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:24:53.236401+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:24:08.643966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T01:24:09.189904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T01:24:13.027323+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:25:12.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.151+0000 7f299aa00640 1 -- 172.21.15.90:0/905478478 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f297006bbd0 msgr2=0x7f297006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:12.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.151+0000 7f299aa00640 1 --2- 172.21.15.90:0/905478478 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f297006bbd0 0x7f297006e090 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2984036000 tx=0x7f2984008030 comp rx=0 tx=0).stop 2024-03-24T01:25:12.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.151+0000 7f299aa00640 1 -- 172.21.15.90:0/905478478 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29941603e0 msgr2=0x7f29940ae060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:12.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.151+0000 7f299aa00640 1 --2- 172.21.15.90:0/905478478 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29941603e0 0x7f29940ae060 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f298c007de0 tx=0x7f298c00e4b0 comp rx=0 tx=0).stop 2024-03-24T01:25:12.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.151+0000 7f299aa00640 1 -- 172.21.15.90:0/905478478 shutdown_connections 2024-03-24T01:25:12.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.151+0000 7f299aa00640 1 --2- 172.21.15.90:0/905478478 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f297006bbd0 0x7f297006e090 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:12.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.151+0000 7f299aa00640 1 --2- 172.21.15.90:0/905478478 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29941603e0 0x7f29940ae060 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:12.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.151+0000 7f299aa00640 1 --2- 172.21.15.90:0/905478478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29940cc150 0x7f29940adb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:12.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.151+0000 7f299aa00640 1 --2- 172.21.15.90:0/905478478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f299406b7d0 0x7f2994135220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:12.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.151+0000 7f299aa00640 1 -- 172.21.15.90:0/905478478 >> 172.21.15.90:0/905478478 conn(0x7f29940b1770 msgr2=0x7f2994143500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:12.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.151+0000 7f299aa00640 1 -- 172.21.15.90:0/905478478 shutdown_connections 2024-03-24T01:25:12.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.151+0000 7f299aa00640 1 -- 172.21.15.90:0/905478478 wait complete. 2024-03-24T01:25:12.170 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T01:24:08.643966+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T01:24:09.189904+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T01:24:13.027323+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T01:25:12.171 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:25:12.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.377+0000 7f37bba00640 1 Processor -- start 2024-03-24T01:25:12.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.377+0000 7f37bba00640 1 -- start start 2024-03-24T01:25:12.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.379+0000 7f37bba00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f37b40aeeb0 0x7f37b40b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:12.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.379+0000 7f37bba00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f37b415bc50 0x7f37b406b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:12.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.379+0000 7f37bba00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37b406dac0 con 0x7f37b406b600 2024-03-24T01:25:12.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.379+0000 7f37bba00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37b406ba70 con 0x7f37b415bc50 2024-03-24T01:25:12.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.379+0000 7f37bba00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37b40b4da0 con 0x7f37b40aeeb0 2024-03-24T01:25:12.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.379+0000 7f37b3e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f37b415bc50 0x7f37b406b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:12.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.379+0000 7f37b3e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f37b415bc50 0x7f37b406b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46236/0 (socket says 172.21.15.90:46236) 2024-03-24T01:25:12.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.379+0000 7f37b3e00640 1 -- 172.21.15.90:0/2693899210 learned_addr learned my addr 172.21.15.90:0/2693899210 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:12.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.380+0000 7f37b9000640 1 --2- 172.21.15.90:0/2693899210 >> v2:172.21.15.120:3300/0 conn(0x7f37b40aeeb0 0x7f37b40b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:12.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.380+0000 7f37b3e00640 1 -- 172.21.15.90:0/2693899210 >> v2:172.21.15.120:3300/0 conn(0x7f37b40aeeb0 msgr2=0x7f37b40b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:12.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.380+0000 7f37b3e00640 1 --2- 172.21.15.90:0/2693899210 >> v2:172.21.15.120:3300/0 conn(0x7f37b40aeeb0 0x7f37b40b44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:12.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.380+0000 7f37b3e00640 1 -- 172.21.15.90:0/2693899210 >> v1:172.21.15.120:6789/0 conn(0x7f37b406b600 legacy=0x7f37b415e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:12.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.380+0000 7f37b3e00640 1 -- 172.21.15.90:0/2693899210 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b406dda0 con 0x7f37b415bc50 2024-03-24T01:25:12.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.381+0000 7f37b3e00640 1 --2- 172.21.15.90:0/2693899210 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37b415bc50 0x7f37b406b0c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f37b40b4f80 tx=0x7f37a002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55756b7ac9f260d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:12.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.381+0000 7f37b3400640 1 -- 172.21.15.90:0/2693899210 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a0038070 con 0x7f37b415bc50 2024-03-24T01:25:12.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.381+0000 7f37b3400640 1 -- 172.21.15.90:0/2693899210 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37a002f050 con 0x7f37b415bc50 2024-03-24T01:25:12.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.381+0000 7f37b3400640 1 -- 172.21.15.90:0/2693899210 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a0031970 con 0x7f37b415bc50 2024-03-24T01:25:12.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.382+0000 7f37bba00640 1 -- 172.21.15.90:0/2693899210 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37b415bc50 msgr2=0x7f37b406b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:12.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.382+0000 7f37bba00640 1 --2- 172.21.15.90:0/2693899210 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37b415bc50 0x7f37b406b0c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f37b40b4f80 tx=0x7f37a002b7f0 comp rx=0 tx=0).stop 2024-03-24T01:25:12.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.382+0000 7f37bba00640 1 -- 172.21.15.90:0/2693899210 shutdown_connections 2024-03-24T01:25:12.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.382+0000 7f37bba00640 1 --2- 172.21.15.90:0/2693899210 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37b415bc50 0x7f37b406b0c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:12.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.382+0000 7f37bba00640 1 --2- 172.21.15.90:0/2693899210 >> v2:172.21.15.120:3300/0 conn(0x7f37b40aeeb0 0x7f37b40b44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:12.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.382+0000 7f37bba00640 1 -- 172.21.15.90:0/2693899210 >> 172.21.15.90:0/2693899210 conn(0x7f37b40b1770 msgr2=0x7f37b40b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:12.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.382+0000 7f37bba00640 1 -- 172.21.15.90:0/2693899210 shutdown_connections 2024-03-24T01:25:12.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.382+0000 7f37bba00640 1 -- 172.21.15.90:0/2693899210 wait complete. 2024-03-24T01:25:12.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.384+0000 7f37bba00640 1 Processor -- start 2024-03-24T01:25:12.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.384+0000 7f37bba00640 1 -- start start 2024-03-24T01:25:12.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.385+0000 7f37bba00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37b406b600 0x7f37b4153c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:12.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.385+0000 7f37bba00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37b40aeeb0 0x7f37b413e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:12.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.385+0000 7f37bba00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37b415bc50 0x7f37b413ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:12.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.385+0000 7f37bba00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b40af290 con 0x7f37b40aeeb0 2024-03-24T01:25:12.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.385+0000 7f37bba00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b406e360 con 0x7f37b415bc50 2024-03-24T01:25:12.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.385+0000 7f37bba00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37b40b4a00 con 0x7f37b406b600 2024-03-24T01:25:12.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.385+0000 7f37b3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37b40aeeb0 0x7f37b413e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:12.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.385+0000 7f37b3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37b40aeeb0 0x7f37b413e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46250/0 (socket says 172.21.15.90:46250) 2024-03-24T01:25:12.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.385+0000 7f37b3e00640 1 -- 172.21.15.90:0/3967381896 learned_addr learned my addr 172.21.15.90:0/3967381896 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:12.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.385+0000 7f37b9a00640 1 --2- 172.21.15.90:0/3967381896 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37b415bc50 0x7f37b413ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:12.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.385+0000 7f37b3e00640 1 -- 172.21.15.90:0/3967381896 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37b406b600 msgr2=0x7f37b4153c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:12.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.385+0000 7f37b9000640 1 --2- 172.21.15.90:0/3967381896 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37b406b600 0x7f37b4153c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:12.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.385+0000 7f37b3e00640 1 --2- 172.21.15.90:0/3967381896 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37b406b600 0x7f37b4153c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:12.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.385+0000 7f37b3e00640 1 -- 172.21.15.90:0/3967381896 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37b415bc50 msgr2=0x7f37b413ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:12.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.385+0000 7f37b3e00640 1 --2- 172.21.15.90:0/3967381896 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37b415bc50 0x7f37b413ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:12.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.385+0000 7f37b3e00640 1 -- 172.21.15.90:0/3967381896 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b4156400 con 0x7f37b40aeeb0 2024-03-24T01:25:12.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.386+0000 7f37b3e00640 1 --2- 172.21.15.90:0/3967381896 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37b40aeeb0 0x7f37b413e910 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f37a002b830 tx=0x7f37a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:12.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.386+0000 7f37b0c00640 1 -- 172.21.15.90:0/3967381896 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a0042070 con 0x7f37b40aeeb0 2024-03-24T01:25:12.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.386+0000 7f37b0c00640 1 -- 172.21.15.90:0/3967381896 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37a003d870 con 0x7f37b40aeeb0 2024-03-24T01:25:12.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.386+0000 7f37b0c00640 1 -- 172.21.15.90:0/3967381896 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a0038040 con 0x7f37b40aeeb0 2024-03-24T01:25:12.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.386+0000 7f37bba00640 1 -- 172.21.15.90:0/3967381896 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37b41566c0 con 0x7f37b40aeeb0 2024-03-24T01:25:12.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.386+0000 7f37bba00640 1 -- 172.21.15.90:0/3967381896 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37b4156c80 con 0x7f37b40aeeb0 2024-03-24T01:25:12.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.387+0000 7f37bba00640 1 -- 172.21.15.90:0/3967381896 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37b404f400 con 0x7f37b40aeeb0 2024-03-24T01:25:12.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.388+0000 7f37b0c00640 1 -- 172.21.15.90:0/3967381896 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f37a0045070 con 0x7f37b40aeeb0 2024-03-24T01:25:12.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.389+0000 7f37b0c00640 1 --2- 172.21.15.90:0/3967381896 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f379406bbd0 0x7f379406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:12.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.389+0000 7f37b0c00640 1 -- 172.21.15.90:0/3967381896 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f37a00acda0 con 0x7f37b40aeeb0 2024-03-24T01:25:12.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.390+0000 7f37b9000640 1 --2- 172.21.15.90:0/3967381896 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f379406bbd0 0x7f379406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:12.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.391+0000 7f37b9000640 1 --2- 172.21.15.90:0/3967381896 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f379406bbd0 0x7f379406e090 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f37a8009f30 tx=0x7f37a800b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:12.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.394+0000 7f37b0c00640 1 -- 172.21.15.90:0/3967381896 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f37a007b4f0 con 0x7f37b40aeeb0 2024-03-24T01:25:12.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.588+0000 7f37bba00640 1 -- 172.21.15.90:0/3967381896 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f37b40b2720 con 0x7f37b40aeeb0 2024-03-24T01:25:12.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.589+0000 7f37b0c00640 1 -- 172.21.15.90:0/3967381896 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f37a007adf0 con 0x7f37b40aeeb0 2024-03-24T01:25:12.590 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:25:12.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.592+0000 7f37bba00640 1 -- 172.21.15.90:0/3967381896 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f379406bbd0 msgr2=0x7f379406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:12.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.592+0000 7f37bba00640 1 --2- 172.21.15.90:0/3967381896 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f379406bbd0 0x7f379406e090 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f37a8009f30 tx=0x7f37a800b040 comp rx=0 tx=0).stop 2024-03-24T01:25:12.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.592+0000 7f37bba00640 1 -- 172.21.15.90:0/3967381896 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37b40aeeb0 msgr2=0x7f37b413e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:12.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.592+0000 7f37bba00640 1 --2- 172.21.15.90:0/3967381896 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37b40aeeb0 0x7f37b413e910 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f37a002b830 tx=0x7f37a00046c0 comp rx=0 tx=0).stop 2024-03-24T01:25:12.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.592+0000 7f37bba00640 1 -- 172.21.15.90:0/3967381896 shutdown_connections 2024-03-24T01:25:12.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.592+0000 7f37bba00640 1 --2- 172.21.15.90:0/3967381896 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f379406bbd0 0x7f379406e090 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:12.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.592+0000 7f37bba00640 1 --2- 172.21.15.90:0/3967381896 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37b40aeeb0 0x7f37b413e910 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:12.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.592+0000 7f37bba00640 1 --2- 172.21.15.90:0/3967381896 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37b415bc50 0x7f37b413ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:12.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.592+0000 7f37bba00640 1 --2- 172.21.15.90:0/3967381896 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f37b406b600 0x7f37b4153c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:12.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.592+0000 7f37bba00640 1 -- 172.21.15.90:0/3967381896 >> 172.21.15.90:0/3967381896 conn(0x7f37b40b1770 msgr2=0x7f37b4159450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:12.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.592+0000 7f37bba00640 1 -- 172.21.15.90:0/3967381896 shutdown_connections 2024-03-24T01:25:12.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.592+0000 7f37bba00640 1 -- 172.21.15.90:0/3967381896 wait complete. 2024-03-24T01:25:12.609 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T01:25:12.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.811+0000 7fe0ac000640 1 Processor -- start 2024-03-24T01:25:12.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.811+0000 7fe0ac000640 1 -- start start 2024-03-24T01:25:12.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.813+0000 7fe0ac000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe0a406b7d0 0x7fe0a40cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:12.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.813+0000 7fe0ac000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe0a40cef80 0x7fe0a40ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:12.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.813+0000 7fe0ac000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a406bca0 con 0x7fe0a4159620 2024-03-24T01:25:12.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.813+0000 7fe0ac000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0a40b4470 con 0x7fe0a40cef80 2024-03-24T01:25:12.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.813+0000 7fe0ac000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0a40b36f0 con 0x7fe0a406b7d0 2024-03-24T01:25:12.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.813+0000 7fe0aa000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe0a40cef80 0x7fe0a40ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:12.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.813+0000 7fe0aa000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe0a40cef80 0x7fe0a40ae530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46256/0 (socket says 172.21.15.90:46256) 2024-03-24T01:25:12.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.813+0000 7fe0aa000640 1 -- 172.21.15.90:0/1422606994 learned_addr learned my addr 172.21.15.90:0/1422606994 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:12.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.813+0000 7fe0a8c00640 1 --2- 172.21.15.90:0/1422606994 >> v2:172.21.15.120:3300/0 conn(0x7fe0a406b7d0 0x7fe0a40cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:12.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.814+0000 7fe0aa000640 1 -- 172.21.15.90:0/1422606994 >> v2:172.21.15.120:3300/0 conn(0x7fe0a406b7d0 msgr2=0x7fe0a40cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:12.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.814+0000 7fe0aa000640 1 --2- 172.21.15.90:0/1422606994 >> v2:172.21.15.120:3300/0 conn(0x7fe0a406b7d0 0x7fe0a40cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:12.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.814+0000 7fe0aa000640 1 -- 172.21.15.90:0/1422606994 >> v1:172.21.15.90:6789/0 conn(0x7fe0a4159620 legacy=0x7fe0a406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:12.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.814+0000 7fe0aa000640 1 -- 172.21.15.90:0/1422606994 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a40b3990 con 0x7fe0a40cef80 2024-03-24T01:25:12.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.815+0000 7fe0aa000640 1 --2- 172.21.15.90:0/1422606994 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0a40cef80 0x7fe0a40ae530 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe0a40cc4b0 tx=0x7fe09c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28ad9866172aec82 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:12.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.815+0000 7fe0a3e00640 1 -- 172.21.15.90:0/1422606994 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09c02e070 con 0x7fe0a40cef80 2024-03-24T01:25:12.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.815+0000 7fe0a3e00640 1 -- 172.21.15.90:0/1422606994 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe09c037070 con 0x7fe0a40cef80 2024-03-24T01:25:12.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.815+0000 7fe0a3e00640 1 -- 172.21.15.90:0/1422606994 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09c031a00 con 0x7fe0a40cef80 2024-03-24T01:25:12.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.816+0000 7fe0ac000640 1 -- 172.21.15.90:0/1422606994 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0a40cef80 msgr2=0x7fe0a40ae530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:12.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.816+0000 7fe0ac000640 1 --2- 172.21.15.90:0/1422606994 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0a40cef80 0x7fe0a40ae530 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe0a40cc4b0 tx=0x7fe09c02b880 comp rx=0 tx=0).stop 2024-03-24T01:25:12.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.816+0000 7fe0ac000640 1 -- 172.21.15.90:0/1422606994 shutdown_connections 2024-03-24T01:25:12.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.816+0000 7fe0ac000640 1 --2- 172.21.15.90:0/1422606994 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0a40cef80 0x7fe0a40ae530 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:12.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.816+0000 7fe0ac000640 1 --2- 172.21.15.90:0/1422606994 >> v2:172.21.15.120:3300/0 conn(0x7fe0a406b7d0 0x7fe0a40cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:12.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.816+0000 7fe0ac000640 1 -- 172.21.15.90:0/1422606994 >> 172.21.15.90:0/1422606994 conn(0x7fe0a40b1770 msgr2=0x7fe0a40b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:12.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.816+0000 7fe0ac000640 1 -- 172.21.15.90:0/1422606994 shutdown_connections 2024-03-24T01:25:12.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.816+0000 7fe0ac000640 1 -- 172.21.15.90:0/1422606994 wait complete. 2024-03-24T01:25:12.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.817+0000 7fe0ac000640 1 Processor -- start 2024-03-24T01:25:12.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.818+0000 7fe0ac000640 1 -- start start 2024-03-24T01:25:12.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.818+0000 7fe0ac000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0a406b7d0 0x7fe0a415bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:12.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.818+0000 7fe0ac000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0a40cef80 0x7fe0a41e5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:12.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.818+0000 7fe0ac000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0a4159620 0x7fe0a41e6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:12.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.818+0000 7fe0ac000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a40cf430 con 0x7fe0a406b7d0 2024-03-24T01:25:12.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.818+0000 7fe0ac000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a40b3f50 con 0x7fe0a40cef80 2024-03-24T01:25:12.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.818+0000 7fe0ac000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a4159a00 con 0x7fe0a4159620 2024-03-24T01:25:12.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.819+0000 7fe0a9600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0a406b7d0 0x7fe0a415bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:12.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.819+0000 7fe0a9600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0a406b7d0 0x7fe0a415bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46272/0 (socket says 172.21.15.90:46272) 2024-03-24T01:25:12.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.819+0000 7fe0a9600640 1 -- 172.21.15.90:0/1078518858 learned_addr learned my addr 172.21.15.90:0/1078518858 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:12.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.819+0000 7fe0a8c00640 1 --2- 172.21.15.90:0/1078518858 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0a40cef80 0x7fe0a41e5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:12.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.819+0000 7fe0a9600640 1 -- 172.21.15.90:0/1078518858 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0a4159620 msgr2=0x7fe0a41e6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:12.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.819+0000 7fe0a9600640 1 --2- 172.21.15.90:0/1078518858 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0a4159620 0x7fe0a41e6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:12.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.819+0000 7fe0a9600640 1 -- 172.21.15.90:0/1078518858 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0a40cef80 msgr2=0x7fe0a41e5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:12.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.819+0000 7fe0a9600640 1 --2- 172.21.15.90:0/1078518858 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0a40cef80 0x7fe0a41e5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:12.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.819+0000 7fe0a9600640 1 -- 172.21.15.90:0/1078518858 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a415a790 con 0x7fe0a406b7d0 2024-03-24T01:25:12.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.819+0000 7fe0a8c00640 1 --2- 172.21.15.90:0/1078518858 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0a40cef80 0x7fe0a41e5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:12.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.819+0000 7fe0a9600640 1 --2- 172.21.15.90:0/1078518858 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0a406b7d0 0x7fe0a415bd80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe09800ba50 tx=0x7fe09800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:12.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.820+0000 7fe0a1600640 1 -- 172.21.15.90:0/1078518858 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe098019070 con 0x7fe0a406b7d0 2024-03-24T01:25:12.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.820+0000 7fe0a1600640 1 -- 172.21.15.90:0/1078518858 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe09800cbc0 con 0x7fe0a406b7d0 2024-03-24T01:25:12.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.820+0000 7fe0a1600640 1 -- 172.21.15.90:0/1078518858 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe098014440 con 0x7fe0a406b7d0 2024-03-24T01:25:12.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.820+0000 7fe0ac000640 1 -- 172.21.15.90:0/1078518858 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0a415aa70 con 0x7fe0a406b7d0 2024-03-24T01:25:12.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.820+0000 7fe0ac000640 1 -- 172.21.15.90:0/1078518858 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0a415b030 con 0x7fe0a406b7d0 2024-03-24T01:25:12.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.821+0000 7fe0ac000640 1 -- 172.21.15.90:0/1078518858 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0a404f400 con 0x7fe0a406b7d0 2024-03-24T01:25:12.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.822+0000 7fe0a1600640 1 -- 172.21.15.90:0/1078518858 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe09801c070 con 0x7fe0a406b7d0 2024-03-24T01:25:12.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.823+0000 7fe0a1600640 1 --2- 172.21.15.90:0/1078518858 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe08406bbd0 0x7fe08406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:12.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.823+0000 7fe0a1600640 1 -- 172.21.15.90:0/1078518858 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7fe09808abc0 con 0x7fe0a406b7d0 2024-03-24T01:25:12.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.823+0000 7fe0a8c00640 1 --2- 172.21.15.90:0/1078518858 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe08406bbd0 0x7fe08406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:12.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.824+0000 7fe0a8c00640 1 --2- 172.21.15.90:0/1078518858 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe08406bbd0 0x7fe08406e090 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe0a40cafc0 tx=0x7fe090008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:12.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:12.827+0000 7fe0a1600640 1 -- 172.21.15.90:0/1078518858 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe098058ba0 con 0x7fe0a406b7d0 2024-03-24T01:25:13.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.013+0000 7fe0ac000640 1 -- 172.21.15.90:0/1078518858 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe0a40b2e90 con 0x7fe0a406b7d0 2024-03-24T01:25:13.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.014+0000 7fe0a1600640 1 -- 172.21.15.90:0/1078518858 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe09805cc40 con 0x7fe0a406b7d0 2024-03-24T01:25:13.015 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:25:13.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.017+0000 7fe0ac000640 1 -- 172.21.15.90:0/1078518858 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe08406bbd0 msgr2=0x7fe08406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:13.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.017+0000 7fe0ac000640 1 --2- 172.21.15.90:0/1078518858 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe08406bbd0 0x7fe08406e090 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe0a40cafc0 tx=0x7fe090008040 comp rx=0 tx=0).stop 2024-03-24T01:25:13.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.017+0000 7fe0ac000640 1 -- 172.21.15.90:0/1078518858 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0a406b7d0 msgr2=0x7fe0a415bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:13.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.017+0000 7fe0ac000640 1 --2- 172.21.15.90:0/1078518858 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0a406b7d0 0x7fe0a415bd80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe09800ba50 tx=0x7fe09800bf10 comp rx=0 tx=0).stop 2024-03-24T01:25:13.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.017+0000 7fe0ac000640 1 -- 172.21.15.90:0/1078518858 shutdown_connections 2024-03-24T01:25:13.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.017+0000 7fe0ac000640 1 --2- 172.21.15.90:0/1078518858 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe08406bbd0 0x7fe08406e090 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:13.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.017+0000 7fe0ac000640 1 --2- 172.21.15.90:0/1078518858 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0a4159620 0x7fe0a41e6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:13.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.017+0000 7fe0ac000640 1 --2- 172.21.15.90:0/1078518858 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0a40cef80 0x7fe0a41e5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:13.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.017+0000 7fe0ac000640 1 --2- 172.21.15.90:0/1078518858 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0a406b7d0 0x7fe0a415bd80 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:13.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.017+0000 7fe0ac000640 1 -- 172.21.15.90:0/1078518858 >> 172.21.15.90:0/1078518858 conn(0x7fe0a40b1770 msgr2=0x7fe0a40cc510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:13.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.017+0000 7fe0ac000640 1 -- 172.21.15.90:0/1078518858 shutdown_connections 2024-03-24T01:25:13.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.017+0000 7fe0ac000640 1 -- 172.21.15.90:0/1078518858 wait complete. 2024-03-24T01:25:13.038 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T01:25:13.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.241+0000 7f1a58200640 1 Processor -- start 2024-03-24T01:25:13.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.241+0000 7f1a58200640 1 -- start start 2024-03-24T01:25:13.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.242+0000 7f1a58200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1a5006b7d0 0x7f1a500cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:13.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.242+0000 7f1a58200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1a500cef80 0x7f1a500ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:13.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.243+0000 7f1a58200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a5006bca0 con 0x7f1a50159620 2024-03-24T01:25:13.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.243+0000 7f1a58200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a500b4470 con 0x7f1a500cef80 2024-03-24T01:25:13.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.243+0000 7f1a58200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a500b36f0 con 0x7f1a5006b7d0 2024-03-24T01:25:13.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.243+0000 7f1a56200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1a500cef80 0x7f1a500ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:13.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.243+0000 7f1a56200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1a500cef80 0x7f1a500ae530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46284/0 (socket says 172.21.15.90:46284) 2024-03-24T01:25:13.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.243+0000 7f1a56200640 1 -- 172.21.15.90:0/3079166006 learned_addr learned my addr 172.21.15.90:0/3079166006 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:13.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.243+0000 7f1a55800640 1 --1- 172.21.15.90:0/3079166006 >> v1:172.21.15.90:6789/0 conn(0x7f1a50159620 0x7f1a5006b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:46680/0 (socket says 172.21.15.90:46680) 2024-03-24T01:25:13.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.243+0000 7f1a54e00640 1 --2- 172.21.15.90:0/3079166006 >> v2:172.21.15.120:3300/0 conn(0x7f1a5006b7d0 0x7f1a500cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:13.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.243+0000 7f1a56200640 1 -- 172.21.15.90:0/3079166006 >> v2:172.21.15.120:3300/0 conn(0x7f1a5006b7d0 msgr2=0x7f1a500cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:13.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.243+0000 7f1a56200640 1 --2- 172.21.15.90:0/3079166006 >> v2:172.21.15.120:3300/0 conn(0x7f1a5006b7d0 0x7f1a500cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:13.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.243+0000 7f1a56200640 1 -- 172.21.15.90:0/3079166006 >> v1:172.21.15.90:6789/0 conn(0x7f1a50159620 legacy=0x7f1a5006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:13.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.244+0000 7f1a56200640 1 -- 172.21.15.90:0/3079166006 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a500b3990 con 0x7f1a500cef80 2024-03-24T01:25:13.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.243+0000 7f1a4fe00640 1 -- 172.21.15.90:0/3079166006 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2928640880 0 0) 0x7f1a5006bca0 con 0x7f1a50159620 2024-03-24T01:25:13.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.245+0000 7f1a56200640 1 --2- 172.21.15.90:0/3079166006 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1a500cef80 0x7f1a500ae530 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1a500cc4b0 tx=0x7f1a4802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b80a8de499b09ae server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.245+0000 7f1a4fe00640 1 -- 172.21.15.90:0/3079166006 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4802ca80 con 0x7f1a500cef80 2024-03-24T01:25:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.245+0000 7f1a4fe00640 1 -- 172.21.15.90:0/3079166006 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a4802cc20 con 0x7f1a500cef80 2024-03-24T01:25:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.245+0000 7f1a4fe00640 1 -- 172.21.15.90:0/3079166006 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a480343f0 con 0x7f1a500cef80 2024-03-24T01:25:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.245+0000 7f1a58200640 1 -- 172.21.15.90:0/3079166006 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1a500cef80 msgr2=0x7f1a500ae530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.245+0000 7f1a58200640 1 --2- 172.21.15.90:0/3079166006 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1a500cef80 0x7f1a500ae530 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1a500cc4b0 tx=0x7f1a4802bee0 comp rx=0 tx=0).stop 2024-03-24T01:25:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.246+0000 7f1a58200640 1 -- 172.21.15.90:0/3079166006 shutdown_connections 2024-03-24T01:25:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.246+0000 7f1a58200640 1 --2- 172.21.15.90:0/3079166006 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1a500cef80 0x7f1a500ae530 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:13.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.246+0000 7f1a58200640 1 --2- 172.21.15.90:0/3079166006 >> v2:172.21.15.120:3300/0 conn(0x7f1a5006b7d0 0x7f1a500cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:13.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.246+0000 7f1a58200640 1 -- 172.21.15.90:0/3079166006 >> 172.21.15.90:0/3079166006 conn(0x7f1a500b1770 msgr2=0x7f1a500b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:13.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.246+0000 7f1a58200640 1 -- 172.21.15.90:0/3079166006 shutdown_connections 2024-03-24T01:25:13.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.246+0000 7f1a58200640 1 -- 172.21.15.90:0/3079166006 wait complete. 2024-03-24T01:25:13.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.247+0000 7f1a58200640 1 Processor -- start 2024-03-24T01:25:13.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.247+0000 7f1a58200640 1 -- start start 2024-03-24T01:25:13.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.248+0000 7f1a58200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a5006b7d0 0x7f1a50143560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:13.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.248+0000 7f1a58200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a500cef80 0x7f1a50156e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:13.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.248+0000 7f1a58200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1a50159620 0x7f1a501573d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:13.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.248+0000 7f1a58200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a500cf430 con 0x7f1a50159620 2024-03-24T01:25:13.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.248+0000 7f1a58200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a500b3f50 con 0x7f1a5006b7d0 2024-03-24T01:25:13.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.248+0000 7f1a58200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a50159a00 con 0x7f1a500cef80 2024-03-24T01:25:13.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.249+0000 7f1a56200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1a50159620 0x7f1a501573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:13.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.249+0000 7f1a56200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1a50159620 0x7f1a501573d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46300/0 (socket says 172.21.15.90:46300) 2024-03-24T01:25:13.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.249+0000 7f1a56200640 1 -- 172.21.15.90:0/3347012863 learned_addr learned my addr 172.21.15.90:0/3347012863 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:13.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.249+0000 7f1a55800640 1 --2- 172.21.15.90:0/3347012863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a5006b7d0 0x7f1a50143560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:13.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.249+0000 7f1a54e00640 1 --2- 172.21.15.90:0/3347012863 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a500cef80 0x7f1a50156e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:13.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.249+0000 7f1a56200640 1 -- 172.21.15.90:0/3347012863 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a500cef80 msgr2=0x7f1a50156e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:13.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.249+0000 7f1a56200640 1 --2- 172.21.15.90:0/3347012863 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a500cef80 0x7f1a50156e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:13.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.249+0000 7f1a56200640 1 -- 172.21.15.90:0/3347012863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a5006b7d0 msgr2=0x7f1a50143560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:13.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.249+0000 7f1a56200640 1 --2- 172.21.15.90:0/3347012863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a5006b7d0 0x7f1a50143560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:13.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.249+0000 7f1a56200640 1 -- 172.21.15.90:0/3347012863 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a501e4300 con 0x7f1a50159620 2024-03-24T01:25:13.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.249+0000 7f1a56200640 1 --2- 172.21.15.90:0/3347012863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1a50159620 0x7f1a501573d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1a48036040 tx=0x7f1a48004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:13.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.250+0000 7f1a4d600640 1 -- 172.21.15.90:0/3347012863 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a48043020 con 0x7f1a50159620 2024-03-24T01:25:13.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.250+0000 7f1a4d600640 1 -- 172.21.15.90:0/3347012863 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a4803e450 con 0x7f1a50159620 2024-03-24T01:25:13.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.250+0000 7f1a58200640 1 -- 172.21.15.90:0/3347012863 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a501f7e10 con 0x7f1a50159620 2024-03-24T01:25:13.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.250+0000 7f1a4d600640 1 -- 172.21.15.90:0/3347012863 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a48009510 con 0x7f1a50159620 2024-03-24T01:25:13.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.250+0000 7f1a58200640 1 -- 172.21.15.90:0/3347012863 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a501e4880 con 0x7f1a50159620 2024-03-24T01:25:13.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.251+0000 7f1a58200640 1 -- 172.21.15.90:0/3347012863 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a5004f400 con 0x7f1a50159620 2024-03-24T01:25:13.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.252+0000 7f1a4d600640 1 -- 172.21.15.90:0/3347012863 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1a48046020 con 0x7f1a50159620 2024-03-24T01:25:13.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.252+0000 7f1a4d600640 1 --2- 172.21.15.90:0/3347012863 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1a3006bbd0 0x7f1a3006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:13.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.253+0000 7f1a4d600640 1 -- 172.21.15.90:0/3347012863 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f1a480ad700 con 0x7f1a50159620 2024-03-24T01:25:13.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.253+0000 7f1a55800640 1 --2- 172.21.15.90:0/3347012863 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1a3006bbd0 0x7f1a3006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:13.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.254+0000 7f1a55800640 1 --2- 172.21.15.90:0/3347012863 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1a3006bbd0 0x7f1a3006e090 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1a44000f60 tx=0x7f1a44007c00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:13.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.257+0000 7f1a4d600640 1 -- 172.21.15.90:0/3347012863 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1a4804d030 con 0x7f1a50159620 2024-03-24T01:25:13.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.466+0000 7f1a58200640 1 -- 172.21.15.90:0/3347012863 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1a50158900 con 0x7f1a50159620 2024-03-24T01:25:13.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.467+0000 7f1a4d600640 1 -- 172.21.15.90:0/3347012863 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1a4807bdc0 con 0x7f1a50159620 2024-03-24T01:25:13.468 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-24T01:25:13.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.470+0000 7f1a58200640 1 -- 172.21.15.90:0/3347012863 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1a3006bbd0 msgr2=0x7f1a3006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:13.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.470+0000 7f1a58200640 1 --2- 172.21.15.90:0/3347012863 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1a3006bbd0 0x7f1a3006e090 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1a44000f60 tx=0x7f1a44007c00 comp rx=0 tx=0).stop 2024-03-24T01:25:13.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.470+0000 7f1a58200640 1 -- 172.21.15.90:0/3347012863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1a50159620 msgr2=0x7f1a501573d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:13.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.470+0000 7f1a58200640 1 --2- 172.21.15.90:0/3347012863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1a50159620 0x7f1a501573d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1a48036040 tx=0x7f1a48004a40 comp rx=0 tx=0).stop 2024-03-24T01:25:13.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.470+0000 7f1a58200640 1 -- 172.21.15.90:0/3347012863 shutdown_connections 2024-03-24T01:25:13.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.470+0000 7f1a58200640 1 --2- 172.21.15.90:0/3347012863 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1a3006bbd0 0x7f1a3006e090 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:13.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.470+0000 7f1a58200640 1 --2- 172.21.15.90:0/3347012863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1a50159620 0x7f1a501573d0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:13.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.470+0000 7f1a58200640 1 --2- 172.21.15.90:0/3347012863 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1a500cef80 0x7f1a50156e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:13.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.470+0000 7f1a58200640 1 --2- 172.21.15.90:0/3347012863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a5006b7d0 0x7f1a50143560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:13.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.470+0000 7f1a58200640 1 -- 172.21.15.90:0/3347012863 >> 172.21.15.90:0/3347012863 conn(0x7f1a500b1770 msgr2=0x7f1a500cba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:13.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.470+0000 7f1a58200640 1 -- 172.21.15.90:0/3347012863 shutdown_connections 2024-03-24T01:25:13.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.471+0000 7f1a58200640 1 -- 172.21.15.90:0/3347012863 wait complete. 2024-03-24T01:25:13.490 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T01:25:13.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.706+0000 7f6cbb600640 1 Processor -- start 2024-03-24T01:25:13.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.706+0000 7f6cbb600640 1 -- start start 2024-03-24T01:25:13.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.707+0000 7f6cbb600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6cb4159620 0x7f6cb406b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:13.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.708+0000 7f6cbb600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6cb40cf150 0x7f6cb40ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:13.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.708+0000 7f6cbb600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb406bca0 con 0x7f6cb406b600 2024-03-24T01:25:13.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.708+0000 7f6cbb600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cb40b4470 con 0x7f6cb4159620 2024-03-24T01:25:13.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.708+0000 7f6cbb600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb40b36f0 con 0x7f6cb40cf150 2024-03-24T01:25:13.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.708+0000 7f6cb8c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6cb4159620 0x7f6cb406b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:13.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.708+0000 7f6cb8c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6cb4159620 0x7f6cb406b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46302/0 (socket says 172.21.15.90:46302) 2024-03-24T01:25:13.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.708+0000 7f6cb8c00640 1 -- 172.21.15.90:0/3066668465 learned_addr learned my addr 172.21.15.90:0/3066668465 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:13.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.708+0000 7f6cb8c00640 1 -- 172.21.15.90:0/3066668465 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6cb40cf150 msgr2=0x7f6cb40ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:13.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.708+0000 7f6cb8c00640 1 --2- 172.21.15.90:0/3066668465 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6cb40cf150 0x7f6cb40ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:13.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.708+0000 7f6cb8c00640 1 -- 172.21.15.90:0/3066668465 >> v1:172.21.15.120:6789/0 conn(0x7f6cb406b600 legacy=0x7f6cb40cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:13.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.708+0000 7f6cb8c00640 1 -- 172.21.15.90:0/3066668465 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb40b3990 con 0x7f6cb4159620 2024-03-24T01:25:13.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.709+0000 7f6cb8c00640 1 --2- 172.21.15.90:0/3066668465 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cb4159620 0x7f6cb406b0c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8009870 tx=0x7f6ca802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5db7365557c4dbf1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:13.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.710+0000 7f6cb3400640 1 -- 172.21.15.90:0/3066668465 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca80317c0 con 0x7f6cb4159620 2024-03-24T01:25:13.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.710+0000 7f6cb3400640 1 -- 172.21.15.90:0/3066668465 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ca8037070 con 0x7f6cb4159620 2024-03-24T01:25:13.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.710+0000 7f6cb3400640 1 -- 172.21.15.90:0/3066668465 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca8031b80 con 0x7f6cb4159620 2024-03-24T01:25:13.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.710+0000 7f6cbb600640 1 -- 172.21.15.90:0/3066668465 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cb4159620 msgr2=0x7f6cb406b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:13.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.710+0000 7f6cbb600640 1 --2- 172.21.15.90:0/3066668465 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cb4159620 0x7f6cb406b0c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8009870 tx=0x7f6ca802b860 comp rx=0 tx=0).stop 2024-03-24T01:25:13.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.710+0000 7f6cbb600640 1 -- 172.21.15.90:0/3066668465 shutdown_connections 2024-03-24T01:25:13.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.710+0000 7f6cbb600640 1 --2- 172.21.15.90:0/3066668465 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6cb40cf150 0x7f6cb40ae530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:13.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.710+0000 7f6cbb600640 1 --2- 172.21.15.90:0/3066668465 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cb4159620 0x7f6cb406b0c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:13.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.710+0000 7f6cbb600640 1 -- 172.21.15.90:0/3066668465 >> 172.21.15.90:0/3066668465 conn(0x7f6cb40b1770 msgr2=0x7f6cb40b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:13.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.710+0000 7f6cbb600640 1 -- 172.21.15.90:0/3066668465 shutdown_connections 2024-03-24T01:25:13.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.710+0000 7f6cbb600640 1 -- 172.21.15.90:0/3066668465 wait complete. 2024-03-24T01:25:13.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.712+0000 7f6cbb600640 1 Processor -- start 2024-03-24T01:25:13.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.712+0000 7f6cbb600640 1 -- start start 2024-03-24T01:25:13.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.713+0000 7f6cbb600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6cb406b600 0x7f6cb41f7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:13.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.713+0000 7f6cbb600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6cb40cf150 0x7f6cb41f4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:13.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.713+0000 7f6cbb600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cb4159620 0x7f6cb41e5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:13.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.713+0000 7f6cbb600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb4159a00 con 0x7f6cb4159620 2024-03-24T01:25:13.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.713+0000 7f6cbb600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb40b3f50 con 0x7f6cb40cf150 2024-03-24T01:25:13.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.713+0000 7f6cbb600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb415bfe0 con 0x7f6cb406b600 2024-03-24T01:25:13.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.713+0000 7f6cb9600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cb4159620 0x7f6cb41e5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:13.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.713+0000 7f6cb9600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cb4159620 0x7f6cb41e5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46318/0 (socket says 172.21.15.90:46318) 2024-03-24T01:25:13.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.713+0000 7f6cb9600640 1 -- 172.21.15.90:0/2344049456 learned_addr learned my addr 172.21.15.90:0/2344049456 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:13.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.714+0000 7f6cb8c00640 1 --2- 172.21.15.90:0/2344049456 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6cb406b600 0x7f6cb41f7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:13.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.714+0000 7f6cb3e00640 1 --2- 172.21.15.90:0/2344049456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6cb40cf150 0x7f6cb41f4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:13.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.714+0000 7f6cb9600640 1 -- 172.21.15.90:0/2344049456 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6cb406b600 msgr2=0x7f6cb41f7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:13.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.714+0000 7f6cb9600640 1 --2- 172.21.15.90:0/2344049456 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6cb406b600 0x7f6cb41f7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:13.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.714+0000 7f6cb9600640 1 -- 172.21.15.90:0/2344049456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6cb40cf150 msgr2=0x7f6cb41f4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:13.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.714+0000 7f6cb9600640 1 --2- 172.21.15.90:0/2344049456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6cb40cf150 0x7f6cb41f4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:13.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.714+0000 7f6cb9600640 1 -- 172.21.15.90:0/2344049456 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb40c8c10 con 0x7f6cb4159620 2024-03-24T01:25:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.714+0000 7f6cb9600640 1 --2- 172.21.15.90:0/2344049456 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cb4159620 0x7f6cb41e5140 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6cac00b970 tx=0x7f6cac00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.714+0000 7f6cb0c00640 1 -- 172.21.15.90:0/2344049456 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cac019070 con 0x7f6cb4159620 2024-03-24T01:25:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.714+0000 7f6cb0c00640 1 -- 172.21.15.90:0/2344049456 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cac00cb60 con 0x7f6cb4159620 2024-03-24T01:25:13.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.714+0000 7f6cbb600640 1 -- 172.21.15.90:0/2344049456 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cb41e5b00 con 0x7f6cb4159620 2024-03-24T01:25:13.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.714+0000 7f6cb0c00640 1 -- 172.21.15.90:0/2344049456 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cac014440 con 0x7f6cb4159620 2024-03-24T01:25:13.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.715+0000 7f6cbb600640 1 -- 172.21.15.90:0/2344049456 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cb40c9190 con 0x7f6cb4159620 2024-03-24T01:25:13.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.716+0000 7f6cbb600640 1 -- 172.21.15.90:0/2344049456 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cb406b3f0 con 0x7f6cb4159620 2024-03-24T01:25:13.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.717+0000 7f6cb0c00640 1 -- 172.21.15.90:0/2344049456 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6cac01c070 con 0x7f6cb4159620 2024-03-24T01:25:13.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.718+0000 7f6cb0c00640 1 --2- 172.21.15.90:0/2344049456 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6c9406bbd0 0x7f6c9406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:13.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.718+0000 7f6cb0c00640 1 -- 172.21.15.90:0/2344049456 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f6cac08ac90 con 0x7f6cb4159620 2024-03-24T01:25:13.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.719+0000 7f6cb8c00640 1 --2- 172.21.15.90:0/2344049456 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6c9406bbd0 0x7f6c9406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:13.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.720+0000 7f6cb8c00640 1 --2- 172.21.15.90:0/2344049456 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6c9406bbd0 0x7f6c9406e090 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8036000 tx=0x7f6ca803d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:13.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.723+0000 7f6cb0c00640 1 -- 172.21.15.90:0/2344049456 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6cb406b3f0 con 0x7f6cb4159620 2024-03-24T01:25:13.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.924+0000 7f6cbb600640 1 -- 172.21.15.90:0/2344049456 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f6cb404f400 con 0x7f6cb4159620 2024-03-24T01:25:13.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.925+0000 7f6cb0c00640 1 -- 172.21.15.90:0/2344049456 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f6cac0592c0 con 0x7f6cb4159620 2024-03-24T01:25:13.926 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:25:13.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.928+0000 7f6cbb600640 1 -- 172.21.15.90:0/2344049456 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6c9406bbd0 msgr2=0x7f6c9406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:13.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.928+0000 7f6cbb600640 1 --2- 172.21.15.90:0/2344049456 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6c9406bbd0 0x7f6c9406e090 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8036000 tx=0x7f6ca803d040 comp rx=0 tx=0).stop 2024-03-24T01:25:13.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.928+0000 7f6cbb600640 1 -- 172.21.15.90:0/2344049456 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cb4159620 msgr2=0x7f6cb41e5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:13.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.928+0000 7f6cbb600640 1 --2- 172.21.15.90:0/2344049456 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cb4159620 0x7f6cb41e5140 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6cac00b970 tx=0x7f6cac00be30 comp rx=0 tx=0).stop 2024-03-24T01:25:13.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.928+0000 7f6cbb600640 1 -- 172.21.15.90:0/2344049456 shutdown_connections 2024-03-24T01:25:13.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.928+0000 7f6cbb600640 1 --2- 172.21.15.90:0/2344049456 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6c9406bbd0 0x7f6c9406e090 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:13.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.928+0000 7f6cbb600640 1 --2- 172.21.15.90:0/2344049456 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cb4159620 0x7f6cb41e5140 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:13.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.928+0000 7f6cbb600640 1 --2- 172.21.15.90:0/2344049456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6cb40cf150 0x7f6cb41f4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:13.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.928+0000 7f6cbb600640 1 --2- 172.21.15.90:0/2344049456 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6cb406b600 0x7f6cb41f7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:13.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.928+0000 7f6cbb600640 1 -- 172.21.15.90:0/2344049456 >> 172.21.15.90:0/2344049456 conn(0x7f6cb40b1770 msgr2=0x7f6cb4159df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:13.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.928+0000 7f6cbb600640 1 -- 172.21.15.90:0/2344049456 shutdown_connections 2024-03-24T01:25:13.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:13.928+0000 7f6cbb600640 1 -- 172.21.15.90:0/2344049456 wait complete. 2024-03-24T01:25:13.980 INFO:tasks.cephfs_test_runner:Starting test: test_inject_to_empty (tasks.cephfs.test_journal_repair.TestJournalRepair) That when some dentries in the journal but nothing is in 2024-03-24T01:25:13.980 INFO:tasks.cephfs_test_runner:test_inject_to_empty (tasks.cephfs.test_journal_repair.TestJournalRepair) 2024-03-24T01:25:13.981 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:25:14.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.194+0000 7f9ee5c00640 1 Processor -- start 2024-03-24T01:25:14.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.195+0000 7f9ee5c00640 1 -- start start 2024-03-24T01:25:14.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.196+0000 7f9ee5c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9ee01603e0 0x7f9ee006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:14.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.196+0000 7f9ee5c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ee00cd490 con 0x7f9ee0145430 2024-03-24T01:25:14.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.196+0000 7f9ee5c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ee015f900 con 0x7f9ee00cc150 2024-03-24T01:25:14.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.196+0000 7f9ee5c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ee00cd270 con 0x7f9ee01603e0 2024-03-24T01:25:14.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.197+0000 7f9edf400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f9ee0145430 0x7f9ee00b3750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:46688/0 (socket says 172.21.15.90:46688) 2024-03-24T01:25:14.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.197+0000 7f9edf400640 1 -- 172.21.15.90:0/1991730226 learned_addr learned my addr 172.21.15.90:0/1991730226 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:14.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.197+0000 7f9ed7e00640 1 --2- 172.21.15.90:0/1991730226 >> v2:172.21.15.90:3300/0 conn(0x7f9ee01603e0 0x7f9ee006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:14.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.197+0000 7f9ed7e00640 1 -- 172.21.15.90:0/1991730226 >> v1:172.21.15.120:6789/0 conn(0x7f9ee00cc150 legacy=0x7f9ee00ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:14.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.197+0000 7f9ed7e00640 1 -- 172.21.15.90:0/1991730226 >> v1:172.21.15.90:6789/0 conn(0x7f9ee0145430 legacy=0x7f9ee00b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:14.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.197+0000 7f9ed7e00640 1 -- 172.21.15.90:0/1991730226 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee015fbe0 con 0x7f9ee01603e0 2024-03-24T01:25:14.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.197+0000 7f9ede000640 1 -- 172.21.15.90:0/1991730226 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3275702789 0 0) 0x7f9ee00cd490 con 0x7f9ee0145430 2024-03-24T01:25:14.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.198+0000 7f9ed7e00640 1 --2- 172.21.15.90:0/1991730226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ee01603e0 0x7f9ee006b0c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9ec80097a0 tx=0x7f9ec802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e0ce9e54b6d73e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:14.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.198+0000 7f9ede000640 1 -- 172.21.15.90:0/1991730226 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec8035020 con 0x7f9ee01603e0 2024-03-24T01:25:14.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.199+0000 7f9ede000640 1 -- 172.21.15.90:0/1991730226 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ec8031830 con 0x7f9ee01603e0 2024-03-24T01:25:14.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.199+0000 7f9ede000640 1 -- 172.21.15.90:0/1991730226 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec8031c00 con 0x7f9ee01603e0 2024-03-24T01:25:14.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.199+0000 7f9ee5c00640 1 -- 172.21.15.90:0/1991730226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ee01603e0 msgr2=0x7f9ee006b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:14.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.199+0000 7f9ee5c00640 1 --2- 172.21.15.90:0/1991730226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ee01603e0 0x7f9ee006b0c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9ec80097a0 tx=0x7f9ec802b8d0 comp rx=0 tx=0).stop 2024-03-24T01:25:14.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.199+0000 7f9ee5c00640 1 -- 172.21.15.90:0/1991730226 shutdown_connections 2024-03-24T01:25:14.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.199+0000 7f9ee5c00640 1 --2- 172.21.15.90:0/1991730226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ee01603e0 0x7f9ee006b0c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:14.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.199+0000 7f9ee5c00640 1 -- 172.21.15.90:0/1991730226 >> 172.21.15.90:0/1991730226 conn(0x7f9ee00b1770 msgr2=0x7f9ee00b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:14.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.199+0000 7f9ee5c00640 1 -- 172.21.15.90:0/1991730226 shutdown_connections 2024-03-24T01:25:14.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.199+0000 7f9ee5c00640 1 -- 172.21.15.90:0/1991730226 wait complete. 2024-03-24T01:25:14.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.201+0000 7f9ee5c00640 1 Processor -- start 2024-03-24T01:25:14.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.201+0000 7f9ee5c00640 1 -- start start 2024-03-24T01:25:14.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.202+0000 7f9ee5c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ee00cc150 0x7f9ee015c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:14.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.202+0000 7f9ee5c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ee0145430 0x7f9ee00b0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:14.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.202+0000 7f9ee5c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ee01603e0 0x7f9ee015e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:14.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.202+0000 7f9ee5c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee006b780 con 0x7f9ee00cc150 2024-03-24T01:25:14.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.202+0000 7f9ee5c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee006b600 con 0x7f9ee0145430 2024-03-24T01:25:14.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.202+0000 7f9ee5c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee006b900 con 0x7f9ee01603e0 2024-03-24T01:25:14.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.202+0000 7f9edea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ee00cc150 0x7f9ee015c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:14.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.202+0000 7f9edea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ee00cc150 0x7f9ee015c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46332/0 (socket says 172.21.15.90:46332) 2024-03-24T01:25:14.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.202+0000 7f9edea00640 1 -- 172.21.15.90:0/3688505327 learned_addr learned my addr 172.21.15.90:0/3688505327 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:14.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.203+0000 7f9edea00640 1 -- 172.21.15.90:0/3688505327 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ee01603e0 msgr2=0x7f9ee015e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:14.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.203+0000 7f9edea00640 1 --2- 172.21.15.90:0/3688505327 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ee01603e0 0x7f9ee015e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:14.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.203+0000 7f9edea00640 1 -- 172.21.15.90:0/3688505327 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ee0145430 msgr2=0x7f9ee00b0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:14.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.203+0000 7f9edea00640 1 --2- 172.21.15.90:0/3688505327 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ee0145430 0x7f9ee00b0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:14.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.203+0000 7f9edea00640 1 -- 172.21.15.90:0/3688505327 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee015c750 con 0x7f9ee00cc150 2024-03-24T01:25:14.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.203+0000 7f9edea00640 1 --2- 172.21.15.90:0/3688505327 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ee00cc150 0x7f9ee015c170 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9ed000c460 tx=0x7f9ed000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:14.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.203+0000 7f9ed6a00640 1 -- 172.21.15.90:0/3688505327 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed0016020 con 0x7f9ee00cc150 2024-03-24T01:25:14.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.203+0000 7f9ed6a00640 1 -- 172.21.15.90:0/3688505327 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ed0012870 con 0x7f9ee00cc150 2024-03-24T01:25:14.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.203+0000 7f9ee5c00640 1 -- 172.21.15.90:0/3688505327 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ee015ca30 con 0x7f9ee00cc150 2024-03-24T01:25:14.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.203+0000 7f9ed6a00640 1 -- 172.21.15.90:0/3688505327 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed0012c60 con 0x7f9ee00cc150 2024-03-24T01:25:14.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.203+0000 7f9ee5c00640 1 -- 172.21.15.90:0/3688505327 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ee015cff0 con 0x7f9ee00cc150 2024-03-24T01:25:14.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.204+0000 7f9ee5c00640 1 -- 172.21.15.90:0/3688505327 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ee0068060 con 0x7f9ee00cc150 2024-03-24T01:25:14.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.205+0000 7f9ed6a00640 1 -- 172.21.15.90:0/3688505327 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9ed0019020 con 0x7f9ee00cc150 2024-03-24T01:25:14.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.206+0000 7f9ed6a00640 1 --2- 172.21.15.90:0/3688505327 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9ebc06bc00 0x7f9ebc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:14.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.206+0000 7f9ed6a00640 1 -- 172.21.15.90:0/3688505327 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f9ed008abd0 con 0x7f9ee00cc150 2024-03-24T01:25:14.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.207+0000 7f9ed7e00640 1 --2- 172.21.15.90:0/3688505327 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9ebc06bc00 0x7f9ebc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:14.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.208+0000 7f9ed7e00640 1 --2- 172.21.15.90:0/3688505327 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9ebc06bc00 0x7f9ebc06e0c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8036000 tx=0x7f9ec8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:14.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.211+0000 7f9ed6a00640 1 -- 172.21.15.90:0/3688505327 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9ed0020030 con 0x7f9ee00cc150 2024-03-24T01:25:14.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.409+0000 7f9ee5c00640 1 -- 172.21.15.90:0/3688505327 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ee00b3220 con 0x7f9ee00cc150 2024-03-24T01:25:14.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.412+0000 7f9ed6a00640 1 -- 172.21.15.90:0/3688505327 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18925 (secure 0 0 0) 0x7f9ed0059200 con 0x7f9ee00cc150 2024-03-24T01:25:14.413 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:25:14.413 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:24:53.236401+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:24:08.643966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T01:24:09.189904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T01:24:13.027323+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:25:14.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.415+0000 7f9ee5c00640 1 -- 172.21.15.90:0/3688505327 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9ebc06bc00 msgr2=0x7f9ebc06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:14.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.415+0000 7f9ee5c00640 1 --2- 172.21.15.90:0/3688505327 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9ebc06bc00 0x7f9ebc06e0c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8036000 tx=0x7f9ec8030000 comp rx=0 tx=0).stop 2024-03-24T01:25:14.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.415+0000 7f9ee5c00640 1 -- 172.21.15.90:0/3688505327 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ee00cc150 msgr2=0x7f9ee015c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:14.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.415+0000 7f9ee5c00640 1 --2- 172.21.15.90:0/3688505327 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ee00cc150 0x7f9ee015c170 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9ed000c460 tx=0x7f9ed000c920 comp rx=0 tx=0).stop 2024-03-24T01:25:14.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.415+0000 7f9ee5c00640 1 -- 172.21.15.90:0/3688505327 shutdown_connections 2024-03-24T01:25:14.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.415+0000 7f9ee5c00640 1 --2- 172.21.15.90:0/3688505327 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9ebc06bc00 0x7f9ebc06e0c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:14.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.415+0000 7f9ee5c00640 1 --2- 172.21.15.90:0/3688505327 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ee01603e0 0x7f9ee015e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:14.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.415+0000 7f9ee5c00640 1 --2- 172.21.15.90:0/3688505327 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ee0145430 0x7f9ee00b0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:14.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.415+0000 7f9ee5c00640 1 --2- 172.21.15.90:0/3688505327 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ee00cc150 0x7f9ee015c170 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:14.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.415+0000 7f9ee5c00640 1 -- 172.21.15.90:0/3688505327 >> 172.21.15.90:0/3688505327 conn(0x7f9ee00b1770 msgr2=0x7f9ee00b47c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:14.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.415+0000 7f9ee5c00640 1 -- 172.21.15.90:0/3688505327 shutdown_connections 2024-03-24T01:25:14.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.415+0000 7f9ee5c00640 1 -- 172.21.15.90:0/3688505327 wait complete. 2024-03-24T01:25:14.433 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T01:24:08.643966+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T01:24:09.189904+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T01:24:13.027323+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T01:25:14.434 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:25:14.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.637+0000 7f63ef800640 1 Processor -- start 2024-03-24T01:25:14.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.637+0000 7f63ef800640 1 -- start start 2024-03-24T01:25:14.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.639+0000 7f63ef800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63e815bb10 0x7f63e8072b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:14.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.639+0000 7f63ef800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63e80b48a0 con 0x7f63e8073090 2024-03-24T01:25:14.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.639+0000 7f63ef800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63e80b3800 con 0x7f63e8162150 2024-03-24T01:25:14.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.639+0000 7f63ef800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e815bef0 con 0x7f63e815bb10 2024-03-24T01:25:14.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.639+0000 7f63e7e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f63e8073090 0x7f63e8161a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:46690/0 (socket says 172.21.15.90:46690) 2024-03-24T01:25:14.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.639+0000 7f63e7e00640 1 -- 172.21.15.90:0/3754832638 learned_addr learned my addr 172.21.15.90:0/3754832638 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:14.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.639+0000 7f63ece00640 1 --2- 172.21.15.90:0/3754832638 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63e815bb10 0x7f63e8072b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:14.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.639+0000 7f63e7400640 1 -- 172.21.15.90:0/3754832638 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2870099926 0 0) 0x7f63e80b48a0 con 0x7f63e8073090 2024-03-24T01:25:14.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.640+0000 7f63e7400640 1 -- 172.21.15.90:0/3754832638 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63d0003520 con 0x7f63e8073090 2024-03-24T01:25:14.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.640+0000 7f63e7400640 1 -- 172.21.15.90:0/3754832638 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4238449252 0 0) 0x7f63d0003520 con 0x7f63e8073090 2024-03-24T01:25:14.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.640+0000 7f63e7400640 1 -- 172.21.15.90:0/3754832638 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63e815bb10 msgr2=0x7f63e8072b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:14.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.640+0000 7f63e7400640 1 --2- 172.21.15.90:0/3754832638 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63e815bb10 0x7f63e8072b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:14.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.640+0000 7f63e7400640 1 -- 172.21.15.90:0/3754832638 >> v1:172.21.15.120:6789/0 conn(0x7f63e8162150 legacy=0x7f63e806b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:14.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.640+0000 7f63e7400640 1 -- 172.21.15.90:0/3754832638 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e80b3ae0 con 0x7f63e8073090 2024-03-24T01:25:14.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.640+0000 7f63e7400640 1 -- 172.21.15.90:0/3754832638 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f63d4001670 con 0x7f63e8073090 2024-03-24T01:25:14.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.640+0000 7f63e7400640 1 -- 172.21.15.90:0/3754832638 >> v1:172.21.15.90:6789/0 conn(0x7f63e8073090 legacy=0x7f63e8161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:14.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.640+0000 7f63e7400640 1 --2- 172.21.15.90:0/3754832638 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63d00039e0 0x7f63d0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:14.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.640+0000 7f63e7400640 1 --2- 172.21.15.90:0/3754832638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63d0004800 0x7f63d0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:14.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.640+0000 7f63e7400640 1 --2- 172.21.15.90:0/3754832638 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63e815bb10 0x7f63d00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:14.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.640+0000 7f63e7400640 1 -- 172.21.15.90:0/3754832638 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e815bef0 con 0x7f63d00039e0 2024-03-24T01:25:14.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.640+0000 7f63e7400640 1 -- 172.21.15.90:0/3754832638 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d000ba70 con 0x7f63d0004800 2024-03-24T01:25:14.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.640+0000 7f63e7400640 1 -- 172.21.15.90:0/3754832638 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63d000bc70 con 0x7f63e815bb10 2024-03-24T01:25:14.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.640+0000 7f63ece00640 1 --2- 172.21.15.90:0/3754832638 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63d00039e0 0x7f63d0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:14.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.641+0000 7f63e7e00640 1 --2- 172.21.15.90:0/3754832638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63d0004800 0x7f63d0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:14.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.641+0000 7f63ed800640 1 --2- 172.21.15.90:0/3754832638 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63e815bb10 0x7f63d00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:14.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.641+0000 7f63ece00640 1 -- 172.21.15.90:0/3754832638 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63e815bb10 msgr2=0x7f63d00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:14.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.641+0000 7f63ece00640 1 --2- 172.21.15.90:0/3754832638 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63e815bb10 0x7f63d00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:14.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.641+0000 7f63ece00640 1 -- 172.21.15.90:0/3754832638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63d0004800 msgr2=0x7f63d0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:14.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.641+0000 7f63ece00640 1 --2- 172.21.15.90:0/3754832638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63d0004800 0x7f63d0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:14.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.641+0000 7f63ece00640 1 -- 172.21.15.90:0/3754832638 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d0003520 con 0x7f63d00039e0 2024-03-24T01:25:14.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.641+0000 7f63ed800640 1 --2- 172.21.15.90:0/3754832638 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63e815bb10 0x7f63d00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:14.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.641+0000 7f63e7e00640 1 --2- 172.21.15.90:0/3754832638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63d0004800 0x7f63d0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:14.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.642+0000 7f63ece00640 1 --2- 172.21.15.90:0/3754832638 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63d00039e0 0x7f63d0003e00 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f63dc009a90 tx=0x7f63d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=431a1f95c3b7ed24 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:14.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.642+0000 7f63e7400640 1 -- 172.21.15.90:0/3754832638 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63dc039070 con 0x7f63d00039e0 2024-03-24T01:25:14.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.642+0000 7f63e7400640 1 -- 172.21.15.90:0/3754832638 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63dc02c660 con 0x7f63d00039e0 2024-03-24T01:25:14.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.642+0000 7f63e7400640 1 -- 172.21.15.90:0/3754832638 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63dc009690 con 0x7f63d00039e0 2024-03-24T01:25:14.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.643+0000 7f63ef800640 1 -- 172.21.15.90:0/3754832638 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63d00039e0 msgr2=0x7f63d0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:14.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.643+0000 7f63ef800640 1 --2- 172.21.15.90:0/3754832638 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63d00039e0 0x7f63d0003e00 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f63dc009a90 tx=0x7f63d0005830 comp rx=0 tx=0).stop 2024-03-24T01:25:14.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.643+0000 7f63ed800640 1 -- 172.21.15.90:0/3754832638 reap_dead start 2024-03-24T01:25:14.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.643+0000 7f63ef800640 1 -- 172.21.15.90:0/3754832638 shutdown_connections 2024-03-24T01:25:14.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.643+0000 7f63ef800640 1 -- 172.21.15.90:0/3754832638 >> 172.21.15.90:0/3754832638 conn(0x7f63e80d00f0 msgr2=0x7f63e80d04d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:14.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.643+0000 7f63ef800640 1 -- 172.21.15.90:0/3754832638 shutdown_connections 2024-03-24T01:25:14.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.643+0000 7f63ef800640 1 -- 172.21.15.90:0/3754832638 wait complete. 2024-03-24T01:25:14.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.645+0000 7f63ef800640 1 Processor -- start 2024-03-24T01:25:14.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.645+0000 7f63ef800640 1 -- start start 2024-03-24T01:25:14.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.646+0000 7f63ef800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63d00039e0 0x7f63e81f7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:14.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.646+0000 7f63ef800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63e80c50f0 0x7f63e8159c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:14.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.646+0000 7f63ef800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63e814fac0 0x7f63e814ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:14.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.646+0000 7f63ef800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e80b3490 con 0x7f63e80c50f0 2024-03-24T01:25:14.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.646+0000 7f63ef800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e80b40a0 con 0x7f63e814fac0 2024-03-24T01:25:14.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.646+0000 7f63ef800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e8072b50 con 0x7f63d00039e0 2024-03-24T01:25:14.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.646+0000 7f63e7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63e80c50f0 0x7f63e8159c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:14.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.646+0000 7f63e7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63e80c50f0 0x7f63e8159c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46342/0 (socket says 172.21.15.90:46342) 2024-03-24T01:25:14.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.646+0000 7f63e7e00640 1 -- 172.21.15.90:0/4155765925 learned_addr learned my addr 172.21.15.90:0/4155765925 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:14.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.647+0000 7f63ece00640 1 --2- 172.21.15.90:0/4155765925 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63d00039e0 0x7f63e81f7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:14.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.647+0000 7f63ed800640 1 --2- 172.21.15.90:0/4155765925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63e814fac0 0x7f63e814ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:14.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.647+0000 7f63e7e00640 1 -- 172.21.15.90:0/4155765925 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63d00039e0 msgr2=0x7f63e81f7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:14.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.647+0000 7f63e7e00640 1 --2- 172.21.15.90:0/4155765925 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63d00039e0 0x7f63e81f7980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:14.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.647+0000 7f63e7e00640 1 -- 172.21.15.90:0/4155765925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63e814fac0 msgr2=0x7f63e814ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:14.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.647+0000 7f63e7e00640 1 --2- 172.21.15.90:0/4155765925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63e814fac0 0x7f63e814ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:14.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.647+0000 7f63e7e00640 1 -- 172.21.15.90:0/4155765925 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e8150480 con 0x7f63e80c50f0 2024-03-24T01:25:14.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.647+0000 7f63e7e00640 1 --2- 172.21.15.90:0/4155765925 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63e80c50f0 0x7f63e8159c80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f63d400d820 tx=0x7f63d400dce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:14.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.647+0000 7f63e4c00640 1 -- 172.21.15.90:0/4155765925 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d4019070 con 0x7f63e80c50f0 2024-03-24T01:25:14.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.647+0000 7f63e4c00640 1 -- 172.21.15.90:0/4155765925 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63d40074e0 con 0x7f63e80c50f0 2024-03-24T01:25:14.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.647+0000 7f63e4c00640 1 -- 172.21.15.90:0/4155765925 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d40049e0 con 0x7f63e80c50f0 2024-03-24T01:25:14.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.647+0000 7f63ef800640 1 -- 172.21.15.90:0/4155765925 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63e80c5570 con 0x7f63e80c50f0 2024-03-24T01:25:14.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.648+0000 7f63ef800640 1 -- 172.21.15.90:0/4155765925 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63e80c5ad0 con 0x7f63e80c50f0 2024-03-24T01:25:14.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.649+0000 7f63ef800640 1 -- 172.21.15.90:0/4155765925 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63e815cfe0 con 0x7f63e80c50f0 2024-03-24T01:25:14.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.650+0000 7f63e4c00640 1 -- 172.21.15.90:0/4155765925 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f63d4004b80 con 0x7f63e80c50f0 2024-03-24T01:25:14.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.650+0000 7f63e4c00640 1 --2- 172.21.15.90:0/4155765925 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f63c806bc00 0x7f63c806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:14.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.651+0000 7f63e4c00640 1 -- 172.21.15.90:0/4155765925 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f63d408a8f0 con 0x7f63e80c50f0 2024-03-24T01:25:14.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.651+0000 7f63ece00640 1 --2- 172.21.15.90:0/4155765925 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f63c806bc00 0x7f63c806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:14.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.652+0000 7f63ece00640 1 --2- 172.21.15.90:0/4155765925 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f63c806bc00 0x7f63c806e0c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f63dc036000 tx=0x7f63dc030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:14.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.655+0000 7f63e4c00640 1 -- 172.21.15.90:0/4155765925 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f63e815cfe0 con 0x7f63e80c50f0 2024-03-24T01:25:14.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.855+0000 7f63ef800640 1 -- 172.21.15.90:0/4155765925 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f63e8066e40 con 0x7f63e80c50f0 2024-03-24T01:25:14.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.856+0000 7f63e4c00640 1 -- 172.21.15.90:0/4155765925 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f63e80c5570 con 0x7f63e80c50f0 2024-03-24T01:25:14.856 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:25:14.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.858+0000 7f63ef800640 1 -- 172.21.15.90:0/4155765925 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f63c806bc00 msgr2=0x7f63c806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:14.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.858+0000 7f63ef800640 1 --2- 172.21.15.90:0/4155765925 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f63c806bc00 0x7f63c806e0c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f63dc036000 tx=0x7f63dc030000 comp rx=0 tx=0).stop 2024-03-24T01:25:14.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.858+0000 7f63ef800640 1 -- 172.21.15.90:0/4155765925 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63e80c50f0 msgr2=0x7f63e8159c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:14.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.858+0000 7f63ef800640 1 --2- 172.21.15.90:0/4155765925 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63e80c50f0 0x7f63e8159c80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f63d400d820 tx=0x7f63d400dce0 comp rx=0 tx=0).stop 2024-03-24T01:25:14.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.858+0000 7f63ef800640 1 -- 172.21.15.90:0/4155765925 shutdown_connections 2024-03-24T01:25:14.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.858+0000 7f63ef800640 1 --2- 172.21.15.90:0/4155765925 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f63c806bc00 0x7f63c806e0c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:14.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.858+0000 7f63ef800640 1 --2- 172.21.15.90:0/4155765925 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63e80c50f0 0x7f63e8159c80 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:14.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.858+0000 7f63ef800640 1 --2- 172.21.15.90:0/4155765925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63e814fac0 0x7f63e814ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:14.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.858+0000 7f63ef800640 1 --2- 172.21.15.90:0/4155765925 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63d00039e0 0x7f63e81f7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:14.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.858+0000 7f63ef800640 1 -- 172.21.15.90:0/4155765925 >> 172.21.15.90:0/4155765925 conn(0x7f63e80d00f0 msgr2=0x7f63e815da80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:14.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.859+0000 7f63ef800640 1 -- 172.21.15.90:0/4155765925 shutdown_connections 2024-03-24T01:25:14.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:14.859+0000 7f63ef800640 1 -- 172.21.15.90:0/4155765925 wait complete. 2024-03-24T01:25:14.879 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T01:25:15.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.094+0000 7f1022800640 1 Processor -- start 2024-03-24T01:25:15.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.094+0000 7f1022800640 1 -- start start 2024-03-24T01:25:15.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.096+0000 7f1022800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f101c1452b0 0x7f101c071990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:15.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.096+0000 7f1022800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f101c0b2890 con 0x7f101c071ed0 2024-03-24T01:25:15.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.096+0000 7f1022800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f101c0b2b70 con 0x7f101c0cf200 2024-03-24T01:25:15.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.096+0000 7f1022800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f101c072330 con 0x7f101c1452b0 2024-03-24T01:25:15.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.096+0000 7f1013e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f101c071ed0 0x7f101c0ceaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:46694/0 (socket says 172.21.15.90:46694) 2024-03-24T01:25:15.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.096+0000 7f1013e00640 1 -- 172.21.15.90:0/2963916463 learned_addr learned my addr 172.21.15.90:0/2963916463 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:15.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.097+0000 7f101b400640 1 --2- 172.21.15.90:0/2963916463 >> v2:172.21.15.120:3300/0 conn(0x7f101c1452b0 0x7f101c071990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:15.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.097+0000 7f101aa00640 1 -- 172.21.15.90:0/2963916463 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 864247672 0 0) 0x7f101c0b2890 con 0x7f101c071ed0 2024-03-24T01:25:15.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.097+0000 7f101aa00640 1 -- 172.21.15.90:0/2963916463 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1000003520 con 0x7f101c071ed0 2024-03-24T01:25:15.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.097+0000 7f101aa00640 1 -- 172.21.15.90:0/2963916463 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1324654076 0 0) 0x7f1000003520 con 0x7f101c071ed0 2024-03-24T01:25:15.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.097+0000 7f101aa00640 1 -- 172.21.15.90:0/2963916463 >> v2:172.21.15.120:3300/0 conn(0x7f101c1452b0 msgr2=0x7f101c071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:15.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.097+0000 7f101aa00640 1 --2- 172.21.15.90:0/2963916463 >> v2:172.21.15.120:3300/0 conn(0x7f101c1452b0 0x7f101c071990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:15.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.097+0000 7f101aa00640 1 -- 172.21.15.90:0/2963916463 >> v1:172.21.15.120:6789/0 conn(0x7f101c0cf200 legacy=0x7f101c0b3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:15.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.097+0000 7f101aa00640 1 -- 172.21.15.90:0/2963916463 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f101c0b2e50 con 0x7f101c071ed0 2024-03-24T01:25:15.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.097+0000 7f101aa00640 1 -- 172.21.15.90:0/2963916463 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f1004001670 con 0x7f101c071ed0 2024-03-24T01:25:15.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.097+0000 7f101aa00640 1 -- 172.21.15.90:0/2963916463 >> v1:172.21.15.90:6789/0 conn(0x7f101c071ed0 legacy=0x7f101c0ceaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:15.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.097+0000 7f101aa00640 1 --2- 172.21.15.90:0/2963916463 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1000003a00 0x7f1000003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:15.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.097+0000 7f101aa00640 1 --2- 172.21.15.90:0/2963916463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1000004800 0x7f1000004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:15.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.098+0000 7f101aa00640 1 --2- 172.21.15.90:0/2963916463 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10000051c0 0x7f100000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:15.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.098+0000 7f101aa00640 1 -- 172.21.15.90:0/2963916463 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100000c090 con 0x7f10000051c0 2024-03-24T01:25:15.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.098+0000 7f101aa00640 1 -- 172.21.15.90:0/2963916463 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100000c240 con 0x7f1000004800 2024-03-24T01:25:15.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.098+0000 7f101aa00640 1 -- 172.21.15.90:0/2963916463 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f100000c440 con 0x7f1000003a00 2024-03-24T01:25:15.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.098+0000 7f101be00640 1 --2- 172.21.15.90:0/2963916463 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10000051c0 0x7f100000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:15.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.098+0000 7f1013e00640 1 --2- 172.21.15.90:0/2963916463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1000004800 0x7f1000004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:15.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.098+0000 7f101b400640 1 --2- 172.21.15.90:0/2963916463 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1000003a00 0x7f1000003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:15.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.098+0000 7f101be00640 1 -- 172.21.15.90:0/2963916463 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1000003a00 msgr2=0x7f1000003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:15.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.098+0000 7f101be00640 1 --2- 172.21.15.90:0/2963916463 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1000003a00 0x7f1000003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:15.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.098+0000 7f101be00640 1 -- 172.21.15.90:0/2963916463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1000004800 msgr2=0x7f1000004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:15.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.098+0000 7f101be00640 1 --2- 172.21.15.90:0/2963916463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1000004800 0x7f1000004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:15.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.099+0000 7f101be00640 1 -- 172.21.15.90:0/2963916463 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1000003520 con 0x7f10000051c0 2024-03-24T01:25:15.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.099+0000 7f1013e00640 1 --2- 172.21.15.90:0/2963916463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1000004800 0x7f1000004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:25:15.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.100+0000 7f101be00640 1 -- 172.21.15.90:0/2963916463 reap_dead start 2024-03-24T01:25:15.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.100+0000 7f101be00640 1 --2- 172.21.15.90:0/2963916463 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10000051c0 0x7f100000b9c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1014001e50 tx=0x7f1000005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=472b618c45762611 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:15.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.100+0000 7f101aa00640 1 -- 172.21.15.90:0/2963916463 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1014001a70 con 0x7f10000051c0 2024-03-24T01:25:15.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.100+0000 7f101aa00640 1 -- 172.21.15.90:0/2963916463 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1014003bb0 con 0x7f10000051c0 2024-03-24T01:25:15.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.100+0000 7f101aa00640 1 -- 172.21.15.90:0/2963916463 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10140050d0 con 0x7f10000051c0 2024-03-24T01:25:15.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.101+0000 7f1022800640 1 -- 172.21.15.90:0/2963916463 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10000051c0 msgr2=0x7f100000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:15.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.101+0000 7f1022800640 1 --2- 172.21.15.90:0/2963916463 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10000051c0 0x7f100000b9c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1014001e50 tx=0x7f1000005830 comp rx=0 tx=0).stop 2024-03-24T01:25:15.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.101+0000 7f1022800640 1 -- 172.21.15.90:0/2963916463 shutdown_connections 2024-03-24T01:25:15.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.101+0000 7f1022800640 1 --2- 172.21.15.90:0/2963916463 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10000051c0 0x7f100000b9c0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:15.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.101+0000 7f1022800640 1 -- 172.21.15.90:0/2963916463 >> 172.21.15.90:0/2963916463 conn(0x7f101c0cf7e0 msgr2=0x7f101c0cfbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:15.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.101+0000 7f1022800640 1 -- 172.21.15.90:0/2963916463 shutdown_connections 2024-03-24T01:25:15.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.101+0000 7f1022800640 1 -- 172.21.15.90:0/2963916463 wait complete. 2024-03-24T01:25:15.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.103+0000 7f1022800640 1 Processor -- start 2024-03-24T01:25:15.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.103+0000 7f1022800640 1 -- start start 2024-03-24T01:25:15.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.104+0000 7f1022800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10000051c0 0x7f101c1f7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:15.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.104+0000 7f1022800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f101c13a820 0x7f101c1398a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:15.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.104+0000 7f1022800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f101c13e380 0x7f101c13eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:15.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.104+0000 7f1022800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f101c0b3410 con 0x7f10000051c0 2024-03-24T01:25:15.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.104+0000 7f1022800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f101c072990 con 0x7f101c13e380 2024-03-24T01:25:15.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.104+0000 7f1022800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f101c145690 con 0x7f101c13a820 2024-03-24T01:25:15.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.104+0000 7f101b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10000051c0 0x7f101c1f7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:15.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.104+0000 7f101b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10000051c0 0x7f101c1f7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46368/0 (socket says 172.21.15.90:46368) 2024-03-24T01:25:15.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.104+0000 7f101b400640 1 -- 172.21.15.90:0/4213055811 learned_addr learned my addr 172.21.15.90:0/4213055811 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:15.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.104+0000 7f101be00640 1 --2- 172.21.15.90:0/4213055811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f101c13e380 0x7f101c13eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:15.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.104+0000 7f1013e00640 1 --2- 172.21.15.90:0/4213055811 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f101c13a820 0x7f101c1398a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:15.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.104+0000 7f101b400640 1 -- 172.21.15.90:0/4213055811 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f101c13a820 msgr2=0x7f101c1398a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:15.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.104+0000 7f101b400640 1 --2- 172.21.15.90:0/4213055811 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f101c13a820 0x7f101c1398a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:15.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.104+0000 7f101b400640 1 -- 172.21.15.90:0/4213055811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f101c13e380 msgr2=0x7f101c13eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:15.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.104+0000 7f101b400640 1 --2- 172.21.15.90:0/4213055811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f101c13e380 0x7f101c13eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:15.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.104+0000 7f101b400640 1 -- 172.21.15.90:0/4213055811 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f101c13d5c0 con 0x7f10000051c0 2024-03-24T01:25:15.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.104+0000 7f101be00640 1 --2- 172.21.15.90:0/4213055811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f101c13e380 0x7f101c13eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:15.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.105+0000 7f101b400640 1 --2- 172.21.15.90:0/4213055811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10000051c0 0x7f101c1f7b80 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f100c011540 tx=0x7f100c011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:15.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.105+0000 7f1013400640 1 -- 172.21.15.90:0/4213055811 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100c015930 con 0x7f10000051c0 2024-03-24T01:25:15.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.105+0000 7f1013400640 1 -- 172.21.15.90:0/4213055811 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f100c015ad0 con 0x7f10000051c0 2024-03-24T01:25:15.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.105+0000 7f1013400640 1 -- 172.21.15.90:0/4213055811 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100c00f460 con 0x7f10000051c0 2024-03-24T01:25:15.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.105+0000 7f1022800640 1 -- 172.21.15.90:0/4213055811 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f101c13d8a0 con 0x7f10000051c0 2024-03-24T01:25:15.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.105+0000 7f1022800640 1 -- 172.21.15.90:0/4213055811 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f101c13de60 con 0x7f10000051c0 2024-03-24T01:25:15.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.106+0000 7f1022800640 1 -- 172.21.15.90:0/4213055811 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f101c04f400 con 0x7f10000051c0 2024-03-24T01:25:15.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.107+0000 7f1013400640 1 -- 172.21.15.90:0/4213055811 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f100c00f600 con 0x7f10000051c0 2024-03-24T01:25:15.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.108+0000 7f1013400640 1 --2- 172.21.15.90:0/4213055811 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0ff806bbd0 0x7f0ff806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:15.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.108+0000 7f1013400640 1 -- 172.21.15.90:0/4213055811 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f100c08b660 con 0x7f10000051c0 2024-03-24T01:25:15.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.109+0000 7f1013e00640 1 --2- 172.21.15.90:0/4213055811 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0ff806bbd0 0x7f0ff806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:15.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.110+0000 7f1013e00640 1 --2- 172.21.15.90:0/4213055811 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0ff806bbd0 0x7f0ff806e090 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1004000ff0 tx=0x7f1004000f40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:15.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.113+0000 7f1013400640 1 -- 172.21.15.90:0/4213055811 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f100c019030 con 0x7f10000051c0 2024-03-24T01:25:15.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.308+0000 7f1022800640 1 -- 172.21.15.90:0/4213055811 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f101c0cbe00 con 0x7f10000051c0 2024-03-24T01:25:15.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.309+0000 7f1013400640 1 -- 172.21.15.90:0/4213055811 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f100c059dc0 con 0x7f10000051c0 2024-03-24T01:25:15.309 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:25:15.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.311+0000 7f1022800640 1 -- 172.21.15.90:0/4213055811 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0ff806bbd0 msgr2=0x7f0ff806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:15.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.311+0000 7f1022800640 1 --2- 172.21.15.90:0/4213055811 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0ff806bbd0 0x7f0ff806e090 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1004000ff0 tx=0x7f1004000f40 comp rx=0 tx=0).stop 2024-03-24T01:25:15.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.311+0000 7f1022800640 1 -- 172.21.15.90:0/4213055811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10000051c0 msgr2=0x7f101c1f7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:15.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.311+0000 7f1022800640 1 --2- 172.21.15.90:0/4213055811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10000051c0 0x7f101c1f7b80 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f100c011540 tx=0x7f100c011a00 comp rx=0 tx=0).stop 2024-03-24T01:25:15.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.312+0000 7f1022800640 1 -- 172.21.15.90:0/4213055811 shutdown_connections 2024-03-24T01:25:15.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.312+0000 7f1022800640 1 --2- 172.21.15.90:0/4213055811 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0ff806bbd0 0x7f0ff806e090 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:15.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.312+0000 7f1022800640 1 --2- 172.21.15.90:0/4213055811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f101c13e380 0x7f101c13eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:15.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.312+0000 7f1022800640 1 --2- 172.21.15.90:0/4213055811 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f101c13a820 0x7f101c1398a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:15.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.312+0000 7f1022800640 1 --2- 172.21.15.90:0/4213055811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10000051c0 0x7f101c1f7b80 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:15.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.312+0000 7f1022800640 1 -- 172.21.15.90:0/4213055811 >> 172.21.15.90:0/4213055811 conn(0x7f101c0cf7e0 msgr2=0x7f101c0b42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:15.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.312+0000 7f1022800640 1 -- 172.21.15.90:0/4213055811 shutdown_connections 2024-03-24T01:25:15.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.312+0000 7f1022800640 1 -- 172.21.15.90:0/4213055811 wait complete. 2024-03-24T01:25:15.330 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T01:25:15.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.534+0000 7fa089e00640 1 Processor -- start 2024-03-24T01:25:15.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.534+0000 7fa089e00640 1 -- start start 2024-03-24T01:25:15.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.536+0000 7fa089e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa08415de10 0x7fa0840acea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:15.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.536+0000 7fa089e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa08406b0c0 0x7fa08406b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:15.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.536+0000 7fa089e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0840af140 con 0x7fa08415da30 2024-03-24T01:25:15.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.536+0000 7fa089e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0840b2c70 con 0x7fa08406b0c0 2024-03-24T01:25:15.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.536+0000 7fa089e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa08415f160 con 0x7fa08415de10 2024-03-24T01:25:15.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.536+0000 7fa083400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa08406b0c0 0x7fa08406b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:15.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.536+0000 7fa082a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa08415da30 0x7fa0840aea30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:46702/0 (socket says 172.21.15.90:46702) 2024-03-24T01:25:15.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.536+0000 7fa082a00640 1 -- 172.21.15.90:0/2330126249 learned_addr learned my addr 172.21.15.90:0/2330126249 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:15.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.536+0000 7fa082000640 1 --2- 172.21.15.90:0/2330126249 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa08415de10 0x7fa0840acea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:15.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.537+0000 7fa083400640 1 -- 172.21.15.90:0/2330126249 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa08415de10 msgr2=0x7fa0840acea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:15.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.537+0000 7fa083400640 1 --2- 172.21.15.90:0/2330126249 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa08415de10 0x7fa0840acea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:15.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.537+0000 7fa083400640 1 -- 172.21.15.90:0/2330126249 >> v1:172.21.15.90:6789/0 conn(0x7fa08415da30 legacy=0x7fa0840aea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:15.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.537+0000 7fa083400640 1 -- 172.21.15.90:0/2330126249 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa08415e550 con 0x7fa08406b0c0 2024-03-24T01:25:15.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.538+0000 7fa083400640 1 --2- 172.21.15.90:0/2330126249 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa08406b0c0 0x7fa08406b550 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa07c009d90 tx=0x7fa07c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b93e08784a69862 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:15.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.538+0000 7fa081600640 1 -- 172.21.15.90:0/2330126249 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa07c039070 con 0x7fa08406b0c0 2024-03-24T01:25:15.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.538+0000 7fa081600640 1 -- 172.21.15.90:0/2330126249 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa07c02cc60 con 0x7fa08406b0c0 2024-03-24T01:25:15.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.538+0000 7fa081600640 1 -- 172.21.15.90:0/2330126249 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa07c034660 con 0x7fa08406b0c0 2024-03-24T01:25:15.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.538+0000 7fa089e00640 1 -- 172.21.15.90:0/2330126249 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa08406b0c0 msgr2=0x7fa08406b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:15.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.538+0000 7fa089e00640 1 --2- 172.21.15.90:0/2330126249 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa08406b0c0 0x7fa08406b550 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa07c009d90 tx=0x7fa07c02beb0 comp rx=0 tx=0).stop 2024-03-24T01:25:15.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.539+0000 7fa089e00640 1 -- 172.21.15.90:0/2330126249 shutdown_connections 2024-03-24T01:25:15.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.539+0000 7fa089e00640 1 --2- 172.21.15.90:0/2330126249 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa08406b0c0 0x7fa08406b550 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:15.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.539+0000 7fa089e00640 1 --2- 172.21.15.90:0/2330126249 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa08415de10 0x7fa0840acea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:15.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.539+0000 7fa089e00640 1 -- 172.21.15.90:0/2330126249 >> 172.21.15.90:0/2330126249 conn(0x7fa0840b1770 msgr2=0x7fa0840b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:15.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.539+0000 7fa089e00640 1 -- 172.21.15.90:0/2330126249 shutdown_connections 2024-03-24T01:25:15.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.539+0000 7fa089e00640 1 -- 172.21.15.90:0/2330126249 wait complete. 2024-03-24T01:25:15.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.540+0000 7fa089e00640 1 Processor -- start 2024-03-24T01:25:15.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.541+0000 7fa089e00640 1 -- start start 2024-03-24T01:25:15.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.541+0000 7fa089e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa08406b0c0 0x7fa084158b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:15.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.541+0000 7fa089e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa08415da30 0x7fa084152bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:15.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.541+0000 7fa089e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa08415de10 0x7fa08415c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:15.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.542+0000 7fa089e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa08415eb10 con 0x7fa08415de10 2024-03-24T01:25:15.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.542+0000 7fa089e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0840ad3e0 con 0x7fa08406b0c0 2024-03-24T01:25:15.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.542+0000 7fa089e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa08415edf0 con 0x7fa08415da30 2024-03-24T01:25:15.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.542+0000 7fa083400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa08415de10 0x7fa08415c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:15.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.542+0000 7fa083400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa08415de10 0x7fa08415c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46396/0 (socket says 172.21.15.90:46396) 2024-03-24T01:25:15.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.542+0000 7fa083400640 1 -- 172.21.15.90:0/1710834492 learned_addr learned my addr 172.21.15.90:0/1710834492 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:15.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.542+0000 7fa082000640 1 --2- 172.21.15.90:0/1710834492 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa08415da30 0x7fa084152bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:15.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.542+0000 7fa082a00640 1 --2- 172.21.15.90:0/1710834492 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa08406b0c0 0x7fa084158b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:15.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.542+0000 7fa083400640 1 -- 172.21.15.90:0/1710834492 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa08415da30 msgr2=0x7fa084152bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:15.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.542+0000 7fa083400640 1 --2- 172.21.15.90:0/1710834492 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa08415da30 0x7fa084152bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:15.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.542+0000 7fa083400640 1 -- 172.21.15.90:0/1710834492 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa08406b0c0 msgr2=0x7fa084158b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:15.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.542+0000 7fa083400640 1 --2- 172.21.15.90:0/1710834492 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa08406b0c0 0x7fa084158b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:15.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.542+0000 7fa083400640 1 -- 172.21.15.90:0/1710834492 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0841f7d90 con 0x7fa08415de10 2024-03-24T01:25:15.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.542+0000 7fa082000640 1 --2- 172.21.15.90:0/1710834492 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa08415da30 0x7fa084152bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:15.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.542+0000 7fa082a00640 1 --2- 172.21.15.90:0/1710834492 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa08406b0c0 0x7fa084158b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:15.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.543+0000 7fa083400640 1 --2- 172.21.15.90:0/1710834492 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa08415de10 0x7fa08415c3a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa07c009d90 tx=0x7fa07c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:15.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.543+0000 7fa07b400640 1 -- 172.21.15.90:0/1710834492 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa07c039050 con 0x7fa08415de10 2024-03-24T01:25:15.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.543+0000 7fa07b400640 1 -- 172.21.15.90:0/1710834492 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa07c045070 con 0x7fa08415de10 2024-03-24T01:25:15.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.543+0000 7fa07b400640 1 -- 172.21.15.90:0/1710834492 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa07c03e830 con 0x7fa08415de10 2024-03-24T01:25:15.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.543+0000 7fa089e00640 1 -- 172.21.15.90:0/1710834492 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0841f8070 con 0x7fa08415de10 2024-03-24T01:25:15.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.543+0000 7fa089e00640 1 -- 172.21.15.90:0/1710834492 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0841f8630 con 0x7fa08415de10 2024-03-24T01:25:15.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.544+0000 7fa089e00640 1 -- 172.21.15.90:0/1710834492 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa08406c2c0 con 0x7fa08415de10 2024-03-24T01:25:15.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.545+0000 7fa07b400640 1 -- 172.21.15.90:0/1710834492 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa07c03e9d0 con 0x7fa08415de10 2024-03-24T01:25:15.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.546+0000 7fa07b400640 1 --2- 172.21.15.90:0/1710834492 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa06406bbd0 0x7fa06406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:15.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.546+0000 7fa07b400640 1 -- 172.21.15.90:0/1710834492 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7fa07c0adb50 con 0x7fa08415de10 2024-03-24T01:25:15.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.546+0000 7fa082a00640 1 --2- 172.21.15.90:0/1710834492 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa06406bbd0 0x7fa06406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:15.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.547+0000 7fa082a00640 1 --2- 172.21.15.90:0/1710834492 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa06406bbd0 0x7fa06406e090 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa074000f60 tx=0x7fa074009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:15.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.550+0000 7fa07b400640 1 -- 172.21.15.90:0/1710834492 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa07c07c3d0 con 0x7fa08415de10 2024-03-24T01:25:15.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.747+0000 7fa089e00640 1 -- 172.21.15.90:0/1710834492 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa08404f400 con 0x7fa08415de10 2024-03-24T01:25:15.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.747+0000 7fa07b400640 1 -- 172.21.15.90:0/1710834492 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa07c0820d0 con 0x7fa08415de10 2024-03-24T01:25:15.748 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-24T01:25:15.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.750+0000 7fa089e00640 1 -- 172.21.15.90:0/1710834492 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa06406bbd0 msgr2=0x7fa06406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:15.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.750+0000 7fa089e00640 1 --2- 172.21.15.90:0/1710834492 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa06406bbd0 0x7fa06406e090 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa074000f60 tx=0x7fa074009040 comp rx=0 tx=0).stop 2024-03-24T01:25:15.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.750+0000 7fa089e00640 1 -- 172.21.15.90:0/1710834492 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa08415de10 msgr2=0x7fa08415c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:15.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.750+0000 7fa089e00640 1 --2- 172.21.15.90:0/1710834492 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa08415de10 0x7fa08415c3a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa07c009d90 tx=0x7fa07c004a40 comp rx=0 tx=0).stop 2024-03-24T01:25:15.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.750+0000 7fa089e00640 1 -- 172.21.15.90:0/1710834492 shutdown_connections 2024-03-24T01:25:15.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.750+0000 7fa089e00640 1 --2- 172.21.15.90:0/1710834492 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa06406bbd0 0x7fa06406e090 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:15.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.750+0000 7fa089e00640 1 --2- 172.21.15.90:0/1710834492 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa08415de10 0x7fa08415c3a0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:15.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.750+0000 7fa089e00640 1 --2- 172.21.15.90:0/1710834492 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa08415da30 0x7fa084152bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:15.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.750+0000 7fa089e00640 1 --2- 172.21.15.90:0/1710834492 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa08406b0c0 0x7fa084158b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:15.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.750+0000 7fa089e00640 1 -- 172.21.15.90:0/1710834492 >> 172.21.15.90:0/1710834492 conn(0x7fa0840b1770 msgr2=0x7fa0841598f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:15.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.750+0000 7fa089e00640 1 -- 172.21.15.90:0/1710834492 shutdown_connections 2024-03-24T01:25:15.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.750+0000 7fa089e00640 1 -- 172.21.15.90:0/1710834492 wait complete. 2024-03-24T01:25:15.769 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T01:25:15.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.983+0000 7f5ca1000640 1 Processor -- start 2024-03-24T01:25:15.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.983+0000 7f5ca1000640 1 -- start start 2024-03-24T01:25:15.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.985+0000 7f5ca1000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c9c15b9f0 0x7f5c9c071990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:15.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.985+0000 7f5ca1000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c9c15f2e0 con 0x7f5c9c071ed0 2024-03-24T01:25:15.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.985+0000 7f5ca1000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c9c06f070 con 0x7f5c9c06eba0 2024-03-24T01:25:15.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.985+0000 7f5ca1000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9c072b00 con 0x7f5c9c15b9f0 2024-03-24T01:25:15.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.985+0000 7f5c93e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f5c9c071ed0 0x7f5c9c06e490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:46712/0 (socket says 172.21.15.90:46712) 2024-03-24T01:25:15.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.985+0000 7f5c93e00640 1 -- 172.21.15.90:0/2695917027 learned_addr learned my addr 172.21.15.90:0/2695917027 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:15.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.985+0000 7f5c99e00640 1 --2- 172.21.15.90:0/2695917027 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c9c15b9f0 0x7f5c9c071990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:15.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.985+0000 7f5c99400640 1 -- 172.21.15.90:0/2695917027 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2339043636 0 0) 0x7f5c9c15f2e0 con 0x7f5c9c071ed0 2024-03-24T01:25:15.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.986+0000 7f5c99400640 1 -- 172.21.15.90:0/2695917027 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c80003520 con 0x7f5c9c071ed0 2024-03-24T01:25:15.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.986+0000 7f5c99400640 1 -- 172.21.15.90:0/2695917027 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1180872859 0 0) 0x7f5c80003520 con 0x7f5c9c071ed0 2024-03-24T01:25:15.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.986+0000 7f5c99400640 1 -- 172.21.15.90:0/2695917027 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c9c15b9f0 msgr2=0x7f5c9c071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:15.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.986+0000 7f5c99400640 1 --2- 172.21.15.90:0/2695917027 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c9c15b9f0 0x7f5c9c071990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:15.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.986+0000 7f5c99400640 1 -- 172.21.15.90:0/2695917027 >> v1:172.21.15.120:6789/0 conn(0x7f5c9c06eba0 legacy=0x7f5c9c0acea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:15.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.986+0000 7f5c99400640 1 -- 172.21.15.90:0/2695917027 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c9c15e520 con 0x7f5c9c071ed0 2024-03-24T01:25:15.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.986+0000 7f5c99400640 1 -- 172.21.15.90:0/2695917027 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f5c84001670 con 0x7f5c9c071ed0 2024-03-24T01:25:15.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.986+0000 7f5c99400640 1 -- 172.21.15.90:0/2695917027 >> v1:172.21.15.90:6789/0 conn(0x7f5c9c071ed0 legacy=0x7f5c9c06e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:15.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.986+0000 7f5c99400640 1 --2- 172.21.15.90:0/2695917027 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c800039e0 0x7f5c80003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:15.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.986+0000 7f5c99400640 1 --2- 172.21.15.90:0/2695917027 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c9c15b9f0 0x7f5c80004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:15.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.986+0000 7f5c99400640 1 --2- 172.21.15.90:0/2695917027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c80004d40 0x7f5c800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:15.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.986+0000 7f5c99400640 1 -- 172.21.15.90:0/2695917027 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8000ba70 con 0x7f5c800039e0 2024-03-24T01:25:15.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.986+0000 7f5c99400640 1 -- 172.21.15.90:0/2695917027 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8000bbf0 con 0x7f5c80004d40 2024-03-24T01:25:15.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.986+0000 7f5c99400640 1 -- 172.21.15.90:0/2695917027 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8000bdf0 con 0x7f5c9c15b9f0 2024-03-24T01:25:15.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.986+0000 7f5c99e00640 1 --2- 172.21.15.90:0/2695917027 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c800039e0 0x7f5c80003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:15.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.987+0000 7f5c99e00640 1 -- 172.21.15.90:0/2695917027 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c9c15b9f0 msgr2=0x7f5c80004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:15.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.987+0000 7f5c99e00640 1 --2- 172.21.15.90:0/2695917027 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c9c15b9f0 0x7f5c80004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:15.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.987+0000 7f5c99e00640 1 -- 172.21.15.90:0/2695917027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c80004d40 msgr2=0x7f5c800051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:15.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.987+0000 7f5c99e00640 1 --2- 172.21.15.90:0/2695917027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c80004d40 0x7f5c800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:15.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.987+0000 7f5c99e00640 1 -- 172.21.15.90:0/2695917027 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c80003520 con 0x7f5c800039e0 2024-03-24T01:25:15.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.988+0000 7f5c99e00640 1 --2- 172.21.15.90:0/2695917027 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c800039e0 0x7f5c80003e00 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c009a90 tx=0x7f5c80005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c4ffbe67d081f99 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:15.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.988+0000 7f5c99400640 1 -- 172.21.15.90:0/2695917027 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c039070 con 0x7f5c800039e0 2024-03-24T01:25:15.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.988+0000 7f5c99400640 1 -- 172.21.15.90:0/2695917027 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c8c02c6b0 con 0x7f5c800039e0 2024-03-24T01:25:15.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.988+0000 7f5c99400640 1 -- 172.21.15.90:0/2695917027 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c009690 con 0x7f5c800039e0 2024-03-24T01:25:15.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.989+0000 7f5ca1000640 1 -- 172.21.15.90:0/2695917027 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c800039e0 msgr2=0x7f5c80003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:15.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.989+0000 7f5ca1000640 1 --2- 172.21.15.90:0/2695917027 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c800039e0 0x7f5c80003e00 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c009a90 tx=0x7f5c80005830 comp rx=0 tx=0).stop 2024-03-24T01:25:15.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.989+0000 7f5c9a800640 1 -- 172.21.15.90:0/2695917027 reap_dead start 2024-03-24T01:25:15.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.989+0000 7f5ca1000640 1 -- 172.21.15.90:0/2695917027 shutdown_connections 2024-03-24T01:25:15.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.989+0000 7f5ca1000640 1 -- 172.21.15.90:0/2695917027 >> 172.21.15.90:0/2695917027 conn(0x7f5c9c0cf7e0 msgr2=0x7f5c9c0cfbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:15.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.989+0000 7f5ca1000640 1 -- 172.21.15.90:0/2695917027 shutdown_connections 2024-03-24T01:25:15.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.989+0000 7f5ca1000640 1 -- 172.21.15.90:0/2695917027 wait complete. 2024-03-24T01:25:15.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.991+0000 7f5ca1000640 1 Processor -- start 2024-03-24T01:25:15.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.991+0000 7f5ca1000640 1 -- start start 2024-03-24T01:25:15.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.992+0000 7f5ca1000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c800039e0 0x7f5c9c1f4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:15.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.992+0000 7f5ca1000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c9c0af0b0 0x7f5c9c1f6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:15.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.992+0000 7f5ca1000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c9c1f2f50 0x7f5c9c0b3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:15.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.992+0000 7f5ca1000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9c15eae0 con 0x7f5c800039e0 2024-03-24T01:25:15.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.992+0000 7f5ca1000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9c15ee50 con 0x7f5c9c0af0b0 2024-03-24T01:25:15.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.992+0000 7f5ca1000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9c071990 con 0x7f5c9c1f2f50 2024-03-24T01:25:15.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.992+0000 7f5c99e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c800039e0 0x7f5c9c1f4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:15.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.992+0000 7f5c99e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c800039e0 0x7f5c9c1f4250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46428/0 (socket says 172.21.15.90:46428) 2024-03-24T01:25:15.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.992+0000 7f5c99e00640 1 -- 172.21.15.90:0/3190629316 learned_addr learned my addr 172.21.15.90:0/3190629316 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:15.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.992+0000 7f5c9a800640 1 --2- 172.21.15.90:0/3190629316 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c9c1f2f50 0x7f5c9c0b3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:15.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.992+0000 7f5c93e00640 1 --2- 172.21.15.90:0/3190629316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c9c0af0b0 0x7f5c9c1f6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:15.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.992+0000 7f5c99e00640 1 -- 172.21.15.90:0/3190629316 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c9c1f2f50 msgr2=0x7f5c9c0b3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:15.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.992+0000 7f5c99e00640 1 --2- 172.21.15.90:0/3190629316 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c9c1f2f50 0x7f5c9c0b3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:15.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.993+0000 7f5c99e00640 1 -- 172.21.15.90:0/3190629316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c9c0af0b0 msgr2=0x7f5c9c1f6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:15.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.993+0000 7f5c99e00640 1 --2- 172.21.15.90:0/3190629316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c9c0af0b0 0x7f5c9c1f6d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:15.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.993+0000 7f5c99e00640 1 -- 172.21.15.90:0/3190629316 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c9c0af530 con 0x7f5c800039e0 2024-03-24T01:25:15.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.993+0000 7f5c9a800640 1 --2- 172.21.15.90:0/3190629316 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c9c1f2f50 0x7f5c9c0b3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:15.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.993+0000 7f5c99e00640 1 --2- 172.21.15.90:0/3190629316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c800039e0 0x7f5c9c1f4250 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c037040 tx=0x7f5c8c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:15.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.993+0000 7f5c92000640 1 -- 172.21.15.90:0/3190629316 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c039050 con 0x7f5c800039e0 2024-03-24T01:25:15.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.993+0000 7f5c92000640 1 -- 172.21.15.90:0/3190629316 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c8c045070 con 0x7f5c800039e0 2024-03-24T01:25:15.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.993+0000 7f5c92000640 1 -- 172.21.15.90:0/3190629316 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c003a30 con 0x7f5c800039e0 2024-03-24T01:25:15.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.993+0000 7f5ca1000640 1 -- 172.21.15.90:0/3190629316 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c9c1f74c0 con 0x7f5c800039e0 2024-03-24T01:25:15.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.993+0000 7f5ca1000640 1 -- 172.21.15.90:0/3190629316 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c9c1f7a80 con 0x7f5c800039e0 2024-03-24T01:25:15.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.995+0000 7f5ca1000640 1 -- 172.21.15.90:0/3190629316 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c9c0cc270 con 0x7f5c800039e0 2024-03-24T01:25:15.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.995+0000 7f5c92000640 1 -- 172.21.15.90:0/3190629316 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5c8c003bd0 con 0x7f5c800039e0 2024-03-24T01:25:15.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.996+0000 7f5c92000640 1 --2- 172.21.15.90:0/3190629316 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5c7806bbd0 0x7f5c7806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:15.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.996+0000 7f5c92000640 1 -- 172.21.15.90:0/3190629316 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f5c8c0ac960 con 0x7f5c800039e0 2024-03-24T01:25:15.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.997+0000 7f5c93e00640 1 --2- 172.21.15.90:0/3190629316 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5c7806bbd0 0x7f5c7806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:15.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:15.998+0000 7f5c93e00640 1 --2- 172.21.15.90:0/3190629316 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5c7806bbd0 0x7f5c7806e090 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5c840017e0 tx=0x7f5c84009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:16.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.001+0000 7f5c92000640 1 -- 172.21.15.90:0/3190629316 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5c8c04e020 con 0x7f5c800039e0 2024-03-24T01:25:16.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.195+0000 7f5ca1000640 1 -- 172.21.15.90:0/3190629316 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f5c9c04f400 con 0x7f5c800039e0 2024-03-24T01:25:16.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.196+0000 7f5c92000640 1 -- 172.21.15.90:0/3190629316 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f5c9c0af530 con 0x7f5c800039e0 2024-03-24T01:25:16.197 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:25:16.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.199+0000 7f5ca1000640 1 -- 172.21.15.90:0/3190629316 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5c7806bbd0 msgr2=0x7f5c7806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:16.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.199+0000 7f5ca1000640 1 --2- 172.21.15.90:0/3190629316 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5c7806bbd0 0x7f5c7806e090 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5c840017e0 tx=0x7f5c84009040 comp rx=0 tx=0).stop 2024-03-24T01:25:16.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.199+0000 7f5ca1000640 1 -- 172.21.15.90:0/3190629316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c800039e0 msgr2=0x7f5c9c1f4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:16.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.199+0000 7f5ca1000640 1 --2- 172.21.15.90:0/3190629316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c800039e0 0x7f5c9c1f4250 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c037040 tx=0x7f5c8c004730 comp rx=0 tx=0).stop 2024-03-24T01:25:16.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.199+0000 7f5ca1000640 1 -- 172.21.15.90:0/3190629316 shutdown_connections 2024-03-24T01:25:16.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.199+0000 7f5ca1000640 1 --2- 172.21.15.90:0/3190629316 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5c7806bbd0 0x7f5c7806e090 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:16.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.199+0000 7f5ca1000640 1 --2- 172.21.15.90:0/3190629316 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5c9c1f2f50 0x7f5c9c0b3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:16.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.199+0000 7f5ca1000640 1 --2- 172.21.15.90:0/3190629316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c9c0af0b0 0x7f5c9c1f6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:16.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.199+0000 7f5ca1000640 1 --2- 172.21.15.90:0/3190629316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c800039e0 0x7f5c9c1f4250 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:16.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.199+0000 7f5ca1000640 1 -- 172.21.15.90:0/3190629316 >> 172.21.15.90:0/3190629316 conn(0x7f5c9c0cf7e0 msgr2=0x7f5c9c15f660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:16.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.199+0000 7f5ca1000640 1 -- 172.21.15.90:0/3190629316 shutdown_connections 2024-03-24T01:25:16.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.199+0000 7f5ca1000640 1 -- 172.21.15.90:0/3190629316 wait complete. 2024-03-24T01:25:16.220 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_journal_repair.TestJournalRepair.test_inject_to_empty' 2024-03-24T01:25:16.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.419+0000 7f9122200640 1 Processor -- start 2024-03-24T01:25:16.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.420+0000 7f9122200640 1 -- start start 2024-03-24T01:25:16.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.421+0000 7f9122200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f911c1454e0 0x7f911c0cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:16.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.421+0000 7f9122200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f911c0cc660 con 0x7f911c06f000 2024-03-24T01:25:16.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.421+0000 7f9122200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f911c0cc920 con 0x7f911c0ce360 2024-03-24T01:25:16.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.421+0000 7f9122200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f911c0cc430 con 0x7f911c1454e0 2024-03-24T01:25:16.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.422+0000 7f911b400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f911c1454e0 0x7f911c0cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:16.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.422+0000 7f911aa00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f911c06f000 0x7f911c0cdc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:46714/0 (socket says 172.21.15.90:46714) 2024-03-24T01:25:16.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.422+0000 7f911aa00640 1 -- 172.21.15.90:0/805830582 learned_addr learned my addr 172.21.15.90:0/805830582 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:16.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.422+0000 7f911b400640 1 -- 172.21.15.90:0/805830582 >> v1:172.21.15.120:6789/0 conn(0x7f911c0ce360 legacy=0x7f911c15c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:16.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.422+0000 7f911a000640 1 -- 172.21.15.90:0/805830582 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 544285494 0 0) 0x7f911c0cc660 con 0x7f911c06f000 2024-03-24T01:25:16.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.422+0000 7f911b400640 1 -- 172.21.15.90:0/805830582 >> v1:172.21.15.90:6789/0 conn(0x7f911c06f000 legacy=0x7f911c0cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:16.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.422+0000 7f911b400640 1 -- 172.21.15.90:0/805830582 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f911c0ccc00 con 0x7f911c1454e0 2024-03-24T01:25:16.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.423+0000 7f911b400640 1 --2- 172.21.15.90:0/805830582 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f911c1454e0 0x7f911c0cd6e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f910c009d90 tx=0x7f910c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7797981324d2d53 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:16.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.423+0000 7f911a000640 1 -- 172.21.15.90:0/805830582 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f910c039070 con 0x7f911c1454e0 2024-03-24T01:25:16.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.423+0000 7f911a000640 1 -- 172.21.15.90:0/805830582 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f910c02cd00 con 0x7f911c1454e0 2024-03-24T01:25:16.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.423+0000 7f911a000640 1 -- 172.21.15.90:0/805830582 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f910c034660 con 0x7f911c1454e0 2024-03-24T01:25:16.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.424+0000 7f9122200640 1 -- 172.21.15.90:0/805830582 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f911c1454e0 msgr2=0x7f911c0cd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:16.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.424+0000 7f9122200640 1 --2- 172.21.15.90:0/805830582 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f911c1454e0 0x7f911c0cd6e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f910c009d90 tx=0x7f910c02bf50 comp rx=0 tx=0).stop 2024-03-24T01:25:16.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.424+0000 7f9122200640 1 -- 172.21.15.90:0/805830582 shutdown_connections 2024-03-24T01:25:16.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.424+0000 7f9122200640 1 --2- 172.21.15.90:0/805830582 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f911c1454e0 0x7f911c0cd6e0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:16.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.424+0000 7f9122200640 1 -- 172.21.15.90:0/805830582 >> 172.21.15.90:0/805830582 conn(0x7f911c06b2a0 msgr2=0x7f911c06b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:16.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.424+0000 7f9122200640 1 -- 172.21.15.90:0/805830582 shutdown_connections 2024-03-24T01:25:16.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.424+0000 7f9122200640 1 -- 172.21.15.90:0/805830582 wait complete. 2024-03-24T01:25:16.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.426+0000 7f9122200640 1 Processor -- start 2024-03-24T01:25:16.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.426+0000 7f9122200640 1 -- start start 2024-03-24T01:25:16.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.427+0000 7f9122200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f911c06f000 0x7f911c158340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:16.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.427+0000 7f9122200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f911c0ce360 0x7f911c12f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:16.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.427+0000 7f9122200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f911c1454e0 0x7f911c12d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:16.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.427+0000 7f9122200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911c06f440 con 0x7f911c06f000 2024-03-24T01:25:16.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.427+0000 7f9122200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911c0cd1c0 con 0x7f911c0ce360 2024-03-24T01:25:16.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.427+0000 7f9122200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f911c06f5c0 con 0x7f911c1454e0 2024-03-24T01:25:16.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.427+0000 7f911b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f911c06f000 0x7f911c158340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:16.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.427+0000 7f911b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f911c06f000 0x7f911c158340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46442/0 (socket says 172.21.15.90:46442) 2024-03-24T01:25:16.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.427+0000 7f911b400640 1 -- 172.21.15.90:0/2748575443 learned_addr learned my addr 172.21.15.90:0/2748575443 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:16.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.427+0000 7f911aa00640 1 --2- 172.21.15.90:0/2748575443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f911c0ce360 0x7f911c12f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:16.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.428+0000 7f911be00640 1 --2- 172.21.15.90:0/2748575443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f911c1454e0 0x7f911c12d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:16.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.428+0000 7f911b400640 1 -- 172.21.15.90:0/2748575443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f911c1454e0 msgr2=0x7f911c12d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:16.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.428+0000 7f911b400640 1 --2- 172.21.15.90:0/2748575443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f911c1454e0 0x7f911c12d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:16.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.428+0000 7f911b400640 1 -- 172.21.15.90:0/2748575443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f911c0ce360 msgr2=0x7f911c12f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:16.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.428+0000 7f911b400640 1 --2- 172.21.15.90:0/2748575443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f911c0ce360 0x7f911c12f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:16.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.428+0000 7f911b400640 1 -- 172.21.15.90:0/2748575443 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f911c12f830 con 0x7f911c06f000 2024-03-24T01:25:16.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.428+0000 7f911b400640 1 --2- 172.21.15.90:0/2748575443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f911c06f000 0x7f911c158340 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f910c009d90 tx=0x7f910c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:16.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.428+0000 7f9113400640 1 -- 172.21.15.90:0/2748575443 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f910c044020 con 0x7f911c06f000 2024-03-24T01:25:16.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.428+0000 7f9113400640 1 -- 172.21.15.90:0/2748575443 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f910c03f670 con 0x7f911c06f000 2024-03-24T01:25:16.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.428+0000 7f9113400640 1 -- 172.21.15.90:0/2748575443 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f910c039040 con 0x7f911c06f000 2024-03-24T01:25:16.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.428+0000 7f9122200640 1 -- 172.21.15.90:0/2748575443 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f911c12e250 con 0x7f911c06f000 2024-03-24T01:25:16.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.429+0000 7f9122200640 1 -- 172.21.15.90:0/2748575443 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f911c12e810 con 0x7f911c06f000 2024-03-24T01:25:16.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.430+0000 7f9122200640 1 -- 172.21.15.90:0/2748575443 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f911c073570 con 0x7f911c06f000 2024-03-24T01:25:16.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.430+0000 7f9113400640 1 -- 172.21.15.90:0/2748575443 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f910c047020 con 0x7f911c06f000 2024-03-24T01:25:16.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.431+0000 7f9113400640 1 --2- 172.21.15.90:0/2748575443 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f90f806bbd0 0x7f90f806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:16.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.431+0000 7f9113400640 1 -- 172.21.15.90:0/2748575443 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f910c0ada70 con 0x7f911c06f000 2024-03-24T01:25:16.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.432+0000 7f911aa00640 1 --2- 172.21.15.90:0/2748575443 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f90f806bbd0 0x7f90f806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:16.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.433+0000 7f911aa00640 1 --2- 172.21.15.90:0/2748575443 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f90f806bbd0 0x7f90f806e090 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f911c073270 tx=0x7f9104007b90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:16.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.435+0000 7f9113400640 1 -- 172.21.15.90:0/2748575443 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f910c04e030 con 0x7f911c06f000 2024-03-24T01:25:16.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:16.707+0000 7f9122200640 1 -- 172.21.15.90:0/2748575443 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_journal_repair.TestJournalRepair.test_inject_to_empty"]} v 0) v1 -- 0x7f911c04f4e0 con 0x7f911c06f000 2024-03-24T01:25:17.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.651+0000 7f9113400640 1 -- 172.21.15.90:0/2748575443 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_journal_repair.TestJournalRepair.test_inject_to_empty"]}]=0 v84) v1 ==== 153+0+0 (secure 0 0 0) 0x7f910c07c1d0 con 0x7f911c06f000 2024-03-24T01:25:17.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.655+0000 7f9122200640 1 -- 172.21.15.90:0/2748575443 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f90f806bbd0 msgr2=0x7f90f806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:17.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.655+0000 7f9122200640 1 --2- 172.21.15.90:0/2748575443 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f90f806bbd0 0x7f90f806e090 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f911c073270 tx=0x7f9104007b90 comp rx=0 tx=0).stop 2024-03-24T01:25:17.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.655+0000 7f9122200640 1 -- 172.21.15.90:0/2748575443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f911c06f000 msgr2=0x7f911c158340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:17.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.655+0000 7f9122200640 1 --2- 172.21.15.90:0/2748575443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f911c06f000 0x7f911c158340 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f910c009d90 tx=0x7f910c004a40 comp rx=0 tx=0).stop 2024-03-24T01:25:17.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.655+0000 7f9122200640 1 -- 172.21.15.90:0/2748575443 shutdown_connections 2024-03-24T01:25:17.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.655+0000 7f9122200640 1 --2- 172.21.15.90:0/2748575443 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f90f806bbd0 0x7f90f806e090 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:17.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.655+0000 7f9122200640 1 --2- 172.21.15.90:0/2748575443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f911c1454e0 0x7f911c12d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:17.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.655+0000 7f9122200640 1 --2- 172.21.15.90:0/2748575443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f911c0ce360 0x7f911c12f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:17.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.655+0000 7f9122200640 1 --2- 172.21.15.90:0/2748575443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f911c06f000 0x7f911c158340 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:17.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.655+0000 7f9122200640 1 -- 172.21.15.90:0/2748575443 >> 172.21.15.90:0/2748575443 conn(0x7f911c06b2a0 msgr2=0x7f911c13c140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:17.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.656+0000 7f9122200640 1 -- 172.21.15.90:0/2748575443 shutdown_connections 2024-03-24T01:25:17.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.656+0000 7f9122200640 1 -- 172.21.15.90:0/2748575443 wait complete. 2024-03-24T01:25:17.677 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-24T01:25:17.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.891+0000 7f5137c00640 1 Processor -- start 2024-03-24T01:25:17.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.891+0000 7f5137c00640 1 -- start start 2024-03-24T01:25:17.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.893+0000 7f5137c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f51300aeeb0 0x7f51300b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:17.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.893+0000 7f5137c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f513006b7d0 0x7f51300b2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:17.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.893+0000 7f5137c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f513006dac0 con 0x7f513015bc50 2024-03-24T01:25:17.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.893+0000 7f5137c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51300b4da0 con 0x7f51300aeeb0 2024-03-24T01:25:17.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.893+0000 7f5137c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f513006dda0 con 0x7f513006b7d0 2024-03-24T01:25:17.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.893+0000 7f5135200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f51300aeeb0 0x7f51300b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:17.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.893+0000 7f512fe00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f513015bc50 0x7f513006b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:46716/0 (socket says 172.21.15.90:46716) 2024-03-24T01:25:17.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.893+0000 7f512fe00640 1 -- 172.21.15.90:0/4043630009 learned_addr learned my addr 172.21.15.90:0/4043630009 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:17.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.893+0000 7f5135c00640 1 --2- 172.21.15.90:0/4043630009 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f513006b7d0 0x7f51300b2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:17.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.893+0000 7f5135200640 1 -- 172.21.15.90:0/4043630009 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f513006b7d0 msgr2=0x7f51300b2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:17.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.893+0000 7f5135200640 1 --2- 172.21.15.90:0/4043630009 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f513006b7d0 0x7f51300b2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:17.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.893+0000 7f5135200640 1 -- 172.21.15.90:0/4043630009 >> v1:172.21.15.90:6789/0 conn(0x7f513015bc50 legacy=0x7f513006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:17.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.894+0000 7f5135200640 1 -- 172.21.15.90:0/4043630009 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513006e080 con 0x7f51300aeeb0 2024-03-24T01:25:17.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.894+0000 7f5135c00640 1 --2- 172.21.15.90:0/4043630009 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f513006b7d0 0x7f51300b2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:17.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.894+0000 7f512f400640 1 -- 172.21.15.90:0/4043630009 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 784173400 0 0) 0x7f513006dac0 con 0x7f513015bc50 2024-03-24T01:25:17.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.895+0000 7f5135200640 1 --2- 172.21.15.90:0/4043630009 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f51300aeeb0 0x7f51300b44c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f51180097a0 tx=0x7f511802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c5f2e4d683569a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:17.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.895+0000 7f512f400640 1 -- 172.21.15.90:0/4043630009 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5118038070 con 0x7f51300aeeb0 2024-03-24T01:25:17.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.895+0000 7f512f400640 1 -- 172.21.15.90:0/4043630009 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5118031700 con 0x7f51300aeeb0 2024-03-24T01:25:17.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.895+0000 7f512f400640 1 -- 172.21.15.90:0/4043630009 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5118031ad0 con 0x7f51300aeeb0 2024-03-24T01:25:17.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.895+0000 7f5137c00640 1 -- 172.21.15.90:0/4043630009 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f51300aeeb0 msgr2=0x7f51300b44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:17.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.895+0000 7f5137c00640 1 --2- 172.21.15.90:0/4043630009 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f51300aeeb0 0x7f51300b44c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f51180097a0 tx=0x7f511802b7a0 comp rx=0 tx=0).stop 2024-03-24T01:25:17.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.896+0000 7f5137c00640 1 -- 172.21.15.90:0/4043630009 shutdown_connections 2024-03-24T01:25:17.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.896+0000 7f5137c00640 1 --2- 172.21.15.90:0/4043630009 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f513006b7d0 0x7f51300b2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:17.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.896+0000 7f5137c00640 1 --2- 172.21.15.90:0/4043630009 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f51300aeeb0 0x7f51300b44c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:17.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.896+0000 7f5137c00640 1 -- 172.21.15.90:0/4043630009 >> 172.21.15.90:0/4043630009 conn(0x7f51300b1770 msgr2=0x7f51300b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:17.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.896+0000 7f5137c00640 1 -- 172.21.15.90:0/4043630009 shutdown_connections 2024-03-24T01:25:17.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.896+0000 7f5137c00640 1 -- 172.21.15.90:0/4043630009 wait complete. 2024-03-24T01:25:17.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.897+0000 7f5137c00640 1 Processor -- start 2024-03-24T01:25:17.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.897+0000 7f5137c00640 1 -- start start 2024-03-24T01:25:17.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.898+0000 7f5137c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f513006b7d0 0x7f513014edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:17.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.898+0000 7f5137c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51300aeeb0 0x7f513014d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:17.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.898+0000 7f5137c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f513015bc50 0x7f51300b2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:17.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.898+0000 7f5137c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51300b4a00 con 0x7f513015bc50 2024-03-24T01:25:17.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.898+0000 7f5137c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51300af290 con 0x7f513006b7d0 2024-03-24T01:25:17.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.898+0000 7f5137c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51300b4b80 con 0x7f51300aeeb0 2024-03-24T01:25:17.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.899+0000 7f5135c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f513015bc50 0x7f51300b2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:17.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.899+0000 7f5135c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f513015bc50 0x7f51300b2680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46454/0 (socket says 172.21.15.90:46454) 2024-03-24T01:25:17.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.899+0000 7f5135c00640 1 -- 172.21.15.90:0/1022828351 learned_addr learned my addr 172.21.15.90:0/1022828351 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:17.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.899+0000 7f5135200640 1 --2- 172.21.15.90:0/1022828351 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f513006b7d0 0x7f513014edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:17.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.899+0000 7f512fe00640 1 --2- 172.21.15.90:0/1022828351 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51300aeeb0 0x7f513014d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:17.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.899+0000 7f5135c00640 1 -- 172.21.15.90:0/1022828351 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51300aeeb0 msgr2=0x7f513014d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:17.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.899+0000 7f5135c00640 1 --2- 172.21.15.90:0/1022828351 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51300aeeb0 0x7f513014d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:17.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.899+0000 7f5135c00640 1 -- 172.21.15.90:0/1022828351 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f513006b7d0 msgr2=0x7f513014edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:17.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.899+0000 7f5135c00640 1 --2- 172.21.15.90:0/1022828351 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f513006b7d0 0x7f513014edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:17.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.899+0000 7f5135c00640 1 -- 172.21.15.90:0/1022828351 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5130150280 con 0x7f513015bc50 2024-03-24T01:25:17.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.900+0000 7f5135c00640 1 --2- 172.21.15.90:0/1022828351 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f513015bc50 0x7f51300b2680 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f512800c820 tx=0x7f512800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:17.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.900+0000 7f512cc00640 1 -- 172.21.15.90:0/1022828351 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5128015070 con 0x7f513015bc50 2024-03-24T01:25:17.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.900+0000 7f512cc00640 1 -- 172.21.15.90:0/1022828351 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5128007a30 con 0x7f513015bc50 2024-03-24T01:25:17.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.900+0000 7f5137c00640 1 -- 172.21.15.90:0/1022828351 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51300ac980 con 0x7f513015bc50 2024-03-24T01:25:17.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.900+0000 7f512cc00640 1 -- 172.21.15.90:0/1022828351 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5128007e20 con 0x7f513015bc50 2024-03-24T01:25:17.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.900+0000 7f5137c00640 1 -- 172.21.15.90:0/1022828351 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51300acf40 con 0x7f513015bc50 2024-03-24T01:25:17.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.901+0000 7f5137c00640 1 -- 172.21.15.90:0/1022828351 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f513004f400 con 0x7f513015bc50 2024-03-24T01:25:17.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.902+0000 7f512cc00640 1 -- 172.21.15.90:0/1022828351 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5128018070 con 0x7f513015bc50 2024-03-24T01:25:17.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.903+0000 7f512cc00640 1 --2- 172.21.15.90:0/1022828351 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f510c06bbd0 0x7f510c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:17.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.903+0000 7f512cc00640 1 -- 172.21.15.90:0/1022828351 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f512808ada0 con 0x7f513015bc50 2024-03-24T01:25:17.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.904+0000 7f5135200640 1 --2- 172.21.15.90:0/1022828351 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f510c06bbd0 0x7f510c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:17.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.905+0000 7f5135200640 1 --2- 172.21.15.90:0/1022828351 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f510c06bbd0 0x7f510c06e090 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5118035000 tx=0x7f5118008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:17.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:17.908+0000 7f512cc00640 1 -- 172.21.15.90:0/1022828351 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5128059500 con 0x7f513015bc50 2024-03-24T01:25:18.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.107+0000 7f5137c00640 1 -- 172.21.15.90:0/1022828351 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f51301576d0 con 0x7f513015bc50 2024-03-24T01:25:18.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.114+0000 7f512cc00640 1 -- 172.21.15.90:0/1022828351 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5128058e00 con 0x7f513015bc50 2024-03-24T01:25:18.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.116+0000 7f5137c00640 1 -- 172.21.15.90:0/1022828351 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f510c06bbd0 msgr2=0x7f510c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:18.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.116+0000 7f5137c00640 1 --2- 172.21.15.90:0/1022828351 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f510c06bbd0 0x7f510c06e090 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5118035000 tx=0x7f5118008030 comp rx=0 tx=0).stop 2024-03-24T01:25:18.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.116+0000 7f5137c00640 1 -- 172.21.15.90:0/1022828351 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f513015bc50 msgr2=0x7f51300b2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:18.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.116+0000 7f5137c00640 1 --2- 172.21.15.90:0/1022828351 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f513015bc50 0x7f51300b2680 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f512800c820 tx=0x7f512800cce0 comp rx=0 tx=0).stop 2024-03-24T01:25:18.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.117+0000 7f5137c00640 1 -- 172.21.15.90:0/1022828351 shutdown_connections 2024-03-24T01:25:18.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.117+0000 7f5137c00640 1 --2- 172.21.15.90:0/1022828351 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f510c06bbd0 0x7f510c06e090 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:18.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.117+0000 7f5137c00640 1 --2- 172.21.15.90:0/1022828351 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f513015bc50 0x7f51300b2680 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:18.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.117+0000 7f5137c00640 1 --2- 172.21.15.90:0/1022828351 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f51300aeeb0 0x7f513014d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:18.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.117+0000 7f5137c00640 1 --2- 172.21.15.90:0/1022828351 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f513006b7d0 0x7f513014edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:18.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.117+0000 7f5137c00640 1 -- 172.21.15.90:0/1022828351 >> 172.21.15.90:0/1022828351 conn(0x7f51300b1770 msgr2=0x7f513006b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:18.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.117+0000 7f5137c00640 1 -- 172.21.15.90:0/1022828351 shutdown_connections 2024-03-24T01:25:18.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.117+0000 7f5137c00640 1 -- 172.21.15.90:0/1022828351 wait complete. 2024-03-24T01:25:18.138 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T01:25:18.138 DEBUG:teuthology.orchestra.run.smithi090:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T01:25:18.245 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T01:25:18.358 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T01:25:18.359 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:25:18.359 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:25:18.415 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:25:18.415 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:25:18.466 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:25:18.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.669+0000 7f740f200640 1 Processor -- start 2024-03-24T01:25:18.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.670+0000 7f740f200640 1 -- start start 2024-03-24T01:25:18.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.671+0000 7f740f200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74100cef60 0x7f741015ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:18.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.671+0000 7f740f200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f741015b980 con 0x7f741015ae50 2024-03-24T01:25:18.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.671+0000 7f740f200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f741015bc60 con 0x7f741015b2d0 2024-03-24T01:25:18.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.671+0000 7f740f200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f741015f390 con 0x7f74100cef60 2024-03-24T01:25:18.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.671+0000 7f7405400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f741015ae50 0x7f74100b2680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:46722/0 (socket says 172.21.15.90:46722) 2024-03-24T01:25:18.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.671+0000 7f7405400640 1 -- 172.21.15.90:0/307617787 learned_addr learned my addr 172.21.15.90:0/307617787 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:18.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.672+0000 7f740de00640 1 --2- 172.21.15.90:0/307617787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74100cef60 0x7f741015ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:18.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.672+0000 7f740d400640 1 -- 172.21.15.90:0/307617787 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3226963760 0 0) 0x7f741015b980 con 0x7f741015ae50 2024-03-24T01:25:18.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.672+0000 7f740d400640 1 -- 172.21.15.90:0/307617787 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73f4003520 con 0x7f741015ae50 2024-03-24T01:25:18.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.672+0000 7f740d400640 1 -- 172.21.15.90:0/307617787 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2616858384 0 0) 0x7f73f4003520 con 0x7f741015ae50 2024-03-24T01:25:18.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.672+0000 7f740d400640 1 -- 172.21.15.90:0/307617787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74100cef60 msgr2=0x7f741015ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:18.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.672+0000 7f740d400640 1 --2- 172.21.15.90:0/307617787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74100cef60 0x7f741015ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:18.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.672+0000 7f740d400640 1 -- 172.21.15.90:0/307617787 >> v1:172.21.15.120:6789/0 conn(0x7f741015b2d0 legacy=0x7f74100cf340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:18.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.672+0000 7f740d400640 1 -- 172.21.15.90:0/307617787 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f741015e650 con 0x7f741015ae50 2024-03-24T01:25:18.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.672+0000 7f740d400640 1 -- 172.21.15.90:0/307617787 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f73f8001670 con 0x7f741015ae50 2024-03-24T01:25:18.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.672+0000 7f740d400640 1 -- 172.21.15.90:0/307617787 >> v1:172.21.15.90:6789/0 conn(0x7f741015ae50 legacy=0x7f74100b2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:18.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.672+0000 7f740d400640 1 --2- 172.21.15.90:0/307617787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74100cef60 0x7f73f40039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:18.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.672+0000 7f740d400640 1 --2- 172.21.15.90:0/307617787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73f4003f60 0x7f73f4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:18.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.672+0000 7f740d400640 1 --2- 172.21.15.90:0/307617787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73f4004d40 0x7f73f40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:18.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.672+0000 7f740d400640 1 -- 172.21.15.90:0/307617787 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741015f390 con 0x7f73f4004d40 2024-03-24T01:25:18.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.672+0000 7f740d400640 1 -- 172.21.15.90:0/307617787 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f400baa0 con 0x7f73f4003f60 2024-03-24T01:25:18.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.672+0000 7f740d400640 1 -- 172.21.15.90:0/307617787 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73f400bca0 con 0x7f74100cef60 2024-03-24T01:25:18.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.673+0000 7f740e800640 1 --2- 172.21.15.90:0/307617787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73f4004d40 0x7f73f40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:18.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.673+0000 7f7405400640 1 --2- 172.21.15.90:0/307617787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73f4003f60 0x7f73f4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:18.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.673+0000 7f740de00640 1 --2- 172.21.15.90:0/307617787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74100cef60 0x7f73f40039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:18.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.673+0000 7f740e800640 1 -- 172.21.15.90:0/307617787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74100cef60 msgr2=0x7f73f40039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:18.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.673+0000 7f740e800640 1 --2- 172.21.15.90:0/307617787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74100cef60 0x7f73f40039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:18.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.673+0000 7f740e800640 1 -- 172.21.15.90:0/307617787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73f4003f60 msgr2=0x7f73f4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:18.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.673+0000 7f740e800640 1 --2- 172.21.15.90:0/307617787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73f4003f60 0x7f73f4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:18.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.673+0000 7f740e800640 1 -- 172.21.15.90:0/307617787 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f4003520 con 0x7f73f4004d40 2024-03-24T01:25:18.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.673+0000 7f740de00640 1 --2- 172.21.15.90:0/307617787 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f74100cef60 0x7f73f40039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:18.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.674+0000 7f740e800640 1 --2- 172.21.15.90:0/307617787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73f4004d40 0x7f73f40051c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7408001e50 tx=0x7f73f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9486397c1e3975bc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:18.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.674+0000 7f740d400640 1 -- 172.21.15.90:0/307617787 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7408001a70 con 0x7f73f4004d40 2024-03-24T01:25:18.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.674+0000 7f740d400640 1 -- 172.21.15.90:0/307617787 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7408003b70 con 0x7f73f4004d40 2024-03-24T01:25:18.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.674+0000 7f740d400640 1 -- 172.21.15.90:0/307617787 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74080050d0 con 0x7f73f4004d40 2024-03-24T01:25:18.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.675+0000 7f740f200640 1 -- 172.21.15.90:0/307617787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73f4004d40 msgr2=0x7f73f40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:18.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.675+0000 7f740f200640 1 --2- 172.21.15.90:0/307617787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73f4004d40 0x7f73f40051c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7408001e50 tx=0x7f73f4005830 comp rx=0 tx=0).stop 2024-03-24T01:25:18.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.675+0000 7f740e800640 1 -- 172.21.15.90:0/307617787 reap_dead start 2024-03-24T01:25:18.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.675+0000 7f740f200640 1 -- 172.21.15.90:0/307617787 shutdown_connections 2024-03-24T01:25:18.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.675+0000 7f740f200640 1 -- 172.21.15.90:0/307617787 >> 172.21.15.90:0/307617787 conn(0x7f741006b0c0 msgr2=0x7f741006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:18.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.675+0000 7f740f200640 1 -- 172.21.15.90:0/307617787 shutdown_connections 2024-03-24T01:25:18.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.675+0000 7f740f200640 1 -- 172.21.15.90:0/307617787 wait complete. 2024-03-24T01:25:18.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.677+0000 7f740f200640 1 Processor -- start 2024-03-24T01:25:18.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.677+0000 7f740f200640 1 -- start start 2024-03-24T01:25:18.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.678+0000 7f740f200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73f4004d40 0x7f74101470a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:18.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.678+0000 7f740f200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74101e66e0 0x7f74100a9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:18.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.678+0000 7f740f200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74100a9850 0x7f74100aa190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:18.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.678+0000 7f740f200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74100cc080 con 0x7f74100a9850 2024-03-24T01:25:18.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.678+0000 7f740f200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741015b720 con 0x7f74101e66e0 2024-03-24T01:25:18.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.678+0000 7f740f200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f741015ee50 con 0x7f73f4004d40 2024-03-24T01:25:18.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.678+0000 7f740e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74100a9850 0x7f74100aa190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:18.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.678+0000 7f740e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74100a9850 0x7f74100aa190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46476/0 (socket says 172.21.15.90:46476) 2024-03-24T01:25:18.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.678+0000 7f740e800640 1 -- 172.21.15.90:0/603510536 learned_addr learned my addr 172.21.15.90:0/603510536 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:18.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.678+0000 7f7405400640 1 --2- 172.21.15.90:0/603510536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74101e66e0 0x7f74100a9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:18.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.678+0000 7f740de00640 1 --2- 172.21.15.90:0/603510536 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73f4004d40 0x7f74101470a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:18.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.678+0000 7f740e800640 1 -- 172.21.15.90:0/603510536 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73f4004d40 msgr2=0x7f74101470a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:18.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.678+0000 7f740e800640 1 --2- 172.21.15.90:0/603510536 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73f4004d40 0x7f74101470a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:18.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.679+0000 7f740e800640 1 -- 172.21.15.90:0/603510536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74101e66e0 msgr2=0x7f74100a9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:18.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.679+0000 7f740e800640 1 --2- 172.21.15.90:0/603510536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74101e66e0 0x7f74100a9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:18.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.679+0000 7f740e800640 1 -- 172.21.15.90:0/603510536 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74101e5640 con 0x7f74100a9850 2024-03-24T01:25:18.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.679+0000 7f7405400640 1 --2- 172.21.15.90:0/603510536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74101e66e0 0x7f74100a9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:18.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.679+0000 7f740e800640 1 --2- 172.21.15.90:0/603510536 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74100a9850 0x7f74100aa190 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7408001f70 tx=0x7f74080014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:18.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.679+0000 7f7406a00640 1 -- 172.21.15.90:0/603510536 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7408007db0 con 0x7f74100a9850 2024-03-24T01:25:18.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.679+0000 7f7406a00640 1 -- 172.21.15.90:0/603510536 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7408003970 con 0x7f74100a9850 2024-03-24T01:25:18.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.679+0000 7f740f200640 1 -- 172.21.15.90:0/603510536 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74101e5920 con 0x7f74100a9850 2024-03-24T01:25:18.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.679+0000 7f7406a00640 1 -- 172.21.15.90:0/603510536 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7408006d90 con 0x7f74100a9850 2024-03-24T01:25:18.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.679+0000 7f740f200640 1 -- 172.21.15.90:0/603510536 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74101e5ee0 con 0x7f74100a9850 2024-03-24T01:25:18.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.680+0000 7f740f200640 1 -- 172.21.15.90:0/603510536 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f741004f400 con 0x7f74100a9850 2024-03-24T01:25:18.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.681+0000 7f7406a00640 1 -- 172.21.15.90:0/603510536 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7408041030 con 0x7f74100a9850 2024-03-24T01:25:18.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.682+0000 7f7406a00640 1 --2- 172.21.15.90:0/603510536 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f73ec06bc00 0x7f73ec06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:18.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.682+0000 7f7406a00640 1 -- 172.21.15.90:0/603510536 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f74080abeb0 con 0x7f74100a9850 2024-03-24T01:25:18.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.683+0000 7f740de00640 1 --2- 172.21.15.90:0/603510536 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f73ec06bc00 0x7f73ec06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:18.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.684+0000 7f740de00640 1 --2- 172.21.15.90:0/603510536 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f73ec06bc00 0x7f73ec06e0c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f74000099c0 tx=0x7f7400008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:18.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.687+0000 7f7406a00640 1 -- 172.21.15.90:0/603510536 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7408033020 con 0x7f74100a9850 2024-03-24T01:25:18.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.885+0000 7f740f200640 1 -- 172.21.15.90:0/603510536 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f741006f9f0 con 0x7f74100a9850 2024-03-24T01:25:18.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.887+0000 7f7406a00640 1 -- 172.21.15.90:0/603510536 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18925 (secure 0 0 0) 0x7f740807a4e0 con 0x7f74100a9850 2024-03-24T01:25:18.890 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:25:18.890 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:24:53.236401+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:24:08.643966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T01:24:09.189904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T01:24:13.027323+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:25:18.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.890+0000 7f740f200640 1 -- 172.21.15.90:0/603510536 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f73ec06bc00 msgr2=0x7f73ec06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:18.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.890+0000 7f740f200640 1 --2- 172.21.15.90:0/603510536 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f73ec06bc00 0x7f73ec06e0c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f74000099c0 tx=0x7f7400008040 comp rx=0 tx=0).stop 2024-03-24T01:25:18.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.890+0000 7f740f200640 1 -- 172.21.15.90:0/603510536 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74100a9850 msgr2=0x7f74100aa190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:18.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.890+0000 7f740f200640 1 --2- 172.21.15.90:0/603510536 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74100a9850 0x7f74100aa190 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7408001f70 tx=0x7f74080014a0 comp rx=0 tx=0).stop 2024-03-24T01:25:18.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.890+0000 7f740f200640 1 -- 172.21.15.90:0/603510536 shutdown_connections 2024-03-24T01:25:18.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.890+0000 7f740f200640 1 --2- 172.21.15.90:0/603510536 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f73ec06bc00 0x7f73ec06e0c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:18.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.890+0000 7f740f200640 1 --2- 172.21.15.90:0/603510536 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74100a9850 0x7f74100aa190 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:18.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.890+0000 7f740f200640 1 --2- 172.21.15.90:0/603510536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74101e66e0 0x7f74100a9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:18.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.890+0000 7f740f200640 1 --2- 172.21.15.90:0/603510536 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73f4004d40 0x7f74101470a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:18.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.890+0000 7f740f200640 1 -- 172.21.15.90:0/603510536 >> 172.21.15.90:0/603510536 conn(0x7f741006b0c0 msgr2=0x7f74100b2ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:18.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.890+0000 7f740f200640 1 -- 172.21.15.90:0/603510536 shutdown_connections 2024-03-24T01:25:18.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:18.890+0000 7f740f200640 1 -- 172.21.15.90:0/603510536 wait complete. 2024-03-24T01:25:18.910 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T01:24:08.643966+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T01:24:09.189904+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T01:24:13.027323+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T01:25:18.911 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:25:19.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.123+0000 7f0d29e00640 1 Processor -- start 2024-03-24T01:25:19.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.123+0000 7f0d29e00640 1 -- start start 2024-03-24T01:25:19.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.125+0000 7f0d29e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0d240ae710 0x7f0d240aeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:19.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.125+0000 7f0d29e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0d240cf660 0x7f0d240cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:19.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.125+0000 7f0d29e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d240af030 con 0x7f0d24071790 2024-03-24T01:25:19.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.125+0000 7f0d29e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d240d0020 con 0x7f0d240cf660 2024-03-24T01:25:19.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.125+0000 7f0d29e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d2415e3b0 con 0x7f0d240ae710 2024-03-24T01:25:19.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.125+0000 7f0d1be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0d240cf660 0x7f0d240cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:19.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.125+0000 7f0d1be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0d240cf660 0x7f0d240cfae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46478/0 (socket says 172.21.15.90:46478) 2024-03-24T01:25:19.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.125+0000 7f0d1be00640 1 -- 172.21.15.90:0/2913143555 learned_addr learned my addr 172.21.15.90:0/2913143555 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:19.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.125+0000 7f0d28a00640 1 --2- 172.21.15.90:0/2913143555 >> v2:172.21.15.120:3300/0 conn(0x7f0d240ae710 0x7f0d240aeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:19.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.126+0000 7f0d1be00640 1 -- 172.21.15.90:0/2913143555 >> v2:172.21.15.120:3300/0 conn(0x7f0d240ae710 msgr2=0x7f0d240aeaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:19.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.126+0000 7f0d1be00640 1 --2- 172.21.15.90:0/2913143555 >> v2:172.21.15.120:3300/0 conn(0x7f0d240ae710 0x7f0d240aeaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:19.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.126+0000 7f0d1be00640 1 -- 172.21.15.90:0/2913143555 >> v1:172.21.15.120:6789/0 conn(0x7f0d24071790 legacy=0x7f0d24071bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:19.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.126+0000 7f0d1be00640 1 -- 172.21.15.90:0/2913143555 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d2415e690 con 0x7f0d240cf660 2024-03-24T01:25:19.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.126+0000 7f0d28a00640 1 --2- 172.21.15.90:0/2913143555 >> v2:172.21.15.120:3300/0 conn(0x7f0d240ae710 0x7f0d240aeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:19.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.127+0000 7f0d1be00640 1 --2- 172.21.15.90:0/2913143555 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d240cf660 0x7f0d240cfae0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0d24072c30 tx=0x7f0d1402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e0617954bd5b8c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:19.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.127+0000 7f0d23e00640 1 -- 172.21.15.90:0/2913143555 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d14035020 con 0x7f0d240cf660 2024-03-24T01:25:19.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.127+0000 7f0d23e00640 1 -- 172.21.15.90:0/2913143555 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d14031870 con 0x7f0d240cf660 2024-03-24T01:25:19.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.127+0000 7f0d23e00640 1 -- 172.21.15.90:0/2913143555 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d14031c40 con 0x7f0d240cf660 2024-03-24T01:25:19.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.128+0000 7f0d29e00640 1 -- 172.21.15.90:0/2913143555 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d240cf660 msgr2=0x7f0d240cfae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:19.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.128+0000 7f0d29e00640 1 --2- 172.21.15.90:0/2913143555 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d240cf660 0x7f0d240cfae0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0d24072c30 tx=0x7f0d1402b910 comp rx=0 tx=0).stop 2024-03-24T01:25:19.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.128+0000 7f0d29e00640 1 -- 172.21.15.90:0/2913143555 shutdown_connections 2024-03-24T01:25:19.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.128+0000 7f0d29e00640 1 --2- 172.21.15.90:0/2913143555 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d240cf660 0x7f0d240cfae0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:19.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.128+0000 7f0d29e00640 1 --2- 172.21.15.90:0/2913143555 >> v2:172.21.15.120:3300/0 conn(0x7f0d240ae710 0x7f0d240aeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:19.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.128+0000 7f0d29e00640 1 -- 172.21.15.90:0/2913143555 >> 172.21.15.90:0/2913143555 conn(0x7f0d2406b0c0 msgr2=0x7f0d2406b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:19.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.128+0000 7f0d29e00640 1 -- 172.21.15.90:0/2913143555 shutdown_connections 2024-03-24T01:25:19.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.128+0000 7f0d29e00640 1 -- 172.21.15.90:0/2913143555 wait complete. 2024-03-24T01:25:19.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.130+0000 7f0d29e00640 1 Processor -- start 2024-03-24T01:25:19.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.130+0000 7f0d29e00640 1 -- start start 2024-03-24T01:25:19.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.131+0000 7f0d29e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d24071790 0x7f0d241556d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:19.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.131+0000 7f0d29e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d240ae710 0x7f0d24158ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:19.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.131+0000 7f0d29e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d240cf660 0x7f0d24154d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:19.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.131+0000 7f0d29e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2415ef30 con 0x7f0d24071790 2024-03-24T01:25:19.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.131+0000 7f0d29e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2415ec50 con 0x7f0d240cf660 2024-03-24T01:25:19.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.131+0000 7f0d29e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d24158530 con 0x7f0d240ae710 2024-03-24T01:25:19.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.131+0000 7f0d28a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d24071790 0x7f0d241556d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:19.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.131+0000 7f0d28a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d24071790 0x7f0d241556d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46494/0 (socket says 172.21.15.90:46494) 2024-03-24T01:25:19.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.131+0000 7f0d28a00640 1 -- 172.21.15.90:0/1975913193 learned_addr learned my addr 172.21.15.90:0/1975913193 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:19.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.131+0000 7f0d28a00640 1 -- 172.21.15.90:0/1975913193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d240ae710 msgr2=0x7f0d24158ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:19.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.131+0000 7f0d28a00640 1 --2- 172.21.15.90:0/1975913193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d240ae710 0x7f0d24158ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:19.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.131+0000 7f0d28a00640 1 -- 172.21.15.90:0/1975913193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d240cf660 msgr2=0x7f0d24154d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:19.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.131+0000 7f0d28a00640 1 --2- 172.21.15.90:0/1975913193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d240cf660 0x7f0d24154d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:19.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.131+0000 7f0d28a00640 1 -- 172.21.15.90:0/1975913193 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d24153cc0 con 0x7f0d24071790 2024-03-24T01:25:19.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.132+0000 7f0d28a00640 1 --2- 172.21.15.90:0/1975913193 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d24071790 0x7f0d241556d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c00c7f0 tx=0x7f0d0c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:19.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.132+0000 7f0d21600640 1 -- 172.21.15.90:0/1975913193 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c015070 con 0x7f0d24071790 2024-03-24T01:25:19.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.132+0000 7f0d21600640 1 -- 172.21.15.90:0/1975913193 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d0c007a30 con 0x7f0d24071790 2024-03-24T01:25:19.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.132+0000 7f0d21600640 1 -- 172.21.15.90:0/1975913193 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c007e00 con 0x7f0d24071790 2024-03-24T01:25:19.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.132+0000 7f0d29e00640 1 -- 172.21.15.90:0/1975913193 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d24153fa0 con 0x7f0d24071790 2024-03-24T01:25:19.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.132+0000 7f0d29e00640 1 -- 172.21.15.90:0/1975913193 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d24154560 con 0x7f0d24071790 2024-03-24T01:25:19.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.133+0000 7f0d29e00640 1 -- 172.21.15.90:0/1975913193 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d2404f400 con 0x7f0d24071790 2024-03-24T01:25:19.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.134+0000 7f0d21600640 1 -- 172.21.15.90:0/1975913193 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0d0c018070 con 0x7f0d24071790 2024-03-24T01:25:19.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.135+0000 7f0d21600640 1 --2- 172.21.15.90:0/1975913193 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0d0006bbd0 0x7f0d0006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:19.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.135+0000 7f0d21600640 1 -- 172.21.15.90:0/1975913193 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f0d0c08af00 con 0x7f0d24071790 2024-03-24T01:25:19.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.136+0000 7f0d1be00640 1 --2- 172.21.15.90:0/1975913193 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0d0006bbd0 0x7f0d0006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:19.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.137+0000 7f0d1be00640 1 --2- 172.21.15.90:0/1975913193 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0d0006bbd0 0x7f0d0006e090 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0d14036000 tx=0x7f0d14030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:19.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.142+0000 7f0d21600640 1 -- 172.21.15.90:0/1975913193 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0d0c059530 con 0x7f0d24071790 2024-03-24T01:25:19.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.340+0000 7f0d29e00640 1 -- 172.21.15.90:0/1975913193 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0d24154a00 con 0x7f0d24071790 2024-03-24T01:25:19.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.341+0000 7f0d21600640 1 -- 172.21.15.90:0/1975913193 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0d0c058e30 con 0x7f0d24071790 2024-03-24T01:25:19.343 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:25:19.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.343+0000 7f0d29e00640 1 -- 172.21.15.90:0/1975913193 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0d0006bbd0 msgr2=0x7f0d0006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:19.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.343+0000 7f0d29e00640 1 --2- 172.21.15.90:0/1975913193 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0d0006bbd0 0x7f0d0006e090 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0d14036000 tx=0x7f0d14030000 comp rx=0 tx=0).stop 2024-03-24T01:25:19.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.344+0000 7f0d29e00640 1 -- 172.21.15.90:0/1975913193 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d24071790 msgr2=0x7f0d241556d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:19.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.344+0000 7f0d29e00640 1 --2- 172.21.15.90:0/1975913193 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d24071790 0x7f0d241556d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c00c7f0 tx=0x7f0d0c00cbf0 comp rx=0 tx=0).stop 2024-03-24T01:25:19.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.344+0000 7f0d29e00640 1 -- 172.21.15.90:0/1975913193 shutdown_connections 2024-03-24T01:25:19.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.344+0000 7f0d29e00640 1 --2- 172.21.15.90:0/1975913193 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0d0006bbd0 0x7f0d0006e090 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:19.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.344+0000 7f0d29e00640 1 --2- 172.21.15.90:0/1975913193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d240cf660 0x7f0d24154d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:19.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.344+0000 7f0d29e00640 1 --2- 172.21.15.90:0/1975913193 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d240ae710 0x7f0d24158ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:19.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.344+0000 7f0d29e00640 1 --2- 172.21.15.90:0/1975913193 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d24071790 0x7f0d241556d0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:19.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.344+0000 7f0d29e00640 1 -- 172.21.15.90:0/1975913193 >> 172.21.15.90:0/1975913193 conn(0x7f0d2406b0c0 msgr2=0x7f0d24071b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:19.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.344+0000 7f0d29e00640 1 -- 172.21.15.90:0/1975913193 shutdown_connections 2024-03-24T01:25:19.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.344+0000 7f0d29e00640 1 -- 172.21.15.90:0/1975913193 wait complete. 2024-03-24T01:25:19.364 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T01:25:19.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.574+0000 7f35e7e00640 1 Processor -- start 2024-03-24T01:25:19.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.574+0000 7f35e7e00640 1 -- start start 2024-03-24T01:25:19.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.576+0000 7f35e7e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f35e006b7d0 0x7f35e00cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:19.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.576+0000 7f35e7e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e006bca0 con 0x7f35e0159620 2024-03-24T01:25:19.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.576+0000 7f35e7e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e00b36f0 con 0x7f35e00cef80 2024-03-24T01:25:19.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.576+0000 7f35e7e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35e00b4470 con 0x7f35e006b7d0 2024-03-24T01:25:19.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.576+0000 7f35e4a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f35e006b7d0 0x7f35e00cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:19.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.576+0000 7f35e4a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f35e006b7d0 0x7f35e00cea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46496/0 (socket says 172.21.15.90:46496) 2024-03-24T01:25:19.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.576+0000 7f35e4a00640 1 -- 172.21.15.90:0/1510558649 learned_addr learned my addr 172.21.15.90:0/1510558649 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:19.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.576+0000 7f35e5400640 1 --1- 172.21.15.90:0/1510558649 >> v1:172.21.15.90:6789/0 conn(0x7f35e0159620 0x7f35e006b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:46730/0 (socket says 172.21.15.90:46730) 2024-03-24T01:25:19.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.577+0000 7f35e4a00640 1 -- 172.21.15.90:0/1510558649 >> v1:172.21.15.120:6789/0 conn(0x7f35e00cef80 legacy=0x7f35e00ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:19.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.577+0000 7f35e4a00640 1 -- 172.21.15.90:0/1510558649 >> v1:172.21.15.90:6789/0 conn(0x7f35e0159620 legacy=0x7f35e006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:19.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.577+0000 7f35e4a00640 1 -- 172.21.15.90:0/1510558649 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e00b3990 con 0x7f35e006b7d0 2024-03-24T01:25:19.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.578+0000 7f35e4a00640 1 --2- 172.21.15.90:0/1510558649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35e006b7d0 0x7f35e00cea40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f35cc009870 tx=0x7f35cc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=785057dee432471e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:19.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.578+0000 7f35dfe00640 1 -- 172.21.15.90:0/1510558649 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35cc034070 con 0x7f35e006b7d0 2024-03-24T01:25:19.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.578+0000 7f35dfe00640 1 -- 172.21.15.90:0/1510558649 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35cc030860 con 0x7f35e006b7d0 2024-03-24T01:25:19.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.578+0000 7f35dfe00640 1 -- 172.21.15.90:0/1510558649 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35cc030c30 con 0x7f35e006b7d0 2024-03-24T01:25:19.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.579+0000 7f35e7e00640 1 -- 172.21.15.90:0/1510558649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35e006b7d0 msgr2=0x7f35e00cea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:19.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.579+0000 7f35e7e00640 1 --2- 172.21.15.90:0/1510558649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35e006b7d0 0x7f35e00cea40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f35cc009870 tx=0x7f35cc02b9a0 comp rx=0 tx=0).stop 2024-03-24T01:25:19.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.579+0000 7f35e7e00640 1 -- 172.21.15.90:0/1510558649 shutdown_connections 2024-03-24T01:25:19.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.579+0000 7f35e7e00640 1 --2- 172.21.15.90:0/1510558649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35e006b7d0 0x7f35e00cea40 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:19.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.579+0000 7f35e7e00640 1 -- 172.21.15.90:0/1510558649 >> 172.21.15.90:0/1510558649 conn(0x7f35e00b1770 msgr2=0x7f35e00b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:19.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.579+0000 7f35e7e00640 1 -- 172.21.15.90:0/1510558649 shutdown_connections 2024-03-24T01:25:19.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.579+0000 7f35e7e00640 1 -- 172.21.15.90:0/1510558649 wait complete. 2024-03-24T01:25:19.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.581+0000 7f35e7e00640 1 Processor -- start 2024-03-24T01:25:19.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.581+0000 7f35e7e00640 1 -- start start 2024-03-24T01:25:19.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.582+0000 7f35e7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35e006b7d0 0x7f35e01519e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:19.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.582+0000 7f35e7e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35e00cef80 0x7f35e006f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:19.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.582+0000 7f35e7e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35e0159620 0x7f35e00c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:19.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.582+0000 7f35e7e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e00cf430 con 0x7f35e006b7d0 2024-03-24T01:25:19.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.582+0000 7f35e7e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e00b3f50 con 0x7f35e00cef80 2024-03-24T01:25:19.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.582+0000 7f35e7e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35e0159a00 con 0x7f35e0159620 2024-03-24T01:25:19.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.582+0000 7f35e5400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35e006b7d0 0x7f35e01519e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:19.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.582+0000 7f35e5400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35e006b7d0 0x7f35e01519e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46504/0 (socket says 172.21.15.90:46504) 2024-03-24T01:25:19.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.582+0000 7f35e5400640 1 -- 172.21.15.90:0/2873460424 learned_addr learned my addr 172.21.15.90:0/2873460424 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:19.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.583+0000 7f35e5400640 1 -- 172.21.15.90:0/2873460424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35e0159620 msgr2=0x7f35e00c8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:19.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.583+0000 7f35e5400640 1 --2- 172.21.15.90:0/2873460424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35e0159620 0x7f35e00c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:19.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.583+0000 7f35e5400640 1 -- 172.21.15.90:0/2873460424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35e00cef80 msgr2=0x7f35e006f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:19.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.583+0000 7f35e4a00640 1 --2- 172.21.15.90:0/2873460424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35e00cef80 0x7f35e006f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:19.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.583+0000 7f35e5400640 1 --2- 172.21.15.90:0/2873460424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35e00cef80 0x7f35e006f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:19.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.583+0000 7f35e5400640 1 -- 172.21.15.90:0/2873460424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e006e430 con 0x7f35e006b7d0 2024-03-24T01:25:19.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.583+0000 7f35e5400640 1 --2- 172.21.15.90:0/2873460424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35e006b7d0 0x7f35e01519e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f35d400c3e0 tx=0x7f35d400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:19.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.583+0000 7f35dd600640 1 -- 172.21.15.90:0/2873460424 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d40074e0 con 0x7f35e006b7d0 2024-03-24T01:25:19.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.583+0000 7f35dd600640 1 -- 172.21.15.90:0/2873460424 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35d4007680 con 0x7f35e006b7d0 2024-03-24T01:25:19.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.583+0000 7f35e7e00640 1 -- 172.21.15.90:0/2873460424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35e006e710 con 0x7f35e006b7d0 2024-03-24T01:25:19.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.583+0000 7f35dd600640 1 -- 172.21.15.90:0/2873460424 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d40049e0 con 0x7f35e006b7d0 2024-03-24T01:25:19.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.583+0000 7f35e7e00640 1 -- 172.21.15.90:0/2873460424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35e006ecd0 con 0x7f35e006b7d0 2024-03-24T01:25:19.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.584+0000 7f35e7e00640 1 -- 172.21.15.90:0/2873460424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35e004f400 con 0x7f35e006b7d0 2024-03-24T01:25:19.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.586+0000 7f35dd600640 1 -- 172.21.15.90:0/2873460424 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f35d401a020 con 0x7f35e006b7d0 2024-03-24T01:25:19.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.587+0000 7f35dd600640 1 --2- 172.21.15.90:0/2873460424 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f35c006bc00 0x7f35c006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:19.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.587+0000 7f35dd600640 1 -- 172.21.15.90:0/2873460424 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f35d408a620 con 0x7f35e006b7d0 2024-03-24T01:25:19.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.588+0000 7f35e4a00640 1 --2- 172.21.15.90:0/2873460424 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f35c006bc00 0x7f35c006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:19.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.589+0000 7f35e4a00640 1 --2- 172.21.15.90:0/2873460424 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f35c006bc00 0x7f35c006e0c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f35cc036000 tx=0x7f35cc03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:19.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.592+0000 7f35dd600640 1 -- 172.21.15.90:0/2873460424 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f35d4010300 con 0x7f35e006b7d0 2024-03-24T01:25:19.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.793+0000 7f35e7e00640 1 -- 172.21.15.90:0/2873460424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f35e015d900 con 0x7f35e006b7d0 2024-03-24T01:25:19.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.794+0000 7f35dd600640 1 -- 172.21.15.90:0/2873460424 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f35d4058ce0 con 0x7f35e006b7d0 2024-03-24T01:25:19.796 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:25:19.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.797+0000 7f35e7e00640 1 -- 172.21.15.90:0/2873460424 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f35c006bc00 msgr2=0x7f35c006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:19.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.797+0000 7f35e7e00640 1 --2- 172.21.15.90:0/2873460424 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f35c006bc00 0x7f35c006e0c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f35cc036000 tx=0x7f35cc03b040 comp rx=0 tx=0).stop 2024-03-24T01:25:19.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.797+0000 7f35e7e00640 1 -- 172.21.15.90:0/2873460424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35e006b7d0 msgr2=0x7f35e01519e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:19.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.797+0000 7f35e7e00640 1 --2- 172.21.15.90:0/2873460424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35e006b7d0 0x7f35e01519e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f35d400c3e0 tx=0x7f35d400c8a0 comp rx=0 tx=0).stop 2024-03-24T01:25:19.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.797+0000 7f35e7e00640 1 -- 172.21.15.90:0/2873460424 shutdown_connections 2024-03-24T01:25:19.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.797+0000 7f35e7e00640 1 --2- 172.21.15.90:0/2873460424 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f35c006bc00 0x7f35c006e0c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:19.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.797+0000 7f35e7e00640 1 --2- 172.21.15.90:0/2873460424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35e0159620 0x7f35e00c8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:19.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.797+0000 7f35e7e00640 1 --2- 172.21.15.90:0/2873460424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35e00cef80 0x7f35e006f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:19.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.797+0000 7f35e7e00640 1 --2- 172.21.15.90:0/2873460424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35e006b7d0 0x7f35e01519e0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:19.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.797+0000 7f35e7e00640 1 -- 172.21.15.90:0/2873460424 >> 172.21.15.90:0/2873460424 conn(0x7f35e00b1770 msgr2=0x7f35e00b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:19.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.797+0000 7f35e7e00640 1 -- 172.21.15.90:0/2873460424 shutdown_connections 2024-03-24T01:25:19.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:19.797+0000 7f35e7e00640 1 -- 172.21.15.90:0/2873460424 wait complete. 2024-03-24T01:25:19.817 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T01:25:20.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.029+0000 7fcb89200640 1 Processor -- start 2024-03-24T01:25:20.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.029+0000 7fcb89200640 1 -- start start 2024-03-24T01:25:20.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.030+0000 7fcb89200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcb840720a0 0x7fcb840b3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:20.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.031+0000 7fcb89200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcb8406b4a0 0x7fcb840b4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:20.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.031+0000 7fcb89200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb840b4950 con 0x7fcb8406b0c0 2024-03-24T01:25:20.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.031+0000 7fcb89200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb8406ebd0 con 0x7fcb8406b4a0 2024-03-24T01:25:20.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.031+0000 7fcb89200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb8406db30 con 0x7fcb840720a0 2024-03-24T01:25:20.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.031+0000 7fcb82800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcb8406b4a0 0x7fcb840b4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:20.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.031+0000 7fcb82800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcb8406b4a0 0x7fcb840b4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42980/0 (socket says 172.21.15.90:42980) 2024-03-24T01:25:20.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.031+0000 7fcb82800640 1 -- 172.21.15.90:0/2236401445 learned_addr learned my addr 172.21.15.90:0/2236401445 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:20.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.031+0000 7fcb82800640 1 -- 172.21.15.90:0/2236401445 >> v2:172.21.15.120:3300/0 conn(0x7fcb840720a0 msgr2=0x7fcb840b3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:20.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.031+0000 7fcb82800640 1 --2- 172.21.15.90:0/2236401445 >> v2:172.21.15.120:3300/0 conn(0x7fcb840720a0 0x7fcb840b3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:20.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.031+0000 7fcb82800640 1 -- 172.21.15.90:0/2236401445 >> v1:172.21.15.120:6789/0 conn(0x7fcb8406b0c0 legacy=0x7fcb84071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:20.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.031+0000 7fcb82800640 1 -- 172.21.15.90:0/2236401445 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb8406de10 con 0x7fcb8406b4a0 2024-03-24T01:25:20.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.032+0000 7fcb82800640 1 --2- 172.21.15.90:0/2236401445 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb8406b4a0 0x7fcb840b4410 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fcb840af4d0 tx=0x7fcb7c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0666be9c6d7664f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:20.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.033+0000 7fcb80a00640 1 -- 172.21.15.90:0/2236401445 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c02e070 con 0x7fcb8406b4a0 2024-03-24T01:25:20.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.033+0000 7fcb80a00640 1 -- 172.21.15.90:0/2236401445 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb7c037070 con 0x7fcb8406b4a0 2024-03-24T01:25:20.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.033+0000 7fcb80a00640 1 -- 172.21.15.90:0/2236401445 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c031a00 con 0x7fcb8406b4a0 2024-03-24T01:25:20.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.033+0000 7fcb89200640 1 -- 172.21.15.90:0/2236401445 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb8406b4a0 msgr2=0x7fcb840b4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:20.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.033+0000 7fcb89200640 1 --2- 172.21.15.90:0/2236401445 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb8406b4a0 0x7fcb840b4410 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fcb840af4d0 tx=0x7fcb7c02b880 comp rx=0 tx=0).stop 2024-03-24T01:25:20.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.033+0000 7fcb89200640 1 -- 172.21.15.90:0/2236401445 shutdown_connections 2024-03-24T01:25:20.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.033+0000 7fcb89200640 1 --2- 172.21.15.90:0/2236401445 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb8406b4a0 0x7fcb840b4410 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:20.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.033+0000 7fcb89200640 1 --2- 172.21.15.90:0/2236401445 >> v2:172.21.15.120:3300/0 conn(0x7fcb840720a0 0x7fcb840b3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:20.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.033+0000 7fcb89200640 1 -- 172.21.15.90:0/2236401445 >> 172.21.15.90:0/2236401445 conn(0x7fcb840b1770 msgr2=0x7fcb840b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:20.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.033+0000 7fcb89200640 1 -- 172.21.15.90:0/2236401445 shutdown_connections 2024-03-24T01:25:20.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.034+0000 7fcb89200640 1 -- 172.21.15.90:0/2236401445 wait complete. 2024-03-24T01:25:20.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.035+0000 7fcb89200640 1 Processor -- start 2024-03-24T01:25:20.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.035+0000 7fcb89200640 1 -- start start 2024-03-24T01:25:20.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.036+0000 7fcb89200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb8406b0c0 0x7fcb84147280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:20.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.036+0000 7fcb89200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb8406b4a0 0x7fcb8414c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:20.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.036+0000 7fcb89200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb840720a0 0x7fcb8414cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:20.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.036+0000 7fcb89200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8406e6b0 con 0x7fcb8406b4a0 2024-03-24T01:25:20.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.036+0000 7fcb89200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8406e3d0 con 0x7fcb8406b0c0 2024-03-24T01:25:20.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.036+0000 7fcb89200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb84164790 con 0x7fcb840720a0 2024-03-24T01:25:20.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.036+0000 7fcb81400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb8406b4a0 0x7fcb8414c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:20.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.036+0000 7fcb81400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb8406b4a0 0x7fcb8414c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42982/0 (socket says 172.21.15.90:42982) 2024-03-24T01:25:20.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.036+0000 7fcb81400640 1 -- 172.21.15.90:0/4022303347 learned_addr learned my addr 172.21.15.90:0/4022303347 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:20.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.037+0000 7fcb81400640 1 -- 172.21.15.90:0/4022303347 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb840720a0 msgr2=0x7fcb8414cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:20.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.037+0000 7fcb81400640 1 --2- 172.21.15.90:0/4022303347 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb840720a0 0x7fcb8414cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:20.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.037+0000 7fcb81400640 1 -- 172.21.15.90:0/4022303347 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb8406b0c0 msgr2=0x7fcb84147280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:20.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.037+0000 7fcb81400640 1 --2- 172.21.15.90:0/4022303347 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb8406b0c0 0x7fcb84147280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:20.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.037+0000 7fcb81400640 1 -- 172.21.15.90:0/4022303347 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb8414d1e0 con 0x7fcb8406b4a0 2024-03-24T01:25:20.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.037+0000 7fcb81400640 1 --2- 172.21.15.90:0/4022303347 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb8406b4a0 0x7fcb8414c670 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fcb7000b440 tx=0x7fcb7000b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:20.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.037+0000 7fcb7a000640 1 -- 172.21.15.90:0/4022303347 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb70015020 con 0x7fcb8406b4a0 2024-03-24T01:25:20.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.037+0000 7fcb7a000640 1 -- 172.21.15.90:0/4022303347 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb70011890 con 0x7fcb8406b4a0 2024-03-24T01:25:20.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.037+0000 7fcb89200640 1 -- 172.21.15.90:0/4022303347 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb8414d4c0 con 0x7fcb8406b4a0 2024-03-24T01:25:20.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.037+0000 7fcb7a000640 1 -- 172.21.15.90:0/4022303347 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb70011c80 con 0x7fcb8406b4a0 2024-03-24T01:25:20.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.038+0000 7fcb89200640 1 -- 172.21.15.90:0/4022303347 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb8414da80 con 0x7fcb8406b4a0 2024-03-24T01:25:20.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.039+0000 7fcb89200640 1 -- 172.21.15.90:0/4022303347 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb8404f400 con 0x7fcb8406b4a0 2024-03-24T01:25:20.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.040+0000 7fcb7a000640 1 -- 172.21.15.90:0/4022303347 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcb70018020 con 0x7fcb8406b4a0 2024-03-24T01:25:20.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.040+0000 7fcb7a000640 1 --2- 172.21.15.90:0/4022303347 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcb6406bbd0 0x7fcb6406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:20.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.041+0000 7fcb7a000640 1 -- 172.21.15.90:0/4022303347 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7fcb7008b060 con 0x7fcb8406b4a0 2024-03-24T01:25:20.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.041+0000 7fcb81e00640 1 --2- 172.21.15.90:0/4022303347 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcb6406bbd0 0x7fcb6406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:20.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.042+0000 7fcb81e00640 1 --2- 172.21.15.90:0/4022303347 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcb6406bbd0 0x7fcb6406e090 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcb740043c0 tx=0x7fcb74007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:20.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.045+0000 7fcb7a000640 1 -- 172.21.15.90:0/4022303347 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcb7001f020 con 0x7fcb8406b4a0 2024-03-24T01:25:20.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.243+0000 7fcb89200640 1 -- 172.21.15.90:0/4022303347 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcb84147f00 con 0x7fcb8406b4a0 2024-03-24T01:25:20.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.244+0000 7fcb7a000640 1 -- 172.21.15.90:0/4022303347 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcb7000d040 con 0x7fcb8406b4a0 2024-03-24T01:25:20.245 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-24T01:25:20.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.246+0000 7fcb89200640 1 -- 172.21.15.90:0/4022303347 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcb6406bbd0 msgr2=0x7fcb6406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:20.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.246+0000 7fcb89200640 1 --2- 172.21.15.90:0/4022303347 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcb6406bbd0 0x7fcb6406e090 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcb740043c0 tx=0x7fcb74007d30 comp rx=0 tx=0).stop 2024-03-24T01:25:20.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.246+0000 7fcb89200640 1 -- 172.21.15.90:0/4022303347 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb8406b4a0 msgr2=0x7fcb8414c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:20.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.246+0000 7fcb89200640 1 --2- 172.21.15.90:0/4022303347 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb8406b4a0 0x7fcb8414c670 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fcb7000b440 tx=0x7fcb7000b900 comp rx=0 tx=0).stop 2024-03-24T01:25:20.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.246+0000 7fcb89200640 1 -- 172.21.15.90:0/4022303347 shutdown_connections 2024-03-24T01:25:20.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.246+0000 7fcb89200640 1 --2- 172.21.15.90:0/4022303347 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcb6406bbd0 0x7fcb6406e090 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:20.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.246+0000 7fcb89200640 1 --2- 172.21.15.90:0/4022303347 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb8406b4a0 0x7fcb8414c670 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:20.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.247+0000 7fcb89200640 1 --2- 172.21.15.90:0/4022303347 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcb840720a0 0x7fcb8414cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:20.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.247+0000 7fcb89200640 1 --2- 172.21.15.90:0/4022303347 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb8406b0c0 0x7fcb84147280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:20.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.247+0000 7fcb89200640 1 -- 172.21.15.90:0/4022303347 >> 172.21.15.90:0/4022303347 conn(0x7fcb840b1770 msgr2=0x7fcb84148420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:20.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.247+0000 7fcb89200640 1 -- 172.21.15.90:0/4022303347 shutdown_connections 2024-03-24T01:25:20.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.247+0000 7fcb89200640 1 -- 172.21.15.90:0/4022303347 wait complete. 2024-03-24T01:25:20.267 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T01:25:20.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.477+0000 7f3c82e00640 1 Processor -- start 2024-03-24T01:25:20.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.477+0000 7f3c82e00640 1 -- start start 2024-03-24T01:25:20.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.478+0000 7f3c82e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3c7c144990 0x7f3c7c06de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:20.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.478+0000 7f3c82e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3c7c144e10 0x7f3c7c06e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:20.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.478+0000 7f3c82e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c7c0cccc0 con 0x7f3c7c1445b0 2024-03-24T01:25:20.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.478+0000 7f3c82e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c7c0cca70 con 0x7f3c7c144990 2024-03-24T01:25:20.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.478+0000 7f3c82e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c7c0cf700 con 0x7f3c7c144e10 2024-03-24T01:25:20.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.479+0000 7f3c81000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3c7c144990 0x7f3c7c06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:20.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.479+0000 7f3c81a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f3c7c1445b0 0x7f3c7c0cfb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:42310/0 (socket says 172.21.15.90:42310) 2024-03-24T01:25:20.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.479+0000 7f3c81a00640 1 -- 172.21.15.90:0/674063531 learned_addr learned my addr 172.21.15.90:0/674063531 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:20.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.479+0000 7f3c82400640 1 --2- 172.21.15.90:0/674063531 >> v2:172.21.15.120:3300/0 conn(0x7f3c7c144e10 0x7f3c7c06e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:20.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.479+0000 7f3c81000640 1 -- 172.21.15.90:0/674063531 >> v2:172.21.15.120:3300/0 conn(0x7f3c7c144e10 msgr2=0x7f3c7c06e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:20.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.479+0000 7f3c81000640 1 --2- 172.21.15.90:0/674063531 >> v2:172.21.15.120:3300/0 conn(0x7f3c7c144e10 0x7f3c7c06e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:20.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.479+0000 7f3c81000640 1 -- 172.21.15.90:0/674063531 >> v1:172.21.15.90:6789/0 conn(0x7f3c7c1445b0 legacy=0x7f3c7c0cfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:20.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.479+0000 7f3c81000640 1 -- 172.21.15.90:0/674063531 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c7c0ccfa0 con 0x7f3c7c144990 2024-03-24T01:25:20.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.479+0000 7f3c82400640 1 --2- 172.21.15.90:0/674063531 >> v2:172.21.15.120:3300/0 conn(0x7f3c7c144e10 0x7f3c7c06e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:20.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.479+0000 7f3c7be00640 1 -- 172.21.15.90:0/674063531 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1585675994 0 0) 0x7f3c7c0cccc0 con 0x7f3c7c1445b0 2024-03-24T01:25:20.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.480+0000 7f3c81000640 1 --2- 172.21.15.90:0/674063531 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3c7c144990 0x7f3c7c06de50 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c0d0620 tx=0x7f3c6802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=468a9f74d3ac0421 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:20.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.481+0000 7f3c7be00640 1 -- 172.21.15.90:0/674063531 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c680315a0 con 0x7f3c7c144990 2024-03-24T01:25:20.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.481+0000 7f3c7be00640 1 -- 172.21.15.90:0/674063531 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c68037070 con 0x7f3c7c144990 2024-03-24T01:25:20.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.481+0000 7f3c7be00640 1 -- 172.21.15.90:0/674063531 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c68031960 con 0x7f3c7c144990 2024-03-24T01:25:20.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.481+0000 7f3c82e00640 1 -- 172.21.15.90:0/674063531 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3c7c144990 msgr2=0x7f3c7c06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:20.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.481+0000 7f3c82e00640 1 --2- 172.21.15.90:0/674063531 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3c7c144990 0x7f3c7c06de50 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c0d0620 tx=0x7f3c6802b840 comp rx=0 tx=0).stop 2024-03-24T01:25:20.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.481+0000 7f3c82e00640 1 -- 172.21.15.90:0/674063531 shutdown_connections 2024-03-24T01:25:20.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.481+0000 7f3c82e00640 1 --2- 172.21.15.90:0/674063531 >> v2:172.21.15.120:3300/0 conn(0x7f3c7c144e10 0x7f3c7c06e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:20.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.481+0000 7f3c82e00640 1 --2- 172.21.15.90:0/674063531 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3c7c144990 0x7f3c7c06de50 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:20.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.481+0000 7f3c82e00640 1 -- 172.21.15.90:0/674063531 >> 172.21.15.90:0/674063531 conn(0x7f3c7c06b0c0 msgr2=0x7f3c7c06b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:20.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.481+0000 7f3c82e00640 1 -- 172.21.15.90:0/674063531 shutdown_connections 2024-03-24T01:25:20.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.481+0000 7f3c82e00640 1 -- 172.21.15.90:0/674063531 wait complete. 2024-03-24T01:25:20.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.483+0000 7f3c82e00640 1 Processor -- start 2024-03-24T01:25:20.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.483+0000 7f3c82e00640 1 -- start start 2024-03-24T01:25:20.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.484+0000 7f3c82e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3c7c1445b0 0x7f3c7c13d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:20.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.484+0000 7f3c82e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c7c144990 0x7f3c7c13ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:20.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.484+0000 7f3c82e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c7c144e10 0x7f3c7c0b32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:20.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.484+0000 7f3c82e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c06e8d0 con 0x7f3c7c1445b0 2024-03-24T01:25:20.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.484+0000 7f3c82e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c0cd280 con 0x7f3c7c144990 2024-03-24T01:25:20.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.484+0000 7f3c82e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c06ea50 con 0x7f3c7c144e10 2024-03-24T01:25:20.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.484+0000 7f3c81a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3c7c1445b0 0x7f3c7c13d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:20.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.484+0000 7f3c81a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3c7c1445b0 0x7f3c7c13d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43004/0 (socket says 172.21.15.90:43004) 2024-03-24T01:25:20.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.484+0000 7f3c81a00640 1 -- 172.21.15.90:0/1644052163 learned_addr learned my addr 172.21.15.90:0/1644052163 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:20.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.485+0000 7f3c82400640 1 --2- 172.21.15.90:0/1644052163 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c7c144e10 0x7f3c7c0b32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:20.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.485+0000 7f3c81000640 1 --2- 172.21.15.90:0/1644052163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c7c144990 0x7f3c7c13ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:20.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.485+0000 7f3c81a00640 1 -- 172.21.15.90:0/1644052163 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c7c144e10 msgr2=0x7f3c7c0b32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:20.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.485+0000 7f3c81a00640 1 --2- 172.21.15.90:0/1644052163 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c7c144e10 0x7f3c7c0b32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:20.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.485+0000 7f3c81a00640 1 -- 172.21.15.90:0/1644052163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c7c144990 msgr2=0x7f3c7c13ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:20.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.485+0000 7f3c81a00640 1 --2- 172.21.15.90:0/1644052163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c7c144990 0x7f3c7c13ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:20.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.485+0000 7f3c81a00640 1 -- 172.21.15.90:0/1644052163 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c7c1f6f50 con 0x7f3c7c1445b0 2024-03-24T01:25:20.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.485+0000 7f3c82400640 1 --2- 172.21.15.90:0/1644052163 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c7c144e10 0x7f3c7c0b32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:20.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.485+0000 7f3c81a00640 1 --2- 172.21.15.90:0/1644052163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3c7c1445b0 0x7f3c7c13d130 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3c7000c4d0 tx=0x7f3c7000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:20.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.485+0000 7f3c79600640 1 -- 172.21.15.90:0/1644052163 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c70018070 con 0x7f3c7c1445b0 2024-03-24T01:25:20.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.485+0000 7f3c79600640 1 -- 172.21.15.90:0/1644052163 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c70007500 con 0x7f3c7c1445b0 2024-03-24T01:25:20.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.485+0000 7f3c79600640 1 -- 172.21.15.90:0/1644052163 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c70004a20 con 0x7f3c7c1445b0 2024-03-24T01:25:20.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.485+0000 7f3c82e00640 1 -- 172.21.15.90:0/1644052163 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c7c1f71f0 con 0x7f3c7c1445b0 2024-03-24T01:25:20.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.486+0000 7f3c82e00640 1 -- 172.21.15.90:0/1644052163 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c7c1f77b0 con 0x7f3c7c1445b0 2024-03-24T01:25:20.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.487+0000 7f3c82e00640 1 -- 172.21.15.90:0/1644052163 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c7c04f400 con 0x7f3c7c1445b0 2024-03-24T01:25:20.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.488+0000 7f3c79600640 1 -- 172.21.15.90:0/1644052163 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3c70004bc0 con 0x7f3c7c1445b0 2024-03-24T01:25:20.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.488+0000 7f3c79600640 1 --2- 172.21.15.90:0/1644052163 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3c5c06bbd0 0x7f3c5c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:20.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.489+0000 7f3c79600640 1 -- 172.21.15.90:0/1644052163 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f3c7008a270 con 0x7f3c7c1445b0 2024-03-24T01:25:20.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.489+0000 7f3c81000640 1 --2- 172.21.15.90:0/1644052163 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3c5c06bbd0 0x7f3c5c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:20.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.490+0000 7f3c81000640 1 --2- 172.21.15.90:0/1644052163 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3c5c06bbd0 0x7f3c5c06e090 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3c68036000 tx=0x7f3c68008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:20.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.493+0000 7f3c79600640 1 -- 172.21.15.90:0/1644052163 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3c700588a0 con 0x7f3c7c1445b0 2024-03-24T01:25:20.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.688+0000 7f3c82e00640 1 -- 172.21.15.90:0/1644052163 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f3c7c072370 con 0x7f3c7c1445b0 2024-03-24T01:25:20.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.689+0000 7f3c79600640 1 -- 172.21.15.90:0/1644052163 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f3c700581a0 con 0x7f3c7c1445b0 2024-03-24T01:25:20.691 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:25:20.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.692+0000 7f3c82e00640 1 -- 172.21.15.90:0/1644052163 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3c5c06bbd0 msgr2=0x7f3c5c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:20.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.692+0000 7f3c82e00640 1 --2- 172.21.15.90:0/1644052163 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3c5c06bbd0 0x7f3c5c06e090 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3c68036000 tx=0x7f3c68008030 comp rx=0 tx=0).stop 2024-03-24T01:25:20.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.692+0000 7f3c82e00640 1 -- 172.21.15.90:0/1644052163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3c7c1445b0 msgr2=0x7f3c7c13d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:20.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.692+0000 7f3c82e00640 1 --2- 172.21.15.90:0/1644052163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3c7c1445b0 0x7f3c7c13d130 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3c7000c4d0 tx=0x7f3c7000c990 comp rx=0 tx=0).stop 2024-03-24T01:25:20.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.692+0000 7f3c82e00640 1 -- 172.21.15.90:0/1644052163 shutdown_connections 2024-03-24T01:25:20.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.692+0000 7f3c82e00640 1 --2- 172.21.15.90:0/1644052163 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3c5c06bbd0 0x7f3c5c06e090 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:20.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.692+0000 7f3c82e00640 1 --2- 172.21.15.90:0/1644052163 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3c7c144e10 0x7f3c7c0b32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:20.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.692+0000 7f3c82e00640 1 --2- 172.21.15.90:0/1644052163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c7c144990 0x7f3c7c13ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:20.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.692+0000 7f3c82e00640 1 --2- 172.21.15.90:0/1644052163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3c7c1445b0 0x7f3c7c13d130 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:20.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.692+0000 7f3c82e00640 1 -- 172.21.15.90:0/1644052163 >> 172.21.15.90:0/1644052163 conn(0x7f3c7c06b0c0 msgr2=0x7f3c7c0ce710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:20.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.692+0000 7f3c82e00640 1 -- 172.21.15.90:0/1644052163 shutdown_connections 2024-03-24T01:25:20.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.692+0000 7f3c82e00640 1 -- 172.21.15.90:0/1644052163 wait complete. 2024-03-24T01:25:20.712 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T01:25:20.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.911+0000 7f049fa00640 1 Processor -- start 2024-03-24T01:25:20.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.911+0000 7f049fa00640 1 -- start start 2024-03-24T01:25:20.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.912+0000 7f049fa00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f049815de60 0x7f04980d1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:20.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.912+0000 7f049fa00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f04980cf1e0 0x7f04980b6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:20.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.912+0000 7f049fa00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04980d2790 con 0x7f04980b97b0 2024-03-24T01:25:20.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.912+0000 7f049fa00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04980d2a70 con 0x7f04980cf1e0 2024-03-24T01:25:20.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.912+0000 7f049fa00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04980d2d50 con 0x7f049815de60 2024-03-24T01:25:20.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.913+0000 7f049da00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f04980cf1e0 0x7f04980b6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:20.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.913+0000 7f049da00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f04980cf1e0 0x7f04980b6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43020/0 (socket says 172.21.15.90:43020) 2024-03-24T01:25:20.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.913+0000 7f049da00640 1 -- 172.21.15.90:0/2037541586 learned_addr learned my addr 172.21.15.90:0/2037541586 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:20.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.913+0000 7f049d000640 1 --2- 172.21.15.90:0/2037541586 >> v2:172.21.15.120:3300/0 conn(0x7f049815de60 0x7f04980d1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:20.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.913+0000 7f049da00640 1 -- 172.21.15.90:0/2037541586 >> v2:172.21.15.120:3300/0 conn(0x7f049815de60 msgr2=0x7f04980d1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:20.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.913+0000 7f049da00640 1 --2- 172.21.15.90:0/2037541586 >> v2:172.21.15.120:3300/0 conn(0x7f049815de60 0x7f04980d1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:20.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.913+0000 7f049da00640 1 -- 172.21.15.90:0/2037541586 >> v1:172.21.15.120:6789/0 conn(0x7f04980b97b0 legacy=0x7f04980cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:20.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.913+0000 7f049da00640 1 -- 172.21.15.90:0/2037541586 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04980d3030 con 0x7f04980cf1e0 2024-03-24T01:25:20.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.913+0000 7f049d000640 1 --2- 172.21.15.90:0/2037541586 >> v2:172.21.15.120:3300/0 conn(0x7f049815de60 0x7f04980d1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:20.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.915+0000 7f049da00640 1 --2- 172.21.15.90:0/2037541586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f04980cf1e0 0x7f04980b6ea0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0490009e50 tx=0x7f049002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98d60ec1376fb0c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:20.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.915+0000 7f0497400640 1 -- 172.21.15.90:0/2037541586 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f049002c9f0 con 0x7f04980cf1e0 2024-03-24T01:25:20.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.915+0000 7f0497400640 1 -- 172.21.15.90:0/2037541586 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f049002cb90 con 0x7f04980cf1e0 2024-03-24T01:25:20.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.915+0000 7f0497400640 1 -- 172.21.15.90:0/2037541586 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04900343f0 con 0x7f04980cf1e0 2024-03-24T01:25:20.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.916+0000 7f049fa00640 1 -- 172.21.15.90:0/2037541586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f04980cf1e0 msgr2=0x7f04980b6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:20.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.916+0000 7f049fa00640 1 --2- 172.21.15.90:0/2037541586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f04980cf1e0 0x7f04980b6ea0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0490009e50 tx=0x7f049002be50 comp rx=0 tx=0).stop 2024-03-24T01:25:20.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.916+0000 7f049fa00640 1 -- 172.21.15.90:0/2037541586 shutdown_connections 2024-03-24T01:25:20.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.916+0000 7f049fa00640 1 --2- 172.21.15.90:0/2037541586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f04980cf1e0 0x7f04980b6ea0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:20.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.916+0000 7f049fa00640 1 --2- 172.21.15.90:0/2037541586 >> v2:172.21.15.120:3300/0 conn(0x7f049815de60 0x7f04980d1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:20.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.916+0000 7f049fa00640 1 -- 172.21.15.90:0/2037541586 >> 172.21.15.90:0/2037541586 conn(0x7f0498072c10 msgr2=0x7f0498073020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:20.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.916+0000 7f049fa00640 1 -- 172.21.15.90:0/2037541586 shutdown_connections 2024-03-24T01:25:20.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.916+0000 7f049fa00640 1 -- 172.21.15.90:0/2037541586 wait complete. 2024-03-24T01:25:20.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.918+0000 7f049fa00640 1 Processor -- start 2024-03-24T01:25:20.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.918+0000 7f049fa00640 1 -- start start 2024-03-24T01:25:20.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.919+0000 7f049fa00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04980cf1e0 0x7f04981f6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:20.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.919+0000 7f049fa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f049815de60 0x7f04981f7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:20.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.919+0000 7f049fa00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04981f38e0 0x7f04981f3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:20.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.919+0000 7f049fa00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049815e3c0 con 0x7f049815de60 2024-03-24T01:25:20.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.919+0000 7f049fa00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049815e240 con 0x7f04981f38e0 2024-03-24T01:25:20.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.919+0000 7f049fa00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04980d2380 con 0x7f04980cf1e0 2024-03-24T01:25:20.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.920+0000 7f0497e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f049815de60 0x7f04981f7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:20.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.920+0000 7f0497e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f049815de60 0x7f04981f7410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43026/0 (socket says 172.21.15.90:43026) 2024-03-24T01:25:20.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.920+0000 7f0497e00640 1 -- 172.21.15.90:0/89500831 learned_addr learned my addr 172.21.15.90:0/89500831 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:20.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.920+0000 7f049d000640 1 --2- 172.21.15.90:0/89500831 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04980cf1e0 0x7f04981f6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:20.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.920+0000 7f049da00640 1 --2- 172.21.15.90:0/89500831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04981f38e0 0x7f04981f3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:20.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.920+0000 7f0497e00640 1 -- 172.21.15.90:0/89500831 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04980cf1e0 msgr2=0x7f04981f6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:20.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.920+0000 7f0497e00640 1 --2- 172.21.15.90:0/89500831 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04980cf1e0 0x7f04981f6ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:20.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.920+0000 7f0497e00640 1 -- 172.21.15.90:0/89500831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04981f38e0 msgr2=0x7f04981f3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:20.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.920+0000 7f0497e00640 1 --2- 172.21.15.90:0/89500831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04981f38e0 0x7f04981f3d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:20.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.920+0000 7f0497e00640 1 -- 172.21.15.90:0/89500831 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04981f4cf0 con 0x7f049815de60 2024-03-24T01:25:20.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.920+0000 7f0497e00640 1 --2- 172.21.15.90:0/89500831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f049815de60 0x7f04981f7410 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f048400c430 tx=0x7f048400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:20.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.921+0000 7f0494c00640 1 -- 172.21.15.90:0/89500831 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0484011790 con 0x7f049815de60 2024-03-24T01:25:20.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.921+0000 7f0494c00640 1 -- 172.21.15.90:0/89500831 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0484017070 con 0x7f049815de60 2024-03-24T01:25:20.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.921+0000 7f0494c00640 1 -- 172.21.15.90:0/89500831 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0484011bb0 con 0x7f049815de60 2024-03-24T01:25:20.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.921+0000 7f049fa00640 1 -- 172.21.15.90:0/89500831 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04981f4fd0 con 0x7f049815de60 2024-03-24T01:25:20.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.921+0000 7f049fa00640 1 -- 172.21.15.90:0/89500831 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04981f5590 con 0x7f049815de60 2024-03-24T01:25:20.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.922+0000 7f049fa00640 1 -- 172.21.15.90:0/89500831 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f049804f400 con 0x7f049815de60 2024-03-24T01:25:20.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.923+0000 7f0494c00640 1 -- 172.21.15.90:0/89500831 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f048401c460 con 0x7f049815de60 2024-03-24T01:25:20.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.924+0000 7f0494c00640 1 --2- 172.21.15.90:0/89500831 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f047806bbd0 0x7f047806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:20.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.924+0000 7f0494c00640 1 -- 172.21.15.90:0/89500831 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f048408ad80 con 0x7f049815de60 2024-03-24T01:25:20.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.924+0000 7f049d000640 1 --2- 172.21.15.90:0/89500831 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f047806bbd0 0x7f047806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:20.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.925+0000 7f049d000640 1 --2- 172.21.15.90:0/89500831 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f047806bbd0 0x7f047806e090 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f048c00a950 tx=0x7f048c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:20.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:20.928+0000 7f0494c00640 1 -- 172.21.15.90:0/89500831 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0484059440 con 0x7f049815de60 2024-03-24T01:25:21.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.141+0000 7f049fa00640 1 -- 172.21.15.90:0/89500831 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0498073e00 con 0x7f049815de60 2024-03-24T01:25:21.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.142+0000 7f0494c00640 1 -- 172.21.15.90:0/89500831 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f0484058d40 con 0x7f049815de60 2024-03-24T01:25:21.144 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-24T01:25:21.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.144+0000 7f049fa00640 1 -- 172.21.15.90:0/89500831 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f047806bbd0 msgr2=0x7f047806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:21.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.144+0000 7f049fa00640 1 --2- 172.21.15.90:0/89500831 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f047806bbd0 0x7f047806e090 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f048c00a950 tx=0x7f048c008040 comp rx=0 tx=0).stop 2024-03-24T01:25:21.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.145+0000 7f049fa00640 1 -- 172.21.15.90:0/89500831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f049815de60 msgr2=0x7f04981f7410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:21.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.145+0000 7f049fa00640 1 --2- 172.21.15.90:0/89500831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f049815de60 0x7f04981f7410 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f048400c430 tx=0x7f048400c8f0 comp rx=0 tx=0).stop 2024-03-24T01:25:21.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.145+0000 7f049fa00640 1 -- 172.21.15.90:0/89500831 shutdown_connections 2024-03-24T01:25:21.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.145+0000 7f049fa00640 1 --2- 172.21.15.90:0/89500831 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f047806bbd0 0x7f047806e090 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:21.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.145+0000 7f049fa00640 1 --2- 172.21.15.90:0/89500831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f049815de60 0x7f04981f7410 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:21.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.145+0000 7f049fa00640 1 --2- 172.21.15.90:0/89500831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04981f38e0 0x7f04981f3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:21.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.145+0000 7f049fa00640 1 --2- 172.21.15.90:0/89500831 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f04980cf1e0 0x7f04981f6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:21.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.145+0000 7f049fa00640 1 -- 172.21.15.90:0/89500831 >> 172.21.15.90:0/89500831 conn(0x7f0498072c10 msgr2=0x7f04980b8670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:21.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.145+0000 7f049fa00640 1 -- 172.21.15.90:0/89500831 shutdown_connections 2024-03-24T01:25:21.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.145+0000 7f049fa00640 1 -- 172.21.15.90:0/89500831 wait complete. 2024-03-24T01:25:21.164 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:25:21.164 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:25:21.217 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:25:21.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.422+0000 7f9672200640 1 Processor -- start 2024-03-24T01:25:21.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.422+0000 7f9672200640 1 -- start start 2024-03-24T01:25:21.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.423+0000 7f9672200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f966c0ce360 0x7f966c1666c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:21.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.424+0000 7f9672200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f966c0cc660 con 0x7f966c0cd6e0 2024-03-24T01:25:21.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.424+0000 7f9672200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f966c0cc920 con 0x7f966c1454e0 2024-03-24T01:25:21.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.424+0000 7f9672200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f966c0cc430 con 0x7f966c0ce360 2024-03-24T01:25:21.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.424+0000 7f966aa00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f966c0cd6e0 0x7f966c0cdc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:42318/0 (socket says 172.21.15.90:42318) 2024-03-24T01:25:21.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.424+0000 7f966aa00640 1 -- 172.21.15.90:0/1469238478 learned_addr learned my addr 172.21.15.90:0/1469238478 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:21.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.424+0000 7f966be00640 1 --2- 172.21.15.90:0/1469238478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f966c0ce360 0x7f966c1666c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:21.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.424+0000 7f966a000640 1 -- 172.21.15.90:0/1469238478 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2808883447 0 0) 0x7f966c0cc660 con 0x7f966c0cd6e0 2024-03-24T01:25:21.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.424+0000 7f966a000640 1 -- 172.21.15.90:0/1469238478 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9650003520 con 0x7f966c0cd6e0 2024-03-24T01:25:21.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.425+0000 7f966a000640 1 -- 172.21.15.90:0/1469238478 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 839672461 0 0) 0x7f9650003520 con 0x7f966c0cd6e0 2024-03-24T01:25:21.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.425+0000 7f966a000640 1 -- 172.21.15.90:0/1469238478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f966c0ce360 msgr2=0x7f966c1666c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:21.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.425+0000 7f966a000640 1 --2- 172.21.15.90:0/1469238478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f966c0ce360 0x7f966c1666c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:21.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.425+0000 7f966a000640 1 -- 172.21.15.90:0/1469238478 >> v1:172.21.15.120:6789/0 conn(0x7f966c1454e0 legacy=0x7f966c06f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:21.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.425+0000 7f966a000640 1 -- 172.21.15.90:0/1469238478 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f966c0ccc00 con 0x7f966c0cd6e0 2024-03-24T01:25:21.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.425+0000 7f966a000640 1 -- 172.21.15.90:0/1469238478 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f9658001670 con 0x7f966c0cd6e0 2024-03-24T01:25:21.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.425+0000 7f966a000640 1 -- 172.21.15.90:0/1469238478 >> v1:172.21.15.90:6789/0 conn(0x7f966c0cd6e0 legacy=0x7f966c0cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:21.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.425+0000 7f966a000640 1 --2- 172.21.15.90:0/1469238478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f966c0ce360 0x7f96500039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:21.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.425+0000 7f966a000640 1 --2- 172.21.15.90:0/1469238478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9650003f60 0x7f9650004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:21.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.425+0000 7f966a000640 1 --2- 172.21.15.90:0/1469238478 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9650004d40 0x7f96500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:21.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.425+0000 7f966a000640 1 -- 172.21.15.90:0/1469238478 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f965000ba70 con 0x7f9650004d40 2024-03-24T01:25:21.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.425+0000 7f966a000640 1 -- 172.21.15.90:0/1469238478 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f965000bc20 con 0x7f9650003f60 2024-03-24T01:25:21.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.425+0000 7f966a000640 1 -- 172.21.15.90:0/1469238478 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f965000be20 con 0x7f966c0ce360 2024-03-24T01:25:21.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.425+0000 7f966be00640 1 --2- 172.21.15.90:0/1469238478 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9650004d40 0x7f96500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:21.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.425+0000 7f966aa00640 1 --2- 172.21.15.90:0/1469238478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9650003f60 0x7f9650004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:21.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.425+0000 7f966b400640 1 --2- 172.21.15.90:0/1469238478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f966c0ce360 0x7f96500039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:21.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.426+0000 7f966be00640 1 -- 172.21.15.90:0/1469238478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f966c0ce360 msgr2=0x7f96500039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:21.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.426+0000 7f966be00640 1 --2- 172.21.15.90:0/1469238478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f966c0ce360 0x7f96500039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:21.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.426+0000 7f966be00640 1 -- 172.21.15.90:0/1469238478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9650003f60 msgr2=0x7f9650004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:21.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.426+0000 7f966be00640 1 --2- 172.21.15.90:0/1469238478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9650003f60 0x7f9650004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:21.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.426+0000 7f966be00640 1 -- 172.21.15.90:0/1469238478 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9650003520 con 0x7f9650004d40 2024-03-24T01:25:21.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.426+0000 7f966aa00640 1 --2- 172.21.15.90:0/1469238478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9650003f60 0x7f9650004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:21.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.426+0000 7f966b400640 1 --2- 172.21.15.90:0/1469238478 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f966c0ce360 0x7f96500039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:21.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.427+0000 7f966be00640 1 --2- 172.21.15.90:0/1469238478 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9650004d40 0x7f96500051c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f9664009f90 tx=0x7f9650005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f34f445764e65381 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:21.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.427+0000 7f966a000640 1 -- 172.21.15.90:0/1469238478 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9664039070 con 0x7f9650004d40 2024-03-24T01:25:21.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.427+0000 7f966a000640 1 -- 172.21.15.90:0/1469238478 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f966402cbd0 con 0x7f9650004d40 2024-03-24T01:25:21.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.427+0000 7f966a000640 1 -- 172.21.15.90:0/1469238478 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9664009c10 con 0x7f9650004d40 2024-03-24T01:25:21.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.427+0000 7f9672200640 1 -- 172.21.15.90:0/1469238478 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9650004d40 msgr2=0x7f96500051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:21.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.427+0000 7f9672200640 1 --2- 172.21.15.90:0/1469238478 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9650004d40 0x7f96500051c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f9664009f90 tx=0x7f9650005830 comp rx=0 tx=0).stop 2024-03-24T01:25:21.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.427+0000 7f966be00640 1 -- 172.21.15.90:0/1469238478 reap_dead start 2024-03-24T01:25:21.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.428+0000 7f9672200640 1 -- 172.21.15.90:0/1469238478 shutdown_connections 2024-03-24T01:25:21.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.428+0000 7f9672200640 1 -- 172.21.15.90:0/1469238478 >> 172.21.15.90:0/1469238478 conn(0x7f966c06b2a0 msgr2=0x7f966c06b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:21.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.428+0000 7f9672200640 1 -- 172.21.15.90:0/1469238478 shutdown_connections 2024-03-24T01:25:21.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.428+0000 7f9672200640 1 -- 172.21.15.90:0/1469238478 wait complete. 2024-03-24T01:25:21.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.429+0000 7f9672200640 1 Processor -- start 2024-03-24T01:25:21.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.430+0000 7f9672200640 1 -- start start 2024-03-24T01:25:21.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.430+0000 7f9672200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9650004d40 0x7f966c1381a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:21.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.431+0000 7f9672200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f966c138810 0x7f966c158e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:21.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.431+0000 7f9672200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f966c1593b0 0x7f966c159860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:21.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.431+0000 7f9672200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966c06f710 con 0x7f966c1593b0 2024-03-24T01:25:21.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.431+0000 7f9672200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966c0cd1c0 con 0x7f966c138810 2024-03-24T01:25:21.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.431+0000 7f9672200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f966c06f890 con 0x7f9650004d40 2024-03-24T01:25:21.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.431+0000 7f966be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f966c1593b0 0x7f966c159860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:21.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.431+0000 7f966be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f966c1593b0 0x7f966c159860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43036/0 (socket says 172.21.15.90:43036) 2024-03-24T01:25:21.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.431+0000 7f966be00640 1 -- 172.21.15.90:0/2135623131 learned_addr learned my addr 172.21.15.90:0/2135623131 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:21.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.431+0000 7f966b400640 1 --2- 172.21.15.90:0/2135623131 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9650004d40 0x7f966c1381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:21.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.431+0000 7f966be00640 1 -- 172.21.15.90:0/2135623131 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9650004d40 msgr2=0x7f966c1381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:21.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.431+0000 7f966be00640 1 --2- 172.21.15.90:0/2135623131 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9650004d40 0x7f966c1381a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:21.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.431+0000 7f966be00640 1 -- 172.21.15.90:0/2135623131 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f966c138810 msgr2=0x7f966c158e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:21.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.431+0000 7f966be00640 1 --2- 172.21.15.90:0/2135623131 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f966c138810 0x7f966c158e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:21.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.431+0000 7f966be00640 1 -- 172.21.15.90:0/2135623131 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f966c13a360 con 0x7f966c1593b0 2024-03-24T01:25:21.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.432+0000 7f966be00640 1 --2- 172.21.15.90:0/2135623131 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f966c1593b0 0x7f966c159860 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f9664009f90 tx=0x7f9664009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:21.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.432+0000 7f9663400640 1 -- 172.21.15.90:0/2135623131 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9664042030 con 0x7f966c1593b0 2024-03-24T01:25:21.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.432+0000 7f9663400640 1 -- 172.21.15.90:0/2135623131 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f966403a040 con 0x7f966c1593b0 2024-03-24T01:25:21.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.432+0000 7f9663400640 1 -- 172.21.15.90:0/2135623131 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9664039070 con 0x7f966c1593b0 2024-03-24T01:25:21.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.432+0000 7f9672200640 1 -- 172.21.15.90:0/2135623131 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f966c157dd0 con 0x7f966c1593b0 2024-03-24T01:25:21.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.432+0000 7f9672200640 1 -- 172.21.15.90:0/2135623131 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f966c158390 con 0x7f966c1593b0 2024-03-24T01:25:21.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.433+0000 7f9672200640 1 -- 172.21.15.90:0/2135623131 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f966c158560 con 0x7f966c1593b0 2024-03-24T01:25:21.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.434+0000 7f9663400640 1 -- 172.21.15.90:0/2135623131 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9664045070 con 0x7f966c1593b0 2024-03-24T01:25:21.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.436+0000 7f9663400640 1 --2- 172.21.15.90:0/2135623131 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f964c06bbd0 0x7f964c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:21.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.436+0000 7f9663400640 1 -- 172.21.15.90:0/2135623131 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f96640aca70 con 0x7f966c1593b0 2024-03-24T01:25:21.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.436+0000 7f966b400640 1 --2- 172.21.15.90:0/2135623131 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f964c06bbd0 0x7f964c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:21.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.437+0000 7f966b400640 1 --2- 172.21.15.90:0/2135623131 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f964c06bbd0 0x7f964c06e090 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f965c007fd0 tx=0x7f965c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:21.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.441+0000 7f9663400640 1 -- 172.21.15.90:0/2135623131 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f966404c020 con 0x7f966c1593b0 2024-03-24T01:25:21.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.646+0000 7f9672200640 1 -- 172.21.15.90:0/2135623131 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f966c0cd400 con 0x7f966c1593b0 2024-03-24T01:25:21.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.649+0000 7f9663400640 1 -- 172.21.15.90:0/2135623131 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18925 (secure 0 0 0) 0x7f966407b130 con 0x7f966c1593b0 2024-03-24T01:25:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:25:21.651 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:24:53.236401+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:24:08.643966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T01:24:09.189904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T01:24:13.027323+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:25:21.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.652+0000 7f9672200640 1 -- 172.21.15.90:0/2135623131 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f964c06bbd0 msgr2=0x7f964c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:21.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.652+0000 7f9672200640 1 --2- 172.21.15.90:0/2135623131 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f964c06bbd0 0x7f964c06e090 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f965c007fd0 tx=0x7f965c009040 comp rx=0 tx=0).stop 2024-03-24T01:25:21.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.652+0000 7f9672200640 1 -- 172.21.15.90:0/2135623131 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f966c1593b0 msgr2=0x7f966c159860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:21.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.652+0000 7f9672200640 1 --2- 172.21.15.90:0/2135623131 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f966c1593b0 0x7f966c159860 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f9664009f90 tx=0x7f9664009750 comp rx=0 tx=0).stop 2024-03-24T01:25:21.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.652+0000 7f9672200640 1 -- 172.21.15.90:0/2135623131 shutdown_connections 2024-03-24T01:25:21.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.652+0000 7f9672200640 1 --2- 172.21.15.90:0/2135623131 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f964c06bbd0 0x7f964c06e090 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:21.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.652+0000 7f9672200640 1 --2- 172.21.15.90:0/2135623131 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f966c1593b0 0x7f966c159860 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:21.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.652+0000 7f9672200640 1 --2- 172.21.15.90:0/2135623131 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f966c138810 0x7f966c158e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:21.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.652+0000 7f9672200640 1 --2- 172.21.15.90:0/2135623131 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9650004d40 0x7f966c1381a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:21.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.652+0000 7f9672200640 1 -- 172.21.15.90:0/2135623131 >> 172.21.15.90:0/2135623131 conn(0x7f966c06b2a0 msgr2=0x7f966c15d0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:21.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.652+0000 7f9672200640 1 -- 172.21.15.90:0/2135623131 shutdown_connections 2024-03-24T01:25:21.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.652+0000 7f9672200640 1 -- 172.21.15.90:0/2135623131 wait complete. 2024-03-24T01:25:21.673 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T01:24:08.643966+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T01:24:09.189904+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T01:24:13.027323+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T01:25:21.674 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:25:21.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.888+0000 7f536d600640 1 Processor -- start 2024-03-24T01:25:21.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.888+0000 7f536d600640 1 -- start start 2024-03-24T01:25:21.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.890+0000 7f536d600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f53680cf150 0x7f53680ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:21.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.890+0000 7f536d600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f536806bca0 con 0x7f536806b7d0 2024-03-24T01:25:21.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.890+0000 7f536d600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53680b36f0 con 0x7f5368159620 2024-03-24T01:25:21.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.890+0000 7f536d600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53680b4470 con 0x7f53680cf150 2024-03-24T01:25:21.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.890+0000 7f5367400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f53680cf150 0x7f53680ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:21.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.890+0000 7f5366000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f536806b7d0 0x7f53680cea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:42320/0 (socket says 172.21.15.90:42320) 2024-03-24T01:25:21.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.890+0000 7f5366000640 1 -- 172.21.15.90:0/3400967220 learned_addr learned my addr 172.21.15.90:0/3400967220 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:21.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.891+0000 7f5367400640 1 -- 172.21.15.90:0/3400967220 >> v1:172.21.15.120:6789/0 conn(0x7f5368159620 legacy=0x7f536806b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:21.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.891+0000 7f5367400640 1 -- 172.21.15.90:0/3400967220 >> v1:172.21.15.90:6789/0 conn(0x7f536806b7d0 legacy=0x7f53680cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:21.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.891+0000 7f5367400640 1 -- 172.21.15.90:0/3400967220 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53680b3990 con 0x7f53680cf150 2024-03-24T01:25:21.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.891+0000 7f5365600640 1 -- 172.21.15.90:0/3400967220 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3294396970 0 0) 0x7f536806bca0 con 0x7f536806b7d0 2024-03-24T01:25:21.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.892+0000 7f5367400640 1 --2- 172.21.15.90:0/3400967220 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53680cf150 0x7f53680ae530 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f53600097a0 tx=0x7f536002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27042186e45a7158 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:21.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.892+0000 7f5365600640 1 -- 172.21.15.90:0/3400967220 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53600315a0 con 0x7f53680cf150 2024-03-24T01:25:21.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.892+0000 7f5365600640 1 -- 172.21.15.90:0/3400967220 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5360037070 con 0x7f53680cf150 2024-03-24T01:25:21.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.892+0000 7f5365600640 1 -- 172.21.15.90:0/3400967220 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5360031960 con 0x7f53680cf150 2024-03-24T01:25:21.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.893+0000 7f536d600640 1 -- 172.21.15.90:0/3400967220 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53680cf150 msgr2=0x7f53680ae530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:21.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.893+0000 7f536d600640 1 --2- 172.21.15.90:0/3400967220 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53680cf150 0x7f53680ae530 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f53600097a0 tx=0x7f536002b840 comp rx=0 tx=0).stop 2024-03-24T01:25:21.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.893+0000 7f536d600640 1 -- 172.21.15.90:0/3400967220 shutdown_connections 2024-03-24T01:25:21.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.893+0000 7f536d600640 1 --2- 172.21.15.90:0/3400967220 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53680cf150 0x7f53680ae530 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:21.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.893+0000 7f536d600640 1 -- 172.21.15.90:0/3400967220 >> 172.21.15.90:0/3400967220 conn(0x7f53680b1770 msgr2=0x7f53680b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:21.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.893+0000 7f536d600640 1 -- 172.21.15.90:0/3400967220 shutdown_connections 2024-03-24T01:25:21.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.893+0000 7f536d600640 1 -- 172.21.15.90:0/3400967220 wait complete. 2024-03-24T01:25:21.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.894+0000 7f536d600640 1 Processor -- start 2024-03-24T01:25:21.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.895+0000 7f536d600640 1 -- start start 2024-03-24T01:25:21.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.895+0000 7f536d600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f536806b7d0 0x7f536806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:21.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.895+0000 7f536d600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53680cf150 0x7f5368152440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:21.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.895+0000 7f536d600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5368159620 0x7f536814f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:21.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.895+0000 7f536d600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5368159a00 con 0x7f53680cf150 2024-03-24T01:25:21.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.895+0000 7f536d600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53680b3f50 con 0x7f5368159620 2024-03-24T01:25:21.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.895+0000 7f536d600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f536815bfe0 con 0x7f536806b7d0 2024-03-24T01:25:21.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.896+0000 7f5366000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53680cf150 0x7f5368152440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:21.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.896+0000 7f5366000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53680cf150 0x7f5368152440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43058/0 (socket says 172.21.15.90:43058) 2024-03-24T01:25:21.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.896+0000 7f5366000640 1 -- 172.21.15.90:0/226467239 learned_addr learned my addr 172.21.15.90:0/226467239 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:21.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.896+0000 7f5367400640 1 --2- 172.21.15.90:0/226467239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5368159620 0x7f536814f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:21.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.896+0000 7f5366000640 1 -- 172.21.15.90:0/226467239 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f536806b7d0 msgr2=0x7f536806d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:21.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.896+0000 7f5366000640 1 --2- 172.21.15.90:0/226467239 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f536806b7d0 0x7f536806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:21.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.896+0000 7f5366000640 1 -- 172.21.15.90:0/226467239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5368159620 msgr2=0x7f536814f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:21.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.896+0000 7f5366000640 1 --2- 172.21.15.90:0/226467239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5368159620 0x7f536814f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:21.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.896+0000 7f5366000640 1 -- 172.21.15.90:0/226467239 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f536806e060 con 0x7f53680cf150 2024-03-24T01:25:21.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.896+0000 7f5367400640 1 --2- 172.21.15.90:0/226467239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5368159620 0x7f536814f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:21.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.896+0000 7f5366000640 1 --2- 172.21.15.90:0/226467239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53680cf150 0x7f5368152440 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f535000c4b0 tx=0x7f535000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:21.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.897+0000 7f535ea00640 1 -- 172.21.15.90:0/226467239 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5350018070 con 0x7f53680cf150 2024-03-24T01:25:21.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.897+0000 7f535ea00640 1 -- 172.21.15.90:0/226467239 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5350007500 con 0x7f53680cf150 2024-03-24T01:25:21.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.897+0000 7f535ea00640 1 -- 172.21.15.90:0/226467239 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5350004a20 con 0x7f53680cf150 2024-03-24T01:25:21.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.897+0000 7f536d600640 1 -- 172.21.15.90:0/226467239 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f536806e340 con 0x7f53680cf150 2024-03-24T01:25:21.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.897+0000 7f536d600640 1 -- 172.21.15.90:0/226467239 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f536806e900 con 0x7f53680cf150 2024-03-24T01:25:21.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.898+0000 7f536d600640 1 -- 172.21.15.90:0/226467239 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f536804f400 con 0x7f53680cf150 2024-03-24T01:25:21.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.899+0000 7f535ea00640 1 -- 172.21.15.90:0/226467239 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5350004bc0 con 0x7f53680cf150 2024-03-24T01:25:21.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.900+0000 7f535ea00640 1 --2- 172.21.15.90:0/226467239 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f534406bbd0 0x7f534406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:21.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.900+0000 7f535ea00640 1 -- 172.21.15.90:0/226467239 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f535008a220 con 0x7f53680cf150 2024-03-24T01:25:21.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.900+0000 7f5366a00640 1 --2- 172.21.15.90:0/226467239 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f534406bbd0 0x7f534406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:21.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.902+0000 7f5366a00640 1 --2- 172.21.15.90:0/226467239 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f534406bbd0 0x7f534406e090 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f535800ab90 tx=0x7f5358009230 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:21.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:21.904+0000 7f535ea00640 1 -- 172.21.15.90:0/226467239 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5350058910 con 0x7f53680cf150 2024-03-24T01:25:22.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.102+0000 7f536d600640 1 -- 172.21.15.90:0/226467239 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f53680b2e90 con 0x7f53680cf150 2024-03-24T01:25:22.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.103+0000 7f535ea00640 1 -- 172.21.15.90:0/226467239 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5350058210 con 0x7f53680cf150 2024-03-24T01:25:22.105 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:25:22.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.106+0000 7f536d600640 1 -- 172.21.15.90:0/226467239 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f534406bbd0 msgr2=0x7f534406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:22.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.106+0000 7f536d600640 1 --2- 172.21.15.90:0/226467239 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f534406bbd0 0x7f534406e090 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f535800ab90 tx=0x7f5358009230 comp rx=0 tx=0).stop 2024-03-24T01:25:22.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.106+0000 7f536d600640 1 -- 172.21.15.90:0/226467239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53680cf150 msgr2=0x7f5368152440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:22.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.106+0000 7f536d600640 1 --2- 172.21.15.90:0/226467239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53680cf150 0x7f5368152440 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f535000c4b0 tx=0x7f535000c970 comp rx=0 tx=0).stop 2024-03-24T01:25:22.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.106+0000 7f536d600640 1 -- 172.21.15.90:0/226467239 shutdown_connections 2024-03-24T01:25:22.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.106+0000 7f536d600640 1 --2- 172.21.15.90:0/226467239 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f534406bbd0 0x7f534406e090 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:22.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.106+0000 7f536d600640 1 --2- 172.21.15.90:0/226467239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53680cf150 0x7f5368152440 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:22.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.106+0000 7f536d600640 1 --2- 172.21.15.90:0/226467239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5368159620 0x7f536814f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:22.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.106+0000 7f536d600640 1 --2- 172.21.15.90:0/226467239 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f536806b7d0 0x7f536806d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:22.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.106+0000 7f536d600640 1 -- 172.21.15.90:0/226467239 >> 172.21.15.90:0/226467239 conn(0x7f53680b1770 msgr2=0x7f53680cf770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:22.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.106+0000 7f536d600640 1 -- 172.21.15.90:0/226467239 shutdown_connections 2024-03-24T01:25:22.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.106+0000 7f536d600640 1 -- 172.21.15.90:0/226467239 wait complete. 2024-03-24T01:25:22.125 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T01:25:22.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.331+0000 7fc056200640 1 Processor -- start 2024-03-24T01:25:22.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.331+0000 7fc056200640 1 -- start start 2024-03-24T01:25:22.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.333+0000 7fc056200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc05006b7d0 0x7fc0500cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:22.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.333+0000 7fc056200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0500cef80 0x7fc0500ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:22.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.333+0000 7fc056200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc05006bca0 con 0x7fc050159620 2024-03-24T01:25:22.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.333+0000 7fc056200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0500b4470 con 0x7fc05006b7d0 2024-03-24T01:25:22.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.333+0000 7fc056200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0500b36f0 con 0x7fc0500cef80 2024-03-24T01:25:22.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.333+0000 7fc04f400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc050159620 0x7fc05006b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:42336/0 (socket says 172.21.15.90:42336) 2024-03-24T01:25:22.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.333+0000 7fc04f400640 1 -- 172.21.15.90:0/1637697906 learned_addr learned my addr 172.21.15.90:0/1637697906 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:22.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.333+0000 7fc04ea00640 1 --2- 172.21.15.90:0/1637697906 >> v2:172.21.15.120:3300/0 conn(0x7fc05006b7d0 0x7fc0500cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:22.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.333+0000 7fc04fe00640 1 --2- 172.21.15.90:0/1637697906 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0500cef80 0x7fc0500ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:22.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.334+0000 7fc04e000640 1 -- 172.21.15.90:0/1637697906 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1272829378 0 0) 0x7fc05006bca0 con 0x7fc050159620 2024-03-24T01:25:22.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.334+0000 7fc04e000640 1 -- 172.21.15.90:0/1637697906 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc034003520 con 0x7fc050159620 2024-03-24T01:25:22.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.334+0000 7fc04ea00640 1 -- 172.21.15.90:0/1637697906 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0500cef80 msgr2=0x7fc0500ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:22.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.334+0000 7fc04ea00640 1 --2- 172.21.15.90:0/1637697906 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0500cef80 0x7fc0500ae530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:22.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.334+0000 7fc04e000640 1 -- 172.21.15.90:0/1637697906 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2799631799 0 0) 0x7fc034003520 con 0x7fc050159620 2024-03-24T01:25:22.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.334+0000 7fc04ea00640 1 -- 172.21.15.90:0/1637697906 >> v1:172.21.15.90:6789/0 conn(0x7fc050159620 legacy=0x7fc05006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:22.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.334+0000 7fc04ea00640 1 -- 172.21.15.90:0/1637697906 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0500b3990 con 0x7fc05006b7d0 2024-03-24T01:25:22.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.336+0000 7fc04ea00640 1 --2- 172.21.15.90:0/1637697906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc05006b7d0 0x7fc0500cea40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc03c009d90 tx=0x7fc03c02bce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ad0d505dd616bf5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:22.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.336+0000 7fc04e000640 1 -- 172.21.15.90:0/1637697906 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03c039070 con 0x7fc05006b7d0 2024-03-24T01:25:22.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.336+0000 7fc04e000640 1 -- 172.21.15.90:0/1637697906 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc03c02c810 con 0x7fc05006b7d0 2024-03-24T01:25:22.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.336+0000 7fc04e000640 1 -- 172.21.15.90:0/1637697906 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc040001670 con 0x7fc05006b7d0 2024-03-24T01:25:22.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.337+0000 7fc056200640 1 -- 172.21.15.90:0/1637697906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc05006b7d0 msgr2=0x7fc0500cea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:22.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.337+0000 7fc056200640 1 --2- 172.21.15.90:0/1637697906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc05006b7d0 0x7fc0500cea40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc03c009d90 tx=0x7fc03c02bce0 comp rx=0 tx=0).stop 2024-03-24T01:25:22.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.337+0000 7fc056200640 1 -- 172.21.15.90:0/1637697906 shutdown_connections 2024-03-24T01:25:22.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.337+0000 7fc056200640 1 --2- 172.21.15.90:0/1637697906 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0500cef80 0x7fc0500ae530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:22.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.337+0000 7fc056200640 1 --2- 172.21.15.90:0/1637697906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc05006b7d0 0x7fc0500cea40 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:22.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.337+0000 7fc056200640 1 -- 172.21.15.90:0/1637697906 >> 172.21.15.90:0/1637697906 conn(0x7fc0500b1770 msgr2=0x7fc0500b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:22.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.337+0000 7fc056200640 1 -- 172.21.15.90:0/1637697906 shutdown_connections 2024-03-24T01:25:22.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.337+0000 7fc056200640 1 -- 172.21.15.90:0/1637697906 wait complete. 2024-03-24T01:25:22.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.339+0000 7fc056200640 1 Processor -- start 2024-03-24T01:25:22.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.339+0000 7fc056200640 1 -- start start 2024-03-24T01:25:22.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.340+0000 7fc056200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc05006b7d0 0x7fc050157f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:22.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.340+0000 7fc056200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0500cef80 0x7fc0500cba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:22.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.340+0000 7fc056200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc050159620 0x7fc0500cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:22.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.340+0000 7fc056200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc050159a00 con 0x7fc050159620 2024-03-24T01:25:22.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.340+0000 7fc056200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0500b3f50 con 0x7fc05006b7d0 2024-03-24T01:25:22.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.340+0000 7fc056200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0500cf400 con 0x7fc0500cef80 2024-03-24T01:25:22.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.340+0000 7fc04fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc050159620 0x7fc0500cbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:22.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.340+0000 7fc04fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc050159620 0x7fc0500cbfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43066/0 (socket says 172.21.15.90:43066) 2024-03-24T01:25:22.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.340+0000 7fc04fe00640 1 -- 172.21.15.90:0/784278247 learned_addr learned my addr 172.21.15.90:0/784278247 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:22.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.341+0000 7fc04ea00640 1 --2- 172.21.15.90:0/784278247 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0500cef80 0x7fc0500cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:22.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.341+0000 7fc04f400640 1 --2- 172.21.15.90:0/784278247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc05006b7d0 0x7fc050157f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:22.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.341+0000 7fc04fe00640 1 -- 172.21.15.90:0/784278247 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0500cef80 msgr2=0x7fc0500cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:22.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.341+0000 7fc04fe00640 1 --2- 172.21.15.90:0/784278247 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0500cef80 0x7fc0500cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:22.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.341+0000 7fc04fe00640 1 -- 172.21.15.90:0/784278247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc05006b7d0 msgr2=0x7fc050157f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:22.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.341+0000 7fc04fe00640 1 --2- 172.21.15.90:0/784278247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc05006b7d0 0x7fc050157f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:22.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.341+0000 7fc04fe00640 1 -- 172.21.15.90:0/784278247 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0501e6470 con 0x7fc050159620 2024-03-24T01:25:22.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.341+0000 7fc04ea00640 1 --2- 172.21.15.90:0/784278247 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0500cef80 0x7fc0500cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:22.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.341+0000 7fc04f400640 1 --2- 172.21.15.90:0/784278247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc05006b7d0 0x7fc050157f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:22.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.341+0000 7fc04fe00640 1 --2- 172.21.15.90:0/784278247 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc050159620 0x7fc0500cbfc0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc04800e930 tx=0x7fc04800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:22.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.341+0000 7fc047400640 1 -- 172.21.15.90:0/784278247 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc048017070 con 0x7fc050159620 2024-03-24T01:25:22.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.341+0000 7fc047400640 1 -- 172.21.15.90:0/784278247 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc04800c980 con 0x7fc050159620 2024-03-24T01:25:22.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.341+0000 7fc047400640 1 -- 172.21.15.90:0/784278247 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc04800cd70 con 0x7fc050159620 2024-03-24T01:25:22.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.341+0000 7fc056200640 1 -- 172.21.15.90:0/784278247 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0501e6750 con 0x7fc050159620 2024-03-24T01:25:22.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.341+0000 7fc056200640 1 -- 172.21.15.90:0/784278247 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0501e6d10 con 0x7fc050159620 2024-03-24T01:25:22.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.343+0000 7fc056200640 1 -- 172.21.15.90:0/784278247 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc05004f400 con 0x7fc050159620 2024-03-24T01:25:22.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.343+0000 7fc047400640 1 -- 172.21.15.90:0/784278247 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc04801a070 con 0x7fc050159620 2024-03-24T01:25:22.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.345+0000 7fc047400640 1 --2- 172.21.15.90:0/784278247 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc03006bbd0 0x7fc03006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:22.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.345+0000 7fc047400640 1 -- 172.21.15.90:0/784278247 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7fc04808aeb0 con 0x7fc050159620 2024-03-24T01:25:22.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.346+0000 7fc04f400640 1 --2- 172.21.15.90:0/784278247 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc03006bbd0 0x7fc03006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:22.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.347+0000 7fc04f400640 1 --2- 172.21.15.90:0/784278247 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc03006bbd0 0x7fc03006e090 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc040001380 tx=0x7fc040009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:22.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.350+0000 7fc047400640 1 -- 172.21.15.90:0/784278247 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc048059630 con 0x7fc050159620 2024-03-24T01:25:22.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.549+0000 7fc056200640 1 -- 172.21.15.90:0/784278247 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc0500b2e90 con 0x7fc050159620 2024-03-24T01:25:22.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.550+0000 7fc047400640 1 -- 172.21.15.90:0/784278247 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc048058f30 con 0x7fc050159620 2024-03-24T01:25:22.551 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:25:22.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.552+0000 7fc056200640 1 -- 172.21.15.90:0/784278247 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc03006bbd0 msgr2=0x7fc03006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:22.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.552+0000 7fc056200640 1 --2- 172.21.15.90:0/784278247 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc03006bbd0 0x7fc03006e090 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc040001380 tx=0x7fc040009040 comp rx=0 tx=0).stop 2024-03-24T01:25:22.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.552+0000 7fc056200640 1 -- 172.21.15.90:0/784278247 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc050159620 msgr2=0x7fc0500cbfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:22.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.552+0000 7fc056200640 1 --2- 172.21.15.90:0/784278247 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc050159620 0x7fc0500cbfc0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc04800e930 tx=0x7fc04800edf0 comp rx=0 tx=0).stop 2024-03-24T01:25:22.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.553+0000 7fc056200640 1 -- 172.21.15.90:0/784278247 shutdown_connections 2024-03-24T01:25:22.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.553+0000 7fc056200640 1 --2- 172.21.15.90:0/784278247 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc03006bbd0 0x7fc03006e090 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:22.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.553+0000 7fc056200640 1 --2- 172.21.15.90:0/784278247 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc050159620 0x7fc0500cbfc0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:22.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.553+0000 7fc056200640 1 --2- 172.21.15.90:0/784278247 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0500cef80 0x7fc0500cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:22.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.553+0000 7fc056200640 1 --2- 172.21.15.90:0/784278247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc05006b7d0 0x7fc050157f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:22.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.553+0000 7fc056200640 1 -- 172.21.15.90:0/784278247 >> 172.21.15.90:0/784278247 conn(0x7fc0500b1770 msgr2=0x7fc0500b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:22.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.553+0000 7fc056200640 1 -- 172.21.15.90:0/784278247 shutdown_connections 2024-03-24T01:25:22.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.553+0000 7fc056200640 1 -- 172.21.15.90:0/784278247 wait complete. 2024-03-24T01:25:22.572 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T01:25:22.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.793+0000 7f5b22400640 1 Processor -- start 2024-03-24T01:25:22.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.793+0000 7f5b22400640 1 -- start start 2024-03-24T01:25:22.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.794+0000 7f5b22400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5b1c1550a0 0x7f5b1c15bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:22.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.794+0000 7f5b22400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5b1c06f5c0 0x7f5b1c0b3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:22.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.794+0000 7f5b22400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5b1c0b3fe0 0x7f5b1c0ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:22.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.794+0000 7f5b22400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b1c0b4430 con 0x7f5b1c1550a0 2024-03-24T01:25:22.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.794+0000 7f5b22400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b1c06fa70 con 0x7f5b1c06f5c0 2024-03-24T01:25:22.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.794+0000 7f5b22400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c155480 con 0x7f5b1c0b3fe0 2024-03-24T01:25:22.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.795+0000 7f5b1b400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5b1c1550a0 0x7f5b1c15bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:22.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.795+0000 7f5b1b400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5b1c1550a0 0x7f5b1c15bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43068/0 (socket says 172.21.15.90:43068) 2024-03-24T01:25:22.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.795+0000 7f5b1b400640 1 -- 172.21.15.90:0/2333420248 learned_addr learned my addr 172.21.15.90:0/2333420248 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:22.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.795+0000 7f5b1be00640 1 --2- 172.21.15.90:0/2333420248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5b1c0b3fe0 0x7f5b1c0ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:22.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.795+0000 7f5b1aa00640 1 --2- 172.21.15.90:0/2333420248 >> v2:172.21.15.120:3300/0 conn(0x7f5b1c06f5c0 0x7f5b1c0b3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:22.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.795+0000 7f5b1b400640 1 -- 172.21.15.90:0/2333420248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5b1c0b3fe0 msgr2=0x7f5b1c0ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:22.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.795+0000 7f5b1b400640 1 --2- 172.21.15.90:0/2333420248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5b1c0b3fe0 0x7f5b1c0ae530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:22.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.795+0000 7f5b1b400640 1 -- 172.21.15.90:0/2333420248 >> v2:172.21.15.120:3300/0 conn(0x7f5b1c06f5c0 msgr2=0x7f5b1c0b3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:22.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.795+0000 7f5b1b400640 1 --2- 172.21.15.90:0/2333420248 >> v2:172.21.15.120:3300/0 conn(0x7f5b1c06f5c0 0x7f5b1c0b3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:22.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.795+0000 7f5b1b400640 1 -- 172.21.15.90:0/2333420248 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1c06fcc0 con 0x7f5b1c1550a0 2024-03-24T01:25:22.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.795+0000 7f5b1aa00640 1 --2- 172.21.15.90:0/2333420248 >> v2:172.21.15.120:3300/0 conn(0x7f5b1c06f5c0 0x7f5b1c0b3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:22.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.796+0000 7f5b1b400640 1 --2- 172.21.15.90:0/2333420248 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b1c1550a0 0x7f5b1c15bb20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c0717e0 tx=0x7f5b0c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a60211c00bb118f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:22.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.797+0000 7f5b1a000640 1 -- 172.21.15.90:0/2333420248 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b0c02e070 con 0x7f5b1c1550a0 2024-03-24T01:25:22.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.797+0000 7f5b1a000640 1 -- 172.21.15.90:0/2333420248 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b0c037070 con 0x7f5b1c1550a0 2024-03-24T01:25:22.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.797+0000 7f5b1a000640 1 -- 172.21.15.90:0/2333420248 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b0c031a00 con 0x7f5b1c1550a0 2024-03-24T01:25:22.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.797+0000 7f5b22400640 1 -- 172.21.15.90:0/2333420248 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b1c1550a0 msgr2=0x7f5b1c15bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:22.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.797+0000 7f5b22400640 1 --2- 172.21.15.90:0/2333420248 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b1c1550a0 0x7f5b1c15bb20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c0717e0 tx=0x7f5b0c02b880 comp rx=0 tx=0).stop 2024-03-24T01:25:22.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.797+0000 7f5b22400640 1 -- 172.21.15.90:0/2333420248 shutdown_connections 2024-03-24T01:25:22.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.797+0000 7f5b22400640 1 --2- 172.21.15.90:0/2333420248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5b1c0b3fe0 0x7f5b1c0ae530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:22.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.797+0000 7f5b22400640 1 --2- 172.21.15.90:0/2333420248 >> v2:172.21.15.120:3300/0 conn(0x7f5b1c06f5c0 0x7f5b1c0b3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:22.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.797+0000 7f5b22400640 1 --2- 172.21.15.90:0/2333420248 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b1c1550a0 0x7f5b1c15bb20 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:22.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.797+0000 7f5b22400640 1 -- 172.21.15.90:0/2333420248 >> 172.21.15.90:0/2333420248 conn(0x7f5b1c0cf7e0 msgr2=0x7f5b1c0cfbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:22.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.797+0000 7f5b22400640 1 -- 172.21.15.90:0/2333420248 shutdown_connections 2024-03-24T01:25:22.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.797+0000 7f5b22400640 1 -- 172.21.15.90:0/2333420248 wait complete. 2024-03-24T01:25:22.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.799+0000 7f5b22400640 1 Processor -- start 2024-03-24T01:25:22.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.799+0000 7f5b22400640 1 -- start start 2024-03-24T01:25:22.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.800+0000 7f5b22400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b1c06f5c0 0x7f5b1c148690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:22.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.800+0000 7f5b22400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5b1c0b3fe0 0x7f5b1c0c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:22.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.800+0000 7f5b22400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b1c1550a0 0x7f5b1c0c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:22.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.800+0000 7f5b22400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c15e9c0 con 0x7f5b1c1550a0 2024-03-24T01:25:22.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.800+0000 7f5b22400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c15e6e0 con 0x7f5b1c06f5c0 2024-03-24T01:25:22.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.800+0000 7f5b22400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c15eca0 con 0x7f5b1c0b3fe0 2024-03-24T01:25:22.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.800+0000 7f5b1be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b1c1550a0 0x7f5b1c0c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:22.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.800+0000 7f5b1be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b1c1550a0 0x7f5b1c0c9150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43070/0 (socket says 172.21.15.90:43070) 2024-03-24T01:25:22.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.800+0000 7f5b1be00640 1 -- 172.21.15.90:0/3855256488 learned_addr learned my addr 172.21.15.90:0/3855256488 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:22.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.801+0000 7f5b1b400640 1 --2- 172.21.15.90:0/3855256488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b1c06f5c0 0x7f5b1c148690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:22.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.801+0000 7f5b1aa00640 1 --2- 172.21.15.90:0/3855256488 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5b1c0b3fe0 0x7f5b1c0c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:22.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.801+0000 7f5b1be00640 1 -- 172.21.15.90:0/3855256488 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5b1c0b3fe0 msgr2=0x7f5b1c0c8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:22.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.801+0000 7f5b1be00640 1 --2- 172.21.15.90:0/3855256488 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5b1c0b3fe0 0x7f5b1c0c8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:22.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.801+0000 7f5b1be00640 1 -- 172.21.15.90:0/3855256488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b1c06f5c0 msgr2=0x7f5b1c148690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:22.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.801+0000 7f5b1be00640 1 --2- 172.21.15.90:0/3855256488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b1c06f5c0 0x7f5b1c148690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:22.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.801+0000 7f5b1be00640 1 -- 172.21.15.90:0/3855256488 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1c1566c0 con 0x7f5b1c1550a0 2024-03-24T01:25:22.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.801+0000 7f5b1be00640 1 --2- 172.21.15.90:0/3855256488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b1c1550a0 0x7f5b1c0c9150 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5b1400cd70 tx=0x7f5b1400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:22.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.801+0000 7f5b13400640 1 -- 172.21.15.90:0/3855256488 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b14003b90 con 0x7f5b1c1550a0 2024-03-24T01:25:22.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.802+0000 7f5b13400640 1 -- 172.21.15.90:0/3855256488 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b14003d30 con 0x7f5b1c1550a0 2024-03-24T01:25:22.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.802+0000 7f5b13400640 1 -- 172.21.15.90:0/3855256488 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b14004240 con 0x7f5b1c1550a0 2024-03-24T01:25:22.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.802+0000 7f5b22400640 1 -- 172.21.15.90:0/3855256488 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b1c1569a0 con 0x7f5b1c1550a0 2024-03-24T01:25:22.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.802+0000 7f5b22400640 1 -- 172.21.15.90:0/3855256488 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b1c156f60 con 0x7f5b1c1550a0 2024-03-24T01:25:22.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.803+0000 7f5b22400640 1 -- 172.21.15.90:0/3855256488 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b1c04f400 con 0x7f5b1c1550a0 2024-03-24T01:25:22.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.804+0000 7f5b13400640 1 -- 172.21.15.90:0/3855256488 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5b140043e0 con 0x7f5b1c1550a0 2024-03-24T01:25:22.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.804+0000 7f5b13400640 1 --2- 172.21.15.90:0/3855256488 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5afc06bbd0 0x7f5afc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:22.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.805+0000 7f5b13400640 1 -- 172.21.15.90:0/3855256488 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f5b1408b900 con 0x7f5b1c1550a0 2024-03-24T01:25:22.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.805+0000 7f5b1b400640 1 --2- 172.21.15.90:0/3855256488 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5afc06bbd0 0x7f5afc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:22.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.806+0000 7f5b1b400640 1 --2- 172.21.15.90:0/3855256488 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5afc06bbd0 0x7f5afc06e090 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c036000 tx=0x7f5b0c03d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:22.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:22.809+0000 7f5b13400640 1 -- 172.21.15.90:0/3855256488 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5b1401d340 con 0x7f5b1c1550a0 2024-03-24T01:25:23.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.004+0000 7f5b22400640 1 -- 172.21.15.90:0/3855256488 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5b1c0cc170 con 0x7f5b1c1550a0 2024-03-24T01:25:23.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.005+0000 7f5b13400640 1 -- 172.21.15.90:0/3855256488 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5b1405a060 con 0x7f5b1c1550a0 2024-03-24T01:25:23.006 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-24T01:25:23.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.008+0000 7f5b22400640 1 -- 172.21.15.90:0/3855256488 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5afc06bbd0 msgr2=0x7f5afc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:23.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.008+0000 7f5b22400640 1 --2- 172.21.15.90:0/3855256488 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5afc06bbd0 0x7f5afc06e090 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c036000 tx=0x7f5b0c03d040 comp rx=0 tx=0).stop 2024-03-24T01:25:23.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.008+0000 7f5b22400640 1 -- 172.21.15.90:0/3855256488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b1c1550a0 msgr2=0x7f5b1c0c9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:23.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.008+0000 7f5b22400640 1 --2- 172.21.15.90:0/3855256488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b1c1550a0 0x7f5b1c0c9150 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5b1400cd70 tx=0x7f5b1400a660 comp rx=0 tx=0).stop 2024-03-24T01:25:23.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.008+0000 7f5b22400640 1 -- 172.21.15.90:0/3855256488 shutdown_connections 2024-03-24T01:25:23.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.008+0000 7f5b22400640 1 --2- 172.21.15.90:0/3855256488 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5afc06bbd0 0x7f5afc06e090 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:23.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.008+0000 7f5b22400640 1 --2- 172.21.15.90:0/3855256488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b1c1550a0 0x7f5b1c0c9150 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:23.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.008+0000 7f5b22400640 1 --2- 172.21.15.90:0/3855256488 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5b1c0b3fe0 0x7f5b1c0c8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:23.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.008+0000 7f5b22400640 1 --2- 172.21.15.90:0/3855256488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b1c06f5c0 0x7f5b1c148690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:23.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.008+0000 7f5b22400640 1 -- 172.21.15.90:0/3855256488 >> 172.21.15.90:0/3855256488 conn(0x7f5b1c0cf7e0 msgr2=0x7f5b1c0cdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:23.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.008+0000 7f5b22400640 1 -- 172.21.15.90:0/3855256488 shutdown_connections 2024-03-24T01:25:23.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.008+0000 7f5b22400640 1 -- 172.21.15.90:0/3855256488 wait complete. 2024-03-24T01:25:23.029 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T01:25:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.241+0000 7f6e4e800640 1 Processor -- start 2024-03-24T01:25:23.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.241+0000 7f6e4e800640 1 -- start start 2024-03-24T01:25:23.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.243+0000 7f6e4e800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6e4806b7d0 0x7f6e480cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:23.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.243+0000 7f6e4e800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6e480cef80 0x7f6e480ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:23.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.243+0000 7f6e4e800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e4806bca0 con 0x7f6e48159620 2024-03-24T01:25:23.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.243+0000 7f6e4e800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e480b4470 con 0x7f6e4806b7d0 2024-03-24T01:25:23.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.243+0000 7f6e4e800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e480b36f0 con 0x7f6e480cef80 2024-03-24T01:25:23.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.243+0000 7f6e3ea00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6e4806b7d0 0x7f6e480cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:23.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.243+0000 7f6e3ea00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6e4806b7d0 0x7f6e480cea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43082/0 (socket says 172.21.15.90:43082) 2024-03-24T01:25:23.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.243+0000 7f6e3ea00640 1 -- 172.21.15.90:0/4031186269 learned_addr learned my addr 172.21.15.90:0/4031186269 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:23.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.244+0000 7f6e47e00640 1 --2- 172.21.15.90:0/4031186269 >> v2:172.21.15.120:3300/0 conn(0x7f6e480cef80 0x7f6e480ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:23.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.244+0000 7f6e3ea00640 1 -- 172.21.15.90:0/4031186269 >> v2:172.21.15.120:3300/0 conn(0x7f6e480cef80 msgr2=0x7f6e480ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:23.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.244+0000 7f6e3ea00640 1 --2- 172.21.15.90:0/4031186269 >> v2:172.21.15.120:3300/0 conn(0x7f6e480cef80 0x7f6e480ae530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:23.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.244+0000 7f6e3ea00640 1 -- 172.21.15.90:0/4031186269 >> v1:172.21.15.90:6789/0 conn(0x7f6e48159620 legacy=0x7f6e4806b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:23.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.244+0000 7f6e3ea00640 1 -- 172.21.15.90:0/4031186269 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e480b3990 con 0x7f6e4806b7d0 2024-03-24T01:25:23.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.245+0000 7f6e3ea00640 1 --2- 172.21.15.90:0/4031186269 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e4806b7d0 0x7f6e480cea40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f6e480b29d0 tx=0x7f6e3002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb8f93012fbd1084 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:23.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.245+0000 7f6e46a00640 1 -- 172.21.15.90:0/4031186269 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e30035020 con 0x7f6e4806b7d0 2024-03-24T01:25:23.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.245+0000 7f6e46a00640 1 -- 172.21.15.90:0/4031186269 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e30031870 con 0x7f6e4806b7d0 2024-03-24T01:25:23.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.245+0000 7f6e46a00640 1 -- 172.21.15.90:0/4031186269 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e30031c40 con 0x7f6e4806b7d0 2024-03-24T01:25:23.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.246+0000 7f6e4e800640 1 -- 172.21.15.90:0/4031186269 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e4806b7d0 msgr2=0x7f6e480cea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:23.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.246+0000 7f6e4e800640 1 --2- 172.21.15.90:0/4031186269 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e4806b7d0 0x7f6e480cea40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f6e480b29d0 tx=0x7f6e3002b910 comp rx=0 tx=0).stop 2024-03-24T01:25:23.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.246+0000 7f6e4e800640 1 -- 172.21.15.90:0/4031186269 shutdown_connections 2024-03-24T01:25:23.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.246+0000 7f6e4e800640 1 --2- 172.21.15.90:0/4031186269 >> v2:172.21.15.120:3300/0 conn(0x7f6e480cef80 0x7f6e480ae530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:23.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.246+0000 7f6e4e800640 1 --2- 172.21.15.90:0/4031186269 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e4806b7d0 0x7f6e480cea40 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:23.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.246+0000 7f6e4e800640 1 -- 172.21.15.90:0/4031186269 >> 172.21.15.90:0/4031186269 conn(0x7f6e480b1770 msgr2=0x7f6e480b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:23.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.246+0000 7f6e4e800640 1 -- 172.21.15.90:0/4031186269 shutdown_connections 2024-03-24T01:25:23.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.246+0000 7f6e4e800640 1 -- 172.21.15.90:0/4031186269 wait complete. 2024-03-24T01:25:23.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.248+0000 7f6e4e800640 1 Processor -- start 2024-03-24T01:25:23.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.248+0000 7f6e4e800640 1 -- start start 2024-03-24T01:25:23.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.249+0000 7f6e4e800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e4806b7d0 0x7f6e481401a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:23.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.249+0000 7f6e4e800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e480cef80 0x7f6e4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:23.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.249+0000 7f6e4e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e48159620 0x7f6e4806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:23.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.249+0000 7f6e4e800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e480cf430 con 0x7f6e48159620 2024-03-24T01:25:23.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.249+0000 7f6e4e800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e480b3f50 con 0x7f6e4806b7d0 2024-03-24T01:25:23.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.249+0000 7f6e4e800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e48159a00 con 0x7f6e480cef80 2024-03-24T01:25:23.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.249+0000 7f6e47e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e48159620 0x7f6e4806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:23.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.249+0000 7f6e47e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e48159620 0x7f6e4806ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43088/0 (socket says 172.21.15.90:43088) 2024-03-24T01:25:23.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.249+0000 7f6e47e00640 1 -- 172.21.15.90:0/653437063 learned_addr learned my addr 172.21.15.90:0/653437063 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:23.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.249+0000 7f6e47400640 1 --2- 172.21.15.90:0/653437063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e4806b7d0 0x7f6e481401a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:23.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.250+0000 7f6e3ea00640 1 --2- 172.21.15.90:0/653437063 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e480cef80 0x7f6e4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:23.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.250+0000 7f6e47e00640 1 -- 172.21.15.90:0/653437063 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e480cef80 msgr2=0x7f6e4806d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:23.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.250+0000 7f6e47e00640 1 --2- 172.21.15.90:0/653437063 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e480cef80 0x7f6e4806d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:23.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.250+0000 7f6e47e00640 1 -- 172.21.15.90:0/653437063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e4806b7d0 msgr2=0x7f6e481401a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:23.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.250+0000 7f6e47e00640 1 --2- 172.21.15.90:0/653437063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e4806b7d0 0x7f6e481401a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:23.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.250+0000 7f6e47e00640 1 -- 172.21.15.90:0/653437063 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e4806e550 con 0x7f6e48159620 2024-03-24T01:25:23.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.250+0000 7f6e3ea00640 1 --2- 172.21.15.90:0/653437063 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e480cef80 0x7f6e4806d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:23.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.250+0000 7f6e47e00640 1 --2- 172.21.15.90:0/653437063 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e48159620 0x7f6e4806ded0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f6e4000cd70 tx=0x7f6e4000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:23.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.250+0000 7f6e3fe00640 1 -- 172.21.15.90:0/653437063 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e40003b90 con 0x7f6e48159620 2024-03-24T01:25:23.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.250+0000 7f6e3fe00640 1 -- 172.21.15.90:0/653437063 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e40003d30 con 0x7f6e48159620 2024-03-24T01:25:23.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.250+0000 7f6e4e800640 1 -- 172.21.15.90:0/653437063 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e4806e830 con 0x7f6e48159620 2024-03-24T01:25:23.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.250+0000 7f6e3fe00640 1 -- 172.21.15.90:0/653437063 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e40004240 con 0x7f6e48159620 2024-03-24T01:25:23.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.250+0000 7f6e4e800640 1 -- 172.21.15.90:0/653437063 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e4806edf0 con 0x7f6e48159620 2024-03-24T01:25:23.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.251+0000 7f6e4e800640 1 -- 172.21.15.90:0/653437063 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e4806f310 con 0x7f6e48159620 2024-03-24T01:25:23.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.252+0000 7f6e3fe00640 1 -- 172.21.15.90:0/653437063 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6e400043e0 con 0x7f6e48159620 2024-03-24T01:25:23.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.253+0000 7f6e3fe00640 1 --2- 172.21.15.90:0/653437063 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6e2406bbd0 0x7f6e2406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:23.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.253+0000 7f6e3fe00640 1 -- 172.21.15.90:0/653437063 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f6e4008b9e0 con 0x7f6e48159620 2024-03-24T01:25:23.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.254+0000 7f6e47400640 1 --2- 172.21.15.90:0/653437063 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6e2406bbd0 0x7f6e2406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:23.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.255+0000 7f6e47400640 1 --2- 172.21.15.90:0/653437063 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6e2406bbd0 0x7f6e2406e090 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6e38007fd0 tx=0x7f6e38009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:23.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.258+0000 7f6e3fe00640 1 -- 172.21.15.90:0/653437063 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6e4001d340 con 0x7f6e48159620 2024-03-24T01:25:23.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.451+0000 7f6e4e800640 1 -- 172.21.15.90:0/653437063 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f6e4804f400 con 0x7f6e48159620 2024-03-24T01:25:23.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.452+0000 7f6e3fe00640 1 -- 172.21.15.90:0/653437063 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f6e4005a0a0 con 0x7f6e48159620 2024-03-24T01:25:23.454 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:25:23.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.455+0000 7f6e4e800640 1 -- 172.21.15.90:0/653437063 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6e2406bbd0 msgr2=0x7f6e2406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:23.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.455+0000 7f6e4e800640 1 --2- 172.21.15.90:0/653437063 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6e2406bbd0 0x7f6e2406e090 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6e38007fd0 tx=0x7f6e38009040 comp rx=0 tx=0).stop 2024-03-24T01:25:23.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.455+0000 7f6e4e800640 1 -- 172.21.15.90:0/653437063 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e48159620 msgr2=0x7f6e4806ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:23.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.455+0000 7f6e4e800640 1 --2- 172.21.15.90:0/653437063 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e48159620 0x7f6e4806ded0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f6e4000cd70 tx=0x7f6e4000a660 comp rx=0 tx=0).stop 2024-03-24T01:25:23.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.455+0000 7f6e4e800640 1 -- 172.21.15.90:0/653437063 shutdown_connections 2024-03-24T01:25:23.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.455+0000 7f6e4e800640 1 --2- 172.21.15.90:0/653437063 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6e2406bbd0 0x7f6e2406e090 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:23.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.455+0000 7f6e4e800640 1 --2- 172.21.15.90:0/653437063 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e48159620 0x7f6e4806ded0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:23.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.455+0000 7f6e4e800640 1 --2- 172.21.15.90:0/653437063 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e480cef80 0x7f6e4806d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:23.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.455+0000 7f6e4e800640 1 --2- 172.21.15.90:0/653437063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e4806b7d0 0x7f6e481401a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:23.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.455+0000 7f6e4e800640 1 -- 172.21.15.90:0/653437063 >> 172.21.15.90:0/653437063 conn(0x7f6e480b1770 msgr2=0x7f6e4815e0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:23.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.455+0000 7f6e4e800640 1 -- 172.21.15.90:0/653437063 shutdown_connections 2024-03-24T01:25:23.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.455+0000 7f6e4e800640 1 -- 172.21.15.90:0/653437063 wait complete. 2024-03-24T01:25:23.473 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T01:25:23.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.671+0000 7f77c1400640 1 Processor -- start 2024-03-24T01:25:23.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.671+0000 7f77c1400640 1 -- start start 2024-03-24T01:25:23.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.673+0000 7f77c1400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f77bc15e020 0x7f77bc0ceab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:23.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.673+0000 7f77c1400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77bc0d2c40 0x7f77bc0d30d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:23.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.673+0000 7f77c1400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77bc06d410 con 0x7f77bc06c5f0 2024-03-24T01:25:23.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.673+0000 7f77c1400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77bc0d27b0 con 0x7f77bc15e020 2024-03-24T01:25:23.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.673+0000 7f77c1400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77bc15e400 con 0x7f77bc0d2c40 2024-03-24T01:25:23.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.673+0000 7f77c0a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f77bc06c5f0 0x7f77bc0b6ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:42360/0 (socket says 172.21.15.90:42360) 2024-03-24T01:25:23.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.673+0000 7f77c0a00640 1 -- 172.21.15.90:0/2523209238 learned_addr learned my addr 172.21.15.90:0/2523209238 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:23.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.674+0000 7f77bb400640 1 --2- 172.21.15.90:0/2523209238 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77bc0d2c40 0x7f77bc0d30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:23.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.674+0000 7f77bbe00640 1 --2- 172.21.15.90:0/2523209238 >> v2:172.21.15.120:3300/0 conn(0x7f77bc15e020 0x7f77bc0ceab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:23.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.674+0000 7f77baa00640 1 -- 172.21.15.90:0/2523209238 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1125603116 0 0) 0x7f77bc06d410 con 0x7f77bc06c5f0 2024-03-24T01:25:23.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.674+0000 7f77baa00640 1 -- 172.21.15.90:0/2523209238 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f779c003520 con 0x7f77bc06c5f0 2024-03-24T01:25:23.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.674+0000 7f77baa00640 1 -- 172.21.15.90:0/2523209238 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1953334416 0 0) 0x7f779c003520 con 0x7f77bc06c5f0 2024-03-24T01:25:23.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.674+0000 7f77baa00640 1 -- 172.21.15.90:0/2523209238 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77bc0d2c40 msgr2=0x7f77bc0d30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:23.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.674+0000 7f77baa00640 1 --2- 172.21.15.90:0/2523209238 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77bc0d2c40 0x7f77bc0d30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:23.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.674+0000 7f77baa00640 1 -- 172.21.15.90:0/2523209238 >> v2:172.21.15.120:3300/0 conn(0x7f77bc15e020 msgr2=0x7f77bc0ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:23.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.674+0000 7f77baa00640 1 --2- 172.21.15.90:0/2523209238 >> v2:172.21.15.120:3300/0 conn(0x7f77bc15e020 0x7f77bc0ceab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:23.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.674+0000 7f77baa00640 1 -- 172.21.15.90:0/2523209238 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77bc0cf2e0 con 0x7f77bc06c5f0 2024-03-24T01:25:23.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.674+0000 7f77baa00640 1 -- 172.21.15.90:0/2523209238 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f77b4001670 con 0x7f77bc06c5f0 2024-03-24T01:25:23.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.674+0000 7f77baa00640 1 -- 172.21.15.90:0/2523209238 >> v1:172.21.15.90:6789/0 conn(0x7f77bc06c5f0 legacy=0x7f77bc0b6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:23.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.674+0000 7f77baa00640 1 --2- 172.21.15.90:0/2523209238 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f779c0039b0 0x7f779c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:23.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.674+0000 7f77baa00640 1 --2- 172.21.15.90:0/2523209238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f779c004800 0x7f779c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:23.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.674+0000 7f77baa00640 1 --2- 172.21.15.90:0/2523209238 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77bc0d2c40 0x7f779c0051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:23.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.674+0000 7f77baa00640 1 -- 172.21.15.90:0/2523209238 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77bc0d27b0 con 0x7f779c004800 2024-03-24T01:25:23.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.674+0000 7f77baa00640 1 -- 172.21.15.90:0/2523209238 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77bc15e400 con 0x7f779c0039b0 2024-03-24T01:25:23.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.674+0000 7f77baa00640 1 -- 172.21.15.90:0/2523209238 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f779c00bb80 con 0x7f77bc0d2c40 2024-03-24T01:25:23.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.675+0000 7f77bb400640 1 --2- 172.21.15.90:0/2523209238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f779c004800 0x7f779c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:23.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.675+0000 7f77bb400640 1 -- 172.21.15.90:0/2523209238 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77bc0d2c40 msgr2=0x7f779c0051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:23.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.675+0000 7f77bb400640 1 --2- 172.21.15.90:0/2523209238 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77bc0d2c40 0x7f779c0051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:23.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.675+0000 7f77bb400640 1 -- 172.21.15.90:0/2523209238 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f779c0039b0 msgr2=0x7f779c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:23.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.675+0000 7f77bb400640 1 --2- 172.21.15.90:0/2523209238 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f779c0039b0 0x7f779c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:23.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.675+0000 7f77bb400640 1 -- 172.21.15.90:0/2523209238 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f779c003520 con 0x7f779c004800 2024-03-24T01:25:23.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.676+0000 7f77bb400640 1 --2- 172.21.15.90:0/2523209238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f779c004800 0x7f779c004c80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f77a4007360 tx=0x7f779c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=501ba0e54b538d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:23.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.676+0000 7f77baa00640 1 -- 172.21.15.90:0/2523209238 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a4039070 con 0x7f779c004800 2024-03-24T01:25:23.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.676+0000 7f77baa00640 1 -- 172.21.15.90:0/2523209238 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77a402ccf0 con 0x7f779c004800 2024-03-24T01:25:23.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.676+0000 7f77baa00640 1 -- 172.21.15.90:0/2523209238 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a4009be0 con 0x7f779c004800 2024-03-24T01:25:23.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.677+0000 7f77c1400640 1 -- 172.21.15.90:0/2523209238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f779c004800 msgr2=0x7f779c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:23.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.677+0000 7f77c1400640 1 --2- 172.21.15.90:0/2523209238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f779c004800 0x7f779c004c80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f77a4007360 tx=0x7f779c005830 comp rx=0 tx=0).stop 2024-03-24T01:25:23.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.677+0000 7f77c0a00640 1 -- 172.21.15.90:0/2523209238 reap_dead start 2024-03-24T01:25:23.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.677+0000 7f77c1400640 1 -- 172.21.15.90:0/2523209238 shutdown_connections 2024-03-24T01:25:23.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.677+0000 7f77c1400640 1 -- 172.21.15.90:0/2523209238 >> 172.21.15.90:0/2523209238 conn(0x7f77bc0d0560 msgr2=0x7f77bc0d0970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:23.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.677+0000 7f77c1400640 1 -- 172.21.15.90:0/2523209238 shutdown_connections 2024-03-24T01:25:23.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.677+0000 7f77c1400640 1 -- 172.21.15.90:0/2523209238 wait complete. 2024-03-24T01:25:23.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.678+0000 7f77c1400640 1 Processor -- start 2024-03-24T01:25:23.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.679+0000 7f77c1400640 1 -- start start 2024-03-24T01:25:23.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.679+0000 7f77c1400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f779c004800 0x7f77bc145990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:23.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.679+0000 7f77c1400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77bc0c9210 0x7f77bc0c9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:23.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.679+0000 7f77c1400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77bc156800 0x7f77bc156c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:23.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.679+0000 7f77c1400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77bc0ceff0 con 0x7f779c004800 2024-03-24T01:25:23.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.679+0000 7f77c1400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77bc0d2440 con 0x7f77bc156800 2024-03-24T01:25:23.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.679+0000 7f77c1400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77bc0cf5c0 con 0x7f77bc0c9210 2024-03-24T01:25:23.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.680+0000 7f77bbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f779c004800 0x7f77bc145990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:23.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.680+0000 7f77bbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f779c004800 0x7f77bc145990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43100/0 (socket says 172.21.15.90:43100) 2024-03-24T01:25:23.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.680+0000 7f77bbe00640 1 -- 172.21.15.90:0/1478372968 learned_addr learned my addr 172.21.15.90:0/1478372968 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:23.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.680+0000 7f77bb400640 1 --2- 172.21.15.90:0/1478372968 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77bc0c9210 0x7f77bc0c9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:23.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.680+0000 7f77bbe00640 1 -- 172.21.15.90:0/1478372968 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77bc0c9210 msgr2=0x7f77bc0c9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:23.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.680+0000 7f77c0a00640 1 --2- 172.21.15.90:0/1478372968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77bc156800 0x7f77bc156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:23.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.680+0000 7f77bbe00640 1 --2- 172.21.15.90:0/1478372968 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77bc0c9210 0x7f77bc0c9690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:23.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.680+0000 7f77bbe00640 1 -- 172.21.15.90:0/1478372968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77bc156800 msgr2=0x7f77bc156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:23.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.680+0000 7f77bbe00640 1 --2- 172.21.15.90:0/1478372968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77bc156800 0x7f77bc156c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:23.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.680+0000 7f77bbe00640 1 -- 172.21.15.90:0/1478372968 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77bc1575a0 con 0x7f779c004800 2024-03-24T01:25:23.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.680+0000 7f77bbe00640 1 --2- 172.21.15.90:0/1478372968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f779c004800 0x7f77bc145990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f77ac00d800 tx=0x7f77ac00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:23.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.681+0000 7f77b3e00640 1 -- 172.21.15.90:0/1478372968 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ac00b6c0 con 0x7f779c004800 2024-03-24T01:25:23.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.681+0000 7f77b3e00640 1 -- 172.21.15.90:0/1478372968 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77ac019070 con 0x7f779c004800 2024-03-24T01:25:23.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.681+0000 7f77b3e00640 1 -- 172.21.15.90:0/1478372968 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ac00baa0 con 0x7f779c004800 2024-03-24T01:25:23.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.681+0000 7f77c1400640 1 -- 172.21.15.90:0/1478372968 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77bc157880 con 0x7f779c004800 2024-03-24T01:25:23.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.681+0000 7f77c1400640 1 -- 172.21.15.90:0/1478372968 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77bc157e40 con 0x7f779c004800 2024-03-24T01:25:23.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.682+0000 7f77c1400640 1 -- 172.21.15.90:0/1478372968 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77bc04f400 con 0x7f779c004800 2024-03-24T01:25:23.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.683+0000 7f77b3e00640 1 -- 172.21.15.90:0/1478372968 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f77ac00bc40 con 0x7f779c004800 2024-03-24T01:25:23.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.684+0000 7f77b3e00640 1 --2- 172.21.15.90:0/1478372968 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f779806bc00 0x7f779806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:23.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.684+0000 7f77b3e00640 1 -- 172.21.15.90:0/1478372968 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f77ac08b3b0 con 0x7f779c004800 2024-03-24T01:25:23.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.685+0000 7f77bb400640 1 --2- 172.21.15.90:0/1478372968 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f779806bc00 0x7f779806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:23.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.686+0000 7f77bb400640 1 --2- 172.21.15.90:0/1478372968 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f779806bc00 0x7f779806e0c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f77a4036000 tx=0x7f77a4030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:23.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.690+0000 7f77b3e00640 1 -- 172.21.15.90:0/1478372968 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f77ac059a80 con 0x7f779c004800 2024-03-24T01:25:23.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.899+0000 7f77c1400640 1 -- 172.21.15.90:0/1478372968 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f77bc0ccf00 con 0x7f779c004800 2024-03-24T01:25:23.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.900+0000 7f77b3e00640 1 -- 172.21.15.90:0/1478372968 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f77ac059380 con 0x7f779c004800 2024-03-24T01:25:23.902 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-24T01:25:23.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.903+0000 7f77c1400640 1 -- 172.21.15.90:0/1478372968 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f779806bc00 msgr2=0x7f779806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:23.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.903+0000 7f77c1400640 1 --2- 172.21.15.90:0/1478372968 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f779806bc00 0x7f779806e0c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f77a4036000 tx=0x7f77a4030000 comp rx=0 tx=0).stop 2024-03-24T01:25:23.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.903+0000 7f77c1400640 1 -- 172.21.15.90:0/1478372968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f779c004800 msgr2=0x7f77bc145990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:23.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.903+0000 7f77c1400640 1 --2- 172.21.15.90:0/1478372968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f779c004800 0x7f77bc145990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f77ac00d800 tx=0x7f77ac00dcc0 comp rx=0 tx=0).stop 2024-03-24T01:25:23.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.903+0000 7f77c1400640 1 -- 172.21.15.90:0/1478372968 shutdown_connections 2024-03-24T01:25:23.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.903+0000 7f77c1400640 1 --2- 172.21.15.90:0/1478372968 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f779806bc00 0x7f779806e0c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:23.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.903+0000 7f77c1400640 1 --2- 172.21.15.90:0/1478372968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77bc156800 0x7f77bc156c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:23.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.903+0000 7f77c1400640 1 --2- 172.21.15.90:0/1478372968 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77bc0c9210 0x7f77bc0c9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:23.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.903+0000 7f77c1400640 1 --2- 172.21.15.90:0/1478372968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f779c004800 0x7f77bc145990 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:23.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.903+0000 7f77c1400640 1 -- 172.21.15.90:0/1478372968 >> 172.21.15.90:0/1478372968 conn(0x7f77bc0d0560 msgr2=0x7f77bc0cb530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:23.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.903+0000 7f77c1400640 1 -- 172.21.15.90:0/1478372968 shutdown_connections 2024-03-24T01:25:23.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:23.903+0000 7f77c1400640 1 -- 172.21.15.90:0/1478372968 wait complete. 2024-03-24T01:25:23.923 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi090.front.sepia.ceph.com... 2024-03-24T01:25:23.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:23.923 DEBUG:teuthology.orchestra.run.smithi090:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-24T01:25:24.009 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:25:24.009 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-03-24T01:25:24.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:24.010 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T01:25:24.063 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:25:24.063 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:25:24.120 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:25:24.120 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:25:24.170 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:25:24.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.383+0000 7fc02b000640 1 Processor -- start 2024-03-24T01:25:24.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.383+0000 7fc02b000640 1 -- start start 2024-03-24T01:25:24.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.384+0000 7fc02b000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0240cc8f0 0x7fc02406d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:24.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.384+0000 7fc02b000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0240cceb0 con 0x7fc024144420 2024-03-24T01:25:24.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.385+0000 7fc02b000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0240cd190 con 0x7fc024144800 2024-03-24T01:25:24.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.385+0000 7fc02b000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0240ceb70 con 0x7fc0240cc8f0 2024-03-24T01:25:24.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.385+0000 7fc029c00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc024144420 0x7fc0240cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:42366/0 (socket says 172.21.15.90:42366) 2024-03-24T01:25:24.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.385+0000 7fc029c00640 1 -- 172.21.15.90:0/2265882489 learned_addr learned my addr 172.21.15.90:0/2265882489 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:24.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.385+0000 7fc02a600640 1 --2- 172.21.15.90:0/2265882489 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0240cc8f0 0x7fc02406d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:24.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.385+0000 7fc023e00640 1 -- 172.21.15.90:0/2265882489 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 855971884 0 0) 0x7fc0240cceb0 con 0x7fc024144420 2024-03-24T01:25:24.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.385+0000 7fc023e00640 1 -- 172.21.15.90:0/2265882489 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc00c003520 con 0x7fc024144420 2024-03-24T01:25:24.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.386+0000 7fc023e00640 1 -- 172.21.15.90:0/2265882489 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 582824550 0 0) 0x7fc00c003520 con 0x7fc024144420 2024-03-24T01:25:24.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.386+0000 7fc023e00640 1 -- 172.21.15.90:0/2265882489 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0240cc8f0 msgr2=0x7fc02406d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:24.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.386+0000 7fc023e00640 1 --2- 172.21.15.90:0/2265882489 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0240cc8f0 0x7fc02406d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:24.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.386+0000 7fc023e00640 1 -- 172.21.15.90:0/2265882489 >> v1:172.21.15.120:6789/0 conn(0x7fc024144800 legacy=0x7fc0240cfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:24.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.386+0000 7fc023e00640 1 -- 172.21.15.90:0/2265882489 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0240cee50 con 0x7fc024144420 2024-03-24T01:25:24.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.386+0000 7fc023e00640 1 -- 172.21.15.90:0/2265882489 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7fc018001670 con 0x7fc024144420 2024-03-24T01:25:24.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.386+0000 7fc023e00640 1 -- 172.21.15.90:0/2265882489 >> v1:172.21.15.90:6789/0 conn(0x7fc024144420 legacy=0x7fc0240cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:24.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.386+0000 7fc023e00640 1 --2- 172.21.15.90:0/2265882489 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc00c0039e0 0x7fc00c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:24.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.386+0000 7fc023e00640 1 --2- 172.21.15.90:0/2265882489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc00c004800 0x7fc00c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:24.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.386+0000 7fc023e00640 1 --2- 172.21.15.90:0/2265882489 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0240cc8f0 0x7fc00c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:24.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.386+0000 7fc023e00640 1 -- 172.21.15.90:0/2265882489 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0240ceb70 con 0x7fc00c0039e0 2024-03-24T01:25:24.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.386+0000 7fc023e00640 1 -- 172.21.15.90:0/2265882489 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc00c00baf0 con 0x7fc00c004800 2024-03-24T01:25:24.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.386+0000 7fc023e00640 1 -- 172.21.15.90:0/2265882489 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc00c00bcf0 con 0x7fc0240cc8f0 2024-03-24T01:25:24.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.386+0000 7fc029c00640 1 --2- 172.21.15.90:0/2265882489 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc00c0039e0 0x7fc00c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:24.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.386+0000 7fc02a600640 1 --2- 172.21.15.90:0/2265882489 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0240cc8f0 0x7fc00c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:24.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.387+0000 7fc029200640 1 --2- 172.21.15.90:0/2265882489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc00c004800 0x7fc00c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:24.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.387+0000 7fc029c00640 1 -- 172.21.15.90:0/2265882489 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0240cc8f0 msgr2=0x7fc00c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:24.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.387+0000 7fc029c00640 1 --2- 172.21.15.90:0/2265882489 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0240cc8f0 0x7fc00c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:24.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.387+0000 7fc029c00640 1 -- 172.21.15.90:0/2265882489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc00c004800 msgr2=0x7fc00c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:24.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.387+0000 7fc029c00640 1 --2- 172.21.15.90:0/2265882489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc00c004800 0x7fc00c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:24.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.387+0000 7fc029c00640 1 -- 172.21.15.90:0/2265882489 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc00c003520 con 0x7fc00c0039e0 2024-03-24T01:25:24.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.387+0000 7fc029200640 1 --2- 172.21.15.90:0/2265882489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc00c004800 0x7fc00c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:24.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.388+0000 7fc029c00640 1 --2- 172.21.15.90:0/2265882489 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc00c0039e0 0x7fc00c003e00 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc01800c240 tx=0x7fc00c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b9ce9fcc71a854c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:24.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.388+0000 7fc023e00640 1 -- 172.21.15.90:0/2265882489 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc018007d10 con 0x7fc00c0039e0 2024-03-24T01:25:24.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.388+0000 7fc023e00640 1 -- 172.21.15.90:0/2265882489 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc018030070 con 0x7fc00c0039e0 2024-03-24T01:25:24.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.388+0000 7fc023e00640 1 -- 172.21.15.90:0/2265882489 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc01802cd60 con 0x7fc00c0039e0 2024-03-24T01:25:24.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.389+0000 7fc02b000640 1 -- 172.21.15.90:0/2265882489 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc00c0039e0 msgr2=0x7fc00c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:24.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.389+0000 7fc02b000640 1 --2- 172.21.15.90:0/2265882489 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc00c0039e0 0x7fc00c003e00 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc01800c240 tx=0x7fc00c005830 comp rx=0 tx=0).stop 2024-03-24T01:25:24.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.389+0000 7fc02a600640 1 -- 172.21.15.90:0/2265882489 reap_dead start 2024-03-24T01:25:24.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.389+0000 7fc02b000640 1 -- 172.21.15.90:0/2265882489 shutdown_connections 2024-03-24T01:25:24.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.389+0000 7fc02b000640 1 -- 172.21.15.90:0/2265882489 >> 172.21.15.90:0/2265882489 conn(0x7fc02406b0c0 msgr2=0x7fc02406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:24.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.389+0000 7fc02b000640 1 -- 172.21.15.90:0/2265882489 shutdown_connections 2024-03-24T01:25:24.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.389+0000 7fc02b000640 1 -- 172.21.15.90:0/2265882489 wait complete. 2024-03-24T01:25:24.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.390+0000 7fc02b000640 1 Processor -- start 2024-03-24T01:25:24.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.391+0000 7fc02b000640 1 -- start start 2024-03-24T01:25:24.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.391+0000 7fc02b000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc00c0039e0 0x7fc024153000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:24.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.392+0000 7fc02b000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0241f1180 0x7fc0240ad450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:24.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.392+0000 7fc02b000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0240ad990 0x7fc0240a93f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:24.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.392+0000 7fc02b000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02406da80 con 0x7fc0241f1180 2024-03-24T01:25:24.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.392+0000 7fc02b000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02406d900 con 0x7fc00c0039e0 2024-03-24T01:25:24.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.392+0000 7fc02b000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc02406dc00 con 0x7fc0240ad990 2024-03-24T01:25:24.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.392+0000 7fc029200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0241f1180 0x7fc0240ad450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:24.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.392+0000 7fc029200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0241f1180 0x7fc0240ad450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43118/0 (socket says 172.21.15.90:43118) 2024-03-24T01:25:24.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.392+0000 7fc029200640 1 -- 172.21.15.90:0/1290370562 learned_addr learned my addr 172.21.15.90:0/1290370562 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:24.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.392+0000 7fc029c00640 1 --2- 172.21.15.90:0/1290370562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc00c0039e0 0x7fc024153000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:24.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.392+0000 7fc02a600640 1 --2- 172.21.15.90:0/1290370562 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0240ad990 0x7fc0240a93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:24.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.392+0000 7fc029200640 1 -- 172.21.15.90:0/1290370562 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0240ad990 msgr2=0x7fc0240a93f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:24.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.392+0000 7fc029200640 1 --2- 172.21.15.90:0/1290370562 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0240ad990 0x7fc0240a93f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:24.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.392+0000 7fc029200640 1 -- 172.21.15.90:0/1290370562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc00c0039e0 msgr2=0x7fc024153000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:24.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.392+0000 7fc029200640 1 --2- 172.21.15.90:0/1290370562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc00c0039e0 0x7fc024153000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:24.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.392+0000 7fc029200640 1 -- 172.21.15.90:0/1290370562 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0240aad20 con 0x7fc0241f1180 2024-03-24T01:25:24.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.393+0000 7fc029200640 1 --2- 172.21.15.90:0/1290370562 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0241f1180 0x7fc0240ad450 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc01000d770 tx=0x7fc01000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:24.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.393+0000 7fc021600640 1 -- 172.21.15.90:0/1290370562 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc010005270 con 0x7fc0241f1180 2024-03-24T01:25:24.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.393+0000 7fc021600640 1 -- 172.21.15.90:0/1290370562 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc010017070 con 0x7fc0241f1180 2024-03-24T01:25:24.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.393+0000 7fc021600640 1 -- 172.21.15.90:0/1290370562 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc01000b6b0 con 0x7fc0241f1180 2024-03-24T01:25:24.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.393+0000 7fc02b000640 1 -- 172.21.15.90:0/1290370562 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc024153540 con 0x7fc0241f1180 2024-03-24T01:25:24.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.393+0000 7fc02b000640 1 -- 172.21.15.90:0/1290370562 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0240ab2a0 con 0x7fc0241f1180 2024-03-24T01:25:24.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.394+0000 7fc02b000640 1 -- 172.21.15.90:0/1290370562 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc02404f400 con 0x7fc0241f1180 2024-03-24T01:25:24.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.395+0000 7fc021600640 1 -- 172.21.15.90:0/1290370562 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc01000b850 con 0x7fc0241f1180 2024-03-24T01:25:24.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.396+0000 7fc021600640 1 --2- 172.21.15.90:0/1290370562 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc00406bbd0 0x7fc00406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:24.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.396+0000 7fc021600640 1 -- 172.21.15.90:0/1290370562 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7fc01008ad40 con 0x7fc0241f1180 2024-03-24T01:25:24.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.397+0000 7fc029c00640 1 --2- 172.21.15.90:0/1290370562 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc00406bbd0 0x7fc00406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:24.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.398+0000 7fc029c00640 1 --2- 172.21.15.90:0/1290370562 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc00406bbd0 0x7fc00406e090 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc018034000 tx=0x7fc018009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:24.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.400+0000 7fc021600640 1 -- 172.21.15.90:0/1290370562 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc010019310 con 0x7fc0241f1180 2024-03-24T01:25:24.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.596+0000 7fc02b000640 1 -- 172.21.15.90:0/1290370562 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0240add70 con 0x7fc0241f1180 2024-03-24T01:25:24.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.598+0000 7fc021600640 1 -- 172.21.15.90:0/1290370562 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18925 (secure 0 0 0) 0x7fc010001230 con 0x7fc0241f1180 2024-03-24T01:25:24.600 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:25:24.600 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:24:53.236401+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:24:08.643966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T01:24:09.189904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T01:24:13.027323+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:25:24.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.601+0000 7fc02b000640 1 -- 172.21.15.90:0/1290370562 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc00406bbd0 msgr2=0x7fc00406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:24.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.601+0000 7fc02b000640 1 --2- 172.21.15.90:0/1290370562 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc00406bbd0 0x7fc00406e090 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc018034000 tx=0x7fc018009030 comp rx=0 tx=0).stop 2024-03-24T01:25:24.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.601+0000 7fc02b000640 1 -- 172.21.15.90:0/1290370562 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0241f1180 msgr2=0x7fc0240ad450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:24.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.601+0000 7fc02b000640 1 --2- 172.21.15.90:0/1290370562 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0241f1180 0x7fc0240ad450 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc01000d770 tx=0x7fc01000dc30 comp rx=0 tx=0).stop 2024-03-24T01:25:24.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.602+0000 7fc02b000640 1 -- 172.21.15.90:0/1290370562 shutdown_connections 2024-03-24T01:25:24.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.602+0000 7fc02b000640 1 --2- 172.21.15.90:0/1290370562 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc00406bbd0 0x7fc00406e090 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:24.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.602+0000 7fc02b000640 1 --2- 172.21.15.90:0/1290370562 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0241f1180 0x7fc0240ad450 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:24.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.602+0000 7fc02b000640 1 --2- 172.21.15.90:0/1290370562 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0240ad990 0x7fc0240a93f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:24.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.602+0000 7fc02b000640 1 --2- 172.21.15.90:0/1290370562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc00c0039e0 0x7fc024153000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:24.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.602+0000 7fc02b000640 1 -- 172.21.15.90:0/1290370562 >> 172.21.15.90:0/1290370562 conn(0x7fc02406b0c0 msgr2=0x7fc024142d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:24.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.602+0000 7fc02b000640 1 -- 172.21.15.90:0/1290370562 shutdown_connections 2024-03-24T01:25:24.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.602+0000 7fc02b000640 1 -- 172.21.15.90:0/1290370562 wait complete. 2024-03-24T01:25:24.620 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T01:24:08.643966+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T01:24:09.189904+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T01:24:13.027323+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T01:25:24.621 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:25:24.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.816+0000 7ff3ab400640 1 Processor -- start 2024-03-24T01:25:24.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.816+0000 7ff3ab400640 1 -- start start 2024-03-24T01:25:24.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.817+0000 7ff3ab400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff3a40aeeb0 0x7ff3a40b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:24.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.817+0000 7ff3ab400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff3a415bc50 0x7ff3a406b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:24.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.817+0000 7ff3ab400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a406dac0 con 0x7ff3a406b600 2024-03-24T01:25:24.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.817+0000 7ff3ab400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3a406ba70 con 0x7ff3a40aeeb0 2024-03-24T01:25:24.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.817+0000 7ff3ab400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3a40b4da0 con 0x7ff3a415bc50 2024-03-24T01:25:24.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.818+0000 7ff3a8a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff3a40aeeb0 0x7ff3a40b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:24.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.818+0000 7ff3a8a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff3a40aeeb0 0x7ff3a40b44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43132/0 (socket says 172.21.15.90:43132) 2024-03-24T01:25:24.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.818+0000 7ff3a8a00640 1 -- 172.21.15.90:0/629921005 learned_addr learned my addr 172.21.15.90:0/629921005 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:24.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.818+0000 7ff3a9400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff3a406b600 0x7ff3a415e590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:42374/0 (socket says 172.21.15.90:42374) 2024-03-24T01:25:24.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.818+0000 7ff39be00640 1 --2- 172.21.15.90:0/629921005 >> v2:172.21.15.120:3300/0 conn(0x7ff3a415bc50 0x7ff3a406b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:24.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.818+0000 7ff3a8a00640 1 -- 172.21.15.90:0/629921005 >> v2:172.21.15.120:3300/0 conn(0x7ff3a415bc50 msgr2=0x7ff3a406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:24.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.818+0000 7ff3a8a00640 1 --2- 172.21.15.90:0/629921005 >> v2:172.21.15.120:3300/0 conn(0x7ff3a415bc50 0x7ff3a406b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:24.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.818+0000 7ff3a8a00640 1 -- 172.21.15.90:0/629921005 >> v1:172.21.15.90:6789/0 conn(0x7ff3a406b600 legacy=0x7ff3a415e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:24.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.818+0000 7ff3a8a00640 1 -- 172.21.15.90:0/629921005 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a406dda0 con 0x7ff3a40aeeb0 2024-03-24T01:25:24.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.819+0000 7ff3a8a00640 1 --2- 172.21.15.90:0/629921005 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3a40aeeb0 0x7ff3a40b44c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff3a415c0e0 tx=0x7ff39402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a58c0e1685b71f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:24.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.820+0000 7ff3a3e00640 1 -- 172.21.15.90:0/629921005 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39402e070 con 0x7ff3a40aeeb0 2024-03-24T01:25:24.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.820+0000 7ff3a3e00640 1 -- 172.21.15.90:0/629921005 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff394037070 con 0x7ff3a40aeeb0 2024-03-24T01:25:24.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.820+0000 7ff3a3e00640 1 -- 172.21.15.90:0/629921005 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff394031a00 con 0x7ff3a40aeeb0 2024-03-24T01:25:24.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.820+0000 7ff3ab400640 1 -- 172.21.15.90:0/629921005 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3a40aeeb0 msgr2=0x7ff3a40b44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:24.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.820+0000 7ff3ab400640 1 --2- 172.21.15.90:0/629921005 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3a40aeeb0 0x7ff3a40b44c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff3a415c0e0 tx=0x7ff39402b880 comp rx=0 tx=0).stop 2024-03-24T01:25:24.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.820+0000 7ff3ab400640 1 -- 172.21.15.90:0/629921005 shutdown_connections 2024-03-24T01:25:24.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.820+0000 7ff3ab400640 1 --2- 172.21.15.90:0/629921005 >> v2:172.21.15.120:3300/0 conn(0x7ff3a415bc50 0x7ff3a406b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:24.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.820+0000 7ff3ab400640 1 --2- 172.21.15.90:0/629921005 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3a40aeeb0 0x7ff3a40b44c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:24.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.820+0000 7ff3ab400640 1 -- 172.21.15.90:0/629921005 >> 172.21.15.90:0/629921005 conn(0x7ff3a40b1770 msgr2=0x7ff3a40b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:24.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.820+0000 7ff3ab400640 1 -- 172.21.15.90:0/629921005 shutdown_connections 2024-03-24T01:25:24.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.821+0000 7ff3ab400640 1 -- 172.21.15.90:0/629921005 wait complete. 2024-03-24T01:25:24.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.822+0000 7ff3ab400640 1 Processor -- start 2024-03-24T01:25:24.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.822+0000 7ff3ab400640 1 -- start start 2024-03-24T01:25:24.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.823+0000 7ff3ab400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3a406b600 0x7ff3a415d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:24.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.823+0000 7ff3ab400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3a40aeeb0 0x7ff3a4141320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:24.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.823+0000 7ff3ab400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3a415bc50 0x7ff3a415c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:24.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.823+0000 7ff3ab400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a40af290 con 0x7ff3a40aeeb0 2024-03-24T01:25:24.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.823+0000 7ff3ab400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a406e360 con 0x7ff3a415bc50 2024-03-24T01:25:24.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.823+0000 7ff3ab400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a40b4a00 con 0x7ff3a406b600 2024-03-24T01:25:24.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.824+0000 7ff39be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3a40aeeb0 0x7ff3a4141320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:24.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.824+0000 7ff39be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3a40aeeb0 0x7ff3a4141320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43144/0 (socket says 172.21.15.90:43144) 2024-03-24T01:25:24.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.824+0000 7ff39be00640 1 -- 172.21.15.90:0/2338566739 learned_addr learned my addr 172.21.15.90:0/2338566739 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:24.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.824+0000 7ff3a9400640 1 --2- 172.21.15.90:0/2338566739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3a415bc50 0x7ff3a415c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:24.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.824+0000 7ff39be00640 1 -- 172.21.15.90:0/2338566739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3a406b600 msgr2=0x7ff3a415d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:24.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.824+0000 7ff39be00640 1 --2- 172.21.15.90:0/2338566739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3a406b600 0x7ff3a415d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:24.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.824+0000 7ff39be00640 1 -- 172.21.15.90:0/2338566739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3a415bc50 msgr2=0x7ff3a415c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:24.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.824+0000 7ff39be00640 1 --2- 172.21.15.90:0/2338566739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3a415bc50 0x7ff3a415c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:24.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.824+0000 7ff39be00640 1 -- 172.21.15.90:0/2338566739 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a415c7e0 con 0x7ff3a40aeeb0 2024-03-24T01:25:24.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.824+0000 7ff3a9400640 1 --2- 172.21.15.90:0/2338566739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3a415bc50 0x7ff3a415c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:24.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.824+0000 7ff39be00640 1 --2- 172.21.15.90:0/2338566739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3a40aeeb0 0x7ff3a4141320 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff38c00d620 tx=0x7ff38c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:24.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.825+0000 7ff3a1600640 1 -- 172.21.15.90:0/2338566739 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38c00b620 con 0x7ff3a40aeeb0 2024-03-24T01:25:24.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.825+0000 7ff3a1600640 1 -- 172.21.15.90:0/2338566739 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff38c019070 con 0x7ff3a40aeeb0 2024-03-24T01:25:24.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.825+0000 7ff3a1600640 1 -- 172.21.15.90:0/2338566739 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38c00ba00 con 0x7ff3a40aeeb0 2024-03-24T01:25:24.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.825+0000 7ff3ab400640 1 -- 172.21.15.90:0/2338566739 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3a415cac0 con 0x7ff3a40aeeb0 2024-03-24T01:25:24.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.825+0000 7ff3ab400640 1 -- 172.21.15.90:0/2338566739 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3a415d080 con 0x7ff3a40aeeb0 2024-03-24T01:25:24.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.826+0000 7ff3ab400640 1 -- 172.21.15.90:0/2338566739 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3a404f400 con 0x7ff3a40aeeb0 2024-03-24T01:25:24.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.827+0000 7ff3a1600640 1 -- 172.21.15.90:0/2338566739 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff38c00bba0 con 0x7ff3a40aeeb0 2024-03-24T01:25:24.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.828+0000 7ff3a1600640 1 --2- 172.21.15.90:0/2338566739 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff38006bbd0 0x7ff38006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:24.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.828+0000 7ff3a1600640 1 -- 172.21.15.90:0/2338566739 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7ff38c08b290 con 0x7ff3a40aeeb0 2024-03-24T01:25:24.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.828+0000 7ff3a8a00640 1 --2- 172.21.15.90:0/2338566739 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff38006bbd0 0x7ff38006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:24.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.829+0000 7ff3a8a00640 1 --2- 172.21.15.90:0/2338566739 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff38006bbd0 0x7ff38006e090 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff394036000 tx=0x7ff394030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:24.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:24.832+0000 7ff3a1600640 1 -- 172.21.15.90:0/2338566739 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff38c059950 con 0x7ff3a40aeeb0 2024-03-24T01:25:25.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.030+0000 7ff3ab400640 1 -- 172.21.15.90:0/2338566739 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff3a4157890 con 0x7ff3a40aeeb0 2024-03-24T01:25:25.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.031+0000 7ff3a1600640 1 -- 172.21.15.90:0/2338566739 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff38c059250 con 0x7ff3a40aeeb0 2024-03-24T01:25:25.032 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:25:25.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.033+0000 7ff3ab400640 1 -- 172.21.15.90:0/2338566739 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff38006bbd0 msgr2=0x7ff38006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:25.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.033+0000 7ff3ab400640 1 --2- 172.21.15.90:0/2338566739 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff38006bbd0 0x7ff38006e090 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff394036000 tx=0x7ff394030000 comp rx=0 tx=0).stop 2024-03-24T01:25:25.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.034+0000 7ff3ab400640 1 -- 172.21.15.90:0/2338566739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3a40aeeb0 msgr2=0x7ff3a4141320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:25.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.034+0000 7ff3ab400640 1 --2- 172.21.15.90:0/2338566739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3a40aeeb0 0x7ff3a4141320 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff38c00d620 tx=0x7ff38c00dae0 comp rx=0 tx=0).stop 2024-03-24T01:25:25.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.034+0000 7ff3ab400640 1 -- 172.21.15.90:0/2338566739 shutdown_connections 2024-03-24T01:25:25.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.034+0000 7ff3ab400640 1 --2- 172.21.15.90:0/2338566739 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff38006bbd0 0x7ff38006e090 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:25.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.034+0000 7ff3ab400640 1 --2- 172.21.15.90:0/2338566739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3a40aeeb0 0x7ff3a4141320 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:25.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.034+0000 7ff3ab400640 1 --2- 172.21.15.90:0/2338566739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3a415bc50 0x7ff3a415c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:25.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.034+0000 7ff3ab400640 1 --2- 172.21.15.90:0/2338566739 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff3a406b600 0x7ff3a415d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:25.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.034+0000 7ff3ab400640 1 -- 172.21.15.90:0/2338566739 >> 172.21.15.90:0/2338566739 conn(0x7ff3a40b1770 msgr2=0x7ff3a4156b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:25.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.034+0000 7ff3ab400640 1 -- 172.21.15.90:0/2338566739 shutdown_connections 2024-03-24T01:25:25.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.034+0000 7ff3ab400640 1 -- 172.21.15.90:0/2338566739 wait complete. 2024-03-24T01:25:25.054 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T01:25:25.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.267+0000 7fc094a00640 1 Processor -- start 2024-03-24T01:25:25.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.267+0000 7fc094a00640 1 -- start start 2024-03-24T01:25:25.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.269+0000 7fc094a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc090145170 0x7fc0900cf810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:25.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.269+0000 7fc094a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc0900cf190 0x7fc0900cfd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:25.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.269+0000 7fc094a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc09006d990 con 0x7fc09006ea30 2024-03-24T01:25:25.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.269+0000 7fc094a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc09006dc70 con 0x7fc0900cf190 2024-03-24T01:25:25.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.269+0000 7fc094a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc09006df50 con 0x7fc090145170 2024-03-24T01:25:25.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.269+0000 7fc08f200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc0900cf190 0x7fc0900cfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:25.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.269+0000 7fc08f200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc0900cf190 0x7fc0900cfd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43160/0 (socket says 172.21.15.90:43160) 2024-03-24T01:25:25.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.269+0000 7fc08f200640 1 -- 172.21.15.90:0/4065043707 learned_addr learned my addr 172.21.15.90:0/4065043707 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:25.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.270+0000 7fc08e800640 1 --2- 172.21.15.90:0/4065043707 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc090145170 0x7fc0900cf810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:25.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.270+0000 7fc08f200640 1 -- 172.21.15.90:0/4065043707 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc090145170 msgr2=0x7fc0900cf810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:25.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.270+0000 7fc08f200640 1 --2- 172.21.15.90:0/4065043707 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc090145170 0x7fc0900cf810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:25.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.270+0000 7fc08f200640 1 -- 172.21.15.90:0/4065043707 >> v1:172.21.15.90:6789/0 conn(0x7fc09006ea30 legacy=0x7fc0900cea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:25.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.270+0000 7fc08f200640 1 -- 172.21.15.90:0/4065043707 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc09006e230 con 0x7fc0900cf190 2024-03-24T01:25:25.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.271+0000 7fc08f200640 1 --2- 172.21.15.90:0/4065043707 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0900cf190 0x7fc0900cfd50 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc088009e50 tx=0x7fc08802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a336c752b5d789f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:25.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.271+0000 7fc08de00640 1 -- 172.21.15.90:0/4065043707 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08802cb00 con 0x7fc0900cf190 2024-03-24T01:25:25.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.271+0000 7fc08de00640 1 -- 172.21.15.90:0/4065043707 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc08802cca0 con 0x7fc0900cf190 2024-03-24T01:25:25.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.271+0000 7fc08de00640 1 -- 172.21.15.90:0/4065043707 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc088034610 con 0x7fc0900cf190 2024-03-24T01:25:25.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.272+0000 7fc094a00640 1 -- 172.21.15.90:0/4065043707 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0900cf190 msgr2=0x7fc0900cfd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:25.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.272+0000 7fc094a00640 1 --2- 172.21.15.90:0/4065043707 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0900cf190 0x7fc0900cfd50 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc088009e50 tx=0x7fc08802bf60 comp rx=0 tx=0).stop 2024-03-24T01:25:25.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.272+0000 7fc094a00640 1 -- 172.21.15.90:0/4065043707 shutdown_connections 2024-03-24T01:25:25.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.272+0000 7fc094a00640 1 --2- 172.21.15.90:0/4065043707 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0900cf190 0x7fc0900cfd50 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:25.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.272+0000 7fc094a00640 1 --2- 172.21.15.90:0/4065043707 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc090145170 0x7fc0900cf810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:25.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.272+0000 7fc094a00640 1 -- 172.21.15.90:0/4065043707 >> 172.21.15.90:0/4065043707 conn(0x7fc09006b0c0 msgr2=0x7fc09006b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:25.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.272+0000 7fc094a00640 1 -- 172.21.15.90:0/4065043707 shutdown_connections 2024-03-24T01:25:25.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.272+0000 7fc094a00640 1 -- 172.21.15.90:0/4065043707 wait complete. 2024-03-24T01:25:25.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.274+0000 7fc094a00640 1 Processor -- start 2024-03-24T01:25:25.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.274+0000 7fc094a00640 1 -- start start 2024-03-24T01:25:25.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.275+0000 7fc094a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc09006ea30 0x7fc0900c69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:25.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.275+0000 7fc094a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0900cf190 0x7fc09013a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:25.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.275+0000 7fc094a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc090145170 0x7fc090141ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:25.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.275+0000 7fc094a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0901456d0 con 0x7fc09006ea30 2024-03-24T01:25:25.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.275+0000 7fc094a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc090145550 con 0x7fc090145170 2024-03-24T01:25:25.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.275+0000 7fc094a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc09015c0a0 con 0x7fc0900cf190 2024-03-24T01:25:25.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.275+0000 7fc08e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc09006ea30 0x7fc0900c69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:25.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.275+0000 7fc08e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc09006ea30 0x7fc0900c69c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43174/0 (socket says 172.21.15.90:43174) 2024-03-24T01:25:25.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.275+0000 7fc08e800640 1 -- 172.21.15.90:0/4170855542 learned_addr learned my addr 172.21.15.90:0/4170855542 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:25.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.275+0000 7fc08e800640 1 -- 172.21.15.90:0/4170855542 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0900cf190 msgr2=0x7fc09013a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:25.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.275+0000 7fc087e00640 1 --2- 172.21.15.90:0/4170855542 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0900cf190 0x7fc09013a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:25.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.276+0000 7fc08f200640 1 --2- 172.21.15.90:0/4170855542 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc090145170 0x7fc090141ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:25.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.276+0000 7fc08e800640 1 --2- 172.21.15.90:0/4170855542 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0900cf190 0x7fc09013a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:25.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.276+0000 7fc08e800640 1 -- 172.21.15.90:0/4170855542 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc090145170 msgr2=0x7fc090141ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:25.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.276+0000 7fc08e800640 1 --2- 172.21.15.90:0/4170855542 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc090145170 0x7fc090141ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:25.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.276+0000 7fc08e800640 1 -- 172.21.15.90:0/4170855542 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0901f3d40 con 0x7fc09006ea30 2024-03-24T01:25:25.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.276+0000 7fc087e00640 1 --2- 172.21.15.90:0/4170855542 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0900cf190 0x7fc09013a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:25.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.276+0000 7fc08e800640 1 --2- 172.21.15.90:0/4170855542 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc09006ea30 0x7fc0900c69c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fc08000d5d0 tx=0x7fc08000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:25.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.276+0000 7fc086a00640 1 -- 172.21.15.90:0/4170855542 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0800125f0 con 0x7fc09006ea30 2024-03-24T01:25:25.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.276+0000 7fc086a00640 1 -- 172.21.15.90:0/4170855542 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc080019070 con 0x7fc09006ea30 2024-03-24T01:25:25.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.276+0000 7fc086a00640 1 -- 172.21.15.90:0/4170855542 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0800129f0 con 0x7fc09006ea30 2024-03-24T01:25:25.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.276+0000 7fc094a00640 1 -- 172.21.15.90:0/4170855542 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc09013acd0 con 0x7fc09006ea30 2024-03-24T01:25:25.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.276+0000 7fc094a00640 1 -- 172.21.15.90:0/4170855542 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc090142ab0 con 0x7fc09006ea30 2024-03-24T01:25:25.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.277+0000 7fc094a00640 1 -- 172.21.15.90:0/4170855542 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc09004f400 con 0x7fc09006ea30 2024-03-24T01:25:25.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.278+0000 7fc086a00640 1 -- 172.21.15.90:0/4170855542 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc080012b90 con 0x7fc09006ea30 2024-03-24T01:25:25.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.279+0000 7fc086a00640 1 --2- 172.21.15.90:0/4170855542 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc06c06bc00 0x7fc06c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:25.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.279+0000 7fc086a00640 1 -- 172.21.15.90:0/4170855542 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7fc08008b380 con 0x7fc09006ea30 2024-03-24T01:25:25.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.280+0000 7fc087e00640 1 --2- 172.21.15.90:0/4170855542 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc06c06bc00 0x7fc06c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:25.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.281+0000 7fc087e00640 1 --2- 172.21.15.90:0/4170855542 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc06c06bc00 0x7fc06c06e0c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc090143210 tx=0x7fc078009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:25.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.283+0000 7fc086a00640 1 -- 172.21.15.90:0/4170855542 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc0800599b0 con 0x7fc09006ea30 2024-03-24T01:25:25.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.486+0000 7fc094a00640 1 -- 172.21.15.90:0/4170855542 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc0900ae2a0 con 0x7fc09006ea30 2024-03-24T01:25:25.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.487+0000 7fc086a00640 1 -- 172.21.15.90:0/4170855542 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc0800592b0 con 0x7fc09006ea30 2024-03-24T01:25:25.488 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:25:25.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.490+0000 7fc094a00640 1 -- 172.21.15.90:0/4170855542 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc06c06bc00 msgr2=0x7fc06c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:25.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.490+0000 7fc094a00640 1 --2- 172.21.15.90:0/4170855542 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc06c06bc00 0x7fc06c06e0c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc090143210 tx=0x7fc078009040 comp rx=0 tx=0).stop 2024-03-24T01:25:25.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.490+0000 7fc094a00640 1 -- 172.21.15.90:0/4170855542 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc09006ea30 msgr2=0x7fc0900c69c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:25.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.490+0000 7fc094a00640 1 --2- 172.21.15.90:0/4170855542 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc09006ea30 0x7fc0900c69c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fc08000d5d0 tx=0x7fc08000da90 comp rx=0 tx=0).stop 2024-03-24T01:25:25.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.490+0000 7fc094a00640 1 -- 172.21.15.90:0/4170855542 shutdown_connections 2024-03-24T01:25:25.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.490+0000 7fc094a00640 1 --2- 172.21.15.90:0/4170855542 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc06c06bc00 0x7fc06c06e0c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:25.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.490+0000 7fc094a00640 1 --2- 172.21.15.90:0/4170855542 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc090145170 0x7fc090141ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:25.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.490+0000 7fc094a00640 1 --2- 172.21.15.90:0/4170855542 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0900cf190 0x7fc09013a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:25.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.490+0000 7fc094a00640 1 --2- 172.21.15.90:0/4170855542 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc09006ea30 0x7fc0900c69c0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:25.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.490+0000 7fc094a00640 1 -- 172.21.15.90:0/4170855542 >> 172.21.15.90:0/4170855542 conn(0x7fc09006b0c0 msgr2=0x7fc0900acc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:25.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.490+0000 7fc094a00640 1 -- 172.21.15.90:0/4170855542 shutdown_connections 2024-03-24T01:25:25.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.490+0000 7fc094a00640 1 -- 172.21.15.90:0/4170855542 wait complete. 2024-03-24T01:25:25.511 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T01:25:25.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.728+0000 7f8f2d800640 1 Processor -- start 2024-03-24T01:25:25.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.728+0000 7f8f2d800640 1 -- start start 2024-03-24T01:25:25.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.730+0000 7f8f2d800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8f2806b7d0 0x7f8f280cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:25.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.730+0000 7f8f2d800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8f280cef80 0x7f8f280ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:25.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.730+0000 7f8f2d800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f2806bca0 con 0x7f8f28159620 2024-03-24T01:25:25.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.730+0000 7f8f2d800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f280b4470 con 0x7f8f2806b7d0 2024-03-24T01:25:25.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.730+0000 7f8f2d800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f280b36f0 con 0x7f8f280cef80 2024-03-24T01:25:25.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.730+0000 7f8f1fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8f2806b7d0 0x7f8f280cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:25.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.730+0000 7f8f1fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8f2806b7d0 0x7f8f280cea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43184/0 (socket says 172.21.15.90:43184) 2024-03-24T01:25:25.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.730+0000 7f8f1fe00640 1 -- 172.21.15.90:0/1561235719 learned_addr learned my addr 172.21.15.90:0/1561235719 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:25.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.730+0000 7f8f27400640 1 --2- 172.21.15.90:0/1561235719 >> v2:172.21.15.120:3300/0 conn(0x7f8f280cef80 0x7f8f280ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:25.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.730+0000 7f8f1fe00640 1 -- 172.21.15.90:0/1561235719 >> v2:172.21.15.120:3300/0 conn(0x7f8f280cef80 msgr2=0x7f8f280ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:25.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.730+0000 7f8f1fe00640 1 --2- 172.21.15.90:0/1561235719 >> v2:172.21.15.120:3300/0 conn(0x7f8f280cef80 0x7f8f280ae530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:25.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.730+0000 7f8f1fe00640 1 -- 172.21.15.90:0/1561235719 >> v1:172.21.15.120:6789/0 conn(0x7f8f28159620 legacy=0x7f8f2806b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:25.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.730+0000 7f8f1fe00640 1 -- 172.21.15.90:0/1561235719 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f280b3990 con 0x7f8f2806b7d0 2024-03-24T01:25:25.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.730+0000 7f8f27400640 1 --2- 172.21.15.90:0/1561235719 >> v2:172.21.15.120:3300/0 conn(0x7f8f280cef80 0x7f8f280ae530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:25.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.732+0000 7f8f1fe00640 1 --2- 172.21.15.90:0/1561235719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f2806b7d0 0x7f8f280cea40 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8f280b29d0 tx=0x7f8f1002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d3dea75c2e19ec3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:25.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.732+0000 7f8f26000640 1 -- 172.21.15.90:0/1561235719 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1002e070 con 0x7f8f2806b7d0 2024-03-24T01:25:25.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.732+0000 7f8f26000640 1 -- 172.21.15.90:0/1561235719 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f10037070 con 0x7f8f2806b7d0 2024-03-24T01:25:25.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.732+0000 7f8f26000640 1 -- 172.21.15.90:0/1561235719 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f10031a00 con 0x7f8f2806b7d0 2024-03-24T01:25:25.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.732+0000 7f8f2d800640 1 -- 172.21.15.90:0/1561235719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f2806b7d0 msgr2=0x7f8f280cea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:25.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.732+0000 7f8f2d800640 1 --2- 172.21.15.90:0/1561235719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f2806b7d0 0x7f8f280cea40 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8f280b29d0 tx=0x7f8f1002b880 comp rx=0 tx=0).stop 2024-03-24T01:25:25.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.733+0000 7f8f2d800640 1 -- 172.21.15.90:0/1561235719 shutdown_connections 2024-03-24T01:25:25.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.733+0000 7f8f2d800640 1 --2- 172.21.15.90:0/1561235719 >> v2:172.21.15.120:3300/0 conn(0x7f8f280cef80 0x7f8f280ae530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:25.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.733+0000 7f8f2d800640 1 --2- 172.21.15.90:0/1561235719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f2806b7d0 0x7f8f280cea40 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:25.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.733+0000 7f8f2d800640 1 -- 172.21.15.90:0/1561235719 >> 172.21.15.90:0/1561235719 conn(0x7f8f280b1770 msgr2=0x7f8f280b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:25.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.733+0000 7f8f2d800640 1 -- 172.21.15.90:0/1561235719 shutdown_connections 2024-03-24T01:25:25.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.733+0000 7f8f2d800640 1 -- 172.21.15.90:0/1561235719 wait complete. 2024-03-24T01:25:25.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.734+0000 7f8f2d800640 1 Processor -- start 2024-03-24T01:25:25.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.735+0000 7f8f2d800640 1 -- start start 2024-03-24T01:25:25.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.735+0000 7f8f2d800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f2806b7d0 0x7f8f28154440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:25.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.735+0000 7f8f2d800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f280cef80 0x7f8f2813e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:25.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.735+0000 7f8f2d800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f28159620 0x7f8f281551d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:25.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.735+0000 7f8f2d800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f280cf430 con 0x7f8f280cef80 2024-03-24T01:25:25.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.736+0000 7f8f2d800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f280b3f50 con 0x7f8f28159620 2024-03-24T01:25:25.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.736+0000 7f8f2d800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f28159a00 con 0x7f8f2806b7d0 2024-03-24T01:25:25.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.736+0000 7f8f1fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f280cef80 0x7f8f2813e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:25.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.736+0000 7f8f1fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f280cef80 0x7f8f2813e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43188/0 (socket says 172.21.15.90:43188) 2024-03-24T01:25:25.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.736+0000 7f8f1fe00640 1 -- 172.21.15.90:0/1359699331 learned_addr learned my addr 172.21.15.90:0/1359699331 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:25.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.736+0000 7f8f27400640 1 --2- 172.21.15.90:0/1359699331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f28159620 0x7f8f281551d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:25.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.736+0000 7f8f1fe00640 1 -- 172.21.15.90:0/1359699331 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f2806b7d0 msgr2=0x7f8f28154440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:25.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.736+0000 7f8f26a00640 1 --2- 172.21.15.90:0/1359699331 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f2806b7d0 0x7f8f28154440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:25.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.736+0000 7f8f1fe00640 1 --2- 172.21.15.90:0/1359699331 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f2806b7d0 0x7f8f28154440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:25.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.736+0000 7f8f1fe00640 1 -- 172.21.15.90:0/1359699331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f28159620 msgr2=0x7f8f281551d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:25.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.736+0000 7f8f1fe00640 1 --2- 172.21.15.90:0/1359699331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f28159620 0x7f8f281551d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:25.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.736+0000 7f8f1fe00640 1 -- 172.21.15.90:0/1359699331 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f281533a0 con 0x7f8f280cef80 2024-03-24T01:25:25.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.736+0000 7f8f27400640 1 --2- 172.21.15.90:0/1359699331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f28159620 0x7f8f281551d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:25.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.737+0000 7f8f1fe00640 1 --2- 172.21.15.90:0/1359699331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f280cef80 0x7f8f2813e180 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8f10000c00 tx=0x7f8f100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:25.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.737+0000 7f8f1ea00640 1 -- 172.21.15.90:0/1359699331 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f10032880 con 0x7f8f280cef80 2024-03-24T01:25:25.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.737+0000 7f8f1ea00640 1 -- 172.21.15.90:0/1359699331 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f10032a20 con 0x7f8f280cef80 2024-03-24T01:25:25.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.737+0000 7f8f1ea00640 1 -- 172.21.15.90:0/1359699331 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f10032e10 con 0x7f8f280cef80 2024-03-24T01:25:25.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.737+0000 7f8f2d800640 1 -- 172.21.15.90:0/1359699331 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f28153680 con 0x7f8f280cef80 2024-03-24T01:25:25.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.737+0000 7f8f2d800640 1 -- 172.21.15.90:0/1359699331 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f28153c40 con 0x7f8f280cef80 2024-03-24T01:25:25.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.738+0000 7f8f2d800640 1 -- 172.21.15.90:0/1359699331 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f2804f400 con 0x7f8f280cef80 2024-03-24T01:25:25.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.739+0000 7f8f1ea00640 1 -- 172.21.15.90:0/1359699331 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8f100453f0 con 0x7f8f280cef80 2024-03-24T01:25:25.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.740+0000 7f8f1ea00640 1 --2- 172.21.15.90:0/1359699331 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8f0406bc00 0x7f8f0406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:25.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.740+0000 7f8f1ea00640 1 -- 172.21.15.90:0/1359699331 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f8f100acec0 con 0x7f8f280cef80 2024-03-24T01:25:25.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.741+0000 7f8f26a00640 1 --2- 172.21.15.90:0/1359699331 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8f0406bc00 0x7f8f0406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:25.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.742+0000 7f8f26a00640 1 --2- 172.21.15.90:0/1359699331 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8f0406bc00 0x7f8f0406e0c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8f1800a750 tx=0x7f8f18008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:25.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.744+0000 7f8f1ea00640 1 -- 172.21.15.90:0/1359699331 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8f10040070 con 0x7f8f280cef80 2024-03-24T01:25:25.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.941+0000 7f8f2d800640 1 -- 172.21.15.90:0/1359699331 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8f280b2e90 con 0x7f8f280cef80 2024-03-24T01:25:25.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.942+0000 7f8f1ea00640 1 -- 172.21.15.90:0/1359699331 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8f1007b580 con 0x7f8f280cef80 2024-03-24T01:25:25.943 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-24T01:25:25.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.945+0000 7f8f2d800640 1 -- 172.21.15.90:0/1359699331 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8f0406bc00 msgr2=0x7f8f0406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:25.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.945+0000 7f8f2d800640 1 --2- 172.21.15.90:0/1359699331 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8f0406bc00 0x7f8f0406e0c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8f1800a750 tx=0x7f8f18008040 comp rx=0 tx=0).stop 2024-03-24T01:25:25.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.945+0000 7f8f2d800640 1 -- 172.21.15.90:0/1359699331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f280cef80 msgr2=0x7f8f2813e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:25.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.945+0000 7f8f2d800640 1 --2- 172.21.15.90:0/1359699331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f280cef80 0x7f8f2813e180 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8f10000c00 tx=0x7f8f100046c0 comp rx=0 tx=0).stop 2024-03-24T01:25:25.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.945+0000 7f8f2d800640 1 -- 172.21.15.90:0/1359699331 shutdown_connections 2024-03-24T01:25:25.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.945+0000 7f8f2d800640 1 --2- 172.21.15.90:0/1359699331 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8f0406bc00 0x7f8f0406e0c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:25.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.945+0000 7f8f2d800640 1 --2- 172.21.15.90:0/1359699331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f280cef80 0x7f8f2813e180 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:25.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.945+0000 7f8f2d800640 1 --2- 172.21.15.90:0/1359699331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f28159620 0x7f8f281551d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:25.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.945+0000 7f8f2d800640 1 --2- 172.21.15.90:0/1359699331 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f2806b7d0 0x7f8f28154440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:25.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.945+0000 7f8f2d800640 1 -- 172.21.15.90:0/1359699331 >> 172.21.15.90:0/1359699331 conn(0x7f8f280b1770 msgr2=0x7f8f280cc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:25.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.945+0000 7f8f2d800640 1 -- 172.21.15.90:0/1359699331 shutdown_connections 2024-03-24T01:25:25.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:25.945+0000 7f8f2d800640 1 -- 172.21.15.90:0/1359699331 wait complete. 2024-03-24T01:25:25.965 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T01:25:26.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.179+0000 7f23b8000640 1 Processor -- start 2024-03-24T01:25:26.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.179+0000 7f23b8000640 1 -- start start 2024-03-24T01:25:26.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.181+0000 7f23b8000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23b006b7d0 0x7f23b0073000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:26.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.181+0000 7f23b8000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f23b00af070 0x7f23b00cddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:26.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.181+0000 7f23b8000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b00ce510 con 0x7f23b0145570 2024-03-24T01:25:26.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.181+0000 7f23b8000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23b0145950 con 0x7f23b00af070 2024-03-24T01:25:26.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.181+0000 7f23b8000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23b00cf120 con 0x7f23b006b7d0 2024-03-24T01:25:26.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.181+0000 7f23b6000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f23b00af070 0x7f23b00cddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:26.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.181+0000 7f23b6000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f23b00af070 0x7f23b00cddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43196/0 (socket says 172.21.15.90:43196) 2024-03-24T01:25:26.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.181+0000 7f23b6000640 1 -- 172.21.15.90:0/4294386674 learned_addr learned my addr 172.21.15.90:0/4294386674 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:26.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.181+0000 7f23b4c00640 1 --2- 172.21.15.90:0/4294386674 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23b006b7d0 0x7f23b0073000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:26.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.181+0000 7f23b6000640 1 -- 172.21.15.90:0/4294386674 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23b006b7d0 msgr2=0x7f23b0073000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:26.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.181+0000 7f23b6000640 1 --2- 172.21.15.90:0/4294386674 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23b006b7d0 0x7f23b0073000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:26.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.182+0000 7f23b6000640 1 -- 172.21.15.90:0/4294386674 >> v1:172.21.15.120:6789/0 conn(0x7f23b0145570 legacy=0x7f23b006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:26.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.182+0000 7f23b6000640 1 -- 172.21.15.90:0/4294386674 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b00ce7f0 con 0x7f23b00af070 2024-03-24T01:25:26.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.183+0000 7f23b6000640 1 --2- 172.21.15.90:0/4294386674 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23b00af070 0x7f23b00cddc0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f23a8009e50 tx=0x7f23a802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ebe583e4a2febf1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:26.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.183+0000 7f23afe00640 1 -- 172.21.15.90:0/4294386674 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a802ca70 con 0x7f23b00af070 2024-03-24T01:25:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.183+0000 7f23afe00640 1 -- 172.21.15.90:0/4294386674 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23a802cc10 con 0x7f23b00af070 2024-03-24T01:25:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.183+0000 7f23afe00640 1 -- 172.21.15.90:0/4294386674 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a80343f0 con 0x7f23b00af070 2024-03-24T01:25:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.183+0000 7f23b8000640 1 -- 172.21.15.90:0/4294386674 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23b00af070 msgr2=0x7f23b00cddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.183+0000 7f23b8000640 1 --2- 172.21.15.90:0/4294386674 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23b00af070 0x7f23b00cddc0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f23a8009e50 tx=0x7f23a802bed0 comp rx=0 tx=0).stop 2024-03-24T01:25:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.184+0000 7f23b8000640 1 -- 172.21.15.90:0/4294386674 shutdown_connections 2024-03-24T01:25:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.184+0000 7f23b8000640 1 --2- 172.21.15.90:0/4294386674 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23b00af070 0x7f23b00cddc0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.184+0000 7f23b8000640 1 --2- 172.21.15.90:0/4294386674 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23b006b7d0 0x7f23b0073000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:26.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.184+0000 7f23b8000640 1 -- 172.21.15.90:0/4294386674 >> 172.21.15.90:0/4294386674 conn(0x7f23b006cec0 msgr2=0x7f23b0099a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:26.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.184+0000 7f23b8000640 1 -- 172.21.15.90:0/4294386674 shutdown_connections 2024-03-24T01:25:26.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.184+0000 7f23b8000640 1 -- 172.21.15.90:0/4294386674 wait complete. 2024-03-24T01:25:26.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.185+0000 7f23b8000640 1 Processor -- start 2024-03-24T01:25:26.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.186+0000 7f23b8000640 1 -- start start 2024-03-24T01:25:26.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.186+0000 7f23b8000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23b006b7d0 0x7f23b00d13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:26.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.186+0000 7f23b8000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23b00af070 0x7f23b0144230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:26.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.186+0000 7f23b8000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23b0145570 0x7f23b0144770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:26.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.186+0000 7f23b8000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b00cf5b0 con 0x7f23b006b7d0 2024-03-24T01:25:26.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.186+0000 7f23b8000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b00cedb0 con 0x7f23b0145570 2024-03-24T01:25:26.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.186+0000 7f23b8000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23b00cf730 con 0x7f23b00af070 2024-03-24T01:25:26.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.187+0000 7f23b5600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23b006b7d0 0x7f23b00d13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:26.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.187+0000 7f23b5600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23b006b7d0 0x7f23b00d13b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43206/0 (socket says 172.21.15.90:43206) 2024-03-24T01:25:26.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.187+0000 7f23b5600640 1 -- 172.21.15.90:0/319094306 learned_addr learned my addr 172.21.15.90:0/319094306 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:26.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.187+0000 7f23b4c00640 1 --2- 172.21.15.90:0/319094306 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23b00af070 0x7f23b0144230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:26.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.187+0000 7f23b5600640 1 -- 172.21.15.90:0/319094306 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23b00af070 msgr2=0x7f23b0144230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:26.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.187+0000 7f23b5600640 1 --2- 172.21.15.90:0/319094306 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23b00af070 0x7f23b0144230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:26.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.187+0000 7f23b5600640 1 -- 172.21.15.90:0/319094306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23b0145570 msgr2=0x7f23b0144770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:26.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.187+0000 7f23b5600640 1 --2- 172.21.15.90:0/319094306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23b0145570 0x7f23b0144770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:26.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.187+0000 7f23b5600640 1 -- 172.21.15.90:0/319094306 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b013d420 con 0x7f23b006b7d0 2024-03-24T01:25:26.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.187+0000 7f23b5600640 1 --2- 172.21.15.90:0/319094306 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23b006b7d0 0x7f23b00d13b0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f23a400b3f0 tx=0x7f23a400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:26.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.188+0000 7f23ad600640 1 -- 172.21.15.90:0/319094306 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a4015020 con 0x7f23b006b7d0 2024-03-24T01:25:26.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.188+0000 7f23ad600640 1 -- 172.21.15.90:0/319094306 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23a4011830 con 0x7f23b006b7d0 2024-03-24T01:25:26.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.188+0000 7f23ad600640 1 -- 172.21.15.90:0/319094306 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a4011c60 con 0x7f23b006b7d0 2024-03-24T01:25:26.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.188+0000 7f23b8000640 1 -- 172.21.15.90:0/319094306 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23b013c380 con 0x7f23b006b7d0 2024-03-24T01:25:26.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.188+0000 7f23b8000640 1 -- 172.21.15.90:0/319094306 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23b013c940 con 0x7f23b006b7d0 2024-03-24T01:25:26.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.189+0000 7f23b8000640 1 -- 172.21.15.90:0/319094306 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23b004f400 con 0x7f23b006b7d0 2024-03-24T01:25:26.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.190+0000 7f23ad600640 1 -- 172.21.15.90:0/319094306 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f23a4018020 con 0x7f23b006b7d0 2024-03-24T01:25:26.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.191+0000 7f23ad600640 1 --2- 172.21.15.90:0/319094306 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f239406bbd0 0x7f239406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:26.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.191+0000 7f23ad600640 1 -- 172.21.15.90:0/319094306 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f23a408ab30 con 0x7f23b006b7d0 2024-03-24T01:25:26.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.192+0000 7f23b4c00640 1 --2- 172.21.15.90:0/319094306 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f239406bbd0 0x7f239406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:26.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.193+0000 7f23b4c00640 1 --2- 172.21.15.90:0/319094306 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f239406bbd0 0x7f239406e090 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f23b01f73f0 tx=0x7f23a0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:26.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.197+0000 7f23ad600640 1 -- 172.21.15.90:0/319094306 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f23a401f030 con 0x7f23b006b7d0 2024-03-24T01:25:26.279 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:25:26.285 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:25:26.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.404+0000 7f23b8000640 1 -- 172.21.15.90:0/319094306 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f23b00cd0e0 con 0x7f23b006b7d0 2024-03-24T01:25:26.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.404+0000 7f23ad600640 1 -- 172.21.15.90:0/319094306 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f23a40172c0 con 0x7f23b006b7d0 2024-03-24T01:25:26.406 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:25:26.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.407+0000 7f23b8000640 1 -- 172.21.15.90:0/319094306 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f239406bbd0 msgr2=0x7f239406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:26.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.407+0000 7f23b8000640 1 --2- 172.21.15.90:0/319094306 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f239406bbd0 0x7f239406e090 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f23b01f73f0 tx=0x7f23a0008040 comp rx=0 tx=0).stop 2024-03-24T01:25:26.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.407+0000 7f23b8000640 1 -- 172.21.15.90:0/319094306 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23b006b7d0 msgr2=0x7f23b00d13b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:26.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.407+0000 7f23b8000640 1 --2- 172.21.15.90:0/319094306 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23b006b7d0 0x7f23b00d13b0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f23a400b3f0 tx=0x7f23a400b8b0 comp rx=0 tx=0).stop 2024-03-24T01:25:26.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.407+0000 7f23b8000640 1 -- 172.21.15.90:0/319094306 shutdown_connections 2024-03-24T01:25:26.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.407+0000 7f23b8000640 1 --2- 172.21.15.90:0/319094306 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f239406bbd0 0x7f239406e090 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:26.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.407+0000 7f23b8000640 1 --2- 172.21.15.90:0/319094306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23b0145570 0x7f23b0144770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:26.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.407+0000 7f23b8000640 1 --2- 172.21.15.90:0/319094306 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23b00af070 0x7f23b0144230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:26.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.407+0000 7f23b8000640 1 --2- 172.21.15.90:0/319094306 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23b006b7d0 0x7f23b00d13b0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:26.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.407+0000 7f23b8000640 1 -- 172.21.15.90:0/319094306 >> 172.21.15.90:0/319094306 conn(0x7f23b006cec0 msgr2=0x7f23b0142920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:26.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.407+0000 7f23b8000640 1 -- 172.21.15.90:0/319094306 shutdown_connections 2024-03-24T01:25:26.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.407+0000 7f23b8000640 1 -- 172.21.15.90:0/319094306 wait complete. 2024-03-24T01:25:26.426 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T01:25:26.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.681+0000 7fd8b7c00640 1 Processor -- start 2024-03-24T01:25:26.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.681+0000 7fd8b7c00640 1 -- start start 2024-03-24T01:25:26.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.684+0000 7fd8b7c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8b015bc50 0x7fd8b006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:26.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.684+0000 7fd8b7c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd8b006b600 0x7fd8b00b2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:26.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.684+0000 7fd8b7c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8b006dac0 con 0x7fd8b00aeeb0 2024-03-24T01:25:26.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.684+0000 7fd8b7c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8b006ba70 con 0x7fd8b006b600 2024-03-24T01:25:26.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.684+0000 7fd8b7c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b00b4da0 con 0x7fd8b015bc50 2024-03-24T01:25:26.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.684+0000 7fd8b5200640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd8b00aeeb0 0x7fd8b00b44c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:42394/0 (socket says 172.21.15.90:42394) 2024-03-24T01:25:26.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.684+0000 7fd8b5200640 1 -- 172.21.15.90:0/135512851 learned_addr learned my addr 172.21.15.90:0/135512851 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:26.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.684+0000 7fd8b5c00640 1 --2- 172.21.15.90:0/135512851 >> v2:172.21.15.120:3300/0 conn(0x7fd8b006b600 0x7fd8b00b2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:26.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.684+0000 7fd8afe00640 1 --2- 172.21.15.90:0/135512851 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8b015bc50 0x7fd8b006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:26.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.684+0000 7fd8af400640 1 -- 172.21.15.90:0/135512851 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1161980948 0 0) 0x7fd8b006dac0 con 0x7fd8b00aeeb0 2024-03-24T01:25:26.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.684+0000 7fd8af400640 1 -- 172.21.15.90:0/135512851 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd894005730 con 0x7fd8b00aeeb0 2024-03-24T01:25:26.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.684+0000 7fd8af400640 1 -- 172.21.15.90:0/135512851 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 783690380 0 0) 0x7fd894005730 con 0x7fd8b00aeeb0 2024-03-24T01:25:26.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.684+0000 7fd8af400640 1 -- 172.21.15.90:0/135512851 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8b015bc50 msgr2=0x7fd8b006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:26.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.684+0000 7fd8af400640 1 --2- 172.21.15.90:0/135512851 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8b015bc50 0x7fd8b006b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:26.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.684+0000 7fd8af400640 1 -- 172.21.15.90:0/135512851 >> v2:172.21.15.120:3300/0 conn(0x7fd8b006b600 msgr2=0x7fd8b00b2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:26.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.684+0000 7fd8af400640 1 --2- 172.21.15.90:0/135512851 >> v2:172.21.15.120:3300/0 conn(0x7fd8b006b600 0x7fd8b00b2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:26.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.685+0000 7fd8af400640 1 -- 172.21.15.90:0/135512851 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b006dda0 con 0x7fd8b00aeeb0 2024-03-24T01:25:26.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.685+0000 7fd8af400640 1 -- 172.21.15.90:0/135512851 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7fd8a00019f0 con 0x7fd8b00aeeb0 2024-03-24T01:25:26.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.685+0000 7fd8af400640 1 -- 172.21.15.90:0/135512851 >> v1:172.21.15.90:6789/0 conn(0x7fd8b00aeeb0 legacy=0x7fd8b00b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:26.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.685+0000 7fd8af400640 1 --2- 172.21.15.90:0/135512851 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd894005bc0 0x7fd894005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:26.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.685+0000 7fd8af400640 1 --2- 172.21.15.90:0/135512851 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd894006a10 0x7fd894006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:26.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.685+0000 7fd8af400640 1 --2- 172.21.15.90:0/135512851 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8b015bc50 0x7fd8940073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:26.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.685+0000 7fd8af400640 1 -- 172.21.15.90:0/135512851 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b006ba70 con 0x7fd894006a10 2024-03-24T01:25:26.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.685+0000 7fd8af400640 1 -- 172.21.15.90:0/135512851 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b00b4da0 con 0x7fd894005bc0 2024-03-24T01:25:26.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.685+0000 7fd8af400640 1 -- 172.21.15.90:0/135512851 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd89400dc80 con 0x7fd8b015bc50 2024-03-24T01:25:26.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.685+0000 7fd8afe00640 1 --2- 172.21.15.90:0/135512851 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd894006a10 0x7fd894006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:26.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.685+0000 7fd8afe00640 1 -- 172.21.15.90:0/135512851 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8b015bc50 msgr2=0x7fd8940073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:26.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.685+0000 7fd8afe00640 1 --2- 172.21.15.90:0/135512851 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8b015bc50 0x7fd8940073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:26.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.685+0000 7fd8afe00640 1 -- 172.21.15.90:0/135512851 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd894005bc0 msgr2=0x7fd894005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:26.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.685+0000 7fd8afe00640 1 --2- 172.21.15.90:0/135512851 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd894005bc0 0x7fd894005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:26.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.685+0000 7fd8afe00640 1 -- 172.21.15.90:0/135512851 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd894005730 con 0x7fd894006a10 2024-03-24T01:25:26.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.686+0000 7fd8afe00640 1 --2- 172.21.15.90:0/135512851 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd894006a10 0x7fd894006e90 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fd898009ac0 tx=0x7fd894007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba4cfec9c0a5d0ae server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:26.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.687+0000 7fd8af400640 1 -- 172.21.15.90:0/135512851 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd898039070 con 0x7fd894006a10 2024-03-24T01:25:26.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.687+0000 7fd8af400640 1 -- 172.21.15.90:0/135512851 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd89802c710 con 0x7fd894006a10 2024-03-24T01:25:26.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.687+0000 7fd8af400640 1 -- 172.21.15.90:0/135512851 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd898009690 con 0x7fd894006a10 2024-03-24T01:25:26.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.687+0000 7fd8b7c00640 1 -- 172.21.15.90:0/135512851 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd894006a10 msgr2=0x7fd894006e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:26.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.687+0000 7fd8b7c00640 1 --2- 172.21.15.90:0/135512851 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd894006a10 0x7fd894006e90 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fd898009ac0 tx=0x7fd894007a40 comp rx=0 tx=0).stop 2024-03-24T01:25:26.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.687+0000 7fd8b5c00640 1 -- 172.21.15.90:0/135512851 reap_dead start 2024-03-24T01:25:26.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.687+0000 7fd8b7c00640 1 -- 172.21.15.90:0/135512851 shutdown_connections 2024-03-24T01:25:26.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.687+0000 7fd8b7c00640 1 -- 172.21.15.90:0/135512851 >> 172.21.15.90:0/135512851 conn(0x7fd8b00b1770 msgr2=0x7fd8b00b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:26.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.687+0000 7fd8b7c00640 1 -- 172.21.15.90:0/135512851 shutdown_connections 2024-03-24T01:25:26.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.687+0000 7fd8b7c00640 1 -- 172.21.15.90:0/135512851 wait complete. 2024-03-24T01:25:26.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.689+0000 7fd8b7c00640 1 Processor -- start 2024-03-24T01:25:26.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.689+0000 7fd8b7c00640 1 -- start start 2024-03-24T01:25:26.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.690+0000 7fd8b7c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd894006a10 0x7fd8b014c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:26.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.690+0000 7fd8b7c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8b00c4800 0x7fd8b014cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:26.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.690+0000 7fd8b7c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8b014d260 0x7fd8b015d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:26.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.690+0000 7fd8b7c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b00af290 con 0x7fd8b00c4800 2024-03-24T01:25:26.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.690+0000 7fd8b7c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b006e360 con 0x7fd8b014d260 2024-03-24T01:25:26.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.690+0000 7fd8b7c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b006e880 con 0x7fd894006a10 2024-03-24T01:25:26.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.690+0000 7fd8afe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8b00c4800 0x7fd8b014cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:26.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.690+0000 7fd8afe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8b00c4800 0x7fd8b014cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43222/0 (socket says 172.21.15.90:43222) 2024-03-24T01:25:26.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.690+0000 7fd8afe00640 1 -- 172.21.15.90:0/4000650420 learned_addr learned my addr 172.21.15.90:0/4000650420 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:26.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.691+0000 7fd8b5200640 1 --2- 172.21.15.90:0/4000650420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd894006a10 0x7fd8b014c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:26.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.691+0000 7fd8b5c00640 1 --2- 172.21.15.90:0/4000650420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8b014d260 0x7fd8b015d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:26.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.691+0000 7fd8afe00640 1 -- 172.21.15.90:0/4000650420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd894006a10 msgr2=0x7fd8b014c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:26.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.691+0000 7fd8afe00640 1 --2- 172.21.15.90:0/4000650420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd894006a10 0x7fd8b014c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:26.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.691+0000 7fd8afe00640 1 -- 172.21.15.90:0/4000650420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8b014d260 msgr2=0x7fd8b015d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:26.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.691+0000 7fd8afe00640 1 --2- 172.21.15.90:0/4000650420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8b014d260 0x7fd8b015d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:26.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.691+0000 7fd8afe00640 1 -- 172.21.15.90:0/4000650420 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b00acca0 con 0x7fd8b00c4800 2024-03-24T01:25:26.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.691+0000 7fd8afe00640 1 --2- 172.21.15.90:0/4000650420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8b00c4800 0x7fd8b014cd20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd898000c00 tx=0x7fd898004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:26.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.691+0000 7fd8acc00640 1 -- 172.21.15.90:0/4000650420 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd898039050 con 0x7fd8b00c4800 2024-03-24T01:25:26.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.692+0000 7fd8acc00640 1 -- 172.21.15.90:0/4000650420 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd898045070 con 0x7fd8b00c4800 2024-03-24T01:25:26.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.692+0000 7fd8acc00640 1 -- 172.21.15.90:0/4000650420 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd898003a50 con 0x7fd8b00c4800 2024-03-24T01:25:26.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.692+0000 7fd8b7c00640 1 -- 172.21.15.90:0/4000650420 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8b00acf80 con 0x7fd8b00c4800 2024-03-24T01:25:26.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.692+0000 7fd8b7c00640 1 -- 172.21.15.90:0/4000650420 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b00ad540 con 0x7fd8b00c4800 2024-03-24T01:25:26.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.693+0000 7fd8b7c00640 1 -- 172.21.15.90:0/4000650420 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8b004f400 con 0x7fd8b00c4800 2024-03-24T01:25:26.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.694+0000 7fd8acc00640 1 -- 172.21.15.90:0/4000650420 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd898003bf0 con 0x7fd8b00c4800 2024-03-24T01:25:26.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.695+0000 7fd8acc00640 1 --2- 172.21.15.90:0/4000650420 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd88c06bbd0 0x7fd88c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:26.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.695+0000 7fd8acc00640 1 -- 172.21.15.90:0/4000650420 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7fd8980ac9d0 con 0x7fd8b00c4800 2024-03-24T01:25:26.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.695+0000 7fd8b5200640 1 --2- 172.21.15.90:0/4000650420 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd88c06bbd0 0x7fd88c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:26.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.696+0000 7fd8b5200640 1 --2- 172.21.15.90:0/4000650420 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd88c06bbd0 0x7fd88c06e090 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0005fd0 tx=0x7fd8a00019a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:26.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.699+0000 7fd8acc00640 1 -- 172.21.15.90:0/4000650420 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd89804e020 con 0x7fd8b00c4800 2024-03-24T01:25:26.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.914+0000 7fd8b7c00640 1 -- 172.21.15.90:0/4000650420 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd8b00b1d20 con 0x7fd8b00c4800 2024-03-24T01:25:26.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.915+0000 7fd8acc00640 1 -- 172.21.15.90:0/4000650420 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd89807b120 con 0x7fd8b00c4800 2024-03-24T01:25:26.916 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-24T01:25:26.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.917+0000 7fd8b7c00640 1 -- 172.21.15.90:0/4000650420 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd88c06bbd0 msgr2=0x7fd88c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:26.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.917+0000 7fd8b7c00640 1 --2- 172.21.15.90:0/4000650420 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd88c06bbd0 0x7fd88c06e090 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0005fd0 tx=0x7fd8a00019a0 comp rx=0 tx=0).stop 2024-03-24T01:25:26.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.917+0000 7fd8b7c00640 1 -- 172.21.15.90:0/4000650420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8b00c4800 msgr2=0x7fd8b014cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:26.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.917+0000 7fd8b7c00640 1 --2- 172.21.15.90:0/4000650420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8b00c4800 0x7fd8b014cd20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd898000c00 tx=0x7fd898004730 comp rx=0 tx=0).stop 2024-03-24T01:25:26.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.918+0000 7fd8b7c00640 1 -- 172.21.15.90:0/4000650420 shutdown_connections 2024-03-24T01:25:26.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.918+0000 7fd8b7c00640 1 --2- 172.21.15.90:0/4000650420 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd88c06bbd0 0x7fd88c06e090 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:26.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.918+0000 7fd8b7c00640 1 --2- 172.21.15.90:0/4000650420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8b00c4800 0x7fd8b014cd20 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:26.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.918+0000 7fd8b7c00640 1 --2- 172.21.15.90:0/4000650420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8b014d260 0x7fd8b015d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:26.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.918+0000 7fd8b7c00640 1 --2- 172.21.15.90:0/4000650420 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd894006a10 0x7fd8b014c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:26.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.918+0000 7fd8b7c00640 1 -- 172.21.15.90:0/4000650420 >> 172.21.15.90:0/4000650420 conn(0x7fd8b00b1770 msgr2=0x7fd8b015c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:26.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.918+0000 7fd8b7c00640 1 -- 172.21.15.90:0/4000650420 shutdown_connections 2024-03-24T01:25:26.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:26.918+0000 7fd8b7c00640 1 -- 172.21.15.90:0/4000650420 wait complete. 2024-03-24T01:25:26.939 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:25:26.940 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:25:26.990 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:25:27.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.204+0000 7ffaf9a00640 1 Processor -- start 2024-03-24T01:25:27.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.204+0000 7ffaf9a00640 1 -- start start 2024-03-24T01:25:27.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.205+0000 7ffaf9a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ffaf406b7d0 0x7ffaf406e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:27.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.205+0000 7ffaf9a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf40acf50 con 0x7ffaf415dd50 2024-03-24T01:25:27.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.205+0000 7ffaf9a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf40b4170 con 0x7ffaf40b3d60 2024-03-24T01:25:27.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.205+0000 7ffaf9a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaf40ad280 con 0x7ffaf406b7d0 2024-03-24T01:25:27.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.206+0000 7ffaf2a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ffaf415dd50 0x7ffaf406b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:42400/0 (socket says 172.21.15.90:42400) 2024-03-24T01:25:27.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.206+0000 7ffaf2a00640 1 -- 172.21.15.90:0/3225208908 learned_addr learned my addr 172.21.15.90:0/3225208908 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:27.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.206+0000 7ffaf2000640 1 --2- 172.21.15.90:0/3225208908 >> v2:172.21.15.90:3300/0 conn(0x7ffaf406b7d0 0x7ffaf406e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.206+0000 7ffaf1600640 1 -- 172.21.15.90:0/3225208908 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2209023963 0 0) 0x7ffaf40acf50 con 0x7ffaf415dd50 2024-03-24T01:25:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.206+0000 7ffaf2000640 1 -- 172.21.15.90:0/3225208908 >> v1:172.21.15.120:6789/0 conn(0x7ffaf40b3d60 legacy=0x7ffaf40ad710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.206+0000 7ffaf2000640 1 -- 172.21.15.90:0/3225208908 >> v1:172.21.15.90:6789/0 conn(0x7ffaf415dd50 legacy=0x7ffaf406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.206+0000 7ffaf2000640 1 -- 172.21.15.90:0/3225208908 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf415e130 con 0x7ffaf406b7d0 2024-03-24T01:25:27.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.207+0000 7ffaf2000640 1 --2- 172.21.15.90:0/3225208908 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffaf406b7d0 0x7ffaf406e020 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ffadc0097a0 tx=0x7ffadc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bfa33da9ede79d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:27.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.208+0000 7ffaf1600640 1 -- 172.21.15.90:0/3225208908 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffadc0315a0 con 0x7ffaf406b7d0 2024-03-24T01:25:27.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.208+0000 7ffaf1600640 1 -- 172.21.15.90:0/3225208908 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffadc037070 con 0x7ffaf406b7d0 2024-03-24T01:25:27.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.208+0000 7ffaf1600640 1 -- 172.21.15.90:0/3225208908 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffadc031960 con 0x7ffaf406b7d0 2024-03-24T01:25:27.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.208+0000 7ffaf9a00640 1 -- 172.21.15.90:0/3225208908 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffaf406b7d0 msgr2=0x7ffaf406e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:27.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.208+0000 7ffaf9a00640 1 --2- 172.21.15.90:0/3225208908 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffaf406b7d0 0x7ffaf406e020 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ffadc0097a0 tx=0x7ffadc02b840 comp rx=0 tx=0).stop 2024-03-24T01:25:27.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.208+0000 7ffaf9a00640 1 -- 172.21.15.90:0/3225208908 shutdown_connections 2024-03-24T01:25:27.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.208+0000 7ffaf9a00640 1 --2- 172.21.15.90:0/3225208908 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffaf406b7d0 0x7ffaf406e020 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:27.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.208+0000 7ffaf9a00640 1 -- 172.21.15.90:0/3225208908 >> 172.21.15.90:0/3225208908 conn(0x7ffaf40b1770 msgr2=0x7ffaf40b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:27.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.208+0000 7ffaf9a00640 1 -- 172.21.15.90:0/3225208908 shutdown_connections 2024-03-24T01:25:27.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.209+0000 7ffaf9a00640 1 -- 172.21.15.90:0/3225208908 wait complete. 2024-03-24T01:25:27.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.210+0000 7ffaf9a00640 1 Processor -- start 2024-03-24T01:25:27.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.210+0000 7ffaf9a00640 1 -- start start 2024-03-24T01:25:27.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.211+0000 7ffaf9a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffaf406b7d0 0x7ffaf415c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:27.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.211+0000 7ffaf9a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaf40b3d60 0x7ffaf4144970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:27.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.211+0000 7ffaf9a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffaf415dd50 0x7ffaf4145210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:27.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.211+0000 7ffaf9a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf41625c0 con 0x7ffaf415dd50 2024-03-24T01:25:27.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.211+0000 7ffaf9a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf41622e0 con 0x7ffaf406b7d0 2024-03-24T01:25:27.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.211+0000 7ffaf9a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf41618d0 con 0x7ffaf40b3d60 2024-03-24T01:25:27.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.211+0000 7ffaf3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffaf415dd50 0x7ffaf4145210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:27.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.212+0000 7ffaf3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffaf415dd50 0x7ffaf4145210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43252/0 (socket says 172.21.15.90:43252) 2024-03-24T01:25:27.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.212+0000 7ffaf3400640 1 -- 172.21.15.90:0/3739953251 learned_addr learned my addr 172.21.15.90:0/3739953251 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:27.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.212+0000 7ffaf2000640 1 --2- 172.21.15.90:0/3739953251 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaf40b3d60 0x7ffaf4144970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:27.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.212+0000 7ffaf2a00640 1 --2- 172.21.15.90:0/3739953251 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffaf406b7d0 0x7ffaf415c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:27.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.212+0000 7ffaf3400640 1 -- 172.21.15.90:0/3739953251 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaf40b3d60 msgr2=0x7ffaf4144970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:27.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.212+0000 7ffaf3400640 1 --2- 172.21.15.90:0/3739953251 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaf40b3d60 0x7ffaf4144970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:27.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.212+0000 7ffaf3400640 1 -- 172.21.15.90:0/3739953251 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffaf406b7d0 msgr2=0x7ffaf415c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:27.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.212+0000 7ffaf3400640 1 --2- 172.21.15.90:0/3739953251 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffaf406b7d0 0x7ffaf415c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:27.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.212+0000 7ffaf3400640 1 -- 172.21.15.90:0/3739953251 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf40cd510 con 0x7ffaf415dd50 2024-03-24T01:25:27.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.212+0000 7ffaf3400640 1 --2- 172.21.15.90:0/3739953251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffaf415dd50 0x7ffaf4145210 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ffaec001eb0 tx=0x7ffaec00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:27.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.212+0000 7ffaeaa00640 1 -- 172.21.15.90:0/3739953251 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaec017070 con 0x7ffaf415dd50 2024-03-24T01:25:27.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.212+0000 7ffaeaa00640 1 -- 172.21.15.90:0/3739953251 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaec0052d0 con 0x7ffaf415dd50 2024-03-24T01:25:27.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.213+0000 7ffaf9a00640 1 -- 172.21.15.90:0/3739953251 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaf415cc70 con 0x7ffaf415dd50 2024-03-24T01:25:27.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.213+0000 7ffaeaa00640 1 -- 172.21.15.90:0/3739953251 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaec003810 con 0x7ffaf415dd50 2024-03-24T01:25:27.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.213+0000 7ffaf9a00640 1 -- 172.21.15.90:0/3739953251 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaf40cda60 con 0x7ffaf415dd50 2024-03-24T01:25:27.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.214+0000 7ffaf9a00640 1 -- 172.21.15.90:0/3739953251 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaf4068060 con 0x7ffaf415dd50 2024-03-24T01:25:27.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.215+0000 7ffaeaa00640 1 -- 172.21.15.90:0/3739953251 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ffaec01a070 con 0x7ffaf415dd50 2024-03-24T01:25:27.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.215+0000 7ffaeaa00640 1 --2- 172.21.15.90:0/3739953251 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ffad006bc00 0x7ffad006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:27.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.216+0000 7ffaeaa00640 1 -- 172.21.15.90:0/3739953251 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7ffaec08a350 con 0x7ffaf415dd50 2024-03-24T01:25:27.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.216+0000 7ffaf2a00640 1 --2- 172.21.15.90:0/3739953251 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ffad006bc00 0x7ffad006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:27.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.217+0000 7ffaf2a00640 1 --2- 172.21.15.90:0/3739953251 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ffad006bc00 0x7ffad006e0c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ffae4007fd0 tx=0x7ffae4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:27.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.220+0000 7ffaeaa00640 1 -- 172.21.15.90:0/3739953251 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ffaec0588f0 con 0x7ffaf415dd50 2024-03-24T01:25:27.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.417+0000 7ffaf9a00640 1 -- 172.21.15.90:0/3739953251 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffaf40ae3e0 con 0x7ffaf415dd50 2024-03-24T01:25:27.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.420+0000 7ffaeaa00640 1 -- 172.21.15.90:0/3739953251 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18925 (secure 0 0 0) 0x7ffaec0581f0 con 0x7ffaf415dd50 2024-03-24T01:25:27.421 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:25:27.421 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:24:53.236401+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:24:08.643966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T01:24:09.189904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T01:24:13.027323+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:25:27.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.422+0000 7ffaf9a00640 1 -- 172.21.15.90:0/3739953251 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ffad006bc00 msgr2=0x7ffad006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:27.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.422+0000 7ffaf9a00640 1 --2- 172.21.15.90:0/3739953251 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ffad006bc00 0x7ffad006e0c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ffae4007fd0 tx=0x7ffae4009040 comp rx=0 tx=0).stop 2024-03-24T01:25:27.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.422+0000 7ffaf9a00640 1 -- 172.21.15.90:0/3739953251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffaf415dd50 msgr2=0x7ffaf4145210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:27.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.422+0000 7ffaf9a00640 1 --2- 172.21.15.90:0/3739953251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffaf415dd50 0x7ffaf4145210 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ffaec001eb0 tx=0x7ffaec00c890 comp rx=0 tx=0).stop 2024-03-24T01:25:27.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.423+0000 7ffaf9a00640 1 -- 172.21.15.90:0/3739953251 shutdown_connections 2024-03-24T01:25:27.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.423+0000 7ffaf9a00640 1 --2- 172.21.15.90:0/3739953251 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ffad006bc00 0x7ffad006e0c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:27.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.423+0000 7ffaf9a00640 1 --2- 172.21.15.90:0/3739953251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffaf415dd50 0x7ffaf4145210 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:27.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.423+0000 7ffaf9a00640 1 --2- 172.21.15.90:0/3739953251 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffaf40b3d60 0x7ffaf4144970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:27.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.423+0000 7ffaf9a00640 1 --2- 172.21.15.90:0/3739953251 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffaf406b7d0 0x7ffaf415c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:27.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.423+0000 7ffaf9a00640 1 -- 172.21.15.90:0/3739953251 >> 172.21.15.90:0/3739953251 conn(0x7ffaf40b1770 msgr2=0x7ffaf4158bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:27.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.423+0000 7ffaf9a00640 1 -- 172.21.15.90:0/3739953251 shutdown_connections 2024-03-24T01:25:27.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.423+0000 7ffaf9a00640 1 -- 172.21.15.90:0/3739953251 wait complete. 2024-03-24T01:25:27.441 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T01:24:08.643966+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T01:24:09.189904+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T01:24:13.027323+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T01:25:27.441 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:25:27.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.653+0000 7f24b1a00640 1 Processor -- start 2024-03-24T01:25:27.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.653+0000 7f24b1a00640 1 -- start start 2024-03-24T01:25:27.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.654+0000 7f24b1a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24ac154d20 0x7f24ac15c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:27.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.654+0000 7f24b1a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f24ac155100 0x7f24ac15adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:27.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.654+0000 7f24b1a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f24ac0b2680 0x7f24ac0cf320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:27.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.654+0000 7f24b1a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24ac0b2ed0 con 0x7f24ac155100 2024-03-24T01:25:27.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.654+0000 7f24b1a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24ac0b31b0 con 0x7f24ac0b2680 2024-03-24T01:25:27.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.654+0000 7f24b1a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24ac0b3490 con 0x7f24ac154d20 2024-03-24T01:25:27.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.655+0000 7f24a3400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f24ac155100 0x7f24ac15adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:27.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.655+0000 7f24a3400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f24ac155100 0x7f24ac15adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43260/0 (socket says 172.21.15.90:43260) 2024-03-24T01:25:27.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.655+0000 7f24a3400640 1 -- 172.21.15.90:0/1167681747 learned_addr learned my addr 172.21.15.90:0/1167681747 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:27.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.655+0000 7f24abe00640 1 --2- 172.21.15.90:0/1167681747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24ac154d20 0x7f24ac15c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:27.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.655+0000 7f24b1000640 1 --2- 172.21.15.90:0/1167681747 >> v2:172.21.15.120:3300/0 conn(0x7f24ac0b2680 0x7f24ac0cf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:27.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.655+0000 7f24a3400640 1 -- 172.21.15.90:0/1167681747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24ac154d20 msgr2=0x7f24ac15c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:27.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.655+0000 7f24a3400640 1 --2- 172.21.15.90:0/1167681747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24ac154d20 0x7f24ac15c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:27.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.655+0000 7f24a3400640 1 -- 172.21.15.90:0/1167681747 >> v2:172.21.15.120:3300/0 conn(0x7f24ac0b2680 msgr2=0x7f24ac0cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:27.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.655+0000 7f24a3400640 1 --2- 172.21.15.90:0/1167681747 >> v2:172.21.15.120:3300/0 conn(0x7f24ac0b2680 0x7f24ac0cf320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:27.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.655+0000 7f24a3400640 1 -- 172.21.15.90:0/1167681747 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24ac0b3770 con 0x7f24ac155100 2024-03-24T01:25:27.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.656+0000 7f24a3400640 1 --2- 172.21.15.90:0/1167681747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24ac155100 0x7f24ac15adf0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f24ac0cf0c0 tx=0x7f249402b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8035a5148334e89 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:27.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.657+0000 7f24ab400640 1 -- 172.21.15.90:0/1167681747 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2494038070 con 0x7f24ac155100 2024-03-24T01:25:27.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.657+0000 7f24ab400640 1 -- 172.21.15.90:0/1167681747 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2494031740 con 0x7f24ac155100 2024-03-24T01:25:27.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.657+0000 7f24ab400640 1 -- 172.21.15.90:0/1167681747 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249402f050 con 0x7f24ac155100 2024-03-24T01:25:27.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.657+0000 7f24b1a00640 1 -- 172.21.15.90:0/1167681747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24ac155100 msgr2=0x7f24ac15adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:27.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.657+0000 7f24b1a00640 1 --2- 172.21.15.90:0/1167681747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24ac155100 0x7f24ac15adf0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f24ac0cf0c0 tx=0x7f249402b7e0 comp rx=0 tx=0).stop 2024-03-24T01:25:27.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.657+0000 7f24b1a00640 1 -- 172.21.15.90:0/1167681747 shutdown_connections 2024-03-24T01:25:27.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.657+0000 7f24b1a00640 1 --2- 172.21.15.90:0/1167681747 >> v2:172.21.15.120:3300/0 conn(0x7f24ac0b2680 0x7f24ac0cf320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:27.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.657+0000 7f24b1a00640 1 --2- 172.21.15.90:0/1167681747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24ac155100 0x7f24ac15adf0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:27.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.657+0000 7f24b1a00640 1 --2- 172.21.15.90:0/1167681747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24ac154d20 0x7f24ac15c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:27.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.657+0000 7f24b1a00640 1 -- 172.21.15.90:0/1167681747 >> 172.21.15.90:0/1167681747 conn(0x7f24ac06b0c0 msgr2=0x7f24ac06b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:27.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.657+0000 7f24b1a00640 1 -- 172.21.15.90:0/1167681747 shutdown_connections 2024-03-24T01:25:27.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.657+0000 7f24b1a00640 1 -- 172.21.15.90:0/1167681747 wait complete. 2024-03-24T01:25:27.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.659+0000 7f24b1a00640 1 Processor -- start 2024-03-24T01:25:27.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.660+0000 7f24b1a00640 1 -- start start 2024-03-24T01:25:27.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.661+0000 7f24b1a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24ac0b2680 0x7f24ac14e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:27.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.661+0000 7f24b1a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24ac154d20 0x7f24ac157ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:27.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.661+0000 7f24b1a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24ac155100 0x7f24ac157ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:27.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.661+0000 7f24b1a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24ac0cf9e0 con 0x7f24ac155100 2024-03-24T01:25:27.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.661+0000 7f24b1a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24ac0cf860 con 0x7f24ac154d20 2024-03-24T01:25:27.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.661+0000 7f24b1a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24ac0cfb60 con 0x7f24ac0b2680 2024-03-24T01:25:27.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.661+0000 7f24b1000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24ac155100 0x7f24ac157ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:27.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.661+0000 7f24abe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24ac0b2680 0x7f24ac14e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:27.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.661+0000 7f24b1000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24ac155100 0x7f24ac157ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43264/0 (socket says 172.21.15.90:43264) 2024-03-24T01:25:27.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.661+0000 7f24b1000640 1 -- 172.21.15.90:0/2291502910 learned_addr learned my addr 172.21.15.90:0/2291502910 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:27.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.661+0000 7f24a3400640 1 --2- 172.21.15.90:0/2291502910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24ac154d20 0x7f24ac157ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:27.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.662+0000 7f24b1000640 1 -- 172.21.15.90:0/2291502910 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24ac0b2680 msgr2=0x7f24ac14e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:27.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.662+0000 7f24b1000640 1 --2- 172.21.15.90:0/2291502910 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24ac0b2680 0x7f24ac14e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:27.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.662+0000 7f24b1000640 1 -- 172.21.15.90:0/2291502910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24ac154d20 msgr2=0x7f24ac157ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:27.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.662+0000 7f24b1000640 1 --2- 172.21.15.90:0/2291502910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24ac154d20 0x7f24ac157ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:27.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.662+0000 7f24b1000640 1 -- 172.21.15.90:0/2291502910 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24ac155850 con 0x7f24ac155100 2024-03-24T01:25:27.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.662+0000 7f24abe00640 1 --2- 172.21.15.90:0/2291502910 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24ac0b2680 0x7f24ac14e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:25:27.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.662+0000 7f24b1000640 1 --2- 172.21.15.90:0/2291502910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24ac155100 0x7f24ac157ff0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f24a4007d20 tx=0x7f24a400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:27.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.663+0000 7f24a8c00640 1 -- 172.21.15.90:0/2291502910 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a4014420 con 0x7f24ac155100 2024-03-24T01:25:27.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.663+0000 7f24a8c00640 1 -- 172.21.15.90:0/2291502910 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a4019070 con 0x7f24ac155100 2024-03-24T01:25:27.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.663+0000 7f24a8c00640 1 -- 172.21.15.90:0/2291502910 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a4014820 con 0x7f24ac155100 2024-03-24T01:25:27.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.663+0000 7f24b1a00640 1 -- 172.21.15.90:0/2291502910 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24ac155b30 con 0x7f24ac155100 2024-03-24T01:25:27.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.663+0000 7f24b1a00640 1 -- 172.21.15.90:0/2291502910 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24ac1560f0 con 0x7f24ac155100 2024-03-24T01:25:27.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.664+0000 7f24b1a00640 1 -- 172.21.15.90:0/2291502910 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24ac04f400 con 0x7f24ac155100 2024-03-24T01:25:27.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.665+0000 7f24a8c00640 1 -- 172.21.15.90:0/2291502910 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f24a40149c0 con 0x7f24ac155100 2024-03-24T01:25:27.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.666+0000 7f24a8c00640 1 --2- 172.21.15.90:0/2291502910 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f248806bbd0 0x7f248806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:27.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.666+0000 7f24a8c00640 1 -- 172.21.15.90:0/2291502910 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f24a408b660 con 0x7f24ac155100 2024-03-24T01:25:27.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.666+0000 7f24abe00640 1 --2- 172.21.15.90:0/2291502910 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f248806bbd0 0x7f248806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:27.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.667+0000 7f24abe00640 1 --2- 172.21.15.90:0/2291502910 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f248806bbd0 0x7f248806e090 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f24ac0cbe10 tx=0x7f249c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:27.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.670+0000 7f24a8c00640 1 -- 172.21.15.90:0/2291502910 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f24a4059d20 con 0x7f24ac155100 2024-03-24T01:25:27.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.871+0000 7f24b1a00640 1 -- 172.21.15.90:0/2291502910 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f24ac15bdf0 con 0x7f24ac155100 2024-03-24T01:25:27.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.872+0000 7f24a8c00640 1 -- 172.21.15.90:0/2291502910 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f24a4059620 con 0x7f24ac155100 2024-03-24T01:25:27.872 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:25:27.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.875+0000 7f24b1a00640 1 -- 172.21.15.90:0/2291502910 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f248806bbd0 msgr2=0x7f248806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:27.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.875+0000 7f24b1a00640 1 --2- 172.21.15.90:0/2291502910 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f248806bbd0 0x7f248806e090 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f24ac0cbe10 tx=0x7f249c008040 comp rx=0 tx=0).stop 2024-03-24T01:25:27.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.875+0000 7f24b1a00640 1 -- 172.21.15.90:0/2291502910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24ac155100 msgr2=0x7f24ac157ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:27.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.875+0000 7f24b1a00640 1 --2- 172.21.15.90:0/2291502910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24ac155100 0x7f24ac157ff0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f24a4007d20 tx=0x7f24a400e4b0 comp rx=0 tx=0).stop 2024-03-24T01:25:27.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.875+0000 7f24b1a00640 1 -- 172.21.15.90:0/2291502910 shutdown_connections 2024-03-24T01:25:27.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.875+0000 7f24b1a00640 1 --2- 172.21.15.90:0/2291502910 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f248806bbd0 0x7f248806e090 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:27.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.875+0000 7f24b1a00640 1 --2- 172.21.15.90:0/2291502910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24ac155100 0x7f24ac157ff0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:27.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.875+0000 7f24b1a00640 1 --2- 172.21.15.90:0/2291502910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24ac154d20 0x7f24ac157ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:27.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.875+0000 7f24b1a00640 1 --2- 172.21.15.90:0/2291502910 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f24ac0b2680 0x7f24ac14e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:27.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.875+0000 7f24b1a00640 1 -- 172.21.15.90:0/2291502910 >> 172.21.15.90:0/2291502910 conn(0x7f24ac06b0c0 msgr2=0x7f24ac06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:27.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.875+0000 7f24b1a00640 1 -- 172.21.15.90:0/2291502910 shutdown_connections 2024-03-24T01:25:27.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:27.875+0000 7f24b1a00640 1 -- 172.21.15.90:0/2291502910 wait complete. 2024-03-24T01:25:27.894 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T01:25:28.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.107+0000 7f43c1c00640 1 Processor -- start 2024-03-24T01:25:28.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.107+0000 7f43c1c00640 1 -- start start 2024-03-24T01:25:28.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.109+0000 7f43c1c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f43bc1550a0 0x7f43bc15bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:28.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.109+0000 7f43c1c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f43bc06f5c0 0x7f43bc0b3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:28.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.109+0000 7f43c1c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43bc0b43f0 con 0x7f43bc0b3fe0 2024-03-24T01:25:28.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.109+0000 7f43c1c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43bc06fa70 con 0x7f43bc1550a0 2024-03-24T01:25:28.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.109+0000 7f43c1c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43bc155480 con 0x7f43bc06f5c0 2024-03-24T01:25:28.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.109+0000 7f43baa00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f43bc1550a0 0x7f43bc15bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:28.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.109+0000 7f43baa00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f43bc1550a0 0x7f43bc15bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43280/0 (socket says 172.21.15.90:43280) 2024-03-24T01:25:28.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.109+0000 7f43baa00640 1 -- 172.21.15.90:0/3996473396 learned_addr learned my addr 172.21.15.90:0/3996473396 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:28.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.109+0000 7f43baa00640 1 -- 172.21.15.90:0/3996473396 >> v2:172.21.15.120:3300/0 conn(0x7f43bc06f5c0 msgr2=0x7f43bc0b3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:28.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.109+0000 7f43baa00640 1 --2- 172.21.15.90:0/3996473396 >> v2:172.21.15.120:3300/0 conn(0x7f43bc06f5c0 0x7f43bc0b3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:28.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.109+0000 7f43baa00640 1 -- 172.21.15.90:0/3996473396 >> v1:172.21.15.120:6789/0 conn(0x7f43bc0b3fe0 legacy=0x7f43bc0ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:28.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.110+0000 7f43baa00640 1 -- 172.21.15.90:0/3996473396 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43bc06fcc0 con 0x7f43bc1550a0 2024-03-24T01:25:28.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.111+0000 7f43baa00640 1 --2- 172.21.15.90:0/3996473396 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43bc1550a0 0x7f43bc15bb20 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f43bc0717e0 tx=0x7f43ac02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f91f6ac916c9984 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:28.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.111+0000 7f43b9600640 1 -- 172.21.15.90:0/3996473396 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43ac02e070 con 0x7f43bc1550a0 2024-03-24T01:25:28.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.111+0000 7f43b9600640 1 -- 172.21.15.90:0/3996473396 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43ac037070 con 0x7f43bc1550a0 2024-03-24T01:25:28.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.111+0000 7f43b9600640 1 -- 172.21.15.90:0/3996473396 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43ac031a00 con 0x7f43bc1550a0 2024-03-24T01:25:28.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.112+0000 7f43c1c00640 1 -- 172.21.15.90:0/3996473396 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43bc1550a0 msgr2=0x7f43bc15bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:28.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.112+0000 7f43c1c00640 1 --2- 172.21.15.90:0/3996473396 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43bc1550a0 0x7f43bc15bb20 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f43bc0717e0 tx=0x7f43ac02b880 comp rx=0 tx=0).stop 2024-03-24T01:25:28.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.112+0000 7f43c1c00640 1 -- 172.21.15.90:0/3996473396 shutdown_connections 2024-03-24T01:25:28.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.112+0000 7f43c1c00640 1 --2- 172.21.15.90:0/3996473396 >> v2:172.21.15.120:3300/0 conn(0x7f43bc06f5c0 0x7f43bc0b3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:28.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.112+0000 7f43c1c00640 1 --2- 172.21.15.90:0/3996473396 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43bc1550a0 0x7f43bc15bb20 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:28.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.112+0000 7f43c1c00640 1 -- 172.21.15.90:0/3996473396 >> 172.21.15.90:0/3996473396 conn(0x7f43bc0cf7e0 msgr2=0x7f43bc0cfbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:28.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.112+0000 7f43c1c00640 1 -- 172.21.15.90:0/3996473396 shutdown_connections 2024-03-24T01:25:28.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.112+0000 7f43c1c00640 1 -- 172.21.15.90:0/3996473396 wait complete. 2024-03-24T01:25:28.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.114+0000 7f43c1c00640 1 Processor -- start 2024-03-24T01:25:28.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.114+0000 7f43c1c00640 1 -- start start 2024-03-24T01:25:28.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.115+0000 7f43c1c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43bc06f5c0 0x7f43bc136970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:28.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.115+0000 7f43c1c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43bc0b3fe0 0x7f43bc14b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:28.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.115+0000 7f43c1c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43bc1550a0 0x7f43bc15d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:28.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.115+0000 7f43c1c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43bc15e9c0 con 0x7f43bc0b3fe0 2024-03-24T01:25:28.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.115+0000 7f43c1c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43bc15e6e0 con 0x7f43bc1550a0 2024-03-24T01:25:28.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.115+0000 7f43c1c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43bc15eca0 con 0x7f43bc06f5c0 2024-03-24T01:25:28.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.115+0000 7f43ba000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43bc0b3fe0 0x7f43bc14b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:28.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.115+0000 7f43ba000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43bc0b3fe0 0x7f43bc14b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43296/0 (socket says 172.21.15.90:43296) 2024-03-24T01:25:28.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.115+0000 7f43ba000640 1 -- 172.21.15.90:0/4211633656 learned_addr learned my addr 172.21.15.90:0/4211633656 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:28.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.115+0000 7f43baa00640 1 --2- 172.21.15.90:0/4211633656 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43bc06f5c0 0x7f43bc136970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:28.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.115+0000 7f43bb400640 1 --2- 172.21.15.90:0/4211633656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43bc1550a0 0x7f43bc15d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:28.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.115+0000 7f43ba000640 1 -- 172.21.15.90:0/4211633656 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43bc06f5c0 msgr2=0x7f43bc136970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:28.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.115+0000 7f43ba000640 1 --2- 172.21.15.90:0/4211633656 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43bc06f5c0 0x7f43bc136970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:28.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.115+0000 7f43ba000640 1 -- 172.21.15.90:0/4211633656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43bc1550a0 msgr2=0x7f43bc15d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:28.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.115+0000 7f43ba000640 1 --2- 172.21.15.90:0/4211633656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43bc1550a0 0x7f43bc15d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:28.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.115+0000 7f43ba000640 1 -- 172.21.15.90:0/4211633656 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43bc14c1f0 con 0x7f43bc0b3fe0 2024-03-24T01:25:28.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.116+0000 7f43baa00640 1 --2- 172.21.15.90:0/4211633656 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43bc06f5c0 0x7f43bc136970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:25:28.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.116+0000 7f43ba000640 1 --2- 172.21.15.90:0/4211633656 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43bc0b3fe0 0x7f43bc14b700 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f43a800b4d0 tx=0x7f43a800b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:28.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.116+0000 7f43b2a00640 1 -- 172.21.15.90:0/4211633656 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a8014070 con 0x7f43bc0b3fe0 2024-03-24T01:25:28.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.116+0000 7f43b2a00640 1 -- 172.21.15.90:0/4211633656 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a8010870 con 0x7f43bc0b3fe0 2024-03-24T01:25:28.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.116+0000 7f43b2a00640 1 -- 172.21.15.90:0/4211633656 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a8010c60 con 0x7f43bc0b3fe0 2024-03-24T01:25:28.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.116+0000 7f43c1c00640 1 -- 172.21.15.90:0/4211633656 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43bc14dea0 con 0x7f43bc0b3fe0 2024-03-24T01:25:28.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.116+0000 7f43c1c00640 1 -- 172.21.15.90:0/4211633656 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43bc14e460 con 0x7f43bc0b3fe0 2024-03-24T01:25:28.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.117+0000 7f43c1c00640 1 -- 172.21.15.90:0/4211633656 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43bc04f400 con 0x7f43bc0b3fe0 2024-03-24T01:25:28.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.118+0000 7f43b2a00640 1 -- 172.21.15.90:0/4211633656 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f43a8017070 con 0x7f43bc0b3fe0 2024-03-24T01:25:28.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.119+0000 7f43b2a00640 1 --2- 172.21.15.90:0/4211633656 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f439c06bbd0 0x7f439c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:28.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.119+0000 7f43b2a00640 1 -- 172.21.15.90:0/4211633656 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f43a808ad30 con 0x7f43bc0b3fe0 2024-03-24T01:25:28.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.120+0000 7f43baa00640 1 --2- 172.21.15.90:0/4211633656 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f439c06bbd0 0x7f439c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:28.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.121+0000 7f43baa00640 1 --2- 172.21.15.90:0/4211633656 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f439c06bbd0 0x7f439c06e090 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f43ac036000 tx=0x7f43ac03d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:28.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.123+0000 7f43b2a00640 1 -- 172.21.15.90:0/4211633656 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f43a8059360 con 0x7f43bc0b3fe0 2024-03-24T01:25:28.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.318+0000 7f43c1c00640 1 -- 172.21.15.90:0/4211633656 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f43bc0d04b0 con 0x7f43bc0b3fe0 2024-03-24T01:25:28.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.319+0000 7f43b2a00640 1 -- 172.21.15.90:0/4211633656 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f43a8058c60 con 0x7f43bc0b3fe0 2024-03-24T01:25:28.319 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:25:28.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.321+0000 7f43c1c00640 1 -- 172.21.15.90:0/4211633656 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f439c06bbd0 msgr2=0x7f439c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:28.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.321+0000 7f43c1c00640 1 --2- 172.21.15.90:0/4211633656 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f439c06bbd0 0x7f439c06e090 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f43ac036000 tx=0x7f43ac03d040 comp rx=0 tx=0).stop 2024-03-24T01:25:28.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.321+0000 7f43c1c00640 1 -- 172.21.15.90:0/4211633656 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43bc0b3fe0 msgr2=0x7f43bc14b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:28.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.321+0000 7f43c1c00640 1 --2- 172.21.15.90:0/4211633656 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43bc0b3fe0 0x7f43bc14b700 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f43a800b4d0 tx=0x7f43a800b990 comp rx=0 tx=0).stop 2024-03-24T01:25:28.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.321+0000 7f43c1c00640 1 -- 172.21.15.90:0/4211633656 shutdown_connections 2024-03-24T01:25:28.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.321+0000 7f43c1c00640 1 --2- 172.21.15.90:0/4211633656 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f439c06bbd0 0x7f439c06e090 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:28.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.321+0000 7f43c1c00640 1 --2- 172.21.15.90:0/4211633656 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43bc0b3fe0 0x7f43bc14b700 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:28.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.321+0000 7f43c1c00640 1 --2- 172.21.15.90:0/4211633656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43bc1550a0 0x7f43bc15d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:28.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.321+0000 7f43c1c00640 1 --2- 172.21.15.90:0/4211633656 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f43bc06f5c0 0x7f43bc136970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:28.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.322+0000 7f43c1c00640 1 -- 172.21.15.90:0/4211633656 >> 172.21.15.90:0/4211633656 conn(0x7f43bc0cf7e0 msgr2=0x7f43bc06d990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:28.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.322+0000 7f43c1c00640 1 -- 172.21.15.90:0/4211633656 shutdown_connections 2024-03-24T01:25:28.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.322+0000 7f43c1c00640 1 -- 172.21.15.90:0/4211633656 wait complete. 2024-03-24T01:25:28.339 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T01:25:28.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.544+0000 7f873b200640 1 Processor -- start 2024-03-24T01:25:28.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.544+0000 7f873b200640 1 -- start start 2024-03-24T01:25:28.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.545+0000 7f873b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8734159620 0x7f873406b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:28.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.546+0000 7f873b200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f873406bca0 con 0x7f87340cf150 2024-03-24T01:25:28.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.546+0000 7f873b200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87340b36f0 con 0x7f873406b600 2024-03-24T01:25:28.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.546+0000 7f873b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87340b4470 con 0x7f8734159620 2024-03-24T01:25:28.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.546+0000 7f8739200640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f87340cf150 0x7f87340ae530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:42416/0 (socket says 172.21.15.90:42416) 2024-03-24T01:25:28.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.546+0000 7f8739200640 1 -- 172.21.15.90:0/3828436744 learned_addr learned my addr 172.21.15.90:0/3828436744 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:28.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.546+0000 7f8733e00640 1 --2- 172.21.15.90:0/3828436744 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8734159620 0x7f873406b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:28.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.546+0000 7f8732a00640 1 -- 172.21.15.90:0/3828436744 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 780277273 0 0) 0x7f873406bca0 con 0x7f87340cf150 2024-03-24T01:25:28.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.547+0000 7f8732a00640 1 -- 172.21.15.90:0/3828436744 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8718003520 con 0x7f87340cf150 2024-03-24T01:25:28.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.547+0000 7f8732a00640 1 -- 172.21.15.90:0/3828436744 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3545665542 0 0) 0x7f8718003520 con 0x7f87340cf150 2024-03-24T01:25:28.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.547+0000 7f8732a00640 1 -- 172.21.15.90:0/3828436744 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8734159620 msgr2=0x7f873406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:28.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.547+0000 7f8732a00640 1 --2- 172.21.15.90:0/3828436744 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8734159620 0x7f873406b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.547+0000 7f8732a00640 1 -- 172.21.15.90:0/3828436744 >> v1:172.21.15.120:6789/0 conn(0x7f873406b600 legacy=0x7f87340cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.547+0000 7f8732a00640 1 -- 172.21.15.90:0/3828436744 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87340b3990 con 0x7f87340cf150 2024-03-24T01:25:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.547+0000 7f8732a00640 1 -- 172.21.15.90:0/3828436744 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f872c0019f0 con 0x7f87340cf150 2024-03-24T01:25:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.547+0000 7f8732a00640 1 -- 172.21.15.90:0/3828436744 >> v1:172.21.15.90:6789/0 conn(0x7f87340cf150 legacy=0x7f87340ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.547+0000 7f8732a00640 1 --2- 172.21.15.90:0/3828436744 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8734159620 0x7f87180039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.547+0000 7f8732a00640 1 --2- 172.21.15.90:0/3828436744 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8718003f20 0x7f8718004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.547+0000 7f8732a00640 1 --2- 172.21.15.90:0/3828436744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8718004d40 0x7f87180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.547+0000 7f8732a00640 1 -- 172.21.15.90:0/3828436744 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87340b4470 con 0x7f8718004d40 2024-03-24T01:25:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.547+0000 7f8732a00640 1 -- 172.21.15.90:0/3828436744 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f871800bb00 con 0x7f8718003f20 2024-03-24T01:25:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.547+0000 7f8732a00640 1 -- 172.21.15.90:0/3828436744 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f871800bd00 con 0x7f8734159620 2024-03-24T01:25:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.547+0000 7f8739200640 1 --2- 172.21.15.90:0/3828436744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8718004d40 0x7f87180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.548+0000 7f8733400640 1 --2- 172.21.15.90:0/3828436744 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8718003f20 0x7f8718004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.548+0000 7f8733e00640 1 --2- 172.21.15.90:0/3828436744 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8734159620 0x7f87180039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.548+0000 7f8739200640 1 -- 172.21.15.90:0/3828436744 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8734159620 msgr2=0x7f87180039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:28.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.548+0000 7f8739200640 1 --2- 172.21.15.90:0/3828436744 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8734159620 0x7f87180039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:28.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.548+0000 7f8739200640 1 -- 172.21.15.90:0/3828436744 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8718003f20 msgr2=0x7f8718004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:28.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.548+0000 7f8739200640 1 --2- 172.21.15.90:0/3828436744 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8718003f20 0x7f8718004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:28.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.548+0000 7f8739200640 1 -- 172.21.15.90:0/3828436744 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8718003520 con 0x7f8718004d40 2024-03-24T01:25:28.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.549+0000 7f8739200640 1 --2- 172.21.15.90:0/3828436744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8718004d40 0x7f87180051c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f872c00a7b0 tx=0x7f8718005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eaf330d05c718b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:28.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.549+0000 7f8732a00640 1 -- 172.21.15.90:0/3828436744 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872c00a3d0 con 0x7f8718004d40 2024-03-24T01:25:28.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.549+0000 7f8732a00640 1 -- 172.21.15.90:0/3828436744 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f872c034630 con 0x7f8718004d40 2024-03-24T01:25:28.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.549+0000 7f8732a00640 1 -- 172.21.15.90:0/3828436744 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872c034a60 con 0x7f8718004d40 2024-03-24T01:25:28.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.550+0000 7f873b200640 1 -- 172.21.15.90:0/3828436744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8718004d40 msgr2=0x7f87180051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:28.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.550+0000 7f873b200640 1 --2- 172.21.15.90:0/3828436744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8718004d40 0x7f87180051c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f872c00a7b0 tx=0x7f8718005830 comp rx=0 tx=0).stop 2024-03-24T01:25:28.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.550+0000 7f8739200640 1 -- 172.21.15.90:0/3828436744 reap_dead start 2024-03-24T01:25:28.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.550+0000 7f873b200640 1 -- 172.21.15.90:0/3828436744 shutdown_connections 2024-03-24T01:25:28.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.550+0000 7f873b200640 1 -- 172.21.15.90:0/3828436744 >> 172.21.15.90:0/3828436744 conn(0x7f87340b1770 msgr2=0x7f87340b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:28.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.550+0000 7f873b200640 1 -- 172.21.15.90:0/3828436744 shutdown_connections 2024-03-24T01:25:28.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.550+0000 7f873b200640 1 -- 172.21.15.90:0/3828436744 wait complete. 2024-03-24T01:25:28.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.552+0000 7f873b200640 1 Processor -- start 2024-03-24T01:25:28.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.552+0000 7f873b200640 1 -- start start 2024-03-24T01:25:28.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.553+0000 7f873b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f873414efe0 0x7f873412f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:28.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.553+0000 7f873b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8734155810 0x7f873415b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:28.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.553+0000 7f873b200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8734155c90 0x7f873415bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:28.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.553+0000 7f873b200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87340b3f50 con 0x7f873414efe0 2024-03-24T01:25:28.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.553+0000 7f873b200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873406ba60 con 0x7f8734155c90 2024-03-24T01:25:28.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.553+0000 7f873b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8734159a00 con 0x7f8734155810 2024-03-24T01:25:28.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.553+0000 7f8733e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f873414efe0 0x7f873412f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:28.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.553+0000 7f8733e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f873414efe0 0x7f873412f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43326/0 (socket says 172.21.15.90:43326) 2024-03-24T01:25:28.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.553+0000 7f8733e00640 1 -- 172.21.15.90:0/877421812 learned_addr learned my addr 172.21.15.90:0/877421812 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:28.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.554+0000 7f8739200640 1 --2- 172.21.15.90:0/877421812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8734155c90 0x7f873415bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:28.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.554+0000 7f8733400640 1 --2- 172.21.15.90:0/877421812 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8734155810 0x7f873415b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:28.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.554+0000 7f8733e00640 1 -- 172.21.15.90:0/877421812 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8734155810 msgr2=0x7f873415b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:28.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.554+0000 7f8733e00640 1 --2- 172.21.15.90:0/877421812 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8734155810 0x7f873415b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:28.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.554+0000 7f8733e00640 1 -- 172.21.15.90:0/877421812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8734155c90 msgr2=0x7f873415bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:28.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.554+0000 7f8733e00640 1 --2- 172.21.15.90:0/877421812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8734155c90 0x7f873415bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:28.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.554+0000 7f8733e00640 1 -- 172.21.15.90:0/877421812 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f873412f950 con 0x7f873414efe0 2024-03-24T01:25:28.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.554+0000 7f8733e00640 1 --2- 172.21.15.90:0/877421812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f873414efe0 0x7f873412f040 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8724011600 tx=0x7f8724011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:28.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.554+0000 7f872be00640 1 -- 172.21.15.90:0/877421812 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872400f920 con 0x7f873414efe0 2024-03-24T01:25:28.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.554+0000 7f872be00640 1 -- 172.21.15.90:0/877421812 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f872400fac0 con 0x7f873414efe0 2024-03-24T01:25:28.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.554+0000 7f872be00640 1 -- 172.21.15.90:0/877421812 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872400fe70 con 0x7f873414efe0 2024-03-24T01:25:28.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.554+0000 7f873b200640 1 -- 172.21.15.90:0/877421812 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87341f2aa0 con 0x7f873414efe0 2024-03-24T01:25:28.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.555+0000 7f873b200640 1 -- 172.21.15.90:0/877421812 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8734154d30 con 0x7f873414efe0 2024-03-24T01:25:28.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.556+0000 7f873b200640 1 -- 172.21.15.90:0/877421812 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87340cd7a0 con 0x7f873414efe0 2024-03-24T01:25:28.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.556+0000 7f872be00640 1 -- 172.21.15.90:0/877421812 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8724015460 con 0x7f873414efe0 2024-03-24T01:25:28.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.557+0000 7f872be00640 1 --2- 172.21.15.90:0/877421812 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f871406bbd0 0x7f871406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:28.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.557+0000 7f872be00640 1 -- 172.21.15.90:0/877421812 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f872408b890 con 0x7f873414efe0 2024-03-24T01:25:28.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.558+0000 7f8733400640 1 --2- 172.21.15.90:0/877421812 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f871406bbd0 0x7f871406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:28.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.559+0000 7f8733400640 1 --2- 172.21.15.90:0/877421812 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f871406bbd0 0x7f871406e090 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f873415e830 tx=0x7f8720009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:28.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.562+0000 7f872be00640 1 -- 172.21.15.90:0/877421812 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f87340cd7a0 con 0x7f873414efe0 2024-03-24T01:25:28.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.760+0000 7f873b200640 1 -- 172.21.15.90:0/877421812 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f87340ce620 con 0x7f873414efe0 2024-03-24T01:25:28.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.761+0000 7f872be00640 1 -- 172.21.15.90:0/877421812 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f87340ce620 con 0x7f873414efe0 2024-03-24T01:25:28.761 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-24T01:25:28.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.764+0000 7f873b200640 1 -- 172.21.15.90:0/877421812 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f871406bbd0 msgr2=0x7f871406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:28.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.764+0000 7f873b200640 1 --2- 172.21.15.90:0/877421812 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f871406bbd0 0x7f871406e090 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f873415e830 tx=0x7f8720009040 comp rx=0 tx=0).stop 2024-03-24T01:25:28.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.764+0000 7f873b200640 1 -- 172.21.15.90:0/877421812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f873414efe0 msgr2=0x7f873412f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:28.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.764+0000 7f873b200640 1 --2- 172.21.15.90:0/877421812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f873414efe0 0x7f873412f040 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8724011600 tx=0x7f8724011ac0 comp rx=0 tx=0).stop 2024-03-24T01:25:28.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.764+0000 7f873b200640 1 -- 172.21.15.90:0/877421812 shutdown_connections 2024-03-24T01:25:28.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.764+0000 7f873b200640 1 --2- 172.21.15.90:0/877421812 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f871406bbd0 0x7f871406e090 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:28.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.764+0000 7f873b200640 1 --2- 172.21.15.90:0/877421812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8734155c90 0x7f873415bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:28.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.764+0000 7f873b200640 1 --2- 172.21.15.90:0/877421812 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8734155810 0x7f873415b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:28.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.764+0000 7f873b200640 1 --2- 172.21.15.90:0/877421812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f873414efe0 0x7f873412f040 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:28.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.764+0000 7f873b200640 1 -- 172.21.15.90:0/877421812 >> 172.21.15.90:0/877421812 conn(0x7f87340b1770 msgr2=0x7f87341561d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:28.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.764+0000 7f873b200640 1 -- 172.21.15.90:0/877421812 shutdown_connections 2024-03-24T01:25:28.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.765+0000 7f873b200640 1 -- 172.21.15.90:0/877421812 wait complete. 2024-03-24T01:25:28.783 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T01:25:28.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.993+0000 7f9c37c00640 1 Processor -- start 2024-03-24T01:25:28.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.993+0000 7f9c37c00640 1 -- start start 2024-03-24T01:25:28.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.995+0000 7f9c37c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9c3006b7d0 0x7f9c300acea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:28.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.995+0000 7f9c37c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c300af140 con 0x7f9c3015de10 2024-03-24T01:25:28.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.995+0000 7f9c37c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c3015e550 con 0x7f9c3015da30 2024-03-24T01:25:28.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.995+0000 7f9c37c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c300b2c70 con 0x7f9c3006b7d0 2024-03-24T01:25:28.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.995+0000 7f9c35c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9c3006b7d0 0x7f9c300acea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:28.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.995+0000 7f9c35c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9c3006b7d0 0x7f9c300acea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43332/0 (socket says 172.21.15.90:43332) 2024-03-24T01:25:28.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.995+0000 7f9c35c00640 1 -- 172.21.15.90:0/3820094077 learned_addr learned my addr 172.21.15.90:0/3820094077 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:28.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.995+0000 7f9c2fe00640 1 --1- 172.21.15.90:0/3820094077 >> v1:172.21.15.90:6789/0 conn(0x7f9c3015de10 0x7f9c3006b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:42428/0 (socket says 172.21.15.90:42428) 2024-03-24T01:25:28.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.996+0000 7f9c35c00640 1 -- 172.21.15.90:0/3820094077 >> v1:172.21.15.120:6789/0 conn(0x7f9c3015da30 legacy=0x7f9c300aea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:28.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.996+0000 7f9c35c00640 1 -- 172.21.15.90:0/3820094077 >> v1:172.21.15.90:6789/0 conn(0x7f9c3015de10 legacy=0x7f9c3006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:28.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.996+0000 7f9c35c00640 1 -- 172.21.15.90:0/3820094077 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c3015e830 con 0x7f9c3006b7d0 2024-03-24T01:25:28.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.997+0000 7f9c35c00640 1 --2- 172.21.15.90:0/3820094077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c3006b7d0 0x7f9c300acea0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f9c28009e50 tx=0x7f9c2802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dff1fd4590917e70 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:28.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.997+0000 7f9c2f400640 1 -- 172.21.15.90:0/3820094077 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c2802cb10 con 0x7f9c3006b7d0 2024-03-24T01:25:28.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.997+0000 7f9c2f400640 1 -- 172.21.15.90:0/3820094077 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c2802ccb0 con 0x7f9c3006b7d0 2024-03-24T01:25:28.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.997+0000 7f9c2f400640 1 -- 172.21.15.90:0/3820094077 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c2802ce60 con 0x7f9c3006b7d0 2024-03-24T01:25:28.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.998+0000 7f9c37c00640 1 -- 172.21.15.90:0/3820094077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c3006b7d0 msgr2=0x7f9c300acea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:28.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.998+0000 7f9c37c00640 1 --2- 172.21.15.90:0/3820094077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c3006b7d0 0x7f9c300acea0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f9c28009e50 tx=0x7f9c2802bf70 comp rx=0 tx=0).stop 2024-03-24T01:25:28.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.998+0000 7f9c37c00640 1 -- 172.21.15.90:0/3820094077 shutdown_connections 2024-03-24T01:25:28.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.998+0000 7f9c37c00640 1 --2- 172.21.15.90:0/3820094077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c3006b7d0 0x7f9c300acea0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:28.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.998+0000 7f9c37c00640 1 -- 172.21.15.90:0/3820094077 >> 172.21.15.90:0/3820094077 conn(0x7f9c300b1770 msgr2=0x7f9c300b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:28.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.998+0000 7f9c37c00640 1 -- 172.21.15.90:0/3820094077 shutdown_connections 2024-03-24T01:25:28.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:28.998+0000 7f9c37c00640 1 -- 172.21.15.90:0/3820094077 wait complete. 2024-03-24T01:25:29.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.000+0000 7f9c37c00640 1 Processor -- start 2024-03-24T01:25:29.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.000+0000 7f9c37c00640 1 -- start start 2024-03-24T01:25:29.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.001+0000 7f9c37c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c3006b7d0 0x7f9c300cbd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:29.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.001+0000 7f9c37c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9c3015da30 0x7f9c301f7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:29.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.001+0000 7f9c37c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c3015de10 0x7f9c301f7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:29.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.001+0000 7f9c37c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c300ad3e0 con 0x7f9c3015de10 2024-03-24T01:25:29.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.001+0000 7f9c37c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3015f160 con 0x7f9c3006b7d0 2024-03-24T01:25:29.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.001+0000 7f9c37c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3015edf0 con 0x7f9c3015da30 2024-03-24T01:25:29.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.001+0000 7f9c35c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c3015de10 0x7f9c301f7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:29.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.001+0000 7f9c35c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c3015de10 0x7f9c301f7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43334/0 (socket says 172.21.15.90:43334) 2024-03-24T01:25:29.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.001+0000 7f9c35c00640 1 -- 172.21.15.90:0/3436203845 learned_addr learned my addr 172.21.15.90:0/3436203845 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:29.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.002+0000 7f9c2fe00640 1 --2- 172.21.15.90:0/3436203845 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9c3015da30 0x7f9c301f7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:29.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.002+0000 7f9c35200640 1 --2- 172.21.15.90:0/3436203845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c3006b7d0 0x7f9c300cbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:29.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.002+0000 7f9c35c00640 1 -- 172.21.15.90:0/3436203845 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9c3015da30 msgr2=0x7f9c301f7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:29.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.002+0000 7f9c35c00640 1 --2- 172.21.15.90:0/3436203845 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9c3015da30 0x7f9c301f7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:29.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.002+0000 7f9c35c00640 1 -- 172.21.15.90:0/3436203845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c3006b7d0 msgr2=0x7f9c300cbd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:29.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.002+0000 7f9c35c00640 1 --2- 172.21.15.90:0/3436203845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c3006b7d0 0x7f9c300cbd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:29.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.002+0000 7f9c35c00640 1 -- 172.21.15.90:0/3436203845 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c3014cd60 con 0x7f9c3015de10 2024-03-24T01:25:29.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.002+0000 7f9c35200640 1 --2- 172.21.15.90:0/3436203845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c3006b7d0 0x7f9c300cbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:29.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.002+0000 7f9c35c00640 1 --2- 172.21.15.90:0/3436203845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c3015de10 0x7f9c301f7760 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f9c28036040 tx=0x7f9c28004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:29.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.002+0000 7f9c2cc00640 1 -- 172.21.15.90:0/3436203845 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c28043020 con 0x7f9c3015de10 2024-03-24T01:25:29.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.002+0000 7f9c2cc00640 1 -- 172.21.15.90:0/3436203845 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c2803e450 con 0x7f9c3015de10 2024-03-24T01:25:29.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.002+0000 7f9c2cc00640 1 -- 172.21.15.90:0/3436203845 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c2802ce60 con 0x7f9c3015de10 2024-03-24T01:25:29.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.002+0000 7f9c37c00640 1 -- 172.21.15.90:0/3436203845 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c301f7e00 con 0x7f9c3015de10 2024-03-24T01:25:29.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.003+0000 7f9c37c00640 1 -- 172.21.15.90:0/3436203845 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c300ceb20 con 0x7f9c3015de10 2024-03-24T01:25:29.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.004+0000 7f9c37c00640 1 -- 172.21.15.90:0/3436203845 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c3004f400 con 0x7f9c3015de10 2024-03-24T01:25:29.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.004+0000 7f9c2cc00640 1 -- 172.21.15.90:0/3436203845 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9c28046020 con 0x7f9c3015de10 2024-03-24T01:25:29.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.005+0000 7f9c2cc00640 1 --2- 172.21.15.90:0/3436203845 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9c1006bbd0 0x7f9c1006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:29.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.005+0000 7f9c2cc00640 1 -- 172.21.15.90:0/3436203845 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f9c280ad550 con 0x7f9c3015de10 2024-03-24T01:25:29.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.006+0000 7f9c35200640 1 --2- 172.21.15.90:0/3436203845 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9c1006bbd0 0x7f9c1006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:29.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.007+0000 7f9c35200640 1 --2- 172.21.15.90:0/3436203845 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9c1006bbd0 0x7f9c1006e090 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9c24003eb0 tx=0x7f9c24009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:29.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.010+0000 7f9c2cc00640 1 -- 172.21.15.90:0/3436203845 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9c2804d030 con 0x7f9c3015de10 2024-03-24T01:25:29.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.205+0000 7f9c37c00640 1 -- 172.21.15.90:0/3436203845 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f9c3006e250 con 0x7f9c3015de10 2024-03-24T01:25:29.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.205+0000 7f9c2cc00640 1 -- 172.21.15.90:0/3436203845 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f9c3014cd60 con 0x7f9c3015de10 2024-03-24T01:25:29.206 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:25:29.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.208+0000 7f9c37c00640 1 -- 172.21.15.90:0/3436203845 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9c1006bbd0 msgr2=0x7f9c1006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:29.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.208+0000 7f9c37c00640 1 --2- 172.21.15.90:0/3436203845 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9c1006bbd0 0x7f9c1006e090 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9c24003eb0 tx=0x7f9c24009040 comp rx=0 tx=0).stop 2024-03-24T01:25:29.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.208+0000 7f9c37c00640 1 -- 172.21.15.90:0/3436203845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c3015de10 msgr2=0x7f9c301f7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:29.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.208+0000 7f9c37c00640 1 --2- 172.21.15.90:0/3436203845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c3015de10 0x7f9c301f7760 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f9c28036040 tx=0x7f9c28004a40 comp rx=0 tx=0).stop 2024-03-24T01:25:29.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.208+0000 7f9c37c00640 1 -- 172.21.15.90:0/3436203845 shutdown_connections 2024-03-24T01:25:29.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.208+0000 7f9c37c00640 1 --2- 172.21.15.90:0/3436203845 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9c1006bbd0 0x7f9c1006e090 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:29.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.208+0000 7f9c37c00640 1 --2- 172.21.15.90:0/3436203845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c3015de10 0x7f9c301f7760 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:29.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.208+0000 7f9c37c00640 1 --2- 172.21.15.90:0/3436203845 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9c3015da30 0x7f9c301f7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:29.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.208+0000 7f9c37c00640 1 --2- 172.21.15.90:0/3436203845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c3006b7d0 0x7f9c300cbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:29.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.208+0000 7f9c37c00640 1 -- 172.21.15.90:0/3436203845 >> 172.21.15.90:0/3436203845 conn(0x7f9c300b1770 msgr2=0x7f9c3006f330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:29.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.208+0000 7f9c37c00640 1 -- 172.21.15.90:0/3436203845 shutdown_connections 2024-03-24T01:25:29.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.208+0000 7f9c37c00640 1 -- 172.21.15.90:0/3436203845 wait complete. 2024-03-24T01:25:29.225 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T01:25:29.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.433+0000 7f2947e00640 1 Processor -- start 2024-03-24T01:25:29.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.433+0000 7f2947e00640 1 -- start start 2024-03-24T01:25:29.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.435+0000 7f2947e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29481631c0 0x7f2948071790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:29.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.435+0000 7f2947e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29480af230 con 0x7f2948071cd0 2024-03-24T01:25:29.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.435+0000 7f2947e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29480b4c00 con 0x7f29480aee50 2024-03-24T01:25:29.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.435+0000 7f2947e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2948072120 con 0x7f29481631c0 2024-03-24T01:25:29.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.435+0000 7f2947400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f2948071cd0 0x7f29480b27b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:42432/0 (socket says 172.21.15.90:42432) 2024-03-24T01:25:29.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.435+0000 7f2947400640 1 -- 172.21.15.90:0/878818625 learned_addr learned my addr 172.21.15.90:0/878818625 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:29.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.435+0000 7f2946000640 1 --2- 172.21.15.90:0/878818625 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29481631c0 0x7f2948071790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:29.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.436+0000 7f2945600640 1 -- 172.21.15.90:0/878818625 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3554053475 0 0) 0x7f29480af230 con 0x7f2948071cd0 2024-03-24T01:25:29.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.436+0000 7f2945600640 1 -- 172.21.15.90:0/878818625 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f292c003520 con 0x7f2948071cd0 2024-03-24T01:25:29.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.436+0000 7f2945600640 1 -- 172.21.15.90:0/878818625 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1330608288 0 0) 0x7f292c003520 con 0x7f2948071cd0 2024-03-24T01:25:29.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.436+0000 7f2945600640 1 -- 172.21.15.90:0/878818625 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29481631c0 msgr2=0x7f2948071790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:29.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.436+0000 7f2945600640 1 --2- 172.21.15.90:0/878818625 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29481631c0 0x7f2948071790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:29.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.436+0000 7f2945600640 1 -- 172.21.15.90:0/878818625 >> v1:172.21.15.120:6789/0 conn(0x7f29480aee50 legacy=0x7f29480b4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:29.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.436+0000 7f2945600640 1 -- 172.21.15.90:0/878818625 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f294806d890 con 0x7f2948071cd0 2024-03-24T01:25:29.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.436+0000 7f2945600640 1 -- 172.21.15.90:0/878818625 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f2940001670 con 0x7f2948071cd0 2024-03-24T01:25:29.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.436+0000 7f2945600640 1 -- 172.21.15.90:0/878818625 >> v1:172.21.15.90:6789/0 conn(0x7f2948071cd0 legacy=0x7f29480b27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:29.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.436+0000 7f2945600640 1 --2- 172.21.15.90:0/878818625 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29481631c0 0x7f292c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:29.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.436+0000 7f2945600640 1 --2- 172.21.15.90:0/878818625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f292c003f60 0x7f292c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:29.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.436+0000 7f2945600640 1 --2- 172.21.15.90:0/878818625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f292c004d40 0x7f292c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:29.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.436+0000 7f2945600640 1 -- 172.21.15.90:0/878818625 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2948072120 con 0x7f292c004d40 2024-03-24T01:25:29.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.436+0000 7f2945600640 1 -- 172.21.15.90:0/878818625 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f292c00baa0 con 0x7f292c003f60 2024-03-24T01:25:29.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.436+0000 7f2945600640 1 -- 172.21.15.90:0/878818625 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f292c00bca0 con 0x7f29481631c0 2024-03-24T01:25:29.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.436+0000 7f2947400640 1 --2- 172.21.15.90:0/878818625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f292c004d40 0x7f292c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:29.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.437+0000 7f2946a00640 1 --2- 172.21.15.90:0/878818625 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29481631c0 0x7f292c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:29.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.437+0000 7f2946000640 1 --2- 172.21.15.90:0/878818625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f292c003f60 0x7f292c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:29.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.437+0000 7f2947400640 1 -- 172.21.15.90:0/878818625 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29481631c0 msgr2=0x7f292c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:29.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.437+0000 7f2947400640 1 --2- 172.21.15.90:0/878818625 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29481631c0 0x7f292c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:29.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.437+0000 7f2947400640 1 -- 172.21.15.90:0/878818625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f292c003f60 msgr2=0x7f292c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:29.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.437+0000 7f2947400640 1 --2- 172.21.15.90:0/878818625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f292c003f60 0x7f292c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:29.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.437+0000 7f2947400640 1 -- 172.21.15.90:0/878818625 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f292c003520 con 0x7f292c004d40 2024-03-24T01:25:29.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.437+0000 7f2946000640 1 --2- 172.21.15.90:0/878818625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f292c003f60 0x7f292c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:29.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.438+0000 7f2947400640 1 --2- 172.21.15.90:0/878818625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f292c004d40 0x7f292c0051c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f294000c240 tx=0x7f292c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86cfe153d8673276 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:29.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.438+0000 7f2945600640 1 -- 172.21.15.90:0/878818625 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2940007d10 con 0x7f292c004d40 2024-03-24T01:25:29.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.438+0000 7f2945600640 1 -- 172.21.15.90:0/878818625 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2940030070 con 0x7f292c004d40 2024-03-24T01:25:29.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.438+0000 7f2945600640 1 -- 172.21.15.90:0/878818625 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f294002cda0 con 0x7f292c004d40 2024-03-24T01:25:29.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.439+0000 7f2947e00640 1 -- 172.21.15.90:0/878818625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f292c004d40 msgr2=0x7f292c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:29.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.439+0000 7f2947e00640 1 --2- 172.21.15.90:0/878818625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f292c004d40 0x7f292c0051c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f294000c240 tx=0x7f292c005830 comp rx=0 tx=0).stop 2024-03-24T01:25:29.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.439+0000 7f2947400640 1 -- 172.21.15.90:0/878818625 reap_dead start 2024-03-24T01:25:29.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.439+0000 7f2947e00640 1 -- 172.21.15.90:0/878818625 shutdown_connections 2024-03-24T01:25:29.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.439+0000 7f2947e00640 1 -- 172.21.15.90:0/878818625 >> 172.21.15.90:0/878818625 conn(0x7f294806b0c0 msgr2=0x7f294806b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:29.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.439+0000 7f2947e00640 1 -- 172.21.15.90:0/878818625 shutdown_connections 2024-03-24T01:25:29.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.439+0000 7f2947e00640 1 -- 172.21.15.90:0/878818625 wait complete. 2024-03-24T01:25:29.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.441+0000 7f2947e00640 1 Processor -- start 2024-03-24T01:25:29.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.441+0000 7f2947e00640 1 -- start start 2024-03-24T01:25:29.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.442+0000 7f2947e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f292c004d40 0x7f2948150430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:29.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.442+0000 7f2947e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29480a8c30 0x7f294814cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:29.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.442+0000 7f2947e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f294814d800 0x7f294814dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:29.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.442+0000 7f2947e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294806e130 con 0x7f292c004d40 2024-03-24T01:25:29.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.442+0000 7f2947e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294806de50 con 0x7f29480a8c30 2024-03-24T01:25:29.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.442+0000 7f2947e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f294806e630 con 0x7f294814d800 2024-03-24T01:25:29.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.442+0000 7f2946a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f292c004d40 0x7f2948150430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:29.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.442+0000 7f2946a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f292c004d40 0x7f2948150430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43352/0 (socket says 172.21.15.90:43352) 2024-03-24T01:25:29.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.442+0000 7f2946a00640 1 -- 172.21.15.90:0/3207638738 learned_addr learned my addr 172.21.15.90:0/3207638738 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:29.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.442+0000 7f2946000640 1 --2- 172.21.15.90:0/3207638738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29480a8c30 0x7f294814cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:29.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.442+0000 7f2947400640 1 --2- 172.21.15.90:0/3207638738 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f294814d800 0x7f294814dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:29.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.443+0000 7f2946a00640 1 -- 172.21.15.90:0/3207638738 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f294814d800 msgr2=0x7f294814dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:29.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.443+0000 7f2946a00640 1 --2- 172.21.15.90:0/3207638738 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f294814d800 0x7f294814dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:29.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.443+0000 7f2946a00640 1 -- 172.21.15.90:0/3207638738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29480a8c30 msgr2=0x7f294814cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:29.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.443+0000 7f2946a00640 1 --2- 172.21.15.90:0/3207638738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29480a8c30 0x7f294814cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:29.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.443+0000 7f2946a00640 1 -- 172.21.15.90:0/3207638738 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29480cd920 con 0x7f292c004d40 2024-03-24T01:25:29.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.443+0000 7f2947400640 1 --2- 172.21.15.90:0/3207638738 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f294814d800 0x7f294814dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:29.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.443+0000 7f2946a00640 1 --2- 172.21.15.90:0/3207638738 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f292c004d40 0x7f2948150430 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f293800e810 tx=0x7f293800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:29.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.443+0000 7f293ea00640 1 -- 172.21.15.90:0/3207638738 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2938003b80 con 0x7f292c004d40 2024-03-24T01:25:29.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.443+0000 7f293ea00640 1 -- 172.21.15.90:0/3207638738 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2938018070 con 0x7f292c004d40 2024-03-24T01:25:29.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.443+0000 7f2947e00640 1 -- 172.21.15.90:0/3207638738 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29480cdc00 con 0x7f292c004d40 2024-03-24T01:25:29.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.443+0000 7f293ea00640 1 -- 172.21.15.90:0/3207638738 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29380050d0 con 0x7f292c004d40 2024-03-24T01:25:29.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.443+0000 7f2947e00640 1 -- 172.21.15.90:0/3207638738 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29480ce1c0 con 0x7f292c004d40 2024-03-24T01:25:29.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.445+0000 7f2947e00640 1 -- 172.21.15.90:0/3207638738 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f294804f400 con 0x7f292c004d40 2024-03-24T01:25:29.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.446+0000 7f293ea00640 1 -- 172.21.15.90:0/3207638738 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2938005270 con 0x7f292c004d40 2024-03-24T01:25:29.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.446+0000 7f293ea00640 1 --2- 172.21.15.90:0/3207638738 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f292406bc00 0x7f292406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:29.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.447+0000 7f293ea00640 1 -- 172.21.15.90:0/3207638738 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f293808aea0 con 0x7f292c004d40 2024-03-24T01:25:29.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.447+0000 7f2946000640 1 --2- 172.21.15.90:0/3207638738 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f292406bc00 0x7f292406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:29.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.448+0000 7f2946000640 1 --2- 172.21.15.90:0/3207638738 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f292406bc00 0x7f292406e0c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f29480ae500 tx=0x7f2930008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:29.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.451+0000 7f293ea00640 1 -- 172.21.15.90:0/3207638738 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2938059570 con 0x7f292c004d40 2024-03-24T01:25:29.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.664+0000 7f2947e00640 1 -- 172.21.15.90:0/3207638738 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f2948161070 con 0x7f292c004d40 2024-03-24T01:25:29.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.664+0000 7f293ea00640 1 -- 172.21.15.90:0/3207638738 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f293801be40 con 0x7f292c004d40 2024-03-24T01:25:29.665 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-24T01:25:29.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.667+0000 7f2947e00640 1 -- 172.21.15.90:0/3207638738 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f292406bc00 msgr2=0x7f292406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:29.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.667+0000 7f2947e00640 1 --2- 172.21.15.90:0/3207638738 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f292406bc00 0x7f292406e0c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f29480ae500 tx=0x7f2930008040 comp rx=0 tx=0).stop 2024-03-24T01:25:29.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.667+0000 7f2947e00640 1 -- 172.21.15.90:0/3207638738 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f292c004d40 msgr2=0x7f2948150430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:29.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.667+0000 7f2947e00640 1 --2- 172.21.15.90:0/3207638738 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f292c004d40 0x7f2948150430 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f293800e810 tx=0x7f293800ecd0 comp rx=0 tx=0).stop 2024-03-24T01:25:29.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.667+0000 7f2947e00640 1 -- 172.21.15.90:0/3207638738 shutdown_connections 2024-03-24T01:25:29.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.667+0000 7f2947e00640 1 --2- 172.21.15.90:0/3207638738 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f292406bc00 0x7f292406e0c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:29.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.667+0000 7f2947e00640 1 --2- 172.21.15.90:0/3207638738 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f294814d800 0x7f294814dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:29.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.667+0000 7f2947e00640 1 --2- 172.21.15.90:0/3207638738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29480a8c30 0x7f294814cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:29.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.667+0000 7f2947e00640 1 --2- 172.21.15.90:0/3207638738 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f292c004d40 0x7f2948150430 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:29.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.667+0000 7f2947e00640 1 -- 172.21.15.90:0/3207638738 >> 172.21.15.90:0/3207638738 conn(0x7f294806b0c0 msgr2=0x7f29481605a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:29.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.667+0000 7f2947e00640 1 -- 172.21.15.90:0/3207638738 shutdown_connections 2024-03-24T01:25:29.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:29.667+0000 7f2947e00640 1 -- 172.21.15.90:0/3207638738 wait complete. 2024-03-24T01:25:29.686 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi120.front.sepia.ceph.com... 2024-03-24T01:25:29.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:29.686 DEBUG:teuthology.orchestra.run.smithi120:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-03-24T01:25:29.783 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:25:29.783 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2024-03-24T01:25:29.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:29.783 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-24T01:25:29.837 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:25:29.837 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:25:29.894 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:25:29.894 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:25:29.946 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:25:30.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.153+0000 7f41ec800640 1 Processor -- start 2024-03-24T01:25:30.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.153+0000 7f41ec800640 1 -- start start 2024-03-24T01:25:30.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.155+0000 7f41ec800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f41e40cc150 0x7f41e40ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:30.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.155+0000 7f41ec800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41e41603e0 0x7f41e406b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:30.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.155+0000 7f41ec800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41e40cd490 con 0x7f41e4145430 2024-03-24T01:25:30.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.155+0000 7f41ec800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41e415f900 con 0x7f41e40cc150 2024-03-24T01:25:30.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.155+0000 7f41ec800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41e415fbe0 con 0x7f41e41603e0 2024-03-24T01:25:30.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.155+0000 7f41e9e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f41e40cc150 0x7f41e40ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:30.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.155+0000 7f41ea800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f41e4145430 0x7f41e4071990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60972/0 (socket says 172.21.15.90:60972) 2024-03-24T01:25:30.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.155+0000 7f41ea800640 1 -- 172.21.15.90:0/38317037 learned_addr learned my addr 172.21.15.90:0/38317037 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:30.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.155+0000 7f41e9400640 1 --2- 172.21.15.90:0/38317037 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41e41603e0 0x7f41e406b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:30.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.156+0000 7f41e9e00640 1 -- 172.21.15.90:0/38317037 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41e41603e0 msgr2=0x7f41e406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:30.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.156+0000 7f41e9e00640 1 --2- 172.21.15.90:0/38317037 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41e41603e0 0x7f41e406b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:30.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.156+0000 7f41e9e00640 1 -- 172.21.15.90:0/38317037 >> v1:172.21.15.90:6789/0 conn(0x7f41e4145430 legacy=0x7f41e4071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:30.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.156+0000 7f41e9e00640 1 -- 172.21.15.90:0/38317037 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e415fec0 con 0x7f41e40cc150 2024-03-24T01:25:30.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.156+0000 7f41e9400640 1 --2- 172.21.15.90:0/38317037 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41e41603e0 0x7f41e406b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:25:30.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.156+0000 7f41e8a00640 1 -- 172.21.15.90:0/38317037 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 954974157 0 0) 0x7f41e40cd490 con 0x7f41e4145430 2024-03-24T01:25:30.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.157+0000 7f41e9e00640 1 --2- 172.21.15.90:0/38317037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f41e40cc150 0x7f41e40ccb60 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f41dc009d90 tx=0x7f41dc02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71ca31645bba7184 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:30.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.157+0000 7f41e8a00640 1 -- 172.21.15.90:0/38317037 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41dc039070 con 0x7f41e40cc150 2024-03-24T01:25:30.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.157+0000 7f41e8a00640 1 -- 172.21.15.90:0/38317037 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41dc02cc50 con 0x7f41e40cc150 2024-03-24T01:25:30.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.157+0000 7f41e8a00640 1 -- 172.21.15.90:0/38317037 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41dc034440 con 0x7f41e40cc150 2024-03-24T01:25:30.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.158+0000 7f41ec800640 1 -- 172.21.15.90:0/38317037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f41e40cc150 msgr2=0x7f41e40ccb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:30.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.158+0000 7f41ec800640 1 --2- 172.21.15.90:0/38317037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f41e40cc150 0x7f41e40ccb60 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f41dc009d90 tx=0x7f41dc02bea0 comp rx=0 tx=0).stop 2024-03-24T01:25:30.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.158+0000 7f41ec800640 1 -- 172.21.15.90:0/38317037 shutdown_connections 2024-03-24T01:25:30.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.158+0000 7f41ec800640 1 --2- 172.21.15.90:0/38317037 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41e41603e0 0x7f41e406b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:30.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.158+0000 7f41ec800640 1 --2- 172.21.15.90:0/38317037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f41e40cc150 0x7f41e40ccb60 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:30.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.158+0000 7f41ec800640 1 -- 172.21.15.90:0/38317037 >> 172.21.15.90:0/38317037 conn(0x7f41e40b1770 msgr2=0x7f41e40b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:30.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.158+0000 7f41ec800640 1 -- 172.21.15.90:0/38317037 shutdown_connections 2024-03-24T01:25:30.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.158+0000 7f41ec800640 1 -- 172.21.15.90:0/38317037 wait complete. 2024-03-24T01:25:30.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.159+0000 7f41ec800640 1 Processor -- start 2024-03-24T01:25:30.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.160+0000 7f41ec800640 1 -- start start 2024-03-24T01:25:30.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.160+0000 7f41ec800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41e40cc150 0x7f41e4139cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:30.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.160+0000 7f41ec800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41e4145430 0x7f41e406f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:30.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.160+0000 7f41ec800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f41e41603e0 0x7f41e415ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:30.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.161+0000 7f41ec800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e406b780 con 0x7f41e41603e0 2024-03-24T01:25:30.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.161+0000 7f41ec800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e406b600 con 0x7f41e40cc150 2024-03-24T01:25:30.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.161+0000 7f41ec800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41e406b900 con 0x7f41e4145430 2024-03-24T01:25:30.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.161+0000 7f41ea800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f41e41603e0 0x7f41e415ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:30.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.161+0000 7f41ea800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f41e41603e0 0x7f41e415ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54964/0 (socket says 172.21.15.90:54964) 2024-03-24T01:25:30.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.161+0000 7f41ea800640 1 -- 172.21.15.90:0/47965737 learned_addr learned my addr 172.21.15.90:0/47965737 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:30.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.161+0000 7f41e9400640 1 --2- 172.21.15.90:0/47965737 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41e4145430 0x7f41e406f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:30.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.161+0000 7f41ea800640 1 -- 172.21.15.90:0/47965737 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41e4145430 msgr2=0x7f41e406f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:30.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.161+0000 7f41ea800640 1 --2- 172.21.15.90:0/47965737 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41e4145430 0x7f41e406f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:30.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.161+0000 7f41e9e00640 1 --2- 172.21.15.90:0/47965737 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41e40cc150 0x7f41e4139cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:30.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.161+0000 7f41ea800640 1 -- 172.21.15.90:0/47965737 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41e40cc150 msgr2=0x7f41e4139cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:30.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.161+0000 7f41ea800640 1 --2- 172.21.15.90:0/47965737 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41e40cc150 0x7f41e4139cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:30.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.161+0000 7f41ea800640 1 -- 172.21.15.90:0/47965737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e413c1f0 con 0x7f41e41603e0 2024-03-24T01:25:30.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.161+0000 7f41e9400640 1 --2- 172.21.15.90:0/47965737 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41e4145430 0x7f41e406f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:30.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.162+0000 7f41ea800640 1 --2- 172.21.15.90:0/47965737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f41e41603e0 0x7f41e415ae00 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f41e000c450 tx=0x7f41e000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:30.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.162+0000 7f41da000640 1 -- 172.21.15.90:0/47965737 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e0018070 con 0x7f41e41603e0 2024-03-24T01:25:30.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.162+0000 7f41da000640 1 -- 172.21.15.90:0/47965737 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41e0007500 con 0x7f41e41603e0 2024-03-24T01:25:30.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.162+0000 7f41da000640 1 -- 172.21.15.90:0/47965737 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e0004a20 con 0x7f41e41603e0 2024-03-24T01:25:30.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.162+0000 7f41ec800640 1 -- 172.21.15.90:0/47965737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41e413c4d0 con 0x7f41e41603e0 2024-03-24T01:25:30.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.162+0000 7f41ec800640 1 -- 172.21.15.90:0/47965737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41e40ad850 con 0x7f41e41603e0 2024-03-24T01:25:30.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.163+0000 7f41ec800640 1 -- 172.21.15.90:0/47965737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41e4143550 con 0x7f41e41603e0 2024-03-24T01:25:30.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.164+0000 7f41da000640 1 -- 172.21.15.90:0/47965737 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f41e001b070 con 0x7f41e41603e0 2024-03-24T01:25:30.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.165+0000 7f41da000640 1 --2- 172.21.15.90:0/47965737 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f41c806bbd0 0x7f41c806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:30.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.165+0000 7f41da000640 1 -- 172.21.15.90:0/47965737 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f41e008a080 con 0x7f41e41603e0 2024-03-24T01:25:30.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.166+0000 7f41e9e00640 1 --2- 172.21.15.90:0/47965737 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f41c806bbd0 0x7f41c806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:30.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.167+0000 7f41e9e00640 1 --2- 172.21.15.90:0/47965737 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f41c806bbd0 0x7f41c806e090 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f41dc036000 tx=0x7f41dc030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:30.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.170+0000 7f41da000640 1 -- 172.21.15.90:0/47965737 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f41e4143550 con 0x7f41e41603e0 2024-03-24T01:25:30.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.372+0000 7f41ec800640 1 -- 172.21.15.90:0/47965737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41e4068060 con 0x7f41e41603e0 2024-03-24T01:25:30.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.374+0000 7f41da000640 1 -- 172.21.15.90:0/47965737 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18925 (secure 0 0 0) 0x7f41e00587c0 con 0x7f41e41603e0 2024-03-24T01:25:30.375 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:25:30.375 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:24:53.236401+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:24:08.643966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T01:24:09.189904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T01:24:13.027323+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:25:30.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.377+0000 7f41ec800640 1 -- 172.21.15.90:0/47965737 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f41c806bbd0 msgr2=0x7f41c806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:30.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.377+0000 7f41ec800640 1 --2- 172.21.15.90:0/47965737 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f41c806bbd0 0x7f41c806e090 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f41dc036000 tx=0x7f41dc030000 comp rx=0 tx=0).stop 2024-03-24T01:25:30.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.377+0000 7f41ec800640 1 -- 172.21.15.90:0/47965737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f41e41603e0 msgr2=0x7f41e415ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:30.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.377+0000 7f41ec800640 1 --2- 172.21.15.90:0/47965737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f41e41603e0 0x7f41e415ae00 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f41e000c450 tx=0x7f41e000c910 comp rx=0 tx=0).stop 2024-03-24T01:25:30.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.377+0000 7f41ec800640 1 -- 172.21.15.90:0/47965737 shutdown_connections 2024-03-24T01:25:30.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.377+0000 7f41ec800640 1 --2- 172.21.15.90:0/47965737 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f41c806bbd0 0x7f41c806e090 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:30.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.377+0000 7f41ec800640 1 --2- 172.21.15.90:0/47965737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f41e41603e0 0x7f41e415ae00 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:30.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.377+0000 7f41ec800640 1 --2- 172.21.15.90:0/47965737 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41e4145430 0x7f41e406f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:30.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.377+0000 7f41ec800640 1 --2- 172.21.15.90:0/47965737 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41e40cc150 0x7f41e4139cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:30.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.377+0000 7f41ec800640 1 -- 172.21.15.90:0/47965737 >> 172.21.15.90:0/47965737 conn(0x7f41e40b1770 msgr2=0x7f41e41442c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:30.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.377+0000 7f41ec800640 1 -- 172.21.15.90:0/47965737 shutdown_connections 2024-03-24T01:25:30.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.378+0000 7f41ec800640 1 -- 172.21.15.90:0/47965737 wait complete. 2024-03-24T01:25:30.397 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T01:24:08.643966+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T01:24:09.189904+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T01:24:13.027323+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T01:25:30.397 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:25:30.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.611+0000 7f1c1b200640 1 Processor -- start 2024-03-24T01:25:30.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.612+0000 7f1c1b200640 1 -- start start 2024-03-24T01:25:30.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.613+0000 7f1c1b200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1c1c06cdd0 0x7f1c1c0b6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:30.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.613+0000 7f1c1b200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c1c0b9460 con 0x7f1c1c0b2720 2024-03-24T01:25:30.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.613+0000 7f1c1b200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c1c0b9740 con 0x7f1c1c15dd90 2024-03-24T01:25:30.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.613+0000 7f1c1b200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c1c0b9a20 con 0x7f1c1c06cdd0 2024-03-24T01:25:30.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.613+0000 7f1c19400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1c1c06cdd0 0x7f1c1c0b6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:30.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.613+0000 7f1c1a800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f1c1c0b2720 0x7f1c1c0d2410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60988/0 (socket says 172.21.15.90:60988) 2024-03-24T01:25:30.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.613+0000 7f1c1a800640 1 -- 172.21.15.90:0/2729516110 learned_addr learned my addr 172.21.15.90:0/2729516110 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:30.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.614+0000 7f1c19400640 1 -- 172.21.15.90:0/2729516110 >> v1:172.21.15.120:6789/0 conn(0x7f1c1c15dd90 legacy=0x7f1c1c06c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:30.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.614+0000 7f1c19400640 1 -- 172.21.15.90:0/2729516110 >> v1:172.21.15.90:6789/0 conn(0x7f1c1c0b2720 legacy=0x7f1c1c0d2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:30.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.614+0000 7f1c19400640 1 -- 172.21.15.90:0/2729516110 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c1c0d1c10 con 0x7f1c1c06cdd0 2024-03-24T01:25:30.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.614+0000 7f1c18a00640 1 -- 172.21.15.90:0/2729516110 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2405414529 0 0) 0x7f1c1c0b9460 con 0x7f1c1c0b2720 2024-03-24T01:25:30.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.615+0000 7f1c19400640 1 --2- 172.21.15.90:0/2729516110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c1c06cdd0 0x7f1c1c0b6af0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f1c04009d90 tx=0x7f1c0402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37ff22553e5269f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:30.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.615+0000 7f1c18a00640 1 -- 172.21.15.90:0/2729516110 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c04039070 con 0x7f1c1c06cdd0 2024-03-24T01:25:30.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.615+0000 7f1c18a00640 1 -- 172.21.15.90:0/2729516110 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c0402cc60 con 0x7f1c1c06cdd0 2024-03-24T01:25:30.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.615+0000 7f1c18a00640 1 -- 172.21.15.90:0/2729516110 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c04034660 con 0x7f1c1c06cdd0 2024-03-24T01:25:30.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.616+0000 7f1c1b200640 1 -- 172.21.15.90:0/2729516110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c1c06cdd0 msgr2=0x7f1c1c0b6af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:30.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.616+0000 7f1c1b200640 1 --2- 172.21.15.90:0/2729516110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c1c06cdd0 0x7f1c1c0b6af0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f1c04009d90 tx=0x7f1c0402beb0 comp rx=0 tx=0).stop 2024-03-24T01:25:30.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.616+0000 7f1c1b200640 1 -- 172.21.15.90:0/2729516110 shutdown_connections 2024-03-24T01:25:30.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.616+0000 7f1c1b200640 1 --2- 172.21.15.90:0/2729516110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c1c06cdd0 0x7f1c1c0b6af0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:30.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.616+0000 7f1c1b200640 1 -- 172.21.15.90:0/2729516110 >> 172.21.15.90:0/2729516110 conn(0x7f1c1c072e20 msgr2=0x7f1c1c0d3a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:30.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.616+0000 7f1c1b200640 1 -- 172.21.15.90:0/2729516110 shutdown_connections 2024-03-24T01:25:30.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.616+0000 7f1c1b200640 1 -- 172.21.15.90:0/2729516110 wait complete. 2024-03-24T01:25:30.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.618+0000 7f1c1b200640 1 Processor -- start 2024-03-24T01:25:30.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.619+0000 7f1c1b200640 1 -- start start 2024-03-24T01:25:30.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.620+0000 7f1c1b200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c1c06cdd0 0x7f1c1c14ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:30.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.620+0000 7f1c1b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c1c0b2720 0x7f1c1c0cb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:30.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.620+0000 7f1c1b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c1c15dd90 0x7f1c1c0ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:30.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.620+0000 7f1c1b200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1c0d3570 con 0x7f1c1c0b2720 2024-03-24T01:25:30.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.620+0000 7f1c1b200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1c0d33f0 con 0x7f1c1c06cdd0 2024-03-24T01:25:30.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.620+0000 7f1c1b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1c0d36f0 con 0x7f1c1c15dd90 2024-03-24T01:25:30.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.620+0000 7f1c19400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c1c0b2720 0x7f1c1c0cb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:30.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.620+0000 7f1c19400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c1c0b2720 0x7f1c1c0cb4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54988/0 (socket says 172.21.15.90:54988) 2024-03-24T01:25:30.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.620+0000 7f1c19400640 1 -- 172.21.15.90:0/905003760 learned_addr learned my addr 172.21.15.90:0/905003760 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:30.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.621+0000 7f1c19e00640 1 --2- 172.21.15.90:0/905003760 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c1c06cdd0 0x7f1c1c14ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:30.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.621+0000 7f1c1a800640 1 --2- 172.21.15.90:0/905003760 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c1c15dd90 0x7f1c1c0ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:30.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.621+0000 7f1c19400640 1 -- 172.21.15.90:0/905003760 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c1c15dd90 msgr2=0x7f1c1c0ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:30.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.621+0000 7f1c19400640 1 --2- 172.21.15.90:0/905003760 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c1c15dd90 0x7f1c1c0ccdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:30.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.621+0000 7f1c19400640 1 -- 172.21.15.90:0/905003760 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c1c06cdd0 msgr2=0x7f1c1c14ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:30.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.621+0000 7f1c19400640 1 --2- 172.21.15.90:0/905003760 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c1c06cdd0 0x7f1c1c14ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:30.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.621+0000 7f1c19400640 1 -- 172.21.15.90:0/905003760 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c1c0cbbb0 con 0x7f1c1c0b2720 2024-03-24T01:25:30.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.621+0000 7f1c19400640 1 --2- 172.21.15.90:0/905003760 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c1c0b2720 0x7f1c1c0cb4d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f1c04009d90 tx=0x7f1c04004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:30.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.621+0000 7f1c12000640 1 -- 172.21.15.90:0/905003760 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c04044020 con 0x7f1c1c0b2720 2024-03-24T01:25:30.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.622+0000 7f1c12000640 1 -- 172.21.15.90:0/905003760 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c0403f670 con 0x7f1c1c0b2720 2024-03-24T01:25:30.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.622+0000 7f1c12000640 1 -- 172.21.15.90:0/905003760 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c04039040 con 0x7f1c1c0b2720 2024-03-24T01:25:30.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.622+0000 7f1c1b200640 1 -- 172.21.15.90:0/905003760 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c1c0b3510 con 0x7f1c1c0b2720 2024-03-24T01:25:30.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.622+0000 7f1c1b200640 1 -- 172.21.15.90:0/905003760 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c1c0b3ad0 con 0x7f1c1c0b2720 2024-03-24T01:25:30.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.623+0000 7f1c1b200640 1 -- 172.21.15.90:0/905003760 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c1c0b3cd0 con 0x7f1c1c0b2720 2024-03-24T01:25:30.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.624+0000 7f1c12000640 1 -- 172.21.15.90:0/905003760 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1c04047020 con 0x7f1c1c0b2720 2024-03-24T01:25:30.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.625+0000 7f1c12000640 1 --2- 172.21.15.90:0/905003760 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1bf806bbd0 0x7f1bf806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:30.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.625+0000 7f1c12000640 1 -- 172.21.15.90:0/905003760 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f1c040adb30 con 0x7f1c1c0b2720 2024-03-24T01:25:30.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.625+0000 7f1c19e00640 1 --2- 172.21.15.90:0/905003760 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1bf806bbd0 0x7f1bf806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:30.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.626+0000 7f1c19e00640 1 --2- 172.21.15.90:0/905003760 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1bf806bbd0 0x7f1bf806e090 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c006fd0 tx=0x7f1c0c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:30.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.629+0000 7f1c12000640 1 -- 172.21.15.90:0/905003760 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1c1c0b3cd0 con 0x7f1c1c0b2720 2024-03-24T01:25:30.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.833+0000 7f1c1b200640 1 -- 172.21.15.90:0/905003760 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1c1c04f400 con 0x7f1c1c0b2720 2024-03-24T01:25:30.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.834+0000 7f1c12000640 1 -- 172.21.15.90:0/905003760 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1c0404e030 con 0x7f1c1c0b2720 2024-03-24T01:25:30.834 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:25:30.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.837+0000 7f1c1b200640 1 -- 172.21.15.90:0/905003760 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1bf806bbd0 msgr2=0x7f1bf806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:30.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.837+0000 7f1c1b200640 1 --2- 172.21.15.90:0/905003760 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1bf806bbd0 0x7f1bf806e090 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c006fd0 tx=0x7f1c0c008040 comp rx=0 tx=0).stop 2024-03-24T01:25:30.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.837+0000 7f1c1b200640 1 -- 172.21.15.90:0/905003760 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c1c0b2720 msgr2=0x7f1c1c0cb4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:30.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.837+0000 7f1c1b200640 1 --2- 172.21.15.90:0/905003760 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c1c0b2720 0x7f1c1c0cb4d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f1c04009d90 tx=0x7f1c04004a40 comp rx=0 tx=0).stop 2024-03-24T01:25:30.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.837+0000 7f1c1b200640 1 -- 172.21.15.90:0/905003760 shutdown_connections 2024-03-24T01:25:30.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.837+0000 7f1c1b200640 1 --2- 172.21.15.90:0/905003760 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1bf806bbd0 0x7f1bf806e090 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:30.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.837+0000 7f1c1b200640 1 --2- 172.21.15.90:0/905003760 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c1c0b2720 0x7f1c1c0cb4d0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:30.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.837+0000 7f1c1b200640 1 --2- 172.21.15.90:0/905003760 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c1c15dd90 0x7f1c1c0ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:30.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.837+0000 7f1c1b200640 1 --2- 172.21.15.90:0/905003760 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c1c06cdd0 0x7f1c1c14ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:30.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.837+0000 7f1c1b200640 1 -- 172.21.15.90:0/905003760 >> 172.21.15.90:0/905003760 conn(0x7f1c1c072e20 msgr2=0x7f1c1c160210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:30.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.837+0000 7f1c1b200640 1 -- 172.21.15.90:0/905003760 shutdown_connections 2024-03-24T01:25:30.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:30.837+0000 7f1c1b200640 1 -- 172.21.15.90:0/905003760 wait complete. 2024-03-24T01:25:30.855 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T01:25:31.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.064+0000 7f86e5c00640 1 Processor -- start 2024-03-24T01:25:31.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.064+0000 7f86e5c00640 1 -- start start 2024-03-24T01:25:31.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.066+0000 7f86e5c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f86e00cf150 0x7f86e00ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:31.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.066+0000 7f86e5c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86e006bca0 con 0x7f86e0159620 2024-03-24T01:25:31.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.066+0000 7f86e5c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86e00b36f0 con 0x7f86e006b7d0 2024-03-24T01:25:31.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.066+0000 7f86e5c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86e00b4470 con 0x7f86e00cf150 2024-03-24T01:25:31.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.066+0000 7f86dea00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f86e0159620 0x7f86e006b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:32772/0 (socket says 172.21.15.90:32772) 2024-03-24T01:25:31.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.066+0000 7f86dea00640 1 -- 172.21.15.90:0/1281534967 learned_addr learned my addr 172.21.15.90:0/1281534967 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.066+0000 7f86dd600640 1 -- 172.21.15.90:0/1281534967 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 582908127 0 0) 0x7f86e006bca0 con 0x7f86e0159620 2024-03-24T01:25:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.067+0000 7f86dd600640 1 -- 172.21.15.90:0/1281534967 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86c4003520 con 0x7f86e0159620 2024-03-24T01:25:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.067+0000 7f86df400640 1 --2- 172.21.15.90:0/1281534967 >> v2:172.21.15.120:3300/0 conn(0x7f86e00cf150 0x7f86e00ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.067+0000 7f86dd600640 1 -- 172.21.15.90:0/1281534967 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2100390457 0 0) 0x7f86c4003520 con 0x7f86e0159620 2024-03-24T01:25:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.067+0000 7f86dd600640 1 -- 172.21.15.90:0/1281534967 >> v2:172.21.15.120:3300/0 conn(0x7f86e00cf150 msgr2=0x7f86e00ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.067+0000 7f86dd600640 1 --2- 172.21.15.90:0/1281534967 >> v2:172.21.15.120:3300/0 conn(0x7f86e00cf150 0x7f86e00ae530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.067+0000 7f86dd600640 1 -- 172.21.15.90:0/1281534967 >> v1:172.21.15.120:6789/0 conn(0x7f86e006b7d0 legacy=0x7f86e00cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.067+0000 7f86dd600640 1 -- 172.21.15.90:0/1281534967 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e00b3990 con 0x7f86e0159620 2024-03-24T01:25:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.067+0000 7f86dd600640 1 -- 172.21.15.90:0/1281534967 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f86d0001670 con 0x7f86e0159620 2024-03-24T01:25:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.067+0000 7f86dd600640 1 -- 172.21.15.90:0/1281534967 >> v1:172.21.15.90:6789/0 conn(0x7f86e0159620 legacy=0x7f86e006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.067+0000 7f86dd600640 1 --2- 172.21.15.90:0/1281534967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86c40039b0 0x7f86c4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.067+0000 7f86dd600640 1 --2- 172.21.15.90:0/1281534967 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86c4004800 0x7f86c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.067+0000 7f86dd600640 1 --2- 172.21.15.90:0/1281534967 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86c40052e0 0x7f86c400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.067+0000 7f86dd600640 1 -- 172.21.15.90:0/1281534967 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e00b4470 con 0x7f86c4004800 2024-03-24T01:25:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.067+0000 7f86dd600640 1 -- 172.21.15.90:0/1281534967 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c400c1b0 con 0x7f86c40039b0 2024-03-24T01:25:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.067+0000 7f86dd600640 1 -- 172.21.15.90:0/1281534967 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86c400c3b0 con 0x7f86c40052e0 2024-03-24T01:25:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.067+0000 7f86de000640 1 --2- 172.21.15.90:0/1281534967 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86c4004800 0x7f86c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.068+0000 7f86dea00640 1 --2- 172.21.15.90:0/1281534967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86c40039b0 0x7f86c4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.068+0000 7f86df400640 1 --2- 172.21.15.90:0/1281534967 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86c40052e0 0x7f86c400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.068+0000 7f86de000640 1 -- 172.21.15.90:0/1281534967 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86c40052e0 msgr2=0x7f86c400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.068+0000 7f86de000640 1 --2- 172.21.15.90:0/1281534967 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86c40052e0 0x7f86c400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.068+0000 7f86de000640 1 -- 172.21.15.90:0/1281534967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86c40039b0 msgr2=0x7f86c4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.068+0000 7f86de000640 1 --2- 172.21.15.90:0/1281534967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86c40039b0 0x7f86c4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.068+0000 7f86de000640 1 -- 172.21.15.90:0/1281534967 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c4003520 con 0x7f86c4004800 2024-03-24T01:25:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.068+0000 7f86df400640 1 -- 172.21.15.90:0/1281534967 reap_dead start 2024-03-24T01:25:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.069+0000 7f86de000640 1 --2- 172.21.15.90:0/1281534967 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86c4004800 0x7f86c4004c80 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f86e00b4f00 tx=0x7f86c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbe5ac06576ee1e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.069+0000 7f86dd600640 1 -- 172.21.15.90:0/1281534967 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86cc001370 con 0x7f86c4004800 2024-03-24T01:25:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.069+0000 7f86dd600640 1 -- 172.21.15.90:0/1281534967 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86cc0052d0 con 0x7f86c4004800 2024-03-24T01:25:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.069+0000 7f86dd600640 1 -- 172.21.15.90:0/1281534967 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86cc003830 con 0x7f86c4004800 2024-03-24T01:25:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.070+0000 7f86e5c00640 1 -- 172.21.15.90:0/1281534967 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86c4004800 msgr2=0x7f86c4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.070+0000 7f86e5c00640 1 --2- 172.21.15.90:0/1281534967 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86c4004800 0x7f86c4004c80 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f86e00b4f00 tx=0x7f86c4005830 comp rx=0 tx=0).stop 2024-03-24T01:25:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.070+0000 7f86e5c00640 1 -- 172.21.15.90:0/1281534967 shutdown_connections 2024-03-24T01:25:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.070+0000 7f86e5c00640 1 --2- 172.21.15.90:0/1281534967 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86c4004800 0x7f86c4004c80 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.070+0000 7f86e5c00640 1 -- 172.21.15.90:0/1281534967 >> 172.21.15.90:0/1281534967 conn(0x7f86e00b1770 msgr2=0x7f86e00b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.070+0000 7f86e5c00640 1 -- 172.21.15.90:0/1281534967 shutdown_connections 2024-03-24T01:25:31.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.070+0000 7f86e5c00640 1 -- 172.21.15.90:0/1281534967 wait complete. 2024-03-24T01:25:31.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.072+0000 7f86e5c00640 1 Processor -- start 2024-03-24T01:25:31.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.072+0000 7f86e5c00640 1 -- start start 2024-03-24T01:25:31.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.073+0000 7f86e5c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86c4004800 0x7f86e014e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:31.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.073+0000 7f86e5c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86e013bfa0 0x7f86e01388b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:31.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.073+0000 7f86e5c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86e0138df0 0x7f86e013b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:31.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.073+0000 7f86e5c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e0159a00 con 0x7f86e0138df0 2024-03-24T01:25:31.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.073+0000 7f86e5c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e00b3f50 con 0x7f86e013bfa0 2024-03-24T01:25:31.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.073+0000 7f86e5c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86e015bfe0 con 0x7f86c4004800 2024-03-24T01:25:31.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.073+0000 7f86df400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86e0138df0 0x7f86e013b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:31.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.073+0000 7f86df400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86e0138df0 0x7f86e013b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55002/0 (socket says 172.21.15.90:55002) 2024-03-24T01:25:31.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.073+0000 7f86df400640 1 -- 172.21.15.90:0/2433154018 learned_addr learned my addr 172.21.15.90:0/2433154018 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:31.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.073+0000 7f86de000640 1 --2- 172.21.15.90:0/2433154018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86e013bfa0 0x7f86e01388b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:31.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.073+0000 7f86dea00640 1 --2- 172.21.15.90:0/2433154018 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86c4004800 0x7f86e014e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:31.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.074+0000 7f86df400640 1 -- 172.21.15.90:0/2433154018 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86c4004800 msgr2=0x7f86e014e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:31.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.074+0000 7f86df400640 1 --2- 172.21.15.90:0/2433154018 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86c4004800 0x7f86e014e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:31.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.074+0000 7f86df400640 1 -- 172.21.15.90:0/2433154018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86e013bfa0 msgr2=0x7f86e01388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:31.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.074+0000 7f86df400640 1 --2- 172.21.15.90:0/2433154018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86e013bfa0 0x7f86e01388b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:31.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.074+0000 7f86df400640 1 -- 172.21.15.90:0/2433154018 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e0150cc0 con 0x7f86e0138df0 2024-03-24T01:25:31.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.074+0000 7f86df400640 1 --2- 172.21.15.90:0/2433154018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86e0138df0 0x7f86e013b390 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f86e00ce580 tx=0x7f86d8009d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:31.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.074+0000 7f86d6a00640 1 -- 172.21.15.90:0/2433154018 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d8019070 con 0x7f86e0138df0 2024-03-24T01:25:31.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.074+0000 7f86d6a00640 1 -- 172.21.15.90:0/2433154018 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86d8007d10 con 0x7f86e0138df0 2024-03-24T01:25:31.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.074+0000 7f86d6a00640 1 -- 172.21.15.90:0/2433154018 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d8014640 con 0x7f86e0138df0 2024-03-24T01:25:31.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.074+0000 7f86e5c00640 1 -- 172.21.15.90:0/2433154018 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86e0150fa0 con 0x7f86e0138df0 2024-03-24T01:25:31.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.074+0000 7f86e5c00640 1 -- 172.21.15.90:0/2433154018 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86e0151560 con 0x7f86e0138df0 2024-03-24T01:25:31.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.075+0000 7f86e5c00640 1 -- 172.21.15.90:0/2433154018 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86e00cdd40 con 0x7f86e0138df0 2024-03-24T01:25:31.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.076+0000 7f86d6a00640 1 -- 172.21.15.90:0/2433154018 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f86d801c070 con 0x7f86e0138df0 2024-03-24T01:25:31.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.078+0000 7f86d6a00640 1 --2- 172.21.15.90:0/2433154018 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f86c006bbd0 0x7f86c006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:31.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.078+0000 7f86d6a00640 1 -- 172.21.15.90:0/2433154018 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f86d808aa20 con 0x7f86e0138df0 2024-03-24T01:25:31.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.078+0000 7f86dea00640 1 --2- 172.21.15.90:0/2433154018 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f86c006bbd0 0x7f86c006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:31.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.079+0000 7f86dea00640 1 --2- 172.21.15.90:0/2433154018 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f86c006bbd0 0x7f86c006e090 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f86d0001eb0 tx=0x7f86d0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:31.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.083+0000 7f86d6a00640 1 -- 172.21.15.90:0/2433154018 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f86e00cdd40 con 0x7f86e0138df0 2024-03-24T01:25:31.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.279+0000 7f86e5c00640 1 -- 172.21.15.90:0/2433154018 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f86e004f400 con 0x7f86e0138df0 2024-03-24T01:25:31.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.280+0000 7f86d6a00640 1 -- 172.21.15.90:0/2433154018 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f86d80590f0 con 0x7f86e0138df0 2024-03-24T01:25:31.280 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:25:31.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.282+0000 7f86e5c00640 1 -- 172.21.15.90:0/2433154018 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f86c006bbd0 msgr2=0x7f86c006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:31.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.282+0000 7f86e5c00640 1 --2- 172.21.15.90:0/2433154018 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f86c006bbd0 0x7f86c006e090 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f86d0001eb0 tx=0x7f86d0009040 comp rx=0 tx=0).stop 2024-03-24T01:25:31.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.282+0000 7f86e5c00640 1 -- 172.21.15.90:0/2433154018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86e0138df0 msgr2=0x7f86e013b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:31.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.282+0000 7f86e5c00640 1 --2- 172.21.15.90:0/2433154018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86e0138df0 0x7f86e013b390 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f86e00ce580 tx=0x7f86d8009d40 comp rx=0 tx=0).stop 2024-03-24T01:25:31.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.282+0000 7f86e5c00640 1 -- 172.21.15.90:0/2433154018 shutdown_connections 2024-03-24T01:25:31.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.283+0000 7f86e5c00640 1 --2- 172.21.15.90:0/2433154018 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f86c006bbd0 0x7f86c006e090 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:31.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.283+0000 7f86e5c00640 1 --2- 172.21.15.90:0/2433154018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86e0138df0 0x7f86e013b390 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:31.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.283+0000 7f86e5c00640 1 --2- 172.21.15.90:0/2433154018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86e013bfa0 0x7f86e01388b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:31.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.283+0000 7f86e5c00640 1 --2- 172.21.15.90:0/2433154018 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f86c4004800 0x7f86e014e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:31.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.283+0000 7f86e5c00640 1 -- 172.21.15.90:0/2433154018 >> 172.21.15.90:0/2433154018 conn(0x7f86e00b1770 msgr2=0x7f86e00cf6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:31.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.283+0000 7f86e5c00640 1 -- 172.21.15.90:0/2433154018 shutdown_connections 2024-03-24T01:25:31.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.283+0000 7f86e5c00640 1 -- 172.21.15.90:0/2433154018 wait complete. 2024-03-24T01:25:31.303 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T01:25:31.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.514+0000 7f73b6600640 1 Processor -- start 2024-03-24T01:25:31.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.514+0000 7f73b6600640 1 -- start start 2024-03-24T01:25:31.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.515+0000 7f73b6600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f73b00d0b60 0x7f73b00d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:31.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.515+0000 7f73b6600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f73b015dfe0 0x7f73b01622c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:31.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.515+0000 7f73b6600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73b00b2e50 0x7f73b00d3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:31.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.515+0000 7f73b6600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73b00d20f0 con 0x7f73b00d0b60 2024-03-24T01:25:31.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.515+0000 7f73b6600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73b00d1480 con 0x7f73b015dfe0 2024-03-24T01:25:31.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.515+0000 7f73b6600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b00ced80 con 0x7f73b00b2e50 2024-03-24T01:25:31.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.516+0000 7f73af400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f73b00d0b60 0x7f73b00d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:31.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.516+0000 7f73af400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f73b00d0b60 0x7f73b00d0f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55018/0 (socket says 172.21.15.90:55018) 2024-03-24T01:25:31.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.516+0000 7f73af400640 1 -- 172.21.15.90:0/2512682149 learned_addr learned my addr 172.21.15.90:0/2512682149 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:31.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.516+0000 7f73af400640 1 -- 172.21.15.90:0/2512682149 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73b00b2e50 msgr2=0x7f73b00d3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:31.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.516+0000 7f73af400640 1 --2- 172.21.15.90:0/2512682149 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73b00b2e50 0x7f73b00d3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:31.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.516+0000 7f73af400640 1 -- 172.21.15.90:0/2512682149 >> v2:172.21.15.120:3300/0 conn(0x7f73b015dfe0 msgr2=0x7f73b01622c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:31.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.516+0000 7f73af400640 1 --2- 172.21.15.90:0/2512682149 >> v2:172.21.15.120:3300/0 conn(0x7f73b015dfe0 0x7f73b01622c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:31.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.516+0000 7f73af400640 1 -- 172.21.15.90:0/2512682149 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b00cf060 con 0x7f73b00d0b60 2024-03-24T01:25:31.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.517+0000 7f73af400640 1 --2- 172.21.15.90:0/2512682149 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73b00d0b60 0x7f73b00d0f40 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f73b00b60f0 tx=0x7f73a002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92cc93e09ed10284 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:31.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.517+0000 7f73aea00640 1 -- 172.21.15.90:0/2512682149 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a0038070 con 0x7f73b00d0b60 2024-03-24T01:25:31.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.518+0000 7f73aea00640 1 -- 172.21.15.90:0/2512682149 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a0031740 con 0x7f73b00d0b60 2024-03-24T01:25:31.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.518+0000 7f73aea00640 1 -- 172.21.15.90:0/2512682149 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a002f050 con 0x7f73b00d0b60 2024-03-24T01:25:31.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.518+0000 7f73b6600640 1 -- 172.21.15.90:0/2512682149 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73b00d0b60 msgr2=0x7f73b00d0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:31.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.518+0000 7f73b6600640 1 --2- 172.21.15.90:0/2512682149 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73b00d0b60 0x7f73b00d0f40 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f73b00b60f0 tx=0x7f73a002b7e0 comp rx=0 tx=0).stop 2024-03-24T01:25:31.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.518+0000 7f73b6600640 1 -- 172.21.15.90:0/2512682149 shutdown_connections 2024-03-24T01:25:31.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.518+0000 7f73b6600640 1 --2- 172.21.15.90:0/2512682149 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73b00b2e50 0x7f73b00d3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:31.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.518+0000 7f73b6600640 1 --2- 172.21.15.90:0/2512682149 >> v2:172.21.15.120:3300/0 conn(0x7f73b015dfe0 0x7f73b01622c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:31.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.518+0000 7f73b6600640 1 --2- 172.21.15.90:0/2512682149 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73b00d0b60 0x7f73b00d0f40 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:31.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.518+0000 7f73b6600640 1 -- 172.21.15.90:0/2512682149 >> 172.21.15.90:0/2512682149 conn(0x7f73b0072c10 msgr2=0x7f73b0073020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:31.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.518+0000 7f73b6600640 1 -- 172.21.15.90:0/2512682149 shutdown_connections 2024-03-24T01:25:31.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.518+0000 7f73b6600640 1 -- 172.21.15.90:0/2512682149 wait complete. 2024-03-24T01:25:31.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.520+0000 7f73b6600640 1 Processor -- start 2024-03-24T01:25:31.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.520+0000 7f73b6600640 1 -- start start 2024-03-24T01:25:31.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.521+0000 7f73b6600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73b00b2e50 0x7f73b00cd710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:31.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.521+0000 7f73b6600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73b00d0b60 0x7f73b015d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:31.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.521+0000 7f73b6600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73b015dfe0 0x7f73b00cca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:31.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.521+0000 7f73b6600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b00cf340 con 0x7f73b00d0b60 2024-03-24T01:25:31.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.521+0000 7f73b6600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b0162800 con 0x7f73b015dfe0 2024-03-24T01:25:31.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.521+0000 7f73b6600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b00b6d90 con 0x7f73b00b2e50 2024-03-24T01:25:31.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.521+0000 7f73a7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73b00d0b60 0x7f73b015d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:31.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.521+0000 7f73a7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73b00d0b60 0x7f73b015d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55030/0 (socket says 172.21.15.90:55030) 2024-03-24T01:25:31.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.521+0000 7f73a7e00640 1 -- 172.21.15.90:0/2556319391 learned_addr learned my addr 172.21.15.90:0/2556319391 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:31.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.522+0000 7f73af400640 1 --2- 172.21.15.90:0/2556319391 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73b00b2e50 0x7f73b00cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:31.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.522+0000 7f73a7e00640 1 -- 172.21.15.90:0/2556319391 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73b00b2e50 msgr2=0x7f73b00cd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:31.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.522+0000 7f73a7e00640 1 --2- 172.21.15.90:0/2556319391 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73b00b2e50 0x7f73b00cd710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:31.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.522+0000 7f73afe00640 1 --2- 172.21.15.90:0/2556319391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73b015dfe0 0x7f73b00cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:31.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.522+0000 7f73a7e00640 1 -- 172.21.15.90:0/2556319391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73b015dfe0 msgr2=0x7f73b00cca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:31.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.522+0000 7f73a7e00640 1 --2- 172.21.15.90:0/2556319391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73b015dfe0 0x7f73b00cca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:31.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.522+0000 7f73a7e00640 1 -- 172.21.15.90:0/2556319391 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b01e84d0 con 0x7f73b00d0b60 2024-03-24T01:25:31.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.522+0000 7f73a7e00640 1 --2- 172.21.15.90:0/2556319391 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73b00d0b60 0x7f73b015d0b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f739800b4a0 tx=0x7f739800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:31.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.522+0000 7f73a7400640 1 -- 172.21.15.90:0/2556319391 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7398014070 con 0x7f73b00d0b60 2024-03-24T01:25:31.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.522+0000 7f73a7400640 1 -- 172.21.15.90:0/2556319391 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7398010610 con 0x7f73b00d0b60 2024-03-24T01:25:31.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.522+0000 7f73b6600640 1 -- 172.21.15.90:0/2556319391 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73b00cbfb0 con 0x7f73b00d0b60 2024-03-24T01:25:31.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.522+0000 7f73a7400640 1 -- 172.21.15.90:0/2556319391 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7398010a00 con 0x7f73b00d0b60 2024-03-24T01:25:31.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.522+0000 7f73b6600640 1 -- 172.21.15.90:0/2556319391 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73b00cc570 con 0x7f73b00d0b60 2024-03-24T01:25:31.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.524+0000 7f73b6600640 1 -- 172.21.15.90:0/2556319391 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73b00cc770 con 0x7f73b00d0b60 2024-03-24T01:25:31.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.524+0000 7f73a7400640 1 -- 172.21.15.90:0/2556319391 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7398017070 con 0x7f73b00d0b60 2024-03-24T01:25:31.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.525+0000 7f73a7400640 1 --2- 172.21.15.90:0/2556319391 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f738c06bbd0 0x7f738c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:31.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.525+0000 7f73a7400640 1 -- 172.21.15.90:0/2556319391 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f739808a980 con 0x7f73b00d0b60 2024-03-24T01:25:31.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.526+0000 7f73af400640 1 --2- 172.21.15.90:0/2556319391 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f738c06bbd0 0x7f738c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:31.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.527+0000 7f73af400640 1 --2- 172.21.15.90:0/2556319391 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f738c06bbd0 0x7f738c06e090 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f73a0035000 tx=0x7f73a0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:31.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.530+0000 7f73a7400640 1 -- 172.21.15.90:0/2556319391 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f73b00cc770 con 0x7f73b00d0b60 2024-03-24T01:25:31.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.724+0000 7f73b6600640 1 -- 172.21.15.90:0/2556319391 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f73b01e8220 con 0x7f73b00d0b60 2024-03-24T01:25:31.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.725+0000 7f73a7400640 1 -- 172.21.15.90:0/2556319391 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7398058fb0 con 0x7f73b00d0b60 2024-03-24T01:25:31.725 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-24T01:25:31.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.727+0000 7f73b6600640 1 -- 172.21.15.90:0/2556319391 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f738c06bbd0 msgr2=0x7f738c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:31.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.727+0000 7f73b6600640 1 --2- 172.21.15.90:0/2556319391 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f738c06bbd0 0x7f738c06e090 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f73a0035000 tx=0x7f73a0008030 comp rx=0 tx=0).stop 2024-03-24T01:25:31.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.727+0000 7f73b6600640 1 -- 172.21.15.90:0/2556319391 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73b00d0b60 msgr2=0x7f73b015d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:31.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.727+0000 7f73b6600640 1 --2- 172.21.15.90:0/2556319391 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73b00d0b60 0x7f73b015d0b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f739800b4a0 tx=0x7f739800b960 comp rx=0 tx=0).stop 2024-03-24T01:25:31.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.727+0000 7f73b6600640 1 -- 172.21.15.90:0/2556319391 shutdown_connections 2024-03-24T01:25:31.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.727+0000 7f73b6600640 1 --2- 172.21.15.90:0/2556319391 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f738c06bbd0 0x7f738c06e090 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:31.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.727+0000 7f73b6600640 1 --2- 172.21.15.90:0/2556319391 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73b00d0b60 0x7f73b015d0b0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:31.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.727+0000 7f73b6600640 1 --2- 172.21.15.90:0/2556319391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73b015dfe0 0x7f73b00cca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:31.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.727+0000 7f73b6600640 1 --2- 172.21.15.90:0/2556319391 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73b00b2e50 0x7f73b00cd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:31.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.727+0000 7f73b6600640 1 -- 172.21.15.90:0/2556319391 >> 172.21.15.90:0/2556319391 conn(0x7f73b0072c10 msgr2=0x7f73b00b4770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:31.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.727+0000 7f73b6600640 1 -- 172.21.15.90:0/2556319391 shutdown_connections 2024-03-24T01:25:31.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.727+0000 7f73b6600640 1 -- 172.21.15.90:0/2556319391 wait complete. 2024-03-24T01:25:31.748 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T01:25:31.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.960+0000 7f4c0b000640 1 Processor -- start 2024-03-24T01:25:31.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.961+0000 7f4c0b000640 1 -- start start 2024-03-24T01:25:31.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.962+0000 7f4c0b000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4c0415da30 0x7f4c0415de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:31.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.962+0000 7f4c0b000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4c0406b0c0 0x7f4c0406b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:31.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.962+0000 7f4c0b000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c040af140 con 0x7f4c040acea0 2024-03-24T01:25:31.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.962+0000 7f4c0b000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c040b2c70 con 0x7f4c0415da30 2024-03-24T01:25:31.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.962+0000 7f4c0b000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c0415f160 con 0x7f4c0406b0c0 2024-03-24T01:25:31.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.963+0000 7f4c03400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4c040acea0 0x7f4c040aea30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:32774/0 (socket says 172.21.15.90:32774) 2024-03-24T01:25:31.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.963+0000 7f4c03e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4c0415da30 0x7f4c0415de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:31.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.963+0000 7f4c03400640 1 -- 172.21.15.90:0/2765675871 learned_addr learned my addr 172.21.15.90:0/2765675871 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:31.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.963+0000 7f4c09000640 1 --2- 172.21.15.90:0/2765675871 >> v2:172.21.15.120:3300/0 conn(0x7f4c0406b0c0 0x7f4c0406b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:31.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.963+0000 7f4c02a00640 1 -- 172.21.15.90:0/2765675871 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2283779097 0 0) 0x7f4c040af140 con 0x7f4c040acea0 2024-03-24T01:25:31.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.963+0000 7f4c02a00640 1 -- 172.21.15.90:0/2765675871 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4be8003520 con 0x7f4c040acea0 2024-03-24T01:25:31.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.963+0000 7f4c03e00640 1 -- 172.21.15.90:0/2765675871 >> v2:172.21.15.120:3300/0 conn(0x7f4c0406b0c0 msgr2=0x7f4c0406b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:31.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.963+0000 7f4c03e00640 1 --2- 172.21.15.90:0/2765675871 >> v2:172.21.15.120:3300/0 conn(0x7f4c0406b0c0 0x7f4c0406b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:31.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.963+0000 7f4c03e00640 1 -- 172.21.15.90:0/2765675871 >> v1:172.21.15.90:6789/0 conn(0x7f4c040acea0 legacy=0x7f4c040aea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:31.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.963+0000 7f4c03e00640 1 -- 172.21.15.90:0/2765675871 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c0415e550 con 0x7f4c0415da30 2024-03-24T01:25:31.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.963+0000 7f4c09000640 1 --2- 172.21.15.90:0/2765675871 >> v2:172.21.15.120:3300/0 conn(0x7f4c0406b0c0 0x7f4c0406b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:25:31.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.964+0000 7f4c03e00640 1 --2- 172.21.15.90:0/2765675871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c0415da30 0x7f4c0415de10 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f4c040ac970 tx=0x7f4bf402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5035c460b334519 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:31.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.965+0000 7f4c02a00640 1 -- 172.21.15.90:0/2765675871 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf4039070 con 0x7f4c0415da30 2024-03-24T01:25:31.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.965+0000 7f4c02a00640 1 -- 172.21.15.90:0/2765675871 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bf402cc80 con 0x7f4c0415da30 2024-03-24T01:25:31.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.965+0000 7f4c02a00640 1 -- 172.21.15.90:0/2765675871 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf4034660 con 0x7f4c0415da30 2024-03-24T01:25:31.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.965+0000 7f4c0b000640 1 -- 172.21.15.90:0/2765675871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c0415da30 msgr2=0x7f4c0415de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:31.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.965+0000 7f4c0b000640 1 --2- 172.21.15.90:0/2765675871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c0415da30 0x7f4c0415de10 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f4c040ac970 tx=0x7f4bf402bed0 comp rx=0 tx=0).stop 2024-03-24T01:25:31.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.965+0000 7f4c0b000640 1 -- 172.21.15.90:0/2765675871 shutdown_connections 2024-03-24T01:25:31.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.965+0000 7f4c0b000640 1 --2- 172.21.15.90:0/2765675871 >> v2:172.21.15.120:3300/0 conn(0x7f4c0406b0c0 0x7f4c0406b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:31.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.965+0000 7f4c0b000640 1 --2- 172.21.15.90:0/2765675871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c0415da30 0x7f4c0415de10 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:31.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.965+0000 7f4c0b000640 1 -- 172.21.15.90:0/2765675871 >> 172.21.15.90:0/2765675871 conn(0x7f4c040b1770 msgr2=0x7f4c040b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:31.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.966+0000 7f4c0b000640 1 -- 172.21.15.90:0/2765675871 shutdown_connections 2024-03-24T01:25:31.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.966+0000 7f4c0b000640 1 -- 172.21.15.90:0/2765675871 wait complete. 2024-03-24T01:25:31.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.967+0000 7f4c0b000640 1 Processor -- start 2024-03-24T01:25:31.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.967+0000 7f4c0b000640 1 -- start start 2024-03-24T01:25:31.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.968+0000 7f4c0b000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c0406b0c0 0x7f4c041474a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:31.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.968+0000 7f4c0b000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c040acea0 0x7f4c0406ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:31.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.968+0000 7f4c0b000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c0415da30 0x7f4c0406f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:31.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.968+0000 7f4c0b000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0415eb10 con 0x7f4c0415da30 2024-03-24T01:25:31.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.968+0000 7f4c0b000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0406ba30 con 0x7f4c040acea0 2024-03-24T01:25:31.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.968+0000 7f4c0b000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0415edf0 con 0x7f4c0406b0c0 2024-03-24T01:25:31.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.969+0000 7f4c09000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c0415da30 0x7f4c0406f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:31.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.969+0000 7f4c09000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c0415da30 0x7f4c0406f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55048/0 (socket says 172.21.15.90:55048) 2024-03-24T01:25:31.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.969+0000 7f4c09000640 1 -- 172.21.15.90:0/33306708 learned_addr learned my addr 172.21.15.90:0/33306708 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:31.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.969+0000 7f4c03400640 1 --2- 172.21.15.90:0/33306708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c040acea0 0x7f4c0406ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:31.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.969+0000 7f4c03e00640 1 --2- 172.21.15.90:0/33306708 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c0406b0c0 0x7f4c041474a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:31.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.969+0000 7f4c09000640 1 -- 172.21.15.90:0/33306708 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c0406b0c0 msgr2=0x7f4c041474a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:31.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.969+0000 7f4c09000640 1 --2- 172.21.15.90:0/33306708 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c0406b0c0 0x7f4c041474a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:31.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.969+0000 7f4c09000640 1 -- 172.21.15.90:0/33306708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c040acea0 msgr2=0x7f4c0406ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:31.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.969+0000 7f4c09000640 1 --2- 172.21.15.90:0/33306708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c040acea0 0x7f4c0406ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:31.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.969+0000 7f4c09000640 1 -- 172.21.15.90:0/33306708 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c0415b3f0 con 0x7f4c0415da30 2024-03-24T01:25:31.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.969+0000 7f4c09000640 1 --2- 172.21.15.90:0/33306708 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c0415da30 0x7f4c0406f350 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc00e810 tx=0x7f4bfc00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:31.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.970+0000 7f4bfbe00640 1 -- 172.21.15.90:0/33306708 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc00f040 con 0x7f4c0415da30 2024-03-24T01:25:31.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.970+0000 7f4bfbe00640 1 -- 172.21.15.90:0/33306708 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bfc004030 con 0x7f4c0415da30 2024-03-24T01:25:31.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.970+0000 7f4bfbe00640 1 -- 172.21.15.90:0/33306708 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc0043e0 con 0x7f4c0415da30 2024-03-24T01:25:31.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.970+0000 7f4c0b000640 1 -- 172.21.15.90:0/33306708 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c0415b6d0 con 0x7f4c0415da30 2024-03-24T01:25:31.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.970+0000 7f4c0b000640 1 -- 172.21.15.90:0/33306708 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c0415bc90 con 0x7f4c0415da30 2024-03-24T01:25:31.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.971+0000 7f4c0b000640 1 -- 172.21.15.90:0/33306708 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c0404f400 con 0x7f4c0415da30 2024-03-24T01:25:31.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.972+0000 7f4bfbe00640 1 -- 172.21.15.90:0/33306708 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4bfc00c6d0 con 0x7f4c0415da30 2024-03-24T01:25:31.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.973+0000 7f4bfbe00640 1 --2- 172.21.15.90:0/33306708 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4be006bbd0 0x7f4be006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:31.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.973+0000 7f4bfbe00640 1 -- 172.21.15.90:0/33306708 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f4bfc08ac40 con 0x7f4c0415da30 2024-03-24T01:25:31.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.974+0000 7f4c03e00640 1 --2- 172.21.15.90:0/33306708 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4be006bbd0 0x7f4be006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:31.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.975+0000 7f4c03e00640 1 --2- 172.21.15.90:0/33306708 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4be006bbd0 0x7f4be006e090 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4bf4036000 tx=0x7f4bf4030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:31.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:31.977+0000 7f4bfbe00640 1 -- 172.21.15.90:0/33306708 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4bfc017020 con 0x7f4c0415da30 2024-03-24T01:25:32.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.171+0000 7f4c0b000640 1 -- 172.21.15.90:0/33306708 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f4c0406e250 con 0x7f4c0415da30 2024-03-24T01:25:32.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.172+0000 7f4bfbe00640 1 -- 172.21.15.90:0/33306708 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f4bfc059270 con 0x7f4c0415da30 2024-03-24T01:25:32.172 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:25:32.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.174+0000 7f4c0b000640 1 -- 172.21.15.90:0/33306708 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4be006bbd0 msgr2=0x7f4be006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:32.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.174+0000 7f4c0b000640 1 --2- 172.21.15.90:0/33306708 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4be006bbd0 0x7f4be006e090 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4bf4036000 tx=0x7f4bf4030000 comp rx=0 tx=0).stop 2024-03-24T01:25:32.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.174+0000 7f4c0b000640 1 -- 172.21.15.90:0/33306708 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c0415da30 msgr2=0x7f4c0406f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:32.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.174+0000 7f4c0b000640 1 --2- 172.21.15.90:0/33306708 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c0415da30 0x7f4c0406f350 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc00e810 tx=0x7f4bfc00ecd0 comp rx=0 tx=0).stop 2024-03-24T01:25:32.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.174+0000 7f4c0b000640 1 -- 172.21.15.90:0/33306708 shutdown_connections 2024-03-24T01:25:32.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.174+0000 7f4c0b000640 1 --2- 172.21.15.90:0/33306708 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4be006bbd0 0x7f4be006e090 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:32.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.174+0000 7f4c0b000640 1 --2- 172.21.15.90:0/33306708 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c0415da30 0x7f4c0406f350 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:32.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.174+0000 7f4c0b000640 1 --2- 172.21.15.90:0/33306708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c040acea0 0x7f4c0406ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:32.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.174+0000 7f4c0b000640 1 --2- 172.21.15.90:0/33306708 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c0406b0c0 0x7f4c041474a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:32.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.174+0000 7f4c0b000640 1 -- 172.21.15.90:0/33306708 >> 172.21.15.90:0/33306708 conn(0x7f4c040b1770 msgr2=0x7f4c040b3730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:32.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.175+0000 7f4c0b000640 1 -- 172.21.15.90:0/33306708 shutdown_connections 2024-03-24T01:25:32.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.175+0000 7f4c0b000640 1 -- 172.21.15.90:0/33306708 wait complete. 2024-03-24T01:25:32.194 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T01:25:32.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.401+0000 7fedeb200640 1 Processor -- start 2024-03-24T01:25:32.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.401+0000 7fedeb200640 1 -- start start 2024-03-24T01:25:32.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.403+0000 7fedeb200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fede40aeeb0 0x7fede40b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:32.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.403+0000 7fedeb200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fede415bc50 0x7fede406b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:32.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.403+0000 7fedeb200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fede406b600 0x7fede415e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:32.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.403+0000 7fedeb200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fede406ba70 con 0x7fede40aeeb0 2024-03-24T01:25:32.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.403+0000 7fedeb200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fede40b4da0 con 0x7fede406b600 2024-03-24T01:25:32.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.403+0000 7fedeb200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fede406dac0 con 0x7fede415bc50 2024-03-24T01:25:32.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.403+0000 7fede3e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fede40aeeb0 0x7fede40b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:32.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.403+0000 7fede3e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fede40aeeb0 0x7fede40b44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55052/0 (socket says 172.21.15.90:55052) 2024-03-24T01:25:32.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.403+0000 7fede3e00640 1 -- 172.21.15.90:0/2387149974 learned_addr learned my addr 172.21.15.90:0/2387149974 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:32.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.404+0000 7feddb400640 1 --2- 172.21.15.90:0/2387149974 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fede415bc50 0x7fede406b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:32.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.404+0000 7fede3e00640 1 -- 172.21.15.90:0/2387149974 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fede415bc50 msgr2=0x7fede406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:32.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.404+0000 7fede3e00640 1 --2- 172.21.15.90:0/2387149974 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fede415bc50 0x7fede406b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:32.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.404+0000 7fede3e00640 1 -- 172.21.15.90:0/2387149974 >> v2:172.21.15.120:3300/0 conn(0x7fede406b600 msgr2=0x7fede415e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:32.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.404+0000 7fede9200640 1 --2- 172.21.15.90:0/2387149974 >> v2:172.21.15.120:3300/0 conn(0x7fede406b600 0x7fede415e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:32.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.404+0000 7fede3e00640 1 --2- 172.21.15.90:0/2387149974 >> v2:172.21.15.120:3300/0 conn(0x7fede406b600 0x7fede415e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:32.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.404+0000 7fede3e00640 1 -- 172.21.15.90:0/2387149974 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede406dda0 con 0x7fede40aeeb0 2024-03-24T01:25:32.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.405+0000 7fede3e00640 1 --2- 172.21.15.90:0/2387149974 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fede40aeeb0 0x7fede40b44c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fede40cc5d0 tx=0x7fedd402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=599a7beb7412b715 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:32.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.405+0000 7fede3400640 1 -- 172.21.15.90:0/2387149974 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd4038070 con 0x7fede40aeeb0 2024-03-24T01:25:32.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.405+0000 7fede3400640 1 -- 172.21.15.90:0/2387149974 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd402f050 con 0x7fede40aeeb0 2024-03-24T01:25:32.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.405+0000 7fede3400640 1 -- 172.21.15.90:0/2387149974 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd4031970 con 0x7fede40aeeb0 2024-03-24T01:25:32.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.406+0000 7fedeb200640 1 -- 172.21.15.90:0/2387149974 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fede40aeeb0 msgr2=0x7fede40b44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:32.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.406+0000 7fedeb200640 1 --2- 172.21.15.90:0/2387149974 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fede40aeeb0 0x7fede40b44c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fede40cc5d0 tx=0x7fedd402b7f0 comp rx=0 tx=0).stop 2024-03-24T01:25:32.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.406+0000 7fedeb200640 1 -- 172.21.15.90:0/2387149974 shutdown_connections 2024-03-24T01:25:32.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.406+0000 7fedeb200640 1 --2- 172.21.15.90:0/2387149974 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fede415bc50 0x7fede406b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:32.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.406+0000 7fedeb200640 1 --2- 172.21.15.90:0/2387149974 >> v2:172.21.15.120:3300/0 conn(0x7fede406b600 0x7fede415e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:32.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.406+0000 7fedeb200640 1 --2- 172.21.15.90:0/2387149974 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fede40aeeb0 0x7fede40b44c0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:32.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.406+0000 7fedeb200640 1 -- 172.21.15.90:0/2387149974 >> 172.21.15.90:0/2387149974 conn(0x7fede40b1770 msgr2=0x7fede40b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:32.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.406+0000 7fedeb200640 1 -- 172.21.15.90:0/2387149974 shutdown_connections 2024-03-24T01:25:32.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.406+0000 7fedeb200640 1 -- 172.21.15.90:0/2387149974 wait complete. 2024-03-24T01:25:32.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.408+0000 7fedeb200640 1 Processor -- start 2024-03-24T01:25:32.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.408+0000 7fedeb200640 1 -- start start 2024-03-24T01:25:32.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.409+0000 7fedeb200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fede406b600 0x7fede40ad2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:32.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.409+0000 7fedeb200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fede40aeeb0 0x7fede40c4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:32.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.409+0000 7fedeb200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fede415bc50 0x7fede4158e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:32.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.409+0000 7fedeb200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede40af290 con 0x7fede406b600 2024-03-24T01:25:32.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.409+0000 7fedeb200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede406e360 con 0x7fede40aeeb0 2024-03-24T01:25:32.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.409+0000 7fedeb200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fede40b4a00 con 0x7fede415bc50 2024-03-24T01:25:32.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.409+0000 7fede3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fede406b600 0x7fede40ad2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:32.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.409+0000 7fede3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fede406b600 0x7fede40ad2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55060/0 (socket says 172.21.15.90:55060) 2024-03-24T01:25:32.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.409+0000 7fede3e00640 1 -- 172.21.15.90:0/2645950436 learned_addr learned my addr 172.21.15.90:0/2645950436 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:32.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.410+0000 7fede9200640 1 --2- 172.21.15.90:0/2645950436 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fede415bc50 0x7fede4158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:32.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.410+0000 7fede3e00640 1 -- 172.21.15.90:0/2645950436 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fede415bc50 msgr2=0x7fede4158e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:32.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.410+0000 7fede3e00640 1 --2- 172.21.15.90:0/2645950436 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fede415bc50 0x7fede4158e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:32.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.410+0000 7fede3e00640 1 -- 172.21.15.90:0/2645950436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fede40aeeb0 msgr2=0x7fede40c4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:32.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.410+0000 7fede3e00640 1 --2- 172.21.15.90:0/2645950436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fede40aeeb0 0x7fede40c4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:32.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.410+0000 7fede3e00640 1 -- 172.21.15.90:0/2645950436 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede414b690 con 0x7fede406b600 2024-03-24T01:25:32.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.410+0000 7fede3e00640 1 --2- 172.21.15.90:0/2645950436 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fede406b600 0x7fede40ad2c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fedd4000c00 tx=0x7fedd40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:32.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.410+0000 7fede0c00640 1 -- 172.21.15.90:0/2645950436 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd4042070 con 0x7fede406b600 2024-03-24T01:25:32.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.410+0000 7fede0c00640 1 -- 172.21.15.90:0/2645950436 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd403d890 con 0x7fede406b600 2024-03-24T01:25:32.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.410+0000 7fede0c00640 1 -- 172.21.15.90:0/2645950436 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd4038040 con 0x7fede406b600 2024-03-24T01:25:32.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.410+0000 7fedeb200640 1 -- 172.21.15.90:0/2645950436 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fede414b970 con 0x7fede406b600 2024-03-24T01:25:32.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.411+0000 7fedeb200640 1 -- 172.21.15.90:0/2645950436 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fede414bf30 con 0x7fede406b600 2024-03-24T01:25:32.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.412+0000 7fedeb200640 1 -- 172.21.15.90:0/2645950436 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fede404f400 con 0x7fede406b600 2024-03-24T01:25:32.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.413+0000 7fede0c00640 1 -- 172.21.15.90:0/2645950436 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fedd4045070 con 0x7fede406b600 2024-03-24T01:25:32.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.413+0000 7fede0c00640 1 --2- 172.21.15.90:0/2645950436 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fedc006bc00 0x7fedc006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:32.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.414+0000 7fede0c00640 1 -- 172.21.15.90:0/2645950436 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7fedd40acbc0 con 0x7fede406b600 2024-03-24T01:25:32.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.414+0000 7feddb400640 1 --2- 172.21.15.90:0/2645950436 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fedc006bc00 0x7fedc006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:32.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.415+0000 7feddb400640 1 --2- 172.21.15.90:0/2645950436 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fedc006bc00 0x7fedc006e0c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fede415d8d0 tx=0x7fedcc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:32.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.418+0000 7fede0c00640 1 -- 172.21.15.90:0/2645950436 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fedd407b3b0 con 0x7fede406b600 2024-03-24T01:25:32.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.612+0000 7fedeb200640 1 -- 172.21.15.90:0/2645950436 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fede40ad800 con 0x7fede406b600 2024-03-24T01:25:32.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.612+0000 7fede0c00640 1 -- 172.21.15.90:0/2645950436 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7fedd407acb0 con 0x7fede406b600 2024-03-24T01:25:32.613 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-24T01:25:32.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.615+0000 7fedeb200640 1 -- 172.21.15.90:0/2645950436 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fedc006bc00 msgr2=0x7fedc006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:32.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.615+0000 7fedeb200640 1 --2- 172.21.15.90:0/2645950436 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fedc006bc00 0x7fedc006e0c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fede415d8d0 tx=0x7fedcc008040 comp rx=0 tx=0).stop 2024-03-24T01:25:32.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.615+0000 7fedeb200640 1 -- 172.21.15.90:0/2645950436 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fede406b600 msgr2=0x7fede40ad2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:32.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.615+0000 7fedeb200640 1 --2- 172.21.15.90:0/2645950436 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fede406b600 0x7fede40ad2c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fedd4000c00 tx=0x7fedd40046c0 comp rx=0 tx=0).stop 2024-03-24T01:25:32.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.615+0000 7fedeb200640 1 -- 172.21.15.90:0/2645950436 shutdown_connections 2024-03-24T01:25:32.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.615+0000 7fedeb200640 1 --2- 172.21.15.90:0/2645950436 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fedc006bc00 0x7fedc006e0c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:32.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.615+0000 7fedeb200640 1 --2- 172.21.15.90:0/2645950436 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fede415bc50 0x7fede4158e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:32.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.615+0000 7fedeb200640 1 --2- 172.21.15.90:0/2645950436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fede40aeeb0 0x7fede40c4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:32.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.615+0000 7fedeb200640 1 --2- 172.21.15.90:0/2645950436 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fede406b600 0x7fede40ad2c0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:32.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.615+0000 7fedeb200640 1 -- 172.21.15.90:0/2645950436 >> 172.21.15.90:0/2645950436 conn(0x7fede40b1770 msgr2=0x7fede40b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:32.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.615+0000 7fedeb200640 1 -- 172.21.15.90:0/2645950436 shutdown_connections 2024-03-24T01:25:32.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.615+0000 7fedeb200640 1 -- 172.21.15.90:0/2645950436 wait complete. 2024-03-24T01:25:32.631 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:25:32.631 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:25:32.684 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:25:32.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.883+0000 7f26ec400640 1 Processor -- start 2024-03-24T01:25:32.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.883+0000 7f26ec400640 1 -- start start 2024-03-24T01:25:32.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.885+0000 7f26ec400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f26e406b7d0 0x7f26e4166a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:32.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.885+0000 7f26ec400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e40cd490 con 0x7f26e41603e0 2024-03-24T01:25:32.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.885+0000 7f26ec400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e415f900 con 0x7f26e40cc150 2024-03-24T01:25:32.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.885+0000 7f26ec400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e40cd270 con 0x7f26e406b7d0 2024-03-24T01:25:32.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.885+0000 7f26e9000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f26e41603e0 0x7f26e406b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:32784/0 (socket says 172.21.15.90:32784) 2024-03-24T01:25:32.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.885+0000 7f26e9000640 1 -- 172.21.15.90:0/2183982259 learned_addr learned my addr 172.21.15.90:0/2183982259 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:32.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.885+0000 7f26ea400640 1 --2- 172.21.15.90:0/2183982259 >> v2:172.21.15.90:3300/0 conn(0x7f26e406b7d0 0x7f26e4166a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:32.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.886+0000 7f26e3e00640 1 -- 172.21.15.90:0/2183982259 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2684255475 0 0) 0x7f26e40cd490 con 0x7f26e41603e0 2024-03-24T01:25:32.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.886+0000 7f26e3e00640 1 -- 172.21.15.90:0/2183982259 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26c8003520 con 0x7f26e41603e0 2024-03-24T01:25:32.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.886+0000 7f26ea400640 1 -- 172.21.15.90:0/2183982259 >> v1:172.21.15.120:6789/0 conn(0x7f26e40cc150 legacy=0x7f26e40ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:32.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.886+0000 7f26ea400640 1 -- 172.21.15.90:0/2183982259 >> v1:172.21.15.90:6789/0 conn(0x7f26e41603e0 legacy=0x7f26e406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:32.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.886+0000 7f26ea400640 1 -- 172.21.15.90:0/2183982259 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e415fbe0 con 0x7f26e406b7d0 2024-03-24T01:25:32.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.887+0000 7f26ea400640 1 --2- 172.21.15.90:0/2183982259 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26e406b7d0 0x7f26e4166a80 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f26dc0097a0 tx=0x7f26dc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=767bfee3b3481bb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:32.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.887+0000 7f26e3e00640 1 -- 172.21.15.90:0/2183982259 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc034070 con 0x7f26e406b7d0 2024-03-24T01:25:32.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.888+0000 7f26e3e00640 1 -- 172.21.15.90:0/2183982259 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26dc030610 con 0x7f26e406b7d0 2024-03-24T01:25:32.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.888+0000 7f26e3e00640 1 -- 172.21.15.90:0/2183982259 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc0309c0 con 0x7f26e406b7d0 2024-03-24T01:25:32.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.888+0000 7f26ec400640 1 -- 172.21.15.90:0/2183982259 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26e406b7d0 msgr2=0x7f26e4166a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:32.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.888+0000 7f26ec400640 1 --2- 172.21.15.90:0/2183982259 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26e406b7d0 0x7f26e4166a80 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f26dc0097a0 tx=0x7f26dc02b960 comp rx=0 tx=0).stop 2024-03-24T01:25:32.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.888+0000 7f26ec400640 1 -- 172.21.15.90:0/2183982259 shutdown_connections 2024-03-24T01:25:32.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.888+0000 7f26ec400640 1 --2- 172.21.15.90:0/2183982259 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26e406b7d0 0x7f26e4166a80 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:32.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.888+0000 7f26ec400640 1 -- 172.21.15.90:0/2183982259 >> 172.21.15.90:0/2183982259 conn(0x7f26e40b1770 msgr2=0x7f26e40b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:32.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.888+0000 7f26ec400640 1 -- 172.21.15.90:0/2183982259 shutdown_connections 2024-03-24T01:25:32.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.888+0000 7f26ec400640 1 -- 172.21.15.90:0/2183982259 wait complete. 2024-03-24T01:25:32.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.890+0000 7f26ec400640 1 Processor -- start 2024-03-24T01:25:32.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.890+0000 7f26ec400640 1 -- start start 2024-03-24T01:25:32.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.891+0000 7f26ec400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26e406b7d0 0x7f26e406f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:32.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.891+0000 7f26ec400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26e40cc150 0x7f26e415a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:32.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.891+0000 7f26ec400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26e41603e0 0x7f26e415d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:32.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.891+0000 7f26ec400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e41455b0 con 0x7f26e41603e0 2024-03-24T01:25:32.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.891+0000 7f26ec400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e4145430 con 0x7f26e406b7d0 2024-03-24T01:25:32.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.891+0000 7f26ec400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e4166fc0 con 0x7f26e40cc150 2024-03-24T01:25:32.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.891+0000 7f26ea400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26e41603e0 0x7f26e415d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:32.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.891+0000 7f26ea400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26e41603e0 0x7f26e415d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55084/0 (socket says 172.21.15.90:55084) 2024-03-24T01:25:32.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.891+0000 7f26ea400640 1 -- 172.21.15.90:0/2080312220 learned_addr learned my addr 172.21.15.90:0/2080312220 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:32.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.891+0000 7f26ea400640 1 -- 172.21.15.90:0/2080312220 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26e40cc150 msgr2=0x7f26e415a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:32.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.891+0000 7f26e9a00640 1 --2- 172.21.15.90:0/2080312220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26e406b7d0 0x7f26e406f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:32.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.891+0000 7f26ea400640 1 --2- 172.21.15.90:0/2080312220 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26e40cc150 0x7f26e415a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:32.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.891+0000 7f26ea400640 1 -- 172.21.15.90:0/2080312220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26e406b7d0 msgr2=0x7f26e406f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:32.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.891+0000 7f26ea400640 1 --2- 172.21.15.90:0/2080312220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26e406b7d0 0x7f26e406f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:32.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.891+0000 7f26ea400640 1 -- 172.21.15.90:0/2080312220 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e40acb30 con 0x7f26e41603e0 2024-03-24T01:25:32.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.891+0000 7f26e9a00640 1 --2- 172.21.15.90:0/2080312220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26e406b7d0 0x7f26e406f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:32.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.892+0000 7f26ea400640 1 --2- 172.21.15.90:0/2080312220 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26e41603e0 0x7f26e415d170 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f26dc02b9a0 tx=0x7f26dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:32.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.892+0000 7f26e1600640 1 -- 172.21.15.90:0/2080312220 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc034040 con 0x7f26e41603e0 2024-03-24T01:25:32.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.892+0000 7f26e1600640 1 -- 172.21.15.90:0/2080312220 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26dc030410 con 0x7f26e41603e0 2024-03-24T01:25:32.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.892+0000 7f26e1600640 1 -- 172.21.15.90:0/2080312220 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc006e10 con 0x7f26e41603e0 2024-03-24T01:25:32.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.892+0000 7f26ec400640 1 -- 172.21.15.90:0/2080312220 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e40ace10 con 0x7f26e41603e0 2024-03-24T01:25:32.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.892+0000 7f26ec400640 1 -- 172.21.15.90:0/2080312220 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e40ad3d0 con 0x7f26e41603e0 2024-03-24T01:25:32.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.893+0000 7f26ec400640 1 -- 172.21.15.90:0/2080312220 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e4068060 con 0x7f26e41603e0 2024-03-24T01:25:32.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.894+0000 7f26e1600640 1 -- 172.21.15.90:0/2080312220 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f26dc04c060 con 0x7f26e41603e0 2024-03-24T01:25:32.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.895+0000 7f26e1600640 1 --2- 172.21.15.90:0/2080312220 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f26c406bc00 0x7f26c406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:32.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.895+0000 7f26e1600640 1 -- 172.21.15.90:0/2080312220 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f26dc0ac8b0 con 0x7f26e41603e0 2024-03-24T01:25:32.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.896+0000 7f26e9a00640 1 --2- 172.21.15.90:0/2080312220 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f26c406bc00 0x7f26c406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:32.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.897+0000 7f26e9a00640 1 --2- 172.21.15.90:0/2080312220 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f26c406bc00 0x7f26c406e0c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f26d4001680 tx=0x7f26d4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:32.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:32.899+0000 7f26e1600640 1 -- 172.21.15.90:0/2080312220 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f26dc04d3b0 con 0x7f26e41603e0 2024-03-24T01:25:33.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.097+0000 7f26ec400640 1 -- 172.21.15.90:0/2080312220 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26e406f980 con 0x7f26e41603e0 2024-03-24T01:25:33.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.099+0000 7f26e1600640 1 -- 172.21.15.90:0/2080312220 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18925 (secure 0 0 0) 0x7f26dc07af70 con 0x7f26e41603e0 2024-03-24T01:25:33.099 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:25:33.099 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:24:53.236401+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:24:08.643966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T01:24:09.189904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T01:24:13.027323+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:25:33.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.102+0000 7f26ec400640 1 -- 172.21.15.90:0/2080312220 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f26c406bc00 msgr2=0x7f26c406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:33.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.102+0000 7f26ec400640 1 --2- 172.21.15.90:0/2080312220 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f26c406bc00 0x7f26c406e0c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f26d4001680 tx=0x7f26d4009040 comp rx=0 tx=0).stop 2024-03-24T01:25:33.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.102+0000 7f26ec400640 1 -- 172.21.15.90:0/2080312220 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26e41603e0 msgr2=0x7f26e415d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:33.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.102+0000 7f26ec400640 1 --2- 172.21.15.90:0/2080312220 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26e41603e0 0x7f26e415d170 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f26dc02b9a0 tx=0x7f26dc0046c0 comp rx=0 tx=0).stop 2024-03-24T01:25:33.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.102+0000 7f26ec400640 1 -- 172.21.15.90:0/2080312220 shutdown_connections 2024-03-24T01:25:33.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.102+0000 7f26ec400640 1 --2- 172.21.15.90:0/2080312220 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f26c406bc00 0x7f26c406e0c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:33.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.102+0000 7f26ec400640 1 --2- 172.21.15.90:0/2080312220 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26e41603e0 0x7f26e415d170 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:33.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.102+0000 7f26ec400640 1 --2- 172.21.15.90:0/2080312220 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26e40cc150 0x7f26e415a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:33.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.102+0000 7f26ec400640 1 --2- 172.21.15.90:0/2080312220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26e406b7d0 0x7f26e406f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:33.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.102+0000 7f26ec400640 1 -- 172.21.15.90:0/2080312220 >> 172.21.15.90:0/2080312220 conn(0x7f26e40b1770 msgr2=0x7f26e406b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:33.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.102+0000 7f26ec400640 1 -- 172.21.15.90:0/2080312220 shutdown_connections 2024-03-24T01:25:33.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.102+0000 7f26ec400640 1 -- 172.21.15.90:0/2080312220 wait complete. 2024-03-24T01:25:33.120 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T01:24:08.643966+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T01:24:09.189904+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T01:24:13.027323+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T01:25:33.121 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:25:33.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.330+0000 7f7c8a800640 1 Processor -- start 2024-03-24T01:25:33.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.330+0000 7f7c8a800640 1 -- start start 2024-03-24T01:25:33.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.332+0000 7f7c8a800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7c8406b7d0 0x7f7c8406ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:33.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.332+0000 7f7c8a800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7c840aefb0 0x7f7c840b4120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:33.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.332+0000 7f7c8a800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c8406e470 con 0x7f7c84159d10 2024-03-24T01:25:33.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.332+0000 7f7c8a800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c8415a0f0 con 0x7f7c840aefb0 2024-03-24T01:25:33.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.332+0000 7f7c8a800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c840b2ba0 con 0x7f7c8406b7d0 2024-03-24T01:25:33.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.332+0000 7f7c83e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7c840aefb0 0x7f7c840b4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:33.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.332+0000 7f7c83e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7c840aefb0 0x7f7c840b4120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55088/0 (socket says 172.21.15.90:55088) 2024-03-24T01:25:33.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.332+0000 7f7c83e00640 1 -- 172.21.15.90:0/3349562031 learned_addr learned my addr 172.21.15.90:0/3349562031 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:33.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.333+0000 7f7c83e00640 1 -- 172.21.15.90:0/3349562031 >> v2:172.21.15.120:3300/0 conn(0x7f7c8406b7d0 msgr2=0x7f7c8406ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:33.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.333+0000 7f7c83e00640 1 --2- 172.21.15.90:0/3349562031 >> v2:172.21.15.120:3300/0 conn(0x7f7c8406b7d0 0x7f7c8406ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:33.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.333+0000 7f7c83e00640 1 -- 172.21.15.90:0/3349562031 >> v1:172.21.15.120:6789/0 conn(0x7f7c84159d10 legacy=0x7f7c8406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:33.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.333+0000 7f7c83e00640 1 -- 172.21.15.90:0/3349562031 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c840b2e80 con 0x7f7c840aefb0 2024-03-24T01:25:33.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.334+0000 7f7c83e00640 1 --2- 172.21.15.90:0/3349562031 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c840aefb0 0x7f7c840b4120 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f7c8406f9b0 tx=0x7f7c7c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2003fa894a96a93 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:33.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.334+0000 7f7c82a00640 1 -- 172.21.15.90:0/3349562031 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7c02c9e0 con 0x7f7c840aefb0 2024-03-24T01:25:33.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.334+0000 7f7c82a00640 1 -- 172.21.15.90:0/3349562031 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c7c02cb80 con 0x7f7c840aefb0 2024-03-24T01:25:33.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.334+0000 7f7c82a00640 1 -- 172.21.15.90:0/3349562031 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7c0343f0 con 0x7f7c840aefb0 2024-03-24T01:25:33.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.335+0000 7f7c8a800640 1 -- 172.21.15.90:0/3349562031 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c840aefb0 msgr2=0x7f7c840b4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:33.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.335+0000 7f7c8a800640 1 --2- 172.21.15.90:0/3349562031 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c840aefb0 0x7f7c840b4120 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f7c8406f9b0 tx=0x7f7c7c02be40 comp rx=0 tx=0).stop 2024-03-24T01:25:33.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.335+0000 7f7c8a800640 1 -- 172.21.15.90:0/3349562031 shutdown_connections 2024-03-24T01:25:33.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.335+0000 7f7c8a800640 1 --2- 172.21.15.90:0/3349562031 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c840aefb0 0x7f7c840b4120 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:33.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.335+0000 7f7c8a800640 1 --2- 172.21.15.90:0/3349562031 >> v2:172.21.15.120:3300/0 conn(0x7f7c8406b7d0 0x7f7c8406ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:33.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.335+0000 7f7c8a800640 1 -- 172.21.15.90:0/3349562031 >> 172.21.15.90:0/3349562031 conn(0x7f7c840b1770 msgr2=0x7f7c840b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:33.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.335+0000 7f7c8a800640 1 -- 172.21.15.90:0/3349562031 shutdown_connections 2024-03-24T01:25:33.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.335+0000 7f7c8a800640 1 -- 172.21.15.90:0/3349562031 wait complete. 2024-03-24T01:25:33.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.337+0000 7f7c8a800640 1 Processor -- start 2024-03-24T01:25:33.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.337+0000 7f7c8a800640 1 -- start start 2024-03-24T01:25:33.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.338+0000 7f7c8a800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c8406b7d0 0x7f7c8415c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:33.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.338+0000 7f7c8a800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c840aefb0 0x7f7c841526d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:33.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.338+0000 7f7c8a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c84159d10 0x7f7c841483b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:33.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.338+0000 7f7c8a800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c840b2890 con 0x7f7c84159d10 2024-03-24T01:25:33.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.338+0000 7f7c8a800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8406e9f0 con 0x7f7c8406b7d0 2024-03-24T01:25:33.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.338+0000 7f7c8a800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8406fa80 con 0x7f7c840aefb0 2024-03-24T01:25:33.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.338+0000 7f7c83e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c84159d10 0x7f7c841483b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:33.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.338+0000 7f7c83e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c84159d10 0x7f7c841483b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55092/0 (socket says 172.21.15.90:55092) 2024-03-24T01:25:33.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.338+0000 7f7c83e00640 1 -- 172.21.15.90:0/2573250213 learned_addr learned my addr 172.21.15.90:0/2573250213 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:33.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.339+0000 7f7c7aa00640 1 --2- 172.21.15.90:0/2573250213 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c840aefb0 0x7f7c841526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:33.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.339+0000 7f7c83400640 1 --2- 172.21.15.90:0/2573250213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c8406b7d0 0x7f7c8415c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:33.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.339+0000 7f7c83e00640 1 -- 172.21.15.90:0/2573250213 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c840aefb0 msgr2=0x7f7c841526d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:33.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.339+0000 7f7c83e00640 1 --2- 172.21.15.90:0/2573250213 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c840aefb0 0x7f7c841526d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:33.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.339+0000 7f7c83e00640 1 -- 172.21.15.90:0/2573250213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c8406b7d0 msgr2=0x7f7c8415c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:33.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.339+0000 7f7c83e00640 1 --2- 172.21.15.90:0/2573250213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c8406b7d0 0x7f7c8415c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:33.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.339+0000 7f7c83e00640 1 -- 172.21.15.90:0/2573250213 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c84154ab0 con 0x7f7c84159d10 2024-03-24T01:25:33.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.339+0000 7f7c7aa00640 1 --2- 172.21.15.90:0/2573250213 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c840aefb0 0x7f7c841526d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:33.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.339+0000 7f7c83400640 1 --2- 172.21.15.90:0/2573250213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c8406b7d0 0x7f7c8415c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:33.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.339+0000 7f7c83e00640 1 --2- 172.21.15.90:0/2573250213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c84159d10 0x7f7c841483b0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c009f10 tx=0x7f7c7c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:33.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.339+0000 7f7c7be00640 1 -- 172.21.15.90:0/2573250213 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7c043020 con 0x7f7c84159d10 2024-03-24T01:25:33.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.339+0000 7f7c7be00640 1 -- 172.21.15.90:0/2573250213 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c7c03e450 con 0x7f7c84159d10 2024-03-24T01:25:33.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.339+0000 7f7c7be00640 1 -- 172.21.15.90:0/2573250213 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7c03e820 con 0x7f7c84159d10 2024-03-24T01:25:33.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.339+0000 7f7c8a800640 1 -- 172.21.15.90:0/2573250213 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c8415bca0 con 0x7f7c84159d10 2024-03-24T01:25:33.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.340+0000 7f7c8a800640 1 -- 172.21.15.90:0/2573250213 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c8415c260 con 0x7f7c84159d10 2024-03-24T01:25:33.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.341+0000 7f7c8a800640 1 -- 172.21.15.90:0/2573250213 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c8404f400 con 0x7f7c84159d10 2024-03-24T01:25:33.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.341+0000 7f7c7be00640 1 -- 172.21.15.90:0/2573250213 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7c7c046020 con 0x7f7c84159d10 2024-03-24T01:25:33.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.342+0000 7f7c7be00640 1 --2- 172.21.15.90:0/2573250213 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f7c6006bbd0 0x7f7c6006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:33.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.342+0000 7f7c7be00640 1 -- 172.21.15.90:0/2573250213 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f7c7c0ad760 con 0x7f7c84159d10 2024-03-24T01:25:33.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.343+0000 7f7c83400640 1 --2- 172.21.15.90:0/2573250213 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f7c6006bbd0 0x7f7c6006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:33.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.344+0000 7f7c83400640 1 --2- 172.21.15.90:0/2573250213 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f7c6006bbd0 0x7f7c6006e090 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7c7400a810 tx=0x7f7c74008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:33.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.347+0000 7f7c7be00640 1 -- 172.21.15.90:0/2573250213 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7c7c04d030 con 0x7f7c84159d10 2024-03-24T01:25:33.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.542+0000 7f7c8a800640 1 -- 172.21.15.90:0/2573250213 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7c8406e250 con 0x7f7c84159d10 2024-03-24T01:25:33.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.543+0000 7f7c7be00640 1 -- 172.21.15.90:0/2573250213 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7c7c07bec0 con 0x7f7c84159d10 2024-03-24T01:25:33.544 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:25:33.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.545+0000 7f7c8a800640 1 -- 172.21.15.90:0/2573250213 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f7c6006bbd0 msgr2=0x7f7c6006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:33.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.545+0000 7f7c8a800640 1 --2- 172.21.15.90:0/2573250213 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f7c6006bbd0 0x7f7c6006e090 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7c7400a810 tx=0x7f7c74008040 comp rx=0 tx=0).stop 2024-03-24T01:25:33.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.545+0000 7f7c8a800640 1 -- 172.21.15.90:0/2573250213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c84159d10 msgr2=0x7f7c841483b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:33.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.545+0000 7f7c8a800640 1 --2- 172.21.15.90:0/2573250213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c84159d10 0x7f7c841483b0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c009f10 tx=0x7f7c7c004a40 comp rx=0 tx=0).stop 2024-03-24T01:25:33.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.545+0000 7f7c8a800640 1 -- 172.21.15.90:0/2573250213 shutdown_connections 2024-03-24T01:25:33.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.545+0000 7f7c8a800640 1 --2- 172.21.15.90:0/2573250213 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f7c6006bbd0 0x7f7c6006e090 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:33.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.545+0000 7f7c8a800640 1 --2- 172.21.15.90:0/2573250213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c84159d10 0x7f7c841483b0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:33.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.545+0000 7f7c8a800640 1 --2- 172.21.15.90:0/2573250213 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7c840aefb0 0x7f7c841526d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:33.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.545+0000 7f7c8a800640 1 --2- 172.21.15.90:0/2573250213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c8406b7d0 0x7f7c8415c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:33.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.546+0000 7f7c8a800640 1 -- 172.21.15.90:0/2573250213 >> 172.21.15.90:0/2573250213 conn(0x7f7c840b1770 msgr2=0x7f7c840b31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:33.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.546+0000 7f7c8a800640 1 -- 172.21.15.90:0/2573250213 shutdown_connections 2024-03-24T01:25:33.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.546+0000 7f7c8a800640 1 -- 172.21.15.90:0/2573250213 wait complete. 2024-03-24T01:25:33.565 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T01:25:33.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.774+0000 7f3825a00640 1 Processor -- start 2024-03-24T01:25:33.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.774+0000 7f3825a00640 1 -- start start 2024-03-24T01:25:33.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.776+0000 7f3825a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38200aeeb0 0x7f38200b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:33.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.776+0000 7f3825a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f382006dac0 con 0x7f382015bc50 2024-03-24T01:25:33.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.776+0000 7f3825a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f382006dda0 con 0x7f382006b7d0 2024-03-24T01:25:33.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.776+0000 7f3825a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38200b4da0 con 0x7f38200aeeb0 2024-03-24T01:25:33.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.776+0000 7f3816000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f382015bc50 0x7f382006b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:32786/0 (socket says 172.21.15.90:32786) 2024-03-24T01:25:33.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.776+0000 7f3816000640 1 -- 172.21.15.90:0/155958775 learned_addr learned my addr 172.21.15.90:0/155958775 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:33.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.777+0000 7f381ea00640 1 --2- 172.21.15.90:0/155958775 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38200aeeb0 0x7f38200b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:33.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.777+0000 7f381e000640 1 -- 172.21.15.90:0/155958775 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3686551058 0 0) 0x7f382006dac0 con 0x7f382015bc50 2024-03-24T01:25:33.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.777+0000 7f381e000640 1 -- 172.21.15.90:0/155958775 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3804005730 con 0x7f382015bc50 2024-03-24T01:25:33.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.777+0000 7f381e000640 1 -- 172.21.15.90:0/155958775 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1664993967 0 0) 0x7f3804005730 con 0x7f382015bc50 2024-03-24T01:25:33.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.777+0000 7f381e000640 1 -- 172.21.15.90:0/155958775 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38200aeeb0 msgr2=0x7f38200b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:33.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.777+0000 7f381e000640 1 --2- 172.21.15.90:0/155958775 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38200aeeb0 0x7f38200b44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:33.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.777+0000 7f381e000640 1 -- 172.21.15.90:0/155958775 >> v1:172.21.15.120:6789/0 conn(0x7f382006b7d0 legacy=0x7f38200b2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:33.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.777+0000 7f381e000640 1 -- 172.21.15.90:0/155958775 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f382006e080 con 0x7f382015bc50 2024-03-24T01:25:33.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.777+0000 7f381e000640 1 -- 172.21.15.90:0/155958775 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f3808001670 con 0x7f382015bc50 2024-03-24T01:25:33.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.777+0000 7f381e000640 1 -- 172.21.15.90:0/155958775 >> v1:172.21.15.90:6789/0 conn(0x7f382015bc50 legacy=0x7f382006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:33.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.777+0000 7f381e000640 1 --2- 172.21.15.90:0/155958775 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3804005bf0 0x7f3804006030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:33.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.777+0000 7f381e000640 1 --2- 172.21.15.90:0/155958775 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38200aeeb0 0x7f3804006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:33.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.777+0000 7f381e000640 1 --2- 172.21.15.90:0/155958775 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3804007070 0x7f380400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:33.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.777+0000 7f381e000640 1 -- 172.21.15.90:0/155958775 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38200b4da0 con 0x7f3804007070 2024-03-24T01:25:33.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.777+0000 7f381e000640 1 -- 172.21.15.90:0/155958775 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3804007830 con 0x7f3804005bf0 2024-03-24T01:25:33.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.777+0000 7f381e000640 1 -- 172.21.15.90:0/155958775 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f380400e110 con 0x7f38200aeeb0 2024-03-24T01:25:33.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.778+0000 7f381f400640 1 --2- 172.21.15.90:0/155958775 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3804007070 0x7f380400dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:33.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.778+0000 7f381ea00640 1 --2- 172.21.15.90:0/155958775 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3804005bf0 0x7f3804006030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:33.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.778+0000 7f3816000640 1 --2- 172.21.15.90:0/155958775 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38200aeeb0 0x7f3804006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:33.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.778+0000 7f381f400640 1 -- 172.21.15.90:0/155958775 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38200aeeb0 msgr2=0x7f3804006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:33.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.778+0000 7f381f400640 1 --2- 172.21.15.90:0/155958775 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38200aeeb0 0x7f3804006a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:33.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.778+0000 7f381f400640 1 -- 172.21.15.90:0/155958775 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3804005bf0 msgr2=0x7f3804006030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:33.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.778+0000 7f381f400640 1 --2- 172.21.15.90:0/155958775 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3804005bf0 0x7f3804006030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:33.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.778+0000 7f381f400640 1 -- 172.21.15.90:0/155958775 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3804005730 con 0x7f3804007070 2024-03-24T01:25:33.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.778+0000 7f381ea00640 1 --2- 172.21.15.90:0/155958775 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3804005bf0 0x7f3804006030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:33.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.779+0000 7f381f400640 1 --2- 172.21.15.90:0/155958775 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3804007070 0x7f380400dbd0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f3818001e50 tx=0x7f3804007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50615b1f0dc68d02 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:33.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.779+0000 7f381e000640 1 -- 172.21.15.90:0/155958775 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3818001a70 con 0x7f3804007070 2024-03-24T01:25:33.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.779+0000 7f381e000640 1 -- 172.21.15.90:0/155958775 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3818003b70 con 0x7f3804007070 2024-03-24T01:25:33.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.779+0000 7f381e000640 1 -- 172.21.15.90:0/155958775 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38180050d0 con 0x7f3804007070 2024-03-24T01:25:33.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.780+0000 7f3825a00640 1 -- 172.21.15.90:0/155958775 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3804007070 msgr2=0x7f380400dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:33.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.780+0000 7f3825a00640 1 --2- 172.21.15.90:0/155958775 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3804007070 0x7f380400dbd0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f3818001e50 tx=0x7f3804007a40 comp rx=0 tx=0).stop 2024-03-24T01:25:33.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.780+0000 7f381f400640 1 -- 172.21.15.90:0/155958775 reap_dead start 2024-03-24T01:25:33.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.780+0000 7f3825a00640 1 -- 172.21.15.90:0/155958775 shutdown_connections 2024-03-24T01:25:33.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.780+0000 7f3825a00640 1 -- 172.21.15.90:0/155958775 >> 172.21.15.90:0/155958775 conn(0x7f38200b1770 msgr2=0x7f38200b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:33.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.780+0000 7f3825a00640 1 -- 172.21.15.90:0/155958775 shutdown_connections 2024-03-24T01:25:33.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.780+0000 7f3825a00640 1 -- 172.21.15.90:0/155958775 wait complete. 2024-03-24T01:25:33.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.782+0000 7f3825a00640 1 Processor -- start 2024-03-24T01:25:33.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.782+0000 7f3825a00640 1 -- start start 2024-03-24T01:25:33.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.783+0000 7f3825a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3804007070 0x7f38201434d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:33.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.783+0000 7f3825a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f382015d100 0x7f382015d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:33.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.783+0000 7f3825a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38201595e0 0x7f3820159a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:33.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.783+0000 7f3825a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382006e880 con 0x7f382015d100 2024-03-24T01:25:33.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.783+0000 7f3825a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38200af290 con 0x7f3804007070 2024-03-24T01:25:33.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.783+0000 7f3825a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f382006ea00 con 0x7f38201595e0 2024-03-24T01:25:33.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.784+0000 7f3816000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f382015d100 0x7f382015d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:33.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.784+0000 7f3816000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f382015d100 0x7f382015d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55114/0 (socket says 172.21.15.90:55114) 2024-03-24T01:25:33.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.784+0000 7f3816000640 1 -- 172.21.15.90:0/1958571545 learned_addr learned my addr 172.21.15.90:0/1958571545 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:33.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.784+0000 7f381ea00640 1 --2- 172.21.15.90:0/1958571545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3804007070 0x7f38201434d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:33.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.784+0000 7f3816000640 1 -- 172.21.15.90:0/1958571545 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38201595e0 msgr2=0x7f3820159a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:33.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.784+0000 7f3816000640 1 --2- 172.21.15.90:0/1958571545 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38201595e0 0x7f3820159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:33.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.784+0000 7f3816000640 1 -- 172.21.15.90:0/1958571545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3804007070 msgr2=0x7f38201434d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:33.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.784+0000 7f3816000640 1 --2- 172.21.15.90:0/1958571545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3804007070 0x7f38201434d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:33.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.784+0000 7f3816000640 1 -- 172.21.15.90:0/1958571545 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f382015a2d0 con 0x7f382015d100 2024-03-24T01:25:33.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.784+0000 7f3816000640 1 --2- 172.21.15.90:0/1958571545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f382015d100 0x7f382015d580 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f380800d730 tx=0x7f380800dbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:33.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.785+0000 7f3817400640 1 -- 172.21.15.90:0/1958571545 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3808019070 con 0x7f382015d100 2024-03-24T01:25:33.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.785+0000 7f3817400640 1 -- 172.21.15.90:0/1958571545 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38080074e0 con 0x7f382015d100 2024-03-24T01:25:33.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.785+0000 7f3817400640 1 -- 172.21.15.90:0/1958571545 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3808004a00 con 0x7f382015d100 2024-03-24T01:25:33.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.785+0000 7f3825a00640 1 -- 172.21.15.90:0/1958571545 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f382015a5b0 con 0x7f382015d100 2024-03-24T01:25:33.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.785+0000 7f3825a00640 1 -- 172.21.15.90:0/1958571545 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3820143fc0 con 0x7f382015d100 2024-03-24T01:25:33.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.786+0000 7f3825a00640 1 -- 172.21.15.90:0/1958571545 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38201441c0 con 0x7f382015d100 2024-03-24T01:25:33.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.788+0000 7f3817400640 1 -- 172.21.15.90:0/1958571545 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3808004ba0 con 0x7f382015d100 2024-03-24T01:25:33.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.788+0000 7f3817400640 1 --2- 172.21.15.90:0/1958571545 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f37fc06bbd0 0x7f37fc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:33.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.789+0000 7f3817400640 1 -- 172.21.15.90:0/1958571545 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f380808a7c0 con 0x7f382015d100 2024-03-24T01:25:33.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.789+0000 7f381ea00640 1 --2- 172.21.15.90:0/1958571545 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f37fc06bbd0 0x7f37fc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:33.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.790+0000 7f381ea00640 1 --2- 172.21.15.90:0/1958571545 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f37fc06bbd0 0x7f37fc06e090 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3810006fd0 tx=0x7f3810008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:33.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.793+0000 7f3817400640 1 -- 172.21.15.90:0/1958571545 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f38201441c0 con 0x7f382015d100 2024-03-24T01:25:33.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.987+0000 7f3825a00640 1 -- 172.21.15.90:0/1958571545 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f382004f400 con 0x7f382015d100 2024-03-24T01:25:33.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.987+0000 7f3817400640 1 -- 172.21.15.90:0/1958571545 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3808058f50 con 0x7f382015d100 2024-03-24T01:25:33.988 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:25:33.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.992+0000 7f3825a00640 1 -- 172.21.15.90:0/1958571545 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f37fc06bbd0 msgr2=0x7f37fc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:33.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.992+0000 7f3825a00640 1 --2- 172.21.15.90:0/1958571545 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f37fc06bbd0 0x7f37fc06e090 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3810006fd0 tx=0x7f3810008040 comp rx=0 tx=0).stop 2024-03-24T01:25:33.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.992+0000 7f3825a00640 1 -- 172.21.15.90:0/1958571545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f382015d100 msgr2=0x7f382015d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:33.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.992+0000 7f3825a00640 1 --2- 172.21.15.90:0/1958571545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f382015d100 0x7f382015d580 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f380800d730 tx=0x7f380800dbf0 comp rx=0 tx=0).stop 2024-03-24T01:25:33.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.992+0000 7f3825a00640 1 -- 172.21.15.90:0/1958571545 shutdown_connections 2024-03-24T01:25:33.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.992+0000 7f3825a00640 1 --2- 172.21.15.90:0/1958571545 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f37fc06bbd0 0x7f37fc06e090 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:33.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.992+0000 7f3825a00640 1 --2- 172.21.15.90:0/1958571545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f382015d100 0x7f382015d580 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:33.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.992+0000 7f3825a00640 1 --2- 172.21.15.90:0/1958571545 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38201595e0 0x7f3820159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:33.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.992+0000 7f3825a00640 1 --2- 172.21.15.90:0/1958571545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3804007070 0x7f38201434d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:33.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.992+0000 7f3825a00640 1 -- 172.21.15.90:0/1958571545 >> 172.21.15.90:0/1958571545 conn(0x7f38200b1770 msgr2=0x7f38200b0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:33.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.992+0000 7f3825a00640 1 -- 172.21.15.90:0/1958571545 shutdown_connections 2024-03-24T01:25:33.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:33.992+0000 7f3825a00640 1 -- 172.21.15.90:0/1958571545 wait complete. 2024-03-24T01:25:34.011 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T01:25:34.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.216+0000 7f807ac00640 1 Processor -- start 2024-03-24T01:25:34.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.217+0000 7f807ac00640 1 -- start start 2024-03-24T01:25:34.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.218+0000 7f807ac00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f807415e0e0 0x7f807406cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:34.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.218+0000 7f807ac00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f80740b2080 0x7f80740b24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:34.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.218+0000 7f807ac00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80740cfb60 0x7f80740cfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:34.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.218+0000 7f807ac00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80740b1350 con 0x7f80740b2080 2024-03-24T01:25:34.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.218+0000 7f807ac00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80740d0530 con 0x7f807415e0e0 2024-03-24T01:25:34.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.218+0000 7f807ac00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f807415e4c0 con 0x7f80740cfb60 2024-03-24T01:25:34.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.218+0000 7f8073400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f80740b2080 0x7f80740b24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:34.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.218+0000 7f8073400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f80740b2080 0x7f80740b24e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55120/0 (socket says 172.21.15.90:55120) 2024-03-24T01:25:34.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.218+0000 7f8073400640 1 -- 172.21.15.90:0/1496957730 learned_addr learned my addr 172.21.15.90:0/1496957730 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:34.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.219+0000 7f8073e00640 1 --2- 172.21.15.90:0/1496957730 >> v2:172.21.15.120:3300/0 conn(0x7f807415e0e0 0x7f807406cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:34.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.219+0000 7f8078c00640 1 --2- 172.21.15.90:0/1496957730 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80740cfb60 0x7f80740cfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:34.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.219+0000 7f8073400640 1 -- 172.21.15.90:0/1496957730 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80740cfb60 msgr2=0x7f80740cfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:34.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.219+0000 7f8073400640 1 --2- 172.21.15.90:0/1496957730 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80740cfb60 0x7f80740cfff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:34.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.219+0000 7f8073400640 1 -- 172.21.15.90:0/1496957730 >> v2:172.21.15.120:3300/0 conn(0x7f807415e0e0 msgr2=0x7f807406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:34.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.219+0000 7f8073400640 1 --2- 172.21.15.90:0/1496957730 >> v2:172.21.15.120:3300/0 conn(0x7f807415e0e0 0x7f807406cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:34.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.219+0000 7f8073400640 1 -- 172.21.15.90:0/1496957730 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f807406d330 con 0x7f80740b2080 2024-03-24T01:25:34.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.220+0000 7f8073400640 1 --2- 172.21.15.90:0/1496957730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80740b2080 0x7f80740b24e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f8074072d50 tx=0x7f805c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bc315c3ba88a952 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:34.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.220+0000 7f8072a00640 1 -- 172.21.15.90:0/1496957730 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805c02ca70 con 0x7f80740b2080 2024-03-24T01:25:34.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.220+0000 7f8072a00640 1 -- 172.21.15.90:0/1496957730 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f805c02cc10 con 0x7f80740b2080 2024-03-24T01:25:34.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.220+0000 7f8072a00640 1 -- 172.21.15.90:0/1496957730 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805c0343f0 con 0x7f80740b2080 2024-03-24T01:25:34.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.221+0000 7f807ac00640 1 -- 172.21.15.90:0/1496957730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80740b2080 msgr2=0x7f80740b24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:34.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.221+0000 7f807ac00640 1 --2- 172.21.15.90:0/1496957730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80740b2080 0x7f80740b24e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f8074072d50 tx=0x7f805c02bed0 comp rx=0 tx=0).stop 2024-03-24T01:25:34.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.221+0000 7f807ac00640 1 -- 172.21.15.90:0/1496957730 shutdown_connections 2024-03-24T01:25:34.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.221+0000 7f807ac00640 1 --2- 172.21.15.90:0/1496957730 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80740cfb60 0x7f80740cfff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:34.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.221+0000 7f807ac00640 1 --2- 172.21.15.90:0/1496957730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80740b2080 0x7f80740b24e0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:34.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.221+0000 7f807ac00640 1 --2- 172.21.15.90:0/1496957730 >> v2:172.21.15.120:3300/0 conn(0x7f807415e0e0 0x7f807406cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:34.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.221+0000 7f807ac00640 1 -- 172.21.15.90:0/1496957730 >> 172.21.15.90:0/1496957730 conn(0x7f80740b34e0 msgr2=0x7f80740b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:34.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.221+0000 7f807ac00640 1 -- 172.21.15.90:0/1496957730 shutdown_connections 2024-03-24T01:25:34.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.221+0000 7f807ac00640 1 -- 172.21.15.90:0/1496957730 wait complete. 2024-03-24T01:25:34.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.223+0000 7f807ac00640 1 Processor -- start 2024-03-24T01:25:34.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.223+0000 7f807ac00640 1 -- start start 2024-03-24T01:25:34.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.224+0000 7f807ac00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80740b2080 0x7f8074149170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:34.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.224+0000 7f807ac00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80740cfb60 0x7f8074160c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:34.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.224+0000 7f807ac00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f807415e0e0 0x7f807406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:34.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.224+0000 7f807ac00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80740b1b60 con 0x7f80740cfb60 2024-03-24T01:25:34.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.224+0000 7f807ac00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80740b1880 con 0x7f807415e0e0 2024-03-24T01:25:34.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.224+0000 7f807ac00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8074161be0 con 0x7f80740b2080 2024-03-24T01:25:34.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.224+0000 7f8073400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80740cfb60 0x7f8074160c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:34.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.224+0000 7f8073400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80740cfb60 0x7f8074160c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55134/0 (socket says 172.21.15.90:55134) 2024-03-24T01:25:34.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.224+0000 7f8073400640 1 -- 172.21.15.90:0/3339156102 learned_addr learned my addr 172.21.15.90:0/3339156102 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:34.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.225+0000 7f8078c00640 1 --2- 172.21.15.90:0/3339156102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f807415e0e0 0x7f807406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:34.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.225+0000 7f8073400640 1 -- 172.21.15.90:0/3339156102 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80740b2080 msgr2=0x7f8074149170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:34.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.225+0000 7f8073400640 1 --2- 172.21.15.90:0/3339156102 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80740b2080 0x7f8074149170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:34.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.225+0000 7f8073400640 1 -- 172.21.15.90:0/3339156102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f807415e0e0 msgr2=0x7f807406d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:34.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.225+0000 7f8073400640 1 --2- 172.21.15.90:0/3339156102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f807415e0e0 0x7f807406d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:34.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.225+0000 7f8073400640 1 -- 172.21.15.90:0/3339156102 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f807414bbc0 con 0x7f80740cfb60 2024-03-24T01:25:34.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.225+0000 7f8073400640 1 --2- 172.21.15.90:0/3339156102 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80740cfb60 0x7f8074160c60 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f805c000f80 tx=0x7f805c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:34.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.225+0000 7f806be00640 1 -- 172.21.15.90:0/3339156102 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805c043020 con 0x7f80740cfb60 2024-03-24T01:25:34.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.225+0000 7f806be00640 1 -- 172.21.15.90:0/3339156102 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f805c03e450 con 0x7f80740cfb60 2024-03-24T01:25:34.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.225+0000 7f806be00640 1 -- 172.21.15.90:0/3339156102 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805c009510 con 0x7f80740cfb60 2024-03-24T01:25:34.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.225+0000 7f807ac00640 1 -- 172.21.15.90:0/3339156102 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f807414bea0 con 0x7f80740cfb60 2024-03-24T01:25:34.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.225+0000 7f807ac00640 1 -- 172.21.15.90:0/3339156102 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f807414c460 con 0x7f80740cfb60 2024-03-24T01:25:34.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.227+0000 7f807ac00640 1 -- 172.21.15.90:0/3339156102 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f807404f400 con 0x7f80740cfb60 2024-03-24T01:25:34.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.227+0000 7f806be00640 1 -- 172.21.15.90:0/3339156102 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f805c046020 con 0x7f80740cfb60 2024-03-24T01:25:34.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.228+0000 7f806be00640 1 --2- 172.21.15.90:0/3339156102 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f805006bbd0 0x7f805006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:34.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.228+0000 7f806be00640 1 -- 172.21.15.90:0/3339156102 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f805c0ad720 con 0x7f80740cfb60 2024-03-24T01:25:34.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.229+0000 7f8073e00640 1 --2- 172.21.15.90:0/3339156102 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f805006bbd0 0x7f805006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:34.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.230+0000 7f8073e00640 1 --2- 172.21.15.90:0/3339156102 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f805006bbd0 0x7f805006e090 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8064006fd0 tx=0x7f8064008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:34.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.233+0000 7f806be00640 1 -- 172.21.15.90:0/3339156102 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f805c04d030 con 0x7f80740cfb60 2024-03-24T01:25:34.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.429+0000 7f807ac00640 1 -- 172.21.15.90:0/3339156102 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f80740b4310 con 0x7f80740cfb60 2024-03-24T01:25:34.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.430+0000 7f806be00640 1 -- 172.21.15.90:0/3339156102 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f805c07bdf0 con 0x7f80740cfb60 2024-03-24T01:25:34.430 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-24T01:25:34.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.432+0000 7f807ac00640 1 -- 172.21.15.90:0/3339156102 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f805006bbd0 msgr2=0x7f805006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:34.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.432+0000 7f807ac00640 1 --2- 172.21.15.90:0/3339156102 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f805006bbd0 0x7f805006e090 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8064006fd0 tx=0x7f8064008040 comp rx=0 tx=0).stop 2024-03-24T01:25:34.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.432+0000 7f807ac00640 1 -- 172.21.15.90:0/3339156102 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80740cfb60 msgr2=0x7f8074160c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:34.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.432+0000 7f807ac00640 1 --2- 172.21.15.90:0/3339156102 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80740cfb60 0x7f8074160c60 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f805c000f80 tx=0x7f805c004a40 comp rx=0 tx=0).stop 2024-03-24T01:25:34.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.432+0000 7f807ac00640 1 -- 172.21.15.90:0/3339156102 shutdown_connections 2024-03-24T01:25:34.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.432+0000 7f807ac00640 1 --2- 172.21.15.90:0/3339156102 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f805006bbd0 0x7f805006e090 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:34.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.432+0000 7f807ac00640 1 --2- 172.21.15.90:0/3339156102 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80740cfb60 0x7f8074160c60 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:34.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.432+0000 7f807ac00640 1 --2- 172.21.15.90:0/3339156102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f807415e0e0 0x7f807406d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:34.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.432+0000 7f807ac00640 1 --2- 172.21.15.90:0/3339156102 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f80740b2080 0x7f8074149170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:34.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.432+0000 7f807ac00640 1 -- 172.21.15.90:0/3339156102 >> 172.21.15.90:0/3339156102 conn(0x7f80740b34e0 msgr2=0x7f807406cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:34.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.433+0000 7f807ac00640 1 -- 172.21.15.90:0/3339156102 shutdown_connections 2024-03-24T01:25:34.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.433+0000 7f807ac00640 1 -- 172.21.15.90:0/3339156102 wait complete. 2024-03-24T01:25:34.452 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T01:25:34.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.666+0000 7f3a77e00640 1 Processor -- start 2024-03-24T01:25:34.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.666+0000 7f3a77e00640 1 -- start start 2024-03-24T01:25:34.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.668+0000 7f3a77e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3a78145170 0x7f3a780cf810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:34.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.668+0000 7f3a77e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a780cf190 0x7f3a780cfd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:34.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.668+0000 7f3a77e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a7806d990 con 0x7f3a7806ea30 2024-03-24T01:25:34.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.668+0000 7f3a77e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a7806dc70 con 0x7f3a78145170 2024-03-24T01:25:34.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.668+0000 7f3a77e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7806df50 con 0x7f3a780cf190 2024-03-24T01:25:34.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.668+0000 7f3a76a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3a78145170 0x7f3a780cf810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:34.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.668+0000 7f3a76000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f3a7806ea30 0x7f3a780cea80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:32798/0 (socket says 172.21.15.90:32798) 2024-03-24T01:25:34.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.668+0000 7f3a76000640 1 -- 172.21.15.90:0/1993237662 learned_addr learned my addr 172.21.15.90:0/1993237662 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:34.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.668+0000 7f3a77400640 1 --2- 172.21.15.90:0/1993237662 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a780cf190 0x7f3a780cfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:34.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.669+0000 7f3a76a00640 1 -- 172.21.15.90:0/1993237662 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a780cf190 msgr2=0x7f3a780cfd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:34.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.669+0000 7f3a76a00640 1 --2- 172.21.15.90:0/1993237662 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a780cf190 0x7f3a780cfd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:34.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.669+0000 7f3a76a00640 1 -- 172.21.15.90:0/1993237662 >> v1:172.21.15.90:6789/0 conn(0x7f3a7806ea30 legacy=0x7f3a780cea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:34.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.669+0000 7f3a76a00640 1 -- 172.21.15.90:0/1993237662 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a7806e230 con 0x7f3a78145170 2024-03-24T01:25:34.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.669+0000 7f3a75600640 1 -- 172.21.15.90:0/1993237662 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1418479820 0 0) 0x7f3a7806d990 con 0x7f3a7806ea30 2024-03-24T01:25:34.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.670+0000 7f3a76a00640 1 --2- 172.21.15.90:0/1993237662 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a78145170 0x7f3a780cf810 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3a680097a0 tx=0x7f3a6802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a351d50295aa8ef0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:34.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.670+0000 7f3a75600640 1 -- 172.21.15.90:0/1993237662 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a68035020 con 0x7f3a78145170 2024-03-24T01:25:34.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.670+0000 7f3a75600640 1 -- 172.21.15.90:0/1993237662 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a68031810 con 0x7f3a78145170 2024-03-24T01:25:34.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.670+0000 7f3a75600640 1 -- 172.21.15.90:0/1993237662 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a68031be0 con 0x7f3a78145170 2024-03-24T01:25:34.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.671+0000 7f3a77e00640 1 -- 172.21.15.90:0/1993237662 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a78145170 msgr2=0x7f3a780cf810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:34.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.671+0000 7f3a77e00640 1 --2- 172.21.15.90:0/1993237662 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a78145170 0x7f3a780cf810 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3a680097a0 tx=0x7f3a6802b8b0 comp rx=0 tx=0).stop 2024-03-24T01:25:34.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.671+0000 7f3a77e00640 1 -- 172.21.15.90:0/1993237662 shutdown_connections 2024-03-24T01:25:34.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.671+0000 7f3a77e00640 1 --2- 172.21.15.90:0/1993237662 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a780cf190 0x7f3a780cfd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:34.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.671+0000 7f3a77e00640 1 --2- 172.21.15.90:0/1993237662 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a78145170 0x7f3a780cf810 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:34.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.671+0000 7f3a77e00640 1 -- 172.21.15.90:0/1993237662 >> 172.21.15.90:0/1993237662 conn(0x7f3a7806b0c0 msgr2=0x7f3a7806b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:34.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.671+0000 7f3a77e00640 1 -- 172.21.15.90:0/1993237662 shutdown_connections 2024-03-24T01:25:34.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.671+0000 7f3a77e00640 1 -- 172.21.15.90:0/1993237662 wait complete. 2024-03-24T01:25:34.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.672+0000 7f3a77e00640 1 Processor -- start 2024-03-24T01:25:34.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.673+0000 7f3a77e00640 1 -- start start 2024-03-24T01:25:34.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.673+0000 7f3a77e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a7806ea30 0x7f3a7813c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:34.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.674+0000 7f3a77e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a780cf190 0x7f3a78140e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:34.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.674+0000 7f3a77e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a78145170 0x7f3a78152170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:34.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.674+0000 7f3a77e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a781456d0 con 0x7f3a78145170 2024-03-24T01:25:34.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.674+0000 7f3a77e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a78145550 con 0x7f3a780cf190 2024-03-24T01:25:34.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.674+0000 7f3a77e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7815c0a0 con 0x7f3a7806ea30 2024-03-24T01:25:34.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.674+0000 7f3a77400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a78145170 0x7f3a78152170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:34.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.674+0000 7f3a77400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a78145170 0x7f3a78152170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55146/0 (socket says 172.21.15.90:55146) 2024-03-24T01:25:34.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.674+0000 7f3a77400640 1 -- 172.21.15.90:0/1340237319 learned_addr learned my addr 172.21.15.90:0/1340237319 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:34.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.674+0000 7f3a76a00640 1 --2- 172.21.15.90:0/1340237319 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a7806ea30 0x7f3a7813c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:34.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.674+0000 7f3a76000640 1 --2- 172.21.15.90:0/1340237319 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a780cf190 0x7f3a78140e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:34.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.674+0000 7f3a77400640 1 -- 172.21.15.90:0/1340237319 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a7806ea30 msgr2=0x7f3a7813c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:34.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.674+0000 7f3a77400640 1 --2- 172.21.15.90:0/1340237319 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a7806ea30 0x7f3a7813c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:34.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.674+0000 7f3a77400640 1 -- 172.21.15.90:0/1340237319 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a780cf190 msgr2=0x7f3a78140e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:34.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.674+0000 7f3a77400640 1 --2- 172.21.15.90:0/1340237319 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a780cf190 0x7f3a78140e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:34.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.674+0000 7f3a77400640 1 -- 172.21.15.90:0/1340237319 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a7813ccd0 con 0x7f3a78145170 2024-03-24T01:25:34.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.674+0000 7f3a76000640 1 --2- 172.21.15.90:0/1340237319 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a780cf190 0x7f3a78140e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:34.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.675+0000 7f3a77400640 1 --2- 172.21.15.90:0/1340237319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a78145170 0x7f3a78152170 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f3a70007e70 tx=0x7f3a7000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:34.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.675+0000 7f3a6ea00640 1 -- 172.21.15.90:0/1340237319 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a70014620 con 0x7f3a78145170 2024-03-24T01:25:34.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.675+0000 7f3a6ea00640 1 -- 172.21.15.90:0/1340237319 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a70019070 con 0x7f3a78145170 2024-03-24T01:25:34.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.675+0000 7f3a6ea00640 1 -- 172.21.15.90:0/1340237319 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a70014a20 con 0x7f3a78145170 2024-03-24T01:25:34.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.675+0000 7f3a77e00640 1 -- 172.21.15.90:0/1340237319 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a7813cfb0 con 0x7f3a78145170 2024-03-24T01:25:34.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.675+0000 7f3a77e00640 1 -- 172.21.15.90:0/1340237319 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a7813d570 con 0x7f3a78145170 2024-03-24T01:25:34.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.676+0000 7f3a77e00640 1 -- 172.21.15.90:0/1340237319 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a7804f400 con 0x7f3a78145170 2024-03-24T01:25:34.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.677+0000 7f3a6ea00640 1 -- 172.21.15.90:0/1340237319 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3a70014bc0 con 0x7f3a78145170 2024-03-24T01:25:34.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.678+0000 7f3a6ea00640 1 --2- 172.21.15.90:0/1340237319 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3a5806bbd0 0x7f3a5806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:34.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.678+0000 7f3a6ea00640 1 -- 172.21.15.90:0/1340237319 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f3a7008b8a0 con 0x7f3a78145170 2024-03-24T01:25:34.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.678+0000 7f3a76a00640 1 --2- 172.21.15.90:0/1340237319 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3a5806bbd0 0x7f3a5806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:34.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.679+0000 7f3a76a00640 1 --2- 172.21.15.90:0/1340237319 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3a5806bbd0 0x7f3a5806e090 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3a68036000 tx=0x7f3a6803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:34.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.682+0000 7f3a6ea00640 1 -- 172.21.15.90:0/1340237319 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3a70059e40 con 0x7f3a78145170 2024-03-24T01:25:34.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.879+0000 7f3a77e00640 1 -- 172.21.15.90:0/1340237319 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f3a780723b0 con 0x7f3a78145170 2024-03-24T01:25:34.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.880+0000 7f3a6ea00640 1 -- 172.21.15.90:0/1340237319 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f3a70059740 con 0x7f3a78145170 2024-03-24T01:25:34.880 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:25:34.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.883+0000 7f3a77e00640 1 -- 172.21.15.90:0/1340237319 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3a5806bbd0 msgr2=0x7f3a5806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:34.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.883+0000 7f3a77e00640 1 --2- 172.21.15.90:0/1340237319 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3a5806bbd0 0x7f3a5806e090 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3a68036000 tx=0x7f3a6803c040 comp rx=0 tx=0).stop 2024-03-24T01:25:34.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.883+0000 7f3a77e00640 1 -- 172.21.15.90:0/1340237319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a78145170 msgr2=0x7f3a78152170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:34.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.883+0000 7f3a77e00640 1 --2- 172.21.15.90:0/1340237319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a78145170 0x7f3a78152170 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f3a70007e70 tx=0x7f3a7000e4b0 comp rx=0 tx=0).stop 2024-03-24T01:25:34.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.883+0000 7f3a77e00640 1 -- 172.21.15.90:0/1340237319 shutdown_connections 2024-03-24T01:25:34.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.883+0000 7f3a77e00640 1 --2- 172.21.15.90:0/1340237319 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3a5806bbd0 0x7f3a5806e090 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:34.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.883+0000 7f3a77e00640 1 --2- 172.21.15.90:0/1340237319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3a78145170 0x7f3a78152170 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:34.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.883+0000 7f3a77e00640 1 --2- 172.21.15.90:0/1340237319 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a780cf190 0x7f3a78140e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:34.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.883+0000 7f3a77e00640 1 --2- 172.21.15.90:0/1340237319 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3a7806ea30 0x7f3a7813c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:34.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.883+0000 7f3a77e00640 1 -- 172.21.15.90:0/1340237319 >> 172.21.15.90:0/1340237319 conn(0x7f3a7806b0c0 msgr2=0x7f3a780cea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:34.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.883+0000 7f3a77e00640 1 -- 172.21.15.90:0/1340237319 shutdown_connections 2024-03-24T01:25:34.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:34.883+0000 7f3a77e00640 1 -- 172.21.15.90:0/1340237319 wait complete. 2024-03-24T01:25:34.906 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T01:25:35.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.121+0000 7f8318a00640 1 Processor -- start 2024-03-24T01:25:35.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.121+0000 7f8318a00640 1 -- start start 2024-03-24T01:25:35.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.123+0000 7f8318a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f831415da30 0x7f831415de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:35.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.123+0000 7f8318a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f83140acea0 0x7f83140aea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:35.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.123+0000 7f8318a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f831415e550 con 0x7f831406b0c0 2024-03-24T01:25:35.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.123+0000 7f8318a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83140b2c70 con 0x7f831415da30 2024-03-24T01:25:35.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.123+0000 7f8318a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f831415f160 con 0x7f83140acea0 2024-03-24T01:25:35.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.123+0000 7f8311e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f831415da30 0x7f831415de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:35.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.123+0000 7f8311e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f831415da30 0x7f831415de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55156/0 (socket says 172.21.15.90:55156) 2024-03-24T01:25:35.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.123+0000 7f8311e00640 1 -- 172.21.15.90:0/3691449205 learned_addr learned my addr 172.21.15.90:0/3691449205 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:35.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.124+0000 7f8311400640 1 --2- 172.21.15.90:0/3691449205 >> v2:172.21.15.120:3300/0 conn(0x7f83140acea0 0x7f83140aea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:35.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.124+0000 7f8311e00640 1 -- 172.21.15.90:0/3691449205 >> v2:172.21.15.120:3300/0 conn(0x7f83140acea0 msgr2=0x7f83140aea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:35.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.124+0000 7f8311e00640 1 --2- 172.21.15.90:0/3691449205 >> v2:172.21.15.120:3300/0 conn(0x7f83140acea0 0x7f83140aea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:35.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.124+0000 7f8311e00640 1 -- 172.21.15.90:0/3691449205 >> v1:172.21.15.120:6789/0 conn(0x7f831406b0c0 legacy=0x7f8314161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:35.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.124+0000 7f8311e00640 1 -- 172.21.15.90:0/3691449205 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f831415e830 con 0x7f831415da30 2024-03-24T01:25:35.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.124+0000 7f8311400640 1 --2- 172.21.15.90:0/3691449205 >> v2:172.21.15.120:3300/0 conn(0x7f83140acea0 0x7f83140aea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:35.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.125+0000 7f8311e00640 1 --2- 172.21.15.90:0/3691449205 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f831415da30 0x7f831415de10 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f831406e220 tx=0x7f830402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e829595244c2542 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:35.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.126+0000 7f8310a00640 1 -- 172.21.15.90:0/3691449205 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8304035020 con 0x7f831415da30 2024-03-24T01:25:35.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.126+0000 7f8310a00640 1 -- 172.21.15.90:0/3691449205 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8304031870 con 0x7f831415da30 2024-03-24T01:25:35.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.126+0000 7f8310a00640 1 -- 172.21.15.90:0/3691449205 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8304031c40 con 0x7f831415da30 2024-03-24T01:25:35.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.126+0000 7f8318a00640 1 -- 172.21.15.90:0/3691449205 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f831415da30 msgr2=0x7f831415de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:35.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.126+0000 7f8318a00640 1 --2- 172.21.15.90:0/3691449205 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f831415da30 0x7f831415de10 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f831406e220 tx=0x7f830402b910 comp rx=0 tx=0).stop 2024-03-24T01:25:35.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.127+0000 7f8318a00640 1 -- 172.21.15.90:0/3691449205 shutdown_connections 2024-03-24T01:25:35.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.127+0000 7f8318a00640 1 --2- 172.21.15.90:0/3691449205 >> v2:172.21.15.120:3300/0 conn(0x7f83140acea0 0x7f83140aea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:35.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.127+0000 7f8318a00640 1 --2- 172.21.15.90:0/3691449205 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f831415da30 0x7f831415de10 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:35.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.127+0000 7f8318a00640 1 -- 172.21.15.90:0/3691449205 >> 172.21.15.90:0/3691449205 conn(0x7f83140b1770 msgr2=0x7f83140b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:35.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.127+0000 7f8318a00640 1 -- 172.21.15.90:0/3691449205 shutdown_connections 2024-03-24T01:25:35.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.127+0000 7f8318a00640 1 -- 172.21.15.90:0/3691449205 wait complete. 2024-03-24T01:25:35.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.128+0000 7f8318a00640 1 Processor -- start 2024-03-24T01:25:35.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.128+0000 7f8318a00640 1 -- start start 2024-03-24T01:25:35.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.129+0000 7f8318a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f831406b0c0 0x7f831414ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:35.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.129+0000 7f8318a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83140acea0 0x7f83140b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:35.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.129+0000 7f8318a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f831415da30 0x7f83140af3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:35.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.129+0000 7f8318a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83140ad300 con 0x7f831415da30 2024-03-24T01:25:35.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.129+0000 7f8318a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f831415edf0 con 0x7f83140acea0 2024-03-24T01:25:35.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.129+0000 7f8318a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83140aef70 con 0x7f831406b0c0 2024-03-24T01:25:35.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.130+0000 7f8312800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f831415da30 0x7f83140af3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:35.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.130+0000 7f8312800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f831415da30 0x7f83140af3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55162/0 (socket says 172.21.15.90:55162) 2024-03-24T01:25:35.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.130+0000 7f8312800640 1 -- 172.21.15.90:0/1292766861 learned_addr learned my addr 172.21.15.90:0/1292766861 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:35.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.130+0000 7f8311e00640 1 --2- 172.21.15.90:0/1292766861 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f831406b0c0 0x7f831414ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:35.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.130+0000 7f8311400640 1 --2- 172.21.15.90:0/1292766861 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83140acea0 0x7f83140b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:35.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.130+0000 7f8312800640 1 -- 172.21.15.90:0/1292766861 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f831406b0c0 msgr2=0x7f831414ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:35.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.130+0000 7f8312800640 1 --2- 172.21.15.90:0/1292766861 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f831406b0c0 0x7f831414ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:35.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.130+0000 7f8312800640 1 -- 172.21.15.90:0/1292766861 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83140acea0 msgr2=0x7f83140b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:35.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.130+0000 7f8312800640 1 --2- 172.21.15.90:0/1292766861 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83140acea0 0x7f83140b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:35.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.130+0000 7f8312800640 1 -- 172.21.15.90:0/1292766861 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83140afdb0 con 0x7f831415da30 2024-03-24T01:25:35.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.130+0000 7f8312800640 1 --2- 172.21.15.90:0/1292766861 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f831415da30 0x7f83140af3d0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f830c00c780 tx=0x7f830c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:35.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.131+0000 7f830a000640 1 -- 172.21.15.90:0/1292766861 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f830c004b60 con 0x7f831415da30 2024-03-24T01:25:35.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.131+0000 7f830a000640 1 -- 172.21.15.90:0/1292766861 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f830c004d00 con 0x7f831415da30 2024-03-24T01:25:35.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.131+0000 7f830a000640 1 -- 172.21.15.90:0/1292766861 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f830c00aa90 con 0x7f831415da30 2024-03-24T01:25:35.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.131+0000 7f8318a00640 1 -- 172.21.15.90:0/1292766861 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83140b0090 con 0x7f831415da30 2024-03-24T01:25:35.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.131+0000 7f8318a00640 1 -- 172.21.15.90:0/1292766861 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83140b0650 con 0x7f831415da30 2024-03-24T01:25:35.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.132+0000 7f8318a00640 1 -- 172.21.15.90:0/1292766861 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f831404f400 con 0x7f831415da30 2024-03-24T01:25:35.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.133+0000 7f830a000640 1 -- 172.21.15.90:0/1292766861 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f830c00ac30 con 0x7f831415da30 2024-03-24T01:25:35.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.134+0000 7f830a000640 1 --2- 172.21.15.90:0/1292766861 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f82f006bbd0 0x7f82f006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:35.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.134+0000 7f830a000640 1 -- 172.21.15.90:0/1292766861 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f830c08b080 con 0x7f831415da30 2024-03-24T01:25:35.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.135+0000 7f8311e00640 1 --2- 172.21.15.90:0/1292766861 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f82f006bbd0 0x7f82f006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:35.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.136+0000 7f8311e00640 1 --2- 172.21.15.90:0/1292766861 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f82f006bbd0 0x7f82f006e090 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8304036000 tx=0x7f8304008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:35.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.138+0000 7f830a000640 1 -- 172.21.15.90:0/1292766861 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f830c020070 con 0x7f831415da30 2024-03-24T01:25:35.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.352+0000 7f8318a00640 1 -- 172.21.15.90:0/1292766861 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f83140684f0 con 0x7f831415da30 2024-03-24T01:25:35.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.352+0000 7f830a000640 1 -- 172.21.15.90:0/1292766861 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f830c0597e0 con 0x7f831415da30 2024-03-24T01:25:35.353 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-24T01:25:35.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.355+0000 7f8318a00640 1 -- 172.21.15.90:0/1292766861 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f82f006bbd0 msgr2=0x7f82f006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:35.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.355+0000 7f8318a00640 1 --2- 172.21.15.90:0/1292766861 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f82f006bbd0 0x7f82f006e090 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8304036000 tx=0x7f8304008030 comp rx=0 tx=0).stop 2024-03-24T01:25:35.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.355+0000 7f8318a00640 1 -- 172.21.15.90:0/1292766861 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f831415da30 msgr2=0x7f83140af3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:35.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.355+0000 7f8318a00640 1 --2- 172.21.15.90:0/1292766861 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f831415da30 0x7f83140af3d0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f830c00c780 tx=0x7f830c00cc40 comp rx=0 tx=0).stop 2024-03-24T01:25:35.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.356+0000 7f8318a00640 1 -- 172.21.15.90:0/1292766861 shutdown_connections 2024-03-24T01:25:35.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.356+0000 7f8318a00640 1 --2- 172.21.15.90:0/1292766861 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f82f006bbd0 0x7f82f006e090 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:35.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.356+0000 7f8318a00640 1 --2- 172.21.15.90:0/1292766861 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f831415da30 0x7f83140af3d0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:35.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.356+0000 7f8318a00640 1 --2- 172.21.15.90:0/1292766861 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83140acea0 0x7f83140b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:35.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.356+0000 7f8318a00640 1 --2- 172.21.15.90:0/1292766861 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f831406b0c0 0x7f831414ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:35.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.356+0000 7f8318a00640 1 -- 172.21.15.90:0/1292766861 >> 172.21.15.90:0/1292766861 conn(0x7f83140b1770 msgr2=0x7f831415fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:35.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.356+0000 7f8318a00640 1 -- 172.21.15.90:0/1292766861 shutdown_connections 2024-03-24T01:25:35.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.356+0000 7f8318a00640 1 -- 172.21.15.90:0/1292766861 wait complete. 2024-03-24T01:25:35.375 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi090.front.sepia.ceph.com... 2024-03-24T01:25:35.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:35.376 DEBUG:teuthology.orchestra.run.smithi090:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-03-24T01:25:35.467 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:25:35.467 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-03-24T01:25:35.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:35.467 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-24T01:25:35.519 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:25:35.519 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:25:35.575 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:25:35.575 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:25:35.622 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:25:35.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.832+0000 7f3e4e200640 1 Processor -- start 2024-03-24T01:25:35.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.832+0000 7f3e4e200640 1 -- start start 2024-03-24T01:25:35.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.833+0000 7f3e4e200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e4815dd50 0x7f3e4806e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:35.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.833+0000 7f3e4e200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e480acf50 con 0x7f3e4806b7d0 2024-03-24T01:25:35.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.834+0000 7f3e4e200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4815e130 con 0x7f3e480b3d60 2024-03-24T01:25:35.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.834+0000 7f3e4e200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e480ad280 con 0x7f3e4815dd50 2024-03-24T01:25:35.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.834+0000 7f3e47e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f3e4806b7d0 0x7f3e480ad710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:32810/0 (socket says 172.21.15.90:32810) 2024-03-24T01:25:35.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.834+0000 7f3e47e00640 1 -- 172.21.15.90:0/1956388837 learned_addr learned my addr 172.21.15.90:0/1956388837 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:35.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.834+0000 7f3e47400640 1 --2- 172.21.15.90:0/1956388837 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e4815dd50 0x7f3e4806e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:35.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.834+0000 7f3e46a00640 1 -- 172.21.15.90:0/1956388837 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3822766783 0 0) 0x7f3e480acf50 con 0x7f3e4806b7d0 2024-03-24T01:25:35.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.834+0000 7f3e46a00640 1 -- 172.21.15.90:0/1956388837 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e2c003520 con 0x7f3e4806b7d0 2024-03-24T01:25:35.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.835+0000 7f3e46a00640 1 -- 172.21.15.90:0/1956388837 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 563991351 0 0) 0x7f3e4815e130 con 0x7f3e480b3d60 2024-03-24T01:25:35.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.835+0000 7f3e46a00640 1 -- 172.21.15.90:0/1956388837 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e480acf50 con 0x7f3e480b3d60 2024-03-24T01:25:35.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.835+0000 7f3e47400640 1 -- 172.21.15.90:0/1956388837 >> v1:172.21.15.120:6789/0 conn(0x7f3e480b3d60 legacy=0x7f3e4806b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:35.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.835+0000 7f3e46a00640 1 -- 172.21.15.90:0/1956388837 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2027068402 0 0) 0x7f3e2c003520 con 0x7f3e4806b7d0 2024-03-24T01:25:35.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.835+0000 7f3e47400640 1 -- 172.21.15.90:0/1956388837 >> v1:172.21.15.90:6789/0 conn(0x7f3e4806b7d0 legacy=0x7f3e480ad710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:35.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.835+0000 7f3e47400640 1 -- 172.21.15.90:0/1956388837 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e48162000 con 0x7f3e4815dd50 2024-03-24T01:25:35.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.836+0000 7f3e47400640 1 --2- 172.21.15.90:0/1956388837 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e4815dd50 0x7f3e4806e020 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3e380097d0 tx=0x7f3e3802b6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4197110c43655b58 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:35.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.836+0000 7f3e46a00640 1 -- 172.21.15.90:0/1956388837 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e38037070 con 0x7f3e4815dd50 2024-03-24T01:25:35.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.836+0000 7f3e46a00640 1 -- 172.21.15.90:0/1956388837 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e38030510 con 0x7f3e4815dd50 2024-03-24T01:25:35.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.836+0000 7f3e46a00640 1 -- 172.21.15.90:0/1956388837 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e400019f0 con 0x7f3e4815dd50 2024-03-24T01:25:35.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.837+0000 7f3e4e200640 1 -- 172.21.15.90:0/1956388837 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e4815dd50 msgr2=0x7f3e4806e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:35.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.837+0000 7f3e4e200640 1 --2- 172.21.15.90:0/1956388837 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e4815dd50 0x7f3e4806e020 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3e380097d0 tx=0x7f3e3802b6e0 comp rx=0 tx=0).stop 2024-03-24T01:25:35.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.837+0000 7f3e4e200640 1 -- 172.21.15.90:0/1956388837 shutdown_connections 2024-03-24T01:25:35.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.837+0000 7f3e4e200640 1 --2- 172.21.15.90:0/1956388837 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e4815dd50 0x7f3e4806e020 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:35.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.837+0000 7f3e4e200640 1 -- 172.21.15.90:0/1956388837 >> 172.21.15.90:0/1956388837 conn(0x7f3e480b1770 msgr2=0x7f3e480b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:35.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.837+0000 7f3e4e200640 1 -- 172.21.15.90:0/1956388837 shutdown_connections 2024-03-24T01:25:35.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.837+0000 7f3e4e200640 1 -- 172.21.15.90:0/1956388837 wait complete. 2024-03-24T01:25:35.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.839+0000 7f3e4e200640 1 Processor -- start 2024-03-24T01:25:35.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.839+0000 7f3e4e200640 1 -- start start 2024-03-24T01:25:35.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.840+0000 7f3e4e200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e4806b7d0 0x7f3e4814e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:35.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.840+0000 7f3e4e200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e480b3d60 0x7f3e481f4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:35.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.840+0000 7f3e4e200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e4815dd50 0x7f3e480cde70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:35.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.840+0000 7f3e4e200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e481625c0 con 0x7f3e4815dd50 2024-03-24T01:25:35.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.840+0000 7f3e4e200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e480b41c0 con 0x7f3e480b3d60 2024-03-24T01:25:35.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.840+0000 7f3e4e200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e481618d0 con 0x7f3e4806b7d0 2024-03-24T01:25:35.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.840+0000 7f3e47e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e4815dd50 0x7f3e480cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:35.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.840+0000 7f3e47e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e4815dd50 0x7f3e480cde70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55172/0 (socket says 172.21.15.90:55172) 2024-03-24T01:25:35.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.840+0000 7f3e47e00640 1 -- 172.21.15.90:0/2166683664 learned_addr learned my addr 172.21.15.90:0/2166683664 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:35.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.841+0000 7f3e3fe00640 1 --2- 172.21.15.90:0/2166683664 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e480b3d60 0x7f3e481f4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:35.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.841+0000 7f3e47400640 1 --2- 172.21.15.90:0/2166683664 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e4806b7d0 0x7f3e4814e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:35.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.841+0000 7f3e47e00640 1 -- 172.21.15.90:0/2166683664 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e4806b7d0 msgr2=0x7f3e4814e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:35.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.841+0000 7f3e47e00640 1 --2- 172.21.15.90:0/2166683664 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e4806b7d0 0x7f3e4814e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:35.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.841+0000 7f3e47e00640 1 -- 172.21.15.90:0/2166683664 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e480b3d60 msgr2=0x7f3e481f4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:35.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.841+0000 7f3e47e00640 1 --2- 172.21.15.90:0/2166683664 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e480b3d60 0x7f3e481f4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:35.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.841+0000 7f3e47e00640 1 -- 172.21.15.90:0/2166683664 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e48152ce0 con 0x7f3e4815dd50 2024-03-24T01:25:35.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.841+0000 7f3e3fe00640 1 --2- 172.21.15.90:0/2166683664 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e480b3d60 0x7f3e481f4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:25:35.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.841+0000 7f3e47e00640 1 --2- 172.21.15.90:0/2166683664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e4815dd50 0x7f3e480cde70 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f3e4000ccb0 tx=0x7f3e4000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:35.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.841+0000 7f3e3f400640 1 -- 172.21.15.90:0/2166683664 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e40017070 con 0x7f3e4815dd50 2024-03-24T01:25:35.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.841+0000 7f3e3f400640 1 -- 172.21.15.90:0/2166683664 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e40012400 con 0x7f3e4815dd50 2024-03-24T01:25:35.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.841+0000 7f3e4e200640 1 -- 172.21.15.90:0/2166683664 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e48155200 con 0x7f3e4815dd50 2024-03-24T01:25:35.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.842+0000 7f3e3f400640 1 -- 172.21.15.90:0/2166683664 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e400127b0 con 0x7f3e4815dd50 2024-03-24T01:25:35.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.842+0000 7f3e4e200640 1 -- 172.21.15.90:0/2166683664 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e4814fe90 con 0x7f3e4815dd50 2024-03-24T01:25:35.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.843+0000 7f3e4e200640 1 -- 172.21.15.90:0/2166683664 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e48068060 con 0x7f3e4815dd50 2024-03-24T01:25:35.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.844+0000 7f3e3f400640 1 -- 172.21.15.90:0/2166683664 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3e4001a070 con 0x7f3e4815dd50 2024-03-24T01:25:35.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.845+0000 7f3e3f400640 1 --2- 172.21.15.90:0/2166683664 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3e2406bbd0 0x7f3e2406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:35.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.845+0000 7f3e3f400640 1 -- 172.21.15.90:0/2166683664 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f3e4008ae10 con 0x7f3e4815dd50 2024-03-24T01:25:35.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.846+0000 7f3e47400640 1 --2- 172.21.15.90:0/2166683664 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3e2406bbd0 0x7f3e2406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:35.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.847+0000 7f3e47400640 1 --2- 172.21.15.90:0/2166683664 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3e2406bbd0 0x7f3e2406e090 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3e38034000 tx=0x7f3e38008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:35.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:35.850+0000 7f3e3f400640 1 -- 172.21.15.90:0/2166683664 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3e400594e0 con 0x7f3e4815dd50 2024-03-24T01:25:36.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.049+0000 7f3e4e200640 1 -- 172.21.15.90:0/2166683664 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e4815e130 con 0x7f3e4815dd50 2024-03-24T01:25:36.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.052+0000 7f3e3f400640 1 -- 172.21.15.90:0/2166683664 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18925 (secure 0 0 0) 0x7f3e40058de0 con 0x7f3e4815dd50 2024-03-24T01:25:36.052 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:25:36.053 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:24:53.236401+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:24:08.643966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T01:24:09.189904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T01:24:13.027323+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:25:36.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.054+0000 7f3e4e200640 1 -- 172.21.15.90:0/2166683664 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3e2406bbd0 msgr2=0x7f3e2406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:36.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.054+0000 7f3e4e200640 1 --2- 172.21.15.90:0/2166683664 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3e2406bbd0 0x7f3e2406e090 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3e38034000 tx=0x7f3e38008030 comp rx=0 tx=0).stop 2024-03-24T01:25:36.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.054+0000 7f3e4e200640 1 -- 172.21.15.90:0/2166683664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e4815dd50 msgr2=0x7f3e480cde70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:36.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.054+0000 7f3e4e200640 1 --2- 172.21.15.90:0/2166683664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e4815dd50 0x7f3e480cde70 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f3e4000ccb0 tx=0x7f3e4000a430 comp rx=0 tx=0).stop 2024-03-24T01:25:36.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.055+0000 7f3e4e200640 1 -- 172.21.15.90:0/2166683664 shutdown_connections 2024-03-24T01:25:36.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.055+0000 7f3e4e200640 1 --2- 172.21.15.90:0/2166683664 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3e2406bbd0 0x7f3e2406e090 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:36.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.055+0000 7f3e4e200640 1 --2- 172.21.15.90:0/2166683664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e4815dd50 0x7f3e480cde70 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:36.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.055+0000 7f3e4e200640 1 --2- 172.21.15.90:0/2166683664 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e480b3d60 0x7f3e481f4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:36.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.055+0000 7f3e4e200640 1 --2- 172.21.15.90:0/2166683664 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e4806b7d0 0x7f3e4814e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:36.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.055+0000 7f3e4e200640 1 -- 172.21.15.90:0/2166683664 >> 172.21.15.90:0/2166683664 conn(0x7f3e480b1770 msgr2=0x7f3e4806f330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:36.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.055+0000 7f3e4e200640 1 -- 172.21.15.90:0/2166683664 shutdown_connections 2024-03-24T01:25:36.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.055+0000 7f3e4e200640 1 -- 172.21.15.90:0/2166683664 wait complete. 2024-03-24T01:25:36.075 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T01:24:08.643966+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T01:24:09.189904+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T01:24:13.027323+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T01:25:36.075 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:25:36.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.288+0000 7f4d6d400640 1 Processor -- start 2024-03-24T01:25:36.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.288+0000 7f4d6d400640 1 -- start start 2024-03-24T01:25:36.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.289+0000 7f4d6d400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4d680aeeb0 0x7f4d680b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:36.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.289+0000 7f4d6d400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d6815bc50 0x7f4d6806b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:36.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.289+0000 7f4d6d400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4d6806b600 0x7f4d6815e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:36.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.289+0000 7f4d6d400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d6806ba70 con 0x7f4d680aeeb0 2024-03-24T01:25:36.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.289+0000 7f4d6d400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d680b4da0 con 0x7f4d6806b600 2024-03-24T01:25:36.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.289+0000 7f4d6d400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6806dac0 con 0x7f4d6815bc50 2024-03-24T01:25:36.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.290+0000 7f4d66800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4d680aeeb0 0x7f4d680b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:36.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.290+0000 7f4d66800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4d680aeeb0 0x7f4d680b44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55186/0 (socket says 172.21.15.90:55186) 2024-03-24T01:25:36.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.290+0000 7f4d66800640 1 -- 172.21.15.90:0/3069684988 learned_addr learned my addr 172.21.15.90:0/3069684988 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:36.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.290+0000 7f4d67200640 1 --2- 172.21.15.90:0/3069684988 >> v2:172.21.15.120:3300/0 conn(0x7f4d6806b600 0x7f4d6815e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:36.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.290+0000 7f4d5de00640 1 --2- 172.21.15.90:0/3069684988 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d6815bc50 0x7f4d6806b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:36.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.290+0000 7f4d66800640 1 -- 172.21.15.90:0/3069684988 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d6815bc50 msgr2=0x7f4d6806b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:36.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.290+0000 7f4d66800640 1 --2- 172.21.15.90:0/3069684988 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d6815bc50 0x7f4d6806b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:36.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.290+0000 7f4d66800640 1 -- 172.21.15.90:0/3069684988 >> v2:172.21.15.120:3300/0 conn(0x7f4d6806b600 msgr2=0x7f4d6815e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:36.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.290+0000 7f4d66800640 1 --2- 172.21.15.90:0/3069684988 >> v2:172.21.15.120:3300/0 conn(0x7f4d6806b600 0x7f4d6815e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:36.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.290+0000 7f4d66800640 1 -- 172.21.15.90:0/3069684988 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d6806dda0 con 0x7f4d680aeeb0 2024-03-24T01:25:36.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.291+0000 7f4d66800640 1 --2- 172.21.15.90:0/3069684988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d680aeeb0 0x7f4d680b44c0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f4d680cc5d0 tx=0x7f4d5802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4febe59650adb995 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:36.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.292+0000 7f4d65e00640 1 -- 172.21.15.90:0/3069684988 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d58038070 con 0x7f4d680aeeb0 2024-03-24T01:25:36.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.292+0000 7f4d65e00640 1 -- 172.21.15.90:0/3069684988 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d5802f050 con 0x7f4d680aeeb0 2024-03-24T01:25:36.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.292+0000 7f4d65e00640 1 -- 172.21.15.90:0/3069684988 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d58031970 con 0x7f4d680aeeb0 2024-03-24T01:25:36.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.292+0000 7f4d6d400640 1 -- 172.21.15.90:0/3069684988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d680aeeb0 msgr2=0x7f4d680b44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:36.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.292+0000 7f4d6d400640 1 --2- 172.21.15.90:0/3069684988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d680aeeb0 0x7f4d680b44c0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f4d680cc5d0 tx=0x7f4d5802b7f0 comp rx=0 tx=0).stop 2024-03-24T01:25:36.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.292+0000 7f4d6d400640 1 -- 172.21.15.90:0/3069684988 shutdown_connections 2024-03-24T01:25:36.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.292+0000 7f4d6d400640 1 --2- 172.21.15.90:0/3069684988 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d6815bc50 0x7f4d6806b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:36.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.292+0000 7f4d6d400640 1 --2- 172.21.15.90:0/3069684988 >> v2:172.21.15.120:3300/0 conn(0x7f4d6806b600 0x7f4d6815e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:36.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.293+0000 7f4d6d400640 1 --2- 172.21.15.90:0/3069684988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d680aeeb0 0x7f4d680b44c0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:36.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.293+0000 7f4d6d400640 1 -- 172.21.15.90:0/3069684988 >> 172.21.15.90:0/3069684988 conn(0x7f4d680b1770 msgr2=0x7f4d680b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:36.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.293+0000 7f4d6d400640 1 -- 172.21.15.90:0/3069684988 shutdown_connections 2024-03-24T01:25:36.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.293+0000 7f4d6d400640 1 -- 172.21.15.90:0/3069684988 wait complete. 2024-03-24T01:25:36.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.294+0000 7f4d6d400640 1 Processor -- start 2024-03-24T01:25:36.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.294+0000 7f4d6d400640 1 -- start start 2024-03-24T01:25:36.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.295+0000 7f4d6d400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d6806b600 0x7f4d680cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:36.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.295+0000 7f4d6d400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d680aeeb0 0x7f4d681f7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:36.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.295+0000 7f4d6d400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d6815bc50 0x7f4d68147b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:36.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.295+0000 7f4d6d400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d680af290 con 0x7f4d6815bc50 2024-03-24T01:25:36.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.295+0000 7f4d6d400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6806e360 con 0x7f4d680aeeb0 2024-03-24T01:25:36.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.295+0000 7f4d6d400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d680b4a00 con 0x7f4d6806b600 2024-03-24T01:25:36.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.296+0000 7f4d67200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d6815bc50 0x7f4d68147b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:36.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.296+0000 7f4d67200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d6815bc50 0x7f4d68147b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55190/0 (socket says 172.21.15.90:55190) 2024-03-24T01:25:36.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.296+0000 7f4d67200640 1 -- 172.21.15.90:0/3900951723 learned_addr learned my addr 172.21.15.90:0/3900951723 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:36.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.296+0000 7f4d66800640 1 --2- 172.21.15.90:0/3900951723 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d6806b600 0x7f4d680cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:36.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.296+0000 7f4d5de00640 1 --2- 172.21.15.90:0/3900951723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d680aeeb0 0x7f4d681f7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:36.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.296+0000 7f4d67200640 1 -- 172.21.15.90:0/3900951723 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d6806b600 msgr2=0x7f4d680cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:36.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.296+0000 7f4d67200640 1 --2- 172.21.15.90:0/3900951723 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d6806b600 0x7f4d680cf1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:36.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.296+0000 7f4d67200640 1 -- 172.21.15.90:0/3900951723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d680aeeb0 msgr2=0x7f4d681f7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:36.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.296+0000 7f4d67200640 1 --2- 172.21.15.90:0/3900951723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d680aeeb0 0x7f4d681f7b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:36.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.296+0000 7f4d67200640 1 -- 172.21.15.90:0/3900951723 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d681484a0 con 0x7f4d6815bc50 2024-03-24T01:25:36.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.296+0000 7f4d67200640 1 --2- 172.21.15.90:0/3900951723 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d6815bc50 0x7f4d68147b30 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f4d60007e90 tx=0x7f4d6000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:36.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.297+0000 7f4d5f400640 1 -- 172.21.15.90:0/3900951723 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d60014420 con 0x7f4d6815bc50 2024-03-24T01:25:36.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.297+0000 7f4d5f400640 1 -- 172.21.15.90:0/3900951723 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d60019070 con 0x7f4d6815bc50 2024-03-24T01:25:36.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.297+0000 7f4d5f400640 1 -- 172.21.15.90:0/3900951723 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d60014800 con 0x7f4d6815bc50 2024-03-24T01:25:36.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.297+0000 7f4d6d400640 1 -- 172.21.15.90:0/3900951723 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d68148780 con 0x7f4d6815bc50 2024-03-24T01:25:36.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.297+0000 7f4d6d400640 1 -- 172.21.15.90:0/3900951723 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d681f80d0 con 0x7f4d6815bc50 2024-03-24T01:25:36.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.298+0000 7f4d6d400640 1 -- 172.21.15.90:0/3900951723 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d6804f400 con 0x7f4d6815bc50 2024-03-24T01:25:36.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.299+0000 7f4d5f400640 1 -- 172.21.15.90:0/3900951723 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4d600149a0 con 0x7f4d6815bc50 2024-03-24T01:25:36.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.300+0000 7f4d5f400640 1 --2- 172.21.15.90:0/3900951723 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4d4406bc00 0x7f4d4406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:36.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.300+0000 7f4d5f400640 1 -- 172.21.15.90:0/3900951723 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f4d6008b6c0 con 0x7f4d6815bc50 2024-03-24T01:25:36.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.301+0000 7f4d66800640 1 --2- 172.21.15.90:0/3900951723 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4d4406bc00 0x7f4d4406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:36.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.302+0000 7f4d66800640 1 --2- 172.21.15.90:0/3900951723 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4d4406bc00 0x7f4d4406e0c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4d58035000 tx=0x7f4d58008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:36.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.305+0000 7f4d5f400640 1 -- 172.21.15.90:0/3900951723 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4d60059d80 con 0x7f4d6815bc50 2024-03-24T01:25:36.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.503+0000 7f4d6d400640 1 -- 172.21.15.90:0/3900951723 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4d6806e910 con 0x7f4d6815bc50 2024-03-24T01:25:36.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.504+0000 7f4d5f400640 1 -- 172.21.15.90:0/3900951723 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4d60059680 con 0x7f4d6815bc50 2024-03-24T01:25:36.504 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:25:36.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.506+0000 7f4d6d400640 1 -- 172.21.15.90:0/3900951723 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4d4406bc00 msgr2=0x7f4d4406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:36.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.506+0000 7f4d6d400640 1 --2- 172.21.15.90:0/3900951723 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4d4406bc00 0x7f4d4406e0c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4d58035000 tx=0x7f4d58008030 comp rx=0 tx=0).stop 2024-03-24T01:25:36.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.506+0000 7f4d6d400640 1 -- 172.21.15.90:0/3900951723 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d6815bc50 msgr2=0x7f4d68147b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:36.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.506+0000 7f4d6d400640 1 --2- 172.21.15.90:0/3900951723 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d6815bc50 0x7f4d68147b30 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f4d60007e90 tx=0x7f4d6000e4b0 comp rx=0 tx=0).stop 2024-03-24T01:25:36.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.506+0000 7f4d6d400640 1 -- 172.21.15.90:0/3900951723 shutdown_connections 2024-03-24T01:25:36.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.506+0000 7f4d6d400640 1 --2- 172.21.15.90:0/3900951723 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4d4406bc00 0x7f4d4406e0c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:36.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.506+0000 7f4d6d400640 1 --2- 172.21.15.90:0/3900951723 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d6815bc50 0x7f4d68147b30 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:36.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.506+0000 7f4d6d400640 1 --2- 172.21.15.90:0/3900951723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d680aeeb0 0x7f4d681f7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:36.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.506+0000 7f4d6d400640 1 --2- 172.21.15.90:0/3900951723 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d6806b600 0x7f4d680cf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:36.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.506+0000 7f4d6d400640 1 -- 172.21.15.90:0/3900951723 >> 172.21.15.90:0/3900951723 conn(0x7f4d680b1770 msgr2=0x7f4d680cfd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:36.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.507+0000 7f4d6d400640 1 -- 172.21.15.90:0/3900951723 shutdown_connections 2024-03-24T01:25:36.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.507+0000 7f4d6d400640 1 -- 172.21.15.90:0/3900951723 wait complete. 2024-03-24T01:25:36.525 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T01:25:36.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.737+0000 7fa996400640 1 Processor -- start 2024-03-24T01:25:36.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.737+0000 7fa996400640 1 -- start start 2024-03-24T01:25:36.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.739+0000 7fa996400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9900cf150 0x7fa9900ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:36.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.739+0000 7fa996400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa99006bca0 con 0x7fa99006b7d0 2024-03-24T01:25:36.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.739+0000 7fa996400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9900b36f0 con 0x7fa990159620 2024-03-24T01:25:36.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.739+0000 7fa996400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9900b4470 con 0x7fa9900cf150 2024-03-24T01:25:36.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.739+0000 7fa986a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa99006b7d0 0x7fa9900cea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:32820/0 (socket says 172.21.15.90:32820) 2024-03-24T01:25:36.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.739+0000 7fa986a00640 1 -- 172.21.15.90:0/2206667231 learned_addr learned my addr 172.21.15.90:0/2206667231 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:36.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.740+0000 7fa98fe00640 1 --2- 172.21.15.90:0/2206667231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9900cf150 0x7fa9900ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:36.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.740+0000 7fa98ea00640 1 -- 172.21.15.90:0/2206667231 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 663606766 0 0) 0x7fa99006bca0 con 0x7fa99006b7d0 2024-03-24T01:25:36.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.740+0000 7fa98ea00640 1 -- 172.21.15.90:0/2206667231 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa974003520 con 0x7fa99006b7d0 2024-03-24T01:25:36.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.740+0000 7fa98ea00640 1 -- 172.21.15.90:0/2206667231 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1490649492 0 0) 0x7fa974003520 con 0x7fa99006b7d0 2024-03-24T01:25:36.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.740+0000 7fa98ea00640 1 -- 172.21.15.90:0/2206667231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9900cf150 msgr2=0x7fa9900ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:36.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.740+0000 7fa98ea00640 1 --2- 172.21.15.90:0/2206667231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9900cf150 0x7fa9900ae530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:36.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.740+0000 7fa98ea00640 1 -- 172.21.15.90:0/2206667231 >> v1:172.21.15.120:6789/0 conn(0x7fa990159620 legacy=0x7fa99006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:36.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.740+0000 7fa98ea00640 1 -- 172.21.15.90:0/2206667231 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9900b3990 con 0x7fa99006b7d0 2024-03-24T01:25:36.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.740+0000 7fa98ea00640 1 -- 172.21.15.90:0/2206667231 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7fa978001670 con 0x7fa99006b7d0 2024-03-24T01:25:36.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.740+0000 7fa98ea00640 1 -- 172.21.15.90:0/2206667231 >> v1:172.21.15.90:6789/0 conn(0x7fa99006b7d0 legacy=0x7fa9900cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:36.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.740+0000 7fa98ea00640 1 --2- 172.21.15.90:0/2206667231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9740039e0 0x7fa974003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:36.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.740+0000 7fa98ea00640 1 --2- 172.21.15.90:0/2206667231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9900cf150 0x7fa974004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:36.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.740+0000 7fa98ea00640 1 --2- 172.21.15.90:0/2206667231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa974004d40 0x7fa9740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:36.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.740+0000 7fa98ea00640 1 -- 172.21.15.90:0/2206667231 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9900b4470 con 0x7fa9740039e0 2024-03-24T01:25:36.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.741+0000 7fa98ea00640 1 -- 172.21.15.90:0/2206667231 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa97400baf0 con 0x7fa974004d40 2024-03-24T01:25:36.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.741+0000 7fa98ea00640 1 -- 172.21.15.90:0/2206667231 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa97400bcf0 con 0x7fa9900cf150 2024-03-24T01:25:36.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.741+0000 7fa98f400640 1 --2- 172.21.15.90:0/2206667231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9740039e0 0x7fa974003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:36.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.741+0000 7fa986a00640 1 --2- 172.21.15.90:0/2206667231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9900cf150 0x7fa974004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:36.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.741+0000 7fa98f400640 1 -- 172.21.15.90:0/2206667231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9900cf150 msgr2=0x7fa974004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:36.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.741+0000 7fa98fe00640 1 --2- 172.21.15.90:0/2206667231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa974004d40 0x7fa9740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:36.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.741+0000 7fa98f400640 1 --2- 172.21.15.90:0/2206667231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9900cf150 0x7fa974004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:36.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.741+0000 7fa98f400640 1 -- 172.21.15.90:0/2206667231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa974004d40 msgr2=0x7fa9740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:36.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.741+0000 7fa98f400640 1 --2- 172.21.15.90:0/2206667231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa974004d40 0x7fa9740051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:36.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.741+0000 7fa98f400640 1 -- 172.21.15.90:0/2206667231 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa974003520 con 0x7fa9740039e0 2024-03-24T01:25:36.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.742+0000 7fa98f400640 1 --2- 172.21.15.90:0/2206667231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9740039e0 0x7fa974003e00 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fa980001e50 tx=0x7fa974005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcbb6e0d37822b4f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:36.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.743+0000 7fa98ea00640 1 -- 172.21.15.90:0/2206667231 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa980001a70 con 0x7fa9740039e0 2024-03-24T01:25:36.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.743+0000 7fa98ea00640 1 -- 172.21.15.90:0/2206667231 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa980003b70 con 0x7fa9740039e0 2024-03-24T01:25:36.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.743+0000 7fa98ea00640 1 -- 172.21.15.90:0/2206667231 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9800050d0 con 0x7fa9740039e0 2024-03-24T01:25:36.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.743+0000 7fa996400640 1 -- 172.21.15.90:0/2206667231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9740039e0 msgr2=0x7fa974003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:36.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.743+0000 7fa996400640 1 --2- 172.21.15.90:0/2206667231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9740039e0 0x7fa974003e00 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fa980001e50 tx=0x7fa974005830 comp rx=0 tx=0).stop 2024-03-24T01:25:36.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.743+0000 7fa98fe00640 1 -- 172.21.15.90:0/2206667231 reap_dead start 2024-03-24T01:25:36.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.743+0000 7fa996400640 1 -- 172.21.15.90:0/2206667231 shutdown_connections 2024-03-24T01:25:36.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.743+0000 7fa996400640 1 -- 172.21.15.90:0/2206667231 >> 172.21.15.90:0/2206667231 conn(0x7fa9900b1770 msgr2=0x7fa9900b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:36.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.743+0000 7fa996400640 1 -- 172.21.15.90:0/2206667231 shutdown_connections 2024-03-24T01:25:36.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.743+0000 7fa996400640 1 -- 172.21.15.90:0/2206667231 wait complete. 2024-03-24T01:25:36.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.745+0000 7fa996400640 1 Processor -- start 2024-03-24T01:25:36.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.745+0000 7fa996400640 1 -- start start 2024-03-24T01:25:36.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.746+0000 7fa996400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9740039e0 0x7fa9901434f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:36.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.746+0000 7fa996400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa99006f2d0 0x7fa9900c9c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:36.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.746+0000 7fa996400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9900ca190 0x7fa9901f3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:36.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.746+0000 7fa996400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa990159a00 con 0x7fa9900ca190 2024-03-24T01:25:36.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.746+0000 7fa996400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9900b3f50 con 0x7fa9740039e0 2024-03-24T01:25:36.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.746+0000 7fa996400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa99015bfe0 con 0x7fa99006f2d0 2024-03-24T01:25:36.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.747+0000 7fa98fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9900ca190 0x7fa9901f3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:36.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.747+0000 7fa98fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9900ca190 0x7fa9901f3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55206/0 (socket says 172.21.15.90:55206) 2024-03-24T01:25:36.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.747+0000 7fa98fe00640 1 -- 172.21.15.90:0/3223208879 learned_addr learned my addr 172.21.15.90:0/3223208879 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:36.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.747+0000 7fa986a00640 1 --2- 172.21.15.90:0/3223208879 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa99006f2d0 0x7fa9900c9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:36.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.747+0000 7fa98f400640 1 --2- 172.21.15.90:0/3223208879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9740039e0 0x7fa9901434f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:36.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.747+0000 7fa98fe00640 1 -- 172.21.15.90:0/3223208879 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa99006f2d0 msgr2=0x7fa9900c9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:36.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.747+0000 7fa98fe00640 1 --2- 172.21.15.90:0/3223208879 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa99006f2d0 0x7fa9900c9c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:36.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.747+0000 7fa98fe00640 1 -- 172.21.15.90:0/3223208879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9740039e0 msgr2=0x7fa9901434f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:36.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.747+0000 7fa98fe00640 1 --2- 172.21.15.90:0/3223208879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9740039e0 0x7fa9901434f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:36.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.747+0000 7fa98fe00640 1 -- 172.21.15.90:0/3223208879 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa99006e510 con 0x7fa9900ca190 2024-03-24T01:25:36.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.747+0000 7fa98fe00640 1 --2- 172.21.15.90:0/3223208879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9900ca190 0x7fa9901f3ec0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fa9900b2e60 tx=0x7fa988010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:36.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.748+0000 7fa987e00640 1 -- 172.21.15.90:0/3223208879 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98800f040 con 0x7fa9900ca190 2024-03-24T01:25:36.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.748+0000 7fa987e00640 1 -- 172.21.15.90:0/3223208879 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa98800e860 con 0x7fa9900ca190 2024-03-24T01:25:36.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.748+0000 7fa987e00640 1 -- 172.21.15.90:0/3223208879 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98800ec50 con 0x7fa9900ca190 2024-03-24T01:25:36.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.748+0000 7fa996400640 1 -- 172.21.15.90:0/3223208879 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa99006e7f0 con 0x7fa9900ca190 2024-03-24T01:25:36.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.748+0000 7fa996400640 1 -- 172.21.15.90:0/3223208879 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa99006edb0 con 0x7fa9900ca190 2024-03-24T01:25:36.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.749+0000 7fa996400640 1 -- 172.21.15.90:0/3223208879 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9900ce620 con 0x7fa9900ca190 2024-03-24T01:25:36.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.750+0000 7fa987e00640 1 -- 172.21.15.90:0/3223208879 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa9880253f0 con 0x7fa9900ca190 2024-03-24T01:25:36.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.751+0000 7fa987e00640 1 --2- 172.21.15.90:0/3223208879 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa96c06bc00 0x7fa96c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:36.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.751+0000 7fa987e00640 1 -- 172.21.15.90:0/3223208879 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7fa98808b080 con 0x7fa9900ca190 2024-03-24T01:25:36.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.751+0000 7fa98f400640 1 --2- 172.21.15.90:0/3223208879 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa96c06bc00 0x7fa96c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:36.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.752+0000 7fa98f400640 1 --2- 172.21.15.90:0/3223208879 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa96c06bc00 0x7fa96c06e0c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa980034000 tx=0x7fa980009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:36.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.755+0000 7fa987e00640 1 -- 172.21.15.90:0/3223208879 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa9900ce620 con 0x7fa9900ca190 2024-03-24T01:25:36.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.953+0000 7fa996400640 1 -- 172.21.15.90:0/3223208879 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa99006efb0 con 0x7fa9900ca190 2024-03-24T01:25:36.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.954+0000 7fa987e00640 1 -- 172.21.15.90:0/3223208879 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa99006efb0 con 0x7fa9900ca190 2024-03-24T01:25:36.955 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:25:36.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.957+0000 7fa996400640 1 -- 172.21.15.90:0/3223208879 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa96c06bc00 msgr2=0x7fa96c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:36.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.957+0000 7fa996400640 1 --2- 172.21.15.90:0/3223208879 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa96c06bc00 0x7fa96c06e0c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa980034000 tx=0x7fa980009030 comp rx=0 tx=0).stop 2024-03-24T01:25:36.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.957+0000 7fa996400640 1 -- 172.21.15.90:0/3223208879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9900ca190 msgr2=0x7fa9901f3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:36.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.957+0000 7fa996400640 1 --2- 172.21.15.90:0/3223208879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9900ca190 0x7fa9901f3ec0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fa9900b2e60 tx=0x7fa988010c20 comp rx=0 tx=0).stop 2024-03-24T01:25:36.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.957+0000 7fa996400640 1 -- 172.21.15.90:0/3223208879 shutdown_connections 2024-03-24T01:25:36.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.957+0000 7fa996400640 1 --2- 172.21.15.90:0/3223208879 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa96c06bc00 0x7fa96c06e0c0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:36.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.957+0000 7fa996400640 1 --2- 172.21.15.90:0/3223208879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9900ca190 0x7fa9901f3ec0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:36.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.957+0000 7fa996400640 1 --2- 172.21.15.90:0/3223208879 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa99006f2d0 0x7fa9900c9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:36.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.957+0000 7fa996400640 1 --2- 172.21.15.90:0/3223208879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9740039e0 0x7fa9901434f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:36.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.957+0000 7fa996400640 1 -- 172.21.15.90:0/3223208879 >> 172.21.15.90:0/3223208879 conn(0x7fa9900b1770 msgr2=0x7fa99015e190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:36.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.957+0000 7fa996400640 1 -- 172.21.15.90:0/3223208879 shutdown_connections 2024-03-24T01:25:36.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:36.957+0000 7fa996400640 1 -- 172.21.15.90:0/3223208879 wait complete. 2024-03-24T01:25:36.977 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T01:25:37.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.186+0000 7f555b200640 1 Processor -- start 2024-03-24T01:25:37.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.186+0000 7f555b200640 1 -- start start 2024-03-24T01:25:37.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.188+0000 7f555b200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5554159620 0x7f555406b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:37.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.188+0000 7f555b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f555406b600 0x7f55540cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:37.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.188+0000 7f555b200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f555406bca0 con 0x7f55540cef80 2024-03-24T01:25:37.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.188+0000 7f555b200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55540b4470 con 0x7f5554159620 2024-03-24T01:25:37.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.188+0000 7f555b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f555406ba60 con 0x7f555406b600 2024-03-24T01:25:37.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.189+0000 7f554b400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f555406b600 0x7f55540cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:37.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.189+0000 7f554b400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f555406b600 0x7f55540cea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.90:58760/0 (socket says 172.21.15.90:58760) 2024-03-24T01:25:37.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.189+0000 7f554b400640 1 -- 172.21.15.90:0/1173434687 learned_addr learned my addr 172.21.15.90:0/1173434687 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:37.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.189+0000 7f5553e00640 1 --2- 172.21.15.90:0/1173434687 >> v2:172.21.15.120:3300/0 conn(0x7f5554159620 0x7f555406b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:37.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.190+0000 7f554b400640 1 -- 172.21.15.90:0/1173434687 >> v2:172.21.15.120:3300/0 conn(0x7f5554159620 msgr2=0x7f555406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:37.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.190+0000 7f554b400640 1 --2- 172.21.15.90:0/1173434687 >> v2:172.21.15.120:3300/0 conn(0x7f5554159620 0x7f555406b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:37.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.190+0000 7f554b400640 1 -- 172.21.15.90:0/1173434687 >> v1:172.21.15.120:6789/0 conn(0x7f55540cef80 legacy=0x7f55540ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:37.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.190+0000 7f554b400640 1 -- 172.21.15.90:0/1173434687 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55540b36f0 con 0x7f555406b600 2024-03-24T01:25:37.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.191+0000 7f554b400640 1 --2- 172.21.15.90:0/1173434687 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f555406b600 0x7f55540cea40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f553c009870 tx=0x7f553c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c762937c13fa72da server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:37.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.191+0000 7f5553400640 1 -- 172.21.15.90:0/1173434687 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f553c035020 con 0x7f555406b600 2024-03-24T01:25:37.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.191+0000 7f5553400640 1 -- 172.21.15.90:0/1173434687 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f553c031640 con 0x7f555406b600 2024-03-24T01:25:37.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.191+0000 7f5553400640 1 -- 172.21.15.90:0/1173434687 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f553c02d040 con 0x7f555406b600 2024-03-24T01:25:37.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.192+0000 7f555b200640 1 -- 172.21.15.90:0/1173434687 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f555406b600 msgr2=0x7f55540cea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:37.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.192+0000 7f555b200640 1 --2- 172.21.15.90:0/1173434687 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f555406b600 0x7f55540cea40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f553c009870 tx=0x7f553c02b8f0 comp rx=0 tx=0).stop 2024-03-24T01:25:37.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.192+0000 7f555b200640 1 -- 172.21.15.90:0/1173434687 shutdown_connections 2024-03-24T01:25:37.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.192+0000 7f555b200640 1 --2- 172.21.15.90:0/1173434687 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f555406b600 0x7f55540cea40 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:37.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.192+0000 7f555b200640 1 --2- 172.21.15.90:0/1173434687 >> v2:172.21.15.120:3300/0 conn(0x7f5554159620 0x7f555406b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:37.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.192+0000 7f555b200640 1 -- 172.21.15.90:0/1173434687 >> 172.21.15.90:0/1173434687 conn(0x7f55540b1770 msgr2=0x7f55540b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:37.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.192+0000 7f555b200640 1 -- 172.21.15.90:0/1173434687 shutdown_connections 2024-03-24T01:25:37.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.192+0000 7f555b200640 1 -- 172.21.15.90:0/1173434687 wait complete. 2024-03-24T01:25:37.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.194+0000 7f555b200640 1 Processor -- start 2024-03-24T01:25:37.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.194+0000 7f555b200640 1 -- start start 2024-03-24T01:25:37.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.195+0000 7f555b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f555406b600 0x7f555413b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:37.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.195+0000 7f555b200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55540cef80 0x7f55541507f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:37.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.195+0000 7f555b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5554159620 0x7f5554150d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:37.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.195+0000 7f555b200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55540b3f50 con 0x7f5554159620 2024-03-24T01:25:37.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.195+0000 7f555b200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55540b3c70 con 0x7f55540cef80 2024-03-24T01:25:37.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.195+0000 7f555b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5554159a00 con 0x7f555406b600 2024-03-24T01:25:37.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.195+0000 7f5559200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5554159620 0x7f5554150d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:37.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.195+0000 7f5559200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5554159620 0x7f5554150d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55212/0 (socket says 172.21.15.90:55212) 2024-03-24T01:25:37.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.195+0000 7f5559200640 1 -- 172.21.15.90:0/2185842344 learned_addr learned my addr 172.21.15.90:0/2185842344 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:37.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.195+0000 7f5553e00640 1 --2- 172.21.15.90:0/2185842344 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f555406b600 0x7f555413b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:37.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.195+0000 7f554b400640 1 --2- 172.21.15.90:0/2185842344 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55540cef80 0x7f55541507f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:37.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.195+0000 7f5559200640 1 -- 172.21.15.90:0/2185842344 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f555406b600 msgr2=0x7f555413b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:37.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.195+0000 7f5559200640 1 --2- 172.21.15.90:0/2185842344 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f555406b600 0x7f555413b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:37.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.195+0000 7f5559200640 1 -- 172.21.15.90:0/2185842344 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55540cef80 msgr2=0x7f55541507f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:37.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.195+0000 7f5559200640 1 --2- 172.21.15.90:0/2185842344 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55540cef80 0x7f55541507f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:37.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.195+0000 7f5559200640 1 -- 172.21.15.90:0/2185842344 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55541f3200 con 0x7f5554159620 2024-03-24T01:25:37.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.195+0000 7f554b400640 1 --2- 172.21.15.90:0/2185842344 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55540cef80 0x7f55541507f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:37.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.196+0000 7f5559200640 1 --2- 172.21.15.90:0/2185842344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5554159620 0x7f5554150d30 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f554c00c7a0 tx=0x7f554c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:37.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.196+0000 7f5550c00640 1 -- 172.21.15.90:0/2185842344 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f554c003b70 con 0x7f5554159620 2024-03-24T01:25:37.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.196+0000 7f5550c00640 1 -- 172.21.15.90:0/2185842344 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f554c003d30 con 0x7f5554159620 2024-03-24T01:25:37.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.196+0000 7f555b200640 1 -- 172.21.15.90:0/2185842344 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55541f34e0 con 0x7f5554159620 2024-03-24T01:25:37.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.196+0000 7f5550c00640 1 -- 172.21.15.90:0/2185842344 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f554c00aa90 con 0x7f5554159620 2024-03-24T01:25:37.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.196+0000 7f555b200640 1 -- 172.21.15.90:0/2185842344 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55541f3aa0 con 0x7f5554159620 2024-03-24T01:25:37.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.197+0000 7f555b200640 1 -- 172.21.15.90:0/2185842344 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f555404f400 con 0x7f5554159620 2024-03-24T01:25:37.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.198+0000 7f5550c00640 1 -- 172.21.15.90:0/2185842344 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f554c00ac30 con 0x7f5554159620 2024-03-24T01:25:37.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.199+0000 7f5550c00640 1 --2- 172.21.15.90:0/2185842344 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f553006bbd0 0x7f553006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:37.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.199+0000 7f5550c00640 1 -- 172.21.15.90:0/2185842344 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f554c08ad40 con 0x7f5554159620 2024-03-24T01:25:37.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.200+0000 7f5553e00640 1 --2- 172.21.15.90:0/2185842344 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f553006bbd0 0x7f553006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:37.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.201+0000 7f5553e00640 1 --2- 172.21.15.90:0/2185842344 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f553006bbd0 0x7f553006e090 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5544009970 tx=0x7f5544008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:37.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.203+0000 7f5550c00640 1 -- 172.21.15.90:0/2185842344 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f554c0594a0 con 0x7f5554159620 2024-03-24T01:25:37.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.403+0000 7f555b200640 1 -- 172.21.15.90:0/2185842344 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f55540b2e90 con 0x7f5554159620 2024-03-24T01:25:37.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.404+0000 7f5550c00640 1 -- 172.21.15.90:0/2185842344 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f554c058da0 con 0x7f5554159620 2024-03-24T01:25:37.404 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-24T01:25:37.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.407+0000 7f555b200640 1 -- 172.21.15.90:0/2185842344 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f553006bbd0 msgr2=0x7f553006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:37.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.407+0000 7f555b200640 1 --2- 172.21.15.90:0/2185842344 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f553006bbd0 0x7f553006e090 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5544009970 tx=0x7f5544008040 comp rx=0 tx=0).stop 2024-03-24T01:25:37.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.407+0000 7f555b200640 1 -- 172.21.15.90:0/2185842344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5554159620 msgr2=0x7f5554150d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:37.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.407+0000 7f555b200640 1 --2- 172.21.15.90:0/2185842344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5554159620 0x7f5554150d30 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f554c00c7a0 tx=0x7f554c00cc60 comp rx=0 tx=0).stop 2024-03-24T01:25:37.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.407+0000 7f555b200640 1 -- 172.21.15.90:0/2185842344 shutdown_connections 2024-03-24T01:25:37.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.407+0000 7f555b200640 1 --2- 172.21.15.90:0/2185842344 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f553006bbd0 0x7f553006e090 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:37.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.407+0000 7f555b200640 1 --2- 172.21.15.90:0/2185842344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5554159620 0x7f5554150d30 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:37.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.407+0000 7f555b200640 1 --2- 172.21.15.90:0/2185842344 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55540cef80 0x7f55541507f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:37.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.407+0000 7f555b200640 1 --2- 172.21.15.90:0/2185842344 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f555406b600 0x7f555413b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:37.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.407+0000 7f555b200640 1 -- 172.21.15.90:0/2185842344 >> 172.21.15.90:0/2185842344 conn(0x7f55540b1770 msgr2=0x7f555415e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:37.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.407+0000 7f555b200640 1 -- 172.21.15.90:0/2185842344 shutdown_connections 2024-03-24T01:25:37.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.407+0000 7f555b200640 1 -- 172.21.15.90:0/2185842344 wait complete. 2024-03-24T01:25:37.427 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T01:25:37.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.635+0000 7f663dc00640 1 Processor -- start 2024-03-24T01:25:37.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.635+0000 7f663dc00640 1 -- start start 2024-03-24T01:25:37.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.636+0000 7f663dc00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f66380aeeb0 0x7f66380b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:37.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.637+0000 7f663dc00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f663806dac0 con 0x7f663806b7d0 2024-03-24T01:25:37.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.637+0000 7f663dc00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f663806dda0 con 0x7f663815bc50 2024-03-24T01:25:37.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.637+0000 7f663dc00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66380b4da0 con 0x7f66380aeeb0 2024-03-24T01:25:37.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.637+0000 7f6636a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f66380aeeb0 0x7f66380b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:37.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.637+0000 7f6637400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f663806b7d0 0x7f66380b2890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:32832/0 (socket says 172.21.15.90:32832) 2024-03-24T01:25:37.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.637+0000 7f6636a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f66380aeeb0 0x7f66380b44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55216/0 (socket says 172.21.15.90:55216) 2024-03-24T01:25:37.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.637+0000 7f6637400640 1 -- 172.21.15.90:0/159686940 learned_addr learned my addr 172.21.15.90:0/159686940 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:37.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.637+0000 7f6636a00640 1 -- 172.21.15.90:0/159686940 >> v1:172.21.15.120:6789/0 conn(0x7f663815bc50 legacy=0x7f663806b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:37.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.637+0000 7f6636a00640 1 -- 172.21.15.90:0/159686940 >> v1:172.21.15.90:6789/0 conn(0x7f663806b7d0 legacy=0x7f66380b2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:37.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.637+0000 7f6636a00640 1 -- 172.21.15.90:0/159686940 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f663806e080 con 0x7f66380aeeb0 2024-03-24T01:25:37.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.637+0000 7f6636000640 1 -- 172.21.15.90:0/159686940 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1475158928 0 0) 0x7f663806dac0 con 0x7f663806b7d0 2024-03-24T01:25:37.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.638+0000 7f6636a00640 1 --2- 172.21.15.90:0/159686940 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66380aeeb0 0x7f66380b44c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f66280097a0 tx=0x7f662802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fa8043d80be0a95 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:37.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.639+0000 7f6636000640 1 -- 172.21.15.90:0/159686940 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66280315a0 con 0x7f66380aeeb0 2024-03-24T01:25:37.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.639+0000 7f6636000640 1 -- 172.21.15.90:0/159686940 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6628037070 con 0x7f66380aeeb0 2024-03-24T01:25:37.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.639+0000 7f6636000640 1 -- 172.21.15.90:0/159686940 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6628031960 con 0x7f66380aeeb0 2024-03-24T01:25:37.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.639+0000 7f663dc00640 1 -- 172.21.15.90:0/159686940 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66380aeeb0 msgr2=0x7f66380b44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:37.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.639+0000 7f663dc00640 1 --2- 172.21.15.90:0/159686940 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66380aeeb0 0x7f66380b44c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f66280097a0 tx=0x7f662802b840 comp rx=0 tx=0).stop 2024-03-24T01:25:37.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.640+0000 7f663dc00640 1 -- 172.21.15.90:0/159686940 shutdown_connections 2024-03-24T01:25:37.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.640+0000 7f663dc00640 1 --2- 172.21.15.90:0/159686940 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66380aeeb0 0x7f66380b44c0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:37.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.640+0000 7f663dc00640 1 -- 172.21.15.90:0/159686940 >> 172.21.15.90:0/159686940 conn(0x7f66380b1770 msgr2=0x7f66380b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:37.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.640+0000 7f663dc00640 1 -- 172.21.15.90:0/159686940 shutdown_connections 2024-03-24T01:25:37.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.640+0000 7f663dc00640 1 -- 172.21.15.90:0/159686940 wait complete. 2024-03-24T01:25:37.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.641+0000 7f663dc00640 1 Processor -- start 2024-03-24T01:25:37.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.641+0000 7f663dc00640 1 -- start start 2024-03-24T01:25:37.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.642+0000 7f663dc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f663806b7d0 0x7f66381506c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:37.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.642+0000 7f663dc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66380aeeb0 0x7f66380b0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:37.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.642+0000 7f663dc00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f663815bc50 0x7f66380ad2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:37.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.642+0000 7f663dc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66380b4a00 con 0x7f66380aeeb0 2024-03-24T01:25:37.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.642+0000 7f663dc00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66380af290 con 0x7f663815bc50 2024-03-24T01:25:37.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.642+0000 7f663dc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66380b4b80 con 0x7f663806b7d0 2024-03-24T01:25:37.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.643+0000 7f662e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66380aeeb0 0x7f66380b0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:37.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.643+0000 7f662e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66380aeeb0 0x7f66380b0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55232/0 (socket says 172.21.15.90:55232) 2024-03-24T01:25:37.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.643+0000 7f662e000640 1 -- 172.21.15.90:0/366284322 learned_addr learned my addr 172.21.15.90:0/366284322 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:37.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.643+0000 7f6636a00640 1 --2- 172.21.15.90:0/366284322 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f663806b7d0 0x7f66381506c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:37.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.643+0000 7f6637400640 1 --2- 172.21.15.90:0/366284322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f663815bc50 0x7f66380ad2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:37.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.643+0000 7f662e000640 1 -- 172.21.15.90:0/366284322 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f663806b7d0 msgr2=0x7f66381506c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:37.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.643+0000 7f662e000640 1 --2- 172.21.15.90:0/366284322 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f663806b7d0 0x7f66381506c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:37.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.643+0000 7f662e000640 1 -- 172.21.15.90:0/366284322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f663815bc50 msgr2=0x7f66380ad2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:37.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.643+0000 7f662e000640 1 --2- 172.21.15.90:0/366284322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f663815bc50 0x7f66380ad2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:37.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.643+0000 7f662e000640 1 -- 172.21.15.90:0/366284322 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f663814f3a0 con 0x7f66380aeeb0 2024-03-24T01:25:37.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.643+0000 7f6637400640 1 --2- 172.21.15.90:0/366284322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f663815bc50 0x7f66380ad2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:37.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.643+0000 7f662e000640 1 --2- 172.21.15.90:0/366284322 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66380aeeb0 0x7f66380b0890 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f662000c400 tx=0x7f662000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:37.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.644+0000 7f662f400640 1 -- 172.21.15.90:0/366284322 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6620017070 con 0x7f66380aeeb0 2024-03-24T01:25:37.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.644+0000 7f662f400640 1 -- 172.21.15.90:0/366284322 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66200052d0 con 0x7f66380aeeb0 2024-03-24T01:25:37.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.644+0000 7f662f400640 1 -- 172.21.15.90:0/366284322 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6620003830 con 0x7f66380aeeb0 2024-03-24T01:25:37.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.644+0000 7f663dc00640 1 -- 172.21.15.90:0/366284322 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f663814f620 con 0x7f66380aeeb0 2024-03-24T01:25:37.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.644+0000 7f663dc00640 1 -- 172.21.15.90:0/366284322 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f663814fbe0 con 0x7f66380aeeb0 2024-03-24T01:25:37.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.645+0000 7f663dc00640 1 -- 172.21.15.90:0/366284322 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f663804f400 con 0x7f66380aeeb0 2024-03-24T01:25:37.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.646+0000 7f662f400640 1 -- 172.21.15.90:0/366284322 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f662001a070 con 0x7f66380aeeb0 2024-03-24T01:25:37.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.647+0000 7f662f400640 1 --2- 172.21.15.90:0/366284322 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f661406bbd0 0x7f661406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:37.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.647+0000 7f662f400640 1 -- 172.21.15.90:0/366284322 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f662008a2d0 con 0x7f66380aeeb0 2024-03-24T01:25:37.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.648+0000 7f6636a00640 1 --2- 172.21.15.90:0/366284322 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f661406bbd0 0x7f661406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:37.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.649+0000 7f6636a00640 1 --2- 172.21.15.90:0/366284322 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f661406bbd0 0x7f661406e090 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6628036000 tx=0x7f6628008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:37.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.651+0000 7f662f400640 1 -- 172.21.15.90:0/366284322 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f663814f3a0 con 0x7f66380aeeb0 2024-03-24T01:25:37.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.856+0000 7f663dc00640 1 -- 172.21.15.90:0/366284322 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f6638156d40 con 0x7f66380aeeb0 2024-03-24T01:25:37.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.857+0000 7f662f400640 1 -- 172.21.15.90:0/366284322 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f6620058900 con 0x7f66380aeeb0 2024-03-24T01:25:37.857 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:25:37.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.859+0000 7f663dc00640 1 -- 172.21.15.90:0/366284322 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f661406bbd0 msgr2=0x7f661406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:37.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.859+0000 7f663dc00640 1 --2- 172.21.15.90:0/366284322 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f661406bbd0 0x7f661406e090 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6628036000 tx=0x7f6628008030 comp rx=0 tx=0).stop 2024-03-24T01:25:37.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.859+0000 7f663dc00640 1 -- 172.21.15.90:0/366284322 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66380aeeb0 msgr2=0x7f66380b0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:37.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.859+0000 7f663dc00640 1 --2- 172.21.15.90:0/366284322 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66380aeeb0 0x7f66380b0890 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f662000c400 tx=0x7f662000c8c0 comp rx=0 tx=0).stop 2024-03-24T01:25:37.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.859+0000 7f663dc00640 1 -- 172.21.15.90:0/366284322 shutdown_connections 2024-03-24T01:25:37.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.859+0000 7f663dc00640 1 --2- 172.21.15.90:0/366284322 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f661406bbd0 0x7f661406e090 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:37.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.859+0000 7f663dc00640 1 --2- 172.21.15.90:0/366284322 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f66380aeeb0 0x7f66380b0890 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:37.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.859+0000 7f663dc00640 1 --2- 172.21.15.90:0/366284322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f663815bc50 0x7f66380ad2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:37.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.859+0000 7f663dc00640 1 --2- 172.21.15.90:0/366284322 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f663806b7d0 0x7f66381506c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:37.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.859+0000 7f663dc00640 1 -- 172.21.15.90:0/366284322 >> 172.21.15.90:0/366284322 conn(0x7f66380b1770 msgr2=0x7f6638157260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:37.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.860+0000 7f663dc00640 1 -- 172.21.15.90:0/366284322 shutdown_connections 2024-03-24T01:25:37.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:37.860+0000 7f663dc00640 1 -- 172.21.15.90:0/366284322 wait complete. 2024-03-24T01:25:37.877 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T01:25:38.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.087+0000 7f29ebc00640 1 Processor -- start 2024-03-24T01:25:38.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.087+0000 7f29ebc00640 1 -- start start 2024-03-24T01:25:38.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.089+0000 7f29ebc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29e4159620 0x7f29e406b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:38.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.089+0000 7f29ebc00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29e406bca0 con 0x7f29e40cf150 2024-03-24T01:25:38.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.089+0000 7f29ebc00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29e40b36f0 con 0x7f29e406b600 2024-03-24T01:25:38.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.089+0000 7f29ebc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e40b4470 con 0x7f29e4159620 2024-03-24T01:25:38.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.089+0000 7f29e9c00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f29e40cf150 0x7f29e40ae530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:32848/0 (socket says 172.21.15.90:32848) 2024-03-24T01:25:38.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.089+0000 7f29e9c00640 1 -- 172.21.15.90:0/2828708631 learned_addr learned my addr 172.21.15.90:0/2828708631 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:38.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.090+0000 7f29e9200640 1 --2- 172.21.15.90:0/2828708631 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29e4159620 0x7f29e406b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:38.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.090+0000 7f29e3400640 1 -- 172.21.15.90:0/2828708631 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 824258277 0 0) 0x7f29e406bca0 con 0x7f29e40cf150 2024-03-24T01:25:38.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.090+0000 7f29e3400640 1 -- 172.21.15.90:0/2828708631 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29cc003520 con 0x7f29e40cf150 2024-03-24T01:25:38.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.090+0000 7f29e3400640 1 -- 172.21.15.90:0/2828708631 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1141398339 0 0) 0x7f29cc003520 con 0x7f29e40cf150 2024-03-24T01:25:38.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.090+0000 7f29e3400640 1 -- 172.21.15.90:0/2828708631 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29e4159620 msgr2=0x7f29e406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:38.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.090+0000 7f29e3400640 1 --2- 172.21.15.90:0/2828708631 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29e4159620 0x7f29e406b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:38.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.090+0000 7f29e3400640 1 -- 172.21.15.90:0/2828708631 >> v1:172.21.15.120:6789/0 conn(0x7f29e406b600 legacy=0x7f29e40cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:38.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.090+0000 7f29e3400640 1 -- 172.21.15.90:0/2828708631 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e40b3990 con 0x7f29e40cf150 2024-03-24T01:25:38.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.090+0000 7f29e3400640 1 -- 172.21.15.90:0/2828708631 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f29dc0019f0 con 0x7f29e40cf150 2024-03-24T01:25:38.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.090+0000 7f29e3400640 1 -- 172.21.15.90:0/2828708631 >> v1:172.21.15.90:6789/0 conn(0x7f29e40cf150 legacy=0x7f29e40ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:38.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.090+0000 7f29e3400640 1 --2- 172.21.15.90:0/2828708631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29cc0039e0 0x7f29cc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:38.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.090+0000 7f29e3400640 1 --2- 172.21.15.90:0/2828708631 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29e4159620 0x7f29cc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:38.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.090+0000 7f29e3400640 1 --2- 172.21.15.90:0/2828708631 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29cc004d40 0x7f29cc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:38.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.090+0000 7f29e3400640 1 -- 172.21.15.90:0/2828708631 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e40b4470 con 0x7f29cc0039e0 2024-03-24T01:25:38.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.090+0000 7f29e3400640 1 -- 172.21.15.90:0/2828708631 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29cc00ba70 con 0x7f29cc004d40 2024-03-24T01:25:38.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.090+0000 7f29e3400640 1 -- 172.21.15.90:0/2828708631 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29cc00bc70 con 0x7f29e4159620 2024-03-24T01:25:38.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.091+0000 7f29e9200640 1 --2- 172.21.15.90:0/2828708631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29cc0039e0 0x7f29cc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:38.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.091+0000 7f29e3e00640 1 --2- 172.21.15.90:0/2828708631 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29e4159620 0x7f29cc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:38.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.091+0000 7f29e9200640 1 -- 172.21.15.90:0/2828708631 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29e4159620 msgr2=0x7f29cc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:38.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.091+0000 7f29e9200640 1 --2- 172.21.15.90:0/2828708631 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29e4159620 0x7f29cc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:38.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.091+0000 7f29e9200640 1 -- 172.21.15.90:0/2828708631 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29cc004d40 msgr2=0x7f29cc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:38.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.091+0000 7f29e9200640 1 --2- 172.21.15.90:0/2828708631 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29cc004d40 0x7f29cc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:38.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.091+0000 7f29e9200640 1 -- 172.21.15.90:0/2828708631 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29cc003520 con 0x7f29cc0039e0 2024-03-24T01:25:38.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.091+0000 7f29e3e00640 1 --2- 172.21.15.90:0/2828708631 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29e4159620 0x7f29cc004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:38.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.092+0000 7f29e9200640 1 --2- 172.21.15.90:0/2828708631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29cc0039e0 0x7f29cc003dc0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f29d8009ac0 tx=0x7f29cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23c9d533207161c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:38.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.092+0000 7f29e3400640 1 -- 172.21.15.90:0/2828708631 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d8039070 con 0x7f29cc0039e0 2024-03-24T01:25:38.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.092+0000 7f29e3400640 1 -- 172.21.15.90:0/2828708631 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d802c6b0 con 0x7f29cc0039e0 2024-03-24T01:25:38.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.092+0000 7f29e3400640 1 -- 172.21.15.90:0/2828708631 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d8009690 con 0x7f29cc0039e0 2024-03-24T01:25:38.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.093+0000 7f29ebc00640 1 -- 172.21.15.90:0/2828708631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29cc0039e0 msgr2=0x7f29cc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:38.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.093+0000 7f29ebc00640 1 --2- 172.21.15.90:0/2828708631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29cc0039e0 0x7f29cc003dc0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f29d8009ac0 tx=0x7f29cc005830 comp rx=0 tx=0).stop 2024-03-24T01:25:38.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.093+0000 7f29e9c00640 1 -- 172.21.15.90:0/2828708631 reap_dead start 2024-03-24T01:25:38.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.093+0000 7f29ebc00640 1 -- 172.21.15.90:0/2828708631 shutdown_connections 2024-03-24T01:25:38.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.093+0000 7f29ebc00640 1 -- 172.21.15.90:0/2828708631 >> 172.21.15.90:0/2828708631 conn(0x7f29e40b1770 msgr2=0x7f29e40b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:38.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.093+0000 7f29ebc00640 1 -- 172.21.15.90:0/2828708631 shutdown_connections 2024-03-24T01:25:38.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.093+0000 7f29ebc00640 1 -- 172.21.15.90:0/2828708631 wait complete. 2024-03-24T01:25:38.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.095+0000 7f29ebc00640 1 Processor -- start 2024-03-24T01:25:38.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.095+0000 7f29ebc00640 1 -- start start 2024-03-24T01:25:38.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.096+0000 7f29ebc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29cc0039e0 0x7f29e40c9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:38.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.096+0000 7f29ebc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29e41f41c0 0x7f29e413dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:38.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.096+0000 7f29ebc00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29e415be00 0x7f29e415c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:38.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.096+0000 7f29ebc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e40b3f50 con 0x7f29e41f41c0 2024-03-24T01:25:38.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.096+0000 7f29ebc00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e406ba60 con 0x7f29e415be00 2024-03-24T01:25:38.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.096+0000 7f29ebc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e4159a00 con 0x7f29cc0039e0 2024-03-24T01:25:38.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.096+0000 7f29e3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29e41f41c0 0x7f29e413dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:38.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.096+0000 7f29e3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29e41f41c0 0x7f29e413dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55238/0 (socket says 172.21.15.90:55238) 2024-03-24T01:25:38.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.096+0000 7f29e3e00640 1 -- 172.21.15.90:0/1074182214 learned_addr learned my addr 172.21.15.90:0/1074182214 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:38.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.097+0000 7f29e9200640 1 --2- 172.21.15.90:0/1074182214 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29cc0039e0 0x7f29e40c9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:38.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.097+0000 7f29e3e00640 1 -- 172.21.15.90:0/1074182214 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29cc0039e0 msgr2=0x7f29e40c9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:38.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.097+0000 7f29e3e00640 1 --2- 172.21.15.90:0/1074182214 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29cc0039e0 0x7f29e40c9620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:38.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.097+0000 7f29e3e00640 1 -- 172.21.15.90:0/1074182214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29e415be00 msgr2=0x7f29e415c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:38.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.097+0000 7f29e3e00640 1 --2- 172.21.15.90:0/1074182214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29e415be00 0x7f29e415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:38.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.097+0000 7f29e3e00640 1 -- 172.21.15.90:0/1074182214 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e41f6f00 con 0x7f29e41f41c0 2024-03-24T01:25:38.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.097+0000 7f29e3e00640 1 --2- 172.21.15.90:0/1074182214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29e41f41c0 0x7f29e413dda0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f29d0000f60 tx=0x7f29d000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:38.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.097+0000 7f29e0c00640 1 -- 172.21.15.90:0/1074182214 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d0005480 con 0x7f29e41f41c0 2024-03-24T01:25:38.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.097+0000 7f29e0c00640 1 -- 172.21.15.90:0/1074182214 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d0017070 con 0x7f29e41f41c0 2024-03-24T01:25:38.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.097+0000 7f29e0c00640 1 -- 172.21.15.90:0/1074182214 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d000b8d0 con 0x7f29e41f41c0 2024-03-24T01:25:38.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.097+0000 7f29ebc00640 1 -- 172.21.15.90:0/1074182214 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29e41f71e0 con 0x7f29e41f41c0 2024-03-24T01:25:38.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.097+0000 7f29ebc00640 1 -- 172.21.15.90:0/1074182214 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29e41f4640 con 0x7f29e41f41c0 2024-03-24T01:25:38.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.098+0000 7f29ebc00640 1 -- 172.21.15.90:0/1074182214 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29e404f400 con 0x7f29e41f41c0 2024-03-24T01:25:38.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.099+0000 7f29e0c00640 1 -- 172.21.15.90:0/1074182214 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f29d000ba70 con 0x7f29e41f41c0 2024-03-24T01:25:38.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.100+0000 7f29e0c00640 1 --2- 172.21.15.90:0/1074182214 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f29c406bbd0 0x7f29c406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:38.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.100+0000 7f29e0c00640 1 -- 172.21.15.90:0/1074182214 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f29d008ae90 con 0x7f29e41f41c0 2024-03-24T01:25:38.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.101+0000 7f29e9200640 1 --2- 172.21.15.90:0/1074182214 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f29c406bbd0 0x7f29c406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:38.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.102+0000 7f29e9200640 1 --2- 172.21.15.90:0/1074182214 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f29c406bbd0 0x7f29c406e090 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f29d8036000 tx=0x7f29d8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:38.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.105+0000 7f29e0c00640 1 -- 172.21.15.90:0/1074182214 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f29d0059560 con 0x7f29e41f41c0 2024-03-24T01:25:38.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.316+0000 7f29ebc00640 1 -- 172.21.15.90:0/1074182214 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f29e40b2e90 con 0x7f29e41f41c0 2024-03-24T01:25:38.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.317+0000 7f29e0c00640 1 -- 172.21.15.90:0/1074182214 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f29d0058e60 con 0x7f29e41f41c0 2024-03-24T01:25:38.317 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-24T01:25:38.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.320+0000 7f29ebc00640 1 -- 172.21.15.90:0/1074182214 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f29c406bbd0 msgr2=0x7f29c406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:38.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.320+0000 7f29ebc00640 1 --2- 172.21.15.90:0/1074182214 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f29c406bbd0 0x7f29c406e090 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f29d8036000 tx=0x7f29d8030000 comp rx=0 tx=0).stop 2024-03-24T01:25:38.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.320+0000 7f29ebc00640 1 -- 172.21.15.90:0/1074182214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29e41f41c0 msgr2=0x7f29e413dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:38.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.320+0000 7f29ebc00640 1 --2- 172.21.15.90:0/1074182214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29e41f41c0 0x7f29e413dda0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f29d0000f60 tx=0x7f29d000dc30 comp rx=0 tx=0).stop 2024-03-24T01:25:38.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.320+0000 7f29ebc00640 1 -- 172.21.15.90:0/1074182214 shutdown_connections 2024-03-24T01:25:38.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.320+0000 7f29ebc00640 1 --2- 172.21.15.90:0/1074182214 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f29c406bbd0 0x7f29c406e090 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:38.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.320+0000 7f29ebc00640 1 --2- 172.21.15.90:0/1074182214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29e41f41c0 0x7f29e413dda0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:38.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.320+0000 7f29ebc00640 1 --2- 172.21.15.90:0/1074182214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29e415be00 0x7f29e415c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:38.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.320+0000 7f29ebc00640 1 --2- 172.21.15.90:0/1074182214 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f29cc0039e0 0x7f29e40c9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:38.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.320+0000 7f29ebc00640 1 -- 172.21.15.90:0/1074182214 >> 172.21.15.90:0/1074182214 conn(0x7f29e40b1770 msgr2=0x7f29e41561d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:38.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.320+0000 7f29ebc00640 1 -- 172.21.15.90:0/1074182214 shutdown_connections 2024-03-24T01:25:38.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.320+0000 7f29ebc00640 1 -- 172.21.15.90:0/1074182214 wait complete. 2024-03-24T01:25:38.337 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:25:38.337 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:25:38.391 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:25:38.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.606+0000 7ffa04a00640 1 Processor -- start 2024-03-24T01:25:38.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.606+0000 7ffa04a00640 1 -- start start 2024-03-24T01:25:38.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.608+0000 7ffa04a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ffa0006b7d0 0x7ffa00166a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:38.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.608+0000 7ffa04a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa000cd490 con 0x7ffa000cc150 2024-03-24T01:25:38.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.608+0000 7ffa04a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa0015f900 con 0x7ffa001603e0 2024-03-24T01:25:38.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.608+0000 7ffa04a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa000cd270 con 0x7ffa0006b7d0 2024-03-24T01:25:38.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.608+0000 7ff9fde00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ffa000cc150 0x7ffa000ccb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:32856/0 (socket says 172.21.15.90:32856) 2024-03-24T01:25:38.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.608+0000 7ff9fde00640 1 -- 172.21.15.90:0/3962484513 learned_addr learned my addr 172.21.15.90:0/3962484513 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:38.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.608+0000 7ff9fe800640 1 --2- 172.21.15.90:0/3962484513 >> v2:172.21.15.90:3300/0 conn(0x7ffa0006b7d0 0x7ffa00166a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:38.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.608+0000 7ff9fd400640 1 -- 172.21.15.90:0/3962484513 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2679360124 0 0) 0x7ffa000cd490 con 0x7ffa000cc150 2024-03-24T01:25:38.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.608+0000 7ff9fe800640 1 -- 172.21.15.90:0/3962484513 >> v1:172.21.15.120:6789/0 conn(0x7ffa001603e0 legacy=0x7ffa0006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:38.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.608+0000 7ff9fe800640 1 -- 172.21.15.90:0/3962484513 >> v1:172.21.15.90:6789/0 conn(0x7ffa000cc150 legacy=0x7ffa000ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:38.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.608+0000 7ff9fe800640 1 -- 172.21.15.90:0/3962484513 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa0015fbe0 con 0x7ffa0006b7d0 2024-03-24T01:25:38.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.609+0000 7ff9fe800640 1 --2- 172.21.15.90:0/3962484513 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa0006b7d0 0x7ffa00166a80 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8009d90 tx=0x7ff9f802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c54eebf26bb6dc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:38.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.610+0000 7ff9fd400640 1 -- 172.21.15.90:0/3962484513 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f8039070 con 0x7ffa0006b7d0 2024-03-24T01:25:38.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.610+0000 7ff9fd400640 1 -- 172.21.15.90:0/3962484513 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9f802cc60 con 0x7ffa0006b7d0 2024-03-24T01:25:38.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.610+0000 7ff9fd400640 1 -- 172.21.15.90:0/3962484513 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f8034660 con 0x7ffa0006b7d0 2024-03-24T01:25:38.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.610+0000 7ffa04a00640 1 -- 172.21.15.90:0/3962484513 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa0006b7d0 msgr2=0x7ffa00166a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:38.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.610+0000 7ffa04a00640 1 --2- 172.21.15.90:0/3962484513 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa0006b7d0 0x7ffa00166a80 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8009d90 tx=0x7ff9f802beb0 comp rx=0 tx=0).stop 2024-03-24T01:25:38.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.610+0000 7ffa04a00640 1 -- 172.21.15.90:0/3962484513 shutdown_connections 2024-03-24T01:25:38.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.610+0000 7ffa04a00640 1 --2- 172.21.15.90:0/3962484513 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa0006b7d0 0x7ffa00166a80 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:38.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.610+0000 7ffa04a00640 1 -- 172.21.15.90:0/3962484513 >> 172.21.15.90:0/3962484513 conn(0x7ffa000b1770 msgr2=0x7ffa000b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:38.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.611+0000 7ffa04a00640 1 -- 172.21.15.90:0/3962484513 shutdown_connections 2024-03-24T01:25:38.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.611+0000 7ffa04a00640 1 -- 172.21.15.90:0/3962484513 wait complete. 2024-03-24T01:25:38.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.612+0000 7ffa04a00640 1 Processor -- start 2024-03-24T01:25:38.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.612+0000 7ffa04a00640 1 -- start start 2024-03-24T01:25:38.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.613+0000 7ffa04a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa0006b7d0 0x7ffa0015ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:38.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.613+0000 7ffa04a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa000cc150 0x7ffa0013b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:38.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.613+0000 7ffa04a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa001603e0 0x7ffa001f20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:38.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.613+0000 7ffa04a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa001455b0 con 0x7ffa0006b7d0 2024-03-24T01:25:38.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.613+0000 7ffa04a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa00145430 con 0x7ffa001603e0 2024-03-24T01:25:38.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.613+0000 7ffa04a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa00166fc0 con 0x7ffa000cc150 2024-03-24T01:25:38.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.613+0000 7ff9fde00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa0006b7d0 0x7ffa0015ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:38.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.613+0000 7ff9fde00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa0006b7d0 0x7ffa0015ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55248/0 (socket says 172.21.15.90:55248) 2024-03-24T01:25:38.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.613+0000 7ff9fde00640 1 -- 172.21.15.90:0/3397622168 learned_addr learned my addr 172.21.15.90:0/3397622168 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:38.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.614+0000 7ff9fde00640 1 -- 172.21.15.90:0/3397622168 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa000cc150 msgr2=0x7ffa0013b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:38.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.614+0000 7ff9fde00640 1 --2- 172.21.15.90:0/3397622168 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa000cc150 0x7ffa0013b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:38.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.614+0000 7ff9fde00640 1 -- 172.21.15.90:0/3397622168 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa001603e0 msgr2=0x7ffa001f20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:38.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.614+0000 7ff9fde00640 1 --2- 172.21.15.90:0/3397622168 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa001603e0 0x7ffa001f20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:38.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.614+0000 7ff9fde00640 1 -- 172.21.15.90:0/3397622168 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa0015efb0 con 0x7ffa0006b7d0 2024-03-24T01:25:38.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.614+0000 7ff9fde00640 1 --2- 172.21.15.90:0/3397622168 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa0006b7d0 0x7ffa0015ea70 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7ff9f000c4d0 tx=0x7ff9f000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:38.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.614+0000 7ff9f6a00640 1 -- 172.21.15.90:0/3397622168 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f0018070 con 0x7ffa0006b7d0 2024-03-24T01:25:38.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.614+0000 7ff9f6a00640 1 -- 172.21.15.90:0/3397622168 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9f0007500 con 0x7ffa0006b7d0 2024-03-24T01:25:38.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.614+0000 7ffa04a00640 1 -- 172.21.15.90:0/3397622168 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa0015d9d0 con 0x7ffa0006b7d0 2024-03-24T01:25:38.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.614+0000 7ff9f6a00640 1 -- 172.21.15.90:0/3397622168 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f0004a20 con 0x7ffa0006b7d0 2024-03-24T01:25:38.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.615+0000 7ffa04a00640 1 -- 172.21.15.90:0/3397622168 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa0015df90 con 0x7ffa0006b7d0 2024-03-24T01:25:38.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.616+0000 7ffa04a00640 1 -- 172.21.15.90:0/3397622168 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa0015e390 con 0x7ffa0006b7d0 2024-03-24T01:25:38.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.617+0000 7ff9f6a00640 1 -- 172.21.15.90:0/3397622168 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff9f001b070 con 0x7ffa0006b7d0 2024-03-24T01:25:38.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.617+0000 7ff9f6a00640 1 --2- 172.21.15.90:0/3397622168 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff9dc06bbd0 0x7ff9dc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:38.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.617+0000 7ff9f6a00640 1 -- 172.21.15.90:0/3397622168 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7ff9f008a230 con 0x7ffa0006b7d0 2024-03-24T01:25:38.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.618+0000 7ff9f5400640 1 --2- 172.21.15.90:0/3397622168 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff9dc06bbd0 0x7ff9dc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:38.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.619+0000 7ff9f5400640 1 --2- 172.21.15.90:0/3397622168 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff9dc06bbd0 0x7ff9dc06e090 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff9e8006fd0 tx=0x7ff9e8006bc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:38.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.622+0000 7ff9f6a00640 1 -- 172.21.15.90:0/3397622168 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff9f0058860 con 0x7ffa0006b7d0 2024-03-24T01:25:38.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.827+0000 7ffa04a00640 1 -- 172.21.15.90:0/3397622168 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa0013cb70 con 0x7ffa0006b7d0 2024-03-24T01:25:38.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.830+0000 7ff9f6a00640 1 -- 172.21.15.90:0/3397622168 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18925 (secure 0 0 0) 0x7ff9f0058160 con 0x7ffa0006b7d0 2024-03-24T01:25:38.831 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:25:38.831 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:24:53.236401+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:24:08.643966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T01:24:09.189904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T01:24:13.027323+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:25:38.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.833+0000 7ffa04a00640 1 -- 172.21.15.90:0/3397622168 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff9dc06bbd0 msgr2=0x7ff9dc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:38.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.833+0000 7ffa04a00640 1 --2- 172.21.15.90:0/3397622168 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff9dc06bbd0 0x7ff9dc06e090 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff9e8006fd0 tx=0x7ff9e8006bc0 comp rx=0 tx=0).stop 2024-03-24T01:25:38.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.834+0000 7ffa04a00640 1 -- 172.21.15.90:0/3397622168 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa0006b7d0 msgr2=0x7ffa0015ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:38.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.834+0000 7ffa04a00640 1 --2- 172.21.15.90:0/3397622168 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa0006b7d0 0x7ffa0015ea70 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7ff9f000c4d0 tx=0x7ff9f000c990 comp rx=0 tx=0).stop 2024-03-24T01:25:38.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.834+0000 7ffa04a00640 1 -- 172.21.15.90:0/3397622168 shutdown_connections 2024-03-24T01:25:38.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.834+0000 7ffa04a00640 1 --2- 172.21.15.90:0/3397622168 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff9dc06bbd0 0x7ff9dc06e090 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:38.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.834+0000 7ffa04a00640 1 --2- 172.21.15.90:0/3397622168 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa001603e0 0x7ffa001f20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:38.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.834+0000 7ffa04a00640 1 --2- 172.21.15.90:0/3397622168 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa000cc150 0x7ffa0013b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:38.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.834+0000 7ffa04a00640 1 --2- 172.21.15.90:0/3397622168 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa0006b7d0 0x7ffa0015ea70 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:38.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.834+0000 7ffa04a00640 1 -- 172.21.15.90:0/3397622168 >> 172.21.15.90:0/3397622168 conn(0x7ffa000b1770 msgr2=0x7ffa0013c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:38.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.834+0000 7ffa04a00640 1 -- 172.21.15.90:0/3397622168 shutdown_connections 2024-03-24T01:25:38.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:38.834+0000 7ffa04a00640 1 -- 172.21.15.90:0/3397622168 wait complete. 2024-03-24T01:25:38.854 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T01:24:08.643966+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T01:24:09.189904+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T01:24:13.027323+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T01:25:38.854 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:25:39.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.063+0000 7f0037800640 1 Processor -- start 2024-03-24T01:25:39.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.063+0000 7f0037800640 1 -- start start 2024-03-24T01:25:39.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.064+0000 7f0037800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f003015bc50 0x7f003006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:39.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.064+0000 7f0037800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f003006dac0 con 0x7f00300aeeb0 2024-03-24T01:25:39.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.065+0000 7f0037800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f003006dda0 con 0x7f003006b600 2024-03-24T01:25:39.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.065+0000 7f0037800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f003006ba70 con 0x7f003015bc50 2024-03-24T01:25:39.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.065+0000 7f0034e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f00300aeeb0 0x7f00300b44c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:32872/0 (socket says 172.21.15.90:32872) 2024-03-24T01:25:39.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.065+0000 7f0034e00640 1 -- 172.21.15.90:0/1558229348 learned_addr learned my addr 172.21.15.90:0/1558229348 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:39.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.065+0000 7f002fe00640 1 -- 172.21.15.90:0/1558229348 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2674589962 0 0) 0x7f003006dac0 con 0x7f00300aeeb0 2024-03-24T01:25:39.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.065+0000 7f002fe00640 1 -- 172.21.15.90:0/1558229348 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0014005730 con 0x7f00300aeeb0 2024-03-24T01:25:39.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.065+0000 7f0027e00640 1 --2- 172.21.15.90:0/1558229348 >> v2:172.21.15.120:3300/0 conn(0x7f003015bc50 0x7f003006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:39.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.066+0000 7f002fe00640 1 -- 172.21.15.90:0/1558229348 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 289841306 0 0) 0x7f0014005730 con 0x7f00300aeeb0 2024-03-24T01:25:39.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.066+0000 7f002fe00640 1 -- 172.21.15.90:0/1558229348 >> v2:172.21.15.120:3300/0 conn(0x7f003015bc50 msgr2=0x7f003006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:39.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.066+0000 7f002fe00640 1 --2- 172.21.15.90:0/1558229348 >> v2:172.21.15.120:3300/0 conn(0x7f003015bc50 0x7f003006b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:39.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.066+0000 7f002fe00640 1 -- 172.21.15.90:0/1558229348 >> v1:172.21.15.120:6789/0 conn(0x7f003006b600 legacy=0x7f00300b2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:39.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.066+0000 7f002fe00640 1 -- 172.21.15.90:0/1558229348 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f003006e080 con 0x7f00300aeeb0 2024-03-24T01:25:39.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.066+0000 7f002fe00640 1 -- 172.21.15.90:0/1558229348 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f0020001670 con 0x7f00300aeeb0 2024-03-24T01:25:39.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.066+0000 7f002fe00640 1 -- 172.21.15.90:0/1558229348 >> v1:172.21.15.90:6789/0 conn(0x7f00300aeeb0 legacy=0x7f00300b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:39.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.066+0000 7f002fe00640 1 --2- 172.21.15.90:0/1558229348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0014005bc0 0x7f0014005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:39.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.066+0000 7f002fe00640 1 --2- 172.21.15.90:0/1558229348 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0014006a10 0x7f0014006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:39.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.066+0000 7f002fe00640 1 --2- 172.21.15.90:0/1558229348 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00140075d0 0x7f001400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:39.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.066+0000 7f002fe00640 1 -- 172.21.15.90:0/1558229348 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f003006ba70 con 0x7f0014006a10 2024-03-24T01:25:39.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.066+0000 7f002fe00640 1 -- 172.21.15.90:0/1558229348 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f001400e460 con 0x7f0014005bc0 2024-03-24T01:25:39.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.066+0000 7f002fe00640 1 -- 172.21.15.90:0/1558229348 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f001400e660 con 0x7f00140075d0 2024-03-24T01:25:39.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.066+0000 7f0027e00640 1 --2- 172.21.15.90:0/1558229348 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0014006a10 0x7f0014006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:39.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.067+0000 7f0034e00640 1 --2- 172.21.15.90:0/1558229348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0014005bc0 0x7f0014005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:39.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.067+0000 7f0027e00640 1 -- 172.21.15.90:0/1558229348 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00140075d0 msgr2=0x7f001400dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:39.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.067+0000 7f0027e00640 1 --2- 172.21.15.90:0/1558229348 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00140075d0 0x7f001400dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:39.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.067+0000 7f0027e00640 1 -- 172.21.15.90:0/1558229348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0014005bc0 msgr2=0x7f0014005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:39.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.067+0000 7f0027e00640 1 --2- 172.21.15.90:0/1558229348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0014005bc0 0x7f0014005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:39.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.067+0000 7f0027e00640 1 -- 172.21.15.90:0/1558229348 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0014005730 con 0x7f0014006a10 2024-03-24T01:25:39.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.067+0000 7f0035800640 1 -- 172.21.15.90:0/1558229348 reap_dead start 2024-03-24T01:25:39.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.067+0000 7f0034e00640 1 --2- 172.21.15.90:0/1558229348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0014005bc0 0x7f0014005fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:39.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.068+0000 7f0027e00640 1 --2- 172.21.15.90:0/1558229348 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0014006a10 0x7f0014006e90 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f001800ba30 tx=0x7f0014007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24893a2cc871877 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:39.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.068+0000 7f002fe00640 1 -- 172.21.15.90:0/1558229348 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0018036020 con 0x7f0014006a10 2024-03-24T01:25:39.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.068+0000 7f002fe00640 1 -- 172.21.15.90:0/1558229348 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0018032680 con 0x7f0014006a10 2024-03-24T01:25:39.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.068+0000 7f002fe00640 1 -- 172.21.15.90:0/1558229348 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f001800b600 con 0x7f0014006a10 2024-03-24T01:25:39.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.069+0000 7f0037800640 1 -- 172.21.15.90:0/1558229348 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0014006a10 msgr2=0x7f0014006e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:39.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.069+0000 7f0037800640 1 --2- 172.21.15.90:0/1558229348 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0014006a10 0x7f0014006e90 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f001800ba30 tx=0x7f0014007a40 comp rx=0 tx=0).stop 2024-03-24T01:25:39.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.069+0000 7f0037800640 1 -- 172.21.15.90:0/1558229348 shutdown_connections 2024-03-24T01:25:39.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.069+0000 7f0037800640 1 --2- 172.21.15.90:0/1558229348 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0014006a10 0x7f0014006e90 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:39.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.069+0000 7f0037800640 1 -- 172.21.15.90:0/1558229348 >> 172.21.15.90:0/1558229348 conn(0x7f00300b1770 msgr2=0x7f00300b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:39.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.069+0000 7f0037800640 1 -- 172.21.15.90:0/1558229348 shutdown_connections 2024-03-24T01:25:39.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.069+0000 7f0037800640 1 -- 172.21.15.90:0/1558229348 wait complete. 2024-03-24T01:25:39.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.071+0000 7f0037800640 1 Processor -- start 2024-03-24T01:25:39.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.071+0000 7f0037800640 1 -- start start 2024-03-24T01:25:39.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.072+0000 7f0037800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0014006a10 0x7f00300b2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:39.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.072+0000 7f0037800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00300ad9f0 0x7f00300ac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:39.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.072+0000 7f0037800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00300b2bc0 0x7f00300ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:39.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.072+0000 7f0037800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00300af290 con 0x7f00300b2bc0 2024-03-24T01:25:39.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.072+0000 7f0037800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00300b4da0 con 0x7f0014006a10 2024-03-24T01:25:39.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.072+0000 7f0037800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f003006e880 con 0x7f00300ad9f0 2024-03-24T01:25:39.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.072+0000 7f0035800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00300b2bc0 0x7f00300ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:39.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.072+0000 7f0035800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00300b2bc0 0x7f00300ace20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55270/0 (socket says 172.21.15.90:55270) 2024-03-24T01:25:39.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.072+0000 7f0035800640 1 -- 172.21.15.90:0/1608140243 learned_addr learned my addr 172.21.15.90:0/1608140243 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:39.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.072+0000 7f0034e00640 1 --2- 172.21.15.90:0/1608140243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0014006a10 0x7f00300b2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:39.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.073+0000 7f0027e00640 1 --2- 172.21.15.90:0/1608140243 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00300ad9f0 0x7f00300ac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:39.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.073+0000 7f0035800640 1 -- 172.21.15.90:0/1608140243 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00300ad9f0 msgr2=0x7f00300ac8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:39.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.073+0000 7f0035800640 1 --2- 172.21.15.90:0/1608140243 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00300ad9f0 0x7f00300ac8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:39.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.073+0000 7f0035800640 1 -- 172.21.15.90:0/1608140243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0014006a10 msgr2=0x7f00300b2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:39.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.073+0000 7f0035800640 1 --2- 172.21.15.90:0/1608140243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0014006a10 0x7f00300b2680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:39.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.073+0000 7f0035800640 1 -- 172.21.15.90:0/1608140243 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00301553b0 con 0x7f00300b2bc0 2024-03-24T01:25:39.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.073+0000 7f0035800640 1 --2- 172.21.15.90:0/1608140243 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00300b2bc0 0x7f00300ace20 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f00300cc5d0 tx=0x7f002800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:39.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.073+0000 7f002d600640 1 -- 172.21.15.90:0/1608140243 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0028017070 con 0x7f00300b2bc0 2024-03-24T01:25:39.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.073+0000 7f002d600640 1 -- 172.21.15.90:0/1608140243 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0028005d60 con 0x7f00300b2bc0 2024-03-24T01:25:39.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.073+0000 7f0037800640 1 -- 172.21.15.90:0/1608140243 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00300b3060 con 0x7f00300b2bc0 2024-03-24T01:25:39.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.073+0000 7f002d600640 1 -- 172.21.15.90:0/1608140243 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f002800a850 con 0x7f00300b2bc0 2024-03-24T01:25:39.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.074+0000 7f0037800640 1 -- 172.21.15.90:0/1608140243 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00300b3620 con 0x7f00300b2bc0 2024-03-24T01:25:39.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.075+0000 7f0037800640 1 -- 172.21.15.90:0/1608140243 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00300ccf40 con 0x7f00300b2bc0 2024-03-24T01:25:39.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.076+0000 7f002d600640 1 -- 172.21.15.90:0/1608140243 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f002801a070 con 0x7f00300b2bc0 2024-03-24T01:25:39.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.077+0000 7f002d600640 1 --2- 172.21.15.90:0/1608140243 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f000c06bbd0 0x7f000c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:39.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.077+0000 7f002d600640 1 -- 172.21.15.90:0/1608140243 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f002808a140 con 0x7f00300b2bc0 2024-03-24T01:25:39.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.077+0000 7f0034e00640 1 --2- 172.21.15.90:0/1608140243 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f000c06bbd0 0x7f000c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:39.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.078+0000 7f0034e00640 1 --2- 172.21.15.90:0/1608140243 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f000c06bbd0 0x7f000c06e090 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0020001200 tx=0x7f0020009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:39.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.081+0000 7f002d600640 1 -- 172.21.15.90:0/1608140243 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f00300ccf40 con 0x7f00300b2bc0 2024-03-24T01:25:39.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.277+0000 7f0037800640 1 -- 172.21.15.90:0/1608140243 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f003006eb80 con 0x7f00300b2bc0 2024-03-24T01:25:39.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.278+0000 7f002d600640 1 -- 172.21.15.90:0/1608140243 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0028058810 con 0x7f00300b2bc0 2024-03-24T01:25:39.278 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:25:39.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.280+0000 7f0037800640 1 -- 172.21.15.90:0/1608140243 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f000c06bbd0 msgr2=0x7f000c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:39.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.280+0000 7f0037800640 1 --2- 172.21.15.90:0/1608140243 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f000c06bbd0 0x7f000c06e090 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0020001200 tx=0x7f0020009040 comp rx=0 tx=0).stop 2024-03-24T01:25:39.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.280+0000 7f0037800640 1 -- 172.21.15.90:0/1608140243 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00300b2bc0 msgr2=0x7f00300ace20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:39.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.280+0000 7f0037800640 1 --2- 172.21.15.90:0/1608140243 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00300b2bc0 0x7f00300ace20 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f00300cc5d0 tx=0x7f002800cb90 comp rx=0 tx=0).stop 2024-03-24T01:25:39.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.281+0000 7f0037800640 1 -- 172.21.15.90:0/1608140243 shutdown_connections 2024-03-24T01:25:39.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.281+0000 7f0037800640 1 --2- 172.21.15.90:0/1608140243 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f000c06bbd0 0x7f000c06e090 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:39.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.281+0000 7f0037800640 1 --2- 172.21.15.90:0/1608140243 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00300b2bc0 0x7f00300ace20 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:39.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.281+0000 7f0037800640 1 --2- 172.21.15.90:0/1608140243 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00300ad9f0 0x7f00300ac8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:39.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.281+0000 7f0037800640 1 --2- 172.21.15.90:0/1608140243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0014006a10 0x7f00300b2680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:39.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.281+0000 7f0037800640 1 -- 172.21.15.90:0/1608140243 >> 172.21.15.90:0/1608140243 conn(0x7f00300b1770 msgr2=0x7f0030156d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:39.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.281+0000 7f0037800640 1 -- 172.21.15.90:0/1608140243 shutdown_connections 2024-03-24T01:25:39.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.281+0000 7f0037800640 1 -- 172.21.15.90:0/1608140243 wait complete. 2024-03-24T01:25:39.301 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T01:25:39.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.515+0000 7f6a40e00640 1 Processor -- start 2024-03-24T01:25:39.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.515+0000 7f6a40e00640 1 -- start start 2024-03-24T01:25:39.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.516+0000 7f6a40e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a3c15a1d0 0x7f6a3c072220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:39.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.516+0000 7f6a40e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6a3c072760 0x7f6a3c06edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:39.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.516+0000 7f6a40e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6a3c06f310 0x7f6a3c06d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:39.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.516+0000 7f6a40e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a3c0b39d0 con 0x7f6a3c06f310 2024-03-24T01:25:39.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.516+0000 7f6a40e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a3c072bc0 con 0x7f6a3c072760 2024-03-24T01:25:39.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.516+0000 7f6a40e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3c0b2930 con 0x7f6a3c15a1d0 2024-03-24T01:25:39.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.517+0000 7f6a3a800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6a3c06f310 0x7f6a3c06d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:39.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.517+0000 7f6a3a800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6a3c06f310 0x7f6a3c06d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55274/0 (socket says 172.21.15.90:55274) 2024-03-24T01:25:39.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.517+0000 7f6a3a800640 1 -- 172.21.15.90:0/869377993 learned_addr learned my addr 172.21.15.90:0/869377993 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:39.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.517+0000 7f6a39e00640 1 --2- 172.21.15.90:0/869377993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a3c15a1d0 0x7f6a3c072220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:39.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.517+0000 7f6a31400640 1 --2- 172.21.15.90:0/869377993 >> v2:172.21.15.120:3300/0 conn(0x7f6a3c072760 0x7f6a3c06edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:39.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.517+0000 7f6a3a800640 1 -- 172.21.15.90:0/869377993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a3c15a1d0 msgr2=0x7f6a3c072220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:39.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.517+0000 7f6a3a800640 1 --2- 172.21.15.90:0/869377993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a3c15a1d0 0x7f6a3c072220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:39.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.517+0000 7f6a3a800640 1 -- 172.21.15.90:0/869377993 >> v2:172.21.15.120:3300/0 conn(0x7f6a3c072760 msgr2=0x7f6a3c06edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:39.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.517+0000 7f6a3a800640 1 --2- 172.21.15.90:0/869377993 >> v2:172.21.15.120:3300/0 conn(0x7f6a3c072760 0x7f6a3c06edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:39.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.517+0000 7f6a3a800640 1 -- 172.21.15.90:0/869377993 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a3c0b2c10 con 0x7f6a3c06f310 2024-03-24T01:25:39.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.517+0000 7f6a39e00640 1 --2- 172.21.15.90:0/869377993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a3c15a1d0 0x7f6a3c072220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:39.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.518+0000 7f6a3a800640 1 --2- 172.21.15.90:0/869377993 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a3c06f310 0x7f6a3c06d9f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c072d80 tx=0x7f6a3402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e6d13847c85c8d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:39.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.519+0000 7f6a39400640 1 -- 172.21.15.90:0/869377993 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a3402ca70 con 0x7f6a3c06f310 2024-03-24T01:25:39.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.519+0000 7f6a39400640 1 -- 172.21.15.90:0/869377993 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a3402cc10 con 0x7f6a3c06f310 2024-03-24T01:25:39.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.519+0000 7f6a39400640 1 -- 172.21.15.90:0/869377993 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a340343f0 con 0x7f6a3c06f310 2024-03-24T01:25:39.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.519+0000 7f6a40e00640 1 -- 172.21.15.90:0/869377993 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a3c06f310 msgr2=0x7f6a3c06d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:39.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.519+0000 7f6a40e00640 1 --2- 172.21.15.90:0/869377993 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a3c06f310 0x7f6a3c06d9f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c072d80 tx=0x7f6a3402bed0 comp rx=0 tx=0).stop 2024-03-24T01:25:39.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.519+0000 7f6a40e00640 1 -- 172.21.15.90:0/869377993 shutdown_connections 2024-03-24T01:25:39.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.519+0000 7f6a40e00640 1 --2- 172.21.15.90:0/869377993 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a3c06f310 0x7f6a3c06d9f0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:39.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.519+0000 7f6a40e00640 1 --2- 172.21.15.90:0/869377993 >> v2:172.21.15.120:3300/0 conn(0x7f6a3c072760 0x7f6a3c06edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:39.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.519+0000 7f6a40e00640 1 --2- 172.21.15.90:0/869377993 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a3c15a1d0 0x7f6a3c072220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:39.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.519+0000 7f6a40e00640 1 -- 172.21.15.90:0/869377993 >> 172.21.15.90:0/869377993 conn(0x7f6a3c0cf7e0 msgr2=0x7f6a3c0cfbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:39.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.520+0000 7f6a40e00640 1 -- 172.21.15.90:0/869377993 shutdown_connections 2024-03-24T01:25:39.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.520+0000 7f6a40e00640 1 -- 172.21.15.90:0/869377993 wait complete. 2024-03-24T01:25:39.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.521+0000 7f6a40e00640 1 Processor -- start 2024-03-24T01:25:39.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.521+0000 7f6a40e00640 1 -- start start 2024-03-24T01:25:39.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.522+0000 7f6a40e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a3c06f310 0x7f6a3c145080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:39.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.522+0000 7f6a40e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a3c072760 0x7f6a3c15ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:39.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.522+0000 7f6a40e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a3c15a1d0 0x7f6a3c15c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:39.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.522+0000 7f6a40e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3c0b34b0 con 0x7f6a3c15a1d0 2024-03-24T01:25:39.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.522+0000 7f6a40e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3c0b31d0 con 0x7f6a3c072760 2024-03-24T01:25:39.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.522+0000 7f6a40e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3c15a5b0 con 0x7f6a3c06f310 2024-03-24T01:25:39.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.523+0000 7f6a3a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a3c15a1d0 0x7f6a3c15c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:39.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.523+0000 7f6a3a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a3c15a1d0 0x7f6a3c15c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55288/0 (socket says 172.21.15.90:55288) 2024-03-24T01:25:39.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.523+0000 7f6a3a800640 1 -- 172.21.15.90:0/4292445501 learned_addr learned my addr 172.21.15.90:0/4292445501 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:39.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.523+0000 7f6a39e00640 1 --2- 172.21.15.90:0/4292445501 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a3c06f310 0x7f6a3c145080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:39.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.523+0000 7f6a3a800640 1 -- 172.21.15.90:0/4292445501 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a3c06f310 msgr2=0x7f6a3c145080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:39.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.523+0000 7f6a31400640 1 --2- 172.21.15.90:0/4292445501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a3c072760 0x7f6a3c15ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:39.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.523+0000 7f6a3a800640 1 --2- 172.21.15.90:0/4292445501 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a3c06f310 0x7f6a3c145080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:39.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.523+0000 7f6a3a800640 1 -- 172.21.15.90:0/4292445501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a3c072760 msgr2=0x7f6a3c15ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:39.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.523+0000 7f6a3a800640 1 --2- 172.21.15.90:0/4292445501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a3c072760 0x7f6a3c15ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:39.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.523+0000 7f6a3a800640 1 -- 172.21.15.90:0/4292445501 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a3c152170 con 0x7f6a3c15a1d0 2024-03-24T01:25:39.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.523+0000 7f6a3a800640 1 --2- 172.21.15.90:0/4292445501 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a3c15a1d0 0x7f6a3c15c670 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f6a34009e50 tx=0x7f6a34004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:39.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.524+0000 7f6a32a00640 1 -- 172.21.15.90:0/4292445501 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a34043020 con 0x7f6a3c15a1d0 2024-03-24T01:25:39.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.524+0000 7f6a32a00640 1 -- 172.21.15.90:0/4292445501 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a3403e450 con 0x7f6a3c15a1d0 2024-03-24T01:25:39.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.524+0000 7f6a32a00640 1 -- 172.21.15.90:0/4292445501 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a34009510 con 0x7f6a3c15a1d0 2024-03-24T01:25:39.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.524+0000 7f6a40e00640 1 -- 172.21.15.90:0/4292445501 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a3c154bc0 con 0x7f6a3c15a1d0 2024-03-24T01:25:39.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.524+0000 7f6a40e00640 1 -- 172.21.15.90:0/4292445501 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a3c150390 con 0x7f6a3c15a1d0 2024-03-24T01:25:39.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.525+0000 7f6a40e00640 1 -- 172.21.15.90:0/4292445501 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a3c04f400 con 0x7f6a3c15a1d0 2024-03-24T01:25:39.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.526+0000 7f6a32a00640 1 -- 172.21.15.90:0/4292445501 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6a34046020 con 0x7f6a3c15a1d0 2024-03-24T01:25:39.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.527+0000 7f6a32a00640 1 --2- 172.21.15.90:0/4292445501 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6a1806bc00 0x7f6a1806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:39.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.527+0000 7f6a32a00640 1 -- 172.21.15.90:0/4292445501 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f6a340ad6a0 con 0x7f6a3c15a1d0 2024-03-24T01:25:39.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.527+0000 7f6a39e00640 1 --2- 172.21.15.90:0/4292445501 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6a1806bc00 0x7f6a1806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:39.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.529+0000 7f6a39e00640 1 --2- 172.21.15.90:0/4292445501 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6a1806bc00 0x7f6a1806e0c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c00b840 tx=0x7f6a2c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:39.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.531+0000 7f6a32a00640 1 -- 172.21.15.90:0/4292445501 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6a3404d030 con 0x7f6a3c15a1d0 2024-03-24T01:25:39.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.728+0000 7f6a40e00640 1 -- 172.21.15.90:0/4292445501 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6a3c151a90 con 0x7f6a3c15a1d0 2024-03-24T01:25:39.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.729+0000 7f6a32a00640 1 -- 172.21.15.90:0/4292445501 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6a3407bd60 con 0x7f6a3c15a1d0 2024-03-24T01:25:39.729 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:25:39.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.731+0000 7f6a40e00640 1 -- 172.21.15.90:0/4292445501 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6a1806bc00 msgr2=0x7f6a1806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:39.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.731+0000 7f6a40e00640 1 --2- 172.21.15.90:0/4292445501 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6a1806bc00 0x7f6a1806e0c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c00b840 tx=0x7f6a2c008040 comp rx=0 tx=0).stop 2024-03-24T01:25:39.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.731+0000 7f6a40e00640 1 -- 172.21.15.90:0/4292445501 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a3c15a1d0 msgr2=0x7f6a3c15c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:39.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.731+0000 7f6a40e00640 1 --2- 172.21.15.90:0/4292445501 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a3c15a1d0 0x7f6a3c15c670 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f6a34009e50 tx=0x7f6a34004a40 comp rx=0 tx=0).stop 2024-03-24T01:25:39.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.731+0000 7f6a40e00640 1 -- 172.21.15.90:0/4292445501 shutdown_connections 2024-03-24T01:25:39.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.731+0000 7f6a40e00640 1 --2- 172.21.15.90:0/4292445501 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6a1806bc00 0x7f6a1806e0c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:39.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.731+0000 7f6a40e00640 1 --2- 172.21.15.90:0/4292445501 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a3c15a1d0 0x7f6a3c15c670 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:39.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.731+0000 7f6a40e00640 1 --2- 172.21.15.90:0/4292445501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a3c072760 0x7f6a3c15ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:39.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.731+0000 7f6a40e00640 1 --2- 172.21.15.90:0/4292445501 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6a3c06f310 0x7f6a3c145080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:39.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.731+0000 7f6a40e00640 1 -- 172.21.15.90:0/4292445501 >> 172.21.15.90:0/4292445501 conn(0x7f6a3c0cf7e0 msgr2=0x7f6a3c0af9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:39.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.732+0000 7f6a40e00640 1 -- 172.21.15.90:0/4292445501 shutdown_connections 2024-03-24T01:25:39.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.732+0000 7f6a40e00640 1 -- 172.21.15.90:0/4292445501 wait complete. 2024-03-24T01:25:39.749 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T01:25:39.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.952+0000 7fee32800640 1 Processor -- start 2024-03-24T01:25:39.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.952+0000 7fee32800640 1 -- start start 2024-03-24T01:25:39.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.953+0000 7fee32800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee2c0720a0 0x7fee2c06e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:39.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.953+0000 7fee32800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fee2c06e9d0 0x7fee2c0acea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:39.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.953+0000 7fee32800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee2c15f2e0 con 0x7fee2c15b9f0 2024-03-24T01:25:39.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.953+0000 7fee32800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee2c072b00 con 0x7fee2c06e9d0 2024-03-24T01:25:39.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.953+0000 7fee32800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c15ee50 con 0x7fee2c0720a0 2024-03-24T01:25:39.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.954+0000 7fee2be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fee2c06e9d0 0x7fee2c0acea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:39.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.954+0000 7fee2be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fee2c06e9d0 0x7fee2c0acea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38694/0 (socket says 172.21.15.90:38694) 2024-03-24T01:25:39.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.954+0000 7fee2be00640 1 -- 172.21.15.90:0/2745799230 learned_addr learned my addr 172.21.15.90:0/2745799230 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:39.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.954+0000 7fee23e00640 1 --2- 172.21.15.90:0/2745799230 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee2c0720a0 0x7fee2c06e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:39.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.954+0000 7fee2be00640 1 -- 172.21.15.90:0/2745799230 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee2c0720a0 msgr2=0x7fee2c06e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:39.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.954+0000 7fee2be00640 1 --2- 172.21.15.90:0/2745799230 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee2c0720a0 0x7fee2c06e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:39.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.954+0000 7fee2be00640 1 -- 172.21.15.90:0/2745799230 >> v1:172.21.15.90:6789/0 conn(0x7fee2c15b9f0 legacy=0x7fee2c071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:39.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.954+0000 7fee2be00640 1 -- 172.21.15.90:0/2745799230 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee2c06f070 con 0x7fee2c06e9d0 2024-03-24T01:25:39.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.956+0000 7fee2be00640 1 --2- 172.21.15.90:0/2745799230 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee2c06e9d0 0x7fee2c0acea0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fee24009e50 tx=0x7fee2402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dd20f80bca7eef1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:39.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.956+0000 7fee2aa00640 1 -- 172.21.15.90:0/2745799230 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee2402cb00 con 0x7fee2c06e9d0 2024-03-24T01:25:39.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.956+0000 7fee2aa00640 1 -- 172.21.15.90:0/2745799230 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee2402cca0 con 0x7fee2c06e9d0 2024-03-24T01:25:39.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.956+0000 7fee2aa00640 1 -- 172.21.15.90:0/2745799230 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee24034610 con 0x7fee2c06e9d0 2024-03-24T01:25:39.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.956+0000 7fee32800640 1 -- 172.21.15.90:0/2745799230 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee2c06e9d0 msgr2=0x7fee2c0acea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:39.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.956+0000 7fee32800640 1 --2- 172.21.15.90:0/2745799230 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee2c06e9d0 0x7fee2c0acea0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fee24009e50 tx=0x7fee2402bf60 comp rx=0 tx=0).stop 2024-03-24T01:25:39.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.957+0000 7fee32800640 1 -- 172.21.15.90:0/2745799230 shutdown_connections 2024-03-24T01:25:39.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.957+0000 7fee32800640 1 --2- 172.21.15.90:0/2745799230 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee2c06e9d0 0x7fee2c0acea0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:39.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.957+0000 7fee32800640 1 --2- 172.21.15.90:0/2745799230 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee2c0720a0 0x7fee2c06e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:39.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.957+0000 7fee32800640 1 -- 172.21.15.90:0/2745799230 >> 172.21.15.90:0/2745799230 conn(0x7fee2c0cf7e0 msgr2=0x7fee2c0cfbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:39.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.957+0000 7fee32800640 1 -- 172.21.15.90:0/2745799230 shutdown_connections 2024-03-24T01:25:39.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.957+0000 7fee32800640 1 -- 172.21.15.90:0/2745799230 wait complete. 2024-03-24T01:25:39.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.958+0000 7fee32800640 1 Processor -- start 2024-03-24T01:25:39.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.958+0000 7fee32800640 1 -- start start 2024-03-24T01:25:39.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.959+0000 7fee32800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee2c06e9d0 0x7fee2c150710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:39.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.959+0000 7fee32800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee2c0720a0 0x7fee2c159a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:39.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.959+0000 7fee32800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee2c15b9f0 0x7fee2c161e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:39.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.959+0000 7fee32800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c15eae0 con 0x7fee2c15b9f0 2024-03-24T01:25:39.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.959+0000 7fee32800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c15e800 con 0x7fee2c0720a0 2024-03-24T01:25:39.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.959+0000 7fee32800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c0ad3e0 con 0x7fee2c06e9d0 2024-03-24T01:25:39.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.960+0000 7fee2be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee2c15b9f0 0x7fee2c161e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:39.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.960+0000 7fee2be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee2c15b9f0 0x7fee2c161e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38704/0 (socket says 172.21.15.90:38704) 2024-03-24T01:25:39.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.960+0000 7fee2be00640 1 -- 172.21.15.90:0/3291972853 learned_addr learned my addr 172.21.15.90:0/3291972853 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:39.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.960+0000 7fee2be00640 1 -- 172.21.15.90:0/3291972853 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee2c06e9d0 msgr2=0x7fee2c150710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:39.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.960+0000 7fee2b400640 1 --2- 172.21.15.90:0/3291972853 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee2c06e9d0 0x7fee2c150710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:39.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.960+0000 7fee2be00640 1 --2- 172.21.15.90:0/3291972853 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee2c06e9d0 0x7fee2c150710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:39.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.960+0000 7fee23e00640 1 --2- 172.21.15.90:0/3291972853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee2c0720a0 0x7fee2c159a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:39.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.960+0000 7fee2be00640 1 -- 172.21.15.90:0/3291972853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee2c0720a0 msgr2=0x7fee2c159a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:39.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.960+0000 7fee2be00640 1 --2- 172.21.15.90:0/3291972853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee2c0720a0 0x7fee2c159a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:39.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.960+0000 7fee2be00640 1 -- 172.21.15.90:0/3291972853 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee2c15a3f0 con 0x7fee2c15b9f0 2024-03-24T01:25:39.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.960+0000 7fee2be00640 1 --2- 172.21.15.90:0/3291972853 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee2c15b9f0 0x7fee2c161e50 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fee24009e50 tx=0x7fee24004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:39.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.961+0000 7fee23400640 1 -- 172.21.15.90:0/3291972853 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee24043020 con 0x7fee2c15b9f0 2024-03-24T01:25:39.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.961+0000 7fee23400640 1 -- 172.21.15.90:0/3291972853 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee2402ce50 con 0x7fee2c15b9f0 2024-03-24T01:25:39.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.961+0000 7fee23400640 1 -- 172.21.15.90:0/3291972853 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee24009510 con 0x7fee2c15b9f0 2024-03-24T01:25:39.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.961+0000 7fee32800640 1 -- 172.21.15.90:0/3291972853 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee2c15a6d0 con 0x7fee2c15b9f0 2024-03-24T01:25:39.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.961+0000 7fee32800640 1 -- 172.21.15.90:0/3291972853 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee2c0cc550 con 0x7fee2c15b9f0 2024-03-24T01:25:39.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.962+0000 7fee32800640 1 -- 172.21.15.90:0/3291972853 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee2c04f400 con 0x7fee2c15b9f0 2024-03-24T01:25:39.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.963+0000 7fee23400640 1 -- 172.21.15.90:0/3291972853 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fee24046020 con 0x7fee2c15b9f0 2024-03-24T01:25:39.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.964+0000 7fee23400640 1 --2- 172.21.15.90:0/3291972853 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fee0806bbd0 0x7fee0806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:39.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.964+0000 7fee23400640 1 -- 172.21.15.90:0/3291972853 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7fee240ad4d0 con 0x7fee2c15b9f0 2024-03-24T01:25:39.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.964+0000 7fee2b400640 1 --2- 172.21.15.90:0/3291972853 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fee0806bbd0 0x7fee0806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:39.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.966+0000 7fee2b400640 1 --2- 172.21.15.90:0/3291972853 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fee0806bbd0 0x7fee0806e090 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fee1c001500 tx=0x7fee1c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:39.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:39.968+0000 7fee23400640 1 -- 172.21.15.90:0/3291972853 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fee2404d030 con 0x7fee2c15b9f0 2024-03-24T01:25:40.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.164+0000 7fee32800640 1 -- 172.21.15.90:0/3291972853 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fee2c072590 con 0x7fee2c15b9f0 2024-03-24T01:25:40.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.164+0000 7fee23400640 1 -- 172.21.15.90:0/3291972853 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fee2c06f070 con 0x7fee2c15b9f0 2024-03-24T01:25:40.165 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-24T01:25:40.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.167+0000 7fee32800640 1 -- 172.21.15.90:0/3291972853 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fee0806bbd0 msgr2=0x7fee0806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:40.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.167+0000 7fee32800640 1 --2- 172.21.15.90:0/3291972853 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fee0806bbd0 0x7fee0806e090 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fee1c001500 tx=0x7fee1c009040 comp rx=0 tx=0).stop 2024-03-24T01:25:40.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.167+0000 7fee32800640 1 -- 172.21.15.90:0/3291972853 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee2c15b9f0 msgr2=0x7fee2c161e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:40.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.167+0000 7fee32800640 1 --2- 172.21.15.90:0/3291972853 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee2c15b9f0 0x7fee2c161e50 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fee24009e50 tx=0x7fee24004a40 comp rx=0 tx=0).stop 2024-03-24T01:25:40.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.167+0000 7fee32800640 1 -- 172.21.15.90:0/3291972853 shutdown_connections 2024-03-24T01:25:40.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.167+0000 7fee32800640 1 --2- 172.21.15.90:0/3291972853 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fee0806bbd0 0x7fee0806e090 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:40.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.167+0000 7fee32800640 1 --2- 172.21.15.90:0/3291972853 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee2c15b9f0 0x7fee2c161e50 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:40.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.167+0000 7fee32800640 1 --2- 172.21.15.90:0/3291972853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee2c0720a0 0x7fee2c159a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:40.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.167+0000 7fee32800640 1 --2- 172.21.15.90:0/3291972853 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fee2c06e9d0 0x7fee2c150710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:40.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.167+0000 7fee32800640 1 -- 172.21.15.90:0/3291972853 >> 172.21.15.90:0/3291972853 conn(0x7fee2c0cf7e0 msgr2=0x7fee2c0ad790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:40.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.167+0000 7fee32800640 1 -- 172.21.15.90:0/3291972853 shutdown_connections 2024-03-24T01:25:40.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.167+0000 7fee32800640 1 -- 172.21.15.90:0/3291972853 wait complete. 2024-03-24T01:25:40.187 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T01:25:40.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.406+0000 7f0019000640 1 Processor -- start 2024-03-24T01:25:40.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.406+0000 7f0019000640 1 -- start start 2024-03-24T01:25:40.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.408+0000 7f0019000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0014159620 0x7f001406b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:40.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.408+0000 7f0019000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00140cf150 0x7f00140ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:40.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.408+0000 7f0019000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f001406bca0 con 0x7f001406b600 2024-03-24T01:25:40.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.408+0000 7f0019000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00140b4470 con 0x7f0014159620 2024-03-24T01:25:40.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.408+0000 7f0019000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00140b36f0 con 0x7f00140cf150 2024-03-24T01:25:40.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.408+0000 7f0011e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0014159620 0x7f001406b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:40.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.408+0000 7f0011e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0014159620 0x7f001406b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38710/0 (socket says 172.21.15.90:38710) 2024-03-24T01:25:40.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.408+0000 7f0011e00640 1 -- 172.21.15.90:0/383869265 learned_addr learned my addr 172.21.15.90:0/383869265 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:40.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.409+0000 7f0011e00640 1 -- 172.21.15.90:0/383869265 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00140cf150 msgr2=0x7f00140ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:40.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.409+0000 7f0011e00640 1 --2- 172.21.15.90:0/383869265 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00140cf150 0x7f00140ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:40.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.409+0000 7f0011e00640 1 -- 172.21.15.90:0/383869265 >> v1:172.21.15.120:6789/0 conn(0x7f001406b600 legacy=0x7f00140cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:40.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.409+0000 7f0011e00640 1 -- 172.21.15.90:0/383869265 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00140b3990 con 0x7f0014159620 2024-03-24T01:25:40.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.410+0000 7f0011e00640 1 --2- 172.21.15.90:0/383869265 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0014159620 0x7f001406b0c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f0004009870 tx=0x7f000402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f575f419d005ab6f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:40.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.411+0000 7f0010a00640 1 -- 172.21.15.90:0/383869265 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00040317c0 con 0x7f0014159620 2024-03-24T01:25:40.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.411+0000 7f0010a00640 1 -- 172.21.15.90:0/383869265 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0004037070 con 0x7f0014159620 2024-03-24T01:25:40.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.411+0000 7f0010a00640 1 -- 172.21.15.90:0/383869265 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0004031b80 con 0x7f0014159620 2024-03-24T01:25:40.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.411+0000 7f0019000640 1 -- 172.21.15.90:0/383869265 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0014159620 msgr2=0x7f001406b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:40.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.411+0000 7f0019000640 1 --2- 172.21.15.90:0/383869265 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0014159620 0x7f001406b0c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f0004009870 tx=0x7f000402b860 comp rx=0 tx=0).stop 2024-03-24T01:25:40.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.411+0000 7f0019000640 1 -- 172.21.15.90:0/383869265 shutdown_connections 2024-03-24T01:25:40.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.412+0000 7f0019000640 1 --2- 172.21.15.90:0/383869265 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00140cf150 0x7f00140ae530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:40.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.412+0000 7f0019000640 1 --2- 172.21.15.90:0/383869265 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0014159620 0x7f001406b0c0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:40.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.412+0000 7f0019000640 1 -- 172.21.15.90:0/383869265 >> 172.21.15.90:0/383869265 conn(0x7f00140b1770 msgr2=0x7f00140b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:40.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.412+0000 7f0019000640 1 -- 172.21.15.90:0/383869265 shutdown_connections 2024-03-24T01:25:40.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.412+0000 7f0019000640 1 -- 172.21.15.90:0/383869265 wait complete. 2024-03-24T01:25:40.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.413+0000 7f0019000640 1 Processor -- start 2024-03-24T01:25:40.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.413+0000 7f0019000640 1 -- start start 2024-03-24T01:25:40.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.414+0000 7f0019000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f001406b600 0x7f001406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:40.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.414+0000 7f0019000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00140cf150 0x7f001414a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:40.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.414+0000 7f0019000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0014159620 0x7f001414d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:40.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.414+0000 7f0019000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0014159a00 con 0x7f001406b600 2024-03-24T01:25:40.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.414+0000 7f0019000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00140b3f50 con 0x7f0014159620 2024-03-24T01:25:40.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.414+0000 7f0019000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f001415bfe0 con 0x7f00140cf150 2024-03-24T01:25:40.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.415+0000 7f0011e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f001406b600 0x7f001406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:40.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.415+0000 7f0011e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f001406b600 0x7f001406d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38726/0 (socket says 172.21.15.90:38726) 2024-03-24T01:25:40.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.415+0000 7f0011e00640 1 -- 172.21.15.90:0/1896376198 learned_addr learned my addr 172.21.15.90:0/1896376198 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:40.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.415+0000 7f0011400640 1 --2- 172.21.15.90:0/1896376198 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00140cf150 0x7f001414a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:40.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.415+0000 7f0011e00640 1 -- 172.21.15.90:0/1896376198 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00140cf150 msgr2=0x7f001414a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:40.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.415+0000 7f0011e00640 1 --2- 172.21.15.90:0/1896376198 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00140cf150 0x7f001414a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:40.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.415+0000 7f0011e00640 1 -- 172.21.15.90:0/1896376198 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0014159620 msgr2=0x7f001414d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:40.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.415+0000 7f0011e00640 1 --2- 172.21.15.90:0/1896376198 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0014159620 0x7f001414d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:40.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.415+0000 7f0011e00640 1 -- 172.21.15.90:0/1896376198 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0014152be0 con 0x7f001406b600 2024-03-24T01:25:40.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.416+0000 7f0011e00640 1 --2- 172.21.15.90:0/1896376198 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f001406b600 0x7f001406d990 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f0004009870 tx=0x7f00040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:40.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.416+0000 7f000a000640 1 -- 172.21.15.90:0/1896376198 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0004032860 con 0x7f001406b600 2024-03-24T01:25:40.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.416+0000 7f000a000640 1 -- 172.21.15.90:0/1896376198 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0004032a00 con 0x7f001406b600 2024-03-24T01:25:40.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.416+0000 7f000a000640 1 -- 172.21.15.90:0/1896376198 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0004032df0 con 0x7f001406b600 2024-03-24T01:25:40.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.416+0000 7f0019000640 1 -- 172.21.15.90:0/1896376198 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f001406e070 con 0x7f001406b600 2024-03-24T01:25:40.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.416+0000 7f0019000640 1 -- 172.21.15.90:0/1896376198 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f001406e630 con 0x7f001406b600 2024-03-24T01:25:40.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.417+0000 7f0019000640 1 -- 172.21.15.90:0/1896376198 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f001404f400 con 0x7f001406b600 2024-03-24T01:25:40.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.418+0000 7f000a000640 1 -- 172.21.15.90:0/1896376198 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f000404c060 con 0x7f001406b600 2024-03-24T01:25:40.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.419+0000 7f000a000640 1 --2- 172.21.15.90:0/1896376198 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7efff406bbd0 0x7efff406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:40.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.419+0000 7f000a000640 1 -- 172.21.15.90:0/1896376198 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f00040ac820 con 0x7f001406b600 2024-03-24T01:25:40.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.419+0000 7f0011400640 1 --2- 172.21.15.90:0/1896376198 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7efff406bbd0 0x7efff406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:40.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.420+0000 7f0011400640 1 --2- 172.21.15.90:0/1896376198 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7efff406bbd0 0x7efff406e090 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f001414b5d0 tx=0x7f0000009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:40.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.423+0000 7f000a000640 1 -- 172.21.15.90:0/1896376198 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0014152be0 con 0x7f001406b600 2024-03-24T01:25:40.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.620+0000 7f0019000640 1 -- 172.21.15.90:0/1896376198 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f00140cbf30 con 0x7f001406b600 2024-03-24T01:25:40.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.621+0000 7f000a000640 1 -- 172.21.15.90:0/1896376198 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f0004040070 con 0x7f001406b600 2024-03-24T01:25:40.622 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:25:40.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.624+0000 7f0019000640 1 -- 172.21.15.90:0/1896376198 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7efff406bbd0 msgr2=0x7efff406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:40.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.624+0000 7f0019000640 1 --2- 172.21.15.90:0/1896376198 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7efff406bbd0 0x7efff406e090 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f001414b5d0 tx=0x7f0000009040 comp rx=0 tx=0).stop 2024-03-24T01:25:40.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.624+0000 7f0019000640 1 -- 172.21.15.90:0/1896376198 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f001406b600 msgr2=0x7f001406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:40.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.624+0000 7f0019000640 1 --2- 172.21.15.90:0/1896376198 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f001406b600 0x7f001406d990 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f0004009870 tx=0x7f00040046c0 comp rx=0 tx=0).stop 2024-03-24T01:25:40.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.624+0000 7f0019000640 1 -- 172.21.15.90:0/1896376198 shutdown_connections 2024-03-24T01:25:40.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.624+0000 7f0019000640 1 --2- 172.21.15.90:0/1896376198 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7efff406bbd0 0x7efff406e090 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:40.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.624+0000 7f0019000640 1 --2- 172.21.15.90:0/1896376198 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0014159620 0x7f001414d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:40.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.624+0000 7f0019000640 1 --2- 172.21.15.90:0/1896376198 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f00140cf150 0x7f001414a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:40.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.624+0000 7f0019000640 1 --2- 172.21.15.90:0/1896376198 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f001406b600 0x7f001406d990 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:40.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.624+0000 7f0019000640 1 -- 172.21.15.90:0/1896376198 >> 172.21.15.90:0/1896376198 conn(0x7f00140b1770 msgr2=0x7f0014159c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:40.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.624+0000 7f0019000640 1 -- 172.21.15.90:0/1896376198 shutdown_connections 2024-03-24T01:25:40.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.624+0000 7f0019000640 1 -- 172.21.15.90:0/1896376198 wait complete. 2024-03-24T01:25:40.642 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T01:25:40.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.850+0000 7fd22a600640 1 Processor -- start 2024-03-24T01:25:40.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.850+0000 7fd22a600640 1 -- start start 2024-03-24T01:25:40.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.852+0000 7fd22a600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd224071790 0x7fd224071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:40.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.852+0000 7fd22a600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2240b4410 0x7fd2240b4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:40.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.852+0000 7fd22a600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2240720b0 con 0x7fd224161800 2024-03-24T01:25:40.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.852+0000 7fd22a600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd22406d7f0 con 0x7fd224071790 2024-03-24T01:25:40.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.852+0000 7fd22a600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd22406dab0 con 0x7fd2240b4410 2024-03-24T01:25:40.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.853+0000 7fd229c00640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fd224161800 0x7fd224161c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.90:36092/0 (socket says 172.21.15.90:36092) 2024-03-24T01:25:40.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.853+0000 7fd229c00640 1 -- 172.21.15.90:0/1768527396 learned_addr learned my addr 172.21.15.90:0/1768527396 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:40.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.853+0000 7fd21be00640 1 --2- 172.21.15.90:0/1768527396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2240b4410 0x7fd2240b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:40.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.853+0000 7fd229200640 1 --2- 172.21.15.90:0/1768527396 >> v2:172.21.15.120:3300/0 conn(0x7fd224071790 0x7fd224071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:40.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.854+0000 7fd223e00640 1 -- 172.21.15.90:0/1768527396 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1622900081 0 0) 0x7fd2240720b0 con 0x7fd224161800 2024-03-24T01:25:40.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.854+0000 7fd223e00640 1 -- 172.21.15.90:0/1768527396 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd204003520 con 0x7fd224161800 2024-03-24T01:25:40.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.854+0000 7fd21be00640 1 -- 172.21.15.90:0/1768527396 >> v2:172.21.15.120:3300/0 conn(0x7fd224071790 msgr2=0x7fd224071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:40.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.854+0000 7fd21be00640 1 --2- 172.21.15.90:0/1768527396 >> v2:172.21.15.120:3300/0 conn(0x7fd224071790 0x7fd224071b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:40.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.854+0000 7fd21be00640 1 -- 172.21.15.90:0/1768527396 >> v1:172.21.15.120:6789/0 conn(0x7fd224161800 legacy=0x7fd224161c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:40.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.854+0000 7fd21be00640 1 -- 172.21.15.90:0/1768527396 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd22406dd90 con 0x7fd2240b4410 2024-03-24T01:25:40.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.854+0000 7fd229200640 1 --2- 172.21.15.90:0/1768527396 >> v2:172.21.15.120:3300/0 conn(0x7fd224071790 0x7fd224071b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T01:25:40.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.855+0000 7fd21be00640 1 --2- 172.21.15.90:0/1768527396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2240b4410 0x7fd2240b4870 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd2140097a0 tx=0x7fd21402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e33e7072a39ad3d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:40.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.856+0000 7fd223e00640 1 -- 172.21.15.90:0/1768527396 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd214035020 con 0x7fd2240b4410 2024-03-24T01:25:40.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.856+0000 7fd223e00640 1 -- 172.21.15.90:0/1768527396 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd214031810 con 0x7fd2240b4410 2024-03-24T01:25:40.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.856+0000 7fd223e00640 1 -- 172.21.15.90:0/1768527396 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd214031be0 con 0x7fd2240b4410 2024-03-24T01:25:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.856+0000 7fd22a600640 1 -- 172.21.15.90:0/1768527396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2240b4410 msgr2=0x7fd2240b4870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.856+0000 7fd22a600640 1 --2- 172.21.15.90:0/1768527396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2240b4410 0x7fd2240b4870 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd2140097a0 tx=0x7fd21402b8b0 comp rx=0 tx=0).stop 2024-03-24T01:25:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.857+0000 7fd22a600640 1 -- 172.21.15.90:0/1768527396 shutdown_connections 2024-03-24T01:25:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.857+0000 7fd22a600640 1 --2- 172.21.15.90:0/1768527396 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2240b4410 0x7fd2240b4870 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.857+0000 7fd22a600640 1 --2- 172.21.15.90:0/1768527396 >> v2:172.21.15.120:3300/0 conn(0x7fd224071790 0x7fd224071b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.857+0000 7fd22a600640 1 -- 172.21.15.90:0/1768527396 >> 172.21.15.90:0/1768527396 conn(0x7fd22406b0c0 msgr2=0x7fd22406b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.857+0000 7fd22a600640 1 -- 172.21.15.90:0/1768527396 shutdown_connections 2024-03-24T01:25:40.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.857+0000 7fd22a600640 1 -- 172.21.15.90:0/1768527396 wait complete. 2024-03-24T01:25:40.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.859+0000 7fd22a600640 1 Processor -- start 2024-03-24T01:25:40.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.859+0000 7fd22a600640 1 -- start start 2024-03-24T01:25:40.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.860+0000 7fd22a600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd224071790 0x7fd2240b2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:40.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.860+0000 7fd22a600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2240b4410 0x7fd2241573d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:40.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.860+0000 7fd22a600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd224161800 0x7fd224157910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:40.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.860+0000 7fd22a600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2240cf460 con 0x7fd224071790 2024-03-24T01:25:40.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.860+0000 7fd22a600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22406e350 con 0x7fd224161800 2024-03-24T01:25:40.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.860+0000 7fd22a600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd22415f050 con 0x7fd2240b4410 2024-03-24T01:25:40.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.860+0000 7fd229200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd224071790 0x7fd2240b2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:40.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.861+0000 7fd229200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd224071790 0x7fd2240b2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38740/0 (socket says 172.21.15.90:38740) 2024-03-24T01:25:40.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.861+0000 7fd229200640 1 -- 172.21.15.90:0/664152058 learned_addr learned my addr 172.21.15.90:0/664152058 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:40.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.861+0000 7fd21be00640 1 --2- 172.21.15.90:0/664152058 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2240b4410 0x7fd2241573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:40.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.861+0000 7fd229200640 1 -- 172.21.15.90:0/664152058 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2240b4410 msgr2=0x7fd2241573d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:40.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.861+0000 7fd229200640 1 --2- 172.21.15.90:0/664152058 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2240b4410 0x7fd2241573d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:40.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.861+0000 7fd229200640 1 -- 172.21.15.90:0/664152058 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd224161800 msgr2=0x7fd224157910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:40.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.861+0000 7fd229200640 1 --2- 172.21.15.90:0/664152058 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd224161800 0x7fd224157910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:40.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.861+0000 7fd229200640 1 -- 172.21.15.90:0/664152058 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2240ae140 con 0x7fd224071790 2024-03-24T01:25:40.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.861+0000 7fd229200640 1 --2- 172.21.15.90:0/664152058 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd224071790 0x7fd2240b2f50 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fd20c00e960 tx=0x7fd20c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:40.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.861+0000 7fd221600640 1 -- 172.21.15.90:0/664152058 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20c017070 con 0x7fd224071790 2024-03-24T01:25:40.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.862+0000 7fd221600640 1 -- 172.21.15.90:0/664152058 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd20c00ca00 con 0x7fd224071790 2024-03-24T01:25:40.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.862+0000 7fd221600640 1 -- 172.21.15.90:0/664152058 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20c00cdd0 con 0x7fd224071790 2024-03-24T01:25:40.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.862+0000 7fd22a600640 1 -- 172.21.15.90:0/664152058 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2241e82e0 con 0x7fd224071790 2024-03-24T01:25:40.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.862+0000 7fd22a600640 1 -- 172.21.15.90:0/664152058 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd224158a90 con 0x7fd224071790 2024-03-24T01:25:40.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.863+0000 7fd22a600640 1 -- 172.21.15.90:0/664152058 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd22415bb60 con 0x7fd224071790 2024-03-24T01:25:40.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.864+0000 7fd221600640 1 -- 172.21.15.90:0/664152058 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd20c01a070 con 0x7fd224071790 2024-03-24T01:25:40.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.864+0000 7fd221600640 1 --2- 172.21.15.90:0/664152058 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd20006bbd0 0x7fd20006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:40.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.865+0000 7fd221600640 1 -- 172.21.15.90:0/664152058 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7fd20c08ae10 con 0x7fd224071790 2024-03-24T01:25:40.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.865+0000 7fd21be00640 1 --2- 172.21.15.90:0/664152058 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd20006bbd0 0x7fd20006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:40.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.866+0000 7fd21be00640 1 --2- 172.21.15.90:0/664152058 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd20006bbd0 0x7fd20006e090 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd214036000 tx=0x7fd214008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:40.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:40.869+0000 7fd221600640 1 -- 172.21.15.90:0/664152058 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd22415bb60 con 0x7fd224071790 2024-03-24T01:25:41.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.079+0000 7fd22a600640 1 -- 172.21.15.90:0/664152058 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd22404f430 con 0x7fd224071790 2024-03-24T01:25:41.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.080+0000 7fd221600640 1 -- 172.21.15.90:0/664152058 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd22404f430 con 0x7fd224071790 2024-03-24T01:25:41.080 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-24T01:25:41.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.082+0000 7fd22a600640 1 -- 172.21.15.90:0/664152058 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd20006bbd0 msgr2=0x7fd20006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:41.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.082+0000 7fd22a600640 1 --2- 172.21.15.90:0/664152058 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd20006bbd0 0x7fd20006e090 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd214036000 tx=0x7fd214008030 comp rx=0 tx=0).stop 2024-03-24T01:25:41.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.082+0000 7fd22a600640 1 -- 172.21.15.90:0/664152058 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd224071790 msgr2=0x7fd2240b2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:41.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.082+0000 7fd22a600640 1 --2- 172.21.15.90:0/664152058 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd224071790 0x7fd2240b2f50 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fd20c00e960 tx=0x7fd20c00ee20 comp rx=0 tx=0).stop 2024-03-24T01:25:41.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.082+0000 7fd22a600640 1 -- 172.21.15.90:0/664152058 shutdown_connections 2024-03-24T01:25:41.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.082+0000 7fd22a600640 1 --2- 172.21.15.90:0/664152058 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd20006bbd0 0x7fd20006e090 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:41.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.082+0000 7fd22a600640 1 --2- 172.21.15.90:0/664152058 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd224161800 0x7fd224157910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:41.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.082+0000 7fd22a600640 1 --2- 172.21.15.90:0/664152058 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2240b4410 0x7fd2241573d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:41.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.082+0000 7fd22a600640 1 --2- 172.21.15.90:0/664152058 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd224071790 0x7fd2240b2f50 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:41.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.082+0000 7fd22a600640 1 -- 172.21.15.90:0/664152058 >> 172.21.15.90:0/664152058 conn(0x7fd22406b0c0 msgr2=0x7fd224071b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:41.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.083+0000 7fd22a600640 1 -- 172.21.15.90:0/664152058 shutdown_connections 2024-03-24T01:25:41.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.083+0000 7fd22a600640 1 -- 172.21.15.90:0/664152058 wait complete. 2024-03-24T01:25:41.103 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi120.front.sepia.ceph.com... 2024-03-24T01:25:41.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:41.103 DEBUG:teuthology.orchestra.run.smithi120:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-03-24T01:25:41.196 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:25:41.196 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2024-03-24T01:25:41.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:25:41.197 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-24T01:25:41.243 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-24T01:25:41.244 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:25:41.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.460+0000 7f2f55a00640 1 Processor -- start 2024-03-24T01:25:41.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.460+0000 7f2f55a00640 1 -- start start 2024-03-24T01:25:41.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.461+0000 7f2f55a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f500cdac0 0x7f2f5006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:41.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.461+0000 7f2f55a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2f5006b600 0x7f2f500ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:41.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.461+0000 7f2f55a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f5015bf70 con 0x7f2f50155030 2024-03-24T01:25:41.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.461+0000 7f2f55a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f5006ba70 con 0x7f2f5006b600 2024-03-24T01:25:41.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.461+0000 7f2f55a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5015b360 con 0x7f2f500cdac0 2024-03-24T01:25:41.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.462+0000 7f2f46000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f500cdac0 0x7f2f5006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:41.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.462+0000 7f2f4ea00640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f2f50155030 0x7f2f5015efc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.90:36098/0 (socket says 172.21.15.90:36098) 2024-03-24T01:25:41.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.462+0000 7f2f4ea00640 1 -- 172.21.15.90:0/4059845683 learned_addr learned my addr 172.21.15.90:0/4059845683 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:41.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.462+0000 7f2f4f400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2f5006b600 0x7f2f500ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:41.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.463+0000 7f2f4e000640 1 -- 172.21.15.90:0/4059845683 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2508248596 0 0) 0x7f2f5015bf70 con 0x7f2f50155030 2024-03-24T01:25:41.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.463+0000 7f2f46000640 1 -- 172.21.15.90:0/4059845683 >> v2:172.21.15.120:3300/0 conn(0x7f2f5006b600 msgr2=0x7f2f500ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:41.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.463+0000 7f2f46000640 1 --2- 172.21.15.90:0/4059845683 >> v2:172.21.15.120:3300/0 conn(0x7f2f5006b600 0x7f2f500ae530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:41.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.463+0000 7f2f46000640 1 -- 172.21.15.90:0/4059845683 >> v1:172.21.15.120:6789/0 conn(0x7f2f50155030 legacy=0x7f2f5015efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:41.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.463+0000 7f2f46000640 1 -- 172.21.15.90:0/4059845683 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f5015b640 con 0x7f2f500cdac0 2024-03-24T01:25:41.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.464+0000 7f2f46000640 1 --2- 172.21.15.90:0/4059845683 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f500cdac0 0x7f2f5006b0c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2f380097a0 tx=0x7f2f3802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=620476f41e2aeb51 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:41.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.465+0000 7f2f4e000640 1 -- 172.21.15.90:0/4059845683 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f38035020 con 0x7f2f500cdac0 2024-03-24T01:25:41.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.465+0000 7f2f4e000640 1 -- 172.21.15.90:0/4059845683 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f38031810 con 0x7f2f500cdac0 2024-03-24T01:25:41.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.465+0000 7f2f4e000640 1 -- 172.21.15.90:0/4059845683 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f38031be0 con 0x7f2f500cdac0 2024-03-24T01:25:41.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.465+0000 7f2f55a00640 1 -- 172.21.15.90:0/4059845683 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f500cdac0 msgr2=0x7f2f5006b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:41.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.465+0000 7f2f55a00640 1 --2- 172.21.15.90:0/4059845683 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f500cdac0 0x7f2f5006b0c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2f380097a0 tx=0x7f2f3802b8b0 comp rx=0 tx=0).stop 2024-03-24T01:25:41.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.465+0000 7f2f55a00640 1 -- 172.21.15.90:0/4059845683 shutdown_connections 2024-03-24T01:25:41.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.466+0000 7f2f55a00640 1 --2- 172.21.15.90:0/4059845683 >> v2:172.21.15.120:3300/0 conn(0x7f2f5006b600 0x7f2f500ae530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:41.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.466+0000 7f2f55a00640 1 --2- 172.21.15.90:0/4059845683 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f500cdac0 0x7f2f5006b0c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:41.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.466+0000 7f2f55a00640 1 -- 172.21.15.90:0/4059845683 >> 172.21.15.90:0/4059845683 conn(0x7f2f500b1770 msgr2=0x7f2f500b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:41.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.466+0000 7f2f55a00640 1 -- 172.21.15.90:0/4059845683 shutdown_connections 2024-03-24T01:25:41.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.466+0000 7f2f55a00640 1 -- 172.21.15.90:0/4059845683 wait complete. 2024-03-24T01:25:41.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.467+0000 7f2f55a00640 1 Processor -- start 2024-03-24T01:25:41.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.468+0000 7f2f55a00640 1 -- start start 2024-03-24T01:25:41.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.468+0000 7f2f55a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f5006b600 0x7f2f501e3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:41.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.468+0000 7f2f55a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f500cdac0 0x7f2f50159d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:41.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.468+0000 7f2f55a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f50155030 0x7f2f5013ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:41.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.469+0000 7f2f55a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f50155410 con 0x7f2f500cdac0 2024-03-24T01:25:41.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.469+0000 7f2f55a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5015bc00 con 0x7f2f5006b600 2024-03-24T01:25:41.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.469+0000 7f2f55a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5015af70 con 0x7f2f50155030 2024-03-24T01:25:41.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.469+0000 7f2f46000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f500cdac0 0x7f2f50159d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:41.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.469+0000 7f2f46000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f500cdac0 0x7f2f50159d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38748/0 (socket says 172.21.15.90:38748) 2024-03-24T01:25:41.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.469+0000 7f2f46000640 1 -- 172.21.15.90:0/2883873926 learned_addr learned my addr 172.21.15.90:0/2883873926 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:41.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.469+0000 7f2f4ea00640 1 --2- 172.21.15.90:0/2883873926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f5006b600 0x7f2f501e3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:41.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.469+0000 7f2f4f400640 1 --2- 172.21.15.90:0/2883873926 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f50155030 0x7f2f5013ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:41.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.469+0000 7f2f46000640 1 -- 172.21.15.90:0/2883873926 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f50155030 msgr2=0x7f2f5013ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:41.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.469+0000 7f2f46000640 1 --2- 172.21.15.90:0/2883873926 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f50155030 0x7f2f5013ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:41.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.469+0000 7f2f46000640 1 -- 172.21.15.90:0/2883873926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f5006b600 msgr2=0x7f2f501e3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:41.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.469+0000 7f2f46000640 1 --2- 172.21.15.90:0/2883873926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f5006b600 0x7f2f501e3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:41.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.469+0000 7f2f46000640 1 -- 172.21.15.90:0/2883873926 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f50158c80 con 0x7f2f500cdac0 2024-03-24T01:25:41.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.469+0000 7f2f4ea00640 1 --2- 172.21.15.90:0/2883873926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f5006b600 0x7f2f501e3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:41.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.470+0000 7f2f46000640 1 --2- 172.21.15.90:0/2883873926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f500cdac0 0x7f2f50159d20 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f2f38000c00 tx=0x7f2f380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:41.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.470+0000 7f2f47400640 1 -- 172.21.15.90:0/2883873926 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f38035050 con 0x7f2f500cdac0 2024-03-24T01:25:41.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.470+0000 7f2f47400640 1 -- 172.21.15.90:0/2883873926 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f3803bcf0 con 0x7f2f500cdac0 2024-03-24T01:25:41.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.470+0000 7f2f47400640 1 -- 172.21.15.90:0/2883873926 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f38006e10 con 0x7f2f500cdac0 2024-03-24T01:25:41.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.470+0000 7f2f55a00640 1 -- 172.21.15.90:0/2883873926 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f50158f60 con 0x7f2f500cdac0 2024-03-24T01:25:41.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.470+0000 7f2f55a00640 1 -- 172.21.15.90:0/2883873926 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f50159520 con 0x7f2f500cdac0 2024-03-24T01:25:41.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.471+0000 7f2f55a00640 1 -- 172.21.15.90:0/2883873926 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f50068060 con 0x7f2f500cdac0 2024-03-24T01:25:41.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.472+0000 7f2f47400640 1 -- 172.21.15.90:0/2883873926 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2f38038020 con 0x7f2f500cdac0 2024-03-24T01:25:41.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.473+0000 7f2f47400640 1 --2- 172.21.15.90:0/2883873926 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f2f2c06bbd0 0x7f2f2c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:41.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.473+0000 7f2f47400640 1 -- 172.21.15.90:0/2883873926 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f2f380ace30 con 0x7f2f500cdac0 2024-03-24T01:25:41.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.474+0000 7f2f4ea00640 1 --2- 172.21.15.90:0/2883873926 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f2f2c06bbd0 0x7f2f2c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:41.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.475+0000 7f2f4ea00640 1 --2- 172.21.15.90:0/2883873926 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f2f2c06bbd0 0x7f2f2c06e090 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2f40000f60 tx=0x7f2f40009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:41.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.478+0000 7f2f47400640 1 -- 172.21.15.90:0/2883873926 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2f38040070 con 0x7f2f500cdac0 2024-03-24T01:25:41.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.717+0000 7f2f55a00640 1 -- 172.21.15.90:0/2883873926 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f5014fbe0 con 0x7f2f500cdac0 2024-03-24T01:25:41.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.718+0000 7f2f47400640 1 -- 172.21.15.90:0/2883873926 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4967 (secure 0 0 0) 0x7f2f3807b4f0 con 0x7f2f500cdac0 2024-03-24T01:25:41.719 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:25:41.719 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/1898972903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":1898972903},{"type":"v1","addr":"172.21.15.120:6837","nonce":1898972903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6834/775759501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":775759501},{"type":"v1","addr":"172.21.15.90:6834","nonce":775759501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/43630575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":43630575},{"type":"v1","addr":"172.21.15.90:6835","nonce":43630575}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T01:24:10.173532+0000","modified":"2024-03-24T01:24:36.608477+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6836/963928022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":963928022},{"type":"v1","addr":"172.21.15.120:6836","nonce":963928022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4257,"qdb_cluster":[4257]},"id":1}]} 2024-03-24T01:25:41.719 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 8 2024-03-24T01:25:41.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.721+0000 7f2f55a00640 1 -- 172.21.15.90:0/2883873926 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f2f2c06bbd0 msgr2=0x7f2f2c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:41.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.721+0000 7f2f55a00640 1 --2- 172.21.15.90:0/2883873926 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f2f2c06bbd0 0x7f2f2c06e090 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2f40000f60 tx=0x7f2f40009040 comp rx=0 tx=0).stop 2024-03-24T01:25:41.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.722+0000 7f2f55a00640 1 -- 172.21.15.90:0/2883873926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f500cdac0 msgr2=0x7f2f50159d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:41.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.722+0000 7f2f55a00640 1 --2- 172.21.15.90:0/2883873926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f500cdac0 0x7f2f50159d20 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f2f38000c00 tx=0x7f2f380046c0 comp rx=0 tx=0).stop 2024-03-24T01:25:41.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.722+0000 7f2f55a00640 1 -- 172.21.15.90:0/2883873926 shutdown_connections 2024-03-24T01:25:41.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.722+0000 7f2f55a00640 1 --2- 172.21.15.90:0/2883873926 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f2f2c06bbd0 0x7f2f2c06e090 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:41.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.722+0000 7f2f55a00640 1 --2- 172.21.15.90:0/2883873926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f500cdac0 0x7f2f50159d20 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:41.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.722+0000 7f2f55a00640 1 --2- 172.21.15.90:0/2883873926 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f50155030 0x7f2f5013ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:41.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.722+0000 7f2f55a00640 1 --2- 172.21.15.90:0/2883873926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f5006b600 0x7f2f501e3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:41.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.722+0000 7f2f55a00640 1 -- 172.21.15.90:0/2883873926 >> 172.21.15.90:0/2883873926 conn(0x7f2f500b1770 msgr2=0x7f2f50071990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:41.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.722+0000 7f2f55a00640 1 -- 172.21.15.90:0/2883873926 shutdown_connections 2024-03-24T01:25:41.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.722+0000 7f2f55a00640 1 -- 172.21.15.90:0/2883873926 wait complete. 2024-03-24T01:25:41.743 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:25:41.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.954+0000 7fdb29e00640 1 Processor -- start 2024-03-24T01:25:41.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.954+0000 7fdb29e00640 1 -- start start 2024-03-24T01:25:41.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.955+0000 7fdb29e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdb2415df80 0x7fdb24162330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:41.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.955+0000 7fdb29e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb240ce4b0 con 0x7fdb240b5c40 2024-03-24T01:25:41.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.955+0000 7fdb29e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb240ce790 con 0x7fdb2415dba0 2024-03-24T01:25:41.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.955+0000 7fdb29e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb24162870 con 0x7fdb2415df80 2024-03-24T01:25:41.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.955+0000 7fdb23e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fdb240b5c40 0x7fdb240d1d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56068/0 (socket says 172.21.15.90:56068) 2024-03-24T01:25:41.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.955+0000 7fdb23e00640 1 -- 172.21.15.90:0/3142563554 learned_addr learned my addr 172.21.15.90:0/3142563554 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:41.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.956+0000 7fdb23400640 1 -- 172.21.15.90:0/3142563554 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2024209387 0 0) 0x7fdb240ce4b0 con 0x7fdb240b5c40 2024-03-24T01:25:41.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.956+0000 7fdb23400640 1 -- 172.21.15.90:0/3142563554 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb08003520 con 0x7fdb240b5c40 2024-03-24T01:25:41.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.956+0000 7fdb29400640 1 --2- 172.21.15.90:0/3142563554 >> v2:172.21.15.120:3300/0 conn(0x7fdb2415df80 0x7fdb24162330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:41.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.956+0000 7fdb23400640 1 -- 172.21.15.90:0/3142563554 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 421875945 0 0) 0x7fdb08003520 con 0x7fdb240b5c40 2024-03-24T01:25:41.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.956+0000 7fdb23400640 1 -- 172.21.15.90:0/3142563554 >> v2:172.21.15.120:3300/0 conn(0x7fdb2415df80 msgr2=0x7fdb24162330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:41.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.956+0000 7fdb23400640 1 --2- 172.21.15.90:0/3142563554 >> v2:172.21.15.120:3300/0 conn(0x7fdb2415df80 0x7fdb24162330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:41.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.956+0000 7fdb23400640 1 -- 172.21.15.90:0/3142563554 >> v1:172.21.15.120:6789/0 conn(0x7fdb2415dba0 legacy=0x7fdb240b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:41.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.956+0000 7fdb23400640 1 -- 172.21.15.90:0/3142563554 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb240cea70 con 0x7fdb240b5c40 2024-03-24T01:25:41.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.956+0000 7fdb23400640 1 -- 172.21.15.90:0/3142563554 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7fdb140019f0 con 0x7fdb240b5c40 2024-03-24T01:25:41.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.956+0000 7fdb23400640 1 -- 172.21.15.90:0/3142563554 >> v1:172.21.15.90:6789/0 conn(0x7fdb240b5c40 legacy=0x7fdb240d1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:41.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.956+0000 7fdb23400640 1 --2- 172.21.15.90:0/3142563554 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb080039b0 0x7fdb08003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:41.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.956+0000 7fdb23400640 1 --2- 172.21.15.90:0/3142563554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb08004800 0x7fdb08004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:41.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.956+0000 7fdb23400640 1 --2- 172.21.15.90:0/3142563554 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb080052e0 0x7fdb0800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:41.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.956+0000 7fdb23400640 1 -- 172.21.15.90:0/3142563554 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb24162870 con 0x7fdb08004800 2024-03-24T01:25:41.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.956+0000 7fdb23400640 1 -- 172.21.15.90:0/3142563554 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0800c1b0 con 0x7fdb080039b0 2024-03-24T01:25:41.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.956+0000 7fdb23400640 1 -- 172.21.15.90:0/3142563554 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0800c3b0 con 0x7fdb080052e0 2024-03-24T01:25:41.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.957+0000 7fdb23e00640 1 --2- 172.21.15.90:0/3142563554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb08004800 0x7fdb08004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:41.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.957+0000 7fdb23e00640 1 -- 172.21.15.90:0/3142563554 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb080052e0 msgr2=0x7fdb0800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:41.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.957+0000 7fdb23e00640 1 --2- 172.21.15.90:0/3142563554 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb080052e0 0x7fdb0800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:41.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.957+0000 7fdb23e00640 1 -- 172.21.15.90:0/3142563554 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb080039b0 msgr2=0x7fdb08003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:41.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.957+0000 7fdb23e00640 1 --2- 172.21.15.90:0/3142563554 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb080039b0 0x7fdb08003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:41.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.957+0000 7fdb23e00640 1 -- 172.21.15.90:0/3142563554 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb08003520 con 0x7fdb08004800 2024-03-24T01:25:41.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.957+0000 7fdb29400640 1 -- 172.21.15.90:0/3142563554 reap_dead start 2024-03-24T01:25:41.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.958+0000 7fdb23e00640 1 --2- 172.21.15.90:0/3142563554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb08004800 0x7fdb08004c80 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fdb14002230 tx=0x7fdb08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee2d35e5fbac5bd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:41.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.958+0000 7fdb23400640 1 -- 172.21.15.90:0/3142563554 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1400a4a0 con 0x7fdb08004800 2024-03-24T01:25:41.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.958+0000 7fdb23400640 1 -- 172.21.15.90:0/3142563554 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb14034650 con 0x7fdb08004800 2024-03-24T01:25:41.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.958+0000 7fdb23400640 1 -- 172.21.15.90:0/3142563554 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb14034a80 con 0x7fdb08004800 2024-03-24T01:25:41.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.959+0000 7fdb29e00640 1 -- 172.21.15.90:0/3142563554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb08004800 msgr2=0x7fdb08004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:41.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.959+0000 7fdb29e00640 1 --2- 172.21.15.90:0/3142563554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb08004800 0x7fdb08004c80 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fdb14002230 tx=0x7fdb08005830 comp rx=0 tx=0).stop 2024-03-24T01:25:41.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.959+0000 7fdb29e00640 1 -- 172.21.15.90:0/3142563554 shutdown_connections 2024-03-24T01:25:41.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.959+0000 7fdb29e00640 1 --2- 172.21.15.90:0/3142563554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb08004800 0x7fdb08004c80 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:41.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.959+0000 7fdb29e00640 1 -- 172.21.15.90:0/3142563554 >> 172.21.15.90:0/3142563554 conn(0x7fdb240d0530 msgr2=0x7fdb240d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:41.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.959+0000 7fdb29e00640 1 -- 172.21.15.90:0/3142563554 shutdown_connections 2024-03-24T01:25:41.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.959+0000 7fdb29e00640 1 -- 172.21.15.90:0/3142563554 wait complete. 2024-03-24T01:25:41.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.961+0000 7fdb29e00640 1 Processor -- start 2024-03-24T01:25:41.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.961+0000 7fdb29e00640 1 -- start start 2024-03-24T01:25:41.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.962+0000 7fdb29e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb08004800 0x7fdb240cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:41.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.962+0000 7fdb29e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb2414aad0 0x7fdb2414cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:41.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.962+0000 7fdb29e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb2414d060 0x7fdb2415a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:41.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.962+0000 7fdb29e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb240b2ae0 con 0x7fdb2414d060 2024-03-24T01:25:41.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.962+0000 7fdb29e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb240b2960 con 0x7fdb08004800 2024-03-24T01:25:41.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.962+0000 7fdb29e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb240b2c60 con 0x7fdb2414aad0 2024-03-24T01:25:41.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.962+0000 7fdb29400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb2414d060 0x7fdb2415a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:41.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.962+0000 7fdb29400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb2414d060 0x7fdb2415a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38768/0 (socket says 172.21.15.90:38768) 2024-03-24T01:25:41.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.962+0000 7fdb29400640 1 -- 172.21.15.90:0/3064201804 learned_addr learned my addr 172.21.15.90:0/3064201804 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:41.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.963+0000 7fdb23e00640 1 --2- 172.21.15.90:0/3064201804 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb2414aad0 0x7fdb2414cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:41.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.963+0000 7fdb29400640 1 -- 172.21.15.90:0/3064201804 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb2414aad0 msgr2=0x7fdb2414cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:41.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.963+0000 7fdb29400640 1 --2- 172.21.15.90:0/3064201804 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb2414aad0 0x7fdb2414cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:41.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.963+0000 7fdb29400640 1 -- 172.21.15.90:0/3064201804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb08004800 msgr2=0x7fdb240cd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:41.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.963+0000 7fdb29400640 1 --2- 172.21.15.90:0/3064201804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb08004800 0x7fdb240cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:41.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.963+0000 7fdb29400640 1 -- 172.21.15.90:0/3064201804 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb2414d720 con 0x7fdb2414d060 2024-03-24T01:25:41.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.963+0000 7fdb23e00640 1 --2- 172.21.15.90:0/3064201804 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb2414aad0 0x7fdb2414cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:41.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.963+0000 7fdb29400640 1 --2- 172.21.15.90:0/3064201804 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb2414d060 0x7fdb2415a540 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fdb2415e3f0 tx=0x7fdb1c00d7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:41.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.963+0000 7fdb20c00640 1 -- 172.21.15.90:0/3064201804 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c004020 con 0x7fdb2414d060 2024-03-24T01:25:41.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.963+0000 7fdb20c00640 1 -- 172.21.15.90:0/3064201804 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb1c004200 con 0x7fdb2414d060 2024-03-24T01:25:41.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.963+0000 7fdb29e00640 1 -- 172.21.15.90:0/3064201804 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb2414da00 con 0x7fdb2414d060 2024-03-24T01:25:41.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.963+0000 7fdb20c00640 1 -- 172.21.15.90:0/3064201804 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c0043f0 con 0x7fdb2414d060 2024-03-24T01:25:41.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.963+0000 7fdb29e00640 1 -- 172.21.15.90:0/3064201804 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb2415f7f0 con 0x7fdb2414d060 2024-03-24T01:25:41.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.964+0000 7fdb29e00640 1 -- 172.21.15.90:0/3064201804 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb2404f400 con 0x7fdb2414d060 2024-03-24T01:25:41.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.965+0000 7fdb20c00640 1 -- 172.21.15.90:0/3064201804 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdb1c01a020 con 0x7fdb2414d060 2024-03-24T01:25:41.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.966+0000 7fdb20c00640 1 --2- 172.21.15.90:0/3064201804 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fdb0006bbd0 0x7fdb0006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:41.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.966+0000 7fdb20c00640 1 -- 172.21.15.90:0/3064201804 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7fdb1c08a6c0 con 0x7fdb2414d060 2024-03-24T01:25:41.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.967+0000 7fdb28a00640 1 --2- 172.21.15.90:0/3064201804 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fdb0006bbd0 0x7fdb0006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:41.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.968+0000 7fdb28a00640 1 --2- 172.21.15.90:0/3064201804 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fdb0006bbd0 0x7fdb0006e090 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c001590 tx=0x7fdb0c007990 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:41.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:41.971+0000 7fdb20c00640 1 -- 172.21.15.90:0/3064201804 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdb1c058d80 con 0x7fdb2414d060 2024-03-24T01:25:42.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.169+0000 7fdb29e00640 1 -- 172.21.15.90:0/3064201804 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb240d17c0 con 0x7fdb2414d060 2024-03-24T01:25:42.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.171+0000 7fdb20c00640 1 -- 172.21.15.90:0/3064201804 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18925 (secure 0 0 0) 0x7fdb1c058680 con 0x7fdb2414d060 2024-03-24T01:25:42.172 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:25:42.172 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:24:53.236401+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:24:08.643966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T01:24:09.189904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T01:24:13.027323+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:25:42.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.174+0000 7fdb29e00640 1 -- 172.21.15.90:0/3064201804 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fdb0006bbd0 msgr2=0x7fdb0006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:42.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.174+0000 7fdb29e00640 1 --2- 172.21.15.90:0/3064201804 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fdb0006bbd0 0x7fdb0006e090 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c001590 tx=0x7fdb0c007990 comp rx=0 tx=0).stop 2024-03-24T01:25:42.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.174+0000 7fdb29e00640 1 -- 172.21.15.90:0/3064201804 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb2414d060 msgr2=0x7fdb2415a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:42.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.174+0000 7fdb29e00640 1 --2- 172.21.15.90:0/3064201804 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb2414d060 0x7fdb2415a540 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fdb2415e3f0 tx=0x7fdb1c00d7e0 comp rx=0 tx=0).stop 2024-03-24T01:25:42.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.174+0000 7fdb29e00640 1 -- 172.21.15.90:0/3064201804 shutdown_connections 2024-03-24T01:25:42.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.174+0000 7fdb29e00640 1 --2- 172.21.15.90:0/3064201804 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fdb0006bbd0 0x7fdb0006e090 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:42.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.174+0000 7fdb29e00640 1 --2- 172.21.15.90:0/3064201804 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb2414d060 0x7fdb2415a540 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:42.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.174+0000 7fdb29e00640 1 --2- 172.21.15.90:0/3064201804 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb2414aad0 0x7fdb2414cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:42.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.174+0000 7fdb29e00640 1 --2- 172.21.15.90:0/3064201804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb08004800 0x7fdb240cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:42.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.174+0000 7fdb29e00640 1 -- 172.21.15.90:0/3064201804 >> 172.21.15.90:0/3064201804 conn(0x7fdb240d0530 msgr2=0x7fdb240b5420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:42.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.175+0000 7fdb29e00640 1 -- 172.21.15.90:0/3064201804 shutdown_connections 2024-03-24T01:25:42.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.175+0000 7fdb29e00640 1 -- 172.21.15.90:0/3064201804 wait complete. 2024-03-24T01:25:42.193 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T01:24:08.643966+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T01:24:09.189904+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T01:24:13.027323+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T01:25:42.193 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:25:42.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.411+0000 7fcce7800640 1 Processor -- start 2024-03-24T01:25:42.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.411+0000 7fcce7800640 1 -- start start 2024-03-24T01:25:42.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.412+0000 7fcce7800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcce00aeeb0 0x7fcce00b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:42.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.412+0000 7fcce7800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcce006b7d0 0x7fcce00b2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:42.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.412+0000 7fcce7800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce006dac0 con 0x7fcce015bc50 2024-03-24T01:25:42.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.412+0000 7fcce7800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcce00b4da0 con 0x7fcce00aeeb0 2024-03-24T01:25:42.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.412+0000 7fcce7800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce006dda0 con 0x7fcce006b7d0 2024-03-24T01:25:42.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.413+0000 7fcce5800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcce006b7d0 0x7fcce00b2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:42.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.413+0000 7fcce5800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcce006b7d0 0x7fcce00b2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.90:49192/0 (socket says 172.21.15.90:49192) 2024-03-24T01:25:42.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.413+0000 7fcce5800640 1 -- 172.21.15.90:0/3567453676 learned_addr learned my addr 172.21.15.90:0/3567453676 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:42.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.413+0000 7fcce4e00640 1 --2- 172.21.15.90:0/3567453676 >> v2:172.21.15.120:3300/0 conn(0x7fcce00aeeb0 0x7fcce00b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:42.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.414+0000 7fcce5800640 1 -- 172.21.15.90:0/3567453676 >> v2:172.21.15.120:3300/0 conn(0x7fcce00aeeb0 msgr2=0x7fcce00b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:42.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.414+0000 7fcce5800640 1 --2- 172.21.15.90:0/3567453676 >> v2:172.21.15.120:3300/0 conn(0x7fcce00aeeb0 0x7fcce00b44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:42.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.414+0000 7fcce5800640 1 -- 172.21.15.90:0/3567453676 >> v1:172.21.15.120:6789/0 conn(0x7fcce015bc50 legacy=0x7fcce006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:42.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.414+0000 7fcce5800640 1 -- 172.21.15.90:0/3567453676 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce006e080 con 0x7fcce006b7d0 2024-03-24T01:25:42.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.415+0000 7fcce5800640 1 --2- 172.21.15.90:0/3567453676 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcce006b7d0 0x7fcce00b2890 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fccd800d150 tx=0x7fccd802e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65762102c70b828a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:42.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.416+0000 7fccdfe00640 1 -- 172.21.15.90:0/3567453676 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd802e9e0 con 0x7fcce006b7d0 2024-03-24T01:25:42.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.416+0000 7fccdfe00640 1 -- 172.21.15.90:0/3567453676 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd802eb80 con 0x7fcce006b7d0 2024-03-24T01:25:42.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.416+0000 7fccdfe00640 1 -- 172.21.15.90:0/3567453676 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd8009600 con 0x7fcce006b7d0 2024-03-24T01:25:42.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.416+0000 7fcce7800640 1 -- 172.21.15.90:0/3567453676 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcce006b7d0 msgr2=0x7fcce00b2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:42.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.416+0000 7fcce7800640 1 --2- 172.21.15.90:0/3567453676 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcce006b7d0 0x7fcce00b2890 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fccd800d150 tx=0x7fccd802e040 comp rx=0 tx=0).stop 2024-03-24T01:25:42.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.417+0000 7fcce7800640 1 -- 172.21.15.90:0/3567453676 shutdown_connections 2024-03-24T01:25:42.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.417+0000 7fcce7800640 1 --2- 172.21.15.90:0/3567453676 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcce006b7d0 0x7fcce00b2890 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:42.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.417+0000 7fcce7800640 1 --2- 172.21.15.90:0/3567453676 >> v2:172.21.15.120:3300/0 conn(0x7fcce00aeeb0 0x7fcce00b44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:42.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.417+0000 7fcce7800640 1 -- 172.21.15.90:0/3567453676 >> 172.21.15.90:0/3567453676 conn(0x7fcce00b1770 msgr2=0x7fcce00b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:42.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.417+0000 7fcce7800640 1 -- 172.21.15.90:0/3567453676 shutdown_connections 2024-03-24T01:25:42.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.417+0000 7fcce7800640 1 -- 172.21.15.90:0/3567453676 wait complete. 2024-03-24T01:25:42.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.418+0000 7fcce7800640 1 Processor -- start 2024-03-24T01:25:42.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.419+0000 7fcce7800640 1 -- start start 2024-03-24T01:25:42.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.419+0000 7fcce7800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcce006b7d0 0x7fcce014b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:42.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.419+0000 7fcce7800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcce00aeeb0 0x7fcce00afeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:42.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.419+0000 7fcce7800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcce015bc50 0x7fcce01f6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:42.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.419+0000 7fcce7800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce00b4a00 con 0x7fcce006b7d0 2024-03-24T01:25:42.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.419+0000 7fcce7800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce00af290 con 0x7fcce015bc50 2024-03-24T01:25:42.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.419+0000 7fcce7800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce00b4b80 con 0x7fcce00aeeb0 2024-03-24T01:25:42.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.420+0000 7fcce4e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcce006b7d0 0x7fcce014b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:42.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.420+0000 7fcce4e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcce006b7d0 0x7fcce014b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38776/0 (socket says 172.21.15.90:38776) 2024-03-24T01:25:42.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.420+0000 7fcce4e00640 1 -- 172.21.15.90:0/1532836161 learned_addr learned my addr 172.21.15.90:0/1532836161 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:42.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.420+0000 7fccd7e00640 1 --2- 172.21.15.90:0/1532836161 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcce00aeeb0 0x7fcce00afeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:42.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.420+0000 7fcce5800640 1 --2- 172.21.15.90:0/1532836161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcce015bc50 0x7fcce01f6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:42.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.420+0000 7fcce4e00640 1 -- 172.21.15.90:0/1532836161 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcce00aeeb0 msgr2=0x7fcce00afeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:42.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.420+0000 7fcce4e00640 1 --2- 172.21.15.90:0/1532836161 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcce00aeeb0 0x7fcce00afeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:42.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.420+0000 7fcce4e00640 1 -- 172.21.15.90:0/1532836161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcce015bc50 msgr2=0x7fcce01f6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:42.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.420+0000 7fcce4e00640 1 --2- 172.21.15.90:0/1532836161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcce015bc50 0x7fcce01f6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:42.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.420+0000 7fcce4e00640 1 -- 172.21.15.90:0/1532836161 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce00b07b0 con 0x7fcce006b7d0 2024-03-24T01:25:42.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.421+0000 7fcce4e00640 1 --2- 172.21.15.90:0/1532836161 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcce006b7d0 0x7fcce014b020 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fccd000e9b0 tx=0x7fccd000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:42.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.421+0000 7fccdd600640 1 -- 172.21.15.90:0/1532836161 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd0017070 con 0x7fcce006b7d0 2024-03-24T01:25:42.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.421+0000 7fccdd600640 1 -- 172.21.15.90:0/1532836161 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd000c9a0 con 0x7fcce006b7d0 2024-03-24T01:25:42.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.421+0000 7fccdd600640 1 -- 172.21.15.90:0/1532836161 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd000cd90 con 0x7fcce006b7d0 2024-03-24T01:25:42.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.421+0000 7fcce7800640 1 -- 172.21.15.90:0/1532836161 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcce00b0a90 con 0x7fcce006b7d0 2024-03-24T01:25:42.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.421+0000 7fcce7800640 1 -- 172.21.15.90:0/1532836161 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcce014b860 con 0x7fcce006b7d0 2024-03-24T01:25:42.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.422+0000 7fcce7800640 1 -- 172.21.15.90:0/1532836161 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcce004f400 con 0x7fcce006b7d0 2024-03-24T01:25:42.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.423+0000 7fccdd600640 1 -- 172.21.15.90:0/1532836161 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fccd001a070 con 0x7fcce006b7d0 2024-03-24T01:25:42.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.424+0000 7fccdd600640 1 --2- 172.21.15.90:0/1532836161 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fccbc06bbd0 0x7fccbc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:42.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.425+0000 7fccdd600640 1 -- 172.21.15.90:0/1532836161 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7fccd008b270 con 0x7fcce006b7d0 2024-03-24T01:25:42.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.425+0000 7fccd7e00640 1 --2- 172.21.15.90:0/1532836161 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fccbc06bbd0 0x7fccbc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:42.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.426+0000 7fccd7e00640 1 --2- 172.21.15.90:0/1532836161 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fccbc06bbd0 0x7fccbc06e090 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcce0158210 tx=0x7fccc8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:42.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.429+0000 7fccdd600640 1 -- 172.21.15.90:0/1532836161 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fccd0059940 con 0x7fcce006b7d0 2024-03-24T01:25:42.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.629+0000 7fcce7800640 1 -- 172.21.15.90:0/1532836161 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcce006ea00 con 0x7fcce006b7d0 2024-03-24T01:25:42.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.630+0000 7fccdd600640 1 -- 172.21.15.90:0/1532836161 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fccd0059240 con 0x7fcce006b7d0 2024-03-24T01:25:42.631 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:25:42.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.633+0000 7fcce7800640 1 -- 172.21.15.90:0/1532836161 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fccbc06bbd0 msgr2=0x7fccbc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:42.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.633+0000 7fcce7800640 1 --2- 172.21.15.90:0/1532836161 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fccbc06bbd0 0x7fccbc06e090 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcce0158210 tx=0x7fccc8009040 comp rx=0 tx=0).stop 2024-03-24T01:25:42.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.633+0000 7fcce7800640 1 -- 172.21.15.90:0/1532836161 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcce006b7d0 msgr2=0x7fcce014b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:42.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.633+0000 7fcce7800640 1 --2- 172.21.15.90:0/1532836161 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcce006b7d0 0x7fcce014b020 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fccd000e9b0 tx=0x7fccd000ee70 comp rx=0 tx=0).stop 2024-03-24T01:25:42.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.633+0000 7fcce7800640 1 -- 172.21.15.90:0/1532836161 shutdown_connections 2024-03-24T01:25:42.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.633+0000 7fcce7800640 1 --2- 172.21.15.90:0/1532836161 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fccbc06bbd0 0x7fccbc06e090 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:42.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.633+0000 7fcce7800640 1 --2- 172.21.15.90:0/1532836161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcce015bc50 0x7fcce01f6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:42.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.633+0000 7fcce7800640 1 --2- 172.21.15.90:0/1532836161 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcce00aeeb0 0x7fcce00afeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:42.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.633+0000 7fcce7800640 1 --2- 172.21.15.90:0/1532836161 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcce006b7d0 0x7fcce014b020 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:42.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.633+0000 7fcce7800640 1 -- 172.21.15.90:0/1532836161 >> 172.21.15.90:0/1532836161 conn(0x7fcce00b1770 msgr2=0x7fcce00ce970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:42.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.633+0000 7fcce7800640 1 -- 172.21.15.90:0/1532836161 shutdown_connections 2024-03-24T01:25:42.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.633+0000 7fcce7800640 1 -- 172.21.15.90:0/1532836161 wait complete. 2024-03-24T01:25:42.653 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T01:25:42.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.856+0000 7f700e000640 1 Processor -- start 2024-03-24T01:25:42.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.857+0000 7f700e000640 1 -- start start 2024-03-24T01:25:42.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.858+0000 7f700e000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7008071990 0x7f7008071d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:42.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.858+0000 7f700e000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f70080b3ed0 0x7f70080b4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:42.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.858+0000 7f700e000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70080722b0 con 0x7f70080b2890 2024-03-24T01:25:42.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.858+0000 7f700e000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f700806ebd0 con 0x7f7008071990 2024-03-24T01:25:42.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.858+0000 7f700e000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f700806db30 con 0x7f70080b3ed0 2024-03-24T01:25:42.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.858+0000 7f7007400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7008071990 0x7f7008071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:42.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.858+0000 7f7007400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7008071990 0x7f7008071d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38778/0 (socket says 172.21.15.90:38778) 2024-03-24T01:25:42.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.858+0000 7f7007400640 1 -- 172.21.15.90:0/3279672207 learned_addr learned my addr 172.21.15.90:0/3279672207 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:42.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.858+0000 7f7007e00640 1 --1- 172.21.15.90:0/3279672207 >> v1:172.21.15.90:6789/0 conn(0x7f70080b2890 0x7f70080b2d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56082/0 (socket says 172.21.15.90:56082) 2024-03-24T01:25:42.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.859+0000 7f6ffea00640 1 --2- 172.21.15.90:0/3279672207 >> v2:172.21.15.120:3300/0 conn(0x7f70080b3ed0 0x7f70080b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:42.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.859+0000 7f7007400640 1 -- 172.21.15.90:0/3279672207 >> v2:172.21.15.120:3300/0 conn(0x7f70080b3ed0 msgr2=0x7f70080b4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:42.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.859+0000 7f7007400640 1 --2- 172.21.15.90:0/3279672207 >> v2:172.21.15.120:3300/0 conn(0x7f70080b3ed0 0x7f70080b4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:42.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.859+0000 7f7007400640 1 -- 172.21.15.90:0/3279672207 >> v1:172.21.15.90:6789/0 conn(0x7f70080b2890 legacy=0x7f70080b2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:42.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.859+0000 7f7007400640 1 -- 172.21.15.90:0/3279672207 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f700806de10 con 0x7f7008071990 2024-03-24T01:25:42.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.859+0000 7f7006a00640 1 -- 172.21.15.90:0/3279672207 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2474696603 0 0) 0x7f70080722b0 con 0x7f70080b2890 2024-03-24T01:25:42.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.860+0000 7f7007400640 1 --2- 172.21.15.90:0/3279672207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7008071990 0x7f7008071d70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8009870 tx=0x7f6ff802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3163d7e55e578ecd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:42.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.860+0000 7f7006a00640 1 -- 172.21.15.90:0/3279672207 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff8035020 con 0x7f7008071990 2024-03-24T01:25:42.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.860+0000 7f7006a00640 1 -- 172.21.15.90:0/3279672207 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff8031870 con 0x7f7008071990 2024-03-24T01:25:42.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.860+0000 7f7006a00640 1 -- 172.21.15.90:0/3279672207 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff8031c40 con 0x7f7008071990 2024-03-24T01:25:42.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.861+0000 7f700e000640 1 -- 172.21.15.90:0/3279672207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7008071990 msgr2=0x7f7008071d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:42.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.861+0000 7f700e000640 1 --2- 172.21.15.90:0/3279672207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7008071990 0x7f7008071d70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8009870 tx=0x7f6ff802b910 comp rx=0 tx=0).stop 2024-03-24T01:25:42.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.861+0000 7f700e000640 1 -- 172.21.15.90:0/3279672207 shutdown_connections 2024-03-24T01:25:42.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.861+0000 7f700e000640 1 --2- 172.21.15.90:0/3279672207 >> v2:172.21.15.120:3300/0 conn(0x7f70080b3ed0 0x7f70080b4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:42.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.861+0000 7f700e000640 1 --2- 172.21.15.90:0/3279672207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7008071990 0x7f7008071d70 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:42.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.861+0000 7f700e000640 1 -- 172.21.15.90:0/3279672207 >> 172.21.15.90:0/3279672207 conn(0x7f70080b1770 msgr2=0x7f70080b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:42.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.861+0000 7f700e000640 1 -- 172.21.15.90:0/3279672207 shutdown_connections 2024-03-24T01:25:42.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.861+0000 7f700e000640 1 -- 172.21.15.90:0/3279672207 wait complete. 2024-03-24T01:25:42.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.863+0000 7f700e000640 1 Processor -- start 2024-03-24T01:25:42.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.863+0000 7f700e000640 1 -- start start 2024-03-24T01:25:42.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.864+0000 7f700e000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7008071990 0x7f700814eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:42.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.864+0000 7f700e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70080b2890 0x7f7008152540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:42.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.864+0000 7f700e000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70080b3ed0 0x7f7008146cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:42.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.864+0000 7f700e000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f700806e6b0 con 0x7f70080b2890 2024-03-24T01:25:42.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.864+0000 7f700e000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f700806e3d0 con 0x7f70080b3ed0 2024-03-24T01:25:42.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.864+0000 7f700e000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70080b4870 con 0x7f7008071990 2024-03-24T01:25:42.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.864+0000 7f6ffea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70080b2890 0x7f7008152540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:42.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.864+0000 7f6ffea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70080b2890 0x7f7008152540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38780/0 (socket says 172.21.15.90:38780) 2024-03-24T01:25:42.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.864+0000 7f6ffea00640 1 -- 172.21.15.90:0/2331858442 learned_addr learned my addr 172.21.15.90:0/2331858442 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:42.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.865+0000 7f7007400640 1 --2- 172.21.15.90:0/2331858442 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7008071990 0x7f700814eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:42.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.865+0000 7f7007e00640 1 --2- 172.21.15.90:0/2331858442 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70080b3ed0 0x7f7008146cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:42.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.865+0000 7f6ffea00640 1 -- 172.21.15.90:0/2331858442 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7008071990 msgr2=0x7f700814eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:42.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.865+0000 7f6ffea00640 1 --2- 172.21.15.90:0/2331858442 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7008071990 0x7f700814eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:42.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.865+0000 7f6ffea00640 1 -- 172.21.15.90:0/2331858442 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70080b3ed0 msgr2=0x7f7008146cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:42.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.865+0000 7f6ffea00640 1 --2- 172.21.15.90:0/2331858442 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70080b3ed0 0x7f7008146cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:42.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.865+0000 7f6ffea00640 1 -- 172.21.15.90:0/2331858442 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f700814f400 con 0x7f70080b2890 2024-03-24T01:25:42.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.865+0000 7f7007400640 1 --2- 172.21.15.90:0/2331858442 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7008071990 0x7f700814eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:42.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.865+0000 7f7007e00640 1 --2- 172.21.15.90:0/2331858442 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70080b3ed0 0x7f7008146cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:42.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.865+0000 7f6ffea00640 1 --2- 172.21.15.90:0/2331858442 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70080b2890 0x7f7008152540 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f6ff000d620 tx=0x7f6ff000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:42.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.865+0000 7f6fffe00640 1 -- 172.21.15.90:0/2331858442 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff000b620 con 0x7f70080b2890 2024-03-24T01:25:42.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.865+0000 7f6fffe00640 1 -- 172.21.15.90:0/2331858442 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff0019070 con 0x7f70080b2890 2024-03-24T01:25:42.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.865+0000 7f6fffe00640 1 -- 172.21.15.90:0/2331858442 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff000ba00 con 0x7f70080b2890 2024-03-24T01:25:42.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.865+0000 7f700e000640 1 -- 172.21.15.90:0/2331858442 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70081310d0 con 0x7f70080b2890 2024-03-24T01:25:42.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.865+0000 7f700e000640 1 -- 172.21.15.90:0/2331858442 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7008131690 con 0x7f70080b2890 2024-03-24T01:25:42.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.866+0000 7f700e000640 1 -- 172.21.15.90:0/2331858442 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f700804f400 con 0x7f70080b2890 2024-03-24T01:25:42.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.867+0000 7f6fffe00640 1 -- 172.21.15.90:0/2331858442 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6ff000bba0 con 0x7f70080b2890 2024-03-24T01:25:42.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.868+0000 7f6fffe00640 1 --2- 172.21.15.90:0/2331858442 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6fe406bbd0 0x7f6fe406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:42.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.868+0000 7f6fffe00640 1 -- 172.21.15.90:0/2331858442 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f6ff008b220 con 0x7f70080b2890 2024-03-24T01:25:42.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.869+0000 7f7007400640 1 --2- 172.21.15.90:0/2331858442 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6fe406bbd0 0x7f6fe406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:42.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.870+0000 7f7007400640 1 --2- 172.21.15.90:0/2331858442 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6fe406bbd0 0x7f6fe406e090 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8036000 tx=0x7f6ff8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:42.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:42.872+0000 7f6fffe00640 1 -- 172.21.15.90:0/2331858442 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6ff0059980 con 0x7f70080b2890 2024-03-24T01:25:43.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.074+0000 7f700e000640 1 -- 172.21.15.90:0/2331858442 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f70080ce6c0 con 0x7f70080b2890 2024-03-24T01:25:43.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.075+0000 7f6fffe00640 1 -- 172.21.15.90:0/2331858442 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6ff0059280 con 0x7f70080b2890 2024-03-24T01:25:43.075 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:25:43.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.078+0000 7f700e000640 1 -- 172.21.15.90:0/2331858442 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6fe406bbd0 msgr2=0x7f6fe406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:43.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.078+0000 7f700e000640 1 --2- 172.21.15.90:0/2331858442 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6fe406bbd0 0x7f6fe406e090 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8036000 tx=0x7f6ff8008030 comp rx=0 tx=0).stop 2024-03-24T01:25:43.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.078+0000 7f700e000640 1 -- 172.21.15.90:0/2331858442 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70080b2890 msgr2=0x7f7008152540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:43.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.078+0000 7f700e000640 1 --2- 172.21.15.90:0/2331858442 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70080b2890 0x7f7008152540 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f6ff000d620 tx=0x7f6ff000dae0 comp rx=0 tx=0).stop 2024-03-24T01:25:43.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.078+0000 7f700e000640 1 -- 172.21.15.90:0/2331858442 shutdown_connections 2024-03-24T01:25:43.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.078+0000 7f700e000640 1 --2- 172.21.15.90:0/2331858442 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6fe406bbd0 0x7f6fe406e090 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:43.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.078+0000 7f700e000640 1 --2- 172.21.15.90:0/2331858442 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70080b2890 0x7f7008152540 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:43.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.078+0000 7f700e000640 1 --2- 172.21.15.90:0/2331858442 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70080b3ed0 0x7f7008146cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:43.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.078+0000 7f700e000640 1 --2- 172.21.15.90:0/2331858442 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7008071990 0x7f700814eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:43.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.078+0000 7f700e000640 1 -- 172.21.15.90:0/2331858442 >> 172.21.15.90:0/2331858442 conn(0x7f70080b1770 msgr2=0x7f7008149fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:43.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.078+0000 7f700e000640 1 -- 172.21.15.90:0/2331858442 shutdown_connections 2024-03-24T01:25:43.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.078+0000 7f700e000640 1 -- 172.21.15.90:0/2331858442 wait complete. 2024-03-24T01:25:43.097 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T01:25:43.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.300+0000 7fdf88600640 1 Processor -- start 2024-03-24T01:25:43.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.300+0000 7fdf88600640 1 -- start start 2024-03-24T01:25:43.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.302+0000 7fdf88600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf800aeeb0 0x7fdf800b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:43.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.302+0000 7fdf88600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdf8015bc50 0x7fdf8006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:43.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.302+0000 7fdf88600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdf8006b600 0x7fdf8015e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:43.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.302+0000 7fdf88600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf8006ba70 con 0x7fdf8015bc50 2024-03-24T01:25:43.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.302+0000 7fdf88600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf800b4da0 con 0x7fdf8006b600 2024-03-24T01:25:43.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.302+0000 7fdf88600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8006dac0 con 0x7fdf800aeeb0 2024-03-24T01:25:43.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.302+0000 7fdf85200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdf8015bc50 0x7fdf8006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:43.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.302+0000 7fdf85200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdf8015bc50 0x7fdf8006b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38788/0 (socket says 172.21.15.90:38788) 2024-03-24T01:25:43.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.302+0000 7fdf85200640 1 -- 172.21.15.90:0/1442990805 learned_addr learned my addr 172.21.15.90:0/1442990805 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:43.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.302+0000 7fdf85c00640 1 --2- 172.21.15.90:0/1442990805 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf800aeeb0 0x7fdf800b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:43.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.303+0000 7fdf85200640 1 -- 172.21.15.90:0/1442990805 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf800aeeb0 msgr2=0x7fdf800b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:43.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.303+0000 7fdf85200640 1 --2- 172.21.15.90:0/1442990805 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf800aeeb0 0x7fdf800b44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:43.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.303+0000 7fdf85200640 1 -- 172.21.15.90:0/1442990805 >> v2:172.21.15.120:3300/0 conn(0x7fdf8006b600 msgr2=0x7fdf8015e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:43.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.303+0000 7fdf86600640 1 --2- 172.21.15.90:0/1442990805 >> v2:172.21.15.120:3300/0 conn(0x7fdf8006b600 0x7fdf8015e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:43.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.303+0000 7fdf85200640 1 --2- 172.21.15.90:0/1442990805 >> v2:172.21.15.120:3300/0 conn(0x7fdf8006b600 0x7fdf8015e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:43.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.303+0000 7fdf85200640 1 -- 172.21.15.90:0/1442990805 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf8006dda0 con 0x7fdf8015bc50 2024-03-24T01:25:43.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.303+0000 7fdf86600640 1 --2- 172.21.15.90:0/1442990805 >> v2:172.21.15.120:3300/0 conn(0x7fdf8006b600 0x7fdf8015e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:43.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.304+0000 7fdf85200640 1 --2- 172.21.15.90:0/1442990805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf8015bc50 0x7fdf8006b0c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fdf800cc5d0 tx=0x7fdf6c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=916e296a690bd901 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:43.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.304+0000 7fdf7fe00640 1 -- 172.21.15.90:0/1442990805 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c038070 con 0x7fdf8015bc50 2024-03-24T01:25:43.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.304+0000 7fdf7fe00640 1 -- 172.21.15.90:0/1442990805 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf6c02f050 con 0x7fdf8015bc50 2024-03-24T01:25:43.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.304+0000 7fdf7fe00640 1 -- 172.21.15.90:0/1442990805 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c031970 con 0x7fdf8015bc50 2024-03-24T01:25:43.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.305+0000 7fdf88600640 1 -- 172.21.15.90:0/1442990805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf8015bc50 msgr2=0x7fdf8006b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:43.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.305+0000 7fdf88600640 1 --2- 172.21.15.90:0/1442990805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf8015bc50 0x7fdf8006b0c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fdf800cc5d0 tx=0x7fdf6c02b7f0 comp rx=0 tx=0).stop 2024-03-24T01:25:43.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.305+0000 7fdf88600640 1 -- 172.21.15.90:0/1442990805 shutdown_connections 2024-03-24T01:25:43.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.305+0000 7fdf88600640 1 --2- 172.21.15.90:0/1442990805 >> v2:172.21.15.120:3300/0 conn(0x7fdf8006b600 0x7fdf8015e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:43.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.305+0000 7fdf88600640 1 --2- 172.21.15.90:0/1442990805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf8015bc50 0x7fdf8006b0c0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:43.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.305+0000 7fdf88600640 1 --2- 172.21.15.90:0/1442990805 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf800aeeb0 0x7fdf800b44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:43.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.305+0000 7fdf88600640 1 -- 172.21.15.90:0/1442990805 >> 172.21.15.90:0/1442990805 conn(0x7fdf800b1770 msgr2=0x7fdf800b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:43.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.305+0000 7fdf88600640 1 -- 172.21.15.90:0/1442990805 shutdown_connections 2024-03-24T01:25:43.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.305+0000 7fdf88600640 1 -- 172.21.15.90:0/1442990805 wait complete. 2024-03-24T01:25:43.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.307+0000 7fdf88600640 1 Processor -- start 2024-03-24T01:25:43.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.307+0000 7fdf88600640 1 -- start start 2024-03-24T01:25:43.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.308+0000 7fdf88600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf8006b600 0x7fdf800b0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:43.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.308+0000 7fdf88600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf800aeeb0 0x7fdf800b3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:43.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.308+0000 7fdf88600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf8015bc50 0x7fdf800b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:43.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.308+0000 7fdf88600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf800af290 con 0x7fdf8015bc50 2024-03-24T01:25:43.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.308+0000 7fdf88600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8006e360 con 0x7fdf8006b600 2024-03-24T01:25:43.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.308+0000 7fdf88600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf800b4a00 con 0x7fdf800aeeb0 2024-03-24T01:25:43.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.308+0000 7fdf86600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf8015bc50 0x7fdf800b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:43.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.308+0000 7fdf86600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf8015bc50 0x7fdf800b3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38804/0 (socket says 172.21.15.90:38804) 2024-03-24T01:25:43.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.308+0000 7fdf86600640 1 -- 172.21.15.90:0/1198340658 learned_addr learned my addr 172.21.15.90:0/1198340658 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:43.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.309+0000 7fdf85c00640 1 --2- 172.21.15.90:0/1198340658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf8006b600 0x7fdf800b0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:43.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.309+0000 7fdf85200640 1 --2- 172.21.15.90:0/1198340658 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf800aeeb0 0x7fdf800b3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:43.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.309+0000 7fdf86600640 1 -- 172.21.15.90:0/1198340658 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf800aeeb0 msgr2=0x7fdf800b3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:43.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.309+0000 7fdf86600640 1 --2- 172.21.15.90:0/1198340658 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf800aeeb0 0x7fdf800b3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:43.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.309+0000 7fdf86600640 1 -- 172.21.15.90:0/1198340658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf8006b600 msgr2=0x7fdf800b0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:43.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.309+0000 7fdf86600640 1 --2- 172.21.15.90:0/1198340658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf8006b600 0x7fdf800b0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:43.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.309+0000 7fdf86600640 1 -- 172.21.15.90:0/1198340658 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf800b4660 con 0x7fdf8015bc50 2024-03-24T01:25:43.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.309+0000 7fdf86600640 1 --2- 172.21.15.90:0/1198340658 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf8015bc50 0x7fdf800b3fc0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fdf7800cd50 tx=0x7fdf7800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:43.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.309+0000 7fdf7d600640 1 -- 172.21.15.90:0/1198340658 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7800f680 con 0x7fdf8015bc50 2024-03-24T01:25:43.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.309+0000 7fdf7d600640 1 -- 172.21.15.90:0/1198340658 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf7800f840 con 0x7fdf8015bc50 2024-03-24T01:25:43.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.309+0000 7fdf88600640 1 -- 172.21.15.90:0/1198340658 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf800b2720 con 0x7fdf8015bc50 2024-03-24T01:25:43.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.309+0000 7fdf7d600640 1 -- 172.21.15.90:0/1198340658 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7800fc30 con 0x7fdf8015bc50 2024-03-24T01:25:43.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.309+0000 7fdf88600640 1 -- 172.21.15.90:0/1198340658 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf800b2ce0 con 0x7fdf8015bc50 2024-03-24T01:25:43.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.311+0000 7fdf88600640 1 -- 172.21.15.90:0/1198340658 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf8004f400 con 0x7fdf8015bc50 2024-03-24T01:25:43.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.311+0000 7fdf7d600640 1 -- 172.21.15.90:0/1198340658 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdf780213f0 con 0x7fdf8015bc50 2024-03-24T01:25:43.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.312+0000 7fdf7d600640 1 --2- 172.21.15.90:0/1198340658 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fdf6006bbd0 0x7fdf6006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:43.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.312+0000 7fdf7d600640 1 -- 172.21.15.90:0/1198340658 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7fdf7808bca0 con 0x7fdf8015bc50 2024-03-24T01:25:43.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.313+0000 7fdf85c00640 1 --2- 172.21.15.90:0/1198340658 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fdf6006bbd0 0x7fdf6006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:43.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.314+0000 7fdf85c00640 1 --2- 172.21.15.90:0/1198340658 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fdf6006bbd0 0x7fdf6006e090 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fdf74006fd0 tx=0x7fdf74008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:43.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.317+0000 7fdf7d600640 1 -- 172.21.15.90:0/1198340658 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdf78014030 con 0x7fdf8015bc50 2024-03-24T01:25:43.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.515+0000 7fdf88600640 1 -- 172.21.15.90:0/1198340658 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdf8006c970 con 0x7fdf8015bc50 2024-03-24T01:25:43.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.516+0000 7fdf7d600640 1 -- 172.21.15.90:0/1198340658 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdf7805a2d0 con 0x7fdf8015bc50 2024-03-24T01:25:43.517 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-24T01:25:43.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.519+0000 7fdf88600640 1 -- 172.21.15.90:0/1198340658 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fdf6006bbd0 msgr2=0x7fdf6006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:43.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.519+0000 7fdf88600640 1 --2- 172.21.15.90:0/1198340658 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fdf6006bbd0 0x7fdf6006e090 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fdf74006fd0 tx=0x7fdf74008040 comp rx=0 tx=0).stop 2024-03-24T01:25:43.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.519+0000 7fdf88600640 1 -- 172.21.15.90:0/1198340658 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf8015bc50 msgr2=0x7fdf800b3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:43.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.519+0000 7fdf88600640 1 --2- 172.21.15.90:0/1198340658 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf8015bc50 0x7fdf800b3fc0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fdf7800cd50 tx=0x7fdf7800a650 comp rx=0 tx=0).stop 2024-03-24T01:25:43.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.519+0000 7fdf88600640 1 -- 172.21.15.90:0/1198340658 shutdown_connections 2024-03-24T01:25:43.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.519+0000 7fdf88600640 1 --2- 172.21.15.90:0/1198340658 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fdf6006bbd0 0x7fdf6006e090 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:43.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.519+0000 7fdf88600640 1 --2- 172.21.15.90:0/1198340658 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf8015bc50 0x7fdf800b3fc0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:43.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.519+0000 7fdf88600640 1 --2- 172.21.15.90:0/1198340658 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdf800aeeb0 0x7fdf800b3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:43.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.519+0000 7fdf88600640 1 --2- 172.21.15.90:0/1198340658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf8006b600 0x7fdf800b0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:43.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.519+0000 7fdf88600640 1 -- 172.21.15.90:0/1198340658 >> 172.21.15.90:0/1198340658 conn(0x7fdf800b1770 msgr2=0x7fdf80072220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:43.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.519+0000 7fdf88600640 1 -- 172.21.15.90:0/1198340658 shutdown_connections 2024-03-24T01:25:43.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.519+0000 7fdf88600640 1 -- 172.21.15.90:0/1198340658 wait complete. 2024-03-24T01:25:43.538 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T01:25:43.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.753+0000 7f36ea800640 1 Processor -- start 2024-03-24T01:25:43.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.753+0000 7f36ea800640 1 -- start start 2024-03-24T01:25:43.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.754+0000 7f36ea800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f36e40aeeb0 0x7f36e40b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:43.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.754+0000 7f36ea800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36e415bc50 0x7f36e406b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:43.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.754+0000 7f36ea800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e406dac0 con 0x7f36e406b600 2024-03-24T01:25:43.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.754+0000 7f36ea800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36e406ba70 con 0x7f36e40aeeb0 2024-03-24T01:25:43.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.754+0000 7f36ea800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36e40b4da0 con 0x7f36e415bc50 2024-03-24T01:25:43.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.755+0000 7f36daa00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36e415bc50 0x7f36e406b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:43.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.755+0000 7f36daa00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36e415bc50 0x7f36e406b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.90:49248/0 (socket says 172.21.15.90:49248) 2024-03-24T01:25:43.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.755+0000 7f36daa00640 1 -- 172.21.15.90:0/2942354671 learned_addr learned my addr 172.21.15.90:0/2942354671 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:43.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.755+0000 7f36e3400640 1 --2- 172.21.15.90:0/2942354671 >> v2:172.21.15.120:3300/0 conn(0x7f36e40aeeb0 0x7f36e40b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:43.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.756+0000 7f36daa00640 1 -- 172.21.15.90:0/2942354671 >> v2:172.21.15.120:3300/0 conn(0x7f36e40aeeb0 msgr2=0x7f36e40b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:43.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.756+0000 7f36daa00640 1 --2- 172.21.15.90:0/2942354671 >> v2:172.21.15.120:3300/0 conn(0x7f36e40aeeb0 0x7f36e40b44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:43.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.756+0000 7f36daa00640 1 -- 172.21.15.90:0/2942354671 >> v1:172.21.15.120:6789/0 conn(0x7f36e406b600 legacy=0x7f36e415e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:43.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.756+0000 7f36daa00640 1 -- 172.21.15.90:0/2942354671 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e406dda0 con 0x7f36e415bc50 2024-03-24T01:25:43.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.757+0000 7f36daa00640 1 --2- 172.21.15.90:0/2942354671 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36e415bc50 0x7f36e406b0c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f36cc009870 tx=0x7f36cc02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e72034c2ca5286ef server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:43.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.757+0000 7f36e2a00640 1 -- 172.21.15.90:0/2942354671 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc0317c0 con 0x7f36e415bc50 2024-03-24T01:25:43.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.757+0000 7f36e2a00640 1 -- 172.21.15.90:0/2942354671 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36cc037070 con 0x7f36e415bc50 2024-03-24T01:25:43.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.757+0000 7f36e2a00640 1 -- 172.21.15.90:0/2942354671 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc031b80 con 0x7f36e415bc50 2024-03-24T01:25:43.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.758+0000 7f36ea800640 1 -- 172.21.15.90:0/2942354671 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36e415bc50 msgr2=0x7f36e406b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:43.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.758+0000 7f36ea800640 1 --2- 172.21.15.90:0/2942354671 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36e415bc50 0x7f36e406b0c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f36cc009870 tx=0x7f36cc02b860 comp rx=0 tx=0).stop 2024-03-24T01:25:43.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.758+0000 7f36ea800640 1 -- 172.21.15.90:0/2942354671 shutdown_connections 2024-03-24T01:25:43.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.758+0000 7f36ea800640 1 --2- 172.21.15.90:0/2942354671 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36e415bc50 0x7f36e406b0c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:43.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.758+0000 7f36ea800640 1 --2- 172.21.15.90:0/2942354671 >> v2:172.21.15.120:3300/0 conn(0x7f36e40aeeb0 0x7f36e40b44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:43.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.758+0000 7f36ea800640 1 -- 172.21.15.90:0/2942354671 >> 172.21.15.90:0/2942354671 conn(0x7f36e40b1770 msgr2=0x7f36e40b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:43.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.758+0000 7f36ea800640 1 -- 172.21.15.90:0/2942354671 shutdown_connections 2024-03-24T01:25:43.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.758+0000 7f36ea800640 1 -- 172.21.15.90:0/2942354671 wait complete. 2024-03-24T01:25:43.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.760+0000 7f36ea800640 1 Processor -- start 2024-03-24T01:25:43.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.760+0000 7f36ea800640 1 -- start start 2024-03-24T01:25:43.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.761+0000 7f36ea800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36e406b600 0x7f36e40ade20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:43.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.761+0000 7f36ea800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36e40aeeb0 0x7f36e414d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:43.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.761+0000 7f36ea800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36e415bc50 0x7f36e40af410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:43.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.761+0000 7f36ea800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e40af290 con 0x7f36e415bc50 2024-03-24T01:25:43.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.761+0000 7f36ea800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e406e360 con 0x7f36e40aeeb0 2024-03-24T01:25:43.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.761+0000 7f36ea800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36e40b4a00 con 0x7f36e406b600 2024-03-24T01:25:43.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.762+0000 7f36e3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36e415bc50 0x7f36e40af410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:43.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.762+0000 7f36e3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36e415bc50 0x7f36e40af410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38818/0 (socket says 172.21.15.90:38818) 2024-03-24T01:25:43.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.762+0000 7f36e3e00640 1 -- 172.21.15.90:0/3992035012 learned_addr learned my addr 172.21.15.90:0/3992035012 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:43.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.762+0000 7f36e3400640 1 --2- 172.21.15.90:0/3992035012 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36e406b600 0x7f36e40ade20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:43.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.762+0000 7f36daa00640 1 --2- 172.21.15.90:0/3992035012 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36e40aeeb0 0x7f36e414d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:43.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.762+0000 7f36e3e00640 1 -- 172.21.15.90:0/3992035012 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36e406b600 msgr2=0x7f36e40ade20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:43.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.762+0000 7f36e3e00640 1 --2- 172.21.15.90:0/3992035012 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36e406b600 0x7f36e40ade20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:43.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.762+0000 7f36e3e00640 1 -- 172.21.15.90:0/3992035012 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36e40aeeb0 msgr2=0x7f36e414d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:43.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.762+0000 7f36e3e00640 1 --2- 172.21.15.90:0/3992035012 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36e40aeeb0 0x7f36e414d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:43.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.762+0000 7f36e3e00640 1 -- 172.21.15.90:0/3992035012 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e414e510 con 0x7f36e415bc50 2024-03-24T01:25:43.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.762+0000 7f36daa00640 1 --2- 172.21.15.90:0/3992035012 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36e40aeeb0 0x7f36e414d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:43.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.763+0000 7f36e3e00640 1 --2- 172.21.15.90:0/3992035012 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36e415bc50 0x7f36e40af410 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f36dc00c3c0 tx=0x7f36dc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:43.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.763+0000 7f36dbe00640 1 -- 172.21.15.90:0/3992035012 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36dc017070 con 0x7f36e415bc50 2024-03-24T01:25:43.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.763+0000 7f36dbe00640 1 -- 172.21.15.90:0/3992035012 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36dc0052d0 con 0x7f36e415bc50 2024-03-24T01:25:43.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.763+0000 7f36dbe00640 1 -- 172.21.15.90:0/3992035012 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36dc003810 con 0x7f36e415bc50 2024-03-24T01:25:43.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.763+0000 7f36ea800640 1 -- 172.21.15.90:0/3992035012 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36e414d6f0 con 0x7f36e415bc50 2024-03-24T01:25:43.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.763+0000 7f36ea800640 1 -- 172.21.15.90:0/3992035012 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36e414dc70 con 0x7f36e415bc50 2024-03-24T01:25:43.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.764+0000 7f36ea800640 1 -- 172.21.15.90:0/3992035012 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36e404f400 con 0x7f36e415bc50 2024-03-24T01:25:43.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.765+0000 7f36dbe00640 1 -- 172.21.15.90:0/3992035012 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f36dc01a070 con 0x7f36e415bc50 2024-03-24T01:25:43.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.766+0000 7f36dbe00640 1 --2- 172.21.15.90:0/3992035012 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f36c006bbd0 0x7f36c006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:43.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.766+0000 7f36dbe00640 1 -- 172.21.15.90:0/3992035012 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f36dc08a140 con 0x7f36e415bc50 2024-03-24T01:25:43.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.767+0000 7f36e3400640 1 --2- 172.21.15.90:0/3992035012 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f36c006bbd0 0x7f36c006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:43.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.768+0000 7f36e3400640 1 --2- 172.21.15.90:0/3992035012 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f36c006bbd0 0x7f36c006e090 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f36e406ecb0 tx=0x7f36d400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:43.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.771+0000 7f36dbe00640 1 -- 172.21.15.90:0/3992035012 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f36dc0588a0 con 0x7f36e415bc50 2024-03-24T01:25:43.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.969+0000 7f36ea800640 1 -- 172.21.15.90:0/3992035012 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f36e41597b0 con 0x7f36e415bc50 2024-03-24T01:25:43.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.970+0000 7f36dbe00640 1 -- 172.21.15.90:0/3992035012 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f36dc0581a0 con 0x7f36e415bc50 2024-03-24T01:25:43.970 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:25:43.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.972+0000 7f36ea800640 1 -- 172.21.15.90:0/3992035012 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f36c006bbd0 msgr2=0x7f36c006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:43.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.972+0000 7f36ea800640 1 --2- 172.21.15.90:0/3992035012 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f36c006bbd0 0x7f36c006e090 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f36e406ecb0 tx=0x7f36d400b040 comp rx=0 tx=0).stop 2024-03-24T01:25:43.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.972+0000 7f36ea800640 1 -- 172.21.15.90:0/3992035012 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36e415bc50 msgr2=0x7f36e40af410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:43.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.972+0000 7f36ea800640 1 --2- 172.21.15.90:0/3992035012 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36e415bc50 0x7f36e40af410 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f36dc00c3c0 tx=0x7f36dc00c880 comp rx=0 tx=0).stop 2024-03-24T01:25:43.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.972+0000 7f36ea800640 1 -- 172.21.15.90:0/3992035012 shutdown_connections 2024-03-24T01:25:43.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.972+0000 7f36ea800640 1 --2- 172.21.15.90:0/3992035012 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f36c006bbd0 0x7f36c006e090 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:43.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.972+0000 7f36ea800640 1 --2- 172.21.15.90:0/3992035012 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36e415bc50 0x7f36e40af410 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:43.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.972+0000 7f36ea800640 1 --2- 172.21.15.90:0/3992035012 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36e40aeeb0 0x7f36e414d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:43.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.972+0000 7f36ea800640 1 --2- 172.21.15.90:0/3992035012 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36e406b600 0x7f36e40ade20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:43.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.972+0000 7f36ea800640 1 -- 172.21.15.90:0/3992035012 >> 172.21.15.90:0/3992035012 conn(0x7f36e40b1770 msgr2=0x7f36e4071990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:43.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.973+0000 7f36ea800640 1 -- 172.21.15.90:0/3992035012 shutdown_connections 2024-03-24T01:25:43.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:43.973+0000 7f36ea800640 1 -- 172.21.15.90:0/3992035012 wait complete. 2024-03-24T01:25:43.991 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:25:44.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.200+0000 7fa6aae00640 1 Processor -- start 2024-03-24T01:25:44.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.200+0000 7fa6aae00640 1 -- start start 2024-03-24T01:25:44.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.201+0000 7fa6aae00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa6a409a370 0x7fa6a40cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:44.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.201+0000 7fa6aae00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6a40b0fc0 con 0x7fa6a40b2760 2024-03-24T01:25:44.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.201+0000 7fa6aae00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6a415f5e0 con 0x7fa6a415f200 2024-03-24T01:25:44.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.201+0000 7fa6aae00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6a40b1270 con 0x7fa6a409a370 2024-03-24T01:25:44.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.202+0000 7fa6a8e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa6a40b2760 0x7fa6a40cfaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56088/0 (socket says 172.21.15.90:56088) 2024-03-24T01:25:44.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.202+0000 7fa6a8e00640 1 -- 172.21.15.90:0/522556147 learned_addr learned my addr 172.21.15.90:0/522556147 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:44.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.202+0000 7fa6a3400640 1 --2- 172.21.15.90:0/522556147 >> v2:172.21.15.120:3300/0 conn(0x7fa6a409a370 0x7fa6a40cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:44.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.202+0000 7fa6a2a00640 1 -- 172.21.15.90:0/522556147 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4232751036 0 0) 0x7fa6a40b0fc0 con 0x7fa6a40b2760 2024-03-24T01:25:44.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.202+0000 7fa6a2a00640 1 -- 172.21.15.90:0/522556147 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa688003520 con 0x7fa6a40b2760 2024-03-24T01:25:44.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.202+0000 7fa6a2a00640 1 -- 172.21.15.90:0/522556147 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3414326643 0 0) 0x7fa688003520 con 0x7fa6a40b2760 2024-03-24T01:25:44.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.202+0000 7fa6a2a00640 1 -- 172.21.15.90:0/522556147 >> v2:172.21.15.120:3300/0 conn(0x7fa6a409a370 msgr2=0x7fa6a40cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:44.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.202+0000 7fa6a2a00640 1 --2- 172.21.15.90:0/522556147 >> v2:172.21.15.120:3300/0 conn(0x7fa6a409a370 0x7fa6a40cf5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:44.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.202+0000 7fa6a2a00640 1 -- 172.21.15.90:0/522556147 >> v1:172.21.15.120:6789/0 conn(0x7fa6a415f200 legacy=0x7fa6a4099c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:44.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.202+0000 7fa6a2a00640 1 -- 172.21.15.90:0/522556147 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a40cd630 con 0x7fa6a40b2760 2024-03-24T01:25:44.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.202+0000 7fa6a2a00640 1 -- 172.21.15.90:0/522556147 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7fa69c0019f0 con 0x7fa6a40b2760 2024-03-24T01:25:44.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.202+0000 7fa6a2a00640 1 -- 172.21.15.90:0/522556147 >> v1:172.21.15.90:6789/0 conn(0x7fa6a40b2760 legacy=0x7fa6a40cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:44.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.202+0000 7fa6a2a00640 1 --2- 172.21.15.90:0/522556147 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa688003a00 0x7fa688003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:44.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.202+0000 7fa6a2a00640 1 --2- 172.21.15.90:0/522556147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa688004800 0x7fa688004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:44.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.203+0000 7fa6a2a00640 1 --2- 172.21.15.90:0/522556147 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6880052e0 0x7fa68800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:44.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.203+0000 7fa6a2a00640 1 -- 172.21.15.90:0/522556147 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a40b1270 con 0x7fa688003a00 2024-03-24T01:25:44.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.203+0000 7fa6a2a00640 1 -- 172.21.15.90:0/522556147 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa68800c1e0 con 0x7fa688004800 2024-03-24T01:25:44.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.203+0000 7fa6a2a00640 1 -- 172.21.15.90:0/522556147 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa68800c3e0 con 0x7fa6880052e0 2024-03-24T01:25:44.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.203+0000 7fa6a3e00640 1 --2- 172.21.15.90:0/522556147 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa688003a00 0x7fa688003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:44.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.203+0000 7fa6a3e00640 1 -- 172.21.15.90:0/522556147 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6880052e0 msgr2=0x7fa68800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:44.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.203+0000 7fa6a3e00640 1 --2- 172.21.15.90:0/522556147 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6880052e0 0x7fa68800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:44.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.203+0000 7fa6a3e00640 1 -- 172.21.15.90:0/522556147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa688004800 msgr2=0x7fa688004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:44.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.203+0000 7fa6a3e00640 1 --2- 172.21.15.90:0/522556147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa688004800 0x7fa688004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:44.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.203+0000 7fa6a3e00640 1 -- 172.21.15.90:0/522556147 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa688003520 con 0x7fa688003a00 2024-03-24T01:25:44.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.203+0000 7fa6a8e00640 1 -- 172.21.15.90:0/522556147 reap_dead start 2024-03-24T01:25:44.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.204+0000 7fa6a3e00640 1 --2- 172.21.15.90:0/522556147 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa688003a00 0x7fa688003e20 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fa694001750 tx=0x7fa688005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc16152bab3a70d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:44.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.205+0000 7fa6a2a00640 1 -- 172.21.15.90:0/522556147 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa694001370 con 0x7fa688003a00 2024-03-24T01:25:44.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.205+0000 7fa6a2a00640 1 -- 172.21.15.90:0/522556147 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6940052d0 con 0x7fa688003a00 2024-03-24T01:25:44.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.205+0000 7fa6a2a00640 1 -- 172.21.15.90:0/522556147 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa694003830 con 0x7fa688003a00 2024-03-24T01:25:44.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.205+0000 7fa6aae00640 1 -- 172.21.15.90:0/522556147 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa688003a00 msgr2=0x7fa688003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:44.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.205+0000 7fa6aae00640 1 --2- 172.21.15.90:0/522556147 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa688003a00 0x7fa688003e20 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fa694001750 tx=0x7fa688005830 comp rx=0 tx=0).stop 2024-03-24T01:25:44.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.205+0000 7fa6aae00640 1 -- 172.21.15.90:0/522556147 shutdown_connections 2024-03-24T01:25:44.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.205+0000 7fa6aae00640 1 --2- 172.21.15.90:0/522556147 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa688003a00 0x7fa688003e20 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:44.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.205+0000 7fa6aae00640 1 -- 172.21.15.90:0/522556147 >> 172.21.15.90:0/522556147 conn(0x7fa6a40b34d0 msgr2=0x7fa6a40b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:44.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.205+0000 7fa6aae00640 1 -- 172.21.15.90:0/522556147 shutdown_connections 2024-03-24T01:25:44.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.205+0000 7fa6aae00640 1 -- 172.21.15.90:0/522556147 wait complete. 2024-03-24T01:25:44.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.207+0000 7fa6aae00640 1 Processor -- start 2024-03-24T01:25:44.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.207+0000 7fa6aae00640 1 -- start start 2024-03-24T01:25:44.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.208+0000 7fa6aae00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa688003a00 0x7fa6a40c8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:44.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.208+0000 7fa6aae00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6a40c7d80 0x7fa6a4158200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:44.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.208+0000 7fa6aae00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6a415d7a0 0x7fa6a415dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:44.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.208+0000 7fa6aae00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a40cded0 con 0x7fa688003a00 2024-03-24T01:25:44.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.208+0000 7fa6aae00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a40cdbf0 con 0x7fa6a40c7d80 2024-03-24T01:25:44.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.208+0000 7fa6aae00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a40cd090 con 0x7fa6a415d7a0 2024-03-24T01:25:44.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.208+0000 7fa6a3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa688003a00 0x7fa6a40c8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:44.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.208+0000 7fa6a3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa688003a00 0x7fa6a40c8ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38844/0 (socket says 172.21.15.90:38844) 2024-03-24T01:25:44.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.208+0000 7fa6a3e00640 1 -- 172.21.15.90:0/1137506834 learned_addr learned my addr 172.21.15.90:0/1137506834 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:44.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.209+0000 7fa6a3400640 1 --2- 172.21.15.90:0/1137506834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6a40c7d80 0x7fa6a4158200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:44.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.209+0000 7fa6a8e00640 1 --2- 172.21.15.90:0/1137506834 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6a415d7a0 0x7fa6a415dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:44.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.209+0000 7fa6a3e00640 1 -- 172.21.15.90:0/1137506834 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6a415d7a0 msgr2=0x7fa6a415dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:44.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.209+0000 7fa6a3e00640 1 --2- 172.21.15.90:0/1137506834 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6a415d7a0 0x7fa6a415dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:44.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.209+0000 7fa6a3e00640 1 -- 172.21.15.90:0/1137506834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6a40c7d80 msgr2=0x7fa6a4158200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:44.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.209+0000 7fa6a3e00640 1 --2- 172.21.15.90:0/1137506834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6a40c7d80 0x7fa6a4158200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:44.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.209+0000 7fa6a3e00640 1 -- 172.21.15.90:0/1137506834 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a40c9930 con 0x7fa688003a00 2024-03-24T01:25:44.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.209+0000 7fa6a3400640 1 --2- 172.21.15.90:0/1137506834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6a40c7d80 0x7fa6a4158200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:44.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.209+0000 7fa6a8e00640 1 --2- 172.21.15.90:0/1137506834 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6a415d7a0 0x7fa6a415dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:44.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.209+0000 7fa6a3e00640 1 --2- 172.21.15.90:0/1137506834 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa688003a00 0x7fa6a40c8ab0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa694001870 tx=0x7fa694001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:44.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.209+0000 7fa69be00640 1 -- 172.21.15.90:0/1137506834 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa694041020 con 0x7fa688003a00 2024-03-24T01:25:44.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.209+0000 7fa69be00640 1 -- 172.21.15.90:0/1137506834 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa694005480 con 0x7fa688003a00 2024-03-24T01:25:44.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.209+0000 7fa69be00640 1 -- 172.21.15.90:0/1137506834 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa694001370 con 0x7fa688003a00 2024-03-24T01:25:44.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.209+0000 7fa6aae00640 1 -- 172.21.15.90:0/1137506834 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6a4158740 con 0x7fa688003a00 2024-03-24T01:25:44.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.210+0000 7fa6aae00640 1 -- 172.21.15.90:0/1137506834 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6a40c9370 con 0x7fa688003a00 2024-03-24T01:25:44.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.211+0000 7fa6aae00640 1 -- 172.21.15.90:0/1137506834 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6a4068770 con 0x7fa688003a00 2024-03-24T01:25:44.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.212+0000 7fa69be00640 1 -- 172.21.15.90:0/1137506834 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa694044020 con 0x7fa688003a00 2024-03-24T01:25:44.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.212+0000 7fa69be00640 1 --2- 172.21.15.90:0/1137506834 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa68406bc00 0x7fa68406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:44.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.213+0000 7fa69be00640 1 -- 172.21.15.90:0/1137506834 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7fa6940ac7e0 con 0x7fa688003a00 2024-03-24T01:25:44.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.213+0000 7fa6a3400640 1 --2- 172.21.15.90:0/1137506834 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa68406bc00 0x7fa68406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:44.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.214+0000 7fa6a3400640 1 --2- 172.21.15.90:0/1137506834 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa68406bc00 0x7fa68406e0c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa6a40d0ad0 tx=0x7fa690008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:44.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.217+0000 7fa69be00640 1 -- 172.21.15.90:0/1137506834 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa69404b030 con 0x7fa688003a00 2024-03-24T01:25:44.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.438+0000 7fa6aae00640 1 -- 172.21.15.90:0/1137506834 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6a40cbea0 con 0x7fa688003a00 2024-03-24T01:25:44.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.438+0000 7fa69be00640 1 -- 172.21.15.90:0/1137506834 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4967 (secure 0 0 0) 0x7fa6940303c0 con 0x7fa688003a00 2024-03-24T01:25:44.439 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:25:44.439 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/1898972903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":1898972903},{"type":"v1","addr":"172.21.15.120:6837","nonce":1898972903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6834/775759501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":775759501},{"type":"v1","addr":"172.21.15.90:6834","nonce":775759501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/43630575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":43630575},{"type":"v1","addr":"172.21.15.90:6835","nonce":43630575}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T01:24:10.173532+0000","modified":"2024-03-24T01:24:36.608477+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6836/963928022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":963928022},{"type":"v1","addr":"172.21.15.120:6836","nonce":963928022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4257,"qdb_cluster":[4257]},"id":1}]} 2024-03-24T01:25:44.439 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 8 2024-03-24T01:25:44.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.441+0000 7fa6aae00640 1 -- 172.21.15.90:0/1137506834 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa68406bc00 msgr2=0x7fa68406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:44.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.441+0000 7fa6aae00640 1 --2- 172.21.15.90:0/1137506834 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa68406bc00 0x7fa68406e0c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa6a40d0ad0 tx=0x7fa690008040 comp rx=0 tx=0).stop 2024-03-24T01:25:44.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.441+0000 7fa6aae00640 1 -- 172.21.15.90:0/1137506834 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa688003a00 msgr2=0x7fa6a40c8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:44.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.441+0000 7fa6aae00640 1 --2- 172.21.15.90:0/1137506834 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa688003a00 0x7fa6a40c8ab0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa694001870 tx=0x7fa694001120 comp rx=0 tx=0).stop 2024-03-24T01:25:44.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.441+0000 7fa6aae00640 1 -- 172.21.15.90:0/1137506834 shutdown_connections 2024-03-24T01:25:44.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.441+0000 7fa6aae00640 1 --2- 172.21.15.90:0/1137506834 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa68406bc00 0x7fa68406e0c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:44.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.441+0000 7fa6aae00640 1 --2- 172.21.15.90:0/1137506834 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa6a415d7a0 0x7fa6a415dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:44.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.441+0000 7fa6aae00640 1 --2- 172.21.15.90:0/1137506834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa6a40c7d80 0x7fa6a4158200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:44.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.441+0000 7fa6aae00640 1 --2- 172.21.15.90:0/1137506834 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa688003a00 0x7fa6a40c8ab0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:44.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.441+0000 7fa6aae00640 1 -- 172.21.15.90:0/1137506834 >> 172.21.15.90:0/1137506834 conn(0x7fa6a40b34d0 msgr2=0x7fa6a40cc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:44.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.441+0000 7fa6aae00640 1 -- 172.21.15.90:0/1137506834 shutdown_connections 2024-03-24T01:25:44.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.441+0000 7fa6aae00640 1 -- 172.21.15.90:0/1137506834 wait complete. 2024-03-24T01:25:44.459 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:25:44.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.662+0000 7faa77200640 1 Processor -- start 2024-03-24T01:25:44.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.662+0000 7faa77200640 1 -- start start 2024-03-24T01:25:44.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.663+0000 7faa77200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa700cdac0 0x7faa7006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:44.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.663+0000 7faa77200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa7015bf70 con 0x7faa7006b600 2024-03-24T01:25:44.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.663+0000 7faa77200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa7015b360 con 0x7faa70155030 2024-03-24T01:25:44.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.663+0000 7faa77200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa7006ba70 con 0x7faa700cdac0 2024-03-24T01:25:44.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.664+0000 7faa75200640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7faa7006b600 0x7faa700ae530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56102/0 (socket says 172.21.15.90:56102) 2024-03-24T01:25:44.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.664+0000 7faa75200640 1 -- 172.21.15.90:0/170299963 learned_addr learned my addr 172.21.15.90:0/170299963 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:44.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.664+0000 7faa6fe00640 1 --1- 172.21.15.90:0/170299963 >> v1:172.21.15.120:6789/0 conn(0x7faa70155030 0x7faa7015efc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.90:36152/0 (socket says 172.21.15.90:36152) 2024-03-24T01:25:44.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.664+0000 7faa67400640 1 --2- 172.21.15.90:0/170299963 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa700cdac0 0x7faa7006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:44.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.664+0000 7faa6f400640 1 -- 172.21.15.90:0/170299963 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181227569 0 0) 0x7faa7015bf70 con 0x7faa7006b600 2024-03-24T01:25:44.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.664+0000 7faa6f400640 1 -- 172.21.15.90:0/170299963 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa54003520 con 0x7faa7006b600 2024-03-24T01:25:44.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.664+0000 7faa6f400640 1 -- 172.21.15.90:0/170299963 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4099678841 0 0) 0x7faa7015b360 con 0x7faa70155030 2024-03-24T01:25:44.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.664+0000 7faa6f400640 1 -- 172.21.15.90:0/170299963 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa7015bf70 con 0x7faa70155030 2024-03-24T01:25:44.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.664+0000 7faa6f400640 1 -- 172.21.15.90:0/170299963 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2542639649 0 0) 0x7faa54003520 con 0x7faa7006b600 2024-03-24T01:25:44.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.664+0000 7faa6f400640 1 -- 172.21.15.90:0/170299963 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa700cdac0 msgr2=0x7faa7006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:44.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.664+0000 7faa6f400640 1 --2- 172.21.15.90:0/170299963 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa700cdac0 0x7faa7006b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:44.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.664+0000 7faa6f400640 1 -- 172.21.15.90:0/170299963 >> v1:172.21.15.120:6789/0 conn(0x7faa70155030 legacy=0x7faa7015efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:44.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.664+0000 7faa6f400640 1 -- 172.21.15.90:0/170299963 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa7015b640 con 0x7faa7006b600 2024-03-24T01:25:44.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.664+0000 7faa6f400640 1 -- 172.21.15.90:0/170299963 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7faa680019f0 con 0x7faa7006b600 2024-03-24T01:25:44.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.664+0000 7faa6f400640 1 -- 172.21.15.90:0/170299963 >> v1:172.21.15.90:6789/0 conn(0x7faa7006b600 legacy=0x7faa700ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:44.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.665+0000 7faa6f400640 1 --2- 172.21.15.90:0/170299963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa54003b70 0x7faa54003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:44.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.665+0000 7faa6f400640 1 --2- 172.21.15.90:0/170299963 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa54004800 0x7faa54004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:44.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.665+0000 7faa6f400640 1 --2- 172.21.15.90:0/170299963 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa700cdac0 0x7faa540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:44.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.665+0000 7faa6f400640 1 -- 172.21.15.90:0/170299963 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7006ba70 con 0x7faa54004800 2024-03-24T01:25:44.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.665+0000 7faa6f400640 1 -- 172.21.15.90:0/170299963 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa5400dc80 con 0x7faa54003b70 2024-03-24T01:25:44.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.665+0000 7faa6f400640 1 -- 172.21.15.90:0/170299963 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa5400de50 con 0x7faa700cdac0 2024-03-24T01:25:44.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.665+0000 7faa67400640 1 --2- 172.21.15.90:0/170299963 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa54004800 0x7faa54004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:44.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.665+0000 7faa75200640 1 --2- 172.21.15.90:0/170299963 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa700cdac0 0x7faa540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:44.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.665+0000 7faa6fe00640 1 --2- 172.21.15.90:0/170299963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa54003b70 0x7faa54003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:44.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.665+0000 7faa67400640 1 -- 172.21.15.90:0/170299963 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa700cdac0 msgr2=0x7faa540051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:44.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.665+0000 7faa67400640 1 --2- 172.21.15.90:0/170299963 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa700cdac0 0x7faa540051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:44.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.665+0000 7faa67400640 1 -- 172.21.15.90:0/170299963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa54003b70 msgr2=0x7faa54003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:44.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.665+0000 7faa67400640 1 --2- 172.21.15.90:0/170299963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa54003b70 0x7faa54003f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:44.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.665+0000 7faa67400640 1 -- 172.21.15.90:0/170299963 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa54003520 con 0x7faa54004800 2024-03-24T01:25:44.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.665+0000 7faa75200640 1 --2- 172.21.15.90:0/170299963 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa700cdac0 0x7faa540051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:44.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.666+0000 7faa67400640 1 --2- 172.21.15.90:0/170299963 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa54004800 0x7faa54004c80 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7faa58009ac0 tx=0x7faa54005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6b9f0e69808f276 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:44.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.666+0000 7faa6f400640 1 -- 172.21.15.90:0/170299963 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa58039070 con 0x7faa54004800 2024-03-24T01:25:44.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.667+0000 7faa6f400640 1 -- 172.21.15.90:0/170299963 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa5802c660 con 0x7faa54004800 2024-03-24T01:25:44.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.667+0000 7faa6f400640 1 -- 172.21.15.90:0/170299963 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa58009690 con 0x7faa54004800 2024-03-24T01:25:44.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.667+0000 7faa77200640 1 -- 172.21.15.90:0/170299963 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa54004800 msgr2=0x7faa54004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:44.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.667+0000 7faa77200640 1 --2- 172.21.15.90:0/170299963 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa54004800 0x7faa54004c80 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7faa58009ac0 tx=0x7faa54005830 comp rx=0 tx=0).stop 2024-03-24T01:25:44.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.667+0000 7faa75200640 1 -- 172.21.15.90:0/170299963 reap_dead start 2024-03-24T01:25:44.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.667+0000 7faa77200640 1 -- 172.21.15.90:0/170299963 shutdown_connections 2024-03-24T01:25:44.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.667+0000 7faa77200640 1 -- 172.21.15.90:0/170299963 >> 172.21.15.90:0/170299963 conn(0x7faa700b1770 msgr2=0x7faa700b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:44.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.667+0000 7faa77200640 1 -- 172.21.15.90:0/170299963 shutdown_connections 2024-03-24T01:25:44.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.667+0000 7faa77200640 1 -- 172.21.15.90:0/170299963 wait complete. 2024-03-24T01:25:44.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.669+0000 7faa77200640 1 Processor -- start 2024-03-24T01:25:44.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.669+0000 7faa77200640 1 -- start start 2024-03-24T01:25:44.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.670+0000 7faa77200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa54004800 0x7faa70143360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:44.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.670+0000 7faa77200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa701e3480 0x7faa701e3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:44.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.670+0000 7faa77200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa70157290 0x7faa70157710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:44.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.670+0000 7faa77200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa70155410 con 0x7faa70157290 2024-03-24T01:25:44.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.670+0000 7faa77200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7015bc00 con 0x7faa701e3480 2024-03-24T01:25:44.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.670+0000 7faa77200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa7015af70 con 0x7faa54004800 2024-03-24T01:25:44.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.670+0000 7faa75200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa70157290 0x7faa70157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:44.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.670+0000 7faa75200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa70157290 0x7faa70157710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38860/0 (socket says 172.21.15.90:38860) 2024-03-24T01:25:44.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.670+0000 7faa75200640 1 -- 172.21.15.90:0/3986511377 learned_addr learned my addr 172.21.15.90:0/3986511377 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:44.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.671+0000 7faa6fe00640 1 --2- 172.21.15.90:0/3986511377 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa54004800 0x7faa70143360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:44.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.671+0000 7faa67400640 1 --2- 172.21.15.90:0/3986511377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa701e3480 0x7faa701e3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:44.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.671+0000 7faa75200640 1 -- 172.21.15.90:0/3986511377 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa54004800 msgr2=0x7faa70143360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:44.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.671+0000 7faa75200640 1 --2- 172.21.15.90:0/3986511377 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa54004800 0x7faa70143360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:44.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.671+0000 7faa75200640 1 -- 172.21.15.90:0/3986511377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa701e3480 msgr2=0x7faa701e3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:44.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.671+0000 7faa75200640 1 --2- 172.21.15.90:0/3986511377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa701e3480 0x7faa701e3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:44.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.671+0000 7faa75200640 1 -- 172.21.15.90:0/3986511377 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa701438a0 con 0x7faa70157290 2024-03-24T01:25:44.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.671+0000 7faa75200640 1 --2- 172.21.15.90:0/3986511377 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa70157290 0x7faa70157710 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7faa7006f120 tx=0x7faa6800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:44.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.671+0000 7faa6cc00640 1 -- 172.21.15.90:0/3986511377 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6800de70 con 0x7faa70157290 2024-03-24T01:25:44.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.671+0000 7faa6cc00640 1 -- 172.21.15.90:0/3986511377 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa6800bb80 con 0x7faa70157290 2024-03-24T01:25:44.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.671+0000 7faa6cc00640 1 -- 172.21.15.90:0/3986511377 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa68007110 con 0x7faa70157290 2024-03-24T01:25:44.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.671+0000 7faa77200640 1 -- 172.21.15.90:0/3986511377 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa70143cc0 con 0x7faa70157290 2024-03-24T01:25:44.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.671+0000 7faa77200640 1 -- 172.21.15.90:0/3986511377 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa70144280 con 0x7faa70157290 2024-03-24T01:25:44.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.672+0000 7faa77200640 1 -- 172.21.15.90:0/3986511377 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa70068060 con 0x7faa70157290 2024-03-24T01:25:44.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.673+0000 7faa6cc00640 1 -- 172.21.15.90:0/3986511377 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7faa6801b020 con 0x7faa70157290 2024-03-24T01:25:44.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.675+0000 7faa6cc00640 1 --2- 172.21.15.90:0/3986511377 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7faa4c06bc00 0x7faa4c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:44.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.675+0000 7faa6cc00640 1 -- 172.21.15.90:0/3986511377 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7faa6808b000 con 0x7faa70157290 2024-03-24T01:25:44.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.675+0000 7faa6fe00640 1 --2- 172.21.15.90:0/3986511377 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7faa4c06bc00 0x7faa4c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:44.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.676+0000 7faa6fe00640 1 --2- 172.21.15.90:0/3986511377 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7faa4c06bc00 0x7faa4c06e0c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7faa60007fd0 tx=0x7faa60009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:44.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.679+0000 7faa6cc00640 1 -- 172.21.15.90:0/3986511377 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7faa68059820 con 0x7faa70157290 2024-03-24T01:25:44.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.877+0000 7faa77200640 1 -- 172.21.15.90:0/3986511377 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa70157fc0 con 0x7faa70157290 2024-03-24T01:25:44.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.880+0000 7faa6cc00640 1 -- 172.21.15.90:0/3986511377 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18925 (secure 0 0 0) 0x7faa68059120 con 0x7faa70157290 2024-03-24T01:25:44.881 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:25:44.881 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:24:53.236401+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:24:08.643966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T01:24:09.189904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T01:24:13.027323+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:25:44.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.883+0000 7faa77200640 1 -- 172.21.15.90:0/3986511377 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7faa4c06bc00 msgr2=0x7faa4c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:44.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.883+0000 7faa77200640 1 --2- 172.21.15.90:0/3986511377 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7faa4c06bc00 0x7faa4c06e0c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7faa60007fd0 tx=0x7faa60009040 comp rx=0 tx=0).stop 2024-03-24T01:25:44.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.883+0000 7faa77200640 1 -- 172.21.15.90:0/3986511377 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa70157290 msgr2=0x7faa70157710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:44.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.883+0000 7faa77200640 1 --2- 172.21.15.90:0/3986511377 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa70157290 0x7faa70157710 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7faa7006f120 tx=0x7faa6800b4a0 comp rx=0 tx=0).stop 2024-03-24T01:25:44.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.883+0000 7faa77200640 1 -- 172.21.15.90:0/3986511377 shutdown_connections 2024-03-24T01:25:44.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.883+0000 7faa77200640 1 --2- 172.21.15.90:0/3986511377 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7faa4c06bc00 0x7faa4c06e0c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:44.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.883+0000 7faa77200640 1 --2- 172.21.15.90:0/3986511377 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa70157290 0x7faa70157710 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:44.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.883+0000 7faa77200640 1 --2- 172.21.15.90:0/3986511377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa701e3480 0x7faa701e3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:44.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.883+0000 7faa77200640 1 --2- 172.21.15.90:0/3986511377 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa54004800 0x7faa70143360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:44.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.883+0000 7faa77200640 1 -- 172.21.15.90:0/3986511377 >> 172.21.15.90:0/3986511377 conn(0x7faa700b1770 msgr2=0x7faa70072220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:44.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.883+0000 7faa77200640 1 -- 172.21.15.90:0/3986511377 shutdown_connections 2024-03-24T01:25:44.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:44.883+0000 7faa77200640 1 -- 172.21.15.90:0/3986511377 wait complete. 2024-03-24T01:25:44.903 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-24T01:25:44.903 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:25:45.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.111+0000 7f8e18a00640 1 Processor -- start 2024-03-24T01:25:45.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.111+0000 7f8e18a00640 1 -- start start 2024-03-24T01:25:45.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.113+0000 7f8e18a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8e140cc150 0x7f8e140ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:45.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.113+0000 7f8e18a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8e141603e0 0x7f8e1406b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:45.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.113+0000 7f8e18a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e140cd490 con 0x7f8e14145430 2024-03-24T01:25:45.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.113+0000 7f8e18a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e1415f900 con 0x7f8e141603e0 2024-03-24T01:25:45.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.113+0000 7f8e18a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e1415fbe0 con 0x7f8e140cc150 2024-03-24T01:25:45.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.113+0000 7f8e09400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8e141603e0 0x7f8e1406b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:45.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.113+0000 7f8e09400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8e141603e0 0x7f8e1406b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38876/0 (socket says 172.21.15.90:38876) 2024-03-24T01:25:45.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.113+0000 7f8e09400640 1 -- 172.21.15.90:0/4030666196 learned_addr learned my addr 172.21.15.90:0/4030666196 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:45.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.114+0000 7f8e11e00640 1 --2- 172.21.15.90:0/4030666196 >> v2:172.21.15.120:3300/0 conn(0x7f8e140cc150 0x7f8e140ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:45.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.114+0000 7f8e09400640 1 -- 172.21.15.90:0/4030666196 >> v2:172.21.15.120:3300/0 conn(0x7f8e140cc150 msgr2=0x7f8e140ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:45.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.114+0000 7f8e09400640 1 --2- 172.21.15.90:0/4030666196 >> v2:172.21.15.120:3300/0 conn(0x7f8e140cc150 0x7f8e140ccb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:45.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.114+0000 7f8e09400640 1 -- 172.21.15.90:0/4030666196 >> v1:172.21.15.90:6789/0 conn(0x7f8e14145430 legacy=0x7f8e14071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:45.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.114+0000 7f8e09400640 1 -- 172.21.15.90:0/4030666196 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e1415fec0 con 0x7f8e141603e0 2024-03-24T01:25:45.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.115+0000 7f8e09400640 1 --2- 172.21.15.90:0/4030666196 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e141603e0 0x7f8e1406b0c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8e1406bd10 tx=0x7f8dfc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed5235a3ab39f154 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:45.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.115+0000 7f8e11400640 1 -- 172.21.15.90:0/4030666196 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dfc034070 con 0x7f8e141603e0 2024-03-24T01:25:45.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.115+0000 7f8e11400640 1 -- 172.21.15.90:0/4030666196 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dfc030860 con 0x7f8e141603e0 2024-03-24T01:25:45.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.115+0000 7f8e11400640 1 -- 172.21.15.90:0/4030666196 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dfc030c30 con 0x7f8e141603e0 2024-03-24T01:25:45.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.116+0000 7f8e18a00640 1 -- 172.21.15.90:0/4030666196 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e141603e0 msgr2=0x7f8e1406b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:45.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.116+0000 7f8e18a00640 1 --2- 172.21.15.90:0/4030666196 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e141603e0 0x7f8e1406b0c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8e1406bd10 tx=0x7f8dfc02b9a0 comp rx=0 tx=0).stop 2024-03-24T01:25:45.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.116+0000 7f8e18a00640 1 -- 172.21.15.90:0/4030666196 shutdown_connections 2024-03-24T01:25:45.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.116+0000 7f8e18a00640 1 --2- 172.21.15.90:0/4030666196 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e141603e0 0x7f8e1406b0c0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:45.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.116+0000 7f8e18a00640 1 --2- 172.21.15.90:0/4030666196 >> v2:172.21.15.120:3300/0 conn(0x7f8e140cc150 0x7f8e140ccb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:45.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.116+0000 7f8e18a00640 1 -- 172.21.15.90:0/4030666196 >> 172.21.15.90:0/4030666196 conn(0x7f8e140b1770 msgr2=0x7f8e140b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:45.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.116+0000 7f8e18a00640 1 -- 172.21.15.90:0/4030666196 shutdown_connections 2024-03-24T01:25:45.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.116+0000 7f8e18a00640 1 -- 172.21.15.90:0/4030666196 wait complete. 2024-03-24T01:25:45.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.118+0000 7f8e18a00640 1 Processor -- start 2024-03-24T01:25:45.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.118+0000 7f8e18a00640 1 -- start start 2024-03-24T01:25:45.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.119+0000 7f8e18a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e140cc150 0x7f8e141f4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:45.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.119+0000 7f8e18a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e14145430 0x7f8e140ac170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:45.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.119+0000 7f8e18a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e141603e0 0x7f8e14142fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:45.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.119+0000 7f8e18a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1406b780 con 0x7f8e141603e0 2024-03-24T01:25:45.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.119+0000 7f8e18a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1406b600 con 0x7f8e14145430 2024-03-24T01:25:45.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.119+0000 7f8e18a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1406b900 con 0x7f8e140cc150 2024-03-24T01:25:45.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.119+0000 7f8e12800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e141603e0 0x7f8e14142fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:45.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.119+0000 7f8e12800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e141603e0 0x7f8e14142fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38882/0 (socket says 172.21.15.90:38882) 2024-03-24T01:25:45.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.119+0000 7f8e12800640 1 -- 172.21.15.90:0/3512335244 learned_addr learned my addr 172.21.15.90:0/3512335244 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:45.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.119+0000 7f8e11e00640 1 --2- 172.21.15.90:0/3512335244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e140cc150 0x7f8e141f4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:45.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.119+0000 7f8e09400640 1 --2- 172.21.15.90:0/3512335244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e14145430 0x7f8e140ac170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:45.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.119+0000 7f8e12800640 1 -- 172.21.15.90:0/3512335244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e140cc150 msgr2=0x7f8e141f4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:45.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.119+0000 7f8e12800640 1 --2- 172.21.15.90:0/3512335244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e140cc150 0x7f8e141f4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:45.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.119+0000 7f8e12800640 1 -- 172.21.15.90:0/3512335244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e14145430 msgr2=0x7f8e140ac170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:45.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.119+0000 7f8e12800640 1 --2- 172.21.15.90:0/3512335244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e14145430 0x7f8e140ac170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:45.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.120+0000 7f8e12800640 1 -- 172.21.15.90:0/3512335244 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e141f4fc0 con 0x7f8e141603e0 2024-03-24T01:25:45.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.120+0000 7f8e12800640 1 --2- 172.21.15.90:0/3512335244 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e141603e0 0x7f8e14142fa0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c00c7a0 tx=0x7f8e0c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:45.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.120+0000 7f8e0aa00640 1 -- 172.21.15.90:0/3512335244 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0c003b70 con 0x7f8e141603e0 2024-03-24T01:25:45.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.120+0000 7f8e0aa00640 1 -- 172.21.15.90:0/3512335244 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e0c003d30 con 0x7f8e141603e0 2024-03-24T01:25:45.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.120+0000 7f8e0aa00640 1 -- 172.21.15.90:0/3512335244 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0c00aa90 con 0x7f8e141603e0 2024-03-24T01:25:45.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.120+0000 7f8e18a00640 1 -- 172.21.15.90:0/3512335244 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e140aca50 con 0x7f8e141603e0 2024-03-24T01:25:45.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.120+0000 7f8e18a00640 1 -- 172.21.15.90:0/3512335244 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e140ad010 con 0x7f8e141603e0 2024-03-24T01:25:45.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.121+0000 7f8e18a00640 1 -- 172.21.15.90:0/3512335244 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e14068060 con 0x7f8e141603e0 2024-03-24T01:25:45.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.122+0000 7f8e0aa00640 1 -- 172.21.15.90:0/3512335244 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8e0c00ac30 con 0x7f8e141603e0 2024-03-24T01:25:45.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.123+0000 7f8e0aa00640 1 --2- 172.21.15.90:0/3512335244 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8df006bbd0 0x7f8df006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:45.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.123+0000 7f8e0aa00640 1 -- 172.21.15.90:0/3512335244 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7f8e0c08ade0 con 0x7f8e141603e0 2024-03-24T01:25:45.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.124+0000 7f8e11e00640 1 --2- 172.21.15.90:0/3512335244 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8df006bbd0 0x7f8df006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:45.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.125+0000 7f8e11e00640 1 --2- 172.21.15.90:0/3512335244 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8df006bbd0 0x7f8df006e090 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8e04006fd0 tx=0x7f8e04008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:45.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.128+0000 7f8e0aa00640 1 -- 172.21.15.90:0/3512335244 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8e0c0594a0 con 0x7f8e141603e0 2024-03-24T01:25:45.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.366+0000 7f8e18a00640 1 -- 172.21.15.90:0/3512335244 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e1415e8d0 con 0x7f8e141603e0 2024-03-24T01:25:45.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.367+0000 7f8e0aa00640 1 -- 172.21.15.90:0/3512335244 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4967 (secure 0 0 0) 0x7f8e0c058da0 con 0x7f8e141603e0 2024-03-24T01:25:45.367 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:25:45.367 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/1898972903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":1898972903},{"type":"v1","addr":"172.21.15.120:6837","nonce":1898972903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6834/775759501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":775759501},{"type":"v1","addr":"172.21.15.90:6834","nonce":775759501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/43630575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":43630575},{"type":"v1","addr":"172.21.15.90:6835","nonce":43630575}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T01:24:10.173532+0000","modified":"2024-03-24T01:24:36.608477+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6836/963928022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":963928022},{"type":"v1","addr":"172.21.15.120:6836","nonce":963928022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4257,"qdb_cluster":[4257]},"id":1}]} 2024-03-24T01:25:45.368 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 8 2024-03-24T01:25:45.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.369+0000 7f8e18a00640 1 -- 172.21.15.90:0/3512335244 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8df006bbd0 msgr2=0x7f8df006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:45.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.369+0000 7f8e18a00640 1 --2- 172.21.15.90:0/3512335244 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8df006bbd0 0x7f8df006e090 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8e04006fd0 tx=0x7f8e04008040 comp rx=0 tx=0).stop 2024-03-24T01:25:45.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.369+0000 7f8e18a00640 1 -- 172.21.15.90:0/3512335244 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e141603e0 msgr2=0x7f8e14142fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:45.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.369+0000 7f8e18a00640 1 --2- 172.21.15.90:0/3512335244 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e141603e0 0x7f8e14142fa0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c00c7a0 tx=0x7f8e0c00cc60 comp rx=0 tx=0).stop 2024-03-24T01:25:45.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.370+0000 7f8e18a00640 1 -- 172.21.15.90:0/3512335244 shutdown_connections 2024-03-24T01:25:45.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.370+0000 7f8e18a00640 1 --2- 172.21.15.90:0/3512335244 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8df006bbd0 0x7f8df006e090 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:45.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.370+0000 7f8e18a00640 1 --2- 172.21.15.90:0/3512335244 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e141603e0 0x7f8e14142fa0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:45.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.370+0000 7f8e18a00640 1 --2- 172.21.15.90:0/3512335244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e14145430 0x7f8e140ac170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:45.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.370+0000 7f8e18a00640 1 --2- 172.21.15.90:0/3512335244 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8e140cc150 0x7f8e141f4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:45.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.370+0000 7f8e18a00640 1 -- 172.21.15.90:0/3512335244 >> 172.21.15.90:0/3512335244 conn(0x7f8e140b1770 msgr2=0x7f8e14144b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:45.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.370+0000 7f8e18a00640 1 -- 172.21.15.90:0/3512335244 shutdown_connections 2024-03-24T01:25:45.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.370+0000 7f8e18a00640 1 -- 172.21.15.90:0/3512335244 wait complete. 2024-03-24T01:25:45.388 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:25:45.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.593+0000 7fc3cda00640 1 Processor -- start 2024-03-24T01:25:45.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.593+0000 7fc3cda00640 1 -- start start 2024-03-24T01:25:45.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.595+0000 7fc3cda00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc3c80afd50 0x7fc3c80674a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:45.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.595+0000 7fc3cda00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c80b5070 con 0x7fc3c815e230 2024-03-24T01:25:45.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.595+0000 7fc3cda00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c80b5350 con 0x7fc3c806c7d0 2024-03-24T01:25:45.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.595+0000 7fc3cda00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3c80b6370 con 0x7fc3c80afd50 2024-03-24T01:25:45.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.595+0000 7fc3c7400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc3c815e230 0x7fc3c80ad1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56122/0 (socket says 172.21.15.90:56122) 2024-03-24T01:25:45.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.595+0000 7fc3c7400640 1 -- 172.21.15.90:0/3793879681 learned_addr learned my addr 172.21.15.90:0/3793879681 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:45.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.596+0000 7fc3c6a00640 1 --2- 172.21.15.90:0/3793879681 >> v2:172.21.15.120:3300/0 conn(0x7fc3c80afd50 0x7fc3c80674a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:45.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.596+0000 7fc3c6000640 1 -- 172.21.15.90:0/3793879681 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3903866581 0 0) 0x7fc3c80b5070 con 0x7fc3c815e230 2024-03-24T01:25:45.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.596+0000 7fc3c6000640 1 -- 172.21.15.90:0/3793879681 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3ac003520 con 0x7fc3c815e230 2024-03-24T01:25:45.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.596+0000 7fc3c6000640 1 -- 172.21.15.90:0/3793879681 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3309556860 0 0) 0x7fc3ac003520 con 0x7fc3c815e230 2024-03-24T01:25:45.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.596+0000 7fc3c6000640 1 -- 172.21.15.90:0/3793879681 >> v2:172.21.15.120:3300/0 conn(0x7fc3c80afd50 msgr2=0x7fc3c80674a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:45.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.596+0000 7fc3c6000640 1 --2- 172.21.15.90:0/3793879681 >> v2:172.21.15.120:3300/0 conn(0x7fc3c80afd50 0x7fc3c80674a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:45.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.596+0000 7fc3c6000640 1 -- 172.21.15.90:0/3793879681 >> v1:172.21.15.120:6789/0 conn(0x7fc3c806c7d0 legacy=0x7fc3c80b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:45.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.596+0000 7fc3c6000640 1 -- 172.21.15.90:0/3793879681 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c80b6090 con 0x7fc3c815e230 2024-03-24T01:25:45.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.596+0000 7fc3c6000640 1 -- 172.21.15.90:0/3793879681 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7fc3c00019f0 con 0x7fc3c815e230 2024-03-24T01:25:45.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.596+0000 7fc3c6000640 1 -- 172.21.15.90:0/3793879681 >> v1:172.21.15.90:6789/0 conn(0x7fc3c815e230 legacy=0x7fc3c80ad1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:45.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.596+0000 7fc3c6000640 1 --2- 172.21.15.90:0/3793879681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3ac0039b0 0x7fc3ac003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:45.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.596+0000 7fc3c6000640 1 --2- 172.21.15.90:0/3793879681 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3ac004800 0x7fc3ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:45.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.596+0000 7fc3c6000640 1 --2- 172.21.15.90:0/3793879681 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3ac0051c0 0x7fc3ac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:45.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.596+0000 7fc3c6000640 1 -- 172.21.15.90:0/3793879681 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ac00c0e0 con 0x7fc3ac004800 2024-03-24T01:25:45.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.596+0000 7fc3c6000640 1 -- 172.21.15.90:0/3793879681 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ac00c290 con 0x7fc3ac0039b0 2024-03-24T01:25:45.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.596+0000 7fc3c6000640 1 -- 172.21.15.90:0/3793879681 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ac00c490 con 0x7fc3ac0051c0 2024-03-24T01:25:45.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.597+0000 7fc3be000640 1 --2- 172.21.15.90:0/3793879681 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3ac004800 0x7fc3ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:45.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.597+0000 7fc3be000640 1 -- 172.21.15.90:0/3793879681 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3ac0051c0 msgr2=0x7fc3ac00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:45.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.597+0000 7fc3be000640 1 --2- 172.21.15.90:0/3793879681 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3ac0051c0 0x7fc3ac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:45.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.597+0000 7fc3be000640 1 -- 172.21.15.90:0/3793879681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3ac0039b0 msgr2=0x7fc3ac003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:45.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.597+0000 7fc3be000640 1 --2- 172.21.15.90:0/3793879681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3ac0039b0 0x7fc3ac003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:45.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.597+0000 7fc3be000640 1 -- 172.21.15.90:0/3793879681 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ac003520 con 0x7fc3ac004800 2024-03-24T01:25:45.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.597+0000 7fc3c7400640 1 -- 172.21.15.90:0/3793879681 reap_dead start 2024-03-24T01:25:45.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.598+0000 7fc3be000640 1 --2- 172.21.15.90:0/3793879681 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3ac004800 0x7fc3ac004c80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fc3b0001720 tx=0x7fc3ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9138f0ec1df767e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:45.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.598+0000 7fc3c6000640 1 -- 172.21.15.90:0/3793879681 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b0001370 con 0x7fc3ac004800 2024-03-24T01:25:45.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.598+0000 7fc3c6000640 1 -- 172.21.15.90:0/3793879681 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3b00052d0 con 0x7fc3ac004800 2024-03-24T01:25:45.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.598+0000 7fc3c6000640 1 -- 172.21.15.90:0/3793879681 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b0003830 con 0x7fc3ac004800 2024-03-24T01:25:45.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.599+0000 7fc3cda00640 1 -- 172.21.15.90:0/3793879681 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3ac004800 msgr2=0x7fc3ac004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:45.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.599+0000 7fc3cda00640 1 --2- 172.21.15.90:0/3793879681 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3ac004800 0x7fc3ac004c80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fc3b0001720 tx=0x7fc3ac005830 comp rx=0 tx=0).stop 2024-03-24T01:25:45.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.599+0000 7fc3cda00640 1 -- 172.21.15.90:0/3793879681 shutdown_connections 2024-03-24T01:25:45.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.599+0000 7fc3cda00640 1 --2- 172.21.15.90:0/3793879681 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3ac004800 0x7fc3ac004c80 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:45.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.599+0000 7fc3cda00640 1 -- 172.21.15.90:0/3793879681 >> 172.21.15.90:0/3793879681 conn(0x7fc3c8099a70 msgr2=0x7fc3c80b5730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:45.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.599+0000 7fc3cda00640 1 -- 172.21.15.90:0/3793879681 shutdown_connections 2024-03-24T01:25:45.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.599+0000 7fc3cda00640 1 -- 172.21.15.90:0/3793879681 wait complete. 2024-03-24T01:25:45.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.601+0000 7fc3cda00640 1 Processor -- start 2024-03-24T01:25:45.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.601+0000 7fc3cda00640 1 -- start start 2024-03-24T01:25:45.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.602+0000 7fc3cda00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3ac004800 0x7fc3c80ad490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:45.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.602+0000 7fc3cda00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3c80aeb10 0x7fc3c8154f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:45.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.602+0000 7fc3cda00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3c8155480 0x7fc3c813ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:45.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.602+0000 7fc3cda00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c80d0f70 con 0x7fc3ac004800 2024-03-24T01:25:45.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.602+0000 7fc3cda00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c80b5e40 con 0x7fc3c8155480 2024-03-24T01:25:45.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.602+0000 7fc3cda00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c80674a0 con 0x7fc3c80aeb10 2024-03-24T01:25:45.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.602+0000 7fc3c6a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3ac004800 0x7fc3c80ad490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:45.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.602+0000 7fc3c6a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3ac004800 0x7fc3c80ad490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38900/0 (socket says 172.21.15.90:38900) 2024-03-24T01:25:45.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.602+0000 7fc3c6a00640 1 -- 172.21.15.90:0/997246132 learned_addr learned my addr 172.21.15.90:0/997246132 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:45.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.602+0000 7fc3be000640 1 --2- 172.21.15.90:0/997246132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3c80aeb10 0x7fc3c8154f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:45.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.602+0000 7fc3c7400640 1 --2- 172.21.15.90:0/997246132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3c8155480 0x7fc3c813ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:45.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.602+0000 7fc3c6a00640 1 -- 172.21.15.90:0/997246132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3c80aeb10 msgr2=0x7fc3c8154f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:45.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.602+0000 7fc3c6a00640 1 --2- 172.21.15.90:0/997246132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3c80aeb10 0x7fc3c8154f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:45.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.602+0000 7fc3c6a00640 1 -- 172.21.15.90:0/997246132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3c8155480 msgr2=0x7fc3c813ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:45.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.602+0000 7fc3c6a00640 1 --2- 172.21.15.90:0/997246132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3c8155480 0x7fc3c813ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:45.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.602+0000 7fc3c6a00640 1 -- 172.21.15.90:0/997246132 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c80ada70 con 0x7fc3ac004800 2024-03-24T01:25:45.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.602+0000 7fc3c7400640 1 --2- 172.21.15.90:0/997246132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3c8155480 0x7fc3c813ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:45.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.603+0000 7fc3c6a00640 1 --2- 172.21.15.90:0/997246132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3ac004800 0x7fc3c80ad490 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fc3b800d560 tx=0x7fc3b800da20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:45.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.603+0000 7fc3bf400640 1 -- 172.21.15.90:0/997246132 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b80125f0 con 0x7fc3ac004800 2024-03-24T01:25:45.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.603+0000 7fc3bf400640 1 -- 172.21.15.90:0/997246132 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3b8019070 con 0x7fc3ac004800 2024-03-24T01:25:45.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.603+0000 7fc3cda00640 1 -- 172.21.15.90:0/997246132 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3c80add50 con 0x7fc3ac004800 2024-03-24T01:25:45.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.603+0000 7fc3bf400640 1 -- 172.21.15.90:0/997246132 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b80129d0 con 0x7fc3ac004800 2024-03-24T01:25:45.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.603+0000 7fc3cda00640 1 -- 172.21.15.90:0/997246132 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3c80ae310 con 0x7fc3ac004800 2024-03-24T01:25:45.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.604+0000 7fc3cda00640 1 -- 172.21.15.90:0/997246132 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3c8068060 con 0x7fc3ac004800 2024-03-24T01:25:45.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.605+0000 7fc3bf400640 1 -- 172.21.15.90:0/997246132 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc3b8012b70 con 0x7fc3ac004800 2024-03-24T01:25:45.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.606+0000 7fc3bf400640 1 --2- 172.21.15.90:0/997246132 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc3a406bbd0 0x7fc3a406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:45.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.606+0000 7fc3bf400640 1 -- 172.21.15.90:0/997246132 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7fc3b808b350 con 0x7fc3ac004800 2024-03-24T01:25:45.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.606+0000 7fc3be000640 1 --2- 172.21.15.90:0/997246132 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc3a406bbd0 0x7fc3a406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:45.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.607+0000 7fc3be000640 1 --2- 172.21.15.90:0/997246132 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc3a406bbd0 0x7fc3a406e090 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc3b0033000 tx=0x7fc3b0009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:45.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.610+0000 7fc3bf400640 1 -- 172.21.15.90:0/997246132 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc3b80598f0 con 0x7fc3ac004800 2024-03-24T01:25:45.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.851+0000 7fc3cda00640 1 -- 172.21.15.90:0/997246132 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3c806ed30 con 0x7fc3ac004800 2024-03-24T01:25:45.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.852+0000 7fc3bf400640 1 -- 172.21.15.90:0/997246132 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4967 (secure 0 0 0) 0x7fc3b80591f0 con 0x7fc3ac004800 2024-03-24T01:25:45.852 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:25:45.853 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/1898972903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":1898972903},{"type":"v1","addr":"172.21.15.120:6837","nonce":1898972903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6834/775759501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":775759501},{"type":"v1","addr":"172.21.15.90:6834","nonce":775759501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/43630575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":43630575},{"type":"v1","addr":"172.21.15.90:6835","nonce":43630575}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T01:24:10.173532+0000","modified":"2024-03-24T01:24:36.608477+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6836/963928022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":963928022},{"type":"v1","addr":"172.21.15.120:6836","nonce":963928022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4257,"qdb_cluster":[4257]},"id":1}]} 2024-03-24T01:25:45.853 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 8 2024-03-24T01:25:45.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.855+0000 7fc3cda00640 1 -- 172.21.15.90:0/997246132 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc3a406bbd0 msgr2=0x7fc3a406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:45.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.855+0000 7fc3cda00640 1 --2- 172.21.15.90:0/997246132 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc3a406bbd0 0x7fc3a406e090 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc3b0033000 tx=0x7fc3b0009030 comp rx=0 tx=0).stop 2024-03-24T01:25:45.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.855+0000 7fc3cda00640 1 -- 172.21.15.90:0/997246132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3ac004800 msgr2=0x7fc3c80ad490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:45.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.855+0000 7fc3cda00640 1 --2- 172.21.15.90:0/997246132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3ac004800 0x7fc3c80ad490 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fc3b800d560 tx=0x7fc3b800da20 comp rx=0 tx=0).stop 2024-03-24T01:25:45.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.855+0000 7fc3cda00640 1 -- 172.21.15.90:0/997246132 shutdown_connections 2024-03-24T01:25:45.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.856+0000 7fc3cda00640 1 --2- 172.21.15.90:0/997246132 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc3a406bbd0 0x7fc3a406e090 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:45.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.856+0000 7fc3cda00640 1 --2- 172.21.15.90:0/997246132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3c8155480 0x7fc3c813ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:45.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.856+0000 7fc3cda00640 1 --2- 172.21.15.90:0/997246132 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3c80aeb10 0x7fc3c8154f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:45.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.856+0000 7fc3cda00640 1 --2- 172.21.15.90:0/997246132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3ac004800 0x7fc3c80ad490 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:45.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.856+0000 7fc3cda00640 1 -- 172.21.15.90:0/997246132 >> 172.21.15.90:0/997246132 conn(0x7fc3c8099a70 msgr2=0x7fc3c80cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:45.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.856+0000 7fc3cda00640 1 -- 172.21.15.90:0/997246132 shutdown_connections 2024-03-24T01:25:45.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:45.856+0000 7fc3cda00640 1 -- 172.21.15.90:0/997246132 wait complete. 2024-03-24T01:25:45.875 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:25:46.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.075+0000 7fbf30a00640 1 Processor -- start 2024-03-24T01:25:46.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.075+0000 7fbf30a00640 1 -- start start 2024-03-24T01:25:46.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.076+0000 7fbf30a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbf2c1454e0 0x7fbf2c0cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:46.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.076+0000 7fbf30a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fbf2c06f000 0x7fbf2c06f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:46.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.076+0000 7fbf30a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf2c0cc660 con 0x7fbf2c0cdc50 2024-03-24T01:25:46.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.076+0000 7fbf30a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf2c0cc430 con 0x7fbf2c06f000 2024-03-24T01:25:46.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.076+0000 7fbf30a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf2c0cc920 con 0x7fbf2c1454e0 2024-03-24T01:25:46.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.077+0000 7fbf23e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fbf2c06f000 0x7fbf2c06f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:46.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.077+0000 7fbf23e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fbf2c06f000 0x7fbf2c06f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38908/0 (socket says 172.21.15.90:38908) 2024-03-24T01:25:46.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.077+0000 7fbf23e00640 1 -- 172.21.15.90:0/966967686 learned_addr learned my addr 172.21.15.90:0/966967686 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:46.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.077+0000 7fbf29e00640 1 --2- 172.21.15.90:0/966967686 >> v2:172.21.15.120:3300/0 conn(0x7fbf2c1454e0 0x7fbf2c0cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:46.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.078+0000 7fbf23e00640 1 -- 172.21.15.90:0/966967686 >> v2:172.21.15.120:3300/0 conn(0x7fbf2c1454e0 msgr2=0x7fbf2c0cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:46.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.078+0000 7fbf23e00640 1 --2- 172.21.15.90:0/966967686 >> v2:172.21.15.120:3300/0 conn(0x7fbf2c1454e0 0x7fbf2c0cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:46.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.078+0000 7fbf23e00640 1 -- 172.21.15.90:0/966967686 >> v1:172.21.15.90:6789/0 conn(0x7fbf2c0cdc50 legacy=0x7fbf2c06d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:46.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.078+0000 7fbf23e00640 1 -- 172.21.15.90:0/966967686 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf2c0ccc00 con 0x7fbf2c06f000 2024-03-24T01:25:46.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.078+0000 7fbf29400640 1 -- 172.21.15.90:0/966967686 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4097493557 0 0) 0x7fbf2c0cc660 con 0x7fbf2c0cdc50 2024-03-24T01:25:46.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.079+0000 7fbf23e00640 1 --2- 172.21.15.90:0/966967686 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf2c06f000 0x7fbf2c06f440 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c0b4df0 tx=0x7fbf1402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7f5a97902bf4593 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:46.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.079+0000 7fbf29400640 1 -- 172.21.15.90:0/966967686 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf14034070 con 0x7fbf2c06f000 2024-03-24T01:25:46.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.079+0000 7fbf29400640 1 -- 172.21.15.90:0/966967686 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf14030860 con 0x7fbf2c06f000 2024-03-24T01:25:46.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.079+0000 7fbf29400640 1 -- 172.21.15.90:0/966967686 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf14030c30 con 0x7fbf2c06f000 2024-03-24T01:25:46.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.080+0000 7fbf30a00640 1 -- 172.21.15.90:0/966967686 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf2c06f000 msgr2=0x7fbf2c06f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:46.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.080+0000 7fbf30a00640 1 --2- 172.21.15.90:0/966967686 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf2c06f000 0x7fbf2c06f440 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c0b4df0 tx=0x7fbf1402b9a0 comp rx=0 tx=0).stop 2024-03-24T01:25:46.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.080+0000 7fbf30a00640 1 -- 172.21.15.90:0/966967686 shutdown_connections 2024-03-24T01:25:46.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.080+0000 7fbf30a00640 1 --2- 172.21.15.90:0/966967686 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf2c06f000 0x7fbf2c06f440 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:46.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.080+0000 7fbf30a00640 1 --2- 172.21.15.90:0/966967686 >> v2:172.21.15.120:3300/0 conn(0x7fbf2c1454e0 0x7fbf2c0cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:46.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.080+0000 7fbf30a00640 1 -- 172.21.15.90:0/966967686 >> 172.21.15.90:0/966967686 conn(0x7fbf2c06b2a0 msgr2=0x7fbf2c06b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:46.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.080+0000 7fbf30a00640 1 -- 172.21.15.90:0/966967686 shutdown_connections 2024-03-24T01:25:46.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.080+0000 7fbf30a00640 1 -- 172.21.15.90:0/966967686 wait complete. 2024-03-24T01:25:46.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.082+0000 7fbf30a00640 1 Processor -- start 2024-03-24T01:25:46.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.082+0000 7fbf30a00640 1 -- start start 2024-03-24T01:25:46.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.083+0000 7fbf30a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf2c06f000 0x7fbf2c0b2b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:46.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.083+0000 7fbf30a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf2c0cdc50 0x7fbf2c0b0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:46.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.083+0000 7fbf30a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf2c1454e0 0x7fbf2c155700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:46.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.083+0000 7fbf30a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2c0ce0c0 con 0x7fbf2c0cdc50 2024-03-24T01:25:46.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.083+0000 7fbf30a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2c0cd1c0 con 0x7fbf2c06f000 2024-03-24T01:25:46.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.083+0000 7fbf30a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2c0ce240 con 0x7fbf2c1454e0 2024-03-24T01:25:46.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.084+0000 7fbf23e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf2c0cdc50 0x7fbf2c0b0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:46.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.084+0000 7fbf23e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf2c0cdc50 0x7fbf2c0b0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38920/0 (socket says 172.21.15.90:38920) 2024-03-24T01:25:46.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.084+0000 7fbf23e00640 1 -- 172.21.15.90:0/3051585178 learned_addr learned my addr 172.21.15.90:0/3051585178 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:46.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.084+0000 7fbf23e00640 1 -- 172.21.15.90:0/3051585178 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf2c1454e0 msgr2=0x7fbf2c155700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:46.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.084+0000 7fbf23e00640 1 --2- 172.21.15.90:0/3051585178 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf2c1454e0 0x7fbf2c155700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:46.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.084+0000 7fbf23e00640 1 -- 172.21.15.90:0/3051585178 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf2c06f000 msgr2=0x7fbf2c0b2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:46.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.084+0000 7fbf23e00640 1 --2- 172.21.15.90:0/3051585178 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf2c06f000 0x7fbf2c0b2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:46.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.084+0000 7fbf23e00640 1 -- 172.21.15.90:0/3051585178 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf2c15a1f0 con 0x7fbf2c0cdc50 2024-03-24T01:25:46.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.084+0000 7fbf23e00640 1 --2- 172.21.15.90:0/3051585178 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf2c0cdc50 0x7fbf2c0b0d20 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fbf140090b0 tx=0x7fbf140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:46.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.084+0000 7fbf22000640 1 -- 172.21.15.90:0/3051585178 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf14034040 con 0x7fbf2c0cdc50 2024-03-24T01:25:46.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.085+0000 7fbf22000640 1 -- 172.21.15.90:0/3051585178 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf14030660 con 0x7fbf2c0cdc50 2024-03-24T01:25:46.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.085+0000 7fbf30a00640 1 -- 172.21.15.90:0/3051585178 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf2c15a4d0 con 0x7fbf2c0cdc50 2024-03-24T01:25:46.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.085+0000 7fbf22000640 1 -- 172.21.15.90:0/3051585178 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf14039d20 con 0x7fbf2c0cdc50 2024-03-24T01:25:46.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.085+0000 7fbf30a00640 1 -- 172.21.15.90:0/3051585178 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf2c15aa90 con 0x7fbf2c0cdc50 2024-03-24T01:25:46.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.086+0000 7fbf30a00640 1 -- 172.21.15.90:0/3051585178 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf2c073570 con 0x7fbf2c0cdc50 2024-03-24T01:25:46.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.087+0000 7fbf22000640 1 -- 172.21.15.90:0/3051585178 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fbf14037070 con 0x7fbf2c0cdc50 2024-03-24T01:25:46.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.088+0000 7fbf22000640 1 --2- 172.21.15.90:0/3051585178 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbf0806bbd0 0x7fbf0806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:46.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.088+0000 7fbf22000640 1 -- 172.21.15.90:0/3051585178 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7fbf140acbf0 con 0x7fbf2c0cdc50 2024-03-24T01:25:46.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.088+0000 7fbf29e00640 1 --2- 172.21.15.90:0/3051585178 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbf0806bbd0 0x7fbf0806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:46.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.089+0000 7fbf29e00640 1 --2- 172.21.15.90:0/3051585178 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbf0806bbd0 0x7fbf0806e090 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c006a90 tx=0x7fbf1c006a20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:46.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.093+0000 7fbf22000640 1 -- 172.21.15.90:0/3051585178 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fbf1407b3d0 con 0x7fbf2c0cdc50 2024-03-24T01:25:46.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.275+0000 7fbf30a00640 1 -- 172.21.15.90:0/3051585178 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf2c04f4e0 con 0x7fbf2c0cdc50 2024-03-24T01:25:46.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.279+0000 7fbf22000640 1 -- 172.21.15.90:0/3051585178 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18925 (secure 0 0 0) 0x7fbf1407acd0 con 0x7fbf2c0cdc50 2024-03-24T01:25:46.279 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:25:46.279 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:24:53.236401+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:24:08.643966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T01:24:09.189904+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T01:24:13.027323+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:25:46.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.281+0000 7fbf30a00640 1 -- 172.21.15.90:0/3051585178 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbf0806bbd0 msgr2=0x7fbf0806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:46.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.281+0000 7fbf30a00640 1 --2- 172.21.15.90:0/3051585178 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbf0806bbd0 0x7fbf0806e090 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c006a90 tx=0x7fbf1c006a20 comp rx=0 tx=0).stop 2024-03-24T01:25:46.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.281+0000 7fbf30a00640 1 -- 172.21.15.90:0/3051585178 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf2c0cdc50 msgr2=0x7fbf2c0b0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:46.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.281+0000 7fbf30a00640 1 --2- 172.21.15.90:0/3051585178 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf2c0cdc50 0x7fbf2c0b0d20 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fbf140090b0 tx=0x7fbf140046c0 comp rx=0 tx=0).stop 2024-03-24T01:25:46.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.281+0000 7fbf30a00640 1 -- 172.21.15.90:0/3051585178 shutdown_connections 2024-03-24T01:25:46.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.281+0000 7fbf30a00640 1 --2- 172.21.15.90:0/3051585178 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbf0806bbd0 0x7fbf0806e090 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:46.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.281+0000 7fbf30a00640 1 --2- 172.21.15.90:0/3051585178 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf2c0cdc50 0x7fbf2c0b0d20 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:46.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.281+0000 7fbf30a00640 1 --2- 172.21.15.90:0/3051585178 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf2c1454e0 0x7fbf2c155700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:46.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.281+0000 7fbf30a00640 1 --2- 172.21.15.90:0/3051585178 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf2c06f000 0x7fbf2c0b2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:46.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.281+0000 7fbf30a00640 1 -- 172.21.15.90:0/3051585178 >> 172.21.15.90:0/3051585178 conn(0x7fbf2c06b2a0 msgr2=0x7fbf2c143c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:46.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.282+0000 7fbf30a00640 1 -- 172.21.15.90:0/3051585178 shutdown_connections 2024-03-24T01:25:46.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.282+0000 7fbf30a00640 1 -- 172.21.15.90:0/3051585178 wait complete. 2024-03-24T01:25:46.300 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T01:25:46.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.498+0000 7fb335800640 1 Processor -- start 2024-03-24T01:25:46.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.498+0000 7fb335800640 1 -- start start 2024-03-24T01:25:46.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.499+0000 7fb335800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb330071ea0 0x7fb330166730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:46.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.499+0000 7fb335800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb33006e0f0 con 0x7fb330144680 2024-03-24T01:25:46.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.499+0000 7fb335800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb33006e3d0 con 0x7fb330145190 2024-03-24T01:25:46.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.499+0000 7fb335800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb330144a60 con 0x7fb330071ea0 2024-03-24T01:25:46.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.500+0000 7fb32fe00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fb330144680 0x7fb33006d7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56136/0 (socket says 172.21.15.90:56136) 2024-03-24T01:25:46.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.500+0000 7fb32fe00640 1 -- 172.21.15.90:0/1132339913 learned_addr learned my addr 172.21.15.90:0/1132339913 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:46.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.500+0000 7fb334e00640 1 --2- 172.21.15.90:0/1132339913 >> v2:172.21.15.90:3300/0 conn(0x7fb330071ea0 0x7fb330166730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:46.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.500+0000 7fb32ea00640 1 -- 172.21.15.90:0/1132339913 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3525946814 0 0) 0x7fb33006e0f0 con 0x7fb330144680 2024-03-24T01:25:46.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.500+0000 7fb32ea00640 1 -- 172.21.15.90:0/1132339913 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb314003520 con 0x7fb330144680 2024-03-24T01:25:46.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.500+0000 7fb334e00640 1 -- 172.21.15.90:0/1132339913 >> v1:172.21.15.120:6789/0 conn(0x7fb330145190 legacy=0x7fb330071790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:46.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.500+0000 7fb334e00640 1 -- 172.21.15.90:0/1132339913 >> v1:172.21.15.90:6789/0 conn(0x7fb330144680 legacy=0x7fb33006d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:46.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.500+0000 7fb334e00640 1 -- 172.21.15.90:0/1132339913 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb33006e6b0 con 0x7fb330071ea0 2024-03-24T01:25:46.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.502+0000 7fb334e00640 1 --2- 172.21.15.90:0/1132339913 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb330071ea0 0x7fb330166730 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fb328009d90 tx=0x7fb32802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cde523d14593b119 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:46.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.502+0000 7fb32ea00640 1 -- 172.21.15.90:0/1132339913 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb328039070 con 0x7fb330071ea0 2024-03-24T01:25:46.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.502+0000 7fb32ea00640 1 -- 172.21.15.90:0/1132339913 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb32802cd00 con 0x7fb330071ea0 2024-03-24T01:25:46.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.502+0000 7fb32ea00640 1 -- 172.21.15.90:0/1132339913 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb328034640 con 0x7fb330071ea0 2024-03-24T01:25:46.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.503+0000 7fb335800640 1 -- 172.21.15.90:0/1132339913 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb330071ea0 msgr2=0x7fb330166730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:46.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.503+0000 7fb335800640 1 --2- 172.21.15.90:0/1132339913 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb330071ea0 0x7fb330166730 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fb328009d90 tx=0x7fb32802bf50 comp rx=0 tx=0).stop 2024-03-24T01:25:46.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.503+0000 7fb335800640 1 -- 172.21.15.90:0/1132339913 shutdown_connections 2024-03-24T01:25:46.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.503+0000 7fb335800640 1 --2- 172.21.15.90:0/1132339913 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb330071ea0 0x7fb330166730 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:46.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.503+0000 7fb335800640 1 -- 172.21.15.90:0/1132339913 >> 172.21.15.90:0/1132339913 conn(0x7fb33006b0c0 msgr2=0x7fb33006b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:46.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.503+0000 7fb335800640 1 -- 172.21.15.90:0/1132339913 shutdown_connections 2024-03-24T01:25:46.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.503+0000 7fb335800640 1 -- 172.21.15.90:0/1132339913 wait complete. 2024-03-24T01:25:46.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.504+0000 7fb335800640 1 Processor -- start 2024-03-24T01:25:46.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.505+0000 7fb335800640 1 -- start start 2024-03-24T01:25:46.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.505+0000 7fb335800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb330071ea0 0x7fb33015af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:46.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.505+0000 7fb335800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb330144680 0x7fb3300ab9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:46.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.505+0000 7fb335800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb330145190 0x7fb3300b3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:46.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.505+0000 7fb335800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3300cc190 con 0x7fb330144680 2024-03-24T01:25:46.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.505+0000 7fb335800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3300cdb20 con 0x7fb330145190 2024-03-24T01:25:46.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.505+0000 7fb335800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3300cc470 con 0x7fb330071ea0 2024-03-24T01:25:46.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.506+0000 7fb32f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb330144680 0x7fb3300ab9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:46.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.506+0000 7fb32f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb330144680 0x7fb3300ab9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38936/0 (socket says 172.21.15.90:38936) 2024-03-24T01:25:46.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.506+0000 7fb32f400640 1 -- 172.21.15.90:0/2712334040 learned_addr learned my addr 172.21.15.90:0/2712334040 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:46.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.506+0000 7fb334e00640 1 --2- 172.21.15.90:0/2712334040 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb330145190 0x7fb3300b3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:46.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.506+0000 7fb32fe00640 1 --2- 172.21.15.90:0/2712334040 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb330071ea0 0x7fb33015af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:46.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.506+0000 7fb32f400640 1 -- 172.21.15.90:0/2712334040 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb330071ea0 msgr2=0x7fb33015af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:46.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.506+0000 7fb32f400640 1 --2- 172.21.15.90:0/2712334040 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb330071ea0 0x7fb33015af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:46.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.506+0000 7fb32f400640 1 -- 172.21.15.90:0/2712334040 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb330145190 msgr2=0x7fb3300b3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:46.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.506+0000 7fb32f400640 1 --2- 172.21.15.90:0/2712334040 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb330145190 0x7fb3300b3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:46.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.506+0000 7fb32f400640 1 -- 172.21.15.90:0/2712334040 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3300abf10 con 0x7fb330144680 2024-03-24T01:25:46.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.506+0000 7fb32fe00640 1 --2- 172.21.15.90:0/2712334040 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb330071ea0 0x7fb33015af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:46.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.506+0000 7fb32f400640 1 --2- 172.21.15.90:0/2712334040 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb330144680 0x7fb3300ab9d0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fb31c00c3c0 tx=0x7fb31c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:46.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.507+0000 7fb327e00640 1 -- 172.21.15.90:0/2712334040 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31c0052d0 con 0x7fb330144680 2024-03-24T01:25:46.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.507+0000 7fb327e00640 1 -- 172.21.15.90:0/2712334040 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb31c005490 con 0x7fb330144680 2024-03-24T01:25:46.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.507+0000 7fb327e00640 1 -- 172.21.15.90:0/2712334040 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31c007260 con 0x7fb330144680 2024-03-24T01:25:46.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.507+0000 7fb335800640 1 -- 172.21.15.90:0/2712334040 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3300a8a30 con 0x7fb330144680 2024-03-24T01:25:46.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.507+0000 7fb335800640 1 -- 172.21.15.90:0/2712334040 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3300a8ff0 con 0x7fb330144680 2024-03-24T01:25:46.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.508+0000 7fb335800640 1 -- 172.21.15.90:0/2712334040 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb33004f400 con 0x7fb330144680 2024-03-24T01:25:46.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.509+0000 7fb327e00640 1 -- 172.21.15.90:0/2712334040 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb31c007400 con 0x7fb330144680 2024-03-24T01:25:46.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.510+0000 7fb327e00640 1 --2- 172.21.15.90:0/2712334040 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb31006bbd0 0x7fb31006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:46.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.510+0000 7fb327e00640 1 -- 172.21.15.90:0/2712334040 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6764+0+0 (secure 0 0 0) 0x7fb31c08a380 con 0x7fb330144680 2024-03-24T01:25:46.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.510+0000 7fb32fe00640 1 --2- 172.21.15.90:0/2712334040 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb31006bbd0 0x7fb31006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:46.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.511+0000 7fb32fe00640 1 --2- 172.21.15.90:0/2712334040 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb31006bbd0 0x7fb31006e090 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb3200011e0 tx=0x7fb320009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:46.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.514+0000 7fb327e00640 1 -- 172.21.15.90:0/2712334040 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb31c058a40 con 0x7fb330144680 2024-03-24T01:25:46.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.751+0000 7fb335800640 1 -- 172.21.15.90:0/2712334040 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fb3300ae760 con 0x7fb330144680 2024-03-24T01:25:46.753 INFO:tasks.ceph.mon.a.smithi090.stderr:2024-03-24T01:25:46.753+0000 7fc6f2400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T01:25:46.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.757+0000 7fb327e00640 1 -- 172.21.15.90:0/2712334040 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7fb31c058340 con 0x7fb330144680 2024-03-24T01:25:46.757 INFO:teuthology.orchestra.run.smithi090.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T01:25:46.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.760+0000 7fb335800640 1 -- 172.21.15.90:0/2712334040 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb31006bbd0 msgr2=0x7fb31006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:46.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.760+0000 7fb335800640 1 --2- 172.21.15.90:0/2712334040 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb31006bbd0 0x7fb31006e090 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb3200011e0 tx=0x7fb320009040 comp rx=0 tx=0).stop 2024-03-24T01:25:46.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.760+0000 7fb335800640 1 -- 172.21.15.90:0/2712334040 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb330144680 msgr2=0x7fb3300ab9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:46.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.760+0000 7fb335800640 1 --2- 172.21.15.90:0/2712334040 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb330144680 0x7fb3300ab9d0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fb31c00c3c0 tx=0x7fb31c00c880 comp rx=0 tx=0).stop 2024-03-24T01:25:46.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.760+0000 7fb335800640 1 -- 172.21.15.90:0/2712334040 shutdown_connections 2024-03-24T01:25:46.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.760+0000 7fb335800640 1 --2- 172.21.15.90:0/2712334040 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb31006bbd0 0x7fb31006e090 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:46.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.760+0000 7fb335800640 1 --2- 172.21.15.90:0/2712334040 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb330144680 0x7fb3300ab9d0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:46.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.760+0000 7fb335800640 1 --2- 172.21.15.90:0/2712334040 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb330145190 0x7fb3300b3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:46.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.760+0000 7fb335800640 1 --2- 172.21.15.90:0/2712334040 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb330071ea0 0x7fb33015af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:46.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.760+0000 7fb335800640 1 -- 172.21.15.90:0/2712334040 >> 172.21.15.90:0/2712334040 conn(0x7fb33006b0c0 msgr2=0x7fb330140030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:46.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.760+0000 7fb335800640 1 -- 172.21.15.90:0/2712334040 shutdown_connections 2024-03-24T01:25:46.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.760+0000 7fb335800640 1 -- 172.21.15.90:0/2712334040 wait complete. 2024-03-24T01:25:46.779 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-24T01:25:46.788 INFO:tasks.ceph.mds.d.smithi120.stderr: -6020> 2024-03-24T01:24:11.669+0000 7ff5bbc00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T01:25:46.861 INFO:tasks.ceph.mds.d.smithi120.stdout:starting mds.d at 2024-03-24T01:25:46.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.983+0000 7fefa4400640 1 Processor -- start 2024-03-24T01:25:46.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.983+0000 7fefa4400640 1 -- start start 2024-03-24T01:25:46.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.985+0000 7fefa4400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fef9c15a1d0 0x7fef9c072220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:46.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.985+0000 7fefa4400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fef9c06f4e0 0x7fef9c0cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:46.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.985+0000 7fefa4400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef9c0b2930 con 0x7fef9c072760 2024-03-24T01:25:46.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.985+0000 7fefa4400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef9c0b39d0 con 0x7fef9c15a1d0 2024-03-24T01:25:46.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.985+0000 7fefa4400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef9c072bc0 con 0x7fef9c06f4e0 2024-03-24T01:25:46.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.985+0000 7fefa1a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fef9c15a1d0 0x7fef9c072220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:46.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.985+0000 7fefa1000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fef9c072760 0x7fef9c06edd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56150/0 (socket says 172.21.15.90:56150) 2024-03-24T01:25:46.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.985+0000 7fefa1000640 1 -- 172.21.15.90:0/1604333272 learned_addr learned my addr 172.21.15.90:0/1604333272 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:46.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.986+0000 7fefa2400640 1 --2- 172.21.15.90:0/1604333272 >> v2:172.21.15.120:3300/0 conn(0x7fef9c06f4e0 0x7fef9c0cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:46.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.986+0000 7fefa1a00640 1 -- 172.21.15.90:0/1604333272 >> v2:172.21.15.120:3300/0 conn(0x7fef9c06f4e0 msgr2=0x7fef9c0cd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:46.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.986+0000 7fefa1a00640 1 --2- 172.21.15.90:0/1604333272 >> v2:172.21.15.120:3300/0 conn(0x7fef9c06f4e0 0x7fef9c0cd1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:46.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.986+0000 7fef9be00640 1 -- 172.21.15.90:0/1604333272 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1683778128 0 0) 0x7fef9c0b2930 con 0x7fef9c072760 2024-03-24T01:25:46.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.986+0000 7fefa1a00640 1 -- 172.21.15.90:0/1604333272 >> v1:172.21.15.90:6789/0 conn(0x7fef9c072760 legacy=0x7fef9c06edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:46.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.986+0000 7fefa1a00640 1 -- 172.21.15.90:0/1604333272 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef9c0b2c10 con 0x7fef9c15a1d0 2024-03-24T01:25:46.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.987+0000 7fefa1a00640 1 --2- 172.21.15.90:0/1604333272 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef9c15a1d0 0x7fef9c072220 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fef9c0af430 tx=0x7fef9002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ccca0d4a974f2c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:46.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.987+0000 7fef9be00640 1 -- 172.21.15.90:0/1604333272 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef90035020 con 0x7fef9c15a1d0 2024-03-24T01:25:46.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.987+0000 7fef9be00640 1 -- 172.21.15.90:0/1604333272 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef90031830 con 0x7fef9c15a1d0 2024-03-24T01:25:46.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.987+0000 7fef9be00640 1 -- 172.21.15.90:0/1604333272 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef90031c00 con 0x7fef9c15a1d0 2024-03-24T01:25:46.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.988+0000 7fefa4400640 1 -- 172.21.15.90:0/1604333272 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef9c15a1d0 msgr2=0x7fef9c072220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:46.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.988+0000 7fefa4400640 1 --2- 172.21.15.90:0/1604333272 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef9c15a1d0 0x7fef9c072220 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fef9c0af430 tx=0x7fef9002b8d0 comp rx=0 tx=0).stop 2024-03-24T01:25:46.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.988+0000 7fefa4400640 1 -- 172.21.15.90:0/1604333272 shutdown_connections 2024-03-24T01:25:46.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.988+0000 7fefa4400640 1 --2- 172.21.15.90:0/1604333272 >> v2:172.21.15.120:3300/0 conn(0x7fef9c06f4e0 0x7fef9c0cd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:46.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.988+0000 7fefa4400640 1 --2- 172.21.15.90:0/1604333272 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef9c15a1d0 0x7fef9c072220 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:46.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.988+0000 7fefa4400640 1 -- 172.21.15.90:0/1604333272 >> 172.21.15.90:0/1604333272 conn(0x7fef9c0cf7e0 msgr2=0x7fef9c0cfbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:46.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.988+0000 7fefa4400640 1 -- 172.21.15.90:0/1604333272 shutdown_connections 2024-03-24T01:25:46.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.988+0000 7fefa4400640 1 -- 172.21.15.90:0/1604333272 wait complete. 2024-03-24T01:25:46.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.990+0000 7fefa4400640 1 Processor -- start 2024-03-24T01:25:46.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.990+0000 7fefa4400640 1 -- start start 2024-03-24T01:25:46.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.991+0000 7fefa4400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef9c06f4e0 0x7fef9c1ed420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:46.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.991+0000 7fefa4400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef9c072760 0x7fef9c1f45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:46.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.991+0000 7fefa4400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef9c15a1d0 0x7fef9c15b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:46.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.991+0000 7fefa4400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c0b34b0 con 0x7fef9c072760 2024-03-24T01:25:46.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.991+0000 7fefa4400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c0b31d0 con 0x7fef9c06f4e0 2024-03-24T01:25:46.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.991+0000 7fefa4400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c15a5b0 con 0x7fef9c15a1d0 2024-03-24T01:25:46.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.991+0000 7fefa1000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef9c072760 0x7fef9c1f45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:46.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.991+0000 7fefa1000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef9c072760 0x7fef9c1f45b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38956/0 (socket says 172.21.15.90:38956) 2024-03-24T01:25:46.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.991+0000 7fefa1000640 1 -- 172.21.15.90:0/32776692 learned_addr learned my addr 172.21.15.90:0/32776692 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:46.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.991+0000 7fefa1000640 1 -- 172.21.15.90:0/32776692 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef9c15a1d0 msgr2=0x7fef9c15b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:46.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.991+0000 7fefa1000640 1 --2- 172.21.15.90:0/32776692 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef9c15a1d0 0x7fef9c15b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:46.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.991+0000 7fefa1000640 1 -- 172.21.15.90:0/32776692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef9c06f4e0 msgr2=0x7fef9c1ed420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:46.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.991+0000 7fefa1000640 1 --2- 172.21.15.90:0/32776692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef9c06f4e0 0x7fef9c1ed420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:46.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.991+0000 7fefa1000640 1 -- 172.21.15.90:0/32776692 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef9c1f43c0 con 0x7fef9c072760 2024-03-24T01:25:46.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.992+0000 7fefa1000640 1 --2- 172.21.15.90:0/32776692 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef9c072760 0x7fef9c1f45b0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fef8800c4d0 tx=0x7fef8800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:46.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.992+0000 7fef99600640 1 -- 172.21.15.90:0/32776692 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef88018070 con 0x7fef9c072760 2024-03-24T01:25:46.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.992+0000 7fef99600640 1 -- 172.21.15.90:0/32776692 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef88007500 con 0x7fef9c072760 2024-03-24T01:25:46.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.992+0000 7fefa4400640 1 -- 172.21.15.90:0/32776692 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef9c1f40e0 con 0x7fef9c072760 2024-03-24T01:25:46.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.992+0000 7fef99600640 1 -- 172.21.15.90:0/32776692 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef88004a20 con 0x7fef9c072760 2024-03-24T01:25:46.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.992+0000 7fefa4400640 1 -- 172.21.15.90:0/32776692 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef9c15ab80 con 0x7fef9c072760 2024-03-24T01:25:46.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.993+0000 7fefa4400640 1 -- 172.21.15.90:0/32776692 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef9c04f400 con 0x7fef9c072760 2024-03-24T01:25:46.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.994+0000 7fef99600640 1 -- 172.21.15.90:0/32776692 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fef8801b070 con 0x7fef9c072760 2024-03-24T01:25:46.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.995+0000 7fef99600640 1 --2- 172.21.15.90:0/32776692 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fef7c06bc00 0x7fef7c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:46.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.995+0000 7fef99600640 1 -- 172.21.15.90:0/32776692 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6850+0+0 (secure 0 0 0) 0x7fef8808a1d0 con 0x7fef9c072760 2024-03-24T01:25:46.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.996+0000 7fefa1a00640 1 --2- 172.21.15.90:0/32776692 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fef7c06bc00 0x7fef7c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:46.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.997+0000 7fefa1a00640 1 --2- 172.21.15.90:0/32776692 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fef7c06bc00 0x7fef7c06e0c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fef90036000 tx=0x7fef90030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:46.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:46.999+0000 7fef99600640 1 -- 172.21.15.90:0/32776692 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fef88058820 con 0x7fef9c072760 2024-03-24T01:25:47.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:47.239+0000 7fefa4400640 1 -- 172.21.15.90:0/32776692 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fef9c06f1e0 con 0x7fef9c072760 2024-03-24T01:25:47.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:47.842+0000 7fef99600640 1 -- 172.21.15.90:0/32776692 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7fef88058120 con 0x7fef9c072760 2024-03-24T01:25:47.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:47.846+0000 7fefa4400640 1 -- 172.21.15.90:0/32776692 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fef7c06bc00 msgr2=0x7fef7c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:47.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:47.846+0000 7fefa4400640 1 --2- 172.21.15.90:0/32776692 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fef7c06bc00 0x7fef7c06e0c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fef90036000 tx=0x7fef90030000 comp rx=0 tx=0).stop 2024-03-24T01:25:47.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:47.846+0000 7fefa4400640 1 -- 172.21.15.90:0/32776692 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef9c072760 msgr2=0x7fef9c1f45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:47.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:47.846+0000 7fefa4400640 1 --2- 172.21.15.90:0/32776692 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef9c072760 0x7fef9c1f45b0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fef8800c4d0 tx=0x7fef8800c990 comp rx=0 tx=0).stop 2024-03-24T01:25:47.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:47.846+0000 7fefa4400640 1 -- 172.21.15.90:0/32776692 shutdown_connections 2024-03-24T01:25:47.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:47.846+0000 7fefa4400640 1 --2- 172.21.15.90:0/32776692 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fef7c06bc00 0x7fef7c06e0c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:47.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:47.846+0000 7fefa4400640 1 --2- 172.21.15.90:0/32776692 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef9c072760 0x7fef9c1f45b0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:47.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:47.846+0000 7fefa4400640 1 --2- 172.21.15.90:0/32776692 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fef9c15a1d0 0x7fef9c15b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:47.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:47.846+0000 7fefa4400640 1 --2- 172.21.15.90:0/32776692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef9c06f4e0 0x7fef9c1ed420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:47.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:47.846+0000 7fefa4400640 1 -- 172.21.15.90:0/32776692 >> 172.21.15.90:0/32776692 conn(0x7fef9c0cf7e0 msgr2=0x7fef9c159310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:47.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:47.846+0000 7fefa4400640 1 -- 172.21.15.90:0/32776692 shutdown_connections 2024-03-24T01:25:47.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:47.846+0000 7fefa4400640 1 -- 172.21.15.90:0/32776692 wait complete. 2024-03-24T01:25:47.866 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-24T01:25:48.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.074+0000 7f233e600640 1 Processor -- start 2024-03-24T01:25:48.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.074+0000 7f233e600640 1 -- start start 2024-03-24T01:25:48.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.075+0000 7f233e600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23380cbe50 0x7f233815f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:48.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.075+0000 7f233e600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2338144060 0x7f23380ce930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:48.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.075+0000 7f233e600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23381444a0 con 0x7f233806e900 2024-03-24T01:25:48.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.075+0000 7f233e600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23380cf180 con 0x7f2338144060 2024-03-24T01:25:48.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.075+0000 7f233e600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23380cf460 con 0x7f23380cbe50 2024-03-24T01:25:48.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.075+0000 7f233dc00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f233806e900 0x7f23380b34a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56158/0 (socket says 172.21.15.90:56158) 2024-03-24T01:25:48.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.075+0000 7f233dc00640 1 -- 172.21.15.90:0/806994300 learned_addr learned my addr 172.21.15.90:0/806994300 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:48.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.076+0000 7f232fe00640 1 --2- 172.21.15.90:0/806994300 >> v2:172.21.15.90:3300/0 conn(0x7f2338144060 0x7f23380ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:48.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.076+0000 7f233d200640 1 --2- 172.21.15.90:0/806994300 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23380cbe50 0x7f233815f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:48.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.076+0000 7f2337e00640 1 -- 172.21.15.90:0/806994300 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1653575693 0 0) 0x7f23381444a0 con 0x7f233806e900 2024-03-24T01:25:48.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.076+0000 7f2337e00640 1 -- 172.21.15.90:0/806994300 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f231c003520 con 0x7f233806e900 2024-03-24T01:25:48.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.076+0000 7f232fe00640 1 -- 172.21.15.90:0/806994300 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23380cbe50 msgr2=0x7f233815f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:48.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.076+0000 7f232fe00640 1 --2- 172.21.15.90:0/806994300 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23380cbe50 0x7f233815f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:48.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.076+0000 7f232fe00640 1 -- 172.21.15.90:0/806994300 >> v1:172.21.15.90:6789/0 conn(0x7f233806e900 legacy=0x7f23380b34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:48.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.076+0000 7f232fe00640 1 -- 172.21.15.90:0/806994300 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f233815e670 con 0x7f2338144060 2024-03-24T01:25:48.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.077+0000 7f232fe00640 1 --2- 172.21.15.90:0/806994300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2338144060 0x7f23380ce930 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f23280097a0 tx=0x7f232802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=532a29981d7a29d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:48.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.078+0000 7f2337e00640 1 -- 172.21.15.90:0/806994300 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2328031780 con 0x7f2338144060 2024-03-24T01:25:48.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.078+0000 7f2337e00640 1 -- 172.21.15.90:0/806994300 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2328037070 con 0x7f2338144060 2024-03-24T01:25:48.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.078+0000 7f2337e00640 1 -- 172.21.15.90:0/806994300 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2328031b40 con 0x7f2338144060 2024-03-24T01:25:48.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.078+0000 7f233e600640 1 -- 172.21.15.90:0/806994300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2338144060 msgr2=0x7f23380ce930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:48.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.078+0000 7f233e600640 1 --2- 172.21.15.90:0/806994300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2338144060 0x7f23380ce930 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f23280097a0 tx=0x7f232802b820 comp rx=0 tx=0).stop 2024-03-24T01:25:48.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.078+0000 7f233e600640 1 -- 172.21.15.90:0/806994300 shutdown_connections 2024-03-24T01:25:48.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.078+0000 7f233e600640 1 --2- 172.21.15.90:0/806994300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2338144060 0x7f23380ce930 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:48.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.078+0000 7f233e600640 1 --2- 172.21.15.90:0/806994300 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f23380cbe50 0x7f233815f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:48.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.078+0000 7f233e600640 1 -- 172.21.15.90:0/806994300 >> 172.21.15.90:0/806994300 conn(0x7f233806b0c0 msgr2=0x7f233806b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:48.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.079+0000 7f233e600640 1 -- 172.21.15.90:0/806994300 shutdown_connections 2024-03-24T01:25:48.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.079+0000 7f233e600640 1 -- 172.21.15.90:0/806994300 wait complete. 2024-03-24T01:25:48.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.080+0000 7f233e600640 1 Processor -- start 2024-03-24T01:25:48.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.080+0000 7f233e600640 1 -- start start 2024-03-24T01:25:48.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.081+0000 7f233e600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f233806e900 0x7f23381f35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:48.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.081+0000 7f233e600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23380cbe50 0x7f23381403a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:48.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.081+0000 7f233e600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2338144060 0x7f23380add40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:48.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.081+0000 7f233e600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23380cee70 con 0x7f23380cbe50 2024-03-24T01:25:48.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.081+0000 7f233e600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f233815ec30 con 0x7f233806e900 2024-03-24T01:25:48.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.081+0000 7f233e600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23380cc230 con 0x7f2338144060 2024-03-24T01:25:48.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.082+0000 7f232fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23380cbe50 0x7f23381403a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:48.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.082+0000 7f232fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23380cbe50 0x7f23381403a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38974/0 (socket says 172.21.15.90:38974) 2024-03-24T01:25:48.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.082+0000 7f232fe00640 1 -- 172.21.15.90:0/1705615989 learned_addr learned my addr 172.21.15.90:0/1705615989 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:48.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.082+0000 7f233dc00640 1 --2- 172.21.15.90:0/1705615989 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2338144060 0x7f23380add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:48.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.082+0000 7f232fe00640 1 -- 172.21.15.90:0/1705615989 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2338144060 msgr2=0x7f23380add40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:48.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.082+0000 7f233d200640 1 --2- 172.21.15.90:0/1705615989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f233806e900 0x7f23381f35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:48.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.082+0000 7f232fe00640 1 --2- 172.21.15.90:0/1705615989 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2338144060 0x7f23380add40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:48.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.082+0000 7f232fe00640 1 -- 172.21.15.90:0/1705615989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f233806e900 msgr2=0x7f23381f35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:48.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.082+0000 7f232fe00640 1 --2- 172.21.15.90:0/1705615989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f233806e900 0x7f23381f35b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:48.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.082+0000 7f232fe00640 1 -- 172.21.15.90:0/1705615989 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23381f4860 con 0x7f23380cbe50 2024-03-24T01:25:48.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.082+0000 7f233dc00640 1 --2- 172.21.15.90:0/1705615989 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2338144060 0x7f23380add40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:48.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.083+0000 7f232fe00640 1 --2- 172.21.15.90:0/1705615989 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23380cbe50 0x7f23381403a0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f2328000c00 tx=0x7f23280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:48.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.083+0000 7f2335600640 1 -- 172.21.15.90:0/1705615989 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2328032610 con 0x7f23380cbe50 2024-03-24T01:25:48.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.083+0000 7f2335600640 1 -- 172.21.15.90:0/1705615989 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23280327d0 con 0x7f23380cbe50 2024-03-24T01:25:48.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.083+0000 7f233e600640 1 -- 172.21.15.90:0/1705615989 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23381f3c50 con 0x7f23380cbe50 2024-03-24T01:25:48.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.083+0000 7f2335600640 1 -- 172.21.15.90:0/1705615989 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2328032bc0 con 0x7f23380cbe50 2024-03-24T01:25:48.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.083+0000 7f233e600640 1 -- 172.21.15.90:0/1705615989 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23381f4210 con 0x7f23380cbe50 2024-03-24T01:25:48.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.084+0000 7f233e600640 1 -- 172.21.15.90:0/1705615989 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f233804f400 con 0x7f23380cbe50 2024-03-24T01:25:48.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.085+0000 7f2335600640 1 -- 172.21.15.90:0/1705615989 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f232804c060 con 0x7f23380cbe50 2024-03-24T01:25:48.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.086+0000 7f2335600640 1 --2- 172.21.15.90:0/1705615989 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f231406bbd0 0x7f231406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:48.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.087+0000 7f2335600640 1 -- 172.21.15.90:0/1705615989 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6850+0+0 (secure 0 0 0) 0x7f23280ac800 con 0x7f23380cbe50 2024-03-24T01:25:48.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.087+0000 7f233d200640 1 --2- 172.21.15.90:0/1705615989 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f231406bbd0 0x7f231406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:48.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.088+0000 7f233d200640 1 --2- 172.21.15.90:0/1705615989 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f231406bbd0 0x7f231406e090 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2320009a10 tx=0x7f2320008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:48.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.091+0000 7f2335600640 1 -- 172.21.15.90:0/1705615989 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2328040070 con 0x7f23380cbe50 2024-03-24T01:25:48.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.288+0000 7f233e600640 1 -- 172.21.15.90:0/1705615989 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f233806b5d0 con 0x7f23380cbe50 2024-03-24T01:25:48.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.855+0000 7f2335600640 1 -- 172.21.15.90:0/1705615989 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v29) v1 ==== 181+0+0 (secure 0 0 0) 0x7f232802f050 con 0x7f23380cbe50 2024-03-24T01:25:48.857 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_metadata' removed 2024-03-24T01:25:48.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.860+0000 7f233e600640 1 -- 172.21.15.90:0/1705615989 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f231406bbd0 msgr2=0x7f231406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:48.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.860+0000 7f233e600640 1 --2- 172.21.15.90:0/1705615989 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f231406bbd0 0x7f231406e090 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2320009a10 tx=0x7f2320008040 comp rx=0 tx=0).stop 2024-03-24T01:25:48.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.860+0000 7f233e600640 1 -- 172.21.15.90:0/1705615989 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23380cbe50 msgr2=0x7f23381403a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:48.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.860+0000 7f233e600640 1 --2- 172.21.15.90:0/1705615989 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23380cbe50 0x7f23381403a0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f2328000c00 tx=0x7f23280046c0 comp rx=0 tx=0).stop 2024-03-24T01:25:48.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.860+0000 7f233e600640 1 -- 172.21.15.90:0/1705615989 shutdown_connections 2024-03-24T01:25:48.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.860+0000 7f233e600640 1 --2- 172.21.15.90:0/1705615989 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f231406bbd0 0x7f231406e090 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:48.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.860+0000 7f233e600640 1 --2- 172.21.15.90:0/1705615989 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23380cbe50 0x7f23381403a0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:48.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.860+0000 7f233e600640 1 --2- 172.21.15.90:0/1705615989 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2338144060 0x7f23380add40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:48.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.860+0000 7f233e600640 1 --2- 172.21.15.90:0/1705615989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f233806e900 0x7f23381f35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:48.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.860+0000 7f233e600640 1 -- 172.21.15.90:0/1705615989 >> 172.21.15.90:0/1705615989 conn(0x7f233806b0c0 msgr2=0x7f23380739c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:48.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.861+0000 7f233e600640 1 -- 172.21.15.90:0/1705615989 shutdown_connections 2024-03-24T01:25:48.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:48.861+0000 7f233e600640 1 -- 172.21.15.90:0/1705615989 wait complete. 2024-03-24T01:25:48.882 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-24T01:25:49.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.081+0000 7f7369600640 1 Processor -- start 2024-03-24T01:25:49.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.081+0000 7f7369600640 1 -- start start 2024-03-24T01:25:49.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.083+0000 7f7369600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f736415dc60 0x7f73640b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:49.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.083+0000 7f7369600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73640b5950 con 0x7f736406bc60 2024-03-24T01:25:49.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.083+0000 7f7369600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73640b5c30 con 0x7f736415e040 2024-03-24T01:25:49.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.083+0000 7f7369600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73640b38b0 con 0x7f736415dc60 2024-03-24T01:25:49.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.083+0000 7f7363400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f736406bc60 0x7f73640b0ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56164/0 (socket says 172.21.15.90:56164) 2024-03-24T01:25:49.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.083+0000 7f7363400640 1 -- 172.21.15.90:0/329714607 learned_addr learned my addr 172.21.15.90:0/329714607 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:49.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.084+0000 7f7362a00640 1 --2- 172.21.15.90:0/329714607 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f736415dc60 0x7f73640b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:49.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.084+0000 7f7362000640 1 -- 172.21.15.90:0/329714607 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484866509 0 0) 0x7f73640b5950 con 0x7f736406bc60 2024-03-24T01:25:49.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.084+0000 7f7362000640 1 -- 172.21.15.90:0/329714607 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7348003520 con 0x7f736406bc60 2024-03-24T01:25:49.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.084+0000 7f7362000640 1 -- 172.21.15.90:0/329714607 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2784502606 0 0) 0x7f7348003520 con 0x7f736406bc60 2024-03-24T01:25:49.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.084+0000 7f7362000640 1 -- 172.21.15.90:0/329714607 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f736415dc60 msgr2=0x7f73640b1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:49.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.084+0000 7f7362000640 1 --2- 172.21.15.90:0/329714607 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f736415dc60 0x7f73640b1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:49.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.084+0000 7f7362000640 1 -- 172.21.15.90:0/329714607 >> v1:172.21.15.120:6789/0 conn(0x7f736415e040 legacy=0x7f73640ad820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:49.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.084+0000 7f7362000640 1 -- 172.21.15.90:0/329714607 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73640b3b00 con 0x7f736406bc60 2024-03-24T01:25:49.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.084+0000 7f7362000640 1 -- 172.21.15.90:0/329714607 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f735c001a10 con 0x7f736406bc60 2024-03-24T01:25:49.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.084+0000 7f7362000640 1 -- 172.21.15.90:0/329714607 >> v1:172.21.15.90:6789/0 conn(0x7f736406bc60 legacy=0x7f73640b0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:49.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.084+0000 7f7362000640 1 --2- 172.21.15.90:0/329714607 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73480039e0 0x7f7348003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:49.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.084+0000 7f7362000640 1 --2- 172.21.15.90:0/329714607 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7348004800 0x7f7348004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:49.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.084+0000 7f7362000640 1 --2- 172.21.15.90:0/329714607 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73480052e0 0x7f734800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:49.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.084+0000 7f7362000640 1 -- 172.21.15.90:0/329714607 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73640b38b0 con 0x7f7348004800 2024-03-24T01:25:49.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.084+0000 7f7362000640 1 -- 172.21.15.90:0/329714607 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f734800c110 con 0x7f73480039e0 2024-03-24T01:25:49.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.084+0000 7f7362000640 1 -- 172.21.15.90:0/329714607 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f734800c310 con 0x7f73480052e0 2024-03-24T01:25:49.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.084+0000 7f7362a00640 1 --2- 172.21.15.90:0/329714607 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f736415dc60 0x7f73640b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:25:49.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.085+0000 7f735be00640 1 --2- 172.21.15.90:0/329714607 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7348004800 0x7f7348004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:49.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.085+0000 7f735be00640 1 -- 172.21.15.90:0/329714607 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73480052e0 msgr2=0x7f734800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:49.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.085+0000 7f735be00640 1 --2- 172.21.15.90:0/329714607 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f73480052e0 0x7f734800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:49.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.085+0000 7f735be00640 1 -- 172.21.15.90:0/329714607 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73480039e0 msgr2=0x7f7348003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:49.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.085+0000 7f735be00640 1 --2- 172.21.15.90:0/329714607 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73480039e0 0x7f7348003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:49.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.085+0000 7f735be00640 1 -- 172.21.15.90:0/329714607 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7348003520 con 0x7f7348004800 2024-03-24T01:25:49.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.086+0000 7f735be00640 1 --2- 172.21.15.90:0/329714607 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7348004800 0x7f7348004c80 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f734c001750 tx=0x7f7348005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5258006655a5ec31 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:49.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.086+0000 7f7362000640 1 -- 172.21.15.90:0/329714607 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c001370 con 0x7f7348004800 2024-03-24T01:25:49.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.086+0000 7f7362000640 1 -- 172.21.15.90:0/329714607 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f734c0052d0 con 0x7f7348004800 2024-03-24T01:25:49.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.086+0000 7f7362000640 1 -- 172.21.15.90:0/329714607 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c003830 con 0x7f7348004800 2024-03-24T01:25:49.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.087+0000 7f7369600640 1 -- 172.21.15.90:0/329714607 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7348004800 msgr2=0x7f7348004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:49.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.087+0000 7f7369600640 1 --2- 172.21.15.90:0/329714607 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7348004800 0x7f7348004c80 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f734c001750 tx=0x7f7348005830 comp rx=0 tx=0).stop 2024-03-24T01:25:49.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.087+0000 7f7363400640 1 -- 172.21.15.90:0/329714607 reap_dead start 2024-03-24T01:25:49.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.087+0000 7f7369600640 1 -- 172.21.15.90:0/329714607 shutdown_connections 2024-03-24T01:25:49.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.087+0000 7f7369600640 1 -- 172.21.15.90:0/329714607 >> 172.21.15.90:0/329714607 conn(0x7f736406cec0 msgr2=0x7f7364099a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:49.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.087+0000 7f7369600640 1 -- 172.21.15.90:0/329714607 shutdown_connections 2024-03-24T01:25:49.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.087+0000 7f7369600640 1 -- 172.21.15.90:0/329714607 wait complete. 2024-03-24T01:25:49.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.089+0000 7f7369600640 1 Processor -- start 2024-03-24T01:25:49.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.089+0000 7f7369600640 1 -- start start 2024-03-24T01:25:49.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.090+0000 7f7369600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7348004800 0x7f736415a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:49.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.090+0000 7f7369600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f736415f6e0 0x7f7364152130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:49.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.090+0000 7f7369600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f736415fb60 0x7f7364152670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:49.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.090+0000 7f7369600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73640adf30 con 0x7f7348004800 2024-03-24T01:25:49.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.090+0000 7f7369600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73640b3e70 con 0x7f736415f6e0 2024-03-24T01:25:49.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.090+0000 7f7369600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f736415a010 con 0x7f736415fb60 2024-03-24T01:25:49.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.090+0000 7f7362a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7348004800 0x7f736415a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:49.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.090+0000 7f7362a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7348004800 0x7f736415a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38994/0 (socket says 172.21.15.90:38994) 2024-03-24T01:25:49.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.090+0000 7f7362a00640 1 -- 172.21.15.90:0/1452083259 learned_addr learned my addr 172.21.15.90:0/1452083259 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:49.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.091+0000 7f735be00640 1 --2- 172.21.15.90:0/1452083259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f736415f6e0 0x7f7364152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:49.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.091+0000 7f7363400640 1 --2- 172.21.15.90:0/1452083259 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f736415fb60 0x7f7364152670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:49.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.091+0000 7f7362a00640 1 -- 172.21.15.90:0/1452083259 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f736415fb60 msgr2=0x7f7364152670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:49.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.091+0000 7f7362a00640 1 --2- 172.21.15.90:0/1452083259 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f736415fb60 0x7f7364152670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:49.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.091+0000 7f7362a00640 1 -- 172.21.15.90:0/1452083259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f736415f6e0 msgr2=0x7f7364152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:49.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.091+0000 7f7362a00640 1 --2- 172.21.15.90:0/1452083259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f736415f6e0 0x7f7364152130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:49.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.091+0000 7f7362a00640 1 -- 172.21.15.90:0/1452083259 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f736415c6d0 con 0x7f7348004800 2024-03-24T01:25:49.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.091+0000 7f7363400640 1 --2- 172.21.15.90:0/1452083259 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f736415fb60 0x7f7364152670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:49.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.091+0000 7f7362a00640 1 --2- 172.21.15.90:0/1452083259 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7348004800 0x7f736415a6f0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f7354006d40 tx=0x7f735400d910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:49.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.091+0000 7f735aa00640 1 -- 172.21.15.90:0/1452083259 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7354004030 con 0x7f7348004800 2024-03-24T01:25:49.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.091+0000 7f735aa00640 1 -- 172.21.15.90:0/1452083259 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73540041f0 con 0x7f7348004800 2024-03-24T01:25:49.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.091+0000 7f7369600640 1 -- 172.21.15.90:0/1452083259 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f736415c9b0 con 0x7f7348004800 2024-03-24T01:25:49.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.092+0000 7f735aa00640 1 -- 172.21.15.90:0/1452083259 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7354003610 con 0x7f7348004800 2024-03-24T01:25:49.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.092+0000 7f7369600640 1 -- 172.21.15.90:0/1452083259 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f736415cf70 con 0x7f7348004800 2024-03-24T01:25:49.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.093+0000 7f7369600640 1 -- 172.21.15.90:0/1452083259 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f736404f400 con 0x7f7348004800 2024-03-24T01:25:49.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.094+0000 7f735aa00640 1 -- 172.21.15.90:0/1452083259 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f73540037b0 con 0x7f7348004800 2024-03-24T01:25:49.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.094+0000 7f735aa00640 1 --2- 172.21.15.90:0/1452083259 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f734006bbd0 0x7f734006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:49.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.095+0000 7f735aa00640 1 -- 172.21.15.90:0/1452083259 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6412+0+0 (secure 0 0 0) 0x7f735408af30 con 0x7f7348004800 2024-03-24T01:25:49.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.095+0000 7f735be00640 1 --2- 172.21.15.90:0/1452083259 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f734006bbd0 0x7f734006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:49.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.096+0000 7f735be00640 1 --2- 172.21.15.90:0/1452083259 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f734006bbd0 0x7f734006e090 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f734c033000 tx=0x7f734c009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:49.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.099+0000 7f735aa00640 1 -- 172.21.15.90:0/1452083259 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7354059750 con 0x7f7348004800 2024-03-24T01:25:49.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.295+0000 7f7369600640 1 -- 172.21.15.90:0/1452083259 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f73641582a0 con 0x7f7348004800 2024-03-24T01:25:49.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.882+0000 7f735aa00640 1 -- 172.21.15.90:0/1452083259 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v30) v1 ==== 169+0+0 (secure 0 0 0) 0x7f7354059050 con 0x7f7348004800 2024-03-24T01:25:49.883 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_data' removed 2024-03-24T01:25:49.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.885+0000 7f7369600640 1 -- 172.21.15.90:0/1452083259 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f734006bbd0 msgr2=0x7f734006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:49.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.885+0000 7f7369600640 1 --2- 172.21.15.90:0/1452083259 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f734006bbd0 0x7f734006e090 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f734c033000 tx=0x7f734c009030 comp rx=0 tx=0).stop 2024-03-24T01:25:49.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.886+0000 7f7369600640 1 -- 172.21.15.90:0/1452083259 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7348004800 msgr2=0x7f736415a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:49.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.886+0000 7f7369600640 1 --2- 172.21.15.90:0/1452083259 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7348004800 0x7f736415a6f0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f7354006d40 tx=0x7f735400d910 comp rx=0 tx=0).stop 2024-03-24T01:25:49.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.886+0000 7f7369600640 1 -- 172.21.15.90:0/1452083259 shutdown_connections 2024-03-24T01:25:49.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.886+0000 7f7369600640 1 --2- 172.21.15.90:0/1452083259 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f734006bbd0 0x7f734006e090 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:49.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.886+0000 7f7369600640 1 --2- 172.21.15.90:0/1452083259 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f736415fb60 0x7f7364152670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:49.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.886+0000 7f7369600640 1 --2- 172.21.15.90:0/1452083259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f736415f6e0 0x7f7364152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:49.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.886+0000 7f7369600640 1 --2- 172.21.15.90:0/1452083259 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7348004800 0x7f736415a6f0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:49.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.886+0000 7f7369600640 1 -- 172.21.15.90:0/1452083259 >> 172.21.15.90:0/1452083259 conn(0x7f736406cec0 msgr2=0x7f736406eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:49.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.887+0000 7f7369600640 1 -- 172.21.15.90:0/1452083259 shutdown_connections 2024-03-24T01:25:49.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:49.887+0000 7f7369600640 1 -- 172.21.15.90:0/1452083259 wait complete. 2024-03-24T01:25:49.912 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-24T01:25:50.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.110+0000 7f1616600640 1 Processor -- start 2024-03-24T01:25:50.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.110+0000 7f1616600640 1 -- start start 2024-03-24T01:25:50.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.111+0000 7f1616600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f161015bc50 0x7f161006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:50.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.111+0000 7f1616600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f161006dac0 con 0x7f16100aeeb0 2024-03-24T01:25:50.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.111+0000 7f1616600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f161006dda0 con 0x7f161006b600 2024-03-24T01:25:50.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.111+0000 7f1616600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f161006ba70 con 0x7f161015bc50 2024-03-24T01:25:50.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.111+0000 7f160f400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f16100aeeb0 0x7f16100b44c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:58122/0 (socket says 172.21.15.90:58122) 2024-03-24T01:25:50.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.111+0000 7f160f400640 1 -- 172.21.15.90:0/1029037791 learned_addr learned my addr 172.21.15.90:0/1029037791 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:50.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.111+0000 7f1606a00640 1 --2- 172.21.15.90:0/1029037791 >> v2:172.21.15.90:3300/0 conn(0x7f161015bc50 0x7f161006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:50.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.112+0000 7f160ea00640 1 -- 172.21.15.90:0/1029037791 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 821631220 0 0) 0x7f161006dac0 con 0x7f16100aeeb0 2024-03-24T01:25:50.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.112+0000 7f160ea00640 1 -- 172.21.15.90:0/1029037791 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15f4003520 con 0x7f16100aeeb0 2024-03-24T01:25:50.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.112+0000 7f1606a00640 1 -- 172.21.15.90:0/1029037791 >> v1:172.21.15.120:6789/0 conn(0x7f161006b600 legacy=0x7f16100b2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:50.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.112+0000 7f1606a00640 1 -- 172.21.15.90:0/1029037791 >> v1:172.21.15.90:6789/0 conn(0x7f16100aeeb0 legacy=0x7f16100b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:50.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.112+0000 7f1606a00640 1 -- 172.21.15.90:0/1029037791 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f161006e080 con 0x7f161015bc50 2024-03-24T01:25:50.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.113+0000 7f1606a00640 1 --2- 172.21.15.90:0/1029037791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f161015bc50 0x7f161006b0c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f15f80097a0 tx=0x7f15f802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e836d84930b2596 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:50.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.113+0000 7f160ea00640 1 -- 172.21.15.90:0/1029037791 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f8038070 con 0x7f161015bc50 2024-03-24T01:25:50.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.113+0000 7f160ea00640 1 -- 172.21.15.90:0/1029037791 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15f8031700 con 0x7f161015bc50 2024-03-24T01:25:50.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.113+0000 7f160ea00640 1 -- 172.21.15.90:0/1029037791 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f8031ad0 con 0x7f161015bc50 2024-03-24T01:25:50.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.114+0000 7f1616600640 1 -- 172.21.15.90:0/1029037791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f161015bc50 msgr2=0x7f161006b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:50.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.114+0000 7f1616600640 1 --2- 172.21.15.90:0/1029037791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f161015bc50 0x7f161006b0c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f15f80097a0 tx=0x7f15f802b7a0 comp rx=0 tx=0).stop 2024-03-24T01:25:50.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.114+0000 7f1616600640 1 -- 172.21.15.90:0/1029037791 shutdown_connections 2024-03-24T01:25:50.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.114+0000 7f1616600640 1 --2- 172.21.15.90:0/1029037791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f161015bc50 0x7f161006b0c0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:50.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.114+0000 7f1616600640 1 -- 172.21.15.90:0/1029037791 >> 172.21.15.90:0/1029037791 conn(0x7f16100b1770 msgr2=0x7f16100b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:50.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.114+0000 7f1616600640 1 -- 172.21.15.90:0/1029037791 shutdown_connections 2024-03-24T01:25:50.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.114+0000 7f1616600640 1 -- 172.21.15.90:0/1029037791 wait complete. 2024-03-24T01:25:50.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.116+0000 7f1616600640 1 Processor -- start 2024-03-24T01:25:50.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.116+0000 7f1616600640 1 -- start start 2024-03-24T01:25:50.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.117+0000 7f1616600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f161006b600 0x7f16100af920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:50.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.117+0000 7f1616600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16100aeeb0 0x7f1610143240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:50.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.117+0000 7f1616600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f161015bc50 0x7f16101f3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:50.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.117+0000 7f1616600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16100af290 con 0x7f16100aeeb0 2024-03-24T01:25:50.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.117+0000 7f1616600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16100b4da0 con 0x7f161015bc50 2024-03-24T01:25:50.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.117+0000 7f1616600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f161006e880 con 0x7f161006b600 2024-03-24T01:25:50.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.117+0000 7f1606a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16100aeeb0 0x7f1610143240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:50.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.117+0000 7f1606a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16100aeeb0 0x7f1610143240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60084/0 (socket says 172.21.15.90:60084) 2024-03-24T01:25:50.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.117+0000 7f1606a00640 1 -- 172.21.15.90:0/3532717657 learned_addr learned my addr 172.21.15.90:0/3532717657 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:50.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.117+0000 7f160f400640 1 --2- 172.21.15.90:0/3532717657 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f161006b600 0x7f16100af920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:50.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.118+0000 7f1606a00640 1 -- 172.21.15.90:0/3532717657 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f161006b600 msgr2=0x7f16100af920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:50.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.118+0000 7f1606a00640 1 --2- 172.21.15.90:0/3532717657 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f161006b600 0x7f16100af920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:50.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.118+0000 7f1606a00640 1 -- 172.21.15.90:0/3532717657 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f161015bc50 msgr2=0x7f16101f3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:50.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.118+0000 7f1606a00640 1 --2- 172.21.15.90:0/3532717657 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f161015bc50 0x7f16101f3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:50.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.118+0000 7f1606a00640 1 -- 172.21.15.90:0/3532717657 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1610143780 con 0x7f16100aeeb0 2024-03-24T01:25:50.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.118+0000 7f1606a00640 1 --2- 172.21.15.90:0/3532717657 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16100aeeb0 0x7f1610143240 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f15f8000c00 tx=0x7f15f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:50.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.118+0000 7f1607e00640 1 -- 172.21.15.90:0/3532717657 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f8042070 con 0x7f16100aeeb0 2024-03-24T01:25:50.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.118+0000 7f1607e00640 1 -- 172.21.15.90:0/3532717657 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15f803d890 con 0x7f16100aeeb0 2024-03-24T01:25:50.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.118+0000 7f1607e00640 1 -- 172.21.15.90:0/3532717657 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f8038040 con 0x7f16100aeeb0 2024-03-24T01:25:50.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.118+0000 7f1616600640 1 -- 172.21.15.90:0/3532717657 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16101520c0 con 0x7f16100aeeb0 2024-03-24T01:25:50.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.118+0000 7f1616600640 1 -- 172.21.15.90:0/3532717657 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16101507a0 con 0x7f16100aeeb0 2024-03-24T01:25:50.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.120+0000 7f1616600640 1 -- 172.21.15.90:0/3532717657 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1610150ae0 con 0x7f16100aeeb0 2024-03-24T01:25:50.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.121+0000 7f1607e00640 1 -- 172.21.15.90:0/3532717657 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f15f8045070 con 0x7f16100aeeb0 2024-03-24T01:25:50.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.121+0000 7f1607e00640 1 --2- 172.21.15.90:0/3532717657 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f15ec06bbd0 0x7f15ec06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:50.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.122+0000 7f1607e00640 1 -- 172.21.15.90:0/3532717657 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f15f80ac630 con 0x7f16100aeeb0 2024-03-24T01:25:50.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.122+0000 7f160f400640 1 --2- 172.21.15.90:0/3532717657 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f15ec06bbd0 0x7f15ec06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:50.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.123+0000 7f160f400640 1 --2- 172.21.15.90:0/3532717657 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f15ec06bbd0 0x7f15ec06e090 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1600005880 tx=0x7f1600009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:50.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.126+0000 7f1607e00640 1 -- 172.21.15.90:0/3532717657 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1610150ae0 con 0x7f16100aeeb0 2024-03-24T01:25:50.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.326+0000 7f1616600640 1 -- 172.21.15.90:0/3532717657 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f16100b1b50 con 0x7f16100aeeb0 2024-03-24T01:25:50.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.895+0000 7f1607e00640 1 -- 172.21.15.90:0/3532717657 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v31) v1 ==== 178+0+0 (secure 0 0 0) 0x7f16100b1b50 con 0x7f16100aeeb0 2024-03-24T01:25:50.896 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_data_ec' removed 2024-03-24T01:25:50.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.899+0000 7f1616600640 1 -- 172.21.15.90:0/3532717657 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f15ec06bbd0 msgr2=0x7f15ec06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:50.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.899+0000 7f1616600640 1 --2- 172.21.15.90:0/3532717657 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f15ec06bbd0 0x7f15ec06e090 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1600005880 tx=0x7f1600009040 comp rx=0 tx=0).stop 2024-03-24T01:25:50.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.899+0000 7f1616600640 1 -- 172.21.15.90:0/3532717657 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16100aeeb0 msgr2=0x7f1610143240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:50.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.899+0000 7f1616600640 1 --2- 172.21.15.90:0/3532717657 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16100aeeb0 0x7f1610143240 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f15f8000c00 tx=0x7f15f80046c0 comp rx=0 tx=0).stop 2024-03-24T01:25:50.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.899+0000 7f1616600640 1 -- 172.21.15.90:0/3532717657 shutdown_connections 2024-03-24T01:25:50.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.899+0000 7f1616600640 1 --2- 172.21.15.90:0/3532717657 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f15ec06bbd0 0x7f15ec06e090 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:50.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.899+0000 7f1616600640 1 --2- 172.21.15.90:0/3532717657 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16100aeeb0 0x7f1610143240 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:50.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.899+0000 7f1616600640 1 --2- 172.21.15.90:0/3532717657 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f161015bc50 0x7f16101f3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:50.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.899+0000 7f1616600640 1 --2- 172.21.15.90:0/3532717657 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f161006b600 0x7f16100af920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:50.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.899+0000 7f1616600640 1 -- 172.21.15.90:0/3532717657 >> 172.21.15.90:0/3532717657 conn(0x7f16100b1770 msgr2=0x7f1610071990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:50.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.899+0000 7f1616600640 1 -- 172.21.15.90:0/3532717657 shutdown_connections 2024-03-24T01:25:50.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:50.899+0000 7f1616600640 1 -- 172.21.15.90:0/3532717657 wait complete. 2024-03-24T01:25:50.923 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-24T01:25:50.923 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-24T01:25:50.923 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-24T01:25:50.923 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:25:50.924 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-24T01:25:50.924 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-24T01:25:50.924 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-24T01:25:50.924 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:25:50.924 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-24T01:25:50.924 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-24T01:25:50.924 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-24T01:25:50.924 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:25:50.925 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-24T01:25:50.925 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-24T01:25:50.925 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-24T01:25:50.925 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:25:50.940 INFO:tasks.ceph.mds.a:Stopped 2024-03-24T01:25:50.941 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-24T01:25:50.943 INFO:tasks.ceph.mds.c:Stopped 2024-03-24T01:25:50.943 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-24T01:25:50.944 INFO:tasks.ceph.mds.d:Stopped 2024-03-24T01:25:50.944 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-24T01:25:50.946 INFO:tasks.ceph.mds.b:Stopped 2024-03-24T01:25:50.946 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-24T01:25:50.990 INFO:tasks.ceph.mds.a:Started 2024-03-24T01:25:50.991 DEBUG:teuthology.parallel:result is None 2024-03-24T01:25:50.992 INFO:tasks.ceph.mds.d:Started 2024-03-24T01:25:50.992 DEBUG:teuthology.parallel:result is None 2024-03-24T01:25:50.997 INFO:tasks.ceph.mds.c:Started 2024-03-24T01:25:50.997 DEBUG:teuthology.parallel:result is None 2024-03-24T01:25:50.998 INFO:tasks.ceph.mds.b:Started 2024-03-24T01:25:50.998 DEBUG:teuthology.parallel:result is None 2024-03-24T01:25:50.999 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-24T01:25:51.146 INFO:tasks.ceph.mds.a.smithi090.stdout:starting mds.a at 2024-03-24T01:25:51.151 INFO:tasks.ceph.mds.c.smithi090.stdout:starting mds.c at 2024-03-24T01:25:51.151 INFO:tasks.ceph.mds.d.smithi120.stdout:starting mds.d at 2024-03-24T01:25:51.162 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2024-03-24T01:25:51.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.188+0000 7f6ee7c00640 1 Processor -- start 2024-03-24T01:25:51.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.188+0000 7f6ee7c00640 1 -- start start 2024-03-24T01:25:51.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.189+0000 7f6ee7c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6ee015dc60 0x7f6ee00b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:51.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.189+0000 7f6ee7c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ee015e040 0x7f6ee00ad820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:51.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.189+0000 7f6ee7c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ee00b5950 con 0x7f6ee006bc60 2024-03-24T01:25:51.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.189+0000 7f6ee7c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ee00b38b0 con 0x7f6ee015dc60 2024-03-24T01:25:51.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.189+0000 7f6ee7c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee00b3e70 con 0x7f6ee015e040 2024-03-24T01:25:51.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.190+0000 7f6ee5200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6ee015dc60 0x7f6ee00b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:51.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.190+0000 7f6ee5c00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f6ee006bc60 0x7f6ee00b0ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:58134/0 (socket says 172.21.15.90:58134) 2024-03-24T01:25:51.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.190+0000 7f6ee5c00640 1 -- 172.21.15.90:0/2407794109 learned_addr learned my addr 172.21.15.90:0/2407794109 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:51.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.190+0000 7f6edfe00640 1 --2- 172.21.15.90:0/2407794109 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ee015e040 0x7f6ee00ad820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:51.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.190+0000 7f6ee5200640 1 -- 172.21.15.90:0/2407794109 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ee015e040 msgr2=0x7f6ee00ad820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:51.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.190+0000 7f6ee5200640 1 --2- 172.21.15.90:0/2407794109 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ee015e040 0x7f6ee00ad820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:51.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.190+0000 7f6ee5200640 1 -- 172.21.15.90:0/2407794109 >> v1:172.21.15.90:6789/0 conn(0x7f6ee006bc60 legacy=0x7f6ee00b0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:51.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.190+0000 7f6ee5200640 1 -- 172.21.15.90:0/2407794109 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee00b5c30 con 0x7f6ee015dc60 2024-03-24T01:25:51.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.190+0000 7f6edfe00640 1 --2- 172.21.15.90:0/2407794109 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ee015e040 0x7f6ee00ad820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:25:51.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.191+0000 7f6ee5200640 1 --2- 172.21.15.90:0/2407794109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ee015dc60 0x7f6ee00b1f60 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f6ed40097a0 tx=0x7f6ed402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=704ce809d0f9ec54 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:51.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.191+0000 7f6edf400640 1 -- 172.21.15.90:0/2407794109 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed4031780 con 0x7f6ee015dc60 2024-03-24T01:25:51.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.191+0000 7f6edf400640 1 -- 172.21.15.90:0/2407794109 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed4037070 con 0x7f6ee015dc60 2024-03-24T01:25:51.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.191+0000 7f6edf400640 1 -- 172.21.15.90:0/2407794109 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed4031b40 con 0x7f6ee015dc60 2024-03-24T01:25:51.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.192+0000 7f6ee7c00640 1 -- 172.21.15.90:0/2407794109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ee015dc60 msgr2=0x7f6ee00b1f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:51.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.192+0000 7f6ee7c00640 1 --2- 172.21.15.90:0/2407794109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ee015dc60 0x7f6ee00b1f60 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f6ed40097a0 tx=0x7f6ed402b820 comp rx=0 tx=0).stop 2024-03-24T01:25:51.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.192+0000 7f6ee7c00640 1 -- 172.21.15.90:0/2407794109 shutdown_connections 2024-03-24T01:25:51.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.192+0000 7f6ee7c00640 1 --2- 172.21.15.90:0/2407794109 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ee015e040 0x7f6ee00ad820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:51.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.192+0000 7f6ee7c00640 1 --2- 172.21.15.90:0/2407794109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ee015dc60 0x7f6ee00b1f60 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:51.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.192+0000 7f6ee7c00640 1 -- 172.21.15.90:0/2407794109 >> 172.21.15.90:0/2407794109 conn(0x7f6ee006cec0 msgr2=0x7f6ee0099a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:51.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.192+0000 7f6ee7c00640 1 -- 172.21.15.90:0/2407794109 shutdown_connections 2024-03-24T01:25:51.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.192+0000 7f6ee7c00640 1 -- 172.21.15.90:0/2407794109 wait complete. 2024-03-24T01:25:51.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.194+0000 7f6ee7c00640 1 Processor -- start 2024-03-24T01:25:51.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.194+0000 7f6ee7c00640 1 -- start start 2024-03-24T01:25:51.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.195+0000 7f6ee7c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ee006bc60 0x7f6ee0158840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:51.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.195+0000 7f6ee7c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ee015dc60 0x7f6ee015bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:51.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.195+0000 7f6ee7c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ee015e040 0x7f6ee015c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:51.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.195+0000 7f6ee7c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee00b5730 con 0x7f6ee015dc60 2024-03-24T01:25:51.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.195+0000 7f6ee7c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee00b55b0 con 0x7f6ee015e040 2024-03-24T01:25:51.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.195+0000 7f6ee7c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee00b15b0 con 0x7f6ee006bc60 2024-03-24T01:25:51.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.195+0000 7f6edfe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ee015dc60 0x7f6ee015bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:51.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.195+0000 7f6edfe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ee015dc60 0x7f6ee015bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60146/0 (socket says 172.21.15.90:60146) 2024-03-24T01:25:51.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.195+0000 7f6edfe00640 1 -- 172.21.15.90:0/2622100522 learned_addr learned my addr 172.21.15.90:0/2622100522 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:51.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.195+0000 7f6ee5200640 1 --2- 172.21.15.90:0/2622100522 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ee006bc60 0x7f6ee0158840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:51.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.195+0000 7f6ee5c00640 1 --2- 172.21.15.90:0/2622100522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ee015e040 0x7f6ee015c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:51.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.196+0000 7f6edfe00640 1 -- 172.21.15.90:0/2622100522 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ee006bc60 msgr2=0x7f6ee0158840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:51.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.196+0000 7f6edfe00640 1 --2- 172.21.15.90:0/2622100522 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ee006bc60 0x7f6ee0158840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:51.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.196+0000 7f6edfe00640 1 -- 172.21.15.90:0/2622100522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ee015e040 msgr2=0x7f6ee015c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:51.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.196+0000 7f6edfe00640 1 --2- 172.21.15.90:0/2622100522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ee015e040 0x7f6ee015c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:51.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.196+0000 7f6edfe00640 1 -- 172.21.15.90:0/2622100522 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee009c420 con 0x7f6ee015dc60 2024-03-24T01:25:51.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.196+0000 7f6edfe00640 1 --2- 172.21.15.90:0/2622100522 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ee015dc60 0x7f6ee015bc90 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc00e7e0 tx=0x7f6ecc00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:51.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.196+0000 7f6edcc00640 1 -- 172.21.15.90:0/2622100522 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc00f040 con 0x7f6ee015dc60 2024-03-24T01:25:51.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.196+0000 7f6edcc00640 1 -- 172.21.15.90:0/2622100522 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ecc004030 con 0x7f6ee015dc60 2024-03-24T01:25:51.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.196+0000 7f6ee7c00640 1 -- 172.21.15.90:0/2622100522 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ee01e9360 con 0x7f6ee015dc60 2024-03-24T01:25:51.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.196+0000 7f6edcc00640 1 -- 172.21.15.90:0/2622100522 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc004400 con 0x7f6ee015dc60 2024-03-24T01:25:51.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.196+0000 7f6ee7c00640 1 -- 172.21.15.90:0/2622100522 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ee009c6c0 con 0x7f6ee015dc60 2024-03-24T01:25:51.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.197+0000 7f6ee7c00640 1 -- 172.21.15.90:0/2622100522 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ee004f400 con 0x7f6ee015dc60 2024-03-24T01:25:51.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.198+0000 7f6edcc00640 1 -- 172.21.15.90:0/2622100522 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6ecc00c6d0 con 0x7f6ee015dc60 2024-03-24T01:25:51.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.199+0000 7f6edcc00640 1 --2- 172.21.15.90:0/2622100522 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6ec006bc00 0x7f6ec006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:51.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.199+0000 7f6edcc00640 1 -- 172.21.15.90:0/2622100522 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f6ecc08a710 con 0x7f6ee015dc60 2024-03-24T01:25:51.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.200+0000 7f6ee5200640 1 --2- 172.21.15.90:0/2622100522 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6ec006bc00 0x7f6ec006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:51.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.201+0000 7f6ee5200640 1 --2- 172.21.15.90:0/2622100522 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6ec006bc00 0x7f6ec006e0c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4036000 tx=0x7f6ed403d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:51.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.204+0000 7f6edcc00640 1 -- 172.21.15.90:0/2622100522 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6ecc017020 con 0x7f6ee015dc60 2024-03-24T01:25:51.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.414+0000 7f6ee7c00640 1 -- 172.21.15.90:0/2622100522 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f6ee00741c0 con 0x7f6ee015dc60 2024-03-24T01:25:51.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.919+0000 7f6edcc00640 1 -- 172.21.15.90:0/2622100522 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v33) v1 ==== 97+0+0 (secure 0 0 0) 0x7f6ecc058bf0 con 0x7f6ee015dc60 2024-03-24T01:25:51.920 INFO:teuthology.orchestra.run.smithi090.stderr: removed all blocklist entries 2024-03-24T01:25:51.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.923+0000 7f6ee7c00640 1 -- 172.21.15.90:0/2622100522 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6ec006bc00 msgr2=0x7f6ec006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:51.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.923+0000 7f6ee7c00640 1 --2- 172.21.15.90:0/2622100522 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6ec006bc00 0x7f6ec006e0c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4036000 tx=0x7f6ed403d040 comp rx=0 tx=0).stop 2024-03-24T01:25:51.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.923+0000 7f6ee7c00640 1 -- 172.21.15.90:0/2622100522 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ee015dc60 msgr2=0x7f6ee015bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:51.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.923+0000 7f6ee7c00640 1 --2- 172.21.15.90:0/2622100522 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ee015dc60 0x7f6ee015bc90 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc00e7e0 tx=0x7f6ecc00eca0 comp rx=0 tx=0).stop 2024-03-24T01:25:51.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.924+0000 7f6ee7c00640 1 -- 172.21.15.90:0/2622100522 shutdown_connections 2024-03-24T01:25:51.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.924+0000 7f6ee7c00640 1 --2- 172.21.15.90:0/2622100522 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6ec006bc00 0x7f6ec006e0c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:51.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.924+0000 7f6ee7c00640 1 --2- 172.21.15.90:0/2622100522 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ee015dc60 0x7f6ee015bc90 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:51.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.924+0000 7f6ee7c00640 1 --2- 172.21.15.90:0/2622100522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ee015e040 0x7f6ee015c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:51.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.924+0000 7f6ee7c00640 1 --2- 172.21.15.90:0/2622100522 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6ee006bc60 0x7f6ee0158840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:51.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.924+0000 7f6ee7c00640 1 -- 172.21.15.90:0/2622100522 >> 172.21.15.90:0/2622100522 conn(0x7f6ee006cec0 msgr2=0x7f6ee0160c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:51.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.924+0000 7f6ee7c00640 1 -- 172.21.15.90:0/2622100522 shutdown_connections 2024-03-24T01:25:51.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:51.924+0000 7f6ee7c00640 1 -- 172.21.15.90:0/2622100522 wait complete. 2024-03-24T01:25:51.952 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-24T01:25:52.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.145+0000 7f5223000640 1 Processor -- start 2024-03-24T01:25:52.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.145+0000 7f5223000640 1 -- start start 2024-03-24T01:25:52.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.146+0000 7f5223000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f521c15ae10 0x7f521c0cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:52.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.146+0000 7f5223000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f521c0cd210 0x7f521c06e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:52.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.146+0000 7f5223000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f521c15b2d0 con 0x7f521c158280 2024-03-24T01:25:52.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.146+0000 7f5223000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f521c15b5b0 con 0x7f521c0cd210 2024-03-24T01:25:52.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.146+0000 7f5223000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f521c15b890 con 0x7f521c15ae10 2024-03-24T01:25:52.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.147+0000 7f5222600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f521c0cd210 0x7f521c06e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:52.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.147+0000 7f5222600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f521c0cd210 0x7f521c06e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60162/0 (socket says 172.21.15.90:60162) 2024-03-24T01:25:52.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.147+0000 7f5222600640 1 -- 172.21.15.90:0/4113211739 learned_addr learned my addr 172.21.15.90:0/4113211739 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:52.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.147+0000 7f5221200640 1 --2- 172.21.15.90:0/4113211739 >> v2:172.21.15.120:3300/0 conn(0x7f521c15ae10 0x7f521c0cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:52.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.147+0000 7f5222600640 1 -- 172.21.15.90:0/4113211739 >> v2:172.21.15.120:3300/0 conn(0x7f521c15ae10 msgr2=0x7f521c0cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:52.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.147+0000 7f5222600640 1 --2- 172.21.15.90:0/4113211739 >> v2:172.21.15.120:3300/0 conn(0x7f521c15ae10 0x7f521c0cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:52.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.147+0000 7f5222600640 1 -- 172.21.15.90:0/4113211739 >> v1:172.21.15.120:6789/0 conn(0x7f521c158280 legacy=0x7f521c0b3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:52.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.147+0000 7f5222600640 1 -- 172.21.15.90:0/4113211739 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f521c15bb70 con 0x7f521c0cd210 2024-03-24T01:25:52.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.148+0000 7f5222600640 1 --2- 172.21.15.90:0/4113211739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f521c0cd210 0x7f521c06e900 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f521c06fc10 tx=0x7f521402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2db2927e9695c63d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:52.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.149+0000 7f521be00640 1 -- 172.21.15.90:0/4113211739 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521402ca80 con 0x7f521c0cd210 2024-03-24T01:25:52.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.149+0000 7f521be00640 1 -- 172.21.15.90:0/4113211739 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521402cc20 con 0x7f521c0cd210 2024-03-24T01:25:52.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.149+0000 7f521be00640 1 -- 172.21.15.90:0/4113211739 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52140343f0 con 0x7f521c0cd210 2024-03-24T01:25:52.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.149+0000 7f5223000640 1 -- 172.21.15.90:0/4113211739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f521c0cd210 msgr2=0x7f521c06e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:52.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.149+0000 7f5223000640 1 --2- 172.21.15.90:0/4113211739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f521c0cd210 0x7f521c06e900 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f521c06fc10 tx=0x7f521402bee0 comp rx=0 tx=0).stop 2024-03-24T01:25:52.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.149+0000 7f5223000640 1 -- 172.21.15.90:0/4113211739 shutdown_connections 2024-03-24T01:25:52.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.149+0000 7f5223000640 1 --2- 172.21.15.90:0/4113211739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f521c0cd210 0x7f521c06e900 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:52.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.149+0000 7f5223000640 1 --2- 172.21.15.90:0/4113211739 >> v2:172.21.15.120:3300/0 conn(0x7f521c15ae10 0x7f521c0cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:52.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.149+0000 7f5223000640 1 -- 172.21.15.90:0/4113211739 >> 172.21.15.90:0/4113211739 conn(0x7f521c06b0c0 msgr2=0x7f521c06b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:52.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.150+0000 7f5223000640 1 -- 172.21.15.90:0/4113211739 shutdown_connections 2024-03-24T01:25:52.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.150+0000 7f5223000640 1 -- 172.21.15.90:0/4113211739 wait complete. 2024-03-24T01:25:52.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.152+0000 7f5223000640 1 Processor -- start 2024-03-24T01:25:52.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.152+0000 7f5223000640 1 -- start start 2024-03-24T01:25:52.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.153+0000 7f5223000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f521c0cd210 0x7f521c1553a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:52.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.153+0000 7f5223000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f521c158280 0x7f521c1588b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:52.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.153+0000 7f5223000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f521c15ae10 0x7f521c0cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:52.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.153+0000 7f5223000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521c06efc0 con 0x7f521c158280 2024-03-24T01:25:52.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.153+0000 7f5223000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521c06ee40 con 0x7f521c0cd210 2024-03-24T01:25:52.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.153+0000 7f5223000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f521c06f140 con 0x7f521c15ae10 2024-03-24T01:25:52.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.153+0000 7f5221200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f521c158280 0x7f521c1588b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:52.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.153+0000 7f5221200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f521c158280 0x7f521c1588b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60172/0 (socket says 172.21.15.90:60172) 2024-03-24T01:25:52.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.153+0000 7f5221200640 1 -- 172.21.15.90:0/2751069650 learned_addr learned my addr 172.21.15.90:0/2751069650 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:52.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.154+0000 7f5221200640 1 -- 172.21.15.90:0/2751069650 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f521c15ae10 msgr2=0x7f521c0cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:52.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.154+0000 7f5221200640 1 --2- 172.21.15.90:0/2751069650 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f521c15ae10 0x7f521c0cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:52.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.154+0000 7f5221200640 1 -- 172.21.15.90:0/2751069650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f521c0cd210 msgr2=0x7f521c1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:52.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.154+0000 7f5221c00640 1 --2- 172.21.15.90:0/2751069650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f521c0cd210 0x7f521c1553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:52.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.154+0000 7f5221200640 1 --2- 172.21.15.90:0/2751069650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f521c0cd210 0x7f521c1553a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:52.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.154+0000 7f5221200640 1 -- 172.21.15.90:0/2751069650 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f521c0cc170 con 0x7f521c158280 2024-03-24T01:25:52.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.154+0000 7f5221200640 1 --2- 172.21.15.90:0/2751069650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f521c158280 0x7f521c1588b0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f520800d5f0 tx=0x7f520800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:52.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.154+0000 7f5219600640 1 -- 172.21.15.90:0/2751069650 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52080125f0 con 0x7f521c158280 2024-03-24T01:25:52.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.154+0000 7f5219600640 1 -- 172.21.15.90:0/2751069650 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5208019070 con 0x7f521c158280 2024-03-24T01:25:52.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.154+0000 7f5223000640 1 -- 172.21.15.90:0/2751069650 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f521c159020 con 0x7f521c158280 2024-03-24T01:25:52.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.155+0000 7f5219600640 1 -- 172.21.15.90:0/2751069650 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52080129d0 con 0x7f521c158280 2024-03-24T01:25:52.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.155+0000 7f5223000640 1 -- 172.21.15.90:0/2751069650 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f521c1595e0 con 0x7f521c158280 2024-03-24T01:25:52.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.156+0000 7f5223000640 1 -- 172.21.15.90:0/2751069650 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f521c04f400 con 0x7f521c158280 2024-03-24T01:25:52.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.157+0000 7f5219600640 1 -- 172.21.15.90:0/2751069650 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5208012b70 con 0x7f521c158280 2024-03-24T01:25:52.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.157+0000 7f5219600640 1 --2- 172.21.15.90:0/2751069650 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f51fc06bbd0 0x7f51fc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:52.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.158+0000 7f5219600640 1 -- 172.21.15.90:0/2751069650 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f520808ad50 con 0x7f521c158280 2024-03-24T01:25:52.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.158+0000 7f5221c00640 1 --2- 172.21.15.90:0/2751069650 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f51fc06bbd0 0x7f51fc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:52.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.159+0000 7f5221c00640 1 --2- 172.21.15.90:0/2751069650 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f51fc06bbd0 0x7f51fc06e090 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5210003d90 tx=0x7f5210009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:52.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.162+0000 7f5219600640 1 -- 172.21.15.90:0/2751069650 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5208059960 con 0x7f521c158280 2024-03-24T01:25:52.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.434+0000 7f5223000640 1 -- 172.21.15.90:0/2751069650 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f521c06ea40 con 0x7f521c158280 2024-03-24T01:25:52.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.436+0000 7f5219600640 1 -- 172.21.15.90:0/2751069650 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f5208059260 con 0x7f521c158280 2024-03-24T01:25:52.436 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:25:52.436 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-24T01:25:52.436 INFO:teuthology.orchestra.run.smithi090.stdout: "auth_dump": [ 2024-03-24T01:25:52.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:25:52.437 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mds.a", 2024-03-24T01:25:52.437 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQBzgP9lEcWmMBAAtUqishOFkZW8T4Hh1Cgj2A==", 2024-03-24T01:25:52.437 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:25:52.437 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-24T01:25:52.437 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow *", 2024-03-24T01:25:52.437 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow *", 2024-03-24T01:25:52.437 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:25:52.437 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:25:52.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:25:52.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:25:52.437 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mds.b", 2024-03-24T01:25:52.437 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQB0gP9lIPZ1FRAAmeGByOKmL/VQTpWm4Y2/Tg==", 2024-03-24T01:25:52.438 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:25:52.438 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-24T01:25:52.438 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow *", 2024-03-24T01:25:52.438 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow *", 2024-03-24T01:25:52.438 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:25:52.438 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:25:52.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:25:52.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:25:52.438 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mds.c", 2024-03-24T01:25:52.438 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQB0gP9lRuOuBBAA3/3PMDba2Tlc6BqaY/499g==", 2024-03-24T01:25:52.438 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:25:52.438 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-24T01:25:52.439 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow *", 2024-03-24T01:25:52.439 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow *", 2024-03-24T01:25:52.439 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:25:52.439 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:25:52.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:25:52.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:25:52.439 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mds.d", 2024-03-24T01:25:52.439 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQB0gP9lOtsLJhAAYUVPFa1eoDHF5MV42s8e6A==", 2024-03-24T01:25:52.439 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:25:52.439 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-24T01:25:52.439 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow *", 2024-03-24T01:25:52.439 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow *", 2024-03-24T01:25:52.440 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:25:52.440 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:25:52.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:25:52.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:25:52.440 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.0", 2024-03-24T01:25:52.440 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQB5gP9lJ2jHGRAAPlZjS638Xdt6yVc+vT2ghw==", 2024-03-24T01:25:52.440 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:25:52.440 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-24T01:25:52.440 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-24T01:25:52.440 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:25:52.440 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:25:52.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:25:52.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:25:52.441 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.1", 2024-03-24T01:25:52.441 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQB8gP9lEN/pIxAAvzVfJNpo7FP6c8KSbZdy9A==", 2024-03-24T01:25:52.441 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:25:52.441 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-24T01:25:52.441 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-24T01:25:52.441 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:25:52.441 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:25:52.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:25:52.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:25:52.441 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.2", 2024-03-24T01:25:52.442 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQB/gP9lKv5mLRAAJXArmoRRVqO07jxWutfMbQ==", 2024-03-24T01:25:52.442 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:25:52.442 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-24T01:25:52.442 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-24T01:25:52.442 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:25:52.442 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:25:52.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:25:52.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:25:52.443 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.3", 2024-03-24T01:25:52.443 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQCCgP9lTZ8YNxAA375ZLJHv6H2CJghmcqfe+A==", 2024-03-24T01:25:52.443 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:25:52.443 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-24T01:25:52.443 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-24T01:25:52.443 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:25:52.443 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:25:52.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:25:52.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:25:52.443 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.4", 2024-03-24T01:25:52.443 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQCKgP9ln1YuBBAAYCNFJGs3fApnhwnY/VYHhw==", 2024-03-24T01:25:52.443 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:25:52.444 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-24T01:25:52.444 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-24T01:25:52.444 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:25:52.444 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:25:52.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:25:52.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:25:52.444 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.5", 2024-03-24T01:25:52.444 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQCNgP9l329zHBAA7VtZVd0XKEK5dzhyzcPFjQ==", 2024-03-24T01:25:52.444 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:25:52.444 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-24T01:25:52.444 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-24T01:25:52.444 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:25:52.444 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:25:52.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:25:52.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:25:52.445 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.6", 2024-03-24T01:25:52.445 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQCQgP9lIv32NBAAZN6sDvBDKUZGB+TnYHO9jA==", 2024-03-24T01:25:52.445 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:25:52.445 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-24T01:25:52.445 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-24T01:25:52.445 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:25:52.445 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:25:52.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:25:52.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:25:52.445 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.7", 2024-03-24T01:25:52.445 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQCUgP9lFpBTBBAAbPF02et1cVIhSfkvJz3R3w==", 2024-03-24T01:25:52.445 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:25:52.445 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-24T01:25:52.445 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-24T01:25:52.446 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:25:52.446 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:25:52.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:25:52.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:25:52.446 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.0", 2024-03-24T01:25:52.446 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQB0gP9lLKWXMxAAwrrRG8zLSylpIEuUgM7q8g==", 2024-03-24T01:25:52.446 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:25:52.446 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-24T01:25:52.446 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow r", 2024-03-24T01:25:52.446 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow rw", 2024-03-24T01:25:52.446 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow rwx" 2024-03-24T01:25:52.446 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:25:52.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:25:52.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:25:52.446 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.1", 2024-03-24T01:25:52.447 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQB1gP9lFlgvDRAAJFhGpBldAqv7oKFr2lIXNQ==", 2024-03-24T01:25:52.447 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:25:52.447 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-24T01:25:52.447 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow r", 2024-03-24T01:25:52.447 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow rw", 2024-03-24T01:25:52.447 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow rwx" 2024-03-24T01:25:52.447 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:25:52.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:25:52.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:25:52.447 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.2", 2024-03-24T01:25:52.447 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQB1gP9lsGI+AhAAPLfzrMVwQpf8iGAfVJX1gA==", 2024-03-24T01:25:52.447 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:25:52.447 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-24T01:25:52.447 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow r", 2024-03-24T01:25:52.448 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow rw", 2024-03-24T01:25:52.448 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow rwx" 2024-03-24T01:25:52.448 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:25:52.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:25:52.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:25:52.448 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.3", 2024-03-24T01:25:52.448 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQB1gP9l1LA0GBAAp6dwwZf+25F64cERHm2/Ew==", 2024-03-24T01:25:52.448 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:25:52.448 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-24T01:25:52.448 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow r", 2024-03-24T01:25:52.448 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow rw", 2024-03-24T01:25:52.448 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow rwx" 2024-03-24T01:25:52.448 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:25:52.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:25:52.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:25:52.449 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.admin", 2024-03-24T01:25:52.449 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQBygP9lwVRtIxAAJ7NWfOKFeLoW5nl0TLtR2w==", 2024-03-24T01:25:52.449 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:25:52.449 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow *", 2024-03-24T01:25:52.449 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow *", 2024-03-24T01:25:52.449 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow *", 2024-03-24T01:25:52.449 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:25:52.449 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:25:52.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:25:52.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:25:52.450 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.bootstrap-mds", 2024-03-24T01:25:52.450 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQCigP9lvyPQDBAASrw3nfyTxPrb4gezzU1qdA==", 2024-03-24T01:25:52.450 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:25:52.450 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile bootstrap-mds" 2024-03-24T01:25:52.450 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:25:52.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:25:52.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:25:52.450 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.bootstrap-mgr", 2024-03-24T01:25:52.450 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQCigP9lrVPQDBAAl/7T9OOqs7f9x5EybHZAWQ==", 2024-03-24T01:25:52.450 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:25:52.450 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile bootstrap-mgr" 2024-03-24T01:25:52.450 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:25:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:25:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:25:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.bootstrap-osd", 2024-03-24T01:25:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQCigP9l8X3QDBAA73CzEjXUREumkK+gt8YLmg==", 2024-03-24T01:25:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:25:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile bootstrap-osd" 2024-03-24T01:25:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:25:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:25:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:25:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.bootstrap-rbd", 2024-03-24T01:25:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQCigP9l46nQDBAAkLO79SS4fgVcOKZw7r14bw==", 2024-03-24T01:25:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:25:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile bootstrap-rbd" 2024-03-24T01:25:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:25:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:25:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:25:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-03-24T01:25:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQCigP9l+NPQDBAAoFv4qzW6bpHtx5DfoVta3A==", 2024-03-24T01:25:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:25:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-03-24T01:25:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:25:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:25:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:25:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.bootstrap-rgw", 2024-03-24T01:25:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQCigP9lXe/QDBAABEXYmYD97WZfs9OzsEPXNQ==", 2024-03-24T01:25:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:25:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile bootstrap-rgw" 2024-03-24T01:25:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:25:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:25:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:25:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mgr.x", 2024-03-24T01:25:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQBzgP9l8GPtJBAAgU+pArQqmyMDPwj0w+W/Dg==", 2024-03-24T01:25:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:25:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow *", 2024-03-24T01:25:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile mgr", 2024-03-24T01:25:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:25:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:25:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:25:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:25:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mgr.y", 2024-03-24T01:25:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQBzgP9l0KzyGBAA9zVfyPbngjHtBQyp+S37iQ==", 2024-03-24T01:25:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:25:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow *", 2024-03-24T01:25:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile mgr", 2024-03-24T01:25:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:25:52.454 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:25:52.454 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:25:52.454 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-03-24T01:25:52.454 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-24T01:25:52.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.438+0000 7f5223000640 1 -- 172.21.15.90:0/2751069650 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f51fc06bbd0 msgr2=0x7f51fc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:52.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.438+0000 7f5223000640 1 --2- 172.21.15.90:0/2751069650 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f51fc06bbd0 0x7f51fc06e090 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5210003d90 tx=0x7f5210009040 comp rx=0 tx=0).stop 2024-03-24T01:25:52.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.439+0000 7f5223000640 1 -- 172.21.15.90:0/2751069650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f521c158280 msgr2=0x7f521c1588b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:52.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.439+0000 7f5223000640 1 --2- 172.21.15.90:0/2751069650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f521c158280 0x7f521c1588b0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f520800d5f0 tx=0x7f520800dab0 comp rx=0 tx=0).stop 2024-03-24T01:25:52.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.439+0000 7f5223000640 1 -- 172.21.15.90:0/2751069650 shutdown_connections 2024-03-24T01:25:52.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.439+0000 7f5223000640 1 --2- 172.21.15.90:0/2751069650 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f51fc06bbd0 0x7f51fc06e090 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:52.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.439+0000 7f5223000640 1 --2- 172.21.15.90:0/2751069650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f521c158280 0x7f521c1588b0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:52.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.439+0000 7f5223000640 1 --2- 172.21.15.90:0/2751069650 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f521c15ae10 0x7f521c0cbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:52.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.439+0000 7f5223000640 1 --2- 172.21.15.90:0/2751069650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f521c0cd210 0x7f521c1553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:52.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.439+0000 7f5223000640 1 -- 172.21.15.90:0/2751069650 >> 172.21.15.90:0/2751069650 conn(0x7f521c06b0c0 msgr2=0x7f521c15d810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:52.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.439+0000 7f5223000640 1 -- 172.21.15.90:0/2751069650 shutdown_connections 2024-03-24T01:25:52.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.439+0000 7f5223000640 1 -- 172.21.15.90:0/2751069650 wait complete. 2024-03-24T01:25:52.457 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-03-24T01:25:52.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.661+0000 7f6356c00640 1 Processor -- start 2024-03-24T01:25:52.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.662+0000 7f6356c00640 1 -- start start 2024-03-24T01:25:52.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.663+0000 7f6356c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6350099b50 0x7f63500674a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:52.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.663+0000 7f6356c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f63500ad450 0x7f63500ad8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:52.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.663+0000 7f6356c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63500cfff0 con 0x7f63500af970 2024-03-24T01:25:52.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.663+0000 7f6356c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f635015e3e0 con 0x7f63500ad450 2024-03-24T01:25:52.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.663+0000 7f6356c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63500d02d0 con 0x7f6350099b50 2024-03-24T01:25:52.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.664+0000 7f634fe00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f63500af970 0x7f63500afd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:58136/0 (socket says 172.21.15.90:58136) 2024-03-24T01:25:52.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.664+0000 7f634fe00640 1 -- 172.21.15.90:0/964101605 learned_addr learned my addr 172.21.15.90:0/964101605 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:52.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.664+0000 7f6354c00640 1 --2- 172.21.15.90:0/964101605 >> v2:172.21.15.90:3300/0 conn(0x7f63500ad450 0x7f63500ad8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:52.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.664+0000 7f634f400640 1 --2- 172.21.15.90:0/964101605 >> v2:172.21.15.120:3300/0 conn(0x7f6350099b50 0x7f63500674a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:52.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.664+0000 7f6354c00640 1 -- 172.21.15.90:0/964101605 >> v2:172.21.15.120:3300/0 conn(0x7f6350099b50 msgr2=0x7f63500674a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:52.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.664+0000 7f6354c00640 1 --2- 172.21.15.90:0/964101605 >> v2:172.21.15.120:3300/0 conn(0x7f6350099b50 0x7f63500674a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:52.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.664+0000 7f6354c00640 1 -- 172.21.15.90:0/964101605 >> v1:172.21.15.90:6789/0 conn(0x7f63500af970 legacy=0x7f63500afd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:52.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.664+0000 7f6354c00640 1 -- 172.21.15.90:0/964101605 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63500d05b0 con 0x7f63500ad450 2024-03-24T01:25:52.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.665+0000 7f6354c00640 1 --2- 172.21.15.90:0/964101605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63500ad450 0x7f63500ad8c0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f63500b0a80 tx=0x7f634802bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6c5a249573002f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:52.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.666+0000 7f634ea00640 1 -- 172.21.15.90:0/964101605 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6348039070 con 0x7f63500ad450 2024-03-24T01:25:52.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.666+0000 7f634ea00640 1 -- 172.21.15.90:0/964101605 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f634802cc70 con 0x7f63500ad450 2024-03-24T01:25:52.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.666+0000 7f634ea00640 1 -- 172.21.15.90:0/964101605 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6348034660 con 0x7f63500ad450 2024-03-24T01:25:52.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.666+0000 7f6356c00640 1 -- 172.21.15.90:0/964101605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63500ad450 msgr2=0x7f63500ad8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:52.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.666+0000 7f6356c00640 1 --2- 172.21.15.90:0/964101605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63500ad450 0x7f63500ad8c0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f63500b0a80 tx=0x7f634802bec0 comp rx=0 tx=0).stop 2024-03-24T01:25:52.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.666+0000 7f6356c00640 1 -- 172.21.15.90:0/964101605 shutdown_connections 2024-03-24T01:25:52.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.666+0000 7f6356c00640 1 --2- 172.21.15.90:0/964101605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63500ad450 0x7f63500ad8c0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:52.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.666+0000 7f6356c00640 1 --2- 172.21.15.90:0/964101605 >> v2:172.21.15.120:3300/0 conn(0x7f6350099b50 0x7f63500674a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:52.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.666+0000 7f6356c00640 1 -- 172.21.15.90:0/964101605 >> 172.21.15.90:0/964101605 conn(0x7f63500b3c90 msgr2=0x7f63500b4ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:52.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.666+0000 7f6356c00640 1 -- 172.21.15.90:0/964101605 shutdown_connections 2024-03-24T01:25:52.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.666+0000 7f6356c00640 1 -- 172.21.15.90:0/964101605 wait complete. 2024-03-24T01:25:52.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.668+0000 7f6356c00640 1 Processor -- start 2024-03-24T01:25:52.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.668+0000 7f6356c00640 1 -- start start 2024-03-24T01:25:52.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.669+0000 7f6356c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6350099b50 0x7f6350152920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:52.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.669+0000 7f6356c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63500ad450 0x7f635009a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:52.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.669+0000 7f6356c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63500af970 0x7f635009dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:52.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.669+0000 7f6356c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63500ade00 con 0x7f63500ad450 2024-03-24T01:25:52.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.669+0000 7f6356c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63500d0b70 con 0x7f63500af970 2024-03-24T01:25:52.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.669+0000 7f6356c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63500adf80 con 0x7f6350099b50 2024-03-24T01:25:52.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.669+0000 7f634f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63500ad450 0x7f635009a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:52.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.670+0000 7f634f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63500ad450 0x7f635009a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60204/0 (socket says 172.21.15.90:60204) 2024-03-24T01:25:52.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.670+0000 7f634f400640 1 -- 172.21.15.90:0/176494552 learned_addr learned my addr 172.21.15.90:0/176494552 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:52.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.670+0000 7f6354c00640 1 --2- 172.21.15.90:0/176494552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63500af970 0x7f635009dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:52.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.670+0000 7f634f400640 1 -- 172.21.15.90:0/176494552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6350099b50 msgr2=0x7f6350152920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:52.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.670+0000 7f634fe00640 1 --2- 172.21.15.90:0/176494552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6350099b50 0x7f6350152920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:52.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.670+0000 7f634f400640 1 --2- 172.21.15.90:0/176494552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6350099b50 0x7f6350152920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:52.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.670+0000 7f634f400640 1 -- 172.21.15.90:0/176494552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63500af970 msgr2=0x7f635009dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:52.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.670+0000 7f634f400640 1 --2- 172.21.15.90:0/176494552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63500af970 0x7f635009dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:52.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.670+0000 7f634f400640 1 -- 172.21.15.90:0/176494552 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6350152e60 con 0x7f63500ad450 2024-03-24T01:25:52.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.670+0000 7f6354c00640 1 --2- 172.21.15.90:0/176494552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63500af970 0x7f635009dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:52.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.670+0000 7f634fe00640 1 --2- 172.21.15.90:0/176494552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6350099b50 0x7f6350152920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:52.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.670+0000 7f634f400640 1 --2- 172.21.15.90:0/176494552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63500ad450 0x7f635009a1f0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f633c00d800 tx=0x7f633c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:52.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.671+0000 7f6347e00640 1 -- 172.21.15.90:0/176494552 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f633c00b6c0 con 0x7f63500ad450 2024-03-24T01:25:52.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.671+0000 7f6347e00640 1 -- 172.21.15.90:0/176494552 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f633c019070 con 0x7f63500ad450 2024-03-24T01:25:52.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.671+0000 7f6347e00640 1 -- 172.21.15.90:0/176494552 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f633c00ba80 con 0x7f63500ad450 2024-03-24T01:25:52.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.671+0000 7f6356c00640 1 -- 172.21.15.90:0/176494552 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6350157de0 con 0x7f63500ad450 2024-03-24T01:25:52.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.671+0000 7f6356c00640 1 -- 172.21.15.90:0/176494552 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63501583a0 con 0x7f63500ad450 2024-03-24T01:25:52.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.672+0000 7f6356c00640 1 -- 172.21.15.90:0/176494552 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f635004f400 con 0x7f63500ad450 2024-03-24T01:25:52.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.673+0000 7f6347e00640 1 -- 172.21.15.90:0/176494552 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f633c00bc20 con 0x7f63500ad450 2024-03-24T01:25:52.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.674+0000 7f6347e00640 1 --2- 172.21.15.90:0/176494552 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f633006bbd0 0x7f633006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:52.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.674+0000 7f6347e00640 1 -- 172.21.15.90:0/176494552 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f633c08acf0 con 0x7f63500ad450 2024-03-24T01:25:52.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.674+0000 7f634fe00640 1 --2- 172.21.15.90:0/176494552 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f633006bbd0 0x7f633006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:52.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.675+0000 7f634fe00640 1 --2- 172.21.15.90:0/176494552 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f633006bbd0 0x7f633006e090 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6340000f60 tx=0x7f6340009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:52.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.678+0000 7f6347e00640 1 -- 172.21.15.90:0/176494552 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f633c08f070 con 0x7f63500ad450 2024-03-24T01:25:52.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.948+0000 7f6356c00640 1 -- 172.21.15.90:0/176494552 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f63500d0f90 con 0x7f63500ad450 2024-03-24T01:25:52.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.952+0000 7f6347e00640 1 -- 172.21.15.90:0/176494552 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7f633c059a30 con 0x7f63500ad450 2024-03-24T01:25:52.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.954+0000 7f6356c00640 1 -- 172.21.15.90:0/176494552 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f633006bbd0 msgr2=0x7f633006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:52.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.954+0000 7f6356c00640 1 --2- 172.21.15.90:0/176494552 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f633006bbd0 0x7f633006e090 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6340000f60 tx=0x7f6340009040 comp rx=0 tx=0).stop 2024-03-24T01:25:52.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.955+0000 7f6356c00640 1 -- 172.21.15.90:0/176494552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63500ad450 msgr2=0x7f635009a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:52.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.955+0000 7f6356c00640 1 --2- 172.21.15.90:0/176494552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63500ad450 0x7f635009a1f0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f633c00d800 tx=0x7f633c00dcc0 comp rx=0 tx=0).stop 2024-03-24T01:25:52.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.955+0000 7f6356c00640 1 -- 172.21.15.90:0/176494552 shutdown_connections 2024-03-24T01:25:52.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.955+0000 7f6356c00640 1 --2- 172.21.15.90:0/176494552 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f633006bbd0 0x7f633006e090 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:52.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.955+0000 7f6356c00640 1 --2- 172.21.15.90:0/176494552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63500ad450 0x7f635009a1f0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:52.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.955+0000 7f6356c00640 1 --2- 172.21.15.90:0/176494552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63500af970 0x7f635009dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:52.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.955+0000 7f6356c00640 1 --2- 172.21.15.90:0/176494552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6350099b50 0x7f6350152920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:52.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.955+0000 7f6356c00640 1 -- 172.21.15.90:0/176494552 >> 172.21.15.90:0/176494552 conn(0x7f63500b3c90 msgr2=0x7f6350153e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:52.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.955+0000 7f6356c00640 1 -- 172.21.15.90:0/176494552 shutdown_connections 2024-03-24T01:25:52.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:52.955+0000 7f6356c00640 1 -- 172.21.15.90:0/176494552 wait complete. 2024-03-24T01:25:52.973 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-03-24T01:25:53.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.169+0000 7f7a78000640 1 Processor -- start 2024-03-24T01:25:53.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.169+0000 7f7a78000640 1 -- start start 2024-03-24T01:25:53.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.170+0000 7f7a78000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7a70159620 0x7f7a7006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:53.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.170+0000 7f7a78000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7a7006b600 0x7f7a700cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:53.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.171+0000 7f7a78000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a7006bca0 con 0x7f7a700cef80 2024-03-24T01:25:53.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.171+0000 7f7a78000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a700b4470 con 0x7f7a7006b600 2024-03-24T01:25:53.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.171+0000 7f7a78000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a700b36f0 con 0x7f7a70159620 2024-03-24T01:25:53.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.171+0000 7f7a74c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7a7006b600 0x7f7a700cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:53.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.171+0000 7f7a74c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7a7006b600 0x7f7a700cea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60214/0 (socket says 172.21.15.90:60214) 2024-03-24T01:25:53.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.171+0000 7f7a74c00640 1 -- 172.21.15.90:0/3550043271 learned_addr learned my addr 172.21.15.90:0/3550043271 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:53.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.171+0000 7f7a75600640 1 --2- 172.21.15.90:0/3550043271 >> v2:172.21.15.120:3300/0 conn(0x7f7a70159620 0x7f7a7006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:53.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.171+0000 7f7a74c00640 1 -- 172.21.15.90:0/3550043271 >> v2:172.21.15.120:3300/0 conn(0x7f7a70159620 msgr2=0x7f7a7006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:53.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.171+0000 7f7a74c00640 1 --2- 172.21.15.90:0/3550043271 >> v2:172.21.15.120:3300/0 conn(0x7f7a70159620 0x7f7a7006b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:53.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.171+0000 7f7a74c00640 1 -- 172.21.15.90:0/3550043271 >> v1:172.21.15.120:6789/0 conn(0x7f7a700cef80 legacy=0x7f7a700ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:53.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.171+0000 7f7a74c00640 1 -- 172.21.15.90:0/3550043271 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a700b3990 con 0x7f7a7006b600 2024-03-24T01:25:53.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.172+0000 7f7a74c00640 1 --2- 172.21.15.90:0/3550043271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a7006b600 0x7f7a700cea40 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f7a7006c2d0 tx=0x7f7a5c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23cd3e4660327a41 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:53.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.173+0000 7f7a6fe00640 1 -- 172.21.15.90:0/3550043271 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a5c02e070 con 0x7f7a7006b600 2024-03-24T01:25:53.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.173+0000 7f7a6fe00640 1 -- 172.21.15.90:0/3550043271 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a5c037070 con 0x7f7a7006b600 2024-03-24T01:25:53.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.173+0000 7f7a6fe00640 1 -- 172.21.15.90:0/3550043271 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a5c031a00 con 0x7f7a7006b600 2024-03-24T01:25:53.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.173+0000 7f7a78000640 1 -- 172.21.15.90:0/3550043271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a7006b600 msgr2=0x7f7a700cea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:53.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.173+0000 7f7a78000640 1 --2- 172.21.15.90:0/3550043271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a7006b600 0x7f7a700cea40 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f7a7006c2d0 tx=0x7f7a5c02b880 comp rx=0 tx=0).stop 2024-03-24T01:25:53.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.173+0000 7f7a78000640 1 -- 172.21.15.90:0/3550043271 shutdown_connections 2024-03-24T01:25:53.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.173+0000 7f7a78000640 1 --2- 172.21.15.90:0/3550043271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a7006b600 0x7f7a700cea40 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:53.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.173+0000 7f7a78000640 1 --2- 172.21.15.90:0/3550043271 >> v2:172.21.15.120:3300/0 conn(0x7f7a70159620 0x7f7a7006b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:53.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.173+0000 7f7a78000640 1 -- 172.21.15.90:0/3550043271 >> 172.21.15.90:0/3550043271 conn(0x7f7a700b1770 msgr2=0x7f7a700b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:53.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.174+0000 7f7a78000640 1 -- 172.21.15.90:0/3550043271 shutdown_connections 2024-03-24T01:25:53.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.174+0000 7f7a78000640 1 -- 172.21.15.90:0/3550043271 wait complete. 2024-03-24T01:25:53.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.175+0000 7f7a78000640 1 Processor -- start 2024-03-24T01:25:53.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.175+0000 7f7a78000640 1 -- start start 2024-03-24T01:25:53.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.176+0000 7f7a78000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a7006b600 0x7f7a70145e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:53.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.176+0000 7f7a78000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a700cef80 0x7f7a701e8f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:53.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.176+0000 7f7a78000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a70159620 0x7f7a7006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:53.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.176+0000 7f7a78000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a70159a00 con 0x7f7a70159620 2024-03-24T01:25:53.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.176+0000 7f7a78000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a700b3f50 con 0x7f7a700cef80 2024-03-24T01:25:53.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.176+0000 7f7a78000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a700cf3f0 con 0x7f7a7006b600 2024-03-24T01:25:53.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.176+0000 7f7a76000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a70159620 0x7f7a7006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:53.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.177+0000 7f7a76000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a70159620 0x7f7a7006e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60224/0 (socket says 172.21.15.90:60224) 2024-03-24T01:25:53.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.177+0000 7f7a76000640 1 -- 172.21.15.90:0/3318768651 learned_addr learned my addr 172.21.15.90:0/3318768651 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:53.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.177+0000 7f7a74c00640 1 --2- 172.21.15.90:0/3318768651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a700cef80 0x7f7a701e8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:53.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.177+0000 7f7a76000640 1 -- 172.21.15.90:0/3318768651 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a7006b600 msgr2=0x7f7a70145e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:53.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.177+0000 7f7a76000640 1 --2- 172.21.15.90:0/3318768651 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a7006b600 0x7f7a70145e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:53.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.177+0000 7f7a76000640 1 -- 172.21.15.90:0/3318768651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a700cef80 msgr2=0x7f7a701e8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:53.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.177+0000 7f7a76000640 1 --2- 172.21.15.90:0/3318768651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a700cef80 0x7f7a701e8f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:53.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.177+0000 7f7a76000640 1 -- 172.21.15.90:0/3318768651 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a70145180 con 0x7f7a70159620 2024-03-24T01:25:53.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.177+0000 7f7a76000640 1 --2- 172.21.15.90:0/3318768651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a70159620 0x7f7a7006e8f0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f7a6800c780 tx=0x7f7a6800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:53.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.178+0000 7f7a6d600640 1 -- 172.21.15.90:0/3318768651 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a68004b60 con 0x7f7a70159620 2024-03-24T01:25:53.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.178+0000 7f7a6d600640 1 -- 172.21.15.90:0/3318768651 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a68004d00 con 0x7f7a70159620 2024-03-24T01:25:53.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.178+0000 7f7a6d600640 1 -- 172.21.15.90:0/3318768651 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6800aa90 con 0x7f7a70159620 2024-03-24T01:25:53.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.178+0000 7f7a78000640 1 -- 172.21.15.90:0/3318768651 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a701464e0 con 0x7f7a70159620 2024-03-24T01:25:53.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.178+0000 7f7a78000640 1 -- 172.21.15.90:0/3318768651 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a70146d20 con 0x7f7a70159620 2024-03-24T01:25:53.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.179+0000 7f7a78000640 1 -- 172.21.15.90:0/3318768651 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a7004f400 con 0x7f7a70159620 2024-03-24T01:25:53.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.180+0000 7f7a6d600640 1 -- 172.21.15.90:0/3318768651 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7a6800ac30 con 0x7f7a70159620 2024-03-24T01:25:53.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.180+0000 7f7a6d600640 1 --2- 172.21.15.90:0/3318768651 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f7a5006bbd0 0x7f7a5006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:53.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.181+0000 7f7a6d600640 1 -- 172.21.15.90:0/3318768651 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f7a6808abd0 con 0x7f7a70159620 2024-03-24T01:25:53.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.181+0000 7f7a75600640 1 --2- 172.21.15.90:0/3318768651 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f7a5006bbd0 0x7f7a5006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:53.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.182+0000 7f7a75600640 1 --2- 172.21.15.90:0/3318768651 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f7a5006bbd0 0x7f7a5006e090 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7a64006fd0 tx=0x7f7a64008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:53.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.185+0000 7f7a6d600640 1 -- 172.21.15.90:0/3318768651 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7a68020070 con 0x7f7a70159620 2024-03-24T01:25:53.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.453+0000 7f7a78000640 1 -- 172.21.15.90:0/3318768651 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f7a7006de80 con 0x7f7a70159620 2024-03-24T01:25:53.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.457+0000 7f7a6d600640 1 -- 172.21.15.90:0/3318768651 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7f7a701464e0 con 0x7f7a70159620 2024-03-24T01:25:53.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.459+0000 7f7a78000640 1 -- 172.21.15.90:0/3318768651 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f7a5006bbd0 msgr2=0x7f7a5006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:53.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.459+0000 7f7a78000640 1 --2- 172.21.15.90:0/3318768651 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f7a5006bbd0 0x7f7a5006e090 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7a64006fd0 tx=0x7f7a64008040 comp rx=0 tx=0).stop 2024-03-24T01:25:53.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.459+0000 7f7a78000640 1 -- 172.21.15.90:0/3318768651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a70159620 msgr2=0x7f7a7006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:53.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.459+0000 7f7a78000640 1 --2- 172.21.15.90:0/3318768651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a70159620 0x7f7a7006e8f0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f7a6800c780 tx=0x7f7a6800cc40 comp rx=0 tx=0).stop 2024-03-24T01:25:53.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.459+0000 7f7a78000640 1 -- 172.21.15.90:0/3318768651 shutdown_connections 2024-03-24T01:25:53.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.459+0000 7f7a78000640 1 --2- 172.21.15.90:0/3318768651 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f7a5006bbd0 0x7f7a5006e090 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:53.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.459+0000 7f7a78000640 1 --2- 172.21.15.90:0/3318768651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a70159620 0x7f7a7006e8f0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:53.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.459+0000 7f7a78000640 1 --2- 172.21.15.90:0/3318768651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7a700cef80 0x7f7a701e8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:53.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.459+0000 7f7a78000640 1 --2- 172.21.15.90:0/3318768651 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7a7006b600 0x7f7a70145e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:53.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.459+0000 7f7a78000640 1 -- 172.21.15.90:0/3318768651 >> 172.21.15.90:0/3318768651 conn(0x7f7a700b1770 msgr2=0x7f7a7015d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:53.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.460+0000 7f7a78000640 1 -- 172.21.15.90:0/3318768651 shutdown_connections 2024-03-24T01:25:53.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.460+0000 7f7a78000640 1 -- 172.21.15.90:0/3318768651 wait complete. 2024-03-24T01:25:53.478 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-03-24T01:25:53.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.683+0000 7f5642c00640 1 Processor -- start 2024-03-24T01:25:53.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.683+0000 7f5642c00640 1 -- start start 2024-03-24T01:25:53.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.684+0000 7f5642c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f563c0b41b0 0x7f563c15bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:53.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.685+0000 7f5642c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f563c06f5c0 0x7f563c0ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:53.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.685+0000 7f5642c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f563c155480 con 0x7f563c1550a0 2024-03-24T01:25:53.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.685+0000 7f5642c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f563c06fcc0 con 0x7f563c0b41b0 2024-03-24T01:25:53.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.685+0000 7f5642c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f563c15f4a0 con 0x7f563c06f5c0 2024-03-24T01:25:53.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.685+0000 7f563b400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f563c0b41b0 0x7f563c15bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:53.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.685+0000 7f563be00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f563c1550a0 0x7f563c0b3aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:58148/0 (socket says 172.21.15.90:58148) 2024-03-24T01:25:53.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.685+0000 7f563be00640 1 -- 172.21.15.90:0/2934279032 learned_addr learned my addr 172.21.15.90:0/2934279032 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:53.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.685+0000 7f563b400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f563c0b41b0 0x7f563c15bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60240/0 (socket says 172.21.15.90:60240) 2024-03-24T01:25:53.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.685+0000 7f563b400640 1 -- 172.21.15.90:0/2934279032 >> v2:172.21.15.120:3300/0 conn(0x7f563c06f5c0 msgr2=0x7f563c0ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:53.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.685+0000 7f563b400640 1 --2- 172.21.15.90:0/2934279032 >> v2:172.21.15.120:3300/0 conn(0x7f563c06f5c0 0x7f563c0ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:53.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.685+0000 7f563b400640 1 -- 172.21.15.90:0/2934279032 >> v1:172.21.15.90:6789/0 conn(0x7f563c1550a0 legacy=0x7f563c0b3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:53.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.685+0000 7f563b400640 1 -- 172.21.15.90:0/2934279032 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f563c06f9f0 con 0x7f563c0b41b0 2024-03-24T01:25:53.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.685+0000 7f563aa00640 1 -- 172.21.15.90:0/2934279032 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2142507660 0 0) 0x7f563c155480 con 0x7f563c1550a0 2024-03-24T01:25:53.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.686+0000 7f563b400640 1 --2- 172.21.15.90:0/2934279032 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f563c0b41b0 0x7f563c15bb20 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f563c158bb0 tx=0x7f562802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d75f5ff10fad7c12 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:53.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.687+0000 7f563aa00640 1 -- 172.21.15.90:0/2934279032 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56280315a0 con 0x7f563c0b41b0 2024-03-24T01:25:53.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.687+0000 7f563aa00640 1 -- 172.21.15.90:0/2934279032 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5628037070 con 0x7f563c0b41b0 2024-03-24T01:25:53.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.687+0000 7f563aa00640 1 -- 172.21.15.90:0/2934279032 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5628031960 con 0x7f563c0b41b0 2024-03-24T01:25:53.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.687+0000 7f5642c00640 1 -- 172.21.15.90:0/2934279032 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f563c0b41b0 msgr2=0x7f563c15bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:53.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.687+0000 7f5642c00640 1 --2- 172.21.15.90:0/2934279032 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f563c0b41b0 0x7f563c15bb20 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f563c158bb0 tx=0x7f562802b840 comp rx=0 tx=0).stop 2024-03-24T01:25:53.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.687+0000 7f5642c00640 1 -- 172.21.15.90:0/2934279032 shutdown_connections 2024-03-24T01:25:53.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.687+0000 7f5642c00640 1 --2- 172.21.15.90:0/2934279032 >> v2:172.21.15.120:3300/0 conn(0x7f563c06f5c0 0x7f563c0ae530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:53.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.687+0000 7f5642c00640 1 --2- 172.21.15.90:0/2934279032 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f563c0b41b0 0x7f563c15bb20 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:53.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.687+0000 7f5642c00640 1 -- 172.21.15.90:0/2934279032 >> 172.21.15.90:0/2934279032 conn(0x7f563c0cf7e0 msgr2=0x7f563c0cfbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:53.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.687+0000 7f5642c00640 1 -- 172.21.15.90:0/2934279032 shutdown_connections 2024-03-24T01:25:53.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.687+0000 7f5642c00640 1 -- 172.21.15.90:0/2934279032 wait complete. 2024-03-24T01:25:53.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.689+0000 7f5642c00640 1 Processor -- start 2024-03-24T01:25:53.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.689+0000 7f5642c00640 1 -- start start 2024-03-24T01:25:53.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.690+0000 7f5642c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f563c06f5c0 0x7f563c15c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:53.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.690+0000 7f5642c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f563c0b41b0 0x7f563c1450f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:53.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.690+0000 7f5642c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f563c1550a0 0x7f563c157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:53.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.690+0000 7f5642c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563c15eca0 con 0x7f563c1550a0 2024-03-24T01:25:53.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.690+0000 7f5642c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563c15e9c0 con 0x7f563c06f5c0 2024-03-24T01:25:53.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.690+0000 7f5642c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f563c15ef80 con 0x7f563c0b41b0 2024-03-24T01:25:53.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.691+0000 7f5640c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f563c1550a0 0x7f563c157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:53.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.691+0000 7f5640c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f563c1550a0 0x7f563c157dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60254/0 (socket says 172.21.15.90:60254) 2024-03-24T01:25:53.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.691+0000 7f5640c00640 1 -- 172.21.15.90:0/1827204256 learned_addr learned my addr 172.21.15.90:0/1827204256 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:53.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.691+0000 7f563be00640 1 --2- 172.21.15.90:0/1827204256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f563c06f5c0 0x7f563c15c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:53.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.691+0000 7f563b400640 1 --2- 172.21.15.90:0/1827204256 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f563c0b41b0 0x7f563c1450f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:53.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.691+0000 7f5640c00640 1 -- 172.21.15.90:0/1827204256 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f563c0b41b0 msgr2=0x7f563c1450f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:53.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.691+0000 7f5640c00640 1 --2- 172.21.15.90:0/1827204256 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f563c0b41b0 0x7f563c1450f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:53.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.691+0000 7f5640c00640 1 -- 172.21.15.90:0/1827204256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f563c06f5c0 msgr2=0x7f563c15c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:53.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.691+0000 7f5640c00640 1 --2- 172.21.15.90:0/1827204256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f563c06f5c0 0x7f563c15c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:53.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.691+0000 7f5640c00640 1 -- 172.21.15.90:0/1827204256 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f563c15cb20 con 0x7f563c1550a0 2024-03-24T01:25:53.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.691+0000 7f563be00640 1 --2- 172.21.15.90:0/1827204256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f563c06f5c0 0x7f563c15c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:53.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.691+0000 7f5640c00640 1 --2- 172.21.15.90:0/1827204256 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f563c1550a0 0x7f563c157dc0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f563400b970 tx=0x7f563400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:53.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.692+0000 7f5633e00640 1 -- 172.21.15.90:0/1827204256 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5634019070 con 0x7f563c1550a0 2024-03-24T01:25:53.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.692+0000 7f5633e00640 1 -- 172.21.15.90:0/1827204256 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563400cb60 con 0x7f563c1550a0 2024-03-24T01:25:53.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.692+0000 7f5642c00640 1 -- 172.21.15.90:0/1827204256 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f563c15cdf0 con 0x7f563c1550a0 2024-03-24T01:25:53.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.692+0000 7f5633e00640 1 -- 172.21.15.90:0/1827204256 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5634014440 con 0x7f563c1550a0 2024-03-24T01:25:53.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.692+0000 7f5642c00640 1 -- 172.21.15.90:0/1827204256 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f563c15d3b0 con 0x7f563c1550a0 2024-03-24T01:25:53.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.693+0000 7f5642c00640 1 -- 172.21.15.90:0/1827204256 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f563c04f400 con 0x7f563c1550a0 2024-03-24T01:25:53.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.694+0000 7f5633e00640 1 -- 172.21.15.90:0/1827204256 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f563401c070 con 0x7f563c1550a0 2024-03-24T01:25:53.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.694+0000 7f5633e00640 1 --2- 172.21.15.90:0/1827204256 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f561c06bbd0 0x7f561c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:53.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.695+0000 7f5633e00640 1 -- 172.21.15.90:0/1827204256 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f563408a610 con 0x7f563c1550a0 2024-03-24T01:25:53.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.695+0000 7f563be00640 1 --2- 172.21.15.90:0/1827204256 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f561c06bbd0 0x7f561c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:53.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.696+0000 7f563be00640 1 --2- 172.21.15.90:0/1827204256 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f561c06bbd0 0x7f561c06e090 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f562c000f60 tx=0x7f562c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:53.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.699+0000 7f5633e00640 1 -- 172.21.15.90:0/1827204256 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f56340592c0 con 0x7f563c1550a0 2024-03-24T01:25:53.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.976+0000 7f5642c00640 1 -- 172.21.15.90:0/1827204256 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f563c0cf3e0 con 0x7f563c1550a0 2024-03-24T01:25:53.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.980+0000 7f5633e00640 1 -- 172.21.15.90:0/1827204256 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7f5634058bc0 con 0x7f563c1550a0 2024-03-24T01:25:53.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.984+0000 7f5642c00640 1 -- 172.21.15.90:0/1827204256 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f561c06bbd0 msgr2=0x7f561c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:53.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.984+0000 7f5642c00640 1 --2- 172.21.15.90:0/1827204256 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f561c06bbd0 0x7f561c06e090 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f562c000f60 tx=0x7f562c009040 comp rx=0 tx=0).stop 2024-03-24T01:25:53.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.984+0000 7f5642c00640 1 -- 172.21.15.90:0/1827204256 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f563c1550a0 msgr2=0x7f563c157dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:53.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.984+0000 7f5642c00640 1 --2- 172.21.15.90:0/1827204256 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f563c1550a0 0x7f563c157dc0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f563400b970 tx=0x7f563400be30 comp rx=0 tx=0).stop 2024-03-24T01:25:53.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.984+0000 7f5642c00640 1 -- 172.21.15.90:0/1827204256 shutdown_connections 2024-03-24T01:25:53.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.984+0000 7f5642c00640 1 --2- 172.21.15.90:0/1827204256 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f561c06bbd0 0x7f561c06e090 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:53.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.984+0000 7f5642c00640 1 --2- 172.21.15.90:0/1827204256 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f563c1550a0 0x7f563c157dc0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:53.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.984+0000 7f5642c00640 1 --2- 172.21.15.90:0/1827204256 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f563c0b41b0 0x7f563c1450f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:53.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.984+0000 7f5642c00640 1 --2- 172.21.15.90:0/1827204256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f563c06f5c0 0x7f563c15c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:53.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.984+0000 7f5642c00640 1 -- 172.21.15.90:0/1827204256 >> 172.21.15.90:0/1827204256 conn(0x7f563c0cf7e0 msgr2=0x7f563c06d990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:53.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.984+0000 7f5642c00640 1 -- 172.21.15.90:0/1827204256 shutdown_connections 2024-03-24T01:25:53.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:53.984+0000 7f5642c00640 1 -- 172.21.15.90:0/1827204256 wait complete. 2024-03-24T01:25:54.004 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-03-24T01:25:54.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.216+0000 7f4063200640 1 Processor -- start 2024-03-24T01:25:54.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.216+0000 7f4063200640 1 -- start start 2024-03-24T01:25:54.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.217+0000 7f4063200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4064144680 0x7f406406d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:54.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.217+0000 7f4063200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f406406e0f0 con 0x7f4064071ea0 2024-03-24T01:25:54.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.217+0000 7f4063200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f406406e3d0 con 0x7f4064145190 2024-03-24T01:25:54.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.217+0000 7f4063200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4064144a60 con 0x7f4064144680 2024-03-24T01:25:54.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.218+0000 7f4062800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4064071ea0 0x7f40640cc900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:58160/0 (socket says 172.21.15.90:58160) 2024-03-24T01:25:54.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.218+0000 7f4062800640 1 -- 172.21.15.90:0/213599893 learned_addr learned my addr 172.21.15.90:0/213599893 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:25:54.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.218+0000 7f4061e00640 1 --2- 172.21.15.90:0/213599893 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4064144680 0x7f406406d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:54.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.218+0000 7f4061400640 1 -- 172.21.15.90:0/213599893 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2852087206 0 0) 0x7f406406e0f0 con 0x7f4064071ea0 2024-03-24T01:25:54.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.218+0000 7f4061400640 1 -- 172.21.15.90:0/213599893 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4048003520 con 0x7f4064071ea0 2024-03-24T01:25:54.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.219+0000 7f4061400640 1 -- 172.21.15.90:0/213599893 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3012384918 0 0) 0x7f4048003520 con 0x7f4064071ea0 2024-03-24T01:25:54.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.219+0000 7f4061400640 1 -- 172.21.15.90:0/213599893 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4064144680 msgr2=0x7f406406d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:54.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.219+0000 7f4061400640 1 --2- 172.21.15.90:0/213599893 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4064144680 0x7f406406d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:54.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.219+0000 7f4061400640 1 -- 172.21.15.90:0/213599893 >> v1:172.21.15.120:6789/0 conn(0x7f4064145190 legacy=0x7f4064071790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:54.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.219+0000 7f4061400640 1 -- 172.21.15.90:0/213599893 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406406e6b0 con 0x7f4064071ea0 2024-03-24T01:25:54.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.219+0000 7f4061400640 1 -- 172.21.15.90:0/213599893 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f405c0019f0 con 0x7f4064071ea0 2024-03-24T01:25:54.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.219+0000 7f4061400640 1 -- 172.21.15.90:0/213599893 >> v1:172.21.15.90:6789/0 conn(0x7f4064071ea0 legacy=0x7f40640cc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:54.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.219+0000 7f4061400640 1 --2- 172.21.15.90:0/213599893 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4064144680 0x7f40480039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:54.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.219+0000 7f4061400640 1 --2- 172.21.15.90:0/213599893 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4048003fb0 0x7f4048004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:54.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.219+0000 7f4061400640 1 --2- 172.21.15.90:0/213599893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4048004d40 0x7f40480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:54.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.219+0000 7f4061400640 1 -- 172.21.15.90:0/213599893 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4064144a60 con 0x7f4048003fb0 2024-03-24T01:25:54.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.219+0000 7f4061400640 1 -- 172.21.15.90:0/213599893 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f404800bb00 con 0x7f4048004d40 2024-03-24T01:25:54.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.219+0000 7f4061400640 1 -- 172.21.15.90:0/213599893 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f404800bd00 con 0x7f4064144680 2024-03-24T01:25:54.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.219+0000 7f4059400640 1 --2- 172.21.15.90:0/213599893 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4048003fb0 0x7f4048004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:54.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.219+0000 7f4061e00640 1 --2- 172.21.15.90:0/213599893 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4064144680 0x7f40480039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:54.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.219+0000 7f4059400640 1 -- 172.21.15.90:0/213599893 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4064144680 msgr2=0x7f40480039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:54.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.219+0000 7f4059400640 1 --2- 172.21.15.90:0/213599893 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4064144680 0x7f40480039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:54.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.219+0000 7f4059400640 1 -- 172.21.15.90:0/213599893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4048004d40 msgr2=0x7f40480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:54.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.219+0000 7f4062800640 1 --2- 172.21.15.90:0/213599893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4048004d40 0x7f40480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:54.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.220+0000 7f4059400640 1 --2- 172.21.15.90:0/213599893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4048004d40 0x7f40480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:54.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.220+0000 7f4059400640 1 -- 172.21.15.90:0/213599893 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4048003520 con 0x7f4048003fb0 2024-03-24T01:25:54.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.220+0000 7f4062800640 1 --2- 172.21.15.90:0/213599893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4048004d40 0x7f40480051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:54.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.221+0000 7f4059400640 1 --2- 172.21.15.90:0/213599893 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4048003fb0 0x7f4048004800 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f404c001750 tx=0x7f4048005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e613f42d37bb5a25 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:54.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.221+0000 7f4061400640 1 -- 172.21.15.90:0/213599893 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404c001370 con 0x7f4048003fb0 2024-03-24T01:25:54.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.221+0000 7f4061400640 1 -- 172.21.15.90:0/213599893 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f404c0052d0 con 0x7f4048003fb0 2024-03-24T01:25:54.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.221+0000 7f4061400640 1 -- 172.21.15.90:0/213599893 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404c003830 con 0x7f4048003fb0 2024-03-24T01:25:54.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.221+0000 7f4063200640 1 -- 172.21.15.90:0/213599893 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4048003fb0 msgr2=0x7f4048004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:54.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.221+0000 7f4063200640 1 --2- 172.21.15.90:0/213599893 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4048003fb0 0x7f4048004800 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f404c001750 tx=0x7f4048005830 comp rx=0 tx=0).stop 2024-03-24T01:25:54.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.221+0000 7f4062800640 1 -- 172.21.15.90:0/213599893 reap_dead start 2024-03-24T01:25:54.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.221+0000 7f4063200640 1 -- 172.21.15.90:0/213599893 shutdown_connections 2024-03-24T01:25:54.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.221+0000 7f4063200640 1 -- 172.21.15.90:0/213599893 >> 172.21.15.90:0/213599893 conn(0x7f406406b0c0 msgr2=0x7f406406b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:54.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.222+0000 7f4063200640 1 -- 172.21.15.90:0/213599893 shutdown_connections 2024-03-24T01:25:54.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.222+0000 7f4063200640 1 -- 172.21.15.90:0/213599893 wait complete. 2024-03-24T01:25:54.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.223+0000 7f4063200640 1 Processor -- start 2024-03-24T01:25:54.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.223+0000 7f4063200640 1 -- start start 2024-03-24T01:25:54.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.224+0000 7f4063200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4048003fb0 0x7f40640b28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:54.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.224+0000 7f4063200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40640b42f0 0x7f40641f6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:54.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.224+0000 7f4063200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40641f7480 0x7f40641431f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:54.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.224+0000 7f4063200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406406db80 con 0x7f4048003fb0 2024-03-24T01:25:54.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.224+0000 7f4063200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406406da00 con 0x7f40641f7480 2024-03-24T01:25:54.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.224+0000 7f4063200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f406406dd00 con 0x7f40640b42f0 2024-03-24T01:25:54.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.225+0000 7f4061e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4048003fb0 0x7f40640b28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:54.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.225+0000 7f4061e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4048003fb0 0x7f40640b28b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60278/0 (socket says 172.21.15.90:60278) 2024-03-24T01:25:54.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.225+0000 7f4061e00640 1 -- 172.21.15.90:0/3563601119 learned_addr learned my addr 172.21.15.90:0/3563601119 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:54.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.225+0000 7f4061e00640 1 -- 172.21.15.90:0/3563601119 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40640b42f0 msgr2=0x7f40641f6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:54.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.225+0000 7f4061e00640 1 --2- 172.21.15.90:0/3563601119 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40640b42f0 0x7f40641f6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:54.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.225+0000 7f4061e00640 1 -- 172.21.15.90:0/3563601119 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40641f7480 msgr2=0x7f40641431f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:54.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.225+0000 7f4062800640 1 --2- 172.21.15.90:0/3563601119 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40641f7480 0x7f40641431f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:54.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.225+0000 7f4061e00640 1 --2- 172.21.15.90:0/3563601119 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40641f7480 0x7f40641431f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:54.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.225+0000 7f4061e00640 1 -- 172.21.15.90:0/3563601119 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40640b2e90 con 0x7f4048003fb0 2024-03-24T01:25:54.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.225+0000 7f4061e00640 1 --2- 172.21.15.90:0/3563601119 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4048003fb0 0x7f40640b28b0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f4054011600 tx=0x7f4054011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:54.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.226+0000 7f405aa00640 1 -- 172.21.15.90:0/3563601119 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f405400f8c0 con 0x7f4048003fb0 2024-03-24T01:25:54.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.226+0000 7f405aa00640 1 -- 172.21.15.90:0/3563601119 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f405400fa60 con 0x7f4048003fb0 2024-03-24T01:25:54.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.226+0000 7f405aa00640 1 -- 172.21.15.90:0/3563601119 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f405400fe10 con 0x7f4048003fb0 2024-03-24T01:25:54.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.226+0000 7f4063200640 1 -- 172.21.15.90:0/3563601119 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40640b3170 con 0x7f4048003fb0 2024-03-24T01:25:54.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.226+0000 7f4063200640 1 -- 172.21.15.90:0/3563601119 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40640b3730 con 0x7f4048003fb0 2024-03-24T01:25:54.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.227+0000 7f4063200640 1 -- 172.21.15.90:0/3563601119 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f406404f400 con 0x7f4048003fb0 2024-03-24T01:25:54.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.228+0000 7f405aa00640 1 -- 172.21.15.90:0/3563601119 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4054015460 con 0x7f4048003fb0 2024-03-24T01:25:54.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.228+0000 7f405aa00640 1 --2- 172.21.15.90:0/3563601119 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f404006bc00 0x7f404006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:54.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.229+0000 7f405aa00640 1 -- 172.21.15.90:0/3563601119 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f405408b2b0 con 0x7f4048003fb0 2024-03-24T01:25:54.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.229+0000 7f4059400640 1 --2- 172.21.15.90:0/3563601119 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f404006bc00 0x7f404006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:54.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.230+0000 7f4059400640 1 --2- 172.21.15.90:0/3563601119 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f404006bc00 0x7f404006e0c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f404c033000 tx=0x7f404c02f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:54.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.233+0000 7f405aa00640 1 -- 172.21.15.90:0/3563601119 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4054019030 con 0x7f4048003fb0 2024-03-24T01:25:54.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.497+0000 7f4063200640 1 -- 172.21.15.90:0/3563601119 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f40641f7860 con 0x7f4048003fb0 2024-03-24T01:25:54.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.502+0000 7f405aa00640 1 -- 172.21.15.90:0/3563601119 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f4054059ec0 con 0x7f4048003fb0 2024-03-24T01:25:54.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.505+0000 7f4063200640 1 -- 172.21.15.90:0/3563601119 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f404006bc00 msgr2=0x7f404006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:54.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.505+0000 7f4063200640 1 --2- 172.21.15.90:0/3563601119 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f404006bc00 0x7f404006e0c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f404c033000 tx=0x7f404c02f000 comp rx=0 tx=0).stop 2024-03-24T01:25:54.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.505+0000 7f4063200640 1 -- 172.21.15.90:0/3563601119 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4048003fb0 msgr2=0x7f40640b28b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:54.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.505+0000 7f4063200640 1 --2- 172.21.15.90:0/3563601119 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4048003fb0 0x7f40640b28b0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f4054011600 tx=0x7f4054011ac0 comp rx=0 tx=0).stop 2024-03-24T01:25:54.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.505+0000 7f4063200640 1 -- 172.21.15.90:0/3563601119 shutdown_connections 2024-03-24T01:25:54.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.505+0000 7f4063200640 1 --2- 172.21.15.90:0/3563601119 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f404006bc00 0x7f404006e0c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:54.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.506+0000 7f4063200640 1 --2- 172.21.15.90:0/3563601119 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40641f7480 0x7f40641431f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:54.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.506+0000 7f4063200640 1 --2- 172.21.15.90:0/3563601119 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40640b42f0 0x7f40641f6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:54.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.506+0000 7f4063200640 1 --2- 172.21.15.90:0/3563601119 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4048003fb0 0x7f40640b28b0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:54.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.506+0000 7f4063200640 1 -- 172.21.15.90:0/3563601119 >> 172.21.15.90:0/3563601119 conn(0x7f406406b0c0 msgr2=0x7f40640b0cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:54.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.506+0000 7f4063200640 1 -- 172.21.15.90:0/3563601119 shutdown_connections 2024-03-24T01:25:54.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.506+0000 7f4063200640 1 -- 172.21.15.90:0/3563601119 wait complete. 2024-03-24T01:25:54.525 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-03-24T01:25:54.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.729+0000 7f5e42c00640 1 Processor -- start 2024-03-24T01:25:54.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.729+0000 7f5e42c00640 1 -- start start 2024-03-24T01:25:54.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.731+0000 7f5e42c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5e3c0717c0 0x7f5e3c071c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:54.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.731+0000 7f5e42c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e3c15d3a0 0x7f5e3c15d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:54.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.731+0000 7f5e42c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e3c15ce00 con 0x7f5e3c1451f0 2024-03-24T01:25:54.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.731+0000 7f5e42c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e3c072160 con 0x7f5e3c0717c0 2024-03-24T01:25:54.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.731+0000 7f5e42c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c15c280 con 0x7f5e3c15d3a0 2024-03-24T01:25:54.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.731+0000 7f5e40e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5e3c0717c0 0x7f5e3c071c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:54.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.731+0000 7f5e40e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5e3c0717c0 0x7f5e3c071c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60292/0 (socket says 172.21.15.90:60292) 2024-03-24T01:25:54.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.731+0000 7f5e40e00640 1 -- 172.21.15.90:0/479148107 learned_addr learned my addr 172.21.15.90:0/479148107 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:54.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.731+0000 7f5e42200640 1 --2- 172.21.15.90:0/479148107 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e3c15d3a0 0x7f5e3c15d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:54.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.731+0000 7f5e40e00640 1 -- 172.21.15.90:0/479148107 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e3c15d3a0 msgr2=0x7f5e3c15d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:54.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.731+0000 7f5e40e00640 1 --2- 172.21.15.90:0/479148107 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e3c15d3a0 0x7f5e3c15d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:54.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.731+0000 7f5e40e00640 1 -- 172.21.15.90:0/479148107 >> v1:172.21.15.120:6789/0 conn(0x7f5e3c1451f0 legacy=0x7f5e3c06ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:54.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.732+0000 7f5e40e00640 1 -- 172.21.15.90:0/479148107 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3c15c560 con 0x7f5e3c0717c0 2024-03-24T01:25:54.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.733+0000 7f5e40e00640 1 --2- 172.21.15.90:0/479148107 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e3c0717c0 0x7f5e3c071c20 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f5e28009870 tx=0x7f5e2802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34a47d4163a572bb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:54.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.733+0000 7f5e3be00640 1 -- 172.21.15.90:0/479148107 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e280317c0 con 0x7f5e3c0717c0 2024-03-24T01:25:54.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.733+0000 7f5e3be00640 1 -- 172.21.15.90:0/479148107 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e28037070 con 0x7f5e3c0717c0 2024-03-24T01:25:54.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.733+0000 7f5e3be00640 1 -- 172.21.15.90:0/479148107 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e28031b80 con 0x7f5e3c0717c0 2024-03-24T01:25:54.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.734+0000 7f5e42c00640 1 -- 172.21.15.90:0/479148107 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e3c0717c0 msgr2=0x7f5e3c071c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:54.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.734+0000 7f5e42c00640 1 --2- 172.21.15.90:0/479148107 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e3c0717c0 0x7f5e3c071c20 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f5e28009870 tx=0x7f5e2802b860 comp rx=0 tx=0).stop 2024-03-24T01:25:54.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.734+0000 7f5e42c00640 1 -- 172.21.15.90:0/479148107 shutdown_connections 2024-03-24T01:25:54.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.734+0000 7f5e42c00640 1 --2- 172.21.15.90:0/479148107 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e3c15d3a0 0x7f5e3c15d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:54.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.734+0000 7f5e42c00640 1 --2- 172.21.15.90:0/479148107 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e3c0717c0 0x7f5e3c071c20 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:54.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.734+0000 7f5e42c00640 1 -- 172.21.15.90:0/479148107 >> 172.21.15.90:0/479148107 conn(0x7f5e3c06b0c0 msgr2=0x7f5e3c06b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:54.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.734+0000 7f5e42c00640 1 -- 172.21.15.90:0/479148107 shutdown_connections 2024-03-24T01:25:54.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.734+0000 7f5e42c00640 1 -- 172.21.15.90:0/479148107 wait complete. 2024-03-24T01:25:54.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.736+0000 7f5e42c00640 1 Processor -- start 2024-03-24T01:25:54.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.736+0000 7f5e42c00640 1 -- start start 2024-03-24T01:25:54.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.737+0000 7f5e42c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e3c0717c0 0x7f5e3c1f57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:54.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.737+0000 7f5e42c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e3c1451f0 0x7f5e3c15b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:54.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.737+0000 7f5e42c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e3c15d3a0 0x7f5e3c0afae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:54.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.737+0000 7f5e42c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c06f330 con 0x7f5e3c1451f0 2024-03-24T01:25:54.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.737+0000 7f5e42c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c15cb20 con 0x7f5e3c0717c0 2024-03-24T01:25:54.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.737+0000 7f5e42c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c1f5d00 con 0x7f5e3c15d3a0 2024-03-24T01:25:54.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.738+0000 7f5e40e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e3c1451f0 0x7f5e3c15b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:54.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.738+0000 7f5e40e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e3c1451f0 0x7f5e3c15b1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60298/0 (socket says 172.21.15.90:60298) 2024-03-24T01:25:54.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.738+0000 7f5e40e00640 1 -- 172.21.15.90:0/2079681924 learned_addr learned my addr 172.21.15.90:0/2079681924 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:54.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.738+0000 7f5e40e00640 1 -- 172.21.15.90:0/2079681924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e3c15d3a0 msgr2=0x7f5e3c0afae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:54.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.738+0000 7f5e40e00640 1 --2- 172.21.15.90:0/2079681924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e3c15d3a0 0x7f5e3c0afae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:54.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.738+0000 7f5e40e00640 1 -- 172.21.15.90:0/2079681924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e3c0717c0 msgr2=0x7f5e3c1f57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:54.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.738+0000 7f5e40e00640 1 --2- 172.21.15.90:0/2079681924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e3c0717c0 0x7f5e3c1f57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:54.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.738+0000 7f5e40e00640 1 -- 172.21.15.90:0/2079681924 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3c140330 con 0x7f5e3c1451f0 2024-03-24T01:25:54.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.738+0000 7f5e40e00640 1 --2- 172.21.15.90:0/2079681924 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e3c1451f0 0x7f5e3c15b1a0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f5e28009870 tx=0x7f5e280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:54.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.739+0000 7f5e39600640 1 -- 172.21.15.90:0/2079681924 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e28032860 con 0x7f5e3c1451f0 2024-03-24T01:25:54.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.739+0000 7f5e39600640 1 -- 172.21.15.90:0/2079681924 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e28032a00 con 0x7f5e3c1451f0 2024-03-24T01:25:54.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.739+0000 7f5e39600640 1 -- 172.21.15.90:0/2079681924 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e28032df0 con 0x7f5e3c1451f0 2024-03-24T01:25:54.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.739+0000 7f5e42c00640 1 -- 172.21.15.90:0/2079681924 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e3c15b6e0 con 0x7f5e3c1451f0 2024-03-24T01:25:54.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.739+0000 7f5e42c00640 1 -- 172.21.15.90:0/2079681924 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e3c12f770 con 0x7f5e3c1451f0 2024-03-24T01:25:54.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.740+0000 7f5e42c00640 1 -- 172.21.15.90:0/2079681924 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e3c04f400 con 0x7f5e3c1451f0 2024-03-24T01:25:54.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.741+0000 7f5e39600640 1 -- 172.21.15.90:0/2079681924 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5e2804c060 con 0x7f5e3c1451f0 2024-03-24T01:25:54.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.742+0000 7f5e39600640 1 --2- 172.21.15.90:0/2079681924 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5e1c06bbd0 0x7f5e1c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:54.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.742+0000 7f5e39600640 1 -- 172.21.15.90:0/2079681924 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f5e280ac3b0 con 0x7f5e3c1451f0 2024-03-24T01:25:54.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.742+0000 7f5e41800640 1 --2- 172.21.15.90:0/2079681924 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5e1c06bbd0 0x7f5e1c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:54.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.743+0000 7f5e41800640 1 --2- 172.21.15.90:0/2079681924 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5e1c06bbd0 0x7f5e1c06e090 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5e30003cc0 tx=0x7f5e300079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:54.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:54.748+0000 7f5e39600640 1 -- 172.21.15.90:0/2079681924 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5e3c140330 con 0x7f5e3c1451f0 2024-03-24T01:25:55.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.018+0000 7f5e42c00640 1 -- 172.21.15.90:0/2079681924 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f5e3c06b820 con 0x7f5e3c1451f0 2024-03-24T01:25:55.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.023+0000 7f5e39600640 1 -- 172.21.15.90:0/2079681924 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) v1 ==== 97+0+0 (secure 0 0 0) 0x7f5e28040070 con 0x7f5e3c1451f0 2024-03-24T01:25:55.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.026+0000 7f5e42c00640 1 -- 172.21.15.90:0/2079681924 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5e1c06bbd0 msgr2=0x7f5e1c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:55.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.026+0000 7f5e42c00640 1 --2- 172.21.15.90:0/2079681924 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5e1c06bbd0 0x7f5e1c06e090 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5e30003cc0 tx=0x7f5e300079b0 comp rx=0 tx=0).stop 2024-03-24T01:25:55.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.026+0000 7f5e42c00640 1 -- 172.21.15.90:0/2079681924 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e3c1451f0 msgr2=0x7f5e3c15b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:55.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.026+0000 7f5e42c00640 1 --2- 172.21.15.90:0/2079681924 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e3c1451f0 0x7f5e3c15b1a0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f5e28009870 tx=0x7f5e280046c0 comp rx=0 tx=0).stop 2024-03-24T01:25:55.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.026+0000 7f5e42c00640 1 -- 172.21.15.90:0/2079681924 shutdown_connections 2024-03-24T01:25:55.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.026+0000 7f5e42c00640 1 --2- 172.21.15.90:0/2079681924 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5e1c06bbd0 0x7f5e1c06e090 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:55.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.026+0000 7f5e42c00640 1 --2- 172.21.15.90:0/2079681924 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e3c1451f0 0x7f5e3c15b1a0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:55.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.026+0000 7f5e42c00640 1 --2- 172.21.15.90:0/2079681924 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e3c15d3a0 0x7f5e3c0afae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:55.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.026+0000 7f5e42c00640 1 --2- 172.21.15.90:0/2079681924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e3c0717c0 0x7f5e3c1f57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:55.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.026+0000 7f5e42c00640 1 -- 172.21.15.90:0/2079681924 >> 172.21.15.90:0/2079681924 conn(0x7f5e3c06b0c0 msgr2=0x7f5e3c1437f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:55.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.026+0000 7f5e42c00640 1 -- 172.21.15.90:0/2079681924 shutdown_connections 2024-03-24T01:25:55.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.026+0000 7f5e42c00640 1 -- 172.21.15.90:0/2079681924 wait complete. 2024-03-24T01:25:55.044 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-03-24T01:25:55.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.244+0000 7f788ec00640 1 Processor -- start 2024-03-24T01:25:55.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.244+0000 7f788ec00640 1 -- start start 2024-03-24T01:25:55.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.246+0000 7f788ec00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f788815da30 0x7f788815de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:55.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.246+0000 7f788ec00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f78880acea0 0x7f78880aea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:55.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.246+0000 7f788ec00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f788806b0c0 0x7f788806b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:55.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.246+0000 7f788ec00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78880b2c70 con 0x7f788806b0c0 2024-03-24T01:25:55.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.246+0000 7f788ec00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f788815f160 con 0x7f78880acea0 2024-03-24T01:25:55.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.246+0000 7f788ec00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f788815e550 con 0x7f788815da30 2024-03-24T01:25:55.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.246+0000 7f788cc00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f788806b0c0 0x7f788806b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:55.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.246+0000 7f788cc00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f788806b0c0 0x7f788806b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60308/0 (socket says 172.21.15.90:60308) 2024-03-24T01:25:55.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.246+0000 7f788cc00640 1 -- 172.21.15.90:0/656980974 learned_addr learned my addr 172.21.15.90:0/656980974 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:55.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.246+0000 7f7887e00640 1 --2- 172.21.15.90:0/656980974 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f788815da30 0x7f788815de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:55.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.246+0000 7f7887400640 1 --2- 172.21.15.90:0/656980974 >> v2:172.21.15.120:3300/0 conn(0x7f78880acea0 0x7f78880aea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:55.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.246+0000 7f788cc00640 1 -- 172.21.15.90:0/656980974 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f788815da30 msgr2=0x7f788815de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:55.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.247+0000 7f788cc00640 1 --2- 172.21.15.90:0/656980974 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f788815da30 0x7f788815de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:55.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.247+0000 7f788cc00640 1 -- 172.21.15.90:0/656980974 >> v2:172.21.15.120:3300/0 conn(0x7f78880acea0 msgr2=0x7f78880aea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:55.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.247+0000 7f788cc00640 1 --2- 172.21.15.90:0/656980974 >> v2:172.21.15.120:3300/0 conn(0x7f78880acea0 0x7f78880aea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:55.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.247+0000 7f788cc00640 1 -- 172.21.15.90:0/656980974 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f788815e830 con 0x7f788806b0c0 2024-03-24T01:25:55.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.247+0000 7f7887e00640 1 --2- 172.21.15.90:0/656980974 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f788815da30 0x7f788815de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:55.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.247+0000 7f7887400640 1 --2- 172.21.15.90:0/656980974 >> v2:172.21.15.120:3300/0 conn(0x7f78880acea0 0x7f78880aea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:55.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.248+0000 7f788cc00640 1 --2- 172.21.15.90:0/656980974 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f788806b0c0 0x7f788806b550 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f7888072ab0 tx=0x7f788002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23384a47fc48f83d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:55.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.248+0000 7f7886a00640 1 -- 172.21.15.90:0/656980974 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788002ca80 con 0x7f788806b0c0 2024-03-24T01:25:55.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.248+0000 7f7886a00640 1 -- 172.21.15.90:0/656980974 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f788002cc20 con 0x7f788806b0c0 2024-03-24T01:25:55.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.248+0000 7f7886a00640 1 -- 172.21.15.90:0/656980974 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78800343f0 con 0x7f788806b0c0 2024-03-24T01:25:55.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.249+0000 7f788ec00640 1 -- 172.21.15.90:0/656980974 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f788806b0c0 msgr2=0x7f788806b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:55.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.249+0000 7f788ec00640 1 --2- 172.21.15.90:0/656980974 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f788806b0c0 0x7f788806b550 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f7888072ab0 tx=0x7f788002bee0 comp rx=0 tx=0).stop 2024-03-24T01:25:55.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.249+0000 7f788ec00640 1 -- 172.21.15.90:0/656980974 shutdown_connections 2024-03-24T01:25:55.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.249+0000 7f788ec00640 1 --2- 172.21.15.90:0/656980974 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f788806b0c0 0x7f788806b550 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:55.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.249+0000 7f788ec00640 1 --2- 172.21.15.90:0/656980974 >> v2:172.21.15.120:3300/0 conn(0x7f78880acea0 0x7f78880aea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:55.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.249+0000 7f788ec00640 1 --2- 172.21.15.90:0/656980974 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f788815da30 0x7f788815de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:55.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.249+0000 7f788ec00640 1 -- 172.21.15.90:0/656980974 >> 172.21.15.90:0/656980974 conn(0x7f78880b1770 msgr2=0x7f78880b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:55.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.249+0000 7f788ec00640 1 -- 172.21.15.90:0/656980974 shutdown_connections 2024-03-24T01:25:55.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.249+0000 7f788ec00640 1 -- 172.21.15.90:0/656980974 wait complete. 2024-03-24T01:25:55.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.250+0000 7f788ec00640 1 Processor -- start 2024-03-24T01:25:55.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.251+0000 7f788ec00640 1 -- start start 2024-03-24T01:25:55.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.251+0000 7f788ec00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f788806b0c0 0x7f78880af3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:55.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.251+0000 7f788ec00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78880acea0 0x7f7888152960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:55.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.251+0000 7f788ec00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f788815da30 0x7f7888151cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:55.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.252+0000 7f788ec00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78880ad300 con 0x7f788815da30 2024-03-24T01:25:55.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.252+0000 7f788ec00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788815edf0 con 0x7f78880acea0 2024-03-24T01:25:55.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.252+0000 7f788ec00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78880aef70 con 0x7f788806b0c0 2024-03-24T01:25:55.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.252+0000 7f788cc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f788815da30 0x7f7888151cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:55.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.252+0000 7f788cc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f788815da30 0x7f7888151cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60324/0 (socket says 172.21.15.90:60324) 2024-03-24T01:25:55.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.252+0000 7f788cc00640 1 -- 172.21.15.90:0/1529883593 learned_addr learned my addr 172.21.15.90:0/1529883593 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:55.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.252+0000 7f7887e00640 1 --2- 172.21.15.90:0/1529883593 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f788806b0c0 0x7f78880af3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:55.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.252+0000 7f7887400640 1 --2- 172.21.15.90:0/1529883593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78880acea0 0x7f7888152960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:55.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.252+0000 7f788cc00640 1 -- 172.21.15.90:0/1529883593 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f788806b0c0 msgr2=0x7f78880af3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:55.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.252+0000 7f788cc00640 1 --2- 172.21.15.90:0/1529883593 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f788806b0c0 0x7f78880af3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:55.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.252+0000 7f788cc00640 1 -- 172.21.15.90:0/1529883593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78880acea0 msgr2=0x7f7888152960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:55.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.252+0000 7f788cc00640 1 --2- 172.21.15.90:0/1529883593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78880acea0 0x7f7888152960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:55.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.252+0000 7f788cc00640 1 -- 172.21.15.90:0/1529883593 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7888154ab0 con 0x7f788815da30 2024-03-24T01:25:55.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.253+0000 7f788cc00640 1 --2- 172.21.15.90:0/1529883593 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f788815da30 0x7f7888151cd0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f7880036040 tx=0x7f7880004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:55.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.253+0000 7f787fe00640 1 -- 172.21.15.90:0/1529883593 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7880043020 con 0x7f788815da30 2024-03-24T01:25:55.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.253+0000 7f787fe00640 1 -- 172.21.15.90:0/1529883593 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f788003e450 con 0x7f788815da30 2024-03-24T01:25:55.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.253+0000 7f787fe00640 1 -- 172.21.15.90:0/1529883593 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7880009510 con 0x7f788815da30 2024-03-24T01:25:55.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.253+0000 7f788ec00640 1 -- 172.21.15.90:0/1529883593 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7888154d90 con 0x7f788815da30 2024-03-24T01:25:55.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.253+0000 7f788ec00640 1 -- 172.21.15.90:0/1529883593 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78880af910 con 0x7f788815da30 2024-03-24T01:25:55.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.254+0000 7f788ec00640 1 -- 172.21.15.90:0/1529883593 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f788804f400 con 0x7f788815da30 2024-03-24T01:25:55.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.255+0000 7f787fe00640 1 -- 172.21.15.90:0/1529883593 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7880046020 con 0x7f788815da30 2024-03-24T01:25:55.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.256+0000 7f787fe00640 1 --2- 172.21.15.90:0/1529883593 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f786806bbd0 0x7f786806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:55.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.256+0000 7f787fe00640 1 -- 172.21.15.90:0/1529883593 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f78800ad5e0 con 0x7f788815da30 2024-03-24T01:25:55.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.257+0000 7f7887e00640 1 --2- 172.21.15.90:0/1529883593 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f786806bbd0 0x7f786806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:55.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.258+0000 7f7887e00640 1 --2- 172.21.15.90:0/1529883593 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f786806bbd0 0x7f786806e090 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7878006fd0 tx=0x7f7878008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:55.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.260+0000 7f787fe00640 1 -- 172.21.15.90:0/1529883593 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f788004d020 con 0x7f788815da30 2024-03-24T01:25:55.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.511+0000 7f788ec00640 1 -- 172.21.15.90:0/1529883593 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f78880ae290 con 0x7f788815da30 2024-03-24T01:25:55.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.516+0000 7f787fe00640 1 -- 172.21.15.90:0/1529883593 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7f788007c3a0 con 0x7f788815da30 2024-03-24T01:25:55.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.519+0000 7f788ec00640 1 -- 172.21.15.90:0/1529883593 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f786806bbd0 msgr2=0x7f786806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:55.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.519+0000 7f788ec00640 1 --2- 172.21.15.90:0/1529883593 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f786806bbd0 0x7f786806e090 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7878006fd0 tx=0x7f7878008040 comp rx=0 tx=0).stop 2024-03-24T01:25:55.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.519+0000 7f788ec00640 1 -- 172.21.15.90:0/1529883593 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f788815da30 msgr2=0x7f7888151cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:55.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.519+0000 7f788ec00640 1 --2- 172.21.15.90:0/1529883593 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f788815da30 0x7f7888151cd0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f7880036040 tx=0x7f7880004a40 comp rx=0 tx=0).stop 2024-03-24T01:25:55.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.519+0000 7f788ec00640 1 -- 172.21.15.90:0/1529883593 shutdown_connections 2024-03-24T01:25:55.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.519+0000 7f788ec00640 1 --2- 172.21.15.90:0/1529883593 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f786806bbd0 0x7f786806e090 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:55.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.519+0000 7f788ec00640 1 --2- 172.21.15.90:0/1529883593 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f788815da30 0x7f7888151cd0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:55.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.519+0000 7f788ec00640 1 --2- 172.21.15.90:0/1529883593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78880acea0 0x7f7888152960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:55.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.519+0000 7f788ec00640 1 --2- 172.21.15.90:0/1529883593 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f788806b0c0 0x7f78880af3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:55.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.519+0000 7f788ec00640 1 -- 172.21.15.90:0/1529883593 >> 172.21.15.90:0/1529883593 conn(0x7f78880b1770 msgr2=0x7f788806e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:55.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.519+0000 7f788ec00640 1 -- 172.21.15.90:0/1529883593 shutdown_connections 2024-03-24T01:25:55.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.519+0000 7f788ec00640 1 -- 172.21.15.90:0/1529883593 wait complete. 2024-03-24T01:25:55.538 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:25:55.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.740+0000 7f778da00640 1 Processor -- start 2024-03-24T01:25:55.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.740+0000 7f778da00640 1 -- start start 2024-03-24T01:25:55.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.742+0000 7f778da00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7788155030 0x7f778815efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:55.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.742+0000 7f778da00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77880cdac0 0x7f778806b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:55.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.742+0000 7f778da00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f778806b600 0x7f77880ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:55.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.742+0000 7f778da00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f778806ba70 con 0x7f7788155030 2024-03-24T01:25:55.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.742+0000 7f778da00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f778815f500 con 0x7f778806b600 2024-03-24T01:25:55.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.742+0000 7f778da00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f778815bf70 con 0x7f77880cdac0 2024-03-24T01:25:55.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.742+0000 7f7786a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7788155030 0x7f778815efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:55.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.742+0000 7f7786a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7788155030 0x7f778815efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60336/0 (socket says 172.21.15.90:60336) 2024-03-24T01:25:55.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.742+0000 7f7786a00640 1 -- 172.21.15.90:0/1036763873 learned_addr learned my addr 172.21.15.90:0/1036763873 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:55.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.742+0000 7f7786a00640 1 -- 172.21.15.90:0/1036763873 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77880cdac0 msgr2=0x7f778806b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:55.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.742+0000 7f7786a00640 1 --2- 172.21.15.90:0/1036763873 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77880cdac0 0x7f778806b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:55.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.742+0000 7f7786a00640 1 -- 172.21.15.90:0/1036763873 >> v2:172.21.15.120:3300/0 conn(0x7f778806b600 msgr2=0x7f77880ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:55.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.742+0000 7f7787400640 1 --2- 172.21.15.90:0/1036763873 >> v2:172.21.15.120:3300/0 conn(0x7f778806b600 0x7f77880ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:55.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.742+0000 7f7786a00640 1 --2- 172.21.15.90:0/1036763873 >> v2:172.21.15.120:3300/0 conn(0x7f778806b600 0x7f77880ae530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:55.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.742+0000 7f7786a00640 1 -- 172.21.15.90:0/1036763873 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f778815b360 con 0x7f7788155030 2024-03-24T01:25:55.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.744+0000 7f7786a00640 1 --2- 172.21.15.90:0/1036763873 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7788155030 0x7f778815efc0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f778806ec70 tx=0x7f777802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=442276be54bad968 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:55.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.744+0000 7f7785600640 1 -- 172.21.15.90:0/1036763873 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7778035020 con 0x7f7788155030 2024-03-24T01:25:55.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.744+0000 7f7785600640 1 -- 172.21.15.90:0/1036763873 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7778031870 con 0x7f7788155030 2024-03-24T01:25:55.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.744+0000 7f7785600640 1 -- 172.21.15.90:0/1036763873 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7778031c40 con 0x7f7788155030 2024-03-24T01:25:55.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.745+0000 7f778da00640 1 -- 172.21.15.90:0/1036763873 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7788155030 msgr2=0x7f778815efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:55.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.745+0000 7f778da00640 1 --2- 172.21.15.90:0/1036763873 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7788155030 0x7f778815efc0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f778806ec70 tx=0x7f777802b910 comp rx=0 tx=0).stop 2024-03-24T01:25:55.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.745+0000 7f778da00640 1 -- 172.21.15.90:0/1036763873 shutdown_connections 2024-03-24T01:25:55.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.745+0000 7f778da00640 1 --2- 172.21.15.90:0/1036763873 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f77880cdac0 0x7f778806b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:55.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.745+0000 7f778da00640 1 --2- 172.21.15.90:0/1036763873 >> v2:172.21.15.120:3300/0 conn(0x7f778806b600 0x7f77880ae530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:55.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.745+0000 7f778da00640 1 --2- 172.21.15.90:0/1036763873 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7788155030 0x7f778815efc0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:55.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.745+0000 7f778da00640 1 -- 172.21.15.90:0/1036763873 >> 172.21.15.90:0/1036763873 conn(0x7f77880b1770 msgr2=0x7f77880b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:55.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.745+0000 7f778da00640 1 -- 172.21.15.90:0/1036763873 shutdown_connections 2024-03-24T01:25:55.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.745+0000 7f778da00640 1 -- 172.21.15.90:0/1036763873 wait complete. 2024-03-24T01:25:55.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.747+0000 7f778da00640 1 Processor -- start 2024-03-24T01:25:55.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.747+0000 7f778da00640 1 -- start start 2024-03-24T01:25:55.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.748+0000 7f778da00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f778806b600 0x7f77880cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:55.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.748+0000 7f778da00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77880cdac0 0x7f7788165460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:55.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.748+0000 7f778da00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7788155030 0x7f7788167070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:55.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.748+0000 7f778da00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778815bc00 con 0x7f778806b600 2024-03-24T01:25:55.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.748+0000 7f778da00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778815b920 con 0x7f77880cdac0 2024-03-24T01:25:55.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.748+0000 7f778da00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7788155410 con 0x7f7788155030 2024-03-24T01:25:55.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.748+0000 7f7786a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f778806b600 0x7f77880cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:55.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.748+0000 7f7786a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f778806b600 0x7f77880cb0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60346/0 (socket says 172.21.15.90:60346) 2024-03-24T01:25:55.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.748+0000 7f7786a00640 1 -- 172.21.15.90:0/1594039090 learned_addr learned my addr 172.21.15.90:0/1594039090 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:55.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.748+0000 7f7786a00640 1 -- 172.21.15.90:0/1594039090 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7788155030 msgr2=0x7f7788167070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:55.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.748+0000 7f7786a00640 1 --2- 172.21.15.90:0/1594039090 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7788155030 0x7f7788167070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:55.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.748+0000 7f7786a00640 1 -- 172.21.15.90:0/1594039090 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77880cdac0 msgr2=0x7f7788165460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:55.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.748+0000 7f7786a00640 1 --2- 172.21.15.90:0/1594039090 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77880cdac0 0x7f7788165460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:55.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.748+0000 7f7786a00640 1 -- 172.21.15.90:0/1594039090 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77880cb620 con 0x7f778806b600 2024-03-24T01:25:55.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.749+0000 7f7786a00640 1 --2- 172.21.15.90:0/1594039090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f778806b600 0x7f77880cb0e0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f777802b950 tx=0x7f77780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:55.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.749+0000 7f777ea00640 1 -- 172.21.15.90:0/1594039090 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7778040070 con 0x7f778806b600 2024-03-24T01:25:55.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.749+0000 7f777ea00640 1 -- 172.21.15.90:0/1594039090 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f777803a420 con 0x7f778806b600 2024-03-24T01:25:55.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.749+0000 7f778da00640 1 -- 172.21.15.90:0/1594039090 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77881646a0 con 0x7f778806b600 2024-03-24T01:25:55.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.749+0000 7f777ea00640 1 -- 172.21.15.90:0/1594039090 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7778035040 con 0x7f778806b600 2024-03-24T01:25:55.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.749+0000 7f778da00640 1 -- 172.21.15.90:0/1594039090 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7788164c60 con 0x7f778806b600 2024-03-24T01:25:55.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.750+0000 7f778da00640 1 -- 172.21.15.90:0/1594039090 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7788068060 con 0x7f778806b600 2024-03-24T01:25:55.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.751+0000 7f777ea00640 1 -- 172.21.15.90:0/1594039090 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7778038020 con 0x7f778806b600 2024-03-24T01:25:55.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.752+0000 7f777ea00640 1 --2- 172.21.15.90:0/1594039090 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f776806bbd0 0x7f776806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:55.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.752+0000 7f777ea00640 1 -- 172.21.15.90:0/1594039090 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f77780ac990 con 0x7f778806b600 2024-03-24T01:25:55.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.753+0000 7f7786000640 1 --2- 172.21.15.90:0/1594039090 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f776806bbd0 0x7f776806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:55.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.754+0000 7f7786000640 1 --2- 172.21.15.90:0/1594039090 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f776806bbd0 0x7f776806e090 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f778814eb40 tx=0x7f7774006c90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:55.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.757+0000 7f777ea00640 1 -- 172.21.15.90:0/1594039090 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f777807b510 con 0x7f778806b600 2024-03-24T01:25:55.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.959+0000 7f778da00640 1 -- 172.21.15.90:0/1594039090 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77880ccdc0 con 0x7f778806b600 2024-03-24T01:25:55.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.960+0000 7f777ea00640 1 -- 172.21.15.90:0/1594039090 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+13140 (secure 0 0 0) 0x7f777807ae10 con 0x7f778806b600 2024-03-24T01:25:55.961 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:25:55.962 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":35,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:25:51.933169+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:25:55.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.963+0000 7f778da00640 1 -- 172.21.15.90:0/1594039090 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f776806bbd0 msgr2=0x7f776806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:55.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.963+0000 7f778da00640 1 --2- 172.21.15.90:0/1594039090 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f776806bbd0 0x7f776806e090 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f778814eb40 tx=0x7f7774006c90 comp rx=0 tx=0).stop 2024-03-24T01:25:55.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.963+0000 7f778da00640 1 -- 172.21.15.90:0/1594039090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f778806b600 msgr2=0x7f77880cb0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:55.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.963+0000 7f778da00640 1 --2- 172.21.15.90:0/1594039090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f778806b600 0x7f77880cb0e0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f777802b950 tx=0x7f77780046c0 comp rx=0 tx=0).stop 2024-03-24T01:25:55.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.963+0000 7f778da00640 1 -- 172.21.15.90:0/1594039090 shutdown_connections 2024-03-24T01:25:55.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.963+0000 7f778da00640 1 --2- 172.21.15.90:0/1594039090 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f776806bbd0 0x7f776806e090 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:55.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.963+0000 7f778da00640 1 --2- 172.21.15.90:0/1594039090 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7788155030 0x7f7788167070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:55.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.963+0000 7f778da00640 1 --2- 172.21.15.90:0/1594039090 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77880cdac0 0x7f7788165460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:55.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.963+0000 7f778da00640 1 --2- 172.21.15.90:0/1594039090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f778806b600 0x7f77880cb0e0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:55.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.963+0000 7f778da00640 1 -- 172.21.15.90:0/1594039090 >> 172.21.15.90:0/1594039090 conn(0x7f77880b1770 msgr2=0x7f778815dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:55.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.963+0000 7f778da00640 1 -- 172.21.15.90:0/1594039090 shutdown_connections 2024-03-24T01:25:55.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:55.963+0000 7f778da00640 1 -- 172.21.15.90:0/1594039090 wait complete. 2024-03-24T01:25:55.982 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T01:25:55.983 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:25:56.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.193+0000 7f9d3da00640 1 Processor -- start 2024-03-24T01:25:56.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.194+0000 7f9d3da00640 1 -- start start 2024-03-24T01:25:56.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.195+0000 7f9d3da00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9d3815dfe0 0x7f9d381622c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:56.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.195+0000 7f9d3da00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d380b2e50 0x7f9d380d28b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:56.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.195+0000 7f9d3da00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d380d20f0 con 0x7f9d380d0b60 2024-03-24T01:25:56.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.195+0000 7f9d3da00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d380ced80 con 0x7f9d3815dfe0 2024-03-24T01:25:56.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.195+0000 7f9d3da00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d380cf060 con 0x7f9d380b2e50 2024-03-24T01:25:56.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.196+0000 7f9d37400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d380b2e50 0x7f9d380d28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:56.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.196+0000 7f9d2e000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9d3815dfe0 0x7f9d381622c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:56.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.196+0000 7f9d2e000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9d3815dfe0 0x7f9d381622c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.90:59950/0 (socket says 172.21.15.90:59950) 2024-03-24T01:25:56.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.196+0000 7f9d2e000640 1 -- 172.21.15.90:0/3872115323 learned_addr learned my addr 172.21.15.90:0/3872115323 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:56.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.196+0000 7f9d36a00640 1 --1- 172.21.15.90:0/3872115323 >> v1:172.21.15.120:6789/0 conn(0x7f9d380d0b60 0x7f9d380d0f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.90:35504/0 (socket says 172.21.15.90:35504) 2024-03-24T01:25:56.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.197+0000 7f9d37400640 1 -- 172.21.15.90:0/3872115323 >> v2:172.21.15.120:3300/0 conn(0x7f9d3815dfe0 msgr2=0x7f9d381622c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:56.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.197+0000 7f9d37400640 1 --2- 172.21.15.90:0/3872115323 >> v2:172.21.15.120:3300/0 conn(0x7f9d3815dfe0 0x7f9d381622c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:56.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.197+0000 7f9d37400640 1 -- 172.21.15.90:0/3872115323 >> v1:172.21.15.120:6789/0 conn(0x7f9d380d0b60 legacy=0x7f9d380d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:56.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.197+0000 7f9d37400640 1 -- 172.21.15.90:0/3872115323 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d380d2390 con 0x7f9d380b2e50 2024-03-24T01:25:56.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.197+0000 7f9d2e000640 1 --2- 172.21.15.90:0/3872115323 >> v2:172.21.15.120:3300/0 conn(0x7f9d3815dfe0 0x7f9d381622c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T01:25:56.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.198+0000 7f9d37400640 1 --2- 172.21.15.90:0/3872115323 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d380b2e50 0x7f9d380d28b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9d380b32c0 tx=0x7f9d3002bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81ac1eab8f74f0a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:56.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.199+0000 7f9d36000640 1 -- 172.21.15.90:0/3872115323 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d30039070 con 0x7f9d380b2e50 2024-03-24T01:25:56.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.199+0000 7f9d36000640 1 -- 172.21.15.90:0/3872115323 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d3002cb20 con 0x7f9d380b2e50 2024-03-24T01:25:56.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.199+0000 7f9d36000640 1 -- 172.21.15.90:0/3872115323 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d30034440 con 0x7f9d380b2e50 2024-03-24T01:25:56.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.199+0000 7f9d3da00640 1 -- 172.21.15.90:0/3872115323 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d380b2e50 msgr2=0x7f9d380d28b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:56.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.199+0000 7f9d3da00640 1 --2- 172.21.15.90:0/3872115323 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d380b2e50 0x7f9d380d28b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9d380b32c0 tx=0x7f9d3002bd70 comp rx=0 tx=0).stop 2024-03-24T01:25:56.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.200+0000 7f9d3da00640 1 -- 172.21.15.90:0/3872115323 shutdown_connections 2024-03-24T01:25:56.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.200+0000 7f9d3da00640 1 --2- 172.21.15.90:0/3872115323 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d380b2e50 0x7f9d380d28b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:56.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.200+0000 7f9d3da00640 1 --2- 172.21.15.90:0/3872115323 >> v2:172.21.15.120:3300/0 conn(0x7f9d3815dfe0 0x7f9d381622c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:56.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.200+0000 7f9d3da00640 1 -- 172.21.15.90:0/3872115323 >> 172.21.15.90:0/3872115323 conn(0x7f9d38072c10 msgr2=0x7f9d38073020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:56.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.200+0000 7f9d3da00640 1 -- 172.21.15.90:0/3872115323 shutdown_connections 2024-03-24T01:25:56.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.200+0000 7f9d3da00640 1 -- 172.21.15.90:0/3872115323 wait complete. 2024-03-24T01:25:56.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.201+0000 7f9d3da00640 1 Processor -- start 2024-03-24T01:25:56.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.201+0000 7f9d3da00640 1 -- start start 2024-03-24T01:25:56.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.202+0000 7f9d3da00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d380b2e50 0x7f9d38150270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:56.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.202+0000 7f9d3da00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d380d0b60 0x7f9d381f5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:56.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.202+0000 7f9d3da00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d3815dfe0 0x7f9d380c9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:56.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.202+0000 7f9d3da00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d380b6d90 con 0x7f9d380d0b60 2024-03-24T01:25:56.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.202+0000 7f9d3da00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d380cf340 con 0x7f9d380b2e50 2024-03-24T01:25:56.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.202+0000 7f9d3da00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d380b6f10 con 0x7f9d3815dfe0 2024-03-24T01:25:56.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.203+0000 7f9d2e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d380d0b60 0x7f9d381f5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:56.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.203+0000 7f9d2e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d380d0b60 0x7f9d381f5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60358/0 (socket says 172.21.15.90:60358) 2024-03-24T01:25:56.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.203+0000 7f9d2e000640 1 -- 172.21.15.90:0/310260466 learned_addr learned my addr 172.21.15.90:0/310260466 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:56.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.203+0000 7f9d36a00640 1 --2- 172.21.15.90:0/310260466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d380b2e50 0x7f9d38150270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:56.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.203+0000 7f9d37400640 1 --2- 172.21.15.90:0/310260466 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d3815dfe0 0x7f9d380c9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:56.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.203+0000 7f9d2e000640 1 -- 172.21.15.90:0/310260466 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d3815dfe0 msgr2=0x7f9d380c9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:56.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.203+0000 7f9d2e000640 1 --2- 172.21.15.90:0/310260466 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d3815dfe0 0x7f9d380c9ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:56.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.203+0000 7f9d2e000640 1 -- 172.21.15.90:0/310260466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d380b2e50 msgr2=0x7f9d38150270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:56.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.203+0000 7f9d2e000640 1 --2- 172.21.15.90:0/310260466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d380b2e50 0x7f9d38150270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:56.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.203+0000 7f9d2e000640 1 -- 172.21.15.90:0/310260466 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d381f5970 con 0x7f9d380d0b60 2024-03-24T01:25:56.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.203+0000 7f9d37400640 1 --2- 172.21.15.90:0/310260466 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d3815dfe0 0x7f9d380c9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:56.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.203+0000 7f9d36a00640 1 --2- 172.21.15.90:0/310260466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d380b2e50 0x7f9d38150270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:56.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.203+0000 7f9d2e000640 1 --2- 172.21.15.90:0/310260466 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d380d0b60 0x7f9d381f5430 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f9d2000e9e0 tx=0x7f9d2000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:56.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.204+0000 7f9d2f400640 1 -- 172.21.15.90:0/310260466 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d20017070 con 0x7f9d380d0b60 2024-03-24T01:25:56.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.204+0000 7f9d2f400640 1 -- 172.21.15.90:0/310260466 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d2000c9d0 con 0x7f9d380d0b60 2024-03-24T01:25:56.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.204+0000 7f9d2f400640 1 -- 172.21.15.90:0/310260466 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2000cd80 con 0x7f9d380d0b60 2024-03-24T01:25:56.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.204+0000 7f9d3da00640 1 -- 172.21.15.90:0/310260466 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d380cba50 con 0x7f9d380d0b60 2024-03-24T01:25:56.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.204+0000 7f9d3da00640 1 -- 172.21.15.90:0/310260466 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d380cc010 con 0x7f9d380d0b60 2024-03-24T01:25:56.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.205+0000 7f9d3da00640 1 -- 172.21.15.90:0/310260466 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d3804f400 con 0x7f9d380d0b60 2024-03-24T01:25:56.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.206+0000 7f9d2f400640 1 -- 172.21.15.90:0/310260466 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9d2001a070 con 0x7f9d380d0b60 2024-03-24T01:25:56.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.207+0000 7f9d2f400640 1 --2- 172.21.15.90:0/310260466 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9d1406bbd0 0x7f9d1406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:56.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.207+0000 7f9d2f400640 1 -- 172.21.15.90:0/310260466 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f9d2008aaa0 con 0x7f9d380d0b60 2024-03-24T01:25:56.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.208+0000 7f9d36a00640 1 --2- 172.21.15.90:0/310260466 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9d1406bbd0 0x7f9d1406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:56.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.209+0000 7f9d36a00640 1 --2- 172.21.15.90:0/310260466 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9d1406bbd0 0x7f9d1406e090 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9d28000f60 tx=0x7f9d28009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:56.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.212+0000 7f9d2f400640 1 -- 172.21.15.90:0/310260466 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9d200597e0 con 0x7f9d380d0b60 2024-03-24T01:25:56.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.408+0000 7f9d3da00640 1 -- 172.21.15.90:0/310260466 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9d380b3330 con 0x7f9d380d0b60 2024-03-24T01:25:56.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.409+0000 7f9d2f400640 1 -- 172.21.15.90:0/310260466 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v35) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9d200590e0 con 0x7f9d380d0b60 2024-03-24T01:25:56.409 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:25:56.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.411+0000 7f9d3da00640 1 -- 172.21.15.90:0/310260466 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9d1406bbd0 msgr2=0x7f9d1406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:56.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.411+0000 7f9d3da00640 1 --2- 172.21.15.90:0/310260466 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9d1406bbd0 0x7f9d1406e090 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9d28000f60 tx=0x7f9d28009040 comp rx=0 tx=0).stop 2024-03-24T01:25:56.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.411+0000 7f9d3da00640 1 -- 172.21.15.90:0/310260466 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d380d0b60 msgr2=0x7f9d381f5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:56.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.411+0000 7f9d3da00640 1 --2- 172.21.15.90:0/310260466 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d380d0b60 0x7f9d381f5430 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f9d2000e9e0 tx=0x7f9d2000eea0 comp rx=0 tx=0).stop 2024-03-24T01:25:56.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.411+0000 7f9d3da00640 1 -- 172.21.15.90:0/310260466 shutdown_connections 2024-03-24T01:25:56.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.411+0000 7f9d3da00640 1 --2- 172.21.15.90:0/310260466 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9d1406bbd0 0x7f9d1406e090 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:56.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.411+0000 7f9d3da00640 1 --2- 172.21.15.90:0/310260466 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d380d0b60 0x7f9d381f5430 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:56.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.411+0000 7f9d3da00640 1 --2- 172.21.15.90:0/310260466 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d3815dfe0 0x7f9d380c9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:56.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.411+0000 7f9d3da00640 1 --2- 172.21.15.90:0/310260466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d380b2e50 0x7f9d38150270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:56.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.411+0000 7f9d3da00640 1 -- 172.21.15.90:0/310260466 >> 172.21.15.90:0/310260466 conn(0x7f9d38072c10 msgr2=0x7f9d380d0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:56.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.411+0000 7f9d3da00640 1 -- 172.21.15.90:0/310260466 shutdown_connections 2024-03-24T01:25:56.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.411+0000 7f9d3da00640 1 -- 172.21.15.90:0/310260466 wait complete. 2024-03-24T01:25:56.430 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-24T01:25:56.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.635+0000 7fb8fb200640 1 Processor -- start 2024-03-24T01:25:56.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.635+0000 7fb8fb200640 1 -- start start 2024-03-24T01:25:56.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.637+0000 7fb8fb200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fb8fc0d1620 0x7fb8fc0cf210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:56.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.637+0000 7fb8fb200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb8fc06cc40 0x7fb8fc0b6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:56.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.637+0000 7fb8fb200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8fc0d1b20 con 0x7fb8fc06c860 2024-03-24T01:25:56.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.637+0000 7fb8fb200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8fc0d2730 con 0x7fb8fc06cc40 2024-03-24T01:25:56.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.637+0000 7fb8fb200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8fc0d1e00 con 0x7fb8fc0d1620 2024-03-24T01:25:56.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.637+0000 7fb8fa800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb8fc06cc40 0x7fb8fc0b6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:56.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.637+0000 7fb8fa800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb8fc06cc40 0x7fb8fc0b6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60372/0 (socket says 172.21.15.90:60372) 2024-03-24T01:25:56.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.637+0000 7fb8fa800640 1 -- 172.21.15.90:0/2590909984 learned_addr learned my addr 172.21.15.90:0/2590909984 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:56.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.638+0000 7fb8f9400640 1 --2- 172.21.15.90:0/2590909984 >> v2:172.21.15.120:3300/0 conn(0x7fb8fc0d1620 0x7fb8fc0cf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:56.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.638+0000 7fb8fa800640 1 -- 172.21.15.90:0/2590909984 >> v2:172.21.15.120:3300/0 conn(0x7fb8fc0d1620 msgr2=0x7fb8fc0cf210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:56.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.638+0000 7fb8fa800640 1 --2- 172.21.15.90:0/2590909984 >> v2:172.21.15.120:3300/0 conn(0x7fb8fc0d1620 0x7fb8fc0cf210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:56.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.638+0000 7fb8fa800640 1 -- 172.21.15.90:0/2590909984 >> v1:172.21.15.120:6789/0 conn(0x7fb8fc06c860 legacy=0x7fb8fc0d2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:56.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.638+0000 7fb8fa800640 1 -- 172.21.15.90:0/2590909984 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8fc0d20e0 con 0x7fb8fc06cc40 2024-03-24T01:25:56.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.639+0000 7fb8fa800640 1 --2- 172.21.15.90:0/2590909984 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8fc06cc40 0x7fb8fc0b6ff0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc0d28b0 tx=0x7fb8f402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4362d5b38dedec1b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:56.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.640+0000 7fb8f8a00640 1 -- 172.21.15.90:0/2590909984 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f4035020 con 0x7fb8fc06cc40 2024-03-24T01:25:56.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.640+0000 7fb8f8a00640 1 -- 172.21.15.90:0/2590909984 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8f4031870 con 0x7fb8fc06cc40 2024-03-24T01:25:56.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.640+0000 7fb8f8a00640 1 -- 172.21.15.90:0/2590909984 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f4031c40 con 0x7fb8fc06cc40 2024-03-24T01:25:56.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.640+0000 7fb8fb200640 1 -- 172.21.15.90:0/2590909984 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8fc06cc40 msgr2=0x7fb8fc0b6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:56.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.640+0000 7fb8fb200640 1 --2- 172.21.15.90:0/2590909984 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8fc06cc40 0x7fb8fc0b6ff0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc0d28b0 tx=0x7fb8f402b910 comp rx=0 tx=0).stop 2024-03-24T01:25:56.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.640+0000 7fb8fb200640 1 -- 172.21.15.90:0/2590909984 shutdown_connections 2024-03-24T01:25:56.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.640+0000 7fb8fb200640 1 --2- 172.21.15.90:0/2590909984 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8fc06cc40 0x7fb8fc0b6ff0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:56.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.640+0000 7fb8fb200640 1 --2- 172.21.15.90:0/2590909984 >> v2:172.21.15.120:3300/0 conn(0x7fb8fc0d1620 0x7fb8fc0cf210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:56.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.640+0000 7fb8fb200640 1 -- 172.21.15.90:0/2590909984 >> 172.21.15.90:0/2590909984 conn(0x7fb8fc0d0560 msgr2=0x7fb8fc0d0970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:56.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.641+0000 7fb8fb200640 1 -- 172.21.15.90:0/2590909984 shutdown_connections 2024-03-24T01:25:56.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.641+0000 7fb8fb200640 1 -- 172.21.15.90:0/2590909984 wait complete. 2024-03-24T01:25:56.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.642+0000 7fb8fb200640 1 Processor -- start 2024-03-24T01:25:56.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.642+0000 7fb8fb200640 1 -- start start 2024-03-24T01:25:56.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.643+0000 7fb8fb200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8fc06c860 0x7fb8fc15e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:56.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.643+0000 7fb8fb200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8fc06cc40 0x7fb8fc160b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:56.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.643+0000 7fb8fb200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8fc0d1620 0x7fb8fc1610a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:56.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.643+0000 7fb8fb200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8fc0c7300 con 0x7fb8fc06c860 2024-03-24T01:25:56.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.643+0000 7fb8fb200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8fc0c7180 con 0x7fb8fc06cc40 2024-03-24T01:25:56.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.643+0000 7fb8fb200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8fc0d32d0 con 0x7fb8fc0d1620 2024-03-24T01:25:56.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.643+0000 7fb8f9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8fc06c860 0x7fb8fc15e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:56.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.644+0000 7fb8f9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8fc06c860 0x7fb8fc15e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60380/0 (socket says 172.21.15.90:60380) 2024-03-24T01:25:56.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.644+0000 7fb8f9e00640 1 -- 172.21.15.90:0/2063701401 learned_addr learned my addr 172.21.15.90:0/2063701401 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:56.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.644+0000 7fb8f9e00640 1 -- 172.21.15.90:0/2063701401 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8fc0d1620 msgr2=0x7fb8fc1610a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:56.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.644+0000 7fb8f9e00640 1 --2- 172.21.15.90:0/2063701401 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8fc0d1620 0x7fb8fc1610a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:56.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.644+0000 7fb8f9e00640 1 -- 172.21.15.90:0/2063701401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8fc06cc40 msgr2=0x7fb8fc160b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:56.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.644+0000 7fb8f9e00640 1 --2- 172.21.15.90:0/2063701401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8fc06cc40 0x7fb8fc160b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:56.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.644+0000 7fb8f9e00640 1 -- 172.21.15.90:0/2063701401 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8fc15eaa0 con 0x7fb8fc06c860 2024-03-24T01:25:56.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.644+0000 7fb8f9e00640 1 --2- 172.21.15.90:0/2063701401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8fc06c860 0x7fb8fc15e560 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec00b3f0 tx=0x7fb8ec00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:56.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.644+0000 7fb8f2000640 1 -- 172.21.15.90:0/2063701401 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec015020 con 0x7fb8fc06c860 2024-03-24T01:25:56.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.644+0000 7fb8f2000640 1 -- 172.21.15.90:0/2063701401 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8ec011830 con 0x7fb8fc06c860 2024-03-24T01:25:56.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.645+0000 7fb8fb200640 1 -- 172.21.15.90:0/2063701401 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8fc15f2c0 con 0x7fb8fc06c860 2024-03-24T01:25:56.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.645+0000 7fb8f2000640 1 -- 172.21.15.90:0/2063701401 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec011c20 con 0x7fb8fc06c860 2024-03-24T01:25:56.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.645+0000 7fb8fb200640 1 -- 172.21.15.90:0/2063701401 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8fc15f880 con 0x7fb8fc06c860 2024-03-24T01:25:56.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.646+0000 7fb8fb200640 1 -- 172.21.15.90:0/2063701401 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8fc04f400 con 0x7fb8fc06c860 2024-03-24T01:25:56.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.647+0000 7fb8f2000640 1 -- 172.21.15.90:0/2063701401 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb8ec018020 con 0x7fb8fc06c860 2024-03-24T01:25:56.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.647+0000 7fb8f2000640 1 --2- 172.21.15.90:0/2063701401 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb8d806bbd0 0x7fb8d806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:56.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.648+0000 7fb8f2000640 1 -- 172.21.15.90:0/2063701401 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fb8ec08a4e0 con 0x7fb8fc06c860 2024-03-24T01:25:56.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.648+0000 7fb8f9400640 1 --2- 172.21.15.90:0/2063701401 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb8d806bbd0 0x7fb8d806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:56.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.649+0000 7fb8f9400640 1 --2- 172.21.15.90:0/2063701401 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb8d806bbd0 0x7fb8d806e090 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb8e40070b0 tx=0x7fb8e4007040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:56.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.652+0000 7fb8f2000640 1 -- 172.21.15.90:0/2063701401 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb8ec01f030 con 0x7fb8fc06c860 2024-03-24T01:25:56.731 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:25:56.738 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:25:56.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.852+0000 7fb8fb200640 1 -- 172.21.15.90:0/2063701401 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb8fc0ce3d0 con 0x7fb8fc06c860 2024-03-24T01:25:56.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.852+0000 7fb8f2000640 1 -- 172.21.15.90:0/2063701401 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v35) v1 ==== 84+0+65 (secure 0 0 0) 0x7fb8ec0172c0 con 0x7fb8fc06c860 2024-03-24T01:25:56.853 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:25:56.853 INFO:teuthology.orchestra.run.smithi090.stdout:[ 2024-03-24T01:25:56.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:25:56.853 INFO:teuthology.orchestra.run.smithi090.stdout: "poolnum": 1, 2024-03-24T01:25:56.853 INFO:teuthology.orchestra.run.smithi090.stdout: "poolname": ".mgr" 2024-03-24T01:25:56.853 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:25:56.853 INFO:teuthology.orchestra.run.smithi090.stdout:] 2024-03-24T01:25:56.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.855+0000 7fb8fb200640 1 -- 172.21.15.90:0/2063701401 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb8d806bbd0 msgr2=0x7fb8d806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:56.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.855+0000 7fb8fb200640 1 --2- 172.21.15.90:0/2063701401 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb8d806bbd0 0x7fb8d806e090 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb8e40070b0 tx=0x7fb8e4007040 comp rx=0 tx=0).stop 2024-03-24T01:25:56.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.855+0000 7fb8fb200640 1 -- 172.21.15.90:0/2063701401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8fc06c860 msgr2=0x7fb8fc15e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:56.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.855+0000 7fb8fb200640 1 --2- 172.21.15.90:0/2063701401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8fc06c860 0x7fb8fc15e560 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec00b3f0 tx=0x7fb8ec00b8b0 comp rx=0 tx=0).stop 2024-03-24T01:25:56.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.855+0000 7fb8fb200640 1 -- 172.21.15.90:0/2063701401 shutdown_connections 2024-03-24T01:25:56.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.855+0000 7fb8fb200640 1 --2- 172.21.15.90:0/2063701401 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb8d806bbd0 0x7fb8d806e090 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:56.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.855+0000 7fb8fb200640 1 --2- 172.21.15.90:0/2063701401 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb8fc0d1620 0x7fb8fc1610a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:56.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.855+0000 7fb8fb200640 1 --2- 172.21.15.90:0/2063701401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8fc06cc40 0x7fb8fc160b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:56.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.855+0000 7fb8fb200640 1 --2- 172.21.15.90:0/2063701401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8fc06c860 0x7fb8fc15e560 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:56.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.855+0000 7fb8fb200640 1 -- 172.21.15.90:0/2063701401 >> 172.21.15.90:0/2063701401 conn(0x7fb8fc0d0560 msgr2=0x7fb8fc0cef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:56.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.856+0000 7fb8fb200640 1 -- 172.21.15.90:0/2063701401 shutdown_connections 2024-03-24T01:25:56.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:56.856+0000 7fb8fb200640 1 -- 172.21.15.90:0/2063701401 wait complete. 2024-03-24T01:25:56.874 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-24T01:25:56.874 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-24T01:25:57.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.130+0000 7fa545400640 1 Processor -- start 2024-03-24T01:25:57.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.130+0000 7fa545400640 1 -- start start 2024-03-24T01:25:57.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.131+0000 7fa545400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa54015f150 0x7fa540144060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:57.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.131+0000 7fa545400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa54006e900 0x7fa54006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:57.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.131+0000 7fa545400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5400cf180 con 0x7fa5400cbe50 2024-03-24T01:25:57.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.131+0000 7fa545400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa54006f2b0 con 0x7fa54015f150 2024-03-24T01:25:57.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.131+0000 7fa545400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5400cf460 con 0x7fa54006e900 2024-03-24T01:25:57.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.132+0000 7fa544a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa54006e900 0x7fa54006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:57.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.132+0000 7fa537400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa54015f150 0x7fa540144060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:57.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.132+0000 7fa537400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa54015f150 0x7fa540144060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.90:59992/0 (socket says 172.21.15.90:59992) 2024-03-24T01:25:57.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.132+0000 7fa537400640 1 -- 172.21.15.90:0/2447850345 learned_addr learned my addr 172.21.15.90:0/2447850345 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:57.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.133+0000 7fa537400640 1 -- 172.21.15.90:0/2447850345 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa54006e900 msgr2=0x7fa54006ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:57.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.133+0000 7fa537400640 1 --2- 172.21.15.90:0/2447850345 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa54006e900 0x7fa54006ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:57.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.133+0000 7fa537400640 1 -- 172.21.15.90:0/2447850345 >> v1:172.21.15.120:6789/0 conn(0x7fa5400cbe50 legacy=0x7fa5400ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:57.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.133+0000 7fa537400640 1 -- 172.21.15.90:0/2447850345 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa54015e670 con 0x7fa54015f150 2024-03-24T01:25:57.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.134+0000 7fa537400640 1 --2- 172.21.15.90:0/2447850345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa54015f150 0x7fa540144060 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa528009870 tx=0x7fa52802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=400fd885a0622f66 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:57.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.135+0000 7fa53f400640 1 -- 172.21.15.90:0/2447850345 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa528034070 con 0x7fa54015f150 2024-03-24T01:25:57.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.135+0000 7fa53f400640 1 -- 172.21.15.90:0/2447850345 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa528030840 con 0x7fa54015f150 2024-03-24T01:25:57.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.135+0000 7fa53f400640 1 -- 172.21.15.90:0/2447850345 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa528030c10 con 0x7fa54015f150 2024-03-24T01:25:57.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.136+0000 7fa545400640 1 -- 172.21.15.90:0/2447850345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa54015f150 msgr2=0x7fa540144060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:57.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.136+0000 7fa545400640 1 --2- 172.21.15.90:0/2447850345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa54015f150 0x7fa540144060 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa528009870 tx=0x7fa52802b980 comp rx=0 tx=0).stop 2024-03-24T01:25:57.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.136+0000 7fa545400640 1 -- 172.21.15.90:0/2447850345 shutdown_connections 2024-03-24T01:25:57.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.136+0000 7fa545400640 1 --2- 172.21.15.90:0/2447850345 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa54006e900 0x7fa54006ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:57.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.136+0000 7fa545400640 1 --2- 172.21.15.90:0/2447850345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa54015f150 0x7fa540144060 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:57.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.136+0000 7fa545400640 1 -- 172.21.15.90:0/2447850345 >> 172.21.15.90:0/2447850345 conn(0x7fa54006b0c0 msgr2=0x7fa54006b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:57.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.136+0000 7fa545400640 1 -- 172.21.15.90:0/2447850345 shutdown_connections 2024-03-24T01:25:57.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.136+0000 7fa545400640 1 -- 172.21.15.90:0/2447850345 wait complete. 2024-03-24T01:25:57.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.138+0000 7fa545400640 1 Processor -- start 2024-03-24T01:25:57.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.138+0000 7fa545400640 1 -- start start 2024-03-24T01:25:57.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.139+0000 7fa545400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa54006e900 0x7fa540141a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:57.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.139+0000 7fa545400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5400cbe50 0x7fa5401f0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:57.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.139+0000 7fa545400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa54015f150 0x7fa540156280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:57.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.139+0000 7fa545400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5400cc230 con 0x7fa54015f150 2024-03-24T01:25:57.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.139+0000 7fa545400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54015ec30 con 0x7fa54006e900 2024-03-24T01:25:57.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.139+0000 7fa545400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5400cc3b0 con 0x7fa5400cbe50 2024-03-24T01:25:57.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.140+0000 7fa544a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa54015f150 0x7fa540156280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:57.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.140+0000 7fa544a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa54015f150 0x7fa540156280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60394/0 (socket says 172.21.15.90:60394) 2024-03-24T01:25:57.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.140+0000 7fa544a00640 1 -- 172.21.15.90:0/611750260 learned_addr learned my addr 172.21.15.90:0/611750260 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:57.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.140+0000 7fa537400640 1 --2- 172.21.15.90:0/611750260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5400cbe50 0x7fa5401f0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:57.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.140+0000 7fa544a00640 1 -- 172.21.15.90:0/611750260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5400cbe50 msgr2=0x7fa5401f0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:57.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.140+0000 7fa544a00640 1 --2- 172.21.15.90:0/611750260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5400cbe50 0x7fa5401f0e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:57.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.140+0000 7fa544a00640 1 -- 172.21.15.90:0/611750260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa54006e900 msgr2=0x7fa540141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:57.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.140+0000 7fa53fe00640 1 --2- 172.21.15.90:0/611750260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa54006e900 0x7fa540141a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:57.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.140+0000 7fa544a00640 1 --2- 172.21.15.90:0/611750260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa54006e900 0x7fa540141a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:57.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.140+0000 7fa544a00640 1 -- 172.21.15.90:0/611750260 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa54012e5a0 con 0x7fa54015f150 2024-03-24T01:25:57.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.140+0000 7fa53fe00640 1 --2- 172.21.15.90:0/611750260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa54006e900 0x7fa540141a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:57.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.140+0000 7fa544a00640 1 --2- 172.21.15.90:0/611750260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa54015f150 0x7fa540156280 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fa53800efc0 tx=0x7fa53800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:57.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.141+0000 7fa53cc00640 1 -- 172.21.15.90:0/611750260 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa538018020 con 0x7fa54015f150 2024-03-24T01:25:57.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.141+0000 7fa53cc00640 1 -- 172.21.15.90:0/611750260 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa538014480 con 0x7fa54015f150 2024-03-24T01:25:57.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.141+0000 7fa53cc00640 1 -- 172.21.15.90:0/611750260 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa538014870 con 0x7fa54015f150 2024-03-24T01:25:57.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.141+0000 7fa545400640 1 -- 172.21.15.90:0/611750260 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa54012e880 con 0x7fa54015f150 2024-03-24T01:25:57.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.141+0000 7fa545400640 1 -- 172.21.15.90:0/611750260 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa54012ee40 con 0x7fa54015f150 2024-03-24T01:25:57.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.142+0000 7fa545400640 1 -- 172.21.15.90:0/611750260 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5400cd6e0 con 0x7fa54015f150 2024-03-24T01:25:57.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.143+0000 7fa53cc00640 1 -- 172.21.15.90:0/611750260 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa538005d10 con 0x7fa54015f150 2024-03-24T01:25:57.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.144+0000 7fa53cc00640 1 --2- 172.21.15.90:0/611750260 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa51c06bbd0 0x7fa51c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:57.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.144+0000 7fa53cc00640 1 -- 172.21.15.90:0/611750260 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fa53808b3e0 con 0x7fa54015f150 2024-03-24T01:25:57.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.145+0000 7fa53fe00640 1 --2- 172.21.15.90:0/611750260 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa51c06bbd0 0x7fa51c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:57.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.146+0000 7fa53fe00640 1 --2- 172.21.15.90:0/611750260 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa51c06bbd0 0x7fa51c06e090 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa530001680 tx=0x7fa530009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:57.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.149+0000 7fa53cc00640 1 -- 172.21.15.90:0/611750260 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa5400cd6e0 con 0x7fa54015f150 2024-03-24T01:25:57.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.346+0000 7fa545400640 1 -- 172.21.15.90:0/611750260 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fa54004f400 con 0x7fa54015f150 2024-03-24T01:25:57.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.946+0000 7fa53cc00640 1 -- 172.21.15.90:0/611750260 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v36) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa538022070 con 0x7fa54015f150 2024-03-24T01:25:57.947 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_metadata' created 2024-03-24T01:25:57.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.950+0000 7fa545400640 1 -- 172.21.15.90:0/611750260 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa51c06bbd0 msgr2=0x7fa51c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:57.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.950+0000 7fa545400640 1 --2- 172.21.15.90:0/611750260 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa51c06bbd0 0x7fa51c06e090 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa530001680 tx=0x7fa530009040 comp rx=0 tx=0).stop 2024-03-24T01:25:57.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.950+0000 7fa545400640 1 -- 172.21.15.90:0/611750260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa54015f150 msgr2=0x7fa540156280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:57.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.950+0000 7fa545400640 1 --2- 172.21.15.90:0/611750260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa54015f150 0x7fa540156280 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fa53800efc0 tx=0x7fa53800c490 comp rx=0 tx=0).stop 2024-03-24T01:25:57.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.950+0000 7fa545400640 1 -- 172.21.15.90:0/611750260 shutdown_connections 2024-03-24T01:25:57.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.950+0000 7fa545400640 1 --2- 172.21.15.90:0/611750260 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa51c06bbd0 0x7fa51c06e090 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:57.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.950+0000 7fa545400640 1 --2- 172.21.15.90:0/611750260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa54015f150 0x7fa540156280 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:57.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.950+0000 7fa545400640 1 --2- 172.21.15.90:0/611750260 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa5400cbe50 0x7fa5401f0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:57.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.950+0000 7fa545400640 1 --2- 172.21.15.90:0/611750260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa54006e900 0x7fa540141a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:57.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.950+0000 7fa545400640 1 -- 172.21.15.90:0/611750260 >> 172.21.15.90:0/611750260 conn(0x7fa54006b0c0 msgr2=0x7fa540071790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:57.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.951+0000 7fa545400640 1 -- 172.21.15.90:0/611750260 shutdown_connections 2024-03-24T01:25:57.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:57.951+0000 7fa545400640 1 -- 172.21.15.90:0/611750260 wait complete. 2024-03-24T01:25:57.979 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-24T01:25:58.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.184+0000 7f0238c00640 1 Processor -- start 2024-03-24T01:25:58.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.185+0000 7f0238c00640 1 -- start start 2024-03-24T01:25:58.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.186+0000 7f0238c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f02340d0b50 0x7f02340d0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:58.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.186+0000 7f0238c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02340b2e40 0x7f02340d28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:58.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.186+0000 7f0238c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02340d20e0 con 0x7f023415dfd0 2024-03-24T01:25:58.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.186+0000 7f0238c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02340d1470 con 0x7f02340d0b50 2024-03-24T01:25:58.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.186+0000 7f0238c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02340ced70 con 0x7f02340b2e40 2024-03-24T01:25:58.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.186+0000 7f0231e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f02340d0b50 0x7f02340d0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:58.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.187+0000 7f0231e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f02340d0b50 0x7f02340d0f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60404/0 (socket says 172.21.15.90:60404) 2024-03-24T01:25:58.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.187+0000 7f0231e00640 1 -- 172.21.15.90:0/630821438 learned_addr learned my addr 172.21.15.90:0/630821438 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:58.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.187+0000 7f0232800640 1 --2- 172.21.15.90:0/630821438 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02340b2e40 0x7f02340d28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:58.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.187+0000 7f0231e00640 1 -- 172.21.15.90:0/630821438 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02340b2e40 msgr2=0x7f02340d28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:58.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.187+0000 7f0231e00640 1 --2- 172.21.15.90:0/630821438 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02340b2e40 0x7f02340d28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:58.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.187+0000 7f0231e00640 1 -- 172.21.15.90:0/630821438 >> v1:172.21.15.120:6789/0 conn(0x7f023415dfd0 legacy=0x7f02341622b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:58.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.187+0000 7f0231e00640 1 -- 172.21.15.90:0/630821438 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02340cf050 con 0x7f02340d0b50 2024-03-24T01:25:58.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.188+0000 7f0231e00640 1 --2- 172.21.15.90:0/630821438 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02340d0b50 0x7f02340d0f30 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f021c009870 tx=0x7f021c02b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25f420656525b66c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:58.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.188+0000 7f0230a00640 1 -- 172.21.15.90:0/630821438 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f021c038070 con 0x7f02340d0b50 2024-03-24T01:25:58.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.189+0000 7f0230a00640 1 -- 172.21.15.90:0/630821438 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f021c031740 con 0x7f02340d0b50 2024-03-24T01:25:58.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.189+0000 7f0230a00640 1 -- 172.21.15.90:0/630821438 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f021c02f050 con 0x7f02340d0b50 2024-03-24T01:25:58.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.189+0000 7f0238c00640 1 -- 172.21.15.90:0/630821438 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02340d0b50 msgr2=0x7f02340d0f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:58.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.189+0000 7f0238c00640 1 --2- 172.21.15.90:0/630821438 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02340d0b50 0x7f02340d0f30 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f021c009870 tx=0x7f021c02b7e0 comp rx=0 tx=0).stop 2024-03-24T01:25:58.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.189+0000 7f0238c00640 1 -- 172.21.15.90:0/630821438 shutdown_connections 2024-03-24T01:25:58.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.189+0000 7f0238c00640 1 --2- 172.21.15.90:0/630821438 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f02340b2e40 0x7f02340d28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:58.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.189+0000 7f0238c00640 1 --2- 172.21.15.90:0/630821438 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02340d0b50 0x7f02340d0f30 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:58.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.189+0000 7f0238c00640 1 -- 172.21.15.90:0/630821438 >> 172.21.15.90:0/630821438 conn(0x7f0234072c10 msgr2=0x7f0234073020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:58.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.189+0000 7f0238c00640 1 -- 172.21.15.90:0/630821438 shutdown_connections 2024-03-24T01:25:58.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.189+0000 7f0238c00640 1 -- 172.21.15.90:0/630821438 wait complete. 2024-03-24T01:25:58.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.191+0000 7f0238c00640 1 Processor -- start 2024-03-24T01:25:58.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.191+0000 7f0238c00640 1 -- start start 2024-03-24T01:25:58.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.192+0000 7f0238c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02340b2e40 0x7f02341e9890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:58.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.192+0000 7f0238c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02340d0b50 0x7f023415f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:58.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.192+0000 7f0238c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f023415dfd0 0x7f023415fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:58.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.192+0000 7f0238c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02340b6d80 con 0x7f02340b2e40 2024-03-24T01:25:58.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.192+0000 7f0238c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02340cf330 con 0x7f02340d0b50 2024-03-24T01:25:58.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.192+0000 7f0238c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02340b6f00 con 0x7f023415dfd0 2024-03-24T01:25:58.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.192+0000 7f0231e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02340b2e40 0x7f02341e9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:58.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.192+0000 7f0231e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02340b2e40 0x7f02341e9890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60418/0 (socket says 172.21.15.90:60418) 2024-03-24T01:25:58.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.192+0000 7f0231e00640 1 -- 172.21.15.90:0/2902261791 learned_addr learned my addr 172.21.15.90:0/2902261791 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:58.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.193+0000 7f0231400640 1 --2- 172.21.15.90:0/2902261791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02340d0b50 0x7f023415f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:58.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.193+0000 7f0232800640 1 --2- 172.21.15.90:0/2902261791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f023415dfd0 0x7f023415fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:58.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.193+0000 7f0231e00640 1 -- 172.21.15.90:0/2902261791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f023415dfd0 msgr2=0x7f023415fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:58.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.193+0000 7f0231e00640 1 --2- 172.21.15.90:0/2902261791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f023415dfd0 0x7f023415fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:58.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.193+0000 7f0231e00640 1 -- 172.21.15.90:0/2902261791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02340d0b50 msgr2=0x7f023415f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:58.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.193+0000 7f0231e00640 1 --2- 172.21.15.90:0/2902261791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02340d0b50 0x7f023415f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:58.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.193+0000 7f0231e00640 1 -- 172.21.15.90:0/2902261791 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02341ea1e0 con 0x7f02340b2e40 2024-03-24T01:25:58.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.193+0000 7f0231e00640 1 --2- 172.21.15.90:0/2902261791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02340b2e40 0x7f02341e9890 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f021c000c00 tx=0x7f021c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:58.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.193+0000 7f022a000640 1 -- 172.21.15.90:0/2902261791 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f021c042070 con 0x7f02340b2e40 2024-03-24T01:25:58.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.194+0000 7f022a000640 1 -- 172.21.15.90:0/2902261791 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f021c031e80 con 0x7f02340b2e40 2024-03-24T01:25:58.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.194+0000 7f0238c00640 1 -- 172.21.15.90:0/2902261791 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02341ea4c0 con 0x7f02340b2e40 2024-03-24T01:25:58.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.194+0000 7f022a000640 1 -- 172.21.15.90:0/2902261791 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f021c038040 con 0x7f02340b2e40 2024-03-24T01:25:58.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.194+0000 7f0238c00640 1 -- 172.21.15.90:0/2902261791 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02341f5890 con 0x7f02340b2e40 2024-03-24T01:25:58.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.195+0000 7f0238c00640 1 -- 172.21.15.90:0/2902261791 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f023404f400 con 0x7f02340b2e40 2024-03-24T01:25:58.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.196+0000 7f022a000640 1 -- 172.21.15.90:0/2902261791 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f021c045070 con 0x7f02340b2e40 2024-03-24T01:25:58.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.197+0000 7f022a000640 1 --2- 172.21.15.90:0/2902261791 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f021006bbd0 0x7f021006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:58.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.197+0000 7f022a000640 1 -- 172.21.15.90:0/2902261791 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f021c0ac750 con 0x7f02340b2e40 2024-03-24T01:25:58.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.197+0000 7f0231400640 1 --2- 172.21.15.90:0/2902261791 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f021006bbd0 0x7f021006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:58.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.198+0000 7f0231400640 1 --2- 172.21.15.90:0/2902261791 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f021006bbd0 0x7f021006e090 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f02340c7430 tx=0x7f02240072f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:58.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.202+0000 7f022a000640 1 -- 172.21.15.90:0/2902261791 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f021c04c030 con 0x7f02340b2e40 2024-03-24T01:25:58.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.400+0000 7f0238c00640 1 -- 172.21.15.90:0/2902261791 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f02340b3320 con 0x7f02340b2e40 2024-03-24T01:25:58.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.962+0000 7f022a000640 1 -- 172.21.15.90:0/2902261791 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v37) v1 ==== 170+0+0 (secure 0 0 0) 0x7f021c07b280 con 0x7f02340b2e40 2024-03-24T01:25:58.963 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_data' created 2024-03-24T01:25:58.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.967+0000 7f0238c00640 1 -- 172.21.15.90:0/2902261791 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f021006bbd0 msgr2=0x7f021006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:58.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.967+0000 7f0238c00640 1 --2- 172.21.15.90:0/2902261791 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f021006bbd0 0x7f021006e090 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f02340c7430 tx=0x7f02240072f0 comp rx=0 tx=0).stop 2024-03-24T01:25:58.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.967+0000 7f0238c00640 1 -- 172.21.15.90:0/2902261791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02340b2e40 msgr2=0x7f02341e9890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:58.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.967+0000 7f0238c00640 1 --2- 172.21.15.90:0/2902261791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02340b2e40 0x7f02341e9890 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f021c000c00 tx=0x7f021c0046c0 comp rx=0 tx=0).stop 2024-03-24T01:25:58.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.967+0000 7f0238c00640 1 -- 172.21.15.90:0/2902261791 shutdown_connections 2024-03-24T01:25:58.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.967+0000 7f0238c00640 1 --2- 172.21.15.90:0/2902261791 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f021006bbd0 0x7f021006e090 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:58.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.967+0000 7f0238c00640 1 --2- 172.21.15.90:0/2902261791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f023415dfd0 0x7f023415fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:58.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.967+0000 7f0238c00640 1 --2- 172.21.15.90:0/2902261791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f02340d0b50 0x7f023415f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:58.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.967+0000 7f0238c00640 1 --2- 172.21.15.90:0/2902261791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02340b2e40 0x7f02341e9890 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:58.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.967+0000 7f0238c00640 1 -- 172.21.15.90:0/2902261791 >> 172.21.15.90:0/2902261791 conn(0x7f0234072c10 msgr2=0x7f023415bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:58.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.968+0000 7f0238c00640 1 -- 172.21.15.90:0/2902261791 shutdown_connections 2024-03-24T01:25:58.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:58.968+0000 7f0238c00640 1 -- 172.21.15.90:0/2902261791 wait complete. 2024-03-24T01:25:59.001 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-24T01:25:59.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.184+0000 7fdb2ee00640 1 Processor -- start 2024-03-24T01:25:59.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.184+0000 7fdb2ee00640 1 -- start start 2024-03-24T01:25:59.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.186+0000 7fdb2ee00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdb28145170 0x7fdb280cf810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:59.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.186+0000 7fdb2ee00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb280cf190 0x7fdb280cfd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:59.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.186+0000 7fdb2ee00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb2806d990 con 0x7fdb2806ea30 2024-03-24T01:25:59.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.186+0000 7fdb2ee00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb2806dc70 con 0x7fdb28145170 2024-03-24T01:25:59.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.186+0000 7fdb2ee00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2806df50 con 0x7fdb280cf190 2024-03-24T01:25:59.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.186+0000 7fdb2da00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdb28145170 0x7fdb280cf810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:59.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.186+0000 7fdb2da00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdb28145170 0x7fdb280cf810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60422/0 (socket says 172.21.15.90:60422) 2024-03-24T01:25:59.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.186+0000 7fdb2da00640 1 -- 172.21.15.90:0/2374022116 learned_addr learned my addr 172.21.15.90:0/2374022116 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:59.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.186+0000 7fdb2da00640 1 -- 172.21.15.90:0/2374022116 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb280cf190 msgr2=0x7fdb280cfd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:59.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.187+0000 7fdb2e400640 1 --2- 172.21.15.90:0/2374022116 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb280cf190 0x7fdb280cfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:59.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.187+0000 7fdb2da00640 1 --2- 172.21.15.90:0/2374022116 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb280cf190 0x7fdb280cfd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:59.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.187+0000 7fdb2da00640 1 -- 172.21.15.90:0/2374022116 >> v1:172.21.15.120:6789/0 conn(0x7fdb2806ea30 legacy=0x7fdb280cea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:25:59.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.187+0000 7fdb2da00640 1 -- 172.21.15.90:0/2374022116 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb2806e230 con 0x7fdb28145170 2024-03-24T01:25:59.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.188+0000 7fdb2da00640 1 --2- 172.21.15.90:0/2374022116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb28145170 0x7fdb280cf810 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c009870 tx=0x7fdb1c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=972434d4b782ab26 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:59.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.188+0000 7fdb27e00640 1 -- 172.21.15.90:0/2374022116 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c035020 con 0x7fdb28145170 2024-03-24T01:25:59.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.188+0000 7fdb27e00640 1 -- 172.21.15.90:0/2374022116 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb1c031640 con 0x7fdb28145170 2024-03-24T01:25:59.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.188+0000 7fdb27e00640 1 -- 172.21.15.90:0/2374022116 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c02d040 con 0x7fdb28145170 2024-03-24T01:25:59.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.189+0000 7fdb2ee00640 1 -- 172.21.15.90:0/2374022116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb28145170 msgr2=0x7fdb280cf810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:59.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.189+0000 7fdb2ee00640 1 --2- 172.21.15.90:0/2374022116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb28145170 0x7fdb280cf810 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c009870 tx=0x7fdb1c02b8f0 comp rx=0 tx=0).stop 2024-03-24T01:25:59.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.189+0000 7fdb2ee00640 1 -- 172.21.15.90:0/2374022116 shutdown_connections 2024-03-24T01:25:59.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.189+0000 7fdb2ee00640 1 --2- 172.21.15.90:0/2374022116 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb280cf190 0x7fdb280cfd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:59.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.189+0000 7fdb2ee00640 1 --2- 172.21.15.90:0/2374022116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb28145170 0x7fdb280cf810 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:59.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.189+0000 7fdb2ee00640 1 -- 172.21.15.90:0/2374022116 >> 172.21.15.90:0/2374022116 conn(0x7fdb2806b0c0 msgr2=0x7fdb2806b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:59.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.189+0000 7fdb2ee00640 1 -- 172.21.15.90:0/2374022116 shutdown_connections 2024-03-24T01:25:59.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.189+0000 7fdb2ee00640 1 -- 172.21.15.90:0/2374022116 wait complete. 2024-03-24T01:25:59.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.191+0000 7fdb2ee00640 1 Processor -- start 2024-03-24T01:25:59.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.191+0000 7fdb2ee00640 1 -- start start 2024-03-24T01:25:59.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.192+0000 7fdb2ee00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb2806ea30 0x7fdb281422b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:59.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.192+0000 7fdb2ee00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb280cf190 0x7fdb281e42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:59.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.192+0000 7fdb2ee00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb28145170 0x7fdb28140060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:59.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.192+0000 7fdb2ee00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb281456d0 con 0x7fdb28145170 2024-03-24T01:25:59.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.192+0000 7fdb2ee00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb28145550 con 0x7fdb280cf190 2024-03-24T01:25:59.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.192+0000 7fdb2ee00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2815c0a0 con 0x7fdb2806ea30 2024-03-24T01:25:59.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.192+0000 7fdb2e400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb28145170 0x7fdb28140060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:59.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.192+0000 7fdb2e400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb28145170 0x7fdb28140060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60424/0 (socket says 172.21.15.90:60424) 2024-03-24T01:25:59.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.192+0000 7fdb2e400640 1 -- 172.21.15.90:0/3894401401 learned_addr learned my addr 172.21.15.90:0/3894401401 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:25:59.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.192+0000 7fdb2da00640 1 --2- 172.21.15.90:0/3894401401 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb2806ea30 0x7fdb281422b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:59.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.192+0000 7fdb2d000640 1 --2- 172.21.15.90:0/3894401401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb280cf190 0x7fdb281e42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:59.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.192+0000 7fdb2e400640 1 -- 172.21.15.90:0/3894401401 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb2806ea30 msgr2=0x7fdb281422b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:59.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.192+0000 7fdb2e400640 1 --2- 172.21.15.90:0/3894401401 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb2806ea30 0x7fdb281422b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:59.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.192+0000 7fdb2e400640 1 -- 172.21.15.90:0/3894401401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb280cf190 msgr2=0x7fdb281e42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:59.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.192+0000 7fdb2e400640 1 --2- 172.21.15.90:0/3894401401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb280cf190 0x7fdb281e42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:59.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.192+0000 7fdb2e400640 1 -- 172.21.15.90:0/3894401401 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb281427f0 con 0x7fdb28145170 2024-03-24T01:25:59.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.192+0000 7fdb2da00640 1 --2- 172.21.15.90:0/3894401401 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb2806ea30 0x7fdb281422b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:25:59.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.193+0000 7fdb2e400640 1 --2- 172.21.15.90:0/3894401401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb28145170 0x7fdb28140060 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fdb2000cd70 tx=0x7fdb2000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:59.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.193+0000 7fdb25600640 1 -- 172.21.15.90:0/3894401401 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb20003b90 con 0x7fdb28145170 2024-03-24T01:25:59.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.193+0000 7fdb25600640 1 -- 172.21.15.90:0/3894401401 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb20003d30 con 0x7fdb28145170 2024-03-24T01:25:59.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.193+0000 7fdb2ee00640 1 -- 172.21.15.90:0/3894401401 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb281f4360 con 0x7fdb28145170 2024-03-24T01:25:59.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.193+0000 7fdb25600640 1 -- 172.21.15.90:0/3894401401 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb20004240 con 0x7fdb28145170 2024-03-24T01:25:59.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.193+0000 7fdb2ee00640 1 -- 172.21.15.90:0/3894401401 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb281f4920 con 0x7fdb28145170 2024-03-24T01:25:59.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.194+0000 7fdb2ee00640 1 -- 172.21.15.90:0/3894401401 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb2806f950 con 0x7fdb28145170 2024-03-24T01:25:59.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.196+0000 7fdb25600640 1 -- 172.21.15.90:0/3894401401 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdb200043e0 con 0x7fdb28145170 2024-03-24T01:25:59.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.197+0000 7fdb25600640 1 --2- 172.21.15.90:0/3894401401 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fdb0806bbd0 0x7fdb0806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:59.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.197+0000 7fdb25600640 1 -- 172.21.15.90:0/3894401401 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6001+0+0 (secure 0 0 0) 0x7fdb2008b740 con 0x7fdb28145170 2024-03-24T01:25:59.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.197+0000 7fdb2da00640 1 --2- 172.21.15.90:0/3894401401 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fdb0806bbd0 0x7fdb0806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:59.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.199+0000 7fdb2da00640 1 --2- 172.21.15.90:0/3894401401 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fdb0806bbd0 0x7fdb0806e090 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c036000 tx=0x7fdb1c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:59.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.201+0000 7fdb25600640 1 -- 172.21.15.90:0/3894401401 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdb2806f950 con 0x7fdb28145170 2024-03-24T01:25:59.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.437+0000 7fdb2ee00640 1 -- 172.21.15.90:0/3894401401 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fdb2804f400 con 0x7fdb28145170 2024-03-24T01:25:59.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.983+0000 7fdb25600640 1 -- 172.21.15.90:0/3894401401 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v13) v1 ==== 172+0+0 (secure 0 0 0) 0x7fdb2005a060 con 0x7fdb28145170 2024-03-24T01:25:59.988 INFO:teuthology.orchestra.run.smithi090.stderr:new fs with metadata pool 5 and data pool 6 2024-03-24T01:25:59.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.996+0000 7fdb2ee00640 1 -- 172.21.15.90:0/3894401401 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fdb0806bbd0 msgr2=0x7fdb0806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:59.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.996+0000 7fdb2ee00640 1 --2- 172.21.15.90:0/3894401401 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fdb0806bbd0 0x7fdb0806e090 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c036000 tx=0x7fdb1c008030 comp rx=0 tx=0).stop 2024-03-24T01:25:59.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.996+0000 7fdb2ee00640 1 -- 172.21.15.90:0/3894401401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb28145170 msgr2=0x7fdb28140060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:25:59.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.996+0000 7fdb2ee00640 1 --2- 172.21.15.90:0/3894401401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb28145170 0x7fdb28140060 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fdb2000cd70 tx=0x7fdb2000a660 comp rx=0 tx=0).stop 2024-03-24T01:25:59.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.996+0000 7fdb2ee00640 1 -- 172.21.15.90:0/3894401401 shutdown_connections 2024-03-24T01:25:59.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.996+0000 7fdb2ee00640 1 --2- 172.21.15.90:0/3894401401 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fdb0806bbd0 0x7fdb0806e090 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:59.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.996+0000 7fdb2ee00640 1 --2- 172.21.15.90:0/3894401401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb28145170 0x7fdb28140060 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:59.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.996+0000 7fdb2ee00640 1 --2- 172.21.15.90:0/3894401401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb280cf190 0x7fdb281e42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:59.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.996+0000 7fdb2ee00640 1 --2- 172.21.15.90:0/3894401401 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdb2806ea30 0x7fdb281422b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:59.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.996+0000 7fdb2ee00640 1 -- 172.21.15.90:0/3894401401 >> 172.21.15.90:0/3894401401 conn(0x7fdb2806b0c0 msgr2=0x7fdb280cf5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:25:59.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.996+0000 7fdb2ee00640 1 -- 172.21.15.90:0/3894401401 shutdown_connections 2024-03-24T01:25:59.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:25:59.997+0000 7fdb2ee00640 1 -- 172.21.15.90:0/3894401401 wait complete. 2024-03-24T01:26:00.033 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:26:00.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.206+0000 7f1521200640 1 Processor -- start 2024-03-24T01:26:00.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.207+0000 7f1521200640 1 -- start start 2024-03-24T01:26:00.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.208+0000 7f1521200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f151c071790 0x7f151c071c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:00.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.208+0000 7f1521200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f151c06f620 con 0x7f151c145260 2024-03-24T01:26:00.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.208+0000 7f1521200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f151c06db00 con 0x7f151c159dc0 2024-03-24T01:26:00.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.208+0000 7f1521200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f151c072140 con 0x7f151c071790 2024-03-24T01:26:00.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.208+0000 7f151b400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f151c145260 0x7f151c06ef10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50062/0 (socket says 172.21.15.90:50062) 2024-03-24T01:26:00.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.208+0000 7f151b400640 1 -- 172.21.15.90:0/3208468718 learned_addr learned my addr 172.21.15.90:0/3208468718 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:26:00.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.209+0000 7f151a000640 1 -- 172.21.15.90:0/3208468718 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2936126796 0 0) 0x7f151c06f620 con 0x7f151c145260 2024-03-24T01:26:00.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.209+0000 7f151aa00640 1 --2- 172.21.15.90:0/3208468718 >> v2:172.21.15.120:3300/0 conn(0x7f151c071790 0x7f151c071c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:00.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.209+0000 7f151a000640 1 -- 172.21.15.90:0/3208468718 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1500003520 con 0x7f151c145260 2024-03-24T01:26:00.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.209+0000 7f151a000640 1 -- 172.21.15.90:0/3208468718 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3475362449 0 0) 0x7f1500003520 con 0x7f151c145260 2024-03-24T01:26:00.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.209+0000 7f151a000640 1 -- 172.21.15.90:0/3208468718 >> v2:172.21.15.120:3300/0 conn(0x7f151c071790 msgr2=0x7f151c071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:00.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.209+0000 7f151a000640 1 --2- 172.21.15.90:0/3208468718 >> v2:172.21.15.120:3300/0 conn(0x7f151c071790 0x7f151c071c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:00.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.209+0000 7f151a000640 1 -- 172.21.15.90:0/3208468718 >> v1:172.21.15.120:6789/0 conn(0x7f151c159dc0 legacy=0x7f151c15a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:00.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.209+0000 7f151a000640 1 -- 172.21.15.90:0/3208468718 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f151c06dde0 con 0x7f151c145260 2024-03-24T01:26:00.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.209+0000 7f151a000640 1 -- 172.21.15.90:0/3208468718 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f150c001670 con 0x7f151c145260 2024-03-24T01:26:00.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.209+0000 7f151a000640 1 -- 172.21.15.90:0/3208468718 >> v1:172.21.15.90:6789/0 conn(0x7f151c145260 legacy=0x7f151c06ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:00.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.209+0000 7f151a000640 1 --2- 172.21.15.90:0/3208468718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1500003a00 0x7f1500003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:00.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.209+0000 7f151a000640 1 --2- 172.21.15.90:0/3208468718 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1500004800 0x7f1500004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:00.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.209+0000 7f151a000640 1 --2- 172.21.15.90:0/3208468718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15000051c0 0x7f150000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:00.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.209+0000 7f151a000640 1 -- 172.21.15.90:0/3208468718 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151c072140 con 0x7f1500003a00 2024-03-24T01:26:00.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.209+0000 7f151a000640 1 -- 172.21.15.90:0/3208468718 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150000c0c0 con 0x7f15000051c0 2024-03-24T01:26:00.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.209+0000 7f151a000640 1 -- 172.21.15.90:0/3208468718 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f150000c2c0 con 0x7f1500004800 2024-03-24T01:26:00.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.210+0000 7f151b400640 1 --2- 172.21.15.90:0/3208468718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1500003a00 0x7f1500003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:00.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.210+0000 7f151b400640 1 -- 172.21.15.90:0/3208468718 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1500004800 msgr2=0x7f1500004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:00.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.210+0000 7f151b400640 1 --2- 172.21.15.90:0/3208468718 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1500004800 0x7f1500004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:00.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.210+0000 7f151b400640 1 -- 172.21.15.90:0/3208468718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15000051c0 msgr2=0x7f150000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:00.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.210+0000 7f151b400640 1 --2- 172.21.15.90:0/3208468718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15000051c0 0x7f150000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:00.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.210+0000 7f151b400640 1 -- 172.21.15.90:0/3208468718 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1500003520 con 0x7f1500003a00 2024-03-24T01:26:00.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.210+0000 7f151be00640 1 -- 172.21.15.90:0/3208468718 reap_dead start 2024-03-24T01:26:00.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.211+0000 7f151b400640 1 --2- 172.21.15.90:0/3208468718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1500003a00 0x7f1500003e20 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f150c00c110 tx=0x7f1500005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1397b54c4ba26d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:00.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.211+0000 7f151a000640 1 -- 172.21.15.90:0/3208468718 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150c007c40 con 0x7f1500003a00 2024-03-24T01:26:00.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.211+0000 7f151a000640 1 -- 172.21.15.90:0/3208468718 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f150c030070 con 0x7f1500003a00 2024-03-24T01:26:00.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.211+0000 7f151a000640 1 -- 172.21.15.90:0/3208468718 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150c005d70 con 0x7f1500003a00 2024-03-24T01:26:00.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.212+0000 7f1521200640 1 -- 172.21.15.90:0/3208468718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1500003a00 msgr2=0x7f1500003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:00.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.212+0000 7f1521200640 1 --2- 172.21.15.90:0/3208468718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1500003a00 0x7f1500003e20 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f150c00c110 tx=0x7f1500005830 comp rx=0 tx=0).stop 2024-03-24T01:26:00.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.212+0000 7f1521200640 1 -- 172.21.15.90:0/3208468718 shutdown_connections 2024-03-24T01:26:00.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.212+0000 7f1521200640 1 --2- 172.21.15.90:0/3208468718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1500003a00 0x7f1500003e20 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:00.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.212+0000 7f1521200640 1 -- 172.21.15.90:0/3208468718 >> 172.21.15.90:0/3208468718 conn(0x7f151c06b0c0 msgr2=0x7f151c06b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:00.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.212+0000 7f1521200640 1 -- 172.21.15.90:0/3208468718 shutdown_connections 2024-03-24T01:26:00.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.212+0000 7f1521200640 1 -- 172.21.15.90:0/3208468718 wait complete. 2024-03-24T01:26:00.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.214+0000 7f1521200640 1 Processor -- start 2024-03-24T01:26:00.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.214+0000 7f1521200640 1 -- start start 2024-03-24T01:26:00.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.215+0000 7f1521200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1500003a00 0x7f151c1591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:00.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.215+0000 7f1521200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f151c0b0470 0x7f151c0b08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:00.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.215+0000 7f1521200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f151c13d250 0x7f151c13d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:00.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.215+0000 7f1521200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151c06e710 con 0x7f1500003a00 2024-03-24T01:26:00.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.215+0000 7f1521200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151c06e3a0 con 0x7f151c13d250 2024-03-24T01:26:00.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.215+0000 7f1521200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f151c06d7f0 con 0x7f151c0b0470 2024-03-24T01:26:00.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.215+0000 7f151b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1500003a00 0x7f151c1591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:00.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.215+0000 7f151b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1500003a00 0x7f151c1591d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59926/0 (socket says 172.21.15.90:59926) 2024-03-24T01:26:00.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.215+0000 7f151b400640 1 -- 172.21.15.90:0/2325714860 learned_addr learned my addr 172.21.15.90:0/2325714860 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:00.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.215+0000 7f151b400640 1 -- 172.21.15.90:0/2325714860 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f151c0b0470 msgr2=0x7f151c0b08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:00.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.215+0000 7f151b400640 1 --2- 172.21.15.90:0/2325714860 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f151c0b0470 0x7f151c0b08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:00.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.215+0000 7f151b400640 1 -- 172.21.15.90:0/2325714860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f151c13d250 msgr2=0x7f151c13d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:00.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.215+0000 7f151b400640 1 --2- 172.21.15.90:0/2325714860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f151c13d250 0x7f151c13d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:00.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.215+0000 7f151b400640 1 -- 172.21.15.90:0/2325714860 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f151c159710 con 0x7f1500003a00 2024-03-24T01:26:00.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.216+0000 7f151b400640 1 --2- 172.21.15.90:0/2325714860 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1500003a00 0x7f151c1591d0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f150c00c200 tx=0x7f150c02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:00.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.216+0000 7f1513400640 1 -- 172.21.15.90:0/2325714860 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150c007c40 con 0x7f1500003a00 2024-03-24T01:26:00.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.216+0000 7f1513400640 1 -- 172.21.15.90:0/2325714860 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f150c004bf0 con 0x7f1500003a00 2024-03-24T01:26:00.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.216+0000 7f1513400640 1 -- 172.21.15.90:0/2325714860 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150c007070 con 0x7f1500003a00 2024-03-24T01:26:00.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.216+0000 7f1521200640 1 -- 172.21.15.90:0/2325714860 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f151c0b0ef0 con 0x7f1500003a00 2024-03-24T01:26:00.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.216+0000 7f1521200640 1 -- 172.21.15.90:0/2325714860 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f151c1393f0 con 0x7f1500003a00 2024-03-24T01:26:00.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.217+0000 7f1521200640 1 -- 172.21.15.90:0/2325714860 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f151c0b1410 con 0x7f1500003a00 2024-03-24T01:26:00.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.218+0000 7f1513400640 1 -- 172.21.15.90:0/2325714860 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f150c048020 con 0x7f1500003a00 2024-03-24T01:26:00.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.219+0000 7f1513400640 1 --2- 172.21.15.90:0/2325714860 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f14fc06bbd0 0x7f14fc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:00.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.219+0000 7f1513400640 1 -- 172.21.15.90:0/2325714860 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f150c0abff0 con 0x7f1500003a00 2024-03-24T01:26:00.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.219+0000 7f151aa00640 1 --2- 172.21.15.90:0/2325714860 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f14fc06bbd0 0x7f14fc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:00.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.220+0000 7f151aa00640 1 --2- 172.21.15.90:0/2325714860 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f14fc06bbd0 0x7f14fc06e090 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f15080093d0 tx=0x7f1508009300 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:00.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.223+0000 7f1513400640 1 -- 172.21.15.90:0/2325714860 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f151c0b1410 con 0x7f1500003a00 2024-03-24T01:26:00.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.413+0000 7f1521200640 1 -- 172.21.15.90:0/2325714860 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f151c04f400 con 0x7f1500003a00 2024-03-24T01:26:00.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.414+0000 7f1513400640 1 -- 172.21.15.90:0/2325714860 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+16931 (secure 0 0 0) 0x7f150c07aa00 con 0x7f1500003a00 2024-03-24T01:26:00.415 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:26:00.415 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":38,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:25:59.439507+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:25:57.398864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-24T01:25:58.451150+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:26:00.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.417+0000 7f1521200640 1 -- 172.21.15.90:0/2325714860 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f14fc06bbd0 msgr2=0x7f14fc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:00.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.417+0000 7f1521200640 1 --2- 172.21.15.90:0/2325714860 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f14fc06bbd0 0x7f14fc06e090 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f15080093d0 tx=0x7f1508009300 comp rx=0 tx=0).stop 2024-03-24T01:26:00.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.417+0000 7f1521200640 1 -- 172.21.15.90:0/2325714860 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1500003a00 msgr2=0x7f151c1591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:00.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.417+0000 7f1521200640 1 --2- 172.21.15.90:0/2325714860 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1500003a00 0x7f151c1591d0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f150c00c200 tx=0x7f150c02e070 comp rx=0 tx=0).stop 2024-03-24T01:26:00.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.417+0000 7f1521200640 1 -- 172.21.15.90:0/2325714860 shutdown_connections 2024-03-24T01:26:00.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.417+0000 7f1521200640 1 --2- 172.21.15.90:0/2325714860 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f14fc06bbd0 0x7f14fc06e090 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:00.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.417+0000 7f1521200640 1 --2- 172.21.15.90:0/2325714860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f151c13d250 0x7f151c13d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:00.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.417+0000 7f1521200640 1 --2- 172.21.15.90:0/2325714860 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f151c0b0470 0x7f151c0b08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:00.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.417+0000 7f1521200640 1 --2- 172.21.15.90:0/2325714860 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1500003a00 0x7f151c1591d0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:00.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.417+0000 7f1521200640 1 -- 172.21.15.90:0/2325714860 >> 172.21.15.90:0/2325714860 conn(0x7f151c06b0c0 msgr2=0x7f151c144430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:00.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.417+0000 7f1521200640 1 -- 172.21.15.90:0/2325714860 shutdown_connections 2024-03-24T01:26:00.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.417+0000 7f1521200640 1 -- 172.21.15.90:0/2325714860 wait complete. 2024-03-24T01:26:00.439 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:26:00.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.635+0000 7f67a9800640 1 Processor -- start 2024-03-24T01:26:00.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.635+0000 7f67a9800640 1 -- start start 2024-03-24T01:26:00.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.637+0000 7f67a9800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67a40cc150 0x7f67a40ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:00.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.637+0000 7f67a9800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f67a41603e0 0x7f67a406b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:00.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.637+0000 7f67a9800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67a40cd490 con 0x7f67a4145430 2024-03-24T01:26:00.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.637+0000 7f67a9800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67a415f900 con 0x7f67a41603e0 2024-03-24T01:26:00.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.637+0000 7f67a9800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67a415fbe0 con 0x7f67a40cc150 2024-03-24T01:26:00.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.637+0000 7f679a000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f67a41603e0 0x7f67a406b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:00.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.637+0000 7f67a2a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67a40cc150 0x7f67a40ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:00.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.637+0000 7f679a000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f67a41603e0 0x7f67a406b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.90:51496/0 (socket says 172.21.15.90:51496) 2024-03-24T01:26:00.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.637+0000 7f679a000640 1 -- 172.21.15.90:0/278704482 learned_addr learned my addr 172.21.15.90:0/278704482 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:00.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.638+0000 7f67a2a00640 1 -- 172.21.15.90:0/278704482 >> v2:172.21.15.120:3300/0 conn(0x7f67a41603e0 msgr2=0x7f67a406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:00.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.638+0000 7f67a2a00640 1 --2- 172.21.15.90:0/278704482 >> v2:172.21.15.120:3300/0 conn(0x7f67a41603e0 0x7f67a406b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:00.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.638+0000 7f67a2a00640 1 -- 172.21.15.90:0/278704482 >> v1:172.21.15.120:6789/0 conn(0x7f67a4145430 legacy=0x7f67a4071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:00.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.638+0000 7f67a2a00640 1 -- 172.21.15.90:0/278704482 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a415fec0 con 0x7f67a40cc150 2024-03-24T01:26:00.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.640+0000 7f67a2a00640 1 --2- 172.21.15.90:0/278704482 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67a40cc150 0x7f67a40ccb60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f67940097a0 tx=0x7f679402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f56294a8609606b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:00.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.641+0000 7f67a2000640 1 -- 172.21.15.90:0/278704482 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6794035020 con 0x7f67a40cc150 2024-03-24T01:26:00.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.641+0000 7f67a2000640 1 -- 172.21.15.90:0/278704482 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6794031830 con 0x7f67a40cc150 2024-03-24T01:26:00.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.641+0000 7f67a2000640 1 -- 172.21.15.90:0/278704482 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6794031c00 con 0x7f67a40cc150 2024-03-24T01:26:00.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.641+0000 7f67a9800640 1 -- 172.21.15.90:0/278704482 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67a40cc150 msgr2=0x7f67a40ccb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:00.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.641+0000 7f67a9800640 1 --2- 172.21.15.90:0/278704482 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67a40cc150 0x7f67a40ccb60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f67940097a0 tx=0x7f679402b8d0 comp rx=0 tx=0).stop 2024-03-24T01:26:00.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.642+0000 7f67a9800640 1 -- 172.21.15.90:0/278704482 shutdown_connections 2024-03-24T01:26:00.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.642+0000 7f67a9800640 1 --2- 172.21.15.90:0/278704482 >> v2:172.21.15.120:3300/0 conn(0x7f67a41603e0 0x7f67a406b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:00.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.642+0000 7f67a9800640 1 --2- 172.21.15.90:0/278704482 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67a40cc150 0x7f67a40ccb60 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:00.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.642+0000 7f67a9800640 1 -- 172.21.15.90:0/278704482 >> 172.21.15.90:0/278704482 conn(0x7f67a40b1770 msgr2=0x7f67a40b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:00.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.642+0000 7f67a9800640 1 -- 172.21.15.90:0/278704482 shutdown_connections 2024-03-24T01:26:00.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.642+0000 7f67a9800640 1 -- 172.21.15.90:0/278704482 wait complete. 2024-03-24T01:26:00.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.643+0000 7f67a9800640 1 Processor -- start 2024-03-24T01:26:00.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.643+0000 7f67a9800640 1 -- start start 2024-03-24T01:26:00.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.644+0000 7f67a9800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67a40cc150 0x7f67a4159e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:00.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.644+0000 7f67a9800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67a4145430 0x7f67a406e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:00.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.644+0000 7f67a9800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67a41603e0 0x7f67a406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:00.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.644+0000 7f67a9800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a406b780 con 0x7f67a41603e0 2024-03-24T01:26:00.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.644+0000 7f67a9800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a406b600 con 0x7f67a4145430 2024-03-24T01:26:00.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.644+0000 7f67a9800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67a406b900 con 0x7f67a40cc150 2024-03-24T01:26:00.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.645+0000 7f67a3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67a41603e0 0x7f67a406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:00.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.645+0000 7f67a3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67a41603e0 0x7f67a406ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59942/0 (socket says 172.21.15.90:59942) 2024-03-24T01:26:00.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.645+0000 7f67a3400640 1 -- 172.21.15.90:0/3158493874 learned_addr learned my addr 172.21.15.90:0/3158493874 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:00.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.645+0000 7f679a000640 1 --2- 172.21.15.90:0/3158493874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67a4145430 0x7f67a406e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:00.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.645+0000 7f67a2a00640 1 --2- 172.21.15.90:0/3158493874 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67a40cc150 0x7f67a4159e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:00.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.645+0000 7f67a3400640 1 -- 172.21.15.90:0/3158493874 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67a40cc150 msgr2=0x7f67a4159e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:00.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.645+0000 7f67a3400640 1 --2- 172.21.15.90:0/3158493874 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67a40cc150 0x7f67a4159e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:00.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.645+0000 7f67a3400640 1 -- 172.21.15.90:0/3158493874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67a4145430 msgr2=0x7f67a406e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:00.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.645+0000 7f67a3400640 1 --2- 172.21.15.90:0/3158493874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67a4145430 0x7f67a406e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:00.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.645+0000 7f67a3400640 1 -- 172.21.15.90:0/3158493874 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a415d6c0 con 0x7f67a41603e0 2024-03-24T01:26:00.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.645+0000 7f67a2a00640 1 --2- 172.21.15.90:0/3158493874 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67a40cc150 0x7f67a4159e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:26:00.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.646+0000 7f67a3400640 1 --2- 172.21.15.90:0/3158493874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67a41603e0 0x7f67a406ef20 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f679c00c7a0 tx=0x7f679c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:00.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.646+0000 7f679b400640 1 -- 172.21.15.90:0/3158493874 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679c003b70 con 0x7f67a41603e0 2024-03-24T01:26:00.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.646+0000 7f679b400640 1 -- 172.21.15.90:0/3158493874 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f679c003d30 con 0x7f67a41603e0 2024-03-24T01:26:00.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.646+0000 7f67a9800640 1 -- 172.21.15.90:0/3158493874 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67a415d9a0 con 0x7f67a41603e0 2024-03-24T01:26:00.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.646+0000 7f679b400640 1 -- 172.21.15.90:0/3158493874 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679c00aa90 con 0x7f67a41603e0 2024-03-24T01:26:00.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.646+0000 7f67a9800640 1 -- 172.21.15.90:0/3158493874 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67a41386b0 con 0x7f67a41603e0 2024-03-24T01:26:00.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.647+0000 7f67a9800640 1 -- 172.21.15.90:0/3158493874 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67a4068060 con 0x7f67a41603e0 2024-03-24T01:26:00.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.648+0000 7f679b400640 1 -- 172.21.15.90:0/3158493874 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f679c00ac30 con 0x7f67a41603e0 2024-03-24T01:26:00.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.649+0000 7f679b400640 1 --2- 172.21.15.90:0/3158493874 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f678006bbd0 0x7f678006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:00.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.649+0000 7f679b400640 1 -- 172.21.15.90:0/3158493874 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f679c08ab80 con 0x7f67a41603e0 2024-03-24T01:26:00.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.650+0000 7f67a2a00640 1 --2- 172.21.15.90:0/3158493874 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f678006bbd0 0x7f678006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:00.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.651+0000 7f67a2a00640 1 --2- 172.21.15.90:0/3158493874 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f678006bbd0 0x7f678006e090 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6794036000 tx=0x7f679403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:00.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.654+0000 7f679b400640 1 -- 172.21.15.90:0/3158493874 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f679c0593c0 con 0x7f67a41603e0 2024-03-24T01:26:00.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.852+0000 7f67a9800640 1 -- 172.21.15.90:0/3158493874 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67a4138880 con 0x7f67a41603e0 2024-03-24T01:26:00.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.855+0000 7f679b400640 1 -- 172.21.15.90:0/3158493874 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+16931 (secure 0 0 0) 0x7f679c058cc0 con 0x7f67a41603e0 2024-03-24T01:26:00.855 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:26:00.856 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":38,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:25:59.439507+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:25:57.398864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-24T01:25:58.451150+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:26:00.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.858+0000 7f67a9800640 1 -- 172.21.15.90:0/3158493874 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f678006bbd0 msgr2=0x7f678006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:00.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.858+0000 7f67a9800640 1 --2- 172.21.15.90:0/3158493874 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f678006bbd0 0x7f678006e090 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6794036000 tx=0x7f679403c040 comp rx=0 tx=0).stop 2024-03-24T01:26:00.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.858+0000 7f67a9800640 1 -- 172.21.15.90:0/3158493874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67a41603e0 msgr2=0x7f67a406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:00.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.858+0000 7f67a9800640 1 --2- 172.21.15.90:0/3158493874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67a41603e0 0x7f67a406ef20 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f679c00c7a0 tx=0x7f679c00cc60 comp rx=0 tx=0).stop 2024-03-24T01:26:00.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.858+0000 7f67a9800640 1 -- 172.21.15.90:0/3158493874 shutdown_connections 2024-03-24T01:26:00.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.858+0000 7f67a9800640 1 --2- 172.21.15.90:0/3158493874 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f678006bbd0 0x7f678006e090 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:00.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.858+0000 7f67a9800640 1 --2- 172.21.15.90:0/3158493874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67a41603e0 0x7f67a406ef20 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:00.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.858+0000 7f67a9800640 1 --2- 172.21.15.90:0/3158493874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67a4145430 0x7f67a406e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:00.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.858+0000 7f67a9800640 1 --2- 172.21.15.90:0/3158493874 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67a40cc150 0x7f67a4159e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:00.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.858+0000 7f67a9800640 1 -- 172.21.15.90:0/3158493874 >> 172.21.15.90:0/3158493874 conn(0x7f67a40b1770 msgr2=0x7f67a40af570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:00.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.858+0000 7f67a9800640 1 -- 172.21.15.90:0/3158493874 shutdown_connections 2024-03-24T01:26:00.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:00.858+0000 7f67a9800640 1 -- 172.21.15.90:0/3158493874 wait complete. 2024-03-24T01:26:00.877 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-24T01:26:01.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.095+0000 7f5603800640 1 Processor -- start 2024-03-24T01:26:01.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.095+0000 7f5603800640 1 -- start start 2024-03-24T01:26:01.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.096+0000 7f5603800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55fc0d0520 0x7f55fc0d09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:01.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.096+0000 7f5603800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f55fc0cede0 0x7f55fc0cf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:01.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.096+0000 7f5603800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55fc0d0ee0 con 0x7f55fc15ab60 2024-03-24T01:26:01.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.097+0000 7f5603800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55fc0cc770 con 0x7f55fc0cede0 2024-03-24T01:26:01.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.097+0000 7f5603800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55fc0ce950 con 0x7f55fc0d0520 2024-03-24T01:26:01.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.097+0000 7f55fbe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55fc0d0520 0x7f55fc0d09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:01.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.097+0000 7f5601800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f55fc0cede0 0x7f55fc0cf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:01.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.097+0000 7f55fbe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55fc0d0520 0x7f55fc0d09a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.90:55788/0 (socket says 172.21.15.90:55788) 2024-03-24T01:26:01.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.097+0000 7f55fbe00640 1 -- 172.21.15.90:0/3497561761 learned_addr learned my addr 172.21.15.90:0/3497561761 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:01.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.098+0000 7f55fbe00640 1 -- 172.21.15.90:0/3497561761 >> v2:172.21.15.120:3300/0 conn(0x7f55fc0cede0 msgr2=0x7f55fc0cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:01.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.098+0000 7f55fbe00640 1 --2- 172.21.15.90:0/3497561761 >> v2:172.21.15.120:3300/0 conn(0x7f55fc0cede0 0x7f55fc0cf270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:01.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.098+0000 7f55fbe00640 1 -- 172.21.15.90:0/3497561761 >> v1:172.21.15.120:6789/0 conn(0x7f55fc15ab60 legacy=0x7f55fc06ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:01.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.098+0000 7f55fbe00640 1 -- 172.21.15.90:0/3497561761 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55fc0cc1b0 con 0x7f55fc0d0520 2024-03-24T01:26:01.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.099+0000 7f55fbe00640 1 --2- 172.21.15.90:0/3497561761 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55fc0d0520 0x7f55fc0d09a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f55e8009870 tx=0x7f55e802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f07c0956fca600f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:01.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.100+0000 7f55fb400640 1 -- 172.21.15.90:0/3497561761 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e8035020 con 0x7f55fc0d0520 2024-03-24T01:26:01.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.100+0000 7f55fb400640 1 -- 172.21.15.90:0/3497561761 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55e8031640 con 0x7f55fc0d0520 2024-03-24T01:26:01.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.100+0000 7f55fb400640 1 -- 172.21.15.90:0/3497561761 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e802d040 con 0x7f55fc0d0520 2024-03-24T01:26:01.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.100+0000 7f5603800640 1 -- 172.21.15.90:0/3497561761 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55fc0d0520 msgr2=0x7f55fc0d09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:01.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.100+0000 7f5603800640 1 --2- 172.21.15.90:0/3497561761 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55fc0d0520 0x7f55fc0d09a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f55e8009870 tx=0x7f55e802b8f0 comp rx=0 tx=0).stop 2024-03-24T01:26:01.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.100+0000 7f5603800640 1 -- 172.21.15.90:0/3497561761 shutdown_connections 2024-03-24T01:26:01.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.100+0000 7f5603800640 1 --2- 172.21.15.90:0/3497561761 >> v2:172.21.15.120:3300/0 conn(0x7f55fc0cede0 0x7f55fc0cf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:01.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.100+0000 7f5603800640 1 --2- 172.21.15.90:0/3497561761 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55fc0d0520 0x7f55fc0d09a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:01.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.100+0000 7f5603800640 1 -- 172.21.15.90:0/3497561761 >> 172.21.15.90:0/3497561761 conn(0x7f55fc099a70 msgr2=0x7f55fc099e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:01.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.100+0000 7f5603800640 1 -- 172.21.15.90:0/3497561761 shutdown_connections 2024-03-24T01:26:01.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.100+0000 7f5603800640 1 -- 172.21.15.90:0/3497561761 wait complete. 2024-03-24T01:26:01.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.102+0000 7f5603800640 1 Processor -- start 2024-03-24T01:26:01.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.102+0000 7f5603800640 1 -- start start 2024-03-24T01:26:01.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.103+0000 7f5603800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f55fc0cede0 0x7f55fc149950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:01.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.103+0000 7f5603800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55fc0d0520 0x7f55fc0c2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:01.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.103+0000 7f5603800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55fc15ab60 0x7f55fc0c32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:01.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.103+0000 7f5603800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55fc06f690 con 0x7f55fc0cede0 2024-03-24T01:26:01.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.103+0000 7f5603800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55fc0ce5e0 con 0x7f55fc0d0520 2024-03-24T01:26:01.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.103+0000 7f5603800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55fc15af40 con 0x7f55fc15ab60 2024-03-24T01:26:01.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.104+0000 7f5600e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f55fc0cede0 0x7f55fc149950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:01.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.104+0000 7f5600e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f55fc0cede0 0x7f55fc149950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59950/0 (socket says 172.21.15.90:59950) 2024-03-24T01:26:01.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.104+0000 7f5600e00640 1 -- 172.21.15.90:0/1683199545 learned_addr learned my addr 172.21.15.90:0/1683199545 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:01.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.104+0000 7f55fbe00640 1 --2- 172.21.15.90:0/1683199545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55fc0d0520 0x7f55fc0c2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:01.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.104+0000 7f5601800640 1 --2- 172.21.15.90:0/1683199545 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55fc15ab60 0x7f55fc0c32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:01.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.104+0000 7f5600e00640 1 -- 172.21.15.90:0/1683199545 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55fc15ab60 msgr2=0x7f55fc0c32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:01.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.104+0000 7f5600e00640 1 --2- 172.21.15.90:0/1683199545 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55fc15ab60 0x7f55fc0c32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:01.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.104+0000 7f5600e00640 1 -- 172.21.15.90:0/1683199545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55fc0d0520 msgr2=0x7f55fc0c2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:01.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.104+0000 7f5600e00640 1 --2- 172.21.15.90:0/1683199545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55fc0d0520 0x7f55fc0c2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:01.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.104+0000 7f5600e00640 1 -- 172.21.15.90:0/1683199545 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55fc0b22f0 con 0x7f55fc0cede0 2024-03-24T01:26:01.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.105+0000 7f5600e00640 1 --2- 172.21.15.90:0/1683199545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f55fc0cede0 0x7f55fc149950 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f55f000c3f0 tx=0x7f55f000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:01.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.105+0000 7f55f8c00640 1 -- 172.21.15.90:0/1683199545 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f00052d0 con 0x7f55fc0cede0 2024-03-24T01:26:01.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.105+0000 7f55f8c00640 1 -- 172.21.15.90:0/1683199545 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55f0005490 con 0x7f55fc0cede0 2024-03-24T01:26:01.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.105+0000 7f55f8c00640 1 -- 172.21.15.90:0/1683199545 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f0007260 con 0x7f55fc0cede0 2024-03-24T01:26:01.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.105+0000 7f5603800640 1 -- 172.21.15.90:0/1683199545 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55fc0b2590 con 0x7f55fc0cede0 2024-03-24T01:26:01.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.105+0000 7f5603800640 1 -- 172.21.15.90:0/1683199545 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55fc0b2830 con 0x7f55fc0cede0 2024-03-24T01:26:01.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.107+0000 7f5603800640 1 -- 172.21.15.90:0/1683199545 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55fc04f400 con 0x7f55fc0cede0 2024-03-24T01:26:01.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.107+0000 7f55f8c00640 1 -- 172.21.15.90:0/1683199545 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f55f0007400 con 0x7f55fc0cede0 2024-03-24T01:26:01.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.108+0000 7f55f8c00640 1 --2- 172.21.15.90:0/1683199545 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f55dc06bbd0 0x7f55dc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:01.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.108+0000 7f55f8c00640 1 -- 172.21.15.90:0/1683199545 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f55f008a0e0 con 0x7f55fc0cede0 2024-03-24T01:26:01.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.109+0000 7f55fbe00640 1 --2- 172.21.15.90:0/1683199545 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f55dc06bbd0 0x7f55dc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:01.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.110+0000 7f55fbe00640 1 --2- 172.21.15.90:0/1683199545 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f55dc06bbd0 0x7f55dc06e090 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f55e8036000 tx=0x7f55e803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:01.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.118+0000 7f55f8c00640 1 -- 172.21.15.90:0/1683199545 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f55f0058a40 con 0x7f55fc0cede0 2024-03-24T01:26:01.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:01.357+0000 7f5603800640 1 -- 172.21.15.90:0/1683199545 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f55fc06ed60 con 0x7f55fc0cede0 2024-03-24T01:26:02.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.005+0000 7f55f8c00640 1 -- 172.21.15.90:0/1683199545 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) v1 ==== 118+0+0 (secure 0 0 0) 0x7f55f0058340 con 0x7f55fc0cede0 2024-03-24T01:26:02.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.008+0000 7f5603800640 1 -- 172.21.15.90:0/1683199545 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f55dc06bbd0 msgr2=0x7f55dc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:02.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.008+0000 7f5603800640 1 --2- 172.21.15.90:0/1683199545 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f55dc06bbd0 0x7f55dc06e090 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f55e8036000 tx=0x7f55e803c040 comp rx=0 tx=0).stop 2024-03-24T01:26:02.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.008+0000 7f5603800640 1 -- 172.21.15.90:0/1683199545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f55fc0cede0 msgr2=0x7f55fc149950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:02.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.008+0000 7f5603800640 1 --2- 172.21.15.90:0/1683199545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f55fc0cede0 0x7f55fc149950 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f55f000c3f0 tx=0x7f55f000c8b0 comp rx=0 tx=0).stop 2024-03-24T01:26:02.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.008+0000 7f5603800640 1 -- 172.21.15.90:0/1683199545 shutdown_connections 2024-03-24T01:26:02.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.008+0000 7f5603800640 1 --2- 172.21.15.90:0/1683199545 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f55dc06bbd0 0x7f55dc06e090 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:02.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.008+0000 7f5603800640 1 --2- 172.21.15.90:0/1683199545 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f55fc15ab60 0x7f55fc0c32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:02.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.008+0000 7f5603800640 1 --2- 172.21.15.90:0/1683199545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55fc0d0520 0x7f55fc0c2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:02.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.008+0000 7f5603800640 1 --2- 172.21.15.90:0/1683199545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f55fc0cede0 0x7f55fc149950 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:02.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.008+0000 7f5603800640 1 -- 172.21.15.90:0/1683199545 >> 172.21.15.90:0/1683199545 conn(0x7f55fc099a70 msgr2=0x7f55fc0d10b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:02.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.009+0000 7f5603800640 1 -- 172.21.15.90:0/1683199545 shutdown_connections 2024-03-24T01:26:02.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.009+0000 7f5603800640 1 -- 172.21.15.90:0/1683199545 wait complete. 2024-03-24T01:26:02.031 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-24T01:26:02.031 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-24T01:26:02.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.261+0000 7f4405000640 1 Processor -- start 2024-03-24T01:26:02.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.261+0000 7f4405000640 1 -- start start 2024-03-24T01:26:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.263+0000 7f4405000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f440009a2d0 0x7f440009ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.263+0000 7f4405000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f44000cd7c0 0x7f44000b15c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.263+0000 7f4405000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44000cf760 con 0x7f440015f4b0 2024-03-24T01:26:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.263+0000 7f4405000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44000cfce0 con 0x7f44000cd7c0 2024-03-24T01:26:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.263+0000 7f4405000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44000b3130 con 0x7f440009a2d0 2024-03-24T01:26:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.263+0000 7f43fe800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f44000cd7c0 0x7f44000b15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.263+0000 7f43fe800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f44000cd7c0 0x7f44000b15c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59964/0 (socket says 172.21.15.90:59964) 2024-03-24T01:26:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.263+0000 7f43fe800640 1 -- 172.21.15.90:0/3817553878 learned_addr learned my addr 172.21.15.90:0/3817553878 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.263+0000 7f43fd400640 1 --2- 172.21.15.90:0/3817553878 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f440009a2d0 0x7f440009ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.264+0000 7f43fe800640 1 -- 172.21.15.90:0/3817553878 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f440009a2d0 msgr2=0x7f440009ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.264+0000 7f43fe800640 1 --2- 172.21.15.90:0/3817553878 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f440009a2d0 0x7f440009ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.264+0000 7f43fe800640 1 -- 172.21.15.90:0/3817553878 >> v1:172.21.15.120:6789/0 conn(0x7f440015f4b0 legacy=0x7f4400099bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.264+0000 7f43fe800640 1 -- 172.21.15.90:0/3817553878 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44000cfa00 con 0x7f44000cd7c0 2024-03-24T01:26:02.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.264+0000 7f43fd400640 1 --2- 172.21.15.90:0/3817553878 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f440009a2d0 0x7f440009ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:02.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.265+0000 7f43fe800640 1 --2- 172.21.15.90:0/3817553878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44000cd7c0 0x7f44000b15c0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f43f8009e50 tx=0x7f43f802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3243b7e32ead0779 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:02.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.265+0000 7f43fca00640 1 -- 172.21.15.90:0/3817553878 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f802c9f0 con 0x7f44000cd7c0 2024-03-24T01:26:02.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.265+0000 7f43fca00640 1 -- 172.21.15.90:0/3817553878 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f802cb90 con 0x7f44000cd7c0 2024-03-24T01:26:02.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.265+0000 7f43fca00640 1 -- 172.21.15.90:0/3817553878 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f80343f0 con 0x7f44000cd7c0 2024-03-24T01:26:02.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.266+0000 7f4405000640 1 -- 172.21.15.90:0/3817553878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44000cd7c0 msgr2=0x7f44000b15c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:02.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.266+0000 7f4405000640 1 --2- 172.21.15.90:0/3817553878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44000cd7c0 0x7f44000b15c0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f43f8009e50 tx=0x7f43f802be50 comp rx=0 tx=0).stop 2024-03-24T01:26:02.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.266+0000 7f4405000640 1 -- 172.21.15.90:0/3817553878 shutdown_connections 2024-03-24T01:26:02.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.266+0000 7f4405000640 1 --2- 172.21.15.90:0/3817553878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44000cd7c0 0x7f44000b15c0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:02.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.266+0000 7f4405000640 1 --2- 172.21.15.90:0/3817553878 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f440009a2d0 0x7f440009ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:02.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.266+0000 7f4405000640 1 -- 172.21.15.90:0/3817553878 >> 172.21.15.90:0/3817553878 conn(0x7f44000b3470 msgr2=0x7f44000b3880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:02.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.266+0000 7f4405000640 1 -- 172.21.15.90:0/3817553878 shutdown_connections 2024-03-24T01:26:02.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.266+0000 7f4405000640 1 -- 172.21.15.90:0/3817553878 wait complete. 2024-03-24T01:26:02.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.268+0000 7f4405000640 1 Processor -- start 2024-03-24T01:26:02.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.268+0000 7f4405000640 1 -- start start 2024-03-24T01:26:02.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.269+0000 7f4405000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f440009a2d0 0x7f44000c8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:02.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.269+0000 7f4405000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44000cd7c0 0x7f44000cb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:02.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.269+0000 7f4405000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f440015f4b0 0x7f44000cc870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:02.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.269+0000 7f4405000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44000b1b00 con 0x7f440015f4b0 2024-03-24T01:26:02.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.269+0000 7f4405000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44000b2dc0 con 0x7f44000cd7c0 2024-03-24T01:26:02.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.269+0000 7f4405000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44000b1c80 con 0x7f440009a2d0 2024-03-24T01:26:02.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.269+0000 7f43fe800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f440015f4b0 0x7f44000cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:02.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.269+0000 7f43fe800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f440015f4b0 0x7f44000cc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59976/0 (socket says 172.21.15.90:59976) 2024-03-24T01:26:02.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.269+0000 7f43fe800640 1 -- 172.21.15.90:0/3637518012 learned_addr learned my addr 172.21.15.90:0/3637518012 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:02.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.269+0000 7f43fe800640 1 -- 172.21.15.90:0/3637518012 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f440009a2d0 msgr2=0x7f44000c8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:02.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.269+0000 7f43fe800640 1 --2- 172.21.15.90:0/3637518012 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f440009a2d0 0x7f44000c8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:02.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.269+0000 7f43fe800640 1 -- 172.21.15.90:0/3637518012 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44000cd7c0 msgr2=0x7f44000cb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:02.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.269+0000 7f43fe800640 1 --2- 172.21.15.90:0/3637518012 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44000cd7c0 0x7f44000cb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:02.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.269+0000 7f43fe800640 1 -- 172.21.15.90:0/3637518012 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44000cb8b0 con 0x7f440015f4b0 2024-03-24T01:26:02.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.270+0000 7f43fe800640 1 --2- 172.21.15.90:0/3637518012 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f440015f4b0 0x7f44000cc870 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f43f8036040 tx=0x7f43f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:02.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.270+0000 7f43f6000640 1 -- 172.21.15.90:0/3637518012 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f8043020 con 0x7f440015f4b0 2024-03-24T01:26:02.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.270+0000 7f43f6000640 1 -- 172.21.15.90:0/3637518012 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f803e450 con 0x7f440015f4b0 2024-03-24T01:26:02.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.270+0000 7f4405000640 1 -- 172.21.15.90:0/3637518012 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44000cb610 con 0x7f440015f4b0 2024-03-24T01:26:02.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.270+0000 7f43f6000640 1 -- 172.21.15.90:0/3637518012 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f803e860 con 0x7f440015f4b0 2024-03-24T01:26:02.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.270+0000 7f4405000640 1 -- 172.21.15.90:0/3637518012 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4400159f90 con 0x7f440015f4b0 2024-03-24T01:26:02.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.271+0000 7f4405000640 1 -- 172.21.15.90:0/3637518012 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f440004f400 con 0x7f440015f4b0 2024-03-24T01:26:02.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.272+0000 7f43f6000640 1 -- 172.21.15.90:0/3637518012 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f43f8046020 con 0x7f440015f4b0 2024-03-24T01:26:02.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.273+0000 7f43f6000640 1 --2- 172.21.15.90:0/3637518012 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f43e006bbd0 0x7f43e006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:02.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.273+0000 7f43f6000640 1 -- 172.21.15.90:0/3637518012 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f43f80ad680 con 0x7f440015f4b0 2024-03-24T01:26:02.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.274+0000 7f43fde00640 1 --2- 172.21.15.90:0/3637518012 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f43e006bbd0 0x7f43e006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:02.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.275+0000 7f43fde00640 1 --2- 172.21.15.90:0/3637518012 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f43e006bbd0 0x7f43e006e090 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f43f0006fd0 tx=0x7f43f0006bc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:02.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.282+0000 7f43f6000640 1 -- 172.21.15.90:0/3637518012 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f43f804d030 con 0x7f440015f4b0 2024-03-24T01:26:02.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:02.564+0000 7f4405000640 1 -- 172.21.15.90:0/3637518012 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f44000cd550 con 0x7f440015f4b0 2024-03-24T01:26:02.926 INFO:tasks.ceph.mds.c.smithi090.stderr:2024-03-24T01:26:02.926+0000 7f983d000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T01:26:03.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.008+0000 7f43f6000640 1 -- 172.21.15.90:0/3637518012 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v17) v1 ==== 122+0+0 (secure 0 0 0) 0x7f43f807beb0 con 0x7f440015f4b0 2024-03-24T01:26:03.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.012+0000 7f4405000640 1 -- 172.21.15.90:0/3637518012 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f43e006bbd0 msgr2=0x7f43e006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:03.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.012+0000 7f4405000640 1 --2- 172.21.15.90:0/3637518012 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f43e006bbd0 0x7f43e006e090 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f43f0006fd0 tx=0x7f43f0006bc0 comp rx=0 tx=0).stop 2024-03-24T01:26:03.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.012+0000 7f4405000640 1 -- 172.21.15.90:0/3637518012 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f440015f4b0 msgr2=0x7f44000cc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:03.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.012+0000 7f4405000640 1 --2- 172.21.15.90:0/3637518012 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f440015f4b0 0x7f44000cc870 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f43f8036040 tx=0x7f43f8004a40 comp rx=0 tx=0).stop 2024-03-24T01:26:03.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.012+0000 7f4405000640 1 -- 172.21.15.90:0/3637518012 shutdown_connections 2024-03-24T01:26:03.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.012+0000 7f4405000640 1 --2- 172.21.15.90:0/3637518012 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f43e006bbd0 0x7f43e006e090 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:03.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.012+0000 7f4405000640 1 --2- 172.21.15.90:0/3637518012 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f440015f4b0 0x7f44000cc870 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:03.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.012+0000 7f4405000640 1 --2- 172.21.15.90:0/3637518012 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44000cd7c0 0x7f44000cb0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:03.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.012+0000 7f4405000640 1 --2- 172.21.15.90:0/3637518012 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f440009a2d0 0x7f44000c8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:03.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.012+0000 7f4405000640 1 -- 172.21.15.90:0/3637518012 >> 172.21.15.90:0/3637518012 conn(0x7f44000b3470 msgr2=0x7f4400071270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:03.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.012+0000 7f4405000640 1 -- 172.21.15.90:0/3637518012 shutdown_connections 2024-03-24T01:26:03.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.012+0000 7f4405000640 1 -- 172.21.15.90:0/3637518012 wait complete. 2024-03-24T01:26:03.032 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-24T01:26:03.032 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:26:03.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.234+0000 7f7109e00640 1 Processor -- start 2024-03-24T01:26:03.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.234+0000 7f7109e00640 1 -- start start 2024-03-24T01:26:03.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.235+0000 7f7109e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f710415dba0 0x7f7104162330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:03.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.235+0000 7f7109e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f710415df80 0x7f71040b5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:03.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.236+0000 7f7109e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f71040b2960 0x7f71040b5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:03.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.236+0000 7f7109e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7104162870 con 0x7f71040b2960 2024-03-24T01:26:03.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.236+0000 7f7109e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71040ce4b0 con 0x7f710415df80 2024-03-24T01:26:03.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.236+0000 7f7109e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71040ce790 con 0x7f710415dba0 2024-03-24T01:26:03.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.236+0000 7f7109400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f71040b2960 0x7f71040b5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:03.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.236+0000 7f7109400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f71040b2960 0x7f71040b5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59986/0 (socket says 172.21.15.90:59986) 2024-03-24T01:26:03.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.236+0000 7f7109400640 1 -- 172.21.15.90:0/560709904 learned_addr learned my addr 172.21.15.90:0/560709904 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:03.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.236+0000 7f7108a00640 1 --2- 172.21.15.90:0/560709904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f710415dba0 0x7f7104162330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:03.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.236+0000 7f7103e00640 1 --2- 172.21.15.90:0/560709904 >> v2:172.21.15.120:3300/0 conn(0x7f710415df80 0x7f71040b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:03.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.236+0000 7f7109400640 1 -- 172.21.15.90:0/560709904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f710415dba0 msgr2=0x7f7104162330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:03.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.236+0000 7f7109400640 1 --2- 172.21.15.90:0/560709904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f710415dba0 0x7f7104162330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:03.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.236+0000 7f7109400640 1 -- 172.21.15.90:0/560709904 >> v2:172.21.15.120:3300/0 conn(0x7f710415df80 msgr2=0x7f71040b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:03.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.236+0000 7f7109400640 1 --2- 172.21.15.90:0/560709904 >> v2:172.21.15.120:3300/0 conn(0x7f710415df80 0x7f71040b5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:03.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.236+0000 7f7109400640 1 -- 172.21.15.90:0/560709904 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71040cea70 con 0x7f71040b2960 2024-03-24T01:26:03.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.236+0000 7f7103e00640 1 --2- 172.21.15.90:0/560709904 >> v2:172.21.15.120:3300/0 conn(0x7f710415df80 0x7f71040b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:03.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.238+0000 7f7109400640 1 --2- 172.21.15.90:0/560709904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71040b2960 0x7f71040b5a70 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f71040d4b50 tx=0x7f70fc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76701f419c694f2a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:03.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.238+0000 7f7103400640 1 -- 172.21.15.90:0/560709904 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70fc02c9e0 con 0x7f71040b2960 2024-03-24T01:26:03.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.238+0000 7f7103400640 1 -- 172.21.15.90:0/560709904 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70fc02cb80 con 0x7f71040b2960 2024-03-24T01:26:03.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.238+0000 7f7103400640 1 -- 172.21.15.90:0/560709904 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70fc0343f0 con 0x7f71040b2960 2024-03-24T01:26:03.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.238+0000 7f7109e00640 1 -- 172.21.15.90:0/560709904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71040b2960 msgr2=0x7f71040b5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:03.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.238+0000 7f7109e00640 1 --2- 172.21.15.90:0/560709904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71040b2960 0x7f71040b5a70 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f71040d4b50 tx=0x7f70fc02be40 comp rx=0 tx=0).stop 2024-03-24T01:26:03.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.239+0000 7f7109e00640 1 -- 172.21.15.90:0/560709904 shutdown_connections 2024-03-24T01:26:03.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.239+0000 7f7109e00640 1 --2- 172.21.15.90:0/560709904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71040b2960 0x7f71040b5a70 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:03.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.239+0000 7f7109e00640 1 --2- 172.21.15.90:0/560709904 >> v2:172.21.15.120:3300/0 conn(0x7f710415df80 0x7f71040b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:03.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.239+0000 7f7109e00640 1 --2- 172.21.15.90:0/560709904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f710415dba0 0x7f7104162330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:03.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.239+0000 7f7109e00640 1 -- 172.21.15.90:0/560709904 >> 172.21.15.90:0/560709904 conn(0x7f71040d0530 msgr2=0x7f71040d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:03.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.239+0000 7f7109e00640 1 -- 172.21.15.90:0/560709904 shutdown_connections 2024-03-24T01:26:03.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.239+0000 7f7109e00640 1 -- 172.21.15.90:0/560709904 wait complete. 2024-03-24T01:26:03.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.240+0000 7f7109e00640 1 Processor -- start 2024-03-24T01:26:03.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.240+0000 7f7109e00640 1 -- start start 2024-03-24T01:26:03.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.241+0000 7f7109e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71040b2960 0x7f71040c94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:03.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.241+0000 7f7109e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f710415dba0 0x7f71041582b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:03.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.241+0000 7f7109e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f710415df80 0x7f7104148a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:03.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.241+0000 7f7109e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f710406c6c0 con 0x7f71040b2960 2024-03-24T01:26:03.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.241+0000 7f7109e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f710406c540 con 0x7f710415df80 2024-03-24T01:26:03.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.241+0000 7f7109e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f710406c840 con 0x7f710415dba0 2024-03-24T01:26:03.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.242+0000 7f7108a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71040b2960 0x7f71040c94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:03.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.242+0000 7f7108a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71040b2960 0x7f71040c94b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59988/0 (socket says 172.21.15.90:59988) 2024-03-24T01:26:03.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.242+0000 7f7108a00640 1 -- 172.21.15.90:0/2488193753 learned_addr learned my addr 172.21.15.90:0/2488193753 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:03.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.242+0000 7f7108a00640 1 -- 172.21.15.90:0/2488193753 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f710415dba0 msgr2=0x7f71041582b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:03.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.242+0000 7f7108a00640 1 --2- 172.21.15.90:0/2488193753 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f710415dba0 0x7f71041582b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:03.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.242+0000 7f7108a00640 1 -- 172.21.15.90:0/2488193753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f710415df80 msgr2=0x7f7104148a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:03.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.242+0000 7f7108a00640 1 --2- 172.21.15.90:0/2488193753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f710415df80 0x7f7104148a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:03.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.242+0000 7f7108a00640 1 -- 172.21.15.90:0/2488193753 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71040c80d0 con 0x7f71040b2960 2024-03-24T01:26:03.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.242+0000 7f7108a00640 1 --2- 172.21.15.90:0/2488193753 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71040b2960 0x7f71040c94b0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f70f800d800 tx=0x7f70f800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:03.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.243+0000 7f7100c00640 1 -- 172.21.15.90:0/2488193753 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f8018020 con 0x7f71040b2960 2024-03-24T01:26:03.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.243+0000 7f7100c00640 1 -- 172.21.15.90:0/2488193753 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70f80096a0 con 0x7f71040b2960 2024-03-24T01:26:03.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.243+0000 7f7100c00640 1 -- 172.21.15.90:0/2488193753 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f8009a70 con 0x7f71040b2960 2024-03-24T01:26:03.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.243+0000 7f7109e00640 1 -- 172.21.15.90:0/2488193753 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71040c83b0 con 0x7f71040b2960 2024-03-24T01:26:03.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.243+0000 7f7109e00640 1 -- 172.21.15.90:0/2488193753 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71040c8970 con 0x7f71040b2960 2024-03-24T01:26:03.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.244+0000 7f7109e00640 1 -- 172.21.15.90:0/2488193753 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f710404f400 con 0x7f71040b2960 2024-03-24T01:26:03.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.245+0000 7f7100c00640 1 -- 172.21.15.90:0/2488193753 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f70f801b020 con 0x7f71040b2960 2024-03-24T01:26:03.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.246+0000 7f7100c00640 1 --2- 172.21.15.90:0/2488193753 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f70e406bbd0 0x7f70e406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:03.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.246+0000 7f7100c00640 1 -- 172.21.15.90:0/2488193753 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f70f808aed0 con 0x7f71040b2960 2024-03-24T01:26:03.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.246+0000 7f7103e00640 1 --2- 172.21.15.90:0/2488193753 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f70e406bbd0 0x7f70e406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:03.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.248+0000 7f7103e00640 1 --2- 172.21.15.90:0/2488193753 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f70e406bbd0 0x7f70e406e090 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f70f0006fd0 tx=0x7f70f0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:03.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.250+0000 7f7100c00640 1 -- 172.21.15.90:0/2488193753 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f70f8022030 con 0x7f71040b2960 2024-03-24T01:26:03.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.489+0000 7f7109e00640 1 -- 172.21.15.90:0/2488193753 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71040d17c0 con 0x7f71040b2960 2024-03-24T01:26:03.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.490+0000 7f7100c00640 1 -- 172.21.15.90:0/2488193753 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f70f80597a0 con 0x7f71040b2960 2024-03-24T01:26:03.490 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:26:03.491 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/296084634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":296084634},{"type":"v1","addr":"172.21.15.120:6835","nonce":296084634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/797786252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":797786252},{"type":"v1","addr":"172.21.15.120:6837","nonce":797786252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5490,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6833/1983177399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1983177399},{"type":"v1","addr":"172.21.15.90:6833","nonce":1983177399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T01:25:59.440431+0000","modified":"2024-03-24T01:26:02.566388+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5496":{"gid":5496,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.90:6835/2166652035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":2166652035},{"type":"v1","addr":"172.21.15.90:6835","nonce":2166652035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5496,"qdb_cluster":[5496]},"id":2}]} 2024-03-24T01:26:03.491 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 17 2024-03-24T01:26:03.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.493+0000 7f7109e00640 1 -- 172.21.15.90:0/2488193753 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f70e406bbd0 msgr2=0x7f70e406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:03.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.493+0000 7f7109e00640 1 --2- 172.21.15.90:0/2488193753 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f70e406bbd0 0x7f70e406e090 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f70f0006fd0 tx=0x7f70f0008040 comp rx=0 tx=0).stop 2024-03-24T01:26:03.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.493+0000 7f7109e00640 1 -- 172.21.15.90:0/2488193753 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71040b2960 msgr2=0x7f71040c94b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:03.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.493+0000 7f7109e00640 1 --2- 172.21.15.90:0/2488193753 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71040b2960 0x7f71040c94b0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f70f800d800 tx=0x7f70f800dcc0 comp rx=0 tx=0).stop 2024-03-24T01:26:03.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.493+0000 7f7109e00640 1 -- 172.21.15.90:0/2488193753 shutdown_connections 2024-03-24T01:26:03.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.493+0000 7f7109e00640 1 --2- 172.21.15.90:0/2488193753 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f70e406bbd0 0x7f70e406e090 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:03.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.493+0000 7f7109e00640 1 --2- 172.21.15.90:0/2488193753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f710415df80 0x7f7104148a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:03.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.493+0000 7f7109e00640 1 --2- 172.21.15.90:0/2488193753 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f710415dba0 0x7f71041582b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:03.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.493+0000 7f7109e00640 1 --2- 172.21.15.90:0/2488193753 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71040b2960 0x7f71040c94b0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:03.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.493+0000 7f7109e00640 1 -- 172.21.15.90:0/2488193753 >> 172.21.15.90:0/2488193753 conn(0x7f71040d0530 msgr2=0x7f71040cf500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:03.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.493+0000 7f7109e00640 1 -- 172.21.15.90:0/2488193753 shutdown_connections 2024-03-24T01:26:03.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.493+0000 7f7109e00640 1 -- 172.21.15.90:0/2488193753 wait complete. 2024-03-24T01:26:03.520 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:26:03.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.723+0000 7fd124400640 1 Processor -- start 2024-03-24T01:26:03.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.723+0000 7fd124400640 1 -- start start 2024-03-24T01:26:03.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.724+0000 7fd124400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd11c1443d0 0x7fd11c0cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:03.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.724+0000 7fd124400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd11c0cf3a0 0x7fd11c0cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:03.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.724+0000 7fd124400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd11c1447b0 con 0x7fd11c0b3310 2024-03-24T01:26:03.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.724+0000 7fd124400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd11c15aef0 con 0x7fd11c0cf3a0 2024-03-24T01:26:03.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.724+0000 7fd124400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd11c15b110 con 0x7fd11c1443d0 2024-03-24T01:26:03.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.725+0000 7fd121a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd11c1443d0 0x7fd11c0cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:03.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.725+0000 7fd122400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd11c0cf3a0 0x7fd11c0cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:03.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.725+0000 7fd121a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd11c1443d0 0x7fd11c0cceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.90:55852/0 (socket says 172.21.15.90:55852) 2024-03-24T01:26:03.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.725+0000 7fd121a00640 1 -- 172.21.15.90:0/1814774457 learned_addr learned my addr 172.21.15.90:0/1814774457 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:03.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.725+0000 7fd122400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd11c0cf3a0 0x7fd11c0cd3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.90:51544/0 (socket says 172.21.15.90:51544) 2024-03-24T01:26:03.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.726+0000 7fd121a00640 1 -- 172.21.15.90:0/1814774457 >> v2:172.21.15.120:3300/0 conn(0x7fd11c0cf3a0 msgr2=0x7fd11c0cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:03.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.726+0000 7fd121a00640 1 --2- 172.21.15.90:0/1814774457 >> v2:172.21.15.120:3300/0 conn(0x7fd11c0cf3a0 0x7fd11c0cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:03.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.726+0000 7fd121a00640 1 -- 172.21.15.90:0/1814774457 >> v1:172.21.15.120:6789/0 conn(0x7fd11c0b3310 legacy=0x7fd11c0cec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:03.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.726+0000 7fd121a00640 1 -- 172.21.15.90:0/1814774457 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd11c15b3f0 con 0x7fd11c1443d0 2024-03-24T01:26:03.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.726+0000 7fd122400640 1 --2- 172.21.15.90:0/1814774457 >> v2:172.21.15.120:3300/0 conn(0x7fd11c0cf3a0 0x7fd11c0cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T01:26:03.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.727+0000 7fd121a00640 1 --2- 172.21.15.90:0/1814774457 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd11c1443d0 0x7fd11c0cceb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd110009870 tx=0x7fd11002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3a95263741e76ec server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:03.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.727+0000 7fd11be00640 1 -- 172.21.15.90:0/1814774457 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd110035020 con 0x7fd11c1443d0 2024-03-24T01:26:03.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.727+0000 7fd11be00640 1 -- 172.21.15.90:0/1814774457 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd110031640 con 0x7fd11c1443d0 2024-03-24T01:26:03.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.727+0000 7fd11be00640 1 -- 172.21.15.90:0/1814774457 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd11002d040 con 0x7fd11c1443d0 2024-03-24T01:26:03.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.728+0000 7fd124400640 1 -- 172.21.15.90:0/1814774457 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd11c1443d0 msgr2=0x7fd11c0cceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:03.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.728+0000 7fd124400640 1 --2- 172.21.15.90:0/1814774457 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd11c1443d0 0x7fd11c0cceb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd110009870 tx=0x7fd11002b8f0 comp rx=0 tx=0).stop 2024-03-24T01:26:03.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.728+0000 7fd124400640 1 -- 172.21.15.90:0/1814774457 shutdown_connections 2024-03-24T01:26:03.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.728+0000 7fd124400640 1 --2- 172.21.15.90:0/1814774457 >> v2:172.21.15.120:3300/0 conn(0x7fd11c0cf3a0 0x7fd11c0cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:03.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.728+0000 7fd124400640 1 --2- 172.21.15.90:0/1814774457 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd11c1443d0 0x7fd11c0cceb0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:03.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.728+0000 7fd124400640 1 -- 172.21.15.90:0/1814774457 >> 172.21.15.90:0/1814774457 conn(0x7fd11c06b2a0 msgr2=0x7fd11c06b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:03.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.728+0000 7fd124400640 1 -- 172.21.15.90:0/1814774457 shutdown_connections 2024-03-24T01:26:03.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.728+0000 7fd124400640 1 -- 172.21.15.90:0/1814774457 wait complete. 2024-03-24T01:26:03.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.730+0000 7fd124400640 1 Processor -- start 2024-03-24T01:26:03.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.730+0000 7fd124400640 1 -- start start 2024-03-24T01:26:03.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.731+0000 7fd124400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd11c0b3310 0x7fd11c1546f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:03.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.731+0000 7fd124400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd11c0cf3a0 0x7fd11c13deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:03.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.731+0000 7fd124400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd11c1443d0 0x7fd11c15d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:03.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.731+0000 7fd124400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd11c15bc90 con 0x7fd11c0cf3a0 2024-03-24T01:26:03.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.731+0000 7fd124400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd11c15b9b0 con 0x7fd11c1443d0 2024-03-24T01:26:03.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.731+0000 7fd124400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd11c15e610 con 0x7fd11c0b3310 2024-03-24T01:26:03.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.731+0000 7fd121000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd11c0cf3a0 0x7fd11c13deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:03.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.731+0000 7fd121000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd11c0cf3a0 0x7fd11c13deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59996/0 (socket says 172.21.15.90:59996) 2024-03-24T01:26:03.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.731+0000 7fd121000640 1 -- 172.21.15.90:0/2267926120 learned_addr learned my addr 172.21.15.90:0/2267926120 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:03.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.731+0000 7fd122400640 1 --2- 172.21.15.90:0/2267926120 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd11c1443d0 0x7fd11c15d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:03.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.731+0000 7fd121a00640 1 --2- 172.21.15.90:0/2267926120 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd11c0b3310 0x7fd11c1546f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:03.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.731+0000 7fd121000640 1 -- 172.21.15.90:0/2267926120 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd11c0b3310 msgr2=0x7fd11c1546f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:03.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.731+0000 7fd121000640 1 --2- 172.21.15.90:0/2267926120 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd11c0b3310 0x7fd11c1546f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:03.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.731+0000 7fd121000640 1 -- 172.21.15.90:0/2267926120 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd11c1443d0 msgr2=0x7fd11c15d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:03.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.731+0000 7fd121000640 1 --2- 172.21.15.90:0/2267926120 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd11c1443d0 0x7fd11c15d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:03.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.731+0000 7fd121000640 1 -- 172.21.15.90:0/2267926120 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd11c13e700 con 0x7fd11c0cf3a0 2024-03-24T01:26:03.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.731+0000 7fd122400640 1 --2- 172.21.15.90:0/2267926120 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd11c1443d0 0x7fd11c15d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:03.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.732+0000 7fd121000640 1 --2- 172.21.15.90:0/2267926120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd11c0cf3a0 0x7fd11c13deb0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fd10800c430 tx=0x7fd10800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:03.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.732+0000 7fd119600640 1 -- 172.21.15.90:0/2267926120 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1080052d0 con 0x7fd11c0cf3a0 2024-03-24T01:26:03.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.732+0000 7fd119600640 1 -- 172.21.15.90:0/2267926120 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd108005490 con 0x7fd11c0cf3a0 2024-03-24T01:26:03.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.732+0000 7fd119600640 1 -- 172.21.15.90:0/2267926120 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd108007260 con 0x7fd11c0cf3a0 2024-03-24T01:26:03.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.732+0000 7fd124400640 1 -- 172.21.15.90:0/2267926120 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd11c13e9e0 con 0x7fd11c0cf3a0 2024-03-24T01:26:03.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.732+0000 7fd124400640 1 -- 172.21.15.90:0/2267926120 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd11c13efa0 con 0x7fd11c0cf3a0 2024-03-24T01:26:03.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.733+0000 7fd124400640 1 -- 172.21.15.90:0/2267926120 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd11c06e9c0 con 0x7fd11c0cf3a0 2024-03-24T01:26:03.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.734+0000 7fd119600640 1 -- 172.21.15.90:0/2267926120 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd108007400 con 0x7fd11c0cf3a0 2024-03-24T01:26:03.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.735+0000 7fd119600640 1 --2- 172.21.15.90:0/2267926120 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd0fc06bbd0 0x7fd0fc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:03.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.735+0000 7fd119600640 1 -- 172.21.15.90:0/2267926120 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd10808a320 con 0x7fd11c0cf3a0 2024-03-24T01:26:03.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.736+0000 7fd121a00640 1 --2- 172.21.15.90:0/2267926120 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd0fc06bbd0 0x7fd0fc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:03.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.737+0000 7fd121a00640 1 --2- 172.21.15.90:0/2267926120 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd0fc06bbd0 0x7fd0fc06e090 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd110036000 tx=0x7fd110008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:03.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.739+0000 7fd119600640 1 -- 172.21.15.90:0/2267926120 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd11c06e9c0 con 0x7fd11c0cf3a0 2024-03-24T01:26:03.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.934+0000 7fd124400640 1 -- 172.21.15.90:0/2267926120 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd11c073570 con 0x7fd11c0cf3a0 2024-03-24T01:26:03.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.937+0000 7fd119600640 1 -- 172.21.15.90:0/2267926120 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+16918 (secure 0 0 0) 0x7fd108058ad0 con 0x7fd11c0cf3a0 2024-03-24T01:26:03.937 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:26:03.938 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":39,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:26:00.983350+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:25:57.398864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-24T01:25:58.451150+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:26:03.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.939+0000 7fd124400640 1 -- 172.21.15.90:0/2267926120 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd0fc06bbd0 msgr2=0x7fd0fc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:03.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.939+0000 7fd124400640 1 --2- 172.21.15.90:0/2267926120 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd0fc06bbd0 0x7fd0fc06e090 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd110036000 tx=0x7fd110008030 comp rx=0 tx=0).stop 2024-03-24T01:26:03.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.939+0000 7fd124400640 1 -- 172.21.15.90:0/2267926120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd11c0cf3a0 msgr2=0x7fd11c13deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:03.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.939+0000 7fd124400640 1 --2- 172.21.15.90:0/2267926120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd11c0cf3a0 0x7fd11c13deb0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fd10800c430 tx=0x7fd10800c8f0 comp rx=0 tx=0).stop 2024-03-24T01:26:03.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.939+0000 7fd124400640 1 -- 172.21.15.90:0/2267926120 shutdown_connections 2024-03-24T01:26:03.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.939+0000 7fd124400640 1 --2- 172.21.15.90:0/2267926120 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd0fc06bbd0 0x7fd0fc06e090 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:03.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.939+0000 7fd124400640 1 --2- 172.21.15.90:0/2267926120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd11c0cf3a0 0x7fd11c13deb0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:03.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.939+0000 7fd124400640 1 --2- 172.21.15.90:0/2267926120 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd11c1443d0 0x7fd11c15d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:03.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.939+0000 7fd124400640 1 --2- 172.21.15.90:0/2267926120 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd11c0b3310 0x7fd11c1546f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:03.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.939+0000 7fd124400640 1 -- 172.21.15.90:0/2267926120 >> 172.21.15.90:0/2267926120 conn(0x7fd11c06b2a0 msgr2=0x7fd11c0b4730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:03.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.939+0000 7fd124400640 1 -- 172.21.15.90:0/2267926120 shutdown_connections 2024-03-24T01:26:03.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:03.939+0000 7fd124400640 1 -- 172.21.15.90:0/2267926120 wait complete. 2024-03-24T01:26:03.956 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-24T01:26:03.956 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T01:26:04.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.148+0000 7f0d55000640 1 Processor -- start 2024-03-24T01:26:04.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.148+0000 7f0d55000640 1 -- start start 2024-03-24T01:26:04.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.149+0000 7f0d55000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0d501603e0 0x7f0d5006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:04.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.149+0000 7f0d55000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d500cd490 con 0x7f0d50145430 2024-03-24T01:26:04.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.149+0000 7f0d55000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d5015f900 con 0x7f0d500cc150 2024-03-24T01:26:04.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.149+0000 7f0d55000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d500cd270 con 0x7f0d501603e0 2024-03-24T01:26:04.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.150+0000 7f0d4e800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f0d50145430 0x7f0d500b3750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50070/0 (socket says 172.21.15.90:50070) 2024-03-24T01:26:04.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.150+0000 7f0d4e800640 1 -- 172.21.15.90:0/3174751038 learned_addr learned my addr 172.21.15.90:0/3174751038 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:26:04.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.150+0000 7f0d4d400640 1 --2- 172.21.15.90:0/3174751038 >> v2:172.21.15.120:3300/0 conn(0x7f0d501603e0 0x7f0d5006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:04.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.150+0000 7f0d4ca00640 1 -- 172.21.15.90:0/3174751038 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2507935554 0 0) 0x7f0d500cd490 con 0x7f0d50145430 2024-03-24T01:26:04.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.150+0000 7f0d4ca00640 1 -- 172.21.15.90:0/3174751038 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d34003520 con 0x7f0d50145430 2024-03-24T01:26:04.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.151+0000 7f0d4ca00640 1 -- 172.21.15.90:0/3174751038 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2423072388 0 0) 0x7f0d34003520 con 0x7f0d50145430 2024-03-24T01:26:04.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.151+0000 7f0d4ca00640 1 -- 172.21.15.90:0/3174751038 >> v2:172.21.15.120:3300/0 conn(0x7f0d501603e0 msgr2=0x7f0d5006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:04.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.151+0000 7f0d4ca00640 1 --2- 172.21.15.90:0/3174751038 >> v2:172.21.15.120:3300/0 conn(0x7f0d501603e0 0x7f0d5006b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:04.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.151+0000 7f0d4ca00640 1 -- 172.21.15.90:0/3174751038 >> v1:172.21.15.120:6789/0 conn(0x7f0d500cc150 legacy=0x7f0d500ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:04.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.151+0000 7f0d4ca00640 1 -- 172.21.15.90:0/3174751038 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d5015fbe0 con 0x7f0d50145430 2024-03-24T01:26:04.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.151+0000 7f0d4ca00640 1 -- 172.21.15.90:0/3174751038 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f0d480019f0 con 0x7f0d50145430 2024-03-24T01:26:04.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.151+0000 7f0d4ca00640 1 -- 172.21.15.90:0/3174751038 >> v1:172.21.15.90:6789/0 conn(0x7f0d50145430 legacy=0x7f0d500b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:04.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.151+0000 7f0d4ca00640 1 --2- 172.21.15.90:0/3174751038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d340039b0 0x7f0d34003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:04.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.151+0000 7f0d4ca00640 1 --2- 172.21.15.90:0/3174751038 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d34004800 0x7f0d34004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:04.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.151+0000 7f0d4ca00640 1 --2- 172.21.15.90:0/3174751038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d340051c0 0x7f0d3400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:04.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.151+0000 7f0d4ca00640 1 -- 172.21.15.90:0/3174751038 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d500cd270 con 0x7f0d340039b0 2024-03-24T01:26:04.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.151+0000 7f0d4ca00640 1 -- 172.21.15.90:0/3174751038 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3400c040 con 0x7f0d340051c0 2024-03-24T01:26:04.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.151+0000 7f0d4ca00640 1 -- 172.21.15.90:0/3174751038 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3400c240 con 0x7f0d34004800 2024-03-24T01:26:04.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.151+0000 7f0d4de00640 1 --2- 172.21.15.90:0/3174751038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d340039b0 0x7f0d34003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:04.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.151+0000 7f0d4de00640 1 -- 172.21.15.90:0/3174751038 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d34004800 msgr2=0x7f0d34004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:04.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.151+0000 7f0d4de00640 1 --2- 172.21.15.90:0/3174751038 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d34004800 0x7f0d34004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:04.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.151+0000 7f0d4de00640 1 -- 172.21.15.90:0/3174751038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d340051c0 msgr2=0x7f0d3400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:04.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.151+0000 7f0d4e800640 1 --2- 172.21.15.90:0/3174751038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d340051c0 0x7f0d3400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:04.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.151+0000 7f0d4de00640 1 --2- 172.21.15.90:0/3174751038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d340051c0 0x7f0d3400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:04.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.151+0000 7f0d4de00640 1 -- 172.21.15.90:0/3174751038 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d34003520 con 0x7f0d340039b0 2024-03-24T01:26:04.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.151+0000 7f0d4e800640 1 -- 172.21.15.90:0/3174751038 reap_dead start 2024-03-24T01:26:04.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.152+0000 7f0d4de00640 1 --2- 172.21.15.90:0/3174751038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d340039b0 0x7f0d34003db0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f0d40003db0 tx=0x7f0d34005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dbb3c20e7e5ab48 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:04.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.153+0000 7f0d4ca00640 1 -- 172.21.15.90:0/3174751038 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d40037070 con 0x7f0d340039b0 2024-03-24T01:26:04.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.153+0000 7f0d4ca00640 1 -- 172.21.15.90:0/3174751038 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d4002be50 con 0x7f0d340039b0 2024-03-24T01:26:04.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.153+0000 7f0d4ca00640 1 -- 172.21.15.90:0/3174751038 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d40003980 con 0x7f0d340039b0 2024-03-24T01:26:04.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.153+0000 7f0d55000640 1 -- 172.21.15.90:0/3174751038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d340039b0 msgr2=0x7f0d34003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:04.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.153+0000 7f0d55000640 1 --2- 172.21.15.90:0/3174751038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d340039b0 0x7f0d34003db0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f0d40003db0 tx=0x7f0d34005830 comp rx=0 tx=0).stop 2024-03-24T01:26:04.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.153+0000 7f0d55000640 1 -- 172.21.15.90:0/3174751038 shutdown_connections 2024-03-24T01:26:04.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.153+0000 7f0d55000640 1 --2- 172.21.15.90:0/3174751038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d340039b0 0x7f0d34003db0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:04.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.153+0000 7f0d55000640 1 -- 172.21.15.90:0/3174751038 >> 172.21.15.90:0/3174751038 conn(0x7f0d500b1770 msgr2=0x7f0d500b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:04.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.153+0000 7f0d55000640 1 -- 172.21.15.90:0/3174751038 shutdown_connections 2024-03-24T01:26:04.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.153+0000 7f0d55000640 1 -- 172.21.15.90:0/3174751038 wait complete. 2024-03-24T01:26:04.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.155+0000 7f0d55000640 1 Processor -- start 2024-03-24T01:26:04.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.155+0000 7f0d55000640 1 -- start start 2024-03-24T01:26:04.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.156+0000 7f0d55000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d340039b0 0x7f0d5015ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:04.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.156+0000 7f0d55000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d500ad7d0 0x7f0d5013be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:04.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.156+0000 7f0d55000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d500adc50 0x7f0d5013c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:04.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.156+0000 7f0d55000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d500ccce0 con 0x7f0d340039b0 2024-03-24T01:26:04.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.156+0000 7f0d55000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d500ccb60 con 0x7f0d500ad7d0 2024-03-24T01:26:04.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.156+0000 7f0d55000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d500cce60 con 0x7f0d500adc50 2024-03-24T01:26:04.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.156+0000 7f0d4de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d340039b0 0x7f0d5015ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:04.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.156+0000 7f0d4de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d340039b0 0x7f0d5015ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60016/0 (socket says 172.21.15.90:60016) 2024-03-24T01:26:04.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.156+0000 7f0d4de00640 1 -- 172.21.15.90:0/3581506488 learned_addr learned my addr 172.21.15.90:0/3581506488 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:04.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.156+0000 7f0d4d400640 1 --2- 172.21.15.90:0/3581506488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d500ad7d0 0x7f0d5013be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:04.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.157+0000 7f0d4de00640 1 -- 172.21.15.90:0/3581506488 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d500adc50 msgr2=0x7f0d5013c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:04.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.157+0000 7f0d4de00640 1 --2- 172.21.15.90:0/3581506488 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d500adc50 0x7f0d5013c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:04.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.157+0000 7f0d4de00640 1 -- 172.21.15.90:0/3581506488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d500ad7d0 msgr2=0x7f0d5013be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:04.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.157+0000 7f0d4de00640 1 --2- 172.21.15.90:0/3581506488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d500ad7d0 0x7f0d5013be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:04.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.157+0000 7f0d4de00640 1 -- 172.21.15.90:0/3581506488 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d501402e0 con 0x7f0d340039b0 2024-03-24T01:26:04.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.157+0000 7f0d4d400640 1 --2- 172.21.15.90:0/3581506488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d500ad7d0 0x7f0d5013be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:04.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.157+0000 7f0d4de00640 1 --2- 172.21.15.90:0/3581506488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d340039b0 0x7f0d5015ea80 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f0d40003db0 tx=0x7f0d400036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:04.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.157+0000 7f0d46000640 1 -- 172.21.15.90:0/3581506488 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d40042030 con 0x7f0d340039b0 2024-03-24T01:26:04.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.157+0000 7f0d46000640 1 -- 172.21.15.90:0/3581506488 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d40007470 con 0x7f0d340039b0 2024-03-24T01:26:04.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.157+0000 7f0d55000640 1 -- 172.21.15.90:0/3581506488 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d501405c0 con 0x7f0d340039b0 2024-03-24T01:26:04.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.157+0000 7f0d46000640 1 -- 172.21.15.90:0/3581506488 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d40037040 con 0x7f0d340039b0 2024-03-24T01:26:04.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.157+0000 7f0d55000640 1 -- 172.21.15.90:0/3581506488 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d50140b80 con 0x7f0d340039b0 2024-03-24T01:26:04.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.158+0000 7f0d55000640 1 -- 172.21.15.90:0/3581506488 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d50068060 con 0x7f0d340039b0 2024-03-24T01:26:04.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.159+0000 7f0d46000640 1 -- 172.21.15.90:0/3581506488 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0d40045070 con 0x7f0d340039b0 2024-03-24T01:26:04.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.161+0000 7f0d46000640 1 --2- 172.21.15.90:0/3581506488 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0d3006bbd0 0x7f0d3006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:04.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.161+0000 7f0d46000640 1 -- 172.21.15.90:0/3581506488 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0d400ac4c0 con 0x7f0d340039b0 2024-03-24T01:26:04.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.162+0000 7f0d4d400640 1 --2- 172.21.15.90:0/3581506488 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0d3006bbd0 0x7f0d3006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:04.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.163+0000 7f0d4d400640 1 --2- 172.21.15.90:0/3581506488 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0d3006bbd0 0x7f0d3006e090 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0d50160810 tx=0x7f0d3c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:04.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.166+0000 7f0d46000640 1 -- 172.21.15.90:0/3581506488 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0d4007ae10 con 0x7f0d340039b0 2024-03-24T01:26:04.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.346+0000 7f0d55000640 1 -- 172.21.15.90:0/3581506488 --> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0d500b3220 con 0x7f0d3006bbd0 2024-03-24T01:26:04.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.366+0000 7f0d46000640 1 -- 172.21.15.90:0/3581506488 <== mgr.4104 v2:172.21.15.120:6832/860392634 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246634 (secure 0 0 0) 0x7f0d500b3220 con 0x7f0d3006bbd0 2024-03-24T01:26:04.370 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:26:04.372 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-24T01:26:04.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.372+0000 7f0d55000640 1 -- 172.21.15.90:0/3581506488 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0d3006bbd0 msgr2=0x7f0d3006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:04.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.372+0000 7f0d55000640 1 --2- 172.21.15.90:0/3581506488 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0d3006bbd0 0x7f0d3006e090 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0d50160810 tx=0x7f0d3c008040 comp rx=0 tx=0).stop 2024-03-24T01:26:04.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.372+0000 7f0d55000640 1 -- 172.21.15.90:0/3581506488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d340039b0 msgr2=0x7f0d5015ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:04.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.372+0000 7f0d55000640 1 --2- 172.21.15.90:0/3581506488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d340039b0 0x7f0d5015ea80 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f0d40003db0 tx=0x7f0d400036f0 comp rx=0 tx=0).stop 2024-03-24T01:26:04.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.372+0000 7f0d55000640 1 -- 172.21.15.90:0/3581506488 shutdown_connections 2024-03-24T01:26:04.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.372+0000 7f0d55000640 1 --2- 172.21.15.90:0/3581506488 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0d3006bbd0 0x7f0d3006e090 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:04.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.372+0000 7f0d55000640 1 --2- 172.21.15.90:0/3581506488 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0d500adc50 0x7f0d5013c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:04.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.372+0000 7f0d55000640 1 --2- 172.21.15.90:0/3581506488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d500ad7d0 0x7f0d5013be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:04.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.372+0000 7f0d55000640 1 --2- 172.21.15.90:0/3581506488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d340039b0 0x7f0d5015ea80 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:04.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.372+0000 7f0d55000640 1 -- 172.21.15.90:0/3581506488 >> 172.21.15.90:0/3581506488 conn(0x7f0d500b1770 msgr2=0x7f0d500b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:04.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.372+0000 7f0d55000640 1 -- 172.21.15.90:0/3581506488 shutdown_connections 2024-03-24T01:26:04.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.373+0000 7f0d55000640 1 -- 172.21.15.90:0/3581506488 wait complete. 2024-03-24T01:26:04.398 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":94,"stamp":"2024-03-24T01:26:03.521564+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220600,"kb_used_data":4428,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754120,"statfs":{"total":773094113280,"available":772868218880,"internally_reserved":0,"allocated":4534272,"data_stored":2615145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12747,"internal_metadata":221236789},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.573169"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.977282+0000","last_change":"2024-03-24T01:25:58.977282+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.977282+0000","last_undegraded":"2024-03-24T01:25:58.977282+0000","last_fullsized":"2024-03-24T01:25:58.977282+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.975953+0000","last_change":"2024-03-24T01:25:58.975953+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.975953+0000","last_undegraded":"2024-03-24T01:25:58.975953+0000","last_fullsized":"2024-03-24T01:25:58.975953+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.979446+0000","last_change":"2024-03-24T01:25:58.979446+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.979446+0000","last_undegraded":"2024-03-24T01:25:58.979446+0000","last_fullsized":"2024-03-24T01:25:58.979446+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.974695+0000","last_change":"2024-03-24T01:25:58.974695+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.974695+0000","last_undegraded":"2024-03-24T01:25:58.974695+0000","last_fullsized":"2024-03-24T01:25:58.974695+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.973754+0000","last_change":"2024-03-24T01:25:58.973754+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.973754+0000","last_undegraded":"2024-03-24T01:25:58.973754+0000","last_fullsized":"2024-03-24T01:25:58.973754+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.986679+0000","last_change":"2024-03-24T01:25:58.986679+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.986679+0000","last_undegraded":"2024-03-24T01:25:58.986679+0000","last_fullsized":"2024-03-24T01:25:58.986679+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.989725+0000","last_change":"2024-03-24T01:25:58.989725+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.989725+0000","last_undegraded":"2024-03-24T01:25:58.989725+0000","last_fullsized":"2024-03-24T01:25:58.989725+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.978405+0000","last_change":"2024-03-24T01:25:58.978405+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.978405+0000","last_undegraded":"2024-03-24T01:25:58.978405+0000","last_fullsized":"2024-03-24T01:25:58.978405+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.978353+0000","last_change":"2024-03-24T01:25:58.978353+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.978353+0000","last_undegraded":"2024-03-24T01:25:58.978353+0000","last_fullsized":"2024-03-24T01:25:58.978353+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.995242+0000","last_change":"2024-03-24T01:25:58.995242+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.995242+0000","last_undegraded":"2024-03-24T01:25:58.995242+0000","last_fullsized":"2024-03-24T01:25:58.995242+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.990081+0000","last_change":"2024-03-24T01:25:58.990081+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.990081+0000","last_undegraded":"2024-03-24T01:25:58.990081+0000","last_fullsized":"2024-03-24T01:25:58.990081+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.976660+0000","last_change":"2024-03-24T01:25:58.976660+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.976660+0000","last_undegraded":"2024-03-24T01:25:58.976660+0000","last_fullsized":"2024-03-24T01:25:58.976660+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.986804+0000","last_change":"2024-03-24T01:25:58.986804+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.986804+0000","last_undegraded":"2024-03-24T01:25:58.986804+0000","last_fullsized":"2024-03-24T01:25:58.986804+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.980568+0000","last_change":"2024-03-24T01:25:58.980568+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.980568+0000","last_undegraded":"2024-03-24T01:25:58.980568+0000","last_fullsized":"2024-03-24T01:25:58.980568+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.986804+0000","last_change":"2024-03-24T01:25:58.986804+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.986804+0000","last_undegraded":"2024-03-24T01:25:58.986804+0000","last_fullsized":"2024-03-24T01:25:58.986804+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.993159+0000","last_change":"2024-03-24T01:25:58.993159+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.993159+0000","last_undegraded":"2024-03-24T01:25:58.993159+0000","last_fullsized":"2024-03-24T01:25:58.993159+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.991992+0000","last_change":"2024-03-24T01:25:58.991992+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.991992+0000","last_undegraded":"2024-03-24T01:25:58.991992+0000","last_fullsized":"2024-03-24T01:25:58.991992+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.974166+0000","last_change":"2024-03-24T01:25:58.974166+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.974166+0000","last_undegraded":"2024-03-24T01:25:58.974166+0000","last_fullsized":"2024-03-24T01:25:58.974166+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.973263+0000","last_change":"2024-03-24T01:25:58.973263+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.973263+0000","last_undegraded":"2024-03-24T01:25:58.973263+0000","last_fullsized":"2024-03-24T01:25:58.973263+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.974422+0000","last_change":"2024-03-24T01:25:58.974422+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.974422+0000","last_undegraded":"2024-03-24T01:25:58.974422+0000","last_fullsized":"2024-03-24T01:25:58.974422+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.986049+0000","last_change":"2024-03-24T01:25:58.986049+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.986049+0000","last_undegraded":"2024-03-24T01:25:58.986049+0000","last_fullsized":"2024-03-24T01:25:58.986049+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.977959+0000","last_change":"2024-03-24T01:25:58.977959+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.977959+0000","last_undegraded":"2024-03-24T01:25:58.977959+0000","last_fullsized":"2024-03-24T01:25:58.977959+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.989831+0000","last_change":"2024-03-24T01:25:58.989831+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.989831+0000","last_undegraded":"2024-03-24T01:25:58.989831+0000","last_fullsized":"2024-03-24T01:25:58.989831+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.972222+0000","last_change":"2024-03-24T01:25:58.972222+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.972222+0000","last_undegraded":"2024-03-24T01:25:58.972222+0000","last_fullsized":"2024-03-24T01:25:58.972222+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.975433+0000","last_change":"2024-03-24T01:25:58.975433+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.975433+0000","last_undegraded":"2024-03-24T01:25:58.975433+0000","last_fullsized":"2024-03-24T01:25:58.975433+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.985886+0000","last_change":"2024-03-24T01:25:58.985886+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.985886+0000","last_undegraded":"2024-03-24T01:25:58.985886+0000","last_fullsized":"2024-03-24T01:25:58.985886+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.977178+0000","last_change":"2024-03-24T01:25:58.977178+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.977178+0000","last_undegraded":"2024-03-24T01:25:58.977178+0000","last_fullsized":"2024-03-24T01:25:58.977178+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.986473+0000","last_change":"2024-03-24T01:25:58.986473+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.986473+0000","last_undegraded":"2024-03-24T01:25:58.986473+0000","last_fullsized":"2024-03-24T01:25:58.986473+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.975161+0000","last_change":"2024-03-24T01:25:58.975161+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.975161+0000","last_undegraded":"2024-03-24T01:25:58.975161+0000","last_fullsized":"2024-03-24T01:25:58.975161+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.983076+0000","last_change":"2024-03-24T01:25:58.983076+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.983076+0000","last_undegraded":"2024-03-24T01:25:58.983076+0000","last_fullsized":"2024-03-24T01:25:58.983076+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.989181+0000","last_change":"2024-03-24T01:25:58.989181+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.989181+0000","last_undegraded":"2024-03-24T01:25:58.989181+0000","last_fullsized":"2024-03-24T01:25:58.989181+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.981936+0000","last_change":"2024-03-24T01:25:58.981936+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.981936+0000","last_undegraded":"2024-03-24T01:25:58.981936+0000","last_fullsized":"2024-03-24T01:25:58.981936+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.994459+0000","last_change":"2024-03-24T01:25:58.994459+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.994459+0000","last_undegraded":"2024-03-24T01:25:58.994459+0000","last_fullsized":"2024-03-24T01:25:58.994459+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.001458+0000","last_change":"2024-03-24T01:25:58.996685+0000","last_active":"2024-03-24T01:25:59.001458+0000","last_peered":"2024-03-24T01:25:59.001458+0000","last_clean":"2024-03-24T01:25:59.001458+0000","last_became_active":"2024-03-24T01:25:58.996398+0000","last_became_peered":"2024-03-24T01:25:58.996398+0000","last_unstale":"2024-03-24T01:25:59.001458+0000","last_undegraded":"2024-03-24T01:25:59.001458+0000","last_fullsized":"2024-03-24T01:25:59.001458+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:53:49.271423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.984299+0000","last_change":"2024-03-24T01:25:58.984299+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.984299+0000","last_undegraded":"2024-03-24T01:25:58.984299+0000","last_fullsized":"2024-03-24T01:25:58.984299+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.004893+0000","last_change":"2024-03-24T01:25:59.002453+0000","last_active":"2024-03-24T01:25:59.004893+0000","last_peered":"2024-03-24T01:25:59.004893+0000","last_clean":"2024-03-24T01:25:59.004893+0000","last_became_active":"2024-03-24T01:25:59.002155+0000","last_became_peered":"2024-03-24T01:25:59.002155+0000","last_unstale":"2024-03-24T01:25:59.004893+0000","last_undegraded":"2024-03-24T01:25:59.004893+0000","last_fullsized":"2024-03-24T01:25:59.004893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:53:49.271423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.978899+0000","last_change":"2024-03-24T01:25:58.978899+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.978899+0000","last_undegraded":"2024-03-24T01:25:58.978899+0000","last_fullsized":"2024-03-24T01:25:58.978899+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.998379+0000","last_change":"2024-03-24T01:25:58.997334+0000","last_active":"2024-03-24T01:25:58.998379+0000","last_peered":"2024-03-24T01:25:58.998379+0000","last_clean":"2024-03-24T01:25:58.998379+0000","last_became_active":"2024-03-24T01:25:58.997033+0000","last_became_peered":"2024-03-24T01:25:58.997033+0000","last_unstale":"2024-03-24T01:25:58.998379+0000","last_undegraded":"2024-03-24T01:25:58.998379+0000","last_fullsized":"2024-03-24T01:25:58.998379+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:45.605005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.977778+0000","last_change":"2024-03-24T01:25:58.977778+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.977778+0000","last_undegraded":"2024-03-24T01:25:58.977778+0000","last_fullsized":"2024-03-24T01:25:58.977778+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.996426+0000","last_change":"2024-03-24T01:25:58.992317+0000","last_active":"2024-03-24T01:25:58.996426+0000","last_peered":"2024-03-24T01:25:58.996426+0000","last_clean":"2024-03-24T01:25:58.996426+0000","last_became_active":"2024-03-24T01:25:58.991652+0000","last_became_peered":"2024-03-24T01:25:58.991652+0000","last_unstale":"2024-03-24T01:25:58.996426+0000","last_undegraded":"2024-03-24T01:25:58.996426+0000","last_fullsized":"2024-03-24T01:25:58.996426+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:53:49.271423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.982673+0000","last_change":"2024-03-24T01:25:58.982673+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.982673+0000","last_undegraded":"2024-03-24T01:25:58.982673+0000","last_fullsized":"2024-03-24T01:25:58.982673+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.004193+0000","last_change":"2024-03-24T01:25:58.998337+0000","last_active":"2024-03-24T01:25:59.004193+0000","last_peered":"2024-03-24T01:25:59.004193+0000","last_clean":"2024-03-24T01:25:59.004193+0000","last_became_active":"2024-03-24T01:25:58.998056+0000","last_became_peered":"2024-03-24T01:25:58.998056+0000","last_unstale":"2024-03-24T01:25:59.004193+0000","last_undegraded":"2024-03-24T01:25:59.004193+0000","last_fullsized":"2024-03-24T01:25:59.004193+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:49:24.128591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.985591+0000","last_change":"2024-03-24T01:25:58.985591+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.985591+0000","last_undegraded":"2024-03-24T01:25:58.985591+0000","last_fullsized":"2024-03-24T01:25:58.985591+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.012523+0000","last_change":"2024-03-24T01:25:59.006913+0000","last_active":"2024-03-24T01:25:59.012523+0000","last_peered":"2024-03-24T01:25:59.012523+0000","last_clean":"2024-03-24T01:25:59.012523+0000","last_became_active":"2024-03-24T01:25:59.006098+0000","last_became_peered":"2024-03-24T01:25:59.006098+0000","last_unstale":"2024-03-24T01:25:59.012523+0000","last_undegraded":"2024-03-24T01:25:59.012523+0000","last_fullsized":"2024-03-24T01:25:59.012523+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:49:24.128591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.976758+0000","last_change":"2024-03-24T01:25:58.976758+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.976758+0000","last_undegraded":"2024-03-24T01:25:58.976758+0000","last_fullsized":"2024-03-24T01:25:58.976758+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.004687+0000","last_change":"2024-03-24T01:25:59.000318+0000","last_active":"2024-03-24T01:25:59.004687+0000","last_peered":"2024-03-24T01:25:59.004687+0000","last_clean":"2024-03-24T01:25:59.004687+0000","last_became_active":"2024-03-24T01:25:59.000085+0000","last_became_peered":"2024-03-24T01:25:59.000085+0000","last_unstale":"2024-03-24T01:25:59.004687+0000","last_undegraded":"2024-03-24T01:25:59.004687+0000","last_fullsized":"2024-03-24T01:25:59.004687+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:28:18.909603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.980649+0000","last_change":"2024-03-24T01:25:58.980649+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.980649+0000","last_undegraded":"2024-03-24T01:25:58.980649+0000","last_fullsized":"2024-03-24T01:25:58.980649+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.999166+0000","last_change":"2024-03-24T01:25:58.994521+0000","last_active":"2024-03-24T01:25:58.999166+0000","last_peered":"2024-03-24T01:25:58.999166+0000","last_clean":"2024-03-24T01:25:58.999166+0000","last_became_active":"2024-03-24T01:25:58.994215+0000","last_became_peered":"2024-03-24T01:25:58.994215+0000","last_unstale":"2024-03-24T01:25:58.999166+0000","last_undegraded":"2024-03-24T01:25:58.999166+0000","last_fullsized":"2024-03-24T01:25:58.999166+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:45.605005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.972522+0000","last_change":"2024-03-24T01:25:58.972522+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.972522+0000","last_undegraded":"2024-03-24T01:25:58.972522+0000","last_fullsized":"2024-03-24T01:25:58.972522+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.996515+0000","last_change":"2024-03-24T01:25:58.995322+0000","last_active":"2024-03-24T01:25:58.996515+0000","last_peered":"2024-03-24T01:25:58.996515+0000","last_clean":"2024-03-24T01:25:58.996515+0000","last_became_active":"2024-03-24T01:25:58.994847+0000","last_became_peered":"2024-03-24T01:25:58.994847+0000","last_unstale":"2024-03-24T01:25:58.996515+0000","last_undegraded":"2024-03-24T01:25:58.996515+0000","last_fullsized":"2024-03-24T01:25:58.996515+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:12:43.140922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.995109+0000","last_change":"2024-03-24T01:25:58.995109+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.995109+0000","last_undegraded":"2024-03-24T01:25:58.995109+0000","last_fullsized":"2024-03-24T01:25:58.995109+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.004224+0000","last_change":"2024-03-24T01:25:58.998109+0000","last_active":"2024-03-24T01:25:59.004224+0000","last_peered":"2024-03-24T01:25:59.004224+0000","last_clean":"2024-03-24T01:25:59.004224+0000","last_became_active":"2024-03-24T01:25:58.997791+0000","last_became_peered":"2024-03-24T01:25:58.997791+0000","last_unstale":"2024-03-24T01:25:59.004224+0000","last_undegraded":"2024-03-24T01:25:59.004224+0000","last_fullsized":"2024-03-24T01:25:59.004224+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:00:42.635663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.976014+0000","last_change":"2024-03-24T01:25:58.976014+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.976014+0000","last_undegraded":"2024-03-24T01:25:58.976014+0000","last_fullsized":"2024-03-24T01:25:58.976014+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.004257+0000","last_change":"2024-03-24T01:25:58.999040+0000","last_active":"2024-03-24T01:25:59.004257+0000","last_peered":"2024-03-24T01:25:59.004257+0000","last_clean":"2024-03-24T01:25:59.004257+0000","last_became_active":"2024-03-24T01:25:58.998593+0000","last_became_peered":"2024-03-24T01:25:58.998593+0000","last_unstale":"2024-03-24T01:25:59.004257+0000","last_undegraded":"2024-03-24T01:25:59.004257+0000","last_fullsized":"2024-03-24T01:25:59.004257+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:53:49.271423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.978116+0000","last_change":"2024-03-24T01:25:58.978116+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.978116+0000","last_undegraded":"2024-03-24T01:25:58.978116+0000","last_fullsized":"2024-03-24T01:25:58.978116+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.007336+0000","last_change":"2024-03-24T01:25:59.002378+0000","last_active":"2024-03-24T01:25:59.007336+0000","last_peered":"2024-03-24T01:25:59.007336+0000","last_clean":"2024-03-24T01:25:59.007336+0000","last_became_active":"2024-03-24T01:25:59.002071+0000","last_became_peered":"2024-03-24T01:25:59.002071+0000","last_unstale":"2024-03-24T01:25:59.007336+0000","last_undegraded":"2024-03-24T01:25:59.007336+0000","last_fullsized":"2024-03-24T01:25:59.007336+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:05:31.709112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.980063+0000","last_change":"2024-03-24T01:25:58.980063+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.980063+0000","last_undegraded":"2024-03-24T01:25:58.980063+0000","last_fullsized":"2024-03-24T01:25:58.980063+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.990060+0000","last_change":"2024-03-24T01:25:58.987270+0000","last_active":"2024-03-24T01:25:58.990060+0000","last_peered":"2024-03-24T01:25:58.990060+0000","last_clean":"2024-03-24T01:25:58.990060+0000","last_became_active":"2024-03-24T01:25:58.986953+0000","last_became_peered":"2024-03-24T01:25:58.986953+0000","last_unstale":"2024-03-24T01:25:58.990060+0000","last_undegraded":"2024-03-24T01:25:58.990060+0000","last_fullsized":"2024-03-24T01:25:58.990060+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:19:04.197281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.981124+0000","last_change":"2024-03-24T01:25:58.981124+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.981124+0000","last_undegraded":"2024-03-24T01:25:58.981124+0000","last_fullsized":"2024-03-24T01:25:58.981124+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.004607+0000","last_change":"2024-03-24T01:25:59.000275+0000","last_active":"2024-03-24T01:25:59.004607+0000","last_peered":"2024-03-24T01:25:59.004607+0000","last_clean":"2024-03-24T01:25:59.004607+0000","last_became_active":"2024-03-24T01:25:58.999948+0000","last_became_peered":"2024-03-24T01:25:58.999948+0000","last_unstale":"2024-03-24T01:25:59.004607+0000","last_undegraded":"2024-03-24T01:25:59.004607+0000","last_fullsized":"2024-03-24T01:25:59.004607+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:44:12.069166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.987147+0000","last_change":"2024-03-24T01:25:58.987147+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.987147+0000","last_undegraded":"2024-03-24T01:25:58.987147+0000","last_fullsized":"2024-03-24T01:25:58.987147+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.995109+0000","last_change":"2024-03-24T01:25:58.991563+0000","last_active":"2024-03-24T01:25:58.995109+0000","last_peered":"2024-03-24T01:25:58.995109+0000","last_clean":"2024-03-24T01:25:58.995109+0000","last_became_active":"2024-03-24T01:25:58.990946+0000","last_became_peered":"2024-03-24T01:25:58.990946+0000","last_unstale":"2024-03-24T01:25:58.995109+0000","last_undegraded":"2024-03-24T01:25:58.995109+0000","last_fullsized":"2024-03-24T01:25:58.995109+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:49:24.128591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.980154+0000","last_change":"2024-03-24T01:25:58.980154+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.980154+0000","last_undegraded":"2024-03-24T01:25:58.980154+0000","last_fullsized":"2024-03-24T01:25:58.980154+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.009845+0000","last_change":"2024-03-24T01:25:59.006569+0000","last_active":"2024-03-24T01:25:59.009845+0000","last_peered":"2024-03-24T01:25:59.009845+0000","last_clean":"2024-03-24T01:25:59.009845+0000","last_became_active":"2024-03-24T01:25:59.006280+0000","last_became_peered":"2024-03-24T01:25:59.006280+0000","last_unstale":"2024-03-24T01:25:59.009845+0000","last_undegraded":"2024-03-24T01:25:59.009845+0000","last_fullsized":"2024-03-24T01:25:59.009845+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:00:42.635663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.976902+0000","last_change":"2024-03-24T01:25:58.976902+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.976902+0000","last_undegraded":"2024-03-24T01:25:58.976902+0000","last_fullsized":"2024-03-24T01:25:58.976902+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.005436+0000","last_change":"2024-03-24T01:25:59.003724+0000","last_active":"2024-03-24T01:25:59.005436+0000","last_peered":"2024-03-24T01:25:59.005436+0000","last_clean":"2024-03-24T01:25:59.005436+0000","last_became_active":"2024-03-24T01:25:59.003410+0000","last_became_peered":"2024-03-24T01:25:59.003410+0000","last_unstale":"2024-03-24T01:25:59.005436+0000","last_undegraded":"2024-03-24T01:25:59.005436+0000","last_fullsized":"2024-03-24T01:25:59.005436+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:08:38.108175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.989181+0000","last_change":"2024-03-24T01:25:58.989181+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.989181+0000","last_undegraded":"2024-03-24T01:25:58.989181+0000","last_fullsized":"2024-03-24T01:25:58.989181+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.002870+0000","last_change":"2024-03-24T01:25:58.999764+0000","last_active":"2024-03-24T01:25:59.002870+0000","last_peered":"2024-03-24T01:25:59.002870+0000","last_clean":"2024-03-24T01:25:59.002870+0000","last_became_active":"2024-03-24T01:25:58.999490+0000","last_became_peered":"2024-03-24T01:25:58.999490+0000","last_unstale":"2024-03-24T01:25:59.002870+0000","last_undegraded":"2024-03-24T01:25:59.002870+0000","last_fullsized":"2024-03-24T01:25:59.002870+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:44:12.069166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.983702+0000","last_change":"2024-03-24T01:25:58.983702+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.983702+0000","last_undegraded":"2024-03-24T01:25:58.983702+0000","last_fullsized":"2024-03-24T01:25:58.983702+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.007469+0000","last_change":"2024-03-24T01:25:59.003238+0000","last_active":"2024-03-24T01:25:59.007469+0000","last_peered":"2024-03-24T01:25:59.007469+0000","last_clean":"2024-03-24T01:25:59.007469+0000","last_became_active":"2024-03-24T01:25:59.002905+0000","last_became_peered":"2024-03-24T01:25:59.002905+0000","last_unstale":"2024-03-24T01:25:59.007469+0000","last_undegraded":"2024-03-24T01:25:59.007469+0000","last_fullsized":"2024-03-24T01:25:59.007469+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:56:29.523110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.972343+0000","last_change":"2024-03-24T01:25:58.972343+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.972343+0000","last_undegraded":"2024-03-24T01:25:58.972343+0000","last_fullsized":"2024-03-24T01:25:58.972343+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.996158+0000","last_change":"2024-03-24T01:25:58.994043+0000","last_active":"2024-03-24T01:25:58.996158+0000","last_peered":"2024-03-24T01:25:58.996158+0000","last_clean":"2024-03-24T01:25:58.996158+0000","last_became_active":"2024-03-24T01:25:58.993746+0000","last_became_peered":"2024-03-24T01:25:58.993746+0000","last_unstale":"2024-03-24T01:25:58.996158+0000","last_undegraded":"2024-03-24T01:25:58.996158+0000","last_fullsized":"2024-03-24T01:25:58.996158+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:08:38.108175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.987330+0000","last_change":"2024-03-24T01:25:58.987330+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.987330+0000","last_undegraded":"2024-03-24T01:25:58.987330+0000","last_fullsized":"2024-03-24T01:25:58.987330+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.991383+0000","last_change":"2024-03-24T01:25:58.988228+0000","last_active":"2024-03-24T01:25:58.991383+0000","last_peered":"2024-03-24T01:25:58.991383+0000","last_clean":"2024-03-24T01:25:58.991383+0000","last_became_active":"2024-03-24T01:25:58.987941+0000","last_became_peered":"2024-03-24T01:25:58.987941+0000","last_unstale":"2024-03-24T01:25:58.991383+0000","last_undegraded":"2024-03-24T01:25:58.991383+0000","last_fullsized":"2024-03-24T01:25:58.991383+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:12:25.446123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.976174+0000","last_change":"2024-03-24T01:25:58.976174+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.976174+0000","last_undegraded":"2024-03-24T01:25:58.976174+0000","last_fullsized":"2024-03-24T01:25:58.976174+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.993352+0000","last_change":"2024-03-24T01:25:58.991840+0000","last_active":"2024-03-24T01:25:58.993352+0000","last_peered":"2024-03-24T01:25:58.993352+0000","last_clean":"2024-03-24T01:25:58.993352+0000","last_became_active":"2024-03-24T01:25:58.991522+0000","last_became_peered":"2024-03-24T01:25:58.991522+0000","last_unstale":"2024-03-24T01:25:58.993352+0000","last_undegraded":"2024-03-24T01:25:58.993352+0000","last_fullsized":"2024-03-24T01:25:58.993352+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:03:47.985379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.985984+0000","last_change":"2024-03-24T01:25:58.985984+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.985984+0000","last_undegraded":"2024-03-24T01:25:58.985984+0000","last_fullsized":"2024-03-24T01:25:58.985984+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.009786+0000","last_change":"2024-03-24T01:25:59.006226+0000","last_active":"2024-03-24T01:25:59.009786+0000","last_peered":"2024-03-24T01:25:59.009786+0000","last_clean":"2024-03-24T01:25:59.009786+0000","last_became_active":"2024-03-24T01:25:59.005739+0000","last_became_peered":"2024-03-24T01:25:59.005739+0000","last_unstale":"2024-03-24T01:25:59.009786+0000","last_undegraded":"2024-03-24T01:25:59.009786+0000","last_fullsized":"2024-03-24T01:25:59.009786+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:07:08.132339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.976537+0000","last_change":"2024-03-24T01:25:58.976537+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.976537+0000","last_undegraded":"2024-03-24T01:25:58.976537+0000","last_fullsized":"2024-03-24T01:25:58.976537+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.991351+0000","last_change":"2024-03-24T01:25:58.988520+0000","last_active":"2024-03-24T01:25:58.991351+0000","last_peered":"2024-03-24T01:25:58.991351+0000","last_clean":"2024-03-24T01:25:58.991351+0000","last_became_active":"2024-03-24T01:25:58.988149+0000","last_became_peered":"2024-03-24T01:25:58.988149+0000","last_unstale":"2024-03-24T01:25:58.991351+0000","last_undegraded":"2024-03-24T01:25:58.991351+0000","last_fullsized":"2024-03-24T01:25:58.991351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:13:07.896803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.983029+0000","last_change":"2024-03-24T01:25:58.983029+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.983029+0000","last_undegraded":"2024-03-24T01:25:58.983029+0000","last_fullsized":"2024-03-24T01:25:58.983029+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.009953+0000","last_change":"2024-03-24T01:25:59.005278+0000","last_active":"2024-03-24T01:25:59.009953+0000","last_peered":"2024-03-24T01:25:59.009953+0000","last_clean":"2024-03-24T01:25:59.009953+0000","last_became_active":"2024-03-24T01:25:59.004896+0000","last_became_peered":"2024-03-24T01:25:59.004896+0000","last_unstale":"2024-03-24T01:25:59.009953+0000","last_undegraded":"2024-03-24T01:25:59.009953+0000","last_fullsized":"2024-03-24T01:25:59.009953+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:45.605005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.989993+0000","last_change":"2024-03-24T01:25:58.989993+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.989993+0000","last_undegraded":"2024-03-24T01:25:58.989993+0000","last_fullsized":"2024-03-24T01:25:58.989993+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.007491+0000","last_change":"2024-03-24T01:25:59.004018+0000","last_active":"2024-03-24T01:25:59.007491+0000","last_peered":"2024-03-24T01:25:59.007491+0000","last_clean":"2024-03-24T01:25:59.007491+0000","last_became_active":"2024-03-24T01:25:59.003661+0000","last_became_peered":"2024-03-24T01:25:59.003661+0000","last_unstale":"2024-03-24T01:25:59.007491+0000","last_undegraded":"2024-03-24T01:25:59.007491+0000","last_fullsized":"2024-03-24T01:25:59.007491+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:35:02.448771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.975406+0000","last_change":"2024-03-24T01:25:58.975406+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.975406+0000","last_undegraded":"2024-03-24T01:25:58.975406+0000","last_fullsized":"2024-03-24T01:25:58.975406+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.998477+0000","last_change":"2024-03-24T01:25:58.995201+0000","last_active":"2024-03-24T01:25:58.998477+0000","last_peered":"2024-03-24T01:25:58.998477+0000","last_clean":"2024-03-24T01:25:58.998477+0000","last_became_active":"2024-03-24T01:25:58.994890+0000","last_became_peered":"2024-03-24T01:25:58.994890+0000","last_unstale":"2024-03-24T01:25:58.998477+0000","last_undegraded":"2024-03-24T01:25:58.998477+0000","last_fullsized":"2024-03-24T01:25:58.998477+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:49:24.128591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.978947+0000","last_change":"2024-03-24T01:25:58.978947+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.978947+0000","last_undegraded":"2024-03-24T01:25:58.978947+0000","last_fullsized":"2024-03-24T01:25:58.978947+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.993897+0000","last_change":"2024-03-24T01:25:58.990591+0000","last_active":"2024-03-24T01:25:58.993897+0000","last_peered":"2024-03-24T01:25:58.993897+0000","last_clean":"2024-03-24T01:25:58.993897+0000","last_became_active":"2024-03-24T01:25:58.990269+0000","last_became_peered":"2024-03-24T01:25:58.990269+0000","last_unstale":"2024-03-24T01:25:58.993897+0000","last_undegraded":"2024-03-24T01:25:58.993897+0000","last_fullsized":"2024-03-24T01:25:58.993897+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:00:42.635663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.979426+0000","last_change":"2024-03-24T01:25:58.979426+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.979426+0000","last_undegraded":"2024-03-24T01:25:58.979426+0000","last_fullsized":"2024-03-24T01:25:58.979426+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.993563+0000","last_change":"2024-03-24T01:25:58.988526+0000","last_active":"2024-03-24T01:25:58.993563+0000","last_peered":"2024-03-24T01:25:58.993563+0000","last_clean":"2024-03-24T01:25:58.993563+0000","last_became_active":"2024-03-24T01:25:58.987932+0000","last_became_peered":"2024-03-24T01:25:58.987932+0000","last_unstale":"2024-03-24T01:25:58.993563+0000","last_undegraded":"2024-03-24T01:25:58.993563+0000","last_fullsized":"2024-03-24T01:25:58.993563+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:00:42.635663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.979324+0000","last_change":"2024-03-24T01:25:58.979324+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.979324+0000","last_undegraded":"2024-03-24T01:25:58.979324+0000","last_fullsized":"2024-03-24T01:25:58.979324+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.995934+0000","last_change":"2024-03-24T01:25:58.992830+0000","last_active":"2024-03-24T01:25:58.995934+0000","last_peered":"2024-03-24T01:25:58.995934+0000","last_clean":"2024-03-24T01:25:58.995934+0000","last_became_active":"2024-03-24T01:25:58.991569+0000","last_became_peered":"2024-03-24T01:25:58.991569+0000","last_unstale":"2024-03-24T01:25:58.995934+0000","last_undegraded":"2024-03-24T01:25:58.995934+0000","last_fullsized":"2024-03-24T01:25:58.995934+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:08:38.108175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.987080+0000","last_change":"2024-03-24T01:25:58.987080+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.987080+0000","last_undegraded":"2024-03-24T01:25:58.987080+0000","last_fullsized":"2024-03-24T01:25:58.987080+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.997322+0000","last_change":"2024-03-24T01:25:58.996128+0000","last_active":"2024-03-24T01:25:58.997322+0000","last_peered":"2024-03-24T01:25:58.997322+0000","last_clean":"2024-03-24T01:25:58.997322+0000","last_became_active":"2024-03-24T01:25:58.994831+0000","last_became_peered":"2024-03-24T01:25:58.994831+0000","last_unstale":"2024-03-24T01:25:58.997322+0000","last_undegraded":"2024-03-24T01:25:58.997322+0000","last_fullsized":"2024-03-24T01:25:58.997322+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:46:49.838842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.991355+0000","last_change":"2024-03-24T01:25:58.991355+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.991355+0000","last_undegraded":"2024-03-24T01:25:58.991355+0000","last_fullsized":"2024-03-24T01:25:58.991355+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.012545+0000","last_change":"2024-03-24T01:25:59.008252+0000","last_active":"2024-03-24T01:25:59.012545+0000","last_peered":"2024-03-24T01:25:59.012545+0000","last_clean":"2024-03-24T01:25:59.012545+0000","last_became_active":"2024-03-24T01:25:59.007969+0000","last_became_peered":"2024-03-24T01:25:59.007969+0000","last_unstale":"2024-03-24T01:25:59.012545+0000","last_undegraded":"2024-03-24T01:25:59.012545+0000","last_fullsized":"2024-03-24T01:25:59.012545+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:53:49.271423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":200,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.999857+0000","last_change":"2024-03-24T01:24:01.545855+0000","last_active":"2024-03-24T01:25:58.999857+0000","last_peered":"2024-03-24T01:25:58.999857+0000","last_clean":"2024-03-24T01:25:58.999857+0000","last_became_active":"2024-03-24T01:24:01.543811+0000","last_became_peered":"2024-03-24T01:24:01.543811+0000","last_unstale":"2024-03-24T01:25:58.999857+0000","last_undegraded":"2024-03-24T01:25:58.999857+0000","last_fullsized":"2024-03-24T01:25:58.999857+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_clean_scrub_stamp":"2024-03-24T01:23:59.861564+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:03.682631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343836,"statfs":{"total":96636764160,"available":96608088064,"internally_reserved":0,"allocated":1007616,"data_stored":767169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574875,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":176801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574876,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":176801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574875,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":425984,"data_stored":183468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":176801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":425984,"data_stored":183468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":176801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343820,"statfs":{"total":96636764160,"available":96608071680,"internally_reserved":0,"allocated":1019904,"data_stored":773836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T01:26:04.402 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T01:26:04.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.653+0000 7fc531000640 1 Processor -- start 2024-03-24T01:26:04.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.653+0000 7fc531000640 1 -- start start 2024-03-24T01:26:04.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.655+0000 7fc531000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc52c0b5c30 0x7fc52c0b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:04.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.655+0000 7fc531000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc52c0b5430 0x7fc52c0ceb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:04.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.655+0000 7fc531000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc52c0b25d0 con 0x7fc52c15e280 2024-03-24T01:26:04.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.655+0000 7fc531000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc52c0b5880 con 0x7fc52c0b5c30 2024-03-24T01:26:04.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.655+0000 7fc531000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc52c0b61a0 con 0x7fc52c0b5430 2024-03-24T01:26:04.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.655+0000 7fc521400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc52c0b5c30 0x7fc52c0b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:04.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.655+0000 7fc529e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc52c15e280 0x7fc52c06b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50076/0 (socket says 172.21.15.90:50076) 2024-03-24T01:26:04.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.655+0000 7fc529e00640 1 -- 172.21.15.90:0/3685526422 learned_addr learned my addr 172.21.15.90:0/3685526422 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:26:04.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.656+0000 7fc52a800640 1 --2- 172.21.15.90:0/3685526422 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc52c0b5430 0x7fc52c0ceb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:04.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.656+0000 7fc521400640 1 -- 172.21.15.90:0/3685526422 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc52c0b5430 msgr2=0x7fc52c0ceb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:04.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.656+0000 7fc529400640 1 -- 172.21.15.90:0/3685526422 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4067412083 0 0) 0x7fc52c0b25d0 con 0x7fc52c15e280 2024-03-24T01:26:04.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.656+0000 7fc521400640 1 --2- 172.21.15.90:0/3685526422 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc52c0b5430 0x7fc52c0ceb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:04.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.656+0000 7fc521400640 1 -- 172.21.15.90:0/3685526422 >> v1:172.21.15.90:6789/0 conn(0x7fc52c15e280 legacy=0x7fc52c06b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:04.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.656+0000 7fc521400640 1 -- 172.21.15.90:0/3685526422 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc52c0b28b0 con 0x7fc52c0b5c30 2024-03-24T01:26:04.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.656+0000 7fc52a800640 1 --2- 172.21.15.90:0/3685526422 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc52c0b5430 0x7fc52c0ceb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:04.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.657+0000 7fc521400640 1 --2- 172.21.15.90:0/3685526422 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc52c0b5c30 0x7fc52c0b4ef0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fc5140097a0 tx=0x7fc51402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cc881f36c779740 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:04.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.657+0000 7fc529400640 1 -- 172.21.15.90:0/3685526422 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc514031780 con 0x7fc52c0b5c30 2024-03-24T01:26:04.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.657+0000 7fc529400640 1 -- 172.21.15.90:0/3685526422 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc514037070 con 0x7fc52c0b5c30 2024-03-24T01:26:04.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.657+0000 7fc529400640 1 -- 172.21.15.90:0/3685526422 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc514031b40 con 0x7fc52c0b5c30 2024-03-24T01:26:04.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.658+0000 7fc531000640 1 -- 172.21.15.90:0/3685526422 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc52c0b5c30 msgr2=0x7fc52c0b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:04.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.658+0000 7fc531000640 1 --2- 172.21.15.90:0/3685526422 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc52c0b5c30 0x7fc52c0b4ef0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fc5140097a0 tx=0x7fc51402b820 comp rx=0 tx=0).stop 2024-03-24T01:26:04.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.658+0000 7fc531000640 1 -- 172.21.15.90:0/3685526422 shutdown_connections 2024-03-24T01:26:04.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.658+0000 7fc531000640 1 --2- 172.21.15.90:0/3685526422 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc52c0b5430 0x7fc52c0ceb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:04.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.658+0000 7fc531000640 1 --2- 172.21.15.90:0/3685526422 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc52c0b5c30 0x7fc52c0b4ef0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:04.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.658+0000 7fc531000640 1 -- 172.21.15.90:0/3685526422 >> 172.21.15.90:0/3685526422 conn(0x7fc52c099a70 msgr2=0x7fc52c0b1d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:04.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.658+0000 7fc531000640 1 -- 172.21.15.90:0/3685526422 shutdown_connections 2024-03-24T01:26:04.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.658+0000 7fc531000640 1 -- 172.21.15.90:0/3685526422 wait complete. 2024-03-24T01:26:04.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.661+0000 7fc531000640 1 Processor -- start 2024-03-24T01:26:04.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.661+0000 7fc531000640 1 -- start start 2024-03-24T01:26:04.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.662+0000 7fc531000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc52c0b5430 0x7fc52c152c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:04.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.662+0000 7fc531000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc52c0b5c30 0x7fc52c144b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:04.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.662+0000 7fc531000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc52c15e280 0x7fc52c0ae660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:04.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.662+0000 7fc531000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc52c066f90 con 0x7fc52c0b5c30 2024-03-24T01:26:04.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.662+0000 7fc531000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc52c0ce6d0 con 0x7fc52c0b5430 2024-03-24T01:26:04.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.662+0000 7fc531000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc52c067110 con 0x7fc52c15e280 2024-03-24T01:26:04.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.663+0000 7fc521400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc52c0b5c30 0x7fc52c144b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:04.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.663+0000 7fc521400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc52c0b5c30 0x7fc52c144b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60038/0 (socket says 172.21.15.90:60038) 2024-03-24T01:26:04.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.663+0000 7fc521400640 1 -- 172.21.15.90:0/519603980 learned_addr learned my addr 172.21.15.90:0/519603980 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:04.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.663+0000 7fc521400640 1 -- 172.21.15.90:0/519603980 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc52c15e280 msgr2=0x7fc52c0ae660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:04.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.663+0000 7fc521400640 1 --2- 172.21.15.90:0/519603980 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc52c15e280 0x7fc52c0ae660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:04.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.663+0000 7fc521400640 1 -- 172.21.15.90:0/519603980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc52c0b5430 msgr2=0x7fc52c152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:04.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.663+0000 7fc529e00640 1 --2- 172.21.15.90:0/519603980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc52c0b5430 0x7fc52c152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:04.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.663+0000 7fc521400640 1 --2- 172.21.15.90:0/519603980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc52c0b5430 0x7fc52c152c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:04.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.663+0000 7fc521400640 1 -- 172.21.15.90:0/519603980 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc52c154070 con 0x7fc52c0b5c30 2024-03-24T01:26:04.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.664+0000 7fc521400640 1 --2- 172.21.15.90:0/519603980 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc52c0b5c30 0x7fc52c144b30 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fc514034040 tx=0x7fc5140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:04.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.664+0000 7fc522a00640 1 -- 172.21.15.90:0/519603980 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc514032820 con 0x7fc52c0b5c30 2024-03-24T01:26:04.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.664+0000 7fc522a00640 1 -- 172.21.15.90:0/519603980 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5140329c0 con 0x7fc52c0b5c30 2024-03-24T01:26:04.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.664+0000 7fc531000640 1 -- 172.21.15.90:0/519603980 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc52c1509f0 con 0x7fc52c0b5c30 2024-03-24T01:26:04.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.664+0000 7fc522a00640 1 -- 172.21.15.90:0/519603980 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc514032db0 con 0x7fc52c0b5c30 2024-03-24T01:26:04.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.664+0000 7fc531000640 1 -- 172.21.15.90:0/519603980 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc52c150fb0 con 0x7fc52c0b5c30 2024-03-24T01:26:04.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.665+0000 7fc531000640 1 -- 172.21.15.90:0/519603980 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc52c09e3f0 con 0x7fc52c0b5c30 2024-03-24T01:26:04.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.666+0000 7fc522a00640 1 -- 172.21.15.90:0/519603980 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc51404e440 con 0x7fc52c0b5c30 2024-03-24T01:26:04.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.667+0000 7fc522a00640 1 --2- 172.21.15.90:0/519603980 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc50806bc00 0x7fc50806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:04.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.668+0000 7fc522a00640 1 -- 172.21.15.90:0/519603980 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc5140acaf0 con 0x7fc52c0b5c30 2024-03-24T01:26:04.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.668+0000 7fc529e00640 1 --2- 172.21.15.90:0/519603980 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc50806bc00 0x7fc50806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:04.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.669+0000 7fc529e00640 1 --2- 172.21.15.90:0/519603980 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc50806bc00 0x7fc50806e0c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc51c0012e0 tx=0x7fc51c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:04.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.672+0000 7fc522a00640 1 -- 172.21.15.90:0/519603980 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc514040070 con 0x7fc52c0b5c30 2024-03-24T01:26:04.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.864+0000 7fc531000640 1 -- 172.21.15.90:0/519603980 --> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc52c151180 con 0x7fc50806bc00 2024-03-24T01:26:04.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.882+0000 7fc522a00640 1 -- 172.21.15.90:0/519603980 <== mgr.4104 v2:172.21.15.120:6832/860392634 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246634 (secure 0 0 0) 0x7fc52c151180 con 0x7fc50806bc00 2024-03-24T01:26:04.886 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:26:04.887 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-24T01:26:04.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.887+0000 7fc531000640 1 -- 172.21.15.90:0/519603980 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc50806bc00 msgr2=0x7fc50806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:04.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.887+0000 7fc531000640 1 --2- 172.21.15.90:0/519603980 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc50806bc00 0x7fc50806e0c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc51c0012e0 tx=0x7fc51c009040 comp rx=0 tx=0).stop 2024-03-24T01:26:04.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.887+0000 7fc531000640 1 -- 172.21.15.90:0/519603980 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc52c0b5c30 msgr2=0x7fc52c144b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:04.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.887+0000 7fc531000640 1 --2- 172.21.15.90:0/519603980 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc52c0b5c30 0x7fc52c144b30 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fc514034040 tx=0x7fc5140046c0 comp rx=0 tx=0).stop 2024-03-24T01:26:04.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.887+0000 7fc531000640 1 -- 172.21.15.90:0/519603980 shutdown_connections 2024-03-24T01:26:04.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.887+0000 7fc531000640 1 --2- 172.21.15.90:0/519603980 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc50806bc00 0x7fc50806e0c0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:04.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.887+0000 7fc531000640 1 --2- 172.21.15.90:0/519603980 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc52c0b5c30 0x7fc52c144b30 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:04.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.887+0000 7fc531000640 1 --2- 172.21.15.90:0/519603980 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc52c15e280 0x7fc52c0ae660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:04.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.887+0000 7fc531000640 1 --2- 172.21.15.90:0/519603980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc52c0b5430 0x7fc52c152c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:04.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.887+0000 7fc531000640 1 -- 172.21.15.90:0/519603980 >> 172.21.15.90:0/519603980 conn(0x7fc52c099a70 msgr2=0x7fc52c06e480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:04.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.887+0000 7fc531000640 1 -- 172.21.15.90:0/519603980 shutdown_connections 2024-03-24T01:26:04.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:04.887+0000 7fc531000640 1 -- 172.21.15.90:0/519603980 wait complete. 2024-03-24T01:26:04.904 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":94,"stamp":"2024-03-24T01:26:03.521564+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220600,"kb_used_data":4428,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754120,"statfs":{"total":773094113280,"available":772868218880,"internally_reserved":0,"allocated":4534272,"data_stored":2615145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12747,"internal_metadata":221236789},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.573169"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.977282+0000","last_change":"2024-03-24T01:25:58.977282+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.977282+0000","last_undegraded":"2024-03-24T01:25:58.977282+0000","last_fullsized":"2024-03-24T01:25:58.977282+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.975953+0000","last_change":"2024-03-24T01:25:58.975953+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.975953+0000","last_undegraded":"2024-03-24T01:25:58.975953+0000","last_fullsized":"2024-03-24T01:25:58.975953+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.979446+0000","last_change":"2024-03-24T01:25:58.979446+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.979446+0000","last_undegraded":"2024-03-24T01:25:58.979446+0000","last_fullsized":"2024-03-24T01:25:58.979446+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.974695+0000","last_change":"2024-03-24T01:25:58.974695+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.974695+0000","last_undegraded":"2024-03-24T01:25:58.974695+0000","last_fullsized":"2024-03-24T01:25:58.974695+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.973754+0000","last_change":"2024-03-24T01:25:58.973754+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.973754+0000","last_undegraded":"2024-03-24T01:25:58.973754+0000","last_fullsized":"2024-03-24T01:25:58.973754+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.986679+0000","last_change":"2024-03-24T01:25:58.986679+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.986679+0000","last_undegraded":"2024-03-24T01:25:58.986679+0000","last_fullsized":"2024-03-24T01:25:58.986679+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.989725+0000","last_change":"2024-03-24T01:25:58.989725+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.989725+0000","last_undegraded":"2024-03-24T01:25:58.989725+0000","last_fullsized":"2024-03-24T01:25:58.989725+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.978405+0000","last_change":"2024-03-24T01:25:58.978405+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.978405+0000","last_undegraded":"2024-03-24T01:25:58.978405+0000","last_fullsized":"2024-03-24T01:25:58.978405+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.978353+0000","last_change":"2024-03-24T01:25:58.978353+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.978353+0000","last_undegraded":"2024-03-24T01:25:58.978353+0000","last_fullsized":"2024-03-24T01:25:58.978353+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.995242+0000","last_change":"2024-03-24T01:25:58.995242+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.995242+0000","last_undegraded":"2024-03-24T01:25:58.995242+0000","last_fullsized":"2024-03-24T01:25:58.995242+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.990081+0000","last_change":"2024-03-24T01:25:58.990081+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.990081+0000","last_undegraded":"2024-03-24T01:25:58.990081+0000","last_fullsized":"2024-03-24T01:25:58.990081+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.976660+0000","last_change":"2024-03-24T01:25:58.976660+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.976660+0000","last_undegraded":"2024-03-24T01:25:58.976660+0000","last_fullsized":"2024-03-24T01:25:58.976660+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.986804+0000","last_change":"2024-03-24T01:25:58.986804+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.986804+0000","last_undegraded":"2024-03-24T01:25:58.986804+0000","last_fullsized":"2024-03-24T01:25:58.986804+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.980568+0000","last_change":"2024-03-24T01:25:58.980568+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.980568+0000","last_undegraded":"2024-03-24T01:25:58.980568+0000","last_fullsized":"2024-03-24T01:25:58.980568+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.986804+0000","last_change":"2024-03-24T01:25:58.986804+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.986804+0000","last_undegraded":"2024-03-24T01:25:58.986804+0000","last_fullsized":"2024-03-24T01:25:58.986804+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.993159+0000","last_change":"2024-03-24T01:25:58.993159+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.993159+0000","last_undegraded":"2024-03-24T01:25:58.993159+0000","last_fullsized":"2024-03-24T01:25:58.993159+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.991992+0000","last_change":"2024-03-24T01:25:58.991992+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.991992+0000","last_undegraded":"2024-03-24T01:25:58.991992+0000","last_fullsized":"2024-03-24T01:25:58.991992+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.974166+0000","last_change":"2024-03-24T01:25:58.974166+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.974166+0000","last_undegraded":"2024-03-24T01:25:58.974166+0000","last_fullsized":"2024-03-24T01:25:58.974166+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.973263+0000","last_change":"2024-03-24T01:25:58.973263+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.973263+0000","last_undegraded":"2024-03-24T01:25:58.973263+0000","last_fullsized":"2024-03-24T01:25:58.973263+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.974422+0000","last_change":"2024-03-24T01:25:58.974422+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.974422+0000","last_undegraded":"2024-03-24T01:25:58.974422+0000","last_fullsized":"2024-03-24T01:25:58.974422+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.986049+0000","last_change":"2024-03-24T01:25:58.986049+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.986049+0000","last_undegraded":"2024-03-24T01:25:58.986049+0000","last_fullsized":"2024-03-24T01:25:58.986049+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.977959+0000","last_change":"2024-03-24T01:25:58.977959+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.977959+0000","last_undegraded":"2024-03-24T01:25:58.977959+0000","last_fullsized":"2024-03-24T01:25:58.977959+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.989831+0000","last_change":"2024-03-24T01:25:58.989831+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.989831+0000","last_undegraded":"2024-03-24T01:25:58.989831+0000","last_fullsized":"2024-03-24T01:25:58.989831+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.972222+0000","last_change":"2024-03-24T01:25:58.972222+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.972222+0000","last_undegraded":"2024-03-24T01:25:58.972222+0000","last_fullsized":"2024-03-24T01:25:58.972222+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.975433+0000","last_change":"2024-03-24T01:25:58.975433+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.975433+0000","last_undegraded":"2024-03-24T01:25:58.975433+0000","last_fullsized":"2024-03-24T01:25:58.975433+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.985886+0000","last_change":"2024-03-24T01:25:58.985886+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.985886+0000","last_undegraded":"2024-03-24T01:25:58.985886+0000","last_fullsized":"2024-03-24T01:25:58.985886+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.977178+0000","last_change":"2024-03-24T01:25:58.977178+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.977178+0000","last_undegraded":"2024-03-24T01:25:58.977178+0000","last_fullsized":"2024-03-24T01:25:58.977178+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.986473+0000","last_change":"2024-03-24T01:25:58.986473+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.986473+0000","last_undegraded":"2024-03-24T01:25:58.986473+0000","last_fullsized":"2024-03-24T01:25:58.986473+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.975161+0000","last_change":"2024-03-24T01:25:58.975161+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.975161+0000","last_undegraded":"2024-03-24T01:25:58.975161+0000","last_fullsized":"2024-03-24T01:25:58.975161+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.983076+0000","last_change":"2024-03-24T01:25:58.983076+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.983076+0000","last_undegraded":"2024-03-24T01:25:58.983076+0000","last_fullsized":"2024-03-24T01:25:58.983076+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.989181+0000","last_change":"2024-03-24T01:25:58.989181+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.989181+0000","last_undegraded":"2024-03-24T01:25:58.989181+0000","last_fullsized":"2024-03-24T01:25:58.989181+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.981936+0000","last_change":"2024-03-24T01:25:58.981936+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.981936+0000","last_undegraded":"2024-03-24T01:25:58.981936+0000","last_fullsized":"2024-03-24T01:25:58.981936+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.994459+0000","last_change":"2024-03-24T01:25:58.994459+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.994459+0000","last_undegraded":"2024-03-24T01:25:58.994459+0000","last_fullsized":"2024-03-24T01:25:58.994459+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.001458+0000","last_change":"2024-03-24T01:25:58.996685+0000","last_active":"2024-03-24T01:25:59.001458+0000","last_peered":"2024-03-24T01:25:59.001458+0000","last_clean":"2024-03-24T01:25:59.001458+0000","last_became_active":"2024-03-24T01:25:58.996398+0000","last_became_peered":"2024-03-24T01:25:58.996398+0000","last_unstale":"2024-03-24T01:25:59.001458+0000","last_undegraded":"2024-03-24T01:25:59.001458+0000","last_fullsized":"2024-03-24T01:25:59.001458+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:53:49.271423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.984299+0000","last_change":"2024-03-24T01:25:58.984299+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.984299+0000","last_undegraded":"2024-03-24T01:25:58.984299+0000","last_fullsized":"2024-03-24T01:25:58.984299+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.004893+0000","last_change":"2024-03-24T01:25:59.002453+0000","last_active":"2024-03-24T01:25:59.004893+0000","last_peered":"2024-03-24T01:25:59.004893+0000","last_clean":"2024-03-24T01:25:59.004893+0000","last_became_active":"2024-03-24T01:25:59.002155+0000","last_became_peered":"2024-03-24T01:25:59.002155+0000","last_unstale":"2024-03-24T01:25:59.004893+0000","last_undegraded":"2024-03-24T01:25:59.004893+0000","last_fullsized":"2024-03-24T01:25:59.004893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:53:49.271423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.978899+0000","last_change":"2024-03-24T01:25:58.978899+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.978899+0000","last_undegraded":"2024-03-24T01:25:58.978899+0000","last_fullsized":"2024-03-24T01:25:58.978899+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.998379+0000","last_change":"2024-03-24T01:25:58.997334+0000","last_active":"2024-03-24T01:25:58.998379+0000","last_peered":"2024-03-24T01:25:58.998379+0000","last_clean":"2024-03-24T01:25:58.998379+0000","last_became_active":"2024-03-24T01:25:58.997033+0000","last_became_peered":"2024-03-24T01:25:58.997033+0000","last_unstale":"2024-03-24T01:25:58.998379+0000","last_undegraded":"2024-03-24T01:25:58.998379+0000","last_fullsized":"2024-03-24T01:25:58.998379+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:45.605005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.977778+0000","last_change":"2024-03-24T01:25:58.977778+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.977778+0000","last_undegraded":"2024-03-24T01:25:58.977778+0000","last_fullsized":"2024-03-24T01:25:58.977778+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.996426+0000","last_change":"2024-03-24T01:25:58.992317+0000","last_active":"2024-03-24T01:25:58.996426+0000","last_peered":"2024-03-24T01:25:58.996426+0000","last_clean":"2024-03-24T01:25:58.996426+0000","last_became_active":"2024-03-24T01:25:58.991652+0000","last_became_peered":"2024-03-24T01:25:58.991652+0000","last_unstale":"2024-03-24T01:25:58.996426+0000","last_undegraded":"2024-03-24T01:25:58.996426+0000","last_fullsized":"2024-03-24T01:25:58.996426+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:53:49.271423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.982673+0000","last_change":"2024-03-24T01:25:58.982673+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.982673+0000","last_undegraded":"2024-03-24T01:25:58.982673+0000","last_fullsized":"2024-03-24T01:25:58.982673+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.004193+0000","last_change":"2024-03-24T01:25:58.998337+0000","last_active":"2024-03-24T01:25:59.004193+0000","last_peered":"2024-03-24T01:25:59.004193+0000","last_clean":"2024-03-24T01:25:59.004193+0000","last_became_active":"2024-03-24T01:25:58.998056+0000","last_became_peered":"2024-03-24T01:25:58.998056+0000","last_unstale":"2024-03-24T01:25:59.004193+0000","last_undegraded":"2024-03-24T01:25:59.004193+0000","last_fullsized":"2024-03-24T01:25:59.004193+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:49:24.128591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.985591+0000","last_change":"2024-03-24T01:25:58.985591+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.985591+0000","last_undegraded":"2024-03-24T01:25:58.985591+0000","last_fullsized":"2024-03-24T01:25:58.985591+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.012523+0000","last_change":"2024-03-24T01:25:59.006913+0000","last_active":"2024-03-24T01:25:59.012523+0000","last_peered":"2024-03-24T01:25:59.012523+0000","last_clean":"2024-03-24T01:25:59.012523+0000","last_became_active":"2024-03-24T01:25:59.006098+0000","last_became_peered":"2024-03-24T01:25:59.006098+0000","last_unstale":"2024-03-24T01:25:59.012523+0000","last_undegraded":"2024-03-24T01:25:59.012523+0000","last_fullsized":"2024-03-24T01:25:59.012523+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:49:24.128591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.976758+0000","last_change":"2024-03-24T01:25:58.976758+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.976758+0000","last_undegraded":"2024-03-24T01:25:58.976758+0000","last_fullsized":"2024-03-24T01:25:58.976758+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.004687+0000","last_change":"2024-03-24T01:25:59.000318+0000","last_active":"2024-03-24T01:25:59.004687+0000","last_peered":"2024-03-24T01:25:59.004687+0000","last_clean":"2024-03-24T01:25:59.004687+0000","last_became_active":"2024-03-24T01:25:59.000085+0000","last_became_peered":"2024-03-24T01:25:59.000085+0000","last_unstale":"2024-03-24T01:25:59.004687+0000","last_undegraded":"2024-03-24T01:25:59.004687+0000","last_fullsized":"2024-03-24T01:25:59.004687+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:28:18.909603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.980649+0000","last_change":"2024-03-24T01:25:58.980649+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.980649+0000","last_undegraded":"2024-03-24T01:25:58.980649+0000","last_fullsized":"2024-03-24T01:25:58.980649+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.999166+0000","last_change":"2024-03-24T01:25:58.994521+0000","last_active":"2024-03-24T01:25:58.999166+0000","last_peered":"2024-03-24T01:25:58.999166+0000","last_clean":"2024-03-24T01:25:58.999166+0000","last_became_active":"2024-03-24T01:25:58.994215+0000","last_became_peered":"2024-03-24T01:25:58.994215+0000","last_unstale":"2024-03-24T01:25:58.999166+0000","last_undegraded":"2024-03-24T01:25:58.999166+0000","last_fullsized":"2024-03-24T01:25:58.999166+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:45.605005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.972522+0000","last_change":"2024-03-24T01:25:58.972522+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.972522+0000","last_undegraded":"2024-03-24T01:25:58.972522+0000","last_fullsized":"2024-03-24T01:25:58.972522+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.996515+0000","last_change":"2024-03-24T01:25:58.995322+0000","last_active":"2024-03-24T01:25:58.996515+0000","last_peered":"2024-03-24T01:25:58.996515+0000","last_clean":"2024-03-24T01:25:58.996515+0000","last_became_active":"2024-03-24T01:25:58.994847+0000","last_became_peered":"2024-03-24T01:25:58.994847+0000","last_unstale":"2024-03-24T01:25:58.996515+0000","last_undegraded":"2024-03-24T01:25:58.996515+0000","last_fullsized":"2024-03-24T01:25:58.996515+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:12:43.140922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.995109+0000","last_change":"2024-03-24T01:25:58.995109+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.995109+0000","last_undegraded":"2024-03-24T01:25:58.995109+0000","last_fullsized":"2024-03-24T01:25:58.995109+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.004224+0000","last_change":"2024-03-24T01:25:58.998109+0000","last_active":"2024-03-24T01:25:59.004224+0000","last_peered":"2024-03-24T01:25:59.004224+0000","last_clean":"2024-03-24T01:25:59.004224+0000","last_became_active":"2024-03-24T01:25:58.997791+0000","last_became_peered":"2024-03-24T01:25:58.997791+0000","last_unstale":"2024-03-24T01:25:59.004224+0000","last_undegraded":"2024-03-24T01:25:59.004224+0000","last_fullsized":"2024-03-24T01:25:59.004224+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:00:42.635663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.976014+0000","last_change":"2024-03-24T01:25:58.976014+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.976014+0000","last_undegraded":"2024-03-24T01:25:58.976014+0000","last_fullsized":"2024-03-24T01:25:58.976014+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.004257+0000","last_change":"2024-03-24T01:25:58.999040+0000","last_active":"2024-03-24T01:25:59.004257+0000","last_peered":"2024-03-24T01:25:59.004257+0000","last_clean":"2024-03-24T01:25:59.004257+0000","last_became_active":"2024-03-24T01:25:58.998593+0000","last_became_peered":"2024-03-24T01:25:58.998593+0000","last_unstale":"2024-03-24T01:25:59.004257+0000","last_undegraded":"2024-03-24T01:25:59.004257+0000","last_fullsized":"2024-03-24T01:25:59.004257+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:53:49.271423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.978116+0000","last_change":"2024-03-24T01:25:58.978116+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.978116+0000","last_undegraded":"2024-03-24T01:25:58.978116+0000","last_fullsized":"2024-03-24T01:25:58.978116+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.007336+0000","last_change":"2024-03-24T01:25:59.002378+0000","last_active":"2024-03-24T01:25:59.007336+0000","last_peered":"2024-03-24T01:25:59.007336+0000","last_clean":"2024-03-24T01:25:59.007336+0000","last_became_active":"2024-03-24T01:25:59.002071+0000","last_became_peered":"2024-03-24T01:25:59.002071+0000","last_unstale":"2024-03-24T01:25:59.007336+0000","last_undegraded":"2024-03-24T01:25:59.007336+0000","last_fullsized":"2024-03-24T01:25:59.007336+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:05:31.709112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.980063+0000","last_change":"2024-03-24T01:25:58.980063+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.980063+0000","last_undegraded":"2024-03-24T01:25:58.980063+0000","last_fullsized":"2024-03-24T01:25:58.980063+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.990060+0000","last_change":"2024-03-24T01:25:58.987270+0000","last_active":"2024-03-24T01:25:58.990060+0000","last_peered":"2024-03-24T01:25:58.990060+0000","last_clean":"2024-03-24T01:25:58.990060+0000","last_became_active":"2024-03-24T01:25:58.986953+0000","last_became_peered":"2024-03-24T01:25:58.986953+0000","last_unstale":"2024-03-24T01:25:58.990060+0000","last_undegraded":"2024-03-24T01:25:58.990060+0000","last_fullsized":"2024-03-24T01:25:58.990060+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:19:04.197281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.981124+0000","last_change":"2024-03-24T01:25:58.981124+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.981124+0000","last_undegraded":"2024-03-24T01:25:58.981124+0000","last_fullsized":"2024-03-24T01:25:58.981124+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.004607+0000","last_change":"2024-03-24T01:25:59.000275+0000","last_active":"2024-03-24T01:25:59.004607+0000","last_peered":"2024-03-24T01:25:59.004607+0000","last_clean":"2024-03-24T01:25:59.004607+0000","last_became_active":"2024-03-24T01:25:58.999948+0000","last_became_peered":"2024-03-24T01:25:58.999948+0000","last_unstale":"2024-03-24T01:25:59.004607+0000","last_undegraded":"2024-03-24T01:25:59.004607+0000","last_fullsized":"2024-03-24T01:25:59.004607+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:44:12.069166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.987147+0000","last_change":"2024-03-24T01:25:58.987147+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.987147+0000","last_undegraded":"2024-03-24T01:25:58.987147+0000","last_fullsized":"2024-03-24T01:25:58.987147+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.995109+0000","last_change":"2024-03-24T01:25:58.991563+0000","last_active":"2024-03-24T01:25:58.995109+0000","last_peered":"2024-03-24T01:25:58.995109+0000","last_clean":"2024-03-24T01:25:58.995109+0000","last_became_active":"2024-03-24T01:25:58.990946+0000","last_became_peered":"2024-03-24T01:25:58.990946+0000","last_unstale":"2024-03-24T01:25:58.995109+0000","last_undegraded":"2024-03-24T01:25:58.995109+0000","last_fullsized":"2024-03-24T01:25:58.995109+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:49:24.128591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.980154+0000","last_change":"2024-03-24T01:25:58.980154+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.980154+0000","last_undegraded":"2024-03-24T01:25:58.980154+0000","last_fullsized":"2024-03-24T01:25:58.980154+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.009845+0000","last_change":"2024-03-24T01:25:59.006569+0000","last_active":"2024-03-24T01:25:59.009845+0000","last_peered":"2024-03-24T01:25:59.009845+0000","last_clean":"2024-03-24T01:25:59.009845+0000","last_became_active":"2024-03-24T01:25:59.006280+0000","last_became_peered":"2024-03-24T01:25:59.006280+0000","last_unstale":"2024-03-24T01:25:59.009845+0000","last_undegraded":"2024-03-24T01:25:59.009845+0000","last_fullsized":"2024-03-24T01:25:59.009845+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:00:42.635663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.976902+0000","last_change":"2024-03-24T01:25:58.976902+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.976902+0000","last_undegraded":"2024-03-24T01:25:58.976902+0000","last_fullsized":"2024-03-24T01:25:58.976902+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.005436+0000","last_change":"2024-03-24T01:25:59.003724+0000","last_active":"2024-03-24T01:25:59.005436+0000","last_peered":"2024-03-24T01:25:59.005436+0000","last_clean":"2024-03-24T01:25:59.005436+0000","last_became_active":"2024-03-24T01:25:59.003410+0000","last_became_peered":"2024-03-24T01:25:59.003410+0000","last_unstale":"2024-03-24T01:25:59.005436+0000","last_undegraded":"2024-03-24T01:25:59.005436+0000","last_fullsized":"2024-03-24T01:25:59.005436+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:08:38.108175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.989181+0000","last_change":"2024-03-24T01:25:58.989181+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.989181+0000","last_undegraded":"2024-03-24T01:25:58.989181+0000","last_fullsized":"2024-03-24T01:25:58.989181+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.002870+0000","last_change":"2024-03-24T01:25:58.999764+0000","last_active":"2024-03-24T01:25:59.002870+0000","last_peered":"2024-03-24T01:25:59.002870+0000","last_clean":"2024-03-24T01:25:59.002870+0000","last_became_active":"2024-03-24T01:25:58.999490+0000","last_became_peered":"2024-03-24T01:25:58.999490+0000","last_unstale":"2024-03-24T01:25:59.002870+0000","last_undegraded":"2024-03-24T01:25:59.002870+0000","last_fullsized":"2024-03-24T01:25:59.002870+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:44:12.069166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.983702+0000","last_change":"2024-03-24T01:25:58.983702+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.983702+0000","last_undegraded":"2024-03-24T01:25:58.983702+0000","last_fullsized":"2024-03-24T01:25:58.983702+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.007469+0000","last_change":"2024-03-24T01:25:59.003238+0000","last_active":"2024-03-24T01:25:59.007469+0000","last_peered":"2024-03-24T01:25:59.007469+0000","last_clean":"2024-03-24T01:25:59.007469+0000","last_became_active":"2024-03-24T01:25:59.002905+0000","last_became_peered":"2024-03-24T01:25:59.002905+0000","last_unstale":"2024-03-24T01:25:59.007469+0000","last_undegraded":"2024-03-24T01:25:59.007469+0000","last_fullsized":"2024-03-24T01:25:59.007469+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:56:29.523110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.972343+0000","last_change":"2024-03-24T01:25:58.972343+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.972343+0000","last_undegraded":"2024-03-24T01:25:58.972343+0000","last_fullsized":"2024-03-24T01:25:58.972343+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.996158+0000","last_change":"2024-03-24T01:25:58.994043+0000","last_active":"2024-03-24T01:25:58.996158+0000","last_peered":"2024-03-24T01:25:58.996158+0000","last_clean":"2024-03-24T01:25:58.996158+0000","last_became_active":"2024-03-24T01:25:58.993746+0000","last_became_peered":"2024-03-24T01:25:58.993746+0000","last_unstale":"2024-03-24T01:25:58.996158+0000","last_undegraded":"2024-03-24T01:25:58.996158+0000","last_fullsized":"2024-03-24T01:25:58.996158+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:08:38.108175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.987330+0000","last_change":"2024-03-24T01:25:58.987330+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.987330+0000","last_undegraded":"2024-03-24T01:25:58.987330+0000","last_fullsized":"2024-03-24T01:25:58.987330+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.991383+0000","last_change":"2024-03-24T01:25:58.988228+0000","last_active":"2024-03-24T01:25:58.991383+0000","last_peered":"2024-03-24T01:25:58.991383+0000","last_clean":"2024-03-24T01:25:58.991383+0000","last_became_active":"2024-03-24T01:25:58.987941+0000","last_became_peered":"2024-03-24T01:25:58.987941+0000","last_unstale":"2024-03-24T01:25:58.991383+0000","last_undegraded":"2024-03-24T01:25:58.991383+0000","last_fullsized":"2024-03-24T01:25:58.991383+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:12:25.446123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.976174+0000","last_change":"2024-03-24T01:25:58.976174+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.976174+0000","last_undegraded":"2024-03-24T01:25:58.976174+0000","last_fullsized":"2024-03-24T01:25:58.976174+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.993352+0000","last_change":"2024-03-24T01:25:58.991840+0000","last_active":"2024-03-24T01:25:58.993352+0000","last_peered":"2024-03-24T01:25:58.993352+0000","last_clean":"2024-03-24T01:25:58.993352+0000","last_became_active":"2024-03-24T01:25:58.991522+0000","last_became_peered":"2024-03-24T01:25:58.991522+0000","last_unstale":"2024-03-24T01:25:58.993352+0000","last_undegraded":"2024-03-24T01:25:58.993352+0000","last_fullsized":"2024-03-24T01:25:58.993352+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:03:47.985379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.985984+0000","last_change":"2024-03-24T01:25:58.985984+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.985984+0000","last_undegraded":"2024-03-24T01:25:58.985984+0000","last_fullsized":"2024-03-24T01:25:58.985984+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.009786+0000","last_change":"2024-03-24T01:25:59.006226+0000","last_active":"2024-03-24T01:25:59.009786+0000","last_peered":"2024-03-24T01:25:59.009786+0000","last_clean":"2024-03-24T01:25:59.009786+0000","last_became_active":"2024-03-24T01:25:59.005739+0000","last_became_peered":"2024-03-24T01:25:59.005739+0000","last_unstale":"2024-03-24T01:25:59.009786+0000","last_undegraded":"2024-03-24T01:25:59.009786+0000","last_fullsized":"2024-03-24T01:25:59.009786+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:07:08.132339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.976537+0000","last_change":"2024-03-24T01:25:58.976537+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.976537+0000","last_undegraded":"2024-03-24T01:25:58.976537+0000","last_fullsized":"2024-03-24T01:25:58.976537+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.991351+0000","last_change":"2024-03-24T01:25:58.988520+0000","last_active":"2024-03-24T01:25:58.991351+0000","last_peered":"2024-03-24T01:25:58.991351+0000","last_clean":"2024-03-24T01:25:58.991351+0000","last_became_active":"2024-03-24T01:25:58.988149+0000","last_became_peered":"2024-03-24T01:25:58.988149+0000","last_unstale":"2024-03-24T01:25:58.991351+0000","last_undegraded":"2024-03-24T01:25:58.991351+0000","last_fullsized":"2024-03-24T01:25:58.991351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:13:07.896803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.983029+0000","last_change":"2024-03-24T01:25:58.983029+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.983029+0000","last_undegraded":"2024-03-24T01:25:58.983029+0000","last_fullsized":"2024-03-24T01:25:58.983029+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.009953+0000","last_change":"2024-03-24T01:25:59.005278+0000","last_active":"2024-03-24T01:25:59.009953+0000","last_peered":"2024-03-24T01:25:59.009953+0000","last_clean":"2024-03-24T01:25:59.009953+0000","last_became_active":"2024-03-24T01:25:59.004896+0000","last_became_peered":"2024-03-24T01:25:59.004896+0000","last_unstale":"2024-03-24T01:25:59.009953+0000","last_undegraded":"2024-03-24T01:25:59.009953+0000","last_fullsized":"2024-03-24T01:25:59.009953+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:45.605005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.989993+0000","last_change":"2024-03-24T01:25:58.989993+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.989993+0000","last_undegraded":"2024-03-24T01:25:58.989993+0000","last_fullsized":"2024-03-24T01:25:58.989993+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.007491+0000","last_change":"2024-03-24T01:25:59.004018+0000","last_active":"2024-03-24T01:25:59.007491+0000","last_peered":"2024-03-24T01:25:59.007491+0000","last_clean":"2024-03-24T01:25:59.007491+0000","last_became_active":"2024-03-24T01:25:59.003661+0000","last_became_peered":"2024-03-24T01:25:59.003661+0000","last_unstale":"2024-03-24T01:25:59.007491+0000","last_undegraded":"2024-03-24T01:25:59.007491+0000","last_fullsized":"2024-03-24T01:25:59.007491+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:35:02.448771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.975406+0000","last_change":"2024-03-24T01:25:58.975406+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.975406+0000","last_undegraded":"2024-03-24T01:25:58.975406+0000","last_fullsized":"2024-03-24T01:25:58.975406+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.998477+0000","last_change":"2024-03-24T01:25:58.995201+0000","last_active":"2024-03-24T01:25:58.998477+0000","last_peered":"2024-03-24T01:25:58.998477+0000","last_clean":"2024-03-24T01:25:58.998477+0000","last_became_active":"2024-03-24T01:25:58.994890+0000","last_became_peered":"2024-03-24T01:25:58.994890+0000","last_unstale":"2024-03-24T01:25:58.998477+0000","last_undegraded":"2024-03-24T01:25:58.998477+0000","last_fullsized":"2024-03-24T01:25:58.998477+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:49:24.128591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.978947+0000","last_change":"2024-03-24T01:25:58.978947+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.978947+0000","last_undegraded":"2024-03-24T01:25:58.978947+0000","last_fullsized":"2024-03-24T01:25:58.978947+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.993897+0000","last_change":"2024-03-24T01:25:58.990591+0000","last_active":"2024-03-24T01:25:58.993897+0000","last_peered":"2024-03-24T01:25:58.993897+0000","last_clean":"2024-03-24T01:25:58.993897+0000","last_became_active":"2024-03-24T01:25:58.990269+0000","last_became_peered":"2024-03-24T01:25:58.990269+0000","last_unstale":"2024-03-24T01:25:58.993897+0000","last_undegraded":"2024-03-24T01:25:58.993897+0000","last_fullsized":"2024-03-24T01:25:58.993897+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:00:42.635663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.979426+0000","last_change":"2024-03-24T01:25:58.979426+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.979426+0000","last_undegraded":"2024-03-24T01:25:58.979426+0000","last_fullsized":"2024-03-24T01:25:58.979426+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.993563+0000","last_change":"2024-03-24T01:25:58.988526+0000","last_active":"2024-03-24T01:25:58.993563+0000","last_peered":"2024-03-24T01:25:58.993563+0000","last_clean":"2024-03-24T01:25:58.993563+0000","last_became_active":"2024-03-24T01:25:58.987932+0000","last_became_peered":"2024-03-24T01:25:58.987932+0000","last_unstale":"2024-03-24T01:25:58.993563+0000","last_undegraded":"2024-03-24T01:25:58.993563+0000","last_fullsized":"2024-03-24T01:25:58.993563+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:00:42.635663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.979324+0000","last_change":"2024-03-24T01:25:58.979324+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.979324+0000","last_undegraded":"2024-03-24T01:25:58.979324+0000","last_fullsized":"2024-03-24T01:25:58.979324+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.995934+0000","last_change":"2024-03-24T01:25:58.992830+0000","last_active":"2024-03-24T01:25:58.995934+0000","last_peered":"2024-03-24T01:25:58.995934+0000","last_clean":"2024-03-24T01:25:58.995934+0000","last_became_active":"2024-03-24T01:25:58.991569+0000","last_became_peered":"2024-03-24T01:25:58.991569+0000","last_unstale":"2024-03-24T01:25:58.995934+0000","last_undegraded":"2024-03-24T01:25:58.995934+0000","last_fullsized":"2024-03-24T01:25:58.995934+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:08:38.108175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.987080+0000","last_change":"2024-03-24T01:25:58.987080+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.987080+0000","last_undegraded":"2024-03-24T01:25:58.987080+0000","last_fullsized":"2024-03-24T01:25:58.987080+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.997322+0000","last_change":"2024-03-24T01:25:58.996128+0000","last_active":"2024-03-24T01:25:58.997322+0000","last_peered":"2024-03-24T01:25:58.997322+0000","last_clean":"2024-03-24T01:25:58.997322+0000","last_became_active":"2024-03-24T01:25:58.994831+0000","last_became_peered":"2024-03-24T01:25:58.994831+0000","last_unstale":"2024-03-24T01:25:58.997322+0000","last_undegraded":"2024-03-24T01:25:58.997322+0000","last_fullsized":"2024-03-24T01:25:58.997322+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:46:49.838842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.991355+0000","last_change":"2024-03-24T01:25:58.991355+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.991355+0000","last_undegraded":"2024-03-24T01:25:58.991355+0000","last_fullsized":"2024-03-24T01:25:58.991355+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.012545+0000","last_change":"2024-03-24T01:25:59.008252+0000","last_active":"2024-03-24T01:25:59.012545+0000","last_peered":"2024-03-24T01:25:59.012545+0000","last_clean":"2024-03-24T01:25:59.012545+0000","last_became_active":"2024-03-24T01:25:59.007969+0000","last_became_peered":"2024-03-24T01:25:59.007969+0000","last_unstale":"2024-03-24T01:25:59.012545+0000","last_undegraded":"2024-03-24T01:25:59.012545+0000","last_fullsized":"2024-03-24T01:25:59.012545+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:53:49.271423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":200,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.999857+0000","last_change":"2024-03-24T01:24:01.545855+0000","last_active":"2024-03-24T01:25:58.999857+0000","last_peered":"2024-03-24T01:25:58.999857+0000","last_clean":"2024-03-24T01:25:58.999857+0000","last_became_active":"2024-03-24T01:24:01.543811+0000","last_became_peered":"2024-03-24T01:24:01.543811+0000","last_unstale":"2024-03-24T01:25:58.999857+0000","last_undegraded":"2024-03-24T01:25:58.999857+0000","last_fullsized":"2024-03-24T01:25:58.999857+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_clean_scrub_stamp":"2024-03-24T01:23:59.861564+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:03.682631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343836,"statfs":{"total":96636764160,"available":96608088064,"internally_reserved":0,"allocated":1007616,"data_stored":767169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574875,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":176801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574876,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":176801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574875,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":425984,"data_stored":183468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":176801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":425984,"data_stored":183468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":176801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343820,"statfs":{"total":96636764160,"available":96608071680,"internally_reserved":0,"allocated":1019904,"data_stored":773836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T01:26:04.908 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T01:26:05.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.139+0000 7f8f58a00640 1 Processor -- start 2024-03-24T01:26:05.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.139+0000 7f8f58a00640 1 -- start start 2024-03-24T01:26:05.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.141+0000 7f8f58a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8f5415df40 0x7f8f54161890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:05.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.141+0000 7f8f58a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8f540b94e0 0x7f8f540ce890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:05.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.141+0000 7f8f58a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f54161dd0 con 0x7f8f540cedd0 2024-03-24T01:26:05.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.141+0000 7f8f58a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f540d1c70 con 0x7f8f5415df40 2024-03-24T01:26:05.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.141+0000 7f8f58a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f541620a0 con 0x7f8f540b94e0 2024-03-24T01:26:05.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.141+0000 7f8f51e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8f5415df40 0x7f8f54161890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:05.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.141+0000 7f8f51e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8f5415df40 0x7f8f54161890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60044/0 (socket says 172.21.15.90:60044) 2024-03-24T01:26:05.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.141+0000 7f8f51e00640 1 -- 172.21.15.90:0/2038857878 learned_addr learned my addr 172.21.15.90:0/2038857878 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:05.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.141+0000 7f8f49400640 1 --2- 172.21.15.90:0/2038857878 >> v2:172.21.15.120:3300/0 conn(0x7f8f540b94e0 0x7f8f540ce890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:05.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.142+0000 7f8f51e00640 1 -- 172.21.15.90:0/2038857878 >> v2:172.21.15.120:3300/0 conn(0x7f8f540b94e0 msgr2=0x7f8f540ce890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:05.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.142+0000 7f8f51e00640 1 --2- 172.21.15.90:0/2038857878 >> v2:172.21.15.120:3300/0 conn(0x7f8f540b94e0 0x7f8f540ce890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:05.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.142+0000 7f8f51e00640 1 -- 172.21.15.90:0/2038857878 >> v1:172.21.15.90:6789/0 conn(0x7f8f540cedd0 legacy=0x7f8f540b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:05.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.142+0000 7f8f51e00640 1 -- 172.21.15.90:0/2038857878 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f54162380 con 0x7f8f5415df40 2024-03-24T01:26:05.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.142+0000 7f8f51400640 1 -- 172.21.15.90:0/2038857878 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 415732284 0 0) 0x7f8f54161dd0 con 0x7f8f540cedd0 2024-03-24T01:26:05.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.143+0000 7f8f51e00640 1 --2- 172.21.15.90:0/2038857878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f5415df40 0x7f8f54161890 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f8f5408cb30 tx=0x7f8f4402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b94e80fcbccb993 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:05.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.143+0000 7f8f51400640 1 -- 172.21.15.90:0/2038857878 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f44034070 con 0x7f8f5415df40 2024-03-24T01:26:05.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.143+0000 7f8f51400640 1 -- 172.21.15.90:0/2038857878 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f44030860 con 0x7f8f5415df40 2024-03-24T01:26:05.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.143+0000 7f8f51400640 1 -- 172.21.15.90:0/2038857878 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f44030c30 con 0x7f8f5415df40 2024-03-24T01:26:05.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.143+0000 7f8f58a00640 1 -- 172.21.15.90:0/2038857878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f5415df40 msgr2=0x7f8f54161890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:05.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.143+0000 7f8f58a00640 1 --2- 172.21.15.90:0/2038857878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f5415df40 0x7f8f54161890 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f8f5408cb30 tx=0x7f8f4402b9a0 comp rx=0 tx=0).stop 2024-03-24T01:26:05.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.144+0000 7f8f58a00640 1 -- 172.21.15.90:0/2038857878 shutdown_connections 2024-03-24T01:26:05.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.144+0000 7f8f58a00640 1 --2- 172.21.15.90:0/2038857878 >> v2:172.21.15.120:3300/0 conn(0x7f8f540b94e0 0x7f8f540ce890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:05.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.144+0000 7f8f58a00640 1 --2- 172.21.15.90:0/2038857878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f5415df40 0x7f8f54161890 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:05.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.144+0000 7f8f58a00640 1 -- 172.21.15.90:0/2038857878 >> 172.21.15.90:0/2038857878 conn(0x7f8f5406c9f0 msgr2=0x7f8f5406ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:05.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.144+0000 7f8f58a00640 1 -- 172.21.15.90:0/2038857878 shutdown_connections 2024-03-24T01:26:05.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.144+0000 7f8f58a00640 1 -- 172.21.15.90:0/2038857878 wait complete. 2024-03-24T01:26:05.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.145+0000 7f8f58a00640 1 Processor -- start 2024-03-24T01:26:05.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.145+0000 7f8f58a00640 1 -- start start 2024-03-24T01:26:05.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.146+0000 7f8f58a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f540b94e0 0x7f8f540b4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:05.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.146+0000 7f8f58a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f540cedd0 0x7f8f540caf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:05.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.146+0000 7f8f58a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f5415df40 0x7f8f54146e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:05.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.146+0000 7f8f58a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f540d33f0 con 0x7f8f5415df40 2024-03-24T01:26:05.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.146+0000 7f8f58a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f540d1ec0 con 0x7f8f540cedd0 2024-03-24T01:26:05.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.146+0000 7f8f58a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f540d3570 con 0x7f8f540b94e0 2024-03-24T01:26:05.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.147+0000 7f8f52800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f5415df40 0x7f8f54146e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:05.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.147+0000 7f8f52800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f5415df40 0x7f8f54146e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60050/0 (socket says 172.21.15.90:60050) 2024-03-24T01:26:05.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.147+0000 7f8f52800640 1 -- 172.21.15.90:0/3182553406 learned_addr learned my addr 172.21.15.90:0/3182553406 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:05.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.147+0000 7f8f52800640 1 -- 172.21.15.90:0/3182553406 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f540b94e0 msgr2=0x7f8f540b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:05.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.147+0000 7f8f52800640 1 --2- 172.21.15.90:0/3182553406 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f540b94e0 0x7f8f540b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:05.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.147+0000 7f8f52800640 1 -- 172.21.15.90:0/3182553406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f540cedd0 msgr2=0x7f8f540caf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:05.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.147+0000 7f8f49400640 1 --2- 172.21.15.90:0/3182553406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f540cedd0 0x7f8f540caf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:05.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.147+0000 7f8f52800640 1 --2- 172.21.15.90:0/3182553406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f540cedd0 0x7f8f540caf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:05.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.147+0000 7f8f52800640 1 -- 172.21.15.90:0/3182553406 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f540b5fd0 con 0x7f8f5415df40 2024-03-24T01:26:05.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.147+0000 7f8f52800640 1 --2- 172.21.15.90:0/3182553406 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f5415df40 0x7f8f54146e40 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c00c760 tx=0x7f8f4c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:05.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.148+0000 7f8f4aa00640 1 -- 172.21.15.90:0/3182553406 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c003da0 con 0x7f8f5415df40 2024-03-24T01:26:05.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.148+0000 7f8f4aa00640 1 -- 172.21.15.90:0/3182553406 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f4c005070 con 0x7f8f5415df40 2024-03-24T01:26:05.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.148+0000 7f8f4aa00640 1 -- 172.21.15.90:0/3182553406 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c005460 con 0x7f8f5415df40 2024-03-24T01:26:05.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.148+0000 7f8f58a00640 1 -- 172.21.15.90:0/3182553406 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f540b4f30 con 0x7f8f5415df40 2024-03-24T01:26:05.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.148+0000 7f8f58a00640 1 -- 172.21.15.90:0/3182553406 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f540b54f0 con 0x7f8f5415df40 2024-03-24T01:26:05.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.149+0000 7f8f58a00640 1 -- 172.21.15.90:0/3182553406 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f54068970 con 0x7f8f5415df40 2024-03-24T01:26:05.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.150+0000 7f8f4aa00640 1 -- 172.21.15.90:0/3182553406 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8f4c005600 con 0x7f8f5415df40 2024-03-24T01:26:05.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.150+0000 7f8f4aa00640 1 --2- 172.21.15.90:0/3182553406 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8f3006bbd0 0x7f8f3006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:05.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.151+0000 7f8f4aa00640 1 -- 172.21.15.90:0/3182553406 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8f4c08ace0 con 0x7f8f5415df40 2024-03-24T01:26:05.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.151+0000 7f8f51e00640 1 --2- 172.21.15.90:0/3182553406 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8f3006bbd0 0x7f8f3006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:05.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.152+0000 7f8f51e00640 1 --2- 172.21.15.90:0/3182553406 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8f3006bbd0 0x7f8f3006e090 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8f44036000 tx=0x7f8f4403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:05.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.155+0000 7f8f4aa00640 1 -- 172.21.15.90:0/3182553406 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8f54068970 con 0x7f8f5415df40 2024-03-24T01:26:05.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.375+0000 7f8f58a00640 1 -- 172.21.15.90:0/3182553406 --> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8f540d1820 con 0x7f8f3006bbd0 2024-03-24T01:26:05.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.394+0000 7f8f4aa00640 1 -- 172.21.15.90:0/3182553406 <== mgr.4104 v2:172.21.15.120:6832/860392634 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246634 (secure 0 0 0) 0x7f8f540d1820 con 0x7f8f3006bbd0 2024-03-24T01:26:05.397 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:26:05.399 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-24T01:26:05.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.400+0000 7f8f58a00640 1 -- 172.21.15.90:0/3182553406 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8f3006bbd0 msgr2=0x7f8f3006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:05.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.400+0000 7f8f58a00640 1 --2- 172.21.15.90:0/3182553406 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8f3006bbd0 0x7f8f3006e090 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8f44036000 tx=0x7f8f4403b040 comp rx=0 tx=0).stop 2024-03-24T01:26:05.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.400+0000 7f8f58a00640 1 -- 172.21.15.90:0/3182553406 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f5415df40 msgr2=0x7f8f54146e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:05.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.400+0000 7f8f58a00640 1 --2- 172.21.15.90:0/3182553406 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f5415df40 0x7f8f54146e40 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c00c760 tx=0x7f8f4c00cc20 comp rx=0 tx=0).stop 2024-03-24T01:26:05.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.400+0000 7f8f58a00640 1 -- 172.21.15.90:0/3182553406 shutdown_connections 2024-03-24T01:26:05.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.400+0000 7f8f58a00640 1 --2- 172.21.15.90:0/3182553406 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8f3006bbd0 0x7f8f3006e090 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:05.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.400+0000 7f8f58a00640 1 --2- 172.21.15.90:0/3182553406 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f5415df40 0x7f8f54146e40 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:05.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.400+0000 7f8f58a00640 1 --2- 172.21.15.90:0/3182553406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f540cedd0 0x7f8f540caf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:05.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.400+0000 7f8f58a00640 1 --2- 172.21.15.90:0/3182553406 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8f540b94e0 0x7f8f540b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:05.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.400+0000 7f8f58a00640 1 -- 172.21.15.90:0/3182553406 >> 172.21.15.90:0/3182553406 conn(0x7f8f5406c9f0 msgr2=0x7f8f540c77f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:05.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.400+0000 7f8f58a00640 1 -- 172.21.15.90:0/3182553406 shutdown_connections 2024-03-24T01:26:05.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.400+0000 7f8f58a00640 1 -- 172.21.15.90:0/3182553406 wait complete. 2024-03-24T01:26:05.419 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":94,"stamp":"2024-03-24T01:26:03.521564+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220600,"kb_used_data":4428,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754120,"statfs":{"total":773094113280,"available":772868218880,"internally_reserved":0,"allocated":4534272,"data_stored":2615145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12747,"internal_metadata":221236789},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.573169"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.977282+0000","last_change":"2024-03-24T01:25:58.977282+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.977282+0000","last_undegraded":"2024-03-24T01:25:58.977282+0000","last_fullsized":"2024-03-24T01:25:58.977282+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.975953+0000","last_change":"2024-03-24T01:25:58.975953+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.975953+0000","last_undegraded":"2024-03-24T01:25:58.975953+0000","last_fullsized":"2024-03-24T01:25:58.975953+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.979446+0000","last_change":"2024-03-24T01:25:58.979446+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.979446+0000","last_undegraded":"2024-03-24T01:25:58.979446+0000","last_fullsized":"2024-03-24T01:25:58.979446+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.974695+0000","last_change":"2024-03-24T01:25:58.974695+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.974695+0000","last_undegraded":"2024-03-24T01:25:58.974695+0000","last_fullsized":"2024-03-24T01:25:58.974695+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.973754+0000","last_change":"2024-03-24T01:25:58.973754+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.973754+0000","last_undegraded":"2024-03-24T01:25:58.973754+0000","last_fullsized":"2024-03-24T01:25:58.973754+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.986679+0000","last_change":"2024-03-24T01:25:58.986679+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.986679+0000","last_undegraded":"2024-03-24T01:25:58.986679+0000","last_fullsized":"2024-03-24T01:25:58.986679+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.989725+0000","last_change":"2024-03-24T01:25:58.989725+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.989725+0000","last_undegraded":"2024-03-24T01:25:58.989725+0000","last_fullsized":"2024-03-24T01:25:58.989725+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.978405+0000","last_change":"2024-03-24T01:25:58.978405+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.978405+0000","last_undegraded":"2024-03-24T01:25:58.978405+0000","last_fullsized":"2024-03-24T01:25:58.978405+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.978353+0000","last_change":"2024-03-24T01:25:58.978353+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.978353+0000","last_undegraded":"2024-03-24T01:25:58.978353+0000","last_fullsized":"2024-03-24T01:25:58.978353+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.995242+0000","last_change":"2024-03-24T01:25:58.995242+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.995242+0000","last_undegraded":"2024-03-24T01:25:58.995242+0000","last_fullsized":"2024-03-24T01:25:58.995242+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.990081+0000","last_change":"2024-03-24T01:25:58.990081+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.990081+0000","last_undegraded":"2024-03-24T01:25:58.990081+0000","last_fullsized":"2024-03-24T01:25:58.990081+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.976660+0000","last_change":"2024-03-24T01:25:58.976660+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.976660+0000","last_undegraded":"2024-03-24T01:25:58.976660+0000","last_fullsized":"2024-03-24T01:25:58.976660+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.986804+0000","last_change":"2024-03-24T01:25:58.986804+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.986804+0000","last_undegraded":"2024-03-24T01:25:58.986804+0000","last_fullsized":"2024-03-24T01:25:58.986804+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.980568+0000","last_change":"2024-03-24T01:25:58.980568+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.980568+0000","last_undegraded":"2024-03-24T01:25:58.980568+0000","last_fullsized":"2024-03-24T01:25:58.980568+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.986804+0000","last_change":"2024-03-24T01:25:58.986804+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.986804+0000","last_undegraded":"2024-03-24T01:25:58.986804+0000","last_fullsized":"2024-03-24T01:25:58.986804+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.993159+0000","last_change":"2024-03-24T01:25:58.993159+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.993159+0000","last_undegraded":"2024-03-24T01:25:58.993159+0000","last_fullsized":"2024-03-24T01:25:58.993159+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.991992+0000","last_change":"2024-03-24T01:25:58.991992+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.991992+0000","last_undegraded":"2024-03-24T01:25:58.991992+0000","last_fullsized":"2024-03-24T01:25:58.991992+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.974166+0000","last_change":"2024-03-24T01:25:58.974166+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.974166+0000","last_undegraded":"2024-03-24T01:25:58.974166+0000","last_fullsized":"2024-03-24T01:25:58.974166+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.973263+0000","last_change":"2024-03-24T01:25:58.973263+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.973263+0000","last_undegraded":"2024-03-24T01:25:58.973263+0000","last_fullsized":"2024-03-24T01:25:58.973263+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.974422+0000","last_change":"2024-03-24T01:25:58.974422+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.974422+0000","last_undegraded":"2024-03-24T01:25:58.974422+0000","last_fullsized":"2024-03-24T01:25:58.974422+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.986049+0000","last_change":"2024-03-24T01:25:58.986049+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.986049+0000","last_undegraded":"2024-03-24T01:25:58.986049+0000","last_fullsized":"2024-03-24T01:25:58.986049+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.977959+0000","last_change":"2024-03-24T01:25:58.977959+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.977959+0000","last_undegraded":"2024-03-24T01:25:58.977959+0000","last_fullsized":"2024-03-24T01:25:58.977959+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.989831+0000","last_change":"2024-03-24T01:25:58.989831+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.989831+0000","last_undegraded":"2024-03-24T01:25:58.989831+0000","last_fullsized":"2024-03-24T01:25:58.989831+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.972222+0000","last_change":"2024-03-24T01:25:58.972222+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.972222+0000","last_undegraded":"2024-03-24T01:25:58.972222+0000","last_fullsized":"2024-03-24T01:25:58.972222+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.975433+0000","last_change":"2024-03-24T01:25:58.975433+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.975433+0000","last_undegraded":"2024-03-24T01:25:58.975433+0000","last_fullsized":"2024-03-24T01:25:58.975433+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.985886+0000","last_change":"2024-03-24T01:25:58.985886+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.985886+0000","last_undegraded":"2024-03-24T01:25:58.985886+0000","last_fullsized":"2024-03-24T01:25:58.985886+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.977178+0000","last_change":"2024-03-24T01:25:58.977178+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.977178+0000","last_undegraded":"2024-03-24T01:25:58.977178+0000","last_fullsized":"2024-03-24T01:25:58.977178+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.986473+0000","last_change":"2024-03-24T01:25:58.986473+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.986473+0000","last_undegraded":"2024-03-24T01:25:58.986473+0000","last_fullsized":"2024-03-24T01:25:58.986473+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.975161+0000","last_change":"2024-03-24T01:25:58.975161+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.975161+0000","last_undegraded":"2024-03-24T01:25:58.975161+0000","last_fullsized":"2024-03-24T01:25:58.975161+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.983076+0000","last_change":"2024-03-24T01:25:58.983076+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.983076+0000","last_undegraded":"2024-03-24T01:25:58.983076+0000","last_fullsized":"2024-03-24T01:25:58.983076+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.989181+0000","last_change":"2024-03-24T01:25:58.989181+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.989181+0000","last_undegraded":"2024-03-24T01:25:58.989181+0000","last_fullsized":"2024-03-24T01:25:58.989181+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.981936+0000","last_change":"2024-03-24T01:25:58.981936+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.981936+0000","last_undegraded":"2024-03-24T01:25:58.981936+0000","last_fullsized":"2024-03-24T01:25:58.981936+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.994459+0000","last_change":"2024-03-24T01:25:58.994459+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.994459+0000","last_undegraded":"2024-03-24T01:25:58.994459+0000","last_fullsized":"2024-03-24T01:25:58.994459+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.001458+0000","last_change":"2024-03-24T01:25:58.996685+0000","last_active":"2024-03-24T01:25:59.001458+0000","last_peered":"2024-03-24T01:25:59.001458+0000","last_clean":"2024-03-24T01:25:59.001458+0000","last_became_active":"2024-03-24T01:25:58.996398+0000","last_became_peered":"2024-03-24T01:25:58.996398+0000","last_unstale":"2024-03-24T01:25:59.001458+0000","last_undegraded":"2024-03-24T01:25:59.001458+0000","last_fullsized":"2024-03-24T01:25:59.001458+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:53:49.271423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.984299+0000","last_change":"2024-03-24T01:25:58.984299+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.984299+0000","last_undegraded":"2024-03-24T01:25:58.984299+0000","last_fullsized":"2024-03-24T01:25:58.984299+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.004893+0000","last_change":"2024-03-24T01:25:59.002453+0000","last_active":"2024-03-24T01:25:59.004893+0000","last_peered":"2024-03-24T01:25:59.004893+0000","last_clean":"2024-03-24T01:25:59.004893+0000","last_became_active":"2024-03-24T01:25:59.002155+0000","last_became_peered":"2024-03-24T01:25:59.002155+0000","last_unstale":"2024-03-24T01:25:59.004893+0000","last_undegraded":"2024-03-24T01:25:59.004893+0000","last_fullsized":"2024-03-24T01:25:59.004893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:53:49.271423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.978899+0000","last_change":"2024-03-24T01:25:58.978899+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.978899+0000","last_undegraded":"2024-03-24T01:25:58.978899+0000","last_fullsized":"2024-03-24T01:25:58.978899+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.998379+0000","last_change":"2024-03-24T01:25:58.997334+0000","last_active":"2024-03-24T01:25:58.998379+0000","last_peered":"2024-03-24T01:25:58.998379+0000","last_clean":"2024-03-24T01:25:58.998379+0000","last_became_active":"2024-03-24T01:25:58.997033+0000","last_became_peered":"2024-03-24T01:25:58.997033+0000","last_unstale":"2024-03-24T01:25:58.998379+0000","last_undegraded":"2024-03-24T01:25:58.998379+0000","last_fullsized":"2024-03-24T01:25:58.998379+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:45.605005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.977778+0000","last_change":"2024-03-24T01:25:58.977778+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.977778+0000","last_undegraded":"2024-03-24T01:25:58.977778+0000","last_fullsized":"2024-03-24T01:25:58.977778+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.996426+0000","last_change":"2024-03-24T01:25:58.992317+0000","last_active":"2024-03-24T01:25:58.996426+0000","last_peered":"2024-03-24T01:25:58.996426+0000","last_clean":"2024-03-24T01:25:58.996426+0000","last_became_active":"2024-03-24T01:25:58.991652+0000","last_became_peered":"2024-03-24T01:25:58.991652+0000","last_unstale":"2024-03-24T01:25:58.996426+0000","last_undegraded":"2024-03-24T01:25:58.996426+0000","last_fullsized":"2024-03-24T01:25:58.996426+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:53:49.271423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.982673+0000","last_change":"2024-03-24T01:25:58.982673+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.982673+0000","last_undegraded":"2024-03-24T01:25:58.982673+0000","last_fullsized":"2024-03-24T01:25:58.982673+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.004193+0000","last_change":"2024-03-24T01:25:58.998337+0000","last_active":"2024-03-24T01:25:59.004193+0000","last_peered":"2024-03-24T01:25:59.004193+0000","last_clean":"2024-03-24T01:25:59.004193+0000","last_became_active":"2024-03-24T01:25:58.998056+0000","last_became_peered":"2024-03-24T01:25:58.998056+0000","last_unstale":"2024-03-24T01:25:59.004193+0000","last_undegraded":"2024-03-24T01:25:59.004193+0000","last_fullsized":"2024-03-24T01:25:59.004193+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:49:24.128591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.985591+0000","last_change":"2024-03-24T01:25:58.985591+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.985591+0000","last_undegraded":"2024-03-24T01:25:58.985591+0000","last_fullsized":"2024-03-24T01:25:58.985591+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.012523+0000","last_change":"2024-03-24T01:25:59.006913+0000","last_active":"2024-03-24T01:25:59.012523+0000","last_peered":"2024-03-24T01:25:59.012523+0000","last_clean":"2024-03-24T01:25:59.012523+0000","last_became_active":"2024-03-24T01:25:59.006098+0000","last_became_peered":"2024-03-24T01:25:59.006098+0000","last_unstale":"2024-03-24T01:25:59.012523+0000","last_undegraded":"2024-03-24T01:25:59.012523+0000","last_fullsized":"2024-03-24T01:25:59.012523+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:49:24.128591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.976758+0000","last_change":"2024-03-24T01:25:58.976758+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.976758+0000","last_undegraded":"2024-03-24T01:25:58.976758+0000","last_fullsized":"2024-03-24T01:25:58.976758+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.004687+0000","last_change":"2024-03-24T01:25:59.000318+0000","last_active":"2024-03-24T01:25:59.004687+0000","last_peered":"2024-03-24T01:25:59.004687+0000","last_clean":"2024-03-24T01:25:59.004687+0000","last_became_active":"2024-03-24T01:25:59.000085+0000","last_became_peered":"2024-03-24T01:25:59.000085+0000","last_unstale":"2024-03-24T01:25:59.004687+0000","last_undegraded":"2024-03-24T01:25:59.004687+0000","last_fullsized":"2024-03-24T01:25:59.004687+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:28:18.909603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.980649+0000","last_change":"2024-03-24T01:25:58.980649+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.980649+0000","last_undegraded":"2024-03-24T01:25:58.980649+0000","last_fullsized":"2024-03-24T01:25:58.980649+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.999166+0000","last_change":"2024-03-24T01:25:58.994521+0000","last_active":"2024-03-24T01:25:58.999166+0000","last_peered":"2024-03-24T01:25:58.999166+0000","last_clean":"2024-03-24T01:25:58.999166+0000","last_became_active":"2024-03-24T01:25:58.994215+0000","last_became_peered":"2024-03-24T01:25:58.994215+0000","last_unstale":"2024-03-24T01:25:58.999166+0000","last_undegraded":"2024-03-24T01:25:58.999166+0000","last_fullsized":"2024-03-24T01:25:58.999166+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:45.605005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.972522+0000","last_change":"2024-03-24T01:25:58.972522+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.972522+0000","last_undegraded":"2024-03-24T01:25:58.972522+0000","last_fullsized":"2024-03-24T01:25:58.972522+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.996515+0000","last_change":"2024-03-24T01:25:58.995322+0000","last_active":"2024-03-24T01:25:58.996515+0000","last_peered":"2024-03-24T01:25:58.996515+0000","last_clean":"2024-03-24T01:25:58.996515+0000","last_became_active":"2024-03-24T01:25:58.994847+0000","last_became_peered":"2024-03-24T01:25:58.994847+0000","last_unstale":"2024-03-24T01:25:58.996515+0000","last_undegraded":"2024-03-24T01:25:58.996515+0000","last_fullsized":"2024-03-24T01:25:58.996515+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:12:43.140922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.995109+0000","last_change":"2024-03-24T01:25:58.995109+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.995109+0000","last_undegraded":"2024-03-24T01:25:58.995109+0000","last_fullsized":"2024-03-24T01:25:58.995109+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.004224+0000","last_change":"2024-03-24T01:25:58.998109+0000","last_active":"2024-03-24T01:25:59.004224+0000","last_peered":"2024-03-24T01:25:59.004224+0000","last_clean":"2024-03-24T01:25:59.004224+0000","last_became_active":"2024-03-24T01:25:58.997791+0000","last_became_peered":"2024-03-24T01:25:58.997791+0000","last_unstale":"2024-03-24T01:25:59.004224+0000","last_undegraded":"2024-03-24T01:25:59.004224+0000","last_fullsized":"2024-03-24T01:25:59.004224+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:00:42.635663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.976014+0000","last_change":"2024-03-24T01:25:58.976014+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.976014+0000","last_undegraded":"2024-03-24T01:25:58.976014+0000","last_fullsized":"2024-03-24T01:25:58.976014+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.004257+0000","last_change":"2024-03-24T01:25:58.999040+0000","last_active":"2024-03-24T01:25:59.004257+0000","last_peered":"2024-03-24T01:25:59.004257+0000","last_clean":"2024-03-24T01:25:59.004257+0000","last_became_active":"2024-03-24T01:25:58.998593+0000","last_became_peered":"2024-03-24T01:25:58.998593+0000","last_unstale":"2024-03-24T01:25:59.004257+0000","last_undegraded":"2024-03-24T01:25:59.004257+0000","last_fullsized":"2024-03-24T01:25:59.004257+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:53:49.271423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.978116+0000","last_change":"2024-03-24T01:25:58.978116+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.978116+0000","last_undegraded":"2024-03-24T01:25:58.978116+0000","last_fullsized":"2024-03-24T01:25:58.978116+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.007336+0000","last_change":"2024-03-24T01:25:59.002378+0000","last_active":"2024-03-24T01:25:59.007336+0000","last_peered":"2024-03-24T01:25:59.007336+0000","last_clean":"2024-03-24T01:25:59.007336+0000","last_became_active":"2024-03-24T01:25:59.002071+0000","last_became_peered":"2024-03-24T01:25:59.002071+0000","last_unstale":"2024-03-24T01:25:59.007336+0000","last_undegraded":"2024-03-24T01:25:59.007336+0000","last_fullsized":"2024-03-24T01:25:59.007336+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:05:31.709112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.980063+0000","last_change":"2024-03-24T01:25:58.980063+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.980063+0000","last_undegraded":"2024-03-24T01:25:58.980063+0000","last_fullsized":"2024-03-24T01:25:58.980063+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.990060+0000","last_change":"2024-03-24T01:25:58.987270+0000","last_active":"2024-03-24T01:25:58.990060+0000","last_peered":"2024-03-24T01:25:58.990060+0000","last_clean":"2024-03-24T01:25:58.990060+0000","last_became_active":"2024-03-24T01:25:58.986953+0000","last_became_peered":"2024-03-24T01:25:58.986953+0000","last_unstale":"2024-03-24T01:25:58.990060+0000","last_undegraded":"2024-03-24T01:25:58.990060+0000","last_fullsized":"2024-03-24T01:25:58.990060+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:19:04.197281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.981124+0000","last_change":"2024-03-24T01:25:58.981124+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.981124+0000","last_undegraded":"2024-03-24T01:25:58.981124+0000","last_fullsized":"2024-03-24T01:25:58.981124+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.004607+0000","last_change":"2024-03-24T01:25:59.000275+0000","last_active":"2024-03-24T01:25:59.004607+0000","last_peered":"2024-03-24T01:25:59.004607+0000","last_clean":"2024-03-24T01:25:59.004607+0000","last_became_active":"2024-03-24T01:25:58.999948+0000","last_became_peered":"2024-03-24T01:25:58.999948+0000","last_unstale":"2024-03-24T01:25:59.004607+0000","last_undegraded":"2024-03-24T01:25:59.004607+0000","last_fullsized":"2024-03-24T01:25:59.004607+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:44:12.069166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.987147+0000","last_change":"2024-03-24T01:25:58.987147+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.987147+0000","last_undegraded":"2024-03-24T01:25:58.987147+0000","last_fullsized":"2024-03-24T01:25:58.987147+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.995109+0000","last_change":"2024-03-24T01:25:58.991563+0000","last_active":"2024-03-24T01:25:58.995109+0000","last_peered":"2024-03-24T01:25:58.995109+0000","last_clean":"2024-03-24T01:25:58.995109+0000","last_became_active":"2024-03-24T01:25:58.990946+0000","last_became_peered":"2024-03-24T01:25:58.990946+0000","last_unstale":"2024-03-24T01:25:58.995109+0000","last_undegraded":"2024-03-24T01:25:58.995109+0000","last_fullsized":"2024-03-24T01:25:58.995109+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:49:24.128591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.980154+0000","last_change":"2024-03-24T01:25:58.980154+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.980154+0000","last_undegraded":"2024-03-24T01:25:58.980154+0000","last_fullsized":"2024-03-24T01:25:58.980154+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.009845+0000","last_change":"2024-03-24T01:25:59.006569+0000","last_active":"2024-03-24T01:25:59.009845+0000","last_peered":"2024-03-24T01:25:59.009845+0000","last_clean":"2024-03-24T01:25:59.009845+0000","last_became_active":"2024-03-24T01:25:59.006280+0000","last_became_peered":"2024-03-24T01:25:59.006280+0000","last_unstale":"2024-03-24T01:25:59.009845+0000","last_undegraded":"2024-03-24T01:25:59.009845+0000","last_fullsized":"2024-03-24T01:25:59.009845+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:00:42.635663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.976902+0000","last_change":"2024-03-24T01:25:58.976902+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.976902+0000","last_undegraded":"2024-03-24T01:25:58.976902+0000","last_fullsized":"2024-03-24T01:25:58.976902+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.005436+0000","last_change":"2024-03-24T01:25:59.003724+0000","last_active":"2024-03-24T01:25:59.005436+0000","last_peered":"2024-03-24T01:25:59.005436+0000","last_clean":"2024-03-24T01:25:59.005436+0000","last_became_active":"2024-03-24T01:25:59.003410+0000","last_became_peered":"2024-03-24T01:25:59.003410+0000","last_unstale":"2024-03-24T01:25:59.005436+0000","last_undegraded":"2024-03-24T01:25:59.005436+0000","last_fullsized":"2024-03-24T01:25:59.005436+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:08:38.108175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.989181+0000","last_change":"2024-03-24T01:25:58.989181+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.989181+0000","last_undegraded":"2024-03-24T01:25:58.989181+0000","last_fullsized":"2024-03-24T01:25:58.989181+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.002870+0000","last_change":"2024-03-24T01:25:58.999764+0000","last_active":"2024-03-24T01:25:59.002870+0000","last_peered":"2024-03-24T01:25:59.002870+0000","last_clean":"2024-03-24T01:25:59.002870+0000","last_became_active":"2024-03-24T01:25:58.999490+0000","last_became_peered":"2024-03-24T01:25:58.999490+0000","last_unstale":"2024-03-24T01:25:59.002870+0000","last_undegraded":"2024-03-24T01:25:59.002870+0000","last_fullsized":"2024-03-24T01:25:59.002870+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:44:12.069166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.983702+0000","last_change":"2024-03-24T01:25:58.983702+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.983702+0000","last_undegraded":"2024-03-24T01:25:58.983702+0000","last_fullsized":"2024-03-24T01:25:58.983702+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.007469+0000","last_change":"2024-03-24T01:25:59.003238+0000","last_active":"2024-03-24T01:25:59.007469+0000","last_peered":"2024-03-24T01:25:59.007469+0000","last_clean":"2024-03-24T01:25:59.007469+0000","last_became_active":"2024-03-24T01:25:59.002905+0000","last_became_peered":"2024-03-24T01:25:59.002905+0000","last_unstale":"2024-03-24T01:25:59.007469+0000","last_undegraded":"2024-03-24T01:25:59.007469+0000","last_fullsized":"2024-03-24T01:25:59.007469+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:56:29.523110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.972343+0000","last_change":"2024-03-24T01:25:58.972343+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.972343+0000","last_undegraded":"2024-03-24T01:25:58.972343+0000","last_fullsized":"2024-03-24T01:25:58.972343+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.996158+0000","last_change":"2024-03-24T01:25:58.994043+0000","last_active":"2024-03-24T01:25:58.996158+0000","last_peered":"2024-03-24T01:25:58.996158+0000","last_clean":"2024-03-24T01:25:58.996158+0000","last_became_active":"2024-03-24T01:25:58.993746+0000","last_became_peered":"2024-03-24T01:25:58.993746+0000","last_unstale":"2024-03-24T01:25:58.996158+0000","last_undegraded":"2024-03-24T01:25:58.996158+0000","last_fullsized":"2024-03-24T01:25:58.996158+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:08:38.108175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.987330+0000","last_change":"2024-03-24T01:25:58.987330+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.987330+0000","last_undegraded":"2024-03-24T01:25:58.987330+0000","last_fullsized":"2024-03-24T01:25:58.987330+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.991383+0000","last_change":"2024-03-24T01:25:58.988228+0000","last_active":"2024-03-24T01:25:58.991383+0000","last_peered":"2024-03-24T01:25:58.991383+0000","last_clean":"2024-03-24T01:25:58.991383+0000","last_became_active":"2024-03-24T01:25:58.987941+0000","last_became_peered":"2024-03-24T01:25:58.987941+0000","last_unstale":"2024-03-24T01:25:58.991383+0000","last_undegraded":"2024-03-24T01:25:58.991383+0000","last_fullsized":"2024-03-24T01:25:58.991383+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:12:25.446123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.976174+0000","last_change":"2024-03-24T01:25:58.976174+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.976174+0000","last_undegraded":"2024-03-24T01:25:58.976174+0000","last_fullsized":"2024-03-24T01:25:58.976174+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.993352+0000","last_change":"2024-03-24T01:25:58.991840+0000","last_active":"2024-03-24T01:25:58.993352+0000","last_peered":"2024-03-24T01:25:58.993352+0000","last_clean":"2024-03-24T01:25:58.993352+0000","last_became_active":"2024-03-24T01:25:58.991522+0000","last_became_peered":"2024-03-24T01:25:58.991522+0000","last_unstale":"2024-03-24T01:25:58.993352+0000","last_undegraded":"2024-03-24T01:25:58.993352+0000","last_fullsized":"2024-03-24T01:25:58.993352+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:03:47.985379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.985984+0000","last_change":"2024-03-24T01:25:58.985984+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.985984+0000","last_undegraded":"2024-03-24T01:25:58.985984+0000","last_fullsized":"2024-03-24T01:25:58.985984+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.009786+0000","last_change":"2024-03-24T01:25:59.006226+0000","last_active":"2024-03-24T01:25:59.009786+0000","last_peered":"2024-03-24T01:25:59.009786+0000","last_clean":"2024-03-24T01:25:59.009786+0000","last_became_active":"2024-03-24T01:25:59.005739+0000","last_became_peered":"2024-03-24T01:25:59.005739+0000","last_unstale":"2024-03-24T01:25:59.009786+0000","last_undegraded":"2024-03-24T01:25:59.009786+0000","last_fullsized":"2024-03-24T01:25:59.009786+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:07:08.132339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.976537+0000","last_change":"2024-03-24T01:25:58.976537+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.976537+0000","last_undegraded":"2024-03-24T01:25:58.976537+0000","last_fullsized":"2024-03-24T01:25:58.976537+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.991351+0000","last_change":"2024-03-24T01:25:58.988520+0000","last_active":"2024-03-24T01:25:58.991351+0000","last_peered":"2024-03-24T01:25:58.991351+0000","last_clean":"2024-03-24T01:25:58.991351+0000","last_became_active":"2024-03-24T01:25:58.988149+0000","last_became_peered":"2024-03-24T01:25:58.988149+0000","last_unstale":"2024-03-24T01:25:58.991351+0000","last_undegraded":"2024-03-24T01:25:58.991351+0000","last_fullsized":"2024-03-24T01:25:58.991351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:13:07.896803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.983029+0000","last_change":"2024-03-24T01:25:58.983029+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.983029+0000","last_undegraded":"2024-03-24T01:25:58.983029+0000","last_fullsized":"2024-03-24T01:25:58.983029+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.009953+0000","last_change":"2024-03-24T01:25:59.005278+0000","last_active":"2024-03-24T01:25:59.009953+0000","last_peered":"2024-03-24T01:25:59.009953+0000","last_clean":"2024-03-24T01:25:59.009953+0000","last_became_active":"2024-03-24T01:25:59.004896+0000","last_became_peered":"2024-03-24T01:25:59.004896+0000","last_unstale":"2024-03-24T01:25:59.009953+0000","last_undegraded":"2024-03-24T01:25:59.009953+0000","last_fullsized":"2024-03-24T01:25:59.009953+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:45.605005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.989993+0000","last_change":"2024-03-24T01:25:58.989993+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.989993+0000","last_undegraded":"2024-03-24T01:25:58.989993+0000","last_fullsized":"2024-03-24T01:25:58.989993+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.007491+0000","last_change":"2024-03-24T01:25:59.004018+0000","last_active":"2024-03-24T01:25:59.007491+0000","last_peered":"2024-03-24T01:25:59.007491+0000","last_clean":"2024-03-24T01:25:59.007491+0000","last_became_active":"2024-03-24T01:25:59.003661+0000","last_became_peered":"2024-03-24T01:25:59.003661+0000","last_unstale":"2024-03-24T01:25:59.007491+0000","last_undegraded":"2024-03-24T01:25:59.007491+0000","last_fullsized":"2024-03-24T01:25:59.007491+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:35:02.448771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.975406+0000","last_change":"2024-03-24T01:25:58.975406+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.975406+0000","last_undegraded":"2024-03-24T01:25:58.975406+0000","last_fullsized":"2024-03-24T01:25:58.975406+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.998477+0000","last_change":"2024-03-24T01:25:58.995201+0000","last_active":"2024-03-24T01:25:58.998477+0000","last_peered":"2024-03-24T01:25:58.998477+0000","last_clean":"2024-03-24T01:25:58.998477+0000","last_became_active":"2024-03-24T01:25:58.994890+0000","last_became_peered":"2024-03-24T01:25:58.994890+0000","last_unstale":"2024-03-24T01:25:58.998477+0000","last_undegraded":"2024-03-24T01:25:58.998477+0000","last_fullsized":"2024-03-24T01:25:58.998477+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:49:24.128591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.978947+0000","last_change":"2024-03-24T01:25:58.978947+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.978947+0000","last_undegraded":"2024-03-24T01:25:58.978947+0000","last_fullsized":"2024-03-24T01:25:58.978947+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.993897+0000","last_change":"2024-03-24T01:25:58.990591+0000","last_active":"2024-03-24T01:25:58.993897+0000","last_peered":"2024-03-24T01:25:58.993897+0000","last_clean":"2024-03-24T01:25:58.993897+0000","last_became_active":"2024-03-24T01:25:58.990269+0000","last_became_peered":"2024-03-24T01:25:58.990269+0000","last_unstale":"2024-03-24T01:25:58.993897+0000","last_undegraded":"2024-03-24T01:25:58.993897+0000","last_fullsized":"2024-03-24T01:25:58.993897+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:00:42.635663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.979426+0000","last_change":"2024-03-24T01:25:58.979426+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.979426+0000","last_undegraded":"2024-03-24T01:25:58.979426+0000","last_fullsized":"2024-03-24T01:25:58.979426+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.993563+0000","last_change":"2024-03-24T01:25:58.988526+0000","last_active":"2024-03-24T01:25:58.993563+0000","last_peered":"2024-03-24T01:25:58.993563+0000","last_clean":"2024-03-24T01:25:58.993563+0000","last_became_active":"2024-03-24T01:25:58.987932+0000","last_became_peered":"2024-03-24T01:25:58.987932+0000","last_unstale":"2024-03-24T01:25:58.993563+0000","last_undegraded":"2024-03-24T01:25:58.993563+0000","last_fullsized":"2024-03-24T01:25:58.993563+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:00:42.635663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.979324+0000","last_change":"2024-03-24T01:25:58.979324+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.979324+0000","last_undegraded":"2024-03-24T01:25:58.979324+0000","last_fullsized":"2024-03-24T01:25:58.979324+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.995934+0000","last_change":"2024-03-24T01:25:58.992830+0000","last_active":"2024-03-24T01:25:58.995934+0000","last_peered":"2024-03-24T01:25:58.995934+0000","last_clean":"2024-03-24T01:25:58.995934+0000","last_became_active":"2024-03-24T01:25:58.991569+0000","last_became_peered":"2024-03-24T01:25:58.991569+0000","last_unstale":"2024-03-24T01:25:58.995934+0000","last_undegraded":"2024-03-24T01:25:58.995934+0000","last_fullsized":"2024-03-24T01:25:58.995934+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:08:38.108175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.987080+0000","last_change":"2024-03-24T01:25:58.987080+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.987080+0000","last_undegraded":"2024-03-24T01:25:58.987080+0000","last_fullsized":"2024-03-24T01:25:58.987080+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.997322+0000","last_change":"2024-03-24T01:25:58.996128+0000","last_active":"2024-03-24T01:25:58.997322+0000","last_peered":"2024-03-24T01:25:58.997322+0000","last_clean":"2024-03-24T01:25:58.997322+0000","last_became_active":"2024-03-24T01:25:58.994831+0000","last_became_peered":"2024-03-24T01:25:58.994831+0000","last_unstale":"2024-03-24T01:25:58.997322+0000","last_undegraded":"2024-03-24T01:25:58.997322+0000","last_fullsized":"2024-03-24T01:25:58.997322+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:46:49.838842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-03-24T01:25:58.991355+0000","last_change":"2024-03-24T01:25:58.991355+0000","last_active":"2024-03-24T01:25:58.948130+0000","last_peered":"2024-03-24T01:25:58.948130+0000","last_clean":"2024-03-24T01:25:58.948130+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:25:58.991355+0000","last_undegraded":"2024-03-24T01:25:58.991355+0000","last_fullsized":"2024-03-24T01:25:58.991355+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:59.012545+0000","last_change":"2024-03-24T01:25:59.008252+0000","last_active":"2024-03-24T01:25:59.012545+0000","last_peered":"2024-03-24T01:25:59.012545+0000","last_clean":"2024-03-24T01:25:59.012545+0000","last_became_active":"2024-03-24T01:25:59.007969+0000","last_became_peered":"2024-03-24T01:25:59.007969+0000","last_unstale":"2024-03-24T01:25:59.012545+0000","last_undegraded":"2024-03-24T01:25:59.012545+0000","last_fullsized":"2024-03-24T01:25:59.012545+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:53:49.271423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":200,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-24T01:25:58.999857+0000","last_change":"2024-03-24T01:24:01.545855+0000","last_active":"2024-03-24T01:25:58.999857+0000","last_peered":"2024-03-24T01:25:58.999857+0000","last_clean":"2024-03-24T01:25:58.999857+0000","last_became_active":"2024-03-24T01:24:01.543811+0000","last_became_peered":"2024-03-24T01:24:01.543811+0000","last_unstale":"2024-03-24T01:25:58.999857+0000","last_undegraded":"2024-03-24T01:25:58.999857+0000","last_fullsized":"2024-03-24T01:25:58.999857+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_clean_scrub_stamp":"2024-03-24T01:23:59.861564+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:03.682631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343836,"statfs":{"total":96636764160,"available":96608088064,"internally_reserved":0,"allocated":1007616,"data_stored":767169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574875,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":176801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574876,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":176801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574875,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":425984,"data_stored":183468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":176801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":425984,"data_stored":183468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":176801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343820,"statfs":{"total":96636764160,"available":96608071680,"internally_reserved":0,"allocated":1019904,"data_stored":773836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T01:26:05.423 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3f is not active+clean 2024-03-24T01:26:05.423 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.977282+0000', 'last_change': '2024-03-24T01:25:58.977282+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.977282+0000', 'last_undegraded': '2024-03-24T01:25:58.977282+0000', 'last_fullsized': '2024-03-24T01:25:58.977282+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T01:26:05.423 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3e is not active+clean 2024-03-24T01:26:05.423 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.975953+0000', 'last_change': '2024-03-24T01:25:58.975953+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.975953+0000', 'last_undegraded': '2024-03-24T01:25:58.975953+0000', 'last_fullsized': '2024-03-24T01:25:58.975953+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T01:26:05.423 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3d is not active+clean 2024-03-24T01:26:05.423 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.979446+0000', 'last_change': '2024-03-24T01:25:58.979446+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.979446+0000', 'last_undegraded': '2024-03-24T01:25:58.979446+0000', 'last_fullsized': '2024-03-24T01:25:58.979446+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T01:26:05.424 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3c is not active+clean 2024-03-24T01:26:05.424 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.974695+0000', 'last_change': '2024-03-24T01:25:58.974695+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.974695+0000', 'last_undegraded': '2024-03-24T01:25:58.974695+0000', 'last_fullsized': '2024-03-24T01:25:58.974695+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-24T01:26:05.424 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3b is not active+clean 2024-03-24T01:26:05.424 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.973754+0000', 'last_change': '2024-03-24T01:25:58.973754+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.973754+0000', 'last_undegraded': '2024-03-24T01:25:58.973754+0000', 'last_fullsized': '2024-03-24T01:25:58.973754+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-24T01:26:05.424 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3a is not active+clean 2024-03-24T01:26:05.424 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.986679+0000', 'last_change': '2024-03-24T01:25:58.986679+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.986679+0000', 'last_undegraded': '2024-03-24T01:25:58.986679+0000', 'last_fullsized': '2024-03-24T01:25:58.986679+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T01:26:05.424 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.39 is not active+clean 2024-03-24T01:26:05.424 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.989725+0000', 'last_change': '2024-03-24T01:25:58.989725+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.989725+0000', 'last_undegraded': '2024-03-24T01:25:58.989725+0000', 'last_fullsized': '2024-03-24T01:25:58.989725+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T01:26:05.424 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.38 is not active+clean 2024-03-24T01:26:05.424 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.978405+0000', 'last_change': '2024-03-24T01:25:58.978405+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.978405+0000', 'last_undegraded': '2024-03-24T01:25:58.978405+0000', 'last_fullsized': '2024-03-24T01:25:58.978405+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T01:26:05.425 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.37 is not active+clean 2024-03-24T01:26:05.425 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.978353+0000', 'last_change': '2024-03-24T01:25:58.978353+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.978353+0000', 'last_undegraded': '2024-03-24T01:25:58.978353+0000', 'last_fullsized': '2024-03-24T01:25:58.978353+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T01:26:05.425 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.36 is not active+clean 2024-03-24T01:26:05.425 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.995242+0000', 'last_change': '2024-03-24T01:25:58.995242+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.995242+0000', 'last_undegraded': '2024-03-24T01:25:58.995242+0000', 'last_fullsized': '2024-03-24T01:25:58.995242+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T01:26:05.425 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.35 is not active+clean 2024-03-24T01:26:05.425 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.990081+0000', 'last_change': '2024-03-24T01:25:58.990081+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.990081+0000', 'last_undegraded': '2024-03-24T01:25:58.990081+0000', 'last_fullsized': '2024-03-24T01:25:58.990081+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T01:26:05.425 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.34 is not active+clean 2024-03-24T01:26:05.425 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.976660+0000', 'last_change': '2024-03-24T01:25:58.976660+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.976660+0000', 'last_undegraded': '2024-03-24T01:25:58.976660+0000', 'last_fullsized': '2024-03-24T01:25:58.976660+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T01:26:05.425 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.33 is not active+clean 2024-03-24T01:26:05.425 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.986804+0000', 'last_change': '2024-03-24T01:25:58.986804+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.986804+0000', 'last_undegraded': '2024-03-24T01:25:58.986804+0000', 'last_fullsized': '2024-03-24T01:25:58.986804+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T01:26:05.425 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.32 is not active+clean 2024-03-24T01:26:05.425 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.980568+0000', 'last_change': '2024-03-24T01:25:58.980568+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.980568+0000', 'last_undegraded': '2024-03-24T01:25:58.980568+0000', 'last_fullsized': '2024-03-24T01:25:58.980568+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T01:26:05.426 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.31 is not active+clean 2024-03-24T01:26:05.426 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.986804+0000', 'last_change': '2024-03-24T01:25:58.986804+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.986804+0000', 'last_undegraded': '2024-03-24T01:25:58.986804+0000', 'last_fullsized': '2024-03-24T01:25:58.986804+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T01:26:05.426 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.30 is not active+clean 2024-03-24T01:26:05.426 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.993159+0000', 'last_change': '2024-03-24T01:25:58.993159+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.993159+0000', 'last_undegraded': '2024-03-24T01:25:58.993159+0000', 'last_fullsized': '2024-03-24T01:25:58.993159+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T01:26:05.426 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2f is not active+clean 2024-03-24T01:26:05.426 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.991992+0000', 'last_change': '2024-03-24T01:25:58.991992+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.991992+0000', 'last_undegraded': '2024-03-24T01:25:58.991992+0000', 'last_fullsized': '2024-03-24T01:25:58.991992+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T01:26:05.426 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2e is not active+clean 2024-03-24T01:26:05.426 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.974166+0000', 'last_change': '2024-03-24T01:25:58.974166+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.974166+0000', 'last_undegraded': '2024-03-24T01:25:58.974166+0000', 'last_fullsized': '2024-03-24T01:25:58.974166+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T01:26:05.426 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2d is not active+clean 2024-03-24T01:26:05.426 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.973263+0000', 'last_change': '2024-03-24T01:25:58.973263+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.973263+0000', 'last_undegraded': '2024-03-24T01:25:58.973263+0000', 'last_fullsized': '2024-03-24T01:25:58.973263+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-24T01:26:05.427 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2c is not active+clean 2024-03-24T01:26:05.427 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.974422+0000', 'last_change': '2024-03-24T01:25:58.974422+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.974422+0000', 'last_undegraded': '2024-03-24T01:25:58.974422+0000', 'last_fullsized': '2024-03-24T01:25:58.974422+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T01:26:05.427 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2b is not active+clean 2024-03-24T01:26:05.427 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.986049+0000', 'last_change': '2024-03-24T01:25:58.986049+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.986049+0000', 'last_undegraded': '2024-03-24T01:25:58.986049+0000', 'last_fullsized': '2024-03-24T01:25:58.986049+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T01:26:05.427 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2a is not active+clean 2024-03-24T01:26:05.427 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.977959+0000', 'last_change': '2024-03-24T01:25:58.977959+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.977959+0000', 'last_undegraded': '2024-03-24T01:25:58.977959+0000', 'last_fullsized': '2024-03-24T01:25:58.977959+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T01:26:05.427 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.29 is not active+clean 2024-03-24T01:26:05.427 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.989831+0000', 'last_change': '2024-03-24T01:25:58.989831+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.989831+0000', 'last_undegraded': '2024-03-24T01:25:58.989831+0000', 'last_fullsized': '2024-03-24T01:25:58.989831+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T01:26:05.427 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.28 is not active+clean 2024-03-24T01:26:05.427 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.972222+0000', 'last_change': '2024-03-24T01:25:58.972222+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.972222+0000', 'last_undegraded': '2024-03-24T01:25:58.972222+0000', 'last_fullsized': '2024-03-24T01:25:58.972222+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-24T01:26:05.427 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.27 is not active+clean 2024-03-24T01:26:05.427 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.975433+0000', 'last_change': '2024-03-24T01:25:58.975433+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.975433+0000', 'last_undegraded': '2024-03-24T01:25:58.975433+0000', 'last_fullsized': '2024-03-24T01:25:58.975433+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T01:26:05.428 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.26 is not active+clean 2024-03-24T01:26:05.428 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.985886+0000', 'last_change': '2024-03-24T01:25:58.985886+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.985886+0000', 'last_undegraded': '2024-03-24T01:25:58.985886+0000', 'last_fullsized': '2024-03-24T01:25:58.985886+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T01:26:05.428 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.25 is not active+clean 2024-03-24T01:26:05.428 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.977178+0000', 'last_change': '2024-03-24T01:25:58.977178+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.977178+0000', 'last_undegraded': '2024-03-24T01:25:58.977178+0000', 'last_fullsized': '2024-03-24T01:25:58.977178+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T01:26:05.428 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.24 is not active+clean 2024-03-24T01:26:05.428 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.986473+0000', 'last_change': '2024-03-24T01:25:58.986473+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.986473+0000', 'last_undegraded': '2024-03-24T01:25:58.986473+0000', 'last_fullsized': '2024-03-24T01:25:58.986473+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T01:26:05.428 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.23 is not active+clean 2024-03-24T01:26:05.428 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.975161+0000', 'last_change': '2024-03-24T01:25:58.975161+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.975161+0000', 'last_undegraded': '2024-03-24T01:25:58.975161+0000', 'last_fullsized': '2024-03-24T01:25:58.975161+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T01:26:05.428 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.22 is not active+clean 2024-03-24T01:26:05.428 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.983076+0000', 'last_change': '2024-03-24T01:25:58.983076+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.983076+0000', 'last_undegraded': '2024-03-24T01:25:58.983076+0000', 'last_fullsized': '2024-03-24T01:25:58.983076+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T01:26:05.428 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.21 is not active+clean 2024-03-24T01:26:05.428 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.989181+0000', 'last_change': '2024-03-24T01:25:58.989181+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.989181+0000', 'last_undegraded': '2024-03-24T01:25:58.989181+0000', 'last_fullsized': '2024-03-24T01:25:58.989181+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T01:26:05.429 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.20 is not active+clean 2024-03-24T01:26:05.429 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.981936+0000', 'last_change': '2024-03-24T01:25:58.981936+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.981936+0000', 'last_undegraded': '2024-03-24T01:25:58.981936+0000', 'last_fullsized': '2024-03-24T01:25:58.981936+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T01:26:05.429 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1c is not active+clean 2024-03-24T01:26:05.429 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.994459+0000', 'last_change': '2024-03-24T01:25:58.994459+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.994459+0000', 'last_undegraded': '2024-03-24T01:25:58.994459+0000', 'last_fullsized': '2024-03-24T01:25:58.994459+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T01:26:05.429 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1d is not active+clean 2024-03-24T01:26:05.429 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.984299+0000', 'last_change': '2024-03-24T01:25:58.984299+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.984299+0000', 'last_undegraded': '2024-03-24T01:25:58.984299+0000', 'last_fullsized': '2024-03-24T01:25:58.984299+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T01:26:05.429 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1e is not active+clean 2024-03-24T01:26:05.429 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.978899+0000', 'last_change': '2024-03-24T01:25:58.978899+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.978899+0000', 'last_undegraded': '2024-03-24T01:25:58.978899+0000', 'last_fullsized': '2024-03-24T01:25:58.978899+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T01:26:05.429 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1f is not active+clean 2024-03-24T01:26:05.429 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.977778+0000', 'last_change': '2024-03-24T01:25:58.977778+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.977778+0000', 'last_undegraded': '2024-03-24T01:25:58.977778+0000', 'last_fullsized': '2024-03-24T01:25:58.977778+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T01:26:05.429 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.18 is not active+clean 2024-03-24T01:26:05.430 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.982673+0000', 'last_change': '2024-03-24T01:25:58.982673+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.982673+0000', 'last_undegraded': '2024-03-24T01:25:58.982673+0000', 'last_fullsized': '2024-03-24T01:25:58.982673+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T01:26:05.430 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.19 is not active+clean 2024-03-24T01:26:05.430 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.985591+0000', 'last_change': '2024-03-24T01:25:58.985591+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.985591+0000', 'last_undegraded': '2024-03-24T01:25:58.985591+0000', 'last_fullsized': '2024-03-24T01:25:58.985591+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T01:26:05.430 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1a is not active+clean 2024-03-24T01:26:05.430 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.976758+0000', 'last_change': '2024-03-24T01:25:58.976758+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.976758+0000', 'last_undegraded': '2024-03-24T01:25:58.976758+0000', 'last_fullsized': '2024-03-24T01:25:58.976758+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T01:26:05.430 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1b is not active+clean 2024-03-24T01:26:05.430 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.980649+0000', 'last_change': '2024-03-24T01:25:58.980649+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.980649+0000', 'last_undegraded': '2024-03-24T01:25:58.980649+0000', 'last_fullsized': '2024-03-24T01:25:58.980649+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T01:26:05.430 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.14 is not active+clean 2024-03-24T01:26:05.430 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.972522+0000', 'last_change': '2024-03-24T01:25:58.972522+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.972522+0000', 'last_undegraded': '2024-03-24T01:25:58.972522+0000', 'last_fullsized': '2024-03-24T01:25:58.972522+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-24T01:26:05.430 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.15 is not active+clean 2024-03-24T01:26:05.430 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.995109+0000', 'last_change': '2024-03-24T01:25:58.995109+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.995109+0000', 'last_undegraded': '2024-03-24T01:25:58.995109+0000', 'last_fullsized': '2024-03-24T01:25:58.995109+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T01:26:05.431 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.16 is not active+clean 2024-03-24T01:26:05.431 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.976014+0000', 'last_change': '2024-03-24T01:25:58.976014+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.976014+0000', 'last_undegraded': '2024-03-24T01:25:58.976014+0000', 'last_fullsized': '2024-03-24T01:25:58.976014+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T01:26:05.431 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.17 is not active+clean 2024-03-24T01:26:05.431 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.978116+0000', 'last_change': '2024-03-24T01:25:58.978116+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.978116+0000', 'last_undegraded': '2024-03-24T01:25:58.978116+0000', 'last_fullsized': '2024-03-24T01:25:58.978116+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T01:26:05.431 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.10 is not active+clean 2024-03-24T01:26:05.431 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.980063+0000', 'last_change': '2024-03-24T01:25:58.980063+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.980063+0000', 'last_undegraded': '2024-03-24T01:25:58.980063+0000', 'last_fullsized': '2024-03-24T01:25:58.980063+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T01:26:05.431 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.11 is not active+clean 2024-03-24T01:26:05.431 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.981124+0000', 'last_change': '2024-03-24T01:25:58.981124+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.981124+0000', 'last_undegraded': '2024-03-24T01:25:58.981124+0000', 'last_fullsized': '2024-03-24T01:25:58.981124+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T01:26:05.431 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.12 is not active+clean 2024-03-24T01:26:05.431 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.987147+0000', 'last_change': '2024-03-24T01:25:58.987147+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.987147+0000', 'last_undegraded': '2024-03-24T01:25:58.987147+0000', 'last_fullsized': '2024-03-24T01:25:58.987147+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T01:26:05.431 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.13 is not active+clean 2024-03-24T01:26:05.432 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.980154+0000', 'last_change': '2024-03-24T01:25:58.980154+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.980154+0000', 'last_undegraded': '2024-03-24T01:25:58.980154+0000', 'last_fullsized': '2024-03-24T01:25:58.980154+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T01:26:05.432 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.c is not active+clean 2024-03-24T01:26:05.432 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.976902+0000', 'last_change': '2024-03-24T01:25:58.976902+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.976902+0000', 'last_undegraded': '2024-03-24T01:25:58.976902+0000', 'last_fullsized': '2024-03-24T01:25:58.976902+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T01:26:05.432 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.d is not active+clean 2024-03-24T01:26:05.432 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.989181+0000', 'last_change': '2024-03-24T01:25:58.989181+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.989181+0000', 'last_undegraded': '2024-03-24T01:25:58.989181+0000', 'last_fullsized': '2024-03-24T01:25:58.989181+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T01:26:05.432 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.e is not active+clean 2024-03-24T01:26:05.432 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.983702+0000', 'last_change': '2024-03-24T01:25:58.983702+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.983702+0000', 'last_undegraded': '2024-03-24T01:25:58.983702+0000', 'last_fullsized': '2024-03-24T01:25:58.983702+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T01:26:05.432 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.f is not active+clean 2024-03-24T01:26:05.432 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.972343+0000', 'last_change': '2024-03-24T01:25:58.972343+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.972343+0000', 'last_undegraded': '2024-03-24T01:25:58.972343+0000', 'last_fullsized': '2024-03-24T01:25:58.972343+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-24T01:26:05.432 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.8 is not active+clean 2024-03-24T01:26:05.432 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.987330+0000', 'last_change': '2024-03-24T01:25:58.987330+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.987330+0000', 'last_undegraded': '2024-03-24T01:25:58.987330+0000', 'last_fullsized': '2024-03-24T01:25:58.987330+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T01:26:05.433 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.9 is not active+clean 2024-03-24T01:26:05.433 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.976174+0000', 'last_change': '2024-03-24T01:25:58.976174+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.976174+0000', 'last_undegraded': '2024-03-24T01:25:58.976174+0000', 'last_fullsized': '2024-03-24T01:25:58.976174+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T01:26:05.433 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.a is not active+clean 2024-03-24T01:26:05.433 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.985984+0000', 'last_change': '2024-03-24T01:25:58.985984+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.985984+0000', 'last_undegraded': '2024-03-24T01:25:58.985984+0000', 'last_fullsized': '2024-03-24T01:25:58.985984+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T01:26:05.433 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.b is not active+clean 2024-03-24T01:26:05.433 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.976537+0000', 'last_change': '2024-03-24T01:25:58.976537+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.976537+0000', 'last_undegraded': '2024-03-24T01:25:58.976537+0000', 'last_fullsized': '2024-03-24T01:25:58.976537+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T01:26:05.433 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.4 is not active+clean 2024-03-24T01:26:05.433 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.983029+0000', 'last_change': '2024-03-24T01:25:58.983029+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.983029+0000', 'last_undegraded': '2024-03-24T01:25:58.983029+0000', 'last_fullsized': '2024-03-24T01:25:58.983029+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T01:26:05.433 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.5 is not active+clean 2024-03-24T01:26:05.433 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.989993+0000', 'last_change': '2024-03-24T01:25:58.989993+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.989993+0000', 'last_undegraded': '2024-03-24T01:25:58.989993+0000', 'last_fullsized': '2024-03-24T01:25:58.989993+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T01:26:05.433 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.6 is not active+clean 2024-03-24T01:26:05.433 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.975406+0000', 'last_change': '2024-03-24T01:25:58.975406+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.975406+0000', 'last_undegraded': '2024-03-24T01:25:58.975406+0000', 'last_fullsized': '2024-03-24T01:25:58.975406+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T01:26:05.434 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.0 is not active+clean 2024-03-24T01:26:05.434 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.978947+0000', 'last_change': '2024-03-24T01:25:58.978947+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.978947+0000', 'last_undegraded': '2024-03-24T01:25:58.978947+0000', 'last_fullsized': '2024-03-24T01:25:58.978947+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T01:26:05.434 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1 is not active+clean 2024-03-24T01:26:05.434 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.979426+0000', 'last_change': '2024-03-24T01:25:58.979426+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.979426+0000', 'last_undegraded': '2024-03-24T01:25:58.979426+0000', 'last_fullsized': '2024-03-24T01:25:58.979426+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T01:26:05.434 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2 is not active+clean 2024-03-24T01:26:05.434 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.979324+0000', 'last_change': '2024-03-24T01:25:58.979324+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.979324+0000', 'last_undegraded': '2024-03-24T01:25:58.979324+0000', 'last_fullsized': '2024-03-24T01:25:58.979324+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T01:26:05.434 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3 is not active+clean 2024-03-24T01:26:05.434 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.987080+0000', 'last_change': '2024-03-24T01:25:58.987080+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.987080+0000', 'last_undegraded': '2024-03-24T01:25:58.987080+0000', 'last_fullsized': '2024-03-24T01:25:58.987080+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T01:26:05.434 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.7 is not active+clean 2024-03-24T01:26:05.435 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-03-24T01:25:58.991355+0000', 'last_change': '2024-03-24T01:25:58.991355+0000', 'last_active': '2024-03-24T01:25:58.948130+0000', 'last_peered': '2024-03-24T01:25:58.948130+0000', 'last_clean': '2024-03-24T01:25:58.948130+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:25:58.991355+0000', 'last_undegraded': '2024-03-24T01:25:58.991355+0000', 'last_fullsized': '2024-03-24T01:25:58.991355+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'last_clean_scrub_stamp': '2024-03-24T01:25:58.948130+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T01:26:05.435 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-24T01:26:05.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.670+0000 7ff5e7e00640 1 Processor -- start 2024-03-24T01:26:05.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.670+0000 7ff5e7e00640 1 -- start start 2024-03-24T01:26:05.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.672+0000 7ff5e7e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff5e00cc150 0x7ff5e00ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:05.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.672+0000 7ff5e7e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5e01603e0 0x7ff5e006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:05.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.672+0000 7ff5e7e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff5e0145430 0x7ff5e006b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:05.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.672+0000 7ff5e7e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5e00cd490 con 0x7ff5e00cc150 2024-03-24T01:26:05.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.672+0000 7ff5e7e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5e015f900 con 0x7ff5e0145430 2024-03-24T01:26:05.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.672+0000 7ff5e7e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e015fbe0 con 0x7ff5e01603e0 2024-03-24T01:26:05.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.672+0000 7ff5e5400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff5e00cc150 0x7ff5e00ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:05.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.672+0000 7ff5e5400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff5e00cc150 0x7ff5e00ccb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60064/0 (socket says 172.21.15.90:60064) 2024-03-24T01:26:05.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.672+0000 7ff5e5400640 1 -- 172.21.15.90:0/3976344639 learned_addr learned my addr 172.21.15.90:0/3976344639 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:05.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.673+0000 7ff5e5400640 1 -- 172.21.15.90:0/3976344639 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5e01603e0 msgr2=0x7ff5e006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:05.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.673+0000 7ff5e5400640 1 --2- 172.21.15.90:0/3976344639 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5e01603e0 0x7ff5e006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:05.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.673+0000 7ff5e5400640 1 -- 172.21.15.90:0/3976344639 >> v2:172.21.15.120:3300/0 conn(0x7ff5e0145430 msgr2=0x7ff5e006b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:05.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.673+0000 7ff5e5400640 1 --2- 172.21.15.90:0/3976344639 >> v2:172.21.15.120:3300/0 conn(0x7ff5e0145430 0x7ff5e006b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:05.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.673+0000 7ff5e5400640 1 -- 172.21.15.90:0/3976344639 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e015fec0 con 0x7ff5e00cc150 2024-03-24T01:26:05.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.674+0000 7ff5e5400640 1 --2- 172.21.15.90:0/3976344639 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5e00cc150 0x7ff5e00ccb60 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0071750 tx=0x7ff5d402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3e8a391ecf6fbd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:05.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.674+0000 7ff5dfe00640 1 -- 172.21.15.90:0/3976344639 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d4035020 con 0x7ff5e00cc150 2024-03-24T01:26:05.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.674+0000 7ff5dfe00640 1 -- 172.21.15.90:0/3976344639 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d402d040 con 0x7ff5e00cc150 2024-03-24T01:26:05.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.674+0000 7ff5dfe00640 1 -- 172.21.15.90:0/3976344639 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d4031a80 con 0x7ff5e00cc150 2024-03-24T01:26:05.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.674+0000 7ff5e7e00640 1 -- 172.21.15.90:0/3976344639 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5e00cc150 msgr2=0x7ff5e00ccb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:05.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.674+0000 7ff5e7e00640 1 --2- 172.21.15.90:0/3976344639 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5e00cc150 0x7ff5e00ccb60 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0071750 tx=0x7ff5d402b900 comp rx=0 tx=0).stop 2024-03-24T01:26:05.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.675+0000 7ff5e7e00640 1 -- 172.21.15.90:0/3976344639 shutdown_connections 2024-03-24T01:26:05.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.675+0000 7ff5e7e00640 1 --2- 172.21.15.90:0/3976344639 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5e01603e0 0x7ff5e006b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:05.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.675+0000 7ff5e7e00640 1 --2- 172.21.15.90:0/3976344639 >> v2:172.21.15.120:3300/0 conn(0x7ff5e0145430 0x7ff5e006b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:05.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.675+0000 7ff5e7e00640 1 --2- 172.21.15.90:0/3976344639 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5e00cc150 0x7ff5e00ccb60 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:05.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.675+0000 7ff5e7e00640 1 -- 172.21.15.90:0/3976344639 >> 172.21.15.90:0/3976344639 conn(0x7ff5e00b1770 msgr2=0x7ff5e00b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:05.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.675+0000 7ff5e7e00640 1 -- 172.21.15.90:0/3976344639 shutdown_connections 2024-03-24T01:26:05.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.675+0000 7ff5e7e00640 1 -- 172.21.15.90:0/3976344639 wait complete. 2024-03-24T01:26:05.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.676+0000 7ff5e7e00640 1 Processor -- start 2024-03-24T01:26:05.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.677+0000 7ff5e7e00640 1 -- start start 2024-03-24T01:26:05.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.677+0000 7ff5e7e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5e00cc150 0x7ff5e015b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:05.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.677+0000 7ff5e7e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5e0145430 0x7ff5e015ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:05.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.677+0000 7ff5e7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5e01603e0 0x7ff5e01e6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:05.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.677+0000 7ff5e7e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e0071b10 con 0x7ff5e01603e0 2024-03-24T01:26:05.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.677+0000 7ff5e7e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e0071990 con 0x7ff5e00cc150 2024-03-24T01:26:05.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.677+0000 7ff5e7e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e0071c90 con 0x7ff5e0145430 2024-03-24T01:26:05.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.678+0000 7ff5e5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5e01603e0 0x7ff5e01e6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:05.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.678+0000 7ff5e5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5e01603e0 0x7ff5e01e6f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60074/0 (socket says 172.21.15.90:60074) 2024-03-24T01:26:05.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.678+0000 7ff5e5e00640 1 -- 172.21.15.90:0/91964319 learned_addr learned my addr 172.21.15.90:0/91964319 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:05.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.678+0000 7ff5e4a00640 1 --2- 172.21.15.90:0/91964319 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5e0145430 0x7ff5e015ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:05.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.678+0000 7ff5e5400640 1 --2- 172.21.15.90:0/91964319 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5e00cc150 0x7ff5e015b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:05.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.678+0000 7ff5e5e00640 1 -- 172.21.15.90:0/91964319 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5e0145430 msgr2=0x7ff5e015ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:05.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.678+0000 7ff5e5e00640 1 --2- 172.21.15.90:0/91964319 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5e0145430 0x7ff5e015ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:05.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.678+0000 7ff5e5e00640 1 -- 172.21.15.90:0/91964319 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5e00cc150 msgr2=0x7ff5e015b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:05.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.678+0000 7ff5e5e00640 1 --2- 172.21.15.90:0/91964319 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5e00cc150 0x7ff5e015b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:05.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.678+0000 7ff5e5e00640 1 -- 172.21.15.90:0/91964319 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e015c780 con 0x7ff5e01603e0 2024-03-24T01:26:05.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.678+0000 7ff5e5e00640 1 --2- 172.21.15.90:0/91964319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5e01603e0 0x7ff5e01e6f20 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7ff5d800b9a0 tx=0x7ff5d800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:05.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.679+0000 7ff5dd600640 1 -- 172.21.15.90:0/91964319 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d8019070 con 0x7ff5e01603e0 2024-03-24T01:26:05.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.679+0000 7ff5dd600640 1 -- 172.21.15.90:0/91964319 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d800cbc0 con 0x7ff5e01603e0 2024-03-24T01:26:05.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.679+0000 7ff5dd600640 1 -- 172.21.15.90:0/91964319 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d8014440 con 0x7ff5e01603e0 2024-03-24T01:26:05.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.679+0000 7ff5e7e00640 1 -- 172.21.15.90:0/91964319 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5e01e7bf0 con 0x7ff5e01603e0 2024-03-24T01:26:05.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.679+0000 7ff5e7e00640 1 -- 172.21.15.90:0/91964319 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5e01e81b0 con 0x7ff5e01603e0 2024-03-24T01:26:05.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.680+0000 7ff5e7e00640 1 -- 172.21.15.90:0/91964319 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5e0068060 con 0x7ff5e01603e0 2024-03-24T01:26:05.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.682+0000 7ff5dd600640 1 -- 172.21.15.90:0/91964319 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff5d801c070 con 0x7ff5e01603e0 2024-03-24T01:26:05.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.683+0000 7ff5dd600640 1 --2- 172.21.15.90:0/91964319 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff5c006bbd0 0x7ff5c006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:05.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.684+0000 7ff5dd600640 1 -- 172.21.15.90:0/91964319 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff5d808a900 con 0x7ff5e01603e0 2024-03-24T01:26:05.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.684+0000 7ff5e5400640 1 --2- 172.21.15.90:0/91964319 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff5c006bbd0 0x7ff5c006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:05.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.685+0000 7ff5e5400640 1 --2- 172.21.15.90:0/91964319 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff5c006bbd0 0x7ff5c006e090 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4036000 tx=0x7ff5d403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:05.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:05.689+0000 7ff5dd600640 1 -- 172.21.15.90:0/91964319 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff5d80592f0 con 0x7ff5e01603e0 2024-03-24T01:26:06.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.004+0000 7ff5e7e00640 1 -- 172.21.15.90:0/91964319 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7ff5e00b3220 con 0x7ff5e01603e0 2024-03-24T01:26:06.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.005+0000 7ff5dd600640 1 -- 172.21.15.90:0/91964319 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1034 (secure 0 0 0) 0x7ff5d8058bf0 con 0x7ff5e01603e0 2024-03-24T01:26:06.006 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:26:06.006 INFO:teuthology.orchestra.run.smithi090.stdout:{"fsid":"24712165-9839-4fcc-8033-988386efea74","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":139,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":8,"num_up_osds":8,"osd_up_since":1711243438,"num_in_osds":8,"osd_in_since":1711243429,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":225894400,"bytes_avail":772868218880,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":17,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"c","status":"up:active","gid":5496}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-03-24T01:25:17.504438+0000","services":{}},"progress_events":{}} 2024-03-24T01:26:06.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.007+0000 7ff5e7e00640 1 -- 172.21.15.90:0/91964319 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff5c006bbd0 msgr2=0x7ff5c006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:06.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.007+0000 7ff5e7e00640 1 --2- 172.21.15.90:0/91964319 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff5c006bbd0 0x7ff5c006e090 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4036000 tx=0x7ff5d403c040 comp rx=0 tx=0).stop 2024-03-24T01:26:06.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.008+0000 7ff5e7e00640 1 -- 172.21.15.90:0/91964319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5e01603e0 msgr2=0x7ff5e01e6f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:06.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.008+0000 7ff5e7e00640 1 --2- 172.21.15.90:0/91964319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5e01603e0 0x7ff5e01e6f20 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7ff5d800b9a0 tx=0x7ff5d800be60 comp rx=0 tx=0).stop 2024-03-24T01:26:06.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.008+0000 7ff5e7e00640 1 -- 172.21.15.90:0/91964319 shutdown_connections 2024-03-24T01:26:06.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.008+0000 7ff5e7e00640 1 --2- 172.21.15.90:0/91964319 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff5c006bbd0 0x7ff5c006e090 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:06.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.008+0000 7ff5e7e00640 1 --2- 172.21.15.90:0/91964319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5e01603e0 0x7ff5e01e6f20 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:06.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.008+0000 7ff5e7e00640 1 --2- 172.21.15.90:0/91964319 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff5e0145430 0x7ff5e015ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:06.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.008+0000 7ff5e7e00640 1 --2- 172.21.15.90:0/91964319 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff5e00cc150 0x7ff5e015b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:06.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.008+0000 7ff5e7e00640 1 -- 172.21.15.90:0/91964319 >> 172.21.15.90:0/91964319 conn(0x7ff5e00b1770 msgr2=0x7ff5e00b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:06.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.008+0000 7ff5e7e00640 1 -- 172.21.15.90:0/91964319 shutdown_connections 2024-03-24T01:26:06.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.008+0000 7ff5e7e00640 1 -- 172.21.15.90:0/91964319 wait complete. 2024-03-24T01:26:06.030 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-24T01:26:06.030 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T01:26:06.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.235+0000 7f16fe600640 1 Processor -- start 2024-03-24T01:26:06.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.236+0000 7f16fe600640 1 -- start start 2024-03-24T01:26:06.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.238+0000 7f16fe600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f16f806c7d0 0x7f16f80674a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:06.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.238+0000 7f16fe600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f80b5070 con 0x7f16f80afd50 2024-03-24T01:26:06.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.238+0000 7f16fe600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f80b5350 con 0x7f16f815e230 2024-03-24T01:26:06.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.238+0000 7f16fe600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16f80b6370 con 0x7f16f806c7d0 2024-03-24T01:26:06.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.239+0000 7f16f7400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f16f80afd50 0x7f16f80b0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50104/0 (socket says 172.21.15.90:50104) 2024-03-24T01:26:06.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.239+0000 7f16f7400640 1 -- 172.21.15.90:0/1364997995 learned_addr learned my addr 172.21.15.90:0/1364997995 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:26:06.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.239+0000 7f16f6000640 1 -- 172.21.15.90:0/1364997995 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2287011797 0 0) 0x7f16f80b5070 con 0x7f16f80afd50 2024-03-24T01:26:06.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.239+0000 7f16f6000640 1 -- 172.21.15.90:0/1364997995 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16dc003520 con 0x7f16f80afd50 2024-03-24T01:26:06.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.239+0000 7f16f6a00640 1 --2- 172.21.15.90:0/1364997995 >> v2:172.21.15.120:3300/0 conn(0x7f16f806c7d0 0x7f16f80674a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:06.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.239+0000 7f16f6000640 1 -- 172.21.15.90:0/1364997995 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 24128941 0 0) 0x7f16dc003520 con 0x7f16f80afd50 2024-03-24T01:26:06.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.239+0000 7f16f6000640 1 -- 172.21.15.90:0/1364997995 >> v2:172.21.15.120:3300/0 conn(0x7f16f806c7d0 msgr2=0x7f16f80674a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:06.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.239+0000 7f16f6000640 1 --2- 172.21.15.90:0/1364997995 >> v2:172.21.15.120:3300/0 conn(0x7f16f806c7d0 0x7f16f80674a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:06.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.239+0000 7f16f6000640 1 -- 172.21.15.90:0/1364997995 >> v1:172.21.15.120:6789/0 conn(0x7f16f815e230 legacy=0x7f16f80ad1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:06.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.239+0000 7f16f6000640 1 -- 172.21.15.90:0/1364997995 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f80b6090 con 0x7f16f80afd50 2024-03-24T01:26:06.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.239+0000 7f16f6000640 1 -- 172.21.15.90:0/1364997995 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f16e8001670 con 0x7f16f80afd50 2024-03-24T01:26:06.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.239+0000 7f16f6000640 1 -- 172.21.15.90:0/1364997995 >> v1:172.21.15.90:6789/0 conn(0x7f16f80afd50 legacy=0x7f16f80b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:06.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.239+0000 7f16f6000640 1 --2- 172.21.15.90:0/1364997995 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16dc0039b0 0x7f16dc003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:06.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.240+0000 7f16f6000640 1 --2- 172.21.15.90:0/1364997995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16dc004800 0x7f16dc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:06.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.240+0000 7f16f6000640 1 --2- 172.21.15.90:0/1364997995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16dc0051c0 0x7f16dc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:06.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.240+0000 7f16f6000640 1 -- 172.21.15.90:0/1364997995 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc00c0c0 con 0x7f16dc0051c0 2024-03-24T01:26:06.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.240+0000 7f16f6000640 1 -- 172.21.15.90:0/1364997995 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc00c270 con 0x7f16dc004800 2024-03-24T01:26:06.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.240+0000 7f16f6000640 1 -- 172.21.15.90:0/1364997995 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc00c470 con 0x7f16dc0039b0 2024-03-24T01:26:06.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.240+0000 7f16f7e00640 1 --2- 172.21.15.90:0/1364997995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16dc0051c0 0x7f16dc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:06.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.240+0000 7f16f7e00640 1 -- 172.21.15.90:0/1364997995 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16dc0039b0 msgr2=0x7f16dc003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:06.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.240+0000 7f16f7e00640 1 --2- 172.21.15.90:0/1364997995 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16dc0039b0 0x7f16dc003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:06.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.240+0000 7f16f6a00640 1 --2- 172.21.15.90:0/1364997995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16dc004800 0x7f16dc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:06.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.240+0000 7f16f7e00640 1 -- 172.21.15.90:0/1364997995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16dc004800 msgr2=0x7f16dc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:06.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.240+0000 7f16f7e00640 1 --2- 172.21.15.90:0/1364997995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16dc004800 0x7f16dc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:06.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.240+0000 7f16f7e00640 1 -- 172.21.15.90:0/1364997995 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16dc003520 con 0x7f16dc0051c0 2024-03-24T01:26:06.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.240+0000 7f16f6a00640 1 --2- 172.21.15.90:0/1364997995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16dc004800 0x7f16dc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:06.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.241+0000 7f16f7e00640 1 -- 172.21.15.90:0/1364997995 reap_dead start 2024-03-24T01:26:06.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.241+0000 7f16f7e00640 1 --2- 172.21.15.90:0/1364997995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16dc0051c0 0x7f16dc00b9c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f16f815e640 tx=0x7f16dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c5d2a189f79b772 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:06.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.242+0000 7f16f6000640 1 -- 172.21.15.90:0/1364997995 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f0001940 con 0x7f16dc0051c0 2024-03-24T01:26:06.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.242+0000 7f16f6000640 1 -- 172.21.15.90:0/1364997995 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16f0003990 con 0x7f16dc0051c0 2024-03-24T01:26:06.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.242+0000 7f16f6000640 1 -- 172.21.15.90:0/1364997995 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f0003d80 con 0x7f16dc0051c0 2024-03-24T01:26:06.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.242+0000 7f16fe600640 1 -- 172.21.15.90:0/1364997995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16dc0051c0 msgr2=0x7f16dc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:06.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.242+0000 7f16fe600640 1 --2- 172.21.15.90:0/1364997995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16dc0051c0 0x7f16dc00b9c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f16f815e640 tx=0x7f16dc005830 comp rx=0 tx=0).stop 2024-03-24T01:26:06.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.243+0000 7f16fe600640 1 -- 172.21.15.90:0/1364997995 shutdown_connections 2024-03-24T01:26:06.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.243+0000 7f16fe600640 1 --2- 172.21.15.90:0/1364997995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16dc0051c0 0x7f16dc00b9c0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:06.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.243+0000 7f16fe600640 1 -- 172.21.15.90:0/1364997995 >> 172.21.15.90:0/1364997995 conn(0x7f16f8099a70 msgr2=0x7f16f80b5730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:06.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.243+0000 7f16fe600640 1 -- 172.21.15.90:0/1364997995 shutdown_connections 2024-03-24T01:26:06.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.243+0000 7f16fe600640 1 -- 172.21.15.90:0/1364997995 wait complete. 2024-03-24T01:26:06.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.244+0000 7f16fe600640 1 Processor -- start 2024-03-24T01:26:06.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.244+0000 7f16fe600640 1 -- start start 2024-03-24T01:26:06.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.245+0000 7f16fe600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16dc0051c0 0x7f16f81500b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:06.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.245+0000 7f16fe600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16f815f560 0x7f16f806f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:06.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.245+0000 7f16fe600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16f815e640 0x7f16f806fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:06.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.245+0000 7f16fe600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f80d0f70 con 0x7f16f815f560 2024-03-24T01:26:06.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.245+0000 7f16fe600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f80b5e40 con 0x7f16f815e640 2024-03-24T01:26:06.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.245+0000 7f16fe600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f80674a0 con 0x7f16dc0051c0 2024-03-24T01:26:06.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.246+0000 7f16f6a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16f815f560 0x7f16f806f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:06.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.246+0000 7f16f6a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16f815f560 0x7f16f806f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60088/0 (socket says 172.21.15.90:60088) 2024-03-24T01:26:06.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.246+0000 7f16f6a00640 1 -- 172.21.15.90:0/2259032616 learned_addr learned my addr 172.21.15.90:0/2259032616 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:06.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.246+0000 7f16f7e00640 1 --2- 172.21.15.90:0/2259032616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16f815e640 0x7f16f806fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:06.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.246+0000 7f16f7400640 1 --2- 172.21.15.90:0/2259032616 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16dc0051c0 0x7f16f81500b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:06.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.246+0000 7f16f6a00640 1 -- 172.21.15.90:0/2259032616 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16dc0051c0 msgr2=0x7f16f81500b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:06.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.246+0000 7f16f6a00640 1 --2- 172.21.15.90:0/2259032616 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16dc0051c0 0x7f16f81500b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:06.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.246+0000 7f16f6a00640 1 -- 172.21.15.90:0/2259032616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16f815e640 msgr2=0x7f16f806fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:06.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.246+0000 7f16f6a00640 1 --2- 172.21.15.90:0/2259032616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16f815e640 0x7f16f806fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:06.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.246+0000 7f16f6a00640 1 -- 172.21.15.90:0/2259032616 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f81505f0 con 0x7f16f815f560 2024-03-24T01:26:06.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.246+0000 7f16f7e00640 1 --2- 172.21.15.90:0/2259032616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16f815e640 0x7f16f806fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:06.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.246+0000 7f16f6a00640 1 --2- 172.21.15.90:0/2259032616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16f815f560 0x7f16f806f5a0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f16e400e810 tx=0x7f16e400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:06.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.247+0000 7f16ef400640 1 -- 172.21.15.90:0/2259032616 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e4017070 con 0x7f16f815f560 2024-03-24T01:26:06.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.247+0000 7f16ef400640 1 -- 172.21.15.90:0/2259032616 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e400c6d0 con 0x7f16f815f560 2024-03-24T01:26:06.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.247+0000 7f16ef400640 1 -- 172.21.15.90:0/2259032616 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e400f040 con 0x7f16f815f560 2024-03-24T01:26:06.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.247+0000 7f16fe600640 1 -- 172.21.15.90:0/2259032616 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f815fa60 con 0x7f16f815f560 2024-03-24T01:26:06.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.247+0000 7f16fe600640 1 -- 172.21.15.90:0/2259032616 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f8160020 con 0x7f16f815f560 2024-03-24T01:26:06.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.248+0000 7f16fe600640 1 -- 172.21.15.90:0/2259032616 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16f8068060 con 0x7f16f815f560 2024-03-24T01:26:06.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.249+0000 7f16ef400640 1 -- 172.21.15.90:0/2259032616 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f16e401a070 con 0x7f16f815f560 2024-03-24T01:26:06.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.250+0000 7f16ef400640 1 --2- 172.21.15.90:0/2259032616 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f16d806bc00 0x7f16d806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:06.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.250+0000 7f16ef400640 1 -- 172.21.15.90:0/2259032616 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f16e408a810 con 0x7f16f815f560 2024-03-24T01:26:06.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.250+0000 7f16f7400640 1 --2- 172.21.15.90:0/2259032616 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f16d806bc00 0x7f16d806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:06.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.252+0000 7f16f7400640 1 --2- 172.21.15.90:0/2259032616 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f16d806bc00 0x7f16d806e0c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f16e8000f60 tx=0x7f16e8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:06.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.254+0000 7f16ef400640 1 -- 172.21.15.90:0/2259032616 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f16e40590d0 con 0x7f16f815f560 2024-03-24T01:26:06.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.448+0000 7f16fe600640 1 -- 172.21.15.90:0/2259032616 --> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f16f806d2a0 con 0x7f16d806bc00 2024-03-24T01:26:06.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.466+0000 7f16ef400640 1 -- 172.21.15.90:0/2259032616 <== mgr.4104 v2:172.21.15.120:6832/860392634 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251959 (secure 0 0 0) 0x7f16f806d2a0 con 0x7f16d806bc00 2024-03-24T01:26:06.468 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:26:06.471 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-24T01:26:06.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.472+0000 7f16fe600640 1 -- 172.21.15.90:0/2259032616 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f16d806bc00 msgr2=0x7f16d806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:06.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.472+0000 7f16fe600640 1 --2- 172.21.15.90:0/2259032616 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f16d806bc00 0x7f16d806e0c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f16e8000f60 tx=0x7f16e8009040 comp rx=0 tx=0).stop 2024-03-24T01:26:06.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.472+0000 7f16fe600640 1 -- 172.21.15.90:0/2259032616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16f815f560 msgr2=0x7f16f806f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:06.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.472+0000 7f16fe600640 1 --2- 172.21.15.90:0/2259032616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16f815f560 0x7f16f806f5a0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f16e400e810 tx=0x7f16e400ecd0 comp rx=0 tx=0).stop 2024-03-24T01:26:06.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.472+0000 7f16fe600640 1 -- 172.21.15.90:0/2259032616 shutdown_connections 2024-03-24T01:26:06.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.472+0000 7f16fe600640 1 --2- 172.21.15.90:0/2259032616 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f16d806bc00 0x7f16d806e0c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:06.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.472+0000 7f16fe600640 1 --2- 172.21.15.90:0/2259032616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16f815f560 0x7f16f806f5a0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:06.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.472+0000 7f16fe600640 1 --2- 172.21.15.90:0/2259032616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16f815e640 0x7f16f806fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:06.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.472+0000 7f16fe600640 1 --2- 172.21.15.90:0/2259032616 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f16dc0051c0 0x7f16f81500b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:06.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.472+0000 7f16fe600640 1 -- 172.21.15.90:0/2259032616 >> 172.21.15.90:0/2259032616 conn(0x7f16f8099a70 msgr2=0x7f16f815bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:06.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.472+0000 7f16fe600640 1 -- 172.21.15.90:0/2259032616 shutdown_connections 2024-03-24T01:26:06.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:06.472+0000 7f16fe600640 1 -- 172.21.15.90:0/2259032616 wait complete. 2024-03-24T01:26:06.491 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2024-03-24T01:26:05.522162+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220912,"kb_used_data":4744,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753808,"statfs":{"total":773094113280,"available":772867899392,"internally_reserved":0,"allocated":4857856,"data_stored":2764172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12747,"internal_metadata":221236789},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.557520"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.998062+0000","last_change":"2024-03-24T01:26:00.019576+0000","last_active":"2024-03-24T01:26:00.998062+0000","last_peered":"2024-03-24T01:26:00.998062+0000","last_clean":"2024-03-24T01:26:00.998062+0000","last_became_active":"2024-03-24T01:26:00.019012+0000","last_became_peered":"2024-03-24T01:26:00.019012+0000","last_unstale":"2024-03-24T01:26:00.998062+0000","last_undegraded":"2024-03-24T01:26:00.998062+0000","last_fullsized":"2024-03-24T01:26:00.998062+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:39:06.645554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.997983+0000","last_change":"2024-03-24T01:26:00.002513+0000","last_active":"2024-03-24T01:26:00.997983+0000","last_peered":"2024-03-24T01:26:00.997983+0000","last_clean":"2024-03-24T01:26:00.997983+0000","last_became_active":"2024-03-24T01:26:00.002206+0000","last_became_peered":"2024-03-24T01:26:00.002206+0000","last_unstale":"2024-03-24T01:26:00.997983+0000","last_undegraded":"2024-03-24T01:26:00.997983+0000","last_fullsized":"2024-03-24T01:26:00.997983+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:02:46.296797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221988+0000","last_change":"2024-03-24T01:26:00.008425+0000","last_active":"2024-03-24T01:26:01.221988+0000","last_peered":"2024-03-24T01:26:01.221988+0000","last_clean":"2024-03-24T01:26:01.221988+0000","last_became_active":"2024-03-24T01:26:00.007866+0000","last_became_peered":"2024-03-24T01:26:00.007866+0000","last_unstale":"2024-03-24T01:26:01.221988+0000","last_undegraded":"2024-03-24T01:26:01.221988+0000","last_fullsized":"2024-03-24T01:26:01.221988+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:30:44.565229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.007149+0000","last_change":"2024-03-24T01:26:00.012605+0000","last_active":"2024-03-24T01:26:01.007149+0000","last_peered":"2024-03-24T01:26:01.007149+0000","last_clean":"2024-03-24T01:26:01.007149+0000","last_became_active":"2024-03-24T01:26:00.009919+0000","last_became_peered":"2024-03-24T01:26:00.009919+0000","last_unstale":"2024-03-24T01:26:01.007149+0000","last_undegraded":"2024-03-24T01:26:01.007149+0000","last_fullsized":"2024-03-24T01:26:01.007149+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:07:09.146157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.006993+0000","last_change":"2024-03-24T01:26:00.006183+0000","last_active":"2024-03-24T01:26:01.006993+0000","last_peered":"2024-03-24T01:26:01.006993+0000","last_clean":"2024-03-24T01:26:01.006993+0000","last_became_active":"2024-03-24T01:26:00.005891+0000","last_became_peered":"2024-03-24T01:26:00.005891+0000","last_unstale":"2024-03-24T01:26:01.006993+0000","last_undegraded":"2024-03-24T01:26:01.006993+0000","last_fullsized":"2024-03-24T01:26:01.006993+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:21:28.890740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221508+0000","last_change":"2024-03-24T01:26:00.009169+0000","last_active":"2024-03-24T01:26:01.221508+0000","last_peered":"2024-03-24T01:26:01.221508+0000","last_clean":"2024-03-24T01:26:01.221508+0000","last_became_active":"2024-03-24T01:26:00.008857+0000","last_became_peered":"2024-03-24T01:26:00.008857+0000","last_unstale":"2024-03-24T01:26:01.221508+0000","last_undegraded":"2024-03-24T01:26:01.221508+0000","last_fullsized":"2024-03-24T01:26:01.221508+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:16:28.904097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.011917+0000","last_change":"2024-03-24T01:26:00.017535+0000","last_active":"2024-03-24T01:26:01.011917+0000","last_peered":"2024-03-24T01:26:01.011917+0000","last_clean":"2024-03-24T01:26:01.011917+0000","last_became_active":"2024-03-24T01:26:00.017258+0000","last_became_peered":"2024-03-24T01:26:00.017258+0000","last_unstale":"2024-03-24T01:26:01.011917+0000","last_undegraded":"2024-03-24T01:26:01.011917+0000","last_fullsized":"2024-03-24T01:26:01.011917+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:56:23.530821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.997252+0000","last_change":"2024-03-24T01:26:00.011615+0000","last_active":"2024-03-24T01:26:00.997252+0000","last_peered":"2024-03-24T01:26:00.997252+0000","last_clean":"2024-03-24T01:26:00.997252+0000","last_became_active":"2024-03-24T01:26:00.011334+0000","last_became_peered":"2024-03-24T01:26:00.011334+0000","last_unstale":"2024-03-24T01:26:00.997252+0000","last_undegraded":"2024-03-24T01:26:00.997252+0000","last_fullsized":"2024-03-24T01:26:00.997252+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:06:55.514360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.222020+0000","last_change":"2024-03-24T01:26:00.008198+0000","last_active":"2024-03-24T01:26:01.222020+0000","last_peered":"2024-03-24T01:26:01.222020+0000","last_clean":"2024-03-24T01:26:01.222020+0000","last_became_active":"2024-03-24T01:26:00.007870+0000","last_became_peered":"2024-03-24T01:26:00.007870+0000","last_unstale":"2024-03-24T01:26:01.222020+0000","last_undegraded":"2024-03-24T01:26:01.222020+0000","last_fullsized":"2024-03-24T01:26:01.222020+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:56:23.530821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.013334+0000","last_change":"2024-03-24T01:26:00.011611+0000","last_active":"2024-03-24T01:26:01.013334+0000","last_peered":"2024-03-24T01:26:01.013334+0000","last_clean":"2024-03-24T01:26:01.013334+0000","last_became_active":"2024-03-24T01:26:00.011272+0000","last_became_peered":"2024-03-24T01:26:00.011272+0000","last_unstale":"2024-03-24T01:26:01.013334+0000","last_undegraded":"2024-03-24T01:26:01.013334+0000","last_fullsized":"2024-03-24T01:26:01.013334+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:17:45.147642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.012882+0000","last_change":"2024-03-24T01:26:00.014118+0000","last_active":"2024-03-24T01:26:01.012882+0000","last_peered":"2024-03-24T01:26:01.012882+0000","last_clean":"2024-03-24T01:26:01.012882+0000","last_became_active":"2024-03-24T01:26:00.013783+0000","last_became_peered":"2024-03-24T01:26:00.013783+0000","last_unstale":"2024-03-24T01:26:01.012882+0000","last_undegraded":"2024-03-24T01:26:01.012882+0000","last_fullsized":"2024-03-24T01:26:01.012882+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:06:55.514360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221697+0000","last_change":"2024-03-24T01:26:00.009436+0000","last_active":"2024-03-24T01:26:01.221697+0000","last_peered":"2024-03-24T01:26:01.221697+0000","last_clean":"2024-03-24T01:26:01.221697+0000","last_became_active":"2024-03-24T01:26:00.009122+0000","last_became_peered":"2024-03-24T01:26:00.009122+0000","last_unstale":"2024-03-24T01:26:01.221697+0000","last_undegraded":"2024-03-24T01:26:01.221697+0000","last_fullsized":"2024-03-24T01:26:01.221697+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:39:38.186024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.012156+0000","last_change":"2024-03-24T01:26:00.003148+0000","last_active":"2024-03-24T01:26:01.012156+0000","last_peered":"2024-03-24T01:26:01.012156+0000","last_clean":"2024-03-24T01:26:01.012156+0000","last_became_active":"2024-03-24T01:26:00.002817+0000","last_became_peered":"2024-03-24T01:26:00.002817+0000","last_unstale":"2024-03-24T01:26:01.012156+0000","last_undegraded":"2024-03-24T01:26:01.012156+0000","last_fullsized":"2024-03-24T01:26:01.012156+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:46:09.862425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.001772+0000","last_change":"2024-03-24T01:26:00.008295+0000","last_active":"2024-03-24T01:26:01.001772+0000","last_peered":"2024-03-24T01:26:01.001772+0000","last_clean":"2024-03-24T01:26:01.001772+0000","last_became_active":"2024-03-24T01:26:00.008007+0000","last_became_peered":"2024-03-24T01:26:00.008007+0000","last_unstale":"2024-03-24T01:26:01.001772+0000","last_undegraded":"2024-03-24T01:26:01.001772+0000","last_fullsized":"2024-03-24T01:26:01.001772+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:02:08.397348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.012677+0000","last_change":"2024-03-24T01:26:00.011211+0000","last_active":"2024-03-24T01:26:01.012677+0000","last_peered":"2024-03-24T01:26:01.012677+0000","last_clean":"2024-03-24T01:26:01.012677+0000","last_became_active":"2024-03-24T01:26:00.010912+0000","last_became_peered":"2024-03-24T01:26:00.010912+0000","last_unstale":"2024-03-24T01:26:01.012677+0000","last_undegraded":"2024-03-24T01:26:01.012677+0000","last_fullsized":"2024-03-24T01:26:01.012677+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:20:17.936211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221276+0000","last_change":"2024-03-24T01:26:00.009781+0000","last_active":"2024-03-24T01:26:01.221276+0000","last_peered":"2024-03-24T01:26:01.221276+0000","last_clean":"2024-03-24T01:26:01.221276+0000","last_became_active":"2024-03-24T01:26:00.009027+0000","last_became_peered":"2024-03-24T01:26:00.009027+0000","last_unstale":"2024-03-24T01:26:01.221276+0000","last_undegraded":"2024-03-24T01:26:01.221276+0000","last_fullsized":"2024-03-24T01:26:01.221276+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:20:43.942659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.222841+0000","last_change":"2024-03-24T01:26:00.011182+0000","last_active":"2024-03-24T01:26:01.222841+0000","last_peered":"2024-03-24T01:26:01.222841+0000","last_clean":"2024-03-24T01:26:01.222841+0000","last_became_active":"2024-03-24T01:26:00.010834+0000","last_became_peered":"2024-03-24T01:26:00.010834+0000","last_unstale":"2024-03-24T01:26:01.222841+0000","last_undegraded":"2024-03-24T01:26:01.222841+0000","last_fullsized":"2024-03-24T01:26:01.222841+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:11:25.571750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.999592+0000","last_change":"2024-03-24T01:26:00.009006+0000","last_active":"2024-03-24T01:26:00.999592+0000","last_peered":"2024-03-24T01:26:00.999592+0000","last_clean":"2024-03-24T01:26:00.999592+0000","last_became_active":"2024-03-24T01:26:00.008723+0000","last_became_peered":"2024-03-24T01:26:00.008723+0000","last_unstale":"2024-03-24T01:26:00.999592+0000","last_undegraded":"2024-03-24T01:26:00.999592+0000","last_fullsized":"2024-03-24T01:26:00.999592+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:05:32.722931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.007473+0000","last_change":"2024-03-24T01:26:00.005802+0000","last_active":"2024-03-24T01:26:01.007473+0000","last_peered":"2024-03-24T01:26:01.007473+0000","last_clean":"2024-03-24T01:26:01.007473+0000","last_became_active":"2024-03-24T01:26:00.005499+0000","last_became_peered":"2024-03-24T01:26:00.005499+0000","last_unstale":"2024-03-24T01:26:01.007473+0000","last_undegraded":"2024-03-24T01:26:01.007473+0000","last_fullsized":"2024-03-24T01:26:01.007473+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:49:25.142409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.999646+0000","last_change":"2024-03-24T01:26:00.008411+0000","last_active":"2024-03-24T01:26:00.999646+0000","last_peered":"2024-03-24T01:26:00.999646+0000","last_clean":"2024-03-24T01:26:00.999646+0000","last_became_active":"2024-03-24T01:26:00.008145+0000","last_became_peered":"2024-03-24T01:26:00.008145+0000","last_unstale":"2024-03-24T01:26:00.999646+0000","last_undegraded":"2024-03-24T01:26:00.999646+0000","last_fullsized":"2024-03-24T01:26:00.999646+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:59:54.117252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.012286+0000","last_change":"2024-03-24T01:26:00.007178+0000","last_active":"2024-03-24T01:26:01.012286+0000","last_peered":"2024-03-24T01:26:01.012286+0000","last_clean":"2024-03-24T01:26:01.012286+0000","last_became_active":"2024-03-24T01:26:00.005004+0000","last_became_peered":"2024-03-24T01:26:00.005004+0000","last_unstale":"2024-03-24T01:26:01.012286+0000","last_undegraded":"2024-03-24T01:26:01.012286+0000","last_fullsized":"2024-03-24T01:26:01.012286+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:07:24.167208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.000663+0000","last_change":"2024-03-24T01:26:00.019905+0000","last_active":"2024-03-24T01:26:01.000663+0000","last_peered":"2024-03-24T01:26:01.000663+0000","last_clean":"2024-03-24T01:26:01.000663+0000","last_became_active":"2024-03-24T01:26:00.019575+0000","last_became_peered":"2024-03-24T01:26:00.019575+0000","last_unstale":"2024-03-24T01:26:01.000663+0000","last_undegraded":"2024-03-24T01:26:01.000663+0000","last_fullsized":"2024-03-24T01:26:01.000663+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:16:28.904097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221506+0000","last_change":"2024-03-24T01:25:59.996770+0000","last_active":"2024-03-24T01:26:01.221506+0000","last_peered":"2024-03-24T01:26:01.221506+0000","last_clean":"2024-03-24T01:26:01.221506+0000","last_became_active":"2024-03-24T01:25:59.996482+0000","last_became_peered":"2024-03-24T01:25:59.996482+0000","last_unstale":"2024-03-24T01:26:01.221506+0000","last_undegraded":"2024-03-24T01:26:01.221506+0000","last_fullsized":"2024-03-24T01:26:01.221506+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:02:08.397348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.006809+0000","last_change":"2024-03-24T01:25:59.989965+0000","last_active":"2024-03-24T01:26:01.006809+0000","last_peered":"2024-03-24T01:26:01.006809+0000","last_clean":"2024-03-24T01:26:01.006809+0000","last_became_active":"2024-03-24T01:25:59.989645+0000","last_became_peered":"2024-03-24T01:25:59.989645+0000","last_unstale":"2024-03-24T01:26:01.006809+0000","last_undegraded":"2024-03-24T01:26:01.006809+0000","last_fullsized":"2024-03-24T01:26:01.006809+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:26:56.752921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.000081+0000","last_change":"2024-03-24T01:26:00.013258+0000","last_active":"2024-03-24T01:26:01.000081+0000","last_peered":"2024-03-24T01:26:01.000081+0000","last_clean":"2024-03-24T01:26:01.000081+0000","last_became_active":"2024-03-24T01:26:00.010223+0000","last_became_peered":"2024-03-24T01:26:00.010223+0000","last_unstale":"2024-03-24T01:26:01.000081+0000","last_undegraded":"2024-03-24T01:26:01.000081+0000","last_fullsized":"2024-03-24T01:26:01.000081+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:24:40.903708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221583+0000","last_change":"2024-03-24T01:26:00.012534+0000","last_active":"2024-03-24T01:26:01.221583+0000","last_peered":"2024-03-24T01:26:01.221583+0000","last_clean":"2024-03-24T01:26:01.221583+0000","last_became_active":"2024-03-24T01:26:00.011982+0000","last_became_peered":"2024-03-24T01:26:00.011982+0000","last_unstale":"2024-03-24T01:26:01.221583+0000","last_undegraded":"2024-03-24T01:26:01.221583+0000","last_fullsized":"2024-03-24T01:26:01.221583+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:30:44.565229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.002400+0000","last_change":"2024-03-24T01:25:59.997124+0000","last_active":"2024-03-24T01:26:01.002400+0000","last_peered":"2024-03-24T01:26:01.002400+0000","last_clean":"2024-03-24T01:26:01.002400+0000","last_became_active":"2024-03-24T01:25:59.996779+0000","last_became_peered":"2024-03-24T01:25:59.996779+0000","last_unstale":"2024-03-24T01:26:01.002400+0000","last_undegraded":"2024-03-24T01:26:01.002400+0000","last_fullsized":"2024-03-24T01:26:01.002400+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:20:43.942659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.012360+0000","last_change":"2024-03-24T01:26:00.016153+0000","last_active":"2024-03-24T01:26:01.012360+0000","last_peered":"2024-03-24T01:26:01.012360+0000","last_clean":"2024-03-24T01:26:01.012360+0000","last_became_active":"2024-03-24T01:26:00.015810+0000","last_became_peered":"2024-03-24T01:26:00.015810+0000","last_unstale":"2024-03-24T01:26:01.012360+0000","last_undegraded":"2024-03-24T01:26:01.012360+0000","last_fullsized":"2024-03-24T01:26:01.012360+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:42:02.641519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.001346+0000","last_change":"2024-03-24T01:26:00.003439+0000","last_active":"2024-03-24T01:26:01.001346+0000","last_peered":"2024-03-24T01:26:01.001346+0000","last_clean":"2024-03-24T01:26:01.001346+0000","last_became_active":"2024-03-24T01:26:00.001832+0000","last_became_peered":"2024-03-24T01:26:00.001832+0000","last_unstale":"2024-03-24T01:26:01.001346+0000","last_undegraded":"2024-03-24T01:26:01.001346+0000","last_fullsized":"2024-03-24T01:26:01.001346+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:08:58.631691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.000243+0000","last_change":"2024-03-24T01:25:59.995363+0000","last_active":"2024-03-24T01:26:01.000243+0000","last_peered":"2024-03-24T01:26:01.000243+0000","last_clean":"2024-03-24T01:26:01.000243+0000","last_became_active":"2024-03-24T01:25:59.995067+0000","last_became_peered":"2024-03-24T01:25:59.995067+0000","last_unstale":"2024-03-24T01:26:01.000243+0000","last_undegraded":"2024-03-24T01:26:01.000243+0000","last_fullsized":"2024-03-24T01:26:01.000243+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:36:04.311730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221948+0000","last_change":"2024-03-24T01:26:00.004303+0000","last_active":"2024-03-24T01:26:01.221948+0000","last_peered":"2024-03-24T01:26:01.221948+0000","last_clean":"2024-03-24T01:26:01.221948+0000","last_became_active":"2024-03-24T01:26:00.002367+0000","last_became_peered":"2024-03-24T01:26:00.002367+0000","last_unstale":"2024-03-24T01:26:01.221948+0000","last_undegraded":"2024-03-24T01:26:01.221948+0000","last_fullsized":"2024-03-24T01:26:01.221948+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:56:23.530821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221269+0000","last_change":"2024-03-24T01:25:59.999709+0000","last_active":"2024-03-24T01:26:01.221269+0000","last_peered":"2024-03-24T01:26:01.221269+0000","last_clean":"2024-03-24T01:26:01.221269+0000","last_became_active":"2024-03-24T01:25:59.999428+0000","last_became_peered":"2024-03-24T01:25:59.999428+0000","last_unstale":"2024-03-24T01:26:01.221269+0000","last_undegraded":"2024-03-24T01:26:01.221269+0000","last_fullsized":"2024-03-24T01:26:01.221269+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:39:38.186024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.222312+0000","last_change":"2024-03-24T01:26:00.008441+0000","last_active":"2024-03-24T01:26:01.222312+0000","last_peered":"2024-03-24T01:26:01.222312+0000","last_clean":"2024-03-24T01:26:01.222312+0000","last_became_active":"2024-03-24T01:26:00.008117+0000","last_became_peered":"2024-03-24T01:26:00.008117+0000","last_unstale":"2024-03-24T01:26:01.222312+0000","last_undegraded":"2024-03-24T01:26:01.222312+0000","last_fullsized":"2024-03-24T01:26:01.222312+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:46:09.862425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":18,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.999605+0000","last_change":"2024-03-24T01:25:58.996685+0000","last_active":"2024-03-24T01:26:00.999605+0000","last_peered":"2024-03-24T01:26:00.999605+0000","last_clean":"2024-03-24T01:26:00.999605+0000","last_became_active":"2024-03-24T01:25:58.996398+0000","last_became_peered":"2024-03-24T01:25:58.996398+0000","last_unstale":"2024-03-24T01:26:00.999605+0000","last_undegraded":"2024-03-24T01:26:00.999605+0000","last_fullsized":"2024-03-24T01:26:00.999605+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:45.605005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221978+0000","last_change":"2024-03-24T01:26:00.015135+0000","last_active":"2024-03-24T01:26:01.221978+0000","last_peered":"2024-03-24T01:26:01.221978+0000","last_clean":"2024-03-24T01:26:01.221978+0000","last_became_active":"2024-03-24T01:26:00.014848+0000","last_became_peered":"2024-03-24T01:26:00.014848+0000","last_unstale":"2024-03-24T01:26:01.221978+0000","last_undegraded":"2024-03-24T01:26:01.221978+0000","last_fullsized":"2024-03-24T01:26:01.221978+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:56:23.530821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.222294+0000","last_change":"2024-03-24T01:25:59.002453+0000","last_active":"2024-03-24T01:26:01.222294+0000","last_peered":"2024-03-24T01:26:01.222294+0000","last_clean":"2024-03-24T01:26:01.222294+0000","last_became_active":"2024-03-24T01:25:59.002155+0000","last_became_peered":"2024-03-24T01:25:59.002155+0000","last_unstale":"2024-03-24T01:26:01.222294+0000","last_undegraded":"2024-03-24T01:26:01.222294+0000","last_fullsized":"2024-03-24T01:26:01.222294+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:45.605005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.002182+0000","last_change":"2024-03-24T01:26:00.010489+0000","last_active":"2024-03-24T01:26:01.002182+0000","last_peered":"2024-03-24T01:26:01.002182+0000","last_clean":"2024-03-24T01:26:01.002182+0000","last_became_active":"2024-03-24T01:26:00.010187+0000","last_became_peered":"2024-03-24T01:26:00.010187+0000","last_unstale":"2024-03-24T01:26:01.002182+0000","last_undegraded":"2024-03-24T01:26:01.002182+0000","last_fullsized":"2024-03-24T01:26:01.002182+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:07:24.167208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.222196+0000","last_change":"2024-03-24T01:25:58.997334+0000","last_active":"2024-03-24T01:26:01.222196+0000","last_peered":"2024-03-24T01:26:01.222196+0000","last_clean":"2024-03-24T01:26:01.222196+0000","last_became_active":"2024-03-24T01:25:58.997033+0000","last_became_peered":"2024-03-24T01:25:58.997033+0000","last_unstale":"2024-03-24T01:26:01.222196+0000","last_undegraded":"2024-03-24T01:26:01.222196+0000","last_fullsized":"2024-03-24T01:26:01.222196+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:00:07.242997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.998005+0000","last_change":"2024-03-24T01:26:00.004685+0000","last_active":"2024-03-24T01:26:00.998005+0000","last_peered":"2024-03-24T01:26:00.998005+0000","last_clean":"2024-03-24T01:26:00.998005+0000","last_became_active":"2024-03-24T01:26:00.004430+0000","last_became_peered":"2024-03-24T01:26:00.004430+0000","last_unstale":"2024-03-24T01:26:00.998005+0000","last_undegraded":"2024-03-24T01:26:00.998005+0000","last_fullsized":"2024-03-24T01:26:00.998005+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:47:33.606184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.001541+0000","last_change":"2024-03-24T01:25:58.992317+0000","last_active":"2024-03-24T01:26:01.001541+0000","last_peered":"2024-03-24T01:26:01.001541+0000","last_clean":"2024-03-24T01:26:01.001541+0000","last_became_active":"2024-03-24T01:25:58.991652+0000","last_became_peered":"2024-03-24T01:25:58.991652+0000","last_unstale":"2024-03-24T01:26:01.001541+0000","last_undegraded":"2024-03-24T01:26:01.001541+0000","last_fullsized":"2024-03-24T01:26:01.001541+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:00:07.242997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.220739+0000","last_change":"2024-03-24T01:25:59.999595+0000","last_active":"2024-03-24T01:26:01.220739+0000","last_peered":"2024-03-24T01:26:01.220739+0000","last_clean":"2024-03-24T01:26:01.220739+0000","last_became_active":"2024-03-24T01:25:59.999225+0000","last_became_peered":"2024-03-24T01:25:59.999225+0000","last_unstale":"2024-03-24T01:26:01.220739+0000","last_undegraded":"2024-03-24T01:26:01.220739+0000","last_fullsized":"2024-03-24T01:26:01.220739+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:36:04.311730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.012655+0000","last_change":"2024-03-24T01:25:58.998337+0000","last_active":"2024-03-24T01:26:01.012655+0000","last_peered":"2024-03-24T01:26:01.012655+0000","last_clean":"2024-03-24T01:26:01.012655+0000","last_became_active":"2024-03-24T01:25:58.998056+0000","last_became_peered":"2024-03-24T01:25:58.998056+0000","last_unstale":"2024-03-24T01:26:01.012655+0000","last_undegraded":"2024-03-24T01:26:01.012655+0000","last_fullsized":"2024-03-24T01:26:01.012655+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:05:31.709112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.012492+0000","last_change":"2024-03-24T01:26:00.004616+0000","last_active":"2024-03-24T01:26:01.012492+0000","last_peered":"2024-03-24T01:26:01.012492+0000","last_clean":"2024-03-24T01:26:01.012492+0000","last_became_active":"2024-03-24T01:26:00.003030+0000","last_became_peered":"2024-03-24T01:26:00.003030+0000","last_unstale":"2024-03-24T01:26:01.012492+0000","last_undegraded":"2024-03-24T01:26:01.012492+0000","last_fullsized":"2024-03-24T01:26:01.012492+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:11:25.571750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221421+0000","last_change":"2024-03-24T01:25:59.006913+0000","last_active":"2024-03-24T01:26:01.221421+0000","last_peered":"2024-03-24T01:26:01.221421+0000","last_clean":"2024-03-24T01:26:01.221421+0000","last_became_active":"2024-03-24T01:25:59.006098+0000","last_became_peered":"2024-03-24T01:25:59.006098+0000","last_unstale":"2024-03-24T01:26:01.221421+0000","last_undegraded":"2024-03-24T01:26:01.221421+0000","last_fullsized":"2024-03-24T01:26:01.221421+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:44:12.069166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.001097+0000","last_change":"2024-03-24T01:26:00.006344+0000","last_active":"2024-03-24T01:26:01.001097+0000","last_peered":"2024-03-24T01:26:01.001097+0000","last_clean":"2024-03-24T01:26:01.001097+0000","last_became_active":"2024-03-24T01:26:00.005891+0000","last_became_peered":"2024-03-24T01:26:00.005891+0000","last_unstale":"2024-03-24T01:26:01.001097+0000","last_undegraded":"2024-03-24T01:26:01.001097+0000","last_fullsized":"2024-03-24T01:26:01.001097+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:30:44.565229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221290+0000","last_change":"2024-03-24T01:25:59.000318+0000","last_active":"2024-03-24T01:26:01.221290+0000","last_peered":"2024-03-24T01:26:01.221290+0000","last_clean":"2024-03-24T01:26:01.221290+0000","last_became_active":"2024-03-24T01:25:59.000085+0000","last_became_peered":"2024-03-24T01:25:59.000085+0000","last_unstale":"2024-03-24T01:26:01.221290+0000","last_undegraded":"2024-03-24T01:26:01.221290+0000","last_fullsized":"2024-03-24T01:26:01.221290+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:12:43.140922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.997472+0000","last_change":"2024-03-24T01:26:00.003084+0000","last_active":"2024-03-24T01:26:00.997472+0000","last_peered":"2024-03-24T01:26:00.997472+0000","last_clean":"2024-03-24T01:26:00.997472+0000","last_became_active":"2024-03-24T01:26:00.002750+0000","last_became_peered":"2024-03-24T01:26:00.002750+0000","last_unstale":"2024-03-24T01:26:00.997472+0000","last_undegraded":"2024-03-24T01:26:00.997472+0000","last_fullsized":"2024-03-24T01:26:00.997472+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:17:45.147642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.999120+0000","last_change":"2024-03-24T01:25:58.994521+0000","last_active":"2024-03-24T01:26:00.999120+0000","last_peered":"2024-03-24T01:26:00.999120+0000","last_clean":"2024-03-24T01:26:00.999120+0000","last_became_active":"2024-03-24T01:25:58.994215+0000","last_became_peered":"2024-03-24T01:25:58.994215+0000","last_unstale":"2024-03-24T01:26:00.999120+0000","last_undegraded":"2024-03-24T01:26:00.999120+0000","last_fullsized":"2024-03-24T01:26:00.999120+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:12:43.140922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.007272+0000","last_change":"2024-03-24T01:26:00.012399+0000","last_active":"2024-03-24T01:26:01.007272+0000","last_peered":"2024-03-24T01:26:01.007272+0000","last_clean":"2024-03-24T01:26:01.007272+0000","last_became_active":"2024-03-24T01:26:00.008358+0000","last_became_peered":"2024-03-24T01:26:00.008358+0000","last_unstale":"2024-03-24T01:26:01.007272+0000","last_undegraded":"2024-03-24T01:26:01.007272+0000","last_fullsized":"2024-03-24T01:26:01.007272+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:00:43.649482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.998097+0000","last_change":"2024-03-24T01:25:58.995322+0000","last_active":"2024-03-24T01:26:00.998097+0000","last_peered":"2024-03-24T01:26:00.998097+0000","last_clean":"2024-03-24T01:26:00.998097+0000","last_became_active":"2024-03-24T01:25:58.994847+0000","last_became_peered":"2024-03-24T01:25:58.994847+0000","last_unstale":"2024-03-24T01:26:00.998097+0000","last_undegraded":"2024-03-24T01:26:00.998097+0000","last_fullsized":"2024-03-24T01:26:00.998097+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:08:57.617873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.222562+0000","last_change":"2024-03-24T01:25:59.997541+0000","last_active":"2024-03-24T01:26:01.222562+0000","last_peered":"2024-03-24T01:26:01.222562+0000","last_clean":"2024-03-24T01:26:01.222562+0000","last_became_active":"2024-03-24T01:25:59.992800+0000","last_became_peered":"2024-03-24T01:25:59.992800+0000","last_unstale":"2024-03-24T01:26:01.222562+0000","last_undegraded":"2024-03-24T01:26:01.222562+0000","last_fullsized":"2024-03-24T01:26:01.222562+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:42:02.641519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.013224+0000","last_change":"2024-03-24T01:25:58.998109+0000","last_active":"2024-03-24T01:26:01.013224+0000","last_peered":"2024-03-24T01:26:01.013224+0000","last_clean":"2024-03-24T01:26:01.013224+0000","last_became_active":"2024-03-24T01:25:58.997791+0000","last_became_peered":"2024-03-24T01:25:58.997791+0000","last_unstale":"2024-03-24T01:26:01.013224+0000","last_undegraded":"2024-03-24T01:26:01.013224+0000","last_fullsized":"2024-03-24T01:26:01.013224+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:19:04.197281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.222240+0000","last_change":"2024-03-24T01:26:00.005147+0000","last_active":"2024-03-24T01:26:01.222240+0000","last_peered":"2024-03-24T01:26:01.222240+0000","last_clean":"2024-03-24T01:26:01.222240+0000","last_became_active":"2024-03-24T01:26:00.004799+0000","last_became_peered":"2024-03-24T01:26:00.004799+0000","last_unstale":"2024-03-24T01:26:01.222240+0000","last_undegraded":"2024-03-24T01:26:01.222240+0000","last_fullsized":"2024-03-24T01:26:01.222240+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:02:08.397348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":17,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.013185+0000","last_change":"2024-03-24T01:25:58.999040+0000","last_active":"2024-03-24T01:26:01.013185+0000","last_peered":"2024-03-24T01:26:01.013185+0000","last_clean":"2024-03-24T01:26:01.013185+0000","last_became_active":"2024-03-24T01:25:58.998593+0000","last_became_peered":"2024-03-24T01:25:58.998593+0000","last_unstale":"2024-03-24T01:26:01.013185+0000","last_undegraded":"2024-03-24T01:26:01.013185+0000","last_fullsized":"2024-03-24T01:26:01.013185+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:00:07.242997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.001205+0000","last_change":"2024-03-24T01:26:00.008616+0000","last_active":"2024-03-24T01:26:01.001205+0000","last_peered":"2024-03-24T01:26:01.001205+0000","last_clean":"2024-03-24T01:26:01.001205+0000","last_became_active":"2024-03-24T01:26:00.008345+0000","last_became_peered":"2024-03-24T01:26:00.008345+0000","last_unstale":"2024-03-24T01:26:01.001205+0000","last_undegraded":"2024-03-24T01:26:01.001205+0000","last_fullsized":"2024-03-24T01:26:01.001205+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:56:23.530821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.998078+0000","last_change":"2024-03-24T01:25:59.002378+0000","last_active":"2024-03-24T01:26:00.998078+0000","last_peered":"2024-03-24T01:26:00.998078+0000","last_clean":"2024-03-24T01:26:00.998078+0000","last_became_active":"2024-03-24T01:25:59.002071+0000","last_became_peered":"2024-03-24T01:25:59.002071+0000","last_unstale":"2024-03-24T01:26:00.998078+0000","last_undegraded":"2024-03-24T01:26:00.998078+0000","last_fullsized":"2024-03-24T01:26:00.998078+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:20:42.928840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.220866+0000","last_change":"2024-03-24T01:26:00.006589+0000","last_active":"2024-03-24T01:26:01.220866+0000","last_peered":"2024-03-24T01:26:01.220866+0000","last_clean":"2024-03-24T01:26:01.220866+0000","last_became_active":"2024-03-24T01:26:00.006332+0000","last_became_peered":"2024-03-24T01:26:00.006332+0000","last_unstale":"2024-03-24T01:26:01.220866+0000","last_undegraded":"2024-03-24T01:26:01.220866+0000","last_fullsized":"2024-03-24T01:26:01.220866+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:08:58.631691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.997732+0000","last_change":"2024-03-24T01:25:58.987270+0000","last_active":"2024-03-24T01:26:00.997732+0000","last_peered":"2024-03-24T01:26:00.997732+0000","last_clean":"2024-03-24T01:26:00.997732+0000","last_became_active":"2024-03-24T01:25:58.986953+0000","last_became_peered":"2024-03-24T01:25:58.986953+0000","last_unstale":"2024-03-24T01:26:00.997732+0000","last_undegraded":"2024-03-24T01:26:00.997732+0000","last_fullsized":"2024-03-24T01:26:00.997732+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:36:03.297911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.997417+0000","last_change":"2024-03-24T01:25:59.999556+0000","last_active":"2024-03-24T01:26:00.997417+0000","last_peered":"2024-03-24T01:26:00.997417+0000","last_clean":"2024-03-24T01:26:00.997417+0000","last_became_active":"2024-03-24T01:25:59.999261+0000","last_became_peered":"2024-03-24T01:25:59.999261+0000","last_unstale":"2024-03-24T01:26:00.997417+0000","last_undegraded":"2024-03-24T01:26:00.997417+0000","last_fullsized":"2024-03-24T01:26:00.997417+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:01:28.134219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221624+0000","last_change":"2024-03-24T01:25:59.000275+0000","last_active":"2024-03-24T01:26:01.221624+0000","last_peered":"2024-03-24T01:26:01.221624+0000","last_clean":"2024-03-24T01:26:01.221624+0000","last_became_active":"2024-03-24T01:25:58.999948+0000","last_became_peered":"2024-03-24T01:25:58.999948+0000","last_unstale":"2024-03-24T01:26:01.221624+0000","last_undegraded":"2024-03-24T01:26:01.221624+0000","last_fullsized":"2024-03-24T01:26:01.221624+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:55.200533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221592+0000","last_change":"2024-03-24T01:25:59.995825+0000","last_active":"2024-03-24T01:26:01.221592+0000","last_peered":"2024-03-24T01:26:01.221592+0000","last_clean":"2024-03-24T01:26:01.221592+0000","last_became_active":"2024-03-24T01:25:59.995551+0000","last_became_peered":"2024-03-24T01:25:59.995551+0000","last_unstale":"2024-03-24T01:26:01.221592+0000","last_undegraded":"2024-03-24T01:26:01.221592+0000","last_fullsized":"2024-03-24T01:26:01.221592+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:39:38.186024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.000968+0000","last_change":"2024-03-24T01:25:58.991563+0000","last_active":"2024-03-24T01:26:01.000968+0000","last_peered":"2024-03-24T01:26:01.000968+0000","last_clean":"2024-03-24T01:26:01.000968+0000","last_became_active":"2024-03-24T01:25:58.990946+0000","last_became_peered":"2024-03-24T01:25:58.990946+0000","last_unstale":"2024-03-24T01:26:01.000968+0000","last_undegraded":"2024-03-24T01:26:01.000968+0000","last_fullsized":"2024-03-24T01:26:01.000968+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:08:38.108175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.997634+0000","last_change":"2024-03-24T01:26:00.019807+0000","last_active":"2024-03-24T01:26:00.997634+0000","last_peered":"2024-03-24T01:26:00.997634+0000","last_clean":"2024-03-24T01:26:00.997634+0000","last_became_active":"2024-03-24T01:26:00.019479+0000","last_became_peered":"2024-03-24T01:26:00.019479+0000","last_unstale":"2024-03-24T01:26:00.997634+0000","last_undegraded":"2024-03-24T01:26:00.997634+0000","last_fullsized":"2024-03-24T01:26:00.997634+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:30:21.708639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":22,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221368+0000","last_change":"2024-03-24T01:25:59.006569+0000","last_active":"2024-03-24T01:26:01.221368+0000","last_peered":"2024-03-24T01:26:01.221368+0000","last_clean":"2024-03-24T01:26:01.221368+0000","last_became_active":"2024-03-24T01:25:59.006280+0000","last_became_peered":"2024-03-24T01:25:59.006280+0000","last_unstale":"2024-03-24T01:26:01.221368+0000","last_undegraded":"2024-03-24T01:26:01.221368+0000","last_fullsized":"2024-03-24T01:26:01.221368+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:45.605005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.997931+0000","last_change":"2024-03-24T01:26:00.005482+0000","last_active":"2024-03-24T01:26:00.997931+0000","last_peered":"2024-03-24T01:26:00.997931+0000","last_clean":"2024-03-24T01:26:00.997931+0000","last_became_active":"2024-03-24T01:26:00.005203+0000","last_became_peered":"2024-03-24T01:26:00.005203+0000","last_unstale":"2024-03-24T01:26:00.997931+0000","last_undegraded":"2024-03-24T01:26:00.997931+0000","last_fullsized":"2024-03-24T01:26:00.997931+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:54:09.379313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.013043+0000","last_change":"2024-03-24T01:25:59.003724+0000","last_active":"2024-03-24T01:26:01.013043+0000","last_peered":"2024-03-24T01:26:01.013043+0000","last_clean":"2024-03-24T01:26:01.013043+0000","last_became_active":"2024-03-24T01:25:59.003410+0000","last_became_peered":"2024-03-24T01:25:59.003410+0000","last_unstale":"2024-03-24T01:26:01.013043+0000","last_undegraded":"2024-03-24T01:26:01.013043+0000","last_fullsized":"2024-03-24T01:26:01.013043+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:12:43.140922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.013119+0000","last_change":"2024-03-24T01:26:00.006656+0000","last_active":"2024-03-24T01:26:01.013119+0000","last_peered":"2024-03-24T01:26:01.013119+0000","last_clean":"2024-03-24T01:26:01.013119+0000","last_became_active":"2024-03-24T01:26:00.006301+0000","last_became_peered":"2024-03-24T01:26:00.006301+0000","last_unstale":"2024-03-24T01:26:01.013119+0000","last_undegraded":"2024-03-24T01:26:01.013119+0000","last_fullsized":"2024-03-24T01:26:01.013119+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:01:28.134219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.002273+0000","last_change":"2024-03-24T01:25:58.999764+0000","last_active":"2024-03-24T01:26:01.002273+0000","last_peered":"2024-03-24T01:26:01.002273+0000","last_clean":"2024-03-24T01:26:01.002273+0000","last_became_active":"2024-03-24T01:25:58.999490+0000","last_became_peered":"2024-03-24T01:25:58.999490+0000","last_unstale":"2024-03-24T01:26:01.002273+0000","last_undegraded":"2024-03-24T01:26:01.002273+0000","last_fullsized":"2024-03-24T01:26:01.002273+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:19:04.197281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.001941+0000","last_change":"2024-03-24T01:26:00.016951+0000","last_active":"2024-03-24T01:26:01.001941+0000","last_peered":"2024-03-24T01:26:01.001941+0000","last_clean":"2024-03-24T01:26:01.001941+0000","last_became_active":"2024-03-24T01:26:00.016500+0000","last_became_peered":"2024-03-24T01:26:00.016500+0000","last_unstale":"2024-03-24T01:26:01.001941+0000","last_undegraded":"2024-03-24T01:26:01.001941+0000","last_fullsized":"2024-03-24T01:26:01.001941+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:46:09.862425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.007538+0000","last_change":"2024-03-24T01:25:59.003238+0000","last_active":"2024-03-24T01:26:01.007538+0000","last_peered":"2024-03-24T01:26:01.007538+0000","last_clean":"2024-03-24T01:26:01.007538+0000","last_became_active":"2024-03-24T01:25:59.002905+0000","last_became_peered":"2024-03-24T01:25:59.002905+0000","last_unstale":"2024-03-24T01:26:01.007538+0000","last_undegraded":"2024-03-24T01:26:01.007538+0000","last_fullsized":"2024-03-24T01:26:01.007538+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:43.279129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.007522+0000","last_change":"2024-03-24T01:26:00.008391+0000","last_active":"2024-03-24T01:26:01.007522+0000","last_peered":"2024-03-24T01:26:01.007522+0000","last_clean":"2024-03-24T01:26:01.007522+0000","last_became_active":"2024-03-24T01:26:00.007343+0000","last_became_peered":"2024-03-24T01:26:00.007343+0000","last_unstale":"2024-03-24T01:26:01.007522+0000","last_undegraded":"2024-03-24T01:26:01.007522+0000","last_fullsized":"2024-03-24T01:26:01.007522+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:53:50.285242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.222165+0000","last_change":"2024-03-24T01:25:58.994043+0000","last_active":"2024-03-24T01:26:01.222165+0000","last_peered":"2024-03-24T01:26:01.222165+0000","last_clean":"2024-03-24T01:26:01.222165+0000","last_became_active":"2024-03-24T01:25:58.993746+0000","last_became_peered":"2024-03-24T01:25:58.993746+0000","last_unstale":"2024-03-24T01:26:01.222165+0000","last_undegraded":"2024-03-24T01:26:01.222165+0000","last_fullsized":"2024-03-24T01:26:01.222165+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:19:04.197281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221838+0000","last_change":"2024-03-24T01:26:00.004263+0000","last_active":"2024-03-24T01:26:01.221838+0000","last_peered":"2024-03-24T01:26:01.221838+0000","last_clean":"2024-03-24T01:26:01.221838+0000","last_became_active":"2024-03-24T01:26:00.000719+0000","last_became_peered":"2024-03-24T01:26:00.000719+0000","last_unstale":"2024-03-24T01:26:01.221838+0000","last_undegraded":"2024-03-24T01:26:01.221838+0000","last_fullsized":"2024-03-24T01:26:01.221838+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:16:28.904097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.007123+0000","last_change":"2024-03-24T01:25:58.988228+0000","last_active":"2024-03-24T01:26:01.007123+0000","last_peered":"2024-03-24T01:26:01.007123+0000","last_clean":"2024-03-24T01:26:01.007123+0000","last_became_active":"2024-03-24T01:25:58.987941+0000","last_became_peered":"2024-03-24T01:25:58.987941+0000","last_unstale":"2024-03-24T01:26:01.007123+0000","last_undegraded":"2024-03-24T01:26:01.007123+0000","last_fullsized":"2024-03-24T01:26:01.007123+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:33:28.206717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221879+0000","last_change":"2024-03-24T01:26:00.000558+0000","last_active":"2024-03-24T01:26:01.221879+0000","last_peered":"2024-03-24T01:26:01.221879+0000","last_clean":"2024-03-24T01:26:01.221879+0000","last_became_active":"2024-03-24T01:26:00.000273+0000","last_became_peered":"2024-03-24T01:26:00.000273+0000","last_unstale":"2024-03-24T01:26:01.221879+0000","last_undegraded":"2024-03-24T01:26:01.221879+0000","last_fullsized":"2024-03-24T01:26:01.221879+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:16:28.904097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.006847+0000","last_change":"2024-03-24T01:25:58.991840+0000","last_active":"2024-03-24T01:26:01.006847+0000","last_peered":"2024-03-24T01:26:01.006847+0000","last_clean":"2024-03-24T01:26:01.006847+0000","last_became_active":"2024-03-24T01:25:58.991522+0000","last_became_peered":"2024-03-24T01:25:58.991522+0000","last_unstale":"2024-03-24T01:26:01.006847+0000","last_undegraded":"2024-03-24T01:26:01.006847+0000","last_fullsized":"2024-03-24T01:26:01.006847+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:29:31.808606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.011986+0000","last_change":"2024-03-24T01:26:00.005519+0000","last_active":"2024-03-24T01:26:01.011986+0000","last_peered":"2024-03-24T01:26:01.011986+0000","last_clean":"2024-03-24T01:26:01.011986+0000","last_became_active":"2024-03-24T01:26:00.005132+0000","last_became_peered":"2024-03-24T01:26:00.005132+0000","last_unstale":"2024-03-24T01:26:01.011986+0000","last_undegraded":"2024-03-24T01:26:01.011986+0000","last_fullsized":"2024-03-24T01:26:01.011986+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:02:08.397348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.222253+0000","last_change":"2024-03-24T01:25:59.006226+0000","last_active":"2024-03-24T01:26:01.222253+0000","last_peered":"2024-03-24T01:26:01.222253+0000","last_clean":"2024-03-24T01:26:01.222253+0000","last_became_active":"2024-03-24T01:25:59.005739+0000","last_became_peered":"2024-03-24T01:25:59.005739+0000","last_unstale":"2024-03-24T01:26:01.222253+0000","last_undegraded":"2024-03-24T01:26:01.222253+0000","last_fullsized":"2024-03-24T01:26:01.222253+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:08:38.108175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.997650+0000","last_change":"2024-03-24T01:26:00.016287+0000","last_active":"2024-03-24T01:26:00.997650+0000","last_peered":"2024-03-24T01:26:00.997650+0000","last_clean":"2024-03-24T01:26:00.997650+0000","last_became_active":"2024-03-24T01:26:00.015993+0000","last_became_peered":"2024-03-24T01:26:00.015993+0000","last_unstale":"2024-03-24T01:26:00.997650+0000","last_undegraded":"2024-03-24T01:26:00.997650+0000","last_fullsized":"2024-03-24T01:26:00.997650+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:29:19.471350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.006694+0000","last_change":"2024-03-24T01:25:58.988520+0000","last_active":"2024-03-24T01:26:01.006694+0000","last_peered":"2024-03-24T01:26:01.006694+0000","last_clean":"2024-03-24T01:26:01.006694+0000","last_became_active":"2024-03-24T01:25:58.988149+0000","last_became_peered":"2024-03-24T01:25:58.988149+0000","last_unstale":"2024-03-24T01:26:01.006694+0000","last_undegraded":"2024-03-24T01:26:01.006694+0000","last_fullsized":"2024-03-24T01:26:01.006694+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:22:00.973282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.222111+0000","last_change":"2024-03-24T01:26:00.005748+0000","last_active":"2024-03-24T01:26:01.222111+0000","last_peered":"2024-03-24T01:26:01.222111+0000","last_clean":"2024-03-24T01:26:01.222111+0000","last_became_active":"2024-03-24T01:26:00.005478+0000","last_became_peered":"2024-03-24T01:26:00.005478+0000","last_unstale":"2024-03-24T01:26:01.222111+0000","last_undegraded":"2024-03-24T01:26:01.222111+0000","last_fullsized":"2024-03-24T01:26:01.222111+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:02:08.397348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.013306+0000","last_change":"2024-03-24T01:25:59.005278+0000","last_active":"2024-03-24T01:26:01.013306+0000","last_peered":"2024-03-24T01:26:01.013306+0000","last_clean":"2024-03-24T01:26:01.013306+0000","last_became_active":"2024-03-24T01:25:59.004896+0000","last_became_peered":"2024-03-24T01:25:59.004896+0000","last_unstale":"2024-03-24T01:26:01.013306+0000","last_undegraded":"2024-03-24T01:26:01.013306+0000","last_fullsized":"2024-03-24T01:26:01.013306+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:24:39.889890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.222522+0000","last_change":"2024-03-24T01:26:00.002565+0000","last_active":"2024-03-24T01:26:01.222522+0000","last_peered":"2024-03-24T01:26:01.222522+0000","last_clean":"2024-03-24T01:26:01.222522+0000","last_became_active":"2024-03-24T01:25:59.993312+0000","last_became_peered":"2024-03-24T01:25:59.993312+0000","last_unstale":"2024-03-24T01:26:01.222522+0000","last_undegraded":"2024-03-24T01:26:01.222522+0000","last_fullsized":"2024-03-24T01:26:01.222522+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:07:24.167208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"39'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.007168+0000","last_change":"2024-03-24T01:25:59.004018+0000","last_active":"2024-03-24T01:26:01.007168+0000","last_peered":"2024-03-24T01:26:01.007168+0000","last_clean":"2024-03-24T01:26:01.007168+0000","last_became_active":"2024-03-24T01:25:59.003661+0000","last_became_peered":"2024-03-24T01:25:59.003661+0000","last_unstale":"2024-03-24T01:26:01.007168+0000","last_undegraded":"2024-03-24T01:26:01.007168+0000","last_fullsized":"2024-03-24T01:26:01.007168+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:35:58.556882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.997888+0000","last_change":"2024-03-24T01:26:00.016797+0000","last_active":"2024-03-24T01:26:00.997888+0000","last_peered":"2024-03-24T01:26:00.997888+0000","last_clean":"2024-03-24T01:26:00.997888+0000","last_became_active":"2024-03-24T01:26:00.016538+0000","last_became_peered":"2024-03-24T01:26:00.016538+0000","last_unstale":"2024-03-24T01:26:00.997888+0000","last_undegraded":"2024-03-24T01:26:00.997888+0000","last_fullsized":"2024-03-24T01:26:00.997888+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:00:25.321525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.222080+0000","last_change":"2024-03-24T01:25:58.995201+0000","last_active":"2024-03-24T01:26:01.222080+0000","last_peered":"2024-03-24T01:26:01.222080+0000","last_clean":"2024-03-24T01:26:01.222080+0000","last_became_active":"2024-03-24T01:25:58.994890+0000","last_became_peered":"2024-03-24T01:25:58.994890+0000","last_unstale":"2024-03-24T01:26:01.222080+0000","last_undegraded":"2024-03-24T01:26:01.222080+0000","last_fullsized":"2024-03-24T01:26:01.222080+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:08:38.108175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221007+0000","last_change":"2024-03-24T01:26:00.012718+0000","last_active":"2024-03-24T01:26:01.221007+0000","last_peered":"2024-03-24T01:26:01.221007+0000","last_clean":"2024-03-24T01:26:01.221007+0000","last_became_active":"2024-03-24T01:26:00.007605+0000","last_became_peered":"2024-03-24T01:26:00.007605+0000","last_unstale":"2024-03-24T01:26:01.221007+0000","last_undegraded":"2024-03-24T01:26:01.221007+0000","last_fullsized":"2024-03-24T01:26:01.221007+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:20:43.942659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221587+0000","last_change":"2024-03-24T01:25:58.990591+0000","last_active":"2024-03-24T01:26:01.221587+0000","last_peered":"2024-03-24T01:26:01.221587+0000","last_clean":"2024-03-24T01:26:01.221587+0000","last_became_active":"2024-03-24T01:25:58.990269+0000","last_became_peered":"2024-03-24T01:25:58.990269+0000","last_unstale":"2024-03-24T01:26:01.221587+0000","last_undegraded":"2024-03-24T01:26:01.221587+0000","last_fullsized":"2024-03-24T01:26:01.221587+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:44:12.069166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221167+0000","last_change":"2024-03-24T01:26:00.003739+0000","last_active":"2024-03-24T01:26:01.221167+0000","last_peered":"2024-03-24T01:26:01.221167+0000","last_clean":"2024-03-24T01:26:01.221167+0000","last_became_active":"2024-03-24T01:26:00.003484+0000","last_became_peered":"2024-03-24T01:26:00.003484+0000","last_unstale":"2024-03-24T01:26:01.221167+0000","last_undegraded":"2024-03-24T01:26:01.221167+0000","last_fullsized":"2024-03-24T01:26:01.221167+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:20:43.942659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.000473+0000","last_change":"2024-03-24T01:25:58.988526+0000","last_active":"2024-03-24T01:26:01.000473+0000","last_peered":"2024-03-24T01:26:01.000473+0000","last_clean":"2024-03-24T01:26:01.000473+0000","last_became_active":"2024-03-24T01:25:58.987932+0000","last_became_peered":"2024-03-24T01:25:58.987932+0000","last_unstale":"2024-03-24T01:26:01.000473+0000","last_undegraded":"2024-03-24T01:26:01.000473+0000","last_fullsized":"2024-03-24T01:26:01.000473+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:44:12.069166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.000093+0000","last_change":"2024-03-24T01:26:00.020959+0000","last_active":"2024-03-24T01:26:01.000093+0000","last_peered":"2024-03-24T01:26:01.000093+0000","last_clean":"2024-03-24T01:26:01.000093+0000","last_became_active":"2024-03-24T01:26:00.020670+0000","last_became_peered":"2024-03-24T01:26:00.020670+0000","last_unstale":"2024-03-24T01:26:01.000093+0000","last_undegraded":"2024-03-24T01:26:01.000093+0000","last_fullsized":"2024-03-24T01:26:01.000093+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:39:38.186024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.000331+0000","last_change":"2024-03-24T01:25:58.992830+0000","last_active":"2024-03-24T01:26:01.000331+0000","last_peered":"2024-03-24T01:26:01.000331+0000","last_clean":"2024-03-24T01:26:01.000331+0000","last_became_active":"2024-03-24T01:25:58.991569+0000","last_became_peered":"2024-03-24T01:25:58.991569+0000","last_unstale":"2024-03-24T01:26:01.000331+0000","last_undegraded":"2024-03-24T01:26:01.000331+0000","last_fullsized":"2024-03-24T01:26:01.000331+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:55.200533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221854+0000","last_change":"2024-03-24T01:26:00.001774+0000","last_active":"2024-03-24T01:26:01.221854+0000","last_peered":"2024-03-24T01:26:01.221854+0000","last_clean":"2024-03-24T01:26:01.221854+0000","last_became_active":"2024-03-24T01:26:00.001470+0000","last_became_peered":"2024-03-24T01:26:00.001470+0000","last_unstale":"2024-03-24T01:26:01.221854+0000","last_undegraded":"2024-03-24T01:26:01.221854+0000","last_fullsized":"2024-03-24T01:26:01.221854+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:30:44.565229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.997557+0000","last_change":"2024-03-24T01:25:58.996128+0000","last_active":"2024-03-24T01:26:00.997557+0000","last_peered":"2024-03-24T01:26:00.997557+0000","last_clean":"2024-03-24T01:26:00.997557+0000","last_became_active":"2024-03-24T01:25:58.994831+0000","last_became_peered":"2024-03-24T01:25:58.994831+0000","last_unstale":"2024-03-24T01:26:00.997557+0000","last_undegraded":"2024-03-24T01:26:00.997557+0000","last_fullsized":"2024-03-24T01:26:00.997557+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:24:39.889890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.012865+0000","last_change":"2024-03-24T01:26:00.018659+0000","last_active":"2024-03-24T01:26:01.012865+0000","last_peered":"2024-03-24T01:26:01.012865+0000","last_clean":"2024-03-24T01:26:01.012865+0000","last_became_active":"2024-03-24T01:26:00.018365+0000","last_became_peered":"2024-03-24T01:26:00.018365+0000","last_unstale":"2024-03-24T01:26:01.012865+0000","last_undegraded":"2024-03-24T01:26:01.012865+0000","last_fullsized":"2024-03-24T01:26:01.012865+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:43:00.446290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.222544+0000","last_change":"2024-03-24T01:25:59.008252+0000","last_active":"2024-03-24T01:26:01.222544+0000","last_peered":"2024-03-24T01:26:01.222544+0000","last_clean":"2024-03-24T01:26:01.222544+0000","last_became_active":"2024-03-24T01:25:59.007969+0000","last_became_peered":"2024-03-24T01:25:59.007969+0000","last_unstale":"2024-03-24T01:26:01.222544+0000","last_undegraded":"2024-03-24T01:26:01.222544+0000","last_fullsized":"2024-03-24T01:26:01.222544+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:28:18.909603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":205,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221970+0000","last_change":"2024-03-24T01:24:01.545855+0000","last_active":"2024-03-24T01:26:01.221970+0000","last_peered":"2024-03-24T01:26:01.221970+0000","last_clean":"2024-03-24T01:26:01.221970+0000","last_became_active":"2024-03-24T01:24:01.543811+0000","last_became_peered":"2024-03-24T01:24:01.543811+0000","last_unstale":"2024-03-24T01:26:01.221970+0000","last_undegraded":"2024-03-24T01:26:01.221970+0000","last_fullsized":"2024-03-24T01:26:01.221970+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_clean_scrub_stamp":"2024-03-24T01:23:59.861564+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:03.682631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574876,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28060,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343780,"statfs":{"total":96636764160,"available":96608030720,"internally_reserved":0,"allocated":1060864,"data_stored":789377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574876,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27464,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344376,"statfs":{"total":96636764160,"available":96608641024,"internally_reserved":0,"allocated":454656,"data_stored":197909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574877,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":458752,"data_stored":197439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574876,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":458752,"data_stored":197931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27476,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":466944,"data_stored":198973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574877,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":454656,"data_stored":197397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574876,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27464,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344376,"statfs":{"total":96636764160,"available":96608641024,"internally_reserved":0,"allocated":454656,"data_stored":197417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574877,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343796,"statfs":{"total":96636764160,"available":96608047104,"internally_reserved":0,"allocated":1048576,"data_stored":787729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T01:26:09.495 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T01:26:09.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.709+0000 7f4c9e000640 1 Processor -- start 2024-03-24T01:26:09.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.709+0000 7f4c9e000640 1 -- start start 2024-03-24T01:26:09.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.711+0000 7f4c9e000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4c980ce390 0x7f4c980b4580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:09.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.711+0000 7f4c9e000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c9815a660 con 0x7f4c9806f790 2024-03-24T01:26:09.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.711+0000 7f4c9e000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c9806dac0 con 0x7f4c9815a280 2024-03-24T01:26:09.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.711+0000 7f4c9e000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c9806dda0 con 0x7f4c980ce390 2024-03-24T01:26:09.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.711+0000 7f4c96a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4c9806f790 0x7f4c980b3e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50118/0 (socket says 172.21.15.90:50118) 2024-03-24T01:26:09.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.711+0000 7f4c96a00640 1 -- 172.21.15.90:0/3593350543 learned_addr learned my addr 172.21.15.90:0/3593350543 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:26:09.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.711+0000 7f4c97e00640 1 --2- 172.21.15.90:0/3593350543 >> v2:172.21.15.90:3300/0 conn(0x7f4c980ce390 0x7f4c980b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:09.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.712+0000 7f4c96000640 1 -- 172.21.15.90:0/3593350543 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1833484078 0 0) 0x7f4c88001110 con 0x7f4c9806f790 2024-03-24T01:26:09.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.712+0000 7f4c96000640 1 -- 172.21.15.90:0/3593350543 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c7c003520 con 0x7f4c9806f790 2024-03-24T01:26:09.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.712+0000 7f4c97e00640 1 -- 172.21.15.90:0/3593350543 >> v1:172.21.15.120:6789/0 conn(0x7f4c9815a280 legacy=0x7f4c9806f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:09.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.712+0000 7f4c96000640 1 -- 172.21.15.90:0/3593350543 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2009435887 0 0) 0x7f4c7c003520 con 0x7f4c9806f790 2024-03-24T01:26:09.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.712+0000 7f4c97e00640 1 -- 172.21.15.90:0/3593350543 >> v1:172.21.15.90:6789/0 conn(0x7f4c9806f790 legacy=0x7f4c980b3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:09.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.712+0000 7f4c97e00640 1 -- 172.21.15.90:0/3593350543 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c9806e080 con 0x7f4c980ce390 2024-03-24T01:26:09.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.713+0000 7f4c97e00640 1 --2- 172.21.15.90:0/3593350543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c980ce390 0x7f4c980b4580 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f4c90009d90 tx=0x7f4c9002bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ca9480328c95d09 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:09.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.714+0000 7f4c96000640 1 -- 172.21.15.90:0/3593350543 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c90039070 con 0x7f4c980ce390 2024-03-24T01:26:09.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.714+0000 7f4c96000640 1 -- 172.21.15.90:0/3593350543 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c9002c8b0 con 0x7f4c980ce390 2024-03-24T01:26:09.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.714+0000 7f4c96000640 1 -- 172.21.15.90:0/3593350543 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c88001840 con 0x7f4c980ce390 2024-03-24T01:26:09.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.714+0000 7f4c9e000640 1 -- 172.21.15.90:0/3593350543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c980ce390 msgr2=0x7f4c980b4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:09.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.714+0000 7f4c9e000640 1 --2- 172.21.15.90:0/3593350543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c980ce390 0x7f4c980b4580 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f4c90009d90 tx=0x7f4c9002bd80 comp rx=0 tx=0).stop 2024-03-24T01:26:09.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.714+0000 7f4c9e000640 1 -- 172.21.15.90:0/3593350543 shutdown_connections 2024-03-24T01:26:09.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.714+0000 7f4c9e000640 1 --2- 172.21.15.90:0/3593350543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c980ce390 0x7f4c980b4580 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:09.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.714+0000 7f4c9e000640 1 -- 172.21.15.90:0/3593350543 >> 172.21.15.90:0/3593350543 conn(0x7f4c980b1770 msgr2=0x7f4c980b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:09.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.714+0000 7f4c9e000640 1 -- 172.21.15.90:0/3593350543 shutdown_connections 2024-03-24T01:26:09.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.714+0000 7f4c9e000640 1 -- 172.21.15.90:0/3593350543 wait complete. 2024-03-24T01:26:09.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.716+0000 7f4c9e000640 1 Processor -- start 2024-03-24T01:26:09.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.716+0000 7f4c9e000640 1 -- start start 2024-03-24T01:26:09.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.717+0000 7f4c9e000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c9806f790 0x7f4c9814b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:09.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.717+0000 7f4c9e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c980ce390 0x7f4c980b0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:09.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.717+0000 7f4c9e000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c9815a280 0x7f4c98147230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:09.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.717+0000 7f4c9e000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9806b0c0 con 0x7f4c980ce390 2024-03-24T01:26:09.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.717+0000 7f4c9e000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9806e640 con 0x7f4c9806f790 2024-03-24T01:26:09.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.717+0000 7f4c9e000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9806b240 con 0x7f4c9815a280 2024-03-24T01:26:09.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.717+0000 7f4c96a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c980ce390 0x7f4c980b0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:09.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.717+0000 7f4c96a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c980ce390 0x7f4c980b0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60110/0 (socket says 172.21.15.90:60110) 2024-03-24T01:26:09.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.717+0000 7f4c96a00640 1 -- 172.21.15.90:0/2678254382 learned_addr learned my addr 172.21.15.90:0/2678254382 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:09.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.718+0000 7f4c96a00640 1 -- 172.21.15.90:0/2678254382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c9815a280 msgr2=0x7f4c98147230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:09.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.718+0000 7f4c96a00640 1 --2- 172.21.15.90:0/2678254382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c9815a280 0x7f4c98147230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:09.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.718+0000 7f4c96a00640 1 -- 172.21.15.90:0/2678254382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c9806f790 msgr2=0x7f4c9814b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:09.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.718+0000 7f4c96a00640 1 --2- 172.21.15.90:0/2678254382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c9806f790 0x7f4c9814b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:09.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.718+0000 7f4c96a00640 1 -- 172.21.15.90:0/2678254382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c9814d940 con 0x7f4c980ce390 2024-03-24T01:26:09.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.718+0000 7f4c96a00640 1 --2- 172.21.15.90:0/2678254382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c980ce390 0x7f4c980b0780 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f4c8800c750 tx=0x7f4c8800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:09.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.718+0000 7f4c8f400640 1 -- 172.21.15.90:0/2678254382 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c88017070 con 0x7f4c980ce390 2024-03-24T01:26:09.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.718+0000 7f4c8f400640 1 -- 172.21.15.90:0/2678254382 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c88005af0 con 0x7f4c980ce390 2024-03-24T01:26:09.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.718+0000 7f4c9e000640 1 -- 172.21.15.90:0/2678254382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c9814dc20 con 0x7f4c980ce390 2024-03-24T01:26:09.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.718+0000 7f4c8f400640 1 -- 172.21.15.90:0/2678254382 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c88007070 con 0x7f4c980ce390 2024-03-24T01:26:09.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.719+0000 7f4c9e000640 1 -- 172.21.15.90:0/2678254382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c9814e4c0 con 0x7f4c980ce390 2024-03-24T01:26:09.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.720+0000 7f4c9e000640 1 -- 172.21.15.90:0/2678254382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c98068ac0 con 0x7f4c980ce390 2024-03-24T01:26:09.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.721+0000 7f4c8f400640 1 -- 172.21.15.90:0/2678254382 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4c8801a070 con 0x7f4c980ce390 2024-03-24T01:26:09.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.722+0000 7f4c8f400640 1 --2- 172.21.15.90:0/2678254382 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4c7806bbd0 0x7f4c7806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:09.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.722+0000 7f4c8f400640 1 -- 172.21.15.90:0/2678254382 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4c8808a170 con 0x7f4c980ce390 2024-03-24T01:26:09.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.722+0000 7f4c97400640 1 --2- 172.21.15.90:0/2678254382 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4c7806bbd0 0x7f4c7806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:09.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.724+0000 7f4c97400640 1 --2- 172.21.15.90:0/2678254382 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4c7806bbd0 0x7f4c7806e090 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4c84000f60 tx=0x7f4c840079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:09.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.731+0000 7f4c8f400640 1 -- 172.21.15.90:0/2678254382 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4c98068ac0 con 0x7f4c980ce390 2024-03-24T01:26:09.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.923+0000 7f4c9e000640 1 -- 172.21.15.90:0/2678254382 --> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4c9814e6c0 con 0x7f4c7806bbd0 2024-03-24T01:26:09.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.944+0000 7f4c8f400640 1 -- 172.21.15.90:0/2678254382 <== mgr.4104 v2:172.21.15.120:6832/860392634 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251959 (secure 0 0 0) 0x7f4c9814e6c0 con 0x7f4c7806bbd0 2024-03-24T01:26:09.949 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:26:09.950 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-24T01:26:09.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.950+0000 7f4c9e000640 1 -- 172.21.15.90:0/2678254382 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4c7806bbd0 msgr2=0x7f4c7806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:09.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.950+0000 7f4c9e000640 1 --2- 172.21.15.90:0/2678254382 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4c7806bbd0 0x7f4c7806e090 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4c84000f60 tx=0x7f4c840079b0 comp rx=0 tx=0).stop 2024-03-24T01:26:09.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.950+0000 7f4c9e000640 1 -- 172.21.15.90:0/2678254382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c980ce390 msgr2=0x7f4c980b0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:09.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.950+0000 7f4c9e000640 1 --2- 172.21.15.90:0/2678254382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c980ce390 0x7f4c980b0780 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f4c8800c750 tx=0x7f4c8800cc10 comp rx=0 tx=0).stop 2024-03-24T01:26:09.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.950+0000 7f4c9e000640 1 -- 172.21.15.90:0/2678254382 shutdown_connections 2024-03-24T01:26:09.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.950+0000 7f4c9e000640 1 --2- 172.21.15.90:0/2678254382 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4c7806bbd0 0x7f4c7806e090 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:09.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.950+0000 7f4c9e000640 1 --2- 172.21.15.90:0/2678254382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c980ce390 0x7f4c980b0780 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:09.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.950+0000 7f4c9e000640 1 --2- 172.21.15.90:0/2678254382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c9815a280 0x7f4c98147230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:09.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.950+0000 7f4c9e000640 1 --2- 172.21.15.90:0/2678254382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c9806f790 0x7f4c9814b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:09.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.950+0000 7f4c9e000640 1 -- 172.21.15.90:0/2678254382 >> 172.21.15.90:0/2678254382 conn(0x7f4c980b1770 msgr2=0x7f4c980b2c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:09.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.950+0000 7f4c9e000640 1 -- 172.21.15.90:0/2678254382 shutdown_connections 2024-03-24T01:26:09.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:09.950+0000 7f4c9e000640 1 -- 172.21.15.90:0/2678254382 wait complete. 2024-03-24T01:26:09.969 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":97,"stamp":"2024-03-24T01:26:09.523732+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220912,"kb_used_data":4744,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753808,"statfs":{"total":773094113280,"available":772867899392,"internally_reserved":0,"allocated":4857856,"data_stored":2764172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12747,"internal_metadata":221236789},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.555115"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.998062+0000","last_change":"2024-03-24T01:26:00.019576+0000","last_active":"2024-03-24T01:26:00.998062+0000","last_peered":"2024-03-24T01:26:00.998062+0000","last_clean":"2024-03-24T01:26:00.998062+0000","last_became_active":"2024-03-24T01:26:00.019012+0000","last_became_peered":"2024-03-24T01:26:00.019012+0000","last_unstale":"2024-03-24T01:26:00.998062+0000","last_undegraded":"2024-03-24T01:26:00.998062+0000","last_fullsized":"2024-03-24T01:26:00.998062+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:39:06.645554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.997983+0000","last_change":"2024-03-24T01:26:00.002513+0000","last_active":"2024-03-24T01:26:00.997983+0000","last_peered":"2024-03-24T01:26:00.997983+0000","last_clean":"2024-03-24T01:26:00.997983+0000","last_became_active":"2024-03-24T01:26:00.002206+0000","last_became_peered":"2024-03-24T01:26:00.002206+0000","last_unstale":"2024-03-24T01:26:00.997983+0000","last_undegraded":"2024-03-24T01:26:00.997983+0000","last_fullsized":"2024-03-24T01:26:00.997983+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:02:46.296797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221988+0000","last_change":"2024-03-24T01:26:00.008425+0000","last_active":"2024-03-24T01:26:01.221988+0000","last_peered":"2024-03-24T01:26:01.221988+0000","last_clean":"2024-03-24T01:26:01.221988+0000","last_became_active":"2024-03-24T01:26:00.007866+0000","last_became_peered":"2024-03-24T01:26:00.007866+0000","last_unstale":"2024-03-24T01:26:01.221988+0000","last_undegraded":"2024-03-24T01:26:01.221988+0000","last_fullsized":"2024-03-24T01:26:01.221988+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:30:44.565229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.007149+0000","last_change":"2024-03-24T01:26:00.012605+0000","last_active":"2024-03-24T01:26:01.007149+0000","last_peered":"2024-03-24T01:26:01.007149+0000","last_clean":"2024-03-24T01:26:01.007149+0000","last_became_active":"2024-03-24T01:26:00.009919+0000","last_became_peered":"2024-03-24T01:26:00.009919+0000","last_unstale":"2024-03-24T01:26:01.007149+0000","last_undegraded":"2024-03-24T01:26:01.007149+0000","last_fullsized":"2024-03-24T01:26:01.007149+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:07:09.146157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.006993+0000","last_change":"2024-03-24T01:26:00.006183+0000","last_active":"2024-03-24T01:26:01.006993+0000","last_peered":"2024-03-24T01:26:01.006993+0000","last_clean":"2024-03-24T01:26:01.006993+0000","last_became_active":"2024-03-24T01:26:00.005891+0000","last_became_peered":"2024-03-24T01:26:00.005891+0000","last_unstale":"2024-03-24T01:26:01.006993+0000","last_undegraded":"2024-03-24T01:26:01.006993+0000","last_fullsized":"2024-03-24T01:26:01.006993+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:21:28.890740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221508+0000","last_change":"2024-03-24T01:26:00.009169+0000","last_active":"2024-03-24T01:26:01.221508+0000","last_peered":"2024-03-24T01:26:01.221508+0000","last_clean":"2024-03-24T01:26:01.221508+0000","last_became_active":"2024-03-24T01:26:00.008857+0000","last_became_peered":"2024-03-24T01:26:00.008857+0000","last_unstale":"2024-03-24T01:26:01.221508+0000","last_undegraded":"2024-03-24T01:26:01.221508+0000","last_fullsized":"2024-03-24T01:26:01.221508+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:16:28.904097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.011917+0000","last_change":"2024-03-24T01:26:00.017535+0000","last_active":"2024-03-24T01:26:01.011917+0000","last_peered":"2024-03-24T01:26:01.011917+0000","last_clean":"2024-03-24T01:26:01.011917+0000","last_became_active":"2024-03-24T01:26:00.017258+0000","last_became_peered":"2024-03-24T01:26:00.017258+0000","last_unstale":"2024-03-24T01:26:01.011917+0000","last_undegraded":"2024-03-24T01:26:01.011917+0000","last_fullsized":"2024-03-24T01:26:01.011917+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:56:23.530821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.997252+0000","last_change":"2024-03-24T01:26:00.011615+0000","last_active":"2024-03-24T01:26:00.997252+0000","last_peered":"2024-03-24T01:26:00.997252+0000","last_clean":"2024-03-24T01:26:00.997252+0000","last_became_active":"2024-03-24T01:26:00.011334+0000","last_became_peered":"2024-03-24T01:26:00.011334+0000","last_unstale":"2024-03-24T01:26:00.997252+0000","last_undegraded":"2024-03-24T01:26:00.997252+0000","last_fullsized":"2024-03-24T01:26:00.997252+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:06:55.514360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.222020+0000","last_change":"2024-03-24T01:26:00.008198+0000","last_active":"2024-03-24T01:26:01.222020+0000","last_peered":"2024-03-24T01:26:01.222020+0000","last_clean":"2024-03-24T01:26:01.222020+0000","last_became_active":"2024-03-24T01:26:00.007870+0000","last_became_peered":"2024-03-24T01:26:00.007870+0000","last_unstale":"2024-03-24T01:26:01.222020+0000","last_undegraded":"2024-03-24T01:26:01.222020+0000","last_fullsized":"2024-03-24T01:26:01.222020+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:56:23.530821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.013334+0000","last_change":"2024-03-24T01:26:00.011611+0000","last_active":"2024-03-24T01:26:01.013334+0000","last_peered":"2024-03-24T01:26:01.013334+0000","last_clean":"2024-03-24T01:26:01.013334+0000","last_became_active":"2024-03-24T01:26:00.011272+0000","last_became_peered":"2024-03-24T01:26:00.011272+0000","last_unstale":"2024-03-24T01:26:01.013334+0000","last_undegraded":"2024-03-24T01:26:01.013334+0000","last_fullsized":"2024-03-24T01:26:01.013334+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:17:45.147642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.012882+0000","last_change":"2024-03-24T01:26:00.014118+0000","last_active":"2024-03-24T01:26:01.012882+0000","last_peered":"2024-03-24T01:26:01.012882+0000","last_clean":"2024-03-24T01:26:01.012882+0000","last_became_active":"2024-03-24T01:26:00.013783+0000","last_became_peered":"2024-03-24T01:26:00.013783+0000","last_unstale":"2024-03-24T01:26:01.012882+0000","last_undegraded":"2024-03-24T01:26:01.012882+0000","last_fullsized":"2024-03-24T01:26:01.012882+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:06:55.514360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221697+0000","last_change":"2024-03-24T01:26:00.009436+0000","last_active":"2024-03-24T01:26:01.221697+0000","last_peered":"2024-03-24T01:26:01.221697+0000","last_clean":"2024-03-24T01:26:01.221697+0000","last_became_active":"2024-03-24T01:26:00.009122+0000","last_became_peered":"2024-03-24T01:26:00.009122+0000","last_unstale":"2024-03-24T01:26:01.221697+0000","last_undegraded":"2024-03-24T01:26:01.221697+0000","last_fullsized":"2024-03-24T01:26:01.221697+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:39:38.186024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.012156+0000","last_change":"2024-03-24T01:26:00.003148+0000","last_active":"2024-03-24T01:26:01.012156+0000","last_peered":"2024-03-24T01:26:01.012156+0000","last_clean":"2024-03-24T01:26:01.012156+0000","last_became_active":"2024-03-24T01:26:00.002817+0000","last_became_peered":"2024-03-24T01:26:00.002817+0000","last_unstale":"2024-03-24T01:26:01.012156+0000","last_undegraded":"2024-03-24T01:26:01.012156+0000","last_fullsized":"2024-03-24T01:26:01.012156+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:46:09.862425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.001772+0000","last_change":"2024-03-24T01:26:00.008295+0000","last_active":"2024-03-24T01:26:01.001772+0000","last_peered":"2024-03-24T01:26:01.001772+0000","last_clean":"2024-03-24T01:26:01.001772+0000","last_became_active":"2024-03-24T01:26:00.008007+0000","last_became_peered":"2024-03-24T01:26:00.008007+0000","last_unstale":"2024-03-24T01:26:01.001772+0000","last_undegraded":"2024-03-24T01:26:01.001772+0000","last_fullsized":"2024-03-24T01:26:01.001772+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:02:08.397348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.012677+0000","last_change":"2024-03-24T01:26:00.011211+0000","last_active":"2024-03-24T01:26:01.012677+0000","last_peered":"2024-03-24T01:26:01.012677+0000","last_clean":"2024-03-24T01:26:01.012677+0000","last_became_active":"2024-03-24T01:26:00.010912+0000","last_became_peered":"2024-03-24T01:26:00.010912+0000","last_unstale":"2024-03-24T01:26:01.012677+0000","last_undegraded":"2024-03-24T01:26:01.012677+0000","last_fullsized":"2024-03-24T01:26:01.012677+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:20:17.936211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221276+0000","last_change":"2024-03-24T01:26:00.009781+0000","last_active":"2024-03-24T01:26:01.221276+0000","last_peered":"2024-03-24T01:26:01.221276+0000","last_clean":"2024-03-24T01:26:01.221276+0000","last_became_active":"2024-03-24T01:26:00.009027+0000","last_became_peered":"2024-03-24T01:26:00.009027+0000","last_unstale":"2024-03-24T01:26:01.221276+0000","last_undegraded":"2024-03-24T01:26:01.221276+0000","last_fullsized":"2024-03-24T01:26:01.221276+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:20:43.942659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.222841+0000","last_change":"2024-03-24T01:26:00.011182+0000","last_active":"2024-03-24T01:26:01.222841+0000","last_peered":"2024-03-24T01:26:01.222841+0000","last_clean":"2024-03-24T01:26:01.222841+0000","last_became_active":"2024-03-24T01:26:00.010834+0000","last_became_peered":"2024-03-24T01:26:00.010834+0000","last_unstale":"2024-03-24T01:26:01.222841+0000","last_undegraded":"2024-03-24T01:26:01.222841+0000","last_fullsized":"2024-03-24T01:26:01.222841+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:11:25.571750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.999592+0000","last_change":"2024-03-24T01:26:00.009006+0000","last_active":"2024-03-24T01:26:00.999592+0000","last_peered":"2024-03-24T01:26:00.999592+0000","last_clean":"2024-03-24T01:26:00.999592+0000","last_became_active":"2024-03-24T01:26:00.008723+0000","last_became_peered":"2024-03-24T01:26:00.008723+0000","last_unstale":"2024-03-24T01:26:00.999592+0000","last_undegraded":"2024-03-24T01:26:00.999592+0000","last_fullsized":"2024-03-24T01:26:00.999592+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:05:32.722931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.007473+0000","last_change":"2024-03-24T01:26:00.005802+0000","last_active":"2024-03-24T01:26:01.007473+0000","last_peered":"2024-03-24T01:26:01.007473+0000","last_clean":"2024-03-24T01:26:01.007473+0000","last_became_active":"2024-03-24T01:26:00.005499+0000","last_became_peered":"2024-03-24T01:26:00.005499+0000","last_unstale":"2024-03-24T01:26:01.007473+0000","last_undegraded":"2024-03-24T01:26:01.007473+0000","last_fullsized":"2024-03-24T01:26:01.007473+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:49:25.142409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.999646+0000","last_change":"2024-03-24T01:26:00.008411+0000","last_active":"2024-03-24T01:26:00.999646+0000","last_peered":"2024-03-24T01:26:00.999646+0000","last_clean":"2024-03-24T01:26:00.999646+0000","last_became_active":"2024-03-24T01:26:00.008145+0000","last_became_peered":"2024-03-24T01:26:00.008145+0000","last_unstale":"2024-03-24T01:26:00.999646+0000","last_undegraded":"2024-03-24T01:26:00.999646+0000","last_fullsized":"2024-03-24T01:26:00.999646+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:59:54.117252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.012286+0000","last_change":"2024-03-24T01:26:00.007178+0000","last_active":"2024-03-24T01:26:01.012286+0000","last_peered":"2024-03-24T01:26:01.012286+0000","last_clean":"2024-03-24T01:26:01.012286+0000","last_became_active":"2024-03-24T01:26:00.005004+0000","last_became_peered":"2024-03-24T01:26:00.005004+0000","last_unstale":"2024-03-24T01:26:01.012286+0000","last_undegraded":"2024-03-24T01:26:01.012286+0000","last_fullsized":"2024-03-24T01:26:01.012286+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:07:24.167208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.000663+0000","last_change":"2024-03-24T01:26:00.019905+0000","last_active":"2024-03-24T01:26:01.000663+0000","last_peered":"2024-03-24T01:26:01.000663+0000","last_clean":"2024-03-24T01:26:01.000663+0000","last_became_active":"2024-03-24T01:26:00.019575+0000","last_became_peered":"2024-03-24T01:26:00.019575+0000","last_unstale":"2024-03-24T01:26:01.000663+0000","last_undegraded":"2024-03-24T01:26:01.000663+0000","last_fullsized":"2024-03-24T01:26:01.000663+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:16:28.904097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221506+0000","last_change":"2024-03-24T01:25:59.996770+0000","last_active":"2024-03-24T01:26:01.221506+0000","last_peered":"2024-03-24T01:26:01.221506+0000","last_clean":"2024-03-24T01:26:01.221506+0000","last_became_active":"2024-03-24T01:25:59.996482+0000","last_became_peered":"2024-03-24T01:25:59.996482+0000","last_unstale":"2024-03-24T01:26:01.221506+0000","last_undegraded":"2024-03-24T01:26:01.221506+0000","last_fullsized":"2024-03-24T01:26:01.221506+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:02:08.397348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.006809+0000","last_change":"2024-03-24T01:25:59.989965+0000","last_active":"2024-03-24T01:26:01.006809+0000","last_peered":"2024-03-24T01:26:01.006809+0000","last_clean":"2024-03-24T01:26:01.006809+0000","last_became_active":"2024-03-24T01:25:59.989645+0000","last_became_peered":"2024-03-24T01:25:59.989645+0000","last_unstale":"2024-03-24T01:26:01.006809+0000","last_undegraded":"2024-03-24T01:26:01.006809+0000","last_fullsized":"2024-03-24T01:26:01.006809+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:26:56.752921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.000081+0000","last_change":"2024-03-24T01:26:00.013258+0000","last_active":"2024-03-24T01:26:01.000081+0000","last_peered":"2024-03-24T01:26:01.000081+0000","last_clean":"2024-03-24T01:26:01.000081+0000","last_became_active":"2024-03-24T01:26:00.010223+0000","last_became_peered":"2024-03-24T01:26:00.010223+0000","last_unstale":"2024-03-24T01:26:01.000081+0000","last_undegraded":"2024-03-24T01:26:01.000081+0000","last_fullsized":"2024-03-24T01:26:01.000081+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:24:40.903708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221583+0000","last_change":"2024-03-24T01:26:00.012534+0000","last_active":"2024-03-24T01:26:01.221583+0000","last_peered":"2024-03-24T01:26:01.221583+0000","last_clean":"2024-03-24T01:26:01.221583+0000","last_became_active":"2024-03-24T01:26:00.011982+0000","last_became_peered":"2024-03-24T01:26:00.011982+0000","last_unstale":"2024-03-24T01:26:01.221583+0000","last_undegraded":"2024-03-24T01:26:01.221583+0000","last_fullsized":"2024-03-24T01:26:01.221583+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:30:44.565229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.002400+0000","last_change":"2024-03-24T01:25:59.997124+0000","last_active":"2024-03-24T01:26:01.002400+0000","last_peered":"2024-03-24T01:26:01.002400+0000","last_clean":"2024-03-24T01:26:01.002400+0000","last_became_active":"2024-03-24T01:25:59.996779+0000","last_became_peered":"2024-03-24T01:25:59.996779+0000","last_unstale":"2024-03-24T01:26:01.002400+0000","last_undegraded":"2024-03-24T01:26:01.002400+0000","last_fullsized":"2024-03-24T01:26:01.002400+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:20:43.942659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.012360+0000","last_change":"2024-03-24T01:26:00.016153+0000","last_active":"2024-03-24T01:26:01.012360+0000","last_peered":"2024-03-24T01:26:01.012360+0000","last_clean":"2024-03-24T01:26:01.012360+0000","last_became_active":"2024-03-24T01:26:00.015810+0000","last_became_peered":"2024-03-24T01:26:00.015810+0000","last_unstale":"2024-03-24T01:26:01.012360+0000","last_undegraded":"2024-03-24T01:26:01.012360+0000","last_fullsized":"2024-03-24T01:26:01.012360+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:42:02.641519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.001346+0000","last_change":"2024-03-24T01:26:00.003439+0000","last_active":"2024-03-24T01:26:01.001346+0000","last_peered":"2024-03-24T01:26:01.001346+0000","last_clean":"2024-03-24T01:26:01.001346+0000","last_became_active":"2024-03-24T01:26:00.001832+0000","last_became_peered":"2024-03-24T01:26:00.001832+0000","last_unstale":"2024-03-24T01:26:01.001346+0000","last_undegraded":"2024-03-24T01:26:01.001346+0000","last_fullsized":"2024-03-24T01:26:01.001346+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:08:58.631691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.000243+0000","last_change":"2024-03-24T01:25:59.995363+0000","last_active":"2024-03-24T01:26:01.000243+0000","last_peered":"2024-03-24T01:26:01.000243+0000","last_clean":"2024-03-24T01:26:01.000243+0000","last_became_active":"2024-03-24T01:25:59.995067+0000","last_became_peered":"2024-03-24T01:25:59.995067+0000","last_unstale":"2024-03-24T01:26:01.000243+0000","last_undegraded":"2024-03-24T01:26:01.000243+0000","last_fullsized":"2024-03-24T01:26:01.000243+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:36:04.311730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221948+0000","last_change":"2024-03-24T01:26:00.004303+0000","last_active":"2024-03-24T01:26:01.221948+0000","last_peered":"2024-03-24T01:26:01.221948+0000","last_clean":"2024-03-24T01:26:01.221948+0000","last_became_active":"2024-03-24T01:26:00.002367+0000","last_became_peered":"2024-03-24T01:26:00.002367+0000","last_unstale":"2024-03-24T01:26:01.221948+0000","last_undegraded":"2024-03-24T01:26:01.221948+0000","last_fullsized":"2024-03-24T01:26:01.221948+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:56:23.530821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221269+0000","last_change":"2024-03-24T01:25:59.999709+0000","last_active":"2024-03-24T01:26:01.221269+0000","last_peered":"2024-03-24T01:26:01.221269+0000","last_clean":"2024-03-24T01:26:01.221269+0000","last_became_active":"2024-03-24T01:25:59.999428+0000","last_became_peered":"2024-03-24T01:25:59.999428+0000","last_unstale":"2024-03-24T01:26:01.221269+0000","last_undegraded":"2024-03-24T01:26:01.221269+0000","last_fullsized":"2024-03-24T01:26:01.221269+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:39:38.186024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.222312+0000","last_change":"2024-03-24T01:26:00.008441+0000","last_active":"2024-03-24T01:26:01.222312+0000","last_peered":"2024-03-24T01:26:01.222312+0000","last_clean":"2024-03-24T01:26:01.222312+0000","last_became_active":"2024-03-24T01:26:00.008117+0000","last_became_peered":"2024-03-24T01:26:00.008117+0000","last_unstale":"2024-03-24T01:26:01.222312+0000","last_undegraded":"2024-03-24T01:26:01.222312+0000","last_fullsized":"2024-03-24T01:26:01.222312+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:46:09.862425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":18,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.999605+0000","last_change":"2024-03-24T01:25:58.996685+0000","last_active":"2024-03-24T01:26:00.999605+0000","last_peered":"2024-03-24T01:26:00.999605+0000","last_clean":"2024-03-24T01:26:00.999605+0000","last_became_active":"2024-03-24T01:25:58.996398+0000","last_became_peered":"2024-03-24T01:25:58.996398+0000","last_unstale":"2024-03-24T01:26:00.999605+0000","last_undegraded":"2024-03-24T01:26:00.999605+0000","last_fullsized":"2024-03-24T01:26:00.999605+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:45.605005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221978+0000","last_change":"2024-03-24T01:26:00.015135+0000","last_active":"2024-03-24T01:26:01.221978+0000","last_peered":"2024-03-24T01:26:01.221978+0000","last_clean":"2024-03-24T01:26:01.221978+0000","last_became_active":"2024-03-24T01:26:00.014848+0000","last_became_peered":"2024-03-24T01:26:00.014848+0000","last_unstale":"2024-03-24T01:26:01.221978+0000","last_undegraded":"2024-03-24T01:26:01.221978+0000","last_fullsized":"2024-03-24T01:26:01.221978+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:56:23.530821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.222294+0000","last_change":"2024-03-24T01:25:59.002453+0000","last_active":"2024-03-24T01:26:01.222294+0000","last_peered":"2024-03-24T01:26:01.222294+0000","last_clean":"2024-03-24T01:26:01.222294+0000","last_became_active":"2024-03-24T01:25:59.002155+0000","last_became_peered":"2024-03-24T01:25:59.002155+0000","last_unstale":"2024-03-24T01:26:01.222294+0000","last_undegraded":"2024-03-24T01:26:01.222294+0000","last_fullsized":"2024-03-24T01:26:01.222294+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:45.605005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.002182+0000","last_change":"2024-03-24T01:26:00.010489+0000","last_active":"2024-03-24T01:26:01.002182+0000","last_peered":"2024-03-24T01:26:01.002182+0000","last_clean":"2024-03-24T01:26:01.002182+0000","last_became_active":"2024-03-24T01:26:00.010187+0000","last_became_peered":"2024-03-24T01:26:00.010187+0000","last_unstale":"2024-03-24T01:26:01.002182+0000","last_undegraded":"2024-03-24T01:26:01.002182+0000","last_fullsized":"2024-03-24T01:26:01.002182+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:07:24.167208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.222196+0000","last_change":"2024-03-24T01:25:58.997334+0000","last_active":"2024-03-24T01:26:01.222196+0000","last_peered":"2024-03-24T01:26:01.222196+0000","last_clean":"2024-03-24T01:26:01.222196+0000","last_became_active":"2024-03-24T01:25:58.997033+0000","last_became_peered":"2024-03-24T01:25:58.997033+0000","last_unstale":"2024-03-24T01:26:01.222196+0000","last_undegraded":"2024-03-24T01:26:01.222196+0000","last_fullsized":"2024-03-24T01:26:01.222196+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:00:07.242997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.998005+0000","last_change":"2024-03-24T01:26:00.004685+0000","last_active":"2024-03-24T01:26:00.998005+0000","last_peered":"2024-03-24T01:26:00.998005+0000","last_clean":"2024-03-24T01:26:00.998005+0000","last_became_active":"2024-03-24T01:26:00.004430+0000","last_became_peered":"2024-03-24T01:26:00.004430+0000","last_unstale":"2024-03-24T01:26:00.998005+0000","last_undegraded":"2024-03-24T01:26:00.998005+0000","last_fullsized":"2024-03-24T01:26:00.998005+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:47:33.606184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.001541+0000","last_change":"2024-03-24T01:25:58.992317+0000","last_active":"2024-03-24T01:26:01.001541+0000","last_peered":"2024-03-24T01:26:01.001541+0000","last_clean":"2024-03-24T01:26:01.001541+0000","last_became_active":"2024-03-24T01:25:58.991652+0000","last_became_peered":"2024-03-24T01:25:58.991652+0000","last_unstale":"2024-03-24T01:26:01.001541+0000","last_undegraded":"2024-03-24T01:26:01.001541+0000","last_fullsized":"2024-03-24T01:26:01.001541+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:00:07.242997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.220739+0000","last_change":"2024-03-24T01:25:59.999595+0000","last_active":"2024-03-24T01:26:01.220739+0000","last_peered":"2024-03-24T01:26:01.220739+0000","last_clean":"2024-03-24T01:26:01.220739+0000","last_became_active":"2024-03-24T01:25:59.999225+0000","last_became_peered":"2024-03-24T01:25:59.999225+0000","last_unstale":"2024-03-24T01:26:01.220739+0000","last_undegraded":"2024-03-24T01:26:01.220739+0000","last_fullsized":"2024-03-24T01:26:01.220739+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:36:04.311730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.012655+0000","last_change":"2024-03-24T01:25:58.998337+0000","last_active":"2024-03-24T01:26:01.012655+0000","last_peered":"2024-03-24T01:26:01.012655+0000","last_clean":"2024-03-24T01:26:01.012655+0000","last_became_active":"2024-03-24T01:25:58.998056+0000","last_became_peered":"2024-03-24T01:25:58.998056+0000","last_unstale":"2024-03-24T01:26:01.012655+0000","last_undegraded":"2024-03-24T01:26:01.012655+0000","last_fullsized":"2024-03-24T01:26:01.012655+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:05:31.709112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.012492+0000","last_change":"2024-03-24T01:26:00.004616+0000","last_active":"2024-03-24T01:26:01.012492+0000","last_peered":"2024-03-24T01:26:01.012492+0000","last_clean":"2024-03-24T01:26:01.012492+0000","last_became_active":"2024-03-24T01:26:00.003030+0000","last_became_peered":"2024-03-24T01:26:00.003030+0000","last_unstale":"2024-03-24T01:26:01.012492+0000","last_undegraded":"2024-03-24T01:26:01.012492+0000","last_fullsized":"2024-03-24T01:26:01.012492+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:11:25.571750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221421+0000","last_change":"2024-03-24T01:25:59.006913+0000","last_active":"2024-03-24T01:26:01.221421+0000","last_peered":"2024-03-24T01:26:01.221421+0000","last_clean":"2024-03-24T01:26:01.221421+0000","last_became_active":"2024-03-24T01:25:59.006098+0000","last_became_peered":"2024-03-24T01:25:59.006098+0000","last_unstale":"2024-03-24T01:26:01.221421+0000","last_undegraded":"2024-03-24T01:26:01.221421+0000","last_fullsized":"2024-03-24T01:26:01.221421+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:44:12.069166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.001097+0000","last_change":"2024-03-24T01:26:00.006344+0000","last_active":"2024-03-24T01:26:01.001097+0000","last_peered":"2024-03-24T01:26:01.001097+0000","last_clean":"2024-03-24T01:26:01.001097+0000","last_became_active":"2024-03-24T01:26:00.005891+0000","last_became_peered":"2024-03-24T01:26:00.005891+0000","last_unstale":"2024-03-24T01:26:01.001097+0000","last_undegraded":"2024-03-24T01:26:01.001097+0000","last_fullsized":"2024-03-24T01:26:01.001097+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:30:44.565229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221290+0000","last_change":"2024-03-24T01:25:59.000318+0000","last_active":"2024-03-24T01:26:01.221290+0000","last_peered":"2024-03-24T01:26:01.221290+0000","last_clean":"2024-03-24T01:26:01.221290+0000","last_became_active":"2024-03-24T01:25:59.000085+0000","last_became_peered":"2024-03-24T01:25:59.000085+0000","last_unstale":"2024-03-24T01:26:01.221290+0000","last_undegraded":"2024-03-24T01:26:01.221290+0000","last_fullsized":"2024-03-24T01:26:01.221290+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:12:43.140922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.997472+0000","last_change":"2024-03-24T01:26:00.003084+0000","last_active":"2024-03-24T01:26:00.997472+0000","last_peered":"2024-03-24T01:26:00.997472+0000","last_clean":"2024-03-24T01:26:00.997472+0000","last_became_active":"2024-03-24T01:26:00.002750+0000","last_became_peered":"2024-03-24T01:26:00.002750+0000","last_unstale":"2024-03-24T01:26:00.997472+0000","last_undegraded":"2024-03-24T01:26:00.997472+0000","last_fullsized":"2024-03-24T01:26:00.997472+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:17:45.147642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.999120+0000","last_change":"2024-03-24T01:25:58.994521+0000","last_active":"2024-03-24T01:26:00.999120+0000","last_peered":"2024-03-24T01:26:00.999120+0000","last_clean":"2024-03-24T01:26:00.999120+0000","last_became_active":"2024-03-24T01:25:58.994215+0000","last_became_peered":"2024-03-24T01:25:58.994215+0000","last_unstale":"2024-03-24T01:26:00.999120+0000","last_undegraded":"2024-03-24T01:26:00.999120+0000","last_fullsized":"2024-03-24T01:26:00.999120+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:12:43.140922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.007272+0000","last_change":"2024-03-24T01:26:00.012399+0000","last_active":"2024-03-24T01:26:01.007272+0000","last_peered":"2024-03-24T01:26:01.007272+0000","last_clean":"2024-03-24T01:26:01.007272+0000","last_became_active":"2024-03-24T01:26:00.008358+0000","last_became_peered":"2024-03-24T01:26:00.008358+0000","last_unstale":"2024-03-24T01:26:01.007272+0000","last_undegraded":"2024-03-24T01:26:01.007272+0000","last_fullsized":"2024-03-24T01:26:01.007272+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:00:43.649482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.998097+0000","last_change":"2024-03-24T01:25:58.995322+0000","last_active":"2024-03-24T01:26:00.998097+0000","last_peered":"2024-03-24T01:26:00.998097+0000","last_clean":"2024-03-24T01:26:00.998097+0000","last_became_active":"2024-03-24T01:25:58.994847+0000","last_became_peered":"2024-03-24T01:25:58.994847+0000","last_unstale":"2024-03-24T01:26:00.998097+0000","last_undegraded":"2024-03-24T01:26:00.998097+0000","last_fullsized":"2024-03-24T01:26:00.998097+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:08:57.617873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.222562+0000","last_change":"2024-03-24T01:25:59.997541+0000","last_active":"2024-03-24T01:26:01.222562+0000","last_peered":"2024-03-24T01:26:01.222562+0000","last_clean":"2024-03-24T01:26:01.222562+0000","last_became_active":"2024-03-24T01:25:59.992800+0000","last_became_peered":"2024-03-24T01:25:59.992800+0000","last_unstale":"2024-03-24T01:26:01.222562+0000","last_undegraded":"2024-03-24T01:26:01.222562+0000","last_fullsized":"2024-03-24T01:26:01.222562+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:42:02.641519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.013224+0000","last_change":"2024-03-24T01:25:58.998109+0000","last_active":"2024-03-24T01:26:01.013224+0000","last_peered":"2024-03-24T01:26:01.013224+0000","last_clean":"2024-03-24T01:26:01.013224+0000","last_became_active":"2024-03-24T01:25:58.997791+0000","last_became_peered":"2024-03-24T01:25:58.997791+0000","last_unstale":"2024-03-24T01:26:01.013224+0000","last_undegraded":"2024-03-24T01:26:01.013224+0000","last_fullsized":"2024-03-24T01:26:01.013224+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:19:04.197281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.222240+0000","last_change":"2024-03-24T01:26:00.005147+0000","last_active":"2024-03-24T01:26:01.222240+0000","last_peered":"2024-03-24T01:26:01.222240+0000","last_clean":"2024-03-24T01:26:01.222240+0000","last_became_active":"2024-03-24T01:26:00.004799+0000","last_became_peered":"2024-03-24T01:26:00.004799+0000","last_unstale":"2024-03-24T01:26:01.222240+0000","last_undegraded":"2024-03-24T01:26:01.222240+0000","last_fullsized":"2024-03-24T01:26:01.222240+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:02:08.397348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":17,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.013185+0000","last_change":"2024-03-24T01:25:58.999040+0000","last_active":"2024-03-24T01:26:01.013185+0000","last_peered":"2024-03-24T01:26:01.013185+0000","last_clean":"2024-03-24T01:26:01.013185+0000","last_became_active":"2024-03-24T01:25:58.998593+0000","last_became_peered":"2024-03-24T01:25:58.998593+0000","last_unstale":"2024-03-24T01:26:01.013185+0000","last_undegraded":"2024-03-24T01:26:01.013185+0000","last_fullsized":"2024-03-24T01:26:01.013185+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:00:07.242997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.001205+0000","last_change":"2024-03-24T01:26:00.008616+0000","last_active":"2024-03-24T01:26:01.001205+0000","last_peered":"2024-03-24T01:26:01.001205+0000","last_clean":"2024-03-24T01:26:01.001205+0000","last_became_active":"2024-03-24T01:26:00.008345+0000","last_became_peered":"2024-03-24T01:26:00.008345+0000","last_unstale":"2024-03-24T01:26:01.001205+0000","last_undegraded":"2024-03-24T01:26:01.001205+0000","last_fullsized":"2024-03-24T01:26:01.001205+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:56:23.530821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.998078+0000","last_change":"2024-03-24T01:25:59.002378+0000","last_active":"2024-03-24T01:26:00.998078+0000","last_peered":"2024-03-24T01:26:00.998078+0000","last_clean":"2024-03-24T01:26:00.998078+0000","last_became_active":"2024-03-24T01:25:59.002071+0000","last_became_peered":"2024-03-24T01:25:59.002071+0000","last_unstale":"2024-03-24T01:26:00.998078+0000","last_undegraded":"2024-03-24T01:26:00.998078+0000","last_fullsized":"2024-03-24T01:26:00.998078+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:20:42.928840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.220866+0000","last_change":"2024-03-24T01:26:00.006589+0000","last_active":"2024-03-24T01:26:01.220866+0000","last_peered":"2024-03-24T01:26:01.220866+0000","last_clean":"2024-03-24T01:26:01.220866+0000","last_became_active":"2024-03-24T01:26:00.006332+0000","last_became_peered":"2024-03-24T01:26:00.006332+0000","last_unstale":"2024-03-24T01:26:01.220866+0000","last_undegraded":"2024-03-24T01:26:01.220866+0000","last_fullsized":"2024-03-24T01:26:01.220866+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:08:58.631691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.997732+0000","last_change":"2024-03-24T01:25:58.987270+0000","last_active":"2024-03-24T01:26:00.997732+0000","last_peered":"2024-03-24T01:26:00.997732+0000","last_clean":"2024-03-24T01:26:00.997732+0000","last_became_active":"2024-03-24T01:25:58.986953+0000","last_became_peered":"2024-03-24T01:25:58.986953+0000","last_unstale":"2024-03-24T01:26:00.997732+0000","last_undegraded":"2024-03-24T01:26:00.997732+0000","last_fullsized":"2024-03-24T01:26:00.997732+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:36:03.297911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.997417+0000","last_change":"2024-03-24T01:25:59.999556+0000","last_active":"2024-03-24T01:26:00.997417+0000","last_peered":"2024-03-24T01:26:00.997417+0000","last_clean":"2024-03-24T01:26:00.997417+0000","last_became_active":"2024-03-24T01:25:59.999261+0000","last_became_peered":"2024-03-24T01:25:59.999261+0000","last_unstale":"2024-03-24T01:26:00.997417+0000","last_undegraded":"2024-03-24T01:26:00.997417+0000","last_fullsized":"2024-03-24T01:26:00.997417+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:01:28.134219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221624+0000","last_change":"2024-03-24T01:25:59.000275+0000","last_active":"2024-03-24T01:26:01.221624+0000","last_peered":"2024-03-24T01:26:01.221624+0000","last_clean":"2024-03-24T01:26:01.221624+0000","last_became_active":"2024-03-24T01:25:58.999948+0000","last_became_peered":"2024-03-24T01:25:58.999948+0000","last_unstale":"2024-03-24T01:26:01.221624+0000","last_undegraded":"2024-03-24T01:26:01.221624+0000","last_fullsized":"2024-03-24T01:26:01.221624+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:55.200533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221592+0000","last_change":"2024-03-24T01:25:59.995825+0000","last_active":"2024-03-24T01:26:01.221592+0000","last_peered":"2024-03-24T01:26:01.221592+0000","last_clean":"2024-03-24T01:26:01.221592+0000","last_became_active":"2024-03-24T01:25:59.995551+0000","last_became_peered":"2024-03-24T01:25:59.995551+0000","last_unstale":"2024-03-24T01:26:01.221592+0000","last_undegraded":"2024-03-24T01:26:01.221592+0000","last_fullsized":"2024-03-24T01:26:01.221592+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:39:38.186024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.000968+0000","last_change":"2024-03-24T01:25:58.991563+0000","last_active":"2024-03-24T01:26:01.000968+0000","last_peered":"2024-03-24T01:26:01.000968+0000","last_clean":"2024-03-24T01:26:01.000968+0000","last_became_active":"2024-03-24T01:25:58.990946+0000","last_became_peered":"2024-03-24T01:25:58.990946+0000","last_unstale":"2024-03-24T01:26:01.000968+0000","last_undegraded":"2024-03-24T01:26:01.000968+0000","last_fullsized":"2024-03-24T01:26:01.000968+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:08:38.108175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.997634+0000","last_change":"2024-03-24T01:26:00.019807+0000","last_active":"2024-03-24T01:26:00.997634+0000","last_peered":"2024-03-24T01:26:00.997634+0000","last_clean":"2024-03-24T01:26:00.997634+0000","last_became_active":"2024-03-24T01:26:00.019479+0000","last_became_peered":"2024-03-24T01:26:00.019479+0000","last_unstale":"2024-03-24T01:26:00.997634+0000","last_undegraded":"2024-03-24T01:26:00.997634+0000","last_fullsized":"2024-03-24T01:26:00.997634+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:30:21.708639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":22,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221368+0000","last_change":"2024-03-24T01:25:59.006569+0000","last_active":"2024-03-24T01:26:01.221368+0000","last_peered":"2024-03-24T01:26:01.221368+0000","last_clean":"2024-03-24T01:26:01.221368+0000","last_became_active":"2024-03-24T01:25:59.006280+0000","last_became_peered":"2024-03-24T01:25:59.006280+0000","last_unstale":"2024-03-24T01:26:01.221368+0000","last_undegraded":"2024-03-24T01:26:01.221368+0000","last_fullsized":"2024-03-24T01:26:01.221368+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:45.605005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.997931+0000","last_change":"2024-03-24T01:26:00.005482+0000","last_active":"2024-03-24T01:26:00.997931+0000","last_peered":"2024-03-24T01:26:00.997931+0000","last_clean":"2024-03-24T01:26:00.997931+0000","last_became_active":"2024-03-24T01:26:00.005203+0000","last_became_peered":"2024-03-24T01:26:00.005203+0000","last_unstale":"2024-03-24T01:26:00.997931+0000","last_undegraded":"2024-03-24T01:26:00.997931+0000","last_fullsized":"2024-03-24T01:26:00.997931+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:54:09.379313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.013043+0000","last_change":"2024-03-24T01:25:59.003724+0000","last_active":"2024-03-24T01:26:01.013043+0000","last_peered":"2024-03-24T01:26:01.013043+0000","last_clean":"2024-03-24T01:26:01.013043+0000","last_became_active":"2024-03-24T01:25:59.003410+0000","last_became_peered":"2024-03-24T01:25:59.003410+0000","last_unstale":"2024-03-24T01:26:01.013043+0000","last_undegraded":"2024-03-24T01:26:01.013043+0000","last_fullsized":"2024-03-24T01:26:01.013043+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:12:43.140922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.013119+0000","last_change":"2024-03-24T01:26:00.006656+0000","last_active":"2024-03-24T01:26:01.013119+0000","last_peered":"2024-03-24T01:26:01.013119+0000","last_clean":"2024-03-24T01:26:01.013119+0000","last_became_active":"2024-03-24T01:26:00.006301+0000","last_became_peered":"2024-03-24T01:26:00.006301+0000","last_unstale":"2024-03-24T01:26:01.013119+0000","last_undegraded":"2024-03-24T01:26:01.013119+0000","last_fullsized":"2024-03-24T01:26:01.013119+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:01:28.134219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.002273+0000","last_change":"2024-03-24T01:25:58.999764+0000","last_active":"2024-03-24T01:26:01.002273+0000","last_peered":"2024-03-24T01:26:01.002273+0000","last_clean":"2024-03-24T01:26:01.002273+0000","last_became_active":"2024-03-24T01:25:58.999490+0000","last_became_peered":"2024-03-24T01:25:58.999490+0000","last_unstale":"2024-03-24T01:26:01.002273+0000","last_undegraded":"2024-03-24T01:26:01.002273+0000","last_fullsized":"2024-03-24T01:26:01.002273+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:19:04.197281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.001941+0000","last_change":"2024-03-24T01:26:00.016951+0000","last_active":"2024-03-24T01:26:01.001941+0000","last_peered":"2024-03-24T01:26:01.001941+0000","last_clean":"2024-03-24T01:26:01.001941+0000","last_became_active":"2024-03-24T01:26:00.016500+0000","last_became_peered":"2024-03-24T01:26:00.016500+0000","last_unstale":"2024-03-24T01:26:01.001941+0000","last_undegraded":"2024-03-24T01:26:01.001941+0000","last_fullsized":"2024-03-24T01:26:01.001941+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:46:09.862425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.007538+0000","last_change":"2024-03-24T01:25:59.003238+0000","last_active":"2024-03-24T01:26:01.007538+0000","last_peered":"2024-03-24T01:26:01.007538+0000","last_clean":"2024-03-24T01:26:01.007538+0000","last_became_active":"2024-03-24T01:25:59.002905+0000","last_became_peered":"2024-03-24T01:25:59.002905+0000","last_unstale":"2024-03-24T01:26:01.007538+0000","last_undegraded":"2024-03-24T01:26:01.007538+0000","last_fullsized":"2024-03-24T01:26:01.007538+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:43.279129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.007522+0000","last_change":"2024-03-24T01:26:00.008391+0000","last_active":"2024-03-24T01:26:01.007522+0000","last_peered":"2024-03-24T01:26:01.007522+0000","last_clean":"2024-03-24T01:26:01.007522+0000","last_became_active":"2024-03-24T01:26:00.007343+0000","last_became_peered":"2024-03-24T01:26:00.007343+0000","last_unstale":"2024-03-24T01:26:01.007522+0000","last_undegraded":"2024-03-24T01:26:01.007522+0000","last_fullsized":"2024-03-24T01:26:01.007522+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:53:50.285242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.222165+0000","last_change":"2024-03-24T01:25:58.994043+0000","last_active":"2024-03-24T01:26:01.222165+0000","last_peered":"2024-03-24T01:26:01.222165+0000","last_clean":"2024-03-24T01:26:01.222165+0000","last_became_active":"2024-03-24T01:25:58.993746+0000","last_became_peered":"2024-03-24T01:25:58.993746+0000","last_unstale":"2024-03-24T01:26:01.222165+0000","last_undegraded":"2024-03-24T01:26:01.222165+0000","last_fullsized":"2024-03-24T01:26:01.222165+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:19:04.197281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221838+0000","last_change":"2024-03-24T01:26:00.004263+0000","last_active":"2024-03-24T01:26:01.221838+0000","last_peered":"2024-03-24T01:26:01.221838+0000","last_clean":"2024-03-24T01:26:01.221838+0000","last_became_active":"2024-03-24T01:26:00.000719+0000","last_became_peered":"2024-03-24T01:26:00.000719+0000","last_unstale":"2024-03-24T01:26:01.221838+0000","last_undegraded":"2024-03-24T01:26:01.221838+0000","last_fullsized":"2024-03-24T01:26:01.221838+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:16:28.904097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.007123+0000","last_change":"2024-03-24T01:25:58.988228+0000","last_active":"2024-03-24T01:26:01.007123+0000","last_peered":"2024-03-24T01:26:01.007123+0000","last_clean":"2024-03-24T01:26:01.007123+0000","last_became_active":"2024-03-24T01:25:58.987941+0000","last_became_peered":"2024-03-24T01:25:58.987941+0000","last_unstale":"2024-03-24T01:26:01.007123+0000","last_undegraded":"2024-03-24T01:26:01.007123+0000","last_fullsized":"2024-03-24T01:26:01.007123+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:33:28.206717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221879+0000","last_change":"2024-03-24T01:26:00.000558+0000","last_active":"2024-03-24T01:26:01.221879+0000","last_peered":"2024-03-24T01:26:01.221879+0000","last_clean":"2024-03-24T01:26:01.221879+0000","last_became_active":"2024-03-24T01:26:00.000273+0000","last_became_peered":"2024-03-24T01:26:00.000273+0000","last_unstale":"2024-03-24T01:26:01.221879+0000","last_undegraded":"2024-03-24T01:26:01.221879+0000","last_fullsized":"2024-03-24T01:26:01.221879+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:16:28.904097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.006847+0000","last_change":"2024-03-24T01:25:58.991840+0000","last_active":"2024-03-24T01:26:01.006847+0000","last_peered":"2024-03-24T01:26:01.006847+0000","last_clean":"2024-03-24T01:26:01.006847+0000","last_became_active":"2024-03-24T01:25:58.991522+0000","last_became_peered":"2024-03-24T01:25:58.991522+0000","last_unstale":"2024-03-24T01:26:01.006847+0000","last_undegraded":"2024-03-24T01:26:01.006847+0000","last_fullsized":"2024-03-24T01:26:01.006847+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:29:31.808606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.011986+0000","last_change":"2024-03-24T01:26:00.005519+0000","last_active":"2024-03-24T01:26:01.011986+0000","last_peered":"2024-03-24T01:26:01.011986+0000","last_clean":"2024-03-24T01:26:01.011986+0000","last_became_active":"2024-03-24T01:26:00.005132+0000","last_became_peered":"2024-03-24T01:26:00.005132+0000","last_unstale":"2024-03-24T01:26:01.011986+0000","last_undegraded":"2024-03-24T01:26:01.011986+0000","last_fullsized":"2024-03-24T01:26:01.011986+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:02:08.397348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.222253+0000","last_change":"2024-03-24T01:25:59.006226+0000","last_active":"2024-03-24T01:26:01.222253+0000","last_peered":"2024-03-24T01:26:01.222253+0000","last_clean":"2024-03-24T01:26:01.222253+0000","last_became_active":"2024-03-24T01:25:59.005739+0000","last_became_peered":"2024-03-24T01:25:59.005739+0000","last_unstale":"2024-03-24T01:26:01.222253+0000","last_undegraded":"2024-03-24T01:26:01.222253+0000","last_fullsized":"2024-03-24T01:26:01.222253+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:08:38.108175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.997650+0000","last_change":"2024-03-24T01:26:00.016287+0000","last_active":"2024-03-24T01:26:00.997650+0000","last_peered":"2024-03-24T01:26:00.997650+0000","last_clean":"2024-03-24T01:26:00.997650+0000","last_became_active":"2024-03-24T01:26:00.015993+0000","last_became_peered":"2024-03-24T01:26:00.015993+0000","last_unstale":"2024-03-24T01:26:00.997650+0000","last_undegraded":"2024-03-24T01:26:00.997650+0000","last_fullsized":"2024-03-24T01:26:00.997650+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:29:19.471350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.006694+0000","last_change":"2024-03-24T01:25:58.988520+0000","last_active":"2024-03-24T01:26:01.006694+0000","last_peered":"2024-03-24T01:26:01.006694+0000","last_clean":"2024-03-24T01:26:01.006694+0000","last_became_active":"2024-03-24T01:25:58.988149+0000","last_became_peered":"2024-03-24T01:25:58.988149+0000","last_unstale":"2024-03-24T01:26:01.006694+0000","last_undegraded":"2024-03-24T01:26:01.006694+0000","last_fullsized":"2024-03-24T01:26:01.006694+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:22:00.973282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.222111+0000","last_change":"2024-03-24T01:26:00.005748+0000","last_active":"2024-03-24T01:26:01.222111+0000","last_peered":"2024-03-24T01:26:01.222111+0000","last_clean":"2024-03-24T01:26:01.222111+0000","last_became_active":"2024-03-24T01:26:00.005478+0000","last_became_peered":"2024-03-24T01:26:00.005478+0000","last_unstale":"2024-03-24T01:26:01.222111+0000","last_undegraded":"2024-03-24T01:26:01.222111+0000","last_fullsized":"2024-03-24T01:26:01.222111+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:02:08.397348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.013306+0000","last_change":"2024-03-24T01:25:59.005278+0000","last_active":"2024-03-24T01:26:01.013306+0000","last_peered":"2024-03-24T01:26:01.013306+0000","last_clean":"2024-03-24T01:26:01.013306+0000","last_became_active":"2024-03-24T01:25:59.004896+0000","last_became_peered":"2024-03-24T01:25:59.004896+0000","last_unstale":"2024-03-24T01:26:01.013306+0000","last_undegraded":"2024-03-24T01:26:01.013306+0000","last_fullsized":"2024-03-24T01:26:01.013306+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:24:39.889890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.222522+0000","last_change":"2024-03-24T01:26:00.002565+0000","last_active":"2024-03-24T01:26:01.222522+0000","last_peered":"2024-03-24T01:26:01.222522+0000","last_clean":"2024-03-24T01:26:01.222522+0000","last_became_active":"2024-03-24T01:25:59.993312+0000","last_became_peered":"2024-03-24T01:25:59.993312+0000","last_unstale":"2024-03-24T01:26:01.222522+0000","last_undegraded":"2024-03-24T01:26:01.222522+0000","last_fullsized":"2024-03-24T01:26:01.222522+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:07:24.167208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"39'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.007168+0000","last_change":"2024-03-24T01:25:59.004018+0000","last_active":"2024-03-24T01:26:01.007168+0000","last_peered":"2024-03-24T01:26:01.007168+0000","last_clean":"2024-03-24T01:26:01.007168+0000","last_became_active":"2024-03-24T01:25:59.003661+0000","last_became_peered":"2024-03-24T01:25:59.003661+0000","last_unstale":"2024-03-24T01:26:01.007168+0000","last_undegraded":"2024-03-24T01:26:01.007168+0000","last_fullsized":"2024-03-24T01:26:01.007168+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:35:58.556882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.997888+0000","last_change":"2024-03-24T01:26:00.016797+0000","last_active":"2024-03-24T01:26:00.997888+0000","last_peered":"2024-03-24T01:26:00.997888+0000","last_clean":"2024-03-24T01:26:00.997888+0000","last_became_active":"2024-03-24T01:26:00.016538+0000","last_became_peered":"2024-03-24T01:26:00.016538+0000","last_unstale":"2024-03-24T01:26:00.997888+0000","last_undegraded":"2024-03-24T01:26:00.997888+0000","last_fullsized":"2024-03-24T01:26:00.997888+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:00:25.321525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.222080+0000","last_change":"2024-03-24T01:25:58.995201+0000","last_active":"2024-03-24T01:26:01.222080+0000","last_peered":"2024-03-24T01:26:01.222080+0000","last_clean":"2024-03-24T01:26:01.222080+0000","last_became_active":"2024-03-24T01:25:58.994890+0000","last_became_peered":"2024-03-24T01:25:58.994890+0000","last_unstale":"2024-03-24T01:26:01.222080+0000","last_undegraded":"2024-03-24T01:26:01.222080+0000","last_fullsized":"2024-03-24T01:26:01.222080+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:08:38.108175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221007+0000","last_change":"2024-03-24T01:26:00.012718+0000","last_active":"2024-03-24T01:26:01.221007+0000","last_peered":"2024-03-24T01:26:01.221007+0000","last_clean":"2024-03-24T01:26:01.221007+0000","last_became_active":"2024-03-24T01:26:00.007605+0000","last_became_peered":"2024-03-24T01:26:00.007605+0000","last_unstale":"2024-03-24T01:26:01.221007+0000","last_undegraded":"2024-03-24T01:26:01.221007+0000","last_fullsized":"2024-03-24T01:26:01.221007+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:20:43.942659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221587+0000","last_change":"2024-03-24T01:25:58.990591+0000","last_active":"2024-03-24T01:26:01.221587+0000","last_peered":"2024-03-24T01:26:01.221587+0000","last_clean":"2024-03-24T01:26:01.221587+0000","last_became_active":"2024-03-24T01:25:58.990269+0000","last_became_peered":"2024-03-24T01:25:58.990269+0000","last_unstale":"2024-03-24T01:26:01.221587+0000","last_undegraded":"2024-03-24T01:26:01.221587+0000","last_fullsized":"2024-03-24T01:26:01.221587+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:44:12.069166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221167+0000","last_change":"2024-03-24T01:26:00.003739+0000","last_active":"2024-03-24T01:26:01.221167+0000","last_peered":"2024-03-24T01:26:01.221167+0000","last_clean":"2024-03-24T01:26:01.221167+0000","last_became_active":"2024-03-24T01:26:00.003484+0000","last_became_peered":"2024-03-24T01:26:00.003484+0000","last_unstale":"2024-03-24T01:26:01.221167+0000","last_undegraded":"2024-03-24T01:26:01.221167+0000","last_fullsized":"2024-03-24T01:26:01.221167+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:20:43.942659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.000473+0000","last_change":"2024-03-24T01:25:58.988526+0000","last_active":"2024-03-24T01:26:01.000473+0000","last_peered":"2024-03-24T01:26:01.000473+0000","last_clean":"2024-03-24T01:26:01.000473+0000","last_became_active":"2024-03-24T01:25:58.987932+0000","last_became_peered":"2024-03-24T01:25:58.987932+0000","last_unstale":"2024-03-24T01:26:01.000473+0000","last_undegraded":"2024-03-24T01:26:01.000473+0000","last_fullsized":"2024-03-24T01:26:01.000473+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:44:12.069166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.000093+0000","last_change":"2024-03-24T01:26:00.020959+0000","last_active":"2024-03-24T01:26:01.000093+0000","last_peered":"2024-03-24T01:26:01.000093+0000","last_clean":"2024-03-24T01:26:01.000093+0000","last_became_active":"2024-03-24T01:26:00.020670+0000","last_became_peered":"2024-03-24T01:26:00.020670+0000","last_unstale":"2024-03-24T01:26:01.000093+0000","last_undegraded":"2024-03-24T01:26:01.000093+0000","last_fullsized":"2024-03-24T01:26:01.000093+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:39:38.186024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.000331+0000","last_change":"2024-03-24T01:25:58.992830+0000","last_active":"2024-03-24T01:26:01.000331+0000","last_peered":"2024-03-24T01:26:01.000331+0000","last_clean":"2024-03-24T01:26:01.000331+0000","last_became_active":"2024-03-24T01:25:58.991569+0000","last_became_peered":"2024-03-24T01:25:58.991569+0000","last_unstale":"2024-03-24T01:26:01.000331+0000","last_undegraded":"2024-03-24T01:26:01.000331+0000","last_fullsized":"2024-03-24T01:26:01.000331+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:55.200533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221854+0000","last_change":"2024-03-24T01:26:00.001774+0000","last_active":"2024-03-24T01:26:01.221854+0000","last_peered":"2024-03-24T01:26:01.221854+0000","last_clean":"2024-03-24T01:26:01.221854+0000","last_became_active":"2024-03-24T01:26:00.001470+0000","last_became_peered":"2024-03-24T01:26:00.001470+0000","last_unstale":"2024-03-24T01:26:01.221854+0000","last_undegraded":"2024-03-24T01:26:01.221854+0000","last_fullsized":"2024-03-24T01:26:01.221854+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:30:44.565229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:00.997557+0000","last_change":"2024-03-24T01:25:58.996128+0000","last_active":"2024-03-24T01:26:00.997557+0000","last_peered":"2024-03-24T01:26:00.997557+0000","last_clean":"2024-03-24T01:26:00.997557+0000","last_became_active":"2024-03-24T01:25:58.994831+0000","last_became_peered":"2024-03-24T01:25:58.994831+0000","last_unstale":"2024-03-24T01:26:00.997557+0000","last_undegraded":"2024-03-24T01:26:00.997557+0000","last_fullsized":"2024-03-24T01:26:00.997557+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:24:39.889890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.012865+0000","last_change":"2024-03-24T01:26:00.018659+0000","last_active":"2024-03-24T01:26:01.012865+0000","last_peered":"2024-03-24T01:26:01.012865+0000","last_clean":"2024-03-24T01:26:01.012865+0000","last_became_active":"2024-03-24T01:26:00.018365+0000","last_became_peered":"2024-03-24T01:26:00.018365+0000","last_unstale":"2024-03-24T01:26:01.012865+0000","last_undegraded":"2024-03-24T01:26:01.012865+0000","last_fullsized":"2024-03-24T01:26:01.012865+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:58.948130+0000","last_clean_scrub_stamp":"2024-03-24T01:25:58.948130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:43:00.446290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.222544+0000","last_change":"2024-03-24T01:25:59.008252+0000","last_active":"2024-03-24T01:26:01.222544+0000","last_peered":"2024-03-24T01:26:01.222544+0000","last_clean":"2024-03-24T01:26:01.222544+0000","last_became_active":"2024-03-24T01:25:59.007969+0000","last_became_peered":"2024-03-24T01:25:59.007969+0000","last_unstale":"2024-03-24T01:26:01.222544+0000","last_undegraded":"2024-03-24T01:26:01.222544+0000","last_fullsized":"2024-03-24T01:26:01.222544+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:25:57.934312+0000","last_clean_scrub_stamp":"2024-03-24T01:25:57.934312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:28:18.909603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":205,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T01:26:01.221970+0000","last_change":"2024-03-24T01:24:01.545855+0000","last_active":"2024-03-24T01:26:01.221970+0000","last_peered":"2024-03-24T01:26:01.221970+0000","last_clean":"2024-03-24T01:26:01.221970+0000","last_became_active":"2024-03-24T01:24:01.543811+0000","last_became_peered":"2024-03-24T01:24:01.543811+0000","last_unstale":"2024-03-24T01:26:01.221970+0000","last_undegraded":"2024-03-24T01:26:01.221970+0000","last_fullsized":"2024-03-24T01:26:01.221970+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_clean_scrub_stamp":"2024-03-24T01:23:59.861564+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:03.682631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574876,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28060,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343780,"statfs":{"total":96636764160,"available":96608030720,"internally_reserved":0,"allocated":1060864,"data_stored":789377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574876,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27464,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344376,"statfs":{"total":96636764160,"available":96608641024,"internally_reserved":0,"allocated":454656,"data_stored":197909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574877,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":458752,"data_stored":197439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574876,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":458752,"data_stored":197931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27476,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":466944,"data_stored":198973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574877,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":454656,"data_stored":197397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574876,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27464,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344376,"statfs":{"total":96636764160,"available":96608641024,"internally_reserved":0,"allocated":454656,"data_stored":197417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574877,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343796,"statfs":{"total":96636764160,"available":96608047104,"internally_reserved":0,"allocated":1048576,"data_stored":787729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T01:26:09.973 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-24T01:26:09.973 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-24T01:26:10.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.230+0000 7fc373800640 1 Processor -- start 2024-03-24T01:26:10.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.231+0000 7fc373800640 1 -- start start 2024-03-24T01:26:10.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.232+0000 7fc373800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc36c0b41b0 0x7fc36c15bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:10.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.232+0000 7fc373800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc36c06f5c0 0x7fc36c0ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:10.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.232+0000 7fc373800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc36c06fa30 con 0x7fc36c1550a0 2024-03-24T01:26:10.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.232+0000 7fc373800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc36c155480 con 0x7fc36c06f5c0 2024-03-24T01:26:10.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.232+0000 7fc373800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc36c06fcc0 con 0x7fc36c0b41b0 2024-03-24T01:26:10.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.233+0000 7fc370e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc36c1550a0 0x7fc36c0b3aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51614/0 (socket says 172.21.15.90:51614) 2024-03-24T01:26:10.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.233+0000 7fc370e00640 1 -- 172.21.15.90:0/637255258 learned_addr learned my addr 172.21.15.90:0/637255258 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:26:10.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.233+0000 7fc36be00640 1 --2- 172.21.15.90:0/637255258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc36c0b41b0 0x7fc36c15bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:10.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.233+0000 7fc371800640 1 --2- 172.21.15.90:0/637255258 >> v2:172.21.15.120:3300/0 conn(0x7fc36c06f5c0 0x7fc36c0ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:10.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.233+0000 7fc36b400640 1 -- 172.21.15.90:0/637255258 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1548317562 0 0) 0x7fc36c06fa30 con 0x7fc36c1550a0 2024-03-24T01:26:10.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.233+0000 7fc36b400640 1 -- 172.21.15.90:0/637255258 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc354003520 con 0x7fc36c1550a0 2024-03-24T01:26:10.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.233+0000 7fc36b400640 1 -- 172.21.15.90:0/637255258 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3930669167 0 0) 0x7fc354003520 con 0x7fc36c1550a0 2024-03-24T01:26:10.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.233+0000 7fc36b400640 1 -- 172.21.15.90:0/637255258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc36c0b41b0 msgr2=0x7fc36c15bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:10.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.233+0000 7fc36b400640 1 --2- 172.21.15.90:0/637255258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc36c0b41b0 0x7fc36c15bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:10.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.233+0000 7fc36b400640 1 -- 172.21.15.90:0/637255258 >> v2:172.21.15.120:3300/0 conn(0x7fc36c06f5c0 msgr2=0x7fc36c0ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:10.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.233+0000 7fc36b400640 1 --2- 172.21.15.90:0/637255258 >> v2:172.21.15.120:3300/0 conn(0x7fc36c06f5c0 0x7fc36c0ae530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:10.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.233+0000 7fc36b400640 1 -- 172.21.15.90:0/637255258 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc36c15f4a0 con 0x7fc36c1550a0 2024-03-24T01:26:10.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.234+0000 7fc36b400640 1 -- 172.21.15.90:0/637255258 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7fc360001670 con 0x7fc36c1550a0 2024-03-24T01:26:10.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.234+0000 7fc36b400640 1 -- 172.21.15.90:0/637255258 >> v1:172.21.15.90:6789/0 conn(0x7fc36c1550a0 legacy=0x7fc36c0b3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:10.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.234+0000 7fc36b400640 1 --2- 172.21.15.90:0/637255258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc36c0b41b0 0x7fc3540039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:10.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.234+0000 7fc36b400640 1 --2- 172.21.15.90:0/637255258 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc354003f80 0x7fc354004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:10.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.234+0000 7fc36b400640 1 --2- 172.21.15.90:0/637255258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc354004d40 0x7fc3540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:10.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.234+0000 7fc36b400640 1 -- 172.21.15.90:0/637255258 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36c155480 con 0x7fc354003f80 2024-03-24T01:26:10.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.234+0000 7fc36b400640 1 -- 172.21.15.90:0/637255258 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36c06fcc0 con 0x7fc354004d40 2024-03-24T01:26:10.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.234+0000 7fc36b400640 1 -- 172.21.15.90:0/637255258 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc35400dc80 con 0x7fc36c0b41b0 2024-03-24T01:26:10.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.234+0000 7fc36be00640 1 --2- 172.21.15.90:0/637255258 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc354003f80 0x7fc354004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:10.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.234+0000 7fc370e00640 1 --2- 172.21.15.90:0/637255258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc36c0b41b0 0x7fc3540039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:10.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.234+0000 7fc371800640 1 --2- 172.21.15.90:0/637255258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc354004d40 0x7fc3540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:10.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.234+0000 7fc36be00640 1 -- 172.21.15.90:0/637255258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc36c0b41b0 msgr2=0x7fc3540039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:10.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.234+0000 7fc36be00640 1 --2- 172.21.15.90:0/637255258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc36c0b41b0 0x7fc3540039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:10.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.234+0000 7fc36be00640 1 -- 172.21.15.90:0/637255258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc354004d40 msgr2=0x7fc3540051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:10.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.234+0000 7fc36be00640 1 --2- 172.21.15.90:0/637255258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc354004d40 0x7fc3540051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:10.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.234+0000 7fc36be00640 1 -- 172.21.15.90:0/637255258 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc354003520 con 0x7fc354003f80 2024-03-24T01:26:10.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.234+0000 7fc370e00640 1 --2- 172.21.15.90:0/637255258 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc36c0b41b0 0x7fc3540039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:10.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.236+0000 7fc36be00640 1 --2- 172.21.15.90:0/637255258 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc354003f80 0x7fc354004800 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fc358009ac0 tx=0x7fc354005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=490266cdd54e3163 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:10.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.236+0000 7fc36b400640 1 -- 172.21.15.90:0/637255258 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc358039070 con 0x7fc354003f80 2024-03-24T01:26:10.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.236+0000 7fc36b400640 1 -- 172.21.15.90:0/637255258 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35802c6b0 con 0x7fc354003f80 2024-03-24T01:26:10.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.236+0000 7fc36b400640 1 -- 172.21.15.90:0/637255258 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc358009690 con 0x7fc354003f80 2024-03-24T01:26:10.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.237+0000 7fc373800640 1 -- 172.21.15.90:0/637255258 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc354003f80 msgr2=0x7fc354004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:10.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.237+0000 7fc373800640 1 --2- 172.21.15.90:0/637255258 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc354003f80 0x7fc354004800 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fc358009ac0 tx=0x7fc354005830 comp rx=0 tx=0).stop 2024-03-24T01:26:10.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.237+0000 7fc371800640 1 -- 172.21.15.90:0/637255258 reap_dead start 2024-03-24T01:26:10.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.237+0000 7fc373800640 1 -- 172.21.15.90:0/637255258 shutdown_connections 2024-03-24T01:26:10.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.237+0000 7fc373800640 1 -- 172.21.15.90:0/637255258 >> 172.21.15.90:0/637255258 conn(0x7fc36c0cf7e0 msgr2=0x7fc36c0cfbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:10.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.237+0000 7fc373800640 1 -- 172.21.15.90:0/637255258 shutdown_connections 2024-03-24T01:26:10.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.237+0000 7fc373800640 1 -- 172.21.15.90:0/637255258 wait complete. 2024-03-24T01:26:10.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.239+0000 7fc373800640 1 Processor -- start 2024-03-24T01:26:10.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.239+0000 7fc373800640 1 -- start start 2024-03-24T01:26:10.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.240+0000 7fc373800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc354003f80 0x7fc36c142930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:10.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.240+0000 7fc373800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc36c157350 0x7fc36c06e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:10.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.240+0000 7fc373800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc36c06eb90 0x7fc36c145190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:10.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.240+0000 7fc373800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36c15eca0 con 0x7fc36c06eb90 2024-03-24T01:26:10.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.240+0000 7fc373800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36c15e9c0 con 0x7fc354003f80 2024-03-24T01:26:10.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.240+0000 7fc373800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc36c15ef80 con 0x7fc36c157350 2024-03-24T01:26:10.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.240+0000 7fc371800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc36c06eb90 0x7fc36c145190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:10.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.240+0000 7fc371800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc36c06eb90 0x7fc36c145190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40038/0 (socket says 172.21.15.90:40038) 2024-03-24T01:26:10.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.240+0000 7fc371800640 1 -- 172.21.15.90:0/2689362007 learned_addr learned my addr 172.21.15.90:0/2689362007 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:10.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.240+0000 7fc370e00640 1 --2- 172.21.15.90:0/2689362007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc354003f80 0x7fc36c142930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:10.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.241+0000 7fc36be00640 1 --2- 172.21.15.90:0/2689362007 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc36c157350 0x7fc36c06e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:10.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.241+0000 7fc371800640 1 -- 172.21.15.90:0/2689362007 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc36c157350 msgr2=0x7fc36c06e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:10.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.241+0000 7fc371800640 1 --2- 172.21.15.90:0/2689362007 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc36c157350 0x7fc36c06e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:10.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.241+0000 7fc371800640 1 -- 172.21.15.90:0/2689362007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc354003f80 msgr2=0x7fc36c142930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:10.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.241+0000 7fc371800640 1 --2- 172.21.15.90:0/2689362007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc354003f80 0x7fc36c142930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:10.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.241+0000 7fc371800640 1 -- 172.21.15.90:0/2689362007 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc36c142e70 con 0x7fc36c06eb90 2024-03-24T01:26:10.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.241+0000 7fc371800640 1 --2- 172.21.15.90:0/2689362007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc36c06eb90 0x7fc36c145190 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fc36c158b80 tx=0x7fc36400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:10.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.241+0000 7fc368c00640 1 -- 172.21.15.90:0/2689362007 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36401c070 con 0x7fc36c06eb90 2024-03-24T01:26:10.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.241+0000 7fc368c00640 1 -- 172.21.15.90:0/2689362007 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc364017400 con 0x7fc36c06eb90 2024-03-24T01:26:10.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.241+0000 7fc368c00640 1 -- 172.21.15.90:0/2689362007 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3640177f0 con 0x7fc36c06eb90 2024-03-24T01:26:10.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.241+0000 7fc373800640 1 -- 172.21.15.90:0/2689362007 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc36c145f00 con 0x7fc36c06eb90 2024-03-24T01:26:10.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.241+0000 7fc373800640 1 -- 172.21.15.90:0/2689362007 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc36c156b50 con 0x7fc36c06eb90 2024-03-24T01:26:10.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.243+0000 7fc373800640 1 -- 172.21.15.90:0/2689362007 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc36c068ad0 con 0x7fc36c06eb90 2024-03-24T01:26:10.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.244+0000 7fc368c00640 1 -- 172.21.15.90:0/2689362007 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc36401f070 con 0x7fc36c06eb90 2024-03-24T01:26:10.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.244+0000 7fc368c00640 1 --2- 172.21.15.90:0/2689362007 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc34c06bb70 0x7fc34c06e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:10.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.245+0000 7fc368c00640 1 -- 172.21.15.90:0/2689362007 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc36408abf0 con 0x7fc36c06eb90 2024-03-24T01:26:10.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.245+0000 7fc370e00640 1 --2- 172.21.15.90:0/2689362007 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc34c06bb70 0x7fc34c06e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:10.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.246+0000 7fc370e00640 1 --2- 172.21.15.90:0/2689362007 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc34c06bb70 0x7fc34c06e030 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fc360001eb0 tx=0x7fc360009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:10.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.253+0000 7fc368c00640 1 -- 172.21.15.90:0/2689362007 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc36c068ad0 con 0x7fc36c06eb90 2024-03-24T01:26:10.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.611+0000 7fc373800640 1 -- 172.21.15.90:0/2689362007 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fc36c04f400 con 0x7fc36c06eb90 2024-03-24T01:26:10.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.615+0000 7fc368c00640 1 -- 172.21.15.90:0/2689362007 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+147 (secure 0 0 0) 0x7fc36405de20 con 0x7fc36c06eb90 2024-03-24T01:26:10.615 INFO:teuthology.orchestra.run.smithi090.stdout:[client.0] 2024-03-24T01:26:10.615 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQB0gP9lLKWXMxAAwrrRG8zLSylpIEuUgM7q8g== 2024-03-24T01:26:10.615 INFO:teuthology.orchestra.run.smithi090.stdout: caps mds = "allow" 2024-03-24T01:26:10.615 INFO:teuthology.orchestra.run.smithi090.stdout: caps mon = "allow r" 2024-03-24T01:26:10.616 INFO:teuthology.orchestra.run.smithi090.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-24T01:26:10.616 INFO:teuthology.orchestra.run.smithi090.stderr:updated caps for client.0 2024-03-24T01:26:10.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.617+0000 7fc373800640 1 -- 172.21.15.90:0/2689362007 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc34c06bb70 msgr2=0x7fc34c06e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:10.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.617+0000 7fc373800640 1 --2- 172.21.15.90:0/2689362007 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc34c06bb70 0x7fc34c06e030 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fc360001eb0 tx=0x7fc360009040 comp rx=0 tx=0).stop 2024-03-24T01:26:10.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.617+0000 7fc373800640 1 -- 172.21.15.90:0/2689362007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc36c06eb90 msgr2=0x7fc36c145190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:10.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.617+0000 7fc373800640 1 --2- 172.21.15.90:0/2689362007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc36c06eb90 0x7fc36c145190 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fc36c158b80 tx=0x7fc36400f4a0 comp rx=0 tx=0).stop 2024-03-24T01:26:10.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.617+0000 7fc373800640 1 -- 172.21.15.90:0/2689362007 shutdown_connections 2024-03-24T01:26:10.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.617+0000 7fc373800640 1 --2- 172.21.15.90:0/2689362007 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc34c06bb70 0x7fc34c06e030 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:10.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.617+0000 7fc373800640 1 --2- 172.21.15.90:0/2689362007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc36c06eb90 0x7fc36c145190 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:10.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.617+0000 7fc373800640 1 --2- 172.21.15.90:0/2689362007 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc36c157350 0x7fc36c06e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:10.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.618+0000 7fc373800640 1 --2- 172.21.15.90:0/2689362007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc354003f80 0x7fc36c142930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:10.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.618+0000 7fc373800640 1 -- 172.21.15.90:0/2689362007 >> 172.21.15.90:0/2689362007 conn(0x7fc36c0cf7e0 msgr2=0x7fc36c1596c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:10.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.618+0000 7fc373800640 1 -- 172.21.15.90:0/2689362007 shutdown_connections 2024-03-24T01:26:10.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.618+0000 7fc373800640 1 -- 172.21.15.90:0/2689362007 wait complete. 2024-03-24T01:26:10.634 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:26:10.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.829+0000 7f40daa00640 1 Processor -- start 2024-03-24T01:26:10.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.830+0000 7f40daa00640 1 -- start start 2024-03-24T01:26:10.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.832+0000 7f40daa00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f40d4145570 0x7f40d415c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:10.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.832+0000 7f40daa00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40d415f7b0 0x7f40d406b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:10.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.832+0000 7f40daa00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40d406bfb0 con 0x7f40d406b600 2024-03-24T01:26:10.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.833+0000 7f40daa00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40d40cfca0 con 0x7f40d4145570 2024-03-24T01:26:10.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.833+0000 7f40daa00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40d415d1b0 con 0x7f40d415f7b0 2024-03-24T01:26:10.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.833+0000 7f40d3400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40d415f7b0 0x7f40d406b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:10.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.833+0000 7f40d3e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f40d4145570 0x7f40d415c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:10.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.833+0000 7f40d3e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f40d4145570 0x7f40d415c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.90:54340/0 (socket says 172.21.15.90:54340) 2024-03-24T01:26:10.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.833+0000 7f40d3e00640 1 -- 172.21.15.90:0/2980234759 learned_addr learned my addr 172.21.15.90:0/2980234759 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:10.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.834+0000 7f40d3400640 1 -- 172.21.15.90:0/2980234759 >> v2:172.21.15.120:3300/0 conn(0x7f40d4145570 msgr2=0x7f40d415c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:10.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.834+0000 7f40d3400640 1 --2- 172.21.15.90:0/2980234759 >> v2:172.21.15.120:3300/0 conn(0x7f40d4145570 0x7f40d415c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:10.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.834+0000 7f40d3400640 1 -- 172.21.15.90:0/2980234759 >> v1:172.21.15.120:6789/0 conn(0x7f40d406b600 legacy=0x7f40d4070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:10.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.834+0000 7f40d3400640 1 -- 172.21.15.90:0/2980234759 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d415c910 con 0x7f40d415f7b0 2024-03-24T01:26:10.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.834+0000 7f40d3e00640 1 --2- 172.21.15.90:0/2980234759 >> v2:172.21.15.120:3300/0 conn(0x7f40d4145570 0x7f40d415c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T01:26:10.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.835+0000 7f40d3400640 1 --2- 172.21.15.90:0/2980234759 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40d415f7b0 0x7f40d406b0c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f40c00097a0 tx=0x7f40c002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6c97ad8101d66d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:10.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.836+0000 7f40d2a00640 1 -- 172.21.15.90:0/2980234759 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c0035020 con 0x7f40d415f7b0 2024-03-24T01:26:10.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.836+0000 7f40d2a00640 1 -- 172.21.15.90:0/2980234759 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40c0031810 con 0x7f40d415f7b0 2024-03-24T01:26:10.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.836+0000 7f40d2a00640 1 -- 172.21.15.90:0/2980234759 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c0031be0 con 0x7f40d415f7b0 2024-03-24T01:26:10.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.837+0000 7f40daa00640 1 -- 172.21.15.90:0/2980234759 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40d415f7b0 msgr2=0x7f40d406b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:10.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.837+0000 7f40daa00640 1 --2- 172.21.15.90:0/2980234759 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40d415f7b0 0x7f40d406b0c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f40c00097a0 tx=0x7f40c002b8b0 comp rx=0 tx=0).stop 2024-03-24T01:26:10.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.837+0000 7f40daa00640 1 -- 172.21.15.90:0/2980234759 shutdown_connections 2024-03-24T01:26:10.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.837+0000 7f40daa00640 1 --2- 172.21.15.90:0/2980234759 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40d415f7b0 0x7f40d406b0c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:10.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.837+0000 7f40daa00640 1 --2- 172.21.15.90:0/2980234759 >> v2:172.21.15.120:3300/0 conn(0x7f40d4145570 0x7f40d415c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:10.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.837+0000 7f40daa00640 1 -- 172.21.15.90:0/2980234759 >> 172.21.15.90:0/2980234759 conn(0x7f40d40b2670 msgr2=0x7f40d40b2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:10.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.837+0000 7f40daa00640 1 -- 172.21.15.90:0/2980234759 shutdown_connections 2024-03-24T01:26:10.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.837+0000 7f40daa00640 1 -- 172.21.15.90:0/2980234759 wait complete. 2024-03-24T01:26:10.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.840+0000 7f40daa00640 1 Processor -- start 2024-03-24T01:26:10.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.840+0000 7f40daa00640 1 -- start start 2024-03-24T01:26:10.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.841+0000 7f40daa00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40d406b600 0x7f40d40b55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:10.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.841+0000 7f40daa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40d4145570 0x7f40d4155760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:10.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.841+0000 7f40daa00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40d415f7b0 0x7f40d40cea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:10.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.841+0000 7f40daa00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d415ced0 con 0x7f40d4145570 2024-03-24T01:26:10.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.841+0000 7f40daa00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d40cff60 con 0x7f40d406b600 2024-03-24T01:26:10.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.841+0000 7f40daa00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40d40d1400 con 0x7f40d415f7b0 2024-03-24T01:26:10.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.842+0000 7f40d3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40d4145570 0x7f40d4155760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:10.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.842+0000 7f40d3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40d4145570 0x7f40d4155760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40048/0 (socket says 172.21.15.90:40048) 2024-03-24T01:26:10.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.842+0000 7f40d3400640 1 -- 172.21.15.90:0/2579873145 learned_addr learned my addr 172.21.15.90:0/2579873145 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:10.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.842+0000 7f40d3e00640 1 --2- 172.21.15.90:0/2579873145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40d406b600 0x7f40d40b55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:10.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.842+0000 7f40d3400640 1 -- 172.21.15.90:0/2579873145 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40d415f7b0 msgr2=0x7f40d40cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:10.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.842+0000 7f40d8a00640 1 --2- 172.21.15.90:0/2579873145 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40d415f7b0 0x7f40d40cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:10.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.842+0000 7f40d3400640 1 --2- 172.21.15.90:0/2579873145 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40d415f7b0 0x7f40d40cea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:10.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.842+0000 7f40d3400640 1 -- 172.21.15.90:0/2579873145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40d406b600 msgr2=0x7f40d40b55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:10.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.842+0000 7f40d3400640 1 --2- 172.21.15.90:0/2579873145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40d406b600 0x7f40d40b55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:10.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.842+0000 7f40d3400640 1 -- 172.21.15.90:0/2579873145 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d4155ca0 con 0x7f40d4145570 2024-03-24T01:26:10.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.842+0000 7f40d3e00640 1 --2- 172.21.15.90:0/2579873145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40d406b600 0x7f40d40b55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:10.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.842+0000 7f40d3400640 1 --2- 172.21.15.90:0/2579873145 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40d4145570 0x7f40d4155760 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f40c0034040 tx=0x7f40c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:10.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.843+0000 7f40cbe00640 1 -- 172.21.15.90:0/2579873145 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c0040070 con 0x7f40d4145570 2024-03-24T01:26:10.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.843+0000 7f40cbe00640 1 -- 172.21.15.90:0/2579873145 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40c0039c70 con 0x7f40d4145570 2024-03-24T01:26:10.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.843+0000 7f40cbe00640 1 -- 172.21.15.90:0/2579873145 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c0035040 con 0x7f40d4145570 2024-03-24T01:26:10.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.843+0000 7f40daa00640 1 -- 172.21.15.90:0/2579873145 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40d40b3fa0 con 0x7f40d4145570 2024-03-24T01:26:10.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.843+0000 7f40daa00640 1 -- 172.21.15.90:0/2579873145 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40d40b4560 con 0x7f40d4145570 2024-03-24T01:26:10.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.844+0000 7f40daa00640 1 -- 172.21.15.90:0/2579873145 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40d415adb0 con 0x7f40d4145570 2024-03-24T01:26:10.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.845+0000 7f40cbe00640 1 -- 172.21.15.90:0/2579873145 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f40c0038020 con 0x7f40d4145570 2024-03-24T01:26:10.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.846+0000 7f40cbe00640 1 --2- 172.21.15.90:0/2579873145 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f40b406bbd0 0x7f40b406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:10.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.846+0000 7f40cbe00640 1 -- 172.21.15.90:0/2579873145 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f40c00ac940 con 0x7f40d4145570 2024-03-24T01:26:10.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.846+0000 7f40d3e00640 1 --2- 172.21.15.90:0/2579873145 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f40b406bbd0 0x7f40b406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:10.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.848+0000 7f40d3e00640 1 --2- 172.21.15.90:0/2579873145 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f40b406bbd0 0x7f40b406e090 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f40c4009630 tx=0x7f40c4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:10.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:10.850+0000 7f40cbe00640 1 -- 172.21.15.90:0/2579873145 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f40c007b2a0 con 0x7f40d4145570 2024-03-24T01:26:11.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.072+0000 7f40daa00640 1 -- 172.21.15.90:0/2579873145 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40d4068770 con 0x7f40d4145570 2024-03-24T01:26:11.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.073+0000 7f40cbe00640 1 -- 172.21.15.90:0/2579873145 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f40c007aba0 con 0x7f40d4145570 2024-03-24T01:26:11.074 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:26:11.074 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/296084634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":296084634},{"type":"v1","addr":"172.21.15.120:6835","nonce":296084634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/797786252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":797786252},{"type":"v1","addr":"172.21.15.120:6837","nonce":797786252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5490,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6833/1983177399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1983177399},{"type":"v1","addr":"172.21.15.90:6833","nonce":1983177399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T01:25:59.440431+0000","modified":"2024-03-24T01:26:02.566388+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5496":{"gid":5496,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.90:6835/2166652035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":2166652035},{"type":"v1","addr":"172.21.15.90:6835","nonce":2166652035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5496,"qdb_cluster":[5496]},"id":2}]} 2024-03-24T01:26:11.074 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 17 2024-03-24T01:26:11.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.076+0000 7f40daa00640 1 -- 172.21.15.90:0/2579873145 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f40b406bbd0 msgr2=0x7f40b406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:11.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.076+0000 7f40daa00640 1 --2- 172.21.15.90:0/2579873145 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f40b406bbd0 0x7f40b406e090 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f40c4009630 tx=0x7f40c4008040 comp rx=0 tx=0).stop 2024-03-24T01:26:11.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.076+0000 7f40daa00640 1 -- 172.21.15.90:0/2579873145 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40d4145570 msgr2=0x7f40d4155760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:11.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.076+0000 7f40daa00640 1 --2- 172.21.15.90:0/2579873145 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40d4145570 0x7f40d4155760 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f40c0034040 tx=0x7f40c00046c0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.076+0000 7f40daa00640 1 -- 172.21.15.90:0/2579873145 shutdown_connections 2024-03-24T01:26:11.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.076+0000 7f40daa00640 1 --2- 172.21.15.90:0/2579873145 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f40b406bbd0 0x7f40b406e090 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.076+0000 7f40daa00640 1 --2- 172.21.15.90:0/2579873145 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40d4145570 0x7f40d4155760 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.076+0000 7f40daa00640 1 --2- 172.21.15.90:0/2579873145 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40d415f7b0 0x7f40d40cea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.076+0000 7f40daa00640 1 --2- 172.21.15.90:0/2579873145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40d406b600 0x7f40d40b55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.076+0000 7f40daa00640 1 -- 172.21.15.90:0/2579873145 >> 172.21.15.90:0/2579873145 conn(0x7f40d40b2670 msgr2=0x7f40d415fc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:11.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.077+0000 7f40daa00640 1 -- 172.21.15.90:0/2579873145 shutdown_connections 2024-03-24T01:26:11.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.077+0000 7f40daa00640 1 -- 172.21.15.90:0/2579873145 wait complete. 2024-03-24T01:26:11.496 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T01:25:59.440431+0000', 'modified': '2024-03-24T01:26:02.566388+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5496}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5496': {'gid': 5496, 'name': 'c', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.90:6835/2166652035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6834', 'nonce': 2166652035}, {'type': 'v1', 'addr': '172.21.15.90:6835', 'nonce': 2166652035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5496, 'qdb_cluster': [5496]} 2024-03-24T01:26:11.496 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T01:26:11.496 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-24T01:26:11.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.705+0000 7fbcd9800640 1 Processor -- start 2024-03-24T01:26:11.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.705+0000 7fbcd9800640 1 -- start start 2024-03-24T01:26:11.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.707+0000 7fbcd9800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbcd40b97b0 0x7fbcd40cead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:11.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.707+0000 7fbcd9800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fbcd40d3830 0x7fbcd40cf010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:11.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.707+0000 7fbcd9800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcd40d2790 con 0x7fbcd415de60 2024-03-24T01:26:11.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.707+0000 7fbcd9800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcd40d2550 con 0x7fbcd40d3830 2024-03-24T01:26:11.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.707+0000 7fbcd9800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcd40b7eb0 con 0x7fbcd40b97b0 2024-03-24T01:26:11.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.707+0000 7fbcd3400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fbcd40d3830 0x7fbcd40cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:11.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.707+0000 7fbcd3400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fbcd40d3830 0x7fbcd40cf010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40064/0 (socket says 172.21.15.90:40064) 2024-03-24T01:26:11.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.707+0000 7fbcd3400640 1 -- 172.21.15.90:0/927277634 learned_addr learned my addr 172.21.15.90:0/927277634 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:11.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.707+0000 7fbcd2000640 1 --2- 172.21.15.90:0/927277634 >> v2:172.21.15.120:3300/0 conn(0x7fbcd40b97b0 0x7fbcd40cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:11.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.707+0000 7fbcd3400640 1 -- 172.21.15.90:0/927277634 >> v2:172.21.15.120:3300/0 conn(0x7fbcd40b97b0 msgr2=0x7fbcd40cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:11.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.707+0000 7fbcd3400640 1 --2- 172.21.15.90:0/927277634 >> v2:172.21.15.120:3300/0 conn(0x7fbcd40b97b0 0x7fbcd40cead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.707+0000 7fbcd3400640 1 -- 172.21.15.90:0/927277634 >> v1:172.21.15.120:6789/0 conn(0x7fbcd415de60 legacy=0x7fbcd40d1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:11.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.707+0000 7fbcd3400640 1 -- 172.21.15.90:0/927277634 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd40d2a70 con 0x7fbcd40d3830 2024-03-24T01:26:11.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.709+0000 7fbcd3400640 1 --2- 172.21.15.90:0/927277634 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbcd40d3830 0x7fbcd40cf010 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fbcd40d04a0 tx=0x7fbccc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f131b6f6a902d04b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:11.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.709+0000 7fbcd1600640 1 -- 172.21.15.90:0/927277634 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbccc02ca80 con 0x7fbcd40d3830 2024-03-24T01:26:11.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.709+0000 7fbcd1600640 1 -- 172.21.15.90:0/927277634 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbccc02cc20 con 0x7fbcd40d3830 2024-03-24T01:26:11.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.709+0000 7fbcd1600640 1 -- 172.21.15.90:0/927277634 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbccc0343f0 con 0x7fbcd40d3830 2024-03-24T01:26:11.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.709+0000 7fbcd9800640 1 -- 172.21.15.90:0/927277634 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbcd40d3830 msgr2=0x7fbcd40cf010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:11.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.709+0000 7fbcd9800640 1 --2- 172.21.15.90:0/927277634 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbcd40d3830 0x7fbcd40cf010 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fbcd40d04a0 tx=0x7fbccc02bee0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.710+0000 7fbcd9800640 1 -- 172.21.15.90:0/927277634 shutdown_connections 2024-03-24T01:26:11.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.710+0000 7fbcd9800640 1 --2- 172.21.15.90:0/927277634 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbcd40d3830 0x7fbcd40cf010 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.710+0000 7fbcd9800640 1 --2- 172.21.15.90:0/927277634 >> v2:172.21.15.120:3300/0 conn(0x7fbcd40b97b0 0x7fbcd40cead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.710+0000 7fbcd9800640 1 -- 172.21.15.90:0/927277634 >> 172.21.15.90:0/927277634 conn(0x7fbcd4072c10 msgr2=0x7fbcd4073020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:11.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.710+0000 7fbcd9800640 1 -- 172.21.15.90:0/927277634 shutdown_connections 2024-03-24T01:26:11.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.710+0000 7fbcd9800640 1 -- 172.21.15.90:0/927277634 wait complete. 2024-03-24T01:26:11.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.711+0000 7fbcd9800640 1 Processor -- start 2024-03-24T01:26:11.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.712+0000 7fbcd9800640 1 -- start start 2024-03-24T01:26:11.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.712+0000 7fbcd9800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbcd40d3830 0x7fbcd40cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:11.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.712+0000 7fbcd9800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbcd415de60 0x7fbcd40c9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:11.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.712+0000 7fbcd9800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbcd4158a50 0x7fbcd41573c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:11.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.712+0000 7fbcd9800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd40d3310 con 0x7fbcd415de60 2024-03-24T01:26:11.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.712+0000 7fbcd9800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd40d3030 con 0x7fbcd4158a50 2024-03-24T01:26:11.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.712+0000 7fbcd9800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd415e240 con 0x7fbcd40d3830 2024-03-24T01:26:11.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.713+0000 7fbcd2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbcd415de60 0x7fbcd40c9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:11.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.713+0000 7fbcd2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbcd415de60 0x7fbcd40c9f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40078/0 (socket says 172.21.15.90:40078) 2024-03-24T01:26:11.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.713+0000 7fbcd2000640 1 -- 172.21.15.90:0/3537362500 learned_addr learned my addr 172.21.15.90:0/3537362500 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:11.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.713+0000 7fbcd2000640 1 -- 172.21.15.90:0/3537362500 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbcd40d3830 msgr2=0x7fbcd40cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:11.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.713+0000 7fbcd2000640 1 --2- 172.21.15.90:0/3537362500 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbcd40d3830 0x7fbcd40cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.713+0000 7fbcd2000640 1 -- 172.21.15.90:0/3537362500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbcd4158a50 msgr2=0x7fbcd41573c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:11.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.713+0000 7fbcd2000640 1 --2- 172.21.15.90:0/3537362500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbcd4158a50 0x7fbcd41573c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.713+0000 7fbcd2000640 1 -- 172.21.15.90:0/3537362500 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd40cae70 con 0x7fbcd415de60 2024-03-24T01:26:11.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.713+0000 7fbcd2000640 1 --2- 172.21.15.90:0/3537362500 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbcd415de60 0x7fbcd40c9f60 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fbcc000d800 tx=0x7fbcc000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:11.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.714+0000 7fbccaa00640 1 -- 172.21.15.90:0/3537362500 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc000b6c0 con 0x7fbcd415de60 2024-03-24T01:26:11.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.714+0000 7fbccaa00640 1 -- 172.21.15.90:0/3537362500 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc0019070 con 0x7fbcd415de60 2024-03-24T01:26:11.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.714+0000 7fbcd9800640 1 -- 172.21.15.90:0/3537362500 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd40cb150 con 0x7fbcd415de60 2024-03-24T01:26:11.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.714+0000 7fbccaa00640 1 -- 172.21.15.90:0/3537362500 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc000ba80 con 0x7fbcd415de60 2024-03-24T01:26:11.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.714+0000 7fbcd9800640 1 -- 172.21.15.90:0/3537362500 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcd40cb710 con 0x7fbcd415de60 2024-03-24T01:26:11.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.716+0000 7fbccaa00640 1 -- 172.21.15.90:0/3537362500 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fbcc000bc20 con 0x7fbcd415de60 2024-03-24T01:26:11.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.717+0000 7fbccaa00640 1 --2- 172.21.15.90:0/3537362500 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbcb406bbd0 0x7fbcb406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:11.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.717+0000 7fbccaa00640 1 -- 172.21.15.90:0/3537362500 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbcc008b110 con 0x7fbcd415de60 2024-03-24T01:26:11.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.717+0000 7fbcd2a00640 1 --2- 172.21.15.90:0/3537362500 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbcb406bbd0 0x7fbcb406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:11.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.719+0000 7fbcd2a00640 1 --2- 172.21.15.90:0/3537362500 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbcb406bbd0 0x7fbcb406e090 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4009650 tx=0x7fbcc4006ba0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:11.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.719+0000 7fbcdc5c9740 1 Processor -- start 2024-03-24T01:26:11.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.720+0000 7fbcdc5c9740 1 -- start start 2024-03-24T01:26:11.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.721+0000 7fbcdc5c9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de6e9d1bd0 0x55de6e9d1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:11.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.721+0000 7fbcdc5c9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de6e9a1bf0 0x55de6e9cee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:11.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.721+0000 7fbcdc5c9740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55de6e9a5750 0x55de6e9cf380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:11.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.721+0000 7fbcdc5c9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de6e90bc90 con 0x55de6e9d1bd0 2024-03-24T01:26:11.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.721+0000 7fbcdc5c9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de6e8b4980 con 0x55de6e9a1bf0 2024-03-24T01:26:11.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.721+0000 7fbcdc5c9740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55de6e907d30 con 0x55de6e9a5750 2024-03-24T01:26:11.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.721+0000 7fbcd2a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de6e9d1bd0 0x55de6e9d1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:11.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.721+0000 7fbcd2a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de6e9d1bd0 0x55de6e9d1fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40088/0 (socket says 172.21.15.90:40088) 2024-03-24T01:26:11.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.721+0000 7fbcd2a00640 1 -- 172.21.15.90:0/2492232204 learned_addr learned my addr 172.21.15.90:0/2492232204 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:11.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.722+0000 7fbcd2a00640 1 -- 172.21.15.90:0/2492232204 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55de6e9a5750 msgr2=0x55de6e9cf380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:11.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.722+0000 7fbcd2a00640 1 --2- 172.21.15.90:0/2492232204 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55de6e9a5750 0x55de6e9cf380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.722+0000 7fbcd2a00640 1 -- 172.21.15.90:0/2492232204 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de6e9a1bf0 msgr2=0x55de6e9cee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:11.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.722+0000 7fbcd2000640 1 --2- 172.21.15.90:0/2492232204 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de6e9a1bf0 0x55de6e9cee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:11.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.722+0000 7fbcd2a00640 1 --2- 172.21.15.90:0/2492232204 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de6e9a1bf0 0x55de6e9cee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.722+0000 7fbcd2a00640 1 -- 172.21.15.90:0/2492232204 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de6e8cc510 con 0x55de6e9d1bd0 2024-03-24T01:26:11.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.722+0000 7fbcd2a00640 1 --2- 172.21.15.90:0/2492232204 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de6e9d1bd0 0x55de6e9d1fb0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4013850 tx=0x7fbcc400e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e20de31e7342099 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:11.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.722+0000 7fbcbf400640 1 -- 172.21.15.90:0/2492232204 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc400bae0 con 0x55de6e9d1bd0 2024-03-24T01:26:11.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.722+0000 7fbcbf400640 1 -- 172.21.15.90:0/2492232204 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc4022070 con 0x55de6e9d1bd0 2024-03-24T01:26:11.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.722+0000 7fbcbf400640 1 -- 172.21.15.90:0/2492232204 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc401d3f0 con 0x55de6e9d1bd0 2024-03-24T01:26:11.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.723+0000 7fbcdc5c9740 1 -- 172.21.15.90:0/2492232204 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de6e9d1bd0 msgr2=0x55de6e9d1fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:11.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.723+0000 7fbcdc5c9740 1 --2- 172.21.15.90:0/2492232204 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de6e9d1bd0 0x55de6e9d1fb0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4013850 tx=0x7fbcc400e000 comp rx=0 tx=0).stop 2024-03-24T01:26:11.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.723+0000 7fbcdc5c9740 1 -- 172.21.15.90:0/2492232204 shutdown_connections 2024-03-24T01:26:11.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.723+0000 7fbcdc5c9740 1 --2- 172.21.15.90:0/2492232204 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55de6e9a5750 0x55de6e9cf380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.723+0000 7fbcdc5c9740 1 --2- 172.21.15.90:0/2492232204 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de6e9a1bf0 0x55de6e9cee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.723+0000 7fbcdc5c9740 1 --2- 172.21.15.90:0/2492232204 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de6e9d1bd0 0x55de6e9d1fb0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.723+0000 7fbcdc5c9740 1 -- 172.21.15.90:0/2492232204 >> 172.21.15.90:0/2492232204 conn(0x55de6e9a3e60 msgr2=0x55de6e9ec9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:11.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.724+0000 7fbcdc5c9740 1 -- 172.21.15.90:0/2492232204 shutdown_connections 2024-03-24T01:26:11.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.724+0000 7fbcdc5c9740 1 -- 172.21.15.90:0/2492232204 wait complete. 2024-03-24T01:26:11.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.724+0000 7fbcdc5c9740 1 -- start start 2024-03-24T01:26:11.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.724+0000 7fbcdc5c9740 1 Processor -- start 2024-03-24T01:26:11.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.725+0000 7fbcdc5c9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de6e9a5750 0x55de6e9c04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:11.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.725+0000 7fbcdc5c9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de6e9d1bd0 0x55de6e9d2510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:11.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.725+0000 7fbcdc5c9740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55de6e9d2a50 0x55de6e9c9480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:11.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.725+0000 7fbcdc5c9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de6e9a6900 con 0x55de6e9d1bd0 2024-03-24T01:26:11.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.725+0000 7fbcdc5c9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de6e8c7dd0 con 0x55de6e9a5750 2024-03-24T01:26:11.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.725+0000 7fbcdc5c9740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55de6e9a5cf0 con 0x55de6e9d2a50 2024-03-24T01:26:11.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.725+0000 7fbcdc5c9740 10 client.0 _ll_register_callbacks cb 0x55de6e61fac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T01:26:11.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.726+0000 7fbcd2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de6e9d1bd0 0x55de6e9d2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:11.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.726+0000 7fbcd2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de6e9d1bd0 0x55de6e9d2510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40102/0 (socket says 172.21.15.90:40102) 2024-03-24T01:26:11.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.726+0000 7fbcd2000640 1 -- 172.21.15.90:0/475542670 learned_addr learned my addr 172.21.15.90:0/475542670 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:11.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.726+0000 7fbcd2000640 1 -- 172.21.15.90:0/475542670 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55de6e9d2a50 msgr2=0x55de6e9c9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:11.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.726+0000 7fbcd3400640 1 --2- 172.21.15.90:0/475542670 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55de6e9d2a50 0x55de6e9c9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:11.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.726+0000 7fbcd2000640 1 --2- 172.21.15.90:0/475542670 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55de6e9d2a50 0x55de6e9c9480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.726+0000 7fbcd2a00640 1 --2- 172.21.15.90:0/475542670 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de6e9a5750 0x55de6e9c04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:11.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.726+0000 7fbcd2000640 1 -- 172.21.15.90:0/475542670 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de6e9a5750 msgr2=0x55de6e9c04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:11.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.726+0000 7fbcd2000640 1 --2- 172.21.15.90:0/475542670 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de6e9a5750 0x55de6e9c04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.726+0000 7fbcd2000640 1 -- 172.21.15.90:0/475542670 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55de6e6202d0 con 0x55de6e9d1bd0 2024-03-24T01:26:11.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.726+0000 7fbcd2000640 1 --2- 172.21.15.90:0/475542670 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de6e9d1bd0 0x55de6e9d2510 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fbcc0061540 tx=0x7fbcc0059b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a982752ec1d30b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:11.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.726+0000 7fbcb3e00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-03-24T01:26:11.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.726+0000 7fbcb3e00640 1 -- 172.21.15.90:0/475542670 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc0061a70 con 0x55de6e9d1bd0 2024-03-24T01:26:11.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.727+0000 7fbcb3e00640 1 -- 172.21.15.90:0/475542670 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbcc0071170 con 0x55de6e9d1bd0 2024-03-24T01:26:11.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.727+0000 7fbcdc5c9740 1 -- 172.21.15.90:0/475542670 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbcc4022070 con 0x55de6e9d1bd0 2024-03-24T01:26:11.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.727+0000 7fbcb3e00640 1 -- 172.21.15.90:0/475542670 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbcc00890d0 con 0x55de6e9d1bd0 2024-03-24T01:26:11.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.727+0000 7fbcdc5c9740 10 client.5682 fetch_fsmap learned FSMap version 17 2024-03-24T01:26:11.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.727+0000 7fbcdc5c9740 1 -- 172.21.15.90:0/475542670 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55de6e6ea6b0 con 0x55de6e9d1bd0 2024-03-24T01:26:11.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.727+0000 7fbcb3e00640 1 -- 172.21.15.90:0/475542670 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc4022070 con 0x55de6e9d1bd0 2024-03-24T01:26:11.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.727+0000 7fbcb3e00640 1 -- 172.21.15.90:0/475542670 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 17) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fbcc0013070 con 0x55de6e9d1bd0 2024-03-24T01:26:11.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.727+0000 7fbcdc5c9740 10 client.5682 fetch_fsmap finished waiting for FSMap version 17 2024-03-24T01:26:11.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.727+0000 7fbcdc5c9740 10 client.5682 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T01:26:11.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.727+0000 7fbcdc5c9740 20 client.5682 populate_metadata read hostname 'smithi090' 2024-03-24T01:26:11.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.727+0000 7fbcdc5c9740 1 --2- 172.21.15.90:0/475542670 >> [v2:172.21.15.90:6834/2166652035,v1:172.21.15.90:6835/2166652035] conn(0x55de6ea005f0 0x55de6ea009d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:11.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.727+0000 7fbcdc5c9740 4 client.5682 mds_command: new command op to 5496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T01:26:11.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.727+0000 7fbcdc5c9740 1 -- 172.21.15.90:0/475542670 --> [v2:172.21.15.90:6834/2166652035,v1:172.21.15.90:6835/2166652035] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55de6e8ad520 con 0x55de6ea005f0 2024-03-24T01:26:11.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.728+0000 7fbcd2a00640 1 --2- 172.21.15.90:0/475542670 >> [v2:172.21.15.90:6834/2166652035,v1:172.21.15.90:6835/2166652035] conn(0x55de6ea005f0 0x55de6ea009d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:11.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.728+0000 7fbcd2a00640 1 --2- 172.21.15.90:0/475542670 >> [v2:172.21.15.90:6834/2166652035,v1:172.21.15.90:6835/2166652035] conn(0x55de6ea005f0 0x55de6ea009d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfecb37f5a499789 server_cookie=82b66d05a287a5b6 in_seq=0 out_seq=0 2024-03-24T01:26:11.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.728+0000 7fbcb3e00640 10 client.5682 ms_handle_connect on v2:172.21.15.90:6834/2166652035 2024-03-24T01:26:11.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.729+0000 7fbcb3e00640 1 -- 172.21.15.90:0/475542670 <== mds.0 v2:172.21.15.90:6834/2166652035 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55de6e8ad520 con 0x55de6ea005f0 2024-03-24T01:26:11.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.729+0000 7fbcb3e00640 10 client.5682 handle_command_reply: tid=0 2024-03-24T01:26:11.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.729+0000 7fbcdc5c9740 1 client.5682 shutdown 2024-03-24T01:26:11.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.729+0000 7fbcdc5c9740 1 -- 172.21.15.90:0/475542670 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de6e9d1bd0 msgr2=0x55de6e9d2510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:11.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.729+0000 7fbcdc5c9740 1 --2- 172.21.15.90:0/475542670 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de6e9d1bd0 0x55de6e9d2510 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fbcc0061540 tx=0x7fbcc0059b20 comp rx=0 tx=0).stop 2024-03-24T01:26:11.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.729+0000 7fbcdc5c9740 1 -- 172.21.15.90:0/475542670 shutdown_connections 2024-03-24T01:26:11.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.729+0000 7fbcdc5c9740 1 --2- 172.21.15.90:0/475542670 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de6e9d1bd0 0x55de6e9d2510 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.729+0000 7fbcdc5c9740 1 --2- 172.21.15.90:0/475542670 >> [v2:172.21.15.90:6834/2166652035,v1:172.21.15.90:6835/2166652035] conn(0x55de6ea005f0 0x55de6ea009d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.729+0000 7fbcdc5c9740 1 --2- 172.21.15.90:0/475542670 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55de6e9d2a50 0x55de6e9c9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.729+0000 7fbcdc5c9740 1 --2- 172.21.15.90:0/475542670 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de6e9a5750 0x55de6e9c04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.729+0000 7fbcdc5c9740 1 -- 172.21.15.90:0/475542670 >> 172.21.15.90:0/475542670 conn(0x55de6e9a1bf0 msgr2=0x55de6e9a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:11.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.730+0000 7fbcb3e00640 0 client.5682 ms_handle_reset on v2:172.21.15.90:6834/2166652035 2024-03-24T01:26:11.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.730+0000 7fbcdc5c9740 1 -- 172.21.15.90:0/475542670 shutdown_connections 2024-03-24T01:26:11.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.730+0000 7fbcdc5c9740 1 -- 172.21.15.90:0/475542670 wait complete. 2024-03-24T01:26:11.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.730+0000 7fbcdc5c9740 20 client.5682 trim_cache size 0 max 16384 2024-03-24T01:26:11.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.766+0000 7fbcdc5c9740 1 Processor -- start 2024-03-24T01:26:11.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.766+0000 7fbcdc5c9740 1 -- start start 2024-03-24T01:26:11.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.768+0000 7fbcdc5c9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de6e9a5750 0x55de6e9d2510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:11.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.768+0000 7fbcdc5c9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de6e9d1bd0 0x55de6ea06e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:11.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.768+0000 7fbcdc5c9740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55de6e9d2a50 0x55de6e9c86d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:11.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.768+0000 7fbcdc5c9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd41ff920 con 0x55de6e9d1bd0 2024-03-24T01:26:11.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.768+0000 7fbcdc5c9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd41fed50 con 0x55de6e9a5750 2024-03-24T01:26:11.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.768+0000 7fbcdc5c9740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd41ff320 con 0x55de6e9d2a50 2024-03-24T01:26:11.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.768+0000 7fbcd2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de6e9d1bd0 0x55de6ea06e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:11.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.768+0000 7fbcd2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de6e9d1bd0 0x55de6ea06e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40118/0 (socket says 172.21.15.90:40118) 2024-03-24T01:26:11.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.768+0000 7fbcd2000640 1 -- 172.21.15.90:0/1871801988 learned_addr learned my addr 172.21.15.90:0/1871801988 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:11.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.769+0000 7fbcd2000640 1 -- 172.21.15.90:0/1871801988 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55de6e9d2a50 msgr2=0x55de6e9c86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:11.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.769+0000 7fbcd2000640 1 --2- 172.21.15.90:0/1871801988 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55de6e9d2a50 0x55de6e9c86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.769+0000 7fbcd2a00640 1 --2- 172.21.15.90:0/1871801988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de6e9a5750 0x55de6e9d2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:11.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.769+0000 7fbcd2000640 1 -- 172.21.15.90:0/1871801988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de6e9a5750 msgr2=0x55de6e9d2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:11.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.769+0000 7fbcd2000640 1 --2- 172.21.15.90:0/1871801988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de6e9a5750 0x55de6e9d2510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.769+0000 7fbcd2000640 1 -- 172.21.15.90:0/1871801988 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de6ea10d60 con 0x55de6e9d1bd0 2024-03-24T01:26:11.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.769+0000 7fbcd2000640 1 --2- 172.21.15.90:0/1871801988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de6e9d1bd0 0x55de6ea06e50 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fbcc00557c0 tx=0x7fbcc0020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8be192dedd9401c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:11.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.769+0000 7fbcbea00640 1 -- 172.21.15.90:0/1871801988 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc0061a70 con 0x55de6e9d1bd0 2024-03-24T01:26:11.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.769+0000 7fbcbea00640 1 -- 172.21.15.90:0/1871801988 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc005d1c0 con 0x55de6e9d1bd0 2024-03-24T01:26:11.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.769+0000 7fbcbea00640 1 -- 172.21.15.90:0/1871801988 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc00559d0 con 0x55de6e9d1bd0 2024-03-24T01:26:11.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.770+0000 7fbcdc5c9740 1 -- 172.21.15.90:0/1871801988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de6e9d1bd0 msgr2=0x55de6ea06e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:11.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.770+0000 7fbcdc5c9740 1 --2- 172.21.15.90:0/1871801988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de6e9d1bd0 0x55de6ea06e50 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fbcc00557c0 tx=0x7fbcc0020000 comp rx=0 tx=0).stop 2024-03-24T01:26:11.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.770+0000 7fbcdc5c9740 1 -- 172.21.15.90:0/1871801988 shutdown_connections 2024-03-24T01:26:11.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.770+0000 7fbcdc5c9740 1 --2- 172.21.15.90:0/1871801988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de6e9d1bd0 0x55de6ea06e50 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.770+0000 7fbcdc5c9740 1 --2- 172.21.15.90:0/1871801988 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55de6e9d2a50 0x55de6e9c86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.770+0000 7fbcdc5c9740 1 --2- 172.21.15.90:0/1871801988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de6e9a5750 0x55de6e9d2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.770+0000 7fbcdc5c9740 1 -- 172.21.15.90:0/1871801988 >> 172.21.15.90:0/1871801988 conn(0x55de6e9a1bf0 msgr2=0x55de6e99cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:11.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.770+0000 7fbcdc5c9740 1 -- 172.21.15.90:0/1871801988 shutdown_connections 2024-03-24T01:26:11.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.770+0000 7fbcdc5c9740 1 -- 172.21.15.90:0/1871801988 wait complete. 2024-03-24T01:26:11.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.771+0000 7fbcdc5c9740 1 -- start start 2024-03-24T01:26:11.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.772+0000 7fbcdc5c9740 1 Processor -- start 2024-03-24T01:26:11.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.773+0000 7fbcdc5c9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de6e9a5750 0x55de6e9cd390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:11.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.773+0000 7fbcdc5c9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de6e9d1bd0 0x55de6e9ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:11.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.773+0000 7fbcdc5c9740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55de6e9d2a50 0x55de6e9ef5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:11.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.773+0000 7fbcdc5c9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de6e9cc460 con 0x55de6e9d1bd0 2024-03-24T01:26:11.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.773+0000 7fbcdc5c9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de6e9d5c70 con 0x55de6e9a5750 2024-03-24T01:26:11.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.773+0000 7fbcdc5c9740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55de6e9cb470 con 0x55de6e9d2a50 2024-03-24T01:26:11.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.773+0000 7fbcdc5c9740 10 client.0 _ll_register_callbacks cb 0x7fbcc00046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T01:26:11.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.774+0000 7fbcd2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de6e9d1bd0 0x55de6e9ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:11.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.774+0000 7fbcd2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de6e9d1bd0 0x55de6e9ca2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40126/0 (socket says 172.21.15.90:40126) 2024-03-24T01:26:11.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.774+0000 7fbcd2000640 1 -- 172.21.15.90:0/1394939261 learned_addr learned my addr 172.21.15.90:0/1394939261 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:11.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.774+0000 7fbcd2000640 1 -- 172.21.15.90:0/1394939261 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55de6e9d2a50 msgr2=0x55de6e9ef5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:11.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.774+0000 7fbcd2000640 1 --2- 172.21.15.90:0/1394939261 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55de6e9d2a50 0x55de6e9ef5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.774+0000 7fbcd2000640 1 -- 172.21.15.90:0/1394939261 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de6e9a5750 msgr2=0x55de6e9cd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:11.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.774+0000 7fbcd2000640 1 --2- 172.21.15.90:0/1394939261 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de6e9a5750 0x55de6e9cd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.774+0000 7fbcd2000640 1 -- 172.21.15.90:0/1394939261 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55de6ea5c270 con 0x55de6e9d1bd0 2024-03-24T01:26:11.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.774+0000 7fbcd2000640 1 --2- 172.21.15.90:0/1394939261 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de6e9d1bd0 0x55de6e9ca2a0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fbcc000b4b0 tx=0x7fbcc0089370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd4d69de9593383 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:11.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.774+0000 7fbcb3e00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-03-24T01:26:11.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.774+0000 7fbcb3e00640 1 -- 172.21.15.90:0/1394939261 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc0061a70 con 0x55de6e9d1bd0 2024-03-24T01:26:11.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.774+0000 7fbcb3e00640 1 -- 172.21.15.90:0/1394939261 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbcc0079e00 con 0x55de6e9d1bd0 2024-03-24T01:26:11.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.774+0000 7fbcdc5c9740 1 -- 172.21.15.90:0/1394939261 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbcc005d1c0 con 0x55de6e9d1bd0 2024-03-24T01:26:11.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.775+0000 7fbcb3e00640 1 -- 172.21.15.90:0/1394939261 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbcc00559d0 con 0x55de6e9d1bd0 2024-03-24T01:26:11.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.775+0000 7fbcdc5c9740 10 client.5688 fetch_fsmap learned FSMap version 17 2024-03-24T01:26:11.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.775+0000 7fbcdc5c9740 1 -- 172.21.15.90:0/1394939261 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55de6ea10250 con 0x55de6e9d1bd0 2024-03-24T01:26:11.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.775+0000 7fbcb3e00640 1 -- 172.21.15.90:0/1394939261 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc005d1c0 con 0x55de6e9d1bd0 2024-03-24T01:26:11.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.775+0000 7fbcb3e00640 1 -- 172.21.15.90:0/1394939261 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 17) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fbcc0081020 con 0x55de6e9d1bd0 2024-03-24T01:26:11.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.775+0000 7fbcdc5c9740 10 client.5688 fetch_fsmap finished waiting for FSMap version 17 2024-03-24T01:26:11.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.775+0000 7fbcdc5c9740 10 client.5688 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T01:26:11.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.775+0000 7fbcdc5c9740 20 client.5688 populate_metadata read hostname 'smithi090' 2024-03-24T01:26:11.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.775+0000 7fbcdc5c9740 1 --2- 172.21.15.90:0/1394939261 >> [v2:172.21.15.90:6834/2166652035,v1:172.21.15.90:6835/2166652035] conn(0x55de6e9ff120 0x55de6e9ff500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:11.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.775+0000 7fbcdc5c9740 4 client.5688 mds_command: new command op to 5496 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T01:26:11.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.775+0000 7fbcdc5c9740 1 -- 172.21.15.90:0/1394939261 --> [v2:172.21.15.90:6834/2166652035,v1:172.21.15.90:6835/2166652035] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55de6e9a23f0 con 0x55de6e9ff120 2024-03-24T01:26:11.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.775+0000 7fbcd2a00640 1 --2- 172.21.15.90:0/1394939261 >> [v2:172.21.15.90:6834/2166652035,v1:172.21.15.90:6835/2166652035] conn(0x55de6e9ff120 0x55de6e9ff500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:11.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.776+0000 7fbcd2a00640 1 --2- 172.21.15.90:0/1394939261 >> [v2:172.21.15.90:6834/2166652035,v1:172.21.15.90:6835/2166652035] conn(0x55de6e9ff120 0x55de6e9ff500 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=852b8abb46961f32 server_cookie=d217dbb4705647ab in_seq=0 out_seq=0 2024-03-24T01:26:11.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.776+0000 7fbcb3e00640 10 client.5688 ms_handle_connect on v2:172.21.15.90:6834/2166652035 2024-03-24T01:26:11.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.776+0000 7fbcb3e00640 1 -- 172.21.15.90:0/1394939261 <== mds.0 v2:172.21.15.90:6834/2166652035 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+319 (crc 0 0 0) 0x55de6e9a23f0 con 0x55de6e9ff120 2024-03-24T01:26:11.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.776+0000 7fbcb3e00640 10 client.5688 handle_command_reply: tid=0 2024-03-24T01:26:11.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.776+0000 7fbcdc5c9740 1 client.5688 shutdown 2024-03-24T01:26:11.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.776+0000 7fbcdc5c9740 1 -- 172.21.15.90:0/1394939261 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de6e9d1bd0 msgr2=0x55de6e9ca2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:11.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.776+0000 7fbcdc5c9740 1 --2- 172.21.15.90:0/1394939261 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de6e9d1bd0 0x55de6e9ca2a0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fbcc000b4b0 tx=0x7fbcc0089370 comp rx=0 tx=0).stop 2024-03-24T01:26:11.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.776+0000 7fbcdc5c9740 1 -- 172.21.15.90:0/1394939261 shutdown_connections 2024-03-24T01:26:11.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.776+0000 7fbcdc5c9740 1 --2- 172.21.15.90:0/1394939261 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55de6e9d1bd0 0x55de6e9ca2a0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.776+0000 7fbcdc5c9740 1 --2- 172.21.15.90:0/1394939261 >> [v2:172.21.15.90:6834/2166652035,v1:172.21.15.90:6835/2166652035] conn(0x55de6e9ff120 0x55de6e9ff500 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.776+0000 7fbcdc5c9740 1 --2- 172.21.15.90:0/1394939261 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55de6e9d2a50 0x55de6e9ef5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.776+0000 7fbcdc5c9740 1 --2- 172.21.15.90:0/1394939261 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de6e9a5750 0x55de6e9cd390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.776+0000 7fbcb3e00640 0 client.5688 ms_handle_reset on v2:172.21.15.90:6834/2166652035 2024-03-24T01:26:11.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.776+0000 7fbcdc5c9740 1 -- 172.21.15.90:0/1394939261 >> 172.21.15.90:0/1394939261 conn(0x55de6e9a1bf0 msgr2=0x55de6e99ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:11.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.776+0000 7fbcdc5c9740 1 -- 172.21.15.90:0/1394939261 shutdown_connections 2024-03-24T01:26:11.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.776+0000 7fbcdc5c9740 1 -- 172.21.15.90:0/1394939261 wait complete. 2024-03-24T01:26:11.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.777+0000 7fbcdc5c9740 20 client.5688 trim_cache size 0 max 16384 2024-03-24T01:26:11.780 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-24T01:26:11.780 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "24712165-9839-4fcc-8033-988386efea74", 2024-03-24T01:26:11.780 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 0, 2024-03-24T01:26:11.780 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 5496, 2024-03-24T01:26:11.780 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-03-24T01:26:11.780 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-03-24T01:26:11.780 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-03-24T01:26:11.780 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 11.790949312, 2024-03-24T01:26:11.780 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 17, 2024-03-24T01:26:11.781 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 39, 2024-03-24T01:26:11.781 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 39, 2024-03-24T01:26:11.781 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 20.620213647 2024-03-24T01:26:11.781 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-24T01:26:11.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.777+0000 7fbcd9800640 1 -- 172.21.15.90:0/3537362500 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbcb406bbd0 msgr2=0x7fbcb406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:11.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.777+0000 7fbcd9800640 1 --2- 172.21.15.90:0/3537362500 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbcb406bbd0 0x7fbcb406e090 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4009650 tx=0x7fbcc4006ba0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.777+0000 7fbcd9800640 1 -- 172.21.15.90:0/3537362500 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbcd415de60 msgr2=0x7fbcd40c9f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:11.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.777+0000 7fbcd9800640 1 --2- 172.21.15.90:0/3537362500 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbcd415de60 0x7fbcd40c9f60 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fbcc000d800 tx=0x7fbcc000dcc0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.777+0000 7fbcd9800640 1 -- 172.21.15.90:0/3537362500 shutdown_connections 2024-03-24T01:26:11.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.777+0000 7fbcd9800640 1 --2- 172.21.15.90:0/3537362500 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbcb406bbd0 0x7fbcb406e090 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.777+0000 7fbcd9800640 1 --2- 172.21.15.90:0/3537362500 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbcd415de60 0x7fbcd40c9f60 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.777+0000 7fbcd9800640 1 --2- 172.21.15.90:0/3537362500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbcd4158a50 0x7fbcd41573c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.777+0000 7fbcd9800640 1 --2- 172.21.15.90:0/3537362500 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbcd40d3830 0x7fbcd40cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.777+0000 7fbcd9800640 1 -- 172.21.15.90:0/3537362500 >> 172.21.15.90:0/3537362500 conn(0x7fbcd4072c10 msgr2=0x7fbcd40b9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:11.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.777+0000 7fbcd9800640 1 -- 172.21.15.90:0/3537362500 shutdown_connections 2024-03-24T01:26:11.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.777+0000 7fbcd9800640 1 -- 172.21.15.90:0/3537362500 wait complete. 2024-03-24T01:26:11.796 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T01:26:11.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.988+0000 7f2db1e00640 1 Processor -- start 2024-03-24T01:26:11.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.988+0000 7f2db1e00640 1 -- start start 2024-03-24T01:26:11.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.989+0000 7f2db1e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2dac15e0a0 0x7f2dac0ce430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:11.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.989+0000 7f2db1e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2dac0d0980 0x7f2dac0d0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:11.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.989+0000 7f2db1e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dac0cea10 con 0x7f2dac15dcc0 2024-03-24T01:26:11.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.989+0000 7f2db1e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dac0d1e60 con 0x7f2dac15e0a0 2024-03-24T01:26:11.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.989+0000 7f2db1e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac0cecf0 con 0x7f2dac0d0980 2024-03-24T01:26:11.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.990+0000 7f2daaa00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f2dac15dcc0 0x7f2dac162450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51628/0 (socket says 172.21.15.90:51628) 2024-03-24T01:26:11.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.990+0000 7f2daaa00640 1 -- 172.21.15.90:0/3599256441 learned_addr learned my addr 172.21.15.90:0/3599256441 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:26:11.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.990+0000 7f2daa000640 1 --2- 172.21.15.90:0/3599256441 >> v2:172.21.15.90:3300/0 conn(0x7f2dac15e0a0 0x7f2dac0ce430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:11.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.990+0000 7f2dab400640 1 --2- 172.21.15.90:0/3599256441 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2dac0d0980 0x7f2dac0d0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:11.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.990+0000 7f2da9600640 1 -- 172.21.15.90:0/3599256441 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 826334781 0 0) 0x7f2dac0cea10 con 0x7f2dac15dcc0 2024-03-24T01:26:11.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.990+0000 7f2da9600640 1 -- 172.21.15.90:0/3599256441 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d90003520 con 0x7f2dac15dcc0 2024-03-24T01:26:11.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.990+0000 7f2daa000640 1 -- 172.21.15.90:0/3599256441 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2dac0d0980 msgr2=0x7f2dac0d0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:11.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.990+0000 7f2daa000640 1 --2- 172.21.15.90:0/3599256441 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2dac0d0980 0x7f2dac0d0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.990+0000 7f2daa000640 1 -- 172.21.15.90:0/3599256441 >> v1:172.21.15.90:6789/0 conn(0x7f2dac15dcc0 legacy=0x7f2dac162450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:11.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.990+0000 7f2daa000640 1 -- 172.21.15.90:0/3599256441 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dac0cefd0 con 0x7f2dac15e0a0 2024-03-24T01:26:11.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.992+0000 7f2daa000640 1 --2- 172.21.15.90:0/3599256441 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2dac15e0a0 0x7f2dac0ce430 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f2d980097a0 tx=0x7f2d9802b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c57d5a293b6265ea server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:11.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.992+0000 7f2da9600640 1 -- 172.21.15.90:0/3599256441 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d98034070 con 0x7f2dac15e0a0 2024-03-24T01:26:11.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.992+0000 7f2da9600640 1 -- 172.21.15.90:0/3599256441 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d98030800 con 0x7f2dac15e0a0 2024-03-24T01:26:11.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.992+0000 7f2da9600640 1 -- 172.21.15.90:0/3599256441 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d98030bd0 con 0x7f2dac15e0a0 2024-03-24T01:26:11.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.992+0000 7f2db1e00640 1 -- 172.21.15.90:0/3599256441 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2dac15e0a0 msgr2=0x7f2dac0ce430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:11.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.992+0000 7f2db1e00640 1 --2- 172.21.15.90:0/3599256441 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2dac15e0a0 0x7f2dac0ce430 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f2d980097a0 tx=0x7f2d9802b940 comp rx=0 tx=0).stop 2024-03-24T01:26:11.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.992+0000 7f2db1e00640 1 -- 172.21.15.90:0/3599256441 shutdown_connections 2024-03-24T01:26:11.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.992+0000 7f2db1e00640 1 --2- 172.21.15.90:0/3599256441 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2dac0d0980 0x7f2dac0d0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.992+0000 7f2db1e00640 1 --2- 172.21.15.90:0/3599256441 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2dac15e0a0 0x7f2dac0ce430 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.992+0000 7f2db1e00640 1 -- 172.21.15.90:0/3599256441 >> 172.21.15.90:0/3599256441 conn(0x7f2dac06cac0 msgr2=0x7f2dac06ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:11.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.993+0000 7f2db1e00640 1 -- 172.21.15.90:0/3599256441 shutdown_connections 2024-03-24T01:26:11.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.993+0000 7f2db1e00640 1 -- 172.21.15.90:0/3599256441 wait complete. 2024-03-24T01:26:11.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.994+0000 7f2db1e00640 1 Processor -- start 2024-03-24T01:26:11.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.994+0000 7f2db1e00640 1 -- start start 2024-03-24T01:26:11.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.995+0000 7f2db1e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2dac0d0980 0x7f2dac156440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:11.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.995+0000 7f2db1e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2dac15dcc0 0x7f2dac15e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:11.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.995+0000 7f2db1e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2dac15e0a0 0x7f2dac0c8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:11.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.995+0000 7f2db1e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac0d2470 con 0x7f2dac15dcc0 2024-03-24T01:26:11.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.995+0000 7f2db1e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac0d22f0 con 0x7f2dac0d0980 2024-03-24T01:26:11.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.995+0000 7f2db1e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac0d25f0 con 0x7f2dac15e0a0 2024-03-24T01:26:11.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.996+0000 7f2daa000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2dac15dcc0 0x7f2dac15e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:11.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.996+0000 7f2daa000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2dac15dcc0 0x7f2dac15e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40144/0 (socket says 172.21.15.90:40144) 2024-03-24T01:26:11.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.996+0000 7f2daa000640 1 -- 172.21.15.90:0/1448839457 learned_addr learned my addr 172.21.15.90:0/1448839457 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:11.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.996+0000 7f2dab400640 1 --2- 172.21.15.90:0/1448839457 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2dac15e0a0 0x7f2dac0c8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:11.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.996+0000 7f2daaa00640 1 --2- 172.21.15.90:0/1448839457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2dac0d0980 0x7f2dac156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:11.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.996+0000 7f2daa000640 1 -- 172.21.15.90:0/1448839457 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2dac15e0a0 msgr2=0x7f2dac0c8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:11.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.996+0000 7f2daa000640 1 --2- 172.21.15.90:0/1448839457 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2dac15e0a0 0x7f2dac0c8b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.996+0000 7f2daa000640 1 -- 172.21.15.90:0/1448839457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2dac0d0980 msgr2=0x7f2dac156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:11.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.996+0000 7f2daa000640 1 --2- 172.21.15.90:0/1448839457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2dac0d0980 0x7f2dac156440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:11.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.996+0000 7f2daa000640 1 -- 172.21.15.90:0/1448839457 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dac15eda0 con 0x7f2dac15dcc0 2024-03-24T01:26:11.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.996+0000 7f2daa000640 1 --2- 172.21.15.90:0/1448839457 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2dac15dcc0 0x7f2dac15e4a0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f2d98009890 tx=0x7f2d980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:11.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.997+0000 7f2da3400640 1 -- 172.21.15.90:0/1448839457 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d98034040 con 0x7f2dac15dcc0 2024-03-24T01:26:11.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.997+0000 7f2da3400640 1 -- 172.21.15.90:0/1448839457 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d9803add0 con 0x7f2dac15dcc0 2024-03-24T01:26:11.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.997+0000 7f2da3400640 1 -- 172.21.15.90:0/1448839457 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d98006e10 con 0x7f2dac15dcc0 2024-03-24T01:26:11.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.997+0000 7f2db1e00640 1 -- 172.21.15.90:0/1448839457 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dac15f080 con 0x7f2dac15dcc0 2024-03-24T01:26:11.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.997+0000 7f2db1e00640 1 -- 172.21.15.90:0/1448839457 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dac15f640 con 0x7f2dac15dcc0 2024-03-24T01:26:11.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.998+0000 7f2db1e00640 1 -- 172.21.15.90:0/1448839457 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dac15d5d0 con 0x7f2dac15dcc0 2024-03-24T01:26:11.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:11.999+0000 7f2da3400640 1 -- 172.21.15.90:0/1448839457 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2d9804b060 con 0x7f2dac15dcc0 2024-03-24T01:26:12.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:12.001+0000 7f2da3400640 1 --2- 172.21.15.90:0/1448839457 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f2d8c06bbd0 0x7f2d8c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:12.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:12.001+0000 7f2da3400640 1 -- 172.21.15.90:0/1448839457 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2d980acbc0 con 0x7f2dac15dcc0 2024-03-24T01:26:12.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:12.001+0000 7f2daaa00640 1 --2- 172.21.15.90:0/1448839457 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f2d8c06bbd0 0x7f2d8c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:12.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:12.002+0000 7f2daaa00640 1 --2- 172.21.15.90:0/1448839457 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f2d8c06bbd0 0x7f2d8c06e090 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c000f60 tx=0x7f2d9c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:12.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:12.006+0000 7f2da3400640 1 -- 172.21.15.90:0/1448839457 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2dac15d5d0 con 0x7f2dac15dcc0 2024-03-24T01:26:12.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:12.246+0000 7f2db1e00640 1 -- 172.21.15.90:0/1448839457 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f2dac0ceab0 con 0x7f2dac15dcc0 2024-03-24T01:26:12.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:12.246+0000 7f2da3400640 1 -- 172.21.15.90:0/1448839457 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+73 (secure 0 0 0) 0x7f2dac0ceab0 con 0x7f2dac15dcc0 2024-03-24T01:26:12.247 INFO:teuthology.orchestra.run.smithi090.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T01:26:12.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:12.248+0000 7f2db1e00640 1 -- 172.21.15.90:0/1448839457 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f2d8c06bbd0 msgr2=0x7f2d8c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:12.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:12.248+0000 7f2db1e00640 1 --2- 172.21.15.90:0/1448839457 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f2d8c06bbd0 0x7f2d8c06e090 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c000f60 tx=0x7f2d9c009040 comp rx=0 tx=0).stop 2024-03-24T01:26:12.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:12.248+0000 7f2db1e00640 1 -- 172.21.15.90:0/1448839457 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2dac15dcc0 msgr2=0x7f2dac15e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:12.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:12.248+0000 7f2db1e00640 1 --2- 172.21.15.90:0/1448839457 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2dac15dcc0 0x7f2dac15e4a0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f2d98009890 tx=0x7f2d980046c0 comp rx=0 tx=0).stop 2024-03-24T01:26:12.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:12.249+0000 7f2db1e00640 1 -- 172.21.15.90:0/1448839457 shutdown_connections 2024-03-24T01:26:12.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:12.249+0000 7f2db1e00640 1 --2- 172.21.15.90:0/1448839457 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f2d8c06bbd0 0x7f2d8c06e090 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:12.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:12.249+0000 7f2db1e00640 1 --2- 172.21.15.90:0/1448839457 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2dac15dcc0 0x7f2dac15e4a0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:12.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:12.249+0000 7f2db1e00640 1 --2- 172.21.15.90:0/1448839457 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2dac15e0a0 0x7f2dac0c8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:12.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:12.249+0000 7f2db1e00640 1 --2- 172.21.15.90:0/1448839457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2dac0d0980 0x7f2dac156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:12.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:12.249+0000 7f2db1e00640 1 -- 172.21.15.90:0/1448839457 >> 172.21.15.90:0/1448839457 conn(0x7f2dac06cac0 msgr2=0x7f2dac15b920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:12.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:12.249+0000 7f2db1e00640 1 -- 172.21.15.90:0/1448839457 shutdown_connections 2024-03-24T01:26:12.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:12.249+0000 7f2db1e00640 1 -- 172.21.15.90:0/1448839457 wait complete. 2024-03-24T01:26:12.266 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T01:26:12.266 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T01:26:12.266 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi090.front.sepia.ceph.com 2024-03-24T01:26:12.266 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T01:26:12.267 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T01:26:12.267 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T01:26:12.267 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T01:26:12.267 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T01:26:12.267 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T01:26:12.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:12.267 DEBUG:teuthology.orchestra.run.smithi090:> ip addr 2024-03-24T01:26:12.315 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T01:26:12.315 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T01:26:12.316 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T01:26:12.316 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:26:12.316 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-03-24T01:26:12.316 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:26:12.316 INFO:teuthology.orchestra.run.smithi090.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T01:26:12.316 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a8 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:26:12.316 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f0 2024-03-24T01:26:12.316 INFO:teuthology.orchestra.run.smithi090.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T01:26:12.316 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a9 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:26:12.316 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f1 2024-03-24T01:26:12.316 INFO:teuthology.orchestra.run.smithi090.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T01:26:12.316 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:66 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:26:12.316 INFO:teuthology.orchestra.run.smithi090.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T01:26:12.317 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:67 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:26:12.317 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.21.15.90/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T01:26:12.317 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft 42410sec preferred_lft 42410sec 2024-03-24T01:26:12.317 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::ec4:7aff:fe8f:d767/64 scope link 2024-03-24T01:26:12.317 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:26:12.317 INFO:teuthology.orchestra.run.smithi090.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T01:26:12.317 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 06:50:1d:bd:ed:70 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:26:12.317 INFO:teuthology.orchestra.run.smithi090.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T01:26:12.317 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:26:12.317 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::a05b:d3ff:fea5:6fd0/64 scope link 2024-03-24T01:26:12.317 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:26:12.317 INFO:teuthology.orchestra.run.smithi090.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T01:26:12.317 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether de:94:0b:7e:c6:41 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:26:12.317 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::dc94:bff:fe7e:c641/64 scope link 2024-03-24T01:26:12.317 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:26:12.318 INFO:teuthology.orchestra.run.smithi090.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T01:26:12.318 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 06:50:1d:bd:ed:70 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:26:12.318 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::450:1dff:febd:ed70/64 scope link 2024-03-24T01:26:12.318 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:26:12.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:12.319 DEBUG:teuthology.orchestra.run.smithi090:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T01:26:12.398 INFO:teuthology.orchestra.run.smithi090.stdout:1 2024-03-24T01:26:12.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:12.402 DEBUG:teuthology.orchestra.run.smithi090:> route 2024-03-24T01:26:12.449 INFO:teuthology.orchestra.run.smithi090.stdout:Kernel IP routing table 2024-03-24T01:26:12.450 INFO:teuthology.orchestra.run.smithi090.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T01:26:12.450 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T01:26:12.450 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T01:26:12.450 INFO:teuthology.orchestra.run.smithi090.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T01:26:12.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:12.451 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2024-03-24T01:26:12.451 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-24T01:26:12.451 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T01:26:12.451 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T01:26:12.451 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T01:26:12.451 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-24T01:26:12.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:12.622 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-03-24T01:26:12.670 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T01:26:12.670 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T01:26:12.671 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T01:26:12.671 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:12.671 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set brx.0 up 2024-03-24T01:26:12.812 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-24T01:26:12.812 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T01:26:12.812 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T01:26:12.864 INFO:teuthology.orchestra.run.smithi090.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T01:26:12.864 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T01:26:12.864 DEBUG:teuthology.orchestra.run.smithi090:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T01:26:12.911 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-24T01:26:13.056 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T01:26:13.106 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-24T01:26:13.106 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-24T01:26:13.106 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-24T01:26:13.106 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-24T01:26:13.106 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-24T01:26:13.106 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-24T01:26:13.106 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-24T01:26:13.106 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-24T01:26:13.107 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-24T01:26:13.107 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-24T01:26:13.107 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-24T01:26:13.107 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/selinux 2024-03-24T01:26:13.107 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T01:26:13.107 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-24T01:26:13.107 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-24T01:26:13.107 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-24T01:26:13.107 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-24T01:26:13.107 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T01:26:13.107 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-24T01:26:13.107 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T01:26:13.107 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-24T01:26:13.107 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T01:26:13.107 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T01:26:13.107 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-24T01:26:13.108 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-24T01:26:13.108 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T01:26:13.108 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T01:26:13.108 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T01:26:13.108 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T01:26:13.108 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-24T01:26:13.108 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:26:13.108 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:26:13.108 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:26:13.108 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:26:13.109 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:13.109 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-24T01:26:13.155 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-24T01:26:13.155 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-24T01:26:13.202 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-24T01:26:13.285 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T01:26:13.327 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-24T01:26:13.327 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-24T01:26:13.327 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-24T01:26:13.327 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-24T01:26:13.327 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-24T01:26:13.327 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-24T01:26:13.327 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-24T01:26:13.327 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-24T01:26:13.327 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-24T01:26:13.327 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-24T01:26:13.327 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-24T01:26:13.327 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/selinux 2024-03-24T01:26:13.327 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T01:26:13.327 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-24T01:26:13.328 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-24T01:26:13.328 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-24T01:26:13.328 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-24T01:26:13.328 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T01:26:13.328 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-24T01:26:13.328 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T01:26:13.328 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-24T01:26:13.328 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T01:26:13.328 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T01:26:13.328 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-24T01:26:13.328 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-24T01:26:13.328 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T01:26:13.328 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T01:26:13.328 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T01:26:13.328 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T01:26:13.328 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-24T01:26:13.328 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:26:13.329 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:26:13.329 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:26:13.329 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:26:13.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:13.330 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-24T01:26:13.389 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:2024-03-24T01:26:13.389+0000 7fd344210480 -1 init, newargv = 0x55d95e818850 newargc=15 2024-03-24T01:26:13.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[53657]: starting ceph client 2024-03-24T01:26:13.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[53657]: starting fuse 2024-03-24T01:26:13.415 INFO:teuthology.orchestra.run.smithi090.stdout:40 2024-03-24T01:26:13.415 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-03-24T01:26:13.416 DEBUG:teuthology.orchestra.run.smithi090:> sudo stdin-killer -- python3 -c ' 2024-03-24T01:26:13.416 DEBUG:teuthology.orchestra.run.smithi090:> import glob 2024-03-24T01:26:13.416 DEBUG:teuthology.orchestra.run.smithi090:> import re 2024-03-24T01:26:13.416 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-24T01:26:13.416 DEBUG:teuthology.orchestra.run.smithi090:> import subprocess 2024-03-24T01:26:13.416 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:26:13.416 DEBUG:teuthology.orchestra.run.smithi090:> def _find_admin_socket(client_name): 2024-03-24T01:26:13.416 DEBUG:teuthology.orchestra.run.smithi090:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-24T01:26:13.416 DEBUG:teuthology.orchestra.run.smithi090:> files = glob.glob(asok_path) 2024-03-24T01:26:13.416 DEBUG:teuthology.orchestra.run.smithi090:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-24T01:26:13.416 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:26:13.417 DEBUG:teuthology.orchestra.run.smithi090:> # Given a non-glob path, it better be there 2024-03-24T01:26:13.417 DEBUG:teuthology.orchestra.run.smithi090:> if "*" not in asok_path: 2024-03-24T01:26:13.417 DEBUG:teuthology.orchestra.run.smithi090:> assert(len(files) == 1) 2024-03-24T01:26:13.417 DEBUG:teuthology.orchestra.run.smithi090:> return files[0] 2024-03-24T01:26:13.417 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:26:13.417 DEBUG:teuthology.orchestra.run.smithi090:> for f in files: 2024-03-24T01:26:13.417 DEBUG:teuthology.orchestra.run.smithi090:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T01:26:13.417 DEBUG:teuthology.orchestra.run.smithi090:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T01:26:13.417 DEBUG:teuthology.orchestra.run.smithi090:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T01:26:13.417 DEBUG:teuthology.orchestra.run.smithi090:> contents = proc_f.read() 2024-03-24T01:26:13.417 DEBUG:teuthology.orchestra.run.smithi090:> if mountpoint in contents: 2024-03-24T01:26:13.417 DEBUG:teuthology.orchestra.run.smithi090:> return f 2024-03-24T01:26:13.417 DEBUG:teuthology.orchestra.run.smithi090:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T01:26:13.417 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:26:13.417 DEBUG:teuthology.orchestra.run.smithi090:> print(_find_admin_socket("client.0")) 2024-03-24T01:26:13.418 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-24T01:26:13.610 INFO:teuthology.orchestra.run.smithi090.stdout:/var/run/ceph/ceph-client.0.53657.asok 2024-03-24T01:26:13.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T01:26:13.630 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.53657.asok 2024-03-24T01:26:13.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:13.631 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.53657.asok status 2024-03-24T01:26:13.860 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-24T01:26:13.860 INFO:teuthology.orchestra.run.smithi090.stdout: "metadata": { 2024-03-24T01:26:13.860 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_sha1": "3be9a5c9bc793e11e2800a8c0c696e8b46742033", 2024-03-24T01:26:13.861 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-24T01:26:13.861 INFO:teuthology.orchestra.run.smithi090.stdout: "entity_id": "0", 2024-03-24T01:26:13.861 INFO:teuthology.orchestra.run.smithi090.stdout: "hostname": "smithi090", 2024-03-24T01:26:13.861 INFO:teuthology.orchestra.run.smithi090.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-24T01:26:13.861 INFO:teuthology.orchestra.run.smithi090.stdout: "pid": "53657", 2024-03-24T01:26:13.861 INFO:teuthology.orchestra.run.smithi090.stdout: "root": "/" 2024-03-24T01:26:13.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:26:13.861 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_count": 0, 2024-03-24T01:26:13.861 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_pinned_count": 0, 2024-03-24T01:26:13.861 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 5706, 2024-03-24T01:26:13.861 INFO:teuthology.orchestra.run.smithi090.stdout: "inst": { 2024-03-24T01:26:13.861 INFO:teuthology.orchestra.run.smithi090.stdout: "name": { 2024-03-24T01:26:13.861 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "client", 2024-03-24T01:26:13.861 INFO:teuthology.orchestra.run.smithi090.stdout: "num": 5706 2024-03-24T01:26:13.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:26:13.862 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-24T01:26:13.862 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-24T01:26:13.862 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-03-24T01:26:13.862 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2963718040 2024-03-24T01:26:13.862 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:26:13.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:26:13.862 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-24T01:26:13.862 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-24T01:26:13.862 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-03-24T01:26:13.862 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2963718040 2024-03-24T01:26:13.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:26:13.862 INFO:teuthology.orchestra.run.smithi090.stdout: "inst_str": "client.5706 192.168.0.1:0/2963718040", 2024-03-24T01:26:13.862 INFO:teuthology.orchestra.run.smithi090.stdout: "addr_str": "192.168.0.1:0/2963718040", 2024-03-24T01:26:13.863 INFO:teuthology.orchestra.run.smithi090.stdout: "inode_count": 1, 2024-03-24T01:26:13.863 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_epoch": 17, 2024-03-24T01:26:13.863 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch": 39, 2024-03-24T01:26:13.863 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch_barrier": 0, 2024-03-24T01:26:13.863 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklisted": false, 2024-03-24T01:26:13.863 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs" 2024-03-24T01:26:13.863 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-24T01:26:13.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:13.878 DEBUG:teuthology.orchestra.run.smithi090:> stat --file-system '--printf=%T 2024-03-24T01:26:13.878 DEBUG:teuthology.orchestra.run.smithi090:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-24T01:26:13.930 INFO:teuthology.orchestra.run.smithi090.stdout:fuseblk 2024-03-24T01:26:13.931 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-24T01:26:13.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:13.931 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T01:26:14.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:14.016 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch rootfile) 2024-03-24T01:26:14.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:14.082 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir) 2024-03-24T01:26:14.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:14.132 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch subdir/subdirfile) 2024-03-24T01:26:14.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:14.186 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir linkdir) 2024-03-24T01:26:14.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:14.242 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch linkdir/link0) 2024-03-24T01:26:14.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:14.294 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f linkdir/link0) 2024-03-24T01:26:14.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:14.349 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln subdir/subdirfile linkdir/link0) 2024-03-24T01:26:14.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:14.400 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln subdir/subdirfile linkdir/link1) 2024-03-24T01:26:14.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:14.452 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln subdir/subdirfile linkdir/link2) 2024-03-24T01:26:14.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:14.502 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f linkdir/link2) 2024-03-24T01:26:14.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:14.557 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch linkdir/link2) 2024-03-24T01:26:14.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:14.611 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln subdir/subdirfile linkdir/link3) 2024-03-24T01:26:14.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:14.674 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f linkdir/link3) 2024-03-24T01:26:14.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:14.734 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln rootfile linkdir/link3) 2024-03-24T01:26:14.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:14.799 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir/subsubdir) 2024-03-24T01:26:14.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:14.847 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec sync) 2024-03-24T01:26:15.107 DEBUG:teuthology.orchestra.run.smithi090:> stdin-killer -- python3 -c ' 2024-03-24T01:26:15.108 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-24T01:26:15.108 DEBUG:teuthology.orchestra.run.smithi090:> import stat 2024-03-24T01:26:15.108 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:26:15.108 DEBUG:teuthology.orchestra.run.smithi090:> print(os.stat("/home/ubuntu/cephtest/mnt.0/rootfile").st_ino) 2024-03-24T01:26:15.108 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-24T01:26:15.239 INFO:teuthology.orchestra.run.smithi090.stdout:1099511627776 2024-03-24T01:26:15.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T01:26:15.255 DEBUG:teuthology.orchestra.run.smithi090:> stdin-killer -- python3 -c ' 2024-03-24T01:26:15.255 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-24T01:26:15.256 DEBUG:teuthology.orchestra.run.smithi090:> import stat 2024-03-24T01:26:15.256 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:26:15.256 DEBUG:teuthology.orchestra.run.smithi090:> print(os.stat("/home/ubuntu/cephtest/mnt.0/subdir").st_ino) 2024-03-24T01:26:15.256 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-24T01:26:15.428 INFO:teuthology.orchestra.run.smithi090.stdout:1099511627777 2024-03-24T01:26:15.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T01:26:15.441 DEBUG:teuthology.orchestra.run.smithi090:> stdin-killer -- python3 -c ' 2024-03-24T01:26:15.441 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-24T01:26:15.441 DEBUG:teuthology.orchestra.run.smithi090:> import stat 2024-03-24T01:26:15.441 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:26:15.441 DEBUG:teuthology.orchestra.run.smithi090:> print(os.stat("/home/ubuntu/cephtest/mnt.0/linkdir").st_ino) 2024-03-24T01:26:15.441 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-24T01:26:15.615 INFO:teuthology.orchestra.run.smithi090.stdout:1099511628279 2024-03-24T01:26:15.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T01:26:15.628 DEBUG:teuthology.orchestra.run.smithi090:> stdin-killer -- python3 -c ' 2024-03-24T01:26:15.628 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-24T01:26:15.629 DEBUG:teuthology.orchestra.run.smithi090:> import stat 2024-03-24T01:26:15.629 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:26:15.629 DEBUG:teuthology.orchestra.run.smithi090:> print(os.stat("/home/ubuntu/cephtest/mnt.0/subdir/subdirfile").st_ino) 2024-03-24T01:26:15.629 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-24T01:26:15.813 INFO:teuthology.orchestra.run.smithi090.stdout:1099511627778 2024-03-24T01:26:15.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T01:26:15.827 DEBUG:teuthology.orchestra.run.smithi090:> stdin-killer -- python3 -c ' 2024-03-24T01:26:15.827 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-24T01:26:15.827 DEBUG:teuthology.orchestra.run.smithi090:> import stat 2024-03-24T01:26:15.827 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:26:15.827 DEBUG:teuthology.orchestra.run.smithi090:> print(os.stat("/home/ubuntu/cephtest/mnt.0/subdir/subsubdir").st_ino) 2024-03-24T01:26:15.827 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-24T01:26:15.998 INFO:teuthology.orchestra.run.smithi090.stdout:1099511628282 2024-03-24T01:26:16.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T01:26:16.013 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:26:16.013 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:26:16.105 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:26:16.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.341+0000 7f2fa8a00640 1 Processor -- start 2024-03-24T01:26:16.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.341+0000 7f2fa8a00640 1 -- start start 2024-03-24T01:26:16.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.343+0000 7f2fa8a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2fa406b7d0 0x7f2fa4166a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:16.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.343+0000 7f2fa8a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fa40cd490 con 0x7f2fa40cc150 2024-03-24T01:26:16.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.343+0000 7f2fa8a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fa415f900 con 0x7f2fa41603e0 2024-03-24T01:26:16.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.343+0000 7f2fa8a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa40cd270 con 0x7f2fa406b7d0 2024-03-24T01:26:16.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.343+0000 7f2fa1e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f2fa40cc150 0x7f2fa40ccb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51636/0 (socket says 172.21.15.90:51636) 2024-03-24T01:26:16.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.343+0000 7f2fa1e00640 1 -- 172.21.15.90:0/4227304038 learned_addr learned my addr 172.21.15.90:0/4227304038 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:26:16.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.343+0000 7f2fa2800640 1 --2- 172.21.15.90:0/4227304038 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2fa406b7d0 0x7f2fa4166a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:16.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.343+0000 7f2fa0a00640 1 -- 172.21.15.90:0/4227304038 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 81860664 0 0) 0x7f2fa40cd490 con 0x7f2fa40cc150 2024-03-24T01:26:16.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.344+0000 7f2fa0a00640 1 -- 172.21.15.90:0/4227304038 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f88003520 con 0x7f2fa40cc150 2024-03-24T01:26:16.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.344+0000 7f2fa0a00640 1 -- 172.21.15.90:0/4227304038 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4291919457 0 0) 0x7f2f88003520 con 0x7f2fa40cc150 2024-03-24T01:26:16.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.344+0000 7f2fa0a00640 1 -- 172.21.15.90:0/4227304038 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2fa406b7d0 msgr2=0x7f2fa4166a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:16.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.344+0000 7f2fa0a00640 1 --2- 172.21.15.90:0/4227304038 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2fa406b7d0 0x7f2fa4166a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:16.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.344+0000 7f2fa0a00640 1 -- 172.21.15.90:0/4227304038 >> v1:172.21.15.120:6789/0 conn(0x7f2fa41603e0 legacy=0x7f2fa406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:16.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.344+0000 7f2fa0a00640 1 -- 172.21.15.90:0/4227304038 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa415fbe0 con 0x7f2fa40cc150 2024-03-24T01:26:16.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.344+0000 7f2fa0a00640 1 -- 172.21.15.90:0/4227304038 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f2f94001670 con 0x7f2fa40cc150 2024-03-24T01:26:16.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.344+0000 7f2fa0a00640 1 -- 172.21.15.90:0/4227304038 >> v1:172.21.15.90:6789/0 conn(0x7f2fa40cc150 legacy=0x7f2fa40ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:16.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.344+0000 7f2fa0a00640 1 --2- 172.21.15.90:0/4227304038 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2fa406b7d0 0x7f2f880039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:16.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.344+0000 7f2fa0a00640 1 --2- 172.21.15.90:0/4227304038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f88003fb0 0x7f2f88004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:16.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.344+0000 7f2fa0a00640 1 --2- 172.21.15.90:0/4227304038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f88004d40 0x7f2f880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:16.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.344+0000 7f2fa0a00640 1 -- 172.21.15.90:0/4227304038 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa40cd270 con 0x7f2f88004d40 2024-03-24T01:26:16.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.344+0000 7f2fa0a00640 1 -- 172.21.15.90:0/4227304038 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8800bb00 con 0x7f2f88003fb0 2024-03-24T01:26:16.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.344+0000 7f2fa0a00640 1 -- 172.21.15.90:0/4227304038 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8800bd00 con 0x7f2fa406b7d0 2024-03-24T01:26:16.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.344+0000 7f2fa2800640 1 --2- 172.21.15.90:0/4227304038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f88004d40 0x7f2f880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:16.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.345+0000 7f2fa1e00640 1 --2- 172.21.15.90:0/4227304038 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2fa406b7d0 0x7f2f880039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:16.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.345+0000 7f2fa1400640 1 --2- 172.21.15.90:0/4227304038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f88003fb0 0x7f2f88004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:16.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.345+0000 7f2fa2800640 1 -- 172.21.15.90:0/4227304038 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2fa406b7d0 msgr2=0x7f2f880039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:16.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.345+0000 7f2fa2800640 1 --2- 172.21.15.90:0/4227304038 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2fa406b7d0 0x7f2f880039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:16.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.345+0000 7f2fa2800640 1 -- 172.21.15.90:0/4227304038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f88003fb0 msgr2=0x7f2f88004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:16.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.345+0000 7f2fa2800640 1 --2- 172.21.15.90:0/4227304038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f88003fb0 0x7f2f88004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:16.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.345+0000 7f2fa2800640 1 -- 172.21.15.90:0/4227304038 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f88003520 con 0x7f2f88004d40 2024-03-24T01:26:16.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.345+0000 7f2fa1400640 1 --2- 172.21.15.90:0/4227304038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f88003fb0 0x7f2f88004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:16.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.346+0000 7f2fa2800640 1 --2- 172.21.15.90:0/4227304038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f88004d40 0x7f2f880051c0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c009f90 tx=0x7f2f88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d718bae5d4cd259c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:16.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.346+0000 7f2fa0a00640 1 -- 172.21.15.90:0/4227304038 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c039070 con 0x7f2f88004d40 2024-03-24T01:26:16.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.346+0000 7f2fa0a00640 1 -- 172.21.15.90:0/4227304038 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f9c02cbd0 con 0x7f2f88004d40 2024-03-24T01:26:16.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.346+0000 7f2fa0a00640 1 -- 172.21.15.90:0/4227304038 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c009c10 con 0x7f2f88004d40 2024-03-24T01:26:16.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.347+0000 7f2fa8a00640 1 -- 172.21.15.90:0/4227304038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f88004d40 msgr2=0x7f2f880051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:16.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.347+0000 7f2fa8a00640 1 --2- 172.21.15.90:0/4227304038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f88004d40 0x7f2f880051c0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c009f90 tx=0x7f2f88005830 comp rx=0 tx=0).stop 2024-03-24T01:26:16.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.347+0000 7f2fa2800640 1 -- 172.21.15.90:0/4227304038 reap_dead start 2024-03-24T01:26:16.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.347+0000 7f2fa8a00640 1 -- 172.21.15.90:0/4227304038 shutdown_connections 2024-03-24T01:26:16.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.347+0000 7f2fa8a00640 1 -- 172.21.15.90:0/4227304038 >> 172.21.15.90:0/4227304038 conn(0x7f2fa40b1770 msgr2=0x7f2fa40b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:16.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.347+0000 7f2fa8a00640 1 -- 172.21.15.90:0/4227304038 shutdown_connections 2024-03-24T01:26:16.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.347+0000 7f2fa8a00640 1 -- 172.21.15.90:0/4227304038 wait complete. 2024-03-24T01:26:16.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.349+0000 7f2fa8a00640 1 Processor -- start 2024-03-24T01:26:16.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.349+0000 7f2fa8a00640 1 -- start start 2024-03-24T01:26:16.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.350+0000 7f2fa8a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f88004d40 0x7f2fa415a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:16.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.350+0000 7f2fa8a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2fa415b320 0x7f2fa413f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:16.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.350+0000 7f2fa8a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fa4139780 0x7f2fa4142f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:16.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.350+0000 7f2fa8a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa41455b0 con 0x7f2fa4139780 2024-03-24T01:26:16.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.350+0000 7f2fa8a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa4145430 con 0x7f2fa415b320 2024-03-24T01:26:16.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.350+0000 7f2fa8a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa406d990 con 0x7f2f88004d40 2024-03-24T01:26:16.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.350+0000 7f2fa2800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fa4139780 0x7f2fa4142f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:16.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.350+0000 7f2fa2800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fa4139780 0x7f2fa4142f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40172/0 (socket says 172.21.15.90:40172) 2024-03-24T01:26:16.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.350+0000 7f2fa2800640 1 -- 172.21.15.90:0/773346936 learned_addr learned my addr 172.21.15.90:0/773346936 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:16.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.350+0000 7f2fa1e00640 1 --2- 172.21.15.90:0/773346936 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f88004d40 0x7f2fa415a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:16.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.350+0000 7f2fa2800640 1 -- 172.21.15.90:0/773346936 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f88004d40 msgr2=0x7f2fa415a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:16.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.350+0000 7f2fa2800640 1 --2- 172.21.15.90:0/773346936 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f88004d40 0x7f2fa415a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:16.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.350+0000 7f2fa1400640 1 --2- 172.21.15.90:0/773346936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2fa415b320 0x7f2fa413f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:16.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.350+0000 7f2fa2800640 1 -- 172.21.15.90:0/773346936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2fa415b320 msgr2=0x7f2fa413f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:16.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.350+0000 7f2fa2800640 1 --2- 172.21.15.90:0/773346936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2fa415b320 0x7f2fa413f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:16.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.350+0000 7f2fa2800640 1 -- 172.21.15.90:0/773346936 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa413f980 con 0x7f2fa4139780 2024-03-24T01:26:16.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.351+0000 7f2fa2800640 1 --2- 172.21.15.90:0/773346936 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fa4139780 0x7f2fa4142f50 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c037040 tx=0x7f2f9c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:16.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.351+0000 7f2f9a000640 1 -- 172.21.15.90:0/773346936 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c039030 con 0x7f2fa4139780 2024-03-24T01:26:16.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.351+0000 7f2f9a000640 1 -- 172.21.15.90:0/773346936 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f9c03a040 con 0x7f2fa4139780 2024-03-24T01:26:16.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.351+0000 7f2f9a000640 1 -- 172.21.15.90:0/773346936 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c009c10 con 0x7f2fa4139780 2024-03-24T01:26:16.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.351+0000 7f2fa8a00640 1 -- 172.21.15.90:0/773346936 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fa413e980 con 0x7f2fa4139780 2024-03-24T01:26:16.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.351+0000 7f2fa8a00640 1 -- 172.21.15.90:0/773346936 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fa413a440 con 0x7f2fa4139780 2024-03-24T01:26:16.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.352+0000 7f2fa8a00640 1 -- 172.21.15.90:0/773346936 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fa4068060 con 0x7f2fa4139780 2024-03-24T01:26:16.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.353+0000 7f2f9a000640 1 -- 172.21.15.90:0/773346936 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2f9c045030 con 0x7f2fa4139780 2024-03-24T01:26:16.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.354+0000 7f2f9a000640 1 --2- 172.21.15.90:0/773346936 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f2f8406bc00 0x7f2f8406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:16.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.354+0000 7f2f9a000640 1 -- 172.21.15.90:0/773346936 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2f9c0ac880 con 0x7f2fa4139780 2024-03-24T01:26:16.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.355+0000 7f2fa1e00640 1 --2- 172.21.15.90:0/773346936 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f2f8406bc00 0x7f2f8406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:16.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.356+0000 7f2fa1e00640 1 --2- 172.21.15.90:0/773346936 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f2f8406bc00 0x7f2f8406e0c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2f94001c00 tx=0x7f2f94000f70 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:16.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.359+0000 7f2f9a000640 1 -- 172.21.15.90:0/773346936 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2f9c04c060 con 0x7f2fa4139780 2024-03-24T01:26:16.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.592+0000 7f2fa8a00640 1 -- 172.21.15.90:0/773346936 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2fa40ccce0 con 0x7f2fa4139780 2024-03-24T01:26:16.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.594+0000 7f2f9a000640 1 -- 172.21.15.90:0/773346936 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+16918 (secure 0 0 0) 0x7f2f9c07b250 con 0x7f2fa4139780 2024-03-24T01:26:16.594 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:26:16.594 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":39,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:26:00.983350+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:25:57.398864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-24T01:25:58.451150+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:26:16.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.597+0000 7f2fa8a00640 1 -- 172.21.15.90:0/773346936 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f2f8406bc00 msgr2=0x7f2f8406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:16.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.597+0000 7f2fa8a00640 1 --2- 172.21.15.90:0/773346936 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f2f8406bc00 0x7f2f8406e0c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2f94001c00 tx=0x7f2f94000f70 comp rx=0 tx=0).stop 2024-03-24T01:26:16.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.597+0000 7f2fa8a00640 1 -- 172.21.15.90:0/773346936 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fa4139780 msgr2=0x7f2fa4142f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:16.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.597+0000 7f2fa8a00640 1 --2- 172.21.15.90:0/773346936 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fa4139780 0x7f2fa4142f50 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c037040 tx=0x7f2f9c009750 comp rx=0 tx=0).stop 2024-03-24T01:26:16.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.598+0000 7f2fa8a00640 1 -- 172.21.15.90:0/773346936 shutdown_connections 2024-03-24T01:26:16.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.598+0000 7f2fa8a00640 1 --2- 172.21.15.90:0/773346936 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f2f8406bc00 0x7f2f8406e0c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:16.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.598+0000 7f2fa8a00640 1 --2- 172.21.15.90:0/773346936 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fa4139780 0x7f2fa4142f50 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:16.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.598+0000 7f2fa8a00640 1 --2- 172.21.15.90:0/773346936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2fa415b320 0x7f2fa413f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:16.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.598+0000 7f2fa8a00640 1 --2- 172.21.15.90:0/773346936 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2f88004d40 0x7f2fa415a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:16.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.598+0000 7f2fa8a00640 1 -- 172.21.15.90:0/773346936 >> 172.21.15.90:0/773346936 conn(0x7f2fa40b1770 msgr2=0x7f2fa40739c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:16.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.598+0000 7f2fa8a00640 1 -- 172.21.15.90:0/773346936 shutdown_connections 2024-03-24T01:26:16.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.598+0000 7f2fa8a00640 1 -- 172.21.15.90:0/773346936 wait complete. 2024-03-24T01:26:16.622 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T01:25:57.398864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T01:25:58.451150+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T01:26:16.622 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:26:16.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.829+0000 7ffa22600640 1 Processor -- start 2024-03-24T01:26:16.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.830+0000 7ffa22600640 1 -- start start 2024-03-24T01:26:16.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.831+0000 7ffa22600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ffa1c1452b0 0x7ffa1c071990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:16.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.831+0000 7ffa22600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa1c0cf200 0x7ffa1c0b3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:16.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.831+0000 7ffa22600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa1c0b2890 con 0x7ffa1c071ed0 2024-03-24T01:26:16.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.831+0000 7ffa22600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa1c072330 con 0x7ffa1c1452b0 2024-03-24T01:26:16.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.831+0000 7ffa22600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1c072990 con 0x7ffa1c0cf200 2024-03-24T01:26:16.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.831+0000 7ffa13e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ffa1c071ed0 0x7ffa1c0ceaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51644/0 (socket says 172.21.15.90:51644) 2024-03-24T01:26:16.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.832+0000 7ffa13e00640 1 -- 172.21.15.90:0/1296377329 learned_addr learned my addr 172.21.15.90:0/1296377329 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:26:16.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.832+0000 7ffa1b400640 1 --2- 172.21.15.90:0/1296377329 >> v2:172.21.15.120:3300/0 conn(0x7ffa1c1452b0 0x7ffa1c071990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:16.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.832+0000 7ffa1be00640 1 --2- 172.21.15.90:0/1296377329 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa1c0cf200 0x7ffa1c0b3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:16.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.832+0000 7ffa1aa00640 1 -- 172.21.15.90:0/1296377329 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2136368239 0 0) 0x7ffa1c0b2890 con 0x7ffa1c071ed0 2024-03-24T01:26:16.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.832+0000 7ffa1aa00640 1 -- 172.21.15.90:0/1296377329 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa00003520 con 0x7ffa1c071ed0 2024-03-24T01:26:16.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.833+0000 7ffa1aa00640 1 -- 172.21.15.90:0/1296377329 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1974564832 0 0) 0x7ffa00003520 con 0x7ffa1c071ed0 2024-03-24T01:26:16.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.833+0000 7ffa1aa00640 1 -- 172.21.15.90:0/1296377329 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa1c0cf200 msgr2=0x7ffa1c0b3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:16.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.833+0000 7ffa1aa00640 1 --2- 172.21.15.90:0/1296377329 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa1c0cf200 0x7ffa1c0b3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:16.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.833+0000 7ffa1aa00640 1 -- 172.21.15.90:0/1296377329 >> v2:172.21.15.120:3300/0 conn(0x7ffa1c1452b0 msgr2=0x7ffa1c071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:16.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.833+0000 7ffa1aa00640 1 --2- 172.21.15.90:0/1296377329 >> v2:172.21.15.120:3300/0 conn(0x7ffa1c1452b0 0x7ffa1c071990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:16.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.833+0000 7ffa1aa00640 1 -- 172.21.15.90:0/1296377329 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa1c0b2b70 con 0x7ffa1c071ed0 2024-03-24T01:26:16.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.833+0000 7ffa1aa00640 1 -- 172.21.15.90:0/1296377329 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7ffa04001670 con 0x7ffa1c071ed0 2024-03-24T01:26:16.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.833+0000 7ffa1aa00640 1 -- 172.21.15.90:0/1296377329 >> v1:172.21.15.90:6789/0 conn(0x7ffa1c071ed0 legacy=0x7ffa1c0ceaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:16.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.833+0000 7ffa1aa00640 1 --2- 172.21.15.90:0/1296377329 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa000039b0 0x7ffa00003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:16.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.833+0000 7ffa1aa00640 1 --2- 172.21.15.90:0/1296377329 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa1c0cf200 0x7ffa00004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:16.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.833+0000 7ffa1aa00640 1 --2- 172.21.15.90:0/1296377329 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa00004d40 0x7ffa000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:16.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.833+0000 7ffa1aa00640 1 -- 172.21.15.90:0/1296377329 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1c072990 con 0x7ffa00004d40 2024-03-24T01:26:16.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.833+0000 7ffa1aa00640 1 -- 172.21.15.90:0/1296377329 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0000b9c0 con 0x7ffa000039b0 2024-03-24T01:26:16.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.833+0000 7ffa1aa00640 1 -- 172.21.15.90:0/1296377329 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0000bb40 con 0x7ffa1c0cf200 2024-03-24T01:26:16.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.833+0000 7ffa1b400640 1 --2- 172.21.15.90:0/1296377329 >> v2:172.21.15.120:3300/0 conn(0x7ffa1c1452b0 0x7ffa1c071990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T01:26:16.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.833+0000 7ffa1be00640 1 --2- 172.21.15.90:0/1296377329 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa00004d40 0x7ffa000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:16.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.833+0000 7ffa13e00640 1 --2- 172.21.15.90:0/1296377329 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa1c0cf200 0x7ffa00004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:16.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.834+0000 7ffa1be00640 1 -- 172.21.15.90:0/1296377329 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa1c0cf200 msgr2=0x7ffa00004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:16.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.834+0000 7ffa1be00640 1 --2- 172.21.15.90:0/1296377329 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa1c0cf200 0x7ffa00004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:16.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.834+0000 7ffa1be00640 1 -- 172.21.15.90:0/1296377329 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa000039b0 msgr2=0x7ffa00003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:16.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.834+0000 7ffa1be00640 1 --2- 172.21.15.90:0/1296377329 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa000039b0 0x7ffa00003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:16.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.834+0000 7ffa1be00640 1 -- 172.21.15.90:0/1296377329 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa00003520 con 0x7ffa00004d40 2024-03-24T01:26:16.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.835+0000 7ffa1be00640 1 --2- 172.21.15.90:0/1296377329 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa00004d40 0x7ffa000051c0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ffa1400e170 tx=0x7ffa00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcd428d0663b391 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:16.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.835+0000 7ffa1aa00640 1 -- 172.21.15.90:0/1296377329 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa14039070 con 0x7ffa00004d40 2024-03-24T01:26:16.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.835+0000 7ffa1aa00640 1 -- 172.21.15.90:0/1296377329 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1402cbf0 con 0x7ffa00004d40 2024-03-24T01:26:16.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.835+0000 7ffa1aa00640 1 -- 172.21.15.90:0/1296377329 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa14009c10 con 0x7ffa00004d40 2024-03-24T01:26:16.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.835+0000 7ffa22600640 1 -- 172.21.15.90:0/1296377329 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa00004d40 msgr2=0x7ffa000051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:16.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.835+0000 7ffa22600640 1 --2- 172.21.15.90:0/1296377329 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa00004d40 0x7ffa000051c0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ffa1400e170 tx=0x7ffa00005830 comp rx=0 tx=0).stop 2024-03-24T01:26:16.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.835+0000 7ffa1be00640 1 -- 172.21.15.90:0/1296377329 reap_dead start 2024-03-24T01:26:16.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.836+0000 7ffa22600640 1 -- 172.21.15.90:0/1296377329 shutdown_connections 2024-03-24T01:26:16.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.836+0000 7ffa22600640 1 -- 172.21.15.90:0/1296377329 >> 172.21.15.90:0/1296377329 conn(0x7ffa1c0cf7e0 msgr2=0x7ffa1c0cfbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:16.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.836+0000 7ffa22600640 1 -- 172.21.15.90:0/1296377329 shutdown_connections 2024-03-24T01:26:16.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.836+0000 7ffa22600640 1 -- 172.21.15.90:0/1296377329 wait complete. 2024-03-24T01:26:16.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.837+0000 7ffa22600640 1 Processor -- start 2024-03-24T01:26:16.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.838+0000 7ffa22600640 1 -- start start 2024-03-24T01:26:16.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.838+0000 7ffa22600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa00004d40 0x7ffa1c0add50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:16.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.838+0000 7ffa22600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa1c1f4010 0x7ffa1c0cca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:16.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.838+0000 7ffa22600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa1c0ccf80 0x7ffa1c1f6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:16.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.838+0000 7ffa22600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1c0b3410 con 0x7ffa1c0ccf80 2024-03-24T01:26:16.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.838+0000 7ffa22600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1c0b3130 con 0x7ffa1c1f4010 2024-03-24T01:26:16.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.838+0000 7ffa22600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1c145690 con 0x7ffa00004d40 2024-03-24T01:26:16.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.839+0000 7ffa1be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa1c0ccf80 0x7ffa1c1f6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:16.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.839+0000 7ffa1be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa1c0ccf80 0x7ffa1c1f6860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40188/0 (socket says 172.21.15.90:40188) 2024-03-24T01:26:16.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.839+0000 7ffa1be00640 1 -- 172.21.15.90:0/3312520176 learned_addr learned my addr 172.21.15.90:0/3312520176 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:16.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.839+0000 7ffa1b400640 1 --2- 172.21.15.90:0/3312520176 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa00004d40 0x7ffa1c0add50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:16.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.839+0000 7ffa13e00640 1 --2- 172.21.15.90:0/3312520176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa1c1f4010 0x7ffa1c0cca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:16.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.839+0000 7ffa1be00640 1 -- 172.21.15.90:0/3312520176 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa00004d40 msgr2=0x7ffa1c0add50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:16.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.839+0000 7ffa1be00640 1 --2- 172.21.15.90:0/3312520176 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa00004d40 0x7ffa1c0add50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:16.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.839+0000 7ffa1be00640 1 -- 172.21.15.90:0/3312520176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa1c1f4010 msgr2=0x7ffa1c0cca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:16.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.839+0000 7ffa1be00640 1 --2- 172.21.15.90:0/3312520176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa1c1f4010 0x7ffa1c0cca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:16.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.839+0000 7ffa1be00640 1 -- 172.21.15.90:0/3312520176 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa1c142dd0 con 0x7ffa1c0ccf80 2024-03-24T01:26:16.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.840+0000 7ffa1be00640 1 --2- 172.21.15.90:0/3312520176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa1c0ccf80 0x7ffa1c1f6860 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7ffa1400e170 tx=0x7ffa14009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:16.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.840+0000 7ffa13400640 1 -- 172.21.15.90:0/3312520176 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa14039030 con 0x7ffa1c0ccf80 2024-03-24T01:26:16.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.840+0000 7ffa13400640 1 -- 172.21.15.90:0/3312520176 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa14043070 con 0x7ffa1c0ccf80 2024-03-24T01:26:16.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.840+0000 7ffa13400640 1 -- 172.21.15.90:0/3312520176 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1403d7d0 con 0x7ffa1c0ccf80 2024-03-24T01:26:16.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.840+0000 7ffa22600640 1 -- 172.21.15.90:0/3312520176 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa1c06ddd0 con 0x7ffa1c0ccf80 2024-03-24T01:26:16.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.840+0000 7ffa22600640 1 -- 172.21.15.90:0/3312520176 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa1c06e390 con 0x7ffa1c0ccf80 2024-03-24T01:26:16.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.841+0000 7ffa22600640 1 -- 172.21.15.90:0/3312520176 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa1c04f400 con 0x7ffa1c0ccf80 2024-03-24T01:26:16.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.842+0000 7ffa13400640 1 -- 172.21.15.90:0/3312520176 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ffa14005d10 con 0x7ffa1c0ccf80 2024-03-24T01:26:16.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.843+0000 7ffa13400640 1 --2- 172.21.15.90:0/3312520176 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff9f806bc00 0x7ff9f806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:16.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.844+0000 7ffa13400640 1 -- 172.21.15.90:0/3312520176 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ffa140ace40 con 0x7ffa1c0ccf80 2024-03-24T01:26:16.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.844+0000 7ffa1b400640 1 --2- 172.21.15.90:0/3312520176 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff9f806bc00 0x7ff9f806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:16.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.845+0000 7ffa1b400640 1 --2- 172.21.15.90:0/3312520176 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff9f806bc00 0x7ff9f806e0c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c004790 tx=0x7ffa0c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:16.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:16.848+0000 7ffa13400640 1 -- 172.21.15.90:0/3312520176 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ffa1407b790 con 0x7ffa1c0ccf80 2024-03-24T01:26:17.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.050+0000 7ffa22600640 1 -- 172.21.15.90:0/3312520176 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffa1c0b47e0 con 0x7ffa1c0ccf80 2024-03-24T01:26:17.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.050+0000 7ffa13400640 1 -- 172.21.15.90:0/3312520176 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v39) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffa1407b090 con 0x7ffa1c0ccf80 2024-03-24T01:26:17.051 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:26:17.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.053+0000 7ffa22600640 1 -- 172.21.15.90:0/3312520176 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff9f806bc00 msgr2=0x7ff9f806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:17.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.053+0000 7ffa22600640 1 --2- 172.21.15.90:0/3312520176 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff9f806bc00 0x7ff9f806e0c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c004790 tx=0x7ffa0c008040 comp rx=0 tx=0).stop 2024-03-24T01:26:17.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.053+0000 7ffa22600640 1 -- 172.21.15.90:0/3312520176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa1c0ccf80 msgr2=0x7ffa1c1f6860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:17.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.053+0000 7ffa22600640 1 --2- 172.21.15.90:0/3312520176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa1c0ccf80 0x7ffa1c1f6860 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7ffa1400e170 tx=0x7ffa14009750 comp rx=0 tx=0).stop 2024-03-24T01:26:17.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.053+0000 7ffa22600640 1 -- 172.21.15.90:0/3312520176 shutdown_connections 2024-03-24T01:26:17.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.053+0000 7ffa22600640 1 --2- 172.21.15.90:0/3312520176 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff9f806bc00 0x7ff9f806e0c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:17.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.053+0000 7ffa22600640 1 --2- 172.21.15.90:0/3312520176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa1c0ccf80 0x7ffa1c1f6860 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:17.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.053+0000 7ffa22600640 1 --2- 172.21.15.90:0/3312520176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa1c1f4010 0x7ffa1c0cca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:17.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.053+0000 7ffa22600640 1 --2- 172.21.15.90:0/3312520176 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa00004d40 0x7ffa1c0add50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:17.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.053+0000 7ffa22600640 1 -- 172.21.15.90:0/3312520176 >> 172.21.15.90:0/3312520176 conn(0x7ffa1c0cf7e0 msgr2=0x7ffa1c0af820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:17.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.053+0000 7ffa22600640 1 -- 172.21.15.90:0/3312520176 shutdown_connections 2024-03-24T01:26:17.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.054+0000 7ffa22600640 1 -- 172.21.15.90:0/3312520176 wait complete. 2024-03-24T01:26:17.072 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T01:26:17.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.271+0000 7f9c4f000640 1 Processor -- start 2024-03-24T01:26:17.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.271+0000 7f9c4f000640 1 -- start start 2024-03-24T01:26:17.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.273+0000 7f9c4f000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9c48159620 0x7f9c4806b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:17.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.273+0000 7f9c4f000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9c480cf150 0x7f9c480ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:17.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.273+0000 7f9c4f000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c4806bca0 con 0x7f9c4806b600 2024-03-24T01:26:17.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.273+0000 7f9c4f000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c480b4470 con 0x7f9c480cf150 2024-03-24T01:26:17.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.273+0000 7f9c4f000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c480b36f0 con 0x7f9c48159620 2024-03-24T01:26:17.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.273+0000 7f9c47400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f9c4806b600 0x7f9c480cea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51646/0 (socket says 172.21.15.90:51646) 2024-03-24T01:26:17.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.273+0000 7f9c47400640 1 -- 172.21.15.90:0/3915074964 learned_addr learned my addr 172.21.15.90:0/3915074964 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:26:17.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.273+0000 7f9c4d000640 1 --2- 172.21.15.90:0/3915074964 >> v2:172.21.15.90:3300/0 conn(0x7f9c480cf150 0x7f9c480ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:17.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.274+0000 7f9c47e00640 1 --2- 172.21.15.90:0/3915074964 >> v2:172.21.15.120:3300/0 conn(0x7f9c48159620 0x7f9c4806b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:17.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.274+0000 7f9c46a00640 1 -- 172.21.15.90:0/3915074964 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4245768851 0 0) 0x7f9c4806bca0 con 0x7f9c4806b600 2024-03-24T01:26:17.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.274+0000 7f9c46a00640 1 -- 172.21.15.90:0/3915074964 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c2c003520 con 0x7f9c4806b600 2024-03-24T01:26:17.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.274+0000 7f9c4d000640 1 -- 172.21.15.90:0/3915074964 >> v2:172.21.15.120:3300/0 conn(0x7f9c48159620 msgr2=0x7f9c4806b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:17.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.274+0000 7f9c4d000640 1 --2- 172.21.15.90:0/3915074964 >> v2:172.21.15.120:3300/0 conn(0x7f9c48159620 0x7f9c4806b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:17.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.274+0000 7f9c4d000640 1 -- 172.21.15.90:0/3915074964 >> v1:172.21.15.90:6789/0 conn(0x7f9c4806b600 legacy=0x7f9c480cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:17.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.274+0000 7f9c4d000640 1 -- 172.21.15.90:0/3915074964 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c480b3990 con 0x7f9c480cf150 2024-03-24T01:26:17.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.274+0000 7f9c4d000640 1 --2- 172.21.15.90:0/3915074964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c480cf150 0x7f9c480ae530 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f9c4806c2d0 tx=0x7f9c4002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28370295c639ec43 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:17.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.275+0000 7f9c46a00640 1 -- 172.21.15.90:0/3915074964 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c40039070 con 0x7f9c480cf150 2024-03-24T01:26:17.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.275+0000 7f9c46a00640 1 -- 172.21.15.90:0/3915074964 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c4002cbf0 con 0x7f9c480cf150 2024-03-24T01:26:17.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.275+0000 7f9c46a00640 1 -- 172.21.15.90:0/3915074964 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c40034440 con 0x7f9c480cf150 2024-03-24T01:26:17.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.275+0000 7f9c4f000640 1 -- 172.21.15.90:0/3915074964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c480cf150 msgr2=0x7f9c480ae530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:17.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.275+0000 7f9c4f000640 1 --2- 172.21.15.90:0/3915074964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c480cf150 0x7f9c480ae530 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f9c4806c2d0 tx=0x7f9c4002be40 comp rx=0 tx=0).stop 2024-03-24T01:26:17.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.275+0000 7f9c4f000640 1 -- 172.21.15.90:0/3915074964 shutdown_connections 2024-03-24T01:26:17.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.275+0000 7f9c4f000640 1 --2- 172.21.15.90:0/3915074964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c480cf150 0x7f9c480ae530 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:17.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.275+0000 7f9c4f000640 1 --2- 172.21.15.90:0/3915074964 >> v2:172.21.15.120:3300/0 conn(0x7f9c48159620 0x7f9c4806b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:17.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.275+0000 7f9c4f000640 1 -- 172.21.15.90:0/3915074964 >> 172.21.15.90:0/3915074964 conn(0x7f9c480b1770 msgr2=0x7f9c480b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:17.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.276+0000 7f9c4f000640 1 -- 172.21.15.90:0/3915074964 shutdown_connections 2024-03-24T01:26:17.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.276+0000 7f9c4f000640 1 -- 172.21.15.90:0/3915074964 wait complete. 2024-03-24T01:26:17.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.277+0000 7f9c4f000640 1 Processor -- start 2024-03-24T01:26:17.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.277+0000 7f9c4f000640 1 -- start start 2024-03-24T01:26:17.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.278+0000 7f9c4f000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c4806b600 0x7f9c480c9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:17.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.278+0000 7f9c4f000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9c480cf150 0x7f9c481e4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:17.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.278+0000 7f9c4f000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c48159620 0x7f9c481f6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:17.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.278+0000 7f9c4f000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c48159a00 con 0x7f9c4806b600 2024-03-24T01:26:17.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.278+0000 7f9c4f000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c480b3f50 con 0x7f9c48159620 2024-03-24T01:26:17.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.278+0000 7f9c4f000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4815bfe0 con 0x7f9c480cf150 2024-03-24T01:26:17.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.279+0000 7f9c47e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c4806b600 0x7f9c480c9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:17.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.279+0000 7f9c47e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c4806b600 0x7f9c480c9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40216/0 (socket says 172.21.15.90:40216) 2024-03-24T01:26:17.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.279+0000 7f9c47e00640 1 -- 172.21.15.90:0/3320193412 learned_addr learned my addr 172.21.15.90:0/3320193412 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:17.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.279+0000 7f9c47e00640 1 -- 172.21.15.90:0/3320193412 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9c480cf150 msgr2=0x7f9c481e4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:17.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.279+0000 7f9c47e00640 1 --2- 172.21.15.90:0/3320193412 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9c480cf150 0x7f9c481e4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:17.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.279+0000 7f9c47e00640 1 -- 172.21.15.90:0/3320193412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c48159620 msgr2=0x7f9c481f6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:17.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.279+0000 7f9c47e00640 1 --2- 172.21.15.90:0/3320193412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c48159620 0x7f9c481f6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:17.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.279+0000 7f9c47e00640 1 -- 172.21.15.90:0/3320193412 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c48144c10 con 0x7f9c4806b600 2024-03-24T01:26:17.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.279+0000 7f9c47e00640 1 --2- 172.21.15.90:0/3320193412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c4806b600 0x7f9c480c9ed0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f9c3800e870 tx=0x7f9c3800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:17.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.280+0000 7f9c3fe00640 1 -- 172.21.15.90:0/3320193412 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c38017070 con 0x7f9c4806b600 2024-03-24T01:26:17.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.280+0000 7f9c3fe00640 1 -- 172.21.15.90:0/3320193412 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c3800c940 con 0x7f9c4806b600 2024-03-24T01:26:17.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.280+0000 7f9c3fe00640 1 -- 172.21.15.90:0/3320193412 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c3800cd30 con 0x7f9c4806b600 2024-03-24T01:26:17.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.280+0000 7f9c4f000640 1 -- 172.21.15.90:0/3320193412 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c481e4ce0 con 0x7f9c4806b600 2024-03-24T01:26:17.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.280+0000 7f9c4f000640 1 -- 172.21.15.90:0/3320193412 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c48144ed0 con 0x7f9c4806b600 2024-03-24T01:26:17.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.281+0000 7f9c4f000640 1 -- 172.21.15.90:0/3320193412 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c4804f400 con 0x7f9c4806b600 2024-03-24T01:26:17.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.283+0000 7f9c3fe00640 1 -- 172.21.15.90:0/3320193412 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9c3801a070 con 0x7f9c4806b600 2024-03-24T01:26:17.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.284+0000 7f9c3fe00640 1 --2- 172.21.15.90:0/3320193412 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9c2806bbd0 0x7f9c2806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:17.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.284+0000 7f9c3fe00640 1 -- 172.21.15.90:0/3320193412 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9c3808aa40 con 0x7f9c4806b600 2024-03-24T01:26:17.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.285+0000 7f9c47400640 1 --2- 172.21.15.90:0/3320193412 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9c2806bbd0 0x7f9c2806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:17.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.286+0000 7f9c47400640 1 --2- 172.21.15.90:0/3320193412 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9c2806bbd0 0x7f9c2806e090 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9c34000f60 tx=0x7f9c340078f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:17.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.289+0000 7f9c3fe00640 1 -- 172.21.15.90:0/3320193412 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9c38059430 con 0x7f9c4806b600 2024-03-24T01:26:17.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.518+0000 7f9c4f000640 1 -- 172.21.15.90:0/3320193412 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9c480b2e90 con 0x7f9c4806b600 2024-03-24T01:26:17.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.519+0000 7f9c3fe00640 1 -- 172.21.15.90:0/3320193412 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v39) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9c38058d30 con 0x7f9c4806b600 2024-03-24T01:26:17.520 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:26:17.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.522+0000 7f9c4f000640 1 -- 172.21.15.90:0/3320193412 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9c2806bbd0 msgr2=0x7f9c2806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:17.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.522+0000 7f9c4f000640 1 --2- 172.21.15.90:0/3320193412 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9c2806bbd0 0x7f9c2806e090 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9c34000f60 tx=0x7f9c340078f0 comp rx=0 tx=0).stop 2024-03-24T01:26:17.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.522+0000 7f9c4f000640 1 -- 172.21.15.90:0/3320193412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c4806b600 msgr2=0x7f9c480c9ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:17.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.522+0000 7f9c4f000640 1 --2- 172.21.15.90:0/3320193412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c4806b600 0x7f9c480c9ed0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f9c3800e870 tx=0x7f9c3800ed30 comp rx=0 tx=0).stop 2024-03-24T01:26:17.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.522+0000 7f9c4f000640 1 -- 172.21.15.90:0/3320193412 shutdown_connections 2024-03-24T01:26:17.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.522+0000 7f9c4f000640 1 --2- 172.21.15.90:0/3320193412 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9c2806bbd0 0x7f9c2806e090 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:17.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.522+0000 7f9c4f000640 1 --2- 172.21.15.90:0/3320193412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c48159620 0x7f9c481f6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:17.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.522+0000 7f9c4f000640 1 --2- 172.21.15.90:0/3320193412 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9c480cf150 0x7f9c481e4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:17.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.522+0000 7f9c4f000640 1 --2- 172.21.15.90:0/3320193412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c4806b600 0x7f9c480c9ed0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:17.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.522+0000 7f9c4f000640 1 -- 172.21.15.90:0/3320193412 >> 172.21.15.90:0/3320193412 conn(0x7f9c480b1770 msgr2=0x7f9c480cc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:17.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.522+0000 7f9c4f000640 1 -- 172.21.15.90:0/3320193412 shutdown_connections 2024-03-24T01:26:17.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.522+0000 7f9c4f000640 1 -- 172.21.15.90:0/3320193412 wait complete. 2024-03-24T01:26:17.542 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T01:26:17.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.746+0000 7f3b91400640 1 Processor -- start 2024-03-24T01:26:17.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.746+0000 7f3b91400640 1 -- start start 2024-03-24T01:26:17.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.749+0000 7f3b91400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3b8c0b2280 0x7f3b8c0b2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:17.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.749+0000 7f3b91400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b8c0b2fc0 con 0x7f3b8c06cbc0 2024-03-24T01:26:17.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.749+0000 7f3b91400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b8c09a780 con 0x7f3b8c0cfd20 2024-03-24T01:26:17.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.749+0000 7f3b91400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b8c09aa60 con 0x7f3b8c0b2280 2024-03-24T01:26:17.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.749+0000 7f3b89e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3b8c0b2280 0x7f3b8c0b2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:17.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.749+0000 7f3b89e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3b8c0b2280 0x7f3b8c0b2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40232/0 (socket says 172.21.15.90:40232) 2024-03-24T01:26:17.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.749+0000 7f3b89e00640 1 -- 172.21.15.90:0/719010740 learned_addr learned my addr 172.21.15.90:0/719010740 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:17.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.750+0000 7f3b89e00640 1 -- 172.21.15.90:0/719010740 >> v1:172.21.15.120:6789/0 conn(0x7f3b8c0cfd20 legacy=0x7f3b8c0d0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:17.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.750+0000 7f3b89e00640 1 -- 172.21.15.90:0/719010740 >> v1:172.21.15.90:6789/0 conn(0x7f3b8c06cbc0 legacy=0x7f3b8c06cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:17.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.750+0000 7f3b89e00640 1 -- 172.21.15.90:0/719010740 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b8c09ad40 con 0x7f3b8c0b2280 2024-03-24T01:26:17.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.751+0000 7f3b89e00640 1 --2- 172.21.15.90:0/719010740 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b8c0b2280 0x7f3b8c0b2920 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f3b78009870 tx=0x7f3b7802b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e40ca6f26535c7b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:17.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.751+0000 7f3b89400640 1 -- 172.21.15.90:0/719010740 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b78034070 con 0x7f3b8c0b2280 2024-03-24T01:26:17.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.751+0000 7f3b89400640 1 -- 172.21.15.90:0/719010740 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b78030850 con 0x7f3b8c0b2280 2024-03-24T01:26:17.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.751+0000 7f3b89400640 1 -- 172.21.15.90:0/719010740 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b78030c20 con 0x7f3b8c0b2280 2024-03-24T01:26:17.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.752+0000 7f3b91400640 1 -- 172.21.15.90:0/719010740 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b8c0b2280 msgr2=0x7f3b8c0b2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:17.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.752+0000 7f3b91400640 1 --2- 172.21.15.90:0/719010740 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b8c0b2280 0x7f3b8c0b2920 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f3b78009870 tx=0x7f3b7802b990 comp rx=0 tx=0).stop 2024-03-24T01:26:17.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.752+0000 7f3b91400640 1 -- 172.21.15.90:0/719010740 shutdown_connections 2024-03-24T01:26:17.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.752+0000 7f3b91400640 1 --2- 172.21.15.90:0/719010740 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b8c0b2280 0x7f3b8c0b2920 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:17.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.752+0000 7f3b91400640 1 -- 172.21.15.90:0/719010740 >> 172.21.15.90:0/719010740 conn(0x7f3b8c0b34e0 msgr2=0x7f3b8c0b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:17.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.752+0000 7f3b91400640 1 -- 172.21.15.90:0/719010740 shutdown_connections 2024-03-24T01:26:17.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.752+0000 7f3b91400640 1 -- 172.21.15.90:0/719010740 wait complete. 2024-03-24T01:26:17.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.754+0000 7f3b91400640 1 Processor -- start 2024-03-24T01:26:17.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.754+0000 7f3b91400640 1 -- start start 2024-03-24T01:26:17.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.755+0000 7f3b91400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b8c06cbc0 0x7f3b8c0b6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:17.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.755+0000 7f3b91400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b8c0b2280 0x7f3b8c070420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:17.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.755+0000 7f3b91400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b8c0cfd20 0x7f3b8c070960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:17.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.755+0000 7f3b91400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8c1617e0 con 0x7f3b8c0cfd20 2024-03-24T01:26:17.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.755+0000 7f3b91400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8c161660 con 0x7f3b8c06cbc0 2024-03-24T01:26:17.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.755+0000 7f3b91400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8c0d0860 con 0x7f3b8c0b2280 2024-03-24T01:26:17.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.755+0000 7f3b8b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b8c0cfd20 0x7f3b8c070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:17.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.755+0000 7f3b8b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b8c0cfd20 0x7f3b8c070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40234/0 (socket says 172.21.15.90:40234) 2024-03-24T01:26:17.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.755+0000 7f3b8b200640 1 -- 172.21.15.90:0/2895191543 learned_addr learned my addr 172.21.15.90:0/2895191543 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:17.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.756+0000 7f3b8a800640 1 --2- 172.21.15.90:0/2895191543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b8c06cbc0 0x7f3b8c0b6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:17.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.756+0000 7f3b8b200640 1 -- 172.21.15.90:0/2895191543 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b8c0b2280 msgr2=0x7f3b8c070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:17.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.756+0000 7f3b89e00640 1 --2- 172.21.15.90:0/2895191543 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b8c0b2280 0x7f3b8c070420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:17.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.756+0000 7f3b8b200640 1 --2- 172.21.15.90:0/2895191543 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b8c0b2280 0x7f3b8c070420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:17.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.756+0000 7f3b8b200640 1 -- 172.21.15.90:0/2895191543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b8c06cbc0 msgr2=0x7f3b8c0b6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:17.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.756+0000 7f3b8b200640 1 --2- 172.21.15.90:0/2895191543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b8c06cbc0 0x7f3b8c0b6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:17.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.756+0000 7f3b8b200640 1 -- 172.21.15.90:0/2895191543 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b8c1f85b0 con 0x7f3b8c0cfd20 2024-03-24T01:26:17.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.756+0000 7f3b8a800640 1 --2- 172.21.15.90:0/2895191543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b8c06cbc0 0x7f3b8c0b6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:17.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.756+0000 7f3b8b200640 1 --2- 172.21.15.90:0/2895191543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b8c0cfd20 0x7f3b8c070960 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f3b8400b970 tx=0x7f3b8400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:17.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.756+0000 7f3b82a00640 1 -- 172.21.15.90:0/2895191543 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b84019070 con 0x7f3b8c0cfd20 2024-03-24T01:26:17.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.756+0000 7f3b82a00640 1 -- 172.21.15.90:0/2895191543 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b8400cb40 con 0x7f3b8c0cfd20 2024-03-24T01:26:17.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.756+0000 7f3b91400640 1 -- 172.21.15.90:0/2895191543 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b8c1f7a30 con 0x7f3b8c0cfd20 2024-03-24T01:26:17.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.756+0000 7f3b82a00640 1 -- 172.21.15.90:0/2895191543 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b84014440 con 0x7f3b8c0cfd20 2024-03-24T01:26:17.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.756+0000 7f3b91400640 1 -- 172.21.15.90:0/2895191543 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b8c1f7ff0 con 0x7f3b8c0cfd20 2024-03-24T01:26:17.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.758+0000 7f3b91400640 1 -- 172.21.15.90:0/2895191543 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b8c04f400 con 0x7f3b8c0cfd20 2024-03-24T01:26:17.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.758+0000 7f3b82a00640 1 -- 172.21.15.90:0/2895191543 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3b8401c070 con 0x7f3b8c0cfd20 2024-03-24T01:26:17.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.760+0000 7f3b82a00640 1 --2- 172.21.15.90:0/2895191543 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3b6c06bbd0 0x7f3b6c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:17.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.760+0000 7f3b82a00640 1 -- 172.21.15.90:0/2895191543 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3b8408a8b0 con 0x7f3b8c0cfd20 2024-03-24T01:26:17.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.761+0000 7f3b8a800640 1 --2- 172.21.15.90:0/2895191543 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3b6c06bbd0 0x7f3b6c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:17.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.762+0000 7f3b8a800640 1 --2- 172.21.15.90:0/2895191543 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3b6c06bbd0 0x7f3b6c06e090 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c071190 tx=0x7f3b7c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:17.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.764+0000 7f3b82a00640 1 -- 172.21.15.90:0/2895191543 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3b84059170 con 0x7f3b8c0cfd20 2024-03-24T01:26:17.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.961+0000 7f3b91400640 1 -- 172.21.15.90:0/2895191543 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3b8c09a540 con 0x7f3b8c0cfd20 2024-03-24T01:26:17.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.962+0000 7f3b82a00640 1 -- 172.21.15.90:0/2895191543 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v39) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3b84058a70 con 0x7f3b8c0cfd20 2024-03-24T01:26:17.963 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-24T01:26:17.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.964+0000 7f3b91400640 1 -- 172.21.15.90:0/2895191543 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3b6c06bbd0 msgr2=0x7f3b6c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:17.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.964+0000 7f3b91400640 1 --2- 172.21.15.90:0/2895191543 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3b6c06bbd0 0x7f3b6c06e090 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c071190 tx=0x7f3b7c009040 comp rx=0 tx=0).stop 2024-03-24T01:26:17.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.964+0000 7f3b91400640 1 -- 172.21.15.90:0/2895191543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b8c0cfd20 msgr2=0x7f3b8c070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:17.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.964+0000 7f3b91400640 1 --2- 172.21.15.90:0/2895191543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b8c0cfd20 0x7f3b8c070960 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f3b8400b970 tx=0x7f3b8400be30 comp rx=0 tx=0).stop 2024-03-24T01:26:17.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.964+0000 7f3b91400640 1 -- 172.21.15.90:0/2895191543 shutdown_connections 2024-03-24T01:26:17.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.964+0000 7f3b91400640 1 --2- 172.21.15.90:0/2895191543 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3b6c06bbd0 0x7f3b6c06e090 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:17.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.964+0000 7f3b91400640 1 --2- 172.21.15.90:0/2895191543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b8c0cfd20 0x7f3b8c070960 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:17.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.964+0000 7f3b91400640 1 --2- 172.21.15.90:0/2895191543 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b8c0b2280 0x7f3b8c070420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:17.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.964+0000 7f3b91400640 1 --2- 172.21.15.90:0/2895191543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b8c06cbc0 0x7f3b8c0b6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:17.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.965+0000 7f3b91400640 1 -- 172.21.15.90:0/2895191543 >> 172.21.15.90:0/2895191543 conn(0x7f3b8c0b34e0 msgr2=0x7f3b8c0d0130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:17.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.965+0000 7f3b91400640 1 -- 172.21.15.90:0/2895191543 shutdown_connections 2024-03-24T01:26:17.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:17.965+0000 7f3b91400640 1 -- 172.21.15.90:0/2895191543 wait complete. 2024-03-24T01:26:17.984 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T01:26:18.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.205+0000 7f85eb200640 1 Processor -- start 2024-03-24T01:26:18.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.205+0000 7f85eb200640 1 -- start start 2024-03-24T01:26:18.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.207+0000 7f85eb200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f85e40aee80 0x7f85e40b33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:18.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.207+0000 7f85eb200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f85e40b3930 0x7f85e40cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:18.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.207+0000 7f85eb200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85e40b3eb0 con 0x7f85e40ccde0 2024-03-24T01:26:18.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.207+0000 7f85eb200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85e406e680 con 0x7f85e40b3930 2024-03-24T01:26:18.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.207+0000 7f85eb200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85e406e960 con 0x7f85e40aee80 2024-03-24T01:26:18.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.207+0000 7f85dbe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f85e40b3930 0x7f85e40cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:18.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.207+0000 7f85dbe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f85e40b3930 0x7f85e40cc8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40244/0 (socket says 172.21.15.90:40244) 2024-03-24T01:26:18.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.207+0000 7f85dbe00640 1 -- 172.21.15.90:0/2990042150 learned_addr learned my addr 172.21.15.90:0/2990042150 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:18.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.207+0000 7f85e3e00640 1 --2- 172.21.15.90:0/2990042150 >> v2:172.21.15.120:3300/0 conn(0x7f85e40aee80 0x7f85e40b33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:18.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.208+0000 7f85dbe00640 1 -- 172.21.15.90:0/2990042150 >> v2:172.21.15.120:3300/0 conn(0x7f85e40aee80 msgr2=0x7f85e40b33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:18.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.208+0000 7f85dbe00640 1 --2- 172.21.15.90:0/2990042150 >> v2:172.21.15.120:3300/0 conn(0x7f85e40aee80 0x7f85e40b33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:18.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.208+0000 7f85dbe00640 1 -- 172.21.15.90:0/2990042150 >> v1:172.21.15.120:6789/0 conn(0x7f85e40ccde0 legacy=0x7f85e406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:18.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.208+0000 7f85dbe00640 1 -- 172.21.15.90:0/2990042150 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e406ec40 con 0x7f85e40b3930 2024-03-24T01:26:18.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.209+0000 7f85dbe00640 1 --2- 172.21.15.90:0/2990042150 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85e40b3930 0x7f85e40cc8a0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f85e4072e30 tx=0x7f85cc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95d5c7c1350627ac server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:18.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.209+0000 7f85e3400640 1 -- 172.21.15.90:0/2990042150 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85cc02e070 con 0x7f85e40b3930 2024-03-24T01:26:18.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.209+0000 7f85e3400640 1 -- 172.21.15.90:0/2990042150 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85cc037070 con 0x7f85e40b3930 2024-03-24T01:26:18.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.209+0000 7f85e3400640 1 -- 172.21.15.90:0/2990042150 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85cc031a00 con 0x7f85e40b3930 2024-03-24T01:26:18.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.210+0000 7f85eb200640 1 -- 172.21.15.90:0/2990042150 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85e40b3930 msgr2=0x7f85e40cc8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:18.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.210+0000 7f85eb200640 1 --2- 172.21.15.90:0/2990042150 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85e40b3930 0x7f85e40cc8a0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f85e4072e30 tx=0x7f85cc02b880 comp rx=0 tx=0).stop 2024-03-24T01:26:18.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.210+0000 7f85eb200640 1 -- 172.21.15.90:0/2990042150 shutdown_connections 2024-03-24T01:26:18.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.210+0000 7f85eb200640 1 --2- 172.21.15.90:0/2990042150 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85e40b3930 0x7f85e40cc8a0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:18.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.210+0000 7f85eb200640 1 --2- 172.21.15.90:0/2990042150 >> v2:172.21.15.120:3300/0 conn(0x7f85e40aee80 0x7f85e40b33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:18.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.210+0000 7f85eb200640 1 -- 172.21.15.90:0/2990042150 >> 172.21.15.90:0/2990042150 conn(0x7f85e40b1770 msgr2=0x7f85e40b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:18.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.210+0000 7f85eb200640 1 -- 172.21.15.90:0/2990042150 shutdown_connections 2024-03-24T01:26:18.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.210+0000 7f85eb200640 1 -- 172.21.15.90:0/2990042150 wait complete. 2024-03-24T01:26:18.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.212+0000 7f85eb200640 1 Processor -- start 2024-03-24T01:26:18.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.212+0000 7f85eb200640 1 -- start start 2024-03-24T01:26:18.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.212+0000 7f85eb200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85e40aee80 0x7f85e4147390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:18.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.213+0000 7f85eb200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85e40b3930 0x7f85e415ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:18.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.213+0000 7f85eb200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85e40ccde0 0x7f85e415cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:18.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.213+0000 7f85eb200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e406b7d0 con 0x7f85e40aee80 2024-03-24T01:26:18.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.213+0000 7f85eb200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e4166ad0 con 0x7f85e40ccde0 2024-03-24T01:26:18.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.213+0000 7f85eb200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85e406b950 con 0x7f85e40b3930 2024-03-24T01:26:18.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.213+0000 7f85e3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85e40aee80 0x7f85e4147390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:18.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.213+0000 7f85e3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85e40aee80 0x7f85e4147390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40258/0 (socket says 172.21.15.90:40258) 2024-03-24T01:26:18.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.213+0000 7f85e3e00640 1 -- 172.21.15.90:0/225611591 learned_addr learned my addr 172.21.15.90:0/225611591 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:18.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.213+0000 7f85e3e00640 1 -- 172.21.15.90:0/225611591 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85e40b3930 msgr2=0x7f85e415ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:18.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.213+0000 7f85e3e00640 1 --2- 172.21.15.90:0/225611591 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85e40b3930 0x7f85e415ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:18.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.213+0000 7f85e3e00640 1 -- 172.21.15.90:0/225611591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85e40ccde0 msgr2=0x7f85e415cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:18.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.213+0000 7f85e3e00640 1 --2- 172.21.15.90:0/225611591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85e40ccde0 0x7f85e415cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:18.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.213+0000 7f85e3e00640 1 -- 172.21.15.90:0/225611591 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e415b800 con 0x7f85e40aee80 2024-03-24T01:26:18.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.214+0000 7f85e3e00640 1 --2- 172.21.15.90:0/225611591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85e40aee80 0x7f85e4147390 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f85d400d800 tx=0x7f85d400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:18.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.214+0000 7f85e0c00640 1 -- 172.21.15.90:0/225611591 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d400b6c0 con 0x7f85e40aee80 2024-03-24T01:26:18.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.214+0000 7f85e0c00640 1 -- 172.21.15.90:0/225611591 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d4019070 con 0x7f85e40aee80 2024-03-24T01:26:18.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.214+0000 7f85eb200640 1 -- 172.21.15.90:0/225611591 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85e415b520 con 0x7f85e40aee80 2024-03-24T01:26:18.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.214+0000 7f85e0c00640 1 -- 172.21.15.90:0/225611591 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d400ba80 con 0x7f85e40aee80 2024-03-24T01:26:18.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.214+0000 7f85eb200640 1 -- 172.21.15.90:0/225611591 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85e4147ed0 con 0x7f85e40aee80 2024-03-24T01:26:18.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.215+0000 7f85eb200640 1 -- 172.21.15.90:0/225611591 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85e404f400 con 0x7f85e40aee80 2024-03-24T01:26:18.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.216+0000 7f85e0c00640 1 -- 172.21.15.90:0/225611591 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f85d400bc20 con 0x7f85e40aee80 2024-03-24T01:26:18.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.217+0000 7f85e0c00640 1 --2- 172.21.15.90:0/225611591 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f85c006bbd0 0x7f85c006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:18.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.217+0000 7f85e0c00640 1 -- 172.21.15.90:0/225611591 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f85d408b160 con 0x7f85e40aee80 2024-03-24T01:26:18.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.218+0000 7f85dbe00640 1 --2- 172.21.15.90:0/225611591 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f85c006bbd0 0x7f85c006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:18.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.219+0000 7f85dbe00640 1 --2- 172.21.15.90:0/225611591 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f85c006bbd0 0x7f85c006e090 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f85cc036000 tx=0x7f85cc030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:18.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.221+0000 7f85e0c00640 1 -- 172.21.15.90:0/225611591 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f85d4059a20 con 0x7f85e40aee80 2024-03-24T01:26:18.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.513+0000 7f85eb200640 1 -- 172.21.15.90:0/225611591 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f85e406e250 con 0x7f85e40aee80 2024-03-24T01:26:18.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.514+0000 7f85e0c00640 1 -- 172.21.15.90:0/225611591 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v39) v1 ==== 80+0+0 (secure 0 0 0) 0x7f85d4059320 con 0x7f85e40aee80 2024-03-24T01:26:18.515 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-24T01:26:18.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.516+0000 7f85eb200640 1 -- 172.21.15.90:0/225611591 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f85c006bbd0 msgr2=0x7f85c006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:18.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.516+0000 7f85eb200640 1 --2- 172.21.15.90:0/225611591 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f85c006bbd0 0x7f85c006e090 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f85cc036000 tx=0x7f85cc030000 comp rx=0 tx=0).stop 2024-03-24T01:26:18.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.516+0000 7f85eb200640 1 -- 172.21.15.90:0/225611591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85e40aee80 msgr2=0x7f85e4147390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:18.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.516+0000 7f85eb200640 1 --2- 172.21.15.90:0/225611591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85e40aee80 0x7f85e4147390 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f85d400d800 tx=0x7f85d400dcc0 comp rx=0 tx=0).stop 2024-03-24T01:26:18.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.517+0000 7f85eb200640 1 -- 172.21.15.90:0/225611591 shutdown_connections 2024-03-24T01:26:18.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.517+0000 7f85eb200640 1 --2- 172.21.15.90:0/225611591 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f85c006bbd0 0x7f85c006e090 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:18.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.517+0000 7f85eb200640 1 --2- 172.21.15.90:0/225611591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85e40ccde0 0x7f85e415cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:18.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.517+0000 7f85eb200640 1 --2- 172.21.15.90:0/225611591 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f85e40b3930 0x7f85e415ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:18.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.517+0000 7f85eb200640 1 --2- 172.21.15.90:0/225611591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85e40aee80 0x7f85e4147390 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:18.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.517+0000 7f85eb200640 1 -- 172.21.15.90:0/225611591 >> 172.21.15.90:0/225611591 conn(0x7f85e40b1770 msgr2=0x7f85e40cd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:18.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.517+0000 7f85eb200640 1 -- 172.21.15.90:0/225611591 shutdown_connections 2024-03-24T01:26:18.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.517+0000 7f85eb200640 1 -- 172.21.15.90:0/225611591 wait complete. 2024-03-24T01:26:18.535 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:26:18.535 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:26:18.577 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:26:18.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.850+0000 7f2b87e00640 1 Processor -- start 2024-03-24T01:26:18.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.850+0000 7f2b87e00640 1 -- start start 2024-03-24T01:26:18.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.851+0000 7f2b87e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f2b88144420 0x7f2b8806d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:18.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.851+0000 7f2b87e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2b88144800 0x7f2b880cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:18.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.851+0000 7f2b87e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b880cceb0 con 0x7f2b880cc8f0 2024-03-24T01:26:18.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.851+0000 7f2b87e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b880cd190 con 0x7f2b88144800 2024-03-24T01:26:18.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.851+0000 7f2b87e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b880ceb70 con 0x7f2b88144420 2024-03-24T01:26:18.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.852+0000 7f2b86000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2b88144800 0x7f2b880cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:18.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.852+0000 7f2b86000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2b88144800 0x7f2b880cf370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40272/0 (socket says 172.21.15.90:40272) 2024-03-24T01:26:18.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.852+0000 7f2b86000640 1 -- 172.21.15.90:0/4222755201 learned_addr learned my addr 172.21.15.90:0/4222755201 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:18.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.852+0000 7f2b87400640 1 --1- 172.21.15.90:0/4222755201 >> v1:172.21.15.90:6789/0 conn(0x7f2b880cc8f0 0x7f2b880cf8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51656/0 (socket says 172.21.15.90:51656) 2024-03-24T01:26:18.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.852+0000 7f2b86a00640 1 --2- 172.21.15.90:0/4222755201 >> v2:172.21.15.120:3300/0 conn(0x7f2b88144420 0x7f2b8806d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:18.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.852+0000 7f2b86000640 1 -- 172.21.15.90:0/4222755201 >> v2:172.21.15.120:3300/0 conn(0x7f2b88144420 msgr2=0x7f2b8806d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:18.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.852+0000 7f2b86000640 1 --2- 172.21.15.90:0/4222755201 >> v2:172.21.15.120:3300/0 conn(0x7f2b88144420 0x7f2b8806d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:18.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.852+0000 7f2b86000640 1 -- 172.21.15.90:0/4222755201 >> v1:172.21.15.90:6789/0 conn(0x7f2b880cc8f0 legacy=0x7f2b880cf8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:18.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.852+0000 7f2b86000640 1 -- 172.21.15.90:0/4222755201 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b880cee50 con 0x7f2b88144800 2024-03-24T01:26:18.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.852+0000 7f2b85600640 1 -- 172.21.15.90:0/4222755201 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1112074222 0 0) 0x7f2b880cceb0 con 0x7f2b880cc8f0 2024-03-24T01:26:18.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.853+0000 7f2b86000640 1 --2- 172.21.15.90:0/4222755201 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b88144800 0x7f2b880cf370 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f2b8806f760 tx=0x7f2b7402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5493d9101ad14bfa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:18.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.854+0000 7f2b85600640 1 -- 172.21.15.90:0/4222755201 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b74035020 con 0x7f2b88144800 2024-03-24T01:26:18.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.854+0000 7f2b85600640 1 -- 172.21.15.90:0/4222755201 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b74031870 con 0x7f2b88144800 2024-03-24T01:26:18.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.854+0000 7f2b85600640 1 -- 172.21.15.90:0/4222755201 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b74031c40 con 0x7f2b88144800 2024-03-24T01:26:18.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.855+0000 7f2b87e00640 1 -- 172.21.15.90:0/4222755201 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b88144800 msgr2=0x7f2b880cf370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:18.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.855+0000 7f2b87e00640 1 --2- 172.21.15.90:0/4222755201 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b88144800 0x7f2b880cf370 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f2b8806f760 tx=0x7f2b7402b910 comp rx=0 tx=0).stop 2024-03-24T01:26:18.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.855+0000 7f2b87e00640 1 -- 172.21.15.90:0/4222755201 shutdown_connections 2024-03-24T01:26:18.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.855+0000 7f2b87e00640 1 --2- 172.21.15.90:0/4222755201 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b88144800 0x7f2b880cf370 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:18.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.855+0000 7f2b87e00640 1 --2- 172.21.15.90:0/4222755201 >> v2:172.21.15.120:3300/0 conn(0x7f2b88144420 0x7f2b8806d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:18.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.855+0000 7f2b87e00640 1 -- 172.21.15.90:0/4222755201 >> 172.21.15.90:0/4222755201 conn(0x7f2b8806b0c0 msgr2=0x7f2b8806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:18.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.855+0000 7f2b87e00640 1 -- 172.21.15.90:0/4222755201 shutdown_connections 2024-03-24T01:26:18.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.855+0000 7f2b87e00640 1 -- 172.21.15.90:0/4222755201 wait complete. 2024-03-24T01:26:18.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.858+0000 7f2b87e00640 1 Processor -- start 2024-03-24T01:26:18.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.858+0000 7f2b87e00640 1 -- start start 2024-03-24T01:26:18.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.860+0000 7f2b87e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b880cc8f0 0x7f2b880a8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:18.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.860+0000 7f2b87e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b88144420 0x7f2b881431a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:18.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.860+0000 7f2b87e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b88144800 0x7f2b881f41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:18.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.860+0000 7f2b87e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b880ce910 con 0x7f2b88144420 2024-03-24T01:26:18.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.860+0000 7f2b87e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b880ce790 con 0x7f2b88144800 2024-03-24T01:26:18.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.860+0000 7f2b87e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8815f540 con 0x7f2b880cc8f0 2024-03-24T01:26:18.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.861+0000 7f2b86000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b88144420 0x7f2b881431a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:18.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.861+0000 7f2b86000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b88144420 0x7f2b881431a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40280/0 (socket says 172.21.15.90:40280) 2024-03-24T01:26:18.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.861+0000 7f2b86000640 1 -- 172.21.15.90:0/1912761011 learned_addr learned my addr 172.21.15.90:0/1912761011 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:18.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.861+0000 7f2b86000640 1 -- 172.21.15.90:0/1912761011 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b880cc8f0 msgr2=0x7f2b880a8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:18.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.861+0000 7f2b86000640 1 --2- 172.21.15.90:0/1912761011 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b880cc8f0 0x7f2b880a8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:18.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.861+0000 7f2b86000640 1 -- 172.21.15.90:0/1912761011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b88144800 msgr2=0x7f2b881f41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:18.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.861+0000 7f2b86000640 1 --2- 172.21.15.90:0/1912761011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b88144800 0x7f2b881f41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:18.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.861+0000 7f2b86000640 1 -- 172.21.15.90:0/1912761011 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b880a9220 con 0x7f2b88144420 2024-03-24T01:26:18.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.861+0000 7f2b86000640 1 --2- 172.21.15.90:0/1912761011 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b88144420 0x7f2b881431a0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f2b74000c00 tx=0x7f2b740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:18.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.862+0000 7f2b7ea00640 1 -- 172.21.15.90:0/1912761011 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b74040070 con 0x7f2b88144420 2024-03-24T01:26:18.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.862+0000 7f2b7ea00640 1 -- 172.21.15.90:0/1912761011 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b74030d00 con 0x7f2b88144420 2024-03-24T01:26:18.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.862+0000 7f2b7ea00640 1 -- 172.21.15.90:0/1912761011 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b74035040 con 0x7f2b88144420 2024-03-24T01:26:18.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.862+0000 7f2b87e00640 1 -- 172.21.15.90:0/1912761011 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b881f3430 con 0x7f2b88144420 2024-03-24T01:26:18.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.862+0000 7f2b87e00640 1 -- 172.21.15.90:0/1912761011 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b881f39f0 con 0x7f2b88144420 2024-03-24T01:26:18.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.863+0000 7f2b87e00640 1 -- 172.21.15.90:0/1912761011 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b8804f400 con 0x7f2b88144420 2024-03-24T01:26:18.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.864+0000 7f2b7ea00640 1 -- 172.21.15.90:0/1912761011 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2b74038020 con 0x7f2b88144420 2024-03-24T01:26:18.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.865+0000 7f2b7ea00640 1 --2- 172.21.15.90:0/1912761011 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f2b6806bbd0 0x7f2b6806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:18.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.865+0000 7f2b7ea00640 1 -- 172.21.15.90:0/1912761011 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2b740ac940 con 0x7f2b88144420 2024-03-24T01:26:18.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.865+0000 7f2b86a00640 1 --2- 172.21.15.90:0/1912761011 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f2b6806bbd0 0x7f2b6806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:18.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.867+0000 7f2b86a00640 1 --2- 172.21.15.90:0/1912761011 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f2b6806bbd0 0x7f2b6806e090 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2b78006a90 tx=0x7f2b78006a20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:18.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:18.870+0000 7f2b7ea00640 1 -- 172.21.15.90:0/1912761011 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2b880a9220 con 0x7f2b88144420 2024-03-24T01:26:19.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.068+0000 7f2b87e00640 1 -- 172.21.15.90:0/1912761011 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b880b2060 con 0x7f2b88144420 2024-03-24T01:26:19.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.070+0000 7f2b7ea00640 1 -- 172.21.15.90:0/1912761011 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+16918 (secure 0 0 0) 0x7f2b7407b330 con 0x7f2b88144420 2024-03-24T01:26:19.072 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:26:19.072 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":39,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:26:00.983350+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:25:57.398864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-24T01:25:58.451150+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:26:19.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.073+0000 7f2b87e00640 1 -- 172.21.15.90:0/1912761011 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f2b6806bbd0 msgr2=0x7f2b6806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:19.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.073+0000 7f2b87e00640 1 --2- 172.21.15.90:0/1912761011 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f2b6806bbd0 0x7f2b6806e090 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2b78006a90 tx=0x7f2b78006a20 comp rx=0 tx=0).stop 2024-03-24T01:26:19.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.073+0000 7f2b87e00640 1 -- 172.21.15.90:0/1912761011 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b88144420 msgr2=0x7f2b881431a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:19.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.073+0000 7f2b87e00640 1 --2- 172.21.15.90:0/1912761011 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b88144420 0x7f2b881431a0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f2b74000c00 tx=0x7f2b740046c0 comp rx=0 tx=0).stop 2024-03-24T01:26:19.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.073+0000 7f2b87e00640 1 -- 172.21.15.90:0/1912761011 shutdown_connections 2024-03-24T01:26:19.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.073+0000 7f2b87e00640 1 --2- 172.21.15.90:0/1912761011 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f2b6806bbd0 0x7f2b6806e090 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:19.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.073+0000 7f2b87e00640 1 --2- 172.21.15.90:0/1912761011 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b88144420 0x7f2b881431a0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:19.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.073+0000 7f2b87e00640 1 --2- 172.21.15.90:0/1912761011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b88144800 0x7f2b881f41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:19.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.073+0000 7f2b87e00640 1 --2- 172.21.15.90:0/1912761011 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2b880cc8f0 0x7f2b880a8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:19.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.073+0000 7f2b87e00640 1 -- 172.21.15.90:0/1912761011 >> 172.21.15.90:0/1912761011 conn(0x7f2b8806b0c0 msgr2=0x7f2b88141f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:19.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.073+0000 7f2b87e00640 1 -- 172.21.15.90:0/1912761011 shutdown_connections 2024-03-24T01:26:19.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.073+0000 7f2b87e00640 1 -- 172.21.15.90:0/1912761011 wait complete. 2024-03-24T01:26:19.094 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T01:25:57.398864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T01:25:58.451150+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T01:26:19.095 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:26:19.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.288+0000 7f6103e00640 1 Processor -- start 2024-03-24T01:26:19.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.288+0000 7f6103e00640 1 -- start start 2024-03-24T01:26:19.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.290+0000 7f6103e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f60fc15df40 0x7f60fc161890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:19.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.290+0000 7f6103e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60fc0b94e0 0x7f60fc0ce890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:19.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.290+0000 7f6103e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60fc161dd0 con 0x7f60fc0cedd0 2024-03-24T01:26:19.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.290+0000 7f6103e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60fc0d1c70 con 0x7f60fc15df40 2024-03-24T01:26:19.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.290+0000 7f6103e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60fc1620a0 con 0x7f60fc0b94e0 2024-03-24T01:26:19.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.290+0000 7f6101e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f60fc0cedd0 0x7f60fc0b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51660/0 (socket says 172.21.15.90:51660) 2024-03-24T01:26:19.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.290+0000 7f6101e00640 1 -- 172.21.15.90:0/3535358750 learned_addr learned my addr 172.21.15.90:0/3535358750 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:26:19.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.290+0000 7f6100a00640 1 --2- 172.21.15.90:0/3535358750 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60fc0b94e0 0x7f60fc0ce890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:19.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.290+0000 7f6101400640 1 --2- 172.21.15.90:0/3535358750 >> v2:172.21.15.120:3300/0 conn(0x7f60fc15df40 0x7f60fc161890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:19.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.291+0000 7f60fbe00640 1 -- 172.21.15.90:0/3535358750 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2630860231 0 0) 0x7f60fc161dd0 con 0x7f60fc0cedd0 2024-03-24T01:26:19.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.291+0000 7f60fbe00640 1 -- 172.21.15.90:0/3535358750 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60e4003520 con 0x7f60fc0cedd0 2024-03-24T01:26:19.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.291+0000 7f60fbe00640 1 -- 172.21.15.90:0/3535358750 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2227618106 0 0) 0x7f60e4003520 con 0x7f60fc0cedd0 2024-03-24T01:26:19.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.291+0000 7f60fbe00640 1 -- 172.21.15.90:0/3535358750 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60fc0b94e0 msgr2=0x7f60fc0ce890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:19.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.291+0000 7f60fbe00640 1 --2- 172.21.15.90:0/3535358750 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60fc0b94e0 0x7f60fc0ce890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:19.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.291+0000 7f60fbe00640 1 -- 172.21.15.90:0/3535358750 >> v2:172.21.15.120:3300/0 conn(0x7f60fc15df40 msgr2=0x7f60fc161890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:19.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.291+0000 7f60fbe00640 1 --2- 172.21.15.90:0/3535358750 >> v2:172.21.15.120:3300/0 conn(0x7f60fc15df40 0x7f60fc161890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:19.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.291+0000 7f60fbe00640 1 -- 172.21.15.90:0/3535358750 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60fc162380 con 0x7f60fc0cedd0 2024-03-24T01:26:19.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.291+0000 7f60fbe00640 1 -- 172.21.15.90:0/3535358750 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f60f40019f0 con 0x7f60fc0cedd0 2024-03-24T01:26:19.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.291+0000 7f60fbe00640 1 -- 172.21.15.90:0/3535358750 >> v1:172.21.15.90:6789/0 conn(0x7f60fc0cedd0 legacy=0x7f60fc0b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:19.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.291+0000 7f60fbe00640 1 --2- 172.21.15.90:0/3535358750 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60fc0b94e0 0x7f60e40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:19.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.291+0000 7f60fbe00640 1 --2- 172.21.15.90:0/3535358750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60e4003f80 0x7f60e4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:19.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.291+0000 7f60fbe00640 1 --2- 172.21.15.90:0/3535358750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60e4004d40 0x7f60e40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:19.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.291+0000 7f60fbe00640 1 -- 172.21.15.90:0/3535358750 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60fc0d1c70 con 0x7f60e4003f80 2024-03-24T01:26:19.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.291+0000 7f60fbe00640 1 -- 172.21.15.90:0/3535358750 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60fc1620a0 con 0x7f60e4004d40 2024-03-24T01:26:19.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.291+0000 7f60fbe00640 1 -- 172.21.15.90:0/3535358750 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60e400b9c0 con 0x7f60fc0b94e0 2024-03-24T01:26:19.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.291+0000 7f6100a00640 1 --2- 172.21.15.90:0/3535358750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60e4003f80 0x7f60e4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:19.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.292+0000 7f6101400640 1 --2- 172.21.15.90:0/3535358750 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60fc0b94e0 0x7f60e40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:19.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.292+0000 7f6100a00640 1 -- 172.21.15.90:0/3535358750 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60fc0b94e0 msgr2=0x7f60e40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:19.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.292+0000 7f6100a00640 1 --2- 172.21.15.90:0/3535358750 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60fc0b94e0 0x7f60e40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:19.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.292+0000 7f6101e00640 1 --2- 172.21.15.90:0/3535358750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60e4004d40 0x7f60e40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:19.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.292+0000 7f6100a00640 1 -- 172.21.15.90:0/3535358750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60e4004d40 msgr2=0x7f60e40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:19.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.292+0000 7f6100a00640 1 --2- 172.21.15.90:0/3535358750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60e4004d40 0x7f60e40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:19.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.292+0000 7f6100a00640 1 -- 172.21.15.90:0/3535358750 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e4003520 con 0x7f60e4003f80 2024-03-24T01:26:19.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.292+0000 7f6101e00640 1 --2- 172.21.15.90:0/3535358750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60e4004d40 0x7f60e40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:19.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.294+0000 7f6100a00640 1 --2- 172.21.15.90:0/3535358750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60e4003f80 0x7f60e4004800 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f60e8009af0 tx=0x7f60e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcf018fa7a3236e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:19.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.294+0000 7f60fbe00640 1 -- 172.21.15.90:0/3535358750 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e8039070 con 0x7f60e4003f80 2024-03-24T01:26:19.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.294+0000 7f60fbe00640 1 -- 172.21.15.90:0/3535358750 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60e802c6b0 con 0x7f60e4003f80 2024-03-24T01:26:19.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.294+0000 7f60fbe00640 1 -- 172.21.15.90:0/3535358750 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e8009690 con 0x7f60e4003f80 2024-03-24T01:26:19.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.295+0000 7f6103e00640 1 -- 172.21.15.90:0/3535358750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60e4003f80 msgr2=0x7f60e4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:19.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.295+0000 7f6103e00640 1 --2- 172.21.15.90:0/3535358750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60e4003f80 0x7f60e4004800 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f60e8009af0 tx=0x7f60e4005830 comp rx=0 tx=0).stop 2024-03-24T01:26:19.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.295+0000 7f6101e00640 1 -- 172.21.15.90:0/3535358750 reap_dead start 2024-03-24T01:26:19.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.295+0000 7f6103e00640 1 -- 172.21.15.90:0/3535358750 shutdown_connections 2024-03-24T01:26:19.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.295+0000 7f6103e00640 1 -- 172.21.15.90:0/3535358750 >> 172.21.15.90:0/3535358750 conn(0x7f60fc06c9f0 msgr2=0x7f60fc06ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:19.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.295+0000 7f6103e00640 1 -- 172.21.15.90:0/3535358750 shutdown_connections 2024-03-24T01:26:19.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.295+0000 7f6103e00640 1 -- 172.21.15.90:0/3535358750 wait complete. 2024-03-24T01:26:19.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.298+0000 7f6103e00640 1 Processor -- start 2024-03-24T01:26:19.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.298+0000 7f6103e00640 1 -- start start 2024-03-24T01:26:19.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.299+0000 7f6103e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60e4003f80 0x7f60fc14a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:19.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.299+0000 7f6103e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60fc15a3a0 0x7f60fc15a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:19.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.299+0000 7f6103e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60fc160320 0x7f60fc1607a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:19.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.299+0000 7f6103e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60fc0d33f0 con 0x7f60fc15a3a0 2024-03-24T01:26:19.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.299+0000 7f6103e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60fc0d1ec0 con 0x7f60fc160320 2024-03-24T01:26:19.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.299+0000 7f6103e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60fc0d3570 con 0x7f60e4003f80 2024-03-24T01:26:19.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.300+0000 7f6100a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60fc15a3a0 0x7f60fc15a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:19.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.300+0000 7f6100a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60fc15a3a0 0x7f60fc15a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40298/0 (socket says 172.21.15.90:40298) 2024-03-24T01:26:19.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.300+0000 7f6100a00640 1 -- 172.21.15.90:0/2222001533 learned_addr learned my addr 172.21.15.90:0/2222001533 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:19.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.300+0000 7f6101400640 1 --2- 172.21.15.90:0/2222001533 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60e4003f80 0x7f60fc14a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:19.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.300+0000 7f6100a00640 1 -- 172.21.15.90:0/2222001533 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60e4003f80 msgr2=0x7f60fc14a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:19.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.300+0000 7f6100a00640 1 --2- 172.21.15.90:0/2222001533 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60e4003f80 0x7f60fc14a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:19.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.300+0000 7f6100a00640 1 -- 172.21.15.90:0/2222001533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60fc160320 msgr2=0x7f60fc1607a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:19.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.300+0000 7f6100a00640 1 --2- 172.21.15.90:0/2222001533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60fc160320 0x7f60fc1607a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:19.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.300+0000 7f6100a00640 1 -- 172.21.15.90:0/2222001533 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60fc14a760 con 0x7f60fc15a3a0 2024-03-24T01:26:19.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.301+0000 7f6100a00640 1 --2- 172.21.15.90:0/2222001533 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60fc15a3a0 0x7f60fc15a7e0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f60e8009af0 tx=0x7f60e8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:19.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.301+0000 7f60f9600640 1 -- 172.21.15.90:0/2222001533 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e8042020 con 0x7f60fc15a3a0 2024-03-24T01:26:19.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.301+0000 7f60f9600640 1 -- 172.21.15.90:0/2222001533 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60e803a040 con 0x7f60fc15a3a0 2024-03-24T01:26:19.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.301+0000 7f6103e00640 1 -- 172.21.15.90:0/2222001533 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60fc15f560 con 0x7f60fc15a3a0 2024-03-24T01:26:19.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.301+0000 7f60f9600640 1 -- 172.21.15.90:0/2222001533 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e8039070 con 0x7f60fc15a3a0 2024-03-24T01:26:19.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.301+0000 7f6103e00640 1 -- 172.21.15.90:0/2222001533 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60fc15fb20 con 0x7f60fc15a3a0 2024-03-24T01:26:19.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.302+0000 7f6103e00640 1 -- 172.21.15.90:0/2222001533 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60fc04f400 con 0x7f60fc15a3a0 2024-03-24T01:26:19.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.303+0000 7f60f9600640 1 -- 172.21.15.90:0/2222001533 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f60e8045020 con 0x7f60fc15a3a0 2024-03-24T01:26:19.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.305+0000 7f60f9600640 1 --2- 172.21.15.90:0/2222001533 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f60dc06bbd0 0x7f60dc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:19.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.305+0000 7f60f9600640 1 -- 172.21.15.90:0/2222001533 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f60e80abf60 con 0x7f60fc15a3a0 2024-03-24T01:26:19.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.306+0000 7f6101400640 1 --2- 172.21.15.90:0/2222001533 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f60dc06bbd0 0x7f60dc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:19.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.307+0000 7f6101400640 1 --2- 172.21.15.90:0/2222001533 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f60dc06bbd0 0x7f60dc06e090 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f60f0009a90 tx=0x7f60f0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:19.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.308+0000 7f60f9600640 1 -- 172.21.15.90:0/2222001533 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f60e804c070 con 0x7f60fc15a3a0 2024-03-24T01:26:19.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.497+0000 7f6103e00640 1 -- 172.21.15.90:0/2222001533 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f60fc0b74a0 con 0x7f60fc15a3a0 2024-03-24T01:26:19.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.498+0000 7f60f9600640 1 -- 172.21.15.90:0/2222001533 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v39) v1 ==== 93+0+10 (secure 0 0 0) 0x7f60e807a950 con 0x7f60fc15a3a0 2024-03-24T01:26:19.500 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:26:19.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.501+0000 7f6103e00640 1 -- 172.21.15.90:0/2222001533 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f60dc06bbd0 msgr2=0x7f60dc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:19.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.501+0000 7f6103e00640 1 --2- 172.21.15.90:0/2222001533 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f60dc06bbd0 0x7f60dc06e090 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f60f0009a90 tx=0x7f60f0008040 comp rx=0 tx=0).stop 2024-03-24T01:26:19.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.501+0000 7f6103e00640 1 -- 172.21.15.90:0/2222001533 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60fc15a3a0 msgr2=0x7f60fc15a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:19.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.501+0000 7f6103e00640 1 --2- 172.21.15.90:0/2222001533 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60fc15a3a0 0x7f60fc15a7e0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f60e8009af0 tx=0x7f60e8004730 comp rx=0 tx=0).stop 2024-03-24T01:26:19.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.501+0000 7f6103e00640 1 -- 172.21.15.90:0/2222001533 shutdown_connections 2024-03-24T01:26:19.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.501+0000 7f6103e00640 1 --2- 172.21.15.90:0/2222001533 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f60dc06bbd0 0x7f60dc06e090 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:19.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.501+0000 7f6103e00640 1 --2- 172.21.15.90:0/2222001533 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60fc15a3a0 0x7f60fc15a7e0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:19.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.501+0000 7f6103e00640 1 --2- 172.21.15.90:0/2222001533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60fc160320 0x7f60fc1607a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:19.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.501+0000 7f6103e00640 1 --2- 172.21.15.90:0/2222001533 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f60e4003f80 0x7f60fc14a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:19.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.501+0000 7f6103e00640 1 -- 172.21.15.90:0/2222001533 >> 172.21.15.90:0/2222001533 conn(0x7f60fc06c9f0 msgr2=0x7f60fc0b3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:19.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.501+0000 7f6103e00640 1 -- 172.21.15.90:0/2222001533 shutdown_connections 2024-03-24T01:26:19.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.501+0000 7f6103e00640 1 -- 172.21.15.90:0/2222001533 wait complete. 2024-03-24T01:26:19.518 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T01:26:19.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.721+0000 7f2252400640 1 Processor -- start 2024-03-24T01:26:19.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.721+0000 7f2252400640 1 -- start start 2024-03-24T01:26:19.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.722+0000 7f2252400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f224c15dd90 0x7f224c0b6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:19.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.723+0000 7f2252400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f224c0b2720 0x7f224c0d33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:19.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.723+0000 7f2252400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f224c0b9460 con 0x7f224c06c6c0 2024-03-24T01:26:19.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.723+0000 7f2252400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f224c0b9a20 con 0x7f224c15dd90 2024-03-24T01:26:19.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.723+0000 7f2252400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f224c0b9740 con 0x7f224c0b2720 2024-03-24T01:26:19.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.723+0000 7f2251000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f224c15dd90 0x7f224c0b6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:19.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.723+0000 7f2251000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f224c15dd90 0x7f224c0b6af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40304/0 (socket says 172.21.15.90:40304) 2024-03-24T01:26:19.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.723+0000 7f2251000640 1 -- 172.21.15.90:0/555183921 learned_addr learned my addr 172.21.15.90:0/555183921 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:19.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.723+0000 7f2251000640 1 -- 172.21.15.90:0/555183921 >> v2:172.21.15.120:3300/0 conn(0x7f224c0b2720 msgr2=0x7f224c0d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:19.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.723+0000 7f2251000640 1 --2- 172.21.15.90:0/555183921 >> v2:172.21.15.120:3300/0 conn(0x7f224c0b2720 0x7f224c0d33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:19.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.723+0000 7f2251000640 1 -- 172.21.15.90:0/555183921 >> v1:172.21.15.90:6789/0 conn(0x7f224c06c6c0 legacy=0x7f224c06caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:19.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.723+0000 7f2251000640 1 -- 172.21.15.90:0/555183921 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f224c0d1c10 con 0x7f224c15dd90 2024-03-24T01:26:19.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.725+0000 7f2251000640 1 --2- 172.21.15.90:0/555183921 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f224c15dd90 0x7f224c0b6af0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f224c0b9ba0 tx=0x7f224002b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93f35da14d45187d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:19.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.725+0000 7f224b400640 1 -- 172.21.15.90:0/555183921 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2240034070 con 0x7f224c15dd90 2024-03-24T01:26:19.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.725+0000 7f224b400640 1 -- 172.21.15.90:0/555183921 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2240030850 con 0x7f224c15dd90 2024-03-24T01:26:19.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.725+0000 7f224b400640 1 -- 172.21.15.90:0/555183921 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2240030c20 con 0x7f224c15dd90 2024-03-24T01:26:19.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.725+0000 7f2252400640 1 -- 172.21.15.90:0/555183921 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f224c15dd90 msgr2=0x7f224c0b6af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:19.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.725+0000 7f2252400640 1 --2- 172.21.15.90:0/555183921 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f224c15dd90 0x7f224c0b6af0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f224c0b9ba0 tx=0x7f224002b990 comp rx=0 tx=0).stop 2024-03-24T01:26:19.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.726+0000 7f2252400640 1 -- 172.21.15.90:0/555183921 shutdown_connections 2024-03-24T01:26:19.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.726+0000 7f2252400640 1 --2- 172.21.15.90:0/555183921 >> v2:172.21.15.120:3300/0 conn(0x7f224c0b2720 0x7f224c0d33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:19.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.726+0000 7f2252400640 1 --2- 172.21.15.90:0/555183921 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f224c15dd90 0x7f224c0b6af0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:19.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.726+0000 7f2252400640 1 -- 172.21.15.90:0/555183921 >> 172.21.15.90:0/555183921 conn(0x7f224c072e20 msgr2=0x7f224c0d3a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:19.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.726+0000 7f2252400640 1 -- 172.21.15.90:0/555183921 shutdown_connections 2024-03-24T01:26:19.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.726+0000 7f2252400640 1 -- 172.21.15.90:0/555183921 wait complete. 2024-03-24T01:26:19.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.727+0000 7f2252400640 1 Processor -- start 2024-03-24T01:26:19.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.727+0000 7f2252400640 1 -- start start 2024-03-24T01:26:19.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.728+0000 7f2252400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f224c06c6c0 0x7f224c14a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:19.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.728+0000 7f2252400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f224c0b2720 0x7f224c0b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:19.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.728+0000 7f2252400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f224c15dd90 0x7f224c0b4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:19.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.728+0000 7f2252400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224c0d2590 con 0x7f224c15dd90 2024-03-24T01:26:19.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.728+0000 7f2252400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224c0d2410 con 0x7f224c06c6c0 2024-03-24T01:26:19.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.728+0000 7f2252400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f224c0d2710 con 0x7f224c0b2720 2024-03-24T01:26:19.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.729+0000 7f2251a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f224c15dd90 0x7f224c0b4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:19.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.729+0000 7f2251a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f224c15dd90 0x7f224c0b4470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40314/0 (socket says 172.21.15.90:40314) 2024-03-24T01:26:19.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.729+0000 7f2251a00640 1 -- 172.21.15.90:0/3611718929 learned_addr learned my addr 172.21.15.90:0/3611718929 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:19.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.729+0000 7f2251a00640 1 -- 172.21.15.90:0/3611718929 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f224c0b2720 msgr2=0x7f224c0b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:19.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.729+0000 7f2251a00640 1 --2- 172.21.15.90:0/3611718929 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f224c0b2720 0x7f224c0b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:19.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.729+0000 7f2251a00640 1 -- 172.21.15.90:0/3611718929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f224c06c6c0 msgr2=0x7f224c14a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:19.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.729+0000 7f2251a00640 1 --2- 172.21.15.90:0/3611718929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f224c06c6c0 0x7f224c14a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:19.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.729+0000 7f2251a00640 1 -- 172.21.15.90:0/3611718929 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f224c0b61d0 con 0x7f224c15dd90 2024-03-24T01:26:19.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.729+0000 7f2251a00640 1 --2- 172.21.15.90:0/3611718929 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f224c15dd90 0x7f224c0b4470 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f224400c7a0 tx=0x7f224400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:19.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.729+0000 7f2248c00640 1 -- 172.21.15.90:0/3611718929 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2244003b70 con 0x7f224c15dd90 2024-03-24T01:26:19.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.730+0000 7f2248c00640 1 -- 172.21.15.90:0/3611718929 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2244003d30 con 0x7f224c15dd90 2024-03-24T01:26:19.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.730+0000 7f2248c00640 1 -- 172.21.15.90:0/3611718929 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224400aa90 con 0x7f224c15dd90 2024-03-24T01:26:19.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.730+0000 7f2252400640 1 -- 172.21.15.90:0/3611718929 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f224c0b4b20 con 0x7f224c15dd90 2024-03-24T01:26:19.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.730+0000 7f2252400640 1 -- 172.21.15.90:0/3611718929 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f224c0b50e0 con 0x7f224c15dd90 2024-03-24T01:26:19.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.731+0000 7f2252400640 1 -- 172.21.15.90:0/3611718929 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f224c04f400 con 0x7f224c15dd90 2024-03-24T01:26:19.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.732+0000 7f2248c00640 1 -- 172.21.15.90:0/3611718929 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f224400ac30 con 0x7f224c15dd90 2024-03-24T01:26:19.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.733+0000 7f2248c00640 1 --2- 172.21.15.90:0/3611718929 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f222c06bbd0 0x7f222c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:19.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.733+0000 7f2248c00640 1 -- 172.21.15.90:0/3611718929 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f224408ab40 con 0x7f224c15dd90 2024-03-24T01:26:19.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.733+0000 7f2251000640 1 --2- 172.21.15.90:0/3611718929 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f222c06bbd0 0x7f222c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:19.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.735+0000 7f2251000640 1 --2- 172.21.15.90:0/3611718929 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f222c06bbd0 0x7f222c06e090 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2240036000 tx=0x7f2240030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:19.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.742+0000 7f2248c00640 1 -- 172.21.15.90:0/3611718929 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f224c0b61d0 con 0x7f224c15dd90 2024-03-24T01:26:19.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.937+0000 7f2252400640 1 -- 172.21.15.90:0/3611718929 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f224c06caa0 con 0x7f224c15dd90 2024-03-24T01:26:19.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.938+0000 7f2248c00640 1 -- 172.21.15.90:0/3611718929 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v39) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2244059490 con 0x7f224c15dd90 2024-03-24T01:26:19.939 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:26:19.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.940+0000 7f2252400640 1 -- 172.21.15.90:0/3611718929 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f222c06bbd0 msgr2=0x7f222c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:19.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.940+0000 7f2252400640 1 --2- 172.21.15.90:0/3611718929 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f222c06bbd0 0x7f222c06e090 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2240036000 tx=0x7f2240030000 comp rx=0 tx=0).stop 2024-03-24T01:26:19.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.940+0000 7f2252400640 1 -- 172.21.15.90:0/3611718929 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f224c15dd90 msgr2=0x7f224c0b4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:19.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.940+0000 7f2252400640 1 --2- 172.21.15.90:0/3611718929 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f224c15dd90 0x7f224c0b4470 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f224400c7a0 tx=0x7f224400cc60 comp rx=0 tx=0).stop 2024-03-24T01:26:19.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.940+0000 7f2252400640 1 -- 172.21.15.90:0/3611718929 shutdown_connections 2024-03-24T01:26:19.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.940+0000 7f2252400640 1 --2- 172.21.15.90:0/3611718929 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f222c06bbd0 0x7f222c06e090 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:19.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.940+0000 7f2252400640 1 --2- 172.21.15.90:0/3611718929 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f224c15dd90 0x7f224c0b4470 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:19.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.940+0000 7f2252400640 1 --2- 172.21.15.90:0/3611718929 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f224c0b2720 0x7f224c0b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:19.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.940+0000 7f2252400640 1 --2- 172.21.15.90:0/3611718929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f224c06c6c0 0x7f224c14a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:19.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.940+0000 7f2252400640 1 -- 172.21.15.90:0/3611718929 >> 172.21.15.90:0/3611718929 conn(0x7f224c072e20 msgr2=0x7f224c0d3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:19.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.941+0000 7f2252400640 1 -- 172.21.15.90:0/3611718929 shutdown_connections 2024-03-24T01:26:19.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:19.941+0000 7f2252400640 1 -- 172.21.15.90:0/3611718929 wait complete. 2024-03-24T01:26:19.959 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T01:26:20.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.163+0000 7fe3a9600640 1 Processor -- start 2024-03-24T01:26:20.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.163+0000 7fe3a9600640 1 -- start start 2024-03-24T01:26:20.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.165+0000 7fe3a9600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe3a40b97b0 0x7fe3a40cead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:20.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.165+0000 7fe3a9600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3a40d3830 0x7fe3a40cf010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:20.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.165+0000 7fe3a9600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a40d2790 con 0x7fe3a415de60 2024-03-24T01:26:20.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.165+0000 7fe3a9600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3a40d2550 con 0x7fe3a40b97b0 2024-03-24T01:26:20.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.165+0000 7fe3a9600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a40b7eb0 con 0x7fe3a40d3830 2024-03-24T01:26:20.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.165+0000 7fe39a000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe3a40b97b0 0x7fe3a40cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:20.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.165+0000 7fe39a000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe3a40b97b0 0x7fe3a40cead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.90:59922/0 (socket says 172.21.15.90:59922) 2024-03-24T01:26:20.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.165+0000 7fe39a000640 1 -- 172.21.15.90:0/506060051 learned_addr learned my addr 172.21.15.90:0/506060051 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:20.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.165+0000 7fe3a2a00640 1 --1- 172.21.15.90:0/506060051 >> v1:172.21.15.120:6789/0 conn(0x7fe3a415de60 0x7fe3a40d1e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.90:35776/0 (socket says 172.21.15.90:35776) 2024-03-24T01:26:20.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.165+0000 7fe3a3400640 1 --2- 172.21.15.90:0/506060051 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3a40d3830 0x7fe3a40cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:20.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.166+0000 7fe39a000640 1 -- 172.21.15.90:0/506060051 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3a40d3830 msgr2=0x7fe3a40cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:20.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.166+0000 7fe39a000640 1 --2- 172.21.15.90:0/506060051 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3a40d3830 0x7fe3a40cf010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:20.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.166+0000 7fe39a000640 1 -- 172.21.15.90:0/506060051 >> v1:172.21.15.120:6789/0 conn(0x7fe3a415de60 legacy=0x7fe3a40d1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:20.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.166+0000 7fe39a000640 1 -- 172.21.15.90:0/506060051 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a40d2a70 con 0x7fe3a40b97b0 2024-03-24T01:26:20.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.167+0000 7fe39a000640 1 --2- 172.21.15.90:0/506060051 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a40b97b0 0x7fe3a40cead0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe38c009870 tx=0x7fe38c02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37c235a07f98b3e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:20.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.168+0000 7fe3a2000640 1 -- 172.21.15.90:0/506060051 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38c035020 con 0x7fe3a40b97b0 2024-03-24T01:26:20.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.168+0000 7fe3a2000640 1 -- 172.21.15.90:0/506060051 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe38c031640 con 0x7fe3a40b97b0 2024-03-24T01:26:20.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.168+0000 7fe3a2000640 1 -- 172.21.15.90:0/506060051 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38c02d040 con 0x7fe3a40b97b0 2024-03-24T01:26:20.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.168+0000 7fe3a9600640 1 -- 172.21.15.90:0/506060051 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a40b97b0 msgr2=0x7fe3a40cead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:20.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.169+0000 7fe3a9600640 1 --2- 172.21.15.90:0/506060051 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a40b97b0 0x7fe3a40cead0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe38c009870 tx=0x7fe38c02b8f0 comp rx=0 tx=0).stop 2024-03-24T01:26:20.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.169+0000 7fe3a9600640 1 -- 172.21.15.90:0/506060051 shutdown_connections 2024-03-24T01:26:20.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.169+0000 7fe3a9600640 1 --2- 172.21.15.90:0/506060051 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3a40d3830 0x7fe3a40cf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:20.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.169+0000 7fe3a9600640 1 --2- 172.21.15.90:0/506060051 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a40b97b0 0x7fe3a40cead0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:20.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.169+0000 7fe3a9600640 1 -- 172.21.15.90:0/506060051 >> 172.21.15.90:0/506060051 conn(0x7fe3a4072c10 msgr2=0x7fe3a4073020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:20.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.169+0000 7fe3a9600640 1 -- 172.21.15.90:0/506060051 shutdown_connections 2024-03-24T01:26:20.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.169+0000 7fe3a9600640 1 -- 172.21.15.90:0/506060051 wait complete. 2024-03-24T01:26:20.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.170+0000 7fe3a9600640 1 Processor -- start 2024-03-24T01:26:20.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.171+0000 7fe3a9600640 1 -- start start 2024-03-24T01:26:20.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.172+0000 7fe3a9600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3a40d3830 0x7fe3a4152080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:20.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.172+0000 7fe3a9600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3a415de60 0x7fe3a4150580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:20.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.172+0000 7fe3a9600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a4157990 0x7fe3a415f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:20.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.172+0000 7fe3a9600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a40d3310 con 0x7fe3a40d3830 2024-03-24T01:26:20.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.172+0000 7fe3a9600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a40d3030 con 0x7fe3a4157990 2024-03-24T01:26:20.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.172+0000 7fe3a9600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a415e240 con 0x7fe3a415de60 2024-03-24T01:26:20.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.172+0000 7fe3a2a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3a40d3830 0x7fe3a4152080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:20.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.172+0000 7fe3a2a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3a40d3830 0x7fe3a4152080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49768/0 (socket says 172.21.15.90:49768) 2024-03-24T01:26:20.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.172+0000 7fe3a2a00640 1 -- 172.21.15.90:0/2728156767 learned_addr learned my addr 172.21.15.90:0/2728156767 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:20.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.172+0000 7fe39a000640 1 --2- 172.21.15.90:0/2728156767 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3a415de60 0x7fe3a4150580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:20.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.172+0000 7fe3a3400640 1 --2- 172.21.15.90:0/2728156767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a4157990 0x7fe3a415f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:20.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.172+0000 7fe3a2a00640 1 -- 172.21.15.90:0/2728156767 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3a415de60 msgr2=0x7fe3a4150580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:20.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.172+0000 7fe3a2a00640 1 --2- 172.21.15.90:0/2728156767 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3a415de60 0x7fe3a4150580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:20.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.172+0000 7fe3a2a00640 1 -- 172.21.15.90:0/2728156767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a4157990 msgr2=0x7fe3a415f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:20.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.172+0000 7fe3a2a00640 1 --2- 172.21.15.90:0/2728156767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a4157990 0x7fe3a415f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:20.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.172+0000 7fe3a2a00640 1 -- 172.21.15.90:0/2728156767 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a415fe50 con 0x7fe3a40d3830 2024-03-24T01:26:20.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.173+0000 7fe3a2a00640 1 --2- 172.21.15.90:0/2728156767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3a40d3830 0x7fe3a4152080 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fe39400c3f0 tx=0x7fe39400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:20.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.173+0000 7fe39b400640 1 -- 172.21.15.90:0/2728156767 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3940074e0 con 0x7fe3a40d3830 2024-03-24T01:26:20.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.173+0000 7fe39b400640 1 -- 172.21.15.90:0/2728156767 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe394007680 con 0x7fe3a40d3830 2024-03-24T01:26:20.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.173+0000 7fe3a9600640 1 -- 172.21.15.90:0/2728156767 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3a4160130 con 0x7fe3a40d3830 2024-03-24T01:26:20.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.173+0000 7fe39b400640 1 -- 172.21.15.90:0/2728156767 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe394004a00 con 0x7fe3a40d3830 2024-03-24T01:26:20.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.173+0000 7fe3a9600640 1 -- 172.21.15.90:0/2728156767 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3a4155bb0 con 0x7fe3a40d3830 2024-03-24T01:26:20.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.174+0000 7fe3a9600640 1 -- 172.21.15.90:0/2728156767 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3a404f400 con 0x7fe3a40d3830 2024-03-24T01:26:20.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.175+0000 7fe39b400640 1 -- 172.21.15.90:0/2728156767 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe39401a020 con 0x7fe3a40d3830 2024-03-24T01:26:20.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.177+0000 7fe39b400640 1 --2- 172.21.15.90:0/2728156767 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe38006bbd0 0x7fe38006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:20.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.177+0000 7fe39b400640 1 -- 172.21.15.90:0/2728156767 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe39408a440 con 0x7fe3a40d3830 2024-03-24T01:26:20.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.177+0000 7fe39a000640 1 --2- 172.21.15.90:0/2728156767 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe38006bbd0 0x7fe38006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:20.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.179+0000 7fe39a000640 1 --2- 172.21.15.90:0/2728156767 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe38006bbd0 0x7fe38006e090 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe38c036000 tx=0x7fe38c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:20.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.181+0000 7fe39b400640 1 -- 172.21.15.90:0/2728156767 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe394010300 con 0x7fe3a40d3830 2024-03-24T01:26:20.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.382+0000 7fe3a9600640 1 -- 172.21.15.90:0/2728156767 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe3a40b79f0 con 0x7fe3a40d3830 2024-03-24T01:26:20.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.383+0000 7fe39b400640 1 -- 172.21.15.90:0/2728156767 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v39) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe394058d00 con 0x7fe3a40d3830 2024-03-24T01:26:20.384 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-24T01:26:20.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.385+0000 7fe3a9600640 1 -- 172.21.15.90:0/2728156767 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe38006bbd0 msgr2=0x7fe38006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:20.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.385+0000 7fe3a9600640 1 --2- 172.21.15.90:0/2728156767 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe38006bbd0 0x7fe38006e090 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe38c036000 tx=0x7fe38c03c040 comp rx=0 tx=0).stop 2024-03-24T01:26:20.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.385+0000 7fe3a9600640 1 -- 172.21.15.90:0/2728156767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3a40d3830 msgr2=0x7fe3a4152080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:20.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.385+0000 7fe3a9600640 1 --2- 172.21.15.90:0/2728156767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3a40d3830 0x7fe3a4152080 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fe39400c3f0 tx=0x7fe39400c8b0 comp rx=0 tx=0).stop 2024-03-24T01:26:20.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.385+0000 7fe3a9600640 1 -- 172.21.15.90:0/2728156767 shutdown_connections 2024-03-24T01:26:20.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.385+0000 7fe3a9600640 1 --2- 172.21.15.90:0/2728156767 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe38006bbd0 0x7fe38006e090 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:20.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.385+0000 7fe3a9600640 1 --2- 172.21.15.90:0/2728156767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a4157990 0x7fe3a415f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:20.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.385+0000 7fe3a9600640 1 --2- 172.21.15.90:0/2728156767 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3a415de60 0x7fe3a4150580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:20.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.385+0000 7fe3a9600640 1 --2- 172.21.15.90:0/2728156767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3a40d3830 0x7fe3a4152080 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:20.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.385+0000 7fe3a9600640 1 -- 172.21.15.90:0/2728156767 >> 172.21.15.90:0/2728156767 conn(0x7fe3a4072c10 msgr2=0x7fe3a40b6e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:20.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.385+0000 7fe3a9600640 1 -- 172.21.15.90:0/2728156767 shutdown_connections 2024-03-24T01:26:20.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.385+0000 7fe3a9600640 1 -- 172.21.15.90:0/2728156767 wait complete. 2024-03-24T01:26:20.406 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T01:26:20.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.609+0000 7f4e75a00640 1 Processor -- start 2024-03-24T01:26:20.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.609+0000 7f4e75a00640 1 -- start start 2024-03-24T01:26:20.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.612+0000 7f4e75a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e7015e260 0x7f4e700acfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:20.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.612+0000 7f4e75a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4e7006b0c0 0x7f4e7006b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:20.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.612+0000 7f4e75a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e701620a0 con 0x7f4e70158430 2024-03-24T01:26:20.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.612+0000 7f4e75a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e7006ba50 con 0x7f4e7006b0c0 2024-03-24T01:26:20.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.612+0000 7f4e75a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e70162380 con 0x7f4e7015e260 2024-03-24T01:26:20.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.612+0000 7f4e6fe00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4e70158430 0x7f4e70161850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53268/0 (socket says 172.21.15.90:53268) 2024-03-24T01:26:20.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.612+0000 7f4e6fe00640 1 -- 172.21.15.90:0/506168305 learned_addr learned my addr 172.21.15.90:0/506168305 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:26:20.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.612+0000 7f4e75000640 1 --2- 172.21.15.90:0/506168305 >> v2:172.21.15.90:3300/0 conn(0x7f4e7006b0c0 0x7f4e7006b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:20.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.613+0000 7f4e67e00640 1 --2- 172.21.15.90:0/506168305 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e7015e260 0x7f4e700acfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:20.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.613+0000 7f4e75000640 1 -- 172.21.15.90:0/506168305 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e7015e260 msgr2=0x7f4e700acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:20.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.613+0000 7f4e75000640 1 --2- 172.21.15.90:0/506168305 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e7015e260 0x7f4e700acfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:20.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.613+0000 7f4e75000640 1 -- 172.21.15.90:0/506168305 >> v1:172.21.15.90:6789/0 conn(0x7f4e70158430 legacy=0x7f4e70161850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:20.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.613+0000 7f4e75000640 1 -- 172.21.15.90:0/506168305 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e70162660 con 0x7f4e7006b0c0 2024-03-24T01:26:20.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.614+0000 7f4e75000640 1 --2- 172.21.15.90:0/506168305 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e7006b0c0 0x7f4e7006b510 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f4e68009d90 tx=0x7f4e6802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b1d0c35d3a5d922 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:20.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.614+0000 7f4e6f400640 1 -- 172.21.15.90:0/506168305 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e68039070 con 0x7f4e7006b0c0 2024-03-24T01:26:20.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.614+0000 7f4e6f400640 1 -- 172.21.15.90:0/506168305 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e6802cbd0 con 0x7f4e7006b0c0 2024-03-24T01:26:20.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.614+0000 7f4e6f400640 1 -- 172.21.15.90:0/506168305 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e68034440 con 0x7f4e7006b0c0 2024-03-24T01:26:20.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.615+0000 7f4e75a00640 1 -- 172.21.15.90:0/506168305 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e7006b0c0 msgr2=0x7f4e7006b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:20.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.615+0000 7f4e75a00640 1 --2- 172.21.15.90:0/506168305 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e7006b0c0 0x7f4e7006b510 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f4e68009d90 tx=0x7f4e6802be20 comp rx=0 tx=0).stop 2024-03-24T01:26:20.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.615+0000 7f4e75a00640 1 -- 172.21.15.90:0/506168305 shutdown_connections 2024-03-24T01:26:20.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.615+0000 7f4e75a00640 1 --2- 172.21.15.90:0/506168305 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e7006b0c0 0x7f4e7006b510 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:20.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.615+0000 7f4e75a00640 1 --2- 172.21.15.90:0/506168305 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e7015e260 0x7f4e700acfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:20.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.615+0000 7f4e75a00640 1 -- 172.21.15.90:0/506168305 >> 172.21.15.90:0/506168305 conn(0x7f4e700cff50 msgr2=0x7f4e700d0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:20.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.615+0000 7f4e75a00640 1 -- 172.21.15.90:0/506168305 shutdown_connections 2024-03-24T01:26:20.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.615+0000 7f4e75a00640 1 -- 172.21.15.90:0/506168305 wait complete. 2024-03-24T01:26:20.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.619+0000 7f4e75a00640 1 Processor -- start 2024-03-24T01:26:20.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.619+0000 7f4e75a00640 1 -- start start 2024-03-24T01:26:20.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.621+0000 7f4e75a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e7006b0c0 0x7f4e7014c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:20.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.621+0000 7f4e75a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e70158430 0x7f4e700b3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:20.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.621+0000 7f4e75a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e7015e260 0x7f4e70155200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:20.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.621+0000 7f4e75a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e700ad6a0 con 0x7f4e7006b0c0 2024-03-24T01:26:20.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.621+0000 7f4e75a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e700ad520 con 0x7f4e70158430 2024-03-24T01:26:20.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.621+0000 7f4e75a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e70072920 con 0x7f4e7015e260 2024-03-24T01:26:20.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.621+0000 7f4e6fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e7006b0c0 0x7f4e7014c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:20.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.621+0000 7f4e6fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e7006b0c0 0x7f4e7014c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49772/0 (socket says 172.21.15.90:49772) 2024-03-24T01:26:20.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.621+0000 7f4e6fe00640 1 -- 172.21.15.90:0/3496931510 learned_addr learned my addr 172.21.15.90:0/3496931510 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:20.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.622+0000 7f4e67e00640 1 --2- 172.21.15.90:0/3496931510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e70158430 0x7f4e700b3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:20.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.622+0000 7f4e75000640 1 --2- 172.21.15.90:0/3496931510 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e7015e260 0x7f4e70155200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:20.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.622+0000 7f4e6fe00640 1 -- 172.21.15.90:0/3496931510 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e7015e260 msgr2=0x7f4e70155200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:20.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.622+0000 7f4e6fe00640 1 --2- 172.21.15.90:0/3496931510 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e7015e260 0x7f4e70155200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:20.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.622+0000 7f4e6fe00640 1 -- 172.21.15.90:0/3496931510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e70158430 msgr2=0x7f4e700b3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:20.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.622+0000 7f4e6fe00640 1 --2- 172.21.15.90:0/3496931510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e70158430 0x7f4e700b3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:20.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.622+0000 7f4e6fe00640 1 -- 172.21.15.90:0/3496931510 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e700cde50 con 0x7f4e7006b0c0 2024-03-24T01:26:20.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.622+0000 7f4e67e00640 1 --2- 172.21.15.90:0/3496931510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e70158430 0x7f4e700b3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:20.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.622+0000 7f4e75000640 1 --2- 172.21.15.90:0/3496931510 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e7015e260 0x7f4e70155200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:20.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.622+0000 7f4e6fe00640 1 --2- 172.21.15.90:0/3496931510 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e7006b0c0 0x7f4e7014c370 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f4e6000c400 tx=0x7f4e6000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:20.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.622+0000 7f4e6cc00640 1 -- 172.21.15.90:0/3496931510 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e60017070 con 0x7f4e7006b0c0 2024-03-24T01:26:20.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.622+0000 7f4e6cc00640 1 -- 172.21.15.90:0/3496931510 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e60005ce0 con 0x7f4e7006b0c0 2024-03-24T01:26:20.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.622+0000 7f4e75a00640 1 -- 172.21.15.90:0/3496931510 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e70154bb0 con 0x7f4e7006b0c0 2024-03-24T01:26:20.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.622+0000 7f4e6cc00640 1 -- 172.21.15.90:0/3496931510 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e600072f0 con 0x7f4e7006b0c0 2024-03-24T01:26:20.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.622+0000 7f4e75a00640 1 -- 172.21.15.90:0/3496931510 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e700ce110 con 0x7f4e7006b0c0 2024-03-24T01:26:20.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.624+0000 7f4e75a00640 1 -- 172.21.15.90:0/3496931510 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e7004f400 con 0x7f4e7006b0c0 2024-03-24T01:26:20.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.625+0000 7f4e6cc00640 1 -- 172.21.15.90:0/3496931510 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4e6001a070 con 0x7f4e7006b0c0 2024-03-24T01:26:20.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.625+0000 7f4e6cc00640 1 --2- 172.21.15.90:0/3496931510 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4e4c06bbd0 0x7f4e4c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:20.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.626+0000 7f4e6cc00640 1 -- 172.21.15.90:0/3496931510 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4e6008a0a0 con 0x7f4e7006b0c0 2024-03-24T01:26:20.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.626+0000 7f4e67e00640 1 --2- 172.21.15.90:0/3496931510 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4e4c06bbd0 0x7f4e4c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:20.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.627+0000 7f4e67e00640 1 --2- 172.21.15.90:0/3496931510 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4e4c06bbd0 0x7f4e4c06e090 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4e58006fd0 tx=0x7f4e58008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:20.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.630+0000 7f4e6cc00640 1 -- 172.21.15.90:0/3496931510 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4e60058960 con 0x7f4e7006b0c0 2024-03-24T01:26:20.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.840+0000 7f4e75a00640 1 -- 172.21.15.90:0/3496931510 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4e7014cf60 con 0x7f4e7006b0c0 2024-03-24T01:26:20.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.841+0000 7f4e6cc00640 1 -- 172.21.15.90:0/3496931510 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v39) v1 ==== 80+0+0 (secure 0 0 0) 0x7f4e60058260 con 0x7f4e7006b0c0 2024-03-24T01:26:20.843 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-24T01:26:20.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.844+0000 7f4e75a00640 1 -- 172.21.15.90:0/3496931510 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4e4c06bbd0 msgr2=0x7f4e4c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:20.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.844+0000 7f4e75a00640 1 --2- 172.21.15.90:0/3496931510 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4e4c06bbd0 0x7f4e4c06e090 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4e58006fd0 tx=0x7f4e58008040 comp rx=0 tx=0).stop 2024-03-24T01:26:20.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.844+0000 7f4e75a00640 1 -- 172.21.15.90:0/3496931510 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e7006b0c0 msgr2=0x7f4e7014c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:20.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.844+0000 7f4e75a00640 1 --2- 172.21.15.90:0/3496931510 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e7006b0c0 0x7f4e7014c370 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f4e6000c400 tx=0x7f4e6000c8c0 comp rx=0 tx=0).stop 2024-03-24T01:26:20.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.844+0000 7f4e75a00640 1 -- 172.21.15.90:0/3496931510 shutdown_connections 2024-03-24T01:26:20.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.844+0000 7f4e75a00640 1 --2- 172.21.15.90:0/3496931510 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4e4c06bbd0 0x7f4e4c06e090 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:20.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.844+0000 7f4e75a00640 1 --2- 172.21.15.90:0/3496931510 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e7015e260 0x7f4e70155200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:20.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.844+0000 7f4e75a00640 1 --2- 172.21.15.90:0/3496931510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e70158430 0x7f4e700b3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:20.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.844+0000 7f4e75a00640 1 --2- 172.21.15.90:0/3496931510 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e7006b0c0 0x7f4e7014c370 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:20.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.844+0000 7f4e75a00640 1 -- 172.21.15.90:0/3496931510 >> 172.21.15.90:0/3496931510 conn(0x7f4e700cff50 msgr2=0x7f4e70072e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:20.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.844+0000 7f4e75a00640 1 -- 172.21.15.90:0/3496931510 shutdown_connections 2024-03-24T01:26:20.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:20.844+0000 7f4e75a00640 1 -- 172.21.15.90:0/3496931510 wait complete. 2024-03-24T01:26:20.864 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi090.front.sepia.ceph.com... 2024-03-24T01:26:20.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:20.865 DEBUG:teuthology.orchestra.run.smithi090:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-24T01:26:20.946 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[53657]: fuse finished with error 0 and tester_r 0 2024-03-24T01:26:20.952 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:26:26.859 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:26:26.867 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:26:27.053 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-03-24T01:26:27.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:27.054 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T01:26:27.111 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T01:26:27.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.319+0000 7f20ba800640 1 Processor -- start 2024-03-24T01:26:27.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.319+0000 7f20ba800640 1 -- start start 2024-03-24T01:26:27.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.320+0000 7f20ba800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20b40b3ed0 0x7f20b40b4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:27.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.320+0000 7f20ba800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f20b40cd140 0x7f20b40cd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:27.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.320+0000 7f20ba800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20b406db30 con 0x7f20b4071990 2024-03-24T01:26:27.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.320+0000 7f20ba800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20b406ebd0 con 0x7f20b40cd140 2024-03-24T01:26:27.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.320+0000 7f20ba800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20b406de10 con 0x7f20b40b3ed0 2024-03-24T01:26:27.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.321+0000 7f20b3e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f20b40cd140 0x7f20b40cd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:27.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.321+0000 7f20abe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20b40b3ed0 0x7f20b40b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:27.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.321+0000 7f20abe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20b40b3ed0 0x7f20b40b4330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.90:49690/0 (socket says 172.21.15.90:49690) 2024-03-24T01:26:27.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.321+0000 7f20abe00640 1 -- 172.21.15.90:0/2475923999 learned_addr learned my addr 172.21.15.90:0/2475923999 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:27.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.322+0000 7f20b3e00640 1 -- 172.21.15.90:0/2475923999 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20b40b3ed0 msgr2=0x7f20b40b4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:27.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.322+0000 7f20b3e00640 1 --2- 172.21.15.90:0/2475923999 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20b40b3ed0 0x7f20b40b4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.322+0000 7f20b3e00640 1 -- 172.21.15.90:0/2475923999 >> v1:172.21.15.120:6789/0 conn(0x7f20b4071990 legacy=0x7f20b4071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:27.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.322+0000 7f20b3e00640 1 -- 172.21.15.90:0/2475923999 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b406e0f0 con 0x7f20b40cd140 2024-03-24T01:26:27.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.322+0000 7f20abe00640 1 --2- 172.21.15.90:0/2475923999 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20b40b3ed0 0x7f20b40b4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T01:26:27.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.324+0000 7f20b3e00640 1 --2- 172.21.15.90:0/2475923999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20b40cd140 0x7f20b40cd5d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f20ac009d90 tx=0x7f20ac02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d36173abf85fdd08 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:27.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.324+0000 7f20b2a00640 1 -- 172.21.15.90:0/2475923999 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ac039070 con 0x7f20b40cd140 2024-03-24T01:26:27.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.324+0000 7f20b2a00640 1 -- 172.21.15.90:0/2475923999 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20ac02cc60 con 0x7f20b40cd140 2024-03-24T01:26:27.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.324+0000 7f20b2a00640 1 -- 172.21.15.90:0/2475923999 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ac034660 con 0x7f20b40cd140 2024-03-24T01:26:27.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.325+0000 7f20ba800640 1 -- 172.21.15.90:0/2475923999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20b40cd140 msgr2=0x7f20b40cd5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:27.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.325+0000 7f20ba800640 1 --2- 172.21.15.90:0/2475923999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20b40cd140 0x7f20b40cd5d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f20ac009d90 tx=0x7f20ac02beb0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.325+0000 7f20ba800640 1 -- 172.21.15.90:0/2475923999 shutdown_connections 2024-03-24T01:26:27.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.325+0000 7f20ba800640 1 --2- 172.21.15.90:0/2475923999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20b40cd140 0x7f20b40cd5d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.325+0000 7f20ba800640 1 --2- 172.21.15.90:0/2475923999 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20b40b3ed0 0x7f20b40b4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.325+0000 7f20ba800640 1 -- 172.21.15.90:0/2475923999 >> 172.21.15.90:0/2475923999 conn(0x7f20b40b1770 msgr2=0x7f20b40b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:27.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.325+0000 7f20ba800640 1 -- 172.21.15.90:0/2475923999 shutdown_connections 2024-03-24T01:26:27.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.325+0000 7f20ba800640 1 -- 172.21.15.90:0/2475923999 wait complete. 2024-03-24T01:26:27.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.328+0000 7f20ba800640 1 Processor -- start 2024-03-24T01:26:27.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.328+0000 7f20ba800640 1 -- start start 2024-03-24T01:26:27.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.328+0000 7f20ba800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20b4071990 0x7f20b414d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:27.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.328+0000 7f20ba800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20b40b3ed0 0x7f20b414ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:27.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.328+0000 7f20ba800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20b40cd140 0x7f20b414f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:27.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.328+0000 7f20ba800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b40b4870 con 0x7f20b4071990 2024-03-24T01:26:27.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.328+0000 7f20ba800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b406e6b0 con 0x7f20b40b3ed0 2024-03-24T01:26:27.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.328+0000 7f20ba800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20b406b0c0 con 0x7f20b40cd140 2024-03-24T01:26:27.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.328+0000 7f20b3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20b4071990 0x7f20b414d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:27.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.328+0000 7f20b3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20b4071990 0x7f20b414d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49774/0 (socket says 172.21.15.90:49774) 2024-03-24T01:26:27.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.328+0000 7f20b3400640 1 -- 172.21.15.90:0/2088189140 learned_addr learned my addr 172.21.15.90:0/2088189140 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:27.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.328+0000 7f20b3e00640 1 --2- 172.21.15.90:0/2088189140 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20b40cd140 0x7f20b414f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:27.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.328+0000 7f20abe00640 1 --2- 172.21.15.90:0/2088189140 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20b40b3ed0 0x7f20b414ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:27.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.328+0000 7f20b3400640 1 -- 172.21.15.90:0/2088189140 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20b40cd140 msgr2=0x7f20b414f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:27.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.328+0000 7f20b3400640 1 --2- 172.21.15.90:0/2088189140 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20b40cd140 0x7f20b414f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.328+0000 7f20b3400640 1 -- 172.21.15.90:0/2088189140 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20b40b3ed0 msgr2=0x7f20b414ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:27.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.328+0000 7f20b3400640 1 --2- 172.21.15.90:0/2088189140 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20b40b3ed0 0x7f20b414ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.328+0000 7f20b3400640 1 -- 172.21.15.90:0/2088189140 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b40b0140 con 0x7f20b4071990 2024-03-24T01:26:27.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.328+0000 7f20b3e00640 1 --2- 172.21.15.90:0/2088189140 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20b40cd140 0x7f20b414f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:27.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.328+0000 7f20abe00640 1 --2- 172.21.15.90:0/2088189140 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20b40b3ed0 0x7f20b414ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:27.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.329+0000 7f20b3400640 1 --2- 172.21.15.90:0/2088189140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20b4071990 0x7f20b414d1b0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f20a400c480 tx=0x7f20a400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:27.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.329+0000 7f20ab400640 1 -- 172.21.15.90:0/2088189140 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a40052d0 con 0x7f20b4071990 2024-03-24T01:26:27.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.329+0000 7f20ab400640 1 -- 172.21.15.90:0/2088189140 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20a4005470 con 0x7f20b4071990 2024-03-24T01:26:27.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.329+0000 7f20ba800640 1 -- 172.21.15.90:0/2088189140 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20b4157b50 con 0x7f20b4071990 2024-03-24T01:26:27.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.329+0000 7f20ab400640 1 -- 172.21.15.90:0/2088189140 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a4012870 con 0x7f20b4071990 2024-03-24T01:26:27.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.329+0000 7f20ba800640 1 -- 172.21.15.90:0/2088189140 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20b4158110 con 0x7f20b4071990 2024-03-24T01:26:27.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.330+0000 7f20ba800640 1 -- 172.21.15.90:0/2088189140 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20b406c2c0 con 0x7f20b4071990 2024-03-24T01:26:27.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.331+0000 7f20ab400640 1 -- 172.21.15.90:0/2088189140 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f20a4012a10 con 0x7f20b4071990 2024-03-24T01:26:27.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.333+0000 7f20ab400640 1 --2- 172.21.15.90:0/2088189140 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f209006bbd0 0x7f209006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:27.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.333+0000 7f20ab400640 1 -- 172.21.15.90:0/2088189140 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f20a408a970 con 0x7f20b4071990 2024-03-24T01:26:27.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.333+0000 7f20abe00640 1 --2- 172.21.15.90:0/2088189140 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f209006bbd0 0x7f209006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:27.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.335+0000 7f20abe00640 1 --2- 172.21.15.90:0/2088189140 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f209006bbd0 0x7f209006e090 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f20b4154b50 tx=0x7f209c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:27.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.338+0000 7f20ab400640 1 -- 172.21.15.90:0/2088189140 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f20a4059240 con 0x7f20b4071990 2024-03-24T01:26:27.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.645+0000 7f20ba800640 1 -- 172.21.15.90:0/2088189140 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f20b404f400 con 0x7f20b4071990 2024-03-24T01:26:27.647 INFO:tasks.ceph.mon.a.smithi090.stderr:2024-03-24T01:26:27.646+0000 7fc6f2400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T01:26:27.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.650+0000 7f20ab400640 1 -- 172.21.15.90:0/2088189140 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v18) v1 ==== 161+0+0 (secure 0 0 0) 0x7f20a4058b40 con 0x7f20b4071990 2024-03-24T01:26:27.652 INFO:teuthology.orchestra.run.smithi090.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T01:26:27.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.653+0000 7f20ba800640 1 -- 172.21.15.90:0/2088189140 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f209006bbd0 msgr2=0x7f209006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:27.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.653+0000 7f20ba800640 1 --2- 172.21.15.90:0/2088189140 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f209006bbd0 0x7f209006e090 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f20b4154b50 tx=0x7f209c008040 comp rx=0 tx=0).stop 2024-03-24T01:26:27.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.653+0000 7f20ba800640 1 -- 172.21.15.90:0/2088189140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20b4071990 msgr2=0x7f20b414d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:27.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.653+0000 7f20ba800640 1 --2- 172.21.15.90:0/2088189140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20b4071990 0x7f20b414d1b0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f20a400c480 tx=0x7f20a400c940 comp rx=0 tx=0).stop 2024-03-24T01:26:27.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.653+0000 7f20ba800640 1 -- 172.21.15.90:0/2088189140 shutdown_connections 2024-03-24T01:26:27.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.653+0000 7f20ba800640 1 --2- 172.21.15.90:0/2088189140 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f209006bbd0 0x7f209006e090 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.653+0000 7f20ba800640 1 --2- 172.21.15.90:0/2088189140 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f20b40cd140 0x7f20b414f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.653+0000 7f20ba800640 1 --2- 172.21.15.90:0/2088189140 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20b40b3ed0 0x7f20b414ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.653+0000 7f20ba800640 1 --2- 172.21.15.90:0/2088189140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20b4071990 0x7f20b414d1b0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.653+0000 7f20ba800640 1 -- 172.21.15.90:0/2088189140 >> 172.21.15.90:0/2088189140 conn(0x7f20b40b1770 msgr2=0x7f20b40b0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:27.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.654+0000 7f20ba800640 1 -- 172.21.15.90:0/2088189140 shutdown_connections 2024-03-24T01:26:27.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.654+0000 7f20ba800640 1 -- 172.21.15.90:0/2088189140 wait complete. 2024-03-24T01:26:27.679 DEBUG:teuthology.orchestra.run.smithi090:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 10000000001.00000000 2024-03-24T01:26:27.758 INFO:tasks.ceph.mds.c.smithi090.stdout:starting mds.c at 2024-03-24T01:26:27.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.775+0000 7f117ec39f40 1 Processor -- start 2024-03-24T01:26:27.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.775+0000 7f117ec39f40 1 -- start start 2024-03-24T01:26:27.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.776+0000 7f117ec39f40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a5f362e170 0x55a5f36caaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:27.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.776+0000 7f117ec39f40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a5f36d0570 0x55a5f3604f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:27.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.776+0000 7f117ec39f40 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a5f36d0950 0x55a5f36d5920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:27.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.776+0000 7f117ec39f40 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5f3630270 con 0x55a5f36d0950 2024-03-24T01:26:27.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.776+0000 7f117ec39f40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5f35a7ff0 con 0x55a5f362e170 2024-03-24T01:26:27.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.776+0000 7f117ec39f40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a5f3630510 con 0x55a5f36d0570 2024-03-24T01:26:27.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.776+0000 7f117c000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a5f36d0950 0x55a5f36d5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:27.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.776+0000 7f117c000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a5f36d0950 0x55a5f36d5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49798/0 (socket says 172.21.15.90:49798) 2024-03-24T01:26:27.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.776+0000 7f117c000640 1 -- 172.21.15.90:0/2504664415 learned_addr learned my addr 172.21.15.90:0/2504664415 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:27.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.776+0000 7f117c000640 1 -- 172.21.15.90:0/2504664415 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a5f36d0570 msgr2=0x55a5f3604f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:27.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.776+0000 7f117c000640 1 --2- 172.21.15.90:0/2504664415 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a5f36d0570 0x55a5f3604f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.776+0000 7f117c000640 1 -- 172.21.15.90:0/2504664415 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a5f362e170 msgr2=0x55a5f36caaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:27.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.776+0000 7f117c000640 1 --2- 172.21.15.90:0/2504664415 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a5f362e170 0x55a5f36caaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.776+0000 7f117c000640 1 -- 172.21.15.90:0/2504664415 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a5f3543dc0 con 0x55a5f36d0950 2024-03-24T01:26:27.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.777+0000 7f117c000640 1 --2- 172.21.15.90:0/2504664415 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a5f36d0950 0x55a5f36d5920 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f1174047040 tx=0x7f117404f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb68f0ecbfa26d78 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:27.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.777+0000 7f1179800640 1 -- 172.21.15.90:0/2504664415 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117405a040 con 0x55a5f36d0950 2024-03-24T01:26:27.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.777+0000 7f1179800640 1 -- 172.21.15.90:0/2504664415 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1174064070 con 0x55a5f36d0950 2024-03-24T01:26:27.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.777+0000 7f1179800640 1 -- 172.21.15.90:0/2504664415 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117404dd80 con 0x55a5f36d0950 2024-03-24T01:26:27.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.777+0000 7f117ec39f40 1 -- 172.21.15.90:0/2504664415 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a5f36d0950 msgr2=0x55a5f36d5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:27.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.777+0000 7f117ec39f40 1 --2- 172.21.15.90:0/2504664415 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a5f36d0950 0x55a5f36d5920 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f1174047040 tx=0x7f117404f000 comp rx=0 tx=0).stop 2024-03-24T01:26:27.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.777+0000 7f117ec39f40 1 -- 172.21.15.90:0/2504664415 shutdown_connections 2024-03-24T01:26:27.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.777+0000 7f117ec39f40 1 --2- 172.21.15.90:0/2504664415 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a5f36d0950 0x55a5f36d5920 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.777+0000 7f117ec39f40 1 --2- 172.21.15.90:0/2504664415 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a5f36d0570 0x55a5f3604f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.777+0000 7f117ec39f40 1 --2- 172.21.15.90:0/2504664415 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a5f362e170 0x55a5f36caaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.777+0000 7f117ec39f40 1 -- 172.21.15.90:0/2504664415 >> 172.21.15.90:0/2504664415 conn(0x55a5f362c640 msgr2=0x55a5f3647c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:27.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.777+0000 7f117ec39f40 1 -- 172.21.15.90:0/2504664415 shutdown_connections 2024-03-24T01:26:27.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.777+0000 7f117ec39f40 1 -- 172.21.15.90:0/2504664415 wait complete. 2024-03-24T01:26:27.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.778+0000 7f117ec39f40 1 Processor -- start 2024-03-24T01:26:27.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.778+0000 7f117ec39f40 1 -- start start 2024-03-24T01:26:27.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.779+0000 7f117ec39f40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a5f362e170 0x55a5f36c81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:27.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.779+0000 7f117ec39f40 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a5f36d0570 0x55a5f3648a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:27.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.779+0000 7f117ec39f40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a5f36d0950 0x55a5f3648f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:27.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.779+0000 7f117ec39f40 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5f36cb570 con 0x55a5f36d0570 2024-03-24T01:26:27.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.779+0000 7f117ec39f40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5f36ccdd0 con 0x55a5f36d0950 2024-03-24T01:26:27.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.779+0000 7f117ec39f40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a5f36042c0 con 0x55a5f362e170 2024-03-24T01:26:27.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.779+0000 7f117ac00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a5f36d0570 0x55a5f3648a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:27.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.780+0000 7f117ac00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a5f36d0570 0x55a5f3648a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49810/0 (socket says 172.21.15.90:49810) 2024-03-24T01:26:27.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.780+0000 7f117ac00640 1 -- 172.21.15.90:0/1495603315 learned_addr learned my addr 172.21.15.90:0/1495603315 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:27.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.780+0000 7f117c000640 1 --2- 172.21.15.90:0/1495603315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a5f36d0950 0x55a5f3648f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:27.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.780+0000 7f117b600640 1 --2- 172.21.15.90:0/1495603315 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a5f362e170 0x55a5f36c81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:27.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.780+0000 7f117ac00640 1 -- 172.21.15.90:0/1495603315 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a5f362e170 msgr2=0x55a5f36c81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:27.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.780+0000 7f117ac00640 1 --2- 172.21.15.90:0/1495603315 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55a5f362e170 0x55a5f36c81a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.780+0000 7f117ac00640 1 -- 172.21.15.90:0/1495603315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a5f36d0950 msgr2=0x55a5f3648f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:27.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.780+0000 7f117ac00640 1 --2- 172.21.15.90:0/1495603315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a5f36d0950 0x55a5f3648f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.780+0000 7f117ac00640 1 -- 172.21.15.90:0/1495603315 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a5f36045a0 con 0x55a5f36d0570 2024-03-24T01:26:27.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.780+0000 7f117ac00640 1 --2- 172.21.15.90:0/1495603315 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a5f36d0570 0x55a5f3648a30 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f116800af00 tx=0x7f1168008690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:27.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.781+0000 7f1173e00640 1 -- 172.21.15.90:0/1495603315 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116800c830 con 0x55a5f36d0570 2024-03-24T01:26:27.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.781+0000 7f1173e00640 1 -- 172.21.15.90:0/1495603315 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f116800c9f0 con 0x55a5f36d0570 2024-03-24T01:26:27.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.781+0000 7f117ec39f40 1 -- 172.21.15.90:0/1495603315 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55a5f36c7120 con 0x55a5f36d0570 2024-03-24T01:26:27.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.781+0000 7f1173e00640 1 -- 172.21.15.90:0/1495603315 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116800cde0 con 0x55a5f36d0570 2024-03-24T01:26:27.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.781+0000 7f117ec39f40 1 -- 172.21.15.90:0/1495603315 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a5f3600cc0 con 0x55a5f36d0570 2024-03-24T01:26:27.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.783+0000 7f1173e00640 1 -- 172.21.15.90:0/1495603315 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1168009040 con 0x55a5f36d0570 2024-03-24T01:26:27.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.784+0000 7f1173e00640 1 --2- 172.21.15.90:0/1495603315 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f115c06a5b0 0x7f115c06ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:27.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.784+0000 7f1173e00640 1 -- 172.21.15.90:0/1495603315 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f1168089780 con 0x55a5f36d0570 2024-03-24T01:26:27.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.784+0000 7f117ec39f40 1 --2- 172.21.15.90:0/1495603315 >> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] conn(0x55a5f36c86e0 0x55a5f3602dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:27.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.784+0000 7f117ec39f40 1 -- 172.21.15.90:0/1495603315 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:1 5.7 5:e2e07930:::10000000001.00000000:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55a5f3604d70 con 0x55a5f36c86e0 2024-03-24T01:26:27.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.784+0000 7f117b600640 1 --2- 172.21.15.90:0/1495603315 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f115c06a5b0 0x7f115c06ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:27.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.785+0000 7f117c000640 1 --2- 172.21.15.90:0/1495603315 >> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] conn(0x55a5f36c86e0 0x55a5f3602dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:27.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.785+0000 7f117b600640 1 --2- 172.21.15.90:0/1495603315 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f115c06a5b0 0x7f115c06ca70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f116c007500 tx=0x7f116c005670 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:27.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.786+0000 7f117c000640 1 --2- 172.21.15.90:0/1495603315 >> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] conn(0x55a5f36c86e0 0x55a5f3602dc0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:27.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.792+0000 7f117c000640 1 -- 172.21.15.90:0/1495603315 <== osd.5 v2:172.21.15.120:6824/1739969887 1 ==== osd_op_reply(1 10000000001.00000000 [omap-get-keys] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 164+0+0 (crc 0 0 0) 0x7f117405d040 con 0x55a5f36c86e0 2024-03-24T01:26:27.795 INFO:teuthology.orchestra.run.smithi090.stderr:error getting omap key set cephfs_metadata/10000000001.00000000: (2) No such file or directory 2024-03-24T01:26:27.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.793+0000 7f117ec39f40 1 -- 172.21.15.90:0/1495603315 >> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] conn(0x55a5f36c86e0 msgr2=0x55a5f3602dc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:27.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.793+0000 7f117ec39f40 1 --2- 172.21.15.90:0/1495603315 >> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] conn(0x55a5f36c86e0 0x55a5f3602dc0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.793+0000 7f117ec39f40 1 -- 172.21.15.90:0/1495603315 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f115c06a5b0 msgr2=0x7f115c06ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:27.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.793+0000 7f117ec39f40 1 --2- 172.21.15.90:0/1495603315 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f115c06a5b0 0x7f115c06ca70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f116c007500 tx=0x7f116c005670 comp rx=0 tx=0).stop 2024-03-24T01:26:27.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.793+0000 7f117ec39f40 1 -- 172.21.15.90:0/1495603315 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a5f36d0570 msgr2=0x55a5f3648a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:27.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.793+0000 7f117ec39f40 1 --2- 172.21.15.90:0/1495603315 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a5f36d0570 0x55a5f3648a30 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f116800af00 tx=0x7f1168008690 comp rx=0 tx=0).stop 2024-03-24T01:26:27.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.793+0000 7f117c000640 1 -- 172.21.15.90:0/1495603315 reap_dead start 2024-03-24T01:26:27.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.793+0000 7f117ec39f40 1 -- 172.21.15.90:0/1495603315 shutdown_connections 2024-03-24T01:26:27.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.793+0000 7f117ec39f40 1 -- 172.21.15.90:0/1495603315 >> 172.21.15.90:0/1495603315 conn(0x55a5f362c640 msgr2=0x55a5f3647c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:27.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.793+0000 7f117ec39f40 1 -- 172.21.15.90:0/1495603315 shutdown_connections 2024-03-24T01:26:27.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.793+0000 7f117ec39f40 1 -- 172.21.15.90:0/1495603315 wait complete. 2024-03-24T01:26:27.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T01:26:27.799 ERROR:tasks.cephfs.filesystem:Command failed on smithi090 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 10000000001.00000000' 2024-03-24T01:26:27.799 DEBUG:teuthology.orchestra.run.smithi090:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 1.00000000 2024-03-24T01:26:27.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.881+0000 7f3c4ae36f40 1 Processor -- start 2024-03-24T01:26:27.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.881+0000 7f3c4ae36f40 1 -- start start 2024-03-24T01:26:27.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.882+0000 7f3c4ae36f40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5605df9c1800 0x5605df9ee810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:27.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.882+0000 7f3c4ae36f40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605df9ef5f0 0x5605dfb2d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:27.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.882+0000 7f3c4ae36f40 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605dfa806d0 0x5605dfb2d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:27.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.882+0000 7f3c4ae36f40 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605dfa9af90 con 0x5605dfa806d0 2024-03-24T01:26:27.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.882+0000 7f3c4ae36f40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605dfa7db70 con 0x5605df9ef5f0 2024-03-24T01:26:27.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.882+0000 7f3c4ae36f40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5605dfa9b230 con 0x5605df9c1800 2024-03-24T01:26:27.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.882+0000 7f3c48200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605dfa806d0 0x5605dfb2d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:27.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.882+0000 7f3c48200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605dfa806d0 0x5605dfb2d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49834/0 (socket says 172.21.15.90:49834) 2024-03-24T01:26:27.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.882+0000 7f3c48200640 1 -- 172.21.15.90:0/2561010034 learned_addr learned my addr 172.21.15.90:0/2561010034 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:27.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.882+0000 7f3c46e00640 1 --2- 172.21.15.90:0/2561010034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605df9ef5f0 0x5605dfb2d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:27.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.882+0000 7f3c47800640 1 --2- 172.21.15.90:0/2561010034 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5605df9c1800 0x5605df9ee810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:27.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.882+0000 7f3c48200640 1 -- 172.21.15.90:0/2561010034 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5605df9c1800 msgr2=0x5605df9ee810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:27.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.882+0000 7f3c48200640 1 --2- 172.21.15.90:0/2561010034 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5605df9c1800 0x5605df9ee810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.882+0000 7f3c48200640 1 -- 172.21.15.90:0/2561010034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605df9ef5f0 msgr2=0x5605dfb2d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:27.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.882+0000 7f3c48200640 1 --2- 172.21.15.90:0/2561010034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605df9ef5f0 0x5605dfb2d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.882+0000 7f3c48200640 1 -- 172.21.15.90:0/2561010034 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5605df8ebad0 con 0x5605dfa806d0 2024-03-24T01:26:27.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.883+0000 7f3c48200640 1 --2- 172.21.15.90:0/2561010034 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605dfa806d0 0x5605dfb2d5d0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f3c4000a5f0 tx=0x7f3c4000a6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3a6dc7791104aa2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:27.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.883+0000 7f3c45a00640 1 -- 172.21.15.90:0/2561010034 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c40018070 con 0x5605dfa806d0 2024-03-24T01:26:27.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.883+0000 7f3c45a00640 1 -- 172.21.15.90:0/2561010034 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c40007600 con 0x5605dfa806d0 2024-03-24T01:26:27.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.883+0000 7f3c45a00640 1 -- 172.21.15.90:0/2561010034 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c400079d0 con 0x5605dfa806d0 2024-03-24T01:26:27.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.883+0000 7f3c4ae36f40 1 -- 172.21.15.90:0/2561010034 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605dfa806d0 msgr2=0x5605dfb2d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:27.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.883+0000 7f3c4ae36f40 1 --2- 172.21.15.90:0/2561010034 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605dfa806d0 0x5605dfb2d5d0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f3c4000a5f0 tx=0x7f3c4000a6e0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.884+0000 7f3c4ae36f40 1 -- 172.21.15.90:0/2561010034 shutdown_connections 2024-03-24T01:26:27.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.884+0000 7f3c4ae36f40 1 --2- 172.21.15.90:0/2561010034 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605dfa806d0 0x5605dfb2d5d0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.884+0000 7f3c4ae36f40 1 --2- 172.21.15.90:0/2561010034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605df9ef5f0 0x5605dfb2d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.884+0000 7f3c4ae36f40 1 --2- 172.21.15.90:0/2561010034 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5605df9c1800 0x5605df9ee810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.884+0000 7f3c4ae36f40 1 -- 172.21.15.90:0/2561010034 >> 172.21.15.90:0/2561010034 conn(0x5605df9ec6d0 msgr2=0x5605df9be0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:27.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.884+0000 7f3c4ae36f40 1 -- 172.21.15.90:0/2561010034 shutdown_connections 2024-03-24T01:26:27.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.884+0000 7f3c4ae36f40 1 -- 172.21.15.90:0/2561010034 wait complete. 2024-03-24T01:26:27.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.884+0000 7f3c4ae36f40 1 Processor -- start 2024-03-24T01:26:27.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.885+0000 7f3c4ae36f40 1 -- start start 2024-03-24T01:26:27.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.885+0000 7f3c4ae36f40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605df9c1800 0x5605dfa79e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:27.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.885+0000 7f3c4ae36f40 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605df9ef5f0 0x5605dfb31f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:27.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.885+0000 7f3c4ae36f40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5605dfa806d0 0x5605df9bf460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:27.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.886+0000 7f3c4ae36f40 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605dfa7fd00 con 0x5605df9ef5f0 2024-03-24T01:26:27.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.886+0000 7f3c4ae36f40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605dfa09840 con 0x5605df9c1800 2024-03-24T01:26:27.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.886+0000 7f3c4ae36f40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5605dfa79800 con 0x5605dfa806d0 2024-03-24T01:26:27.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.886+0000 7f3c46e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605df9ef5f0 0x5605dfb31f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:27.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.886+0000 7f3c46e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605df9ef5f0 0x5605dfb31f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49846/0 (socket says 172.21.15.90:49846) 2024-03-24T01:26:27.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.886+0000 7f3c46e00640 1 -- 172.21.15.90:0/1231849152 learned_addr learned my addr 172.21.15.90:0/1231849152 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:27.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.886+0000 7f3c46e00640 1 -- 172.21.15.90:0/1231849152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5605dfa806d0 msgr2=0x5605df9bf460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:27.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.886+0000 7f3c46e00640 1 --2- 172.21.15.90:0/1231849152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5605dfa806d0 0x5605df9bf460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.886+0000 7f3c46e00640 1 -- 172.21.15.90:0/1231849152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605df9c1800 msgr2=0x5605dfa79e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:27.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.886+0000 7f3c46e00640 1 --2- 172.21.15.90:0/1231849152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605df9c1800 0x5605dfa79e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.886+0000 7f3c46e00640 1 -- 172.21.15.90:0/1231849152 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5605df901dc0 con 0x5605df9ef5f0 2024-03-24T01:26:27.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.886+0000 7f3c46e00640 1 --2- 172.21.15.90:0/1231849152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605df9ef5f0 0x5605dfb31f70 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f3c3000be30 tx=0x7f3c3000bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:27.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.887+0000 7f3c3fe00640 1 -- 172.21.15.90:0/1231849152 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c30019070 con 0x5605df9ef5f0 2024-03-24T01:26:27.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.887+0000 7f3c3fe00640 1 -- 172.21.15.90:0/1231849152 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c30007b50 con 0x5605df9ef5f0 2024-03-24T01:26:27.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.887+0000 7f3c4ae36f40 1 -- 172.21.15.90:0/1231849152 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5605dfa7b830 con 0x5605df9ef5f0 2024-03-24T01:26:27.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.887+0000 7f3c3fe00640 1 -- 172.21.15.90:0/1231849152 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c30014410 con 0x5605df9ef5f0 2024-03-24T01:26:27.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.887+0000 7f3c4ae36f40 1 -- 172.21.15.90:0/1231849152 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5605dfa7a3d0 con 0x5605df9ef5f0 2024-03-24T01:26:27.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.889+0000 7f3c3fe00640 1 -- 172.21.15.90:0/1231849152 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3c3001c070 con 0x5605df9ef5f0 2024-03-24T01:26:27.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.889+0000 7f3c3fe00640 1 --2- 172.21.15.90:0/1231849152 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3c2406a5b0 0x7f3c2406ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:27.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.890+0000 7f3c3fe00640 1 -- 172.21.15.90:0/1231849152 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f3c30088940 con 0x5605df9ef5f0 2024-03-24T01:26:27.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.890+0000 7f3c4ae36f40 1 --2- 172.21.15.90:0/1231849152 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x5605dfb324b0 0x5605dfa7ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:27.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.890+0000 7f3c4ae36f40 1 -- 172.21.15.90:0/1231849152 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:1 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x5605dfb30db0 con 0x5605dfb324b0 2024-03-24T01:26:27.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.890+0000 7f3c47800640 1 --2- 172.21.15.90:0/1231849152 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3c2406a5b0 0x7f3c2406ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:27.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.891+0000 7f3c48200640 1 --2- 172.21.15.90:0/1231849152 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x5605dfb324b0 0x5605dfa7ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:27.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.891+0000 7f3c47800640 1 --2- 172.21.15.90:0/1231849152 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3c2406a5b0 0x7f3c2406ca70 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3c38055000 tx=0x7f3c3804f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:27.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.891+0000 7f3c48200640 1 --2- 172.21.15.90:0/1231849152 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x5605dfb324b0 0x5605dfa7ca80 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:27.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.892+0000 7f3c48200640 1 -- 172.21.15.90:0/1231849152 <== osd.6 v2:172.21.15.120:6816/1454745630 1 ==== osd_op_reply(1 1.00000000 [omap-get-keys out=5b] v0'0 uv2 ondisk = 0) v8 ==== 154+0+5 (crc 0 0 0) 0x7f3c4001b040 con 0x5605dfb324b0 2024-03-24T01:26:27.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.892+0000 7f3c4ae36f40 1 -- 172.21.15.90:0/1231849152 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x5605dfb324b0 msgr2=0x5605dfa7ca80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:27.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.892+0000 7f3c4ae36f40 1 --2- 172.21.15.90:0/1231849152 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x5605dfb324b0 0x5605dfa7ca80 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.892+0000 7f3c4ae36f40 1 -- 172.21.15.90:0/1231849152 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3c2406a5b0 msgr2=0x7f3c2406ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:27.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.892+0000 7f3c4ae36f40 1 --2- 172.21.15.90:0/1231849152 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3c2406a5b0 0x7f3c2406ca70 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3c38055000 tx=0x7f3c3804f000 comp rx=0 tx=0).stop 2024-03-24T01:26:27.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.892+0000 7f3c4ae36f40 1 -- 172.21.15.90:0/1231849152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605df9ef5f0 msgr2=0x5605dfb31f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:27.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.893+0000 7f3c4ae36f40 1 --2- 172.21.15.90:0/1231849152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5605df9ef5f0 0x5605dfb31f70 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f3c3000be30 tx=0x7f3c3000bf20 comp rx=0 tx=0).stop 2024-03-24T01:26:27.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.893+0000 7f3c48200640 1 -- 172.21.15.90:0/1231849152 reap_dead start 2024-03-24T01:26:27.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.893+0000 7f3c4ae36f40 1 -- 172.21.15.90:0/1231849152 shutdown_connections 2024-03-24T01:26:27.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.893+0000 7f3c4ae36f40 1 -- 172.21.15.90:0/1231849152 >> 172.21.15.90:0/1231849152 conn(0x5605df9ec6d0 msgr2=0x5605dfa94a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:27.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.893+0000 7f3c4ae36f40 1 -- 172.21.15.90:0/1231849152 shutdown_connections 2024-03-24T01:26:27.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.893+0000 7f3c4ae36f40 1 -- 172.21.15.90:0/1231849152 wait complete. 2024-03-24T01:26:27.899 DEBUG:teuthology.orchestra.run.smithi090:> cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank cephfs:0 event recover_dentries list 2024-03-24T01:26:27.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.979+0000 7f6bae60e280 1 -- start start 2024-03-24T01:26:27.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.979+0000 7f6bae60e280 1 Processor -- start 2024-03-24T01:26:27.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.980+0000 7f6bae60e280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55917b660c00 0x55917b6ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:27.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.980+0000 7f6bae60e280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55917b661800 0x55917b6cab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:27.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.980+0000 7f6bae60e280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55917b661c00 0x55917b6cb080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:27.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.980+0000 7f6bae60e280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55917c436600 con 0x55917b660c00 2024-03-24T01:26:27.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.980+0000 7f6bae60e280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55917c436480 con 0x55917b661c00 2024-03-24T01:26:27.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.980+0000 7f6bae60e280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55917c436300 con 0x55917b661800 2024-03-24T01:26:27.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.980+0000 7f6baba00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55917b660c00 0x55917b6ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:27.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.980+0000 7f6baba00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55917b660c00 0x55917b6ca580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49872/0 (socket says 172.21.15.90:49872) 2024-03-24T01:26:27.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.980+0000 7f6baba00640 1 -- 172.21.15.90:0/1686742849 learned_addr learned my addr 172.21.15.90:0/1686742849 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:27.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.980+0000 7f6bab000640 1 --2- 172.21.15.90:0/1686742849 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55917b661800 0x55917b6cab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:27.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.980+0000 7f6bac400640 1 --2- 172.21.15.90:0/1686742849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55917b661c00 0x55917b6cb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:27.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.980+0000 7f6baba00640 1 -- 172.21.15.90:0/1686742849 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55917b661800 msgr2=0x55917b6cab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:27.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.980+0000 7f6baba00640 1 --2- 172.21.15.90:0/1686742849 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55917b661800 0x55917b6cab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.980+0000 7f6baba00640 1 -- 172.21.15.90:0/1686742849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55917b661c00 msgr2=0x55917b6cb080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:27.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.980+0000 7f6baba00640 1 --2- 172.21.15.90:0/1686742849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55917b661c00 0x55917b6cb080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.981+0000 7f6baba00640 1 -- 172.21.15.90:0/1686742849 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55917b694780 con 0x55917b660c00 2024-03-24T01:26:27.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.981+0000 7f6baba00640 1 --2- 172.21.15.90:0/1686742849 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55917b660c00 0x55917b6ca580 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x55917c471530 tx=0x55917c3f6600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c70ac8c60df2b88 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:27.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.981+0000 7f6ba8800640 1 -- 172.21.15.90:0/1686742849 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55917b686680 con 0x55917b660c00 2024-03-24T01:26:27.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.981+0000 7f6ba8800640 1 -- 172.21.15.90:0/1686742849 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55917b68ce00 con 0x55917b660c00 2024-03-24T01:26:27.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.981+0000 7f6ba8800640 1 -- 172.21.15.90:0/1686742849 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55917b686b60 con 0x55917b660c00 2024-03-24T01:26:27.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.981+0000 7f6bae60e280 1 -- 172.21.15.90:0/1686742849 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55917b6954a0 con 0x55917b660c00 2024-03-24T01:26:27.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.981+0000 7f6bae60e280 1 -- 172.21.15.90:0/1686742849 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x55917b694000 con 0x55917b660c00 2024-03-24T01:26:27.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.981+0000 7f6bae60e280 4 waiting for MDS map... 2024-03-24T01:26:27.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.981+0000 7f6ba8800640 1 -- 172.21.15.90:0/1686742849 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x55917c52c600 con 0x55917b660c00 2024-03-24T01:26:27.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.982+0000 7f6ba8800640 1 -- 172.21.15.90:0/1686742849 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 18) v1 ==== 2735+0+0 (secure 0 0 0) 0x55917b6bb180 con 0x55917b660c00 2024-03-24T01:26:27.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.982+0000 7f6bae60e280 4 Got MDS map 18 2024-03-24T01:26:27.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.982+0000 7f6bae60e280 10 main: JournalTool::main 2024-03-24T01:26:27.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.982+0000 7f6bae60e280 4 main: JournalTool: connecting to RADOS... 2024-03-24T01:26:27.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.982+0000 7f6bae60e280 1 Processor -- start 2024-03-24T01:26:27.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.983+0000 7f6bae60e280 1 -- start start 2024-03-24T01:26:27.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.983+0000 7f6bae60e280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55917c52f400 0x55917b6cb600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:27.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.983+0000 7f6bae60e280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55917c52f800 0x55917b6cbb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:27.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.983+0000 7f6bae60e280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55917c52fc00 0x55917b6cc100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:27.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.983+0000 7f6bae60e280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55917c436180 con 0x55917c52fc00 2024-03-24T01:26:27.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.983+0000 7f6bae60e280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55917b6c2480 con 0x55917c52f800 2024-03-24T01:26:27.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.983+0000 7f6bae60e280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55917b6c2900 con 0x55917c52f400 2024-03-24T01:26:27.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.984+0000 7f6bab000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55917c52fc00 0x55917b6cc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:27.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.984+0000 7f6bab000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55917c52fc00 0x55917b6cc100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49874/0 (socket says 172.21.15.90:49874) 2024-03-24T01:26:27.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.984+0000 7f6bab000640 1 -- 172.21.15.90:0/3461737871 learned_addr learned my addr 172.21.15.90:0/3461737871 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:27.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.984+0000 7f6bab000640 1 -- 172.21.15.90:0/3461737871 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55917c52f400 msgr2=0x55917b6cb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:27.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.984+0000 7f6bab000640 1 --2- 172.21.15.90:0/3461737871 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55917c52f400 0x55917b6cb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.984+0000 7f6bab000640 1 -- 172.21.15.90:0/3461737871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55917c52f800 msgr2=0x55917b6cbb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:27.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.984+0000 7f6bab000640 1 --2- 172.21.15.90:0/3461737871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55917c52f800 0x55917b6cbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.984+0000 7f6bab000640 1 -- 172.21.15.90:0/3461737871 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55917b6943c0 con 0x55917c52fc00 2024-03-24T01:26:27.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.984+0000 7f6bab000640 1 --2- 172.21.15.90:0/3461737871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55917c52fc00 0x55917b6cc100 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x55917c470a20 tx=0x55917c3f67e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab13a031fea8a120 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:27.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.984+0000 7f6ba6000640 1 -- 172.21.15.90:0/3461737871 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55917b6869c0 con 0x55917c52fc00 2024-03-24T01:26:27.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.984+0000 7f6ba6000640 1 -- 172.21.15.90:0/3461737871 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55917b68cc40 con 0x55917c52fc00 2024-03-24T01:26:27.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.984+0000 7f6ba6000640 1 -- 172.21.15.90:0/3461737871 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55917b686680 con 0x55917c52fc00 2024-03-24T01:26:27.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.985+0000 7f6bae60e280 1 -- 172.21.15.90:0/3461737871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55917c52fc00 msgr2=0x55917b6cc100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:27.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.985+0000 7f6bae60e280 1 --2- 172.21.15.90:0/3461737871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55917c52fc00 0x55917b6cc100 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x55917c470a20 tx=0x55917c3f67e0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.985+0000 7f6bae60e280 1 -- 172.21.15.90:0/3461737871 shutdown_connections 2024-03-24T01:26:27.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.985+0000 7f6bae60e280 1 --2- 172.21.15.90:0/3461737871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55917c52fc00 0x55917b6cc100 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.985+0000 7f6bae60e280 1 --2- 172.21.15.90:0/3461737871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55917c52f800 0x55917b6cbb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.985+0000 7f6bae60e280 1 --2- 172.21.15.90:0/3461737871 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55917c52f400 0x55917b6cb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.985+0000 7f6bae60e280 1 -- 172.21.15.90:0/3461737871 >> 172.21.15.90:0/3461737871 conn(0x55917c52f000 msgr2=0x55917b746800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:27.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.985+0000 7f6bae60e280 1 -- 172.21.15.90:0/3461737871 shutdown_connections 2024-03-24T01:26:27.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.985+0000 7f6bae60e280 1 -- 172.21.15.90:0/3461737871 wait complete. 2024-03-24T01:26:27.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.986+0000 7f6bae60e280 1 Processor -- start 2024-03-24T01:26:27.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.986+0000 7f6bae60e280 1 -- start start 2024-03-24T01:26:27.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.987+0000 7f6bae60e280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55917c52f400 0x55917b6cb600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:27.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.987+0000 7f6bae60e280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55917c52f800 0x55917b6cbb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:27.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.987+0000 7f6bae60e280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55917c52fc00 0x55917b6cc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:27.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.987+0000 7f6bae60e280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55917c437200 con 0x55917c52fc00 2024-03-24T01:26:27.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.987+0000 7f6bae60e280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55917c436180 con 0x55917c52f800 2024-03-24T01:26:27.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.987+0000 7f6bae60e280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55917c437380 con 0x55917c52f400 2024-03-24T01:26:27.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.987+0000 7f6bab000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55917c52fc00 0x55917b6cc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:27.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.987+0000 7f6bab000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55917c52fc00 0x55917b6cc680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49888/0 (socket says 172.21.15.90:49888) 2024-03-24T01:26:27.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.987+0000 7f6bab000640 1 -- 172.21.15.90:0/225243987 learned_addr learned my addr 172.21.15.90:0/225243987 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:27.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.987+0000 7f6baba00640 1 --2- 172.21.15.90:0/225243987 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55917c52f800 0x55917b6cbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:27.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.987+0000 7f6bac400640 1 --2- 172.21.15.90:0/225243987 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55917c52f400 0x55917b6cb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:27.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.987+0000 7f6bab000640 1 -- 172.21.15.90:0/225243987 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55917c52f400 msgr2=0x55917b6cb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:27.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.987+0000 7f6bab000640 1 --2- 172.21.15.90:0/225243987 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55917c52f400 0x55917b6cb600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.987+0000 7f6bab000640 1 -- 172.21.15.90:0/225243987 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55917c52f800 msgr2=0x55917b6cbb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:27.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.987+0000 7f6bab000640 1 --2- 172.21.15.90:0/225243987 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55917c52f800 0x55917b6cbb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:27.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.987+0000 7f6bab000640 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55917b694780 con 0x55917c52fc00 2024-03-24T01:26:27.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.988+0000 7f6bac400640 1 --2- 172.21.15.90:0/225243987 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55917c52f400 0x55917b6cb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:27.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.988+0000 7f6baba00640 1 --2- 172.21.15.90:0/225243987 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55917c52f800 0x55917b6cbb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:27.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.988+0000 7f6bab000640 1 --2- 172.21.15.90:0/225243987 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55917c52fc00 0x55917b6cc680 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x55917c544990 tx=0x55917c3f6a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:27.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.988+0000 7f6ba4200640 1 -- 172.21.15.90:0/225243987 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55917b686b60 con 0x55917c52fc00 2024-03-24T01:26:27.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.988+0000 7f6ba4200640 1 -- 172.21.15.90:0/225243987 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55917b68c1c0 con 0x55917c52fc00 2024-03-24T01:26:27.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.988+0000 7f6ba4200640 1 -- 172.21.15.90:0/225243987 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55917b6864e0 con 0x55917c52fc00 2024-03-24T01:26:27.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.988+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55917b6954a0 con 0x55917c52fc00 2024-03-24T01:26:27.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.988+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55917b694960 con 0x55917c52fc00 2024-03-24T01:26:27.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.988+0000 7f6bae60e280 4 main: JournalTool: resolving pool 5 2024-03-24T01:26:27.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.990+0000 7f6ba4200640 1 -- 172.21.15.90:0/225243987 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x55917c448c00 con 0x55917c52fc00 2024-03-24T01:26:27.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.990+0000 7f6ba4200640 1 --2- 172.21.15.90:0/225243987 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55917c59b400 0x55917b6cd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:27.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.990+0000 7f6ba4200640 1 -- 172.21.15.90:0/225243987 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x55917c59c600 con 0x55917c52fc00 2024-03-24T01:26:27.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.991+0000 7f6bae60e280 4 main: JournalTool: creating IoCtx.. 2024-03-24T01:26:27.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.991+0000 7f6bae60e280 4 main: Executing for rank 0 2024-03-24T01:26:27.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.991+0000 7f6bae60e280 1 --2- 172.21.15.90:0/225243987 >> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] conn(0x55917c439400 0x55917b6ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:27.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.991+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:1 5.f 5:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c439000 con 0x55917c439400 2024-03-24T01:26:27.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.991+0000 7f6bac400640 1 --2- 172.21.15.90:0/225243987 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55917c59b400 0x55917b6cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:27.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.991+0000 7f6baba00640 1 --2- 172.21.15.90:0/225243987 >> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] conn(0x55917c439400 0x55917b6ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:27.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.992+0000 7f6baba00640 1 --2- 172.21.15.90:0/225243987 >> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] conn(0x55917c439400 0x55917b6ccc00 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:27.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.992+0000 7f6bac400640 1 --2- 172.21.15.90:0/225243987 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55917c59b400 0x55917b6cd700 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x55917c3f27b0 tx=0x55917b9d0060 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:27.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.993+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x55917b7e2000 con 0x55917c439400 2024-03-24T01:26:27.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.993+0000 7f6bae60e280 4 Pointer 400.00000000 is readable 2024-03-24T01:26:27.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.994+0000 7f6bae60e280 4 JournalScanner::scan: reading header object '200.00000000' 2024-03-24T01:26:27.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.994+0000 7f6bae60e280 1 --2- 172.21.15.90:0/225243987 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55917c438c00 0x55917b6cd180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:27.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.994+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:2 5.14 5:292cf221:::200.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c438800 con 0x55917c438c00 2024-03-24T01:26:27.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.994+0000 7f6bac400640 1 --2- 172.21.15.90:0/225243987 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55917c438c00 0x55917b6cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:27.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.994+0000 7f6bac400640 1 --2- 172.21.15.90:0/225243987 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55917c438c00 0x55917b6cd180 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:27.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.995+0000 7f6bac400640 1 -- 172.21.15.90:0/225243987 <== osd.3 v2:172.21.15.90:6808/2999931983 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv23 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x55917b7e2500 con 0x55917c438c00 2024-03-24T01:26:27.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.995+0000 7f6bae60e280 10 Header 0x400000 0x400000 0x40a9d3 2024-03-24T01:26:27.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.995+0000 7f6bae60e280 10 Starting journal scan from offset 0x400000 2024-03-24T01:26:27.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.995+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:3 5.14 5:2e2fa760:::200.00000001:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4000 con 0x55917c438c00 2024-03-24T01:26:27.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bac400640 1 -- 172.21.15.90:0/225243987 <== osd.3 v2:172.21.15.90:6808/2999931983 2 ==== osd_op_reply(3 200.00000001 [read 0~56200 out=56200b] v0'0 uv29 ondisk = 0) v8 ==== 156+0+56200 (crc 0 0 0) 0x55917b7e2500 con 0x55917c438c00 2024-03-24T01:26:27.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 4 Read 0xdb88 bytes from 200.00000001 gap=0 2024-03-24T01:26:27.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Parsing data, 0xdb88 bytes available 2024-03-24T01:26:27.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Attempting decode at 0x400000 2024-03-24T01:26:27.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Consumed 0x30 bytes 2024-03-24T01:26:27.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Valid entry at 0x400000 2024-03-24T01:26:27.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Attempting decode at 0x400030 2024-03-24T01:26:27.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Consumed 0x37c bytes 2024-03-24T01:26:27.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Valid entry at 0x400030 2024-03-24T01:26:27.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Attempting decode at 0x4003ac 2024-03-24T01:26:27.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Consumed 0x1b9 bytes 2024-03-24T01:26:28.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Valid entry at 0x4003ac 2024-03-24T01:26:28.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Attempting decode at 0x400565 2024-03-24T01:26:28.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Consumed 0x32f bytes 2024-03-24T01:26:28.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Valid entry at 0x400565 2024-03-24T01:26:28.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Attempting decode at 0x400894 2024-03-24T01:26:28.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Consumed 0x6b2 bytes 2024-03-24T01:26:28.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Valid entry at 0x400894 2024-03-24T01:26:28.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Attempting decode at 0x400f46 2024-03-24T01:26:28.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Consumed 0x7ce bytes 2024-03-24T01:26:28.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Valid entry at 0x400f46 2024-03-24T01:26:28.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Attempting decode at 0x401714 2024-03-24T01:26:28.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Consumed 0x7ac bytes 2024-03-24T01:26:28.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Valid entry at 0x401714 2024-03-24T01:26:28.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Attempting decode at 0x401ec0 2024-03-24T01:26:28.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Consumed 0x7df bytes 2024-03-24T01:26:28.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Valid entry at 0x401ec0 2024-03-24T01:26:28.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Attempting decode at 0x40269f 2024-03-24T01:26:28.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Consumed 0x7a8 bytes 2024-03-24T01:26:28.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Valid entry at 0x40269f 2024-03-24T01:26:28.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Attempting decode at 0x402e47 2024-03-24T01:26:28.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Consumed 0x787 bytes 2024-03-24T01:26:28.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Valid entry at 0x402e47 2024-03-24T01:26:28.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Attempting decode at 0x4035ce 2024-03-24T01:26:28.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Consumed 0xc66 bytes 2024-03-24T01:26:28.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Valid entry at 0x4035ce 2024-03-24T01:26:28.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Attempting decode at 0x404234 2024-03-24T01:26:28.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Consumed 0x472 bytes 2024-03-24T01:26:28.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Valid entry at 0x404234 2024-03-24T01:26:28.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Attempting decode at 0x4046a6 2024-03-24T01:26:28.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Consumed 0x78b bytes 2024-03-24T01:26:28.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Valid entry at 0x4046a6 2024-03-24T01:26:28.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Attempting decode at 0x404e31 2024-03-24T01:26:28.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Consumed 0xb60 bytes 2024-03-24T01:26:28.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Valid entry at 0x404e31 2024-03-24T01:26:28.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Attempting decode at 0x405991 2024-03-24T01:26:28.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Consumed 0x7e6 bytes 2024-03-24T01:26:28.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Valid entry at 0x405991 2024-03-24T01:26:28.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Attempting decode at 0x406177 2024-03-24T01:26:28.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Consumed 0x7c0 bytes 2024-03-24T01:26:28.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Valid entry at 0x406177 2024-03-24T01:26:28.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Attempting decode at 0x406937 2024-03-24T01:26:28.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Consumed 0xb60 bytes 2024-03-24T01:26:28.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Valid entry at 0x406937 2024-03-24T01:26:28.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Attempting decode at 0x407497 2024-03-24T01:26:28.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Consumed 0xb60 bytes 2024-03-24T01:26:28.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Valid entry at 0x407497 2024-03-24T01:26:28.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Attempting decode at 0x407ff7 2024-03-24T01:26:28.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Consumed 0xb63 bytes 2024-03-24T01:26:28.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Valid entry at 0x407ff7 2024-03-24T01:26:28.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Attempting decode at 0x408b5a 2024-03-24T01:26:28.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Consumed 0x7a8 bytes 2024-03-24T01:26:28.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Valid entry at 0x408b5a 2024-03-24T01:26:28.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Attempting decode at 0x409302 2024-03-24T01:26:28.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Consumed 0xb6e bytes 2024-03-24T01:26:28.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Valid entry at 0x409302 2024-03-24T01:26:28.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Attempting decode at 0x409e70 2024-03-24T01:26:28.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Consumed 0xb63 bytes 2024-03-24T01:26:28.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Valid entry at 0x409e70 2024-03-24T01:26:28.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Attempting decode at 0x40a9d3 2024-03-24T01:26:28.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Consumed 0x681 bytes 2024-03-24T01:26:28.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Valid entry at 0x40a9d3 2024-03-24T01:26:28.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Attempting decode at 0x40b054 2024-03-24T01:26:28.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Consumed 0xa56 bytes 2024-03-24T01:26:28.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Valid entry at 0x40b054 2024-03-24T01:26:28.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Attempting decode at 0x40baaa 2024-03-24T01:26:28.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Consumed 0x8d9 bytes 2024-03-24T01:26:28.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Valid entry at 0x40baaa 2024-03-24T01:26:28.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Attempting decode at 0x40c383 2024-03-24T01:26:28.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Consumed 0x257 bytes 2024-03-24T01:26:28.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Valid entry at 0x40c383 2024-03-24T01:26:28.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Attempting decode at 0x40c5da 2024-03-24T01:26:28.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Consumed 0x9d0 bytes 2024-03-24T01:26:28.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Valid entry at 0x40c5da 2024-03-24T01:26:28.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Attempting decode at 0x40cfaa 2024-03-24T01:26:28.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Consumed 0x673 bytes 2024-03-24T01:26:28.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Valid entry at 0x40cfaa 2024-03-24T01:26:28.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Attempting decode at 0x40d61d 2024-03-24T01:26:28.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Consumed 0x472 bytes 2024-03-24T01:26:28.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Valid entry at 0x40d61d 2024-03-24T01:26:28.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Attempting decode at 0x40da8f 2024-03-24T01:26:28.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Consumed 0xad bytes 2024-03-24T01:26:28.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Valid entry at 0x40da8f 2024-03-24T01:26:28.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Attempting decode at 0x40db3c 2024-03-24T01:26:28.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Consumed 0x4c bytes 2024-03-24T01:26:28.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 10 Valid entry at 0x40db3c 2024-03-24T01:26:28.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.996+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:4 5.1b 5:ddb4e4d7:::200.00000002:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4400 con 0x55917c439400 2024-03-24T01:26:28.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.997+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 2 ==== osd_op_reply(4 200.00000002 [read 0~2147483647] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x55917b7e2000 con 0x55917c439400 2024-03-24T01:26:28.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.997+0000 7f6bae60e280 4 Reached end of journal objects 2024-03-24T01:26:28.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.997+0000 7f6bae60e280 4 Scanned objects, 0 missing, 1 valid 2024-03-24T01:26:28.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.997+0000 7f6bae60e280 4 Events scanned, 0 gaps 2024-03-24T01:26:28.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.997+0000 7f6bae60e280 4 Found 31 valid events 2024-03-24T01:26:28.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.997+0000 7f6bae60e280 4 Selected 31 events events for processing 2024-03-24T01:26:28.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.997+0000 7f6bae60e280 4 recover_dentries: inspecting lump 1.00000000 2024-03-24T01:26:28.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.997+0000 7f6bae60e280 1 --2- 172.21.15.90:0/225243987 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x55917b9e4800 0x55917ba04000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:28.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.997+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:5 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4c00 con 0x55917b9e4800 2024-03-24T01:26:28.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.998+0000 7f6baba00640 1 --2- 172.21.15.90:0/225243987 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x55917b9e4800 0x55917ba04000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:28.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:27.999+0000 7f6baba00640 1 --2- 172.21.15.90:0/225243987 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x55917b9e4800 0x55917ba04000 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:28.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.000+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 1 ==== osd_op_reply(5 1.00000000 [omap-get-header out=274b] v0'0 uv2 ondisk = 0) v8 ==== 154+0+274 (crc 0 0 0) 0x55917b7e2000 con 0x55917b9e4800 2024-03-24T01:26:28.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.000+0000 7f6bae60e280 4 recover_dentries: frag 1.00000000 fnode old v1 vs new v1 2024-03-24T01:26:28.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.000+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:6 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5000 con 0x55917b9e4800 2024-03-24T01:26:28.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.001+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 2 ==== osd_op_reply(6 1.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv2 ondisk = 0) v8 ==== 154+0+4 (crc 0 0 0) 0x55917b7e2000 con 0x55917b9e4800 2024-03-24T01:26:28.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.001+0000 7f6bae60e280 4 recover_dentries: inspecting lump 100.00000000 2024-03-24T01:26:28.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.001+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:7 5.13 5:cd5a64a3:::100.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5400 con 0x55917c438c00 2024-03-24T01:26:28.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.001+0000 7f6bac400640 1 -- 172.21.15.90:0/225243987 <== osd.3 v2:172.21.15.90:6808/2999931983 3 ==== osd_op_reply(7 100.00000000 [omap-get-header out=274b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+274 (crc 0 0 0) 0x55917b7e2500 con 0x55917c438c00 2024-03-24T01:26:28.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.002+0000 7f6bae60e280 4 recover_dentries: frag 100.00000000 fnode old v1 vs new v1 2024-03-24T01:26:28.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.002+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:8 5.13 5:cd5a64a3:::100.00000000:head [omap-get-vals-by-keys in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4400 con 0x55917c438c00 2024-03-24T01:26:28.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.002+0000 7f6bac400640 1 -- 172.21.15.90:0/225243987 <== osd.3 v2:172.21.15.90:6808/2999931983 4 ==== osd_op_reply(8 100.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+4 (crc 0 0 0) 0x55917b7e2500 con 0x55917c438c00 2024-03-24T01:26:28.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.002+0000 7f6bae60e280 4 recover_dentries: updating root 0x0x1 2024-03-24T01:26:28.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.002+0000 7f6bae60e280 4 recover_dentries: object id 1.00000000.inode 2024-03-24T01:26:28.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.002+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:9 5.14 5:287034c4:::1.00000000.inode:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4c00 con 0x55917c438c00 2024-03-24T01:26:28.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.002+0000 7f6bac400640 1 -- 172.21.15.90:0/225243987 <== osd.3 v2:172.21.15.90:6808/2999931983 5 ==== osd_op_reply(9 1.00000000.inode [read 0~582 out=582b] v0'0 uv4 ondisk = 0) v8 ==== 160+0+582 (crc 0 0 0) 0x55917b7e2500 con 0x55917c438c00 2024-03-24T01:26:28.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.002+0000 7f6bae60e280 4 recover_dentries: root exists, will modify (582) 2024-03-24T01:26:28.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.002+0000 7f6bae60e280 4 recover_dentries: magic ok 2024-03-24T01:26:28.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.002+0000 7f6bae60e280 4 recover_dentries: writing root ino 1.00000000.inode version 2 2024-03-24T01:26:28.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.002+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:10 5.14 5:287034c4:::1.00000000.inode:head [writefull 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5000 con 0x55917c438c00 2024-03-24T01:26:28.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.006+0000 7f6bac400640 1 -- 172.21.15.90:0/225243987 <== osd.3 v2:172.21.15.90:6808/2999931983 6 ==== osd_op_reply(10 1.00000000.inode [writefull 0~582] v40'30 uv30 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x55917b7e2500 con 0x55917c438c00 2024-03-24T01:26:28.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.006+0000 7f6bae60e280 4 recover_dentries: inspecting lump 1.00000000 2024-03-24T01:26:28.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.006+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:11 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5800 con 0x55917b9e4800 2024-03-24T01:26:28.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.007+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 3 ==== osd_op_reply(11 1.00000000 [omap-get-header out=274b] v0'0 uv2 ondisk = 0) v8 ==== 154+0+274 (crc 0 0 0) 0x55917b7e2000 con 0x55917b9e4800 2024-03-24T01:26:28.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.007+0000 7f6bae60e280 4 recover_dentries: frag 1.00000000 fnode old v1 vs new v3 2024-03-24T01:26:28.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.007+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.007+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:12 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5c00 con 0x55917b9e4800 2024-03-24T01:26:28.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.012+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 4 ==== osd_op_reply(12 1.00000000 [omap-set-header] v40'5 uv5 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917b7e2000 con 0x55917b9e4800 2024-03-24T01:26:28.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.012+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:13 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=21b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c439000 con 0x55917b9e4800 2024-03-24T01:26:28.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.014+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 5 ==== osd_op_reply(13 1.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv5 ondisk = 0) v8 ==== 154+0+4 (crc 0 0 0) 0x55917b7e2000 con 0x55917b9e4800 2024-03-24T01:26:28.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.014+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 1.00000000/rootfile 2024-03-24T01:26:28.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.014+0000 7f6bae60e280 4 recover_dentries: dentry did not already exist, will create 2024-03-24T01:26:28.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.014+0000 7f6bae60e280 4 recover_dentries: writing I dentry rootfile_head into frag 1.00000000 2024-03-24T01:26:28.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.014+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:14 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-vals in=547b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c52e800 con 0x55917b9e4800 2024-03-24T01:26:28.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.017+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 6 ==== osd_op_reply(14 1.00000000 [omap-set-vals] v40'6 uv6 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917b7e2000 con 0x55917b9e4800 2024-03-24T01:26:28.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.017+0000 7f6bae60e280 4 recover_dentries: updating root 0x0x1 2024-03-24T01:26:28.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.017+0000 7f6bae60e280 4 recover_dentries: object id 1.00000000.inode 2024-03-24T01:26:28.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.017+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:15 5.14 5:287034c4:::1.00000000.inode:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4000 con 0x55917c438c00 2024-03-24T01:26:28.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.017+0000 7f6bac400640 1 -- 172.21.15.90:0/225243987 <== osd.3 v2:172.21.15.90:6808/2999931983 7 ==== osd_op_reply(15 1.00000000.inode [read 0~582 out=582b] v0'0 uv30 ondisk = 0) v8 ==== 160+0+582 (crc 0 0 0) 0x55917b7e2500 con 0x55917c438c00 2024-03-24T01:26:28.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.017+0000 7f6bae60e280 4 recover_dentries: root exists, will modify (582) 2024-03-24T01:26:28.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.017+0000 7f6bae60e280 4 recover_dentries: magic ok 2024-03-24T01:26:28.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.017+0000 7f6bae60e280 4 recover_dentries: writing root ino 1.00000000.inode version 3 2024-03-24T01:26:28.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.017+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:16 5.14 5:287034c4:::1.00000000.inode:head [writefull 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5400 con 0x55917c438c00 2024-03-24T01:26:28.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.020+0000 7f6bac400640 1 -- 172.21.15.90:0/225243987 <== osd.3 v2:172.21.15.90:6808/2999931983 8 ==== osd_op_reply(16 1.00000000.inode [writefull 0~582] v40'31 uv31 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x55917b7e2500 con 0x55917c438c00 2024-03-24T01:26:28.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.020+0000 7f6bae60e280 4 recover_dentries: inspecting lump 1.00000000 2024-03-24T01:26:28.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.020+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:17 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4400 con 0x55917b9e4800 2024-03-24T01:26:28.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.021+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 7 ==== osd_op_reply(17 1.00000000 [omap-get-header out=274b] v0'0 uv6 ondisk = 0) v8 ==== 154+0+274 (crc 0 0 0) 0x55917b7e2000 con 0x55917b9e4800 2024-03-24T01:26:28.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.022+0000 7f6bae60e280 4 recover_dentries: frag 1.00000000 fnode old v3 vs new v5 2024-03-24T01:26:28.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.022+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.022+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:18 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c438800 con 0x55917b9e4800 2024-03-24T01:26:28.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.025+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 8 ==== osd_op_reply(18 1.00000000 [omap-set-header] v40'7 uv7 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917b7e2000 con 0x55917b9e4800 2024-03-24T01:26:28.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.025+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:19 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c439000 con 0x55917b9e4800 2024-03-24T01:26:28.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.026+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 9 ==== osd_op_reply(19 1.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv7 ondisk = 0) v8 ==== 154+0+4 (crc 0 0 0) 0x55917b7e2000 con 0x55917b9e4800 2024-03-24T01:26:28.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.026+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 1.00000000/subdir 2024-03-24T01:26:28.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.026+0000 7f6bae60e280 4 recover_dentries: dentry did not already exist, will create 2024-03-24T01:26:28.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.026+0000 7f6bae60e280 4 recover_dentries: writing I dentry subdir_head into frag 1.00000000 2024-03-24T01:26:28.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.026+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:20 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-vals in=507b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c438800 con 0x55917b9e4800 2024-03-24T01:26:28.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.031+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 10 ==== osd_op_reply(20 1.00000000 [omap-set-vals] v40'8 uv8 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917b7e2000 con 0x55917b9e4800 2024-03-24T01:26:28.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.031+0000 7f6bae60e280 4 recover_dentries: inspecting lump 10000000001.00000000 2024-03-24T01:26:28.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.031+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:21 5.7 5:e2e07930:::10000000001.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5800 con 0x55917c439400 2024-03-24T01:26:28.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.031+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 3 ==== osd_op_reply(21 10000000001.00000000 [omap-get-header] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 164+0+0 (crc 0 0 0) 0x55917b9fcc80 con 0x55917c439400 2024-03-24T01:26:28.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.031+0000 7f6bae60e280 4 recover_dentries: failed to read OMAP header from directory fragment 10000000001.00000000 (2) No such file or directory 2024-03-24T01:26:28.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.031+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.031+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:22 5.7 5:e2e07930:::10000000001.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5c00 con 0x55917c439400 2024-03-24T01:26:28.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.036+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 4 ==== osd_op_reply(22 10000000001.00000000 [omap-set-header] v40'3 uv3 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917b9fcc80 con 0x55917c439400 2024-03-24T01:26:28.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.036+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:23 5.7 5:e2e07930:::10000000001.00000000:head [omap-get-vals-by-keys in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4400 con 0x55917c439400 2024-03-24T01:26:28.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.036+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 5 ==== osd_op_reply(23 10000000001.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+4 (crc 0 0 0) 0x55917b9fcc80 con 0x55917c439400 2024-03-24T01:26:28.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.036+0000 7f6bae60e280 4 recover_dentries: updating root 0x0x1 2024-03-24T01:26:28.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.036+0000 7f6bae60e280 4 recover_dentries: object id 1.00000000.inode 2024-03-24T01:26:28.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.036+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:24 5.14 5:287034c4:::1.00000000.inode:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c52e800 con 0x55917c438c00 2024-03-24T01:26:28.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.036+0000 7f6bac400640 1 -- 172.21.15.90:0/225243987 <== osd.3 v2:172.21.15.90:6808/2999931983 9 ==== osd_op_reply(24 1.00000000.inode [read 0~582 out=582b] v0'0 uv31 ondisk = 0) v8 ==== 160+0+582 (crc 0 0 0) 0x55917b7e2500 con 0x55917c438c00 2024-03-24T01:26:28.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.036+0000 7f6bae60e280 4 recover_dentries: root exists, will modify (582) 2024-03-24T01:26:28.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.036+0000 7f6bae60e280 4 recover_dentries: magic ok 2024-03-24T01:26:28.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.036+0000 7f6bae60e280 4 recover_dentries: writing root ino 1.00000000.inode version 4 2024-03-24T01:26:28.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.036+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:25 5.14 5:287034c4:::1.00000000.inode:head [writefull 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c52e800 con 0x55917c438c00 2024-03-24T01:26:28.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.040+0000 7f6bac400640 1 -- 172.21.15.90:0/225243987 <== osd.3 v2:172.21.15.90:6808/2999931983 10 ==== osd_op_reply(25 1.00000000.inode [writefull 0~582] v40'32 uv32 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x55917b7e2500 con 0x55917c438c00 2024-03-24T01:26:28.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.040+0000 7f6bae60e280 4 recover_dentries: inspecting lump 1.00000000 2024-03-24T01:26:28.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.040+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:26 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4c00 con 0x55917b9e4800 2024-03-24T01:26:28.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.042+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 11 ==== osd_op_reply(26 1.00000000 [omap-get-header out=274b] v0'0 uv8 ondisk = 0) v8 ==== 154+0+274 (crc 0 0 0) 0x55917b9fcc80 con 0x55917b9e4800 2024-03-24T01:26:28.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.042+0000 7f6bae60e280 4 recover_dentries: frag 1.00000000 fnode old v5 vs new v7 2024-03-24T01:26:28.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.042+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.042+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:27 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5000 con 0x55917b9e4800 2024-03-24T01:26:28.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.045+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 12 ==== osd_op_reply(27 1.00000000 [omap-set-header] v40'9 uv9 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917b9fcc80 con 0x55917b9e4800 2024-03-24T01:26:28.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.045+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:28 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4000 con 0x55917b9e4800 2024-03-24T01:26:28.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.047+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 13 ==== osd_op_reply(28 1.00000000 [omap-get-vals-by-keys out=507b] v0'0 uv9 ondisk = 0) v8 ==== 154+0+507 (crc 0 0 0) 0x55917b9fcc80 con 0x55917b9e4800 2024-03-24T01:26:28.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.047+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 1.00000000/subdir 2024-03-24T01:26:28.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.047+0000 7f6bae60e280 4 recover_dentries: dentry subdir_head existed already 2024-03-24T01:26:28.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.047+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.047+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 4 vs fullbit version 6 2024-03-24T01:26:28.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.047+0000 7f6bae60e280 4 recover_dentries: writing I dentry subdir_head into frag 1.00000000 2024-03-24T01:26:28.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.047+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:29 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-vals in=507b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5400 con 0x55917b9e4800 2024-03-24T01:26:28.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.051+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 14 ==== osd_op_reply(29 1.00000000 [omap-set-vals] v40'10 uv10 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917b9fcc80 con 0x55917b9e4800 2024-03-24T01:26:28.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.051+0000 7f6bae60e280 4 recover_dentries: inspecting lump 10000000001.00000000 2024-03-24T01:26:28.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.051+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:30 5.7 5:e2e07930:::10000000001.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5800 con 0x55917c439400 2024-03-24T01:26:28.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.052+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 6 ==== osd_op_reply(30 10000000001.00000000 [omap-get-header out=274b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55917b9fcc80 con 0x55917c439400 2024-03-24T01:26:28.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.052+0000 7f6bae60e280 4 recover_dentries: frag 10000000001.00000000 fnode old v1 vs new v3 2024-03-24T01:26:28.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.052+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.052+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:31 5.7 5:e2e07930:::10000000001.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5c00 con 0x55917c439400 2024-03-24T01:26:28.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.055+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 7 ==== osd_op_reply(31 10000000001.00000000 [omap-set-header] v40'4 uv4 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917b9fcc80 con 0x55917c439400 2024-03-24T01:26:28.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.055+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:32 5.7 5:e2e07930:::10000000001.00000000:head [omap-get-vals-by-keys in=23b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4400 con 0x55917c439400 2024-03-24T01:26:28.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.058+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 8 ==== osd_op_reply(32 10000000001.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv4 ondisk = 0) v8 ==== 164+0+4 (crc 0 0 0) 0x55917b9fcc80 con 0x55917c439400 2024-03-24T01:26:28.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.058+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 10000000001.00000000/subdirfile 2024-03-24T01:26:28.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.058+0000 7f6bae60e280 4 recover_dentries: dentry did not already exist, will create 2024-03-24T01:26:28.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.058+0000 7f6bae60e280 4 recover_dentries: writing I dentry subdirfile_head into frag 10000000001.00000000 2024-03-24T01:26:28.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.058+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:33 5.7 5:e2e07930:::10000000001.00000000:head [omap-set-vals in=549b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4c00 con 0x55917c439400 2024-03-24T01:26:28.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.061+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 9 ==== osd_op_reply(33 10000000001.00000000 [omap-set-vals] v40'5 uv5 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917b9fcc80 con 0x55917c439400 2024-03-24T01:26:28.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.061+0000 7f6bae60e280 4 recover_dentries: inspecting lump 1.00000000 2024-03-24T01:26:28.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.061+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:34 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5000 con 0x55917b9e4800 2024-03-24T01:26:28.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.062+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 15 ==== osd_op_reply(34 1.00000000 [omap-get-header out=274b] v0'0 uv10 ondisk = 0) v8 ==== 154+0+274 (crc 0 0 0) 0x55917b9fcf00 con 0x55917b9e4800 2024-03-24T01:26:28.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.062+0000 7f6bae60e280 4 recover_dentries: frag 1.00000000 fnode old v7 vs new v9 2024-03-24T01:26:28.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.062+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.062+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:35 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c439000 con 0x55917b9e4800 2024-03-24T01:26:28.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.067+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 16 ==== osd_op_reply(35 1.00000000 [omap-set-header] v40'11 uv11 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917b9fcf00 con 0x55917b9e4800 2024-03-24T01:26:28.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.067+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:36 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c438800 con 0x55917b9e4800 2024-03-24T01:26:28.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.068+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 17 ==== osd_op_reply(36 1.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv11 ondisk = 0) v8 ==== 154+0+4 (crc 0 0 0) 0x55917b9fcf00 con 0x55917b9e4800 2024-03-24T01:26:28.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.068+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 1.00000000/linkdir 2024-03-24T01:26:28.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.068+0000 7f6bae60e280 4 recover_dentries: dentry did not already exist, will create 2024-03-24T01:26:28.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.068+0000 7f6bae60e280 4 recover_dentries: writing I dentry linkdir_head into frag 1.00000000 2024-03-24T01:26:28.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.068+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:37 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-vals in=508b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c439000 con 0x55917b9e4800 2024-03-24T01:26:28.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.071+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 18 ==== osd_op_reply(37 1.00000000 [omap-set-vals] v40'12 uv12 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917b9fcf00 con 0x55917b9e4800 2024-03-24T01:26:28.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.071+0000 7f6bae60e280 4 recover_dentries: inspecting lump 100000001f7.00000000 2024-03-24T01:26:28.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.071+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:38 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4000 con 0x55917c439400 2024-03-24T01:26:28.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.072+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 10 ==== osd_op_reply(38 100000001f7.00000000 [omap-get-header] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 164+0+0 (crc 0 0 0) 0x55917b9fcf00 con 0x55917c439400 2024-03-24T01:26:28.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.072+0000 7f6bae60e280 4 recover_dentries: failed to read OMAP header from directory fragment 100000001f7.00000000 (2) No such file or directory 2024-03-24T01:26:28.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.072+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.072+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:39 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5400 con 0x55917c439400 2024-03-24T01:26:28.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.077+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 11 ==== osd_op_reply(39 100000001f7.00000000 [omap-set-header] v40'1 uv1 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917b9fcf00 con 0x55917c439400 2024-03-24T01:26:28.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.077+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:40 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-get-vals-by-keys in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5800 con 0x55917c439400 2024-03-24T01:26:28.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.078+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 12 ==== osd_op_reply(40 100000001f7.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv1 ondisk = 0) v8 ==== 164+0+4 (crc 0 0 0) 0x55917b9fcf00 con 0x55917c439400 2024-03-24T01:26:28.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.078+0000 7f6bae60e280 4 recover_dentries: updating root 0x0x1 2024-03-24T01:26:28.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.078+0000 7f6bae60e280 4 recover_dentries: object id 1.00000000.inode 2024-03-24T01:26:28.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.078+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:41 5.14 5:287034c4:::1.00000000.inode:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5c00 con 0x55917c438c00 2024-03-24T01:26:28.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.079+0000 7f6bac400640 1 -- 172.21.15.90:0/225243987 <== osd.3 v2:172.21.15.90:6808/2999931983 11 ==== osd_op_reply(41 1.00000000.inode [read 0~582 out=582b] v0'0 uv32 ondisk = 0) v8 ==== 160+0+582 (crc 0 0 0) 0x55917b7e2500 con 0x55917c438c00 2024-03-24T01:26:28.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.079+0000 7f6bae60e280 4 recover_dentries: root exists, will modify (582) 2024-03-24T01:26:28.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.079+0000 7f6bae60e280 4 recover_dentries: magic ok 2024-03-24T01:26:28.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.079+0000 7f6bae60e280 4 recover_dentries: writing root ino 1.00000000.inode version 5 2024-03-24T01:26:28.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.079+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:42 5.14 5:287034c4:::1.00000000.inode:head [writefull 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4400 con 0x55917c438c00 2024-03-24T01:26:28.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.084+0000 7f6bac400640 1 -- 172.21.15.90:0/225243987 <== osd.3 v2:172.21.15.90:6808/2999931983 12 ==== osd_op_reply(42 1.00000000.inode [writefull 0~582] v40'33 uv33 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x55917b7e2500 con 0x55917c438c00 2024-03-24T01:26:28.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.084+0000 7f6bae60e280 4 recover_dentries: inspecting lump 1.00000000 2024-03-24T01:26:28.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.084+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:43 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4c00 con 0x55917b9e4800 2024-03-24T01:26:28.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.084+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 19 ==== osd_op_reply(43 1.00000000 [omap-get-header out=274b] v0'0 uv12 ondisk = 0) v8 ==== 154+0+274 (crc 0 0 0) 0x55917b9fcf00 con 0x55917b9e4800 2024-03-24T01:26:28.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.084+0000 7f6bae60e280 4 recover_dentries: frag 1.00000000 fnode old v9 vs new v11 2024-03-24T01:26:28.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.084+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.084+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:44 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5000 con 0x55917b9e4800 2024-03-24T01:26:28.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.089+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 20 ==== osd_op_reply(44 1.00000000 [omap-set-header] v40'13 uv13 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917b9fcf00 con 0x55917b9e4800 2024-03-24T01:26:28.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.089+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:45 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba16000 con 0x55917b9e4800 2024-03-24T01:26:28.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.090+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 21 ==== osd_op_reply(45 1.00000000 [omap-get-vals-by-keys out=508b] v0'0 uv13 ondisk = 0) v8 ==== 154+0+508 (crc 0 0 0) 0x55917b9fcf00 con 0x55917b9e4800 2024-03-24T01:26:28.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.090+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 1.00000000/linkdir 2024-03-24T01:26:28.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.090+0000 7f6bae60e280 4 recover_dentries: dentry linkdir_head existed already 2024-03-24T01:26:28.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.090+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.090+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 8 vs fullbit version 10 2024-03-24T01:26:28.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.090+0000 7f6bae60e280 4 recover_dentries: writing I dentry linkdir_head into frag 1.00000000 2024-03-24T01:26:28.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.090+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:46 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-vals in=508b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba16400 con 0x55917b9e4800 2024-03-24T01:26:28.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.094+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 22 ==== osd_op_reply(46 1.00000000 [omap-set-vals] v40'14 uv14 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917b9fcf00 con 0x55917b9e4800 2024-03-24T01:26:28.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.094+0000 7f6bae60e280 4 recover_dentries: inspecting lump 100000001f7.00000000 2024-03-24T01:26:28.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.094+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:47 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba16800 con 0x55917c439400 2024-03-24T01:26:28.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.095+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 13 ==== osd_op_reply(47 100000001f7.00000000 [omap-get-header out=274b] v0'0 uv1 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55917b9fcc80 con 0x55917c439400 2024-03-24T01:26:28.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.095+0000 7f6bae60e280 4 recover_dentries: frag 100000001f7.00000000 fnode old v1 vs new v3 2024-03-24T01:26:28.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.095+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.095+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:48 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba16c00 con 0x55917c439400 2024-03-24T01:26:28.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.098+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 14 ==== osd_op_reply(48 100000001f7.00000000 [omap-set-header] v40'2 uv2 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917b9fcc80 con 0x55917c439400 2024-03-24T01:26:28.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.098+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:49 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-get-vals-by-keys in=18b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba17000 con 0x55917c439400 2024-03-24T01:26:28.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.100+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 15 ==== osd_op_reply(49 100000001f7.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv2 ondisk = 0) v8 ==== 164+0+4 (crc 0 0 0) 0x55917b9fcc80 con 0x55917c439400 2024-03-24T01:26:28.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.101+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 100000001f7.00000000/link0 2024-03-24T01:26:28.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.101+0000 7f6bae60e280 4 recover_dentries: dentry did not already exist, will create 2024-03-24T01:26:28.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.101+0000 7f6bae60e280 4 recover_dentries: writing I dentry link0_head into frag 100000001f7.00000000 2024-03-24T01:26:28.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.101+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:50 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-set-vals in=544b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba17400 con 0x55917c439400 2024-03-24T01:26:28.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.105+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 16 ==== osd_op_reply(50 100000001f7.00000000 [omap-set-vals] v40'3 uv3 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917b9fcc80 con 0x55917c439400 2024-03-24T01:26:28.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.105+0000 7f6bae60e280 4 recover_dentries: inspecting lump 1.00000000 2024-03-24T01:26:28.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.105+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:51 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba17800 con 0x55917b9e4800 2024-03-24T01:26:28.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.106+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 23 ==== osd_op_reply(51 1.00000000 [omap-get-header out=274b] v0'0 uv14 ondisk = 0) v8 ==== 154+0+274 (crc 0 0 0) 0x55917b9fcc80 con 0x55917b9e4800 2024-03-24T01:26:28.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.106+0000 7f6bae60e280 4 recover_dentries: frag 1.00000000 fnode old v11 vs new v13 2024-03-24T01:26:28.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.106+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.106+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:52 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba17c00 con 0x55917b9e4800 2024-03-24T01:26:28.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.111+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 24 ==== osd_op_reply(52 1.00000000 [omap-set-header] v40'15 uv15 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917b9fcc80 con 0x55917b9e4800 2024-03-24T01:26:28.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.111+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:53 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba16000 con 0x55917b9e4800 2024-03-24T01:26:28.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.111+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 25 ==== osd_op_reply(53 1.00000000 [omap-get-vals-by-keys out=508b] v0'0 uv15 ondisk = 0) v8 ==== 154+0+508 (crc 0 0 0) 0x55917b9fcc80 con 0x55917b9e4800 2024-03-24T01:26:28.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.111+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 1.00000000/linkdir 2024-03-24T01:26:28.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.111+0000 7f6bae60e280 4 recover_dentries: dentry linkdir_head existed already 2024-03-24T01:26:28.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.111+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.111+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 10 vs fullbit version 12 2024-03-24T01:26:28.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.111+0000 7f6bae60e280 4 recover_dentries: writing I dentry linkdir_head into frag 1.00000000 2024-03-24T01:26:28.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.111+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:54 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-vals in=508b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba16400 con 0x55917b9e4800 2024-03-24T01:26:28.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.115+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 26 ==== osd_op_reply(54 1.00000000 [omap-set-vals] v40'16 uv16 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917b9fcc80 con 0x55917b9e4800 2024-03-24T01:26:28.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.115+0000 7f6bae60e280 4 recover_dentries: inspecting lump 100000001f7.00000000 2024-03-24T01:26:28.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.115+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:55 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4000 con 0x55917c439400 2024-03-24T01:26:28.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.116+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 17 ==== osd_op_reply(55 100000001f7.00000000 [omap-get-header out=274b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55917ba1e000 con 0x55917c439400 2024-03-24T01:26:28.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.116+0000 7f6bae60e280 4 recover_dentries: frag 100000001f7.00000000 fnode old v3 vs new v5 2024-03-24T01:26:28.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.116+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.116+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:56 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5400 con 0x55917c439400 2024-03-24T01:26:28.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.121+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 18 ==== osd_op_reply(56 100000001f7.00000000 [omap-set-header] v40'4 uv4 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1e000 con 0x55917c439400 2024-03-24T01:26:28.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.121+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:57 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-get-vals-by-keys in=18b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5800 con 0x55917c439400 2024-03-24T01:26:28.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.122+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 19 ==== osd_op_reply(57 100000001f7.00000000 [omap-get-vals-by-keys out=544b] v0'0 uv4 ondisk = 0) v8 ==== 164+0+544 (crc 0 0 0) 0x55917ba1e000 con 0x55917c439400 2024-03-24T01:26:28.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.122+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 100000001f7.00000000/link0 2024-03-24T01:26:28.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.122+0000 7f6bae60e280 4 recover_dentries: dentry link0_head existed already 2024-03-24T01:26:28.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.122+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.122+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 2 vs fullbit version 4 2024-03-24T01:26:28.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.122+0000 7f6bae60e280 4 recover_dentries: writing I dentry link0_head into frag 100000001f7.00000000 2024-03-24T01:26:28.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.122+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:58 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-set-vals in=506b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4c00 con 0x55917c439400 2024-03-24T01:26:28.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.125+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 20 ==== osd_op_reply(58 100000001f7.00000000 [omap-set-vals] v40'5 uv5 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1e000 con 0x55917c439400 2024-03-24T01:26:28.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.125+0000 7f6bae60e280 4 recover_dentries: inspecting lump 1.00000000 2024-03-24T01:26:28.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.125+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:59 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5000 con 0x55917b9e4800 2024-03-24T01:26:28.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.126+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 27 ==== osd_op_reply(59 1.00000000 [omap-get-header out=274b] v0'0 uv16 ondisk = 0) v8 ==== 154+0+274 (crc 0 0 0) 0x55917ba1e000 con 0x55917b9e4800 2024-03-24T01:26:28.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.127+0000 7f6bae60e280 4 recover_dentries: frag 1.00000000 fnode old v13 vs new v15 2024-03-24T01:26:28.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.127+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.127+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:60 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c438800 con 0x55917b9e4800 2024-03-24T01:26:28.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.131+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 28 ==== osd_op_reply(60 1.00000000 [omap-set-header] v40'17 uv17 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917ba1e000 con 0x55917b9e4800 2024-03-24T01:26:28.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.131+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:61 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c439000 con 0x55917b9e4800 2024-03-24T01:26:28.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.132+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 29 ==== osd_op_reply(61 1.00000000 [omap-get-vals-by-keys out=508b] v0'0 uv17 ondisk = 0) v8 ==== 154+0+508 (crc 0 0 0) 0x55917ba1e000 con 0x55917b9e4800 2024-03-24T01:26:28.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.132+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 1.00000000/linkdir 2024-03-24T01:26:28.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.132+0000 7f6bae60e280 4 recover_dentries: dentry linkdir_head existed already 2024-03-24T01:26:28.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.132+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.132+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 12 vs fullbit version 14 2024-03-24T01:26:28.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.132+0000 7f6bae60e280 4 recover_dentries: writing I dentry linkdir_head into frag 1.00000000 2024-03-24T01:26:28.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.132+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:62 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-vals in=508b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4000 con 0x55917b9e4800 2024-03-24T01:26:28.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.138+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 30 ==== osd_op_reply(62 1.00000000 [omap-set-vals] v40'18 uv18 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917ba1e000 con 0x55917b9e4800 2024-03-24T01:26:28.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.138+0000 7f6bae60e280 4 recover_dentries: inspecting lump 100.00000000 2024-03-24T01:26:28.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.138+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:63 5.13 5:cd5a64a3:::100.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5400 con 0x55917c438c00 2024-03-24T01:26:28.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.138+0000 7f6bac400640 1 -- 172.21.15.90:0/225243987 <== osd.3 v2:172.21.15.90:6808/2999931983 13 ==== osd_op_reply(63 100.00000000 [omap-get-header out=274b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+274 (crc 0 0 0) 0x55917b7e2500 con 0x55917c438c00 2024-03-24T01:26:28.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.138+0000 7f6bae60e280 4 recover_dentries: frag 100.00000000 fnode old v1 vs new v3 2024-03-24T01:26:28.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.138+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.138+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:64 5.13 5:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba16000 con 0x55917c438c00 2024-03-24T01:26:28.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.144+0000 7f6bac400640 1 -- 172.21.15.90:0/225243987 <== osd.3 v2:172.21.15.90:6808/2999931983 14 ==== osd_op_reply(64 100.00000000 [omap-set-header] v40'3 uv3 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55917b7e2500 con 0x55917c438c00 2024-03-24T01:26:28.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.144+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:65 5.13 5:cd5a64a3:::100.00000000:head [omap-get-vals-by-keys in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba16400 con 0x55917c438c00 2024-03-24T01:26:28.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.145+0000 7f6bac400640 1 -- 172.21.15.90:0/225243987 <== osd.3 v2:172.21.15.90:6808/2999931983 15 ==== osd_op_reply(65 100.00000000 [omap-get-vals-by-keys out=523b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+523 (crc 0 0 0) 0x55917b7e2500 con 0x55917c438c00 2024-03-24T01:26:28.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.145+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 100.00000000/stray2 2024-03-24T01:26:28.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.145+0000 7f6bae60e280 4 recover_dentries: dentry stray2_head existed already 2024-03-24T01:26:28.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.145+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.145+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 1 vs fullbit version 2 2024-03-24T01:26:28.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.145+0000 7f6bae60e280 4 recover_dentries: writing I dentry stray2_head into frag 100.00000000 2024-03-24T01:26:28.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.145+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:66 5.13 5:cd5a64a3:::100.00000000:head [omap-set-vals in=507b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba16800 con 0x55917c438c00 2024-03-24T01:26:28.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.149+0000 7f6bac400640 1 -- 172.21.15.90:0/225243987 <== osd.3 v2:172.21.15.90:6808/2999931983 16 ==== osd_op_reply(66 100.00000000 [omap-set-vals] v40'4 uv4 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55917b7e2500 con 0x55917c438c00 2024-03-24T01:26:28.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.149+0000 7f6bae60e280 4 recover_dentries: inspecting lump 602.00000000 2024-03-24T01:26:28.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.149+0000 7f6bae60e280 1 --2- 172.21.15.90:0/225243987 >> [v2:172.21.15.120:6808/1045862942,v1:172.21.15.120:6809/1045862942] conn(0x55917ba16c00 0x55917ba04580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:28.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.149+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6808/1045862942,v1:172.21.15.120:6809/1045862942] -- osd_op(unknown.0.0:67 5.10 5:0bd6d154:::602.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba17000 con 0x55917ba16c00 2024-03-24T01:26:28.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.150+0000 7f6bab000640 1 --2- 172.21.15.90:0/225243987 >> [v2:172.21.15.120:6808/1045862942,v1:172.21.15.120:6809/1045862942] conn(0x55917ba16c00 0x55917ba04580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:28.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.151+0000 7f6bab000640 1 --2- 172.21.15.90:0/225243987 >> [v2:172.21.15.120:6808/1045862942,v1:172.21.15.120:6809/1045862942] conn(0x55917ba16c00 0x55917ba04580 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:28.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.152+0000 7f6bab000640 1 -- 172.21.15.90:0/225243987 <== osd.7 v2:172.21.15.120:6808/1045862942 1 ==== osd_op_reply(67 602.00000000 [omap-get-header out=274b] v0'0 uv7 ondisk = 0) v8 ==== 156+0+274 (crc 0 0 0) 0x55917b7e2000 con 0x55917ba16c00 2024-03-24T01:26:28.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.152+0000 7f6bae60e280 4 recover_dentries: frag 602.00000000 fnode old v1 vs new v6 2024-03-24T01:26:28.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.152+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.152+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6808/1045862942,v1:172.21.15.120:6809/1045862942] -- osd_op(unknown.0.0:68 5.10 5:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba17400 con 0x55917ba16c00 2024-03-24T01:26:28.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.158+0000 7f6bab000640 1 -- 172.21.15.90:0/225243987 <== osd.7 v2:172.21.15.120:6808/1045862942 2 ==== osd_op_reply(68 602.00000000 [omap-set-header] v40'9 uv9 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55917b7e2000 con 0x55917ba16c00 2024-03-24T01:26:28.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.158+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6808/1045862942,v1:172.21.15.120:6809/1045862942] -- osd_op(unknown.0.0:69 5.10 5:0bd6d154:::602.00000000:head [omap-get-vals-by-keys in=24b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba17800 con 0x55917ba16c00 2024-03-24T01:26:28.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.158+0000 7f6bab000640 1 -- 172.21.15.90:0/225243987 <== osd.7 v2:172.21.15.120:6808/1045862942 3 ==== osd_op_reply(69 602.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv9 ondisk = 0) v8 ==== 156+0+4 (crc 0 0 0) 0x55917b7e2000 con 0x55917ba16c00 2024-03-24T01:26:28.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.158+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 602.00000000/100000001f8 2024-03-24T01:26:28.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.158+0000 7f6bae60e280 4 recover_dentries: dentry did not already exist, will create 2024-03-24T01:26:28.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.158+0000 7f6bae60e280 4 recover_dentries: writing I dentry 100000001f8_head into frag 602.00000000 2024-03-24T01:26:28.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.158+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6808/1045862942,v1:172.21.15.120:6809/1045862942] -- osd_op(unknown.0.0:70 5.10 5:0bd6d154:::602.00000000:head [omap-set-vals in=526b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba17c00 con 0x55917ba16c00 2024-03-24T01:26:28.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.162+0000 7f6bab000640 1 -- 172.21.15.90:0/225243987 <== osd.7 v2:172.21.15.120:6808/1045862942 4 ==== osd_op_reply(70 602.00000000 [omap-set-vals] v40'10 uv10 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55917b7e2000 con 0x55917ba16c00 2024-03-24T01:26:28.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.162+0000 7f6bae60e280 4 recover_dentries: inspecting lump 100000001f7.00000000 2024-03-24T01:26:28.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.162+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:71 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba16000 con 0x55917c439400 2024-03-24T01:26:28.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.164+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 21 ==== osd_op_reply(71 100000001f7.00000000 [omap-get-header out=274b] v0'0 uv5 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.164+0000 7f6bae60e280 4 recover_dentries: frag 100000001f7.00000000 fnode old v5 vs new v7 2024-03-24T01:26:28.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.164+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.164+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:72 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba16400 con 0x55917c439400 2024-03-24T01:26:28.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.167+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 22 ==== osd_op_reply(72 100000001f7.00000000 [omap-set-header] v40'6 uv6 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.167+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:73 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-get-vals-by-keys in=18b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5c00 con 0x55917c439400 2024-03-24T01:26:28.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.168+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 23 ==== osd_op_reply(73 100000001f7.00000000 [omap-get-vals-by-keys out=506b] v0'0 uv6 ondisk = 0) v8 ==== 164+0+506 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.168+0000 7f6bae60e280 4 recover_dentries: inspecting nullbit 100000001f7.00000000/link0 2024-03-24T01:26:28.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.168+0000 7f6bae60e280 4 recover_dentries: dentry exists, will remove 2024-03-24T01:26:28.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.168+0000 7f6bae60e280 10 recover_dentries: Existing full inode in slot to be (maybe) removed by null journal dn 'link0' with lump fnode version 7vs existing fnode version 5 2024-03-24T01:26:28.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.168+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:74 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-rm-keys in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4400 con 0x55917c439400 2024-03-24T01:26:28.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.171+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 24 ==== osd_op_reply(74 100000001f7.00000000 [omap-rm-keys] v40'7 uv7 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.171+0000 7f6bae60e280 4 recover_dentries: inspecting lump 1.00000000 2024-03-24T01:26:28.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.172+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:75 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5400 con 0x55917b9e4800 2024-03-24T01:26:28.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.174+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 31 ==== osd_op_reply(75 1.00000000 [omap-get-header out=274b] v0'0 uv18 ondisk = 0) v8 ==== 154+0+274 (crc 0 0 0) 0x55917ba1e280 con 0x55917b9e4800 2024-03-24T01:26:28.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.174+0000 7f6bae60e280 4 recover_dentries: frag 1.00000000 fnode old v15 vs new v16 2024-03-24T01:26:28.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.174+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.174+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:76 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c52e800 con 0x55917b9e4800 2024-03-24T01:26:28.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.177+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 32 ==== osd_op_reply(76 1.00000000 [omap-set-header] v40'19 uv19 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917b9e4800 2024-03-24T01:26:28.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.177+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:77 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5800 con 0x55917b9e4800 2024-03-24T01:26:28.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.179+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 33 ==== osd_op_reply(77 1.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv19 ondisk = 0) v8 ==== 154+0+4 (crc 0 0 0) 0x55917ba1e280 con 0x55917b9e4800 2024-03-24T01:26:28.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.179+0000 7f6bae60e280 4 recover_dentries: updating root 0x0x1 2024-03-24T01:26:28.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.179+0000 7f6bae60e280 4 recover_dentries: object id 1.00000000.inode 2024-03-24T01:26:28.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.179+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:78 5.14 5:287034c4:::1.00000000.inode:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4c00 con 0x55917c438c00 2024-03-24T01:26:28.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.179+0000 7f6bac400640 1 -- 172.21.15.90:0/225243987 <== osd.3 v2:172.21.15.90:6808/2999931983 17 ==== osd_op_reply(78 1.00000000.inode [read 0~582 out=582b] v0'0 uv33 ondisk = 0) v8 ==== 160+0+582 (crc 0 0 0) 0x55917b7e2500 con 0x55917c438c00 2024-03-24T01:26:28.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.179+0000 7f6bae60e280 4 recover_dentries: root exists, will modify (582) 2024-03-24T01:26:28.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.179+0000 7f6bae60e280 4 recover_dentries: magic ok 2024-03-24T01:26:28.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.179+0000 7f6bae60e280 4 recover_dentries: writing root ino 1.00000000.inode version 6 2024-03-24T01:26:28.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.179+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:79 5.14 5:287034c4:::1.00000000.inode:head [writefull 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5000 con 0x55917c438c00 2024-03-24T01:26:28.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.183+0000 7f6bac400640 1 -- 172.21.15.90:0/225243987 <== osd.3 v2:172.21.15.90:6808/2999931983 18 ==== osd_op_reply(79 1.00000000.inode [writefull 0~582] v40'34 uv34 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x55917b7e2500 con 0x55917c438c00 2024-03-24T01:26:28.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.183+0000 7f6bae60e280 4 recover_dentries: inspecting lump 1.00000000 2024-03-24T01:26:28.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.183+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:80 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4000 con 0x55917b9e4800 2024-03-24T01:26:28.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.185+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 34 ==== osd_op_reply(80 1.00000000 [omap-get-header out=274b] v0'0 uv19 ondisk = 0) v8 ==== 154+0+274 (crc 0 0 0) 0x55917ba1e280 con 0x55917b9e4800 2024-03-24T01:26:28.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.186+0000 7f6bae60e280 4 recover_dentries: frag 1.00000000 fnode old v16 vs new v18 2024-03-24T01:26:28.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.186+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.186+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:81 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5c00 con 0x55917b9e4800 2024-03-24T01:26:28.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.190+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 35 ==== osd_op_reply(81 1.00000000 [omap-set-header] v40'20 uv20 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917b9e4800 2024-03-24T01:26:28.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.190+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:82 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4400 con 0x55917b9e4800 2024-03-24T01:26:28.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.190+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 36 ==== osd_op_reply(82 1.00000000 [omap-get-vals-by-keys out=507b] v0'0 uv20 ondisk = 0) v8 ==== 154+0+507 (crc 0 0 0) 0x55917ba1e280 con 0x55917b9e4800 2024-03-24T01:26:28.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.190+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 1.00000000/subdir 2024-03-24T01:26:28.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.190+0000 7f6bae60e280 4 recover_dentries: dentry subdir_head existed already 2024-03-24T01:26:28.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.190+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.190+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 6 vs fullbit version 17 2024-03-24T01:26:28.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.190+0000 7f6bae60e280 4 recover_dentries: writing I dentry subdir_head into frag 1.00000000 2024-03-24T01:26:28.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.190+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:83 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-vals in=507b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5400 con 0x55917b9e4800 2024-03-24T01:26:28.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.194+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 37 ==== osd_op_reply(83 1.00000000 [omap-set-vals] v40'21 uv21 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917b9e4800 2024-03-24T01:26:28.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.194+0000 7f6bae60e280 4 recover_dentries: inspecting lump 10000000001.00000000 2024-03-24T01:26:28.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.194+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:84 5.7 5:e2e07930:::10000000001.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba16000 con 0x55917c439400 2024-03-24T01:26:28.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.194+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 25 ==== osd_op_reply(84 10000000001.00000000 [omap-get-header out=274b] v0'0 uv5 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.194+0000 7f6bae60e280 4 recover_dentries: frag 10000000001.00000000 fnode old v3 vs new v5 2024-03-24T01:26:28.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.194+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.194+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:85 5.7 5:e2e07930:::10000000001.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba16400 con 0x55917c439400 2024-03-24T01:26:28.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.200+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 26 ==== osd_op_reply(85 10000000001.00000000 [omap-set-header] v40'6 uv6 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.200+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:86 5.7 5:e2e07930:::10000000001.00000000:head [omap-get-vals-by-keys in=23b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c438800 con 0x55917c439400 2024-03-24T01:26:28.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.201+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 27 ==== osd_op_reply(86 10000000001.00000000 [omap-get-vals-by-keys out=549b] v0'0 uv6 ondisk = 0) v8 ==== 164+0+549 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.201+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 10000000001.00000000/subdirfile 2024-03-24T01:26:28.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.201+0000 7f6bae60e280 4 recover_dentries: dentry subdirfile_head existed already 2024-03-24T01:26:28.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.201+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.201+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 2 vs fullbit version 4 2024-03-24T01:26:28.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.201+0000 7f6bae60e280 4 recover_dentries: writing I dentry subdirfile_head into frag 10000000001.00000000 2024-03-24T01:26:28.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.201+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:87 5.7 5:e2e07930:::10000000001.00000000:head [omap-set-vals in=511b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c439000 con 0x55917c439400 2024-03-24T01:26:28.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.204+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 28 ==== osd_op_reply(87 10000000001.00000000 [omap-set-vals] v40'7 uv7 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.204+0000 7f6bae60e280 4 recover_dentries: inspecting lump 1.00000000 2024-03-24T01:26:28.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.204+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:88 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba38000 con 0x55917b9e4800 2024-03-24T01:26:28.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.205+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 38 ==== osd_op_reply(88 1.00000000 [omap-get-header out=274b] v0'0 uv21 ondisk = 0) v8 ==== 154+0+274 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.205+0000 7f6bae60e280 4 recover_dentries: frag 1.00000000 fnode old v18 vs new v22 2024-03-24T01:26:28.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.205+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.205+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:89 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba38400 con 0x55917b9e4800 2024-03-24T01:26:28.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.209+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 39 ==== osd_op_reply(89 1.00000000 [omap-set-header] v40'22 uv22 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.209+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:90 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=35b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba38800 con 0x55917b9e4800 2024-03-24T01:26:28.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.209+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 40 ==== osd_op_reply(90 1.00000000 [omap-get-vals-by-keys out=1011b] v0'0 uv22 ondisk = 0) v8 ==== 154+0+1011 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.210+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 1.00000000/linkdir 2024-03-24T01:26:28.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.210+0000 7f6bae60e280 4 recover_dentries: dentry linkdir_head existed already 2024-03-24T01:26:28.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.210+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.210+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 14 vs fullbit version 19 2024-03-24T01:26:28.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.210+0000 7f6bae60e280 4 recover_dentries: writing I dentry linkdir_head into frag 1.00000000 2024-03-24T01:26:28.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.210+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 1.00000000/subdir 2024-03-24T01:26:28.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.210+0000 7f6bae60e280 4 recover_dentries: dentry subdir_head existed already 2024-03-24T01:26:28.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.210+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.210+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 17 vs fullbit version 21 2024-03-24T01:26:28.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.210+0000 7f6bae60e280 4 recover_dentries: writing I dentry subdir_head into frag 1.00000000 2024-03-24T01:26:28.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.210+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:91 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-vals in=1011b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba38c00 con 0x55917b9e4800 2024-03-24T01:26:28.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.214+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 41 ==== osd_op_reply(91 1.00000000 [omap-set-vals] v40'23 uv23 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.214+0000 7f6bae60e280 4 recover_dentries: inspecting lump 100000001f7.00000000 2024-03-24T01:26:28.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.214+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:92 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba39000 con 0x55917c439400 2024-03-24T01:26:28.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.216+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 29 ==== osd_op_reply(92 100000001f7.00000000 [omap-get-header out=274b] v0'0 uv7 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.216+0000 7f6bae60e280 4 recover_dentries: frag 100000001f7.00000000 fnode old v7 vs new v9 2024-03-24T01:26:28.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.216+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.216+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:93 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba39400 con 0x55917c439400 2024-03-24T01:26:28.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.219+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 30 ==== osd_op_reply(93 100000001f7.00000000 [omap-set-header] v40'8 uv8 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.219+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:94 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-get-vals-by-keys in=18b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba39800 con 0x55917c439400 2024-03-24T01:26:28.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.220+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 31 ==== osd_op_reply(94 100000001f7.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv8 ondisk = 0) v8 ==== 164+0+4 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.220+0000 7f6bae60e280 4 recover_dentries: inspecting remotebit 100000001f7.00000000/link0 2024-03-24T01:26:28.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.220+0000 7f6bae60e280 4 recover_dentries: dentry did not already exist, will create 2024-03-24T01:26:28.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.220+0000 7f6bae60e280 4 recover_dentries: writing L dentry link0_head into frag 100000001f7.00000000 2024-03-24T01:26:28.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.220+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:95 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-set-vals in=40b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba39c00 con 0x55917c439400 2024-03-24T01:26:28.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.224+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 32 ==== osd_op_reply(95 100000001f7.00000000 [omap-set-vals] v40'9 uv9 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.224+0000 7f6bae60e280 4 recover_dentries: inspecting lump 10000000001.00000000 2024-03-24T01:26:28.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.224+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:96 5.7 5:e2e07930:::10000000001.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba38000 con 0x55917c439400 2024-03-24T01:26:28.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.226+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 33 ==== osd_op_reply(96 10000000001.00000000 [omap-get-header out=274b] v0'0 uv7 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.226+0000 7f6bae60e280 4 recover_dentries: frag 10000000001.00000000 fnode old v5 vs new v7 2024-03-24T01:26:28.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.226+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.226+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:97 5.7 5:e2e07930:::10000000001.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba38400 con 0x55917c439400 2024-03-24T01:26:28.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.228+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 34 ==== osd_op_reply(97 10000000001.00000000 [omap-set-header] v40'8 uv8 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.228+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:98 5.7 5:e2e07930:::10000000001.00000000:head [omap-get-vals-by-keys in=23b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c438800 con 0x55917c439400 2024-03-24T01:26:28.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.229+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 35 ==== osd_op_reply(98 10000000001.00000000 [omap-get-vals-by-keys out=511b] v0'0 uv8 ondisk = 0) v8 ==== 164+0+511 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.229+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 10000000001.00000000/subdirfile 2024-03-24T01:26:28.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.229+0000 7f6bae60e280 4 recover_dentries: dentry subdirfile_head existed already 2024-03-24T01:26:28.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.229+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.229+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 4 vs fullbit version 6 2024-03-24T01:26:28.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.229+0000 7f6bae60e280 4 recover_dentries: writing I dentry subdirfile_head into frag 10000000001.00000000 2024-03-24T01:26:28.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.229+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:99 5.7 5:e2e07930:::10000000001.00000000:head [omap-set-vals in=589b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c439000 con 0x55917c439400 2024-03-24T01:26:28.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.233+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 36 ==== osd_op_reply(99 10000000001.00000000 [omap-set-vals] v40'9 uv9 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.234+0000 7f6bae60e280 4 recover_dentries: inspecting lump 1.00000000 2024-03-24T01:26:28.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.234+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:100 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba16000 con 0x55917b9e4800 2024-03-24T01:26:28.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.235+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 42 ==== osd_op_reply(100 1.00000000 [omap-get-header out=274b] v0'0 uv23 ondisk = 0) v8 ==== 154+0+274 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.235+0000 7f6bae60e280 4 recover_dentries: frag 1.00000000 fnode old v22 vs new v24 2024-03-24T01:26:28.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.235+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.235+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:101 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba16400 con 0x55917b9e4800 2024-03-24T01:26:28.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.239+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 43 ==== osd_op_reply(101 1.00000000 [omap-set-header] v40'24 uv24 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.239+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:102 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5800 con 0x55917b9e4800 2024-03-24T01:26:28.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.239+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 44 ==== osd_op_reply(102 1.00000000 [omap-get-vals-by-keys out=507b] v0'0 uv24 ondisk = 0) v8 ==== 154+0+507 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.239+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 1.00000000/subdir 2024-03-24T01:26:28.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.239+0000 7f6bae60e280 4 recover_dentries: dentry subdir_head existed already 2024-03-24T01:26:28.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.239+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.239+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 21 vs fullbit version 23 2024-03-24T01:26:28.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.239+0000 7f6bae60e280 4 recover_dentries: writing I dentry subdir_head into frag 1.00000000 2024-03-24T01:26:28.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.239+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:103 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-vals in=507b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4000 con 0x55917b9e4800 2024-03-24T01:26:28.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.243+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 45 ==== osd_op_reply(103 1.00000000 [omap-set-vals] v40'25 uv25 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.243+0000 7f6bae60e280 4 recover_dentries: inspecting lump 10000000001.00000000 2024-03-24T01:26:28.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.243+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:104 5.7 5:e2e07930:::10000000001.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5c00 con 0x55917c439400 2024-03-24T01:26:28.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.244+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 37 ==== osd_op_reply(104 10000000001.00000000 [omap-get-header out=274b] v0'0 uv9 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.245+0000 7f6bae60e280 4 recover_dentries: frag 10000000001.00000000 fnode old v7 vs new v9 2024-03-24T01:26:28.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.245+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.245+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:105 5.7 5:e2e07930:::10000000001.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4400 con 0x55917c439400 2024-03-24T01:26:28.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.248+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 38 ==== osd_op_reply(105 10000000001.00000000 [omap-set-header] v40'10 uv10 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.249+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:106 5.7 5:e2e07930:::10000000001.00000000:head [omap-get-vals-by-keys in=23b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5400 con 0x55917c439400 2024-03-24T01:26:28.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.250+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 39 ==== osd_op_reply(106 10000000001.00000000 [omap-get-vals-by-keys out=589b] v0'0 uv10 ondisk = 0) v8 ==== 164+0+589 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.250+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 10000000001.00000000/subdirfile 2024-03-24T01:26:28.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.250+0000 7f6bae60e280 4 recover_dentries: dentry subdirfile_head existed already 2024-03-24T01:26:28.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.250+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.250+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 6 vs fullbit version 8 2024-03-24T01:26:28.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.250+0000 7f6bae60e280 4 recover_dentries: writing I dentry subdirfile_head into frag 10000000001.00000000 2024-03-24T01:26:28.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.250+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:107 5.7 5:e2e07930:::10000000001.00000000:head [omap-set-vals in=627b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c52e800 con 0x55917c439400 2024-03-24T01:26:28.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.253+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 40 ==== osd_op_reply(107 10000000001.00000000 [omap-set-vals] v40'11 uv11 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.254+0000 7f6bae60e280 4 recover_dentries: inspecting lump 1.00000000 2024-03-24T01:26:28.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.254+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:108 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5800 con 0x55917b9e4800 2024-03-24T01:26:28.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.256+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 46 ==== osd_op_reply(108 1.00000000 [omap-get-header out=274b] v0'0 uv25 ondisk = 0) v8 ==== 154+0+274 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.256+0000 7f6bae60e280 4 recover_dentries: frag 1.00000000 fnode old v24 vs new v26 2024-03-24T01:26:28.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.256+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.256+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:109 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba16000 con 0x55917b9e4800 2024-03-24T01:26:28.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.259+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 47 ==== osd_op_reply(109 1.00000000 [omap-set-header] v40'26 uv26 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.259+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:110 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba16400 con 0x55917b9e4800 2024-03-24T01:26:28.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.259+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 48 ==== osd_op_reply(110 1.00000000 [omap-get-vals-by-keys out=507b] v0'0 uv26 ondisk = 0) v8 ==== 154+0+507 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.259+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 1.00000000/subdir 2024-03-24T01:26:28.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.259+0000 7f6bae60e280 4 recover_dentries: dentry subdir_head existed already 2024-03-24T01:26:28.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.259+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.259+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 23 vs fullbit version 25 2024-03-24T01:26:28.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.259+0000 7f6bae60e280 4 recover_dentries: writing I dentry subdir_head into frag 1.00000000 2024-03-24T01:26:28.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.259+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:111 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-vals in=507b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c438800 con 0x55917b9e4800 2024-03-24T01:26:28.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.264+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 49 ==== osd_op_reply(111 1.00000000 [omap-set-vals] v40'27 uv27 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.264+0000 7f6bae60e280 4 recover_dentries: inspecting lump 10000000001.00000000 2024-03-24T01:26:28.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.264+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:112 5.7 5:e2e07930:::10000000001.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c439000 con 0x55917c439400 2024-03-24T01:26:28.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.264+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 41 ==== osd_op_reply(112 10000000001.00000000 [omap-get-header out=274b] v0'0 uv11 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.264+0000 7f6bae60e280 4 recover_dentries: frag 10000000001.00000000 fnode old v9 vs new v11 2024-03-24T01:26:28.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.264+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.264+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:113 5.7 5:e2e07930:::10000000001.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba38000 con 0x55917c439400 2024-03-24T01:26:28.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.269+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 42 ==== osd_op_reply(113 10000000001.00000000 [omap-set-header] v40'12 uv12 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.269+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:114 5.7 5:e2e07930:::10000000001.00000000:head [omap-get-vals-by-keys in=23b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba38400 con 0x55917c439400 2024-03-24T01:26:28.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.271+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 43 ==== osd_op_reply(114 10000000001.00000000 [omap-get-vals-by-keys out=627b] v0'0 uv12 ondisk = 0) v8 ==== 164+0+627 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.271+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 10000000001.00000000/subdirfile 2024-03-24T01:26:28.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.271+0000 7f6bae60e280 4 recover_dentries: dentry subdirfile_head existed already 2024-03-24T01:26:28.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.271+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.271+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 8 vs fullbit version 10 2024-03-24T01:26:28.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.271+0000 7f6bae60e280 4 recover_dentries: writing I dentry subdirfile_head into frag 10000000001.00000000 2024-03-24T01:26:28.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.271+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:115 5.7 5:e2e07930:::10000000001.00000000:head [omap-set-vals in=589b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba38800 con 0x55917c439400 2024-03-24T01:26:28.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.273+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 44 ==== osd_op_reply(115 10000000001.00000000 [omap-set-vals] v40'13 uv13 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.273+0000 7f6bae60e280 4 recover_dentries: inspecting lump 1.00000000 2024-03-24T01:26:28.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.273+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:116 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba38c00 con 0x55917b9e4800 2024-03-24T01:26:28.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.275+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 50 ==== osd_op_reply(116 1.00000000 [omap-get-header out=274b] v0'0 uv27 ondisk = 0) v8 ==== 154+0+274 (crc 0 0 0) 0x55917ba1e280 con 0x55917b9e4800 2024-03-24T01:26:28.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.275+0000 7f6bae60e280 4 recover_dentries: frag 1.00000000 fnode old v26 vs new v30 2024-03-24T01:26:28.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.275+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.275+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:117 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba39000 con 0x55917b9e4800 2024-03-24T01:26:28.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.279+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 51 ==== osd_op_reply(117 1.00000000 [omap-set-header] v40'28 uv28 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917b9e4800 2024-03-24T01:26:28.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.279+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:118 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=35b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba39400 con 0x55917b9e4800 2024-03-24T01:26:28.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.280+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 52 ==== osd_op_reply(118 1.00000000 [omap-get-vals-by-keys out=1011b] v0'0 uv28 ondisk = 0) v8 ==== 154+0+1011 (crc 0 0 0) 0x55917ba1e280 con 0x55917b9e4800 2024-03-24T01:26:28.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.280+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 1.00000000/linkdir 2024-03-24T01:26:28.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.280+0000 7f6bae60e280 4 recover_dentries: dentry linkdir_head existed already 2024-03-24T01:26:28.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.280+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.280+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 19 vs fullbit version 27 2024-03-24T01:26:28.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.280+0000 7f6bae60e280 4 recover_dentries: writing I dentry linkdir_head into frag 1.00000000 2024-03-24T01:26:28.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.280+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 1.00000000/subdir 2024-03-24T01:26:28.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.280+0000 7f6bae60e280 4 recover_dentries: dentry subdir_head existed already 2024-03-24T01:26:28.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.280+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.280+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 25 vs fullbit version 29 2024-03-24T01:26:28.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.280+0000 7f6bae60e280 4 recover_dentries: writing I dentry subdir_head into frag 1.00000000 2024-03-24T01:26:28.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.280+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:119 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-vals in=1011b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba39800 con 0x55917b9e4800 2024-03-24T01:26:28.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.284+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 53 ==== osd_op_reply(119 1.00000000 [omap-set-vals] v40'29 uv29 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917b9e4800 2024-03-24T01:26:28.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.284+0000 7f6bae60e280 4 recover_dentries: inspecting lump 100000001f7.00000000 2024-03-24T01:26:28.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.284+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:120 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba39c00 con 0x55917c439400 2024-03-24T01:26:28.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.286+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 45 ==== osd_op_reply(120 100000001f7.00000000 [omap-get-header out=274b] v0'0 uv9 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.286+0000 7f6bae60e280 4 recover_dentries: frag 100000001f7.00000000 fnode old v9 vs new v11 2024-03-24T01:26:28.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.286+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.286+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:121 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba38000 con 0x55917c439400 2024-03-24T01:26:28.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.289+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 46 ==== osd_op_reply(121 100000001f7.00000000 [omap-set-header] v40'10 uv10 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.289+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:122 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-get-vals-by-keys in=18b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba38400 con 0x55917c439400 2024-03-24T01:26:28.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.291+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 47 ==== osd_op_reply(122 100000001f7.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv10 ondisk = 0) v8 ==== 164+0+4 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.291+0000 7f6bae60e280 4 recover_dentries: inspecting remotebit 100000001f7.00000000/link1 2024-03-24T01:26:28.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.291+0000 7f6bae60e280 4 recover_dentries: dentry did not already exist, will create 2024-03-24T01:26:28.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.291+0000 7f6bae60e280 4 recover_dentries: writing L dentry link1_head into frag 100000001f7.00000000 2024-03-24T01:26:28.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.291+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:123 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-set-vals in=40b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba38800 con 0x55917c439400 2024-03-24T01:26:28.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.295+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 48 ==== osd_op_reply(123 100000001f7.00000000 [omap-set-vals] v40'11 uv11 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.295+0000 7f6bae60e280 4 recover_dentries: inspecting lump 10000000001.00000000 2024-03-24T01:26:28.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.295+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:124 5.7 5:e2e07930:::10000000001.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba38c00 con 0x55917c439400 2024-03-24T01:26:28.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.296+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 49 ==== osd_op_reply(124 10000000001.00000000 [omap-get-header out=274b] v0'0 uv13 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.296+0000 7f6bae60e280 4 recover_dentries: frag 10000000001.00000000 fnode old v11 vs new v13 2024-03-24T01:26:28.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.296+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.296+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:125 5.7 5:e2e07930:::10000000001.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c438800 con 0x55917c439400 2024-03-24T01:26:28.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.299+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 50 ==== osd_op_reply(125 10000000001.00000000 [omap-set-header] v40'14 uv14 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.299+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:126 5.7 5:e2e07930:::10000000001.00000000:head [omap-get-vals-by-keys in=23b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c439000 con 0x55917c439400 2024-03-24T01:26:28.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.300+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 51 ==== osd_op_reply(126 10000000001.00000000 [omap-get-vals-by-keys out=589b] v0'0 uv14 ondisk = 0) v8 ==== 164+0+589 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.300+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 10000000001.00000000/subdirfile 2024-03-24T01:26:28.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.300+0000 7f6bae60e280 4 recover_dentries: dentry subdirfile_head existed already 2024-03-24T01:26:28.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.300+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.300+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 10 vs fullbit version 12 2024-03-24T01:26:28.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.300+0000 7f6bae60e280 4 recover_dentries: writing I dentry subdirfile_head into frag 10000000001.00000000 2024-03-24T01:26:28.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.301+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:127 5.7 5:e2e07930:::10000000001.00000000:head [omap-set-vals in=589b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba16000 con 0x55917c439400 2024-03-24T01:26:28.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.305+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 52 ==== osd_op_reply(127 10000000001.00000000 [omap-set-vals] v40'15 uv15 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.305+0000 7f6bae60e280 4 recover_dentries: inspecting lump 1.00000000 2024-03-24T01:26:28.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.305+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:128 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba16400 con 0x55917b9e4800 2024-03-24T01:26:28.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.306+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 54 ==== osd_op_reply(128 1.00000000 [omap-get-header out=274b] v0'0 uv29 ondisk = 0) v8 ==== 154+0+274 (crc 0 0 0) 0x55917ba1e280 con 0x55917b9e4800 2024-03-24T01:26:28.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.306+0000 7f6bae60e280 4 recover_dentries: frag 1.00000000 fnode old v30 vs new v34 2024-03-24T01:26:28.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.306+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.306+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:129 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4000 con 0x55917b9e4800 2024-03-24T01:26:28.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.310+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 55 ==== osd_op_reply(129 1.00000000 [omap-set-header] v40'30 uv30 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917b9e4800 2024-03-24T01:26:28.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.310+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:130 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=35b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5c00 con 0x55917b9e4800 2024-03-24T01:26:28.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.310+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 56 ==== osd_op_reply(130 1.00000000 [omap-get-vals-by-keys out=1011b] v0'0 uv30 ondisk = 0) v8 ==== 154+0+1011 (crc 0 0 0) 0x55917ba1e280 con 0x55917b9e4800 2024-03-24T01:26:28.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.310+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 1.00000000/linkdir 2024-03-24T01:26:28.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.310+0000 7f6bae60e280 4 recover_dentries: dentry linkdir_head existed already 2024-03-24T01:26:28.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.310+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.310+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 27 vs fullbit version 31 2024-03-24T01:26:28.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.310+0000 7f6bae60e280 4 recover_dentries: writing I dentry linkdir_head into frag 1.00000000 2024-03-24T01:26:28.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.310+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 1.00000000/subdir 2024-03-24T01:26:28.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.310+0000 7f6bae60e280 4 recover_dentries: dentry subdir_head existed already 2024-03-24T01:26:28.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.310+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.310+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 29 vs fullbit version 33 2024-03-24T01:26:28.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.310+0000 7f6bae60e280 4 recover_dentries: writing I dentry subdir_head into frag 1.00000000 2024-03-24T01:26:28.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.310+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:131 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-vals in=1011b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4400 con 0x55917b9e4800 2024-03-24T01:26:28.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.315+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 57 ==== osd_op_reply(131 1.00000000 [omap-set-vals] v40'31 uv31 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917b9e4800 2024-03-24T01:26:28.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.315+0000 7f6bae60e280 4 recover_dentries: inspecting lump 100000001f7.00000000 2024-03-24T01:26:28.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.315+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:132 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5400 con 0x55917c439400 2024-03-24T01:26:28.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.317+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 53 ==== osd_op_reply(132 100000001f7.00000000 [omap-get-header out=274b] v0'0 uv11 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.317+0000 7f6bae60e280 4 recover_dentries: frag 100000001f7.00000000 fnode old v11 vs new v13 2024-03-24T01:26:28.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.317+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.317+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:133 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5800 con 0x55917c439400 2024-03-24T01:26:28.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.319+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 54 ==== osd_op_reply(133 100000001f7.00000000 [omap-set-header] v40'12 uv12 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.320+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:134 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-get-vals-by-keys in=18b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c52e800 con 0x55917c439400 2024-03-24T01:26:28.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.321+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 55 ==== osd_op_reply(134 100000001f7.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv12 ondisk = 0) v8 ==== 164+0+4 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.321+0000 7f6bae60e280 4 recover_dentries: inspecting remotebit 100000001f7.00000000/link2 2024-03-24T01:26:28.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.321+0000 7f6bae60e280 4 recover_dentries: dentry did not already exist, will create 2024-03-24T01:26:28.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.321+0000 7f6bae60e280 4 recover_dentries: writing L dentry link2_head into frag 100000001f7.00000000 2024-03-24T01:26:28.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.321+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:135 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-set-vals in=40b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4000 con 0x55917c439400 2024-03-24T01:26:28.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.326+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 56 ==== osd_op_reply(135 100000001f7.00000000 [omap-set-vals] v40'13 uv13 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.326+0000 7f6bae60e280 4 recover_dentries: inspecting lump 10000000001.00000000 2024-03-24T01:26:28.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.326+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:136 5.7 5:e2e07930:::10000000001.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5c00 con 0x55917c439400 2024-03-24T01:26:28.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.326+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 57 ==== osd_op_reply(136 10000000001.00000000 [omap-get-header out=274b] v0'0 uv15 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.326+0000 7f6bae60e280 4 recover_dentries: frag 10000000001.00000000 fnode old v13 vs new v15 2024-03-24T01:26:28.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.326+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.326+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:137 5.7 5:e2e07930:::10000000001.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4400 con 0x55917c439400 2024-03-24T01:26:28.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.330+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 58 ==== osd_op_reply(137 10000000001.00000000 [omap-set-header] v40'16 uv16 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.330+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:138 5.7 5:e2e07930:::10000000001.00000000:head [omap-get-vals-by-keys in=23b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba16000 con 0x55917c439400 2024-03-24T01:26:28.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.331+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 59 ==== osd_op_reply(138 10000000001.00000000 [omap-get-vals-by-keys out=589b] v0'0 uv16 ondisk = 0) v8 ==== 164+0+589 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.331+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 10000000001.00000000/subdirfile 2024-03-24T01:26:28.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.331+0000 7f6bae60e280 4 recover_dentries: dentry subdirfile_head existed already 2024-03-24T01:26:28.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.331+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.331+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 12 vs fullbit version 14 2024-03-24T01:26:28.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.331+0000 7f6bae60e280 4 recover_dentries: writing I dentry subdirfile_head into frag 10000000001.00000000 2024-03-24T01:26:28.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.331+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:139 5.7 5:e2e07930:::10000000001.00000000:head [omap-set-vals in=589b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba16400 con 0x55917c439400 2024-03-24T01:26:28.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.336+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 60 ==== osd_op_reply(139 10000000001.00000000 [omap-set-vals] v40'17 uv17 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.336+0000 7f6bae60e280 4 recover_dentries: inspecting lump 1.00000000 2024-03-24T01:26:28.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.336+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:140 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c438800 con 0x55917b9e4800 2024-03-24T01:26:28.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.336+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 58 ==== osd_op_reply(140 1.00000000 [omap-get-header out=274b] v0'0 uv31 ondisk = 0) v8 ==== 154+0+274 (crc 0 0 0) 0x55917ba1e280 con 0x55917b9e4800 2024-03-24T01:26:28.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.336+0000 7f6bae60e280 4 recover_dentries: frag 1.00000000 fnode old v34 vs new v38 2024-03-24T01:26:28.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.336+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.336+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:141 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c439000 con 0x55917b9e4800 2024-03-24T01:26:28.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.341+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 59 ==== osd_op_reply(141 1.00000000 [omap-set-header] v40'32 uv32 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917b9e4800 2024-03-24T01:26:28.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.341+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:142 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=35b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba58000 con 0x55917b9e4800 2024-03-24T01:26:28.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.342+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 60 ==== osd_op_reply(142 1.00000000 [omap-get-vals-by-keys out=1011b] v0'0 uv32 ondisk = 0) v8 ==== 154+0+1011 (crc 0 0 0) 0x55917ba1e280 con 0x55917b9e4800 2024-03-24T01:26:28.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.342+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 1.00000000/linkdir 2024-03-24T01:26:28.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.342+0000 7f6bae60e280 4 recover_dentries: dentry linkdir_head existed already 2024-03-24T01:26:28.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.342+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.342+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 31 vs fullbit version 35 2024-03-24T01:26:28.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.342+0000 7f6bae60e280 4 recover_dentries: writing I dentry linkdir_head into frag 1.00000000 2024-03-24T01:26:28.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.342+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 1.00000000/subdir 2024-03-24T01:26:28.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.342+0000 7f6bae60e280 4 recover_dentries: dentry subdir_head existed already 2024-03-24T01:26:28.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.342+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.342+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 33 vs fullbit version 37 2024-03-24T01:26:28.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.342+0000 7f6bae60e280 4 recover_dentries: writing I dentry subdir_head into frag 1.00000000 2024-03-24T01:26:28.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.342+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:143 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-vals in=1011b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba58400 con 0x55917b9e4800 2024-03-24T01:26:28.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.346+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 61 ==== osd_op_reply(143 1.00000000 [omap-set-vals] v40'33 uv33 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917b9e4800 2024-03-24T01:26:28.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.346+0000 7f6bae60e280 4 recover_dentries: inspecting lump 10000000001.00000000 2024-03-24T01:26:28.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.346+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:144 5.7 5:e2e07930:::10000000001.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba58800 con 0x55917c439400 2024-03-24T01:26:28.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.348+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 61 ==== osd_op_reply(144 10000000001.00000000 [omap-get-header out=274b] v0'0 uv17 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.348+0000 7f6bae60e280 4 recover_dentries: frag 10000000001.00000000 fnode old v15 vs new v17 2024-03-24T01:26:28.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.348+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.348+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:145 5.7 5:e2e07930:::10000000001.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba58c00 con 0x55917c439400 2024-03-24T01:26:28.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.352+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 62 ==== osd_op_reply(145 10000000001.00000000 [omap-set-header] v40'18 uv18 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.352+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:146 5.7 5:e2e07930:::10000000001.00000000:head [omap-get-vals-by-keys in=23b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba59000 con 0x55917c439400 2024-03-24T01:26:28.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.354+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 63 ==== osd_op_reply(146 10000000001.00000000 [omap-get-vals-by-keys out=589b] v0'0 uv18 ondisk = 0) v8 ==== 164+0+589 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.354+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 10000000001.00000000/subdirfile 2024-03-24T01:26:28.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.354+0000 7f6bae60e280 4 recover_dentries: dentry subdirfile_head existed already 2024-03-24T01:26:28.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.354+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.354+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 14 vs fullbit version 16 2024-03-24T01:26:28.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.354+0000 7f6bae60e280 4 recover_dentries: writing I dentry subdirfile_head into frag 10000000001.00000000 2024-03-24T01:26:28.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.354+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:147 5.7 5:e2e07930:::10000000001.00000000:head [omap-set-vals in=603b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba59400 con 0x55917c439400 2024-03-24T01:26:28.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.357+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 64 ==== osd_op_reply(147 10000000001.00000000 [omap-set-vals] v40'19 uv19 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.357+0000 7f6bae60e280 4 recover_dentries: inspecting lump 100000001f7.00000000 2024-03-24T01:26:28.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.357+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:148 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba59800 con 0x55917c439400 2024-03-24T01:26:28.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.359+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 65 ==== osd_op_reply(148 100000001f7.00000000 [omap-get-header out=274b] v0'0 uv13 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.359+0000 7f6bae60e280 4 recover_dentries: frag 100000001f7.00000000 fnode old v13 vs new v15 2024-03-24T01:26:28.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.359+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.359+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:149 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba59c00 con 0x55917c439400 2024-03-24T01:26:28.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.362+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 66 ==== osd_op_reply(149 100000001f7.00000000 [omap-set-header] v40'14 uv14 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.362+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:150 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-get-vals-by-keys in=18b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba58000 con 0x55917c439400 2024-03-24T01:26:28.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.363+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 67 ==== osd_op_reply(150 100000001f7.00000000 [omap-get-vals-by-keys out=40b] v0'0 uv14 ondisk = 0) v8 ==== 164+0+40 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.363+0000 7f6bae60e280 4 recover_dentries: inspecting nullbit 100000001f7.00000000/link2 2024-03-24T01:26:28.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.363+0000 7f6bae60e280 4 recover_dentries: dentry exists, will remove 2024-03-24T01:26:28.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.363+0000 7f6bae60e280 10 recover_dentries: Existing hardlink inode in slot to be (maybe) removed by null journal dn 'link2' with lump fnode version 15vs existing fnode version 13 2024-03-24T01:26:28.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.363+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:151 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-rm-keys in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba58400 con 0x55917c439400 2024-03-24T01:26:28.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.367+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 68 ==== osd_op_reply(151 100000001f7.00000000 [omap-rm-keys] v40'15 uv15 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.367+0000 7f6bae60e280 4 recover_dentries: inspecting lump 1.00000000 2024-03-24T01:26:28.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.368+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:152 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba58800 con 0x55917b9e4800 2024-03-24T01:26:28.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.370+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 62 ==== osd_op_reply(152 1.00000000 [omap-get-header out=274b] v0'0 uv33 ondisk = 0) v8 ==== 154+0+274 (crc 0 0 0) 0x55917ba1e280 con 0x55917b9e4800 2024-03-24T01:26:28.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.370+0000 7f6bae60e280 4 recover_dentries: frag 1.00000000 fnode old v38 vs new v40 2024-03-24T01:26:28.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.370+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.370+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:153 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba58c00 con 0x55917b9e4800 2024-03-24T01:26:28.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.373+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 63 ==== osd_op_reply(153 1.00000000 [omap-set-header] v40'34 uv34 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917b9e4800 2024-03-24T01:26:28.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.373+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:154 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba59000 con 0x55917b9e4800 2024-03-24T01:26:28.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.374+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 64 ==== osd_op_reply(154 1.00000000 [omap-get-vals-by-keys out=508b] v0'0 uv34 ondisk = 0) v8 ==== 154+0+508 (crc 0 0 0) 0x55917ba1e280 con 0x55917b9e4800 2024-03-24T01:26:28.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.374+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 1.00000000/linkdir 2024-03-24T01:26:28.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.374+0000 7f6bae60e280 4 recover_dentries: dentry linkdir_head existed already 2024-03-24T01:26:28.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.374+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.374+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 35 vs fullbit version 39 2024-03-24T01:26:28.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.374+0000 7f6bae60e280 4 recover_dentries: writing I dentry linkdir_head into frag 1.00000000 2024-03-24T01:26:28.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.374+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:155 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-vals in=508b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba59400 con 0x55917b9e4800 2024-03-24T01:26:28.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.378+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 65 ==== osd_op_reply(155 1.00000000 [omap-set-vals] v40'35 uv35 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917b9e4800 2024-03-24T01:26:28.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.378+0000 7f6bae60e280 4 recover_dentries: inspecting lump 100000001f7.00000000 2024-03-24T01:26:28.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.378+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:156 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c438800 con 0x55917c439400 2024-03-24T01:26:28.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.378+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 69 ==== osd_op_reply(156 100000001f7.00000000 [omap-get-header out=274b] v0'0 uv15 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.378+0000 7f6bae60e280 4 recover_dentries: frag 100000001f7.00000000 fnode old v15 vs new v17 2024-03-24T01:26:28.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.378+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.378+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:157 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c439000 con 0x55917c439400 2024-03-24T01:26:28.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.383+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 70 ==== osd_op_reply(157 100000001f7.00000000 [omap-set-header] v40'16 uv16 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.383+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:158 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-get-vals-by-keys in=18b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba16000 con 0x55917c439400 2024-03-24T01:26:28.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.385+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 71 ==== osd_op_reply(158 100000001f7.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv16 ondisk = 0) v8 ==== 164+0+4 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.385+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 100000001f7.00000000/link2 2024-03-24T01:26:28.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.385+0000 7f6bae60e280 4 recover_dentries: dentry did not already exist, will create 2024-03-24T01:26:28.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.385+0000 7f6bae60e280 4 recover_dentries: writing I dentry link2_head into frag 100000001f7.00000000 2024-03-24T01:26:28.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.385+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:159 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-set-vals in=544b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba16400 con 0x55917c439400 2024-03-24T01:26:28.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.389+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 72 ==== osd_op_reply(159 100000001f7.00000000 [omap-set-vals] v40'17 uv17 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.389+0000 7f6bae60e280 4 recover_dentries: inspecting lump 1.00000000 2024-03-24T01:26:28.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.389+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:160 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5400 con 0x55917b9e4800 2024-03-24T01:26:28.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.389+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 66 ==== osd_op_reply(160 1.00000000 [omap-get-header out=274b] v0'0 uv35 ondisk = 0) v8 ==== 154+0+274 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.389+0000 7f6bae60e280 4 recover_dentries: frag 1.00000000 fnode old v40 vs new v44 2024-03-24T01:26:28.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.389+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.389+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:161 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5800 con 0x55917b9e4800 2024-03-24T01:26:28.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.394+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 67 ==== osd_op_reply(161 1.00000000 [omap-set-header] v40'36 uv36 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.394+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:162 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=35b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4000 con 0x55917b9e4800 2024-03-24T01:26:28.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.394+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 68 ==== osd_op_reply(162 1.00000000 [omap-get-vals-by-keys out=1011b] v0'0 uv36 ondisk = 0) v8 ==== 154+0+1011 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.395+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 1.00000000/linkdir 2024-03-24T01:26:28.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.395+0000 7f6bae60e280 4 recover_dentries: dentry linkdir_head existed already 2024-03-24T01:26:28.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.395+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.395+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 39 vs fullbit version 41 2024-03-24T01:26:28.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.395+0000 7f6bae60e280 4 recover_dentries: writing I dentry linkdir_head into frag 1.00000000 2024-03-24T01:26:28.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.395+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 1.00000000/subdir 2024-03-24T01:26:28.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.395+0000 7f6bae60e280 4 recover_dentries: dentry subdir_head existed already 2024-03-24T01:26:28.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.395+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.395+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 37 vs fullbit version 43 2024-03-24T01:26:28.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.395+0000 7f6bae60e280 4 recover_dentries: writing I dentry subdir_head into frag 1.00000000 2024-03-24T01:26:28.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.395+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:163 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-vals in=1011b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5c00 con 0x55917b9e4800 2024-03-24T01:26:28.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.399+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 69 ==== osd_op_reply(163 1.00000000 [omap-set-vals] v40'37 uv37 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.399+0000 7f6bae60e280 4 recover_dentries: inspecting lump 100000001f7.00000000 2024-03-24T01:26:28.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.399+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:164 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4400 con 0x55917c439400 2024-03-24T01:26:28.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.401+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 73 ==== osd_op_reply(164 100000001f7.00000000 [omap-get-header out=274b] v0'0 uv17 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.401+0000 7f6bae60e280 4 recover_dentries: frag 100000001f7.00000000 fnode old v17 vs new v19 2024-03-24T01:26:28.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.401+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.401+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:165 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c52e800 con 0x55917c439400 2024-03-24T01:26:28.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.404+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 74 ==== osd_op_reply(165 100000001f7.00000000 [omap-set-header] v40'18 uv18 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.404+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:166 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-get-vals-by-keys in=18b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5400 con 0x55917c439400 2024-03-24T01:26:28.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.405+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 75 ==== osd_op_reply(166 100000001f7.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv18 ondisk = 0) v8 ==== 164+0+4 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.405+0000 7f6bae60e280 4 recover_dentries: inspecting remotebit 100000001f7.00000000/link3 2024-03-24T01:26:28.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.405+0000 7f6bae60e280 4 recover_dentries: dentry did not already exist, will create 2024-03-24T01:26:28.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.405+0000 7f6bae60e280 4 recover_dentries: writing L dentry link3_head into frag 100000001f7.00000000 2024-03-24T01:26:28.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.405+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:167 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-set-vals in=40b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5800 con 0x55917c439400 2024-03-24T01:26:28.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.408+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 76 ==== osd_op_reply(167 100000001f7.00000000 [omap-set-vals] v40'19 uv19 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.408+0000 7f6bae60e280 4 recover_dentries: inspecting lump 10000000001.00000000 2024-03-24T01:26:28.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.408+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:168 5.7 5:e2e07930:::10000000001.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4000 con 0x55917c439400 2024-03-24T01:26:28.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.410+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 77 ==== osd_op_reply(168 10000000001.00000000 [omap-get-header out=274b] v0'0 uv19 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.410+0000 7f6bae60e280 4 recover_dentries: frag 10000000001.00000000 fnode old v17 vs new v19 2024-03-24T01:26:28.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.410+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.410+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:169 5.7 5:e2e07930:::10000000001.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5c00 con 0x55917c439400 2024-03-24T01:26:28.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.413+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 78 ==== osd_op_reply(169 10000000001.00000000 [omap-set-header] v40'20 uv20 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.413+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:170 5.7 5:e2e07930:::10000000001.00000000:head [omap-get-vals-by-keys in=23b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4400 con 0x55917c439400 2024-03-24T01:26:28.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.415+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 79 ==== osd_op_reply(170 10000000001.00000000 [omap-get-vals-by-keys out=603b] v0'0 uv20 ondisk = 0) v8 ==== 164+0+603 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.415+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 10000000001.00000000/subdirfile 2024-03-24T01:26:28.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.415+0000 7f6bae60e280 4 recover_dentries: dentry subdirfile_head existed already 2024-03-24T01:26:28.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.415+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.415+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 16 vs fullbit version 18 2024-03-24T01:26:28.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.415+0000 7f6bae60e280 4 recover_dentries: writing I dentry subdirfile_head into frag 10000000001.00000000 2024-03-24T01:26:28.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.415+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:171 5.7 5:e2e07930:::10000000001.00000000:head [omap-set-vals in=603b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba16000 con 0x55917c439400 2024-03-24T01:26:28.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.419+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 80 ==== osd_op_reply(171 10000000001.00000000 [omap-set-vals] v40'21 uv21 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.419+0000 7f6bae60e280 4 recover_dentries: inspecting lump 1.00000000 2024-03-24T01:26:28.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.419+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:172 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba16400 con 0x55917b9e4800 2024-03-24T01:26:28.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.422+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 70 ==== osd_op_reply(172 1.00000000 [omap-get-header out=274b] v0'0 uv37 ondisk = 0) v8 ==== 154+0+274 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.422+0000 7f6bae60e280 4 recover_dentries: frag 1.00000000 fnode old v44 vs new v48 2024-03-24T01:26:28.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.422+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.422+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:173 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c438800 con 0x55917b9e4800 2024-03-24T01:26:28.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.424+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 71 ==== osd_op_reply(173 1.00000000 [omap-set-header] v40'38 uv38 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.424+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:174 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=35b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c439000 con 0x55917b9e4800 2024-03-24T01:26:28.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.427+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 72 ==== osd_op_reply(174 1.00000000 [omap-get-vals-by-keys out=1011b] v0'0 uv38 ondisk = 0) v8 ==== 154+0+1011 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.427+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 1.00000000/linkdir 2024-03-24T01:26:28.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.427+0000 7f6bae60e280 4 recover_dentries: dentry linkdir_head existed already 2024-03-24T01:26:28.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.427+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.427+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 41 vs fullbit version 45 2024-03-24T01:26:28.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.427+0000 7f6bae60e280 4 recover_dentries: writing I dentry linkdir_head into frag 1.00000000 2024-03-24T01:26:28.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.427+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 1.00000000/subdir 2024-03-24T01:26:28.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.427+0000 7f6bae60e280 4 recover_dentries: dentry subdir_head existed already 2024-03-24T01:26:28.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.427+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.427+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 43 vs fullbit version 47 2024-03-24T01:26:28.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.427+0000 7f6bae60e280 4 recover_dentries: writing I dentry subdir_head into frag 1.00000000 2024-03-24T01:26:28.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.427+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:175 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-vals in=1011b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba58000 con 0x55917b9e4800 2024-03-24T01:26:28.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.431+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 73 ==== osd_op_reply(175 1.00000000 [omap-set-vals] v40'39 uv39 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.431+0000 7f6bae60e280 4 recover_dentries: inspecting lump 10000000001.00000000 2024-03-24T01:26:28.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.431+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:176 5.7 5:e2e07930:::10000000001.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba58400 con 0x55917c439400 2024-03-24T01:26:28.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.432+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 81 ==== osd_op_reply(176 10000000001.00000000 [omap-get-header out=274b] v0'0 uv21 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.432+0000 7f6bae60e280 4 recover_dentries: frag 10000000001.00000000 fnode old v19 vs new v21 2024-03-24T01:26:28.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.432+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.432+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:177 5.7 5:e2e07930:::10000000001.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba58800 con 0x55917c439400 2024-03-24T01:26:28.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.436+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 82 ==== osd_op_reply(177 10000000001.00000000 [omap-set-header] v40'22 uv22 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.436+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:178 5.7 5:e2e07930:::10000000001.00000000:head [omap-get-vals-by-keys in=23b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba58c00 con 0x55917c439400 2024-03-24T01:26:28.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.437+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 83 ==== osd_op_reply(178 10000000001.00000000 [omap-get-vals-by-keys out=603b] v0'0 uv22 ondisk = 0) v8 ==== 164+0+603 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.437+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 10000000001.00000000/subdirfile 2024-03-24T01:26:28.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.437+0000 7f6bae60e280 4 recover_dentries: dentry subdirfile_head existed already 2024-03-24T01:26:28.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.437+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.437+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 18 vs fullbit version 20 2024-03-24T01:26:28.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.437+0000 7f6bae60e280 4 recover_dentries: writing I dentry subdirfile_head into frag 10000000001.00000000 2024-03-24T01:26:28.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.437+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:179 5.7 5:e2e07930:::10000000001.00000000:head [omap-set-vals in=603b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba59000 con 0x55917c439400 2024-03-24T01:26:28.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.441+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 84 ==== osd_op_reply(179 10000000001.00000000 [omap-set-vals] v40'23 uv23 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.442+0000 7f6bae60e280 4 recover_dentries: inspecting lump 100000001f7.00000000 2024-03-24T01:26:28.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.442+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:180 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba59400 con 0x55917c439400 2024-03-24T01:26:28.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.443+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 85 ==== osd_op_reply(180 100000001f7.00000000 [omap-get-header out=274b] v0'0 uv19 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.443+0000 7f6bae60e280 4 recover_dentries: frag 100000001f7.00000000 fnode old v19 vs new v21 2024-03-24T01:26:28.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.443+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.443+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:181 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba59800 con 0x55917c439400 2024-03-24T01:26:28.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.447+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 86 ==== osd_op_reply(181 100000001f7.00000000 [omap-set-header] v40'20 uv20 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.447+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:182 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-get-vals-by-keys in=18b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba59c00 con 0x55917c439400 2024-03-24T01:26:28.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.447+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 87 ==== osd_op_reply(182 100000001f7.00000000 [omap-get-vals-by-keys out=40b] v0'0 uv20 ondisk = 0) v8 ==== 164+0+40 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.447+0000 7f6bae60e280 4 recover_dentries: inspecting nullbit 100000001f7.00000000/link3 2024-03-24T01:26:28.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.447+0000 7f6bae60e280 4 recover_dentries: dentry exists, will remove 2024-03-24T01:26:28.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.447+0000 7f6bae60e280 10 recover_dentries: Existing hardlink inode in slot to be (maybe) removed by null journal dn 'link3' with lump fnode version 21vs existing fnode version 19 2024-03-24T01:26:28.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.447+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:183 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-rm-keys in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c438800 con 0x55917c439400 2024-03-24T01:26:28.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.450+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 88 ==== osd_op_reply(183 100000001f7.00000000 [omap-rm-keys] v40'21 uv21 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.450+0000 7f6bae60e280 4 recover_dentries: inspecting lump 1.00000000 2024-03-24T01:26:28.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.450+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:184 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c439000 con 0x55917b9e4800 2024-03-24T01:26:28.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.453+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 74 ==== osd_op_reply(184 1.00000000 [omap-get-header out=274b] v0'0 uv39 ondisk = 0) v8 ==== 154+0+274 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.453+0000 7f6bae60e280 4 recover_dentries: frag 1.00000000 fnode old v48 vs new v50 2024-03-24T01:26:28.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.453+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.453+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:185 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba16000 con 0x55917b9e4800 2024-03-24T01:26:28.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.456+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 75 ==== osd_op_reply(185 1.00000000 [omap-set-header] v40'40 uv40 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.456+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:186 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=21b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba16400 con 0x55917b9e4800 2024-03-24T01:26:28.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.458+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 76 ==== osd_op_reply(186 1.00000000 [omap-get-vals-by-keys out=547b] v0'0 uv40 ondisk = 0) v8 ==== 154+0+547 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.458+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 1.00000000/rootfile 2024-03-24T01:26:28.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.458+0000 7f6bae60e280 4 recover_dentries: dentry rootfile_head existed already 2024-03-24T01:26:28.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.458+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.458+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 2 vs fullbit version 49 2024-03-24T01:26:28.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.458+0000 7f6bae60e280 4 recover_dentries: writing I dentry rootfile_head into frag 1.00000000 2024-03-24T01:26:28.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.458+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:187 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-vals in=509b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5400 con 0x55917b9e4800 2024-03-24T01:26:28.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.462+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 77 ==== osd_op_reply(187 1.00000000 [omap-set-vals] v40'41 uv41 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.462+0000 7f6bae60e280 4 recover_dentries: updating root 0x0x1 2024-03-24T01:26:28.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.462+0000 7f6bae60e280 4 recover_dentries: object id 1.00000000.inode 2024-03-24T01:26:28.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.462+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:188 5.14 5:287034c4:::1.00000000.inode:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5800 con 0x55917c438c00 2024-03-24T01:26:28.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.462+0000 7f6bac400640 1 -- 172.21.15.90:0/225243987 <== osd.3 v2:172.21.15.90:6808/2999931983 19 ==== osd_op_reply(188 1.00000000.inode [read 0~582 out=582b] v0'0 uv34 ondisk = 0) v8 ==== 160+0+582 (crc 0 0 0) 0x55917b7e2500 con 0x55917c438c00 2024-03-24T01:26:28.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.462+0000 7f6bae60e280 4 recover_dentries: root exists, will modify (582) 2024-03-24T01:26:28.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.462+0000 7f6bae60e280 4 recover_dentries: magic ok 2024-03-24T01:26:28.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.462+0000 7f6bae60e280 4 recover_dentries: writing root ino 1.00000000.inode version 7 2024-03-24T01:26:28.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.462+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:189 5.14 5:287034c4:::1.00000000.inode:head [writefull 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4000 con 0x55917c438c00 2024-03-24T01:26:28.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.466+0000 7f6bac400640 1 -- 172.21.15.90:0/225243987 <== osd.3 v2:172.21.15.90:6808/2999931983 20 ==== osd_op_reply(189 1.00000000.inode [writefull 0~582] v40'35 uv35 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x55917b7e2500 con 0x55917c438c00 2024-03-24T01:26:28.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.466+0000 7f6bae60e280 4 recover_dentries: inspecting lump 1.00000000 2024-03-24T01:26:28.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.466+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:190 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5c00 con 0x55917b9e4800 2024-03-24T01:26:28.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.468+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 78 ==== osd_op_reply(190 1.00000000 [omap-get-header out=274b] v0'0 uv41 ondisk = 0) v8 ==== 154+0+274 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.468+0000 7f6bae60e280 4 recover_dentries: frag 1.00000000 fnode old v50 vs new v54 2024-03-24T01:26:28.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.468+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.468+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:191 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4400 con 0x55917b9e4800 2024-03-24T01:26:28.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.472+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 79 ==== osd_op_reply(191 1.00000000 [omap-set-header] v40'42 uv42 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.472+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:192 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=37b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c52e800 con 0x55917b9e4800 2024-03-24T01:26:28.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.472+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 80 ==== osd_op_reply(192 1.00000000 [omap-get-vals-by-keys out=1013b] v0'0 uv42 ondisk = 0) v8 ==== 154+0+1013 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.472+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 1.00000000/linkdir 2024-03-24T01:26:28.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.472+0000 7f6bae60e280 4 recover_dentries: dentry linkdir_head existed already 2024-03-24T01:26:28.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.472+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.472+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 45 vs fullbit version 52 2024-03-24T01:26:28.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.472+0000 7f6bae60e280 4 recover_dentries: writing I dentry linkdir_head into frag 1.00000000 2024-03-24T01:26:28.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.472+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 1.00000000/rootfile 2024-03-24T01:26:28.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.472+0000 7f6bae60e280 4 recover_dentries: dentry rootfile_head existed already 2024-03-24T01:26:28.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.472+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.472+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 49 vs fullbit version 51 2024-03-24T01:26:28.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.472+0000 7f6bae60e280 4 recover_dentries: writing I dentry rootfile_head into frag 1.00000000 2024-03-24T01:26:28.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.472+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:193 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-vals in=1091b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba78000 con 0x55917b9e4800 2024-03-24T01:26:28.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.477+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 81 ==== osd_op_reply(193 1.00000000 [omap-set-vals] v40'43 uv43 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.477+0000 7f6bae60e280 4 recover_dentries: inspecting lump 100000001f7.00000000 2024-03-24T01:26:28.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.477+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:194 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba78400 con 0x55917c439400 2024-03-24T01:26:28.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.478+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 89 ==== osd_op_reply(194 100000001f7.00000000 [omap-get-header out=274b] v0'0 uv21 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.479+0000 7f6bae60e280 4 recover_dentries: frag 100000001f7.00000000 fnode old v21 vs new v23 2024-03-24T01:26:28.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.479+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.479+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:195 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba78800 con 0x55917c439400 2024-03-24T01:26:28.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.481+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 90 ==== osd_op_reply(195 100000001f7.00000000 [omap-set-header] v40'22 uv22 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.481+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:196 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-get-vals-by-keys in=18b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba78c00 con 0x55917c439400 2024-03-24T01:26:28.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.484+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 91 ==== osd_op_reply(196 100000001f7.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv22 ondisk = 0) v8 ==== 164+0+4 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.484+0000 7f6bae60e280 4 recover_dentries: inspecting remotebit 100000001f7.00000000/link3 2024-03-24T01:26:28.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.484+0000 7f6bae60e280 4 recover_dentries: dentry did not already exist, will create 2024-03-24T01:26:28.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.484+0000 7f6bae60e280 4 recover_dentries: writing L dentry link3_head into frag 100000001f7.00000000 2024-03-24T01:26:28.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.484+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:197 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-set-vals in=40b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba79000 con 0x55917c439400 2024-03-24T01:26:28.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.487+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 92 ==== osd_op_reply(197 100000001f7.00000000 [omap-set-vals] v40'23 uv23 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.487+0000 7f6bae60e280 4 recover_dentries: updating root 0x0x1 2024-03-24T01:26:28.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.487+0000 7f6bae60e280 4 recover_dentries: object id 1.00000000.inode 2024-03-24T01:26:28.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.487+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:198 5.14 5:287034c4:::1.00000000.inode:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba79400 con 0x55917c438c00 2024-03-24T01:26:28.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.489+0000 7f6bac400640 1 -- 172.21.15.90:0/225243987 <== osd.3 v2:172.21.15.90:6808/2999931983 21 ==== osd_op_reply(198 1.00000000.inode [read 0~582 out=582b] v0'0 uv35 ondisk = 0) v8 ==== 160+0+582 (crc 0 0 0) 0x55917b7e2500 con 0x55917c438c00 2024-03-24T01:26:28.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.489+0000 7f6bae60e280 4 recover_dentries: root exists, will modify (582) 2024-03-24T01:26:28.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.489+0000 7f6bae60e280 4 recover_dentries: magic ok 2024-03-24T01:26:28.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.489+0000 7f6bae60e280 4 recover_dentries: writing root ino 1.00000000.inode version 8 2024-03-24T01:26:28.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.489+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:199 5.14 5:287034c4:::1.00000000.inode:head [writefull 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba79800 con 0x55917c438c00 2024-03-24T01:26:28.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.493+0000 7f6bac400640 1 -- 172.21.15.90:0/225243987 <== osd.3 v2:172.21.15.90:6808/2999931983 22 ==== osd_op_reply(199 1.00000000.inode [writefull 0~582] v40'36 uv36 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x55917b7e2500 con 0x55917c438c00 2024-03-24T01:26:28.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.493+0000 7f6bae60e280 4 recover_dentries: inspecting lump 1.00000000 2024-03-24T01:26:28.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.493+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:200 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba79c00 con 0x55917b9e4800 2024-03-24T01:26:28.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.494+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 82 ==== osd_op_reply(200 1.00000000 [omap-get-header out=274b] v0'0 uv43 ondisk = 0) v8 ==== 154+0+274 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.494+0000 7f6bae60e280 4 recover_dentries: frag 1.00000000 fnode old v54 vs new v56 2024-03-24T01:26:28.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.494+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.494+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:201 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba79400 con 0x55917b9e4800 2024-03-24T01:26:28.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.500+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 83 ==== osd_op_reply(201 1.00000000 [omap-set-header] v40'44 uv44 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.500+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:202 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba79800 con 0x55917b9e4800 2024-03-24T01:26:28.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.502+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 84 ==== osd_op_reply(202 1.00000000 [omap-get-vals-by-keys out=507b] v0'0 uv44 ondisk = 0) v8 ==== 154+0+507 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.502+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 1.00000000/subdir 2024-03-24T01:26:28.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.502+0000 7f6bae60e280 4 recover_dentries: dentry subdir_head existed already 2024-03-24T01:26:28.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.502+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.502+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 47 vs fullbit version 55 2024-03-24T01:26:28.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.502+0000 7f6bae60e280 4 recover_dentries: writing I dentry subdir_head into frag 1.00000000 2024-03-24T01:26:28.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.502+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:203 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-vals in=507b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4c00 con 0x55917b9e4800 2024-03-24T01:26:28.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.505+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 85 ==== osd_op_reply(203 1.00000000 [omap-set-vals] v40'45 uv45 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.505+0000 7f6bae60e280 4 recover_dentries: inspecting lump 10000000001.00000000 2024-03-24T01:26:28.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.506+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:204 5.7 5:e2e07930:::10000000001.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5000 con 0x55917c439400 2024-03-24T01:26:28.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.507+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 93 ==== osd_op_reply(204 10000000001.00000000 [omap-get-header out=274b] v0'0 uv23 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.507+0000 7f6bae60e280 4 recover_dentries: frag 10000000001.00000000 fnode old v21 vs new v23 2024-03-24T01:26:28.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.507+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.507+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:205 5.7 5:e2e07930:::10000000001.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5800 con 0x55917c439400 2024-03-24T01:26:28.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.511+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 94 ==== osd_op_reply(205 10000000001.00000000 [omap-set-header] v40'24 uv24 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.511+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:206 5.7 5:e2e07930:::10000000001.00000000:head [omap-get-vals-by-keys in=22b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4000 con 0x55917c439400 2024-03-24T01:26:28.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.512+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 95 ==== osd_op_reply(206 10000000001.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv24 ondisk = 0) v8 ==== 164+0+4 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.512+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 10000000001.00000000/subsubdir 2024-03-24T01:26:28.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.512+0000 7f6bae60e280 4 recover_dentries: dentry did not already exist, will create 2024-03-24T01:26:28.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.512+0000 7f6bae60e280 4 recover_dentries: writing I dentry subsubdir_head into frag 10000000001.00000000 2024-03-24T01:26:28.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.512+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:207 5.7 5:e2e07930:::10000000001.00000000:head [omap-set-vals in=510b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba16800 con 0x55917c439400 2024-03-24T01:26:28.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.515+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 96 ==== osd_op_reply(207 10000000001.00000000 [omap-set-vals] v40'25 uv25 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917c439400 2024-03-24T01:26:28.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.515+0000 7f6bae60e280 4 recover_dentries: inspecting lump 100000001fa.00000000 2024-03-24T01:26:28.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.515+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:208 5.1f 5:feb1c803:::100000001fa.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5400 con 0x55917b9e4800 2024-03-24T01:26:28.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.517+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 86 ==== osd_op_reply(208 100000001fa.00000000 [omap-get-header] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917b9e4800 2024-03-24T01:26:28.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.517+0000 7f6bae60e280 4 recover_dentries: failed to read OMAP header from directory fragment 100000001fa.00000000 (2) No such file or directory 2024-03-24T01:26:28.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.517+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.517+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:209 5.1f 5:feb1c803:::100000001fa.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5c00 con 0x55917b9e4800 2024-03-24T01:26:28.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.520+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 87 ==== osd_op_reply(209 100000001fa.00000000 [omap-set-header] v40'46 uv46 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917b9e4800 2024-03-24T01:26:28.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.520+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:210 5.1f 5:feb1c803:::100000001fa.00000000:head [omap-get-vals-by-keys in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4400 con 0x55917b9e4800 2024-03-24T01:26:28.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.522+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 88 ==== osd_op_reply(210 100000001fa.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv46 ondisk = 0) v8 ==== 164+0+4 (crc 0 0 0) 0x55917ba1e280 con 0x55917b9e4800 2024-03-24T01:26:28.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.522+0000 7f6bae60e280 4 recover_dentries: inspecting lump 602.00000000 2024-03-24T01:26:28.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.522+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6808/1045862942,v1:172.21.15.120:6809/1045862942] -- osd_op(unknown.0.0:211 5.10 5:0bd6d154:::602.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4c00 con 0x55917ba16c00 2024-03-24T01:26:28.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.523+0000 7f6bab000640 1 -- 172.21.15.90:0/225243987 <== osd.7 v2:172.21.15.120:6808/1045862942 5 ==== osd_op_reply(211 602.00000000 [omap-get-header out=274b] v0'0 uv10 ondisk = 0) v8 ==== 156+0+274 (crc 0 0 0) 0x55917b7e2000 con 0x55917ba16c00 2024-03-24T01:26:28.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.523+0000 7f6bae60e280 4 recover_dentries: frag 602.00000000 fnode old v6 vs new v8 2024-03-24T01:26:28.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.523+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.523+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6808/1045862942,v1:172.21.15.120:6809/1045862942] -- osd_op(unknown.0.0:212 5.10 5:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5000 con 0x55917ba16c00 2024-03-24T01:26:28.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.527+0000 7f6bab000640 1 -- 172.21.15.90:0/225243987 <== osd.7 v2:172.21.15.120:6808/1045862942 6 ==== osd_op_reply(212 602.00000000 [omap-set-header] v40'11 uv11 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55917b7e2000 con 0x55917ba16c00 2024-03-24T01:26:28.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.528+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6808/1045862942,v1:172.21.15.120:6809/1045862942] -- osd_op(unknown.0.0:213 5.10 5:0bd6d154:::602.00000000:head [omap-get-vals-by-keys in=24b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e5800 con 0x55917ba16c00 2024-03-24T01:26:28.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.529+0000 7f6bab000640 1 -- 172.21.15.90:0/225243987 <== osd.7 v2:172.21.15.120:6808/1045862942 7 ==== osd_op_reply(213 602.00000000 [omap-get-vals-by-keys out=526b] v0'0 uv11 ondisk = 0) v8 ==== 156+0+526 (crc 0 0 0) 0x55917b7e2000 con 0x55917ba16c00 2024-03-24T01:26:28.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.529+0000 7f6bae60e280 4 recover_dentries: inspecting nullbit 602.00000000/100000001f8 2024-03-24T01:26:28.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.529+0000 7f6bae60e280 4 recover_dentries: dentry exists, will remove 2024-03-24T01:26:28.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.529+0000 7f6bae60e280 10 recover_dentries: Existing full inode in slot to be (maybe) removed by null journal dn '100000001f8' with lump fnode version 8vs existing fnode version 6 2024-03-24T01:26:28.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.529+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6808/1045862942,v1:172.21.15.120:6809/1045862942] -- osd_op(unknown.0.0:214 5.10 5:0bd6d154:::602.00000000:head [omap-rm-keys in=24b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c52e800 con 0x55917ba16c00 2024-03-24T01:26:28.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.532+0000 7f6bab000640 1 -- 172.21.15.90:0/225243987 <== osd.7 v2:172.21.15.120:6808/1045862942 8 ==== osd_op_reply(214 602.00000000 [omap-rm-keys] v40'12 uv12 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55917b7e2000 con 0x55917ba16c00 2024-03-24T01:26:28.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.532+0000 7f6bae60e280 4 recover_dentries: inspecting lump 1.00000000 2024-03-24T01:26:28.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.532+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:215 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba16000 con 0x55917b9e4800 2024-03-24T01:26:28.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.534+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 89 ==== osd_op_reply(215 1.00000000 [omap-get-header out=274b] v0'0 uv45 ondisk = 0) v8 ==== 154+0+274 (crc 0 0 0) 0x55917ba1e280 con 0x55917b9e4800 2024-03-24T01:26:28.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.534+0000 7f6bae60e280 4 recover_dentries: frag 1.00000000 fnode old v56 vs new v58 2024-03-24T01:26:28.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.534+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.534+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:216 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba16400 con 0x55917b9e4800 2024-03-24T01:26:28.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.537+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 90 ==== osd_op_reply(216 1.00000000 [omap-set-header] v40'47 uv47 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917b9e4800 2024-03-24T01:26:28.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.537+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:217 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c438800 con 0x55917b9e4800 2024-03-24T01:26:28.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.539+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 91 ==== osd_op_reply(217 1.00000000 [omap-get-vals-by-keys out=508b] v0'0 uv47 ondisk = 0) v8 ==== 154+0+508 (crc 0 0 0) 0x55917ba1e280 con 0x55917b9e4800 2024-03-24T01:26:28.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.539+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 1.00000000/linkdir 2024-03-24T01:26:28.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.539+0000 7f6bae60e280 4 recover_dentries: dentry linkdir_head existed already 2024-03-24T01:26:28.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.539+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.539+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 52 vs fullbit version 57 2024-03-24T01:26:28.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.539+0000 7f6bae60e280 4 recover_dentries: writing I dentry linkdir_head into frag 1.00000000 2024-03-24T01:26:28.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.539+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:218 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-vals in=508b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c439000 con 0x55917b9e4800 2024-03-24T01:26:28.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.543+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 92 ==== osd_op_reply(218 1.00000000 [omap-set-vals] v40'48 uv48 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917ba1e280 con 0x55917b9e4800 2024-03-24T01:26:28.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.543+0000 7f6bae60e280 4 recover_dentries: inspecting lump 100000001f7.00000000 2024-03-24T01:26:28.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.543+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:219 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917b9e4c00 con 0x55917c439400 2024-03-24T01:26:28.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.544+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 97 ==== osd_op_reply(219 100000001f7.00000000 [omap-get-header out=274b] v0'0 uv23 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.544+0000 7f6bae60e280 4 recover_dentries: frag 100000001f7.00000000 fnode old v23 vs new v25 2024-03-24T01:26:28.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.544+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.544+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:220 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba17000 con 0x55917c439400 2024-03-24T01:26:28.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.548+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 98 ==== osd_op_reply(220 100000001f7.00000000 [omap-set-header] v40'24 uv24 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.548+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:221 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-get-vals-by-keys in=18b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba17400 con 0x55917c439400 2024-03-24T01:26:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.549+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 99 ==== osd_op_reply(221 100000001f7.00000000 [omap-get-vals-by-keys out=544b] v0'0 uv24 ondisk = 0) v8 ==== 164+0+544 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.549+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 100000001f7.00000000/link2 2024-03-24T01:26:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.549+0000 7f6bae60e280 4 recover_dentries: dentry link2_head existed already 2024-03-24T01:26:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.549+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.549+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 16 vs fullbit version 24 2024-03-24T01:26:28.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.549+0000 7f6bae60e280 4 recover_dentries: writing I dentry link2_head into frag 100000001f7.00000000 2024-03-24T01:26:28.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.549+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:222 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-set-vals in=506b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba17800 con 0x55917c439400 2024-03-24T01:26:28.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.554+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.5 v2:172.21.15.120:6824/1739969887 100 ==== osd_op_reply(222 100000001f7.00000000 [omap-set-vals] v40'25 uv25 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917c439400 2024-03-24T01:26:28.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.554+0000 7f6bae60e280 4 recover_dentries: updating root 0x0x1 2024-03-24T01:26:28.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.554+0000 7f6bae60e280 4 recover_dentries: object id 1.00000000.inode 2024-03-24T01:26:28.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.554+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:223 5.14 5:287034c4:::1.00000000.inode:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba17c00 con 0x55917c438c00 2024-03-24T01:26:28.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.555+0000 7f6bac400640 1 -- 172.21.15.90:0/225243987 <== osd.3 v2:172.21.15.90:6808/2999931983 23 ==== osd_op_reply(223 1.00000000.inode [read 0~582 out=582b] v0'0 uv36 ondisk = 0) v8 ==== 160+0+582 (crc 0 0 0) 0x55917b7e2500 con 0x55917c438c00 2024-03-24T01:26:28.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.555+0000 7f6bae60e280 4 recover_dentries: root exists, will modify (582) 2024-03-24T01:26:28.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.555+0000 7f6bae60e280 4 recover_dentries: magic ok 2024-03-24T01:26:28.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.555+0000 7f6bae60e280 4 recover_dentries: writing root ino 1.00000000.inode version 9 2024-03-24T01:26:28.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.555+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:224 5.14 5:287034c4:::1.00000000.inode:head [writefull 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c59b000 con 0x55917c438c00 2024-03-24T01:26:28.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.559+0000 7f6bac400640 1 -- 172.21.15.90:0/225243987 <== osd.3 v2:172.21.15.90:6808/2999931983 24 ==== osd_op_reply(224 1.00000000.inode [writefull 0~582] v40'37 uv37 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x55917b7e2500 con 0x55917c438c00 2024-03-24T01:26:28.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.559+0000 7f6bae60e280 4 recover_dentries: inspecting lump 100.00000000 2024-03-24T01:26:28.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.559+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:225 5.13 5:cd5a64a3:::100.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917c59ac00 con 0x55917c438c00 2024-03-24T01:26:28.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.560+0000 7f6bac400640 1 -- 172.21.15.90:0/225243987 <== osd.3 v2:172.21.15.90:6808/2999931983 25 ==== osd_op_reply(225 100.00000000 [omap-get-header out=274b] v0'0 uv4 ondisk = 0) v8 ==== 156+0+274 (crc 0 0 0) 0x55917b7e2500 con 0x55917c438c00 2024-03-24T01:26:28.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.560+0000 7f6bae60e280 4 recover_dentries: frag 100.00000000 fnode old v3 vs new v5 2024-03-24T01:26:28.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.560+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.560+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:226 5.13 5:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba78000 con 0x55917c438c00 2024-03-24T01:26:28.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.563+0000 7f6bac400640 1 -- 172.21.15.90:0/225243987 <== osd.3 v2:172.21.15.90:6808/2999931983 26 ==== osd_op_reply(226 100.00000000 [omap-set-header] v40'5 uv5 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55917b7e2500 con 0x55917c438c00 2024-03-24T01:26:28.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.564+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:227 5.13 5:cd5a64a3:::100.00000000:head [omap-get-vals-by-keys in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba78400 con 0x55917c438c00 2024-03-24T01:26:28.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.565+0000 7f6bac400640 1 -- 172.21.15.90:0/225243987 <== osd.3 v2:172.21.15.90:6808/2999931983 27 ==== osd_op_reply(227 100.00000000 [omap-get-vals-by-keys out=507b] v0'0 uv5 ondisk = 0) v8 ==== 156+0+507 (crc 0 0 0) 0x55917b7e2500 con 0x55917c438c00 2024-03-24T01:26:28.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.565+0000 7f6bae60e280 4 recover_dentries: inspecting fullbit 100.00000000/stray2 2024-03-24T01:26:28.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.565+0000 7f6bae60e280 4 recover_dentries: dentry stray2_head existed already 2024-03-24T01:26:28.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.565+0000 7f6bae60e280 4 recover_dentries: dentry exists, checking versions... 2024-03-24T01:26:28.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.565+0000 7f6bae60e280 4 recover_dentries: decoded embedded inode version 2 vs fullbit version 4 2024-03-24T01:26:28.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.565+0000 7f6bae60e280 4 recover_dentries: writing I dentry stray2_head into frag 100.00000000 2024-03-24T01:26:28.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.565+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:228 5.13 5:cd5a64a3:::100.00000000:head [omap-set-vals in=507b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba78800 con 0x55917c438c00 2024-03-24T01:26:28.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.568+0000 7f6bac400640 1 -- 172.21.15.90:0/225243987 <== osd.3 v2:172.21.15.90:6808/2999931983 28 ==== osd_op_reply(228 100.00000000 [omap-set-vals] v40'6 uv6 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55917b7e2500 con 0x55917c438c00 2024-03-24T01:26:28.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.568+0000 7f6bae60e280 4 recover_dentries: updating root 0x0x100 2024-03-24T01:26:28.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.568+0000 7f6bae60e280 4 recover_dentries: object id 100.00000000.inode 2024-03-24T01:26:28.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.568+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:229 5.1f 5:fe07bba1:::100.00000000.inode:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba78c00 con 0x55917b9e4800 2024-03-24T01:26:28.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.570+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 93 ==== osd_op_reply(229 100.00000000.inode [read 0~582 out=582b] v0'0 uv3 ondisk = 0) v8 ==== 162+0+582 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.570+0000 7f6bae60e280 4 recover_dentries: root exists, will modify (582) 2024-03-24T01:26:28.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.570+0000 7f6bae60e280 4 recover_dentries: magic ok 2024-03-24T01:26:28.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.570+0000 7f6bae60e280 4 recover_dentries: writing root ino 100.00000000.inode version 2 2024-03-24T01:26:28.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.570+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:230 5.1f 5:fe07bba1:::100.00000000.inode:head [writefull 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba79000 con 0x55917b9e4800 2024-03-24T01:26:28.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.574+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 94 ==== osd_op_reply(230 100.00000000.inode [writefull 0~582] v40'49 uv49 ondisk = 0) v8 ==== 162+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.574+0000 7f6bae60e280 4 recover_dentries: inspecting lump 1.00000000 2024-03-24T01:26:28.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.574+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:231 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba79400 con 0x55917b9e4800 2024-03-24T01:26:28.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.576+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 95 ==== osd_op_reply(231 1.00000000 [omap-get-header out=274b] v0'0 uv48 ondisk = 0) v8 ==== 154+0+274 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.576+0000 7f6bae60e280 4 recover_dentries: frag 1.00000000 fnode old v58 vs new v59 2024-03-24T01:26:28.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.576+0000 7f6bae60e280 4 recover_dentries: writing fnode to omap header 2024-03-24T01:26:28.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.576+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:232 5.1f 5:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba79800 con 0x55917b9e4800 2024-03-24T01:26:28.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.579+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 96 ==== osd_op_reply(232 1.00000000 [omap-set-header] v40'50 uv50 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.579+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:233 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba79c00 con 0x55917b9e4800 2024-03-24T01:26:28.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.581+0000 7f6baba00640 1 -- 172.21.15.90:0/225243987 <== osd.6 v2:172.21.15.120:6816/1454745630 97 ==== osd_op_reply(233 1.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv50 ondisk = 0) v8 ==== 154+0+4 (crc 0 0 0) 0x55917ba1f900 con 0x55917b9e4800 2024-03-24T01:26:28.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.581+0000 7f6bae60e280 4 recover_dentries: updating root 0x0x1 2024-03-24T01:26:28.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.581+0000 7f6bae60e280 4 recover_dentries: object id 1.00000000.inode 2024-03-24T01:26:28.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.581+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:234 5.14 5:287034c4:::1.00000000.inode:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba46000 con 0x55917c438c00 2024-03-24T01:26:28.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.582+0000 7f6bac400640 1 -- 172.21.15.90:0/225243987 <== osd.3 v2:172.21.15.90:6808/2999931983 29 ==== osd_op_reply(234 1.00000000.inode [read 0~582 out=582b] v0'0 uv37 ondisk = 0) v8 ==== 160+0+582 (crc 0 0 0) 0x55917b7e2500 con 0x55917c438c00 2024-03-24T01:26:28.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.582+0000 7f6bae60e280 4 recover_dentries: root exists, will modify (582) 2024-03-24T01:26:28.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.582+0000 7f6bae60e280 4 recover_dentries: magic ok 2024-03-24T01:26:28.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.582+0000 7f6bae60e280 4 recover_dentries: writing root ino 1.00000000.inode version 10 2024-03-24T01:26:28.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.582+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:235 5.14 5:287034c4:::1.00000000.inode:head [writefull 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba46400 con 0x55917c438c00 2024-03-24T01:26:28.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.586+0000 7f6bac400640 1 -- 172.21.15.90:0/225243987 <== osd.3 v2:172.21.15.90:6808/2999931983 30 ==== osd_op_reply(235 1.00000000.inode [writefull 0~582] v40'38 uv38 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x55917b7e2500 con 0x55917c438c00 2024-03-24T01:26:28.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.586+0000 7f6bae60e280 4 main_event: consumed 8 inodes 2024-03-24T01:26:28.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.586+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:236 5.13 5:c91d4a1d:::mds0_inotable:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba46800 con 0x55917c438c00 2024-03-24T01:26:28.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.587+0000 7f6bac400640 1 -- 172.21.15.90:0/225243987 <== osd.3 v2:172.21.15.90:6808/2999931983 31 ==== osd_op_reply(236 mds0_inotable [read 0~34 out=34b] v0'0 uv2 ondisk = 0) v8 ==== 157+0+34 (crc 0 0 0) 0x55917b7e2500 con 0x55917c438c00 2024-03-24T01:26:28.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.587+0000 7f6bae60e280 4 consume_inos: Used ino 0x0x10000000000 requires inotable update 2024-03-24T01:26:28.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.587+0000 7f6bae60e280 4 consume_inos: Used ino 0x0x10000000001 requires inotable update 2024-03-24T01:26:28.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.587+0000 7f6bae60e280 4 consume_inos: Used ino 0x0x10000000002 requires inotable update 2024-03-24T01:26:28.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.587+0000 7f6bae60e280 4 consume_inos: Used ino 0x0x100000001f7 requires inotable update 2024-03-24T01:26:28.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.587+0000 7f6bae60e280 4 consume_inos: Used ino 0x0x100000001f8 requires inotable update 2024-03-24T01:26:28.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.587+0000 7f6bae60e280 4 consume_inos: Used ino 0x0x100000001f9 requires inotable update 2024-03-24T01:26:28.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.587+0000 7f6bae60e280 4 consume_inos: Used ino 0x0x100000001fa requires inotable update 2024-03-24T01:26:28.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.587+0000 7f6bae60e280 4 consume_inos: writing modified inotable version 1 2024-03-24T01:26:28.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.587+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:237 5.13 5:c91d4a1d:::mds0_inotable:head [writefull 0~50 in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x55917ba46c00 con 0x55917c438c00 2024-03-24T01:26:28.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.591+0000 7f6bac400640 1 -- 172.21.15.90:0/225243987 <== osd.3 v2:172.21.15.90:6808/2999931983 32 ==== osd_op_reply(237 mds0_inotable [writefull 0~50] v40'7 uv7 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x55917b7e2500 con 0x55917c438c00 2024-03-24T01:26:28.593 INFO:teuthology.orchestra.run.smithi090.stdout:0.000000 0x400000 LID: () 2024-03-24T01:26:28.593 INFO:teuthology.orchestra.run.smithi090.stdout:2024-03-24T01:25:59.996337+0000 0x400030 SUBTREEMAP: () 2024-03-24T01:26:28.593 INFO:teuthology.orchestra.run.smithi090.stdout:2024-03-24T01:26:13.394658+0000 0x4003ac SESSION: () 2024-03-24T01:26:28.593 INFO:teuthology.orchestra.run.smithi090.stdout:2024-03-24T01:26:14.012037+0000 0x400565 UPDATE: (setattr) 2024-03-24T01:26:28.593 INFO:teuthology.orchestra.run.smithi090.stdout: / 2024-03-24T01:26:28.593 INFO:teuthology.orchestra.run.smithi090.stdout:2024-03-24T01:26:14.075564+0000 0x400894 UPDATE: (openc) 2024-03-24T01:26:28.593 INFO:teuthology.orchestra.run.smithi090.stdout: rootfile 2024-03-24T01:26:28.593 INFO:teuthology.orchestra.run.smithi090.stdout:2024-03-24T01:26:14.131485+0000 0x400f46 UPDATE: (mkdir) 2024-03-24T01:26:28.593 INFO:teuthology.orchestra.run.smithi090.stdout: subdir 2024-03-24T01:26:28.594 INFO:teuthology.orchestra.run.smithi090.stdout:2024-03-24T01:26:14.185402+0000 0x401714 UPDATE: (openc) 2024-03-24T01:26:28.594 INFO:teuthology.orchestra.run.smithi090.stdout: subdir/subdirfile 2024-03-24T01:26:28.594 INFO:teuthology.orchestra.run.smithi090.stdout:2024-03-24T01:26:14.240979+0000 0x401ec0 UPDATE: (mkdir) 2024-03-24T01:26:28.594 INFO:teuthology.orchestra.run.smithi090.stdout: linkdir 2024-03-24T01:26:28.594 INFO:teuthology.orchestra.run.smithi090.stdout:2024-03-24T01:26:14.292410+0000 0x40269f UPDATE: (openc) 2024-03-24T01:26:28.594 INFO:teuthology.orchestra.run.smithi090.stdout: linkdir/link0 2024-03-24T01:26:28.594 INFO:teuthology.orchestra.run.smithi090.stdout:2024-03-24T01:26:14.342247+0000 0x402e47 UPDATE: (cap update) 2024-03-24T01:26:28.594 INFO:teuthology.orchestra.run.smithi090.stdout: linkdir/link0 2024-03-24T01:26:28.594 INFO:teuthology.orchestra.run.smithi090.stdout:2024-03-24T01:26:14.348108+0000 0x4035ce UPDATE: (unlink_local) 2024-03-24T01:26:28.594 INFO:teuthology.orchestra.run.smithi090.stdout: stray2/100000001f8 2024-03-24T01:26:28.594 INFO:teuthology.orchestra.run.smithi090.stdout: linkdir/link0 2024-03-24T01:26:28.594 INFO:teuthology.orchestra.run.smithi090.stdout:2024-03-24T01:26:14.397169+0000 0x404234 UPDATE: (scatter_writebehind) 2024-03-24T01:26:28.595 INFO:teuthology.orchestra.run.smithi090.stdout:2024-03-24T01:26:14.398287+0000 0x4046a6 UPDATE: (cap update) 2024-03-24T01:26:28.595 INFO:teuthology.orchestra.run.smithi090.stdout: subdir/subdirfile 2024-03-24T01:26:28.595 INFO:teuthology.orchestra.run.smithi090.stdout:2024-03-24T01:26:14.398996+0000 0x404e31 UPDATE: (link_local) 2024-03-24T01:26:28.595 INFO:teuthology.orchestra.run.smithi090.stdout: subdir/subdirfile 2024-03-24T01:26:28.595 INFO:teuthology.orchestra.run.smithi090.stdout: linkdir/link0 2024-03-24T01:26:28.595 INFO:teuthology.orchestra.run.smithi090.stdout:2024-03-24T01:26:14.449993+0000 0x405991 UPDATE: (cap update) 2024-03-24T01:26:28.595 INFO:teuthology.orchestra.run.smithi090.stdout: subdir/subdirfile 2024-03-24T01:26:28.595 INFO:teuthology.orchestra.run.smithi090.stdout:2024-03-24T01:26:14.450651+0000 0x406177 UPDATE: (cap update) 2024-03-24T01:26:28.595 INFO:teuthology.orchestra.run.smithi090.stdout: subdir/subdirfile 2024-03-24T01:26:28.595 INFO:teuthology.orchestra.run.smithi090.stdout:2024-03-24T01:26:14.451541+0000 0x406937 UPDATE: (link_local) 2024-03-24T01:26:28.595 INFO:teuthology.orchestra.run.smithi090.stdout: subdir/subdirfile 2024-03-24T01:26:28.595 INFO:teuthology.orchestra.run.smithi090.stdout: linkdir/link1 2024-03-24T01:26:28.595 INFO:teuthology.orchestra.run.smithi090.stdout:2024-03-24T01:26:14.500907+0000 0x407497 UPDATE: (link_local) 2024-03-24T01:26:28.595 INFO:teuthology.orchestra.run.smithi090.stdout: subdir/subdirfile 2024-03-24T01:26:28.595 INFO:teuthology.orchestra.run.smithi090.stdout: linkdir/link2 2024-03-24T01:26:28.596 INFO:teuthology.orchestra.run.smithi090.stdout:2024-03-24T01:26:14.555062+0000 0x407ff7 UPDATE: (unlink_local) 2024-03-24T01:26:28.596 INFO:teuthology.orchestra.run.smithi090.stdout: subdir/subdirfile 2024-03-24T01:26:28.596 INFO:teuthology.orchestra.run.smithi090.stdout: linkdir/link2 2024-03-24T01:26:28.596 INFO:teuthology.orchestra.run.smithi090.stdout:2024-03-24T01:26:14.609004+0000 0x408b5a UPDATE: (openc) 2024-03-24T01:26:28.596 INFO:teuthology.orchestra.run.smithi090.stdout: linkdir/link2 2024-03-24T01:26:28.596 INFO:teuthology.orchestra.run.smithi090.stdout:2024-03-24T01:26:14.672302+0000 0x409302 UPDATE: (link_local) 2024-03-24T01:26:28.596 INFO:teuthology.orchestra.run.smithi090.stdout: subdir/subdirfile 2024-03-24T01:26:28.596 INFO:teuthology.orchestra.run.smithi090.stdout: linkdir/link3 2024-03-24T01:26:28.596 INFO:teuthology.orchestra.run.smithi090.stdout:2024-03-24T01:26:14.733339+0000 0x409e70 UPDATE: (unlink_local) 2024-03-24T01:26:28.596 INFO:teuthology.orchestra.run.smithi090.stdout: subdir/subdirfile 2024-03-24T01:26:28.596 INFO:teuthology.orchestra.run.smithi090.stdout: linkdir/link3 2024-03-24T01:26:28.596 INFO:teuthology.orchestra.run.smithi090.stdout:2024-03-24T01:26:14.797905+0000 0x40a9d3 UPDATE: (cap update) 2024-03-24T01:26:28.596 INFO:teuthology.orchestra.run.smithi090.stdout: rootfile 2024-03-24T01:26:28.596 INFO:teuthology.orchestra.run.smithi090.stdout:2024-03-24T01:26:14.798525+0000 0x40b054 UPDATE: (link_local) 2024-03-24T01:26:28.596 INFO:teuthology.orchestra.run.smithi090.stdout: rootfile 2024-03-24T01:26:28.597 INFO:teuthology.orchestra.run.smithi090.stdout: linkdir/link3 2024-03-24T01:26:28.597 INFO:teuthology.orchestra.run.smithi090.stdout:2024-03-24T01:26:14.846156+0000 0x40baaa UPDATE: (mkdir) 2024-03-24T01:26:28.597 INFO:teuthology.orchestra.run.smithi090.stdout: subdir/subsubdir 2024-03-24T01:26:28.597 INFO:teuthology.orchestra.run.smithi090.stdout:2024-03-24T01:26:17.028764+0000 0x40c383 UPDATE: (purge_stray) 2024-03-24T01:26:28.597 INFO:teuthology.orchestra.run.smithi090.stdout: 100000001f8 2024-03-24T01:26:28.597 INFO:teuthology.orchestra.run.smithi090.stdout:2024-03-24T01:26:20.396048+0000 0x40c5da UPDATE: (cap update) 2024-03-24T01:26:28.597 INFO:teuthology.orchestra.run.smithi090.stdout: linkdir/link2 2024-03-24T01:26:28.597 INFO:teuthology.orchestra.run.smithi090.stdout:2024-03-24T01:26:21.161248+0000 0x40cfaa UPDATE: (scatter_writebehind) 2024-03-24T01:26:28.597 INFO:teuthology.orchestra.run.smithi090.stdout: stray2 2024-03-24T01:26:28.597 INFO:teuthology.orchestra.run.smithi090.stdout:2024-03-24T01:26:21.167143+0000 0x40d61d UPDATE: (scatter_writebehind) 2024-03-24T01:26:28.597 INFO:teuthology.orchestra.run.smithi090.stdout:2024-03-24T01:26:21.168468+0000 0x40da8f SESSION: () 2024-03-24T01:26:28.597 INFO:teuthology.orchestra.run.smithi090.stdout:0.000000 0x40db3c PURGED: () 2024-03-24T01:26:28.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.592+0000 7f6bae60e280 4 31 events 2024-03-24T01:26:28.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.592+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55917c438c00 msgr2=0x55917b6cd180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:28.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.592+0000 7f6bae60e280 1 --2- 172.21.15.90:0/225243987 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55917c438c00 0x55917b6cd180 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.592+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 >> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] conn(0x55917c439400 msgr2=0x55917b6ccc00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:28.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.592+0000 7f6bae60e280 1 --2- 172.21.15.90:0/225243987 >> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] conn(0x55917c439400 0x55917b6ccc00 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.592+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x55917b9e4800 msgr2=0x55917ba04000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:28.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.592+0000 7f6bae60e280 1 --2- 172.21.15.90:0/225243987 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x55917b9e4800 0x55917ba04000 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.592+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 >> [v2:172.21.15.120:6808/1045862942,v1:172.21.15.120:6809/1045862942] conn(0x55917ba16c00 msgr2=0x55917ba04580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:28.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.592+0000 7f6bae60e280 1 --2- 172.21.15.90:0/225243987 >> [v2:172.21.15.120:6808/1045862942,v1:172.21.15.120:6809/1045862942] conn(0x55917ba16c00 0x55917ba04580 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.592+0000 7f6bab000640 1 -- 172.21.15.90:0/225243987 reap_dead start 2024-03-24T01:26:28.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.592+0000 7f6bab000640 1 -- 172.21.15.90:0/225243987 reap_dead start 2024-03-24T01:26:28.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.593+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55917c59b400 msgr2=0x55917b6cd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:28.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.593+0000 7f6bae60e280 1 --2- 172.21.15.90:0/225243987 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55917c59b400 0x55917b6cd700 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x55917c3f27b0 tx=0x55917b9d0060 comp rx=0 tx=0).stop 2024-03-24T01:26:28.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.593+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55917c52fc00 msgr2=0x55917b6cc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:28.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.593+0000 7f6bae60e280 1 --2- 172.21.15.90:0/225243987 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55917c52fc00 0x55917b6cc680 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x55917c544990 tx=0x55917c3f6a20 comp rx=0 tx=0).stop 2024-03-24T01:26:28.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.593+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 shutdown_connections 2024-03-24T01:26:28.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.593+0000 7f6bae60e280 1 --2- 172.21.15.90:0/225243987 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55917c59b400 0x55917b6cd700 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.593+0000 7f6bae60e280 1 --2- 172.21.15.90:0/225243987 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55917c52fc00 0x55917b6cc680 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.593+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 >> 172.21.15.90:0/225243987 conn(0x55917c52f000 msgr2=0x55917b746800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:28.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.593+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 shutdown_connections 2024-03-24T01:26:28.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.594+0000 7f6bae60e280 1 -- 172.21.15.90:0/225243987 wait complete. 2024-03-24T01:26:28.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.595+0000 7f6bae60e280 1 -- 172.21.15.90:0/1686742849 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55917b660c00 msgr2=0x55917b6ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:28.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.595+0000 7f6bae60e280 1 --2- 172.21.15.90:0/1686742849 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55917b660c00 0x55917b6ca580 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x55917c471530 tx=0x55917c3f6600 comp rx=0 tx=0).stop 2024-03-24T01:26:28.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.595+0000 7f6bae60e280 1 -- 172.21.15.90:0/1686742849 shutdown_connections 2024-03-24T01:26:28.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.595+0000 7f6bae60e280 1 --2- 172.21.15.90:0/1686742849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55917b661c00 0x55917b6cb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.595+0000 7f6bae60e280 1 --2- 172.21.15.90:0/1686742849 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55917b661800 0x55917b6cab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.595+0000 7f6bae60e280 1 --2- 172.21.15.90:0/1686742849 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55917b660c00 0x55917b6ca580 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.595+0000 7f6bae60e280 1 -- 172.21.15.90:0/1686742849 >> 172.21.15.90:0/1686742849 conn(0x55917b661000 msgr2=0x55917b747000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:28.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.595+0000 7f6bae60e280 1 -- 172.21.15.90:0/1686742849 shutdown_connections 2024-03-24T01:26:28.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.595+0000 7f6bae60e280 1 -- 172.21.15.90:0/1686742849 wait complete. 2024-03-24T01:26:28.605 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-journal-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '--rank', 'cephfs:0', 'event', 'recover_dentries', 'list'] in time 0:00:00.706346, result: 0.000000 0x400000 LID: () 2024-03-24T01:25:59.996337+0000 0x400030 SUBTREEMAP: () 2024-03-24T01:26:13.394658+0000 0x4003ac SESSION: () 2024-03-24T01:26:14.012037+0000 0x400565 UPDATE: (setattr) / 2024-03-24T01:26:14.075564+0000 0x400894 UPDATE: (openc) rootfile 2024-03-24T01:26:14.131485+0000 0x400f46 UPDATE: (mkdir) subdir 2024-03-24T01:26:14.185402+0000 0x401714 UPDATE: (openc) subdir/subdirfile 2024-03-24T01:26:14.240979+0000 0x401ec0 UPDATE: (mkdir) linkdir 2024-03-24T01:26:14.292410+0000 0x40269f UPDATE: (openc) linkdir/link0 2024-03-24T01:26:14.342247+0000 0x402e47 UPDATE: (cap update) linkdir/link0 2024-03-24T01:26:14.348108+0000 0x4035ce UPDATE: (unlink_local) stray2/100000001f8 linkdir/link0 2024-03-24T01:26:14.397169+0000 0x404234 UPDATE: (scatter_writebehind) 2024-03-24T01:26:14.398287+0000 0x4046a6 UPDATE: (cap update) subdir/subdirfile 2024-03-24T01:26:14.398996+0000 0x404e31 UPDATE: (link_local) subdir/subdirfile linkdir/link0 2024-03-24T01:26:14.449993+0000 0x405991 UPDATE: (cap update) subdir/subdirfile 2024-03-24T01:26:14.450651+0000 0x406177 UPDATE: (cap update) subdir/subdirfile 2024-03-24T01:26:14.451541+0000 0x406937 UPDATE: (link_local) subdir/subdirfile linkdir/link1 2024-03-24T01:26:14.500907+0000 0x407497 UPDATE: (link_local) subdir/subdirfile linkdir/link2 2024-03-24T01:26:14.555062+0000 0x407ff7 UPDATE: (unlink_local) subdir/subdirfile linkdir/link2 2024-03-24T01:26:14.609004+0000 0x408b5a UPDATE: (openc) linkdir/link2 2024-03-24T01:26:14.672302+0000 0x409302 UPDATE: (link_local) subdir/subdirfile linkdir/link3 2024-03-24T01:26:14.733339+0000 0x409e70 UPDATE: (unlink_local) subdir/subdirfile linkdir/link3 2024-03-24T01:26:14.797905+0000 0x40a9d3 UPDATE: (cap update) rootfile 2024-03-24T01:26:14.798525+0000 0x40b054 UPDATE: (link_local) rootfile linkdir/link3 2024-03-24T01:26:14.846156+0000 0x40baaa UPDATE: (mkdir) subdir/subsubdir 2024-03-24T01:26:17.028764+0000 0x40c383 UPDATE: (purge_stray) 100000001f8 2024-03-24T01:26:20.396048+0000 0x40c5da UPDATE: (cap update) linkdir/link2 2024-03-24T01:26:21.161248+0000 0x40cfaa UPDATE: (scatter_writebehind) stray2 2024-03-24T01:26:21.167143+0000 0x40d61d UPDATE: (scatter_writebehind) 2024-03-24T01:26:21.168468+0000 0x40da8f SESSION: () 0.000000 0x40db3c PURGED: () 2024-03-24T01:26:28.606 DEBUG:teuthology.orchestra.run.smithi090:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 1.00000000 2024-03-24T01:26:28.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.754+0000 7fae37940f40 1 Processor -- start 2024-03-24T01:26:28.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.754+0000 7fae37940f40 1 -- start start 2024-03-24T01:26:28.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.755+0000 7fae37940f40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b06e56bb80 0x55b06e4e1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:28.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.755+0000 7fae37940f40 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b06e580be0 0x55b06e4dd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:28.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.755+0000 7fae37940f40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b06e5874d0 0x55b06e4ae400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:28.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.755+0000 7fae37940f40 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b06e455ff0 con 0x55b06e580be0 2024-03-24T01:26:28.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.755+0000 7fae37940f40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b06e56c4d0 con 0x55b06e56bb80 2024-03-24T01:26:28.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.755+0000 7fae37940f40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b06e569860 con 0x55b06e5874d0 2024-03-24T01:26:28.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.755+0000 7fae34200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b06e580be0 0x55b06e4dd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:28.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.755+0000 7fae34200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b06e580be0 0x55b06e4dd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49900/0 (socket says 172.21.15.90:49900) 2024-03-24T01:26:28.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.755+0000 7fae34200640 1 -- 172.21.15.90:0/2374754445 learned_addr learned my addr 172.21.15.90:0/2374754445 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:28.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.755+0000 7fae34200640 1 -- 172.21.15.90:0/2374754445 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b06e5874d0 msgr2=0x55b06e4ae400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:28.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.755+0000 7fae35600640 1 --2- 172.21.15.90:0/2374754445 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b06e5874d0 0x55b06e4ae400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:28.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.755+0000 7fae34c00640 1 --2- 172.21.15.90:0/2374754445 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b06e56bb80 0x55b06e4e1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:28.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.755+0000 7fae34200640 1 --2- 172.21.15.90:0/2374754445 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b06e5874d0 0x55b06e4ae400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.755+0000 7fae34200640 1 -- 172.21.15.90:0/2374754445 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b06e56bb80 msgr2=0x55b06e4e1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:28.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.755+0000 7fae34200640 1 --2- 172.21.15.90:0/2374754445 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b06e56bb80 0x55b06e4e1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.755+0000 7fae34200640 1 -- 172.21.15.90:0/2374754445 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b06e3f1dc0 con 0x55b06e580be0 2024-03-24T01:26:28.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.756+0000 7fae34200640 1 --2- 172.21.15.90:0/2374754445 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b06e580be0 0x55b06e4dd3a0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fae1c00a430 tx=0x7fae1c00a8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a46016c6af74b77 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:28.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.756+0000 7fae32e00640 1 -- 172.21.15.90:0/2374754445 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1c00b920 con 0x55b06e580be0 2024-03-24T01:26:28.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.756+0000 7fae32e00640 1 -- 172.21.15.90:0/2374754445 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae1c00f030 con 0x55b06e580be0 2024-03-24T01:26:28.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.756+0000 7fae32e00640 1 -- 172.21.15.90:0/2374754445 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1c00bd20 con 0x55b06e580be0 2024-03-24T01:26:28.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.756+0000 7fae37940f40 1 -- 172.21.15.90:0/2374754445 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b06e580be0 msgr2=0x55b06e4dd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:28.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.756+0000 7fae37940f40 1 --2- 172.21.15.90:0/2374754445 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b06e580be0 0x55b06e4dd3a0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fae1c00a430 tx=0x7fae1c00a8f0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.756+0000 7fae37940f40 1 -- 172.21.15.90:0/2374754445 shutdown_connections 2024-03-24T01:26:28.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.756+0000 7fae37940f40 1 --2- 172.21.15.90:0/2374754445 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b06e580be0 0x55b06e4dd3a0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.756+0000 7fae37940f40 1 --2- 172.21.15.90:0/2374754445 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b06e5874d0 0x55b06e4ae400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.756+0000 7fae37940f40 1 --2- 172.21.15.90:0/2374754445 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b06e56bb80 0x55b06e4e1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.756+0000 7fae37940f40 1 -- 172.21.15.90:0/2374754445 >> 172.21.15.90:0/2374754445 conn(0x55b06e4fcf60 msgr2=0x55b06e4dde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:28.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.757+0000 7fae37940f40 1 -- 172.21.15.90:0/2374754445 shutdown_connections 2024-03-24T01:26:28.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.757+0000 7fae37940f40 1 -- 172.21.15.90:0/2374754445 wait complete. 2024-03-24T01:26:28.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.757+0000 7fae37940f40 1 Processor -- start 2024-03-24T01:26:28.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.758+0000 7fae37940f40 1 -- start start 2024-03-24T01:26:28.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.758+0000 7fae37940f40 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b06e56bb80 0x55b06e5685c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:28.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.758+0000 7fae37940f40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b06e580be0 0x55b06e5837a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:28.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.758+0000 7fae37940f40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b06e5874d0 0x55b06e583ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:28.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.758+0000 7fae37940f40 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b06e5695c0 con 0x55b06e56bb80 2024-03-24T01:26:28.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.758+0000 7fae37940f40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b06e57f4f0 con 0x55b06e580be0 2024-03-24T01:26:28.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.758+0000 7fae37940f40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b06e57ee90 con 0x55b06e5874d0 2024-03-24T01:26:28.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.759+0000 7fae34c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b06e56bb80 0x55b06e5685c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:28.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.759+0000 7fae34c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b06e56bb80 0x55b06e5685c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49916/0 (socket says 172.21.15.90:49916) 2024-03-24T01:26:28.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.759+0000 7fae34c00640 1 -- 172.21.15.90:0/4190125477 learned_addr learned my addr 172.21.15.90:0/4190125477 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:28.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.759+0000 7fae34c00640 1 -- 172.21.15.90:0/4190125477 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b06e5874d0 msgr2=0x55b06e583ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:28.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.759+0000 7fae35600640 1 --2- 172.21.15.90:0/4190125477 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b06e5874d0 0x55b06e583ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:28.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.759+0000 7fae34200640 1 --2- 172.21.15.90:0/4190125477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b06e580be0 0x55b06e5837a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:28.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.759+0000 7fae34c00640 1 --2- 172.21.15.90:0/4190125477 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b06e5874d0 0x55b06e583ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.759+0000 7fae34c00640 1 -- 172.21.15.90:0/4190125477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b06e580be0 msgr2=0x55b06e5837a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:28.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.759+0000 7fae34c00640 1 --2- 172.21.15.90:0/4190125477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b06e580be0 0x55b06e5837a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.759+0000 7fae34c00640 1 -- 172.21.15.90:0/4190125477 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b06e4fbf00 con 0x55b06e56bb80 2024-03-24T01:26:28.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.759+0000 7fae34c00640 1 --2- 172.21.15.90:0/4190125477 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b06e56bb80 0x55b06e5685c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fae2400a4a0 tx=0x7fae2400a960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:28.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.760+0000 7fae31a00640 1 -- 172.21.15.90:0/4190125477 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae24015020 con 0x55b06e56bb80 2024-03-24T01:26:28.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.760+0000 7fae31a00640 1 -- 172.21.15.90:0/4190125477 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae240118b0 con 0x55b06e56bb80 2024-03-24T01:26:28.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.760+0000 7fae31a00640 1 -- 172.21.15.90:0/4190125477 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae24011ca0 con 0x55b06e56bb80 2024-03-24T01:26:28.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.760+0000 7fae37940f40 1 -- 172.21.15.90:0/4190125477 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55b06e4fc1e0 con 0x55b06e56bb80 2024-03-24T01:26:28.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.760+0000 7fae37940f40 1 -- 172.21.15.90:0/4190125477 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b06e566f80 con 0x55b06e56bb80 2024-03-24T01:26:28.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.762+0000 7fae31a00640 1 -- 172.21.15.90:0/4190125477 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fae24018020 con 0x55b06e56bb80 2024-03-24T01:26:28.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.763+0000 7fae31a00640 1 --2- 172.21.15.90:0/4190125477 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fae1006a5b0 0x7fae1006ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:28.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.763+0000 7fae31a00640 1 -- 172.21.15.90:0/4190125477 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fae24089f10 con 0x55b06e56bb80 2024-03-24T01:26:28.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.763+0000 7fae37940f40 1 --2- 172.21.15.90:0/4190125477 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x55b06e4f18a0 0x55b06e582380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:28.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.763+0000 7fae37940f40 1 -- 172.21.15.90:0/4190125477 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:1 5.1f 5:ff5b34d6:::1.00000000:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x55b06e568b00 con 0x55b06e4f18a0 2024-03-24T01:26:28.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.763+0000 7fae34200640 1 --2- 172.21.15.90:0/4190125477 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fae1006a5b0 0x7fae1006ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:28.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.764+0000 7fae35600640 1 --2- 172.21.15.90:0/4190125477 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x55b06e4f18a0 0x55b06e582380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:28.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.765+0000 7fae34200640 1 --2- 172.21.15.90:0/4190125477 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fae1006a5b0 0x7fae1006ca70 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x55b06e566040 tx=0x7fae1c016040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:28.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.765+0000 7fae35600640 1 --2- 172.21.15.90:0/4190125477 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x55b06e4f18a0 0x55b06e582380 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:28.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.766+0000 7fae35600640 1 -- 172.21.15.90:0/4190125477 <== osd.6 v2:172.21.15.120:6816/1454745630 1 ==== osd_op_reply(1 1.00000000 [omap-get-keys out=53b] v0'0 uv50 ondisk = 0) v8 ==== 154+0+53 (crc 0 0 0) 0x7fae2c05d040 con 0x55b06e4f18a0 2024-03-24T01:26:28.766 INFO:teuthology.orchestra.run.smithi090.stdout:linkdir_head 2024-03-24T01:26:28.766 INFO:teuthology.orchestra.run.smithi090.stdout:rootfile_head 2024-03-24T01:26:28.766 INFO:teuthology.orchestra.run.smithi090.stdout:subdir_head 2024-03-24T01:26:28.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.766+0000 7fae37940f40 1 -- 172.21.15.90:0/4190125477 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x55b06e4f18a0 msgr2=0x55b06e582380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:28.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.766+0000 7fae37940f40 1 --2- 172.21.15.90:0/4190125477 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x55b06e4f18a0 0x55b06e582380 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.766+0000 7fae37940f40 1 -- 172.21.15.90:0/4190125477 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fae1006a5b0 msgr2=0x7fae1006ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:28.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.766+0000 7fae37940f40 1 --2- 172.21.15.90:0/4190125477 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fae1006a5b0 0x7fae1006ca70 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x55b06e566040 tx=0x7fae1c016040 comp rx=0 tx=0).stop 2024-03-24T01:26:28.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.766+0000 7fae37940f40 1 -- 172.21.15.90:0/4190125477 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b06e56bb80 msgr2=0x55b06e5685c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:28.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.766+0000 7fae37940f40 1 --2- 172.21.15.90:0/4190125477 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b06e56bb80 0x55b06e5685c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fae2400a4a0 tx=0x7fae2400a960 comp rx=0 tx=0).stop 2024-03-24T01:26:28.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.766+0000 7fae35600640 1 -- 172.21.15.90:0/4190125477 reap_dead start 2024-03-24T01:26:28.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.766+0000 7fae37940f40 1 -- 172.21.15.90:0/4190125477 shutdown_connections 2024-03-24T01:26:28.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.766+0000 7fae37940f40 1 -- 172.21.15.90:0/4190125477 >> 172.21.15.90:0/4190125477 conn(0x55b06e4fcf60 msgr2=0x55b06e57f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:28.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.766+0000 7fae37940f40 1 -- 172.21.15.90:0/4190125477 shutdown_connections 2024-03-24T01:26:28.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.766+0000 7fae37940f40 1 -- 172.21.15.90:0/4190125477 wait complete. 2024-03-24T01:26:28.771 DEBUG:teuthology.orchestra.run.smithi090:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 10000000001.00000000 2024-03-24T01:26:28.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.858+0000 7f07e9eb4f40 1 Processor -- start 2024-03-24T01:26:28.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.858+0000 7f07e9eb4f40 1 -- start start 2024-03-24T01:26:28.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.859+0000 7f07e9eb4f40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5560bf8d6b80 0x5560bf84bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:28.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.859+0000 7f07e9eb4f40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5560bf8ebbe0 0x5560bf8d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:28.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.859+0000 7f07e9eb4f40 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560bf8f24d0 0x5560bf8f3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:28.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.859+0000 7f07e9eb4f40 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560bf7c0ff0 con 0x5560bf8f24d0 2024-03-24T01:26:28.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.859+0000 7f07e9eb4f40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560bf8d74d0 con 0x5560bf8d6b80 2024-03-24T01:26:28.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.859+0000 7f07e9eb4f40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5560bf84d400 con 0x5560bf8ebbe0 2024-03-24T01:26:28.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.859+0000 7f07e7a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560bf8f24d0 0x5560bf8f3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:28.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.859+0000 7f07e7a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560bf8f24d0 0x5560bf8f3c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49946/0 (socket says 172.21.15.90:49946) 2024-03-24T01:26:28.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.859+0000 7f07e7a00640 1 -- 172.21.15.90:0/3683227886 learned_addr learned my addr 172.21.15.90:0/3683227886 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:28.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.859+0000 7f07e7a00640 1 -- 172.21.15.90:0/3683227886 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5560bf8ebbe0 msgr2=0x5560bf8d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:28.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.859+0000 7f07e7a00640 1 --2- 172.21.15.90:0/3683227886 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5560bf8ebbe0 0x5560bf8d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.859+0000 7f07e7a00640 1 -- 172.21.15.90:0/3683227886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5560bf8d6b80 msgr2=0x5560bf84bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:28.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.859+0000 7f07e7a00640 1 --2- 172.21.15.90:0/3683227886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5560bf8d6b80 0x5560bf84bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.859+0000 7f07e7a00640 1 -- 172.21.15.90:0/3683227886 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560bf75cdc0 con 0x5560bf8f24d0 2024-03-24T01:26:28.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.860+0000 7f07e7a00640 1 --2- 172.21.15.90:0/3683227886 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560bf8f24d0 0x5560bf8f3c60 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f07e000c4b0 tx=0x7f07e000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6875add1e9f86819 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:28.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.860+0000 7f07e5c00640 1 -- 172.21.15.90:0/3683227886 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e00116b0 con 0x5560bf8f24d0 2024-03-24T01:26:28.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.860+0000 7f07e5c00640 1 -- 172.21.15.90:0/3683227886 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07e0017070 con 0x5560bf8f24d0 2024-03-24T01:26:28.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.860+0000 7f07e5c00640 1 -- 172.21.15.90:0/3683227886 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e0011a90 con 0x5560bf8f24d0 2024-03-24T01:26:28.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.860+0000 7f07e9eb4f40 1 -- 172.21.15.90:0/3683227886 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560bf8f24d0 msgr2=0x5560bf8f3c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:28.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.860+0000 7f07e9eb4f40 1 --2- 172.21.15.90:0/3683227886 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560bf8f24d0 0x5560bf8f3c60 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f07e000c4b0 tx=0x7f07e000c970 comp rx=0 tx=0).stop 2024-03-24T01:26:28.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.860+0000 7f07e9eb4f40 1 -- 172.21.15.90:0/3683227886 shutdown_connections 2024-03-24T01:26:28.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.860+0000 7f07e9eb4f40 1 --2- 172.21.15.90:0/3683227886 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560bf8f24d0 0x5560bf8f3c60 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.860+0000 7f07e9eb4f40 1 --2- 172.21.15.90:0/3683227886 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5560bf8ebbe0 0x5560bf8d18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.860+0000 7f07e9eb4f40 1 --2- 172.21.15.90:0/3683227886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5560bf8d6b80 0x5560bf84bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.860+0000 7f07e9eb4f40 1 -- 172.21.15.90:0/3683227886 >> 172.21.15.90:0/3683227886 conn(0x5560bf867f60 msgr2=0x5560bf84a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:28.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.861+0000 7f07e9eb4f40 1 -- 172.21.15.90:0/3683227886 shutdown_connections 2024-03-24T01:26:28.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.861+0000 7f07e9eb4f40 1 -- 172.21.15.90:0/3683227886 wait complete. 2024-03-24T01:26:28.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.861+0000 7f07e9eb4f40 1 Processor -- start 2024-03-24T01:26:28.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.862+0000 7f07e9eb4f40 1 -- start start 2024-03-24T01:26:28.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.862+0000 7f07e9eb4f40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5560bf8d6b80 0x5560bf8e7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:28.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.862+0000 7f07e9eb4f40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5560bf8ebbe0 0x5560bf8f30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:28.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.862+0000 7f07e9eb4f40 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560bf8f24d0 0x5560bf8f3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:28.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.862+0000 7f07e9eb4f40 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560bf8eafb0 con 0x5560bf8f24d0 2024-03-24T01:26:28.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.862+0000 7f07e9eb4f40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560bf8f5870 con 0x5560bf8ebbe0 2024-03-24T01:26:28.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.862+0000 7f07e9eb4f40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5560bf8eb250 con 0x5560bf8d6b80 2024-03-24T01:26:28.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.863+0000 7f07e7a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560bf8f24d0 0x5560bf8f3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:28.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.863+0000 7f07e7a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560bf8f24d0 0x5560bf8f3610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49956/0 (socket says 172.21.15.90:49956) 2024-03-24T01:26:28.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.863+0000 7f07e7a00640 1 -- 172.21.15.90:0/969582967 learned_addr learned my addr 172.21.15.90:0/969582967 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:28.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.863+0000 7f07e7000640 1 --2- 172.21.15.90:0/969582967 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5560bf8d6b80 0x5560bf8e7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:28.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.863+0000 7f07e7a00640 1 -- 172.21.15.90:0/969582967 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5560bf8d6b80 msgr2=0x5560bf8e7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:28.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.863+0000 7f07dfe00640 1 --2- 172.21.15.90:0/969582967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5560bf8ebbe0 0x5560bf8f30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:28.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.863+0000 7f07e7a00640 1 --2- 172.21.15.90:0/969582967 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5560bf8d6b80 0x5560bf8e7640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.863+0000 7f07e7a00640 1 -- 172.21.15.90:0/969582967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5560bf8ebbe0 msgr2=0x5560bf8f30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:28.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.863+0000 7f07e7a00640 1 --2- 172.21.15.90:0/969582967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5560bf8ebbe0 0x5560bf8f30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.863+0000 7f07e7a00640 1 -- 172.21.15.90:0/969582967 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560bf84b2c0 con 0x5560bf8f24d0 2024-03-24T01:26:28.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.863+0000 7f07dfe00640 1 --2- 172.21.15.90:0/969582967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5560bf8ebbe0 0x5560bf8f30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:28.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.863+0000 7f07e7a00640 1 --2- 172.21.15.90:0/969582967 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560bf8f24d0 0x5560bf8f3610 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f07e000c210 tx=0x7f07e00033a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:28.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.864+0000 7f07df400640 1 -- 172.21.15.90:0/969582967 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e0012bc0 con 0x5560bf8f24d0 2024-03-24T01:26:28.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.864+0000 7f07df400640 1 -- 172.21.15.90:0/969582967 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07e0012d60 con 0x5560bf8f24d0 2024-03-24T01:26:28.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.864+0000 7f07e9eb4f40 1 -- 172.21.15.90:0/969582967 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5560bf819530 con 0x5560bf8f24d0 2024-03-24T01:26:28.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.864+0000 7f07df400640 1 -- 172.21.15.90:0/969582967 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e0025650 con 0x5560bf8f24d0 2024-03-24T01:26:28.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.864+0000 7f07e9eb4f40 1 -- 172.21.15.90:0/969582967 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5560bf8ea0c0 con 0x5560bf8f24d0 2024-03-24T01:26:28.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.866+0000 7f07df400640 1 -- 172.21.15.90:0/969582967 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f07e0023020 con 0x5560bf8f24d0 2024-03-24T01:26:28.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.867+0000 7f07df400640 1 --2- 172.21.15.90:0/969582967 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f07c406a5b0 0x7f07c406ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:28.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.867+0000 7f07df400640 1 -- 172.21.15.90:0/969582967 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f07e008b0f0 con 0x5560bf8f24d0 2024-03-24T01:26:28.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.867+0000 7f07e9eb4f40 1 --2- 172.21.15.90:0/969582967 >> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] conn(0x5560bf866ea0 0x5560bf8ce470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:28.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.867+0000 7f07e9eb4f40 1 -- 172.21.15.90:0/969582967 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:1 5.7 5:e2e07930:::10000000001.00000000:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x5560bf8f5060 con 0x5560bf866ea0 2024-03-24T01:26:28.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.868+0000 7f07e7000640 1 --2- 172.21.15.90:0/969582967 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f07c406a5b0 0x7f07c406ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:28.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.868+0000 7f07dfe00640 1 --2- 172.21.15.90:0/969582967 >> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] conn(0x5560bf866ea0 0x5560bf8ce470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:28.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.869+0000 7f07e7000640 1 --2- 172.21.15.90:0/969582967 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f07c406a5b0 0x7f07c406ca70 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f07d8057000 tx=0x7f07d804f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:28.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.869+0000 7f07dfe00640 1 --2- 172.21.15.90:0/969582967 >> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] conn(0x5560bf866ea0 0x5560bf8ce470 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:28.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.870+0000 7f07dfe00640 1 -- 172.21.15.90:0/969582967 <== osd.5 v2:172.21.15.120:6824/1739969887 1 ==== osd_op_reply(1 10000000001.00000000 [omap-get-keys out=42b] v0'0 uv25 ondisk = 0) v8 ==== 164+0+42 (crc 0 0 0) 0x7f07d000f3a0 con 0x5560bf866ea0 2024-03-24T01:26:28.871 INFO:teuthology.orchestra.run.smithi090.stdout:subdirfile_head 2024-03-24T01:26:28.871 INFO:teuthology.orchestra.run.smithi090.stdout:subsubdir_head 2024-03-24T01:26:28.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.870+0000 7f07e9eb4f40 1 -- 172.21.15.90:0/969582967 >> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] conn(0x5560bf866ea0 msgr2=0x5560bf8ce470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:28.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.870+0000 7f07e9eb4f40 1 --2- 172.21.15.90:0/969582967 >> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] conn(0x5560bf866ea0 0x5560bf8ce470 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.871+0000 7f07e9eb4f40 1 -- 172.21.15.90:0/969582967 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f07c406a5b0 msgr2=0x7f07c406ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:28.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.871+0000 7f07e9eb4f40 1 --2- 172.21.15.90:0/969582967 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f07c406a5b0 0x7f07c406ca70 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f07d8057000 tx=0x7f07d804f000 comp rx=0 tx=0).stop 2024-03-24T01:26:28.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.871+0000 7f07e9eb4f40 1 -- 172.21.15.90:0/969582967 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560bf8f24d0 msgr2=0x5560bf8f3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:28.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.871+0000 7f07e9eb4f40 1 --2- 172.21.15.90:0/969582967 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5560bf8f24d0 0x5560bf8f3610 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f07e000c210 tx=0x7f07e00033a0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.871+0000 7f07e7a00640 1 -- 172.21.15.90:0/969582967 reap_dead start 2024-03-24T01:26:28.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.871+0000 7f07e9eb4f40 1 -- 172.21.15.90:0/969582967 shutdown_connections 2024-03-24T01:26:28.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.871+0000 7f07e9eb4f40 1 -- 172.21.15.90:0/969582967 >> 172.21.15.90:0/969582967 conn(0x5560bf867f60 msgr2=0x5560bf84a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:28.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.871+0000 7f07e9eb4f40 1 -- 172.21.15.90:0/969582967 shutdown_connections 2024-03-24T01:26:28.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.871+0000 7f07e9eb4f40 1 -- 172.21.15.90:0/969582967 wait complete. 2024-03-24T01:26:28.877 DEBUG:teuthology.orchestra.run.smithi090:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 100000001f7.00000000 2024-03-24T01:26:28.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.977+0000 7f8c7fa49f40 1 Processor -- start 2024-03-24T01:26:28.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.977+0000 7f8c7fa49f40 1 -- start start 2024-03-24T01:26:28.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.978+0000 7f8c7fa49f40 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558dd1659090 0x558dd16f5c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:28.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.978+0000 7f8c7fa49f40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558dd1672370 0x558dd1701730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:28.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.978+0000 7f8c7fa49f40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558dd1702970 0x558dd1701c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:28.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.978+0000 7f8c7fa49f40 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dd1671cb0 con 0x558dd1659090 2024-03-24T01:26:28.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.978+0000 7f8c7fa49f40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dd15ccff0 con 0x558dd1672370 2024-03-24T01:26:28.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.978+0000 7f8c7fa49f40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x558dd1671f10 con 0x558dd1702970 2024-03-24T01:26:28.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.978+0000 7f8c7c400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558dd1659090 0x558dd16f5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:28.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.979+0000 7f8c7c400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558dd1659090 0x558dd16f5c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49972/0 (socket says 172.21.15.90:49972) 2024-03-24T01:26:28.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.979+0000 7f8c7c400640 1 -- 172.21.15.90:0/3055547809 learned_addr learned my addr 172.21.15.90:0/3055547809 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:28.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.979+0000 7f8c7c400640 1 -- 172.21.15.90:0/3055547809 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558dd1702970 msgr2=0x558dd1701c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:28.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.979+0000 7f8c7c400640 1 --2- 172.21.15.90:0/3055547809 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558dd1702970 0x558dd1701c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.979+0000 7f8c7c400640 1 -- 172.21.15.90:0/3055547809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558dd1672370 msgr2=0x558dd1701730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:28.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.979+0000 7f8c7c400640 1 --2- 172.21.15.90:0/3055547809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558dd1672370 0x558dd1701730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.979+0000 7f8c7c400640 1 -- 172.21.15.90:0/3055547809 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558dd1568dc0 con 0x558dd1659090 2024-03-24T01:26:28.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.979+0000 7f8c7c400640 1 --2- 172.21.15.90:0/3055547809 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558dd1659090 0x558dd16f5c50 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c047040 tx=0x7f8c6c04f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4853c5a2465a579d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:28.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.980+0000 7f8c7a600640 1 -- 172.21.15.90:0/3055547809 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c04d8a0 con 0x558dd1659090 2024-03-24T01:26:28.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.980+0000 7f8c7a600640 1 -- 172.21.15.90:0/3055547809 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c6c064070 con 0x558dd1659090 2024-03-24T01:26:28.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.980+0000 7f8c7a600640 1 -- 172.21.15.90:0/3055547809 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c04dc80 con 0x558dd1659090 2024-03-24T01:26:28.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.980+0000 7f8c7fa49f40 1 -- 172.21.15.90:0/3055547809 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558dd1659090 msgr2=0x558dd16f5c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:28.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.980+0000 7f8c7fa49f40 1 --2- 172.21.15.90:0/3055547809 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558dd1659090 0x558dd16f5c50 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c047040 tx=0x7f8c6c04f000 comp rx=0 tx=0).stop 2024-03-24T01:26:28.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.980+0000 7f8c7fa49f40 1 -- 172.21.15.90:0/3055547809 shutdown_connections 2024-03-24T01:26:28.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.980+0000 7f8c7fa49f40 1 --2- 172.21.15.90:0/3055547809 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558dd1702970 0x558dd1701c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.980+0000 7f8c7fa49f40 1 --2- 172.21.15.90:0/3055547809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558dd1672370 0x558dd1701730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.980+0000 7f8c7fa49f40 1 --2- 172.21.15.90:0/3055547809 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558dd1659090 0x558dd16f5c50 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.980+0000 7f8c7fa49f40 1 -- 172.21.15.90:0/3055547809 >> 172.21.15.90:0/3055547809 conn(0x558dd1673f60 msgr2=0x558dd16580e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:28.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.980+0000 7f8c7fa49f40 1 -- 172.21.15.90:0/3055547809 shutdown_connections 2024-03-24T01:26:28.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.980+0000 7f8c7fa49f40 1 -- 172.21.15.90:0/3055547809 wait complete. 2024-03-24T01:26:28.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.981+0000 7f8c7fa49f40 1 Processor -- start 2024-03-24T01:26:28.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.981+0000 7f8c7fa49f40 1 -- start start 2024-03-24T01:26:28.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.982+0000 7f8c7fa49f40 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558dd1659090 0x558dd16e2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:28.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.982+0000 7f8c7fa49f40 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558dd1672370 0x558dd17018c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:28.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.982+0000 7f8c7fa49f40 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558dd1702970 0x558dd1700cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:28.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.982+0000 7f8c7fa49f40 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dd16530a0 con 0x558dd1672370 2024-03-24T01:26:28.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.982+0000 7f8c7fa49f40 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x558dd1704960 con 0x558dd1659090 2024-03-24T01:26:28.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.982+0000 7f8c7fa49f40 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x558dd1653340 con 0x558dd1702970 2024-03-24T01:26:28.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.982+0000 7f8c7ba00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558dd1672370 0x558dd17018c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:28.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.982+0000 7f8c7ba00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558dd1672370 0x558dd17018c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49988/0 (socket says 172.21.15.90:49988) 2024-03-24T01:26:28.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.982+0000 7f8c7ba00640 1 -- 172.21.15.90:0/260483069 learned_addr learned my addr 172.21.15.90:0/260483069 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:28.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.983+0000 7f8c7c400640 1 --2- 172.21.15.90:0/260483069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558dd1659090 0x558dd16e2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:28.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.983+0000 7f8c7ce00640 1 --2- 172.21.15.90:0/260483069 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558dd1702970 0x558dd1700cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:28.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.983+0000 7f8c7ba00640 1 -- 172.21.15.90:0/260483069 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558dd1702970 msgr2=0x558dd1700cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:28.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.983+0000 7f8c7ba00640 1 --2- 172.21.15.90:0/260483069 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558dd1702970 0x558dd1700cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.983+0000 7f8c7ba00640 1 -- 172.21.15.90:0/260483069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558dd1659090 msgr2=0x558dd16e2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:28.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.983+0000 7f8c7ba00640 1 --2- 172.21.15.90:0/260483069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558dd1659090 0x558dd16e2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.983+0000 7f8c7ba00640 1 -- 172.21.15.90:0/260483069 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558dd16ffcd0 con 0x558dd1672370 2024-03-24T01:26:28.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.983+0000 7f8c7ba00640 1 --2- 172.21.15.90:0/260483069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558dd1672370 0x558dd17018c0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f8c6400af50 tx=0x7f8c640087f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:28.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.983+0000 7f8c79200640 1 -- 172.21.15.90:0/260483069 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6400b860 con 0x558dd1672370 2024-03-24T01:26:28.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.983+0000 7f8c79200640 1 -- 172.21.15.90:0/260483069 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c6400f070 con 0x558dd1672370 2024-03-24T01:26:28.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.983+0000 7f8c79200640 1 -- 172.21.15.90:0/260483069 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6400bc60 con 0x558dd1672370 2024-03-24T01:26:28.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.984+0000 7f8c7fa49f40 1 -- 172.21.15.90:0/260483069 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x558dd16fff70 con 0x558dd1672370 2024-03-24T01:26:28.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.984+0000 7f8c7fa49f40 1 -- 172.21.15.90:0/260483069 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558dd17004f0 con 0x558dd1672370 2024-03-24T01:26:28.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.986+0000 7f8c79200640 1 -- 172.21.15.90:0/260483069 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8c6401d3a0 con 0x558dd1672370 2024-03-24T01:26:28.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.986+0000 7f8c79200640 1 --2- 172.21.15.90:0/260483069 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8c5806a5b0 0x7f8c5806ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:28.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.987+0000 7f8c79200640 1 -- 172.21.15.90:0/260483069 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f8c64089920 con 0x558dd1672370 2024-03-24T01:26:28.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.987+0000 7f8c7fa49f40 1 --2- 172.21.15.90:0/260483069 >> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] conn(0x558dd1701230 0x558dd1653ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:28.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.987+0000 7f8c7fa49f40 1 -- 172.21.15.90:0/260483069 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:1 5.16 5:6fe38dc2:::100000001f7.00000000:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x558dd16faa30 con 0x558dd1701230 2024-03-24T01:26:28.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.987+0000 7f8c7c400640 1 --2- 172.21.15.90:0/260483069 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8c5806a5b0 0x7f8c5806ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:28.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.988+0000 7f8c7ce00640 1 --2- 172.21.15.90:0/260483069 >> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] conn(0x558dd1701230 0x558dd1653ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:28.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.988+0000 7f8c7c400640 1 --2- 172.21.15.90:0/260483069 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8c5806a5b0 0x7f8c5806ca70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c047040 tx=0x7f8c6c007030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:28.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.989+0000 7f8c7ce00640 1 --2- 172.21.15.90:0/260483069 >> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] conn(0x558dd1701230 0x558dd1653ab0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:28.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.989+0000 7f8c7ce00640 1 -- 172.21.15.90:0/260483069 <== osd.5 v2:172.21.15.120:6824/1739969887 1 ==== osd_op_reply(1 100000001f7.00000000 [omap-get-keys out=61b] v0'0 uv25 ondisk = 0) v8 ==== 164+0+61 (crc 0 0 0) 0x7f8c740074d0 con 0x558dd1701230 2024-03-24T01:26:28.990 INFO:teuthology.orchestra.run.smithi090.stdout:link0_head 2024-03-24T01:26:28.990 INFO:teuthology.orchestra.run.smithi090.stdout:link1_head 2024-03-24T01:26:28.990 INFO:teuthology.orchestra.run.smithi090.stdout:link2_head 2024-03-24T01:26:28.990 INFO:teuthology.orchestra.run.smithi090.stdout:link3_head 2024-03-24T01:26:28.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.990+0000 7f8c7fa49f40 1 -- 172.21.15.90:0/260483069 >> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] conn(0x558dd1701230 msgr2=0x558dd1653ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:28.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.990+0000 7f8c7fa49f40 1 --2- 172.21.15.90:0/260483069 >> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] conn(0x558dd1701230 0x558dd1653ab0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.990+0000 7f8c7fa49f40 1 -- 172.21.15.90:0/260483069 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8c5806a5b0 msgr2=0x7f8c5806ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:28.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.990+0000 7f8c7fa49f40 1 --2- 172.21.15.90:0/260483069 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8c5806a5b0 0x7f8c5806ca70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c047040 tx=0x7f8c6c007030 comp rx=0 tx=0).stop 2024-03-24T01:26:28.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.990+0000 7f8c7fa49f40 1 -- 172.21.15.90:0/260483069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558dd1672370 msgr2=0x558dd17018c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:28.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.990+0000 7f8c7fa49f40 1 --2- 172.21.15.90:0/260483069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558dd1672370 0x558dd17018c0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f8c6400af50 tx=0x7f8c640087f0 comp rx=0 tx=0).stop 2024-03-24T01:26:28.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.990+0000 7f8c7ce00640 1 -- 172.21.15.90:0/260483069 reap_dead start 2024-03-24T01:26:28.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.990+0000 7f8c7fa49f40 1 -- 172.21.15.90:0/260483069 shutdown_connections 2024-03-24T01:26:28.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.990+0000 7f8c7fa49f40 1 -- 172.21.15.90:0/260483069 >> 172.21.15.90:0/260483069 conn(0x558dd1673f60 msgr2=0x558dd1654e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:28.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.990+0000 7f8c7fa49f40 1 -- 172.21.15.90:0/260483069 shutdown_connections 2024-03-24T01:26:28.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:28.990+0000 7f8c7fa49f40 1 -- 172.21.15.90:0/260483069 wait complete. 2024-03-24T01:26:28.995 DEBUG:teuthology.orchestra.run.smithi090:> cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank cephfs:0 journal reset 2024-03-24T01:26:29.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.083+0000 7f1b37a42280 1 -- start start 2024-03-24T01:26:29.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.084+0000 7f1b37a42280 1 Processor -- start 2024-03-24T01:26:29.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.084+0000 7f1b37a42280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557c55fde400 0x557c552e5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:29.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.084+0000 7f1b37a42280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557c55fde800 0x557c552e4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:29.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.084+0000 7f1b37a42280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c55fdfc00 0x557c552e5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:29.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.084+0000 7f1b37a42280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c5605e000 con 0x557c55fde400 2024-03-24T01:26:29.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.084+0000 7f1b37a42280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c5605e180 con 0x557c55fdfc00 2024-03-24T01:26:29.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.084+0000 7f1b37a42280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x557c5605e300 con 0x557c55fde800 2024-03-24T01:26:29.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.085+0000 7f1b35200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557c55fde400 0x557c552e5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:29.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.085+0000 7f1b35200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557c55fde400 0x557c552e5600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49996/0 (socket says 172.21.15.90:49996) 2024-03-24T01:26:29.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.085+0000 7f1b35200640 1 -- 172.21.15.90:0/4051793528 learned_addr learned my addr 172.21.15.90:0/4051793528 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:29.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.085+0000 7f1b35c00640 1 --2- 172.21.15.90:0/4051793528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c55fdfc00 0x557c552e5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:29.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.085+0000 7f1b34800640 1 --2- 172.21.15.90:0/4051793528 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557c55fde800 0x557c552e4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:29.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.085+0000 7f1b35200640 1 -- 172.21.15.90:0/4051793528 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557c55fde800 msgr2=0x557c552e4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:29.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.085+0000 7f1b35200640 1 --2- 172.21.15.90:0/4051793528 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557c55fde800 0x557c552e4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:29.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.085+0000 7f1b35200640 1 -- 172.21.15.90:0/4051793528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c55fdfc00 msgr2=0x557c552e5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:29.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.085+0000 7f1b35200640 1 --2- 172.21.15.90:0/4051793528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c55fdfc00 0x557c552e5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:29.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.085+0000 7f1b35200640 1 -- 172.21.15.90:0/4051793528 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557c552ae780 con 0x557c55fde400 2024-03-24T01:26:29.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.085+0000 7f1b35c00640 1 --2- 172.21.15.90:0/4051793528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c55fdfc00 0x557c552e5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:29.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.085+0000 7f1b35200640 1 --2- 172.21.15.90:0/4051793528 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557c55fde400 0x557c552e5600 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x557c560a9350 tx=0x557c56012e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2ccfc5dd46875e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:29.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.086+0000 7f1b32000640 1 -- 172.21.15.90:0/4051793528 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x557c552a0d00 con 0x557c55fde400 2024-03-24T01:26:29.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.086+0000 7f1b32000640 1 -- 172.21.15.90:0/4051793528 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x557c552a6a80 con 0x557c55fde400 2024-03-24T01:26:29.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.086+0000 7f1b32000640 1 -- 172.21.15.90:0/4051793528 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x557c552a0680 con 0x557c55fde400 2024-03-24T01:26:29.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.086+0000 7f1b37a42280 1 -- 172.21.15.90:0/4051793528 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557c552ae000 con 0x557c55fde400 2024-03-24T01:26:29.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.086+0000 7f1b37a42280 1 -- 172.21.15.90:0/4051793528 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x557c552afa40 con 0x557c55fde400 2024-03-24T01:26:29.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.086+0000 7f1b37a42280 4 waiting for MDS map... 2024-03-24T01:26:29.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.086+0000 7f1b32000640 1 -- 172.21.15.90:0/4051793528 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x557c5608c400 con 0x557c55fde400 2024-03-24T01:26:29.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.086+0000 7f1b32000640 1 -- 172.21.15.90:0/4051793528 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 19) v1 ==== 3310+0+0 (secure 0 0 0) 0x557c552d5180 con 0x557c55fde400 2024-03-24T01:26:29.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.086+0000 7f1b37a42280 4 Got MDS map 19 2024-03-24T01:26:29.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.086+0000 7f1b37a42280 10 main: JournalTool::main 2024-03-24T01:26:29.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.087+0000 7f1b37a42280 4 main: JournalTool: connecting to RADOS... 2024-03-24T01:26:29.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.087+0000 7f1b37a42280 1 Processor -- start 2024-03-24T01:26:29.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.087+0000 7f1b37a42280 1 -- start start 2024-03-24T01:26:29.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.088+0000 7f1b37a42280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557c55fdf800 0x557c552e5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:29.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.088+0000 7f1b37a42280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c5608f000 0x557c552e4580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:29.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.088+0000 7f1b37a42280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557c5608f400 0x557c552e6100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:29.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.088+0000 7f1b37a42280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c5605e480 con 0x557c5608f400 2024-03-24T01:26:29.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.088+0000 7f1b37a42280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c5605e600 con 0x557c5608f000 2024-03-24T01:26:29.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.088+0000 7f1b37a42280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x557c5605e780 con 0x557c55fdf800 2024-03-24T01:26:29.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.088+0000 7f1b34800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557c5608f400 0x557c552e6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:29.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.088+0000 7f1b34800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557c5608f400 0x557c552e6100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50006/0 (socket says 172.21.15.90:50006) 2024-03-24T01:26:29.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.088+0000 7f1b34800640 1 -- 172.21.15.90:0/1842778381 learned_addr learned my addr 172.21.15.90:0/1842778381 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:29.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.089+0000 7f1b34800640 1 -- 172.21.15.90:0/1842778381 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557c55fdf800 msgr2=0x557c552e5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:29.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.089+0000 7f1b34800640 1 --2- 172.21.15.90:0/1842778381 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557c55fdf800 0x557c552e5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:29.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.089+0000 7f1b34800640 1 -- 172.21.15.90:0/1842778381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c5608f000 msgr2=0x557c552e4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:29.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.089+0000 7f1b34800640 1 --2- 172.21.15.90:0/1842778381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c5608f000 0x557c552e4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:29.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.089+0000 7f1b34800640 1 -- 172.21.15.90:0/1842778381 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557c552af860 con 0x557c5608f400 2024-03-24T01:26:29.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.089+0000 7f1b34800640 1 --2- 172.21.15.90:0/1842778381 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557c5608f400 0x557c552e6100 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x557c5615c750 tx=0x557c5615ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c106b0de3876a13 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:29.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.089+0000 7f1b2f800640 1 -- 172.21.15.90:0/1842778381 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x557c552a0d00 con 0x557c5608f400 2024-03-24T01:26:29.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.089+0000 7f1b2f800640 1 -- 172.21.15.90:0/1842778381 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x557c552a6e00 con 0x557c5608f400 2024-03-24T01:26:29.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.089+0000 7f1b2f800640 1 -- 172.21.15.90:0/1842778381 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x557c552a0680 con 0x557c5608f400 2024-03-24T01:26:29.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.090+0000 7f1b37a42280 1 -- 172.21.15.90:0/1842778381 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557c5608f400 msgr2=0x557c552e6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:29.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.090+0000 7f1b37a42280 1 --2- 172.21.15.90:0/1842778381 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557c5608f400 0x557c552e6100 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x557c5615c750 tx=0x557c5615ec00 comp rx=0 tx=0).stop 2024-03-24T01:26:29.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.090+0000 7f1b37a42280 1 -- 172.21.15.90:0/1842778381 shutdown_connections 2024-03-24T01:26:29.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.090+0000 7f1b37a42280 1 --2- 172.21.15.90:0/1842778381 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557c5608f400 0x557c552e6100 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:29.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.090+0000 7f1b37a42280 1 --2- 172.21.15.90:0/1842778381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c5608f000 0x557c552e4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:29.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.090+0000 7f1b37a42280 1 --2- 172.21.15.90:0/1842778381 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557c55fdf800 0x557c552e5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:29.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.090+0000 7f1b37a42280 1 -- 172.21.15.90:0/1842778381 >> 172.21.15.90:0/1842778381 conn(0x557c55fdf400 msgr2=0x557c55372800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:29.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.090+0000 7f1b37a42280 1 -- 172.21.15.90:0/1842778381 shutdown_connections 2024-03-24T01:26:29.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.090+0000 7f1b37a42280 1 -- 172.21.15.90:0/1842778381 wait complete. 2024-03-24T01:26:29.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.091+0000 7f1b37a42280 1 Processor -- start 2024-03-24T01:26:29.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.091+0000 7f1b37a42280 1 -- start start 2024-03-24T01:26:29.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.091+0000 7f1b37a42280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c55fdf800 0x557c552e7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:29.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.091+0000 7f1b37a42280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557c5608f000 0x557c552e6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:29.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.091+0000 7f1b37a42280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557c5608f400 0x557c552e6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:29.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.091+0000 7f1b37a42280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c5605ed80 con 0x557c5608f400 2024-03-24T01:26:29.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.091+0000 7f1b37a42280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c5605ef00 con 0x557c55fdf800 2024-03-24T01:26:29.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.091+0000 7f1b37a42280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x557c5605f080 con 0x557c5608f000 2024-03-24T01:26:29.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.092+0000 7f1b34800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557c5608f400 0x557c552e6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:29.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.092+0000 7f1b34800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557c5608f400 0x557c552e6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50016/0 (socket says 172.21.15.90:50016) 2024-03-24T01:26:29.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.092+0000 7f1b34800640 1 -- 172.21.15.90:0/1666264281 learned_addr learned my addr 172.21.15.90:0/1666264281 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:29.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.092+0000 7f1b34800640 1 -- 172.21.15.90:0/1666264281 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557c5608f000 msgr2=0x557c552e6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:29.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.092+0000 7f1b34800640 1 --2- 172.21.15.90:0/1666264281 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557c5608f000 0x557c552e6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:29.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.092+0000 7f1b34800640 1 -- 172.21.15.90:0/1666264281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c55fdf800 msgr2=0x557c552e7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:29.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.092+0000 7f1b34800640 1 --2- 172.21.15.90:0/1666264281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c55fdf800 0x557c552e7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:29.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.092+0000 7f1b34800640 1 -- 172.21.15.90:0/1666264281 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557c552ae780 con 0x557c5608f400 2024-03-24T01:26:29.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.092+0000 7f1b34800640 1 --2- 172.21.15.90:0/1666264281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557c5608f400 0x557c552e6680 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x557c5615d410 tx=0x557c56012cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:29.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.093+0000 7f1b2da00640 1 -- 172.21.15.90:0/1666264281 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x557c552a0d00 con 0x557c5608f400 2024-03-24T01:26:29.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.093+0000 7f1b2da00640 1 -- 172.21.15.90:0/1666264281 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x557c552a6c40 con 0x557c5608f400 2024-03-24T01:26:29.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.093+0000 7f1b37a42280 1 -- 172.21.15.90:0/1666264281 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x557c552ae000 con 0x557c5608f400 2024-03-24T01:26:29.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.093+0000 7f1b2da00640 1 -- 172.21.15.90:0/1666264281 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x557c552a0680 con 0x557c5608f400 2024-03-24T01:26:29.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.093+0000 7f1b37a42280 1 -- 172.21.15.90:0/1666264281 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557c552af680 con 0x557c5608f400 2024-03-24T01:26:29.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.093+0000 7f1b37a42280 4 main: JournalTool: resolving pool 5 2024-03-24T01:26:29.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.094+0000 7f1b2da00640 1 -- 172.21.15.90:0/1666264281 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x557c56172c00 con 0x557c5608f400 2024-03-24T01:26:29.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.095+0000 7f1b2da00640 1 --2- 172.21.15.90:0/1666264281 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x557c561b6c00 0x557c552e7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:29.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.095+0000 7f1b2da00640 1 -- 172.21.15.90:0/1666264281 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x557c561bca00 con 0x557c5608f400 2024-03-24T01:26:29.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.095+0000 7f1b37a42280 4 main: JournalTool: creating IoCtx.. 2024-03-24T01:26:29.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.095+0000 7f1b37a42280 4 main: Executing for rank 0 2024-03-24T01:26:29.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.096+0000 7f1b35c00640 1 --2- 172.21.15.90:0/1666264281 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x557c561b6c00 0x557c552e7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:29.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.097+0000 7f1b37a42280 1 -- start start 2024-03-24T01:26:29.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.097+0000 7f1b37a42280 1 Processor -- start 2024-03-24T01:26:29.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.097+0000 7f1b35c00640 1 --2- 172.21.15.90:0/1666264281 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x557c561b6c00 0x557c552e7700 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x557c55ffc870 tx=0x557c56012ea0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:29.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.098+0000 7f1b37a42280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557c5608fc00 0x557c552e6100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:29.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.098+0000 7f1b37a42280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c561b7000 0x557c552e4580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:29.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.098+0000 7f1b37a42280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557c561b7400 0x557c552e5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:29.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.098+0000 7f1b37a42280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c5605e480 con 0x557c5608fc00 2024-03-24T01:26:29.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.098+0000 7f1b37a42280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c5605e600 con 0x557c561b7000 2024-03-24T01:26:29.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.098+0000 7f1b37a42280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x557c5605e780 con 0x557c561b7400 2024-03-24T01:26:29.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.098+0000 7f1b35c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557c5608fc00 0x557c552e6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:29.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.098+0000 7f1b35c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557c5608fc00 0x557c552e6100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50018/0 (socket says 172.21.15.90:50018) 2024-03-24T01:26:29.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.098+0000 7f1b35c00640 1 -- 172.21.15.90:0/1230509315 learned_addr learned my addr 172.21.15.90:0/1230509315 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:29.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.098+0000 7f1b35200640 1 --2- 172.21.15.90:0/1230509315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c561b7000 0x557c552e4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:29.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.098+0000 7f1b34800640 1 --2- 172.21.15.90:0/1230509315 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557c561b7400 0x557c552e5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:29.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.098+0000 7f1b35c00640 1 -- 172.21.15.90:0/1230509315 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557c561b7400 msgr2=0x557c552e5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:29.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.098+0000 7f1b35c00640 1 --2- 172.21.15.90:0/1230509315 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557c561b7400 0x557c552e5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:29.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.098+0000 7f1b35c00640 1 -- 172.21.15.90:0/1230509315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c561b7000 msgr2=0x557c552e4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:29.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.098+0000 7f1b35c00640 1 --2- 172.21.15.90:0/1230509315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c561b7000 0x557c552e4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:29.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.098+0000 7f1b35c00640 1 -- 172.21.15.90:0/1230509315 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557c552ae3c0 con 0x557c5608fc00 2024-03-24T01:26:29.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.098+0000 7f1b35200640 1 --2- 172.21.15.90:0/1230509315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c561b7000 0x557c552e4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:29.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.099+0000 7f1b35c00640 1 --2- 172.21.15.90:0/1230509315 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557c5608fc00 0x557c552e6100 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x557c555a9620 tx=0x557c56012f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33b8698a515119e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:29.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.099+0000 7f1b29e00640 1 -- 172.21.15.90:0/1230509315 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x557c552a0b60 con 0x557c5608fc00 2024-03-24T01:26:29.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.099+0000 7f1b29e00640 1 -- 172.21.15.90:0/1230509315 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x557c552a61c0 con 0x557c5608fc00 2024-03-24T01:26:29.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.099+0000 7f1b29e00640 1 -- 172.21.15.90:0/1230509315 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x557c561f7ba0 con 0x557c5608fc00 2024-03-24T01:26:29.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.099+0000 7f1b37a42280 1 -- 172.21.15.90:0/1230509315 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557c552af4a0 con 0x557c5608fc00 2024-03-24T01:26:29.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.099+0000 7f1b37a42280 1 -- 172.21.15.90:0/1230509315 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x557c552ae960 con 0x557c5608fc00 2024-03-24T01:26:29.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.099+0000 7f1b37a42280 4 waiting for MDS map... 2024-03-24T01:26:29.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.099+0000 7f1b29e00640 1 -- 172.21.15.90:0/1230509315 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x557c55654400 con 0x557c5608fc00 2024-03-24T01:26:29.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.099+0000 7f1b29e00640 1 -- 172.21.15.90:0/1230509315 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 19) v1 ==== 3310+0+0 (secure 0 0 0) 0x557c552d5500 con 0x557c5608fc00 2024-03-24T01:26:29.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.099+0000 7f1b37a42280 4 Got MDS map 19 2024-03-24T01:26:29.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.100+0000 7f1b37a42280 1 --2- 172.21.15.90:0/1230509315 >> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] conn(0x557c561b7800 0x557c5566a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:29.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.100+0000 7f1b37a42280 1 -- 172.21.15.90:0/1230509315 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:1 5.f 5:f1f69726:::400.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x557c561b7c00 con 0x557c561b7800 2024-03-24T01:26:29.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.100+0000 7f1b35200640 1 --2- 172.21.15.90:0/1230509315 >> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] conn(0x557c561b7800 0x557c5566a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:29.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.101+0000 7f1b35200640 1 --2- 172.21.15.90:0/1230509315 >> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] conn(0x557c561b7800 0x557c5566a000 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=d361aee996c1754f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:29.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.102+0000 7f1b35200640 1 -- 172.21.15.90:0/1230509315 <== osd.5 v2:172.21.15.120:6824/1739969887 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x557c55ffa000 con 0x557c561b7800 2024-03-24T01:26:29.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.102+0000 7f1b37a42280 1 --2- 172.21.15.90:0/1230509315 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x557c55653c00 0x557c5566a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:29.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.102+0000 7f1b37a42280 1 -- 172.21.15.90:0/1230509315 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:2 5.14 5:292cf221:::200.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40) v9 -- 0x557c5567c000 con 0x557c55653c00 2024-03-24T01:26:29.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.102+0000 7f1b34800640 1 --2- 172.21.15.90:0/1230509315 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x557c55653c00 0x557c5566a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:29.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.103+0000 7f1b34800640 1 --2- 172.21.15.90:0/1230509315 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x557c55653c00 0x557c5566a580 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=c0ad264eccaa9451 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:29.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.103+0000 7f1b34800640 1 -- 172.21.15.90:0/1230509315 <== osd.3 v2:172.21.15.90:6808/2999931983 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv23 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x557c55ffa500 con 0x557c55653c00 2024-03-24T01:26:29.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.103+0000 7f1b28000640 1 -- 172.21.15.90:0/1230509315 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:3 5.14 5:2e2fa760:::200.00000001:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40) v9 -- 0x557c5567e000 con 0x557c55653c00 2024-03-24T01:26:29.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.103+0000 7f1b28000640 1 -- 172.21.15.90:0/1230509315 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:4 5.1b 5:ddb4e4d7:::200.00000002:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40) v9 -- 0x557c5567e400 con 0x557c561b7800 2024-03-24T01:26:29.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.103+0000 7f1b34800640 1 -- 172.21.15.90:0/1230509315 <== osd.3 v2:172.21.15.90:6808/2999931983 2 ==== osd_op_reply(3 200.00000001 [stat out=16b] v0'0 uv29 ondisk = 0) v8 ==== 156+0+16 (crc 0 0 0) 0x557c55ffa500 con 0x557c55653c00 2024-03-24T01:26:29.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.104+0000 7f1b35200640 1 -- 172.21.15.90:0/1230509315 <== osd.5 v2:172.21.15.120:6824/1739969887 2 ==== osd_op_reply(4 200.00000002 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x557c55ffa000 con 0x557c561b7800 2024-03-24T01:26:29.106 INFO:teuthology.orchestra.run.smithi090.stdout:old journal was 4194304~56200 2024-03-24T01:26:29.106 INFO:teuthology.orchestra.run.smithi090.stdout:new journal start will be 8388608 (4138104 bytes past old end) 2024-03-24T01:26:29.106 INFO:teuthology.orchestra.run.smithi090.stdout:writing journal head 2024-03-24T01:26:29.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.104+0000 7f1b37a42280 1 -- 172.21.15.90:0/1230509315 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:5 5.14 5:292cf221:::200.00000000:head [writefull 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x557c5567c400 con 0x557c55653c00 2024-03-24T01:26:29.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.107+0000 7f1b34800640 1 -- 172.21.15.90:0/1230509315 <== osd.3 v2:172.21.15.90:6808/2999931983 3 ==== osd_op_reply(5 200.00000000 [writefull 0~90] v40'39 uv39 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x557c55ffa500 con 0x557c55653c00 2024-03-24T01:26:29.107 INFO:teuthology.orchestra.run.smithi090.stdout:writing EResetJournal entry 2024-03-24T01:26:29.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.107+0000 7f1b37a42280 1 -- 172.21.15.90:0/1230509315 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:6 5.1b 5:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x557c5567c800 con 0x557c561b7800 2024-03-24T01:26:29.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.107+0000 7f1b37a42280 1 --2- 172.21.15.90:0/1230509315 >> [v2:172.21.15.90:6817/865053620,v1:172.21.15.90:6820/865053620] conn(0x557c5567cc00 0x557c5566ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:29.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.107+0000 7f1b37a42280 1 -- 172.21.15.90:0/1230509315 --> [v2:172.21.15.90:6817/865053620,v1:172.21.15.90:6820/865053620] -- osd_op(unknown.0.0:7 5.b 5:d5c7a900:::200.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x557c5567d000 con 0x557c5567cc00 2024-03-24T01:26:29.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.107+0000 7f1b37a42280 1 --2- 172.21.15.90:0/1230509315 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x557c5567d400 0x557c5566b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:29.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.107+0000 7f1b37a42280 1 -- 172.21.15.90:0/1230509315 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:8 5.1f 5:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x557c5567d800 con 0x557c5567d400 2024-03-24T01:26:29.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.107+0000 7f1b37a42280 1 --2- 172.21.15.90:0/1230509315 >> [v2:172.21.15.120:6808/1045862942,v1:172.21.15.120:6809/1045862942] conn(0x557c5567dc00 0x557c5566b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:29.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.107+0000 7f1b37a42280 1 -- 172.21.15.90:0/1230509315 --> [v2:172.21.15.120:6808/1045862942,v1:172.21.15.120:6809/1045862942] -- osd_op(unknown.0.0:9 5.1a 5:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x557c5567ec00 con 0x557c5567dc00 2024-03-24T01:26:29.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.107+0000 7f1b37a42280 1 --2- 172.21.15.90:0/1230509315 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x557c5567f000 0x557c5566bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:29.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.107+0000 7f1b37a42280 1 -- 172.21.15.90:0/1230509315 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:10 5.1e 5:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x557c5567f400 con 0x557c5567f000 2024-03-24T01:26:29.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.107+0000 7f1b37a42280 1 --2- 172.21.15.90:0/1230509315 >> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] conn(0x557c5567f800 0x557c5566c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:29.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.107+0000 7f1b37a42280 1 -- 172.21.15.90:0/1230509315 --> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] -- osd_op(unknown.0.0:11 5.1d 5:bec1251c:::200.00000007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x557c5567fc00 con 0x557c5567f800 2024-03-24T01:26:29.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.108+0000 7f1b35200640 1 --2- 172.21.15.90:0/1230509315 >> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] conn(0x557c5567f800 0x557c5566c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:29.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.108+0000 7f1b35c00640 1 --2- 172.21.15.90:0/1230509315 >> [v2:172.21.15.90:6817/865053620,v1:172.21.15.90:6820/865053620] conn(0x557c5567cc00 0x557c5566ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:29.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.108+0000 7f1b35c00640 1 --2- 172.21.15.90:0/1230509315 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x557c5567f000 0x557c5566bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:29.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.108+0000 7f1b34800640 1 --2- 172.21.15.90:0/1230509315 >> [v2:172.21.15.120:6808/1045862942,v1:172.21.15.120:6809/1045862942] conn(0x557c5567dc00 0x557c5566b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:29.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.108+0000 7f1b35200640 1 --2- 172.21.15.90:0/1230509315 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x557c5567d400 0x557c5566b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:29.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.108+0000 7f1b35200640 1 --2- 172.21.15.90:0/1230509315 >> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] conn(0x557c5567f800 0x557c5566c100 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=ab8b9c73076ccf65 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:29.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.109+0000 7f1b35c00640 1 --2- 172.21.15.90:0/1230509315 >> [v2:172.21.15.90:6817/865053620,v1:172.21.15.90:6820/865053620] conn(0x557c5567cc00 0x557c5566ab00 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=e80837dfc6fb453f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:29.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.109+0000 7f1b35c00640 1 --2- 172.21.15.90:0/1230509315 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x557c5567f000 0x557c5566bb80 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=369492c7ce0f8119 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:29.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.110+0000 7f1b34800640 1 --2- 172.21.15.90:0/1230509315 >> [v2:172.21.15.120:6808/1045862942,v1:172.21.15.120:6809/1045862942] conn(0x557c5567dc00 0x557c5566b600 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=48f579c5b4469506 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:29.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.110+0000 7f1b35200640 1 --2- 172.21.15.90:0/1230509315 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x557c5567d400 0x557c5566b080 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=f1277371e57e7e74 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:29.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.111+0000 7f1b35c00640 1 -- 172.21.15.90:0/1230509315 <== osd.2 v2:172.21.15.90:6817/865053620 1 ==== osd_op_reply(7 200.00000003 [delete] v40'4 uv1 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x557c55ffa780 con 0x557c5567cc00 2024-03-24T01:26:29.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.111+0000 7f1b35200640 1 -- 172.21.15.90:0/1230509315 <== osd.5 v2:172.21.15.120:6824/1739969887 3 ==== osd_op_reply(6 200.00000002 [delete] v40'2 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x557c55ffa000 con 0x557c561b7800 2024-03-24T01:26:29.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.111+0000 7f1b28000640 1 -- 172.21.15.90:0/1230509315 --> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] -- osd_op(unknown.0.0:12 5.1b 5:ddb4e4d7:::200.00000002:head [write 0~48 in=48b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40) v9 -- 0x557c5567e800 con 0x557c561b7800 2024-03-24T01:26:29.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.112+0000 7f1b35c00640 1 -- 172.21.15.90:0/1230509315 <== osd.0 v2:172.21.15.90:6800/307045587 1 ==== osd_op_reply(10 200.00000006 [delete] v40'2 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x557c55ffa780 con 0x557c5567f000 2024-03-24T01:26:29.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.113+0000 7f1b35200640 1 -- 172.21.15.90:0/1230509315 <== osd.1 v2:172.21.15.90:6816/133441085 1 ==== osd_op_reply(11 200.00000007 [delete] v40'4 uv2 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x557c55ffa000 con 0x557c5567f800 2024-03-24T01:26:29.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.115+0000 7f1b34800640 1 -- 172.21.15.90:0/1230509315 <== osd.7 v2:172.21.15.120:6808/1045862942 1 ==== osd_op_reply(9 200.00000005 [delete] v40'2 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x557c55ffa500 con 0x557c5567dc00 2024-03-24T01:26:29.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.115+0000 7f1b35200640 1 -- 172.21.15.90:0/1230509315 <== osd.5 v2:172.21.15.120:6824/1739969887 4 ==== osd_op_reply(12 200.00000002 [write 0~48] v40'3 uv3 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x557c55ffa000 con 0x557c561b7800 2024-03-24T01:26:29.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.115+0000 7f1b35200640 1 -- 172.21.15.90:0/1230509315 <== osd.6 v2:172.21.15.120:6816/1454745630 1 ==== osd_op_reply(8 200.00000004 [delete] v40'51 uv50 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x557c55ffa000 con 0x557c5567d400 2024-03-24T01:26:29.115 INFO:teuthology.orchestra.run.smithi090.stdout:done 2024-03-24T01:26:29.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.115+0000 7f1b37a42280 1 -- 172.21.15.90:0/1230509315 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x557c5567f000 msgr2=0x557c5566bb80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:29.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.115+0000 7f1b37a42280 1 --2- 172.21.15.90:0/1230509315 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x557c5567f000 0x557c5566bb80 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:29.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.115+0000 7f1b37a42280 1 -- 172.21.15.90:0/1230509315 >> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] conn(0x557c5567f800 msgr2=0x557c5566c100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:29.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.115+0000 7f1b37a42280 1 --2- 172.21.15.90:0/1230509315 >> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] conn(0x557c5567f800 0x557c5566c100 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:29.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.115+0000 7f1b37a42280 1 -- 172.21.15.90:0/1230509315 >> [v2:172.21.15.90:6817/865053620,v1:172.21.15.90:6820/865053620] conn(0x557c5567cc00 msgr2=0x557c5566ab00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:29.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.115+0000 7f1b37a42280 1 --2- 172.21.15.90:0/1230509315 >> [v2:172.21.15.90:6817/865053620,v1:172.21.15.90:6820/865053620] conn(0x557c5567cc00 0x557c5566ab00 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:29.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.115+0000 7f1b37a42280 1 -- 172.21.15.90:0/1230509315 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x557c55653c00 msgr2=0x557c5566a580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:29.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.115+0000 7f1b37a42280 1 --2- 172.21.15.90:0/1230509315 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x557c55653c00 0x557c5566a580 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:29.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.115+0000 7f1b37a42280 1 -- 172.21.15.90:0/1230509315 >> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] conn(0x557c561b7800 msgr2=0x557c5566a000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:29.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.115+0000 7f1b37a42280 1 --2- 172.21.15.90:0/1230509315 >> [v2:172.21.15.120:6824/1739969887,v1:172.21.15.120:6825/1739969887] conn(0x557c561b7800 0x557c5566a000 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:29.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.115+0000 7f1b37a42280 1 -- 172.21.15.90:0/1230509315 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x557c5567d400 msgr2=0x557c5566b080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:29.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.115+0000 7f1b37a42280 1 --2- 172.21.15.90:0/1230509315 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x557c5567d400 0x557c5566b080 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:29.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.115+0000 7f1b37a42280 1 -- 172.21.15.90:0/1230509315 >> [v2:172.21.15.120:6808/1045862942,v1:172.21.15.120:6809/1045862942] conn(0x557c5567dc00 msgr2=0x557c5566b600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:29.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.115+0000 7f1b37a42280 1 --2- 172.21.15.90:0/1230509315 >> [v2:172.21.15.120:6808/1045862942,v1:172.21.15.120:6809/1045862942] conn(0x557c5567dc00 0x557c5566b600 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:29.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.115+0000 7f1b35200640 1 -- 172.21.15.90:0/1230509315 reap_dead start 2024-03-24T01:26:29.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.115+0000 7f1b37a42280 1 -- 172.21.15.90:0/1230509315 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557c5608fc00 msgr2=0x557c552e6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:29.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.115+0000 7f1b37a42280 1 --2- 172.21.15.90:0/1230509315 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557c5608fc00 0x557c552e6100 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x557c555a9620 tx=0x557c56012f60 comp rx=0 tx=0).stop 2024-03-24T01:26:29.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.115+0000 7f1b35200640 1 -- 172.21.15.90:0/1230509315 reap_dead start 2024-03-24T01:26:29.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.115+0000 7f1b35200640 1 -- 172.21.15.90:0/1230509315 reap_dead start 2024-03-24T01:26:29.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.115+0000 7f1b35200640 1 -- 172.21.15.90:0/1230509315 reap_dead start 2024-03-24T01:26:29.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.115+0000 7f1b35200640 1 -- 172.21.15.90:0/1230509315 reap_dead start 2024-03-24T01:26:29.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.115+0000 7f1b37a42280 1 -- 172.21.15.90:0/1230509315 shutdown_connections 2024-03-24T01:26:29.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.115+0000 7f1b37a42280 1 -- 172.21.15.90:0/1230509315 >> 172.21.15.90:0/1230509315 conn(0x557c5608f800 msgr2=0x557c55371000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:29.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.115+0000 7f1b37a42280 1 -- 172.21.15.90:0/1230509315 shutdown_connections 2024-03-24T01:26:29.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.116+0000 7f1b37a42280 1 -- 172.21.15.90:0/1230509315 wait complete. 2024-03-24T01:26:29.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.116+0000 7f1b37a42280 1 -- 172.21.15.90:0/1666264281 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x557c561b6c00 msgr2=0x557c552e7700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:29.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.116+0000 7f1b37a42280 1 --2- 172.21.15.90:0/1666264281 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x557c561b6c00 0x557c552e7700 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x557c55ffc870 tx=0x557c56012ea0 comp rx=0 tx=0).stop 2024-03-24T01:26:29.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.116+0000 7f1b37a42280 1 -- 172.21.15.90:0/1666264281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557c5608f400 msgr2=0x557c552e6680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:29.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.116+0000 7f1b37a42280 1 --2- 172.21.15.90:0/1666264281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557c5608f400 0x557c552e6680 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x557c5615d410 tx=0x557c56012cc0 comp rx=0 tx=0).stop 2024-03-24T01:26:29.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.116+0000 7f1b37a42280 1 -- 172.21.15.90:0/1666264281 shutdown_connections 2024-03-24T01:26:29.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.116+0000 7f1b37a42280 1 --2- 172.21.15.90:0/1666264281 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x557c561b6c00 0x557c552e7700 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:29.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.116+0000 7f1b37a42280 1 --2- 172.21.15.90:0/1666264281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557c5608f400 0x557c552e6680 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:29.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.116+0000 7f1b37a42280 1 --2- 172.21.15.90:0/1666264281 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557c5608f000 0x557c552e6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:29.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.116+0000 7f1b37a42280 1 --2- 172.21.15.90:0/1666264281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c55fdf800 0x557c552e7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:29.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.116+0000 7f1b37a42280 1 -- 172.21.15.90:0/1666264281 >> 172.21.15.90:0/1666264281 conn(0x557c55fdf400 msgr2=0x557c55372800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:29.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.116+0000 7f1b37a42280 1 -- 172.21.15.90:0/1666264281 shutdown_connections 2024-03-24T01:26:29.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.116+0000 7f1b37a42280 1 -- 172.21.15.90:0/1666264281 wait complete. 2024-03-24T01:26:29.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.117+0000 7f1b37a42280 1 -- 172.21.15.90:0/4051793528 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557c55fde400 msgr2=0x557c552e5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:29.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.117+0000 7f1b37a42280 1 --2- 172.21.15.90:0/4051793528 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557c55fde400 0x557c552e5600 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x557c560a9350 tx=0x557c56012e40 comp rx=0 tx=0).stop 2024-03-24T01:26:29.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.117+0000 7f1b37a42280 1 -- 172.21.15.90:0/4051793528 shutdown_connections 2024-03-24T01:26:29.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.117+0000 7f1b37a42280 1 --2- 172.21.15.90:0/4051793528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c55fdfc00 0x557c552e5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:29.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.117+0000 7f1b37a42280 1 --2- 172.21.15.90:0/4051793528 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557c55fde800 0x557c552e4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:29.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.117+0000 7f1b37a42280 1 --2- 172.21.15.90:0/4051793528 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557c55fde400 0x557c552e5600 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:29.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.117+0000 7f1b37a42280 1 -- 172.21.15.90:0/4051793528 >> 172.21.15.90:0/4051793528 conn(0x557c5527b000 msgr2=0x557c55373000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:29.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.117+0000 7f1b37a42280 1 -- 172.21.15.90:0/4051793528 shutdown_connections 2024-03-24T01:26:29.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.117+0000 7f1b37a42280 1 -- 172.21.15.90:0/4051793528 wait complete. 2024-03-24T01:26:29.123 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-journal-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '--rank', 'cephfs:0', 'journal', 'reset'] in time 0:00:00.128356, result: old journal was 4194304~56200 new journal start will be 8388608 (4138104 bytes past old end) writing journal head writing EResetJournal entry done 2024-03-24T01:26:29.123 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-03-24T01:26:29.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.323+0000 7f9561600640 1 Processor -- start 2024-03-24T01:26:29.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.324+0000 7f9561600640 1 -- start start 2024-03-24T01:26:29.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.325+0000 7f9561600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f955c161660 0x7f955c0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:29.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.325+0000 7f9561600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f955c099c60 0x7f955c09a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:29.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.325+0000 7f9561600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f955c0d13c0 con 0x7f955c0d0900 2024-03-24T01:26:29.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.325+0000 7f9561600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f955c06c8c0 con 0x7f955c099c60 2024-03-24T01:26:29.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.325+0000 7f9561600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f955c0d10e0 con 0x7f955c161660 2024-03-24T01:26:29.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.326+0000 7f955a000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f955c0d0900 0x7f955c0cc0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53304/0 (socket says 172.21.15.90:53304) 2024-03-24T01:26:29.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.326+0000 7f955a000640 1 -- 172.21.15.90:0/1601475082 learned_addr learned my addr 172.21.15.90:0/1601475082 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:26:29.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.326+0000 7f955aa00640 1 --2- 172.21.15.90:0/1601475082 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f955c161660 0x7f955c0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:29.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.326+0000 7f955b400640 1 --2- 172.21.15.90:0/1601475082 >> v2:172.21.15.120:3300/0 conn(0x7f955c099c60 0x7f955c09a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:29.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.326+0000 7f9559600640 1 -- 172.21.15.90:0/1601475082 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3764114865 0 0) 0x7f955c0d13c0 con 0x7f955c0d0900 2024-03-24T01:26:29.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.326+0000 7f9559600640 1 -- 172.21.15.90:0/1601475082 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9540003520 con 0x7f955c0d0900 2024-03-24T01:26:29.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.326+0000 7f9559600640 1 -- 172.21.15.90:0/1601475082 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1224652745 0 0) 0x7f9540003520 con 0x7f955c0d0900 2024-03-24T01:26:29.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.326+0000 7f9559600640 1 -- 172.21.15.90:0/1601475082 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f955c161660 msgr2=0x7f955c0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:29.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.326+0000 7f9559600640 1 --2- 172.21.15.90:0/1601475082 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f955c161660 0x7f955c0b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:29.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.326+0000 7f9559600640 1 -- 172.21.15.90:0/1601475082 >> v2:172.21.15.120:3300/0 conn(0x7f955c099c60 msgr2=0x7f955c09a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:29.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.326+0000 7f9559600640 1 --2- 172.21.15.90:0/1601475082 >> v2:172.21.15.120:3300/0 conn(0x7f955c099c60 0x7f955c09a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:29.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.326+0000 7f9559600640 1 -- 172.21.15.90:0/1601475082 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f955c06c270 con 0x7f955c0d0900 2024-03-24T01:26:29.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.327+0000 7f9559600640 1 -- 172.21.15.90:0/1601475082 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f9544001670 con 0x7f955c0d0900 2024-03-24T01:26:29.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.327+0000 7f9559600640 1 -- 172.21.15.90:0/1601475082 >> v1:172.21.15.90:6789/0 conn(0x7f955c0d0900 legacy=0x7f955c0cc0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:29.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.327+0000 7f9559600640 1 --2- 172.21.15.90:0/1601475082 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95400039b0 0x7f9540003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:29.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.327+0000 7f9559600640 1 --2- 172.21.15.90:0/1601475082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9540004800 0x7f9540004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:29.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.327+0000 7f9559600640 1 --2- 172.21.15.90:0/1601475082 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f955c161660 0x7f95400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:29.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.327+0000 7f9559600640 1 -- 172.21.15.90:0/1601475082 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f955c06c8c0 con 0x7f95400039b0 2024-03-24T01:26:29.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.327+0000 7f9559600640 1 -- 172.21.15.90:0/1601475082 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f955c0d10e0 con 0x7f9540004800 2024-03-24T01:26:29.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.327+0000 7f9559600640 1 -- 172.21.15.90:0/1601475082 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f954000b9c0 con 0x7f955c161660 2024-03-24T01:26:29.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.327+0000 7f955aa00640 1 --2- 172.21.15.90:0/1601475082 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95400039b0 0x7f9540003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:29.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.327+0000 7f955a000640 1 --2- 172.21.15.90:0/1601475082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9540004800 0x7f9540004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:29.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.327+0000 7f955aa00640 1 -- 172.21.15.90:0/1601475082 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f955c161660 msgr2=0x7f95400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:29.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.327+0000 7f955aa00640 1 --2- 172.21.15.90:0/1601475082 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f955c161660 0x7f95400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:29.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.327+0000 7f955aa00640 1 -- 172.21.15.90:0/1601475082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9540004800 msgr2=0x7f9540004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:29.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.327+0000 7f955aa00640 1 --2- 172.21.15.90:0/1601475082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9540004800 0x7f9540004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:29.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.327+0000 7f955aa00640 1 -- 172.21.15.90:0/1601475082 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9540003520 con 0x7f95400039b0 2024-03-24T01:26:29.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.327+0000 7f955a000640 1 --2- 172.21.15.90:0/1601475082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9540004800 0x7f9540004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:29.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.328+0000 7f955aa00640 1 --2- 172.21.15.90:0/1601475082 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95400039b0 0x7f9540003dd0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f954c009af0 tx=0x7f9540005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=303c29303a782ca6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:29.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.329+0000 7f9559600640 1 -- 172.21.15.90:0/1601475082 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954c039070 con 0x7f95400039b0 2024-03-24T01:26:29.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.329+0000 7f9559600640 1 -- 172.21.15.90:0/1601475082 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f954c02c710 con 0x7f95400039b0 2024-03-24T01:26:29.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.329+0000 7f9559600640 1 -- 172.21.15.90:0/1601475082 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954c009690 con 0x7f95400039b0 2024-03-24T01:26:29.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.329+0000 7f9561600640 1 -- 172.21.15.90:0/1601475082 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95400039b0 msgr2=0x7f9540003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:29.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.329+0000 7f9561600640 1 --2- 172.21.15.90:0/1601475082 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95400039b0 0x7f9540003dd0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f954c009af0 tx=0x7f9540005830 comp rx=0 tx=0).stop 2024-03-24T01:26:29.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.329+0000 7f955b400640 1 -- 172.21.15.90:0/1601475082 reap_dead start 2024-03-24T01:26:29.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.329+0000 7f9561600640 1 -- 172.21.15.90:0/1601475082 shutdown_connections 2024-03-24T01:26:29.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.329+0000 7f9561600640 1 -- 172.21.15.90:0/1601475082 >> 172.21.15.90:0/1601475082 conn(0x7f955c0b3570 msgr2=0x7f955c0b3980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:29.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.329+0000 7f9561600640 1 -- 172.21.15.90:0/1601475082 shutdown_connections 2024-03-24T01:26:29.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.330+0000 7f9561600640 1 -- 172.21.15.90:0/1601475082 wait complete. 2024-03-24T01:26:29.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.331+0000 7f9561600640 1 Processor -- start 2024-03-24T01:26:29.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.331+0000 7f9561600640 1 -- start start 2024-03-24T01:26:29.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.332+0000 7f9561600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95400039b0 0x7f955c14b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:29.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.332+0000 7f9561600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f955c0c7a10 0x7f955c160690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:29.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.332+0000 7f9561600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f955c160bd0 0x7f955c15d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:29.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.332+0000 7f9561600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f955c0b2950 con 0x7f95400039b0 2024-03-24T01:26:29.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.332+0000 7f9561600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f955c0d0d80 con 0x7f955c160bd0 2024-03-24T01:26:29.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.332+0000 7f9561600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f955c0b2ad0 con 0x7f955c0c7a10 2024-03-24T01:26:29.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.332+0000 7f955aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95400039b0 0x7f955c14b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:29.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.333+0000 7f955aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95400039b0 0x7f955c14b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50042/0 (socket says 172.21.15.90:50042) 2024-03-24T01:26:29.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.333+0000 7f955aa00640 1 -- 172.21.15.90:0/22051904 learned_addr learned my addr 172.21.15.90:0/22051904 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:29.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.333+0000 7f955aa00640 1 -- 172.21.15.90:0/22051904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f955c0c7a10 msgr2=0x7f955c160690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:29.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.333+0000 7f955aa00640 1 --2- 172.21.15.90:0/22051904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f955c0c7a10 0x7f955c160690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:29.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.333+0000 7f955aa00640 1 -- 172.21.15.90:0/22051904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f955c160bd0 msgr2=0x7f955c15d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:29.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.333+0000 7f955aa00640 1 --2- 172.21.15.90:0/22051904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f955c160bd0 0x7f955c15d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:29.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.333+0000 7f955aa00640 1 -- 172.21.15.90:0/22051904 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f955c0c6970 con 0x7f95400039b0 2024-03-24T01:26:29.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.333+0000 7f955aa00640 1 --2- 172.21.15.90:0/22051904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95400039b0 0x7f955c14b620 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f954c037040 tx=0x7f954c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:29.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.334+0000 7f9552a00640 1 -- 172.21.15.90:0/22051904 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954c042020 con 0x7f95400039b0 2024-03-24T01:26:29.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.334+0000 7f9552a00640 1 -- 172.21.15.90:0/22051904 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f954c03a040 con 0x7f95400039b0 2024-03-24T01:26:29.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.334+0000 7f9552a00640 1 -- 172.21.15.90:0/22051904 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954c039070 con 0x7f95400039b0 2024-03-24T01:26:29.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.334+0000 7f9561600640 1 -- 172.21.15.90:0/22051904 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f955c0c6c50 con 0x7f95400039b0 2024-03-24T01:26:29.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.334+0000 7f9561600640 1 -- 172.21.15.90:0/22051904 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f955c0c7210 con 0x7f95400039b0 2024-03-24T01:26:29.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.335+0000 7f9561600640 1 -- 172.21.15.90:0/22051904 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f955c06e4b0 con 0x7f95400039b0 2024-03-24T01:26:29.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.336+0000 7f9552a00640 1 -- 172.21.15.90:0/22051904 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f954c045020 con 0x7f95400039b0 2024-03-24T01:26:29.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.337+0000 7f9552a00640 1 --2- 172.21.15.90:0/22051904 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f953806bbd0 0x7f953806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:29.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.337+0000 7f9552a00640 1 -- 172.21.15.90:0/22051904 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f954c0abd00 con 0x7f95400039b0 2024-03-24T01:26:29.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.338+0000 7f955a000640 1 --2- 172.21.15.90:0/22051904 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f953806bbd0 0x7f953806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:29.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.339+0000 7f955a000640 1 --2- 172.21.15.90:0/22051904 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f953806bbd0 0x7f953806e090 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9544000c00 tx=0x7f9544009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:29.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.343+0000 7f9552a00640 1 -- 172.21.15.90:0/22051904 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f954c04c020 con 0x7f95400039b0 2024-03-24T01:26:29.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:29.585+0000 7f9561600640 1 -- 172.21.15.90:0/22051904 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) v1 -- 0x7f955c09a790 con 0x7f95400039b0 2024-03-24T01:26:30.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.139+0000 7f9552a00640 1 -- 172.21.15.90:0/22051904 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v20) v1 ==== 162+0+0 (secure 0 0 0) 0x7f954c07a600 con 0x7f95400039b0 2024-03-24T01:26:30.140 INFO:teuthology.orchestra.run.smithi090.stderr:cephfs marked joinable; MDS may join as newly active. 2024-03-24T01:26:30.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.142+0000 7f9561600640 1 -- 172.21.15.90:0/22051904 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f953806bbd0 msgr2=0x7f953806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:30.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.142+0000 7f9561600640 1 --2- 172.21.15.90:0/22051904 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f953806bbd0 0x7f953806e090 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9544000c00 tx=0x7f9544009040 comp rx=0 tx=0).stop 2024-03-24T01:26:30.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.143+0000 7f9561600640 1 -- 172.21.15.90:0/22051904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95400039b0 msgr2=0x7f955c14b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:30.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.143+0000 7f9561600640 1 --2- 172.21.15.90:0/22051904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95400039b0 0x7f955c14b620 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f954c037040 tx=0x7f954c004730 comp rx=0 tx=0).stop 2024-03-24T01:26:30.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.143+0000 7f9561600640 1 -- 172.21.15.90:0/22051904 shutdown_connections 2024-03-24T01:26:30.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.143+0000 7f9561600640 1 --2- 172.21.15.90:0/22051904 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f953806bbd0 0x7f953806e090 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:30.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.143+0000 7f9561600640 1 --2- 172.21.15.90:0/22051904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f955c160bd0 0x7f955c15d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:30.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.143+0000 7f9561600640 1 --2- 172.21.15.90:0/22051904 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f955c0c7a10 0x7f955c160690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:30.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.143+0000 7f9561600640 1 --2- 172.21.15.90:0/22051904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95400039b0 0x7f955c14b620 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:30.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.143+0000 7f9561600640 1 -- 172.21.15.90:0/22051904 >> 172.21.15.90:0/22051904 conn(0x7f955c0b3570 msgr2=0x7f955c0ce270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:30.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.143+0000 7f9561600640 1 -- 172.21.15.90:0/22051904 shutdown_connections 2024-03-24T01:26:30.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.143+0000 7f9561600640 1 -- 172.21.15.90:0/22051904 wait complete. 2024-03-24T01:26:30.166 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:26:30.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.354+0000 7f9d82400640 1 Processor -- start 2024-03-24T01:26:30.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.355+0000 7f9d82400640 1 -- start start 2024-03-24T01:26:30.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.356+0000 7f9d82400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d7c06c860 0x7f9d7c0cf210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:30.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.356+0000 7f9d82400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f9d7c0d1620 0x7f9d7c0d2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:30.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.356+0000 7f9d82400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d7c0d1b20 con 0x7f9d7c06cc40 2024-03-24T01:26:30.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.356+0000 7f9d82400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d7c0d2730 con 0x7f9d7c0d1620 2024-03-24T01:26:30.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.356+0000 7f9d82400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c0d1e00 con 0x7f9d7c06c860 2024-03-24T01:26:30.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.356+0000 7f9d81a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f9d7c06cc40 0x7f9d7c0b6ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:34568/0 (socket says 172.21.15.90:34568) 2024-03-24T01:26:30.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.356+0000 7f9d81a00640 1 -- 172.21.15.90:0/1971177762 learned_addr learned my addr 172.21.15.90:0/1971177762 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:26:30.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.357+0000 7f9d81000640 1 --2- 172.21.15.90:0/1971177762 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d7c06c860 0x7f9d7c0cf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:30.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.357+0000 7f9d7be00640 1 --2- 172.21.15.90:0/1971177762 >> v2:172.21.15.120:3300/0 conn(0x7f9d7c0d1620 0x7f9d7c0d2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:30.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.357+0000 7f9d7b400640 1 -- 172.21.15.90:0/1971177762 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1625587600 0 0) 0x7f9d7c0d1b20 con 0x7f9d7c06cc40 2024-03-24T01:26:30.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.357+0000 7f9d7b400640 1 -- 172.21.15.90:0/1971177762 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d60003520 con 0x7f9d7c06cc40 2024-03-24T01:26:30.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.357+0000 7f9d7b400640 1 -- 172.21.15.90:0/1971177762 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2667152602 0 0) 0x7f9d60003520 con 0x7f9d7c06cc40 2024-03-24T01:26:30.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.357+0000 7f9d7b400640 1 -- 172.21.15.90:0/1971177762 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d7c06c860 msgr2=0x7f9d7c0cf210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:30.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.357+0000 7f9d7b400640 1 --2- 172.21.15.90:0/1971177762 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d7c06c860 0x7f9d7c0cf210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:30.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.357+0000 7f9d7b400640 1 -- 172.21.15.90:0/1971177762 >> v2:172.21.15.120:3300/0 conn(0x7f9d7c0d1620 msgr2=0x7f9d7c0d2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:30.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.357+0000 7f9d7b400640 1 --2- 172.21.15.90:0/1971177762 >> v2:172.21.15.120:3300/0 conn(0x7f9d7c0d1620 0x7f9d7c0d2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:30.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.357+0000 7f9d7b400640 1 -- 172.21.15.90:0/1971177762 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d7c0d20e0 con 0x7f9d7c06cc40 2024-03-24T01:26:30.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.357+0000 7f9d7b400640 1 -- 172.21.15.90:0/1971177762 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f9d740019f0 con 0x7f9d7c06cc40 2024-03-24T01:26:30.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.357+0000 7f9d7b400640 1 -- 172.21.15.90:0/1971177762 >> v1:172.21.15.90:6789/0 conn(0x7f9d7c06cc40 legacy=0x7f9d7c0b6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:30.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.358+0000 7f9d7b400640 1 --2- 172.21.15.90:0/1971177762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d600039b0 0x7f9d60003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:30.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.358+0000 7f9d7b400640 1 --2- 172.21.15.90:0/1971177762 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d7c06c860 0x7f9d60004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:30.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.358+0000 7f9d7b400640 1 --2- 172.21.15.90:0/1971177762 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d60004d40 0x7f9d600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:30.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.358+0000 7f9d7b400640 1 -- 172.21.15.90:0/1971177762 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c0d2730 con 0x7f9d60004d40 2024-03-24T01:26:30.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.358+0000 7f9d7b400640 1 -- 172.21.15.90:0/1971177762 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c0d1e00 con 0x7f9d600039b0 2024-03-24T01:26:30.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.358+0000 7f9d7b400640 1 -- 172.21.15.90:0/1971177762 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6000b9c0 con 0x7f9d7c06c860 2024-03-24T01:26:30.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.358+0000 7f9d81a00640 1 --2- 172.21.15.90:0/1971177762 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d60004d40 0x7f9d600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:30.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.358+0000 7f9d81000640 1 --2- 172.21.15.90:0/1971177762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d600039b0 0x7f9d60003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:30.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.358+0000 7f9d7be00640 1 --2- 172.21.15.90:0/1971177762 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d7c06c860 0x7f9d60004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:30.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.358+0000 7f9d81a00640 1 -- 172.21.15.90:0/1971177762 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d7c06c860 msgr2=0x7f9d60004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:30.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.358+0000 7f9d81a00640 1 --2- 172.21.15.90:0/1971177762 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d7c06c860 0x7f9d60004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:30.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.358+0000 7f9d81a00640 1 -- 172.21.15.90:0/1971177762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d600039b0 msgr2=0x7f9d60003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:30.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.358+0000 7f9d81a00640 1 --2- 172.21.15.90:0/1971177762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d600039b0 0x7f9d60003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:30.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.358+0000 7f9d81a00640 1 -- 172.21.15.90:0/1971177762 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d60003520 con 0x7f9d60004d40 2024-03-24T01:26:30.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.359+0000 7f9d81a00640 1 --2- 172.21.15.90:0/1971177762 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d60004d40 0x7f9d600051c0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f9d7400a7b0 tx=0x7f9d60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da16e0b7bf9e8d62 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:30.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.360+0000 7f9d7b400640 1 -- 172.21.15.90:0/1971177762 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7400a3d0 con 0x7f9d60004d40 2024-03-24T01:26:30.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.360+0000 7f9d7b400640 1 -- 172.21.15.90:0/1971177762 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d74034630 con 0x7f9d60004d40 2024-03-24T01:26:30.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.360+0000 7f9d7b400640 1 -- 172.21.15.90:0/1971177762 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d74034a60 con 0x7f9d60004d40 2024-03-24T01:26:30.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.360+0000 7f9d82400640 1 -- 172.21.15.90:0/1971177762 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d60004d40 msgr2=0x7f9d600051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:30.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.360+0000 7f9d82400640 1 --2- 172.21.15.90:0/1971177762 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d60004d40 0x7f9d600051c0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f9d7400a7b0 tx=0x7f9d60005830 comp rx=0 tx=0).stop 2024-03-24T01:26:30.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.360+0000 7f9d81a00640 1 -- 172.21.15.90:0/1971177762 reap_dead start 2024-03-24T01:26:30.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.360+0000 7f9d82400640 1 -- 172.21.15.90:0/1971177762 shutdown_connections 2024-03-24T01:26:30.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.360+0000 7f9d82400640 1 -- 172.21.15.90:0/1971177762 >> 172.21.15.90:0/1971177762 conn(0x7f9d7c0d0560 msgr2=0x7f9d7c0d0970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:30.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.361+0000 7f9d82400640 1 -- 172.21.15.90:0/1971177762 shutdown_connections 2024-03-24T01:26:30.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.361+0000 7f9d82400640 1 -- 172.21.15.90:0/1971177762 wait complete. 2024-03-24T01:26:30.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.362+0000 7f9d82400640 1 Processor -- start 2024-03-24T01:26:30.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.362+0000 7f9d82400640 1 -- start start 2024-03-24T01:26:30.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.363+0000 7f9d82400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d60004d40 0x7f9d7c1f7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:30.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.363+0000 7f9d82400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d7c14cfa0 0x7f9d7c1604d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:30.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.363+0000 7f9d82400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d7c0ccfb0 0x7f9d7c0cd410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:30.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.363+0000 7f9d82400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c0c7300 con 0x7f9d7c0ccfb0 2024-03-24T01:26:30.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.363+0000 7f9d82400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c0c7180 con 0x7f9d60004d40 2024-03-24T01:26:30.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.363+0000 7f9d82400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c0b7700 con 0x7f9d7c14cfa0 2024-03-24T01:26:30.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.364+0000 7f9d81a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d7c0ccfb0 0x7f9d7c0cd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:30.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.364+0000 7f9d81a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d7c0ccfb0 0x7f9d7c0cd410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59526/0 (socket says 172.21.15.90:59526) 2024-03-24T01:26:30.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.364+0000 7f9d81a00640 1 -- 172.21.15.90:0/914962456 learned_addr learned my addr 172.21.15.90:0/914962456 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:30.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.364+0000 7f9d81000640 1 --2- 172.21.15.90:0/914962456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d60004d40 0x7f9d7c1f7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:30.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.364+0000 7f9d81a00640 1 -- 172.21.15.90:0/914962456 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d7c14cfa0 msgr2=0x7f9d7c1604d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:30.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.364+0000 7f9d81a00640 1 --2- 172.21.15.90:0/914962456 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d7c14cfa0 0x7f9d7c1604d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:30.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.364+0000 7f9d81a00640 1 -- 172.21.15.90:0/914962456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d60004d40 msgr2=0x7f9d7c1f7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:30.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.364+0000 7f9d81a00640 1 --2- 172.21.15.90:0/914962456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d60004d40 0x7f9d7c1f7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:30.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.364+0000 7f9d81a00640 1 -- 172.21.15.90:0/914962456 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d7c0aade0 con 0x7f9d7c0ccfb0 2024-03-24T01:26:30.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.364+0000 7f9d81a00640 1 --2- 172.21.15.90:0/914962456 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d7c0ccfb0 0x7f9d7c0cd410 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f9d740019c0 tx=0x7f9d7c0d1cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:30.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.365+0000 7f9d78c00640 1 -- 172.21.15.90:0/914962456 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d74045070 con 0x7f9d7c0ccfb0 2024-03-24T01:26:30.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.365+0000 7f9d78c00640 1 -- 172.21.15.90:0/914962456 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d7403a040 con 0x7f9d7c0ccfb0 2024-03-24T01:26:30.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.365+0000 7f9d78c00640 1 -- 172.21.15.90:0/914962456 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7400a3d0 con 0x7f9d7c0ccfb0 2024-03-24T01:26:30.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.365+0000 7f9d82400640 1 -- 172.21.15.90:0/914962456 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d7c14d420 con 0x7f9d7c0ccfb0 2024-03-24T01:26:30.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.365+0000 7f9d82400640 1 -- 172.21.15.90:0/914962456 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d7c0ab050 con 0x7f9d7c0ccfb0 2024-03-24T01:26:30.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.366+0000 7f9d82400640 1 -- 172.21.15.90:0/914962456 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d7c0cd950 con 0x7f9d7c0ccfb0 2024-03-24T01:26:30.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.367+0000 7f9d78c00640 1 -- 172.21.15.90:0/914962456 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9d74048070 con 0x7f9d7c0ccfb0 2024-03-24T01:26:30.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.368+0000 7f9d78c00640 1 --2- 172.21.15.90:0/914962456 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9d5c06bbd0 0x7f9d5c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:30.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.368+0000 7f9d78c00640 1 -- 172.21.15.90:0/914962456 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f9d740aca00 con 0x7f9d7c0ccfb0 2024-03-24T01:26:30.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.368+0000 7f9d81000640 1 --2- 172.21.15.90:0/914962456 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9d5c06bbd0 0x7f9d5c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:30.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.369+0000 7f9d81000640 1 --2- 172.21.15.90:0/914962456 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9d5c06bbd0 0x7f9d5c06e090 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c0098d0 tx=0x7f9d6c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:30.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.372+0000 7f9d78c00640 1 -- 172.21.15.90:0/914962456 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9d7407b270 con 0x7f9d7c0ccfb0 2024-03-24T01:26:30.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.605+0000 7f9d82400640 1 -- 172.21.15.90:0/914962456 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d7c04f400 con 0x7f9d7c0ccfb0 2024-03-24T01:26:30.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.607+0000 7f9d78c00640 1 -- 172.21.15.90:0/914962456 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+4965 (secure 0 0 0) 0x7f9d7407ab70 con 0x7f9d7c0ccfb0 2024-03-24T01:26:30.607 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:26:30.607 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":21,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/296084634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":296084634},{"type":"v1","addr":"172.21.15.120:6835","nonce":296084634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/797786252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":797786252},{"type":"v1","addr":"172.21.15.120:6837","nonce":797786252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5490,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6833/1983177399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1983177399},{"type":"v1","addr":"172.21.15.90:6833","nonce":1983177399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T01:25:59.440431+0000","modified":"2024-03-24T01:26:30.144653+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5787},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5787":{"gid":5787,"name":"c","rank":0,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.90:6835/11525832","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":11525832},{"type":"v1","addr":"172.21.15.90:6835","nonce":11525832}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-24T01:26:30.607 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 21 2024-03-24T01:26:30.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.609+0000 7f9d82400640 1 -- 172.21.15.90:0/914962456 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9d5c06bbd0 msgr2=0x7f9d5c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:30.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.609+0000 7f9d82400640 1 --2- 172.21.15.90:0/914962456 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9d5c06bbd0 0x7f9d5c06e090 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c0098d0 tx=0x7f9d6c008040 comp rx=0 tx=0).stop 2024-03-24T01:26:30.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.609+0000 7f9d82400640 1 -- 172.21.15.90:0/914962456 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d7c0ccfb0 msgr2=0x7f9d7c0cd410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:30.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.609+0000 7f9d82400640 1 --2- 172.21.15.90:0/914962456 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d7c0ccfb0 0x7f9d7c0cd410 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f9d740019c0 tx=0x7f9d7c0d1cf0 comp rx=0 tx=0).stop 2024-03-24T01:26:30.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.609+0000 7f9d82400640 1 -- 172.21.15.90:0/914962456 shutdown_connections 2024-03-24T01:26:30.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.609+0000 7f9d82400640 1 --2- 172.21.15.90:0/914962456 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9d5c06bbd0 0x7f9d5c06e090 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:30.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.609+0000 7f9d82400640 1 --2- 172.21.15.90:0/914962456 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d7c0ccfb0 0x7f9d7c0cd410 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:30.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.609+0000 7f9d82400640 1 --2- 172.21.15.90:0/914962456 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9d7c14cfa0 0x7f9d7c1604d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:30.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.609+0000 7f9d82400640 1 --2- 172.21.15.90:0/914962456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d60004d40 0x7f9d7c1f7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:30.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.609+0000 7f9d82400640 1 -- 172.21.15.90:0/914962456 >> 172.21.15.90:0/914962456 conn(0x7f9d7c0d0560 msgr2=0x7f9d7c0b8920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:30.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.609+0000 7f9d82400640 1 -- 172.21.15.90:0/914962456 shutdown_connections 2024-03-24T01:26:30.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:30.609+0000 7f9d82400640 1 -- 172.21.15.90:0/914962456 wait complete. 2024-03-24T01:26:31.029 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T01:25:59.440431+0000', 'modified': '2024-03-24T01:26:30.144653+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 40, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5787}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5787': {'gid': 5787, 'name': 'c', 'rank': 0, 'incarnation': 21, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.90:6835/11525832', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6834', 'nonce': 11525832}, {'type': 'v1', 'addr': '172.21.15.90:6835', 'nonce': 11525832}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-24T01:26:31.029 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5787:up:replay 2024-03-24T01:26:32.029 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:26:32.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.245+0000 7fddd8a00640 1 Processor -- start 2024-03-24T01:26:32.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.245+0000 7fddd8a00640 1 -- start start 2024-03-24T01:26:32.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.246+0000 7fddd8a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fddd415a280 0x7fddd406b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:32.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.246+0000 7fddd8a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fddd406f080 0x7fddd406f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:32.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.247+0000 7fddd8a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddd40b3e70 0x7fddd40b4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:32.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.247+0000 7fddd8a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddd406fa20 con 0x7fddd415a280 2024-03-24T01:26:32.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.247+0000 7fddd8a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddd415a660 con 0x7fddd406f080 2024-03-24T01:26:32.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.247+0000 7fddd8a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddd406dac0 con 0x7fddd40b3e70 2024-03-24T01:26:32.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.247+0000 7fddd1e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fddd415a280 0x7fddd406b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:32.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.247+0000 7fddd1e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fddd415a280 0x7fddd406b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59530/0 (socket says 172.21.15.90:59530) 2024-03-24T01:26:32.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.247+0000 7fddd1e00640 1 -- 172.21.15.90:0/1805040690 learned_addr learned my addr 172.21.15.90:0/1805040690 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:32.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.247+0000 7fddd1e00640 1 -- 172.21.15.90:0/1805040690 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddd40b3e70 msgr2=0x7fddd40b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:32.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.247+0000 7fddd1e00640 1 --2- 172.21.15.90:0/1805040690 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddd40b3e70 0x7fddd40b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:32.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.247+0000 7fddd1e00640 1 -- 172.21.15.90:0/1805040690 >> v2:172.21.15.120:3300/0 conn(0x7fddd406f080 msgr2=0x7fddd406f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:32.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.247+0000 7fddd1e00640 1 --2- 172.21.15.90:0/1805040690 >> v2:172.21.15.120:3300/0 conn(0x7fddd406f080 0x7fddd406f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:32.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.247+0000 7fddd1e00640 1 -- 172.21.15.90:0/1805040690 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd406dda0 con 0x7fddd415a280 2024-03-24T01:26:32.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.248+0000 7fddd1e00640 1 --2- 172.21.15.90:0/1805040690 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fddd415a280 0x7fddd406b0c0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fddd406bbc0 tx=0x7fddc402e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48370687084048b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:32.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.249+0000 7fddd0a00640 1 -- 172.21.15.90:0/1805040690 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc403b070 con 0x7fddd415a280 2024-03-24T01:26:32.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.249+0000 7fddd0a00640 1 -- 172.21.15.90:0/1805040690 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddc402ee00 con 0x7fddd415a280 2024-03-24T01:26:32.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.249+0000 7fddd0a00640 1 -- 172.21.15.90:0/1805040690 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc4036660 con 0x7fddd415a280 2024-03-24T01:26:32.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.249+0000 7fddd8a00640 1 -- 172.21.15.90:0/1805040690 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fddd415a280 msgr2=0x7fddd406b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:32.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.249+0000 7fddd8a00640 1 --2- 172.21.15.90:0/1805040690 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fddd415a280 0x7fddd406b0c0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fddd406bbc0 tx=0x7fddc402e050 comp rx=0 tx=0).stop 2024-03-24T01:26:32.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.249+0000 7fddd8a00640 1 -- 172.21.15.90:0/1805040690 shutdown_connections 2024-03-24T01:26:32.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.249+0000 7fddd8a00640 1 --2- 172.21.15.90:0/1805040690 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddd40b3e70 0x7fddd40b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:32.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.249+0000 7fddd8a00640 1 --2- 172.21.15.90:0/1805040690 >> v2:172.21.15.120:3300/0 conn(0x7fddd406f080 0x7fddd406f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:32.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.249+0000 7fddd8a00640 1 --2- 172.21.15.90:0/1805040690 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fddd415a280 0x7fddd406b0c0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:32.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.249+0000 7fddd8a00640 1 -- 172.21.15.90:0/1805040690 >> 172.21.15.90:0/1805040690 conn(0x7fddd40b1770 msgr2=0x7fddd40b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:32.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.249+0000 7fddd8a00640 1 -- 172.21.15.90:0/1805040690 shutdown_connections 2024-03-24T01:26:32.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.250+0000 7fddd8a00640 1 -- 172.21.15.90:0/1805040690 wait complete. 2024-03-24T01:26:32.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.251+0000 7fddd8a00640 1 Processor -- start 2024-03-24T01:26:32.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.251+0000 7fddd8a00640 1 -- start start 2024-03-24T01:26:32.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.252+0000 7fddd8a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddd406f080 0x7fddd4149990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:32.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.252+0000 7fddd8a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fddd40b3e70 0x7fddd4146370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:32.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.252+0000 7fddd8a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddd415a280 0x7fddd4151050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:32.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.252+0000 7fddd8a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd406e640 con 0x7fddd40b3e70 2024-03-24T01:26:32.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.252+0000 7fddd8a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd406e360 con 0x7fddd406f080 2024-03-24T01:26:32.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.252+0000 7fddd8a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddd406b600 con 0x7fddd415a280 2024-03-24T01:26:32.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.252+0000 7fddd1400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fddd40b3e70 0x7fddd4146370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:32.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.253+0000 7fddd1400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fddd40b3e70 0x7fddd4146370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59532/0 (socket says 172.21.15.90:59532) 2024-03-24T01:26:32.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.253+0000 7fddd1400640 1 -- 172.21.15.90:0/1307738424 learned_addr learned my addr 172.21.15.90:0/1307738424 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:32.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.253+0000 7fddd1400640 1 -- 172.21.15.90:0/1307738424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddd415a280 msgr2=0x7fddd4151050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:32.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.253+0000 7fddd1400640 1 --2- 172.21.15.90:0/1307738424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddd415a280 0x7fddd4151050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:32.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.253+0000 7fddd1e00640 1 --2- 172.21.15.90:0/1307738424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddd406f080 0x7fddd4149990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:32.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.253+0000 7fddd1400640 1 -- 172.21.15.90:0/1307738424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddd406f080 msgr2=0x7fddd4149990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:32.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.253+0000 7fddd1400640 1 --2- 172.21.15.90:0/1307738424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddd406f080 0x7fddd4149990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:32.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.253+0000 7fddd1400640 1 -- 172.21.15.90:0/1307738424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd40ac8e0 con 0x7fddd40b3e70 2024-03-24T01:26:32.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.253+0000 7fddd1400640 1 --2- 172.21.15.90:0/1307738424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fddd40b3e70 0x7fddd4146370 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fddc000b410 tx=0x7fddc000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:32.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.254+0000 7fddca000640 1 -- 172.21.15.90:0/1307738424 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc0015020 con 0x7fddd40b3e70 2024-03-24T01:26:32.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.254+0000 7fddca000640 1 -- 172.21.15.90:0/1307738424 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddc0011850 con 0x7fddd40b3e70 2024-03-24T01:26:32.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.254+0000 7fddd8a00640 1 -- 172.21.15.90:0/1307738424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddd415aba0 con 0x7fddd40b3e70 2024-03-24T01:26:32.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.254+0000 7fddca000640 1 -- 172.21.15.90:0/1307738424 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc0011c40 con 0x7fddd40b3e70 2024-03-24T01:26:32.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.254+0000 7fddd8a00640 1 -- 172.21.15.90:0/1307738424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddd40acb80 con 0x7fddd40b3e70 2024-03-24T01:26:32.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.255+0000 7fddd8a00640 1 -- 172.21.15.90:0/1307738424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddd4068ac0 con 0x7fddd40b3e70 2024-03-24T01:26:32.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.256+0000 7fddca000640 1 -- 172.21.15.90:0/1307738424 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fddc0018020 con 0x7fddd40b3e70 2024-03-24T01:26:32.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.257+0000 7fddca000640 1 --2- 172.21.15.90:0/1307738424 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fddb406bbd0 0x7fddb406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:32.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.257+0000 7fddca000640 1 -- 172.21.15.90:0/1307738424 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fddc008ac10 con 0x7fddd40b3e70 2024-03-24T01:26:32.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.258+0000 7fddd1e00640 1 --2- 172.21.15.90:0/1307738424 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fddb406bbd0 0x7fddb406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:32.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.259+0000 7fddd1e00640 1 --2- 172.21.15.90:0/1307738424 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fddb406bbd0 0x7fddb406e090 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fddc4038000 tx=0x7fddc4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:32.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.262+0000 7fddca000640 1 -- 172.21.15.90:0/1307738424 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fddd4068ac0 con 0x7fddd40b3e70 2024-03-24T01:26:32.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.500+0000 7fddd8a00640 1 -- 172.21.15.90:0/1307738424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fddd4068060 con 0x7fddd40b3e70 2024-03-24T01:26:32.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.501+0000 7fddca000640 1 -- 172.21.15.90:0/1307738424 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+4965 (secure 0 0 0) 0x7fddc001f020 con 0x7fddd40b3e70 2024-03-24T01:26:32.501 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:26:32.501 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/296084634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":296084634},{"type":"v1","addr":"172.21.15.120:6835","nonce":296084634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/797786252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":797786252},{"type":"v1","addr":"172.21.15.120:6837","nonce":797786252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5490,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6833/1983177399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1983177399},{"type":"v1","addr":"172.21.15.90:6833","nonce":1983177399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T01:25:59.440431+0000","modified":"2024-03-24T01:26:31.156801+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5787},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5787":{"gid":5787,"name":"c","rank":0,"incarnation":21,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.90:6835/11525832","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":11525832},{"type":"v1","addr":"172.21.15.90:6835","nonce":11525832}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-03-24T01:26:32.502 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 23 2024-03-24T01:26:32.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.503+0000 7fddd8a00640 1 -- 172.21.15.90:0/1307738424 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fddb406bbd0 msgr2=0x7fddb406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:32.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.504+0000 7fddd8a00640 1 --2- 172.21.15.90:0/1307738424 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fddb406bbd0 0x7fddb406e090 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fddc4038000 tx=0x7fddc4008040 comp rx=0 tx=0).stop 2024-03-24T01:26:32.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.504+0000 7fddd8a00640 1 -- 172.21.15.90:0/1307738424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fddd40b3e70 msgr2=0x7fddd4146370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:32.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.504+0000 7fddd8a00640 1 --2- 172.21.15.90:0/1307738424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fddd40b3e70 0x7fddd4146370 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fddc000b410 tx=0x7fddc000b8d0 comp rx=0 tx=0).stop 2024-03-24T01:26:32.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.504+0000 7fddd8a00640 1 -- 172.21.15.90:0/1307738424 shutdown_connections 2024-03-24T01:26:32.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.504+0000 7fddd8a00640 1 --2- 172.21.15.90:0/1307738424 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fddb406bbd0 0x7fddb406e090 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:32.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.504+0000 7fddd8a00640 1 --2- 172.21.15.90:0/1307738424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fddd40b3e70 0x7fddd4146370 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:32.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.504+0000 7fddd8a00640 1 --2- 172.21.15.90:0/1307738424 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fddd415a280 0x7fddd4151050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:32.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.504+0000 7fddd8a00640 1 --2- 172.21.15.90:0/1307738424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddd406f080 0x7fddd4149990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:32.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.504+0000 7fddd8a00640 1 -- 172.21.15.90:0/1307738424 >> 172.21.15.90:0/1307738424 conn(0x7fddd40b1770 msgr2=0x7fddd40b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:32.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.504+0000 7fddd8a00640 1 -- 172.21.15.90:0/1307738424 shutdown_connections 2024-03-24T01:26:32.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:32.504+0000 7fddd8a00640 1 -- 172.21.15.90:0/1307738424 wait complete. 2024-03-24T01:26:32.926 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T01:25:59.440431+0000', 'modified': '2024-03-24T01:26:31.156801+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 40, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5787}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5787': {'gid': 5787, 'name': 'c', 'rank': 0, 'incarnation': 21, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.90:6835/11525832', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6834', 'nonce': 11525832}, {'type': 'v1', 'addr': '172.21.15.90:6835', 'nonce': 11525832}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-24T01:26:32.926 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5787:up:rejoin 2024-03-24T01:26:33.669 INFO:tasks.ceph.mds.c.smithi090.stderr:2024-03-24T01:26:33.668+0000 7f40cfc00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T01:26:33.927 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:26:34.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.138+0000 7f11af000640 1 Processor -- start 2024-03-24T01:26:34.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.139+0000 7f11af000640 1 -- start start 2024-03-24T01:26:34.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.140+0000 7f11af000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f11a80cc150 0x7f11a80ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:34.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.140+0000 7f11af000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f11a81603e0 0x7f11a806b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:34.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.140+0000 7f11af000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11a80cd490 con 0x7f11a8145430 2024-03-24T01:26:34.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.140+0000 7f11af000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11a815f900 con 0x7f11a80cc150 2024-03-24T01:26:34.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.140+0000 7f11af000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11a815fbe0 con 0x7f11a81603e0 2024-03-24T01:26:34.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.140+0000 7f11a7e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f11a80cc150 0x7f11a80ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:34.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.141+0000 7f11a7e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f11a80cc150 0x7f11a80ccb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59542/0 (socket says 172.21.15.90:59542) 2024-03-24T01:26:34.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.141+0000 7f11a7e00640 1 -- 172.21.15.90:0/1173178299 learned_addr learned my addr 172.21.15.90:0/1173178299 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:34.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.141+0000 7f119f400640 1 --2- 172.21.15.90:0/1173178299 >> v2:172.21.15.120:3300/0 conn(0x7f11a81603e0 0x7f11a806b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:34.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.141+0000 7f11a7e00640 1 -- 172.21.15.90:0/1173178299 >> v2:172.21.15.120:3300/0 conn(0x7f11a81603e0 msgr2=0x7f11a806b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:34.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.141+0000 7f11a7e00640 1 --2- 172.21.15.90:0/1173178299 >> v2:172.21.15.120:3300/0 conn(0x7f11a81603e0 0x7f11a806b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:34.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.141+0000 7f11a7e00640 1 -- 172.21.15.90:0/1173178299 >> v1:172.21.15.120:6789/0 conn(0x7f11a8145430 legacy=0x7f11a8071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:34.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.141+0000 7f11a7e00640 1 -- 172.21.15.90:0/1173178299 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a815fec0 con 0x7f11a80cc150 2024-03-24T01:26:34.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.143+0000 7f11a7e00640 1 --2- 172.21.15.90:0/1173178299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11a80cc150 0x7f11a80ccb60 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f11a806c2d0 tx=0x7f119802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2594733dd3e3372 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:34.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.143+0000 7f11a7400640 1 -- 172.21.15.90:0/1173178299 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1198035020 con 0x7f11a80cc150 2024-03-24T01:26:34.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.143+0000 7f11a7400640 1 -- 172.21.15.90:0/1173178299 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1198031870 con 0x7f11a80cc150 2024-03-24T01:26:34.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.143+0000 7f11a7400640 1 -- 172.21.15.90:0/1173178299 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1198031c40 con 0x7f11a80cc150 2024-03-24T01:26:34.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.144+0000 7f11af000640 1 -- 172.21.15.90:0/1173178299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11a80cc150 msgr2=0x7f11a80ccb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:34.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.144+0000 7f11af000640 1 --2- 172.21.15.90:0/1173178299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11a80cc150 0x7f11a80ccb60 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f11a806c2d0 tx=0x7f119802b910 comp rx=0 tx=0).stop 2024-03-24T01:26:34.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.144+0000 7f11af000640 1 -- 172.21.15.90:0/1173178299 shutdown_connections 2024-03-24T01:26:34.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.144+0000 7f11af000640 1 --2- 172.21.15.90:0/1173178299 >> v2:172.21.15.120:3300/0 conn(0x7f11a81603e0 0x7f11a806b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:34.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.144+0000 7f11af000640 1 --2- 172.21.15.90:0/1173178299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11a80cc150 0x7f11a80ccb60 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:34.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.144+0000 7f11af000640 1 -- 172.21.15.90:0/1173178299 >> 172.21.15.90:0/1173178299 conn(0x7f11a80b1770 msgr2=0x7f11a80b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:34.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.144+0000 7f11af000640 1 -- 172.21.15.90:0/1173178299 shutdown_connections 2024-03-24T01:26:34.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.144+0000 7f11af000640 1 -- 172.21.15.90:0/1173178299 wait complete. 2024-03-24T01:26:34.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.147+0000 7f11af000640 1 Processor -- start 2024-03-24T01:26:34.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.147+0000 7f11af000640 1 -- start start 2024-03-24T01:26:34.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.148+0000 7f11af000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11a80cc150 0x7f11a80ad850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:34.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.149+0000 7f11af000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11a8145430 0x7f11a80a93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:34.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.149+0000 7f11af000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11a81603e0 0x7f11a80a9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:34.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.149+0000 7f11af000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a806b780 con 0x7f11a81603e0 2024-03-24T01:26:34.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.149+0000 7f11af000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a806b600 con 0x7f11a8145430 2024-03-24T01:26:34.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.149+0000 7f11af000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11a806b900 con 0x7f11a80cc150 2024-03-24T01:26:34.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.149+0000 7f11ad000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11a81603e0 0x7f11a80a9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:34.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.149+0000 7f11ad000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11a81603e0 0x7f11a80a9900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59548/0 (socket says 172.21.15.90:59548) 2024-03-24T01:26:34.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.149+0000 7f11ad000640 1 -- 172.21.15.90:0/2259287673 learned_addr learned my addr 172.21.15.90:0/2259287673 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:34.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.149+0000 7f11a7e00640 1 --2- 172.21.15.90:0/2259287673 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11a80cc150 0x7f11a80ad850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:34.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.149+0000 7f119f400640 1 --2- 172.21.15.90:0/2259287673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11a8145430 0x7f11a80a93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:34.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.149+0000 7f11ad000640 1 -- 172.21.15.90:0/2259287673 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11a80cc150 msgr2=0x7f11a80ad850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:34.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.149+0000 7f11ad000640 1 --2- 172.21.15.90:0/2259287673 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11a80cc150 0x7f11a80ad850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:34.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.149+0000 7f11ad000640 1 -- 172.21.15.90:0/2259287673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11a8145430 msgr2=0x7f11a80a93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:34.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.149+0000 7f11ad000640 1 --2- 172.21.15.90:0/2259287673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11a8145430 0x7f11a80a93c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:34.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.149+0000 7f11ad000640 1 -- 172.21.15.90:0/2259287673 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a8132160 con 0x7f11a81603e0 2024-03-24T01:26:34.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.150+0000 7f11ad000640 1 --2- 172.21.15.90:0/2259287673 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11a81603e0 0x7f11a80a9900 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f11a000c780 tx=0x7f11a000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:34.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.150+0000 7f11a4c00640 1 -- 172.21.15.90:0/2259287673 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a0004b60 con 0x7f11a81603e0 2024-03-24T01:26:34.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.150+0000 7f11a4c00640 1 -- 172.21.15.90:0/2259287673 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11a0004d00 con 0x7f11a81603e0 2024-03-24T01:26:34.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.150+0000 7f11a4c00640 1 -- 172.21.15.90:0/2259287673 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a000aa90 con 0x7f11a81603e0 2024-03-24T01:26:34.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.150+0000 7f11af000640 1 -- 172.21.15.90:0/2259287673 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11a8132440 con 0x7f11a81603e0 2024-03-24T01:26:34.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.150+0000 7f11af000640 1 -- 172.21.15.90:0/2259287673 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11a80a9e40 con 0x7f11a81603e0 2024-03-24T01:26:34.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.152+0000 7f11af000640 1 -- 172.21.15.90:0/2259287673 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11a8068060 con 0x7f11a81603e0 2024-03-24T01:26:34.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.153+0000 7f11a4c00640 1 -- 172.21.15.90:0/2259287673 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f11a000ac30 con 0x7f11a81603e0 2024-03-24T01:26:34.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.154+0000 7f11a4c00640 1 --2- 172.21.15.90:0/2259287673 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f118406bbd0 0x7f118406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:34.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.154+0000 7f11a4c00640 1 -- 172.21.15.90:0/2259287673 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f11a008af40 con 0x7f11a81603e0 2024-03-24T01:26:34.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.155+0000 7f11a7e00640 1 --2- 172.21.15.90:0/2259287673 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f118406bbd0 0x7f118406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:34.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.156+0000 7f11a7e00640 1 --2- 172.21.15.90:0/2259287673 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f118406bbd0 0x7f118406e090 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1198036000 tx=0x7f119803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:34.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.160+0000 7f11a4c00640 1 -- 172.21.15.90:0/2259287673 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f11a0020070 con 0x7f11a81603e0 2024-03-24T01:26:34.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.501+0000 7f11af000640 1 -- 172.21.15.90:0/2259287673 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11a80b3220 con 0x7f11a81603e0 2024-03-24T01:26:34.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.502+0000 7f11a4c00640 1 -- 172.21.15.90:0/2259287673 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4972 (secure 0 0 0) 0x7f11a0059850 con 0x7f11a81603e0 2024-03-24T01:26:34.503 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:26:34.503 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":24,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/296084634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":296084634},{"type":"v1","addr":"172.21.15.120:6835","nonce":296084634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/797786252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":797786252},{"type":"v1","addr":"172.21.15.120:6837","nonce":797786252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5490,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6833/1983177399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1983177399},{"type":"v1","addr":"172.21.15.90:6833","nonce":1983177399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T01:25:59.440431+0000","modified":"2024-03-24T01:26:33.165178+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5787},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5787":{"gid":5787,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":4,"addr":"172.21.15.90:6835/11525832","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":11525832},{"type":"v1","addr":"172.21.15.90:6835","nonce":11525832}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5787,"qdb_cluster":[5787]},"id":2}]} 2024-03-24T01:26:34.503 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 24 2024-03-24T01:26:34.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.505+0000 7f11af000640 1 -- 172.21.15.90:0/2259287673 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f118406bbd0 msgr2=0x7f118406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:34.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.505+0000 7f11af000640 1 --2- 172.21.15.90:0/2259287673 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f118406bbd0 0x7f118406e090 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1198036000 tx=0x7f119803c040 comp rx=0 tx=0).stop 2024-03-24T01:26:34.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.505+0000 7f11af000640 1 -- 172.21.15.90:0/2259287673 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11a81603e0 msgr2=0x7f11a80a9900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:34.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.505+0000 7f11af000640 1 --2- 172.21.15.90:0/2259287673 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11a81603e0 0x7f11a80a9900 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f11a000c780 tx=0x7f11a000cc40 comp rx=0 tx=0).stop 2024-03-24T01:26:34.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.505+0000 7f11af000640 1 -- 172.21.15.90:0/2259287673 shutdown_connections 2024-03-24T01:26:34.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.505+0000 7f11af000640 1 --2- 172.21.15.90:0/2259287673 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f118406bbd0 0x7f118406e090 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:34.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.505+0000 7f11af000640 1 --2- 172.21.15.90:0/2259287673 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11a81603e0 0x7f11a80a9900 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:34.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.505+0000 7f11af000640 1 --2- 172.21.15.90:0/2259287673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11a8145430 0x7f11a80a93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:34.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.505+0000 7f11af000640 1 --2- 172.21.15.90:0/2259287673 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11a80cc150 0x7f11a80ad850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:34.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.505+0000 7f11af000640 1 -- 172.21.15.90:0/2259287673 >> 172.21.15.90:0/2259287673 conn(0x7f11a80b1770 msgr2=0x7f11a80af500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:34.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.505+0000 7f11af000640 1 -- 172.21.15.90:0/2259287673 shutdown_connections 2024-03-24T01:26:34.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:34.505+0000 7f11af000640 1 -- 172.21.15.90:0/2259287673 wait complete. 2024-03-24T01:26:34.925 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T01:25:59.440431+0000', 'modified': '2024-03-24T01:26:33.165178+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 40, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5787}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5787': {'gid': 5787, 'name': 'c', 'rank': 0, 'incarnation': 21, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.90:6835/11525832', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6834', 'nonce': 11525832}, {'type': 'v1', 'addr': '172.21.15.90:6835', 'nonce': 11525832}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5787, 'qdb_cluster': [5787]} 2024-03-24T01:26:34.925 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T01:26:34.925 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-24T01:26:35.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.135+0000 7fe098400640 1 Processor -- start 2024-03-24T01:26:35.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.135+0000 7fe098400640 1 -- start start 2024-03-24T01:26:35.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.138+0000 7fe098400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0900aeeb0 0x7fe0900b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:35.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.138+0000 7fe098400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe09015bc50 0x7fe09006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:35.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.138+0000 7fe098400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe09006dac0 con 0x7fe09006b600 2024-03-24T01:26:35.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.138+0000 7fe098400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe09006ba70 con 0x7fe09015bc50 2024-03-24T01:26:35.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.138+0000 7fe098400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0900b4da0 con 0x7fe0900aeeb0 2024-03-24T01:26:35.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.138+0000 7fe095000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe09015bc50 0x7fe09006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:35.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.138+0000 7fe095a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0900aeeb0 0x7fe0900b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:35.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.138+0000 7fe095000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe09015bc50 0x7fe09006b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.90:52912/0 (socket says 172.21.15.90:52912) 2024-03-24T01:26:35.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.138+0000 7fe095000640 1 -- 172.21.15.90:0/1808827982 learned_addr learned my addr 172.21.15.90:0/1808827982 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:35.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.139+0000 7fe095000640 1 -- 172.21.15.90:0/1808827982 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0900aeeb0 msgr2=0x7fe0900b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:35.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.139+0000 7fe095000640 1 --2- 172.21.15.90:0/1808827982 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0900aeeb0 0x7fe0900b44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.139+0000 7fe095000640 1 -- 172.21.15.90:0/1808827982 >> v1:172.21.15.120:6789/0 conn(0x7fe09006b600 legacy=0x7fe09015e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:35.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.139+0000 7fe095000640 1 -- 172.21.15.90:0/1808827982 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe09006dda0 con 0x7fe09015bc50 2024-03-24T01:26:35.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.141+0000 7fe095000640 1 --2- 172.21.15.90:0/1808827982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe09015bc50 0x7fe09006b0c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe07c009870 tx=0x7fe07c02b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb277602aacf2791 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:35.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.141+0000 7fe08fe00640 1 -- 172.21.15.90:0/1808827982 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07c0317c0 con 0x7fe09015bc50 2024-03-24T01:26:35.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.141+0000 7fe08fe00640 1 -- 172.21.15.90:0/1808827982 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe07c037070 con 0x7fe09015bc50 2024-03-24T01:26:35.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.141+0000 7fe08fe00640 1 -- 172.21.15.90:0/1808827982 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07c031b80 con 0x7fe09015bc50 2024-03-24T01:26:35.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.142+0000 7fe098400640 1 -- 172.21.15.90:0/1808827982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe09015bc50 msgr2=0x7fe09006b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:35.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.142+0000 7fe098400640 1 --2- 172.21.15.90:0/1808827982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe09015bc50 0x7fe09006b0c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe07c009870 tx=0x7fe07c02b860 comp rx=0 tx=0).stop 2024-03-24T01:26:35.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.142+0000 7fe098400640 1 -- 172.21.15.90:0/1808827982 shutdown_connections 2024-03-24T01:26:35.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.142+0000 7fe098400640 1 --2- 172.21.15.90:0/1808827982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe09015bc50 0x7fe09006b0c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.142+0000 7fe098400640 1 --2- 172.21.15.90:0/1808827982 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe0900aeeb0 0x7fe0900b44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.142+0000 7fe098400640 1 -- 172.21.15.90:0/1808827982 >> 172.21.15.90:0/1808827982 conn(0x7fe0900b1770 msgr2=0x7fe0900b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:35.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.142+0000 7fe098400640 1 -- 172.21.15.90:0/1808827982 shutdown_connections 2024-03-24T01:26:35.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.142+0000 7fe098400640 1 -- 172.21.15.90:0/1808827982 wait complete. 2024-03-24T01:26:35.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.144+0000 7fe098400640 1 Processor -- start 2024-03-24T01:26:35.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.145+0000 7fe098400640 1 -- start start 2024-03-24T01:26:35.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.146+0000 7fe098400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe09006b600 0x7fe090145350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:35.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.146+0000 7fe098400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0900aeeb0 0x7fe090159d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:35.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.146+0000 7fe098400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe09015bc50 0x7fe09015a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:35.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.146+0000 7fe098400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0900af290 con 0x7fe09015bc50 2024-03-24T01:26:35.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.146+0000 7fe098400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe09006e360 con 0x7fe0900aeeb0 2024-03-24T01:26:35.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.146+0000 7fe098400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0900b4a00 con 0x7fe09006b600 2024-03-24T01:26:35.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.146+0000 7fe096400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe09015bc50 0x7fe09015a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:35.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.147+0000 7fe096400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe09015bc50 0x7fe09015a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59560/0 (socket says 172.21.15.90:59560) 2024-03-24T01:26:35.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.147+0000 7fe096400640 1 -- 172.21.15.90:0/3341543089 learned_addr learned my addr 172.21.15.90:0/3341543089 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:35.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.147+0000 7fe095000640 1 --2- 172.21.15.90:0/3341543089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0900aeeb0 0x7fe090159d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:35.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.147+0000 7fe095a00640 1 --2- 172.21.15.90:0/3341543089 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe09006b600 0x7fe090145350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:35.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.147+0000 7fe096400640 1 -- 172.21.15.90:0/3341543089 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe09006b600 msgr2=0x7fe090145350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:35.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.147+0000 7fe096400640 1 --2- 172.21.15.90:0/3341543089 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe09006b600 0x7fe090145350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.147+0000 7fe096400640 1 -- 172.21.15.90:0/3341543089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0900aeeb0 msgr2=0x7fe090159d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:35.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.147+0000 7fe096400640 1 --2- 172.21.15.90:0/3341543089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0900aeeb0 0x7fe090159d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.147+0000 7fe096400640 1 -- 172.21.15.90:0/3341543089 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe090154b90 con 0x7fe09015bc50 2024-03-24T01:26:35.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.147+0000 7fe096400640 1 --2- 172.21.15.90:0/3341543089 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe09015bc50 0x7fe09015a2d0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fe08800c6b0 tx=0x7fe08800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:35.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.147+0000 7fe08d600640 1 -- 172.21.15.90:0/3341543089 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe088003b70 con 0x7fe09015bc50 2024-03-24T01:26:35.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.148+0000 7fe08d600640 1 -- 172.21.15.90:0/3341543089 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe088003d30 con 0x7fe09015bc50 2024-03-24T01:26:35.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.148+0000 7fe08d600640 1 -- 172.21.15.90:0/3341543089 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe088005310 con 0x7fe09015bc50 2024-03-24T01:26:35.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.148+0000 7fe098400640 1 -- 172.21.15.90:0/3341543089 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe090154e70 con 0x7fe09015bc50 2024-03-24T01:26:35.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.148+0000 7fe098400640 1 -- 172.21.15.90:0/3341543089 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe090155430 con 0x7fe09015bc50 2024-03-24T01:26:35.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.150+0000 7fe08d600640 1 -- 172.21.15.90:0/3341543089 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe0880054b0 con 0x7fe09015bc50 2024-03-24T01:26:35.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.151+0000 7fe08d600640 1 --2- 172.21.15.90:0/3341543089 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe07006bbd0 0x7fe07006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:35.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.151+0000 7fe08d600640 1 -- 172.21.15.90:0/3341543089 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fe08808ac50 con 0x7fe09015bc50 2024-03-24T01:26:35.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.151+0000 7fe09b073740 1 Processor -- start 2024-03-24T01:26:35.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.151+0000 7fe09b073740 1 -- start start 2024-03-24T01:26:35.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.151+0000 7fe095a00640 1 --2- 172.21.15.90:0/3341543089 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe07006bbd0 0x7fe07006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:35.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.152+0000 7fe09b073740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e7c3298900 0x55e7c327a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:35.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.152+0000 7fe09b073740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7c3296420 0x55e7c327af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:35.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.152+0000 7fe09b073740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7c3278010 0x55e7c324e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:35.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.152+0000 7fe09b073740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7c31b4c90 con 0x55e7c3278010 2024-03-24T01:26:35.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.152+0000 7fe09b073740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7c315d980 con 0x55e7c3296420 2024-03-24T01:26:35.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.152+0000 7fe09b073740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e7c31b0d30 con 0x55e7c3298900 2024-03-24T01:26:35.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.152+0000 7fe096400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7c3278010 0x55e7c324e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:35.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.152+0000 7fe096400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7c3278010 0x55e7c324e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59564/0 (socket says 172.21.15.90:59564) 2024-03-24T01:26:35.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.152+0000 7fe096400640 1 -- 172.21.15.90:0/629243386 learned_addr learned my addr 172.21.15.90:0/629243386 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:35.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.153+0000 7fe095a00640 1 --2- 172.21.15.90:0/3341543089 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe07006bbd0 0x7fe07006e090 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe084009ad0 tx=0x7fe084008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:35.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.153+0000 7fe095000640 1 --2- 172.21.15.90:0/629243386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7c3296420 0x55e7c327af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:35.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.153+0000 7fe096400640 1 -- 172.21.15.90:0/629243386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e7c3298900 msgr2=0x55e7c327a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:35.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.153+0000 7fe095a00640 1 --2- 172.21.15.90:0/629243386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e7c3298900 0x55e7c327a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:35.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.153+0000 7fe096400640 1 --2- 172.21.15.90:0/629243386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e7c3298900 0x55e7c327a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.153+0000 7fe096400640 1 -- 172.21.15.90:0/629243386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7c3296420 msgr2=0x55e7c327af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:35.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.153+0000 7fe096400640 1 --2- 172.21.15.90:0/629243386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7c3296420 0x55e7c327af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.153+0000 7fe096400640 1 -- 172.21.15.90:0/629243386 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e7c3175510 con 0x55e7c3278010 2024-03-24T01:26:35.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.153+0000 7fe096400640 1 --2- 172.21.15.90:0/629243386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7c3278010 0x55e7c324e750 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fe088001440 tx=0x7fe088058790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70cdcf91ecb93dc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:35.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.153+0000 7fe082000640 1 -- 172.21.15.90:0/629243386 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe088086020 con 0x55e7c3278010 2024-03-24T01:26:35.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.153+0000 7fe082000640 1 -- 172.21.15.90:0/629243386 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0880614d0 con 0x55e7c3278010 2024-03-24T01:26:35.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.153+0000 7fe082000640 1 -- 172.21.15.90:0/629243386 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08806f070 con 0x55e7c3278010 2024-03-24T01:26:35.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.154+0000 7fe09b073740 1 -- 172.21.15.90:0/629243386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7c3278010 msgr2=0x55e7c324e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:35.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.154+0000 7fe09b073740 1 --2- 172.21.15.90:0/629243386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7c3278010 0x55e7c324e750 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fe088001440 tx=0x7fe088058790 comp rx=0 tx=0).stop 2024-03-24T01:26:35.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.154+0000 7fe09b073740 1 -- 172.21.15.90:0/629243386 shutdown_connections 2024-03-24T01:26:35.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.154+0000 7fe09b073740 1 --2- 172.21.15.90:0/629243386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7c3278010 0x55e7c324e750 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.154+0000 7fe09b073740 1 --2- 172.21.15.90:0/629243386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7c3296420 0x55e7c327af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.154+0000 7fe09b073740 1 --2- 172.21.15.90:0/629243386 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e7c3298900 0x55e7c327a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.154+0000 7fe09b073740 1 -- 172.21.15.90:0/629243386 >> 172.21.15.90:0/629243386 conn(0x55e7c324ce60 msgr2=0x55e7c32959e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:35.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.154+0000 7fe09b073740 1 -- 172.21.15.90:0/629243386 shutdown_connections 2024-03-24T01:26:35.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.154+0000 7fe09b073740 1 -- 172.21.15.90:0/629243386 wait complete. 2024-03-24T01:26:35.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.155+0000 7fe09b073740 1 -- start start 2024-03-24T01:26:35.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.155+0000 7fe09b073740 1 Processor -- start 2024-03-24T01:26:35.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.156+0000 7fe09b073740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e7c3296420 0x55e7c3271ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:35.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.156+0000 7fe09b073740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7c3298900 0x55e7c327db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:35.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.156+0000 7fe09b073740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7c3297ae0 0x55e7c3297f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:35.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.156+0000 7fe09b073740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7c324abf0 con 0x55e7c3297ae0 2024-03-24T01:26:35.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.156+0000 7fe09b073740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7c3170dd0 con 0x55e7c3298900 2024-03-24T01:26:35.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.156+0000 7fe09b073740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e7c3250a80 con 0x55e7c3296420 2024-03-24T01:26:35.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.156+0000 7fe09b073740 10 client.0 _ll_register_callbacks cb 0x55e7c2ec8ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T01:26:35.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.156+0000 7fe096400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7c3297ae0 0x55e7c3297f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:35.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.156+0000 7fe096400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7c3297ae0 0x55e7c3297f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59576/0 (socket says 172.21.15.90:59576) 2024-03-24T01:26:35.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.156+0000 7fe096400640 1 -- 172.21.15.90:0/2625289778 learned_addr learned my addr 172.21.15.90:0/2625289778 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:35.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.157+0000 7fe095000640 1 --2- 172.21.15.90:0/2625289778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7c3298900 0x55e7c327db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:35.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.157+0000 7fe096400640 1 -- 172.21.15.90:0/2625289778 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e7c3296420 msgr2=0x55e7c3271ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:35.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.157+0000 7fe096400640 1 --2- 172.21.15.90:0/2625289778 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e7c3296420 0x55e7c3271ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.157+0000 7fe096400640 1 -- 172.21.15.90:0/2625289778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7c3298900 msgr2=0x55e7c327db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:35.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.157+0000 7fe096400640 1 --2- 172.21.15.90:0/2625289778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7c3298900 0x55e7c327db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.157+0000 7fe096400640 1 -- 172.21.15.90:0/2625289778 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e7c2ec92d0 con 0x55e7c3297ae0 2024-03-24T01:26:35.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.157+0000 7fe096400640 1 --2- 172.21.15.90:0/2625289778 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7c3297ae0 0x55e7c3297f60 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fe08800c680 tx=0x7fe088001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af9d4ae4c0cd54e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:35.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.157+0000 7fe076a00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-03-24T01:26:35.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.157+0000 7fe076a00640 1 -- 172.21.15.90:0/2625289778 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe088086040 con 0x55e7c3297ae0 2024-03-24T01:26:35.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.157+0000 7fe076a00640 1 -- 172.21.15.90:0/2625289778 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fe088058180 con 0x55e7c3297ae0 2024-03-24T01:26:35.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.157+0000 7fe09b073740 1 -- 172.21.15.90:0/2625289778 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe0880614d0 con 0x55e7c3297ae0 2024-03-24T01:26:35.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.157+0000 7fe076a00640 1 -- 172.21.15.90:0/2625289778 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe0880583a0 con 0x55e7c3297ae0 2024-03-24T01:26:35.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.157+0000 7fe09b073740 10 client.5898 fetch_fsmap learned FSMap version 24 2024-03-24T01:26:35.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.157+0000 7fe09b073740 1 -- 172.21.15.90:0/2625289778 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x55e7c2f936b0 con 0x55e7c3297ae0 2024-03-24T01:26:35.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.158+0000 7fe076a00640 1 -- 172.21.15.90:0/2625289778 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0880614d0 con 0x55e7c3297ae0 2024-03-24T01:26:35.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.158+0000 7fe076a00640 1 -- 172.21.15.90:0/2625289778 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 24) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fe088054a80 con 0x55e7c3297ae0 2024-03-24T01:26:35.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.158+0000 7fe09b073740 10 client.5898 fetch_fsmap finished waiting for FSMap version 24 2024-03-24T01:26:35.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.158+0000 7fe09b073740 10 client.5898 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T01:26:35.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.158+0000 7fe09b073740 20 client.5898 populate_metadata read hostname 'smithi090' 2024-03-24T01:26:35.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.158+0000 7fe09b073740 1 --2- 172.21.15.90:0/2625289778 >> [v2:172.21.15.90:6834/11525832,v1:172.21.15.90:6835/11525832] conn(0x55e7c32aaee0 0x55e7c327ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:35.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.158+0000 7fe09b073740 4 client.5898 mds_command: new command op to 5787 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T01:26:35.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.158+0000 7fe09b073740 1 -- 172.21.15.90:0/2625289778 --> [v2:172.21.15.90:6834/11525832,v1:172.21.15.90:6835/11525832] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e7c3156520 con 0x55e7c32aaee0 2024-03-24T01:26:35.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.158+0000 7fe095a00640 1 --2- 172.21.15.90:0/2625289778 >> [v2:172.21.15.90:6834/11525832,v1:172.21.15.90:6835/11525832] conn(0x55e7c32aaee0 0x55e7c327ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:35.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.159+0000 7fe095a00640 1 --2- 172.21.15.90:0/2625289778 >> [v2:172.21.15.90:6834/11525832,v1:172.21.15.90:6835/11525832] conn(0x55e7c32aaee0 0x55e7c327ba20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a0515f23fb3dc5a server_cookie=96ad76d251f47673 in_seq=0 out_seq=0 2024-03-24T01:26:35.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.159+0000 7fe076a00640 10 client.5898 ms_handle_connect on v2:172.21.15.90:6834/11525832 2024-03-24T01:26:35.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.160+0000 7fe076a00640 1 -- 172.21.15.90:0/2625289778 <== mds.0 v2:172.21.15.90:6834/11525832 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e7c3156520 con 0x55e7c32aaee0 2024-03-24T01:26:35.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.160+0000 7fe076a00640 10 client.5898 handle_command_reply: tid=0 2024-03-24T01:26:35.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.160+0000 7fe09b073740 1 client.5898 shutdown 2024-03-24T01:26:35.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.160+0000 7fe09b073740 1 -- 172.21.15.90:0/2625289778 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7c3297ae0 msgr2=0x55e7c3297f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:35.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.160+0000 7fe09b073740 1 --2- 172.21.15.90:0/2625289778 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7c3297ae0 0x55e7c3297f60 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fe08800c680 tx=0x7fe088001250 comp rx=0 tx=0).stop 2024-03-24T01:26:35.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.160+0000 7fe09b073740 1 -- 172.21.15.90:0/2625289778 shutdown_connections 2024-03-24T01:26:35.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.160+0000 7fe09b073740 1 --2- 172.21.15.90:0/2625289778 >> [v2:172.21.15.90:6834/11525832,v1:172.21.15.90:6835/11525832] conn(0x55e7c32aaee0 0x55e7c327ba20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.160+0000 7fe09b073740 1 --2- 172.21.15.90:0/2625289778 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7c3297ae0 0x55e7c3297f60 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.160+0000 7fe09b073740 1 --2- 172.21.15.90:0/2625289778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7c3298900 0x55e7c327db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.160+0000 7fe09b073740 1 --2- 172.21.15.90:0/2625289778 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e7c3296420 0x55e7c3271ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.160+0000 7fe076a00640 0 client.5898 ms_handle_reset on v2:172.21.15.90:6834/11525832 2024-03-24T01:26:35.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.160+0000 7fe09b073740 1 -- 172.21.15.90:0/2625289778 >> 172.21.15.90:0/2625289778 conn(0x55e7c3278010 msgr2=0x55e7c324b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:35.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.160+0000 7fe09b073740 1 -- 172.21.15.90:0/2625289778 shutdown_connections 2024-03-24T01:26:35.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.160+0000 7fe09b073740 1 -- 172.21.15.90:0/2625289778 wait complete. 2024-03-24T01:26:35.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.160+0000 7fe09b073740 20 client.5898 trim_cache size 0 max 16384 2024-03-24T01:26:35.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.178+0000 7fe09b073740 1 Processor -- start 2024-03-24T01:26:35.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.178+0000 7fe09b073740 1 -- start start 2024-03-24T01:26:35.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.179+0000 7fe09b073740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e7c3296420 0x55e7c324bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:35.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.179+0000 7fe09b073740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7c3297ae0 0x55e7c327aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:35.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.179+0000 7fe09b073740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7c3298900 0x55e7c327cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:35.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.179+0000 7fe09b073740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0901fc520 con 0x55e7c3298900 2024-03-24T01:26:35.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.179+0000 7fe09b073740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0901fb640 con 0x55e7c3297ae0 2024-03-24T01:26:35.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.179+0000 7fe09b073740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0901fb040 con 0x55e7c3296420 2024-03-24T01:26:35.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.179+0000 7fe096400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7c3298900 0x55e7c327cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:35.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.179+0000 7fe096400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7c3298900 0x55e7c327cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59582/0 (socket says 172.21.15.90:59582) 2024-03-24T01:26:35.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.179+0000 7fe096400640 1 -- 172.21.15.90:0/2158143827 learned_addr learned my addr 172.21.15.90:0/2158143827 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:35.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.180+0000 7fe095000640 1 --2- 172.21.15.90:0/2158143827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7c3297ae0 0x55e7c327aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:35.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.180+0000 7fe095a00640 1 --2- 172.21.15.90:0/2158143827 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e7c3296420 0x55e7c324bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:35.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.180+0000 7fe096400640 1 -- 172.21.15.90:0/2158143827 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e7c3296420 msgr2=0x55e7c324bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:35.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.180+0000 7fe096400640 1 --2- 172.21.15.90:0/2158143827 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e7c3296420 0x55e7c324bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.180+0000 7fe096400640 1 -- 172.21.15.90:0/2158143827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7c3297ae0 msgr2=0x55e7c327aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:35.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.180+0000 7fe096400640 1 --2- 172.21.15.90:0/2158143827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7c3297ae0 0x55e7c327aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.180+0000 7fe096400640 1 -- 172.21.15.90:0/2158143827 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e7c324c5c0 con 0x55e7c3298900 2024-03-24T01:26:35.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.180+0000 7fe096400640 1 --2- 172.21.15.90:0/2158143827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7c3298900 0x55e7c327cb70 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fe08800cfd0 tx=0x7fe088075000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99e4ea9ca03c91c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:35.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.182+0000 7fe081600640 1 -- 172.21.15.90:0/2158143827 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe088086040 con 0x55e7c3298900 2024-03-24T01:26:35.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.182+0000 7fe081600640 1 -- 172.21.15.90:0/2158143827 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0880614d0 con 0x55e7c3298900 2024-03-24T01:26:35.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.182+0000 7fe081600640 1 -- 172.21.15.90:0/2158143827 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe088058b00 con 0x55e7c3298900 2024-03-24T01:26:35.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.183+0000 7fe09b073740 1 -- 172.21.15.90:0/2158143827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7c3298900 msgr2=0x55e7c327cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:35.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.183+0000 7fe09b073740 1 --2- 172.21.15.90:0/2158143827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7c3298900 0x55e7c327cb70 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fe08800cfd0 tx=0x7fe088075000 comp rx=0 tx=0).stop 2024-03-24T01:26:35.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.183+0000 7fe09b073740 1 -- 172.21.15.90:0/2158143827 shutdown_connections 2024-03-24T01:26:35.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.183+0000 7fe09b073740 1 --2- 172.21.15.90:0/2158143827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7c3298900 0x55e7c327cb70 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.183+0000 7fe09b073740 1 --2- 172.21.15.90:0/2158143827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7c3297ae0 0x55e7c327aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.183+0000 7fe09b073740 1 --2- 172.21.15.90:0/2158143827 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e7c3296420 0x55e7c324bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.183+0000 7fe09b073740 1 -- 172.21.15.90:0/2158143827 >> 172.21.15.90:0/2158143827 conn(0x55e7c3278010 msgr2=0x55e7c32719b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:35.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.183+0000 7fe09b073740 1 -- 172.21.15.90:0/2158143827 shutdown_connections 2024-03-24T01:26:35.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.183+0000 7fe09b073740 1 -- 172.21.15.90:0/2158143827 wait complete. 2024-03-24T01:26:35.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.184+0000 7fe09b073740 1 -- start start 2024-03-24T01:26:35.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.184+0000 7fe09b073740 1 Processor -- start 2024-03-24T01:26:35.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.185+0000 7fe09b073740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7c3296420 0x55e7c32a8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:35.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.185+0000 7fe09b073740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7c3297ae0 0x55e7c32f1800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:35.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.185+0000 7fe09b073740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e7c3298900 0x55e7c327ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:35.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.185+0000 7fe09b073740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7c324abf0 con 0x55e7c3296420 2024-03-24T01:26:35.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.185+0000 7fe09b073740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7c3250a80 con 0x55e7c3297ae0 2024-03-24T01:26:35.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.185+0000 7fe09b073740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e7c3170dd0 con 0x55e7c3298900 2024-03-24T01:26:35.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.185+0000 7fe09b073740 10 client.0 _ll_register_callbacks cb 0x7fe088058690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T01:26:35.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.185+0000 7fe095a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7c3296420 0x55e7c32a8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:35.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.185+0000 7fe095a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7c3296420 0x55e7c32a8ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59588/0 (socket says 172.21.15.90:59588) 2024-03-24T01:26:35.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.185+0000 7fe095a00640 1 -- 172.21.15.90:0/3171277336 learned_addr learned my addr 172.21.15.90:0/3171277336 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:35.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.186+0000 7fe095a00640 1 -- 172.21.15.90:0/3171277336 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e7c3298900 msgr2=0x55e7c327ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:35.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.186+0000 7fe095a00640 1 --2- 172.21.15.90:0/3171277336 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e7c3298900 0x55e7c327ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.186+0000 7fe095a00640 1 -- 172.21.15.90:0/3171277336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7c3297ae0 msgr2=0x55e7c32f1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:35.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.186+0000 7fe095a00640 1 --2- 172.21.15.90:0/3171277336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7c3297ae0 0x55e7c32f1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.186+0000 7fe095a00640 1 -- 172.21.15.90:0/3171277336 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e7c327c890 con 0x55e7c3296420 2024-03-24T01:26:35.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.186+0000 7fe095a00640 1 --2- 172.21.15.90:0/3171277336 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7c3296420 0x55e7c32a8ac0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x55e7c32aae60 tx=0x7fe084009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=498f70b22a22fb81 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:35.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.186+0000 7fe076a00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-03-24T01:26:35.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.186+0000 7fe076a00640 1 -- 172.21.15.90:0/3171277336 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe084011d30 con 0x55e7c3296420 2024-03-24T01:26:35.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.186+0000 7fe076a00640 1 -- 172.21.15.90:0/3171277336 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fe084022880 con 0x55e7c3296420 2024-03-24T01:26:35.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.186+0000 7fe09b073740 1 -- 172.21.15.90:0/3171277336 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe0880614d0 con 0x55e7c3296420 2024-03-24T01:26:35.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.186+0000 7fe076a00640 1 -- 172.21.15.90:0/3171277336 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe084036050 con 0x55e7c3296420 2024-03-24T01:26:35.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.186+0000 7fe09b073740 10 client.5904 fetch_fsmap learned FSMap version 24 2024-03-24T01:26:35.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.186+0000 7fe09b073740 1 -- 172.21.15.90:0/3171277336 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x55e7c32f2090 con 0x55e7c3296420 2024-03-24T01:26:35.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.187+0000 7fe076a00640 1 -- 172.21.15.90:0/3171277336 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0880614d0 con 0x55e7c3296420 2024-03-24T01:26:35.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.187+0000 7fe076a00640 1 -- 172.21.15.90:0/3171277336 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 24) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fe084039020 con 0x55e7c3296420 2024-03-24T01:26:35.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.187+0000 7fe09b073740 10 client.5904 fetch_fsmap finished waiting for FSMap version 24 2024-03-24T01:26:35.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.187+0000 7fe09b073740 10 client.5904 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T01:26:35.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.187+0000 7fe09b073740 20 client.5904 populate_metadata read hostname 'smithi090' 2024-03-24T01:26:35.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.187+0000 7fe09b073740 1 --2- 172.21.15.90:0/3171277336 >> [v2:172.21.15.90:6834/11525832,v1:172.21.15.90:6835/11525832] conn(0x55e7c327cfd0 0x55e7c324e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:35.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.187+0000 7fe09b073740 4 client.5904 mds_command: new command op to 5787 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T01:26:35.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.187+0000 7fe09b073740 1 -- 172.21.15.90:0/3171277336 --> [v2:172.21.15.90:6834/11525832,v1:172.21.15.90:6835/11525832] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e7c31c21d0 con 0x55e7c327cfd0 2024-03-24T01:26:35.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.187+0000 7fe095000640 1 --2- 172.21.15.90:0/3171277336 >> [v2:172.21.15.90:6834/11525832,v1:172.21.15.90:6835/11525832] conn(0x55e7c327cfd0 0x55e7c324e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:35.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.188+0000 7fe095000640 1 --2- 172.21.15.90:0/3171277336 >> [v2:172.21.15.90:6834/11525832,v1:172.21.15.90:6835/11525832] conn(0x55e7c327cfd0 0x55e7c324e750 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9277540966aa0d8 server_cookie=e9b5d3f89a0b3132 in_seq=0 out_seq=0 2024-03-24T01:26:35.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.188+0000 7fe076a00640 10 client.5904 ms_handle_connect on v2:172.21.15.90:6834/11525832 2024-03-24T01:26:35.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.188+0000 7fe076a00640 1 -- 172.21.15.90:0/3171277336 <== mds.0 v2:172.21.15.90:6834/11525832 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x55e7c31c21d0 con 0x55e7c327cfd0 2024-03-24T01:26:35.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.188+0000 7fe076a00640 10 client.5904 handle_command_reply: tid=0 2024-03-24T01:26:35.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.188+0000 7fe09b073740 1 client.5904 shutdown 2024-03-24T01:26:35.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.188+0000 7fe09b073740 1 -- 172.21.15.90:0/3171277336 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7c3296420 msgr2=0x55e7c32a8ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:35.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.188+0000 7fe09b073740 1 --2- 172.21.15.90:0/3171277336 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7c3296420 0x55e7c32a8ac0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x55e7c32aae60 tx=0x7fe084009190 comp rx=0 tx=0).stop 2024-03-24T01:26:35.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.188+0000 7fe09b073740 1 -- 172.21.15.90:0/3171277336 shutdown_connections 2024-03-24T01:26:35.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.188+0000 7fe09b073740 1 --2- 172.21.15.90:0/3171277336 >> [v2:172.21.15.90:6834/11525832,v1:172.21.15.90:6835/11525832] conn(0x55e7c327cfd0 0x55e7c324e750 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.188+0000 7fe09b073740 1 --2- 172.21.15.90:0/3171277336 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e7c3298900 0x55e7c327ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.188+0000 7fe09b073740 1 --2- 172.21.15.90:0/3171277336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7c3297ae0 0x55e7c32f1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.188+0000 7fe09b073740 1 --2- 172.21.15.90:0/3171277336 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7c3296420 0x55e7c32a8ac0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.188+0000 7fe09b073740 1 -- 172.21.15.90:0/3171277336 >> 172.21.15.90:0/3171277336 conn(0x55e7c3278010 msgr2=0x55e7c3270b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:35.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.188+0000 7fe076a00640 0 client.5904 ms_handle_reset on v2:172.21.15.90:6834/11525832 2024-03-24T01:26:35.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.188+0000 7fe09b073740 1 -- 172.21.15.90:0/3171277336 shutdown_connections 2024-03-24T01:26:35.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.188+0000 7fe09b073740 1 -- 172.21.15.90:0/3171277336 wait complete. 2024-03-24T01:26:35.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.188+0000 7fe09b073740 20 client.5904 trim_cache size 0 max 16384 2024-03-24T01:26:35.192 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-24T01:26:35.192 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "24712165-9839-4fcc-8033-988386efea74", 2024-03-24T01:26:35.192 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 0, 2024-03-24T01:26:35.192 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 5787, 2024-03-24T01:26:35.192 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-03-24T01:26:35.192 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-03-24T01:26:35.192 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-03-24T01:26:35.192 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 5.0403320169999999, 2024-03-24T01:26:35.192 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 24, 2024-03-24T01:26:35.192 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 40, 2024-03-24T01:26:35.192 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 40, 2024-03-24T01:26:35.192 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 7.426029422 2024-03-24T01:26:35.193 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-24T01:26:35.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.189+0000 7fe098400640 1 -- 172.21.15.90:0/3341543089 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe07006bbd0 msgr2=0x7fe07006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:35.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.189+0000 7fe098400640 1 --2- 172.21.15.90:0/3341543089 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe07006bbd0 0x7fe07006e090 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe084009ad0 tx=0x7fe084008040 comp rx=0 tx=0).stop 2024-03-24T01:26:35.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.189+0000 7fe098400640 1 -- 172.21.15.90:0/3341543089 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe09015bc50 msgr2=0x7fe09015a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:35.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.189+0000 7fe098400640 1 --2- 172.21.15.90:0/3341543089 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe09015bc50 0x7fe09015a2d0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fe08800c6b0 tx=0x7fe08800cb70 comp rx=0 tx=0).stop 2024-03-24T01:26:35.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.189+0000 7fe098400640 1 -- 172.21.15.90:0/3341543089 shutdown_connections 2024-03-24T01:26:35.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.189+0000 7fe098400640 1 --2- 172.21.15.90:0/3341543089 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe07006bbd0 0x7fe07006e090 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.189+0000 7fe098400640 1 --2- 172.21.15.90:0/3341543089 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe09015bc50 0x7fe09015a2d0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.189+0000 7fe098400640 1 --2- 172.21.15.90:0/3341543089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe0900aeeb0 0x7fe090159d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.189+0000 7fe098400640 1 --2- 172.21.15.90:0/3341543089 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe09006b600 0x7fe090145350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.189+0000 7fe098400640 1 -- 172.21.15.90:0/3341543089 >> 172.21.15.90:0/3341543089 conn(0x7fe0900b1770 msgr2=0x7fe0900af410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:35.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.189+0000 7fe098400640 1 -- 172.21.15.90:0/3341543089 shutdown_connections 2024-03-24T01:26:35.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.189+0000 7fe098400640 1 -- 172.21.15.90:0/3341543089 wait complete. 2024-03-24T01:26:35.207 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T01:26:35.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.389+0000 7f0f1b200640 1 Processor -- start 2024-03-24T01:26:35.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.390+0000 7f0f1b200640 1 -- start start 2024-03-24T01:26:35.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.391+0000 7f0f1b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f1415dc10 0x7f0f140aff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:35.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.391+0000 7f0f1b200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0f1415dff0 0x7f0f1406b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:35.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.392+0000 7f0f1b200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1406eb60 con 0x7f0f14161940 2024-03-24T01:26:35.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.392+0000 7f0f1b200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f1406ee40 con 0x7f0f1415dff0 2024-03-24T01:26:35.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.392+0000 7f0f1b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1406f120 con 0x7f0f1415dc10 2024-03-24T01:26:35.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.392+0000 7f0f13e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f1415dc10 0x7f0f140aff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:35.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.392+0000 7f0f0be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0f1415dff0 0x7f0f1406b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:35.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.392+0000 7f0f13e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f1415dc10 0x7f0f140aff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.90:57928/0 (socket says 172.21.15.90:57928) 2024-03-24T01:26:35.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.392+0000 7f0f13e00640 1 -- 172.21.15.90:0/3389299769 learned_addr learned my addr 172.21.15.90:0/3389299769 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:35.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.393+0000 7f0f13e00640 1 -- 172.21.15.90:0/3389299769 >> v2:172.21.15.120:3300/0 conn(0x7f0f1415dff0 msgr2=0x7f0f1406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:35.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.393+0000 7f0f13e00640 1 --2- 172.21.15.90:0/3389299769 >> v2:172.21.15.120:3300/0 conn(0x7f0f1415dff0 0x7f0f1406b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.393+0000 7f0f13e00640 1 -- 172.21.15.90:0/3389299769 >> v1:172.21.15.120:6789/0 conn(0x7f0f14161940 legacy=0x7f0f140702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:35.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.393+0000 7f0f13e00640 1 -- 172.21.15.90:0/3389299769 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1406f400 con 0x7f0f1415dc10 2024-03-24T01:26:35.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.394+0000 7f0f13e00640 1 --2- 172.21.15.90:0/3389299769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f1415dc10 0x7f0f140aff50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0f04009870 tx=0x7f0f0402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5436f40c76fe772 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:35.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.395+0000 7f0f13400640 1 -- 172.21.15.90:0/3389299769 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f040317c0 con 0x7f0f1415dc10 2024-03-24T01:26:35.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.395+0000 7f0f13400640 1 -- 172.21.15.90:0/3389299769 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f04037070 con 0x7f0f1415dc10 2024-03-24T01:26:35.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.395+0000 7f0f13400640 1 -- 172.21.15.90:0/3389299769 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f04031b80 con 0x7f0f1415dc10 2024-03-24T01:26:35.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.395+0000 7f0f1b200640 1 -- 172.21.15.90:0/3389299769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f1415dc10 msgr2=0x7f0f140aff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:35.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.395+0000 7f0f1b200640 1 --2- 172.21.15.90:0/3389299769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f1415dc10 0x7f0f140aff50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0f04009870 tx=0x7f0f0402b860 comp rx=0 tx=0).stop 2024-03-24T01:26:35.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.395+0000 7f0f1b200640 1 -- 172.21.15.90:0/3389299769 shutdown_connections 2024-03-24T01:26:35.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.395+0000 7f0f1b200640 1 --2- 172.21.15.90:0/3389299769 >> v2:172.21.15.120:3300/0 conn(0x7f0f1415dff0 0x7f0f1406b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.395+0000 7f0f1b200640 1 --2- 172.21.15.90:0/3389299769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f1415dc10 0x7f0f140aff50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.395+0000 7f0f1b200640 1 -- 172.21.15.90:0/3389299769 >> 172.21.15.90:0/3389299769 conn(0x7f0f1406c1d0 msgr2=0x7f0f1406c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:35.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.395+0000 7f0f1b200640 1 -- 172.21.15.90:0/3389299769 shutdown_connections 2024-03-24T01:26:35.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.396+0000 7f0f1b200640 1 -- 172.21.15.90:0/3389299769 wait complete. 2024-03-24T01:26:35.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.397+0000 7f0f1b200640 1 Processor -- start 2024-03-24T01:26:35.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.398+0000 7f0f1b200640 1 -- start start 2024-03-24T01:26:35.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.399+0000 7f0f1b200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f1415dc10 0x7f0f14150460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:35.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.399+0000 7f0f1b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f14161940 0x7f0f140b2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:35.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.399+0000 7f0f1b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f140b3160 0x7f0f140cfdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:35.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.399+0000 7f0f1b200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f140b4590 con 0x7f0f140b3160 2024-03-24T01:26:35.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.399+0000 7f0f1b200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f140b4410 con 0x7f0f1415dc10 2024-03-24T01:26:35.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.399+0000 7f0f1b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f140aff30 con 0x7f0f14161940 2024-03-24T01:26:35.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.399+0000 7f0f19200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f140b3160 0x7f0f140cfdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:35.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.399+0000 7f0f19200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f140b3160 0x7f0f140cfdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59596/0 (socket says 172.21.15.90:59596) 2024-03-24T01:26:35.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.399+0000 7f0f19200640 1 -- 172.21.15.90:0/4087805621 learned_addr learned my addr 172.21.15.90:0/4087805621 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:35.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.399+0000 7f0f13e00640 1 --2- 172.21.15.90:0/4087805621 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f1415dc10 0x7f0f14150460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:35.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.399+0000 7f0f0be00640 1 --2- 172.21.15.90:0/4087805621 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f14161940 0x7f0f140b2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:35.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.400+0000 7f0f19200640 1 -- 172.21.15.90:0/4087805621 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f14161940 msgr2=0x7f0f140b2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:35.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.400+0000 7f0f19200640 1 --2- 172.21.15.90:0/4087805621 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f14161940 0x7f0f140b2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.400+0000 7f0f19200640 1 -- 172.21.15.90:0/4087805621 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f1415dc10 msgr2=0x7f0f14150460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:35.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.400+0000 7f0f19200640 1 --2- 172.21.15.90:0/4087805621 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f1415dc10 0x7f0f14150460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.400+0000 7f0f19200640 1 -- 172.21.15.90:0/4087805621 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f141509a0 con 0x7f0f140b3160 2024-03-24T01:26:35.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.400+0000 7f0f19200640 1 --2- 172.21.15.90:0/4087805621 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f140b3160 0x7f0f140cfdf0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c00c680 tx=0x7f0f0c00cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:35.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.400+0000 7f0f10c00640 1 -- 172.21.15.90:0/4087805621 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c003b70 con 0x7f0f140b3160 2024-03-24T01:26:35.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.400+0000 7f0f10c00640 1 -- 172.21.15.90:0/4087805621 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f0c003d10 con 0x7f0f140b3160 2024-03-24T01:26:35.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.401+0000 7f0f10c00640 1 -- 172.21.15.90:0/4087805621 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c005310 con 0x7f0f140b3160 2024-03-24T01:26:35.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.401+0000 7f0f1b200640 1 -- 172.21.15.90:0/4087805621 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f14153a20 con 0x7f0f140b3160 2024-03-24T01:26:35.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.401+0000 7f0f1b200640 1 -- 172.21.15.90:0/4087805621 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f14153fe0 con 0x7f0f140b3160 2024-03-24T01:26:35.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.402+0000 7f0f1b200640 1 -- 172.21.15.90:0/4087805621 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f141541e0 con 0x7f0f140b3160 2024-03-24T01:26:35.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.403+0000 7f0f10c00640 1 -- 172.21.15.90:0/4087805621 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0f0c0054b0 con 0x7f0f140b3160 2024-03-24T01:26:35.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.404+0000 7f0f10c00640 1 --2- 172.21.15.90:0/4087805621 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0ef006bbd0 0x7f0ef006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:35.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.404+0000 7f0f10c00640 1 -- 172.21.15.90:0/4087805621 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f0f0c08aa00 con 0x7f0f140b3160 2024-03-24T01:26:35.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.405+0000 7f0f13e00640 1 --2- 172.21.15.90:0/4087805621 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0ef006bbd0 0x7f0ef006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:35.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.406+0000 7f0f13e00640 1 --2- 172.21.15.90:0/4087805621 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0ef006bbd0 0x7f0ef006e090 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0f04036000 tx=0x7f0f0403d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:35.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.409+0000 7f0f10c00640 1 -- 172.21.15.90:0/4087805621 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0f0c059310 con 0x7f0f140b3160 2024-03-24T01:26:35.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.646+0000 7f0f1b200640 1 -- 172.21.15.90:0/4087805621 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0f140b4710 con 0x7f0f140b3160 2024-03-24T01:26:35.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.647+0000 7f0f10c00640 1 -- 172.21.15.90:0/4087805621 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) v1 ==== 53+0+73 (secure 0 0 0) 0x7f0f0c058c10 con 0x7f0f140b3160 2024-03-24T01:26:35.648 INFO:teuthology.orchestra.run.smithi090.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T01:26:35.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.650+0000 7f0f1b200640 1 -- 172.21.15.90:0/4087805621 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0ef006bbd0 msgr2=0x7f0ef006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:35.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.650+0000 7f0f1b200640 1 --2- 172.21.15.90:0/4087805621 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0ef006bbd0 0x7f0ef006e090 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0f04036000 tx=0x7f0f0403d040 comp rx=0 tx=0).stop 2024-03-24T01:26:35.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.650+0000 7f0f1b200640 1 -- 172.21.15.90:0/4087805621 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f140b3160 msgr2=0x7f0f140cfdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:35.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.650+0000 7f0f1b200640 1 --2- 172.21.15.90:0/4087805621 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f140b3160 0x7f0f140cfdf0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c00c680 tx=0x7f0f0c00cb40 comp rx=0 tx=0).stop 2024-03-24T01:26:35.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.650+0000 7f0f1b200640 1 -- 172.21.15.90:0/4087805621 shutdown_connections 2024-03-24T01:26:35.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.650+0000 7f0f1b200640 1 --2- 172.21.15.90:0/4087805621 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0ef006bbd0 0x7f0ef006e090 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.650+0000 7f0f1b200640 1 --2- 172.21.15.90:0/4087805621 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f140b3160 0x7f0f140cfdf0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.650+0000 7f0f1b200640 1 --2- 172.21.15.90:0/4087805621 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f14161940 0x7f0f140b2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.650+0000 7f0f1b200640 1 --2- 172.21.15.90:0/4087805621 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f1415dc10 0x7f0f14150460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:35.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.650+0000 7f0f1b200640 1 -- 172.21.15.90:0/4087805621 >> 172.21.15.90:0/4087805621 conn(0x7f0f1406c1d0 msgr2=0x7f0f140739c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:35.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.650+0000 7f0f1b200640 1 -- 172.21.15.90:0/4087805621 shutdown_connections 2024-03-24T01:26:35.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:35.650+0000 7f0f1b200640 1 -- 172.21.15.90:0/4087805621 wait complete. 2024-03-24T01:26:35.667 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T01:26:35.667 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T01:26:35.667 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi090.front.sepia.ceph.com 2024-03-24T01:26:35.667 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T01:26:35.667 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T01:26:35.668 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T01:26:35.668 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T01:26:35.668 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T01:26:35.668 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T01:26:35.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:35.668 DEBUG:teuthology.orchestra.run.smithi090:> ip addr 2024-03-24T01:26:35.717 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T01:26:35.718 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T01:26:35.718 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T01:26:35.718 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:26:35.718 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-03-24T01:26:35.718 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:26:35.718 INFO:teuthology.orchestra.run.smithi090.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T01:26:35.718 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a8 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:26:35.718 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f0 2024-03-24T01:26:35.718 INFO:teuthology.orchestra.run.smithi090.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T01:26:35.718 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a9 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:26:35.718 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f1 2024-03-24T01:26:35.718 INFO:teuthology.orchestra.run.smithi090.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T01:26:35.718 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:66 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:26:35.719 INFO:teuthology.orchestra.run.smithi090.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T01:26:35.719 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:67 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:26:35.719 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.21.15.90/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T01:26:35.719 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft 42386sec preferred_lft 42386sec 2024-03-24T01:26:35.719 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::ec4:7aff:fe8f:d767/64 scope link 2024-03-24T01:26:35.719 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:26:35.719 INFO:teuthology.orchestra.run.smithi090.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T01:26:35.719 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 06:50:1d:bd:ed:70 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:26:35.719 INFO:teuthology.orchestra.run.smithi090.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T01:26:35.719 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:26:35.719 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::a05b:d3ff:fea5:6fd0/64 scope link 2024-03-24T01:26:35.719 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:26:35.719 INFO:teuthology.orchestra.run.smithi090.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T01:26:35.719 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether de:94:0b:7e:c6:41 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:26:35.720 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::dc94:bff:fe7e:c641/64 scope link 2024-03-24T01:26:35.720 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:26:35.720 INFO:teuthology.orchestra.run.smithi090.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T01:26:35.720 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 06:50:1d:bd:ed:70 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:26:35.720 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::450:1dff:febd:ed70/64 scope link 2024-03-24T01:26:35.720 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:26:35.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:35.720 DEBUG:teuthology.orchestra.run.smithi090:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T01:26:35.846 INFO:teuthology.orchestra.run.smithi090.stdout:1 2024-03-24T01:26:35.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:35.850 DEBUG:teuthology.orchestra.run.smithi090:> route 2024-03-24T01:26:35.901 INFO:teuthology.orchestra.run.smithi090.stdout:Kernel IP routing table 2024-03-24T01:26:35.901 INFO:teuthology.orchestra.run.smithi090.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T01:26:35.901 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T01:26:35.902 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T01:26:35.902 INFO:teuthology.orchestra.run.smithi090.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T01:26:35.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:35.902 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2024-03-24T01:26:35.902 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-24T01:26:35.902 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T01:26:35.902 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T01:26:35.902 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T01:26:35.902 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-24T01:26:36.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:36.079 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-03-24T01:26:36.123 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T01:26:36.123 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T01:26:36.124 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T01:26:36.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:36.124 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set brx.0 up 2024-03-24T01:26:36.251 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-24T01:26:36.251 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T01:26:36.251 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T01:26:36.300 INFO:teuthology.orchestra.run.smithi090.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T01:26:36.301 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T01:26:36.301 DEBUG:teuthology.orchestra.run.smithi090:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T01:26:36.391 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-24T01:26:36.473 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T01:26:36.526 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-24T01:26:36.526 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-24T01:26:36.526 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-24T01:26:36.526 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-24T01:26:36.526 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-24T01:26:36.526 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-24T01:26:36.526 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-24T01:26:36.527 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-24T01:26:36.527 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-24T01:26:36.527 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-24T01:26:36.527 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-24T01:26:36.527 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/selinux 2024-03-24T01:26:36.527 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T01:26:36.527 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-24T01:26:36.527 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-24T01:26:36.527 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-24T01:26:36.527 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-24T01:26:36.527 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T01:26:36.527 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-24T01:26:36.527 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T01:26:36.527 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-24T01:26:36.527 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T01:26:36.527 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T01:26:36.528 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-24T01:26:36.528 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-24T01:26:36.528 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T01:26:36.528 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T01:26:36.528 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T01:26:36.528 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T01:26:36.528 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-24T01:26:36.528 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:26:36.528 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:26:36.528 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:26:36.528 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:26:36.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:36.529 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-24T01:26:36.576 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-24T01:26:36.577 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-24T01:26:36.626 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-24T01:26:36.702 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T01:26:36.744 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-24T01:26:36.744 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-24T01:26:36.745 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-24T01:26:36.745 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-24T01:26:36.745 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-24T01:26:36.745 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-24T01:26:36.745 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-24T01:26:36.745 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-24T01:26:36.745 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-24T01:26:36.745 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-24T01:26:36.745 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-24T01:26:36.745 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/selinux 2024-03-24T01:26:36.745 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T01:26:36.745 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-24T01:26:36.745 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-24T01:26:36.745 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-24T01:26:36.745 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-24T01:26:36.746 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T01:26:36.746 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-24T01:26:36.746 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T01:26:36.746 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-24T01:26:36.746 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T01:26:36.746 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T01:26:36.746 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-24T01:26:36.746 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-24T01:26:36.746 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T01:26:36.746 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T01:26:36.746 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T01:26:36.746 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T01:26:36.746 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-24T01:26:36.746 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:26:36.747 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:26:36.747 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:26:36.747 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:26:36.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:36.748 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-24T01:26:36.812 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:2024-03-24T01:26:36.812+0000 7f60940d3480 -1 init, newargv = 0x561a3596d990 newargc=15 2024-03-24T01:26:36.813 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[55523]: starting ceph client 2024-03-24T01:26:36.823 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[55523]: starting fuse 2024-03-24T01:26:36.834 INFO:teuthology.orchestra.run.smithi090.stdout:40 2024-03-24T01:26:36.834 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-03-24T01:26:36.835 DEBUG:teuthology.orchestra.run.smithi090:> sudo stdin-killer -- python3 -c ' 2024-03-24T01:26:36.835 DEBUG:teuthology.orchestra.run.smithi090:> import glob 2024-03-24T01:26:36.835 DEBUG:teuthology.orchestra.run.smithi090:> import re 2024-03-24T01:26:36.835 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-24T01:26:36.835 DEBUG:teuthology.orchestra.run.smithi090:> import subprocess 2024-03-24T01:26:36.835 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:26:36.835 DEBUG:teuthology.orchestra.run.smithi090:> def _find_admin_socket(client_name): 2024-03-24T01:26:36.835 DEBUG:teuthology.orchestra.run.smithi090:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-24T01:26:36.835 DEBUG:teuthology.orchestra.run.smithi090:> files = glob.glob(asok_path) 2024-03-24T01:26:36.835 DEBUG:teuthology.orchestra.run.smithi090:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-24T01:26:36.835 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:26:36.835 DEBUG:teuthology.orchestra.run.smithi090:> # Given a non-glob path, it better be there 2024-03-24T01:26:36.835 DEBUG:teuthology.orchestra.run.smithi090:> if "*" not in asok_path: 2024-03-24T01:26:36.836 DEBUG:teuthology.orchestra.run.smithi090:> assert(len(files) == 1) 2024-03-24T01:26:36.836 DEBUG:teuthology.orchestra.run.smithi090:> return files[0] 2024-03-24T01:26:36.836 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:26:36.836 DEBUG:teuthology.orchestra.run.smithi090:> for f in files: 2024-03-24T01:26:36.836 DEBUG:teuthology.orchestra.run.smithi090:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T01:26:36.836 DEBUG:teuthology.orchestra.run.smithi090:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T01:26:36.836 DEBUG:teuthology.orchestra.run.smithi090:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T01:26:36.836 DEBUG:teuthology.orchestra.run.smithi090:> contents = proc_f.read() 2024-03-24T01:26:36.836 DEBUG:teuthology.orchestra.run.smithi090:> if mountpoint in contents: 2024-03-24T01:26:36.836 DEBUG:teuthology.orchestra.run.smithi090:> return f 2024-03-24T01:26:36.836 DEBUG:teuthology.orchestra.run.smithi090:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T01:26:36.836 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:26:36.836 DEBUG:teuthology.orchestra.run.smithi090:> print(_find_admin_socket("client.0")) 2024-03-24T01:26:36.836 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-24T01:26:37.025 INFO:teuthology.orchestra.run.smithi090.stdout:/var/run/ceph/ceph-client.0.55523.asok 2024-03-24T01:26:37.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T01:26:37.043 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.55523.asok 2024-03-24T01:26:37.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:37.043 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.55523.asok status 2024-03-24T01:26:37.282 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-24T01:26:37.282 INFO:teuthology.orchestra.run.smithi090.stdout: "metadata": { 2024-03-24T01:26:37.282 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_sha1": "3be9a5c9bc793e11e2800a8c0c696e8b46742033", 2024-03-24T01:26:37.282 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-24T01:26:37.282 INFO:teuthology.orchestra.run.smithi090.stdout: "entity_id": "0", 2024-03-24T01:26:37.282 INFO:teuthology.orchestra.run.smithi090.stdout: "hostname": "smithi090", 2024-03-24T01:26:37.282 INFO:teuthology.orchestra.run.smithi090.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-24T01:26:37.283 INFO:teuthology.orchestra.run.smithi090.stdout: "pid": "55523", 2024-03-24T01:26:37.283 INFO:teuthology.orchestra.run.smithi090.stdout: "root": "/" 2024-03-24T01:26:37.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:26:37.283 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_count": 0, 2024-03-24T01:26:37.283 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_pinned_count": 0, 2024-03-24T01:26:37.283 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 5913, 2024-03-24T01:26:37.283 INFO:teuthology.orchestra.run.smithi090.stdout: "inst": { 2024-03-24T01:26:37.283 INFO:teuthology.orchestra.run.smithi090.stdout: "name": { 2024-03-24T01:26:37.283 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "client", 2024-03-24T01:26:37.283 INFO:teuthology.orchestra.run.smithi090.stdout: "num": 5913 2024-03-24T01:26:37.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:26:37.283 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-24T01:26:37.283 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-24T01:26:37.283 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-03-24T01:26:37.283 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 4209630582 2024-03-24T01:26:37.284 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:26:37.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:26:37.284 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-24T01:26:37.284 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-24T01:26:37.284 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-03-24T01:26:37.284 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 4209630582 2024-03-24T01:26:37.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:26:37.284 INFO:teuthology.orchestra.run.smithi090.stdout: "inst_str": "client.5913 192.168.0.1:0/4209630582", 2024-03-24T01:26:37.284 INFO:teuthology.orchestra.run.smithi090.stdout: "addr_str": "192.168.0.1:0/4209630582", 2024-03-24T01:26:37.284 INFO:teuthology.orchestra.run.smithi090.stdout: "inode_count": 1, 2024-03-24T01:26:37.284 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_epoch": 24, 2024-03-24T01:26:37.284 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch": 40, 2024-03-24T01:26:37.284 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch_barrier": 0, 2024-03-24T01:26:37.284 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklisted": false, 2024-03-24T01:26:37.284 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs" 2024-03-24T01:26:37.285 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-24T01:26:37.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:37.296 DEBUG:teuthology.orchestra.run.smithi090:> stat --file-system '--printf=%T 2024-03-24T01:26:37.296 DEBUG:teuthology.orchestra.run.smithi090:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-24T01:26:37.345 INFO:teuthology.orchestra.run.smithi090.stdout:fuseblk 2024-03-24T01:26:37.345 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-24T01:26:37.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:37.345 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T01:26:37.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:37.425 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -R) 2024-03-24T01:26:37.483 INFO:tasks.ceph.mds.c.smithi090.stderr:2024-03-24T01:26:37.483+0000 7f40cac00640 -1 log_channel(cluster) log [ERR] : bad backtrace on directory inode 0x100000001f7 2024-03-24T01:26:37.491 INFO:tasks.ceph.mds.c.smithi090.stderr:2024-03-24T01:26:37.491+0000 7f40cac00640 -1 log_channel(cluster) log [ERR] : bad backtrace on directory inode 0x10000000001 2024-03-24T01:26:37.495 INFO:teuthology.orchestra.run.smithi090.stdout:.: 2024-03-24T01:26:37.495 INFO:teuthology.orchestra.run.smithi090.stdout:linkdir 2024-03-24T01:26:37.495 INFO:teuthology.orchestra.run.smithi090.stdout:rootfile 2024-03-24T01:26:37.495 INFO:teuthology.orchestra.run.smithi090.stdout:subdir 2024-03-24T01:26:37.495 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:26:37.495 INFO:teuthology.orchestra.run.smithi090.stdout:./linkdir: 2024-03-24T01:26:37.495 INFO:teuthology.orchestra.run.smithi090.stdout:link2 2024-03-24T01:26:37.495 INFO:teuthology.orchestra.run.smithi090.stdout:link3 2024-03-24T01:26:37.495 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:26:37.495 INFO:teuthology.orchestra.run.smithi090.stdout:./subdir: 2024-03-24T01:26:37.495 INFO:teuthology.orchestra.run.smithi090.stdout:subdirfile 2024-03-24T01:26:37.496 INFO:teuthology.orchestra.run.smithi090.stdout:subsubdir 2024-03-24T01:26:37.496 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:26:37.496 INFO:teuthology.orchestra.run.smithi090.stdout:./subdir/subsubdir: 2024-03-24T01:26:37.497 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:26:37.497 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:26:37.545 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:26:37.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.793+0000 7f3901400640 1 Processor -- start 2024-03-24T01:26:37.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.793+0000 7f3901400640 1 -- start start 2024-03-24T01:26:37.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.795+0000 7f3901400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38fc06b0c0 0x7f38fc06b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:37.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.795+0000 7f3901400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38fc15ef40 con 0x7f38fc147880 2024-03-24T01:26:37.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.795+0000 7f3901400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38fc0b3de0 con 0x7f38fc15f9e0 2024-03-24T01:26:37.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.795+0000 7f3901400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38fc06ba40 con 0x7f38fc06b0c0 2024-03-24T01:26:37.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.795+0000 7f38fa800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f38fc147880 0x7f38fc0b57e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:34580/0 (socket says 172.21.15.90:34580) 2024-03-24T01:26:37.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.795+0000 7f38fa800640 1 -- 172.21.15.90:0/758250574 learned_addr learned my addr 172.21.15.90:0/758250574 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:26:37.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.795+0000 7f38f3e00640 1 --2- 172.21.15.90:0/758250574 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38fc06b0c0 0x7f38fc06b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:37.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.796+0000 7f38f9e00640 1 -- 172.21.15.90:0/758250574 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3456406705 0 0) 0x7f38fc15ef40 con 0x7f38fc147880 2024-03-24T01:26:37.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.796+0000 7f38f9e00640 1 -- 172.21.15.90:0/758250574 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38e0003520 con 0x7f38fc147880 2024-03-24T01:26:37.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.796+0000 7f38f9e00640 1 -- 172.21.15.90:0/758250574 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 235348404 0 0) 0x7f38e0003520 con 0x7f38fc147880 2024-03-24T01:26:37.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.796+0000 7f38f9e00640 1 -- 172.21.15.90:0/758250574 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38fc06b0c0 msgr2=0x7f38fc06b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:37.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.796+0000 7f38f9e00640 1 --2- 172.21.15.90:0/758250574 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38fc06b0c0 0x7f38fc06b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:37.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.796+0000 7f38f9e00640 1 -- 172.21.15.90:0/758250574 >> v1:172.21.15.120:6789/0 conn(0x7f38fc15f9e0 legacy=0x7f38fc15fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:37.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.796+0000 7f38f9e00640 1 -- 172.21.15.90:0/758250574 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38fc0b40c0 con 0x7f38fc147880 2024-03-24T01:26:37.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.796+0000 7f38f9e00640 1 -- 172.21.15.90:0/758250574 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f38ec001670 con 0x7f38fc147880 2024-03-24T01:26:37.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.796+0000 7f38f9e00640 1 -- 172.21.15.90:0/758250574 >> v1:172.21.15.90:6789/0 conn(0x7f38fc147880 legacy=0x7f38fc0b57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:37.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.796+0000 7f38f9e00640 1 --2- 172.21.15.90:0/758250574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38e00039e0 0x7f38e0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:37.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.796+0000 7f38f9e00640 1 --2- 172.21.15.90:0/758250574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38e0004800 0x7f38e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:37.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.796+0000 7f38f9e00640 1 --2- 172.21.15.90:0/758250574 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38fc06b0c0 0x7f38e0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:37.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.796+0000 7f38f9e00640 1 -- 172.21.15.90:0/758250574 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38fc06ba40 con 0x7f38e0004800 2024-03-24T01:26:37.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.796+0000 7f38f9e00640 1 -- 172.21.15.90:0/758250574 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38e000bbb0 con 0x7f38e00039e0 2024-03-24T01:26:37.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.796+0000 7f38f9e00640 1 -- 172.21.15.90:0/758250574 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38e000bdb0 con 0x7f38fc06b0c0 2024-03-24T01:26:37.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.797+0000 7f38f3e00640 1 --2- 172.21.15.90:0/758250574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38e0004800 0x7f38e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:37.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.797+0000 7f38fa800640 1 --2- 172.21.15.90:0/758250574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38e00039e0 0x7f38e0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:37.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.797+0000 7f38f3e00640 1 -- 172.21.15.90:0/758250574 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38fc06b0c0 msgr2=0x7f38e0005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:37.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.797+0000 7f38fb200640 1 --2- 172.21.15.90:0/758250574 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38fc06b0c0 0x7f38e0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:37.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.797+0000 7f38f3e00640 1 --2- 172.21.15.90:0/758250574 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38fc06b0c0 0x7f38e0005200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:37.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.797+0000 7f38f3e00640 1 -- 172.21.15.90:0/758250574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38e00039e0 msgr2=0x7f38e0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:37.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.797+0000 7f38f3e00640 1 --2- 172.21.15.90:0/758250574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38e00039e0 0x7f38e0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:37.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.797+0000 7f38f3e00640 1 -- 172.21.15.90:0/758250574 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38e0003520 con 0x7f38e0004800 2024-03-24T01:26:37.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.797+0000 7f38fa800640 1 --2- 172.21.15.90:0/758250574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38e00039e0 0x7f38e0003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:37.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.798+0000 7f38f3e00640 1 --2- 172.21.15.90:0/758250574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38e0004800 0x7f38e0004c80 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f38e400ba30 tx=0x7f38e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fcf1a618c5fe3bd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:37.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.798+0000 7f38f9e00640 1 -- 172.21.15.90:0/758250574 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e4036020 con 0x7f38e0004800 2024-03-24T01:26:37.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.798+0000 7f38f9e00640 1 -- 172.21.15.90:0/758250574 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38e4032640 con 0x7f38e0004800 2024-03-24T01:26:37.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.798+0000 7f38f9e00640 1 -- 172.21.15.90:0/758250574 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e400b600 con 0x7f38e0004800 2024-03-24T01:26:37.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.799+0000 7f3901400640 1 -- 172.21.15.90:0/758250574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38e0004800 msgr2=0x7f38e0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:37.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.799+0000 7f3901400640 1 --2- 172.21.15.90:0/758250574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38e0004800 0x7f38e0004c80 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f38e400ba30 tx=0x7f38e0005830 comp rx=0 tx=0).stop 2024-03-24T01:26:37.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.799+0000 7f38fb200640 1 -- 172.21.15.90:0/758250574 reap_dead start 2024-03-24T01:26:37.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.799+0000 7f3901400640 1 -- 172.21.15.90:0/758250574 shutdown_connections 2024-03-24T01:26:37.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.799+0000 7f3901400640 1 -- 172.21.15.90:0/758250574 >> 172.21.15.90:0/758250574 conn(0x7f38fc073170 msgr2=0x7f38fc099a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:37.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.799+0000 7f3901400640 1 -- 172.21.15.90:0/758250574 shutdown_connections 2024-03-24T01:26:37.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.799+0000 7f3901400640 1 -- 172.21.15.90:0/758250574 wait complete. 2024-03-24T01:26:37.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.801+0000 7f3901400640 1 Processor -- start 2024-03-24T01:26:37.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.801+0000 7f3901400640 1 -- start start 2024-03-24T01:26:37.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.802+0000 7f3901400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38e0004800 0x7f38fc13e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:37.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.802+0000 7f3901400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38fc13ee10 0x7f38fc13f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:37.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.802+0000 7f3901400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38fc140480 0x7f38fc140900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:37.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.802+0000 7f3901400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38fc15f4c0 con 0x7f38fc140480 2024-03-24T01:26:37.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.802+0000 7f3901400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38fc0b5ef0 con 0x7f38e0004800 2024-03-24T01:26:37.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.802+0000 7f3901400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38fc0b3a00 con 0x7f38fc13ee10 2024-03-24T01:26:37.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.802+0000 7f38fb200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38fc140480 0x7f38fc140900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:37.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.802+0000 7f38fb200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38fc140480 0x7f38fc140900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59618/0 (socket says 172.21.15.90:59618) 2024-03-24T01:26:37.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.802+0000 7f38fb200640 1 -- 172.21.15.90:0/3159788085 learned_addr learned my addr 172.21.15.90:0/3159788085 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:37.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.803+0000 7f38fa800640 1 --2- 172.21.15.90:0/3159788085 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38e0004800 0x7f38fc13e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:37.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.803+0000 7f38f3e00640 1 --2- 172.21.15.90:0/3159788085 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38fc13ee10 0x7f38fc13f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:37.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.803+0000 7f38fb200640 1 -- 172.21.15.90:0/3159788085 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38fc13ee10 msgr2=0x7f38fc13f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:37.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.803+0000 7f38fb200640 1 --2- 172.21.15.90:0/3159788085 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38fc13ee10 0x7f38fc13f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:37.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.803+0000 7f38fb200640 1 -- 172.21.15.90:0/3159788085 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38e0004800 msgr2=0x7f38fc13e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:37.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.803+0000 7f38fb200640 1 --2- 172.21.15.90:0/3159788085 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38e0004800 0x7f38fc13e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:37.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.803+0000 7f38fb200640 1 -- 172.21.15.90:0/3159788085 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38fc0b0490 con 0x7f38fc140480 2024-03-24T01:26:37.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.803+0000 7f38fb200640 1 --2- 172.21.15.90:0/3159788085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38fc140480 0x7f38fc140900 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f38fc15fdc0 tx=0x7f38f400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:37.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.803+0000 7f38f2a00640 1 -- 172.21.15.90:0/3159788085 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f4018070 con 0x7f38fc140480 2024-03-24T01:26:37.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.804+0000 7f38f2a00640 1 -- 172.21.15.90:0/3159788085 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38f4003b70 con 0x7f38fc140480 2024-03-24T01:26:37.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.804+0000 7f3901400640 1 -- 172.21.15.90:0/3159788085 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38fc13f8d0 con 0x7f38fc140480 2024-03-24T01:26:37.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.804+0000 7f38f2a00640 1 -- 172.21.15.90:0/3159788085 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f4005070 con 0x7f38fc140480 2024-03-24T01:26:37.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.804+0000 7f3901400640 1 -- 172.21.15.90:0/3159788085 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38fc0b0730 con 0x7f38fc140480 2024-03-24T01:26:37.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.805+0000 7f3901400640 1 -- 172.21.15.90:0/3159788085 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38fc09ecf0 con 0x7f38fc140480 2024-03-24T01:26:37.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.807+0000 7f38f2a00640 1 -- 172.21.15.90:0/3159788085 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f38f401b070 con 0x7f38fc140480 2024-03-24T01:26:37.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.808+0000 7f38f2a00640 1 --2- 172.21.15.90:0/3159788085 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f38d806bc00 0x7f38d806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:37.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.808+0000 7f38f2a00640 1 -- 172.21.15.90:0/3159788085 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f38f408a5b0 con 0x7f38fc140480 2024-03-24T01:26:37.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.809+0000 7f38fa800640 1 --2- 172.21.15.90:0/3159788085 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f38d806bc00 0x7f38d806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:37.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.810+0000 7f38fa800640 1 --2- 172.21.15.90:0/3159788085 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f38d806bc00 0x7f38d806e0c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f38ec0017e0 tx=0x7f38ec009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:37.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:37.812+0000 7f38f2a00640 1 -- 172.21.15.90:0/3159788085 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f38f4058f30 con 0x7f38fc140480 2024-03-24T01:26:38.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.009+0000 7f3901400640 1 -- 172.21.15.90:0/3159788085 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38fc04f4e0 con 0x7f38fc140480 2024-03-24T01:26:38.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.011+0000 7f38f2a00640 1 -- 172.21.15.90:0/3159788085 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+17047 (secure 0 0 0) 0x7f38f4058830 con 0x7f38fc140480 2024-03-24T01:26:38.011 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:26:38.011 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":40,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:26:27.648103+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:25:57.398864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-24T01:25:58.451150+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:6835/2166652035":"2024-03-25T01:26:27.648052+0000","172.21.15.90:6834/2166652035":"2024-03-25T01:26:27.648052+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:26:38.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.013+0000 7f3901400640 1 -- 172.21.15.90:0/3159788085 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f38d806bc00 msgr2=0x7f38d806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:38.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.013+0000 7f3901400640 1 --2- 172.21.15.90:0/3159788085 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f38d806bc00 0x7f38d806e0c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f38ec0017e0 tx=0x7f38ec009040 comp rx=0 tx=0).stop 2024-03-24T01:26:38.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.013+0000 7f3901400640 1 -- 172.21.15.90:0/3159788085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38fc140480 msgr2=0x7f38fc140900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:38.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.013+0000 7f3901400640 1 --2- 172.21.15.90:0/3159788085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38fc140480 0x7f38fc140900 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f38fc15fdc0 tx=0x7f38f400dc00 comp rx=0 tx=0).stop 2024-03-24T01:26:38.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.014+0000 7f3901400640 1 -- 172.21.15.90:0/3159788085 shutdown_connections 2024-03-24T01:26:38.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.014+0000 7f3901400640 1 --2- 172.21.15.90:0/3159788085 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f38d806bc00 0x7f38d806e0c0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:38.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.014+0000 7f3901400640 1 --2- 172.21.15.90:0/3159788085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38fc140480 0x7f38fc140900 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:38.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.014+0000 7f3901400640 1 --2- 172.21.15.90:0/3159788085 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f38fc13ee10 0x7f38fc13f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:38.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.014+0000 7f3901400640 1 --2- 172.21.15.90:0/3159788085 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38e0004800 0x7f38fc13e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:38.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.014+0000 7f3901400640 1 -- 172.21.15.90:0/3159788085 >> 172.21.15.90:0/3159788085 conn(0x7f38fc073170 msgr2=0x7f38fc070610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:38.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.014+0000 7f3901400640 1 -- 172.21.15.90:0/3159788085 shutdown_connections 2024-03-24T01:26:38.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.014+0000 7f3901400640 1 -- 172.21.15.90:0/3159788085 wait complete. 2024-03-24T01:26:38.034 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T01:25:57.398864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T01:25:58.451150+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T01:26:38.035 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:26:38.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.242+0000 7f3d9b200640 1 Processor -- start 2024-03-24T01:26:38.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.242+0000 7f3d9b200640 1 -- start start 2024-03-24T01:26:38.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.243+0000 7f3d9b200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3d940aeeb0 0x7f3d940b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:38.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.243+0000 7f3d9b200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d9406dac0 con 0x7f3d9415bc50 2024-03-24T01:26:38.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.243+0000 7f3d9b200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d9406dda0 con 0x7f3d9406b7d0 2024-03-24T01:26:38.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.243+0000 7f3d9b200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d940b4da0 con 0x7f3d940aeeb0 2024-03-24T01:26:38.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.243+0000 7f3d93400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f3d9415bc50 0x7f3d9406b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:34590/0 (socket says 172.21.15.90:34590) 2024-03-24T01:26:38.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.243+0000 7f3d93400640 1 -- 172.21.15.90:0/3588825654 learned_addr learned my addr 172.21.15.90:0/3588825654 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:26:38.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.244+0000 7f3d93e00640 1 --2- 172.21.15.90:0/3588825654 >> v2:172.21.15.120:3300/0 conn(0x7f3d940aeeb0 0x7f3d940b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:38.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.244+0000 7f3d92a00640 1 -- 172.21.15.90:0/3588825654 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1780775017 0 0) 0x7f3d9406dac0 con 0x7f3d9415bc50 2024-03-24T01:26:38.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.244+0000 7f3d92a00640 1 -- 172.21.15.90:0/3588825654 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d78003520 con 0x7f3d9415bc50 2024-03-24T01:26:38.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.244+0000 7f3d92a00640 1 -- 172.21.15.90:0/3588825654 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1024991252 0 0) 0x7f3d78003520 con 0x7f3d9415bc50 2024-03-24T01:26:38.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.244+0000 7f3d92a00640 1 -- 172.21.15.90:0/3588825654 >> v2:172.21.15.120:3300/0 conn(0x7f3d940aeeb0 msgr2=0x7f3d940b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:38.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.244+0000 7f3d92a00640 1 --2- 172.21.15.90:0/3588825654 >> v2:172.21.15.120:3300/0 conn(0x7f3d940aeeb0 0x7f3d940b44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:38.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.244+0000 7f3d92a00640 1 -- 172.21.15.90:0/3588825654 >> v1:172.21.15.120:6789/0 conn(0x7f3d9406b7d0 legacy=0x7f3d940b2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:38.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.244+0000 7f3d92a00640 1 -- 172.21.15.90:0/3588825654 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d9406e080 con 0x7f3d9415bc50 2024-03-24T01:26:38.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.244+0000 7f3d92a00640 1 -- 172.21.15.90:0/3588825654 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f3d80001670 con 0x7f3d9415bc50 2024-03-24T01:26:38.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.244+0000 7f3d92a00640 1 -- 172.21.15.90:0/3588825654 >> v1:172.21.15.90:6789/0 conn(0x7f3d9415bc50 legacy=0x7f3d9406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:38.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.244+0000 7f3d92a00640 1 --2- 172.21.15.90:0/3588825654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d780039b0 0x7f3d78003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:38.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.244+0000 7f3d92a00640 1 --2- 172.21.15.90:0/3588825654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d78004800 0x7f3d78004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:38.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.244+0000 7f3d92a00640 1 --2- 172.21.15.90:0/3588825654 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d780052e0 0x7f3d7800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:38.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.244+0000 7f3d92a00640 1 -- 172.21.15.90:0/3588825654 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d940b4da0 con 0x7f3d78004800 2024-03-24T01:26:38.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.244+0000 7f3d92a00640 1 -- 172.21.15.90:0/3588825654 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7800c1b0 con 0x7f3d780039b0 2024-03-24T01:26:38.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.244+0000 7f3d92a00640 1 -- 172.21.15.90:0/3588825654 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7800c3b0 con 0x7f3d780052e0 2024-03-24T01:26:38.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.245+0000 7f3d93400640 1 --2- 172.21.15.90:0/3588825654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d78004800 0x7f3d78004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:38.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.245+0000 7f3d93e00640 1 --2- 172.21.15.90:0/3588825654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d780039b0 0x7f3d78003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:38.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.245+0000 7f3d93400640 1 -- 172.21.15.90:0/3588825654 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d780052e0 msgr2=0x7f3d7800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:38.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.245+0000 7f3d99200640 1 --2- 172.21.15.90:0/3588825654 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d780052e0 0x7f3d7800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:38.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.245+0000 7f3d93400640 1 --2- 172.21.15.90:0/3588825654 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d780052e0 0x7f3d7800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:38.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.245+0000 7f3d93400640 1 -- 172.21.15.90:0/3588825654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d780039b0 msgr2=0x7f3d78003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:38.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.245+0000 7f3d93400640 1 --2- 172.21.15.90:0/3588825654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d780039b0 0x7f3d78003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:38.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.245+0000 7f3d93400640 1 -- 172.21.15.90:0/3588825654 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d78003520 con 0x7f3d78004800 2024-03-24T01:26:38.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.245+0000 7f3d93e00640 1 --2- 172.21.15.90:0/3588825654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d780039b0 0x7f3d78003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:26:38.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.245+0000 7f3d99200640 1 --2- 172.21.15.90:0/3588825654 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d780052e0 0x7f3d7800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:38.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.245+0000 7f3d99200640 1 -- 172.21.15.90:0/3588825654 reap_dead start 2024-03-24T01:26:38.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.246+0000 7f3d93400640 1 --2- 172.21.15.90:0/3588825654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d78004800 0x7f3d78004c80 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f3d8000c210 tx=0x7f3d78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25148535b8198b4c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:38.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.247+0000 7f3d92a00640 1 -- 172.21.15.90:0/3588825654 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d80007d10 con 0x7f3d78004800 2024-03-24T01:26:38.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.247+0000 7f3d92a00640 1 -- 172.21.15.90:0/3588825654 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d80030070 con 0x7f3d78004800 2024-03-24T01:26:38.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.247+0000 7f3d92a00640 1 -- 172.21.15.90:0/3588825654 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d80005d90 con 0x7f3d78004800 2024-03-24T01:26:38.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.247+0000 7f3d9b200640 1 -- 172.21.15.90:0/3588825654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d78004800 msgr2=0x7f3d78004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:38.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.247+0000 7f3d9b200640 1 --2- 172.21.15.90:0/3588825654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d78004800 0x7f3d78004c80 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f3d8000c210 tx=0x7f3d78005830 comp rx=0 tx=0).stop 2024-03-24T01:26:38.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.247+0000 7f3d9b200640 1 -- 172.21.15.90:0/3588825654 shutdown_connections 2024-03-24T01:26:38.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.247+0000 7f3d9b200640 1 --2- 172.21.15.90:0/3588825654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d78004800 0x7f3d78004c80 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:38.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.247+0000 7f3d9b200640 1 -- 172.21.15.90:0/3588825654 >> 172.21.15.90:0/3588825654 conn(0x7f3d940b1770 msgr2=0x7f3d940b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:38.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.247+0000 7f3d9b200640 1 -- 172.21.15.90:0/3588825654 shutdown_connections 2024-03-24T01:26:38.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.248+0000 7f3d9b200640 1 -- 172.21.15.90:0/3588825654 wait complete. 2024-03-24T01:26:38.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.249+0000 7f3d9b200640 1 Processor -- start 2024-03-24T01:26:38.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.249+0000 7f3d9b200640 1 -- start start 2024-03-24T01:26:38.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.250+0000 7f3d9b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d78004800 0x7f3d940b2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:38.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.250+0000 7f3d9b200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d9413edb0 0x7f3d941523e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:38.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.250+0000 7f3d9b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d94152920 0x7f3d94141660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:38.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.250+0000 7f3d9b200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9406e880 con 0x7f3d78004800 2024-03-24T01:26:38.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.250+0000 7f3d9b200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d940af290 con 0x7f3d9413edb0 2024-03-24T01:26:38.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.250+0000 7f3d9b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9406ea00 con 0x7f3d94152920 2024-03-24T01:26:38.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.250+0000 7f3d93e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d78004800 0x7f3d940b2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:38.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.251+0000 7f3d93e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d78004800 0x7f3d940b2680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59644/0 (socket says 172.21.15.90:59644) 2024-03-24T01:26:38.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.251+0000 7f3d93e00640 1 -- 172.21.15.90:0/554890469 learned_addr learned my addr 172.21.15.90:0/554890469 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:38.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.251+0000 7f3d93400640 1 --2- 172.21.15.90:0/554890469 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d9413edb0 0x7f3d941523e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:38.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.251+0000 7f3d99200640 1 --2- 172.21.15.90:0/554890469 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d94152920 0x7f3d94141660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:38.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.251+0000 7f3d93e00640 1 -- 172.21.15.90:0/554890469 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d94152920 msgr2=0x7f3d94141660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:38.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.251+0000 7f3d93e00640 1 --2- 172.21.15.90:0/554890469 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d94152920 0x7f3d94141660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:38.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.251+0000 7f3d93e00640 1 -- 172.21.15.90:0/554890469 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d9413edb0 msgr2=0x7f3d941523e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:38.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.251+0000 7f3d93e00640 1 --2- 172.21.15.90:0/554890469 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d9413edb0 0x7f3d941523e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:38.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.251+0000 7f3d93e00640 1 -- 172.21.15.90:0/554890469 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d94152da0 con 0x7f3d78004800 2024-03-24T01:26:38.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.251+0000 7f3d93400640 1 --2- 172.21.15.90:0/554890469 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d9413edb0 0x7f3d941523e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:38.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.251+0000 7f3d93e00640 1 --2- 172.21.15.90:0/554890469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d78004800 0x7f3d940b2680 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f3d840116d0 tx=0x7f3d84011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:38.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.252+0000 7f3d8be00640 1 -- 172.21.15.90:0/554890469 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8400f9c0 con 0x7f3d78004800 2024-03-24T01:26:38.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.252+0000 7f3d8be00640 1 -- 172.21.15.90:0/554890469 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d8400fb60 con 0x7f3d78004800 2024-03-24T01:26:38.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.252+0000 7f3d8be00640 1 -- 172.21.15.90:0/554890469 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d84015460 con 0x7f3d78004800 2024-03-24T01:26:38.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.252+0000 7f3d9b200640 1 -- 172.21.15.90:0/554890469 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d940b2eb0 con 0x7f3d78004800 2024-03-24T01:26:38.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.252+0000 7f3d9b200640 1 -- 172.21.15.90:0/554890469 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d940b3470 con 0x7f3d78004800 2024-03-24T01:26:38.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.253+0000 7f3d9b200640 1 -- 172.21.15.90:0/554890469 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d940b3b80 con 0x7f3d78004800 2024-03-24T01:26:38.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.254+0000 7f3d8be00640 1 -- 172.21.15.90:0/554890469 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3d84005990 con 0x7f3d78004800 2024-03-24T01:26:38.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.255+0000 7f3d8be00640 1 --2- 172.21.15.90:0/554890469 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3d7406bc00 0x7f3d7406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:38.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.255+0000 7f3d8be00640 1 -- 172.21.15.90:0/554890469 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f3d8408b350 con 0x7f3d78004800 2024-03-24T01:26:38.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.255+0000 7f3d93400640 1 --2- 172.21.15.90:0/554890469 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3d7406bc00 0x7f3d7406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:38.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.256+0000 7f3d93400640 1 --2- 172.21.15.90:0/554890469 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3d7406bc00 0x7f3d7406e0c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3d80034000 tx=0x7f3d80009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:38.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.259+0000 7f3d8be00640 1 -- 172.21.15.90:0/554890469 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3d84019030 con 0x7f3d78004800 2024-03-24T01:26:38.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.621+0000 7f3d9b200640 1 -- 172.21.15.90:0/554890469 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3d9404f400 con 0x7f3d78004800 2024-03-24T01:26:38.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.622+0000 7f3d8be00640 1 -- 172.21.15.90:0/554890469 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3d8401b300 con 0x7f3d78004800 2024-03-24T01:26:38.622 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:26:38.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.625+0000 7f3d9b200640 1 -- 172.21.15.90:0/554890469 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3d7406bc00 msgr2=0x7f3d7406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:38.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.625+0000 7f3d9b200640 1 --2- 172.21.15.90:0/554890469 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3d7406bc00 0x7f3d7406e0c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3d80034000 tx=0x7f3d80009030 comp rx=0 tx=0).stop 2024-03-24T01:26:38.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.625+0000 7f3d9b200640 1 -- 172.21.15.90:0/554890469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d78004800 msgr2=0x7f3d940b2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:38.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.625+0000 7f3d9b200640 1 --2- 172.21.15.90:0/554890469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d78004800 0x7f3d940b2680 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f3d840116d0 tx=0x7f3d84011b90 comp rx=0 tx=0).stop 2024-03-24T01:26:38.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.626+0000 7f3d9b200640 1 -- 172.21.15.90:0/554890469 shutdown_connections 2024-03-24T01:26:38.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.626+0000 7f3d9b200640 1 --2- 172.21.15.90:0/554890469 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3d7406bc00 0x7f3d7406e0c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:38.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.626+0000 7f3d9b200640 1 --2- 172.21.15.90:0/554890469 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3d94152920 0x7f3d94141660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:38.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.626+0000 7f3d9b200640 1 --2- 172.21.15.90:0/554890469 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d9413edb0 0x7f3d941523e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:38.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.626+0000 7f3d9b200640 1 --2- 172.21.15.90:0/554890469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d78004800 0x7f3d940b2680 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:38.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.626+0000 7f3d9b200640 1 -- 172.21.15.90:0/554890469 >> 172.21.15.90:0/554890469 conn(0x7f3d940b1770 msgr2=0x7f3d9415e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:38.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.626+0000 7f3d9b200640 1 -- 172.21.15.90:0/554890469 shutdown_connections 2024-03-24T01:26:38.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.626+0000 7f3d9b200640 1 -- 172.21.15.90:0/554890469 wait complete. 2024-03-24T01:26:38.650 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T01:26:38.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.843+0000 7f272cc00640 1 Processor -- start 2024-03-24T01:26:38.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.843+0000 7f272cc00640 1 -- start start 2024-03-24T01:26:38.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.844+0000 7f272cc00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f272815da30 0x7f272815de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:38.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.844+0000 7f272cc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27280acea0 0x7f27280aea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:38.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.844+0000 7f272cc00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f272815e550 con 0x7f272806b0c0 2024-03-24T01:26:38.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.844+0000 7f272cc00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27280b2c70 con 0x7f272815da30 2024-03-24T01:26:38.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.844+0000 7f272cc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f272815f160 con 0x7f27280acea0 2024-03-24T01:26:38.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.845+0000 7f2725e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f272815da30 0x7f272815de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:38.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.845+0000 7f2725e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f272815da30 0x7f272815de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59648/0 (socket says 172.21.15.90:59648) 2024-03-24T01:26:38.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.845+0000 7f2725e00640 1 -- 172.21.15.90:0/2628828679 learned_addr learned my addr 172.21.15.90:0/2628828679 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:38.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.845+0000 7f2725400640 1 --2- 172.21.15.90:0/2628828679 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27280acea0 0x7f27280aea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:38.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.845+0000 7f2725e00640 1 -- 172.21.15.90:0/2628828679 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27280acea0 msgr2=0x7f27280aea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:38.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.845+0000 7f2725e00640 1 --2- 172.21.15.90:0/2628828679 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27280acea0 0x7f27280aea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:38.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.845+0000 7f2725e00640 1 -- 172.21.15.90:0/2628828679 >> v1:172.21.15.120:6789/0 conn(0x7f272806b0c0 legacy=0x7f2728161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:38.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.845+0000 7f2725e00640 1 -- 172.21.15.90:0/2628828679 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f272815e830 con 0x7f272815da30 2024-03-24T01:26:38.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.846+0000 7f2725e00640 1 --2- 172.21.15.90:0/2628828679 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f272815da30 0x7f272815de10 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f2718009870 tx=0x7f271802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c260dc4bc0bda832 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:38.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.847+0000 7f2724a00640 1 -- 172.21.15.90:0/2628828679 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2718035020 con 0x7f272815da30 2024-03-24T01:26:38.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.847+0000 7f2724a00640 1 -- 172.21.15.90:0/2628828679 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2718031640 con 0x7f272815da30 2024-03-24T01:26:38.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.847+0000 7f2724a00640 1 -- 172.21.15.90:0/2628828679 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f271802d040 con 0x7f272815da30 2024-03-24T01:26:38.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.847+0000 7f272cc00640 1 -- 172.21.15.90:0/2628828679 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f272815da30 msgr2=0x7f272815de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:38.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.847+0000 7f272cc00640 1 --2- 172.21.15.90:0/2628828679 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f272815da30 0x7f272815de10 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f2718009870 tx=0x7f271802b8f0 comp rx=0 tx=0).stop 2024-03-24T01:26:38.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.847+0000 7f272cc00640 1 -- 172.21.15.90:0/2628828679 shutdown_connections 2024-03-24T01:26:38.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.847+0000 7f272cc00640 1 --2- 172.21.15.90:0/2628828679 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27280acea0 0x7f27280aea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:38.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.847+0000 7f272cc00640 1 --2- 172.21.15.90:0/2628828679 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f272815da30 0x7f272815de10 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:38.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.847+0000 7f272cc00640 1 -- 172.21.15.90:0/2628828679 >> 172.21.15.90:0/2628828679 conn(0x7f27280b1770 msgr2=0x7f27280b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:38.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.848+0000 7f272cc00640 1 -- 172.21.15.90:0/2628828679 shutdown_connections 2024-03-24T01:26:38.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.848+0000 7f272cc00640 1 -- 172.21.15.90:0/2628828679 wait complete. 2024-03-24T01:26:38.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.849+0000 7f272cc00640 1 Processor -- start 2024-03-24T01:26:38.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.849+0000 7f272cc00640 1 -- start start 2024-03-24T01:26:38.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.851+0000 7f272cc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f272806b0c0 0x7f272814c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:38.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.851+0000 7f272cc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27280acea0 0x7f27280cbb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:38.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.851+0000 7f272cc00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f272815da30 0x7f27281492a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:38.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.851+0000 7f272cc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27280ad300 con 0x7f27280acea0 2024-03-24T01:26:38.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.851+0000 7f272cc00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f272815edf0 con 0x7f272815da30 2024-03-24T01:26:38.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.851+0000 7f272cc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27280aef70 con 0x7f272806b0c0 2024-03-24T01:26:38.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.851+0000 7f2725400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27280acea0 0x7f27280cbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:38.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.851+0000 7f2725400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27280acea0 0x7f27280cbb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59652/0 (socket says 172.21.15.90:59652) 2024-03-24T01:26:38.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.851+0000 7f2725400640 1 -- 172.21.15.90:0/4185882935 learned_addr learned my addr 172.21.15.90:0/4185882935 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:38.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.852+0000 7f2725e00640 1 --2- 172.21.15.90:0/4185882935 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f272806b0c0 0x7f272814c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:38.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.852+0000 7f2725400640 1 -- 172.21.15.90:0/4185882935 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f272806b0c0 msgr2=0x7f272814c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:38.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.852+0000 7f2725400640 1 --2- 172.21.15.90:0/4185882935 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f272806b0c0 0x7f272814c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:38.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.852+0000 7f2726800640 1 --2- 172.21.15.90:0/4185882935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f272815da30 0x7f27281492a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:38.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.852+0000 7f2725400640 1 -- 172.21.15.90:0/4185882935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f272815da30 msgr2=0x7f27281492a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:38.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.852+0000 7f2725400640 1 --2- 172.21.15.90:0/4185882935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f272815da30 0x7f27281492a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:38.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.852+0000 7f2725400640 1 -- 172.21.15.90:0/4185882935 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27280cc260 con 0x7f27280acea0 2024-03-24T01:26:38.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.852+0000 7f2726800640 1 --2- 172.21.15.90:0/4185882935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f272815da30 0x7f27281492a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:38.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.852+0000 7f2725400640 1 --2- 172.21.15.90:0/4185882935 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27280acea0 0x7f27280cbb10 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f271400c820 tx=0x7f271400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:38.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.852+0000 7f271e000640 1 -- 172.21.15.90:0/4185882935 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2714015070 con 0x7f27280acea0 2024-03-24T01:26:38.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.852+0000 7f271e000640 1 -- 172.21.15.90:0/4185882935 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2714004240 con 0x7f27280acea0 2024-03-24T01:26:38.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.852+0000 7f272cc00640 1 -- 172.21.15.90:0/4185882935 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27280cc540 con 0x7f27280acea0 2024-03-24T01:26:38.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.852+0000 7f271e000640 1 -- 172.21.15.90:0/4185882935 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f271400ce10 con 0x7f27280acea0 2024-03-24T01:26:38.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.852+0000 7f272cc00640 1 -- 172.21.15.90:0/4185882935 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27280ccb00 con 0x7f27280acea0 2024-03-24T01:26:38.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.854+0000 7f272cc00640 1 -- 172.21.15.90:0/4185882935 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f272804f400 con 0x7f27280acea0 2024-03-24T01:26:38.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.854+0000 7f271e000640 1 -- 172.21.15.90:0/4185882935 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2714007830 con 0x7f27280acea0 2024-03-24T01:26:38.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.856+0000 7f271e000640 1 --2- 172.21.15.90:0/4185882935 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f270806bbd0 0x7f270806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:38.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.857+0000 7f271e000640 1 -- 172.21.15.90:0/4185882935 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f271408ab30 con 0x7f27280acea0 2024-03-24T01:26:38.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.857+0000 7f2725e00640 1 --2- 172.21.15.90:0/4185882935 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f270806bbd0 0x7f270806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:38.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.858+0000 7f2725e00640 1 --2- 172.21.15.90:0/4185882935 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f270806bbd0 0x7f270806e090 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f2718036000 tx=0x7f2718008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:38.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:38.860+0000 7f271e000640 1 -- 172.21.15.90:0/4185882935 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2714059440 con 0x7f27280acea0 2024-03-24T01:26:39.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.051+0000 7f272cc00640 1 -- 172.21.15.90:0/4185882935 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f272806e250 con 0x7f27280acea0 2024-03-24T01:26:39.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.052+0000 7f271e000640 1 -- 172.21.15.90:0/4185882935 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v40) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2714058d40 con 0x7f27280acea0 2024-03-24T01:26:39.052 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:26:39.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.054+0000 7f272cc00640 1 -- 172.21.15.90:0/4185882935 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f270806bbd0 msgr2=0x7f270806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:39.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.054+0000 7f272cc00640 1 --2- 172.21.15.90:0/4185882935 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f270806bbd0 0x7f270806e090 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f2718036000 tx=0x7f2718008030 comp rx=0 tx=0).stop 2024-03-24T01:26:39.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.054+0000 7f272cc00640 1 -- 172.21.15.90:0/4185882935 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27280acea0 msgr2=0x7f27280cbb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:39.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.054+0000 7f272cc00640 1 --2- 172.21.15.90:0/4185882935 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27280acea0 0x7f27280cbb10 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f271400c820 tx=0x7f271400cce0 comp rx=0 tx=0).stop 2024-03-24T01:26:39.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.054+0000 7f272cc00640 1 -- 172.21.15.90:0/4185882935 shutdown_connections 2024-03-24T01:26:39.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.054+0000 7f272cc00640 1 --2- 172.21.15.90:0/4185882935 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f270806bbd0 0x7f270806e090 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:39.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.054+0000 7f272cc00640 1 --2- 172.21.15.90:0/4185882935 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27280acea0 0x7f27280cbb10 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:39.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.054+0000 7f272cc00640 1 --2- 172.21.15.90:0/4185882935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f272815da30 0x7f27281492a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:39.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.054+0000 7f272cc00640 1 --2- 172.21.15.90:0/4185882935 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f272806b0c0 0x7f272814c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:39.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.054+0000 7f272cc00640 1 -- 172.21.15.90:0/4185882935 >> 172.21.15.90:0/4185882935 conn(0x7f27280b1770 msgr2=0x7f27280b2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:39.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.054+0000 7f272cc00640 1 -- 172.21.15.90:0/4185882935 shutdown_connections 2024-03-24T01:26:39.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.054+0000 7f272cc00640 1 -- 172.21.15.90:0/4185882935 wait complete. 2024-03-24T01:26:39.072 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T01:26:39.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.285+0000 7f4731000640 1 Processor -- start 2024-03-24T01:26:39.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.285+0000 7f4731000640 1 -- start start 2024-03-24T01:26:39.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.287+0000 7f4731000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f472c0728d0 0x7f472c0b4e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:39.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.287+0000 7f4731000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f472c0cf770 0x7f472c0cdab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:39.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.287+0000 7f4731000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f472c0cfc70 con 0x7f472c15a300 2024-03-24T01:26:39.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.287+0000 7f4731000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f472c0b3150 con 0x7f472c0cf770 2024-03-24T01:26:39.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.287+0000 7f4731000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f472c0b39f0 con 0x7f472c0728d0 2024-03-24T01:26:39.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.287+0000 7f4729e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f472c15a300 0x7f472c0721c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:34594/0 (socket says 172.21.15.90:34594) 2024-03-24T01:26:39.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.287+0000 7f4729e00640 1 -- 172.21.15.90:0/970266596 learned_addr learned my addr 172.21.15.90:0/970266596 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:26:39.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.287+0000 7f472a800640 1 --2- 172.21.15.90:0/970266596 >> v2:172.21.15.90:3300/0 conn(0x7f472c0cf770 0x7f472c0cdab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:39.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.288+0000 7f4721400640 1 --2- 172.21.15.90:0/970266596 >> v2:172.21.15.120:3300/0 conn(0x7f472c0728d0 0x7f472c0b4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:39.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.288+0000 7f472a800640 1 -- 172.21.15.90:0/970266596 >> v2:172.21.15.120:3300/0 conn(0x7f472c0728d0 msgr2=0x7f472c0b4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:39.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.288+0000 7f472a800640 1 --2- 172.21.15.90:0/970266596 >> v2:172.21.15.120:3300/0 conn(0x7f472c0728d0 0x7f472c0b4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:39.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.288+0000 7f472a800640 1 -- 172.21.15.90:0/970266596 >> v1:172.21.15.90:6789/0 conn(0x7f472c15a300 legacy=0x7f472c0721c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:39.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.288+0000 7f472a800640 1 -- 172.21.15.90:0/970266596 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f472c0b33a0 con 0x7f472c0cf770 2024-03-24T01:26:39.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.288+0000 7f4729400640 1 -- 172.21.15.90:0/970266596 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 541521817 0 0) 0x7f472c0cfc70 con 0x7f472c15a300 2024-03-24T01:26:39.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.289+0000 7f472a800640 1 --2- 172.21.15.90:0/970266596 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f472c0cf770 0x7f472c0cdab0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f472c0cd540 tx=0x7f472402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b36c7f9b62ac3aaf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:39.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.289+0000 7f4729400640 1 -- 172.21.15.90:0/970266596 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4724039070 con 0x7f472c0cf770 2024-03-24T01:26:39.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.289+0000 7f4729400640 1 -- 172.21.15.90:0/970266596 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f472402cc80 con 0x7f472c0cf770 2024-03-24T01:26:39.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.289+0000 7f4729400640 1 -- 172.21.15.90:0/970266596 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4724034660 con 0x7f472c0cf770 2024-03-24T01:26:39.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.290+0000 7f4731000640 1 -- 172.21.15.90:0/970266596 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f472c0cf770 msgr2=0x7f472c0cdab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:39.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.290+0000 7f4731000640 1 --2- 172.21.15.90:0/970266596 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f472c0cf770 0x7f472c0cdab0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f472c0cd540 tx=0x7f472402bed0 comp rx=0 tx=0).stop 2024-03-24T01:26:39.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.290+0000 7f4731000640 1 -- 172.21.15.90:0/970266596 shutdown_connections 2024-03-24T01:26:39.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.290+0000 7f4731000640 1 --2- 172.21.15.90:0/970266596 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f472c0cf770 0x7f472c0cdab0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:39.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.290+0000 7f4731000640 1 --2- 172.21.15.90:0/970266596 >> v2:172.21.15.120:3300/0 conn(0x7f472c0728d0 0x7f472c0b4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:39.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.290+0000 7f4731000640 1 -- 172.21.15.90:0/970266596 >> 172.21.15.90:0/970266596 conn(0x7f472c0b1e50 msgr2=0x7f472c0b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:39.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.290+0000 7f4731000640 1 -- 172.21.15.90:0/970266596 shutdown_connections 2024-03-24T01:26:39.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.290+0000 7f4731000640 1 -- 172.21.15.90:0/970266596 wait complete. 2024-03-24T01:26:39.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.292+0000 7f4731000640 1 Processor -- start 2024-03-24T01:26:39.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.292+0000 7f4731000640 1 -- start start 2024-03-24T01:26:39.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.293+0000 7f4731000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f472c0728d0 0x7f472c146dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:39.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.293+0000 7f4731000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f472c0cf770 0x7f472c143b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:39.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.293+0000 7f4731000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f472c15a300 0x7f472c144090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:39.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.293+0000 7f4731000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472c0b3e80 con 0x7f472c15a300 2024-03-24T01:26:39.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.293+0000 7f4731000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472c06c410 con 0x7f472c0728d0 2024-03-24T01:26:39.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.293+0000 7f4731000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f472c0b4000 con 0x7f472c0cf770 2024-03-24T01:26:39.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.293+0000 7f472a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f472c15a300 0x7f472c144090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:39.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.293+0000 7f472a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f472c15a300 0x7f472c144090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59682/0 (socket says 172.21.15.90:59682) 2024-03-24T01:26:39.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.293+0000 7f472a800640 1 -- 172.21.15.90:0/2426769955 learned_addr learned my addr 172.21.15.90:0/2426769955 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:39.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.293+0000 7f4729e00640 1 --2- 172.21.15.90:0/2426769955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f472c0728d0 0x7f472c146dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:39.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.293+0000 7f4721400640 1 --2- 172.21.15.90:0/2426769955 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f472c0cf770 0x7f472c143b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:39.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.293+0000 7f472a800640 1 -- 172.21.15.90:0/2426769955 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f472c0cf770 msgr2=0x7f472c143b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:39.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.293+0000 7f472a800640 1 --2- 172.21.15.90:0/2426769955 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f472c0cf770 0x7f472c143b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:39.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.293+0000 7f472a800640 1 -- 172.21.15.90:0/2426769955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f472c0728d0 msgr2=0x7f472c146dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:39.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.293+0000 7f472a800640 1 --2- 172.21.15.90:0/2426769955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f472c0728d0 0x7f472c146dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:39.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.293+0000 7f472a800640 1 -- 172.21.15.90:0/2426769955 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f472c140e50 con 0x7f472c15a300 2024-03-24T01:26:39.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.294+0000 7f472a800640 1 --2- 172.21.15.90:0/2426769955 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f472c15a300 0x7f472c144090 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f472402c4e0 tx=0x7f4724004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:39.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.294+0000 7f4722a00640 1 -- 172.21.15.90:0/2426769955 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4724044020 con 0x7f472c15a300 2024-03-24T01:26:39.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.294+0000 7f4722a00640 1 -- 172.21.15.90:0/2426769955 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f472403f470 con 0x7f472c15a300 2024-03-24T01:26:39.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.294+0000 7f4731000640 1 -- 172.21.15.90:0/2426769955 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f472c15bfd0 con 0x7f472c15a300 2024-03-24T01:26:39.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.294+0000 7f4722a00640 1 -- 172.21.15.90:0/2426769955 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4724039040 con 0x7f472c15a300 2024-03-24T01:26:39.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.294+0000 7f4731000640 1 -- 172.21.15.90:0/2426769955 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f472c15c2b0 con 0x7f472c15a300 2024-03-24T01:26:39.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.296+0000 7f4731000640 1 -- 172.21.15.90:0/2426769955 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f472c04f400 con 0x7f472c15a300 2024-03-24T01:26:39.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.296+0000 7f4722a00640 1 -- 172.21.15.90:0/2426769955 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4724047020 con 0x7f472c15a300 2024-03-24T01:26:39.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.297+0000 7f4722a00640 1 --2- 172.21.15.90:0/2426769955 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f470806bc00 0x7f470806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:39.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.297+0000 7f4722a00640 1 -- 172.21.15.90:0/2426769955 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f47240ad5c0 con 0x7f472c15a300 2024-03-24T01:26:39.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.298+0000 7f4729e00640 1 --2- 172.21.15.90:0/2426769955 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f470806bc00 0x7f470806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:39.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.299+0000 7f4729e00640 1 --2- 172.21.15.90:0/2426769955 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f470806bc00 0x7f470806e0c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f471c000f60 tx=0x7f471c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:39.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.302+0000 7f4722a00640 1 -- 172.21.15.90:0/2426769955 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f472404e070 con 0x7f472c15a300 2024-03-24T01:26:39.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.502+0000 7f4731000640 1 -- 172.21.15.90:0/2426769955 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f472c06e2b0 con 0x7f472c15a300 2024-03-24T01:26:39.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.503+0000 7f4722a00640 1 -- 172.21.15.90:0/2426769955 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v40) v1 ==== 100+0+11 (secure 0 0 0) 0x7f472407be30 con 0x7f472c15a300 2024-03-24T01:26:39.503 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-24T01:26:39.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.506+0000 7f4731000640 1 -- 172.21.15.90:0/2426769955 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f470806bc00 msgr2=0x7f470806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:39.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.506+0000 7f4731000640 1 --2- 172.21.15.90:0/2426769955 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f470806bc00 0x7f470806e0c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f471c000f60 tx=0x7f471c009040 comp rx=0 tx=0).stop 2024-03-24T01:26:39.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.506+0000 7f4731000640 1 -- 172.21.15.90:0/2426769955 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f472c15a300 msgr2=0x7f472c144090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:39.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.506+0000 7f4731000640 1 --2- 172.21.15.90:0/2426769955 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f472c15a300 0x7f472c144090 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f472402c4e0 tx=0x7f4724004a40 comp rx=0 tx=0).stop 2024-03-24T01:26:39.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.506+0000 7f4731000640 1 -- 172.21.15.90:0/2426769955 shutdown_connections 2024-03-24T01:26:39.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.506+0000 7f4731000640 1 --2- 172.21.15.90:0/2426769955 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f470806bc00 0x7f470806e0c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:39.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.506+0000 7f4731000640 1 --2- 172.21.15.90:0/2426769955 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f472c15a300 0x7f472c144090 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:39.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.506+0000 7f4731000640 1 --2- 172.21.15.90:0/2426769955 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f472c0cf770 0x7f472c143b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:39.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.506+0000 7f4731000640 1 --2- 172.21.15.90:0/2426769955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f472c0728d0 0x7f472c146dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:39.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.506+0000 7f4731000640 1 -- 172.21.15.90:0/2426769955 >> 172.21.15.90:0/2426769955 conn(0x7f472c0b1e50 msgr2=0x7f472c159040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:39.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.506+0000 7f4731000640 1 -- 172.21.15.90:0/2426769955 shutdown_connections 2024-03-24T01:26:39.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.506+0000 7f4731000640 1 -- 172.21.15.90:0/2426769955 wait complete. 2024-03-24T01:26:39.534 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T01:26:39.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.728+0000 7fc9ac600640 1 Processor -- start 2024-03-24T01:26:39.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.728+0000 7fc9ac600640 1 -- start start 2024-03-24T01:26:39.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.729+0000 7fc9ac600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc9a415dcd0 0x7fc9a406eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:39.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.729+0000 7fc9ac600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9a406f710 0x7fc9a406fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:39.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.729+0000 7fc9ac600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a406c630 con 0x7fc9a4162350 2024-03-24T01:26:39.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.729+0000 7fc9ac600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9a4070080 con 0x7fc9a415dcd0 2024-03-24T01:26:39.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.729+0000 7fc9ac600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a4162850 con 0x7fc9a406f710 2024-03-24T01:26:39.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.730+0000 7fc9a9c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc9a415dcd0 0x7fc9a406eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:39.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.730+0000 7fc9a9c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc9a415dcd0 0x7fc9a406eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59694/0 (socket says 172.21.15.90:59694) 2024-03-24T01:26:39.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.730+0000 7fc9a9c00640 1 -- 172.21.15.90:0/24562168 learned_addr learned my addr 172.21.15.90:0/24562168 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:39.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.730+0000 7fc9aa600640 1 --2- 172.21.15.90:0/24562168 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9a406f710 0x7fc9a406fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:39.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.730+0000 7fc9a9c00640 1 -- 172.21.15.90:0/24562168 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9a406f710 msgr2=0x7fc9a406fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:39.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.730+0000 7fc9a9c00640 1 --2- 172.21.15.90:0/24562168 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9a406f710 0x7fc9a406fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:39.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.730+0000 7fc9a9c00640 1 -- 172.21.15.90:0/24562168 >> v1:172.21.15.90:6789/0 conn(0x7fc9a4162350 legacy=0x7fc9a406bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:39.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.730+0000 7fc9a9c00640 1 -- 172.21.15.90:0/24562168 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a406b160 con 0x7fc9a415dcd0 2024-03-24T01:26:39.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.730+0000 7fc9aa600640 1 --2- 172.21.15.90:0/24562168 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9a406f710 0x7fc9a406fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:39.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.732+0000 7fc9a9c00640 1 --2- 172.21.15.90:0/24562168 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9a415dcd0 0x7fc9a406eef0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fc998009870 tx=0x7fc99802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b038abf9f0b14f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:39.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.732+0000 7fc9a3e00640 1 -- 172.21.15.90:0/24562168 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc998034070 con 0x7fc9a415dcd0 2024-03-24T01:26:39.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.732+0000 7fc9a3e00640 1 -- 172.21.15.90:0/24562168 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc998030840 con 0x7fc9a415dcd0 2024-03-24T01:26:39.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.732+0000 7fc9a3e00640 1 -- 172.21.15.90:0/24562168 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc998030c10 con 0x7fc9a415dcd0 2024-03-24T01:26:39.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.733+0000 7fc9ac600640 1 -- 172.21.15.90:0/24562168 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9a415dcd0 msgr2=0x7fc9a406eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:39.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.733+0000 7fc9ac600640 1 --2- 172.21.15.90:0/24562168 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9a415dcd0 0x7fc9a406eef0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fc998009870 tx=0x7fc99802b980 comp rx=0 tx=0).stop 2024-03-24T01:26:39.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.733+0000 7fc9ac600640 1 -- 172.21.15.90:0/24562168 shutdown_connections 2024-03-24T01:26:39.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.733+0000 7fc9ac600640 1 --2- 172.21.15.90:0/24562168 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9a406f710 0x7fc9a406fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:39.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.733+0000 7fc9ac600640 1 --2- 172.21.15.90:0/24562168 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9a415dcd0 0x7fc9a406eef0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:39.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.733+0000 7fc9ac600640 1 -- 172.21.15.90:0/24562168 >> 172.21.15.90:0/24562168 conn(0x7fc9a406e180 msgr2=0x7fc9a406e560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:39.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.733+0000 7fc9ac600640 1 -- 172.21.15.90:0/24562168 shutdown_connections 2024-03-24T01:26:39.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.733+0000 7fc9ac600640 1 -- 172.21.15.90:0/24562168 wait complete. 2024-03-24T01:26:39.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.735+0000 7fc9ac600640 1 Processor -- start 2024-03-24T01:26:39.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.735+0000 7fc9ac600640 1 -- start start 2024-03-24T01:26:39.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.736+0000 7fc9ac600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9a406f710 0x7fc9a40b0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:39.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.736+0000 7fc9ac600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9a415dcd0 0x7fc9a414d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:39.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.736+0000 7fc9ac600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9a4162350 0x7fc9a40ad1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:39.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.736+0000 7fc9ac600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a406ba00 con 0x7fc9a4162350 2024-03-24T01:26:39.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.736+0000 7fc9ac600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a406b720 con 0x7fc9a415dcd0 2024-03-24T01:26:39.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.736+0000 7fc9ac600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a4070680 con 0x7fc9a406f710 2024-03-24T01:26:39.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.736+0000 7fc9aa600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9a4162350 0x7fc9a40ad1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:39.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.736+0000 7fc9aa600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9a4162350 0x7fc9a40ad1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59702/0 (socket says 172.21.15.90:59702) 2024-03-24T01:26:39.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.736+0000 7fc9aa600640 1 -- 172.21.15.90:0/760720091 learned_addr learned my addr 172.21.15.90:0/760720091 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:39.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.736+0000 7fc9a9c00640 1 --2- 172.21.15.90:0/760720091 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9a406f710 0x7fc9a40b0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:39.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.737+0000 7fc9aa600640 1 -- 172.21.15.90:0/760720091 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9a406f710 msgr2=0x7fc9a40b0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:39.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.737+0000 7fc9aa600640 1 --2- 172.21.15.90:0/760720091 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9a406f710 0x7fc9a40b0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:39.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.737+0000 7fc9aa600640 1 -- 172.21.15.90:0/760720091 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9a415dcd0 msgr2=0x7fc9a414d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:39.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.737+0000 7fc9aa600640 1 --2- 172.21.15.90:0/760720091 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9a415dcd0 0x7fc9a414d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:39.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.737+0000 7fc9aa600640 1 -- 172.21.15.90:0/760720091 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a40ada20 con 0x7fc9a4162350 2024-03-24T01:26:39.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.737+0000 7fc9aa600640 1 --2- 172.21.15.90:0/760720091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9a4162350 0x7fc9a40ad1f0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fc99c00cf20 tx=0x7fc99c00a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:39.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.737+0000 7fc9a1600640 1 -- 172.21.15.90:0/760720091 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99c01a070 con 0x7fc9a4162350 2024-03-24T01:26:39.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.737+0000 7fc9a1600640 1 -- 172.21.15.90:0/760720091 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc99c00d840 con 0x7fc9a4162350 2024-03-24T01:26:39.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.737+0000 7fc9ac600640 1 -- 172.21.15.90:0/760720091 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9a40add00 con 0x7fc9a4162350 2024-03-24T01:26:39.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.737+0000 7fc9a1600640 1 -- 172.21.15.90:0/760720091 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99c00dc30 con 0x7fc9a4162350 2024-03-24T01:26:39.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.737+0000 7fc9ac600640 1 -- 172.21.15.90:0/760720091 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9a40ae2c0 con 0x7fc9a4162350 2024-03-24T01:26:39.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.738+0000 7fc9ac600640 1 -- 172.21.15.90:0/760720091 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9a404f400 con 0x7fc9a4162350 2024-03-24T01:26:39.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.739+0000 7fc9a1600640 1 -- 172.21.15.90:0/760720091 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc99c01d070 con 0x7fc9a4162350 2024-03-24T01:26:39.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.740+0000 7fc9a1600640 1 --2- 172.21.15.90:0/760720091 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc98406bbd0 0x7fc98406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:39.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.740+0000 7fc9a1600640 1 -- 172.21.15.90:0/760720091 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fc99c08b480 con 0x7fc9a4162350 2024-03-24T01:26:39.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.741+0000 7fc9a9c00640 1 --2- 172.21.15.90:0/760720091 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc98406bbd0 0x7fc98406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:39.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.742+0000 7fc9a9c00640 1 --2- 172.21.15.90:0/760720091 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc98406bbd0 0x7fc98406e090 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc998036000 tx=0x7fc99803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:39.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.745+0000 7fc9a1600640 1 -- 172.21.15.90:0/760720091 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc99c059e20 con 0x7fc9a4162350 2024-03-24T01:26:39.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.954+0000 7fc9ac600640 1 -- 172.21.15.90:0/760720091 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc9a406ecd0 con 0x7fc9a4162350 2024-03-24T01:26:39.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.955+0000 7fc9a1600640 1 -- 172.21.15.90:0/760720091 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v40) v1 ==== 80+0+122 (secure 0 0 0) 0x7fc99c059720 con 0x7fc9a4162350 2024-03-24T01:26:39.956 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6835/2166652035 2024-03-25T01:26:27.648052+0000 2024-03-24T01:26:39.956 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6834/2166652035 2024-03-25T01:26:27.648052+0000 2024-03-24T01:26:39.956 INFO:teuthology.orchestra.run.smithi090.stderr:listed 2 entries 2024-03-24T01:26:39.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.958+0000 7fc9ac600640 1 -- 172.21.15.90:0/760720091 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc98406bbd0 msgr2=0x7fc98406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:39.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.958+0000 7fc9ac600640 1 --2- 172.21.15.90:0/760720091 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc98406bbd0 0x7fc98406e090 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc998036000 tx=0x7fc99803b040 comp rx=0 tx=0).stop 2024-03-24T01:26:39.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.958+0000 7fc9ac600640 1 -- 172.21.15.90:0/760720091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9a4162350 msgr2=0x7fc9a40ad1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:39.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.958+0000 7fc9ac600640 1 --2- 172.21.15.90:0/760720091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9a4162350 0x7fc9a40ad1f0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fc99c00cf20 tx=0x7fc99c00a7d0 comp rx=0 tx=0).stop 2024-03-24T01:26:39.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.958+0000 7fc9ac600640 1 -- 172.21.15.90:0/760720091 shutdown_connections 2024-03-24T01:26:39.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.958+0000 7fc9ac600640 1 --2- 172.21.15.90:0/760720091 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc98406bbd0 0x7fc98406e090 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:39.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.958+0000 7fc9ac600640 1 --2- 172.21.15.90:0/760720091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9a4162350 0x7fc9a40ad1f0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:39.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.958+0000 7fc9ac600640 1 --2- 172.21.15.90:0/760720091 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9a415dcd0 0x7fc9a414d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:39.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.958+0000 7fc9ac600640 1 --2- 172.21.15.90:0/760720091 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc9a406f710 0x7fc9a40b0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:39.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.958+0000 7fc9ac600640 1 -- 172.21.15.90:0/760720091 >> 172.21.15.90:0/760720091 conn(0x7fc9a406e180 msgr2=0x7fc9a415cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:39.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.958+0000 7fc9ac600640 1 -- 172.21.15.90:0/760720091 shutdown_connections 2024-03-24T01:26:39.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:39.958+0000 7fc9ac600640 1 -- 172.21.15.90:0/760720091 wait complete. 2024-03-24T01:26:39.977 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:26:39.977 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:26:40.027 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:26:40.206 INFO:tasks.ceph.mon.a.smithi090.stderr:2024-03-24T01:26:40.206+0000 7fc6f5600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-03-24T01:26:40.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.227+0000 7f3614e00640 1 Processor -- start 2024-03-24T01:26:40.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.227+0000 7f3614e00640 1 -- start start 2024-03-24T01:26:40.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.229+0000 7f3614e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36101603e0 0x7f361006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:40.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.229+0000 7f3614e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3610145430 0x7f361006b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:40.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.229+0000 7f3614e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36100cd490 con 0x7f36100cc150 2024-03-24T01:26:40.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.229+0000 7f3614e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36100cd270 con 0x7f3610145430 2024-03-24T01:26:40.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.229+0000 7f3614e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f361015f900 con 0x7f36101603e0 2024-03-24T01:26:40.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.229+0000 7f360de00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f36100cc150 0x7f36100ccb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:54914/0 (socket says 172.21.15.90:54914) 2024-03-24T01:26:40.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.229+0000 7f360de00640 1 -- 172.21.15.90:0/937089062 learned_addr learned my addr 172.21.15.90:0/937089062 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:26:40.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.229+0000 7f360ca00640 1 -- 172.21.15.90:0/937089062 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2043763754 0 0) 0x7f36100cd490 con 0x7f36100cc150 2024-03-24T01:26:40.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.230+0000 7f360e800640 1 --2- 172.21.15.90:0/937089062 >> v2:172.21.15.120:3300/0 conn(0x7f3610145430 0x7f361006b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:40.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.230+0000 7f360d400640 1 --2- 172.21.15.90:0/937089062 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36101603e0 0x7f361006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:40.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.230+0000 7f360ca00640 1 -- 172.21.15.90:0/937089062 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35f4003520 con 0x7f36100cc150 2024-03-24T01:26:40.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.230+0000 7f360ca00640 1 -- 172.21.15.90:0/937089062 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3591570023 0 0) 0x7f35f4003520 con 0x7f36100cc150 2024-03-24T01:26:40.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.230+0000 7f360ca00640 1 -- 172.21.15.90:0/937089062 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36101603e0 msgr2=0x7f361006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:40.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.230+0000 7f360ca00640 1 --2- 172.21.15.90:0/937089062 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36101603e0 0x7f361006b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:40.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.230+0000 7f360ca00640 1 -- 172.21.15.90:0/937089062 >> v2:172.21.15.120:3300/0 conn(0x7f3610145430 msgr2=0x7f361006b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:40.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.230+0000 7f360ca00640 1 --2- 172.21.15.90:0/937089062 >> v2:172.21.15.120:3300/0 conn(0x7f3610145430 0x7f361006b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:40.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.230+0000 7f360ca00640 1 -- 172.21.15.90:0/937089062 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f361015fbe0 con 0x7f36100cc150 2024-03-24T01:26:40.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.230+0000 7f360ca00640 1 -- 172.21.15.90:0/937089062 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f3600001670 con 0x7f36100cc150 2024-03-24T01:26:40.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.230+0000 7f360ca00640 1 -- 172.21.15.90:0/937089062 >> v1:172.21.15.90:6789/0 conn(0x7f36100cc150 legacy=0x7f36100ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:40.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.230+0000 7f360ca00640 1 --2- 172.21.15.90:0/937089062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35f40039b0 0x7f35f4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:40.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.230+0000 7f360ca00640 1 --2- 172.21.15.90:0/937089062 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35f4004800 0x7f35f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:40.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.230+0000 7f360ca00640 1 --2- 172.21.15.90:0/937089062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f40051c0 0x7f35f400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:40.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.230+0000 7f360ca00640 1 -- 172.21.15.90:0/937089062 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36100cd270 con 0x7f35f40051c0 2024-03-24T01:26:40.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.230+0000 7f360ca00640 1 -- 172.21.15.90:0/937089062 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361015f900 con 0x7f35f40039b0 2024-03-24T01:26:40.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.230+0000 7f360ca00640 1 -- 172.21.15.90:0/937089062 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f400bf30 con 0x7f35f4004800 2024-03-24T01:26:40.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.230+0000 7f360d400640 1 --2- 172.21.15.90:0/937089062 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f36101603e0 0x7f361006b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:26:40.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.230+0000 7f360e800640 1 --2- 172.21.15.90:0/937089062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f40051c0 0x7f35f400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:40.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.231+0000 7f360de00640 1 --2- 172.21.15.90:0/937089062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35f40039b0 0x7f35f4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:40.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.231+0000 7f360d400640 1 --2- 172.21.15.90:0/937089062 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35f4004800 0x7f35f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:40.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.231+0000 7f360e800640 1 -- 172.21.15.90:0/937089062 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35f4004800 msgr2=0x7f35f4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:40.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.231+0000 7f360e800640 1 --2- 172.21.15.90:0/937089062 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35f4004800 0x7f35f4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:40.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.231+0000 7f360e800640 1 -- 172.21.15.90:0/937089062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35f40039b0 msgr2=0x7f35f4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:40.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.231+0000 7f360e800640 1 --2- 172.21.15.90:0/937089062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35f40039b0 0x7f35f4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:40.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.231+0000 7f360e800640 1 -- 172.21.15.90:0/937089062 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f4003520 con 0x7f35f40051c0 2024-03-24T01:26:40.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.232+0000 7f360e800640 1 --2- 172.21.15.90:0/937089062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f40051c0 0x7f35f400b9c0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f3608009fc0 tx=0x7f35f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e55ee344dbbd7cb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:40.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.232+0000 7f360ca00640 1 -- 172.21.15.90:0/937089062 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3608039070 con 0x7f35f40051c0 2024-03-24T01:26:40.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.232+0000 7f360ca00640 1 -- 172.21.15.90:0/937089062 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f360802cc60 con 0x7f35f40051c0 2024-03-24T01:26:40.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.232+0000 7f360ca00640 1 -- 172.21.15.90:0/937089062 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3608009c10 con 0x7f35f40051c0 2024-03-24T01:26:40.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.233+0000 7f3614e00640 1 -- 172.21.15.90:0/937089062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f40051c0 msgr2=0x7f35f400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:40.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.233+0000 7f3614e00640 1 --2- 172.21.15.90:0/937089062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f40051c0 0x7f35f400b9c0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f3608009fc0 tx=0x7f35f4005830 comp rx=0 tx=0).stop 2024-03-24T01:26:40.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.233+0000 7f360e800640 1 -- 172.21.15.90:0/937089062 reap_dead start 2024-03-24T01:26:40.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.233+0000 7f3614e00640 1 -- 172.21.15.90:0/937089062 shutdown_connections 2024-03-24T01:26:40.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.233+0000 7f3614e00640 1 -- 172.21.15.90:0/937089062 >> 172.21.15.90:0/937089062 conn(0x7f36100b1770 msgr2=0x7f36100b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:40.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.233+0000 7f3614e00640 1 -- 172.21.15.90:0/937089062 shutdown_connections 2024-03-24T01:26:40.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.233+0000 7f3614e00640 1 -- 172.21.15.90:0/937089062 wait complete. 2024-03-24T01:26:40.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.235+0000 7f3614e00640 1 Processor -- start 2024-03-24T01:26:40.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.235+0000 7f3614e00640 1 -- start start 2024-03-24T01:26:40.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.236+0000 7f3614e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f40051c0 0x7f36100ac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:40.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.236+0000 7f3614e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3610139fc0 0x7f36100ae010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:40.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.236+0000 7f3614e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f361015a890 0x7f361015e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:40.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.236+0000 7f3614e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361006b240 con 0x7f35f40051c0 2024-03-24T01:26:40.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.236+0000 7f3614e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361006b0c0 con 0x7f361015a890 2024-03-24T01:26:40.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.236+0000 7f3614e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f361006b3c0 con 0x7f3610139fc0 2024-03-24T01:26:40.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.236+0000 7f360de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f40051c0 0x7f36100ac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:40.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.236+0000 7f360de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f40051c0 0x7f36100ac8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53222/0 (socket says 172.21.15.90:53222) 2024-03-24T01:26:40.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.236+0000 7f360de00640 1 -- 172.21.15.90:0/229224179 learned_addr learned my addr 172.21.15.90:0/229224179 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:40.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.237+0000 7f360d400640 1 --2- 172.21.15.90:0/229224179 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3610139fc0 0x7f36100ae010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:40.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.237+0000 7f360de00640 1 -- 172.21.15.90:0/229224179 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3610139fc0 msgr2=0x7f36100ae010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:40.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.237+0000 7f360de00640 1 --2- 172.21.15.90:0/229224179 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3610139fc0 0x7f36100ae010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:40.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.237+0000 7f360de00640 1 -- 172.21.15.90:0/229224179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f361015a890 msgr2=0x7f361015e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:40.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.237+0000 7f360e800640 1 --2- 172.21.15.90:0/229224179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f361015a890 0x7f361015e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:40.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.237+0000 7f360de00640 1 --2- 172.21.15.90:0/229224179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f361015a890 0x7f361015e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:40.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.237+0000 7f360de00640 1 -- 172.21.15.90:0/229224179 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f361013cc60 con 0x7f35f40051c0 2024-03-24T01:26:40.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.237+0000 7f360d400640 1 --2- 172.21.15.90:0/229224179 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3610139fc0 0x7f36100ae010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:40.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.237+0000 7f360de00640 1 --2- 172.21.15.90:0/229224179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f40051c0 0x7f36100ac8e0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f360000e8a0 tx=0x7f360000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:40.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.237+0000 7f3606000640 1 -- 172.21.15.90:0/229224179 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360000c8d0 con 0x7f35f40051c0 2024-03-24T01:26:40.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.237+0000 7f3606000640 1 -- 172.21.15.90:0/229224179 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3600018070 con 0x7f35f40051c0 2024-03-24T01:26:40.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.237+0000 7f3606000640 1 -- 172.21.15.90:0/229224179 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360000ccd0 con 0x7f35f40051c0 2024-03-24T01:26:40.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.237+0000 7f3614e00640 1 -- 172.21.15.90:0/229224179 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36100acf70 con 0x7f35f40051c0 2024-03-24T01:26:40.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.238+0000 7f3614e00640 1 -- 172.21.15.90:0/229224179 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36100ad530 con 0x7f35f40051c0 2024-03-24T01:26:40.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.239+0000 7f3614e00640 1 -- 172.21.15.90:0/229224179 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3610068060 con 0x7f35f40051c0 2024-03-24T01:26:40.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.239+0000 7f3606000640 1 -- 172.21.15.90:0/229224179 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f360001d460 con 0x7f35f40051c0 2024-03-24T01:26:40.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.240+0000 7f3606000640 1 --2- 172.21.15.90:0/229224179 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f35f006bbd0 0x7f35f006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:40.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.240+0000 7f3606000640 1 -- 172.21.15.90:0/229224179 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f360008acc0 con 0x7f35f40051c0 2024-03-24T01:26:40.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.241+0000 7f360d400640 1 --2- 172.21.15.90:0/229224179 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f35f006bbd0 0x7f35f006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:40.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.242+0000 7f360d400640 1 --2- 172.21.15.90:0/229224179 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f35f006bbd0 0x7f35f006e090 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f361013b620 tx=0x7f35fc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:40.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.245+0000 7f3606000640 1 -- 172.21.15.90:0/229224179 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3600091080 con 0x7f35f40051c0 2024-03-24T01:26:40.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.456+0000 7f3614e00640 1 -- 172.21.15.90:0/229224179 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36100cbcc0 con 0x7f35f40051c0 2024-03-24T01:26:40.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.457+0000 7f3606000640 1 -- 172.21.15.90:0/229224179 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+17047 (secure 0 0 0) 0x7f3600059530 con 0x7f35f40051c0 2024-03-24T01:26:40.458 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:26:40.458 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":40,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:26:27.648103+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:25:57.398864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-24T01:25:58.451150+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:6835/2166652035":"2024-03-25T01:26:27.648052+0000","172.21.15.90:6834/2166652035":"2024-03-25T01:26:27.648052+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:26:40.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.460+0000 7f3614e00640 1 -- 172.21.15.90:0/229224179 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f35f006bbd0 msgr2=0x7f35f006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:40.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.460+0000 7f3614e00640 1 --2- 172.21.15.90:0/229224179 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f35f006bbd0 0x7f35f006e090 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f361013b620 tx=0x7f35fc008040 comp rx=0 tx=0).stop 2024-03-24T01:26:40.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.460+0000 7f3614e00640 1 -- 172.21.15.90:0/229224179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f40051c0 msgr2=0x7f36100ac8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:40.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.461+0000 7f3614e00640 1 --2- 172.21.15.90:0/229224179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f40051c0 0x7f36100ac8e0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f360000e8a0 tx=0x7f360000ed60 comp rx=0 tx=0).stop 2024-03-24T01:26:40.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.461+0000 7f3614e00640 1 -- 172.21.15.90:0/229224179 shutdown_connections 2024-03-24T01:26:40.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.461+0000 7f3614e00640 1 --2- 172.21.15.90:0/229224179 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f35f006bbd0 0x7f35f006e090 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:40.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.461+0000 7f3614e00640 1 --2- 172.21.15.90:0/229224179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f361015a890 0x7f361015e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:40.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.461+0000 7f3614e00640 1 --2- 172.21.15.90:0/229224179 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3610139fc0 0x7f36100ae010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:40.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.461+0000 7f3614e00640 1 --2- 172.21.15.90:0/229224179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35f40051c0 0x7f36100ac8e0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:40.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.461+0000 7f3614e00640 1 -- 172.21.15.90:0/229224179 >> 172.21.15.90:0/229224179 conn(0x7f36100b1770 msgr2=0x7f36100ceb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:40.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.461+0000 7f3614e00640 1 -- 172.21.15.90:0/229224179 shutdown_connections 2024-03-24T01:26:40.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.461+0000 7f3614e00640 1 -- 172.21.15.90:0/229224179 wait complete. 2024-03-24T01:26:40.481 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T01:25:57.398864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T01:25:58.451150+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T01:26:40.482 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:26:40.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.675+0000 7f7993000640 1 Processor -- start 2024-03-24T01:26:40.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.675+0000 7f7993000640 1 -- start start 2024-03-24T01:26:40.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.676+0000 7f7993000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f798c1445b0 0x7f798c0cfb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:40.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.676+0000 7f7993000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f798c144e10 0x7f798c06e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:40.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.676+0000 7f7993000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f798c0d00d0 con 0x7f798c144990 2024-03-24T01:26:40.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.676+0000 7f7993000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f798c0cca70 con 0x7f798c144e10 2024-03-24T01:26:40.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.676+0000 7f7993000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f798c0cf700 con 0x7f798c1445b0 2024-03-24T01:26:40.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.676+0000 7f7992600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f798c144e10 0x7f798c06e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:40.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.677+0000 7f7991200640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f798c144990 0x7f798c06de50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:54916/0 (socket says 172.21.15.90:54916) 2024-03-24T01:26:40.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.677+0000 7f7991200640 1 -- 172.21.15.90:0/1149302175 learned_addr learned my addr 172.21.15.90:0/1149302175 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:26:40.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.677+0000 7f7991c00640 1 --2- 172.21.15.90:0/1149302175 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f798c1445b0 0x7f798c0cfb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:40.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.677+0000 7f7992600640 1 -- 172.21.15.90:0/1149302175 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f798c1445b0 msgr2=0x7f798c0cfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:40.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.677+0000 7f7992600640 1 --2- 172.21.15.90:0/1149302175 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f798c1445b0 0x7f798c0cfb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:40.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.677+0000 7f7992600640 1 -- 172.21.15.90:0/1149302175 >> v1:172.21.15.90:6789/0 conn(0x7f798c144990 legacy=0x7f798c06de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:40.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.677+0000 7f7992600640 1 -- 172.21.15.90:0/1149302175 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f798c0cccc0 con 0x7f798c144e10 2024-03-24T01:26:40.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.679+0000 7f7992600640 1 --2- 172.21.15.90:0/1149302175 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f798c144e10 0x7f798c06e560 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f7984009dc0 tx=0x7f798402bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb60f72dc886cab1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:40.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.679+0000 7f798be00640 1 -- 172.21.15.90:0/1149302175 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7984039070 con 0x7f798c144e10 2024-03-24T01:26:40.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.679+0000 7f798be00640 1 -- 172.21.15.90:0/1149302175 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f798402cba0 con 0x7f798c144e10 2024-03-24T01:26:40.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.679+0000 7f798be00640 1 -- 172.21.15.90:0/1149302175 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7984034440 con 0x7f798c144e10 2024-03-24T01:26:40.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.680+0000 7f7993000640 1 -- 172.21.15.90:0/1149302175 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f798c144e10 msgr2=0x7f798c06e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:40.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.680+0000 7f7993000640 1 --2- 172.21.15.90:0/1149302175 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f798c144e10 0x7f798c06e560 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f7984009dc0 tx=0x7f798402bdf0 comp rx=0 tx=0).stop 2024-03-24T01:26:40.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.680+0000 7f7993000640 1 -- 172.21.15.90:0/1149302175 shutdown_connections 2024-03-24T01:26:40.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.680+0000 7f7993000640 1 --2- 172.21.15.90:0/1149302175 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f798c144e10 0x7f798c06e560 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:40.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.680+0000 7f7993000640 1 --2- 172.21.15.90:0/1149302175 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f798c1445b0 0x7f798c0cfb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:40.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.680+0000 7f7993000640 1 -- 172.21.15.90:0/1149302175 >> 172.21.15.90:0/1149302175 conn(0x7f798c06b0c0 msgr2=0x7f798c06b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:40.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.680+0000 7f7993000640 1 -- 172.21.15.90:0/1149302175 shutdown_connections 2024-03-24T01:26:40.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.680+0000 7f7993000640 1 -- 172.21.15.90:0/1149302175 wait complete. 2024-03-24T01:26:40.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.682+0000 7f7993000640 1 Processor -- start 2024-03-24T01:26:40.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.682+0000 7f7993000640 1 -- start start 2024-03-24T01:26:40.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.683+0000 7f7993000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f798c1445b0 0x7f798c0a8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:40.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.683+0000 7f7993000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f798c144990 0x7f798c13e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:40.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.683+0000 7f7993000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f798c144e10 0x7f798c1f42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:40.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.683+0000 7f7993000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798c0cd280 con 0x7f798c1445b0 2024-03-24T01:26:40.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.683+0000 7f7993000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798c0cf390 con 0x7f798c144e10 2024-03-24T01:26:40.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.683+0000 7f7993000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f798c15f540 con 0x7f798c144990 2024-03-24T01:26:40.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.683+0000 7f7991c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f798c1445b0 0x7f798c0a8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:40.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.683+0000 7f7991c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f798c1445b0 0x7f798c0a8e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53234/0 (socket says 172.21.15.90:53234) 2024-03-24T01:26:40.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.683+0000 7f7991c00640 1 -- 172.21.15.90:0/1065046013 learned_addr learned my addr 172.21.15.90:0/1065046013 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:40.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.684+0000 7f7991200640 1 --2- 172.21.15.90:0/1065046013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f798c144990 0x7f798c13e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:40.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.684+0000 7f7992600640 1 --2- 172.21.15.90:0/1065046013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f798c144e10 0x7f798c1f42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:40.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.684+0000 7f7991c00640 1 -- 172.21.15.90:0/1065046013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f798c144990 msgr2=0x7f798c13e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:40.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.684+0000 7f7991c00640 1 --2- 172.21.15.90:0/1065046013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f798c144990 0x7f798c13e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:40.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.684+0000 7f7991c00640 1 -- 172.21.15.90:0/1065046013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f798c144e10 msgr2=0x7f798c1f42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:40.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.684+0000 7f7991c00640 1 --2- 172.21.15.90:0/1065046013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f798c144e10 0x7f798c1f42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:40.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.684+0000 7f7991c00640 1 -- 172.21.15.90:0/1065046013 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f798c0a8bc0 con 0x7f798c1445b0 2024-03-24T01:26:40.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.684+0000 7f7991c00640 1 --2- 172.21.15.90:0/1065046013 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f798c1445b0 0x7f798c0a8e60 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f798000c7f0 tx=0x7f798000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:40.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.684+0000 7f7989600640 1 -- 172.21.15.90:0/1065046013 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7980015070 con 0x7f798c1445b0 2024-03-24T01:26:40.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.685+0000 7f7989600640 1 -- 172.21.15.90:0/1065046013 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7980004240 con 0x7f798c1445b0 2024-03-24T01:26:40.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.685+0000 7f7989600640 1 -- 172.21.15.90:0/1065046013 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7980007620 con 0x7f798c1445b0 2024-03-24T01:26:40.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.685+0000 7f7993000640 1 -- 172.21.15.90:0/1065046013 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f798c0a8600 con 0x7f798c1445b0 2024-03-24T01:26:40.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.685+0000 7f7993000640 1 -- 172.21.15.90:0/1065046013 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f798c1f3af0 con 0x7f798c1445b0 2024-03-24T01:26:40.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.686+0000 7f7993000640 1 -- 172.21.15.90:0/1065046013 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f798c04f400 con 0x7f798c1445b0 2024-03-24T01:26:40.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.688+0000 7f7989600640 1 -- 172.21.15.90:0/1065046013 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f79800079d0 con 0x7f798c1445b0 2024-03-24T01:26:40.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.689+0000 7f7989600640 1 --2- 172.21.15.90:0/1065046013 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f796c06bbd0 0x7f796c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:40.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.689+0000 7f7989600640 1 -- 172.21.15.90:0/1065046013 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f798008b060 con 0x7f798c1445b0 2024-03-24T01:26:40.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.689+0000 7f7991200640 1 --2- 172.21.15.90:0/1065046013 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f796c06bbd0 0x7f796c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:40.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.690+0000 7f7991200640 1 --2- 172.21.15.90:0/1065046013 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f796c06bbd0 0x7f796c06e090 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f798c0cf1d0 tx=0x7f7978009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:40.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.693+0000 7f7989600640 1 -- 172.21.15.90:0/1065046013 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7980059960 con 0x7f798c1445b0 2024-03-24T01:26:40.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.905+0000 7f7993000640 1 -- 172.21.15.90:0/1065046013 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f798c0b43f0 con 0x7f798c1445b0 2024-03-24T01:26:40.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.906+0000 7f7989600640 1 -- 172.21.15.90:0/1065046013 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7980059260 con 0x7f798c1445b0 2024-03-24T01:26:40.906 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:26:40.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.908+0000 7f7993000640 1 -- 172.21.15.90:0/1065046013 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f796c06bbd0 msgr2=0x7f796c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:40.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.908+0000 7f7993000640 1 --2- 172.21.15.90:0/1065046013 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f796c06bbd0 0x7f796c06e090 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f798c0cf1d0 tx=0x7f7978009040 comp rx=0 tx=0).stop 2024-03-24T01:26:40.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.908+0000 7f7993000640 1 -- 172.21.15.90:0/1065046013 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f798c1445b0 msgr2=0x7f798c0a8e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:40.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.908+0000 7f7993000640 1 --2- 172.21.15.90:0/1065046013 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f798c1445b0 0x7f798c0a8e60 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f798000c7f0 tx=0x7f798000ccb0 comp rx=0 tx=0).stop 2024-03-24T01:26:40.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.908+0000 7f7993000640 1 -- 172.21.15.90:0/1065046013 shutdown_connections 2024-03-24T01:26:40.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.908+0000 7f7993000640 1 --2- 172.21.15.90:0/1065046013 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f796c06bbd0 0x7f796c06e090 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:40.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.908+0000 7f7993000640 1 --2- 172.21.15.90:0/1065046013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f798c144e10 0x7f798c1f42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:40.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.908+0000 7f7993000640 1 --2- 172.21.15.90:0/1065046013 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f798c144990 0x7f798c13e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:40.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.908+0000 7f7993000640 1 --2- 172.21.15.90:0/1065046013 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f798c1445b0 0x7f798c0a8e60 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:40.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.908+0000 7f7993000640 1 -- 172.21.15.90:0/1065046013 >> 172.21.15.90:0/1065046013 conn(0x7f798c06b0c0 msgr2=0x7f798c06b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:40.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.908+0000 7f7993000640 1 -- 172.21.15.90:0/1065046013 shutdown_connections 2024-03-24T01:26:40.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:40.908+0000 7f7993000640 1 -- 172.21.15.90:0/1065046013 wait complete. 2024-03-24T01:26:40.926 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T01:26:41.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.125+0000 7f3f91a00640 1 Processor -- start 2024-03-24T01:26:41.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.125+0000 7f3f91a00640 1 -- start start 2024-03-24T01:26:41.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.127+0000 7f3f91a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3f8c0ce2f0 0x7f3f8c0cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:41.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.127+0000 7f3f91a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f8c0cf980 0x7f3f8c0cfe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:41.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.127+0000 7f3f91a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f8c0ccae0 con 0x7f3f8c145410 2024-03-24T01:26:41.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.127+0000 7f3f91a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f8c0cc8a0 con 0x7f3f8c0ce2f0 2024-03-24T01:26:41.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.127+0000 7f3f91a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8c0ccdc0 con 0x7f3f8c0cf980 2024-03-24T01:26:41.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.127+0000 7f3f8be00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f3f8c145410 0x7f3f8c0cc190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:54930/0 (socket says 172.21.15.90:54930) 2024-03-24T01:26:41.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.127+0000 7f3f8be00640 1 -- 172.21.15.90:0/3923748526 learned_addr learned my addr 172.21.15.90:0/3923748526 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:26:41.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.128+0000 7f3f8b400640 1 --2- 172.21.15.90:0/3923748526 >> v2:172.21.15.120:3300/0 conn(0x7f3f8c0ce2f0 0x7f3f8c0cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:41.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.128+0000 7f3f91000640 1 --2- 172.21.15.90:0/3923748526 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f8c0cf980 0x7f3f8c0cfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:41.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.128+0000 7f3f8aa00640 1 -- 172.21.15.90:0/3923748526 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2617382294 0 0) 0x7f3f8c0ccae0 con 0x7f3f8c145410 2024-03-24T01:26:41.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.128+0000 7f3f8aa00640 1 -- 172.21.15.90:0/3923748526 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f70003520 con 0x7f3f8c145410 2024-03-24T01:26:41.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.128+0000 7f3f8aa00640 1 -- 172.21.15.90:0/3923748526 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2022374138 0 0) 0x7f3f70003520 con 0x7f3f8c145410 2024-03-24T01:26:41.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.128+0000 7f3f8aa00640 1 -- 172.21.15.90:0/3923748526 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f8c0cf980 msgr2=0x7f3f8c0cfe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:41.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.128+0000 7f3f8aa00640 1 --2- 172.21.15.90:0/3923748526 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f8c0cf980 0x7f3f8c0cfe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:41.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.128+0000 7f3f8aa00640 1 -- 172.21.15.90:0/3923748526 >> v2:172.21.15.120:3300/0 conn(0x7f3f8c0ce2f0 msgr2=0x7f3f8c0cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:41.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.128+0000 7f3f8aa00640 1 --2- 172.21.15.90:0/3923748526 >> v2:172.21.15.120:3300/0 conn(0x7f3f8c0ce2f0 0x7f3f8c0cdb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:41.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.128+0000 7f3f8aa00640 1 -- 172.21.15.90:0/3923748526 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f8c0cd0a0 con 0x7f3f8c145410 2024-03-24T01:26:41.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.128+0000 7f3f8aa00640 1 -- 172.21.15.90:0/3923748526 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f3f7c001670 con 0x7f3f8c145410 2024-03-24T01:26:41.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.128+0000 7f3f8aa00640 1 -- 172.21.15.90:0/3923748526 >> v1:172.21.15.90:6789/0 conn(0x7f3f8c145410 legacy=0x7f3f8c0cc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:41.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.128+0000 7f3f8aa00640 1 --2- 172.21.15.90:0/3923748526 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f700039b0 0x7f3f70003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:41.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.128+0000 7f3f8aa00640 1 --2- 172.21.15.90:0/3923748526 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f8c0cf980 0x7f3f70004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:41.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.128+0000 7f3f8aa00640 1 --2- 172.21.15.90:0/3923748526 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f70004d40 0x7f3f700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:41.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.128+0000 7f3f8aa00640 1 -- 172.21.15.90:0/3923748526 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8c0cc8a0 con 0x7f3f700039b0 2024-03-24T01:26:41.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.128+0000 7f3f8aa00640 1 -- 172.21.15.90:0/3923748526 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8c0ccdc0 con 0x7f3f70004d40 2024-03-24T01:26:41.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.128+0000 7f3f8aa00640 1 -- 172.21.15.90:0/3923748526 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7000ba70 con 0x7f3f8c0cf980 2024-03-24T01:26:41.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.128+0000 7f3f8b400640 1 --2- 172.21.15.90:0/3923748526 >> v2:172.21.15.120:3300/0 conn(0x7f3f8c0ce2f0 0x7f3f8c0cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:26:41.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.129+0000 7f3f8be00640 1 --2- 172.21.15.90:0/3923748526 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f700039b0 0x7f3f70003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:41.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.129+0000 7f3f91000640 1 --2- 172.21.15.90:0/3923748526 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f70004d40 0x7f3f700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:41.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.129+0000 7f3f8be00640 1 -- 172.21.15.90:0/3923748526 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f8c0cf980 msgr2=0x7f3f70004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:41.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.129+0000 7f3f8be00640 1 --2- 172.21.15.90:0/3923748526 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f8c0cf980 0x7f3f70004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:41.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.129+0000 7f3f8be00640 1 -- 172.21.15.90:0/3923748526 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f70004d40 msgr2=0x7f3f700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:41.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.129+0000 7f3f8be00640 1 --2- 172.21.15.90:0/3923748526 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f70004d40 0x7f3f700051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:41.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.129+0000 7f3f8be00640 1 -- 172.21.15.90:0/3923748526 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f70003520 con 0x7f3f700039b0 2024-03-24T01:26:41.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.130+0000 7f3f8be00640 1 --2- 172.21.15.90:0/3923748526 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f700039b0 0x7f3f70003dd0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c00c240 tx=0x7f3f70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7f472f0637da89d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:41.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.130+0000 7f3f8aa00640 1 -- 172.21.15.90:0/3923748526 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c007d50 con 0x7f3f700039b0 2024-03-24T01:26:41.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.131+0000 7f3f8aa00640 1 -- 172.21.15.90:0/3923748526 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f7c030070 con 0x7f3f700039b0 2024-03-24T01:26:41.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.131+0000 7f3f8aa00640 1 -- 172.21.15.90:0/3923748526 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c02cd80 con 0x7f3f700039b0 2024-03-24T01:26:41.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.131+0000 7f3f91a00640 1 -- 172.21.15.90:0/3923748526 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f700039b0 msgr2=0x7f3f70003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:41.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.131+0000 7f3f91a00640 1 --2- 172.21.15.90:0/3923748526 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f700039b0 0x7f3f70003dd0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c00c240 tx=0x7f3f70005830 comp rx=0 tx=0).stop 2024-03-24T01:26:41.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.131+0000 7f3f91000640 1 -- 172.21.15.90:0/3923748526 reap_dead start 2024-03-24T01:26:41.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.131+0000 7f3f91a00640 1 -- 172.21.15.90:0/3923748526 shutdown_connections 2024-03-24T01:26:41.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.131+0000 7f3f91a00640 1 -- 172.21.15.90:0/3923748526 >> 172.21.15.90:0/3923748526 conn(0x7f3f8c0b23d0 msgr2=0x7f3f8c0b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:41.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.131+0000 7f3f91a00640 1 -- 172.21.15.90:0/3923748526 shutdown_connections 2024-03-24T01:26:41.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.131+0000 7f3f91a00640 1 -- 172.21.15.90:0/3923748526 wait complete. 2024-03-24T01:26:41.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.133+0000 7f3f91a00640 1 Processor -- start 2024-03-24T01:26:41.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.133+0000 7f3f91a00640 1 -- start start 2024-03-24T01:26:41.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.134+0000 7f3f91a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f700039b0 0x7f3f8c15ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:41.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.134+0000 7f3f91a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f8c0b6050 0x7f3f8c13ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:41.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.134+0000 7f3f91a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f8c0b34c0 0x7f3f8c0b3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:41.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.134+0000 7f3f91a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8c0cfe10 con 0x7f3f8c0b6050 2024-03-24T01:26:41.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.134+0000 7f3f91a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8c0cd660 con 0x7f3f700039b0 2024-03-24T01:26:41.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.134+0000 7f3f91a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8c0cff90 con 0x7f3f8c0b34c0 2024-03-24T01:26:41.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.134+0000 7f3f8b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f8c0b6050 0x7f3f8c13ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:41.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.134+0000 7f3f8b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f8c0b6050 0x7f3f8c13ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53246/0 (socket says 172.21.15.90:53246) 2024-03-24T01:26:41.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.134+0000 7f3f8b400640 1 -- 172.21.15.90:0/1908808384 learned_addr learned my addr 172.21.15.90:0/1908808384 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:41.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.135+0000 7f3f91000640 1 --2- 172.21.15.90:0/1908808384 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f8c0b34c0 0x7f3f8c0b3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:41.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.135+0000 7f3f8be00640 1 --2- 172.21.15.90:0/1908808384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f700039b0 0x7f3f8c15ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:41.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.135+0000 7f3f8b400640 1 -- 172.21.15.90:0/1908808384 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f8c0b34c0 msgr2=0x7f3f8c0b3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:41.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.135+0000 7f3f8b400640 1 --2- 172.21.15.90:0/1908808384 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f8c0b34c0 0x7f3f8c0b3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:41.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.135+0000 7f3f8b400640 1 -- 172.21.15.90:0/1908808384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f700039b0 msgr2=0x7f3f8c15ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:41.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.135+0000 7f3f8b400640 1 --2- 172.21.15.90:0/1908808384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f700039b0 0x7f3f8c15ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:41.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.135+0000 7f3f8b400640 1 -- 172.21.15.90:0/1908808384 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f8c15b2d0 con 0x7f3f8c0b6050 2024-03-24T01:26:41.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.135+0000 7f3f8b400640 1 --2- 172.21.15.90:0/1908808384 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f8c0b6050 0x7f3f8c13ee40 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f3f7800e810 tx=0x7f3f7800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:41.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.135+0000 7f3f83e00640 1 -- 172.21.15.90:0/1908808384 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7800f040 con 0x7f3f8c0b6050 2024-03-24T01:26:41.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.135+0000 7f3f83e00640 1 -- 172.21.15.90:0/1908808384 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f78004230 con 0x7f3f8c0b6050 2024-03-24T01:26:41.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.135+0000 7f3f83e00640 1 -- 172.21.15.90:0/1908808384 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7800c6d0 con 0x7f3f8c0b6050 2024-03-24T01:26:41.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.136+0000 7f3f91a00640 1 -- 172.21.15.90:0/1908808384 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f8c15ea90 con 0x7f3f8c0b6050 2024-03-24T01:26:41.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.136+0000 7f3f91a00640 1 -- 172.21.15.90:0/1908808384 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f8c13e920 con 0x7f3f8c0b6050 2024-03-24T01:26:41.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.137+0000 7f3f91a00640 1 -- 172.21.15.90:0/1908808384 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f8c04f400 con 0x7f3f8c0b6050 2024-03-24T01:26:41.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.138+0000 7f3f83e00640 1 -- 172.21.15.90:0/1908808384 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3f7800c870 con 0x7f3f8c0b6050 2024-03-24T01:26:41.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.138+0000 7f3f83e00640 1 --2- 172.21.15.90:0/1908808384 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3f6c06bbd0 0x7f3f6c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:41.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.139+0000 7f3f83e00640 1 -- 172.21.15.90:0/1908808384 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f3f7808a960 con 0x7f3f8c0b6050 2024-03-24T01:26:41.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.139+0000 7f3f8be00640 1 --2- 172.21.15.90:0/1908808384 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3f6c06bbd0 0x7f3f6c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:41.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.140+0000 7f3f8be00640 1 --2- 172.21.15.90:0/1908808384 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3f6c06bbd0 0x7f3f6c06e090 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c034000 tx=0x7f3f7c039040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:41.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.143+0000 7f3f83e00640 1 -- 172.21.15.90:0/1908808384 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3f78017020 con 0x7f3f8c0b6050 2024-03-24T01:26:41.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.410+0000 7f3f91a00640 1 -- 172.21.15.90:0/1908808384 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3f8c144720 con 0x7f3f8c0b6050 2024-03-24T01:26:41.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.411+0000 7f3f83e00640 1 -- 172.21.15.90:0/1908808384 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v40) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3f78059260 con 0x7f3f8c0b6050 2024-03-24T01:26:41.411 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:26:41.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.414+0000 7f3f91a00640 1 -- 172.21.15.90:0/1908808384 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3f6c06bbd0 msgr2=0x7f3f6c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:41.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.414+0000 7f3f91a00640 1 --2- 172.21.15.90:0/1908808384 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3f6c06bbd0 0x7f3f6c06e090 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c034000 tx=0x7f3f7c039040 comp rx=0 tx=0).stop 2024-03-24T01:26:41.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.414+0000 7f3f91a00640 1 -- 172.21.15.90:0/1908808384 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f8c0b6050 msgr2=0x7f3f8c13ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:41.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.414+0000 7f3f91a00640 1 --2- 172.21.15.90:0/1908808384 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f8c0b6050 0x7f3f8c13ee40 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f3f7800e810 tx=0x7f3f7800ecd0 comp rx=0 tx=0).stop 2024-03-24T01:26:41.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.414+0000 7f3f91a00640 1 -- 172.21.15.90:0/1908808384 shutdown_connections 2024-03-24T01:26:41.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.414+0000 7f3f91a00640 1 --2- 172.21.15.90:0/1908808384 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3f6c06bbd0 0x7f3f6c06e090 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:41.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.414+0000 7f3f91a00640 1 --2- 172.21.15.90:0/1908808384 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f8c0b6050 0x7f3f8c13ee40 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:41.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.414+0000 7f3f91a00640 1 --2- 172.21.15.90:0/1908808384 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3f8c0b34c0 0x7f3f8c0b3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:41.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.414+0000 7f3f91a00640 1 --2- 172.21.15.90:0/1908808384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3f700039b0 0x7f3f8c15ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:41.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.414+0000 7f3f91a00640 1 -- 172.21.15.90:0/1908808384 >> 172.21.15.90:0/1908808384 conn(0x7f3f8c0b23d0 msgr2=0x7f3f8c06b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:41.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.414+0000 7f3f91a00640 1 -- 172.21.15.90:0/1908808384 shutdown_connections 2024-03-24T01:26:41.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.414+0000 7f3f91a00640 1 -- 172.21.15.90:0/1908808384 wait complete. 2024-03-24T01:26:41.432 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T01:26:41.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.632+0000 7fc94d400640 1 Processor -- start 2024-03-24T01:26:41.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.633+0000 7fc94d400640 1 -- start start 2024-03-24T01:26:41.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.634+0000 7fc94d400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc9480d0270 0x7fc948168f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:41.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.634+0000 7fc94d400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc94815e4c0 con 0x7fc94815e0e0 2024-03-24T01:26:41.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.634+0000 7fc94d400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc94806d330 con 0x7fc94806cbe0 2024-03-24T01:26:41.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.634+0000 7fc94d400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9480b1350 con 0x7fc9480d0270 2024-03-24T01:26:41.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.634+0000 7fc946800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc94815e0e0 0x7fc9480b2080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:54936/0 (socket says 172.21.15.90:54936) 2024-03-24T01:26:41.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.634+0000 7fc946800640 1 -- 172.21.15.90:0/1905691036 learned_addr learned my addr 172.21.15.90:0/1905691036 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:26:41.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.635+0000 7fc947200640 1 --2- 172.21.15.90:0/1905691036 >> v2:172.21.15.90:3300/0 conn(0x7fc9480d0270 0x7fc948168f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:41.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.635+0000 7fc945e00640 1 -- 172.21.15.90:0/1905691036 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1536944414 0 0) 0x7fc94815e4c0 con 0x7fc94815e0e0 2024-03-24T01:26:41.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.635+0000 7fc945e00640 1 -- 172.21.15.90:0/1905691036 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc92c003520 con 0x7fc94815e0e0 2024-03-24T01:26:41.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.635+0000 7fc947200640 1 -- 172.21.15.90:0/1905691036 >> v1:172.21.15.120:6789/0 conn(0x7fc94806cbe0 legacy=0x7fc9480cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:41.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.635+0000 7fc947200640 1 -- 172.21.15.90:0/1905691036 >> v1:172.21.15.90:6789/0 conn(0x7fc94815e0e0 legacy=0x7fc9480b2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:41.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.635+0000 7fc947200640 1 -- 172.21.15.90:0/1905691036 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9480b15a0 con 0x7fc9480d0270 2024-03-24T01:26:41.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.636+0000 7fc947200640 1 --2- 172.21.15.90:0/1905691036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9480d0270 0x7fc948168f10 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fc940009d90 tx=0x7fc94002bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d40e3895931445b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:41.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.637+0000 7fc945e00640 1 -- 172.21.15.90:0/1905691036 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc940039070 con 0x7fc9480d0270 2024-03-24T01:26:41.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.637+0000 7fc945e00640 1 -- 172.21.15.90:0/1905691036 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc94002ccf0 con 0x7fc9480d0270 2024-03-24T01:26:41.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.637+0000 7fc945e00640 1 -- 172.21.15.90:0/1905691036 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc940034660 con 0x7fc9480d0270 2024-03-24T01:26:41.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.637+0000 7fc94d400640 1 -- 172.21.15.90:0/1905691036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9480d0270 msgr2=0x7fc948168f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:41.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.637+0000 7fc94d400640 1 --2- 172.21.15.90:0/1905691036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9480d0270 0x7fc948168f10 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fc940009d90 tx=0x7fc94002bf40 comp rx=0 tx=0).stop 2024-03-24T01:26:41.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.637+0000 7fc94d400640 1 -- 172.21.15.90:0/1905691036 shutdown_connections 2024-03-24T01:26:41.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.637+0000 7fc94d400640 1 --2- 172.21.15.90:0/1905691036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9480d0270 0x7fc948168f10 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:41.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.637+0000 7fc94d400640 1 -- 172.21.15.90:0/1905691036 >> 172.21.15.90:0/1905691036 conn(0x7fc9480b34e0 msgr2=0x7fc9480b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:41.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.638+0000 7fc94d400640 1 -- 172.21.15.90:0/1905691036 shutdown_connections 2024-03-24T01:26:41.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.638+0000 7fc94d400640 1 -- 172.21.15.90:0/1905691036 wait complete. 2024-03-24T01:26:41.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.639+0000 7fc94d400640 1 Processor -- start 2024-03-24T01:26:41.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.639+0000 7fc94d400640 1 -- start start 2024-03-24T01:26:41.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.640+0000 7fc94d400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc94806cbe0 0x7fc9480c96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:41.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.640+0000 7fc94d400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9480d0270 0x7fc948151e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:41.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.640+0000 7fc94d400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc94815e0e0 0x7fc94815c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:41.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.640+0000 7fc94d400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94806d040 con 0x7fc9480d0270 2024-03-24T01:26:41.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.640+0000 7fc94d400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9480b1b60 con 0x7fc94815e0e0 2024-03-24T01:26:41.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.640+0000 7fc94d400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9480b2790 con 0x7fc94806cbe0 2024-03-24T01:26:41.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.641+0000 7fc93de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9480d0270 0x7fc948151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:41.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.641+0000 7fc93de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9480d0270 0x7fc948151e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53272/0 (socket says 172.21.15.90:53272) 2024-03-24T01:26:41.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.641+0000 7fc93de00640 1 -- 172.21.15.90:0/3661273742 learned_addr learned my addr 172.21.15.90:0/3661273742 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:41.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.641+0000 7fc947200640 1 --2- 172.21.15.90:0/3661273742 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc94815e0e0 0x7fc94815c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:41.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.641+0000 7fc93de00640 1 -- 172.21.15.90:0/3661273742 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc94806cbe0 msgr2=0x7fc9480c96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:41.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.641+0000 7fc93de00640 1 --2- 172.21.15.90:0/3661273742 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc94806cbe0 0x7fc9480c96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:41.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.641+0000 7fc93de00640 1 -- 172.21.15.90:0/3661273742 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc94815e0e0 msgr2=0x7fc94815c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:41.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.641+0000 7fc93de00640 1 --2- 172.21.15.90:0/3661273742 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc94815e0e0 0x7fc94815c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:41.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.641+0000 7fc93de00640 1 -- 172.21.15.90:0/3661273742 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9480c9c10 con 0x7fc9480d0270 2024-03-24T01:26:41.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.641+0000 7fc93de00640 1 --2- 172.21.15.90:0/3661273742 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9480d0270 0x7fc948151e20 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fc93000c480 tx=0x7fc93000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:41.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.642+0000 7fc93f400640 1 -- 172.21.15.90:0/3661273742 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9300052d0 con 0x7fc9480d0270 2024-03-24T01:26:41.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.642+0000 7fc93f400640 1 -- 172.21.15.90:0/3661273742 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc930005470 con 0x7fc9480d0270 2024-03-24T01:26:41.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.642+0000 7fc94d400640 1 -- 172.21.15.90:0/3661273742 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc94815d290 con 0x7fc9480d0270 2024-03-24T01:26:41.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.642+0000 7fc93f400640 1 -- 172.21.15.90:0/3661273742 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc930012870 con 0x7fc9480d0270 2024-03-24T01:26:41.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.642+0000 7fc94d400640 1 -- 172.21.15.90:0/3661273742 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9480c8ed0 con 0x7fc9480d0270 2024-03-24T01:26:41.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.643+0000 7fc94d400640 1 -- 172.21.15.90:0/3661273742 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc94804f400 con 0x7fc9480d0270 2024-03-24T01:26:41.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.644+0000 7fc93f400640 1 -- 172.21.15.90:0/3661273742 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc930012a10 con 0x7fc9480d0270 2024-03-24T01:26:41.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.645+0000 7fc93f400640 1 --2- 172.21.15.90:0/3661273742 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc92406bbd0 0x7fc92406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:41.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.645+0000 7fc93f400640 1 -- 172.21.15.90:0/3661273742 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fc93008a8f0 con 0x7fc9480d0270 2024-03-24T01:26:41.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.646+0000 7fc946800640 1 --2- 172.21.15.90:0/3661273742 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc92406bbd0 0x7fc92406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:41.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.647+0000 7fc946800640 1 --2- 172.21.15.90:0/3661273742 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc92406bbd0 0x7fc92406e090 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fc938001570 tx=0x7fc938007930 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:41.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.654+0000 7fc93f400640 1 -- 172.21.15.90:0/3661273742 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc9300591f0 con 0x7fc9480d0270 2024-03-24T01:26:41.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.850+0000 7fc94d400640 1 -- 172.21.15.90:0/3661273742 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc9480b4310 con 0x7fc9480d0270 2024-03-24T01:26:41.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.850+0000 7fc93f400640 1 -- 172.21.15.90:0/3661273742 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v40) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc930058af0 con 0x7fc9480d0270 2024-03-24T01:26:41.851 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-24T01:26:41.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.853+0000 7fc94d400640 1 -- 172.21.15.90:0/3661273742 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc92406bbd0 msgr2=0x7fc92406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:41.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.853+0000 7fc94d400640 1 --2- 172.21.15.90:0/3661273742 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc92406bbd0 0x7fc92406e090 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fc938001570 tx=0x7fc938007930 comp rx=0 tx=0).stop 2024-03-24T01:26:41.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.853+0000 7fc94d400640 1 -- 172.21.15.90:0/3661273742 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9480d0270 msgr2=0x7fc948151e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:41.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.853+0000 7fc94d400640 1 --2- 172.21.15.90:0/3661273742 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9480d0270 0x7fc948151e20 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fc93000c480 tx=0x7fc93000c940 comp rx=0 tx=0).stop 2024-03-24T01:26:41.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.853+0000 7fc94d400640 1 -- 172.21.15.90:0/3661273742 shutdown_connections 2024-03-24T01:26:41.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.853+0000 7fc94d400640 1 --2- 172.21.15.90:0/3661273742 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc92406bbd0 0x7fc92406e090 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:41.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.853+0000 7fc94d400640 1 --2- 172.21.15.90:0/3661273742 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc9480d0270 0x7fc948151e20 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:41.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.853+0000 7fc94d400640 1 --2- 172.21.15.90:0/3661273742 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc94815e0e0 0x7fc94815c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:41.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.853+0000 7fc94d400640 1 --2- 172.21.15.90:0/3661273742 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc94806cbe0 0x7fc9480c96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:41.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.853+0000 7fc94d400640 1 -- 172.21.15.90:0/3661273742 >> 172.21.15.90:0/3661273742 conn(0x7fc9480b34e0 msgr2=0x7fc9480739c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:41.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.853+0000 7fc94d400640 1 -- 172.21.15.90:0/3661273742 shutdown_connections 2024-03-24T01:26:41.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:41.853+0000 7fc94d400640 1 -- 172.21.15.90:0/3661273742 wait complete. 2024-03-24T01:26:41.878 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T01:26:42.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.082+0000 7ffabf400640 1 Processor -- start 2024-03-24T01:26:42.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.082+0000 7ffabf400640 1 -- start start 2024-03-24T01:26:42.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.084+0000 7ffabf400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffab80cf150 0x7ffab80ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:42.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.084+0000 7ffabf400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffab806bca0 con 0x7ffab8159620 2024-03-24T01:26:42.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.084+0000 7ffabf400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffab80b36f0 con 0x7ffab806b7d0 2024-03-24T01:26:42.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.084+0000 7ffabf400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffab80b4470 con 0x7ffab80cf150 2024-03-24T01:26:42.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.084+0000 7ffabca00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ffab8159620 0x7ffab806b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:54950/0 (socket says 172.21.15.90:54950) 2024-03-24T01:26:42.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.084+0000 7ffabca00640 1 -- 172.21.15.90:0/3047219613 learned_addr learned my addr 172.21.15.90:0/3047219613 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:26:42.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.085+0000 7ffabd400640 1 --2- 172.21.15.90:0/3047219613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffab80cf150 0x7ffab80ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:42.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.085+0000 7ffab7e00640 1 -- 172.21.15.90:0/3047219613 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2711577944 0 0) 0x7ffab806bca0 con 0x7ffab8159620 2024-03-24T01:26:42.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.085+0000 7ffab7e00640 1 -- 172.21.15.90:0/3047219613 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa9c003520 con 0x7ffab8159620 2024-03-24T01:26:42.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.085+0000 7ffab7e00640 1 -- 172.21.15.90:0/3047219613 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1699894544 0 0) 0x7ffa9c003520 con 0x7ffab8159620 2024-03-24T01:26:42.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.085+0000 7ffab7e00640 1 -- 172.21.15.90:0/3047219613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffab80cf150 msgr2=0x7ffab80ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:42.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.085+0000 7ffab7e00640 1 --2- 172.21.15.90:0/3047219613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffab80cf150 0x7ffab80ae530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:42.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.085+0000 7ffab7e00640 1 -- 172.21.15.90:0/3047219613 >> v1:172.21.15.120:6789/0 conn(0x7ffab806b7d0 legacy=0x7ffab80cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:42.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.085+0000 7ffab7e00640 1 -- 172.21.15.90:0/3047219613 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab80b3990 con 0x7ffab8159620 2024-03-24T01:26:42.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.085+0000 7ffab7e00640 1 -- 172.21.15.90:0/3047219613 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7ffaa8001670 con 0x7ffab8159620 2024-03-24T01:26:42.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.085+0000 7ffab7e00640 1 -- 172.21.15.90:0/3047219613 >> v1:172.21.15.90:6789/0 conn(0x7ffab8159620 legacy=0x7ffab806b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:42.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.085+0000 7ffab7e00640 1 --2- 172.21.15.90:0/3047219613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa9c0039e0 0x7ffa9c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:42.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.085+0000 7ffab7e00640 1 --2- 172.21.15.90:0/3047219613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffab80cf150 0x7ffa9c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:42.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.085+0000 7ffab7e00640 1 --2- 172.21.15.90:0/3047219613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa9c004d40 0x7ffa9c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:42.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.085+0000 7ffab7e00640 1 -- 172.21.15.90:0/3047219613 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab80b4470 con 0x7ffa9c0039e0 2024-03-24T01:26:42.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.085+0000 7ffab7e00640 1 -- 172.21.15.90:0/3047219613 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9c00baf0 con 0x7ffa9c004d40 2024-03-24T01:26:42.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.085+0000 7ffab7e00640 1 -- 172.21.15.90:0/3047219613 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9c00bcf0 con 0x7ffab80cf150 2024-03-24T01:26:42.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.086+0000 7ffabca00640 1 --2- 172.21.15.90:0/3047219613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa9c0039e0 0x7ffa9c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:42.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.086+0000 7ffabca00640 1 -- 172.21.15.90:0/3047219613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffab80cf150 msgr2=0x7ffa9c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:42.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.086+0000 7ffabca00640 1 --2- 172.21.15.90:0/3047219613 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffab80cf150 0x7ffa9c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:42.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.086+0000 7ffabca00640 1 -- 172.21.15.90:0/3047219613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa9c004d40 msgr2=0x7ffa9c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:42.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.086+0000 7ffabca00640 1 --2- 172.21.15.90:0/3047219613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa9c004d40 0x7ffa9c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:42.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.086+0000 7ffabca00640 1 -- 172.21.15.90:0/3047219613 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa9c003520 con 0x7ffa9c0039e0 2024-03-24T01:26:42.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.087+0000 7ffabca00640 1 --2- 172.21.15.90:0/3047219613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa9c0039e0 0x7ffa9c003e00 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7ffaa800c240 tx=0x7ffa9c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b75e9d116bbc24d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:42.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.087+0000 7ffab7e00640 1 -- 172.21.15.90:0/3047219613 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa8007d10 con 0x7ffa9c0039e0 2024-03-24T01:26:42.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.087+0000 7ffab7e00640 1 -- 172.21.15.90:0/3047219613 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaa8030070 con 0x7ffa9c0039e0 2024-03-24T01:26:42.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.087+0000 7ffab7e00640 1 -- 172.21.15.90:0/3047219613 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa8005d30 con 0x7ffa9c0039e0 2024-03-24T01:26:42.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.088+0000 7ffabf400640 1 -- 172.21.15.90:0/3047219613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa9c0039e0 msgr2=0x7ffa9c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:42.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.088+0000 7ffabf400640 1 --2- 172.21.15.90:0/3047219613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa9c0039e0 0x7ffa9c003e00 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7ffaa800c240 tx=0x7ffa9c005830 comp rx=0 tx=0).stop 2024-03-24T01:26:42.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.088+0000 7ffabd400640 1 -- 172.21.15.90:0/3047219613 reap_dead start 2024-03-24T01:26:42.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.088+0000 7ffabf400640 1 -- 172.21.15.90:0/3047219613 shutdown_connections 2024-03-24T01:26:42.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.088+0000 7ffabf400640 1 -- 172.21.15.90:0/3047219613 >> 172.21.15.90:0/3047219613 conn(0x7ffab80b1770 msgr2=0x7ffab80b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:42.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.088+0000 7ffabf400640 1 -- 172.21.15.90:0/3047219613 shutdown_connections 2024-03-24T01:26:42.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.088+0000 7ffabf400640 1 -- 172.21.15.90:0/3047219613 wait complete. 2024-03-24T01:26:42.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.090+0000 7ffabf400640 1 Processor -- start 2024-03-24T01:26:42.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.090+0000 7ffabf400640 1 -- start start 2024-03-24T01:26:42.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.091+0000 7ffabf400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa9c0039e0 0x7ffab81e5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:42.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.091+0000 7ffabf400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffab8159de0 0x7ffab8143020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:42.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.091+0000 7ffabf400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffab815a240 0x7ffab8143560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:42.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.091+0000 7ffabf400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab8159a00 con 0x7ffab815a240 2024-03-24T01:26:42.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.091+0000 7ffabf400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab80b3f50 con 0x7ffab8159de0 2024-03-24T01:26:42.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.091+0000 7ffabf400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffab815bfe0 con 0x7ffa9c0039e0 2024-03-24T01:26:42.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.091+0000 7ffabd400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffab815a240 0x7ffab8143560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:42.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.091+0000 7ffabd400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffab815a240 0x7ffab8143560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53296/0 (socket says 172.21.15.90:53296) 2024-03-24T01:26:42.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.091+0000 7ffabd400640 1 -- 172.21.15.90:0/2080221929 learned_addr learned my addr 172.21.15.90:0/2080221929 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:42.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.092+0000 7ffaafe00640 1 --2- 172.21.15.90:0/2080221929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffab8159de0 0x7ffab8143020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:42.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.092+0000 7ffabca00640 1 --2- 172.21.15.90:0/2080221929 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa9c0039e0 0x7ffab81e5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:42.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.092+0000 7ffabd400640 1 -- 172.21.15.90:0/2080221929 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa9c0039e0 msgr2=0x7ffab81e5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:42.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.092+0000 7ffabd400640 1 --2- 172.21.15.90:0/2080221929 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa9c0039e0 0x7ffab81e5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:42.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.092+0000 7ffabd400640 1 -- 172.21.15.90:0/2080221929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffab8159de0 msgr2=0x7ffab8143020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:42.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.092+0000 7ffabd400640 1 --2- 172.21.15.90:0/2080221929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffab8159de0 0x7ffab8143020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:42.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.092+0000 7ffabd400640 1 -- 172.21.15.90:0/2080221929 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab815b7f0 con 0x7ffab815a240 2024-03-24T01:26:42.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.092+0000 7ffabd400640 1 --2- 172.21.15.90:0/2080221929 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffab815a240 0x7ffab8143560 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7ffab80b2e60 tx=0x7ffab000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:42.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.092+0000 7ffab5600640 1 -- 172.21.15.90:0/2080221929 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab0018020 con 0x7ffab815a240 2024-03-24T01:26:42.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.092+0000 7ffab5600640 1 -- 172.21.15.90:0/2080221929 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab000bae0 con 0x7ffab815a240 2024-03-24T01:26:42.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.093+0000 7ffabf400640 1 -- 172.21.15.90:0/2080221929 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffab815a750 con 0x7ffab815a240 2024-03-24T01:26:42.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.093+0000 7ffab5600640 1 -- 172.21.15.90:0/2080221929 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab0014400 con 0x7ffab815a240 2024-03-24T01:26:42.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.093+0000 7ffabf400640 1 -- 172.21.15.90:0/2080221929 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffab815ad10 con 0x7ffab815a240 2024-03-24T01:26:42.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.094+0000 7ffabf400640 1 -- 172.21.15.90:0/2080221929 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffab804f400 con 0x7ffab815a240 2024-03-24T01:26:42.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.095+0000 7ffab5600640 1 -- 172.21.15.90:0/2080221929 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ffab0005d10 con 0x7ffab815a240 2024-03-24T01:26:42.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.096+0000 7ffab5600640 1 --2- 172.21.15.90:0/2080221929 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ffa9406bbd0 0x7ffa9406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:42.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.096+0000 7ffab5600640 1 -- 172.21.15.90:0/2080221929 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7ffab008b220 con 0x7ffab815a240 2024-03-24T01:26:42.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.097+0000 7ffabca00640 1 --2- 172.21.15.90:0/2080221929 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ffa9406bbd0 0x7ffa9406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:42.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.098+0000 7ffabca00640 1 --2- 172.21.15.90:0/2080221929 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ffa9406bbd0 0x7ffa9406e090 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ffaa8034000 tx=0x7ffaa8009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:42.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.101+0000 7ffab5600640 1 -- 172.21.15.90:0/2080221929 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ffab0022030 con 0x7ffab815a240 2024-03-24T01:26:42.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.381+0000 7ffabf400640 1 -- 172.21.15.90:0/2080221929 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ffab806c970 con 0x7ffab815a240 2024-03-24T01:26:42.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.382+0000 7ffab5600640 1 -- 172.21.15.90:0/2080221929 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v40) v1 ==== 80+0+122 (secure 0 0 0) 0x7ffab0059b20 con 0x7ffab815a240 2024-03-24T01:26:42.382 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6835/2166652035 2024-03-25T01:26:27.648052+0000 2024-03-24T01:26:42.382 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6834/2166652035 2024-03-25T01:26:27.648052+0000 2024-03-24T01:26:42.383 INFO:teuthology.orchestra.run.smithi090.stderr:listed 2 entries 2024-03-24T01:26:42.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.385+0000 7ffabf400640 1 -- 172.21.15.90:0/2080221929 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ffa9406bbd0 msgr2=0x7ffa9406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:42.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.385+0000 7ffabf400640 1 --2- 172.21.15.90:0/2080221929 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ffa9406bbd0 0x7ffa9406e090 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ffaa8034000 tx=0x7ffaa8009030 comp rx=0 tx=0).stop 2024-03-24T01:26:42.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.385+0000 7ffabf400640 1 -- 172.21.15.90:0/2080221929 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffab815a240 msgr2=0x7ffab8143560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:42.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.385+0000 7ffabf400640 1 --2- 172.21.15.90:0/2080221929 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffab815a240 0x7ffab8143560 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7ffab80b2e60 tx=0x7ffab000b480 comp rx=0 tx=0).stop 2024-03-24T01:26:42.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.385+0000 7ffabf400640 1 -- 172.21.15.90:0/2080221929 shutdown_connections 2024-03-24T01:26:42.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.385+0000 7ffabf400640 1 --2- 172.21.15.90:0/2080221929 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ffa9406bbd0 0x7ffa9406e090 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:42.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.385+0000 7ffabf400640 1 --2- 172.21.15.90:0/2080221929 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffab815a240 0x7ffab8143560 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:42.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.385+0000 7ffabf400640 1 --2- 172.21.15.90:0/2080221929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffab8159de0 0x7ffab8143020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:42.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.385+0000 7ffabf400640 1 --2- 172.21.15.90:0/2080221929 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffa9c0039e0 0x7ffab81e5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:42.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.385+0000 7ffabf400640 1 -- 172.21.15.90:0/2080221929 >> 172.21.15.90:0/2080221929 conn(0x7ffab80b1770 msgr2=0x7ffab80b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:42.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.385+0000 7ffabf400640 1 -- 172.21.15.90:0/2080221929 shutdown_connections 2024-03-24T01:26:42.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:42.385+0000 7ffabf400640 1 -- 172.21.15.90:0/2080221929 wait complete. 2024-03-24T01:26:42.402 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi090.front.sepia.ceph.com... 2024-03-24T01:26:42.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:42.402 DEBUG:teuthology.orchestra.run.smithi090:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-24T01:26:42.482 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[55523]: fuse finished with error 0 and tester_r 0 2024-03-24T01:26:42.487 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:26:48.589 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-03-24T01:26:48.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:48.590 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T01:26:48.648 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T01:26:48.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.851+0000 7fd084a00640 1 Processor -- start 2024-03-24T01:26:48.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.851+0000 7fd084a00640 1 -- start start 2024-03-24T01:26:48.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.852+0000 7fd084a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd08015dba0 0x7fd080162330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:48.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.852+0000 7fd084a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd08015df80 0x7fd0800b5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:48.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.852+0000 7fd084a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0800ce4b0 con 0x7fd0800b2960 2024-03-24T01:26:48.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.852+0000 7fd084a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd080162870 con 0x7fd08015dba0 2024-03-24T01:26:48.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.852+0000 7fd084a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0800ce790 con 0x7fd08015df80 2024-03-24T01:26:48.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.853+0000 7fd07e800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd08015dba0 0x7fd080162330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:48.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.853+0000 7fd07f200640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd0800b2960 0x7fd08006c540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:54962/0 (socket says 172.21.15.90:54962) 2024-03-24T01:26:48.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.853+0000 7fd07f200640 1 -- 172.21.15.90:0/635614373 learned_addr learned my addr 172.21.15.90:0/635614373 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:26:48.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.853+0000 7fd07de00640 1 --2- 172.21.15.90:0/635614373 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd08015df80 0x7fd0800b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:48.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.853+0000 7fd07e800640 1 -- 172.21.15.90:0/635614373 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd08015df80 msgr2=0x7fd0800b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:48.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.853+0000 7fd07e800640 1 --2- 172.21.15.90:0/635614373 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd08015df80 0x7fd0800b5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:48.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.853+0000 7fd07e800640 1 -- 172.21.15.90:0/635614373 >> v1:172.21.15.90:6789/0 conn(0x7fd0800b2960 legacy=0x7fd08006c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:48.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.853+0000 7fd07e800640 1 -- 172.21.15.90:0/635614373 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0800cea70 con 0x7fd08015dba0 2024-03-24T01:26:48.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.853+0000 7fd07de00640 1 --2- 172.21.15.90:0/635614373 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd08015df80 0x7fd0800b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:48.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.853+0000 7fd07d400640 1 -- 172.21.15.90:0/635614373 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1615608493 0 0) 0x7fd0800ce4b0 con 0x7fd0800b2960 2024-03-24T01:26:48.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.854+0000 7fd07e800640 1 --2- 172.21.15.90:0/635614373 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd08015dba0 0x7fd080162330 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fd0700097a0 tx=0x7fd07002b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87f0ec66149ede78 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:48.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.855+0000 7fd07d400640 1 -- 172.21.15.90:0/635614373 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd070031780 con 0x7fd08015dba0 2024-03-24T01:26:48.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.855+0000 7fd07d400640 1 -- 172.21.15.90:0/635614373 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd070037070 con 0x7fd08015dba0 2024-03-24T01:26:48.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.855+0000 7fd07d400640 1 -- 172.21.15.90:0/635614373 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd070031b40 con 0x7fd08015dba0 2024-03-24T01:26:48.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.855+0000 7fd084a00640 1 -- 172.21.15.90:0/635614373 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd08015dba0 msgr2=0x7fd080162330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:48.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.855+0000 7fd084a00640 1 --2- 172.21.15.90:0/635614373 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd08015dba0 0x7fd080162330 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fd0700097a0 tx=0x7fd07002b820 comp rx=0 tx=0).stop 2024-03-24T01:26:48.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.855+0000 7fd084a00640 1 -- 172.21.15.90:0/635614373 shutdown_connections 2024-03-24T01:26:48.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.855+0000 7fd084a00640 1 --2- 172.21.15.90:0/635614373 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd08015df80 0x7fd0800b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:48.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.855+0000 7fd084a00640 1 --2- 172.21.15.90:0/635614373 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd08015dba0 0x7fd080162330 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:48.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.855+0000 7fd084a00640 1 -- 172.21.15.90:0/635614373 >> 172.21.15.90:0/635614373 conn(0x7fd0800d0530 msgr2=0x7fd0800d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:48.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.855+0000 7fd084a00640 1 -- 172.21.15.90:0/635614373 shutdown_connections 2024-03-24T01:26:48.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.855+0000 7fd084a00640 1 -- 172.21.15.90:0/635614373 wait complete. 2024-03-24T01:26:48.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.858+0000 7fd084a00640 1 Processor -- start 2024-03-24T01:26:48.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.858+0000 7fd084a00640 1 -- start start 2024-03-24T01:26:48.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.859+0000 7fd084a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0800b2960 0x7fd080160b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:48.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.859+0000 7fd084a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd08015dba0 0x7fd0800cab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:48.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.859+0000 7fd084a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd08015df80 0x7fd08014afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:48.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.859+0000 7fd084a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0800d2ea0 con 0x7fd08015dba0 2024-03-24T01:26:48.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.859+0000 7fd084a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0800d2d20 con 0x7fd0800b2960 2024-03-24T01:26:48.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.859+0000 7fd084a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0800d3020 con 0x7fd08015df80 2024-03-24T01:26:48.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.859+0000 7fd07de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd08015dba0 0x7fd0800cab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:48.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.859+0000 7fd07de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd08015dba0 0x7fd0800cab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53318/0 (socket says 172.21.15.90:53318) 2024-03-24T01:26:48.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.859+0000 7fd07de00640 1 -- 172.21.15.90:0/1125248189 learned_addr learned my addr 172.21.15.90:0/1125248189 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:48.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.859+0000 7fd07e800640 1 --2- 172.21.15.90:0/1125248189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0800b2960 0x7fd080160b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:48.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.859+0000 7fd07de00640 1 -- 172.21.15.90:0/1125248189 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd08015df80 msgr2=0x7fd08014afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:48.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.859+0000 7fd07f200640 1 --2- 172.21.15.90:0/1125248189 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd08015df80 0x7fd08014afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:48.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.859+0000 7fd07de00640 1 --2- 172.21.15.90:0/1125248189 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd08015df80 0x7fd08014afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:48.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.859+0000 7fd07de00640 1 -- 172.21.15.90:0/1125248189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0800b2960 msgr2=0x7fd080160b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:48.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.859+0000 7fd07de00640 1 --2- 172.21.15.90:0/1125248189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0800b2960 0x7fd080160b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:48.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.859+0000 7fd07de00640 1 -- 172.21.15.90:0/1125248189 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0800cb210 con 0x7fd08015dba0 2024-03-24T01:26:48.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.859+0000 7fd07e800640 1 --2- 172.21.15.90:0/1125248189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0800b2960 0x7fd080160b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:48.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.860+0000 7fd07de00640 1 --2- 172.21.15.90:0/1125248189 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd08015dba0 0x7fd0800cab80 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fd06c00c820 tx=0x7fd06c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:48.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.860+0000 7fd076a00640 1 -- 172.21.15.90:0/1125248189 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06c015070 con 0x7fd08015dba0 2024-03-24T01:26:48.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.860+0000 7fd076a00640 1 -- 172.21.15.90:0/1125248189 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd06c007a30 con 0x7fd08015dba0 2024-03-24T01:26:48.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.860+0000 7fd084a00640 1 -- 172.21.15.90:0/1125248189 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0800cb4f0 con 0x7fd08015dba0 2024-03-24T01:26:48.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.860+0000 7fd076a00640 1 -- 172.21.15.90:0/1125248189 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06c007de0 con 0x7fd08015dba0 2024-03-24T01:26:48.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.860+0000 7fd084a00640 1 -- 172.21.15.90:0/1125248189 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0800cbab0 con 0x7fd08015dba0 2024-03-24T01:26:48.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.861+0000 7fd084a00640 1 -- 172.21.15.90:0/1125248189 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd08004f400 con 0x7fd08015dba0 2024-03-24T01:26:48.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.862+0000 7fd076a00640 1 -- 172.21.15.90:0/1125248189 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd06c018070 con 0x7fd08015dba0 2024-03-24T01:26:48.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.864+0000 7fd076a00640 1 --2- 172.21.15.90:0/1125248189 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd06006bbd0 0x7fd06006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:48.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.864+0000 7fd076a00640 1 -- 172.21.15.90:0/1125248189 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fd06c08abb0 con 0x7fd08015dba0 2024-03-24T01:26:48.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.864+0000 7fd07e800640 1 --2- 172.21.15.90:0/1125248189 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd06006bbd0 0x7fd06006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:48.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.866+0000 7fd07e800640 1 --2- 172.21.15.90:0/1125248189 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd06006bbd0 0x7fd06006e090 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fd070036000 tx=0x7fd070008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:48.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:48.868+0000 7fd076a00640 1 -- 172.21.15.90:0/1125248189 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd06c0594b0 con 0x7fd08015dba0 2024-03-24T01:26:49.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:49.227+0000 7fd084a00640 1 -- 172.21.15.90:0/1125248189 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd0800b64b0 con 0x7fd08015dba0 2024-03-24T01:26:49.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:49.228+0000 7fd076a00640 1 -- 172.21.15.90:0/1125248189 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd06c05de20 con 0x7fd08015dba0 2024-03-24T01:26:49.229 INFO:teuthology.orchestra.run.smithi090.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T01:26:49.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:49.230+0000 7fd084a00640 1 -- 172.21.15.90:0/1125248189 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd06006bbd0 msgr2=0x7fd06006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:49.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:49.230+0000 7fd084a00640 1 --2- 172.21.15.90:0/1125248189 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd06006bbd0 0x7fd06006e090 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fd070036000 tx=0x7fd070008030 comp rx=0 tx=0).stop 2024-03-24T01:26:49.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:49.230+0000 7fd084a00640 1 -- 172.21.15.90:0/1125248189 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd08015dba0 msgr2=0x7fd0800cab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:49.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:49.230+0000 7fd084a00640 1 --2- 172.21.15.90:0/1125248189 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd08015dba0 0x7fd0800cab80 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fd06c00c820 tx=0x7fd06c00cce0 comp rx=0 tx=0).stop 2024-03-24T01:26:49.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:49.230+0000 7fd084a00640 1 -- 172.21.15.90:0/1125248189 shutdown_connections 2024-03-24T01:26:49.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:49.230+0000 7fd084a00640 1 --2- 172.21.15.90:0/1125248189 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd06006bbd0 0x7fd06006e090 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:49.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:49.230+0000 7fd084a00640 1 --2- 172.21.15.90:0/1125248189 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd08015dba0 0x7fd0800cab80 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:49.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:49.230+0000 7fd084a00640 1 --2- 172.21.15.90:0/1125248189 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd08015df80 0x7fd08014afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:49.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:49.230+0000 7fd084a00640 1 --2- 172.21.15.90:0/1125248189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0800b2960 0x7fd080160b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:49.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:49.230+0000 7fd084a00640 1 -- 172.21.15.90:0/1125248189 >> 172.21.15.90:0/1125248189 conn(0x7fd0800d0530 msgr2=0x7fd0800d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:49.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:49.231+0000 7fd084a00640 1 -- 172.21.15.90:0/1125248189 shutdown_connections 2024-03-24T01:26:49.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:49.231+0000 7fd084a00640 1 -- 172.21.15.90:0/1125248189 wait complete. 2024-03-24T01:26:49.248 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T01:26:49.249 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T01:26:49.249 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi090.front.sepia.ceph.com 2024-03-24T01:26:49.249 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T01:26:49.249 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T01:26:49.249 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T01:26:49.249 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T01:26:49.249 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T01:26:49.249 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T01:26:49.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:49.250 DEBUG:teuthology.orchestra.run.smithi090:> ip addr 2024-03-24T01:26:49.294 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T01:26:49.294 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T01:26:49.294 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T01:26:49.294 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:26:49.295 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-03-24T01:26:49.295 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:26:49.295 INFO:teuthology.orchestra.run.smithi090.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T01:26:49.295 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a8 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:26:49.295 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f0 2024-03-24T01:26:49.295 INFO:teuthology.orchestra.run.smithi090.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T01:26:49.295 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a9 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:26:49.295 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f1 2024-03-24T01:26:49.295 INFO:teuthology.orchestra.run.smithi090.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T01:26:49.295 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:66 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:26:49.295 INFO:teuthology.orchestra.run.smithi090.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T01:26:49.295 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:67 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:26:49.295 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.21.15.90/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T01:26:49.295 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft 42373sec preferred_lft 42373sec 2024-03-24T01:26:49.295 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::ec4:7aff:fe8f:d767/64 scope link 2024-03-24T01:26:49.296 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:26:49.296 INFO:teuthology.orchestra.run.smithi090.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T01:26:49.296 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 06:50:1d:bd:ed:70 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:26:49.296 INFO:teuthology.orchestra.run.smithi090.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T01:26:49.296 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:26:49.296 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::a05b:d3ff:fea5:6fd0/64 scope link 2024-03-24T01:26:49.296 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:26:49.296 INFO:teuthology.orchestra.run.smithi090.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T01:26:49.296 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether de:94:0b:7e:c6:41 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:26:49.296 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::dc94:bff:fe7e:c641/64 scope link 2024-03-24T01:26:49.296 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:26:49.296 INFO:teuthology.orchestra.run.smithi090.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T01:26:49.297 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 06:50:1d:bd:ed:70 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:26:49.297 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::450:1dff:febd:ed70/64 scope link 2024-03-24T01:26:49.297 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:26:49.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:49.298 DEBUG:teuthology.orchestra.run.smithi090:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T01:26:49.424 INFO:teuthology.orchestra.run.smithi090.stdout:1 2024-03-24T01:26:49.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:49.429 DEBUG:teuthology.orchestra.run.smithi090:> route 2024-03-24T01:26:49.481 INFO:teuthology.orchestra.run.smithi090.stdout:Kernel IP routing table 2024-03-24T01:26:49.481 INFO:teuthology.orchestra.run.smithi090.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T01:26:49.481 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T01:26:49.482 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T01:26:49.482 INFO:teuthology.orchestra.run.smithi090.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T01:26:49.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:49.482 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2024-03-24T01:26:49.482 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-24T01:26:49.482 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T01:26:49.482 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T01:26:49.482 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T01:26:49.482 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-24T01:26:49.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:49.660 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-03-24T01:26:49.706 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T01:26:49.707 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T01:26:49.707 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T01:26:49.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:49.707 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set brx.0 up 2024-03-24T01:26:49.842 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-24T01:26:49.842 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T01:26:49.842 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T01:26:49.891 INFO:teuthology.orchestra.run.smithi090.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T01:26:49.891 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T01:26:49.891 DEBUG:teuthology.orchestra.run.smithi090:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T01:26:49.937 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-24T01:26:50.075 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T01:26:50.124 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-24T01:26:50.124 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-24T01:26:50.124 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-24T01:26:50.124 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-24T01:26:50.124 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-24T01:26:50.124 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-24T01:26:50.124 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-24T01:26:50.124 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-24T01:26:50.124 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-24T01:26:50.124 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-24T01:26:50.125 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-24T01:26:50.125 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/selinux 2024-03-24T01:26:50.125 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T01:26:50.125 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-24T01:26:50.125 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-24T01:26:50.125 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-24T01:26:50.125 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-24T01:26:50.125 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T01:26:50.125 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-24T01:26:50.125 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T01:26:50.125 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-24T01:26:50.125 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T01:26:50.125 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T01:26:50.125 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-24T01:26:50.125 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-24T01:26:50.126 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T01:26:50.126 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T01:26:50.126 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T01:26:50.126 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T01:26:50.126 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-24T01:26:50.126 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:26:50.126 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:26:50.126 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:26:50.126 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:26:50.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:50.127 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-24T01:26:50.168 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-24T01:26:50.169 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-24T01:26:50.217 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-24T01:26:50.294 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T01:26:50.335 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-24T01:26:50.336 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-24T01:26:50.336 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-24T01:26:50.336 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-24T01:26:50.336 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-24T01:26:50.336 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-24T01:26:50.336 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-24T01:26:50.336 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-24T01:26:50.336 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-24T01:26:50.336 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-24T01:26:50.336 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-24T01:26:50.336 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/selinux 2024-03-24T01:26:50.336 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T01:26:50.337 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-24T01:26:50.337 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-24T01:26:50.337 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-24T01:26:50.337 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-24T01:26:50.337 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T01:26:50.337 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-24T01:26:50.337 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T01:26:50.337 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-24T01:26:50.337 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T01:26:50.337 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T01:26:50.337 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-24T01:26:50.337 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-24T01:26:50.337 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T01:26:50.337 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T01:26:50.338 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T01:26:50.338 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T01:26:50.338 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-24T01:26:50.338 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:26:50.338 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:26:50.338 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:26:50.338 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:26:50.339 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:50.339 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-24T01:26:50.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:2024-03-24T01:26:50.405+0000 7f1029d70480 -1 init, newargv = 0x5649945409f0 newargc=15 2024-03-24T01:26:50.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[56445]: starting ceph client 2024-03-24T01:26:50.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[56445]: starting fuse 2024-03-24T01:26:50.426 INFO:teuthology.orchestra.run.smithi090.stdout:40 2024-03-24T01:26:50.426 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-03-24T01:26:50.426 DEBUG:teuthology.orchestra.run.smithi090:> sudo stdin-killer -- python3 -c ' 2024-03-24T01:26:50.426 DEBUG:teuthology.orchestra.run.smithi090:> import glob 2024-03-24T01:26:50.426 DEBUG:teuthology.orchestra.run.smithi090:> import re 2024-03-24T01:26:50.426 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-24T01:26:50.426 DEBUG:teuthology.orchestra.run.smithi090:> import subprocess 2024-03-24T01:26:50.426 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:26:50.427 DEBUG:teuthology.orchestra.run.smithi090:> def _find_admin_socket(client_name): 2024-03-24T01:26:50.427 DEBUG:teuthology.orchestra.run.smithi090:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-24T01:26:50.427 DEBUG:teuthology.orchestra.run.smithi090:> files = glob.glob(asok_path) 2024-03-24T01:26:50.427 DEBUG:teuthology.orchestra.run.smithi090:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-24T01:26:50.427 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:26:50.427 DEBUG:teuthology.orchestra.run.smithi090:> # Given a non-glob path, it better be there 2024-03-24T01:26:50.427 DEBUG:teuthology.orchestra.run.smithi090:> if "*" not in asok_path: 2024-03-24T01:26:50.427 DEBUG:teuthology.orchestra.run.smithi090:> assert(len(files) == 1) 2024-03-24T01:26:50.427 DEBUG:teuthology.orchestra.run.smithi090:> return files[0] 2024-03-24T01:26:50.427 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:26:50.427 DEBUG:teuthology.orchestra.run.smithi090:> for f in files: 2024-03-24T01:26:50.427 DEBUG:teuthology.orchestra.run.smithi090:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T01:26:50.427 DEBUG:teuthology.orchestra.run.smithi090:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T01:26:50.427 DEBUG:teuthology.orchestra.run.smithi090:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T01:26:50.427 DEBUG:teuthology.orchestra.run.smithi090:> contents = proc_f.read() 2024-03-24T01:26:50.427 DEBUG:teuthology.orchestra.run.smithi090:> if mountpoint in contents: 2024-03-24T01:26:50.427 DEBUG:teuthology.orchestra.run.smithi090:> return f 2024-03-24T01:26:50.428 DEBUG:teuthology.orchestra.run.smithi090:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T01:26:50.428 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:26:50.428 DEBUG:teuthology.orchestra.run.smithi090:> print(_find_admin_socket("client.0")) 2024-03-24T01:26:50.428 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-24T01:26:50.616 INFO:teuthology.orchestra.run.smithi090.stdout:/var/run/ceph/ceph-client.0.56445.asok 2024-03-24T01:26:50.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T01:26:50.633 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.56445.asok 2024-03-24T01:26:50.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:50.634 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.56445.asok status 2024-03-24T01:26:50.862 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-24T01:26:50.862 INFO:teuthology.orchestra.run.smithi090.stdout: "metadata": { 2024-03-24T01:26:50.862 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_sha1": "3be9a5c9bc793e11e2800a8c0c696e8b46742033", 2024-03-24T01:26:50.863 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-24T01:26:50.863 INFO:teuthology.orchestra.run.smithi090.stdout: "entity_id": "0", 2024-03-24T01:26:50.863 INFO:teuthology.orchestra.run.smithi090.stdout: "hostname": "smithi090", 2024-03-24T01:26:50.863 INFO:teuthology.orchestra.run.smithi090.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-24T01:26:50.863 INFO:teuthology.orchestra.run.smithi090.stdout: "pid": "56445", 2024-03-24T01:26:50.863 INFO:teuthology.orchestra.run.smithi090.stdout: "root": "/" 2024-03-24T01:26:50.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:26:50.863 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_count": 0, 2024-03-24T01:26:50.863 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_pinned_count": 0, 2024-03-24T01:26:50.863 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 6003, 2024-03-24T01:26:50.863 INFO:teuthology.orchestra.run.smithi090.stdout: "inst": { 2024-03-24T01:26:50.863 INFO:teuthology.orchestra.run.smithi090.stdout: "name": { 2024-03-24T01:26:50.863 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "client", 2024-03-24T01:26:50.863 INFO:teuthology.orchestra.run.smithi090.stdout: "num": 6003 2024-03-24T01:26:50.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:26:50.864 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-24T01:26:50.864 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-24T01:26:50.865 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-03-24T01:26:50.865 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 4239060849 2024-03-24T01:26:50.865 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:26:50.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:26:50.865 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-24T01:26:50.865 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-24T01:26:50.865 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-03-24T01:26:50.865 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 4239060849 2024-03-24T01:26:50.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:26:50.865 INFO:teuthology.orchestra.run.smithi090.stdout: "inst_str": "client.6003 192.168.0.1:0/4239060849", 2024-03-24T01:26:50.865 INFO:teuthology.orchestra.run.smithi090.stdout: "addr_str": "192.168.0.1:0/4239060849", 2024-03-24T01:26:50.865 INFO:teuthology.orchestra.run.smithi090.stdout: "inode_count": 1, 2024-03-24T01:26:50.865 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_epoch": 25, 2024-03-24T01:26:50.866 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch": 40, 2024-03-24T01:26:50.866 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch_barrier": 0, 2024-03-24T01:26:50.866 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklisted": false, 2024-03-24T01:26:50.866 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs" 2024-03-24T01:26:50.866 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-24T01:26:50.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:50.877 DEBUG:teuthology.orchestra.run.smithi090:> stat --file-system '--printf=%T 2024-03-24T01:26:50.877 DEBUG:teuthology.orchestra.run.smithi090:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-24T01:26:50.925 INFO:teuthology.orchestra.run.smithi090.stdout:fuseblk 2024-03-24T01:26:50.925 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-24T01:26:50.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:50.926 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T01:26:51.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:51.050 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -R) 2024-03-24T01:26:51.118 INFO:teuthology.orchestra.run.smithi090.stdout:.: 2024-03-24T01:26:51.118 INFO:teuthology.orchestra.run.smithi090.stdout:linkdir 2024-03-24T01:26:51.118 INFO:teuthology.orchestra.run.smithi090.stdout:rootfile 2024-03-24T01:26:51.118 INFO:teuthology.orchestra.run.smithi090.stdout:subdir 2024-03-24T01:26:51.118 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:26:51.118 INFO:teuthology.orchestra.run.smithi090.stdout:./linkdir: 2024-03-24T01:26:51.118 INFO:teuthology.orchestra.run.smithi090.stdout:link0 2024-03-24T01:26:51.118 INFO:teuthology.orchestra.run.smithi090.stdout:link1 2024-03-24T01:26:51.119 INFO:teuthology.orchestra.run.smithi090.stdout:link2 2024-03-24T01:26:51.119 INFO:teuthology.orchestra.run.smithi090.stdout:link3 2024-03-24T01:26:51.119 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:26:51.119 INFO:teuthology.orchestra.run.smithi090.stdout:./subdir: 2024-03-24T01:26:51.119 INFO:teuthology.orchestra.run.smithi090.stdout:subdirfile 2024-03-24T01:26:51.119 INFO:teuthology.orchestra.run.smithi090.stdout:subsubdir 2024-03-24T01:26:51.119 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:26:51.119 INFO:teuthology.orchestra.run.smithi090.stdout:./subdir/subsubdir: 2024-03-24T01:26:51.120 DEBUG:teuthology.orchestra.run.smithi090:> stdin-killer -- python3 -c ' 2024-03-24T01:26:51.120 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-24T01:26:51.120 DEBUG:teuthology.orchestra.run.smithi090:> import stat 2024-03-24T01:26:51.120 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:26:51.121 DEBUG:teuthology.orchestra.run.smithi090:> print(os.stat("/home/ubuntu/cephtest/mnt.0/rootfile").st_ino) 2024-03-24T01:26:51.121 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-24T01:26:51.238 INFO:teuthology.orchestra.run.smithi090.stdout:1099511627776 2024-03-24T01:26:51.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T01:26:51.253 DEBUG:teuthology.orchestra.run.smithi090:> stdin-killer -- python3 -c ' 2024-03-24T01:26:51.253 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-24T01:26:51.253 DEBUG:teuthology.orchestra.run.smithi090:> import stat 2024-03-24T01:26:51.253 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:26:51.253 DEBUG:teuthology.orchestra.run.smithi090:> print(os.stat("/home/ubuntu/cephtest/mnt.0/subdir").st_ino) 2024-03-24T01:26:51.253 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-24T01:26:51.423 INFO:teuthology.orchestra.run.smithi090.stdout:1099511627777 2024-03-24T01:26:51.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T01:26:51.436 DEBUG:teuthology.orchestra.run.smithi090:> stdin-killer -- python3 -c ' 2024-03-24T01:26:51.436 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-24T01:26:51.436 DEBUG:teuthology.orchestra.run.smithi090:> import stat 2024-03-24T01:26:51.436 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:26:51.436 DEBUG:teuthology.orchestra.run.smithi090:> print(os.stat("/home/ubuntu/cephtest/mnt.0/subdir/subdirfile").st_ino) 2024-03-24T01:26:51.436 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-24T01:26:51.610 INFO:teuthology.orchestra.run.smithi090.stdout:1099511627778 2024-03-24T01:26:51.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T01:26:51.625 DEBUG:teuthology.orchestra.run.smithi090:> stdin-killer -- python3 -c ' 2024-03-24T01:26:51.625 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-24T01:26:51.625 DEBUG:teuthology.orchestra.run.smithi090:> import stat 2024-03-24T01:26:51.625 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:26:51.625 DEBUG:teuthology.orchestra.run.smithi090:> print(os.stat("/home/ubuntu/cephtest/mnt.0/subdir/subsubdir").st_ino) 2024-03-24T01:26:51.626 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-24T01:26:51.794 INFO:teuthology.orchestra.run.smithi090.stdout:1099511628282 2024-03-24T01:26:51.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T01:26:51.807 DEBUG:teuthology.orchestra.run.smithi090:> stdin-killer -- python3 -c ' 2024-03-24T01:26:51.808 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-24T01:26:51.808 DEBUG:teuthology.orchestra.run.smithi090:> import stat 2024-03-24T01:26:51.808 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:26:51.808 DEBUG:teuthology.orchestra.run.smithi090:> print(os.stat("/home/ubuntu/cephtest/mnt.0/linkdir/link0").st_ino) 2024-03-24T01:26:51.808 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-24T01:26:51.972 INFO:teuthology.orchestra.run.smithi090.stdout:1099511627778 2024-03-24T01:26:51.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T01:26:51.985 DEBUG:teuthology.orchestra.run.smithi090:> stdin-killer -- python3 -c ' 2024-03-24T01:26:51.985 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-24T01:26:51.985 DEBUG:teuthology.orchestra.run.smithi090:> import stat 2024-03-24T01:26:51.985 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:26:51.985 DEBUG:teuthology.orchestra.run.smithi090:> print(os.stat("/home/ubuntu/cephtest/mnt.0/linkdir/link1").st_ino) 2024-03-24T01:26:51.985 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-24T01:26:52.171 INFO:teuthology.orchestra.run.smithi090.stdout:1099511627778 2024-03-24T01:26:52.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T01:26:52.184 DEBUG:teuthology.orchestra.run.smithi090:> stdin-killer -- python3 -c ' 2024-03-24T01:26:52.185 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-24T01:26:52.185 DEBUG:teuthology.orchestra.run.smithi090:> import stat 2024-03-24T01:26:52.185 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:26:52.185 DEBUG:teuthology.orchestra.run.smithi090:> print(os.stat("/home/ubuntu/cephtest/mnt.0/linkdir/link2").st_ino) 2024-03-24T01:26:52.185 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-24T01:26:52.376 INFO:teuthology.orchestra.run.smithi090.stdout:1099511628281 2024-03-24T01:26:52.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T01:26:52.393 DEBUG:teuthology.orchestra.run.smithi090:> stdin-killer -- python3 -c ' 2024-03-24T01:26:52.394 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-24T01:26:52.394 DEBUG:teuthology.orchestra.run.smithi090:> import stat 2024-03-24T01:26:52.394 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:26:52.394 DEBUG:teuthology.orchestra.run.smithi090:> print(os.stat("/home/ubuntu/cephtest/mnt.0/linkdir/link3").st_ino) 2024-03-24T01:26:52.394 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-24T01:26:52.573 INFO:teuthology.orchestra.run.smithi090.stdout:1099511627776 2024-03-24T01:26:52.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T01:26:52.587 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:52.587 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch afterwards) 2024-03-24T01:26:52.698 DEBUG:teuthology.orchestra.run.smithi090:> stdin-killer -- python3 -c ' 2024-03-24T01:26:52.698 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-24T01:26:52.698 DEBUG:teuthology.orchestra.run.smithi090:> import stat 2024-03-24T01:26:52.698 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:26:52.698 DEBUG:teuthology.orchestra.run.smithi090:> print(os.stat("/home/ubuntu/cephtest/mnt.0/afterwards").st_ino) 2024-03-24T01:26:52.698 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-24T01:26:52.838 INFO:teuthology.orchestra.run.smithi090.stdout:1099521627779 2024-03-24T01:26:52.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T01:26:52.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:52.846 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch subdir/subsubdir/subsubdirfile) 2024-03-24T01:26:52.943 INFO:tasks.ceph.mds.c.smithi090.stderr:2024-03-24T01:26:52.943+0000 7f40cac00640 -1 log_channel(cluster) log [ERR] : bad backtrace on directory inode 0x100000001fa 2024-03-24T01:26:52.949 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T01:26:52.949 DEBUG:teuthology.orchestra.run.smithi090:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T01:26:53.054 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T01:26:53.176 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T01:26:53.176 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:26:53.176 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:26:53.234 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:26:53.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.444+0000 7fc0d0a00640 1 Processor -- start 2024-03-24T01:26:53.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.444+0000 7fc0d0a00640 1 -- start start 2024-03-24T01:26:53.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.445+0000 7fc0d0a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0cc0d1620 0x7fc0cc0cf210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:53.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.445+0000 7fc0d0a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0cc0d1b20 con 0x7fc0cc06c860 2024-03-24T01:26:53.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.446+0000 7fc0d0a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0cc0d1e00 con 0x7fc0cc06cc40 2024-03-24T01:26:53.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.446+0000 7fc0d0a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0cc0d2730 con 0x7fc0cc0d1620 2024-03-24T01:26:53.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.446+0000 7fc0ca800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc0cc06c860 0x7fc0cc0d2bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53118/0 (socket says 172.21.15.90:53118) 2024-03-24T01:26:53.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.446+0000 7fc0ca800640 1 -- 172.21.15.90:0/179499587 learned_addr learned my addr 172.21.15.90:0/179499587 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:26:53.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.446+0000 7fc0c9e00640 1 --2- 172.21.15.90:0/179499587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0cc0d1620 0x7fc0cc0cf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:53.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.446+0000 7fc0c9400640 1 -- 172.21.15.90:0/179499587 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 775863131 0 0) 0x7fc0cc0d1b20 con 0x7fc0cc06c860 2024-03-24T01:26:53.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.446+0000 7fc0c9400640 1 -- 172.21.15.90:0/179499587 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0b0003520 con 0x7fc0cc06c860 2024-03-24T01:26:53.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.447+0000 7fc0c9400640 1 -- 172.21.15.90:0/179499587 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3528192470 0 0) 0x7fc0b0003520 con 0x7fc0cc06c860 2024-03-24T01:26:53.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.447+0000 7fc0c9400640 1 -- 172.21.15.90:0/179499587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0cc0d1620 msgr2=0x7fc0cc0cf210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:53.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.447+0000 7fc0c9400640 1 --2- 172.21.15.90:0/179499587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0cc0d1620 0x7fc0cc0cf210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:53.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.447+0000 7fc0c9400640 1 -- 172.21.15.90:0/179499587 >> v1:172.21.15.120:6789/0 conn(0x7fc0cc06cc40 legacy=0x7fc0cc0b6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:53.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.447+0000 7fc0c9400640 1 -- 172.21.15.90:0/179499587 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0cc0d20e0 con 0x7fc0cc06c860 2024-03-24T01:26:53.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.447+0000 7fc0c9400640 1 -- 172.21.15.90:0/179499587 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7fc0bc001670 con 0x7fc0cc06c860 2024-03-24T01:26:53.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.447+0000 7fc0c9400640 1 -- 172.21.15.90:0/179499587 >> v1:172.21.15.90:6789/0 conn(0x7fc0cc06c860 legacy=0x7fc0cc0d2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:53.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.447+0000 7fc0c9400640 1 --2- 172.21.15.90:0/179499587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0cc0d1620 0x7fc0b00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:53.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.447+0000 7fc0c9400640 1 --2- 172.21.15.90:0/179499587 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0b0004060 0x7fc0b0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:53.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.447+0000 7fc0c9400640 1 --2- 172.21.15.90:0/179499587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0b0004d40 0x7fc0b00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:53.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.447+0000 7fc0c9400640 1 -- 172.21.15.90:0/179499587 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0cc0d2730 con 0x7fc0b0004060 2024-03-24T01:26:53.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.447+0000 7fc0c9400640 1 -- 172.21.15.90:0/179499587 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b000bc00 con 0x7fc0b0004d40 2024-03-24T01:26:53.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.447+0000 7fc0c9400640 1 -- 172.21.15.90:0/179499587 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b000be00 con 0x7fc0cc0d1620 2024-03-24T01:26:53.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.447+0000 7fc0c9e00640 1 --2- 172.21.15.90:0/179499587 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0b0004060 0x7fc0b0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:53.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.448+0000 7fc0ca800640 1 --2- 172.21.15.90:0/179499587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0cc0d1620 0x7fc0b00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:53.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.448+0000 7fc0cb200640 1 --2- 172.21.15.90:0/179499587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0b0004d40 0x7fc0b00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:53.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.448+0000 7fc0c9e00640 1 -- 172.21.15.90:0/179499587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0cc0d1620 msgr2=0x7fc0b00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:53.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.448+0000 7fc0c9e00640 1 --2- 172.21.15.90:0/179499587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0cc0d1620 0x7fc0b00039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:53.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.448+0000 7fc0c9e00640 1 -- 172.21.15.90:0/179499587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0b0004d40 msgr2=0x7fc0b00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:53.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.448+0000 7fc0c9e00640 1 --2- 172.21.15.90:0/179499587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0b0004d40 0x7fc0b00051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:53.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.448+0000 7fc0c9e00640 1 -- 172.21.15.90:0/179499587 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b0003520 con 0x7fc0b0004060 2024-03-24T01:26:53.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.449+0000 7fc0c9e00640 1 --2- 172.21.15.90:0/179499587 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0b0004060 0x7fc0b0004800 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fc0b800ba30 tx=0x7fc0b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc27413199b88eb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:53.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.449+0000 7fc0c9400640 1 -- 172.21.15.90:0/179499587 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b8036020 con 0x7fc0b0004060 2024-03-24T01:26:53.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.449+0000 7fc0c9400640 1 -- 172.21.15.90:0/179499587 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b8032640 con 0x7fc0b0004060 2024-03-24T01:26:53.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.449+0000 7fc0c9400640 1 -- 172.21.15.90:0/179499587 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b800b600 con 0x7fc0b0004060 2024-03-24T01:26:53.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.449+0000 7fc0d0a00640 1 -- 172.21.15.90:0/179499587 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0b0004060 msgr2=0x7fc0b0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:53.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.449+0000 7fc0d0a00640 1 --2- 172.21.15.90:0/179499587 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0b0004060 0x7fc0b0004800 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fc0b800ba30 tx=0x7fc0b0005830 comp rx=0 tx=0).stop 2024-03-24T01:26:53.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.450+0000 7fc0cb200640 1 -- 172.21.15.90:0/179499587 reap_dead start 2024-03-24T01:26:53.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.450+0000 7fc0d0a00640 1 -- 172.21.15.90:0/179499587 shutdown_connections 2024-03-24T01:26:53.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.450+0000 7fc0d0a00640 1 -- 172.21.15.90:0/179499587 >> 172.21.15.90:0/179499587 conn(0x7fc0cc0d0560 msgr2=0x7fc0cc0d0970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:53.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.450+0000 7fc0d0a00640 1 -- 172.21.15.90:0/179499587 shutdown_connections 2024-03-24T01:26:53.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.450+0000 7fc0d0a00640 1 -- 172.21.15.90:0/179499587 wait complete. 2024-03-24T01:26:53.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.452+0000 7fc0d0a00640 1 Processor -- start 2024-03-24T01:26:53.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.453+0000 7fc0d0a00640 1 -- start start 2024-03-24T01:26:53.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.454+0000 7fc0d0a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0b0004060 0x7fc0cc0b5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:53.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.454+0000 7fc0d0a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0cc160b10 0x7fc0cc160f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:53.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.454+0000 7fc0d0a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0cc1e93d0 0x7fc0cc1e9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:53.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.454+0000 7fc0d0a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0cc0c7300 con 0x7fc0b0004060 2024-03-24T01:26:53.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.454+0000 7fc0d0a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0cc0c7180 con 0x7fc0cc160b10 2024-03-24T01:26:53.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.454+0000 7fc0d0a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0cc0d32d0 con 0x7fc0cc1e93d0 2024-03-24T01:26:53.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.454+0000 7fc0ca800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0b0004060 0x7fc0cc0b5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:53.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.455+0000 7fc0ca800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0b0004060 0x7fc0cc0b5cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35518/0 (socket says 172.21.15.90:35518) 2024-03-24T01:26:53.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.455+0000 7fc0ca800640 1 -- 172.21.15.90:0/1006230309 learned_addr learned my addr 172.21.15.90:0/1006230309 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:53.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.455+0000 7fc0c9e00640 1 --2- 172.21.15.90:0/1006230309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0cc160b10 0x7fc0cc160f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:53.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.455+0000 7fc0cb200640 1 --2- 172.21.15.90:0/1006230309 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0cc1e93d0 0x7fc0cc1e9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:53.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.455+0000 7fc0ca800640 1 -- 172.21.15.90:0/1006230309 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0cc1e93d0 msgr2=0x7fc0cc1e9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:53.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.455+0000 7fc0ca800640 1 --2- 172.21.15.90:0/1006230309 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0cc1e93d0 0x7fc0cc1e9850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:53.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.455+0000 7fc0ca800640 1 -- 172.21.15.90:0/1006230309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0cc160b10 msgr2=0x7fc0cc160f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:53.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.455+0000 7fc0ca800640 1 --2- 172.21.15.90:0/1006230309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0cc160b10 0x7fc0cc160f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:53.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.455+0000 7fc0ca800640 1 -- 172.21.15.90:0/1006230309 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0cc147f10 con 0x7fc0b0004060 2024-03-24T01:26:53.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.455+0000 7fc0c9e00640 1 --2- 172.21.15.90:0/1006230309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0cc160b10 0x7fc0cc160f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:53.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.455+0000 7fc0ca800640 1 --2- 172.21.15.90:0/1006230309 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0b0004060 0x7fc0cc0b5cb0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc0018a0 tx=0x7fc0bc00dc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:53.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.456+0000 7fc0c2a00640 1 -- 172.21.15.90:0/1006230309 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0bc019070 con 0x7fc0b0004060 2024-03-24T01:26:53.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.456+0000 7fc0c2a00640 1 -- 172.21.15.90:0/1006230309 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0bc0074e0 con 0x7fc0b0004060 2024-03-24T01:26:53.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.456+0000 7fc0d0a00640 1 -- 172.21.15.90:0/1006230309 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0cc1481f0 con 0x7fc0b0004060 2024-03-24T01:26:53.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.456+0000 7fc0c2a00640 1 -- 172.21.15.90:0/1006230309 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0bc004a00 con 0x7fc0b0004060 2024-03-24T01:26:53.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.456+0000 7fc0d0a00640 1 -- 172.21.15.90:0/1006230309 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0cc0b61f0 con 0x7fc0b0004060 2024-03-24T01:26:53.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.457+0000 7fc0d0a00640 1 -- 172.21.15.90:0/1006230309 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0cc04f400 con 0x7fc0b0004060 2024-03-24T01:26:53.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.459+0000 7fc0c2a00640 1 -- 172.21.15.90:0/1006230309 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc0bc01c070 con 0x7fc0b0004060 2024-03-24T01:26:53.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.460+0000 7fc0c2a00640 1 --2- 172.21.15.90:0/1006230309 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc0ac06bc00 0x7fc0ac06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:53.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.460+0000 7fc0c2a00640 1 -- 172.21.15.90:0/1006230309 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6216+0+0 (secure 0 0 0) 0x7fc0bc08a660 con 0x7fc0b0004060 2024-03-24T01:26:53.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.460+0000 7fc0c9e00640 1 --2- 172.21.15.90:0/1006230309 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc0ac06bc00 0x7fc0ac06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:53.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.462+0000 7fc0c9e00640 1 --2- 172.21.15.90:0/1006230309 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc0ac06bc00 0x7fc0ac06e0c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc0b800ba00 tx=0x7fc0b8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:53.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.464+0000 7fc0c2a00640 1 -- 172.21.15.90:0/1006230309 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc0bc058fc0 con 0x7fc0b0004060 2024-03-24T01:26:53.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.679+0000 7fc0d0a00640 1 -- 172.21.15.90:0/1006230309 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0cc0ce3d0 con 0x7fc0b0004060 2024-03-24T01:26:53.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.682+0000 7fc0c2a00640 1 -- 172.21.15.90:0/1006230309 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+17096 (secure 0 0 0) 0x7fc0bc0588c0 con 0x7fc0b0004060 2024-03-24T01:26:53.683 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:26:53.683 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":42,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:26:53.308852+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:25:57.398864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-24T01:25:58.451150+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.30","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:6835/2166652035":"2024-03-25T01:26:27.648052+0000","172.21.15.90:6834/2166652035":"2024-03-25T01:26:27.648052+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:26:53.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.686+0000 7fc0d0a00640 1 -- 172.21.15.90:0/1006230309 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc0ac06bc00 msgr2=0x7fc0ac06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:53.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.686+0000 7fc0d0a00640 1 --2- 172.21.15.90:0/1006230309 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc0ac06bc00 0x7fc0ac06e0c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc0b800ba00 tx=0x7fc0b8030000 comp rx=0 tx=0).stop 2024-03-24T01:26:53.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.686+0000 7fc0d0a00640 1 -- 172.21.15.90:0/1006230309 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0b0004060 msgr2=0x7fc0cc0b5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:53.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.686+0000 7fc0d0a00640 1 --2- 172.21.15.90:0/1006230309 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0b0004060 0x7fc0cc0b5cb0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc0018a0 tx=0x7fc0bc00dc50 comp rx=0 tx=0).stop 2024-03-24T01:26:53.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.686+0000 7fc0d0a00640 1 -- 172.21.15.90:0/1006230309 shutdown_connections 2024-03-24T01:26:53.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.686+0000 7fc0d0a00640 1 --2- 172.21.15.90:0/1006230309 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc0ac06bc00 0x7fc0ac06e0c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:53.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.686+0000 7fc0d0a00640 1 --2- 172.21.15.90:0/1006230309 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0cc1e93d0 0x7fc0cc1e9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:53.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.686+0000 7fc0d0a00640 1 --2- 172.21.15.90:0/1006230309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0cc160b10 0x7fc0cc160f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:53.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.686+0000 7fc0d0a00640 1 --2- 172.21.15.90:0/1006230309 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0b0004060 0x7fc0cc0b5cb0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:53.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.686+0000 7fc0d0a00640 1 -- 172.21.15.90:0/1006230309 >> 172.21.15.90:0/1006230309 conn(0x7fc0cc0d0560 msgr2=0x7fc0cc0cd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:53.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.686+0000 7fc0d0a00640 1 -- 172.21.15.90:0/1006230309 shutdown_connections 2024-03-24T01:26:53.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.686+0000 7fc0d0a00640 1 -- 172.21.15.90:0/1006230309 wait complete. 2024-03-24T01:26:53.714 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T01:25:57.398864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T01:25:58.451150+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T01:26:53.714 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:26:53.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.923+0000 7fb9a8e00640 1 Processor -- start 2024-03-24T01:26:53.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.923+0000 7fb9a8e00640 1 -- start start 2024-03-24T01:26:53.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.925+0000 7fb9a8e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9a4145170 0x7fb9a40cf810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:53.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.925+0000 7fb9a8e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb9a406ea30 0x7fb9a40cfd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:53.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.925+0000 7fb9a8e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a406d990 con 0x7fb9a40cea80 2024-03-24T01:26:53.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.925+0000 7fb9a8e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9a406dc70 con 0x7fb9a406ea30 2024-03-24T01:26:53.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.925+0000 7fb9a8e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a406df50 con 0x7fb9a4145170 2024-03-24T01:26:53.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.925+0000 7fb9a2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb9a406ea30 0x7fb9a40cfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:53.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.925+0000 7fb9a3e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fb9a40cea80 0x7fb9a40cee90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53120/0 (socket says 172.21.15.90:53120) 2024-03-24T01:26:53.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.925+0000 7fb9a3e00640 1 -- 172.21.15.90:0/1351996481 learned_addr learned my addr 172.21.15.90:0/1351996481 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:26:53.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.925+0000 7fb9a3400640 1 --2- 172.21.15.90:0/1351996481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9a4145170 0x7fb9a40cf810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:53.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.925+0000 7fb9a2a00640 1 -- 172.21.15.90:0/1351996481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9a4145170 msgr2=0x7fb9a40cf810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:53.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.925+0000 7fb9a2a00640 1 --2- 172.21.15.90:0/1351996481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9a4145170 0x7fb9a40cf810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:53.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.925+0000 7fb9a2a00640 1 -- 172.21.15.90:0/1351996481 >> v1:172.21.15.90:6789/0 conn(0x7fb9a40cea80 legacy=0x7fb9a40cee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:53.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.925+0000 7fb9a2a00640 1 -- 172.21.15.90:0/1351996481 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a406e230 con 0x7fb9a406ea30 2024-03-24T01:26:53.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.927+0000 7fb9a2a00640 1 --2- 172.21.15.90:0/1351996481 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9a406ea30 0x7fb9a40cfd50 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fb9900097a0 tx=0x7fb99002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e217a43b7403f58 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:53.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.927+0000 7fb9a2000640 1 -- 172.21.15.90:0/1351996481 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb990035020 con 0x7fb9a406ea30 2024-03-24T01:26:53.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.927+0000 7fb9a2000640 1 -- 172.21.15.90:0/1351996481 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb990031810 con 0x7fb9a406ea30 2024-03-24T01:26:53.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.927+0000 7fb9a2000640 1 -- 172.21.15.90:0/1351996481 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb990031be0 con 0x7fb9a406ea30 2024-03-24T01:26:53.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.927+0000 7fb9a8e00640 1 -- 172.21.15.90:0/1351996481 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9a406ea30 msgr2=0x7fb9a40cfd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:53.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.927+0000 7fb9a8e00640 1 --2- 172.21.15.90:0/1351996481 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9a406ea30 0x7fb9a40cfd50 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fb9900097a0 tx=0x7fb99002b8b0 comp rx=0 tx=0).stop 2024-03-24T01:26:53.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.928+0000 7fb9a8e00640 1 -- 172.21.15.90:0/1351996481 shutdown_connections 2024-03-24T01:26:53.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.928+0000 7fb9a8e00640 1 --2- 172.21.15.90:0/1351996481 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9a406ea30 0x7fb9a40cfd50 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:53.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.928+0000 7fb9a8e00640 1 --2- 172.21.15.90:0/1351996481 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9a4145170 0x7fb9a40cf810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:53.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.928+0000 7fb9a8e00640 1 -- 172.21.15.90:0/1351996481 >> 172.21.15.90:0/1351996481 conn(0x7fb9a406b0c0 msgr2=0x7fb9a406b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:53.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.928+0000 7fb9a8e00640 1 -- 172.21.15.90:0/1351996481 shutdown_connections 2024-03-24T01:26:53.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.928+0000 7fb9a8e00640 1 -- 172.21.15.90:0/1351996481 wait complete. 2024-03-24T01:26:53.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.929+0000 7fb9a8e00640 1 Processor -- start 2024-03-24T01:26:53.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.929+0000 7fb9a8e00640 1 -- start start 2024-03-24T01:26:53.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.930+0000 7fb9a8e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9a406ea30 0x7fb9a41f3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:53.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.930+0000 7fb9a8e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9a40cea80 0x7fb9a40aff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:53.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.930+0000 7fb9a8e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9a4145170 0x7fb9a40b3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:53.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.930+0000 7fb9a8e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a41456d0 con 0x7fb9a406ea30 2024-03-24T01:26:53.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.930+0000 7fb9a8e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a4145550 con 0x7fb9a4145170 2024-03-24T01:26:53.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.930+0000 7fb9a8e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a415c0a0 con 0x7fb9a40cea80 2024-03-24T01:26:53.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.931+0000 7fb9a3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9a406ea30 0x7fb9a41f3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:53.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.931+0000 7fb9a3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9a406ea30 0x7fb9a41f3a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35532/0 (socket says 172.21.15.90:35532) 2024-03-24T01:26:53.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.931+0000 7fb9a3400640 1 -- 172.21.15.90:0/640972948 learned_addr learned my addr 172.21.15.90:0/640972948 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:53.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.931+0000 7fb9a3400640 1 -- 172.21.15.90:0/640972948 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9a40cea80 msgr2=0x7fb9a40aff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:53.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.931+0000 7fb9a3400640 1 --2- 172.21.15.90:0/640972948 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9a40cea80 0x7fb9a40aff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:53.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.931+0000 7fb9a3400640 1 -- 172.21.15.90:0/640972948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9a4145170 msgr2=0x7fb9a40b3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:53.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.931+0000 7fb9a3400640 1 --2- 172.21.15.90:0/640972948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9a4145170 0x7fb9a40b3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:53.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.931+0000 7fb9a3400640 1 -- 172.21.15.90:0/640972948 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a40c8bd0 con 0x7fb9a406ea30 2024-03-24T01:26:53.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.931+0000 7fb9a3400640 1 --2- 172.21.15.90:0/640972948 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9a406ea30 0x7fb9a41f3a40 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fb99400d5f0 tx=0x7fb99400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:53.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.932+0000 7fb99b400640 1 -- 172.21.15.90:0/640972948 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9940125f0 con 0x7fb9a406ea30 2024-03-24T01:26:53.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.932+0000 7fb99b400640 1 -- 172.21.15.90:0/640972948 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb994019070 con 0x7fb9a406ea30 2024-03-24T01:26:53.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.932+0000 7fb9a8e00640 1 -- 172.21.15.90:0/640972948 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9a40c8eb0 con 0x7fb9a406ea30 2024-03-24T01:26:53.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.932+0000 7fb99b400640 1 -- 172.21.15.90:0/640972948 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9940129d0 con 0x7fb9a406ea30 2024-03-24T01:26:53.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.932+0000 7fb9a8e00640 1 -- 172.21.15.90:0/640972948 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9a40c9470 con 0x7fb9a406ea30 2024-03-24T01:26:53.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.933+0000 7fb9a8e00640 1 -- 172.21.15.90:0/640972948 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9a404f400 con 0x7fb9a406ea30 2024-03-24T01:26:53.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.934+0000 7fb99b400640 1 -- 172.21.15.90:0/640972948 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb994012b70 con 0x7fb9a406ea30 2024-03-24T01:26:53.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.935+0000 7fb99b400640 1 --2- 172.21.15.90:0/640972948 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb98406bbd0 0x7fb98406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:53.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.935+0000 7fb99b400640 1 -- 172.21.15.90:0/640972948 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6216+0+0 (secure 0 0 0) 0x7fb99408b140 con 0x7fb9a406ea30 2024-03-24T01:26:53.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.936+0000 7fb9a2a00640 1 --2- 172.21.15.90:0/640972948 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb98406bbd0 0x7fb98406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:53.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.937+0000 7fb9a2a00640 1 --2- 172.21.15.90:0/640972948 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb98406bbd0 0x7fb98406e090 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb990036000 tx=0x7fb990030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:53.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:53.940+0000 7fb99b400640 1 -- 172.21.15.90:0/640972948 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb994059990 con 0x7fb9a406ea30 2024-03-24T01:26:54.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.136+0000 7fb9a8e00640 1 -- 172.21.15.90:0/640972948 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb9a40cbc30 con 0x7fb9a406ea30 2024-03-24T01:26:54.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.137+0000 7fb99b400640 1 -- 172.21.15.90:0/640972948 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb994059290 con 0x7fb9a406ea30 2024-03-24T01:26:54.138 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:26:54.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.139+0000 7fb9a8e00640 1 -- 172.21.15.90:0/640972948 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb98406bbd0 msgr2=0x7fb98406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:54.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.139+0000 7fb9a8e00640 1 --2- 172.21.15.90:0/640972948 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb98406bbd0 0x7fb98406e090 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb990036000 tx=0x7fb990030000 comp rx=0 tx=0).stop 2024-03-24T01:26:54.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.140+0000 7fb9a8e00640 1 -- 172.21.15.90:0/640972948 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9a406ea30 msgr2=0x7fb9a41f3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:54.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.140+0000 7fb9a8e00640 1 --2- 172.21.15.90:0/640972948 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9a406ea30 0x7fb9a41f3a40 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fb99400d5f0 tx=0x7fb99400dab0 comp rx=0 tx=0).stop 2024-03-24T01:26:54.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.140+0000 7fb9a8e00640 1 -- 172.21.15.90:0/640972948 shutdown_connections 2024-03-24T01:26:54.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.140+0000 7fb9a8e00640 1 --2- 172.21.15.90:0/640972948 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb98406bbd0 0x7fb98406e090 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:54.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.140+0000 7fb9a8e00640 1 --2- 172.21.15.90:0/640972948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9a4145170 0x7fb9a40b3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:54.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.140+0000 7fb9a8e00640 1 --2- 172.21.15.90:0/640972948 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb9a40cea80 0x7fb9a40aff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:54.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.140+0000 7fb9a8e00640 1 --2- 172.21.15.90:0/640972948 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9a406ea30 0x7fb9a41f3a40 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:54.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.140+0000 7fb9a8e00640 1 -- 172.21.15.90:0/640972948 >> 172.21.15.90:0/640972948 conn(0x7fb9a406b0c0 msgr2=0x7fb9a40b1d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:54.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.140+0000 7fb9a8e00640 1 -- 172.21.15.90:0/640972948 shutdown_connections 2024-03-24T01:26:54.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.140+0000 7fb9a8e00640 1 -- 172.21.15.90:0/640972948 wait complete. 2024-03-24T01:26:54.158 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T01:26:54.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.354+0000 7f1684400640 1 Processor -- start 2024-03-24T01:26:54.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.354+0000 7f1684400640 1 -- start start 2024-03-24T01:26:54.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.356+0000 7f1684400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f167c15e0e0 0x7f167c099cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:54.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.356+0000 7f1684400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f167c0d0870 0x7f167c09a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:54.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.356+0000 7f1684400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f167c0cc690 0x7f167c0ccb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:54.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.356+0000 7f1684400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f167c0cc200 con 0x7f167c15e0e0 2024-03-24T01:26:54.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.356+0000 7f1684400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f167c0d1600 con 0x7f167c0d0870 2024-03-24T01:26:54.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.356+0000 7f1684400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f167c0d0d60 con 0x7f167c0cc690 2024-03-24T01:26:54.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.356+0000 7f1681a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f167c15e0e0 0x7f167c099cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:54.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.356+0000 7f1681a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f167c15e0e0 0x7f167c099cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35538/0 (socket says 172.21.15.90:35538) 2024-03-24T01:26:54.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.356+0000 7f1681a00640 1 -- 172.21.15.90:0/2788072491 learned_addr learned my addr 172.21.15.90:0/2788072491 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:54.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.357+0000 7f1682400640 1 --2- 172.21.15.90:0/2788072491 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f167c0cc690 0x7f167c0ccb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:54.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.357+0000 7f1681000640 1 --2- 172.21.15.90:0/2788072491 >> v2:172.21.15.120:3300/0 conn(0x7f167c0d0870 0x7f167c09a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:54.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.357+0000 7f1681a00640 1 -- 172.21.15.90:0/2788072491 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f167c0cc690 msgr2=0x7f167c0ccb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:54.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.357+0000 7f1681a00640 1 --2- 172.21.15.90:0/2788072491 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f167c0cc690 0x7f167c0ccb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:54.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.357+0000 7f1681a00640 1 -- 172.21.15.90:0/2788072491 >> v2:172.21.15.120:3300/0 conn(0x7f167c0d0870 msgr2=0x7f167c09a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:54.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.357+0000 7f1681a00640 1 --2- 172.21.15.90:0/2788072491 >> v2:172.21.15.120:3300/0 conn(0x7f167c0d0870 0x7f167c09a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:54.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.357+0000 7f1681a00640 1 -- 172.21.15.90:0/2788072491 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f167c0d1040 con 0x7f167c15e0e0 2024-03-24T01:26:54.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.357+0000 7f1682400640 1 --2- 172.21.15.90:0/2788072491 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f167c0cc690 0x7f167c0ccb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:54.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.357+0000 7f1681000640 1 --2- 172.21.15.90:0/2788072491 >> v2:172.21.15.120:3300/0 conn(0x7f167c0d0870 0x7f167c09a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:54.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.358+0000 7f1681a00640 1 --2- 172.21.15.90:0/2788072491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f167c15e0e0 0x7f167c099cd0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f167c0ce440 tx=0x7f167002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7cbc0b0c483c46b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:54.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.358+0000 7f167be00640 1 -- 172.21.15.90:0/2788072491 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167002c9e0 con 0x7f167c15e0e0 2024-03-24T01:26:54.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.358+0000 7f167be00640 1 -- 172.21.15.90:0/2788072491 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f167002cb80 con 0x7f167c15e0e0 2024-03-24T01:26:54.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.358+0000 7f167be00640 1 -- 172.21.15.90:0/2788072491 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16700343f0 con 0x7f167c15e0e0 2024-03-24T01:26:54.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.359+0000 7f1684400640 1 -- 172.21.15.90:0/2788072491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f167c15e0e0 msgr2=0x7f167c099cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:54.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.359+0000 7f1684400640 1 --2- 172.21.15.90:0/2788072491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f167c15e0e0 0x7f167c099cd0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f167c0ce440 tx=0x7f167002be40 comp rx=0 tx=0).stop 2024-03-24T01:26:54.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.359+0000 7f1684400640 1 -- 172.21.15.90:0/2788072491 shutdown_connections 2024-03-24T01:26:54.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.359+0000 7f1684400640 1 --2- 172.21.15.90:0/2788072491 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f167c0cc690 0x7f167c0ccb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:54.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.359+0000 7f1684400640 1 --2- 172.21.15.90:0/2788072491 >> v2:172.21.15.120:3300/0 conn(0x7f167c0d0870 0x7f167c09a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:54.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.359+0000 7f1684400640 1 --2- 172.21.15.90:0/2788072491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f167c15e0e0 0x7f167c099cd0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:54.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.359+0000 7f1684400640 1 -- 172.21.15.90:0/2788072491 >> 172.21.15.90:0/2788072491 conn(0x7f167c0b34e0 msgr2=0x7f167c0b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:54.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.359+0000 7f1684400640 1 -- 172.21.15.90:0/2788072491 shutdown_connections 2024-03-24T01:26:54.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.359+0000 7f1684400640 1 -- 172.21.15.90:0/2788072491 wait complete. 2024-03-24T01:26:54.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.361+0000 7f1684400640 1 Processor -- start 2024-03-24T01:26:54.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.361+0000 7f1684400640 1 -- start start 2024-03-24T01:26:54.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.362+0000 7f1684400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f167c0cc690 0x7f167c151f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:54.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.362+0000 7f1684400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f167c0d0870 0x7f167c0b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:54.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.362+0000 7f1684400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f167c15e0e0 0x7f167c0b1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:54.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.362+0000 7f1684400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f167c15e4c0 con 0x7f167c0d0870 2024-03-24T01:26:54.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.362+0000 7f1684400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f167c0b2f70 con 0x7f167c0cc690 2024-03-24T01:26:54.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.362+0000 7f1684400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f167c15e640 con 0x7f167c15e0e0 2024-03-24T01:26:54.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.362+0000 7f1681000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f167c0d0870 0x7f167c0b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:54.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.362+0000 7f1681000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f167c0d0870 0x7f167c0b0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35546/0 (socket says 172.21.15.90:35546) 2024-03-24T01:26:54.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.362+0000 7f1681000640 1 -- 172.21.15.90:0/2859443090 learned_addr learned my addr 172.21.15.90:0/2859443090 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:54.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.363+0000 7f1682400640 1 --2- 172.21.15.90:0/2859443090 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f167c15e0e0 0x7f167c0b1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:54.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.363+0000 7f1681a00640 1 --2- 172.21.15.90:0/2859443090 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f167c0cc690 0x7f167c151f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:54.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.363+0000 7f1681000640 1 -- 172.21.15.90:0/2859443090 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f167c15e0e0 msgr2=0x7f167c0b1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:54.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.363+0000 7f1681000640 1 --2- 172.21.15.90:0/2859443090 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f167c15e0e0 0x7f167c0b1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:54.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.363+0000 7f1681000640 1 -- 172.21.15.90:0/2859443090 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f167c0cc690 msgr2=0x7f167c151f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:54.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.363+0000 7f1681000640 1 --2- 172.21.15.90:0/2859443090 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f167c0cc690 0x7f167c151f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:54.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.363+0000 7f1681000640 1 -- 172.21.15.90:0/2859443090 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f167c0af450 con 0x7f167c0d0870 2024-03-24T01:26:54.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.363+0000 7f1682400640 1 --2- 172.21.15.90:0/2859443090 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f167c15e0e0 0x7f167c0b1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:54.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.363+0000 7f1681000640 1 --2- 172.21.15.90:0/2859443090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f167c0d0870 0x7f167c0b0e40 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f166800c7f0 tx=0x7f166800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:54.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.363+0000 7f1679600640 1 -- 172.21.15.90:0/2859443090 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1668015070 con 0x7f167c0d0870 2024-03-24T01:26:54.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.363+0000 7f1679600640 1 -- 172.21.15.90:0/2859443090 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1668007820 con 0x7f167c0d0870 2024-03-24T01:26:54.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.363+0000 7f1679600640 1 -- 172.21.15.90:0/2859443090 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1668007bd0 con 0x7f167c0d0870 2024-03-24T01:26:54.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.363+0000 7f1684400640 1 -- 172.21.15.90:0/2859443090 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f167c0af730 con 0x7f167c0d0870 2024-03-24T01:26:54.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.363+0000 7f1684400640 1 -- 172.21.15.90:0/2859443090 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f167c0afcf0 con 0x7f167c0d0870 2024-03-24T01:26:54.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.365+0000 7f1684400640 1 -- 172.21.15.90:0/2859443090 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f167c04f400 con 0x7f167c0d0870 2024-03-24T01:26:54.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.366+0000 7f1679600640 1 -- 172.21.15.90:0/2859443090 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1668018070 con 0x7f167c0d0870 2024-03-24T01:26:54.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.366+0000 7f1679600640 1 --2- 172.21.15.90:0/2859443090 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f165c06bbd0 0x7f165c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:54.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.366+0000 7f1679600640 1 -- 172.21.15.90:0/2859443090 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f166808acb0 con 0x7f167c0d0870 2024-03-24T01:26:54.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.367+0000 7f1681a00640 1 --2- 172.21.15.90:0/2859443090 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f165c06bbd0 0x7f165c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:54.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.368+0000 7f1681a00640 1 --2- 172.21.15.90:0/2859443090 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f165c06bbd0 0x7f165c06e090 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1670035000 tx=0x7f1670030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:54.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.371+0000 7f1679600640 1 -- 172.21.15.90:0/2859443090 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1668059630 con 0x7f167c0d0870 2024-03-24T01:26:54.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.566+0000 7f1684400640 1 -- 172.21.15.90:0/2859443090 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f167c0b3220 con 0x7f167c0d0870 2024-03-24T01:26:54.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.567+0000 7f1679600640 1 -- 172.21.15.90:0/2859443090 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1668058f30 con 0x7f167c0d0870 2024-03-24T01:26:54.568 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:26:54.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.570+0000 7f1684400640 1 -- 172.21.15.90:0/2859443090 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f165c06bbd0 msgr2=0x7f165c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:54.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.570+0000 7f1684400640 1 --2- 172.21.15.90:0/2859443090 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f165c06bbd0 0x7f165c06e090 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1670035000 tx=0x7f1670030000 comp rx=0 tx=0).stop 2024-03-24T01:26:54.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.570+0000 7f1684400640 1 -- 172.21.15.90:0/2859443090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f167c0d0870 msgr2=0x7f167c0b0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:54.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.570+0000 7f1684400640 1 --2- 172.21.15.90:0/2859443090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f167c0d0870 0x7f167c0b0e40 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f166800c7f0 tx=0x7f166800ccb0 comp rx=0 tx=0).stop 2024-03-24T01:26:54.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.570+0000 7f1684400640 1 -- 172.21.15.90:0/2859443090 shutdown_connections 2024-03-24T01:26:54.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.570+0000 7f1684400640 1 --2- 172.21.15.90:0/2859443090 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f165c06bbd0 0x7f165c06e090 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:54.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.570+0000 7f1684400640 1 --2- 172.21.15.90:0/2859443090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f167c0d0870 0x7f167c0b0e40 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:54.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.570+0000 7f1684400640 1 --2- 172.21.15.90:0/2859443090 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f167c15e0e0 0x7f167c0b1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:54.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.570+0000 7f1684400640 1 --2- 172.21.15.90:0/2859443090 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f167c0cc690 0x7f167c151f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:54.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.570+0000 7f1684400640 1 -- 172.21.15.90:0/2859443090 >> 172.21.15.90:0/2859443090 conn(0x7f167c0b34e0 msgr2=0x7f167c070a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:54.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.570+0000 7f1684400640 1 -- 172.21.15.90:0/2859443090 shutdown_connections 2024-03-24T01:26:54.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.570+0000 7f1684400640 1 -- 172.21.15.90:0/2859443090 wait complete. 2024-03-24T01:26:54.589 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T01:26:54.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.794+0000 7fd474800640 1 Processor -- start 2024-03-24T01:26:54.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.794+0000 7fd474800640 1 -- start start 2024-03-24T01:26:54.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.795+0000 7fd474800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd46c0cc340 0x7fd46c06f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:54.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.795+0000 7fd474800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd46c067120 con 0x7fd46c159620 2024-03-24T01:26:54.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.795+0000 7fd474800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd46c06c9f0 con 0x7fd46c06fc20 2024-03-24T01:26:54.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.795+0000 7fd474800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd46c070070 con 0x7fd46c0cc340 2024-03-24T01:26:54.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.796+0000 7fd471e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd46c159620 0x7fd46c06c2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53132/0 (socket says 172.21.15.90:53132) 2024-03-24T01:26:54.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.796+0000 7fd471e00640 1 -- 172.21.15.90:0/1782520351 learned_addr learned my addr 172.21.15.90:0/1782520351 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:26:54.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.796+0000 7fd470a00640 1 -- 172.21.15.90:0/1782520351 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4264168976 0 0) 0x7fd46c067120 con 0x7fd46c159620 2024-03-24T01:26:54.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.796+0000 7fd470a00640 1 -- 172.21.15.90:0/1782520351 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd454003520 con 0x7fd46c159620 2024-03-24T01:26:54.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.796+0000 7fd471400640 1 --2- 172.21.15.90:0/1782520351 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd46c0cc340 0x7fd46c06f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:54.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.796+0000 7fd470a00640 1 -- 172.21.15.90:0/1782520351 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2851201784 0 0) 0x7fd454003520 con 0x7fd46c159620 2024-03-24T01:26:54.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.796+0000 7fd470a00640 1 -- 172.21.15.90:0/1782520351 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd46c0cc340 msgr2=0x7fd46c06f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:54.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.796+0000 7fd470a00640 1 --2- 172.21.15.90:0/1782520351 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd46c0cc340 0x7fd46c06f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:54.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.796+0000 7fd470a00640 1 -- 172.21.15.90:0/1782520351 >> v1:172.21.15.120:6789/0 conn(0x7fd46c06fc20 legacy=0x7fd46c15c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:54.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.796+0000 7fd470a00640 1 -- 172.21.15.90:0/1782520351 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd46c0cc780 con 0x7fd46c159620 2024-03-24T01:26:54.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.796+0000 7fd470a00640 1 -- 172.21.15.90:0/1782520351 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7fd460001690 con 0x7fd46c159620 2024-03-24T01:26:54.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.796+0000 7fd470a00640 1 -- 172.21.15.90:0/1782520351 >> v1:172.21.15.90:6789/0 conn(0x7fd46c159620 legacy=0x7fd46c06c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:54.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.797+0000 7fd470a00640 1 --2- 172.21.15.90:0/1782520351 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd46c0cc340 0x7fd4540039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:54.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.797+0000 7fd470a00640 1 --2- 172.21.15.90:0/1782520351 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd454004010 0x7fd454004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:54.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.797+0000 7fd470a00640 1 --2- 172.21.15.90:0/1782520351 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd454004d40 0x7fd4540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:54.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.797+0000 7fd470a00640 1 -- 172.21.15.90:0/1782520351 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46c070070 con 0x7fd454004010 2024-03-24T01:26:54.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.797+0000 7fd470a00640 1 -- 172.21.15.90:0/1782520351 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd45400bc00 con 0x7fd454004d40 2024-03-24T01:26:54.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.797+0000 7fd470a00640 1 -- 172.21.15.90:0/1782520351 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd45400be00 con 0x7fd46c0cc340 2024-03-24T01:26:54.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.797+0000 7fd471400640 1 --2- 172.21.15.90:0/1782520351 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd454004010 0x7fd454004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:54.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.797+0000 7fd471e00640 1 --2- 172.21.15.90:0/1782520351 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd46c0cc340 0x7fd4540039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:54.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.797+0000 7fd471400640 1 -- 172.21.15.90:0/1782520351 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd46c0cc340 msgr2=0x7fd4540039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:54.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.797+0000 7fd471400640 1 --2- 172.21.15.90:0/1782520351 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd46c0cc340 0x7fd4540039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:54.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.797+0000 7fd471400640 1 -- 172.21.15.90:0/1782520351 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd454004d40 msgr2=0x7fd4540051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:54.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.797+0000 7fd471400640 1 --2- 172.21.15.90:0/1782520351 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd454004d40 0x7fd4540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:54.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.797+0000 7fd471400640 1 -- 172.21.15.90:0/1782520351 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd454003520 con 0x7fd454004010 2024-03-24T01:26:54.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.797+0000 7fd471e00640 1 --2- 172.21.15.90:0/1782520351 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd46c0cc340 0x7fd4540039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:54.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.798+0000 7fd471400640 1 --2- 172.21.15.90:0/1782520351 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd454004010 0x7fd454004800 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fd45c00ba30 tx=0x7fd454005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72677b0305ddd576 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:54.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.799+0000 7fd470a00640 1 -- 172.21.15.90:0/1782520351 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45c036020 con 0x7fd454004010 2024-03-24T01:26:54.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.799+0000 7fd470a00640 1 -- 172.21.15.90:0/1782520351 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd45c032640 con 0x7fd454004010 2024-03-24T01:26:54.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.799+0000 7fd470a00640 1 -- 172.21.15.90:0/1782520351 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45c00b600 con 0x7fd454004010 2024-03-24T01:26:54.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.799+0000 7fd474800640 1 -- 172.21.15.90:0/1782520351 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd454004010 msgr2=0x7fd454004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:54.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.799+0000 7fd474800640 1 --2- 172.21.15.90:0/1782520351 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd454004010 0x7fd454004800 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fd45c00ba30 tx=0x7fd454005830 comp rx=0 tx=0).stop 2024-03-24T01:26:54.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.799+0000 7fd472800640 1 -- 172.21.15.90:0/1782520351 reap_dead start 2024-03-24T01:26:54.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.799+0000 7fd474800640 1 -- 172.21.15.90:0/1782520351 shutdown_connections 2024-03-24T01:26:54.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.799+0000 7fd474800640 1 -- 172.21.15.90:0/1782520351 >> 172.21.15.90:0/1782520351 conn(0x7fd46c06e1a0 msgr2=0x7fd46c06e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:54.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.799+0000 7fd474800640 1 -- 172.21.15.90:0/1782520351 shutdown_connections 2024-03-24T01:26:54.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.799+0000 7fd474800640 1 -- 172.21.15.90:0/1782520351 wait complete. 2024-03-24T01:26:54.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.801+0000 7fd474800640 1 Processor -- start 2024-03-24T01:26:54.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.801+0000 7fd474800640 1 -- start start 2024-03-24T01:26:54.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.802+0000 7fd474800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd454004010 0x7fd46c145ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:54.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.802+0000 7fd474800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd46c0c94d0 0x7fd46c157990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:54.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.802+0000 7fd474800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd46c157ed0 0x7fd46c0ca330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:54.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.802+0000 7fd474800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46c15fe90 con 0x7fd46c157ed0 2024-03-24T01:26:54.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.802+0000 7fd474800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46c15fbb0 con 0x7fd46c0c94d0 2024-03-24T01:26:54.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.802+0000 7fd474800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd46c160170 con 0x7fd454004010 2024-03-24T01:26:54.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.802+0000 7fd472800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd46c157ed0 0x7fd46c0ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:54.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.802+0000 7fd472800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd46c157ed0 0x7fd46c0ca330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35566/0 (socket says 172.21.15.90:35566) 2024-03-24T01:26:54.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.802+0000 7fd472800640 1 -- 172.21.15.90:0/2869276141 learned_addr learned my addr 172.21.15.90:0/2869276141 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:54.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.803+0000 7fd472800640 1 -- 172.21.15.90:0/2869276141 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd454004010 msgr2=0x7fd46c145ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:54.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.803+0000 7fd472800640 1 --2- 172.21.15.90:0/2869276141 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd454004010 0x7fd46c145ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:54.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.803+0000 7fd472800640 1 -- 172.21.15.90:0/2869276141 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd46c0c94d0 msgr2=0x7fd46c157990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:54.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.803+0000 7fd471400640 1 --2- 172.21.15.90:0/2869276141 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd46c0c94d0 0x7fd46c157990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:54.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.803+0000 7fd472800640 1 --2- 172.21.15.90:0/2869276141 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd46c0c94d0 0x7fd46c157990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:54.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.803+0000 7fd472800640 1 -- 172.21.15.90:0/2869276141 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd46c146020 con 0x7fd46c157ed0 2024-03-24T01:26:54.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.803+0000 7fd472800640 1 --2- 172.21.15.90:0/2869276141 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd46c157ed0 0x7fd46c0ca330 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fd46c071b80 tx=0x7fd46800c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:54.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.803+0000 7fd466000640 1 -- 172.21.15.90:0/2869276141 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd468007070 con 0x7fd46c157ed0 2024-03-24T01:26:54.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.803+0000 7fd466000640 1 -- 172.21.15.90:0/2869276141 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd468007230 con 0x7fd46c157ed0 2024-03-24T01:26:54.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.803+0000 7fd474800640 1 -- 172.21.15.90:0/2869276141 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd46c1568f0 con 0x7fd46c157ed0 2024-03-24T01:26:54.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.803+0000 7fd466000640 1 -- 172.21.15.90:0/2869276141 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd468007660 con 0x7fd46c157ed0 2024-03-24T01:26:54.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.804+0000 7fd474800640 1 -- 172.21.15.90:0/2869276141 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd46c156eb0 con 0x7fd46c157ed0 2024-03-24T01:26:54.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.805+0000 7fd474800640 1 -- 172.21.15.90:0/2869276141 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd46c04f400 con 0x7fd46c157ed0 2024-03-24T01:26:54.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.806+0000 7fd466000640 1 -- 172.21.15.90:0/2869276141 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd468011070 con 0x7fd46c157ed0 2024-03-24T01:26:54.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.806+0000 7fd466000640 1 --2- 172.21.15.90:0/2869276141 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd45006bbd0 0x7fd45006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:54.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.806+0000 7fd466000640 1 -- 172.21.15.90:0/2869276141 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6216+0+0 (secure 0 0 0) 0x7fd468089f70 con 0x7fd46c157ed0 2024-03-24T01:26:54.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.807+0000 7fd471e00640 1 --2- 172.21.15.90:0/2869276141 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd45006bbd0 0x7fd45006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:54.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.808+0000 7fd471e00640 1 --2- 172.21.15.90:0/2869276141 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd45006bbd0 0x7fd45006e090 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd460001eb0 tx=0x7fd460009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:54.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:54.811+0000 7fd466000640 1 -- 172.21.15.90:0/2869276141 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd46c146020 con 0x7fd46c157ed0 2024-03-24T01:26:55.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.010+0000 7fd474800640 1 -- 172.21.15.90:0/2869276141 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd46c06d230 con 0x7fd46c157ed0 2024-03-24T01:26:55.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.011+0000 7fd466000640 1 -- 172.21.15.90:0/2869276141 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd468058850 con 0x7fd46c157ed0 2024-03-24T01:26:55.011 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-24T01:26:55.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.013+0000 7fd474800640 1 -- 172.21.15.90:0/2869276141 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd45006bbd0 msgr2=0x7fd45006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:55.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.013+0000 7fd474800640 1 --2- 172.21.15.90:0/2869276141 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd45006bbd0 0x7fd45006e090 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd460001eb0 tx=0x7fd460009040 comp rx=0 tx=0).stop 2024-03-24T01:26:55.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.013+0000 7fd474800640 1 -- 172.21.15.90:0/2869276141 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd46c157ed0 msgr2=0x7fd46c0ca330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:55.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.013+0000 7fd474800640 1 --2- 172.21.15.90:0/2869276141 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd46c157ed0 0x7fd46c0ca330 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fd46c071b80 tx=0x7fd46800c980 comp rx=0 tx=0).stop 2024-03-24T01:26:55.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.014+0000 7fd474800640 1 -- 172.21.15.90:0/2869276141 shutdown_connections 2024-03-24T01:26:55.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.014+0000 7fd474800640 1 --2- 172.21.15.90:0/2869276141 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd45006bbd0 0x7fd45006e090 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:55.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.014+0000 7fd474800640 1 --2- 172.21.15.90:0/2869276141 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd46c157ed0 0x7fd46c0ca330 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:55.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.014+0000 7fd474800640 1 --2- 172.21.15.90:0/2869276141 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd46c0c94d0 0x7fd46c157990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:55.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.014+0000 7fd474800640 1 --2- 172.21.15.90:0/2869276141 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd454004010 0x7fd46c145ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:55.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.014+0000 7fd474800640 1 -- 172.21.15.90:0/2869276141 >> 172.21.15.90:0/2869276141 conn(0x7fd46c06e1a0 msgr2=0x7fd46c0cc9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:55.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.014+0000 7fd474800640 1 -- 172.21.15.90:0/2869276141 shutdown_connections 2024-03-24T01:26:55.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.014+0000 7fd474800640 1 -- 172.21.15.90:0/2869276141 wait complete. 2024-03-24T01:26:55.033 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T01:26:55.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.236+0000 7fd820e00640 1 Processor -- start 2024-03-24T01:26:55.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.236+0000 7fd820e00640 1 -- start start 2024-03-24T01:26:55.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.238+0000 7fd820e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd81c0720a0 0x7fd81c072530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:55.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.238+0000 7fd820e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd81c0b2930 con 0x7fd81c06f4e0 2024-03-24T01:26:55.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.238+0000 7fd820e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd81c0b2c10 con 0x7fd81c15a1d0 2024-03-24T01:26:55.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.238+0000 7fd820e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c0b39d0 con 0x7fd81c0720a0 2024-03-24T01:26:55.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.239+0000 7fd819400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd81c06f4e0 0x7fd81c071990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53142/0 (socket says 172.21.15.90:53142) 2024-03-24T01:26:55.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.239+0000 7fd819400640 1 -- 172.21.15.90:0/3663323382 learned_addr learned my addr 172.21.15.90:0/3663323382 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:26:55.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.239+0000 7fd81a800640 1 --2- 172.21.15.90:0/3663323382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd81c0720a0 0x7fd81c072530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:55.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.239+0000 7fd818a00640 1 -- 172.21.15.90:0/3663323382 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1106703478 0 0) 0x7fd81c0b2930 con 0x7fd81c06f4e0 2024-03-24T01:26:55.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.239+0000 7fd818a00640 1 -- 172.21.15.90:0/3663323382 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd800003520 con 0x7fd81c06f4e0 2024-03-24T01:26:55.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.239+0000 7fd818a00640 1 -- 172.21.15.90:0/3663323382 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 287012989 0 0) 0x7fd800003520 con 0x7fd81c06f4e0 2024-03-24T01:26:55.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.240+0000 7fd818a00640 1 -- 172.21.15.90:0/3663323382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd81c0720a0 msgr2=0x7fd81c072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:55.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.240+0000 7fd818a00640 1 --2- 172.21.15.90:0/3663323382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd81c0720a0 0x7fd81c072530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:55.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.240+0000 7fd818a00640 1 -- 172.21.15.90:0/3663323382 >> v1:172.21.15.120:6789/0 conn(0x7fd81c15a1d0 legacy=0x7fd81c06edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:55.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.240+0000 7fd818a00640 1 -- 172.21.15.90:0/3663323382 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd81c0b2ef0 con 0x7fd81c06f4e0 2024-03-24T01:26:55.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.240+0000 7fd818a00640 1 -- 172.21.15.90:0/3663323382 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7fd808001670 con 0x7fd81c06f4e0 2024-03-24T01:26:55.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.240+0000 7fd818a00640 1 -- 172.21.15.90:0/3663323382 >> v1:172.21.15.90:6789/0 conn(0x7fd81c06f4e0 legacy=0x7fd81c071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:55.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.240+0000 7fd818a00640 1 --2- 172.21.15.90:0/3663323382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8000039e0 0x7fd800003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:55.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.240+0000 7fd818a00640 1 --2- 172.21.15.90:0/3663323382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd800004800 0x7fd800004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:55.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.240+0000 7fd818a00640 1 --2- 172.21.15.90:0/3663323382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8000051c0 0x7fd80000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:55.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.240+0000 7fd818a00640 1 -- 172.21.15.90:0/3663323382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c0b39d0 con 0x7fd8000051c0 2024-03-24T01:26:55.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.240+0000 7fd818a00640 1 -- 172.21.15.90:0/3663323382 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd80000c0c0 con 0x7fd800004800 2024-03-24T01:26:55.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.240+0000 7fd818a00640 1 -- 172.21.15.90:0/3663323382 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd80000c2c0 con 0x7fd8000039e0 2024-03-24T01:26:55.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.240+0000 7fd81a800640 1 --2- 172.21.15.90:0/3663323382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd81c0720a0 0x7fd81c072530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:26:55.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.240+0000 7fd81a800640 1 --2- 172.21.15.90:0/3663323382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8000051c0 0x7fd80000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:55.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.240+0000 7fd819e00640 1 --2- 172.21.15.90:0/3663323382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8000039e0 0x7fd800003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:55.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.240+0000 7fd819400640 1 --2- 172.21.15.90:0/3663323382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd800004800 0x7fd800004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:55.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.241+0000 7fd81a800640 1 -- 172.21.15.90:0/3663323382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8000039e0 msgr2=0x7fd800003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:55.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.241+0000 7fd81a800640 1 --2- 172.21.15.90:0/3663323382 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd8000039e0 0x7fd800003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:55.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.241+0000 7fd81a800640 1 -- 172.21.15.90:0/3663323382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd800004800 msgr2=0x7fd800004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:55.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.241+0000 7fd81a800640 1 --2- 172.21.15.90:0/3663323382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd800004800 0x7fd800004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:55.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.241+0000 7fd81a800640 1 -- 172.21.15.90:0/3663323382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd800003520 con 0x7fd8000051c0 2024-03-24T01:26:55.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.241+0000 7fd819400640 1 --2- 172.21.15.90:0/3663323382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd800004800 0x7fd800004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:55.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.242+0000 7fd81a800640 1 --2- 172.21.15.90:0/3663323382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8000051c0 0x7fd80000b9c0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fd81c072480 tx=0x7fd800005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d466244daa41367 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:55.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.242+0000 7fd818a00640 1 -- 172.21.15.90:0/3663323382 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd814032070 con 0x7fd8000051c0 2024-03-24T01:26:55.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.242+0000 7fd818a00640 1 -- 172.21.15.90:0/3663323382 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd814030070 con 0x7fd8000051c0 2024-03-24T01:26:55.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.242+0000 7fd818a00640 1 -- 172.21.15.90:0/3663323382 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd814009b50 con 0x7fd8000051c0 2024-03-24T01:26:55.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.243+0000 7fd820e00640 1 -- 172.21.15.90:0/3663323382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8000051c0 msgr2=0x7fd80000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:55.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.243+0000 7fd820e00640 1 --2- 172.21.15.90:0/3663323382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8000051c0 0x7fd80000b9c0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fd81c072480 tx=0x7fd800005830 comp rx=0 tx=0).stop 2024-03-24T01:26:55.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.243+0000 7fd81a800640 1 -- 172.21.15.90:0/3663323382 reap_dead start 2024-03-24T01:26:55.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.243+0000 7fd820e00640 1 -- 172.21.15.90:0/3663323382 shutdown_connections 2024-03-24T01:26:55.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.243+0000 7fd820e00640 1 -- 172.21.15.90:0/3663323382 >> 172.21.15.90:0/3663323382 conn(0x7fd81c0cf7e0 msgr2=0x7fd81c0cfbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:55.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.243+0000 7fd820e00640 1 -- 172.21.15.90:0/3663323382 shutdown_connections 2024-03-24T01:26:55.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.243+0000 7fd820e00640 1 -- 172.21.15.90:0/3663323382 wait complete. 2024-03-24T01:26:55.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.245+0000 7fd820e00640 1 Processor -- start 2024-03-24T01:26:55.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.246+0000 7fd820e00640 1 -- start start 2024-03-24T01:26:55.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.247+0000 7fd820e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8000051c0 0x7fd81c152a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:55.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.247+0000 7fd820e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd81c14e410 0x7fd81c0b4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:55.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.247+0000 7fd820e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd81c0cbd10 0x7fd81c0b4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:55.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.247+0000 7fd820e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c15a5b0 con 0x7fd81c0cbd10 2024-03-24T01:26:55.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.247+0000 7fd820e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c0b34b0 con 0x7fd8000051c0 2024-03-24T01:26:55.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.247+0000 7fd820e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c0ce300 con 0x7fd81c14e410 2024-03-24T01:26:55.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.247+0000 7fd81a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd81c0cbd10 0x7fd81c0b4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:55.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.247+0000 7fd81a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd81c0cbd10 0x7fd81c0b4540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35578/0 (socket says 172.21.15.90:35578) 2024-03-24T01:26:55.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.247+0000 7fd81a800640 1 -- 172.21.15.90:0/2349116950 learned_addr learned my addr 172.21.15.90:0/2349116950 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:55.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.247+0000 7fd819400640 1 --2- 172.21.15.90:0/2349116950 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd81c14e410 0x7fd81c0b4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:55.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.247+0000 7fd819e00640 1 --2- 172.21.15.90:0/2349116950 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8000051c0 0x7fd81c152a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:55.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.248+0000 7fd81a800640 1 -- 172.21.15.90:0/2349116950 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd81c14e410 msgr2=0x7fd81c0b4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:55.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.248+0000 7fd81a800640 1 --2- 172.21.15.90:0/2349116950 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd81c14e410 0x7fd81c0b4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:55.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.248+0000 7fd81a800640 1 -- 172.21.15.90:0/2349116950 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8000051c0 msgr2=0x7fd81c152a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:55.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.248+0000 7fd81a800640 1 --2- 172.21.15.90:0/2349116950 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8000051c0 0x7fd81c152a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:55.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.248+0000 7fd81a800640 1 -- 172.21.15.90:0/2349116950 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd81c14e980 con 0x7fd81c0cbd10 2024-03-24T01:26:55.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.248+0000 7fd81a800640 1 --2- 172.21.15.90:0/2349116950 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd81c0cbd10 0x7fd81c0b4540 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fd814003e10 tx=0x7fd814004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:55.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.248+0000 7fd812000640 1 -- 172.21.15.90:0/2349116950 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd814009b50 con 0x7fd81c0cbd10 2024-03-24T01:26:55.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.248+0000 7fd812000640 1 -- 172.21.15.90:0/2349116950 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd814004440 con 0x7fd81c0cbd10 2024-03-24T01:26:55.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.248+0000 7fd820e00640 1 -- 172.21.15.90:0/2349116950 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd81c14ec60 con 0x7fd81c0cbd10 2024-03-24T01:26:55.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.248+0000 7fd812000640 1 -- 172.21.15.90:0/2349116950 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd814004810 con 0x7fd81c0cbd10 2024-03-24T01:26:55.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.248+0000 7fd820e00640 1 -- 172.21.15.90:0/2349116950 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd81c14f220 con 0x7fd81c0cbd10 2024-03-24T01:26:55.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.250+0000 7fd820e00640 1 -- 172.21.15.90:0/2349116950 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd81c04f400 con 0x7fd81c0cbd10 2024-03-24T01:26:55.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.251+0000 7fd812000640 1 -- 172.21.15.90:0/2349116950 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd814037460 con 0x7fd81c0cbd10 2024-03-24T01:26:55.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.252+0000 7fd812000640 1 --2- 172.21.15.90:0/2349116950 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd7fc06bbd0 0x7fd7fc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:55.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.252+0000 7fd812000640 1 -- 172.21.15.90:0/2349116950 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6216+0+0 (secure 0 0 0) 0x7fd8140abed0 con 0x7fd81c0cbd10 2024-03-24T01:26:55.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.252+0000 7fd819e00640 1 --2- 172.21.15.90:0/2349116950 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd7fc06bbd0 0x7fd7fc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:55.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.254+0000 7fd819e00640 1 --2- 172.21.15.90:0/2349116950 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd7fc06bbd0 0x7fd7fc06e090 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd80c001550 tx=0x7fd80c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:55.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.256+0000 7fd812000640 1 -- 172.21.15.90:0/2349116950 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd81407a7c0 con 0x7fd81c0cbd10 2024-03-24T01:26:55.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.508+0000 7fd820e00640 1 -- 172.21.15.90:0/2349116950 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd81c06c970 con 0x7fd81c0cbd10 2024-03-24T01:26:55.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.509+0000 7fd812000640 1 -- 172.21.15.90:0/2349116950 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v42) v1 ==== 80+0+122 (secure 0 0 0) 0x7fd81407a0c0 con 0x7fd81c0cbd10 2024-03-24T01:26:55.509 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6835/2166652035 2024-03-25T01:26:27.648052+0000 2024-03-24T01:26:55.510 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6834/2166652035 2024-03-25T01:26:27.648052+0000 2024-03-24T01:26:55.510 INFO:teuthology.orchestra.run.smithi090.stderr:listed 2 entries 2024-03-24T01:26:55.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.511+0000 7fd820e00640 1 -- 172.21.15.90:0/2349116950 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd7fc06bbd0 msgr2=0x7fd7fc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:55.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.511+0000 7fd820e00640 1 --2- 172.21.15.90:0/2349116950 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd7fc06bbd0 0x7fd7fc06e090 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd80c001550 tx=0x7fd80c009040 comp rx=0 tx=0).stop 2024-03-24T01:26:55.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.512+0000 7fd820e00640 1 -- 172.21.15.90:0/2349116950 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd81c0cbd10 msgr2=0x7fd81c0b4540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:55.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.512+0000 7fd820e00640 1 --2- 172.21.15.90:0/2349116950 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd81c0cbd10 0x7fd81c0b4540 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fd814003e10 tx=0x7fd814004ab0 comp rx=0 tx=0).stop 2024-03-24T01:26:55.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.512+0000 7fd820e00640 1 -- 172.21.15.90:0/2349116950 shutdown_connections 2024-03-24T01:26:55.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.512+0000 7fd820e00640 1 --2- 172.21.15.90:0/2349116950 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd7fc06bbd0 0x7fd7fc06e090 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:55.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.512+0000 7fd820e00640 1 --2- 172.21.15.90:0/2349116950 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd81c0cbd10 0x7fd81c0b4540 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:55.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.512+0000 7fd820e00640 1 --2- 172.21.15.90:0/2349116950 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd81c14e410 0x7fd81c0b4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:55.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.512+0000 7fd820e00640 1 --2- 172.21.15.90:0/2349116950 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8000051c0 0x7fd81c152a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:55.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.512+0000 7fd820e00640 1 -- 172.21.15.90:0/2349116950 >> 172.21.15.90:0/2349116950 conn(0x7fd81c0cf7e0 msgr2=0x7fd81c0cf050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:55.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.512+0000 7fd820e00640 1 -- 172.21.15.90:0/2349116950 shutdown_connections 2024-03-24T01:26:55.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:55.512+0000 7fd820e00640 1 -- 172.21.15.90:0/2349116950 wait complete. 2024-03-24T01:26:55.530 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi090.front.sepia.ceph.com... 2024-03-24T01:26:55.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:55.530 DEBUG:teuthology.orchestra.run.smithi090:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-24T01:26:55.605 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[56445]: fuse finished with error 0 and tester_r 0 2024-03-24T01:26:55.611 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-03-24T01:26:55.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:55.611 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T01:26:55.794 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:26:55.794 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:26:55.844 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T01:26:55.844 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2024-03-24T01:26:55.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:55.844 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-24T01:26:55.938 INFO:teuthology.orchestra.run.smithi120.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-24T01:26:55.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T01:26:55.939 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:26:55.939 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:26:55.987 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053414,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498340k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249168k,nr_inodes=812292,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498340k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T01:26:55.987 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-03-24T01:26:55.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:55.987 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-24T01:26:56.074 INFO:teuthology.orchestra.run.smithi090.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-24T01:26:56.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T01:26:56.075 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:26:56.075 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:26:56.130 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T01:26:56.131 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2024-03-24T01:26:56.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:26:56.131 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-24T01:26:56.180 INFO:teuthology.orchestra.run.smithi120.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-24T01:26:56.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T01:26:56.181 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:26:56.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.394+0000 7ff1ec600640 1 Processor -- start 2024-03-24T01:26:56.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.394+0000 7ff1ec600640 1 -- start start 2024-03-24T01:26:56.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.397+0000 7ff1ec600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff1e406b7d0 0x7ff1e41629e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:56.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.397+0000 7ff1ec600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1e4165ed0 con 0x7ff1e40af040 2024-03-24T01:26:56.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.397+0000 7ff1ec600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1e41661b0 con 0x7ff1e4162f20 2024-03-24T01:26:56.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.397+0000 7ff1ec600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1e4166490 con 0x7ff1e406b7d0 2024-03-24T01:26:56.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.397+0000 7ff1e9c00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff1e40af040 0x7ff1e406b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53152/0 (socket says 172.21.15.90:53152) 2024-03-24T01:26:56.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.397+0000 7ff1e9c00640 1 -- 172.21.15.90:0/1764686009 learned_addr learned my addr 172.21.15.90:0/1764686009 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:26:56.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.398+0000 7ff1e9200640 1 --2- 172.21.15.90:0/1764686009 >> v2:172.21.15.120:3300/0 conn(0x7ff1e406b7d0 0x7ff1e41629e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:56.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.398+0000 7ff1e3e00640 1 -- 172.21.15.90:0/1764686009 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3278745701 0 0) 0x7ff1e4165ed0 con 0x7ff1e40af040 2024-03-24T01:26:56.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.398+0000 7ff1e3e00640 1 -- 172.21.15.90:0/1764686009 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1cc003520 con 0x7ff1e40af040 2024-03-24T01:26:56.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.398+0000 7ff1e3e00640 1 -- 172.21.15.90:0/1764686009 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 992476211 0 0) 0x7ff1cc003520 con 0x7ff1e40af040 2024-03-24T01:26:56.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.398+0000 7ff1e3e00640 1 -- 172.21.15.90:0/1764686009 >> v2:172.21.15.120:3300/0 conn(0x7ff1e406b7d0 msgr2=0x7ff1e41629e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:56.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.398+0000 7ff1e3e00640 1 --2- 172.21.15.90:0/1764686009 >> v2:172.21.15.120:3300/0 conn(0x7ff1e406b7d0 0x7ff1e41629e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:56.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.398+0000 7ff1e3e00640 1 -- 172.21.15.90:0/1764686009 >> v1:172.21.15.120:6789/0 conn(0x7ff1e4162f20 legacy=0x7ff1e415f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:56.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.398+0000 7ff1e3e00640 1 -- 172.21.15.90:0/1764686009 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e4166770 con 0x7ff1e40af040 2024-03-24T01:26:56.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.398+0000 7ff1e3e00640 1 -- 172.21.15.90:0/1764686009 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7ff1d8001670 con 0x7ff1e40af040 2024-03-24T01:26:56.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.398+0000 7ff1e3e00640 1 -- 172.21.15.90:0/1764686009 >> v1:172.21.15.90:6789/0 conn(0x7ff1e40af040 legacy=0x7ff1e406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:56.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.398+0000 7ff1e3e00640 1 --2- 172.21.15.90:0/1764686009 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1cc003a00 0x7ff1cc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:56.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.398+0000 7ff1e3e00640 1 --2- 172.21.15.90:0/1764686009 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1cc004800 0x7ff1cc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:56.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.398+0000 7ff1e3e00640 1 --2- 172.21.15.90:0/1764686009 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1cc0051c0 0x7ff1cc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:56.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.399+0000 7ff1e3e00640 1 -- 172.21.15.90:0/1764686009 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e4166490 con 0x7ff1cc004800 2024-03-24T01:26:56.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.399+0000 7ff1e3e00640 1 -- 172.21.15.90:0/1764686009 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc00c0c0 con 0x7ff1cc003a00 2024-03-24T01:26:56.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.399+0000 7ff1e3e00640 1 -- 172.21.15.90:0/1764686009 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc00c2c0 con 0x7ff1cc0051c0 2024-03-24T01:26:56.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.399+0000 7ff1e9200640 1 --2- 172.21.15.90:0/1764686009 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1cc004800 0x7ff1cc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:56.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.399+0000 7ff1e9c00640 1 --2- 172.21.15.90:0/1764686009 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1cc003a00 0x7ff1cc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:56.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.399+0000 7ff1e9200640 1 -- 172.21.15.90:0/1764686009 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1cc0051c0 msgr2=0x7ff1cc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:56.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.399+0000 7ff1ea600640 1 --2- 172.21.15.90:0/1764686009 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1cc0051c0 0x7ff1cc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:56.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.399+0000 7ff1e9200640 1 --2- 172.21.15.90:0/1764686009 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1cc0051c0 0x7ff1cc00b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:56.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.399+0000 7ff1e9200640 1 -- 172.21.15.90:0/1764686009 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1cc003a00 msgr2=0x7ff1cc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:56.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.399+0000 7ff1e9200640 1 --2- 172.21.15.90:0/1764686009 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1cc003a00 0x7ff1cc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:56.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.399+0000 7ff1e9200640 1 -- 172.21.15.90:0/1764686009 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1cc003520 con 0x7ff1cc004800 2024-03-24T01:26:56.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.399+0000 7ff1ea600640 1 -- 172.21.15.90:0/1764686009 reap_dead start 2024-03-24T01:26:56.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.400+0000 7ff1e9200640 1 --2- 172.21.15.90:0/1764686009 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1cc004800 0x7ff1cc004c80 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0009ac0 tx=0x7ff1cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31b25ca922f9fa20 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:56.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.401+0000 7ff1e3e00640 1 -- 172.21.15.90:0/1764686009 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d0039070 con 0x7ff1cc004800 2024-03-24T01:26:56.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.401+0000 7ff1e3e00640 1 -- 172.21.15.90:0/1764686009 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1d002c6b0 con 0x7ff1cc004800 2024-03-24T01:26:56.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.401+0000 7ff1e3e00640 1 -- 172.21.15.90:0/1764686009 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d0009690 con 0x7ff1cc004800 2024-03-24T01:26:56.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.401+0000 7ff1ec600640 1 -- 172.21.15.90:0/1764686009 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1cc004800 msgr2=0x7ff1cc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:56.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.401+0000 7ff1ec600640 1 --2- 172.21.15.90:0/1764686009 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1cc004800 0x7ff1cc004c80 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0009ac0 tx=0x7ff1cc005830 comp rx=0 tx=0).stop 2024-03-24T01:26:56.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.401+0000 7ff1ec600640 1 -- 172.21.15.90:0/1764686009 shutdown_connections 2024-03-24T01:26:56.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.401+0000 7ff1ec600640 1 --2- 172.21.15.90:0/1764686009 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1cc004800 0x7ff1cc004c80 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:56.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.401+0000 7ff1ec600640 1 -- 172.21.15.90:0/1764686009 >> 172.21.15.90:0/1764686009 conn(0x7ff1e40b1770 msgr2=0x7ff1e40b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:56.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.401+0000 7ff1ec600640 1 -- 172.21.15.90:0/1764686009 shutdown_connections 2024-03-24T01:26:56.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.401+0000 7ff1ec600640 1 -- 172.21.15.90:0/1764686009 wait complete. 2024-03-24T01:26:56.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.403+0000 7ff1ec600640 1 Processor -- start 2024-03-24T01:26:56.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.403+0000 7ff1ec600640 1 -- start start 2024-03-24T01:26:56.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.404+0000 7ff1ec600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1cc004800 0x7ff1e41ec4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:56.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.404+0000 7ff1ec600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1e41651a0 0x7ff1e4156260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:56.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.404+0000 7ff1ec600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1e41ed210 0x7ff1e40afcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:56.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.404+0000 7ff1ec600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e4162b60 con 0x7ff1e41651a0 2024-03-24T01:26:56.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.404+0000 7ff1ec600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e41629e0 con 0x7ff1e41ed210 2024-03-24T01:26:56.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.404+0000 7ff1ec600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e4162ce0 con 0x7ff1cc004800 2024-03-24T01:26:56.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.404+0000 7ff1e9200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1e41651a0 0x7ff1e4156260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:56.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.404+0000 7ff1e9200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1e41651a0 0x7ff1e4156260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35592/0 (socket says 172.21.15.90:35592) 2024-03-24T01:26:56.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.404+0000 7ff1e9200640 1 -- 172.21.15.90:0/1771755178 learned_addr learned my addr 172.21.15.90:0/1771755178 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:56.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.405+0000 7ff1ea600640 1 --2- 172.21.15.90:0/1771755178 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1e41ed210 0x7ff1e40afcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:56.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.405+0000 7ff1e9c00640 1 --2- 172.21.15.90:0/1771755178 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1cc004800 0x7ff1e41ec4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:56.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.405+0000 7ff1e9200640 1 -- 172.21.15.90:0/1771755178 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1cc004800 msgr2=0x7ff1e41ec4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:56.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.405+0000 7ff1e9200640 1 --2- 172.21.15.90:0/1771755178 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1cc004800 0x7ff1e41ec4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:56.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.405+0000 7ff1e9200640 1 -- 172.21.15.90:0/1771755178 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1e41ed210 msgr2=0x7ff1e40afcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:56.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.405+0000 7ff1e9200640 1 --2- 172.21.15.90:0/1771755178 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1e41ed210 0x7ff1e40afcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:56.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.405+0000 7ff1e9200640 1 -- 172.21.15.90:0/1771755178 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e4165620 con 0x7ff1e41651a0 2024-03-24T01:26:56.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.405+0000 7ff1ea600640 1 --2- 172.21.15.90:0/1771755178 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1e41ed210 0x7ff1e40afcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:56.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.405+0000 7ff1e9200640 1 --2- 172.21.15.90:0/1771755178 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1e41651a0 0x7ff1e4156260 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0000c00 tx=0x7ff1d0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:56.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.406+0000 7ff1e1600640 1 -- 172.21.15.90:0/1771755178 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d0042020 con 0x7ff1e41651a0 2024-03-24T01:26:56.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.406+0000 7ff1e1600640 1 -- 172.21.15.90:0/1771755178 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1d003a040 con 0x7ff1e41651a0 2024-03-24T01:26:56.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.406+0000 7ff1e1600640 1 -- 172.21.15.90:0/1771755178 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d0039070 con 0x7ff1e41651a0 2024-03-24T01:26:56.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.406+0000 7ff1ec600640 1 -- 172.21.15.90:0/1771755178 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1e41ecb60 con 0x7ff1e41651a0 2024-03-24T01:26:56.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.406+0000 7ff1ec600640 1 -- 172.21.15.90:0/1771755178 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1e4158ee0 con 0x7ff1e41651a0 2024-03-24T01:26:56.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.407+0000 7ff1ec600640 1 -- 172.21.15.90:0/1771755178 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1e4068060 con 0x7ff1e41651a0 2024-03-24T01:26:56.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.408+0000 7ff1e1600640 1 -- 172.21.15.90:0/1771755178 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff1d0045020 con 0x7ff1e41651a0 2024-03-24T01:26:56.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.409+0000 7ff1e1600640 1 --2- 172.21.15.90:0/1771755178 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff1c406bbd0 0x7ff1c406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:56.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.409+0000 7ff1e1600640 1 -- 172.21.15.90:0/1771755178 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6216+0+0 (secure 0 0 0) 0x7ff1d00ac050 con 0x7ff1e41651a0 2024-03-24T01:26:56.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.409+0000 7ff1e9c00640 1 --2- 172.21.15.90:0/1771755178 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff1c406bbd0 0x7ff1c406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:56.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.410+0000 7ff1e9c00640 1 --2- 172.21.15.90:0/1771755178 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff1c406bbd0 0x7ff1c406e090 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8001050 tx=0x7ff1d8001b60 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:56.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.419+0000 7ff1e1600640 1 -- 172.21.15.90:0/1771755178 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff1d004c070 con 0x7ff1e41651a0 2024-03-24T01:26:56.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.655+0000 7ff1ec600640 1 -- 172.21.15.90:0/1771755178 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1e40b1370 con 0x7ff1e41651a0 2024-03-24T01:26:56.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.656+0000 7ff1e1600640 1 -- 172.21.15.90:0/1771755178 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4972 (secure 0 0 0) 0x7ff1d007a9d0 con 0x7ff1e41651a0 2024-03-24T01:26:56.656 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:26:56.656 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/296084634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":296084634},{"type":"v1","addr":"172.21.15.120:6835","nonce":296084634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/797786252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":797786252},{"type":"v1","addr":"172.21.15.120:6837","nonce":797786252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5490,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6833/1983177399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1983177399},{"type":"v1","addr":"172.21.15.90:6833","nonce":1983177399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T01:25:59.440431+0000","modified":"2024-03-24T01:26:40.166132+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5787},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5787":{"gid":5787,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.90:6835/11525832","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":11525832},{"type":"v1","addr":"172.21.15.90:6835","nonce":11525832}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5787,"qdb_cluster":[5787]},"id":2}]} 2024-03-24T01:26:56.656 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 25 2024-03-24T01:26:56.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.658+0000 7ff1ec600640 1 -- 172.21.15.90:0/1771755178 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff1c406bbd0 msgr2=0x7ff1c406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:56.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.658+0000 7ff1ec600640 1 --2- 172.21.15.90:0/1771755178 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff1c406bbd0 0x7ff1c406e090 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8001050 tx=0x7ff1d8001b60 comp rx=0 tx=0).stop 2024-03-24T01:26:56.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.658+0000 7ff1ec600640 1 -- 172.21.15.90:0/1771755178 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1e41651a0 msgr2=0x7ff1e4156260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:56.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.658+0000 7ff1ec600640 1 --2- 172.21.15.90:0/1771755178 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1e41651a0 0x7ff1e4156260 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0000c00 tx=0x7ff1d0004730 comp rx=0 tx=0).stop 2024-03-24T01:26:56.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.658+0000 7ff1ec600640 1 -- 172.21.15.90:0/1771755178 shutdown_connections 2024-03-24T01:26:56.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.658+0000 7ff1ec600640 1 --2- 172.21.15.90:0/1771755178 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff1c406bbd0 0x7ff1c406e090 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:56.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.658+0000 7ff1ec600640 1 --2- 172.21.15.90:0/1771755178 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1e41651a0 0x7ff1e4156260 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:56.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.658+0000 7ff1ec600640 1 --2- 172.21.15.90:0/1771755178 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1e41ed210 0x7ff1e40afcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:56.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.658+0000 7ff1ec600640 1 --2- 172.21.15.90:0/1771755178 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1cc004800 0x7ff1e41ec4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:56.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.658+0000 7ff1ec600640 1 -- 172.21.15.90:0/1771755178 >> 172.21.15.90:0/1771755178 conn(0x7ff1e40b1770 msgr2=0x7ff1e4160dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:56.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.658+0000 7ff1ec600640 1 -- 172.21.15.90:0/1771755178 shutdown_connections 2024-03-24T01:26:56.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.658+0000 7ff1ec600640 1 -- 172.21.15.90:0/1771755178 wait complete. 2024-03-24T01:26:56.678 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:26:56.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.891+0000 7f9cedc00640 1 Processor -- start 2024-03-24T01:26:56.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.891+0000 7f9cedc00640 1 -- start start 2024-03-24T01:26:56.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.892+0000 7f9cedc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ce80d10c0 0x7f9ce80ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:56.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.893+0000 7f9cedc00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce8161dd0 con 0x7f9ce80ce960 2024-03-24T01:26:56.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.893+0000 7f9cedc00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce80cf000 con 0x7f9ce815ded0 2024-03-24T01:26:56.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.893+0000 7f9cedc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce80d2410 con 0x7f9ce80d10c0 2024-03-24T01:26:56.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.893+0000 7f9ce7400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f9ce80ce960 0x7f9ce80d28a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53158/0 (socket says 172.21.15.90:53158) 2024-03-24T01:26:56.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.893+0000 7f9ce7400640 1 -- 172.21.15.90:0/1219753277 learned_addr learned my addr 172.21.15.90:0/1219753277 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:26:56.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.893+0000 7f9cde000640 1 --2- 172.21.15.90:0/1219753277 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ce80d10c0 0x7f9ce80ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:56.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.893+0000 7f9ce6000640 1 -- 172.21.15.90:0/1219753277 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3237133068 0 0) 0x7f9ce0001490 con 0x7f9ce80ce960 2024-03-24T01:26:56.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.894+0000 7f9ce6000640 1 -- 172.21.15.90:0/1219753277 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ccc003520 con 0x7f9ce80ce960 2024-03-24T01:26:56.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.894+0000 7f9ce6000640 1 -- 172.21.15.90:0/1219753277 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3848032105 0 0) 0x7f9ccc003520 con 0x7f9ce80ce960 2024-03-24T01:26:56.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.894+0000 7f9ce6000640 1 -- 172.21.15.90:0/1219753277 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ce80d10c0 msgr2=0x7f9ce80ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:56.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.894+0000 7f9ce6000640 1 --2- 172.21.15.90:0/1219753277 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ce80d10c0 0x7f9ce80ce420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:56.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.894+0000 7f9ce6000640 1 -- 172.21.15.90:0/1219753277 >> v1:172.21.15.120:6789/0 conn(0x7f9ce815ded0 legacy=0x7f9ce80d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:56.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.894+0000 7f9ce6000640 1 -- 172.21.15.90:0/1219753277 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce80cf2e0 con 0x7f9ce80ce960 2024-03-24T01:26:56.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.894+0000 7f9ce6000640 1 -- 172.21.15.90:0/1219753277 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f9ce0001bc0 con 0x7f9ce80ce960 2024-03-24T01:26:56.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.894+0000 7f9ce6000640 1 -- 172.21.15.90:0/1219753277 >> v1:172.21.15.90:6789/0 conn(0x7f9ce80ce960 legacy=0x7f9ce80d28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:56.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.894+0000 7f9ce6000640 1 --2- 172.21.15.90:0/1219753277 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ccc0039e0 0x7f9ccc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:56.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.894+0000 7f9ce6000640 1 --2- 172.21.15.90:0/1219753277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ccc004800 0x7f9ccc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:56.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.894+0000 7f9ce6000640 1 --2- 172.21.15.90:0/1219753277 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ce80d10c0 0x7f9ccc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:56.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.894+0000 7f9ce6000640 1 -- 172.21.15.90:0/1219753277 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce80d2410 con 0x7f9ccc0039e0 2024-03-24T01:26:56.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.894+0000 7f9ce6000640 1 -- 172.21.15.90:0/1219753277 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ccc00ba70 con 0x7f9ccc004800 2024-03-24T01:26:56.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.894+0000 7f9ce6000640 1 -- 172.21.15.90:0/1219753277 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ccc00bc70 con 0x7f9ce80d10c0 2024-03-24T01:26:56.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.894+0000 7f9ce6a00640 1 --2- 172.21.15.90:0/1219753277 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ccc0039e0 0x7f9ccc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:56.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.895+0000 7f9ce7400640 1 --2- 172.21.15.90:0/1219753277 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ce80d10c0 0x7f9ccc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:56.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.895+0000 7f9cde000640 1 --2- 172.21.15.90:0/1219753277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ccc004800 0x7f9ccc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:56.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.895+0000 7f9ce6a00640 1 -- 172.21.15.90:0/1219753277 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ce80d10c0 msgr2=0x7f9ccc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:56.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.895+0000 7f9ce6a00640 1 --2- 172.21.15.90:0/1219753277 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ce80d10c0 0x7f9ccc0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:56.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.895+0000 7f9ce6a00640 1 -- 172.21.15.90:0/1219753277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ccc004800 msgr2=0x7f9ccc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:56.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.895+0000 7f9ce6a00640 1 --2- 172.21.15.90:0/1219753277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ccc004800 0x7f9ccc004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:56.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.895+0000 7f9ce6a00640 1 -- 172.21.15.90:0/1219753277 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ccc003520 con 0x7f9ccc0039e0 2024-03-24T01:26:56.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.896+0000 7f9ce6a00640 1 --2- 172.21.15.90:0/1219753277 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ccc0039e0 0x7f9ccc003e00 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8001720 tx=0x7f9ccc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afc2e84e0b39d206 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:56.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.896+0000 7f9ce6000640 1 -- 172.21.15.90:0/1219753277 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd8001370 con 0x7f9ccc0039e0 2024-03-24T01:26:56.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.896+0000 7f9ce6000640 1 -- 172.21.15.90:0/1219753277 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cd80052d0 con 0x7f9ccc0039e0 2024-03-24T01:26:56.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.896+0000 7f9ce6000640 1 -- 172.21.15.90:0/1219753277 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd8003830 con 0x7f9ccc0039e0 2024-03-24T01:26:56.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.897+0000 7f9cedc00640 1 -- 172.21.15.90:0/1219753277 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ccc0039e0 msgr2=0x7f9ccc003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:56.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.897+0000 7f9cedc00640 1 --2- 172.21.15.90:0/1219753277 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ccc0039e0 0x7f9ccc003e00 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8001720 tx=0x7f9ccc005830 comp rx=0 tx=0).stop 2024-03-24T01:26:56.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.897+0000 7f9ce7400640 1 -- 172.21.15.90:0/1219753277 reap_dead start 2024-03-24T01:26:56.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.897+0000 7f9cedc00640 1 -- 172.21.15.90:0/1219753277 shutdown_connections 2024-03-24T01:26:56.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.897+0000 7f9cedc00640 1 -- 172.21.15.90:0/1219753277 >> 172.21.15.90:0/1219753277 conn(0x7f9ce806c9c0 msgr2=0x7f9ce806cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:56.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.897+0000 7f9cedc00640 1 -- 172.21.15.90:0/1219753277 shutdown_connections 2024-03-24T01:26:56.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.897+0000 7f9cedc00640 1 -- 172.21.15.90:0/1219753277 wait complete. 2024-03-24T01:26:56.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.899+0000 7f9cedc00640 1 Processor -- start 2024-03-24T01:26:56.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.899+0000 7f9cedc00640 1 -- start start 2024-03-24T01:26:56.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.900+0000 7f9cedc00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ccc0039e0 0x7f9ce80ac4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:56.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.900+0000 7f9cedc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ce8155a00 0x7f9ce815be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:56.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.900+0000 7f9cedc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ce815c3b0 0x7f9ce80aa710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:56.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.900+0000 7f9cedc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce815e2b0 con 0x7f9ce815c3b0 2024-03-24T01:26:56.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.900+0000 7f9cedc00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce80d20a0 con 0x7f9ccc0039e0 2024-03-24T01:26:56.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.900+0000 7f9cedc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce80b6df0 con 0x7f9ce8155a00 2024-03-24T01:26:56.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.900+0000 7f9ce7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ce815c3b0 0x7f9ce80aa710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:56.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.900+0000 7f9ce7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ce815c3b0 0x7f9ce80aa710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35606/0 (socket says 172.21.15.90:35606) 2024-03-24T01:26:56.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.900+0000 7f9ce7400640 1 -- 172.21.15.90:0/761974374 learned_addr learned my addr 172.21.15.90:0/761974374 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:56.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.900+0000 7f9ce6a00640 1 --2- 172.21.15.90:0/761974374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ccc0039e0 0x7f9ce80ac4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:56.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.900+0000 7f9cde000640 1 --2- 172.21.15.90:0/761974374 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ce8155a00 0x7f9ce815be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:56.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.901+0000 7f9ce7400640 1 -- 172.21.15.90:0/761974374 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ce8155a00 msgr2=0x7f9ce815be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:56.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.901+0000 7f9ce7400640 1 --2- 172.21.15.90:0/761974374 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ce8155a00 0x7f9ce815be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:56.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.901+0000 7f9ce7400640 1 -- 172.21.15.90:0/761974374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ccc0039e0 msgr2=0x7f9ce80ac4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:56.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.901+0000 7f9ce7400640 1 --2- 172.21.15.90:0/761974374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ccc0039e0 0x7f9ce80ac4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:56.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.901+0000 7f9ce7400640 1 -- 172.21.15.90:0/761974374 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce80ab440 con 0x7f9ce815c3b0 2024-03-24T01:26:56.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.901+0000 7f9ce7400640 1 --2- 172.21.15.90:0/761974374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ce815c3b0 0x7f9ce80aa710 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f9ce0001750 tx=0x7f9ce000b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:56.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.901+0000 7f9cdf400640 1 -- 172.21.15.90:0/761974374 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce000bb80 con 0x7f9ce815c3b0 2024-03-24T01:26:56.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.901+0000 7f9cdf400640 1 -- 172.21.15.90:0/761974374 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ce000bd40 con 0x7f9ce815c3b0 2024-03-24T01:26:56.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.901+0000 7f9cedc00640 1 -- 172.21.15.90:0/761974374 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ce80ab720 con 0x7f9ce815c3b0 2024-03-24T01:26:56.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.901+0000 7f9cdf400640 1 -- 172.21.15.90:0/761974374 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce00072b0 con 0x7f9ce815c3b0 2024-03-24T01:26:56.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.902+0000 7f9cedc00640 1 -- 172.21.15.90:0/761974374 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ce80abce0 con 0x7f9ce815c3b0 2024-03-24T01:26:56.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.903+0000 7f9cedc00640 1 -- 172.21.15.90:0/761974374 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ce815c830 con 0x7f9ce815c3b0 2024-03-24T01:26:56.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.903+0000 7f9cdf400640 1 -- 172.21.15.90:0/761974374 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f9ce001b060 con 0x7f9ce815c3b0 2024-03-24T01:26:56.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.904+0000 7f9cdf400640 1 --2- 172.21.15.90:0/761974374 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9cc406bbd0 0x7f9cc406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:56.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.904+0000 7f9cdf400640 1 -- 172.21.15.90:0/761974374 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f9ce008a870 con 0x7f9ce815c3b0 2024-03-24T01:26:56.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.905+0000 7f9ce6a00640 1 --2- 172.21.15.90:0/761974374 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9cc406bbd0 0x7f9cc406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:56.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.906+0000 7f9ce6a00640 1 --2- 172.21.15.90:0/761974374 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9cc406bbd0 0x7f9cc406e090 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8033000 tx=0x7f9cd8009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:56.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:56.909+0000 7f9cdf400640 1 -- 172.21.15.90:0/761974374 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f9ce8161dd0 con 0x7f9ce815c3b0 2024-03-24T01:26:56.998 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:26:57.006 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:26:57.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.109+0000 7f9cedc00640 1 -- 172.21.15.90:0/761974374 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ce80d18d0 con 0x7f9ce815c3b0 2024-03-24T01:26:57.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.111+0000 7f9cdf400640 1 -- 172.21.15.90:0/761974374 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+17096 (secure 0 0 0) 0x7f9ce0059160 con 0x7f9ce815c3b0 2024-03-24T01:26:57.111 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:26:57.112 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":42,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:26:53.308852+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:25:57.398864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-24T01:25:58.451150+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.30","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:6835/2166652035":"2024-03-25T01:26:27.648052+0000","172.21.15.90:6834/2166652035":"2024-03-25T01:26:27.648052+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:26:57.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.113+0000 7f9cedc00640 1 -- 172.21.15.90:0/761974374 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9cc406bbd0 msgr2=0x7f9cc406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:57.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.113+0000 7f9cedc00640 1 --2- 172.21.15.90:0/761974374 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9cc406bbd0 0x7f9cc406e090 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8033000 tx=0x7f9cd8009030 comp rx=0 tx=0).stop 2024-03-24T01:26:57.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.113+0000 7f9cedc00640 1 -- 172.21.15.90:0/761974374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ce815c3b0 msgr2=0x7f9ce80aa710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:57.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.113+0000 7f9cedc00640 1 --2- 172.21.15.90:0/761974374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ce815c3b0 0x7f9ce80aa710 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f9ce0001750 tx=0x7f9ce000b4a0 comp rx=0 tx=0).stop 2024-03-24T01:26:57.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.113+0000 7f9cedc00640 1 -- 172.21.15.90:0/761974374 shutdown_connections 2024-03-24T01:26:57.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.113+0000 7f9cedc00640 1 --2- 172.21.15.90:0/761974374 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f9cc406bbd0 0x7f9cc406e090 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:57.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.113+0000 7f9cedc00640 1 --2- 172.21.15.90:0/761974374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ce815c3b0 0x7f9ce80aa710 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:57.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.113+0000 7f9cedc00640 1 --2- 172.21.15.90:0/761974374 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9ce8155a00 0x7f9ce815be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:57.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.113+0000 7f9cedc00640 1 --2- 172.21.15.90:0/761974374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ccc0039e0 0x7f9ce80ac4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:57.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.113+0000 7f9cedc00640 1 -- 172.21.15.90:0/761974374 >> 172.21.15.90:0/761974374 conn(0x7f9ce806c9c0 msgr2=0x7f9ce815f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:57.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.113+0000 7f9cedc00640 1 -- 172.21.15.90:0/761974374 shutdown_connections 2024-03-24T01:26:57.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.113+0000 7f9cedc00640 1 -- 172.21.15.90:0/761974374 wait complete. 2024-03-24T01:26:57.130 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T01:25:57.398864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T01:25:58.451150+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T01:26:57.131 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:26:57.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.387+0000 7faa1ec00640 1 Processor -- start 2024-03-24T01:26:57.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.387+0000 7faa1ec00640 1 -- start start 2024-03-24T01:26:57.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.389+0000 7faa1ec00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7faa1815ba60 0x7faa1806b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:57.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.389+0000 7faa1ec00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7faa180b2a70 0x7faa1806b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:57.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.389+0000 7faa1ec00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa1809a0a0 0x7faa180ce3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:57.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.389+0000 7faa1ec00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa18160210 con 0x7faa180b2a70 2024-03-24T01:26:57.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.389+0000 7faa1ec00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa1815f600 con 0x7faa1815ba60 2024-03-24T01:26:57.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.389+0000 7faa1ec00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa1815f8e0 con 0x7faa1809a0a0 2024-03-24T01:26:57.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.389+0000 7faa1ce00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7faa180b2a70 0x7faa1806b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:57.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.389+0000 7faa1ce00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7faa180b2a70 0x7faa1806b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35610/0 (socket says 172.21.15.90:35610) 2024-03-24T01:26:57.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.389+0000 7faa1ce00640 1 -- 172.21.15.90:0/3744236457 learned_addr learned my addr 172.21.15.90:0/3744236457 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:57.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.389+0000 7faa1d800640 1 --2- 172.21.15.90:0/3744236457 >> v2:172.21.15.120:3300/0 conn(0x7faa1815ba60 0x7faa1806b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:57.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.389+0000 7faa1e200640 1 --2- 172.21.15.90:0/3744236457 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa1809a0a0 0x7faa180ce3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:57.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.390+0000 7faa1ce00640 1 -- 172.21.15.90:0/3744236457 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa1809a0a0 msgr2=0x7faa180ce3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:57.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.390+0000 7faa1ce00640 1 --2- 172.21.15.90:0/3744236457 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa1809a0a0 0x7faa180ce3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:57.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.390+0000 7faa1ce00640 1 -- 172.21.15.90:0/3744236457 >> v2:172.21.15.120:3300/0 conn(0x7faa1815ba60 msgr2=0x7faa1806b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:57.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.390+0000 7faa1ce00640 1 --2- 172.21.15.90:0/3744236457 >> v2:172.21.15.120:3300/0 conn(0x7faa1815ba60 0x7faa1806b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:57.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.390+0000 7faa1ce00640 1 -- 172.21.15.90:0/3744236457 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa1815fbc0 con 0x7faa180b2a70 2024-03-24T01:26:57.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.390+0000 7faa1d800640 1 --2- 172.21.15.90:0/3744236457 >> v2:172.21.15.120:3300/0 conn(0x7faa1815ba60 0x7faa1806b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:26:57.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.391+0000 7faa1ce00640 1 --2- 172.21.15.90:0/3744236457 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa180b2a70 0x7faa1806b600 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7faa180cbd80 tx=0x7faa0402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fb345eaa69a3b92 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:57.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.391+0000 7faa17e00640 1 -- 172.21.15.90:0/3744236457 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa04035020 con 0x7faa180b2a70 2024-03-24T01:26:57.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.391+0000 7faa17e00640 1 -- 172.21.15.90:0/3744236457 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa04031870 con 0x7faa180b2a70 2024-03-24T01:26:57.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.391+0000 7faa17e00640 1 -- 172.21.15.90:0/3744236457 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa04031c40 con 0x7faa180b2a70 2024-03-24T01:26:57.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.392+0000 7faa1ec00640 1 -- 172.21.15.90:0/3744236457 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa180b2a70 msgr2=0x7faa1806b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:57.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.392+0000 7faa1ec00640 1 --2- 172.21.15.90:0/3744236457 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa180b2a70 0x7faa1806b600 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7faa180cbd80 tx=0x7faa0402b910 comp rx=0 tx=0).stop 2024-03-24T01:26:57.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.392+0000 7faa1ec00640 1 -- 172.21.15.90:0/3744236457 shutdown_connections 2024-03-24T01:26:57.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.392+0000 7faa1ec00640 1 --2- 172.21.15.90:0/3744236457 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa1809a0a0 0x7faa180ce3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:57.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.392+0000 7faa1ec00640 1 --2- 172.21.15.90:0/3744236457 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa180b2a70 0x7faa1806b600 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:57.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.392+0000 7faa1ec00640 1 --2- 172.21.15.90:0/3744236457 >> v2:172.21.15.120:3300/0 conn(0x7faa1815ba60 0x7faa1806b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:57.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.392+0000 7faa1ec00640 1 -- 172.21.15.90:0/3744236457 >> 172.21.15.90:0/3744236457 conn(0x7faa18099a70 msgr2=0x7faa180b2360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:57.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.392+0000 7faa1ec00640 1 -- 172.21.15.90:0/3744236457 shutdown_connections 2024-03-24T01:26:57.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.392+0000 7faa1ec00640 1 -- 172.21.15.90:0/3744236457 wait complete. 2024-03-24T01:26:57.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.394+0000 7faa1ec00640 1 Processor -- start 2024-03-24T01:26:57.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.394+0000 7faa1ec00640 1 -- start start 2024-03-24T01:26:57.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.395+0000 7faa1ec00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa1809a0a0 0x7faa1814af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:57.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.395+0000 7faa1ec00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa180b2a70 0x7faa181f3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:57.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.395+0000 7faa1ec00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa1815ba60 0x7faa180c5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:57.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.395+0000 7faa1ec00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa180d0390 con 0x7faa1815ba60 2024-03-24T01:26:57.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.395+0000 7faa1ec00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1815be40 con 0x7faa1809a0a0 2024-03-24T01:26:57.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.395+0000 7faa1ec00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa180d0510 con 0x7faa180b2a70 2024-03-24T01:26:57.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.395+0000 7faa1e200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa1815ba60 0x7faa180c5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:57.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.395+0000 7faa1e200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa1815ba60 0x7faa180c5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35622/0 (socket says 172.21.15.90:35622) 2024-03-24T01:26:57.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.395+0000 7faa1e200640 1 -- 172.21.15.90:0/4149459699 learned_addr learned my addr 172.21.15.90:0/4149459699 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:57.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.395+0000 7faa1d800640 1 --2- 172.21.15.90:0/4149459699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa1809a0a0 0x7faa1814af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:57.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.395+0000 7faa1ce00640 1 --2- 172.21.15.90:0/4149459699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa180b2a70 0x7faa181f3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:57.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.396+0000 7faa1e200640 1 -- 172.21.15.90:0/4149459699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa180b2a70 msgr2=0x7faa181f3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:57.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.396+0000 7faa1e200640 1 --2- 172.21.15.90:0/4149459699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa180b2a70 0x7faa181f3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:57.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.396+0000 7faa1e200640 1 -- 172.21.15.90:0/4149459699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa1809a0a0 msgr2=0x7faa1814af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:57.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.396+0000 7faa1e200640 1 --2- 172.21.15.90:0/4149459699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa1809a0a0 0x7faa1814af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:57.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.396+0000 7faa1e200640 1 -- 172.21.15.90:0/4149459699 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa180c6180 con 0x7faa1815ba60 2024-03-24T01:26:57.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.396+0000 7faa1e200640 1 --2- 172.21.15.90:0/4149459699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa1815ba60 0x7faa180c5c40 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7faa1000d800 tx=0x7faa1000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:57.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.396+0000 7faa15600640 1 -- 172.21.15.90:0/4149459699 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1000b6c0 con 0x7faa1815ba60 2024-03-24T01:26:57.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.396+0000 7faa15600640 1 -- 172.21.15.90:0/4149459699 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa10019070 con 0x7faa1815ba60 2024-03-24T01:26:57.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.396+0000 7faa15600640 1 -- 172.21.15.90:0/4149459699 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1000bac0 con 0x7faa1815ba60 2024-03-24T01:26:57.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.396+0000 7faa1ec00640 1 -- 172.21.15.90:0/4149459699 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa180c6440 con 0x7faa1815ba60 2024-03-24T01:26:57.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.397+0000 7faa1ec00640 1 -- 172.21.15.90:0/4149459699 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa180c6a00 con 0x7faa1815ba60 2024-03-24T01:26:57.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.398+0000 7faa1ec00640 1 -- 172.21.15.90:0/4149459699 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa181565a0 con 0x7faa1815ba60 2024-03-24T01:26:57.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.399+0000 7faa15600640 1 -- 172.21.15.90:0/4149459699 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7faa1000bc60 con 0x7faa1815ba60 2024-03-24T01:26:57.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.400+0000 7faa15600640 1 --2- 172.21.15.90:0/4149459699 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa9f806bbd0 0x7fa9f806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:57.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.400+0000 7faa15600640 1 -- 172.21.15.90:0/4149459699 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6216+0+0 (secure 0 0 0) 0x7faa1008aff0 con 0x7faa1815ba60 2024-03-24T01:26:57.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.400+0000 7faa1d800640 1 --2- 172.21.15.90:0/4149459699 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa9f806bbd0 0x7fa9f806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:57.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.402+0000 7faa1d800640 1 --2- 172.21.15.90:0/4149459699 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa9f806bbd0 0x7fa9f806e090 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7faa08003aa0 tx=0x7faa08009870 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:57.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.404+0000 7faa15600640 1 -- 172.21.15.90:0/4149459699 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7faa181565a0 con 0x7faa1815ba60 2024-03-24T01:26:57.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.651+0000 7faa1ec00640 1 -- 172.21.15.90:0/4149459699 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faa1804f400 con 0x7faa1815ba60 2024-03-24T01:26:57.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.652+0000 7faa15600640 1 -- 172.21.15.90:0/4149459699 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) v1 ==== 93+0+10 (secure 0 0 0) 0x7faa10059210 con 0x7faa1815ba60 2024-03-24T01:26:57.652 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:26:57.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.655+0000 7faa1ec00640 1 -- 172.21.15.90:0/4149459699 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa9f806bbd0 msgr2=0x7fa9f806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:57.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.655+0000 7faa1ec00640 1 --2- 172.21.15.90:0/4149459699 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa9f806bbd0 0x7fa9f806e090 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7faa08003aa0 tx=0x7faa08009870 comp rx=0 tx=0).stop 2024-03-24T01:26:57.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.655+0000 7faa1ec00640 1 -- 172.21.15.90:0/4149459699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa1815ba60 msgr2=0x7faa180c5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:57.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.655+0000 7faa1ec00640 1 --2- 172.21.15.90:0/4149459699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa1815ba60 0x7faa180c5c40 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7faa1000d800 tx=0x7faa1000dcc0 comp rx=0 tx=0).stop 2024-03-24T01:26:57.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.655+0000 7faa1ec00640 1 -- 172.21.15.90:0/4149459699 shutdown_connections 2024-03-24T01:26:57.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.655+0000 7faa1ec00640 1 --2- 172.21.15.90:0/4149459699 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa9f806bbd0 0x7fa9f806e090 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:57.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.655+0000 7faa1ec00640 1 --2- 172.21.15.90:0/4149459699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa1815ba60 0x7faa180c5c40 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:57.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.655+0000 7faa1ec00640 1 --2- 172.21.15.90:0/4149459699 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7faa180b2a70 0x7faa181f3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:57.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.655+0000 7faa1ec00640 1 --2- 172.21.15.90:0/4149459699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa1809a0a0 0x7faa1814af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:57.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.655+0000 7faa1ec00640 1 -- 172.21.15.90:0/4149459699 >> 172.21.15.90:0/4149459699 conn(0x7faa18099a70 msgr2=0x7faa1806e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:57.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.655+0000 7faa1ec00640 1 -- 172.21.15.90:0/4149459699 shutdown_connections 2024-03-24T01:26:57.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.655+0000 7faa1ec00640 1 -- 172.21.15.90:0/4149459699 wait complete. 2024-03-24T01:26:57.675 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T01:26:57.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.882+0000 7fc0f8800640 1 Processor -- start 2024-03-24T01:26:57.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.882+0000 7fc0f8800640 1 -- start start 2024-03-24T01:26:57.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.884+0000 7fc0f8800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0f00cfcb0 0x7fc0f01666e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:57.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.884+0000 7fc0f8800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc0f00b4580 0x7fc0f00b31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:57.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.884+0000 7fc0f8800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0f00b4d30 con 0x7fc0f00cf8d0 2024-03-24T01:26:57.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.884+0000 7fc0f8800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0f00b4fd0 con 0x7fc0f00b4580 2024-03-24T01:26:57.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.884+0000 7fc0f8800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f00b52b0 con 0x7fc0f00cfcb0 2024-03-24T01:26:57.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.884+0000 7fc0f6800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc0f00b4580 0x7fc0f00b31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:57.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.885+0000 7fc0f5e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc0f00cf8d0 0x7fc0f0162e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53160/0 (socket says 172.21.15.90:53160) 2024-03-24T01:26:57.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.885+0000 7fc0f5e00640 1 -- 172.21.15.90:0/2507686206 learned_addr learned my addr 172.21.15.90:0/2507686206 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:26:57.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.885+0000 7fc0f6800640 1 -- 172.21.15.90:0/2507686206 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0f00cfcb0 msgr2=0x7fc0f01666e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:57.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.885+0000 7fc0f6800640 1 --2- 172.21.15.90:0/2507686206 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0f00cfcb0 0x7fc0f01666e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:57.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.885+0000 7fc0f6800640 1 -- 172.21.15.90:0/2507686206 >> v1:172.21.15.90:6789/0 conn(0x7fc0f00cf8d0 legacy=0x7fc0f0162e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:57.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.885+0000 7fc0f6800640 1 -- 172.21.15.90:0/2507686206 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f01628f0 con 0x7fc0f00b4580 2024-03-24T01:26:57.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.885+0000 7fc0f4a00640 1 -- 172.21.15.90:0/2507686206 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 795034615 0 0) 0x7fc0f00b4d30 con 0x7fc0f00cf8d0 2024-03-24T01:26:57.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.886+0000 7fc0f6800640 1 --2- 172.21.15.90:0/2507686206 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0f00b4580 0x7fc0f00b31f0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec009d90 tx=0x7fc0ec02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7ab8b4d1fe218f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:57.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.887+0000 7fc0f4a00640 1 -- 172.21.15.90:0/2507686206 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec039070 con 0x7fc0f00b4580 2024-03-24T01:26:57.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.887+0000 7fc0f4a00640 1 -- 172.21.15.90:0/2507686206 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0ec02cc50 con 0x7fc0f00b4580 2024-03-24T01:26:57.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.887+0000 7fc0f4a00640 1 -- 172.21.15.90:0/2507686206 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec034440 con 0x7fc0f00b4580 2024-03-24T01:26:57.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.887+0000 7fc0f8800640 1 -- 172.21.15.90:0/2507686206 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0f00b4580 msgr2=0x7fc0f00b31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:57.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.887+0000 7fc0f8800640 1 --2- 172.21.15.90:0/2507686206 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0f00b4580 0x7fc0f00b31f0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec009d90 tx=0x7fc0ec02bea0 comp rx=0 tx=0).stop 2024-03-24T01:26:57.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.887+0000 7fc0f8800640 1 -- 172.21.15.90:0/2507686206 shutdown_connections 2024-03-24T01:26:57.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.887+0000 7fc0f8800640 1 --2- 172.21.15.90:0/2507686206 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0f00b4580 0x7fc0f00b31f0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:57.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.887+0000 7fc0f8800640 1 --2- 172.21.15.90:0/2507686206 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0f00cfcb0 0x7fc0f01666e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:57.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.887+0000 7fc0f8800640 1 -- 172.21.15.90:0/2507686206 >> 172.21.15.90:0/2507686206 conn(0x7fc0f006f150 msgr2=0x7fc0f00721e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:57.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.888+0000 7fc0f8800640 1 -- 172.21.15.90:0/2507686206 shutdown_connections 2024-03-24T01:26:57.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.888+0000 7fc0f8800640 1 -- 172.21.15.90:0/2507686206 wait complete. 2024-03-24T01:26:57.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.890+0000 7fc0f8800640 1 Processor -- start 2024-03-24T01:26:57.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.890+0000 7fc0f8800640 1 -- start start 2024-03-24T01:26:57.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.891+0000 7fc0f8800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0f00b4580 0x7fc0f01f7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:57.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.891+0000 7fc0f8800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0f00cf8d0 0x7fc0f0157e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:57.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.891+0000 7fc0f8800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0f00cfcb0 0x7fc0f0158350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:57.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.891+0000 7fc0f8800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f01636a0 con 0x7fc0f00b4580 2024-03-24T01:26:57.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.891+0000 7fc0f8800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f0163520 con 0x7fc0f00cfcb0 2024-03-24T01:26:57.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.891+0000 7fc0f8800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f00b3730 con 0x7fc0f00cf8d0 2024-03-24T01:26:57.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.891+0000 7fc0f5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0f00b4580 0x7fc0f01f7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:57.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.891+0000 7fc0f5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0f00b4580 0x7fc0f01f7ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35640/0 (socket says 172.21.15.90:35640) 2024-03-24T01:26:57.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.891+0000 7fc0f5e00640 1 -- 172.21.15.90:0/675578245 learned_addr learned my addr 172.21.15.90:0/675578245 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:57.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.891+0000 7fc0f5400640 1 --2- 172.21.15.90:0/675578245 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0f00cf8d0 0x7fc0f0157e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:57.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.891+0000 7fc0f6800640 1 --2- 172.21.15.90:0/675578245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0f00cfcb0 0x7fc0f0158350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:57.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.892+0000 7fc0f5e00640 1 -- 172.21.15.90:0/675578245 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0f00cf8d0 msgr2=0x7fc0f0157e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:57.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.892+0000 7fc0f5e00640 1 --2- 172.21.15.90:0/675578245 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0f00cf8d0 0x7fc0f0157e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:57.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.892+0000 7fc0f5e00640 1 -- 172.21.15.90:0/675578245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0f00cfcb0 msgr2=0x7fc0f0158350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:57.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.892+0000 7fc0f5e00640 1 --2- 172.21.15.90:0/675578245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0f00cfcb0 0x7fc0f0158350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:57.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.892+0000 7fc0f5e00640 1 -- 172.21.15.90:0/675578245 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f00b1ae0 con 0x7fc0f00b4580 2024-03-24T01:26:57.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.892+0000 7fc0f6800640 1 --2- 172.21.15.90:0/675578245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0f00cfcb0 0x7fc0f0158350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:26:57.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.892+0000 7fc0f5e00640 1 --2- 172.21.15.90:0/675578245 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0f00b4580 0x7fc0f01f7ac0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fc0e400c500 tx=0x7fc0e400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:57.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.892+0000 7fc0ea000640 1 -- 172.21.15.90:0/675578245 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e4018070 con 0x7fc0f00b4580 2024-03-24T01:26:57.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.892+0000 7fc0ea000640 1 -- 172.21.15.90:0/675578245 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0e4007580 con 0x7fc0f00b4580 2024-03-24T01:26:57.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.892+0000 7fc0f8800640 1 -- 172.21.15.90:0/675578245 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0f0156880 con 0x7fc0f00b4580 2024-03-24T01:26:57.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.893+0000 7fc0ea000640 1 -- 172.21.15.90:0/675578245 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e4004a20 con 0x7fc0f00b4580 2024-03-24T01:26:57.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.893+0000 7fc0f8800640 1 -- 172.21.15.90:0/675578245 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0f0156e00 con 0x7fc0f00b4580 2024-03-24T01:26:57.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.894+0000 7fc0f8800640 1 -- 172.21.15.90:0/675578245 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0f004f400 con 0x7fc0f00b4580 2024-03-24T01:26:57.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.896+0000 7fc0ea000640 1 -- 172.21.15.90:0/675578245 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc0e401b070 con 0x7fc0f00b4580 2024-03-24T01:26:57.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.897+0000 7fc0ea000640 1 --2- 172.21.15.90:0/675578245 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc0d406bbd0 0x7fc0d406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:57.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.897+0000 7fc0ea000640 1 -- 172.21.15.90:0/675578245 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6216+0+0 (secure 0 0 0) 0x7fc0e4089ec0 con 0x7fc0f00b4580 2024-03-24T01:26:57.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.897+0000 7fc0f5400640 1 --2- 172.21.15.90:0/675578245 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc0d406bbd0 0x7fc0d406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:57.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.898+0000 7fc0f5400640 1 --2- 172.21.15.90:0/675578245 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc0d406bbd0 0x7fc0d406e090 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0066ef0 tx=0x7fc0e0009060 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:57.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:57.902+0000 7fc0ea000640 1 -- 172.21.15.90:0/675578245 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc0e40581f0 con 0x7fc0f00b4580 2024-03-24T01:26:58.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.100+0000 7fc0f8800640 1 -- 172.21.15.90:0/675578245 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc0f00b1cb0 con 0x7fc0f00b4580 2024-03-24T01:26:58.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.101+0000 7fc0ea000640 1 -- 172.21.15.90:0/675578245 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc0e405e030 con 0x7fc0f00b4580 2024-03-24T01:26:58.101 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:26:58.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.103+0000 7fc0f8800640 1 -- 172.21.15.90:0/675578245 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc0d406bbd0 msgr2=0x7fc0d406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:58.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.103+0000 7fc0f8800640 1 --2- 172.21.15.90:0/675578245 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc0d406bbd0 0x7fc0d406e090 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0066ef0 tx=0x7fc0e0009060 comp rx=0 tx=0).stop 2024-03-24T01:26:58.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.103+0000 7fc0f8800640 1 -- 172.21.15.90:0/675578245 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0f00b4580 msgr2=0x7fc0f01f7ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:58.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.103+0000 7fc0f8800640 1 --2- 172.21.15.90:0/675578245 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0f00b4580 0x7fc0f01f7ac0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fc0e400c500 tx=0x7fc0e400c9c0 comp rx=0 tx=0).stop 2024-03-24T01:26:58.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.103+0000 7fc0f8800640 1 -- 172.21.15.90:0/675578245 shutdown_connections 2024-03-24T01:26:58.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.103+0000 7fc0f8800640 1 --2- 172.21.15.90:0/675578245 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc0d406bbd0 0x7fc0d406e090 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:58.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.103+0000 7fc0f8800640 1 --2- 172.21.15.90:0/675578245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0f00cfcb0 0x7fc0f0158350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:58.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.103+0000 7fc0f8800640 1 --2- 172.21.15.90:0/675578245 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc0f00cf8d0 0x7fc0f0157e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:58.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.103+0000 7fc0f8800640 1 --2- 172.21.15.90:0/675578245 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc0f00b4580 0x7fc0f01f7ac0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:58.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.103+0000 7fc0f8800640 1 -- 172.21.15.90:0/675578245 >> 172.21.15.90:0/675578245 conn(0x7fc0f006f150 msgr2=0x7fc0f0160930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:58.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.103+0000 7fc0f8800640 1 -- 172.21.15.90:0/675578245 shutdown_connections 2024-03-24T01:26:58.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.103+0000 7fc0f8800640 1 -- 172.21.15.90:0/675578245 wait complete. 2024-03-24T01:26:58.130 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T01:26:58.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.355+0000 7f413de00640 1 Processor -- start 2024-03-24T01:26:58.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.356+0000 7f413de00640 1 -- start start 2024-03-24T01:26:58.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.358+0000 7f413de00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f413806b0c0 0x7f41380b1a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:58.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.358+0000 7f413de00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413815d500 0x7f413815d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:58.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.358+0000 7f413de00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41380d0610 con 0x7f41380ce450 2024-03-24T01:26:58.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.358+0000 7f413de00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41380b4260 con 0x7f413806b0c0 2024-03-24T01:26:58.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.358+0000 7f413de00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41380b1f90 con 0x7f413815d500 2024-03-24T01:26:58.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.358+0000 7f4136000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f41380ce450 0x7f413806b4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53168/0 (socket says 172.21.15.90:53168) 2024-03-24T01:26:58.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.358+0000 7f4136000640 1 -- 172.21.15.90:0/3889738658 learned_addr learned my addr 172.21.15.90:0/3889738658 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:26:58.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.358+0000 7f4136a00640 1 --2- 172.21.15.90:0/3889738658 >> v2:172.21.15.120:3300/0 conn(0x7f413806b0c0 0x7f41380b1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:58.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.359+0000 7f4137400640 1 --2- 172.21.15.90:0/3889738658 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413815d500 0x7f413815d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:58.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.359+0000 7f4135600640 1 -- 172.21.15.90:0/3889738658 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 822824697 0 0) 0x7f41380d0610 con 0x7f41380ce450 2024-03-24T01:26:58.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.359+0000 7f4135600640 1 -- 172.21.15.90:0/3889738658 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f411c003520 con 0x7f41380ce450 2024-03-24T01:26:58.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.359+0000 7f4136a00640 1 -- 172.21.15.90:0/3889738658 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413815d500 msgr2=0x7f413815d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:58.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.359+0000 7f4136a00640 1 --2- 172.21.15.90:0/3889738658 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413815d500 0x7f413815d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:58.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.359+0000 7f4136a00640 1 -- 172.21.15.90:0/3889738658 >> v1:172.21.15.90:6789/0 conn(0x7f41380ce450 legacy=0x7f413806b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:58.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.359+0000 7f4136a00640 1 -- 172.21.15.90:0/3889738658 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41380b10c0 con 0x7f413806b0c0 2024-03-24T01:26:58.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.360+0000 7f4136a00640 1 --2- 172.21.15.90:0/3889738658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f413806b0c0 0x7f41380b1a50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4128009d90 tx=0x7f412802bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8dba2fd8be4ffb00 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:58.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.360+0000 7f4135600640 1 -- 172.21.15.90:0/3889738658 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4128039070 con 0x7f413806b0c0 2024-03-24T01:26:58.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.360+0000 7f4135600640 1 -- 172.21.15.90:0/3889738658 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f412802cce0 con 0x7f413806b0c0 2024-03-24T01:26:58.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.361+0000 7f4135600640 1 -- 172.21.15.90:0/3889738658 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4128034640 con 0x7f413806b0c0 2024-03-24T01:26:58.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.361+0000 7f413de00640 1 -- 172.21.15.90:0/3889738658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f413806b0c0 msgr2=0x7f41380b1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:58.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.361+0000 7f413de00640 1 --2- 172.21.15.90:0/3889738658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f413806b0c0 0x7f41380b1a50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4128009d90 tx=0x7f412802bf30 comp rx=0 tx=0).stop 2024-03-24T01:26:58.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.361+0000 7f413de00640 1 -- 172.21.15.90:0/3889738658 shutdown_connections 2024-03-24T01:26:58.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.361+0000 7f413de00640 1 --2- 172.21.15.90:0/3889738658 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413815d500 0x7f413815d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:58.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.361+0000 7f413de00640 1 --2- 172.21.15.90:0/3889738658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f413806b0c0 0x7f41380b1a50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:58.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.361+0000 7f413de00640 1 -- 172.21.15.90:0/3889738658 >> 172.21.15.90:0/3889738658 conn(0x7f41380d0880 msgr2=0x7f41380d0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:58.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.361+0000 7f413de00640 1 -- 172.21.15.90:0/3889738658 shutdown_connections 2024-03-24T01:26:58.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.361+0000 7f413de00640 1 -- 172.21.15.90:0/3889738658 wait complete. 2024-03-24T01:26:58.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.363+0000 7f413de00640 1 Processor -- start 2024-03-24T01:26:58.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.364+0000 7f413de00640 1 -- start start 2024-03-24T01:26:58.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.364+0000 7f413de00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413806b0c0 0x7f413814fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:58.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.365+0000 7f413de00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41380ce450 0x7f41380ad610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:58.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.365+0000 7f413de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f413815d500 0x7f41380adb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:58.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.365+0000 7f413de00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413815ded0 con 0x7f413815d500 2024-03-24T01:26:58.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.365+0000 7f413de00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41380d0330 con 0x7f41380ce450 2024-03-24T01:26:58.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.365+0000 7f413de00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f413815e050 con 0x7f413806b0c0 2024-03-24T01:26:58.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.365+0000 7f4137400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f413815d500 0x7f41380adb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:58.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.365+0000 7f4137400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f413815d500 0x7f41380adb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35656/0 (socket says 172.21.15.90:35656) 2024-03-24T01:26:58.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.365+0000 7f4137400640 1 -- 172.21.15.90:0/3432723250 learned_addr learned my addr 172.21.15.90:0/3432723250 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:58.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.365+0000 7f4136000640 1 --2- 172.21.15.90:0/3432723250 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41380ce450 0x7f41380ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:58.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.365+0000 7f4136a00640 1 --2- 172.21.15.90:0/3432723250 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413806b0c0 0x7f413814fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:58.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.365+0000 7f4137400640 1 -- 172.21.15.90:0/3432723250 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413806b0c0 msgr2=0x7f413814fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:58.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.365+0000 7f4137400640 1 --2- 172.21.15.90:0/3432723250 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413806b0c0 0x7f413814fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:58.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.365+0000 7f4137400640 1 -- 172.21.15.90:0/3432723250 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41380ce450 msgr2=0x7f41380ad610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:58.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.365+0000 7f4137400640 1 --2- 172.21.15.90:0/3432723250 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41380ce450 0x7f41380ad610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:58.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.365+0000 7f4137400640 1 -- 172.21.15.90:0/3432723250 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41381e9010 con 0x7f413815d500 2024-03-24T01:26:58.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.366+0000 7f4137400640 1 --2- 172.21.15.90:0/3432723250 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f413815d500 0x7f41380adb50 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f413000d800 tx=0x7f413000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:58.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.366+0000 7f412f400640 1 -- 172.21.15.90:0/3432723250 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f413000b6c0 con 0x7f413815d500 2024-03-24T01:26:58.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.366+0000 7f412f400640 1 -- 172.21.15.90:0/3432723250 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4130019070 con 0x7f413815d500 2024-03-24T01:26:58.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.366+0000 7f412f400640 1 -- 172.21.15.90:0/3432723250 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f413000bac0 con 0x7f413815d500 2024-03-24T01:26:58.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.366+0000 7f413de00640 1 -- 172.21.15.90:0/3432723250 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41381e92f0 con 0x7f413815d500 2024-03-24T01:26:58.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.366+0000 7f413de00640 1 -- 172.21.15.90:0/3432723250 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4138157540 con 0x7f413815d500 2024-03-24T01:26:58.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.367+0000 7f413de00640 1 -- 172.21.15.90:0/3432723250 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f413804f400 con 0x7f413815d500 2024-03-24T01:26:58.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.368+0000 7f412f400640 1 -- 172.21.15.90:0/3432723250 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f413000bc60 con 0x7f413815d500 2024-03-24T01:26:58.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.370+0000 7f412f400640 1 --2- 172.21.15.90:0/3432723250 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f411406bc00 0x7f411406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:58.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.370+0000 7f412f400640 1 -- 172.21.15.90:0/3432723250 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f413008b140 con 0x7f413815d500 2024-03-24T01:26:58.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.370+0000 7f4136a00640 1 --2- 172.21.15.90:0/3432723250 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f411406bc00 0x7f411406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:58.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.371+0000 7f4136a00640 1 --2- 172.21.15.90:0/3432723250 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f411406bc00 0x7f411406e0c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f4128036000 tx=0x7f4128030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:58.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.374+0000 7f412f400640 1 -- 172.21.15.90:0/3432723250 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4130059900 con 0x7f413815d500 2024-03-24T01:26:58.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.569+0000 7f413de00640 1 -- 172.21.15.90:0/3432723250 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f41380b1620 con 0x7f413815d500 2024-03-24T01:26:58.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.570+0000 7f412f400640 1 -- 172.21.15.90:0/3432723250 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4130059200 con 0x7f413815d500 2024-03-24T01:26:58.571 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-24T01:26:58.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.573+0000 7f413de00640 1 -- 172.21.15.90:0/3432723250 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f411406bc00 msgr2=0x7f411406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:58.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.573+0000 7f413de00640 1 --2- 172.21.15.90:0/3432723250 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f411406bc00 0x7f411406e0c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f4128036000 tx=0x7f4128030000 comp rx=0 tx=0).stop 2024-03-24T01:26:58.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.573+0000 7f413de00640 1 -- 172.21.15.90:0/3432723250 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f413815d500 msgr2=0x7f41380adb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:58.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.573+0000 7f413de00640 1 --2- 172.21.15.90:0/3432723250 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f413815d500 0x7f41380adb50 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f413000d800 tx=0x7f413000dcc0 comp rx=0 tx=0).stop 2024-03-24T01:26:58.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.573+0000 7f413de00640 1 -- 172.21.15.90:0/3432723250 shutdown_connections 2024-03-24T01:26:58.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.573+0000 7f413de00640 1 --2- 172.21.15.90:0/3432723250 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f411406bc00 0x7f411406e0c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:58.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.573+0000 7f413de00640 1 --2- 172.21.15.90:0/3432723250 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f413815d500 0x7f41380adb50 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:58.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.573+0000 7f413de00640 1 --2- 172.21.15.90:0/3432723250 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41380ce450 0x7f41380ad610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:58.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.573+0000 7f413de00640 1 --2- 172.21.15.90:0/3432723250 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f413806b0c0 0x7f413814fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:58.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.573+0000 7f413de00640 1 -- 172.21.15.90:0/3432723250 >> 172.21.15.90:0/3432723250 conn(0x7f41380d0880 msgr2=0x7f41380b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:58.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.574+0000 7f413de00640 1 -- 172.21.15.90:0/3432723250 shutdown_connections 2024-03-24T01:26:58.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.574+0000 7f413de00640 1 -- 172.21.15.90:0/3432723250 wait complete. 2024-03-24T01:26:58.591 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:26:58.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.773+0000 7f5b3b400640 1 Processor -- start 2024-03-24T01:26:58.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.773+0000 7f5b3b400640 1 -- start start 2024-03-24T01:26:58.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.776+0000 7f5b3b400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5b3406c690 0x7f5b3406caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:58.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.776+0000 7f5b3b400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5b340d2970 0x7f5b340d2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:58.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.776+0000 7f5b3b400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b3415ee70 con 0x7f5b3415e080 2024-03-24T01:26:58.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.776+0000 7f5b3b400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b340b6c10 con 0x7f5b340d2970 2024-03-24T01:26:58.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.776+0000 7f5b3b400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b3406d030 con 0x7f5b3406c690 2024-03-24T01:26:58.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.776+0000 7f5b38a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f5b3415e080 0x7f5b3415e460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53182/0 (socket says 172.21.15.90:53182) 2024-03-24T01:26:58.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.776+0000 7f5b38a00640 1 -- 172.21.15.90:0/2473084570 learned_addr learned my addr 172.21.15.90:0/2473084570 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:26:58.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.776+0000 7f5b39400640 1 --2- 172.21.15.90:0/2473084570 >> v2:172.21.15.90:3300/0 conn(0x7f5b340d2970 0x7f5b340d2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:58.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.777+0000 7f5b33e00640 1 --2- 172.21.15.90:0/2473084570 >> v2:172.21.15.120:3300/0 conn(0x7f5b3406c690 0x7f5b3406caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:58.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.777+0000 7f5b33400640 1 -- 172.21.15.90:0/2473084570 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2276167030 0 0) 0x7f5b3415ee70 con 0x7f5b3415e080 2024-03-24T01:26:58.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.777+0000 7f5b33400640 1 -- 172.21.15.90:0/2473084570 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b1c003520 con 0x7f5b3415e080 2024-03-24T01:26:58.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.777+0000 7f5b39400640 1 -- 172.21.15.90:0/2473084570 >> v2:172.21.15.120:3300/0 conn(0x7f5b3406c690 msgr2=0x7f5b3406caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:58.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.777+0000 7f5b39400640 1 --2- 172.21.15.90:0/2473084570 >> v2:172.21.15.120:3300/0 conn(0x7f5b3406c690 0x7f5b3406caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:58.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.777+0000 7f5b39400640 1 -- 172.21.15.90:0/2473084570 >> v1:172.21.15.90:6789/0 conn(0x7f5b3415e080 legacy=0x7f5b3415e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:58.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.777+0000 7f5b39400640 1 -- 172.21.15.90:0/2473084570 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b3415f150 con 0x7f5b340d2970 2024-03-24T01:26:58.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.778+0000 7f5b39400640 1 --2- 172.21.15.90:0/2473084570 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b340d2970 0x7f5b340d2e00 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f5b340b9250 tx=0x7f5b2c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d08c9a523bbc2d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:58.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.778+0000 7f5b33400640 1 -- 172.21.15.90:0/2473084570 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b2c039070 con 0x7f5b340d2970 2024-03-24T01:26:58.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.778+0000 7f5b33400640 1 -- 172.21.15.90:0/2473084570 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b2c02cd00 con 0x7f5b340d2970 2024-03-24T01:26:58.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.779+0000 7f5b33400640 1 -- 172.21.15.90:0/2473084570 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b2c034640 con 0x7f5b340d2970 2024-03-24T01:26:58.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.779+0000 7f5b3b400640 1 -- 172.21.15.90:0/2473084570 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b340d2970 msgr2=0x7f5b340d2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:58.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.779+0000 7f5b3b400640 1 --2- 172.21.15.90:0/2473084570 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b340d2970 0x7f5b340d2e00 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f5b340b9250 tx=0x7f5b2c02bf50 comp rx=0 tx=0).stop 2024-03-24T01:26:58.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.779+0000 7f5b3b400640 1 -- 172.21.15.90:0/2473084570 shutdown_connections 2024-03-24T01:26:58.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.779+0000 7f5b3b400640 1 --2- 172.21.15.90:0/2473084570 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b340d2970 0x7f5b340d2e00 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:58.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.779+0000 7f5b3b400640 1 --2- 172.21.15.90:0/2473084570 >> v2:172.21.15.120:3300/0 conn(0x7f5b3406c690 0x7f5b3406caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:58.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.779+0000 7f5b3b400640 1 -- 172.21.15.90:0/2473084570 >> 172.21.15.90:0/2473084570 conn(0x7f5b34072c40 msgr2=0x7f5b34073050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:58.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.779+0000 7f5b3b400640 1 -- 172.21.15.90:0/2473084570 shutdown_connections 2024-03-24T01:26:58.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.780+0000 7f5b3b400640 1 -- 172.21.15.90:0/2473084570 wait complete. 2024-03-24T01:26:58.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.782+0000 7f5b3b400640 1 Processor -- start 2024-03-24T01:26:58.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.782+0000 7f5b3b400640 1 -- start start 2024-03-24T01:26:58.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.784+0000 7f5b3b400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b3406c690 0x7f5b340b2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:58.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.784+0000 7f5b3b400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5b340d2970 0x7f5b3414d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:58.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.784+0000 7f5b3b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b3415e080 0x7f5b3414dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:58.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.784+0000 7f5b3b400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b3415eb70 con 0x7f5b3415e080 2024-03-24T01:26:58.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.784+0000 7f5b3b400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b340d2450 con 0x7f5b3406c690 2024-03-24T01:26:58.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.784+0000 7f5b3b400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b3415f5c0 con 0x7f5b340d2970 2024-03-24T01:26:58.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.784+0000 7f5b39400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b3415e080 0x7f5b3414dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:58.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.784+0000 7f5b39400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b3415e080 0x7f5b3414dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35672/0 (socket says 172.21.15.90:35672) 2024-03-24T01:26:58.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.784+0000 7f5b39400640 1 -- 172.21.15.90:0/3810413349 learned_addr learned my addr 172.21.15.90:0/3810413349 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:58.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.785+0000 7f5b38a00640 1 --2- 172.21.15.90:0/3810413349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b3406c690 0x7f5b340b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:58.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.785+0000 7f5b33e00640 1 --2- 172.21.15.90:0/3810413349 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5b340d2970 0x7f5b3414d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:58.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.785+0000 7f5b39400640 1 -- 172.21.15.90:0/3810413349 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5b340d2970 msgr2=0x7f5b3414d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:58.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.785+0000 7f5b39400640 1 --2- 172.21.15.90:0/3810413349 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5b340d2970 0x7f5b3414d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:58.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.785+0000 7f5b39400640 1 -- 172.21.15.90:0/3810413349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b3406c690 msgr2=0x7f5b340b2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:58.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.785+0000 7f5b39400640 1 --2- 172.21.15.90:0/3810413349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b3406c690 0x7f5b340b2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:58.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.785+0000 7f5b39400640 1 -- 172.21.15.90:0/3810413349 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b3414e4f0 con 0x7f5b3415e080 2024-03-24T01:26:58.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.785+0000 7f5b33e00640 1 --2- 172.21.15.90:0/3810413349 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5b340d2970 0x7f5b3414d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:58.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.785+0000 7f5b39400640 1 --2- 172.21.15.90:0/3810413349 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b3415e080 0x7f5b3414dc80 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c02c560 tx=0x7f5b2c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:58.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.785+0000 7f5b30c00640 1 -- 172.21.15.90:0/3810413349 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b2c039050 con 0x7f5b3415e080 2024-03-24T01:26:58.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.785+0000 7f5b30c00640 1 -- 172.21.15.90:0/3810413349 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b2c045070 con 0x7f5b3415e080 2024-03-24T01:26:58.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.785+0000 7f5b3b400640 1 -- 172.21.15.90:0/3810413349 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b3414e7d0 con 0x7f5b3415e080 2024-03-24T01:26:58.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.785+0000 7f5b30c00640 1 -- 172.21.15.90:0/3810413349 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b2c009450 con 0x7f5b3415e080 2024-03-24T01:26:58.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.786+0000 7f5b3b400640 1 -- 172.21.15.90:0/3810413349 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b3414f070 con 0x7f5b3415e080 2024-03-24T01:26:58.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.787+0000 7f5b3b400640 1 -- 172.21.15.90:0/3810413349 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b3414f270 con 0x7f5b3415e080 2024-03-24T01:26:58.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.788+0000 7f5b30c00640 1 -- 172.21.15.90:0/3810413349 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5b2c03e830 con 0x7f5b3415e080 2024-03-24T01:26:58.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.788+0000 7f5b30c00640 1 --2- 172.21.15.90:0/3810413349 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5b1406bbd0 0x7f5b1406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:58.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.789+0000 7f5b30c00640 1 -- 172.21.15.90:0/3810413349 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f5b2c0adae0 con 0x7f5b3415e080 2024-03-24T01:26:58.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.789+0000 7f5b38a00640 1 --2- 172.21.15.90:0/3810413349 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5b1406bbd0 0x7f5b1406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:58.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.790+0000 7f5b38a00640 1 --2- 172.21.15.90:0/3810413349 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5b1406bbd0 0x7f5b1406e090 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5b28000f60 tx=0x7f5b28009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:58.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:58.793+0000 7f5b30c00640 1 -- 172.21.15.90:0/3810413349 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5b3414f270 con 0x7f5b3415e080 2024-03-24T01:26:59.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.030+0000 7f5b3b400640 1 -- 172.21.15.90:0/3810413349 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b34068060 con 0x7f5b3415e080 2024-03-24T01:26:59.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.031+0000 7f5b30c00640 1 -- 172.21.15.90:0/3810413349 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4972 (secure 0 0 0) 0x7f5b2c032e20 con 0x7f5b3415e080 2024-03-24T01:26:59.031 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:26:59.032 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/296084634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":296084634},{"type":"v1","addr":"172.21.15.120:6835","nonce":296084634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/797786252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":797786252},{"type":"v1","addr":"172.21.15.120:6837","nonce":797786252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5490,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6833/1983177399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1983177399},{"type":"v1","addr":"172.21.15.90:6833","nonce":1983177399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T01:25:59.440431+0000","modified":"2024-03-24T01:26:40.166132+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5787},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5787":{"gid":5787,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.90:6835/11525832","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":11525832},{"type":"v1","addr":"172.21.15.90:6835","nonce":11525832}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5787,"qdb_cluster":[5787]},"id":2}]} 2024-03-24T01:26:59.032 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 25 2024-03-24T01:26:59.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.034+0000 7f5b3b400640 1 -- 172.21.15.90:0/3810413349 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5b1406bbd0 msgr2=0x7f5b1406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:59.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.034+0000 7f5b3b400640 1 --2- 172.21.15.90:0/3810413349 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5b1406bbd0 0x7f5b1406e090 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5b28000f60 tx=0x7f5b28009040 comp rx=0 tx=0).stop 2024-03-24T01:26:59.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.034+0000 7f5b3b400640 1 -- 172.21.15.90:0/3810413349 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b3415e080 msgr2=0x7f5b3414dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:59.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.034+0000 7f5b3b400640 1 --2- 172.21.15.90:0/3810413349 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b3415e080 0x7f5b3414dc80 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c02c560 tx=0x7f5b2c004a40 comp rx=0 tx=0).stop 2024-03-24T01:26:59.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.034+0000 7f5b3b400640 1 -- 172.21.15.90:0/3810413349 shutdown_connections 2024-03-24T01:26:59.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.034+0000 7f5b3b400640 1 --2- 172.21.15.90:0/3810413349 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5b1406bbd0 0x7f5b1406e090 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:59.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.034+0000 7f5b3b400640 1 --2- 172.21.15.90:0/3810413349 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b3415e080 0x7f5b3414dc80 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:59.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.034+0000 7f5b3b400640 1 --2- 172.21.15.90:0/3810413349 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5b340d2970 0x7f5b3414d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:59.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.034+0000 7f5b3b400640 1 --2- 172.21.15.90:0/3810413349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b3406c690 0x7f5b340b2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:59.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.034+0000 7f5b3b400640 1 -- 172.21.15.90:0/3810413349 >> 172.21.15.90:0/3810413349 conn(0x7f5b34072c40 msgr2=0x7f5b340b8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:59.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.035+0000 7f5b3b400640 1 -- 172.21.15.90:0/3810413349 shutdown_connections 2024-03-24T01:26:59.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.035+0000 7f5b3b400640 1 -- 172.21.15.90:0/3810413349 wait complete. 2024-03-24T01:26:59.053 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:26:59.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.262+0000 7fbc34600640 1 Processor -- start 2024-03-24T01:26:59.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.262+0000 7fbc34600640 1 -- start start 2024-03-24T01:26:59.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.264+0000 7fbc34600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fbc2c1454e0 0x7fbc2c0cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:59.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.264+0000 7fbc34600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbc2c06f000 0x7fbc2c06f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:59.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.264+0000 7fbc34600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc2c0cc660 con 0x7fbc2c0cdc50 2024-03-24T01:26:59.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.264+0000 7fbc34600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc2c0cc430 con 0x7fbc2c1454e0 2024-03-24T01:26:59.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.264+0000 7fbc34600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc2c0cc920 con 0x7fbc2c06f000 2024-03-24T01:26:59.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.264+0000 7fbc31c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fbc2c1454e0 0x7fbc2c0cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:59.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.264+0000 7fbc32600640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fbc2c0cdc50 0x7fbc2c06d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:53190/0 (socket says 172.21.15.90:53190) 2024-03-24T01:26:59.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.264+0000 7fbc32600640 1 -- 172.21.15.90:0/1521382595 learned_addr learned my addr 172.21.15.90:0/1521382595 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:26:59.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.264+0000 7fbc31200640 1 --2- 172.21.15.90:0/1521382595 >> v2:172.21.15.120:3300/0 conn(0x7fbc2c06f000 0x7fbc2c06f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:59.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.265+0000 7fbc31c00640 1 -- 172.21.15.90:0/1521382595 >> v2:172.21.15.120:3300/0 conn(0x7fbc2c06f000 msgr2=0x7fbc2c06f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:59.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.265+0000 7fbc31c00640 1 --2- 172.21.15.90:0/1521382595 >> v2:172.21.15.120:3300/0 conn(0x7fbc2c06f000 0x7fbc2c06f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:59.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.265+0000 7fbc31c00640 1 -- 172.21.15.90:0/1521382595 >> v1:172.21.15.90:6789/0 conn(0x7fbc2c0cdc50 legacy=0x7fbc2c06d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:59.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.265+0000 7fbc31c00640 1 -- 172.21.15.90:0/1521382595 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc2c0ccc00 con 0x7fbc2c1454e0 2024-03-24T01:26:59.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.266+0000 7fbc31c00640 1 --2- 172.21.15.90:0/1521382595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc2c1454e0 0x7fbc2c0cd6e0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c0b4ef0 tx=0x7fbc2002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f28a3b3e5849e67d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:59.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.266+0000 7fbc2be00640 1 -- 172.21.15.90:0/1521382595 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc20035020 con 0x7fbc2c1454e0 2024-03-24T01:26:59.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.266+0000 7fbc2be00640 1 -- 172.21.15.90:0/1521382595 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc20031830 con 0x7fbc2c1454e0 2024-03-24T01:26:59.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.266+0000 7fbc2be00640 1 -- 172.21.15.90:0/1521382595 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc20031c00 con 0x7fbc2c1454e0 2024-03-24T01:26:59.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.267+0000 7fbc34600640 1 -- 172.21.15.90:0/1521382595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc2c1454e0 msgr2=0x7fbc2c0cd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:59.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.267+0000 7fbc34600640 1 --2- 172.21.15.90:0/1521382595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc2c1454e0 0x7fbc2c0cd6e0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c0b4ef0 tx=0x7fbc2002b8d0 comp rx=0 tx=0).stop 2024-03-24T01:26:59.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.267+0000 7fbc34600640 1 -- 172.21.15.90:0/1521382595 shutdown_connections 2024-03-24T01:26:59.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.267+0000 7fbc34600640 1 --2- 172.21.15.90:0/1521382595 >> v2:172.21.15.120:3300/0 conn(0x7fbc2c06f000 0x7fbc2c06f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:59.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.267+0000 7fbc34600640 1 --2- 172.21.15.90:0/1521382595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc2c1454e0 0x7fbc2c0cd6e0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:59.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.267+0000 7fbc34600640 1 -- 172.21.15.90:0/1521382595 >> 172.21.15.90:0/1521382595 conn(0x7fbc2c06b2a0 msgr2=0x7fbc2c06b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:59.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.267+0000 7fbc34600640 1 -- 172.21.15.90:0/1521382595 shutdown_connections 2024-03-24T01:26:59.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.267+0000 7fbc34600640 1 -- 172.21.15.90:0/1521382595 wait complete. 2024-03-24T01:26:59.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.269+0000 7fbc34600640 1 Processor -- start 2024-03-24T01:26:59.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.269+0000 7fbc34600640 1 -- start start 2024-03-24T01:26:59.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.270+0000 7fbc34600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc2c06f000 0x7fbc2c1e5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:59.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.270+0000 7fbc34600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc2c0cdc50 0x7fbc2c0b3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:59.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.270+0000 7fbc34600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc2c1454e0 0x7fbc2c155310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:59.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.270+0000 7fbc34600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2c0ce0c0 con 0x7fbc2c06f000 2024-03-24T01:26:59.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.270+0000 7fbc34600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2c0cd1c0 con 0x7fbc2c1454e0 2024-03-24T01:26:59.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.270+0000 7fbc34600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2c0ce240 con 0x7fbc2c0cdc50 2024-03-24T01:26:59.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.270+0000 7fbc31c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc2c06f000 0x7fbc2c1e5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:59.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.270+0000 7fbc31c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc2c06f000 0x7fbc2c1e5d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35682/0 (socket says 172.21.15.90:35682) 2024-03-24T01:26:59.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.270+0000 7fbc31c00640 1 -- 172.21.15.90:0/2788327649 learned_addr learned my addr 172.21.15.90:0/2788327649 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:59.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.270+0000 7fbc31200640 1 --2- 172.21.15.90:0/2788327649 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc2c0cdc50 0x7fbc2c0b3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:59.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.270+0000 7fbc32600640 1 --2- 172.21.15.90:0/2788327649 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc2c1454e0 0x7fbc2c155310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:59.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.270+0000 7fbc31c00640 1 -- 172.21.15.90:0/2788327649 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc2c0cdc50 msgr2=0x7fbc2c0b3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:59.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.270+0000 7fbc31c00640 1 --2- 172.21.15.90:0/2788327649 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc2c0cdc50 0x7fbc2c0b3560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:59.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.270+0000 7fbc31c00640 1 -- 172.21.15.90:0/2788327649 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc2c1454e0 msgr2=0x7fbc2c155310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:59.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.270+0000 7fbc31c00640 1 --2- 172.21.15.90:0/2788327649 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc2c1454e0 0x7fbc2c155310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:59.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.270+0000 7fbc31c00640 1 -- 172.21.15.90:0/2788327649 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc2c0b3aa0 con 0x7fbc2c06f000 2024-03-24T01:26:59.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.270+0000 7fbc32600640 1 --2- 172.21.15.90:0/2788327649 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc2c1454e0 0x7fbc2c155310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:26:59.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.271+0000 7fbc31c00640 1 --2- 172.21.15.90:0/2788327649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc2c06f000 0x7fbc2c1e5d90 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fbc20000c00 tx=0x7fbc200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:59.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.271+0000 7fbc29600640 1 -- 172.21.15.90:0/2788327649 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc20040070 con 0x7fbc2c06f000 2024-03-24T01:26:59.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.271+0000 7fbc29600640 1 -- 172.21.15.90:0/2788327649 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc20039a20 con 0x7fbc2c06f000 2024-03-24T01:26:59.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.271+0000 7fbc29600640 1 -- 172.21.15.90:0/2788327649 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc20035040 con 0x7fbc2c06f000 2024-03-24T01:26:59.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.271+0000 7fbc34600640 1 -- 172.21.15.90:0/2788327649 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc2c143f10 con 0x7fbc2c06f000 2024-03-24T01:26:59.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.271+0000 7fbc34600640 1 -- 172.21.15.90:0/2788327649 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc2c1444d0 con 0x7fbc2c06f000 2024-03-24T01:26:59.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.272+0000 7fbc34600640 1 -- 172.21.15.90:0/2788327649 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc2c073570 con 0x7fbc2c06f000 2024-03-24T01:26:59.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.273+0000 7fbc29600640 1 -- 172.21.15.90:0/2788327649 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fbc2004b060 con 0x7fbc2c06f000 2024-03-24T01:26:59.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.275+0000 7fbc29600640 1 --2- 172.21.15.90:0/2788327649 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbc0c06bbd0 0x7fbc0c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:59.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.275+0000 7fbc29600640 1 -- 172.21.15.90:0/2788327649 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6216+0+0 (secure 0 0 0) 0x7fbc200ac600 con 0x7fbc2c06f000 2024-03-24T01:26:59.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.276+0000 7fbc31200640 1 --2- 172.21.15.90:0/2788327649 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbc0c06bbd0 0x7fbc0c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:59.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.277+0000 7fbc31200640 1 --2- 172.21.15.90:0/2788327649 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbc0c06bbd0 0x7fbc0c06e090 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c131d70 tx=0x7fbc18008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:59.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.279+0000 7fbc29600640 1 -- 172.21.15.90:0/2788327649 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fbc2c0b3aa0 con 0x7fbc2c06f000 2024-03-24T01:26:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.479+0000 7fbc34600640 1 -- 172.21.15.90:0/2788327649 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc2c04f4e0 con 0x7fbc2c06f000 2024-03-24T01:26:59.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.482+0000 7fbc29600640 1 -- 172.21.15.90:0/2788327649 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+17096 (secure 0 0 0) 0x7fbc2007ae50 con 0x7fbc2c06f000 2024-03-24T01:26:59.482 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:26:59.482 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":42,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:26:53.308852+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:25:57.398864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-24T01:25:58.451150+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.30","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:6835/2166652035":"2024-03-25T01:26:27.648052+0000","172.21.15.90:6834/2166652035":"2024-03-25T01:26:27.648052+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:26:59.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.484+0000 7fbc34600640 1 -- 172.21.15.90:0/2788327649 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbc0c06bbd0 msgr2=0x7fbc0c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:59.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.484+0000 7fbc34600640 1 --2- 172.21.15.90:0/2788327649 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbc0c06bbd0 0x7fbc0c06e090 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c131d70 tx=0x7fbc18008040 comp rx=0 tx=0).stop 2024-03-24T01:26:59.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.484+0000 7fbc34600640 1 -- 172.21.15.90:0/2788327649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc2c06f000 msgr2=0x7fbc2c1e5d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:59.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.484+0000 7fbc34600640 1 --2- 172.21.15.90:0/2788327649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc2c06f000 0x7fbc2c1e5d90 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fbc20000c00 tx=0x7fbc200046c0 comp rx=0 tx=0).stop 2024-03-24T01:26:59.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.484+0000 7fbc34600640 1 -- 172.21.15.90:0/2788327649 shutdown_connections 2024-03-24T01:26:59.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.484+0000 7fbc34600640 1 --2- 172.21.15.90:0/2788327649 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbc0c06bbd0 0x7fbc0c06e090 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:59.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.484+0000 7fbc34600640 1 --2- 172.21.15.90:0/2788327649 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc2c1454e0 0x7fbc2c155310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:59.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.484+0000 7fbc34600640 1 --2- 172.21.15.90:0/2788327649 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbc2c0cdc50 0x7fbc2c0b3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:59.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.484+0000 7fbc34600640 1 --2- 172.21.15.90:0/2788327649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc2c06f000 0x7fbc2c1e5d90 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:59.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.484+0000 7fbc34600640 1 -- 172.21.15.90:0/2788327649 >> 172.21.15.90:0/2788327649 conn(0x7fbc2c06b2a0 msgr2=0x7fbc2c15bb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:59.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.484+0000 7fbc34600640 1 -- 172.21.15.90:0/2788327649 shutdown_connections 2024-03-24T01:26:59.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.484+0000 7fbc34600640 1 -- 172.21.15.90:0/2788327649 wait complete. 2024-03-24T01:26:59.504 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-24T01:26:59.504 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:26:59.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.706+0000 7feb9e200640 1 Processor -- start 2024-03-24T01:26:59.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.706+0000 7feb9e200640 1 -- start start 2024-03-24T01:26:59.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.707+0000 7feb9e200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7feb9815dba0 0x7feb98162330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:59.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.707+0000 7feb9e200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb9815df80 0x7feb980b5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:59.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.708+0000 7feb9e200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7feb980b2960 0x7feb980b5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:59.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.708+0000 7feb9e200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb98162870 con 0x7feb980b2960 2024-03-24T01:26:59.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.708+0000 7feb9e200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb980ce4b0 con 0x7feb9815dba0 2024-03-24T01:26:59.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.708+0000 7feb9e200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb980ce790 con 0x7feb9815df80 2024-03-24T01:26:59.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.708+0000 7feb9d800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7feb980b2960 0x7feb980b5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:59.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.708+0000 7feb9d800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7feb980b2960 0x7feb980b5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35686/0 (socket says 172.21.15.90:35686) 2024-03-24T01:26:59.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.708+0000 7feb9d800640 1 -- 172.21.15.90:0/2901591554 learned_addr learned my addr 172.21.15.90:0/2901591554 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:59.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.708+0000 7feb9ce00640 1 --2- 172.21.15.90:0/2901591554 >> v2:172.21.15.120:3300/0 conn(0x7feb9815dba0 0x7feb98162330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:59.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.708+0000 7feb97e00640 1 --2- 172.21.15.90:0/2901591554 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb9815df80 0x7feb980b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:59.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.709+0000 7feb9d800640 1 -- 172.21.15.90:0/2901591554 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb9815df80 msgr2=0x7feb980b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:59.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.709+0000 7feb9d800640 1 --2- 172.21.15.90:0/2901591554 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb9815df80 0x7feb980b5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:59.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.709+0000 7feb9d800640 1 -- 172.21.15.90:0/2901591554 >> v2:172.21.15.120:3300/0 conn(0x7feb9815dba0 msgr2=0x7feb98162330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:26:59.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.709+0000 7feb9d800640 1 --2- 172.21.15.90:0/2901591554 >> v2:172.21.15.120:3300/0 conn(0x7feb9815dba0 0x7feb98162330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:59.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.709+0000 7feb9d800640 1 -- 172.21.15.90:0/2901591554 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb980cea70 con 0x7feb980b2960 2024-03-24T01:26:59.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.710+0000 7feb9d800640 1 --2- 172.21.15.90:0/2901591554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb980b2960 0x7feb980b5a70 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7feb9806d660 tx=0x7feb9002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8016db3eb90a28f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:59.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.710+0000 7feb97400640 1 -- 172.21.15.90:0/2901591554 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9002c9e0 con 0x7feb980b2960 2024-03-24T01:26:59.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.710+0000 7feb97400640 1 -- 172.21.15.90:0/2901591554 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb9002cb80 con 0x7feb980b2960 2024-03-24T01:26:59.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.710+0000 7feb97400640 1 -- 172.21.15.90:0/2901591554 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb900343f0 con 0x7feb980b2960 2024-03-24T01:26:59.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.711+0000 7feb9e200640 1 -- 172.21.15.90:0/2901591554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb980b2960 msgr2=0x7feb980b5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:59.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.711+0000 7feb9e200640 1 --2- 172.21.15.90:0/2901591554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb980b2960 0x7feb980b5a70 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7feb9806d660 tx=0x7feb9002be40 comp rx=0 tx=0).stop 2024-03-24T01:26:59.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.711+0000 7feb9e200640 1 -- 172.21.15.90:0/2901591554 shutdown_connections 2024-03-24T01:26:59.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.711+0000 7feb9e200640 1 --2- 172.21.15.90:0/2901591554 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb9815df80 0x7feb980b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:59.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.711+0000 7feb9e200640 1 --2- 172.21.15.90:0/2901591554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb980b2960 0x7feb980b5a70 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:59.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.711+0000 7feb9e200640 1 --2- 172.21.15.90:0/2901591554 >> v2:172.21.15.120:3300/0 conn(0x7feb9815dba0 0x7feb98162330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:59.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.711+0000 7feb9e200640 1 -- 172.21.15.90:0/2901591554 >> 172.21.15.90:0/2901591554 conn(0x7feb980d0530 msgr2=0x7feb980d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:59.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.711+0000 7feb9e200640 1 -- 172.21.15.90:0/2901591554 shutdown_connections 2024-03-24T01:26:59.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.711+0000 7feb9e200640 1 -- 172.21.15.90:0/2901591554 wait complete. 2024-03-24T01:26:59.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.713+0000 7feb9e200640 1 Processor -- start 2024-03-24T01:26:59.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.713+0000 7feb9e200640 1 -- start start 2024-03-24T01:26:59.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.714+0000 7feb9e200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb980b2960 0x7feb98151fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:59.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.714+0000 7feb9e200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb9815dba0 0x7feb9815fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:59.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.714+0000 7feb9e200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb9815df80 0x7feb980c92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:59.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.714+0000 7feb9e200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb9806c6c0 con 0x7feb9815df80 2024-03-24T01:26:59.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.714+0000 7feb9e200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb9806c540 con 0x7feb980b2960 2024-03-24T01:26:59.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.714+0000 7feb9e200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb9806c840 con 0x7feb9815dba0 2024-03-24T01:26:59.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.714+0000 7feb9d800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb9815df80 0x7feb980c92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:59.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.714+0000 7feb9d800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb9815df80 0x7feb980c92b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35696/0 (socket says 172.21.15.90:35696) 2024-03-24T01:26:59.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.714+0000 7feb9d800640 1 -- 172.21.15.90:0/1657301580 learned_addr learned my addr 172.21.15.90:0/1657301580 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:26:59.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.714+0000 7feb9ce00640 1 --2- 172.21.15.90:0/1657301580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb980b2960 0x7feb98151fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:59.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.714+0000 7feb97e00640 1 --2- 172.21.15.90:0/1657301580 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb9815dba0 0x7feb9815fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:59.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.714+0000 7feb9d800640 1 -- 172.21.15.90:0/1657301580 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb9815dba0 msgr2=0x7feb9815fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:59.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.714+0000 7feb9d800640 1 --2- 172.21.15.90:0/1657301580 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb9815dba0 0x7feb9815fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:59.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.714+0000 7feb9d800640 1 -- 172.21.15.90:0/1657301580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb980b2960 msgr2=0x7feb98151fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:59.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.714+0000 7feb9d800640 1 --2- 172.21.15.90:0/1657301580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb980b2960 0x7feb98151fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:59.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.714+0000 7feb9d800640 1 -- 172.21.15.90:0/1657301580 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb980b35b0 con 0x7feb9815df80 2024-03-24T01:26:59.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.715+0000 7feb9d800640 1 --2- 172.21.15.90:0/1657301580 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb9815df80 0x7feb980c92b0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7feb90036040 tx=0x7feb90004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:59.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.715+0000 7feb94c00640 1 -- 172.21.15.90:0/1657301580 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb90043020 con 0x7feb9815df80 2024-03-24T01:26:59.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.715+0000 7feb94c00640 1 -- 172.21.15.90:0/1657301580 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb9003e450 con 0x7feb9815df80 2024-03-24T01:26:59.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.715+0000 7feb9e200640 1 -- 172.21.15.90:0/1657301580 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb980b3850 con 0x7feb9815df80 2024-03-24T01:26:59.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.715+0000 7feb94c00640 1 -- 172.21.15.90:0/1657301580 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb90009510 con 0x7feb9815df80 2024-03-24T01:26:59.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.715+0000 7feb9e200640 1 -- 172.21.15.90:0/1657301580 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb980b3e10 con 0x7feb9815df80 2024-03-24T01:26:59.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.717+0000 7feb9e200640 1 -- 172.21.15.90:0/1657301580 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb980b25f0 con 0x7feb9815df80 2024-03-24T01:26:59.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.717+0000 7feb94c00640 1 -- 172.21.15.90:0/1657301580 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7feb90046020 con 0x7feb9815df80 2024-03-24T01:26:59.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.719+0000 7feb94c00640 1 --2- 172.21.15.90:0/1657301580 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7feb7806bbd0 0x7feb7806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:26:59.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.719+0000 7feb94c00640 1 -- 172.21.15.90:0/1657301580 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6216+0+0 (secure 0 0 0) 0x7feb900ad4e0 con 0x7feb9815df80 2024-03-24T01:26:59.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.720+0000 7feb9ce00640 1 --2- 172.21.15.90:0/1657301580 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7feb7806bbd0 0x7feb7806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:26:59.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.721+0000 7feb9ce00640 1 --2- 172.21.15.90:0/1657301580 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7feb7806bbd0 0x7feb7806e090 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7feb8c006fd0 tx=0x7feb8c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:26:59.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.723+0000 7feb94c00640 1 -- 172.21.15.90:0/1657301580 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7feb980b25f0 con 0x7feb9815df80 2024-03-24T01:26:59.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.961+0000 7feb9e200640 1 -- 172.21.15.90:0/1657301580 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb9804f400 con 0x7feb9815df80 2024-03-24T01:26:59.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.962+0000 7feb94c00640 1 -- 172.21.15.90:0/1657301580 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4972 (secure 0 0 0) 0x7feb9004d030 con 0x7feb9815df80 2024-03-24T01:26:59.963 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:26:59.963 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/296084634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":296084634},{"type":"v1","addr":"172.21.15.120:6835","nonce":296084634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/797786252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":797786252},{"type":"v1","addr":"172.21.15.120:6837","nonce":797786252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5490,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6833/1983177399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1983177399},{"type":"v1","addr":"172.21.15.90:6833","nonce":1983177399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T01:25:59.440431+0000","modified":"2024-03-24T01:26:40.166132+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5787},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5787":{"gid":5787,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.90:6835/11525832","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":11525832},{"type":"v1","addr":"172.21.15.90:6835","nonce":11525832}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5787,"qdb_cluster":[5787]},"id":2}]} 2024-03-24T01:26:59.963 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 25 2024-03-24T01:26:59.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.965+0000 7feb9e200640 1 -- 172.21.15.90:0/1657301580 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7feb7806bbd0 msgr2=0x7feb7806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:59.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.965+0000 7feb9e200640 1 --2- 172.21.15.90:0/1657301580 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7feb7806bbd0 0x7feb7806e090 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7feb8c006fd0 tx=0x7feb8c008040 comp rx=0 tx=0).stop 2024-03-24T01:26:59.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.965+0000 7feb9e200640 1 -- 172.21.15.90:0/1657301580 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb9815df80 msgr2=0x7feb980c92b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:26:59.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.965+0000 7feb9e200640 1 --2- 172.21.15.90:0/1657301580 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb9815df80 0x7feb980c92b0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7feb90036040 tx=0x7feb90004a40 comp rx=0 tx=0).stop 2024-03-24T01:26:59.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.965+0000 7feb9e200640 1 -- 172.21.15.90:0/1657301580 shutdown_connections 2024-03-24T01:26:59.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.965+0000 7feb9e200640 1 --2- 172.21.15.90:0/1657301580 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7feb7806bbd0 0x7feb7806e090 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:59.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.965+0000 7feb9e200640 1 --2- 172.21.15.90:0/1657301580 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb9815df80 0x7feb980c92b0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:59.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.965+0000 7feb9e200640 1 --2- 172.21.15.90:0/1657301580 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb9815dba0 0x7feb9815fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:59.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.965+0000 7feb9e200640 1 --2- 172.21.15.90:0/1657301580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb980b2960 0x7feb98151fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:26:59.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.965+0000 7feb9e200640 1 -- 172.21.15.90:0/1657301580 >> 172.21.15.90:0/1657301580 conn(0x7feb980d0530 msgr2=0x7feb980cf500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:26:59.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.965+0000 7feb9e200640 1 -- 172.21.15.90:0/1657301580 shutdown_connections 2024-03-24T01:26:59.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:26:59.965+0000 7feb9e200640 1 -- 172.21.15.90:0/1657301580 wait complete. 2024-03-24T01:26:59.983 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:27:00.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.193+0000 7ff151a00640 1 Processor -- start 2024-03-24T01:27:00.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.193+0000 7ff151a00640 1 -- start start 2024-03-24T01:27:00.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.194+0000 7ff151a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff14c0d1620 0x7ff14c0cf210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:00.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.194+0000 7ff151a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff14c06cc40 0x7ff14c0b6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:00.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.194+0000 7ff151a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14c0d1b20 con 0x7ff14c06c860 2024-03-24T01:27:00.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.194+0000 7ff151a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff14c0d2730 con 0x7ff14c0d1620 2024-03-24T01:27:00.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.194+0000 7ff151a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff14c0d1e00 con 0x7ff14c06cc40 2024-03-24T01:27:00.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.194+0000 7ff143e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff14c0d1620 0x7ff14c0cf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:00.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.194+0000 7ff14be00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff14c06c860 0x7ff14c0d2bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40664/0 (socket says 172.21.15.90:40664) 2024-03-24T01:27:00.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.194+0000 7ff14be00640 1 -- 172.21.15.90:0/1892187135 learned_addr learned my addr 172.21.15.90:0/1892187135 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:27:00.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.195+0000 7ff151000640 1 --2- 172.21.15.90:0/1892187135 >> v2:172.21.15.120:3300/0 conn(0x7ff14c06cc40 0x7ff14c0b6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:00.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.195+0000 7ff143e00640 1 -- 172.21.15.90:0/1892187135 >> v2:172.21.15.120:3300/0 conn(0x7ff14c06cc40 msgr2=0x7ff14c0b6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:00.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.195+0000 7ff143e00640 1 --2- 172.21.15.90:0/1892187135 >> v2:172.21.15.120:3300/0 conn(0x7ff14c06cc40 0x7ff14c0b6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:00.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.195+0000 7ff143e00640 1 -- 172.21.15.90:0/1892187135 >> v1:172.21.15.90:6789/0 conn(0x7ff14c06c860 legacy=0x7ff14c0d2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:00.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.195+0000 7ff143e00640 1 -- 172.21.15.90:0/1892187135 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14c0d20e0 con 0x7ff14c0d1620 2024-03-24T01:27:00.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.195+0000 7ff14b400640 1 -- 172.21.15.90:0/1892187135 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3373222101 0 0) 0x7ff14c0d1b20 con 0x7ff14c06c860 2024-03-24T01:27:00.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.196+0000 7ff143e00640 1 --2- 172.21.15.90:0/1892187135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff14c0d1620 0x7ff14c0cf210 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7ff14c0d28b0 tx=0x7ff13402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=531083e676466fee server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:00.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.197+0000 7ff14b400640 1 -- 172.21.15.90:0/1892187135 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff134035020 con 0x7ff14c0d1620 2024-03-24T01:27:00.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.197+0000 7ff14b400640 1 -- 172.21.15.90:0/1892187135 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff134031830 con 0x7ff14c0d1620 2024-03-24T01:27:00.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.197+0000 7ff14b400640 1 -- 172.21.15.90:0/1892187135 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff134031c00 con 0x7ff14c0d1620 2024-03-24T01:27:00.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.197+0000 7ff151a00640 1 -- 172.21.15.90:0/1892187135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff14c0d1620 msgr2=0x7ff14c0cf210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:00.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.197+0000 7ff151a00640 1 --2- 172.21.15.90:0/1892187135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff14c0d1620 0x7ff14c0cf210 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7ff14c0d28b0 tx=0x7ff13402b8d0 comp rx=0 tx=0).stop 2024-03-24T01:27:00.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.197+0000 7ff151a00640 1 -- 172.21.15.90:0/1892187135 shutdown_connections 2024-03-24T01:27:00.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.197+0000 7ff151a00640 1 --2- 172.21.15.90:0/1892187135 >> v2:172.21.15.120:3300/0 conn(0x7ff14c06cc40 0x7ff14c0b6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:00.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.197+0000 7ff151a00640 1 --2- 172.21.15.90:0/1892187135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff14c0d1620 0x7ff14c0cf210 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:00.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.197+0000 7ff151a00640 1 -- 172.21.15.90:0/1892187135 >> 172.21.15.90:0/1892187135 conn(0x7ff14c0d0560 msgr2=0x7ff14c0d0970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:00.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.197+0000 7ff151a00640 1 -- 172.21.15.90:0/1892187135 shutdown_connections 2024-03-24T01:27:00.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.197+0000 7ff151a00640 1 -- 172.21.15.90:0/1892187135 wait complete. 2024-03-24T01:27:00.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.200+0000 7ff151a00640 1 Processor -- start 2024-03-24T01:27:00.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.200+0000 7ff151a00640 1 -- start start 2024-03-24T01:27:00.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.202+0000 7ff151a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff14c06c860 0x7ff14c0c5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:00.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.202+0000 7ff151a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff14c06cc40 0x7ff14c1537f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:00.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.202+0000 7ff151a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff14c0d1620 0x7ff14c153d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:00.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.202+0000 7ff151a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c0c7300 con 0x7ff14c06c860 2024-03-24T01:27:00.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.202+0000 7ff151a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c0c7180 con 0x7ff14c06cc40 2024-03-24T01:27:00.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.202+0000 7ff151a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c0d32d0 con 0x7ff14c0d1620 2024-03-24T01:27:00.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.202+0000 7ff14be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff14c06c860 0x7ff14c0c5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:00.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.202+0000 7ff14be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff14c06c860 0x7ff14c0c5ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49340/0 (socket says 172.21.15.90:49340) 2024-03-24T01:27:00.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.202+0000 7ff14be00640 1 -- 172.21.15.90:0/3975908426 learned_addr learned my addr 172.21.15.90:0/3975908426 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:00.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.202+0000 7ff14be00640 1 -- 172.21.15.90:0/3975908426 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff14c0d1620 msgr2=0x7ff14c153d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:00.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.202+0000 7ff14be00640 1 --2- 172.21.15.90:0/3975908426 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff14c0d1620 0x7ff14c153d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:00.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.202+0000 7ff14be00640 1 -- 172.21.15.90:0/3975908426 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff14c06cc40 msgr2=0x7ff14c1537f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:00.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.203+0000 7ff14be00640 1 --2- 172.21.15.90:0/3975908426 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff14c06cc40 0x7ff14c1537f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:00.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.203+0000 7ff14be00640 1 -- 172.21.15.90:0/3975908426 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14c158fa0 con 0x7ff14c06c860 2024-03-24T01:27:00.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.203+0000 7ff14be00640 1 --2- 172.21.15.90:0/3975908426 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff14c06c860 0x7ff14c0c5ef0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7ff13c00c430 tx=0x7ff13c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:00.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.203+0000 7ff148c00640 1 -- 172.21.15.90:0/3975908426 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13c0074e0 con 0x7ff14c06c860 2024-03-24T01:27:00.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.203+0000 7ff148c00640 1 -- 172.21.15.90:0/3975908426 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff13c0047e0 con 0x7ff14c06c860 2024-03-24T01:27:00.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.203+0000 7ff148c00640 1 -- 172.21.15.90:0/3975908426 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13c004bd0 con 0x7ff14c06c860 2024-03-24T01:27:00.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.203+0000 7ff151a00640 1 -- 172.21.15.90:0/3975908426 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff14c0ca6a0 con 0x7ff14c06c860 2024-03-24T01:27:00.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.203+0000 7ff151a00640 1 -- 172.21.15.90:0/3975908426 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff14c1567d0 con 0x7ff14c06c860 2024-03-24T01:27:00.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.204+0000 7ff151a00640 1 -- 172.21.15.90:0/3975908426 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff14c04f400 con 0x7ff14c06c860 2024-03-24T01:27:00.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.205+0000 7ff148c00640 1 -- 172.21.15.90:0/3975908426 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff13c011070 con 0x7ff14c06c860 2024-03-24T01:27:00.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.206+0000 7ff148c00640 1 --2- 172.21.15.90:0/3975908426 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff12806bbd0 0x7ff12806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:00.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.206+0000 7ff148c00640 1 -- 172.21.15.90:0/3975908426 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6216+0+0 (secure 0 0 0) 0x7ff13c08aa00 con 0x7ff14c06c860 2024-03-24T01:27:00.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.207+0000 7ff143e00640 1 --2- 172.21.15.90:0/3975908426 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff12806bbd0 0x7ff12806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:00.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.208+0000 7ff143e00640 1 --2- 172.21.15.90:0/3975908426 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff12806bbd0 0x7ff12806e090 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff134036000 tx=0x7ff134030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:00.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.211+0000 7ff148c00640 1 -- 172.21.15.90:0/3975908426 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff13c059250 con 0x7ff14c06c860 2024-03-24T01:27:00.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.449+0000 7ff151a00640 1 -- 172.21.15.90:0/3975908426 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff14c0ce3d0 con 0x7ff14c06c860 2024-03-24T01:27:00.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.450+0000 7ff148c00640 1 -- 172.21.15.90:0/3975908426 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4972 (secure 0 0 0) 0x7ff13c058b50 con 0x7ff14c06c860 2024-03-24T01:27:00.451 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:27:00.451 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/296084634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":296084634},{"type":"v1","addr":"172.21.15.120:6835","nonce":296084634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/797786252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":797786252},{"type":"v1","addr":"172.21.15.120:6837","nonce":797786252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5490,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6833/1983177399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1983177399},{"type":"v1","addr":"172.21.15.90:6833","nonce":1983177399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T01:25:59.440431+0000","modified":"2024-03-24T01:26:40.166132+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5787},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5787":{"gid":5787,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.90:6835/11525832","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":11525832},{"type":"v1","addr":"172.21.15.90:6835","nonce":11525832}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5787,"qdb_cluster":[5787]},"id":2}]} 2024-03-24T01:27:00.452 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 25 2024-03-24T01:27:00.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.453+0000 7ff151a00640 1 -- 172.21.15.90:0/3975908426 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff12806bbd0 msgr2=0x7ff12806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:00.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.453+0000 7ff151a00640 1 --2- 172.21.15.90:0/3975908426 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff12806bbd0 0x7ff12806e090 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff134036000 tx=0x7ff134030000 comp rx=0 tx=0).stop 2024-03-24T01:27:00.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.453+0000 7ff151a00640 1 -- 172.21.15.90:0/3975908426 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff14c06c860 msgr2=0x7ff14c0c5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:00.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.453+0000 7ff151a00640 1 --2- 172.21.15.90:0/3975908426 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff14c06c860 0x7ff14c0c5ef0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7ff13c00c430 tx=0x7ff13c00c8f0 comp rx=0 tx=0).stop 2024-03-24T01:27:00.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.453+0000 7ff151a00640 1 -- 172.21.15.90:0/3975908426 shutdown_connections 2024-03-24T01:27:00.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.453+0000 7ff151a00640 1 --2- 172.21.15.90:0/3975908426 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff12806bbd0 0x7ff12806e090 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:00.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.453+0000 7ff151a00640 1 --2- 172.21.15.90:0/3975908426 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff14c0d1620 0x7ff14c153d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:00.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.453+0000 7ff151a00640 1 --2- 172.21.15.90:0/3975908426 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff14c06cc40 0x7ff14c1537f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:00.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.453+0000 7ff151a00640 1 --2- 172.21.15.90:0/3975908426 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff14c06c860 0x7ff14c0c5ef0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:00.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.453+0000 7ff151a00640 1 -- 172.21.15.90:0/3975908426 >> 172.21.15.90:0/3975908426 conn(0x7ff14c0d0560 msgr2=0x7ff14c0d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:00.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.453+0000 7ff151a00640 1 -- 172.21.15.90:0/3975908426 shutdown_connections 2024-03-24T01:27:00.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.453+0000 7ff151a00640 1 -- 172.21.15.90:0/3975908426 wait complete. 2024-03-24T01:27:00.473 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:27:00.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.679+0000 7f1239a00640 1 Processor -- start 2024-03-24T01:27:00.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.679+0000 7f1239a00640 1 -- start start 2024-03-24T01:27:00.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.680+0000 7f1239a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f12340cc150 0x7f12340ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:00.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.680+0000 7f1239a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f123406b7d0 0x7f12340b3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:00.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.680+0000 7f1239a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12340cd490 con 0x7f12341603e0 2024-03-24T01:27:00.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.680+0000 7f1239a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f123415f900 con 0x7f12340cc150 2024-03-24T01:27:00.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.680+0000 7f1239a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f123415fbe0 con 0x7f123406b7d0 2024-03-24T01:27:00.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.681+0000 7f1232a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f12340cc150 0x7f12340ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:00.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.681+0000 7f1232a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f12340cc150 0x7f12340ccb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49350/0 (socket says 172.21.15.90:49350) 2024-03-24T01:27:00.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.681+0000 7f1232a00640 1 -- 172.21.15.90:0/1216470001 learned_addr learned my addr 172.21.15.90:0/1216470001 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:00.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.681+0000 7f1233400640 1 --2- 172.21.15.90:0/1216470001 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f123406b7d0 0x7f12340b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:00.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.681+0000 7f1232a00640 1 -- 172.21.15.90:0/1216470001 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f123406b7d0 msgr2=0x7f12340b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:00.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.681+0000 7f1232a00640 1 --2- 172.21.15.90:0/1216470001 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f123406b7d0 0x7f12340b3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:00.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.681+0000 7f1232a00640 1 -- 172.21.15.90:0/1216470001 >> v1:172.21.15.90:6789/0 conn(0x7f12341603e0 legacy=0x7f123406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:00.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.681+0000 7f1232a00640 1 -- 172.21.15.90:0/1216470001 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f123415fec0 con 0x7f12340cc150 2024-03-24T01:27:00.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.681+0000 7f1232000640 1 -- 172.21.15.90:0/1216470001 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1020845672 0 0) 0x7f12340cd490 con 0x7f12341603e0 2024-03-24T01:27:00.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.682+0000 7f1232a00640 1 --2- 172.21.15.90:0/1216470001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12340cc150 0x7f12340ccb60 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f1224009e50 tx=0x7f122402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4395f7e247339940 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:00.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.683+0000 7f1232000640 1 -- 172.21.15.90:0/1216470001 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122402cb00 con 0x7f12340cc150 2024-03-24T01:27:00.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.683+0000 7f1232000640 1 -- 172.21.15.90:0/1216470001 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f122402cca0 con 0x7f12340cc150 2024-03-24T01:27:00.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.683+0000 7f1232000640 1 -- 172.21.15.90:0/1216470001 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1224034610 con 0x7f12340cc150 2024-03-24T01:27:00.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.683+0000 7f1239a00640 1 -- 172.21.15.90:0/1216470001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12340cc150 msgr2=0x7f12340ccb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:00.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.683+0000 7f1239a00640 1 --2- 172.21.15.90:0/1216470001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12340cc150 0x7f12340ccb60 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f1224009e50 tx=0x7f122402bf60 comp rx=0 tx=0).stop 2024-03-24T01:27:00.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.683+0000 7f1239a00640 1 -- 172.21.15.90:0/1216470001 shutdown_connections 2024-03-24T01:27:00.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.683+0000 7f1239a00640 1 --2- 172.21.15.90:0/1216470001 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f123406b7d0 0x7f12340b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:00.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.683+0000 7f1239a00640 1 --2- 172.21.15.90:0/1216470001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12340cc150 0x7f12340ccb60 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:00.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.683+0000 7f1239a00640 1 -- 172.21.15.90:0/1216470001 >> 172.21.15.90:0/1216470001 conn(0x7f12340b1770 msgr2=0x7f12340b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:00.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.683+0000 7f1239a00640 1 -- 172.21.15.90:0/1216470001 shutdown_connections 2024-03-24T01:27:00.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.683+0000 7f1239a00640 1 -- 172.21.15.90:0/1216470001 wait complete. 2024-03-24T01:27:00.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.686+0000 7f1239a00640 1 Processor -- start 2024-03-24T01:27:00.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.686+0000 7f1239a00640 1 -- start start 2024-03-24T01:27:00.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.688+0000 7f1239a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f123406b7d0 0x7f12340ad2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:00.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.688+0000 7f1239a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12340cc150 0x7f123415b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:00.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.688+0000 7f1239a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12341603e0 0x7f12341411e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:00.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.688+0000 7f1239a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12341455b0 con 0x7f123406b7d0 2024-03-24T01:27:00.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.688+0000 7f1239a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1234145430 con 0x7f12340cc150 2024-03-24T01:27:00.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.688+0000 7f1239a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12340b3c90 con 0x7f12341603e0 2024-03-24T01:27:00.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.688+0000 7f1232a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f123406b7d0 0x7f12340ad2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:00.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.688+0000 7f1232a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f123406b7d0 0x7f12340ad2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49366/0 (socket says 172.21.15.90:49366) 2024-03-24T01:27:00.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.688+0000 7f1232a00640 1 -- 172.21.15.90:0/527743022 learned_addr learned my addr 172.21.15.90:0/527743022 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:00.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.688+0000 7f1232a00640 1 -- 172.21.15.90:0/527743022 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12341603e0 msgr2=0x7f12341411e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:00.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.688+0000 7f122a000640 1 --2- 172.21.15.90:0/527743022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12340cc150 0x7f123415b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:00.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.688+0000 7f1233400640 1 --2- 172.21.15.90:0/527743022 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12341603e0 0x7f12341411e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:00.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.688+0000 7f1232a00640 1 --2- 172.21.15.90:0/527743022 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12341603e0 0x7f12341411e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:00.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.688+0000 7f1232a00640 1 -- 172.21.15.90:0/527743022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12340cc150 msgr2=0x7f123415b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:00.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.688+0000 7f1232a00640 1 --2- 172.21.15.90:0/527743022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12340cc150 0x7f123415b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:00.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.688+0000 7f1232a00640 1 -- 172.21.15.90:0/527743022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12340adb20 con 0x7f123406b7d0 2024-03-24T01:27:00.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.688+0000 7f122a000640 1 --2- 172.21.15.90:0/527743022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12340cc150 0x7f123415b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:27:00.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.689+0000 7f1232a00640 1 --2- 172.21.15.90:0/527743022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f123406b7d0 0x7f12340ad2f0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f1224009e50 tx=0x7f1224004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:00.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.689+0000 7f122b400640 1 -- 172.21.15.90:0/527743022 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1224043020 con 0x7f123406b7d0 2024-03-24T01:27:00.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.689+0000 7f122b400640 1 -- 172.21.15.90:0/527743022 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f122402ce50 con 0x7f123406b7d0 2024-03-24T01:27:00.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.689+0000 7f122b400640 1 -- 172.21.15.90:0/527743022 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122403e690 con 0x7f123406b7d0 2024-03-24T01:27:00.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.689+0000 7f1239a00640 1 -- 172.21.15.90:0/527743022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12340ade00 con 0x7f123406b7d0 2024-03-24T01:27:00.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.689+0000 7f1239a00640 1 -- 172.21.15.90:0/527743022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12340ae3c0 con 0x7f123406b7d0 2024-03-24T01:27:00.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.690+0000 7f1239a00640 1 -- 172.21.15.90:0/527743022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1234068060 con 0x7f123406b7d0 2024-03-24T01:27:00.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.691+0000 7f122b400640 1 -- 172.21.15.90:0/527743022 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1224046020 con 0x7f123406b7d0 2024-03-24T01:27:00.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.692+0000 7f122b400640 1 --2- 172.21.15.90:0/527743022 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f121006bbd0 0x7f121006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:00.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.692+0000 7f122b400640 1 -- 172.21.15.90:0/527743022 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f12240ad530 con 0x7f123406b7d0 2024-03-24T01:27:00.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.693+0000 7f122a000640 1 --2- 172.21.15.90:0/527743022 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f121006bbd0 0x7f121006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:00.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.694+0000 7f122a000640 1 --2- 172.21.15.90:0/527743022 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f121006bbd0 0x7f121006e090 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f12340cc9b0 tx=0x7f121c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:00.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.701+0000 7f122b400640 1 -- 172.21.15.90:0/527743022 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f122404d030 con 0x7f123406b7d0 2024-03-24T01:27:00.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.887+0000 7f1239a00640 1 -- 172.21.15.90:0/527743022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12340cd9e0 con 0x7f123406b7d0 2024-03-24T01:27:00.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.890+0000 7f122b400640 1 -- 172.21.15.90:0/527743022 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+17096 (secure 0 0 0) 0x7f122407bd80 con 0x7f123406b7d0 2024-03-24T01:27:00.891 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:27:00.891 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":42,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:26:53.308852+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:25:57.398864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-24T01:25:58.451150+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.30","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:6835/2166652035":"2024-03-25T01:26:27.648052+0000","172.21.15.90:6834/2166652035":"2024-03-25T01:26:27.648052+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:27:00.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.893+0000 7f1239a00640 1 -- 172.21.15.90:0/527743022 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f121006bbd0 msgr2=0x7f121006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:00.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.893+0000 7f1239a00640 1 --2- 172.21.15.90:0/527743022 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f121006bbd0 0x7f121006e090 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f12340cc9b0 tx=0x7f121c009040 comp rx=0 tx=0).stop 2024-03-24T01:27:00.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.893+0000 7f1239a00640 1 -- 172.21.15.90:0/527743022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f123406b7d0 msgr2=0x7f12340ad2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:00.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.893+0000 7f1239a00640 1 --2- 172.21.15.90:0/527743022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f123406b7d0 0x7f12340ad2f0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f1224009e50 tx=0x7f1224004a40 comp rx=0 tx=0).stop 2024-03-24T01:27:00.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.894+0000 7f1239a00640 1 -- 172.21.15.90:0/527743022 shutdown_connections 2024-03-24T01:27:00.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.894+0000 7f1239a00640 1 --2- 172.21.15.90:0/527743022 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f121006bbd0 0x7f121006e090 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:00.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.894+0000 7f1239a00640 1 --2- 172.21.15.90:0/527743022 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f12341603e0 0x7f12341411e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:00.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.894+0000 7f1239a00640 1 --2- 172.21.15.90:0/527743022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12340cc150 0x7f123415b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:00.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.894+0000 7f1239a00640 1 --2- 172.21.15.90:0/527743022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f123406b7d0 0x7f12340ad2f0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:00.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.894+0000 7f1239a00640 1 -- 172.21.15.90:0/527743022 >> 172.21.15.90:0/527743022 conn(0x7f12340b1770 msgr2=0x7f1234142b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:00.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.894+0000 7f1239a00640 1 -- 172.21.15.90:0/527743022 shutdown_connections 2024-03-24T01:27:00.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:00.894+0000 7f1239a00640 1 -- 172.21.15.90:0/527743022 wait complete. 2024-03-24T01:27:00.918 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T01:27:01.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.119+0000 7f096de00640 1 Processor -- start 2024-03-24T01:27:01.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.119+0000 7f096de00640 1 -- start start 2024-03-24T01:27:01.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.120+0000 7f096de00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f09680b97b0 0x7f09680cead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:01.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.120+0000 7f096de00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f09680d3830 0x7f09680cf010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:01.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.120+0000 7f096de00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09680d2790 con 0x7f096815de60 2024-03-24T01:27:01.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.120+0000 7f096de00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09680d2550 con 0x7f09680b97b0 2024-03-24T01:27:01.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.120+0000 7f096de00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09680b7eb0 con 0x7f09680d3830 2024-03-24T01:27:01.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.121+0000 7f095fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f09680b97b0 0x7f09680cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:01.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.121+0000 7f095fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f09680b97b0 0x7f09680cead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49380/0 (socket says 172.21.15.90:49380) 2024-03-24T01:27:01.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.121+0000 7f095fe00640 1 -- 172.21.15.90:0/1301255966 learned_addr learned my addr 172.21.15.90:0/1301255966 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:01.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.121+0000 7f0967400640 1 --2- 172.21.15.90:0/1301255966 >> v2:172.21.15.120:3300/0 conn(0x7f09680d3830 0x7f09680cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:01.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.121+0000 7f095fe00640 1 -- 172.21.15.90:0/1301255966 >> v2:172.21.15.120:3300/0 conn(0x7f09680d3830 msgr2=0x7f09680cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:01.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.121+0000 7f095fe00640 1 --2- 172.21.15.90:0/1301255966 >> v2:172.21.15.120:3300/0 conn(0x7f09680d3830 0x7f09680cf010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:01.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.121+0000 7f095fe00640 1 -- 172.21.15.90:0/1301255966 >> v1:172.21.15.90:6789/0 conn(0x7f096815de60 legacy=0x7f09680d1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:01.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.121+0000 7f095fe00640 1 -- 172.21.15.90:0/1301255966 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09680d2a70 con 0x7f09680b97b0 2024-03-24T01:27:01.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.121+0000 7f0967400640 1 --2- 172.21.15.90:0/1301255966 >> v2:172.21.15.120:3300/0 conn(0x7f09680d3830 0x7f09680cf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:27:01.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.121+0000 7f0966000640 1 -- 172.21.15.90:0/1301255966 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1118768811 0 0) 0x7f09680d2790 con 0x7f096815de60 2024-03-24T01:27:01.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.123+0000 7f095fe00640 1 --2- 172.21.15.90:0/1301255966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09680b97b0 0x7f09680cead0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f09680cfbd0 tx=0x7f095002b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc9d246fef15ada1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:01.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.123+0000 7f0966000640 1 -- 172.21.15.90:0/1301255966 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0950034070 con 0x7f09680b97b0 2024-03-24T01:27:01.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.123+0000 7f0966000640 1 -- 172.21.15.90:0/1301255966 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0950030870 con 0x7f09680b97b0 2024-03-24T01:27:01.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.123+0000 7f0966000640 1 -- 172.21.15.90:0/1301255966 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0950030c40 con 0x7f09680b97b0 2024-03-24T01:27:01.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.123+0000 7f096de00640 1 -- 172.21.15.90:0/1301255966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09680b97b0 msgr2=0x7f09680cead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:01.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.123+0000 7f096de00640 1 --2- 172.21.15.90:0/1301255966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09680b97b0 0x7f09680cead0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f09680cfbd0 tx=0x7f095002b9b0 comp rx=0 tx=0).stop 2024-03-24T01:27:01.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.124+0000 7f096de00640 1 -- 172.21.15.90:0/1301255966 shutdown_connections 2024-03-24T01:27:01.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.124+0000 7f096de00640 1 --2- 172.21.15.90:0/1301255966 >> v2:172.21.15.120:3300/0 conn(0x7f09680d3830 0x7f09680cf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:01.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.124+0000 7f096de00640 1 --2- 172.21.15.90:0/1301255966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09680b97b0 0x7f09680cead0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:01.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.124+0000 7f096de00640 1 -- 172.21.15.90:0/1301255966 >> 172.21.15.90:0/1301255966 conn(0x7f0968072c10 msgr2=0x7f0968073020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:01.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.124+0000 7f096de00640 1 -- 172.21.15.90:0/1301255966 shutdown_connections 2024-03-24T01:27:01.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.124+0000 7f096de00640 1 -- 172.21.15.90:0/1301255966 wait complete. 2024-03-24T01:27:01.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.126+0000 7f096de00640 1 Processor -- start 2024-03-24T01:27:01.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.127+0000 7f096de00640 1 -- start start 2024-03-24T01:27:01.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.128+0000 7f096de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09680d3830 0x7f0968154460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:01.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.128+0000 7f096de00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f096815de60 0x7f0968156370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:01.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.128+0000 7f096de00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09680c9a40 0x7f0968159e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:01.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.128+0000 7f096de00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09680d3310 con 0x7f09680d3830 2024-03-24T01:27:01.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.128+0000 7f096de00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09680d3030 con 0x7f09680c9a40 2024-03-24T01:27:01.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.128+0000 7f096de00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f096815e240 con 0x7f096815de60 2024-03-24T01:27:01.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.128+0000 7f0966a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09680d3830 0x7f0968154460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:01.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.128+0000 7f0966a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09680d3830 0x7f0968154460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49392/0 (socket says 172.21.15.90:49392) 2024-03-24T01:27:01.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.128+0000 7f0966a00640 1 -- 172.21.15.90:0/4062149971 learned_addr learned my addr 172.21.15.90:0/4062149971 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:01.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.129+0000 7f0966a00640 1 -- 172.21.15.90:0/4062149971 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f096815de60 msgr2=0x7f0968156370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:01.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.129+0000 7f0966a00640 1 --2- 172.21.15.90:0/4062149971 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f096815de60 0x7f0968156370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:01.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.129+0000 7f0966a00640 1 -- 172.21.15.90:0/4062149971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09680c9a40 msgr2=0x7f0968159e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:01.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.129+0000 7f0966a00640 1 --2- 172.21.15.90:0/4062149971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09680c9a40 0x7f0968159e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:01.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.129+0000 7f0966a00640 1 -- 172.21.15.90:0/4062149971 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0968159240 con 0x7f09680d3830 2024-03-24T01:27:01.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.129+0000 7f0966a00640 1 --2- 172.21.15.90:0/4062149971 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09680d3830 0x7f0968154460 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f095800c470 tx=0x7f095800c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:01.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.129+0000 7f095f400640 1 -- 172.21.15.90:0/4062149971 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0958017070 con 0x7f09680d3830 2024-03-24T01:27:01.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.129+0000 7f095f400640 1 -- 172.21.15.90:0/4062149971 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09580035f0 con 0x7f09680d3830 2024-03-24T01:27:01.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.129+0000 7f096de00640 1 -- 172.21.15.90:0/4062149971 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09681569e0 con 0x7f09680d3830 2024-03-24T01:27:01.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.129+0000 7f095f400640 1 -- 172.21.15.90:0/4062149971 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09580039e0 con 0x7f09680d3830 2024-03-24T01:27:01.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.129+0000 7f096de00640 1 -- 172.21.15.90:0/4062149971 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0968156fa0 con 0x7f09680d3830 2024-03-24T01:27:01.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.131+0000 7f096de00640 1 -- 172.21.15.90:0/4062149971 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f096804f400 con 0x7f09680d3830 2024-03-24T01:27:01.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.132+0000 7f095f400640 1 -- 172.21.15.90:0/4062149971 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f095801a070 con 0x7f09680d3830 2024-03-24T01:27:01.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.133+0000 7f095f400640 1 --2- 172.21.15.90:0/4062149971 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f094406bc00 0x7f094406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:01.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.133+0000 7f095f400640 1 -- 172.21.15.90:0/4062149971 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f0958089f00 con 0x7f09680d3830 2024-03-24T01:27:01.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.133+0000 7f095fe00640 1 --2- 172.21.15.90:0/4062149971 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f094406bc00 0x7f094406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:01.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.134+0000 7f095fe00640 1 --2- 172.21.15.90:0/4062149971 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f094406bc00 0x7f094406e0c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f0950036000 tx=0x7f095003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:01.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.137+0000 7f095f400640 1 -- 172.21.15.90:0/4062149971 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0958058880 con 0x7f09680d3830 2024-03-24T01:27:01.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.381+0000 7f096de00640 1 -- 172.21.15.90:0/4062149971 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f09680d1ee0 con 0x7f09680d3830 2024-03-24T01:27:01.383 INFO:tasks.ceph.mon.a.smithi090.stderr:2024-03-24T01:27:01.383+0000 7fc6f2400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T01:27:01.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.387+0000 7f095f400640 1 -- 172.21.15.90:0/4062149971 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v26) v1 ==== 161+0+0 (secure 0 0 0) 0x7f0958058180 con 0x7f09680d3830 2024-03-24T01:27:01.387 INFO:teuthology.orchestra.run.smithi090.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T01:27:01.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.390+0000 7f096de00640 1 -- 172.21.15.90:0/4062149971 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f094406bc00 msgr2=0x7f094406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:01.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.390+0000 7f096de00640 1 --2- 172.21.15.90:0/4062149971 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f094406bc00 0x7f094406e0c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f0950036000 tx=0x7f095003b040 comp rx=0 tx=0).stop 2024-03-24T01:27:01.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.390+0000 7f096de00640 1 -- 172.21.15.90:0/4062149971 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09680d3830 msgr2=0x7f0968154460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:01.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.390+0000 7f096de00640 1 --2- 172.21.15.90:0/4062149971 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09680d3830 0x7f0968154460 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f095800c470 tx=0x7f095800c930 comp rx=0 tx=0).stop 2024-03-24T01:27:01.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.391+0000 7f096de00640 1 -- 172.21.15.90:0/4062149971 shutdown_connections 2024-03-24T01:27:01.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.391+0000 7f096de00640 1 --2- 172.21.15.90:0/4062149971 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f094406bc00 0x7f094406e0c0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:01.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.391+0000 7f096de00640 1 --2- 172.21.15.90:0/4062149971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09680c9a40 0x7f0968159e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:01.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.391+0000 7f096de00640 1 --2- 172.21.15.90:0/4062149971 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f096815de60 0x7f0968156370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:01.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.391+0000 7f096de00640 1 --2- 172.21.15.90:0/4062149971 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09680d3830 0x7f0968154460 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:01.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.391+0000 7f096de00640 1 -- 172.21.15.90:0/4062149971 >> 172.21.15.90:0/4062149971 conn(0x7f0968072c10 msgr2=0x7f09680ca170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:01.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.391+0000 7f096de00640 1 -- 172.21.15.90:0/4062149971 shutdown_connections 2024-03-24T01:27:01.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.392+0000 7f096de00640 1 -- 172.21.15.90:0/4062149971 wait complete. 2024-03-24T01:27:01.419 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-24T01:27:01.426 INFO:tasks.ceph.mds.c.smithi090.stderr: -5129> 2024-03-24T01:26:33.668+0000 7f40cfc00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T01:27:01.429 INFO:tasks.ceph.mds.c.smithi090.stderr: -4580> 2024-03-24T01:26:37.483+0000 7f40cac00640 -1 log_channel(cluster) log [ERR] : bad backtrace on directory inode 0x100000001f7 2024-03-24T01:27:01.431 INFO:tasks.ceph.mds.c.smithi090.stderr: -4152> 2024-03-24T01:26:37.491+0000 7f40cac00640 -1 log_channel(cluster) log [ERR] : bad backtrace on directory inode 0x10000000001 2024-03-24T01:27:01.439 INFO:tasks.ceph.mds.c.smithi090.stderr: -1911> 2024-03-24T01:26:52.943+0000 7f40cac00640 -1 log_channel(cluster) log [ERR] : bad backtrace on directory inode 0x100000001fa 2024-03-24T01:27:01.491 INFO:tasks.ceph.mds.c.smithi090.stdout:starting mds.c at 2024-03-24T01:27:01.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.606+0000 7fd29b200640 1 Processor -- start 2024-03-24T01:27:01.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.606+0000 7fd29b200640 1 -- start start 2024-03-24T01:27:01.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.607+0000 7fd29b200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd29406b7d0 0x7fd2940cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:01.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.607+0000 7fd29b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2940cef80 0x7fd2940ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:01.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.607+0000 7fd29b200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd29406bca0 con 0x7fd294159620 2024-03-24T01:27:01.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.607+0000 7fd29b200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2940b4470 con 0x7fd29406b7d0 2024-03-24T01:27:01.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.608+0000 7fd29b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2940b36f0 con 0x7fd2940cef80 2024-03-24T01:27:01.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.608+0000 7fd293e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd294159620 0x7fd29406b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40680/0 (socket says 172.21.15.90:40680) 2024-03-24T01:27:01.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.608+0000 7fd293e00640 1 -- 172.21.15.90:0/1628389437 learned_addr learned my addr 172.21.15.90:0/1628389437 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:27:01.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.608+0000 7fd293400640 1 --2- 172.21.15.90:0/1628389437 >> v2:172.21.15.90:3300/0 conn(0x7fd29406b7d0 0x7fd2940cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:01.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.608+0000 7fd299200640 1 --2- 172.21.15.90:0/1628389437 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2940cef80 0x7fd2940ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:01.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.608+0000 7fd292a00640 1 -- 172.21.15.90:0/1628389437 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3854674856 0 0) 0x7fd29406bca0 con 0x7fd294159620 2024-03-24T01:27:01.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.608+0000 7fd292a00640 1 -- 172.21.15.90:0/1628389437 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd278003520 con 0x7fd294159620 2024-03-24T01:27:01.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.608+0000 7fd293400640 1 -- 172.21.15.90:0/1628389437 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2940cef80 msgr2=0x7fd2940ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:01.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.609+0000 7fd293400640 1 --2- 172.21.15.90:0/1628389437 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2940cef80 0x7fd2940ae530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:01.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.609+0000 7fd293400640 1 -- 172.21.15.90:0/1628389437 >> v1:172.21.15.90:6789/0 conn(0x7fd294159620 legacy=0x7fd29406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:01.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.609+0000 7fd293400640 1 -- 172.21.15.90:0/1628389437 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2940b3990 con 0x7fd29406b7d0 2024-03-24T01:27:01.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.610+0000 7fd293400640 1 --2- 172.21.15.90:0/1628389437 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd29406b7d0 0x7fd2940cea40 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fd280009d90 tx=0x7fd28002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd3f094815fbe9d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:01.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.610+0000 7fd292a00640 1 -- 172.21.15.90:0/1628389437 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd280039070 con 0x7fd29406b7d0 2024-03-24T01:27:01.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.610+0000 7fd292a00640 1 -- 172.21.15.90:0/1628389437 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28002cbc0 con 0x7fd29406b7d0 2024-03-24T01:27:01.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.610+0000 7fd292a00640 1 -- 172.21.15.90:0/1628389437 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd280034440 con 0x7fd29406b7d0 2024-03-24T01:27:01.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.610+0000 7fd29b200640 1 -- 172.21.15.90:0/1628389437 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd29406b7d0 msgr2=0x7fd2940cea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:01.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.610+0000 7fd29b200640 1 --2- 172.21.15.90:0/1628389437 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd29406b7d0 0x7fd2940cea40 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fd280009d90 tx=0x7fd28002be10 comp rx=0 tx=0).stop 2024-03-24T01:27:01.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.611+0000 7fd29b200640 1 -- 172.21.15.90:0/1628389437 shutdown_connections 2024-03-24T01:27:01.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.611+0000 7fd29b200640 1 --2- 172.21.15.90:0/1628389437 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2940cef80 0x7fd2940ae530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:01.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.611+0000 7fd29b200640 1 --2- 172.21.15.90:0/1628389437 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd29406b7d0 0x7fd2940cea40 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:01.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.611+0000 7fd29b200640 1 -- 172.21.15.90:0/1628389437 >> 172.21.15.90:0/1628389437 conn(0x7fd2940b1770 msgr2=0x7fd2940b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:01.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.611+0000 7fd29b200640 1 -- 172.21.15.90:0/1628389437 shutdown_connections 2024-03-24T01:27:01.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.611+0000 7fd29b200640 1 -- 172.21.15.90:0/1628389437 wait complete. 2024-03-24T01:27:01.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.612+0000 7fd29b200640 1 Processor -- start 2024-03-24T01:27:01.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.613+0000 7fd29b200640 1 -- start start 2024-03-24T01:27:01.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.613+0000 7fd29b200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd29406b7d0 0x7fd2940ca100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:01.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.613+0000 7fd29b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2940cef80 0x7fd294146e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:01.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.613+0000 7fd29b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd294159620 0x7fd294147390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:01.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.613+0000 7fd29b200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd294159a00 con 0x7fd294159620 2024-03-24T01:27:01.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.613+0000 7fd29b200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2940b3f50 con 0x7fd29406b7d0 2024-03-24T01:27:01.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.613+0000 7fd29b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2940cf400 con 0x7fd2940cef80 2024-03-24T01:27:01.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.614+0000 7fd299200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd294159620 0x7fd294147390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:01.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.614+0000 7fd299200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd294159620 0x7fd294147390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49426/0 (socket says 172.21.15.90:49426) 2024-03-24T01:27:01.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.614+0000 7fd299200640 1 -- 172.21.15.90:0/4014278146 learned_addr learned my addr 172.21.15.90:0/4014278146 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:01.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.614+0000 7fd293400640 1 --2- 172.21.15.90:0/4014278146 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2940cef80 0x7fd294146e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:01.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.614+0000 7fd299200640 1 -- 172.21.15.90:0/4014278146 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2940cef80 msgr2=0x7fd294146e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:01.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.614+0000 7fd299200640 1 --2- 172.21.15.90:0/4014278146 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2940cef80 0x7fd294146e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:01.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.614+0000 7fd299200640 1 -- 172.21.15.90:0/4014278146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd29406b7d0 msgr2=0x7fd2940ca100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:01.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.614+0000 7fd293e00640 1 --2- 172.21.15.90:0/4014278146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd29406b7d0 0x7fd2940ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:01.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.614+0000 7fd299200640 1 --2- 172.21.15.90:0/4014278146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd29406b7d0 0x7fd2940ca100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:01.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.614+0000 7fd299200640 1 -- 172.21.15.90:0/4014278146 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2940ca640 con 0x7fd294159620 2024-03-24T01:27:01.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.615+0000 7fd299200640 1 --2- 172.21.15.90:0/4014278146 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd294159620 0x7fd294147390 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fd28c00d800 tx=0x7fd28c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:01.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.615+0000 7fd28be00640 1 -- 172.21.15.90:0/4014278146 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28c00b6c0 con 0x7fd294159620 2024-03-24T01:27:01.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.615+0000 7fd28be00640 1 -- 172.21.15.90:0/4014278146 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28c019070 con 0x7fd294159620 2024-03-24T01:27:01.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.615+0000 7fd28be00640 1 -- 172.21.15.90:0/4014278146 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28c00bac0 con 0x7fd294159620 2024-03-24T01:27:01.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.615+0000 7fd29b200640 1 -- 172.21.15.90:0/4014278146 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd294152380 con 0x7fd294159620 2024-03-24T01:27:01.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.615+0000 7fd29b200640 1 -- 172.21.15.90:0/4014278146 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2940c9be0 con 0x7fd294159620 2024-03-24T01:27:01.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.616+0000 7fd29b200640 1 -- 172.21.15.90:0/4014278146 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd29404f400 con 0x7fd294159620 2024-03-24T01:27:01.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.617+0000 7fd28be00640 1 -- 172.21.15.90:0/4014278146 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd28c00bc60 con 0x7fd294159620 2024-03-24T01:27:01.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.618+0000 7fd28be00640 1 --2- 172.21.15.90:0/4014278146 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd27406bc00 0x7fd27406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:01.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.618+0000 7fd28be00640 1 -- 172.21.15.90:0/4014278146 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6302+0+0 (secure 0 0 0) 0x7fd28c08b080 con 0x7fd294159620 2024-03-24T01:27:01.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.619+0000 7fd293e00640 1 --2- 172.21.15.90:0/4014278146 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd27406bc00 0x7fd27406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:01.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.620+0000 7fd293e00640 1 --2- 172.21.15.90:0/4014278146 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd27406bc00 0x7fd27406e0c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd284000f60 tx=0x7fd284009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:01.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.623+0000 7fd28be00640 1 -- 172.21.15.90:0/4014278146 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd2940ca640 con 0x7fd294159620 2024-03-24T01:27:01.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:01.857+0000 7fd29b200640 1 -- 172.21.15.90:0/4014278146 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd2940b2e90 con 0x7fd294159620 2024-03-24T01:27:02.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.390+0000 7fd28be00640 1 -- 172.21.15.90:0/4014278146 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd28c0599b0 con 0x7fd294159620 2024-03-24T01:27:02.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.394+0000 7fd29b200640 1 -- 172.21.15.90:0/4014278146 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd27406bc00 msgr2=0x7fd27406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:02.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.394+0000 7fd29b200640 1 --2- 172.21.15.90:0/4014278146 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd27406bc00 0x7fd27406e0c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd284000f60 tx=0x7fd284009040 comp rx=0 tx=0).stop 2024-03-24T01:27:02.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.394+0000 7fd29b200640 1 -- 172.21.15.90:0/4014278146 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd294159620 msgr2=0x7fd294147390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:02.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.394+0000 7fd29b200640 1 --2- 172.21.15.90:0/4014278146 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd294159620 0x7fd294147390 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fd28c00d800 tx=0x7fd28c00dcc0 comp rx=0 tx=0).stop 2024-03-24T01:27:02.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.395+0000 7fd29b200640 1 -- 172.21.15.90:0/4014278146 shutdown_connections 2024-03-24T01:27:02.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.395+0000 7fd29b200640 1 --2- 172.21.15.90:0/4014278146 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd27406bc00 0x7fd27406e0c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:02.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.395+0000 7fd29b200640 1 --2- 172.21.15.90:0/4014278146 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd294159620 0x7fd294147390 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:02.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.395+0000 7fd29b200640 1 --2- 172.21.15.90:0/4014278146 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd2940cef80 0x7fd294146e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:02.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.395+0000 7fd29b200640 1 --2- 172.21.15.90:0/4014278146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd29406b7d0 0x7fd2940ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:02.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.395+0000 7fd29b200640 1 -- 172.21.15.90:0/4014278146 >> 172.21.15.90:0/4014278146 conn(0x7fd2940b1770 msgr2=0x7fd2940720e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:02.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.395+0000 7fd29b200640 1 -- 172.21.15.90:0/4014278146 shutdown_connections 2024-03-24T01:27:02.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.395+0000 7fd29b200640 1 -- 172.21.15.90:0/4014278146 wait complete. 2024-03-24T01:27:02.417 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-24T01:27:02.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.670+0000 7f3bc2400640 1 Processor -- start 2024-03-24T01:27:02.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.670+0000 7f3bc2400640 1 -- start start 2024-03-24T01:27:02.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.671+0000 7f3bc2400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bbc15b9f0 0x7f3bbc071990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:02.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.671+0000 7f3bc2400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3bbc071ed0 0x7f3bbc06e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:02.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.671+0000 7f3bc2400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3bbc06e9d0 0x7f3bbc161900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:02.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.671+0000 7f3bc2400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bbc072b00 con 0x7f3bbc071ed0 2024-03-24T01:27:02.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.671+0000 7f3bc2400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bbc15ee50 con 0x7f3bbc06e9d0 2024-03-24T01:27:02.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.671+0000 7f3bc2400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bbc15f2e0 con 0x7f3bbc15b9f0 2024-03-24T01:27:02.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.672+0000 7f3bb3e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3bbc071ed0 0x7f3bbc06e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:02.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.672+0000 7f3bb3e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3bbc071ed0 0x7f3bbc06e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49432/0 (socket says 172.21.15.90:49432) 2024-03-24T01:27:02.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.672+0000 7f3bb3e00640 1 -- 172.21.15.90:0/1630183299 learned_addr learned my addr 172.21.15.90:0/1630183299 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:02.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.672+0000 7f3bbb400640 1 --2- 172.21.15.90:0/1630183299 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bbc15b9f0 0x7f3bbc071990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:02.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.672+0000 7f3bbbe00640 1 --2- 172.21.15.90:0/1630183299 >> v2:172.21.15.120:3300/0 conn(0x7f3bbc06e9d0 0x7f3bbc161900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:02.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.672+0000 7f3bb3e00640 1 -- 172.21.15.90:0/1630183299 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bbc15b9f0 msgr2=0x7f3bbc071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:02.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.672+0000 7f3bb3e00640 1 --2- 172.21.15.90:0/1630183299 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bbc15b9f0 0x7f3bbc071990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:02.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.672+0000 7f3bb3e00640 1 -- 172.21.15.90:0/1630183299 >> v2:172.21.15.120:3300/0 conn(0x7f3bbc06e9d0 msgr2=0x7f3bbc161900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:02.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.672+0000 7f3bb3e00640 1 --2- 172.21.15.90:0/1630183299 >> v2:172.21.15.120:3300/0 conn(0x7f3bbc06e9d0 0x7f3bbc161900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:02.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.672+0000 7f3bb3e00640 1 -- 172.21.15.90:0/1630183299 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bbc06f070 con 0x7f3bbc071ed0 2024-03-24T01:27:02.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.674+0000 7f3bb3e00640 1 --2- 172.21.15.90:0/1630183299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3bbc071ed0 0x7f3bbc06e490 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc06fe80 tx=0x7f3ba402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35da21a83047aebf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:02.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.674+0000 7f3bbaa00640 1 -- 172.21.15.90:0/1630183299 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba4035020 con 0x7f3bbc071ed0 2024-03-24T01:27:02.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.674+0000 7f3bbaa00640 1 -- 172.21.15.90:0/1630183299 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ba402d040 con 0x7f3bbc071ed0 2024-03-24T01:27:02.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.674+0000 7f3bbaa00640 1 -- 172.21.15.90:0/1630183299 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba4031a80 con 0x7f3bbc071ed0 2024-03-24T01:27:02.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.675+0000 7f3bc2400640 1 -- 172.21.15.90:0/1630183299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3bbc071ed0 msgr2=0x7f3bbc06e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:02.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.675+0000 7f3bc2400640 1 --2- 172.21.15.90:0/1630183299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3bbc071ed0 0x7f3bbc06e490 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc06fe80 tx=0x7f3ba402b900 comp rx=0 tx=0).stop 2024-03-24T01:27:02.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.675+0000 7f3bc2400640 1 -- 172.21.15.90:0/1630183299 shutdown_connections 2024-03-24T01:27:02.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.675+0000 7f3bc2400640 1 --2- 172.21.15.90:0/1630183299 >> v2:172.21.15.120:3300/0 conn(0x7f3bbc06e9d0 0x7f3bbc161900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:02.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.675+0000 7f3bc2400640 1 --2- 172.21.15.90:0/1630183299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3bbc071ed0 0x7f3bbc06e490 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:02.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.675+0000 7f3bc2400640 1 --2- 172.21.15.90:0/1630183299 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bbc15b9f0 0x7f3bbc071990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:02.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.675+0000 7f3bc2400640 1 -- 172.21.15.90:0/1630183299 >> 172.21.15.90:0/1630183299 conn(0x7f3bbc0cf7e0 msgr2=0x7f3bbc0cfbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:02.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.675+0000 7f3bc2400640 1 -- 172.21.15.90:0/1630183299 shutdown_connections 2024-03-24T01:27:02.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.675+0000 7f3bc2400640 1 -- 172.21.15.90:0/1630183299 wait complete. 2024-03-24T01:27:02.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.677+0000 7f3bc2400640 1 Processor -- start 2024-03-24T01:27:02.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.677+0000 7f3bc2400640 1 -- start start 2024-03-24T01:27:02.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.678+0000 7f3bc2400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3bbc06e9d0 0x7f3bbc1f2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:02.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.678+0000 7f3bc2400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bbc071ed0 0x7f3bbc157a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:02.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.678+0000 7f3bc2400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3bbc15b9f0 0x7f3bbc157fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:02.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.678+0000 7f3bc2400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bbc15eae0 con 0x7f3bbc15b9f0 2024-03-24T01:27:02.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.678+0000 7f3bc2400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bbc15e800 con 0x7f3bbc06e9d0 2024-03-24T01:27:02.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.678+0000 7f3bc2400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bbc161e40 con 0x7f3bbc071ed0 2024-03-24T01:27:02.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.678+0000 7f3bbbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3bbc15b9f0 0x7f3bbc157fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:02.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.678+0000 7f3bbbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3bbc15b9f0 0x7f3bbc157fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49438/0 (socket says 172.21.15.90:49438) 2024-03-24T01:27:02.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.678+0000 7f3bbbe00640 1 -- 172.21.15.90:0/642919408 learned_addr learned my addr 172.21.15.90:0/642919408 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:02.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.678+0000 7f3bb3e00640 1 --2- 172.21.15.90:0/642919408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bbc071ed0 0x7f3bbc157a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:02.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.678+0000 7f3bbb400640 1 --2- 172.21.15.90:0/642919408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3bbc06e9d0 0x7f3bbc1f2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:02.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.678+0000 7f3bbbe00640 1 -- 172.21.15.90:0/642919408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bbc071ed0 msgr2=0x7f3bbc157a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:02.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.678+0000 7f3bbbe00640 1 --2- 172.21.15.90:0/642919408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bbc071ed0 0x7f3bbc157a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:02.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.678+0000 7f3bbbe00640 1 -- 172.21.15.90:0/642919408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3bbc06e9d0 msgr2=0x7f3bbc1f2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:02.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.678+0000 7f3bbbe00640 1 --2- 172.21.15.90:0/642919408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3bbc06e9d0 0x7f3bbc1f2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:02.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.678+0000 7f3bbbe00640 1 -- 172.21.15.90:0/642919408 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bbc146ed0 con 0x7f3bbc15b9f0 2024-03-24T01:27:02.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.678+0000 7f3bbb400640 1 --2- 172.21.15.90:0/642919408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3bbc06e9d0 0x7f3bbc1f2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:27:02.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.679+0000 7f3bbbe00640 1 --2- 172.21.15.90:0/642919408 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3bbc15b9f0 0x7f3bbc157fa0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4007e60 tx=0x7f3bb400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:02.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.679+0000 7f3bb3400640 1 -- 172.21.15.90:0/642919408 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb4014620 con 0x7f3bbc15b9f0 2024-03-24T01:27:02.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.679+0000 7f3bb3400640 1 -- 172.21.15.90:0/642919408 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bb4019070 con 0x7f3bbc15b9f0 2024-03-24T01:27:02.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.679+0000 7f3bb3400640 1 -- 172.21.15.90:0/642919408 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb4014a20 con 0x7f3bbc15b9f0 2024-03-24T01:27:02.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.679+0000 7f3bc2400640 1 -- 172.21.15.90:0/642919408 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bbc147170 con 0x7f3bbc15b9f0 2024-03-24T01:27:02.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.679+0000 7f3bc2400640 1 -- 172.21.15.90:0/642919408 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bbc147730 con 0x7f3bbc15b9f0 2024-03-24T01:27:02.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.680+0000 7f3bc2400640 1 -- 172.21.15.90:0/642919408 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bbc04f400 con 0x7f3bbc15b9f0 2024-03-24T01:27:02.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.681+0000 7f3bb3400640 1 -- 172.21.15.90:0/642919408 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3bb4014bc0 con 0x7f3bbc15b9f0 2024-03-24T01:27:02.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.682+0000 7f3bb3400640 1 --2- 172.21.15.90:0/642919408 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3b9806bc00 0x7f3b9806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:02.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.682+0000 7f3bb3400640 1 -- 172.21.15.90:0/642919408 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6302+0+0 (secure 0 0 0) 0x7f3bb408b600 con 0x7f3bbc15b9f0 2024-03-24T01:27:02.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.682+0000 7f3bbb400640 1 --2- 172.21.15.90:0/642919408 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3b9806bc00 0x7f3b9806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:02.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.683+0000 7f3bbb400640 1 --2- 172.21.15.90:0/642919408 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3b9806bc00 0x7f3b9806e0c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f3bac006fd0 tx=0x7f3bac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:02.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.686+0000 7f3bb3400640 1 -- 172.21.15.90:0/642919408 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3bb4059e00 con 0x7f3bbc15b9f0 2024-03-24T01:27:02.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:02.884+0000 7f3bc2400640 1 -- 172.21.15.90:0/642919408 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3bbc072590 con 0x7f3bbc15b9f0 2024-03-24T01:27:03.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.401+0000 7f3bb3400640 1 -- 172.21.15.90:0/642919408 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v44) v1 ==== 181+0+0 (secure 0 0 0) 0x7f3bb4059700 con 0x7f3bbc15b9f0 2024-03-24T01:27:03.401 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_metadata' removed 2024-03-24T01:27:03.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.404+0000 7f3bc2400640 1 -- 172.21.15.90:0/642919408 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3b9806bc00 msgr2=0x7f3b9806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:03.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.404+0000 7f3bc2400640 1 --2- 172.21.15.90:0/642919408 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3b9806bc00 0x7f3b9806e0c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f3bac006fd0 tx=0x7f3bac008040 comp rx=0 tx=0).stop 2024-03-24T01:27:03.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.404+0000 7f3bc2400640 1 -- 172.21.15.90:0/642919408 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3bbc15b9f0 msgr2=0x7f3bbc157fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:03.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.404+0000 7f3bc2400640 1 --2- 172.21.15.90:0/642919408 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3bbc15b9f0 0x7f3bbc157fa0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4007e60 tx=0x7f3bb400e4b0 comp rx=0 tx=0).stop 2024-03-24T01:27:03.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.404+0000 7f3bc2400640 1 -- 172.21.15.90:0/642919408 shutdown_connections 2024-03-24T01:27:03.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.404+0000 7f3bc2400640 1 --2- 172.21.15.90:0/642919408 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3b9806bc00 0x7f3b9806e0c0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:03.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.404+0000 7f3bc2400640 1 --2- 172.21.15.90:0/642919408 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3bbc15b9f0 0x7f3bbc157fa0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:03.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.404+0000 7f3bc2400640 1 --2- 172.21.15.90:0/642919408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3bbc071ed0 0x7f3bbc157a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:03.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.404+0000 7f3bc2400640 1 --2- 172.21.15.90:0/642919408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3bbc06e9d0 0x7f3bbc1f2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:03.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.405+0000 7f3bc2400640 1 -- 172.21.15.90:0/642919408 >> 172.21.15.90:0/642919408 conn(0x7f3bbc0cf7e0 msgr2=0x7f3bbc15d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:03.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.405+0000 7f3bc2400640 1 -- 172.21.15.90:0/642919408 shutdown_connections 2024-03-24T01:27:03.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.405+0000 7f3bc2400640 1 -- 172.21.15.90:0/642919408 wait complete. 2024-03-24T01:27:03.428 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-24T01:27:03.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.623+0000 7fe79fe00640 1 Processor -- start 2024-03-24T01:27:03.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.624+0000 7fe79fe00640 1 -- start start 2024-03-24T01:27:03.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.625+0000 7fe79fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe7a0144680 0x7fe7a006d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:03.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.625+0000 7fe79fe00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe7a0145190 0x7fe7a0071790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:03.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.625+0000 7fe79fe00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7a006e0f0 con 0x7fe7a0071cd0 2024-03-24T01:27:03.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.625+0000 7fe79fe00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7a0072140 con 0x7fe7a0144680 2024-03-24T01:27:03.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.625+0000 7fe79fe00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7a0144a60 con 0x7fe7a0145190 2024-03-24T01:27:03.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.626+0000 7fe79ea00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe7a0144680 0x7fe7a006d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:03.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.626+0000 7fe79ea00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe7a0144680 0x7fe7a006d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49446/0 (socket says 172.21.15.90:49446) 2024-03-24T01:27:03.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.626+0000 7fe79ea00640 1 -- 172.21.15.90:0/1920749952 learned_addr learned my addr 172.21.15.90:0/1920749952 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:03.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.626+0000 7fe79ea00640 1 -- 172.21.15.90:0/1920749952 >> v2:172.21.15.120:3300/0 conn(0x7fe7a0145190 msgr2=0x7fe7a0071790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:03.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.626+0000 7fe796000640 1 --2- 172.21.15.90:0/1920749952 >> v2:172.21.15.120:3300/0 conn(0x7fe7a0145190 0x7fe7a0071790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:03.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.626+0000 7fe79ea00640 1 --2- 172.21.15.90:0/1920749952 >> v2:172.21.15.120:3300/0 conn(0x7fe7a0145190 0x7fe7a0071790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:03.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.626+0000 7fe79ea00640 1 -- 172.21.15.90:0/1920749952 >> v1:172.21.15.120:6789/0 conn(0x7fe7a0071cd0 legacy=0x7fe7a015f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:03.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.626+0000 7fe79ea00640 1 -- 172.21.15.90:0/1920749952 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7a006e3d0 con 0x7fe7a0144680 2024-03-24T01:27:03.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.626+0000 7fe796000640 1 --2- 172.21.15.90:0/1920749952 >> v2:172.21.15.120:3300/0 conn(0x7fe7a0145190 0x7fe7a0071790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:27:03.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.628+0000 7fe79ea00640 1 --2- 172.21.15.90:0/1920749952 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7a0144680 0x7fe7a006d7f0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fe7a00cf1b0 tx=0x7fe79002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=737d12bc62bf9eb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:03.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.628+0000 7fe79e000640 1 -- 172.21.15.90:0/1920749952 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe790035020 con 0x7fe7a0144680 2024-03-24T01:27:03.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.628+0000 7fe79e000640 1 -- 172.21.15.90:0/1920749952 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe790031870 con 0x7fe7a0144680 2024-03-24T01:27:03.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.628+0000 7fe79e000640 1 -- 172.21.15.90:0/1920749952 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe790031c40 con 0x7fe7a0144680 2024-03-24T01:27:03.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.629+0000 7fe79fe00640 1 -- 172.21.15.90:0/1920749952 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7a0144680 msgr2=0x7fe7a006d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:03.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.629+0000 7fe79fe00640 1 --2- 172.21.15.90:0/1920749952 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7a0144680 0x7fe7a006d7f0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fe7a00cf1b0 tx=0x7fe79002b910 comp rx=0 tx=0).stop 2024-03-24T01:27:03.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.629+0000 7fe79fe00640 1 -- 172.21.15.90:0/1920749952 shutdown_connections 2024-03-24T01:27:03.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.629+0000 7fe79fe00640 1 --2- 172.21.15.90:0/1920749952 >> v2:172.21.15.120:3300/0 conn(0x7fe7a0145190 0x7fe7a0071790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:03.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.629+0000 7fe79fe00640 1 --2- 172.21.15.90:0/1920749952 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7a0144680 0x7fe7a006d7f0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:03.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.629+0000 7fe79fe00640 1 -- 172.21.15.90:0/1920749952 >> 172.21.15.90:0/1920749952 conn(0x7fe7a006b0c0 msgr2=0x7fe7a006b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:03.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.629+0000 7fe79fe00640 1 -- 172.21.15.90:0/1920749952 shutdown_connections 2024-03-24T01:27:03.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.629+0000 7fe79fe00640 1 -- 172.21.15.90:0/1920749952 wait complete. 2024-03-24T01:27:03.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.631+0000 7fe79fe00640 1 Processor -- start 2024-03-24T01:27:03.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.631+0000 7fe79fe00640 1 -- start start 2024-03-24T01:27:03.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.632+0000 7fe79fe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7a0071cd0 0x7fe7a015b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:03.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.632+0000 7fe79fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7a0144680 0x7fe7a0131980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:03.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.632+0000 7fe79fe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7a0145190 0x7fe7a01f7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:03.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.632+0000 7fe79fe00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a00cc470 con 0x7fe7a0144680 2024-03-24T01:27:03.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.632+0000 7fe79fe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a00cc190 con 0x7fe7a0145190 2024-03-24T01:27:03.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.632+0000 7fe79fe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a006dd30 con 0x7fe7a0071cd0 2024-03-24T01:27:03.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.632+0000 7fe796000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7a0144680 0x7fe7a0131980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:03.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.632+0000 7fe796000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7a0144680 0x7fe7a0131980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49460/0 (socket says 172.21.15.90:49460) 2024-03-24T01:27:03.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.632+0000 7fe796000640 1 -- 172.21.15.90:0/54537640 learned_addr learned my addr 172.21.15.90:0/54537640 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:03.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.632+0000 7fe79ea00640 1 --2- 172.21.15.90:0/54537640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7a0071cd0 0x7fe7a015b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:03.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.632+0000 7fe79f400640 1 --2- 172.21.15.90:0/54537640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7a0145190 0x7fe7a01f7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:03.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.632+0000 7fe796000640 1 -- 172.21.15.90:0/54537640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7a0071cd0 msgr2=0x7fe7a015b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:03.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.632+0000 7fe796000640 1 --2- 172.21.15.90:0/54537640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7a0071cd0 0x7fe7a015b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:03.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.633+0000 7fe796000640 1 -- 172.21.15.90:0/54537640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7a0145190 msgr2=0x7fe7a01f7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:03.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.633+0000 7fe796000640 1 --2- 172.21.15.90:0/54537640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7a0145190 0x7fe7a01f7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:03.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.633+0000 7fe796000640 1 -- 172.21.15.90:0/54537640 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7a01321d0 con 0x7fe7a0144680 2024-03-24T01:27:03.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.633+0000 7fe796000640 1 --2- 172.21.15.90:0/54537640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7a0144680 0x7fe7a0131980 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fe78800c7f0 tx=0x7fe78800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:03.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.633+0000 7fe797400640 1 -- 172.21.15.90:0/54537640 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe788015070 con 0x7fe7a0144680 2024-03-24T01:27:03.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.633+0000 7fe797400640 1 -- 172.21.15.90:0/54537640 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe788007820 con 0x7fe7a0144680 2024-03-24T01:27:03.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.633+0000 7fe79fe00640 1 -- 172.21.15.90:0/54537640 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7a01324b0 con 0x7fe7a0144680 2024-03-24T01:27:03.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.633+0000 7fe797400640 1 -- 172.21.15.90:0/54537640 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe788007bd0 con 0x7fe7a0144680 2024-03-24T01:27:03.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.633+0000 7fe79fe00640 1 -- 172.21.15.90:0/54537640 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7a015b590 con 0x7fe7a0144680 2024-03-24T01:27:03.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.635+0000 7fe79fe00640 1 -- 172.21.15.90:0/54537640 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7a004f400 con 0x7fe7a0144680 2024-03-24T01:27:03.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.635+0000 7fe797400640 1 -- 172.21.15.90:0/54537640 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe788018070 con 0x7fe7a0144680 2024-03-24T01:27:03.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.637+0000 7fe797400640 1 --2- 172.21.15.90:0/54537640 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe77c06bbd0 0x7fe77c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:03.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.637+0000 7fe797400640 1 -- 172.21.15.90:0/54537640 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5864+0+0 (secure 0 0 0) 0x7fe78808ac00 con 0x7fe7a0144680 2024-03-24T01:27:03.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.637+0000 7fe79ea00640 1 --2- 172.21.15.90:0/54537640 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe77c06bbd0 0x7fe77c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:03.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.638+0000 7fe79ea00640 1 --2- 172.21.15.90:0/54537640 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe77c06bbd0 0x7fe77c06e090 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe790036000 tx=0x7fe79003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:03.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.642+0000 7fe797400640 1 -- 172.21.15.90:0/54537640 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe788091080 con 0x7fe7a0144680 2024-03-24T01:27:03.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:03.840+0000 7fe79fe00640 1 -- 172.21.15.90:0/54537640 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe7a0144f20 con 0x7fe7a0144680 2024-03-24T01:27:04.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.413+0000 7fe797400640 1 -- 172.21.15.90:0/54537640 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v45) v1 ==== 169+0+0 (secure 0 0 0) 0x7fe7880596e0 con 0x7fe7a0144680 2024-03-24T01:27:04.414 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_data' removed 2024-03-24T01:27:04.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.417+0000 7fe79fe00640 1 -- 172.21.15.90:0/54537640 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe77c06bbd0 msgr2=0x7fe77c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:04.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.417+0000 7fe79fe00640 1 --2- 172.21.15.90:0/54537640 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe77c06bbd0 0x7fe77c06e090 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe790036000 tx=0x7fe79003c040 comp rx=0 tx=0).stop 2024-03-24T01:27:04.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.417+0000 7fe79fe00640 1 -- 172.21.15.90:0/54537640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7a0144680 msgr2=0x7fe7a0131980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:04.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.417+0000 7fe79fe00640 1 --2- 172.21.15.90:0/54537640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7a0144680 0x7fe7a0131980 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fe78800c7f0 tx=0x7fe78800ccb0 comp rx=0 tx=0).stop 2024-03-24T01:27:04.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.417+0000 7fe79fe00640 1 -- 172.21.15.90:0/54537640 shutdown_connections 2024-03-24T01:27:04.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.417+0000 7fe79fe00640 1 --2- 172.21.15.90:0/54537640 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe77c06bbd0 0x7fe77c06e090 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:04.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.417+0000 7fe79fe00640 1 --2- 172.21.15.90:0/54537640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7a0144680 0x7fe7a0131980 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:04.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.417+0000 7fe79fe00640 1 --2- 172.21.15.90:0/54537640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7a0145190 0x7fe7a01f7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:04.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.417+0000 7fe79fe00640 1 --2- 172.21.15.90:0/54537640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7a0071cd0 0x7fe7a015b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:04.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.417+0000 7fe79fe00640 1 -- 172.21.15.90:0/54537640 >> 172.21.15.90:0/54537640 conn(0x7fe7a006b0c0 msgr2=0x7fe7a00b05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:04.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.417+0000 7fe79fe00640 1 -- 172.21.15.90:0/54537640 shutdown_connections 2024-03-24T01:27:04.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.418+0000 7fe79fe00640 1 -- 172.21.15.90:0/54537640 wait complete. 2024-03-24T01:27:04.444 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-24T01:27:04.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.646+0000 7fc128000640 1 Processor -- start 2024-03-24T01:27:04.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.646+0000 7fc128000640 1 -- start start 2024-03-24T01:27:04.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.648+0000 7fc128000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc1200aeeb0 0x7fc1200b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:04.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.648+0000 7fc128000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc12015bc50 0x7fc12006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:04.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.648+0000 7fc128000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc12006dac0 con 0x7fc12006b600 2024-03-24T01:27:04.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.648+0000 7fc128000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc12006ba70 con 0x7fc12015bc50 2024-03-24T01:27:04.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.648+0000 7fc128000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1200b4da0 con 0x7fc1200aeeb0 2024-03-24T01:27:04.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.648+0000 7fc124c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc12015bc50 0x7fc12006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:04.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.648+0000 7fc126000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc12006b600 0x7fc12015e590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40694/0 (socket says 172.21.15.90:40694) 2024-03-24T01:27:04.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.648+0000 7fc124c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc12015bc50 0x7fc12006b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49468/0 (socket says 172.21.15.90:49468) 2024-03-24T01:27:04.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.648+0000 7fc126000640 1 -- 172.21.15.90:0/3356341325 learned_addr learned my addr 172.21.15.90:0/3356341325 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:27:04.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.649+0000 7fc125600640 1 --2- 172.21.15.90:0/3356341325 >> v2:172.21.15.120:3300/0 conn(0x7fc1200aeeb0 0x7fc1200b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:04.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.649+0000 7fc124c00640 1 -- 172.21.15.90:0/3356341325 >> v2:172.21.15.120:3300/0 conn(0x7fc1200aeeb0 msgr2=0x7fc1200b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:04.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.649+0000 7fc124c00640 1 --2- 172.21.15.90:0/3356341325 >> v2:172.21.15.120:3300/0 conn(0x7fc1200aeeb0 0x7fc1200b44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:04.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.649+0000 7fc124c00640 1 -- 172.21.15.90:0/3356341325 >> v1:172.21.15.90:6789/0 conn(0x7fc12006b600 legacy=0x7fc12015e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:04.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.649+0000 7fc124c00640 1 -- 172.21.15.90:0/3356341325 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12006dda0 con 0x7fc12015bc50 2024-03-24T01:27:04.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.650+0000 7fc124c00640 1 --2- 172.21.15.90:0/3356341325 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc12015bc50 0x7fc12006b0c0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fc1200b4f80 tx=0x7fc10c02b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96a6b4a95aa55b92 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:04.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.650+0000 7fc11fe00640 1 -- 172.21.15.90:0/3356341325 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc10c038070 con 0x7fc12015bc50 2024-03-24T01:27:04.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.650+0000 7fc11fe00640 1 -- 172.21.15.90:0/3356341325 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc10c031710 con 0x7fc12015bc50 2024-03-24T01:27:04.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.650+0000 7fc11fe00640 1 -- 172.21.15.90:0/3356341325 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc10c031ae0 con 0x7fc12015bc50 2024-03-24T01:27:04.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.651+0000 7fc128000640 1 -- 172.21.15.90:0/3356341325 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc12015bc50 msgr2=0x7fc12006b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:04.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.651+0000 7fc128000640 1 --2- 172.21.15.90:0/3356341325 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc12015bc50 0x7fc12006b0c0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fc1200b4f80 tx=0x7fc10c02b7b0 comp rx=0 tx=0).stop 2024-03-24T01:27:04.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.651+0000 7fc128000640 1 -- 172.21.15.90:0/3356341325 shutdown_connections 2024-03-24T01:27:04.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.651+0000 7fc128000640 1 --2- 172.21.15.90:0/3356341325 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc12015bc50 0x7fc12006b0c0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:04.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.651+0000 7fc128000640 1 --2- 172.21.15.90:0/3356341325 >> v2:172.21.15.120:3300/0 conn(0x7fc1200aeeb0 0x7fc1200b44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:04.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.651+0000 7fc128000640 1 -- 172.21.15.90:0/3356341325 >> 172.21.15.90:0/3356341325 conn(0x7fc1200b1770 msgr2=0x7fc1200b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:04.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.651+0000 7fc128000640 1 -- 172.21.15.90:0/3356341325 shutdown_connections 2024-03-24T01:27:04.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.651+0000 7fc128000640 1 -- 172.21.15.90:0/3356341325 wait complete. 2024-03-24T01:27:04.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.653+0000 7fc128000640 1 Processor -- start 2024-03-24T01:27:04.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.653+0000 7fc128000640 1 -- start start 2024-03-24T01:27:04.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.654+0000 7fc128000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc12006b600 0x7fc1201f6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:04.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.654+0000 7fc128000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1200aeeb0 0x7fc1201e3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:04.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.654+0000 7fc128000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc12015bc50 0x7fc1201e5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:04.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.654+0000 7fc128000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1200af290 con 0x7fc12006b600 2024-03-24T01:27:04.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.654+0000 7fc128000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12006e360 con 0x7fc12015bc50 2024-03-24T01:27:04.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.654+0000 7fc128000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1200b4a00 con 0x7fc1200aeeb0 2024-03-24T01:27:04.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.654+0000 7fc125600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc12006b600 0x7fc1201f6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:04.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.654+0000 7fc125600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc12006b600 0x7fc1201f6480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49482/0 (socket says 172.21.15.90:49482) 2024-03-24T01:27:04.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.654+0000 7fc125600640 1 -- 172.21.15.90:0/538392283 learned_addr learned my addr 172.21.15.90:0/538392283 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:04.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.654+0000 7fc125600640 1 -- 172.21.15.90:0/538392283 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1200aeeb0 msgr2=0x7fc1201e3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:04.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.654+0000 7fc125600640 1 --2- 172.21.15.90:0/538392283 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1200aeeb0 0x7fc1201e3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:04.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.654+0000 7fc125600640 1 -- 172.21.15.90:0/538392283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc12015bc50 msgr2=0x7fc1201e5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:04.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.654+0000 7fc125600640 1 --2- 172.21.15.90:0/538392283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc12015bc50 0x7fc1201e5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:04.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.655+0000 7fc125600640 1 -- 172.21.15.90:0/538392283 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1201e4600 con 0x7fc12006b600 2024-03-24T01:27:04.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.655+0000 7fc125600640 1 --2- 172.21.15.90:0/538392283 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc12006b600 0x7fc1201f6480 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fc11400d830 tx=0x7fc11400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:04.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.655+0000 7fc11d600640 1 -- 172.21.15.90:0/538392283 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11400b660 con 0x7fc12006b600 2024-03-24T01:27:04.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.655+0000 7fc128000640 1 -- 172.21.15.90:0/538392283 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1201e48e0 con 0x7fc12006b600 2024-03-24T01:27:04.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.655+0000 7fc11d600640 1 -- 172.21.15.90:0/538392283 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc114019070 con 0x7fc12006b600 2024-03-24T01:27:04.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.655+0000 7fc11d600640 1 -- 172.21.15.90:0/538392283 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11400ba20 con 0x7fc12006b600 2024-03-24T01:27:04.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.655+0000 7fc128000640 1 -- 172.21.15.90:0/538392283 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1201e4ea0 con 0x7fc12006b600 2024-03-24T01:27:04.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.656+0000 7fc128000640 1 -- 172.21.15.90:0/538392283 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc12004f400 con 0x7fc12006b600 2024-03-24T01:27:04.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.658+0000 7fc11d600640 1 -- 172.21.15.90:0/538392283 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc11401e460 con 0x7fc12006b600 2024-03-24T01:27:04.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.658+0000 7fc11d600640 1 --2- 172.21.15.90:0/538392283 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc10006bc00 0x7fc10006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:04.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.658+0000 7fc11d600640 1 -- 172.21.15.90:0/538392283 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fc11408afe0 con 0x7fc12006b600 2024-03-24T01:27:04.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.659+0000 7fc124c00640 1 --2- 172.21.15.90:0/538392283 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc10006bc00 0x7fc10006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:04.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.660+0000 7fc124c00640 1 --2- 172.21.15.90:0/538392283 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc10006bc00 0x7fc10006e0c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc10c035000 tx=0x7fc10c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:04.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.663+0000 7fc11d600640 1 -- 172.21.15.90:0/538392283 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc114059b60 con 0x7fc12006b600 2024-03-24T01:27:04.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.862+0000 7fc128000640 1 -- 172.21.15.90:0/538392283 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fc12006c970 con 0x7fc12006b600 2024-03-24T01:27:04.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.870+0000 7fc11d600640 1 -- 172.21.15.90:0/538392283 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc114059460 con 0x7fc12006b600 2024-03-24T01:27:04.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.873+0000 7fc128000640 1 -- 172.21.15.90:0/538392283 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc10006bc00 msgr2=0x7fc10006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:04.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.873+0000 7fc128000640 1 --2- 172.21.15.90:0/538392283 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc10006bc00 0x7fc10006e0c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc10c035000 tx=0x7fc10c030000 comp rx=0 tx=0).stop 2024-03-24T01:27:04.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.873+0000 7fc128000640 1 -- 172.21.15.90:0/538392283 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc12006b600 msgr2=0x7fc1201f6480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:04.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.873+0000 7fc128000640 1 --2- 172.21.15.90:0/538392283 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc12006b600 0x7fc1201f6480 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fc11400d830 tx=0x7fc11400dcf0 comp rx=0 tx=0).stop 2024-03-24T01:27:04.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.873+0000 7fc128000640 1 -- 172.21.15.90:0/538392283 shutdown_connections 2024-03-24T01:27:04.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.873+0000 7fc128000640 1 --2- 172.21.15.90:0/538392283 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc10006bc00 0x7fc10006e0c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:04.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.873+0000 7fc128000640 1 --2- 172.21.15.90:0/538392283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc12015bc50 0x7fc1201e5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:04.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.873+0000 7fc128000640 1 --2- 172.21.15.90:0/538392283 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1200aeeb0 0x7fc1201e3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:04.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.873+0000 7fc128000640 1 --2- 172.21.15.90:0/538392283 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc12006b600 0x7fc1201f6480 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:04.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.873+0000 7fc128000640 1 -- 172.21.15.90:0/538392283 >> 172.21.15.90:0/538392283 conn(0x7fc1200b1770 msgr2=0x7fc120071990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:04.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.873+0000 7fc128000640 1 -- 172.21.15.90:0/538392283 shutdown_connections 2024-03-24T01:27:04.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:04.873+0000 7fc128000640 1 -- 172.21.15.90:0/538392283 wait complete. 2024-03-24T01:27:04.892 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_journal_repair.TestJournalRepair.test_inject_to_empty' 2024-03-24T01:27:05.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.096+0000 7f23a2a00640 1 Processor -- start 2024-03-24T01:27:05.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.097+0000 7f23a2a00640 1 -- start start 2024-03-24T01:27:05.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.098+0000 7f23a2a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f239c0d1090 0x7f239c0ce430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:05.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.098+0000 7f23a2a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f239c0cea10 con 0x7f239c15dcc0 2024-03-24T01:27:05.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.098+0000 7f23a2a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f239c0cecf0 con 0x7f239c15e0a0 2024-03-24T01:27:05.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.098+0000 7f23a2a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f239c0d1e60 con 0x7f239c0d1090 2024-03-24T01:27:05.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.098+0000 7f239be00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f239c15dcc0 0x7f239c162450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40706/0 (socket says 172.21.15.90:40706) 2024-03-24T01:27:05.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.098+0000 7f239be00640 1 -- 172.21.15.90:0/2849172074 learned_addr learned my addr 172.21.15.90:0/2849172074 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:27:05.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.099+0000 7f23a0a00640 1 --2- 172.21.15.90:0/2849172074 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f239c0d1090 0x7f239c0ce430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:05.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.099+0000 7f239aa00640 1 -- 172.21.15.90:0/2849172074 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3934682260 0 0) 0x7f239c0cea10 con 0x7f239c15dcc0 2024-03-24T01:27:05.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.099+0000 7f239aa00640 1 -- 172.21.15.90:0/2849172074 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2380003520 con 0x7f239c15dcc0 2024-03-24T01:27:05.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.099+0000 7f239aa00640 1 -- 172.21.15.90:0/2849172074 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3962184213 0 0) 0x7f2380003520 con 0x7f239c15dcc0 2024-03-24T01:27:05.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.099+0000 7f239aa00640 1 -- 172.21.15.90:0/2849172074 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f239c0d1090 msgr2=0x7f239c0ce430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:05.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.099+0000 7f239aa00640 1 --2- 172.21.15.90:0/2849172074 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f239c0d1090 0x7f239c0ce430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:05.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.099+0000 7f239aa00640 1 -- 172.21.15.90:0/2849172074 >> v1:172.21.15.120:6789/0 conn(0x7f239c15e0a0 legacy=0x7f239c0d0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:05.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.099+0000 7f239aa00640 1 -- 172.21.15.90:0/2849172074 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f239c0cefd0 con 0x7f239c15dcc0 2024-03-24T01:27:05.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.099+0000 7f239aa00640 1 -- 172.21.15.90:0/2849172074 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f238c001670 con 0x7f239c15dcc0 2024-03-24T01:27:05.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.099+0000 7f239aa00640 1 -- 172.21.15.90:0/2849172074 >> v1:172.21.15.90:6789/0 conn(0x7f239c15dcc0 legacy=0x7f239c162450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:05.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.099+0000 7f239aa00640 1 --2- 172.21.15.90:0/2849172074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23800039e0 0x7f2380003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:05.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.099+0000 7f239aa00640 1 --2- 172.21.15.90:0/2849172074 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f239c0d1090 0x7f2380004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:05.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.100+0000 7f239aa00640 1 --2- 172.21.15.90:0/2849172074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2380004d40 0x7f23800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:05.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.100+0000 7f239aa00640 1 -- 172.21.15.90:0/2849172074 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239c0d1e60 con 0x7f23800039e0 2024-03-24T01:27:05.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.100+0000 7f239aa00640 1 -- 172.21.15.90:0/2849172074 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238000bb90 con 0x7f2380004d40 2024-03-24T01:27:05.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.100+0000 7f239aa00640 1 -- 172.21.15.90:0/2849172074 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f238000bd90 con 0x7f239c0d1090 2024-03-24T01:27:05.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.100+0000 7f239be00640 1 --2- 172.21.15.90:0/2849172074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23800039e0 0x7f2380003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:05.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.100+0000 7f239b400640 1 --2- 172.21.15.90:0/2849172074 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f239c0d1090 0x7f2380004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:05.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.100+0000 7f239be00640 1 -- 172.21.15.90:0/2849172074 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f239c0d1090 msgr2=0x7f2380004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:05.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.100+0000 7f239be00640 1 --2- 172.21.15.90:0/2849172074 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f239c0d1090 0x7f2380004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:05.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.100+0000 7f239be00640 1 -- 172.21.15.90:0/2849172074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2380004d40 msgr2=0x7f23800051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:05.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.100+0000 7f239be00640 1 --2- 172.21.15.90:0/2849172074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2380004d40 0x7f23800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:05.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.100+0000 7f239be00640 1 -- 172.21.15.90:0/2849172074 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2380003520 con 0x7f23800039e0 2024-03-24T01:27:05.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.100+0000 7f239b400640 1 --2- 172.21.15.90:0/2849172074 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f239c0d1090 0x7f2380004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:27:05.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.101+0000 7f239be00640 1 --2- 172.21.15.90:0/2849172074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23800039e0 0x7f2380003e20 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f238c00c0e0 tx=0x7f2380005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c5386f6ee35c71b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:05.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.101+0000 7f239aa00640 1 -- 172.21.15.90:0/2849172074 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238c007c40 con 0x7f23800039e0 2024-03-24T01:27:05.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.101+0000 7f239aa00640 1 -- 172.21.15.90:0/2849172074 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f238c030070 con 0x7f23800039e0 2024-03-24T01:27:05.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.101+0000 7f239aa00640 1 -- 172.21.15.90:0/2849172074 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238c005d30 con 0x7f23800039e0 2024-03-24T01:27:05.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.102+0000 7f23a2a00640 1 -- 172.21.15.90:0/2849172074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23800039e0 msgr2=0x7f2380003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:05.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.102+0000 7f23a2a00640 1 --2- 172.21.15.90:0/2849172074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23800039e0 0x7f2380003e20 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f238c00c0e0 tx=0x7f2380005830 comp rx=0 tx=0).stop 2024-03-24T01:27:05.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.102+0000 7f23a0a00640 1 -- 172.21.15.90:0/2849172074 reap_dead start 2024-03-24T01:27:05.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.102+0000 7f23a2a00640 1 -- 172.21.15.90:0/2849172074 shutdown_connections 2024-03-24T01:27:05.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.102+0000 7f23a2a00640 1 -- 172.21.15.90:0/2849172074 >> 172.21.15.90:0/2849172074 conn(0x7f239c06cac0 msgr2=0x7f239c06ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:05.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.102+0000 7f23a2a00640 1 -- 172.21.15.90:0/2849172074 shutdown_connections 2024-03-24T01:27:05.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.102+0000 7f23a2a00640 1 -- 172.21.15.90:0/2849172074 wait complete. 2024-03-24T01:27:05.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.104+0000 7f23a2a00640 1 Processor -- start 2024-03-24T01:27:05.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.104+0000 7f23a2a00640 1 -- start start 2024-03-24T01:27:05.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.105+0000 7f23a2a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23800039e0 0x7f239c0cd720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:05.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.105+0000 7f23a2a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f239c0cd310 0x7f239c1604b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:05.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.105+0000 7f23a2a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f239c15ef80 0x7f239c15d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:05.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.105+0000 7f23a2a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239c0d3480 con 0x7f239c15ef80 2024-03-24T01:27:05.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.105+0000 7f23a2a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239c0d3300 con 0x7f23800039e0 2024-03-24T01:27:05.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.105+0000 7f23a2a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f239c0d3600 con 0x7f239c0cd310 2024-03-24T01:27:05.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.105+0000 7f23a0a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f239c15ef80 0x7f239c15d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:05.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.105+0000 7f23a0a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f239c15ef80 0x7f239c15d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49492/0 (socket says 172.21.15.90:49492) 2024-03-24T01:27:05.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.105+0000 7f23a0a00640 1 -- 172.21.15.90:0/4024762095 learned_addr learned my addr 172.21.15.90:0/4024762095 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:05.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.106+0000 7f239be00640 1 --2- 172.21.15.90:0/4024762095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23800039e0 0x7f239c0cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:05.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.106+0000 7f239b400640 1 --2- 172.21.15.90:0/4024762095 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f239c0cd310 0x7f239c1604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:05.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.106+0000 7f23a0a00640 1 -- 172.21.15.90:0/4024762095 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f239c0cd310 msgr2=0x7f239c1604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:05.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.106+0000 7f23a0a00640 1 --2- 172.21.15.90:0/4024762095 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f239c0cd310 0x7f239c1604b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:05.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.106+0000 7f23a0a00640 1 -- 172.21.15.90:0/4024762095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23800039e0 msgr2=0x7f239c0cd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:05.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.106+0000 7f23a0a00640 1 --2- 172.21.15.90:0/4024762095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23800039e0 0x7f239c0cd720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:05.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.106+0000 7f23a0a00640 1 -- 172.21.15.90:0/4024762095 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f239c1609f0 con 0x7f239c15ef80 2024-03-24T01:27:05.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.106+0000 7f239be00640 1 --2- 172.21.15.90:0/4024762095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23800039e0 0x7f239c0cd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:27:05.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.106+0000 7f23a0a00640 1 --2- 172.21.15.90:0/4024762095 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f239c15ef80 0x7f239c15d0e0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f239c08cba0 tx=0x7f239400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:05.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.106+0000 7f2393e00640 1 -- 172.21.15.90:0/4024762095 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239400bae0 con 0x7f239c15ef80 2024-03-24T01:27:05.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.106+0000 7f2393e00640 1 -- 172.21.15.90:0/4024762095 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f239401a070 con 0x7f239c15ef80 2024-03-24T01:27:05.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.106+0000 7f23a2a00640 1 -- 172.21.15.90:0/4024762095 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f239c15f410 con 0x7f239c15ef80 2024-03-24T01:27:05.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.106+0000 7f2393e00640 1 -- 172.21.15.90:0/4024762095 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2394014400 con 0x7f239c15ef80 2024-03-24T01:27:05.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.106+0000 7f23a2a00640 1 -- 172.21.15.90:0/4024762095 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f239c15f9d0 con 0x7f239c15ef80 2024-03-24T01:27:05.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.108+0000 7f23a2a00640 1 -- 172.21.15.90:0/4024762095 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f239c08b000 con 0x7f239c15ef80 2024-03-24T01:27:05.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.108+0000 7f2393e00640 1 -- 172.21.15.90:0/4024762095 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2394005d10 con 0x7f239c15ef80 2024-03-24T01:27:05.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.109+0000 7f2393e00640 1 --2- 172.21.15.90:0/4024762095 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f237806bbd0 0x7f237806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:05.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.109+0000 7f2393e00640 1 -- 172.21.15.90:0/4024762095 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f239408a890 con 0x7f239c15ef80 2024-03-24T01:27:05.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.110+0000 7f239be00640 1 --2- 172.21.15.90:0/4024762095 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f237806bbd0 0x7f237806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:05.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.111+0000 7f239be00640 1 --2- 172.21.15.90:0/4024762095 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f237806bbd0 0x7f237806e090 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f238c034000 tx=0x7f238c009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:05.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.114+0000 7f2393e00640 1 -- 172.21.15.90:0/4024762095 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2394006000 con 0x7f239c15ef80 2024-03-24T01:27:05.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.384+0000 7f23a2a00640 1 -- 172.21.15.90:0/4024762095 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_journal_repair.TestJournalRepair.test_inject_to_empty"]} v 0) v1 -- 0x7f239c0d18e0 con 0x7f239c15ef80 2024-03-24T01:27:05.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.416+0000 7f2393e00640 1 -- 172.21.15.90:0/4024762095 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_journal_repair.TestJournalRepair.test_inject_to_empty"]}]=0 v181) v1 ==== 150+0+0 (secure 0 0 0) 0x7f2394059530 con 0x7f239c15ef80 2024-03-24T01:27:05.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.419+0000 7f23a2a00640 1 -- 172.21.15.90:0/4024762095 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f237806bbd0 msgr2=0x7f237806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:05.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.419+0000 7f23a2a00640 1 --2- 172.21.15.90:0/4024762095 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f237806bbd0 0x7f237806e090 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f238c034000 tx=0x7f238c009030 comp rx=0 tx=0).stop 2024-03-24T01:27:05.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.419+0000 7f23a2a00640 1 -- 172.21.15.90:0/4024762095 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f239c15ef80 msgr2=0x7f239c15d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:05.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.419+0000 7f23a2a00640 1 --2- 172.21.15.90:0/4024762095 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f239c15ef80 0x7f239c15d0e0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f239c08cba0 tx=0x7f239400b480 comp rx=0 tx=0).stop 2024-03-24T01:27:05.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.419+0000 7f23a2a00640 1 -- 172.21.15.90:0/4024762095 shutdown_connections 2024-03-24T01:27:05.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.419+0000 7f23a2a00640 1 --2- 172.21.15.90:0/4024762095 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f237806bbd0 0x7f237806e090 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:05.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.419+0000 7f23a2a00640 1 --2- 172.21.15.90:0/4024762095 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f239c15ef80 0x7f239c15d0e0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:05.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.419+0000 7f23a2a00640 1 --2- 172.21.15.90:0/4024762095 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f239c0cd310 0x7f239c1604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:05.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.419+0000 7f23a2a00640 1 --2- 172.21.15.90:0/4024762095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23800039e0 0x7f239c0cd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:05.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.419+0000 7f23a2a00640 1 -- 172.21.15.90:0/4024762095 >> 172.21.15.90:0/4024762095 conn(0x7f239c06cac0 msgr2=0x7f239c0b4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:05.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.419+0000 7f23a2a00640 1 -- 172.21.15.90:0/4024762095 shutdown_connections 2024-03-24T01:27:05.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.419+0000 7f23a2a00640 1 -- 172.21.15.90:0/4024762095 wait complete. 2024-03-24T01:27:05.437 INFO:tasks.cephfs_test_runner:That when some dentries in the journal but nothing is in ... ok 2024-03-24T01:27:05.438 INFO:tasks.cephfs_test_runner:Starting test: test_journal_smoke (tasks.cephfs.test_journal_repair.TestJournalRepair) 2024-03-24T01:27:05.438 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:27:05.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.641+0000 7fb390400640 1 Processor -- start 2024-03-24T01:27:05.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.641+0000 7fb390400640 1 -- start start 2024-03-24T01:27:05.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.642+0000 7fb390400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb388155030 0x7fb38815efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:05.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.643+0000 7fb390400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb38815f500 con 0x7fb38806b7d0 2024-03-24T01:27:05.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.643+0000 7fb390400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb38815bf70 con 0x7fb3880cdac0 2024-03-24T01:27:05.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.643+0000 7fb390400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb38815b360 con 0x7fb388155030 2024-03-24T01:27:05.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.643+0000 7fb38da00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb388155030 0x7fb38815efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:05.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.643+0000 7fb38da00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb388155030 0x7fb38815efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49498/0 (socket says 172.21.15.90:49498) 2024-03-24T01:27:05.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.643+0000 7fb38da00640 1 -- 172.21.15.90:0/2781339677 learned_addr learned my addr 172.21.15.90:0/2781339677 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:05.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.643+0000 7fb38da00640 1 -- 172.21.15.90:0/2781339677 >> v1:172.21.15.120:6789/0 conn(0x7fb3880cdac0 legacy=0x7fb38806b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:05.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.643+0000 7fb38da00640 1 -- 172.21.15.90:0/2781339677 >> v1:172.21.15.90:6789/0 conn(0x7fb38806b7d0 legacy=0x7fb3880ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:05.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.643+0000 7fb38da00640 1 -- 172.21.15.90:0/2781339677 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb38815b640 con 0x7fb388155030 2024-03-24T01:27:05.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.643+0000 7fb387e00640 1 -- 172.21.15.90:0/2781339677 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3258871743 0 0) 0x7fb380001490 con 0x7fb38806b7d0 2024-03-24T01:27:05.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.644+0000 7fb38da00640 1 --2- 172.21.15.90:0/2781339677 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb388155030 0x7fb38815efc0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fb37c009870 tx=0x7fb37c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efd35ef47cb45850 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:05.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.645+0000 7fb387e00640 1 -- 172.21.15.90:0/2781339677 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37c034070 con 0x7fb388155030 2024-03-24T01:27:05.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.645+0000 7fb387e00640 1 -- 172.21.15.90:0/2781339677 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb37c030860 con 0x7fb388155030 2024-03-24T01:27:05.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.645+0000 7fb387e00640 1 -- 172.21.15.90:0/2781339677 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37c030c30 con 0x7fb388155030 2024-03-24T01:27:05.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.645+0000 7fb390400640 1 -- 172.21.15.90:0/2781339677 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb388155030 msgr2=0x7fb38815efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:05.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.645+0000 7fb390400640 1 --2- 172.21.15.90:0/2781339677 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb388155030 0x7fb38815efc0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fb37c009870 tx=0x7fb37c02b9a0 comp rx=0 tx=0).stop 2024-03-24T01:27:05.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.645+0000 7fb390400640 1 -- 172.21.15.90:0/2781339677 shutdown_connections 2024-03-24T01:27:05.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.645+0000 7fb390400640 1 --2- 172.21.15.90:0/2781339677 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb388155030 0x7fb38815efc0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:05.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.645+0000 7fb390400640 1 -- 172.21.15.90:0/2781339677 >> 172.21.15.90:0/2781339677 conn(0x7fb3880b1770 msgr2=0x7fb3880b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:05.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.646+0000 7fb390400640 1 -- 172.21.15.90:0/2781339677 shutdown_connections 2024-03-24T01:27:05.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.646+0000 7fb390400640 1 -- 172.21.15.90:0/2781339677 wait complete. 2024-03-24T01:27:05.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.647+0000 7fb390400640 1 Processor -- start 2024-03-24T01:27:05.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.647+0000 7fb390400640 1 -- start start 2024-03-24T01:27:05.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.648+0000 7fb390400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb38806b7d0 0x7fb3880c9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:05.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.648+0000 7fb390400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3880cdac0 0x7fb3881e4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:05.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.648+0000 7fb390400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb388155030 0x7fb388147bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:05.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.648+0000 7fb390400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb388155410 con 0x7fb388155030 2024-03-24T01:27:05.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.648+0000 7fb390400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38815bc00 con 0x7fb38806b7d0 2024-03-24T01:27:05.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.648+0000 7fb390400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb38815af70 con 0x7fb3880cdac0 2024-03-24T01:27:05.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.649+0000 7fb38e400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb388155030 0x7fb388147bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:05.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.649+0000 7fb38e400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb388155030 0x7fb388147bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49504/0 (socket says 172.21.15.90:49504) 2024-03-24T01:27:05.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.649+0000 7fb38e400640 1 -- 172.21.15.90:0/1370676571 learned_addr learned my addr 172.21.15.90:0/1370676571 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:05.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.649+0000 7fb38da00640 1 --2- 172.21.15.90:0/1370676571 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb38806b7d0 0x7fb3880c9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:05.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.649+0000 7fb38d000640 1 --2- 172.21.15.90:0/1370676571 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3880cdac0 0x7fb3881e4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:05.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.649+0000 7fb38e400640 1 -- 172.21.15.90:0/1370676571 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3880cdac0 msgr2=0x7fb3881e4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:05.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.649+0000 7fb38e400640 1 --2- 172.21.15.90:0/1370676571 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3880cdac0 0x7fb3881e4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:05.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.649+0000 7fb38e400640 1 -- 172.21.15.90:0/1370676571 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb38806b7d0 msgr2=0x7fb3880c9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:05.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.649+0000 7fb38e400640 1 --2- 172.21.15.90:0/1370676571 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb38806b7d0 0x7fb3880c9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:05.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.649+0000 7fb38e400640 1 -- 172.21.15.90:0/1370676571 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3880c9d10 con 0x7fb388155030 2024-03-24T01:27:05.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.649+0000 7fb38e400640 1 --2- 172.21.15.90:0/1370676571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb388155030 0x7fb388147bb0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fb38000c7c0 tx=0x7fb38000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:05.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.650+0000 7fb385600640 1 -- 172.21.15.90:0/1370676571 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb380003da0 con 0x7fb388155030 2024-03-24T01:27:05.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.650+0000 7fb385600640 1 -- 172.21.15.90:0/1370676571 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb380005070 con 0x7fb388155030 2024-03-24T01:27:05.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.650+0000 7fb385600640 1 -- 172.21.15.90:0/1370676571 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb380005460 con 0x7fb388155030 2024-03-24T01:27:05.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.650+0000 7fb390400640 1 -- 172.21.15.90:0/1370676571 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3880c9ff0 con 0x7fb388155030 2024-03-24T01:27:05.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.650+0000 7fb390400640 1 -- 172.21.15.90:0/1370676571 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3880ca5b0 con 0x7fb388155030 2024-03-24T01:27:05.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.651+0000 7fb390400640 1 -- 172.21.15.90:0/1370676571 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3880d0680 con 0x7fb388155030 2024-03-24T01:27:05.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.652+0000 7fb385600640 1 -- 172.21.15.90:0/1370676571 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb380005600 con 0x7fb388155030 2024-03-24T01:27:05.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.653+0000 7fb385600640 1 --2- 172.21.15.90:0/1370676571 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb36806bbd0 0x7fb36806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:05.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.653+0000 7fb385600640 1 -- 172.21.15.90:0/1370676571 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fb38008a870 con 0x7fb388155030 2024-03-24T01:27:05.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.653+0000 7fb38da00640 1 --2- 172.21.15.90:0/1370676571 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb36806bbd0 0x7fb36806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:05.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.655+0000 7fb38da00640 1 --2- 172.21.15.90:0/1370676571 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb36806bbd0 0x7fb36806e090 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fb37c036000 tx=0x7fb37c03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:05.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.657+0000 7fb385600640 1 -- 172.21.15.90:0/1370676571 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb38815f500 con 0x7fb388155030 2024-03-24T01:27:05.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.855+0000 7fb390400640 1 -- 172.21.15.90:0/1370676571 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb3880caa50 con 0x7fb388155030 2024-03-24T01:27:05.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.857+0000 7fb385600640 1 -- 172.21.15.90:0/1370676571 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+13395 (secure 0 0 0) 0x7fb3800594f0 con 0x7fb388155030 2024-03-24T01:27:05.857 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:27:05.858 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":45,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:27:04.402373+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:6835/11525832":"2024-03-25T01:27:01.383400+0000","172.21.15.90:6834/11525832":"2024-03-25T01:27:01.383400+0000","172.21.15.90:6835/2166652035":"2024-03-25T01:26:27.648052+0000","172.21.15.90:6834/2166652035":"2024-03-25T01:26:27.648052+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:27:05.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.859+0000 7fb390400640 1 -- 172.21.15.90:0/1370676571 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb36806bbd0 msgr2=0x7fb36806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:05.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.859+0000 7fb390400640 1 --2- 172.21.15.90:0/1370676571 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb36806bbd0 0x7fb36806e090 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fb37c036000 tx=0x7fb37c03b040 comp rx=0 tx=0).stop 2024-03-24T01:27:05.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.859+0000 7fb390400640 1 -- 172.21.15.90:0/1370676571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb388155030 msgr2=0x7fb388147bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:05.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.859+0000 7fb390400640 1 --2- 172.21.15.90:0/1370676571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb388155030 0x7fb388147bb0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fb38000c7c0 tx=0x7fb38000cc80 comp rx=0 tx=0).stop 2024-03-24T01:27:05.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.859+0000 7fb390400640 1 -- 172.21.15.90:0/1370676571 shutdown_connections 2024-03-24T01:27:05.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.859+0000 7fb390400640 1 --2- 172.21.15.90:0/1370676571 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb36806bbd0 0x7fb36806e090 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:05.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.859+0000 7fb390400640 1 --2- 172.21.15.90:0/1370676571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb388155030 0x7fb388147bb0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:05.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.859+0000 7fb390400640 1 --2- 172.21.15.90:0/1370676571 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb3880cdac0 0x7fb3881e4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:05.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.859+0000 7fb390400640 1 --2- 172.21.15.90:0/1370676571 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb38806b7d0 0x7fb3880c9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:05.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.860+0000 7fb390400640 1 -- 172.21.15.90:0/1370676571 >> 172.21.15.90:0/1370676571 conn(0x7fb3880b1770 msgr2=0x7fb3880cff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:05.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.860+0000 7fb390400640 1 -- 172.21.15.90:0/1370676571 shutdown_connections 2024-03-24T01:27:05.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:05.860+0000 7fb390400640 1 -- 172.21.15.90:0/1370676571 wait complete. 2024-03-24T01:27:05.879 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T01:27:05.879 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:27:06.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.091+0000 7f4a93400640 1 Processor -- start 2024-03-24T01:27:06.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.092+0000 7f4a93400640 1 -- start start 2024-03-24T01:27:06.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.093+0000 7f4a93400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a8c06cbc0 0x7f4a8c0b2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:06.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.093+0000 7f4a93400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4a8c0d0d30 0x7f4a8c0b1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:06.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.093+0000 7f4a93400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a8c0b2fc0 con 0x7f4a8c06cfa0 2024-03-24T01:27:06.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.093+0000 7f4a93400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a8c09a780 con 0x7f4a8c0d0d30 2024-03-24T01:27:06.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.093+0000 7f4a93400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8c09aa60 con 0x7f4a8c06cbc0 2024-03-24T01:27:06.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.094+0000 7f4a91400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4a8c0d0d30 0x7f4a8c0b1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:06.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.094+0000 7f4a91400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4a8c0d0d30 0x7f4a8c0b1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.90:48040/0 (socket says 172.21.15.90:48040) 2024-03-24T01:27:06.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.094+0000 7f4a91400640 1 -- 172.21.15.90:0/1952248668 learned_addr learned my addr 172.21.15.90:0/1952248668 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:06.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.094+0000 7f4a90a00640 1 --2- 172.21.15.90:0/1952248668 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a8c06cbc0 0x7f4a8c0b2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:06.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.095+0000 7f4a91400640 1 -- 172.21.15.90:0/1952248668 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a8c06cbc0 msgr2=0x7f4a8c0b2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:06.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.095+0000 7f4a91400640 1 --2- 172.21.15.90:0/1952248668 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a8c06cbc0 0x7f4a8c0b2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:06.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.095+0000 7f4a91400640 1 -- 172.21.15.90:0/1952248668 >> v1:172.21.15.120:6789/0 conn(0x7f4a8c06cfa0 legacy=0x7f4a8c0b1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:06.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.095+0000 7f4a91400640 1 -- 172.21.15.90:0/1952248668 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a8c09ad40 con 0x7f4a8c0d0d30 2024-03-24T01:27:06.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.096+0000 7f4a91400640 1 --2- 172.21.15.90:0/1952248668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a8c0d0d30 0x7f4a8c0b1980 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4a84009870 tx=0x7f4a8402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9fe661b39c99c998 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:06.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.097+0000 7f4a8b400640 1 -- 172.21.15.90:0/1952248668 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a84034070 con 0x7f4a8c0d0d30 2024-03-24T01:27:06.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.097+0000 7f4a8b400640 1 -- 172.21.15.90:0/1952248668 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a84030840 con 0x7f4a8c0d0d30 2024-03-24T01:27:06.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.097+0000 7f4a8b400640 1 -- 172.21.15.90:0/1952248668 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a84030c10 con 0x7f4a8c0d0d30 2024-03-24T01:27:06.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.097+0000 7f4a93400640 1 -- 172.21.15.90:0/1952248668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a8c0d0d30 msgr2=0x7f4a8c0b1980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:06.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.097+0000 7f4a93400640 1 --2- 172.21.15.90:0/1952248668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a8c0d0d30 0x7f4a8c0b1980 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4a84009870 tx=0x7f4a8402b980 comp rx=0 tx=0).stop 2024-03-24T01:27:06.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.097+0000 7f4a93400640 1 -- 172.21.15.90:0/1952248668 shutdown_connections 2024-03-24T01:27:06.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.097+0000 7f4a93400640 1 --2- 172.21.15.90:0/1952248668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a8c0d0d30 0x7f4a8c0b1980 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:06.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.097+0000 7f4a93400640 1 --2- 172.21.15.90:0/1952248668 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a8c06cbc0 0x7f4a8c0b2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:06.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.097+0000 7f4a93400640 1 -- 172.21.15.90:0/1952248668 >> 172.21.15.90:0/1952248668 conn(0x7f4a8c0b34e0 msgr2=0x7f4a8c0b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:06.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.097+0000 7f4a93400640 1 -- 172.21.15.90:0/1952248668 shutdown_connections 2024-03-24T01:27:06.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.097+0000 7f4a93400640 1 -- 172.21.15.90:0/1952248668 wait complete. 2024-03-24T01:27:06.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.099+0000 7f4a93400640 1 Processor -- start 2024-03-24T01:27:06.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.099+0000 7f4a93400640 1 -- start start 2024-03-24T01:27:06.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.100+0000 7f4a93400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a8c06cbc0 0x7f4a8c14ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:06.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.100+0000 7f4a93400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a8c06cfa0 0x7f4a8c15ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:06.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.100+0000 7f4a93400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a8c0d0d30 0x7f4a8c0b4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:06.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.100+0000 7f4a93400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8c1617e0 con 0x7f4a8c06cfa0 2024-03-24T01:27:06.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.100+0000 7f4a93400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8c161660 con 0x7f4a8c0d0d30 2024-03-24T01:27:06.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.100+0000 7f4a93400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8c06d3a0 con 0x7f4a8c06cbc0 2024-03-24T01:27:06.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.100+0000 7f4a8be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a8c06cfa0 0x7f4a8c15ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:06.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.101+0000 7f4a8be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a8c06cfa0 0x7f4a8c15ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49520/0 (socket says 172.21.15.90:49520) 2024-03-24T01:27:06.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.101+0000 7f4a8be00640 1 -- 172.21.15.90:0/1073155990 learned_addr learned my addr 172.21.15.90:0/1073155990 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:06.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.101+0000 7f4a90a00640 1 --2- 172.21.15.90:0/1073155990 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a8c06cbc0 0x7f4a8c14ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:06.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.101+0000 7f4a91400640 1 --2- 172.21.15.90:0/1073155990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a8c0d0d30 0x7f4a8c0b4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:06.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.101+0000 7f4a8be00640 1 -- 172.21.15.90:0/1073155990 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a8c06cbc0 msgr2=0x7f4a8c14ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:06.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.101+0000 7f4a8be00640 1 --2- 172.21.15.90:0/1073155990 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a8c06cbc0 0x7f4a8c14ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:06.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.101+0000 7f4a8be00640 1 -- 172.21.15.90:0/1073155990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a8c0d0d30 msgr2=0x7f4a8c0b4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:06.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.101+0000 7f4a8be00640 1 --2- 172.21.15.90:0/1073155990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a8c0d0d30 0x7f4a8c0b4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:06.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.101+0000 7f4a8be00640 1 -- 172.21.15.90:0/1073155990 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a8c15ca20 con 0x7f4a8c06cfa0 2024-03-24T01:27:06.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.101+0000 7f4a8be00640 1 --2- 172.21.15.90:0/1073155990 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a8c06cfa0 0x7f4a8c15ad50 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f4a7400c3c0 tx=0x7f4a7400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:06.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.102+0000 7f4a88c00640 1 -- 172.21.15.90:0/1073155990 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a740052d0 con 0x7f4a8c06cfa0 2024-03-24T01:27:06.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.102+0000 7f4a88c00640 1 -- 172.21.15.90:0/1073155990 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a74005490 con 0x7f4a8c06cfa0 2024-03-24T01:27:06.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.102+0000 7f4a88c00640 1 -- 172.21.15.90:0/1073155990 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a74007260 con 0x7f4a8c06cfa0 2024-03-24T01:27:06.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.102+0000 7f4a93400640 1 -- 172.21.15.90:0/1073155990 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a8c15cd00 con 0x7f4a8c06cfa0 2024-03-24T01:27:06.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.102+0000 7f4a93400640 1 -- 172.21.15.90:0/1073155990 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a8c15d2c0 con 0x7f4a8c06cfa0 2024-03-24T01:27:06.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.103+0000 7f4a93400640 1 -- 172.21.15.90:0/1073155990 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a8c04f400 con 0x7f4a8c06cfa0 2024-03-24T01:27:06.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.104+0000 7f4a88c00640 1 -- 172.21.15.90:0/1073155990 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4a74007400 con 0x7f4a8c06cfa0 2024-03-24T01:27:06.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.105+0000 7f4a88c00640 1 --2- 172.21.15.90:0/1073155990 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4a6806bbd0 0x7f4a6806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:06.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.105+0000 7f4a88c00640 1 -- 172.21.15.90:0/1073155990 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f4a74089e40 con 0x7f4a8c06cfa0 2024-03-24T01:27:06.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.106+0000 7f4a90a00640 1 --2- 172.21.15.90:0/1073155990 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4a6806bbd0 0x7f4a6806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:06.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.107+0000 7f4a90a00640 1 --2- 172.21.15.90:0/1073155990 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4a6806bbd0 0x7f4a6806e090 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c0073f0 tx=0x7f4a7c00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:06.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.110+0000 7f4a88c00640 1 -- 172.21.15.90:0/1073155990 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4a74058a40 con 0x7f4a8c06cfa0 2024-03-24T01:27:06.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.318+0000 7f4a93400640 1 -- 172.21.15.90:0/1073155990 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4a8c0d1760 con 0x7f4a8c06cfa0 2024-03-24T01:27:06.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.319+0000 7f4a88c00640 1 -- 172.21.15.90:0/1073155990 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v45) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4a74058340 con 0x7f4a8c06cfa0 2024-03-24T01:27:06.320 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:27:06.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.322+0000 7f4a93400640 1 -- 172.21.15.90:0/1073155990 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4a6806bbd0 msgr2=0x7f4a6806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:06.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.322+0000 7f4a93400640 1 --2- 172.21.15.90:0/1073155990 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4a6806bbd0 0x7f4a6806e090 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c0073f0 tx=0x7f4a7c00b040 comp rx=0 tx=0).stop 2024-03-24T01:27:06.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.322+0000 7f4a93400640 1 -- 172.21.15.90:0/1073155990 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a8c06cfa0 msgr2=0x7f4a8c15ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:06.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.322+0000 7f4a93400640 1 --2- 172.21.15.90:0/1073155990 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a8c06cfa0 0x7f4a8c15ad50 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f4a7400c3c0 tx=0x7f4a7400c880 comp rx=0 tx=0).stop 2024-03-24T01:27:06.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.322+0000 7f4a93400640 1 -- 172.21.15.90:0/1073155990 shutdown_connections 2024-03-24T01:27:06.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.322+0000 7f4a93400640 1 --2- 172.21.15.90:0/1073155990 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4a6806bbd0 0x7f4a6806e090 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:06.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.322+0000 7f4a93400640 1 --2- 172.21.15.90:0/1073155990 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a8c06cfa0 0x7f4a8c15ad50 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:06.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.322+0000 7f4a93400640 1 --2- 172.21.15.90:0/1073155990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a8c0d0d30 0x7f4a8c0b4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:06.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.322+0000 7f4a93400640 1 --2- 172.21.15.90:0/1073155990 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a8c06cbc0 0x7f4a8c14ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:06.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.322+0000 7f4a93400640 1 -- 172.21.15.90:0/1073155990 >> 172.21.15.90:0/1073155990 conn(0x7f4a8c0b34e0 msgr2=0x7f4a8c071460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:06.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.322+0000 7f4a93400640 1 -- 172.21.15.90:0/1073155990 shutdown_connections 2024-03-24T01:27:06.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.322+0000 7f4a93400640 1 -- 172.21.15.90:0/1073155990 wait complete. 2024-03-24T01:27:06.340 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_journal_repair.TestJournalRepair.test_journal_smoke' 2024-03-24T01:27:06.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.553+0000 7f0588600640 1 Processor -- start 2024-03-24T01:27:06.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.554+0000 7f0588600640 1 -- start start 2024-03-24T01:27:06.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.555+0000 7f0588600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f05800cc150 0x7f05800ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:06.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.555+0000 7f0588600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f058006b7d0 0x7f05800b3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:06.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.555+0000 7f0588600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05800cd490 con 0x7f05801603e0 2024-03-24T01:27:06.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.555+0000 7f0588600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f058015f900 con 0x7f05800cc150 2024-03-24T01:27:06.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.555+0000 7f0588600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f058015fbe0 con 0x7f058006b7d0 2024-03-24T01:27:06.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.556+0000 7f0585200640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f05801603e0 0x7f058006b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40720/0 (socket says 172.21.15.90:40720) 2024-03-24T01:27:06.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.556+0000 7f0585200640 1 -- 172.21.15.90:0/2301465457 learned_addr learned my addr 172.21.15.90:0/2301465457 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:27:06.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.556+0000 7f0586600640 1 --2- 172.21.15.90:0/2301465457 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f058006b7d0 0x7f05800b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:06.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.556+0000 7f0585c00640 1 --2- 172.21.15.90:0/2301465457 >> v2:172.21.15.120:3300/0 conn(0x7f05800cc150 0x7f05800ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:06.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.556+0000 7f057fe00640 1 -- 172.21.15.90:0/2301465457 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2278988009 0 0) 0x7f05800cd490 con 0x7f05801603e0 2024-03-24T01:27:06.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.556+0000 7f057fe00640 1 -- 172.21.15.90:0/2301465457 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0568003520 con 0x7f05801603e0 2024-03-24T01:27:06.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.557+0000 7f057fe00640 1 -- 172.21.15.90:0/2301465457 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1992981787 0 0) 0x7f0568003520 con 0x7f05801603e0 2024-03-24T01:27:06.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.557+0000 7f0585c00640 1 -- 172.21.15.90:0/2301465457 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f058006b7d0 msgr2=0x7f05800b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:06.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.557+0000 7f0585c00640 1 --2- 172.21.15.90:0/2301465457 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f058006b7d0 0x7f05800b3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:06.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.557+0000 7f0585c00640 1 -- 172.21.15.90:0/2301465457 >> v1:172.21.15.90:6789/0 conn(0x7f05801603e0 legacy=0x7f058006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:06.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.557+0000 7f0585c00640 1 -- 172.21.15.90:0/2301465457 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f058015fec0 con 0x7f05800cc150 2024-03-24T01:27:06.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.557+0000 7f0586600640 1 --2- 172.21.15.90:0/2301465457 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f058006b7d0 0x7f05800b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T01:27:06.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.558+0000 7f0585c00640 1 --2- 172.21.15.90:0/2301465457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05800cc150 0x7f05800ccb60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f05740097a0 tx=0x7f057402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4711dc65aef2f556 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:06.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.559+0000 7f057fe00640 1 -- 172.21.15.90:0/2301465457 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0574037070 con 0x7f05800cc150 2024-03-24T01:27:06.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.559+0000 7f057fe00640 1 -- 172.21.15.90:0/2301465457 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0574030510 con 0x7f05800cc150 2024-03-24T01:27:06.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.559+0000 7f057fe00640 1 -- 172.21.15.90:0/2301465457 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c001670 con 0x7f05800cc150 2024-03-24T01:27:06.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.560+0000 7f0588600640 1 -- 172.21.15.90:0/2301465457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05800cc150 msgr2=0x7f05800ccb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:06.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.560+0000 7f0588600640 1 --2- 172.21.15.90:0/2301465457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05800cc150 0x7f05800ccb60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f05740097a0 tx=0x7f057402b790 comp rx=0 tx=0).stop 2024-03-24T01:27:06.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.560+0000 7f0588600640 1 -- 172.21.15.90:0/2301465457 shutdown_connections 2024-03-24T01:27:06.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.560+0000 7f0588600640 1 --2- 172.21.15.90:0/2301465457 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f058006b7d0 0x7f05800b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:06.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.560+0000 7f0588600640 1 --2- 172.21.15.90:0/2301465457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05800cc150 0x7f05800ccb60 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:06.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.560+0000 7f0588600640 1 -- 172.21.15.90:0/2301465457 >> 172.21.15.90:0/2301465457 conn(0x7f05800b1770 msgr2=0x7f05800b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:06.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.560+0000 7f0588600640 1 -- 172.21.15.90:0/2301465457 shutdown_connections 2024-03-24T01:27:06.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.560+0000 7f0588600640 1 -- 172.21.15.90:0/2301465457 wait complete. 2024-03-24T01:27:06.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.562+0000 7f0588600640 1 Processor -- start 2024-03-24T01:27:06.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.562+0000 7f0588600640 1 -- start start 2024-03-24T01:27:06.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.563+0000 7f0588600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f058006b7d0 0x7f058015aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:06.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.563+0000 7f0588600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05800cc150 0x7f05801f4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:06.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.564+0000 7f0588600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05801603e0 0x7f05800add50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:06.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.564+0000 7f0588600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05801455b0 con 0x7f05801603e0 2024-03-24T01:27:06.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.564+0000 7f0588600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0580145430 con 0x7f05800cc150 2024-03-24T01:27:06.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.564+0000 7f0588600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05800b3c90 con 0x7f058006b7d0 2024-03-24T01:27:06.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.564+0000 7f0586600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05801603e0 0x7f05800add50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:06.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.564+0000 7f0586600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05801603e0 0x7f05800add50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49536/0 (socket says 172.21.15.90:49536) 2024-03-24T01:27:06.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.564+0000 7f0586600640 1 -- 172.21.15.90:0/1815838827 learned_addr learned my addr 172.21.15.90:0/1815838827 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:06.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.564+0000 7f0585200640 1 --2- 172.21.15.90:0/1815838827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05800cc150 0x7f05801f4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:06.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.564+0000 7f0585c00640 1 --2- 172.21.15.90:0/1815838827 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f058006b7d0 0x7f058015aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:06.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.565+0000 7f0586600640 1 -- 172.21.15.90:0/1815838827 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f058006b7d0 msgr2=0x7f058015aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:06.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.565+0000 7f0586600640 1 --2- 172.21.15.90:0/1815838827 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f058006b7d0 0x7f058015aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:06.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.565+0000 7f0586600640 1 -- 172.21.15.90:0/1815838827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05800cc150 msgr2=0x7f05801f4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:06.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.565+0000 7f0586600640 1 --2- 172.21.15.90:0/1815838827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05800cc150 0x7f05801f4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:06.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.565+0000 7f0586600640 1 -- 172.21.15.90:0/1815838827 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05800accb0 con 0x7f05801603e0 2024-03-24T01:27:06.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.565+0000 7f0585200640 1 --2- 172.21.15.90:0/1815838827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05800cc150 0x7f05801f4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:27:06.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.565+0000 7f0586600640 1 --2- 172.21.15.90:0/1815838827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05801603e0 0x7f05800add50 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f0578009eb0 tx=0x7f057800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:06.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.565+0000 7f057d600640 1 -- 172.21.15.90:0/1815838827 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0578018020 con 0x7f05801603e0 2024-03-24T01:27:06.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.565+0000 7f057d600640 1 -- 172.21.15.90:0/1815838827 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0578014480 con 0x7f05801603e0 2024-03-24T01:27:06.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.565+0000 7f057d600640 1 -- 172.21.15.90:0/1815838827 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0578014850 con 0x7f05801603e0 2024-03-24T01:27:06.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.565+0000 7f0588600640 1 -- 172.21.15.90:0/1815838827 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05800acf90 con 0x7f05801603e0 2024-03-24T01:27:06.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.565+0000 7f0588600640 1 -- 172.21.15.90:0/1815838827 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05800ad550 con 0x7f05801603e0 2024-03-24T01:27:06.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.567+0000 7f0588600640 1 -- 172.21.15.90:0/1815838827 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0580068060 con 0x7f05801603e0 2024-03-24T01:27:06.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.568+0000 7f057d600640 1 -- 172.21.15.90:0/1815838827 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f057801b020 con 0x7f05801603e0 2024-03-24T01:27:06.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.568+0000 7f057d600640 1 --2- 172.21.15.90:0/1815838827 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f056006bc00 0x7f056006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:06.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.569+0000 7f057d600640 1 -- 172.21.15.90:0/1815838827 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f057808b840 con 0x7f05801603e0 2024-03-24T01:27:06.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.569+0000 7f0585c00640 1 --2- 172.21.15.90:0/1815838827 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f056006bc00 0x7f056006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:06.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.571+0000 7f0585c00640 1 --2- 172.21.15.90:0/1815838827 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f056006bc00 0x7f056006e0c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f0574034000 tx=0x7f057402f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:06.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.573+0000 7f057d600640 1 -- 172.21.15.90:0/1815838827 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0578022020 con 0x7f05801603e0 2024-03-24T01:27:06.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:06.841+0000 7f0588600640 1 -- 172.21.15.90:0/1815838827 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_journal_repair.TestJournalRepair.test_journal_smoke"]} v 0) v1 -- 0x7f058015b670 con 0x7f05801603e0 2024-03-24T01:27:07.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.428+0000 7f057d600640 1 -- 172.21.15.90:0/1815838827 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_journal_repair.TestJournalRepair.test_journal_smoke"]}]=0 v183) v1 ==== 151+0+0 (secure 0 0 0) 0x7f0578010040 con 0x7f05801603e0 2024-03-24T01:27:07.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.431+0000 7f0588600640 1 -- 172.21.15.90:0/1815838827 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f056006bc00 msgr2=0x7f056006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:07.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.431+0000 7f0588600640 1 --2- 172.21.15.90:0/1815838827 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f056006bc00 0x7f056006e0c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f0574034000 tx=0x7f057402f000 comp rx=0 tx=0).stop 2024-03-24T01:27:07.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.431+0000 7f0588600640 1 -- 172.21.15.90:0/1815838827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05801603e0 msgr2=0x7f05800add50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:07.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.431+0000 7f0588600640 1 --2- 172.21.15.90:0/1815838827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05801603e0 0x7f05800add50 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f0578009eb0 tx=0x7f057800c490 comp rx=0 tx=0).stop 2024-03-24T01:27:07.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.431+0000 7f0588600640 1 -- 172.21.15.90:0/1815838827 shutdown_connections 2024-03-24T01:27:07.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.431+0000 7f0588600640 1 --2- 172.21.15.90:0/1815838827 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f056006bc00 0x7f056006e0c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:07.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.431+0000 7f0588600640 1 --2- 172.21.15.90:0/1815838827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05801603e0 0x7f05800add50 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:07.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.431+0000 7f0588600640 1 --2- 172.21.15.90:0/1815838827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05800cc150 0x7f05801f4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:07.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.431+0000 7f0588600640 1 --2- 172.21.15.90:0/1815838827 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f058006b7d0 0x7f058015aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:07.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.432+0000 7f0588600640 1 -- 172.21.15.90:0/1815838827 >> 172.21.15.90:0/1815838827 conn(0x7f05800b1770 msgr2=0x7f05800cc530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:07.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.432+0000 7f0588600640 1 -- 172.21.15.90:0/1815838827 shutdown_connections 2024-03-24T01:27:07.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.432+0000 7f0588600640 1 -- 172.21.15.90:0/1815838827 wait complete. 2024-03-24T01:27:07.452 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-24T01:27:07.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.660+0000 7f32d3400640 1 Processor -- start 2024-03-24T01:27:07.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.661+0000 7f32d3400640 1 -- start start 2024-03-24T01:27:07.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.662+0000 7f32d3400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f32cc0cca70 0x7f32cc15cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:07.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.662+0000 7f32d3400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32cc066e40 0x7f32cc0cce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:07.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.662+0000 7f32d3400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32cc15d240 con 0x7f32cc160110 2024-03-24T01:27:07.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.662+0000 7f32d3400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32cc15fc80 con 0x7f32cc0cca70 2024-03-24T01:27:07.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.662+0000 7f32d3400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc0cd390 con 0x7f32cc066e40 2024-03-24T01:27:07.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.663+0000 7f32d0a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f32cc0cca70 0x7f32cc15cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:07.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.663+0000 7f32d0a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f32cc0cca70 0x7f32cc15cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49538/0 (socket says 172.21.15.90:49538) 2024-03-24T01:27:07.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.663+0000 7f32d0a00640 1 -- 172.21.15.90:0/2626747711 learned_addr learned my addr 172.21.15.90:0/2626747711 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:07.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.663+0000 7f32d0a00640 1 -- 172.21.15.90:0/2626747711 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32cc066e40 msgr2=0x7f32cc0cce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:07.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.663+0000 7f32d0a00640 1 --2- 172.21.15.90:0/2626747711 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32cc066e40 0x7f32cc0cce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:07.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.663+0000 7f32d0a00640 1 -- 172.21.15.90:0/2626747711 >> v1:172.21.15.120:6789/0 conn(0x7f32cc160110 legacy=0x7f32cc06b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:07.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.663+0000 7f32d0a00640 1 -- 172.21.15.90:0/2626747711 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32cc0b25d0 con 0x7f32cc0cca70 2024-03-24T01:27:07.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.664+0000 7f32d0a00640 1 --2- 172.21.15.90:0/2626747711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32cc0cca70 0x7f32cc15cd00 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f32bc009e50 tx=0x7f32bc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8d3e26ada1efeb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:07.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.664+0000 7f32cb400640 1 -- 172.21.15.90:0/2626747711 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32bc02c9e0 con 0x7f32cc0cca70 2024-03-24T01:27:07.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.664+0000 7f32cb400640 1 -- 172.21.15.90:0/2626747711 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32bc02cb80 con 0x7f32cc0cca70 2024-03-24T01:27:07.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.664+0000 7f32cb400640 1 -- 172.21.15.90:0/2626747711 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32bc0343f0 con 0x7f32cc0cca70 2024-03-24T01:27:07.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.665+0000 7f32d3400640 1 -- 172.21.15.90:0/2626747711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32cc0cca70 msgr2=0x7f32cc15cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:07.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.665+0000 7f32d3400640 1 --2- 172.21.15.90:0/2626747711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32cc0cca70 0x7f32cc15cd00 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f32bc009e50 tx=0x7f32bc02be40 comp rx=0 tx=0).stop 2024-03-24T01:27:07.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.665+0000 7f32d3400640 1 -- 172.21.15.90:0/2626747711 shutdown_connections 2024-03-24T01:27:07.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.665+0000 7f32d3400640 1 --2- 172.21.15.90:0/2626747711 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32cc066e40 0x7f32cc0cce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:07.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.665+0000 7f32d3400640 1 --2- 172.21.15.90:0/2626747711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32cc0cca70 0x7f32cc15cd00 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:07.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.665+0000 7f32d3400640 1 -- 172.21.15.90:0/2626747711 >> 172.21.15.90:0/2626747711 conn(0x7f32cc099a70 msgr2=0x7f32cc0b1d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:07.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.665+0000 7f32d3400640 1 -- 172.21.15.90:0/2626747711 shutdown_connections 2024-03-24T01:27:07.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.665+0000 7f32d3400640 1 -- 172.21.15.90:0/2626747711 wait complete. 2024-03-24T01:27:07.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.667+0000 7f32d3400640 1 Processor -- start 2024-03-24T01:27:07.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.667+0000 7f32d3400640 1 -- start start 2024-03-24T01:27:07.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.667+0000 7f32d3400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32cc066e40 0x7f32cc0cf840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:07.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.668+0000 7f32d3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32cc0cca70 0x7f32cc1559e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:07.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.668+0000 7f32d3400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32cc160110 0x7f32cc151740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:07.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.668+0000 7f32d3400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc15f910 con 0x7f32cc0cca70 2024-03-24T01:27:07.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.668+0000 7f32d3400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc0b2b90 con 0x7f32cc066e40 2024-03-24T01:27:07.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.668+0000 7f32d3400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc160580 con 0x7f32cc160110 2024-03-24T01:27:07.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.668+0000 7f32cbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32cc0cca70 0x7f32cc1559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:07.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.668+0000 7f32cbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32cc0cca70 0x7f32cc1559e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49554/0 (socket says 172.21.15.90:49554) 2024-03-24T01:27:07.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.668+0000 7f32cbe00640 1 -- 172.21.15.90:0/4164331152 learned_addr learned my addr 172.21.15.90:0/4164331152 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:07.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.668+0000 7f32cbe00640 1 -- 172.21.15.90:0/4164331152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32cc160110 msgr2=0x7f32cc151740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:07.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.668+0000 7f32d0a00640 1 --2- 172.21.15.90:0/4164331152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32cc066e40 0x7f32cc0cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:07.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.668+0000 7f32cbe00640 1 --2- 172.21.15.90:0/4164331152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32cc160110 0x7f32cc151740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:07.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.668+0000 7f32cbe00640 1 -- 172.21.15.90:0/4164331152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32cc066e40 msgr2=0x7f32cc0cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:07.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.668+0000 7f32cbe00640 1 --2- 172.21.15.90:0/4164331152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32cc066e40 0x7f32cc0cf840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:07.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.668+0000 7f32cbe00640 1 -- 172.21.15.90:0/4164331152 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32cc0ca170 con 0x7f32cc0cca70 2024-03-24T01:27:07.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.669+0000 7f32cbe00640 1 --2- 172.21.15.90:0/4164331152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32cc0cca70 0x7f32cc1559e0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f32b800b4a0 tx=0x7f32b800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:07.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.669+0000 7f32c8c00640 1 -- 172.21.15.90:0/4164331152 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b8014070 con 0x7f32cc0cca70 2024-03-24T01:27:07.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.669+0000 7f32c8c00640 1 -- 172.21.15.90:0/4164331152 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b8010610 con 0x7f32cc0cca70 2024-03-24T01:27:07.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.669+0000 7f32c8c00640 1 -- 172.21.15.90:0/4164331152 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b8010a00 con 0x7f32cc0cca70 2024-03-24T01:27:07.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.669+0000 7f32d3400640 1 -- 172.21.15.90:0/4164331152 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32cc0cddb0 con 0x7f32cc0cca70 2024-03-24T01:27:07.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.669+0000 7f32d3400640 1 -- 172.21.15.90:0/4164331152 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32cc0ce370 con 0x7f32cc0cca70 2024-03-24T01:27:07.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.670+0000 7f32d3400640 1 -- 172.21.15.90:0/4164331152 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32cc04f400 con 0x7f32cc0cca70 2024-03-24T01:27:07.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.671+0000 7f32c8c00640 1 -- 172.21.15.90:0/4164331152 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f32b8017070 con 0x7f32cc0cca70 2024-03-24T01:27:07.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.672+0000 7f32c8c00640 1 --2- 172.21.15.90:0/4164331152 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f32ac06bbd0 0x7f32ac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:07.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.672+0000 7f32c8c00640 1 -- 172.21.15.90:0/4164331152 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f32b808a3b0 con 0x7f32cc0cca70 2024-03-24T01:27:07.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.673+0000 7f32d0a00640 1 --2- 172.21.15.90:0/4164331152 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f32ac06bbd0 0x7f32ac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:07.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.674+0000 7f32d0a00640 1 --2- 172.21.15.90:0/4164331152 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f32ac06bbd0 0x7f32ac06e090 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f32bc035000 tx=0x7f32bc030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:07.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.676+0000 7f32c8c00640 1 -- 172.21.15.90:0/4164331152 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f32b8058fb0 con 0x7f32cc0cca70 2024-03-24T01:27:07.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.874+0000 7f32d3400640 1 -- 172.21.15.90:0/4164331152 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f32cc0ce540 con 0x7f32cc0cca70 2024-03-24T01:27:07.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.880+0000 7f32c8c00640 1 -- 172.21.15.90:0/4164331152 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7f32b80588b0 con 0x7f32cc0cca70 2024-03-24T01:27:07.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.882+0000 7f32d3400640 1 -- 172.21.15.90:0/4164331152 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f32ac06bbd0 msgr2=0x7f32ac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:07.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.883+0000 7f32d3400640 1 --2- 172.21.15.90:0/4164331152 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f32ac06bbd0 0x7f32ac06e090 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f32bc035000 tx=0x7f32bc030000 comp rx=0 tx=0).stop 2024-03-24T01:27:07.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.883+0000 7f32d3400640 1 -- 172.21.15.90:0/4164331152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32cc0cca70 msgr2=0x7f32cc1559e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:07.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.883+0000 7f32d3400640 1 --2- 172.21.15.90:0/4164331152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32cc0cca70 0x7f32cc1559e0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f32b800b4a0 tx=0x7f32b800b960 comp rx=0 tx=0).stop 2024-03-24T01:27:07.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.883+0000 7f32d3400640 1 -- 172.21.15.90:0/4164331152 shutdown_connections 2024-03-24T01:27:07.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.883+0000 7f32d3400640 1 --2- 172.21.15.90:0/4164331152 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f32ac06bbd0 0x7f32ac06e090 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:07.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.883+0000 7f32d3400640 1 --2- 172.21.15.90:0/4164331152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32cc0cca70 0x7f32cc1559e0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:07.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.883+0000 7f32d3400640 1 --2- 172.21.15.90:0/4164331152 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f32cc160110 0x7f32cc151740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:07.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.883+0000 7f32d3400640 1 --2- 172.21.15.90:0/4164331152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32cc066e40 0x7f32cc0cf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:07.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.883+0000 7f32d3400640 1 -- 172.21.15.90:0/4164331152 >> 172.21.15.90:0/4164331152 conn(0x7f32cc099a70 msgr2=0x7f32cc15b160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:07.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.883+0000 7f32d3400640 1 -- 172.21.15.90:0/4164331152 shutdown_connections 2024-03-24T01:27:07.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:07.883+0000 7f32d3400640 1 -- 172.21.15.90:0/4164331152 wait complete. 2024-03-24T01:27:07.904 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T01:27:07.904 DEBUG:teuthology.orchestra.run.smithi090:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T01:27:08.016 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T01:27:08.147 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T01:27:08.147 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:27:08.147 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:27:08.207 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053414,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498340k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249168k,nr_inodes=812292,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498340k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T01:27:08.207 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:27:08.207 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:27:08.265 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T01:27:08.266 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:27:08.266 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:27:08.326 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053414,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498340k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249168k,nr_inodes=812292,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498340k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T01:27:08.326 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:27:08.326 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:27:08.383 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T01:27:08.384 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-24T01:27:08.384 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:27:08.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.594+0000 7f675fe00640 1 Processor -- start 2024-03-24T01:27:08.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.594+0000 7f675fe00640 1 -- start start 2024-03-24T01:27:08.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.596+0000 7f675fe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67580b2ab0 0x7f67580b24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:08.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.596+0000 7f675fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f67580cc040 0x7f67580cc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:08.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.596+0000 7f675fe00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f675806c5b0 con 0x7f675806cbc0 2024-03-24T01:27:08.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.596+0000 7f675fe00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67580d0e20 con 0x7f67580cc040 2024-03-24T01:27:08.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.596+0000 7f675fe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f675806c890 con 0x7f67580b2ab0 2024-03-24T01:27:08.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.596+0000 7f675d400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f675806cbc0 0x7f675806cfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40734/0 (socket says 172.21.15.90:40734) 2024-03-24T01:27:08.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.596+0000 7f675d400640 1 -- 172.21.15.90:0/222590795 learned_addr learned my addr 172.21.15.90:0/222590795 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:27:08.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.596+0000 7f675de00640 1 --2- 172.21.15.90:0/222590795 >> v2:172.21.15.90:3300/0 conn(0x7f67580cc040 0x7f67580cc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:08.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.597+0000 7f675ca00640 1 --2- 172.21.15.90:0/222590795 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67580b2ab0 0x7f67580b24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:08.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.597+0000 7f6757e00640 1 -- 172.21.15.90:0/222590795 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1831573726 0 0) 0x7f675806c5b0 con 0x7f675806cbc0 2024-03-24T01:27:08.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.597+0000 7f6757e00640 1 -- 172.21.15.90:0/222590795 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6740003520 con 0x7f675806cbc0 2024-03-24T01:27:08.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.597+0000 7f675de00640 1 -- 172.21.15.90:0/222590795 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67580b2ab0 msgr2=0x7f67580b24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:08.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.597+0000 7f675de00640 1 --2- 172.21.15.90:0/222590795 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67580b2ab0 0x7f67580b24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:08.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.597+0000 7f675de00640 1 -- 172.21.15.90:0/222590795 >> v1:172.21.15.90:6789/0 conn(0x7f675806cbc0 legacy=0x7f675806cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:08.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.597+0000 7f675de00640 1 -- 172.21.15.90:0/222590795 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67580d07d0 con 0x7f67580cc040 2024-03-24T01:27:08.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.598+0000 7f675de00640 1 --2- 172.21.15.90:0/222590795 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67580cc040 0x7f67580cc4d0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f6750009dc0 tx=0x7f675002be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29c914ef79d749e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:08.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.598+0000 7f6757e00640 1 -- 172.21.15.90:0/222590795 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6750039070 con 0x7f67580cc040 2024-03-24T01:27:08.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.598+0000 7f6757e00640 1 -- 172.21.15.90:0/222590795 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f675002cbb0 con 0x7f67580cc040 2024-03-24T01:27:08.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.598+0000 7f6757e00640 1 -- 172.21.15.90:0/222590795 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6750034440 con 0x7f67580cc040 2024-03-24T01:27:08.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.599+0000 7f675fe00640 1 -- 172.21.15.90:0/222590795 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67580cc040 msgr2=0x7f67580cc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:08.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.599+0000 7f675fe00640 1 --2- 172.21.15.90:0/222590795 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67580cc040 0x7f67580cc4d0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f6750009dc0 tx=0x7f675002be00 comp rx=0 tx=0).stop 2024-03-24T01:27:08.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.599+0000 7f675fe00640 1 -- 172.21.15.90:0/222590795 shutdown_connections 2024-03-24T01:27:08.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.599+0000 7f675fe00640 1 --2- 172.21.15.90:0/222590795 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67580cc040 0x7f67580cc4d0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:08.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.599+0000 7f675fe00640 1 --2- 172.21.15.90:0/222590795 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67580b2ab0 0x7f67580b24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:08.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.599+0000 7f675fe00640 1 -- 172.21.15.90:0/222590795 >> 172.21.15.90:0/222590795 conn(0x7f67580b34e0 msgr2=0x7f67580b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:08.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.599+0000 7f675fe00640 1 -- 172.21.15.90:0/222590795 shutdown_connections 2024-03-24T01:27:08.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.599+0000 7f675fe00640 1 -- 172.21.15.90:0/222590795 wait complete. 2024-03-24T01:27:08.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.601+0000 7f675fe00640 1 Processor -- start 2024-03-24T01:27:08.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.601+0000 7f675fe00640 1 -- start start 2024-03-24T01:27:08.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.602+0000 7f675fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f675806cbc0 0x7f6758152f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:08.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.602+0000 7f675fe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67580b2ab0 0x7f67580b0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:08.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.602+0000 7f675fe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67580cc040 0x7f67580ce470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:08.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.602+0000 7f675fe00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67580d12b0 con 0x7f675806cbc0 2024-03-24T01:27:08.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.602+0000 7f675fe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675806c1d0 con 0x7f67580b2ab0 2024-03-24T01:27:08.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.602+0000 7f675fe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67580b0bb0 con 0x7f67580cc040 2024-03-24T01:27:08.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.602+0000 7f675d400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f675806cbc0 0x7f6758152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:08.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.602+0000 7f675d400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f675806cbc0 0x7f6758152f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49582/0 (socket says 172.21.15.90:49582) 2024-03-24T01:27:08.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.602+0000 7f675d400640 1 -- 172.21.15.90:0/4294010398 learned_addr learned my addr 172.21.15.90:0/4294010398 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:08.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.602+0000 7f675d400640 1 -- 172.21.15.90:0/4294010398 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67580cc040 msgr2=0x7f67580ce470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:08.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.602+0000 7f675d400640 1 --2- 172.21.15.90:0/4294010398 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67580cc040 0x7f67580ce470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:08.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.602+0000 7f675d400640 1 -- 172.21.15.90:0/4294010398 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67580b2ab0 msgr2=0x7f67580b0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:08.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.602+0000 7f675d400640 1 --2- 172.21.15.90:0/4294010398 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67580b2ab0 0x7f67580b0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:08.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.602+0000 7f675d400640 1 -- 172.21.15.90:0/4294010398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67580b0e90 con 0x7f675806cbc0 2024-03-24T01:27:08.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.603+0000 7f675d400640 1 --2- 172.21.15.90:0/4294010398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f675806cbc0 0x7f6758152f10 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f674c00c390 tx=0x7f674c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:08.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.603+0000 7f6755600640 1 -- 172.21.15.90:0/4294010398 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674c017070 con 0x7f675806cbc0 2024-03-24T01:27:08.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.603+0000 7f6755600640 1 -- 172.21.15.90:0/4294010398 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f674c005ce0 con 0x7f675806cbc0 2024-03-24T01:27:08.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.603+0000 7f6755600640 1 -- 172.21.15.90:0/4294010398 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674c0072f0 con 0x7f675806cbc0 2024-03-24T01:27:08.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.603+0000 7f675fe00640 1 -- 172.21.15.90:0/4294010398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67580b1170 con 0x7f675806cbc0 2024-03-24T01:27:08.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.603+0000 7f675fe00640 1 -- 172.21.15.90:0/4294010398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67580b1730 con 0x7f675806cbc0 2024-03-24T01:27:08.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.604+0000 7f675fe00640 1 -- 172.21.15.90:0/4294010398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6758068060 con 0x7f675806cbc0 2024-03-24T01:27:08.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.605+0000 7f6755600640 1 -- 172.21.15.90:0/4294010398 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f674c01a070 con 0x7f675806cbc0 2024-03-24T01:27:08.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.606+0000 7f6755600640 1 --2- 172.21.15.90:0/4294010398 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f673806bbd0 0x7f673806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:08.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.606+0000 7f6755600640 1 -- 172.21.15.90:0/4294010398 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f674c089d90 con 0x7f675806cbc0 2024-03-24T01:27:08.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.607+0000 7f675ca00640 1 --2- 172.21.15.90:0/4294010398 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f673806bbd0 0x7f673806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:08.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.608+0000 7f675ca00640 1 --2- 172.21.15.90:0/4294010398 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f673806bbd0 0x7f673806e090 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6744006fd0 tx=0x7f6744008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:08.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.611+0000 7f6755600640 1 -- 172.21.15.90:0/4294010398 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f674c058990 con 0x7f675806cbc0 2024-03-24T01:27:08.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.846+0000 7f675fe00640 1 -- 172.21.15.90:0/4294010398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67580b1900 con 0x7f675806cbc0 2024-03-24T01:27:08.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.848+0000 7f6755600640 1 -- 172.21.15.90:0/4294010398 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+3699 (secure 0 0 0) 0x7f674c058290 con 0x7f675806cbc0 2024-03-24T01:27:08.848 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:27:08.849 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4763,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/296084634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":296084634},{"type":"v1","addr":"172.21.15.120:6835","nonce":296084634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5026,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/797786252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":797786252},{"type":"v1","addr":"172.21.15.120:6837","nonce":797786252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5490,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6833/1983177399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1983177399},{"type":"v1","addr":"172.21.15.90:6833","nonce":1983177399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":6129,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/2605492639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":2605492639},{"type":"v1","addr":"172.21.15.90:6835","nonce":2605492639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[]} 2024-03-24T01:27:08.849 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-03-24T01:27:08.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.851+0000 7f675fe00640 1 -- 172.21.15.90:0/4294010398 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f673806bbd0 msgr2=0x7f673806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:08.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.851+0000 7f675fe00640 1 --2- 172.21.15.90:0/4294010398 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f673806bbd0 0x7f673806e090 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6744006fd0 tx=0x7f6744008040 comp rx=0 tx=0).stop 2024-03-24T01:27:08.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.851+0000 7f675fe00640 1 -- 172.21.15.90:0/4294010398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f675806cbc0 msgr2=0x7f6758152f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:08.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.851+0000 7f675fe00640 1 --2- 172.21.15.90:0/4294010398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f675806cbc0 0x7f6758152f10 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f674c00c390 tx=0x7f674c00c850 comp rx=0 tx=0).stop 2024-03-24T01:27:08.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.851+0000 7f675fe00640 1 -- 172.21.15.90:0/4294010398 shutdown_connections 2024-03-24T01:27:08.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.851+0000 7f675fe00640 1 --2- 172.21.15.90:0/4294010398 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f673806bbd0 0x7f673806e090 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:08.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.851+0000 7f675fe00640 1 --2- 172.21.15.90:0/4294010398 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f67580cc040 0x7f67580ce470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:08.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.851+0000 7f675fe00640 1 --2- 172.21.15.90:0/4294010398 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67580b2ab0 0x7f67580b0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:08.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.851+0000 7f675fe00640 1 --2- 172.21.15.90:0/4294010398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f675806cbc0 0x7f6758152f10 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:08.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.851+0000 7f675fe00640 1 -- 172.21.15.90:0/4294010398 >> 172.21.15.90:0/4294010398 conn(0x7f67580b34e0 msgr2=0x7f67580caad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:08.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.851+0000 7f675fe00640 1 -- 172.21.15.90:0/4294010398 shutdown_connections 2024-03-24T01:27:08.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:08.851+0000 7f675fe00640 1 -- 172.21.15.90:0/4294010398 wait complete. 2024-03-24T01:27:08.870 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-24T01:27:08.870 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-24T01:27:08.870 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-24T01:27:08.870 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:27:08.870 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-24T01:27:08.870 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-24T01:27:08.870 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-24T01:27:08.871 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:27:08.871 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-24T01:27:08.871 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-24T01:27:08.871 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-24T01:27:08.871 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:27:08.871 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-24T01:27:08.871 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-24T01:27:08.871 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-24T01:27:08.872 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:27:08.887 INFO:tasks.ceph.mds.a:Stopped 2024-03-24T01:27:08.887 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-24T01:27:08.892 INFO:tasks.ceph.mds.d:Stopped 2024-03-24T01:27:08.892 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-24T01:27:08.894 INFO:tasks.ceph.mds.c:Stopped 2024-03-24T01:27:08.894 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-24T01:27:08.900 INFO:tasks.ceph.mds.b:Stopped 2024-03-24T01:27:08.900 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-24T01:27:08.941 INFO:tasks.ceph.mds.a:Started 2024-03-24T01:27:08.941 DEBUG:teuthology.parallel:result is None 2024-03-24T01:27:08.944 INFO:tasks.ceph.mds.d:Started 2024-03-24T01:27:08.945 DEBUG:teuthology.parallel:result is None 2024-03-24T01:27:08.947 INFO:tasks.ceph.mds.c:Started 2024-03-24T01:27:08.947 DEBUG:teuthology.parallel:result is None 2024-03-24T01:27:08.950 INFO:tasks.ceph.mds.b:Started 2024-03-24T01:27:08.950 DEBUG:teuthology.parallel:result is None 2024-03-24T01:27:08.951 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-24T01:27:09.095 INFO:tasks.ceph.mds.a.smithi090.stdout:starting mds.a at 2024-03-24T01:27:09.096 INFO:tasks.ceph.mds.c.smithi090.stdout:starting mds.c at 2024-03-24T01:27:09.106 INFO:tasks.ceph.mds.d.smithi120.stdout:starting mds.d at 2024-03-24T01:27:09.107 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2024-03-24T01:27:09.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.134+0000 7f3e0ac00640 1 Processor -- start 2024-03-24T01:27:09.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.134+0000 7f3e0ac00640 1 -- start start 2024-03-24T01:27:09.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.136+0000 7f3e0ac00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e040aee80 0x7f3e040b33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:09.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.136+0000 7f3e0ac00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3e0406b0c0 0x7f3e0406b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:09.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.136+0000 7f3e0ac00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e040b3eb0 con 0x7f3e040b3930 2024-03-24T01:27:09.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.136+0000 7f3e0ac00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e0406e680 con 0x7f3e0406b0c0 2024-03-24T01:27:09.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.136+0000 7f3e0ac00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0406e960 con 0x7f3e040aee80 2024-03-24T01:27:09.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.136+0000 7f3e08c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3e0406b0c0 0x7f3e0406b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:09.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.136+0000 7f3e08c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3e0406b0c0 0x7f3e0406b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49620/0 (socket says 172.21.15.90:49620) 2024-03-24T01:27:09.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.136+0000 7f3e08c00640 1 -- 172.21.15.90:0/1082180421 learned_addr learned my addr 172.21.15.90:0/1082180421 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:09.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.136+0000 7f3dfb400640 1 --1- 172.21.15.90:0/1082180421 >> v1:172.21.15.120:6789/0 conn(0x7f3e040b3930 0x7f3e040cc8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.90:34898/0 (socket says 172.21.15.90:34898) 2024-03-24T01:27:09.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.136+0000 7f3e03e00640 1 --2- 172.21.15.90:0/1082180421 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e040aee80 0x7f3e040b33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:09.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.136+0000 7f3e08c00640 1 -- 172.21.15.90:0/1082180421 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e040aee80 msgr2=0x7f3e040b33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:09.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.137+0000 7f3e08c00640 1 --2- 172.21.15.90:0/1082180421 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e040aee80 0x7f3e040b33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:09.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.137+0000 7f3e08c00640 1 -- 172.21.15.90:0/1082180421 >> v1:172.21.15.120:6789/0 conn(0x7f3e040b3930 legacy=0x7f3e040cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:09.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.137+0000 7f3e08c00640 1 -- 172.21.15.90:0/1082180421 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e0406ec40 con 0x7f3e0406b0c0 2024-03-24T01:27:09.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.138+0000 7f3e08c00640 1 --2- 172.21.15.90:0/1082180421 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e0406b0c0 0x7f3e0406b550 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc009e50 tx=0x7f3dfc02bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55cbc7ec94372e77 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:09.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.138+0000 7f3e03400640 1 -- 172.21.15.90:0/1082180421 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc02ca60 con 0x7f3e0406b0c0 2024-03-24T01:27:09.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.138+0000 7f3e03400640 1 -- 172.21.15.90:0/1082180421 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dfc02cc00 con 0x7f3e0406b0c0 2024-03-24T01:27:09.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.138+0000 7f3e03400640 1 -- 172.21.15.90:0/1082180421 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc0343f0 con 0x7f3e0406b0c0 2024-03-24T01:27:09.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.138+0000 7f3e0ac00640 1 -- 172.21.15.90:0/1082180421 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e0406b0c0 msgr2=0x7f3e0406b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:09.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.138+0000 7f3e0ac00640 1 --2- 172.21.15.90:0/1082180421 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e0406b0c0 0x7f3e0406b550 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc009e50 tx=0x7f3dfc02bec0 comp rx=0 tx=0).stop 2024-03-24T01:27:09.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.138+0000 7f3e0ac00640 1 -- 172.21.15.90:0/1082180421 shutdown_connections 2024-03-24T01:27:09.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.138+0000 7f3e0ac00640 1 --2- 172.21.15.90:0/1082180421 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e0406b0c0 0x7f3e0406b550 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:09.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.138+0000 7f3e0ac00640 1 --2- 172.21.15.90:0/1082180421 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e040aee80 0x7f3e040b33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:09.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.138+0000 7f3e0ac00640 1 -- 172.21.15.90:0/1082180421 >> 172.21.15.90:0/1082180421 conn(0x7f3e040b1770 msgr2=0x7f3e040b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:09.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.139+0000 7f3e0ac00640 1 -- 172.21.15.90:0/1082180421 shutdown_connections 2024-03-24T01:27:09.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.139+0000 7f3e0ac00640 1 -- 172.21.15.90:0/1082180421 wait complete. 2024-03-24T01:27:09.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.140+0000 7f3e0ac00640 1 Processor -- start 2024-03-24T01:27:09.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.140+0000 7f3e0ac00640 1 -- start start 2024-03-24T01:27:09.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.141+0000 7f3e0ac00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e0406b0c0 0x7f3e041fa920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:09.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.141+0000 7f3e0ac00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e040aee80 0x7f3e04164610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:09.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.141+0000 7f3e0ac00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e040b3930 0x7f3e04164b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:09.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.141+0000 7f3e0ac00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e040ccfb0 con 0x7f3e040aee80 2024-03-24T01:27:09.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.141+0000 7f3e0ac00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e04166ad0 con 0x7f3e0406b0c0 2024-03-24T01:27:09.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.141+0000 7f3e0ac00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e040cd130 con 0x7f3e040b3930 2024-03-24T01:27:09.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.141+0000 7f3dfb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e040aee80 0x7f3e04164610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:09.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.142+0000 7f3dfb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e040aee80 0x7f3e04164610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49632/0 (socket says 172.21.15.90:49632) 2024-03-24T01:27:09.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.142+0000 7f3dfb400640 1 -- 172.21.15.90:0/749003371 learned_addr learned my addr 172.21.15.90:0/749003371 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:09.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.142+0000 7f3e03e00640 1 --2- 172.21.15.90:0/749003371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e0406b0c0 0x7f3e041fa920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:09.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.142+0000 7f3e08c00640 1 --2- 172.21.15.90:0/749003371 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e040b3930 0x7f3e04164b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:09.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.142+0000 7f3dfb400640 1 -- 172.21.15.90:0/749003371 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e040b3930 msgr2=0x7f3e04164b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:09.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.142+0000 7f3dfb400640 1 --2- 172.21.15.90:0/749003371 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e040b3930 0x7f3e04164b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:09.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.142+0000 7f3dfb400640 1 -- 172.21.15.90:0/749003371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e0406b0c0 msgr2=0x7f3e041fa920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:09.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.142+0000 7f3dfb400640 1 --2- 172.21.15.90:0/749003371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e0406b0c0 0x7f3e041fa920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:09.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.142+0000 7f3dfb400640 1 -- 172.21.15.90:0/749003371 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e04163380 con 0x7f3e040aee80 2024-03-24T01:27:09.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.142+0000 7f3dfb400640 1 --2- 172.21.15.90:0/749003371 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e040aee80 0x7f3e04164610 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f3dec00c400 tx=0x7f3dec00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:09.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.143+0000 7f3e00c00640 1 -- 172.21.15.90:0/749003371 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dec0074e0 con 0x7f3e040aee80 2024-03-24T01:27:09.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.143+0000 7f3e00c00640 1 -- 172.21.15.90:0/749003371 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dec007680 con 0x7f3e040aee80 2024-03-24T01:27:09.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.143+0000 7f3e00c00640 1 -- 172.21.15.90:0/749003371 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dec004a00 con 0x7f3e040aee80 2024-03-24T01:27:09.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.143+0000 7f3e0ac00640 1 -- 172.21.15.90:0/749003371 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e04163660 con 0x7f3e040aee80 2024-03-24T01:27:09.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.143+0000 7f3e0ac00640 1 -- 172.21.15.90:0/749003371 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e04163c20 con 0x7f3e040aee80 2024-03-24T01:27:09.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.144+0000 7f3e0ac00640 1 -- 172.21.15.90:0/749003371 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e0404f400 con 0x7f3e040aee80 2024-03-24T01:27:09.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.145+0000 7f3e00c00640 1 -- 172.21.15.90:0/749003371 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3dec01a020 con 0x7f3e040aee80 2024-03-24T01:27:09.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.146+0000 7f3e00c00640 1 --2- 172.21.15.90:0/749003371 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3de006bbd0 0x7f3de006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:09.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.146+0000 7f3e00c00640 1 -- 172.21.15.90:0/749003371 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f3dec08a170 con 0x7f3e040aee80 2024-03-24T01:27:09.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.146+0000 7f3e03e00640 1 --2- 172.21.15.90:0/749003371 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3de006bbd0 0x7f3de006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:09.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.147+0000 7f3e03e00640 1 --2- 172.21.15.90:0/749003371 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3de006bbd0 0x7f3de006e090 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3df4009b40 tx=0x7f3df4009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:09.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.150+0000 7f3e00c00640 1 -- 172.21.15.90:0/749003371 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3dec058ce0 con 0x7f3e040aee80 2024-03-24T01:27:09.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.367+0000 7f3e0ac00640 1 -- 172.21.15.90:0/749003371 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f3e040cec20 con 0x7f3e040aee80 2024-03-24T01:27:09.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.465+0000 7f3e00c00640 1 -- 172.21.15.90:0/749003371 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v47) v1 ==== 97+0+0 (secure 0 0 0) 0x7f3dec0585e0 con 0x7f3e040aee80 2024-03-24T01:27:09.466 INFO:teuthology.orchestra.run.smithi090.stderr: removed all blocklist entries 2024-03-24T01:27:09.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.469+0000 7f3e0ac00640 1 -- 172.21.15.90:0/749003371 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3de006bbd0 msgr2=0x7f3de006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:09.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.469+0000 7f3e0ac00640 1 --2- 172.21.15.90:0/749003371 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3de006bbd0 0x7f3de006e090 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3df4009b40 tx=0x7f3df4009190 comp rx=0 tx=0).stop 2024-03-24T01:27:09.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.469+0000 7f3e0ac00640 1 -- 172.21.15.90:0/749003371 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e040aee80 msgr2=0x7f3e04164610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:09.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.469+0000 7f3e0ac00640 1 --2- 172.21.15.90:0/749003371 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e040aee80 0x7f3e04164610 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f3dec00c400 tx=0x7f3dec00c8c0 comp rx=0 tx=0).stop 2024-03-24T01:27:09.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.469+0000 7f3e0ac00640 1 -- 172.21.15.90:0/749003371 shutdown_connections 2024-03-24T01:27:09.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.469+0000 7f3e0ac00640 1 --2- 172.21.15.90:0/749003371 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3de006bbd0 0x7f3de006e090 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:09.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.469+0000 7f3e0ac00640 1 --2- 172.21.15.90:0/749003371 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e040aee80 0x7f3e04164610 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:09.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.469+0000 7f3e0ac00640 1 --2- 172.21.15.90:0/749003371 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3e040b3930 0x7f3e04164b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:09.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.469+0000 7f3e0ac00640 1 --2- 172.21.15.90:0/749003371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e0406b0c0 0x7f3e041fa920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:09.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.469+0000 7f3e0ac00640 1 -- 172.21.15.90:0/749003371 >> 172.21.15.90:0/749003371 conn(0x7f3e040b1770 msgr2=0x7f3e040cc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:09.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.469+0000 7f3e0ac00640 1 -- 172.21.15.90:0/749003371 shutdown_connections 2024-03-24T01:27:09.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.469+0000 7f3e0ac00640 1 -- 172.21.15.90:0/749003371 wait complete. 2024-03-24T01:27:09.493 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-24T01:27:09.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.698+0000 7f1392200640 1 Processor -- start 2024-03-24T01:27:09.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.698+0000 7f1392200640 1 -- start start 2024-03-24T01:27:09.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.700+0000 7f1392200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f138c0d0700 0x7f138c0b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:09.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.700+0000 7f1392200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f138c067750 con 0x7f138c15bde0 2024-03-24T01:27:09.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.700+0000 7f1392200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f138c0cfa00 con 0x7f138c0b19c0 2024-03-24T01:27:09.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.700+0000 7f1392200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f138c0b1530 con 0x7f138c0d0700 2024-03-24T01:27:09.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.700+0000 7f138b400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f138c15bde0 0x7f138c0cfff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40756/0 (socket says 172.21.15.90:40756) 2024-03-24T01:27:09.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.700+0000 7f138b400640 1 -- 172.21.15.90:0/228889028 learned_addr learned my addr 172.21.15.90:0/228889028 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:27:09.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.700+0000 7f138aa00640 1 --2- 172.21.15.90:0/228889028 >> v2:172.21.15.90:3300/0 conn(0x7f138c0d0700 0x7f138c0b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:09.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.701+0000 7f138a000640 1 -- 172.21.15.90:0/228889028 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3946817344 0 0) 0x7f138c067750 con 0x7f138c15bde0 2024-03-24T01:27:09.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.701+0000 7f138a000640 1 -- 172.21.15.90:0/228889028 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1370003520 con 0x7f138c15bde0 2024-03-24T01:27:09.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.701+0000 7f138aa00640 1 -- 172.21.15.90:0/228889028 >> v1:172.21.15.120:6789/0 conn(0x7f138c0b19c0 legacy=0x7f138c06b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:09.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.701+0000 7f138aa00640 1 -- 172.21.15.90:0/228889028 >> v1:172.21.15.90:6789/0 conn(0x7f138c15bde0 legacy=0x7f138c0cfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:09.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.701+0000 7f138aa00640 1 -- 172.21.15.90:0/228889028 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f138c0cfce0 con 0x7f138c0d0700 2024-03-24T01:27:09.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.702+0000 7f138aa00640 1 --2- 172.21.15.90:0/228889028 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f138c0d0700 0x7f138c0b4f90 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f137c0097a0 tx=0x7f137c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=338a734ebf077a01 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:09.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.702+0000 7f138a000640 1 -- 172.21.15.90:0/228889028 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f137c035020 con 0x7f138c0d0700 2024-03-24T01:27:09.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.702+0000 7f138a000640 1 -- 172.21.15.90:0/228889028 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f137c031830 con 0x7f138c0d0700 2024-03-24T01:27:09.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.702+0000 7f138a000640 1 -- 172.21.15.90:0/228889028 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f137c031c00 con 0x7f138c0d0700 2024-03-24T01:27:09.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.703+0000 7f1392200640 1 -- 172.21.15.90:0/228889028 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f138c0d0700 msgr2=0x7f138c0b4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:09.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.703+0000 7f1392200640 1 --2- 172.21.15.90:0/228889028 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f138c0d0700 0x7f138c0b4f90 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f137c0097a0 tx=0x7f137c02b8d0 comp rx=0 tx=0).stop 2024-03-24T01:27:09.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.703+0000 7f1392200640 1 -- 172.21.15.90:0/228889028 shutdown_connections 2024-03-24T01:27:09.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.703+0000 7f1392200640 1 --2- 172.21.15.90:0/228889028 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f138c0d0700 0x7f138c0b4f90 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:09.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.703+0000 7f1392200640 1 -- 172.21.15.90:0/228889028 >> 172.21.15.90:0/228889028 conn(0x7f138c099a70 msgr2=0x7f138c0b5650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:09.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.703+0000 7f1392200640 1 -- 172.21.15.90:0/228889028 shutdown_connections 2024-03-24T01:27:09.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.703+0000 7f1392200640 1 -- 172.21.15.90:0/228889028 wait complete. 2024-03-24T01:27:09.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.706+0000 7f1392200640 1 Processor -- start 2024-03-24T01:27:09.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.706+0000 7f1392200640 1 -- start start 2024-03-24T01:27:09.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.707+0000 7f1392200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f138c0b19c0 0x7f138c143510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:09.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.707+0000 7f1392200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f138c0d0700 0x7f138c142970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:09.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.707+0000 7f1392200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f138c15bde0 0x7f138c0ce700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:09.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.707+0000 7f1392200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138c0b10c0 con 0x7f138c0d0700 2024-03-24T01:27:09.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.707+0000 7f1392200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138c0b6260 con 0x7f138c0b19c0 2024-03-24T01:27:09.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.707+0000 7f1392200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f138c0b1240 con 0x7f138c15bde0 2024-03-24T01:27:09.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.707+0000 7f138aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f138c0d0700 0x7f138c142970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:09.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.707+0000 7f138aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f138c0d0700 0x7f138c142970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49650/0 (socket says 172.21.15.90:49650) 2024-03-24T01:27:09.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.707+0000 7f138aa00640 1 -- 172.21.15.90:0/1423299173 learned_addr learned my addr 172.21.15.90:0/1423299173 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:09.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.708+0000 7f138b400640 1 --2- 172.21.15.90:0/1423299173 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f138c0b19c0 0x7f138c143510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:09.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.708+0000 7f138aa00640 1 -- 172.21.15.90:0/1423299173 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f138c15bde0 msgr2=0x7f138c0ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:09.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.708+0000 7f138aa00640 1 --2- 172.21.15.90:0/1423299173 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f138c15bde0 0x7f138c0ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:09.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.708+0000 7f138aa00640 1 -- 172.21.15.90:0/1423299173 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f138c0b19c0 msgr2=0x7f138c143510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:09.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.708+0000 7f138aa00640 1 --2- 172.21.15.90:0/1423299173 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f138c0b19c0 0x7f138c143510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:09.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.708+0000 7f138aa00640 1 -- 172.21.15.90:0/1423299173 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f138c14ebc0 con 0x7f138c0d0700 2024-03-24T01:27:09.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.708+0000 7f138aa00640 1 --2- 172.21.15.90:0/1423299173 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f138c0d0700 0x7f138c142970 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f137c02bee0 tx=0x7f137c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:09.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.708+0000 7f1383400640 1 -- 172.21.15.90:0/1423299173 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f137c035040 con 0x7f138c0d0700 2024-03-24T01:27:09.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.708+0000 7f1383400640 1 -- 172.21.15.90:0/1423299173 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f137c03b920 con 0x7f138c0d0700 2024-03-24T01:27:09.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.708+0000 7f1383400640 1 -- 172.21.15.90:0/1423299173 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f137c03bd10 con 0x7f138c0d0700 2024-03-24T01:27:09.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.708+0000 7f1392200640 1 -- 172.21.15.90:0/1423299173 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f138c14eea0 con 0x7f138c0d0700 2024-03-24T01:27:09.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.709+0000 7f1392200640 1 -- 172.21.15.90:0/1423299173 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f138c14f460 con 0x7f138c0d0700 2024-03-24T01:27:09.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.710+0000 7f1392200640 1 -- 172.21.15.90:0/1423299173 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f138c072eb0 con 0x7f138c0d0700 2024-03-24T01:27:09.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.711+0000 7f1383400640 1 -- 172.21.15.90:0/1423299173 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f137c038020 con 0x7f138c0d0700 2024-03-24T01:27:09.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.712+0000 7f1383400640 1 --2- 172.21.15.90:0/1423299173 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f136c06bbd0 0x7f136c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:09.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.712+0000 7f1383400640 1 -- 172.21.15.90:0/1423299173 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f137c0acac0 con 0x7f138c0d0700 2024-03-24T01:27:09.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.712+0000 7f138b400640 1 --2- 172.21.15.90:0/1423299173 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f136c06bbd0 0x7f136c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:09.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.714+0000 7f138b400640 1 --2- 172.21.15.90:0/1423299173 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f136c06bbd0 0x7f136c06e090 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f13780011e0 tx=0x7f1378009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:09.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.716+0000 7f1383400640 1 -- 172.21.15.90:0/1423299173 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f138c072eb0 con 0x7f138c0d0700 2024-03-24T01:27:09.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.985+0000 7f1392200640 1 -- 172.21.15.90:0/1423299173 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f138c09e490 con 0x7f138c0d0700 2024-03-24T01:27:09.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.986+0000 7f1383400640 1 -- 172.21.15.90:0/1423299173 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f137c07b6e0 con 0x7f138c0d0700 2024-03-24T01:27:09.987 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:27:09.987 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-24T01:27:09.987 INFO:teuthology.orchestra.run.smithi090.stdout: "auth_dump": [ 2024-03-24T01:27:09.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:27:09.987 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mds.a", 2024-03-24T01:27:09.987 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQBzgP9lEcWmMBAAtUqishOFkZW8T4Hh1Cgj2A==", 2024-03-24T01:27:09.987 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:27:09.987 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-24T01:27:09.987 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow *", 2024-03-24T01:27:09.987 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow *", 2024-03-24T01:27:09.987 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:27:09.987 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:27:09.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:27:09.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:27:09.988 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mds.b", 2024-03-24T01:27:09.988 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQB0gP9lIPZ1FRAAmeGByOKmL/VQTpWm4Y2/Tg==", 2024-03-24T01:27:09.988 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:27:09.988 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-24T01:27:09.988 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow *", 2024-03-24T01:27:09.988 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow *", 2024-03-24T01:27:09.988 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:27:09.988 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:27:09.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:27:09.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:27:09.988 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mds.c", 2024-03-24T01:27:09.988 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQB0gP9lRuOuBBAA3/3PMDba2Tlc6BqaY/499g==", 2024-03-24T01:27:09.989 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:27:09.989 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-24T01:27:09.989 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow *", 2024-03-24T01:27:09.989 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow *", 2024-03-24T01:27:09.989 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:27:09.989 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:27:09.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:27:09.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:27:09.989 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mds.d", 2024-03-24T01:27:09.989 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQB0gP9lOtsLJhAAYUVPFa1eoDHF5MV42s8e6A==", 2024-03-24T01:27:09.989 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:27:09.989 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-24T01:27:09.989 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow *", 2024-03-24T01:27:09.989 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow *", 2024-03-24T01:27:09.989 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:27:09.990 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:27:09.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:27:09.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:27:09.990 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.0", 2024-03-24T01:27:09.990 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQB5gP9lJ2jHGRAAPlZjS638Xdt6yVc+vT2ghw==", 2024-03-24T01:27:09.990 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:27:09.990 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-24T01:27:09.990 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-24T01:27:09.990 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:27:09.990 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:27:09.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:27:09.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:27:09.990 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.1", 2024-03-24T01:27:09.990 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQB8gP9lEN/pIxAAvzVfJNpo7FP6c8KSbZdy9A==", 2024-03-24T01:27:09.990 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:27:09.991 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-24T01:27:09.991 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-24T01:27:09.991 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:27:09.991 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:27:09.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:27:09.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:27:09.991 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.2", 2024-03-24T01:27:09.991 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQB/gP9lKv5mLRAAJXArmoRRVqO07jxWutfMbQ==", 2024-03-24T01:27:09.991 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:27:09.991 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-24T01:27:09.991 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-24T01:27:09.991 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:27:09.991 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:27:09.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:27:09.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:27:09.992 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.3", 2024-03-24T01:27:09.992 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQCCgP9lTZ8YNxAA375ZLJHv6H2CJghmcqfe+A==", 2024-03-24T01:27:09.992 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:27:09.992 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-24T01:27:09.992 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-24T01:27:09.992 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:27:09.992 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:27:09.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:27:09.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:27:09.992 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.4", 2024-03-24T01:27:09.992 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQCKgP9ln1YuBBAAYCNFJGs3fApnhwnY/VYHhw==", 2024-03-24T01:27:09.992 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:27:09.992 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-24T01:27:09.992 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-24T01:27:09.993 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:27:09.993 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:27:09.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:27:09.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:27:09.993 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.5", 2024-03-24T01:27:09.993 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQCNgP9l329zHBAA7VtZVd0XKEK5dzhyzcPFjQ==", 2024-03-24T01:27:09.993 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:27:09.993 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-24T01:27:09.993 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-24T01:27:09.993 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:27:09.993 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:27:09.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:27:09.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:27:09.993 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.6", 2024-03-24T01:27:09.994 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQCQgP9lIv32NBAAZN6sDvBDKUZGB+TnYHO9jA==", 2024-03-24T01:27:09.994 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:27:09.994 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-24T01:27:09.994 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-24T01:27:09.994 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:27:09.994 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:27:09.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:27:09.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:27:09.994 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.7", 2024-03-24T01:27:09.994 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQCUgP9lFpBTBBAAbPF02et1cVIhSfkvJz3R3w==", 2024-03-24T01:27:09.994 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:27:09.994 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-24T01:27:09.994 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-24T01:27:09.994 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:27:09.994 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:27:09.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:27:09.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:27:09.995 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.0", 2024-03-24T01:27:09.995 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQB0gP9lLKWXMxAAwrrRG8zLSylpIEuUgM7q8g==", 2024-03-24T01:27:09.995 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:27:09.995 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-24T01:27:09.995 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow r", 2024-03-24T01:27:09.995 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-24T01:27:09.995 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:27:09.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:27:09.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:27:09.995 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.1", 2024-03-24T01:27:09.995 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQB1gP9lFlgvDRAAJFhGpBldAqv7oKFr2lIXNQ==", 2024-03-24T01:27:09.995 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:27:09.995 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-24T01:27:09.996 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow r", 2024-03-24T01:27:09.996 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow rw", 2024-03-24T01:27:09.996 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow rwx" 2024-03-24T01:27:09.996 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:27:09.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:27:09.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:27:09.996 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.2", 2024-03-24T01:27:09.996 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQB1gP9lsGI+AhAAPLfzrMVwQpf8iGAfVJX1gA==", 2024-03-24T01:27:09.996 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:27:09.996 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-24T01:27:09.996 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow r", 2024-03-24T01:27:09.996 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow rw", 2024-03-24T01:27:09.996 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow rwx" 2024-03-24T01:27:09.996 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:27:09.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:27:09.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:27:09.997 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.3", 2024-03-24T01:27:09.997 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQB1gP9l1LA0GBAAp6dwwZf+25F64cERHm2/Ew==", 2024-03-24T01:27:09.997 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:27:09.997 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-24T01:27:09.997 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow r", 2024-03-24T01:27:09.997 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow rw", 2024-03-24T01:27:09.997 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow rwx" 2024-03-24T01:27:09.997 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:27:09.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:27:09.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:27:09.997 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.admin", 2024-03-24T01:27:09.997 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQBygP9lwVRtIxAAJ7NWfOKFeLoW5nl0TLtR2w==", 2024-03-24T01:27:09.997 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:27:09.997 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow *", 2024-03-24T01:27:09.997 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow *", 2024-03-24T01:27:09.998 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow *", 2024-03-24T01:27:09.998 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:27:09.998 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:27:09.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:27:09.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:27:09.998 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mgr.x", 2024-03-24T01:27:09.998 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQBzgP9l8GPtJBAAgU+pArQqmyMDPwj0w+W/Dg==", 2024-03-24T01:27:09.998 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:27:09.998 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow *", 2024-03-24T01:27:09.998 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile mgr", 2024-03-24T01:27:09.998 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:27:09.998 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:27:09.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:27:09.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:27:09.999 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mgr.y", 2024-03-24T01:27:09.999 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQBzgP9l0KzyGBAA9zVfyPbngjHtBQyp+S37iQ==", 2024-03-24T01:27:09.999 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:27:09.999 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow *", 2024-03-24T01:27:09.999 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile mgr", 2024-03-24T01:27:09.999 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:27:09.999 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:27:09.999 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:27:09.999 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-03-24T01:27:09.999 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-24T01:27:10.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.989+0000 7f1392200640 1 -- 172.21.15.90:0/1423299173 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f136c06bbd0 msgr2=0x7f136c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:10.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.989+0000 7f1392200640 1 --2- 172.21.15.90:0/1423299173 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f136c06bbd0 0x7f136c06e090 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f13780011e0 tx=0x7f1378009040 comp rx=0 tx=0).stop 2024-03-24T01:27:10.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.989+0000 7f1392200640 1 -- 172.21.15.90:0/1423299173 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f138c0d0700 msgr2=0x7f138c142970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:10.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.989+0000 7f1392200640 1 --2- 172.21.15.90:0/1423299173 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f138c0d0700 0x7f138c142970 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f137c02bee0 tx=0x7f137c0046c0 comp rx=0 tx=0).stop 2024-03-24T01:27:10.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.989+0000 7f1392200640 1 -- 172.21.15.90:0/1423299173 shutdown_connections 2024-03-24T01:27:10.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.989+0000 7f1392200640 1 --2- 172.21.15.90:0/1423299173 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f136c06bbd0 0x7f136c06e090 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:10.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.989+0000 7f1392200640 1 --2- 172.21.15.90:0/1423299173 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f138c0d0700 0x7f138c142970 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:10.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.989+0000 7f1392200640 1 --2- 172.21.15.90:0/1423299173 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f138c15bde0 0x7f138c0ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:10.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.989+0000 7f1392200640 1 --2- 172.21.15.90:0/1423299173 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f138c0b19c0 0x7f138c143510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:10.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.989+0000 7f1392200640 1 -- 172.21.15.90:0/1423299173 >> 172.21.15.90:0/1423299173 conn(0x7f138c099a70 msgr2=0x7f138c1581e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:10.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.989+0000 7f1392200640 1 -- 172.21.15.90:0/1423299173 shutdown_connections 2024-03-24T01:27:10.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:09.989+0000 7f1392200640 1 -- 172.21.15.90:0/1423299173 wait complete. 2024-03-24T01:27:10.006 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:27:10.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.199+0000 7fbf7a000640 1 Processor -- start 2024-03-24T01:27:10.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.199+0000 7fbf7a000640 1 -- start start 2024-03-24T01:27:10.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.200+0000 7fbf7a000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbf7406b7d0 0x7fbf741629e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:10.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.200+0000 7fbf7a000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fbf74162f20 0x7fbf7415f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:10.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.200+0000 7fbf7a000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf74165ed0 con 0x7fbf740af040 2024-03-24T01:27:10.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.200+0000 7fbf7a000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf741661b0 con 0x7fbf74162f20 2024-03-24T01:27:10.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.200+0000 7fbf7a000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf74166490 con 0x7fbf7406b7d0 2024-03-24T01:27:10.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.201+0000 7fbf73e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fbf74162f20 0x7fbf7415f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:10.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.201+0000 7fbf73e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fbf74162f20 0x7fbf7415f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58722/0 (socket says 172.21.15.90:58722) 2024-03-24T01:27:10.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.201+0000 7fbf73e00640 1 -- 172.21.15.90:0/1084093327 learned_addr learned my addr 172.21.15.90:0/1084093327 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:10.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.201+0000 7fbf73e00640 1 -- 172.21.15.90:0/1084093327 >> v2:172.21.15.120:3300/0 conn(0x7fbf7406b7d0 msgr2=0x7fbf741629e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:10.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.201+0000 7fbf73e00640 1 --2- 172.21.15.90:0/1084093327 >> v2:172.21.15.120:3300/0 conn(0x7fbf7406b7d0 0x7fbf741629e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:10.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.201+0000 7fbf73e00640 1 -- 172.21.15.90:0/1084093327 >> v1:172.21.15.120:6789/0 conn(0x7fbf740af040 legacy=0x7fbf7406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:10.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.201+0000 7fbf73e00640 1 -- 172.21.15.90:0/1084093327 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf74166770 con 0x7fbf74162f20 2024-03-24T01:27:10.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.202+0000 7fbf73e00640 1 --2- 172.21.15.90:0/1084093327 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf74162f20 0x7fbf7415f2a0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fbf74072b50 tx=0x7fbf6c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4827a33623fb2038 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:10.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.202+0000 7fbf72a00640 1 -- 172.21.15.90:0/1084093327 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c02ca80 con 0x7fbf74162f20 2024-03-24T01:27:10.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.202+0000 7fbf72a00640 1 -- 172.21.15.90:0/1084093327 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6c02cc20 con 0x7fbf74162f20 2024-03-24T01:27:10.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.202+0000 7fbf72a00640 1 -- 172.21.15.90:0/1084093327 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c0343f0 con 0x7fbf74162f20 2024-03-24T01:27:10.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.203+0000 7fbf7a000640 1 -- 172.21.15.90:0/1084093327 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf74162f20 msgr2=0x7fbf7415f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:10.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.203+0000 7fbf7a000640 1 --2- 172.21.15.90:0/1084093327 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf74162f20 0x7fbf7415f2a0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fbf74072b50 tx=0x7fbf6c02bee0 comp rx=0 tx=0).stop 2024-03-24T01:27:10.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.203+0000 7fbf7a000640 1 -- 172.21.15.90:0/1084093327 shutdown_connections 2024-03-24T01:27:10.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.203+0000 7fbf7a000640 1 --2- 172.21.15.90:0/1084093327 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf74162f20 0x7fbf7415f2a0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:10.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.203+0000 7fbf7a000640 1 --2- 172.21.15.90:0/1084093327 >> v2:172.21.15.120:3300/0 conn(0x7fbf7406b7d0 0x7fbf741629e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:10.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.203+0000 7fbf7a000640 1 -- 172.21.15.90:0/1084093327 >> 172.21.15.90:0/1084093327 conn(0x7fbf740b1770 msgr2=0x7fbf740b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:10.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.203+0000 7fbf7a000640 1 -- 172.21.15.90:0/1084093327 shutdown_connections 2024-03-24T01:27:10.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.203+0000 7fbf7a000640 1 -- 172.21.15.90:0/1084093327 wait complete. 2024-03-24T01:27:10.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.205+0000 7fbf7a000640 1 Processor -- start 2024-03-24T01:27:10.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.205+0000 7fbf7a000640 1 -- start start 2024-03-24T01:27:10.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.206+0000 7fbf7a000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf7406b7d0 0x7fbf74161a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:10.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.206+0000 7fbf7a000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf740af040 0x7fbf740ce500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:10.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.206+0000 7fbf7a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf74162f20 0x7fbf74150470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:10.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.206+0000 7fbf7a000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7415f960 con 0x7fbf74162f20 2024-03-24T01:27:10.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.206+0000 7fbf7a000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7415f7e0 con 0x7fbf7406b7d0 2024-03-24T01:27:10.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.206+0000 7fbf7a000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7415fae0 con 0x7fbf740af040 2024-03-24T01:27:10.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.206+0000 7fbf73e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf74162f20 0x7fbf74150470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:10.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.206+0000 7fbf73e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf74162f20 0x7fbf74150470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58736/0 (socket says 172.21.15.90:58736) 2024-03-24T01:27:10.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.206+0000 7fbf73e00640 1 -- 172.21.15.90:0/2418302537 learned_addr learned my addr 172.21.15.90:0/2418302537 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:10.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.207+0000 7fbf73e00640 1 -- 172.21.15.90:0/2418302537 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf740af040 msgr2=0x7fbf740ce500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:10.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.207+0000 7fbf6aa00640 1 --2- 172.21.15.90:0/2418302537 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf740af040 0x7fbf740ce500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:10.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.207+0000 7fbf73e00640 1 --2- 172.21.15.90:0/2418302537 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf740af040 0x7fbf740ce500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:10.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.207+0000 7fbf73400640 1 --2- 172.21.15.90:0/2418302537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf7406b7d0 0x7fbf74161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:10.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.207+0000 7fbf73e00640 1 -- 172.21.15.90:0/2418302537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf7406b7d0 msgr2=0x7fbf74161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:10.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.207+0000 7fbf73e00640 1 --2- 172.21.15.90:0/2418302537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf7406b7d0 0x7fbf74161a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:10.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.207+0000 7fbf73e00640 1 -- 172.21.15.90:0/2418302537 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf740cdd60 con 0x7fbf74162f20 2024-03-24T01:27:10.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.207+0000 7fbf73e00640 1 --2- 172.21.15.90:0/2418302537 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf74162f20 0x7fbf74150470 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c036040 tx=0x7fbf6c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:10.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.207+0000 7fbf6be00640 1 -- 172.21.15.90:0/2418302537 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c043020 con 0x7fbf74162f20 2024-03-24T01:27:10.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.207+0000 7fbf6be00640 1 -- 172.21.15.90:0/2418302537 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6c03e450 con 0x7fbf74162f20 2024-03-24T01:27:10.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.207+0000 7fbf7a000640 1 -- 172.21.15.90:0/2418302537 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf740cccc0 con 0x7fbf74162f20 2024-03-24T01:27:10.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.207+0000 7fbf6be00640 1 -- 172.21.15.90:0/2418302537 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c03e820 con 0x7fbf74162f20 2024-03-24T01:27:10.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.208+0000 7fbf7a000640 1 -- 172.21.15.90:0/2418302537 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf740cd560 con 0x7fbf74162f20 2024-03-24T01:27:10.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.208+0000 7fbf68c00640 1 -- 172.21.15.90:0/2418302537 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf74068060 con 0x7fbf74162f20 2024-03-24T01:27:10.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.210+0000 7fbf6be00640 1 -- 172.21.15.90:0/2418302537 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fbf6c046020 con 0x7fbf74162f20 2024-03-24T01:27:10.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.211+0000 7fbf6be00640 1 --2- 172.21.15.90:0/2418302537 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbf5006bbd0 0x7fbf5006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:10.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.211+0000 7fbf6be00640 1 -- 172.21.15.90:0/2418302537 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fbf6c0ad230 con 0x7fbf74162f20 2024-03-24T01:27:10.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.212+0000 7fbf73400640 1 --2- 172.21.15.90:0/2418302537 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbf5006bbd0 0x7fbf5006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:10.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.213+0000 7fbf73400640 1 --2- 172.21.15.90:0/2418302537 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbf5006bbd0 0x7fbf5006e090 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fbf640015d0 tx=0x7fbf64009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:10.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.216+0000 7fbf6be00640 1 -- 172.21.15.90:0/2418302537 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fbf6c04d030 con 0x7fbf74162f20 2024-03-24T01:27:10.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.411+0000 7fbf68c00640 1 -- 172.21.15.90:0/2418302537 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf7415f300 con 0x7fbf74162f20 2024-03-24T01:27:10.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.413+0000 7fbf6be00640 1 -- 172.21.15.90:0/2418302537 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v49) v1 ==== 74+0+13140 (secure 0 0 0) 0x7fbf740cdd60 con 0x7fbf74162f20 2024-03-24T01:27:10.414 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:27:10.414 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":49,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:27:09.480107+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:27:10.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.415+0000 7fbf68c00640 1 -- 172.21.15.90:0/2418302537 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbf5006bbd0 msgr2=0x7fbf5006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:10.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.415+0000 7fbf68c00640 1 --2- 172.21.15.90:0/2418302537 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbf5006bbd0 0x7fbf5006e090 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fbf640015d0 tx=0x7fbf64009040 comp rx=0 tx=0).stop 2024-03-24T01:27:10.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.415+0000 7fbf68c00640 1 -- 172.21.15.90:0/2418302537 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf74162f20 msgr2=0x7fbf74150470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:10.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.415+0000 7fbf68c00640 1 --2- 172.21.15.90:0/2418302537 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf74162f20 0x7fbf74150470 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c036040 tx=0x7fbf6c004a40 comp rx=0 tx=0).stop 2024-03-24T01:27:10.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.416+0000 7fbf68c00640 1 -- 172.21.15.90:0/2418302537 shutdown_connections 2024-03-24T01:27:10.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.416+0000 7fbf68c00640 1 --2- 172.21.15.90:0/2418302537 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbf5006bbd0 0x7fbf5006e090 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:10.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.416+0000 7fbf68c00640 1 --2- 172.21.15.90:0/2418302537 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf74162f20 0x7fbf74150470 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:10.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.416+0000 7fbf68c00640 1 --2- 172.21.15.90:0/2418302537 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbf740af040 0x7fbf740ce500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:10.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.416+0000 7fbf68c00640 1 --2- 172.21.15.90:0/2418302537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf7406b7d0 0x7fbf74161a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:10.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.416+0000 7fbf68c00640 1 -- 172.21.15.90:0/2418302537 >> 172.21.15.90:0/2418302537 conn(0x7fbf740b1770 msgr2=0x7fbf740720e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:10.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.416+0000 7fbf68c00640 1 -- 172.21.15.90:0/2418302537 shutdown_connections 2024-03-24T01:27:10.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.416+0000 7fbf68c00640 1 -- 172.21.15.90:0/2418302537 wait complete. 2024-03-24T01:27:10.433 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T01:27:10.433 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:27:10.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.645+0000 7f1805000640 1 Processor -- start 2024-03-24T01:27:10.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.645+0000 7f1805000640 1 -- start start 2024-03-24T01:27:10.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.646+0000 7f1805000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1800159620 0x7f180006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:10.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.646+0000 7f1805000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f18000cf150 0x7f18000ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:10.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.646+0000 7f1805000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f180006bca0 con 0x7f180006b600 2024-03-24T01:27:10.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.646+0000 7f1805000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18000b4470 con 0x7f18000cf150 2024-03-24T01:27:10.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.646+0000 7f1805000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18000b36f0 con 0x7f1800159620 2024-03-24T01:27:10.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.647+0000 7f17fe800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f18000cf150 0x7f18000ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:10.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.647+0000 7f17f5400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f180006b600 0x7f18000cea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60256/0 (socket says 172.21.15.90:60256) 2024-03-24T01:27:10.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.647+0000 7f17f5400640 1 -- 172.21.15.90:0/3938566286 learned_addr learned my addr 172.21.15.90:0/3938566286 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:27:10.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.647+0000 7f17fde00640 1 --2- 172.21.15.90:0/3938566286 >> v2:172.21.15.120:3300/0 conn(0x7f1800159620 0x7f180006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:10.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.647+0000 7f17fe800640 1 -- 172.21.15.90:0/3938566286 >> v2:172.21.15.120:3300/0 conn(0x7f1800159620 msgr2=0x7f180006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:10.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.647+0000 7f17fe800640 1 --2- 172.21.15.90:0/3938566286 >> v2:172.21.15.120:3300/0 conn(0x7f1800159620 0x7f180006b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:10.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.647+0000 7f17fe800640 1 -- 172.21.15.90:0/3938566286 >> v1:172.21.15.90:6789/0 conn(0x7f180006b600 legacy=0x7f18000cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:10.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.647+0000 7f17fe800640 1 -- 172.21.15.90:0/3938566286 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18000b3990 con 0x7f18000cf150 2024-03-24T01:27:10.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.647+0000 7f17fd400640 1 -- 172.21.15.90:0/3938566286 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 615323596 0 0) 0x7f180006bca0 con 0x7f180006b600 2024-03-24T01:27:10.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.648+0000 7f17fe800640 1 --2- 172.21.15.90:0/3938566286 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18000cf150 0x7f18000ae530 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f180006c2d0 tx=0x7f17f802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b15b4bc8fc46a13 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:10.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.648+0000 7f17fd400640 1 -- 172.21.15.90:0/3938566286 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f8039070 con 0x7f18000cf150 2024-03-24T01:27:10.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.648+0000 7f17fd400640 1 -- 172.21.15.90:0/3938566286 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17f802cbf0 con 0x7f18000cf150 2024-03-24T01:27:10.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.648+0000 7f17fd400640 1 -- 172.21.15.90:0/3938566286 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f8034440 con 0x7f18000cf150 2024-03-24T01:27:10.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.649+0000 7f1805000640 1 -- 172.21.15.90:0/3938566286 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18000cf150 msgr2=0x7f18000ae530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:10.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.649+0000 7f1805000640 1 --2- 172.21.15.90:0/3938566286 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18000cf150 0x7f18000ae530 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f180006c2d0 tx=0x7f17f802be40 comp rx=0 tx=0).stop 2024-03-24T01:27:10.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.649+0000 7f1805000640 1 -- 172.21.15.90:0/3938566286 shutdown_connections 2024-03-24T01:27:10.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.649+0000 7f1805000640 1 --2- 172.21.15.90:0/3938566286 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18000cf150 0x7f18000ae530 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:10.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.649+0000 7f1805000640 1 --2- 172.21.15.90:0/3938566286 >> v2:172.21.15.120:3300/0 conn(0x7f1800159620 0x7f180006b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:10.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.649+0000 7f1805000640 1 -- 172.21.15.90:0/3938566286 >> 172.21.15.90:0/3938566286 conn(0x7f18000b1770 msgr2=0x7f18000b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:10.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.649+0000 7f1805000640 1 -- 172.21.15.90:0/3938566286 shutdown_connections 2024-03-24T01:27:10.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.649+0000 7f1805000640 1 -- 172.21.15.90:0/3938566286 wait complete. 2024-03-24T01:27:10.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.651+0000 7f1805000640 1 Processor -- start 2024-03-24T01:27:10.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.651+0000 7f1805000640 1 -- start start 2024-03-24T01:27:10.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.652+0000 7f1805000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f180006b600 0x7f18001443e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:10.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.652+0000 7f1805000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18000cf150 0x7f18001513b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:10.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.652+0000 7f1805000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1800159620 0x7f180015d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:10.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.652+0000 7f1805000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1800159a00 con 0x7f180006b600 2024-03-24T01:27:10.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.652+0000 7f1805000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18000b3f50 con 0x7f1800159620 2024-03-24T01:27:10.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.652+0000 7f1805000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f180015bfe0 con 0x7f18000cf150 2024-03-24T01:27:10.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.652+0000 7f17fde00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f180006b600 0x7f18001443e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:10.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.652+0000 7f17fde00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f180006b600 0x7f18001443e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58754/0 (socket says 172.21.15.90:58754) 2024-03-24T01:27:10.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.652+0000 7f17fde00640 1 -- 172.21.15.90:0/2117190073 learned_addr learned my addr 172.21.15.90:0/2117190073 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:10.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.653+0000 7f17f5400640 1 --2- 172.21.15.90:0/2117190073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18000cf150 0x7f18001513b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:10.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.653+0000 7f17fde00640 1 -- 172.21.15.90:0/2117190073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18000cf150 msgr2=0x7f18001513b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:10.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.653+0000 7f17fde00640 1 --2- 172.21.15.90:0/2117190073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18000cf150 0x7f18001513b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:10.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.653+0000 7f17fde00640 1 -- 172.21.15.90:0/2117190073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1800159620 msgr2=0x7f180015d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:10.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.653+0000 7f17fde00640 1 --2- 172.21.15.90:0/2117190073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1800159620 0x7f180015d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:10.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.653+0000 7f17fde00640 1 -- 172.21.15.90:0/2117190073 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1800152aa0 con 0x7f180006b600 2024-03-24T01:27:10.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.653+0000 7f17f5400640 1 --2- 172.21.15.90:0/2117190073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18000cf150 0x7f18001513b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:27:10.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.653+0000 7f17fde00640 1 --2- 172.21.15.90:0/2117190073 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f180006b600 0x7f18001443e0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f17f000d5f0 tx=0x7f17f000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:10.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.653+0000 7f17f6a00640 1 -- 172.21.15.90:0/2117190073 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f00125f0 con 0x7f180006b600 2024-03-24T01:27:10.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.653+0000 7f17f6a00640 1 -- 172.21.15.90:0/2117190073 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17f0019070 con 0x7f180006b600 2024-03-24T01:27:10.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.653+0000 7f17f6a00640 1 -- 172.21.15.90:0/2117190073 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f00129d0 con 0x7f180006b600 2024-03-24T01:27:10.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.653+0000 7f1805000640 1 -- 172.21.15.90:0/2117190073 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1800152d80 con 0x7f180006b600 2024-03-24T01:27:10.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.653+0000 7f1805000640 1 -- 172.21.15.90:0/2117190073 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1800153340 con 0x7f180006b600 2024-03-24T01:27:10.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.655+0000 7f1805000640 1 -- 172.21.15.90:0/2117190073 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f180004f400 con 0x7f180006b600 2024-03-24T01:27:10.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.655+0000 7f17f6a00640 1 -- 172.21.15.90:0/2117190073 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f17f0012b70 con 0x7f180006b600 2024-03-24T01:27:10.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.656+0000 7f17f6a00640 1 --2- 172.21.15.90:0/2117190073 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f17dc06bbd0 0x7f17dc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:10.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.656+0000 7f17f6a00640 1 -- 172.21.15.90:0/2117190073 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f17f008ad50 con 0x7f180006b600 2024-03-24T01:27:10.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.657+0000 7f17f5400640 1 --2- 172.21.15.90:0/2117190073 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f17dc06bbd0 0x7f17dc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:10.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.658+0000 7f17f5400640 1 --2- 172.21.15.90:0/2117190073 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f17dc06bbd0 0x7f17dc06e090 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f1800068c60 tx=0x7f17e8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:10.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.660+0000 7f17f6a00640 1 -- 172.21.15.90:0/2117190073 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f17f0059960 con 0x7f180006b600 2024-03-24T01:27:10.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.857+0000 7f1805000640 1 -- 172.21.15.90:0/2117190073 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f18000b2e90 con 0x7f180006b600 2024-03-24T01:27:10.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.858+0000 7f17f6a00640 1 -- 172.21.15.90:0/2117190073 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v49) v1 ==== 93+0+10 (secure 0 0 0) 0x7f17f0059260 con 0x7f180006b600 2024-03-24T01:27:10.859 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:27:10.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.861+0000 7f1805000640 1 -- 172.21.15.90:0/2117190073 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f17dc06bbd0 msgr2=0x7f17dc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:10.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.861+0000 7f1805000640 1 --2- 172.21.15.90:0/2117190073 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f17dc06bbd0 0x7f17dc06e090 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f1800068c60 tx=0x7f17e8009040 comp rx=0 tx=0).stop 2024-03-24T01:27:10.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.861+0000 7f1805000640 1 -- 172.21.15.90:0/2117190073 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f180006b600 msgr2=0x7f18001443e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:10.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.861+0000 7f1805000640 1 --2- 172.21.15.90:0/2117190073 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f180006b600 0x7f18001443e0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f17f000d5f0 tx=0x7f17f000dab0 comp rx=0 tx=0).stop 2024-03-24T01:27:10.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.861+0000 7f1805000640 1 -- 172.21.15.90:0/2117190073 shutdown_connections 2024-03-24T01:27:10.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.861+0000 7f1805000640 1 --2- 172.21.15.90:0/2117190073 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f17dc06bbd0 0x7f17dc06e090 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:10.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.861+0000 7f1805000640 1 --2- 172.21.15.90:0/2117190073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1800159620 0x7f180015d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:10.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.861+0000 7f1805000640 1 --2- 172.21.15.90:0/2117190073 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f18000cf150 0x7f18001513b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:10.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.861+0000 7f1805000640 1 --2- 172.21.15.90:0/2117190073 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f180006b600 0x7f18001443e0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:10.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.861+0000 7f1805000640 1 -- 172.21.15.90:0/2117190073 >> 172.21.15.90:0/2117190073 conn(0x7f18000b1770 msgr2=0x7f18000cc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:10.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.861+0000 7f1805000640 1 -- 172.21.15.90:0/2117190073 shutdown_connections 2024-03-24T01:27:10.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:10.862+0000 7f1805000640 1 -- 172.21.15.90:0/2117190073 wait complete. 2024-03-24T01:27:10.880 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-24T01:27:11.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.070+0000 7f014fe00640 1 Processor -- start 2024-03-24T01:27:11.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.071+0000 7f014fe00640 1 -- start start 2024-03-24T01:27:11.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.072+0000 7f014fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0150144420 0x7f015006d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:11.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.072+0000 7f014fe00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0150144800 0x7f01500cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:11.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.072+0000 7f014fe00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01500cceb0 con 0x7f01500cc8f0 2024-03-24T01:27:11.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.072+0000 7f014fe00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01500cd190 con 0x7f0150144420 2024-03-24T01:27:11.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.072+0000 7f014fe00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01500ceb70 con 0x7f0150144800 2024-03-24T01:27:11.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.072+0000 7f014ea00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0150144420 0x7f015006d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:11.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.072+0000 7f014ea00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0150144420 0x7f015006d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58768/0 (socket says 172.21.15.90:58768) 2024-03-24T01:27:11.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.072+0000 7f014ea00640 1 -- 172.21.15.90:0/1902651664 learned_addr learned my addr 172.21.15.90:0/1902651664 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:11.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.073+0000 7f0146000640 1 --2- 172.21.15.90:0/1902651664 >> v2:172.21.15.120:3300/0 conn(0x7f0150144800 0x7f01500cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:11.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.073+0000 7f014ea00640 1 -- 172.21.15.90:0/1902651664 >> v2:172.21.15.120:3300/0 conn(0x7f0150144800 msgr2=0x7f01500cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:11.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.073+0000 7f014ea00640 1 --2- 172.21.15.90:0/1902651664 >> v2:172.21.15.120:3300/0 conn(0x7f0150144800 0x7f01500cf370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:11.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.073+0000 7f014ea00640 1 -- 172.21.15.90:0/1902651664 >> v1:172.21.15.120:6789/0 conn(0x7f01500cc8f0 legacy=0x7f01500cf8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:11.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.073+0000 7f014ea00640 1 -- 172.21.15.90:0/1902651664 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01500cee50 con 0x7f0150144420 2024-03-24T01:27:11.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.074+0000 7f014ea00640 1 --2- 172.21.15.90:0/1902651664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0150144420 0x7f015006d900 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f015006bf50 tx=0x7f014002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5a0f0263b6c9aa9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:11.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.075+0000 7f014e000640 1 -- 172.21.15.90:0/1902651664 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014002e070 con 0x7f0150144420 2024-03-24T01:27:11.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.075+0000 7f014e000640 1 -- 172.21.15.90:0/1902651664 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0140037070 con 0x7f0150144420 2024-03-24T01:27:11.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.075+0000 7f014e000640 1 -- 172.21.15.90:0/1902651664 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0140031a00 con 0x7f0150144420 2024-03-24T01:27:11.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.075+0000 7f014fe00640 1 -- 172.21.15.90:0/1902651664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0150144420 msgr2=0x7f015006d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:11.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.075+0000 7f014fe00640 1 --2- 172.21.15.90:0/1902651664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0150144420 0x7f015006d900 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f015006bf50 tx=0x7f014002b880 comp rx=0 tx=0).stop 2024-03-24T01:27:11.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.075+0000 7f014fe00640 1 -- 172.21.15.90:0/1902651664 shutdown_connections 2024-03-24T01:27:11.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.075+0000 7f014fe00640 1 --2- 172.21.15.90:0/1902651664 >> v2:172.21.15.120:3300/0 conn(0x7f0150144800 0x7f01500cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:11.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.075+0000 7f014fe00640 1 --2- 172.21.15.90:0/1902651664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0150144420 0x7f015006d900 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:11.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.075+0000 7f014fe00640 1 -- 172.21.15.90:0/1902651664 >> 172.21.15.90:0/1902651664 conn(0x7f015006b0c0 msgr2=0x7f015006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:11.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.075+0000 7f014fe00640 1 -- 172.21.15.90:0/1902651664 shutdown_connections 2024-03-24T01:27:11.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.075+0000 7f014fe00640 1 -- 172.21.15.90:0/1902651664 wait complete. 2024-03-24T01:27:11.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.077+0000 7f014fe00640 1 Processor -- start 2024-03-24T01:27:11.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.077+0000 7f014fe00640 1 -- start start 2024-03-24T01:27:11.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.078+0000 7f014fe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01500cc8f0 0x7f015015b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:11.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.078+0000 7f014fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0150144420 0x7f01500b1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:11.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.078+0000 7f014fe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0150144800 0x7f0150164050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:11.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.078+0000 7f014fe00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01500ce910 con 0x7f0150144420 2024-03-24T01:27:11.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.078+0000 7f014fe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01500ce790 con 0x7f0150144800 2024-03-24T01:27:11.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.078+0000 7f014fe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f015015f540 con 0x7f01500cc8f0 2024-03-24T01:27:11.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.078+0000 7f0146000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0150144420 0x7f01500b1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:11.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.078+0000 7f0146000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0150144420 0x7f01500b1240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58774/0 (socket says 172.21.15.90:58774) 2024-03-24T01:27:11.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.078+0000 7f0146000640 1 -- 172.21.15.90:0/146037846 learned_addr learned my addr 172.21.15.90:0/146037846 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:11.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.079+0000 7f014ea00640 1 --2- 172.21.15.90:0/146037846 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01500cc8f0 0x7f015015b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:11.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.079+0000 7f0146000640 1 -- 172.21.15.90:0/146037846 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01500cc8f0 msgr2=0x7f015015b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:11.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.079+0000 7f0146000640 1 --2- 172.21.15.90:0/146037846 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01500cc8f0 0x7f015015b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:11.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.079+0000 7f0146000640 1 -- 172.21.15.90:0/146037846 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0150144800 msgr2=0x7f0150164050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:11.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.079+0000 7f014f400640 1 --2- 172.21.15.90:0/146037846 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0150144800 0x7f0150164050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:11.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.079+0000 7f0146000640 1 --2- 172.21.15.90:0/146037846 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0150144800 0x7f0150164050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:11.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.079+0000 7f0146000640 1 -- 172.21.15.90:0/146037846 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01501617b0 con 0x7f0150144420 2024-03-24T01:27:11.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.079+0000 7f014ea00640 1 --2- 172.21.15.90:0/146037846 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01500cc8f0 0x7f015015b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:27:11.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.079+0000 7f0146000640 1 --2- 172.21.15.90:0/146037846 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0150144420 0x7f01500b1240 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f013800c820 tx=0x7f013800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:11.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.079+0000 7f0147400640 1 -- 172.21.15.90:0/146037846 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0138015070 con 0x7f0150144420 2024-03-24T01:27:11.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.079+0000 7f0147400640 1 -- 172.21.15.90:0/146037846 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0138004020 con 0x7f0150144420 2024-03-24T01:27:11.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.079+0000 7f014fe00640 1 -- 172.21.15.90:0/146037846 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0150161a90 con 0x7f0150144420 2024-03-24T01:27:11.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.079+0000 7f0147400640 1 -- 172.21.15.90:0/146037846 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01380043f0 con 0x7f0150144420 2024-03-24T01:27:11.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.080+0000 7f014fe00640 1 -- 172.21.15.90:0/146037846 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0150162050 con 0x7f0150144420 2024-03-24T01:27:11.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.081+0000 7f014fe00640 1 -- 172.21.15.90:0/146037846 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f015004f400 con 0x7f0150144420 2024-03-24T01:27:11.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.082+0000 7f0147400640 1 -- 172.21.15.90:0/146037846 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0138007830 con 0x7f0150144420 2024-03-24T01:27:11.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.082+0000 7f0147400640 1 --2- 172.21.15.90:0/146037846 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f012c06bbd0 0x7f012c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:11.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.082+0000 7f0147400640 1 -- 172.21.15.90:0/146037846 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f013808a740 con 0x7f0150144420 2024-03-24T01:27:11.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.083+0000 7f014ea00640 1 --2- 172.21.15.90:0/146037846 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f012c06bbd0 0x7f012c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:11.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.084+0000 7f014ea00640 1 --2- 172.21.15.90:0/146037846 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f012c06bbd0 0x7f012c06e090 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0140036000 tx=0x7f0140008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:11.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.087+0000 7f0147400640 1 -- 172.21.15.90:0/146037846 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0138059350 con 0x7f0150144420 2024-03-24T01:27:11.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.275+0000 7f014fe00640 1 -- 172.21.15.90:0/146037846 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f015006b640 con 0x7f0150144420 2024-03-24T01:27:11.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.276+0000 7f0147400640 1 -- 172.21.15.90:0/146037846 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v49) v1 ==== 84+0+65 (secure 0 0 0) 0x7f013808a030 con 0x7f0150144420 2024-03-24T01:27:11.276 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:27:11.276 INFO:teuthology.orchestra.run.smithi090.stdout:[ 2024-03-24T01:27:11.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:27:11.277 INFO:teuthology.orchestra.run.smithi090.stdout: "poolnum": 1, 2024-03-24T01:27:11.277 INFO:teuthology.orchestra.run.smithi090.stdout: "poolname": ".mgr" 2024-03-24T01:27:11.277 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:27:11.277 INFO:teuthology.orchestra.run.smithi090.stdout:] 2024-03-24T01:27:11.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.278+0000 7f014fe00640 1 -- 172.21.15.90:0/146037846 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f012c06bbd0 msgr2=0x7f012c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:11.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.278+0000 7f014fe00640 1 --2- 172.21.15.90:0/146037846 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f012c06bbd0 0x7f012c06e090 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0140036000 tx=0x7f0140008030 comp rx=0 tx=0).stop 2024-03-24T01:27:11.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.278+0000 7f014fe00640 1 -- 172.21.15.90:0/146037846 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0150144420 msgr2=0x7f01500b1240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:11.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.278+0000 7f014fe00640 1 --2- 172.21.15.90:0/146037846 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0150144420 0x7f01500b1240 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f013800c820 tx=0x7f013800cce0 comp rx=0 tx=0).stop 2024-03-24T01:27:11.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.278+0000 7f014fe00640 1 -- 172.21.15.90:0/146037846 shutdown_connections 2024-03-24T01:27:11.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.278+0000 7f014fe00640 1 --2- 172.21.15.90:0/146037846 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f012c06bbd0 0x7f012c06e090 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:11.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.278+0000 7f014fe00640 1 --2- 172.21.15.90:0/146037846 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0150144420 0x7f01500b1240 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:11.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.278+0000 7f014fe00640 1 --2- 172.21.15.90:0/146037846 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0150144800 0x7f0150164050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:11.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.278+0000 7f014fe00640 1 --2- 172.21.15.90:0/146037846 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f01500cc8f0 0x7f015015b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:11.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.279+0000 7f014fe00640 1 -- 172.21.15.90:0/146037846 >> 172.21.15.90:0/146037846 conn(0x7f015006b0c0 msgr2=0x7f01500b2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:11.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.279+0000 7f014fe00640 1 -- 172.21.15.90:0/146037846 shutdown_connections 2024-03-24T01:27:11.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.279+0000 7f014fe00640 1 -- 172.21.15.90:0/146037846 wait complete. 2024-03-24T01:27:11.297 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-24T01:27:11.298 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-24T01:27:11.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.509+0000 7fd3ede00640 1 Processor -- start 2024-03-24T01:27:11.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.510+0000 7fd3ede00640 1 -- start start 2024-03-24T01:27:11.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.511+0000 7fd3ede00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3e80aee80 0x7fd3e80b33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:11.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.511+0000 7fd3ede00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd3e806b0c0 0x7fd3e806b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:11.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.511+0000 7fd3ede00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3e80b3eb0 con 0x7fd3e80b3930 2024-03-24T01:27:11.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.511+0000 7fd3ede00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3e806e680 con 0x7fd3e806b0c0 2024-03-24T01:27:11.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.511+0000 7fd3ede00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e806e960 con 0x7fd3e80aee80 2024-03-24T01:27:11.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.511+0000 7fd3e6000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd3e80b3930 0x7fd3e80cc8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60264/0 (socket says 172.21.15.90:60264) 2024-03-24T01:27:11.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.511+0000 7fd3e6000640 1 -- 172.21.15.90:0/822597552 learned_addr learned my addr 172.21.15.90:0/822597552 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:27:11.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.512+0000 7fd3e6a00640 1 --2- 172.21.15.90:0/822597552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3e80aee80 0x7fd3e80b33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:11.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.512+0000 7fd3e7400640 1 --2- 172.21.15.90:0/822597552 >> v2:172.21.15.120:3300/0 conn(0x7fd3e806b0c0 0x7fd3e806b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:11.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.512+0000 7fd3e5600640 1 -- 172.21.15.90:0/822597552 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3262959377 0 0) 0x7fd3e80b3eb0 con 0x7fd3e80b3930 2024-03-24T01:27:11.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.512+0000 7fd3e5600640 1 -- 172.21.15.90:0/822597552 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3cc003520 con 0x7fd3e80b3930 2024-03-24T01:27:11.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.512+0000 7fd3e5600640 1 -- 172.21.15.90:0/822597552 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2966300819 0 0) 0x7fd3cc003520 con 0x7fd3e80b3930 2024-03-24T01:27:11.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.512+0000 7fd3e5600640 1 -- 172.21.15.90:0/822597552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3e80aee80 msgr2=0x7fd3e80b33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:11.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.512+0000 7fd3e5600640 1 --2- 172.21.15.90:0/822597552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3e80aee80 0x7fd3e80b33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:11.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.512+0000 7fd3e5600640 1 -- 172.21.15.90:0/822597552 >> v2:172.21.15.120:3300/0 conn(0x7fd3e806b0c0 msgr2=0x7fd3e806b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:11.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.512+0000 7fd3e5600640 1 --2- 172.21.15.90:0/822597552 >> v2:172.21.15.120:3300/0 conn(0x7fd3e806b0c0 0x7fd3e806b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:11.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.512+0000 7fd3e5600640 1 -- 172.21.15.90:0/822597552 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e806ec40 con 0x7fd3e80b3930 2024-03-24T01:27:11.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.512+0000 7fd3e5600640 1 -- 172.21.15.90:0/822597552 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7fd3d0001670 con 0x7fd3e80b3930 2024-03-24T01:27:11.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.512+0000 7fd3e5600640 1 -- 172.21.15.90:0/822597552 >> v1:172.21.15.90:6789/0 conn(0x7fd3e80b3930 legacy=0x7fd3e80cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:11.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.512+0000 7fd3e5600640 1 --2- 172.21.15.90:0/822597552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3cc0039b0 0x7fd3cc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:11.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.513+0000 7fd3e5600640 1 --2- 172.21.15.90:0/822597552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3e80aee80 0x7fd3cc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:11.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.513+0000 7fd3e5600640 1 --2- 172.21.15.90:0/822597552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3cc004d40 0x7fd3cc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:11.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.513+0000 7fd3e5600640 1 -- 172.21.15.90:0/822597552 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e806e680 con 0x7fd3cc004d40 2024-03-24T01:27:11.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.513+0000 7fd3e5600640 1 -- 172.21.15.90:0/822597552 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e806e960 con 0x7fd3cc0039b0 2024-03-24T01:27:11.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.513+0000 7fd3e5600640 1 -- 172.21.15.90:0/822597552 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3cc00baa0 con 0x7fd3e80aee80 2024-03-24T01:27:11.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.513+0000 7fd3e7400640 1 --2- 172.21.15.90:0/822597552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3cc004d40 0x7fd3cc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:11.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.513+0000 7fd3e6a00640 1 --2- 172.21.15.90:0/822597552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3cc0039b0 0x7fd3cc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:11.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.513+0000 7fd3e6000640 1 --2- 172.21.15.90:0/822597552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3e80aee80 0x7fd3cc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:11.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.513+0000 7fd3e7400640 1 -- 172.21.15.90:0/822597552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3e80aee80 msgr2=0x7fd3cc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:11.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.513+0000 7fd3e7400640 1 --2- 172.21.15.90:0/822597552 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3e80aee80 0x7fd3cc004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:11.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.513+0000 7fd3e7400640 1 -- 172.21.15.90:0/822597552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3cc0039b0 msgr2=0x7fd3cc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:11.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.513+0000 7fd3e7400640 1 --2- 172.21.15.90:0/822597552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3cc0039b0 0x7fd3cc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:11.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.513+0000 7fd3e7400640 1 -- 172.21.15.90:0/822597552 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3cc003520 con 0x7fd3cc004d40 2024-03-24T01:27:11.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.513+0000 7fd3e6a00640 1 --2- 172.21.15.90:0/822597552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3cc0039b0 0x7fd3cc003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:27:11.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.514+0000 7fd3e7400640 1 --2- 172.21.15.90:0/822597552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3cc004d40 0x7fd3cc0051c0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fd3e0007360 tx=0x7fd3cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2a0032843e20bbf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:11.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.515+0000 7fd3e5600640 1 -- 172.21.15.90:0/822597552 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e0039070 con 0x7fd3cc004d40 2024-03-24T01:27:11.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.515+0000 7fd3e5600640 1 -- 172.21.15.90:0/822597552 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3e002ccf0 con 0x7fd3cc004d40 2024-03-24T01:27:11.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.515+0000 7fd3e5600640 1 -- 172.21.15.90:0/822597552 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e0009be0 con 0x7fd3cc004d40 2024-03-24T01:27:11.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.515+0000 7fd3ede00640 1 -- 172.21.15.90:0/822597552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3cc004d40 msgr2=0x7fd3cc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:11.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.515+0000 7fd3ede00640 1 --2- 172.21.15.90:0/822597552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3cc004d40 0x7fd3cc0051c0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fd3e0007360 tx=0x7fd3cc005830 comp rx=0 tx=0).stop 2024-03-24T01:27:11.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.515+0000 7fd3e7400640 1 -- 172.21.15.90:0/822597552 reap_dead start 2024-03-24T01:27:11.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.515+0000 7fd3ede00640 1 -- 172.21.15.90:0/822597552 shutdown_connections 2024-03-24T01:27:11.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.515+0000 7fd3ede00640 1 -- 172.21.15.90:0/822597552 >> 172.21.15.90:0/822597552 conn(0x7fd3e80b1770 msgr2=0x7fd3e80b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:11.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.515+0000 7fd3ede00640 1 -- 172.21.15.90:0/822597552 shutdown_connections 2024-03-24T01:27:11.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.515+0000 7fd3ede00640 1 -- 172.21.15.90:0/822597552 wait complete. 2024-03-24T01:27:11.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.517+0000 7fd3ede00640 1 Processor -- start 2024-03-24T01:27:11.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.517+0000 7fd3ede00640 1 -- start start 2024-03-24T01:27:11.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.518+0000 7fd3ede00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3cc004d40 0x7fd3e815f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:11.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.518+0000 7fd3ede00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3e80ad160 0x7fd3e80a8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:11.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.518+0000 7fd3ede00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3e8164230 0x7fd3e80ad830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:11.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.518+0000 7fd3ede00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e80ccfb0 con 0x7fd3e80ad160 2024-03-24T01:27:11.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.518+0000 7fd3ede00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e8166ad0 con 0x7fd3cc004d40 2024-03-24T01:27:11.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.518+0000 7fd3ede00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e80cd130 con 0x7fd3e8164230 2024-03-24T01:27:11.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.518+0000 7fd3e6000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3e80ad160 0x7fd3e80a8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:11.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.518+0000 7fd3e6000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3e80ad160 0x7fd3e80a8ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58804/0 (socket says 172.21.15.90:58804) 2024-03-24T01:27:11.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.519+0000 7fd3e6000640 1 -- 172.21.15.90:0/3730890106 learned_addr learned my addr 172.21.15.90:0/3730890106 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:11.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.519+0000 7fd3e6000640 1 -- 172.21.15.90:0/3730890106 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3e8164230 msgr2=0x7fd3e80ad830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:11.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.519+0000 7fd3e6000640 1 --2- 172.21.15.90:0/3730890106 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3e8164230 0x7fd3e80ad830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:11.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.519+0000 7fd3e6000640 1 -- 172.21.15.90:0/3730890106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3cc004d40 msgr2=0x7fd3e815f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:11.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.519+0000 7fd3e6000640 1 --2- 172.21.15.90:0/3730890106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3cc004d40 0x7fd3e815f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:11.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.519+0000 7fd3e6000640 1 -- 172.21.15.90:0/3730890106 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e80a9420 con 0x7fd3e80ad160 2024-03-24T01:27:11.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.519+0000 7fd3e6000640 1 --2- 172.21.15.90:0/3730890106 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3e80ad160 0x7fd3e80a8ee0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd3d000d840 tx=0x7fd3d000dd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:11.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.519+0000 7fd3df400640 1 -- 172.21.15.90:0/3730890106 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d0019070 con 0x7fd3e80ad160 2024-03-24T01:27:11.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.519+0000 7fd3df400640 1 -- 172.21.15.90:0/3730890106 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3d00074e0 con 0x7fd3e80ad160 2024-03-24T01:27:11.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.519+0000 7fd3df400640 1 -- 172.21.15.90:0/3730890106 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d0004a20 con 0x7fd3e80ad160 2024-03-24T01:27:11.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.519+0000 7fd3ede00640 1 -- 172.21.15.90:0/3730890106 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3e80ac3a0 con 0x7fd3e80ad160 2024-03-24T01:27:11.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.520+0000 7fd3ede00640 1 -- 172.21.15.90:0/3730890106 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3e80ac960 con 0x7fd3e80ad160 2024-03-24T01:27:11.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.521+0000 7fd3ede00640 1 -- 172.21.15.90:0/3730890106 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3e804f400 con 0x7fd3e80ad160 2024-03-24T01:27:11.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.522+0000 7fd3df400640 1 -- 172.21.15.90:0/3730890106 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd3d0004bc0 con 0x7fd3e80ad160 2024-03-24T01:27:11.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.522+0000 7fd3df400640 1 --2- 172.21.15.90:0/3730890106 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd3c406bbd0 0x7fd3c406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:11.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.523+0000 7fd3df400640 1 -- 172.21.15.90:0/3730890106 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fd3d008a2e0 con 0x7fd3e80ad160 2024-03-24T01:27:11.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.523+0000 7fd3e6a00640 1 --2- 172.21.15.90:0/3730890106 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd3c406bbd0 0x7fd3c406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:11.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.524+0000 7fd3e6a00640 1 --2- 172.21.15.90:0/3730890106 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd3c406bbd0 0x7fd3c406e090 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8009a00 tx=0x7fd3d8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:11.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.527+0000 7fd3df400640 1 -- 172.21.15.90:0/3730890106 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd3e80a9420 con 0x7fd3e80ad160 2024-03-24T01:27:11.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:11.723+0000 7fd3ede00640 1 -- 172.21.15.90:0/3730890106 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd3e806e250 con 0x7fd3e80ad160 2024-03-24T01:27:12.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.484+0000 7fd3df400640 1 -- 172.21.15.90:0/3730890106 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v50) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd3d0058e60 con 0x7fd3e80ad160 2024-03-24T01:27:12.485 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_metadata' created 2024-03-24T01:27:12.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.488+0000 7fd3ede00640 1 -- 172.21.15.90:0/3730890106 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd3c406bbd0 msgr2=0x7fd3c406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:12.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.488+0000 7fd3ede00640 1 --2- 172.21.15.90:0/3730890106 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd3c406bbd0 0x7fd3c406e090 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8009a00 tx=0x7fd3d8008040 comp rx=0 tx=0).stop 2024-03-24T01:27:12.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.488+0000 7fd3ede00640 1 -- 172.21.15.90:0/3730890106 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3e80ad160 msgr2=0x7fd3e80a8ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:12.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.488+0000 7fd3ede00640 1 --2- 172.21.15.90:0/3730890106 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3e80ad160 0x7fd3e80a8ee0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd3d000d840 tx=0x7fd3d000dd00 comp rx=0 tx=0).stop 2024-03-24T01:27:12.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.488+0000 7fd3ede00640 1 -- 172.21.15.90:0/3730890106 shutdown_connections 2024-03-24T01:27:12.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.488+0000 7fd3ede00640 1 --2- 172.21.15.90:0/3730890106 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd3c406bbd0 0x7fd3c406e090 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:12.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.488+0000 7fd3ede00640 1 --2- 172.21.15.90:0/3730890106 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3e80ad160 0x7fd3e80a8ee0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:12.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.488+0000 7fd3ede00640 1 --2- 172.21.15.90:0/3730890106 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3e8164230 0x7fd3e80ad830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:12.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.488+0000 7fd3ede00640 1 --2- 172.21.15.90:0/3730890106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3cc004d40 0x7fd3e815f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:12.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.488+0000 7fd3ede00640 1 -- 172.21.15.90:0/3730890106 >> 172.21.15.90:0/3730890106 conn(0x7fd3e80b1770 msgr2=0x7fd3e8160550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:12.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.489+0000 7fd3ede00640 1 -- 172.21.15.90:0/3730890106 shutdown_connections 2024-03-24T01:27:12.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.489+0000 7fd3ede00640 1 -- 172.21.15.90:0/3730890106 wait complete. 2024-03-24T01:27:12.517 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-24T01:27:12.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.721+0000 7fcc60a00640 1 Processor -- start 2024-03-24T01:27:12.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.721+0000 7fcc60a00640 1 -- start start 2024-03-24T01:27:12.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.723+0000 7fcc60a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc5c159d90 0x7fcc5c0b2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:12.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.723+0000 7fcc60a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcc5c0ae960 0x7fcc5c071990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:12.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.723+0000 7fcc60a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcc5c071ed0 0x7fcc5c15d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:12.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.723+0000 7fcc60a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc5c0af2f0 con 0x7fcc5c0ae960 2024-03-24T01:27:12.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.723+0000 7fcc60a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc5c15a170 con 0x7fcc5c071ed0 2024-03-24T01:27:12.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.723+0000 7fcc60a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5c0b2f70 con 0x7fcc5c159d90 2024-03-24T01:27:12.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.723+0000 7fcc59400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcc5c0ae960 0x7fcc5c071990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:12.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.723+0000 7fcc59400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcc5c0ae960 0x7fcc5c071990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58816/0 (socket says 172.21.15.90:58816) 2024-03-24T01:27:12.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.723+0000 7fcc59400640 1 -- 172.21.15.90:0/475009780 learned_addr learned my addr 172.21.15.90:0/475009780 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:12.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.724+0000 7fcc5a800640 1 --2- 172.21.15.90:0/475009780 >> v2:172.21.15.120:3300/0 conn(0x7fcc5c071ed0 0x7fcc5c15d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:12.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.724+0000 7fcc59e00640 1 --2- 172.21.15.90:0/475009780 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc5c159d90 0x7fcc5c0b2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:12.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.724+0000 7fcc59400640 1 -- 172.21.15.90:0/475009780 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc5c159d90 msgr2=0x7fcc5c0b2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:12.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.724+0000 7fcc59400640 1 --2- 172.21.15.90:0/475009780 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc5c159d90 0x7fcc5c0b2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:12.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.724+0000 7fcc59400640 1 -- 172.21.15.90:0/475009780 >> v2:172.21.15.120:3300/0 conn(0x7fcc5c071ed0 msgr2=0x7fcc5c15d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:12.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.724+0000 7fcc59400640 1 --2- 172.21.15.90:0/475009780 >> v2:172.21.15.120:3300/0 conn(0x7fcc5c071ed0 0x7fcc5c15d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:12.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.724+0000 7fcc59400640 1 -- 172.21.15.90:0/475009780 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc5c06da90 con 0x7fcc5c0ae960 2024-03-24T01:27:12.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.725+0000 7fcc59400640 1 --2- 172.21.15.90:0/475009780 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc5c0ae960 0x7fcc5c071990 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c071940 tx=0x7fcc4802b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=633e0dc470fcf4b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:12.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.725+0000 7fcc58a00640 1 -- 172.21.15.90:0/475009780 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc480317d0 con 0x7fcc5c0ae960 2024-03-24T01:27:12.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.725+0000 7fcc58a00640 1 -- 172.21.15.90:0/475009780 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc48037070 con 0x7fcc5c0ae960 2024-03-24T01:27:12.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.725+0000 7fcc58a00640 1 -- 172.21.15.90:0/475009780 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc48031b90 con 0x7fcc5c0ae960 2024-03-24T01:27:12.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.726+0000 7fcc60a00640 1 -- 172.21.15.90:0/475009780 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc5c0ae960 msgr2=0x7fcc5c071990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:12.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.726+0000 7fcc60a00640 1 --2- 172.21.15.90:0/475009780 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc5c0ae960 0x7fcc5c071990 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c071940 tx=0x7fcc4802b870 comp rx=0 tx=0).stop 2024-03-24T01:27:12.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.726+0000 7fcc60a00640 1 -- 172.21.15.90:0/475009780 shutdown_connections 2024-03-24T01:27:12.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.726+0000 7fcc60a00640 1 --2- 172.21.15.90:0/475009780 >> v2:172.21.15.120:3300/0 conn(0x7fcc5c071ed0 0x7fcc5c15d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:12.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.726+0000 7fcc60a00640 1 --2- 172.21.15.90:0/475009780 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc5c0ae960 0x7fcc5c071990 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:12.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.726+0000 7fcc60a00640 1 --2- 172.21.15.90:0/475009780 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc5c159d90 0x7fcc5c0b2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:12.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.726+0000 7fcc60a00640 1 -- 172.21.15.90:0/475009780 >> 172.21.15.90:0/475009780 conn(0x7fcc5c0cf7e0 msgr2=0x7fcc5c0cfbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:12.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.726+0000 7fcc60a00640 1 -- 172.21.15.90:0/475009780 shutdown_connections 2024-03-24T01:27:12.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.726+0000 7fcc60a00640 1 -- 172.21.15.90:0/475009780 wait complete. 2024-03-24T01:27:12.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.729+0000 7fcc60a00640 1 Processor -- start 2024-03-24T01:27:12.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.729+0000 7fcc60a00640 1 -- start start 2024-03-24T01:27:12.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.730+0000 7fcc60a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc5c071ed0 0x7fcc5c151520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:12.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.730+0000 7fcc60a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc5c0ae960 0x7fcc5c15ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:12.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.730+0000 7fcc60a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc5c159d90 0x7fcc5c15b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:12.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.730+0000 7fcc60a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5c06e330 con 0x7fcc5c159d90 2024-03-24T01:27:12.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.730+0000 7fcc60a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5c06e050 con 0x7fcc5c071ed0 2024-03-24T01:27:12.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.730+0000 7fcc60a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5c0ccd60 con 0x7fcc5c0ae960 2024-03-24T01:27:12.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.730+0000 7fcc5a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc5c159d90 0x7fcc5c15b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:12.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.730+0000 7fcc5a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc5c159d90 0x7fcc5c15b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58818/0 (socket says 172.21.15.90:58818) 2024-03-24T01:27:12.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.730+0000 7fcc5a800640 1 -- 172.21.15.90:0/754800149 learned_addr learned my addr 172.21.15.90:0/754800149 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:12.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.731+0000 7fcc59e00640 1 --2- 172.21.15.90:0/754800149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc5c071ed0 0x7fcc5c151520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:12.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.731+0000 7fcc59400640 1 --2- 172.21.15.90:0/754800149 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc5c0ae960 0x7fcc5c15ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:12.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.731+0000 7fcc5a800640 1 -- 172.21.15.90:0/754800149 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc5c0ae960 msgr2=0x7fcc5c15ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:12.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.731+0000 7fcc5a800640 1 --2- 172.21.15.90:0/754800149 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc5c0ae960 0x7fcc5c15ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:12.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.731+0000 7fcc5a800640 1 -- 172.21.15.90:0/754800149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc5c071ed0 msgr2=0x7fcc5c151520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:12.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.731+0000 7fcc5a800640 1 --2- 172.21.15.90:0/754800149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc5c071ed0 0x7fcc5c151520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:12.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.731+0000 7fcc5a800640 1 -- 172.21.15.90:0/754800149 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc5c154810 con 0x7fcc5c159d90 2024-03-24T01:27:12.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.731+0000 7fcc59e00640 1 --2- 172.21.15.90:0/754800149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc5c071ed0 0x7fcc5c151520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:27:12.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.731+0000 7fcc5a800640 1 --2- 172.21.15.90:0/754800149 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc5c159d90 0x7fcc5c15b2e0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fcc54007d80 tx=0x7fcc5400d490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:12.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.731+0000 7fcc52000640 1 -- 172.21.15.90:0/754800149 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc54019070 con 0x7fcc5c159d90 2024-03-24T01:27:12.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.731+0000 7fcc52000640 1 -- 172.21.15.90:0/754800149 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc5400dda0 con 0x7fcc5c159d90 2024-03-24T01:27:12.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.732+0000 7fcc52000640 1 -- 172.21.15.90:0/754800149 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc54003bd0 con 0x7fcc5c159d90 2024-03-24T01:27:12.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.732+0000 7fcc60a00640 1 -- 172.21.15.90:0/754800149 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc5c154ab0 con 0x7fcc5c159d90 2024-03-24T01:27:12.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.732+0000 7fcc60a00640 1 -- 172.21.15.90:0/754800149 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc5c155070 con 0x7fcc5c159d90 2024-03-24T01:27:12.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.733+0000 7fcc60a00640 1 -- 172.21.15.90:0/754800149 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc5c155730 con 0x7fcc5c159d90 2024-03-24T01:27:12.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.734+0000 7fcc52000640 1 -- 172.21.15.90:0/754800149 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcc5401c070 con 0x7fcc5c159d90 2024-03-24T01:27:12.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.736+0000 7fcc52000640 1 --2- 172.21.15.90:0/754800149 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcc3c06bc00 0x7fcc3c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:12.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.736+0000 7fcc52000640 1 -- 172.21.15.90:0/754800149 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5619+0+0 (secure 0 0 0) 0x7fcc5408aa90 con 0x7fcc5c159d90 2024-03-24T01:27:12.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.736+0000 7fcc59e00640 1 --2- 172.21.15.90:0/754800149 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcc3c06bc00 0x7fcc3c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:12.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.737+0000 7fcc59e00640 1 --2- 172.21.15.90:0/754800149 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcc3c06bc00 0x7fcc3c06e0c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c00b970 tx=0x7fcc4c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:12.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.741+0000 7fcc52000640 1 -- 172.21.15.90:0/754800149 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcc5c155730 con 0x7fcc5c159d90 2024-03-24T01:27:12.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:12.940+0000 7fcc60a00640 1 -- 172.21.15.90:0/754800149 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fcc5c04f400 con 0x7fcc5c159d90 2024-03-24T01:27:13.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.491+0000 7fcc52000640 1 -- 172.21.15.90:0/754800149 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v51) v1 ==== 170+0+0 (secure 0 0 0) 0x7fcc540595c0 con 0x7fcc5c159d90 2024-03-24T01:27:13.491 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_data' created 2024-03-24T01:27:13.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.494+0000 7fcc60a00640 1 -- 172.21.15.90:0/754800149 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcc3c06bc00 msgr2=0x7fcc3c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:13.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.494+0000 7fcc60a00640 1 --2- 172.21.15.90:0/754800149 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcc3c06bc00 0x7fcc3c06e0c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c00b970 tx=0x7fcc4c008040 comp rx=0 tx=0).stop 2024-03-24T01:27:13.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.494+0000 7fcc60a00640 1 -- 172.21.15.90:0/754800149 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc5c159d90 msgr2=0x7fcc5c15b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:13.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.494+0000 7fcc60a00640 1 --2- 172.21.15.90:0/754800149 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc5c159d90 0x7fcc5c15b2e0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fcc54007d80 tx=0x7fcc5400d490 comp rx=0 tx=0).stop 2024-03-24T01:27:13.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.495+0000 7fcc60a00640 1 -- 172.21.15.90:0/754800149 shutdown_connections 2024-03-24T01:27:13.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.495+0000 7fcc60a00640 1 --2- 172.21.15.90:0/754800149 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcc3c06bc00 0x7fcc3c06e0c0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:13.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.495+0000 7fcc60a00640 1 --2- 172.21.15.90:0/754800149 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc5c159d90 0x7fcc5c15b2e0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:13.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.495+0000 7fcc60a00640 1 --2- 172.21.15.90:0/754800149 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc5c0ae960 0x7fcc5c15ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:13.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.495+0000 7fcc60a00640 1 --2- 172.21.15.90:0/754800149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc5c071ed0 0x7fcc5c151520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:13.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.495+0000 7fcc60a00640 1 -- 172.21.15.90:0/754800149 >> 172.21.15.90:0/754800149 conn(0x7fcc5c0cf7e0 msgr2=0x7fcc5c153460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:13.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.495+0000 7fcc60a00640 1 -- 172.21.15.90:0/754800149 shutdown_connections 2024-03-24T01:27:13.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.495+0000 7fcc60a00640 1 -- 172.21.15.90:0/754800149 wait complete. 2024-03-24T01:27:13.528 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-24T01:27:13.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.707+0000 7f7d61600640 1 Processor -- start 2024-03-24T01:27:13.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.707+0000 7f7d61600640 1 -- start start 2024-03-24T01:27:13.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.708+0000 7f7d61600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7d5c06f4e0 0x7f7d5c071990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:13.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.708+0000 7f7d61600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f7d5c071ed0 0x7f7d5c072320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:13.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.708+0000 7f7d61600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d5c0b2930 con 0x7f7d5c15a1d0 2024-03-24T01:27:13.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.708+0000 7f7d61600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d5c0b39d0 con 0x7f7d5c06f4e0 2024-03-24T01:27:13.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.708+0000 7f7d61600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d5c0b2c10 con 0x7f7d5c071ed0 2024-03-24T01:27:13.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.709+0000 7f7d52000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7d5c06f4e0 0x7f7d5c071990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:13.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.709+0000 7f7d52000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7d5c06f4e0 0x7f7d5c071990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58820/0 (socket says 172.21.15.90:58820) 2024-03-24T01:27:13.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.709+0000 7f7d52000640 1 -- 172.21.15.90:0/40299552 learned_addr learned my addr 172.21.15.90:0/40299552 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:13.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.709+0000 7f7d5aa00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f7d5c15a1d0 0x7f7d5c06edd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60268/0 (socket says 172.21.15.90:60268) 2024-03-24T01:27:13.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.709+0000 7f7d5b400640 1 --2- 172.21.15.90:0/40299552 >> v2:172.21.15.120:3300/0 conn(0x7f7d5c071ed0 0x7f7d5c072320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:13.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.709+0000 7f7d52000640 1 -- 172.21.15.90:0/40299552 >> v2:172.21.15.120:3300/0 conn(0x7f7d5c071ed0 msgr2=0x7f7d5c072320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:13.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.709+0000 7f7d52000640 1 --2- 172.21.15.90:0/40299552 >> v2:172.21.15.120:3300/0 conn(0x7f7d5c071ed0 0x7f7d5c072320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:13.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.709+0000 7f7d52000640 1 -- 172.21.15.90:0/40299552 >> v1:172.21.15.90:6789/0 conn(0x7f7d5c15a1d0 legacy=0x7f7d5c06edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:13.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.709+0000 7f7d52000640 1 -- 172.21.15.90:0/40299552 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d5c0b2ef0 con 0x7f7d5c06f4e0 2024-03-24T01:27:13.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.709+0000 7f7d5a000640 1 -- 172.21.15.90:0/40299552 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3291141961 0 0) 0x7f7d5c0b2930 con 0x7f7d5c15a1d0 2024-03-24T01:27:13.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.710+0000 7f7d52000640 1 --2- 172.21.15.90:0/40299552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d5c06f4e0 0x7f7d5c071990 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c0af430 tx=0x7f7d4402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=344d645eb8bc5969 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:13.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.711+0000 7f7d5a000640 1 -- 172.21.15.90:0/40299552 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4402cb20 con 0x7f7d5c06f4e0 2024-03-24T01:27:13.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.711+0000 7f7d5a000640 1 -- 172.21.15.90:0/40299552 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d4402ccc0 con 0x7f7d5c06f4e0 2024-03-24T01:27:13.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.711+0000 7f7d5a000640 1 -- 172.21.15.90:0/40299552 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d44034610 con 0x7f7d5c06f4e0 2024-03-24T01:27:13.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.711+0000 7f7d61600640 1 -- 172.21.15.90:0/40299552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d5c06f4e0 msgr2=0x7f7d5c071990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:13.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.711+0000 7f7d61600640 1 --2- 172.21.15.90:0/40299552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d5c06f4e0 0x7f7d5c071990 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c0af430 tx=0x7f7d4402bf80 comp rx=0 tx=0).stop 2024-03-24T01:27:13.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.711+0000 7f7d61600640 1 -- 172.21.15.90:0/40299552 shutdown_connections 2024-03-24T01:27:13.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.711+0000 7f7d61600640 1 --2- 172.21.15.90:0/40299552 >> v2:172.21.15.120:3300/0 conn(0x7f7d5c071ed0 0x7f7d5c072320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:13.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.711+0000 7f7d61600640 1 --2- 172.21.15.90:0/40299552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d5c06f4e0 0x7f7d5c071990 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:13.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.711+0000 7f7d61600640 1 -- 172.21.15.90:0/40299552 >> 172.21.15.90:0/40299552 conn(0x7f7d5c0cf7e0 msgr2=0x7f7d5c0cfbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:13.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.711+0000 7f7d61600640 1 -- 172.21.15.90:0/40299552 shutdown_connections 2024-03-24T01:27:13.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.711+0000 7f7d61600640 1 -- 172.21.15.90:0/40299552 wait complete. 2024-03-24T01:27:13.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.713+0000 7f7d61600640 1 Processor -- start 2024-03-24T01:27:13.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.713+0000 7f7d61600640 1 -- start start 2024-03-24T01:27:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.714+0000 7f7d61600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7d5c06f4e0 0x7f7d5c1441a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.714+0000 7f7d61600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7d5c071ed0 0x7f7d5c15a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.714+0000 7f7d61600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d5c15a1d0 0x7f7d5c159070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.714+0000 7f7d61600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5c15a5b0 con 0x7f7d5c15a1d0 2024-03-24T01:27:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.714+0000 7f7d61600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5c0b34b0 con 0x7f7d5c06f4e0 2024-03-24T01:27:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.714+0000 7f7d61600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5c072860 con 0x7f7d5c071ed0 2024-03-24T01:27:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.714+0000 7f7d5b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d5c15a1d0 0x7f7d5c159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.714+0000 7f7d5b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d5c15a1d0 0x7f7d5c159070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58822/0 (socket says 172.21.15.90:58822) 2024-03-24T01:27:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.714+0000 7f7d5b400640 1 -- 172.21.15.90:0/992276718 learned_addr learned my addr 172.21.15.90:0/992276718 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.715+0000 7f7d5aa00640 1 --2- 172.21.15.90:0/992276718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7d5c06f4e0 0x7f7d5c1441a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.715+0000 7f7d5b400640 1 -- 172.21.15.90:0/992276718 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7d5c071ed0 msgr2=0x7f7d5c15a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.715+0000 7f7d52000640 1 --2- 172.21.15.90:0/992276718 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7d5c071ed0 0x7f7d5c15a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:13.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.715+0000 7f7d5b400640 1 --2- 172.21.15.90:0/992276718 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7d5c071ed0 0x7f7d5c15a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:13.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.715+0000 7f7d5b400640 1 -- 172.21.15.90:0/992276718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7d5c06f4e0 msgr2=0x7f7d5c1441a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:13.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.715+0000 7f7d5b400640 1 --2- 172.21.15.90:0/992276718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7d5c06f4e0 0x7f7d5c1441a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:13.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.715+0000 7f7d5b400640 1 -- 172.21.15.90:0/992276718 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d5c15c0d0 con 0x7f7d5c15a1d0 2024-03-24T01:27:13.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.715+0000 7f7d5aa00640 1 --2- 172.21.15.90:0/992276718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7d5c06f4e0 0x7f7d5c1441a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:27:13.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.715+0000 7f7d5b400640 1 --2- 172.21.15.90:0/992276718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d5c15a1d0 0x7f7d5c159070 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f7d5400d800 tx=0x7f7d5400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:13.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.715+0000 7f7d53400640 1 -- 172.21.15.90:0/992276718 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d5400b6c0 con 0x7f7d5c15a1d0 2024-03-24T01:27:13.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.715+0000 7f7d53400640 1 -- 172.21.15.90:0/992276718 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d54019070 con 0x7f7d5c15a1d0 2024-03-24T01:27:13.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.715+0000 7f7d53400640 1 -- 172.21.15.90:0/992276718 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d5400bac0 con 0x7f7d5c15a1d0 2024-03-24T01:27:13.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.715+0000 7f7d61600640 1 -- 172.21.15.90:0/992276718 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d5c15b030 con 0x7f7d5c15a1d0 2024-03-24T01:27:13.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.715+0000 7f7d61600640 1 -- 172.21.15.90:0/992276718 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d5c15b5f0 con 0x7f7d5c15a1d0 2024-03-24T01:27:13.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.717+0000 7f7d61600640 1 -- 172.21.15.90:0/992276718 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d5c04f400 con 0x7f7d5c15a1d0 2024-03-24T01:27:13.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.718+0000 7f7d53400640 1 -- 172.21.15.90:0/992276718 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7d5400bc60 con 0x7f7d5c15a1d0 2024-03-24T01:27:13.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.719+0000 7f7d53400640 1 --2- 172.21.15.90:0/992276718 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f7d3806bbd0 0x7f7d3806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:13.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.719+0000 7f7d53400640 1 -- 172.21.15.90:0/992276718 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f7d5408b0c0 con 0x7f7d5c15a1d0 2024-03-24T01:27:13.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.720+0000 7f7d5aa00640 1 --2- 172.21.15.90:0/992276718 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f7d3806bbd0 0x7f7d3806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:13.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.721+0000 7f7d5aa00640 1 --2- 172.21.15.90:0/992276718 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f7d3806bbd0 0x7f7d3806e090 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c000f60 tx=0x7f7d4c00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:13.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.724+0000 7f7d53400640 1 -- 172.21.15.90:0/992276718 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7d540599e0 con 0x7f7d5c15a1d0 2024-03-24T01:27:13.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:13.959+0000 7f7d61600640 1 -- 172.21.15.90:0/992276718 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f7d5c06edd0 con 0x7f7d5c15a1d0 2024-03-24T01:27:14.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.533+0000 7f7d53400640 1 -- 172.21.15.90:0/992276718 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 7 and data pool 8 v30) v1 ==== 172+0+0 (secure 0 0 0) 0x7f7d540592e0 con 0x7f7d5c15a1d0 2024-03-24T01:27:14.534 INFO:teuthology.orchestra.run.smithi090.stderr:new fs with metadata pool 7 and data pool 8 2024-03-24T01:27:14.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.543+0000 7f7d61600640 1 -- 172.21.15.90:0/992276718 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f7d3806bbd0 msgr2=0x7f7d3806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:14.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.543+0000 7f7d61600640 1 --2- 172.21.15.90:0/992276718 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f7d3806bbd0 0x7f7d3806e090 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c000f60 tx=0x7f7d4c00b040 comp rx=0 tx=0).stop 2024-03-24T01:27:14.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.543+0000 7f7d61600640 1 -- 172.21.15.90:0/992276718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d5c15a1d0 msgr2=0x7f7d5c159070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:14.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.543+0000 7f7d61600640 1 --2- 172.21.15.90:0/992276718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d5c15a1d0 0x7f7d5c159070 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f7d5400d800 tx=0x7f7d5400dcc0 comp rx=0 tx=0).stop 2024-03-24T01:27:14.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.543+0000 7f7d61600640 1 -- 172.21.15.90:0/992276718 shutdown_connections 2024-03-24T01:27:14.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.543+0000 7f7d61600640 1 --2- 172.21.15.90:0/992276718 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f7d3806bbd0 0x7f7d3806e090 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:14.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.543+0000 7f7d61600640 1 --2- 172.21.15.90:0/992276718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d5c15a1d0 0x7f7d5c159070 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:14.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.543+0000 7f7d61600640 1 --2- 172.21.15.90:0/992276718 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7d5c071ed0 0x7f7d5c15a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:14.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.543+0000 7f7d61600640 1 --2- 172.21.15.90:0/992276718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7d5c06f4e0 0x7f7d5c1441a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:14.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.543+0000 7f7d61600640 1 -- 172.21.15.90:0/992276718 >> 172.21.15.90:0/992276718 conn(0x7f7d5c0cf7e0 msgr2=0x7f7d5c0cfbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:14.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.543+0000 7f7d61600640 1 -- 172.21.15.90:0/992276718 shutdown_connections 2024-03-24T01:27:14.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.545+0000 7f7d61600640 1 -- 172.21.15.90:0/992276718 wait complete. 2024-03-24T01:27:14.578 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:27:14.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.749+0000 7f61b9a00640 1 Processor -- start 2024-03-24T01:27:14.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.749+0000 7f61b9a00640 1 -- start start 2024-03-24T01:27:14.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.750+0000 7f61b9a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f61b406b7d0 0x7f61b4166a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:14.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.750+0000 7f61b9a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b40cd490 con 0x7f61b40cc150 2024-03-24T01:27:14.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.750+0000 7f61b9a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b415f900 con 0x7f61b41603e0 2024-03-24T01:27:14.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.750+0000 7f61b9a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61b40cd270 con 0x7f61b406b7d0 2024-03-24T01:27:14.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.750+0000 7f61b2a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f61b40cc150 0x7f61b40ccb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60278/0 (socket says 172.21.15.90:60278) 2024-03-24T01:27:14.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.751+0000 7f61b2a00640 1 -- 172.21.15.90:0/471013083 learned_addr learned my addr 172.21.15.90:0/471013083 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:27:14.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.751+0000 7f61b3400640 1 --2- 172.21.15.90:0/471013083 >> v2:172.21.15.90:3300/0 conn(0x7f61b406b7d0 0x7f61b4166a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:14.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.751+0000 7f61b1600640 1 -- 172.21.15.90:0/471013083 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2633788852 0 0) 0x7f61b40cd490 con 0x7f61b40cc150 2024-03-24T01:27:14.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.751+0000 7f61b1600640 1 -- 172.21.15.90:0/471013083 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6198003520 con 0x7f61b40cc150 2024-03-24T01:27:14.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.751+0000 7f61b3400640 1 -- 172.21.15.90:0/471013083 >> v1:172.21.15.120:6789/0 conn(0x7f61b41603e0 legacy=0x7f61b406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:14.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.751+0000 7f61b3400640 1 -- 172.21.15.90:0/471013083 >> v1:172.21.15.90:6789/0 conn(0x7f61b40cc150 legacy=0x7f61b40ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:14.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.751+0000 7f61b3400640 1 -- 172.21.15.90:0/471013083 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b415fbe0 con 0x7f61b406b7d0 2024-03-24T01:27:14.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.752+0000 7f61b3400640 1 --2- 172.21.15.90:0/471013083 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61b406b7d0 0x7f61b4166a80 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f61ac009d90 tx=0x7f61ac02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dfb1e8f8b3ca129 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:14.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.753+0000 7f61b1600640 1 -- 172.21.15.90:0/471013083 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ac039070 con 0x7f61b406b7d0 2024-03-24T01:27:14.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.753+0000 7f61b1600640 1 -- 172.21.15.90:0/471013083 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61ac02cc60 con 0x7f61b406b7d0 2024-03-24T01:27:14.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.753+0000 7f61b1600640 1 -- 172.21.15.90:0/471013083 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ac034660 con 0x7f61b406b7d0 2024-03-24T01:27:14.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.753+0000 7f61b9a00640 1 -- 172.21.15.90:0/471013083 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61b406b7d0 msgr2=0x7f61b4166a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:14.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.753+0000 7f61b9a00640 1 --2- 172.21.15.90:0/471013083 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61b406b7d0 0x7f61b4166a80 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f61ac009d90 tx=0x7f61ac02beb0 comp rx=0 tx=0).stop 2024-03-24T01:27:14.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.753+0000 7f61b9a00640 1 -- 172.21.15.90:0/471013083 shutdown_connections 2024-03-24T01:27:14.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.753+0000 7f61b9a00640 1 --2- 172.21.15.90:0/471013083 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61b406b7d0 0x7f61b4166a80 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:14.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.753+0000 7f61b9a00640 1 -- 172.21.15.90:0/471013083 >> 172.21.15.90:0/471013083 conn(0x7f61b40b1770 msgr2=0x7f61b40b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:14.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.753+0000 7f61b9a00640 1 -- 172.21.15.90:0/471013083 shutdown_connections 2024-03-24T01:27:14.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.754+0000 7f61b9a00640 1 -- 172.21.15.90:0/471013083 wait complete. 2024-03-24T01:27:14.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.755+0000 7f61b9a00640 1 Processor -- start 2024-03-24T01:27:14.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.755+0000 7f61b9a00640 1 -- start start 2024-03-24T01:27:14.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.756+0000 7f61b9a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61b406b7d0 0x7f61b4136270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:14.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.756+0000 7f61b9a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61b40cc150 0x7f61b413a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:14.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.756+0000 7f61b9a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61b41603e0 0x7f61b4136990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:14.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.756+0000 7f61b9a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b41455b0 con 0x7f61b406b7d0 2024-03-24T01:27:14.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.756+0000 7f61b9a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b4145430 con 0x7f61b40cc150 2024-03-24T01:27:14.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.756+0000 7f61b9a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b4166fc0 con 0x7f61b41603e0 2024-03-24T01:27:14.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.756+0000 7f61b2a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61b406b7d0 0x7f61b4136270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:14.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.756+0000 7f61b2a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61b406b7d0 0x7f61b4136270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58846/0 (socket says 172.21.15.90:58846) 2024-03-24T01:27:14.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.756+0000 7f61b2a00640 1 -- 172.21.15.90:0/3254343419 learned_addr learned my addr 172.21.15.90:0/3254343419 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:14.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.757+0000 7f61b2a00640 1 -- 172.21.15.90:0/3254343419 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61b41603e0 msgr2=0x7f61b4136990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:14.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.757+0000 7f61b3400640 1 --2- 172.21.15.90:0/3254343419 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61b41603e0 0x7f61b4136990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:14.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.757+0000 7f61b2000640 1 --2- 172.21.15.90:0/3254343419 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61b40cc150 0x7f61b413a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:14.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.757+0000 7f61b2a00640 1 --2- 172.21.15.90:0/3254343419 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61b41603e0 0x7f61b4136990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:14.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.757+0000 7f61b2a00640 1 -- 172.21.15.90:0/3254343419 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61b40cc150 msgr2=0x7f61b413a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:14.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.757+0000 7f61b2a00640 1 --2- 172.21.15.90:0/3254343419 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61b40cc150 0x7f61b413a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:14.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.757+0000 7f61b2a00640 1 -- 172.21.15.90:0/3254343419 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b413b8c0 con 0x7f61b406b7d0 2024-03-24T01:27:14.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.757+0000 7f61b2a00640 1 --2- 172.21.15.90:0/3254343419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61b406b7d0 0x7f61b4136270 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f61a400c430 tx=0x7f61a400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:14.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.757+0000 7f61aaa00640 1 -- 172.21.15.90:0/3254343419 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a4017070 con 0x7f61b406b7d0 2024-03-24T01:27:14.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.758+0000 7f61aaa00640 1 -- 172.21.15.90:0/3254343419 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a4005ce0 con 0x7f61b406b7d0 2024-03-24T01:27:14.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.758+0000 7f61b9a00640 1 -- 172.21.15.90:0/3254343419 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61b413b300 con 0x7f61b406b7d0 2024-03-24T01:27:14.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.758+0000 7f61aaa00640 1 -- 172.21.15.90:0/3254343419 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a40072f0 con 0x7f61b406b7d0 2024-03-24T01:27:14.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.758+0000 7f61b9a00640 1 -- 172.21.15.90:0/3254343419 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61b415cc70 con 0x7f61b406b7d0 2024-03-24T01:27:14.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.759+0000 7f61b9a00640 1 -- 172.21.15.90:0/3254343419 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61b4068060 con 0x7f61b406b7d0 2024-03-24T01:27:14.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.760+0000 7f61aaa00640 1 -- 172.21.15.90:0/3254343419 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f61a401a070 con 0x7f61b406b7d0 2024-03-24T01:27:14.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.760+0000 7f61aaa00640 1 --2- 172.21.15.90:0/3254343419 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f619406bc00 0x7f619406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:14.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.761+0000 7f61aaa00640 1 -- 172.21.15.90:0/3254343419 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f61a408a0a0 con 0x7f61b406b7d0 2024-03-24T01:27:14.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.761+0000 7f61b2000640 1 --2- 172.21.15.90:0/3254343419 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f619406bc00 0x7f619406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:14.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.762+0000 7f61b2000640 1 --2- 172.21.15.90:0/3254343419 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f619406bc00 0x7f619406e0c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f61b4136940 tx=0x7f61a0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:14.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.765+0000 7f61aaa00640 1 -- 172.21.15.90:0/3254343419 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f61a40589e0 con 0x7f61b406b7d0 2024-03-24T01:27:14.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.948+0000 7f61b9a00640 1 -- 172.21.15.90:0/3254343419 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61b41405d0 con 0x7f61b406b7d0 2024-03-24T01:27:14.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.951+0000 7f61aaa00640 1 -- 172.21.15.90:0/3254343419 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+16931 (secure 0 0 0) 0x7f61a40582e0 con 0x7f61b406b7d0 2024-03-24T01:27:14.952 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:27:14.952 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":52,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:27:13.961036+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:27:11.774313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-24T01:27:12.992083+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:27:14.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.954+0000 7f61b9a00640 1 -- 172.21.15.90:0/3254343419 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f619406bc00 msgr2=0x7f619406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:14.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.954+0000 7f61b9a00640 1 --2- 172.21.15.90:0/3254343419 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f619406bc00 0x7f619406e0c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f61b4136940 tx=0x7f61a0008040 comp rx=0 tx=0).stop 2024-03-24T01:27:14.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.954+0000 7f61b9a00640 1 -- 172.21.15.90:0/3254343419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61b406b7d0 msgr2=0x7f61b4136270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:14.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.954+0000 7f61b9a00640 1 --2- 172.21.15.90:0/3254343419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61b406b7d0 0x7f61b4136270 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f61a400c430 tx=0x7f61a400c8f0 comp rx=0 tx=0).stop 2024-03-24T01:27:14.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.954+0000 7f61b9a00640 1 -- 172.21.15.90:0/3254343419 shutdown_connections 2024-03-24T01:27:14.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.954+0000 7f61b9a00640 1 --2- 172.21.15.90:0/3254343419 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f619406bc00 0x7f619406e0c0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:14.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.954+0000 7f61b9a00640 1 --2- 172.21.15.90:0/3254343419 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61b41603e0 0x7f61b4136990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:14.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.954+0000 7f61b9a00640 1 --2- 172.21.15.90:0/3254343419 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61b40cc150 0x7f61b413a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:14.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.954+0000 7f61b9a00640 1 --2- 172.21.15.90:0/3254343419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61b406b7d0 0x7f61b4136270 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:14.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.954+0000 7f61b9a00640 1 -- 172.21.15.90:0/3254343419 >> 172.21.15.90:0/3254343419 conn(0x7f61b40b1770 msgr2=0x7f61b406b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:14.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.954+0000 7f61b9a00640 1 -- 172.21.15.90:0/3254343419 shutdown_connections 2024-03-24T01:27:14.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:14.954+0000 7f61b9a00640 1 -- 172.21.15.90:0/3254343419 wait complete. 2024-03-24T01:27:14.972 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:27:15.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.178+0000 7f8a5fe00640 1 Processor -- start 2024-03-24T01:27:15.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.178+0000 7f8a5fe00640 1 -- start start 2024-03-24T01:27:15.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.179+0000 7f8a5fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8a6015dba0 0x7f8a60162330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:15.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.179+0000 7f8a5fe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a600b5c40 0x7f8a600d1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:15.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.179+0000 7f8a5fe00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a600ce4b0 con 0x7f8a6015df80 2024-03-24T01:27:15.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.179+0000 7f8a5fe00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a60162870 con 0x7f8a6015dba0 2024-03-24T01:27:15.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.179+0000 7f8a5fe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a600ce790 con 0x7f8a600b5c40 2024-03-24T01:27:15.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.180+0000 7f8a5ea00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8a6015dba0 0x7f8a60162330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:15.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.180+0000 7f8a5ea00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8a6015dba0 0x7f8a60162330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58862/0 (socket says 172.21.15.90:58862) 2024-03-24T01:27:15.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.180+0000 7f8a5ea00640 1 -- 172.21.15.90:0/2418469875 learned_addr learned my addr 172.21.15.90:0/2418469875 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:15.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.180+0000 7f8a5f400640 1 --2- 172.21.15.90:0/2418469875 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a600b5c40 0x7f8a600d1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:15.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.180+0000 7f8a5ea00640 1 -- 172.21.15.90:0/2418469875 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a600b5c40 msgr2=0x7f8a600d1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:15.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.180+0000 7f8a5ea00640 1 --2- 172.21.15.90:0/2418469875 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a600b5c40 0x7f8a600d1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:15.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.180+0000 7f8a5ea00640 1 -- 172.21.15.90:0/2418469875 >> v1:172.21.15.120:6789/0 conn(0x7f8a6015df80 legacy=0x7f8a600b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:15.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.180+0000 7f8a5ea00640 1 -- 172.21.15.90:0/2418469875 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a600cea70 con 0x7f8a6015dba0 2024-03-24T01:27:15.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.180+0000 7f8a5f400640 1 --2- 172.21.15.90:0/2418469875 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a600b5c40 0x7f8a600d1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:27:15.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.182+0000 7f8a5ea00640 1 --2- 172.21.15.90:0/2418469875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a6015dba0 0x7f8a60162330 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f8a50009870 tx=0x7f8a5002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bea5de49fcb4f023 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:15.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.182+0000 7f8a5d600640 1 -- 172.21.15.90:0/2418469875 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a500317c0 con 0x7f8a6015dba0 2024-03-24T01:27:15.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.182+0000 7f8a5d600640 1 -- 172.21.15.90:0/2418469875 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a50037070 con 0x7f8a6015dba0 2024-03-24T01:27:15.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.182+0000 7f8a5d600640 1 -- 172.21.15.90:0/2418469875 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a50031b80 con 0x7f8a6015dba0 2024-03-24T01:27:15.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.182+0000 7f8a5fe00640 1 -- 172.21.15.90:0/2418469875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a6015dba0 msgr2=0x7f8a60162330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:15.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.182+0000 7f8a5fe00640 1 --2- 172.21.15.90:0/2418469875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a6015dba0 0x7f8a60162330 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f8a50009870 tx=0x7f8a5002b860 comp rx=0 tx=0).stop 2024-03-24T01:27:15.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.183+0000 7f8a5fe00640 1 -- 172.21.15.90:0/2418469875 shutdown_connections 2024-03-24T01:27:15.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.183+0000 7f8a5fe00640 1 --2- 172.21.15.90:0/2418469875 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a600b5c40 0x7f8a600d1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:15.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.183+0000 7f8a5fe00640 1 --2- 172.21.15.90:0/2418469875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a6015dba0 0x7f8a60162330 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:15.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.183+0000 7f8a5fe00640 1 -- 172.21.15.90:0/2418469875 >> 172.21.15.90:0/2418469875 conn(0x7f8a600d0530 msgr2=0x7f8a600d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:15.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.183+0000 7f8a5fe00640 1 -- 172.21.15.90:0/2418469875 shutdown_connections 2024-03-24T01:27:15.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.183+0000 7f8a5fe00640 1 -- 172.21.15.90:0/2418469875 wait complete. 2024-03-24T01:27:15.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.184+0000 7f8a5fe00640 1 Processor -- start 2024-03-24T01:27:15.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.184+0000 7f8a5fe00640 1 -- start start 2024-03-24T01:27:15.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.185+0000 7f8a5fe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a600b5c40 0x7f8a601f6f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:15.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.185+0000 7f8a5fe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a6015dba0 0x7f8a601f0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:15.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.185+0000 7f8a5fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a6015df80 0x7f8a601f42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:15.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.185+0000 7f8a5fe00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a600b2ae0 con 0x7f8a6015df80 2024-03-24T01:27:15.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.185+0000 7f8a5fe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a600b2960 con 0x7f8a6015dba0 2024-03-24T01:27:15.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.185+0000 7f8a5fe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a600b2c60 con 0x7f8a600b5c40 2024-03-24T01:27:15.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.186+0000 7f8a5f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a6015df80 0x7f8a601f42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:15.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.186+0000 7f8a5f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a6015df80 0x7f8a601f42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58872/0 (socket says 172.21.15.90:58872) 2024-03-24T01:27:15.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.186+0000 7f8a5f400640 1 -- 172.21.15.90:0/1159388248 learned_addr learned my addr 172.21.15.90:0/1159388248 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:15.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.186+0000 7f8a5ea00640 1 --2- 172.21.15.90:0/1159388248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a600b5c40 0x7f8a601f6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:15.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.186+0000 7f8a5e000640 1 --2- 172.21.15.90:0/1159388248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a6015dba0 0x7f8a601f0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:15.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.186+0000 7f8a5f400640 1 -- 172.21.15.90:0/1159388248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a600b5c40 msgr2=0x7f8a601f6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:15.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.186+0000 7f8a5f400640 1 --2- 172.21.15.90:0/1159388248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a600b5c40 0x7f8a601f6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:15.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.186+0000 7f8a5f400640 1 -- 172.21.15.90:0/1159388248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a6015dba0 msgr2=0x7f8a601f0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:15.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.186+0000 7f8a5f400640 1 --2- 172.21.15.90:0/1159388248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a6015dba0 0x7f8a601f0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:15.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.186+0000 7f8a5f400640 1 -- 172.21.15.90:0/1159388248 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a601f77f0 con 0x7f8a6015df80 2024-03-24T01:27:15.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.186+0000 7f8a5ea00640 1 --2- 172.21.15.90:0/1159388248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a600b5c40 0x7f8a601f6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:27:15.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.186+0000 7f8a5e000640 1 --2- 172.21.15.90:0/1159388248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a6015dba0 0x7f8a601f0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:27:15.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.186+0000 7f8a5f400640 1 --2- 172.21.15.90:0/1159388248 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a6015df80 0x7f8a601f42a0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f8a5800c7f0 tx=0x7f8a5800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:15.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.187+0000 7f8a56a00640 1 -- 172.21.15.90:0/1159388248 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a58015070 con 0x7f8a6015df80 2024-03-24T01:27:15.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.187+0000 7f8a56a00640 1 -- 172.21.15.90:0/1159388248 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a58007820 con 0x7f8a6015df80 2024-03-24T01:27:15.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.187+0000 7f8a56a00640 1 -- 172.21.15.90:0/1159388248 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a58007c10 con 0x7f8a6015df80 2024-03-24T01:27:15.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.187+0000 7f8a5fe00640 1 -- 172.21.15.90:0/1159388248 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a601f7ad0 con 0x7f8a6015df80 2024-03-24T01:27:15.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.187+0000 7f8a5fe00640 1 -- 172.21.15.90:0/1159388248 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a601f3aa0 con 0x7f8a6015df80 2024-03-24T01:27:15.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.188+0000 7f8a5fe00640 1 -- 172.21.15.90:0/1159388248 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a6004f400 con 0x7f8a6015df80 2024-03-24T01:27:15.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.189+0000 7f8a56a00640 1 -- 172.21.15.90:0/1159388248 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8a58018070 con 0x7f8a6015df80 2024-03-24T01:27:15.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.191+0000 7f8a56a00640 1 --2- 172.21.15.90:0/1159388248 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8a3c06bbd0 0x7f8a3c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:15.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.191+0000 7f8a56a00640 1 -- 172.21.15.90:0/1159388248 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8a5808ac10 con 0x7f8a6015df80 2024-03-24T01:27:15.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.191+0000 7f8a5ea00640 1 --2- 172.21.15.90:0/1159388248 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8a3c06bbd0 0x7f8a3c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:15.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.192+0000 7f8a5ea00640 1 --2- 172.21.15.90:0/1159388248 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8a3c06bbd0 0x7f8a3c06e090 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8a50036000 tx=0x7f8a50008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:15.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.195+0000 7f8a56a00640 1 -- 172.21.15.90:0/1159388248 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8a580594d0 con 0x7f8a6015df80 2024-03-24T01:27:15.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.393+0000 7f8a5fe00640 1 -- 172.21.15.90:0/1159388248 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a600d17c0 con 0x7f8a6015df80 2024-03-24T01:27:15.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.395+0000 7f8a56a00640 1 -- 172.21.15.90:0/1159388248 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+16931 (secure 0 0 0) 0x7f8a58058dd0 con 0x7f8a6015df80 2024-03-24T01:27:15.395 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:27:15.396 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":52,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:27:13.961036+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:27:11.774313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-24T01:27:12.992083+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:27:15.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.397+0000 7f8a5fe00640 1 -- 172.21.15.90:0/1159388248 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8a3c06bbd0 msgr2=0x7f8a3c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:15.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.397+0000 7f8a5fe00640 1 --2- 172.21.15.90:0/1159388248 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8a3c06bbd0 0x7f8a3c06e090 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8a50036000 tx=0x7f8a50008030 comp rx=0 tx=0).stop 2024-03-24T01:27:15.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.397+0000 7f8a5fe00640 1 -- 172.21.15.90:0/1159388248 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a6015df80 msgr2=0x7f8a601f42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:15.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.397+0000 7f8a5fe00640 1 --2- 172.21.15.90:0/1159388248 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a6015df80 0x7f8a601f42a0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f8a5800c7f0 tx=0x7f8a5800ccb0 comp rx=0 tx=0).stop 2024-03-24T01:27:15.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.398+0000 7f8a5fe00640 1 -- 172.21.15.90:0/1159388248 shutdown_connections 2024-03-24T01:27:15.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.398+0000 7f8a5fe00640 1 --2- 172.21.15.90:0/1159388248 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8a3c06bbd0 0x7f8a3c06e090 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:15.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.398+0000 7f8a5fe00640 1 --2- 172.21.15.90:0/1159388248 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a6015df80 0x7f8a601f42a0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:15.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.398+0000 7f8a5fe00640 1 --2- 172.21.15.90:0/1159388248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a6015dba0 0x7f8a601f0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:15.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.398+0000 7f8a5fe00640 1 --2- 172.21.15.90:0/1159388248 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8a600b5c40 0x7f8a601f6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:15.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.398+0000 7f8a5fe00640 1 -- 172.21.15.90:0/1159388248 >> 172.21.15.90:0/1159388248 conn(0x7f8a600d0530 msgr2=0x7f8a600b7570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:15.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.398+0000 7f8a5fe00640 1 -- 172.21.15.90:0/1159388248 shutdown_connections 2024-03-24T01:27:15.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.398+0000 7f8a5fe00640 1 -- 172.21.15.90:0/1159388248 wait complete. 2024-03-24T01:27:15.417 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-24T01:27:15.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.614+0000 7f27fb200640 1 Processor -- start 2024-03-24T01:27:15.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.614+0000 7f27fb200640 1 -- start start 2024-03-24T01:27:15.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.615+0000 7f27fb200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27f40cc8a0 0x7f27f40ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:15.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.616+0000 7f27fb200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27f40b3eb0 con 0x7f27f406b0c0 2024-03-24T01:27:15.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.616+0000 7f27fb200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27f406e680 con 0x7f27f40aee80 2024-03-24T01:27:15.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.616+0000 7f27fb200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27f406e960 con 0x7f27f40cc8a0 2024-03-24T01:27:15.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.616+0000 7f27f9200640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f27f406b0c0 0x7f27f406f160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60282/0 (socket says 172.21.15.90:60282) 2024-03-24T01:27:15.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.616+0000 7f27f9200640 1 -- 172.21.15.90:0/477524166 learned_addr learned my addr 172.21.15.90:0/477524166 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:27:15.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.616+0000 7f27ebe00640 1 --2- 172.21.15.90:0/477524166 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27f40cc8a0 0x7f27f40ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:15.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.616+0000 7f27f3400640 1 -- 172.21.15.90:0/477524166 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 171326493 0 0) 0x7f27f40b3eb0 con 0x7f27f406b0c0 2024-03-24T01:27:15.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.616+0000 7f27f3400640 1 -- 172.21.15.90:0/477524166 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27d8003520 con 0x7f27f406b0c0 2024-03-24T01:27:15.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.617+0000 7f27f3400640 1 -- 172.21.15.90:0/477524166 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4181517538 0 0) 0x7f27d8003520 con 0x7f27f406b0c0 2024-03-24T01:27:15.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.617+0000 7f27f3400640 1 -- 172.21.15.90:0/477524166 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27f40cc8a0 msgr2=0x7f27f40ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:15.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.617+0000 7f27f3400640 1 --2- 172.21.15.90:0/477524166 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27f40cc8a0 0x7f27f40ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:15.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.617+0000 7f27f3400640 1 -- 172.21.15.90:0/477524166 >> v1:172.21.15.120:6789/0 conn(0x7f27f40aee80 legacy=0x7f27f40b33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:15.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.617+0000 7f27f3400640 1 -- 172.21.15.90:0/477524166 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f406ec40 con 0x7f27f406b0c0 2024-03-24T01:27:15.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.617+0000 7f27f3400640 1 -- 172.21.15.90:0/477524166 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f27ec0019f0 con 0x7f27f406b0c0 2024-03-24T01:27:15.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.617+0000 7f27f3400640 1 -- 172.21.15.90:0/477524166 >> v1:172.21.15.90:6789/0 conn(0x7f27f406b0c0 legacy=0x7f27f406f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:15.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.617+0000 7f27f3400640 1 --2- 172.21.15.90:0/477524166 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27f40cc8a0 0x7f27d80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:15.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.617+0000 7f27f3400640 1 --2- 172.21.15.90:0/477524166 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27d8003f20 0x7f27d8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:15.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.617+0000 7f27f3400640 1 --2- 172.21.15.90:0/477524166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27d8004d40 0x7f27d80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:15.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.617+0000 7f27f3400640 1 -- 172.21.15.90:0/477524166 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f406e960 con 0x7f27d8003f20 2024-03-24T01:27:15.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.617+0000 7f27f3400640 1 -- 172.21.15.90:0/477524166 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d800bad0 con 0x7f27d8004d40 2024-03-24T01:27:15.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.617+0000 7f27f3400640 1 -- 172.21.15.90:0/477524166 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27d800bcd0 con 0x7f27f40cc8a0 2024-03-24T01:27:15.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.617+0000 7f27ebe00640 1 --2- 172.21.15.90:0/477524166 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27d8003f20 0x7f27d8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:15.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.618+0000 7f27f9200640 1 --2- 172.21.15.90:0/477524166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27d8004d40 0x7f27d80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:15.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.618+0000 7f27ebe00640 1 -- 172.21.15.90:0/477524166 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27f40cc8a0 msgr2=0x7f27d80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:15.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.618+0000 7f27ebe00640 1 --2- 172.21.15.90:0/477524166 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27f40cc8a0 0x7f27d80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:15.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.618+0000 7f27ebe00640 1 -- 172.21.15.90:0/477524166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27d8004d40 msgr2=0x7f27d80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:15.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.618+0000 7f27ebe00640 1 --2- 172.21.15.90:0/477524166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27d8004d40 0x7f27d80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:15.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.618+0000 7f27ebe00640 1 -- 172.21.15.90:0/477524166 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d8003520 con 0x7f27d8003f20 2024-03-24T01:27:15.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.618+0000 7f27f9200640 1 --2- 172.21.15.90:0/477524166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27d8004d40 0x7f27d80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:27:15.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.619+0000 7f27ebe00640 1 --2- 172.21.15.90:0/477524166 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27d8003f20 0x7f27d8004800 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f27dc009ac0 tx=0x7f27d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bbf8ff22c0a28b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:15.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.619+0000 7f27f3400640 1 -- 172.21.15.90:0/477524166 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27dc039070 con 0x7f27d8003f20 2024-03-24T01:27:15.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.619+0000 7f27f3400640 1 -- 172.21.15.90:0/477524166 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27dc02c710 con 0x7f27d8003f20 2024-03-24T01:27:15.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.619+0000 7f27f3400640 1 -- 172.21.15.90:0/477524166 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27dc009690 con 0x7f27d8003f20 2024-03-24T01:27:15.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.619+0000 7f27fb200640 1 -- 172.21.15.90:0/477524166 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27d8003f20 msgr2=0x7f27d8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:15.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.619+0000 7f27fb200640 1 --2- 172.21.15.90:0/477524166 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27d8003f20 0x7f27d8004800 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f27dc009ac0 tx=0x7f27d8005830 comp rx=0 tx=0).stop 2024-03-24T01:27:15.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.619+0000 7f27f9200640 1 -- 172.21.15.90:0/477524166 reap_dead start 2024-03-24T01:27:15.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.620+0000 7f27fb200640 1 -- 172.21.15.90:0/477524166 shutdown_connections 2024-03-24T01:27:15.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.620+0000 7f27fb200640 1 -- 172.21.15.90:0/477524166 >> 172.21.15.90:0/477524166 conn(0x7f27f40b1770 msgr2=0x7f27f40b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:15.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.620+0000 7f27fb200640 1 -- 172.21.15.90:0/477524166 shutdown_connections 2024-03-24T01:27:15.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.620+0000 7f27fb200640 1 -- 172.21.15.90:0/477524166 wait complete. 2024-03-24T01:27:15.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.621+0000 7f27fb200640 1 Processor -- start 2024-03-24T01:27:15.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.622+0000 7f27fb200640 1 -- start start 2024-03-24T01:27:15.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.622+0000 7f27fb200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27d8003f20 0x7f27f414cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:15.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.622+0000 7f27fb200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27f4159670 0x7f27f4159ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:15.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.622+0000 7f27fb200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27f4162870 0x7f27f4162cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:15.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.622+0000 7f27fb200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f40b3b00 con 0x7f27d8003f20 2024-03-24T01:27:15.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.622+0000 7f27fb200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f4166ad0 con 0x7f27f4162870 2024-03-24T01:27:15.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.622+0000 7f27fb200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27f415a010 con 0x7f27f4159670 2024-03-24T01:27:15.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.623+0000 7f27f3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27d8003f20 0x7f27f414cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:15.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.623+0000 7f27f3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27d8003f20 0x7f27f414cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58900/0 (socket says 172.21.15.90:58900) 2024-03-24T01:27:15.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.623+0000 7f27f3e00640 1 -- 172.21.15.90:0/3547290011 learned_addr learned my addr 172.21.15.90:0/3547290011 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:15.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.623+0000 7f27ebe00640 1 --2- 172.21.15.90:0/3547290011 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27f4159670 0x7f27f4159ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:15.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.623+0000 7f27f9200640 1 --2- 172.21.15.90:0/3547290011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27f4162870 0x7f27f4162cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:15.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.623+0000 7f27f3e00640 1 -- 172.21.15.90:0/3547290011 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27f4159670 msgr2=0x7f27f4159ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:15.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.623+0000 7f27f3e00640 1 --2- 172.21.15.90:0/3547290011 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27f4159670 0x7f27f4159ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:15.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.623+0000 7f27f3e00640 1 -- 172.21.15.90:0/3547290011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27f4162870 msgr2=0x7f27f4162cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:15.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.623+0000 7f27f3e00640 1 --2- 172.21.15.90:0/3547290011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27f4162870 0x7f27f4162cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:15.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.623+0000 7f27f3e00640 1 -- 172.21.15.90:0/3547290011 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f41635a0 con 0x7f27d8003f20 2024-03-24T01:27:15.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.623+0000 7f27f9200640 1 --2- 172.21.15.90:0/3547290011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27f4162870 0x7f27f4162cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:27:15.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.623+0000 7f27f3e00640 1 --2- 172.21.15.90:0/3547290011 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27d8003f20 0x7f27f414cb60 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f27e400c550 tx=0x7f27e400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:15.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.624+0000 7f27f0c00640 1 -- 172.21.15.90:0/3547290011 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e4017070 con 0x7f27d8003f20 2024-03-24T01:27:15.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.624+0000 7f27f0c00640 1 -- 172.21.15.90:0/3547290011 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e4005480 con 0x7f27d8003f20 2024-03-24T01:27:15.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.624+0000 7f27f0c00640 1 -- 172.21.15.90:0/3547290011 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e4003830 con 0x7f27d8003f20 2024-03-24T01:27:15.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.624+0000 7f27fb200640 1 -- 172.21.15.90:0/3547290011 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27f4163880 con 0x7f27d8003f20 2024-03-24T01:27:15.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.624+0000 7f27fb200640 1 -- 172.21.15.90:0/3547290011 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27f4163e40 con 0x7f27d8003f20 2024-03-24T01:27:15.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.625+0000 7f27fb200640 1 -- 172.21.15.90:0/3547290011 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27f404f400 con 0x7f27d8003f20 2024-03-24T01:27:15.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.626+0000 7f27f0c00640 1 -- 172.21.15.90:0/3547290011 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f27e401a070 con 0x7f27d8003f20 2024-03-24T01:27:15.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.627+0000 7f27f0c00640 1 --2- 172.21.15.90:0/3547290011 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f27d006bc00 0x7f27d006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:15.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.627+0000 7f27f0c00640 1 -- 172.21.15.90:0/3547290011 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f27e408a1d0 con 0x7f27d8003f20 2024-03-24T01:27:15.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.628+0000 7f27ebe00640 1 --2- 172.21.15.90:0/3547290011 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f27d006bc00 0x7f27d006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:15.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.629+0000 7f27ebe00640 1 --2- 172.21.15.90:0/3547290011 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f27d006bc00 0x7f27d006e0c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f27dc036000 tx=0x7f27dc030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:15.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.632+0000 7f27f0c00640 1 -- 172.21.15.90:0/3547290011 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f27e4058a00 con 0x7f27d8003f20 2024-03-24T01:27:15.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:15.851+0000 7f27fb200640 1 -- 172.21.15.90:0/3547290011 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f27f40684f0 con 0x7f27d8003f20 2024-03-24T01:27:16.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.555+0000 7f27f0c00640 1 -- 172.21.15.90:0/3547290011 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v33) v1 ==== 118+0+0 (secure 0 0 0) 0x7f27e4058300 con 0x7f27d8003f20 2024-03-24T01:27:16.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.558+0000 7f27fb200640 1 -- 172.21.15.90:0/3547290011 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f27d006bc00 msgr2=0x7f27d006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:16.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.558+0000 7f27fb200640 1 --2- 172.21.15.90:0/3547290011 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f27d006bc00 0x7f27d006e0c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f27dc036000 tx=0x7f27dc030000 comp rx=0 tx=0).stop 2024-03-24T01:27:16.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.558+0000 7f27fb200640 1 -- 172.21.15.90:0/3547290011 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27d8003f20 msgr2=0x7f27f414cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:16.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.558+0000 7f27fb200640 1 --2- 172.21.15.90:0/3547290011 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27d8003f20 0x7f27f414cb60 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f27e400c550 tx=0x7f27e400ca10 comp rx=0 tx=0).stop 2024-03-24T01:27:16.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.558+0000 7f27fb200640 1 -- 172.21.15.90:0/3547290011 shutdown_connections 2024-03-24T01:27:16.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.558+0000 7f27fb200640 1 --2- 172.21.15.90:0/3547290011 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f27d006bc00 0x7f27d006e0c0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:16.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.558+0000 7f27fb200640 1 --2- 172.21.15.90:0/3547290011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27f4162870 0x7f27f4162cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:16.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.558+0000 7f27fb200640 1 --2- 172.21.15.90:0/3547290011 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f27f4159670 0x7f27f4159ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:16.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.558+0000 7f27fb200640 1 --2- 172.21.15.90:0/3547290011 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27d8003f20 0x7f27f414cb60 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:16.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.558+0000 7f27fb200640 1 -- 172.21.15.90:0/3547290011 >> 172.21.15.90:0/3547290011 conn(0x7f27f40b1770 msgr2=0x7f27f4166190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:16.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.559+0000 7f27fb200640 1 -- 172.21.15.90:0/3547290011 shutdown_connections 2024-03-24T01:27:16.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.559+0000 7f27fb200640 1 -- 172.21.15.90:0/3547290011 wait complete. 2024-03-24T01:27:16.581 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-24T01:27:16.582 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-24T01:27:16.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.790+0000 7f5253a00640 1 Processor -- start 2024-03-24T01:27:16.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.790+0000 7f5253a00640 1 -- start start 2024-03-24T01:27:16.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.791+0000 7f5253a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f524c15f370 0x7f524c06c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:16.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.791+0000 7f5253a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f524c0cc310 0x7f524c0cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:16.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.791+0000 7f5253a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f524c0d0a10 con 0x7f524c0b2930 2024-03-24T01:27:16.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.791+0000 7f5253a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f524c0d07c0 con 0x7f524c15f370 2024-03-24T01:27:16.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.791+0000 7f5253a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f524c0d0cf0 con 0x7f524c0cc310 2024-03-24T01:27:16.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.792+0000 7f5251000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f524c15f370 0x7f524c06c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:16.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.792+0000 7f524be00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f524c0b2930 0x7f524c06c710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60298/0 (socket says 172.21.15.90:60298) 2024-03-24T01:27:16.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.792+0000 7f5251000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f524c15f370 0x7f524c06c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58912/0 (socket says 172.21.15.90:58912) 2024-03-24T01:27:16.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.792+0000 7f524be00640 1 -- 172.21.15.90:0/2329572219 learned_addr learned my addr 172.21.15.90:0/2329572219 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:27:16.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.792+0000 7f5251a00640 1 --2- 172.21.15.90:0/2329572219 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f524c0cc310 0x7f524c0cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:16.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.792+0000 7f5251000640 1 -- 172.21.15.90:0/2329572219 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f524c0cc310 msgr2=0x7f524c0cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:16.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.792+0000 7f5251000640 1 --2- 172.21.15.90:0/2329572219 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f524c0cc310 0x7f524c0cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:16.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.792+0000 7f5251000640 1 -- 172.21.15.90:0/2329572219 >> v1:172.21.15.90:6789/0 conn(0x7f524c0b2930 legacy=0x7f524c06c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:16.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.792+0000 7f5251000640 1 -- 172.21.15.90:0/2329572219 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f524c0d0fd0 con 0x7f524c15f370 2024-03-24T01:27:16.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.793+0000 7f5251000640 1 --2- 172.21.15.90:0/2329572219 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f524c15f370 0x7f524c06c1d0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f523c0097a0 tx=0x7f523c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46263152d248c7e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:16.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.794+0000 7f524b400640 1 -- 172.21.15.90:0/2329572219 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523c035020 con 0x7f524c15f370 2024-03-24T01:27:16.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.794+0000 7f524b400640 1 -- 172.21.15.90:0/2329572219 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f523c031810 con 0x7f524c15f370 2024-03-24T01:27:16.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.794+0000 7f524b400640 1 -- 172.21.15.90:0/2329572219 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523c031be0 con 0x7f524c15f370 2024-03-24T01:27:16.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.794+0000 7f5253a00640 1 -- 172.21.15.90:0/2329572219 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f524c15f370 msgr2=0x7f524c06c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:16.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.794+0000 7f5253a00640 1 --2- 172.21.15.90:0/2329572219 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f524c15f370 0x7f524c06c1d0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f523c0097a0 tx=0x7f523c02b8b0 comp rx=0 tx=0).stop 2024-03-24T01:27:16.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.794+0000 7f5253a00640 1 -- 172.21.15.90:0/2329572219 shutdown_connections 2024-03-24T01:27:16.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.794+0000 7f5253a00640 1 --2- 172.21.15.90:0/2329572219 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f524c0cc310 0x7f524c0cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:16.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.794+0000 7f5253a00640 1 --2- 172.21.15.90:0/2329572219 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f524c15f370 0x7f524c06c1d0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:16.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.794+0000 7f5253a00640 1 -- 172.21.15.90:0/2329572219 >> 172.21.15.90:0/2329572219 conn(0x7f524c0b34d0 msgr2=0x7f524c0b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:16.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.794+0000 7f5253a00640 1 -- 172.21.15.90:0/2329572219 shutdown_connections 2024-03-24T01:27:16.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.795+0000 7f5253a00640 1 -- 172.21.15.90:0/2329572219 wait complete. 2024-03-24T01:27:16.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.796+0000 7f5253a00640 1 Processor -- start 2024-03-24T01:27:16.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.796+0000 7f5253a00640 1 -- start start 2024-03-24T01:27:16.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.797+0000 7f5253a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f524c0b2930 0x7f524c150f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:16.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.797+0000 7f5253a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f524c0cc310 0x7f524c0b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:16.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.797+0000 7f5253a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f524c15f370 0x7f524c0b6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:16.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.797+0000 7f5253a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f524c06ce20 con 0x7f524c15f370 2024-03-24T01:27:16.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.797+0000 7f5253a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f524c0d1590 con 0x7f524c0b2930 2024-03-24T01:27:16.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.797+0000 7f5253a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f524c06cfa0 con 0x7f524c0cc310 2024-03-24T01:27:16.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.797+0000 7f5251a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f524c15f370 0x7f524c0b6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:16.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.797+0000 7f5251a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f524c15f370 0x7f524c0b6730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58914/0 (socket says 172.21.15.90:58914) 2024-03-24T01:27:16.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.797+0000 7f5251a00640 1 -- 172.21.15.90:0/1246531749 learned_addr learned my addr 172.21.15.90:0/1246531749 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:16.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.798+0000 7f524be00640 1 --2- 172.21.15.90:0/1246531749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f524c0cc310 0x7f524c0b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:16.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.798+0000 7f5251000640 1 --2- 172.21.15.90:0/1246531749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f524c0b2930 0x7f524c150f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:16.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.798+0000 7f5251a00640 1 -- 172.21.15.90:0/1246531749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f524c0cc310 msgr2=0x7f524c0b61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:16.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.798+0000 7f5251a00640 1 --2- 172.21.15.90:0/1246531749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f524c0cc310 0x7f524c0b61f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:16.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.798+0000 7f5251a00640 1 -- 172.21.15.90:0/1246531749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f524c0b2930 msgr2=0x7f524c150f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.798+0000 7f5251a00640 1 --2- 172.21.15.90:0/1246531749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f524c0b2930 0x7f524c150f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.798+0000 7f5251a00640 1 -- 172.21.15.90:0/1246531749 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f524c0c60a0 con 0x7f524c15f370 2024-03-24T01:27:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.798+0000 7f5251a00640 1 --2- 172.21.15.90:0/1246531749 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f524c15f370 0x7f524c0b6730 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f5244007e40 tx=0x7f524400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.798+0000 7f5248c00640 1 -- 172.21.15.90:0/1246531749 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5244014620 con 0x7f524c15f370 2024-03-24T01:27:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.798+0000 7f5248c00640 1 -- 172.21.15.90:0/1246531749 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5244019070 con 0x7f524c15f370 2024-03-24T01:27:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.798+0000 7f5248c00640 1 -- 172.21.15.90:0/1246531749 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5244014a20 con 0x7f524c15f370 2024-03-24T01:27:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.798+0000 7f5253a00640 1 -- 172.21.15.90:0/1246531749 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f524c0c6380 con 0x7f524c15f370 2024-03-24T01:27:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.799+0000 7f5253a00640 1 -- 172.21.15.90:0/1246531749 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f524c0c6940 con 0x7f524c15f370 2024-03-24T01:27:16.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.800+0000 7f5253a00640 1 -- 172.21.15.90:0/1246531749 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f524c04f400 con 0x7f524c15f370 2024-03-24T01:27:16.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.800+0000 7f5248c00640 1 -- 172.21.15.90:0/1246531749 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5244014bc0 con 0x7f524c15f370 2024-03-24T01:27:16.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.801+0000 7f5248c00640 1 --2- 172.21.15.90:0/1246531749 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f522806bbd0 0x7f522806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:16.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.801+0000 7f5248c00640 1 -- 172.21.15.90:0/1246531749 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f524408b500 con 0x7f524c15f370 2024-03-24T01:27:16.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.802+0000 7f5251000640 1 --2- 172.21.15.90:0/1246531749 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f522806bbd0 0x7f522806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:16.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.803+0000 7f5251000640 1 --2- 172.21.15.90:0/1246531749 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f522806bbd0 0x7f522806e090 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f523c036000 tx=0x7f523c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:16.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:16.806+0000 7f5248c00640 1 -- 172.21.15.90:0/1246531749 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5244059e50 con 0x7f524c15f370 2024-03-24T01:27:17.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.042+0000 7f5253a00640 1 -- 172.21.15.90:0/1246531749 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f524c0c7ee0 con 0x7f524c15f370 2024-03-24T01:27:17.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.558+0000 7f5248c00640 1 -- 172.21.15.90:0/1246531749 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5244059750 con 0x7f524c15f370 2024-03-24T01:27:17.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.561+0000 7f5253a00640 1 -- 172.21.15.90:0/1246531749 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f522806bbd0 msgr2=0x7f522806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:17.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.561+0000 7f5253a00640 1 --2- 172.21.15.90:0/1246531749 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f522806bbd0 0x7f522806e090 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f523c036000 tx=0x7f523c008030 comp rx=0 tx=0).stop 2024-03-24T01:27:17.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.561+0000 7f5253a00640 1 -- 172.21.15.90:0/1246531749 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f524c15f370 msgr2=0x7f524c0b6730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:17.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.561+0000 7f5253a00640 1 --2- 172.21.15.90:0/1246531749 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f524c15f370 0x7f524c0b6730 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f5244007e40 tx=0x7f524400e4b0 comp rx=0 tx=0).stop 2024-03-24T01:27:17.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.561+0000 7f5253a00640 1 -- 172.21.15.90:0/1246531749 shutdown_connections 2024-03-24T01:27:17.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.561+0000 7f5253a00640 1 --2- 172.21.15.90:0/1246531749 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f522806bbd0 0x7f522806e090 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:17.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.561+0000 7f5253a00640 1 --2- 172.21.15.90:0/1246531749 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f524c15f370 0x7f524c0b6730 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:17.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.561+0000 7f5253a00640 1 --2- 172.21.15.90:0/1246531749 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f524c0cc310 0x7f524c0b61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:17.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.561+0000 7f5253a00640 1 --2- 172.21.15.90:0/1246531749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f524c0b2930 0x7f524c150f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:17.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.561+0000 7f5253a00640 1 -- 172.21.15.90:0/1246531749 >> 172.21.15.90:0/1246531749 conn(0x7f524c0b34d0 msgr2=0x7f524c0cc740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:17.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.562+0000 7f5253a00640 1 -- 172.21.15.90:0/1246531749 shutdown_connections 2024-03-24T01:27:17.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.562+0000 7f5253a00640 1 -- 172.21.15.90:0/1246531749 wait complete. 2024-03-24T01:27:17.582 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-24T01:27:17.582 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:27:17.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.781+0000 7fc206400640 1 Processor -- start 2024-03-24T01:27:17.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.781+0000 7fc206400640 1 -- start start 2024-03-24T01:27:17.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.783+0000 7fc206400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc2000c5c50 0x7fc200166b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:17.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.783+0000 7fc206400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2000cd4a0 con 0x7fc2000cd010 2024-03-24T01:27:17.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.783+0000 7fc206400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2000cd760 con 0x7fc20015f0f0 2024-03-24T01:27:17.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.783+0000 7fc206400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc200099cd0 con 0x7fc2000c5c50 2024-03-24T01:27:17.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.783+0000 7fc1f6a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc2000cd010 0x7fc2000c5540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60310/0 (socket says 172.21.15.90:60310) 2024-03-24T01:27:17.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.783+0000 7fc1f6a00640 1 -- 172.21.15.90:0/2888769017 learned_addr learned my addr 172.21.15.90:0/2888769017 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:27:17.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.784+0000 7fc1fea00640 1 -- 172.21.15.90:0/2888769017 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4094163826 0 0) 0x7fc2000cd4a0 con 0x7fc2000cd010 2024-03-24T01:27:17.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.784+0000 7fc1ffe00640 1 --2- 172.21.15.90:0/2888769017 >> v2:172.21.15.120:3300/0 conn(0x7fc2000c5c50 0x7fc200166b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:17.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.784+0000 7fc1fea00640 1 -- 172.21.15.90:0/2888769017 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1e4003520 con 0x7fc2000cd010 2024-03-24T01:27:17.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.784+0000 7fc1fea00640 1 -- 172.21.15.90:0/2888769017 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2939675543 0 0) 0x7fc1e4003520 con 0x7fc2000cd010 2024-03-24T01:27:17.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.784+0000 7fc1fea00640 1 -- 172.21.15.90:0/2888769017 >> v2:172.21.15.120:3300/0 conn(0x7fc2000c5c50 msgr2=0x7fc200166b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:17.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.784+0000 7fc1fea00640 1 --2- 172.21.15.90:0/2888769017 >> v2:172.21.15.120:3300/0 conn(0x7fc2000c5c50 0x7fc200166b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:17.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.784+0000 7fc1fea00640 1 -- 172.21.15.90:0/2888769017 >> v1:172.21.15.120:6789/0 conn(0x7fc20015f0f0 legacy=0x7fc20009aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:17.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.784+0000 7fc1fea00640 1 -- 172.21.15.90:0/2888769017 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc200099f20 con 0x7fc2000cd010 2024-03-24T01:27:17.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.784+0000 7fc1fea00640 1 -- 172.21.15.90:0/2888769017 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7fc1e8001670 con 0x7fc2000cd010 2024-03-24T01:27:17.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.784+0000 7fc1fea00640 1 -- 172.21.15.90:0/2888769017 >> v1:172.21.15.90:6789/0 conn(0x7fc2000cd010 legacy=0x7fc2000c5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:17.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.784+0000 7fc1fea00640 1 --2- 172.21.15.90:0/2888769017 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1e40039b0 0x7fc1e4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:17.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.784+0000 7fc1fea00640 1 --2- 172.21.15.90:0/2888769017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1e4004800 0x7fc1e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:17.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.784+0000 7fc1fea00640 1 --2- 172.21.15.90:0/2888769017 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1e40052e0 0x7fc1e400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:17.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.784+0000 7fc1fea00640 1 -- 172.21.15.90:0/2888769017 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc200099cd0 con 0x7fc1e40039b0 2024-03-24T01:27:17.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.784+0000 7fc1fea00640 1 -- 172.21.15.90:0/2888769017 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e400c1b0 con 0x7fc1e4004800 2024-03-24T01:27:17.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.784+0000 7fc1fea00640 1 -- 172.21.15.90:0/2888769017 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e400c3b0 con 0x7fc1e40052e0 2024-03-24T01:27:17.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.784+0000 7fc1ff400640 1 --2- 172.21.15.90:0/2888769017 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1e40039b0 0x7fc1e4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:17.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.785+0000 7fc1f6a00640 1 --2- 172.21.15.90:0/2888769017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1e4004800 0x7fc1e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:17.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.785+0000 7fc1ffe00640 1 --2- 172.21.15.90:0/2888769017 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1e40052e0 0x7fc1e400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:17.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.785+0000 7fc1ff400640 1 -- 172.21.15.90:0/2888769017 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1e40052e0 msgr2=0x7fc1e400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:17.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.785+0000 7fc1ff400640 1 --2- 172.21.15.90:0/2888769017 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1e40052e0 0x7fc1e400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:17.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.785+0000 7fc1ff400640 1 -- 172.21.15.90:0/2888769017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1e4004800 msgr2=0x7fc1e4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:17.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.785+0000 7fc1ff400640 1 --2- 172.21.15.90:0/2888769017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1e4004800 0x7fc1e4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:17.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.785+0000 7fc1ff400640 1 -- 172.21.15.90:0/2888769017 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e4003520 con 0x7fc1e40039b0 2024-03-24T01:27:17.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.785+0000 7fc1ffe00640 1 -- 172.21.15.90:0/2888769017 reap_dead start 2024-03-24T01:27:17.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.785+0000 7fc1f6a00640 1 --2- 172.21.15.90:0/2888769017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1e4004800 0x7fc1e4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:27:17.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.786+0000 7fc1ff400640 1 --2- 172.21.15.90:0/2888769017 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1e40039b0 0x7fc1e4003dd0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fc1f0001720 tx=0x7fc1e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64a9bd77f506dc30 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:17.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.786+0000 7fc1fea00640 1 -- 172.21.15.90:0/2888769017 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f0001370 con 0x7fc1e40039b0 2024-03-24T01:27:17.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.786+0000 7fc1fea00640 1 -- 172.21.15.90:0/2888769017 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1f00052d0 con 0x7fc1e40039b0 2024-03-24T01:27:17.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.786+0000 7fc1fea00640 1 -- 172.21.15.90:0/2888769017 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f0003830 con 0x7fc1e40039b0 2024-03-24T01:27:17.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.787+0000 7fc206400640 1 -- 172.21.15.90:0/2888769017 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1e40039b0 msgr2=0x7fc1e4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:17.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.787+0000 7fc206400640 1 --2- 172.21.15.90:0/2888769017 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1e40039b0 0x7fc1e4003dd0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fc1f0001720 tx=0x7fc1e4005830 comp rx=0 tx=0).stop 2024-03-24T01:27:17.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.787+0000 7fc206400640 1 -- 172.21.15.90:0/2888769017 shutdown_connections 2024-03-24T01:27:17.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.787+0000 7fc206400640 1 --2- 172.21.15.90:0/2888769017 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1e40039b0 0x7fc1e4003dd0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:17.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.787+0000 7fc206400640 1 -- 172.21.15.90:0/2888769017 >> 172.21.15.90:0/2888769017 conn(0x7fc2000b34e0 msgr2=0x7fc2000b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:17.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.787+0000 7fc206400640 1 -- 172.21.15.90:0/2888769017 shutdown_connections 2024-03-24T01:27:17.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.787+0000 7fc206400640 1 -- 172.21.15.90:0/2888769017 wait complete. 2024-03-24T01:27:17.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.789+0000 7fc206400640 1 Processor -- start 2024-03-24T01:27:17.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.789+0000 7fc206400640 1 -- start start 2024-03-24T01:27:17.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.790+0000 7fc206400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1e40039b0 0x7fc200158220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:17.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.790+0000 7fc206400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc200158930 0x7fc2000cadc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:17.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.790+0000 7fc206400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2000b4830 0x7fc2000b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:17.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.790+0000 7fc206400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20015f4d0 con 0x7fc200158930 2024-03-24T01:27:17.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.790+0000 7fc206400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20009a4e0 con 0x7fc2000b4830 2024-03-24T01:27:17.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.790+0000 7fc206400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2000c5540 con 0x7fc1e40039b0 2024-03-24T01:27:17.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.790+0000 7fc1f6a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc200158930 0x7fc2000cadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:17.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.790+0000 7fc1f6a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc200158930 0x7fc2000cadc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58936/0 (socket says 172.21.15.90:58936) 2024-03-24T01:27:17.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.790+0000 7fc1f6a00640 1 -- 172.21.15.90:0/2799636827 learned_addr learned my addr 172.21.15.90:0/2799636827 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:17.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.790+0000 7fc1ff400640 1 --2- 172.21.15.90:0/2799636827 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1e40039b0 0x7fc200158220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:17.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.791+0000 7fc1f6a00640 1 -- 172.21.15.90:0/2799636827 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1e40039b0 msgr2=0x7fc200158220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:17.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.791+0000 7fc1f6a00640 1 --2- 172.21.15.90:0/2799636827 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1e40039b0 0x7fc200158220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:17.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.791+0000 7fc1f6a00640 1 -- 172.21.15.90:0/2799636827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2000b4830 msgr2=0x7fc2000b4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:17.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.791+0000 7fc1f6a00640 1 --2- 172.21.15.90:0/2799636827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2000b4830 0x7fc2000b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:17.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.791+0000 7fc1f6a00640 1 -- 172.21.15.90:0/2799636827 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc200158e30 con 0x7fc200158930 2024-03-24T01:27:17.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.791+0000 7fc1f6a00640 1 --2- 172.21.15.90:0/2799636827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc200158930 0x7fc2000cadc0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fc1e800e7b0 tx=0x7fc1e800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:17.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.791+0000 7fc1f7e00640 1 -- 172.21.15.90:0/2799636827 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e800ee60 con 0x7fc200158930 2024-03-24T01:27:17.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.791+0000 7fc1f7e00640 1 -- 172.21.15.90:0/2799636827 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1e8018070 con 0x7fc200158930 2024-03-24T01:27:17.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.791+0000 7fc1f7e00640 1 -- 172.21.15.90:0/2799636827 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e800ca70 con 0x7fc200158930 2024-03-24T01:27:17.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.791+0000 7fc206400640 1 -- 172.21.15.90:0/2799636827 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc200159110 con 0x7fc200158930 2024-03-24T01:27:17.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.791+0000 7fc206400640 1 -- 172.21.15.90:0/2799636827 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2001596d0 con 0x7fc200158930 2024-03-24T01:27:17.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.792+0000 7fc206400640 1 -- 172.21.15.90:0/2799636827 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc200068060 con 0x7fc200158930 2024-03-24T01:27:17.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.793+0000 7fc1f7e00640 1 -- 172.21.15.90:0/2799636827 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc1e800cc10 con 0x7fc200158930 2024-03-24T01:27:17.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.794+0000 7fc1f7e00640 1 --2- 172.21.15.90:0/2799636827 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc1dc06bc00 0x7fc1dc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:17.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.794+0000 7fc1f7e00640 1 -- 172.21.15.90:0/2799636827 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc1e808aaa0 con 0x7fc200158930 2024-03-24T01:27:17.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.795+0000 7fc1ff400640 1 --2- 172.21.15.90:0/2799636827 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc1dc06bc00 0x7fc1dc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:17.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.796+0000 7fc1ff400640 1 --2- 172.21.15.90:0/2799636827 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc1dc06bc00 0x7fc1dc06e0c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fc1f0033000 tx=0x7fc1f0009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:17.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:17.799+0000 7fc1f7e00640 1 -- 172.21.15.90:0/2799636827 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc1e8059360 con 0x7fc200158930 2024-03-24T01:27:18.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.028+0000 7fc206400640 1 -- 172.21.15.90:0/2799636827 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2000cd4a0 con 0x7fc200158930 2024-03-24T01:27:18.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.029+0000 7fc1f7e00640 1 -- 172.21.15.90:0/2799636827 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4977 (secure 0 0 0) 0x7fc1e8058c60 con 0x7fc200158930 2024-03-24T01:27:18.030 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:27:18.030 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":34,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4985,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6834/3475131973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":3475131973},{"type":"v1","addr":"172.21.15.90:6834","nonce":3475131973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":4994,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/3151604088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":3151604088},{"type":"v1","addr":"172.21.15.120:6837","nonce":3151604088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6219,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/226826781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":226826781},{"type":"v1","addr":"172.21.15.90:6835","nonce":226826781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T01:27:13.961373+0000","modified":"2024-03-24T01:27:17.043944+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6222},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6222":{"gid":6222,"name":"b","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6836/266172244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":266172244},{"type":"v1","addr":"172.21.15.120:6836","nonce":266172244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6222,"qdb_cluster":[6222]},"id":3}]} 2024-03-24T01:27:18.030 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 34 2024-03-24T01:27:18.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.032+0000 7fc206400640 1 -- 172.21.15.90:0/2799636827 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc1dc06bc00 msgr2=0x7fc1dc06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:18.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.032+0000 7fc206400640 1 --2- 172.21.15.90:0/2799636827 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc1dc06bc00 0x7fc1dc06e0c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fc1f0033000 tx=0x7fc1f0009030 comp rx=0 tx=0).stop 2024-03-24T01:27:18.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.032+0000 7fc206400640 1 -- 172.21.15.90:0/2799636827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc200158930 msgr2=0x7fc2000cadc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:18.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.032+0000 7fc206400640 1 --2- 172.21.15.90:0/2799636827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc200158930 0x7fc2000cadc0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fc1e800e7b0 tx=0x7fc1e800ec70 comp rx=0 tx=0).stop 2024-03-24T01:27:18.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.032+0000 7fc206400640 1 -- 172.21.15.90:0/2799636827 shutdown_connections 2024-03-24T01:27:18.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.032+0000 7fc206400640 1 --2- 172.21.15.90:0/2799636827 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc1dc06bc00 0x7fc1dc06e0c0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:18.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.032+0000 7fc206400640 1 --2- 172.21.15.90:0/2799636827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc200158930 0x7fc2000cadc0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:18.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.032+0000 7fc206400640 1 --2- 172.21.15.90:0/2799636827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2000b4830 0x7fc2000b4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:18.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.032+0000 7fc206400640 1 --2- 172.21.15.90:0/2799636827 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1e40039b0 0x7fc200158220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:18.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.032+0000 7fc206400640 1 -- 172.21.15.90:0/2799636827 >> 172.21.15.90:0/2799636827 conn(0x7fc2000b34e0 msgr2=0x7fc20006fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:18.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.032+0000 7fc206400640 1 -- 172.21.15.90:0/2799636827 shutdown_connections 2024-03-24T01:27:18.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.032+0000 7fc206400640 1 -- 172.21.15.90:0/2799636827 wait complete. 2024-03-24T01:27:18.051 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:27:18.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.249+0000 7fa305e00640 1 Processor -- start 2024-03-24T01:27:18.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.249+0000 7fa305e00640 1 -- start start 2024-03-24T01:27:18.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.251+0000 7fa305e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa3000cc150 0x7fa3000ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:18.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.251+0000 7fa305e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa3001603e0 0x7fa30006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:18.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.251+0000 7fa305e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3000cd490 con 0x7fa300145430 2024-03-24T01:27:18.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.251+0000 7fa305e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa30015f900 con 0x7fa3001603e0 2024-03-24T01:27:18.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.251+0000 7fa305e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa30015fbe0 con 0x7fa3000cc150 2024-03-24T01:27:18.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.251+0000 7fa2f6000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa3001603e0 0x7fa30006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:18.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.251+0000 7fa2f6000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa3001603e0 0x7fa30006b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58948/0 (socket says 172.21.15.90:58948) 2024-03-24T01:27:18.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.251+0000 7fa2f6000640 1 -- 172.21.15.90:0/2076541515 learned_addr learned my addr 172.21.15.90:0/2076541515 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:18.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.252+0000 7fa2fea00640 1 --2- 172.21.15.90:0/2076541515 >> v2:172.21.15.120:3300/0 conn(0x7fa3000cc150 0x7fa3000ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:18.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.252+0000 7fa2f6000640 1 -- 172.21.15.90:0/2076541515 >> v2:172.21.15.120:3300/0 conn(0x7fa3000cc150 msgr2=0x7fa3000ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:18.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.252+0000 7fa2f6000640 1 --2- 172.21.15.90:0/2076541515 >> v2:172.21.15.120:3300/0 conn(0x7fa3000cc150 0x7fa3000ccb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:18.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.252+0000 7fa2f6000640 1 -- 172.21.15.90:0/2076541515 >> v1:172.21.15.120:6789/0 conn(0x7fa300145430 legacy=0x7fa300071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:18.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.252+0000 7fa2f6000640 1 -- 172.21.15.90:0/2076541515 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa30015fec0 con 0x7fa3001603e0 2024-03-24T01:27:18.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.253+0000 7fa2f6000640 1 --2- 172.21.15.90:0/2076541515 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3001603e0 0x7fa30006b0c0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fa30006bd10 tx=0x7fa2e802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e92720a5eb082347 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:18.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.254+0000 7fa2fe000640 1 -- 172.21.15.90:0/2076541515 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e8034070 con 0x7fa3001603e0 2024-03-24T01:27:18.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.254+0000 7fa2fe000640 1 -- 172.21.15.90:0/2076541515 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2e8030860 con 0x7fa3001603e0 2024-03-24T01:27:18.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.254+0000 7fa2fe000640 1 -- 172.21.15.90:0/2076541515 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e8030c30 con 0x7fa3001603e0 2024-03-24T01:27:18.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.254+0000 7fa305e00640 1 -- 172.21.15.90:0/2076541515 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3001603e0 msgr2=0x7fa30006b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:18.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.254+0000 7fa305e00640 1 --2- 172.21.15.90:0/2076541515 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3001603e0 0x7fa30006b0c0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fa30006bd10 tx=0x7fa2e802b9a0 comp rx=0 tx=0).stop 2024-03-24T01:27:18.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.254+0000 7fa305e00640 1 -- 172.21.15.90:0/2076541515 shutdown_connections 2024-03-24T01:27:18.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.254+0000 7fa305e00640 1 --2- 172.21.15.90:0/2076541515 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3001603e0 0x7fa30006b0c0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:18.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.254+0000 7fa305e00640 1 --2- 172.21.15.90:0/2076541515 >> v2:172.21.15.120:3300/0 conn(0x7fa3000cc150 0x7fa3000ccb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:18.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.254+0000 7fa305e00640 1 -- 172.21.15.90:0/2076541515 >> 172.21.15.90:0/2076541515 conn(0x7fa3000b1770 msgr2=0x7fa3000b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:18.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.255+0000 7fa305e00640 1 -- 172.21.15.90:0/2076541515 shutdown_connections 2024-03-24T01:27:18.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.255+0000 7fa305e00640 1 -- 172.21.15.90:0/2076541515 wait complete. 2024-03-24T01:27:18.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.256+0000 7fa305e00640 1 Processor -- start 2024-03-24T01:27:18.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.256+0000 7fa305e00640 1 -- start start 2024-03-24T01:27:18.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.257+0000 7fa305e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3000cc150 0x7fa3001f4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:18.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.257+0000 7fa305e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa300145430 0x7fa3000ac170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:18.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.257+0000 7fa305e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3001603e0 0x7fa300142fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:18.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.257+0000 7fa305e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa30006b780 con 0x7fa300145430 2024-03-24T01:27:18.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.257+0000 7fa305e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa30006b600 con 0x7fa3001603e0 2024-03-24T01:27:18.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.257+0000 7fa305e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa30006b900 con 0x7fa3000cc150 2024-03-24T01:27:18.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.257+0000 7fa2f6000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa300145430 0x7fa3000ac170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:18.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.258+0000 7fa2f6000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa300145430 0x7fa3000ac170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58954/0 (socket says 172.21.15.90:58954) 2024-03-24T01:27:18.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.258+0000 7fa2f6000640 1 -- 172.21.15.90:0/2011943504 learned_addr learned my addr 172.21.15.90:0/2011943504 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:18.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.258+0000 7fa2f6000640 1 -- 172.21.15.90:0/2011943504 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3000cc150 msgr2=0x7fa3001f4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:18.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.258+0000 7fa2f6000640 1 --2- 172.21.15.90:0/2011943504 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3000cc150 0x7fa3001f4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:18.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.258+0000 7fa2f6000640 1 -- 172.21.15.90:0/2011943504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3001603e0 msgr2=0x7fa300142fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:18.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.258+0000 7fa2f6000640 1 --2- 172.21.15.90:0/2011943504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3001603e0 0x7fa300142fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:18.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.258+0000 7fa2f6000640 1 -- 172.21.15.90:0/2011943504 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3001f4fc0 con 0x7fa300145430 2024-03-24T01:27:18.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.258+0000 7fa2f6000640 1 --2- 172.21.15.90:0/2011943504 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa300145430 0x7fa3000ac170 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fa2e802b9e0 tx=0x7fa2e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:18.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.258+0000 7fa2f7e00640 1 -- 172.21.15.90:0/2011943504 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e8034040 con 0x7fa300145430 2024-03-24T01:27:18.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.259+0000 7fa2f7e00640 1 -- 172.21.15.90:0/2011943504 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2e803ad20 con 0x7fa300145430 2024-03-24T01:27:18.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.259+0000 7fa2f7e00640 1 -- 172.21.15.90:0/2011943504 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e8044610 con 0x7fa300145430 2024-03-24T01:27:18.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.259+0000 7fa305e00640 1 -- 172.21.15.90:0/2011943504 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3000aca50 con 0x7fa300145430 2024-03-24T01:27:18.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.259+0000 7fa305e00640 1 -- 172.21.15.90:0/2011943504 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3000ad010 con 0x7fa300145430 2024-03-24T01:27:18.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.260+0000 7fa305e00640 1 -- 172.21.15.90:0/2011943504 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa300068060 con 0x7fa300145430 2024-03-24T01:27:18.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.261+0000 7fa2f7e00640 1 -- 172.21.15.90:0/2011943504 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa2e804b060 con 0x7fa300145430 2024-03-24T01:27:18.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.262+0000 7fa2f7e00640 1 --2- 172.21.15.90:0/2011943504 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa2dc06bbd0 0x7fa2dc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:18.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.262+0000 7fa2f7e00640 1 -- 172.21.15.90:0/2011943504 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa2e80acda0 con 0x7fa300145430 2024-03-24T01:27:18.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.262+0000 7fa2fea00640 1 --2- 172.21.15.90:0/2011943504 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa2dc06bbd0 0x7fa2dc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:18.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.263+0000 7fa2fea00640 1 --2- 172.21.15.90:0/2011943504 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa2dc06bbd0 0x7fa2dc06e090 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa2f00098f0 tx=0x7fa2f0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:18.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.267+0000 7fa2f7e00640 1 -- 172.21.15.90:0/2011943504 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa2e803f070 con 0x7fa300145430 2024-03-24T01:27:18.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.466+0000 7fa305e00640 1 -- 172.21.15.90:0/2011943504 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa30015e8d0 con 0x7fa300145430 2024-03-24T01:27:18.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.468+0000 7fa2f7e00640 1 -- 172.21.15.90:0/2011943504 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+16918 (secure 0 0 0) 0x7fa2e8037360 con 0x7fa300145430 2024-03-24T01:27:18.469 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:27:18.469 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":53,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:27:15.534037+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:27:11.774313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-24T01:27:12.992083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:27:18.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.471+0000 7fa305e00640 1 -- 172.21.15.90:0/2011943504 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa2dc06bbd0 msgr2=0x7fa2dc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:18.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.471+0000 7fa305e00640 1 --2- 172.21.15.90:0/2011943504 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa2dc06bbd0 0x7fa2dc06e090 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa2f00098f0 tx=0x7fa2f0008040 comp rx=0 tx=0).stop 2024-03-24T01:27:18.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.471+0000 7fa305e00640 1 -- 172.21.15.90:0/2011943504 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa300145430 msgr2=0x7fa3000ac170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:18.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.471+0000 7fa305e00640 1 --2- 172.21.15.90:0/2011943504 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa300145430 0x7fa3000ac170 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fa2e802b9e0 tx=0x7fa2e80046c0 comp rx=0 tx=0).stop 2024-03-24T01:27:18.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.471+0000 7fa305e00640 1 -- 172.21.15.90:0/2011943504 shutdown_connections 2024-03-24T01:27:18.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.471+0000 7fa305e00640 1 --2- 172.21.15.90:0/2011943504 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa2dc06bbd0 0x7fa2dc06e090 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:18.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.471+0000 7fa305e00640 1 --2- 172.21.15.90:0/2011943504 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa300145430 0x7fa3000ac170 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:18.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.471+0000 7fa305e00640 1 --2- 172.21.15.90:0/2011943504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3001603e0 0x7fa300142fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:18.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.471+0000 7fa305e00640 1 --2- 172.21.15.90:0/2011943504 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa3000cc150 0x7fa3001f4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:18.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.471+0000 7fa305e00640 1 -- 172.21.15.90:0/2011943504 >> 172.21.15.90:0/2011943504 conn(0x7fa3000b1770 msgr2=0x7fa300144b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:18.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.471+0000 7fa305e00640 1 -- 172.21.15.90:0/2011943504 shutdown_connections 2024-03-24T01:27:18.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.471+0000 7fa305e00640 1 -- 172.21.15.90:0/2011943504 wait complete. 2024-03-24T01:27:18.490 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-24T01:27:18.490 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T01:27:18.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.699+0000 7f6e94000640 1 Processor -- start 2024-03-24T01:27:18.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.699+0000 7f6e94000640 1 -- start start 2024-03-24T01:27:18.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.700+0000 7f6e94000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6e8c06fd00 0x7f6e8c06db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:18.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.700+0000 7f6e94000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e8c0702e0 con 0x7f6e8c070710 2024-03-24T01:27:18.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.700+0000 7f6e94000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e8c0cc1d0 con 0x7f6e8c09aca0 2024-03-24T01:27:18.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.700+0000 7f6e94000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e8c06c1d0 con 0x7f6e8c06fd00 2024-03-24T01:27:18.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.701+0000 7f6e91600640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f6e8c070710 0x7f6e8c070af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60314/0 (socket says 172.21.15.90:60314) 2024-03-24T01:27:18.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.701+0000 7f6e91600640 1 -- 172.21.15.90:0/901828782 learned_addr learned my addr 172.21.15.90:0/901828782 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:27:18.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.701+0000 7f6e8be00640 1 -- 172.21.15.90:0/901828782 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 534828253 0 0) 0x7f6e8c0702e0 con 0x7f6e8c070710 2024-03-24T01:27:18.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.701+0000 7f6e8be00640 1 -- 172.21.15.90:0/901828782 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e74003520 con 0x7f6e8c070710 2024-03-24T01:27:18.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.701+0000 7f6e92000640 1 --2- 172.21.15.90:0/901828782 >> v2:172.21.15.120:3300/0 conn(0x7f6e8c06fd00 0x7f6e8c06db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:18.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.701+0000 7f6e8be00640 1 -- 172.21.15.90:0/901828782 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3547880739 0 0) 0x7f6e74003520 con 0x7f6e8c070710 2024-03-24T01:27:18.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.701+0000 7f6e8be00640 1 -- 172.21.15.90:0/901828782 >> v2:172.21.15.120:3300/0 conn(0x7f6e8c06fd00 msgr2=0x7f6e8c06db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:18.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.701+0000 7f6e8be00640 1 --2- 172.21.15.90:0/901828782 >> v2:172.21.15.120:3300/0 conn(0x7f6e8c06fd00 0x7f6e8c06db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:18.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.701+0000 7f6e8be00640 1 -- 172.21.15.90:0/901828782 >> v1:172.21.15.120:6789/0 conn(0x7f6e8c09aca0 legacy=0x7f6e8c06d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:18.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.701+0000 7f6e8be00640 1 -- 172.21.15.90:0/901828782 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e8c0cc4b0 con 0x7f6e8c070710 2024-03-24T01:27:18.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.702+0000 7f6e8be00640 1 -- 172.21.15.90:0/901828782 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f6e80001670 con 0x7f6e8c070710 2024-03-24T01:27:18.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.702+0000 7f6e8be00640 1 -- 172.21.15.90:0/901828782 >> v1:172.21.15.90:6789/0 conn(0x7f6e8c070710 legacy=0x7f6e8c070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:18.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.702+0000 7f6e8be00640 1 --2- 172.21.15.90:0/901828782 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e740039b0 0x7f6e74003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:18.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.702+0000 7f6e8be00640 1 --2- 172.21.15.90:0/901828782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e74004800 0x7f6e74004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:18.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.702+0000 7f6e8be00640 1 --2- 172.21.15.90:0/901828782 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e740052e0 0x7f6e7400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:18.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.702+0000 7f6e8be00640 1 -- 172.21.15.90:0/901828782 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7400c180 con 0x7f6e740039b0 2024-03-24T01:27:18.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.702+0000 7f6e8be00640 1 -- 172.21.15.90:0/901828782 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7400c330 con 0x7f6e74004800 2024-03-24T01:27:18.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.702+0000 7f6e8be00640 1 -- 172.21.15.90:0/901828782 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7400c530 con 0x7f6e740052e0 2024-03-24T01:27:18.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.702+0000 7f6e91600640 1 --2- 172.21.15.90:0/901828782 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e740039b0 0x7f6e74003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:18.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.702+0000 7f6e90c00640 1 --2- 172.21.15.90:0/901828782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e74004800 0x7f6e74004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:18.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.702+0000 7f6e91600640 1 -- 172.21.15.90:0/901828782 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e740052e0 msgr2=0x7f6e7400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:18.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.702+0000 7f6e91600640 1 --2- 172.21.15.90:0/901828782 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e740052e0 0x7f6e7400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:18.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.702+0000 7f6e91600640 1 -- 172.21.15.90:0/901828782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e74004800 msgr2=0x7f6e74004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:18.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.702+0000 7f6e91600640 1 --2- 172.21.15.90:0/901828782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e74004800 0x7f6e74004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:18.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.702+0000 7f6e92000640 1 -- 172.21.15.90:0/901828782 reap_dead start 2024-03-24T01:27:18.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.702+0000 7f6e91600640 1 -- 172.21.15.90:0/901828782 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e74003520 con 0x7f6e740039b0 2024-03-24T01:27:18.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.702+0000 7f6e90c00640 1 --2- 172.21.15.90:0/901828782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e74004800 0x7f6e74004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:27:18.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.703+0000 7f6e91600640 1 --2- 172.21.15.90:0/901828782 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e740039b0 0x7f6e74003dd0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f6e80001eb0 tx=0x7f6e74005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac177b92c3086d8f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:18.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.704+0000 7f6e8be00640 1 -- 172.21.15.90:0/901828782 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e80007d30 con 0x7f6e740039b0 2024-03-24T01:27:18.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.704+0000 7f6e8be00640 1 -- 172.21.15.90:0/901828782 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e80030070 con 0x7f6e740039b0 2024-03-24T01:27:18.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.704+0000 7f6e8be00640 1 -- 172.21.15.90:0/901828782 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e80005d70 con 0x7f6e740039b0 2024-03-24T01:27:18.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.704+0000 7f6e94000640 1 -- 172.21.15.90:0/901828782 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e740039b0 msgr2=0x7f6e74003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:18.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.704+0000 7f6e94000640 1 --2- 172.21.15.90:0/901828782 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e740039b0 0x7f6e74003dd0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f6e80001eb0 tx=0x7f6e74005830 comp rx=0 tx=0).stop 2024-03-24T01:27:18.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.704+0000 7f6e94000640 1 -- 172.21.15.90:0/901828782 shutdown_connections 2024-03-24T01:27:18.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.704+0000 7f6e94000640 1 --2- 172.21.15.90:0/901828782 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e740039b0 0x7f6e74003dd0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:18.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.704+0000 7f6e94000640 1 -- 172.21.15.90:0/901828782 >> 172.21.15.90:0/901828782 conn(0x7f6e8c0b34d0 msgr2=0x7f6e8c0b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:18.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.704+0000 7f6e94000640 1 -- 172.21.15.90:0/901828782 shutdown_connections 2024-03-24T01:27:18.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.704+0000 7f6e94000640 1 -- 172.21.15.90:0/901828782 wait complete. 2024-03-24T01:27:18.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.706+0000 7f6e94000640 1 Processor -- start 2024-03-24T01:27:18.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.706+0000 7f6e94000640 1 -- start start 2024-03-24T01:27:18.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.707+0000 7f6e94000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e740039b0 0x7f6e8c1575f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:18.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.707+0000 7f6e94000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e8c160330 0x7f6e8c0b65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:18.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.707+0000 7f6e94000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e8c0b6af0 0x7f6e8c0ca790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:18.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.707+0000 7f6e94000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8c15e140 con 0x7f6e8c160330 2024-03-24T01:27:18.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.707+0000 7f6e94000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8c0d0f30 con 0x7f6e740039b0 2024-03-24T01:27:18.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.707+0000 7f6e94000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8c15e2c0 con 0x7f6e8c0b6af0 2024-03-24T01:27:18.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.707+0000 7f6e90c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e8c160330 0x7f6e8c0b65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:18.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.707+0000 7f6e90c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e8c160330 0x7f6e8c0b65b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58974/0 (socket says 172.21.15.90:58974) 2024-03-24T01:27:18.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.707+0000 7f6e90c00640 1 -- 172.21.15.90:0/1834450225 learned_addr learned my addr 172.21.15.90:0/1834450225 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:18.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.708+0000 7f6e91600640 1 --2- 172.21.15.90:0/1834450225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e740039b0 0x7f6e8c1575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:18.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.708+0000 7f6e92000640 1 --2- 172.21.15.90:0/1834450225 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e8c0b6af0 0x7f6e8c0ca790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:18.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.708+0000 7f6e90c00640 1 -- 172.21.15.90:0/1834450225 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e8c0b6af0 msgr2=0x7f6e8c0ca790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:18.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.708+0000 7f6e90c00640 1 --2- 172.21.15.90:0/1834450225 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e8c0b6af0 0x7f6e8c0ca790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:18.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.708+0000 7f6e90c00640 1 -- 172.21.15.90:0/1834450225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e740039b0 msgr2=0x7f6e8c1575f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:18.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.708+0000 7f6e90c00640 1 --2- 172.21.15.90:0/1834450225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e740039b0 0x7f6e8c1575f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:18.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.708+0000 7f6e90c00640 1 -- 172.21.15.90:0/1834450225 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e8c15f2d0 con 0x7f6e8c160330 2024-03-24T01:27:18.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.708+0000 7f6e91600640 1 --2- 172.21.15.90:0/1834450225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e740039b0 0x7f6e8c1575f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:27:18.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.708+0000 7f6e90c00640 1 --2- 172.21.15.90:0/1834450225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e8c160330 0x7f6e8c0b65b0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f6e7800d430 tx=0x7f6e7800d8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:18.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.708+0000 7f6e89600640 1 -- 172.21.15.90:0/1834450225 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e78018070 con 0x7f6e8c160330 2024-03-24T01:27:18.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.708+0000 7f6e89600640 1 -- 172.21.15.90:0/1834450225 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e78003610 con 0x7f6e8c160330 2024-03-24T01:27:18.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.708+0000 7f6e89600640 1 -- 172.21.15.90:0/1834450225 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e780039e0 con 0x7f6e8c160330 2024-03-24T01:27:18.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.708+0000 7f6e94000640 1 -- 172.21.15.90:0/1834450225 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e8c157b30 con 0x7f6e8c160330 2024-03-24T01:27:18.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.708+0000 7f6e94000640 1 -- 172.21.15.90:0/1834450225 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e8c15f850 con 0x7f6e8c160330 2024-03-24T01:27:18.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.709+0000 7f6e94000640 1 -- 172.21.15.90:0/1834450225 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e8c09fb10 con 0x7f6e8c160330 2024-03-24T01:27:18.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.710+0000 7f6e89600640 1 -- 172.21.15.90:0/1834450225 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6e7801b070 con 0x7f6e8c160330 2024-03-24T01:27:18.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.711+0000 7f6e89600640 1 --2- 172.21.15.90:0/1834450225 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6e6c06bbd0 0x7f6e6c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:18.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.711+0000 7f6e89600640 1 -- 172.21.15.90:0/1834450225 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6e7808a2e0 con 0x7f6e8c160330 2024-03-24T01:27:18.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.712+0000 7f6e91600640 1 --2- 172.21.15.90:0/1834450225 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6e6c06bbd0 0x7f6e6c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:18.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.713+0000 7f6e91600640 1 --2- 172.21.15.90:0/1834450225 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6e6c06bbd0 0x7f6e6c06e090 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f6e80034000 tx=0x7f6e80039040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:18.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.715+0000 7f6e89600640 1 -- 172.21.15.90:0/1834450225 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6e78058550 con 0x7f6e8c160330 2024-03-24T01:27:18.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.911+0000 7f6e94000640 1 -- 172.21.15.90:0/1834450225 --> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6e8c0b6ed0 con 0x7f6e6c06bbd0 2024-03-24T01:27:18.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.931+0000 7f6e89600640 1 -- 172.21.15.90:0/1834450225 <== mgr.4104 v2:172.21.15.120:6832/860392634 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+250916 (secure 0 0 0) 0x7f6e8c0b6ed0 con 0x7f6e6c06bbd0 2024-03-24T01:27:18.935 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:27:18.937 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-24T01:27:18.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.936+0000 7f6e94000640 1 -- 172.21.15.90:0/1834450225 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6e6c06bbd0 msgr2=0x7f6e6c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:18.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.936+0000 7f6e94000640 1 --2- 172.21.15.90:0/1834450225 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6e6c06bbd0 0x7f6e6c06e090 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f6e80034000 tx=0x7f6e80039040 comp rx=0 tx=0).stop 2024-03-24T01:27:18.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.936+0000 7f6e94000640 1 -- 172.21.15.90:0/1834450225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e8c160330 msgr2=0x7f6e8c0b65b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:18.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.936+0000 7f6e94000640 1 --2- 172.21.15.90:0/1834450225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e8c160330 0x7f6e8c0b65b0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f6e7800d430 tx=0x7f6e7800d8f0 comp rx=0 tx=0).stop 2024-03-24T01:27:18.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.936+0000 7f6e94000640 1 -- 172.21.15.90:0/1834450225 shutdown_connections 2024-03-24T01:27:18.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.936+0000 7f6e94000640 1 --2- 172.21.15.90:0/1834450225 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6e6c06bbd0 0x7f6e6c06e090 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:18.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.936+0000 7f6e94000640 1 --2- 172.21.15.90:0/1834450225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e8c160330 0x7f6e8c0b65b0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:18.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.936+0000 7f6e94000640 1 --2- 172.21.15.90:0/1834450225 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e8c0b6af0 0x7f6e8c0ca790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:18.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.936+0000 7f6e94000640 1 --2- 172.21.15.90:0/1834450225 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e740039b0 0x7f6e8c1575f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:18.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.936+0000 7f6e94000640 1 -- 172.21.15.90:0/1834450225 >> 172.21.15.90:0/1834450225 conn(0x7f6e8c0b34d0 msgr2=0x7f6e8c0b2be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:18.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.936+0000 7f6e94000640 1 -- 172.21.15.90:0/1834450225 shutdown_connections 2024-03-24T01:27:18.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:18.936+0000 7f6e94000640 1 -- 172.21.15.90:0/1834450225 wait complete. 2024-03-24T01:27:18.957 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":145,"stamp":"2024-03-24T01:27:17.549064+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":179,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132,"ondisk_log_size":132,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222104,"kb_used_data":5908,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752616,"statfs":{"total":773094113280,"available":772866678784,"internally_reserved":0,"allocated":6049792,"data_stored":3354391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12747,"internal_metadata":221236789},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":672,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.062753"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.547093+0000","last_change":"2024-03-24T01:27:14.540765+0000","last_active":"2024-03-24T01:27:14.547093+0000","last_peered":"2024-03-24T01:27:14.547093+0000","last_clean":"2024-03-24T01:27:14.547093+0000","last_became_active":"2024-03-24T01:27:14.539541+0000","last_became_peered":"2024-03-24T01:27:14.539541+0000","last_unstale":"2024-03-24T01:27:14.547093+0000","last_undegraded":"2024-03-24T01:27:14.547093+0000","last_fullsized":"2024-03-24T01:27:14.547093+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:30:50.055523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.559986+0000","last_change":"2024-03-24T01:27:14.559986+0000","last_active":"2024-03-24T01:27:14.559986+0000","last_peered":"2024-03-24T01:27:14.559986+0000","last_clean":"2024-03-24T01:27:14.559986+0000","last_became_active":"2024-03-24T01:27:14.559649+0000","last_became_peered":"2024-03-24T01:27:14.559649+0000","last_unstale":"2024-03-24T01:27:14.559986+0000","last_undegraded":"2024-03-24T01:27:14.559986+0000","last_fullsized":"2024-03-24T01:27:14.559986+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:23.971971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.538645+0000","last_change":"2024-03-24T01:27:14.538645+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.538645+0000","last_undegraded":"2024-03-24T01:27:14.538645+0000","last_fullsized":"2024-03-24T01:27:14.538645+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.559476+0000","last_change":"2024-03-24T01:27:14.559476+0000","last_active":"2024-03-24T01:27:14.559476+0000","last_peered":"2024-03-24T01:27:14.559476+0000","last_clean":"2024-03-24T01:27:14.559476+0000","last_became_active":"2024-03-24T01:27:14.559176+0000","last_became_peered":"2024-03-24T01:27:14.559176+0000","last_unstale":"2024-03-24T01:27:14.559476+0000","last_undegraded":"2024-03-24T01:27:14.559476+0000","last_fullsized":"2024-03-24T01:27:14.559476+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:03.448771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.531210+0000","last_change":"2024-03-24T01:27:14.531210+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.531210+0000","last_undegraded":"2024-03-24T01:27:14.531210+0000","last_fullsized":"2024-03-24T01:27:14.531210+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.555238+0000","last_change":"2024-03-24T01:27:14.551494+0000","last_active":"2024-03-24T01:27:14.555238+0000","last_peered":"2024-03-24T01:27:14.555238+0000","last_clean":"2024-03-24T01:27:14.555238+0000","last_became_active":"2024-03-24T01:27:14.551175+0000","last_became_peered":"2024-03-24T01:27:14.551175+0000","last_unstale":"2024-03-24T01:27:14.555238+0000","last_undegraded":"2024-03-24T01:27:14.555238+0000","last_fullsized":"2024-03-24T01:27:14.555238+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:06:34.262662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.552294+0000","last_change":"2024-03-24T01:27:14.552294+0000","last_active":"2024-03-24T01:27:14.552294+0000","last_peered":"2024-03-24T01:27:14.552294+0000","last_clean":"2024-03-24T01:27:14.552294+0000","last_became_active":"2024-03-24T01:27:14.551839+0000","last_became_peered":"2024-03-24T01:27:14.551839+0000","last_unstale":"2024-03-24T01:27:14.552294+0000","last_undegraded":"2024-03-24T01:27:14.552294+0000","last_fullsized":"2024-03-24T01:27:14.552294+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:01:09.219193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.562359+0000","last_change":"2024-03-24T01:27:14.558536+0000","last_active":"2024-03-24T01:27:14.562359+0000","last_peered":"2024-03-24T01:27:14.562359+0000","last_clean":"2024-03-24T01:27:14.562359+0000","last_became_active":"2024-03-24T01:27:14.558211+0000","last_became_peered":"2024-03-24T01:27:14.558211+0000","last_unstale":"2024-03-24T01:27:14.562359+0000","last_undegraded":"2024-03-24T01:27:14.562359+0000","last_fullsized":"2024-03-24T01:27:14.562359+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:44:39.116139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.539698+0000","last_change":"2024-03-24T01:27:14.539698+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.539698+0000","last_undegraded":"2024-03-24T01:27:14.539698+0000","last_fullsized":"2024-03-24T01:27:14.539698+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.547500+0000","last_change":"2024-03-24T01:27:14.541578+0000","last_active":"2024-03-24T01:27:14.547500+0000","last_peered":"2024-03-24T01:27:14.547500+0000","last_clean":"2024-03-24T01:27:14.547500+0000","last_became_active":"2024-03-24T01:27:14.541289+0000","last_became_peered":"2024-03-24T01:27:14.541289+0000","last_unstale":"2024-03-24T01:27:14.547500+0000","last_undegraded":"2024-03-24T01:27:14.547500+0000","last_fullsized":"2024-03-24T01:27:14.547500+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:40.646803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.528427+0000","last_change":"2024-03-24T01:27:14.528427+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.528427+0000","last_undegraded":"2024-03-24T01:27:14.528427+0000","last_fullsized":"2024-03-24T01:27:14.528427+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.558861+0000","last_change":"2024-03-24T01:27:14.552072+0000","last_active":"2024-03-24T01:27:14.558861+0000","last_peered":"2024-03-24T01:27:14.558861+0000","last_clean":"2024-03-24T01:27:14.558861+0000","last_became_active":"2024-03-24T01:27:14.550084+0000","last_became_peered":"2024-03-24T01:27:14.550084+0000","last_unstale":"2024-03-24T01:27:14.558861+0000","last_undegraded":"2024-03-24T01:27:14.558861+0000","last_fullsized":"2024-03-24T01:27:14.558861+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:45:39.507934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.550970+0000","last_change":"2024-03-24T01:27:14.544185+0000","last_active":"2024-03-24T01:27:14.550970+0000","last_peered":"2024-03-24T01:27:14.550970+0000","last_clean":"2024-03-24T01:27:14.550970+0000","last_became_active":"2024-03-24T01:27:14.543895+0000","last_became_peered":"2024-03-24T01:27:14.543895+0000","last_unstale":"2024-03-24T01:27:14.550970+0000","last_undegraded":"2024-03-24T01:27:14.550970+0000","last_fullsized":"2024-03-24T01:27:14.550970+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:47:39.461292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.550120+0000","last_change":"2024-03-24T01:27:14.550120+0000","last_active":"2024-03-24T01:27:14.550120+0000","last_peered":"2024-03-24T01:27:14.550120+0000","last_clean":"2024-03-24T01:27:14.550120+0000","last_became_active":"2024-03-24T01:27:14.549329+0000","last_became_peered":"2024-03-24T01:27:14.549329+0000","last_unstale":"2024-03-24T01:27:14.550120+0000","last_undegraded":"2024-03-24T01:27:14.550120+0000","last_fullsized":"2024-03-24T01:27:14.550120+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:32:58.788010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.557855+0000","last_change":"2024-03-24T01:27:14.557855+0000","last_active":"2024-03-24T01:27:14.557855+0000","last_peered":"2024-03-24T01:27:14.557855+0000","last_clean":"2024-03-24T01:27:14.557855+0000","last_became_active":"2024-03-24T01:27:14.557606+0000","last_became_peered":"2024-03-24T01:27:14.557606+0000","last_unstale":"2024-03-24T01:27:14.557855+0000","last_undegraded":"2024-03-24T01:27:14.557855+0000","last_fullsized":"2024-03-24T01:27:14.557855+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:28:36.027521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.550788+0000","last_change":"2024-03-24T01:27:14.542073+0000","last_active":"2024-03-24T01:27:14.550788+0000","last_peered":"2024-03-24T01:27:14.550788+0000","last_clean":"2024-03-24T01:27:14.550788+0000","last_became_active":"2024-03-24T01:27:14.540636+0000","last_became_peered":"2024-03-24T01:27:14.540636+0000","last_unstale":"2024-03-24T01:27:14.550788+0000","last_undegraded":"2024-03-24T01:27:14.550788+0000","last_fullsized":"2024-03-24T01:27:14.550788+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:02:12.837672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.556042+0000","last_change":"2024-03-24T01:27:14.556042+0000","last_active":"2024-03-24T01:27:14.556042+0000","last_peered":"2024-03-24T01:27:14.556042+0000","last_clean":"2024-03-24T01:27:14.556042+0000","last_became_active":"2024-03-24T01:27:14.555743+0000","last_became_peered":"2024-03-24T01:27:14.555743+0000","last_unstale":"2024-03-24T01:27:14.556042+0000","last_undegraded":"2024-03-24T01:27:14.556042+0000","last_fullsized":"2024-03-24T01:27:14.556042+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:14:16.916597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.561148+0000","last_change":"2024-03-24T01:27:14.561148+0000","last_active":"2024-03-24T01:27:14.561148+0000","last_peered":"2024-03-24T01:27:14.561148+0000","last_clean":"2024-03-24T01:27:14.561148+0000","last_became_active":"2024-03-24T01:27:14.560509+0000","last_became_peered":"2024-03-24T01:27:14.560509+0000","last_unstale":"2024-03-24T01:27:14.561148+0000","last_undegraded":"2024-03-24T01:27:14.561148+0000","last_fullsized":"2024-03-24T01:27:14.561148+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:09:58.411238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.553533+0000","last_change":"2024-03-24T01:27:14.553533+0000","last_active":"2024-03-24T01:27:14.553533+0000","last_peered":"2024-03-24T01:27:14.553533+0000","last_clean":"2024-03-24T01:27:14.553533+0000","last_became_active":"2024-03-24T01:27:14.553252+0000","last_became_peered":"2024-03-24T01:27:14.553252+0000","last_unstale":"2024-03-24T01:27:14.553533+0000","last_undegraded":"2024-03-24T01:27:14.553533+0000","last_fullsized":"2024-03-24T01:27:14.553533+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:46:45.759885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.546096+0000","last_change":"2024-03-24T01:27:14.541674+0000","last_active":"2024-03-24T01:27:14.546096+0000","last_peered":"2024-03-24T01:27:14.546096+0000","last_clean":"2024-03-24T01:27:14.546096+0000","last_became_active":"2024-03-24T01:27:14.541300+0000","last_became_peered":"2024-03-24T01:27:14.541300+0000","last_unstale":"2024-03-24T01:27:14.546096+0000","last_undegraded":"2024-03-24T01:27:14.546096+0000","last_fullsized":"2024-03-24T01:27:14.546096+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:12:15.274601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.527732+0000","last_change":"2024-03-24T01:27:14.527732+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.527732+0000","last_undegraded":"2024-03-24T01:27:14.527732+0000","last_fullsized":"2024-03-24T01:27:14.527732+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.558260+0000","last_change":"2024-03-24T01:27:14.554032+0000","last_active":"2024-03-24T01:27:14.558260+0000","last_peered":"2024-03-24T01:27:14.558260+0000","last_clean":"2024-03-24T01:27:14.558260+0000","last_became_active":"2024-03-24T01:27:14.553732+0000","last_became_peered":"2024-03-24T01:27:14.553732+0000","last_unstale":"2024-03-24T01:27:14.558260+0000","last_undegraded":"2024-03-24T01:27:14.558260+0000","last_fullsized":"2024-03-24T01:27:14.558260+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:04:45.650427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.551791+0000","last_change":"2024-03-24T01:27:14.551791+0000","last_active":"2024-03-24T01:27:14.551791+0000","last_peered":"2024-03-24T01:27:14.551791+0000","last_clean":"2024-03-24T01:27:14.551791+0000","last_became_active":"2024-03-24T01:27:14.551503+0000","last_became_peered":"2024-03-24T01:27:14.551503+0000","last_unstale":"2024-03-24T01:27:14.551791+0000","last_undegraded":"2024-03-24T01:27:14.551791+0000","last_fullsized":"2024-03-24T01:27:14.551791+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:18:59.684453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.556979+0000","last_change":"2024-03-24T01:27:14.545843+0000","last_active":"2024-03-24T01:27:14.556979+0000","last_peered":"2024-03-24T01:27:14.556979+0000","last_clean":"2024-03-24T01:27:14.556979+0000","last_became_active":"2024-03-24T01:27:14.545534+0000","last_became_peered":"2024-03-24T01:27:14.545534+0000","last_unstale":"2024-03-24T01:27:14.556979+0000","last_undegraded":"2024-03-24T01:27:14.556979+0000","last_fullsized":"2024-03-24T01:27:14.556979+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:06:34.262662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.548640+0000","last_change":"2024-03-24T01:27:14.548640+0000","last_active":"2024-03-24T01:27:14.548640+0000","last_peered":"2024-03-24T01:27:14.548640+0000","last_clean":"2024-03-24T01:27:14.548640+0000","last_became_active":"2024-03-24T01:27:14.548364+0000","last_became_peered":"2024-03-24T01:27:14.548364+0000","last_unstale":"2024-03-24T01:27:14.548640+0000","last_undegraded":"2024-03-24T01:27:14.548640+0000","last_fullsized":"2024-03-24T01:27:14.548640+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:28:36.027521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.544765+0000","last_change":"2024-03-24T01:27:14.541461+0000","last_active":"2024-03-24T01:27:14.544765+0000","last_peered":"2024-03-24T01:27:14.544765+0000","last_clean":"2024-03-24T01:27:14.544765+0000","last_became_active":"2024-03-24T01:27:14.541156+0000","last_became_peered":"2024-03-24T01:27:14.541156+0000","last_unstale":"2024-03-24T01:27:14.544765+0000","last_undegraded":"2024-03-24T01:27:14.544765+0000","last_fullsized":"2024-03-24T01:27:14.544765+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:14:09.820632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.556242+0000","last_change":"2024-03-24T01:27:14.556242+0000","last_active":"2024-03-24T01:27:14.556242+0000","last_peered":"2024-03-24T01:27:14.556242+0000","last_clean":"2024-03-24T01:27:14.556242+0000","last_became_active":"2024-03-24T01:27:14.552642+0000","last_became_peered":"2024-03-24T01:27:14.552642+0000","last_unstale":"2024-03-24T01:27:14.556242+0000","last_undegraded":"2024-03-24T01:27:14.556242+0000","last_fullsized":"2024-03-24T01:27:14.556242+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:28:36.027521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.546588+0000","last_change":"2024-03-24T01:27:14.542335+0000","last_active":"2024-03-24T01:27:14.546588+0000","last_peered":"2024-03-24T01:27:14.546588+0000","last_clean":"2024-03-24T01:27:14.546588+0000","last_became_active":"2024-03-24T01:27:14.541006+0000","last_became_peered":"2024-03-24T01:27:14.541006+0000","last_unstale":"2024-03-24T01:27:14.546588+0000","last_undegraded":"2024-03-24T01:27:14.546588+0000","last_fullsized":"2024-03-24T01:27:14.546588+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:44.340192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.550220+0000","last_change":"2024-03-24T01:27:14.550220+0000","last_active":"2024-03-24T01:27:14.550220+0000","last_peered":"2024-03-24T01:27:14.550220+0000","last_clean":"2024-03-24T01:27:14.550220+0000","last_became_active":"2024-03-24T01:27:14.549812+0000","last_became_peered":"2024-03-24T01:27:14.549812+0000","last_unstale":"2024-03-24T01:27:14.550220+0000","last_undegraded":"2024-03-24T01:27:14.550220+0000","last_fullsized":"2024-03-24T01:27:14.550220+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:09:58.411238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.556235+0000","last_change":"2024-03-24T01:27:14.556235+0000","last_active":"2024-03-24T01:27:14.556235+0000","last_peered":"2024-03-24T01:27:14.556235+0000","last_clean":"2024-03-24T01:27:14.556235+0000","last_became_active":"2024-03-24T01:27:14.555888+0000","last_became_peered":"2024-03-24T01:27:14.555888+0000","last_unstale":"2024-03-24T01:27:14.556235+0000","last_undegraded":"2024-03-24T01:27:14.556235+0000","last_fullsized":"2024-03-24T01:27:14.556235+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:43:45.838352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.552212+0000","last_change":"2024-03-24T01:27:14.552212+0000","last_active":"2024-03-24T01:27:14.552212+0000","last_peered":"2024-03-24T01:27:14.552212+0000","last_clean":"2024-03-24T01:27:14.552212+0000","last_became_active":"2024-03-24T01:27:14.551775+0000","last_became_peered":"2024-03-24T01:27:14.551775+0000","last_unstale":"2024-03-24T01:27:14.552212+0000","last_undegraded":"2024-03-24T01:27:14.552212+0000","last_fullsized":"2024-03-24T01:27:14.552212+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:23.971971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.550504+0000","last_change":"2024-03-24T01:27:14.542056+0000","last_active":"2024-03-24T01:27:14.550504+0000","last_peered":"2024-03-24T01:27:14.550504+0000","last_clean":"2024-03-24T01:27:14.550504+0000","last_became_active":"2024-03-24T01:27:14.541695+0000","last_became_peered":"2024-03-24T01:27:14.541695+0000","last_unstale":"2024-03-24T01:27:14.550504+0000","last_undegraded":"2024-03-24T01:27:14.550504+0000","last_fullsized":"2024-03-24T01:27:14.550504+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:32.037863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.559666+0000","last_change":"2024-03-24T01:27:14.559666+0000","last_active":"2024-03-24T01:27:14.559666+0000","last_peered":"2024-03-24T01:27:14.559666+0000","last_clean":"2024-03-24T01:27:14.559666+0000","last_became_active":"2024-03-24T01:27:14.559356+0000","last_became_peered":"2024-03-24T01:27:14.559356+0000","last_unstale":"2024-03-24T01:27:14.559666+0000","last_undegraded":"2024-03-24T01:27:14.559666+0000","last_fullsized":"2024-03-24T01:27:14.559666+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:32.037863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.529333+0000","last_change":"2024-03-24T01:27:13.522918+0000","last_active":"2024-03-24T01:27:14.529333+0000","last_peered":"2024-03-24T01:27:14.529333+0000","last_clean":"2024-03-24T01:27:14.529333+0000","last_became_active":"2024-03-24T01:27:13.520866+0000","last_became_peered":"2024-03-24T01:27:13.520866+0000","last_unstale":"2024-03-24T01:27:14.529333+0000","last_undegraded":"2024-03-24T01:27:14.529333+0000","last_fullsized":"2024-03-24T01:27:14.529333+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:30.183145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.557567+0000","last_change":"2024-03-24T01:27:14.551119+0000","last_active":"2024-03-24T01:27:14.557567+0000","last_peered":"2024-03-24T01:27:14.557567+0000","last_clean":"2024-03-24T01:27:14.557567+0000","last_became_active":"2024-03-24T01:27:14.550800+0000","last_became_peered":"2024-03-24T01:27:14.550800+0000","last_unstale":"2024-03-24T01:27:14.557567+0000","last_undegraded":"2024-03-24T01:27:14.557567+0000","last_fullsized":"2024-03-24T01:27:14.557567+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:09:58.411238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.518296+0000","last_change":"2024-03-24T01:27:13.531689+0000","last_active":"2024-03-24T01:27:14.518296+0000","last_peered":"2024-03-24T01:27:14.518296+0000","last_clean":"2024-03-24T01:27:14.518296+0000","last_became_active":"2024-03-24T01:27:13.530929+0000","last_became_peered":"2024-03-24T01:27:13.530929+0000","last_unstale":"2024-03-24T01:27:14.518296+0000","last_undegraded":"2024-03-24T01:27:14.518296+0000","last_fullsized":"2024-03-24T01:27:14.518296+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:00:24.458371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.552432+0000","last_change":"2024-03-24T01:27:14.552432+0000","last_active":"2024-03-24T01:27:14.552432+0000","last_peered":"2024-03-24T01:27:14.552432+0000","last_clean":"2024-03-24T01:27:14.552432+0000","last_became_active":"2024-03-24T01:27:14.551697+0000","last_became_peered":"2024-03-24T01:27:14.551697+0000","last_unstale":"2024-03-24T01:27:14.552432+0000","last_undegraded":"2024-03-24T01:27:14.552432+0000","last_fullsized":"2024-03-24T01:27:14.552432+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:47:39.461292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.518290+0000","last_change":"2024-03-24T01:27:13.528068+0000","last_active":"2024-03-24T01:27:14.518290+0000","last_peered":"2024-03-24T01:27:14.518290+0000","last_clean":"2024-03-24T01:27:14.518290+0000","last_became_active":"2024-03-24T01:27:13.527750+0000","last_became_peered":"2024-03-24T01:27:13.527750+0000","last_unstale":"2024-03-24T01:27:14.518290+0000","last_undegraded":"2024-03-24T01:27:14.518290+0000","last_fullsized":"2024-03-24T01:27:14.518290+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:37:15.291837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.549966+0000","last_change":"2024-03-24T01:27:14.548464+0000","last_active":"2024-03-24T01:27:14.549966+0000","last_peered":"2024-03-24T01:27:14.549966+0000","last_clean":"2024-03-24T01:27:14.549966+0000","last_became_active":"2024-03-24T01:27:14.548113+0000","last_became_peered":"2024-03-24T01:27:14.548113+0000","last_unstale":"2024-03-24T01:27:14.549966+0000","last_undegraded":"2024-03-24T01:27:14.549966+0000","last_fullsized":"2024-03-24T01:27:14.549966+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:47:57.893749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.523180+0000","last_change":"2024-03-24T01:27:13.527296+0000","last_active":"2024-03-24T01:27:14.523180+0000","last_peered":"2024-03-24T01:27:14.523180+0000","last_clean":"2024-03-24T01:27:14.523180+0000","last_became_active":"2024-03-24T01:27:13.526656+0000","last_became_peered":"2024-03-24T01:27:13.526656+0000","last_unstale":"2024-03-24T01:27:14.523180+0000","last_undegraded":"2024-03-24T01:27:14.523180+0000","last_fullsized":"2024-03-24T01:27:14.523180+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:28:35.013061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.531475+0000","last_change":"2024-03-24T01:27:14.531475+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.531475+0000","last_undegraded":"2024-03-24T01:27:14.531475+0000","last_fullsized":"2024-03-24T01:27:14.531475+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.532174+0000","last_change":"2024-03-24T01:27:13.518516+0000","last_active":"2024-03-24T01:27:14.532174+0000","last_peered":"2024-03-24T01:27:14.532174+0000","last_clean":"2024-03-24T01:27:14.532174+0000","last_became_active":"2024-03-24T01:27:13.518276+0000","last_became_peered":"2024-03-24T01:27:13.518276+0000","last_unstale":"2024-03-24T01:27:14.532174+0000","last_undegraded":"2024-03-24T01:27:14.532174+0000","last_fullsized":"2024-03-24T01:27:14.532174+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:39.632343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.559916+0000","last_change":"2024-03-24T01:27:14.559916+0000","last_active":"2024-03-24T01:27:14.559916+0000","last_peered":"2024-03-24T01:27:14.559916+0000","last_clean":"2024-03-24T01:27:14.559916+0000","last_became_active":"2024-03-24T01:27:14.559597+0000","last_became_peered":"2024-03-24T01:27:14.559597+0000","last_unstale":"2024-03-24T01:27:14.559916+0000","last_undegraded":"2024-03-24T01:27:14.559916+0000","last_fullsized":"2024-03-24T01:27:14.559916+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:32:58.788010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.535218+0000","last_change":"2024-03-24T01:27:13.527482+0000","last_active":"2024-03-24T01:27:14.535218+0000","last_peered":"2024-03-24T01:27:14.535218+0000","last_clean":"2024-03-24T01:27:14.535218+0000","last_became_active":"2024-03-24T01:27:13.527205+0000","last_became_peered":"2024-03-24T01:27:13.527205+0000","last_unstale":"2024-03-24T01:27:14.535218+0000","last_undegraded":"2024-03-24T01:27:14.535218+0000","last_fullsized":"2024-03-24T01:27:14.535218+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:39.632343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.544743+0000","last_change":"2024-03-24T01:27:14.541241+0000","last_active":"2024-03-24T01:27:14.544743+0000","last_peered":"2024-03-24T01:27:14.544743+0000","last_clean":"2024-03-24T01:27:14.544743+0000","last_became_active":"2024-03-24T01:27:14.540872+0000","last_became_peered":"2024-03-24T01:27:14.540872+0000","last_unstale":"2024-03-24T01:27:14.544743+0000","last_undegraded":"2024-03-24T01:27:14.544743+0000","last_fullsized":"2024-03-24T01:27:14.544743+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:16:39.371501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.527060+0000","last_change":"2024-03-24T01:27:13.531379+0000","last_active":"2024-03-24T01:27:14.527060+0000","last_peered":"2024-03-24T01:27:14.527060+0000","last_clean":"2024-03-24T01:27:14.527060+0000","last_became_active":"2024-03-24T01:27:13.530618+0000","last_became_peered":"2024-03-24T01:27:13.530618+0000","last_unstale":"2024-03-24T01:27:14.527060+0000","last_undegraded":"2024-03-24T01:27:14.527060+0000","last_fullsized":"2024-03-24T01:27:14.527060+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:00.227179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.548287+0000","last_change":"2024-03-24T01:27:14.548287+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.548287+0000","last_undegraded":"2024-03-24T01:27:14.548287+0000","last_fullsized":"2024-03-24T01:27:14.548287+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.527604+0000","last_change":"2024-03-24T01:27:13.529478+0000","last_active":"2024-03-24T01:27:14.527604+0000","last_peered":"2024-03-24T01:27:14.527604+0000","last_clean":"2024-03-24T01:27:14.527604+0000","last_became_active":"2024-03-24T01:27:13.529197+0000","last_became_peered":"2024-03-24T01:27:13.529197+0000","last_unstale":"2024-03-24T01:27:14.527604+0000","last_undegraded":"2024-03-24T01:27:14.527604+0000","last_fullsized":"2024-03-24T01:27:14.527604+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:30:49.041062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.547194+0000","last_change":"2024-03-24T01:27:14.544605+0000","last_active":"2024-03-24T01:27:14.547194+0000","last_peered":"2024-03-24T01:27:14.547194+0000","last_clean":"2024-03-24T01:27:14.547194+0000","last_became_active":"2024-03-24T01:27:14.544189+0000","last_became_peered":"2024-03-24T01:27:14.544189+0000","last_unstale":"2024-03-24T01:27:14.547194+0000","last_undegraded":"2024-03-24T01:27:14.547194+0000","last_fullsized":"2024-03-24T01:27:14.547194+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:12:15.274601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.548563+0000","last_change":"2024-03-24T01:27:13.533618+0000","last_active":"2024-03-24T01:27:14.548563+0000","last_peered":"2024-03-24T01:27:14.548563+0000","last_clean":"2024-03-24T01:27:14.548563+0000","last_became_active":"2024-03-24T01:27:13.531829+0000","last_became_peered":"2024-03-24T01:27:13.531829+0000","last_unstale":"2024-03-24T01:27:14.548563+0000","last_undegraded":"2024-03-24T01:27:14.548563+0000","last_fullsized":"2024-03-24T01:27:14.548563+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:06:33.248202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.552880+0000","last_change":"2024-03-24T01:27:14.541924+0000","last_active":"2024-03-24T01:27:14.552880+0000","last_peered":"2024-03-24T01:27:14.552880+0000","last_clean":"2024-03-24T01:27:14.552880+0000","last_became_active":"2024-03-24T01:27:14.541512+0000","last_became_peered":"2024-03-24T01:27:14.541512+0000","last_unstale":"2024-03-24T01:27:14.552880+0000","last_undegraded":"2024-03-24T01:27:14.552880+0000","last_fullsized":"2024-03-24T01:27:14.552880+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:03.448771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.536520+0000","last_change":"2024-03-24T01:27:13.524033+0000","last_active":"2024-03-24T01:27:14.536520+0000","last_peered":"2024-03-24T01:27:14.536520+0000","last_clean":"2024-03-24T01:27:14.536520+0000","last_became_active":"2024-03-24T01:27:13.523502+0000","last_became_peered":"2024-03-24T01:27:13.523502+0000","last_unstale":"2024-03-24T01:27:14.536520+0000","last_undegraded":"2024-03-24T01:27:14.536520+0000","last_fullsized":"2024-03-24T01:27:14.536520+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:02:11.823211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.552947+0000","last_change":"2024-03-24T01:27:14.552947+0000","last_active":"2024-03-24T01:27:14.552947+0000","last_peered":"2024-03-24T01:27:14.552947+0000","last_clean":"2024-03-24T01:27:14.552947+0000","last_became_active":"2024-03-24T01:27:14.552583+0000","last_became_peered":"2024-03-24T01:27:14.552583+0000","last_unstale":"2024-03-24T01:27:14.552947+0000","last_undegraded":"2024-03-24T01:27:14.552947+0000","last_fullsized":"2024-03-24T01:27:14.552947+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:58.630025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"52'2","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.548205+0000","last_change":"2024-03-24T01:27:13.530329+0000","last_active":"2024-03-24T01:27:14.548205+0000","last_peered":"2024-03-24T01:27:14.548205+0000","last_clean":"2024-03-24T01:27:14.548205+0000","last_became_active":"2024-03-24T01:27:13.529972+0000","last_became_peered":"2024-03-24T01:27:13.529972+0000","last_unstale":"2024-03-24T01:27:14.548205+0000","last_undegraded":"2024-03-24T01:27:14.548205+0000","last_fullsized":"2024-03-24T01:27:14.548205+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:43.325731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.552486+0000","last_change":"2024-03-24T01:27:14.552486+0000","last_active":"2024-03-24T01:27:14.552486+0000","last_peered":"2024-03-24T01:27:14.552486+0000","last_clean":"2024-03-24T01:27:14.552486+0000","last_became_active":"2024-03-24T01:27:14.551883+0000","last_became_peered":"2024-03-24T01:27:14.551883+0000","last_unstale":"2024-03-24T01:27:14.552486+0000","last_undegraded":"2024-03-24T01:27:14.552486+0000","last_fullsized":"2024-03-24T01:27:14.552486+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:44.340192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"52'5","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.535485+0000","last_change":"2024-03-24T01:27:13.521128+0000","last_active":"2024-03-24T01:27:14.535485+0000","last_peered":"2024-03-24T01:27:14.535485+0000","last_clean":"2024-03-24T01:27:14.535485+0000","last_became_active":"2024-03-24T01:27:13.520358+0000","last_became_peered":"2024-03-24T01:27:13.520358+0000","last_unstale":"2024-03-24T01:27:14.535485+0000","last_undegraded":"2024-03-24T01:27:14.535485+0000","last_fullsized":"2024-03-24T01:27:14.535485+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:01:08.204732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.522002+0000","last_change":"2024-03-24T01:27:14.522002+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.522002+0000","last_undegraded":"2024-03-24T01:27:14.522002+0000","last_fullsized":"2024-03-24T01:27:14.522002+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.532355+0000","last_change":"2024-03-24T01:27:13.532029+0000","last_active":"2024-03-24T01:27:14.532355+0000","last_peered":"2024-03-24T01:27:14.532355+0000","last_clean":"2024-03-24T01:27:14.532355+0000","last_became_active":"2024-03-24T01:27:13.531765+0000","last_became_peered":"2024-03-24T01:27:13.531765+0000","last_unstale":"2024-03-24T01:27:14.532355+0000","last_undegraded":"2024-03-24T01:27:14.532355+0000","last_fullsized":"2024-03-24T01:27:14.532355+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:02:11.823211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.537552+0000","last_change":"2024-03-24T01:27:14.537552+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.537552+0000","last_undegraded":"2024-03-24T01:27:14.537552+0000","last_fullsized":"2024-03-24T01:27:14.537552+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.527543+0000","last_change":"2024-03-24T01:27:13.530136+0000","last_active":"2024-03-24T01:27:14.527543+0000","last_peered":"2024-03-24T01:27:14.527543+0000","last_clean":"2024-03-24T01:27:14.527543+0000","last_became_active":"2024-03-24T01:27:13.529811+0000","last_became_peered":"2024-03-24T01:27:13.529811+0000","last_unstale":"2024-03-24T01:27:14.527543+0000","last_undegraded":"2024-03-24T01:27:14.527543+0000","last_fullsized":"2024-03-24T01:27:14.527543+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:37:15.291837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.563469+0000","last_change":"2024-03-24T01:27:14.561422+0000","last_active":"2024-03-24T01:27:14.563469+0000","last_peered":"2024-03-24T01:27:14.563469+0000","last_clean":"2024-03-24T01:27:14.563469+0000","last_became_active":"2024-03-24T01:27:14.559580+0000","last_became_peered":"2024-03-24T01:27:14.559580+0000","last_unstale":"2024-03-24T01:27:14.563469+0000","last_undegraded":"2024-03-24T01:27:14.563469+0000","last_fullsized":"2024-03-24T01:27:14.563469+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:18:51.495804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.546920+0000","last_change":"2024-03-24T01:27:13.532579+0000","last_active":"2024-03-24T01:27:14.546920+0000","last_peered":"2024-03-24T01:27:14.546920+0000","last_clean":"2024-03-24T01:27:14.546920+0000","last_became_active":"2024-03-24T01:27:13.532264+0000","last_became_peered":"2024-03-24T01:27:13.532264+0000","last_unstale":"2024-03-24T01:27:14.546920+0000","last_undegraded":"2024-03-24T01:27:14.546920+0000","last_fullsized":"2024-03-24T01:27:14.546920+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:12:14.260140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.545505+0000","last_change":"2024-03-24T01:27:14.545505+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.545505+0000","last_undegraded":"2024-03-24T01:27:14.545505+0000","last_fullsized":"2024-03-24T01:27:14.545505+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"52'8","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.536546+0000","last_change":"2024-03-24T01:27:13.518331+0000","last_active":"2024-03-24T01:27:14.536546+0000","last_peered":"2024-03-24T01:27:14.536546+0000","last_clean":"2024-03-24T01:27:14.536546+0000","last_became_active":"2024-03-24T01:27:13.518017+0000","last_became_peered":"2024-03-24T01:27:13.518017+0000","last_unstale":"2024-03-24T01:27:14.536546+0000","last_undegraded":"2024-03-24T01:27:14.536546+0000","last_fullsized":"2024-03-24T01:27:14.536546+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:02:41.582283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.547545+0000","last_change":"2024-03-24T01:27:14.543017+0000","last_active":"2024-03-24T01:27:14.547545+0000","last_peered":"2024-03-24T01:27:14.547545+0000","last_clean":"2024-03-24T01:27:14.547545+0000","last_became_active":"2024-03-24T01:27:14.542730+0000","last_became_peered":"2024-03-24T01:27:14.542730+0000","last_unstale":"2024-03-24T01:27:14.547545+0000","last_undegraded":"2024-03-24T01:27:14.547545+0000","last_fullsized":"2024-03-24T01:27:14.547545+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:19:18.467994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.545934+0000","last_change":"2024-03-24T01:27:13.530251+0000","last_active":"2024-03-24T01:27:14.545934+0000","last_peered":"2024-03-24T01:27:14.545934+0000","last_clean":"2024-03-24T01:27:14.545934+0000","last_became_active":"2024-03-24T01:27:13.529767+0000","last_became_peered":"2024-03-24T01:27:13.529767+0000","last_unstale":"2024-03-24T01:27:14.545934+0000","last_undegraded":"2024-03-24T01:27:14.545934+0000","last_fullsized":"2024-03-24T01:27:14.545934+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:43:44.823891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.557227+0000","last_change":"2024-03-24T01:27:14.554474+0000","last_active":"2024-03-24T01:27:14.557227+0000","last_peered":"2024-03-24T01:27:14.557227+0000","last_clean":"2024-03-24T01:27:14.557227+0000","last_became_active":"2024-03-24T01:27:14.554189+0000","last_became_peered":"2024-03-24T01:27:14.554189+0000","last_unstale":"2024-03-24T01:27:14.557227+0000","last_undegraded":"2024-03-24T01:27:14.557227+0000","last_fullsized":"2024-03-24T01:27:14.557227+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:43:45.838352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.529435+0000","last_change":"2024-03-24T01:27:13.526531+0000","last_active":"2024-03-24T01:27:14.529435+0000","last_peered":"2024-03-24T01:27:14.529435+0000","last_clean":"2024-03-24T01:27:14.529435+0000","last_became_active":"2024-03-24T01:27:13.526212+0000","last_became_peered":"2024-03-24T01:27:13.526212+0000","last_unstale":"2024-03-24T01:27:14.529435+0000","last_undegraded":"2024-03-24T01:27:14.529435+0000","last_fullsized":"2024-03-24T01:27:14.529435+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:02:41.656569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.531995+0000","last_change":"2024-03-24T01:27:14.531995+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.531995+0000","last_undegraded":"2024-03-24T01:27:14.531995+0000","last_fullsized":"2024-03-24T01:27:14.531995+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.529356+0000","last_change":"2024-03-24T01:27:13.526615+0000","last_active":"2024-03-24T01:27:14.529356+0000","last_peered":"2024-03-24T01:27:14.529356+0000","last_clean":"2024-03-24T01:27:14.529356+0000","last_became_active":"2024-03-24T01:27:13.526286+0000","last_became_peered":"2024-03-24T01:27:13.526286+0000","last_unstale":"2024-03-24T01:27:14.529356+0000","last_undegraded":"2024-03-24T01:27:14.529356+0000","last_fullsized":"2024-03-24T01:27:14.529356+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:42:00.908936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.547136+0000","last_change":"2024-03-24T01:27:14.539239+0000","last_active":"2024-03-24T01:27:14.547136+0000","last_peered":"2024-03-24T01:27:14.547136+0000","last_clean":"2024-03-24T01:27:14.547136+0000","last_became_active":"2024-03-24T01:27:14.538525+0000","last_became_peered":"2024-03-24T01:27:14.538525+0000","last_unstale":"2024-03-24T01:27:14.547136+0000","last_undegraded":"2024-03-24T01:27:14.547136+0000","last_fullsized":"2024-03-24T01:27:14.547136+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:42:01.923397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.528640+0000","last_change":"2024-03-24T01:27:13.523480+0000","last_active":"2024-03-24T01:27:14.528640+0000","last_peered":"2024-03-24T01:27:14.528640+0000","last_clean":"2024-03-24T01:27:14.528640+0000","last_became_active":"2024-03-24T01:27:13.523203+0000","last_became_peered":"2024-03-24T01:27:13.523203+0000","last_unstale":"2024-03-24T01:27:14.528640+0000","last_undegraded":"2024-03-24T01:27:14.528640+0000","last_fullsized":"2024-03-24T01:27:14.528640+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:00:24.458371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.557519+0000","last_change":"2024-03-24T01:27:14.549953+0000","last_active":"2024-03-24T01:27:14.557519+0000","last_peered":"2024-03-24T01:27:14.557519+0000","last_clean":"2024-03-24T01:27:14.557519+0000","last_became_active":"2024-03-24T01:27:14.549616+0000","last_became_peered":"2024-03-24T01:27:14.549616+0000","last_unstale":"2024-03-24T01:27:14.557519+0000","last_undegraded":"2024-03-24T01:27:14.557519+0000","last_fullsized":"2024-03-24T01:27:14.557519+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:32:58.788010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.518549+0000","last_change":"2024-03-24T01:27:13.526041+0000","last_active":"2024-03-24T01:27:14.518549+0000","last_peered":"2024-03-24T01:27:14.518549+0000","last_clean":"2024-03-24T01:27:14.518549+0000","last_became_active":"2024-03-24T01:27:13.525587+0000","last_became_peered":"2024-03-24T01:27:13.525587+0000","last_unstale":"2024-03-24T01:27:14.518549+0000","last_undegraded":"2024-03-24T01:27:14.518549+0000","last_fullsized":"2024-03-24T01:27:14.518549+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:30.183145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.546589+0000","last_change":"2024-03-24T01:27:14.539787+0000","last_active":"2024-03-24T01:27:14.546589+0000","last_peered":"2024-03-24T01:27:14.546589+0000","last_clean":"2024-03-24T01:27:14.546589+0000","last_became_active":"2024-03-24T01:27:14.539446+0000","last_became_peered":"2024-03-24T01:27:14.539446+0000","last_unstale":"2024-03-24T01:27:14.546589+0000","last_undegraded":"2024-03-24T01:27:14.546589+0000","last_fullsized":"2024-03-24T01:27:14.546589+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:02:12.837672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.535722+0000","last_change":"2024-03-24T01:27:13.519780+0000","last_active":"2024-03-24T01:27:14.535722+0000","last_peered":"2024-03-24T01:27:14.535722+0000","last_clean":"2024-03-24T01:27:14.535722+0000","last_became_active":"2024-03-24T01:27:13.519458+0000","last_became_peered":"2024-03-24T01:27:13.519458+0000","last_unstale":"2024-03-24T01:27:14.535722+0000","last_undegraded":"2024-03-24T01:27:14.535722+0000","last_fullsized":"2024-03-24T01:27:14.535722+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:46:44.745424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.549326+0000","last_change":"2024-03-24T01:27:14.549326+0000","last_active":"2024-03-24T01:27:14.549326+0000","last_peered":"2024-03-24T01:27:14.549326+0000","last_clean":"2024-03-24T01:27:14.549326+0000","last_became_active":"2024-03-24T01:27:14.548886+0000","last_became_peered":"2024-03-24T01:27:14.548886+0000","last_unstale":"2024-03-24T01:27:14.549326+0000","last_undegraded":"2024-03-24T01:27:14.549326+0000","last_fullsized":"2024-03-24T01:27:14.549326+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:03.448771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.545176+0000","last_change":"2024-03-24T01:27:13.521689+0000","last_active":"2024-03-24T01:27:14.545176+0000","last_peered":"2024-03-24T01:27:14.545176+0000","last_clean":"2024-03-24T01:27:14.545176+0000","last_became_active":"2024-03-24T01:27:13.521436+0000","last_became_peered":"2024-03-24T01:27:13.521436+0000","last_unstale":"2024-03-24T01:27:14.545176+0000","last_undegraded":"2024-03-24T01:27:14.545176+0000","last_fullsized":"2024-03-24T01:27:14.545176+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:47:38.446831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.551289+0000","last_change":"2024-03-24T01:27:14.543585+0000","last_active":"2024-03-24T01:27:14.551289+0000","last_peered":"2024-03-24T01:27:14.551289+0000","last_clean":"2024-03-24T01:27:14.551289+0000","last_became_active":"2024-03-24T01:27:14.543301+0000","last_became_peered":"2024-03-24T01:27:14.543301+0000","last_unstale":"2024-03-24T01:27:14.551289+0000","last_undegraded":"2024-03-24T01:27:14.551289+0000","last_fullsized":"2024-03-24T01:27:14.551289+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:12:15.274601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.527295+0000","last_change":"2024-03-24T01:27:13.522613+0000","last_active":"2024-03-24T01:27:14.527295+0000","last_peered":"2024-03-24T01:27:14.527295+0000","last_clean":"2024-03-24T01:27:14.527295+0000","last_became_active":"2024-03-24T01:27:13.522331+0000","last_became_peered":"2024-03-24T01:27:13.522331+0000","last_unstale":"2024-03-24T01:27:14.527295+0000","last_undegraded":"2024-03-24T01:27:14.527295+0000","last_fullsized":"2024-03-24T01:27:14.527295+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:08:09.036710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.558092+0000","last_change":"2024-03-24T01:27:14.550330+0000","last_active":"2024-03-24T01:27:14.558092+0000","last_peered":"2024-03-24T01:27:14.558092+0000","last_clean":"2024-03-24T01:27:14.558092+0000","last_became_active":"2024-03-24T01:27:14.550056+0000","last_became_peered":"2024-03-24T01:27:14.550056+0000","last_unstale":"2024-03-24T01:27:14.558092+0000","last_undegraded":"2024-03-24T01:27:14.558092+0000","last_fullsized":"2024-03-24T01:27:14.558092+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:23.971971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.529686+0000","last_change":"2024-03-24T01:27:13.533418+0000","last_active":"2024-03-24T01:27:14.529686+0000","last_peered":"2024-03-24T01:27:14.529686+0000","last_clean":"2024-03-24T01:27:14.529686+0000","last_became_active":"2024-03-24T01:27:13.533164+0000","last_became_peered":"2024-03-24T01:27:13.533164+0000","last_unstale":"2024-03-24T01:27:14.529686+0000","last_undegraded":"2024-03-24T01:27:14.529686+0000","last_fullsized":"2024-03-24T01:27:14.529686+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:30:49.041062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.551209+0000","last_change":"2024-03-24T01:27:14.543832+0000","last_active":"2024-03-24T01:27:14.551209+0000","last_peered":"2024-03-24T01:27:14.551209+0000","last_clean":"2024-03-24T01:27:14.551209+0000","last_became_active":"2024-03-24T01:27:14.543520+0000","last_became_peered":"2024-03-24T01:27:14.543520+0000","last_unstale":"2024-03-24T01:27:14.551209+0000","last_undegraded":"2024-03-24T01:27:14.551209+0000","last_fullsized":"2024-03-24T01:27:14.551209+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:44.340192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.548373+0000","last_change":"2024-03-24T01:27:13.530300+0000","last_active":"2024-03-24T01:27:14.548373+0000","last_peered":"2024-03-24T01:27:14.548373+0000","last_clean":"2024-03-24T01:27:14.548373+0000","last_became_active":"2024-03-24T01:27:13.529919+0000","last_became_peered":"2024-03-24T01:27:13.529919+0000","last_unstale":"2024-03-24T01:27:14.548373+0000","last_undegraded":"2024-03-24T01:27:14.548373+0000","last_fullsized":"2024-03-24T01:27:14.548373+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:47:38.446831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.548304+0000","last_change":"2024-03-24T01:27:14.544735+0000","last_active":"2024-03-24T01:27:14.548304+0000","last_peered":"2024-03-24T01:27:14.548304+0000","last_clean":"2024-03-24T01:27:14.548304+0000","last_became_active":"2024-03-24T01:27:14.544460+0000","last_became_peered":"2024-03-24T01:27:14.544460+0000","last_unstale":"2024-03-24T01:27:14.548304+0000","last_undegraded":"2024-03-24T01:27:14.548304+0000","last_fullsized":"2024-03-24T01:27:14.548304+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:06.757918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.536983+0000","last_change":"2024-03-24T01:27:13.525558+0000","last_active":"2024-03-24T01:27:14.536983+0000","last_peered":"2024-03-24T01:27:14.536983+0000","last_clean":"2024-03-24T01:27:14.536983+0000","last_became_active":"2024-03-24T01:27:13.525223+0000","last_became_peered":"2024-03-24T01:27:13.525223+0000","last_unstale":"2024-03-24T01:27:14.536983+0000","last_undegraded":"2024-03-24T01:27:14.536983+0000","last_fullsized":"2024-03-24T01:27:14.536983+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:43.325731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.549747+0000","last_change":"2024-03-24T01:27:14.549747+0000","last_active":"2024-03-24T01:27:14.549747+0000","last_peered":"2024-03-24T01:27:14.549747+0000","last_clean":"2024-03-24T01:27:14.549747+0000","last_became_active":"2024-03-24T01:27:14.548344+0000","last_became_peered":"2024-03-24T01:27:14.548344+0000","last_unstale":"2024-03-24T01:27:14.549747+0000","last_undegraded":"2024-03-24T01:27:14.549747+0000","last_fullsized":"2024-03-24T01:27:14.549747+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:32.037863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.535645+0000","last_change":"2024-03-24T01:27:13.519855+0000","last_active":"2024-03-24T01:27:14.535645+0000","last_peered":"2024-03-24T01:27:14.535645+0000","last_clean":"2024-03-24T01:27:14.535645+0000","last_became_active":"2024-03-24T01:27:13.519566+0000","last_became_peered":"2024-03-24T01:27:13.519566+0000","last_unstale":"2024-03-24T01:27:14.535645+0000","last_undegraded":"2024-03-24T01:27:14.535645+0000","last_fullsized":"2024-03-24T01:27:14.535645+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:57.615564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.522676+0000","last_change":"2024-03-24T01:27:14.522676+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.522676+0000","last_undegraded":"2024-03-24T01:27:14.522676+0000","last_fullsized":"2024-03-24T01:27:14.522676+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.535933+0000","last_change":"2024-03-24T01:27:13.524780+0000","last_active":"2024-03-24T01:27:14.535933+0000","last_peered":"2024-03-24T01:27:14.535933+0000","last_clean":"2024-03-24T01:27:14.535933+0000","last_became_active":"2024-03-24T01:27:13.524518+0000","last_became_peered":"2024-03-24T01:27:13.524518+0000","last_unstale":"2024-03-24T01:27:14.535933+0000","last_undegraded":"2024-03-24T01:27:14.535933+0000","last_fullsized":"2024-03-24T01:27:14.535933+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:14:15.902136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.534874+0000","last_change":"2024-03-24T01:27:14.534874+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.534874+0000","last_undegraded":"2024-03-24T01:27:14.534874+0000","last_fullsized":"2024-03-24T01:27:14.534874+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.534655+0000","last_change":"2024-03-24T01:27:13.519549+0000","last_active":"2024-03-24T01:27:14.534655+0000","last_peered":"2024-03-24T01:27:14.534655+0000","last_clean":"2024-03-24T01:27:14.534655+0000","last_became_active":"2024-03-24T01:27:13.519174+0000","last_became_peered":"2024-03-24T01:27:13.519174+0000","last_unstale":"2024-03-24T01:27:14.534655+0000","last_undegraded":"2024-03-24T01:27:14.534655+0000","last_fullsized":"2024-03-24T01:27:14.534655+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:30:49.041062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.549972+0000","last_change":"2024-03-24T01:27:14.548520+0000","last_active":"2024-03-24T01:27:14.549972+0000","last_peered":"2024-03-24T01:27:14.549972+0000","last_clean":"2024-03-24T01:27:14.549972+0000","last_became_active":"2024-03-24T01:27:14.547968+0000","last_became_peered":"2024-03-24T01:27:14.547968+0000","last_unstale":"2024-03-24T01:27:14.549972+0000","last_undegraded":"2024-03-24T01:27:14.549972+0000","last_fullsized":"2024-03-24T01:27:14.549972+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:36.294802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.520961+0000","last_change":"2024-03-24T01:27:13.524804+0000","last_active":"2024-03-24T01:27:14.520961+0000","last_peered":"2024-03-24T01:27:14.520961+0000","last_clean":"2024-03-24T01:27:14.520961+0000","last_became_active":"2024-03-24T01:27:13.524487+0000","last_became_peered":"2024-03-24T01:27:13.524487+0000","last_unstale":"2024-03-24T01:27:14.520961+0000","last_undegraded":"2024-03-24T01:27:14.520961+0000","last_fullsized":"2024-03-24T01:27:14.520961+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:43:44.823891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.546611+0000","last_change":"2024-03-24T01:27:14.539180+0000","last_active":"2024-03-24T01:27:14.546611+0000","last_peered":"2024-03-24T01:27:14.546611+0000","last_clean":"2024-03-24T01:27:14.546611+0000","last_became_active":"2024-03-24T01:27:14.538800+0000","last_became_peered":"2024-03-24T01:27:14.538800+0000","last_unstale":"2024-03-24T01:27:14.546611+0000","last_undegraded":"2024-03-24T01:27:14.546611+0000","last_fullsized":"2024-03-24T01:27:14.546611+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:40.646803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.535286+0000","last_change":"2024-03-24T01:27:13.532470+0000","last_active":"2024-03-24T01:27:14.535286+0000","last_peered":"2024-03-24T01:27:14.535286+0000","last_clean":"2024-03-24T01:27:14.535286+0000","last_became_active":"2024-03-24T01:27:13.530705+0000","last_became_peered":"2024-03-24T01:27:13.530705+0000","last_unstale":"2024-03-24T01:27:14.535286+0000","last_undegraded":"2024-03-24T01:27:14.535286+0000","last_fullsized":"2024-03-24T01:27:14.535286+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:12:14.260140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":229,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.545758+0000","last_change":"2024-03-24T01:24:01.545855+0000","last_active":"2024-03-24T01:27:14.545758+0000","last_peered":"2024-03-24T01:27:14.545758+0000","last_clean":"2024-03-24T01:27:14.545758+0000","last_became_active":"2024-03-24T01:24:01.543811+0000","last_became_peered":"2024-03-24T01:24:01.543811+0000","last_unstale":"2024-03-24T01:27:14.545758+0000","last_undegraded":"2024-03-24T01:27:14.545758+0000","last_fullsized":"2024-03-24T01:27:14.545758+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_clean_scrub_stamp":"2024-03-24T01:23:59.861564+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:03.682631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":672,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15,"ondisk_log_size":15,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574890,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28196,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343644,"statfs":{"total":96636764160,"available":96607891456,"internally_reserved":0,"allocated":1200128,"data_stored":861179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":270811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":270811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574890,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":270811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":270811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574891,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":270811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":618496,"data_stored":277978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574891,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28196,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343644,"statfs":{"total":96636764160,"available":96607891456,"internally_reserved":0,"allocated":1200128,"data_stored":861179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T01:27:18.962 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T01:27:19.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.207+0000 7f4a10600640 1 Processor -- start 2024-03-24T01:27:19.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.207+0000 7f4a10600640 1 -- start start 2024-03-24T01:27:19.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.209+0000 7f4a10600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4a080b94e0 0x7f4a080ce890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:19.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.209+0000 7f4a10600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4a080cedd0 0x7f4a080d23e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:19.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.209+0000 7f4a10600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a081620a0 con 0x7f4a0815df40 2024-03-24T01:27:19.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.209+0000 7f4a10600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a080d1c70 con 0x7f4a080b94e0 2024-03-24T01:27:19.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.209+0000 7f4a10600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a08162380 con 0x7f4a080cedd0 2024-03-24T01:27:19.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.209+0000 7f4a0d200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4a080b94e0 0x7f4a080ce890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:19.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.209+0000 7f4a0d200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4a080b94e0 0x7f4a080ce890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58980/0 (socket says 172.21.15.90:58980) 2024-03-24T01:27:19.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.209+0000 7f4a0d200640 1 -- 172.21.15.90:0/1713892225 learned_addr learned my addr 172.21.15.90:0/1713892225 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:19.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.209+0000 7f4a0d200640 1 -- 172.21.15.90:0/1713892225 >> v2:172.21.15.120:3300/0 conn(0x7f4a080cedd0 msgr2=0x7f4a080d23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:19.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.209+0000 7f4a0d200640 1 --2- 172.21.15.90:0/1713892225 >> v2:172.21.15.120:3300/0 conn(0x7f4a080cedd0 0x7f4a080d23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:19.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.209+0000 7f4a0d200640 1 -- 172.21.15.90:0/1713892225 >> v1:172.21.15.120:6789/0 conn(0x7f4a0815df40 legacy=0x7f4a08161890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:19.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.209+0000 7f4a0d200640 1 -- 172.21.15.90:0/1713892225 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a08162660 con 0x7f4a080b94e0 2024-03-24T01:27:19.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.211+0000 7f4a0d200640 1 --2- 172.21.15.90:0/1713892225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a080b94e0 0x7f4a080ce890 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f4a080d05f0 tx=0x7f49f802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc1ab7d00bcd6d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:19.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.211+0000 7f4a07e00640 1 -- 172.21.15.90:0/1713892225 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f802ca80 con 0x7f4a080b94e0 2024-03-24T01:27:19.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.211+0000 7f4a07e00640 1 -- 172.21.15.90:0/1713892225 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49f802cc20 con 0x7f4a080b94e0 2024-03-24T01:27:19.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.211+0000 7f4a07e00640 1 -- 172.21.15.90:0/1713892225 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f80343f0 con 0x7f4a080b94e0 2024-03-24T01:27:19.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.212+0000 7f4a10600640 1 -- 172.21.15.90:0/1713892225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a080b94e0 msgr2=0x7f4a080ce890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:19.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.212+0000 7f4a10600640 1 --2- 172.21.15.90:0/1713892225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a080b94e0 0x7f4a080ce890 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f4a080d05f0 tx=0x7f49f802bee0 comp rx=0 tx=0).stop 2024-03-24T01:27:19.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.212+0000 7f4a10600640 1 -- 172.21.15.90:0/1713892225 shutdown_connections 2024-03-24T01:27:19.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.212+0000 7f4a10600640 1 --2- 172.21.15.90:0/1713892225 >> v2:172.21.15.120:3300/0 conn(0x7f4a080cedd0 0x7f4a080d23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:19.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.212+0000 7f4a10600640 1 --2- 172.21.15.90:0/1713892225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a080b94e0 0x7f4a080ce890 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:19.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.212+0000 7f4a10600640 1 -- 172.21.15.90:0/1713892225 >> 172.21.15.90:0/1713892225 conn(0x7f4a0806c9f0 msgr2=0x7f4a0806ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:19.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.212+0000 7f4a10600640 1 -- 172.21.15.90:0/1713892225 shutdown_connections 2024-03-24T01:27:19.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.212+0000 7f4a10600640 1 -- 172.21.15.90:0/1713892225 wait complete. 2024-03-24T01:27:19.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.213+0000 7f4a10600640 1 Processor -- start 2024-03-24T01:27:19.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.214+0000 7f4a10600640 1 -- start start 2024-03-24T01:27:19.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.214+0000 7f4a10600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a080b94e0 0x7f4a081f7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:19.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.214+0000 7f4a10600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a080cedd0 0x7f4a081f7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:19.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.214+0000 7f4a10600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a0815df40 0x7f4a0814de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:19.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.214+0000 7f4a10600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a080d2aa0 con 0x7f4a080cedd0 2024-03-24T01:27:19.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.214+0000 7f4a10600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a080d2920 con 0x7f4a0815df40 2024-03-24T01:27:19.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.214+0000 7f4a10600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a080d2c20 con 0x7f4a080b94e0 2024-03-24T01:27:19.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.215+0000 7f4a0d200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a080cedd0 0x7f4a081f7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:19.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.215+0000 7f4a0d200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a080cedd0 0x7f4a081f7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58982/0 (socket says 172.21.15.90:58982) 2024-03-24T01:27:19.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.215+0000 7f4a0d200640 1 -- 172.21.15.90:0/3452667464 learned_addr learned my addr 172.21.15.90:0/3452667464 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:19.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.215+0000 7f4a0d200640 1 -- 172.21.15.90:0/3452667464 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a080b94e0 msgr2=0x7f4a081f7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:19.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.215+0000 7f4a0d200640 1 --2- 172.21.15.90:0/3452667464 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a080b94e0 0x7f4a081f7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:19.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.215+0000 7f4a0d200640 1 -- 172.21.15.90:0/3452667464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a0815df40 msgr2=0x7f4a0814de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:19.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.215+0000 7f4a0d200640 1 --2- 172.21.15.90:0/3452667464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a0815df40 0x7f4a0814de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:19.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.215+0000 7f4a0d200640 1 -- 172.21.15.90:0/3452667464 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a0814cde0 con 0x7f4a080cedd0 2024-03-24T01:27:19.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.215+0000 7f4a0d200640 1 --2- 172.21.15.90:0/3452667464 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a080cedd0 0x7f4a081f7d70 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f49f80093e0 tx=0x7f49f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:19.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.216+0000 7f4a05600640 1 -- 172.21.15.90:0/3452667464 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f8043020 con 0x7f4a080cedd0 2024-03-24T01:27:19.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.216+0000 7f4a05600640 1 -- 172.21.15.90:0/3452667464 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49f803e450 con 0x7f4a080cedd0 2024-03-24T01:27:19.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.216+0000 7f4a05600640 1 -- 172.21.15.90:0/3452667464 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f803e840 con 0x7f4a080cedd0 2024-03-24T01:27:19.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.216+0000 7f4a10600640 1 -- 172.21.15.90:0/3452667464 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a0814d080 con 0x7f4a080cedd0 2024-03-24T01:27:19.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.216+0000 7f4a10600640 1 -- 172.21.15.90:0/3452667464 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a0814d640 con 0x7f4a080cedd0 2024-03-24T01:27:19.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.217+0000 7f4a10600640 1 -- 172.21.15.90:0/3452667464 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a08068060 con 0x7f4a080cedd0 2024-03-24T01:27:19.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.218+0000 7f4a05600640 1 -- 172.21.15.90:0/3452667464 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f49f8046020 con 0x7f4a080cedd0 2024-03-24T01:27:19.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.219+0000 7f4a05600640 1 --2- 172.21.15.90:0/3452667464 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f49ec06bc00 0x7f49ec06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:19.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.219+0000 7f4a05600640 1 -- 172.21.15.90:0/3452667464 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f49f80ad3c0 con 0x7f4a080cedd0 2024-03-24T01:27:19.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.219+0000 7f4a0dc00640 1 --2- 172.21.15.90:0/3452667464 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f49ec06bc00 0x7f49ec06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:19.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.220+0000 7f4a0dc00640 1 --2- 172.21.15.90:0/3452667464 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f49ec06bc00 0x7f49ec06e0c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f4a080b2610 tx=0x7f49fc006ba0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:19.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.223+0000 7f4a05600640 1 -- 172.21.15.90:0/3452667464 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f49f804d070 con 0x7f4a080cedd0 2024-03-24T01:27:19.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.415+0000 7f4a10600640 1 -- 172.21.15.90:0/3452667464 --> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4a08161350 con 0x7f49ec06bc00 2024-03-24T01:27:19.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.434+0000 7f4a05600640 1 -- 172.21.15.90:0/3452667464 <== mgr.4104 v2:172.21.15.120:6832/860392634 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+250916 (secure 0 0 0) 0x7f4a08161350 con 0x7f49ec06bc00 2024-03-24T01:27:19.438 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:27:19.440 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-24T01:27:19.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.439+0000 7f4a10600640 1 -- 172.21.15.90:0/3452667464 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f49ec06bc00 msgr2=0x7f49ec06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:19.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.439+0000 7f4a10600640 1 --2- 172.21.15.90:0/3452667464 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f49ec06bc00 0x7f49ec06e0c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f4a080b2610 tx=0x7f49fc006ba0 comp rx=0 tx=0).stop 2024-03-24T01:27:19.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.439+0000 7f4a10600640 1 -- 172.21.15.90:0/3452667464 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a080cedd0 msgr2=0x7f4a081f7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:19.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.439+0000 7f4a10600640 1 --2- 172.21.15.90:0/3452667464 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a080cedd0 0x7f4a081f7d70 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f49f80093e0 tx=0x7f49f8004a40 comp rx=0 tx=0).stop 2024-03-24T01:27:19.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.439+0000 7f4a10600640 1 -- 172.21.15.90:0/3452667464 shutdown_connections 2024-03-24T01:27:19.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.439+0000 7f4a10600640 1 --2- 172.21.15.90:0/3452667464 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f49ec06bc00 0x7f49ec06e0c0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:19.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.439+0000 7f4a10600640 1 --2- 172.21.15.90:0/3452667464 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a080cedd0 0x7f4a081f7d70 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:19.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.439+0000 7f4a10600640 1 --2- 172.21.15.90:0/3452667464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a0815df40 0x7f4a0814de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:19.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.439+0000 7f4a10600640 1 --2- 172.21.15.90:0/3452667464 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a080b94e0 0x7f4a081f7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:19.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.439+0000 7f4a10600640 1 -- 172.21.15.90:0/3452667464 >> 172.21.15.90:0/3452667464 conn(0x7f4a0806c9f0 msgr2=0x7f4a080c78e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:19.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.439+0000 7f4a10600640 1 -- 172.21.15.90:0/3452667464 shutdown_connections 2024-03-24T01:27:19.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.439+0000 7f4a10600640 1 -- 172.21.15.90:0/3452667464 wait complete. 2024-03-24T01:27:19.459 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":145,"stamp":"2024-03-24T01:27:17.549064+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":179,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132,"ondisk_log_size":132,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222104,"kb_used_data":5908,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752616,"statfs":{"total":773094113280,"available":772866678784,"internally_reserved":0,"allocated":6049792,"data_stored":3354391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12747,"internal_metadata":221236789},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":672,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.062753"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.547093+0000","last_change":"2024-03-24T01:27:14.540765+0000","last_active":"2024-03-24T01:27:14.547093+0000","last_peered":"2024-03-24T01:27:14.547093+0000","last_clean":"2024-03-24T01:27:14.547093+0000","last_became_active":"2024-03-24T01:27:14.539541+0000","last_became_peered":"2024-03-24T01:27:14.539541+0000","last_unstale":"2024-03-24T01:27:14.547093+0000","last_undegraded":"2024-03-24T01:27:14.547093+0000","last_fullsized":"2024-03-24T01:27:14.547093+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:30:50.055523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.559986+0000","last_change":"2024-03-24T01:27:14.559986+0000","last_active":"2024-03-24T01:27:14.559986+0000","last_peered":"2024-03-24T01:27:14.559986+0000","last_clean":"2024-03-24T01:27:14.559986+0000","last_became_active":"2024-03-24T01:27:14.559649+0000","last_became_peered":"2024-03-24T01:27:14.559649+0000","last_unstale":"2024-03-24T01:27:14.559986+0000","last_undegraded":"2024-03-24T01:27:14.559986+0000","last_fullsized":"2024-03-24T01:27:14.559986+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:23.971971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.538645+0000","last_change":"2024-03-24T01:27:14.538645+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.538645+0000","last_undegraded":"2024-03-24T01:27:14.538645+0000","last_fullsized":"2024-03-24T01:27:14.538645+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.559476+0000","last_change":"2024-03-24T01:27:14.559476+0000","last_active":"2024-03-24T01:27:14.559476+0000","last_peered":"2024-03-24T01:27:14.559476+0000","last_clean":"2024-03-24T01:27:14.559476+0000","last_became_active":"2024-03-24T01:27:14.559176+0000","last_became_peered":"2024-03-24T01:27:14.559176+0000","last_unstale":"2024-03-24T01:27:14.559476+0000","last_undegraded":"2024-03-24T01:27:14.559476+0000","last_fullsized":"2024-03-24T01:27:14.559476+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:03.448771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.531210+0000","last_change":"2024-03-24T01:27:14.531210+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.531210+0000","last_undegraded":"2024-03-24T01:27:14.531210+0000","last_fullsized":"2024-03-24T01:27:14.531210+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.555238+0000","last_change":"2024-03-24T01:27:14.551494+0000","last_active":"2024-03-24T01:27:14.555238+0000","last_peered":"2024-03-24T01:27:14.555238+0000","last_clean":"2024-03-24T01:27:14.555238+0000","last_became_active":"2024-03-24T01:27:14.551175+0000","last_became_peered":"2024-03-24T01:27:14.551175+0000","last_unstale":"2024-03-24T01:27:14.555238+0000","last_undegraded":"2024-03-24T01:27:14.555238+0000","last_fullsized":"2024-03-24T01:27:14.555238+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:06:34.262662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.552294+0000","last_change":"2024-03-24T01:27:14.552294+0000","last_active":"2024-03-24T01:27:14.552294+0000","last_peered":"2024-03-24T01:27:14.552294+0000","last_clean":"2024-03-24T01:27:14.552294+0000","last_became_active":"2024-03-24T01:27:14.551839+0000","last_became_peered":"2024-03-24T01:27:14.551839+0000","last_unstale":"2024-03-24T01:27:14.552294+0000","last_undegraded":"2024-03-24T01:27:14.552294+0000","last_fullsized":"2024-03-24T01:27:14.552294+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:01:09.219193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.562359+0000","last_change":"2024-03-24T01:27:14.558536+0000","last_active":"2024-03-24T01:27:14.562359+0000","last_peered":"2024-03-24T01:27:14.562359+0000","last_clean":"2024-03-24T01:27:14.562359+0000","last_became_active":"2024-03-24T01:27:14.558211+0000","last_became_peered":"2024-03-24T01:27:14.558211+0000","last_unstale":"2024-03-24T01:27:14.562359+0000","last_undegraded":"2024-03-24T01:27:14.562359+0000","last_fullsized":"2024-03-24T01:27:14.562359+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:44:39.116139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.539698+0000","last_change":"2024-03-24T01:27:14.539698+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.539698+0000","last_undegraded":"2024-03-24T01:27:14.539698+0000","last_fullsized":"2024-03-24T01:27:14.539698+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.547500+0000","last_change":"2024-03-24T01:27:14.541578+0000","last_active":"2024-03-24T01:27:14.547500+0000","last_peered":"2024-03-24T01:27:14.547500+0000","last_clean":"2024-03-24T01:27:14.547500+0000","last_became_active":"2024-03-24T01:27:14.541289+0000","last_became_peered":"2024-03-24T01:27:14.541289+0000","last_unstale":"2024-03-24T01:27:14.547500+0000","last_undegraded":"2024-03-24T01:27:14.547500+0000","last_fullsized":"2024-03-24T01:27:14.547500+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:40.646803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.528427+0000","last_change":"2024-03-24T01:27:14.528427+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.528427+0000","last_undegraded":"2024-03-24T01:27:14.528427+0000","last_fullsized":"2024-03-24T01:27:14.528427+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.558861+0000","last_change":"2024-03-24T01:27:14.552072+0000","last_active":"2024-03-24T01:27:14.558861+0000","last_peered":"2024-03-24T01:27:14.558861+0000","last_clean":"2024-03-24T01:27:14.558861+0000","last_became_active":"2024-03-24T01:27:14.550084+0000","last_became_peered":"2024-03-24T01:27:14.550084+0000","last_unstale":"2024-03-24T01:27:14.558861+0000","last_undegraded":"2024-03-24T01:27:14.558861+0000","last_fullsized":"2024-03-24T01:27:14.558861+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:45:39.507934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.550970+0000","last_change":"2024-03-24T01:27:14.544185+0000","last_active":"2024-03-24T01:27:14.550970+0000","last_peered":"2024-03-24T01:27:14.550970+0000","last_clean":"2024-03-24T01:27:14.550970+0000","last_became_active":"2024-03-24T01:27:14.543895+0000","last_became_peered":"2024-03-24T01:27:14.543895+0000","last_unstale":"2024-03-24T01:27:14.550970+0000","last_undegraded":"2024-03-24T01:27:14.550970+0000","last_fullsized":"2024-03-24T01:27:14.550970+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:47:39.461292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.550120+0000","last_change":"2024-03-24T01:27:14.550120+0000","last_active":"2024-03-24T01:27:14.550120+0000","last_peered":"2024-03-24T01:27:14.550120+0000","last_clean":"2024-03-24T01:27:14.550120+0000","last_became_active":"2024-03-24T01:27:14.549329+0000","last_became_peered":"2024-03-24T01:27:14.549329+0000","last_unstale":"2024-03-24T01:27:14.550120+0000","last_undegraded":"2024-03-24T01:27:14.550120+0000","last_fullsized":"2024-03-24T01:27:14.550120+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:32:58.788010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.557855+0000","last_change":"2024-03-24T01:27:14.557855+0000","last_active":"2024-03-24T01:27:14.557855+0000","last_peered":"2024-03-24T01:27:14.557855+0000","last_clean":"2024-03-24T01:27:14.557855+0000","last_became_active":"2024-03-24T01:27:14.557606+0000","last_became_peered":"2024-03-24T01:27:14.557606+0000","last_unstale":"2024-03-24T01:27:14.557855+0000","last_undegraded":"2024-03-24T01:27:14.557855+0000","last_fullsized":"2024-03-24T01:27:14.557855+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:28:36.027521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.550788+0000","last_change":"2024-03-24T01:27:14.542073+0000","last_active":"2024-03-24T01:27:14.550788+0000","last_peered":"2024-03-24T01:27:14.550788+0000","last_clean":"2024-03-24T01:27:14.550788+0000","last_became_active":"2024-03-24T01:27:14.540636+0000","last_became_peered":"2024-03-24T01:27:14.540636+0000","last_unstale":"2024-03-24T01:27:14.550788+0000","last_undegraded":"2024-03-24T01:27:14.550788+0000","last_fullsized":"2024-03-24T01:27:14.550788+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:02:12.837672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.556042+0000","last_change":"2024-03-24T01:27:14.556042+0000","last_active":"2024-03-24T01:27:14.556042+0000","last_peered":"2024-03-24T01:27:14.556042+0000","last_clean":"2024-03-24T01:27:14.556042+0000","last_became_active":"2024-03-24T01:27:14.555743+0000","last_became_peered":"2024-03-24T01:27:14.555743+0000","last_unstale":"2024-03-24T01:27:14.556042+0000","last_undegraded":"2024-03-24T01:27:14.556042+0000","last_fullsized":"2024-03-24T01:27:14.556042+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:14:16.916597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.561148+0000","last_change":"2024-03-24T01:27:14.561148+0000","last_active":"2024-03-24T01:27:14.561148+0000","last_peered":"2024-03-24T01:27:14.561148+0000","last_clean":"2024-03-24T01:27:14.561148+0000","last_became_active":"2024-03-24T01:27:14.560509+0000","last_became_peered":"2024-03-24T01:27:14.560509+0000","last_unstale":"2024-03-24T01:27:14.561148+0000","last_undegraded":"2024-03-24T01:27:14.561148+0000","last_fullsized":"2024-03-24T01:27:14.561148+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:09:58.411238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.553533+0000","last_change":"2024-03-24T01:27:14.553533+0000","last_active":"2024-03-24T01:27:14.553533+0000","last_peered":"2024-03-24T01:27:14.553533+0000","last_clean":"2024-03-24T01:27:14.553533+0000","last_became_active":"2024-03-24T01:27:14.553252+0000","last_became_peered":"2024-03-24T01:27:14.553252+0000","last_unstale":"2024-03-24T01:27:14.553533+0000","last_undegraded":"2024-03-24T01:27:14.553533+0000","last_fullsized":"2024-03-24T01:27:14.553533+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:46:45.759885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.546096+0000","last_change":"2024-03-24T01:27:14.541674+0000","last_active":"2024-03-24T01:27:14.546096+0000","last_peered":"2024-03-24T01:27:14.546096+0000","last_clean":"2024-03-24T01:27:14.546096+0000","last_became_active":"2024-03-24T01:27:14.541300+0000","last_became_peered":"2024-03-24T01:27:14.541300+0000","last_unstale":"2024-03-24T01:27:14.546096+0000","last_undegraded":"2024-03-24T01:27:14.546096+0000","last_fullsized":"2024-03-24T01:27:14.546096+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:12:15.274601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.527732+0000","last_change":"2024-03-24T01:27:14.527732+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.527732+0000","last_undegraded":"2024-03-24T01:27:14.527732+0000","last_fullsized":"2024-03-24T01:27:14.527732+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.558260+0000","last_change":"2024-03-24T01:27:14.554032+0000","last_active":"2024-03-24T01:27:14.558260+0000","last_peered":"2024-03-24T01:27:14.558260+0000","last_clean":"2024-03-24T01:27:14.558260+0000","last_became_active":"2024-03-24T01:27:14.553732+0000","last_became_peered":"2024-03-24T01:27:14.553732+0000","last_unstale":"2024-03-24T01:27:14.558260+0000","last_undegraded":"2024-03-24T01:27:14.558260+0000","last_fullsized":"2024-03-24T01:27:14.558260+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:04:45.650427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.551791+0000","last_change":"2024-03-24T01:27:14.551791+0000","last_active":"2024-03-24T01:27:14.551791+0000","last_peered":"2024-03-24T01:27:14.551791+0000","last_clean":"2024-03-24T01:27:14.551791+0000","last_became_active":"2024-03-24T01:27:14.551503+0000","last_became_peered":"2024-03-24T01:27:14.551503+0000","last_unstale":"2024-03-24T01:27:14.551791+0000","last_undegraded":"2024-03-24T01:27:14.551791+0000","last_fullsized":"2024-03-24T01:27:14.551791+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:18:59.684453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.556979+0000","last_change":"2024-03-24T01:27:14.545843+0000","last_active":"2024-03-24T01:27:14.556979+0000","last_peered":"2024-03-24T01:27:14.556979+0000","last_clean":"2024-03-24T01:27:14.556979+0000","last_became_active":"2024-03-24T01:27:14.545534+0000","last_became_peered":"2024-03-24T01:27:14.545534+0000","last_unstale":"2024-03-24T01:27:14.556979+0000","last_undegraded":"2024-03-24T01:27:14.556979+0000","last_fullsized":"2024-03-24T01:27:14.556979+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:06:34.262662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.548640+0000","last_change":"2024-03-24T01:27:14.548640+0000","last_active":"2024-03-24T01:27:14.548640+0000","last_peered":"2024-03-24T01:27:14.548640+0000","last_clean":"2024-03-24T01:27:14.548640+0000","last_became_active":"2024-03-24T01:27:14.548364+0000","last_became_peered":"2024-03-24T01:27:14.548364+0000","last_unstale":"2024-03-24T01:27:14.548640+0000","last_undegraded":"2024-03-24T01:27:14.548640+0000","last_fullsized":"2024-03-24T01:27:14.548640+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:28:36.027521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.544765+0000","last_change":"2024-03-24T01:27:14.541461+0000","last_active":"2024-03-24T01:27:14.544765+0000","last_peered":"2024-03-24T01:27:14.544765+0000","last_clean":"2024-03-24T01:27:14.544765+0000","last_became_active":"2024-03-24T01:27:14.541156+0000","last_became_peered":"2024-03-24T01:27:14.541156+0000","last_unstale":"2024-03-24T01:27:14.544765+0000","last_undegraded":"2024-03-24T01:27:14.544765+0000","last_fullsized":"2024-03-24T01:27:14.544765+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:14:09.820632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.556242+0000","last_change":"2024-03-24T01:27:14.556242+0000","last_active":"2024-03-24T01:27:14.556242+0000","last_peered":"2024-03-24T01:27:14.556242+0000","last_clean":"2024-03-24T01:27:14.556242+0000","last_became_active":"2024-03-24T01:27:14.552642+0000","last_became_peered":"2024-03-24T01:27:14.552642+0000","last_unstale":"2024-03-24T01:27:14.556242+0000","last_undegraded":"2024-03-24T01:27:14.556242+0000","last_fullsized":"2024-03-24T01:27:14.556242+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:28:36.027521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.546588+0000","last_change":"2024-03-24T01:27:14.542335+0000","last_active":"2024-03-24T01:27:14.546588+0000","last_peered":"2024-03-24T01:27:14.546588+0000","last_clean":"2024-03-24T01:27:14.546588+0000","last_became_active":"2024-03-24T01:27:14.541006+0000","last_became_peered":"2024-03-24T01:27:14.541006+0000","last_unstale":"2024-03-24T01:27:14.546588+0000","last_undegraded":"2024-03-24T01:27:14.546588+0000","last_fullsized":"2024-03-24T01:27:14.546588+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:44.340192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.550220+0000","last_change":"2024-03-24T01:27:14.550220+0000","last_active":"2024-03-24T01:27:14.550220+0000","last_peered":"2024-03-24T01:27:14.550220+0000","last_clean":"2024-03-24T01:27:14.550220+0000","last_became_active":"2024-03-24T01:27:14.549812+0000","last_became_peered":"2024-03-24T01:27:14.549812+0000","last_unstale":"2024-03-24T01:27:14.550220+0000","last_undegraded":"2024-03-24T01:27:14.550220+0000","last_fullsized":"2024-03-24T01:27:14.550220+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:09:58.411238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.556235+0000","last_change":"2024-03-24T01:27:14.556235+0000","last_active":"2024-03-24T01:27:14.556235+0000","last_peered":"2024-03-24T01:27:14.556235+0000","last_clean":"2024-03-24T01:27:14.556235+0000","last_became_active":"2024-03-24T01:27:14.555888+0000","last_became_peered":"2024-03-24T01:27:14.555888+0000","last_unstale":"2024-03-24T01:27:14.556235+0000","last_undegraded":"2024-03-24T01:27:14.556235+0000","last_fullsized":"2024-03-24T01:27:14.556235+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:43:45.838352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.552212+0000","last_change":"2024-03-24T01:27:14.552212+0000","last_active":"2024-03-24T01:27:14.552212+0000","last_peered":"2024-03-24T01:27:14.552212+0000","last_clean":"2024-03-24T01:27:14.552212+0000","last_became_active":"2024-03-24T01:27:14.551775+0000","last_became_peered":"2024-03-24T01:27:14.551775+0000","last_unstale":"2024-03-24T01:27:14.552212+0000","last_undegraded":"2024-03-24T01:27:14.552212+0000","last_fullsized":"2024-03-24T01:27:14.552212+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:23.971971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.550504+0000","last_change":"2024-03-24T01:27:14.542056+0000","last_active":"2024-03-24T01:27:14.550504+0000","last_peered":"2024-03-24T01:27:14.550504+0000","last_clean":"2024-03-24T01:27:14.550504+0000","last_became_active":"2024-03-24T01:27:14.541695+0000","last_became_peered":"2024-03-24T01:27:14.541695+0000","last_unstale":"2024-03-24T01:27:14.550504+0000","last_undegraded":"2024-03-24T01:27:14.550504+0000","last_fullsized":"2024-03-24T01:27:14.550504+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:32.037863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.559666+0000","last_change":"2024-03-24T01:27:14.559666+0000","last_active":"2024-03-24T01:27:14.559666+0000","last_peered":"2024-03-24T01:27:14.559666+0000","last_clean":"2024-03-24T01:27:14.559666+0000","last_became_active":"2024-03-24T01:27:14.559356+0000","last_became_peered":"2024-03-24T01:27:14.559356+0000","last_unstale":"2024-03-24T01:27:14.559666+0000","last_undegraded":"2024-03-24T01:27:14.559666+0000","last_fullsized":"2024-03-24T01:27:14.559666+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:32.037863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.529333+0000","last_change":"2024-03-24T01:27:13.522918+0000","last_active":"2024-03-24T01:27:14.529333+0000","last_peered":"2024-03-24T01:27:14.529333+0000","last_clean":"2024-03-24T01:27:14.529333+0000","last_became_active":"2024-03-24T01:27:13.520866+0000","last_became_peered":"2024-03-24T01:27:13.520866+0000","last_unstale":"2024-03-24T01:27:14.529333+0000","last_undegraded":"2024-03-24T01:27:14.529333+0000","last_fullsized":"2024-03-24T01:27:14.529333+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:30.183145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.557567+0000","last_change":"2024-03-24T01:27:14.551119+0000","last_active":"2024-03-24T01:27:14.557567+0000","last_peered":"2024-03-24T01:27:14.557567+0000","last_clean":"2024-03-24T01:27:14.557567+0000","last_became_active":"2024-03-24T01:27:14.550800+0000","last_became_peered":"2024-03-24T01:27:14.550800+0000","last_unstale":"2024-03-24T01:27:14.557567+0000","last_undegraded":"2024-03-24T01:27:14.557567+0000","last_fullsized":"2024-03-24T01:27:14.557567+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:09:58.411238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.518296+0000","last_change":"2024-03-24T01:27:13.531689+0000","last_active":"2024-03-24T01:27:14.518296+0000","last_peered":"2024-03-24T01:27:14.518296+0000","last_clean":"2024-03-24T01:27:14.518296+0000","last_became_active":"2024-03-24T01:27:13.530929+0000","last_became_peered":"2024-03-24T01:27:13.530929+0000","last_unstale":"2024-03-24T01:27:14.518296+0000","last_undegraded":"2024-03-24T01:27:14.518296+0000","last_fullsized":"2024-03-24T01:27:14.518296+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:00:24.458371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.552432+0000","last_change":"2024-03-24T01:27:14.552432+0000","last_active":"2024-03-24T01:27:14.552432+0000","last_peered":"2024-03-24T01:27:14.552432+0000","last_clean":"2024-03-24T01:27:14.552432+0000","last_became_active":"2024-03-24T01:27:14.551697+0000","last_became_peered":"2024-03-24T01:27:14.551697+0000","last_unstale":"2024-03-24T01:27:14.552432+0000","last_undegraded":"2024-03-24T01:27:14.552432+0000","last_fullsized":"2024-03-24T01:27:14.552432+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:47:39.461292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.518290+0000","last_change":"2024-03-24T01:27:13.528068+0000","last_active":"2024-03-24T01:27:14.518290+0000","last_peered":"2024-03-24T01:27:14.518290+0000","last_clean":"2024-03-24T01:27:14.518290+0000","last_became_active":"2024-03-24T01:27:13.527750+0000","last_became_peered":"2024-03-24T01:27:13.527750+0000","last_unstale":"2024-03-24T01:27:14.518290+0000","last_undegraded":"2024-03-24T01:27:14.518290+0000","last_fullsized":"2024-03-24T01:27:14.518290+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:37:15.291837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.549966+0000","last_change":"2024-03-24T01:27:14.548464+0000","last_active":"2024-03-24T01:27:14.549966+0000","last_peered":"2024-03-24T01:27:14.549966+0000","last_clean":"2024-03-24T01:27:14.549966+0000","last_became_active":"2024-03-24T01:27:14.548113+0000","last_became_peered":"2024-03-24T01:27:14.548113+0000","last_unstale":"2024-03-24T01:27:14.549966+0000","last_undegraded":"2024-03-24T01:27:14.549966+0000","last_fullsized":"2024-03-24T01:27:14.549966+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:47:57.893749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.523180+0000","last_change":"2024-03-24T01:27:13.527296+0000","last_active":"2024-03-24T01:27:14.523180+0000","last_peered":"2024-03-24T01:27:14.523180+0000","last_clean":"2024-03-24T01:27:14.523180+0000","last_became_active":"2024-03-24T01:27:13.526656+0000","last_became_peered":"2024-03-24T01:27:13.526656+0000","last_unstale":"2024-03-24T01:27:14.523180+0000","last_undegraded":"2024-03-24T01:27:14.523180+0000","last_fullsized":"2024-03-24T01:27:14.523180+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:28:35.013061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.531475+0000","last_change":"2024-03-24T01:27:14.531475+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.531475+0000","last_undegraded":"2024-03-24T01:27:14.531475+0000","last_fullsized":"2024-03-24T01:27:14.531475+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.532174+0000","last_change":"2024-03-24T01:27:13.518516+0000","last_active":"2024-03-24T01:27:14.532174+0000","last_peered":"2024-03-24T01:27:14.532174+0000","last_clean":"2024-03-24T01:27:14.532174+0000","last_became_active":"2024-03-24T01:27:13.518276+0000","last_became_peered":"2024-03-24T01:27:13.518276+0000","last_unstale":"2024-03-24T01:27:14.532174+0000","last_undegraded":"2024-03-24T01:27:14.532174+0000","last_fullsized":"2024-03-24T01:27:14.532174+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:39.632343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.559916+0000","last_change":"2024-03-24T01:27:14.559916+0000","last_active":"2024-03-24T01:27:14.559916+0000","last_peered":"2024-03-24T01:27:14.559916+0000","last_clean":"2024-03-24T01:27:14.559916+0000","last_became_active":"2024-03-24T01:27:14.559597+0000","last_became_peered":"2024-03-24T01:27:14.559597+0000","last_unstale":"2024-03-24T01:27:14.559916+0000","last_undegraded":"2024-03-24T01:27:14.559916+0000","last_fullsized":"2024-03-24T01:27:14.559916+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:32:58.788010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.535218+0000","last_change":"2024-03-24T01:27:13.527482+0000","last_active":"2024-03-24T01:27:14.535218+0000","last_peered":"2024-03-24T01:27:14.535218+0000","last_clean":"2024-03-24T01:27:14.535218+0000","last_became_active":"2024-03-24T01:27:13.527205+0000","last_became_peered":"2024-03-24T01:27:13.527205+0000","last_unstale":"2024-03-24T01:27:14.535218+0000","last_undegraded":"2024-03-24T01:27:14.535218+0000","last_fullsized":"2024-03-24T01:27:14.535218+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:39.632343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.544743+0000","last_change":"2024-03-24T01:27:14.541241+0000","last_active":"2024-03-24T01:27:14.544743+0000","last_peered":"2024-03-24T01:27:14.544743+0000","last_clean":"2024-03-24T01:27:14.544743+0000","last_became_active":"2024-03-24T01:27:14.540872+0000","last_became_peered":"2024-03-24T01:27:14.540872+0000","last_unstale":"2024-03-24T01:27:14.544743+0000","last_undegraded":"2024-03-24T01:27:14.544743+0000","last_fullsized":"2024-03-24T01:27:14.544743+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:16:39.371501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.527060+0000","last_change":"2024-03-24T01:27:13.531379+0000","last_active":"2024-03-24T01:27:14.527060+0000","last_peered":"2024-03-24T01:27:14.527060+0000","last_clean":"2024-03-24T01:27:14.527060+0000","last_became_active":"2024-03-24T01:27:13.530618+0000","last_became_peered":"2024-03-24T01:27:13.530618+0000","last_unstale":"2024-03-24T01:27:14.527060+0000","last_undegraded":"2024-03-24T01:27:14.527060+0000","last_fullsized":"2024-03-24T01:27:14.527060+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:00.227179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.548287+0000","last_change":"2024-03-24T01:27:14.548287+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.548287+0000","last_undegraded":"2024-03-24T01:27:14.548287+0000","last_fullsized":"2024-03-24T01:27:14.548287+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.527604+0000","last_change":"2024-03-24T01:27:13.529478+0000","last_active":"2024-03-24T01:27:14.527604+0000","last_peered":"2024-03-24T01:27:14.527604+0000","last_clean":"2024-03-24T01:27:14.527604+0000","last_became_active":"2024-03-24T01:27:13.529197+0000","last_became_peered":"2024-03-24T01:27:13.529197+0000","last_unstale":"2024-03-24T01:27:14.527604+0000","last_undegraded":"2024-03-24T01:27:14.527604+0000","last_fullsized":"2024-03-24T01:27:14.527604+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:30:49.041062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.547194+0000","last_change":"2024-03-24T01:27:14.544605+0000","last_active":"2024-03-24T01:27:14.547194+0000","last_peered":"2024-03-24T01:27:14.547194+0000","last_clean":"2024-03-24T01:27:14.547194+0000","last_became_active":"2024-03-24T01:27:14.544189+0000","last_became_peered":"2024-03-24T01:27:14.544189+0000","last_unstale":"2024-03-24T01:27:14.547194+0000","last_undegraded":"2024-03-24T01:27:14.547194+0000","last_fullsized":"2024-03-24T01:27:14.547194+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:12:15.274601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.548563+0000","last_change":"2024-03-24T01:27:13.533618+0000","last_active":"2024-03-24T01:27:14.548563+0000","last_peered":"2024-03-24T01:27:14.548563+0000","last_clean":"2024-03-24T01:27:14.548563+0000","last_became_active":"2024-03-24T01:27:13.531829+0000","last_became_peered":"2024-03-24T01:27:13.531829+0000","last_unstale":"2024-03-24T01:27:14.548563+0000","last_undegraded":"2024-03-24T01:27:14.548563+0000","last_fullsized":"2024-03-24T01:27:14.548563+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:06:33.248202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.552880+0000","last_change":"2024-03-24T01:27:14.541924+0000","last_active":"2024-03-24T01:27:14.552880+0000","last_peered":"2024-03-24T01:27:14.552880+0000","last_clean":"2024-03-24T01:27:14.552880+0000","last_became_active":"2024-03-24T01:27:14.541512+0000","last_became_peered":"2024-03-24T01:27:14.541512+0000","last_unstale":"2024-03-24T01:27:14.552880+0000","last_undegraded":"2024-03-24T01:27:14.552880+0000","last_fullsized":"2024-03-24T01:27:14.552880+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:03.448771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.536520+0000","last_change":"2024-03-24T01:27:13.524033+0000","last_active":"2024-03-24T01:27:14.536520+0000","last_peered":"2024-03-24T01:27:14.536520+0000","last_clean":"2024-03-24T01:27:14.536520+0000","last_became_active":"2024-03-24T01:27:13.523502+0000","last_became_peered":"2024-03-24T01:27:13.523502+0000","last_unstale":"2024-03-24T01:27:14.536520+0000","last_undegraded":"2024-03-24T01:27:14.536520+0000","last_fullsized":"2024-03-24T01:27:14.536520+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:02:11.823211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.552947+0000","last_change":"2024-03-24T01:27:14.552947+0000","last_active":"2024-03-24T01:27:14.552947+0000","last_peered":"2024-03-24T01:27:14.552947+0000","last_clean":"2024-03-24T01:27:14.552947+0000","last_became_active":"2024-03-24T01:27:14.552583+0000","last_became_peered":"2024-03-24T01:27:14.552583+0000","last_unstale":"2024-03-24T01:27:14.552947+0000","last_undegraded":"2024-03-24T01:27:14.552947+0000","last_fullsized":"2024-03-24T01:27:14.552947+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:58.630025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"52'2","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.548205+0000","last_change":"2024-03-24T01:27:13.530329+0000","last_active":"2024-03-24T01:27:14.548205+0000","last_peered":"2024-03-24T01:27:14.548205+0000","last_clean":"2024-03-24T01:27:14.548205+0000","last_became_active":"2024-03-24T01:27:13.529972+0000","last_became_peered":"2024-03-24T01:27:13.529972+0000","last_unstale":"2024-03-24T01:27:14.548205+0000","last_undegraded":"2024-03-24T01:27:14.548205+0000","last_fullsized":"2024-03-24T01:27:14.548205+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:43.325731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.552486+0000","last_change":"2024-03-24T01:27:14.552486+0000","last_active":"2024-03-24T01:27:14.552486+0000","last_peered":"2024-03-24T01:27:14.552486+0000","last_clean":"2024-03-24T01:27:14.552486+0000","last_became_active":"2024-03-24T01:27:14.551883+0000","last_became_peered":"2024-03-24T01:27:14.551883+0000","last_unstale":"2024-03-24T01:27:14.552486+0000","last_undegraded":"2024-03-24T01:27:14.552486+0000","last_fullsized":"2024-03-24T01:27:14.552486+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:44.340192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"52'5","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.535485+0000","last_change":"2024-03-24T01:27:13.521128+0000","last_active":"2024-03-24T01:27:14.535485+0000","last_peered":"2024-03-24T01:27:14.535485+0000","last_clean":"2024-03-24T01:27:14.535485+0000","last_became_active":"2024-03-24T01:27:13.520358+0000","last_became_peered":"2024-03-24T01:27:13.520358+0000","last_unstale":"2024-03-24T01:27:14.535485+0000","last_undegraded":"2024-03-24T01:27:14.535485+0000","last_fullsized":"2024-03-24T01:27:14.535485+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:01:08.204732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.522002+0000","last_change":"2024-03-24T01:27:14.522002+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.522002+0000","last_undegraded":"2024-03-24T01:27:14.522002+0000","last_fullsized":"2024-03-24T01:27:14.522002+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.532355+0000","last_change":"2024-03-24T01:27:13.532029+0000","last_active":"2024-03-24T01:27:14.532355+0000","last_peered":"2024-03-24T01:27:14.532355+0000","last_clean":"2024-03-24T01:27:14.532355+0000","last_became_active":"2024-03-24T01:27:13.531765+0000","last_became_peered":"2024-03-24T01:27:13.531765+0000","last_unstale":"2024-03-24T01:27:14.532355+0000","last_undegraded":"2024-03-24T01:27:14.532355+0000","last_fullsized":"2024-03-24T01:27:14.532355+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:02:11.823211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.537552+0000","last_change":"2024-03-24T01:27:14.537552+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.537552+0000","last_undegraded":"2024-03-24T01:27:14.537552+0000","last_fullsized":"2024-03-24T01:27:14.537552+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.527543+0000","last_change":"2024-03-24T01:27:13.530136+0000","last_active":"2024-03-24T01:27:14.527543+0000","last_peered":"2024-03-24T01:27:14.527543+0000","last_clean":"2024-03-24T01:27:14.527543+0000","last_became_active":"2024-03-24T01:27:13.529811+0000","last_became_peered":"2024-03-24T01:27:13.529811+0000","last_unstale":"2024-03-24T01:27:14.527543+0000","last_undegraded":"2024-03-24T01:27:14.527543+0000","last_fullsized":"2024-03-24T01:27:14.527543+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:37:15.291837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.563469+0000","last_change":"2024-03-24T01:27:14.561422+0000","last_active":"2024-03-24T01:27:14.563469+0000","last_peered":"2024-03-24T01:27:14.563469+0000","last_clean":"2024-03-24T01:27:14.563469+0000","last_became_active":"2024-03-24T01:27:14.559580+0000","last_became_peered":"2024-03-24T01:27:14.559580+0000","last_unstale":"2024-03-24T01:27:14.563469+0000","last_undegraded":"2024-03-24T01:27:14.563469+0000","last_fullsized":"2024-03-24T01:27:14.563469+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:18:51.495804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.546920+0000","last_change":"2024-03-24T01:27:13.532579+0000","last_active":"2024-03-24T01:27:14.546920+0000","last_peered":"2024-03-24T01:27:14.546920+0000","last_clean":"2024-03-24T01:27:14.546920+0000","last_became_active":"2024-03-24T01:27:13.532264+0000","last_became_peered":"2024-03-24T01:27:13.532264+0000","last_unstale":"2024-03-24T01:27:14.546920+0000","last_undegraded":"2024-03-24T01:27:14.546920+0000","last_fullsized":"2024-03-24T01:27:14.546920+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:12:14.260140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.545505+0000","last_change":"2024-03-24T01:27:14.545505+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.545505+0000","last_undegraded":"2024-03-24T01:27:14.545505+0000","last_fullsized":"2024-03-24T01:27:14.545505+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"52'8","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.536546+0000","last_change":"2024-03-24T01:27:13.518331+0000","last_active":"2024-03-24T01:27:14.536546+0000","last_peered":"2024-03-24T01:27:14.536546+0000","last_clean":"2024-03-24T01:27:14.536546+0000","last_became_active":"2024-03-24T01:27:13.518017+0000","last_became_peered":"2024-03-24T01:27:13.518017+0000","last_unstale":"2024-03-24T01:27:14.536546+0000","last_undegraded":"2024-03-24T01:27:14.536546+0000","last_fullsized":"2024-03-24T01:27:14.536546+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:02:41.582283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.547545+0000","last_change":"2024-03-24T01:27:14.543017+0000","last_active":"2024-03-24T01:27:14.547545+0000","last_peered":"2024-03-24T01:27:14.547545+0000","last_clean":"2024-03-24T01:27:14.547545+0000","last_became_active":"2024-03-24T01:27:14.542730+0000","last_became_peered":"2024-03-24T01:27:14.542730+0000","last_unstale":"2024-03-24T01:27:14.547545+0000","last_undegraded":"2024-03-24T01:27:14.547545+0000","last_fullsized":"2024-03-24T01:27:14.547545+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:19:18.467994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.545934+0000","last_change":"2024-03-24T01:27:13.530251+0000","last_active":"2024-03-24T01:27:14.545934+0000","last_peered":"2024-03-24T01:27:14.545934+0000","last_clean":"2024-03-24T01:27:14.545934+0000","last_became_active":"2024-03-24T01:27:13.529767+0000","last_became_peered":"2024-03-24T01:27:13.529767+0000","last_unstale":"2024-03-24T01:27:14.545934+0000","last_undegraded":"2024-03-24T01:27:14.545934+0000","last_fullsized":"2024-03-24T01:27:14.545934+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:43:44.823891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.557227+0000","last_change":"2024-03-24T01:27:14.554474+0000","last_active":"2024-03-24T01:27:14.557227+0000","last_peered":"2024-03-24T01:27:14.557227+0000","last_clean":"2024-03-24T01:27:14.557227+0000","last_became_active":"2024-03-24T01:27:14.554189+0000","last_became_peered":"2024-03-24T01:27:14.554189+0000","last_unstale":"2024-03-24T01:27:14.557227+0000","last_undegraded":"2024-03-24T01:27:14.557227+0000","last_fullsized":"2024-03-24T01:27:14.557227+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:43:45.838352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.529435+0000","last_change":"2024-03-24T01:27:13.526531+0000","last_active":"2024-03-24T01:27:14.529435+0000","last_peered":"2024-03-24T01:27:14.529435+0000","last_clean":"2024-03-24T01:27:14.529435+0000","last_became_active":"2024-03-24T01:27:13.526212+0000","last_became_peered":"2024-03-24T01:27:13.526212+0000","last_unstale":"2024-03-24T01:27:14.529435+0000","last_undegraded":"2024-03-24T01:27:14.529435+0000","last_fullsized":"2024-03-24T01:27:14.529435+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:02:41.656569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.531995+0000","last_change":"2024-03-24T01:27:14.531995+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.531995+0000","last_undegraded":"2024-03-24T01:27:14.531995+0000","last_fullsized":"2024-03-24T01:27:14.531995+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.529356+0000","last_change":"2024-03-24T01:27:13.526615+0000","last_active":"2024-03-24T01:27:14.529356+0000","last_peered":"2024-03-24T01:27:14.529356+0000","last_clean":"2024-03-24T01:27:14.529356+0000","last_became_active":"2024-03-24T01:27:13.526286+0000","last_became_peered":"2024-03-24T01:27:13.526286+0000","last_unstale":"2024-03-24T01:27:14.529356+0000","last_undegraded":"2024-03-24T01:27:14.529356+0000","last_fullsized":"2024-03-24T01:27:14.529356+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:42:00.908936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.547136+0000","last_change":"2024-03-24T01:27:14.539239+0000","last_active":"2024-03-24T01:27:14.547136+0000","last_peered":"2024-03-24T01:27:14.547136+0000","last_clean":"2024-03-24T01:27:14.547136+0000","last_became_active":"2024-03-24T01:27:14.538525+0000","last_became_peered":"2024-03-24T01:27:14.538525+0000","last_unstale":"2024-03-24T01:27:14.547136+0000","last_undegraded":"2024-03-24T01:27:14.547136+0000","last_fullsized":"2024-03-24T01:27:14.547136+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:42:01.923397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.528640+0000","last_change":"2024-03-24T01:27:13.523480+0000","last_active":"2024-03-24T01:27:14.528640+0000","last_peered":"2024-03-24T01:27:14.528640+0000","last_clean":"2024-03-24T01:27:14.528640+0000","last_became_active":"2024-03-24T01:27:13.523203+0000","last_became_peered":"2024-03-24T01:27:13.523203+0000","last_unstale":"2024-03-24T01:27:14.528640+0000","last_undegraded":"2024-03-24T01:27:14.528640+0000","last_fullsized":"2024-03-24T01:27:14.528640+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:00:24.458371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.557519+0000","last_change":"2024-03-24T01:27:14.549953+0000","last_active":"2024-03-24T01:27:14.557519+0000","last_peered":"2024-03-24T01:27:14.557519+0000","last_clean":"2024-03-24T01:27:14.557519+0000","last_became_active":"2024-03-24T01:27:14.549616+0000","last_became_peered":"2024-03-24T01:27:14.549616+0000","last_unstale":"2024-03-24T01:27:14.557519+0000","last_undegraded":"2024-03-24T01:27:14.557519+0000","last_fullsized":"2024-03-24T01:27:14.557519+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:32:58.788010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.518549+0000","last_change":"2024-03-24T01:27:13.526041+0000","last_active":"2024-03-24T01:27:14.518549+0000","last_peered":"2024-03-24T01:27:14.518549+0000","last_clean":"2024-03-24T01:27:14.518549+0000","last_became_active":"2024-03-24T01:27:13.525587+0000","last_became_peered":"2024-03-24T01:27:13.525587+0000","last_unstale":"2024-03-24T01:27:14.518549+0000","last_undegraded":"2024-03-24T01:27:14.518549+0000","last_fullsized":"2024-03-24T01:27:14.518549+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:30.183145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.546589+0000","last_change":"2024-03-24T01:27:14.539787+0000","last_active":"2024-03-24T01:27:14.546589+0000","last_peered":"2024-03-24T01:27:14.546589+0000","last_clean":"2024-03-24T01:27:14.546589+0000","last_became_active":"2024-03-24T01:27:14.539446+0000","last_became_peered":"2024-03-24T01:27:14.539446+0000","last_unstale":"2024-03-24T01:27:14.546589+0000","last_undegraded":"2024-03-24T01:27:14.546589+0000","last_fullsized":"2024-03-24T01:27:14.546589+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:02:12.837672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.535722+0000","last_change":"2024-03-24T01:27:13.519780+0000","last_active":"2024-03-24T01:27:14.535722+0000","last_peered":"2024-03-24T01:27:14.535722+0000","last_clean":"2024-03-24T01:27:14.535722+0000","last_became_active":"2024-03-24T01:27:13.519458+0000","last_became_peered":"2024-03-24T01:27:13.519458+0000","last_unstale":"2024-03-24T01:27:14.535722+0000","last_undegraded":"2024-03-24T01:27:14.535722+0000","last_fullsized":"2024-03-24T01:27:14.535722+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:46:44.745424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.549326+0000","last_change":"2024-03-24T01:27:14.549326+0000","last_active":"2024-03-24T01:27:14.549326+0000","last_peered":"2024-03-24T01:27:14.549326+0000","last_clean":"2024-03-24T01:27:14.549326+0000","last_became_active":"2024-03-24T01:27:14.548886+0000","last_became_peered":"2024-03-24T01:27:14.548886+0000","last_unstale":"2024-03-24T01:27:14.549326+0000","last_undegraded":"2024-03-24T01:27:14.549326+0000","last_fullsized":"2024-03-24T01:27:14.549326+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:03.448771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.545176+0000","last_change":"2024-03-24T01:27:13.521689+0000","last_active":"2024-03-24T01:27:14.545176+0000","last_peered":"2024-03-24T01:27:14.545176+0000","last_clean":"2024-03-24T01:27:14.545176+0000","last_became_active":"2024-03-24T01:27:13.521436+0000","last_became_peered":"2024-03-24T01:27:13.521436+0000","last_unstale":"2024-03-24T01:27:14.545176+0000","last_undegraded":"2024-03-24T01:27:14.545176+0000","last_fullsized":"2024-03-24T01:27:14.545176+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:47:38.446831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.551289+0000","last_change":"2024-03-24T01:27:14.543585+0000","last_active":"2024-03-24T01:27:14.551289+0000","last_peered":"2024-03-24T01:27:14.551289+0000","last_clean":"2024-03-24T01:27:14.551289+0000","last_became_active":"2024-03-24T01:27:14.543301+0000","last_became_peered":"2024-03-24T01:27:14.543301+0000","last_unstale":"2024-03-24T01:27:14.551289+0000","last_undegraded":"2024-03-24T01:27:14.551289+0000","last_fullsized":"2024-03-24T01:27:14.551289+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:12:15.274601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.527295+0000","last_change":"2024-03-24T01:27:13.522613+0000","last_active":"2024-03-24T01:27:14.527295+0000","last_peered":"2024-03-24T01:27:14.527295+0000","last_clean":"2024-03-24T01:27:14.527295+0000","last_became_active":"2024-03-24T01:27:13.522331+0000","last_became_peered":"2024-03-24T01:27:13.522331+0000","last_unstale":"2024-03-24T01:27:14.527295+0000","last_undegraded":"2024-03-24T01:27:14.527295+0000","last_fullsized":"2024-03-24T01:27:14.527295+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:08:09.036710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.558092+0000","last_change":"2024-03-24T01:27:14.550330+0000","last_active":"2024-03-24T01:27:14.558092+0000","last_peered":"2024-03-24T01:27:14.558092+0000","last_clean":"2024-03-24T01:27:14.558092+0000","last_became_active":"2024-03-24T01:27:14.550056+0000","last_became_peered":"2024-03-24T01:27:14.550056+0000","last_unstale":"2024-03-24T01:27:14.558092+0000","last_undegraded":"2024-03-24T01:27:14.558092+0000","last_fullsized":"2024-03-24T01:27:14.558092+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:23.971971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.529686+0000","last_change":"2024-03-24T01:27:13.533418+0000","last_active":"2024-03-24T01:27:14.529686+0000","last_peered":"2024-03-24T01:27:14.529686+0000","last_clean":"2024-03-24T01:27:14.529686+0000","last_became_active":"2024-03-24T01:27:13.533164+0000","last_became_peered":"2024-03-24T01:27:13.533164+0000","last_unstale":"2024-03-24T01:27:14.529686+0000","last_undegraded":"2024-03-24T01:27:14.529686+0000","last_fullsized":"2024-03-24T01:27:14.529686+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:30:49.041062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.551209+0000","last_change":"2024-03-24T01:27:14.543832+0000","last_active":"2024-03-24T01:27:14.551209+0000","last_peered":"2024-03-24T01:27:14.551209+0000","last_clean":"2024-03-24T01:27:14.551209+0000","last_became_active":"2024-03-24T01:27:14.543520+0000","last_became_peered":"2024-03-24T01:27:14.543520+0000","last_unstale":"2024-03-24T01:27:14.551209+0000","last_undegraded":"2024-03-24T01:27:14.551209+0000","last_fullsized":"2024-03-24T01:27:14.551209+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:44.340192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.548373+0000","last_change":"2024-03-24T01:27:13.530300+0000","last_active":"2024-03-24T01:27:14.548373+0000","last_peered":"2024-03-24T01:27:14.548373+0000","last_clean":"2024-03-24T01:27:14.548373+0000","last_became_active":"2024-03-24T01:27:13.529919+0000","last_became_peered":"2024-03-24T01:27:13.529919+0000","last_unstale":"2024-03-24T01:27:14.548373+0000","last_undegraded":"2024-03-24T01:27:14.548373+0000","last_fullsized":"2024-03-24T01:27:14.548373+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:47:38.446831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.548304+0000","last_change":"2024-03-24T01:27:14.544735+0000","last_active":"2024-03-24T01:27:14.548304+0000","last_peered":"2024-03-24T01:27:14.548304+0000","last_clean":"2024-03-24T01:27:14.548304+0000","last_became_active":"2024-03-24T01:27:14.544460+0000","last_became_peered":"2024-03-24T01:27:14.544460+0000","last_unstale":"2024-03-24T01:27:14.548304+0000","last_undegraded":"2024-03-24T01:27:14.548304+0000","last_fullsized":"2024-03-24T01:27:14.548304+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:06.757918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.536983+0000","last_change":"2024-03-24T01:27:13.525558+0000","last_active":"2024-03-24T01:27:14.536983+0000","last_peered":"2024-03-24T01:27:14.536983+0000","last_clean":"2024-03-24T01:27:14.536983+0000","last_became_active":"2024-03-24T01:27:13.525223+0000","last_became_peered":"2024-03-24T01:27:13.525223+0000","last_unstale":"2024-03-24T01:27:14.536983+0000","last_undegraded":"2024-03-24T01:27:14.536983+0000","last_fullsized":"2024-03-24T01:27:14.536983+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:43.325731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.549747+0000","last_change":"2024-03-24T01:27:14.549747+0000","last_active":"2024-03-24T01:27:14.549747+0000","last_peered":"2024-03-24T01:27:14.549747+0000","last_clean":"2024-03-24T01:27:14.549747+0000","last_became_active":"2024-03-24T01:27:14.548344+0000","last_became_peered":"2024-03-24T01:27:14.548344+0000","last_unstale":"2024-03-24T01:27:14.549747+0000","last_undegraded":"2024-03-24T01:27:14.549747+0000","last_fullsized":"2024-03-24T01:27:14.549747+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:32.037863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.535645+0000","last_change":"2024-03-24T01:27:13.519855+0000","last_active":"2024-03-24T01:27:14.535645+0000","last_peered":"2024-03-24T01:27:14.535645+0000","last_clean":"2024-03-24T01:27:14.535645+0000","last_became_active":"2024-03-24T01:27:13.519566+0000","last_became_peered":"2024-03-24T01:27:13.519566+0000","last_unstale":"2024-03-24T01:27:14.535645+0000","last_undegraded":"2024-03-24T01:27:14.535645+0000","last_fullsized":"2024-03-24T01:27:14.535645+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:57.615564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.522676+0000","last_change":"2024-03-24T01:27:14.522676+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.522676+0000","last_undegraded":"2024-03-24T01:27:14.522676+0000","last_fullsized":"2024-03-24T01:27:14.522676+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.535933+0000","last_change":"2024-03-24T01:27:13.524780+0000","last_active":"2024-03-24T01:27:14.535933+0000","last_peered":"2024-03-24T01:27:14.535933+0000","last_clean":"2024-03-24T01:27:14.535933+0000","last_became_active":"2024-03-24T01:27:13.524518+0000","last_became_peered":"2024-03-24T01:27:13.524518+0000","last_unstale":"2024-03-24T01:27:14.535933+0000","last_undegraded":"2024-03-24T01:27:14.535933+0000","last_fullsized":"2024-03-24T01:27:14.535933+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:14:15.902136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.534874+0000","last_change":"2024-03-24T01:27:14.534874+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.534874+0000","last_undegraded":"2024-03-24T01:27:14.534874+0000","last_fullsized":"2024-03-24T01:27:14.534874+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.534655+0000","last_change":"2024-03-24T01:27:13.519549+0000","last_active":"2024-03-24T01:27:14.534655+0000","last_peered":"2024-03-24T01:27:14.534655+0000","last_clean":"2024-03-24T01:27:14.534655+0000","last_became_active":"2024-03-24T01:27:13.519174+0000","last_became_peered":"2024-03-24T01:27:13.519174+0000","last_unstale":"2024-03-24T01:27:14.534655+0000","last_undegraded":"2024-03-24T01:27:14.534655+0000","last_fullsized":"2024-03-24T01:27:14.534655+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:30:49.041062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.549972+0000","last_change":"2024-03-24T01:27:14.548520+0000","last_active":"2024-03-24T01:27:14.549972+0000","last_peered":"2024-03-24T01:27:14.549972+0000","last_clean":"2024-03-24T01:27:14.549972+0000","last_became_active":"2024-03-24T01:27:14.547968+0000","last_became_peered":"2024-03-24T01:27:14.547968+0000","last_unstale":"2024-03-24T01:27:14.549972+0000","last_undegraded":"2024-03-24T01:27:14.549972+0000","last_fullsized":"2024-03-24T01:27:14.549972+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:36.294802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.520961+0000","last_change":"2024-03-24T01:27:13.524804+0000","last_active":"2024-03-24T01:27:14.520961+0000","last_peered":"2024-03-24T01:27:14.520961+0000","last_clean":"2024-03-24T01:27:14.520961+0000","last_became_active":"2024-03-24T01:27:13.524487+0000","last_became_peered":"2024-03-24T01:27:13.524487+0000","last_unstale":"2024-03-24T01:27:14.520961+0000","last_undegraded":"2024-03-24T01:27:14.520961+0000","last_fullsized":"2024-03-24T01:27:14.520961+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:43:44.823891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.546611+0000","last_change":"2024-03-24T01:27:14.539180+0000","last_active":"2024-03-24T01:27:14.546611+0000","last_peered":"2024-03-24T01:27:14.546611+0000","last_clean":"2024-03-24T01:27:14.546611+0000","last_became_active":"2024-03-24T01:27:14.538800+0000","last_became_peered":"2024-03-24T01:27:14.538800+0000","last_unstale":"2024-03-24T01:27:14.546611+0000","last_undegraded":"2024-03-24T01:27:14.546611+0000","last_fullsized":"2024-03-24T01:27:14.546611+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:40.646803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.535286+0000","last_change":"2024-03-24T01:27:13.532470+0000","last_active":"2024-03-24T01:27:14.535286+0000","last_peered":"2024-03-24T01:27:14.535286+0000","last_clean":"2024-03-24T01:27:14.535286+0000","last_became_active":"2024-03-24T01:27:13.530705+0000","last_became_peered":"2024-03-24T01:27:13.530705+0000","last_unstale":"2024-03-24T01:27:14.535286+0000","last_undegraded":"2024-03-24T01:27:14.535286+0000","last_fullsized":"2024-03-24T01:27:14.535286+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:12:14.260140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":229,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.545758+0000","last_change":"2024-03-24T01:24:01.545855+0000","last_active":"2024-03-24T01:27:14.545758+0000","last_peered":"2024-03-24T01:27:14.545758+0000","last_clean":"2024-03-24T01:27:14.545758+0000","last_became_active":"2024-03-24T01:24:01.543811+0000","last_became_peered":"2024-03-24T01:24:01.543811+0000","last_unstale":"2024-03-24T01:27:14.545758+0000","last_undegraded":"2024-03-24T01:27:14.545758+0000","last_fullsized":"2024-03-24T01:27:14.545758+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_clean_scrub_stamp":"2024-03-24T01:23:59.861564+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:03.682631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":672,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15,"ondisk_log_size":15,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574890,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28196,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343644,"statfs":{"total":96636764160,"available":96607891456,"internally_reserved":0,"allocated":1200128,"data_stored":861179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":270811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":270811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574890,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":270811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":270811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574891,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":270811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":618496,"data_stored":277978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574891,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28196,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343644,"statfs":{"total":96636764160,"available":96607891456,"internally_reserved":0,"allocated":1200128,"data_stored":861179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T01:27:19.463 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T01:27:19.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.712+0000 7f0f33e00640 1 Processor -- start 2024-03-24T01:27:19.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.712+0000 7f0f33e00640 1 -- start start 2024-03-24T01:27:19.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.713+0000 7f0f33e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f34144420 0x7f0f3406d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:19.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.713+0000 7f0f33e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f0f34144800 0x7f0f340cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:19.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.713+0000 7f0f33e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f340cceb0 con 0x7f0f340cc8f0 2024-03-24T01:27:19.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.713+0000 7f0f33e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f340cd190 con 0x7f0f34144800 2024-03-24T01:27:19.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.713+0000 7f0f33e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f340ceb70 con 0x7f0f34144420 2024-03-24T01:27:19.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.714+0000 7f0f33400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f0f340cc8f0 0x7f0f340cf8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60318/0 (socket says 172.21.15.90:60318) 2024-03-24T01:27:19.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.714+0000 7f0f33400640 1 -- 172.21.15.90:0/830028592 learned_addr learned my addr 172.21.15.90:0/830028592 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:27:19.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.714+0000 7f0f32000640 1 --2- 172.21.15.90:0/830028592 >> v2:172.21.15.120:3300/0 conn(0x7f0f34144800 0x7f0f340cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:19.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.714+0000 7f0f32a00640 1 --2- 172.21.15.90:0/830028592 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f34144420 0x7f0f3406d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:19.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.714+0000 7f0f31600640 1 -- 172.21.15.90:0/830028592 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3726180923 0 0) 0x7f0f340cceb0 con 0x7f0f340cc8f0 2024-03-24T01:27:19.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.714+0000 7f0f31600640 1 -- 172.21.15.90:0/830028592 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f18003520 con 0x7f0f340cc8f0 2024-03-24T01:27:19.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.714+0000 7f0f31600640 1 -- 172.21.15.90:0/830028592 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3307747095 0 0) 0x7f0f18003520 con 0x7f0f340cc8f0 2024-03-24T01:27:19.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.715+0000 7f0f31600640 1 -- 172.21.15.90:0/830028592 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f34144420 msgr2=0x7f0f3406d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:19.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.715+0000 7f0f31600640 1 --2- 172.21.15.90:0/830028592 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f34144420 0x7f0f3406d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:19.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.715+0000 7f0f31600640 1 -- 172.21.15.90:0/830028592 >> v2:172.21.15.120:3300/0 conn(0x7f0f34144800 msgr2=0x7f0f340cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:19.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.715+0000 7f0f31600640 1 --2- 172.21.15.90:0/830028592 >> v2:172.21.15.120:3300/0 conn(0x7f0f34144800 0x7f0f340cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:19.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.715+0000 7f0f31600640 1 -- 172.21.15.90:0/830028592 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f340cee50 con 0x7f0f340cc8f0 2024-03-24T01:27:19.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.715+0000 7f0f31600640 1 -- 172.21.15.90:0/830028592 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f0f2c0019f0 con 0x7f0f340cc8f0 2024-03-24T01:27:19.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.715+0000 7f0f31600640 1 -- 172.21.15.90:0/830028592 >> v1:172.21.15.90:6789/0 conn(0x7f0f340cc8f0 legacy=0x7f0f340cf8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:19.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.715+0000 7f0f31600640 1 --2- 172.21.15.90:0/830028592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f180039b0 0x7f0f18003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:19.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.715+0000 7f0f31600640 1 --2- 172.21.15.90:0/830028592 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f34144420 0x7f0f18004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:19.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.715+0000 7f0f31600640 1 --2- 172.21.15.90:0/830028592 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f18004d40 0x7f0f180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:19.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.715+0000 7f0f31600640 1 -- 172.21.15.90:0/830028592 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f340cd190 con 0x7f0f18004d40 2024-03-24T01:27:19.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.715+0000 7f0f31600640 1 -- 172.21.15.90:0/830028592 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f340ceb70 con 0x7f0f180039b0 2024-03-24T01:27:19.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.715+0000 7f0f31600640 1 -- 172.21.15.90:0/830028592 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1800ba40 con 0x7f0f34144420 2024-03-24T01:27:19.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.715+0000 7f0f33400640 1 --2- 172.21.15.90:0/830028592 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f18004d40 0x7f0f180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:19.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.715+0000 7f0f33400640 1 -- 172.21.15.90:0/830028592 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f34144420 msgr2=0x7f0f18004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:19.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.715+0000 7f0f32000640 1 --2- 172.21.15.90:0/830028592 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f34144420 0x7f0f18004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:19.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.715+0000 7f0f32a00640 1 --2- 172.21.15.90:0/830028592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f180039b0 0x7f0f18003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:19.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.715+0000 7f0f33400640 1 --2- 172.21.15.90:0/830028592 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f34144420 0x7f0f18004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:19.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.715+0000 7f0f33400640 1 -- 172.21.15.90:0/830028592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f180039b0 msgr2=0x7f0f18003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:19.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.715+0000 7f0f33400640 1 --2- 172.21.15.90:0/830028592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f180039b0 0x7f0f18003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:19.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.715+0000 7f0f33400640 1 -- 172.21.15.90:0/830028592 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f18003520 con 0x7f0f18004d40 2024-03-24T01:27:19.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.716+0000 7f0f33400640 1 --2- 172.21.15.90:0/830028592 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f18004d40 0x7f0f180051c0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c00a920 tx=0x7f0f18005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20baeb9f1c9b1dd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:19.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.717+0000 7f0f31600640 1 -- 172.21.15.90:0/830028592 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c00a4c0 con 0x7f0f18004d40 2024-03-24T01:27:19.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.717+0000 7f0f31600640 1 -- 172.21.15.90:0/830028592 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f2c034650 con 0x7f0f18004d40 2024-03-24T01:27:19.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.717+0000 7f0f31600640 1 -- 172.21.15.90:0/830028592 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c034a40 con 0x7f0f18004d40 2024-03-24T01:27:19.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.717+0000 7f0f33e00640 1 -- 172.21.15.90:0/830028592 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f18004d40 msgr2=0x7f0f180051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:19.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.717+0000 7f0f33e00640 1 --2- 172.21.15.90:0/830028592 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f18004d40 0x7f0f180051c0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c00a920 tx=0x7f0f18005830 comp rx=0 tx=0).stop 2024-03-24T01:27:19.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.717+0000 7f0f33400640 1 -- 172.21.15.90:0/830028592 reap_dead start 2024-03-24T01:27:19.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.717+0000 7f0f33e00640 1 -- 172.21.15.90:0/830028592 shutdown_connections 2024-03-24T01:27:19.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.717+0000 7f0f33e00640 1 -- 172.21.15.90:0/830028592 >> 172.21.15.90:0/830028592 conn(0x7f0f3406b0c0 msgr2=0x7f0f3406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:19.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.717+0000 7f0f33e00640 1 -- 172.21.15.90:0/830028592 shutdown_connections 2024-03-24T01:27:19.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.718+0000 7f0f33e00640 1 -- 172.21.15.90:0/830028592 wait complete. 2024-03-24T01:27:19.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.719+0000 7f0f33e00640 1 Processor -- start 2024-03-24T01:27:19.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.719+0000 7f0f33e00640 1 -- start start 2024-03-24T01:27:19.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.720+0000 7f0f33e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f18004d40 0x7f0f341f6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:19.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.720+0000 7f0f33e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f341419e0 0x7f0f341f4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:19.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.720+0000 7f0f33e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f341e2ff0 0x7f0f341e34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:19.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.720+0000 7f0f33e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f340ce910 con 0x7f0f341419e0 2024-03-24T01:27:19.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.720+0000 7f0f33e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f340ce790 con 0x7f0f18004d40 2024-03-24T01:27:19.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.720+0000 7f0f33e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3406d900 con 0x7f0f341e2ff0 2024-03-24T01:27:19.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.721+0000 7f0f32000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f341419e0 0x7f0f341f4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:19.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.721+0000 7f0f32000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f341419e0 0x7f0f341f4230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59000/0 (socket says 172.21.15.90:59000) 2024-03-24T01:27:19.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.721+0000 7f0f32000640 1 -- 172.21.15.90:0/1908687502 learned_addr learned my addr 172.21.15.90:0/1908687502 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:19.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.721+0000 7f0f33400640 1 --2- 172.21.15.90:0/1908687502 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f341e2ff0 0x7f0f341e34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:19.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.721+0000 7f0f32000640 1 -- 172.21.15.90:0/1908687502 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f341e2ff0 msgr2=0x7f0f341e34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:19.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.721+0000 7f0f32a00640 1 --2- 172.21.15.90:0/1908687502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f18004d40 0x7f0f341f6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:19.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.721+0000 7f0f32000640 1 --2- 172.21.15.90:0/1908687502 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f341e2ff0 0x7f0f341e34a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:19.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.721+0000 7f0f32000640 1 -- 172.21.15.90:0/1908687502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f18004d40 msgr2=0x7f0f341f6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:19.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.721+0000 7f0f32000640 1 --2- 172.21.15.90:0/1908687502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f18004d40 0x7f0f341f6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:19.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.721+0000 7f0f32000640 1 -- 172.21.15.90:0/1908687502 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f341f4770 con 0x7f0f341419e0 2024-03-24T01:27:19.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.721+0000 7f0f32000640 1 --2- 172.21.15.90:0/1908687502 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f341419e0 0x7f0f341f4230 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f0f20010910 tx=0x7f0f20010dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:19.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.721+0000 7f0f2b400640 1 -- 172.21.15.90:0/1908687502 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2000f040 con 0x7f0f341419e0 2024-03-24T01:27:19.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.722+0000 7f0f2b400640 1 -- 172.21.15.90:0/1908687502 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f2000ec20 con 0x7f0f341419e0 2024-03-24T01:27:19.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.722+0000 7f0f2b400640 1 -- 172.21.15.90:0/1908687502 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2001c400 con 0x7f0f341419e0 2024-03-24T01:27:19.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.722+0000 7f0f33e00640 1 -- 172.21.15.90:0/1908687502 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f34141e80 con 0x7f0f341419e0 2024-03-24T01:27:19.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.722+0000 7f0f33e00640 1 -- 172.21.15.90:0/1908687502 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f34142440 con 0x7f0f341419e0 2024-03-24T01:27:19.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.723+0000 7f0f33e00640 1 -- 172.21.15.90:0/1908687502 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f3404f400 con 0x7f0f341419e0 2024-03-24T01:27:19.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.724+0000 7f0f2b400640 1 -- 172.21.15.90:0/1908687502 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0f2001c5a0 con 0x7f0f341419e0 2024-03-24T01:27:19.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.725+0000 7f0f2b400640 1 --2- 172.21.15.90:0/1908687502 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0f1406bbd0 0x7f0f1406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:19.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.725+0000 7f0f2b400640 1 -- 172.21.15.90:0/1908687502 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0f2008b320 con 0x7f0f341419e0 2024-03-24T01:27:19.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.725+0000 7f0f32a00640 1 --2- 172.21.15.90:0/1908687502 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0f1406bbd0 0x7f0f1406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:19.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.726+0000 7f0f32a00640 1 --2- 172.21.15.90:0/1908687502 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0f1406bbd0 0x7f0f1406e090 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f0f24009910 tx=0x7f0f24008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:19.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.729+0000 7f0f2b400640 1 -- 172.21.15.90:0/1908687502 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0f341f4770 con 0x7f0f341419e0 2024-03-24T01:27:19.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.926+0000 7f0f33e00640 1 -- 172.21.15.90:0/1908687502 --> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0f340cd5f0 con 0x7f0f1406bbd0 2024-03-24T01:27:19.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.945+0000 7f0f2b400640 1 -- 172.21.15.90:0/1908687502 <== mgr.4104 v2:172.21.15.120:6832/860392634 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+250916 (secure 0 0 0) 0x7f0f340cd5f0 con 0x7f0f1406bbd0 2024-03-24T01:27:19.948 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:27:19.951 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-24T01:27:19.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.951+0000 7f0f33e00640 1 -- 172.21.15.90:0/1908687502 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0f1406bbd0 msgr2=0x7f0f1406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:19.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.951+0000 7f0f33e00640 1 --2- 172.21.15.90:0/1908687502 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0f1406bbd0 0x7f0f1406e090 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f0f24009910 tx=0x7f0f24008040 comp rx=0 tx=0).stop 2024-03-24T01:27:19.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.951+0000 7f0f33e00640 1 -- 172.21.15.90:0/1908687502 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f341419e0 msgr2=0x7f0f341f4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:19.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.951+0000 7f0f33e00640 1 --2- 172.21.15.90:0/1908687502 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f341419e0 0x7f0f341f4230 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f0f20010910 tx=0x7f0f20010dd0 comp rx=0 tx=0).stop 2024-03-24T01:27:19.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.951+0000 7f0f33e00640 1 -- 172.21.15.90:0/1908687502 shutdown_connections 2024-03-24T01:27:19.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.951+0000 7f0f33e00640 1 --2- 172.21.15.90:0/1908687502 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0f1406bbd0 0x7f0f1406e090 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:19.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.951+0000 7f0f33e00640 1 --2- 172.21.15.90:0/1908687502 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f341419e0 0x7f0f341f4230 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:19.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.951+0000 7f0f33e00640 1 --2- 172.21.15.90:0/1908687502 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0f341e2ff0 0x7f0f341e34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:19.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.951+0000 7f0f33e00640 1 --2- 172.21.15.90:0/1908687502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f18004d40 0x7f0f341f6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:19.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.951+0000 7f0f33e00640 1 -- 172.21.15.90:0/1908687502 >> 172.21.15.90:0/1908687502 conn(0x7f0f3406b0c0 msgr2=0x7f0f340cbbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:19.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.952+0000 7f0f33e00640 1 -- 172.21.15.90:0/1908687502 shutdown_connections 2024-03-24T01:27:19.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:19.952+0000 7f0f33e00640 1 -- 172.21.15.90:0/1908687502 wait complete. 2024-03-24T01:27:19.973 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":146,"stamp":"2024-03-24T01:27:19.549613+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":179,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132,"ondisk_log_size":132,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222104,"kb_used_data":5908,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752616,"statfs":{"total":773094113280,"available":772866678784,"internally_reserved":0,"allocated":6049792,"data_stored":3354391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12747,"internal_metadata":221236789},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":672,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.055385"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.547093+0000","last_change":"2024-03-24T01:27:14.540765+0000","last_active":"2024-03-24T01:27:14.547093+0000","last_peered":"2024-03-24T01:27:14.547093+0000","last_clean":"2024-03-24T01:27:14.547093+0000","last_became_active":"2024-03-24T01:27:14.539541+0000","last_became_peered":"2024-03-24T01:27:14.539541+0000","last_unstale":"2024-03-24T01:27:14.547093+0000","last_undegraded":"2024-03-24T01:27:14.547093+0000","last_fullsized":"2024-03-24T01:27:14.547093+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:30:50.055523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.559986+0000","last_change":"2024-03-24T01:27:14.559986+0000","last_active":"2024-03-24T01:27:14.559986+0000","last_peered":"2024-03-24T01:27:14.559986+0000","last_clean":"2024-03-24T01:27:14.559986+0000","last_became_active":"2024-03-24T01:27:14.559649+0000","last_became_peered":"2024-03-24T01:27:14.559649+0000","last_unstale":"2024-03-24T01:27:14.559986+0000","last_undegraded":"2024-03-24T01:27:14.559986+0000","last_fullsized":"2024-03-24T01:27:14.559986+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:23.971971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.538645+0000","last_change":"2024-03-24T01:27:14.538645+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.538645+0000","last_undegraded":"2024-03-24T01:27:14.538645+0000","last_fullsized":"2024-03-24T01:27:14.538645+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.559476+0000","last_change":"2024-03-24T01:27:14.559476+0000","last_active":"2024-03-24T01:27:14.559476+0000","last_peered":"2024-03-24T01:27:14.559476+0000","last_clean":"2024-03-24T01:27:14.559476+0000","last_became_active":"2024-03-24T01:27:14.559176+0000","last_became_peered":"2024-03-24T01:27:14.559176+0000","last_unstale":"2024-03-24T01:27:14.559476+0000","last_undegraded":"2024-03-24T01:27:14.559476+0000","last_fullsized":"2024-03-24T01:27:14.559476+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:03.448771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.531210+0000","last_change":"2024-03-24T01:27:14.531210+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.531210+0000","last_undegraded":"2024-03-24T01:27:14.531210+0000","last_fullsized":"2024-03-24T01:27:14.531210+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.555238+0000","last_change":"2024-03-24T01:27:14.551494+0000","last_active":"2024-03-24T01:27:14.555238+0000","last_peered":"2024-03-24T01:27:14.555238+0000","last_clean":"2024-03-24T01:27:14.555238+0000","last_became_active":"2024-03-24T01:27:14.551175+0000","last_became_peered":"2024-03-24T01:27:14.551175+0000","last_unstale":"2024-03-24T01:27:14.555238+0000","last_undegraded":"2024-03-24T01:27:14.555238+0000","last_fullsized":"2024-03-24T01:27:14.555238+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:06:34.262662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.552294+0000","last_change":"2024-03-24T01:27:14.552294+0000","last_active":"2024-03-24T01:27:14.552294+0000","last_peered":"2024-03-24T01:27:14.552294+0000","last_clean":"2024-03-24T01:27:14.552294+0000","last_became_active":"2024-03-24T01:27:14.551839+0000","last_became_peered":"2024-03-24T01:27:14.551839+0000","last_unstale":"2024-03-24T01:27:14.552294+0000","last_undegraded":"2024-03-24T01:27:14.552294+0000","last_fullsized":"2024-03-24T01:27:14.552294+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:01:09.219193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.562359+0000","last_change":"2024-03-24T01:27:14.558536+0000","last_active":"2024-03-24T01:27:14.562359+0000","last_peered":"2024-03-24T01:27:14.562359+0000","last_clean":"2024-03-24T01:27:14.562359+0000","last_became_active":"2024-03-24T01:27:14.558211+0000","last_became_peered":"2024-03-24T01:27:14.558211+0000","last_unstale":"2024-03-24T01:27:14.562359+0000","last_undegraded":"2024-03-24T01:27:14.562359+0000","last_fullsized":"2024-03-24T01:27:14.562359+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:44:39.116139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.539698+0000","last_change":"2024-03-24T01:27:14.539698+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.539698+0000","last_undegraded":"2024-03-24T01:27:14.539698+0000","last_fullsized":"2024-03-24T01:27:14.539698+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.547500+0000","last_change":"2024-03-24T01:27:14.541578+0000","last_active":"2024-03-24T01:27:14.547500+0000","last_peered":"2024-03-24T01:27:14.547500+0000","last_clean":"2024-03-24T01:27:14.547500+0000","last_became_active":"2024-03-24T01:27:14.541289+0000","last_became_peered":"2024-03-24T01:27:14.541289+0000","last_unstale":"2024-03-24T01:27:14.547500+0000","last_undegraded":"2024-03-24T01:27:14.547500+0000","last_fullsized":"2024-03-24T01:27:14.547500+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:40.646803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.528427+0000","last_change":"2024-03-24T01:27:14.528427+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.528427+0000","last_undegraded":"2024-03-24T01:27:14.528427+0000","last_fullsized":"2024-03-24T01:27:14.528427+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.558861+0000","last_change":"2024-03-24T01:27:14.552072+0000","last_active":"2024-03-24T01:27:14.558861+0000","last_peered":"2024-03-24T01:27:14.558861+0000","last_clean":"2024-03-24T01:27:14.558861+0000","last_became_active":"2024-03-24T01:27:14.550084+0000","last_became_peered":"2024-03-24T01:27:14.550084+0000","last_unstale":"2024-03-24T01:27:14.558861+0000","last_undegraded":"2024-03-24T01:27:14.558861+0000","last_fullsized":"2024-03-24T01:27:14.558861+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:45:39.507934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.550970+0000","last_change":"2024-03-24T01:27:14.544185+0000","last_active":"2024-03-24T01:27:14.550970+0000","last_peered":"2024-03-24T01:27:14.550970+0000","last_clean":"2024-03-24T01:27:14.550970+0000","last_became_active":"2024-03-24T01:27:14.543895+0000","last_became_peered":"2024-03-24T01:27:14.543895+0000","last_unstale":"2024-03-24T01:27:14.550970+0000","last_undegraded":"2024-03-24T01:27:14.550970+0000","last_fullsized":"2024-03-24T01:27:14.550970+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:47:39.461292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.550120+0000","last_change":"2024-03-24T01:27:14.550120+0000","last_active":"2024-03-24T01:27:14.550120+0000","last_peered":"2024-03-24T01:27:14.550120+0000","last_clean":"2024-03-24T01:27:14.550120+0000","last_became_active":"2024-03-24T01:27:14.549329+0000","last_became_peered":"2024-03-24T01:27:14.549329+0000","last_unstale":"2024-03-24T01:27:14.550120+0000","last_undegraded":"2024-03-24T01:27:14.550120+0000","last_fullsized":"2024-03-24T01:27:14.550120+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:32:58.788010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.557855+0000","last_change":"2024-03-24T01:27:14.557855+0000","last_active":"2024-03-24T01:27:14.557855+0000","last_peered":"2024-03-24T01:27:14.557855+0000","last_clean":"2024-03-24T01:27:14.557855+0000","last_became_active":"2024-03-24T01:27:14.557606+0000","last_became_peered":"2024-03-24T01:27:14.557606+0000","last_unstale":"2024-03-24T01:27:14.557855+0000","last_undegraded":"2024-03-24T01:27:14.557855+0000","last_fullsized":"2024-03-24T01:27:14.557855+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:28:36.027521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.550788+0000","last_change":"2024-03-24T01:27:14.542073+0000","last_active":"2024-03-24T01:27:14.550788+0000","last_peered":"2024-03-24T01:27:14.550788+0000","last_clean":"2024-03-24T01:27:14.550788+0000","last_became_active":"2024-03-24T01:27:14.540636+0000","last_became_peered":"2024-03-24T01:27:14.540636+0000","last_unstale":"2024-03-24T01:27:14.550788+0000","last_undegraded":"2024-03-24T01:27:14.550788+0000","last_fullsized":"2024-03-24T01:27:14.550788+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:02:12.837672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.556042+0000","last_change":"2024-03-24T01:27:14.556042+0000","last_active":"2024-03-24T01:27:14.556042+0000","last_peered":"2024-03-24T01:27:14.556042+0000","last_clean":"2024-03-24T01:27:14.556042+0000","last_became_active":"2024-03-24T01:27:14.555743+0000","last_became_peered":"2024-03-24T01:27:14.555743+0000","last_unstale":"2024-03-24T01:27:14.556042+0000","last_undegraded":"2024-03-24T01:27:14.556042+0000","last_fullsized":"2024-03-24T01:27:14.556042+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:14:16.916597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.561148+0000","last_change":"2024-03-24T01:27:14.561148+0000","last_active":"2024-03-24T01:27:14.561148+0000","last_peered":"2024-03-24T01:27:14.561148+0000","last_clean":"2024-03-24T01:27:14.561148+0000","last_became_active":"2024-03-24T01:27:14.560509+0000","last_became_peered":"2024-03-24T01:27:14.560509+0000","last_unstale":"2024-03-24T01:27:14.561148+0000","last_undegraded":"2024-03-24T01:27:14.561148+0000","last_fullsized":"2024-03-24T01:27:14.561148+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:09:58.411238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.553533+0000","last_change":"2024-03-24T01:27:14.553533+0000","last_active":"2024-03-24T01:27:14.553533+0000","last_peered":"2024-03-24T01:27:14.553533+0000","last_clean":"2024-03-24T01:27:14.553533+0000","last_became_active":"2024-03-24T01:27:14.553252+0000","last_became_peered":"2024-03-24T01:27:14.553252+0000","last_unstale":"2024-03-24T01:27:14.553533+0000","last_undegraded":"2024-03-24T01:27:14.553533+0000","last_fullsized":"2024-03-24T01:27:14.553533+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:46:45.759885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.546096+0000","last_change":"2024-03-24T01:27:14.541674+0000","last_active":"2024-03-24T01:27:14.546096+0000","last_peered":"2024-03-24T01:27:14.546096+0000","last_clean":"2024-03-24T01:27:14.546096+0000","last_became_active":"2024-03-24T01:27:14.541300+0000","last_became_peered":"2024-03-24T01:27:14.541300+0000","last_unstale":"2024-03-24T01:27:14.546096+0000","last_undegraded":"2024-03-24T01:27:14.546096+0000","last_fullsized":"2024-03-24T01:27:14.546096+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:12:15.274601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.527732+0000","last_change":"2024-03-24T01:27:14.527732+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.527732+0000","last_undegraded":"2024-03-24T01:27:14.527732+0000","last_fullsized":"2024-03-24T01:27:14.527732+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.558260+0000","last_change":"2024-03-24T01:27:14.554032+0000","last_active":"2024-03-24T01:27:14.558260+0000","last_peered":"2024-03-24T01:27:14.558260+0000","last_clean":"2024-03-24T01:27:14.558260+0000","last_became_active":"2024-03-24T01:27:14.553732+0000","last_became_peered":"2024-03-24T01:27:14.553732+0000","last_unstale":"2024-03-24T01:27:14.558260+0000","last_undegraded":"2024-03-24T01:27:14.558260+0000","last_fullsized":"2024-03-24T01:27:14.558260+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:04:45.650427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.551791+0000","last_change":"2024-03-24T01:27:14.551791+0000","last_active":"2024-03-24T01:27:14.551791+0000","last_peered":"2024-03-24T01:27:14.551791+0000","last_clean":"2024-03-24T01:27:14.551791+0000","last_became_active":"2024-03-24T01:27:14.551503+0000","last_became_peered":"2024-03-24T01:27:14.551503+0000","last_unstale":"2024-03-24T01:27:14.551791+0000","last_undegraded":"2024-03-24T01:27:14.551791+0000","last_fullsized":"2024-03-24T01:27:14.551791+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:18:59.684453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.556979+0000","last_change":"2024-03-24T01:27:14.545843+0000","last_active":"2024-03-24T01:27:14.556979+0000","last_peered":"2024-03-24T01:27:14.556979+0000","last_clean":"2024-03-24T01:27:14.556979+0000","last_became_active":"2024-03-24T01:27:14.545534+0000","last_became_peered":"2024-03-24T01:27:14.545534+0000","last_unstale":"2024-03-24T01:27:14.556979+0000","last_undegraded":"2024-03-24T01:27:14.556979+0000","last_fullsized":"2024-03-24T01:27:14.556979+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:06:34.262662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.548640+0000","last_change":"2024-03-24T01:27:14.548640+0000","last_active":"2024-03-24T01:27:14.548640+0000","last_peered":"2024-03-24T01:27:14.548640+0000","last_clean":"2024-03-24T01:27:14.548640+0000","last_became_active":"2024-03-24T01:27:14.548364+0000","last_became_peered":"2024-03-24T01:27:14.548364+0000","last_unstale":"2024-03-24T01:27:14.548640+0000","last_undegraded":"2024-03-24T01:27:14.548640+0000","last_fullsized":"2024-03-24T01:27:14.548640+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:28:36.027521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.544765+0000","last_change":"2024-03-24T01:27:14.541461+0000","last_active":"2024-03-24T01:27:14.544765+0000","last_peered":"2024-03-24T01:27:14.544765+0000","last_clean":"2024-03-24T01:27:14.544765+0000","last_became_active":"2024-03-24T01:27:14.541156+0000","last_became_peered":"2024-03-24T01:27:14.541156+0000","last_unstale":"2024-03-24T01:27:14.544765+0000","last_undegraded":"2024-03-24T01:27:14.544765+0000","last_fullsized":"2024-03-24T01:27:14.544765+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:14:09.820632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.556242+0000","last_change":"2024-03-24T01:27:14.556242+0000","last_active":"2024-03-24T01:27:14.556242+0000","last_peered":"2024-03-24T01:27:14.556242+0000","last_clean":"2024-03-24T01:27:14.556242+0000","last_became_active":"2024-03-24T01:27:14.552642+0000","last_became_peered":"2024-03-24T01:27:14.552642+0000","last_unstale":"2024-03-24T01:27:14.556242+0000","last_undegraded":"2024-03-24T01:27:14.556242+0000","last_fullsized":"2024-03-24T01:27:14.556242+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:28:36.027521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.546588+0000","last_change":"2024-03-24T01:27:14.542335+0000","last_active":"2024-03-24T01:27:14.546588+0000","last_peered":"2024-03-24T01:27:14.546588+0000","last_clean":"2024-03-24T01:27:14.546588+0000","last_became_active":"2024-03-24T01:27:14.541006+0000","last_became_peered":"2024-03-24T01:27:14.541006+0000","last_unstale":"2024-03-24T01:27:14.546588+0000","last_undegraded":"2024-03-24T01:27:14.546588+0000","last_fullsized":"2024-03-24T01:27:14.546588+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:44.340192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.550220+0000","last_change":"2024-03-24T01:27:14.550220+0000","last_active":"2024-03-24T01:27:14.550220+0000","last_peered":"2024-03-24T01:27:14.550220+0000","last_clean":"2024-03-24T01:27:14.550220+0000","last_became_active":"2024-03-24T01:27:14.549812+0000","last_became_peered":"2024-03-24T01:27:14.549812+0000","last_unstale":"2024-03-24T01:27:14.550220+0000","last_undegraded":"2024-03-24T01:27:14.550220+0000","last_fullsized":"2024-03-24T01:27:14.550220+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:09:58.411238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.556235+0000","last_change":"2024-03-24T01:27:14.556235+0000","last_active":"2024-03-24T01:27:14.556235+0000","last_peered":"2024-03-24T01:27:14.556235+0000","last_clean":"2024-03-24T01:27:14.556235+0000","last_became_active":"2024-03-24T01:27:14.555888+0000","last_became_peered":"2024-03-24T01:27:14.555888+0000","last_unstale":"2024-03-24T01:27:14.556235+0000","last_undegraded":"2024-03-24T01:27:14.556235+0000","last_fullsized":"2024-03-24T01:27:14.556235+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:43:45.838352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.552212+0000","last_change":"2024-03-24T01:27:14.552212+0000","last_active":"2024-03-24T01:27:14.552212+0000","last_peered":"2024-03-24T01:27:14.552212+0000","last_clean":"2024-03-24T01:27:14.552212+0000","last_became_active":"2024-03-24T01:27:14.551775+0000","last_became_peered":"2024-03-24T01:27:14.551775+0000","last_unstale":"2024-03-24T01:27:14.552212+0000","last_undegraded":"2024-03-24T01:27:14.552212+0000","last_fullsized":"2024-03-24T01:27:14.552212+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:23.971971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.550504+0000","last_change":"2024-03-24T01:27:14.542056+0000","last_active":"2024-03-24T01:27:14.550504+0000","last_peered":"2024-03-24T01:27:14.550504+0000","last_clean":"2024-03-24T01:27:14.550504+0000","last_became_active":"2024-03-24T01:27:14.541695+0000","last_became_peered":"2024-03-24T01:27:14.541695+0000","last_unstale":"2024-03-24T01:27:14.550504+0000","last_undegraded":"2024-03-24T01:27:14.550504+0000","last_fullsized":"2024-03-24T01:27:14.550504+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:32.037863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.559666+0000","last_change":"2024-03-24T01:27:14.559666+0000","last_active":"2024-03-24T01:27:14.559666+0000","last_peered":"2024-03-24T01:27:14.559666+0000","last_clean":"2024-03-24T01:27:14.559666+0000","last_became_active":"2024-03-24T01:27:14.559356+0000","last_became_peered":"2024-03-24T01:27:14.559356+0000","last_unstale":"2024-03-24T01:27:14.559666+0000","last_undegraded":"2024-03-24T01:27:14.559666+0000","last_fullsized":"2024-03-24T01:27:14.559666+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:32.037863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.529333+0000","last_change":"2024-03-24T01:27:13.522918+0000","last_active":"2024-03-24T01:27:14.529333+0000","last_peered":"2024-03-24T01:27:14.529333+0000","last_clean":"2024-03-24T01:27:14.529333+0000","last_became_active":"2024-03-24T01:27:13.520866+0000","last_became_peered":"2024-03-24T01:27:13.520866+0000","last_unstale":"2024-03-24T01:27:14.529333+0000","last_undegraded":"2024-03-24T01:27:14.529333+0000","last_fullsized":"2024-03-24T01:27:14.529333+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:30.183145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.557567+0000","last_change":"2024-03-24T01:27:14.551119+0000","last_active":"2024-03-24T01:27:14.557567+0000","last_peered":"2024-03-24T01:27:14.557567+0000","last_clean":"2024-03-24T01:27:14.557567+0000","last_became_active":"2024-03-24T01:27:14.550800+0000","last_became_peered":"2024-03-24T01:27:14.550800+0000","last_unstale":"2024-03-24T01:27:14.557567+0000","last_undegraded":"2024-03-24T01:27:14.557567+0000","last_fullsized":"2024-03-24T01:27:14.557567+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:09:58.411238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.518296+0000","last_change":"2024-03-24T01:27:13.531689+0000","last_active":"2024-03-24T01:27:14.518296+0000","last_peered":"2024-03-24T01:27:14.518296+0000","last_clean":"2024-03-24T01:27:14.518296+0000","last_became_active":"2024-03-24T01:27:13.530929+0000","last_became_peered":"2024-03-24T01:27:13.530929+0000","last_unstale":"2024-03-24T01:27:14.518296+0000","last_undegraded":"2024-03-24T01:27:14.518296+0000","last_fullsized":"2024-03-24T01:27:14.518296+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:00:24.458371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.552432+0000","last_change":"2024-03-24T01:27:14.552432+0000","last_active":"2024-03-24T01:27:14.552432+0000","last_peered":"2024-03-24T01:27:14.552432+0000","last_clean":"2024-03-24T01:27:14.552432+0000","last_became_active":"2024-03-24T01:27:14.551697+0000","last_became_peered":"2024-03-24T01:27:14.551697+0000","last_unstale":"2024-03-24T01:27:14.552432+0000","last_undegraded":"2024-03-24T01:27:14.552432+0000","last_fullsized":"2024-03-24T01:27:14.552432+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:47:39.461292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.518290+0000","last_change":"2024-03-24T01:27:13.528068+0000","last_active":"2024-03-24T01:27:14.518290+0000","last_peered":"2024-03-24T01:27:14.518290+0000","last_clean":"2024-03-24T01:27:14.518290+0000","last_became_active":"2024-03-24T01:27:13.527750+0000","last_became_peered":"2024-03-24T01:27:13.527750+0000","last_unstale":"2024-03-24T01:27:14.518290+0000","last_undegraded":"2024-03-24T01:27:14.518290+0000","last_fullsized":"2024-03-24T01:27:14.518290+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:37:15.291837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.549966+0000","last_change":"2024-03-24T01:27:14.548464+0000","last_active":"2024-03-24T01:27:14.549966+0000","last_peered":"2024-03-24T01:27:14.549966+0000","last_clean":"2024-03-24T01:27:14.549966+0000","last_became_active":"2024-03-24T01:27:14.548113+0000","last_became_peered":"2024-03-24T01:27:14.548113+0000","last_unstale":"2024-03-24T01:27:14.549966+0000","last_undegraded":"2024-03-24T01:27:14.549966+0000","last_fullsized":"2024-03-24T01:27:14.549966+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:47:57.893749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.523180+0000","last_change":"2024-03-24T01:27:13.527296+0000","last_active":"2024-03-24T01:27:14.523180+0000","last_peered":"2024-03-24T01:27:14.523180+0000","last_clean":"2024-03-24T01:27:14.523180+0000","last_became_active":"2024-03-24T01:27:13.526656+0000","last_became_peered":"2024-03-24T01:27:13.526656+0000","last_unstale":"2024-03-24T01:27:14.523180+0000","last_undegraded":"2024-03-24T01:27:14.523180+0000","last_fullsized":"2024-03-24T01:27:14.523180+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:28:35.013061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.531475+0000","last_change":"2024-03-24T01:27:14.531475+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.531475+0000","last_undegraded":"2024-03-24T01:27:14.531475+0000","last_fullsized":"2024-03-24T01:27:14.531475+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.532174+0000","last_change":"2024-03-24T01:27:13.518516+0000","last_active":"2024-03-24T01:27:14.532174+0000","last_peered":"2024-03-24T01:27:14.532174+0000","last_clean":"2024-03-24T01:27:14.532174+0000","last_became_active":"2024-03-24T01:27:13.518276+0000","last_became_peered":"2024-03-24T01:27:13.518276+0000","last_unstale":"2024-03-24T01:27:14.532174+0000","last_undegraded":"2024-03-24T01:27:14.532174+0000","last_fullsized":"2024-03-24T01:27:14.532174+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:39.632343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.559916+0000","last_change":"2024-03-24T01:27:14.559916+0000","last_active":"2024-03-24T01:27:14.559916+0000","last_peered":"2024-03-24T01:27:14.559916+0000","last_clean":"2024-03-24T01:27:14.559916+0000","last_became_active":"2024-03-24T01:27:14.559597+0000","last_became_peered":"2024-03-24T01:27:14.559597+0000","last_unstale":"2024-03-24T01:27:14.559916+0000","last_undegraded":"2024-03-24T01:27:14.559916+0000","last_fullsized":"2024-03-24T01:27:14.559916+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:32:58.788010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.535218+0000","last_change":"2024-03-24T01:27:13.527482+0000","last_active":"2024-03-24T01:27:14.535218+0000","last_peered":"2024-03-24T01:27:14.535218+0000","last_clean":"2024-03-24T01:27:14.535218+0000","last_became_active":"2024-03-24T01:27:13.527205+0000","last_became_peered":"2024-03-24T01:27:13.527205+0000","last_unstale":"2024-03-24T01:27:14.535218+0000","last_undegraded":"2024-03-24T01:27:14.535218+0000","last_fullsized":"2024-03-24T01:27:14.535218+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:39.632343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.544743+0000","last_change":"2024-03-24T01:27:14.541241+0000","last_active":"2024-03-24T01:27:14.544743+0000","last_peered":"2024-03-24T01:27:14.544743+0000","last_clean":"2024-03-24T01:27:14.544743+0000","last_became_active":"2024-03-24T01:27:14.540872+0000","last_became_peered":"2024-03-24T01:27:14.540872+0000","last_unstale":"2024-03-24T01:27:14.544743+0000","last_undegraded":"2024-03-24T01:27:14.544743+0000","last_fullsized":"2024-03-24T01:27:14.544743+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:16:39.371501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.527060+0000","last_change":"2024-03-24T01:27:13.531379+0000","last_active":"2024-03-24T01:27:14.527060+0000","last_peered":"2024-03-24T01:27:14.527060+0000","last_clean":"2024-03-24T01:27:14.527060+0000","last_became_active":"2024-03-24T01:27:13.530618+0000","last_became_peered":"2024-03-24T01:27:13.530618+0000","last_unstale":"2024-03-24T01:27:14.527060+0000","last_undegraded":"2024-03-24T01:27:14.527060+0000","last_fullsized":"2024-03-24T01:27:14.527060+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:00.227179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.548287+0000","last_change":"2024-03-24T01:27:14.548287+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.548287+0000","last_undegraded":"2024-03-24T01:27:14.548287+0000","last_fullsized":"2024-03-24T01:27:14.548287+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.527604+0000","last_change":"2024-03-24T01:27:13.529478+0000","last_active":"2024-03-24T01:27:14.527604+0000","last_peered":"2024-03-24T01:27:14.527604+0000","last_clean":"2024-03-24T01:27:14.527604+0000","last_became_active":"2024-03-24T01:27:13.529197+0000","last_became_peered":"2024-03-24T01:27:13.529197+0000","last_unstale":"2024-03-24T01:27:14.527604+0000","last_undegraded":"2024-03-24T01:27:14.527604+0000","last_fullsized":"2024-03-24T01:27:14.527604+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:30:49.041062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.547194+0000","last_change":"2024-03-24T01:27:14.544605+0000","last_active":"2024-03-24T01:27:14.547194+0000","last_peered":"2024-03-24T01:27:14.547194+0000","last_clean":"2024-03-24T01:27:14.547194+0000","last_became_active":"2024-03-24T01:27:14.544189+0000","last_became_peered":"2024-03-24T01:27:14.544189+0000","last_unstale":"2024-03-24T01:27:14.547194+0000","last_undegraded":"2024-03-24T01:27:14.547194+0000","last_fullsized":"2024-03-24T01:27:14.547194+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:12:15.274601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.548563+0000","last_change":"2024-03-24T01:27:13.533618+0000","last_active":"2024-03-24T01:27:14.548563+0000","last_peered":"2024-03-24T01:27:14.548563+0000","last_clean":"2024-03-24T01:27:14.548563+0000","last_became_active":"2024-03-24T01:27:13.531829+0000","last_became_peered":"2024-03-24T01:27:13.531829+0000","last_unstale":"2024-03-24T01:27:14.548563+0000","last_undegraded":"2024-03-24T01:27:14.548563+0000","last_fullsized":"2024-03-24T01:27:14.548563+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:06:33.248202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.552880+0000","last_change":"2024-03-24T01:27:14.541924+0000","last_active":"2024-03-24T01:27:14.552880+0000","last_peered":"2024-03-24T01:27:14.552880+0000","last_clean":"2024-03-24T01:27:14.552880+0000","last_became_active":"2024-03-24T01:27:14.541512+0000","last_became_peered":"2024-03-24T01:27:14.541512+0000","last_unstale":"2024-03-24T01:27:14.552880+0000","last_undegraded":"2024-03-24T01:27:14.552880+0000","last_fullsized":"2024-03-24T01:27:14.552880+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:03.448771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.536520+0000","last_change":"2024-03-24T01:27:13.524033+0000","last_active":"2024-03-24T01:27:14.536520+0000","last_peered":"2024-03-24T01:27:14.536520+0000","last_clean":"2024-03-24T01:27:14.536520+0000","last_became_active":"2024-03-24T01:27:13.523502+0000","last_became_peered":"2024-03-24T01:27:13.523502+0000","last_unstale":"2024-03-24T01:27:14.536520+0000","last_undegraded":"2024-03-24T01:27:14.536520+0000","last_fullsized":"2024-03-24T01:27:14.536520+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:02:11.823211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.552947+0000","last_change":"2024-03-24T01:27:14.552947+0000","last_active":"2024-03-24T01:27:14.552947+0000","last_peered":"2024-03-24T01:27:14.552947+0000","last_clean":"2024-03-24T01:27:14.552947+0000","last_became_active":"2024-03-24T01:27:14.552583+0000","last_became_peered":"2024-03-24T01:27:14.552583+0000","last_unstale":"2024-03-24T01:27:14.552947+0000","last_undegraded":"2024-03-24T01:27:14.552947+0000","last_fullsized":"2024-03-24T01:27:14.552947+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:58.630025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"52'2","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.548205+0000","last_change":"2024-03-24T01:27:13.530329+0000","last_active":"2024-03-24T01:27:14.548205+0000","last_peered":"2024-03-24T01:27:14.548205+0000","last_clean":"2024-03-24T01:27:14.548205+0000","last_became_active":"2024-03-24T01:27:13.529972+0000","last_became_peered":"2024-03-24T01:27:13.529972+0000","last_unstale":"2024-03-24T01:27:14.548205+0000","last_undegraded":"2024-03-24T01:27:14.548205+0000","last_fullsized":"2024-03-24T01:27:14.548205+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:43.325731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.552486+0000","last_change":"2024-03-24T01:27:14.552486+0000","last_active":"2024-03-24T01:27:14.552486+0000","last_peered":"2024-03-24T01:27:14.552486+0000","last_clean":"2024-03-24T01:27:14.552486+0000","last_became_active":"2024-03-24T01:27:14.551883+0000","last_became_peered":"2024-03-24T01:27:14.551883+0000","last_unstale":"2024-03-24T01:27:14.552486+0000","last_undegraded":"2024-03-24T01:27:14.552486+0000","last_fullsized":"2024-03-24T01:27:14.552486+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:44.340192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"52'5","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.535485+0000","last_change":"2024-03-24T01:27:13.521128+0000","last_active":"2024-03-24T01:27:14.535485+0000","last_peered":"2024-03-24T01:27:14.535485+0000","last_clean":"2024-03-24T01:27:14.535485+0000","last_became_active":"2024-03-24T01:27:13.520358+0000","last_became_peered":"2024-03-24T01:27:13.520358+0000","last_unstale":"2024-03-24T01:27:14.535485+0000","last_undegraded":"2024-03-24T01:27:14.535485+0000","last_fullsized":"2024-03-24T01:27:14.535485+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:01:08.204732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.522002+0000","last_change":"2024-03-24T01:27:14.522002+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.522002+0000","last_undegraded":"2024-03-24T01:27:14.522002+0000","last_fullsized":"2024-03-24T01:27:14.522002+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.532355+0000","last_change":"2024-03-24T01:27:13.532029+0000","last_active":"2024-03-24T01:27:14.532355+0000","last_peered":"2024-03-24T01:27:14.532355+0000","last_clean":"2024-03-24T01:27:14.532355+0000","last_became_active":"2024-03-24T01:27:13.531765+0000","last_became_peered":"2024-03-24T01:27:13.531765+0000","last_unstale":"2024-03-24T01:27:14.532355+0000","last_undegraded":"2024-03-24T01:27:14.532355+0000","last_fullsized":"2024-03-24T01:27:14.532355+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:02:11.823211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.537552+0000","last_change":"2024-03-24T01:27:14.537552+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.537552+0000","last_undegraded":"2024-03-24T01:27:14.537552+0000","last_fullsized":"2024-03-24T01:27:14.537552+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.527543+0000","last_change":"2024-03-24T01:27:13.530136+0000","last_active":"2024-03-24T01:27:14.527543+0000","last_peered":"2024-03-24T01:27:14.527543+0000","last_clean":"2024-03-24T01:27:14.527543+0000","last_became_active":"2024-03-24T01:27:13.529811+0000","last_became_peered":"2024-03-24T01:27:13.529811+0000","last_unstale":"2024-03-24T01:27:14.527543+0000","last_undegraded":"2024-03-24T01:27:14.527543+0000","last_fullsized":"2024-03-24T01:27:14.527543+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:37:15.291837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.563469+0000","last_change":"2024-03-24T01:27:14.561422+0000","last_active":"2024-03-24T01:27:14.563469+0000","last_peered":"2024-03-24T01:27:14.563469+0000","last_clean":"2024-03-24T01:27:14.563469+0000","last_became_active":"2024-03-24T01:27:14.559580+0000","last_became_peered":"2024-03-24T01:27:14.559580+0000","last_unstale":"2024-03-24T01:27:14.563469+0000","last_undegraded":"2024-03-24T01:27:14.563469+0000","last_fullsized":"2024-03-24T01:27:14.563469+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:18:51.495804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.546920+0000","last_change":"2024-03-24T01:27:13.532579+0000","last_active":"2024-03-24T01:27:14.546920+0000","last_peered":"2024-03-24T01:27:14.546920+0000","last_clean":"2024-03-24T01:27:14.546920+0000","last_became_active":"2024-03-24T01:27:13.532264+0000","last_became_peered":"2024-03-24T01:27:13.532264+0000","last_unstale":"2024-03-24T01:27:14.546920+0000","last_undegraded":"2024-03-24T01:27:14.546920+0000","last_fullsized":"2024-03-24T01:27:14.546920+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:12:14.260140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.545505+0000","last_change":"2024-03-24T01:27:14.545505+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.545505+0000","last_undegraded":"2024-03-24T01:27:14.545505+0000","last_fullsized":"2024-03-24T01:27:14.545505+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"52'8","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.536546+0000","last_change":"2024-03-24T01:27:13.518331+0000","last_active":"2024-03-24T01:27:14.536546+0000","last_peered":"2024-03-24T01:27:14.536546+0000","last_clean":"2024-03-24T01:27:14.536546+0000","last_became_active":"2024-03-24T01:27:13.518017+0000","last_became_peered":"2024-03-24T01:27:13.518017+0000","last_unstale":"2024-03-24T01:27:14.536546+0000","last_undegraded":"2024-03-24T01:27:14.536546+0000","last_fullsized":"2024-03-24T01:27:14.536546+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:02:41.582283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.547545+0000","last_change":"2024-03-24T01:27:14.543017+0000","last_active":"2024-03-24T01:27:14.547545+0000","last_peered":"2024-03-24T01:27:14.547545+0000","last_clean":"2024-03-24T01:27:14.547545+0000","last_became_active":"2024-03-24T01:27:14.542730+0000","last_became_peered":"2024-03-24T01:27:14.542730+0000","last_unstale":"2024-03-24T01:27:14.547545+0000","last_undegraded":"2024-03-24T01:27:14.547545+0000","last_fullsized":"2024-03-24T01:27:14.547545+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:19:18.467994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.545934+0000","last_change":"2024-03-24T01:27:13.530251+0000","last_active":"2024-03-24T01:27:14.545934+0000","last_peered":"2024-03-24T01:27:14.545934+0000","last_clean":"2024-03-24T01:27:14.545934+0000","last_became_active":"2024-03-24T01:27:13.529767+0000","last_became_peered":"2024-03-24T01:27:13.529767+0000","last_unstale":"2024-03-24T01:27:14.545934+0000","last_undegraded":"2024-03-24T01:27:14.545934+0000","last_fullsized":"2024-03-24T01:27:14.545934+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:43:44.823891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.557227+0000","last_change":"2024-03-24T01:27:14.554474+0000","last_active":"2024-03-24T01:27:14.557227+0000","last_peered":"2024-03-24T01:27:14.557227+0000","last_clean":"2024-03-24T01:27:14.557227+0000","last_became_active":"2024-03-24T01:27:14.554189+0000","last_became_peered":"2024-03-24T01:27:14.554189+0000","last_unstale":"2024-03-24T01:27:14.557227+0000","last_undegraded":"2024-03-24T01:27:14.557227+0000","last_fullsized":"2024-03-24T01:27:14.557227+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:43:45.838352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.529435+0000","last_change":"2024-03-24T01:27:13.526531+0000","last_active":"2024-03-24T01:27:14.529435+0000","last_peered":"2024-03-24T01:27:14.529435+0000","last_clean":"2024-03-24T01:27:14.529435+0000","last_became_active":"2024-03-24T01:27:13.526212+0000","last_became_peered":"2024-03-24T01:27:13.526212+0000","last_unstale":"2024-03-24T01:27:14.529435+0000","last_undegraded":"2024-03-24T01:27:14.529435+0000","last_fullsized":"2024-03-24T01:27:14.529435+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:02:41.656569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.531995+0000","last_change":"2024-03-24T01:27:14.531995+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.531995+0000","last_undegraded":"2024-03-24T01:27:14.531995+0000","last_fullsized":"2024-03-24T01:27:14.531995+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.529356+0000","last_change":"2024-03-24T01:27:13.526615+0000","last_active":"2024-03-24T01:27:14.529356+0000","last_peered":"2024-03-24T01:27:14.529356+0000","last_clean":"2024-03-24T01:27:14.529356+0000","last_became_active":"2024-03-24T01:27:13.526286+0000","last_became_peered":"2024-03-24T01:27:13.526286+0000","last_unstale":"2024-03-24T01:27:14.529356+0000","last_undegraded":"2024-03-24T01:27:14.529356+0000","last_fullsized":"2024-03-24T01:27:14.529356+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:42:00.908936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.547136+0000","last_change":"2024-03-24T01:27:14.539239+0000","last_active":"2024-03-24T01:27:14.547136+0000","last_peered":"2024-03-24T01:27:14.547136+0000","last_clean":"2024-03-24T01:27:14.547136+0000","last_became_active":"2024-03-24T01:27:14.538525+0000","last_became_peered":"2024-03-24T01:27:14.538525+0000","last_unstale":"2024-03-24T01:27:14.547136+0000","last_undegraded":"2024-03-24T01:27:14.547136+0000","last_fullsized":"2024-03-24T01:27:14.547136+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:42:01.923397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.528640+0000","last_change":"2024-03-24T01:27:13.523480+0000","last_active":"2024-03-24T01:27:14.528640+0000","last_peered":"2024-03-24T01:27:14.528640+0000","last_clean":"2024-03-24T01:27:14.528640+0000","last_became_active":"2024-03-24T01:27:13.523203+0000","last_became_peered":"2024-03-24T01:27:13.523203+0000","last_unstale":"2024-03-24T01:27:14.528640+0000","last_undegraded":"2024-03-24T01:27:14.528640+0000","last_fullsized":"2024-03-24T01:27:14.528640+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:00:24.458371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.557519+0000","last_change":"2024-03-24T01:27:14.549953+0000","last_active":"2024-03-24T01:27:14.557519+0000","last_peered":"2024-03-24T01:27:14.557519+0000","last_clean":"2024-03-24T01:27:14.557519+0000","last_became_active":"2024-03-24T01:27:14.549616+0000","last_became_peered":"2024-03-24T01:27:14.549616+0000","last_unstale":"2024-03-24T01:27:14.557519+0000","last_undegraded":"2024-03-24T01:27:14.557519+0000","last_fullsized":"2024-03-24T01:27:14.557519+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:32:58.788010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.518549+0000","last_change":"2024-03-24T01:27:13.526041+0000","last_active":"2024-03-24T01:27:14.518549+0000","last_peered":"2024-03-24T01:27:14.518549+0000","last_clean":"2024-03-24T01:27:14.518549+0000","last_became_active":"2024-03-24T01:27:13.525587+0000","last_became_peered":"2024-03-24T01:27:13.525587+0000","last_unstale":"2024-03-24T01:27:14.518549+0000","last_undegraded":"2024-03-24T01:27:14.518549+0000","last_fullsized":"2024-03-24T01:27:14.518549+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:30.183145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.546589+0000","last_change":"2024-03-24T01:27:14.539787+0000","last_active":"2024-03-24T01:27:14.546589+0000","last_peered":"2024-03-24T01:27:14.546589+0000","last_clean":"2024-03-24T01:27:14.546589+0000","last_became_active":"2024-03-24T01:27:14.539446+0000","last_became_peered":"2024-03-24T01:27:14.539446+0000","last_unstale":"2024-03-24T01:27:14.546589+0000","last_undegraded":"2024-03-24T01:27:14.546589+0000","last_fullsized":"2024-03-24T01:27:14.546589+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:02:12.837672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.535722+0000","last_change":"2024-03-24T01:27:13.519780+0000","last_active":"2024-03-24T01:27:14.535722+0000","last_peered":"2024-03-24T01:27:14.535722+0000","last_clean":"2024-03-24T01:27:14.535722+0000","last_became_active":"2024-03-24T01:27:13.519458+0000","last_became_peered":"2024-03-24T01:27:13.519458+0000","last_unstale":"2024-03-24T01:27:14.535722+0000","last_undegraded":"2024-03-24T01:27:14.535722+0000","last_fullsized":"2024-03-24T01:27:14.535722+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:46:44.745424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.549326+0000","last_change":"2024-03-24T01:27:14.549326+0000","last_active":"2024-03-24T01:27:14.549326+0000","last_peered":"2024-03-24T01:27:14.549326+0000","last_clean":"2024-03-24T01:27:14.549326+0000","last_became_active":"2024-03-24T01:27:14.548886+0000","last_became_peered":"2024-03-24T01:27:14.548886+0000","last_unstale":"2024-03-24T01:27:14.549326+0000","last_undegraded":"2024-03-24T01:27:14.549326+0000","last_fullsized":"2024-03-24T01:27:14.549326+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:03.448771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.545176+0000","last_change":"2024-03-24T01:27:13.521689+0000","last_active":"2024-03-24T01:27:14.545176+0000","last_peered":"2024-03-24T01:27:14.545176+0000","last_clean":"2024-03-24T01:27:14.545176+0000","last_became_active":"2024-03-24T01:27:13.521436+0000","last_became_peered":"2024-03-24T01:27:13.521436+0000","last_unstale":"2024-03-24T01:27:14.545176+0000","last_undegraded":"2024-03-24T01:27:14.545176+0000","last_fullsized":"2024-03-24T01:27:14.545176+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:47:38.446831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.551289+0000","last_change":"2024-03-24T01:27:14.543585+0000","last_active":"2024-03-24T01:27:14.551289+0000","last_peered":"2024-03-24T01:27:14.551289+0000","last_clean":"2024-03-24T01:27:14.551289+0000","last_became_active":"2024-03-24T01:27:14.543301+0000","last_became_peered":"2024-03-24T01:27:14.543301+0000","last_unstale":"2024-03-24T01:27:14.551289+0000","last_undegraded":"2024-03-24T01:27:14.551289+0000","last_fullsized":"2024-03-24T01:27:14.551289+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:12:15.274601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.527295+0000","last_change":"2024-03-24T01:27:13.522613+0000","last_active":"2024-03-24T01:27:14.527295+0000","last_peered":"2024-03-24T01:27:14.527295+0000","last_clean":"2024-03-24T01:27:14.527295+0000","last_became_active":"2024-03-24T01:27:13.522331+0000","last_became_peered":"2024-03-24T01:27:13.522331+0000","last_unstale":"2024-03-24T01:27:14.527295+0000","last_undegraded":"2024-03-24T01:27:14.527295+0000","last_fullsized":"2024-03-24T01:27:14.527295+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:08:09.036710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.558092+0000","last_change":"2024-03-24T01:27:14.550330+0000","last_active":"2024-03-24T01:27:14.558092+0000","last_peered":"2024-03-24T01:27:14.558092+0000","last_clean":"2024-03-24T01:27:14.558092+0000","last_became_active":"2024-03-24T01:27:14.550056+0000","last_became_peered":"2024-03-24T01:27:14.550056+0000","last_unstale":"2024-03-24T01:27:14.558092+0000","last_undegraded":"2024-03-24T01:27:14.558092+0000","last_fullsized":"2024-03-24T01:27:14.558092+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:23.971971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.529686+0000","last_change":"2024-03-24T01:27:13.533418+0000","last_active":"2024-03-24T01:27:14.529686+0000","last_peered":"2024-03-24T01:27:14.529686+0000","last_clean":"2024-03-24T01:27:14.529686+0000","last_became_active":"2024-03-24T01:27:13.533164+0000","last_became_peered":"2024-03-24T01:27:13.533164+0000","last_unstale":"2024-03-24T01:27:14.529686+0000","last_undegraded":"2024-03-24T01:27:14.529686+0000","last_fullsized":"2024-03-24T01:27:14.529686+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:30:49.041062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.551209+0000","last_change":"2024-03-24T01:27:14.543832+0000","last_active":"2024-03-24T01:27:14.551209+0000","last_peered":"2024-03-24T01:27:14.551209+0000","last_clean":"2024-03-24T01:27:14.551209+0000","last_became_active":"2024-03-24T01:27:14.543520+0000","last_became_peered":"2024-03-24T01:27:14.543520+0000","last_unstale":"2024-03-24T01:27:14.551209+0000","last_undegraded":"2024-03-24T01:27:14.551209+0000","last_fullsized":"2024-03-24T01:27:14.551209+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:44.340192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.548373+0000","last_change":"2024-03-24T01:27:13.530300+0000","last_active":"2024-03-24T01:27:14.548373+0000","last_peered":"2024-03-24T01:27:14.548373+0000","last_clean":"2024-03-24T01:27:14.548373+0000","last_became_active":"2024-03-24T01:27:13.529919+0000","last_became_peered":"2024-03-24T01:27:13.529919+0000","last_unstale":"2024-03-24T01:27:14.548373+0000","last_undegraded":"2024-03-24T01:27:14.548373+0000","last_fullsized":"2024-03-24T01:27:14.548373+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:47:38.446831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.548304+0000","last_change":"2024-03-24T01:27:14.544735+0000","last_active":"2024-03-24T01:27:14.548304+0000","last_peered":"2024-03-24T01:27:14.548304+0000","last_clean":"2024-03-24T01:27:14.548304+0000","last_became_active":"2024-03-24T01:27:14.544460+0000","last_became_peered":"2024-03-24T01:27:14.544460+0000","last_unstale":"2024-03-24T01:27:14.548304+0000","last_undegraded":"2024-03-24T01:27:14.548304+0000","last_fullsized":"2024-03-24T01:27:14.548304+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:06.757918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.536983+0000","last_change":"2024-03-24T01:27:13.525558+0000","last_active":"2024-03-24T01:27:14.536983+0000","last_peered":"2024-03-24T01:27:14.536983+0000","last_clean":"2024-03-24T01:27:14.536983+0000","last_became_active":"2024-03-24T01:27:13.525223+0000","last_became_peered":"2024-03-24T01:27:13.525223+0000","last_unstale":"2024-03-24T01:27:14.536983+0000","last_undegraded":"2024-03-24T01:27:14.536983+0000","last_fullsized":"2024-03-24T01:27:14.536983+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:43.325731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.549747+0000","last_change":"2024-03-24T01:27:14.549747+0000","last_active":"2024-03-24T01:27:14.549747+0000","last_peered":"2024-03-24T01:27:14.549747+0000","last_clean":"2024-03-24T01:27:14.549747+0000","last_became_active":"2024-03-24T01:27:14.548344+0000","last_became_peered":"2024-03-24T01:27:14.548344+0000","last_unstale":"2024-03-24T01:27:14.549747+0000","last_undegraded":"2024-03-24T01:27:14.549747+0000","last_fullsized":"2024-03-24T01:27:14.549747+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:32.037863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.535645+0000","last_change":"2024-03-24T01:27:13.519855+0000","last_active":"2024-03-24T01:27:14.535645+0000","last_peered":"2024-03-24T01:27:14.535645+0000","last_clean":"2024-03-24T01:27:14.535645+0000","last_became_active":"2024-03-24T01:27:13.519566+0000","last_became_peered":"2024-03-24T01:27:13.519566+0000","last_unstale":"2024-03-24T01:27:14.535645+0000","last_undegraded":"2024-03-24T01:27:14.535645+0000","last_fullsized":"2024-03-24T01:27:14.535645+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:57.615564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.522676+0000","last_change":"2024-03-24T01:27:14.522676+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.522676+0000","last_undegraded":"2024-03-24T01:27:14.522676+0000","last_fullsized":"2024-03-24T01:27:14.522676+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.535933+0000","last_change":"2024-03-24T01:27:13.524780+0000","last_active":"2024-03-24T01:27:14.535933+0000","last_peered":"2024-03-24T01:27:14.535933+0000","last_clean":"2024-03-24T01:27:14.535933+0000","last_became_active":"2024-03-24T01:27:13.524518+0000","last_became_peered":"2024-03-24T01:27:13.524518+0000","last_unstale":"2024-03-24T01:27:14.535933+0000","last_undegraded":"2024-03-24T01:27:14.535933+0000","last_fullsized":"2024-03-24T01:27:14.535933+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:14:15.902136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.534874+0000","last_change":"2024-03-24T01:27:14.534874+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.534874+0000","last_undegraded":"2024-03-24T01:27:14.534874+0000","last_fullsized":"2024-03-24T01:27:14.534874+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.534655+0000","last_change":"2024-03-24T01:27:13.519549+0000","last_active":"2024-03-24T01:27:14.534655+0000","last_peered":"2024-03-24T01:27:14.534655+0000","last_clean":"2024-03-24T01:27:14.534655+0000","last_became_active":"2024-03-24T01:27:13.519174+0000","last_became_peered":"2024-03-24T01:27:13.519174+0000","last_unstale":"2024-03-24T01:27:14.534655+0000","last_undegraded":"2024-03-24T01:27:14.534655+0000","last_fullsized":"2024-03-24T01:27:14.534655+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:30:49.041062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.549972+0000","last_change":"2024-03-24T01:27:14.548520+0000","last_active":"2024-03-24T01:27:14.549972+0000","last_peered":"2024-03-24T01:27:14.549972+0000","last_clean":"2024-03-24T01:27:14.549972+0000","last_became_active":"2024-03-24T01:27:14.547968+0000","last_became_peered":"2024-03-24T01:27:14.547968+0000","last_unstale":"2024-03-24T01:27:14.549972+0000","last_undegraded":"2024-03-24T01:27:14.549972+0000","last_fullsized":"2024-03-24T01:27:14.549972+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:36.294802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.520961+0000","last_change":"2024-03-24T01:27:13.524804+0000","last_active":"2024-03-24T01:27:14.520961+0000","last_peered":"2024-03-24T01:27:14.520961+0000","last_clean":"2024-03-24T01:27:14.520961+0000","last_became_active":"2024-03-24T01:27:13.524487+0000","last_became_peered":"2024-03-24T01:27:13.524487+0000","last_unstale":"2024-03-24T01:27:14.520961+0000","last_undegraded":"2024-03-24T01:27:14.520961+0000","last_fullsized":"2024-03-24T01:27:14.520961+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:43:44.823891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.546611+0000","last_change":"2024-03-24T01:27:14.539180+0000","last_active":"2024-03-24T01:27:14.546611+0000","last_peered":"2024-03-24T01:27:14.546611+0000","last_clean":"2024-03-24T01:27:14.546611+0000","last_became_active":"2024-03-24T01:27:14.538800+0000","last_became_peered":"2024-03-24T01:27:14.538800+0000","last_unstale":"2024-03-24T01:27:14.546611+0000","last_undegraded":"2024-03-24T01:27:14.546611+0000","last_fullsized":"2024-03-24T01:27:14.546611+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:40.646803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.535286+0000","last_change":"2024-03-24T01:27:13.532470+0000","last_active":"2024-03-24T01:27:14.535286+0000","last_peered":"2024-03-24T01:27:14.535286+0000","last_clean":"2024-03-24T01:27:14.535286+0000","last_became_active":"2024-03-24T01:27:13.530705+0000","last_became_peered":"2024-03-24T01:27:13.530705+0000","last_unstale":"2024-03-24T01:27:14.535286+0000","last_undegraded":"2024-03-24T01:27:14.535286+0000","last_fullsized":"2024-03-24T01:27:14.535286+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:12:14.260140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":229,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.545758+0000","last_change":"2024-03-24T01:24:01.545855+0000","last_active":"2024-03-24T01:27:14.545758+0000","last_peered":"2024-03-24T01:27:14.545758+0000","last_clean":"2024-03-24T01:27:14.545758+0000","last_became_active":"2024-03-24T01:24:01.543811+0000","last_became_peered":"2024-03-24T01:24:01.543811+0000","last_unstale":"2024-03-24T01:27:14.545758+0000","last_undegraded":"2024-03-24T01:27:14.545758+0000","last_fullsized":"2024-03-24T01:27:14.545758+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_clean_scrub_stamp":"2024-03-24T01:23:59.861564+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:03.682631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":672,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15,"ondisk_log_size":15,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574890,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28196,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343644,"statfs":{"total":96636764160,"available":96607891456,"internally_reserved":0,"allocated":1200128,"data_stored":861179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":270811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":270811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574890,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":270811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":270811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574891,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":270811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":618496,"data_stored":277978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574891,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28196,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343644,"statfs":{"total":96636764160,"available":96607891456,"internally_reserved":0,"allocated":1200128,"data_stored":861179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T01:27:19.977 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3d is not active+clean 2024-03-24T01:27:19.977 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3d', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 52, 'state': 'creating+activating', 'last_fresh': '2024-03-24T01:27:14.538645+0000', 'last_change': '2024-03-24T01:27:14.538645+0000', 'last_active': '2024-03-24T01:27:13.484941+0000', 'last_peered': '2024-03-24T01:27:13.484941+0000', 'last_clean': '2024-03-24T01:27:13.484941+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:27:14.538645+0000', 'last_undegraded': '2024-03-24T01:27:14.538645+0000', 'last_fullsized': '2024-03-24T01:27:14.538645+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'last_clean_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': ['0', '2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T01:27:19.978 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3b is not active+clean 2024-03-24T01:27:19.978 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3b', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 52, 'state': 'creating+activating', 'last_fresh': '2024-03-24T01:27:14.531210+0000', 'last_change': '2024-03-24T01:27:14.531210+0000', 'last_active': '2024-03-24T01:27:13.484941+0000', 'last_peered': '2024-03-24T01:27:13.484941+0000', 'last_clean': '2024-03-24T01:27:13.484941+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:27:14.531210+0000', 'last_undegraded': '2024-03-24T01:27:14.531210+0000', 'last_fullsized': '2024-03-24T01:27:14.531210+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'last_clean_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': ['4', '7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T01:27:19.978 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.37 is not active+clean 2024-03-24T01:27:19.978 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.37', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 52, 'state': 'creating+activating', 'last_fresh': '2024-03-24T01:27:14.539698+0000', 'last_change': '2024-03-24T01:27:14.539698+0000', 'last_active': '2024-03-24T01:27:13.484941+0000', 'last_peered': '2024-03-24T01:27:13.484941+0000', 'last_clean': '2024-03-24T01:27:13.484941+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:27:14.539698+0000', 'last_undegraded': '2024-03-24T01:27:14.539698+0000', 'last_fullsized': '2024-03-24T01:27:14.539698+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'last_clean_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': ['3', '0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T01:27:19.978 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.35 is not active+clean 2024-03-24T01:27:19.978 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.35', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 52, 'state': 'creating+activating', 'last_fresh': '2024-03-24T01:27:14.528427+0000', 'last_change': '2024-03-24T01:27:14.528427+0000', 'last_active': '2024-03-24T01:27:13.484941+0000', 'last_peered': '2024-03-24T01:27:13.484941+0000', 'last_clean': '2024-03-24T01:27:13.484941+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:27:14.528427+0000', 'last_undegraded': '2024-03-24T01:27:14.528427+0000', 'last_fullsized': '2024-03-24T01:27:14.528427+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'last_clean_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': ['2', '7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-24T01:27:19.978 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2b is not active+clean 2024-03-24T01:27:19.978 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2b', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 52, 'state': 'creating+activating', 'last_fresh': '2024-03-24T01:27:14.527732+0000', 'last_change': '2024-03-24T01:27:14.527732+0000', 'last_active': '2024-03-24T01:27:13.484941+0000', 'last_peered': '2024-03-24T01:27:13.484941+0000', 'last_clean': '2024-03-24T01:27:13.484941+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:27:14.527732+0000', 'last_undegraded': '2024-03-24T01:27:14.527732+0000', 'last_fullsized': '2024-03-24T01:27:14.527732+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'last_clean_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': ['6', '7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T01:27:19.978 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.14 is not active+clean 2024-03-24T01:27:19.978 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.14', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 52, 'state': 'creating+activating', 'last_fresh': '2024-03-24T01:27:14.531475+0000', 'last_change': '2024-03-24T01:27:14.531475+0000', 'last_active': '2024-03-24T01:27:13.484941+0000', 'last_peered': '2024-03-24T01:27:13.484941+0000', 'last_clean': '2024-03-24T01:27:13.484941+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:27:14.531475+0000', 'last_undegraded': '2024-03-24T01:27:14.531475+0000', 'last_fullsized': '2024-03-24T01:27:14.531475+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'last_clean_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': ['4', '0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T01:27:19.978 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.17 is not active+clean 2024-03-24T01:27:19.979 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.17', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 52, 'state': 'creating+activating', 'last_fresh': '2024-03-24T01:27:14.548287+0000', 'last_change': '2024-03-24T01:27:14.548287+0000', 'last_active': '2024-03-24T01:27:13.484941+0000', 'last_peered': '2024-03-24T01:27:13.484941+0000', 'last_clean': '2024-03-24T01:27:13.484941+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:27:14.548287+0000', 'last_undegraded': '2024-03-24T01:27:14.548287+0000', 'last_fullsized': '2024-03-24T01:27:14.548287+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'last_clean_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': ['7', '0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T01:27:19.979 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1c is not active+clean 2024-03-24T01:27:19.979 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1c', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 52, 'state': 'creating+activating', 'last_fresh': '2024-03-24T01:27:14.522002+0000', 'last_change': '2024-03-24T01:27:14.522002+0000', 'last_active': '2024-03-24T01:27:13.484941+0000', 'last_peered': '2024-03-24T01:27:13.484941+0000', 'last_clean': '2024-03-24T01:27:13.484941+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:27:14.522002+0000', 'last_undegraded': '2024-03-24T01:27:14.522002+0000', 'last_fullsized': '2024-03-24T01:27:14.522002+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'last_clean_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': ['5', '7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T01:27:19.979 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1d is not active+clean 2024-03-24T01:27:19.979 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1d', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 52, 'state': 'creating+activating', 'last_fresh': '2024-03-24T01:27:14.537552+0000', 'last_change': '2024-03-24T01:27:14.537552+0000', 'last_active': '2024-03-24T01:27:13.484941+0000', 'last_peered': '2024-03-24T01:27:13.484941+0000', 'last_clean': '2024-03-24T01:27:13.484941+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:27:14.537552+0000', 'last_undegraded': '2024-03-24T01:27:14.537552+0000', 'last_fullsized': '2024-03-24T01:27:14.537552+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'last_clean_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': ['0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T01:27:19.979 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1f is not active+clean 2024-03-24T01:27:19.979 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1f', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 52, 'state': 'creating+activating', 'last_fresh': '2024-03-24T01:27:14.545505+0000', 'last_change': '2024-03-24T01:27:14.545505+0000', 'last_active': '2024-03-24T01:27:13.484941+0000', 'last_peered': '2024-03-24T01:27:13.484941+0000', 'last_clean': '2024-03-24T01:27:13.484941+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:27:14.545505+0000', 'last_undegraded': '2024-03-24T01:27:14.545505+0000', 'last_fullsized': '2024-03-24T01:27:14.545505+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'last_clean_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': ['0', '5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T01:27:19.979 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2 is not active+clean 2024-03-24T01:27:19.979 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 52, 'state': 'creating+activating', 'last_fresh': '2024-03-24T01:27:14.531995+0000', 'last_change': '2024-03-24T01:27:14.531995+0000', 'last_active': '2024-03-24T01:27:13.484941+0000', 'last_peered': '2024-03-24T01:27:13.484941+0000', 'last_clean': '2024-03-24T01:27:13.484941+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:27:14.531995+0000', 'last_undegraded': '2024-03-24T01:27:14.531995+0000', 'last_fullsized': '2024-03-24T01:27:14.531995+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'last_clean_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': ['4', '0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T01:27:19.979 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.d is not active+clean 2024-03-24T01:27:19.979 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.d', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 52, 'state': 'creating+activating', 'last_fresh': '2024-03-24T01:27:14.522676+0000', 'last_change': '2024-03-24T01:27:14.522676+0000', 'last_active': '2024-03-24T01:27:13.484941+0000', 'last_peered': '2024-03-24T01:27:13.484941+0000', 'last_clean': '2024-03-24T01:27:13.484941+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:27:14.522676+0000', 'last_undegraded': '2024-03-24T01:27:14.522676+0000', 'last_fullsized': '2024-03-24T01:27:14.522676+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'last_clean_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': ['5', '7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T01:27:19.980 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.e is not active+clean 2024-03-24T01:27:19.980 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.e', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 52, 'state': 'creating+activating', 'last_fresh': '2024-03-24T01:27:14.534874+0000', 'last_change': '2024-03-24T01:27:14.534874+0000', 'last_active': '2024-03-24T01:27:13.484941+0000', 'last_peered': '2024-03-24T01:27:13.484941+0000', 'last_clean': '2024-03-24T01:27:13.484941+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T01:27:14.534874+0000', 'last_undegraded': '2024-03-24T01:27:14.534874+0000', 'last_fullsized': '2024-03-24T01:27:14.534874+0000', 'mapping_epoch': 51, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 51, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'last_clean_scrub_stamp': '2024-03-24T01:27:13.484941+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': ['0', '2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T01:27:19.980 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-24T01:27:20.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.227+0000 7feba3400640 1 Processor -- start 2024-03-24T01:27:20.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.228+0000 7feba3400640 1 -- start start 2024-03-24T01:27:20.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.229+0000 7feba3400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7feb9c166a20 0x7feb9c071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:20.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.229+0000 7feba3400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7feb9c0b3990 0x7feb9c072220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:20.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.229+0000 7feba3400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb9c0b2bd0 con 0x7feb9c1620f0 2024-03-24T01:27:20.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.229+0000 7feba3400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb9c0b4df0 con 0x7feb9c0b3990 2024-03-24T01:27:20.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.229+0000 7feba3400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb9c0b3500 con 0x7feb9c166a20 2024-03-24T01:27:20.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.229+0000 7feba1400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7feb9c0b3990 0x7feb9c072220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:20.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.229+0000 7feba1400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7feb9c0b3990 0x7feb9c072220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57574/0 (socket says 172.21.15.90:57574) 2024-03-24T01:27:20.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.229+0000 7feba1400640 1 -- 172.21.15.90:0/2672746321 learned_addr learned my addr 172.21.15.90:0/2672746321 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:20.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.230+0000 7feba1400640 1 -- 172.21.15.90:0/2672746321 >> v2:172.21.15.120:3300/0 conn(0x7feb9c166a20 msgr2=0x7feb9c071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:20.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.230+0000 7feba1400640 1 --2- 172.21.15.90:0/2672746321 >> v2:172.21.15.120:3300/0 conn(0x7feb9c166a20 0x7feb9c071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:20.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.230+0000 7feba1400640 1 -- 172.21.15.90:0/2672746321 >> v1:172.21.15.120:6789/0 conn(0x7feb9c1620f0 legacy=0x7feb9c0b46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:20.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.230+0000 7feba1400640 1 -- 172.21.15.90:0/2672746321 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb9c0b2eb0 con 0x7feb9c0b3990 2024-03-24T01:27:20.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.231+0000 7feba1400640 1 --2- 172.21.15.90:0/2672746321 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb9c0b3990 0x7feb9c072220 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7feb9c0cf850 tx=0x7feb9402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b50e0b1e85c29939 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:20.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.231+0000 7feb9be00640 1 -- 172.21.15.90:0/2672746321 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9402ca80 con 0x7feb9c0b3990 2024-03-24T01:27:20.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.231+0000 7feb9be00640 1 -- 172.21.15.90:0/2672746321 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb9402cc20 con 0x7feb9c0b3990 2024-03-24T01:27:20.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.231+0000 7feb9be00640 1 -- 172.21.15.90:0/2672746321 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb940343f0 con 0x7feb9c0b3990 2024-03-24T01:27:20.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.232+0000 7feba3400640 1 -- 172.21.15.90:0/2672746321 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb9c0b3990 msgr2=0x7feb9c072220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:20.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.232+0000 7feba3400640 1 --2- 172.21.15.90:0/2672746321 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb9c0b3990 0x7feb9c072220 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7feb9c0cf850 tx=0x7feb9402bee0 comp rx=0 tx=0).stop 2024-03-24T01:27:20.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.232+0000 7feba3400640 1 -- 172.21.15.90:0/2672746321 shutdown_connections 2024-03-24T01:27:20.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.232+0000 7feba3400640 1 --2- 172.21.15.90:0/2672746321 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb9c0b3990 0x7feb9c072220 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:20.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.232+0000 7feba3400640 1 --2- 172.21.15.90:0/2672746321 >> v2:172.21.15.120:3300/0 conn(0x7feb9c166a20 0x7feb9c071ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:20.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.232+0000 7feba3400640 1 -- 172.21.15.90:0/2672746321 >> 172.21.15.90:0/2672746321 conn(0x7feb9c06b2a0 msgr2=0x7feb9c06b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:20.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.232+0000 7feba3400640 1 -- 172.21.15.90:0/2672746321 shutdown_connections 2024-03-24T01:27:20.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.232+0000 7feba3400640 1 -- 172.21.15.90:0/2672746321 wait complete. 2024-03-24T01:27:20.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.234+0000 7feba3400640 1 Processor -- start 2024-03-24T01:27:20.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.234+0000 7feba3400640 1 -- start start 2024-03-24T01:27:20.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.235+0000 7feba3400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb9c0b3990 0x7feb9c149d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:20.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.235+0000 7feba3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb9c1620f0 0x7feb9c162650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:20.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.235+0000 7feba3400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb9c166a20 0x7feb9c0ceb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:20.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.235+0000 7feba3400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb9c0cdb40 con 0x7feb9c1620f0 2024-03-24T01:27:20.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.235+0000 7feba3400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb9c1624d0 con 0x7feb9c166a20 2024-03-24T01:27:20.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.235+0000 7feba3400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb9c0cdcc0 con 0x7feb9c0b3990 2024-03-24T01:27:20.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.235+0000 7feb93e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb9c1620f0 0x7feb9c162650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:20.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.235+0000 7feb93e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb9c1620f0 0x7feb9c162650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57590/0 (socket says 172.21.15.90:57590) 2024-03-24T01:27:20.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.235+0000 7feb93e00640 1 -- 172.21.15.90:0/843175022 learned_addr learned my addr 172.21.15.90:0/843175022 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:20.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.235+0000 7feba0a00640 1 --2- 172.21.15.90:0/843175022 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb9c0b3990 0x7feb9c149d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:20.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.235+0000 7feba1400640 1 --2- 172.21.15.90:0/843175022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb9c166a20 0x7feb9c0ceb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:20.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.236+0000 7feb93e00640 1 -- 172.21.15.90:0/843175022 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb9c0b3990 msgr2=0x7feb9c149d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:20.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.236+0000 7feb93e00640 1 --2- 172.21.15.90:0/843175022 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb9c0b3990 0x7feb9c149d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:20.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.236+0000 7feb93e00640 1 -- 172.21.15.90:0/843175022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb9c166a20 msgr2=0x7feb9c0ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:20.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.236+0000 7feb93e00640 1 --2- 172.21.15.90:0/843175022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb9c166a20 0x7feb9c0ceb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:20.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.236+0000 7feb93e00640 1 -- 172.21.15.90:0/843175022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb9c163110 con 0x7feb9c1620f0 2024-03-24T01:27:20.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.236+0000 7feb93e00640 1 --2- 172.21.15.90:0/843175022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb9c1620f0 0x7feb9c162650 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7feb8400b410 tx=0x7feb8400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:20.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.236+0000 7feb99600640 1 -- 172.21.15.90:0/843175022 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb84015020 con 0x7feb9c1620f0 2024-03-24T01:27:20.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.236+0000 7feb99600640 1 -- 172.21.15.90:0/843175022 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb84011850 con 0x7feb9c1620f0 2024-03-24T01:27:20.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.236+0000 7feb99600640 1 -- 172.21.15.90:0/843175022 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb84011c40 con 0x7feb9c1620f0 2024-03-24T01:27:20.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.236+0000 7feba3400640 1 -- 172.21.15.90:0/843175022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb9c1633f0 con 0x7feb9c1620f0 2024-03-24T01:27:20.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.236+0000 7feba3400640 1 -- 172.21.15.90:0/843175022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb9c154740 con 0x7feb9c1620f0 2024-03-24T01:27:20.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.237+0000 7feba3400640 1 -- 172.21.15.90:0/843175022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb9c073570 con 0x7feb9c1620f0 2024-03-24T01:27:20.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.238+0000 7feb99600640 1 -- 172.21.15.90:0/843175022 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7feb84018020 con 0x7feb9c1620f0 2024-03-24T01:27:20.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.239+0000 7feb99600640 1 --2- 172.21.15.90:0/843175022 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7feb7806bbd0 0x7feb7806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:20.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.239+0000 7feb99600640 1 -- 172.21.15.90:0/843175022 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7feb8408aa30 con 0x7feb9c1620f0 2024-03-24T01:27:20.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.240+0000 7feba0a00640 1 --2- 172.21.15.90:0/843175022 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7feb7806bbd0 0x7feb7806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:20.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.241+0000 7feba0a00640 1 --2- 172.21.15.90:0/843175022 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7feb7806bbd0 0x7feb7806e090 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7feb9c0afad0 tx=0x7feb8c007e30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:20.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.244+0000 7feb99600640 1 -- 172.21.15.90:0/843175022 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7feb8401f020 con 0x7feb9c1620f0 2024-03-24T01:27:20.481 INFO:tasks.ceph.mds.b.smithi120.stderr:2024-03-24T01:27:20.480+0000 7f3266400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T01:27:20.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.513+0000 7feba3400640 1 -- 172.21.15.90:0/843175022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7feb9c04f4e0 con 0x7feb9c1620f0 2024-03-24T01:27:20.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.514+0000 7feb99600640 1 -- 172.21.15.90:0/843175022 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1100 (secure 0 0 0) 0x7feb84059400 con 0x7feb9c1620f0 2024-03-24T01:27:20.515 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:27:20.515 INFO:teuthology.orchestra.run.smithi090.stdout:{"fsid":"24712165-9839-4fcc-8033-988386efea74","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":214,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":53,"num_osds":8,"num_up_osds":8,"osd_up_since":1711243438,"num_in_osds":8,"osd_in_since":1711243429,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":84},{"state_name":"creating+activating","count":13}],"num_pgs":97,"num_pools":3,"num_objects":10,"data_bytes":591040,"bytes_used":227434496,"bytes_avail":772866678784,"bytes_total":773094113280,"inactive_pgs_ratio":0.1340206116437912,"write_bytes_sec":338,"read_op_per_sec":0,"write_op_per_sec":3},"fsmap":{"epoch":34,"id":3,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":3,"rank":0,"name":"b","status":"up:active","gid":6222}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-03-24T01:27:01.544438+0000","services":{}},"progress_events":{}} 2024-03-24T01:27:20.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.516+0000 7feba3400640 1 -- 172.21.15.90:0/843175022 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7feb7806bbd0 msgr2=0x7feb7806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:20.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.516+0000 7feba3400640 1 --2- 172.21.15.90:0/843175022 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7feb7806bbd0 0x7feb7806e090 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7feb9c0afad0 tx=0x7feb8c007e30 comp rx=0 tx=0).stop 2024-03-24T01:27:20.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.516+0000 7feba3400640 1 -- 172.21.15.90:0/843175022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb9c1620f0 msgr2=0x7feb9c162650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:20.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.516+0000 7feba3400640 1 --2- 172.21.15.90:0/843175022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb9c1620f0 0x7feb9c162650 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7feb8400b410 tx=0x7feb8400b8d0 comp rx=0 tx=0).stop 2024-03-24T01:27:20.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.516+0000 7feba3400640 1 -- 172.21.15.90:0/843175022 shutdown_connections 2024-03-24T01:27:20.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.516+0000 7feba3400640 1 --2- 172.21.15.90:0/843175022 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7feb7806bbd0 0x7feb7806e090 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:20.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.516+0000 7feba3400640 1 --2- 172.21.15.90:0/843175022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb9c1620f0 0x7feb9c162650 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:20.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.516+0000 7feba3400640 1 --2- 172.21.15.90:0/843175022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb9c166a20 0x7feb9c0ceb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:20.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.516+0000 7feba3400640 1 --2- 172.21.15.90:0/843175022 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7feb9c0b3990 0x7feb9c149d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:20.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.517+0000 7feba3400640 1 -- 172.21.15.90:0/843175022 >> 172.21.15.90:0/843175022 conn(0x7feb9c06b2a0 msgr2=0x7feb9c0b46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:20.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.517+0000 7feba3400640 1 -- 172.21.15.90:0/843175022 shutdown_connections 2024-03-24T01:27:20.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.517+0000 7feba3400640 1 -- 172.21.15.90:0/843175022 wait complete. 2024-03-24T01:27:20.535 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-24T01:27:20.535 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T01:27:20.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.742+0000 7f71e7400640 1 Processor -- start 2024-03-24T01:27:20.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.743+0000 7f71e7400640 1 -- start start 2024-03-24T01:27:20.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.744+0000 7f71e7400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f71e01603e0 0x7f71e006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:20.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.744+0000 7f71e7400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71e00cd490 con 0x7f71e00cc150 2024-03-24T01:27:20.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.744+0000 7f71e7400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71e015f900 con 0x7f71e0145430 2024-03-24T01:27:20.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.744+0000 7f71e7400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71e00cd270 con 0x7f71e01603e0 2024-03-24T01:27:20.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.745+0000 7f71e4a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f71e00cc150 0x7f71e00ccb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:34674/0 (socket says 172.21.15.90:34674) 2024-03-24T01:27:20.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.745+0000 7f71e4a00640 1 -- 172.21.15.90:0/2786525682 learned_addr learned my addr 172.21.15.90:0/2786525682 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:27:20.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.745+0000 7f71dfe00640 1 -- 172.21.15.90:0/2786525682 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3910795438 0 0) 0x7f71e00cd490 con 0x7f71e00cc150 2024-03-24T01:27:20.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.745+0000 7f71dfe00640 1 -- 172.21.15.90:0/2786525682 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71c4003520 con 0x7f71e00cc150 2024-03-24T01:27:20.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.745+0000 7f71d7e00640 1 --2- 172.21.15.90:0/2786525682 >> v2:172.21.15.120:3300/0 conn(0x7f71e01603e0 0x7f71e006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:20.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.745+0000 7f71dfe00640 1 -- 172.21.15.90:0/2786525682 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1847395000 0 0) 0x7f71c4003520 con 0x7f71e00cc150 2024-03-24T01:27:20.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.745+0000 7f71dfe00640 1 -- 172.21.15.90:0/2786525682 >> v2:172.21.15.120:3300/0 conn(0x7f71e01603e0 msgr2=0x7f71e006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:20.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.745+0000 7f71dfe00640 1 --2- 172.21.15.90:0/2786525682 >> v2:172.21.15.120:3300/0 conn(0x7f71e01603e0 0x7f71e006b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:20.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.745+0000 7f71dfe00640 1 -- 172.21.15.90:0/2786525682 >> v1:172.21.15.120:6789/0 conn(0x7f71e0145430 legacy=0x7f71e00b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:20.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.745+0000 7f71dfe00640 1 -- 172.21.15.90:0/2786525682 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e015fbe0 con 0x7f71e00cc150 2024-03-24T01:27:20.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.745+0000 7f71dfe00640 1 -- 172.21.15.90:0/2786525682 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f71d0001670 con 0x7f71e00cc150 2024-03-24T01:27:20.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.746+0000 7f71dfe00640 1 -- 172.21.15.90:0/2786525682 >> v1:172.21.15.90:6789/0 conn(0x7f71e00cc150 legacy=0x7f71e00ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:20.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.746+0000 7f71dfe00640 1 --2- 172.21.15.90:0/2786525682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71c40039b0 0x7f71c4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:20.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.746+0000 7f71dfe00640 1 --2- 172.21.15.90:0/2786525682 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71c4004800 0x7f71c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:20.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.746+0000 7f71dfe00640 1 --2- 172.21.15.90:0/2786525682 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71c40051c0 0x7f71c400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:20.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.746+0000 7f71dfe00640 1 -- 172.21.15.90:0/2786525682 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e00cd270 con 0x7f71c4004800 2024-03-24T01:27:20.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.746+0000 7f71dfe00640 1 -- 172.21.15.90:0/2786525682 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c400c110 con 0x7f71c40039b0 2024-03-24T01:27:20.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.746+0000 7f71dfe00640 1 -- 172.21.15.90:0/2786525682 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c400c310 con 0x7f71c40051c0 2024-03-24T01:27:20.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.746+0000 7f71d7e00640 1 --2- 172.21.15.90:0/2786525682 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71c4004800 0x7f71c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:20.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.746+0000 7f71e4a00640 1 --2- 172.21.15.90:0/2786525682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71c40039b0 0x7f71c4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:20.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.746+0000 7f71d7e00640 1 -- 172.21.15.90:0/2786525682 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71c40051c0 msgr2=0x7f71c400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:20.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.746+0000 7f71d7e00640 1 --2- 172.21.15.90:0/2786525682 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71c40051c0 0x7f71c400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:20.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.746+0000 7f71d7e00640 1 -- 172.21.15.90:0/2786525682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71c40039b0 msgr2=0x7f71c4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:20.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.746+0000 7f71d7e00640 1 --2- 172.21.15.90:0/2786525682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71c40039b0 0x7f71c4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:20.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.746+0000 7f71d7e00640 1 -- 172.21.15.90:0/2786525682 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c4003520 con 0x7f71c4004800 2024-03-24T01:27:20.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.746+0000 7f71e5400640 1 -- 172.21.15.90:0/2786525682 reap_dead start 2024-03-24T01:27:20.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.747+0000 7f71d7e00640 1 --2- 172.21.15.90:0/2786525682 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71c4004800 0x7f71c4004c80 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f71c800ba00 tx=0x7f71c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b3e78371794ed35 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:20.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.748+0000 7f71dfe00640 1 -- 172.21.15.90:0/2786525682 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c8036020 con 0x7f71c4004800 2024-03-24T01:27:20.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.748+0000 7f71dfe00640 1 -- 172.21.15.90:0/2786525682 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71c8032640 con 0x7f71c4004800 2024-03-24T01:27:20.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.748+0000 7f71dfe00640 1 -- 172.21.15.90:0/2786525682 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c800b600 con 0x7f71c4004800 2024-03-24T01:27:20.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.748+0000 7f71e7400640 1 -- 172.21.15.90:0/2786525682 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71c4004800 msgr2=0x7f71c4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:20.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.748+0000 7f71e7400640 1 --2- 172.21.15.90:0/2786525682 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71c4004800 0x7f71c4004c80 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f71c800ba00 tx=0x7f71c4005830 comp rx=0 tx=0).stop 2024-03-24T01:27:20.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.748+0000 7f71e7400640 1 -- 172.21.15.90:0/2786525682 shutdown_connections 2024-03-24T01:27:20.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.748+0000 7f71e7400640 1 --2- 172.21.15.90:0/2786525682 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71c4004800 0x7f71c4004c80 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:20.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.748+0000 7f71e7400640 1 -- 172.21.15.90:0/2786525682 >> 172.21.15.90:0/2786525682 conn(0x7f71e00b1770 msgr2=0x7f71e00b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:20.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.749+0000 7f71e7400640 1 -- 172.21.15.90:0/2786525682 shutdown_connections 2024-03-24T01:27:20.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.749+0000 7f71e7400640 1 -- 172.21.15.90:0/2786525682 wait complete. 2024-03-24T01:27:20.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.750+0000 7f71e7400640 1 Processor -- start 2024-03-24T01:27:20.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.750+0000 7f71e7400640 1 -- start start 2024-03-24T01:27:20.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.751+0000 7f71e7400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71c4004800 0x7f71e015db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:20.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.751+0000 7f71e7400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71e01f7c50 0x7f71e01532f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:20.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.751+0000 7f71e7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71e00ad960 0x7f71e00adde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:20.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.751+0000 7f71e7400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e00ccce0 con 0x7f71e00ad960 2024-03-24T01:27:20.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.751+0000 7f71e7400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e00ccb60 con 0x7f71e01f7c50 2024-03-24T01:27:20.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.751+0000 7f71e7400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71e00cce60 con 0x7f71c4004800 2024-03-24T01:27:20.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.752+0000 7f71e5400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71e00ad960 0x7f71e00adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:20.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.752+0000 7f71e5400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71e00ad960 0x7f71e00adde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57612/0 (socket says 172.21.15.90:57612) 2024-03-24T01:27:20.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.752+0000 7f71e5400640 1 -- 172.21.15.90:0/3246278615 learned_addr learned my addr 172.21.15.90:0/3246278615 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:20.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.752+0000 7f71e4a00640 1 --2- 172.21.15.90:0/3246278615 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71c4004800 0x7f71e015db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:20.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.752+0000 7f71d7e00640 1 --2- 172.21.15.90:0/3246278615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71e01f7c50 0x7f71e01532f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:20.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.753+0000 7f71e5400640 1 -- 172.21.15.90:0/3246278615 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71c4004800 msgr2=0x7f71e015db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:20.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.753+0000 7f71e5400640 1 --2- 172.21.15.90:0/3246278615 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71c4004800 0x7f71e015db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:20.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.753+0000 7f71e5400640 1 -- 172.21.15.90:0/3246278615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71e01f7c50 msgr2=0x7f71e01532f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:20.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.753+0000 7f71e5400640 1 --2- 172.21.15.90:0/3246278615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71e01f7c50 0x7f71e01532f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:20.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.753+0000 7f71e5400640 1 -- 172.21.15.90:0/3246278615 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e015e1f0 con 0x7f71e00ad960 2024-03-24T01:27:20.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.753+0000 7f71e5400640 1 --2- 172.21.15.90:0/3246278615 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71e00ad960 0x7f71e00adde0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f71e0071920 tx=0x7f71d800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:20.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.753+0000 7f71dd600640 1 -- 172.21.15.90:0/3246278615 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d8017070 con 0x7f71e00ad960 2024-03-24T01:27:20.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.753+0000 7f71dd600640 1 -- 172.21.15.90:0/3246278615 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71d8005d40 con 0x7f71e00ad960 2024-03-24T01:27:20.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.753+0000 7f71e7400640 1 -- 172.21.15.90:0/3246278615 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71e015e4d0 con 0x7f71e00ad960 2024-03-24T01:27:20.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.753+0000 7f71dd600640 1 -- 172.21.15.90:0/3246278615 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d800a850 con 0x7f71e00ad960 2024-03-24T01:27:20.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.753+0000 7f71e7400640 1 -- 172.21.15.90:0/3246278615 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71e015ea90 con 0x7f71e00ad960 2024-03-24T01:27:20.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.754+0000 7f71e7400640 1 -- 172.21.15.90:0/3246278615 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71e0068060 con 0x7f71e00ad960 2024-03-24T01:27:20.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.755+0000 7f71dd600640 1 -- 172.21.15.90:0/3246278615 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f71d801a070 con 0x7f71e00ad960 2024-03-24T01:27:20.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.756+0000 7f71dd600640 1 --2- 172.21.15.90:0/3246278615 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f71bc06bbd0 0x7f71bc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:20.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.756+0000 7f71dd600640 1 -- 172.21.15.90:0/3246278615 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f71d8089f30 con 0x7f71e00ad960 2024-03-24T01:27:20.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.757+0000 7f71e4a00640 1 --2- 172.21.15.90:0/3246278615 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f71bc06bbd0 0x7f71bc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:20.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.758+0000 7f71e4a00640 1 --2- 172.21.15.90:0/3246278615 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f71bc06bbd0 0x7f71bc06e090 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f71d0001eb0 tx=0x7f71d0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:20.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.761+0000 7f71dd600640 1 -- 172.21.15.90:0/3246278615 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f71d80588b0 con 0x7f71e00ad960 2024-03-24T01:27:20.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.955+0000 7f71e7400640 1 -- 172.21.15.90:0/3246278615 --> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f71e00b3220 con 0x7f71bc06bbd0 2024-03-24T01:27:20.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.974+0000 7f71dd600640 1 -- 172.21.15.90:0/3246278615 <== mgr.4104 v2:172.21.15.120:6832/860392634 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+250916 (secure 0 0 0) 0x7f71e00b3220 con 0x7f71bc06bbd0 2024-03-24T01:27:20.977 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:27:20.980 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-24T01:27:20.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.980+0000 7f71e7400640 1 -- 172.21.15.90:0/3246278615 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f71bc06bbd0 msgr2=0x7f71bc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:20.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.980+0000 7f71e7400640 1 --2- 172.21.15.90:0/3246278615 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f71bc06bbd0 0x7f71bc06e090 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f71d0001eb0 tx=0x7f71d0009040 comp rx=0 tx=0).stop 2024-03-24T01:27:20.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.980+0000 7f71e7400640 1 -- 172.21.15.90:0/3246278615 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71e00ad960 msgr2=0x7f71e00adde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:20.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.980+0000 7f71e7400640 1 --2- 172.21.15.90:0/3246278615 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71e00ad960 0x7f71e00adde0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f71e0071920 tx=0x7f71d800cb90 comp rx=0 tx=0).stop 2024-03-24T01:27:20.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.980+0000 7f71e7400640 1 -- 172.21.15.90:0/3246278615 shutdown_connections 2024-03-24T01:27:20.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.980+0000 7f71e7400640 1 --2- 172.21.15.90:0/3246278615 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f71bc06bbd0 0x7f71bc06e090 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:20.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.980+0000 7f71e7400640 1 --2- 172.21.15.90:0/3246278615 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71e00ad960 0x7f71e00adde0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:20.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.980+0000 7f71e7400640 1 --2- 172.21.15.90:0/3246278615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71e01f7c50 0x7f71e01532f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:20.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.980+0000 7f71e7400640 1 --2- 172.21.15.90:0/3246278615 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71c4004800 0x7f71e015db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:20.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.980+0000 7f71e7400640 1 -- 172.21.15.90:0/3246278615 >> 172.21.15.90:0/3246278615 conn(0x7f71e00b1770 msgr2=0x7f71e00ac8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:20.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.980+0000 7f71e7400640 1 -- 172.21.15.90:0/3246278615 shutdown_connections 2024-03-24T01:27:20.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:20.980+0000 7f71e7400640 1 -- 172.21.15.90:0/3246278615 wait complete. 2024-03-24T01:27:21.001 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":146,"stamp":"2024-03-24T01:27:19.549613+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":179,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132,"ondisk_log_size":132,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222104,"kb_used_data":5908,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752616,"statfs":{"total":773094113280,"available":772866678784,"internally_reserved":0,"allocated":6049792,"data_stored":3354391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12747,"internal_metadata":221236789},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":672,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.055385"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.547093+0000","last_change":"2024-03-24T01:27:14.540765+0000","last_active":"2024-03-24T01:27:14.547093+0000","last_peered":"2024-03-24T01:27:14.547093+0000","last_clean":"2024-03-24T01:27:14.547093+0000","last_became_active":"2024-03-24T01:27:14.539541+0000","last_became_peered":"2024-03-24T01:27:14.539541+0000","last_unstale":"2024-03-24T01:27:14.547093+0000","last_undegraded":"2024-03-24T01:27:14.547093+0000","last_fullsized":"2024-03-24T01:27:14.547093+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:30:50.055523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.559986+0000","last_change":"2024-03-24T01:27:14.559986+0000","last_active":"2024-03-24T01:27:14.559986+0000","last_peered":"2024-03-24T01:27:14.559986+0000","last_clean":"2024-03-24T01:27:14.559986+0000","last_became_active":"2024-03-24T01:27:14.559649+0000","last_became_peered":"2024-03-24T01:27:14.559649+0000","last_unstale":"2024-03-24T01:27:14.559986+0000","last_undegraded":"2024-03-24T01:27:14.559986+0000","last_fullsized":"2024-03-24T01:27:14.559986+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:23.971971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.538645+0000","last_change":"2024-03-24T01:27:14.538645+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.538645+0000","last_undegraded":"2024-03-24T01:27:14.538645+0000","last_fullsized":"2024-03-24T01:27:14.538645+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.559476+0000","last_change":"2024-03-24T01:27:14.559476+0000","last_active":"2024-03-24T01:27:14.559476+0000","last_peered":"2024-03-24T01:27:14.559476+0000","last_clean":"2024-03-24T01:27:14.559476+0000","last_became_active":"2024-03-24T01:27:14.559176+0000","last_became_peered":"2024-03-24T01:27:14.559176+0000","last_unstale":"2024-03-24T01:27:14.559476+0000","last_undegraded":"2024-03-24T01:27:14.559476+0000","last_fullsized":"2024-03-24T01:27:14.559476+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:03.448771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.531210+0000","last_change":"2024-03-24T01:27:14.531210+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.531210+0000","last_undegraded":"2024-03-24T01:27:14.531210+0000","last_fullsized":"2024-03-24T01:27:14.531210+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.555238+0000","last_change":"2024-03-24T01:27:14.551494+0000","last_active":"2024-03-24T01:27:14.555238+0000","last_peered":"2024-03-24T01:27:14.555238+0000","last_clean":"2024-03-24T01:27:14.555238+0000","last_became_active":"2024-03-24T01:27:14.551175+0000","last_became_peered":"2024-03-24T01:27:14.551175+0000","last_unstale":"2024-03-24T01:27:14.555238+0000","last_undegraded":"2024-03-24T01:27:14.555238+0000","last_fullsized":"2024-03-24T01:27:14.555238+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:06:34.262662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.552294+0000","last_change":"2024-03-24T01:27:14.552294+0000","last_active":"2024-03-24T01:27:14.552294+0000","last_peered":"2024-03-24T01:27:14.552294+0000","last_clean":"2024-03-24T01:27:14.552294+0000","last_became_active":"2024-03-24T01:27:14.551839+0000","last_became_peered":"2024-03-24T01:27:14.551839+0000","last_unstale":"2024-03-24T01:27:14.552294+0000","last_undegraded":"2024-03-24T01:27:14.552294+0000","last_fullsized":"2024-03-24T01:27:14.552294+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:01:09.219193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.562359+0000","last_change":"2024-03-24T01:27:14.558536+0000","last_active":"2024-03-24T01:27:14.562359+0000","last_peered":"2024-03-24T01:27:14.562359+0000","last_clean":"2024-03-24T01:27:14.562359+0000","last_became_active":"2024-03-24T01:27:14.558211+0000","last_became_peered":"2024-03-24T01:27:14.558211+0000","last_unstale":"2024-03-24T01:27:14.562359+0000","last_undegraded":"2024-03-24T01:27:14.562359+0000","last_fullsized":"2024-03-24T01:27:14.562359+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:44:39.116139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.539698+0000","last_change":"2024-03-24T01:27:14.539698+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.539698+0000","last_undegraded":"2024-03-24T01:27:14.539698+0000","last_fullsized":"2024-03-24T01:27:14.539698+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.547500+0000","last_change":"2024-03-24T01:27:14.541578+0000","last_active":"2024-03-24T01:27:14.547500+0000","last_peered":"2024-03-24T01:27:14.547500+0000","last_clean":"2024-03-24T01:27:14.547500+0000","last_became_active":"2024-03-24T01:27:14.541289+0000","last_became_peered":"2024-03-24T01:27:14.541289+0000","last_unstale":"2024-03-24T01:27:14.547500+0000","last_undegraded":"2024-03-24T01:27:14.547500+0000","last_fullsized":"2024-03-24T01:27:14.547500+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:40.646803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.528427+0000","last_change":"2024-03-24T01:27:14.528427+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.528427+0000","last_undegraded":"2024-03-24T01:27:14.528427+0000","last_fullsized":"2024-03-24T01:27:14.528427+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.558861+0000","last_change":"2024-03-24T01:27:14.552072+0000","last_active":"2024-03-24T01:27:14.558861+0000","last_peered":"2024-03-24T01:27:14.558861+0000","last_clean":"2024-03-24T01:27:14.558861+0000","last_became_active":"2024-03-24T01:27:14.550084+0000","last_became_peered":"2024-03-24T01:27:14.550084+0000","last_unstale":"2024-03-24T01:27:14.558861+0000","last_undegraded":"2024-03-24T01:27:14.558861+0000","last_fullsized":"2024-03-24T01:27:14.558861+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:45:39.507934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.550970+0000","last_change":"2024-03-24T01:27:14.544185+0000","last_active":"2024-03-24T01:27:14.550970+0000","last_peered":"2024-03-24T01:27:14.550970+0000","last_clean":"2024-03-24T01:27:14.550970+0000","last_became_active":"2024-03-24T01:27:14.543895+0000","last_became_peered":"2024-03-24T01:27:14.543895+0000","last_unstale":"2024-03-24T01:27:14.550970+0000","last_undegraded":"2024-03-24T01:27:14.550970+0000","last_fullsized":"2024-03-24T01:27:14.550970+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:47:39.461292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.550120+0000","last_change":"2024-03-24T01:27:14.550120+0000","last_active":"2024-03-24T01:27:14.550120+0000","last_peered":"2024-03-24T01:27:14.550120+0000","last_clean":"2024-03-24T01:27:14.550120+0000","last_became_active":"2024-03-24T01:27:14.549329+0000","last_became_peered":"2024-03-24T01:27:14.549329+0000","last_unstale":"2024-03-24T01:27:14.550120+0000","last_undegraded":"2024-03-24T01:27:14.550120+0000","last_fullsized":"2024-03-24T01:27:14.550120+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:32:58.788010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.557855+0000","last_change":"2024-03-24T01:27:14.557855+0000","last_active":"2024-03-24T01:27:14.557855+0000","last_peered":"2024-03-24T01:27:14.557855+0000","last_clean":"2024-03-24T01:27:14.557855+0000","last_became_active":"2024-03-24T01:27:14.557606+0000","last_became_peered":"2024-03-24T01:27:14.557606+0000","last_unstale":"2024-03-24T01:27:14.557855+0000","last_undegraded":"2024-03-24T01:27:14.557855+0000","last_fullsized":"2024-03-24T01:27:14.557855+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:28:36.027521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.550788+0000","last_change":"2024-03-24T01:27:14.542073+0000","last_active":"2024-03-24T01:27:14.550788+0000","last_peered":"2024-03-24T01:27:14.550788+0000","last_clean":"2024-03-24T01:27:14.550788+0000","last_became_active":"2024-03-24T01:27:14.540636+0000","last_became_peered":"2024-03-24T01:27:14.540636+0000","last_unstale":"2024-03-24T01:27:14.550788+0000","last_undegraded":"2024-03-24T01:27:14.550788+0000","last_fullsized":"2024-03-24T01:27:14.550788+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:02:12.837672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.556042+0000","last_change":"2024-03-24T01:27:14.556042+0000","last_active":"2024-03-24T01:27:14.556042+0000","last_peered":"2024-03-24T01:27:14.556042+0000","last_clean":"2024-03-24T01:27:14.556042+0000","last_became_active":"2024-03-24T01:27:14.555743+0000","last_became_peered":"2024-03-24T01:27:14.555743+0000","last_unstale":"2024-03-24T01:27:14.556042+0000","last_undegraded":"2024-03-24T01:27:14.556042+0000","last_fullsized":"2024-03-24T01:27:14.556042+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:14:16.916597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.561148+0000","last_change":"2024-03-24T01:27:14.561148+0000","last_active":"2024-03-24T01:27:14.561148+0000","last_peered":"2024-03-24T01:27:14.561148+0000","last_clean":"2024-03-24T01:27:14.561148+0000","last_became_active":"2024-03-24T01:27:14.560509+0000","last_became_peered":"2024-03-24T01:27:14.560509+0000","last_unstale":"2024-03-24T01:27:14.561148+0000","last_undegraded":"2024-03-24T01:27:14.561148+0000","last_fullsized":"2024-03-24T01:27:14.561148+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:09:58.411238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.553533+0000","last_change":"2024-03-24T01:27:14.553533+0000","last_active":"2024-03-24T01:27:14.553533+0000","last_peered":"2024-03-24T01:27:14.553533+0000","last_clean":"2024-03-24T01:27:14.553533+0000","last_became_active":"2024-03-24T01:27:14.553252+0000","last_became_peered":"2024-03-24T01:27:14.553252+0000","last_unstale":"2024-03-24T01:27:14.553533+0000","last_undegraded":"2024-03-24T01:27:14.553533+0000","last_fullsized":"2024-03-24T01:27:14.553533+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:46:45.759885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.546096+0000","last_change":"2024-03-24T01:27:14.541674+0000","last_active":"2024-03-24T01:27:14.546096+0000","last_peered":"2024-03-24T01:27:14.546096+0000","last_clean":"2024-03-24T01:27:14.546096+0000","last_became_active":"2024-03-24T01:27:14.541300+0000","last_became_peered":"2024-03-24T01:27:14.541300+0000","last_unstale":"2024-03-24T01:27:14.546096+0000","last_undegraded":"2024-03-24T01:27:14.546096+0000","last_fullsized":"2024-03-24T01:27:14.546096+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:12:15.274601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.527732+0000","last_change":"2024-03-24T01:27:14.527732+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.527732+0000","last_undegraded":"2024-03-24T01:27:14.527732+0000","last_fullsized":"2024-03-24T01:27:14.527732+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.558260+0000","last_change":"2024-03-24T01:27:14.554032+0000","last_active":"2024-03-24T01:27:14.558260+0000","last_peered":"2024-03-24T01:27:14.558260+0000","last_clean":"2024-03-24T01:27:14.558260+0000","last_became_active":"2024-03-24T01:27:14.553732+0000","last_became_peered":"2024-03-24T01:27:14.553732+0000","last_unstale":"2024-03-24T01:27:14.558260+0000","last_undegraded":"2024-03-24T01:27:14.558260+0000","last_fullsized":"2024-03-24T01:27:14.558260+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:04:45.650427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.551791+0000","last_change":"2024-03-24T01:27:14.551791+0000","last_active":"2024-03-24T01:27:14.551791+0000","last_peered":"2024-03-24T01:27:14.551791+0000","last_clean":"2024-03-24T01:27:14.551791+0000","last_became_active":"2024-03-24T01:27:14.551503+0000","last_became_peered":"2024-03-24T01:27:14.551503+0000","last_unstale":"2024-03-24T01:27:14.551791+0000","last_undegraded":"2024-03-24T01:27:14.551791+0000","last_fullsized":"2024-03-24T01:27:14.551791+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:18:59.684453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.556979+0000","last_change":"2024-03-24T01:27:14.545843+0000","last_active":"2024-03-24T01:27:14.556979+0000","last_peered":"2024-03-24T01:27:14.556979+0000","last_clean":"2024-03-24T01:27:14.556979+0000","last_became_active":"2024-03-24T01:27:14.545534+0000","last_became_peered":"2024-03-24T01:27:14.545534+0000","last_unstale":"2024-03-24T01:27:14.556979+0000","last_undegraded":"2024-03-24T01:27:14.556979+0000","last_fullsized":"2024-03-24T01:27:14.556979+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:06:34.262662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.548640+0000","last_change":"2024-03-24T01:27:14.548640+0000","last_active":"2024-03-24T01:27:14.548640+0000","last_peered":"2024-03-24T01:27:14.548640+0000","last_clean":"2024-03-24T01:27:14.548640+0000","last_became_active":"2024-03-24T01:27:14.548364+0000","last_became_peered":"2024-03-24T01:27:14.548364+0000","last_unstale":"2024-03-24T01:27:14.548640+0000","last_undegraded":"2024-03-24T01:27:14.548640+0000","last_fullsized":"2024-03-24T01:27:14.548640+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:28:36.027521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.544765+0000","last_change":"2024-03-24T01:27:14.541461+0000","last_active":"2024-03-24T01:27:14.544765+0000","last_peered":"2024-03-24T01:27:14.544765+0000","last_clean":"2024-03-24T01:27:14.544765+0000","last_became_active":"2024-03-24T01:27:14.541156+0000","last_became_peered":"2024-03-24T01:27:14.541156+0000","last_unstale":"2024-03-24T01:27:14.544765+0000","last_undegraded":"2024-03-24T01:27:14.544765+0000","last_fullsized":"2024-03-24T01:27:14.544765+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:14:09.820632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.556242+0000","last_change":"2024-03-24T01:27:14.556242+0000","last_active":"2024-03-24T01:27:14.556242+0000","last_peered":"2024-03-24T01:27:14.556242+0000","last_clean":"2024-03-24T01:27:14.556242+0000","last_became_active":"2024-03-24T01:27:14.552642+0000","last_became_peered":"2024-03-24T01:27:14.552642+0000","last_unstale":"2024-03-24T01:27:14.556242+0000","last_undegraded":"2024-03-24T01:27:14.556242+0000","last_fullsized":"2024-03-24T01:27:14.556242+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:28:36.027521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.546588+0000","last_change":"2024-03-24T01:27:14.542335+0000","last_active":"2024-03-24T01:27:14.546588+0000","last_peered":"2024-03-24T01:27:14.546588+0000","last_clean":"2024-03-24T01:27:14.546588+0000","last_became_active":"2024-03-24T01:27:14.541006+0000","last_became_peered":"2024-03-24T01:27:14.541006+0000","last_unstale":"2024-03-24T01:27:14.546588+0000","last_undegraded":"2024-03-24T01:27:14.546588+0000","last_fullsized":"2024-03-24T01:27:14.546588+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:44.340192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.550220+0000","last_change":"2024-03-24T01:27:14.550220+0000","last_active":"2024-03-24T01:27:14.550220+0000","last_peered":"2024-03-24T01:27:14.550220+0000","last_clean":"2024-03-24T01:27:14.550220+0000","last_became_active":"2024-03-24T01:27:14.549812+0000","last_became_peered":"2024-03-24T01:27:14.549812+0000","last_unstale":"2024-03-24T01:27:14.550220+0000","last_undegraded":"2024-03-24T01:27:14.550220+0000","last_fullsized":"2024-03-24T01:27:14.550220+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:09:58.411238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.556235+0000","last_change":"2024-03-24T01:27:14.556235+0000","last_active":"2024-03-24T01:27:14.556235+0000","last_peered":"2024-03-24T01:27:14.556235+0000","last_clean":"2024-03-24T01:27:14.556235+0000","last_became_active":"2024-03-24T01:27:14.555888+0000","last_became_peered":"2024-03-24T01:27:14.555888+0000","last_unstale":"2024-03-24T01:27:14.556235+0000","last_undegraded":"2024-03-24T01:27:14.556235+0000","last_fullsized":"2024-03-24T01:27:14.556235+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:43:45.838352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.552212+0000","last_change":"2024-03-24T01:27:14.552212+0000","last_active":"2024-03-24T01:27:14.552212+0000","last_peered":"2024-03-24T01:27:14.552212+0000","last_clean":"2024-03-24T01:27:14.552212+0000","last_became_active":"2024-03-24T01:27:14.551775+0000","last_became_peered":"2024-03-24T01:27:14.551775+0000","last_unstale":"2024-03-24T01:27:14.552212+0000","last_undegraded":"2024-03-24T01:27:14.552212+0000","last_fullsized":"2024-03-24T01:27:14.552212+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:23.971971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.550504+0000","last_change":"2024-03-24T01:27:14.542056+0000","last_active":"2024-03-24T01:27:14.550504+0000","last_peered":"2024-03-24T01:27:14.550504+0000","last_clean":"2024-03-24T01:27:14.550504+0000","last_became_active":"2024-03-24T01:27:14.541695+0000","last_became_peered":"2024-03-24T01:27:14.541695+0000","last_unstale":"2024-03-24T01:27:14.550504+0000","last_undegraded":"2024-03-24T01:27:14.550504+0000","last_fullsized":"2024-03-24T01:27:14.550504+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:32.037863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.559666+0000","last_change":"2024-03-24T01:27:14.559666+0000","last_active":"2024-03-24T01:27:14.559666+0000","last_peered":"2024-03-24T01:27:14.559666+0000","last_clean":"2024-03-24T01:27:14.559666+0000","last_became_active":"2024-03-24T01:27:14.559356+0000","last_became_peered":"2024-03-24T01:27:14.559356+0000","last_unstale":"2024-03-24T01:27:14.559666+0000","last_undegraded":"2024-03-24T01:27:14.559666+0000","last_fullsized":"2024-03-24T01:27:14.559666+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:32.037863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.529333+0000","last_change":"2024-03-24T01:27:13.522918+0000","last_active":"2024-03-24T01:27:14.529333+0000","last_peered":"2024-03-24T01:27:14.529333+0000","last_clean":"2024-03-24T01:27:14.529333+0000","last_became_active":"2024-03-24T01:27:13.520866+0000","last_became_peered":"2024-03-24T01:27:13.520866+0000","last_unstale":"2024-03-24T01:27:14.529333+0000","last_undegraded":"2024-03-24T01:27:14.529333+0000","last_fullsized":"2024-03-24T01:27:14.529333+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:30.183145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.557567+0000","last_change":"2024-03-24T01:27:14.551119+0000","last_active":"2024-03-24T01:27:14.557567+0000","last_peered":"2024-03-24T01:27:14.557567+0000","last_clean":"2024-03-24T01:27:14.557567+0000","last_became_active":"2024-03-24T01:27:14.550800+0000","last_became_peered":"2024-03-24T01:27:14.550800+0000","last_unstale":"2024-03-24T01:27:14.557567+0000","last_undegraded":"2024-03-24T01:27:14.557567+0000","last_fullsized":"2024-03-24T01:27:14.557567+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:09:58.411238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.518296+0000","last_change":"2024-03-24T01:27:13.531689+0000","last_active":"2024-03-24T01:27:14.518296+0000","last_peered":"2024-03-24T01:27:14.518296+0000","last_clean":"2024-03-24T01:27:14.518296+0000","last_became_active":"2024-03-24T01:27:13.530929+0000","last_became_peered":"2024-03-24T01:27:13.530929+0000","last_unstale":"2024-03-24T01:27:14.518296+0000","last_undegraded":"2024-03-24T01:27:14.518296+0000","last_fullsized":"2024-03-24T01:27:14.518296+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:00:24.458371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.552432+0000","last_change":"2024-03-24T01:27:14.552432+0000","last_active":"2024-03-24T01:27:14.552432+0000","last_peered":"2024-03-24T01:27:14.552432+0000","last_clean":"2024-03-24T01:27:14.552432+0000","last_became_active":"2024-03-24T01:27:14.551697+0000","last_became_peered":"2024-03-24T01:27:14.551697+0000","last_unstale":"2024-03-24T01:27:14.552432+0000","last_undegraded":"2024-03-24T01:27:14.552432+0000","last_fullsized":"2024-03-24T01:27:14.552432+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:47:39.461292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.518290+0000","last_change":"2024-03-24T01:27:13.528068+0000","last_active":"2024-03-24T01:27:14.518290+0000","last_peered":"2024-03-24T01:27:14.518290+0000","last_clean":"2024-03-24T01:27:14.518290+0000","last_became_active":"2024-03-24T01:27:13.527750+0000","last_became_peered":"2024-03-24T01:27:13.527750+0000","last_unstale":"2024-03-24T01:27:14.518290+0000","last_undegraded":"2024-03-24T01:27:14.518290+0000","last_fullsized":"2024-03-24T01:27:14.518290+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:37:15.291837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.549966+0000","last_change":"2024-03-24T01:27:14.548464+0000","last_active":"2024-03-24T01:27:14.549966+0000","last_peered":"2024-03-24T01:27:14.549966+0000","last_clean":"2024-03-24T01:27:14.549966+0000","last_became_active":"2024-03-24T01:27:14.548113+0000","last_became_peered":"2024-03-24T01:27:14.548113+0000","last_unstale":"2024-03-24T01:27:14.549966+0000","last_undegraded":"2024-03-24T01:27:14.549966+0000","last_fullsized":"2024-03-24T01:27:14.549966+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:47:57.893749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.523180+0000","last_change":"2024-03-24T01:27:13.527296+0000","last_active":"2024-03-24T01:27:14.523180+0000","last_peered":"2024-03-24T01:27:14.523180+0000","last_clean":"2024-03-24T01:27:14.523180+0000","last_became_active":"2024-03-24T01:27:13.526656+0000","last_became_peered":"2024-03-24T01:27:13.526656+0000","last_unstale":"2024-03-24T01:27:14.523180+0000","last_undegraded":"2024-03-24T01:27:14.523180+0000","last_fullsized":"2024-03-24T01:27:14.523180+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:28:35.013061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.531475+0000","last_change":"2024-03-24T01:27:14.531475+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.531475+0000","last_undegraded":"2024-03-24T01:27:14.531475+0000","last_fullsized":"2024-03-24T01:27:14.531475+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.532174+0000","last_change":"2024-03-24T01:27:13.518516+0000","last_active":"2024-03-24T01:27:14.532174+0000","last_peered":"2024-03-24T01:27:14.532174+0000","last_clean":"2024-03-24T01:27:14.532174+0000","last_became_active":"2024-03-24T01:27:13.518276+0000","last_became_peered":"2024-03-24T01:27:13.518276+0000","last_unstale":"2024-03-24T01:27:14.532174+0000","last_undegraded":"2024-03-24T01:27:14.532174+0000","last_fullsized":"2024-03-24T01:27:14.532174+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:39.632343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.559916+0000","last_change":"2024-03-24T01:27:14.559916+0000","last_active":"2024-03-24T01:27:14.559916+0000","last_peered":"2024-03-24T01:27:14.559916+0000","last_clean":"2024-03-24T01:27:14.559916+0000","last_became_active":"2024-03-24T01:27:14.559597+0000","last_became_peered":"2024-03-24T01:27:14.559597+0000","last_unstale":"2024-03-24T01:27:14.559916+0000","last_undegraded":"2024-03-24T01:27:14.559916+0000","last_fullsized":"2024-03-24T01:27:14.559916+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:32:58.788010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.535218+0000","last_change":"2024-03-24T01:27:13.527482+0000","last_active":"2024-03-24T01:27:14.535218+0000","last_peered":"2024-03-24T01:27:14.535218+0000","last_clean":"2024-03-24T01:27:14.535218+0000","last_became_active":"2024-03-24T01:27:13.527205+0000","last_became_peered":"2024-03-24T01:27:13.527205+0000","last_unstale":"2024-03-24T01:27:14.535218+0000","last_undegraded":"2024-03-24T01:27:14.535218+0000","last_fullsized":"2024-03-24T01:27:14.535218+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:39.632343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.544743+0000","last_change":"2024-03-24T01:27:14.541241+0000","last_active":"2024-03-24T01:27:14.544743+0000","last_peered":"2024-03-24T01:27:14.544743+0000","last_clean":"2024-03-24T01:27:14.544743+0000","last_became_active":"2024-03-24T01:27:14.540872+0000","last_became_peered":"2024-03-24T01:27:14.540872+0000","last_unstale":"2024-03-24T01:27:14.544743+0000","last_undegraded":"2024-03-24T01:27:14.544743+0000","last_fullsized":"2024-03-24T01:27:14.544743+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:16:39.371501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.527060+0000","last_change":"2024-03-24T01:27:13.531379+0000","last_active":"2024-03-24T01:27:14.527060+0000","last_peered":"2024-03-24T01:27:14.527060+0000","last_clean":"2024-03-24T01:27:14.527060+0000","last_became_active":"2024-03-24T01:27:13.530618+0000","last_became_peered":"2024-03-24T01:27:13.530618+0000","last_unstale":"2024-03-24T01:27:14.527060+0000","last_undegraded":"2024-03-24T01:27:14.527060+0000","last_fullsized":"2024-03-24T01:27:14.527060+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:00.227179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.548287+0000","last_change":"2024-03-24T01:27:14.548287+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.548287+0000","last_undegraded":"2024-03-24T01:27:14.548287+0000","last_fullsized":"2024-03-24T01:27:14.548287+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.527604+0000","last_change":"2024-03-24T01:27:13.529478+0000","last_active":"2024-03-24T01:27:14.527604+0000","last_peered":"2024-03-24T01:27:14.527604+0000","last_clean":"2024-03-24T01:27:14.527604+0000","last_became_active":"2024-03-24T01:27:13.529197+0000","last_became_peered":"2024-03-24T01:27:13.529197+0000","last_unstale":"2024-03-24T01:27:14.527604+0000","last_undegraded":"2024-03-24T01:27:14.527604+0000","last_fullsized":"2024-03-24T01:27:14.527604+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:30:49.041062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.547194+0000","last_change":"2024-03-24T01:27:14.544605+0000","last_active":"2024-03-24T01:27:14.547194+0000","last_peered":"2024-03-24T01:27:14.547194+0000","last_clean":"2024-03-24T01:27:14.547194+0000","last_became_active":"2024-03-24T01:27:14.544189+0000","last_became_peered":"2024-03-24T01:27:14.544189+0000","last_unstale":"2024-03-24T01:27:14.547194+0000","last_undegraded":"2024-03-24T01:27:14.547194+0000","last_fullsized":"2024-03-24T01:27:14.547194+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:12:15.274601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.548563+0000","last_change":"2024-03-24T01:27:13.533618+0000","last_active":"2024-03-24T01:27:14.548563+0000","last_peered":"2024-03-24T01:27:14.548563+0000","last_clean":"2024-03-24T01:27:14.548563+0000","last_became_active":"2024-03-24T01:27:13.531829+0000","last_became_peered":"2024-03-24T01:27:13.531829+0000","last_unstale":"2024-03-24T01:27:14.548563+0000","last_undegraded":"2024-03-24T01:27:14.548563+0000","last_fullsized":"2024-03-24T01:27:14.548563+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:06:33.248202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.552880+0000","last_change":"2024-03-24T01:27:14.541924+0000","last_active":"2024-03-24T01:27:14.552880+0000","last_peered":"2024-03-24T01:27:14.552880+0000","last_clean":"2024-03-24T01:27:14.552880+0000","last_became_active":"2024-03-24T01:27:14.541512+0000","last_became_peered":"2024-03-24T01:27:14.541512+0000","last_unstale":"2024-03-24T01:27:14.552880+0000","last_undegraded":"2024-03-24T01:27:14.552880+0000","last_fullsized":"2024-03-24T01:27:14.552880+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:03.448771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.536520+0000","last_change":"2024-03-24T01:27:13.524033+0000","last_active":"2024-03-24T01:27:14.536520+0000","last_peered":"2024-03-24T01:27:14.536520+0000","last_clean":"2024-03-24T01:27:14.536520+0000","last_became_active":"2024-03-24T01:27:13.523502+0000","last_became_peered":"2024-03-24T01:27:13.523502+0000","last_unstale":"2024-03-24T01:27:14.536520+0000","last_undegraded":"2024-03-24T01:27:14.536520+0000","last_fullsized":"2024-03-24T01:27:14.536520+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:02:11.823211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.552947+0000","last_change":"2024-03-24T01:27:14.552947+0000","last_active":"2024-03-24T01:27:14.552947+0000","last_peered":"2024-03-24T01:27:14.552947+0000","last_clean":"2024-03-24T01:27:14.552947+0000","last_became_active":"2024-03-24T01:27:14.552583+0000","last_became_peered":"2024-03-24T01:27:14.552583+0000","last_unstale":"2024-03-24T01:27:14.552947+0000","last_undegraded":"2024-03-24T01:27:14.552947+0000","last_fullsized":"2024-03-24T01:27:14.552947+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:58.630025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"52'2","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.548205+0000","last_change":"2024-03-24T01:27:13.530329+0000","last_active":"2024-03-24T01:27:14.548205+0000","last_peered":"2024-03-24T01:27:14.548205+0000","last_clean":"2024-03-24T01:27:14.548205+0000","last_became_active":"2024-03-24T01:27:13.529972+0000","last_became_peered":"2024-03-24T01:27:13.529972+0000","last_unstale":"2024-03-24T01:27:14.548205+0000","last_undegraded":"2024-03-24T01:27:14.548205+0000","last_fullsized":"2024-03-24T01:27:14.548205+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:43.325731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.552486+0000","last_change":"2024-03-24T01:27:14.552486+0000","last_active":"2024-03-24T01:27:14.552486+0000","last_peered":"2024-03-24T01:27:14.552486+0000","last_clean":"2024-03-24T01:27:14.552486+0000","last_became_active":"2024-03-24T01:27:14.551883+0000","last_became_peered":"2024-03-24T01:27:14.551883+0000","last_unstale":"2024-03-24T01:27:14.552486+0000","last_undegraded":"2024-03-24T01:27:14.552486+0000","last_fullsized":"2024-03-24T01:27:14.552486+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:44.340192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"52'5","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.535485+0000","last_change":"2024-03-24T01:27:13.521128+0000","last_active":"2024-03-24T01:27:14.535485+0000","last_peered":"2024-03-24T01:27:14.535485+0000","last_clean":"2024-03-24T01:27:14.535485+0000","last_became_active":"2024-03-24T01:27:13.520358+0000","last_became_peered":"2024-03-24T01:27:13.520358+0000","last_unstale":"2024-03-24T01:27:14.535485+0000","last_undegraded":"2024-03-24T01:27:14.535485+0000","last_fullsized":"2024-03-24T01:27:14.535485+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:01:08.204732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.522002+0000","last_change":"2024-03-24T01:27:14.522002+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.522002+0000","last_undegraded":"2024-03-24T01:27:14.522002+0000","last_fullsized":"2024-03-24T01:27:14.522002+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.532355+0000","last_change":"2024-03-24T01:27:13.532029+0000","last_active":"2024-03-24T01:27:14.532355+0000","last_peered":"2024-03-24T01:27:14.532355+0000","last_clean":"2024-03-24T01:27:14.532355+0000","last_became_active":"2024-03-24T01:27:13.531765+0000","last_became_peered":"2024-03-24T01:27:13.531765+0000","last_unstale":"2024-03-24T01:27:14.532355+0000","last_undegraded":"2024-03-24T01:27:14.532355+0000","last_fullsized":"2024-03-24T01:27:14.532355+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:02:11.823211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.537552+0000","last_change":"2024-03-24T01:27:14.537552+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.537552+0000","last_undegraded":"2024-03-24T01:27:14.537552+0000","last_fullsized":"2024-03-24T01:27:14.537552+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.527543+0000","last_change":"2024-03-24T01:27:13.530136+0000","last_active":"2024-03-24T01:27:14.527543+0000","last_peered":"2024-03-24T01:27:14.527543+0000","last_clean":"2024-03-24T01:27:14.527543+0000","last_became_active":"2024-03-24T01:27:13.529811+0000","last_became_peered":"2024-03-24T01:27:13.529811+0000","last_unstale":"2024-03-24T01:27:14.527543+0000","last_undegraded":"2024-03-24T01:27:14.527543+0000","last_fullsized":"2024-03-24T01:27:14.527543+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:37:15.291837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.563469+0000","last_change":"2024-03-24T01:27:14.561422+0000","last_active":"2024-03-24T01:27:14.563469+0000","last_peered":"2024-03-24T01:27:14.563469+0000","last_clean":"2024-03-24T01:27:14.563469+0000","last_became_active":"2024-03-24T01:27:14.559580+0000","last_became_peered":"2024-03-24T01:27:14.559580+0000","last_unstale":"2024-03-24T01:27:14.563469+0000","last_undegraded":"2024-03-24T01:27:14.563469+0000","last_fullsized":"2024-03-24T01:27:14.563469+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:18:51.495804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.546920+0000","last_change":"2024-03-24T01:27:13.532579+0000","last_active":"2024-03-24T01:27:14.546920+0000","last_peered":"2024-03-24T01:27:14.546920+0000","last_clean":"2024-03-24T01:27:14.546920+0000","last_became_active":"2024-03-24T01:27:13.532264+0000","last_became_peered":"2024-03-24T01:27:13.532264+0000","last_unstale":"2024-03-24T01:27:14.546920+0000","last_undegraded":"2024-03-24T01:27:14.546920+0000","last_fullsized":"2024-03-24T01:27:14.546920+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:12:14.260140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.545505+0000","last_change":"2024-03-24T01:27:14.545505+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.545505+0000","last_undegraded":"2024-03-24T01:27:14.545505+0000","last_fullsized":"2024-03-24T01:27:14.545505+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"52'8","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.536546+0000","last_change":"2024-03-24T01:27:13.518331+0000","last_active":"2024-03-24T01:27:14.536546+0000","last_peered":"2024-03-24T01:27:14.536546+0000","last_clean":"2024-03-24T01:27:14.536546+0000","last_became_active":"2024-03-24T01:27:13.518017+0000","last_became_peered":"2024-03-24T01:27:13.518017+0000","last_unstale":"2024-03-24T01:27:14.536546+0000","last_undegraded":"2024-03-24T01:27:14.536546+0000","last_fullsized":"2024-03-24T01:27:14.536546+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:02:41.582283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.547545+0000","last_change":"2024-03-24T01:27:14.543017+0000","last_active":"2024-03-24T01:27:14.547545+0000","last_peered":"2024-03-24T01:27:14.547545+0000","last_clean":"2024-03-24T01:27:14.547545+0000","last_became_active":"2024-03-24T01:27:14.542730+0000","last_became_peered":"2024-03-24T01:27:14.542730+0000","last_unstale":"2024-03-24T01:27:14.547545+0000","last_undegraded":"2024-03-24T01:27:14.547545+0000","last_fullsized":"2024-03-24T01:27:14.547545+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:19:18.467994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.545934+0000","last_change":"2024-03-24T01:27:13.530251+0000","last_active":"2024-03-24T01:27:14.545934+0000","last_peered":"2024-03-24T01:27:14.545934+0000","last_clean":"2024-03-24T01:27:14.545934+0000","last_became_active":"2024-03-24T01:27:13.529767+0000","last_became_peered":"2024-03-24T01:27:13.529767+0000","last_unstale":"2024-03-24T01:27:14.545934+0000","last_undegraded":"2024-03-24T01:27:14.545934+0000","last_fullsized":"2024-03-24T01:27:14.545934+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:43:44.823891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.557227+0000","last_change":"2024-03-24T01:27:14.554474+0000","last_active":"2024-03-24T01:27:14.557227+0000","last_peered":"2024-03-24T01:27:14.557227+0000","last_clean":"2024-03-24T01:27:14.557227+0000","last_became_active":"2024-03-24T01:27:14.554189+0000","last_became_peered":"2024-03-24T01:27:14.554189+0000","last_unstale":"2024-03-24T01:27:14.557227+0000","last_undegraded":"2024-03-24T01:27:14.557227+0000","last_fullsized":"2024-03-24T01:27:14.557227+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:43:45.838352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.529435+0000","last_change":"2024-03-24T01:27:13.526531+0000","last_active":"2024-03-24T01:27:14.529435+0000","last_peered":"2024-03-24T01:27:14.529435+0000","last_clean":"2024-03-24T01:27:14.529435+0000","last_became_active":"2024-03-24T01:27:13.526212+0000","last_became_peered":"2024-03-24T01:27:13.526212+0000","last_unstale":"2024-03-24T01:27:14.529435+0000","last_undegraded":"2024-03-24T01:27:14.529435+0000","last_fullsized":"2024-03-24T01:27:14.529435+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:02:41.656569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.531995+0000","last_change":"2024-03-24T01:27:14.531995+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.531995+0000","last_undegraded":"2024-03-24T01:27:14.531995+0000","last_fullsized":"2024-03-24T01:27:14.531995+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.529356+0000","last_change":"2024-03-24T01:27:13.526615+0000","last_active":"2024-03-24T01:27:14.529356+0000","last_peered":"2024-03-24T01:27:14.529356+0000","last_clean":"2024-03-24T01:27:14.529356+0000","last_became_active":"2024-03-24T01:27:13.526286+0000","last_became_peered":"2024-03-24T01:27:13.526286+0000","last_unstale":"2024-03-24T01:27:14.529356+0000","last_undegraded":"2024-03-24T01:27:14.529356+0000","last_fullsized":"2024-03-24T01:27:14.529356+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:42:00.908936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.547136+0000","last_change":"2024-03-24T01:27:14.539239+0000","last_active":"2024-03-24T01:27:14.547136+0000","last_peered":"2024-03-24T01:27:14.547136+0000","last_clean":"2024-03-24T01:27:14.547136+0000","last_became_active":"2024-03-24T01:27:14.538525+0000","last_became_peered":"2024-03-24T01:27:14.538525+0000","last_unstale":"2024-03-24T01:27:14.547136+0000","last_undegraded":"2024-03-24T01:27:14.547136+0000","last_fullsized":"2024-03-24T01:27:14.547136+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:42:01.923397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.528640+0000","last_change":"2024-03-24T01:27:13.523480+0000","last_active":"2024-03-24T01:27:14.528640+0000","last_peered":"2024-03-24T01:27:14.528640+0000","last_clean":"2024-03-24T01:27:14.528640+0000","last_became_active":"2024-03-24T01:27:13.523203+0000","last_became_peered":"2024-03-24T01:27:13.523203+0000","last_unstale":"2024-03-24T01:27:14.528640+0000","last_undegraded":"2024-03-24T01:27:14.528640+0000","last_fullsized":"2024-03-24T01:27:14.528640+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:00:24.458371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.557519+0000","last_change":"2024-03-24T01:27:14.549953+0000","last_active":"2024-03-24T01:27:14.557519+0000","last_peered":"2024-03-24T01:27:14.557519+0000","last_clean":"2024-03-24T01:27:14.557519+0000","last_became_active":"2024-03-24T01:27:14.549616+0000","last_became_peered":"2024-03-24T01:27:14.549616+0000","last_unstale":"2024-03-24T01:27:14.557519+0000","last_undegraded":"2024-03-24T01:27:14.557519+0000","last_fullsized":"2024-03-24T01:27:14.557519+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:32:58.788010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.518549+0000","last_change":"2024-03-24T01:27:13.526041+0000","last_active":"2024-03-24T01:27:14.518549+0000","last_peered":"2024-03-24T01:27:14.518549+0000","last_clean":"2024-03-24T01:27:14.518549+0000","last_became_active":"2024-03-24T01:27:13.525587+0000","last_became_peered":"2024-03-24T01:27:13.525587+0000","last_unstale":"2024-03-24T01:27:14.518549+0000","last_undegraded":"2024-03-24T01:27:14.518549+0000","last_fullsized":"2024-03-24T01:27:14.518549+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:30.183145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.546589+0000","last_change":"2024-03-24T01:27:14.539787+0000","last_active":"2024-03-24T01:27:14.546589+0000","last_peered":"2024-03-24T01:27:14.546589+0000","last_clean":"2024-03-24T01:27:14.546589+0000","last_became_active":"2024-03-24T01:27:14.539446+0000","last_became_peered":"2024-03-24T01:27:14.539446+0000","last_unstale":"2024-03-24T01:27:14.546589+0000","last_undegraded":"2024-03-24T01:27:14.546589+0000","last_fullsized":"2024-03-24T01:27:14.546589+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:02:12.837672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.535722+0000","last_change":"2024-03-24T01:27:13.519780+0000","last_active":"2024-03-24T01:27:14.535722+0000","last_peered":"2024-03-24T01:27:14.535722+0000","last_clean":"2024-03-24T01:27:14.535722+0000","last_became_active":"2024-03-24T01:27:13.519458+0000","last_became_peered":"2024-03-24T01:27:13.519458+0000","last_unstale":"2024-03-24T01:27:14.535722+0000","last_undegraded":"2024-03-24T01:27:14.535722+0000","last_fullsized":"2024-03-24T01:27:14.535722+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:46:44.745424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.549326+0000","last_change":"2024-03-24T01:27:14.549326+0000","last_active":"2024-03-24T01:27:14.549326+0000","last_peered":"2024-03-24T01:27:14.549326+0000","last_clean":"2024-03-24T01:27:14.549326+0000","last_became_active":"2024-03-24T01:27:14.548886+0000","last_became_peered":"2024-03-24T01:27:14.548886+0000","last_unstale":"2024-03-24T01:27:14.549326+0000","last_undegraded":"2024-03-24T01:27:14.549326+0000","last_fullsized":"2024-03-24T01:27:14.549326+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:03.448771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.545176+0000","last_change":"2024-03-24T01:27:13.521689+0000","last_active":"2024-03-24T01:27:14.545176+0000","last_peered":"2024-03-24T01:27:14.545176+0000","last_clean":"2024-03-24T01:27:14.545176+0000","last_became_active":"2024-03-24T01:27:13.521436+0000","last_became_peered":"2024-03-24T01:27:13.521436+0000","last_unstale":"2024-03-24T01:27:14.545176+0000","last_undegraded":"2024-03-24T01:27:14.545176+0000","last_fullsized":"2024-03-24T01:27:14.545176+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:47:38.446831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.551289+0000","last_change":"2024-03-24T01:27:14.543585+0000","last_active":"2024-03-24T01:27:14.551289+0000","last_peered":"2024-03-24T01:27:14.551289+0000","last_clean":"2024-03-24T01:27:14.551289+0000","last_became_active":"2024-03-24T01:27:14.543301+0000","last_became_peered":"2024-03-24T01:27:14.543301+0000","last_unstale":"2024-03-24T01:27:14.551289+0000","last_undegraded":"2024-03-24T01:27:14.551289+0000","last_fullsized":"2024-03-24T01:27:14.551289+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:12:15.274601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.527295+0000","last_change":"2024-03-24T01:27:13.522613+0000","last_active":"2024-03-24T01:27:14.527295+0000","last_peered":"2024-03-24T01:27:14.527295+0000","last_clean":"2024-03-24T01:27:14.527295+0000","last_became_active":"2024-03-24T01:27:13.522331+0000","last_became_peered":"2024-03-24T01:27:13.522331+0000","last_unstale":"2024-03-24T01:27:14.527295+0000","last_undegraded":"2024-03-24T01:27:14.527295+0000","last_fullsized":"2024-03-24T01:27:14.527295+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:08:09.036710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.558092+0000","last_change":"2024-03-24T01:27:14.550330+0000","last_active":"2024-03-24T01:27:14.558092+0000","last_peered":"2024-03-24T01:27:14.558092+0000","last_clean":"2024-03-24T01:27:14.558092+0000","last_became_active":"2024-03-24T01:27:14.550056+0000","last_became_peered":"2024-03-24T01:27:14.550056+0000","last_unstale":"2024-03-24T01:27:14.558092+0000","last_undegraded":"2024-03-24T01:27:14.558092+0000","last_fullsized":"2024-03-24T01:27:14.558092+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:23.971971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.529686+0000","last_change":"2024-03-24T01:27:13.533418+0000","last_active":"2024-03-24T01:27:14.529686+0000","last_peered":"2024-03-24T01:27:14.529686+0000","last_clean":"2024-03-24T01:27:14.529686+0000","last_became_active":"2024-03-24T01:27:13.533164+0000","last_became_peered":"2024-03-24T01:27:13.533164+0000","last_unstale":"2024-03-24T01:27:14.529686+0000","last_undegraded":"2024-03-24T01:27:14.529686+0000","last_fullsized":"2024-03-24T01:27:14.529686+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:30:49.041062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.551209+0000","last_change":"2024-03-24T01:27:14.543832+0000","last_active":"2024-03-24T01:27:14.551209+0000","last_peered":"2024-03-24T01:27:14.551209+0000","last_clean":"2024-03-24T01:27:14.551209+0000","last_became_active":"2024-03-24T01:27:14.543520+0000","last_became_peered":"2024-03-24T01:27:14.543520+0000","last_unstale":"2024-03-24T01:27:14.551209+0000","last_undegraded":"2024-03-24T01:27:14.551209+0000","last_fullsized":"2024-03-24T01:27:14.551209+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:44.340192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.548373+0000","last_change":"2024-03-24T01:27:13.530300+0000","last_active":"2024-03-24T01:27:14.548373+0000","last_peered":"2024-03-24T01:27:14.548373+0000","last_clean":"2024-03-24T01:27:14.548373+0000","last_became_active":"2024-03-24T01:27:13.529919+0000","last_became_peered":"2024-03-24T01:27:13.529919+0000","last_unstale":"2024-03-24T01:27:14.548373+0000","last_undegraded":"2024-03-24T01:27:14.548373+0000","last_fullsized":"2024-03-24T01:27:14.548373+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:47:38.446831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.548304+0000","last_change":"2024-03-24T01:27:14.544735+0000","last_active":"2024-03-24T01:27:14.548304+0000","last_peered":"2024-03-24T01:27:14.548304+0000","last_clean":"2024-03-24T01:27:14.548304+0000","last_became_active":"2024-03-24T01:27:14.544460+0000","last_became_peered":"2024-03-24T01:27:14.544460+0000","last_unstale":"2024-03-24T01:27:14.548304+0000","last_undegraded":"2024-03-24T01:27:14.548304+0000","last_fullsized":"2024-03-24T01:27:14.548304+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:06.757918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.536983+0000","last_change":"2024-03-24T01:27:13.525558+0000","last_active":"2024-03-24T01:27:14.536983+0000","last_peered":"2024-03-24T01:27:14.536983+0000","last_clean":"2024-03-24T01:27:14.536983+0000","last_became_active":"2024-03-24T01:27:13.525223+0000","last_became_peered":"2024-03-24T01:27:13.525223+0000","last_unstale":"2024-03-24T01:27:14.536983+0000","last_undegraded":"2024-03-24T01:27:14.536983+0000","last_fullsized":"2024-03-24T01:27:14.536983+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:43.325731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":8,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.549747+0000","last_change":"2024-03-24T01:27:14.549747+0000","last_active":"2024-03-24T01:27:14.549747+0000","last_peered":"2024-03-24T01:27:14.549747+0000","last_clean":"2024-03-24T01:27:14.549747+0000","last_became_active":"2024-03-24T01:27:14.548344+0000","last_became_peered":"2024-03-24T01:27:14.548344+0000","last_unstale":"2024-03-24T01:27:14.549747+0000","last_undegraded":"2024-03-24T01:27:14.549747+0000","last_fullsized":"2024-03-24T01:27:14.549747+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:32.037863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.535645+0000","last_change":"2024-03-24T01:27:13.519855+0000","last_active":"2024-03-24T01:27:14.535645+0000","last_peered":"2024-03-24T01:27:14.535645+0000","last_clean":"2024-03-24T01:27:14.535645+0000","last_became_active":"2024-03-24T01:27:13.519566+0000","last_became_peered":"2024-03-24T01:27:13.519566+0000","last_unstale":"2024-03-24T01:27:14.535645+0000","last_undegraded":"2024-03-24T01:27:14.535645+0000","last_fullsized":"2024-03-24T01:27:14.535645+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:57.615564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.522676+0000","last_change":"2024-03-24T01:27:14.522676+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.522676+0000","last_undegraded":"2024-03-24T01:27:14.522676+0000","last_fullsized":"2024-03-24T01:27:14.522676+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.535933+0000","last_change":"2024-03-24T01:27:13.524780+0000","last_active":"2024-03-24T01:27:14.535933+0000","last_peered":"2024-03-24T01:27:14.535933+0000","last_clean":"2024-03-24T01:27:14.535933+0000","last_became_active":"2024-03-24T01:27:13.524518+0000","last_became_peered":"2024-03-24T01:27:13.524518+0000","last_unstale":"2024-03-24T01:27:14.535933+0000","last_undegraded":"2024-03-24T01:27:14.535933+0000","last_fullsized":"2024-03-24T01:27:14.535933+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:14:15.902136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":3,"reported_epoch":52,"state":"creating+activating","last_fresh":"2024-03-24T01:27:14.534874+0000","last_change":"2024-03-24T01:27:14.534874+0000","last_active":"2024-03-24T01:27:13.484941+0000","last_peered":"2024-03-24T01:27:13.484941+0000","last_clean":"2024-03-24T01:27:13.484941+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:27:14.534874+0000","last_undegraded":"2024-03-24T01:27:14.534874+0000","last_fullsized":"2024-03-24T01:27:14.534874+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.534655+0000","last_change":"2024-03-24T01:27:13.519549+0000","last_active":"2024-03-24T01:27:14.534655+0000","last_peered":"2024-03-24T01:27:14.534655+0000","last_clean":"2024-03-24T01:27:14.534655+0000","last_became_active":"2024-03-24T01:27:13.519174+0000","last_became_peered":"2024-03-24T01:27:13.519174+0000","last_unstale":"2024-03-24T01:27:14.534655+0000","last_undegraded":"2024-03-24T01:27:14.534655+0000","last_fullsized":"2024-03-24T01:27:14.534655+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:30:49.041062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.549972+0000","last_change":"2024-03-24T01:27:14.548520+0000","last_active":"2024-03-24T01:27:14.549972+0000","last_peered":"2024-03-24T01:27:14.549972+0000","last_clean":"2024-03-24T01:27:14.549972+0000","last_became_active":"2024-03-24T01:27:14.547968+0000","last_became_peered":"2024-03-24T01:27:14.547968+0000","last_unstale":"2024-03-24T01:27:14.549972+0000","last_undegraded":"2024-03-24T01:27:14.549972+0000","last_fullsized":"2024-03-24T01:27:14.549972+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:36.294802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.520961+0000","last_change":"2024-03-24T01:27:13.524804+0000","last_active":"2024-03-24T01:27:14.520961+0000","last_peered":"2024-03-24T01:27:14.520961+0000","last_clean":"2024-03-24T01:27:14.520961+0000","last_became_active":"2024-03-24T01:27:13.524487+0000","last_became_peered":"2024-03-24T01:27:13.524487+0000","last_unstale":"2024-03-24T01:27:14.520961+0000","last_undegraded":"2024-03-24T01:27:14.520961+0000","last_fullsized":"2024-03-24T01:27:14.520961+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:43:44.823891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.546611+0000","last_change":"2024-03-24T01:27:14.539180+0000","last_active":"2024-03-24T01:27:14.546611+0000","last_peered":"2024-03-24T01:27:14.546611+0000","last_clean":"2024-03-24T01:27:14.546611+0000","last_became_active":"2024-03-24T01:27:14.538800+0000","last_became_peered":"2024-03-24T01:27:14.538800+0000","last_unstale":"2024-03-24T01:27:14.546611+0000","last_undegraded":"2024-03-24T01:27:14.546611+0000","last_fullsized":"2024-03-24T01:27:14.546611+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:40.646803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.535286+0000","last_change":"2024-03-24T01:27:13.532470+0000","last_active":"2024-03-24T01:27:14.535286+0000","last_peered":"2024-03-24T01:27:14.535286+0000","last_clean":"2024-03-24T01:27:14.535286+0000","last_became_active":"2024-03-24T01:27:13.530705+0000","last_became_peered":"2024-03-24T01:27:13.530705+0000","last_unstale":"2024-03-24T01:27:14.535286+0000","last_undegraded":"2024-03-24T01:27:14.535286+0000","last_fullsized":"2024-03-24T01:27:14.535286+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:12:14.260140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":229,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-24T01:27:14.545758+0000","last_change":"2024-03-24T01:24:01.545855+0000","last_active":"2024-03-24T01:27:14.545758+0000","last_peered":"2024-03-24T01:27:14.545758+0000","last_clean":"2024-03-24T01:27:14.545758+0000","last_became_active":"2024-03-24T01:24:01.543811+0000","last_became_peered":"2024-03-24T01:24:01.543811+0000","last_unstale":"2024-03-24T01:27:14.545758+0000","last_undegraded":"2024-03-24T01:27:14.545758+0000","last_fullsized":"2024-03-24T01:27:14.545758+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_clean_scrub_stamp":"2024-03-24T01:23:59.861564+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:03.682631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":672,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15,"ondisk_log_size":15,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574890,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28196,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343644,"statfs":{"total":96636764160,"available":96607891456,"internally_reserved":0,"allocated":1200128,"data_stored":861179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":270811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":270811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574890,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":270811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":270811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574891,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":270811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":618496,"data_stored":277978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574891,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28196,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343644,"statfs":{"total":96636764160,"available":96607891456,"internally_reserved":0,"allocated":1200128,"data_stored":861179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T01:27:24.006 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T01:27:24.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.220+0000 7fbdcfe00640 1 Processor -- start 2024-03-24T01:27:24.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.220+0000 7fbdcfe00640 1 -- start start 2024-03-24T01:27:24.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.221+0000 7fbdcfe00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fbdc8155030 0x7fbdc815efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:24.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.221+0000 7fbdcfe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fbdc80cdac0 0x7fbdc806b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:24.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.221+0000 7fbdcfe00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdc815f500 con 0x7fbdc806b600 2024-03-24T01:27:24.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.221+0000 7fbdcfe00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdc815bf70 con 0x7fbdc80cdac0 2024-03-24T01:27:24.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.221+0000 7fbdcfe00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdc806ba00 con 0x7fbdc8155030 2024-03-24T01:27:24.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.222+0000 7fbdcca00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fbdc80cdac0 0x7fbdc806b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:24.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.222+0000 7fbdcca00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fbdc80cdac0 0x7fbdc806b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57624/0 (socket says 172.21.15.90:57624) 2024-03-24T01:27:24.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.222+0000 7fbdcca00640 1 -- 172.21.15.90:0/3893764630 learned_addr learned my addr 172.21.15.90:0/3893764630 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:24.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.222+0000 7fbdcde00640 1 --1- 172.21.15.90:0/3893764630 >> v1:172.21.15.90:6789/0 conn(0x7fbdc806b600 0x7fbdc80ae530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:34676/0 (socket says 172.21.15.90:34676) 2024-03-24T01:27:24.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.222+0000 7fbdcd400640 1 --2- 172.21.15.90:0/3893764630 >> v2:172.21.15.120:3300/0 conn(0x7fbdc8155030 0x7fbdc815efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:24.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.222+0000 7fbdcca00640 1 -- 172.21.15.90:0/3893764630 >> v2:172.21.15.120:3300/0 conn(0x7fbdc8155030 msgr2=0x7fbdc815efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:24.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.222+0000 7fbdcca00640 1 --2- 172.21.15.90:0/3893764630 >> v2:172.21.15.120:3300/0 conn(0x7fbdc8155030 0x7fbdc815efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:24.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.222+0000 7fbdcca00640 1 -- 172.21.15.90:0/3893764630 >> v1:172.21.15.90:6789/0 conn(0x7fbdc806b600 legacy=0x7fbdc80ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:24.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.222+0000 7fbdcca00640 1 -- 172.21.15.90:0/3893764630 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc815b360 con 0x7fbdc80cdac0 2024-03-24T01:27:24.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.223+0000 7fbdcca00640 1 --2- 172.21.15.90:0/3893764630 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdc80cdac0 0x7fbdc806b0c0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fbdc806ecc0 tx=0x7fbdb402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=901c0dd903346b17 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:24.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.224+0000 7fbdc7e00640 1 -- 172.21.15.90:0/3893764630 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb4034070 con 0x7fbdc80cdac0 2024-03-24T01:27:24.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.224+0000 7fbdc7e00640 1 -- 172.21.15.90:0/3893764630 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdb4030860 con 0x7fbdc80cdac0 2024-03-24T01:27:24.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.224+0000 7fbdc7e00640 1 -- 172.21.15.90:0/3893764630 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb4030c30 con 0x7fbdc80cdac0 2024-03-24T01:27:24.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.224+0000 7fbdcfe00640 1 -- 172.21.15.90:0/3893764630 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdc80cdac0 msgr2=0x7fbdc806b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:24.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.224+0000 7fbdcfe00640 1 --2- 172.21.15.90:0/3893764630 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdc80cdac0 0x7fbdc806b0c0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fbdc806ecc0 tx=0x7fbdb402b9a0 comp rx=0 tx=0).stop 2024-03-24T01:27:24.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.224+0000 7fbdcfe00640 1 -- 172.21.15.90:0/3893764630 shutdown_connections 2024-03-24T01:27:24.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.224+0000 7fbdcfe00640 1 --2- 172.21.15.90:0/3893764630 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdc80cdac0 0x7fbdc806b0c0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:24.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.224+0000 7fbdcfe00640 1 --2- 172.21.15.90:0/3893764630 >> v2:172.21.15.120:3300/0 conn(0x7fbdc8155030 0x7fbdc815efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:24.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.224+0000 7fbdcfe00640 1 -- 172.21.15.90:0/3893764630 >> 172.21.15.90:0/3893764630 conn(0x7fbdc80b1770 msgr2=0x7fbdc80b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:24.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.224+0000 7fbdcfe00640 1 -- 172.21.15.90:0/3893764630 shutdown_connections 2024-03-24T01:27:24.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.224+0000 7fbdcfe00640 1 -- 172.21.15.90:0/3893764630 wait complete. 2024-03-24T01:27:24.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.226+0000 7fbdcfe00640 1 Processor -- start 2024-03-24T01:27:24.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.226+0000 7fbdcfe00640 1 -- start start 2024-03-24T01:27:24.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.227+0000 7fbdcfe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbdc806b600 0x7fbdc81454f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:24.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.227+0000 7fbdcfe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdc80cdac0 0x7fbdc80c6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:24.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.227+0000 7fbdcfe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbdc8155030 0x7fbdc80c4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:24.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.227+0000 7fbdcfe00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc815bc00 con 0x7fbdc80cdac0 2024-03-24T01:27:24.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.227+0000 7fbdcfe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc815b920 con 0x7fbdc806b600 2024-03-24T01:27:24.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.227+0000 7fbdcfe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc8155410 con 0x7fbdc8155030 2024-03-24T01:27:24.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.227+0000 7fbdcca00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdc80cdac0 0x7fbdc80c6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:24.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.227+0000 7fbdcca00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdc80cdac0 0x7fbdc80c6890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57632/0 (socket says 172.21.15.90:57632) 2024-03-24T01:27:24.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.227+0000 7fbdcca00640 1 -- 172.21.15.90:0/1054663146 learned_addr learned my addr 172.21.15.90:0/1054663146 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:24.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.228+0000 7fbdcd400640 1 --2- 172.21.15.90:0/1054663146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbdc806b600 0x7fbdc81454f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:24.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.228+0000 7fbdcca00640 1 -- 172.21.15.90:0/1054663146 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbdc8155030 msgr2=0x7fbdc80c4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:24.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.228+0000 7fbdcca00640 1 --2- 172.21.15.90:0/1054663146 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbdc8155030 0x7fbdc80c4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:24.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.228+0000 7fbdcca00640 1 -- 172.21.15.90:0/1054663146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbdc806b600 msgr2=0x7fbdc81454f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:24.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.228+0000 7fbdcca00640 1 --2- 172.21.15.90:0/1054663146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbdc806b600 0x7fbdc81454f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:24.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.228+0000 7fbdcca00640 1 -- 172.21.15.90:0/1054663146 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc8145a30 con 0x7fbdc80cdac0 2024-03-24T01:27:24.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.228+0000 7fbdcca00640 1 --2- 172.21.15.90:0/1054663146 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdc80cdac0 0x7fbdc80c6890 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fbdb402b9e0 tx=0x7fbdb4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:24.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.228+0000 7fbdc5600640 1 -- 172.21.15.90:0/1054663146 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb4048070 con 0x7fbdc80cdac0 2024-03-24T01:27:24.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.228+0000 7fbdc5600640 1 -- 172.21.15.90:0/1054663146 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdb4030660 con 0x7fbdc80cdac0 2024-03-24T01:27:24.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.228+0000 7fbdc5600640 1 -- 172.21.15.90:0/1054663146 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb4034040 con 0x7fbdc80cdac0 2024-03-24T01:27:24.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.229+0000 7fbdcfe00640 1 -- 172.21.15.90:0/1054663146 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdc8142f50 con 0x7fbdc80cdac0 2024-03-24T01:27:24.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.229+0000 7fbdcfe00640 1 -- 172.21.15.90:0/1054663146 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdc8143510 con 0x7fbdc80cdac0 2024-03-24T01:27:24.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.230+0000 7fbdcfe00640 1 -- 172.21.15.90:0/1054663146 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdc8068060 con 0x7fbdc80cdac0 2024-03-24T01:27:24.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.231+0000 7fbdc5600640 1 -- 172.21.15.90:0/1054663146 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fbdb4037070 con 0x7fbdc80cdac0 2024-03-24T01:27:24.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.231+0000 7fbdc5600640 1 --2- 172.21.15.90:0/1054663146 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbda806bbd0 0x7fbda806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:24.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.232+0000 7fbdc5600640 1 -- 172.21.15.90:0/1054663146 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbdb40acb70 con 0x7fbdc80cdac0 2024-03-24T01:27:24.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.232+0000 7fbdcd400640 1 --2- 172.21.15.90:0/1054663146 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbda806bbd0 0x7fbda806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:24.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.233+0000 7fbdcd400640 1 --2- 172.21.15.90:0/1054663146 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbda806bbd0 0x7fbda806e090 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc006fd0 tx=0x7fbdbc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:24.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.237+0000 7fbdc5600640 1 -- 172.21.15.90:0/1054663146 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fbdc8145a30 con 0x7fbdc80cdac0 2024-03-24T01:27:24.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.431+0000 7fbdcfe00640 1 -- 172.21.15.90:0/1054663146 --> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbdc80c6dd0 con 0x7fbda806bbd0 2024-03-24T01:27:24.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.449+0000 7fbdc5600640 1 -- 172.21.15.90:0/1054663146 <== mgr.4104 v2:172.21.15.120:6832/860392634 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251961 (secure 0 0 0) 0x7fbdc80c6dd0 con 0x7fbda806bbd0 2024-03-24T01:27:24.454 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:27:24.455 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-24T01:27:24.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.454+0000 7fbdcfe00640 1 -- 172.21.15.90:0/1054663146 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbda806bbd0 msgr2=0x7fbda806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:24.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.454+0000 7fbdcfe00640 1 --2- 172.21.15.90:0/1054663146 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbda806bbd0 0x7fbda806e090 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc006fd0 tx=0x7fbdbc008040 comp rx=0 tx=0).stop 2024-03-24T01:27:24.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.454+0000 7fbdcfe00640 1 -- 172.21.15.90:0/1054663146 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdc80cdac0 msgr2=0x7fbdc80c6890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:24.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.454+0000 7fbdcfe00640 1 --2- 172.21.15.90:0/1054663146 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdc80cdac0 0x7fbdc80c6890 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fbdb402b9e0 tx=0x7fbdb4004730 comp rx=0 tx=0).stop 2024-03-24T01:27:24.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.455+0000 7fbdcfe00640 1 -- 172.21.15.90:0/1054663146 shutdown_connections 2024-03-24T01:27:24.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.455+0000 7fbdcfe00640 1 --2- 172.21.15.90:0/1054663146 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fbda806bbd0 0x7fbda806e090 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:24.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.455+0000 7fbdcfe00640 1 --2- 172.21.15.90:0/1054663146 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdc80cdac0 0x7fbdc80c6890 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:24.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.455+0000 7fbdcfe00640 1 --2- 172.21.15.90:0/1054663146 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fbdc8155030 0x7fbdc80c4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:24.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.455+0000 7fbdcfe00640 1 --2- 172.21.15.90:0/1054663146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbdc806b600 0x7fbdc81454f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:24.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.455+0000 7fbdcfe00640 1 -- 172.21.15.90:0/1054663146 >> 172.21.15.90:0/1054663146 conn(0x7fbdc80b1770 msgr2=0x7fbdc80b0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:24.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.455+0000 7fbdcfe00640 1 -- 172.21.15.90:0/1054663146 shutdown_connections 2024-03-24T01:27:24.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.455+0000 7fbdcfe00640 1 -- 172.21.15.90:0/1054663146 wait complete. 2024-03-24T01:27:24.475 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":148,"stamp":"2024-03-24T01:27:23.551150+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222312,"kb_used_data":6152,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752408,"statfs":{"total":773094113280,"available":772866465792,"internally_reserved":0,"allocated":6299648,"data_stored":3462916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12747,"internal_metadata":221236789},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.037806"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.551588+0000","last_change":"2024-03-24T01:27:14.540765+0000","last_active":"2024-03-24T01:27:15.551588+0000","last_peered":"2024-03-24T01:27:15.551588+0000","last_clean":"2024-03-24T01:27:15.551588+0000","last_became_active":"2024-03-24T01:27:14.539541+0000","last_became_peered":"2024-03-24T01:27:14.539541+0000","last_unstale":"2024-03-24T01:27:15.551588+0000","last_undegraded":"2024-03-24T01:27:15.551588+0000","last_fullsized":"2024-03-24T01:27:15.551588+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:14:16.916597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.565573+0000","last_change":"2024-03-24T01:27:14.559986+0000","last_active":"2024-03-24T01:27:15.565573+0000","last_peered":"2024-03-24T01:27:15.565573+0000","last_clean":"2024-03-24T01:27:15.565573+0000","last_became_active":"2024-03-24T01:27:14.559649+0000","last_became_peered":"2024-03-24T01:27:14.559649+0000","last_unstale":"2024-03-24T01:27:15.565573+0000","last_undegraded":"2024-03-24T01:27:15.565573+0000","last_fullsized":"2024-03-24T01:27:15.565573+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:14:09.820632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.559355+0000","last_change":"2024-03-24T01:27:14.563380+0000","last_active":"2024-03-24T01:27:15.559355+0000","last_peered":"2024-03-24T01:27:15.559355+0000","last_clean":"2024-03-24T01:27:15.559355+0000","last_became_active":"2024-03-24T01:27:14.562140+0000","last_became_peered":"2024-03-24T01:27:14.562140+0000","last_unstale":"2024-03-24T01:27:15.559355+0000","last_undegraded":"2024-03-24T01:27:15.559355+0000","last_fullsized":"2024-03-24T01:27:15.559355+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:18:51.495804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.565418+0000","last_change":"2024-03-24T01:27:14.559476+0000","last_active":"2024-03-24T01:27:15.565418+0000","last_peered":"2024-03-24T01:27:15.565418+0000","last_clean":"2024-03-24T01:27:15.565418+0000","last_became_active":"2024-03-24T01:27:14.559176+0000","last_became_peered":"2024-03-24T01:27:14.559176+0000","last_unstale":"2024-03-24T01:27:15.565418+0000","last_undegraded":"2024-03-24T01:27:15.565418+0000","last_fullsized":"2024-03-24T01:27:15.565418+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:16:39.371501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.976640+0000","last_change":"2024-03-24T01:27:14.559920+0000","last_active":"2024-03-24T01:27:15.976640+0000","last_peered":"2024-03-24T01:27:15.976640+0000","last_clean":"2024-03-24T01:27:15.976640+0000","last_became_active":"2024-03-24T01:27:14.559643+0000","last_became_peered":"2024-03-24T01:27:14.559643+0000","last_unstale":"2024-03-24T01:27:15.976640+0000","last_undegraded":"2024-03-24T01:27:15.976640+0000","last_fullsized":"2024-03-24T01:27:15.976640+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:32.037863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.550937+0000","last_change":"2024-03-24T01:27:14.551494+0000","last_active":"2024-03-24T01:27:15.550937+0000","last_peered":"2024-03-24T01:27:15.550937+0000","last_clean":"2024-03-24T01:27:15.550937+0000","last_became_active":"2024-03-24T01:27:14.551175+0000","last_became_peered":"2024-03-24T01:27:14.551175+0000","last_unstale":"2024-03-24T01:27:15.550937+0000","last_undegraded":"2024-03-24T01:27:15.550937+0000","last_fullsized":"2024-03-24T01:27:15.550937+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:09:58.411238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.558990+0000","last_change":"2024-03-24T01:27:14.552294+0000","last_active":"2024-03-24T01:27:15.558990+0000","last_peered":"2024-03-24T01:27:15.558990+0000","last_clean":"2024-03-24T01:27:15.558990+0000","last_became_active":"2024-03-24T01:27:14.551839+0000","last_became_peered":"2024-03-24T01:27:14.551839+0000","last_unstale":"2024-03-24T01:27:15.558990+0000","last_undegraded":"2024-03-24T01:27:15.558990+0000","last_fullsized":"2024-03-24T01:27:15.558990+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:45:39.507934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.976892+0000","last_change":"2024-03-24T01:27:14.558536+0000","last_active":"2024-03-24T01:27:15.976892+0000","last_peered":"2024-03-24T01:27:15.976892+0000","last_clean":"2024-03-24T01:27:15.976892+0000","last_became_active":"2024-03-24T01:27:14.558211+0000","last_became_peered":"2024-03-24T01:27:14.558211+0000","last_unstale":"2024-03-24T01:27:15.976892+0000","last_undegraded":"2024-03-24T01:27:15.976892+0000","last_fullsized":"2024-03-24T01:27:15.976892+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:26.907326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.978374+0000","last_change":"2024-03-24T01:27:14.567915+0000","last_active":"2024-03-24T01:27:15.978374+0000","last_peered":"2024-03-24T01:27:15.978374+0000","last_clean":"2024-03-24T01:27:15.978374+0000","last_became_active":"2024-03-24T01:27:14.567632+0000","last_became_peered":"2024-03-24T01:27:14.567632+0000","last_unstale":"2024-03-24T01:27:15.978374+0000","last_undegraded":"2024-03-24T01:27:15.978374+0000","last_fullsized":"2024-03-24T01:27:15.978374+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:56:15.426216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.551405+0000","last_change":"2024-03-24T01:27:14.541578+0000","last_active":"2024-03-24T01:27:15.551405+0000","last_peered":"2024-03-24T01:27:15.551405+0000","last_clean":"2024-03-24T01:27:15.551405+0000","last_became_active":"2024-03-24T01:27:14.541289+0000","last_became_peered":"2024-03-24T01:27:14.541289+0000","last_unstale":"2024-03-24T01:27:15.551405+0000","last_undegraded":"2024-03-24T01:27:15.551405+0000","last_fullsized":"2024-03-24T01:27:15.551405+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:46:45.759885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.976758+0000","last_change":"2024-03-24T01:27:14.557130+0000","last_active":"2024-03-24T01:27:15.976758+0000","last_peered":"2024-03-24T01:27:15.976758+0000","last_clean":"2024-03-24T01:27:15.976758+0000","last_became_active":"2024-03-24T01:27:14.556873+0000","last_became_peered":"2024-03-24T01:27:14.556873+0000","last_unstale":"2024-03-24T01:27:15.976758+0000","last_undegraded":"2024-03-24T01:27:15.976758+0000","last_fullsized":"2024-03-24T01:27:15.976758+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:01:39.858336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.977838+0000","last_change":"2024-03-24T01:27:14.552072+0000","last_active":"2024-03-24T01:27:15.977838+0000","last_peered":"2024-03-24T01:27:15.977838+0000","last_clean":"2024-03-24T01:27:15.977838+0000","last_became_active":"2024-03-24T01:27:14.550084+0000","last_became_peered":"2024-03-24T01:27:14.550084+0000","last_unstale":"2024-03-24T01:27:15.977838+0000","last_undegraded":"2024-03-24T01:27:15.977838+0000","last_fullsized":"2024-03-24T01:27:15.977838+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:54:52.883656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.550994+0000","last_change":"2024-03-24T01:27:14.544185+0000","last_active":"2024-03-24T01:27:15.550994+0000","last_peered":"2024-03-24T01:27:15.550994+0000","last_clean":"2024-03-24T01:27:15.550994+0000","last_became_active":"2024-03-24T01:27:14.543895+0000","last_became_peered":"2024-03-24T01:27:14.543895+0000","last_unstale":"2024-03-24T01:27:15.550994+0000","last_undegraded":"2024-03-24T01:27:15.550994+0000","last_fullsized":"2024-03-24T01:27:15.550994+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:01:09.219193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.558941+0000","last_change":"2024-03-24T01:27:14.550120+0000","last_active":"2024-03-24T01:27:15.558941+0000","last_peered":"2024-03-24T01:27:15.558941+0000","last_clean":"2024-03-24T01:27:15.558941+0000","last_became_active":"2024-03-24T01:27:14.549329+0000","last_became_peered":"2024-03-24T01:27:14.549329+0000","last_unstale":"2024-03-24T01:27:15.558941+0000","last_undegraded":"2024-03-24T01:27:15.558941+0000","last_fullsized":"2024-03-24T01:27:15.558941+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:47:57.893749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.564927+0000","last_change":"2024-03-24T01:27:14.557855+0000","last_active":"2024-03-24T01:27:15.564927+0000","last_peered":"2024-03-24T01:27:15.564927+0000","last_clean":"2024-03-24T01:27:15.564927+0000","last_became_active":"2024-03-24T01:27:14.557606+0000","last_became_peered":"2024-03-24T01:27:14.557606+0000","last_unstale":"2024-03-24T01:27:15.564927+0000","last_undegraded":"2024-03-24T01:27:15.564927+0000","last_fullsized":"2024-03-24T01:27:15.564927+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:46:45.759885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.550251+0000","last_change":"2024-03-24T01:27:14.542073+0000","last_active":"2024-03-24T01:27:15.550251+0000","last_peered":"2024-03-24T01:27:15.550251+0000","last_clean":"2024-03-24T01:27:15.550251+0000","last_became_active":"2024-03-24T01:27:14.540636+0000","last_became_peered":"2024-03-24T01:27:14.540636+0000","last_unstale":"2024-03-24T01:27:15.550251+0000","last_undegraded":"2024-03-24T01:27:15.550251+0000","last_fullsized":"2024-03-24T01:27:15.550251+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:32.037863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.560254+0000","last_change":"2024-03-24T01:27:14.556042+0000","last_active":"2024-03-24T01:27:15.560254+0000","last_peered":"2024-03-24T01:27:15.560254+0000","last_clean":"2024-03-24T01:27:15.560254+0000","last_became_active":"2024-03-24T01:27:14.555743+0000","last_became_peered":"2024-03-24T01:27:14.555743+0000","last_unstale":"2024-03-24T01:27:15.560254+0000","last_undegraded":"2024-03-24T01:27:15.560254+0000","last_fullsized":"2024-03-24T01:27:15.560254+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:07:29.856046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.565648+0000","last_change":"2024-03-24T01:27:14.561148+0000","last_active":"2024-03-24T01:27:15.565648+0000","last_peered":"2024-03-24T01:27:15.565648+0000","last_clean":"2024-03-24T01:27:15.565648+0000","last_became_active":"2024-03-24T01:27:14.560509+0000","last_became_peered":"2024-03-24T01:27:14.560509+0000","last_unstale":"2024-03-24T01:27:15.565648+0000","last_undegraded":"2024-03-24T01:27:15.565648+0000","last_fullsized":"2024-03-24T01:27:15.565648+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:06.757918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.559561+0000","last_change":"2024-03-24T01:27:14.553533+0000","last_active":"2024-03-24T01:27:15.559561+0000","last_peered":"2024-03-24T01:27:15.559561+0000","last_clean":"2024-03-24T01:27:15.559561+0000","last_became_active":"2024-03-24T01:27:14.553252+0000","last_became_peered":"2024-03-24T01:27:14.553252+0000","last_unstale":"2024-03-24T01:27:15.559561+0000","last_undegraded":"2024-03-24T01:27:15.559561+0000","last_fullsized":"2024-03-24T01:27:15.559561+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:59:27.554575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.551503+0000","last_change":"2024-03-24T01:27:14.541674+0000","last_active":"2024-03-24T01:27:15.551503+0000","last_peered":"2024-03-24T01:27:15.551503+0000","last_clean":"2024-03-24T01:27:15.551503+0000","last_became_active":"2024-03-24T01:27:14.541300+0000","last_became_peered":"2024-03-24T01:27:14.541300+0000","last_unstale":"2024-03-24T01:27:15.551503+0000","last_undegraded":"2024-03-24T01:27:15.551503+0000","last_fullsized":"2024-03-24T01:27:15.551503+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:32:58.788010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.551344+0000","last_change":"2024-03-24T01:27:14.560160+0000","last_active":"2024-03-24T01:27:15.551344+0000","last_peered":"2024-03-24T01:27:15.551344+0000","last_clean":"2024-03-24T01:27:15.551344+0000","last_became_active":"2024-03-24T01:27:14.559765+0000","last_became_peered":"2024-03-24T01:27:14.559765+0000","last_unstale":"2024-03-24T01:27:15.551344+0000","last_undegraded":"2024-03-24T01:27:15.551344+0000","last_fullsized":"2024-03-24T01:27:15.551344+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:32.037863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.976701+0000","last_change":"2024-03-24T01:27:14.554032+0000","last_active":"2024-03-24T01:27:15.976701+0000","last_peered":"2024-03-24T01:27:15.976701+0000","last_clean":"2024-03-24T01:27:15.976701+0000","last_became_active":"2024-03-24T01:27:14.553732+0000","last_became_peered":"2024-03-24T01:27:14.553732+0000","last_unstale":"2024-03-24T01:27:15.976701+0000","last_undegraded":"2024-03-24T01:27:15.976701+0000","last_fullsized":"2024-03-24T01:27:15.976701+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:55:30.524954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.975732+0000","last_change":"2024-03-24T01:27:14.551791+0000","last_active":"2024-03-24T01:27:15.975732+0000","last_peered":"2024-03-24T01:27:15.975732+0000","last_clean":"2024-03-24T01:27:15.975732+0000","last_became_active":"2024-03-24T01:27:14.551503+0000","last_became_peered":"2024-03-24T01:27:14.551503+0000","last_unstale":"2024-03-24T01:27:15.975732+0000","last_undegraded":"2024-03-24T01:27:15.975732+0000","last_fullsized":"2024-03-24T01:27:15.975732+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:30:34.008161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.558990+0000","last_change":"2024-03-24T01:27:14.545843+0000","last_active":"2024-03-24T01:27:15.558990+0000","last_peered":"2024-03-24T01:27:15.558990+0000","last_clean":"2024-03-24T01:27:15.558990+0000","last_became_active":"2024-03-24T01:27:14.545534+0000","last_became_peered":"2024-03-24T01:27:14.545534+0000","last_unstale":"2024-03-24T01:27:15.558990+0000","last_undegraded":"2024-03-24T01:27:15.558990+0000","last_fullsized":"2024-03-24T01:27:15.558990+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:36.294802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.560455+0000","last_change":"2024-03-24T01:27:14.548640+0000","last_active":"2024-03-24T01:27:15.560455+0000","last_peered":"2024-03-24T01:27:15.560455+0000","last_clean":"2024-03-24T01:27:15.560455+0000","last_became_active":"2024-03-24T01:27:14.548364+0000","last_became_peered":"2024-03-24T01:27:14.548364+0000","last_unstale":"2024-03-24T01:27:15.560455+0000","last_undegraded":"2024-03-24T01:27:15.560455+0000","last_fullsized":"2024-03-24T01:27:15.560455+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:46:50.633787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.977991+0000","last_change":"2024-03-24T01:27:14.541461+0000","last_active":"2024-03-24T01:27:15.977991+0000","last_peered":"2024-03-24T01:27:15.977991+0000","last_clean":"2024-03-24T01:27:15.977991+0000","last_became_active":"2024-03-24T01:27:14.541156+0000","last_became_peered":"2024-03-24T01:27:14.541156+0000","last_unstale":"2024-03-24T01:27:15.977991+0000","last_undegraded":"2024-03-24T01:27:15.977991+0000","last_fullsized":"2024-03-24T01:27:15.977991+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:46:50.633787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.551880+0000","last_change":"2024-03-24T01:27:14.556242+0000","last_active":"2024-03-24T01:27:15.551880+0000","last_peered":"2024-03-24T01:27:15.551880+0000","last_clean":"2024-03-24T01:27:15.551880+0000","last_became_active":"2024-03-24T01:27:14.552642+0000","last_became_peered":"2024-03-24T01:27:14.552642+0000","last_unstale":"2024-03-24T01:27:15.551880+0000","last_undegraded":"2024-03-24T01:27:15.551880+0000","last_fullsized":"2024-03-24T01:27:15.551880+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:16:39.371501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.551665+0000","last_change":"2024-03-24T01:27:14.542335+0000","last_active":"2024-03-24T01:27:15.551665+0000","last_peered":"2024-03-24T01:27:15.551665+0000","last_clean":"2024-03-24T01:27:15.551665+0000","last_became_active":"2024-03-24T01:27:14.541006+0000","last_became_peered":"2024-03-24T01:27:14.541006+0000","last_unstale":"2024-03-24T01:27:15.551665+0000","last_undegraded":"2024-03-24T01:27:15.551665+0000","last_fullsized":"2024-03-24T01:27:15.551665+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:23.971971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.559454+0000","last_change":"2024-03-24T01:27:14.550220+0000","last_active":"2024-03-24T01:27:15.559454+0000","last_peered":"2024-03-24T01:27:15.559454+0000","last_clean":"2024-03-24T01:27:15.559454+0000","last_became_active":"2024-03-24T01:27:14.549812+0000","last_became_peered":"2024-03-24T01:27:14.549812+0000","last_unstale":"2024-03-24T01:27:15.559454+0000","last_undegraded":"2024-03-24T01:27:15.559454+0000","last_fullsized":"2024-03-24T01:27:15.559454+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:12:03.363072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.565209+0000","last_change":"2024-03-24T01:27:14.556235+0000","last_active":"2024-03-24T01:27:15.565209+0000","last_peered":"2024-03-24T01:27:15.565209+0000","last_clean":"2024-03-24T01:27:15.565209+0000","last_became_active":"2024-03-24T01:27:14.555888+0000","last_became_peered":"2024-03-24T01:27:14.555888+0000","last_unstale":"2024-03-24T01:27:15.565209+0000","last_undegraded":"2024-03-24T01:27:15.565209+0000","last_fullsized":"2024-03-24T01:27:15.565209+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:14:16.916597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.559206+0000","last_change":"2024-03-24T01:27:14.552212+0000","last_active":"2024-03-24T01:27:15.559206+0000","last_peered":"2024-03-24T01:27:15.559206+0000","last_clean":"2024-03-24T01:27:15.559206+0000","last_became_active":"2024-03-24T01:27:14.551775+0000","last_became_peered":"2024-03-24T01:27:14.551775+0000","last_unstale":"2024-03-24T01:27:15.559206+0000","last_undegraded":"2024-03-24T01:27:15.559206+0000","last_fullsized":"2024-03-24T01:27:15.559206+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:44:39.116139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.550458+0000","last_change":"2024-03-24T01:27:14.542056+0000","last_active":"2024-03-24T01:27:15.550458+0000","last_peered":"2024-03-24T01:27:15.550458+0000","last_clean":"2024-03-24T01:27:15.550458+0000","last_became_active":"2024-03-24T01:27:14.541695+0000","last_became_peered":"2024-03-24T01:27:14.541695+0000","last_unstale":"2024-03-24T01:27:15.550458+0000","last_undegraded":"2024-03-24T01:27:15.550458+0000","last_fullsized":"2024-03-24T01:27:15.550458+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:46:45.759885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.564818+0000","last_change":"2024-03-24T01:27:14.559666+0000","last_active":"2024-03-24T01:27:15.564818+0000","last_peered":"2024-03-24T01:27:15.564818+0000","last_clean":"2024-03-24T01:27:15.564818+0000","last_became_active":"2024-03-24T01:27:14.559356+0000","last_became_peered":"2024-03-24T01:27:14.559356+0000","last_unstale":"2024-03-24T01:27:15.564818+0000","last_undegraded":"2024-03-24T01:27:15.564818+0000","last_fullsized":"2024-03-24T01:27:15.564818+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:58.630025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"52'4","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.551878+0000","last_change":"2024-03-24T01:27:13.522918+0000","last_active":"2024-03-24T01:27:15.551878+0000","last_peered":"2024-03-24T01:27:15.551878+0000","last_clean":"2024-03-24T01:27:15.551878+0000","last_became_active":"2024-03-24T01:27:13.520866+0000","last_became_peered":"2024-03-24T01:27:13.520866+0000","last_unstale":"2024-03-24T01:27:15.551878+0000","last_undegraded":"2024-03-24T01:27:15.551878+0000","last_fullsized":"2024-03-24T01:27:15.551878+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:30.183145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.551748+0000","last_change":"2024-03-24T01:27:14.551119+0000","last_active":"2024-03-24T01:27:15.551748+0000","last_peered":"2024-03-24T01:27:15.551748+0000","last_clean":"2024-03-24T01:27:15.551748+0000","last_became_active":"2024-03-24T01:27:14.550800+0000","last_became_peered":"2024-03-24T01:27:14.550800+0000","last_unstale":"2024-03-24T01:27:15.551748+0000","last_undegraded":"2024-03-24T01:27:15.551748+0000","last_fullsized":"2024-03-24T01:27:15.551748+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:16:39.371501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"52'1","reported_seq":15,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.551448+0000","last_change":"2024-03-24T01:27:13.531689+0000","last_active":"2024-03-24T01:27:15.551448+0000","last_peered":"2024-03-24T01:27:15.551448+0000","last_clean":"2024-03-24T01:27:15.551448+0000","last_became_active":"2024-03-24T01:27:13.530929+0000","last_became_peered":"2024-03-24T01:27:13.530929+0000","last_unstale":"2024-03-24T01:27:15.551448+0000","last_undegraded":"2024-03-24T01:27:15.551448+0000","last_fullsized":"2024-03-24T01:27:15.551448+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:00:24.458371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.551321+0000","last_change":"2024-03-24T01:27:14.552432+0000","last_active":"2024-03-24T01:27:15.551321+0000","last_peered":"2024-03-24T01:27:15.551321+0000","last_clean":"2024-03-24T01:27:15.551321+0000","last_became_active":"2024-03-24T01:27:14.551697+0000","last_became_peered":"2024-03-24T01:27:14.551697+0000","last_unstale":"2024-03-24T01:27:15.551321+0000","last_undegraded":"2024-03-24T01:27:15.551321+0000","last_fullsized":"2024-03-24T01:27:15.551321+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:03.448771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"52'2","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.551987+0000","last_change":"2024-03-24T01:27:13.528068+0000","last_active":"2024-03-24T01:27:15.551987+0000","last_peered":"2024-03-24T01:27:15.551987+0000","last_clean":"2024-03-24T01:27:15.551987+0000","last_became_active":"2024-03-24T01:27:13.527750+0000","last_became_peered":"2024-03-24T01:27:13.527750+0000","last_unstale":"2024-03-24T01:27:15.551987+0000","last_undegraded":"2024-03-24T01:27:15.551987+0000","last_fullsized":"2024-03-24T01:27:15.551987+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:37:15.291837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.976635+0000","last_change":"2024-03-24T01:27:14.548464+0000","last_active":"2024-03-24T01:27:15.976635+0000","last_peered":"2024-03-24T01:27:15.976635+0000","last_clean":"2024-03-24T01:27:15.976635+0000","last_became_active":"2024-03-24T01:27:14.548113+0000","last_became_peered":"2024-03-24T01:27:14.548113+0000","last_unstale":"2024-03-24T01:27:15.976635+0000","last_undegraded":"2024-03-24T01:27:15.976635+0000","last_fullsized":"2024-03-24T01:27:15.976635+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:22:28.066386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"52'2","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.552578+0000","last_change":"2024-03-24T01:27:13.527296+0000","last_active":"2024-03-24T01:27:15.552578+0000","last_peered":"2024-03-24T01:27:15.552578+0000","last_clean":"2024-03-24T01:27:15.552578+0000","last_became_active":"2024-03-24T01:27:13.526656+0000","last_became_peered":"2024-03-24T01:27:13.526656+0000","last_unstale":"2024-03-24T01:27:15.552578+0000","last_undegraded":"2024-03-24T01:27:15.552578+0000","last_fullsized":"2024-03-24T01:27:15.552578+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:28:35.013061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.976760+0000","last_change":"2024-03-24T01:27:14.555039+0000","last_active":"2024-03-24T01:27:15.976760+0000","last_peered":"2024-03-24T01:27:15.976760+0000","last_clean":"2024-03-24T01:27:15.976760+0000","last_became_active":"2024-03-24T01:27:14.554761+0000","last_became_peered":"2024-03-24T01:27:14.554761+0000","last_unstale":"2024-03-24T01:27:15.976760+0000","last_undegraded":"2024-03-24T01:27:15.976760+0000","last_fullsized":"2024-03-24T01:27:15.976760+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:23.971971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"52'1","reported_seq":15,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.559427+0000","last_change":"2024-03-24T01:27:13.518516+0000","last_active":"2024-03-24T01:27:15.559427+0000","last_peered":"2024-03-24T01:27:15.559427+0000","last_clean":"2024-03-24T01:27:15.559427+0000","last_became_active":"2024-03-24T01:27:13.518276+0000","last_became_peered":"2024-03-24T01:27:13.518276+0000","last_unstale":"2024-03-24T01:27:15.559427+0000","last_undegraded":"2024-03-24T01:27:15.559427+0000","last_fullsized":"2024-03-24T01:27:15.559427+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:39.632343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.565413+0000","last_change":"2024-03-24T01:27:14.559916+0000","last_active":"2024-03-24T01:27:15.565413+0000","last_peered":"2024-03-24T01:27:15.565413+0000","last_clean":"2024-03-24T01:27:15.565413+0000","last_became_active":"2024-03-24T01:27:14.559597+0000","last_became_peered":"2024-03-24T01:27:14.559597+0000","last_unstale":"2024-03-24T01:27:15.565413+0000","last_undegraded":"2024-03-24T01:27:15.565413+0000","last_fullsized":"2024-03-24T01:27:15.565413+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:02:42.596744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"52'1","reported_seq":15,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.975977+0000","last_change":"2024-03-24T01:27:13.527482+0000","last_active":"2024-03-24T01:27:15.975977+0000","last_peered":"2024-03-24T01:27:15.975977+0000","last_clean":"2024-03-24T01:27:15.975977+0000","last_became_active":"2024-03-24T01:27:13.527205+0000","last_became_peered":"2024-03-24T01:27:13.527205+0000","last_unstale":"2024-03-24T01:27:15.975977+0000","last_undegraded":"2024-03-24T01:27:15.975977+0000","last_fullsized":"2024-03-24T01:27:15.975977+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:39.632343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.978359+0000","last_change":"2024-03-24T01:27:14.541241+0000","last_active":"2024-03-24T01:27:15.978359+0000","last_peered":"2024-03-24T01:27:15.978359+0000","last_clean":"2024-03-24T01:27:15.978359+0000","last_became_active":"2024-03-24T01:27:14.540872+0000","last_became_peered":"2024-03-24T01:27:14.540872+0000","last_unstale":"2024-03-24T01:27:15.978359+0000","last_undegraded":"2024-03-24T01:27:15.978359+0000","last_fullsized":"2024-03-24T01:27:15.978359+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:01:40.597597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.550808+0000","last_change":"2024-03-24T01:27:13.531379+0000","last_active":"2024-03-24T01:27:15.550808+0000","last_peered":"2024-03-24T01:27:15.550808+0000","last_clean":"2024-03-24T01:27:15.550808+0000","last_became_active":"2024-03-24T01:27:13.530618+0000","last_became_peered":"2024-03-24T01:27:13.530618+0000","last_unstale":"2024-03-24T01:27:15.550808+0000","last_undegraded":"2024-03-24T01:27:15.550808+0000","last_fullsized":"2024-03-24T01:27:15.550808+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:00.227179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.565602+0000","last_change":"2024-03-24T01:27:14.570633+0000","last_active":"2024-03-24T01:27:15.565602+0000","last_peered":"2024-03-24T01:27:15.565602+0000","last_clean":"2024-03-24T01:27:15.565602+0000","last_became_active":"2024-03-24T01:27:14.570275+0000","last_became_peered":"2024-03-24T01:27:14.570275+0000","last_unstale":"2024-03-24T01:27:15.565602+0000","last_undegraded":"2024-03-24T01:27:15.565602+0000","last_fullsized":"2024-03-24T01:27:15.565602+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:19:18.467994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.558905+0000","last_change":"2024-03-24T01:27:13.529478+0000","last_active":"2024-03-24T01:27:15.558905+0000","last_peered":"2024-03-24T01:27:15.558905+0000","last_clean":"2024-03-24T01:27:15.558905+0000","last_became_active":"2024-03-24T01:27:13.529197+0000","last_became_peered":"2024-03-24T01:27:13.529197+0000","last_unstale":"2024-03-24T01:27:15.558905+0000","last_undegraded":"2024-03-24T01:27:15.558905+0000","last_fullsized":"2024-03-24T01:27:15.558905+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:30:49.041062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.976109+0000","last_change":"2024-03-24T01:27:14.544605+0000","last_active":"2024-03-24T01:27:15.976109+0000","last_peered":"2024-03-24T01:27:15.976109+0000","last_clean":"2024-03-24T01:27:15.976109+0000","last_became_active":"2024-03-24T01:27:14.544189+0000","last_became_peered":"2024-03-24T01:27:14.544189+0000","last_unstale":"2024-03-24T01:27:15.976109+0000","last_undegraded":"2024-03-24T01:27:15.976109+0000","last_fullsized":"2024-03-24T01:27:15.976109+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:28:36.027521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.565690+0000","last_change":"2024-03-24T01:27:13.533618+0000","last_active":"2024-03-24T01:27:15.565690+0000","last_peered":"2024-03-24T01:27:15.565690+0000","last_clean":"2024-03-24T01:27:15.565690+0000","last_became_active":"2024-03-24T01:27:13.531829+0000","last_became_peered":"2024-03-24T01:27:13.531829+0000","last_unstale":"2024-03-24T01:27:15.565690+0000","last_undegraded":"2024-03-24T01:27:15.565690+0000","last_fullsized":"2024-03-24T01:27:15.565690+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:06:33.248202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.551618+0000","last_change":"2024-03-24T01:27:14.541924+0000","last_active":"2024-03-24T01:27:15.551618+0000","last_peered":"2024-03-24T01:27:15.551618+0000","last_clean":"2024-03-24T01:27:15.551618+0000","last_became_active":"2024-03-24T01:27:14.541512+0000","last_became_peered":"2024-03-24T01:27:14.541512+0000","last_unstale":"2024-03-24T01:27:15.551618+0000","last_undegraded":"2024-03-24T01:27:15.551618+0000","last_fullsized":"2024-03-24T01:27:15.551618+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:02:42.596744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.976704+0000","last_change":"2024-03-24T01:27:13.524033+0000","last_active":"2024-03-24T01:27:15.976704+0000","last_peered":"2024-03-24T01:27:15.976704+0000","last_clean":"2024-03-24T01:27:15.976704+0000","last_became_active":"2024-03-24T01:27:13.523502+0000","last_became_peered":"2024-03-24T01:27:13.523502+0000","last_unstale":"2024-03-24T01:27:15.976704+0000","last_undegraded":"2024-03-24T01:27:15.976704+0000","last_fullsized":"2024-03-24T01:27:15.976704+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:02:11.823211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.559097+0000","last_change":"2024-03-24T01:27:14.552947+0000","last_active":"2024-03-24T01:27:15.559097+0000","last_peered":"2024-03-24T01:27:15.559097+0000","last_clean":"2024-03-24T01:27:15.559097+0000","last_became_active":"2024-03-24T01:27:14.552583+0000","last_became_peered":"2024-03-24T01:27:14.552583+0000","last_unstale":"2024-03-24T01:27:15.559097+0000","last_undegraded":"2024-03-24T01:27:15.559097+0000","last_fullsized":"2024-03-24T01:27:15.559097+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:04:45.650427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"52'3","reported_seq":17,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.565679+0000","last_change":"2024-03-24T01:27:13.530329+0000","last_active":"2024-03-24T01:27:15.565679+0000","last_peered":"2024-03-24T01:27:15.565679+0000","last_clean":"2024-03-24T01:27:15.565679+0000","last_became_active":"2024-03-24T01:27:13.529972+0000","last_became_peered":"2024-03-24T01:27:13.529972+0000","last_unstale":"2024-03-24T01:27:15.565679+0000","last_undegraded":"2024-03-24T01:27:15.565679+0000","last_fullsized":"2024-03-24T01:27:15.565679+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:43.325731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.976669+0000","last_change":"2024-03-24T01:27:14.552486+0000","last_active":"2024-03-24T01:27:15.976669+0000","last_peered":"2024-03-24T01:27:15.976669+0000","last_clean":"2024-03-24T01:27:15.976669+0000","last_became_active":"2024-03-24T01:27:14.551883+0000","last_became_peered":"2024-03-24T01:27:14.551883+0000","last_unstale":"2024-03-24T01:27:15.976669+0000","last_undegraded":"2024-03-24T01:27:15.976669+0000","last_fullsized":"2024-03-24T01:27:15.976669+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:32:58.788010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"52'6","reported_seq":20,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.977915+0000","last_change":"2024-03-24T01:27:13.521128+0000","last_active":"2024-03-24T01:27:15.977915+0000","last_peered":"2024-03-24T01:27:15.977915+0000","last_clean":"2024-03-24T01:27:15.977915+0000","last_became_active":"2024-03-24T01:27:13.520358+0000","last_became_peered":"2024-03-24T01:27:13.520358+0000","last_unstale":"2024-03-24T01:27:15.977915+0000","last_undegraded":"2024-03-24T01:27:15.977915+0000","last_fullsized":"2024-03-24T01:27:15.977915+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:01:08.204732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.552270+0000","last_change":"2024-03-24T01:27:14.561717+0000","last_active":"2024-03-24T01:27:15.552270+0000","last_peered":"2024-03-24T01:27:15.552270+0000","last_clean":"2024-03-24T01:27:15.552270+0000","last_became_active":"2024-03-24T01:27:14.561448+0000","last_became_peered":"2024-03-24T01:27:14.561448+0000","last_unstale":"2024-03-24T01:27:15.552270+0000","last_undegraded":"2024-03-24T01:27:15.552270+0000","last_fullsized":"2024-03-24T01:27:15.552270+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:14:09.820632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"52'2","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.559687+0000","last_change":"2024-03-24T01:27:13.532029+0000","last_active":"2024-03-24T01:27:15.559687+0000","last_peered":"2024-03-24T01:27:15.559687+0000","last_clean":"2024-03-24T01:27:15.559687+0000","last_became_active":"2024-03-24T01:27:13.531765+0000","last_became_peered":"2024-03-24T01:27:13.531765+0000","last_unstale":"2024-03-24T01:27:15.559687+0000","last_undegraded":"2024-03-24T01:27:15.559687+0000","last_fullsized":"2024-03-24T01:27:15.559687+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:02:11.823211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.559718+0000","last_change":"2024-03-24T01:27:14.563597+0000","last_active":"2024-03-24T01:27:15.559718+0000","last_peered":"2024-03-24T01:27:15.559718+0000","last_clean":"2024-03-24T01:27:15.559718+0000","last_became_active":"2024-03-24T01:27:14.563117+0000","last_became_peered":"2024-03-24T01:27:14.563117+0000","last_unstale":"2024-03-24T01:27:15.559718+0000","last_undegraded":"2024-03-24T01:27:15.559718+0000","last_fullsized":"2024-03-24T01:27:15.559718+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:22:28.066386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.551474+0000","last_change":"2024-03-24T01:27:13.530136+0000","last_active":"2024-03-24T01:27:15.551474+0000","last_peered":"2024-03-24T01:27:15.551474+0000","last_clean":"2024-03-24T01:27:15.551474+0000","last_became_active":"2024-03-24T01:27:13.529811+0000","last_became_peered":"2024-03-24T01:27:13.529811+0000","last_unstale":"2024-03-24T01:27:15.551474+0000","last_undegraded":"2024-03-24T01:27:15.551474+0000","last_fullsized":"2024-03-24T01:27:15.551474+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:37:15.291837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.978157+0000","last_change":"2024-03-24T01:27:14.561422+0000","last_active":"2024-03-24T01:27:15.978157+0000","last_peered":"2024-03-24T01:27:15.978157+0000","last_clean":"2024-03-24T01:27:15.978157+0000","last_became_active":"2024-03-24T01:27:14.559580+0000","last_became_peered":"2024-03-24T01:27:14.559580+0000","last_unstale":"2024-03-24T01:27:15.978157+0000","last_undegraded":"2024-03-24T01:27:15.978157+0000","last_fullsized":"2024-03-24T01:27:15.978157+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:11:33.733576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.565118+0000","last_change":"2024-03-24T01:27:13.532579+0000","last_active":"2024-03-24T01:27:15.565118+0000","last_peered":"2024-03-24T01:27:15.565118+0000","last_clean":"2024-03-24T01:27:15.565118+0000","last_became_active":"2024-03-24T01:27:13.532264+0000","last_became_peered":"2024-03-24T01:27:13.532264+0000","last_unstale":"2024-03-24T01:27:15.565118+0000","last_undegraded":"2024-03-24T01:27:15.565118+0000","last_fullsized":"2024-03-24T01:27:15.565118+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:12:14.260140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.560220+0000","last_change":"2024-03-24T01:27:14.567679+0000","last_active":"2024-03-24T01:27:15.560220+0000","last_peered":"2024-03-24T01:27:15.560220+0000","last_clean":"2024-03-24T01:27:15.560220+0000","last_became_active":"2024-03-24T01:27:14.566777+0000","last_became_peered":"2024-03-24T01:27:14.566777+0000","last_unstale":"2024-03-24T01:27:15.560220+0000","last_undegraded":"2024-03-24T01:27:15.560220+0000","last_fullsized":"2024-03-24T01:27:15.560220+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:55:01.380185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"52'8","reported_seq":22,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.977159+0000","last_change":"2024-03-24T01:27:13.518331+0000","last_active":"2024-03-24T01:27:15.977159+0000","last_peered":"2024-03-24T01:27:15.977159+0000","last_clean":"2024-03-24T01:27:15.977159+0000","last_became_active":"2024-03-24T01:27:13.518017+0000","last_became_peered":"2024-03-24T01:27:13.518017+0000","last_unstale":"2024-03-24T01:27:15.977159+0000","last_undegraded":"2024-03-24T01:27:15.977159+0000","last_fullsized":"2024-03-24T01:27:15.977159+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:02:41.582283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.977051+0000","last_change":"2024-03-24T01:27:14.543017+0000","last_active":"2024-03-24T01:27:15.977051+0000","last_peered":"2024-03-24T01:27:15.977051+0000","last_clean":"2024-03-24T01:27:15.977051+0000","last_became_active":"2024-03-24T01:27:14.542730+0000","last_became_peered":"2024-03-24T01:27:14.542730+0000","last_unstale":"2024-03-24T01:27:15.977051+0000","last_undegraded":"2024-03-24T01:27:15.977051+0000","last_fullsized":"2024-03-24T01:27:15.977051+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:55:01.380185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"52'1","reported_seq":15,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.560453+0000","last_change":"2024-03-24T01:27:13.530251+0000","last_active":"2024-03-24T01:27:15.560453+0000","last_peered":"2024-03-24T01:27:15.560453+0000","last_clean":"2024-03-24T01:27:15.560453+0000","last_became_active":"2024-03-24T01:27:13.529767+0000","last_became_peered":"2024-03-24T01:27:13.529767+0000","last_unstale":"2024-03-24T01:27:15.560453+0000","last_undegraded":"2024-03-24T01:27:15.560453+0000","last_fullsized":"2024-03-24T01:27:15.560453+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:43:44.823891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.550523+0000","last_change":"2024-03-24T01:27:14.554474+0000","last_active":"2024-03-24T01:27:15.550523+0000","last_peered":"2024-03-24T01:27:15.550523+0000","last_clean":"2024-03-24T01:27:15.550523+0000","last_became_active":"2024-03-24T01:27:14.554189+0000","last_became_peered":"2024-03-24T01:27:14.554189+0000","last_unstale":"2024-03-24T01:27:15.550523+0000","last_undegraded":"2024-03-24T01:27:15.550523+0000","last_fullsized":"2024-03-24T01:27:15.550523+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:32:58.788010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"52'2","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.977103+0000","last_change":"2024-03-24T01:27:13.526531+0000","last_active":"2024-03-24T01:27:15.977103+0000","last_peered":"2024-03-24T01:27:15.977103+0000","last_clean":"2024-03-24T01:27:15.977103+0000","last_became_active":"2024-03-24T01:27:13.526212+0000","last_became_peered":"2024-03-24T01:27:13.526212+0000","last_unstale":"2024-03-24T01:27:15.977103+0000","last_undegraded":"2024-03-24T01:27:15.977103+0000","last_fullsized":"2024-03-24T01:27:15.977103+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:02:41.656569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.976334+0000","last_change":"2024-03-24T01:27:14.567932+0000","last_active":"2024-03-24T01:27:15.976334+0000","last_peered":"2024-03-24T01:27:15.976334+0000","last_clean":"2024-03-24T01:27:15.976334+0000","last_became_active":"2024-03-24T01:27:14.567624+0000","last_became_peered":"2024-03-24T01:27:14.567624+0000","last_unstale":"2024-03-24T01:27:15.976334+0000","last_undegraded":"2024-03-24T01:27:15.976334+0000","last_fullsized":"2024-03-24T01:27:15.976334+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:03.448771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"52'1","reported_seq":15,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.551948+0000","last_change":"2024-03-24T01:27:13.526615+0000","last_active":"2024-03-24T01:27:15.551948+0000","last_peered":"2024-03-24T01:27:15.551948+0000","last_clean":"2024-03-24T01:27:15.551948+0000","last_became_active":"2024-03-24T01:27:13.526286+0000","last_became_peered":"2024-03-24T01:27:13.526286+0000","last_unstale":"2024-03-24T01:27:15.551948+0000","last_undegraded":"2024-03-24T01:27:15.551948+0000","last_fullsized":"2024-03-24T01:27:15.551948+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:42:00.908936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.550995+0000","last_change":"2024-03-24T01:27:14.539239+0000","last_active":"2024-03-24T01:27:15.550995+0000","last_peered":"2024-03-24T01:27:15.550995+0000","last_clean":"2024-03-24T01:27:15.550995+0000","last_became_active":"2024-03-24T01:27:14.538525+0000","last_became_peered":"2024-03-24T01:27:14.538525+0000","last_unstale":"2024-03-24T01:27:15.550995+0000","last_undegraded":"2024-03-24T01:27:15.550995+0000","last_fullsized":"2024-03-24T01:27:15.550995+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:58.630025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.551723+0000","last_change":"2024-03-24T01:27:13.523480+0000","last_active":"2024-03-24T01:27:15.551723+0000","last_peered":"2024-03-24T01:27:15.551723+0000","last_clean":"2024-03-24T01:27:15.551723+0000","last_became_active":"2024-03-24T01:27:13.523203+0000","last_became_peered":"2024-03-24T01:27:13.523203+0000","last_unstale":"2024-03-24T01:27:15.551723+0000","last_undegraded":"2024-03-24T01:27:15.551723+0000","last_fullsized":"2024-03-24T01:27:15.551723+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:00:24.458371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.552605+0000","last_change":"2024-03-24T01:27:14.549953+0000","last_active":"2024-03-24T01:27:15.552605+0000","last_peered":"2024-03-24T01:27:15.552605+0000","last_clean":"2024-03-24T01:27:15.552605+0000","last_became_active":"2024-03-24T01:27:14.549616+0000","last_became_peered":"2024-03-24T01:27:14.549616+0000","last_unstale":"2024-03-24T01:27:15.552605+0000","last_undegraded":"2024-03-24T01:27:15.552605+0000","last_fullsized":"2024-03-24T01:27:15.552605+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:06.757918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"52'2","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.551171+0000","last_change":"2024-03-24T01:27:13.526041+0000","last_active":"2024-03-24T01:27:15.551171+0000","last_peered":"2024-03-24T01:27:15.551171+0000","last_clean":"2024-03-24T01:27:15.551171+0000","last_became_active":"2024-03-24T01:27:13.525587+0000","last_became_peered":"2024-03-24T01:27:13.525587+0000","last_unstale":"2024-03-24T01:27:15.551171+0000","last_undegraded":"2024-03-24T01:27:15.551171+0000","last_fullsized":"2024-03-24T01:27:15.551171+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:30.183145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.551715+0000","last_change":"2024-03-24T01:27:14.539787+0000","last_active":"2024-03-24T01:27:15.551715+0000","last_peered":"2024-03-24T01:27:15.551715+0000","last_clean":"2024-03-24T01:27:15.551715+0000","last_became_active":"2024-03-24T01:27:14.539446+0000","last_became_peered":"2024-03-24T01:27:14.539446+0000","last_unstale":"2024-03-24T01:27:15.551715+0000","last_undegraded":"2024-03-24T01:27:15.551715+0000","last_fullsized":"2024-03-24T01:27:15.551715+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:09:58.411238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.977307+0000","last_change":"2024-03-24T01:27:13.519780+0000","last_active":"2024-03-24T01:27:15.977307+0000","last_peered":"2024-03-24T01:27:15.977307+0000","last_clean":"2024-03-24T01:27:15.977307+0000","last_became_active":"2024-03-24T01:27:13.519458+0000","last_became_peered":"2024-03-24T01:27:13.519458+0000","last_unstale":"2024-03-24T01:27:15.977307+0000","last_undegraded":"2024-03-24T01:27:15.977307+0000","last_fullsized":"2024-03-24T01:27:15.977307+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:46:44.745424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.560258+0000","last_change":"2024-03-24T01:27:14.549326+0000","last_active":"2024-03-24T01:27:15.560258+0000","last_peered":"2024-03-24T01:27:15.560258+0000","last_clean":"2024-03-24T01:27:15.560258+0000","last_became_active":"2024-03-24T01:27:14.548886+0000","last_became_peered":"2024-03-24T01:27:14.548886+0000","last_unstale":"2024-03-24T01:27:15.560258+0000","last_undegraded":"2024-03-24T01:27:15.560258+0000","last_fullsized":"2024-03-24T01:27:15.560258+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:54:52.883656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.559273+0000","last_change":"2024-03-24T01:27:13.521689+0000","last_active":"2024-03-24T01:27:15.559273+0000","last_peered":"2024-03-24T01:27:15.559273+0000","last_clean":"2024-03-24T01:27:15.559273+0000","last_became_active":"2024-03-24T01:27:13.521436+0000","last_became_peered":"2024-03-24T01:27:13.521436+0000","last_unstale":"2024-03-24T01:27:15.559273+0000","last_undegraded":"2024-03-24T01:27:15.559273+0000","last_fullsized":"2024-03-24T01:27:15.559273+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:47:38.446831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.551643+0000","last_change":"2024-03-24T01:27:14.543585+0000","last_active":"2024-03-24T01:27:15.551643+0000","last_peered":"2024-03-24T01:27:15.551643+0000","last_clean":"2024-03-24T01:27:15.551643+0000","last_became_active":"2024-03-24T01:27:14.543301+0000","last_became_peered":"2024-03-24T01:27:14.543301+0000","last_unstale":"2024-03-24T01:27:15.551643+0000","last_undegraded":"2024-03-24T01:27:15.551643+0000","last_fullsized":"2024-03-24T01:27:15.551643+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:02:42.596744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.975426+0000","last_change":"2024-03-24T01:27:13.522613+0000","last_active":"2024-03-24T01:27:15.975426+0000","last_peered":"2024-03-24T01:27:15.975426+0000","last_clean":"2024-03-24T01:27:15.975426+0000","last_became_active":"2024-03-24T01:27:13.522331+0000","last_became_peered":"2024-03-24T01:27:13.522331+0000","last_unstale":"2024-03-24T01:27:15.975426+0000","last_undegraded":"2024-03-24T01:27:15.975426+0000","last_fullsized":"2024-03-24T01:27:15.975426+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:08:09.036710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.550773+0000","last_change":"2024-03-24T01:27:14.550330+0000","last_active":"2024-03-24T01:27:15.550773+0000","last_peered":"2024-03-24T01:27:15.550773+0000","last_clean":"2024-03-24T01:27:15.550773+0000","last_became_active":"2024-03-24T01:27:14.550056+0000","last_became_peered":"2024-03-24T01:27:14.550056+0000","last_unstale":"2024-03-24T01:27:15.550773+0000","last_undegraded":"2024-03-24T01:27:15.550773+0000","last_fullsized":"2024-03-24T01:27:15.550773+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:14:16.916597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"52'2","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.551953+0000","last_change":"2024-03-24T01:27:13.533418+0000","last_active":"2024-03-24T01:27:15.551953+0000","last_peered":"2024-03-24T01:27:15.551953+0000","last_clean":"2024-03-24T01:27:15.551953+0000","last_became_active":"2024-03-24T01:27:13.533164+0000","last_became_peered":"2024-03-24T01:27:13.533164+0000","last_unstale":"2024-03-24T01:27:15.551953+0000","last_undegraded":"2024-03-24T01:27:15.551953+0000","last_fullsized":"2024-03-24T01:27:15.551953+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:30:49.041062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.550784+0000","last_change":"2024-03-24T01:27:14.543832+0000","last_active":"2024-03-24T01:27:15.550784+0000","last_peered":"2024-03-24T01:27:15.550784+0000","last_clean":"2024-03-24T01:27:15.550784+0000","last_became_active":"2024-03-24T01:27:14.543520+0000","last_became_peered":"2024-03-24T01:27:14.543520+0000","last_unstale":"2024-03-24T01:27:15.550784+0000","last_undegraded":"2024-03-24T01:27:15.550784+0000","last_fullsized":"2024-03-24T01:27:15.550784+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:23.971971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.565729+0000","last_change":"2024-03-24T01:27:13.530300+0000","last_active":"2024-03-24T01:27:15.565729+0000","last_peered":"2024-03-24T01:27:15.565729+0000","last_clean":"2024-03-24T01:27:15.565729+0000","last_became_active":"2024-03-24T01:27:13.529919+0000","last_became_peered":"2024-03-24T01:27:13.529919+0000","last_unstale":"2024-03-24T01:27:15.565729+0000","last_undegraded":"2024-03-24T01:27:15.565729+0000","last_fullsized":"2024-03-24T01:27:15.565729+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:47:38.446831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.976923+0000","last_change":"2024-03-24T01:27:14.544735+0000","last_active":"2024-03-24T01:27:15.976923+0000","last_peered":"2024-03-24T01:27:15.976923+0000","last_clean":"2024-03-24T01:27:15.976923+0000","last_became_active":"2024-03-24T01:27:14.544460+0000","last_became_peered":"2024-03-24T01:27:14.544460+0000","last_unstale":"2024-03-24T01:27:15.976923+0000","last_undegraded":"2024-03-24T01:27:15.976923+0000","last_fullsized":"2024-03-24T01:27:15.976923+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:07:29.856046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.559844+0000","last_change":"2024-03-24T01:27:13.525558+0000","last_active":"2024-03-24T01:27:15.559844+0000","last_peered":"2024-03-24T01:27:15.559844+0000","last_clean":"2024-03-24T01:27:15.559844+0000","last_became_active":"2024-03-24T01:27:13.525223+0000","last_became_peered":"2024-03-24T01:27:13.525223+0000","last_unstale":"2024-03-24T01:27:15.559844+0000","last_undegraded":"2024-03-24T01:27:15.559844+0000","last_fullsized":"2024-03-24T01:27:15.559844+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:26:43.325731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.559961+0000","last_change":"2024-03-24T01:27:14.549747+0000","last_active":"2024-03-24T01:27:15.559961+0000","last_peered":"2024-03-24T01:27:15.559961+0000","last_clean":"2024-03-24T01:27:15.559961+0000","last_became_active":"2024-03-24T01:27:14.548344+0000","last_became_peered":"2024-03-24T01:27:14.548344+0000","last_unstale":"2024-03-24T01:27:15.559961+0000","last_undegraded":"2024-03-24T01:27:15.559961+0000","last_fullsized":"2024-03-24T01:27:15.559961+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:26.907326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.976969+0000","last_change":"2024-03-24T01:27:13.519855+0000","last_active":"2024-03-24T01:27:15.976969+0000","last_peered":"2024-03-24T01:27:15.976969+0000","last_clean":"2024-03-24T01:27:15.976969+0000","last_became_active":"2024-03-24T01:27:13.519566+0000","last_became_peered":"2024-03-24T01:27:13.519566+0000","last_unstale":"2024-03-24T01:27:15.976969+0000","last_undegraded":"2024-03-24T01:27:15.976969+0000","last_fullsized":"2024-03-24T01:27:15.976969+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:57.615564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.552436+0000","last_change":"2024-03-24T01:27:14.563611+0000","last_active":"2024-03-24T01:27:15.552436+0000","last_peered":"2024-03-24T01:27:15.552436+0000","last_clean":"2024-03-24T01:27:15.552436+0000","last_became_active":"2024-03-24T01:27:14.563327+0000","last_became_peered":"2024-03-24T01:27:14.563327+0000","last_unstale":"2024-03-24T01:27:15.552436+0000","last_undegraded":"2024-03-24T01:27:15.552436+0000","last_fullsized":"2024-03-24T01:27:15.552436+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:19:18.467994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.977481+0000","last_change":"2024-03-24T01:27:13.524780+0000","last_active":"2024-03-24T01:27:15.977481+0000","last_peered":"2024-03-24T01:27:15.977481+0000","last_clean":"2024-03-24T01:27:15.977481+0000","last_became_active":"2024-03-24T01:27:13.524518+0000","last_became_peered":"2024-03-24T01:27:13.524518+0000","last_unstale":"2024-03-24T01:27:15.977481+0000","last_undegraded":"2024-03-24T01:27:15.977481+0000","last_fullsized":"2024-03-24T01:27:15.977481+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:14:15.902136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.559700+0000","last_change":"2024-03-24T01:27:14.571076+0000","last_active":"2024-03-24T01:27:15.559700+0000","last_peered":"2024-03-24T01:27:15.559700+0000","last_clean":"2024-03-24T01:27:15.559700+0000","last_became_active":"2024-03-24T01:27:14.570781+0000","last_became_peered":"2024-03-24T01:27:14.570781+0000","last_unstale":"2024-03-24T01:27:15.559700+0000","last_undegraded":"2024-03-24T01:27:15.559700+0000","last_fullsized":"2024-03-24T01:27:15.559700+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:55:30.524954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.975960+0000","last_change":"2024-03-24T01:27:13.519549+0000","last_active":"2024-03-24T01:27:15.975960+0000","last_peered":"2024-03-24T01:27:15.975960+0000","last_clean":"2024-03-24T01:27:15.975960+0000","last_became_active":"2024-03-24T01:27:13.519174+0000","last_became_peered":"2024-03-24T01:27:13.519174+0000","last_unstale":"2024-03-24T01:27:15.975960+0000","last_undegraded":"2024-03-24T01:27:15.975960+0000","last_fullsized":"2024-03-24T01:27:15.975960+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:30:49.041062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.978535+0000","last_change":"2024-03-24T01:27:14.548520+0000","last_active":"2024-03-24T01:27:15.978535+0000","last_peered":"2024-03-24T01:27:15.978535+0000","last_clean":"2024-03-24T01:27:15.978535+0000","last_became_active":"2024-03-24T01:27:14.547968+0000","last_became_peered":"2024-03-24T01:27:14.547968+0000","last_unstale":"2024-03-24T01:27:15.978535+0000","last_undegraded":"2024-03-24T01:27:15.978535+0000","last_fullsized":"2024-03-24T01:27:15.978535+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:19:52.286477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.550948+0000","last_change":"2024-03-24T01:27:13.524804+0000","last_active":"2024-03-24T01:27:15.550948+0000","last_peered":"2024-03-24T01:27:15.550948+0000","last_clean":"2024-03-24T01:27:15.550948+0000","last_became_active":"2024-03-24T01:27:13.524487+0000","last_became_peered":"2024-03-24T01:27:13.524487+0000","last_unstale":"2024-03-24T01:27:15.550948+0000","last_undegraded":"2024-03-24T01:27:15.550948+0000","last_fullsized":"2024-03-24T01:27:15.550948+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:43:44.823891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.550669+0000","last_change":"2024-03-24T01:27:14.539180+0000","last_active":"2024-03-24T01:27:15.550669+0000","last_peered":"2024-03-24T01:27:15.550669+0000","last_clean":"2024-03-24T01:27:15.550669+0000","last_became_active":"2024-03-24T01:27:14.538800+0000","last_became_peered":"2024-03-24T01:27:14.538800+0000","last_unstale":"2024-03-24T01:27:15.550669+0000","last_undegraded":"2024-03-24T01:27:15.550669+0000","last_fullsized":"2024-03-24T01:27:15.550669+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:13.484941+0000","last_clean_scrub_stamp":"2024-03-24T01:27:13.484941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:28:36.027521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"53'1","reported_seq":15,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.560443+0000","last_change":"2024-03-24T01:27:13.532470+0000","last_active":"2024-03-24T01:27:15.560443+0000","last_peered":"2024-03-24T01:27:15.560443+0000","last_clean":"2024-03-24T01:27:15.560443+0000","last_became_active":"2024-03-24T01:27:13.530705+0000","last_became_peered":"2024-03-24T01:27:13.530705+0000","last_unstale":"2024-03-24T01:27:15.560443+0000","last_undegraded":"2024-03-24T01:27:15.560443+0000","last_fullsized":"2024-03-24T01:27:15.560443+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:27:12.470480+0000","last_clean_scrub_stamp":"2024-03-24T01:27:12.470480+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:12:14.260140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":231,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-24T01:27:15.564864+0000","last_change":"2024-03-24T01:24:01.545855+0000","last_active":"2024-03-24T01:27:15.564864+0000","last_peered":"2024-03-24T01:27:15.564864+0000","last_clean":"2024-03-24T01:27:15.564864+0000","last_became_active":"2024-03-24T01:24:01.543811+0000","last_became_peered":"2024-03-24T01:24:01.543811+0000","last_unstale":"2024-03-24T01:27:15.564864+0000","last_undegraded":"2024-03-24T01:27:15.564864+0000","last_fullsized":"2024-03-24T01:27:15.564864+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_clean_scrub_stamp":"2024-03-24T01:23:59.861564+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:03.682631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574891,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28224,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343616,"statfs":{"total":96636764160,"available":96607862784,"internally_reserved":0,"allocated":1232896,"data_stored":875162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574891,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":285322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574892,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27636,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344204,"statfs":{"total":96636764160,"available":96608464896,"internally_reserved":0,"allocated":630784,"data_stored":284670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574891,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27640,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":634880,"data_stored":285252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574891,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27652,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344188,"statfs":{"total":96636764160,"available":96608448512,"internally_reserved":0,"allocated":647168,"data_stored":286304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574892,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27648,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":643072,"data_stored":286282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574891,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":284830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574892,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28224,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343616,"statfs":{"total":96636764160,"available":96607862784,"internally_reserved":0,"allocated":1232896,"data_stored":875094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T01:27:24.479 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-24T01:27:24.479 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-24T01:27:24.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.737+0000 7f5346000640 1 Processor -- start 2024-03-24T01:27:24.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.737+0000 7f5346000640 1 -- start start 2024-03-24T01:27:24.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.739+0000 7f5346000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f534015da30 0x7f534015de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:24.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.739+0000 7f5346000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f534006b0c0 0x7f534006b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:24.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.739+0000 7f5346000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53400af140 con 0x7f53400acea0 2024-03-24T01:27:24.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.739+0000 7f5346000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53400b2c70 con 0x7f534015da30 2024-03-24T01:27:24.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.739+0000 7f5346000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f534015f160 con 0x7f534006b0c0 2024-03-24T01:27:24.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.739+0000 7f533f400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f534015da30 0x7f534015de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:24.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.740+0000 7f533f400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f534015da30 0x7f534015de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57640/0 (socket says 172.21.15.90:57640) 2024-03-24T01:27:24.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.740+0000 7f533f400640 1 -- 172.21.15.90:0/361667747 learned_addr learned my addr 172.21.15.90:0/361667747 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:24.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.740+0000 7f533fe00640 1 --2- 172.21.15.90:0/361667747 >> v2:172.21.15.120:3300/0 conn(0x7f534006b0c0 0x7f534006b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:24.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.740+0000 7f533f400640 1 -- 172.21.15.90:0/361667747 >> v2:172.21.15.120:3300/0 conn(0x7f534006b0c0 msgr2=0x7f534006b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:24.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.740+0000 7f533f400640 1 --2- 172.21.15.90:0/361667747 >> v2:172.21.15.120:3300/0 conn(0x7f534006b0c0 0x7f534006b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:24.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.740+0000 7f533f400640 1 -- 172.21.15.90:0/361667747 >> v1:172.21.15.120:6789/0 conn(0x7f53400acea0 legacy=0x7f53400aea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:24.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.740+0000 7f533f400640 1 -- 172.21.15.90:0/361667747 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f534015e550 con 0x7f534015da30 2024-03-24T01:27:24.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.740+0000 7f533fe00640 1 --2- 172.21.15.90:0/361667747 >> v2:172.21.15.120:3300/0 conn(0x7f534006b0c0 0x7f534006b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:27:24.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.741+0000 7f533f400640 1 --2- 172.21.15.90:0/361667747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f534015da30 0x7f534015de10 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f53400ac970 tx=0x7f533002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d341c13fb6faf92 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:24.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.742+0000 7f533ea00640 1 -- 172.21.15.90:0/361667747 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5330035020 con 0x7f534015da30 2024-03-24T01:27:24.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.742+0000 7f533ea00640 1 -- 172.21.15.90:0/361667747 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5330031870 con 0x7f534015da30 2024-03-24T01:27:24.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.742+0000 7f533ea00640 1 -- 172.21.15.90:0/361667747 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5330031c40 con 0x7f534015da30 2024-03-24T01:27:24.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.742+0000 7f5346000640 1 -- 172.21.15.90:0/361667747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f534015da30 msgr2=0x7f534015de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:24.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.742+0000 7f5346000640 1 --2- 172.21.15.90:0/361667747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f534015da30 0x7f534015de10 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f53400ac970 tx=0x7f533002b910 comp rx=0 tx=0).stop 2024-03-24T01:27:24.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.742+0000 7f5346000640 1 -- 172.21.15.90:0/361667747 shutdown_connections 2024-03-24T01:27:24.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.742+0000 7f5346000640 1 --2- 172.21.15.90:0/361667747 >> v2:172.21.15.120:3300/0 conn(0x7f534006b0c0 0x7f534006b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:24.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.742+0000 7f5346000640 1 --2- 172.21.15.90:0/361667747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f534015da30 0x7f534015de10 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:24.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.742+0000 7f5346000640 1 -- 172.21.15.90:0/361667747 >> 172.21.15.90:0/361667747 conn(0x7f53400b1770 msgr2=0x7f53400b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:24.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.742+0000 7f5346000640 1 -- 172.21.15.90:0/361667747 shutdown_connections 2024-03-24T01:27:24.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.742+0000 7f5346000640 1 -- 172.21.15.90:0/361667747 wait complete. 2024-03-24T01:27:24.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.744+0000 7f5346000640 1 Processor -- start 2024-03-24T01:27:24.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.744+0000 7f5346000640 1 -- start start 2024-03-24T01:27:24.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.745+0000 7f5346000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f534006b0c0 0x7f5340150660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:24.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.745+0000 7f5346000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53400acea0 0x7f53401431b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:24.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.745+0000 7f5346000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f534015da30 0x7f534015c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:24.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.745+0000 7f5346000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f534015eb10 con 0x7f534006b0c0 2024-03-24T01:27:24.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.745+0000 7f5346000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f534006ba30 con 0x7f53400acea0 2024-03-24T01:27:24.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.745+0000 7f5346000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f534015edf0 con 0x7f534015da30 2024-03-24T01:27:24.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.745+0000 7f533f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f534006b0c0 0x7f5340150660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:24.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.745+0000 7f533f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f534006b0c0 0x7f5340150660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57648/0 (socket says 172.21.15.90:57648) 2024-03-24T01:27:24.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.745+0000 7f533f400640 1 -- 172.21.15.90:0/3744956877 learned_addr learned my addr 172.21.15.90:0/3744956877 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:24.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.746+0000 7f5337e00640 1 --2- 172.21.15.90:0/3744956877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53400acea0 0x7f53401431b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:24.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.746+0000 7f533fe00640 1 --2- 172.21.15.90:0/3744956877 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f534015da30 0x7f534015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:24.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.746+0000 7f533f400640 1 -- 172.21.15.90:0/3744956877 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f534015da30 msgr2=0x7f534015c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:24.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.746+0000 7f533f400640 1 --2- 172.21.15.90:0/3744956877 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f534015da30 0x7f534015c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:24.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.746+0000 7f533f400640 1 -- 172.21.15.90:0/3744956877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53400acea0 msgr2=0x7f53401431b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:24.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.746+0000 7f533f400640 1 --2- 172.21.15.90:0/3744956877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53400acea0 0x7f53401431b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:24.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.746+0000 7f533f400640 1 -- 172.21.15.90:0/3744956877 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5340150dd0 con 0x7f534006b0c0 2024-03-24T01:27:24.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.747+0000 7f533f400640 1 --2- 172.21.15.90:0/3744956877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f534006b0c0 0x7f5340150660 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f533002b950 tx=0x7f53300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:24.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.747+0000 7f5337400640 1 -- 172.21.15.90:0/3744956877 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5330040070 con 0x7f534006b0c0 2024-03-24T01:27:24.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.747+0000 7f5337400640 1 -- 172.21.15.90:0/3744956877 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f533003a420 con 0x7f534006b0c0 2024-03-24T01:27:24.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.747+0000 7f5337400640 1 -- 172.21.15.90:0/3744956877 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5330035040 con 0x7f534006b0c0 2024-03-24T01:27:24.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.747+0000 7f5346000640 1 -- 172.21.15.90:0/3744956877 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53401510b0 con 0x7f534006b0c0 2024-03-24T01:27:24.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.747+0000 7f5346000640 1 -- 172.21.15.90:0/3744956877 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5340151670 con 0x7f534006b0c0 2024-03-24T01:27:24.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.748+0000 7f5346000640 1 -- 172.21.15.90:0/3744956877 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f534004f400 con 0x7f534006b0c0 2024-03-24T01:27:24.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.749+0000 7f5337400640 1 -- 172.21.15.90:0/3744956877 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5330038020 con 0x7f534006b0c0 2024-03-24T01:27:24.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.750+0000 7f5337400640 1 --2- 172.21.15.90:0/3744956877 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f531c06bbd0 0x7f531c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:24.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.750+0000 7f5337400640 1 -- 172.21.15.90:0/3744956877 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f53300acc30 con 0x7f534006b0c0 2024-03-24T01:27:24.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.751+0000 7f5337e00640 1 --2- 172.21.15.90:0/3744956877 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f531c06bbd0 0x7f531c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:24.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.752+0000 7f5337e00640 1 --2- 172.21.15.90:0/3744956877 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f531c06bbd0 0x7f531c06e090 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f5340146db0 tx=0x7f5328009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:24.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:24.754+0000 7f5337400640 1 -- 172.21.15.90:0/3744956877 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f533007b580 con 0x7f534006b0c0 2024-03-24T01:27:25.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.025+0000 7f5346000640 1 -- 172.21.15.90:0/3744956877 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f534006e250 con 0x7f534006b0c0 2024-03-24T01:27:25.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.030+0000 7f5337400640 1 -- 172.21.15.90:0/3744956877 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) v1 ==== 190+0+147 (secure 0 0 0) 0x7f533007ae80 con 0x7f534006b0c0 2024-03-24T01:27:25.031 INFO:teuthology.orchestra.run.smithi090.stdout:[client.0] 2024-03-24T01:27:25.031 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQB0gP9lLKWXMxAAwrrRG8zLSylpIEuUgM7q8g== 2024-03-24T01:27:25.031 INFO:teuthology.orchestra.run.smithi090.stdout: caps mds = "allow" 2024-03-24T01:27:25.032 INFO:teuthology.orchestra.run.smithi090.stdout: caps mon = "allow r" 2024-03-24T01:27:25.032 INFO:teuthology.orchestra.run.smithi090.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-24T01:27:25.032 INFO:teuthology.orchestra.run.smithi090.stderr:updated caps for client.0 2024-03-24T01:27:25.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.032+0000 7f5346000640 1 -- 172.21.15.90:0/3744956877 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f531c06bbd0 msgr2=0x7f531c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:25.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.032+0000 7f5346000640 1 --2- 172.21.15.90:0/3744956877 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f531c06bbd0 0x7f531c06e090 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f5340146db0 tx=0x7f5328009040 comp rx=0 tx=0).stop 2024-03-24T01:27:25.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.033+0000 7f5346000640 1 -- 172.21.15.90:0/3744956877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f534006b0c0 msgr2=0x7f5340150660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:25.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.033+0000 7f5346000640 1 --2- 172.21.15.90:0/3744956877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f534006b0c0 0x7f5340150660 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f533002b950 tx=0x7f53300046c0 comp rx=0 tx=0).stop 2024-03-24T01:27:25.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.033+0000 7f5346000640 1 -- 172.21.15.90:0/3744956877 shutdown_connections 2024-03-24T01:27:25.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.033+0000 7f5346000640 1 --2- 172.21.15.90:0/3744956877 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f531c06bbd0 0x7f531c06e090 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:25.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.033+0000 7f5346000640 1 --2- 172.21.15.90:0/3744956877 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f534015da30 0x7f534015c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:25.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.033+0000 7f5346000640 1 --2- 172.21.15.90:0/3744956877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53400acea0 0x7f53401431b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:25.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.033+0000 7f5346000640 1 --2- 172.21.15.90:0/3744956877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f534006b0c0 0x7f5340150660 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:25.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.033+0000 7f5346000640 1 -- 172.21.15.90:0/3744956877 >> 172.21.15.90:0/3744956877 conn(0x7f53400b1770 msgr2=0x7f5340159750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:25.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.033+0000 7f5346000640 1 -- 172.21.15.90:0/3744956877 shutdown_connections 2024-03-24T01:27:25.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.033+0000 7f5346000640 1 -- 172.21.15.90:0/3744956877 wait complete. 2024-03-24T01:27:25.054 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:27:25.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.257+0000 7f01a1c00640 1 Processor -- start 2024-03-24T01:27:25.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.257+0000 7f01a1c00640 1 -- start start 2024-03-24T01:27:25.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.259+0000 7f01a1c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f019c1443d0 0x7f019c0cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:25.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.259+0000 7f01a1c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f019c0b3310 0x7f019c0cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:25.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.259+0000 7f01a1c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f019c1447b0 con 0x7f019c0cec90 2024-03-24T01:27:25.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.259+0000 7f01a1c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f019c15aef0 con 0x7f019c1443d0 2024-03-24T01:27:25.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.259+0000 7f01a1c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f019c15b110 con 0x7f019c0b3310 2024-03-24T01:27:25.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.259+0000 7f019aa00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f019c1443d0 0x7f019c0cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:25.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.259+0000 7f019aa00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f019c1443d0 0x7f019c0cceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57652/0 (socket says 172.21.15.90:57652) 2024-03-24T01:27:25.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.259+0000 7f019aa00640 1 -- 172.21.15.90:0/1428552455 learned_addr learned my addr 172.21.15.90:0/1428552455 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:25.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.260+0000 7f019aa00640 1 -- 172.21.15.90:0/1428552455 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f019c0b3310 msgr2=0x7f019c0cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:25.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.260+0000 7f019aa00640 1 --2- 172.21.15.90:0/1428552455 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f019c0b3310 0x7f019c0cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:25.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.260+0000 7f019aa00640 1 -- 172.21.15.90:0/1428552455 >> v1:172.21.15.120:6789/0 conn(0x7f019c0cec90 legacy=0x7f019c0cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:25.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.260+0000 7f019aa00640 1 -- 172.21.15.90:0/1428552455 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f019c15b3f0 con 0x7f019c1443d0 2024-03-24T01:27:25.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.261+0000 7f019aa00640 1 --2- 172.21.15.90:0/1428552455 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f019c1443d0 0x7f019c0cceb0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f018c009870 tx=0x7f018c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2c1f080ea25536f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:25.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.261+0000 7f0199600640 1 -- 172.21.15.90:0/1428552455 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f018c0317c0 con 0x7f019c1443d0 2024-03-24T01:27:25.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.261+0000 7f0199600640 1 -- 172.21.15.90:0/1428552455 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f018c037070 con 0x7f019c1443d0 2024-03-24T01:27:25.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.261+0000 7f0199600640 1 -- 172.21.15.90:0/1428552455 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f018c031b80 con 0x7f019c1443d0 2024-03-24T01:27:25.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.262+0000 7f01a1c00640 1 -- 172.21.15.90:0/1428552455 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f019c1443d0 msgr2=0x7f019c0cceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:25.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.262+0000 7f01a1c00640 1 --2- 172.21.15.90:0/1428552455 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f019c1443d0 0x7f019c0cceb0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f018c009870 tx=0x7f018c02b860 comp rx=0 tx=0).stop 2024-03-24T01:27:25.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.262+0000 7f01a1c00640 1 -- 172.21.15.90:0/1428552455 shutdown_connections 2024-03-24T01:27:25.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.262+0000 7f01a1c00640 1 --2- 172.21.15.90:0/1428552455 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f019c0b3310 0x7f019c0cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:25.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.262+0000 7f01a1c00640 1 --2- 172.21.15.90:0/1428552455 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f019c1443d0 0x7f019c0cceb0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:25.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.262+0000 7f01a1c00640 1 -- 172.21.15.90:0/1428552455 >> 172.21.15.90:0/1428552455 conn(0x7f019c06b2a0 msgr2=0x7f019c06b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:25.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.262+0000 7f01a1c00640 1 -- 172.21.15.90:0/1428552455 shutdown_connections 2024-03-24T01:27:25.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.262+0000 7f01a1c00640 1 -- 172.21.15.90:0/1428552455 wait complete. 2024-03-24T01:27:25.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.264+0000 7f01a1c00640 1 Processor -- start 2024-03-24T01:27:25.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.264+0000 7f01a1c00640 1 -- start start 2024-03-24T01:27:25.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.265+0000 7f01a1c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f019c0b3310 0x7f019c1e4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:25.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.265+0000 7f01a1c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f019c0cec90 0x7f019c1e5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:25.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.265+0000 7f01a1c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f019c1443d0 0x7f019c1e3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:25.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.265+0000 7f01a1c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f019c15bc90 con 0x7f019c1443d0 2024-03-24T01:27:25.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.265+0000 7f01a1c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f019c15b9b0 con 0x7f019c0b3310 2024-03-24T01:27:25.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.265+0000 7f01a1c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f019c15e610 con 0x7f019c0cec90 2024-03-24T01:27:25.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.265+0000 7f019b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f019c1443d0 0x7f019c1e3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:25.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.265+0000 7f019b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f019c1443d0 0x7f019c1e3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57656/0 (socket says 172.21.15.90:57656) 2024-03-24T01:27:25.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.265+0000 7f019b400640 1 -- 172.21.15.90:0/1644896764 learned_addr learned my addr 172.21.15.90:0/1644896764 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:25.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.265+0000 7f019aa00640 1 --2- 172.21.15.90:0/1644896764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f019c0b3310 0x7f019c1e4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:25.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.265+0000 7f019a000640 1 --2- 172.21.15.90:0/1644896764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f019c0cec90 0x7f019c1e5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:25.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.266+0000 7f019b400640 1 -- 172.21.15.90:0/1644896764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f019c0cec90 msgr2=0x7f019c1e5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:25.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.266+0000 7f019b400640 1 --2- 172.21.15.90:0/1644896764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f019c0cec90 0x7f019c1e5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:25.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.266+0000 7f019b400640 1 -- 172.21.15.90:0/1644896764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f019c0b3310 msgr2=0x7f019c1e4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:25.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.266+0000 7f019b400640 1 --2- 172.21.15.90:0/1644896764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f019c0b3310 0x7f019c1e4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:25.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.266+0000 7f019b400640 1 -- 172.21.15.90:0/1644896764 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f019c13e670 con 0x7f019c1443d0 2024-03-24T01:27:25.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.266+0000 7f019aa00640 1 --2- 172.21.15.90:0/1644896764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f019c0b3310 0x7f019c1e4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:27:25.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.266+0000 7f019b400640 1 --2- 172.21.15.90:0/1644896764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f019c1443d0 0x7f019c1e3690 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f019400c440 tx=0x7f019400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:25.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.266+0000 7f0192a00640 1 -- 172.21.15.90:0/1644896764 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0194011790 con 0x7f019c1443d0 2024-03-24T01:27:25.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.266+0000 7f0192a00640 1 -- 172.21.15.90:0/1644896764 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0194017070 con 0x7f019c1443d0 2024-03-24T01:27:25.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.266+0000 7f0192a00640 1 -- 172.21.15.90:0/1644896764 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0194011b70 con 0x7f019c1443d0 2024-03-24T01:27:25.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.266+0000 7f01a1c00640 1 -- 172.21.15.90:0/1644896764 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f019c1e42a0 con 0x7f019c1443d0 2024-03-24T01:27:25.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.266+0000 7f01a1c00640 1 -- 172.21.15.90:0/1644896764 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f019c13e910 con 0x7f019c1443d0 2024-03-24T01:27:25.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.268+0000 7f01a1c00640 1 -- 172.21.15.90:0/1644896764 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f019c140d50 con 0x7f019c1443d0 2024-03-24T01:27:25.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.269+0000 7f0192a00640 1 -- 172.21.15.90:0/1644896764 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0194011d10 con 0x7f019c1443d0 2024-03-24T01:27:25.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.270+0000 7f0192a00640 1 --2- 172.21.15.90:0/1644896764 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f017806bbd0 0x7f017806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:25.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.270+0000 7f0192a00640 1 -- 172.21.15.90:0/1644896764 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f019408a9f0 con 0x7f019c1443d0 2024-03-24T01:27:25.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.271+0000 7f019aa00640 1 --2- 172.21.15.90:0/1644896764 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f017806bbd0 0x7f017806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:25.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.272+0000 7f019aa00640 1 --2- 172.21.15.90:0/1644896764 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f017806bbd0 0x7f017806e090 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f018c036000 tx=0x7f018c03d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:25.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.275+0000 7f0192a00640 1 -- 172.21.15.90:0/1644896764 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0194059350 con 0x7f019c1443d0 2024-03-24T01:27:25.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.507+0000 7f01a1c00640 1 -- 172.21.15.90:0/1644896764 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f019c04f4e0 con 0x7f019c1443d0 2024-03-24T01:27:25.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.508+0000 7f0192a00640 1 -- 172.21.15.90:0/1644896764 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f019408a020 con 0x7f019c1443d0 2024-03-24T01:27:25.509 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:27:25.509 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":34,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4985,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6834/3475131973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":3475131973},{"type":"v1","addr":"172.21.15.90:6834","nonce":3475131973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":4994,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/3151604088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":3151604088},{"type":"v1","addr":"172.21.15.120:6837","nonce":3151604088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6219,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/226826781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":226826781},{"type":"v1","addr":"172.21.15.90:6835","nonce":226826781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T01:27:13.961373+0000","modified":"2024-03-24T01:27:17.043944+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6222},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6222":{"gid":6222,"name":"b","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6836/266172244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":266172244},{"type":"v1","addr":"172.21.15.120:6836","nonce":266172244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6222,"qdb_cluster":[6222]},"id":3}]} 2024-03-24T01:27:25.509 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 34 2024-03-24T01:27:25.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.510+0000 7f01a1c00640 1 -- 172.21.15.90:0/1644896764 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f017806bbd0 msgr2=0x7f017806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:25.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.510+0000 7f01a1c00640 1 --2- 172.21.15.90:0/1644896764 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f017806bbd0 0x7f017806e090 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f018c036000 tx=0x7f018c03d040 comp rx=0 tx=0).stop 2024-03-24T01:27:25.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.510+0000 7f01a1c00640 1 -- 172.21.15.90:0/1644896764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f019c1443d0 msgr2=0x7f019c1e3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:25.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.510+0000 7f01a1c00640 1 --2- 172.21.15.90:0/1644896764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f019c1443d0 0x7f019c1e3690 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f019400c440 tx=0x7f019400c900 comp rx=0 tx=0).stop 2024-03-24T01:27:25.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.510+0000 7f01a1c00640 1 -- 172.21.15.90:0/1644896764 shutdown_connections 2024-03-24T01:27:25.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.510+0000 7f01a1c00640 1 --2- 172.21.15.90:0/1644896764 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f017806bbd0 0x7f017806e090 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:25.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.510+0000 7f01a1c00640 1 --2- 172.21.15.90:0/1644896764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f019c1443d0 0x7f019c1e3690 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:25.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.510+0000 7f01a1c00640 1 --2- 172.21.15.90:0/1644896764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f019c0cec90 0x7f019c1e5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:25.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.510+0000 7f01a1c00640 1 --2- 172.21.15.90:0/1644896764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f019c0b3310 0x7f019c1e4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:25.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.510+0000 7f01a1c00640 1 -- 172.21.15.90:0/1644896764 >> 172.21.15.90:0/1644896764 conn(0x7f019c06b2a0 msgr2=0x7f019c0b45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:25.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.511+0000 7f01a1c00640 1 -- 172.21.15.90:0/1644896764 shutdown_connections 2024-03-24T01:27:25.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:25.511+0000 7f01a1c00640 1 -- 172.21.15.90:0/1644896764 wait complete. 2024-03-24T01:27:25.930 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 34, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T01:27:13.961373+0000', 'modified': '2024-03-24T01:27:17.043944+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6222}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6222': {'gid': 6222, 'name': 'b', 'rank': 0, 'incarnation': 30, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.120:6836/266172244', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6834', 'nonce': 266172244}, {'type': 'v1', 'addr': '172.21.15.120:6836', 'nonce': 266172244}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6222, 'qdb_cluster': [6222]} 2024-03-24T01:27:25.931 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T01:27:25.931 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-24T01:27:26.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.140+0000 7f036ec00640 1 Processor -- start 2024-03-24T01:27:26.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.141+0000 7f036ec00640 1 -- start start 2024-03-24T01:27:26.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.142+0000 7f036ec00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f036815de50 0x7f03680d1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:26.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.142+0000 7f036ec00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03680d35b0 0x7f03680b9540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:26.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.142+0000 7f036ec00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03680d2510 con 0x7f03680b5550 2024-03-24T01:27:26.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.142+0000 7f036ec00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03680b9a80 con 0x7f036815de50 2024-03-24T01:27:26.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.142+0000 7f036ec00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03680b7c10 con 0x7f03680d35b0 2024-03-24T01:27:26.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.143+0000 7f036ce00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f036815de50 0x7f03680d1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:26.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.143+0000 7f036ce00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f036815de50 0x7f03680d1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57672/0 (socket says 172.21.15.90:57672) 2024-03-24T01:27:26.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.143+0000 7f036ce00640 1 -- 172.21.15.90:0/288526826 learned_addr learned my addr 172.21.15.90:0/288526826 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:26.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.143+0000 7f036e200640 1 --2- 172.21.15.90:0/288526826 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03680d35b0 0x7f03680b9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:26.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.143+0000 7f036ce00640 1 -- 172.21.15.90:0/288526826 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03680d35b0 msgr2=0x7f03680b9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:26.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.143+0000 7f036ce00640 1 --2- 172.21.15.90:0/288526826 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03680d35b0 0x7f03680b9540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.143+0000 7f036ce00640 1 -- 172.21.15.90:0/288526826 >> v1:172.21.15.120:6789/0 conn(0x7f03680b5550 legacy=0x7f03680b5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:26.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.143+0000 7f036ce00640 1 -- 172.21.15.90:0/288526826 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03680d27f0 con 0x7f036815de50 2024-03-24T01:27:26.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.143+0000 7f036e200640 1 --2- 172.21.15.90:0/288526826 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03680d35b0 0x7f03680b9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:27:26.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.144+0000 7f036ce00640 1 --2- 172.21.15.90:0/288526826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f036815de50 0x7f03680d1ea0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f0358009e50 tx=0x7f035802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ada27a5c55b804f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:26.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.145+0000 7f0367e00640 1 -- 172.21.15.90:0/288526826 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035802c9f0 con 0x7f036815de50 2024-03-24T01:27:26.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.145+0000 7f0367e00640 1 -- 172.21.15.90:0/288526826 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f035802cb90 con 0x7f036815de50 2024-03-24T01:27:26.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.145+0000 7f0367e00640 1 -- 172.21.15.90:0/288526826 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03580343f0 con 0x7f036815de50 2024-03-24T01:27:26.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.145+0000 7f036ec00640 1 -- 172.21.15.90:0/288526826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f036815de50 msgr2=0x7f03680d1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:26.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.145+0000 7f036ec00640 1 --2- 172.21.15.90:0/288526826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f036815de50 0x7f03680d1ea0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f0358009e50 tx=0x7f035802be50 comp rx=0 tx=0).stop 2024-03-24T01:27:26.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.145+0000 7f036ec00640 1 -- 172.21.15.90:0/288526826 shutdown_connections 2024-03-24T01:27:26.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.145+0000 7f036ec00640 1 --2- 172.21.15.90:0/288526826 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03680d35b0 0x7f03680b9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.145+0000 7f036ec00640 1 --2- 172.21.15.90:0/288526826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f036815de50 0x7f03680d1ea0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.145+0000 7f036ec00640 1 -- 172.21.15.90:0/288526826 >> 172.21.15.90:0/288526826 conn(0x7f03680d0550 msgr2=0x7f03680d0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:26.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.145+0000 7f036ec00640 1 -- 172.21.15.90:0/288526826 shutdown_connections 2024-03-24T01:27:26.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.145+0000 7f036ec00640 1 -- 172.21.15.90:0/288526826 wait complete. 2024-03-24T01:27:26.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.147+0000 7f036ec00640 1 Processor -- start 2024-03-24T01:27:26.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.147+0000 7f036ec00640 1 -- start start 2024-03-24T01:27:26.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.148+0000 7f036ec00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03680b5550 0x7f036815e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:26.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.148+0000 7f036ec00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03680d35b0 0x7f03680c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:26.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.148+0000 7f036ec00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f036815de50 0x7f03681f34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:26.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.148+0000 7f036ec00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03680d3090 con 0x7f036815de50 2024-03-24T01:27:26.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.148+0000 7f036ec00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03680d2db0 con 0x7f03680b5550 2024-03-24T01:27:26.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.148+0000 7f036ec00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03680b6c00 con 0x7f03680d35b0 2024-03-24T01:27:26.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.148+0000 7f036e200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f036815de50 0x7f03681f34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:26.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.148+0000 7f036e200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f036815de50 0x7f03681f34a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57684/0 (socket says 172.21.15.90:57684) 2024-03-24T01:27:26.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.148+0000 7f036e200640 1 -- 172.21.15.90:0/1644085716 learned_addr learned my addr 172.21.15.90:0/1644085716 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:26.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.149+0000 7f036e200640 1 -- 172.21.15.90:0/1644085716 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03680d35b0 msgr2=0x7f03680c7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:26.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.149+0000 7f036e200640 1 --2- 172.21.15.90:0/1644085716 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03680d35b0 0x7f03680c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.149+0000 7f036e200640 1 -- 172.21.15.90:0/1644085716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03680b5550 msgr2=0x7f036815e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:26.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.149+0000 7f036e200640 1 --2- 172.21.15.90:0/1644085716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03680b5550 0x7f036815e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.149+0000 7f036e200640 1 -- 172.21.15.90:0/1644085716 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03680c7db0 con 0x7f036815de50 2024-03-24T01:27:26.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.149+0000 7f036e200640 1 --2- 172.21.15.90:0/1644085716 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f036815de50 0x7f03681f34a0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f036000c820 tx=0x7f036000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:26.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.149+0000 7f0365600640 1 -- 172.21.15.90:0/1644085716 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0360015070 con 0x7f036815de50 2024-03-24T01:27:26.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.149+0000 7f0365600640 1 -- 172.21.15.90:0/1644085716 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0360007a30 con 0x7f036815de50 2024-03-24T01:27:26.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.149+0000 7f036ec00640 1 -- 172.21.15.90:0/1644085716 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f036815ec20 con 0x7f036815de50 2024-03-24T01:27:26.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.149+0000 7f0365600640 1 -- 172.21.15.90:0/1644085716 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0360007e20 con 0x7f036815de50 2024-03-24T01:27:26.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.150+0000 7f036ec00640 1 -- 172.21.15.90:0/1644085716 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f036815f1e0 con 0x7f036815de50 2024-03-24T01:27:26.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.152+0000 7f0365600640 1 -- 172.21.15.90:0/1644085716 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0360018070 con 0x7f036815de50 2024-03-24T01:27:26.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.153+0000 7f0365600640 1 --2- 172.21.15.90:0/1644085716 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f034c06bbd0 0x7f034c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:26.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.153+0000 7f0365600640 1 -- 172.21.15.90:0/1644085716 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f036008ab20 con 0x7f036815de50 2024-03-24T01:27:26.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.153+0000 7f036d800640 1 --2- 172.21.15.90:0/1644085716 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f034c06bbd0 0x7f034c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:26.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.153+0000 7f0372f79740 1 Processor -- start 2024-03-24T01:27:26.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.154+0000 7f0372f79740 1 -- start start 2024-03-24T01:27:26.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.154+0000 7f036d800640 1 --2- 172.21.15.90:0/1644085716 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f034c06bbd0 0x7f034c06e090 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f035c000f60 tx=0x7f035c0079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:26.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.155+0000 7f0372f79740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5572de0ccdb0 0x5572de0cd190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:26.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.155+0000 7f0372f79740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572de09cdd0 0x5572de0ca020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:26.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.155+0000 7f0372f79740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572de0a0930 0x5572de0ca560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:26.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.155+0000 7f0372f79740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572de006c90 con 0x5572de0a0930 2024-03-24T01:27:26.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.155+0000 7f0372f79740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572ddfaf980 con 0x5572de09cdd0 2024-03-24T01:27:26.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.155+0000 7f0372f79740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5572de002d30 con 0x5572de0ccdb0 2024-03-24T01:27:26.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.155+0000 7f036e200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572de0a0930 0x5572de0ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:26.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.155+0000 7f036e200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572de0a0930 0x5572de0ca560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57698/0 (socket says 172.21.15.90:57698) 2024-03-24T01:27:26.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.155+0000 7f036e200640 1 -- 172.21.15.90:0/1468990641 learned_addr learned my addr 172.21.15.90:0/1468990641 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:26.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.155+0000 7f036e200640 1 -- 172.21.15.90:0/1468990641 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5572de0ccdb0 msgr2=0x5572de0cd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:26.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.155+0000 7f036e200640 1 --2- 172.21.15.90:0/1468990641 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5572de0ccdb0 0x5572de0cd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.155+0000 7f036e200640 1 -- 172.21.15.90:0/1468990641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572de09cdd0 msgr2=0x5572de0ca020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:26.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.155+0000 7f036e200640 1 --2- 172.21.15.90:0/1468990641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572de09cdd0 0x5572de0ca020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.155+0000 7f036e200640 1 -- 172.21.15.90:0/1468990641 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5572ddfc7510 con 0x5572de0a0930 2024-03-24T01:27:26.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.155+0000 7f036e200640 1 --2- 172.21.15.90:0/1468990641 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572de0a0930 0x5572de0ca560 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f0360060f60 tx=0x7f0360058ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=695f20eaa359379d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:26.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.156+0000 7f0356000640 1 -- 172.21.15.90:0/1468990641 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0360087070 con 0x5572de0a0930 2024-03-24T01:27:26.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.156+0000 7f0356000640 1 -- 172.21.15.90:0/1468990641 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f036000ce10 con 0x5572de0a0930 2024-03-24T01:27:26.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.156+0000 7f0356000640 1 -- 172.21.15.90:0/1468990641 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036006f030 con 0x5572de0a0930 2024-03-24T01:27:26.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.156+0000 7f0372f79740 1 -- 172.21.15.90:0/1468990641 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572de0a0930 msgr2=0x5572de0ca560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:26.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.156+0000 7f0372f79740 1 --2- 172.21.15.90:0/1468990641 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572de0a0930 0x5572de0ca560 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f0360060f60 tx=0x7f0360058ec0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.156+0000 7f0372f79740 1 -- 172.21.15.90:0/1468990641 shutdown_connections 2024-03-24T01:27:26.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.156+0000 7f0372f79740 1 --2- 172.21.15.90:0/1468990641 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572de0a0930 0x5572de0ca560 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.156+0000 7f0372f79740 1 --2- 172.21.15.90:0/1468990641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572de09cdd0 0x5572de0ca020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.156+0000 7f0372f79740 1 --2- 172.21.15.90:0/1468990641 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5572de0ccdb0 0x5572de0cd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.156+0000 7f0372f79740 1 -- 172.21.15.90:0/1468990641 >> 172.21.15.90:0/1468990641 conn(0x5572de09f040 msgr2=0x5572de0e7bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:26.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.156+0000 7f0372f79740 1 -- 172.21.15.90:0/1468990641 shutdown_connections 2024-03-24T01:27:26.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.156+0000 7f0372f79740 1 -- 172.21.15.90:0/1468990641 wait complete. 2024-03-24T01:27:26.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.157+0000 7f0372f79740 1 -- start start 2024-03-24T01:27:26.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.157+0000 7f0372f79740 1 Processor -- start 2024-03-24T01:27:26.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.158+0000 7f0372f79740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572de0a0930 0x5572de0d00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:26.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.158+0000 7f0372f79740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572de0ccdb0 0x5572de0c70e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:26.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.158+0000 7f0372f79740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5572de0c2760 0x5572de0c7620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:26.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.158+0000 7f0372f79740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572de0a1ae0 con 0x5572de0ccdb0 2024-03-24T01:27:26.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.158+0000 7f0372f79740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572ddfc2dd0 con 0x5572de0a0930 2024-03-24T01:27:26.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.158+0000 7f0372f79740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5572de0a0ed0 con 0x5572de0c2760 2024-03-24T01:27:26.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.158+0000 7f0372f79740 10 client.0 _ll_register_callbacks cb 0x5572ddd1aac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T01:27:26.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.159+0000 7f036ce00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572de0ccdb0 0x5572de0c70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:26.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.159+0000 7f036ce00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572de0ccdb0 0x5572de0c70e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57714/0 (socket says 172.21.15.90:57714) 2024-03-24T01:27:26.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.159+0000 7f036ce00640 1 -- 172.21.15.90:0/558804359 learned_addr learned my addr 172.21.15.90:0/558804359 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:26.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.159+0000 7f036d800640 1 --2- 172.21.15.90:0/558804359 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572de0a0930 0x5572de0d00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:26.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.159+0000 7f036ce00640 1 -- 172.21.15.90:0/558804359 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5572de0c2760 msgr2=0x5572de0c7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:26.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.159+0000 7f036e200640 1 --2- 172.21.15.90:0/558804359 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5572de0c2760 0x5572de0c7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:26.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.159+0000 7f036ce00640 1 --2- 172.21.15.90:0/558804359 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5572de0c2760 0x5572de0c7620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.159+0000 7f036ce00640 1 -- 172.21.15.90:0/558804359 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572de0a0930 msgr2=0x5572de0d00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:26.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.159+0000 7f036ce00640 1 --2- 172.21.15.90:0/558804359 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572de0a0930 0x5572de0d00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.159+0000 7f036ce00640 1 -- 172.21.15.90:0/558804359 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5572ddd1b2d0 con 0x5572de0ccdb0 2024-03-24T01:27:26.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.159+0000 7f036d800640 1 --2- 172.21.15.90:0/558804359 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572de0a0930 0x5572de0d00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:27:26.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.159+0000 7f036ce00640 1 --2- 172.21.15.90:0/558804359 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572de0ccdb0 0x5572de0c70e0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f0358009040 tx=0x7f035800c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce48526fa31a46d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:26.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.160+0000 7f034aa00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-03-24T01:27:26.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.160+0000 7f034aa00640 1 -- 172.21.15.90:0/558804359 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0358043070 con 0x5572de0ccdb0 2024-03-24T01:27:26.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.160+0000 7f0372f79740 1 -- 172.21.15.90:0/558804359 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f036000ce10 con 0x5572de0ccdb0 2024-03-24T01:27:26.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.160+0000 7f034aa00640 1 -- 172.21.15.90:0/558804359 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f035803f850 con 0x5572de0ccdb0 2024-03-24T01:27:26.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.160+0000 7f034aa00640 1 -- 172.21.15.90:0/558804359 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f035804d050 con 0x5572de0ccdb0 2024-03-24T01:27:26.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.160+0000 7f0372f79740 10 client.6387 fetch_fsmap learned FSMap version 34 2024-03-24T01:27:26.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.160+0000 7f0372f79740 1 -- 172.21.15.90:0/558804359 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) v3 -- 0x5572ddde56b0 con 0x5572de0ccdb0 2024-03-24T01:27:26.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.160+0000 7f034aa00640 1 -- 172.21.15.90:0/558804359 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f036000ce10 con 0x5572de0ccdb0 2024-03-24T01:27:26.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.161+0000 7f034aa00640 1 -- 172.21.15.90:0/558804359 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 34) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f0358050020 con 0x5572de0ccdb0 2024-03-24T01:27:26.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.161+0000 7f0372f79740 10 client.6387 fetch_fsmap finished waiting for FSMap version 34 2024-03-24T01:27:26.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.161+0000 7f0372f79740 10 client.6387 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T01:27:26.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.161+0000 7f0372f79740 20 client.6387 populate_metadata read hostname 'smithi090' 2024-03-24T01:27:26.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.161+0000 7f0372f79740 1 --2- 172.21.15.90:0/558804359 >> [v2:172.21.15.120:6834/266172244,v1:172.21.15.120:6836/266172244] conn(0x5572de0cda90 0x5572de0cde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:26.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.161+0000 7f0372f79740 4 client.6387 mds_command: new command op to 6222 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T01:27:26.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.161+0000 7f0372f79740 1 -- 172.21.15.90:0/558804359 --> [v2:172.21.15.120:6834/266172244,v1:172.21.15.120:6836/266172244] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5572ddfa8520 con 0x5572de0cda90 2024-03-24T01:27:26.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.161+0000 7f036d800640 1 --2- 172.21.15.90:0/558804359 >> [v2:172.21.15.120:6834/266172244,v1:172.21.15.120:6836/266172244] conn(0x5572de0cda90 0x5572de0cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:26.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.162+0000 7f034aa00640 10 client.6387 ms_handle_connect on v2:172.21.15.120:6834/266172244 2024-03-24T01:27:26.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.162+0000 7f036d800640 1 --2- 172.21.15.90:0/558804359 >> [v2:172.21.15.120:6834/266172244,v1:172.21.15.120:6836/266172244] conn(0x5572de0cda90 0x5572de0cde70 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30b1a5f2fb46067d server_cookie=a46cdf093d9cc973 in_seq=0 out_seq=0 2024-03-24T01:27:26.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.164+0000 7f034aa00640 1 -- 172.21.15.90:0/558804359 <== mds.0 v2:172.21.15.120:6834/266172244 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5572ddfa8520 con 0x5572de0cda90 2024-03-24T01:27:26.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.164+0000 7f034aa00640 10 client.6387 handle_command_reply: tid=0 2024-03-24T01:27:26.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.164+0000 7f0372f79740 1 client.6387 shutdown 2024-03-24T01:27:26.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.165+0000 7f0372f79740 1 -- 172.21.15.90:0/558804359 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572de0ccdb0 msgr2=0x5572de0c70e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:26.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.165+0000 7f0372f79740 1 --2- 172.21.15.90:0/558804359 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572de0ccdb0 0x5572de0c70e0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f0358009040 tx=0x7f035800c040 comp rx=0 tx=0).stop 2024-03-24T01:27:26.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.165+0000 7f0372f79740 1 -- 172.21.15.90:0/558804359 shutdown_connections 2024-03-24T01:27:26.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.165+0000 7f0372f79740 1 --2- 172.21.15.90:0/558804359 >> [v2:172.21.15.120:6834/266172244,v1:172.21.15.120:6836/266172244] conn(0x5572de0cda90 0x5572de0cde70 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.165+0000 7f0372f79740 1 --2- 172.21.15.90:0/558804359 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572de0ccdb0 0x5572de0c70e0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.165+0000 7f034aa00640 0 client.6387 ms_handle_reset on v2:172.21.15.120:6834/266172244 2024-03-24T01:27:26.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.165+0000 7f0372f79740 1 --2- 172.21.15.90:0/558804359 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5572de0c2760 0x5572de0c7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.165+0000 7f0372f79740 1 --2- 172.21.15.90:0/558804359 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572de0a0930 0x5572de0d00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.165+0000 7f0372f79740 1 -- 172.21.15.90:0/558804359 >> 172.21.15.90:0/558804359 conn(0x5572de09cdd0 msgr2=0x5572de0eb0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:26.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.165+0000 7f0372f79740 1 -- 172.21.15.90:0/558804359 shutdown_connections 2024-03-24T01:27:26.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.165+0000 7f0372f79740 1 -- 172.21.15.90:0/558804359 wait complete. 2024-03-24T01:27:26.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.165+0000 7f0372f79740 20 client.6387 trim_cache size 0 max 16384 2024-03-24T01:27:26.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.183+0000 7f0372f79740 1 Processor -- start 2024-03-24T01:27:26.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.183+0000 7f0372f79740 1 -- start start 2024-03-24T01:27:26.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.184+0000 7f0372f79740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5572de0a0930 0x5572de0c1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:26.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.184+0000 7f0372f79740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572de0c2760 0x5572de0e7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:26.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.184+0000 7f0372f79740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572de0ccdb0 0x5572de0eb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:26.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.184+0000 7f0372f79740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03681ff690 con 0x5572de0ccdb0 2024-03-24T01:27:26.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.184+0000 7f0372f79740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0368160860 con 0x5572de0c2760 2024-03-24T01:27:26.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.184+0000 7f0372f79740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0368160e60 con 0x5572de0a0930 2024-03-24T01:27:26.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.184+0000 7f036e200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572de0ccdb0 0x5572de0eb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:26.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.184+0000 7f036e200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572de0ccdb0 0x5572de0eb0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57722/0 (socket says 172.21.15.90:57722) 2024-03-24T01:27:26.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.184+0000 7f036e200640 1 -- 172.21.15.90:0/413502458 learned_addr learned my addr 172.21.15.90:0/413502458 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:26.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.185+0000 7f036d800640 1 --2- 172.21.15.90:0/413502458 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5572de0a0930 0x5572de0c1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:26.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.185+0000 7f036ce00640 1 --2- 172.21.15.90:0/413502458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572de0c2760 0x5572de0e7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:26.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.185+0000 7f036e200640 1 -- 172.21.15.90:0/413502458 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5572de0a0930 msgr2=0x5572de0c1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:26.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.185+0000 7f036e200640 1 --2- 172.21.15.90:0/413502458 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5572de0a0930 0x5572de0c1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.185+0000 7f036e200640 1 -- 172.21.15.90:0/413502458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572de0c2760 msgr2=0x5572de0e7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:26.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.185+0000 7f036e200640 1 --2- 172.21.15.90:0/413502458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572de0c2760 0x5572de0e7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.185+0000 7f036e200640 1 -- 172.21.15.90:0/413502458 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5572de0fbe60 con 0x5572de0ccdb0 2024-03-24T01:27:26.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.185+0000 7f036d800640 1 --2- 172.21.15.90:0/413502458 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5572de0a0930 0x5572de0c1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:27:26.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.185+0000 7f036ce00640 1 --2- 172.21.15.90:0/413502458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572de0c2760 0x5572de0e7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:27:26.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.185+0000 7f036e200640 1 --2- 172.21.15.90:0/413502458 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572de0ccdb0 0x5572de0eb0d0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f03600551e0 tx=0x7f036005f050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be1476aaeaa69e98 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:26.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.185+0000 7f0355600640 1 -- 172.21.15.90:0/413502458 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0360076070 con 0x5572de0ccdb0 2024-03-24T01:27:26.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.185+0000 7f0355600640 1 -- 172.21.15.90:0/413502458 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0360056e20 con 0x5572de0ccdb0 2024-03-24T01:27:26.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.185+0000 7f0355600640 1 -- 172.21.15.90:0/413502458 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0360087040 con 0x5572de0ccdb0 2024-03-24T01:27:26.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.186+0000 7f0372f79740 1 -- 172.21.15.90:0/413502458 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572de0ccdb0 msgr2=0x5572de0eb0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:26.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.186+0000 7f0372f79740 1 --2- 172.21.15.90:0/413502458 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572de0ccdb0 0x5572de0eb0d0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f03600551e0 tx=0x7f036005f050 comp rx=0 tx=0).stop 2024-03-24T01:27:26.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.186+0000 7f0372f79740 1 -- 172.21.15.90:0/413502458 shutdown_connections 2024-03-24T01:27:26.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.186+0000 7f0372f79740 1 --2- 172.21.15.90:0/413502458 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572de0ccdb0 0x5572de0eb0d0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.186+0000 7f0372f79740 1 --2- 172.21.15.90:0/413502458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572de0c2760 0x5572de0e7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.186+0000 7f0372f79740 1 --2- 172.21.15.90:0/413502458 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5572de0a0930 0x5572de0c1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.186+0000 7f0372f79740 1 -- 172.21.15.90:0/413502458 >> 172.21.15.90:0/413502458 conn(0x5572de09cdd0 msgr2=0x5572de0c9f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:26.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.186+0000 7f0372f79740 1 -- 172.21.15.90:0/413502458 shutdown_connections 2024-03-24T01:27:26.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.186+0000 7f0372f79740 1 -- 172.21.15.90:0/413502458 wait complete. 2024-03-24T01:27:26.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.187+0000 7f0372f79740 1 -- start start 2024-03-24T01:27:26.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.187+0000 7f0372f79740 1 Processor -- start 2024-03-24T01:27:26.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.188+0000 7f0372f79740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572de0a0930 0x5572de1568b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:26.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.188+0000 7f0372f79740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572de0c2760 0x5572de0bb280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:26.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.188+0000 7f0372f79740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5572de0ccdb0 0x5572de0bb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:26.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.188+0000 7f0372f79740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572de0fb650 con 0x5572de0a0930 2024-03-24T01:27:26.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.188+0000 7f0372f79740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572de0c3330 con 0x5572de0c2760 2024-03-24T01:27:26.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.188+0000 7f0372f79740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5572de0fbb90 con 0x5572de0ccdb0 2024-03-24T01:27:26.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.188+0000 7f0372f79740 10 client.0 _ll_register_callbacks cb 0x7f0358009d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T01:27:26.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.188+0000 7f036d800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572de0a0930 0x5572de1568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:26.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.188+0000 7f036d800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572de0a0930 0x5572de1568b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57738/0 (socket says 172.21.15.90:57738) 2024-03-24T01:27:26.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.188+0000 7f036d800640 1 -- 172.21.15.90:0/965240164 learned_addr learned my addr 172.21.15.90:0/965240164 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:26.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.188+0000 7f036d800640 1 -- 172.21.15.90:0/965240164 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5572de0ccdb0 msgr2=0x5572de0bb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:26.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.188+0000 7f036d800640 1 --2- 172.21.15.90:0/965240164 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5572de0ccdb0 0x5572de0bb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.189+0000 7f036d800640 1 -- 172.21.15.90:0/965240164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572de0c2760 msgr2=0x5572de0bb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:26.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.189+0000 7f036d800640 1 --2- 172.21.15.90:0/965240164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572de0c2760 0x5572de0bb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.189+0000 7f036d800640 1 -- 172.21.15.90:0/965240164 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5572de0fae80 con 0x5572de0a0930 2024-03-24T01:27:26.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.189+0000 7f036d800640 1 --2- 172.21.15.90:0/965240164 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572de0a0930 0x5572de1568b0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x5572de0c7de0 tx=0x7f035c0218d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b07ce6766e0c03fa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:26.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.189+0000 7f034aa00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-03-24T01:27:26.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.189+0000 7f034aa00640 1 -- 172.21.15.90:0/965240164 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035c027060 con 0x5572de0a0930 2024-03-24T01:27:26.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.189+0000 7f034aa00640 1 -- 172.21.15.90:0/965240164 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f035c003a00 con 0x5572de0a0930 2024-03-24T01:27:26.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.189+0000 7f0372f79740 1 -- 172.21.15.90:0/965240164 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0360056e20 con 0x5572de0a0930 2024-03-24T01:27:26.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.189+0000 7f034aa00640 1 -- 172.21.15.90:0/965240164 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f035c02d050 con 0x5572de0a0930 2024-03-24T01:27:26.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.189+0000 7f0372f79740 10 client.6393 fetch_fsmap learned FSMap version 34 2024-03-24T01:27:26.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.189+0000 7f0372f79740 1 -- 172.21.15.90:0/965240164 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) v3 -- 0x5572de0bbe80 con 0x5572de0a0930 2024-03-24T01:27:26.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.190+0000 7f034aa00640 1 -- 172.21.15.90:0/965240164 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0360056e20 con 0x5572de0a0930 2024-03-24T01:27:26.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.190+0000 7f034aa00640 1 -- 172.21.15.90:0/965240164 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 34) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f035c030020 con 0x5572de0a0930 2024-03-24T01:27:26.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.190+0000 7f0372f79740 10 client.6393 fetch_fsmap finished waiting for FSMap version 34 2024-03-24T01:27:26.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.190+0000 7f0372f79740 10 client.6393 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T01:27:26.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.190+0000 7f0372f79740 20 client.6393 populate_metadata read hostname 'smithi090' 2024-03-24T01:27:26.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.190+0000 7f0372f79740 1 --2- 172.21.15.90:0/965240164 >> [v2:172.21.15.120:6834/266172244,v1:172.21.15.120:6836/266172244] conn(0x5572de0a20e0 0x5572de0c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:26.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.190+0000 7f0372f79740 4 client.6393 mds_command: new command op to 6222 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T01:27:26.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.190+0000 7f0372f79740 1 -- 172.21.15.90:0/965240164 --> [v2:172.21.15.120:6834/266172244,v1:172.21.15.120:6836/266172244] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5572de0141d0 con 0x5572de0a20e0 2024-03-24T01:27:26.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.190+0000 7f036ce00640 1 --2- 172.21.15.90:0/965240164 >> [v2:172.21.15.120:6834/266172244,v1:172.21.15.120:6836/266172244] conn(0x5572de0a20e0 0x5572de0c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:26.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.191+0000 7f036ce00640 1 --2- 172.21.15.90:0/965240164 >> [v2:172.21.15.120:6834/266172244,v1:172.21.15.120:6836/266172244] conn(0x5572de0a20e0 0x5572de0c7e10 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=afd9ea8858046687 server_cookie=54ae66fcb91e9998 in_seq=0 out_seq=0 2024-03-24T01:27:26.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.191+0000 7f034aa00640 10 client.6393 ms_handle_connect on v2:172.21.15.120:6834/266172244 2024-03-24T01:27:26.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.192+0000 7f034aa00640 1 -- 172.21.15.90:0/965240164 <== mds.0 v2:172.21.15.120:6834/266172244 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x5572de0141d0 con 0x5572de0a20e0 2024-03-24T01:27:26.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.192+0000 7f034aa00640 10 client.6393 handle_command_reply: tid=0 2024-03-24T01:27:26.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.192+0000 7f0372f79740 1 client.6393 shutdown 2024-03-24T01:27:26.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.192+0000 7f0372f79740 1 -- 172.21.15.90:0/965240164 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572de0a0930 msgr2=0x5572de1568b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:26.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.192+0000 7f0372f79740 1 --2- 172.21.15.90:0/965240164 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572de0a0930 0x5572de1568b0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x5572de0c7de0 tx=0x7f035c0218d0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.192+0000 7f0372f79740 1 -- 172.21.15.90:0/965240164 shutdown_connections 2024-03-24T01:27:26.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.192+0000 7f0372f79740 1 --2- 172.21.15.90:0/965240164 >> [v2:172.21.15.120:6834/266172244,v1:172.21.15.120:6836/266172244] conn(0x5572de0a20e0 0x5572de0c7e10 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.192+0000 7f0372f79740 1 --2- 172.21.15.90:0/965240164 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5572de0ccdb0 0x5572de0bb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.192+0000 7f0372f79740 1 --2- 172.21.15.90:0/965240164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572de0c2760 0x5572de0bb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.192+0000 7f034aa00640 0 client.6393 ms_handle_reset on v2:172.21.15.120:6834/266172244 2024-03-24T01:27:26.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.192+0000 7f0372f79740 1 --2- 172.21.15.90:0/965240164 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5572de0a0930 0x5572de1568b0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.192+0000 7f0372f79740 1 -- 172.21.15.90:0/965240164 >> 172.21.15.90:0/965240164 conn(0x5572de09cdd0 msgr2=0x5572de0ca100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:26.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.192+0000 7f0372f79740 1 -- 172.21.15.90:0/965240164 shutdown_connections 2024-03-24T01:27:26.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.192+0000 7f0372f79740 1 -- 172.21.15.90:0/965240164 wait complete. 2024-03-24T01:27:26.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.192+0000 7f0372f79740 20 client.6393 trim_cache size 0 max 16384 2024-03-24T01:27:26.197 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-24T01:27:26.197 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "24712165-9839-4fcc-8033-988386efea74", 2024-03-24T01:27:26.197 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 0, 2024-03-24T01:27:26.197 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 6222, 2024-03-24T01:27:26.197 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-03-24T01:27:26.197 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-03-24T01:27:26.197 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-03-24T01:27:26.198 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 11.656877642, 2024-03-24T01:27:26.198 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 34, 2024-03-24T01:27:26.198 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 53, 2024-03-24T01:27:26.198 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 53, 2024-03-24T01:27:26.198 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 17.084087308000001 2024-03-24T01:27:26.198 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-24T01:27:26.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.193+0000 7f036ec00640 1 -- 172.21.15.90:0/1644085716 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f034c06bbd0 msgr2=0x7f034c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:26.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.193+0000 7f036ec00640 1 --2- 172.21.15.90:0/1644085716 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f034c06bbd0 0x7f034c06e090 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f035c000f60 tx=0x7f035c0079b0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.193+0000 7f036ec00640 1 -- 172.21.15.90:0/1644085716 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f036815de50 msgr2=0x7f03681f34a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:26.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.193+0000 7f036ec00640 1 --2- 172.21.15.90:0/1644085716 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f036815de50 0x7f03681f34a0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f036000c820 tx=0x7f036000cce0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.193+0000 7f036ec00640 1 -- 172.21.15.90:0/1644085716 shutdown_connections 2024-03-24T01:27:26.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.193+0000 7f036ec00640 1 --2- 172.21.15.90:0/1644085716 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f034c06bbd0 0x7f034c06e090 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.193+0000 7f036ec00640 1 --2- 172.21.15.90:0/1644085716 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f036815de50 0x7f03681f34a0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.193+0000 7f036ec00640 1 --2- 172.21.15.90:0/1644085716 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f03680d35b0 0x7f03680c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.193+0000 7f036ec00640 1 --2- 172.21.15.90:0/1644085716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03680b5550 0x7f036815e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.193+0000 7f036ec00640 1 -- 172.21.15.90:0/1644085716 >> 172.21.15.90:0/1644085716 conn(0x7f03680d0550 msgr2=0x7f036806cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:26.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.193+0000 7f036ec00640 1 -- 172.21.15.90:0/1644085716 shutdown_connections 2024-03-24T01:27:26.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.193+0000 7f036ec00640 1 -- 172.21.15.90:0/1644085716 wait complete. 2024-03-24T01:27:26.211 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T01:27:26.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.404+0000 7f4d46000640 1 Processor -- start 2024-03-24T01:27:26.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.404+0000 7f4d46000640 1 -- start start 2024-03-24T01:27:26.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.405+0000 7f4d46000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d401603e0 0x7f4d4006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:26.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.406+0000 7f4d46000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4d40145430 0x7f4d4006b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:26.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.406+0000 7f4d46000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d400cd490 con 0x7f4d400cc150 2024-03-24T01:27:26.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.406+0000 7f4d46000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d400cd270 con 0x7f4d40145430 2024-03-24T01:27:26.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.406+0000 7f4d46000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4015f900 con 0x7f4d401603e0 2024-03-24T01:27:26.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.406+0000 7f4d3f400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4d400cc150 0x7f4d400ccb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:34688/0 (socket says 172.21.15.90:34688) 2024-03-24T01:27:26.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.406+0000 7f4d3f400640 1 -- 172.21.15.90:0/3679926038 learned_addr learned my addr 172.21.15.90:0/3679926038 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:27:26.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.406+0000 7f4d37e00640 1 --2- 172.21.15.90:0/3679926038 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d401603e0 0x7f4d4006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:26.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.406+0000 7f4d3fe00640 1 --2- 172.21.15.90:0/3679926038 >> v2:172.21.15.120:3300/0 conn(0x7f4d40145430 0x7f4d4006b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:26.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.406+0000 7f4d3ea00640 1 -- 172.21.15.90:0/3679926038 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 440294444 0 0) 0x7f4d400cd490 con 0x7f4d400cc150 2024-03-24T01:27:26.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.406+0000 7f4d3ea00640 1 -- 172.21.15.90:0/3679926038 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d24003520 con 0x7f4d400cc150 2024-03-24T01:27:26.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.407+0000 7f4d3ea00640 1 -- 172.21.15.90:0/3679926038 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3556462451 0 0) 0x7f4d24003520 con 0x7f4d400cc150 2024-03-24T01:27:26.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.407+0000 7f4d3ea00640 1 -- 172.21.15.90:0/3679926038 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d401603e0 msgr2=0x7f4d4006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:26.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.407+0000 7f4d3ea00640 1 --2- 172.21.15.90:0/3679926038 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d401603e0 0x7f4d4006b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.407+0000 7f4d3ea00640 1 -- 172.21.15.90:0/3679926038 >> v2:172.21.15.120:3300/0 conn(0x7f4d40145430 msgr2=0x7f4d4006b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:26.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.407+0000 7f4d3ea00640 1 --2- 172.21.15.90:0/3679926038 >> v2:172.21.15.120:3300/0 conn(0x7f4d40145430 0x7f4d4006b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.407+0000 7f4d3ea00640 1 -- 172.21.15.90:0/3679926038 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d4015fbe0 con 0x7f4d400cc150 2024-03-24T01:27:26.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.407+0000 7f4d3ea00640 1 -- 172.21.15.90:0/3679926038 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f4d30001670 con 0x7f4d400cc150 2024-03-24T01:27:26.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.407+0000 7f4d3ea00640 1 -- 172.21.15.90:0/3679926038 >> v1:172.21.15.90:6789/0 conn(0x7f4d400cc150 legacy=0x7f4d400ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:26.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.407+0000 7f4d3ea00640 1 --2- 172.21.15.90:0/3679926038 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d401603e0 0x7f4d240039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:26.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.407+0000 7f4d3ea00640 1 --2- 172.21.15.90:0/3679926038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d24003f80 0x7f4d24004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:26.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.407+0000 7f4d3ea00640 1 --2- 172.21.15.90:0/3679926038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d24004d40 0x7f4d240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:26.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.407+0000 7f4d3ea00640 1 -- 172.21.15.90:0/3679926038 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d400cd270 con 0x7f4d24003f80 2024-03-24T01:27:26.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.407+0000 7f4d3ea00640 1 -- 172.21.15.90:0/3679926038 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4015f900 con 0x7f4d24004d40 2024-03-24T01:27:26.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.407+0000 7f4d3ea00640 1 -- 172.21.15.90:0/3679926038 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2400b9c0 con 0x7f4d401603e0 2024-03-24T01:27:26.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.407+0000 7f4d37e00640 1 --2- 172.21.15.90:0/3679926038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d24003f80 0x7f4d24004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:26.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.407+0000 7f4d3f400640 1 --2- 172.21.15.90:0/3679926038 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d401603e0 0x7f4d240039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:26.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.408+0000 7f4d3fe00640 1 --2- 172.21.15.90:0/3679926038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d24004d40 0x7f4d240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:26.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.408+0000 7f4d37e00640 1 -- 172.21.15.90:0/3679926038 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d401603e0 msgr2=0x7f4d240039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:26.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.408+0000 7f4d37e00640 1 --2- 172.21.15.90:0/3679926038 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d401603e0 0x7f4d240039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.408+0000 7f4d37e00640 1 -- 172.21.15.90:0/3679926038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d24004d40 msgr2=0x7f4d240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:27:26.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.408+0000 7f4d37e00640 1 --2- 172.21.15.90:0/3679926038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d24004d40 0x7f4d240051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.408+0000 7f4d37e00640 1 -- 172.21.15.90:0/3679926038 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d24003520 con 0x7f4d24003f80 2024-03-24T01:27:26.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.408+0000 7f4d3fe00640 1 --2- 172.21.15.90:0/3679926038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d24004d40 0x7f4d240051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:27:26.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.409+0000 7f4d37e00640 1 --2- 172.21.15.90:0/3679926038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d24003f80 0x7f4d24004800 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f4d28009af0 tx=0x7f4d24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81458a7ff3fb977f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:26.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.409+0000 7f4d3ea00640 1 -- 172.21.15.90:0/3679926038 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d28039070 con 0x7f4d24003f80 2024-03-24T01:27:26.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.409+0000 7f4d3ea00640 1 -- 172.21.15.90:0/3679926038 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d2802c6b0 con 0x7f4d24003f80 2024-03-24T01:27:26.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.409+0000 7f4d3ea00640 1 -- 172.21.15.90:0/3679926038 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d28009690 con 0x7f4d24003f80 2024-03-24T01:27:26.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.409+0000 7f4d46000640 1 -- 172.21.15.90:0/3679926038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d24003f80 msgr2=0x7f4d24004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:26.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.409+0000 7f4d46000640 1 --2- 172.21.15.90:0/3679926038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d24003f80 0x7f4d24004800 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f4d28009af0 tx=0x7f4d24005830 comp rx=0 tx=0).stop 2024-03-24T01:27:26.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.409+0000 7f4d3fe00640 1 -- 172.21.15.90:0/3679926038 reap_dead start 2024-03-24T01:27:26.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.410+0000 7f4d46000640 1 -- 172.21.15.90:0/3679926038 shutdown_connections 2024-03-24T01:27:26.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.410+0000 7f4d46000640 1 -- 172.21.15.90:0/3679926038 >> 172.21.15.90:0/3679926038 conn(0x7f4d400b1770 msgr2=0x7f4d400b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:26.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.410+0000 7f4d46000640 1 -- 172.21.15.90:0/3679926038 shutdown_connections 2024-03-24T01:27:26.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.410+0000 7f4d46000640 1 -- 172.21.15.90:0/3679926038 wait complete. 2024-03-24T01:27:26.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.411+0000 7f4d46000640 1 Processor -- start 2024-03-24T01:27:26.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.411+0000 7f4d46000640 1 -- start start 2024-03-24T01:27:26.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.412+0000 7f4d46000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d24003f80 0x7f4d4015d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:26.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.412+0000 7f4d46000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d4013dd90 0x7f4d4015cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:26.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.412+0000 7f4d46000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d4015abc0 0x7f4d4015b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:26.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.412+0000 7f4d46000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4006b240 con 0x7f4d24003f80 2024-03-24T01:27:26.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.412+0000 7f4d46000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4006b0c0 con 0x7f4d4015abc0 2024-03-24T01:27:26.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.412+0000 7f4d46000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4006b3c0 con 0x7f4d4013dd90 2024-03-24T01:27:26.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.413+0000 7f4d3f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d24003f80 0x7f4d4015d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:26.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.413+0000 7f4d3f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d24003f80 0x7f4d4015d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57752/0 (socket says 172.21.15.90:57752) 2024-03-24T01:27:26.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.413+0000 7f4d3f400640 1 -- 172.21.15.90:0/1137346474 learned_addr learned my addr 172.21.15.90:0/1137346474 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:27:26.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.413+0000 7f4d37e00640 1 --2- 172.21.15.90:0/1137346474 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d4013dd90 0x7f4d4015cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:26.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.413+0000 7f4d3f400640 1 -- 172.21.15.90:0/1137346474 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d4013dd90 msgr2=0x7f4d4015cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:26.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.413+0000 7f4d3f400640 1 --2- 172.21.15.90:0/1137346474 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d4013dd90 0x7f4d4015cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.413+0000 7f4d3f400640 1 -- 172.21.15.90:0/1137346474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d4015abc0 msgr2=0x7f4d4015b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:26.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.413+0000 7f4d3f400640 1 --2- 172.21.15.90:0/1137346474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d4015abc0 0x7f4d4015b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.413+0000 7f4d3f400640 1 -- 172.21.15.90:0/1137346474 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d4013e210 con 0x7f4d24003f80 2024-03-24T01:27:26.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.413+0000 7f4d3f400640 1 --2- 172.21.15.90:0/1137346474 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d24003f80 0x7f4d4015d460 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f4d3000e730 tx=0x7f4d3000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:26.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.414+0000 7f4d37400640 1 -- 172.21.15.90:0/1137346474 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3000ee40 con 0x7f4d24003f80 2024-03-24T01:27:26.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.414+0000 7f4d37400640 1 -- 172.21.15.90:0/1137346474 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d30018070 con 0x7f4d24003f80 2024-03-24T01:27:26.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.414+0000 7f4d46000640 1 -- 172.21.15.90:0/1137346474 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d4015db60 con 0x7f4d24003f80 2024-03-24T01:27:26.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.414+0000 7f4d37400640 1 -- 172.21.15.90:0/1137346474 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3000ca70 con 0x7f4d24003f80 2024-03-24T01:27:26.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.414+0000 7f4d46000640 1 -- 172.21.15.90:0/1137346474 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d4015c140 con 0x7f4d24003f80 2024-03-24T01:27:26.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.415+0000 7f4d46000640 1 -- 172.21.15.90:0/1137346474 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d40068060 con 0x7f4d24003f80 2024-03-24T01:27:26.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.416+0000 7f4d37400640 1 -- 172.21.15.90:0/1137346474 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4d3000cc10 con 0x7f4d24003f80 2024-03-24T01:27:26.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.417+0000 7f4d37400640 1 --2- 172.21.15.90:0/1137346474 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4d1c06bc00 0x7f4d1c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:27:26.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.417+0000 7f4d37400640 1 -- 172.21.15.90:0/1137346474 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4d3008ab20 con 0x7f4d24003f80 2024-03-24T01:27:26.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.417+0000 7f4d37e00640 1 --2- 172.21.15.90:0/1137346474 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4d1c06bc00 0x7f4d1c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:27:26.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.418+0000 7f4d37e00640 1 --2- 172.21.15.90:0/1137346474 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4d1c06bc00 0x7f4d1c06e0c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f4d28036000 tx=0x7f4d28030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:27:26.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.421+0000 7f4d37400640 1 -- 172.21.15.90:0/1137346474 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4d300593e0 con 0x7f4d24003f80 2024-03-24T01:27:26.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.642+0000 7f4d46000640 1 -- 172.21.15.90:0/1137346474 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f4d400b3220 con 0x7f4d24003f80 2024-03-24T01:27:26.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.642+0000 7f4d37400640 1 -- 172.21.15.90:0/1137346474 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v34) v1 ==== 53+0+73 (secure 0 0 0) 0x7f4d30058ce0 con 0x7f4d24003f80 2024-03-24T01:27:26.644 INFO:teuthology.orchestra.run.smithi090.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T01:27:26.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.645+0000 7f4d46000640 1 -- 172.21.15.90:0/1137346474 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4d1c06bc00 msgr2=0x7f4d1c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:26.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.645+0000 7f4d46000640 1 --2- 172.21.15.90:0/1137346474 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4d1c06bc00 0x7f4d1c06e0c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f4d28036000 tx=0x7f4d28030000 comp rx=0 tx=0).stop 2024-03-24T01:27:26.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.645+0000 7f4d46000640 1 -- 172.21.15.90:0/1137346474 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d24003f80 msgr2=0x7f4d4015d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:27:26.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.645+0000 7f4d46000640 1 --2- 172.21.15.90:0/1137346474 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d24003f80 0x7f4d4015d460 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f4d3000e730 tx=0x7f4d3000ebf0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.645+0000 7f4d46000640 1 -- 172.21.15.90:0/1137346474 shutdown_connections 2024-03-24T01:27:26.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.645+0000 7f4d46000640 1 --2- 172.21.15.90:0/1137346474 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4d1c06bc00 0x7f4d1c06e0c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.645+0000 7f4d46000640 1 --2- 172.21.15.90:0/1137346474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d4015abc0 0x7f4d4015b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.645+0000 7f4d46000640 1 --2- 172.21.15.90:0/1137346474 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4d4013dd90 0x7f4d4015cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.645+0000 7f4d46000640 1 --2- 172.21.15.90:0/1137346474 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d24003f80 0x7f4d4015d460 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:27:26.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.645+0000 7f4d46000640 1 -- 172.21.15.90:0/1137346474 >> 172.21.15.90:0/1137346474 conn(0x7f4d400b1770 msgr2=0x7f4d40143550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:27:26.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.645+0000 7f4d46000640 1 -- 172.21.15.90:0/1137346474 shutdown_connections 2024-03-24T01:27:26.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:26.645+0000 7f4d46000640 1 -- 172.21.15.90:0/1137346474 wait complete. 2024-03-24T01:27:26.664 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T01:27:26.664 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T01:27:26.664 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi090.front.sepia.ceph.com 2024-03-24T01:27:26.664 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T01:27:26.664 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T01:27:26.664 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T01:27:26.664 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T01:27:26.664 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T01:27:26.664 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T01:27:26.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:27:26.664 DEBUG:teuthology.orchestra.run.smithi090:> ip addr 2024-03-24T01:27:26.716 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T01:27:26.716 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T01:27:26.716 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T01:27:26.717 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:27:26.717 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-03-24T01:27:26.717 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:27:26.717 INFO:teuthology.orchestra.run.smithi090.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T01:27:26.717 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a8 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:27:26.717 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f0 2024-03-24T01:27:26.717 INFO:teuthology.orchestra.run.smithi090.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T01:27:26.717 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a9 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:27:26.717 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f1 2024-03-24T01:27:26.717 INFO:teuthology.orchestra.run.smithi090.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T01:27:26.717 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:66 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:27:26.717 INFO:teuthology.orchestra.run.smithi090.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T01:27:26.717 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:67 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:27:26.717 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.21.15.90/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T01:27:26.717 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft 42335sec preferred_lft 42335sec 2024-03-24T01:27:26.718 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::ec4:7aff:fe8f:d767/64 scope link 2024-03-24T01:27:26.718 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:27:26.718 INFO:teuthology.orchestra.run.smithi090.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T01:27:26.718 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 06:50:1d:bd:ed:70 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:27:26.718 INFO:teuthology.orchestra.run.smithi090.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T01:27:26.718 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:27:26.718 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::a05b:d3ff:fea5:6fd0/64 scope link 2024-03-24T01:27:26.718 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:27:26.718 INFO:teuthology.orchestra.run.smithi090.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T01:27:26.718 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether de:94:0b:7e:c6:41 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:27:26.718 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::dc94:bff:fe7e:c641/64 scope link 2024-03-24T01:27:26.718 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:27:26.718 INFO:teuthology.orchestra.run.smithi090.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T01:27:26.718 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 06:50:1d:bd:ed:70 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:27:26.719 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::450:1dff:febd:ed70/64 scope link 2024-03-24T01:27:26.719 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:27:26.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:27:26.719 DEBUG:teuthology.orchestra.run.smithi090:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T01:27:26.801 INFO:teuthology.orchestra.run.smithi090.stdout:1 2024-03-24T01:27:26.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:27:26.805 DEBUG:teuthology.orchestra.run.smithi090:> route 2024-03-24T01:27:26.858 INFO:teuthology.orchestra.run.smithi090.stdout:Kernel IP routing table 2024-03-24T01:27:26.858 INFO:teuthology.orchestra.run.smithi090.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T01:27:26.858 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T01:27:26.859 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T01:27:26.859 INFO:teuthology.orchestra.run.smithi090.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T01:27:26.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:27:26.859 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2024-03-24T01:27:26.859 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-24T01:27:26.859 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T01:27:26.859 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T01:27:26.859 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T01:27:26.859 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-24T01:27:27.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:27:27.039 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-03-24T01:27:27.087 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T01:27:27.087 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T01:27:27.088 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T01:27:27.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:27:27.088 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set brx.0 up 2024-03-24T01:27:27.145 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:27:27.151 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:27:27.170 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-24T01:27:27.170 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T01:27:27.170 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T01:27:27.258 INFO:teuthology.orchestra.run.smithi090.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T01:27:27.258 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T01:27:27.259 DEBUG:teuthology.orchestra.run.smithi090:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T01:27:27.361 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-24T01:27:27.459 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T01:27:27.513 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-24T01:27:27.513 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-24T01:27:27.513 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-24T01:27:27.513 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-24T01:27:27.514 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-24T01:27:27.514 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-24T01:27:27.514 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-24T01:27:27.514 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-24T01:27:27.514 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-24T01:27:27.514 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-24T01:27:27.514 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-24T01:27:27.514 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/selinux 2024-03-24T01:27:27.514 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T01:27:27.514 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-24T01:27:27.514 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-24T01:27:27.514 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-24T01:27:27.514 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-24T01:27:27.514 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T01:27:27.515 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-24T01:27:27.515 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T01:27:27.515 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-24T01:27:27.515 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T01:27:27.515 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T01:27:27.515 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-24T01:27:27.515 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-24T01:27:27.515 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T01:27:27.515 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T01:27:27.515 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T01:27:27.515 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T01:27:27.515 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-24T01:27:27.515 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:27:27.515 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:27:27.515 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:27:27.515 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:27:27.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:27:27.516 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-24T01:27:27.568 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-24T01:27:27.568 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-24T01:27:27.574 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-24T01:27:27.645 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T01:27:27.685 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-24T01:27:27.685 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-24T01:27:27.685 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-24T01:27:27.685 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-24T01:27:27.685 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-24T01:27:27.686 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-24T01:27:27.686 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-24T01:27:27.686 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-24T01:27:27.686 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-24T01:27:27.686 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-24T01:27:27.686 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-24T01:27:27.686 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/selinux 2024-03-24T01:27:27.686 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T01:27:27.686 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-24T01:27:27.686 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-24T01:27:27.686 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-24T01:27:27.686 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-24T01:27:27.686 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T01:27:27.687 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-24T01:27:27.687 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T01:27:27.687 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-24T01:27:27.687 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T01:27:27.687 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T01:27:27.687 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-24T01:27:27.687 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-24T01:27:27.687 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T01:27:27.687 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T01:27:27.687 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T01:27:27.687 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T01:27:27.687 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-24T01:27:27.687 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:27:27.687 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:27:27.687 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:27:27.688 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:27:27.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:27:27.689 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-24T01:27:27.762 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:2024-03-24T01:27:27.761+0000 7f679d249480 -1 init, newargv = 0x55a0d9073030 newargc=15 2024-03-24T01:27:27.762 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[59444]: starting ceph client 2024-03-24T01:27:27.772 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[59444]: starting fuse 2024-03-24T01:27:27.777 INFO:teuthology.orchestra.run.smithi090.stdout:40 2024-03-24T01:27:27.778 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-03-24T01:27:27.778 DEBUG:teuthology.orchestra.run.smithi090:> sudo stdin-killer -- python3 -c ' 2024-03-24T01:27:27.778 DEBUG:teuthology.orchestra.run.smithi090:> import glob 2024-03-24T01:27:27.778 DEBUG:teuthology.orchestra.run.smithi090:> import re 2024-03-24T01:27:27.778 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-24T01:27:27.778 DEBUG:teuthology.orchestra.run.smithi090:> import subprocess 2024-03-24T01:27:27.778 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:27:27.778 DEBUG:teuthology.orchestra.run.smithi090:> def _find_admin_socket(client_name): 2024-03-24T01:27:27.778 DEBUG:teuthology.orchestra.run.smithi090:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-24T01:27:27.779 DEBUG:teuthology.orchestra.run.smithi090:> files = glob.glob(asok_path) 2024-03-24T01:27:27.779 DEBUG:teuthology.orchestra.run.smithi090:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-24T01:27:27.779 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:27:27.779 DEBUG:teuthology.orchestra.run.smithi090:> # Given a non-glob path, it better be there 2024-03-24T01:27:27.779 DEBUG:teuthology.orchestra.run.smithi090:> if "*" not in asok_path: 2024-03-24T01:27:27.779 DEBUG:teuthology.orchestra.run.smithi090:> assert(len(files) == 1) 2024-03-24T01:27:27.779 DEBUG:teuthology.orchestra.run.smithi090:> return files[0] 2024-03-24T01:27:27.779 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:27:27.779 DEBUG:teuthology.orchestra.run.smithi090:> for f in files: 2024-03-24T01:27:27.779 DEBUG:teuthology.orchestra.run.smithi090:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T01:27:27.779 DEBUG:teuthology.orchestra.run.smithi090:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T01:27:27.779 DEBUG:teuthology.orchestra.run.smithi090:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T01:27:27.779 DEBUG:teuthology.orchestra.run.smithi090:> contents = proc_f.read() 2024-03-24T01:27:27.779 DEBUG:teuthology.orchestra.run.smithi090:> if mountpoint in contents: 2024-03-24T01:27:27.779 DEBUG:teuthology.orchestra.run.smithi090:> return f 2024-03-24T01:27:27.779 DEBUG:teuthology.orchestra.run.smithi090:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T01:27:27.780 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:27:27.780 DEBUG:teuthology.orchestra.run.smithi090:> print(_find_admin_socket("client.0")) 2024-03-24T01:27:27.780 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-24T01:27:27.999 INFO:teuthology.orchestra.run.smithi090.stdout:/var/run/ceph/ceph-client.0.59444.asok 2024-03-24T01:27:28.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:27:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T01:27:28.016 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.59444.asok 2024-03-24T01:27:28.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:27:28.016 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.59444.asok status 2024-03-24T01:27:28.253 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-24T01:27:28.253 INFO:teuthology.orchestra.run.smithi090.stdout: "metadata": { 2024-03-24T01:27:28.253 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_sha1": "3be9a5c9bc793e11e2800a8c0c696e8b46742033", 2024-03-24T01:27:28.253 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-24T01:27:28.253 INFO:teuthology.orchestra.run.smithi090.stdout: "entity_id": "0", 2024-03-24T01:27:28.253 INFO:teuthology.orchestra.run.smithi090.stdout: "hostname": "smithi090", 2024-03-24T01:27:28.253 INFO:teuthology.orchestra.run.smithi090.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-24T01:27:28.253 INFO:teuthology.orchestra.run.smithi090.stdout: "pid": "59444", 2024-03-24T01:27:28.254 INFO:teuthology.orchestra.run.smithi090.stdout: "root": "/" 2024-03-24T01:27:28.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:27:28.254 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_count": 0, 2024-03-24T01:27:28.254 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_pinned_count": 0, 2024-03-24T01:27:28.254 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 6405, 2024-03-24T01:27:28.254 INFO:teuthology.orchestra.run.smithi090.stdout: "inst": { 2024-03-24T01:27:28.254 INFO:teuthology.orchestra.run.smithi090.stdout: "name": { 2024-03-24T01:27:28.254 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "client", 2024-03-24T01:27:28.254 INFO:teuthology.orchestra.run.smithi090.stdout: "num": 6405 2024-03-24T01:27:28.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:27:28.254 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-24T01:27:28.254 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-24T01:27:28.254 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-03-24T01:27:28.254 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3003695619 2024-03-24T01:27:28.255 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:27:28.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:27:28.255 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-24T01:27:28.255 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-24T01:27:28.255 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-03-24T01:27:28.255 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3003695619 2024-03-24T01:27:28.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:27:28.255 INFO:teuthology.orchestra.run.smithi090.stdout: "inst_str": "client.6405 192.168.0.1:0/3003695619", 2024-03-24T01:27:28.255 INFO:teuthology.orchestra.run.smithi090.stdout: "addr_str": "192.168.0.1:0/3003695619", 2024-03-24T01:27:28.255 INFO:teuthology.orchestra.run.smithi090.stdout: "inode_count": 1, 2024-03-24T01:27:28.255 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_epoch": 34, 2024-03-24T01:27:28.255 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch": 53, 2024-03-24T01:27:28.255 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch_barrier": 0, 2024-03-24T01:27:28.255 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklisted": false, 2024-03-24T01:27:28.256 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs" 2024-03-24T01:27:28.256 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-24T01:27:28.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:27:28.268 DEBUG:teuthology.orchestra.run.smithi090:> stat --file-system '--printf=%T 2024-03-24T01:27:28.268 DEBUG:teuthology.orchestra.run.smithi090:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-24T01:27:28.322 INFO:teuthology.orchestra.run.smithi090.stdout:fuseblk 2024-03-24T01:27:28.322 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-24T01:27:28.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:27:28.322 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T01:27:28.416 INFO:tasks.workunit:Pulling workunits from ref 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T01:27:28.416 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-24T01:27:28.417 DEBUG:teuthology.orchestra.run.smithi090:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-24T01:27:28.472 INFO:teuthology.orchestra.run.smithi090.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-24T01:27:28.472 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-03-24T01:27:28.472 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 28h/40d Inode: 1 Links: 2 2024-03-24T01:27:28.472 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T01:27:28.472 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-24T01:27:28.472 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-03-24T01:27:28.472 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-03-24 01:27:14.539944686 +0000 2024-03-24T01:27:28.472 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-03-24 01:27:28.410600515 +0000 2024-03-24T01:27:28.472 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: - 2024-03-24T01:27:28.472 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-24T01:27:28.473 DEBUG:teuthology.orchestra.run.smithi090:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-24T01:27:28.588 INFO:tasks.workunit:timeout=1h 2024-03-24T01:27:28.588 INFO:tasks.workunit:cleanup=True 2024-03-24T01:27:28.589 DEBUG:teuthology.orchestra.run.smithi090:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T01:27:28.648 INFO:tasks.workunit.client.0.smithi090.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-24T01:27:57.276 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:27:57.286 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:28:27.417 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:28:27.424 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:28:51.413 INFO:tasks.workunit.client.0.smithi090.stderr:Updating files: 63% (7845/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-24T01:28:51.984 INFO:tasks.workunit.client.0.smithi090.stderr:Note: switching to '3be9a5c9bc793e11e2800a8c0c696e8b46742033'. 2024-03-24T01:28:51.984 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-03-24T01:28:51.984 INFO:tasks.workunit.client.0.smithi090.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-24T01:28:51.985 INFO:tasks.workunit.client.0.smithi090.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-24T01:28:51.985 INFO:tasks.workunit.client.0.smithi090.stderr:state without impacting any branches by switching back to a branch. 2024-03-24T01:28:51.985 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-03-24T01:28:51.985 INFO:tasks.workunit.client.0.smithi090.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-24T01:28:51.985 INFO:tasks.workunit.client.0.smithi090.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-24T01:28:51.985 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-03-24T01:28:51.985 INFO:tasks.workunit.client.0.smithi090.stderr: git switch -c 2024-03-24T01:28:51.985 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-03-24T01:28:51.985 INFO:tasks.workunit.client.0.smithi090.stderr:Or undo this operation with: 2024-03-24T01:28:51.985 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-03-24T01:28:51.985 INFO:tasks.workunit.client.0.smithi090.stderr: git switch - 2024-03-24T01:28:51.985 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-03-24T01:28:51.985 INFO:tasks.workunit.client.0.smithi090.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-24T01:28:51.986 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-03-24T01:28:51.986 INFO:tasks.workunit.client.0.smithi090.stderr:HEAD is now at 3be9a5c9bc7 dummy commit #1 2024-03-24T01:28:51.993 DEBUG:teuthology.orchestra.run.smithi090:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-24T01:28:52.050 INFO:tasks.workunit.client.0.smithi090.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-24T01:28:52.056 INFO:tasks.workunit.client.0.smithi090.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-24T01:28:52.056 INFO:tasks.workunit.client.0.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-24T01:28:53.053 INFO:tasks.workunit.client.0.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-24T01:28:53.199 INFO:tasks.workunit.client.0.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-24T01:28:53.277 INFO:tasks.workunit.client.0.smithi090.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-24T01:28:53.282 INFO:tasks.workunit.client.0.smithi090.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-24T01:28:53.283 INFO:tasks.workunit.client.0.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-24T01:28:53.365 INFO:tasks.workunit.client.0.smithi090.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-24T01:28:53.375 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:28:53.375 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-24T01:28:53.425 INFO:tasks.workunit:Running workunits matching fs/misc/trivial_sync.sh on client.0... 2024-03-24T01:28:53.426 INFO:tasks.workunit:Running workunit fs/misc/trivial_sync.sh... 2024-03-24T01:28:53.426 DEBUG:teuthology.orchestra.run.smithi090:workunit test fs/misc/trivial_sync.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/trivial_sync.sh 2024-03-24T01:28:55.992 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-24T01:28:55.993 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-24T01:28:56.148 INFO:tasks.workunit:Stopping ['fs/misc/trivial_sync.sh'] on client.0... 2024-03-24T01:28:56.148 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-24T01:28:57.151 DEBUG:teuthology.parallel:result is None 2024-03-24T01:28:57.151 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-24T01:28:57.251 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-24T01:28:57.252 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:28:57.252 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:28:57.305 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:28:57.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.508+0000 7ff221a00640 1 Processor -- start 2024-03-24T01:28:57.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.508+0000 7ff221a00640 1 -- start start 2024-03-24T01:28:57.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.510+0000 7ff221a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff21c0cd6e0 0x7ff21c0cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:28:57.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.510+0000 7ff221a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff21c0ce190 0x7ff21c15c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:28:57.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.510+0000 7ff221a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff21c0cc660 con 0x7ff21c1454e0 2024-03-24T01:28:57.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.510+0000 7ff221a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff21c0cc430 con 0x7ff21c0ce190 2024-03-24T01:28:57.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.510+0000 7ff221a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff21c0cc920 con 0x7ff21c0cd6e0 2024-03-24T01:28:57.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.510+0000 7ff21b400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff21c0ce190 0x7ff21c15c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:28:57.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.510+0000 7ff21b400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff21c0ce190 0x7ff21c15c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54512/0 (socket says 172.21.15.90:54512) 2024-03-24T01:28:57.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.510+0000 7ff21b400640 1 -- 172.21.15.90:0/3470288731 learned_addr learned my addr 172.21.15.90:0/3470288731 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:28:57.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.510+0000 7ff21aa00640 1 --1- 172.21.15.90:0/3470288731 >> v1:172.21.15.90:6789/0 conn(0x7ff21c1454e0 0x7ff21c06f000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:32806/0 (socket says 172.21.15.90:32806) 2024-03-24T01:28:57.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.510+0000 7ff21a000640 1 --2- 172.21.15.90:0/3470288731 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff21c0cd6e0 0x7ff21c0cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:28:57.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.511+0000 7ff21b400640 1 -- 172.21.15.90:0/3470288731 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff21c0cd6e0 msgr2=0x7ff21c0cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:28:57.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.511+0000 7ff21b400640 1 --2- 172.21.15.90:0/3470288731 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff21c0cd6e0 0x7ff21c0cdc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:57.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.511+0000 7ff21b400640 1 -- 172.21.15.90:0/3470288731 >> v1:172.21.15.90:6789/0 conn(0x7ff21c1454e0 legacy=0x7ff21c06f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:28:57.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.511+0000 7ff21b400640 1 -- 172.21.15.90:0/3470288731 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff21c0ccc00 con 0x7ff21c0ce190 2024-03-24T01:28:57.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.511+0000 7ff219600640 1 -- 172.21.15.90:0/3470288731 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2970145121 0 0) 0x7ff21c0cc660 con 0x7ff21c1454e0 2024-03-24T01:28:57.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.512+0000 7ff21b400640 1 --2- 172.21.15.90:0/3470288731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff21c0ce190 0x7ff21c15c090 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7ff214009e50 tx=0x7ff21402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b50fe6c7f9b3d4ba server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:28:57.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.512+0000 7ff219600640 1 -- 172.21.15.90:0/3470288731 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff21402cb00 con 0x7ff21c0ce190 2024-03-24T01:28:57.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.512+0000 7ff219600640 1 -- 172.21.15.90:0/3470288731 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff21402cca0 con 0x7ff21c0ce190 2024-03-24T01:28:57.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.512+0000 7ff219600640 1 -- 172.21.15.90:0/3470288731 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff214034610 con 0x7ff21c0ce190 2024-03-24T01:28:57.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.512+0000 7ff221a00640 1 -- 172.21.15.90:0/3470288731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff21c0ce190 msgr2=0x7ff21c15c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:28:57.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.512+0000 7ff221a00640 1 --2- 172.21.15.90:0/3470288731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff21c0ce190 0x7ff21c15c090 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7ff214009e50 tx=0x7ff21402bf60 comp rx=0 tx=0).stop 2024-03-24T01:28:57.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.513+0000 7ff221a00640 1 -- 172.21.15.90:0/3470288731 shutdown_connections 2024-03-24T01:28:57.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.513+0000 7ff221a00640 1 --2- 172.21.15.90:0/3470288731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff21c0ce190 0x7ff21c15c090 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:57.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.513+0000 7ff221a00640 1 --2- 172.21.15.90:0/3470288731 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff21c0cd6e0 0x7ff21c0cdc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:57.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.513+0000 7ff221a00640 1 -- 172.21.15.90:0/3470288731 >> 172.21.15.90:0/3470288731 conn(0x7ff21c06b2a0 msgr2=0x7ff21c06b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:28:57.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.513+0000 7ff221a00640 1 -- 172.21.15.90:0/3470288731 shutdown_connections 2024-03-24T01:28:57.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.513+0000 7ff221a00640 1 -- 172.21.15.90:0/3470288731 wait complete. 2024-03-24T01:28:57.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.514+0000 7ff221a00640 1 Processor -- start 2024-03-24T01:28:57.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.515+0000 7ff221a00640 1 -- start start 2024-03-24T01:28:57.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.515+0000 7ff221a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff21c0cd6e0 0x7ff21c156e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:28:57.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.515+0000 7ff221a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff21c0ce190 0x7ff21c0ae1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:28:57.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.515+0000 7ff221a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff21c1454e0 0x7ff21c15bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:28:57.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.515+0000 7ff221a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21c06f710 con 0x7ff21c0cd6e0 2024-03-24T01:28:57.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.515+0000 7ff221a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21c0cd1c0 con 0x7ff21c0ce190 2024-03-24T01:28:57.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.515+0000 7ff221a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff21c06f890 con 0x7ff21c1454e0 2024-03-24T01:28:57.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.516+0000 7ff21aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff21c0cd6e0 0x7ff21c156e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:28:57.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.516+0000 7ff21aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff21c0cd6e0 0x7ff21c156e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54516/0 (socket says 172.21.15.90:54516) 2024-03-24T01:28:57.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.516+0000 7ff21aa00640 1 -- 172.21.15.90:0/3023250546 learned_addr learned my addr 172.21.15.90:0/3023250546 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:28:57.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.516+0000 7ff21aa00640 1 -- 172.21.15.90:0/3023250546 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff21c1454e0 msgr2=0x7ff21c15bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:28:57.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.516+0000 7ff21aa00640 1 --2- 172.21.15.90:0/3023250546 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff21c1454e0 0x7ff21c15bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:57.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.516+0000 7ff21aa00640 1 -- 172.21.15.90:0/3023250546 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff21c0ce190 msgr2=0x7ff21c0ae1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:28:57.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.516+0000 7ff21aa00640 1 --2- 172.21.15.90:0/3023250546 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff21c0ce190 0x7ff21c0ae1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:57.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.516+0000 7ff21aa00640 1 -- 172.21.15.90:0/3023250546 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff21c1573b0 con 0x7ff21c0cd6e0 2024-03-24T01:28:57.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.516+0000 7ff21aa00640 1 --2- 172.21.15.90:0/3023250546 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff21c0cd6e0 0x7ff21c156e70 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7ff20c00c4d0 tx=0x7ff20c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:28:57.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.517+0000 7ff212a00640 1 -- 172.21.15.90:0/3023250546 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20c018070 con 0x7ff21c0cd6e0 2024-03-24T01:28:57.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.517+0000 7ff212a00640 1 -- 172.21.15.90:0/3023250546 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff20c0074e0 con 0x7ff21c0cd6e0 2024-03-24T01:28:57.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.517+0000 7ff212a00640 1 -- 172.21.15.90:0/3023250546 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20c004a20 con 0x7ff21c0cd6e0 2024-03-24T01:28:57.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.517+0000 7ff221a00640 1 -- 172.21.15.90:0/3023250546 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff21c13d230 con 0x7ff21c0cd6e0 2024-03-24T01:28:57.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.517+0000 7ff221a00640 1 -- 172.21.15.90:0/3023250546 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff21c13d7f0 con 0x7ff21c0cd6e0 2024-03-24T01:28:57.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.518+0000 7ff221a00640 1 -- 172.21.15.90:0/3023250546 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff21c13cfc0 con 0x7ff21c0cd6e0 2024-03-24T01:28:57.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.519+0000 7ff212a00640 1 -- 172.21.15.90:0/3023250546 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff20c004bc0 con 0x7ff21c0cd6e0 2024-03-24T01:28:57.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.520+0000 7ff212a00640 1 --2- 172.21.15.90:0/3023250546 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff1fc06bbd0 0x7ff1fc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:28:57.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.520+0000 7ff212a00640 1 -- 172.21.15.90:0/3023250546 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6159+0+0 (secure 0 0 0) 0x7ff20c089ff0 con 0x7ff21c0cd6e0 2024-03-24T01:28:57.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.520+0000 7ff21a000640 1 --2- 172.21.15.90:0/3023250546 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff1fc06bbd0 0x7ff1fc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:28:57.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.521+0000 7ff21a000640 1 --2- 172.21.15.90:0/3023250546 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff1fc06bbd0 0x7ff1fc06e090 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff208006fd0 tx=0x7ff208008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:28:57.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.525+0000 7ff212a00640 1 -- 172.21.15.90:0/3023250546 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff21c1573b0 con 0x7ff21c0cd6e0 2024-03-24T01:28:57.562 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:28:57.566 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:28:57.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.725+0000 7ff221a00640 1 -- 172.21.15.90:0/3023250546 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff21c04f4e0 con 0x7ff21c0cd6e0 2024-03-24T01:28:57.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.728+0000 7ff212a00640 1 -- 172.21.15.90:0/3023250546 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+16958 (secure 0 0 0) 0x7ff20c058850 con 0x7ff21c0cd6e0 2024-03-24T01:28:57.728 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:28:57.728 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":55,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:27:54.129944+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:27:11.774313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-24T01:27:12.992083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.23","mappings":[{"from":0,"to":1}]},{"pgid":"8.37","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:28:57.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.730+0000 7ff221a00640 1 -- 172.21.15.90:0/3023250546 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff1fc06bbd0 msgr2=0x7ff1fc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:28:57.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.730+0000 7ff221a00640 1 --2- 172.21.15.90:0/3023250546 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff1fc06bbd0 0x7ff1fc06e090 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff208006fd0 tx=0x7ff208008040 comp rx=0 tx=0).stop 2024-03-24T01:28:57.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.730+0000 7ff221a00640 1 -- 172.21.15.90:0/3023250546 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff21c0cd6e0 msgr2=0x7ff21c156e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:28:57.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.730+0000 7ff221a00640 1 --2- 172.21.15.90:0/3023250546 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff21c0cd6e0 0x7ff21c156e70 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7ff20c00c4d0 tx=0x7ff20c00c990 comp rx=0 tx=0).stop 2024-03-24T01:28:57.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.730+0000 7ff221a00640 1 -- 172.21.15.90:0/3023250546 shutdown_connections 2024-03-24T01:28:57.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.730+0000 7ff221a00640 1 --2- 172.21.15.90:0/3023250546 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff1fc06bbd0 0x7ff1fc06e090 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:57.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.730+0000 7ff221a00640 1 --2- 172.21.15.90:0/3023250546 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff21c1454e0 0x7ff21c15bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:57.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.730+0000 7ff221a00640 1 --2- 172.21.15.90:0/3023250546 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff21c0ce190 0x7ff21c0ae1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:57.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.730+0000 7ff221a00640 1 --2- 172.21.15.90:0/3023250546 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff21c0cd6e0 0x7ff21c156e70 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:57.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.730+0000 7ff221a00640 1 -- 172.21.15.90:0/3023250546 >> 172.21.15.90:0/3023250546 conn(0x7ff21c06b2a0 msgr2=0x7ff21c06d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:28:57.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.730+0000 7ff221a00640 1 -- 172.21.15.90:0/3023250546 shutdown_connections 2024-03-24T01:28:57.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.731+0000 7ff221a00640 1 -- 172.21.15.90:0/3023250546 wait complete. 2024-03-24T01:28:57.748 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T01:27:11.774313+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T01:27:12.992083+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T01:28:57.749 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:28:57.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.947+0000 7fde2b200640 1 Processor -- start 2024-03-24T01:28:57.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.948+0000 7fde2b200640 1 -- start start 2024-03-24T01:28:57.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.949+0000 7fde2b200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fde24145170 0x7fde240cf810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:28:57.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.949+0000 7fde2b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fde240cf190 0x7fde240cfd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:28:57.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.949+0000 7fde2b200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde2406d990 con 0x7fde2406ea30 2024-03-24T01:28:57.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.949+0000 7fde2b200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde2406dc70 con 0x7fde24145170 2024-03-24T01:28:57.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.949+0000 7fde2b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde2406df50 con 0x7fde240cf190 2024-03-24T01:28:57.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.949+0000 7fde29e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fde24145170 0x7fde240cf810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:28:57.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.950+0000 7fde29e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fde24145170 0x7fde240cf810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54528/0 (socket says 172.21.15.90:54528) 2024-03-24T01:28:57.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.950+0000 7fde29e00640 1 -- 172.21.15.90:0/3582499817 learned_addr learned my addr 172.21.15.90:0/3582499817 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:28:57.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.950+0000 7fde2a800640 1 --2- 172.21.15.90:0/3582499817 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fde240cf190 0x7fde240cfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:28:57.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.950+0000 7fde29e00640 1 -- 172.21.15.90:0/3582499817 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fde240cf190 msgr2=0x7fde240cfd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:28:57.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.950+0000 7fde29e00640 1 --2- 172.21.15.90:0/3582499817 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fde240cf190 0x7fde240cfd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:57.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.950+0000 7fde29e00640 1 -- 172.21.15.90:0/3582499817 >> v1:172.21.15.120:6789/0 conn(0x7fde2406ea30 legacy=0x7fde240cea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:28:57.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.950+0000 7fde29e00640 1 -- 172.21.15.90:0/3582499817 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde2406e230 con 0x7fde24145170 2024-03-24T01:28:57.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.951+0000 7fde29e00640 1 --2- 172.21.15.90:0/3582499817 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde24145170 0x7fde240cf810 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fde18009870 tx=0x7fde1802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=350d03d5feba5793 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:28:57.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.951+0000 7fde28a00640 1 -- 172.21.15.90:0/3582499817 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde18035020 con 0x7fde24145170 2024-03-24T01:28:57.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.951+0000 7fde28a00640 1 -- 172.21.15.90:0/3582499817 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde18031640 con 0x7fde24145170 2024-03-24T01:28:57.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.951+0000 7fde28a00640 1 -- 172.21.15.90:0/3582499817 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1802d040 con 0x7fde24145170 2024-03-24T01:28:57.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.952+0000 7fde2b200640 1 -- 172.21.15.90:0/3582499817 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde24145170 msgr2=0x7fde240cf810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:28:57.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.952+0000 7fde2b200640 1 --2- 172.21.15.90:0/3582499817 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde24145170 0x7fde240cf810 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fde18009870 tx=0x7fde1802b8f0 comp rx=0 tx=0).stop 2024-03-24T01:28:57.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.952+0000 7fde2b200640 1 -- 172.21.15.90:0/3582499817 shutdown_connections 2024-03-24T01:28:57.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.952+0000 7fde2b200640 1 --2- 172.21.15.90:0/3582499817 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fde240cf190 0x7fde240cfd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:57.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.952+0000 7fde2b200640 1 --2- 172.21.15.90:0/3582499817 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde24145170 0x7fde240cf810 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:57.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.952+0000 7fde2b200640 1 -- 172.21.15.90:0/3582499817 >> 172.21.15.90:0/3582499817 conn(0x7fde2406b0c0 msgr2=0x7fde2406b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:28:57.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.952+0000 7fde2b200640 1 -- 172.21.15.90:0/3582499817 shutdown_connections 2024-03-24T01:28:57.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.952+0000 7fde2b200640 1 -- 172.21.15.90:0/3582499817 wait complete. 2024-03-24T01:28:57.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.954+0000 7fde2b200640 1 Processor -- start 2024-03-24T01:28:57.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.954+0000 7fde2b200640 1 -- start start 2024-03-24T01:28:57.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.955+0000 7fde2b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde2406ea30 0x7fde241422b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:28:57.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.955+0000 7fde2b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fde240cf190 0x7fde241e42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:28:57.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.955+0000 7fde2b200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde24145170 0x7fde24140060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:28:57.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.955+0000 7fde2b200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde241456d0 con 0x7fde2406ea30 2024-03-24T01:28:57.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.955+0000 7fde2b200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde24145550 con 0x7fde24145170 2024-03-24T01:28:57.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.955+0000 7fde2b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde2415c0a0 con 0x7fde240cf190 2024-03-24T01:28:57.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.955+0000 7fde29e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde2406ea30 0x7fde241422b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:28:57.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.955+0000 7fde29e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde2406ea30 0x7fde241422b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54536/0 (socket says 172.21.15.90:54536) 2024-03-24T01:28:57.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.955+0000 7fde29e00640 1 -- 172.21.15.90:0/3947285251 learned_addr learned my addr 172.21.15.90:0/3947285251 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:28:57.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.956+0000 7fde29e00640 1 -- 172.21.15.90:0/3947285251 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fde240cf190 msgr2=0x7fde241e42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:28:57.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.956+0000 7fde29e00640 1 --2- 172.21.15.90:0/3947285251 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fde240cf190 0x7fde241e42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:57.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.956+0000 7fde29e00640 1 -- 172.21.15.90:0/3947285251 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde24145170 msgr2=0x7fde24140060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:28:57.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.956+0000 7fde29e00640 1 --2- 172.21.15.90:0/3947285251 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde24145170 0x7fde24140060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:57.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.956+0000 7fde29e00640 1 -- 172.21.15.90:0/3947285251 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde241427f0 con 0x7fde2406ea30 2024-03-24T01:28:57.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.956+0000 7fde29e00640 1 --2- 172.21.15.90:0/3947285251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde2406ea30 0x7fde241422b0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fde1802b930 tx=0x7fde180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:28:57.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.956+0000 7fde1e000640 1 -- 172.21.15.90:0/3947285251 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde18040070 con 0x7fde2406ea30 2024-03-24T01:28:57.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.956+0000 7fde1e000640 1 -- 172.21.15.90:0/3947285251 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde18039c10 con 0x7fde2406ea30 2024-03-24T01:28:57.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.956+0000 7fde2b200640 1 -- 172.21.15.90:0/3947285251 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde241f4360 con 0x7fde2406ea30 2024-03-24T01:28:57.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.956+0000 7fde1e000640 1 -- 172.21.15.90:0/3947285251 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde18035040 con 0x7fde2406ea30 2024-03-24T01:28:57.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.957+0000 7fde2b200640 1 -- 172.21.15.90:0/3947285251 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde241f4920 con 0x7fde2406ea30 2024-03-24T01:28:57.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.958+0000 7fde2b200640 1 -- 172.21.15.90:0/3947285251 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde2406f950 con 0x7fde2406ea30 2024-03-24T01:28:57.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.959+0000 7fde1e000640 1 -- 172.21.15.90:0/3947285251 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fde1804b060 con 0x7fde2406ea30 2024-03-24T01:28:57.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.959+0000 7fde1e000640 1 --2- 172.21.15.90:0/3947285251 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fde0806bbd0 0x7fde0806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:28:57.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.960+0000 7fde1e000640 1 -- 172.21.15.90:0/3947285251 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fde180ac7b0 con 0x7fde2406ea30 2024-03-24T01:28:57.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.960+0000 7fde29400640 1 --2- 172.21.15.90:0/3947285251 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fde0806bbd0 0x7fde0806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:28:57.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.961+0000 7fde29400640 1 --2- 172.21.15.90:0/3947285251 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fde0806bbd0 0x7fde0806e090 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fde241e6be0 tx=0x7fde140079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:28:57.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:57.964+0000 7fde1e000640 1 -- 172.21.15.90:0/3947285251 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fde2406f950 con 0x7fde2406ea30 2024-03-24T01:28:58.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.149+0000 7fde2b200640 1 -- 172.21.15.90:0/3947285251 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fde2404f400 con 0x7fde2406ea30 2024-03-24T01:28:58.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.149+0000 7fde1e000640 1 -- 172.21.15.90:0/3947285251 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v55) v1 ==== 93+0+10 (secure 0 0 0) 0x7fde1802f300 con 0x7fde2406ea30 2024-03-24T01:28:58.150 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:28:58.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.152+0000 7fde2b200640 1 -- 172.21.15.90:0/3947285251 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fde0806bbd0 msgr2=0x7fde0806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:28:58.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.152+0000 7fde2b200640 1 --2- 172.21.15.90:0/3947285251 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fde0806bbd0 0x7fde0806e090 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fde241e6be0 tx=0x7fde140079b0 comp rx=0 tx=0).stop 2024-03-24T01:28:58.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.152+0000 7fde2b200640 1 -- 172.21.15.90:0/3947285251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde2406ea30 msgr2=0x7fde241422b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:28:58.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.152+0000 7fde2b200640 1 --2- 172.21.15.90:0/3947285251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde2406ea30 0x7fde241422b0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fde1802b930 tx=0x7fde180046c0 comp rx=0 tx=0).stop 2024-03-24T01:28:58.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.152+0000 7fde2b200640 1 -- 172.21.15.90:0/3947285251 shutdown_connections 2024-03-24T01:28:58.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.152+0000 7fde2b200640 1 --2- 172.21.15.90:0/3947285251 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fde0806bbd0 0x7fde0806e090 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:58.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.152+0000 7fde2b200640 1 --2- 172.21.15.90:0/3947285251 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde24145170 0x7fde24140060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:58.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.152+0000 7fde2b200640 1 --2- 172.21.15.90:0/3947285251 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fde240cf190 0x7fde241e42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:58.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.152+0000 7fde2b200640 1 --2- 172.21.15.90:0/3947285251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde2406ea30 0x7fde241422b0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:58.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.152+0000 7fde2b200640 1 -- 172.21.15.90:0/3947285251 >> 172.21.15.90:0/3947285251 conn(0x7fde2406b0c0 msgr2=0x7fde240cf5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:28:58.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.152+0000 7fde2b200640 1 -- 172.21.15.90:0/3947285251 shutdown_connections 2024-03-24T01:28:58.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.152+0000 7fde2b200640 1 -- 172.21.15.90:0/3947285251 wait complete. 2024-03-24T01:28:58.170 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T01:28:58.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.374+0000 7fc3cea00640 1 Processor -- start 2024-03-24T01:28:58.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.374+0000 7fc3cea00640 1 -- start start 2024-03-24T01:28:58.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.376+0000 7fc3cea00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc3c80aeeb0 0x7fc3c80b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:28:58.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.376+0000 7fc3cea00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3c806b7d0 0x7fc3c80b2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:28:58.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.376+0000 7fc3cea00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c806dac0 con 0x7fc3c815bc50 2024-03-24T01:28:58.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.376+0000 7fc3cea00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3c80b4da0 con 0x7fc3c80aeeb0 2024-03-24T01:28:58.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.376+0000 7fc3cea00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c806dda0 con 0x7fc3c806b7d0 2024-03-24T01:28:58.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.376+0000 7fc3bf400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc3c815bc50 0x7fc3c806b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:32816/0 (socket says 172.21.15.90:32816) 2024-03-24T01:28:58.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.376+0000 7fc3bf400640 1 -- 172.21.15.90:0/311163818 learned_addr learned my addr 172.21.15.90:0/311163818 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:28:58.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.376+0000 7fc3c7e00640 1 --2- 172.21.15.90:0/311163818 >> v2:172.21.15.120:3300/0 conn(0x7fc3c80aeeb0 0x7fc3c80b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:28:58.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.376+0000 7fc3cca00640 1 --2- 172.21.15.90:0/311163818 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3c806b7d0 0x7fc3c80b2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:28:58.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.377+0000 7fc3c7400640 1 -- 172.21.15.90:0/311163818 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 584581899 0 0) 0x7fc3c806dac0 con 0x7fc3c815bc50 2024-03-24T01:28:58.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.377+0000 7fc3c7400640 1 -- 172.21.15.90:0/311163818 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3ac003520 con 0x7fc3c815bc50 2024-03-24T01:28:58.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.377+0000 7fc3cca00640 1 -- 172.21.15.90:0/311163818 >> v2:172.21.15.120:3300/0 conn(0x7fc3c80aeeb0 msgr2=0x7fc3c80b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:28:58.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.377+0000 7fc3cca00640 1 --2- 172.21.15.90:0/311163818 >> v2:172.21.15.120:3300/0 conn(0x7fc3c80aeeb0 0x7fc3c80b44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:58.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.377+0000 7fc3cca00640 1 -- 172.21.15.90:0/311163818 >> v1:172.21.15.90:6789/0 conn(0x7fc3c815bc50 legacy=0x7fc3c806b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:28:58.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.377+0000 7fc3cca00640 1 -- 172.21.15.90:0/311163818 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c806e080 con 0x7fc3c806b7d0 2024-03-24T01:28:58.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.377+0000 7fc3c7e00640 1 --2- 172.21.15.90:0/311163818 >> v2:172.21.15.120:3300/0 conn(0x7fc3c80aeeb0 0x7fc3c80b44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T01:28:58.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.378+0000 7fc3cca00640 1 --2- 172.21.15.90:0/311163818 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3c806b7d0 0x7fc3c80b2890 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc3c0009d90 tx=0x7fc3c002bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe640a363a28ae55 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:28:58.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.379+0000 7fc3c7400640 1 -- 172.21.15.90:0/311163818 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c0039070 con 0x7fc3c806b7d0 2024-03-24T01:28:58.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.379+0000 7fc3c7400640 1 -- 172.21.15.90:0/311163818 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3c002cb30 con 0x7fc3c806b7d0 2024-03-24T01:28:58.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.379+0000 7fc3c7400640 1 -- 172.21.15.90:0/311163818 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c0034440 con 0x7fc3c806b7d0 2024-03-24T01:28:58.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.379+0000 7fc3cea00640 1 -- 172.21.15.90:0/311163818 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3c806b7d0 msgr2=0x7fc3c80b2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:28:58.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.379+0000 7fc3cea00640 1 --2- 172.21.15.90:0/311163818 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3c806b7d0 0x7fc3c80b2890 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc3c0009d90 tx=0x7fc3c002bd80 comp rx=0 tx=0).stop 2024-03-24T01:28:58.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.379+0000 7fc3cea00640 1 -- 172.21.15.90:0/311163818 shutdown_connections 2024-03-24T01:28:58.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.379+0000 7fc3cea00640 1 --2- 172.21.15.90:0/311163818 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3c806b7d0 0x7fc3c80b2890 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:58.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.379+0000 7fc3cea00640 1 --2- 172.21.15.90:0/311163818 >> v2:172.21.15.120:3300/0 conn(0x7fc3c80aeeb0 0x7fc3c80b44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:58.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.379+0000 7fc3cea00640 1 -- 172.21.15.90:0/311163818 >> 172.21.15.90:0/311163818 conn(0x7fc3c80b1770 msgr2=0x7fc3c80b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:28:58.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.379+0000 7fc3cea00640 1 -- 172.21.15.90:0/311163818 shutdown_connections 2024-03-24T01:28:58.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.379+0000 7fc3cea00640 1 -- 172.21.15.90:0/311163818 wait complete. 2024-03-24T01:28:58.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.381+0000 7fc3cea00640 1 Processor -- start 2024-03-24T01:28:58.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.381+0000 7fc3cea00640 1 -- start start 2024-03-24T01:28:58.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.382+0000 7fc3cea00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3c806b7d0 0x7fc3c8155a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:28:58.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.382+0000 7fc3cea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3c80aeeb0 0x7fc3c813a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:28:58.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.382+0000 7fc3cea00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3c815bc50 0x7fc3c8153a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:28:58.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.382+0000 7fc3cea00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c80b4a00 con 0x7fc3c80aeeb0 2024-03-24T01:28:58.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.382+0000 7fc3cea00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c80af290 con 0x7fc3c815bc50 2024-03-24T01:28:58.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.382+0000 7fc3cea00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c80b4b80 con 0x7fc3c806b7d0 2024-03-24T01:28:58.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.382+0000 7fc3bf400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3c80aeeb0 0x7fc3c813a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:28:58.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.382+0000 7fc3bf400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3c80aeeb0 0x7fc3c813a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54542/0 (socket says 172.21.15.90:54542) 2024-03-24T01:28:58.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.382+0000 7fc3bf400640 1 -- 172.21.15.90:0/1478565504 learned_addr learned my addr 172.21.15.90:0/1478565504 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:28:58.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.382+0000 7fc3c7e00640 1 --2- 172.21.15.90:0/1478565504 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3c806b7d0 0x7fc3c8155a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:28:58.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.383+0000 7fc3cca00640 1 --2- 172.21.15.90:0/1478565504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3c815bc50 0x7fc3c8153a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:28:58.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.383+0000 7fc3bf400640 1 -- 172.21.15.90:0/1478565504 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3c806b7d0 msgr2=0x7fc3c8155a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:28:58.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.383+0000 7fc3bf400640 1 --2- 172.21.15.90:0/1478565504 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3c806b7d0 0x7fc3c8155a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:58.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.383+0000 7fc3bf400640 1 -- 172.21.15.90:0/1478565504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3c815bc50 msgr2=0x7fc3c8153a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:28:58.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.383+0000 7fc3bf400640 1 --2- 172.21.15.90:0/1478565504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3c815bc50 0x7fc3c8153a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:58.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.383+0000 7fc3bf400640 1 -- 172.21.15.90:0/1478565504 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c81543b0 con 0x7fc3c80aeeb0 2024-03-24T01:28:58.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.383+0000 7fc3bf400640 1 --2- 172.21.15.90:0/1478565504 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3c80aeeb0 0x7fc3c813a080 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fc3b000c390 tx=0x7fc3b000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:28:58.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.383+0000 7fc3c4c00640 1 -- 172.21.15.90:0/1478565504 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b0017070 con 0x7fc3c80aeeb0 2024-03-24T01:28:58.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.383+0000 7fc3c4c00640 1 -- 172.21.15.90:0/1478565504 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3b0005ce0 con 0x7fc3c80aeeb0 2024-03-24T01:28:58.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.383+0000 7fc3c4c00640 1 -- 172.21.15.90:0/1478565504 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b00072f0 con 0x7fc3c80aeeb0 2024-03-24T01:28:58.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.383+0000 7fc3cea00640 1 -- 172.21.15.90:0/1478565504 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3c814ee30 con 0x7fc3c80aeeb0 2024-03-24T01:28:58.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.384+0000 7fc3cea00640 1 -- 172.21.15.90:0/1478565504 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3c814f3f0 con 0x7fc3c80aeeb0 2024-03-24T01:28:58.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.385+0000 7fc3cea00640 1 -- 172.21.15.90:0/1478565504 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3c814ec40 con 0x7fc3c80aeeb0 2024-03-24T01:28:58.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.385+0000 7fc3c4c00640 1 -- 172.21.15.90:0/1478565504 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc3b001a070 con 0x7fc3c80aeeb0 2024-03-24T01:28:58.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.386+0000 7fc3c4c00640 1 --2- 172.21.15.90:0/1478565504 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc3a406bbd0 0x7fc3a406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:28:58.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.386+0000 7fc3c4c00640 1 -- 172.21.15.90:0/1478565504 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fc3b0089fe0 con 0x7fc3c80aeeb0 2024-03-24T01:28:58.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.387+0000 7fc3c7e00640 1 --2- 172.21.15.90:0/1478565504 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc3a406bbd0 0x7fc3a406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:28:58.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.388+0000 7fc3c7e00640 1 --2- 172.21.15.90:0/1478565504 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc3a406bbd0 0x7fc3a406e090 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8152b10 tx=0x7fc3b8009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:28:58.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.391+0000 7fc3c4c00640 1 -- 172.21.15.90:0/1478565504 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc3b00582b0 con 0x7fc3c80aeeb0 2024-03-24T01:28:58.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.592+0000 7fc3cea00640 1 -- 172.21.15.90:0/1478565504 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc3c804f400 con 0x7fc3c80aeeb0 2024-03-24T01:28:58.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.593+0000 7fc3c4c00640 1 -- 172.21.15.90:0/1478565504 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v55) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc3b005c350 con 0x7fc3c80aeeb0 2024-03-24T01:28:58.594 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:28:58.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.596+0000 7fc3cea00640 1 -- 172.21.15.90:0/1478565504 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc3a406bbd0 msgr2=0x7fc3a406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:28:58.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.596+0000 7fc3cea00640 1 --2- 172.21.15.90:0/1478565504 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc3a406bbd0 0x7fc3a406e090 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8152b10 tx=0x7fc3b8009190 comp rx=0 tx=0).stop 2024-03-24T01:28:58.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.596+0000 7fc3cea00640 1 -- 172.21.15.90:0/1478565504 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3c80aeeb0 msgr2=0x7fc3c813a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:28:58.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.596+0000 7fc3cea00640 1 --2- 172.21.15.90:0/1478565504 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3c80aeeb0 0x7fc3c813a080 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fc3b000c390 tx=0x7fc3b000c850 comp rx=0 tx=0).stop 2024-03-24T01:28:58.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.596+0000 7fc3cea00640 1 -- 172.21.15.90:0/1478565504 shutdown_connections 2024-03-24T01:28:58.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.596+0000 7fc3cea00640 1 --2- 172.21.15.90:0/1478565504 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc3a406bbd0 0x7fc3a406e090 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:58.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.596+0000 7fc3cea00640 1 --2- 172.21.15.90:0/1478565504 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3c80aeeb0 0x7fc3c813a080 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:58.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.596+0000 7fc3cea00640 1 --2- 172.21.15.90:0/1478565504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3c815bc50 0x7fc3c8153a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:58.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.596+0000 7fc3cea00640 1 --2- 172.21.15.90:0/1478565504 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3c806b7d0 0x7fc3c8155a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:58.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.596+0000 7fc3cea00640 1 -- 172.21.15.90:0/1478565504 >> 172.21.15.90:0/1478565504 conn(0x7fc3c80b1770 msgr2=0x7fc3c80cd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:28:58.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.596+0000 7fc3cea00640 1 -- 172.21.15.90:0/1478565504 shutdown_connections 2024-03-24T01:28:58.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.596+0000 7fc3cea00640 1 -- 172.21.15.90:0/1478565504 wait complete. 2024-03-24T01:28:58.613 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T01:28:58.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.826+0000 7f387f000640 1 Processor -- start 2024-03-24T01:28:58.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.826+0000 7f387f000640 1 -- start start 2024-03-24T01:28:58.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.828+0000 7f387f000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3878159620 0x7f387806b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:28:58.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.828+0000 7f387f000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f38780cf150 0x7f38780ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:28:58.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.828+0000 7f387f000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f387806bca0 con 0x7f387806b600 2024-03-24T01:28:58.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.828+0000 7f387f000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38780b4470 con 0x7f3878159620 2024-03-24T01:28:58.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.828+0000 7f387f000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38780b36f0 con 0x7f38780cf150 2024-03-24T01:28:58.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.828+0000 7f3877e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3878159620 0x7f387806b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:28:58.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.828+0000 7f386f400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f387806b600 0x7f38780cea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:32822/0 (socket says 172.21.15.90:32822) 2024-03-24T01:28:58.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.828+0000 7f3877e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3878159620 0x7f387806b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54544/0 (socket says 172.21.15.90:54544) 2024-03-24T01:28:58.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.828+0000 7f386f400640 1 -- 172.21.15.90:0/1511988233 learned_addr learned my addr 172.21.15.90:0/1511988233 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:28:58.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.828+0000 7f387d000640 1 --2- 172.21.15.90:0/1511988233 >> v2:172.21.15.120:3300/0 conn(0x7f38780cf150 0x7f38780ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:28:58.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.829+0000 7f3877e00640 1 -- 172.21.15.90:0/1511988233 >> v2:172.21.15.120:3300/0 conn(0x7f38780cf150 msgr2=0x7f38780ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:28:58.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.829+0000 7f3877e00640 1 --2- 172.21.15.90:0/1511988233 >> v2:172.21.15.120:3300/0 conn(0x7f38780cf150 0x7f38780ae530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:58.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.829+0000 7f3877e00640 1 -- 172.21.15.90:0/1511988233 >> v1:172.21.15.90:6789/0 conn(0x7f387806b600 legacy=0x7f38780cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:28:58.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.829+0000 7f3877e00640 1 -- 172.21.15.90:0/1511988233 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38780b3990 con 0x7f3878159620 2024-03-24T01:28:58.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.829+0000 7f3877400640 1 -- 172.21.15.90:0/1511988233 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 355836050 0 0) 0x7f387806bca0 con 0x7f387806b600 2024-03-24T01:28:58.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.830+0000 7f3877e00640 1 --2- 172.21.15.90:0/1511988233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3878159620 0x7f387806b0c0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f38780cc4e0 tx=0x7f386802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b47591aa53892115 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:28:58.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.830+0000 7f3877400640 1 -- 172.21.15.90:0/1511988233 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38680315a0 con 0x7f3878159620 2024-03-24T01:28:58.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.830+0000 7f3877400640 1 -- 172.21.15.90:0/1511988233 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3868037070 con 0x7f3878159620 2024-03-24T01:28:58.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.830+0000 7f3877400640 1 -- 172.21.15.90:0/1511988233 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3868031960 con 0x7f3878159620 2024-03-24T01:28:58.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.830+0000 7f387f000640 1 -- 172.21.15.90:0/1511988233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3878159620 msgr2=0x7f387806b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:28:58.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.830+0000 7f387f000640 1 --2- 172.21.15.90:0/1511988233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3878159620 0x7f387806b0c0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f38780cc4e0 tx=0x7f386802b840 comp rx=0 tx=0).stop 2024-03-24T01:28:58.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.831+0000 7f387f000640 1 -- 172.21.15.90:0/1511988233 shutdown_connections 2024-03-24T01:28:58.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.831+0000 7f387f000640 1 --2- 172.21.15.90:0/1511988233 >> v2:172.21.15.120:3300/0 conn(0x7f38780cf150 0x7f38780ae530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:58.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.831+0000 7f387f000640 1 --2- 172.21.15.90:0/1511988233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3878159620 0x7f387806b0c0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:58.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.831+0000 7f387f000640 1 -- 172.21.15.90:0/1511988233 >> 172.21.15.90:0/1511988233 conn(0x7f38780b1770 msgr2=0x7f38780b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:28:58.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.831+0000 7f387f000640 1 -- 172.21.15.90:0/1511988233 shutdown_connections 2024-03-24T01:28:58.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.831+0000 7f387f000640 1 -- 172.21.15.90:0/1511988233 wait complete. 2024-03-24T01:28:58.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.832+0000 7f387f000640 1 Processor -- start 2024-03-24T01:28:58.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.832+0000 7f387f000640 1 -- start start 2024-03-24T01:28:58.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.833+0000 7f387f000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f387806b600 0x7f387813c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:28:58.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.833+0000 7f387f000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38780cf150 0x7f3878154870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:28:58.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.833+0000 7f387f000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3878159620 0x7f3878154db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:28:58.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.833+0000 7f387f000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3878159a00 con 0x7f3878159620 2024-03-24T01:28:58.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.833+0000 7f387f000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38780b3f50 con 0x7f38780cf150 2024-03-24T01:28:58.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.833+0000 7f387f000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f387815bfe0 con 0x7f387806b600 2024-03-24T01:28:58.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.834+0000 7f387d000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3878159620 0x7f3878154db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:28:58.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.834+0000 7f387d000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3878159620 0x7f3878154db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54554/0 (socket says 172.21.15.90:54554) 2024-03-24T01:28:58.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.834+0000 7f387d000640 1 -- 172.21.15.90:0/1469411409 learned_addr learned my addr 172.21.15.90:0/1469411409 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:28:58.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.834+0000 7f387d000640 1 -- 172.21.15.90:0/1469411409 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f387806b600 msgr2=0x7f387813c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:28:58.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.834+0000 7f3877e00640 1 --2- 172.21.15.90:0/1469411409 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f387806b600 0x7f387813c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:28:58.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.834+0000 7f386f400640 1 --2- 172.21.15.90:0/1469411409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38780cf150 0x7f3878154870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:28:58.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.834+0000 7f387d000640 1 --2- 172.21.15.90:0/1469411409 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f387806b600 0x7f387813c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:58.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.834+0000 7f387d000640 1 -- 172.21.15.90:0/1469411409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38780cf150 msgr2=0x7f3878154870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:28:58.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.834+0000 7f387d000640 1 --2- 172.21.15.90:0/1469411409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38780cf150 0x7f3878154870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:58.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.834+0000 7f387d000640 1 -- 172.21.15.90:0/1469411409 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3878152c80 con 0x7f3878159620 2024-03-24T01:28:58.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.834+0000 7f387d000640 1 --2- 172.21.15.90:0/1469411409 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3878159620 0x7f3878154db0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f3870007e40 tx=0x7f387000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:28:58.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.835+0000 7f3874c00640 1 -- 172.21.15.90:0/1469411409 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3870014620 con 0x7f3878159620 2024-03-24T01:28:58.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.835+0000 7f3874c00640 1 -- 172.21.15.90:0/1469411409 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3870019070 con 0x7f3878159620 2024-03-24T01:28:58.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.835+0000 7f3874c00640 1 -- 172.21.15.90:0/1469411409 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3870014a20 con 0x7f3878159620 2024-03-24T01:28:58.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.835+0000 7f387f000640 1 -- 172.21.15.90:0/1469411409 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3878150cf0 con 0x7f3878159620 2024-03-24T01:28:58.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.835+0000 7f387f000640 1 -- 172.21.15.90:0/1469411409 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38781531d0 con 0x7f3878159620 2024-03-24T01:28:58.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.836+0000 7f387f000640 1 -- 172.21.15.90:0/1469411409 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f387804f400 con 0x7f3878159620 2024-03-24T01:28:58.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.837+0000 7f3874c00640 1 -- 172.21.15.90:0/1469411409 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3870014bc0 con 0x7f3878159620 2024-03-24T01:28:58.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.838+0000 7f3874c00640 1 --2- 172.21.15.90:0/1469411409 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f385406bbd0 0x7f385406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:28:58.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.838+0000 7f3874c00640 1 -- 172.21.15.90:0/1469411409 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f387008b610 con 0x7f3878159620 2024-03-24T01:28:58.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.838+0000 7f3877e00640 1 --2- 172.21.15.90:0/1469411409 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f385406bbd0 0x7f385406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:28:58.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.839+0000 7f3877e00640 1 --2- 172.21.15.90:0/1469411409 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f385406bbd0 0x7f385406e090 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3868036000 tx=0x7f3868008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:28:58.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:58.842+0000 7f3874c00640 1 -- 172.21.15.90:0/1469411409 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3870059ea0 con 0x7f3878159620 2024-03-24T01:28:59.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.042+0000 7f387f000640 1 -- 172.21.15.90:0/1469411409 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f38781533a0 con 0x7f3878159620 2024-03-24T01:28:59.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.043+0000 7f3874c00640 1 -- 172.21.15.90:0/1469411409 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v55) v1 ==== 100+0+11 (secure 0 0 0) 0x7f38700597a0 con 0x7f3878159620 2024-03-24T01:28:59.044 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-24T01:28:59.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.046+0000 7f387f000640 1 -- 172.21.15.90:0/1469411409 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f385406bbd0 msgr2=0x7f385406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:28:59.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.046+0000 7f387f000640 1 --2- 172.21.15.90:0/1469411409 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f385406bbd0 0x7f385406e090 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3868036000 tx=0x7f3868008030 comp rx=0 tx=0).stop 2024-03-24T01:28:59.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.046+0000 7f387f000640 1 -- 172.21.15.90:0/1469411409 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3878159620 msgr2=0x7f3878154db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:28:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.046+0000 7f387f000640 1 --2- 172.21.15.90:0/1469411409 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3878159620 0x7f3878154db0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f3870007e40 tx=0x7f387000e4b0 comp rx=0 tx=0).stop 2024-03-24T01:28:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.046+0000 7f387f000640 1 -- 172.21.15.90:0/1469411409 shutdown_connections 2024-03-24T01:28:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.046+0000 7f387f000640 1 --2- 172.21.15.90:0/1469411409 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f385406bbd0 0x7f385406e090 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.046+0000 7f387f000640 1 --2- 172.21.15.90:0/1469411409 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3878159620 0x7f3878154db0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.046+0000 7f387f000640 1 --2- 172.21.15.90:0/1469411409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38780cf150 0x7f3878154870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.046+0000 7f387f000640 1 --2- 172.21.15.90:0/1469411409 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f387806b600 0x7f387813c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.046+0000 7f387f000640 1 -- 172.21.15.90:0/1469411409 >> 172.21.15.90:0/1469411409 conn(0x7f38780b1770 msgr2=0x7f38780cf770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:28:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.046+0000 7f387f000640 1 -- 172.21.15.90:0/1469411409 shutdown_connections 2024-03-24T01:28:59.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.046+0000 7f387f000640 1 -- 172.21.15.90:0/1469411409 wait complete. 2024-03-24T01:28:59.066 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T01:28:59.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.276+0000 7fd73fe00640 1 Processor -- start 2024-03-24T01:28:59.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.276+0000 7fd73fe00640 1 -- start start 2024-03-24T01:28:59.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.278+0000 7fd73fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd73815e0e0 0x7fd73806cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:28:59.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.278+0000 7fd73fe00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd73815e4c0 con 0x7fd7380b2080 2024-03-24T01:28:59.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.278+0000 7fd73fe00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd73806d330 con 0x7fd7380d0270 2024-03-24T01:28:59.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.278+0000 7fd73fe00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7380b1350 con 0x7fd73815e0e0 2024-03-24T01:28:59.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.278+0000 7fd73d400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd73815e0e0 0x7fd73806cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:28:59.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.278+0000 7fd73d400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd73815e0e0 0x7fd73806cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54562/0 (socket says 172.21.15.90:54562) 2024-03-24T01:28:59.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.278+0000 7fd73d400640 1 -- 172.21.15.90:0/722381259 learned_addr learned my addr 172.21.15.90:0/722381259 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:28:59.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.278+0000 7fd73ca00640 1 --1- 172.21.15.90:0/722381259 >> v1:172.21.15.90:6789/0 conn(0x7fd7380b2080 0x7fd7380cfb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:32830/0 (socket says 172.21.15.90:32830) 2024-03-24T01:28:59.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.279+0000 7fd73d400640 1 -- 172.21.15.90:0/722381259 >> v1:172.21.15.120:6789/0 conn(0x7fd7380d0270 legacy=0x7fd738099c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:28:59.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.279+0000 7fd73d400640 1 -- 172.21.15.90:0/722381259 >> v1:172.21.15.90:6789/0 conn(0x7fd7380b2080 legacy=0x7fd7380cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:28:59.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.279+0000 7fd73d400640 1 -- 172.21.15.90:0/722381259 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7380b15a0 con 0x7fd73815e0e0 2024-03-24T01:28:59.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.280+0000 7fd73d400640 1 --2- 172.21.15.90:0/722381259 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd73815e0e0 0x7fd73806cbe0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fd72c009870 tx=0x7fd72c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13087becd7de6eaf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:28:59.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.280+0000 7fd737e00640 1 -- 172.21.15.90:0/722381259 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd72c034070 con 0x7fd73815e0e0 2024-03-24T01:28:59.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.280+0000 7fd737e00640 1 -- 172.21.15.90:0/722381259 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd72c030860 con 0x7fd73815e0e0 2024-03-24T01:28:59.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.280+0000 7fd737e00640 1 -- 172.21.15.90:0/722381259 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd72c030c30 con 0x7fd73815e0e0 2024-03-24T01:28:59.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.281+0000 7fd73fe00640 1 -- 172.21.15.90:0/722381259 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd73815e0e0 msgr2=0x7fd73806cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:28:59.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.281+0000 7fd73fe00640 1 --2- 172.21.15.90:0/722381259 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd73815e0e0 0x7fd73806cbe0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fd72c009870 tx=0x7fd72c02b9a0 comp rx=0 tx=0).stop 2024-03-24T01:28:59.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.281+0000 7fd73fe00640 1 -- 172.21.15.90:0/722381259 shutdown_connections 2024-03-24T01:28:59.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.281+0000 7fd73fe00640 1 --2- 172.21.15.90:0/722381259 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd73815e0e0 0x7fd73806cbe0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:59.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.281+0000 7fd73fe00640 1 -- 172.21.15.90:0/722381259 >> 172.21.15.90:0/722381259 conn(0x7fd7380b34e0 msgr2=0x7fd7380b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:28:59.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.281+0000 7fd73fe00640 1 -- 172.21.15.90:0/722381259 shutdown_connections 2024-03-24T01:28:59.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.281+0000 7fd73fe00640 1 -- 172.21.15.90:0/722381259 wait complete. 2024-03-24T01:28:59.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.283+0000 7fd73fe00640 1 Processor -- start 2024-03-24T01:28:59.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.283+0000 7fd73fe00640 1 -- start start 2024-03-24T01:28:59.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.284+0000 7fd73fe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7380b2080 0x7fd7380cb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:28:59.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.284+0000 7fd73fe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7380d0270 0x7fd738154c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:28:59.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.284+0000 7fd73fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd73815e0e0 0x7fd73815a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:28:59.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.284+0000 7fd73fe00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7380b24e0 con 0x7fd73815e0e0 2024-03-24T01:28:59.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.284+0000 7fd73fe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7380b1b60 con 0x7fd7380d0270 2024-03-24T01:28:59.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.284+0000 7fd73fe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7380b2660 con 0x7fd7380b2080 2024-03-24T01:28:59.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.284+0000 7fd73de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd73815e0e0 0x7fd73815a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:28:59.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.284+0000 7fd73de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd73815e0e0 0x7fd73815a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54566/0 (socket says 172.21.15.90:54566) 2024-03-24T01:28:59.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.284+0000 7fd73de00640 1 -- 172.21.15.90:0/903482490 learned_addr learned my addr 172.21.15.90:0/903482490 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:28:59.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.284+0000 7fd73de00640 1 -- 172.21.15.90:0/903482490 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7380b2080 msgr2=0x7fd7380cb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:28:59.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.284+0000 7fd73de00640 1 --2- 172.21.15.90:0/903482490 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7380b2080 0x7fd7380cb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:59.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.284+0000 7fd73de00640 1 -- 172.21.15.90:0/903482490 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7380d0270 msgr2=0x7fd738154c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:28:59.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.284+0000 7fd73ca00640 1 --2- 172.21.15.90:0/903482490 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7380d0270 0x7fd738154c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:28:59.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.284+0000 7fd73de00640 1 --2- 172.21.15.90:0/903482490 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7380d0270 0x7fd738154c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:59.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.285+0000 7fd73de00640 1 -- 172.21.15.90:0/903482490 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd73815ca60 con 0x7fd73815e0e0 2024-03-24T01:28:59.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.285+0000 7fd73de00640 1 --2- 172.21.15.90:0/903482490 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd73815e0e0 0x7fd73815a9f0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fd73000c3e0 tx=0x7fd73000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:28:59.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.285+0000 7fd735600640 1 -- 172.21.15.90:0/903482490 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd730011790 con 0x7fd73815e0e0 2024-03-24T01:28:59.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.285+0000 7fd735600640 1 -- 172.21.15.90:0/903482490 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd730017070 con 0x7fd73815e0e0 2024-03-24T01:28:59.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.285+0000 7fd735600640 1 -- 172.21.15.90:0/903482490 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd730011b70 con 0x7fd73815e0e0 2024-03-24T01:28:59.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.285+0000 7fd73fe00640 1 -- 172.21.15.90:0/903482490 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7380ca8f0 con 0x7fd73815e0e0 2024-03-24T01:28:59.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.285+0000 7fd73fe00640 1 -- 172.21.15.90:0/903482490 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7380caeb0 con 0x7fd73815e0e0 2024-03-24T01:28:59.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.286+0000 7fd73fe00640 1 -- 172.21.15.90:0/903482490 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd73804f400 con 0x7fd73815e0e0 2024-03-24T01:28:59.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.287+0000 7fd735600640 1 -- 172.21.15.90:0/903482490 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd730011d10 con 0x7fd73815e0e0 2024-03-24T01:28:59.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.288+0000 7fd735600640 1 --2- 172.21.15.90:0/903482490 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd71806bbd0 0x7fd71806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:28:59.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.288+0000 7fd735600640 1 -- 172.21.15.90:0/903482490 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fd73008aa90 con 0x7fd73815e0e0 2024-03-24T01:28:59.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.289+0000 7fd73d400640 1 --2- 172.21.15.90:0/903482490 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd71806bbd0 0x7fd71806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:28:59.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.290+0000 7fd73d400640 1 --2- 172.21.15.90:0/903482490 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd71806bbd0 0x7fd71806e090 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fd72c036000 tx=0x7fd72c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:28:59.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.293+0000 7fd735600640 1 -- 172.21.15.90:0/903482490 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd730059320 con 0x7fd73815e0e0 2024-03-24T01:28:59.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.506+0000 7fd73fe00640 1 -- 172.21.15.90:0/903482490 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd7380b4310 con 0x7fd73815e0e0 2024-03-24T01:28:59.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.506+0000 7fd735600640 1 -- 172.21.15.90:0/903482490 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v55) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd730058c20 con 0x7fd73815e0e0 2024-03-24T01:28:59.507 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-24T01:28:59.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.509+0000 7fd73fe00640 1 -- 172.21.15.90:0/903482490 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd71806bbd0 msgr2=0x7fd71806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:28:59.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.509+0000 7fd73fe00640 1 --2- 172.21.15.90:0/903482490 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd71806bbd0 0x7fd71806e090 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fd72c036000 tx=0x7fd72c030000 comp rx=0 tx=0).stop 2024-03-24T01:28:59.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.509+0000 7fd73fe00640 1 -- 172.21.15.90:0/903482490 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd73815e0e0 msgr2=0x7fd73815a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:28:59.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.509+0000 7fd73fe00640 1 --2- 172.21.15.90:0/903482490 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd73815e0e0 0x7fd73815a9f0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fd73000c3e0 tx=0x7fd73000c8a0 comp rx=0 tx=0).stop 2024-03-24T01:28:59.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.509+0000 7fd73fe00640 1 -- 172.21.15.90:0/903482490 shutdown_connections 2024-03-24T01:28:59.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.509+0000 7fd73fe00640 1 --2- 172.21.15.90:0/903482490 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd71806bbd0 0x7fd71806e090 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:59.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.509+0000 7fd73fe00640 1 --2- 172.21.15.90:0/903482490 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd73815e0e0 0x7fd73815a9f0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:59.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.509+0000 7fd73fe00640 1 --2- 172.21.15.90:0/903482490 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7380d0270 0x7fd738154c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:59.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.509+0000 7fd73fe00640 1 --2- 172.21.15.90:0/903482490 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7380b2080 0x7fd7380cb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:59.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.509+0000 7fd73fe00640 1 -- 172.21.15.90:0/903482490 >> 172.21.15.90:0/903482490 conn(0x7fd7380b34e0 msgr2=0x7fd7380ce470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:28:59.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.509+0000 7fd73fe00640 1 -- 172.21.15.90:0/903482490 shutdown_connections 2024-03-24T01:28:59.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.509+0000 7fd73fe00640 1 -- 172.21.15.90:0/903482490 wait complete. 2024-03-24T01:28:59.529 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:28:59.529 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:28:59.580 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:28:59.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.834+0000 7fdd26800640 1 Processor -- start 2024-03-24T01:28:59.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.834+0000 7fdd26800640 1 -- start start 2024-03-24T01:28:59.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.835+0000 7fdd26800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdd20144800 0x7fdd2006d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:28:59.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.835+0000 7fdd26800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdd200cc8f0 0x7fdd200cfa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:28:59.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.835+0000 7fdd26800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd200cceb0 con 0x7fdd20144420 2024-03-24T01:28:59.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.835+0000 7fdd26800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd200cd190 con 0x7fdd200cc8f0 2024-03-24T01:28:59.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.835+0000 7fdd26800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd200ceb70 con 0x7fdd20144800 2024-03-24T01:28:59.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.836+0000 7fdd25e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdd200cc8f0 0x7fdd200cfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:28:59.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.836+0000 7fdd25e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdd200cc8f0 0x7fdd200cfa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40408/0 (socket says 172.21.15.90:40408) 2024-03-24T01:28:59.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.836+0000 7fdd25e00640 1 -- 172.21.15.90:0/3907101297 learned_addr learned my addr 172.21.15.90:0/3907101297 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:28:59.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.836+0000 7fdd25e00640 1 -- 172.21.15.90:0/3907101297 >> v2:172.21.15.120:3300/0 conn(0x7fdd20144800 msgr2=0x7fdd2006d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:28:59.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.836+0000 7fdd25e00640 1 --2- 172.21.15.90:0/3907101297 >> v2:172.21.15.120:3300/0 conn(0x7fdd20144800 0x7fdd2006d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:59.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.836+0000 7fdd25e00640 1 -- 172.21.15.90:0/3907101297 >> v1:172.21.15.120:6789/0 conn(0x7fdd20144420 legacy=0x7fdd200cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:28:59.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.836+0000 7fdd25e00640 1 -- 172.21.15.90:0/3907101297 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd200cee50 con 0x7fdd200cc8f0 2024-03-24T01:28:59.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.837+0000 7fdd25e00640 1 --2- 172.21.15.90:0/3907101297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd200cc8f0 0x7fdd200cfa80 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fdd2006bf50 tx=0x7fdd1802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19e8f15f30da0056 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:28:59.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.837+0000 7fdd1fe00640 1 -- 172.21.15.90:0/3907101297 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1802c9f0 con 0x7fdd200cc8f0 2024-03-24T01:28:59.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.837+0000 7fdd1fe00640 1 -- 172.21.15.90:0/3907101297 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd1802cb90 con 0x7fdd200cc8f0 2024-03-24T01:28:59.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.837+0000 7fdd1fe00640 1 -- 172.21.15.90:0/3907101297 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd180343f0 con 0x7fdd200cc8f0 2024-03-24T01:28:59.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.838+0000 7fdd26800640 1 -- 172.21.15.90:0/3907101297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd200cc8f0 msgr2=0x7fdd200cfa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:28:59.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.838+0000 7fdd26800640 1 --2- 172.21.15.90:0/3907101297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd200cc8f0 0x7fdd200cfa80 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fdd2006bf50 tx=0x7fdd1802be50 comp rx=0 tx=0).stop 2024-03-24T01:28:59.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.838+0000 7fdd26800640 1 -- 172.21.15.90:0/3907101297 shutdown_connections 2024-03-24T01:28:59.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.838+0000 7fdd26800640 1 --2- 172.21.15.90:0/3907101297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd200cc8f0 0x7fdd200cfa80 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:59.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.838+0000 7fdd26800640 1 --2- 172.21.15.90:0/3907101297 >> v2:172.21.15.120:3300/0 conn(0x7fdd20144800 0x7fdd2006d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:59.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.838+0000 7fdd26800640 1 -- 172.21.15.90:0/3907101297 >> 172.21.15.90:0/3907101297 conn(0x7fdd2006b0c0 msgr2=0x7fdd2006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:28:59.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.838+0000 7fdd26800640 1 -- 172.21.15.90:0/3907101297 shutdown_connections 2024-03-24T01:28:59.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.838+0000 7fdd26800640 1 -- 172.21.15.90:0/3907101297 wait complete. 2024-03-24T01:28:59.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.840+0000 7fdd26800640 1 Processor -- start 2024-03-24T01:28:59.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.840+0000 7fdd26800640 1 -- start start 2024-03-24T01:28:59.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.841+0000 7fdd26800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd200cc8f0 0x7fdd2013e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:28:59.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.841+0000 7fdd26800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd20144420 0x7fdd200ac5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:28:59.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.841+0000 7fdd26800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd20144800 0x7fdd2015dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:28:59.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.841+0000 7fdd26800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd200ce910 con 0x7fdd20144800 2024-03-24T01:28:59.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.841+0000 7fdd26800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd200ce790 con 0x7fdd20144420 2024-03-24T01:28:59.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.841+0000 7fdd26800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2015f540 con 0x7fdd200cc8f0 2024-03-24T01:28:59.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.841+0000 7fdd25e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd20144800 0x7fdd2015dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:28:59.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.841+0000 7fdd25e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd20144800 0x7fdd2015dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40418/0 (socket says 172.21.15.90:40418) 2024-03-24T01:28:59.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.841+0000 7fdd25e00640 1 -- 172.21.15.90:0/2842140135 learned_addr learned my addr 172.21.15.90:0/2842140135 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:28:59.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.841+0000 7fdd25400640 1 --2- 172.21.15.90:0/2842140135 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd200cc8f0 0x7fdd2013e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:28:59.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.841+0000 7fdd24a00640 1 --2- 172.21.15.90:0/2842140135 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd20144420 0x7fdd200ac5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:28:59.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.842+0000 7fdd25e00640 1 -- 172.21.15.90:0/2842140135 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd200cc8f0 msgr2=0x7fdd2013e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:28:59.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.842+0000 7fdd25e00640 1 --2- 172.21.15.90:0/2842140135 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd200cc8f0 0x7fdd2013e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:59.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.842+0000 7fdd25e00640 1 -- 172.21.15.90:0/2842140135 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd20144420 msgr2=0x7fdd200ac5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:28:59.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.842+0000 7fdd25e00640 1 --2- 172.21.15.90:0/2842140135 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd20144420 0x7fdd200ac5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:28:59.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.842+0000 7fdd25e00640 1 -- 172.21.15.90:0/2842140135 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd2015d0e0 con 0x7fdd20144800 2024-03-24T01:28:59.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.842+0000 7fdd25400640 1 --2- 172.21.15.90:0/2842140135 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd200cc8f0 0x7fdd2013e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:28:59.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.842+0000 7fdd25e00640 1 --2- 172.21.15.90:0/2842140135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd20144800 0x7fdd2015dea0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fdd18036040 tx=0x7fdd18004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:28:59.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.842+0000 7fdd1d600640 1 -- 172.21.15.90:0/2842140135 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd18043020 con 0x7fdd20144800 2024-03-24T01:28:59.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.842+0000 7fdd1d600640 1 -- 172.21.15.90:0/2842140135 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd1803e450 con 0x7fdd20144800 2024-03-24T01:28:59.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.842+0000 7fdd1d600640 1 -- 172.21.15.90:0/2842140135 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd18009510 con 0x7fdd20144800 2024-03-24T01:28:59.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.842+0000 7fdd26800640 1 -- 172.21.15.90:0/2842140135 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd2015d3c0 con 0x7fdd20144800 2024-03-24T01:28:59.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.842+0000 7fdd26800640 1 -- 172.21.15.90:0/2842140135 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd2015d980 con 0x7fdd20144800 2024-03-24T01:28:59.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.844+0000 7fdd26800640 1 -- 172.21.15.90:0/2842140135 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd2004f400 con 0x7fdd20144800 2024-03-24T01:28:59.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.845+0000 7fdd1d600640 1 -- 172.21.15.90:0/2842140135 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdd18046020 con 0x7fdd20144800 2024-03-24T01:28:59.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.845+0000 7fdd1d600640 1 --2- 172.21.15.90:0/2842140135 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fdd0006bbd0 0x7fdd0006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:28:59.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.846+0000 7fdd1d600640 1 -- 172.21.15.90:0/2842140135 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fdd180ad5f0 con 0x7fdd20144800 2024-03-24T01:28:59.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.846+0000 7fdd25400640 1 --2- 172.21.15.90:0/2842140135 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fdd0006bbd0 0x7fdd0006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:28:59.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.848+0000 7fdd25400640 1 --2- 172.21.15.90:0/2842140135 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fdd0006bbd0 0x7fdd0006e090 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fdd14001750 tx=0x7fdd14009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:28:59.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:28:59.850+0000 7fdd1d600640 1 -- 172.21.15.90:0/2842140135 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdd1804d030 con 0x7fdd20144800 2024-03-24T01:29:00.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.049+0000 7fdd26800640 1 -- 172.21.15.90:0/2842140135 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd2006ded0 con 0x7fdd20144800 2024-03-24T01:29:00.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.051+0000 7fdd1d600640 1 -- 172.21.15.90:0/2842140135 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+16958 (secure 0 0 0) 0x7fdd1807bf10 con 0x7fdd20144800 2024-03-24T01:29:00.052 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:29:00.052 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":55,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:27:54.129944+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:27:11.774313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-24T01:27:12.992083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.23","mappings":[{"from":0,"to":1}]},{"pgid":"8.37","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:29:00.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.054+0000 7fdd26800640 1 -- 172.21.15.90:0/2842140135 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fdd0006bbd0 msgr2=0x7fdd0006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:29:00.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.054+0000 7fdd26800640 1 --2- 172.21.15.90:0/2842140135 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fdd0006bbd0 0x7fdd0006e090 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fdd14001750 tx=0x7fdd14009040 comp rx=0 tx=0).stop 2024-03-24T01:29:00.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.054+0000 7fdd26800640 1 -- 172.21.15.90:0/2842140135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd20144800 msgr2=0x7fdd2015dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:29:00.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.054+0000 7fdd26800640 1 --2- 172.21.15.90:0/2842140135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd20144800 0x7fdd2015dea0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fdd18036040 tx=0x7fdd18004a40 comp rx=0 tx=0).stop 2024-03-24T01:29:00.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.054+0000 7fdd26800640 1 -- 172.21.15.90:0/2842140135 shutdown_connections 2024-03-24T01:29:00.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.054+0000 7fdd26800640 1 --2- 172.21.15.90:0/2842140135 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fdd0006bbd0 0x7fdd0006e090 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:00.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.054+0000 7fdd26800640 1 --2- 172.21.15.90:0/2842140135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd20144800 0x7fdd2015dea0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:00.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.054+0000 7fdd26800640 1 --2- 172.21.15.90:0/2842140135 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd20144420 0x7fdd200ac5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:00.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.054+0000 7fdd26800640 1 --2- 172.21.15.90:0/2842140135 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdd200cc8f0 0x7fdd2013e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:00.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.054+0000 7fdd26800640 1 -- 172.21.15.90:0/2842140135 >> 172.21.15.90:0/2842140135 conn(0x7fdd2006b0c0 msgr2=0x7fdd200739c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:29:00.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.054+0000 7fdd26800640 1 -- 172.21.15.90:0/2842140135 shutdown_connections 2024-03-24T01:29:00.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.054+0000 7fdd26800640 1 -- 172.21.15.90:0/2842140135 wait complete. 2024-03-24T01:29:00.074 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T01:27:11.774313+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T01:27:12.992083+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T01:29:00.075 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:29:00.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.279+0000 7ffae3e00640 1 Processor -- start 2024-03-24T01:29:00.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.279+0000 7ffae3e00640 1 -- start start 2024-03-24T01:29:00.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.281+0000 7ffae3e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffadc1452b0 0x7ffadc071990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:00.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.281+0000 7ffae3e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ffadc071ed0 0x7ffadc0ceaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:00.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.281+0000 7ffae3e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffadc0b2890 con 0x7ffadc0cf030 2024-03-24T01:29:00.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.281+0000 7ffae3e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffadc072330 con 0x7ffadc071ed0 2024-03-24T01:29:00.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.281+0000 7ffae3e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffadc072990 con 0x7ffadc1452b0 2024-03-24T01:29:00.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.281+0000 7ffae1e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ffadc0cf030 0x7ffadc15f5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:33446/0 (socket says 172.21.15.90:33446) 2024-03-24T01:29:00.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.281+0000 7ffae1e00640 1 -- 172.21.15.90:0/3836953160 learned_addr learned my addr 172.21.15.90:0/3836953160 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:29:00.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.282+0000 7ffae1400640 1 --2- 172.21.15.90:0/3836953160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffadc1452b0 0x7ffadc071990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:29:00.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.282+0000 7ffae0a00640 1 --2- 172.21.15.90:0/3836953160 >> v2:172.21.15.120:3300/0 conn(0x7ffadc071ed0 0x7ffadc0ceaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:29:00.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.282+0000 7ffadbe00640 1 -- 172.21.15.90:0/3836953160 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013275517 0 0) 0x7ffadc0b2890 con 0x7ffadc0cf030 2024-03-24T01:29:00.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.282+0000 7ffadbe00640 1 -- 172.21.15.90:0/3836953160 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffac4003520 con 0x7ffadc0cf030 2024-03-24T01:29:00.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.282+0000 7ffadbe00640 1 -- 172.21.15.90:0/3836953160 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2859855899 0 0) 0x7ffac4003520 con 0x7ffadc0cf030 2024-03-24T01:29:00.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.282+0000 7ffadbe00640 1 -- 172.21.15.90:0/3836953160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffadc1452b0 msgr2=0x7ffadc071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:29:00.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.282+0000 7ffadbe00640 1 --2- 172.21.15.90:0/3836953160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffadc1452b0 0x7ffadc071990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:00.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.282+0000 7ffadbe00640 1 -- 172.21.15.90:0/3836953160 >> v2:172.21.15.120:3300/0 conn(0x7ffadc071ed0 msgr2=0x7ffadc0ceaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:29:00.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.282+0000 7ffadbe00640 1 --2- 172.21.15.90:0/3836953160 >> v2:172.21.15.120:3300/0 conn(0x7ffadc071ed0 0x7ffadc0ceaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:00.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.282+0000 7ffadbe00640 1 -- 172.21.15.90:0/3836953160 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffadc0b2b70 con 0x7ffadc0cf030 2024-03-24T01:29:00.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.282+0000 7ffadbe00640 1 -- 172.21.15.90:0/3836953160 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7ffad40019f0 con 0x7ffadc0cf030 2024-03-24T01:29:00.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.282+0000 7ffadbe00640 1 -- 172.21.15.90:0/3836953160 >> v1:172.21.15.90:6789/0 conn(0x7ffadc0cf030 legacy=0x7ffadc15f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:29:00.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.282+0000 7ffadbe00640 1 --2- 172.21.15.90:0/3836953160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffadc1452b0 0x7ffac40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:00.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.282+0000 7ffadbe00640 1 --2- 172.21.15.90:0/3836953160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffac4003f30 0x7ffac4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:00.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.282+0000 7ffadbe00640 1 --2- 172.21.15.90:0/3836953160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffac4004d40 0x7ffac40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:00.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.282+0000 7ffadbe00640 1 -- 172.21.15.90:0/3836953160 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffadc072990 con 0x7ffac4004d40 2024-03-24T01:29:00.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.282+0000 7ffadbe00640 1 -- 172.21.15.90:0/3836953160 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac400b9c0 con 0x7ffac4003f30 2024-03-24T01:29:00.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.282+0000 7ffadbe00640 1 -- 172.21.15.90:0/3836953160 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffac400bb90 con 0x7ffadc1452b0 2024-03-24T01:29:00.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.283+0000 7ffae1e00640 1 --2- 172.21.15.90:0/3836953160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffac4004d40 0x7ffac40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:29:00.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.283+0000 7ffae1400640 1 --2- 172.21.15.90:0/3836953160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffadc1452b0 0x7ffac40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:29:00.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.283+0000 7ffae0a00640 1 --2- 172.21.15.90:0/3836953160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffac4003f30 0x7ffac4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:29:00.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.283+0000 7ffae1e00640 1 -- 172.21.15.90:0/3836953160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffadc1452b0 msgr2=0x7ffac40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:29:00.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.283+0000 7ffae1e00640 1 --2- 172.21.15.90:0/3836953160 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffadc1452b0 0x7ffac40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:00.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.283+0000 7ffae1e00640 1 -- 172.21.15.90:0/3836953160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffac4003f30 msgr2=0x7ffac4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:29:00.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.283+0000 7ffae1e00640 1 --2- 172.21.15.90:0/3836953160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffac4003f30 0x7ffac4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:00.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.283+0000 7ffae1e00640 1 -- 172.21.15.90:0/3836953160 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac4003520 con 0x7ffac4004d40 2024-03-24T01:29:00.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.283+0000 7ffae0a00640 1 --2- 172.21.15.90:0/3836953160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffac4003f30 0x7ffac4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:29:00.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.284+0000 7ffae1e00640 1 --2- 172.21.15.90:0/3836953160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffac4004d40 0x7ffac40051c0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7ffad40013d0 tx=0x7ffac4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d05cf656ccf1205e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:29:00.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.284+0000 7ffadbe00640 1 -- 172.21.15.90:0/3836953160 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad400a4e0 con 0x7ffac4004d40 2024-03-24T01:29:00.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.284+0000 7ffadbe00640 1 -- 172.21.15.90:0/3836953160 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffad4034650 con 0x7ffac4004d40 2024-03-24T01:29:00.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.284+0000 7ffadbe00640 1 -- 172.21.15.90:0/3836953160 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad4034a40 con 0x7ffac4004d40 2024-03-24T01:29:00.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.285+0000 7ffae3e00640 1 -- 172.21.15.90:0/3836953160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffac4004d40 msgr2=0x7ffac40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:29:00.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.285+0000 7ffae3e00640 1 --2- 172.21.15.90:0/3836953160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffac4004d40 0x7ffac40051c0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7ffad40013d0 tx=0x7ffac4005830 comp rx=0 tx=0).stop 2024-03-24T01:29:00.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.285+0000 7ffae1e00640 1 -- 172.21.15.90:0/3836953160 reap_dead start 2024-03-24T01:29:00.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.285+0000 7ffae3e00640 1 -- 172.21.15.90:0/3836953160 shutdown_connections 2024-03-24T01:29:00.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.285+0000 7ffae3e00640 1 -- 172.21.15.90:0/3836953160 >> 172.21.15.90:0/3836953160 conn(0x7ffadc0cf7e0 msgr2=0x7ffadc0cfbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:29:00.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.285+0000 7ffae3e00640 1 -- 172.21.15.90:0/3836953160 shutdown_connections 2024-03-24T01:29:00.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.285+0000 7ffae3e00640 1 -- 172.21.15.90:0/3836953160 wait complete. 2024-03-24T01:29:00.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.287+0000 7ffae3e00640 1 Processor -- start 2024-03-24T01:29:00.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.287+0000 7ffae3e00640 1 -- start start 2024-03-24T01:29:00.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.288+0000 7ffae3e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffac4004d40 0x7ffadc1f13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:00.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.288+0000 7ffae3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffadc0afb10 0x7ffadc0cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:00.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.288+0000 7ffae3e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffadc0b3830 0x7ffadc0b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:00.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.288+0000 7ffae3e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffadc0b3410 con 0x7ffadc0afb10 2024-03-24T01:29:00.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.288+0000 7ffae3e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffadc0b3130 con 0x7ffac4004d40 2024-03-24T01:29:00.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.288+0000 7ffae3e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffadc0b4940 con 0x7ffadc0b3830 2024-03-24T01:29:00.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.288+0000 7ffae0a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffadc0afb10 0x7ffadc0cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:29:00.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.288+0000 7ffae0a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffadc0afb10 0x7ffadc0cb0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40434/0 (socket says 172.21.15.90:40434) 2024-03-24T01:29:00.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.288+0000 7ffae0a00640 1 -- 172.21.15.90:0/304050090 learned_addr learned my addr 172.21.15.90:0/304050090 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:29:00.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.288+0000 7ffae1400640 1 --2- 172.21.15.90:0/304050090 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffac4004d40 0x7ffadc1f13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:29:00.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.288+0000 7ffae0a00640 1 -- 172.21.15.90:0/304050090 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffadc0b3830 msgr2=0x7ffadc0b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:29:00.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.288+0000 7ffae1e00640 1 --2- 172.21.15.90:0/304050090 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffadc0b3830 0x7ffadc0b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:29:00.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.288+0000 7ffae0a00640 1 --2- 172.21.15.90:0/304050090 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffadc0b3830 0x7ffadc0b3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:00.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.289+0000 7ffae0a00640 1 -- 172.21.15.90:0/304050090 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffac4004d40 msgr2=0x7ffadc1f13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:29:00.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.289+0000 7ffae0a00640 1 --2- 172.21.15.90:0/304050090 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffac4004d40 0x7ffadc1f13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:00.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.289+0000 7ffae0a00640 1 -- 172.21.15.90:0/304050090 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffadc0cb5f0 con 0x7ffadc0afb10 2024-03-24T01:29:00.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.289+0000 7ffae1400640 1 --2- 172.21.15.90:0/304050090 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffac4004d40 0x7ffadc1f13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:29:00.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.289+0000 7ffae0a00640 1 --2- 172.21.15.90:0/304050090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffadc0afb10 0x7ffadc0cb0b0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7ffac800e7e0 tx=0x7ffac800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:29:00.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.289+0000 7ffad9600640 1 -- 172.21.15.90:0/304050090 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac8017070 con 0x7ffadc0afb10 2024-03-24T01:29:00.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.289+0000 7ffad9600640 1 -- 172.21.15.90:0/304050090 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffac800ee90 con 0x7ffadc0afb10 2024-03-24T01:29:00.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.289+0000 7ffad9600640 1 -- 172.21.15.90:0/304050090 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac800f040 con 0x7ffadc0afb10 2024-03-24T01:29:00.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.289+0000 7ffae3e00640 1 -- 172.21.15.90:0/304050090 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffadc1f1b10 con 0x7ffadc0afb10 2024-03-24T01:29:00.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.289+0000 7ffae3e00640 1 -- 172.21.15.90:0/304050090 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffadc1f20d0 con 0x7ffadc0afb10 2024-03-24T01:29:00.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.290+0000 7ffae3e00640 1 -- 172.21.15.90:0/304050090 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffadc0b41f0 con 0x7ffadc0afb10 2024-03-24T01:29:00.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.291+0000 7ffad9600640 1 -- 172.21.15.90:0/304050090 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ffac801a070 con 0x7ffadc0afb10 2024-03-24T01:29:00.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.292+0000 7ffad9600640 1 --2- 172.21.15.90:0/304050090 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ffabc06bbd0 0x7ffabc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:00.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.292+0000 7ffad9600640 1 -- 172.21.15.90:0/304050090 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6159+0+0 (secure 0 0 0) 0x7ffac808a8a0 con 0x7ffadc0afb10 2024-03-24T01:29:00.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.293+0000 7ffae1400640 1 --2- 172.21.15.90:0/304050090 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ffabc06bbd0 0x7ffabc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:29:00.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.294+0000 7ffae1400640 1 --2- 172.21.15.90:0/304050090 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ffabc06bbd0 0x7ffabc06e090 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ffadc0b10b0 tx=0x7ffad0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:29:00.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.297+0000 7ffad9600640 1 -- 172.21.15.90:0/304050090 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ffac80591c0 con 0x7ffadc0afb10 2024-03-24T01:29:00.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.494+0000 7ffae3e00640 1 -- 172.21.15.90:0/304050090 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffadc04f400 con 0x7ffadc0afb10 2024-03-24T01:29:00.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.495+0000 7ffad9600640 1 -- 172.21.15.90:0/304050090 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v55) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffac8058ac0 con 0x7ffadc0afb10 2024-03-24T01:29:00.495 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:29:00.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.497+0000 7ffae3e00640 1 -- 172.21.15.90:0/304050090 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ffabc06bbd0 msgr2=0x7ffabc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:29:00.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.497+0000 7ffae3e00640 1 --2- 172.21.15.90:0/304050090 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ffabc06bbd0 0x7ffabc06e090 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ffadc0b10b0 tx=0x7ffad0008040 comp rx=0 tx=0).stop 2024-03-24T01:29:00.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.497+0000 7ffae3e00640 1 -- 172.21.15.90:0/304050090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffadc0afb10 msgr2=0x7ffadc0cb0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:29:00.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.497+0000 7ffae3e00640 1 --2- 172.21.15.90:0/304050090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffadc0afb10 0x7ffadc0cb0b0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7ffac800e7e0 tx=0x7ffac800eca0 comp rx=0 tx=0).stop 2024-03-24T01:29:00.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.497+0000 7ffae3e00640 1 -- 172.21.15.90:0/304050090 shutdown_connections 2024-03-24T01:29:00.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.497+0000 7ffae3e00640 1 --2- 172.21.15.90:0/304050090 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ffabc06bbd0 0x7ffabc06e090 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:00.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.497+0000 7ffae3e00640 1 --2- 172.21.15.90:0/304050090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffadc0afb10 0x7ffadc0cb0b0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:00.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.497+0000 7ffae3e00640 1 --2- 172.21.15.90:0/304050090 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ffadc0b3830 0x7ffadc0b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:00.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.497+0000 7ffae3e00640 1 --2- 172.21.15.90:0/304050090 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffac4004d40 0x7ffadc1f13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:00.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.497+0000 7ffae3e00640 1 -- 172.21.15.90:0/304050090 >> 172.21.15.90:0/304050090 conn(0x7ffadc0cf7e0 msgr2=0x7ffadc0cc510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:29:00.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.497+0000 7ffae3e00640 1 -- 172.21.15.90:0/304050090 shutdown_connections 2024-03-24T01:29:00.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.498+0000 7ffae3e00640 1 -- 172.21.15.90:0/304050090 wait complete. 2024-03-24T01:29:00.516 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T01:29:00.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.726+0000 7f118ee00640 1 Processor -- start 2024-03-24T01:29:00.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.726+0000 7f118ee00640 1 -- start start 2024-03-24T01:29:00.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.728+0000 7f118ee00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11880aee80 0x7f11880b33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:00.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.728+0000 7f118ee00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11880b3eb0 con 0x7f118806b0c0 2024-03-24T01:29:00.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.728+0000 7f118ee00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118806e680 con 0x7f11880b3930 2024-03-24T01:29:00.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.728+0000 7f118ee00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f118806e960 con 0x7f11880aee80 2024-03-24T01:29:00.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.728+0000 7f118ce00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f118806b0c0 0x7f118806f160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:33452/0 (socket says 172.21.15.90:33452) 2024-03-24T01:29:00.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.728+0000 7f118ce00640 1 -- 172.21.15.90:0/4049550778 learned_addr learned my addr 172.21.15.90:0/4049550778 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:29:00.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.729+0000 7f1187e00640 1 --2- 172.21.15.90:0/4049550778 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11880aee80 0x7f11880b33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:29:00.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.729+0000 7f1186a00640 1 -- 172.21.15.90:0/4049550778 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 221463719 0 0) 0x7f11880b3eb0 con 0x7f118806b0c0 2024-03-24T01:29:00.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.729+0000 7f1186a00640 1 -- 172.21.15.90:0/4049550778 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f116c003520 con 0x7f118806b0c0 2024-03-24T01:29:00.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.729+0000 7f1186a00640 1 -- 172.21.15.90:0/4049550778 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1839654080 0 0) 0x7f118806e680 con 0x7f11880b3930 2024-03-24T01:29:00.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.729+0000 7f1186a00640 1 -- 172.21.15.90:0/4049550778 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11880b3eb0 con 0x7f11880b3930 2024-03-24T01:29:00.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.729+0000 7f1186a00640 1 -- 172.21.15.90:0/4049550778 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1618097929 0 0) 0x7f116c003520 con 0x7f118806b0c0 2024-03-24T01:29:00.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.729+0000 7f1186a00640 1 -- 172.21.15.90:0/4049550778 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11880aee80 msgr2=0x7f11880b33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:29:00.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.729+0000 7f1186a00640 1 --2- 172.21.15.90:0/4049550778 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11880aee80 0x7f11880b33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:00.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.729+0000 7f1186a00640 1 -- 172.21.15.90:0/4049550778 >> v1:172.21.15.120:6789/0 conn(0x7f11880b3930 legacy=0x7f11880cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:29:00.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.729+0000 7f1186a00640 1 -- 172.21.15.90:0/4049550778 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f118806ec40 con 0x7f118806b0c0 2024-03-24T01:29:00.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.729+0000 7f1186a00640 1 -- 172.21.15.90:0/4049550778 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f11800019f0 con 0x7f118806b0c0 2024-03-24T01:29:00.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.729+0000 7f1186a00640 1 -- 172.21.15.90:0/4049550778 >> v1:172.21.15.90:6789/0 conn(0x7f118806b0c0 legacy=0x7f118806f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:29:00.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.729+0000 7f1186a00640 1 --2- 172.21.15.90:0/4049550778 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f116c003b70 0x7f116c003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:00.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.729+0000 7f1186a00640 1 --2- 172.21.15.90:0/4049550778 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f116c004800 0x7f116c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:00.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.729+0000 7f1186a00640 1 --2- 172.21.15.90:0/4049550778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f116c005210 0x7f116c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:00.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.729+0000 7f1186a00640 1 -- 172.21.15.90:0/4049550778 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f118806e960 con 0x7f116c004800 2024-03-24T01:29:00.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.729+0000 7f1186a00640 1 -- 172.21.15.90:0/4049550778 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116c00c110 con 0x7f116c005210 2024-03-24T01:29:00.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.729+0000 7f1186a00640 1 -- 172.21.15.90:0/4049550778 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f116c00c310 con 0x7f116c003b70 2024-03-24T01:29:00.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.729+0000 7f1187400640 1 --2- 172.21.15.90:0/4049550778 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f116c004800 0x7f116c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:29:00.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.730+0000 7f1187e00640 1 --2- 172.21.15.90:0/4049550778 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f116c003b70 0x7f116c003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:29:00.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.730+0000 7f1187400640 1 -- 172.21.15.90:0/4049550778 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f116c003b70 msgr2=0x7f116c003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:29:00.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.730+0000 7f1187400640 1 --2- 172.21.15.90:0/4049550778 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f116c003b70 0x7f116c003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:00.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.730+0000 7f1187400640 1 -- 172.21.15.90:0/4049550778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f116c005210 msgr2=0x7f116c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:29:00.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.730+0000 7f1187400640 1 --2- 172.21.15.90:0/4049550778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f116c005210 0x7f116c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:00.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.730+0000 7f1187400640 1 -- 172.21.15.90:0/4049550778 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f116c003520 con 0x7f116c004800 2024-03-24T01:29:00.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.730+0000 7f1187e00640 1 --2- 172.21.15.90:0/4049550778 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f116c003b70 0x7f116c003f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:29:00.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.731+0000 7f1187400640 1 --2- 172.21.15.90:0/4049550778 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f116c004800 0x7f116c004c80 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f11740018d0 tx=0x7f116c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15f0a27dfab5ca94 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:29:00.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.731+0000 7f1186a00640 1 -- 172.21.15.90:0/4049550778 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1174037070 con 0x7f116c004800 2024-03-24T01:29:00.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.731+0000 7f1186a00640 1 -- 172.21.15.90:0/4049550778 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11740052d0 con 0x7f116c004800 2024-03-24T01:29:00.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.731+0000 7f1186a00640 1 -- 172.21.15.90:0/4049550778 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11740014f0 con 0x7f116c004800 2024-03-24T01:29:00.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.732+0000 7f118ee00640 1 -- 172.21.15.90:0/4049550778 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f116c004800 msgr2=0x7f116c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:29:00.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.732+0000 7f118ee00640 1 --2- 172.21.15.90:0/4049550778 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f116c004800 0x7f116c004c80 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f11740018d0 tx=0x7f116c005830 comp rx=0 tx=0).stop 2024-03-24T01:29:00.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.732+0000 7f118ce00640 1 -- 172.21.15.90:0/4049550778 reap_dead start 2024-03-24T01:29:00.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.732+0000 7f118ee00640 1 -- 172.21.15.90:0/4049550778 shutdown_connections 2024-03-24T01:29:00.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.732+0000 7f118ee00640 1 -- 172.21.15.90:0/4049550778 >> 172.21.15.90:0/4049550778 conn(0x7f11880b1770 msgr2=0x7f11880b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:29:00.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.732+0000 7f118ee00640 1 -- 172.21.15.90:0/4049550778 shutdown_connections 2024-03-24T01:29:00.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.732+0000 7f118ee00640 1 -- 172.21.15.90:0/4049550778 wait complete. 2024-03-24T01:29:00.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.734+0000 7f118ee00640 1 Processor -- start 2024-03-24T01:29:00.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.734+0000 7f118ee00640 1 -- start start 2024-03-24T01:29:00.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.735+0000 7f118ee00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f116c004800 0x7f11881508e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:00.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.735+0000 7f118ee00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1188164f80 0x7f11881f82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:00.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.735+0000 7f118ee00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11881653e0 0x7f11881f8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:00.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.735+0000 7f118ee00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11880ccfb0 con 0x7f1188164f80 2024-03-24T01:29:00.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.735+0000 7f118ee00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1188166ad0 con 0x7f116c004800 2024-03-24T01:29:00.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.735+0000 7f118ee00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11880cd130 con 0x7f11881653e0 2024-03-24T01:29:00.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.735+0000 7f1187400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1188164f80 0x7f11881f82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:29:00.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.735+0000 7f1187400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1188164f80 0x7f11881f82d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40458/0 (socket says 172.21.15.90:40458) 2024-03-24T01:29:00.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.735+0000 7f1187400640 1 -- 172.21.15.90:0/46724190 learned_addr learned my addr 172.21.15.90:0/46724190 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:29:00.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.735+0000 7f118ce00640 1 --2- 172.21.15.90:0/46724190 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11881653e0 0x7f11881f8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:29:00.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.736+0000 7f1187e00640 1 --2- 172.21.15.90:0/46724190 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f116c004800 0x7f11881508e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:29:00.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.736+0000 7f1187400640 1 -- 172.21.15.90:0/46724190 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11881653e0 msgr2=0x7f11881f8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:29:00.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.736+0000 7f1187400640 1 --2- 172.21.15.90:0/46724190 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11881653e0 0x7f11881f8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:00.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.736+0000 7f1187400640 1 -- 172.21.15.90:0/46724190 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f116c004800 msgr2=0x7f11881508e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:29:00.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.736+0000 7f1187400640 1 --2- 172.21.15.90:0/46724190 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f116c004800 0x7f11881508e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:00.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.736+0000 7f1187400640 1 -- 172.21.15.90:0/46724190 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1188164720 con 0x7f1188164f80 2024-03-24T01:29:00.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.736+0000 7f118ce00640 1 --2- 172.21.15.90:0/46724190 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11881653e0 0x7f11881f8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:29:00.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.736+0000 7f1187e00640 1 --2- 172.21.15.90:0/46724190 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f116c004800 0x7f11881508e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:29:00.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.736+0000 7f1187400640 1 --2- 172.21.15.90:0/46724190 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1188164f80 0x7f11881f82d0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f1174035040 tx=0x7f11740012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:29:00.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.736+0000 7f117fe00640 1 -- 172.21.15.90:0/46724190 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1174042020 con 0x7f1188164f80 2024-03-24T01:29:00.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.736+0000 7f117fe00640 1 -- 172.21.15.90:0/46724190 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1174038040 con 0x7f1188164f80 2024-03-24T01:29:00.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.736+0000 7f117fe00640 1 -- 172.21.15.90:0/46724190 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1174037070 con 0x7f1188164f80 2024-03-24T01:29:00.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.736+0000 7f118ee00640 1 -- 172.21.15.90:0/46724190 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1188150e20 con 0x7f1188164f80 2024-03-24T01:29:00.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.736+0000 7f118ee00640 1 -- 172.21.15.90:0/46724190 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1188163960 con 0x7f1188164f80 2024-03-24T01:29:00.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.738+0000 7f118ee00640 1 -- 172.21.15.90:0/46724190 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f118804f400 con 0x7f1188164f80 2024-03-24T01:29:00.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.739+0000 7f117fe00640 1 -- 172.21.15.90:0/46724190 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1174045020 con 0x7f1188164f80 2024-03-24T01:29:00.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.739+0000 7f117fe00640 1 --2- 172.21.15.90:0/46724190 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f116806bbd0 0x7f116806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:00.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.740+0000 7f117fe00640 1 -- 172.21.15.90:0/46724190 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f11740ac560 con 0x7f1188164f80 2024-03-24T01:29:00.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.740+0000 7f1187e00640 1 --2- 172.21.15.90:0/46724190 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f116806bbd0 0x7f116806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:29:00.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.741+0000 7f1187e00640 1 --2- 172.21.15.90:0/46724190 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f116806bbd0 0x7f116806e090 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1178006fd0 tx=0x7f1178008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:29:00.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.744+0000 7f117fe00640 1 -- 172.21.15.90:0/46724190 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f117404c030 con 0x7f1188164f80 2024-03-24T01:29:00.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.943+0000 7f118ee00640 1 -- 172.21.15.90:0/46724190 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f118806e250 con 0x7f1188164f80 2024-03-24T01:29:00.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.944+0000 7f117fe00640 1 -- 172.21.15.90:0/46724190 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v55) v1 ==== 104+0+11 (secure 0 0 0) 0x7f117407ae00 con 0x7f1188164f80 2024-03-24T01:29:00.944 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:29:00.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.946+0000 7f118ee00640 1 -- 172.21.15.90:0/46724190 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f116806bbd0 msgr2=0x7f116806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:29:00.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.946+0000 7f118ee00640 1 --2- 172.21.15.90:0/46724190 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f116806bbd0 0x7f116806e090 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1178006fd0 tx=0x7f1178008040 comp rx=0 tx=0).stop 2024-03-24T01:29:00.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.946+0000 7f118ee00640 1 -- 172.21.15.90:0/46724190 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1188164f80 msgr2=0x7f11881f82d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:29:00.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.946+0000 7f118ee00640 1 --2- 172.21.15.90:0/46724190 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1188164f80 0x7f11881f82d0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f1174035040 tx=0x7f11740012d0 comp rx=0 tx=0).stop 2024-03-24T01:29:00.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.947+0000 7f118ee00640 1 -- 172.21.15.90:0/46724190 shutdown_connections 2024-03-24T01:29:00.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.947+0000 7f118ee00640 1 --2- 172.21.15.90:0/46724190 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f116806bbd0 0x7f116806e090 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:00.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.947+0000 7f118ee00640 1 --2- 172.21.15.90:0/46724190 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1188164f80 0x7f11881f82d0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:00.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.947+0000 7f118ee00640 1 --2- 172.21.15.90:0/46724190 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f11881653e0 0x7f11881f8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:00.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.947+0000 7f118ee00640 1 --2- 172.21.15.90:0/46724190 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f116c004800 0x7f11881508e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:00.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.947+0000 7f118ee00640 1 -- 172.21.15.90:0/46724190 >> 172.21.15.90:0/46724190 conn(0x7f11880b1770 msgr2=0x7f118806f060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:29:00.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.947+0000 7f118ee00640 1 -- 172.21.15.90:0/46724190 shutdown_connections 2024-03-24T01:29:00.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:00.947+0000 7f118ee00640 1 -- 172.21.15.90:0/46724190 wait complete. 2024-03-24T01:29:00.965 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T01:29:01.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.177+0000 7f35d7200640 1 Processor -- start 2024-03-24T01:29:01.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.177+0000 7f35d7200640 1 -- start start 2024-03-24T01:29:01.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.179+0000 7f35d7200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f35d015df50 0x7f35d00d2e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:01.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.179+0000 7f35d7200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35d00d1d90 con 0x7f35d006cdd0 2024-03-24T01:29:01.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.179+0000 7f35d7200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35d00d2070 con 0x7f35d00b9440 2024-03-24T01:29:01.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.179+0000 7f35d7200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35d00d2350 con 0x7f35d015df50 2024-03-24T01:29:01.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.179+0000 7f35d5e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f35d015df50 0x7f35d00d2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:29:01.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.179+0000 7f35d6800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f35d006cdd0 0x7f35d00b6c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:33464/0 (socket says 172.21.15.90:33464) 2024-03-24T01:29:01.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.179+0000 7f35d6800640 1 -- 172.21.15.90:0/3555143345 learned_addr learned my addr 172.21.15.90:0/3555143345 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:29:01.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.180+0000 7f35d4a00640 1 -- 172.21.15.90:0/3555143345 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 459316023 0 0) 0x7f35d00d1d90 con 0x7f35d006cdd0 2024-03-24T01:29:01.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.180+0000 7f35d4a00640 1 -- 172.21.15.90:0/3555143345 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35b8003520 con 0x7f35d006cdd0 2024-03-24T01:29:01.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.180+0000 7f35d5e00640 1 -- 172.21.15.90:0/3555143345 >> v1:172.21.15.120:6789/0 conn(0x7f35d00b9440 legacy=0x7f35d006c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:29:01.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.180+0000 7f35d5e00640 1 -- 172.21.15.90:0/3555143345 >> v1:172.21.15.90:6789/0 conn(0x7f35d006cdd0 legacy=0x7f35d00b6c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:29:01.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.180+0000 7f35d5e00640 1 -- 172.21.15.90:0/3555143345 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d00d2630 con 0x7f35d015df50 2024-03-24T01:29:01.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.182+0000 7f35d5e00640 1 --2- 172.21.15.90:0/3555143345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35d015df50 0x7f35d00d2e30 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f35c40097a0 tx=0x7f35c402b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33ce228f7546246b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:29:01.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.182+0000 7f35d4a00640 1 -- 172.21.15.90:0/3555143345 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c402f810 con 0x7f35d015df50 2024-03-24T01:29:01.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.182+0000 7f35d4a00640 1 -- 172.21.15.90:0/3555143345 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c402f9b0 con 0x7f35d015df50 2024-03-24T01:29:01.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.182+0000 7f35d4a00640 1 -- 172.21.15.90:0/3555143345 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c402fd60 con 0x7f35d015df50 2024-03-24T01:29:01.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.183+0000 7f35d7200640 1 -- 172.21.15.90:0/3555143345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35d015df50 msgr2=0x7f35d00d2e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:29:01.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.183+0000 7f35d7200640 1 --2- 172.21.15.90:0/3555143345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35d015df50 0x7f35d00d2e30 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f35c40097a0 tx=0x7f35c402b9f0 comp rx=0 tx=0).stop 2024-03-24T01:29:01.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.183+0000 7f35d7200640 1 -- 172.21.15.90:0/3555143345 shutdown_connections 2024-03-24T01:29:01.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.183+0000 7f35d7200640 1 --2- 172.21.15.90:0/3555143345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35d015df50 0x7f35d00d2e30 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:01.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.183+0000 7f35d7200640 1 -- 172.21.15.90:0/3555143345 >> 172.21.15.90:0/3555143345 conn(0x7f35d0072c10 msgr2=0x7f35d0073020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:29:01.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.183+0000 7f35d7200640 1 -- 172.21.15.90:0/3555143345 shutdown_connections 2024-03-24T01:29:01.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.183+0000 7f35d7200640 1 -- 172.21.15.90:0/3555143345 wait complete. 2024-03-24T01:29:01.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.184+0000 7f35d7200640 1 Processor -- start 2024-03-24T01:29:01.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.185+0000 7f35d7200640 1 -- start start 2024-03-24T01:29:01.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.186+0000 7f35d7200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35d006cdd0 0x7f35d015b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:01.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.186+0000 7f35d7200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35d00b9440 0x7f35d00b5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:01.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.186+0000 7f35d7200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35d00b5b90 0x7f35d00b3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:01.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.186+0000 7f35d7200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d00b9a00 con 0x7f35d00b9440 2024-03-24T01:29:01.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.186+0000 7f35d7200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d00b9880 con 0x7f35d006cdd0 2024-03-24T01:29:01.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.186+0000 7f35d7200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35d00b7360 con 0x7f35d00b5b90 2024-03-24T01:29:01.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.186+0000 7f35d5400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35d00b9440 0x7f35d00b5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:29:01.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.186+0000 7f35d5400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35d00b9440 0x7f35d00b5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40478/0 (socket says 172.21.15.90:40478) 2024-03-24T01:29:01.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.186+0000 7f35d5400640 1 -- 172.21.15.90:0/2441780236 learned_addr learned my addr 172.21.15.90:0/2441780236 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:29:01.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.186+0000 7f35d5e00640 1 --2- 172.21.15.90:0/2441780236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35d006cdd0 0x7f35d015b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:29:01.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.186+0000 7f35d6800640 1 --2- 172.21.15.90:0/2441780236 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35d00b5b90 0x7f35d00b3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:29:01.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.186+0000 7f35d5400640 1 -- 172.21.15.90:0/2441780236 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35d00b5b90 msgr2=0x7f35d00b3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:29:01.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.186+0000 7f35d5400640 1 --2- 172.21.15.90:0/2441780236 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35d00b5b90 0x7f35d00b3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:01.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.186+0000 7f35d5400640 1 -- 172.21.15.90:0/2441780236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35d006cdd0 msgr2=0x7f35d015b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:29:01.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.186+0000 7f35d5400640 1 --2- 172.21.15.90:0/2441780236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35d006cdd0 0x7f35d015b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:01.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.187+0000 7f35d5400640 1 -- 172.21.15.90:0/2441780236 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d00b4230 con 0x7f35d00b9440 2024-03-24T01:29:01.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.187+0000 7f35d6800640 1 --2- 172.21.15.90:0/2441780236 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35d00b5b90 0x7f35d00b3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:29:01.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.187+0000 7f35d5e00640 1 --2- 172.21.15.90:0/2441780236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35d006cdd0 0x7f35d015b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:29:01.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.187+0000 7f35d5400640 1 --2- 172.21.15.90:0/2441780236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35d00b9440 0x7f35d00b5650 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f35c000c400 tx=0x7f35c000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:29:01.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.187+0000 7f35ca000640 1 -- 172.21.15.90:0/2441780236 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c00052d0 con 0x7f35d00b9440 2024-03-24T01:29:01.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.187+0000 7f35ca000640 1 -- 172.21.15.90:0/2441780236 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c0005490 con 0x7f35d00b9440 2024-03-24T01:29:01.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.187+0000 7f35ca000640 1 -- 172.21.15.90:0/2441780236 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c0007260 con 0x7f35d00b9440 2024-03-24T01:29:01.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.187+0000 7f35d7200640 1 -- 172.21.15.90:0/2441780236 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35d00b4510 con 0x7f35d00b9440 2024-03-24T01:29:01.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.187+0000 7f35d7200640 1 -- 172.21.15.90:0/2441780236 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35d00b4ad0 con 0x7f35d00b9440 2024-03-24T01:29:01.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.188+0000 7f35d7200640 1 -- 172.21.15.90:0/2441780236 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35d00b5fa0 con 0x7f35d00b9440 2024-03-24T01:29:01.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.189+0000 7f35ca000640 1 -- 172.21.15.90:0/2441780236 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f35c0007400 con 0x7f35d00b9440 2024-03-24T01:29:01.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.190+0000 7f35ca000640 1 --2- 172.21.15.90:0/2441780236 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f35b406bbd0 0x7f35b406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:01.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.190+0000 7f35ca000640 1 -- 172.21.15.90:0/2441780236 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f35c008a1b0 con 0x7f35d00b9440 2024-03-24T01:29:01.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.191+0000 7f35d5e00640 1 --2- 172.21.15.90:0/2441780236 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f35b406bbd0 0x7f35b406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:29:01.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.192+0000 7f35d5e00640 1 --2- 172.21.15.90:0/2441780236 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f35b406bbd0 0x7f35b406e090 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f35c4036000 tx=0x7f35c4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:29:01.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.195+0000 7f35ca000640 1 -- 172.21.15.90:0/2441780236 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f35d00b5fa0 con 0x7f35d00b9440 2024-03-24T01:29:01.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.392+0000 7f35d7200640 1 -- 172.21.15.90:0/2441780236 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f35d0073190 con 0x7f35d00b9440 2024-03-24T01:29:01.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.393+0000 7f35ca000640 1 -- 172.21.15.90:0/2441780236 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v55) v1 ==== 100+0+11 (secure 0 0 0) 0x7f35c0058ad0 con 0x7f35d00b9440 2024-03-24T01:29:01.393 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-24T01:29:01.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.395+0000 7f35d7200640 1 -- 172.21.15.90:0/2441780236 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f35b406bbd0 msgr2=0x7f35b406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:29:01.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.395+0000 7f35d7200640 1 --2- 172.21.15.90:0/2441780236 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f35b406bbd0 0x7f35b406e090 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f35c4036000 tx=0x7f35c4008030 comp rx=0 tx=0).stop 2024-03-24T01:29:01.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.395+0000 7f35d7200640 1 -- 172.21.15.90:0/2441780236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35d00b9440 msgr2=0x7f35d00b5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:29:01.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.395+0000 7f35d7200640 1 --2- 172.21.15.90:0/2441780236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35d00b9440 0x7f35d00b5650 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f35c000c400 tx=0x7f35c000c8c0 comp rx=0 tx=0).stop 2024-03-24T01:29:01.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.396+0000 7f35d7200640 1 -- 172.21.15.90:0/2441780236 shutdown_connections 2024-03-24T01:29:01.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.396+0000 7f35d7200640 1 --2- 172.21.15.90:0/2441780236 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f35b406bbd0 0x7f35b406e090 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:01.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.396+0000 7f35d7200640 1 --2- 172.21.15.90:0/2441780236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35d00b9440 0x7f35d00b5650 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:01.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.396+0000 7f35d7200640 1 --2- 172.21.15.90:0/2441780236 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35d00b5b90 0x7f35d00b3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:01.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.396+0000 7f35d7200640 1 --2- 172.21.15.90:0/2441780236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35d006cdd0 0x7f35d015b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:01.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.396+0000 7f35d7200640 1 -- 172.21.15.90:0/2441780236 >> 172.21.15.90:0/2441780236 conn(0x7f35d0072c10 msgr2=0x7f35d00cb670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:29:01.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.396+0000 7f35d7200640 1 -- 172.21.15.90:0/2441780236 shutdown_connections 2024-03-24T01:29:01.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.396+0000 7f35d7200640 1 -- 172.21.15.90:0/2441780236 wait complete. 2024-03-24T01:29:01.415 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T01:29:01.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.627+0000 7fe318c00640 1 Processor -- start 2024-03-24T01:29:01.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.627+0000 7fe318c00640 1 -- start start 2024-03-24T01:29:01.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.629+0000 7fe318c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe314145190 0x7fe314071790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:01.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.629+0000 7fe318c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe314071cd0 0x7fe3140cc900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:01.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.629+0000 7fe318c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe31406e0f0 con 0x7fe314144680 2024-03-24T01:29:01.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.629+0000 7fe318c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe314072140 con 0x7fe314071cd0 2024-03-24T01:29:01.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.629+0000 7fe318c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe314144a60 con 0x7fe314145190 2024-03-24T01:29:01.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.629+0000 7fe313e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe314071cd0 0x7fe3140cc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:29:01.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.629+0000 7fe313e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe314071cd0 0x7fe3140cc900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40486/0 (socket says 172.21.15.90:40486) 2024-03-24T01:29:01.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.629+0000 7fe313e00640 1 -- 172.21.15.90:0/3283755177 learned_addr learned my addr 172.21.15.90:0/3283755177 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:29:01.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.629+0000 7fe313e00640 1 -- 172.21.15.90:0/3283755177 >> v2:172.21.15.120:3300/0 conn(0x7fe314145190 msgr2=0x7fe314071790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:29:01.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.629+0000 7fe313e00640 1 --2- 172.21.15.90:0/3283755177 >> v2:172.21.15.120:3300/0 conn(0x7fe314145190 0x7fe314071790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:01.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.629+0000 7fe313e00640 1 -- 172.21.15.90:0/3283755177 >> v1:172.21.15.120:6789/0 conn(0x7fe314144680 legacy=0x7fe31406d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:29:01.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.629+0000 7fe313e00640 1 -- 172.21.15.90:0/3283755177 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe31406e3d0 con 0x7fe314071cd0 2024-03-24T01:29:01.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.631+0000 7fe313e00640 1 --2- 172.21.15.90:0/3283755177 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe314071cd0 0x7fe3140cc900 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fe3140cf1b0 tx=0x7fe30c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2df55b1b568c1ae4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:29:01.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.631+0000 7fe312a00640 1 -- 172.21.15.90:0/3283755177 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30c02ca80 con 0x7fe314071cd0 2024-03-24T01:29:01.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.631+0000 7fe312a00640 1 -- 172.21.15.90:0/3283755177 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe30c02cc20 con 0x7fe314071cd0 2024-03-24T01:29:01.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.631+0000 7fe312a00640 1 -- 172.21.15.90:0/3283755177 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30c0343f0 con 0x7fe314071cd0 2024-03-24T01:29:01.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.632+0000 7fe318c00640 1 -- 172.21.15.90:0/3283755177 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe314071cd0 msgr2=0x7fe3140cc900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:29:01.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.632+0000 7fe318c00640 1 --2- 172.21.15.90:0/3283755177 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe314071cd0 0x7fe3140cc900 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fe3140cf1b0 tx=0x7fe30c02bee0 comp rx=0 tx=0).stop 2024-03-24T01:29:01.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.632+0000 7fe318c00640 1 -- 172.21.15.90:0/3283755177 shutdown_connections 2024-03-24T01:29:01.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.632+0000 7fe318c00640 1 --2- 172.21.15.90:0/3283755177 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe314071cd0 0x7fe3140cc900 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:01.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.632+0000 7fe318c00640 1 --2- 172.21.15.90:0/3283755177 >> v2:172.21.15.120:3300/0 conn(0x7fe314145190 0x7fe314071790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:01.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.632+0000 7fe318c00640 1 -- 172.21.15.90:0/3283755177 >> 172.21.15.90:0/3283755177 conn(0x7fe31406b0c0 msgr2=0x7fe31406b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:29:01.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.632+0000 7fe318c00640 1 -- 172.21.15.90:0/3283755177 shutdown_connections 2024-03-24T01:29:01.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.632+0000 7fe318c00640 1 -- 172.21.15.90:0/3283755177 wait complete. 2024-03-24T01:29:01.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.634+0000 7fe318c00640 1 Processor -- start 2024-03-24T01:29:01.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.634+0000 7fe318c00640 1 -- start start 2024-03-24T01:29:01.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.635+0000 7fe318c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe314071cd0 0x7fe314141000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:01.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.635+0000 7fe318c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe314144680 0x7fe314142de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:01.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.635+0000 7fe318c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe314145190 0x7fe3141f1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:01.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.635+0000 7fe318c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3140cc470 con 0x7fe314144680 2024-03-24T01:29:01.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.635+0000 7fe318c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3140cc190 con 0x7fe314071cd0 2024-03-24T01:29:01.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.635+0000 7fe318c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3140cce40 con 0x7fe314145190 2024-03-24T01:29:01.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.635+0000 7fe30aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe314144680 0x7fe314142de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:29:01.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.635+0000 7fe30aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe314144680 0x7fe314142de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40498/0 (socket says 172.21.15.90:40498) 2024-03-24T01:29:01.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.635+0000 7fe30aa00640 1 -- 172.21.15.90:0/509933112 learned_addr learned my addr 172.21.15.90:0/509933112 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:29:01.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.635+0000 7fe313400640 1 --2- 172.21.15.90:0/509933112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe314071cd0 0x7fe314141000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:29:01.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.636+0000 7fe30aa00640 1 -- 172.21.15.90:0/509933112 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe314145190 msgr2=0x7fe3141f1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:29:01.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.636+0000 7fe30aa00640 1 --2- 172.21.15.90:0/509933112 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe314145190 0x7fe3141f1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:01.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.636+0000 7fe30aa00640 1 -- 172.21.15.90:0/509933112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe314071cd0 msgr2=0x7fe314141000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:29:01.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.636+0000 7fe30aa00640 1 --2- 172.21.15.90:0/509933112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe314071cd0 0x7fe314141000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:01.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.636+0000 7fe30aa00640 1 -- 172.21.15.90:0/509933112 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3141f2b20 con 0x7fe314144680 2024-03-24T01:29:01.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.636+0000 7fe30aa00640 1 --2- 172.21.15.90:0/509933112 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe314144680 0x7fe314142de0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc00b410 tx=0x7fe2fc00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:29:01.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.636+0000 7fe30be00640 1 -- 172.21.15.90:0/509933112 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc015020 con 0x7fe314144680 2024-03-24T01:29:01.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.636+0000 7fe30be00640 1 -- 172.21.15.90:0/509933112 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2fc011850 con 0x7fe314144680 2024-03-24T01:29:01.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.636+0000 7fe318c00640 1 -- 172.21.15.90:0/509933112 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3141f2e00 con 0x7fe314144680 2024-03-24T01:29:01.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.636+0000 7fe30be00640 1 -- 172.21.15.90:0/509933112 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc011c40 con 0x7fe314144680 2024-03-24T01:29:01.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.636+0000 7fe318c00640 1 -- 172.21.15.90:0/509933112 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3141f33c0 con 0x7fe314144680 2024-03-24T01:29:01.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.637+0000 7fe318c00640 1 -- 172.21.15.90:0/509933112 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe31404f400 con 0x7fe314144680 2024-03-24T01:29:01.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.638+0000 7fe30be00640 1 -- 172.21.15.90:0/509933112 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe2fc018020 con 0x7fe314144680 2024-03-24T01:29:01.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.639+0000 7fe30be00640 1 --2- 172.21.15.90:0/509933112 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe2f006bbd0 0x7fe2f006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:01.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.640+0000 7fe30be00640 1 -- 172.21.15.90:0/509933112 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fe2fc08ac30 con 0x7fe314144680 2024-03-24T01:29:01.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.640+0000 7fe313400640 1 --2- 172.21.15.90:0/509933112 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe2f006bbd0 0x7fe2f006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:29:01.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.641+0000 7fe313400640 1 --2- 172.21.15.90:0/509933112 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe2f006bbd0 0x7fe2f006e090 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fe304004ea0 tx=0x7fe304009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:29:01.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.644+0000 7fe30be00640 1 -- 172.21.15.90:0/509933112 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe2fc01f020 con 0x7fe314144680 2024-03-24T01:29:01.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.852+0000 7fe318c00640 1 -- 172.21.15.90:0/509933112 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe31406b540 con 0x7fe314144680 2024-03-24T01:29:01.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.853+0000 7fe30be00640 1 -- 172.21.15.90:0/509933112 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v55) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe2fc0172c0 con 0x7fe314144680 2024-03-24T01:29:01.853 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-24T01:29:01.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.856+0000 7fe318c00640 1 -- 172.21.15.90:0/509933112 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe2f006bbd0 msgr2=0x7fe2f006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:29:01.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.856+0000 7fe318c00640 1 --2- 172.21.15.90:0/509933112 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe2f006bbd0 0x7fe2f006e090 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fe304004ea0 tx=0x7fe304009040 comp rx=0 tx=0).stop 2024-03-24T01:29:01.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.856+0000 7fe318c00640 1 -- 172.21.15.90:0/509933112 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe314144680 msgr2=0x7fe314142de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:29:01.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.856+0000 7fe318c00640 1 --2- 172.21.15.90:0/509933112 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe314144680 0x7fe314142de0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc00b410 tx=0x7fe2fc00b8d0 comp rx=0 tx=0).stop 2024-03-24T01:29:01.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.856+0000 7fe318c00640 1 -- 172.21.15.90:0/509933112 shutdown_connections 2024-03-24T01:29:01.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.856+0000 7fe318c00640 1 --2- 172.21.15.90:0/509933112 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe2f006bbd0 0x7fe2f006e090 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:01.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.856+0000 7fe318c00640 1 --2- 172.21.15.90:0/509933112 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe314144680 0x7fe314142de0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:01.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.856+0000 7fe318c00640 1 --2- 172.21.15.90:0/509933112 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe314145190 0x7fe3141f1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:01.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.856+0000 7fe318c00640 1 --2- 172.21.15.90:0/509933112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe314071cd0 0x7fe314141000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:01.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.856+0000 7fe318c00640 1 -- 172.21.15.90:0/509933112 >> 172.21.15.90:0/509933112 conn(0x7fe31406b0c0 msgr2=0x7fe31406d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:29:01.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.856+0000 7fe318c00640 1 -- 172.21.15.90:0/509933112 shutdown_connections 2024-03-24T01:29:01.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:01.856+0000 7fe318c00640 1 -- 172.21.15.90:0/509933112 wait complete. 2024-03-24T01:29:01.876 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi090.front.sepia.ceph.com... 2024-03-24T01:29:01.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:29:01.876 DEBUG:teuthology.orchestra.run.smithi090:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-24T01:29:01.964 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[59444]: fuse finished with error 0 and tester_r 0 2024-03-24T01:29:01.968 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:29:08.069 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-03-24T01:29:08.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:29:08.070 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T01:29:08.129 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:29:08.129 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:29:08.182 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T01:29:08.182 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.1 is not mounted at ubuntu@smithi120.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1 2024-03-24T01:29:08.182 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2024-03-24T01:29:08.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:29:08.182 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-24T01:29:08.231 INFO:teuthology.orchestra.run.smithi120.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-24T01:29:08.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T01:29:08.232 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:29:08.232 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:29:08.289 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053414,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498340k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249168k,nr_inodes=812292,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498340k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T01:29:08.289 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.2 is not mounted at ubuntu@smithi090.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.2 2024-03-24T01:29:08.289 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-03-24T01:29:08.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:29:08.290 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-24T01:29:08.342 INFO:teuthology.orchestra.run.smithi090.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-24T01:29:08.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T01:29:08.343 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:29:08.343 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:29:08.404 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T01:29:08.405 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.3 is not mounted at ubuntu@smithi120.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.3 2024-03-24T01:29:08.405 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2024-03-24T01:29:08.405 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:29:08.405 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-24T01:29:08.454 INFO:teuthology.orchestra.run.smithi120.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-24T01:29:08.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T01:29:08.455 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T01:29:08.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.673+0000 7f4a32c00640 1 Processor -- start 2024-03-24T01:29:08.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.673+0000 7f4a32c00640 1 -- start start 2024-03-24T01:29:08.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.675+0000 7f4a32c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4a2c15b9f0 0x7f4a2c071990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:08.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.675+0000 7f4a32c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a2c06eba0 0x7f4a2c0acea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:08.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.675+0000 7f4a32c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a2c15f2e0 con 0x7f4a2c071ed0 2024-03-24T01:29:08.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.675+0000 7f4a32c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a2c072b00 con 0x7f4a2c15b9f0 2024-03-24T01:29:08.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.675+0000 7f4a32c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c15ee50 con 0x7f4a2c06eba0 2024-03-24T01:29:08.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.675+0000 7f4a23400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4a2c071ed0 0x7f4a2c06e490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:33466/0 (socket says 172.21.15.90:33466) 2024-03-24T01:29:08.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.675+0000 7f4a23400640 1 -- 172.21.15.90:0/2251860548 learned_addr learned my addr 172.21.15.90:0/2251860548 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:29:08.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.675+0000 7f4a30c00640 1 --2- 172.21.15.90:0/2251860548 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a2c06eba0 0x7f4a2c0acea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:29:08.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.675+0000 7f4a2b400640 1 -- 172.21.15.90:0/2251860548 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 52791475 0 0) 0x7f4a2c15f2e0 con 0x7f4a2c071ed0 2024-03-24T01:29:08.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.675+0000 7f4a2b400640 1 -- 172.21.15.90:0/2251860548 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a10003520 con 0x7f4a2c071ed0 2024-03-24T01:29:08.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.676+0000 7f4a2b400640 1 -- 172.21.15.90:0/2251860548 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3922656194 0 0) 0x7f4a10003520 con 0x7f4a2c071ed0 2024-03-24T01:29:08.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.676+0000 7f4a2b400640 1 -- 172.21.15.90:0/2251860548 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a2c06eba0 msgr2=0x7f4a2c0acea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:29:08.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.676+0000 7f4a2b400640 1 --2- 172.21.15.90:0/2251860548 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a2c06eba0 0x7f4a2c0acea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:08.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.676+0000 7f4a2be00640 1 --2- 172.21.15.90:0/2251860548 >> v2:172.21.15.120:3300/0 conn(0x7f4a2c15b9f0 0x7f4a2c071990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:29:08.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.676+0000 7f4a2b400640 1 -- 172.21.15.90:0/2251860548 >> v2:172.21.15.120:3300/0 conn(0x7f4a2c15b9f0 msgr2=0x7f4a2c071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:29:08.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.676+0000 7f4a2b400640 1 --2- 172.21.15.90:0/2251860548 >> v2:172.21.15.120:3300/0 conn(0x7f4a2c15b9f0 0x7f4a2c071990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:08.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.676+0000 7f4a2b400640 1 -- 172.21.15.90:0/2251860548 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a2c06f070 con 0x7f4a2c071ed0 2024-03-24T01:29:08.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.676+0000 7f4a2b400640 1 -- 172.21.15.90:0/2251860548 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f4a14001670 con 0x7f4a2c071ed0 2024-03-24T01:29:08.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.676+0000 7f4a2b400640 1 -- 172.21.15.90:0/2251860548 >> v1:172.21.15.90:6789/0 conn(0x7f4a2c071ed0 legacy=0x7f4a2c06e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:29:08.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.676+0000 7f4a2b400640 1 --2- 172.21.15.90:0/2251860548 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a100039b0 0x7f4a10003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:08.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.676+0000 7f4a2b400640 1 --2- 172.21.15.90:0/2251860548 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a10004800 0x7f4a10004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:08.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.676+0000 7f4a2b400640 1 --2- 172.21.15.90:0/2251860548 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a2c06eba0 0x7f4a100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:08.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.676+0000 7f4a2b400640 1 -- 172.21.15.90:0/2251860548 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c15ee50 con 0x7f4a100039b0 2024-03-24T01:29:08.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.676+0000 7f4a2b400640 1 -- 172.21.15.90:0/2251860548 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1000ba70 con 0x7f4a10004800 2024-03-24T01:29:08.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.676+0000 7f4a2b400640 1 -- 172.21.15.90:0/2251860548 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1000bc20 con 0x7f4a2c06eba0 2024-03-24T01:29:08.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.676+0000 7f4a2be00640 1 --2- 172.21.15.90:0/2251860548 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a100039b0 0x7f4a10003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:29:08.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.676+0000 7f4a30c00640 1 --2- 172.21.15.90:0/2251860548 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a2c06eba0 0x7f4a100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:29:08.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.676+0000 7f4a23400640 1 --2- 172.21.15.90:0/2251860548 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a10004800 0x7f4a10004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:29:08.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.677+0000 7f4a2be00640 1 -- 172.21.15.90:0/2251860548 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a2c06eba0 msgr2=0x7f4a100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:29:08.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.677+0000 7f4a2be00640 1 --2- 172.21.15.90:0/2251860548 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a2c06eba0 0x7f4a100051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:08.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.677+0000 7f4a2be00640 1 -- 172.21.15.90:0/2251860548 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a10004800 msgr2=0x7f4a10004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:29:08.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.677+0000 7f4a2be00640 1 --2- 172.21.15.90:0/2251860548 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a10004800 0x7f4a10004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:08.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.677+0000 7f4a2be00640 1 -- 172.21.15.90:0/2251860548 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a10003520 con 0x7f4a100039b0 2024-03-24T01:29:08.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.678+0000 7f4a2be00640 1 --2- 172.21.15.90:0/2251860548 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a100039b0 0x7f4a10003dd0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c00a970 tx=0x7f4a10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9092fa0d8073903e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:29:08.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.678+0000 7f4a2b400640 1 -- 172.21.15.90:0/2251860548 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c034070 con 0x7f4a100039b0 2024-03-24T01:29:08.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.678+0000 7f4a2b400640 1 -- 172.21.15.90:0/2251860548 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a1c030400 con 0x7f4a100039b0 2024-03-24T01:29:08.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.678+0000 7f4a2b400640 1 -- 172.21.15.90:0/2251860548 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c00a570 con 0x7f4a100039b0 2024-03-24T01:29:08.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.678+0000 7f4a32c00640 1 -- 172.21.15.90:0/2251860548 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a100039b0 msgr2=0x7f4a10003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:29:08.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.678+0000 7f4a32c00640 1 --2- 172.21.15.90:0/2251860548 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a100039b0 0x7f4a10003dd0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c00a970 tx=0x7f4a10005830 comp rx=0 tx=0).stop 2024-03-24T01:29:08.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.679+0000 7f4a30c00640 1 -- 172.21.15.90:0/2251860548 reap_dead start 2024-03-24T01:29:08.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.679+0000 7f4a32c00640 1 -- 172.21.15.90:0/2251860548 shutdown_connections 2024-03-24T01:29:08.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.679+0000 7f4a32c00640 1 -- 172.21.15.90:0/2251860548 >> 172.21.15.90:0/2251860548 conn(0x7f4a2c0cf7e0 msgr2=0x7f4a2c0cfbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:29:08.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.679+0000 7f4a32c00640 1 -- 172.21.15.90:0/2251860548 shutdown_connections 2024-03-24T01:29:08.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.679+0000 7f4a32c00640 1 -- 172.21.15.90:0/2251860548 wait complete. 2024-03-24T01:29:08.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.680+0000 7f4a32c00640 1 Processor -- start 2024-03-24T01:29:08.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.681+0000 7f4a32c00640 1 -- start start 2024-03-24T01:29:08.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.681+0000 7f4a32c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a100039b0 0x7f4a2c0cc270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:08.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.681+0000 7f4a32c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a2c153130 0x7f4a2c1f74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:08.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.681+0000 7f4a32c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a2c14f350 0x7f4a2c0b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:08.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.681+0000 7f4a32c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c15eae0 con 0x7f4a2c153130 2024-03-24T01:29:08.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.681+0000 7f4a32c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c15e800 con 0x7f4a100039b0 2024-03-24T01:29:08.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.681+0000 7f4a32c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c0acea0 con 0x7f4a2c14f350 2024-03-24T01:29:08.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.682+0000 7f4a23400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a2c153130 0x7f4a2c1f74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:29:08.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.682+0000 7f4a23400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a2c153130 0x7f4a2c1f74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40518/0 (socket says 172.21.15.90:40518) 2024-03-24T01:29:08.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.682+0000 7f4a23400640 1 -- 172.21.15.90:0/1332700076 learned_addr learned my addr 172.21.15.90:0/1332700076 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:29:08.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.682+0000 7f4a30c00640 1 --2- 172.21.15.90:0/1332700076 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a2c14f350 0x7f4a2c0b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:29:08.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.682+0000 7f4a2be00640 1 --2- 172.21.15.90:0/1332700076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a100039b0 0x7f4a2c0cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:29:08.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.682+0000 7f4a23400640 1 -- 172.21.15.90:0/1332700076 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a2c14f350 msgr2=0x7f4a2c0b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:29:08.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.682+0000 7f4a23400640 1 --2- 172.21.15.90:0/1332700076 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a2c14f350 0x7f4a2c0b36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:08.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.682+0000 7f4a23400640 1 -- 172.21.15.90:0/1332700076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a100039b0 msgr2=0x7f4a2c0cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:29:08.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.682+0000 7f4a23400640 1 --2- 172.21.15.90:0/1332700076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a100039b0 0x7f4a2c0cc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:08.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.682+0000 7f4a23400640 1 -- 172.21.15.90:0/1332700076 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a2c1f6ce0 con 0x7f4a2c153130 2024-03-24T01:29:08.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.682+0000 7f4a23400640 1 --2- 172.21.15.90:0/1332700076 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a2c153130 0x7f4a2c1f74a0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f4a14001200 tx=0x7f4a1400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:29:08.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.683+0000 7f4a28c00640 1 -- 172.21.15.90:0/1332700076 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1400ee80 con 0x7f4a2c153130 2024-03-24T01:29:08.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.683+0000 7f4a28c00640 1 -- 172.21.15.90:0/1332700076 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a14018070 con 0x7f4a2c153130 2024-03-24T01:29:08.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.683+0000 7f4a28c00640 1 -- 172.21.15.90:0/1332700076 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1400cb10 con 0x7f4a2c153130 2024-03-24T01:29:08.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.683+0000 7f4a32c00640 1 -- 172.21.15.90:0/1332700076 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a2c0cca10 con 0x7f4a2c153130 2024-03-24T01:29:08.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.683+0000 7f4a32c00640 1 -- 172.21.15.90:0/1332700076 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a2c1f6f80 con 0x7f4a2c153130 2024-03-24T01:29:08.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.684+0000 7f4a32c00640 1 -- 172.21.15.90:0/1332700076 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a2c04f400 con 0x7f4a2c153130 2024-03-24T01:29:08.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.685+0000 7f4a28c00640 1 -- 172.21.15.90:0/1332700076 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4a1400ccb0 con 0x7f4a2c153130 2024-03-24T01:29:08.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.685+0000 7f4a28c00640 1 --2- 172.21.15.90:0/1332700076 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4a0806bc50 0x7f4a0806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:29:08.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.686+0000 7f4a28c00640 1 -- 172.21.15.90:0/1332700076 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f4a1408ab60 con 0x7f4a2c153130 2024-03-24T01:29:08.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.686+0000 7f4a2be00640 1 --2- 172.21.15.90:0/1332700076 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4a0806bc50 0x7f4a0806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:29:08.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.688+0000 7f4a2be00640 1 --2- 172.21.15.90:0/1332700076 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4a0806bc50 0x7f4a0806e110 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c036000 tx=0x7f4a1c03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:29:08.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.691+0000 7f4a28c00640 1 -- 172.21.15.90:0/1332700076 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4a2c06f070 con 0x7f4a2c153130 2024-03-24T01:29:08.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.933+0000 7f4a32c00640 1 -- 172.21.15.90:0/1332700076 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f4a2c1f7150 con 0x7f4a2c153130 2024-03-24T01:29:08.935 INFO:tasks.ceph.mon.a.smithi090.stderr:2024-03-24T01:29:08.935+0000 7fc6f2400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T01:29:08.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.939+0000 7f4a28c00640 1 -- 172.21.15.90:0/1332700076 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v35) v1 ==== 161+0+0 (secure 0 0 0) 0x7f4a14059470 con 0x7f4a2c153130 2024-03-24T01:29:08.939 INFO:teuthology.orchestra.run.smithi090.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T01:29:08.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.942+0000 7f4a32c00640 1 -- 172.21.15.90:0/1332700076 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4a0806bc50 msgr2=0x7f4a0806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:29:08.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.942+0000 7f4a32c00640 1 --2- 172.21.15.90:0/1332700076 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4a0806bc50 0x7f4a0806e110 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c036000 tx=0x7f4a1c03b040 comp rx=0 tx=0).stop 2024-03-24T01:29:08.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.942+0000 7f4a32c00640 1 -- 172.21.15.90:0/1332700076 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a2c153130 msgr2=0x7f4a2c1f74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:29:08.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.942+0000 7f4a32c00640 1 --2- 172.21.15.90:0/1332700076 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a2c153130 0x7f4a2c1f74a0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f4a14001200 tx=0x7f4a1400ec30 comp rx=0 tx=0).stop 2024-03-24T01:29:08.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.942+0000 7f4a32c00640 1 -- 172.21.15.90:0/1332700076 shutdown_connections 2024-03-24T01:29:08.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.942+0000 7f4a32c00640 1 --2- 172.21.15.90:0/1332700076 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4a0806bc50 0x7f4a0806e110 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:08.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.942+0000 7f4a32c00640 1 --2- 172.21.15.90:0/1332700076 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a2c153130 0x7f4a2c1f74a0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:08.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.942+0000 7f4a32c00640 1 --2- 172.21.15.90:0/1332700076 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4a2c14f350 0x7f4a2c0b36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:08.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.942+0000 7f4a32c00640 1 --2- 172.21.15.90:0/1332700076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a100039b0 0x7f4a2c0cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:29:08.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.942+0000 7f4a32c00640 1 -- 172.21.15.90:0/1332700076 >> 172.21.15.90:0/1332700076 conn(0x7f4a2c0cf7e0 msgr2=0x7f4a2c0b05e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:29:08.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.942+0000 7f4a32c00640 1 -- 172.21.15.90:0/1332700076 shutdown_connections 2024-03-24T01:29:08.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:29:08.942+0000 7f4a32c00640 1 -- 172.21.15.90:0/1332700076 wait complete. 2024-03-24T01:29:08.962 INFO:tasks.workunit:Pulling workunits from ref 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T01:29:08.962 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-24T01:29:08.963 DEBUG:teuthology.orchestra.run.smithi090:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-24T01:29:09.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T01:29:09.007 INFO:teuthology.orchestra.run.smithi090.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-24T01:29:09.008 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-03-24T01:29:09.061 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2024-03-24T01:29:09.103 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-03-24T01:29:09.104 DEBUG:teuthology.orchestra.run.smithi090:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-03-24T01:29:09.152 INFO:tasks.workunit:timeout=1h 2024-03-24T01:29:09.152 INFO:tasks.workunit:cleanup=True 2024-03-24T01:29:09.152 DEBUG:teuthology.orchestra.run.smithi090:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T01:29:09.203 INFO:tasks.workunit.client.0.smithi090.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-24T01:29:27.934 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:29:27.943 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:29:58.073 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:29:58.082 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:29:59.999 INFO:tasks.ceph.mon.a.smithi090.stderr:2024-03-24T01:29:59.999+0000 7fc6f5600640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-24T01:30:29.477 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:30:29.485 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:30:31.550 INFO:tasks.workunit.client.0.smithi090.stderr:Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-24T01:30:32.132 INFO:tasks.workunit.client.0.smithi090.stderr:Note: switching to '3be9a5c9bc793e11e2800a8c0c696e8b46742033'. 2024-03-24T01:30:32.133 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-03-24T01:30:32.133 INFO:tasks.workunit.client.0.smithi090.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-24T01:30:32.133 INFO:tasks.workunit.client.0.smithi090.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-24T01:30:32.134 INFO:tasks.workunit.client.0.smithi090.stderr:state without impacting any branches by switching back to a branch. 2024-03-24T01:30:32.134 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-03-24T01:30:32.135 INFO:tasks.workunit.client.0.smithi090.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-24T01:30:32.135 INFO:tasks.workunit.client.0.smithi090.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-24T01:30:32.135 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-03-24T01:30:32.135 INFO:tasks.workunit.client.0.smithi090.stderr: git switch -c 2024-03-24T01:30:32.135 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-03-24T01:30:32.135 INFO:tasks.workunit.client.0.smithi090.stderr:Or undo this operation with: 2024-03-24T01:30:32.136 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-03-24T01:30:32.136 INFO:tasks.workunit.client.0.smithi090.stderr: git switch - 2024-03-24T01:30:32.136 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-03-24T01:30:32.136 INFO:tasks.workunit.client.0.smithi090.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-24T01:30:32.136 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-03-24T01:30:32.136 INFO:tasks.workunit.client.0.smithi090.stderr:HEAD is now at 3be9a5c9bc7 dummy commit #1 2024-03-24T01:30:32.141 DEBUG:teuthology.orchestra.run.smithi090:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-24T01:30:32.190 INFO:tasks.workunit.client.0.smithi090.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-24T01:30:32.196 INFO:tasks.workunit.client.0.smithi090.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-24T01:30:32.196 INFO:tasks.workunit.client.0.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-24T01:30:32.288 INFO:tasks.workunit.client.0.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-24T01:30:32.385 INFO:tasks.workunit.client.0.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-24T01:30:32.463 INFO:tasks.workunit.client.0.smithi090.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-24T01:30:32.469 INFO:tasks.workunit.client.0.smithi090.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-24T01:30:32.469 INFO:tasks.workunit.client.0.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-24T01:30:32.549 INFO:tasks.workunit.client.0.smithi090.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-24T01:30:32.557 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:30:32.557 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-24T01:30:32.605 INFO:tasks.workunit:Running workunits matching suites/cephfs_journal_tool_smoke.sh on client.0... 2024-03-24T01:30:32.606 INFO:tasks.workunit:Running workunit suites/cephfs_journal_tool_smoke.sh... 2024-03-24T01:30:32.607 DEBUG:teuthology.orchestra.run.smithi090:workunit test suites/cephfs_journal_tool_smoke.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/cephfs_journal_tool_smoke.sh 2024-03-24T01:30:32.719 INFO:tasks.workunit.client.0.smithi090.stderr:+ export 'BIN=cephfs-journal-tool --rank=cephfs:0' 2024-03-24T01:30:32.719 INFO:tasks.workunit.client.0.smithi090.stderr:+ BIN='cephfs-journal-tool --rank=cephfs:0' 2024-03-24T01:30:32.719 INFO:tasks.workunit.client.0.smithi090.stderr:+ export JOURNAL_FILE=/tmp/journal.bin 2024-03-24T01:30:32.719 INFO:tasks.workunit.client.0.smithi090.stderr:+ JOURNAL_FILE=/tmp/journal.bin 2024-03-24T01:30:32.719 INFO:tasks.workunit.client.0.smithi090.stderr:+ export JSON_OUTPUT=/tmp/json.tmp 2024-03-24T01:30:32.719 INFO:tasks.workunit.client.0.smithi090.stderr:+ JSON_OUTPUT=/tmp/json.tmp 2024-03-24T01:30:32.720 INFO:tasks.workunit.client.0.smithi090.stderr:+ export BINARY_OUTPUT=/tmp/binary.tmp 2024-03-24T01:30:32.720 INFO:tasks.workunit.client.0.smithi090.stderr:+ BINARY_OUTPUT=/tmp/binary.tmp 2024-03-24T01:30:32.720 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' -d /tmp/binary.tmp ']' 2024-03-24T01:30:32.720 INFO:tasks.workunit.client.0.smithi090.stdout:Testing that export/import cycle preserves state 2024-03-24T01:30:32.720 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'Testing that export/import cycle preserves state' 2024-03-24T01:30:32.721 INFO:tasks.workunit.client.0.smithi090.stderr:++ cephfs-journal-tool --rank=cephfs:0 header get 2024-03-24T01:30:32.771 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.770+0000 7f0b4703a280 1 -- start start 2024-03-24T01:30:32.771 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.770+0000 7f0b4703a280 1 Processor -- start 2024-03-24T01:30:32.774 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.771+0000 7f0b4703a280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c9e9eda400 0x55c9e915f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.774 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.771+0000 7f0b4703a280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9e9edb000 0x55c9e915eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.774 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.771+0000 7f0b4703a280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9e9edb400 0x55c9e915f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.774 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.771+0000 7f0b4703a280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9e9fb0000 con 0x55c9e9edb000 2024-03-24T01:30:32.774 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.771+0000 7f0b4703a280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9e9fb0180 con 0x55c9e9edb400 2024-03-24T01:30:32.774 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.771+0000 7f0b4703a280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c9e9fb0300 con 0x55c9e9eda400 2024-03-24T01:30:32.774 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.771+0000 7f0b44000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9e9edb000 0x55c9e915eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.774 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.771+0000 7f0b44000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9e9edb000 0x55c9e915eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35982/0 (socket says 172.21.15.90:35982) 2024-03-24T01:30:32.775 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.771+0000 7f0b44000640 1 -- 172.21.15.90:0/1622405940 learned_addr learned my addr 172.21.15.90:0/1622405940 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:32.775 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.772+0000 7f0b45400640 1 --2- 172.21.15.90:0/1622405940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9e9edb400 0x55c9e915f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.775 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.772+0000 7f0b44a00640 1 --2- 172.21.15.90:0/1622405940 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c9e9eda400 0x55c9e915f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.775 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.772+0000 7f0b44000640 1 -- 172.21.15.90:0/1622405940 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c9e9eda400 msgr2=0x55c9e915f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:32.775 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.772+0000 7f0b44000640 1 --2- 172.21.15.90:0/1622405940 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c9e9eda400 0x55c9e915f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.775 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.772+0000 7f0b44000640 1 -- 172.21.15.90:0/1622405940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9e9edb400 msgr2=0x55c9e915f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:32.775 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.772+0000 7f0b44000640 1 --2- 172.21.15.90:0/1622405940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9e9edb400 0x55c9e915f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.775 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.772+0000 7f0b44000640 1 -- 172.21.15.90:0/1622405940 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c9e9128780 con 0x55c9e9edb000 2024-03-24T01:30:32.775 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.772+0000 7f0b45400640 1 --2- 172.21.15.90:0/1622405940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9e9edb400 0x55c9e915f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:30:32.775 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.772+0000 7f0b44000640 1 --2- 172.21.15.90:0/1622405940 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9e9edb000 0x55c9e915eb00 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x55c9e9f18a20 tx=0x55c9e9eccf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e18d803b1b88fb8d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:32.775 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.772+0000 7f0b41800640 1 -- 172.21.15.90:0/1622405940 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55c9e911ab60 con 0x55c9e9edb000 2024-03-24T01:30:32.775 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.772+0000 7f0b41800640 1 -- 172.21.15.90:0/1622405940 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c9e9120700 con 0x55c9e9edb000 2024-03-24T01:30:32.775 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.772+0000 7f0b41800640 1 -- 172.21.15.90:0/1622405940 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55c9e911ad00 con 0x55c9e9edb000 2024-03-24T01:30:32.776 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.772+0000 7f0b4703a280 1 -- 172.21.15.90:0/1622405940 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c9e9128000 con 0x55c9e9edb000 2024-03-24T01:30:32.776 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.772+0000 7f0b4703a280 1 -- 172.21.15.90:0/1622405940 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x55c9e9129a40 con 0x55c9e9edb000 2024-03-24T01:30:32.776 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.773+0000 7f0b41800640 1 -- 172.21.15.90:0/1622405940 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x55c9e9e9ac00 con 0x55c9e9edb000 2024-03-24T01:30:32.776 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.773+0000 7f0b41800640 1 -- 172.21.15.90:0/1622405940 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 36) v1 ==== 3310+0+0 (secure 0 0 0) 0x55c9e914f180 con 0x55c9e9edb000 2024-03-24T01:30:32.788 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.774+0000 7f0b4703a280 1 Processor -- start 2024-03-24T01:30:32.788 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.774+0000 7f0b4703a280 1 -- start start 2024-03-24T01:30:32.789 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.775+0000 7f0b4703a280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9e9edbc00 0x55c9e915e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.789 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.775+0000 7f0b4703a280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9e9fd6400 0x55c9e9160100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.789 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.775+0000 7f0b4703a280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c9e9fd6800 0x55c9e9160680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.789 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.775+0000 7f0b4703a280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9e9fb0480 con 0x55c9e9edbc00 2024-03-24T01:30:32.789 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.775+0000 7f0b4703a280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9e9fb0600 con 0x55c9e9fd6400 2024-03-24T01:30:32.789 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.775+0000 7f0b4703a280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c9e9fb0780 con 0x55c9e9fd6800 2024-03-24T01:30:32.789 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.775+0000 7f0b45400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9e9edbc00 0x55c9e915e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.789 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.775+0000 7f0b45400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9e9edbc00 0x55c9e915e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35986/0 (socket says 172.21.15.90:35986) 2024-03-24T01:30:32.789 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.775+0000 7f0b45400640 1 -- 172.21.15.90:0/1579685741 learned_addr learned my addr 172.21.15.90:0/1579685741 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:32.789 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.775+0000 7f0b45400640 1 -- 172.21.15.90:0/1579685741 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c9e9fd6800 msgr2=0x55c9e9160680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:32.789 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.775+0000 7f0b45400640 1 --2- 172.21.15.90:0/1579685741 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c9e9fd6800 0x55c9e9160680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.789 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.775+0000 7f0b45400640 1 -- 172.21.15.90:0/1579685741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9e9fd6400 msgr2=0x55c9e9160100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:32.789 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.775+0000 7f0b45400640 1 --2- 172.21.15.90:0/1579685741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9e9fd6400 0x55c9e9160100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.790 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.775+0000 7f0b45400640 1 -- 172.21.15.90:0/1579685741 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c9e9129860 con 0x55c9e9edbc00 2024-03-24T01:30:32.790 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.776+0000 7f0b45400640 1 --2- 172.21.15.90:0/1579685741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9e9edbc00 0x55c9e915e580 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x55c9e9fd2a20 tx=0x55c9e9fd8cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24e45f7f86e2827f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:32.790 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.776+0000 7f0b3f000640 1 -- 172.21.15.90:0/1579685741 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55c9e911ab60 con 0x55c9e9edbc00 2024-03-24T01:30:32.790 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.776+0000 7f0b3f000640 1 -- 172.21.15.90:0/1579685741 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c9e9120c40 con 0x55c9e9edbc00 2024-03-24T01:30:32.790 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.776+0000 7f0b3f000640 1 -- 172.21.15.90:0/1579685741 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55c9e911ad00 con 0x55c9e9edbc00 2024-03-24T01:30:32.790 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.776+0000 7f0b4703a280 1 -- 172.21.15.90:0/1579685741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9e9edbc00 msgr2=0x55c9e915e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:32.790 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.776+0000 7f0b4703a280 1 --2- 172.21.15.90:0/1579685741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9e9edbc00 0x55c9e915e580 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x55c9e9fd2a20 tx=0x55c9e9fd8cc0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.790 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.776+0000 7f0b4703a280 1 -- 172.21.15.90:0/1579685741 shutdown_connections 2024-03-24T01:30:32.790 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.776+0000 7f0b4703a280 1 --2- 172.21.15.90:0/1579685741 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c9e9fd6800 0x55c9e9160680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.790 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.776+0000 7f0b4703a280 1 --2- 172.21.15.90:0/1579685741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9e9fd6400 0x55c9e9160100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.790 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.776+0000 7f0b4703a280 1 --2- 172.21.15.90:0/1579685741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9e9edbc00 0x55c9e915e580 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.790 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.776+0000 7f0b4703a280 1 -- 172.21.15.90:0/1579685741 >> 172.21.15.90:0/1579685741 conn(0x55c9e9edb800 msgr2=0x55c9e91ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:32.790 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.776+0000 7f0b4703a280 1 -- 172.21.15.90:0/1579685741 shutdown_connections 2024-03-24T01:30:32.790 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.776+0000 7f0b4703a280 1 -- 172.21.15.90:0/1579685741 wait complete. 2024-03-24T01:30:32.791 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.777+0000 7f0b4703a280 1 Processor -- start 2024-03-24T01:30:32.791 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.777+0000 7f0b4703a280 1 -- start start 2024-03-24T01:30:32.791 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.778+0000 7f0b4703a280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9e9edb800 0x55c9e9161700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.791 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.778+0000 7f0b4703a280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9e9fd7400 0x55c9e9161180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.791 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.778+0000 7f0b4703a280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c9e9fd7000 0x55c9e9160c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.791 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.778+0000 7f0b4703a280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9e9fb1b00 con 0x55c9e9fd7400 2024-03-24T01:30:32.791 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.778+0000 7f0b4703a280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9e9fb1980 con 0x55c9e9edb800 2024-03-24T01:30:32.791 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.778+0000 7f0b4703a280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c9e9fb0c00 con 0x55c9e9fd7000 2024-03-24T01:30:32.791 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.778+0000 7f0b44a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9e9fd7400 0x55c9e9161180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.791 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.778+0000 7f0b44a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9e9fd7400 0x55c9e9161180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36000/0 (socket says 172.21.15.90:36000) 2024-03-24T01:30:32.791 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.778+0000 7f0b44a00640 1 -- 172.21.15.90:0/2112149533 learned_addr learned my addr 172.21.15.90:0/2112149533 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:32.791 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.779+0000 7f0b44000640 1 --2- 172.21.15.90:0/2112149533 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c9e9fd7000 0x55c9e9160c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.791 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.779+0000 7f0b45400640 1 --2- 172.21.15.90:0/2112149533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9e9edb800 0x55c9e9161700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.791 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.779+0000 7f0b44a00640 1 -- 172.21.15.90:0/2112149533 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c9e9fd7000 msgr2=0x55c9e9160c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:32.792 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.779+0000 7f0b44a00640 1 --2- 172.21.15.90:0/2112149533 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c9e9fd7000 0x55c9e9160c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.792 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.779+0000 7f0b44a00640 1 -- 172.21.15.90:0/2112149533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9e9edb800 msgr2=0x55c9e9161700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:32.792 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.779+0000 7f0b44a00640 1 --2- 172.21.15.90:0/2112149533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9e9edb800 0x55c9e9161700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.792 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.779+0000 7f0b44a00640 1 -- 172.21.15.90:0/2112149533 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c9e9129680 con 0x55c9e9fd7400 2024-03-24T01:30:32.792 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.779+0000 7f0b44000640 1 --2- 172.21.15.90:0/2112149533 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c9e9fd7000 0x55c9e9160c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:30:32.792 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.779+0000 7f0b45400640 1 --2- 172.21.15.90:0/2112149533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9e9edb800 0x55c9e9161700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:30:32.792 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.779+0000 7f0b44a00640 1 --2- 172.21.15.90:0/2112149533 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9e9fd7400 0x55c9e9161180 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x55c9e9fd3830 tx=0x55c9e9fd8de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:32.792 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.779+0000 7f0b3d200640 1 -- 172.21.15.90:0/2112149533 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55c9e911ab60 con 0x55c9e9fd7400 2024-03-24T01:30:32.792 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.779+0000 7f0b3d200640 1 -- 172.21.15.90:0/2112149533 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c9e91208c0 con 0x55c9e9fd7400 2024-03-24T01:30:32.792 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.779+0000 7f0b3d200640 1 -- 172.21.15.90:0/2112149533 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55c9e911ad00 con 0x55c9e9fd7400 2024-03-24T01:30:32.792 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.779+0000 7f0b4703a280 1 -- 172.21.15.90:0/2112149533 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55c9e91283c0 con 0x55c9e9fd7400 2024-03-24T01:30:32.792 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.779+0000 7f0b4703a280 1 -- 172.21.15.90:0/2112149533 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c9e9129c20 con 0x55c9e9fd7400 2024-03-24T01:30:32.792 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.781+0000 7f0b3d200640 1 -- 172.21.15.90:0/2112149533 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x55c9e9feac00 con 0x55c9e9fd7400 2024-03-24T01:30:32.792 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.782+0000 7f0b3d200640 1 --2- 172.21.15.90:0/2112149533 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55c9ea047000 0x55c9e9470000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.793 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.782+0000 7f0b3d200640 1 -- 172.21.15.90:0/2112149533 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x55c9ea038c00 con 0x55c9e9fd7400 2024-03-24T01:30:32.793 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.782+0000 7f0b4703a280 1 --2- 172.21.15.90:0/2112149533 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55c9e9fd6c00 0x55c9e9160680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.793 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.782+0000 7f0b4703a280 1 -- 172.21.15.90:0/2112149533 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:1 7.f 7:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55c9e9fd6800 con 0x55c9e9fd6c00 2024-03-24T01:30:32.793 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.782+0000 7f0b44000640 1 --2- 172.21.15.90:0/2112149533 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55c9e9fd6c00 0x55c9e9160680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.793 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.783+0000 7f0b45400640 1 --2- 172.21.15.90:0/2112149533 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55c9ea047000 0x55c9e9470000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.793 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.783+0000 7f0b44000640 1 --2- 172.21.15.90:0/2112149533 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55c9e9fd6c00 0x55c9e9160680 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:32.793 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.784+0000 7f0b44000640 1 -- 172.21.15.90:0/2112149533 <== osd.0 v2:172.21.15.90:6800/307045587 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x55c9e9e78000 con 0x55c9e9fd6c00 2024-03-24T01:30:32.793 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.784+0000 7f0b45400640 1 --2- 172.21.15.90:0/2112149533 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55c9ea047000 0x55c9e9470000 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x55c9e9fd2a20 tx=0x55c9e9fd8c00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:32.793 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.784+0000 7f0b4703a280 1 --2- 172.21.15.90:0/2112149533 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55c9e9fd6400 0x55c9e9160100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.793 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.784+0000 7f0b4703a280 1 -- 172.21.15.90:0/2112149533 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:2 7.14 7:292cf221:::200.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55c9ea047400 con 0x55c9e9fd6400 2024-03-24T01:30:32.793 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.784+0000 7f0b45400640 1 --2- 172.21.15.90:0/2112149533 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55c9e9fd6400 0x55c9e9160100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.793 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.784+0000 7f0b45400640 1 --2- 172.21.15.90:0/2112149533 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55c9e9fd6400 0x55c9e9160100 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:32.793 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.785+0000 7f0b45400640 1 -- 172.21.15.90:0/2112149533 <== osd.3 v2:172.21.15.90:6808/2999931983 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv12 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x55c9e9e78500 con 0x55c9e9fd6400 2024-03-24T01:30:32.793 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.785+0000 7f0b4703a280 1 -- 172.21.15.90:0/2112149533 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55c9e9fd6c00 msgr2=0x55c9e9160680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:32.794 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.785+0000 7f0b4703a280 1 --2- 172.21.15.90:0/2112149533 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55c9e9fd6c00 0x55c9e9160680 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.794 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.785+0000 7f0b4703a280 1 -- 172.21.15.90:0/2112149533 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55c9e9fd6400 msgr2=0x55c9e9160100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:32.794 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.785+0000 7f0b4703a280 1 --2- 172.21.15.90:0/2112149533 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55c9e9fd6400 0x55c9e9160100 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.794 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.785+0000 7f0b4703a280 1 -- 172.21.15.90:0/2112149533 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55c9ea047000 msgr2=0x55c9e9470000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:32.794 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.785+0000 7f0b4703a280 1 --2- 172.21.15.90:0/2112149533 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55c9ea047000 0x55c9e9470000 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x55c9e9fd2a20 tx=0x55c9e9fd8c00 comp rx=0 tx=0).stop 2024-03-24T01:30:32.794 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.785+0000 7f0b4703a280 1 -- 172.21.15.90:0/2112149533 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9e9fd7400 msgr2=0x55c9e9161180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:32.794 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.785+0000 7f0b4703a280 1 --2- 172.21.15.90:0/2112149533 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9e9fd7400 0x55c9e9161180 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x55c9e9fd3830 tx=0x55c9e9fd8de0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.794 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.785+0000 7f0b44a00640 1 -- 172.21.15.90:0/2112149533 reap_dead start 2024-03-24T01:30:32.794 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.785+0000 7f0b44a00640 1 -- 172.21.15.90:0/2112149533 reap_dead start 2024-03-24T01:30:32.794 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.785+0000 7f0b4703a280 1 -- 172.21.15.90:0/2112149533 shutdown_connections 2024-03-24T01:30:32.794 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.785+0000 7f0b4703a280 1 -- 172.21.15.90:0/2112149533 >> 172.21.15.90:0/2112149533 conn(0x55c9e9edbc00 msgr2=0x55c9e91ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:32.794 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.785+0000 7f0b4703a280 1 -- 172.21.15.90:0/2112149533 shutdown_connections 2024-03-24T01:30:32.794 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.785+0000 7f0b4703a280 1 -- 172.21.15.90:0/2112149533 wait complete. 2024-03-24T01:30:32.794 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.786+0000 7f0b4703a280 1 -- 172.21.15.90:0/1622405940 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9e9edb000 msgr2=0x55c9e915eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:32.795 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.786+0000 7f0b4703a280 1 --2- 172.21.15.90:0/1622405940 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9e9edb000 0x55c9e915eb00 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x55c9e9f18a20 tx=0x55c9e9eccf60 comp rx=0 tx=0).stop 2024-03-24T01:30:32.795 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.786+0000 7f0b4703a280 1 -- 172.21.15.90:0/1622405940 shutdown_connections 2024-03-24T01:30:32.795 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.786+0000 7f0b4703a280 1 --2- 172.21.15.90:0/1622405940 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9e9edb000 0x55c9e915eb00 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.795 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.786+0000 7f0b4703a280 1 --2- 172.21.15.90:0/1622405940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9e9edb400 0x55c9e915f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.795 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.786+0000 7f0b4703a280 1 --2- 172.21.15.90:0/1622405940 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c9e9eda400 0x55c9e915f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.795 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.786+0000 7f0b4703a280 1 -- 172.21.15.90:0/1622405940 >> 172.21.15.90:0/1622405940 conn(0x55c9e90f5000 msgr2=0x55c9e91fe800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:32.795 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.786+0000 7f0b4703a280 1 -- 172.21.15.90:0/1622405940 shutdown_connections 2024-03-24T01:30:32.795 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.786+0000 7f0b4703a280 1 -- 172.21.15.90:0/1622405940 wait complete. 2024-03-24T01:30:32.796 INFO:tasks.workunit.client.0.smithi090.stderr:+ HEADER_STATE='{ 2024-03-24T01:30:32.796 INFO:tasks.workunit.client.0.smithi090.stderr: "magic": "ceph fs volume v011", 2024-03-24T01:30:32.796 INFO:tasks.workunit.client.0.smithi090.stderr: "write_pos": 4200185, 2024-03-24T01:30:32.796 INFO:tasks.workunit.client.0.smithi090.stderr: "expire_pos": 4194304, 2024-03-24T01:30:32.796 INFO:tasks.workunit.client.0.smithi090.stderr: "trimmed_pos": 4194304, 2024-03-24T01:30:32.796 INFO:tasks.workunit.client.0.smithi090.stderr: "stream_format": 1, 2024-03-24T01:30:32.796 INFO:tasks.workunit.client.0.smithi090.stderr: "layout": { 2024-03-24T01:30:32.796 INFO:tasks.workunit.client.0.smithi090.stderr: "stripe_unit": 4194304, 2024-03-24T01:30:32.796 INFO:tasks.workunit.client.0.smithi090.stderr: "stripe_count": 1, 2024-03-24T01:30:32.796 INFO:tasks.workunit.client.0.smithi090.stderr: "object_size": 4194304, 2024-03-24T01:30:32.796 INFO:tasks.workunit.client.0.smithi090.stderr: "pool_id": 7, 2024-03-24T01:30:32.796 INFO:tasks.workunit.client.0.smithi090.stderr: "pool_ns": "" 2024-03-24T01:30:32.796 INFO:tasks.workunit.client.0.smithi090.stderr: } 2024-03-24T01:30:32.797 INFO:tasks.workunit.client.0.smithi090.stderr:}' 2024-03-24T01:30:32.797 INFO:tasks.workunit.client.0.smithi090.stderr:++ cephfs-journal-tool --rank=cephfs:0 event get list 2024-03-24T01:30:32.837 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.836+0000 7fdfef6cb280 1 -- start start 2024-03-24T01:30:32.838 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.836+0000 7fdfef6cb280 1 Processor -- start 2024-03-24T01:30:32.840 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.837+0000 7fdfef6cb280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556f381f2c00 0x556f3825c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.840 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.837+0000 7fdfef6cb280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556f381f3800 0x556f3825cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.840 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.837+0000 7fdfef6cb280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f381f3c00 0x556f3825d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.840 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.837+0000 7fdfef6cb280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f38fda180 con 0x556f381f3800 2024-03-24T01:30:32.840 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.837+0000 7fdfef6cb280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f38fda300 con 0x556f381f3c00 2024-03-24T01:30:32.840 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.837+0000 7fdfef6cb280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x556f38fda480 con 0x556f381f2c00 2024-03-24T01:30:32.840 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.837+0000 7fdfece00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556f381f3800 0x556f3825cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.840 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.837+0000 7fdfece00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556f381f3800 0x556f3825cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36028/0 (socket says 172.21.15.90:36028) 2024-03-24T01:30:32.840 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.837+0000 7fdfece00640 1 -- 172.21.15.90:0/4228428489 learned_addr learned my addr 172.21.15.90:0/4228428489 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:32.840 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.838+0000 7fdfece00640 1 -- 172.21.15.90:0/4228428489 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556f381f2c00 msgr2=0x556f3825c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:32.841 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.838+0000 7fdfece00640 1 --2- 172.21.15.90:0/4228428489 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556f381f2c00 0x556f3825c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.841 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.838+0000 7fdfece00640 1 -- 172.21.15.90:0/4228428489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f381f3c00 msgr2=0x556f3825d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:32.841 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.838+0000 7fdfece00640 1 --2- 172.21.15.90:0/4228428489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f381f3c00 0x556f3825d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.841 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.838+0000 7fdfece00640 1 -- 172.21.15.90:0/4228428489 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556f38226780 con 0x556f381f3800 2024-03-24T01:30:32.841 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.838+0000 7fdfece00640 1 --2- 172.21.15.90:0/4228428489 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556f381f3800 0x556f3825cb00 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x556f39017440 tx=0x556f38fc8de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7b99600d36b4d0f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:32.841 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.838+0000 7fdfea600640 1 -- 172.21.15.90:0/4228428489 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x556f38218680 con 0x556f381f3800 2024-03-24T01:30:32.841 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.838+0000 7fdfea600640 1 -- 172.21.15.90:0/4228428489 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x556f3821e700 con 0x556f381f3800 2024-03-24T01:30:32.841 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.838+0000 7fdfea600640 1 -- 172.21.15.90:0/4228428489 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x556f38218b60 con 0x556f381f3800 2024-03-24T01:30:32.841 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.838+0000 7fdfef6cb280 1 -- 172.21.15.90:0/4228428489 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556f382274a0 con 0x556f381f3800 2024-03-24T01:30:32.841 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.838+0000 7fdfef6cb280 1 -- 172.21.15.90:0/4228428489 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x556f38226000 con 0x556f381f3800 2024-03-24T01:30:32.841 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.839+0000 7fdfea600640 1 -- 172.21.15.90:0/4228428489 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x556f390b8600 con 0x556f381f3800 2024-03-24T01:30:32.841 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.839+0000 7fdfea600640 1 -- 172.21.15.90:0/4228428489 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 36) v1 ==== 3310+0+0 (secure 0 0 0) 0x556f3824d180 con 0x556f381f3800 2024-03-24T01:30:32.844 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.840+0000 7fdfef6cb280 1 Processor -- start 2024-03-24T01:30:32.844 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.840+0000 7fdfef6cb280 1 -- start start 2024-03-24T01:30:32.844 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.840+0000 7fdfef6cb280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556f390bb400 0x556f3825d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.844 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.840+0000 7fdfef6cb280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f390bb800 0x556f3825db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.844 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.840+0000 7fdfef6cb280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556f390bbc00 0x556f3825e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.844 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.841+0000 7fdfef6cb280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f38fda600 con 0x556f390bbc00 2024-03-24T01:30:32.844 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.841+0000 7fdfef6cb280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f38fda780 con 0x556f390bb800 2024-03-24T01:30:32.844 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.841+0000 7fdfef6cb280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x556f38fda900 con 0x556f390bb400 2024-03-24T01:30:32.845 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.841+0000 7fdfece00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556f390bbc00 0x556f3825e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.845 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.841+0000 7fdfece00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556f390bbc00 0x556f3825e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36038/0 (socket says 172.21.15.90:36038) 2024-03-24T01:30:32.845 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.841+0000 7fdfece00640 1 -- 172.21.15.90:0/3693279665 learned_addr learned my addr 172.21.15.90:0/3693279665 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:32.845 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.841+0000 7fdfee200640 1 --2- 172.21.15.90:0/3693279665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556f390bb400 0x556f3825d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.845 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.841+0000 7fdfed800640 1 --2- 172.21.15.90:0/3693279665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f390bb800 0x556f3825db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.845 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.841+0000 7fdfece00640 1 -- 172.21.15.90:0/3693279665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556f390bb400 msgr2=0x556f3825d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:32.845 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.841+0000 7fdfece00640 1 --2- 172.21.15.90:0/3693279665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556f390bb400 0x556f3825d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.845 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.841+0000 7fdfece00640 1 -- 172.21.15.90:0/3693279665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f390bb800 msgr2=0x556f3825db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:32.845 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.841+0000 7fdfece00640 1 --2- 172.21.15.90:0/3693279665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f390bb800 0x556f3825db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.845 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.841+0000 7fdfece00640 1 -- 172.21.15.90:0/3693279665 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556f382263c0 con 0x556f390bbc00 2024-03-24T01:30:32.845 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.841+0000 7fdfee200640 1 --2- 172.21.15.90:0/3693279665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556f390bb400 0x556f3825d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:30:32.845 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.842+0000 7fdfece00640 1 --2- 172.21.15.90:0/3693279665 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556f390bbc00 0x556f3825e100 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x556f390d6930 tx=0x556f38fc8120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a39c39ba3ccc23e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:32.845 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.842+0000 7fdfe7e00640 1 -- 172.21.15.90:0/3693279665 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x556f38218820 con 0x556f390bbc00 2024-03-24T01:30:32.845 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.842+0000 7fdfe7e00640 1 -- 172.21.15.90:0/3693279665 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x556f3821efc0 con 0x556f390bbc00 2024-03-24T01:30:32.845 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.842+0000 7fdfe7e00640 1 -- 172.21.15.90:0/3693279665 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x556f38218680 con 0x556f390bbc00 2024-03-24T01:30:32.846 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.842+0000 7fdfef6cb280 1 -- 172.21.15.90:0/3693279665 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556f390bbc00 msgr2=0x556f3825e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:32.846 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.842+0000 7fdfef6cb280 1 --2- 172.21.15.90:0/3693279665 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556f390bbc00 0x556f3825e100 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x556f390d6930 tx=0x556f38fc8120 comp rx=0 tx=0).stop 2024-03-24T01:30:32.846 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.842+0000 7fdfef6cb280 1 -- 172.21.15.90:0/3693279665 shutdown_connections 2024-03-24T01:30:32.846 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.842+0000 7fdfef6cb280 1 --2- 172.21.15.90:0/3693279665 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556f390bbc00 0x556f3825e100 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.846 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.842+0000 7fdfef6cb280 1 --2- 172.21.15.90:0/3693279665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f390bb800 0x556f3825db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.846 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.842+0000 7fdfef6cb280 1 --2- 172.21.15.90:0/3693279665 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556f390bb400 0x556f3825d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.846 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.842+0000 7fdfef6cb280 1 -- 172.21.15.90:0/3693279665 >> 172.21.15.90:0/3693279665 conn(0x556f390bb000 msgr2=0x556f382fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:32.846 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.842+0000 7fdfef6cb280 1 -- 172.21.15.90:0/3693279665 shutdown_connections 2024-03-24T01:30:32.846 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.842+0000 7fdfef6cb280 1 -- 172.21.15.90:0/3693279665 wait complete. 2024-03-24T01:30:32.846 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.843+0000 7fdfef6cb280 1 Processor -- start 2024-03-24T01:30:32.849 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.843+0000 7fdfef6cb280 1 -- start start 2024-03-24T01:30:32.849 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.844+0000 7fdfef6cb280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556f390bb400 0x556f3825f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.849 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.844+0000 7fdfef6cb280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556f390bb800 0x556f3825ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.849 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.844+0000 7fdfef6cb280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f390bbc00 0x556f3825e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.849 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.844+0000 7fdfef6cb280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f38fdbc80 con 0x556f390bb800 2024-03-24T01:30:32.849 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.844+0000 7fdfef6cb280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f38fdaa80 con 0x556f390bbc00 2024-03-24T01:30:32.849 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.844+0000 7fdfef6cb280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x556f38fdbb00 con 0x556f390bb400 2024-03-24T01:30:32.849 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.844+0000 7fdfed800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556f390bb800 0x556f3825ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.849 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.844+0000 7fdfed800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556f390bb800 0x556f3825ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36050/0 (socket says 172.21.15.90:36050) 2024-03-24T01:30:32.849 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.844+0000 7fdfed800640 1 -- 172.21.15.90:0/1921159379 learned_addr learned my addr 172.21.15.90:0/1921159379 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:32.850 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.844+0000 7fdfed800640 1 -- 172.21.15.90:0/1921159379 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556f390bb400 msgr2=0x556f3825f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:32.850 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.844+0000 7fdfed800640 1 --2- 172.21.15.90:0/1921159379 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556f390bb400 0x556f3825f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.850 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.844+0000 7fdfed800640 1 -- 172.21.15.90:0/1921159379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f390bbc00 msgr2=0x556f3825e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:32.850 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.844+0000 7fdfed800640 1 --2- 172.21.15.90:0/1921159379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f390bbc00 0x556f3825e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.850 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.844+0000 7fdfed800640 1 -- 172.21.15.90:0/1921159379 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556f38226780 con 0x556f390bb800 2024-03-24T01:30:32.850 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.845+0000 7fdfed800640 1 --2- 172.21.15.90:0/1921159379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556f390bb800 0x556f3825ec00 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x556f390d75f0 tx=0x556f390db260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:32.850 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.845+0000 7fdfe6000640 1 -- 172.21.15.90:0/1921159379 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x556f38218820 con 0x556f390bb800 2024-03-24T01:30:32.850 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.845+0000 7fdfe6000640 1 -- 172.21.15.90:0/1921159379 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x556f3821ec40 con 0x556f390bb800 2024-03-24T01:30:32.850 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.845+0000 7fdfe6000640 1 -- 172.21.15.90:0/1921159379 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x556f38218680 con 0x556f390bb800 2024-03-24T01:30:32.850 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.845+0000 7fdfef6cb280 1 -- 172.21.15.90:0/1921159379 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x556f382274a0 con 0x556f390bb800 2024-03-24T01:30:32.850 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.845+0000 7fdfef6cb280 1 -- 172.21.15.90:0/1921159379 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556f38226960 con 0x556f390bb800 2024-03-24T01:30:32.851 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.847+0000 7fdfe6000640 1 -- 172.21.15.90:0/1921159379 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x556f39132c00 con 0x556f390bb800 2024-03-24T01:30:32.851 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.847+0000 7fdfe6000640 1 --2- 172.21.15.90:0/1921159379 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x556f3856c000 0x556f3825f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.851 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.847+0000 7fdfe6000640 1 -- 172.21.15.90:0/1921159379 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x556f3913aa00 con 0x556f390bb800 2024-03-24T01:30:32.851 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.847+0000 7fdfef6cb280 1 --2- 172.21.15.90:0/1921159379 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x556f3856c400 0x556f3825e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.851 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.847+0000 7fdfef6cb280 1 -- 172.21.15.90:0/1921159379 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:1 7.f 7:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x556f3856c800 con 0x556f3856c400 2024-03-24T01:30:32.851 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.848+0000 7fdfece00640 1 --2- 172.21.15.90:0/1921159379 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x556f3856c400 0x556f3825e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.854 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.848+0000 7fdfee200640 1 --2- 172.21.15.90:0/1921159379 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x556f3856c000 0x556f3825f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.854 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.848+0000 7fdfece00640 1 --2- 172.21.15.90:0/1921159379 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x556f3856c400 0x556f3825e100 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:32.854 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.849+0000 7fdfece00640 1 -- 172.21.15.90:0/1921159379 <== osd.0 v2:172.21.15.90:6800/307045587 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x556f38f76000 con 0x556f3856c400 2024-03-24T01:30:32.854 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.849+0000 7fdfef6cb280 1 --2- 172.21.15.90:0/1921159379 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x556f3856cc00 0x556f3825db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.854 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.849+0000 7fdfef6cb280 1 -- 172.21.15.90:0/1921159379 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:2 7.14 7:292cf221:::200.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x556f3856d000 con 0x556f3856cc00 2024-03-24T01:30:32.854 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.849+0000 7fdfee200640 1 --2- 172.21.15.90:0/1921159379 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x556f3856cc00 0x556f3825db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.854 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.849+0000 7fdfee200640 1 --2- 172.21.15.90:0/1921159379 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x556f3856c000 0x556f3825f700 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x556f38225a70 tx=0x556f390dad20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:32.855 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.849+0000 7fdfee200640 1 --2- 172.21.15.90:0/1921159379 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x556f3856cc00 0x556f3825db80 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:32.855 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.850+0000 7fdfee200640 1 -- 172.21.15.90:0/1921159379 <== osd.3 v2:172.21.15.90:6808/2999931983 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv12 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x556f38f76500 con 0x556f3856cc00 2024-03-24T01:30:32.855 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.850+0000 7fdfef6cb280 1 -- 172.21.15.90:0/1921159379 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:3 7.14 7:2e2fa760:::200.00000001:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x556f3856d400 con 0x556f3856cc00 2024-03-24T01:30:32.855 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.850+0000 7fdfee200640 1 -- 172.21.15.90:0/1921159379 <== osd.3 v2:172.21.15.90:6808/2999931983 2 ==== osd_op_reply(3 200.00000001 [read 0~35827 out=35827b] v0'0 uv23 ondisk = 0) v8 ==== 156+0+35827 (crc 0 0 0) 0x556f38f76500 con 0x556f3856cc00 2024-03-24T01:30:32.855 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.851+0000 7fdfef6cb280 1 -- 172.21.15.90:0/1921159379 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:4 7.1b 7:ddb4e4d7:::200.00000002:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x556f3856c800 con 0x556f3856c400 2024-03-24T01:30:32.855 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.851+0000 7fdfece00640 1 -- 172.21.15.90:0/1921159379 <== osd.0 v2:172.21.15.90:6800/307045587 2 ==== osd_op_reply(4 200.00000002 [read 0~2147483647] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x556f38f76000 con 0x556f3856c400 2024-03-24T01:30:32.855 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.851+0000 7fdfef6cb280 1 -- 172.21.15.90:0/1921159379 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x556f3856c400 msgr2=0x556f3825e100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:32.855 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.851+0000 7fdfef6cb280 1 --2- 172.21.15.90:0/1921159379 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x556f3856c400 0x556f3825e100 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.855 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.851+0000 7fdfef6cb280 1 -- 172.21.15.90:0/1921159379 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x556f3856cc00 msgr2=0x556f3825db80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:32.855 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.851+0000 7fdfef6cb280 1 --2- 172.21.15.90:0/1921159379 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x556f3856cc00 0x556f3825db80 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.855 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.851+0000 7fdfef6cb280 1 -- 172.21.15.90:0/1921159379 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x556f3856c000 msgr2=0x556f3825f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:32.855 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.851+0000 7fdfef6cb280 1 --2- 172.21.15.90:0/1921159379 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x556f3856c000 0x556f3825f700 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x556f38225a70 tx=0x556f390dad20 comp rx=0 tx=0).stop 2024-03-24T01:30:32.855 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.851+0000 7fdfef6cb280 1 -- 172.21.15.90:0/1921159379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556f390bb800 msgr2=0x556f3825ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:32.855 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.851+0000 7fdfef6cb280 1 --2- 172.21.15.90:0/1921159379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556f390bb800 0x556f3825ec00 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x556f390d75f0 tx=0x556f390db260 comp rx=0 tx=0).stop 2024-03-24T01:30:32.856 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.851+0000 7fdfed800640 1 -- 172.21.15.90:0/1921159379 reap_dead start 2024-03-24T01:30:32.856 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.851+0000 7fdfed800640 1 -- 172.21.15.90:0/1921159379 reap_dead start 2024-03-24T01:30:32.856 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.851+0000 7fdfef6cb280 1 -- 172.21.15.90:0/1921159379 shutdown_connections 2024-03-24T01:30:32.856 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.851+0000 7fdfef6cb280 1 -- 172.21.15.90:0/1921159379 >> 172.21.15.90:0/1921159379 conn(0x556f390bb000 msgr2=0x556f382fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:32.856 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.852+0000 7fdfef6cb280 1 -- 172.21.15.90:0/1921159379 shutdown_connections 2024-03-24T01:30:32.856 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.852+0000 7fdfef6cb280 1 -- 172.21.15.90:0/1921159379 wait complete. 2024-03-24T01:30:32.856 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.852+0000 7fdfef6cb280 1 -- 172.21.15.90:0/4228428489 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556f381f3800 msgr2=0x556f3825cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:32.856 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.852+0000 7fdfef6cb280 1 --2- 172.21.15.90:0/4228428489 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556f381f3800 0x556f3825cb00 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x556f39017440 tx=0x556f38fc8de0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.856 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.852+0000 7fdfef6cb280 1 -- 172.21.15.90:0/4228428489 shutdown_connections 2024-03-24T01:30:32.856 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.852+0000 7fdfef6cb280 1 --2- 172.21.15.90:0/4228428489 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556f381f3800 0x556f3825cb00 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.856 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.852+0000 7fdfef6cb280 1 --2- 172.21.15.90:0/4228428489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f381f3c00 0x556f3825d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.856 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.852+0000 7fdfef6cb280 1 --2- 172.21.15.90:0/4228428489 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x556f381f2c00 0x556f3825c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.856 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.852+0000 7fdfef6cb280 1 -- 172.21.15.90:0/4228428489 >> 172.21.15.90:0/4228428489 conn(0x556f381f3000 msgr2=0x556f382fc800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:32.856 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.852+0000 7fdfef6cb280 1 -- 172.21.15.90:0/4228428489 shutdown_connections 2024-03-24T01:30:32.856 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.852+0000 7fdfef6cb280 1 -- 172.21.15.90:0/4228428489 wait complete. 2024-03-24T01:30:32.858 INFO:tasks.workunit.client.0.smithi090.stderr:+ EVENT_LIST='0.000000 0x400000 LID: () 2024-03-24T01:30:32.858 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:27:14.548044+0000 0x400030 SUBTREEMAP: () 2024-03-24T01:30:32.858 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:27:27.768607+0000 0x4003ac SESSION: () 2024-03-24T01:30:32.858 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:27:28.411755+0000 0x400565 UPDATE: (setattr) 2024-03-24T01:30:32.858 INFO:tasks.workunit.client.0.smithi090.stderr: / 2024-03-24T01:30:32.858 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:27:28.576042+0000 0x400894 UPDATE: (mkdir) 2024-03-24T01:30:32.858 INFO:tasks.workunit.client.0.smithi090.stderr: client.0 2024-03-24T01:30:32.858 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:27:33.769077+0000 0x401074 UPDATE: (cap update) 2024-03-24T01:30:32.858 INFO:tasks.workunit.client.0.smithi090.stderr: client.0 2024-03-24T01:30:32.858 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:53.486798+0000 0x4016f9 UPDATE: (mkdir) 2024-03-24T01:30:32.859 INFO:tasks.workunit.client.0.smithi090.stderr: client.0/tmp 2024-03-24T01:30:32.859 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:53.536138+0000 0x402217 UPDATE: (mkdir) 2024-03-24T01:30:32.859 INFO:tasks.workunit.client.0.smithi090.stderr: tmp/foo 2024-03-24T01:30:32.859 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:53.549102+0000 0x402ae7 UPDATE: (openc) 2024-03-24T01:30:32.859 INFO:tasks.workunit.client.0.smithi090.stderr: tmp/bar 2024-03-24T01:30:32.859 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:56.112617+0000 0x403289 UPDATE: (unlink_local) 2024-03-24T01:30:32.859 INFO:tasks.workunit.client.0.smithi090.stderr: stray2/10000000002 2024-03-24T01:30:32.859 INFO:tasks.workunit.client.0.smithi090.stderr: client.0/tmp/foo 2024-03-24T01:30:32.859 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:56.118494+0000 0x40448c UPDATE: (cap update) 2024-03-24T01:30:32.859 INFO:tasks.workunit.client.0.smithi090.stderr: tmp/bar 2024-03-24T01:30:32.859 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:56.127543+0000 0x404c0d UPDATE: (unlink_local) 2024-03-24T01:30:32.859 INFO:tasks.workunit.client.0.smithi090.stderr: stray2/10000000003 2024-03-24T01:30:32.859 INFO:tasks.workunit.client.0.smithi090.stderr: tmp/bar 2024-03-24T01:30:32.859 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:56.142108+0000 0x405870 UPDATE: (unlink_local) 2024-03-24T01:30:32.859 INFO:tasks.workunit.client.0.smithi090.stderr: stray2/10000000001 2024-03-24T01:30:32.860 INFO:tasks.workunit.client.0.smithi090.stderr: client.0/tmp 2024-03-24T01:30:32.860 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:57.246240+0000 0x4064d8 UPDATE: (unlink_local) 2024-03-24T01:30:32.860 INFO:tasks.workunit.client.0.smithi090.stderr: client.0 2024-03-24T01:30:32.860 INFO:tasks.workunit.client.0.smithi090.stderr: stray2/10000000000 2024-03-24T01:30:32.860 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:58.733720+0000 0x407037 UPDATE: (purge_stray) 2024-03-24T01:30:32.860 INFO:tasks.workunit.client.0.smithi090.stderr: 10000000002 2024-03-24T01:30:32.860 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:58.733860+0000 0x40728e UPDATE: (purge_stray) 2024-03-24T01:30:32.860 INFO:tasks.workunit.client.0.smithi090.stderr: 10000000003 2024-03-24T01:30:32.860 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:59.116268+0000 0x4074e5 UPDATE: (scatter_writebehind) 2024-03-24T01:30:32.860 INFO:tasks.workunit.client.0.smithi090.stderr: stray2/10000000000 2024-03-24T01:30:32.860 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:59.119801+0000 0x407dbe UPDATE: (scatter_writebehind) 2024-03-24T01:30:32.860 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:59.732514+0000 0x408230 UPDATE: (purge_stray) 2024-03-24T01:30:32.860 INFO:tasks.workunit.client.0.smithi090.stderr: 10000000001 2024-03-24T01:30:32.860 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:29:01.965453+0000 0x408487 SESSION: () 2024-03-24T01:30:32.860 INFO:tasks.workunit.client.0.smithi090.stderr:0.000000 0x408534 PURGED: () 2024-03-24T01:30:32.860 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:29:04.115764+0000 0x408580 UPDATE: (scatter_writebehind) 2024-03-24T01:30:32.861 INFO:tasks.workunit.client.0.smithi090.stderr: stray2' 2024-03-24T01:30:32.861 INFO:tasks.workunit.client.0.smithi090.stderr:+ cephfs-journal-tool --rank=cephfs:0 journal export /tmp/journal.bin 2024-03-24T01:30:32.904 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.902+0000 7ff9d7411280 1 -- start start 2024-03-24T01:30:32.904 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.902+0000 7ff9d7411280 1 Processor -- start 2024-03-24T01:30:32.905 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.903+0000 7ff9d7411280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e417080000 0x55e416323080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.905 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.903+0000 7ff9d7411280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e4162b9c00 0x55e416322b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.905 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.903+0000 7ff9d7411280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e417080800 0x55e416323b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.905 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.903+0000 7ff9d7411280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e41717a000 con 0x55e417080800 2024-03-24T01:30:32.905 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.903+0000 7ff9d7411280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e41717a180 con 0x55e4162b9c00 2024-03-24T01:30:32.905 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.903+0000 7ff9d7411280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e41717a300 con 0x55e417080000 2024-03-24T01:30:32.905 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.903+0000 7ff9d5200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e417080800 0x55e416323b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.905 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.904+0000 7ff9d5200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e417080800 0x55e416323b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36066/0 (socket says 172.21.15.90:36066) 2024-03-24T01:30:32.905 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.904+0000 7ff9d5200640 1 -- 172.21.15.90:0/500395457 learned_addr learned my addr 172.21.15.90:0/500395457 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:32.907 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.904+0000 7ff9d5200640 1 -- 172.21.15.90:0/500395457 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e417080000 msgr2=0x55e416323080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:32.907 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.904+0000 7ff9d5200640 1 --2- 172.21.15.90:0/500395457 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e417080000 0x55e416323080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.907 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.904+0000 7ff9d5200640 1 -- 172.21.15.90:0/500395457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e4162b9c00 msgr2=0x55e416322b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:32.907 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.904+0000 7ff9d5200640 1 --2- 172.21.15.90:0/500395457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e4162b9c00 0x55e416322b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.907 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.904+0000 7ff9d5200640 1 -- 172.21.15.90:0/500395457 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e4162ec780 con 0x55e417080800 2024-03-24T01:30:32.907 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.904+0000 7ff9d5200640 1 --2- 172.21.15.90:0/500395457 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e417080800 0x55e416323b80 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x55e4170d7290 tx=0x55e4163a0de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a2c4d76bc74b316 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:32.907 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.904+0000 7ff9d1600640 1 -- 172.21.15.90:0/500395457 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e4162ded00 con 0x55e417080800 2024-03-24T01:30:32.907 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.904+0000 7ff9d1600640 1 -- 172.21.15.90:0/500395457 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e4162e4e00 con 0x55e417080800 2024-03-24T01:30:32.907 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.904+0000 7ff9d1600640 1 -- 172.21.15.90:0/500395457 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e4162de680 con 0x55e417080800 2024-03-24T01:30:32.908 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.904+0000 7ff9d7411280 1 -- 172.21.15.90:0/500395457 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e4162ec000 con 0x55e417080800 2024-03-24T01:30:32.908 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.904+0000 7ff9d7411280 1 -- 172.21.15.90:0/500395457 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x55e4162eda40 con 0x55e417080800 2024-03-24T01:30:32.908 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.905+0000 7ff9d1600640 1 -- 172.21.15.90:0/500395457 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x55e417184400 con 0x55e417080800 2024-03-24T01:30:32.908 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.905+0000 7ff9d1600640 1 -- 172.21.15.90:0/500395457 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 36) v1 ==== 3310+0+0 (secure 0 0 0) 0x55e416313180 con 0x55e417080800 2024-03-24T01:30:32.908 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.906+0000 7ff9d7411280 1 Processor -- start 2024-03-24T01:30:32.908 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.906+0000 7ff9d7411280 1 -- start start 2024-03-24T01:30:32.910 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.907+0000 7ff9d7411280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e417186c00 0x55e416323600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.910 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.907+0000 7ff9d7411280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e417187000 0x55e416322580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.910 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.907+0000 7ff9d7411280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e417187400 0x55e416324100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.910 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.907+0000 7ff9d7411280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e41717a480 con 0x55e417187400 2024-03-24T01:30:32.910 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.907+0000 7ff9d7411280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e41717a600 con 0x55e417186c00 2024-03-24T01:30:32.910 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.907+0000 7ff9d7411280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e41717a780 con 0x55e417187000 2024-03-24T01:30:32.910 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.907+0000 7ff9d3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e417187400 0x55e416324100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.910 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.907+0000 7ff9d3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e417187400 0x55e416324100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36076/0 (socket says 172.21.15.90:36076) 2024-03-24T01:30:32.910 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.907+0000 7ff9d3e00640 1 -- 172.21.15.90:0/2003476377 learned_addr learned my addr 172.21.15.90:0/2003476377 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:32.910 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.907+0000 7ff9d3e00640 1 -- 172.21.15.90:0/2003476377 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e417187000 msgr2=0x55e416322580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:32.911 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.907+0000 7ff9d3e00640 1 --2- 172.21.15.90:0/2003476377 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e417187000 0x55e416322580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.911 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.907+0000 7ff9d3e00640 1 -- 172.21.15.90:0/2003476377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e417186c00 msgr2=0x55e416323600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:32.911 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.907+0000 7ff9d3e00640 1 --2- 172.21.15.90:0/2003476377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e417186c00 0x55e416323600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.911 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.907+0000 7ff9d3e00640 1 -- 172.21.15.90:0/2003476377 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e4162ed860 con 0x55e417187400 2024-03-24T01:30:32.911 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.908+0000 7ff9d3e00640 1 --2- 172.21.15.90:0/2003476377 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e417187400 0x55e416324100 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x55e4171f4120 tx=0x55e4171a2d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c46faf93ef40058 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:32.911 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.908+0000 7ff9cee00640 1 -- 172.21.15.90:0/2003476377 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e4162ded00 con 0x55e417187400 2024-03-24T01:30:32.911 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.908+0000 7ff9cee00640 1 -- 172.21.15.90:0/2003476377 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e4162e4700 con 0x55e417187400 2024-03-24T01:30:32.911 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.908+0000 7ff9cee00640 1 -- 172.21.15.90:0/2003476377 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e4162de680 con 0x55e417187400 2024-03-24T01:30:32.911 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.908+0000 7ff9d7411280 1 -- 172.21.15.90:0/2003476377 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e417187400 msgr2=0x55e416324100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:32.911 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.908+0000 7ff9d7411280 1 --2- 172.21.15.90:0/2003476377 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e417187400 0x55e416324100 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x55e4171f4120 tx=0x55e4171a2d20 comp rx=0 tx=0).stop 2024-03-24T01:30:32.911 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.908+0000 7ff9d7411280 1 -- 172.21.15.90:0/2003476377 shutdown_connections 2024-03-24T01:30:32.911 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.908+0000 7ff9d7411280 1 --2- 172.21.15.90:0/2003476377 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e417187400 0x55e416324100 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.911 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.908+0000 7ff9d7411280 1 --2- 172.21.15.90:0/2003476377 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e417187000 0x55e416322580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.911 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.908+0000 7ff9d7411280 1 --2- 172.21.15.90:0/2003476377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e417186c00 0x55e416323600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.912 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.908+0000 7ff9d7411280 1 -- 172.21.15.90:0/2003476377 >> 172.21.15.90:0/2003476377 conn(0x55e417080400 msgr2=0x55e417037000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:32.912 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.908+0000 7ff9d7411280 1 -- 172.21.15.90:0/2003476377 shutdown_connections 2024-03-24T01:30:32.912 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.909+0000 7ff9d7411280 1 -- 172.21.15.90:0/2003476377 wait complete. 2024-03-24T01:30:32.914 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.909+0000 7ff9d7411280 1 Processor -- start 2024-03-24T01:30:32.914 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.909+0000 7ff9d7411280 1 -- start start 2024-03-24T01:30:32.914 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.910+0000 7ff9d7411280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e417186c00 0x55e416325180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.914 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.910+0000 7ff9d7411280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e417187000 0x55e416324c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.914 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.910+0000 7ff9d7411280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e417187400 0x55e416324680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.914 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.910+0000 7ff9d7411280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e41717ad80 con 0x55e417186c00 2024-03-24T01:30:32.914 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.910+0000 7ff9d7411280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e41717af00 con 0x55e417187000 2024-03-24T01:30:32.914 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.910+0000 7ff9d7411280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e41717b080 con 0x55e417187400 2024-03-24T01:30:32.914 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.910+0000 7ff9d3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e417186c00 0x55e416325180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.914 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.910+0000 7ff9d3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e417186c00 0x55e416325180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36092/0 (socket says 172.21.15.90:36092) 2024-03-24T01:30:32.914 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.910+0000 7ff9d3e00640 1 -- 172.21.15.90:0/1577094894 learned_addr learned my addr 172.21.15.90:0/1577094894 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:32.915 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.911+0000 7ff9d3e00640 1 -- 172.21.15.90:0/1577094894 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e417187400 msgr2=0x55e416324680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:32.915 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.911+0000 7ff9d3e00640 1 --2- 172.21.15.90:0/1577094894 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e417187400 0x55e416324680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.915 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.911+0000 7ff9d3e00640 1 -- 172.21.15.90:0/1577094894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e417187000 msgr2=0x55e416324c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:32.915 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.911+0000 7ff9d3e00640 1 --2- 172.21.15.90:0/1577094894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e417187000 0x55e416324c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.915 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.911+0000 7ff9d3e00640 1 -- 172.21.15.90:0/1577094894 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e4162ed4a0 con 0x55e417186c00 2024-03-24T01:30:32.915 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.911+0000 7ff9d3e00640 1 --2- 172.21.15.90:0/1577094894 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e417186c00 0x55e416325180 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x55e4171f53e0 tx=0x55e4171a2c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:32.915 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.911+0000 7ff9cd000640 1 -- 172.21.15.90:0/1577094894 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e4162de820 con 0x55e417186c00 2024-03-24T01:30:32.915 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.911+0000 7ff9cd000640 1 -- 172.21.15.90:0/1577094894 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e4162e41c0 con 0x55e417186c00 2024-03-24T01:30:32.915 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.911+0000 7ff9cd000640 1 -- 172.21.15.90:0/1577094894 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e4162ded00 con 0x55e417186c00 2024-03-24T01:30:32.915 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.911+0000 7ff9d7411280 1 -- 172.21.15.90:0/1577094894 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55e4162ec960 con 0x55e417186c00 2024-03-24T01:30:32.915 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.911+0000 7ff9d7411280 1 -- 172.21.15.90:0/1577094894 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e4162ec780 con 0x55e417186c00 2024-03-24T01:30:32.915 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.913+0000 7ff9cd000640 1 -- 172.21.15.90:0/1577094894 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x55e417206c00 con 0x55e417186c00 2024-03-24T01:30:32.917 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.913+0000 7ff9cd000640 1 --2- 172.21.15.90:0/1577094894 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55e4171ff000 0x55e416325700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.917 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.913+0000 7ff9cd000640 1 -- 172.21.15.90:0/1577094894 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x55e417204a00 con 0x55e417186c00 2024-03-24T01:30:32.917 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.914+0000 7ff9d7411280 1 --2- 172.21.15.90:0/1577094894 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55e417187800 0x55e416324100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.917 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.914+0000 7ff9d7411280 1 -- 172.21.15.90:0/1577094894 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:1 7.f 7:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e417187c00 con 0x55e417187800 2024-03-24T01:30:32.918 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.914+0000 7ff9d3e00640 1 --2- 172.21.15.90:0/1577094894 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55e417187800 0x55e416324100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.918 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.914+0000 7ff9d4800640 1 --2- 172.21.15.90:0/1577094894 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55e4171ff000 0x55e416325700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.918 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.914+0000 7ff9d3e00640 1 --2- 172.21.15.90:0/1577094894 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55e417187800 0x55e416324100 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:32.918 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.915+0000 7ff9d3e00640 1 -- 172.21.15.90:0/1577094894 <== osd.0 v2:172.21.15.90:6800/307045587 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x55e41703c000 con 0x55e417187800 2024-03-24T01:30:32.918 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.915+0000 7ff9d7411280 1 --2- 172.21.15.90:0/1577094894 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55e4171ff400 0x55e416322580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.918 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.915+0000 7ff9d7411280 1 -- 172.21.15.90:0/1577094894 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:2 7.14 7:292cf221:::200.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e4171ff800 con 0x55e4171ff400 2024-03-24T01:30:32.918 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.915+0000 7ff9d4800640 1 --2- 172.21.15.90:0/1577094894 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55e4171ff000 0x55e416325700 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x55e41639aab0 tx=0x55e41663c120 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:32.918 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.915+0000 7ff9d5200640 1 --2- 172.21.15.90:0/1577094894 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55e4171ff400 0x55e416322580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.918 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.916+0000 7ff9d5200640 1 --2- 172.21.15.90:0/1577094894 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55e4171ff400 0x55e416322580 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:32.918 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.916+0000 7ff9d5200640 1 -- 172.21.15.90:0/1577094894 <== osd.3 v2:172.21.15.90:6808/2999931983 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv12 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x55e41703c500 con 0x55e4171ff400 2024-03-24T01:30:32.918 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.916+0000 7ff9d7411280 1 -- 172.21.15.90:0/1577094894 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:3 7.14 7:2e2fa760:::200.00000001:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e4171ffc00 con 0x55e4171ff400 2024-03-24T01:30:32.918 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.916+0000 7ff9d5200640 1 -- 172.21.15.90:0/1577094894 <== osd.3 v2:172.21.15.90:6808/2999931983 2 ==== osd_op_reply(3 200.00000001 [read 0~35827 out=35827b] v0'0 uv23 ondisk = 0) v8 ==== 156+0+35827 (crc 0 0 0) 0x55e41703c500 con 0x55e4171ff400 2024-03-24T01:30:32.919 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.917+0000 7ff9d7411280 1 -- 172.21.15.90:0/1577094894 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:4 7.1b 7:ddb4e4d7:::200.00000002:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e416658000 con 0x55e417187800 2024-03-24T01:30:32.919 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.917+0000 7ff9d3e00640 1 -- 172.21.15.90:0/1577094894 <== osd.0 v2:172.21.15.90:6800/307045587 2 ==== osd_op_reply(4 200.00000002 [read 0~2147483647] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x55e41703c000 con 0x55e417187800 2024-03-24T01:30:32.919 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.918+0000 7ff9d7411280 1 -- start start 2024-03-24T01:30:32.919 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.918+0000 7ff9d7411280 1 Processor -- start 2024-03-24T01:30:32.921 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.919+0000 7ff9d7411280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e416658800 0x55e416323600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.921 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.919+0000 7ff9d7411280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e416658c00 0x55e4166a8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.921 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.919+0000 7ff9d7411280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e416659000 0x55e4166a8580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.921 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.919+0000 7ff9d7411280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e41717a780 con 0x55e416658800 2024-03-24T01:30:32.921 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.919+0000 7ff9d7411280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e41717a900 con 0x55e416659000 2024-03-24T01:30:32.921 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.919+0000 7ff9d7411280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e41717aa80 con 0x55e416658c00 2024-03-24T01:30:32.921 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.919+0000 7ff9d3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e416658800 0x55e416323600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.921 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.919+0000 7ff9d3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e416658800 0x55e416323600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36104/0 (socket says 172.21.15.90:36104) 2024-03-24T01:30:32.921 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.919+0000 7ff9d3e00640 1 -- 172.21.15.90:0/3190184623 learned_addr learned my addr 172.21.15.90:0/3190184623 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:32.921 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.919+0000 7ff9d3e00640 1 -- 172.21.15.90:0/3190184623 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e416658c00 msgr2=0x55e4166a8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:32.922 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.919+0000 7ff9d3e00640 1 --2- 172.21.15.90:0/3190184623 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e416658c00 0x55e4166a8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.922 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.919+0000 7ff9d3e00640 1 -- 172.21.15.90:0/3190184623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e416659000 msgr2=0x55e4166a8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:32.923 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.919+0000 7ff9d3e00640 1 --2- 172.21.15.90:0/3190184623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e416659000 0x55e4166a8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.923 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.919+0000 7ff9d3e00640 1 -- 172.21.15.90:0/3190184623 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e4162ec000 con 0x55e416658800 2024-03-24T01:30:32.923 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.920+0000 7ff9d3e00640 1 --2- 172.21.15.90:0/3190184623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e416658800 0x55e416323600 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x55e4165f7800 tx=0x55e4171a36e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4847c1371c9e4fa3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:32.927 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.920+0000 7ff9c9400640 1 -- 172.21.15.90:0/3190184623 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e4165beb60 con 0x55e416658800 2024-03-24T01:30:32.927 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.920+0000 7ff9c9400640 1 -- 172.21.15.90:0/3190184623 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e4162e48c0 con 0x55e416658800 2024-03-24T01:30:32.927 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.920+0000 7ff9c9400640 1 -- 172.21.15.90:0/3190184623 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e4165bed00 con 0x55e416658800 2024-03-24T01:30:32.927 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.920+0000 7ff9d7411280 1 -- 172.21.15.90:0/3190184623 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e4162ec960 con 0x55e416658800 2024-03-24T01:30:32.927 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.920+0000 7ff9d7411280 1 -- 172.21.15.90:0/3190184623 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x55e4162ec780 con 0x55e416658800 2024-03-24T01:30:32.927 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.920+0000 7ff9c9400640 1 -- 172.21.15.90:0/3190184623 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x55e4166ac600 con 0x55e416658800 2024-03-24T01:30:32.927 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.920+0000 7ff9c9400640 1 -- 172.21.15.90:0/3190184623 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 36) v1 ==== 3310+0+0 (secure 0 0 0) 0x55e416313500 con 0x55e416658800 2024-03-24T01:30:32.927 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.921+0000 7ff9d7411280 1 --2- 172.21.15.90:0/3190184623 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55e416659400 0x55e4166a8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.928 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.921+0000 7ff9d7411280 1 -- 172.21.15.90:0/3190184623 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:1 7.f 7:f1f69726:::400.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e416659800 con 0x55e416659400 2024-03-24T01:30:32.928 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.921+0000 7ff9d5200640 1 --2- 172.21.15.90:0/3190184623 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55e416659400 0x55e4166a8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.928 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.921+0000 7ff9d5200640 1 --2- 172.21.15.90:0/3190184623 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55e416659400 0x55e4166a8b00 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=faabf07371c26f26 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:32.928 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.922+0000 7ff9d5200640 1 -- 172.21.15.90:0/3190184623 <== osd.0 v2:172.21.15.90:6800/307045587 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x55e41703c500 con 0x55e416659400 2024-03-24T01:30:32.928 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.922+0000 7ff9d7411280 1 --2- 172.21.15.90:0/3190184623 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55e4166bf400 0x55e4166a9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.928 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.922+0000 7ff9d7411280 1 -- 172.21.15.90:0/3190184623 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:2 7.14 7:292cf221:::200.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e4166bf800 con 0x55e4166bf400 2024-03-24T01:30:32.928 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.922+0000 7ff9d4800640 1 --2- 172.21.15.90:0/3190184623 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55e4166bf400 0x55e4166a9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.928 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.922+0000 7ff9d4800640 1 --2- 172.21.15.90:0/3190184623 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55e4166bf400 0x55e4166a9080 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=97b94c4aad9cd5db server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:32.928 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.923+0000 7ff9d4800640 1 -- 172.21.15.90:0/3190184623 <== osd.3 v2:172.21.15.90:6808/2999931983 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv12 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x55e4166d0000 con 0x55e4166bf400 2024-03-24T01:30:32.928 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.923+0000 7ff9c7600640 1 -- 172.21.15.90:0/3190184623 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:3 7.14 7:2e2fa760:::200.00000001:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e4166d2000 con 0x55e4166bf400 2024-03-24T01:30:32.928 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.923+0000 7ff9c7600640 1 -- 172.21.15.90:0/3190184623 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:4 7.1b 7:ddb4e4d7:::200.00000002:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e4166d2400 con 0x55e416659400 2024-03-24T01:30:32.928 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.923+0000 7ff9d5200640 1 -- 172.21.15.90:0/3190184623 <== osd.0 v2:172.21.15.90:6800/307045587 2 ==== osd_op_reply(4 200.00000002 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x55e41703c500 con 0x55e416659400 2024-03-24T01:30:32.928 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.923+0000 7ff9d4800640 1 -- 172.21.15.90:0/3190184623 <== osd.3 v2:172.21.15.90:6808/2999931983 2 ==== osd_op_reply(3 200.00000001 [stat out=16b] v0'0 uv23 ondisk = 0) v8 ==== 156+0+16 (crc 0 0 0) 0x55e4166d0000 con 0x55e4166bf400 2024-03-24T01:30:32.929 INFO:tasks.workunit.client.0.smithi090.stdout:journal is 4194304~35827 2024-03-24T01:30:32.929 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.923+0000 7ff9d7411280 1 -- 172.21.15.90:0/3190184623 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:5 7.14 7:2e2fa760:::200.00000001:head [read 0~35827] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e4166bfc00 con 0x55e4166bf400 2024-03-24T01:30:32.929 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.924+0000 7ff9d4800640 1 -- 172.21.15.90:0/3190184623 <== osd.3 v2:172.21.15.90:6808/2999931983 3 ==== osd_op_reply(5 200.00000001 [read 0~35827 out=35827b] v0'0 uv23 ondisk = 0) v8 ==== 156+0+35827 (crc 0 0 0) 0x55e4166d0000 con 0x55e4166bf400 2024-03-24T01:30:32.929 INFO:tasks.workunit.client.0.smithi090.stdout:wrote 35827 bytes at offset 4194304 to /tmp/journal.bin 2024-03-24T01:30:32.929 INFO:tasks.workunit.client.0.smithi090.stdout:NOTE: this is a _sparse_ file; you can 2024-03-24T01:30:32.930 INFO:tasks.workunit.client.0.smithi090.stdout: $ tar cSzf /tmp/journal.bin.tgz /tmp/journal.bin 2024-03-24T01:30:32.930 INFO:tasks.workunit.client.0.smithi090.stdout: to efficiently compress it while preserving sparseness. 2024-03-24T01:30:32.932 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.924+0000 7ff9d7411280 1 -- 172.21.15.90:0/3190184623 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55e416659400 msgr2=0x55e4166a8b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:32.932 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.924+0000 7ff9d7411280 1 --2- 172.21.15.90:0/3190184623 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55e416659400 0x55e4166a8b00 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.932 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.924+0000 7ff9d7411280 1 -- 172.21.15.90:0/3190184623 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55e4166bf400 msgr2=0x55e4166a9080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:32.933 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.924+0000 7ff9d7411280 1 --2- 172.21.15.90:0/3190184623 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55e4166bf400 0x55e4166a9080 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.933 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.924+0000 7ff9d7411280 1 -- 172.21.15.90:0/3190184623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e416658800 msgr2=0x55e416323600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:32.933 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.924+0000 7ff9d7411280 1 --2- 172.21.15.90:0/3190184623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e416658800 0x55e416323600 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x55e4165f7800 tx=0x55e4171a36e0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.933 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.924+0000 7ff9d4800640 1 -- 172.21.15.90:0/3190184623 reap_dead start 2024-03-24T01:30:32.933 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.924+0000 7ff9d7411280 1 -- 172.21.15.90:0/3190184623 shutdown_connections 2024-03-24T01:30:32.933 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.924+0000 7ff9d7411280 1 -- 172.21.15.90:0/3190184623 >> 172.21.15.90:0/3190184623 conn(0x55e416658400 msgr2=0x55e417036800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:32.933 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.924+0000 7ff9d7411280 1 -- 172.21.15.90:0/3190184623 shutdown_connections 2024-03-24T01:30:32.933 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.924+0000 7ff9d7411280 1 -- 172.21.15.90:0/3190184623 wait complete. 2024-03-24T01:30:32.933 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.924+0000 7ff9d7411280 1 -- 172.21.15.90:0/1577094894 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55e417187800 msgr2=0x55e416324100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:32.934 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.924+0000 7ff9d7411280 1 --2- 172.21.15.90:0/1577094894 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55e417187800 0x55e416324100 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.934 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.924+0000 7ff9d7411280 1 -- 172.21.15.90:0/1577094894 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55e4171ff400 msgr2=0x55e416322580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:32.934 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.924+0000 7ff9d7411280 1 --2- 172.21.15.90:0/1577094894 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55e4171ff400 0x55e416322580 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.934 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.924+0000 7ff9d7411280 1 -- 172.21.15.90:0/1577094894 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55e4171ff000 msgr2=0x55e416325700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:32.934 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.924+0000 7ff9d7411280 1 --2- 172.21.15.90:0/1577094894 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55e4171ff000 0x55e416325700 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x55e41639aab0 tx=0x55e41663c120 comp rx=0 tx=0).stop 2024-03-24T01:30:32.934 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.924+0000 7ff9d7411280 1 -- 172.21.15.90:0/1577094894 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e417186c00 msgr2=0x55e416325180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:32.935 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.924+0000 7ff9d7411280 1 --2- 172.21.15.90:0/1577094894 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e417186c00 0x55e416325180 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x55e4171f53e0 tx=0x55e4171a2c00 comp rx=0 tx=0).stop 2024-03-24T01:30:32.935 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.925+0000 7ff9d4800640 1 -- 172.21.15.90:0/1577094894 reap_dead start 2024-03-24T01:30:32.935 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.925+0000 7ff9d4800640 1 -- 172.21.15.90:0/1577094894 reap_dead start 2024-03-24T01:30:32.935 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.925+0000 7ff9d7411280 1 -- 172.21.15.90:0/1577094894 shutdown_connections 2024-03-24T01:30:32.935 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.925+0000 7ff9d7411280 1 -- 172.21.15.90:0/1577094894 >> 172.21.15.90:0/1577094894 conn(0x55e417080400 msgr2=0x55e417037000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:32.936 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.925+0000 7ff9d7411280 1 -- 172.21.15.90:0/1577094894 shutdown_connections 2024-03-24T01:30:32.936 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.925+0000 7ff9d7411280 1 -- 172.21.15.90:0/1577094894 wait complete. 2024-03-24T01:30:32.936 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.925+0000 7ff9d7411280 1 -- 172.21.15.90:0/500395457 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e417080800 msgr2=0x55e416323b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:32.936 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.925+0000 7ff9d7411280 1 --2- 172.21.15.90:0/500395457 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e417080800 0x55e416323b80 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x55e4170d7290 tx=0x55e4163a0de0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.936 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.925+0000 7ff9d7411280 1 -- 172.21.15.90:0/500395457 shutdown_connections 2024-03-24T01:30:32.937 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.925+0000 7ff9d7411280 1 --2- 172.21.15.90:0/500395457 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e417080800 0x55e416323b80 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.937 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.925+0000 7ff9d7411280 1 --2- 172.21.15.90:0/500395457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e4162b9c00 0x55e416322b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.937 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.925+0000 7ff9d7411280 1 --2- 172.21.15.90:0/500395457 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e417080000 0x55e416323080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.937 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.925+0000 7ff9d7411280 1 -- 172.21.15.90:0/500395457 >> 172.21.15.90:0/500395457 conn(0x55e4162b9000 msgr2=0x55e417038000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:32.937 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.925+0000 7ff9d7411280 1 -- 172.21.15.90:0/500395457 shutdown_connections 2024-03-24T01:30:32.937 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.925+0000 7ff9d7411280 1 -- 172.21.15.90:0/500395457 wait complete. 2024-03-24T01:30:32.937 INFO:tasks.workunit.client.0.smithi090.stderr:+ cephfs-journal-tool --rank=cephfs:0 journal import /tmp/journal.bin 2024-03-24T01:30:32.974 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.972+0000 7f818a8fa280 1 -- start start 2024-03-24T01:30:32.974 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.973+0000 7f818a8fa280 1 Processor -- start 2024-03-24T01:30:32.976 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.973+0000 7f818a8fa280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e94649ec00 0x55e946509b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.976 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.973+0000 7f818a8fa280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94649f800 0x55e946509600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.976 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.973+0000 7f818a8fa280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e94649fc00 0x55e946509080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.976 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.973+0000 7f818a8fa280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e94727a180 con 0x55e94649f800 2024-03-24T01:30:32.976 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.973+0000 7f818a8fa280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e946500480 con 0x55e94649fc00 2024-03-24T01:30:32.976 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.973+0000 7f818a8fa280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e946500900 con 0x55e94649ec00 2024-03-24T01:30:32.976 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.974+0000 7f8188000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94649f800 0x55e946509600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.976 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.974+0000 7f8188000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94649f800 0x55e946509600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36124/0 (socket says 172.21.15.90:36124) 2024-03-24T01:30:32.977 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.974+0000 7f8188000640 1 -- 172.21.15.90:0/1127488165 learned_addr learned my addr 172.21.15.90:0/1127488165 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:32.977 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.974+0000 7f8188000640 1 -- 172.21.15.90:0/1127488165 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e94649ec00 msgr2=0x55e946509b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:32.977 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.974+0000 7f8188000640 1 --2- 172.21.15.90:0/1127488165 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e94649ec00 0x55e946509b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.977 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.974+0000 7f8188000640 1 -- 172.21.15.90:0/1127488165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e94649fc00 msgr2=0x55e946509080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:32.977 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.974+0000 7f8188000640 1 --2- 172.21.15.90:0/1127488165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e94649fc00 0x55e946509080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.977 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.974+0000 7f8188000640 1 -- 172.21.15.90:0/1127488165 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e9464d2780 con 0x55e94649f800 2024-03-24T01:30:32.977 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.974+0000 7f8188000640 1 --2- 172.21.15.90:0/1127488165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94649f800 0x55e946509600 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x55e9472b5350 tx=0x55e94726ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=912cdb15eb15f6d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:32.977 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.974+0000 7f8185800640 1 -- 172.21.15.90:0/1127488165 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e9464c4680 con 0x55e94649f800 2024-03-24T01:30:32.977 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.975+0000 7f8185800640 1 -- 172.21.15.90:0/1127488165 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e9464ca700 con 0x55e94649f800 2024-03-24T01:30:32.977 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.975+0000 7f8185800640 1 -- 172.21.15.90:0/1127488165 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e9464c4b60 con 0x55e94649f800 2024-03-24T01:30:32.977 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.975+0000 7f818a8fa280 1 -- 172.21.15.90:0/1127488165 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e9464d34a0 con 0x55e94649f800 2024-03-24T01:30:32.977 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.975+0000 7f818a8fa280 1 -- 172.21.15.90:0/1127488165 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x55e9464d2000 con 0x55e94649f800 2024-03-24T01:30:32.977 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.975+0000 7f8185800640 1 -- 172.21.15.90:0/1127488165 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x55e94735c600 con 0x55e94649f800 2024-03-24T01:30:32.977 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.975+0000 7f8185800640 1 -- 172.21.15.90:0/1127488165 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 36) v1 ==== 3310+0+0 (secure 0 0 0) 0x55e9464f9180 con 0x55e94649f800 2024-03-24T01:30:32.980 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.976+0000 7f818a8fa280 1 Processor -- start 2024-03-24T01:30:32.980 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.976+0000 7f818a8fa280 1 -- start start 2024-03-24T01:30:32.980 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.977+0000 7f818a8fa280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e94735f400 0x55e946508b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.980 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.977+0000 7f818a8fa280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e94735f800 0x55e946508580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.981 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.977+0000 7f818a8fa280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94735fc00 0x55e94650a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.981 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.977+0000 7f818a8fa280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e946500180 con 0x55e94735fc00 2024-03-24T01:30:32.981 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.977+0000 7f818a8fa280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e94727b500 con 0x55e94735f400 2024-03-24T01:30:32.981 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.977+0000 7f818a8fa280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e94727b380 con 0x55e94735f800 2024-03-24T01:30:32.981 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.977+0000 7f8188000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94735fc00 0x55e94650a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.981 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.977+0000 7f8188000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94735fc00 0x55e94650a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36140/0 (socket says 172.21.15.90:36140) 2024-03-24T01:30:32.981 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.977+0000 7f8188000640 1 -- 172.21.15.90:0/4121226891 learned_addr learned my addr 172.21.15.90:0/4121226891 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:32.981 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.977+0000 7f8188000640 1 -- 172.21.15.90:0/4121226891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e94735f800 msgr2=0x55e946508580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:32.981 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.977+0000 7f8188000640 1 --2- 172.21.15.90:0/4121226891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e94735f800 0x55e946508580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.981 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.977+0000 7f8188000640 1 -- 172.21.15.90:0/4121226891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e94735f400 msgr2=0x55e946508b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:32.981 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.977+0000 7f8189400640 1 --2- 172.21.15.90:0/4121226891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e94735f400 0x55e946508b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.981 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.977+0000 7f8188000640 1 --2- 172.21.15.90:0/4121226891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e94735f400 0x55e946508b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.981 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.977+0000 7f8188000640 1 -- 172.21.15.90:0/4121226891 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e9464d23c0 con 0x55e94735fc00 2024-03-24T01:30:32.981 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.978+0000 7f8188000640 1 --2- 172.21.15.90:0/4121226891 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94735fc00 0x55e94650a100 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x55e94737a6c0 tx=0x55e94726cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a017915e563f0bf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:32.982 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.978+0000 7f8183000640 1 -- 172.21.15.90:0/4121226891 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e9464c4820 con 0x55e94735fc00 2024-03-24T01:30:32.982 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.978+0000 7f8183000640 1 -- 172.21.15.90:0/4121226891 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e9464cafc0 con 0x55e94735fc00 2024-03-24T01:30:32.982 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.978+0000 7f8183000640 1 -- 172.21.15.90:0/4121226891 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e9464c4680 con 0x55e94735fc00 2024-03-24T01:30:32.982 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.978+0000 7f818a8fa280 1 -- 172.21.15.90:0/4121226891 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94735fc00 msgr2=0x55e94650a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:32.982 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.978+0000 7f818a8fa280 1 --2- 172.21.15.90:0/4121226891 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94735fc00 0x55e94650a100 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x55e94737a6c0 tx=0x55e94726cd80 comp rx=0 tx=0).stop 2024-03-24T01:30:32.982 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.978+0000 7f818a8fa280 1 -- 172.21.15.90:0/4121226891 shutdown_connections 2024-03-24T01:30:32.982 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.978+0000 7f818a8fa280 1 --2- 172.21.15.90:0/4121226891 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94735fc00 0x55e94650a100 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.982 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.978+0000 7f818a8fa280 1 --2- 172.21.15.90:0/4121226891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e94735f800 0x55e946508580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.982 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.978+0000 7f818a8fa280 1 --2- 172.21.15.90:0/4121226891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e94735f400 0x55e946508b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.982 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.978+0000 7f818a8fa280 1 -- 172.21.15.90:0/4121226891 >> 172.21.15.90:0/4121226891 conn(0x55e94735f000 msgr2=0x55e94721f000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:32.982 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.979+0000 7f818a8fa280 1 -- 172.21.15.90:0/4121226891 shutdown_connections 2024-03-24T01:30:32.982 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.979+0000 7f818a8fa280 1 -- 172.21.15.90:0/4121226891 wait complete. 2024-03-24T01:30:32.982 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.979+0000 7f818a8fa280 1 Processor -- start 2024-03-24T01:30:32.982 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.979+0000 7f818a8fa280 1 -- start start 2024-03-24T01:30:32.984 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.980+0000 7f818a8fa280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e94735f400 0x55e946508b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.984 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.980+0000 7f818a8fa280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e94735f800 0x55e946508580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.984 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.980+0000 7f818a8fa280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94735fc00 0x55e94650a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.985 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.980+0000 7f818a8fa280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e94727a780 con 0x55e94735fc00 2024-03-24T01:30:32.985 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.980+0000 7f818a8fa280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e94727a900 con 0x55e94735f400 2024-03-24T01:30:32.985 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.980+0000 7f818a8fa280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e94727a600 con 0x55e94735f800 2024-03-24T01:30:32.985 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.980+0000 7f8188000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94735fc00 0x55e94650a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.985 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.980+0000 7f8188000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94735fc00 0x55e94650a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36146/0 (socket says 172.21.15.90:36146) 2024-03-24T01:30:32.985 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.980+0000 7f8188000640 1 -- 172.21.15.90:0/3320168891 learned_addr learned my addr 172.21.15.90:0/3320168891 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:32.985 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.981+0000 7f8189400640 1 --2- 172.21.15.90:0/3320168891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e94735f400 0x55e946508b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.985 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.981+0000 7f8188000640 1 -- 172.21.15.90:0/3320168891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e94735f800 msgr2=0x55e946508580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:32.985 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.981+0000 7f8188000640 1 --2- 172.21.15.90:0/3320168891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e94735f800 0x55e946508580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.985 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.981+0000 7f8188000640 1 -- 172.21.15.90:0/3320168891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e94735f400 msgr2=0x55e946508b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:32.985 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.981+0000 7f8188000640 1 --2- 172.21.15.90:0/3320168891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e94735f400 0x55e946508b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.985 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.981+0000 7f8188000640 1 -- 172.21.15.90:0/3320168891 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e9464d2780 con 0x55e94735fc00 2024-03-24T01:30:32.985 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.981+0000 7f8188000640 1 --2- 172.21.15.90:0/3320168891 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94735fc00 0x55e94650a100 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x55e94737b380 tx=0x55e94737cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:32.986 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.981+0000 7f8181200640 1 -- 172.21.15.90:0/3320168891 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e9464c4b60 con 0x55e94735fc00 2024-03-24T01:30:32.986 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.981+0000 7f8181200640 1 -- 172.21.15.90:0/3320168891 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e9464cae00 con 0x55e94735fc00 2024-03-24T01:30:32.986 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.981+0000 7f818a8fa280 1 -- 172.21.15.90:0/3320168891 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55e9464d34a0 con 0x55e94735fc00 2024-03-24T01:30:32.986 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.981+0000 7f8181200640 1 -- 172.21.15.90:0/3320168891 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e9464c44e0 con 0x55e94735fc00 2024-03-24T01:30:32.986 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.981+0000 7f818a8fa280 1 -- 172.21.15.90:0/3320168891 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e9464d2960 con 0x55e94735fc00 2024-03-24T01:30:32.986 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.983+0000 7f8181200640 1 -- 172.21.15.90:0/3320168891 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x55e9473dac00 con 0x55e94735fc00 2024-03-24T01:30:32.986 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.983+0000 7f8181200640 1 --2- 172.21.15.90:0/3320168891 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55e9473d9400 0x55e94680e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.986 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.983+0000 7f8181200640 1 -- 172.21.15.90:0/3320168891 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x55e9473de600 con 0x55e94735fc00 2024-03-24T01:30:32.989 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.984+0000 7f8189400640 1 --2- 172.21.15.90:0/3320168891 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55e9473d9400 0x55e94680e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.989 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.984+0000 7f818a8fa280 1 -- start start 2024-03-24T01:30:32.989 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.984+0000 7f818a8fa280 1 Processor -- start 2024-03-24T01:30:32.989 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.985+0000 7f8189400640 1 --2- 172.21.15.90:0/3320168891 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55e9473d9400 0x55e94680e000 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x55e9464d1ef0 tx=0x55e94737cd20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:32.989 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.985+0000 7f818a8fa280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9473d9c00 0x55e94650a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.989 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.985+0000 7f818a8fa280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94728e000 0x55e94650ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.989 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.985+0000 7f818a8fa280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e94728e400 0x55e94650b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.989 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.985+0000 7f818a8fa280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e94727b500 con 0x55e94728e000 2024-03-24T01:30:32.989 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.985+0000 7f818a8fa280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e94727a600 con 0x55e9473d9c00 2024-03-24T01:30:32.989 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.985+0000 7f818a8fa280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e94727b800 con 0x55e94728e400 2024-03-24T01:30:32.990 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.986+0000 7f8188a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94728e000 0x55e94650ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.990 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.986+0000 7f8188a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94728e000 0x55e94650ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36160/0 (socket says 172.21.15.90:36160) 2024-03-24T01:30:32.990 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.986+0000 7f8188a00640 1 -- 172.21.15.90:0/2297652756 learned_addr learned my addr 172.21.15.90:0/2297652756 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:32.990 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.986+0000 7f8189400640 1 --2- 172.21.15.90:0/2297652756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9473d9c00 0x55e94650a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.990 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.986+0000 7f8188000640 1 --2- 172.21.15.90:0/2297652756 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e94728e400 0x55e94650b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.990 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.986+0000 7f8188a00640 1 -- 172.21.15.90:0/2297652756 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e94728e400 msgr2=0x55e94650b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:32.990 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.986+0000 7f8188a00640 1 --2- 172.21.15.90:0/2297652756 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e94728e400 0x55e94650b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.990 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.986+0000 7f8188a00640 1 -- 172.21.15.90:0/2297652756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9473d9c00 msgr2=0x55e94650a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:32.990 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.986+0000 7f8188a00640 1 --2- 172.21.15.90:0/2297652756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9473d9c00 0x55e94650a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.990 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.986+0000 7f8188a00640 1 -- 172.21.15.90:0/2297652756 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e9464d3680 con 0x55e94728e000 2024-03-24T01:30:32.990 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.986+0000 7f8188a00640 1 --2- 172.21.15.90:0/2297652756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94728e000 0x55e94650ac00 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x55e9467bb890 tx=0x55e94681b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99cd17d0e87b3eb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:32.991 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.987+0000 7f817d600640 1 -- 172.21.15.90:0/2297652756 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e947404000 con 0x55e94728e000 2024-03-24T01:30:32.991 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.987+0000 7f817d600640 1 -- 172.21.15.90:0/2297652756 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e9464caa80 con 0x55e94728e000 2024-03-24T01:30:32.991 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.987+0000 7f817d600640 1 -- 172.21.15.90:0/2297652756 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e94740fa00 con 0x55e94728e000 2024-03-24T01:30:32.991 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.987+0000 7f818a8fa280 1 -- 172.21.15.90:0/2297652756 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e9464d2000 con 0x55e94728e000 2024-03-24T01:30:32.991 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.987+0000 7f818a8fa280 1 -- 172.21.15.90:0/2297652756 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x55e9464d23c0 con 0x55e94728e000 2024-03-24T01:30:32.991 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.987+0000 7f817d600640 1 -- 172.21.15.90:0/2297652756 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x55e946872600 con 0x55e94728e000 2024-03-24T01:30:32.991 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.987+0000 7f817d600640 1 -- 172.21.15.90:0/2297652756 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 36) v1 ==== 3310+0+0 (secure 0 0 0) 0x55e9464f9500 con 0x55e94728e000 2024-03-24T01:30:32.991 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.987+0000 7f818a8fa280 1 --2- 172.21.15.90:0/2297652756 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55e94728e800 0x55e94650b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.991 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.987+0000 7f818a8fa280 1 -- 172.21.15.90:0/2297652756 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:1 7.f 7:f1f69726:::400.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e94728ec00 con 0x55e94728e800 2024-03-24T01:30:32.991 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.987+0000 7f8189400640 1 --2- 172.21.15.90:0/2297652756 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55e94728e800 0x55e94650b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.991 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.988+0000 7f8189400640 1 --2- 172.21.15.90:0/2297652756 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55e94728e800 0x55e94650b700 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=c2a2d9301839b003 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:32.992 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.988+0000 7f8189400640 1 -- 172.21.15.90:0/2297652756 <== osd.0 v2:172.21.15.90:6800/307045587 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x55e94661e000 con 0x55e94728e800 2024-03-24T01:30:32.992 INFO:tasks.workunit.client.0.smithi090.stdout:undump /tmp/journal.bin 2024-03-24T01:30:32.993 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.988+0000 7f818a8fa280 1 --2- 172.21.15.90:0/2297652756 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55e94728f000 0x55e94680e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:32.993 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.988+0000 7f818a8fa280 1 -- 172.21.15.90:0/2297652756 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:2 7.14 7:292cf221:::200.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e94728f400 con 0x55e94728f000 2024-03-24T01:30:32.993 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.988+0000 7f8188000640 1 --2- 172.21.15.90:0/2297652756 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55e94728f000 0x55e94680e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:32.993 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.989+0000 7f8188000640 1 --2- 172.21.15.90:0/2297652756 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55e94728f000 0x55e94680e580 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=3600c2517fa89ac0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:32.993 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.989+0000 7f8188000640 1 -- 172.21.15.90:0/2297652756 <== osd.3 v2:172.21.15.90:6808/2999931983 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv12 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x55e94661e500 con 0x55e94728f000 2024-03-24T01:30:32.993 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.989+0000 7f817b800640 1 -- 172.21.15.90:0/2297652756 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:3 7.14 7:2e2fa760:::200.00000001:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e946875000 con 0x55e94728f000 2024-03-24T01:30:32.993 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.989+0000 7f817b800640 1 -- 172.21.15.90:0/2297652756 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:4 7.1b 7:ddb4e4d7:::200.00000002:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e946875400 con 0x55e94728e800 2024-03-24T01:30:32.993 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.990+0000 7f8188000640 1 -- 172.21.15.90:0/2297652756 <== osd.3 v2:172.21.15.90:6808/2999931983 2 ==== osd_op_reply(3 200.00000001 [stat out=16b] v0'0 uv23 ondisk = 0) v8 ==== 156+0+16 (crc 0 0 0) 0x55e94661e500 con 0x55e94728f000 2024-03-24T01:30:32.993 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.990+0000 7f8189400640 1 -- 172.21.15.90:0/2297652756 <== osd.0 v2:172.21.15.90:6800/307045587 2 ==== osd_op_reply(4 200.00000002 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x55e94661e000 con 0x55e94728e800 2024-03-24T01:30:32.994 INFO:tasks.workunit.client.0.smithi090.stdout:start 4194304 len 35827 write_pos 4200185 format 1 trimmed_pos 4194304 stripe_unit 4194304 stripe_count 1 object_size 4194304 2024-03-24T01:30:32.994 INFO:tasks.workunit.client.0.smithi090.stdout:writing header 200.00000000 2024-03-24T01:30:32.994 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.990+0000 7f818a8fa280 1 -- 172.21.15.90:0/2297652756 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:5 7.14 7:292cf221:::200.00000000:head [writefull 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e946875c00 con 0x55e94728f000 2024-03-24T01:30:32.994 INFO:tasks.workunit.client.0.smithi090.stdout:Purging 2 objects from 1 2024-03-24T01:30:32.995 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.991+0000 7f8188000640 1 -- 172.21.15.90:0/2297652756 <== osd.3 v2:172.21.15.90:6808/2999931983 3 ==== osd_op_reply(5 200.00000000 [writefull 0~90] v56'24 uv24 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55e94661e500 con 0x55e94728f000 2024-03-24T01:30:32.995 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.991+0000 7f818a8fa280 1 -- 172.21.15.90:0/2297652756 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:6 7.14 7:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e946894000 con 0x55e94728f000 2024-03-24T01:30:32.995 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.991+0000 7f818a8fa280 1 -- 172.21.15.90:0/2297652756 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:7 7.1b 7:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e946894400 con 0x55e94728e800 2024-03-24T01:30:32.995 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.993+0000 7f8189400640 1 -- 172.21.15.90:0/2297652756 <== osd.0 v2:172.21.15.90:6800/307045587 3 ==== osd_op_reply(7 200.00000002 [delete] v56'2 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x55e94661e000 con 0x55e94728e800 2024-03-24T01:30:32.995 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.994+0000 7f8188000640 1 -- 172.21.15.90:0/2297652756 <== osd.3 v2:172.21.15.90:6808/2999931983 4 ==== osd_op_reply(6 200.00000001 [delete] v56'25 uv25 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55e94661e500 con 0x55e94728f000 2024-03-24T01:30:32.996 INFO:tasks.workunit.client.0.smithi090.stdout: writing 4194304~35827 2024-03-24T01:30:32.996 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.994+0000 7f818a8fa280 1 -- 172.21.15.90:0/2297652756 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:8 7.14 7:2e2fa760:::200.00000001:head [write 0~35827 in=35827b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e946894800 con 0x55e94728f000 2024-03-24T01:30:32.996 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.995+0000 7f8188000640 1 -- 172.21.15.90:0/2297652756 <== osd.3 v2:172.21.15.90:6808/2999931983 5 ==== osd_op_reply(8 200.00000001 [write 0~35827] v56'26 uv26 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55e94661e500 con 0x55e94728f000 2024-03-24T01:30:32.997 INFO:tasks.workunit.client.0.smithi090.stdout:done. 2024-03-24T01:30:32.998 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.995+0000 7f818a8fa280 1 -- 172.21.15.90:0/2297652756 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55e94728e800 msgr2=0x55e94650b700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:32.998 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.995+0000 7f818a8fa280 1 --2- 172.21.15.90:0/2297652756 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55e94728e800 0x55e94650b700 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.999 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.995+0000 7f818a8fa280 1 -- 172.21.15.90:0/2297652756 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55e94728f000 msgr2=0x55e94680e580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:32.999 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.995+0000 7f818a8fa280 1 --2- 172.21.15.90:0/2297652756 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55e94728f000 0x55e94680e580 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.999 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.995+0000 7f818a8fa280 1 -- 172.21.15.90:0/2297652756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94728e000 msgr2=0x55e94650ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:32.999 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.995+0000 7f818a8fa280 1 --2- 172.21.15.90:0/2297652756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94728e000 0x55e94650ac00 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x55e9467bb890 tx=0x55e94681b8c0 comp rx=0 tx=0).stop 2024-03-24T01:30:32.999 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.995+0000 7f8188a00640 1 -- 172.21.15.90:0/2297652756 reap_dead start 2024-03-24T01:30:32.999 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.996+0000 7f818a8fa280 1 -- 172.21.15.90:0/2297652756 shutdown_connections 2024-03-24T01:30:32.999 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.996+0000 7f818a8fa280 1 -- 172.21.15.90:0/2297652756 >> 172.21.15.90:0/2297652756 conn(0x55e9473d9800 msgr2=0x55e94721f800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:32.999 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.996+0000 7f818a8fa280 1 -- 172.21.15.90:0/2297652756 shutdown_connections 2024-03-24T01:30:32.999 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.996+0000 7f818a8fa280 1 -- 172.21.15.90:0/2297652756 wait complete. 2024-03-24T01:30:32.999 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.996+0000 7f818a8fa280 1 -- 172.21.15.90:0/3320168891 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55e9473d9400 msgr2=0x55e94680e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:32.999 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.996+0000 7f818a8fa280 1 --2- 172.21.15.90:0/3320168891 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55e9473d9400 0x55e94680e000 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x55e9464d1ef0 tx=0x55e94737cd20 comp rx=0 tx=0).stop 2024-03-24T01:30:32.999 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.996+0000 7f818a8fa280 1 -- 172.21.15.90:0/3320168891 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94735fc00 msgr2=0x55e94650a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:32.999 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.996+0000 7f818a8fa280 1 --2- 172.21.15.90:0/3320168891 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94735fc00 0x55e94650a100 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x55e94737b380 tx=0x55e94737cf00 comp rx=0 tx=0).stop 2024-03-24T01:30:33.000 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.996+0000 7f818a8fa280 1 -- 172.21.15.90:0/3320168891 shutdown_connections 2024-03-24T01:30:33.000 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.996+0000 7f818a8fa280 1 --2- 172.21.15.90:0/3320168891 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55e9473d9400 0x55e94680e000 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.000 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.996+0000 7f818a8fa280 1 --2- 172.21.15.90:0/3320168891 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94735fc00 0x55e94650a100 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.000 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.996+0000 7f818a8fa280 1 --2- 172.21.15.90:0/3320168891 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e94735f800 0x55e946508580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.000 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.996+0000 7f818a8fa280 1 --2- 172.21.15.90:0/3320168891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e94735f400 0x55e946508b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.000 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.996+0000 7f818a8fa280 1 -- 172.21.15.90:0/3320168891 >> 172.21.15.90:0/3320168891 conn(0x55e94735f000 msgr2=0x55e94721f000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.000 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.996+0000 7f818a8fa280 1 -- 172.21.15.90:0/3320168891 shutdown_connections 2024-03-24T01:30:33.000 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.996+0000 7f818a8fa280 1 -- 172.21.15.90:0/3320168891 wait complete. 2024-03-24T01:30:33.000 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.997+0000 7f818a8fa280 1 -- 172.21.15.90:0/1127488165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94649f800 msgr2=0x55e946509600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.000 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.997+0000 7f818a8fa280 1 --2- 172.21.15.90:0/1127488165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94649f800 0x55e946509600 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x55e9472b5350 tx=0x55e94726ce40 comp rx=0 tx=0).stop 2024-03-24T01:30:33.000 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.997+0000 7f818a8fa280 1 -- 172.21.15.90:0/1127488165 shutdown_connections 2024-03-24T01:30:33.000 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.997+0000 7f818a8fa280 1 --2- 172.21.15.90:0/1127488165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e94649f800 0x55e946509600 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.000 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.997+0000 7f818a8fa280 1 --2- 172.21.15.90:0/1127488165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e94649fc00 0x55e946509080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.000 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.997+0000 7f818a8fa280 1 --2- 172.21.15.90:0/1127488165 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e94649ec00 0x55e946509b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.001 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.997+0000 7f818a8fa280 1 -- 172.21.15.90:0/1127488165 >> 172.21.15.90:0/1127488165 conn(0x55e94649f000 msgr2=0x55e94721e800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.001 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.997+0000 7f818a8fa280 1 -- 172.21.15.90:0/1127488165 shutdown_connections 2024-03-24T01:30:33.001 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:32.997+0000 7f818a8fa280 1 -- 172.21.15.90:0/1127488165 wait complete. 2024-03-24T01:30:33.003 INFO:tasks.workunit.client.0.smithi090.stderr:++ cephfs-journal-tool --rank=cephfs:0 header get 2024-03-24T01:30:33.046 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.045+0000 7fc60dc36280 1 -- start start 2024-03-24T01:30:33.046 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.045+0000 7fc60dc36280 1 Processor -- start 2024-03-24T01:30:33.049 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.046+0000 7fc60dc36280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e39c85ac00 0x55e39c8c4580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.050 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.046+0000 7fc60dc36280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e39c85b800 0x55e39c8c4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.050 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.046+0000 7fc60dc36280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e39c85bc00 0x55e39c8c5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.050 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.046+0000 7fc60dc36280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e39c958600 con 0x55e39c85b800 2024-03-24T01:30:33.050 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.046+0000 7fc60dc36280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e39c958480 con 0x55e39c85bc00 2024-03-24T01:30:33.051 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.046+0000 7fc60dc36280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e39c958300 con 0x55e39c85ac00 2024-03-24T01:30:33.051 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.046+0000 7fc60ac00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e39c85b800 0x55e39c8c4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.052 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.046+0000 7fc60ac00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e39c85b800 0x55e39c8c4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36180/0 (socket says 172.21.15.90:36180) 2024-03-24T01:30:33.053 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.046+0000 7fc60ac00640 1 -- 172.21.15.90:0/1432063662 learned_addr learned my addr 172.21.15.90:0/1432063662 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.053 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.047+0000 7fc60c000640 1 --2- 172.21.15.90:0/1432063662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e39c85bc00 0x55e39c8c5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.053 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.047+0000 7fc60ac00640 1 -- 172.21.15.90:0/1432063662 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e39c85ac00 msgr2=0x55e39c8c4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.053 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.047+0000 7fc60ac00640 1 --2- 172.21.15.90:0/1432063662 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e39c85ac00 0x55e39c8c4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.053 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.047+0000 7fc60ac00640 1 -- 172.21.15.90:0/1432063662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e39c85bc00 msgr2=0x55e39c8c5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.053 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.047+0000 7fc60ac00640 1 --2- 172.21.15.90:0/1432063662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e39c85bc00 0x55e39c8c5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.053 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.047+0000 7fc60ac00640 1 -- 172.21.15.90:0/1432063662 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e39c88e780 con 0x55e39c85b800 2024-03-24T01:30:33.053 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.047+0000 7fc60ac00640 1 --2- 172.21.15.90:0/1432063662 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e39c85b800 0x55e39c8c4b00 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x55e39d677380 tx=0x55e39d632e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b41886829bfa7ca server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.053 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.047+0000 7fc608400640 1 -- 172.21.15.90:0/1432063662 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e39c880680 con 0x55e39c85b800 2024-03-24T01:30:33.054 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.047+0000 7fc608400640 1 -- 172.21.15.90:0/1432063662 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e39c886700 con 0x55e39c85b800 2024-03-24T01:30:33.054 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.047+0000 7fc608400640 1 -- 172.21.15.90:0/1432063662 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e39c880b60 con 0x55e39c85b800 2024-03-24T01:30:33.054 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.047+0000 7fc60dc36280 1 -- 172.21.15.90:0/1432063662 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e39c88e3c0 con 0x55e39c85b800 2024-03-24T01:30:33.054 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.047+0000 7fc60dc36280 1 -- 172.21.15.90:0/1432063662 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x55e39c88f680 con 0x55e39c85b800 2024-03-24T01:30:33.054 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.048+0000 7fc608400640 1 -- 172.21.15.90:0/1432063662 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x55e39d720600 con 0x55e39c85b800 2024-03-24T01:30:33.054 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.048+0000 7fc608400640 1 -- 172.21.15.90:0/1432063662 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 36) v1 ==== 3310+0+0 (secure 0 0 0) 0x55e39c8b5180 con 0x55e39c85b800 2024-03-24T01:30:33.059 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.049+0000 7fc60dc36280 1 Processor -- start 2024-03-24T01:30:33.059 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.049+0000 7fc60dc36280 1 -- start start 2024-03-24T01:30:33.059 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.049+0000 7fc60dc36280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e39d723400 0x55e39c8c5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.059 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.049+0000 7fc60dc36280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e39d723800 0x55e39c8c5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.059 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.049+0000 7fc60dc36280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e39d723c00 0x55e39c8c6100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.059 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.049+0000 7fc60dc36280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e39c958180 con 0x55e39d723400 2024-03-24T01:30:33.059 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.049+0000 7fc60dc36280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e39c8bc900 con 0x55e39d723c00 2024-03-24T01:30:33.060 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.049+0000 7fc60dc36280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e39c8bc480 con 0x55e39d723800 2024-03-24T01:30:33.060 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.050+0000 7fc60c000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e39d723400 0x55e39c8c5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.060 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.050+0000 7fc60c000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e39d723400 0x55e39c8c5600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36182/0 (socket says 172.21.15.90:36182) 2024-03-24T01:30:33.060 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.050+0000 7fc60c000640 1 -- 172.21.15.90:0/2996232170 learned_addr learned my addr 172.21.15.90:0/2996232170 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.060 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.050+0000 7fc60c000640 1 -- 172.21.15.90:0/2996232170 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e39d723800 msgr2=0x55e39c8c5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.060 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.050+0000 7fc60c000640 1 --2- 172.21.15.90:0/2996232170 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e39d723800 0x55e39c8c5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.060 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.050+0000 7fc60c000640 1 -- 172.21.15.90:0/2996232170 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e39d723c00 msgr2=0x55e39c8c6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.060 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.050+0000 7fc60c000640 1 --2- 172.21.15.90:0/2996232170 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e39d723c00 0x55e39c8c6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.060 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.050+0000 7fc60c000640 1 -- 172.21.15.90:0/2996232170 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e39c88f4a0 con 0x55e39d723400 2024-03-24T01:30:33.060 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.050+0000 7fc60c000640 1 --2- 172.21.15.90:0/2996232170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e39d723400 0x55e39c8c5600 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x55e39d738840 tx=0x55e39d73acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32d80432bc980732 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.060 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.051+0000 7fc605c00640 1 -- 172.21.15.90:0/2996232170 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e39c880680 con 0x55e39d723400 2024-03-24T01:30:33.060 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.051+0000 7fc605c00640 1 -- 172.21.15.90:0/2996232170 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e39c886e00 con 0x55e39d723400 2024-03-24T01:30:33.060 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.051+0000 7fc605c00640 1 -- 172.21.15.90:0/2996232170 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e39c880b60 con 0x55e39d723400 2024-03-24T01:30:33.060 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.051+0000 7fc60dc36280 1 -- 172.21.15.90:0/2996232170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e39d723400 msgr2=0x55e39c8c5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.061 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.051+0000 7fc60dc36280 1 --2- 172.21.15.90:0/2996232170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e39d723400 0x55e39c8c5600 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x55e39d738840 tx=0x55e39d73acc0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.061 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.051+0000 7fc60dc36280 1 -- 172.21.15.90:0/2996232170 shutdown_connections 2024-03-24T01:30:33.061 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.051+0000 7fc60dc36280 1 --2- 172.21.15.90:0/2996232170 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e39d723c00 0x55e39c8c6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.061 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.051+0000 7fc60dc36280 1 --2- 172.21.15.90:0/2996232170 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e39d723800 0x55e39c8c5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.061 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.051+0000 7fc60dc36280 1 --2- 172.21.15.90:0/2996232170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e39d723400 0x55e39c8c5600 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.061 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.051+0000 7fc60dc36280 1 -- 172.21.15.90:0/2996232170 >> 172.21.15.90:0/2996232170 conn(0x55e39d723000 msgr2=0x55e39d5d8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.061 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.051+0000 7fc60dc36280 1 -- 172.21.15.90:0/2996232170 shutdown_connections 2024-03-24T01:30:33.061 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.051+0000 7fc60dc36280 1 -- 172.21.15.90:0/2996232170 wait complete. 2024-03-24T01:30:33.061 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.052+0000 7fc60dc36280 1 Processor -- start 2024-03-24T01:30:33.061 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.052+0000 7fc60dc36280 1 -- start start 2024-03-24T01:30:33.061 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.053+0000 7fc60dc36280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e39d723400 0x55e39c8c7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.061 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.053+0000 7fc60dc36280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e39d723800 0x55e39c8c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.062 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.053+0000 7fc60dc36280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e39d723c00 0x55e39c8c6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.062 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.053+0000 7fc60dc36280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e39c959800 con 0x55e39d723c00 2024-03-24T01:30:33.062 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.053+0000 7fc60dc36280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e39c958180 con 0x55e39d723400 2024-03-24T01:30:33.062 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.053+0000 7fc60dc36280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e39c958d80 con 0x55e39d723800 2024-03-24T01:30:33.062 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.053+0000 7fc60ac00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e39d723c00 0x55e39c8c6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.062 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.053+0000 7fc60ac00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e39d723c00 0x55e39c8c6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36184/0 (socket says 172.21.15.90:36184) 2024-03-24T01:30:33.062 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.053+0000 7fc60ac00640 1 -- 172.21.15.90:0/1969580965 learned_addr learned my addr 172.21.15.90:0/1969580965 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.062 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.053+0000 7fc60ac00640 1 -- 172.21.15.90:0/1969580965 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e39d723800 msgr2=0x55e39c8c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.062 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.053+0000 7fc60ac00640 1 --2- 172.21.15.90:0/1969580965 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e39d723800 0x55e39c8c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.062 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.053+0000 7fc60ac00640 1 -- 172.21.15.90:0/1969580965 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e39d723400 msgr2=0x55e39c8c7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.062 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.053+0000 7fc60c000640 1 --2- 172.21.15.90:0/1969580965 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e39d723400 0x55e39c8c7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.063 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.053+0000 7fc60ac00640 1 --2- 172.21.15.90:0/1969580965 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e39d723400 0x55e39c8c7180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.063 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.053+0000 7fc60ac00640 1 -- 172.21.15.90:0/1969580965 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e39c88e780 con 0x55e39d723c00 2024-03-24T01:30:33.063 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.054+0000 7fc60ac00640 1 --2- 172.21.15.90:0/1969580965 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e39d723c00 0x55e39c8c6680 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x55e39d7386f0 tx=0x55e39d632d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.063 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.054+0000 7fc603e00640 1 -- 172.21.15.90:0/1969580965 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e39c880820 con 0x55e39d723c00 2024-03-24T01:30:33.063 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.054+0000 7fc603e00640 1 -- 172.21.15.90:0/1969580965 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e39c8861c0 con 0x55e39d723c00 2024-03-24T01:30:33.064 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.054+0000 7fc603e00640 1 -- 172.21.15.90:0/1969580965 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e39c880680 con 0x55e39d723c00 2024-03-24T01:30:33.064 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.054+0000 7fc60dc36280 1 -- 172.21.15.90:0/1969580965 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55e39c88e3c0 con 0x55e39d723c00 2024-03-24T01:30:33.064 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.054+0000 7fc60dc36280 1 -- 172.21.15.90:0/1969580965 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e39c88e000 con 0x55e39d723c00 2024-03-24T01:30:33.064 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.055+0000 7fc603e00640 1 -- 172.21.15.90:0/1969580965 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x55e39d792c00 con 0x55e39d723c00 2024-03-24T01:30:33.064 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.056+0000 7fc603e00640 1 --2- 172.21.15.90:0/1969580965 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55e39cbcc000 0x55e39c8c7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.064 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.056+0000 7fc603e00640 1 -- 172.21.15.90:0/1969580965 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x55e39d790c00 con 0x55e39d723c00 2024-03-24T01:30:33.064 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.056+0000 7fc60dc36280 1 --2- 172.21.15.90:0/1969580965 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55e39cbcc400 0x55e39c8c6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.064 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.056+0000 7fc60dc36280 1 -- 172.21.15.90:0/1969580965 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:1 7.f 7:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e39cbcc800 con 0x55e39cbcc400 2024-03-24T01:30:33.065 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.057+0000 7fc60b600640 1 --2- 172.21.15.90:0/1969580965 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55e39cbcc400 0x55e39c8c6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.065 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.057+0000 7fc60c000640 1 --2- 172.21.15.90:0/1969580965 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55e39cbcc000 0x55e39c8c7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.065 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.057+0000 7fc60b600640 1 --2- 172.21.15.90:0/1969580965 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55e39cbcc400 0x55e39c8c6100 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.065 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.057+0000 7fc60b600640 1 -- 172.21.15.90:0/1969580965 <== osd.0 v2:172.21.15.90:6800/307045587 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x55e39c9da000 con 0x55e39cbcc400 2024-03-24T01:30:33.066 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.057+0000 7fc60dc36280 1 --2- 172.21.15.90:0/1969580965 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55e39cbccc00 0x55e39c8c5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.066 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.057+0000 7fc60dc36280 1 -- 172.21.15.90:0/1969580965 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:2 7.14 7:292cf221:::200.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e39cbcd000 con 0x55e39cbccc00 2024-03-24T01:30:33.067 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.058+0000 7fc60c000640 1 --2- 172.21.15.90:0/1969580965 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55e39cbcc000 0x55e39c8c7700 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x55e39d738840 tx=0x55e39d73ac00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.067 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.058+0000 7fc60c000640 1 --2- 172.21.15.90:0/1969580965 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55e39cbccc00 0x55e39c8c5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.067 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.058+0000 7fc60c000640 1 --2- 172.21.15.90:0/1969580965 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55e39cbccc00 0x55e39c8c5b80 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.069 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.058+0000 7fc60c000640 1 -- 172.21.15.90:0/1969580965 <== osd.3 v2:172.21.15.90:6808/2999931983 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv24 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x55e39c9da500 con 0x55e39cbccc00 2024-03-24T01:30:33.069 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.058+0000 7fc60dc36280 1 -- 172.21.15.90:0/1969580965 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55e39cbcc400 msgr2=0x55e39c8c6100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.069 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.059+0000 7fc60dc36280 1 --2- 172.21.15.90:0/1969580965 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55e39cbcc400 0x55e39c8c6100 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.069 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.059+0000 7fc60dc36280 1 -- 172.21.15.90:0/1969580965 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55e39cbccc00 msgr2=0x55e39c8c5b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.069 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.059+0000 7fc60dc36280 1 --2- 172.21.15.90:0/1969580965 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55e39cbccc00 0x55e39c8c5b80 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.069 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.059+0000 7fc60dc36280 1 -- 172.21.15.90:0/1969580965 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55e39cbcc000 msgr2=0x55e39c8c7700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.069 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.059+0000 7fc60dc36280 1 --2- 172.21.15.90:0/1969580965 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55e39cbcc000 0x55e39c8c7700 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x55e39d738840 tx=0x55e39d73ac00 comp rx=0 tx=0).stop 2024-03-24T01:30:33.069 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.059+0000 7fc60dc36280 1 -- 172.21.15.90:0/1969580965 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e39d723c00 msgr2=0x55e39c8c6680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.069 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.059+0000 7fc60dc36280 1 --2- 172.21.15.90:0/1969580965 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e39d723c00 0x55e39c8c6680 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x55e39d7386f0 tx=0x55e39d632d20 comp rx=0 tx=0).stop 2024-03-24T01:30:33.069 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.059+0000 7fc60b600640 1 -- 172.21.15.90:0/1969580965 reap_dead start 2024-03-24T01:30:33.069 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.059+0000 7fc60dc36280 1 -- 172.21.15.90:0/1969580965 shutdown_connections 2024-03-24T01:30:33.070 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.059+0000 7fc60dc36280 1 -- 172.21.15.90:0/1969580965 >> 172.21.15.90:0/1969580965 conn(0x55e39d723000 msgr2=0x55e39d5d8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.070 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.059+0000 7fc60dc36280 1 -- 172.21.15.90:0/1969580965 shutdown_connections 2024-03-24T01:30:33.070 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.059+0000 7fc60dc36280 1 -- 172.21.15.90:0/1969580965 wait complete. 2024-03-24T01:30:33.070 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.059+0000 7fc60dc36280 1 -- 172.21.15.90:0/1432063662 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e39c85b800 msgr2=0x55e39c8c4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.070 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.059+0000 7fc60dc36280 1 --2- 172.21.15.90:0/1432063662 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e39c85b800 0x55e39c8c4b00 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x55e39d677380 tx=0x55e39d632e40 comp rx=0 tx=0).stop 2024-03-24T01:30:33.070 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.059+0000 7fc60dc36280 1 -- 172.21.15.90:0/1432063662 shutdown_connections 2024-03-24T01:30:33.070 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.059+0000 7fc60dc36280 1 --2- 172.21.15.90:0/1432063662 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e39c85b800 0x55e39c8c4b00 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.070 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.059+0000 7fc60dc36280 1 --2- 172.21.15.90:0/1432063662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e39c85bc00 0x55e39c8c5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.070 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.059+0000 7fc60dc36280 1 --2- 172.21.15.90:0/1432063662 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e39c85ac00 0x55e39c8c4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.070 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.059+0000 7fc60dc36280 1 -- 172.21.15.90:0/1432063662 >> 172.21.15.90:0/1432063662 conn(0x55e39c85b000 msgr2=0x55e39d5d9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.070 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.059+0000 7fc60dc36280 1 -- 172.21.15.90:0/1432063662 shutdown_connections 2024-03-24T01:30:33.070 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.059+0000 7fc60dc36280 1 -- 172.21.15.90:0/1432063662 wait complete. 2024-03-24T01:30:33.070 INFO:tasks.workunit.client.0.smithi090.stderr:+ NEW_HEADER_STATE='{ 2024-03-24T01:30:33.070 INFO:tasks.workunit.client.0.smithi090.stderr: "magic": "ceph fs volume v011", 2024-03-24T01:30:33.071 INFO:tasks.workunit.client.0.smithi090.stderr: "write_pos": 4200185, 2024-03-24T01:30:33.071 INFO:tasks.workunit.client.0.smithi090.stderr: "expire_pos": 4194304, 2024-03-24T01:30:33.071 INFO:tasks.workunit.client.0.smithi090.stderr: "trimmed_pos": 4194304, 2024-03-24T01:30:33.071 INFO:tasks.workunit.client.0.smithi090.stderr: "stream_format": 1, 2024-03-24T01:30:33.071 INFO:tasks.workunit.client.0.smithi090.stderr: "layout": { 2024-03-24T01:30:33.071 INFO:tasks.workunit.client.0.smithi090.stderr: "stripe_unit": 4194304, 2024-03-24T01:30:33.071 INFO:tasks.workunit.client.0.smithi090.stderr: "stripe_count": 1, 2024-03-24T01:30:33.071 INFO:tasks.workunit.client.0.smithi090.stderr: "object_size": 4194304, 2024-03-24T01:30:33.071 INFO:tasks.workunit.client.0.smithi090.stderr: "pool_id": 7, 2024-03-24T01:30:33.071 INFO:tasks.workunit.client.0.smithi090.stderr: "pool_ns": "" 2024-03-24T01:30:33.071 INFO:tasks.workunit.client.0.smithi090.stderr: } 2024-03-24T01:30:33.071 INFO:tasks.workunit.client.0.smithi090.stderr:}' 2024-03-24T01:30:33.071 INFO:tasks.workunit.client.0.smithi090.stderr:++ cephfs-journal-tool --rank=cephfs:0 event get list 2024-03-24T01:30:33.110 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.108+0000 7fcea5e64280 1 -- start start 2024-03-24T01:30:33.110 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.108+0000 7fcea5e64280 1 Processor -- start 2024-03-24T01:30:33.111 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.109+0000 7fcea5e64280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576bdda4000 0x5576bd037080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.111 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.109+0000 7fcea5e64280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576bcfcdc00 0x5576bd036b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.111 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.109+0000 7fcea5e64280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5576bdda4800 0x5576bd037b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.111 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.109+0000 7fcea5e64280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576bde92000 con 0x5576bcfcdc00 2024-03-24T01:30:33.111 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.109+0000 7fcea5e64280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576bde92180 con 0x5576bdda4000 2024-03-24T01:30:33.111 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.109+0000 7fcea5e64280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5576bde92300 con 0x5576bdda4800 2024-03-24T01:30:33.111 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.110+0000 7fcea3600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576bcfcdc00 0x5576bd036b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.111 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.110+0000 7fcea3600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576bcfcdc00 0x5576bd036b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36204/0 (socket says 172.21.15.90:36204) 2024-03-24T01:30:33.111 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.110+0000 7fcea3600640 1 -- 172.21.15.90:0/2104133007 learned_addr learned my addr 172.21.15.90:0/2104133007 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.113 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.110+0000 7fcea3600640 1 -- 172.21.15.90:0/2104133007 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5576bdda4800 msgr2=0x5576bd037b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.113 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.110+0000 7fcea3600640 1 --2- 172.21.15.90:0/2104133007 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5576bdda4800 0x5576bd037b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.113 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.110+0000 7fcea3600640 1 -- 172.21.15.90:0/2104133007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576bdda4000 msgr2=0x5576bd037080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.113 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.110+0000 7fcea3600640 1 --2- 172.21.15.90:0/2104133007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576bdda4000 0x5576bd037080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.113 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.110+0000 7fcea3600640 1 -- 172.21.15.90:0/2104133007 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5576bd000780 con 0x5576bcfcdc00 2024-03-24T01:30:33.113 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.110+0000 7fcea3600640 1 --2- 172.21.15.90:0/2104133007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576bcfcdc00 0x5576bd036b00 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x5576bddf0d80 tx=0x5576bdd98f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5c159a4fac62052 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.113 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.110+0000 7fcea0e00640 1 -- 172.21.15.90:0/2104133007 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5576bcff2b60 con 0x5576bcfcdc00 2024-03-24T01:30:33.113 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.110+0000 7fcea0e00640 1 -- 172.21.15.90:0/2104133007 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5576bcff8e00 con 0x5576bcfcdc00 2024-03-24T01:30:33.113 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.110+0000 7fcea0e00640 1 -- 172.21.15.90:0/2104133007 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5576bcff2820 con 0x5576bcfcdc00 2024-03-24T01:30:33.113 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.111+0000 7fcea5e64280 1 -- 172.21.15.90:0/2104133007 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5576bd0014a0 con 0x5576bcfcdc00 2024-03-24T01:30:33.114 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.111+0000 7fcea5e64280 1 -- 172.21.15.90:0/2104133007 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x5576bd000960 con 0x5576bcfcdc00 2024-03-24T01:30:33.114 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.111+0000 7fcea0e00640 1 -- 172.21.15.90:0/2104133007 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x5576bde98200 con 0x5576bcfcdc00 2024-03-24T01:30:33.114 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.111+0000 7fcea0e00640 1 -- 172.21.15.90:0/2104133007 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 36) v1 ==== 3310+0+0 (secure 0 0 0) 0x5576bd027180 con 0x5576bcfcdc00 2024-03-24T01:30:33.114 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.112+0000 7fcea5e64280 1 Processor -- start 2024-03-24T01:30:33.114 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.112+0000 7fcea5e64280 1 -- start start 2024-03-24T01:30:33.116 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.113+0000 7fcea5e64280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576bdead400 0x5576bd037600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.116 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.113+0000 7fcea5e64280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5576bdead800 0x5576bd036580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.116 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.113+0000 7fcea5e64280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576bdeadc00 0x5576bd038100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.116 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.113+0000 7fcea5e64280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576bde92480 con 0x5576bdead400 2024-03-24T01:30:33.116 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.113+0000 7fcea5e64280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576bde92600 con 0x5576bdeadc00 2024-03-24T01:30:33.116 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.113+0000 7fcea5e64280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5576bde92780 con 0x5576bdead800 2024-03-24T01:30:33.116 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.113+0000 7fcea4a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576bdead400 0x5576bd037600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.117 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.113+0000 7fcea4a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576bdead400 0x5576bd037600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36216/0 (socket says 172.21.15.90:36216) 2024-03-24T01:30:33.117 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.113+0000 7fcea4a00640 1 -- 172.21.15.90:0/872311840 learned_addr learned my addr 172.21.15.90:0/872311840 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.117 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.113+0000 7fcea4a00640 1 -- 172.21.15.90:0/872311840 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5576bdead800 msgr2=0x5576bd036580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.117 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.113+0000 7fcea4a00640 1 --2- 172.21.15.90:0/872311840 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5576bdead800 0x5576bd036580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.117 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.113+0000 7fcea4a00640 1 -- 172.21.15.90:0/872311840 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576bdeadc00 msgr2=0x5576bd038100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.117 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.113+0000 7fcea4a00640 1 --2- 172.21.15.90:0/872311840 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576bdeadc00 0x5576bd038100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.117 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.113+0000 7fcea4a00640 1 -- 172.21.15.90:0/872311840 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5576bd0003c0 con 0x5576bdead400 2024-03-24T01:30:33.117 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.114+0000 7fcea4a00640 1 --2- 172.21.15.90:0/872311840 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576bdead400 0x5576bd037600 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x5576bdeb6b10 tx=0x5576bdeb4d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8977e12294de1406 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.117 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.114+0000 7fce9e600640 1 -- 172.21.15.90:0/872311840 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5576bcff2b60 con 0x5576bdead400 2024-03-24T01:30:33.117 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.114+0000 7fce9e600640 1 -- 172.21.15.90:0/872311840 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5576bcff8700 con 0x5576bdead400 2024-03-24T01:30:33.117 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.114+0000 7fce9e600640 1 -- 172.21.15.90:0/872311840 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5576bcff2820 con 0x5576bdead400 2024-03-24T01:30:33.117 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.114+0000 7fcea5e64280 1 -- 172.21.15.90:0/872311840 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576bdead400 msgr2=0x5576bd037600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.117 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.114+0000 7fcea5e64280 1 --2- 172.21.15.90:0/872311840 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576bdead400 0x5576bd037600 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x5576bdeb6b10 tx=0x5576bdeb4d20 comp rx=0 tx=0).stop 2024-03-24T01:30:33.117 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.114+0000 7fcea5e64280 1 -- 172.21.15.90:0/872311840 shutdown_connections 2024-03-24T01:30:33.117 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.114+0000 7fcea5e64280 1 --2- 172.21.15.90:0/872311840 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576bdeadc00 0x5576bd038100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.118 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.114+0000 7fcea5e64280 1 --2- 172.21.15.90:0/872311840 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5576bdead800 0x5576bd036580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.118 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.114+0000 7fcea5e64280 1 --2- 172.21.15.90:0/872311840 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576bdead400 0x5576bd037600 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.118 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.114+0000 7fcea5e64280 1 -- 172.21.15.90:0/872311840 >> 172.21.15.90:0/872311840 conn(0x5576bdda4400 msgr2=0x5576bd0d1000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.118 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.114+0000 7fcea5e64280 1 -- 172.21.15.90:0/872311840 shutdown_connections 2024-03-24T01:30:33.118 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.114+0000 7fcea5e64280 1 -- 172.21.15.90:0/872311840 wait complete. 2024-03-24T01:30:33.118 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.115+0000 7fcea5e64280 1 Processor -- start 2024-03-24T01:30:33.118 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.115+0000 7fcea5e64280 1 -- start start 2024-03-24T01:30:33.120 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.116+0000 7fcea5e64280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576bdead400 0x5576bd039180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.120 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.116+0000 7fcea5e64280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5576bdead800 0x5576bd038c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.120 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.116+0000 7fcea5e64280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576bdeadc00 0x5576bd038680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.121 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.116+0000 7fcea5e64280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576bde93b00 con 0x5576bdeadc00 2024-03-24T01:30:33.121 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.116+0000 7fcea5e64280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576bde93980 con 0x5576bdead400 2024-03-24T01:30:33.121 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.116+0000 7fcea5e64280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5576bde92c00 con 0x5576bdead800 2024-03-24T01:30:33.121 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.116+0000 7fcea3600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576bdeadc00 0x5576bd038680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.121 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.116+0000 7fcea3600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576bdeadc00 0x5576bd038680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36222/0 (socket says 172.21.15.90:36222) 2024-03-24T01:30:33.121 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.116+0000 7fcea3600640 1 -- 172.21.15.90:0/3449038547 learned_addr learned my addr 172.21.15.90:0/3449038547 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.121 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.116+0000 7fcea4a00640 1 --2- 172.21.15.90:0/3449038547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576bdead400 0x5576bd039180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.121 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.116+0000 7fcea3600640 1 -- 172.21.15.90:0/3449038547 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5576bdead800 msgr2=0x5576bd038c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.121 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.116+0000 7fcea4000640 1 --2- 172.21.15.90:0/3449038547 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5576bdead800 0x5576bd038c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.121 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.116+0000 7fcea3600640 1 --2- 172.21.15.90:0/3449038547 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5576bdead800 0x5576bd038c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.121 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.117+0000 7fcea3600640 1 -- 172.21.15.90:0/3449038547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576bdead400 msgr2=0x5576bd039180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.121 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.117+0000 7fcea3600640 1 --2- 172.21.15.90:0/3449038547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576bdead400 0x5576bd039180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.121 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.117+0000 7fcea3600640 1 -- 172.21.15.90:0/3449038547 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5576bd000780 con 0x5576bdeadc00 2024-03-24T01:30:33.121 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.117+0000 7fcea3600640 1 --2- 172.21.15.90:0/3449038547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576bdeadc00 0x5576bd038680 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x5576bdeb6750 tx=0x5576bdd98fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.122 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.117+0000 7fce9c800640 1 -- 172.21.15.90:0/3449038547 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5576bcff2b60 con 0x5576bdeadc00 2024-03-24T01:30:33.122 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.117+0000 7fce9c800640 1 -- 172.21.15.90:0/3449038547 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5576bcff8c40 con 0x5576bdeadc00 2024-03-24T01:30:33.122 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.117+0000 7fce9c800640 1 -- 172.21.15.90:0/3449038547 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5576bcff2820 con 0x5576bdeadc00 2024-03-24T01:30:33.122 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.117+0000 7fcea5e64280 1 -- 172.21.15.90:0/3449038547 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5576bd000000 con 0x5576bdeadc00 2024-03-24T01:30:33.122 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.117+0000 7fcea5e64280 1 -- 172.21.15.90:0/3449038547 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5576bd001680 con 0x5576bdeadc00 2024-03-24T01:30:33.122 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.119+0000 7fce9c800640 1 -- 172.21.15.90:0/3449038547 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x5576bdf0ac00 con 0x5576bdeadc00 2024-03-24T01:30:33.122 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.119+0000 7fce9c800640 1 --2- 172.21.15.90:0/3449038547 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x5576bdf14400 0x5576bd039700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.122 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.119+0000 7fce9c800640 1 -- 172.21.15.90:0/3449038547 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x5576bdf03200 con 0x5576bdeadc00 2024-03-24T01:30:33.122 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.119+0000 7fcea5e64280 1 --2- 172.21.15.90:0/3449038547 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x5576bdf14800 0x5576bd038100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.122 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.119+0000 7fcea5e64280 1 -- 172.21.15.90:0/3449038547 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:1 7.f 7:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x5576bdf14c00 con 0x5576bdf14800 2024-03-24T01:30:33.124 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.120+0000 7fcea4000640 1 --2- 172.21.15.90:0/3449038547 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x5576bdf14800 0x5576bd038100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.124 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.120+0000 7fcea4a00640 1 --2- 172.21.15.90:0/3449038547 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x5576bdf14400 0x5576bd039700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.124 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.120+0000 7fcea4000640 1 --2- 172.21.15.90:0/3449038547 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x5576bdf14800 0x5576bd038100 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.124 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.121+0000 7fcea4000640 1 -- 172.21.15.90:0/3449038547 <== osd.0 v2:172.21.15.90:6800/307045587 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x5576bdd50000 con 0x5576bdf14800 2024-03-24T01:30:33.124 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.121+0000 7fcea5e64280 1 --2- 172.21.15.90:0/3449038547 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x5576bdf15000 0x5576bd036580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.124 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.121+0000 7fcea5e64280 1 -- 172.21.15.90:0/3449038547 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:2 7.14 7:292cf221:::200.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x5576bdf15400 con 0x5576bdf15000 2024-03-24T01:30:33.124 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.121+0000 7fcea4a00640 1 --2- 172.21.15.90:0/3449038547 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x5576bdf14400 0x5576bd039700 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x5576bdeb6b10 tx=0x5576bdeb4c00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.124 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.121+0000 7fcea4a00640 1 --2- 172.21.15.90:0/3449038547 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x5576bdf15000 0x5576bd036580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.124 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.121+0000 7fcea4a00640 1 --2- 172.21.15.90:0/3449038547 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x5576bdf15000 0x5576bd036580 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.124 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.122+0000 7fcea4a00640 1 -- 172.21.15.90:0/3449038547 <== osd.3 v2:172.21.15.90:6808/2999931983 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv24 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x5576bdd50500 con 0x5576bdf15000 2024-03-24T01:30:33.124 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.122+0000 7fcea5e64280 1 -- 172.21.15.90:0/3449038547 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:3 7.14 7:2e2fa760:::200.00000001:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x5576bdf15800 con 0x5576bdf15000 2024-03-24T01:30:33.125 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.123+0000 7fcea4a00640 1 -- 172.21.15.90:0/3449038547 <== osd.3 v2:172.21.15.90:6808/2999931983 2 ==== osd_op_reply(3 200.00000001 [read 0~35827 out=35827b] v0'0 uv26 ondisk = 0) v8 ==== 156+0+35827 (crc 0 0 0) 0x5576bdd50500 con 0x5576bdf15000 2024-03-24T01:30:33.125 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.123+0000 7fcea5e64280 1 -- 172.21.15.90:0/3449038547 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:4 7.1b 7:ddb4e4d7:::200.00000002:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x5576bdf15c00 con 0x5576bdf14800 2024-03-24T01:30:33.126 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.123+0000 7fcea4000640 1 -- 172.21.15.90:0/3449038547 <== osd.0 v2:172.21.15.90:6800/307045587 2 ==== osd_op_reply(4 200.00000002 [read 0~2147483647] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x5576bdd50000 con 0x5576bdf14800 2024-03-24T01:30:33.126 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.123+0000 7fcea5e64280 1 -- 172.21.15.90:0/3449038547 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x5576bdf14800 msgr2=0x5576bd038100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.126 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.123+0000 7fcea5e64280 1 --2- 172.21.15.90:0/3449038547 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x5576bdf14800 0x5576bd038100 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.127 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.123+0000 7fcea5e64280 1 -- 172.21.15.90:0/3449038547 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x5576bdf15000 msgr2=0x5576bd036580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.127 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.123+0000 7fcea5e64280 1 --2- 172.21.15.90:0/3449038547 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x5576bdf15000 0x5576bd036580 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.127 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.124+0000 7fcea5e64280 1 -- 172.21.15.90:0/3449038547 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x5576bdf14400 msgr2=0x5576bd039700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.127 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.124+0000 7fcea5e64280 1 --2- 172.21.15.90:0/3449038547 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x5576bdf14400 0x5576bd039700 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x5576bdeb6b10 tx=0x5576bdeb4c00 comp rx=0 tx=0).stop 2024-03-24T01:30:33.127 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.124+0000 7fcea5e64280 1 -- 172.21.15.90:0/3449038547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576bdeadc00 msgr2=0x5576bd038680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.127 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.124+0000 7fcea5e64280 1 --2- 172.21.15.90:0/3449038547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576bdeadc00 0x5576bd038680 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x5576bdeb6750 tx=0x5576bdd98fc0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.127 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.124+0000 7fcea4000640 1 -- 172.21.15.90:0/3449038547 reap_dead start 2024-03-24T01:30:33.127 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.124+0000 7fcea5e64280 1 -- 172.21.15.90:0/3449038547 shutdown_connections 2024-03-24T01:30:33.127 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.124+0000 7fcea5e64280 1 -- 172.21.15.90:0/3449038547 >> 172.21.15.90:0/3449038547 conn(0x5576bdda4400 msgr2=0x5576bd0d1000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.127 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.124+0000 7fcea5e64280 1 -- 172.21.15.90:0/3449038547 shutdown_connections 2024-03-24T01:30:33.127 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.124+0000 7fcea5e64280 1 -- 172.21.15.90:0/3449038547 wait complete. 2024-03-24T01:30:33.127 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.124+0000 7fcea5e64280 1 -- 172.21.15.90:0/2104133007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576bcfcdc00 msgr2=0x5576bd036b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.127 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.124+0000 7fcea5e64280 1 --2- 172.21.15.90:0/2104133007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576bcfcdc00 0x5576bd036b00 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x5576bddf0d80 tx=0x5576bdd98f00 comp rx=0 tx=0).stop 2024-03-24T01:30:33.127 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.124+0000 7fcea5e64280 1 -- 172.21.15.90:0/2104133007 shutdown_connections 2024-03-24T01:30:33.128 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.124+0000 7fcea5e64280 1 --2- 172.21.15.90:0/2104133007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576bcfcdc00 0x5576bd036b00 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.128 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.124+0000 7fcea5e64280 1 --2- 172.21.15.90:0/2104133007 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5576bdda4800 0x5576bd037b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.128 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.124+0000 7fcea5e64280 1 --2- 172.21.15.90:0/2104133007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576bdda4000 0x5576bd037080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.128 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.124+0000 7fcea5e64280 1 -- 172.21.15.90:0/2104133007 >> 172.21.15.90:0/2104133007 conn(0x5576bcfcd000 msgr2=0x5576bd0d2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.128 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.124+0000 7fcea5e64280 1 -- 172.21.15.90:0/2104133007 shutdown_connections 2024-03-24T01:30:33.128 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.124+0000 7fcea5e64280 1 -- 172.21.15.90:0/2104133007 wait complete. 2024-03-24T01:30:33.130 INFO:tasks.workunit.client.0.smithi090.stderr:+ NEW_EVENT_LIST='0.000000 0x400000 LID: () 2024-03-24T01:30:33.130 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:27:14.548044+0000 0x400030 SUBTREEMAP: () 2024-03-24T01:30:33.130 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:27:27.768607+0000 0x4003ac SESSION: () 2024-03-24T01:30:33.130 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:27:28.411755+0000 0x400565 UPDATE: (setattr) 2024-03-24T01:30:33.130 INFO:tasks.workunit.client.0.smithi090.stderr: / 2024-03-24T01:30:33.130 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:27:28.576042+0000 0x400894 UPDATE: (mkdir) 2024-03-24T01:30:33.130 INFO:tasks.workunit.client.0.smithi090.stderr: client.0 2024-03-24T01:30:33.131 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:27:33.769077+0000 0x401074 UPDATE: (cap update) 2024-03-24T01:30:33.131 INFO:tasks.workunit.client.0.smithi090.stderr: client.0 2024-03-24T01:30:33.131 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:53.486798+0000 0x4016f9 UPDATE: (mkdir) 2024-03-24T01:30:33.131 INFO:tasks.workunit.client.0.smithi090.stderr: client.0/tmp 2024-03-24T01:30:33.131 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:53.536138+0000 0x402217 UPDATE: (mkdir) 2024-03-24T01:30:33.131 INFO:tasks.workunit.client.0.smithi090.stderr: tmp/foo 2024-03-24T01:30:33.131 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:53.549102+0000 0x402ae7 UPDATE: (openc) 2024-03-24T01:30:33.131 INFO:tasks.workunit.client.0.smithi090.stderr: tmp/bar 2024-03-24T01:30:33.131 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:56.112617+0000 0x403289 UPDATE: (unlink_local) 2024-03-24T01:30:33.131 INFO:tasks.workunit.client.0.smithi090.stderr: stray2/10000000002 2024-03-24T01:30:33.131 INFO:tasks.workunit.client.0.smithi090.stderr: client.0/tmp/foo 2024-03-24T01:30:33.131 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:56.118494+0000 0x40448c UPDATE: (cap update) 2024-03-24T01:30:33.131 INFO:tasks.workunit.client.0.smithi090.stderr: tmp/bar 2024-03-24T01:30:33.131 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:56.127543+0000 0x404c0d UPDATE: (unlink_local) 2024-03-24T01:30:33.132 INFO:tasks.workunit.client.0.smithi090.stderr: stray2/10000000003 2024-03-24T01:30:33.132 INFO:tasks.workunit.client.0.smithi090.stderr: tmp/bar 2024-03-24T01:30:33.132 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:56.142108+0000 0x405870 UPDATE: (unlink_local) 2024-03-24T01:30:33.132 INFO:tasks.workunit.client.0.smithi090.stderr: stray2/10000000001 2024-03-24T01:30:33.132 INFO:tasks.workunit.client.0.smithi090.stderr: client.0/tmp 2024-03-24T01:30:33.132 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:57.246240+0000 0x4064d8 UPDATE: (unlink_local) 2024-03-24T01:30:33.132 INFO:tasks.workunit.client.0.smithi090.stderr: client.0 2024-03-24T01:30:33.132 INFO:tasks.workunit.client.0.smithi090.stderr: stray2/10000000000 2024-03-24T01:30:33.132 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:58.733720+0000 0x407037 UPDATE: (purge_stray) 2024-03-24T01:30:33.132 INFO:tasks.workunit.client.0.smithi090.stderr: 10000000002 2024-03-24T01:30:33.132 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:58.733860+0000 0x40728e UPDATE: (purge_stray) 2024-03-24T01:30:33.132 INFO:tasks.workunit.client.0.smithi090.stderr: 10000000003 2024-03-24T01:30:33.132 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:59.116268+0000 0x4074e5 UPDATE: (scatter_writebehind) 2024-03-24T01:30:33.132 INFO:tasks.workunit.client.0.smithi090.stderr: stray2/10000000000 2024-03-24T01:30:33.132 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:59.119801+0000 0x407dbe UPDATE: (scatter_writebehind) 2024-03-24T01:30:33.133 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:59.732514+0000 0x408230 UPDATE: (purge_stray) 2024-03-24T01:30:33.133 INFO:tasks.workunit.client.0.smithi090.stderr: 10000000001 2024-03-24T01:30:33.133 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:29:01.965453+0000 0x408487 SESSION: () 2024-03-24T01:30:33.133 INFO:tasks.workunit.client.0.smithi090.stderr:0.000000 0x408534 PURGED: () 2024-03-24T01:30:33.133 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:29:04.115764+0000 0x408580 UPDATE: (scatter_writebehind) 2024-03-24T01:30:33.133 INFO:tasks.workunit.client.0.smithi090.stderr: stray2' 2024-03-24T01:30:33.133 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' '!' '{ 2024-03-24T01:30:33.133 INFO:tasks.workunit.client.0.smithi090.stderr: "magic": "ceph fs volume v011", 2024-03-24T01:30:33.133 INFO:tasks.workunit.client.0.smithi090.stderr: "write_pos": 4200185, 2024-03-24T01:30:33.133 INFO:tasks.workunit.client.0.smithi090.stderr: "expire_pos": 4194304, 2024-03-24T01:30:33.133 INFO:tasks.workunit.client.0.smithi090.stderr: "trimmed_pos": 4194304, 2024-03-24T01:30:33.133 INFO:tasks.workunit.client.0.smithi090.stderr: "stream_format": 1, 2024-03-24T01:30:33.133 INFO:tasks.workunit.client.0.smithi090.stderr: "layout": { 2024-03-24T01:30:33.133 INFO:tasks.workunit.client.0.smithi090.stderr: "stripe_unit": 4194304, 2024-03-24T01:30:33.133 INFO:tasks.workunit.client.0.smithi090.stderr: "stripe_count": 1, 2024-03-24T01:30:33.134 INFO:tasks.workunit.client.0.smithi090.stderr: "object_size": 4194304, 2024-03-24T01:30:33.134 INFO:tasks.workunit.client.0.smithi090.stderr: "pool_id": 7, 2024-03-24T01:30:33.134 INFO:tasks.workunit.client.0.smithi090.stderr: "pool_ns": "" 2024-03-24T01:30:33.134 INFO:tasks.workunit.client.0.smithi090.stderr: } 2024-03-24T01:30:33.134 INFO:tasks.workunit.client.0.smithi090.stderr:}' = '{ 2024-03-24T01:30:33.134 INFO:tasks.workunit.client.0.smithi090.stderr: "magic": "ceph fs volume v011", 2024-03-24T01:30:33.134 INFO:tasks.workunit.client.0.smithi090.stderr: "write_pos": 4200185, 2024-03-24T01:30:33.134 INFO:tasks.workunit.client.0.smithi090.stderr: "expire_pos": 4194304, 2024-03-24T01:30:33.134 INFO:tasks.workunit.client.0.smithi090.stderr: "trimmed_pos": 4194304, 2024-03-24T01:30:33.134 INFO:tasks.workunit.client.0.smithi090.stderr: "stream_format": 1, 2024-03-24T01:30:33.134 INFO:tasks.workunit.client.0.smithi090.stderr: "layout": { 2024-03-24T01:30:33.134 INFO:tasks.workunit.client.0.smithi090.stderr: "stripe_unit": 4194304, 2024-03-24T01:30:33.134 INFO:tasks.workunit.client.0.smithi090.stderr: "stripe_count": 1, 2024-03-24T01:30:33.134 INFO:tasks.workunit.client.0.smithi090.stderr: "object_size": 4194304, 2024-03-24T01:30:33.134 INFO:tasks.workunit.client.0.smithi090.stderr: "pool_id": 7, 2024-03-24T01:30:33.135 INFO:tasks.workunit.client.0.smithi090.stderr: "pool_ns": "" 2024-03-24T01:30:33.135 INFO:tasks.workunit.client.0.smithi090.stderr: } 2024-03-24T01:30:33.135 INFO:tasks.workunit.client.0.smithi090.stderr:}' ']' 2024-03-24T01:30:33.135 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' '!' '0.000000 0x400000 LID: () 2024-03-24T01:30:33.135 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:27:14.548044+0000 0x400030 SUBTREEMAP: () 2024-03-24T01:30:33.135 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:27:27.768607+0000 0x4003ac SESSION: () 2024-03-24T01:30:33.135 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:27:28.411755+0000 0x400565 UPDATE: (setattr) 2024-03-24T01:30:33.135 INFO:tasks.workunit.client.0.smithi090.stderr: / 2024-03-24T01:30:33.135 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:27:28.576042+0000 0x400894 UPDATE: (mkdir) 2024-03-24T01:30:33.135 INFO:tasks.workunit.client.0.smithi090.stderr: client.0 2024-03-24T01:30:33.135 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:27:33.769077+0000 0x401074 UPDATE: (cap update) 2024-03-24T01:30:33.135 INFO:tasks.workunit.client.0.smithi090.stderr: client.0 2024-03-24T01:30:33.135 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:53.486798+0000 0x4016f9 UPDATE: (mkdir) 2024-03-24T01:30:33.135 INFO:tasks.workunit.client.0.smithi090.stderr: client.0/tmp 2024-03-24T01:30:33.136 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:53.536138+0000 0x402217 UPDATE: (mkdir) 2024-03-24T01:30:33.136 INFO:tasks.workunit.client.0.smithi090.stderr: tmp/foo 2024-03-24T01:30:33.136 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:53.549102+0000 0x402ae7 UPDATE: (openc) 2024-03-24T01:30:33.136 INFO:tasks.workunit.client.0.smithi090.stderr: tmp/bar 2024-03-24T01:30:33.136 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:56.112617+0000 0x403289 UPDATE: (unlink_local) 2024-03-24T01:30:33.136 INFO:tasks.workunit.client.0.smithi090.stderr: stray2/10000000002 2024-03-24T01:30:33.136 INFO:tasks.workunit.client.0.smithi090.stderr: client.0/tmp/foo 2024-03-24T01:30:33.136 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:56.118494+0000 0x40448c UPDATE: (cap update) 2024-03-24T01:30:33.136 INFO:tasks.workunit.client.0.smithi090.stderr: tmp/bar 2024-03-24T01:30:33.136 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:56.127543+0000 0x404c0d UPDATE: (unlink_local) 2024-03-24T01:30:33.136 INFO:tasks.workunit.client.0.smithi090.stderr: stray2/10000000003 2024-03-24T01:30:33.136 INFO:tasks.workunit.client.0.smithi090.stderr: tmp/bar 2024-03-24T01:30:33.136 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:56.142108+0000 0x405870 UPDATE: (unlink_local) 2024-03-24T01:30:33.136 INFO:tasks.workunit.client.0.smithi090.stderr: stray2/10000000001 2024-03-24T01:30:33.136 INFO:tasks.workunit.client.0.smithi090.stderr: client.0/tmp 2024-03-24T01:30:33.137 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:57.246240+0000 0x4064d8 UPDATE: (unlink_local) 2024-03-24T01:30:33.137 INFO:tasks.workunit.client.0.smithi090.stderr: client.0 2024-03-24T01:30:33.137 INFO:tasks.workunit.client.0.smithi090.stderr: stray2/10000000000 2024-03-24T01:30:33.137 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:58.733720+0000 0x407037 UPDATE: (purge_stray) 2024-03-24T01:30:33.137 INFO:tasks.workunit.client.0.smithi090.stderr: 10000000002 2024-03-24T01:30:33.137 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:58.733860+0000 0x40728e UPDATE: (purge_stray) 2024-03-24T01:30:33.137 INFO:tasks.workunit.client.0.smithi090.stderr: 10000000003 2024-03-24T01:30:33.137 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:59.116268+0000 0x4074e5 UPDATE: (scatter_writebehind) 2024-03-24T01:30:33.137 INFO:tasks.workunit.client.0.smithi090.stderr: stray2/10000000000 2024-03-24T01:30:33.137 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:59.119801+0000 0x407dbe UPDATE: (scatter_writebehind) 2024-03-24T01:30:33.137 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:59.732514+0000 0x408230 UPDATE: (purge_stray) 2024-03-24T01:30:33.137 INFO:tasks.workunit.client.0.smithi090.stderr: 10000000001 2024-03-24T01:30:33.137 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:29:01.965453+0000 0x408487 SESSION: () 2024-03-24T01:30:33.137 INFO:tasks.workunit.client.0.smithi090.stderr:0.000000 0x408534 PURGED: () 2024-03-24T01:30:33.137 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:29:04.115764+0000 0x408580 UPDATE: (scatter_writebehind) 2024-03-24T01:30:33.138 INFO:tasks.workunit.client.0.smithi090.stderr: stray2' = '0.000000 0x400000 LID: () 2024-03-24T01:30:33.138 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:27:14.548044+0000 0x400030 SUBTREEMAP: () 2024-03-24T01:30:33.138 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:27:27.768607+0000 0x4003ac SESSION: () 2024-03-24T01:30:33.138 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:27:28.411755+0000 0x400565 UPDATE: (setattr) 2024-03-24T01:30:33.138 INFO:tasks.workunit.client.0.smithi090.stderr: / 2024-03-24T01:30:33.138 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:27:28.576042+0000 0x400894 UPDATE: (mkdir) 2024-03-24T01:30:33.138 INFO:tasks.workunit.client.0.smithi090.stderr: client.0 2024-03-24T01:30:33.138 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:27:33.769077+0000 0x401074 UPDATE: (cap update) 2024-03-24T01:30:33.138 INFO:tasks.workunit.client.0.smithi090.stderr: client.0 2024-03-24T01:30:33.138 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:53.486798+0000 0x4016f9 UPDATE: (mkdir) 2024-03-24T01:30:33.138 INFO:tasks.workunit.client.0.smithi090.stderr: client.0/tmp 2024-03-24T01:30:33.138 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:53.536138+0000 0x402217 UPDATE: (mkdir) 2024-03-24T01:30:33.138 INFO:tasks.workunit.client.0.smithi090.stderr: tmp/foo 2024-03-24T01:30:33.138 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:53.549102+0000 0x402ae7 UPDATE: (openc) 2024-03-24T01:30:33.138 INFO:tasks.workunit.client.0.smithi090.stderr: tmp/bar 2024-03-24T01:30:33.139 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:56.112617+0000 0x403289 UPDATE: (unlink_local) 2024-03-24T01:30:33.139 INFO:tasks.workunit.client.0.smithi090.stderr: stray2/10000000002 2024-03-24T01:30:33.139 INFO:tasks.workunit.client.0.smithi090.stderr: client.0/tmp/foo 2024-03-24T01:30:33.139 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:56.118494+0000 0x40448c UPDATE: (cap update) 2024-03-24T01:30:33.139 INFO:tasks.workunit.client.0.smithi090.stderr: tmp/bar 2024-03-24T01:30:33.139 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:56.127543+0000 0x404c0d UPDATE: (unlink_local) 2024-03-24T01:30:33.139 INFO:tasks.workunit.client.0.smithi090.stderr: stray2/10000000003 2024-03-24T01:30:33.139 INFO:tasks.workunit.client.0.smithi090.stderr: tmp/bar 2024-03-24T01:30:33.139 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:56.142108+0000 0x405870 UPDATE: (unlink_local) 2024-03-24T01:30:33.139 INFO:tasks.workunit.client.0.smithi090.stderr: stray2/10000000001 2024-03-24T01:30:33.139 INFO:tasks.workunit.client.0.smithi090.stderr: client.0/tmp 2024-03-24T01:30:33.139 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:57.246240+0000 0x4064d8 UPDATE: (unlink_local) 2024-03-24T01:30:33.139 INFO:tasks.workunit.client.0.smithi090.stderr: client.0 2024-03-24T01:30:33.139 INFO:tasks.workunit.client.0.smithi090.stderr: stray2/10000000000 2024-03-24T01:30:33.139 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:58.733720+0000 0x407037 UPDATE: (purge_stray) 2024-03-24T01:30:33.139 INFO:tasks.workunit.client.0.smithi090.stderr: 10000000002 2024-03-24T01:30:33.140 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:58.733860+0000 0x40728e UPDATE: (purge_stray) 2024-03-24T01:30:33.140 INFO:tasks.workunit.client.0.smithi090.stderr: 10000000003 2024-03-24T01:30:33.140 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:59.116268+0000 0x4074e5 UPDATE: (scatter_writebehind) 2024-03-24T01:30:33.140 INFO:tasks.workunit.client.0.smithi090.stderr: stray2/10000000000 2024-03-24T01:30:33.140 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:59.119801+0000 0x407dbe UPDATE: (scatter_writebehind) 2024-03-24T01:30:33.140 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:28:59.732514+0000 0x408230 UPDATE: (purge_stray) 2024-03-24T01:30:33.140 INFO:tasks.workunit.client.0.smithi090.stderr: 10000000001 2024-03-24T01:30:33.140 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:29:01.965453+0000 0x408487 SESSION: () 2024-03-24T01:30:33.140 INFO:tasks.workunit.client.0.smithi090.stderr:0.000000 0x408534 PURGED: () 2024-03-24T01:30:33.140 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:29:04.115764+0000 0x408580 UPDATE: (scatter_writebehind) 2024-03-24T01:30:33.140 INFO:tasks.workunit.client.0.smithi090.stderr: stray2' ']' 2024-03-24T01:30:33.140 INFO:tasks.workunit.client.0.smithi090.stdout:Testing 'journal' commands... 2024-03-24T01:30:33.141 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'Testing '\''journal'\'' commands...' 2024-03-24T01:30:33.141 INFO:tasks.workunit.client.0.smithi090.stderr:+ cephfs-journal-tool --rank=cephfs:0 journal inspect 2024-03-24T01:30:33.175 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.174+0000 7f039160c280 1 -- start start 2024-03-24T01:30:33.175 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.174+0000 7f039160c280 1 Processor -- start 2024-03-24T01:30:33.176 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.175+0000 7f039160c280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b045a26c00 0x55b045a90580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.176 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.175+0000 7f039160c280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b045a27800 0x55b045a90b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.176 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.175+0000 7f039160c280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b045a27c00 0x55b045a91080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.176 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.175+0000 7f039160c280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0467fc180 con 0x55b045a26c00 2024-03-24T01:30:33.176 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.175+0000 7f039160c280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0467fc300 con 0x55b045a27800 2024-03-24T01:30:33.176 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.175+0000 7f039160c280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b0467fc480 con 0x55b045a27c00 2024-03-24T01:30:33.178 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.175+0000 7f038ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b045a26c00 0x55b045a90580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.178 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.175+0000 7f038e000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b045a27800 0x55b045a90b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.178 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.175+0000 7f038f400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b045a27c00 0x55b045a91080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.178 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.175+0000 7f038e000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b045a27800 0x55b045a90b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.90:52384/0 (socket says 172.21.15.90:52384) 2024-03-24T01:30:33.178 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.176+0000 7f038e000640 1 -- 172.21.15.90:0/2422712800 learned_addr learned my addr 172.21.15.90:0/2422712800 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.178 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.176+0000 7f038ea00640 1 -- 172.21.15.90:0/2422712800 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b045a27c00 msgr2=0x55b045a91080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.178 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.176+0000 7f038ea00640 1 --2- 172.21.15.90:0/2422712800 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b045a27c00 0x55b045a91080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.178 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.176+0000 7f038ea00640 1 -- 172.21.15.90:0/2422712800 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b045a27800 msgr2=0x55b045a90b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.178 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.176+0000 7f038ea00640 1 --2- 172.21.15.90:0/2422712800 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b045a27800 0x55b045a90b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.178 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.176+0000 7f038ea00640 1 -- 172.21.15.90:0/2422712800 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b045a5a780 con 0x55b045a26c00 2024-03-24T01:30:33.178 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.176+0000 7f038ea00640 1 --2- 172.21.15.90:0/2422712800 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b045a26c00 0x55b045a90580 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x55b04684d440 tx=0x55b046800f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9abaeb1b605972b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.179 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.177+0000 7f038b800640 1 -- 172.21.15.90:0/2422712800 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55b045a4c9c0 con 0x55b045a26c00 2024-03-24T01:30:33.179 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.177+0000 7f038b800640 1 -- 172.21.15.90:0/2422712800 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b045a52fc0 con 0x55b045a26c00 2024-03-24T01:30:33.179 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.177+0000 7f038b800640 1 -- 172.21.15.90:0/2422712800 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55b045a4c680 con 0x55b045a26c00 2024-03-24T01:30:33.179 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.177+0000 7f039160c280 1 -- 172.21.15.90:0/2422712800 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b045a5b4a0 con 0x55b045a26c00 2024-03-24T01:30:33.179 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.177+0000 7f039160c280 1 -- 172.21.15.90:0/2422712800 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x55b045a5a000 con 0x55b045a26c00 2024-03-24T01:30:33.180 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.177+0000 7f038b800640 1 -- 172.21.15.90:0/2422712800 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x55b045b0ea00 con 0x55b045a26c00 2024-03-24T01:30:33.180 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.177+0000 7f038b800640 1 -- 172.21.15.90:0/2422712800 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 36) v1 ==== 3310+0+0 (secure 0 0 0) 0x55b045a81180 con 0x55b045a26c00 2024-03-24T01:30:33.180 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.178+0000 7f039160c280 1 Processor -- start 2024-03-24T01:30:33.180 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.178+0000 7f039160c280 1 -- start start 2024-03-24T01:30:33.182 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.179+0000 7f039160c280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b046950000 0x55b045a91600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.182 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.179+0000 7f039160c280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b046950400 0x55b045a91b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.182 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.179+0000 7f039160c280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b046950800 0x55b045a92100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.182 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.179+0000 7f039160c280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0467fc600 con 0x55b046950400 2024-03-24T01:30:33.182 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.179+0000 7f039160c280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0467fc780 con 0x55b046950000 2024-03-24T01:30:33.182 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.179+0000 7f039160c280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b0467fc900 con 0x55b046950800 2024-03-24T01:30:33.182 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.179+0000 7f038ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b046950400 0x55b045a91b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.182 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.179+0000 7f038ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b046950400 0x55b045a91b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36230/0 (socket says 172.21.15.90:36230) 2024-03-24T01:30:33.182 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.179+0000 7f038ea00640 1 -- 172.21.15.90:0/86672279 learned_addr learned my addr 172.21.15.90:0/86672279 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.182 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.179+0000 7f038ea00640 1 -- 172.21.15.90:0/86672279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b046950800 msgr2=0x55b045a92100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.182 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.179+0000 7f038ea00640 1 --2- 172.21.15.90:0/86672279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b046950800 0x55b045a92100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.183 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.179+0000 7f038ea00640 1 -- 172.21.15.90:0/86672279 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b046950000 msgr2=0x55b045a91600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.183 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.179+0000 7f038ea00640 1 --2- 172.21.15.90:0/86672279 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b046950000 0x55b045a91600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.183 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.180+0000 7f038ea00640 1 -- 172.21.15.90:0/86672279 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b045a5a3c0 con 0x55b046950400 2024-03-24T01:30:33.183 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.180+0000 7f038ea00640 1 --2- 172.21.15.90:0/86672279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b046950400 0x55b045a91b80 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x55b04684d890 tx=0x55b046800fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16db42e14e9b12c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.183 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.180+0000 7f0389000640 1 -- 172.21.15.90:0/86672279 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55b045a4cb60 con 0x55b046950400 2024-03-24T01:30:33.183 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.180+0000 7f0389000640 1 -- 172.21.15.90:0/86672279 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b045a52e00 con 0x55b046950400 2024-03-24T01:30:33.183 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.180+0000 7f0389000640 1 -- 172.21.15.90:0/86672279 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55b045a4c820 con 0x55b046950400 2024-03-24T01:30:33.183 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.180+0000 7f039160c280 1 -- 172.21.15.90:0/86672279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b046950400 msgr2=0x55b045a91b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.183 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.180+0000 7f039160c280 1 --2- 172.21.15.90:0/86672279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b046950400 0x55b045a91b80 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x55b04684d890 tx=0x55b046800fc0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.183 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.180+0000 7f039160c280 1 -- 172.21.15.90:0/86672279 shutdown_connections 2024-03-24T01:30:33.183 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.180+0000 7f039160c280 1 --2- 172.21.15.90:0/86672279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b046950400 0x55b045a91b80 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.183 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.180+0000 7f039160c280 1 --2- 172.21.15.90:0/86672279 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b046950800 0x55b045a92100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.183 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.180+0000 7f039160c280 1 --2- 172.21.15.90:0/86672279 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b046950000 0x55b045a91600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.183 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.180+0000 7f039160c280 1 -- 172.21.15.90:0/86672279 >> 172.21.15.90:0/86672279 conn(0x55b046813c00 msgr2=0x55b045b2f800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.184 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.181+0000 7f039160c280 1 -- 172.21.15.90:0/86672279 shutdown_connections 2024-03-24T01:30:33.184 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.181+0000 7f039160c280 1 -- 172.21.15.90:0/86672279 wait complete. 2024-03-24T01:30:33.185 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.181+0000 7f039160c280 1 Processor -- start 2024-03-24T01:30:33.185 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.181+0000 7f039160c280 1 -- start start 2024-03-24T01:30:33.186 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.182+0000 7f039160c280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b046950000 0x55b045a93180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.186 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.182+0000 7f039160c280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b046950400 0x55b045a92c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.186 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.182+0000 7f039160c280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b046950800 0x55b045a92680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.186 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.182+0000 7f039160c280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0467fdc80 con 0x55b046950400 2024-03-24T01:30:33.186 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.182+0000 7f039160c280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0467fdb00 con 0x55b046950000 2024-03-24T01:30:33.186 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.182+0000 7f039160c280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b0467fd980 con 0x55b046950800 2024-03-24T01:30:33.186 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.182+0000 7f038ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b046950400 0x55b045a92c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.186 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.182+0000 7f038ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b046950400 0x55b045a92c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36244/0 (socket says 172.21.15.90:36244) 2024-03-24T01:30:33.186 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.182+0000 7f038ea00640 1 -- 172.21.15.90:0/3266636595 learned_addr learned my addr 172.21.15.90:0/3266636595 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.186 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.183+0000 7f038ea00640 1 -- 172.21.15.90:0/3266636595 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b046950800 msgr2=0x55b045a92680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.186 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.183+0000 7f038ea00640 1 --2- 172.21.15.90:0/3266636595 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b046950800 0x55b045a92680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.186 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.183+0000 7f038ea00640 1 -- 172.21.15.90:0/3266636595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b046950000 msgr2=0x55b045a93180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.186 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.183+0000 7f038ea00640 1 --2- 172.21.15.90:0/3266636595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b046950000 0x55b045a93180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.186 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.183+0000 7f038ea00640 1 -- 172.21.15.90:0/3266636595 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b045a5a960 con 0x55b046950400 2024-03-24T01:30:33.187 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.183+0000 7f038ea00640 1 --2- 172.21.15.90:0/3266636595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b046950400 0x55b045a92c00 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x55b04682c5d0 tx=0x55b046800e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.187 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.183+0000 7f0387200640 1 -- 172.21.15.90:0/3266636595 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55b045a4cb60 con 0x55b046950400 2024-03-24T01:30:33.187 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.183+0000 7f0387200640 1 -- 172.21.15.90:0/3266636595 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b045a52c40 con 0x55b046950400 2024-03-24T01:30:33.187 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.183+0000 7f0387200640 1 -- 172.21.15.90:0/3266636595 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55b045a4c820 con 0x55b046950400 2024-03-24T01:30:33.187 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.183+0000 7f039160c280 1 -- 172.21.15.90:0/3266636595 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55b045a5b680 con 0x55b046950400 2024-03-24T01:30:33.187 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.183+0000 7f039160c280 1 -- 172.21.15.90:0/3266636595 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b045a5b860 con 0x55b046950400 2024-03-24T01:30:33.188 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.185+0000 7f0387200640 1 -- 172.21.15.90:0/3266636595 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x55b046912c00 con 0x55b046950400 2024-03-24T01:30:33.189 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.185+0000 7f0387200640 1 --2- 172.21.15.90:0/3266636595 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55b046962400 0x55b045a93700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.189 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.186+0000 7f0387200640 1 -- 172.21.15.90:0/3266636595 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x55b04695cc00 con 0x55b046950400 2024-03-24T01:30:33.189 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.186+0000 7f039160c280 1 --2- 172.21.15.90:0/3266636595 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55b046950c00 0x55b045a92100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.189 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.186+0000 7f039160c280 1 -- 172.21.15.90:0/3266636595 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:1 7.f 7:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b046951000 con 0x55b046950c00 2024-03-24T01:30:33.189 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.186+0000 7f038e000640 1 --2- 172.21.15.90:0/3266636595 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55b046950c00 0x55b045a92100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.189 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.186+0000 7f038f400640 1 --2- 172.21.15.90:0/3266636595 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55b046962400 0x55b045a93700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.189 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.187+0000 7f038e000640 1 --2- 172.21.15.90:0/3266636595 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55b046950c00 0x55b045a92100 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.189 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.187+0000 7f038e000640 1 -- 172.21.15.90:0/3266636595 <== osd.0 v2:172.21.15.90:6800/307045587 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x55b0467aa000 con 0x55b046950c00 2024-03-24T01:30:33.189 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.187+0000 7f039160c280 1 --2- 172.21.15.90:0/3266636595 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55b046963400 0x55b045a91b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.189 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.187+0000 7f039160c280 1 -- 172.21.15.90:0/3266636595 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:2 7.14 7:292cf221:::200.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b046963800 con 0x55b046963400 2024-03-24T01:30:33.189 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.187+0000 7f038f400640 1 --2- 172.21.15.90:0/3266636595 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55b046962400 0x55b045a93700 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x55b045ba6900 tx=0x55b045d1ac00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.189 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.187+0000 7f038f400640 1 --2- 172.21.15.90:0/3266636595 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55b046963400 0x55b045a91b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.192 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.188+0000 7f038f400640 1 --2- 172.21.15.90:0/3266636595 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55b046963400 0x55b045a91b80 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.192 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.188+0000 7f038f400640 1 -- 172.21.15.90:0/3266636595 <== osd.3 v2:172.21.15.90:6808/2999931983 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv24 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x55b0467aa500 con 0x55b046963400 2024-03-24T01:30:33.192 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.188+0000 7f039160c280 1 -- 172.21.15.90:0/3266636595 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:3 7.14 7:2e2fa760:::200.00000001:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b046963c00 con 0x55b046963400 2024-03-24T01:30:33.192 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.189+0000 7f038f400640 1 -- 172.21.15.90:0/3266636595 <== osd.3 v2:172.21.15.90:6808/2999931983 2 ==== osd_op_reply(3 200.00000001 [read 0~35827 out=35827b] v0'0 uv26 ondisk = 0) v8 ==== 156+0+35827 (crc 0 0 0) 0x55b0467aa500 con 0x55b046963400 2024-03-24T01:30:33.192 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.189+0000 7f039160c280 1 -- 172.21.15.90:0/3266636595 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:4 7.1b 7:ddb4e4d7:::200.00000002:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b045dc8000 con 0x55b046950c00 2024-03-24T01:30:33.192 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.189+0000 7f038e000640 1 -- 172.21.15.90:0/3266636595 <== osd.0 v2:172.21.15.90:6800/307045587 2 ==== osd_op_reply(4 200.00000002 [read 0~2147483647] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x55b0467aa000 con 0x55b046950c00 2024-03-24T01:30:33.193 INFO:tasks.workunit.client.0.smithi090.stdout:Overall journal integrity: OK 2024-03-24T01:30:33.195 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.189+0000 7f039160c280 1 -- 172.21.15.90:0/3266636595 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55b046950c00 msgr2=0x55b045a92100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.195 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.189+0000 7f039160c280 1 --2- 172.21.15.90:0/3266636595 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55b046950c00 0x55b045a92100 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.195 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.189+0000 7f039160c280 1 -- 172.21.15.90:0/3266636595 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55b046963400 msgr2=0x55b045a91b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.195 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.189+0000 7f039160c280 1 --2- 172.21.15.90:0/3266636595 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55b046963400 0x55b045a91b80 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.195 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.189+0000 7f039160c280 1 -- 172.21.15.90:0/3266636595 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55b046962400 msgr2=0x55b045a93700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.195 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.189+0000 7f039160c280 1 --2- 172.21.15.90:0/3266636595 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55b046962400 0x55b045a93700 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x55b045ba6900 tx=0x55b045d1ac00 comp rx=0 tx=0).stop 2024-03-24T01:30:33.195 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.189+0000 7f039160c280 1 -- 172.21.15.90:0/3266636595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b046950400 msgr2=0x55b045a92c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.195 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.189+0000 7f039160c280 1 --2- 172.21.15.90:0/3266636595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b046950400 0x55b045a92c00 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x55b04682c5d0 tx=0x55b046800e40 comp rx=0 tx=0).stop 2024-03-24T01:30:33.195 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.189+0000 7f038e000640 1 -- 172.21.15.90:0/3266636595 reap_dead start 2024-03-24T01:30:33.195 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.189+0000 7f039160c280 1 -- 172.21.15.90:0/3266636595 shutdown_connections 2024-03-24T01:30:33.195 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.189+0000 7f039160c280 1 -- 172.21.15.90:0/3266636595 >> 172.21.15.90:0/3266636595 conn(0x55b046813c00 msgr2=0x55b045b2f800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.195 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.190+0000 7f039160c280 1 -- 172.21.15.90:0/3266636595 shutdown_connections 2024-03-24T01:30:33.195 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.190+0000 7f039160c280 1 -- 172.21.15.90:0/3266636595 wait complete. 2024-03-24T01:30:33.196 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.190+0000 7f039160c280 1 -- 172.21.15.90:0/2422712800 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b045a26c00 msgr2=0x55b045a90580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.196 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.190+0000 7f039160c280 1 --2- 172.21.15.90:0/2422712800 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b045a26c00 0x55b045a90580 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x55b04684d440 tx=0x55b046800f00 comp rx=0 tx=0).stop 2024-03-24T01:30:33.196 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.190+0000 7f039160c280 1 -- 172.21.15.90:0/2422712800 shutdown_connections 2024-03-24T01:30:33.196 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.190+0000 7f039160c280 1 --2- 172.21.15.90:0/2422712800 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b045a27c00 0x55b045a91080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.196 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.190+0000 7f039160c280 1 --2- 172.21.15.90:0/2422712800 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b045a27800 0x55b045a90b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.196 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.190+0000 7f039160c280 1 --2- 172.21.15.90:0/2422712800 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b045a26c00 0x55b045a90580 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.196 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.190+0000 7f039160c280 1 -- 172.21.15.90:0/2422712800 >> 172.21.15.90:0/2422712800 conn(0x55b045a27000 msgr2=0x55b045b2f000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.196 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.190+0000 7f039160c280 1 -- 172.21.15.90:0/2422712800 shutdown_connections 2024-03-24T01:30:33.196 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.190+0000 7f039160c280 1 -- 172.21.15.90:0/2422712800 wait complete. 2024-03-24T01:30:33.196 INFO:tasks.workunit.client.0.smithi090.stderr:+ cephfs-journal-tool --rank=cephfs:0 header get 2024-03-24T01:30:33.239 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.238+0000 7f32a77db280 1 -- start start 2024-03-24T01:30:33.239 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.238+0000 7f32a77db280 1 Processor -- start 2024-03-24T01:30:33.241 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.239+0000 7f32a77db280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617e5a52c00 0x5617e5abc580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.241 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.239+0000 7f32a77db280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5617e5a53800 0x5617e5abcb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.241 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.239+0000 7f32a77db280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617e5a53c00 0x5617e5abd080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.241 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.239+0000 7f32a77db280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617e6834180 con 0x5617e5a52c00 2024-03-24T01:30:33.241 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.239+0000 7f32a77db280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617e6834300 con 0x5617e5a53c00 2024-03-24T01:30:33.241 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.239+0000 7f32a77db280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5617e6834480 con 0x5617e5a53800 2024-03-24T01:30:33.241 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.239+0000 7f32a5800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617e5a52c00 0x5617e5abc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.241 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.239+0000 7f32a5800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617e5a52c00 0x5617e5abc580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36258/0 (socket says 172.21.15.90:36258) 2024-03-24T01:30:33.242 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.239+0000 7f32a5800640 1 -- 172.21.15.90:0/2275491120 learned_addr learned my addr 172.21.15.90:0/2275491120 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.242 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.240+0000 7f32a5800640 1 -- 172.21.15.90:0/2275491120 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5617e5a53800 msgr2=0x5617e5abcb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.242 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.240+0000 7f32a5800640 1 --2- 172.21.15.90:0/2275491120 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5617e5a53800 0x5617e5abcb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.242 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.240+0000 7f32a5800640 1 -- 172.21.15.90:0/2275491120 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617e5a53c00 msgr2=0x5617e5abd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.242 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.240+0000 7f32a5800640 1 --2- 172.21.15.90:0/2275491120 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617e5a53c00 0x5617e5abd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.242 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.240+0000 7f32a5800640 1 -- 172.21.15.90:0/2275491120 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5617e5a86780 con 0x5617e5a52c00 2024-03-24T01:30:33.242 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.240+0000 7f32a5800640 1 --2- 172.21.15.90:0/2275491120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617e5a52c00 0x5617e5abc580 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x5617e687d380 tx=0x5617e6828f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d22291dfeea732c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.244 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.240+0000 7f32a2600640 1 -- 172.21.15.90:0/2275491120 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5617e5a78680 con 0x5617e5a52c00 2024-03-24T01:30:33.244 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.240+0000 7f32a2600640 1 -- 172.21.15.90:0/2275491120 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5617e5a7e700 con 0x5617e5a52c00 2024-03-24T01:30:33.244 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.240+0000 7f32a2600640 1 -- 172.21.15.90:0/2275491120 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5617e5a78b60 con 0x5617e5a52c00 2024-03-24T01:30:33.244 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.240+0000 7f32a77db280 1 -- 172.21.15.90:0/2275491120 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5617e5a863c0 con 0x5617e5a52c00 2024-03-24T01:30:33.244 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.240+0000 7f32a77db280 1 -- 172.21.15.90:0/2275491120 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x5617e5a87680 con 0x5617e5a52c00 2024-03-24T01:30:33.244 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.241+0000 7f32a2600640 1 -- 172.21.15.90:0/2275491120 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x5617e67c2a00 con 0x5617e5a52c00 2024-03-24T01:30:33.244 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.241+0000 7f32a2600640 1 -- 172.21.15.90:0/2275491120 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 36) v1 ==== 3310+0+0 (secure 0 0 0) 0x5617e5aad180 con 0x5617e5a52c00 2024-03-24T01:30:33.244 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.242+0000 7f32a77db280 1 Processor -- start 2024-03-24T01:30:33.244 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.242+0000 7f32a77db280 1 -- start start 2024-03-24T01:30:33.244 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.242+0000 7f32a77db280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617e691b400 0x5617e5abd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.244 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.242+0000 7f32a77db280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5617e691b800 0x5617e5abdb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.245 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.242+0000 7f32a77db280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617e691bc00 0x5617e5abe100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.245 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.242+0000 7f32a77db280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617e6834600 con 0x5617e691bc00 2024-03-24T01:30:33.245 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.242+0000 7f32a77db280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617e6834780 con 0x5617e691b400 2024-03-24T01:30:33.245 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.242+0000 7f32a77db280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5617e6834900 con 0x5617e691b800 2024-03-24T01:30:33.245 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.243+0000 7f32a4e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617e691bc00 0x5617e5abe100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.245 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.243+0000 7f32a4e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617e691bc00 0x5617e5abe100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36266/0 (socket says 172.21.15.90:36266) 2024-03-24T01:30:33.245 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.243+0000 7f32a4e00640 1 -- 172.21.15.90:0/1171360385 learned_addr learned my addr 172.21.15.90:0/1171360385 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.248 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.243+0000 7f32a4e00640 1 -- 172.21.15.90:0/1171360385 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5617e691b800 msgr2=0x5617e5abdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.248 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.243+0000 7f32a4e00640 1 --2- 172.21.15.90:0/1171360385 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5617e691b800 0x5617e5abdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.248 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.243+0000 7f32a4e00640 1 -- 172.21.15.90:0/1171360385 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617e691b400 msgr2=0x5617e5abd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.248 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.243+0000 7f32a4e00640 1 --2- 172.21.15.90:0/1171360385 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617e691b400 0x5617e5abd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.248 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.243+0000 7f32a4e00640 1 -- 172.21.15.90:0/1171360385 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5617e5a874a0 con 0x5617e691bc00 2024-03-24T01:30:33.249 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.243+0000 7f32a4e00640 1 --2- 172.21.15.90:0/1171360385 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617e691bc00 0x5617e5abe100 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x5617e69345a0 tx=0x5617e6936d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44aceb285652be9a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.249 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.244+0000 7f329fe00640 1 -- 172.21.15.90:0/1171360385 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5617e5a789c0 con 0x5617e691bc00 2024-03-24T01:30:33.249 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.244+0000 7f329fe00640 1 -- 172.21.15.90:0/1171360385 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5617e5a7ee00 con 0x5617e691bc00 2024-03-24T01:30:33.249 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.244+0000 7f329fe00640 1 -- 172.21.15.90:0/1171360385 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5617e5a78680 con 0x5617e691bc00 2024-03-24T01:30:33.249 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.244+0000 7f32a77db280 1 -- 172.21.15.90:0/1171360385 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617e691bc00 msgr2=0x5617e5abe100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.249 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.244+0000 7f32a77db280 1 --2- 172.21.15.90:0/1171360385 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617e691bc00 0x5617e5abe100 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x5617e69345a0 tx=0x5617e6936d20 comp rx=0 tx=0).stop 2024-03-24T01:30:33.249 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.244+0000 7f32a77db280 1 -- 172.21.15.90:0/1171360385 shutdown_connections 2024-03-24T01:30:33.249 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.244+0000 7f32a77db280 1 --2- 172.21.15.90:0/1171360385 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617e691bc00 0x5617e5abe100 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.249 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.244+0000 7f32a77db280 1 --2- 172.21.15.90:0/1171360385 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5617e691b800 0x5617e5abdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.249 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.244+0000 7f32a77db280 1 --2- 172.21.15.90:0/1171360385 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617e691b400 0x5617e5abd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.249 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.244+0000 7f32a77db280 1 -- 172.21.15.90:0/1171360385 >> 172.21.15.90:0/1171360385 conn(0x5617e691b000 msgr2=0x5617e67d3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.249 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.244+0000 7f32a77db280 1 -- 172.21.15.90:0/1171360385 shutdown_connections 2024-03-24T01:30:33.249 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.244+0000 7f32a77db280 1 -- 172.21.15.90:0/1171360385 wait complete. 2024-03-24T01:30:33.249 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.245+0000 7f32a77db280 1 Processor -- start 2024-03-24T01:30:33.250 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.245+0000 7f32a77db280 1 -- start start 2024-03-24T01:30:33.250 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.246+0000 7f32a77db280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617e691b400 0x5617e5abf180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.250 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.246+0000 7f32a77db280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5617e691b800 0x5617e5abec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.250 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.246+0000 7f32a77db280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617e691bc00 0x5617e5abe680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.250 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.246+0000 7f32a77db280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617e5ab4180 con 0x5617e691bc00 2024-03-24T01:30:33.250 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.246+0000 7f32a77db280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617e6835e00 con 0x5617e691b400 2024-03-24T01:30:33.250 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.246+0000 7f32a77db280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5617e6834d80 con 0x5617e691b800 2024-03-24T01:30:33.250 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.246+0000 7f32a4e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617e691bc00 0x5617e5abe680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.250 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.246+0000 7f32a4e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617e691bc00 0x5617e5abe680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36276/0 (socket says 172.21.15.90:36276) 2024-03-24T01:30:33.250 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.246+0000 7f32a4e00640 1 -- 172.21.15.90:0/1820222606 learned_addr learned my addr 172.21.15.90:0/1820222606 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.250 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.246+0000 7f32a4e00640 1 -- 172.21.15.90:0/1820222606 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5617e691b800 msgr2=0x5617e5abec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.250 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.246+0000 7f32a4e00640 1 --2- 172.21.15.90:0/1820222606 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5617e691b800 0x5617e5abec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.250 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.246+0000 7f32a4e00640 1 -- 172.21.15.90:0/1820222606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617e691b400 msgr2=0x5617e5abf180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.251 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.246+0000 7f32a6200640 1 --2- 172.21.15.90:0/1820222606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617e691b400 0x5617e5abf180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.251 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.246+0000 7f32a4e00640 1 --2- 172.21.15.90:0/1820222606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617e691b400 0x5617e5abf180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.251 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.246+0000 7f32a4e00640 1 -- 172.21.15.90:0/1820222606 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5617e5a86780 con 0x5617e691bc00 2024-03-24T01:30:33.251 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.247+0000 7f32a4e00640 1 --2- 172.21.15.90:0/1820222606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617e691bc00 0x5617e5abe680 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x5617e69348d0 tx=0x5617e6936c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.251 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.247+0000 7f329e000640 1 -- 172.21.15.90:0/1820222606 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5617e5a78b60 con 0x5617e691bc00 2024-03-24T01:30:33.251 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.247+0000 7f329e000640 1 -- 172.21.15.90:0/1820222606 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5617e5a7ec40 con 0x5617e691bc00 2024-03-24T01:30:33.251 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.247+0000 7f329e000640 1 -- 172.21.15.90:0/1820222606 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5617e5a784e0 con 0x5617e691bc00 2024-03-24T01:30:33.251 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.247+0000 7f32a77db280 1 -- 172.21.15.90:0/1820222606 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5617e5a863c0 con 0x5617e691bc00 2024-03-24T01:30:33.251 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.247+0000 7f32a77db280 1 -- 172.21.15.90:0/1820222606 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5617e5a87680 con 0x5617e691bc00 2024-03-24T01:30:33.253 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.249+0000 7f329e000640 1 -- 172.21.15.90:0/1820222606 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x5617e6854c00 con 0x5617e691bc00 2024-03-24T01:30:33.253 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.249+0000 7f329e000640 1 --2- 172.21.15.90:0/1820222606 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x5617e685c400 0x5617e5abf700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.253 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.249+0000 7f329e000640 1 -- 172.21.15.90:0/1820222606 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x5617e685a600 con 0x5617e691bc00 2024-03-24T01:30:33.253 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.250+0000 7f32a77db280 1 --2- 172.21.15.90:0/1820222606 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x5617e685c800 0x5617e5abe100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.253 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.250+0000 7f32a77db280 1 -- 172.21.15.90:0/1820222606 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:1 7.f 7:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x5617e685cc00 con 0x5617e685c800 2024-03-24T01:30:33.253 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.250+0000 7f32a5800640 1 --2- 172.21.15.90:0/1820222606 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x5617e685c800 0x5617e5abe100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.253 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.250+0000 7f32a6200640 1 --2- 172.21.15.90:0/1820222606 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x5617e685c400 0x5617e5abf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.253 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.250+0000 7f32a5800640 1 --2- 172.21.15.90:0/1820222606 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x5617e685c800 0x5617e5abe100 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.254 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.251+0000 7f32a5800640 1 -- 172.21.15.90:0/1820222606 <== osd.0 v2:172.21.15.90:6800/307045587 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x5617e67d6000 con 0x5617e685c800 2024-03-24T01:30:33.254 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.251+0000 7f32a77db280 1 --2- 172.21.15.90:0/1820222606 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x5617e685d000 0x5617e5abdb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.254 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.251+0000 7f32a77db280 1 -- 172.21.15.90:0/1820222606 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:2 7.14 7:292cf221:::200.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x5617e685d400 con 0x5617e685d000 2024-03-24T01:30:33.254 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.251+0000 7f32a6200640 1 --2- 172.21.15.90:0/1820222606 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x5617e685d000 0x5617e5abdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.254 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.251+0000 7f32a6200640 1 --2- 172.21.15.90:0/1820222606 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x5617e685c400 0x5617e5abf700 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x5617e5b34690 tx=0x5617e5dce000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.254 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.251+0000 7f32a6200640 1 --2- 172.21.15.90:0/1820222606 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x5617e685d000 0x5617e5abdb80 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.254 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.252+0000 7f32a6200640 1 -- 172.21.15.90:0/1820222606 <== osd.3 v2:172.21.15.90:6808/2999931983 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv24 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x5617e67d6500 con 0x5617e685d000 2024-03-24T01:30:33.254 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2024-03-24T01:30:33.254 INFO:tasks.workunit.client.0.smithi090.stdout: "magic": "ceph fs volume v011", 2024-03-24T01:30:33.254 INFO:tasks.workunit.client.0.smithi090.stdout: "write_pos": 4200185, 2024-03-24T01:30:33.255 INFO:tasks.workunit.client.0.smithi090.stdout: "expire_pos": 4194304, 2024-03-24T01:30:33.255 INFO:tasks.workunit.client.0.smithi090.stdout: "trimmed_pos": 4194304, 2024-03-24T01:30:33.255 INFO:tasks.workunit.client.0.smithi090.stdout: "stream_format": 1, 2024-03-24T01:30:33.255 INFO:tasks.workunit.client.0.smithi090.stdout: "layout": { 2024-03-24T01:30:33.255 INFO:tasks.workunit.client.0.smithi090.stdout: "stripe_unit": 4194304, 2024-03-24T01:30:33.255 INFO:tasks.workunit.client.0.smithi090.stdout: "stripe_count": 1, 2024-03-24T01:30:33.255 INFO:tasks.workunit.client.0.smithi090.stdout: "object_size": 4194304, 2024-03-24T01:30:33.255 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_id": 7, 2024-03-24T01:30:33.255 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_ns": "" 2024-03-24T01:30:33.255 INFO:tasks.workunit.client.0.smithi090.stdout: } 2024-03-24T01:30:33.255 INFO:tasks.workunit.client.0.smithi090.stdout:} 2024-03-24T01:30:33.255 INFO:tasks.workunit.client.0.smithi090.stdout: 2024-03-24T01:30:33.257 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.252+0000 7f32a77db280 1 -- 172.21.15.90:0/1820222606 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x5617e685c800 msgr2=0x5617e5abe100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.257 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.252+0000 7f32a77db280 1 --2- 172.21.15.90:0/1820222606 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x5617e685c800 0x5617e5abe100 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.257 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.252+0000 7f32a77db280 1 -- 172.21.15.90:0/1820222606 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x5617e685d000 msgr2=0x5617e5abdb80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.257 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.252+0000 7f32a77db280 1 --2- 172.21.15.90:0/1820222606 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x5617e685d000 0x5617e5abdb80 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.257 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.252+0000 7f32a77db280 1 -- 172.21.15.90:0/1820222606 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x5617e685c400 msgr2=0x5617e5abf700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.257 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.252+0000 7f32a77db280 1 --2- 172.21.15.90:0/1820222606 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x5617e685c400 0x5617e5abf700 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x5617e5b34690 tx=0x5617e5dce000 comp rx=0 tx=0).stop 2024-03-24T01:30:33.257 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.252+0000 7f32a77db280 1 -- 172.21.15.90:0/1820222606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617e691bc00 msgr2=0x5617e5abe680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.257 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.252+0000 7f32a77db280 1 --2- 172.21.15.90:0/1820222606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617e691bc00 0x5617e5abe680 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x5617e69348d0 tx=0x5617e6936c00 comp rx=0 tx=0).stop 2024-03-24T01:30:33.257 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.252+0000 7f32a4e00640 1 -- 172.21.15.90:0/1820222606 reap_dead start 2024-03-24T01:30:33.257 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.252+0000 7f32a4e00640 1 -- 172.21.15.90:0/1820222606 reap_dead start 2024-03-24T01:30:33.257 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.252+0000 7f32a77db280 1 -- 172.21.15.90:0/1820222606 shutdown_connections 2024-03-24T01:30:33.258 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.252+0000 7f32a77db280 1 -- 172.21.15.90:0/1820222606 >> 172.21.15.90:0/1820222606 conn(0x5617e691b000 msgr2=0x5617e67d3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.258 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.252+0000 7f32a77db280 1 -- 172.21.15.90:0/1820222606 shutdown_connections 2024-03-24T01:30:33.258 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.252+0000 7f32a77db280 1 -- 172.21.15.90:0/1820222606 wait complete. 2024-03-24T01:30:33.258 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.253+0000 7f32a77db280 1 -- 172.21.15.90:0/2275491120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617e5a52c00 msgr2=0x5617e5abc580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.258 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.253+0000 7f32a77db280 1 --2- 172.21.15.90:0/2275491120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617e5a52c00 0x5617e5abc580 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x5617e687d380 tx=0x5617e6828f00 comp rx=0 tx=0).stop 2024-03-24T01:30:33.258 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.253+0000 7f32a77db280 1 -- 172.21.15.90:0/2275491120 shutdown_connections 2024-03-24T01:30:33.258 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.253+0000 7f32a77db280 1 --2- 172.21.15.90:0/2275491120 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617e5a53c00 0x5617e5abd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.258 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.253+0000 7f32a77db280 1 --2- 172.21.15.90:0/2275491120 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5617e5a53800 0x5617e5abcb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.258 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.253+0000 7f32a77db280 1 --2- 172.21.15.90:0/2275491120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617e5a52c00 0x5617e5abc580 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.258 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.253+0000 7f32a77db280 1 -- 172.21.15.90:0/2275491120 >> 172.21.15.90:0/2275491120 conn(0x5617e5a53000 msgr2=0x5617e67d2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.258 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.253+0000 7f32a77db280 1 -- 172.21.15.90:0/2275491120 shutdown_connections 2024-03-24T01:30:33.258 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.253+0000 7f32a77db280 1 -- 172.21.15.90:0/2275491120 wait complete. 2024-03-24T01:30:33.259 INFO:tasks.workunit.client.0.smithi090.stderr:+ cephfs-journal-tool --rank=cephfs:0 journal export /tmp/journal.bin 2024-03-24T01:30:33.302 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.300+0000 7fdf0e266280 1 -- start start 2024-03-24T01:30:33.302 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.300+0000 7fdf0e266280 1 Processor -- start 2024-03-24T01:30:33.304 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.301+0000 7fdf0e266280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e06f696c00 0x55e06f700580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.304 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.301+0000 7fdf0e266280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e06f697800 0x55e06f700b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.304 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.301+0000 7fdf0e266280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e06f697c00 0x55e06f701080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.304 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.301+0000 7fdf0e266280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e070480180 con 0x55e06f697c00 2024-03-24T01:30:33.304 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.301+0000 7fdf0e266280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e070480300 con 0x55e06f696c00 2024-03-24T01:30:33.304 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.301+0000 7fdf0e266280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e070480480 con 0x55e06f697800 2024-03-24T01:30:33.304 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.301+0000 7fdf0ce00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e06f697c00 0x55e06f701080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.304 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.301+0000 7fdf0ce00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e06f697c00 0x55e06f701080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36298/0 (socket says 172.21.15.90:36298) 2024-03-24T01:30:33.304 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.301+0000 7fdf0ce00640 1 -- 172.21.15.90:0/3178584004 learned_addr learned my addr 172.21.15.90:0/3178584004 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.305 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.302+0000 7fdf0ce00640 1 -- 172.21.15.90:0/3178584004 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e06f697800 msgr2=0x55e06f700b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.305 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.302+0000 7fdf0ce00640 1 --2- 172.21.15.90:0/3178584004 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e06f697800 0x55e06f700b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.305 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.302+0000 7fdf0ce00640 1 -- 172.21.15.90:0/3178584004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e06f696c00 msgr2=0x55e06f700580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.305 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.302+0000 7fdf0ce00640 1 --2- 172.21.15.90:0/3178584004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e06f696c00 0x55e06f700580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.305 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.302+0000 7fdf0ce00640 1 -- 172.21.15.90:0/3178584004 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e06f6ca780 con 0x55e06f697c00 2024-03-24T01:30:33.305 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.302+0000 7fdf0ce00640 1 --2- 172.21.15.90:0/3178584004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e06f697c00 0x55e06f701080 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x55e0704bf440 tx=0x55e07046ede0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed66e6c647d9cce2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.305 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.302+0000 7fdf09200640 1 -- 172.21.15.90:0/3178584004 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e06f6bc680 con 0x55e06f697c00 2024-03-24T01:30:33.305 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.302+0000 7fdf09200640 1 -- 172.21.15.90:0/3178584004 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e06f6c2700 con 0x55e06f697c00 2024-03-24T01:30:33.305 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.302+0000 7fdf09200640 1 -- 172.21.15.90:0/3178584004 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e06f6bcb60 con 0x55e06f697c00 2024-03-24T01:30:33.305 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.302+0000 7fdf0e266280 1 -- 172.21.15.90:0/3178584004 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e06f6ca3c0 con 0x55e06f697c00 2024-03-24T01:30:33.305 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.302+0000 7fdf0e266280 1 -- 172.21.15.90:0/3178584004 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x55e06f6cb680 con 0x55e06f697c00 2024-03-24T01:30:33.305 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.302+0000 7fdf09200640 1 -- 172.21.15.90:0/3178584004 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x55e06f77f000 con 0x55e06f697c00 2024-03-24T01:30:33.305 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.303+0000 7fdf09200640 1 -- 172.21.15.90:0/3178584004 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 36) v1 ==== 3310+0+0 (secure 0 0 0) 0x55e06f6f1180 con 0x55e06f697c00 2024-03-24T01:30:33.308 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.303+0000 7fdf0e266280 1 Processor -- start 2024-03-24T01:30:33.308 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.304+0000 7fdf0e266280 1 -- start start 2024-03-24T01:30:33.308 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.304+0000 7fdf0e266280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e07055d400 0x55e06f701600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.308 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.304+0000 7fdf0e266280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e07055d800 0x55e06f701b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.308 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.304+0000 7fdf0e266280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e07055dc00 0x55e06f702100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.308 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.304+0000 7fdf0e266280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e070480600 con 0x55e07055d400 2024-03-24T01:30:33.308 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.304+0000 7fdf0e266280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e070480780 con 0x55e07055dc00 2024-03-24T01:30:33.309 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.304+0000 7fdf0e266280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e070480900 con 0x55e07055d800 2024-03-24T01:30:33.309 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.305+0000 7fdf0ba00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e07055d400 0x55e06f701600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.309 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.305+0000 7fdf0ba00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e07055d400 0x55e06f701600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36302/0 (socket says 172.21.15.90:36302) 2024-03-24T01:30:33.309 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.305+0000 7fdf0ba00640 1 -- 172.21.15.90:0/3052768718 learned_addr learned my addr 172.21.15.90:0/3052768718 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.309 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.305+0000 7fdf0ba00640 1 -- 172.21.15.90:0/3052768718 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e07055d800 msgr2=0x55e06f701b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.309 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.305+0000 7fdf0ba00640 1 --2- 172.21.15.90:0/3052768718 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e07055d800 0x55e06f701b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.309 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.305+0000 7fdf0ba00640 1 -- 172.21.15.90:0/3052768718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e07055dc00 msgr2=0x55e06f702100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.309 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.305+0000 7fdf0ba00640 1 --2- 172.21.15.90:0/3052768718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e07055dc00 0x55e06f702100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.309 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.305+0000 7fdf0ba00640 1 -- 172.21.15.90:0/3052768718 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e06f6cb4a0 con 0x55e07055d400 2024-03-24T01:30:33.309 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.305+0000 7fdf0ba00640 1 --2- 172.21.15.90:0/3052768718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e07055d400 0x55e06f701600 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x55e070574330 tx=0x55e07057cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e0a1231f6398afa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.309 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.306+0000 7fdf06a00640 1 -- 172.21.15.90:0/3052768718 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e06f6bc680 con 0x55e07055d400 2024-03-24T01:30:33.309 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.306+0000 7fdf06a00640 1 -- 172.21.15.90:0/3052768718 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e06f6c2fc0 con 0x55e07055d400 2024-03-24T01:30:33.309 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.306+0000 7fdf06a00640 1 -- 172.21.15.90:0/3052768718 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e06f6bcb60 con 0x55e07055d400 2024-03-24T01:30:33.309 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.306+0000 7fdf0e266280 1 -- 172.21.15.90:0/3052768718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e07055d400 msgr2=0x55e06f701600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.310 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.306+0000 7fdf0e266280 1 --2- 172.21.15.90:0/3052768718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e07055d400 0x55e06f701600 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x55e070574330 tx=0x55e07057cd20 comp rx=0 tx=0).stop 2024-03-24T01:30:33.310 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.306+0000 7fdf0e266280 1 -- 172.21.15.90:0/3052768718 shutdown_connections 2024-03-24T01:30:33.310 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.306+0000 7fdf0e266280 1 --2- 172.21.15.90:0/3052768718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e07055dc00 0x55e06f702100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.310 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.306+0000 7fdf0e266280 1 --2- 172.21.15.90:0/3052768718 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e07055d800 0x55e06f701b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.310 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.306+0000 7fdf0e266280 1 --2- 172.21.15.90:0/3052768718 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e07055d400 0x55e06f701600 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.310 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.306+0000 7fdf0e266280 1 -- 172.21.15.90:0/3052768718 >> 172.21.15.90:0/3052768718 conn(0x55e07055d000 msgr2=0x55e06f79f800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.310 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.306+0000 7fdf0e266280 1 -- 172.21.15.90:0/3052768718 shutdown_connections 2024-03-24T01:30:33.310 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.306+0000 7fdf0e266280 1 -- 172.21.15.90:0/3052768718 wait complete. 2024-03-24T01:30:33.310 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.307+0000 7fdf0e266280 1 Processor -- start 2024-03-24T01:30:33.310 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.307+0000 7fdf0e266280 1 -- start start 2024-03-24T01:30:33.312 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.308+0000 7fdf0e266280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e07055d400 0x55e06f703180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.312 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.308+0000 7fdf0e266280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e07055d800 0x55e06f702c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.312 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.308+0000 7fdf0e266280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e07055dc00 0x55e06f702680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.312 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.308+0000 7fdf0e266280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e070480900 con 0x55e07055d400 2024-03-24T01:30:33.312 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.308+0000 7fdf0e266280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e070480a80 con 0x55e07055d800 2024-03-24T01:30:33.312 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.308+0000 7fdf0e266280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e070480780 con 0x55e07055dc00 2024-03-24T01:30:33.312 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.308+0000 7fdf0ba00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e07055d400 0x55e06f703180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.312 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.308+0000 7fdf0ba00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e07055d400 0x55e06f703180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36310/0 (socket says 172.21.15.90:36310) 2024-03-24T01:30:33.312 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.308+0000 7fdf0ba00640 1 -- 172.21.15.90:0/3091624812 learned_addr learned my addr 172.21.15.90:0/3091624812 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.313 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.308+0000 7fdf0ba00640 1 -- 172.21.15.90:0/3091624812 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e07055dc00 msgr2=0x55e06f702680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.313 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.308+0000 7fdf0ba00640 1 --2- 172.21.15.90:0/3091624812 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e07055dc00 0x55e06f702680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.313 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.308+0000 7fdf0ba00640 1 -- 172.21.15.90:0/3091624812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e07055d800 msgr2=0x55e06f702c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.313 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.308+0000 7fdf0ba00640 1 --2- 172.21.15.90:0/3091624812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e07055d800 0x55e06f702c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.313 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.308+0000 7fdf0ba00640 1 -- 172.21.15.90:0/3091624812 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e06f6ca780 con 0x55e07055d400 2024-03-24T01:30:33.313 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.308+0000 7fdf0ba00640 1 --2- 172.21.15.90:0/3091624812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e07055d400 0x55e06f703180 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x55e0705755f0 tx=0x55e07057cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.313 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.309+0000 7fdf04c00640 1 -- 172.21.15.90:0/3091624812 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e06f6bc4e0 con 0x55e07055d400 2024-03-24T01:30:33.313 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.309+0000 7fdf04c00640 1 -- 172.21.15.90:0/3091624812 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e06f6c2e00 con 0x55e07055d400 2024-03-24T01:30:33.313 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.309+0000 7fdf04c00640 1 -- 172.21.15.90:0/3091624812 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e06f6bc680 con 0x55e07055d400 2024-03-24T01:30:33.313 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.309+0000 7fdf0e266280 1 -- 172.21.15.90:0/3091624812 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55e06f6ca3c0 con 0x55e07055d400 2024-03-24T01:30:33.313 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.309+0000 7fdf0e266280 1 -- 172.21.15.90:0/3091624812 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e06f6ca000 con 0x55e07055d400 2024-03-24T01:30:33.313 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.311+0000 7fdf04c00640 1 -- 172.21.15.90:0/3091624812 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x55e070594c00 con 0x55e07055d400 2024-03-24T01:30:33.315 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.311+0000 7fdf04c00640 1 --2- 172.21.15.90:0/3091624812 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55e0705db000 0x55e06f703700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.315 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.311+0000 7fdf04c00640 1 -- 172.21.15.90:0/3091624812 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x55e0705d0a00 con 0x55e07055d400 2024-03-24T01:30:33.316 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.311+0000 7fdf0e266280 1 --2- 172.21.15.90:0/3091624812 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55e0705db400 0x55e06f702100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.316 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.311+0000 7fdf0e266280 1 -- 172.21.15.90:0/3091624812 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:1 7.f 7:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e0705db800 con 0x55e0705db400 2024-03-24T01:30:33.316 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.312+0000 7fdf0ba00640 1 --2- 172.21.15.90:0/3091624812 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55e0705db400 0x55e06f702100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.316 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.312+0000 7fdf0c400640 1 --2- 172.21.15.90:0/3091624812 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55e0705db000 0x55e06f703700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.316 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.312+0000 7fdf0ba00640 1 --2- 172.21.15.90:0/3091624812 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55e0705db400 0x55e06f702100 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.316 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.313+0000 7fdf0ba00640 1 -- 172.21.15.90:0/3091624812 <== osd.0 v2:172.21.15.90:6800/307045587 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x55e07041a000 con 0x55e0705db400 2024-03-24T01:30:33.316 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.313+0000 7fdf0e266280 1 --2- 172.21.15.90:0/3091624812 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55e0705dbc00 0x55e06f701b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.316 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.313+0000 7fdf0e266280 1 -- 172.21.15.90:0/3091624812 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:2 7.14 7:292cf221:::200.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e06fa28000 con 0x55e0705dbc00 2024-03-24T01:30:33.316 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.313+0000 7fdf0ce00640 1 --2- 172.21.15.90:0/3091624812 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55e0705dbc00 0x55e06f701b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.316 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.313+0000 7fdf0c400640 1 --2- 172.21.15.90:0/3091624812 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55e0705db000 0x55e06f703700 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x55e06f816390 tx=0x55e06fa18120 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.316 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.313+0000 7fdf0ce00640 1 --2- 172.21.15.90:0/3091624812 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55e0705dbc00 0x55e06f701b80 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.316 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.314+0000 7fdf0ce00640 1 -- 172.21.15.90:0/3091624812 <== osd.3 v2:172.21.15.90:6808/2999931983 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv24 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x55e07041a500 con 0x55e0705dbc00 2024-03-24T01:30:33.316 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.314+0000 7fdf0e266280 1 -- 172.21.15.90:0/3091624812 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:3 7.14 7:2e2fa760:::200.00000001:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e06fa28400 con 0x55e0705dbc00 2024-03-24T01:30:33.316 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.314+0000 7fdf0ce00640 1 -- 172.21.15.90:0/3091624812 <== osd.3 v2:172.21.15.90:6808/2999931983 2 ==== osd_op_reply(3 200.00000001 [read 0~35827 out=35827b] v0'0 uv26 ondisk = 0) v8 ==== 156+0+35827 (crc 0 0 0) 0x55e07041a500 con 0x55e0705dbc00 2024-03-24T01:30:33.317 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.314+0000 7fdf0e266280 1 -- 172.21.15.90:0/3091624812 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:4 7.1b 7:ddb4e4d7:::200.00000002:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e06fa28800 con 0x55e0705db400 2024-03-24T01:30:33.317 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.315+0000 7fdf0ba00640 1 -- 172.21.15.90:0/3091624812 <== osd.0 v2:172.21.15.90:6800/307045587 2 ==== osd_op_reply(4 200.00000002 [read 0~2147483647] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x55e07041a000 con 0x55e0705db400 2024-03-24T01:30:33.317 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.316+0000 7fdf0e266280 1 -- start start 2024-03-24T01:30:33.317 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.316+0000 7fdf0e266280 1 Processor -- start 2024-03-24T01:30:33.319 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.317+0000 7fdf0e266280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e06fa29000 0x55e06f701600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.320 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.317+0000 7fdf0e266280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e06fa29400 0x55e06fa8c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.320 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.317+0000 7fdf0e266280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e06fa29800 0x55e06fa8c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.320 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.317+0000 7fdf0e266280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e070480f00 con 0x55e06fa29000 2024-03-24T01:30:33.320 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.317+0000 7fdf0e266280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e070480d80 con 0x55e06fa29800 2024-03-24T01:30:33.320 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.317+0000 7fdf0e266280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e070481e00 con 0x55e06fa29400 2024-03-24T01:30:33.320 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.317+0000 7fdf0ba00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e06fa29000 0x55e06f701600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.320 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.317+0000 7fdf0ba00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e06fa29000 0x55e06f701600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36312/0 (socket says 172.21.15.90:36312) 2024-03-24T01:30:33.320 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.317+0000 7fdf0ba00640 1 -- 172.21.15.90:0/2432189418 learned_addr learned my addr 172.21.15.90:0/2432189418 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.320 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.317+0000 7fdf0ba00640 1 -- 172.21.15.90:0/2432189418 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e06fa29400 msgr2=0x55e06fa8c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.320 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.317+0000 7fdf0ba00640 1 --2- 172.21.15.90:0/2432189418 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e06fa29400 0x55e06fa8c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.320 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.317+0000 7fdf0ba00640 1 -- 172.21.15.90:0/2432189418 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e06fa29800 msgr2=0x55e06fa8c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.320 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.317+0000 7fdf0ba00640 1 --2- 172.21.15.90:0/2432189418 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e06fa29800 0x55e06fa8c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.321 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.317+0000 7fdf0ba00640 1 -- 172.21.15.90:0/2432189418 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e06f6ca960 con 0x55e06fa29000 2024-03-24T01:30:33.321 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.318+0000 7fdf0ba00640 1 --2- 172.21.15.90:0/2432189418 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e06fa29000 0x55e06f701600 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x55e0706155f0 tx=0x55e07057d6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed13346d5b64f578 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.321 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.318+0000 7fdf01000640 1 -- 172.21.15.90:0/2432189418 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e06f9a4d00 con 0x55e06fa29000 2024-03-24T01:30:33.321 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.318+0000 7fdf01000640 1 -- 172.21.15.90:0/2432189418 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e06f6c2c40 con 0x55e06fa29000 2024-03-24T01:30:33.321 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.318+0000 7fdf01000640 1 -- 172.21.15.90:0/2432189418 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e06f9a4ea0 con 0x55e06fa29000 2024-03-24T01:30:33.321 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.318+0000 7fdf0e266280 1 -- 172.21.15.90:0/2432189418 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e06f6cb4a0 con 0x55e06fa29000 2024-03-24T01:30:33.321 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.318+0000 7fdf0e266280 1 -- 172.21.15.90:0/2432189418 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x55e06f6ca780 con 0x55e06fa29000 2024-03-24T01:30:33.321 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.318+0000 7fdf01000640 1 -- 172.21.15.90:0/2432189418 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x55e06fa94600 con 0x55e06fa29000 2024-03-24T01:30:33.321 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.318+0000 7fdf01000640 1 -- 172.21.15.90:0/2432189418 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 36) v1 ==== 3310+0+0 (secure 0 0 0) 0x55e06f6f1500 con 0x55e06fa29000 2024-03-24T01:30:33.321 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.318+0000 7fdf0e266280 1 --2- 172.21.15.90:0/2432189418 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55e06fa29c00 0x55e06fa8cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.321 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.318+0000 7fdf0e266280 1 -- 172.21.15.90:0/2432189418 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:1 7.f 7:f1f69726:::400.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e06fa97000 con 0x55e06fa29c00 2024-03-24T01:30:33.321 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.319+0000 7fdf0ce00640 1 --2- 172.21.15.90:0/2432189418 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55e06fa29c00 0x55e06fa8cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.323 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.319+0000 7fdf0ce00640 1 --2- 172.21.15.90:0/2432189418 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55e06fa29c00 0x55e06fa8cb00 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=5e24b3497e5bd6e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.323 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.319+0000 7fdf0ce00640 1 -- 172.21.15.90:0/2432189418 <== osd.0 v2:172.21.15.90:6800/307045587 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x55e07041a500 con 0x55e06fa29c00 2024-03-24T01:30:33.323 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.319+0000 7fdf0e266280 1 --2- 172.21.15.90:0/2432189418 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55e06fa97400 0x55e06fa8d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.323 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.319+0000 7fdf0e266280 1 -- 172.21.15.90:0/2432189418 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:2 7.14 7:292cf221:::200.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e06fa97800 con 0x55e06fa97400 2024-03-24T01:30:33.323 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.320+0000 7fdf0c400640 1 --2- 172.21.15.90:0/2432189418 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55e06fa97400 0x55e06fa8d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.323 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.320+0000 7fdf0c400640 1 --2- 172.21.15.90:0/2432189418 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55e06fa97400 0x55e06fa8d080 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=470734eab82a9abd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.323 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.320+0000 7fdf0c400640 1 -- 172.21.15.90:0/2432189418 <== osd.3 v2:172.21.15.90:6808/2999931983 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv24 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x55e06fab6000 con 0x55e06fa97400 2024-03-24T01:30:33.323 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.320+0000 7fdeff200640 1 -- 172.21.15.90:0/2432189418 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:3 7.14 7:2e2fa760:::200.00000001:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e06fab8000 con 0x55e06fa97400 2024-03-24T01:30:33.323 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.320+0000 7fdeff200640 1 -- 172.21.15.90:0/2432189418 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:4 7.1b 7:ddb4e4d7:::200.00000002:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e06fab8400 con 0x55e06fa29c00 2024-03-24T01:30:33.323 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.321+0000 7fdf0ce00640 1 -- 172.21.15.90:0/2432189418 <== osd.0 v2:172.21.15.90:6800/307045587 2 ==== osd_op_reply(4 200.00000002 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x55e07041a500 con 0x55e06fa29c00 2024-03-24T01:30:33.323 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.321+0000 7fdf0c400640 1 -- 172.21.15.90:0/2432189418 <== osd.3 v2:172.21.15.90:6808/2999931983 2 ==== osd_op_reply(3 200.00000001 [stat out=16b] v0'0 uv26 ondisk = 0) v8 ==== 156+0+16 (crc 0 0 0) 0x55e06fab6000 con 0x55e06fa97400 2024-03-24T01:30:33.324 INFO:tasks.workunit.client.0.smithi090.stdout:journal is 4194304~35827 2024-03-24T01:30:33.324 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.321+0000 7fdf0e266280 1 -- 172.21.15.90:0/2432189418 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:5 7.14 7:2e2fa760:::200.00000001:head [read 0~35827] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e06fa97c00 con 0x55e06fa97400 2024-03-24T01:30:33.324 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.321+0000 7fdf0c400640 1 -- 172.21.15.90:0/2432189418 <== osd.3 v2:172.21.15.90:6808/2999931983 3 ==== osd_op_reply(5 200.00000001 [read 0~35827 out=35827b] v0'0 uv26 ondisk = 0) v8 ==== 156+0+35827 (crc 0 0 0) 0x55e06fab6000 con 0x55e06fa97400 2024-03-24T01:30:33.325 INFO:tasks.workunit.client.0.smithi090.stdout:wrote 35827 bytes at offset 4194304 to /tmp/journal.bin 2024-03-24T01:30:33.325 INFO:tasks.workunit.client.0.smithi090.stdout:NOTE: this is a _sparse_ file; you can 2024-03-24T01:30:33.325 INFO:tasks.workunit.client.0.smithi090.stdout: $ tar cSzf /tmp/journal.bin.tgz /tmp/journal.bin 2024-03-24T01:30:33.325 INFO:tasks.workunit.client.0.smithi090.stdout: to efficiently compress it while preserving sparseness. 2024-03-24T01:30:33.327 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.322+0000 7fdf0e266280 1 -- 172.21.15.90:0/2432189418 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55e06fa29c00 msgr2=0x55e06fa8cb00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.327 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.322+0000 7fdf0e266280 1 --2- 172.21.15.90:0/2432189418 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55e06fa29c00 0x55e06fa8cb00 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.327 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.322+0000 7fdf0e266280 1 -- 172.21.15.90:0/2432189418 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55e06fa97400 msgr2=0x55e06fa8d080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.327 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.322+0000 7fdf0e266280 1 --2- 172.21.15.90:0/2432189418 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55e06fa97400 0x55e06fa8d080 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.327 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.322+0000 7fdf0e266280 1 -- 172.21.15.90:0/2432189418 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e06fa29000 msgr2=0x55e06f701600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.327 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.322+0000 7fdf0e266280 1 --2- 172.21.15.90:0/2432189418 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e06fa29000 0x55e06f701600 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x55e0706155f0 tx=0x55e07057d6e0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.327 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.322+0000 7fdf0c400640 1 -- 172.21.15.90:0/2432189418 reap_dead start 2024-03-24T01:30:33.328 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.322+0000 7fdf0e266280 1 -- 172.21.15.90:0/2432189418 shutdown_connections 2024-03-24T01:30:33.328 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.322+0000 7fdf0e266280 1 -- 172.21.15.90:0/2432189418 >> 172.21.15.90:0/2432189418 conn(0x55e06fa28c00 msgr2=0x55e06f7a1800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.328 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.322+0000 7fdf0e266280 1 -- 172.21.15.90:0/2432189418 shutdown_connections 2024-03-24T01:30:33.328 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.322+0000 7fdf0e266280 1 -- 172.21.15.90:0/2432189418 wait complete. 2024-03-24T01:30:33.328 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.322+0000 7fdf0e266280 1 -- 172.21.15.90:0/3091624812 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55e0705db400 msgr2=0x55e06f702100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.328 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.322+0000 7fdf0e266280 1 --2- 172.21.15.90:0/3091624812 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55e0705db400 0x55e06f702100 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.328 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.322+0000 7fdf0e266280 1 -- 172.21.15.90:0/3091624812 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55e0705dbc00 msgr2=0x55e06f701b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.328 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.322+0000 7fdf0e266280 1 --2- 172.21.15.90:0/3091624812 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55e0705dbc00 0x55e06f701b80 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.328 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.322+0000 7fdf0e266280 1 -- 172.21.15.90:0/3091624812 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55e0705db000 msgr2=0x55e06f703700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.328 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.322+0000 7fdf0e266280 1 --2- 172.21.15.90:0/3091624812 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55e0705db000 0x55e06f703700 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x55e06f816390 tx=0x55e06fa18120 comp rx=0 tx=0).stop 2024-03-24T01:30:33.328 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.322+0000 7fdf0e266280 1 -- 172.21.15.90:0/3091624812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e07055d400 msgr2=0x55e06f703180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.328 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.322+0000 7fdf0e266280 1 --2- 172.21.15.90:0/3091624812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e07055d400 0x55e06f703180 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x55e0705755f0 tx=0x55e07057cc00 comp rx=0 tx=0).stop 2024-03-24T01:30:33.329 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.322+0000 7fdf0c400640 1 -- 172.21.15.90:0/3091624812 reap_dead start 2024-03-24T01:30:33.329 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.322+0000 7fdf0e266280 1 -- 172.21.15.90:0/3091624812 shutdown_connections 2024-03-24T01:30:33.329 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.322+0000 7fdf0e266280 1 --2- 172.21.15.90:0/3091624812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e07055d400 0x55e06f703180 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.329 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.322+0000 7fdf0e266280 1 -- 172.21.15.90:0/3091624812 >> 172.21.15.90:0/3091624812 conn(0x55e07055d000 msgr2=0x55e06f79f800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.329 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.323+0000 7fdf0e266280 1 -- 172.21.15.90:0/3091624812 shutdown_connections 2024-03-24T01:30:33.329 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.323+0000 7fdf0e266280 1 -- 172.21.15.90:0/3091624812 wait complete. 2024-03-24T01:30:33.329 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.323+0000 7fdf0e266280 1 -- 172.21.15.90:0/3178584004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e06f697c00 msgr2=0x55e06f701080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.329 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.323+0000 7fdf0e266280 1 --2- 172.21.15.90:0/3178584004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e06f697c00 0x55e06f701080 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x55e0704bf440 tx=0x55e07046ede0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.329 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.323+0000 7fdf0e266280 1 -- 172.21.15.90:0/3178584004 shutdown_connections 2024-03-24T01:30:33.329 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.323+0000 7fdf0e266280 1 --2- 172.21.15.90:0/3178584004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e06f697c00 0x55e06f701080 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.329 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.323+0000 7fdf0e266280 1 --2- 172.21.15.90:0/3178584004 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e06f697800 0x55e06f700b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.329 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.323+0000 7fdf0e266280 1 --2- 172.21.15.90:0/3178584004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e06f696c00 0x55e06f700580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.329 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.323+0000 7fdf0e266280 1 -- 172.21.15.90:0/3178584004 >> 172.21.15.90:0/3178584004 conn(0x55e06f697000 msgr2=0x55e06f79f000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.329 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.323+0000 7fdf0e266280 1 -- 172.21.15.90:0/3178584004 shutdown_connections 2024-03-24T01:30:33.330 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.323+0000 7fdf0e266280 1 -- 172.21.15.90:0/3178584004 wait complete. 2024-03-24T01:30:33.331 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' '!' -s /tmp/journal.bin ']' 2024-03-24T01:30:33.331 INFO:tasks.workunit.client.0.smithi090.stderr:+ cephfs-journal-tool --rank=cephfs:0 journal reset 2024-03-24T01:30:33.371 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.370+0000 7f130c3b8280 1 -- start start 2024-03-24T01:30:33.371 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.370+0000 7f130c3b8280 1 Processor -- start 2024-03-24T01:30:33.373 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.370+0000 7f130c3b8280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a74fe4c00 0x563a7504e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.373 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.371+0000 7f130c3b8280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a74fe5800 0x563a7504eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.373 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.371+0000 7f130c3b8280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563a74fe5c00 0x563a7504f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.373 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.371+0000 7f130c3b8280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a75db8180 con 0x563a74fe5800 2024-03-24T01:30:33.374 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.371+0000 7f130c3b8280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a75db8300 con 0x563a74fe4c00 2024-03-24T01:30:33.374 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.371+0000 7f130c3b8280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x563a75db8480 con 0x563a74fe5c00 2024-03-24T01:30:33.374 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.371+0000 7f1309a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a74fe5800 0x563a7504eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.374 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.371+0000 7f1309a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a74fe5800 0x563a7504eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36334/0 (socket says 172.21.15.90:36334) 2024-03-24T01:30:33.374 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.371+0000 7f1309a00640 1 -- 172.21.15.90:0/3892851206 learned_addr learned my addr 172.21.15.90:0/3892851206 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.374 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.371+0000 7f130a400640 1 --2- 172.21.15.90:0/3892851206 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a74fe4c00 0x563a7504e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.374 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.371+0000 7f1309a00640 1 -- 172.21.15.90:0/3892851206 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563a74fe5c00 msgr2=0x563a7504f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.374 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.371+0000 7f130ae00640 1 --2- 172.21.15.90:0/3892851206 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563a74fe5c00 0x563a7504f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.374 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.371+0000 7f1309a00640 1 --2- 172.21.15.90:0/3892851206 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563a74fe5c00 0x563a7504f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.374 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.371+0000 7f1309a00640 1 -- 172.21.15.90:0/3892851206 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a74fe4c00 msgr2=0x563a7504e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.374 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.371+0000 7f1309a00640 1 --2- 172.21.15.90:0/3892851206 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a74fe4c00 0x563a7504e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.374 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.371+0000 7f1309a00640 1 -- 172.21.15.90:0/3892851206 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563a75018780 con 0x563a74fe5800 2024-03-24T01:30:33.374 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.371+0000 7f130a400640 1 --2- 172.21.15.90:0/3892851206 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a74fe4c00 0x563a7504e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:30:33.375 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.372+0000 7f1309a00640 1 --2- 172.21.15.90:0/3892851206 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a74fe5800 0x563a7504eb00 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x563a75dfd290 tx=0x563a750cce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dafd9a991384fd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.375 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.372+0000 7f1307200640 1 -- 172.21.15.90:0/3892851206 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x563a7500a9c0 con 0x563a74fe5800 2024-03-24T01:30:33.375 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.372+0000 7f1307200640 1 -- 172.21.15.90:0/3892851206 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563a75010fc0 con 0x563a74fe5800 2024-03-24T01:30:33.375 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.372+0000 7f1307200640 1 -- 172.21.15.90:0/3892851206 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x563a7500a680 con 0x563a74fe5800 2024-03-24T01:30:33.375 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.372+0000 7f130c3b8280 1 -- 172.21.15.90:0/3892851206 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563a750183c0 con 0x563a74fe5800 2024-03-24T01:30:33.375 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.372+0000 7f130c3b8280 1 -- 172.21.15.90:0/3892851206 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x563a75019680 con 0x563a74fe5800 2024-03-24T01:30:33.375 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.372+0000 7f1307200640 1 -- 172.21.15.90:0/3892851206 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x563a75ea6200 con 0x563a74fe5800 2024-03-24T01:30:33.375 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.372+0000 7f1307200640 1 -- 172.21.15.90:0/3892851206 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 36) v1 ==== 3310+0+0 (secure 0 0 0) 0x563a7503f180 con 0x563a74fe5800 2024-03-24T01:30:33.378 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.373+0000 7f130c3b8280 1 Processor -- start 2024-03-24T01:30:33.378 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.373+0000 7f130c3b8280 1 -- start start 2024-03-24T01:30:33.378 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.374+0000 7f130c3b8280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a75f04000 0x563a7504f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.378 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.374+0000 7f130c3b8280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563a75f04400 0x563a7504fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.378 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.374+0000 7f130c3b8280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a75f04800 0x563a75050100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.378 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.374+0000 7f130c3b8280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a75db8600 con 0x563a75f04000 2024-03-24T01:30:33.378 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.374+0000 7f130c3b8280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a75db8780 con 0x563a75f04800 2024-03-24T01:30:33.378 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.374+0000 7f130c3b8280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x563a75db8900 con 0x563a75f04400 2024-03-24T01:30:33.378 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.374+0000 7f130ae00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a75f04000 0x563a7504f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.378 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.374+0000 7f130ae00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a75f04000 0x563a7504f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36344/0 (socket says 172.21.15.90:36344) 2024-03-24T01:30:33.379 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.374+0000 7f130ae00640 1 -- 172.21.15.90:0/473006215 learned_addr learned my addr 172.21.15.90:0/473006215 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.379 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.375+0000 7f130ae00640 1 -- 172.21.15.90:0/473006215 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563a75f04400 msgr2=0x563a7504fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.379 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.375+0000 7f130ae00640 1 --2- 172.21.15.90:0/473006215 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563a75f04400 0x563a7504fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.379 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.375+0000 7f130ae00640 1 -- 172.21.15.90:0/473006215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a75f04800 msgr2=0x563a75050100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.379 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.375+0000 7f130ae00640 1 --2- 172.21.15.90:0/473006215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a75f04800 0x563a75050100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.379 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.375+0000 7f130ae00640 1 -- 172.21.15.90:0/473006215 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563a750194a0 con 0x563a75f04000 2024-03-24T01:30:33.379 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.375+0000 7f130ae00640 1 --2- 172.21.15.90:0/473006215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a75f04000 0x563a7504f600 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x563a75ddc8a0 tx=0x563a75eb4cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1d31c666e916cb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.379 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.375+0000 7f1304a00640 1 -- 172.21.15.90:0/473006215 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x563a7500a9c0 con 0x563a75f04000 2024-03-24T01:30:33.379 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.375+0000 7f1304a00640 1 -- 172.21.15.90:0/473006215 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563a75010a80 con 0x563a75f04000 2024-03-24T01:30:33.379 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.375+0000 7f1304a00640 1 -- 172.21.15.90:0/473006215 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x563a7500a680 con 0x563a75f04000 2024-03-24T01:30:33.379 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.376+0000 7f130c3b8280 1 -- 172.21.15.90:0/473006215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a75f04000 msgr2=0x563a7504f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.379 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.376+0000 7f130c3b8280 1 --2- 172.21.15.90:0/473006215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a75f04000 0x563a7504f600 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x563a75ddc8a0 tx=0x563a75eb4cc0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.379 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.376+0000 7f130c3b8280 1 -- 172.21.15.90:0/473006215 shutdown_connections 2024-03-24T01:30:33.379 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.376+0000 7f130c3b8280 1 --2- 172.21.15.90:0/473006215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a75f04800 0x563a75050100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.380 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.376+0000 7f130c3b8280 1 --2- 172.21.15.90:0/473006215 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563a75f04400 0x563a7504fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.380 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.376+0000 7f130c3b8280 1 --2- 172.21.15.90:0/473006215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a75f04000 0x563a7504f600 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.380 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.376+0000 7f130c3b8280 1 -- 172.21.15.90:0/473006215 >> 172.21.15.90:0/473006215 conn(0x563a75dbdc00 msgr2=0x563a75d65000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.380 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.376+0000 7f130c3b8280 1 -- 172.21.15.90:0/473006215 shutdown_connections 2024-03-24T01:30:33.380 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.376+0000 7f130c3b8280 1 -- 172.21.15.90:0/473006215 wait complete. 2024-03-24T01:30:33.380 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.376+0000 7f130c3b8280 1 Processor -- start 2024-03-24T01:30:33.380 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.376+0000 7f130c3b8280 1 -- start start 2024-03-24T01:30:33.382 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.377+0000 7f130c3b8280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563a75f04000 0x563a75051180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.382 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.377+0000 7f130c3b8280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a75f04400 0x563a75050c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.382 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.377+0000 7f130c3b8280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a75f04800 0x563a75050680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.382 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.377+0000 7f130c3b8280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a75db9980 con 0x563a75f04800 2024-03-24T01:30:33.382 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.377+0000 7f130c3b8280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a75db8300 con 0x563a75f04400 2024-03-24T01:30:33.382 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.377+0000 7f130c3b8280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x563a75db8c00 con 0x563a75f04000 2024-03-24T01:30:33.382 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.377+0000 7f1309a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a75f04800 0x563a75050680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.382 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.377+0000 7f1309a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a75f04800 0x563a75050680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36346/0 (socket says 172.21.15.90:36346) 2024-03-24T01:30:33.382 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.377+0000 7f1309a00640 1 -- 172.21.15.90:0/3666402165 learned_addr learned my addr 172.21.15.90:0/3666402165 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.382 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.378+0000 7f1309a00640 1 -- 172.21.15.90:0/3666402165 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563a75f04000 msgr2=0x563a75051180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.383 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.378+0000 7f1309a00640 1 --2- 172.21.15.90:0/3666402165 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563a75f04000 0x563a75051180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.383 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.378+0000 7f1309a00640 1 -- 172.21.15.90:0/3666402165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a75f04400 msgr2=0x563a75050c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.383 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.378+0000 7f1309a00640 1 --2- 172.21.15.90:0/3666402165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a75f04400 0x563a75050c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.383 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.378+0000 7f1309a00640 1 -- 172.21.15.90:0/3666402165 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563a75018000 con 0x563a75f04800 2024-03-24T01:30:33.383 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.378+0000 7f1309a00640 1 --2- 172.21.15.90:0/3666402165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a75f04800 0x563a75050680 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x563a75dfd710 tx=0x563a750ccf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.383 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.378+0000 7f1302c00640 1 -- 172.21.15.90:0/3666402165 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x563a7500ab60 con 0x563a75f04800 2024-03-24T01:30:33.383 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.378+0000 7f1302c00640 1 -- 172.21.15.90:0/3666402165 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563a75010e00 con 0x563a75f04800 2024-03-24T01:30:33.383 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.378+0000 7f130c3b8280 1 -- 172.21.15.90:0/3666402165 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x563a75018960 con 0x563a75f04800 2024-03-24T01:30:33.383 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.378+0000 7f1302c00640 1 -- 172.21.15.90:0/3666402165 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x563a7500a820 con 0x563a75f04800 2024-03-24T01:30:33.383 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.378+0000 7f130c3b8280 1 -- 172.21.15.90:0/3666402165 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563a75019860 con 0x563a75f04800 2024-03-24T01:30:33.383 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.380+0000 7f1302c00640 1 -- 172.21.15.90:0/3666402165 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x563a75f16c00 con 0x563a75f04800 2024-03-24T01:30:33.383 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.380+0000 7f1302c00640 1 --2- 172.21.15.90:0/3666402165 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x563a75f1f400 0x563a75051700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.383 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.381+0000 7f1302c00640 1 -- 172.21.15.90:0/3666402165 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x563a75f0b200 con 0x563a75f04800 2024-03-24T01:30:33.386 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.381+0000 7f130ae00640 1 --2- 172.21.15.90:0/3666402165 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x563a75f1f400 0x563a75051700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.386 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.382+0000 7f130c3b8280 1 -- start start 2024-03-24T01:30:33.386 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.382+0000 7f130c3b8280 1 Processor -- start 2024-03-24T01:30:33.386 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.382+0000 7f130ae00640 1 --2- 172.21.15.90:0/3666402165 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x563a75f1f400 0x563a75051700 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x563a75ddc8a0 tx=0x563a75eb4c00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.386 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.383+0000 7f130c3b8280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a75f05000 0x563a75050100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.386 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.383+0000 7f130c3b8280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a75f1f800 0x563a7504fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.386 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.383+0000 7f130c3b8280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563a75f1fc00 0x563a7504f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.386 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.383+0000 7f130c3b8280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a75db9680 con 0x563a75f1f800 2024-03-24T01:30:33.386 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.383+0000 7f130c3b8280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a75db8600 con 0x563a75f05000 2024-03-24T01:30:33.386 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.383+0000 7f130c3b8280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x563a75db8780 con 0x563a75f1fc00 2024-03-24T01:30:33.387 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.383+0000 7f130a400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a75f1f800 0x563a7504fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.387 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.383+0000 7f130a400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a75f1f800 0x563a7504fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36358/0 (socket says 172.21.15.90:36358) 2024-03-24T01:30:33.387 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.383+0000 7f130a400640 1 -- 172.21.15.90:0/1867739188 learned_addr learned my addr 172.21.15.90:0/1867739188 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.387 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.383+0000 7f130a400640 1 -- 172.21.15.90:0/1867739188 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563a75f1fc00 msgr2=0x563a7504f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.387 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.383+0000 7f130a400640 1 --2- 172.21.15.90:0/1867739188 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563a75f1fc00 0x563a7504f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.387 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.383+0000 7f130a400640 1 -- 172.21.15.90:0/1867739188 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a75f05000 msgr2=0x563a75050100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.387 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.383+0000 7f130a400640 1 --2- 172.21.15.90:0/1867739188 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a75f05000 0x563a75050100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.387 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.383+0000 7f130a400640 1 -- 172.21.15.90:0/1867739188 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563a75019a40 con 0x563a75f1f800 2024-03-24T01:30:33.387 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.384+0000 7f130a400640 1 --2- 172.21.15.90:0/1867739188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a75f1f800 0x563a7504fb80 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x563a752ff710 tx=0x563a753a80c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db9ac0022b0039f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.387 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.384+0000 7f12ff000640 1 -- 172.21.15.90:0/1867739188 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x563a75f4ba00 con 0x563a75f1f800 2024-03-24T01:30:33.387 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.384+0000 7f12ff000640 1 -- 172.21.15.90:0/1867739188 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563a75010c40 con 0x563a75f1f800 2024-03-24T01:30:33.387 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.384+0000 7f12ff000640 1 -- 172.21.15.90:0/1867739188 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x563a75f4b860 con 0x563a75f1f800 2024-03-24T01:30:33.387 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.384+0000 7f130c3b8280 1 -- 172.21.15.90:0/1867739188 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563a75018780 con 0x563a75f1f800 2024-03-24T01:30:33.388 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.384+0000 7f130c3b8280 1 -- 172.21.15.90:0/1867739188 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x563a750183c0 con 0x563a75f1f800 2024-03-24T01:30:33.388 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.384+0000 7f12ff000640 1 -- 172.21.15.90:0/1867739188 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x563a753ac600 con 0x563a75f1f800 2024-03-24T01:30:33.388 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.384+0000 7f12ff000640 1 -- 172.21.15.90:0/1867739188 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 36) v1 ==== 3310+0+0 (secure 0 0 0) 0x563a7503f500 con 0x563a75f1f800 2024-03-24T01:30:33.388 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.384+0000 7f130c3b8280 1 --2- 172.21.15.90:0/1867739188 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x563a753abc00 0x563a753c2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.388 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.384+0000 7f130c3b8280 1 -- 172.21.15.90:0/1867739188 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:1 7.f 7:f1f69726:::400.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x563a753c6000 con 0x563a753abc00 2024-03-24T01:30:33.388 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.385+0000 7f130ae00640 1 --2- 172.21.15.90:0/1867739188 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x563a753abc00 0x563a753c2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.389 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.385+0000 7f130ae00640 1 --2- 172.21.15.90:0/1867739188 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x563a753abc00 0x563a753c2000 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=8fa4486b168e0ed0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.389 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.385+0000 7f130ae00640 1 -- 172.21.15.90:0/1867739188 <== osd.0 v2:172.21.15.90:6800/307045587 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x563a75d68000 con 0x563a753abc00 2024-03-24T01:30:33.389 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.386+0000 7f130c3b8280 1 --2- 172.21.15.90:0/1867739188 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x563a753c6400 0x563a753c2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.389 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.386+0000 7f130c3b8280 1 -- 172.21.15.90:0/1867739188 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:2 7.14 7:292cf221:::200.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x563a753c6800 con 0x563a753c6400 2024-03-24T01:30:33.390 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.386+0000 7f1309a00640 1 --2- 172.21.15.90:0/1867739188 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x563a753c6400 0x563a753c2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.390 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.386+0000 7f1309a00640 1 --2- 172.21.15.90:0/1867739188 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x563a753c6400 0x563a753c2580 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=6f4a025a8a9ab80f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.390 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.387+0000 7f1309a00640 1 -- 172.21.15.90:0/1867739188 <== osd.3 v2:172.21.15.90:6808/2999931983 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv24 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x563a75d68500 con 0x563a753c6400 2024-03-24T01:30:33.390 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.387+0000 7f12fd200640 1 -- 172.21.15.90:0/1867739188 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:3 7.14 7:2e2fa760:::200.00000001:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56) v9 -- 0x563a753d4000 con 0x563a753c6400 2024-03-24T01:30:33.390 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.387+0000 7f12fd200640 1 -- 172.21.15.90:0/1867739188 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:4 7.1b 7:ddb4e4d7:::200.00000002:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56) v9 -- 0x563a753d4400 con 0x563a753abc00 2024-03-24T01:30:33.390 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.387+0000 7f1309a00640 1 -- 172.21.15.90:0/1867739188 <== osd.3 v2:172.21.15.90:6808/2999931983 2 ==== osd_op_reply(3 200.00000001 [stat out=16b] v0'0 uv26 ondisk = 0) v8 ==== 156+0+16 (crc 0 0 0) 0x563a75d68500 con 0x563a753c6400 2024-03-24T01:30:33.390 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.387+0000 7f130ae00640 1 -- 172.21.15.90:0/1867739188 <== osd.0 v2:172.21.15.90:6800/307045587 2 ==== osd_op_reply(4 200.00000002 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x563a75d68000 con 0x563a753abc00 2024-03-24T01:30:33.390 INFO:tasks.workunit.client.0.smithi090.stdout:old journal was 4194304~35827 2024-03-24T01:30:33.390 INFO:tasks.workunit.client.0.smithi090.stdout:new journal start will be 8388608 (4158477 bytes past old end) 2024-03-24T01:30:33.390 INFO:tasks.workunit.client.0.smithi090.stdout:writing journal head 2024-03-24T01:30:33.391 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.387+0000 7f130c3b8280 1 -- 172.21.15.90:0/1867739188 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:5 7.14 7:292cf221:::200.00000000:head [writefull 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x563a753c6c00 con 0x563a753c6400 2024-03-24T01:30:33.391 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.389+0000 7f1309a00640 1 -- 172.21.15.90:0/1867739188 <== osd.3 v2:172.21.15.90:6808/2999931983 3 ==== osd_op_reply(5 200.00000000 [writefull 0~90] v56'27 uv27 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x563a75d68500 con 0x563a753c6400 2024-03-24T01:30:33.391 INFO:tasks.workunit.client.0.smithi090.stdout:writing EResetJournal entry 2024-03-24T01:30:33.393 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.389+0000 7f130c3b8280 1 -- 172.21.15.90:0/1867739188 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:6 7.1b 7:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x563a753c7000 con 0x563a753abc00 2024-03-24T01:30:33.393 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.389+0000 7f130c3b8280 1 --2- 172.21.15.90:0/1867739188 >> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] conn(0x563a753c7400 0x563a753c2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.393 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.389+0000 7f130c3b8280 1 -- 172.21.15.90:0/1867739188 --> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] -- osd_op(unknown.0.0:7 7.b 7:d5c7a900:::200.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x563a753c7800 con 0x563a753c7400 2024-03-24T01:30:33.393 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.389+0000 7f130c3b8280 1 --2- 172.21.15.90:0/1867739188 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x563a753c7c00 0x563a753c3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.393 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.389+0000 7f130c3b8280 1 -- 172.21.15.90:0/1867739188 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:8 7.1f 7:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x563a753d4400 con 0x563a753c7c00 2024-03-24T01:30:33.393 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.389+0000 7f130c3b8280 1 --2- 172.21.15.90:0/1867739188 >> [v2:172.21.15.120:6800/2150486081,v1:172.21.15.120:6801/2150486081] conn(0x563a753d4c00 0x563a753c3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.393 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.389+0000 7f130c3b8280 1 -- 172.21.15.90:0/1867739188 --> [v2:172.21.15.120:6800/2150486081,v1:172.21.15.120:6801/2150486081] -- osd_op(unknown.0.0:9 7.1a 7:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x563a753d5000 con 0x563a753d4c00 2024-03-24T01:30:33.393 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.389+0000 7f130c3b8280 1 -- 172.21.15.90:0/1867739188 --> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] -- osd_op(unknown.0.0:10 7.1e 7:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x563a753d5400 con 0x563a753c7400 2024-03-24T01:30:33.393 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.389+0000 7f130c3b8280 1 -- 172.21.15.90:0/1867739188 --> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] -- osd_op(unknown.0.0:11 7.1d 7:bec1251c:::200.00000007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x563a753d5800 con 0x563a753c7400 2024-03-24T01:30:33.393 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.389+0000 7f130ae00640 1 --2- 172.21.15.90:0/1867739188 >> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] conn(0x563a753c7400 0x563a753c2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.393 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.390+0000 7f130a400640 1 --2- 172.21.15.90:0/1867739188 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x563a753c7c00 0x563a753c3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.394 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.390+0000 7f1309a00640 1 --2- 172.21.15.90:0/1867739188 >> [v2:172.21.15.120:6800/2150486081,v1:172.21.15.120:6801/2150486081] conn(0x563a753d4c00 0x563a753c3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.394 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.390+0000 7f130ae00640 1 --2- 172.21.15.90:0/1867739188 >> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] conn(0x563a753c7400 0x563a753c2b00 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=24e156e9789d4520 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.394 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.390+0000 7f130ae00640 1 -- 172.21.15.90:0/1867739188 <== osd.0 v2:172.21.15.90:6800/307045587 3 ==== osd_op_reply(6 200.00000002 [delete] v56'3 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x563a75d68000 con 0x563a753abc00 2024-03-24T01:30:33.394 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.391+0000 7f130a400640 1 --2- 172.21.15.90:0/1867739188 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x563a753c7c00 0x563a753c3080 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=da7d802660de2f3c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.394 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.391+0000 7f1309a00640 1 --2- 172.21.15.90:0/1867739188 >> [v2:172.21.15.120:6800/2150486081,v1:172.21.15.120:6801/2150486081] conn(0x563a753d4c00 0x563a753c3600 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=b3e4b0ad8686a05e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.394 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.392+0000 7f130ae00640 1 -- 172.21.15.90:0/1867739188 <== osd.1 v2:172.21.15.90:6816/133441085 1 ==== osd_op_reply(10 200.00000006 [delete] v56'2 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x563a75d68000 con 0x563a753c7400 2024-03-24T01:30:33.394 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.393+0000 7f130ae00640 1 -- 172.21.15.90:0/1867739188 <== osd.1 v2:172.21.15.90:6816/133441085 2 ==== osd_op_reply(11 200.00000007 [delete] v56'4 uv2 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x563a75d68000 con 0x563a753c7400 2024-03-24T01:30:33.395 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.394+0000 7f130ae00640 1 -- 172.21.15.90:0/1867739188 <== osd.1 v2:172.21.15.90:6816/133441085 3 ==== osd_op_reply(7 200.00000003 [delete] v56'4 uv1 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x563a75d68000 con 0x563a753c7400 2024-03-24T01:30:33.395 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.394+0000 7f12fd200640 1 -- 172.21.15.90:0/1867739188 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:12 7.1b 7:ddb4e4d7:::200.00000002:head [write 0~48 in=48b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x563a753d4800 con 0x563a753abc00 2024-03-24T01:30:33.396 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.395+0000 7f130ae00640 1 -- 172.21.15.90:0/1867739188 <== osd.0 v2:172.21.15.90:6800/307045587 4 ==== osd_op_reply(12 200.00000002 [write 0~48] v56'4 uv4 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x563a75d68000 con 0x563a753abc00 2024-03-24T01:30:33.397 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.396+0000 7f130a400640 1 -- 172.21.15.90:0/1867739188 <== osd.6 v2:172.21.15.120:6816/1454745630 1 ==== osd_op_reply(8 200.00000004 [delete] v56'5 uv3 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x563a75d68780 con 0x563a753c7c00 2024-03-24T01:30:33.397 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.396+0000 7f1309a00640 1 -- 172.21.15.90:0/1867739188 <== osd.4 v2:172.21.15.120:6800/2150486081 1 ==== osd_op_reply(9 200.00000005 [delete] v56'2 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x563a75d68500 con 0x563a753d4c00 2024-03-24T01:30:33.397 INFO:tasks.workunit.client.0.smithi090.stdout:done 2024-03-24T01:30:33.399 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.396+0000 7f130c3b8280 1 -- 172.21.15.90:0/1867739188 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x563a753abc00 msgr2=0x563a753c2000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.399 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.396+0000 7f130c3b8280 1 --2- 172.21.15.90:0/1867739188 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x563a753abc00 0x563a753c2000 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.400 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.396+0000 7f130c3b8280 1 -- 172.21.15.90:0/1867739188 >> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] conn(0x563a753c7400 msgr2=0x563a753c2b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.400 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.396+0000 7f130c3b8280 1 --2- 172.21.15.90:0/1867739188 >> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] conn(0x563a753c7400 0x563a753c2b00 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.400 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.396+0000 7f130c3b8280 1 -- 172.21.15.90:0/1867739188 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x563a753c6400 msgr2=0x563a753c2580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.400 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.396+0000 7f130c3b8280 1 --2- 172.21.15.90:0/1867739188 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x563a753c6400 0x563a753c2580 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.400 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.396+0000 7f130c3b8280 1 -- 172.21.15.90:0/1867739188 >> [v2:172.21.15.120:6800/2150486081,v1:172.21.15.120:6801/2150486081] conn(0x563a753d4c00 msgr2=0x563a753c3600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.400 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.396+0000 7f130c3b8280 1 --2- 172.21.15.90:0/1867739188 >> [v2:172.21.15.120:6800/2150486081,v1:172.21.15.120:6801/2150486081] conn(0x563a753d4c00 0x563a753c3600 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.400 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.396+0000 7f130a400640 1 -- 172.21.15.90:0/1867739188 reap_dead start 2024-03-24T01:30:33.400 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.396+0000 7f130c3b8280 1 -- 172.21.15.90:0/1867739188 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x563a753c7c00 msgr2=0x563a753c3080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.400 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.396+0000 7f130c3b8280 1 --2- 172.21.15.90:0/1867739188 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x563a753c7c00 0x563a753c3080 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.400 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.396+0000 7f130a400640 1 -- 172.21.15.90:0/1867739188 reap_dead start 2024-03-24T01:30:33.400 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.396+0000 7f130c3b8280 1 -- 172.21.15.90:0/1867739188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a75f1f800 msgr2=0x563a7504fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.400 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.396+0000 7f130c3b8280 1 --2- 172.21.15.90:0/1867739188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a75f1f800 0x563a7504fb80 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x563a752ff710 tx=0x563a753a80c0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.400 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.396+0000 7f130c3b8280 1 -- 172.21.15.90:0/1867739188 shutdown_connections 2024-03-24T01:30:33.400 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.396+0000 7f130c3b8280 1 --2- 172.21.15.90:0/1867739188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a75f1f800 0x563a7504fb80 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.401 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.396+0000 7f130c3b8280 1 -- 172.21.15.90:0/1867739188 >> 172.21.15.90:0/1867739188 conn(0x563a75f04c00 msgr2=0x563a75d65800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.401 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.396+0000 7f130c3b8280 1 -- 172.21.15.90:0/1867739188 shutdown_connections 2024-03-24T01:30:33.401 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.396+0000 7f130c3b8280 1 -- 172.21.15.90:0/1867739188 wait complete. 2024-03-24T01:30:33.401 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.397+0000 7f130c3b8280 1 -- 172.21.15.90:0/3666402165 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x563a75f1f400 msgr2=0x563a75051700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.401 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.397+0000 7f130c3b8280 1 --2- 172.21.15.90:0/3666402165 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x563a75f1f400 0x563a75051700 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x563a75ddc8a0 tx=0x563a75eb4c00 comp rx=0 tx=0).stop 2024-03-24T01:30:33.401 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.397+0000 7f130c3b8280 1 -- 172.21.15.90:0/3666402165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a75f04800 msgr2=0x563a75050680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.401 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.397+0000 7f130c3b8280 1 --2- 172.21.15.90:0/3666402165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a75f04800 0x563a75050680 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x563a75dfd710 tx=0x563a750ccf00 comp rx=0 tx=0).stop 2024-03-24T01:30:33.401 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.397+0000 7f130c3b8280 1 -- 172.21.15.90:0/3666402165 shutdown_connections 2024-03-24T01:30:33.401 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.397+0000 7f130c3b8280 1 --2- 172.21.15.90:0/3666402165 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x563a75f1f400 0x563a75051700 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.401 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.397+0000 7f130c3b8280 1 --2- 172.21.15.90:0/3666402165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a75f04800 0x563a75050680 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.401 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.397+0000 7f130c3b8280 1 --2- 172.21.15.90:0/3666402165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a75f04400 0x563a75050c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.401 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.397+0000 7f130c3b8280 1 --2- 172.21.15.90:0/3666402165 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563a75f04000 0x563a75051180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.401 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.397+0000 7f130c3b8280 1 -- 172.21.15.90:0/3666402165 >> 172.21.15.90:0/3666402165 conn(0x563a75dbdc00 msgr2=0x563a75d65000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.401 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.397+0000 7f130c3b8280 1 -- 172.21.15.90:0/3666402165 shutdown_connections 2024-03-24T01:30:33.402 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.397+0000 7f130c3b8280 1 -- 172.21.15.90:0/3666402165 wait complete. 2024-03-24T01:30:33.402 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.397+0000 7f130c3b8280 1 -- 172.21.15.90:0/3892851206 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a74fe5800 msgr2=0x563a7504eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.402 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.397+0000 7f130c3b8280 1 --2- 172.21.15.90:0/3892851206 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a74fe5800 0x563a7504eb00 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x563a75dfd290 tx=0x563a750cce40 comp rx=0 tx=0).stop 2024-03-24T01:30:33.402 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.397+0000 7f130c3b8280 1 -- 172.21.15.90:0/3892851206 shutdown_connections 2024-03-24T01:30:33.402 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.397+0000 7f130c3b8280 1 --2- 172.21.15.90:0/3892851206 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a74fe5800 0x563a7504eb00 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.402 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.397+0000 7f130c3b8280 1 --2- 172.21.15.90:0/3892851206 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563a74fe5c00 0x563a7504f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.402 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.397+0000 7f130c3b8280 1 --2- 172.21.15.90:0/3892851206 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a74fe4c00 0x563a7504e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.402 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.397+0000 7f130c3b8280 1 -- 172.21.15.90:0/3892851206 >> 172.21.15.90:0/3892851206 conn(0x563a74fe5000 msgr2=0x563a75d64800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.402 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.397+0000 7f130c3b8280 1 -- 172.21.15.90:0/3892851206 shutdown_connections 2024-03-24T01:30:33.402 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.398+0000 7f130c3b8280 1 -- 172.21.15.90:0/3892851206 wait complete. 2024-03-24T01:30:33.403 INFO:tasks.workunit.client.0.smithi090.stderr:+ cephfs-journal-tool --rank=cephfs:0 journal inspect 2024-03-24T01:30:33.447 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.446+0000 7effa54fa280 1 -- start start 2024-03-24T01:30:33.447 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.446+0000 7effa54fa280 1 Processor -- start 2024-03-24T01:30:33.449 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.447+0000 7effa54fa280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ab4b7ec00 0x557ab4be8580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.449 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.447+0000 7effa54fa280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557ab4b7f800 0x557ab4be8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.449 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.447+0000 7effa54fa280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557ab4b7fc00 0x557ab4be9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.449 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.447+0000 7effa54fa280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ab5964600 con 0x557ab4b7ec00 2024-03-24T01:30:33.450 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.447+0000 7effa54fa280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ab5964480 con 0x557ab4b7fc00 2024-03-24T01:30:33.450 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.447+0000 7effa54fa280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x557ab5964300 con 0x557ab4b7f800 2024-03-24T01:30:33.450 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.447+0000 7effa3600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ab4b7ec00 0x557ab4be8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.450 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.447+0000 7effa3600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ab4b7ec00 0x557ab4be8580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36382/0 (socket says 172.21.15.90:36382) 2024-03-24T01:30:33.450 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.447+0000 7effa3600640 1 -- 172.21.15.90:0/1199445161 learned_addr learned my addr 172.21.15.90:0/1199445161 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.450 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.447+0000 7effa3600640 1 -- 172.21.15.90:0/1199445161 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557ab4b7f800 msgr2=0x557ab4be8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.450 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.447+0000 7effa3600640 1 --2- 172.21.15.90:0/1199445161 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557ab4b7f800 0x557ab4be8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.450 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.447+0000 7effa3600640 1 -- 172.21.15.90:0/1199445161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557ab4b7fc00 msgr2=0x557ab4be9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.450 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.447+0000 7effa3600640 1 --2- 172.21.15.90:0/1199445161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557ab4b7fc00 0x557ab4be9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.450 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.447+0000 7effa3600640 1 -- 172.21.15.90:0/1199445161 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557ab4bb2780 con 0x557ab4b7ec00 2024-03-24T01:30:33.450 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.448+0000 7effa3600640 1 --2- 172.21.15.90:0/1199445161 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ab4b7ec00 0x557ab4be8580 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x557ab59a1530 tx=0x557ab5956de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b48031b5c69aebd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.450 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.448+0000 7effa0400640 1 -- 172.21.15.90:0/1199445161 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x557ab4ba4680 con 0x557ab4b7ec00 2024-03-24T01:30:33.451 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.448+0000 7effa0400640 1 -- 172.21.15.90:0/1199445161 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x557ab4baa700 con 0x557ab4b7ec00 2024-03-24T01:30:33.451 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.448+0000 7effa0400640 1 -- 172.21.15.90:0/1199445161 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x557ab4ba49c0 con 0x557ab4b7ec00 2024-03-24T01:30:33.451 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.448+0000 7effa54fa280 1 -- 172.21.15.90:0/1199445161 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557ab4bb3680 con 0x557ab4b7ec00 2024-03-24T01:30:33.451 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.448+0000 7effa54fa280 1 -- 172.21.15.90:0/1199445161 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x557ab4bb2000 con 0x557ab4b7ec00 2024-03-24T01:30:33.451 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.448+0000 7effa0400640 1 -- 172.21.15.90:0/1199445161 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x557ab4c66e00 con 0x557ab4b7ec00 2024-03-24T01:30:33.451 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.448+0000 7effa0400640 1 -- 172.21.15.90:0/1199445161 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 36) v1 ==== 3310+0+0 (secure 0 0 0) 0x557ab4bd9180 con 0x557ab4b7ec00 2024-03-24T01:30:33.454 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.449+0000 7effa54fa280 1 Processor -- start 2024-03-24T01:30:33.454 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.449+0000 7effa54fa280 1 -- start start 2024-03-24T01:30:33.454 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.450+0000 7effa54fa280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557ab5a53400 0x557ab4be9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.454 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.450+0000 7effa54fa280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557ab5a53800 0x557ab4be9b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.454 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.450+0000 7effa54fa280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ab5a53c00 0x557ab4bea100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.454 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.450+0000 7effa54fa280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ab5964180 con 0x557ab5a53c00 2024-03-24T01:30:33.454 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.450+0000 7effa54fa280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ab4be0900 con 0x557ab5a53800 2024-03-24T01:30:33.454 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.450+0000 7effa54fa280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x557ab4be0480 con 0x557ab5a53400 2024-03-24T01:30:33.454 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.450+0000 7effa2c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ab5a53c00 0x557ab4bea100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.454 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.450+0000 7effa2c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ab5a53c00 0x557ab4bea100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36398/0 (socket says 172.21.15.90:36398) 2024-03-24T01:30:33.454 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.450+0000 7effa2c00640 1 -- 172.21.15.90:0/3815273747 learned_addr learned my addr 172.21.15.90:0/3815273747 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.455 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.451+0000 7effa2c00640 1 -- 172.21.15.90:0/3815273747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557ab5a53400 msgr2=0x557ab4be9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.455 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.451+0000 7effa2c00640 1 --2- 172.21.15.90:0/3815273747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557ab5a53400 0x557ab4be9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.455 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.451+0000 7effa2c00640 1 -- 172.21.15.90:0/3815273747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557ab5a53800 msgr2=0x557ab4be9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.455 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.451+0000 7effa3600640 1 --2- 172.21.15.90:0/3815273747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557ab5a53800 0x557ab4be9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.455 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.451+0000 7effa2c00640 1 --2- 172.21.15.90:0/3815273747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557ab5a53800 0x557ab4be9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.455 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.451+0000 7effa2c00640 1 -- 172.21.15.90:0/3815273747 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557ab4bb23c0 con 0x557ab5a53c00 2024-03-24T01:30:33.455 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.451+0000 7effa2c00640 1 --2- 172.21.15.90:0/3815273747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ab5a53c00 0x557ab4bea100 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x557ab5a68300 tx=0x557ab5a6cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a70e73432f2b52ea server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.455 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.451+0000 7eff9dc00640 1 -- 172.21.15.90:0/3815273747 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x557ab4ba4680 con 0x557ab5a53c00 2024-03-24T01:30:33.455 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.451+0000 7eff9dc00640 1 -- 172.21.15.90:0/3815273747 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x557ab4baae00 con 0x557ab5a53c00 2024-03-24T01:30:33.455 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.451+0000 7eff9dc00640 1 -- 172.21.15.90:0/3815273747 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x557ab4ba49c0 con 0x557ab5a53c00 2024-03-24T01:30:33.455 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.452+0000 7effa54fa280 1 -- 172.21.15.90:0/3815273747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ab5a53c00 msgr2=0x557ab4bea100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.455 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.452+0000 7effa54fa280 1 --2- 172.21.15.90:0/3815273747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ab5a53c00 0x557ab4bea100 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x557ab5a68300 tx=0x557ab5a6cd20 comp rx=0 tx=0).stop 2024-03-24T01:30:33.455 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.452+0000 7effa54fa280 1 -- 172.21.15.90:0/3815273747 shutdown_connections 2024-03-24T01:30:33.456 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.452+0000 7effa54fa280 1 --2- 172.21.15.90:0/3815273747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ab5a53c00 0x557ab4bea100 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.456 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.452+0000 7effa54fa280 1 --2- 172.21.15.90:0/3815273747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557ab5a53800 0x557ab4be9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.456 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.452+0000 7effa54fa280 1 --2- 172.21.15.90:0/3815273747 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557ab5a53400 0x557ab4be9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.456 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.452+0000 7effa54fa280 1 -- 172.21.15.90:0/3815273747 >> 172.21.15.90:0/3815273747 conn(0x557ab5a53000 msgr2=0x557ab4c87800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.456 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.452+0000 7effa54fa280 1 -- 172.21.15.90:0/3815273747 shutdown_connections 2024-03-24T01:30:33.456 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.452+0000 7effa54fa280 1 -- 172.21.15.90:0/3815273747 wait complete. 2024-03-24T01:30:33.456 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.452+0000 7effa54fa280 1 Processor -- start 2024-03-24T01:30:33.456 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.453+0000 7effa54fa280 1 -- start start 2024-03-24T01:30:33.459 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.453+0000 7effa54fa280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ab5a53400 0x557ab4beb180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.459 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.453+0000 7effa54fa280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557ab5a53800 0x557ab4beac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.459 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.453+0000 7effa54fa280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557ab5a53c00 0x557ab4bea680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.459 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.453+0000 7effa54fa280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ab5965800 con 0x557ab5a53400 2024-03-24T01:30:33.459 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.453+0000 7effa54fa280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ab5964180 con 0x557ab5a53800 2024-03-24T01:30:33.459 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.453+0000 7effa54fa280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x557ab5964d80 con 0x557ab5a53c00 2024-03-24T01:30:33.459 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.453+0000 7effa4000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ab5a53400 0x557ab4beb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.460 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.453+0000 7effa4000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ab5a53400 0x557ab4beb180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36404/0 (socket says 172.21.15.90:36404) 2024-03-24T01:30:33.460 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.453+0000 7effa4000640 1 -- 172.21.15.90:0/661711584 learned_addr learned my addr 172.21.15.90:0/661711584 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.460 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.454+0000 7effa4000640 1 -- 172.21.15.90:0/661711584 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557ab5a53c00 msgr2=0x557ab4bea680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.460 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.454+0000 7effa4000640 1 --2- 172.21.15.90:0/661711584 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557ab5a53c00 0x557ab4bea680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.460 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.454+0000 7effa4000640 1 -- 172.21.15.90:0/661711584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557ab5a53800 msgr2=0x557ab4beac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.460 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.454+0000 7effa4000640 1 --2- 172.21.15.90:0/661711584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557ab5a53800 0x557ab4beac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.460 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.454+0000 7effa4000640 1 -- 172.21.15.90:0/661711584 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557ab4bb2780 con 0x557ab5a53400 2024-03-24T01:30:33.460 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.454+0000 7effa4000640 1 --2- 172.21.15.90:0/661711584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ab5a53400 0x557ab4beb180 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x557ab5a69680 tx=0x557ab5a6cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.460 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.454+0000 7eff9be00640 1 -- 172.21.15.90:0/661711584 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x557ab4ba4680 con 0x557ab5a53400 2024-03-24T01:30:33.460 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.454+0000 7eff9be00640 1 -- 172.21.15.90:0/661711584 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x557ab4baa1c0 con 0x557ab5a53400 2024-03-24T01:30:33.460 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.454+0000 7eff9be00640 1 -- 172.21.15.90:0/661711584 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x557ab4ba49c0 con 0x557ab5a53400 2024-03-24T01:30:33.460 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.454+0000 7effa54fa280 1 -- 172.21.15.90:0/661711584 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x557ab4bb3680 con 0x557ab5a53400 2024-03-24T01:30:33.460 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.454+0000 7effa54fa280 1 -- 172.21.15.90:0/661711584 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557ab4bb2960 con 0x557ab5a53400 2024-03-24T01:30:33.460 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.456+0000 7eff9be00640 1 -- 172.21.15.90:0/661711584 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x557ab5a7ec00 con 0x557ab5a53400 2024-03-24T01:30:33.461 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.456+0000 7eff9be00640 1 --2- 172.21.15.90:0/661711584 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x557ab5ac7000 0x557ab4beb700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.461 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.457+0000 7eff9be00640 1 -- 172.21.15.90:0/661711584 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x557ab5acca00 con 0x557ab5a53400 2024-03-24T01:30:33.461 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.457+0000 7effa54fa280 1 --2- 172.21.15.90:0/661711584 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x557ab5ac7400 0x557ab4bea100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.461 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.457+0000 7effa54fa280 1 -- 172.21.15.90:0/661711584 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:1 7.f 7:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x557ab5ac7800 con 0x557ab5ac7400 2024-03-24T01:30:33.461 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.457+0000 7effa2c00640 1 --2- 172.21.15.90:0/661711584 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x557ab5ac7400 0x557ab4bea100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.461 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.457+0000 7effa3600640 1 --2- 172.21.15.90:0/661711584 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x557ab5ac7000 0x557ab4beb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.461 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.457+0000 7effa2c00640 1 --2- 172.21.15.90:0/661711584 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x557ab5ac7400 0x557ab4bea100 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.461 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.458+0000 7effa2c00640 1 -- 172.21.15.90:0/661711584 <== osd.0 v2:172.21.15.90:6800/307045587 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x557ab5902000 con 0x557ab5ac7400 2024-03-24T01:30:33.461 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.458+0000 7effa54fa280 1 --2- 172.21.15.90:0/661711584 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x557ab5ac7c00 0x557ab4be9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.461 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.458+0000 7effa54fa280 1 -- 172.21.15.90:0/661711584 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:2 7.14 7:292cf221:::200.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x557ab4f00000 con 0x557ab5ac7c00 2024-03-24T01:30:33.461 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.458+0000 7effa4000640 1 --2- 172.21.15.90:0/661711584 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x557ab5ac7c00 0x557ab4be9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.462 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.458+0000 7effa3600640 1 --2- 172.21.15.90:0/661711584 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x557ab5ac7000 0x557ab4beb700 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x557ab5a683f0 tx=0x557ab5956d20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.462 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.459+0000 7effa4000640 1 --2- 172.21.15.90:0/661711584 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x557ab5ac7c00 0x557ab4be9b80 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.462 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.459+0000 7effa4000640 1 -- 172.21.15.90:0/661711584 <== osd.3 v2:172.21.15.90:6808/2999931983 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv27 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x557ab5902500 con 0x557ab5ac7c00 2024-03-24T01:30:33.462 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.459+0000 7effa54fa280 1 -- 172.21.15.90:0/661711584 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:3 7.1b 7:ddb4e4d7:::200.00000002:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x557ab4f00400 con 0x557ab5ac7400 2024-03-24T01:30:33.463 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.462+0000 7effa2c00640 1 -- 172.21.15.90:0/661711584 <== osd.0 v2:172.21.15.90:6800/307045587 2 ==== osd_op_reply(3 200.00000002 [read 0~48 out=48b] v0'0 uv4 ondisk = 0) v8 ==== 156+0+48 (crc 0 0 0) 0x557ab5902000 con 0x557ab5ac7400 2024-03-24T01:30:33.464 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.462+0000 7effa54fa280 1 --2- 172.21.15.90:0/661711584 >> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] conn(0x557ab4f00800 0x557ab4be9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.464 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.462+0000 7effa54fa280 1 -- 172.21.15.90:0/661711584 --> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] -- osd_op(unknown.0.0:4 7.b 7:d5c7a900:::200.00000003:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x557ab4f00c00 con 0x557ab4f00800 2024-03-24T01:30:33.464 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.463+0000 7effa3600640 1 --2- 172.21.15.90:0/661711584 >> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] conn(0x557ab4f00800 0x557ab4be9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.464 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.463+0000 7effa3600640 1 --2- 172.21.15.90:0/661711584 >> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] conn(0x557ab4f00800 0x557ab4be9600 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.464 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.463+0000 7effa3600640 1 -- 172.21.15.90:0/661711584 <== osd.1 v2:172.21.15.90:6816/133441085 1 ==== osd_op_reply(4 200.00000003 [read 0~2147483647] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x557ab5902780 con 0x557ab4f00800 2024-03-24T01:30:33.465 INFO:tasks.workunit.client.0.smithi090.stdout:Overall journal integrity: OK 2024-03-24T01:30:33.466 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.463+0000 7effa54fa280 1 -- 172.21.15.90:0/661711584 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x557ab5ac7400 msgr2=0x557ab4bea100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.466 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.463+0000 7effa54fa280 1 --2- 172.21.15.90:0/661711584 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x557ab5ac7400 0x557ab4bea100 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.466 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.463+0000 7effa54fa280 1 -- 172.21.15.90:0/661711584 >> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] conn(0x557ab4f00800 msgr2=0x557ab4be9600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.467 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.463+0000 7effa54fa280 1 --2- 172.21.15.90:0/661711584 >> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] conn(0x557ab4f00800 0x557ab4be9600 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.467 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.463+0000 7effa54fa280 1 -- 172.21.15.90:0/661711584 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x557ab5ac7c00 msgr2=0x557ab4be9b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.467 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.463+0000 7effa54fa280 1 --2- 172.21.15.90:0/661711584 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x557ab5ac7c00 0x557ab4be9b80 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.467 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.464+0000 7effa2c00640 1 -- 172.21.15.90:0/661711584 reap_dead start 2024-03-24T01:30:33.467 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.464+0000 7effa54fa280 1 -- 172.21.15.90:0/661711584 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x557ab5ac7000 msgr2=0x557ab4beb700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.467 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.464+0000 7effa54fa280 1 --2- 172.21.15.90:0/661711584 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x557ab5ac7000 0x557ab4beb700 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x557ab5a683f0 tx=0x557ab5956d20 comp rx=0 tx=0).stop 2024-03-24T01:30:33.467 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.464+0000 7effa54fa280 1 -- 172.21.15.90:0/661711584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ab5a53400 msgr2=0x557ab4beb180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.467 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.464+0000 7effa54fa280 1 --2- 172.21.15.90:0/661711584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ab5a53400 0x557ab4beb180 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x557ab5a69680 tx=0x557ab5a6cf60 comp rx=0 tx=0).stop 2024-03-24T01:30:33.467 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.464+0000 7effa54fa280 1 -- 172.21.15.90:0/661711584 shutdown_connections 2024-03-24T01:30:33.467 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.464+0000 7effa54fa280 1 --2- 172.21.15.90:0/661711584 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x557ab5ac7000 0x557ab4beb700 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.467 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.464+0000 7effa54fa280 1 --2- 172.21.15.90:0/661711584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ab5a53400 0x557ab4beb180 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.467 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.464+0000 7effa54fa280 1 -- 172.21.15.90:0/661711584 >> 172.21.15.90:0/661711584 conn(0x557ab5a53000 msgr2=0x557ab4c87800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.467 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.464+0000 7effa54fa280 1 -- 172.21.15.90:0/661711584 shutdown_connections 2024-03-24T01:30:33.467 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.464+0000 7effa54fa280 1 -- 172.21.15.90:0/661711584 wait complete. 2024-03-24T01:30:33.467 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.464+0000 7effa54fa280 1 -- 172.21.15.90:0/1199445161 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ab4b7ec00 msgr2=0x557ab4be8580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.468 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.464+0000 7effa54fa280 1 --2- 172.21.15.90:0/1199445161 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ab4b7ec00 0x557ab4be8580 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x557ab59a1530 tx=0x557ab5956de0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.468 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.464+0000 7effa54fa280 1 -- 172.21.15.90:0/1199445161 shutdown_connections 2024-03-24T01:30:33.468 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.464+0000 7effa54fa280 1 --2- 172.21.15.90:0/1199445161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557ab4b7fc00 0x557ab4be9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.468 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.464+0000 7effa54fa280 1 --2- 172.21.15.90:0/1199445161 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557ab4b7f800 0x557ab4be8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.468 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.464+0000 7effa54fa280 1 --2- 172.21.15.90:0/1199445161 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ab4b7ec00 0x557ab4be8580 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.468 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.464+0000 7effa54fa280 1 -- 172.21.15.90:0/1199445161 >> 172.21.15.90:0/1199445161 conn(0x557ab4b7f000 msgr2=0x557ab4c87000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.468 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.464+0000 7effa54fa280 1 -- 172.21.15.90:0/1199445161 shutdown_connections 2024-03-24T01:30:33.468 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.465+0000 7effa54fa280 1 -- 172.21.15.90:0/1199445161 wait complete. 2024-03-24T01:30:33.470 INFO:tasks.workunit.client.0.smithi090.stderr:+ cephfs-journal-tool --rank=cephfs:0 header get 2024-03-24T01:30:33.514 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.512+0000 7f4827d29280 1 -- start start 2024-03-24T01:30:33.514 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.512+0000 7f4827d29280 1 Processor -- start 2024-03-24T01:30:33.516 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.513+0000 7f4827d29280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558f7e6bcc00 0x558f7e726580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.516 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.513+0000 7f4827d29280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558f7e6bd800 0x558f7e726b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.516 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.513+0000 7f4827d29280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f7e6bdc00 0x558f7e727080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.516 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.513+0000 7f4827d29280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f7f498780 con 0x558f7e6bcc00 2024-03-24T01:30:33.516 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.513+0000 7f4827d29280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f7f498600 con 0x558f7e6bdc00 2024-03-24T01:30:33.516 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.513+0000 7f4827d29280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x558f7f498480 con 0x558f7e6bd800 2024-03-24T01:30:33.516 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.514+0000 7f4825e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558f7e6bcc00 0x558f7e726580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.517 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.514+0000 7f4825e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558f7e6bcc00 0x558f7e726580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36418/0 (socket says 172.21.15.90:36418) 2024-03-24T01:30:33.517 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.514+0000 7f4825e00640 1 -- 172.21.15.90:0/3992673769 learned_addr learned my addr 172.21.15.90:0/3992673769 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.517 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.514+0000 7f4826800640 1 --2- 172.21.15.90:0/3992673769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f7e6bdc00 0x558f7e727080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.517 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.514+0000 7f4825e00640 1 -- 172.21.15.90:0/3992673769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558f7e6bd800 msgr2=0x558f7e726b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.517 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.514+0000 7f4825400640 1 --2- 172.21.15.90:0/3992673769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558f7e6bd800 0x558f7e726b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.517 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.514+0000 7f4825e00640 1 --2- 172.21.15.90:0/3992673769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558f7e6bd800 0x558f7e726b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.517 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.514+0000 7f4825e00640 1 -- 172.21.15.90:0/3992673769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f7e6bdc00 msgr2=0x558f7e727080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.517 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.514+0000 7f4825e00640 1 --2- 172.21.15.90:0/3992673769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f7e6bdc00 0x558f7e727080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.517 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.514+0000 7f4825e00640 1 -- 172.21.15.90:0/3992673769 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f7e6f0780 con 0x558f7e6bcc00 2024-03-24T01:30:33.517 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.514+0000 7f4826800640 1 --2- 172.21.15.90:0/3992673769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f7e6bdc00 0x558f7e727080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:30:33.517 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.514+0000 7f4825e00640 1 --2- 172.21.15.90:0/3992673769 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558f7e6bcc00 0x558f7e726580 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x558f7f4d5290 tx=0x558f7f48af00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20ee591efc81450a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.517 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.514+0000 7f4822c00640 1 -- 172.21.15.90:0/3992673769 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x558f7e6e29c0 con 0x558f7e6bcc00 2024-03-24T01:30:33.517 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.514+0000 7f4822c00640 1 -- 172.21.15.90:0/3992673769 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558f7e6e8fc0 con 0x558f7e6bcc00 2024-03-24T01:30:33.517 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.514+0000 7f4822c00640 1 -- 172.21.15.90:0/3992673769 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x558f7e6e2680 con 0x558f7e6bcc00 2024-03-24T01:30:33.518 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.515+0000 7f4827d29280 1 -- 172.21.15.90:0/3992673769 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558f7e6f03c0 con 0x558f7e6bcc00 2024-03-24T01:30:33.518 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.515+0000 7f4827d29280 1 -- 172.21.15.90:0/3992673769 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x558f7e6f1680 con 0x558f7e6bcc00 2024-03-24T01:30:33.518 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.515+0000 7f4822c00640 1 -- 172.21.15.90:0/3992673769 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x558f7f586200 con 0x558f7e6bcc00 2024-03-24T01:30:33.518 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.515+0000 7f4822c00640 1 -- 172.21.15.90:0/3992673769 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 36) v1 ==== 3310+0+0 (secure 0 0 0) 0x558f7e717180 con 0x558f7e6bcc00 2024-03-24T01:30:33.521 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.516+0000 7f4827d29280 1 Processor -- start 2024-03-24T01:30:33.521 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.516+0000 7f4827d29280 1 -- start start 2024-03-24T01:30:33.521 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.517+0000 7f4827d29280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f7f5de000 0x558f7e727600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.521 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.517+0000 7f4827d29280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558f7f5de400 0x558f7e727b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.521 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.517+0000 7f4827d29280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558f7f5de800 0x558f7e728100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.521 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.517+0000 7f4827d29280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f7f498300 con 0x558f7f5de800 2024-03-24T01:30:33.521 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.517+0000 7f4827d29280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f7f498000 con 0x558f7f5de000 2024-03-24T01:30:33.521 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.517+0000 7f4827d29280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x558f7e71e480 con 0x558f7f5de400 2024-03-24T01:30:33.521 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.517+0000 7f4825400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558f7f5de800 0x558f7e728100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.522 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.517+0000 7f4825400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558f7f5de800 0x558f7e728100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36432/0 (socket says 172.21.15.90:36432) 2024-03-24T01:30:33.522 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.517+0000 7f4825400640 1 -- 172.21.15.90:0/2911269321 learned_addr learned my addr 172.21.15.90:0/2911269321 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.522 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.517+0000 7f4826800640 1 --2- 172.21.15.90:0/2911269321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f7f5de000 0x558f7e727600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.522 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.517+0000 7f4825e00640 1 --2- 172.21.15.90:0/2911269321 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558f7f5de400 0x558f7e727b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.522 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.517+0000 7f4825400640 1 -- 172.21.15.90:0/2911269321 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558f7f5de400 msgr2=0x558f7e727b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.522 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.517+0000 7f4825400640 1 --2- 172.21.15.90:0/2911269321 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558f7f5de400 0x558f7e727b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.522 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.517+0000 7f4825400640 1 -- 172.21.15.90:0/2911269321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f7f5de000 msgr2=0x558f7e727600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.522 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.517+0000 7f4825400640 1 --2- 172.21.15.90:0/2911269321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f7f5de000 0x558f7e727600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.522 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.517+0000 7f4825400640 1 -- 172.21.15.90:0/2911269321 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f7e6f14a0 con 0x558f7f5de800 2024-03-24T01:30:33.522 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.518+0000 7f4825400640 1 --2- 172.21.15.90:0/2911269321 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558f7f5de800 0x558f7e728100 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x558f7f5948a0 tx=0x558f7f598cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f5f3eab8e9ae606 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.522 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.518+0000 7f4820400640 1 -- 172.21.15.90:0/2911269321 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x558f7e6e29c0 con 0x558f7f5de800 2024-03-24T01:30:33.522 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.518+0000 7f4820400640 1 -- 172.21.15.90:0/2911269321 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558f7e6e8a80 con 0x558f7f5de800 2024-03-24T01:30:33.522 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.518+0000 7f4820400640 1 -- 172.21.15.90:0/2911269321 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x558f7e6e2680 con 0x558f7f5de800 2024-03-24T01:30:33.523 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.518+0000 7f4827d29280 1 -- 172.21.15.90:0/2911269321 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558f7f5de800 msgr2=0x558f7e728100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.523 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.518+0000 7f4827d29280 1 --2- 172.21.15.90:0/2911269321 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558f7f5de800 0x558f7e728100 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x558f7f5948a0 tx=0x558f7f598cc0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.523 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.518+0000 7f4827d29280 1 -- 172.21.15.90:0/2911269321 shutdown_connections 2024-03-24T01:30:33.523 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.518+0000 7f4827d29280 1 --2- 172.21.15.90:0/2911269321 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558f7f5de800 0x558f7e728100 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.523 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.518+0000 7f4827d29280 1 --2- 172.21.15.90:0/2911269321 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558f7f5de400 0x558f7e727b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.523 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.518+0000 7f4827d29280 1 --2- 172.21.15.90:0/2911269321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f7f5de000 0x558f7e727600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.523 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.518+0000 7f4827d29280 1 -- 172.21.15.90:0/2911269321 >> 172.21.15.90:0/2911269321 conn(0x558f7f487c00 msgr2=0x558f7f43d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.523 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.519+0000 7f4827d29280 1 -- 172.21.15.90:0/2911269321 shutdown_connections 2024-03-24T01:30:33.523 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.519+0000 7f4827d29280 1 -- 172.21.15.90:0/2911269321 wait complete. 2024-03-24T01:30:33.523 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.519+0000 7f4827d29280 1 Processor -- start 2024-03-24T01:30:33.523 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.519+0000 7f4827d29280 1 -- start start 2024-03-24T01:30:33.526 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.520+0000 7f4827d29280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f7f5de000 0x558f7e729180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.526 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.520+0000 7f4827d29280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558f7f5de400 0x558f7e728c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.527 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.520+0000 7f4827d29280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558f7f5de800 0x558f7e728680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.527 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.520+0000 7f4827d29280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f7f498300 con 0x558f7f5de800 2024-03-24T01:30:33.527 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.520+0000 7f4827d29280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f7f498000 con 0x558f7f5de000 2024-03-24T01:30:33.527 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.520+0000 7f4827d29280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x558f7f499500 con 0x558f7f5de400 2024-03-24T01:30:33.527 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.520+0000 7f4825400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558f7f5de800 0x558f7e728680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.527 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.520+0000 7f4825400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558f7f5de800 0x558f7e728680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36442/0 (socket says 172.21.15.90:36442) 2024-03-24T01:30:33.527 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.520+0000 7f4825400640 1 -- 172.21.15.90:0/2579204007 learned_addr learned my addr 172.21.15.90:0/2579204007 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.527 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.521+0000 7f4826800640 1 --2- 172.21.15.90:0/2579204007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f7f5de000 0x558f7e729180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.527 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.521+0000 7f4825e00640 1 --2- 172.21.15.90:0/2579204007 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558f7f5de400 0x558f7e728c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.527 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.521+0000 7f4825400640 1 -- 172.21.15.90:0/2579204007 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558f7f5de400 msgr2=0x558f7e728c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.527 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.521+0000 7f4825400640 1 --2- 172.21.15.90:0/2579204007 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558f7f5de400 0x558f7e728c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.528 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.521+0000 7f4825400640 1 -- 172.21.15.90:0/2579204007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f7f5de000 msgr2=0x558f7e729180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.528 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.521+0000 7f4825400640 1 --2- 172.21.15.90:0/2579204007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f7f5de000 0x558f7e729180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.528 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.521+0000 7f4825400640 1 -- 172.21.15.90:0/2579204007 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f7e6f0000 con 0x558f7f5de800 2024-03-24T01:30:33.528 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.521+0000 7f4825400640 1 --2- 172.21.15.90:0/2579204007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558f7f5de800 0x558f7e728680 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x558f7f595560 tx=0x558f7f598c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.528 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.521+0000 7f481e600640 1 -- 172.21.15.90:0/2579204007 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x558f7e6e24e0 con 0x558f7f5de800 2024-03-24T01:30:33.528 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.521+0000 7f481e600640 1 -- 172.21.15.90:0/2579204007 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558f7e6e8c40 con 0x558f7f5de800 2024-03-24T01:30:33.528 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.521+0000 7f4827d29280 1 -- 172.21.15.90:0/2579204007 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x558f7e6f0960 con 0x558f7f5de800 2024-03-24T01:30:33.528 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.521+0000 7f481e600640 1 -- 172.21.15.90:0/2579204007 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x558f7e6e29c0 con 0x558f7f5de800 2024-03-24T01:30:33.528 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.521+0000 7f4827d29280 1 -- 172.21.15.90:0/2579204007 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558f7e6f1860 con 0x558f7f5de800 2024-03-24T01:30:33.528 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.523+0000 7f481e600640 1 -- 172.21.15.90:0/2579204007 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x558f7f494c00 con 0x558f7f5de800 2024-03-24T01:30:33.528 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.523+0000 7f481e600640 1 --2- 172.21.15.90:0/2579204007 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x558f7f5f8400 0x558f7e729700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.528 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.523+0000 7f481e600640 1 -- 172.21.15.90:0/2579204007 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x558f7f5e7400 con 0x558f7f5de800 2024-03-24T01:30:33.528 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.523+0000 7f4827d29280 1 --2- 172.21.15.90:0/2579204007 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x558f7f5dec00 0x558f7e728100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.529 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.523+0000 7f4827d29280 1 -- 172.21.15.90:0/2579204007 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:1 7.f 7:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x558f7f5df000 con 0x558f7f5dec00 2024-03-24T01:30:33.529 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.524+0000 7f4825e00640 1 --2- 172.21.15.90:0/2579204007 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x558f7f5dec00 0x558f7e728100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.529 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.524+0000 7f4826800640 1 --2- 172.21.15.90:0/2579204007 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x558f7f5f8400 0x558f7e729700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.529 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.524+0000 7f4825e00640 1 --2- 172.21.15.90:0/2579204007 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x558f7f5dec00 0x558f7e728100 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.529 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.525+0000 7f4825e00640 1 -- 172.21.15.90:0/2579204007 <== osd.0 v2:172.21.15.90:6800/307045587 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x558f7f440000 con 0x558f7f5dec00 2024-03-24T01:30:33.529 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.525+0000 7f4827d29280 1 --2- 172.21.15.90:0/2579204007 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x558f7f5f9400 0x558f7e727b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.529 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.525+0000 7f4827d29280 1 -- 172.21.15.90:0/2579204007 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:2 7.14 7:292cf221:::200.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x558f7f5f9800 con 0x558f7f5f9400 2024-03-24T01:30:33.529 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.525+0000 7f4826800640 1 --2- 172.21.15.90:0/2579204007 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x558f7f5f8400 0x558f7e729700 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x558f7e79e6f0 tx=0x558f7ea2c000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.529 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.525+0000 7f4826800640 1 --2- 172.21.15.90:0/2579204007 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x558f7f5f9400 0x558f7e727b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.529 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.525+0000 7f4826800640 1 --2- 172.21.15.90:0/2579204007 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x558f7f5f9400 0x558f7e727b80 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.530 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.526+0000 7f4826800640 1 -- 172.21.15.90:0/2579204007 <== osd.3 v2:172.21.15.90:6808/2999931983 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv27 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x558f7f440500 con 0x558f7f5f9400 2024-03-24T01:30:33.530 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2024-03-24T01:30:33.530 INFO:tasks.workunit.client.0.smithi090.stdout: "magic": "ceph fs volume v011", 2024-03-24T01:30:33.530 INFO:tasks.workunit.client.0.smithi090.stdout: "write_pos": 8388608, 2024-03-24T01:30:33.530 INFO:tasks.workunit.client.0.smithi090.stdout: "expire_pos": 8388608, 2024-03-24T01:30:33.530 INFO:tasks.workunit.client.0.smithi090.stdout: "trimmed_pos": 8388608, 2024-03-24T01:30:33.530 INFO:tasks.workunit.client.0.smithi090.stdout: "stream_format": 1, 2024-03-24T01:30:33.530 INFO:tasks.workunit.client.0.smithi090.stdout: "layout": { 2024-03-24T01:30:33.531 INFO:tasks.workunit.client.0.smithi090.stdout: "stripe_unit": 4194304, 2024-03-24T01:30:33.531 INFO:tasks.workunit.client.0.smithi090.stdout: "stripe_count": 1, 2024-03-24T01:30:33.531 INFO:tasks.workunit.client.0.smithi090.stdout: "object_size": 4194304, 2024-03-24T01:30:33.531 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_id": 7, 2024-03-24T01:30:33.531 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_ns": "" 2024-03-24T01:30:33.531 INFO:tasks.workunit.client.0.smithi090.stdout: } 2024-03-24T01:30:33.531 INFO:tasks.workunit.client.0.smithi090.stdout:} 2024-03-24T01:30:33.531 INFO:tasks.workunit.client.0.smithi090.stdout: 2024-03-24T01:30:33.532 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.526+0000 7f4827d29280 1 -- 172.21.15.90:0/2579204007 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x558f7f5dec00 msgr2=0x558f7e728100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.532 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.526+0000 7f4827d29280 1 --2- 172.21.15.90:0/2579204007 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x558f7f5dec00 0x558f7e728100 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.532 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.526+0000 7f4827d29280 1 -- 172.21.15.90:0/2579204007 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x558f7f5f9400 msgr2=0x558f7e727b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.532 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.526+0000 7f4827d29280 1 --2- 172.21.15.90:0/2579204007 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x558f7f5f9400 0x558f7e727b80 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.533 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.526+0000 7f4827d29280 1 -- 172.21.15.90:0/2579204007 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x558f7f5f8400 msgr2=0x558f7e729700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.533 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.526+0000 7f4827d29280 1 --2- 172.21.15.90:0/2579204007 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x558f7f5f8400 0x558f7e729700 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x558f7e79e6f0 tx=0x558f7ea2c000 comp rx=0 tx=0).stop 2024-03-24T01:30:33.533 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.526+0000 7f4827d29280 1 -- 172.21.15.90:0/2579204007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558f7f5de800 msgr2=0x558f7e728680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.533 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.526+0000 7f4827d29280 1 --2- 172.21.15.90:0/2579204007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558f7f5de800 0x558f7e728680 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x558f7f595560 tx=0x558f7f598c00 comp rx=0 tx=0).stop 2024-03-24T01:30:33.533 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.526+0000 7f4825400640 1 -- 172.21.15.90:0/2579204007 reap_dead start 2024-03-24T01:30:33.533 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.526+0000 7f4825400640 1 -- 172.21.15.90:0/2579204007 reap_dead start 2024-03-24T01:30:33.533 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.526+0000 7f4827d29280 1 -- 172.21.15.90:0/2579204007 shutdown_connections 2024-03-24T01:30:33.533 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.526+0000 7f4827d29280 1 -- 172.21.15.90:0/2579204007 >> 172.21.15.90:0/2579204007 conn(0x558f7f487c00 msgr2=0x558f7f43d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.533 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.526+0000 7f4827d29280 1 -- 172.21.15.90:0/2579204007 shutdown_connections 2024-03-24T01:30:33.533 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.526+0000 7f4827d29280 1 -- 172.21.15.90:0/2579204007 wait complete. 2024-03-24T01:30:33.533 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.527+0000 7f4827d29280 1 -- 172.21.15.90:0/3992673769 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558f7e6bcc00 msgr2=0x558f7e726580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.533 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.527+0000 7f4827d29280 1 --2- 172.21.15.90:0/3992673769 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558f7e6bcc00 0x558f7e726580 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x558f7f4d5290 tx=0x558f7f48af00 comp rx=0 tx=0).stop 2024-03-24T01:30:33.533 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.527+0000 7f4827d29280 1 -- 172.21.15.90:0/3992673769 shutdown_connections 2024-03-24T01:30:33.533 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.527+0000 7f4827d29280 1 --2- 172.21.15.90:0/3992673769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f7e6bdc00 0x558f7e727080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.533 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.527+0000 7f4827d29280 1 --2- 172.21.15.90:0/3992673769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558f7e6bd800 0x558f7e726b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.534 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.527+0000 7f4827d29280 1 --2- 172.21.15.90:0/3992673769 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558f7e6bcc00 0x558f7e726580 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.534 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.527+0000 7f4827d29280 1 -- 172.21.15.90:0/3992673769 >> 172.21.15.90:0/3992673769 conn(0x558f7e6bd000 msgr2=0x558f7f43c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.534 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.527+0000 7f4827d29280 1 -- 172.21.15.90:0/3992673769 shutdown_connections 2024-03-24T01:30:33.534 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.527+0000 7f4827d29280 1 -- 172.21.15.90:0/3992673769 wait complete. 2024-03-24T01:30:33.534 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'Rolling back journal to original state...' 2024-03-24T01:30:33.534 INFO:tasks.workunit.client.0.smithi090.stdout:Rolling back journal to original state... 2024-03-24T01:30:33.535 INFO:tasks.workunit.client.0.smithi090.stderr:+ cephfs-journal-tool --rank=cephfs:0 journal import /tmp/journal.bin 2024-03-24T01:30:33.578 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.577+0000 7fa6fd046280 1 -- start start 2024-03-24T01:30:33.578 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.577+0000 7fa6fd046280 1 Processor -- start 2024-03-24T01:30:33.580 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.577+0000 7fa6fd046280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563de146a000 0x563de06fd080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.580 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.577+0000 7fa6fd046280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563de146b000 0x563de06fcb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.580 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.577+0000 7fa6fd046280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563de146b400 0x563de06fdb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.580 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.577+0000 7fa6fd046280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x563de155c000 con 0x563de146b000 2024-03-24T01:30:33.580 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.577+0000 7fa6fd046280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x563de155c180 con 0x563de146b400 2024-03-24T01:30:33.581 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.577+0000 7fa6fd046280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x563de155c300 con 0x563de146a000 2024-03-24T01:30:33.581 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.578+0000 7fa6f9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563de146b000 0x563de06fcb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.581 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.578+0000 7fa6f9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563de146b000 0x563de06fcb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36456/0 (socket says 172.21.15.90:36456) 2024-03-24T01:30:33.581 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.578+0000 7fa6f9e00640 1 -- 172.21.15.90:0/1178895828 learned_addr learned my addr 172.21.15.90:0/1178895828 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.581 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.578+0000 7fa6f9e00640 1 -- 172.21.15.90:0/1178895828 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563de146a000 msgr2=0x563de06fd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.581 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.578+0000 7fa6f9e00640 1 --2- 172.21.15.90:0/1178895828 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563de146a000 0x563de06fd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.581 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.578+0000 7fa6f9e00640 1 -- 172.21.15.90:0/1178895828 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563de146b400 msgr2=0x563de06fdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.581 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.578+0000 7fa6f9e00640 1 --2- 172.21.15.90:0/1178895828 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563de146b400 0x563de06fdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.581 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.578+0000 7fa6f9e00640 1 -- 172.21.15.90:0/1178895828 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563de06c6780 con 0x563de146b000 2024-03-24T01:30:33.581 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.578+0000 7fa6f9e00640 1 --2- 172.21.15.90:0/1178895828 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563de146b000 0x563de06fcb00 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x563de14c75f0 tx=0x563de146ede0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b3aece706430d5b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.581 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.579+0000 7fa6f7600640 1 -- 172.21.15.90:0/1178895828 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x563de06b8d00 con 0x563de146b000 2024-03-24T01:30:33.581 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.579+0000 7fa6f7600640 1 -- 172.21.15.90:0/1178895828 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563de06be700 con 0x563de146b000 2024-03-24T01:30:33.581 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.579+0000 7fa6f7600640 1 -- 172.21.15.90:0/1178895828 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x563de06b8680 con 0x563de146b000 2024-03-24T01:30:33.581 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.579+0000 7fa6fd046280 1 -- 172.21.15.90:0/1178895828 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563de06c6000 con 0x563de146b000 2024-03-24T01:30:33.582 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.579+0000 7fa6fd046280 1 -- 172.21.15.90:0/1178895828 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x563de06c7a40 con 0x563de146b000 2024-03-24T01:30:33.582 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.579+0000 7fa6f7600640 1 -- 172.21.15.90:0/1178895828 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x563de156ca00 con 0x563de146b000 2024-03-24T01:30:33.582 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.579+0000 7fa6f7600640 1 -- 172.21.15.90:0/1178895828 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 36) v1 ==== 3310+0+0 (secure 0 0 0) 0x563de06ed180 con 0x563de146b000 2024-03-24T01:30:33.585 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.580+0000 7fa6fd046280 1 Processor -- start 2024-03-24T01:30:33.585 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.580+0000 7fa6fd046280 1 -- start start 2024-03-24T01:30:33.585 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.581+0000 7fa6fd046280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563de146bc00 0x563de06fd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.585 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.581+0000 7fa6fd046280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563de156bc00 0x563de06fe100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.585 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.581+0000 7fa6fd046280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563de15d8000 0x563de06fe680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.585 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.581+0000 7fa6fd046280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x563de155c480 con 0x563de146bc00 2024-03-24T01:30:33.585 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.581+0000 7fa6fd046280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x563de155c600 con 0x563de156bc00 2024-03-24T01:30:33.585 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.581+0000 7fa6fd046280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x563de155c780 con 0x563de15d8000 2024-03-24T01:30:33.585 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.581+0000 7fa6fb200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563de146bc00 0x563de06fd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.585 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.581+0000 7fa6fb200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563de146bc00 0x563de06fd600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36472/0 (socket says 172.21.15.90:36472) 2024-03-24T01:30:33.585 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.581+0000 7fa6fb200640 1 -- 172.21.15.90:0/2517481915 learned_addr learned my addr 172.21.15.90:0/2517481915 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.585 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.581+0000 7fa6fa800640 1 --2- 172.21.15.90:0/2517481915 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563de156bc00 0x563de06fe100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.586 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.581+0000 7fa6fb200640 1 -- 172.21.15.90:0/2517481915 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563de15d8000 msgr2=0x563de06fe680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.586 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.581+0000 7fa6f9e00640 1 --2- 172.21.15.90:0/2517481915 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563de15d8000 0x563de06fe680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.586 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.581+0000 7fa6fb200640 1 --2- 172.21.15.90:0/2517481915 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563de15d8000 0x563de06fe680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.586 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.581+0000 7fa6fb200640 1 -- 172.21.15.90:0/2517481915 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563de156bc00 msgr2=0x563de06fe100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.586 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.581+0000 7fa6fb200640 1 --2- 172.21.15.90:0/2517481915 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563de156bc00 0x563de06fe100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.586 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.582+0000 7fa6fb200640 1 -- 172.21.15.90:0/2517481915 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563de06c7860 con 0x563de146bc00 2024-03-24T01:30:33.586 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.582+0000 7fa6fa800640 1 --2- 172.21.15.90:0/2517481915 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563de156bc00 0x563de06fe100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:30:33.586 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.582+0000 7fa6fb200640 1 --2- 172.21.15.90:0/2517481915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563de146bc00 0x563de06fd600 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x563de1584990 tx=0x563de158b020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8c883d1656d0961 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.586 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.582+0000 7fa6f4e00640 1 -- 172.21.15.90:0/2517481915 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x563de06b8d00 con 0x563de146bc00 2024-03-24T01:30:33.586 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.582+0000 7fa6f4e00640 1 -- 172.21.15.90:0/2517481915 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563de06bee00 con 0x563de146bc00 2024-03-24T01:30:33.586 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.582+0000 7fa6f4e00640 1 -- 172.21.15.90:0/2517481915 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x563de06b8680 con 0x563de146bc00 2024-03-24T01:30:33.586 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.582+0000 7fa6fd046280 1 -- 172.21.15.90:0/2517481915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563de146bc00 msgr2=0x563de06fd600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.586 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.582+0000 7fa6fd046280 1 --2- 172.21.15.90:0/2517481915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563de146bc00 0x563de06fd600 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x563de1584990 tx=0x563de158b020 comp rx=0 tx=0).stop 2024-03-24T01:30:33.586 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.583+0000 7fa6fd046280 1 -- 172.21.15.90:0/2517481915 shutdown_connections 2024-03-24T01:30:33.587 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.583+0000 7fa6fd046280 1 --2- 172.21.15.90:0/2517481915 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563de15d8000 0x563de06fe680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.587 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.583+0000 7fa6fd046280 1 --2- 172.21.15.90:0/2517481915 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563de156bc00 0x563de06fe100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.587 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.583+0000 7fa6fd046280 1 --2- 172.21.15.90:0/2517481915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563de146bc00 0x563de06fd600 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.587 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.583+0000 7fa6fd046280 1 -- 172.21.15.90:0/2517481915 >> 172.21.15.90:0/2517481915 conn(0x563de146b800 msgr2=0x563de0799000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.587 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.583+0000 7fa6fd046280 1 -- 172.21.15.90:0/2517481915 shutdown_connections 2024-03-24T01:30:33.587 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.583+0000 7fa6fd046280 1 -- 172.21.15.90:0/2517481915 wait complete. 2024-03-24T01:30:33.587 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.583+0000 7fa6fd046280 1 Processor -- start 2024-03-24T01:30:33.587 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.583+0000 7fa6fd046280 1 -- start start 2024-03-24T01:30:33.589 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.584+0000 7fa6fd046280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563de146bc00 0x563de06ff700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.589 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.584+0000 7fa6fd046280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563de156bc00 0x563de06ff180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.589 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.584+0000 7fa6fd046280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563de15d9400 0x563de06fec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.589 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.584+0000 7fa6fd046280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x563de155cd80 con 0x563de146bc00 2024-03-24T01:30:33.589 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.584+0000 7fa6fd046280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x563de155cf00 con 0x563de15d9400 2024-03-24T01:30:33.589 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.584+0000 7fa6fd046280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x563de155d080 con 0x563de156bc00 2024-03-24T01:30:33.589 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.584+0000 7fa6fb200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563de146bc00 0x563de06ff700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.589 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.584+0000 7fa6fb200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563de146bc00 0x563de06ff700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36478/0 (socket says 172.21.15.90:36478) 2024-03-24T01:30:33.589 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.584+0000 7fa6fb200640 1 -- 172.21.15.90:0/2511835220 learned_addr learned my addr 172.21.15.90:0/2511835220 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.589 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.585+0000 7fa6fb200640 1 -- 172.21.15.90:0/2511835220 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563de156bc00 msgr2=0x563de06ff180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.589 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.585+0000 7fa6fb200640 1 --2- 172.21.15.90:0/2511835220 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563de156bc00 0x563de06ff180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.590 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.585+0000 7fa6fb200640 1 -- 172.21.15.90:0/2511835220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563de15d9400 msgr2=0x563de06fec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.590 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.585+0000 7fa6fb200640 1 --2- 172.21.15.90:0/2511835220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563de15d9400 0x563de06fec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.590 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.585+0000 7fa6fb200640 1 -- 172.21.15.90:0/2511835220 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563de06c6780 con 0x563de146bc00 2024-03-24T01:30:33.590 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.585+0000 7fa6fb200640 1 --2- 172.21.15.90:0/2511835220 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563de146bc00 0x563de06ff700 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x563de14c7650 tx=0x563de158ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.590 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.585+0000 7fa6f3000640 1 -- 172.21.15.90:0/2511835220 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x563de06b8d00 con 0x563de146bc00 2024-03-24T01:30:33.590 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.585+0000 7fa6f3000640 1 -- 172.21.15.90:0/2511835220 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563de06bec40 con 0x563de146bc00 2024-03-24T01:30:33.590 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.585+0000 7fa6fd046280 1 -- 172.21.15.90:0/2511835220 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x563de06c6000 con 0x563de146bc00 2024-03-24T01:30:33.590 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.585+0000 7fa6f3000640 1 -- 172.21.15.90:0/2511835220 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x563de06b8680 con 0x563de146bc00 2024-03-24T01:30:33.590 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.585+0000 7fa6fd046280 1 -- 172.21.15.90:0/2511835220 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563de06c7680 con 0x563de146bc00 2024-03-24T01:30:33.590 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.587+0000 7fa6f3000640 1 -- 172.21.15.90:0/2511835220 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x563de159cc00 con 0x563de146bc00 2024-03-24T01:30:33.590 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.587+0000 7fa6f3000640 1 --2- 172.21.15.90:0/2511835220 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x563de15ed400 0x563de0a20000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.590 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.587+0000 7fa6f3000640 1 -- 172.21.15.90:0/2511835220 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x563de1438400 con 0x563de146bc00 2024-03-24T01:30:33.593 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.588+0000 7fa6fa800640 1 --2- 172.21.15.90:0/2511835220 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x563de15ed400 0x563de0a20000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.593 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.588+0000 7fa6fd046280 1 -- start start 2024-03-24T01:30:33.593 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.588+0000 7fa6fd046280 1 Processor -- start 2024-03-24T01:30:33.593 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.589+0000 7fa6fa800640 1 --2- 172.21.15.90:0/2511835220 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x563de15ed400 0x563de0a20000 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x563de141eb40 tx=0x563de158ae40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.593 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.589+0000 7fa6fd046280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563de15d8c00 0x563de06fe680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.593 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.589+0000 7fa6fd046280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563de15d8800 0x563de06fe100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.594 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.589+0000 7fa6fd046280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563de15d8400 0x563de06fd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.594 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.589+0000 7fa6fd046280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x563de155c480 con 0x563de15d8c00 2024-03-24T01:30:33.594 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.589+0000 7fa6fd046280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x563de155c600 con 0x563de15d8400 2024-03-24T01:30:33.594 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.589+0000 7fa6fd046280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x563de155c780 con 0x563de15d8800 2024-03-24T01:30:33.594 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.590+0000 7fa6fb200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563de15d8c00 0x563de06fe680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.594 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.590+0000 7fa6fb200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563de15d8c00 0x563de06fe680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36482/0 (socket says 172.21.15.90:36482) 2024-03-24T01:30:33.594 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.590+0000 7fa6fb200640 1 -- 172.21.15.90:0/901287978 learned_addr learned my addr 172.21.15.90:0/901287978 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.594 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.590+0000 7fa6fb200640 1 -- 172.21.15.90:0/901287978 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563de15d8800 msgr2=0x563de06fe100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.594 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.590+0000 7fa6f9e00640 1 --2- 172.21.15.90:0/901287978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563de15d8400 0x563de06fd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.594 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.590+0000 7fa6fb200640 1 --2- 172.21.15.90:0/901287978 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563de15d8800 0x563de06fe100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.594 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.590+0000 7fa6fb200640 1 -- 172.21.15.90:0/901287978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563de15d8400 msgr2=0x563de06fd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.594 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.590+0000 7fa6fb200640 1 --2- 172.21.15.90:0/901287978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563de15d8400 0x563de06fd600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.594 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.590+0000 7fa6fb200640 1 -- 172.21.15.90:0/901287978 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563de06c63c0 con 0x563de15d8c00 2024-03-24T01:30:33.595 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.590+0000 7fa6fb200640 1 --2- 172.21.15.90:0/901287978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563de15d8c00 0x563de06fe680 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x563de09cb890 tx=0x563de158b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ebd63128c722026 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.595 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.590+0000 7fa6ef400640 1 -- 172.21.15.90:0/901287978 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x563de09a8d00 con 0x563de15d8c00 2024-03-24T01:30:33.595 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.590+0000 7fa6ef400640 1 -- 172.21.15.90:0/901287978 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563de06be1c0 con 0x563de15d8c00 2024-03-24T01:30:33.595 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.590+0000 7fa6fd046280 1 -- 172.21.15.90:0/901287978 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563de06c74a0 con 0x563de15d8c00 2024-03-24T01:30:33.595 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.590+0000 7fa6fd046280 1 -- 172.21.15.90:0/901287978 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x563de06c6960 con 0x563de15d8c00 2024-03-24T01:30:33.595 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.590+0000 7fa6ef400640 1 -- 172.21.15.90:0/901287978 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x563de09a8ea0 con 0x563de15d8c00 2024-03-24T01:30:33.595 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.591+0000 7fa6ef400640 1 -- 172.21.15.90:0/901287978 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x563de0a74800 con 0x563de15d8c00 2024-03-24T01:30:33.595 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.591+0000 7fa6ef400640 1 -- 172.21.15.90:0/901287978 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 36) v1 ==== 3310+0+0 (secure 0 0 0) 0x563de06ed500 con 0x563de15d8c00 2024-03-24T01:30:33.595 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.591+0000 7fa6fd046280 1 --2- 172.21.15.90:0/901287978 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x563de15d8000 0x563de0a20580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.595 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.591+0000 7fa6fd046280 1 -- 172.21.15.90:0/901287978 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:1 7.f 7:f1f69726:::400.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x563de0a94400 con 0x563de15d8000 2024-03-24T01:30:33.595 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.591+0000 7fa6fa800640 1 --2- 172.21.15.90:0/901287978 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x563de15d8000 0x563de0a20580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.595 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.592+0000 7fa6fa800640 1 --2- 172.21.15.90:0/901287978 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x563de15d8000 0x563de0a20580 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=4c7672f7b7133712 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.595 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.592+0000 7fa6fa800640 1 -- 172.21.15.90:0/901287978 <== osd.0 v2:172.21.15.90:6800/307045587 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x563de1416000 con 0x563de15d8000 2024-03-24T01:30:33.596 INFO:tasks.workunit.client.0.smithi090.stdout:undump /tmp/journal.bin 2024-03-24T01:30:33.597 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.592+0000 7fa6fd046280 1 --2- 172.21.15.90:0/901287978 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x563de0a94800 0x563de0a20b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.597 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.592+0000 7fa6fd046280 1 -- 172.21.15.90:0/901287978 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:2 7.14 7:292cf221:::200.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x563de0a94c00 con 0x563de0a94800 2024-03-24T01:30:33.597 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.592+0000 7fa6f9e00640 1 --2- 172.21.15.90:0/901287978 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x563de0a94800 0x563de0a20b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.597 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.593+0000 7fa6f9e00640 1 --2- 172.21.15.90:0/901287978 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x563de0a94800 0x563de0a20b00 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=2a2e1eec4b2862c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.597 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.593+0000 7fa6f9e00640 1 -- 172.21.15.90:0/901287978 <== osd.3 v2:172.21.15.90:6808/2999931983 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv27 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x563de1416500 con 0x563de0a94800 2024-03-24T01:30:33.597 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.593+0000 7fa6ed600640 1 -- 172.21.15.90:0/901287978 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:3 7.1b 7:ddb4e4d7:::200.00000002:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56) v9 -- 0x563de0aa4000 con 0x563de15d8000 2024-03-24T01:30:33.597 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.593+0000 7fa6fa800640 1 -- 172.21.15.90:0/901287978 <== osd.0 v2:172.21.15.90:6800/307045587 2 ==== osd_op_reply(3 200.00000002 [stat out=16b] v0'0 uv4 ondisk = 0) v8 ==== 156+0+16 (crc 0 0 0) 0x563de1416000 con 0x563de15d8000 2024-03-24T01:30:33.597 INFO:tasks.workunit.client.0.smithi090.stdout:start 4194304 len 35827 write_pos 4200185 format 1 trimmed_pos 4194304 stripe_unit 4194304 stripe_count 1 object_size 4194304 2024-03-24T01:30:33.597 INFO:tasks.workunit.client.0.smithi090.stdout:writing header 200.00000000 2024-03-24T01:30:33.598 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.594+0000 7fa6fd046280 1 -- 172.21.15.90:0/901287978 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:4 7.14 7:292cf221:::200.00000000:head [writefull 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x563de0a95000 con 0x563de0a94800 2024-03-24T01:30:33.598 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.596+0000 7fa6f9e00640 1 -- 172.21.15.90:0/901287978 <== osd.3 v2:172.21.15.90:6808/2999931983 2 ==== osd_op_reply(4 200.00000000 [writefull 0~90] v56'28 uv28 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x563de1416500 con 0x563de0a94800 2024-03-24T01:30:33.598 INFO:tasks.workunit.client.0.smithi090.stdout:Purging 2 objects from 1 2024-03-24T01:30:33.598 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.596+0000 7fa6fd046280 1 -- 172.21.15.90:0/901287978 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:5 7.14 7:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x563de0a95400 con 0x563de0a94800 2024-03-24T01:30:33.599 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.596+0000 7fa6fd046280 1 -- 172.21.15.90:0/901287978 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:6 7.1b 7:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x563de0a95800 con 0x563de15d8000 2024-03-24T01:30:33.599 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.597+0000 7fa6f9e00640 1 -- 172.21.15.90:0/901287978 <== osd.3 v2:172.21.15.90:6808/2999931983 3 ==== osd_op_reply(5 200.00000001 [delete] v56'29 uv29 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x563de1416500 con 0x563de0a94800 2024-03-24T01:30:33.599 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.598+0000 7fa6fa800640 1 -- 172.21.15.90:0/901287978 <== osd.0 v2:172.21.15.90:6800/307045587 3 ==== osd_op_reply(6 200.00000002 [delete] v56'5 uv5 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x563de1416000 con 0x563de15d8000 2024-03-24T01:30:33.599 INFO:tasks.workunit.client.0.smithi090.stdout: writing 4194304~35827 2024-03-24T01:30:33.600 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.598+0000 7fa6fd046280 1 -- 172.21.15.90:0/901287978 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:7 7.14 7:2e2fa760:::200.00000001:head [write 0~35827 in=35827b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x563de0a95c00 con 0x563de0a94800 2024-03-24T01:30:33.601 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.600+0000 7fa6f9e00640 1 -- 172.21.15.90:0/901287978 <== osd.3 v2:172.21.15.90:6808/2999931983 4 ==== osd_op_reply(7 200.00000001 [write 0~35827] v56'30 uv30 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x563de1416500 con 0x563de0a94800 2024-03-24T01:30:33.601 INFO:tasks.workunit.client.0.smithi090.stdout:done. 2024-03-24T01:30:33.603 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.600+0000 7fa6fd046280 1 -- 172.21.15.90:0/901287978 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x563de15d8000 msgr2=0x563de0a20580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.603 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.600+0000 7fa6fd046280 1 --2- 172.21.15.90:0/901287978 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x563de15d8000 0x563de0a20580 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.603 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.600+0000 7fa6fd046280 1 -- 172.21.15.90:0/901287978 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x563de0a94800 msgr2=0x563de0a20b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.603 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.600+0000 7fa6fd046280 1 --2- 172.21.15.90:0/901287978 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x563de0a94800 0x563de0a20b00 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.603 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.600+0000 7fa6fd046280 1 -- 172.21.15.90:0/901287978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563de15d8c00 msgr2=0x563de06fe680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.603 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.600+0000 7fa6fd046280 1 --2- 172.21.15.90:0/901287978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563de15d8c00 0x563de06fe680 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x563de09cb890 tx=0x563de158b980 comp rx=0 tx=0).stop 2024-03-24T01:30:33.604 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.600+0000 7fa6f9e00640 1 -- 172.21.15.90:0/901287978 reap_dead start 2024-03-24T01:30:33.604 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.600+0000 7fa6fd046280 1 -- 172.21.15.90:0/901287978 shutdown_connections 2024-03-24T01:30:33.604 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.600+0000 7fa6fd046280 1 -- 172.21.15.90:0/901287978 >> 172.21.15.90:0/901287978 conn(0x563de15d9000 msgr2=0x563de0799800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.604 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.600+0000 7fa6fd046280 1 -- 172.21.15.90:0/901287978 shutdown_connections 2024-03-24T01:30:33.604 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.600+0000 7fa6fd046280 1 -- 172.21.15.90:0/901287978 wait complete. 2024-03-24T01:30:33.604 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.601+0000 7fa6fd046280 1 -- 172.21.15.90:0/2511835220 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x563de15ed400 msgr2=0x563de0a20000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.604 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.601+0000 7fa6fd046280 1 --2- 172.21.15.90:0/2511835220 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x563de15ed400 0x563de0a20000 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x563de141eb40 tx=0x563de158ae40 comp rx=0 tx=0).stop 2024-03-24T01:30:33.604 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.601+0000 7fa6fd046280 1 -- 172.21.15.90:0/2511835220 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563de146bc00 msgr2=0x563de06ff700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.604 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.601+0000 7fa6fd046280 1 --2- 172.21.15.90:0/2511835220 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563de146bc00 0x563de06ff700 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x563de14c7650 tx=0x563de158ac00 comp rx=0 tx=0).stop 2024-03-24T01:30:33.604 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.601+0000 7fa6fd046280 1 -- 172.21.15.90:0/2511835220 shutdown_connections 2024-03-24T01:30:33.604 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.601+0000 7fa6fd046280 1 --2- 172.21.15.90:0/2511835220 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x563de15ed400 0x563de0a20000 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.604 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.601+0000 7fa6fd046280 1 --2- 172.21.15.90:0/2511835220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563de15d9400 0x563de06fec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.605 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.601+0000 7fa6fd046280 1 --2- 172.21.15.90:0/2511835220 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563de156bc00 0x563de06ff180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.605 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.601+0000 7fa6fd046280 1 --2- 172.21.15.90:0/2511835220 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563de146bc00 0x563de06ff700 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.605 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.601+0000 7fa6fd046280 1 -- 172.21.15.90:0/2511835220 >> 172.21.15.90:0/2511835220 conn(0x563de146b800 msgr2=0x563de0799000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.605 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.601+0000 7fa6fd046280 1 -- 172.21.15.90:0/2511835220 shutdown_connections 2024-03-24T01:30:33.605 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.601+0000 7fa6fd046280 1 -- 172.21.15.90:0/2511835220 wait complete. 2024-03-24T01:30:33.605 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.601+0000 7fa6fd046280 1 -- 172.21.15.90:0/1178895828 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563de146b000 msgr2=0x563de06fcb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.605 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.601+0000 7fa6fd046280 1 --2- 172.21.15.90:0/1178895828 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563de146b000 0x563de06fcb00 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x563de14c75f0 tx=0x563de146ede0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.605 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.601+0000 7fa6fd046280 1 -- 172.21.15.90:0/1178895828 shutdown_connections 2024-03-24T01:30:33.605 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.601+0000 7fa6fd046280 1 --2- 172.21.15.90:0/1178895828 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563de146b000 0x563de06fcb00 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.605 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.601+0000 7fa6fd046280 1 --2- 172.21.15.90:0/1178895828 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563de146b400 0x563de06fdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.605 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.601+0000 7fa6fd046280 1 --2- 172.21.15.90:0/1178895828 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x563de146a000 0x563de06fd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.605 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.601+0000 7fa6fd046280 1 -- 172.21.15.90:0/1178895828 >> 172.21.15.90:0/1178895828 conn(0x563de0693000 msgr2=0x563de0798800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.605 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.601+0000 7fa6fd046280 1 -- 172.21.15.90:0/1178895828 shutdown_connections 2024-03-24T01:30:33.605 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.601+0000 7fa6fd046280 1 -- 172.21.15.90:0/1178895828 wait complete. 2024-03-24T01:30:33.607 INFO:tasks.workunit.client.0.smithi090.stdout:Testing 'header' commands... 2024-03-24T01:30:33.607 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'Testing '\''header'\'' commands...' 2024-03-24T01:30:33.607 INFO:tasks.workunit.client.0.smithi090.stderr:+ cephfs-journal-tool --rank=cephfs:0 header get 2024-03-24T01:30:33.651 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.649+0000 7f28b0e2c280 1 -- start start 2024-03-24T01:30:33.651 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.650+0000 7f28b0e2c280 1 Processor -- start 2024-03-24T01:30:33.653 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.650+0000 7f28b0e2c280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56520087ac00 0x5652008e4580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.653 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.650+0000 7f28b0e2c280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56520087b800 0x5652008e4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.654 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.650+0000 7f28b0e2c280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56520087bc00 0x5652008e5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.654 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.650+0000 7f28b0e2c280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x565201640180 con 0x56520087bc00 2024-03-24T01:30:33.654 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.650+0000 7f28b0e2c280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x565201640300 con 0x56520087b800 2024-03-24T01:30:33.654 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.650+0000 7f28b0e2c280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x565201640480 con 0x56520087ac00 2024-03-24T01:30:33.654 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.651+0000 7f28aea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56520087bc00 0x5652008e5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.654 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.651+0000 7f28aea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56520087bc00 0x5652008e5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36496/0 (socket says 172.21.15.90:36496) 2024-03-24T01:30:33.654 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.651+0000 7f28aea00640 1 -- 172.21.15.90:0/3138486512 learned_addr learned my addr 172.21.15.90:0/3138486512 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.654 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.651+0000 7f28ae000640 1 --2- 172.21.15.90:0/3138486512 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56520087ac00 0x5652008e4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.654 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.651+0000 7f28aea00640 1 -- 172.21.15.90:0/3138486512 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56520087ac00 msgr2=0x5652008e4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.654 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.651+0000 7f28ad600640 1 --2- 172.21.15.90:0/3138486512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56520087b800 0x5652008e4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.654 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.651+0000 7f28aea00640 1 --2- 172.21.15.90:0/3138486512 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56520087ac00 0x5652008e4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.655 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.651+0000 7f28aea00640 1 -- 172.21.15.90:0/3138486512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56520087b800 msgr2=0x5652008e4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.655 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.651+0000 7f28aea00640 1 --2- 172.21.15.90:0/3138486512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56520087b800 0x5652008e4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.655 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.651+0000 7f28aea00640 1 -- 172.21.15.90:0/3138486512 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652008ae780 con 0x56520087bc00 2024-03-24T01:30:33.655 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.651+0000 7f28ae000640 1 --2- 172.21.15.90:0/3138486512 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56520087ac00 0x5652008e4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:30:33.655 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.651+0000 7f28ad600640 1 --2- 172.21.15.90:0/3138486512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56520087b800 0x5652008e4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:30:33.655 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.651+0000 7f28aea00640 1 --2- 172.21.15.90:0/3138486512 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56520087bc00 0x5652008e5080 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x5652016953e0 tx=0x565200963140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ea55ce5fa57f42a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.655 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.652+0000 7f28aae00640 1 -- 172.21.15.90:0/3138486512 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5652008a0680 con 0x56520087bc00 2024-03-24T01:30:33.655 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.652+0000 7f28aae00640 1 -- 172.21.15.90:0/3138486512 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5652008a6fc0 con 0x56520087bc00 2024-03-24T01:30:33.655 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.652+0000 7f28aae00640 1 -- 172.21.15.90:0/3138486512 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5652008a0b60 con 0x56520087bc00 2024-03-24T01:30:33.655 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.652+0000 7f28b0e2c280 1 -- 172.21.15.90:0/3138486512 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5652008af4a0 con 0x56520087bc00 2024-03-24T01:30:33.655 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.652+0000 7f28b0e2c280 1 -- 172.21.15.90:0/3138486512 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x5652008ae000 con 0x56520087bc00 2024-03-24T01:30:33.655 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.652+0000 7f28aae00640 1 -- 172.21.15.90:0/3138486512 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x565201740600 con 0x56520087bc00 2024-03-24T01:30:33.655 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.652+0000 7f28aae00640 1 -- 172.21.15.90:0/3138486512 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 36) v1 ==== 3310+0+0 (secure 0 0 0) 0x5652008d5180 con 0x56520087bc00 2024-03-24T01:30:33.658 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.653+0000 7f28b0e2c280 1 Processor -- start 2024-03-24T01:30:33.658 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.653+0000 7f28b0e2c280 1 -- start start 2024-03-24T01:30:33.658 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.654+0000 7f28b0e2c280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56520174b400 0x5652008e5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.658 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.654+0000 7f28b0e2c280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56520174b800 0x5652008e5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.658 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.654+0000 7f28b0e2c280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56520174bc00 0x5652008e6100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.658 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.654+0000 7f28b0e2c280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x565201640600 con 0x56520174bc00 2024-03-24T01:30:33.658 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.654+0000 7f28b0e2c280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x565201640780 con 0x56520174b400 2024-03-24T01:30:33.658 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.654+0000 7f28b0e2c280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x565201640900 con 0x56520174b800 2024-03-24T01:30:33.658 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.654+0000 7f28ad600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56520174bc00 0x5652008e6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.659 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.654+0000 7f28ad600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56520174bc00 0x5652008e6100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36508/0 (socket says 172.21.15.90:36508) 2024-03-24T01:30:33.659 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.654+0000 7f28ad600640 1 -- 172.21.15.90:0/3277170662 learned_addr learned my addr 172.21.15.90:0/3277170662 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.659 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.654+0000 7f28ad600640 1 --2- 172.21.15.90:0/3277170662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56520174b400 0x5652008e5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.659 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.654+0000 7f28ae000640 1 --2- 172.21.15.90:0/3277170662 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56520174b800 0x5652008e5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.659 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.655+0000 7f28ad600640 1 -- 172.21.15.90:0/3277170662 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56520174b800 msgr2=0x5652008e5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.659 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.655+0000 7f28ad600640 1 --2- 172.21.15.90:0/3277170662 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56520174b800 0x5652008e5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.659 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.655+0000 7f28ad600640 1 -- 172.21.15.90:0/3277170662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56520174b400 msgr2=0x5652008e5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.659 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.655+0000 7f28ad600640 1 --2- 172.21.15.90:0/3277170662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56520174b400 0x5652008e5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.659 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.655+0000 7f28ad600640 1 -- 172.21.15.90:0/3277170662 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652008ae3c0 con 0x56520174bc00 2024-03-24T01:30:33.659 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.655+0000 7f28ad600640 1 --2- 172.21.15.90:0/3277170662 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56520174bc00 0x5652008e6100 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x565201694ae0 tx=0x565200962fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84ba00ba0c25b774 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.659 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.655+0000 7f28a8600640 1 -- 172.21.15.90:0/3277170662 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5652008a09c0 con 0x56520174bc00 2024-03-24T01:30:33.659 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.655+0000 7f28a8600640 1 -- 172.21.15.90:0/3277170662 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5652008a6e00 con 0x56520174bc00 2024-03-24T01:30:33.659 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.655+0000 7f28a8600640 1 -- 172.21.15.90:0/3277170662 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5652008a0680 con 0x56520174bc00 2024-03-24T01:30:33.659 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.656+0000 7f28b0e2c280 1 -- 172.21.15.90:0/3277170662 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56520174bc00 msgr2=0x5652008e6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.660 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.656+0000 7f28b0e2c280 1 --2- 172.21.15.90:0/3277170662 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56520174bc00 0x5652008e6100 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x565201694ae0 tx=0x565200962fc0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.660 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.656+0000 7f28b0e2c280 1 -- 172.21.15.90:0/3277170662 shutdown_connections 2024-03-24T01:30:33.660 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.656+0000 7f28b0e2c280 1 --2- 172.21.15.90:0/3277170662 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56520174bc00 0x5652008e6100 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.660 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.656+0000 7f28b0e2c280 1 --2- 172.21.15.90:0/3277170662 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56520174b800 0x5652008e5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.660 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.656+0000 7f28b0e2c280 1 --2- 172.21.15.90:0/3277170662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56520174b400 0x5652008e5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.660 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.656+0000 7f28b0e2c280 1 -- 172.21.15.90:0/3277170662 >> 172.21.15.90:0/3277170662 conn(0x56520174b000 msgr2=0x5652015fb000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.660 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.656+0000 7f28b0e2c280 1 -- 172.21.15.90:0/3277170662 shutdown_connections 2024-03-24T01:30:33.660 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.656+0000 7f28b0e2c280 1 -- 172.21.15.90:0/3277170662 wait complete. 2024-03-24T01:30:33.660 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.656+0000 7f28b0e2c280 1 Processor -- start 2024-03-24T01:30:33.660 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.657+0000 7f28b0e2c280 1 -- start start 2024-03-24T01:30:33.663 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.657+0000 7f28b0e2c280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56520174b400 0x5652008e7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.663 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.657+0000 7f28b0e2c280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56520174b800 0x5652008e6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.663 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.657+0000 7f28b0e2c280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56520174bc00 0x5652008e6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.663 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.657+0000 7f28b0e2c280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652008dc900 con 0x56520174b400 2024-03-24T01:30:33.664 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.657+0000 7f28b0e2c280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x565201641e00 con 0x56520174b800 2024-03-24T01:30:33.664 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.657+0000 7f28b0e2c280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x565201640d80 con 0x56520174bc00 2024-03-24T01:30:33.664 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.657+0000 7f28ad600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56520174b400 0x5652008e7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.664 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.658+0000 7f28ad600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56520174b400 0x5652008e7180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36524/0 (socket says 172.21.15.90:36524) 2024-03-24T01:30:33.664 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.658+0000 7f28ad600640 1 -- 172.21.15.90:0/1846431946 learned_addr learned my addr 172.21.15.90:0/1846431946 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.664 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.658+0000 7f28ad600640 1 -- 172.21.15.90:0/1846431946 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56520174bc00 msgr2=0x5652008e6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.664 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.658+0000 7f28ad600640 1 --2- 172.21.15.90:0/1846431946 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56520174bc00 0x5652008e6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.664 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.658+0000 7f28ad600640 1 -- 172.21.15.90:0/1846431946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56520174b800 msgr2=0x5652008e6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.664 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.658+0000 7f28ad600640 1 --2- 172.21.15.90:0/1846431946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56520174b800 0x5652008e6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.664 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.658+0000 7f28ad600640 1 -- 172.21.15.90:0/1846431946 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652008ae780 con 0x56520174b400 2024-03-24T01:30:33.664 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.658+0000 7f28ad600640 1 --2- 172.21.15.90:0/1846431946 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56520174b400 0x5652008e7180 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x565201762870 tx=0x565200962e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.664 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.658+0000 7f28a6800640 1 -- 172.21.15.90:0/1846431946 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5652008a0b60 con 0x56520174b400 2024-03-24T01:30:33.664 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.658+0000 7f28a6800640 1 -- 172.21.15.90:0/1846431946 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5652008a6c40 con 0x56520174b400 2024-03-24T01:30:33.665 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.658+0000 7f28a6800640 1 -- 172.21.15.90:0/1846431946 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5652008a04e0 con 0x56520174b400 2024-03-24T01:30:33.665 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.658+0000 7f28b0e2c280 1 -- 172.21.15.90:0/1846431946 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5652008af4a0 con 0x56520174b400 2024-03-24T01:30:33.665 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.658+0000 7f28b0e2c280 1 -- 172.21.15.90:0/1846431946 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5652008ae960 con 0x56520174b400 2024-03-24T01:30:33.665 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.660+0000 7f28a6800640 1 -- 172.21.15.90:0/1846431946 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x5652017bcc00 con 0x56520174b400 2024-03-24T01:30:33.665 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.661+0000 7f28a6800640 1 --2- 172.21.15.90:0/1846431946 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x5652017b4400 0x5652008e7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.665 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.661+0000 7f28a6800640 1 -- 172.21.15.90:0/1846431946 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x5652017b2600 con 0x56520174b400 2024-03-24T01:30:33.665 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.661+0000 7f28b0e2c280 1 --2- 172.21.15.90:0/1846431946 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x5652017b4800 0x5652008e6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.665 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.661+0000 7f28b0e2c280 1 -- 172.21.15.90:0/1846431946 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:1 7.f 7:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x5652017b4c00 con 0x5652017b4800 2024-03-24T01:30:33.665 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.661+0000 7f28ad600640 1 --2- 172.21.15.90:0/1846431946 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x5652017b4800 0x5652008e6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.665 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.661+0000 7f28ae000640 1 --2- 172.21.15.90:0/1846431946 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x5652017b4400 0x5652008e7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.665 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.661+0000 7f28ad600640 1 --2- 172.21.15.90:0/1846431946 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x5652017b4800 0x5652008e6100 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.665 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.662+0000 7f28ad600640 1 -- 172.21.15.90:0/1846431946 <== osd.0 v2:172.21.15.90:6800/307045587 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x5652015fe000 con 0x5652017b4800 2024-03-24T01:30:33.665 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.662+0000 7f28b0e2c280 1 --2- 172.21.15.90:0/1846431946 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x5652017b5000 0x5652008e5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.665 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.662+0000 7f28b0e2c280 1 -- 172.21.15.90:0/1846431946 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:2 7.14 7:292cf221:::200.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x5652017b5400 con 0x5652017b5000 2024-03-24T01:30:33.666 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.662+0000 7f28aea00640 1 --2- 172.21.15.90:0/1846431946 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x5652017b5000 0x5652008e5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.666 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.662+0000 7f28ae000640 1 --2- 172.21.15.90:0/1846431946 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x5652017b4400 0x5652008e7700 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x56520095c810 tx=0x565200b70f00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.666 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.663+0000 7f28aea00640 1 --2- 172.21.15.90:0/1846431946 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x5652017b5000 0x5652008e5b80 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.666 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.663+0000 7f28aea00640 1 -- 172.21.15.90:0/1846431946 <== osd.3 v2:172.21.15.90:6808/2999931983 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv28 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x5652015fe500 con 0x5652017b5000 2024-03-24T01:30:33.667 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2024-03-24T01:30:33.667 INFO:tasks.workunit.client.0.smithi090.stdout: "magic": "ceph fs volume v011", 2024-03-24T01:30:33.667 INFO:tasks.workunit.client.0.smithi090.stdout: "write_pos": 4200185, 2024-03-24T01:30:33.667 INFO:tasks.workunit.client.0.smithi090.stdout: "expire_pos": 4194304, 2024-03-24T01:30:33.667 INFO:tasks.workunit.client.0.smithi090.stdout: "trimmed_pos": 4194304, 2024-03-24T01:30:33.667 INFO:tasks.workunit.client.0.smithi090.stdout: "stream_format": 1, 2024-03-24T01:30:33.667 INFO:tasks.workunit.client.0.smithi090.stdout: "layout": { 2024-03-24T01:30:33.667 INFO:tasks.workunit.client.0.smithi090.stdout: "stripe_unit": 4194304, 2024-03-24T01:30:33.667 INFO:tasks.workunit.client.0.smithi090.stdout: "stripe_count": 1, 2024-03-24T01:30:33.667 INFO:tasks.workunit.client.0.smithi090.stdout: "object_size": 4194304, 2024-03-24T01:30:33.667 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_id": 7, 2024-03-24T01:30:33.667 INFO:tasks.workunit.client.0.smithi090.stdout: "pool_ns": "" 2024-03-24T01:30:33.667 INFO:tasks.workunit.client.0.smithi090.stdout: } 2024-03-24T01:30:33.668 INFO:tasks.workunit.client.0.smithi090.stdout:} 2024-03-24T01:30:33.668 INFO:tasks.workunit.client.0.smithi090.stdout: 2024-03-24T01:30:33.669 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.663+0000 7f28b0e2c280 1 -- 172.21.15.90:0/1846431946 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x5652017b4800 msgr2=0x5652008e6100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.669 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.663+0000 7f28b0e2c280 1 --2- 172.21.15.90:0/1846431946 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x5652017b4800 0x5652008e6100 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.669 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.663+0000 7f28b0e2c280 1 -- 172.21.15.90:0/1846431946 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x5652017b5000 msgr2=0x5652008e5b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.669 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.663+0000 7f28b0e2c280 1 --2- 172.21.15.90:0/1846431946 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x5652017b5000 0x5652008e5b80 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.669 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.663+0000 7f28b0e2c280 1 -- 172.21.15.90:0/1846431946 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x5652017b4400 msgr2=0x5652008e7700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.670 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.663+0000 7f28b0e2c280 1 --2- 172.21.15.90:0/1846431946 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x5652017b4400 0x5652008e7700 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x56520095c810 tx=0x565200b70f00 comp rx=0 tx=0).stop 2024-03-24T01:30:33.670 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.663+0000 7f28b0e2c280 1 -- 172.21.15.90:0/1846431946 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56520174b400 msgr2=0x5652008e7180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.670 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.663+0000 7f28b0e2c280 1 --2- 172.21.15.90:0/1846431946 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56520174b400 0x5652008e7180 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x565201762870 tx=0x565200962e40 comp rx=0 tx=0).stop 2024-03-24T01:30:33.670 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.663+0000 7f28ae000640 1 -- 172.21.15.90:0/1846431946 reap_dead start 2024-03-24T01:30:33.670 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.663+0000 7f28ae000640 1 -- 172.21.15.90:0/1846431946 reap_dead start 2024-03-24T01:30:33.670 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.663+0000 7f28b0e2c280 1 -- 172.21.15.90:0/1846431946 shutdown_connections 2024-03-24T01:30:33.670 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.663+0000 7f28b0e2c280 1 -- 172.21.15.90:0/1846431946 >> 172.21.15.90:0/1846431946 conn(0x56520174b000 msgr2=0x5652015fb000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.670 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.663+0000 7f28b0e2c280 1 -- 172.21.15.90:0/1846431946 shutdown_connections 2024-03-24T01:30:33.670 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.663+0000 7f28b0e2c280 1 -- 172.21.15.90:0/1846431946 wait complete. 2024-03-24T01:30:33.670 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.664+0000 7f28b0e2c280 1 -- 172.21.15.90:0/3138486512 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56520087bc00 msgr2=0x5652008e5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.670 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.664+0000 7f28b0e2c280 1 --2- 172.21.15.90:0/3138486512 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56520087bc00 0x5652008e5080 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x5652016953e0 tx=0x565200963140 comp rx=0 tx=0).stop 2024-03-24T01:30:33.670 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.664+0000 7f28b0e2c280 1 -- 172.21.15.90:0/3138486512 shutdown_connections 2024-03-24T01:30:33.670 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.664+0000 7f28b0e2c280 1 --2- 172.21.15.90:0/3138486512 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56520087bc00 0x5652008e5080 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.670 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.664+0000 7f28b0e2c280 1 --2- 172.21.15.90:0/3138486512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56520087b800 0x5652008e4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.671 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.664+0000 7f28b0e2c280 1 --2- 172.21.15.90:0/3138486512 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56520087ac00 0x5652008e4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.671 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.664+0000 7f28b0e2c280 1 -- 172.21.15.90:0/3138486512 >> 172.21.15.90:0/3138486512 conn(0x56520087b000 msgr2=0x5652015fa800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.671 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.664+0000 7f28b0e2c280 1 -- 172.21.15.90:0/3138486512 shutdown_connections 2024-03-24T01:30:33.671 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.664+0000 7f28b0e2c280 1 -- 172.21.15.90:0/3138486512 wait complete. 2024-03-24T01:30:33.671 INFO:tasks.workunit.client.0.smithi090.stderr:+ cephfs-journal-tool --rank=cephfs:0 header set write_pos 123 2024-03-24T01:30:33.715 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.714+0000 7f25fb811280 1 -- start start 2024-03-24T01:30:33.715 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.714+0000 7f25fb811280 1 Processor -- start 2024-03-24T01:30:33.717 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.715+0000 7f25fb811280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644f50c0800 0x5644f43c9b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.717 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.715+0000 7f25fb811280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5644f435fc00 0x5644f43c8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.717 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.715+0000 7f25fb811280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5644f50c1c00 0x5644f43c9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.717 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.715+0000 7f25fb811280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644f5224000 con 0x5644f50c0800 2024-03-24T01:30:33.717 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.715+0000 7f25fb811280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644f5224180 con 0x5644f435fc00 2024-03-24T01:30:33.717 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.715+0000 7f25fb811280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5644f5224300 con 0x5644f50c1c00 2024-03-24T01:30:33.717 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.715+0000 7f25f8c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644f50c0800 0x5644f43c9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.717 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.715+0000 7f25f8c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644f50c0800 0x5644f43c9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36546/0 (socket says 172.21.15.90:36546) 2024-03-24T01:30:33.717 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.715+0000 7f25f8c00640 1 -- 172.21.15.90:0/2177304324 learned_addr learned my addr 172.21.15.90:0/2177304324 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.718 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.715+0000 7f25f9600640 1 --2- 172.21.15.90:0/2177304324 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5644f50c1c00 0x5644f43c9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.718 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.715+0000 7f25f8200640 1 --2- 172.21.15.90:0/2177304324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5644f435fc00 0x5644f43c8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.718 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.715+0000 7f25f8c00640 1 -- 172.21.15.90:0/2177304324 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5644f50c1c00 msgr2=0x5644f43c9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.718 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.715+0000 7f25f8c00640 1 --2- 172.21.15.90:0/2177304324 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5644f50c1c00 0x5644f43c9600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.718 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.715+0000 7f25f8c00640 1 -- 172.21.15.90:0/2177304324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5644f435fc00 msgr2=0x5644f43c8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.718 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.715+0000 7f25f8c00640 1 --2- 172.21.15.90:0/2177304324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5644f435fc00 0x5644f43c8b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.718 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.715+0000 7f25f8c00640 1 -- 172.21.15.90:0/2177304324 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5644f4392780 con 0x5644f50c0800 2024-03-24T01:30:33.718 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.716+0000 7f25f8c00640 1 --2- 172.21.15.90:0/2177304324 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644f50c0800 0x5644f43c9b80 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x5644f51895f0 tx=0x5644f5133200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f802c5a31c190d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.718 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.716+0000 7f25f5a00640 1 -- 172.21.15.90:0/2177304324 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5644f4384d00 con 0x5644f50c0800 2024-03-24T01:30:33.718 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.716+0000 7f25f5a00640 1 -- 172.21.15.90:0/2177304324 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5644f438a700 con 0x5644f50c0800 2024-03-24T01:30:33.718 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.716+0000 7f25f5a00640 1 -- 172.21.15.90:0/2177304324 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5644f4384680 con 0x5644f50c0800 2024-03-24T01:30:33.718 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.716+0000 7f25fb811280 1 -- 172.21.15.90:0/2177304324 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5644f4392000 con 0x5644f50c0800 2024-03-24T01:30:33.718 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.716+0000 7f25fb811280 1 -- 172.21.15.90:0/2177304324 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x5644f4393680 con 0x5644f50c0800 2024-03-24T01:30:33.721 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.716+0000 7f25f5a00640 1 -- 172.21.15.90:0/2177304324 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x5644f4447000 con 0x5644f50c0800 2024-03-24T01:30:33.721 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.716+0000 7f25f5a00640 1 -- 172.21.15.90:0/2177304324 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 36) v1 ==== 3310+0+0 (secure 0 0 0) 0x5644f43b9180 con 0x5644f50c0800 2024-03-24T01:30:33.721 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.717+0000 7f25fb811280 1 Processor -- start 2024-03-24T01:30:33.721 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.717+0000 7f25fb811280 1 -- start start 2024-03-24T01:30:33.721 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.718+0000 7f25fb811280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5644f50c0000 0x5644f43c9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.721 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.718+0000 7f25fb811280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644f5233800 0x5644f43ca100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.721 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.718+0000 7f25fb811280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5644f5233c00 0x5644f43ca680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.721 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.718+0000 7f25fb811280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644f5224480 con 0x5644f5233800 2024-03-24T01:30:33.721 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.718+0000 7f25fb811280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644f5224600 con 0x5644f50c0000 2024-03-24T01:30:33.721 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.718+0000 7f25fb811280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5644f5224780 con 0x5644f5233c00 2024-03-24T01:30:33.722 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.718+0000 7f25f8c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644f5233800 0x5644f43ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.722 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.718+0000 7f25f8c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644f5233800 0x5644f43ca100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36562/0 (socket says 172.21.15.90:36562) 2024-03-24T01:30:33.722 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.718+0000 7f25f8c00640 1 -- 172.21.15.90:0/2672133860 learned_addr learned my addr 172.21.15.90:0/2672133860 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.722 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.719+0000 7f25f9600640 1 --2- 172.21.15.90:0/2672133860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5644f50c0000 0x5644f43c9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.722 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.719+0000 7f25f8c00640 1 -- 172.21.15.90:0/2672133860 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5644f5233c00 msgr2=0x5644f43ca680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.722 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.719+0000 7f25f8c00640 1 --2- 172.21.15.90:0/2672133860 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5644f5233c00 0x5644f43ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.722 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.719+0000 7f25f8c00640 1 -- 172.21.15.90:0/2672133860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5644f50c0000 msgr2=0x5644f43c9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.722 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.719+0000 7f25f8c00640 1 --2- 172.21.15.90:0/2672133860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5644f50c0000 0x5644f43c9080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.722 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.719+0000 7f25f8c00640 1 -- 172.21.15.90:0/2672133860 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5644f4393860 con 0x5644f5233800 2024-03-24T01:30:33.722 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.719+0000 7f25f8c00640 1 --2- 172.21.15.90:0/2672133860 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644f5233800 0x5644f43ca100 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x5644f524e900 tx=0x5644f51322a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3318d8d57c24a5ed server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.722 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.719+0000 7f25f3200640 1 -- 172.21.15.90:0/2672133860 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5644f4384b60 con 0x5644f5233800 2024-03-24T01:30:33.722 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.719+0000 7f25f3200640 1 -- 172.21.15.90:0/2672133860 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5644f438ac40 con 0x5644f5233800 2024-03-24T01:30:33.722 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.719+0000 7f25f3200640 1 -- 172.21.15.90:0/2672133860 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5644f4384d00 con 0x5644f5233800 2024-03-24T01:30:33.723 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.719+0000 7f25fb811280 1 -- 172.21.15.90:0/2672133860 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644f5233800 msgr2=0x5644f43ca100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.723 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.719+0000 7f25fb811280 1 --2- 172.21.15.90:0/2672133860 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644f5233800 0x5644f43ca100 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x5644f524e900 tx=0x5644f51322a0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.723 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.719+0000 7f25fb811280 1 -- 172.21.15.90:0/2672133860 shutdown_connections 2024-03-24T01:30:33.723 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.719+0000 7f25fb811280 1 --2- 172.21.15.90:0/2672133860 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644f5233800 0x5644f43ca100 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.723 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.719+0000 7f25fb811280 1 --2- 172.21.15.90:0/2672133860 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5644f5233c00 0x5644f43ca680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.723 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.719+0000 7f25fb811280 1 --2- 172.21.15.90:0/2672133860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5644f50c0000 0x5644f43c9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.723 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.719+0000 7f25fb811280 1 -- 172.21.15.90:0/2672133860 >> 172.21.15.90:0/2672133860 conn(0x5644f50c0400 msgr2=0x5644f4468800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.723 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.720+0000 7f25fb811280 1 -- 172.21.15.90:0/2672133860 shutdown_connections 2024-03-24T01:30:33.723 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.720+0000 7f25fb811280 1 -- 172.21.15.90:0/2672133860 wait complete. 2024-03-24T01:30:33.725 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.720+0000 7f25fb811280 1 Processor -- start 2024-03-24T01:30:33.725 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.720+0000 7f25fb811280 1 -- start start 2024-03-24T01:30:33.725 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.721+0000 7f25fb811280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5644f50c0000 0x5644f43cb700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.725 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.721+0000 7f25fb811280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5644f5233800 0x5644f43cb180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.725 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.721+0000 7f25fb811280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644f5233c00 0x5644f43cac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.725 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.721+0000 7f25fb811280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644f5224a80 con 0x5644f5233c00 2024-03-24T01:30:33.726 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.721+0000 7f25fb811280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644f5224900 con 0x5644f50c0000 2024-03-24T01:30:33.726 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.721+0000 7f25fb811280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5644f5225c80 con 0x5644f5233800 2024-03-24T01:30:33.726 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.721+0000 7f25f8200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644f5233c00 0x5644f43cac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.726 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.721+0000 7f25f8200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644f5233c00 0x5644f43cac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36566/0 (socket says 172.21.15.90:36566) 2024-03-24T01:30:33.726 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.721+0000 7f25f8200640 1 -- 172.21.15.90:0/4175396611 learned_addr learned my addr 172.21.15.90:0/4175396611 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.726 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.722+0000 7f25f8c00640 1 --2- 172.21.15.90:0/4175396611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5644f5233800 0x5644f43cb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.726 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.722+0000 7f25f8200640 1 -- 172.21.15.90:0/4175396611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5644f5233800 msgr2=0x5644f43cb180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.726 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.722+0000 7f25f9600640 1 --2- 172.21.15.90:0/4175396611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5644f50c0000 0x5644f43cb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.726 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.722+0000 7f25f8200640 1 --2- 172.21.15.90:0/4175396611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5644f5233800 0x5644f43cb180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.726 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.722+0000 7f25f8200640 1 -- 172.21.15.90:0/4175396611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5644f50c0000 msgr2=0x5644f43cb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.726 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.722+0000 7f25f8200640 1 --2- 172.21.15.90:0/4175396611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5644f50c0000 0x5644f43cb700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.726 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.722+0000 7f25f8200640 1 -- 172.21.15.90:0/4175396611 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5644f4392780 con 0x5644f5233c00 2024-03-24T01:30:33.726 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.722+0000 7f25f8c00640 1 --2- 172.21.15.90:0/4175396611 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5644f5233800 0x5644f43cb180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:30:33.726 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.722+0000 7f25f8200640 1 --2- 172.21.15.90:0/4175396611 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644f5233c00 0x5644f43cac00 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x5644f524f590 tx=0x5644f5255a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.727 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.722+0000 7f25f1400640 1 -- 172.21.15.90:0/4175396611 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5644f4384b60 con 0x5644f5233c00 2024-03-24T01:30:33.727 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.722+0000 7f25f1400640 1 -- 172.21.15.90:0/4175396611 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5644f438a1c0 con 0x5644f5233c00 2024-03-24T01:30:33.727 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.722+0000 7f25f1400640 1 -- 172.21.15.90:0/4175396611 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5644f4384d00 con 0x5644f5233c00 2024-03-24T01:30:33.727 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.722+0000 7f25fb811280 1 -- 172.21.15.90:0/4175396611 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5644f4392000 con 0x5644f5233c00 2024-03-24T01:30:33.727 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.722+0000 7f25fb811280 1 -- 172.21.15.90:0/4175396611 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5644f4393a40 con 0x5644f5233c00 2024-03-24T01:30:33.727 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.724+0000 7f25f1400640 1 -- 172.21.15.90:0/4175396611 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x5644f5262c00 con 0x5644f5233c00 2024-03-24T01:30:33.728 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.724+0000 7f25f1400640 1 --2- 172.21.15.90:0/4175396611 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x5644f52b1000 0x5644f46e4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.729 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.724+0000 7f25f1400640 1 -- 172.21.15.90:0/4175396611 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x5644f52b2800 con 0x5644f5233c00 2024-03-24T01:30:33.729 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.725+0000 7f25fb811280 1 --2- 172.21.15.90:0/4175396611 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x5644f52b1400 0x5644f43ca680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.729 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.725+0000 7f25fb811280 1 -- 172.21.15.90:0/4175396611 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:1 7.f 7:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x5644f52b1800 con 0x5644f52b1400 2024-03-24T01:30:33.729 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.725+0000 7f25f8c00640 1 --2- 172.21.15.90:0/4175396611 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x5644f52b1400 0x5644f43ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.729 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.725+0000 7f25f9600640 1 --2- 172.21.15.90:0/4175396611 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x5644f52b1000 0x5644f46e4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.729 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.725+0000 7f25f8c00640 1 --2- 172.21.15.90:0/4175396611 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x5644f52b1400 0x5644f43ca680 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.729 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.726+0000 7f25f8c00640 1 -- 172.21.15.90:0/4175396611 <== osd.0 v2:172.21.15.90:6800/307045587 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x5644f50e2000 con 0x5644f52b1400 2024-03-24T01:30:33.729 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.726+0000 7f25fb811280 1 --2- 172.21.15.90:0/4175396611 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x5644f52b1c00 0x5644f43ca100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.729 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.726+0000 7f25fb811280 1 -- 172.21.15.90:0/4175396611 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:2 7.14 7:292cf221:::200.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x5644f5162000 con 0x5644f52b1c00 2024-03-24T01:30:33.729 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.726+0000 7f25f9600640 1 --2- 172.21.15.90:0/4175396611 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x5644f52b1c00 0x5644f43ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.729 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.726+0000 7f25f9600640 1 --2- 172.21.15.90:0/4175396611 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x5644f52b1000 0x5644f46e4000 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x5644f44dea50 tx=0x5644f5133020 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.729 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.726+0000 7f25f9600640 1 --2- 172.21.15.90:0/4175396611 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x5644f52b1c00 0x5644f43ca100 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.729 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.727+0000 7f25f9600640 1 -- 172.21.15.90:0/4175396611 <== osd.3 v2:172.21.15.90:6808/2999931983 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv28 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x5644f50e2500 con 0x5644f52b1c00 2024-03-24T01:30:33.730 INFO:tasks.workunit.client.0.smithi090.stdout:Updating write_pos 0x4016f9 -> 0x7b 2024-03-24T01:30:33.730 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.727+0000 7f25fb811280 1 -- 172.21.15.90:0/4175396611 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:3 7.14 7:292cf221:::200.00000000:head [writefull 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x5644f5162400 con 0x5644f52b1c00 2024-03-24T01:30:33.730 INFO:tasks.workunit.client.0.smithi090.stdout:Successfully updated header. 2024-03-24T01:30:33.732 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.729+0000 7f25f9600640 1 -- 172.21.15.90:0/4175396611 <== osd.3 v2:172.21.15.90:6808/2999931983 2 ==== osd_op_reply(3 200.00000000 [writefull 0~90] v56'31 uv31 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x5644f50e2500 con 0x5644f52b1c00 2024-03-24T01:30:33.732 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.729+0000 7f25fb811280 1 -- 172.21.15.90:0/4175396611 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x5644f52b1400 msgr2=0x5644f43ca680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.732 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.729+0000 7f25fb811280 1 --2- 172.21.15.90:0/4175396611 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x5644f52b1400 0x5644f43ca680 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.732 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.729+0000 7f25fb811280 1 -- 172.21.15.90:0/4175396611 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x5644f52b1c00 msgr2=0x5644f43ca100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.732 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.729+0000 7f25fb811280 1 --2- 172.21.15.90:0/4175396611 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x5644f52b1c00 0x5644f43ca100 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.732 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.729+0000 7f25fb811280 1 -- 172.21.15.90:0/4175396611 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x5644f52b1000 msgr2=0x5644f46e4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.732 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.729+0000 7f25fb811280 1 --2- 172.21.15.90:0/4175396611 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x5644f52b1000 0x5644f46e4000 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x5644f44dea50 tx=0x5644f5133020 comp rx=0 tx=0).stop 2024-03-24T01:30:33.732 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.729+0000 7f25fb811280 1 -- 172.21.15.90:0/4175396611 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644f5233c00 msgr2=0x5644f43cac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.733 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.729+0000 7f25fb811280 1 --2- 172.21.15.90:0/4175396611 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644f5233c00 0x5644f43cac00 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x5644f524f590 tx=0x5644f5255a40 comp rx=0 tx=0).stop 2024-03-24T01:30:33.733 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.729+0000 7f25f8200640 1 -- 172.21.15.90:0/4175396611 reap_dead start 2024-03-24T01:30:33.733 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.729+0000 7f25f8200640 1 -- 172.21.15.90:0/4175396611 reap_dead start 2024-03-24T01:30:33.733 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.729+0000 7f25fb811280 1 -- 172.21.15.90:0/4175396611 shutdown_connections 2024-03-24T01:30:33.733 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.729+0000 7f25fb811280 1 -- 172.21.15.90:0/4175396611 >> 172.21.15.90:0/4175396611 conn(0x5644f50c0400 msgr2=0x5644f4468800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.733 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.729+0000 7f25fb811280 1 -- 172.21.15.90:0/4175396611 shutdown_connections 2024-03-24T01:30:33.733 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.729+0000 7f25fb811280 1 -- 172.21.15.90:0/4175396611 wait complete. 2024-03-24T01:30:33.733 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.730+0000 7f25fb811280 1 -- 172.21.15.90:0/2177304324 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644f50c0800 msgr2=0x5644f43c9b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.733 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.730+0000 7f25fb811280 1 --2- 172.21.15.90:0/2177304324 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644f50c0800 0x5644f43c9b80 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x5644f51895f0 tx=0x5644f5133200 comp rx=0 tx=0).stop 2024-03-24T01:30:33.733 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.730+0000 7f25fb811280 1 -- 172.21.15.90:0/2177304324 shutdown_connections 2024-03-24T01:30:33.733 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.730+0000 7f25fb811280 1 --2- 172.21.15.90:0/2177304324 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5644f50c1c00 0x5644f43c9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.733 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.730+0000 7f25fb811280 1 --2- 172.21.15.90:0/2177304324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5644f435fc00 0x5644f43c8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.733 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.730+0000 7f25fb811280 1 --2- 172.21.15.90:0/2177304324 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644f50c0800 0x5644f43c9b80 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.733 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.730+0000 7f25fb811280 1 -- 172.21.15.90:0/2177304324 >> 172.21.15.90:0/2177304324 conn(0x5644f435f000 msgr2=0x5644f4469000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.734 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.730+0000 7f25fb811280 1 -- 172.21.15.90:0/2177304324 shutdown_connections 2024-03-24T01:30:33.734 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.730+0000 7f25fb811280 1 -- 172.21.15.90:0/2177304324 wait complete. 2024-03-24T01:30:33.736 INFO:tasks.workunit.client.0.smithi090.stderr:+ cephfs-journal-tool --rank=cephfs:0 header set expire_pos 123 2024-03-24T01:30:33.781 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.779+0000 7f75e2046280 1 -- start start 2024-03-24T01:30:33.781 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.779+0000 7f75e2046280 1 Processor -- start 2024-03-24T01:30:33.782 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.780+0000 7f75e2046280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d10c45c400 0x55d10b6f7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.782 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.780+0000 7f75e2046280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d10b68dc00 0x55d10b6f7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.782 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.780+0000 7f75e2046280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d10c45c800 0x55d10b6f7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.782 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.780+0000 7f75e2046280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d10c472480 con 0x55d10c45c800 2024-03-24T01:30:33.782 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.780+0000 7f75e2046280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d10c472600 con 0x55d10b68dc00 2024-03-24T01:30:33.782 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.780+0000 7f75e2046280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d10c472780 con 0x55d10c45c400 2024-03-24T01:30:33.782 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.781+0000 7f75e0400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d10c45c800 0x55d10b6f7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.782 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.781+0000 7f75e0400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d10c45c800 0x55d10b6f7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36582/0 (socket says 172.21.15.90:36582) 2024-03-24T01:30:33.782 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.781+0000 7f75e0400640 1 -- 172.21.15.90:0/2609290697 learned_addr learned my addr 172.21.15.90:0/2609290697 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.784 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.781+0000 7f75e0400640 1 -- 172.21.15.90:0/2609290697 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d10c45c400 msgr2=0x55d10b6f7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.784 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.781+0000 7f75dfa00640 1 --2- 172.21.15.90:0/2609290697 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d10c45c400 0x55d10b6f7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.784 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.781+0000 7f75df000640 1 --2- 172.21.15.90:0/2609290697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d10b68dc00 0x55d10b6f7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.784 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.781+0000 7f75e0400640 1 --2- 172.21.15.90:0/2609290697 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d10c45c400 0x55d10b6f7600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.784 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.781+0000 7f75e0400640 1 -- 172.21.15.90:0/2609290697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d10b68dc00 msgr2=0x55d10b6f7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.784 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.781+0000 7f75e0400640 1 --2- 172.21.15.90:0/2609290697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d10b68dc00 0x55d10b6f7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.784 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.781+0000 7f75e0400640 1 -- 172.21.15.90:0/2609290697 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d10b6c0780 con 0x55d10c45c800 2024-03-24T01:30:33.785 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.781+0000 7f75e0400640 1 --2- 172.21.15.90:0/2609290697 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d10c45c800 0x55d10b6f7080 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x55d10c4b1320 tx=0x55d10b775140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e1d01342c00a7a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.785 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.782+0000 7f75dc800640 1 -- 172.21.15.90:0/2609290697 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55d10b6b2d00 con 0x55d10c45c800 2024-03-24T01:30:33.785 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.782+0000 7f75dc800640 1 -- 172.21.15.90:0/2609290697 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55d10b6b8700 con 0x55d10c45c800 2024-03-24T01:30:33.785 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.782+0000 7f75dc800640 1 -- 172.21.15.90:0/2609290697 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55d10b6b2680 con 0x55d10c45c800 2024-03-24T01:30:33.785 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.782+0000 7f75e2046280 1 -- 172.21.15.90:0/2609290697 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d10b6c0000 con 0x55d10c45c800 2024-03-24T01:30:33.785 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.782+0000 7f75e2046280 1 -- 172.21.15.90:0/2609290697 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x55d10b6c1a40 con 0x55d10c45c800 2024-03-24T01:30:33.785 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.782+0000 7f75dc800640 1 -- 172.21.15.90:0/2609290697 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x55d10c55a600 con 0x55d10c45c800 2024-03-24T01:30:33.785 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.782+0000 7f75dc800640 1 -- 172.21.15.90:0/2609290697 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 36) v1 ==== 3310+0+0 (secure 0 0 0) 0x55d10b6e7180 con 0x55d10c45c800 2024-03-24T01:30:33.785 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.783+0000 7f75e2046280 1 Processor -- start 2024-03-24T01:30:33.785 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.783+0000 7f75e2046280 1 -- start start 2024-03-24T01:30:33.787 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.784+0000 7f75e2046280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d10c55cc00 0x55d10b6f6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.787 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.784+0000 7f75e2046280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d10c55d000 0x55d10b6f8100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.787 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.784+0000 7f75e2046280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d10c55d400 0x55d10b6f8680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.788 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.784+0000 7f75e2046280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d10b6ee180 con 0x55d10c55cc00 2024-03-24T01:30:33.788 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.784+0000 7f75e2046280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d10c473680 con 0x55d10c55d400 2024-03-24T01:30:33.788 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.784+0000 7f75e2046280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d10c473500 con 0x55d10c55d000 2024-03-24T01:30:33.788 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.784+0000 7f75df000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d10c55cc00 0x55d10b6f6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.788 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.784+0000 7f75df000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d10c55cc00 0x55d10b6f6b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36590/0 (socket says 172.21.15.90:36590) 2024-03-24T01:30:33.788 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.784+0000 7f75df000640 1 -- 172.21.15.90:0/587456793 learned_addr learned my addr 172.21.15.90:0/587456793 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.788 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.784+0000 7f75df000640 1 -- 172.21.15.90:0/587456793 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d10c55d000 msgr2=0x55d10b6f8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.788 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.784+0000 7f75df000640 1 --2- 172.21.15.90:0/587456793 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d10c55d000 0x55d10b6f8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.788 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.784+0000 7f75df000640 1 -- 172.21.15.90:0/587456793 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d10c55d400 msgr2=0x55d10b6f8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.788 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.784+0000 7f75df000640 1 --2- 172.21.15.90:0/587456793 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d10c55d400 0x55d10b6f8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.788 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.784+0000 7f75df000640 1 -- 172.21.15.90:0/587456793 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d10b6c1860 con 0x55d10c55cc00 2024-03-24T01:30:33.788 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.784+0000 7f75df000640 1 --2- 172.21.15.90:0/587456793 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d10c55cc00 0x55d10b6f6b00 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x55d10c5745d0 tx=0x55d10b774f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fed584c06caab8ad server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.788 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.785+0000 7f75da000640 1 -- 172.21.15.90:0/587456793 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55d10b6b2b60 con 0x55d10c55cc00 2024-03-24T01:30:33.788 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.785+0000 7f75da000640 1 -- 172.21.15.90:0/587456793 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55d10b6b8fc0 con 0x55d10c55cc00 2024-03-24T01:30:33.789 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.785+0000 7f75da000640 1 -- 172.21.15.90:0/587456793 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55d10b6b2d00 con 0x55d10c55cc00 2024-03-24T01:30:33.789 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.785+0000 7f75e2046280 1 -- 172.21.15.90:0/587456793 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d10c55cc00 msgr2=0x55d10b6f6b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.789 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.785+0000 7f75e2046280 1 --2- 172.21.15.90:0/587456793 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d10c55cc00 0x55d10b6f6b00 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x55d10c5745d0 tx=0x55d10b774f60 comp rx=0 tx=0).stop 2024-03-24T01:30:33.789 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.785+0000 7f75e2046280 1 -- 172.21.15.90:0/587456793 shutdown_connections 2024-03-24T01:30:33.789 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.785+0000 7f75e2046280 1 --2- 172.21.15.90:0/587456793 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d10c55d400 0x55d10b6f8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.789 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.785+0000 7f75e2046280 1 --2- 172.21.15.90:0/587456793 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d10c55d000 0x55d10b6f8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.789 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.785+0000 7f75e2046280 1 --2- 172.21.15.90:0/587456793 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d10c55cc00 0x55d10b6f6b00 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.789 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.785+0000 7f75e2046280 1 -- 172.21.15.90:0/587456793 >> 172.21.15.90:0/587456793 conn(0x55d10c45c000 msgr2=0x55d10c40d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.789 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.785+0000 7f75e2046280 1 -- 172.21.15.90:0/587456793 shutdown_connections 2024-03-24T01:30:33.789 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.785+0000 7f75e2046280 1 -- 172.21.15.90:0/587456793 wait complete. 2024-03-24T01:30:33.789 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.786+0000 7f75e2046280 1 Processor -- start 2024-03-24T01:30:33.789 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.786+0000 7f75e2046280 1 -- start start 2024-03-24T01:30:33.792 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.787+0000 7f75e2046280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d10c55cc00 0x55d10b6f6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.792 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.787+0000 7f75e2046280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d10c55d000 0x55d10b6f8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.792 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.787+0000 7f75e2046280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d10c55d400 0x55d10b6f8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.792 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.787+0000 7f75e2046280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d10c472900 con 0x55d10c55d000 2024-03-24T01:30:33.792 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.787+0000 7f75e2046280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d10c472a80 con 0x55d10c55d400 2024-03-24T01:30:33.792 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.787+0000 7f75e2046280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d10c472180 con 0x55d10c55cc00 2024-03-24T01:30:33.792 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.787+0000 7f75dfa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d10c55d000 0x55d10b6f8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.792 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.787+0000 7f75dfa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d10c55d000 0x55d10b6f8100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36592/0 (socket says 172.21.15.90:36592) 2024-03-24T01:30:33.792 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.787+0000 7f75dfa00640 1 -- 172.21.15.90:0/2518687606 learned_addr learned my addr 172.21.15.90:0/2518687606 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.792 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.787+0000 7f75dfa00640 1 -- 172.21.15.90:0/2518687606 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d10c55cc00 msgr2=0x55d10b6f6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.792 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.787+0000 7f75dfa00640 1 --2- 172.21.15.90:0/2518687606 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d10c55cc00 0x55d10b6f6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.793 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.787+0000 7f75dfa00640 1 -- 172.21.15.90:0/2518687606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d10c55d400 msgr2=0x55d10b6f8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.793 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.787+0000 7f75dfa00640 1 --2- 172.21.15.90:0/2518687606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d10c55d400 0x55d10b6f8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.793 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.787+0000 7f75dfa00640 1 -- 172.21.15.90:0/2518687606 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d10b6c0780 con 0x55d10c55d000 2024-03-24T01:30:33.793 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.788+0000 7f75dfa00640 1 --2- 172.21.15.90:0/2518687606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d10c55d000 0x55d10b6f8100 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x55d10c575320 tx=0x55d10c5778c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.793 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.788+0000 7f75d8200640 1 -- 172.21.15.90:0/2518687606 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55d10b6b2b60 con 0x55d10c55d000 2024-03-24T01:30:33.793 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.788+0000 7f75d8200640 1 -- 172.21.15.90:0/2518687606 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55d10b6b8c40 con 0x55d10c55d000 2024-03-24T01:30:33.793 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.788+0000 7f75d8200640 1 -- 172.21.15.90:0/2518687606 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55d10b6b2d00 con 0x55d10c55d000 2024-03-24T01:30:33.793 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.788+0000 7f75e2046280 1 -- 172.21.15.90:0/2518687606 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55d10b6c0000 con 0x55d10c55d000 2024-03-24T01:30:33.793 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.788+0000 7f75e2046280 1 -- 172.21.15.90:0/2518687606 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d10b6c1680 con 0x55d10c55d000 2024-03-24T01:30:33.793 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.789+0000 7f75d8200640 1 -- 172.21.15.90:0/2518687606 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x55d10c48cc00 con 0x55d10c55d000 2024-03-24T01:30:33.793 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.790+0000 7f75d8200640 1 --2- 172.21.15.90:0/2518687606 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55d10c5cb000 0x55d10b6f9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.793 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.790+0000 7f75d8200640 1 -- 172.21.15.90:0/2518687606 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x55d10c48aa00 con 0x55d10c55d000 2024-03-24T01:30:33.794 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.790+0000 7f75e2046280 1 --2- 172.21.15.90:0/2518687606 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55d10c55d800 0x55d10b6f8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.794 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.790+0000 7f75e2046280 1 -- 172.21.15.90:0/2518687606 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:1 7.f 7:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55d10c55dc00 con 0x55d10c55d800 2024-03-24T01:30:33.794 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.790+0000 7f75df000640 1 --2- 172.21.15.90:0/2518687606 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55d10c55d800 0x55d10b6f8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.794 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.791+0000 7f75df000640 1 --2- 172.21.15.90:0/2518687606 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55d10c5cb000 0x55d10b6f9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.795 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.791+0000 7f75df000640 1 --2- 172.21.15.90:0/2518687606 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55d10c55d800 0x55d10b6f8c00 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.795 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.791+0000 7f75df000640 1 -- 172.21.15.90:0/2518687606 <== osd.0 v2:172.21.15.90:6800/307045587 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x55d10c410000 con 0x55d10c55d800 2024-03-24T01:30:33.795 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.791+0000 7f75e2046280 1 --2- 172.21.15.90:0/2518687606 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55d10c5cb400 0x55d10b6f9180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.795 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.791+0000 7f75e2046280 1 -- 172.21.15.90:0/2518687606 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:2 7.14 7:292cf221:::200.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55d10c5cb800 con 0x55d10c5cb400 2024-03-24T01:30:33.795 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.792+0000 7f75df000640 1 --2- 172.21.15.90:0/2518687606 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55d10c5cb000 0x55d10b6f9700 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x55d10c574840 tx=0x55d10b774e40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.795 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.792+0000 7f75e0400640 1 --2- 172.21.15.90:0/2518687606 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55d10c5cb400 0x55d10b6f9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.795 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.792+0000 7f75e0400640 1 --2- 172.21.15.90:0/2518687606 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55d10c5cb400 0x55d10b6f9180 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.795 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.792+0000 7f75e0400640 1 -- 172.21.15.90:0/2518687606 <== osd.3 v2:172.21.15.90:6808/2999931983 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv31 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x55d10c410500 con 0x55d10c5cb400 2024-03-24T01:30:33.796 INFO:tasks.workunit.client.0.smithi090.stdout:Updating expire_pos 0x400000 -> 0x7b 2024-03-24T01:30:33.796 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.792+0000 7f75e2046280 -1 Header is invalid (inconsistent offsets) 2024-03-24T01:30:33.796 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.793+0000 7f75e2046280 1 -- 172.21.15.90:0/2518687606 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:3 7.14 7:292cf221:::200.00000000:head [writefull 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55d10c5cbc00 con 0x55d10c5cb400 2024-03-24T01:30:33.797 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.795+0000 7f75e0400640 1 -- 172.21.15.90:0/2518687606 <== osd.3 v2:172.21.15.90:6808/2999931983 2 ==== osd_op_reply(3 200.00000000 [writefull 0~90] v56'32 uv32 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55d10c410500 con 0x55d10c5cb400 2024-03-24T01:30:33.797 INFO:tasks.workunit.client.0.smithi090.stdout:Successfully updated header. 2024-03-24T01:30:33.798 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.795+0000 7f75e2046280 1 -- 172.21.15.90:0/2518687606 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55d10c55d800 msgr2=0x55d10b6f8c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.798 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.795+0000 7f75e2046280 1 --2- 172.21.15.90:0/2518687606 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55d10c55d800 0x55d10b6f8c00 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.798 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.795+0000 7f75e2046280 1 -- 172.21.15.90:0/2518687606 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55d10c5cb400 msgr2=0x55d10b6f9180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.798 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.795+0000 7f75e2046280 1 --2- 172.21.15.90:0/2518687606 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55d10c5cb400 0x55d10b6f9180 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.798 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.795+0000 7f75e2046280 1 -- 172.21.15.90:0/2518687606 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55d10c5cb000 msgr2=0x55d10b6f9700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.798 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.795+0000 7f75e2046280 1 --2- 172.21.15.90:0/2518687606 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55d10c5cb000 0x55d10b6f9700 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x55d10c574840 tx=0x55d10b774e40 comp rx=0 tx=0).stop 2024-03-24T01:30:33.798 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.795+0000 7f75e2046280 1 -- 172.21.15.90:0/2518687606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d10c55d000 msgr2=0x55d10b6f8100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.798 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.795+0000 7f75e2046280 1 --2- 172.21.15.90:0/2518687606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d10c55d000 0x55d10b6f8100 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x55d10c575320 tx=0x55d10c5778c0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.799 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.795+0000 7f75dfa00640 1 -- 172.21.15.90:0/2518687606 reap_dead start 2024-03-24T01:30:33.799 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.795+0000 7f75dfa00640 1 -- 172.21.15.90:0/2518687606 reap_dead start 2024-03-24T01:30:33.799 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.795+0000 7f75e2046280 1 -- 172.21.15.90:0/2518687606 shutdown_connections 2024-03-24T01:30:33.799 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.795+0000 7f75e2046280 1 -- 172.21.15.90:0/2518687606 >> 172.21.15.90:0/2518687606 conn(0x55d10c45c000 msgr2=0x55d10c40d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.799 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.795+0000 7f75e2046280 1 -- 172.21.15.90:0/2518687606 shutdown_connections 2024-03-24T01:30:33.799 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.795+0000 7f75e2046280 1 -- 172.21.15.90:0/2518687606 wait complete. 2024-03-24T01:30:33.799 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.795+0000 7f75e2046280 1 -- 172.21.15.90:0/2609290697 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d10c45c800 msgr2=0x55d10b6f7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.799 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.795+0000 7f75e2046280 1 --2- 172.21.15.90:0/2609290697 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d10c45c800 0x55d10b6f7080 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x55d10c4b1320 tx=0x55d10b775140 comp rx=0 tx=0).stop 2024-03-24T01:30:33.799 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.795+0000 7f75e2046280 1 -- 172.21.15.90:0/2609290697 shutdown_connections 2024-03-24T01:30:33.799 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.795+0000 7f75e2046280 1 --2- 172.21.15.90:0/2609290697 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d10c45c800 0x55d10b6f7080 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.799 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.796+0000 7f75e2046280 1 --2- 172.21.15.90:0/2609290697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d10b68dc00 0x55d10b6f7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.799 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.796+0000 7f75e2046280 1 --2- 172.21.15.90:0/2609290697 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55d10c45c400 0x55d10b6f7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.799 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.796+0000 7f75e2046280 1 -- 172.21.15.90:0/2609290697 >> 172.21.15.90:0/2609290697 conn(0x55d10b68d000 msgr2=0x55d10c40d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.799 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.796+0000 7f75e2046280 1 -- 172.21.15.90:0/2609290697 shutdown_connections 2024-03-24T01:30:33.800 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.796+0000 7f75e2046280 1 -- 172.21.15.90:0/2609290697 wait complete. 2024-03-24T01:30:33.801 INFO:tasks.workunit.client.0.smithi090.stderr:+ cephfs-journal-tool --rank=cephfs:0 header set trimmed_pos 123 2024-03-24T01:30:33.845 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.844+0000 7f4443850280 1 -- start start 2024-03-24T01:30:33.846 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.844+0000 7f4443850280 1 Processor -- start 2024-03-24T01:30:33.848 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.845+0000 7f4443850280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617c86fa000 0x5617c799d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.848 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.845+0000 7f4443850280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5617c86fb000 0x5617c799cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.848 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.845+0000 7f4443850280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617c86fb400 0x5617c799d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.848 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.845+0000 7f4443850280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617c8702000 con 0x5617c86fa000 2024-03-24T01:30:33.848 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.845+0000 7f4443850280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617c8702180 con 0x5617c86fb400 2024-03-24T01:30:33.848 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.845+0000 7f4443850280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5617c8702300 con 0x5617c86fb000 2024-03-24T01:30:33.848 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.845+0000 7f4441000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617c86fa000 0x5617c799d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.848 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.845+0000 7f4441000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617c86fa000 0x5617c799d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36612/0 (socket says 172.21.15.90:36612) 2024-03-24T01:30:33.848 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.845+0000 7f4441000640 1 -- 172.21.15.90:0/1969272641 learned_addr learned my addr 172.21.15.90:0/1969272641 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.848 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.846+0000 7f4441000640 1 -- 172.21.15.90:0/1969272641 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5617c86fb000 msgr2=0x5617c799cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.848 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.846+0000 7f4441000640 1 --2- 172.21.15.90:0/1969272641 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5617c86fb000 0x5617c799cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.848 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.846+0000 7f4441000640 1 -- 172.21.15.90:0/1969272641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617c86fb400 msgr2=0x5617c799d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.848 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.846+0000 7f4441000640 1 --2- 172.21.15.90:0/1969272641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617c86fb400 0x5617c799d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.849 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.846+0000 7f4441000640 1 -- 172.21.15.90:0/1969272641 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5617c7966780 con 0x5617c86fa000 2024-03-24T01:30:33.849 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.846+0000 7f4441000640 1 --2- 172.21.15.90:0/1969272641 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617c86fa000 0x5617c799d080 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x5617c875d200 tx=0x5617c7a1af60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e090a748ef6e2cf0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.849 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.846+0000 7f443de00640 1 -- 172.21.15.90:0/1969272641 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5617c7958680 con 0x5617c86fa000 2024-03-24T01:30:33.849 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.846+0000 7f443de00640 1 -- 172.21.15.90:0/1969272641 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5617c795ea80 con 0x5617c86fa000 2024-03-24T01:30:33.849 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.846+0000 7f443de00640 1 -- 172.21.15.90:0/1969272641 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5617c79589c0 con 0x5617c86fa000 2024-03-24T01:30:33.849 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.846+0000 7f4443850280 1 -- 172.21.15.90:0/1969272641 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5617c7966000 con 0x5617c86fa000 2024-03-24T01:30:33.849 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.846+0000 7f4443850280 1 -- 172.21.15.90:0/1969272641 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x5617c7967a40 con 0x5617c86fa000 2024-03-24T01:30:33.849 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.847+0000 7f443de00640 1 -- 172.21.15.90:0/1969272641 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x5617c8730400 con 0x5617c86fa000 2024-03-24T01:30:33.849 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.847+0000 7f443de00640 1 -- 172.21.15.90:0/1969272641 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 36) v1 ==== 3310+0+0 (secure 0 0 0) 0x5617c798d180 con 0x5617c86fa000 2024-03-24T01:30:33.851 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.848+0000 7f4443850280 1 Processor -- start 2024-03-24T01:30:33.851 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.848+0000 7f4443850280 1 -- start start 2024-03-24T01:30:33.852 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.848+0000 7f4443850280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617c86fbc00 0x5617c799c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.852 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.848+0000 7f4443850280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5617c8810400 0x5617c799e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.852 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.848+0000 7f4443850280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617c8810800 0x5617c799e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.852 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.848+0000 7f4443850280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617c8702480 con 0x5617c8810800 2024-03-24T01:30:33.852 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.848+0000 7f4443850280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617c8702600 con 0x5617c86fbc00 2024-03-24T01:30:33.852 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.848+0000 7f4443850280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5617c8702780 con 0x5617c8810400 2024-03-24T01:30:33.852 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.849+0000 7f4440600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617c8810800 0x5617c799e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.852 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.849+0000 7f4440600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617c8810800 0x5617c799e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36624/0 (socket says 172.21.15.90:36624) 2024-03-24T01:30:33.852 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.849+0000 7f4440600640 1 -- 172.21.15.90:0/270267333 learned_addr learned my addr 172.21.15.90:0/270267333 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.852 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.849+0000 7f4440600640 1 -- 172.21.15.90:0/270267333 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5617c8810400 msgr2=0x5617c799e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.852 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.849+0000 7f4440600640 1 --2- 172.21.15.90:0/270267333 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5617c8810400 0x5617c799e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.852 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.849+0000 7f4440600640 1 -- 172.21.15.90:0/270267333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617c86fbc00 msgr2=0x5617c799c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.852 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.849+0000 7f4440600640 1 --2- 172.21.15.90:0/270267333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617c86fbc00 0x5617c799c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.852 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.849+0000 7f4440600640 1 -- 172.21.15.90:0/270267333 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5617c7967860 con 0x5617c8810800 2024-03-24T01:30:33.853 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.849+0000 7f4440600640 1 --2- 172.21.15.90:0/270267333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617c8810800 0x5617c799e680 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x5617c880e660 tx=0x5617c8812c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d17c0f4409af9d06 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.853 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.849+0000 7f443b600640 1 -- 172.21.15.90:0/270267333 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5617c7958680 con 0x5617c8810800 2024-03-24T01:30:33.853 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.850+0000 7f443b600640 1 -- 172.21.15.90:0/270267333 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5617c795ec40 con 0x5617c8810800 2024-03-24T01:30:33.853 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.850+0000 7f443b600640 1 -- 172.21.15.90:0/270267333 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5617c79589c0 con 0x5617c8810800 2024-03-24T01:30:33.853 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.850+0000 7f4443850280 1 -- 172.21.15.90:0/270267333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617c8810800 msgr2=0x5617c799e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.853 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.850+0000 7f4443850280 1 --2- 172.21.15.90:0/270267333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617c8810800 0x5617c799e680 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x5617c880e660 tx=0x5617c8812c00 comp rx=0 tx=0).stop 2024-03-24T01:30:33.853 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.850+0000 7f4443850280 1 -- 172.21.15.90:0/270267333 shutdown_connections 2024-03-24T01:30:33.853 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.850+0000 7f4443850280 1 --2- 172.21.15.90:0/270267333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617c8810800 0x5617c799e680 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.853 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.850+0000 7f4443850280 1 --2- 172.21.15.90:0/270267333 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5617c8810400 0x5617c799e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.853 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.850+0000 7f4443850280 1 --2- 172.21.15.90:0/270267333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617c86fbc00 0x5617c799c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.853 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.850+0000 7f4443850280 1 -- 172.21.15.90:0/270267333 >> 172.21.15.90:0/270267333 conn(0x5617c86fb800 msgr2=0x5617c86b3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.853 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.850+0000 7f4443850280 1 -- 172.21.15.90:0/270267333 shutdown_connections 2024-03-24T01:30:33.854 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.850+0000 7f4443850280 1 -- 172.21.15.90:0/270267333 wait complete. 2024-03-24T01:30:33.855 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.851+0000 7f4443850280 1 Processor -- start 2024-03-24T01:30:33.856 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.851+0000 7f4443850280 1 -- start start 2024-03-24T01:30:33.856 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.852+0000 7f4443850280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5617c86fb800 0x5617c799f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.856 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.852+0000 7f4443850280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617c8811400 0x5617c799f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.856 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.852+0000 7f4443850280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617c8811000 0x5617c799ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.856 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.852+0000 7f4443850280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617c8703b00 con 0x5617c8811000 2024-03-24T01:30:33.856 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.852+0000 7f4443850280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617c8703980 con 0x5617c8811400 2024-03-24T01:30:33.856 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.852+0000 7f4443850280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5617c8702c00 con 0x5617c86fb800 2024-03-24T01:30:33.856 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.852+0000 7f4440600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617c8811000 0x5617c799ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.856 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.852+0000 7f4440600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617c8811000 0x5617c799ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36626/0 (socket says 172.21.15.90:36626) 2024-03-24T01:30:33.856 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.852+0000 7f4440600640 1 -- 172.21.15.90:0/1019307452 learned_addr learned my addr 172.21.15.90:0/1019307452 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.856 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.852+0000 7f4440600640 1 -- 172.21.15.90:0/1019307452 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5617c86fb800 msgr2=0x5617c799f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.856 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.852+0000 7f4440600640 1 --2- 172.21.15.90:0/1019307452 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5617c86fb800 0x5617c799f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.856 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.852+0000 7f4440600640 1 -- 172.21.15.90:0/1019307452 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617c8811400 msgr2=0x5617c799f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.857 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.852+0000 7f4440600640 1 --2- 172.21.15.90:0/1019307452 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617c8811400 0x5617c799f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.857 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.852+0000 7f4440600640 1 -- 172.21.15.90:0/1019307452 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5617c7967680 con 0x5617c8811000 2024-03-24T01:30:33.857 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.852+0000 7f4440600640 1 --2- 172.21.15.90:0/1019307452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617c8811000 0x5617c799ec00 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x5617c880f320 tx=0x5617c7a1a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.857 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.853+0000 7f4439800640 1 -- 172.21.15.90:0/1019307452 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5617c79584e0 con 0x5617c8811000 2024-03-24T01:30:33.857 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.853+0000 7f4439800640 1 -- 172.21.15.90:0/1019307452 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5617c795e1c0 con 0x5617c8811000 2024-03-24T01:30:33.857 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.853+0000 7f4439800640 1 -- 172.21.15.90:0/1019307452 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5617c7958680 con 0x5617c8811000 2024-03-24T01:30:33.857 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.853+0000 7f4443850280 1 -- 172.21.15.90:0/1019307452 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5617c79663c0 con 0x5617c8811000 2024-03-24T01:30:33.857 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.853+0000 7f4443850280 1 -- 172.21.15.90:0/1019307452 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5617c7967c20 con 0x5617c8811000 2024-03-24T01:30:33.857 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.854+0000 7f4439800640 1 -- 172.21.15.90:0/1019307452 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x5617c886ac00 con 0x5617c8811000 2024-03-24T01:30:33.859 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.855+0000 7f4439800640 1 --2- 172.21.15.90:0/1019307452 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x5617c8871000 0x5617c7c9e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.859 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.855+0000 7f4439800640 1 -- 172.21.15.90:0/1019307452 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x5617c8864a00 con 0x5617c8811000 2024-03-24T01:30:33.859 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.855+0000 7f4443850280 1 --2- 172.21.15.90:0/1019307452 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x5617c8810c00 0x5617c799e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.859 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.855+0000 7f4443850280 1 -- 172.21.15.90:0/1019307452 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:1 7.f 7:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x5617c8810800 con 0x5617c8810c00 2024-03-24T01:30:33.859 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.855+0000 7f4441000640 1 --2- 172.21.15.90:0/1019307452 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x5617c8810c00 0x5617c799e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.859 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.855+0000 7f4441a00640 1 --2- 172.21.15.90:0/1019307452 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x5617c8871000 0x5617c7c9e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.859 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.856+0000 7f4441000640 1 --2- 172.21.15.90:0/1019307452 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x5617c8810c00 0x5617c799e680 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.859 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.856+0000 7f4441000640 1 -- 172.21.15.90:0/1019307452 <== osd.0 v2:172.21.15.90:6800/307045587 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x5617c86b6000 con 0x5617c8810c00 2024-03-24T01:30:33.860 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.856+0000 7f4443850280 1 --2- 172.21.15.90:0/1019307452 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x5617c8810400 0x5617c799e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.860 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.856+0000 7f4443850280 1 -- 172.21.15.90:0/1019307452 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:2 7.14 7:292cf221:::200.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x5617c7cbc000 con 0x5617c8810400 2024-03-24T01:30:33.860 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.857+0000 7f4441a00640 1 --2- 172.21.15.90:0/1019307452 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x5617c8810400 0x5617c799e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.860 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.857+0000 7f4441a00640 1 --2- 172.21.15.90:0/1019307452 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x5617c8871000 0x5617c7c9e000 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x5617c7a148a0 tx=0x5617c7caa0c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.860 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.857+0000 7f4441a00640 1 --2- 172.21.15.90:0/1019307452 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x5617c8810400 0x5617c799e100 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.860 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.857+0000 7f4441a00640 1 -- 172.21.15.90:0/1019307452 <== osd.3 v2:172.21.15.90:6808/2999931983 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv32 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x5617c86b6500 con 0x5617c8810400 2024-03-24T01:30:33.860 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.857+0000 7f4443850280 -1 Header is invalid (inconsistent offsets) 2024-03-24T01:30:33.860 INFO:tasks.workunit.client.0.smithi090.stdout:Updating trimmed_pos 0x400000 -> 0x7b 2024-03-24T01:30:33.861 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.857+0000 7f4443850280 1 -- 172.21.15.90:0/1019307452 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:3 7.14 7:292cf221:::200.00000000:head [writefull 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x5617c7cbc400 con 0x5617c8810400 2024-03-24T01:30:33.861 INFO:tasks.workunit.client.0.smithi090.stdout:Successfully updated header. 2024-03-24T01:30:33.862 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.859+0000 7f4441a00640 1 -- 172.21.15.90:0/1019307452 <== osd.3 v2:172.21.15.90:6808/2999931983 2 ==== osd_op_reply(3 200.00000000 [writefull 0~90] v56'33 uv33 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x5617c86b6500 con 0x5617c8810400 2024-03-24T01:30:33.862 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.859+0000 7f4443850280 1 -- 172.21.15.90:0/1019307452 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x5617c8810c00 msgr2=0x5617c799e680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.863 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.859+0000 7f4443850280 1 --2- 172.21.15.90:0/1019307452 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x5617c8810c00 0x5617c799e680 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.863 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.859+0000 7f4443850280 1 -- 172.21.15.90:0/1019307452 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x5617c8810400 msgr2=0x5617c799e100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.863 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.859+0000 7f4443850280 1 --2- 172.21.15.90:0/1019307452 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x5617c8810400 0x5617c799e100 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.863 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.859+0000 7f4443850280 1 -- 172.21.15.90:0/1019307452 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x5617c8871000 msgr2=0x5617c7c9e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.863 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.859+0000 7f4443850280 1 --2- 172.21.15.90:0/1019307452 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x5617c8871000 0x5617c7c9e000 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x5617c7a148a0 tx=0x5617c7caa0c0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.863 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.859+0000 7f4443850280 1 -- 172.21.15.90:0/1019307452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617c8811000 msgr2=0x5617c799ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.863 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.859+0000 7f4443850280 1 --2- 172.21.15.90:0/1019307452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617c8811000 0x5617c799ec00 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x5617c880f320 tx=0x5617c7a1a000 comp rx=0 tx=0).stop 2024-03-24T01:30:33.863 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.859+0000 7f4440600640 1 -- 172.21.15.90:0/1019307452 reap_dead start 2024-03-24T01:30:33.863 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.859+0000 7f4440600640 1 -- 172.21.15.90:0/1019307452 reap_dead start 2024-03-24T01:30:33.863 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.859+0000 7f4443850280 1 -- 172.21.15.90:0/1019307452 shutdown_connections 2024-03-24T01:30:33.863 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.859+0000 7f4443850280 1 -- 172.21.15.90:0/1019307452 >> 172.21.15.90:0/1019307452 conn(0x5617c86fbc00 msgr2=0x5617c86b3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.863 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.859+0000 7f4443850280 1 -- 172.21.15.90:0/1019307452 shutdown_connections 2024-03-24T01:30:33.863 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.859+0000 7f4443850280 1 -- 172.21.15.90:0/1019307452 wait complete. 2024-03-24T01:30:33.864 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.860+0000 7f4443850280 1 -- 172.21.15.90:0/1969272641 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617c86fa000 msgr2=0x5617c799d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.864 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.860+0000 7f4443850280 1 --2- 172.21.15.90:0/1969272641 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617c86fa000 0x5617c799d080 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x5617c875d200 tx=0x5617c7a1af60 comp rx=0 tx=0).stop 2024-03-24T01:30:33.864 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.860+0000 7f4443850280 1 -- 172.21.15.90:0/1969272641 shutdown_connections 2024-03-24T01:30:33.864 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.860+0000 7f4443850280 1 --2- 172.21.15.90:0/1969272641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617c86fb400 0x5617c799d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.864 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.860+0000 7f4443850280 1 --2- 172.21.15.90:0/1969272641 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5617c86fb000 0x5617c799cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.864 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.860+0000 7f4443850280 1 --2- 172.21.15.90:0/1969272641 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5617c86fa000 0x5617c799d080 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.864 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.860+0000 7f4443850280 1 -- 172.21.15.90:0/1969272641 >> 172.21.15.90:0/1969272641 conn(0x5617c7933000 msgr2=0x5617c86b2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.864 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.860+0000 7f4443850280 1 -- 172.21.15.90:0/1969272641 shutdown_connections 2024-03-24T01:30:33.864 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.860+0000 7f4443850280 1 -- 172.21.15.90:0/1969272641 wait complete. 2024-03-24T01:30:33.866 INFO:tasks.workunit.client.0.smithi090.stdout:Rolling back journal to original state... 2024-03-24T01:30:33.866 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'Rolling back journal to original state...' 2024-03-24T01:30:33.866 INFO:tasks.workunit.client.0.smithi090.stderr:+ cephfs-journal-tool --rank=cephfs:0 journal import /tmp/journal.bin 2024-03-24T01:30:33.909 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.908+0000 7f9b0fc0f280 1 -- start start 2024-03-24T01:30:33.909 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.908+0000 7f9b0fc0f280 1 Processor -- start 2024-03-24T01:30:33.911 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.909+0000 7f9b0fc0f280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e56149cc00 0x55e561506580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.911 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.909+0000 7f9b0fc0f280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e56149d800 0x55e561506b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.911 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.909+0000 7f9b0fc0f280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e56149dc00 0x55e561507080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.911 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.909+0000 7f9b0fc0f280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e562262180 con 0x55e56149dc00 2024-03-24T01:30:33.911 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.909+0000 7f9b0fc0f280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e562262300 con 0x55e56149cc00 2024-03-24T01:30:33.912 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.909+0000 7f9b0fc0f280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e562262480 con 0x55e56149d800 2024-03-24T01:30:33.912 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.909+0000 7f9b0dc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e56149dc00 0x55e561507080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.912 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.909+0000 7f9b0dc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e56149dc00 0x55e561507080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36652/0 (socket says 172.21.15.90:36652) 2024-03-24T01:30:33.912 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.909+0000 7f9b0dc00640 1 -- 172.21.15.90:0/2207529143 learned_addr learned my addr 172.21.15.90:0/2207529143 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.912 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.909+0000 7f9b0dc00640 1 -- 172.21.15.90:0/2207529143 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e56149d800 msgr2=0x55e561506b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.912 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.909+0000 7f9b0dc00640 1 --2- 172.21.15.90:0/2207529143 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e56149d800 0x55e561506b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.912 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.909+0000 7f9b0dc00640 1 -- 172.21.15.90:0/2207529143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e56149cc00 msgr2=0x55e561506580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.912 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.909+0000 7f9b0dc00640 1 --2- 172.21.15.90:0/2207529143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e56149cc00 0x55e561506580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.912 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.909+0000 7f9b0dc00640 1 -- 172.21.15.90:0/2207529143 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e5614d0780 con 0x55e56149dc00 2024-03-24T01:30:33.912 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.909+0000 7f9b0dc00640 1 --2- 172.21.15.90:0/2207529143 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e56149dc00 0x55e561507080 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x55e5622b3440 tx=0x55e561584e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c989dc1f4d80dd2d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.912 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.910+0000 7f9b0a000640 1 -- 172.21.15.90:0/2207529143 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e5614c2680 con 0x55e56149dc00 2024-03-24T01:30:33.912 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.910+0000 7f9b0a000640 1 -- 172.21.15.90:0/2207529143 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e5614c8700 con 0x55e56149dc00 2024-03-24T01:30:33.912 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.910+0000 7f9b0a000640 1 -- 172.21.15.90:0/2207529143 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e5614c2b60 con 0x55e56149dc00 2024-03-24T01:30:33.912 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.910+0000 7f9b0fc0f280 1 -- 172.21.15.90:0/2207529143 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e5614d14a0 con 0x55e56149dc00 2024-03-24T01:30:33.913 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.910+0000 7f9b0fc0f280 1 -- 172.21.15.90:0/2207529143 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x55e5614d0000 con 0x55e56149dc00 2024-03-24T01:30:33.913 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.910+0000 7f9b0a000640 1 -- 172.21.15.90:0/2207529143 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x55e562362600 con 0x55e56149dc00 2024-03-24T01:30:33.913 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.910+0000 7f9b0a000640 1 -- 172.21.15.90:0/2207529143 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 36) v1 ==== 3310+0+0 (secure 0 0 0) 0x55e5614f7180 con 0x55e56149dc00 2024-03-24T01:30:33.915 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.911+0000 7f9b0fc0f280 1 Processor -- start 2024-03-24T01:30:33.915 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.911+0000 7f9b0fc0f280 1 -- start start 2024-03-24T01:30:33.915 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.912+0000 7f9b0fc0f280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e562365400 0x55e561507600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.915 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.912+0000 7f9b0fc0f280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e562365800 0x55e561507b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.915 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.912+0000 7f9b0fc0f280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e562365c00 0x55e561508100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.915 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.912+0000 7f9b0fc0f280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e562262600 con 0x55e562365800 2024-03-24T01:30:33.915 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.912+0000 7f9b0fc0f280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e562262780 con 0x55e562365c00 2024-03-24T01:30:33.916 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.912+0000 7f9b0fc0f280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e562262900 con 0x55e562365400 2024-03-24T01:30:33.916 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.912+0000 7f9b0d200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e562365800 0x55e561507b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.916 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.912+0000 7f9b0d200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e562365800 0x55e561507b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36662/0 (socket says 172.21.15.90:36662) 2024-03-24T01:30:33.916 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.912+0000 7f9b0d200640 1 -- 172.21.15.90:0/1756230707 learned_addr learned my addr 172.21.15.90:0/1756230707 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.916 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.913+0000 7f9b0d200640 1 -- 172.21.15.90:0/1756230707 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e562365400 msgr2=0x55e561507600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.916 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.913+0000 7f9b0d200640 1 --2- 172.21.15.90:0/1756230707 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e562365400 0x55e561507600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.916 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.913+0000 7f9b0d200640 1 -- 172.21.15.90:0/1756230707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e562365c00 msgr2=0x55e561508100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.916 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.913+0000 7f9b0d200640 1 --2- 172.21.15.90:0/1756230707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e562365c00 0x55e561508100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.916 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.913+0000 7f9b0d200640 1 -- 172.21.15.90:0/1756230707 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e5614d03c0 con 0x55e562365800 2024-03-24T01:30:33.916 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.913+0000 7f9b0d200640 1 --2- 172.21.15.90:0/1756230707 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e562365800 0x55e561507b80 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x55e5622b2a50 tx=0x55e562382d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83c9287c07df7a91 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.916 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.913+0000 7f9b07800640 1 -- 172.21.15.90:0/1756230707 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e5614c29c0 con 0x55e562365800 2024-03-24T01:30:33.916 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.913+0000 7f9b07800640 1 -- 172.21.15.90:0/1756230707 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e5614c8fc0 con 0x55e562365800 2024-03-24T01:30:33.916 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.913+0000 7f9b07800640 1 -- 172.21.15.90:0/1756230707 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e5614c2680 con 0x55e562365800 2024-03-24T01:30:33.916 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.913+0000 7f9b0fc0f280 1 -- 172.21.15.90:0/1756230707 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e562365800 msgr2=0x55e561507b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.917 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.913+0000 7f9b0fc0f280 1 --2- 172.21.15.90:0/1756230707 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e562365800 0x55e561507b80 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x55e5622b2a50 tx=0x55e562382d20 comp rx=0 tx=0).stop 2024-03-24T01:30:33.917 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.913+0000 7f9b0fc0f280 1 -- 172.21.15.90:0/1756230707 shutdown_connections 2024-03-24T01:30:33.917 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.913+0000 7f9b0fc0f280 1 --2- 172.21.15.90:0/1756230707 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e562365800 0x55e561507b80 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.917 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.913+0000 7f9b0fc0f280 1 --2- 172.21.15.90:0/1756230707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e562365c00 0x55e561508100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.917 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.913+0000 7f9b0fc0f280 1 --2- 172.21.15.90:0/1756230707 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e562365400 0x55e561507600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.917 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.913+0000 7f9b0fc0f280 1 -- 172.21.15.90:0/1756230707 >> 172.21.15.90:0/1756230707 conn(0x55e562365000 msgr2=0x55e56221d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.917 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.914+0000 7f9b0fc0f280 1 -- 172.21.15.90:0/1756230707 shutdown_connections 2024-03-24T01:30:33.917 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.914+0000 7f9b0fc0f280 1 -- 172.21.15.90:0/1756230707 wait complete. 2024-03-24T01:30:33.917 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.914+0000 7f9b0fc0f280 1 Processor -- start 2024-03-24T01:30:33.919 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.914+0000 7f9b0fc0f280 1 -- start start 2024-03-24T01:30:33.919 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.915+0000 7f9b0fc0f280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e562365400 0x55e561509180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.919 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.915+0000 7f9b0fc0f280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e562365800 0x55e561508c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.919 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.915+0000 7f9b0fc0f280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e562365c00 0x55e561508680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.919 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.915+0000 7f9b0fc0f280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5614fe180 con 0x55e562365c00 2024-03-24T01:30:33.919 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.915+0000 7f9b0fc0f280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e562263e00 con 0x55e562365400 2024-03-24T01:30:33.919 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.915+0000 7f9b0fc0f280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e562262d80 con 0x55e562365800 2024-03-24T01:30:33.919 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.915+0000 7f9b0c800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e562365c00 0x55e561508680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.919 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.915+0000 7f9b0c800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e562365c00 0x55e561508680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36666/0 (socket says 172.21.15.90:36666) 2024-03-24T01:30:33.919 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.915+0000 7f9b0c800640 1 -- 172.21.15.90:0/1359510666 learned_addr learned my addr 172.21.15.90:0/1359510666 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.919 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.916+0000 7f9b0c800640 1 -- 172.21.15.90:0/1359510666 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e562365800 msgr2=0x55e561508c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.919 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.916+0000 7f9b0c800640 1 --2- 172.21.15.90:0/1359510666 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e562365800 0x55e561508c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.920 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.916+0000 7f9b0c800640 1 -- 172.21.15.90:0/1359510666 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e562365400 msgr2=0x55e561509180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.920 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.916+0000 7f9b0c800640 1 --2- 172.21.15.90:0/1359510666 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e562365400 0x55e561509180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.920 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.916+0000 7f9b0c800640 1 -- 172.21.15.90:0/1359510666 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e5614d0780 con 0x55e562365c00 2024-03-24T01:30:33.920 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.916+0000 7f9b0c800640 1 --2- 172.21.15.90:0/1359510666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e562365c00 0x55e561508680 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x55e5623816b0 tx=0x55e562382fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.920 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.916+0000 7f9b05a00640 1 -- 172.21.15.90:0/1359510666 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e5614c29c0 con 0x55e562365c00 2024-03-24T01:30:33.920 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.916+0000 7f9b05a00640 1 -- 172.21.15.90:0/1359510666 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e5614c8e00 con 0x55e562365c00 2024-03-24T01:30:33.920 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.916+0000 7f9b05a00640 1 -- 172.21.15.90:0/1359510666 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e5614c2680 con 0x55e562365c00 2024-03-24T01:30:33.920 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.916+0000 7f9b0fc0f280 1 -- 172.21.15.90:0/1359510666 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55e5614d14a0 con 0x55e562365c00 2024-03-24T01:30:33.920 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.916+0000 7f9b0fc0f280 1 -- 172.21.15.90:0/1359510666 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e5614d0960 con 0x55e562365c00 2024-03-24T01:30:33.921 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.918+0000 7f9b05a00640 1 -- 172.21.15.90:0/1359510666 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x55e56228cc00 con 0x55e562365c00 2024-03-24T01:30:33.921 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.919+0000 7f9b05a00640 1 --2- 172.21.15.90:0/1359510666 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55e562291000 0x55e561509700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.921 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.919+0000 7f9b05a00640 1 -- 172.21.15.90:0/1359510666 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x55e5623de600 con 0x55e562365c00 2024-03-24T01:30:33.922 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.919+0000 7f9b0d200640 1 --2- 172.21.15.90:0/1359510666 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55e562291000 0x55e561509700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.922 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.920+0000 7f9b0fc0f280 1 -- start start 2024-03-24T01:30:33.922 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.920+0000 7f9b0fc0f280 1 Processor -- start 2024-03-24T01:30:33.922 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.920+0000 7f9b0d200640 1 --2- 172.21.15.90:0/1359510666 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55e562291000 0x55e561509700 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x55e5622b2a50 tx=0x55e562382c00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.924 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.921+0000 7f9b0fc0f280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e562291800 0x55e561508100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.924 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.921+0000 7f9b0fc0f280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e562291c00 0x55e561507b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.924 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.921+0000 7f9b0fc0f280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e561862000 0x55e561507600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.924 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.921+0000 7f9b0fc0f280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e562263080 con 0x55e561862000 2024-03-24T01:30:33.924 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.921+0000 7f9b0fc0f280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e562263b00 con 0x55e562291c00 2024-03-24T01:30:33.924 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.921+0000 7f9b0fc0f280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e562263c80 con 0x55e562291800 2024-03-24T01:30:33.924 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.921+0000 7f9b0c800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e561862000 0x55e561507600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.924 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.921+0000 7f9b0c800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e561862000 0x55e561507600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36676/0 (socket says 172.21.15.90:36676) 2024-03-24T01:30:33.925 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.921+0000 7f9b0c800640 1 -- 172.21.15.90:0/1167583077 learned_addr learned my addr 172.21.15.90:0/1167583077 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.925 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.921+0000 7f9b0dc00640 1 --2- 172.21.15.90:0/1167583077 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e562291800 0x55e561508100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.925 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.922+0000 7f9b0c800640 1 -- 172.21.15.90:0/1167583077 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e562291800 msgr2=0x55e561508100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.925 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.922+0000 7f9b0c800640 1 --2- 172.21.15.90:0/1167583077 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e562291800 0x55e561508100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.925 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.922+0000 7f9b0c800640 1 -- 172.21.15.90:0/1167583077 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e562291c00 msgr2=0x55e561507b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.925 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.922+0000 7f9b0c800640 1 --2- 172.21.15.90:0/1167583077 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e562291c00 0x55e561507b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.925 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.922+0000 7f9b0c800640 1 -- 172.21.15.90:0/1167583077 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e5614d1680 con 0x55e561862000 2024-03-24T01:30:33.925 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.922+0000 7f9b0c800640 1 --2- 172.21.15.90:0/1167583077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e561862000 0x55e561507600 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x55e5624076e0 tx=0x55e562383920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d38b387ce429b49d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.925 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.922+0000 7f9b01e00640 1 -- 172.21.15.90:0/1167583077 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e561798ea0 con 0x55e561862000 2024-03-24T01:30:33.925 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.922+0000 7f9b01e00640 1 -- 172.21.15.90:0/1167583077 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e5614c8c40 con 0x55e561862000 2024-03-24T01:30:33.925 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.922+0000 7f9b01e00640 1 -- 172.21.15.90:0/1167583077 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e561799040 con 0x55e561862000 2024-03-24T01:30:33.925 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.922+0000 7f9b0fc0f280 1 -- 172.21.15.90:0/1167583077 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e5614d0780 con 0x55e561862000 2024-03-24T01:30:33.925 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.922+0000 7f9b0fc0f280 1 -- 172.21.15.90:0/1167583077 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x55e5614d14a0 con 0x55e561862000 2024-03-24T01:30:33.925 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.922+0000 7f9b01e00640 1 -- 172.21.15.90:0/1167583077 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x55e56186c200 con 0x55e561862000 2024-03-24T01:30:33.926 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.923+0000 7f9b01e00640 1 -- 172.21.15.90:0/1167583077 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 36) v1 ==== 3310+0+0 (secure 0 0 0) 0x55e5614f7500 con 0x55e561862000 2024-03-24T01:30:33.926 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.923+0000 7f9b0fc0f280 1 --2- 172.21.15.90:0/1167583077 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55e561862400 0x55e561884000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.926 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.923+0000 7f9b0fc0f280 1 -- 172.21.15.90:0/1167583077 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:1 7.f 7:f1f69726:::400.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e561862800 con 0x55e561862400 2024-03-24T01:30:33.926 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.923+0000 7f9b0dc00640 1 --2- 172.21.15.90:0/1167583077 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55e561862400 0x55e561884000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.926 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.923+0000 7f9b0dc00640 1 --2- 172.21.15.90:0/1167583077 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55e561862400 0x55e561884000 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=6bc862dcb285200a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.926 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.924+0000 7f9b0dc00640 1 -- 172.21.15.90:0/1167583077 <== osd.0 v2:172.21.15.90:6800/307045587 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x55e562220000 con 0x55e561862400 2024-03-24T01:30:33.927 INFO:tasks.workunit.client.0.smithi090.stdout:undump /tmp/journal.bin 2024-03-24T01:30:33.928 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.924+0000 7f9b0fc0f280 1 --2- 172.21.15.90:0/1167583077 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55e561862c00 0x55e561884580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.928 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.924+0000 7f9b0fc0f280 1 -- 172.21.15.90:0/1167583077 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:2 7.14 7:292cf221:::200.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e561863000 con 0x55e561862c00 2024-03-24T01:30:33.928 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.924+0000 7f9b0d200640 1 --2- 172.21.15.90:0/1167583077 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55e561862c00 0x55e561884580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.928 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.924+0000 7f9b0d200640 1 --2- 172.21.15.90:0/1167583077 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55e561862c00 0x55e561884580 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=37178010f357879 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.928 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.925+0000 7f9b0d200640 1 -- 172.21.15.90:0/1167583077 <== osd.3 v2:172.21.15.90:6808/2999931983 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv33 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x55e562220500 con 0x55e561862c00 2024-03-24T01:30:33.928 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.925+0000 7f9b00000640 1 -- 172.21.15.90:0/1167583077 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:3 7.14 7:292cf221:::200.00000000:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e56186f400 con 0x55e561862c00 2024-03-24T01:30:33.928 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.925+0000 7f9b00000640 1 -- 172.21.15.90:0/1167583077 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:4 7.14 7:2e2fa760:::200.00000001:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e56186f800 con 0x55e561862c00 2024-03-24T01:30:33.928 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.925+0000 7f9b0d200640 1 -- 172.21.15.90:0/1167583077 <== osd.3 v2:172.21.15.90:6808/2999931983 2 ==== osd_op_reply(3 200.00000000 [stat out=16b] v0'0 uv33 ondisk = 0) v8 ==== 156+0+16 (crc 0 0 0) 0x55e562220500 con 0x55e561862c00 2024-03-24T01:30:33.929 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.925+0000 7f9b0d200640 1 -- 172.21.15.90:0/1167583077 <== osd.3 v2:172.21.15.90:6808/2999931983 3 ==== osd_op_reply(4 200.00000001 [stat out=16b] v0'0 uv30 ondisk = 0) v8 ==== 156+0+16 (crc 0 0 0) 0x55e562220500 con 0x55e561862c00 2024-03-24T01:30:33.929 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.925+0000 7f9b00000640 1 -- 172.21.15.90:0/1167583077 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:5 7.1b 7:ddb4e4d7:::200.00000002:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e56186fc00 con 0x55e561862400 2024-03-24T01:30:33.929 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.926+0000 7f9b0dc00640 1 -- 172.21.15.90:0/1167583077 <== osd.0 v2:172.21.15.90:6800/307045587 2 ==== osd_op_reply(5 200.00000002 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x55e562220000 con 0x55e561862400 2024-03-24T01:30:33.929 INFO:tasks.workunit.client.0.smithi090.stdout:start 4194304 len 35827 write_pos 4200185 format 1 trimmed_pos 4194304 stripe_unit 4194304 stripe_count 1 object_size 4194304 2024-03-24T01:30:33.929 INFO:tasks.workunit.client.0.smithi090.stdout:writing header 200.00000000 2024-03-24T01:30:33.929 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.926+0000 7f9b0fc0f280 1 -- 172.21.15.90:0/1167583077 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:6 7.14 7:292cf221:::200.00000000:head [writefull 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e561863400 con 0x55e561862c00 2024-03-24T01:30:33.930 INFO:tasks.workunit.client.0.smithi090.stdout:Purging 2 objects from 1 2024-03-24T01:30:33.930 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.927+0000 7f9b0d200640 1 -- 172.21.15.90:0/1167583077 <== osd.3 v2:172.21.15.90:6808/2999931983 4 ==== osd_op_reply(6 200.00000000 [writefull 0~90] v56'34 uv34 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55e562220500 con 0x55e561862c00 2024-03-24T01:30:33.930 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.927+0000 7f9b0fc0f280 1 -- 172.21.15.90:0/1167583077 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:7 7.14 7:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e56186f400 con 0x55e561862c00 2024-03-24T01:30:33.930 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.927+0000 7f9b0fc0f280 1 -- 172.21.15.90:0/1167583077 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:8 7.1b 7:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e56186f800 con 0x55e561862400 2024-03-24T01:30:33.931 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.929+0000 7f9b0d200640 1 -- 172.21.15.90:0/1167583077 <== osd.3 v2:172.21.15.90:6808/2999931983 5 ==== osd_op_reply(7 200.00000001 [delete] v56'35 uv35 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55e562220500 con 0x55e561862c00 2024-03-24T01:30:33.931 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.930+0000 7f9b0dc00640 1 -- 172.21.15.90:0/1167583077 <== osd.0 v2:172.21.15.90:6800/307045587 3 ==== osd_op_reply(8 200.00000002 [delete] v56'6 uv5 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x55e562220000 con 0x55e561862400 2024-03-24T01:30:33.931 INFO:tasks.workunit.client.0.smithi090.stdout: writing 4194304~35827 2024-03-24T01:30:33.931 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.930+0000 7f9b0fc0f280 1 -- 172.21.15.90:0/1167583077 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:9 7.14 7:2e2fa760:::200.00000001:head [write 0~35827 in=35827b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e561863000 con 0x55e561862c00 2024-03-24T01:30:33.934 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.933+0000 7f9b0d200640 1 -- 172.21.15.90:0/1167583077 <== osd.3 v2:172.21.15.90:6808/2999931983 6 ==== osd_op_reply(9 200.00000001 [write 0~35827] v56'36 uv36 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55e562220500 con 0x55e561862c00 2024-03-24T01:30:33.934 INFO:tasks.workunit.client.0.smithi090.stdout:done. 2024-03-24T01:30:33.936 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.933+0000 7f9b0fc0f280 1 -- 172.21.15.90:0/1167583077 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55e561862400 msgr2=0x55e561884000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.936 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.933+0000 7f9b0fc0f280 1 --2- 172.21.15.90:0/1167583077 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55e561862400 0x55e561884000 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.936 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.933+0000 7f9b0fc0f280 1 -- 172.21.15.90:0/1167583077 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55e561862c00 msgr2=0x55e561884580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.937 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.933+0000 7f9b0fc0f280 1 --2- 172.21.15.90:0/1167583077 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55e561862c00 0x55e561884580 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.937 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.933+0000 7f9b0fc0f280 1 -- 172.21.15.90:0/1167583077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e561862000 msgr2=0x55e561507600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.937 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.933+0000 7f9b0fc0f280 1 --2- 172.21.15.90:0/1167583077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e561862000 0x55e561507600 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x55e5624076e0 tx=0x55e562383920 comp rx=0 tx=0).stop 2024-03-24T01:30:33.937 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.933+0000 7f9b0c800640 1 -- 172.21.15.90:0/1167583077 reap_dead start 2024-03-24T01:30:33.937 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.933+0000 7f9b0fc0f280 1 -- 172.21.15.90:0/1167583077 shutdown_connections 2024-03-24T01:30:33.937 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.933+0000 7f9b0fc0f280 1 -- 172.21.15.90:0/1167583077 >> 172.21.15.90:0/1167583077 conn(0x55e562291400 msgr2=0x55e56221d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.937 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.933+0000 7f9b0fc0f280 1 -- 172.21.15.90:0/1167583077 shutdown_connections 2024-03-24T01:30:33.937 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.933+0000 7f9b0fc0f280 1 -- 172.21.15.90:0/1167583077 wait complete. 2024-03-24T01:30:33.937 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.933+0000 7f9b0fc0f280 1 -- 172.21.15.90:0/1359510666 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55e562291000 msgr2=0x55e561509700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.937 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.933+0000 7f9b0fc0f280 1 --2- 172.21.15.90:0/1359510666 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55e562291000 0x55e561509700 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x55e5622b2a50 tx=0x55e562382c00 comp rx=0 tx=0).stop 2024-03-24T01:30:33.937 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.933+0000 7f9b0fc0f280 1 -- 172.21.15.90:0/1359510666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e562365c00 msgr2=0x55e561508680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.937 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.933+0000 7f9b0fc0f280 1 --2- 172.21.15.90:0/1359510666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e562365c00 0x55e561508680 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x55e5623816b0 tx=0x55e562382fc0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.937 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.933+0000 7f9b0fc0f280 1 -- 172.21.15.90:0/1359510666 shutdown_connections 2024-03-24T01:30:33.937 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.933+0000 7f9b0fc0f280 1 --2- 172.21.15.90:0/1359510666 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55e562291000 0x55e561509700 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.937 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.933+0000 7f9b0fc0f280 1 --2- 172.21.15.90:0/1359510666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e562365c00 0x55e561508680 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.937 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.933+0000 7f9b0fc0f280 1 --2- 172.21.15.90:0/1359510666 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e562365800 0x55e561508c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.938 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.933+0000 7f9b0fc0f280 1 --2- 172.21.15.90:0/1359510666 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e562365400 0x55e561509180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.938 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.933+0000 7f9b0fc0f280 1 -- 172.21.15.90:0/1359510666 >> 172.21.15.90:0/1359510666 conn(0x55e562365000 msgr2=0x55e56221d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.938 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.934+0000 7f9b0fc0f280 1 -- 172.21.15.90:0/1359510666 shutdown_connections 2024-03-24T01:30:33.938 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.934+0000 7f9b0fc0f280 1 -- 172.21.15.90:0/1359510666 wait complete. 2024-03-24T01:30:33.938 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.934+0000 7f9b0fc0f280 1 -- 172.21.15.90:0/2207529143 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e56149dc00 msgr2=0x55e561507080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.938 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.934+0000 7f9b0fc0f280 1 --2- 172.21.15.90:0/2207529143 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e56149dc00 0x55e561507080 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x55e5622b3440 tx=0x55e561584e40 comp rx=0 tx=0).stop 2024-03-24T01:30:33.938 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.934+0000 7f9b0fc0f280 1 -- 172.21.15.90:0/2207529143 shutdown_connections 2024-03-24T01:30:33.938 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.934+0000 7f9b0fc0f280 1 --2- 172.21.15.90:0/2207529143 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e56149dc00 0x55e561507080 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.938 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.934+0000 7f9b0fc0f280 1 --2- 172.21.15.90:0/2207529143 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e56149d800 0x55e561506b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.938 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.934+0000 7f9b0fc0f280 1 --2- 172.21.15.90:0/2207529143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e56149cc00 0x55e561506580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.938 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.934+0000 7f9b0fc0f280 1 -- 172.21.15.90:0/2207529143 >> 172.21.15.90:0/2207529143 conn(0x55e56149d000 msgr2=0x55e56221c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.938 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.934+0000 7f9b0fc0f280 1 -- 172.21.15.90:0/2207529143 shutdown_connections 2024-03-24T01:30:33.938 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.934+0000 7f9b0fc0f280 1 -- 172.21.15.90:0/2207529143 wait complete. 2024-03-24T01:30:33.939 INFO:tasks.workunit.client.0.smithi090.stdout:Testing 'event' commands... 2024-03-24T01:30:33.940 INFO:tasks.workunit.client.0.smithi090.stderr:+ echo 'Testing '\''event'\'' commands...' 2024-03-24T01:30:33.940 INFO:tasks.workunit.client.0.smithi090.stderr:+ cephfs-journal-tool --rank=cephfs:0 event get summary 2024-03-24T01:30:33.984 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.983+0000 7f7f4682c280 1 -- start start 2024-03-24T01:30:33.984 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.983+0000 7f7f4682c280 1 Processor -- start 2024-03-24T01:30:33.987 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.984+0000 7f7f4682c280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e47f0ac00 0x555e47f74580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.987 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.984+0000 7f7f4682c280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555e47f0b800 0x555e47f74b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.987 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.984+0000 7f7f4682c280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e47f0bc00 0x555e47f75080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.987 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.984+0000 7f7f4682c280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e48dba180 con 0x555e47f0ac00 2024-03-24T01:30:33.987 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.984+0000 7f7f4682c280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e48dba300 con 0x555e47f0bc00 2024-03-24T01:30:33.987 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.984+0000 7f7f4682c280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x555e48dba480 con 0x555e47f0b800 2024-03-24T01:30:33.987 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.984+0000 7f7f43a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e47f0ac00 0x555e47f74580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.987 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.984+0000 7f7f43a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e47f0ac00 0x555e47f74580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36686/0 (socket says 172.21.15.90:36686) 2024-03-24T01:30:33.987 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.984+0000 7f7f43a00640 1 -- 172.21.15.90:0/1373315952 learned_addr learned my addr 172.21.15.90:0/1373315952 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.987 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.984+0000 7f7f43a00640 1 -- 172.21.15.90:0/1373315952 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555e47f0b800 msgr2=0x555e47f74b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.987 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.984+0000 7f7f43a00640 1 --2- 172.21.15.90:0/1373315952 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555e47f0b800 0x555e47f74b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.988 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.984+0000 7f7f43a00640 1 -- 172.21.15.90:0/1373315952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e47f0bc00 msgr2=0x555e47f75080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.988 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.984+0000 7f7f44400640 1 --2- 172.21.15.90:0/1373315952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e47f0bc00 0x555e47f75080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.988 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.984+0000 7f7f43a00640 1 --2- 172.21.15.90:0/1373315952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e47f0bc00 0x555e47f75080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.988 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.984+0000 7f7f43a00640 1 -- 172.21.15.90:0/1373315952 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e47f3e780 con 0x555e47f0ac00 2024-03-24T01:30:33.988 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.985+0000 7f7f43a00640 1 --2- 172.21.15.90:0/1373315952 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e47f0ac00 0x555e47f74580 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x555e48d21440 tx=0x555e47ff2e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4db91c77b114b75d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.988 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.985+0000 7f7f40800640 1 -- 172.21.15.90:0/1373315952 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x555e47f30680 con 0x555e47f0ac00 2024-03-24T01:30:33.988 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.985+0000 7f7f40800640 1 -- 172.21.15.90:0/1373315952 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x555e47f36700 con 0x555e47f0ac00 2024-03-24T01:30:33.988 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.985+0000 7f7f40800640 1 -- 172.21.15.90:0/1373315952 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x555e47f30b60 con 0x555e47f0ac00 2024-03-24T01:30:33.988 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.985+0000 7f7f4682c280 1 -- 172.21.15.90:0/1373315952 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555e47f3f4a0 con 0x555e47f0ac00 2024-03-24T01:30:33.988 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.985+0000 7f7f4682c280 1 -- 172.21.15.90:0/1373315952 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x555e47f3e000 con 0x555e47f0ac00 2024-03-24T01:30:33.988 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.985+0000 7f7f40800640 1 -- 172.21.15.90:0/1373315952 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x555e48dd0600 con 0x555e47f0ac00 2024-03-24T01:30:33.988 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.985+0000 7f7f40800640 1 -- 172.21.15.90:0/1373315952 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 36) v1 ==== 3310+0+0 (secure 0 0 0) 0x555e47f65180 con 0x555e47f0ac00 2024-03-24T01:30:33.993 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.986+0000 7f7f4682c280 1 Processor -- start 2024-03-24T01:30:33.993 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.986+0000 7f7f4682c280 1 -- start start 2024-03-24T01:30:33.993 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.987+0000 7f7f4682c280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e48dd7400 0x555e47f75600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.993 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.987+0000 7f7f4682c280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e48dd7800 0x555e47f75b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.993 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.987+0000 7f7f4682c280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555e48dd7c00 0x555e47f76100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.993 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.987+0000 7f7f4682c280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e48dba600 con 0x555e48dd7400 2024-03-24T01:30:33.993 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.987+0000 7f7f4682c280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e48dba780 con 0x555e48dd7800 2024-03-24T01:30:33.993 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.987+0000 7f7f4682c280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x555e48dba900 con 0x555e48dd7c00 2024-03-24T01:30:33.993 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.987+0000 7f7f44400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e48dd7400 0x555e47f75600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.993 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.987+0000 7f7f44400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e48dd7400 0x555e47f75600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36696/0 (socket says 172.21.15.90:36696) 2024-03-24T01:30:33.993 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.987+0000 7f7f44400640 1 -- 172.21.15.90:0/2105959003 learned_addr learned my addr 172.21.15.90:0/2105959003 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.994 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.988+0000 7f7f44400640 1 -- 172.21.15.90:0/2105959003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555e48dd7c00 msgr2=0x555e47f76100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.994 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.988+0000 7f7f44400640 1 --2- 172.21.15.90:0/2105959003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555e48dd7c00 0x555e47f76100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.994 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.988+0000 7f7f44400640 1 -- 172.21.15.90:0/2105959003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e48dd7800 msgr2=0x555e47f75b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:33.994 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.988+0000 7f7f44400640 1 --2- 172.21.15.90:0/2105959003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e48dd7800 0x555e47f75b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.994 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.988+0000 7f7f44400640 1 -- 172.21.15.90:0/2105959003 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e47f3e3c0 con 0x555e48dd7400 2024-03-24T01:30:33.994 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.988+0000 7f7f44400640 1 --2- 172.21.15.90:0/2105959003 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e48dd7400 0x555e47f75600 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x555e48dc8a80 tx=0x555e48deccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85bd45d7037589e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.994 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.988+0000 7f7f3e000640 1 -- 172.21.15.90:0/2105959003 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x555e47f30680 con 0x555e48dd7400 2024-03-24T01:30:33.994 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.988+0000 7f7f3e000640 1 -- 172.21.15.90:0/2105959003 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x555e47f36fc0 con 0x555e48dd7400 2024-03-24T01:30:33.994 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.988+0000 7f7f3e000640 1 -- 172.21.15.90:0/2105959003 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x555e47f30b60 con 0x555e48dd7400 2024-03-24T01:30:33.994 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.989+0000 7f7f4682c280 1 -- 172.21.15.90:0/2105959003 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e48dd7400 msgr2=0x555e47f75600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.994 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.989+0000 7f7f4682c280 1 --2- 172.21.15.90:0/2105959003 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e48dd7400 0x555e47f75600 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x555e48dc8a80 tx=0x555e48deccc0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.994 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.989+0000 7f7f4682c280 1 -- 172.21.15.90:0/2105959003 shutdown_connections 2024-03-24T01:30:33.995 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.989+0000 7f7f4682c280 1 --2- 172.21.15.90:0/2105959003 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555e48dd7c00 0x555e47f76100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.995 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.989+0000 7f7f4682c280 1 --2- 172.21.15.90:0/2105959003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e48dd7800 0x555e47f75b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.995 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.989+0000 7f7f4682c280 1 --2- 172.21.15.90:0/2105959003 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e48dd7400 0x555e47f75600 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.995 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.989+0000 7f7f4682c280 1 -- 172.21.15.90:0/2105959003 >> 172.21.15.90:0/2105959003 conn(0x555e48dd7000 msgr2=0x555e48c8b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:33.995 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.989+0000 7f7f4682c280 1 -- 172.21.15.90:0/2105959003 shutdown_connections 2024-03-24T01:30:33.995 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.989+0000 7f7f4682c280 1 -- 172.21.15.90:0/2105959003 wait complete. 2024-03-24T01:30:33.995 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.989+0000 7f7f4682c280 1 Processor -- start 2024-03-24T01:30:33.995 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.990+0000 7f7f4682c280 1 -- start start 2024-03-24T01:30:33.995 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.990+0000 7f7f4682c280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e48dd7400 0x555e47f77180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.995 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.990+0000 7f7f4682c280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555e48dd7800 0x555e47f76c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.995 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.990+0000 7f7f4682c280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e48dd7c00 0x555e47f76680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.995 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.990+0000 7f7f4682c280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e48dba900 con 0x555e48dd7400 2024-03-24T01:30:33.995 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.990+0000 7f7f4682c280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e48dbaa80 con 0x555e48dd7c00 2024-03-24T01:30:33.996 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.990+0000 7f7f4682c280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x555e48dba780 con 0x555e48dd7800 2024-03-24T01:30:33.996 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.990+0000 7f7f44400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e48dd7400 0x555e47f77180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.996 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.990+0000 7f7f44400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e48dd7400 0x555e47f77180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36710/0 (socket says 172.21.15.90:36710) 2024-03-24T01:30:33.996 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.991+0000 7f7f44400640 1 -- 172.21.15.90:0/3130417009 learned_addr learned my addr 172.21.15.90:0/3130417009 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:33.996 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.991+0000 7f7f44400640 1 -- 172.21.15.90:0/3130417009 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555e48dd7800 msgr2=0x555e47f76c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.996 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.991+0000 7f7f44400640 1 --2- 172.21.15.90:0/3130417009 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555e48dd7800 0x555e47f76c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.996 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.991+0000 7f7f44400640 1 -- 172.21.15.90:0/3130417009 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e48dd7c00 msgr2=0x555e47f76680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:33.996 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.991+0000 7f7f44400640 1 --2- 172.21.15.90:0/3130417009 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e48dd7c00 0x555e47f76680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:33.996 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.991+0000 7f7f44400640 1 -- 172.21.15.90:0/3130417009 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e47f3e780 con 0x555e48dd7400 2024-03-24T01:30:33.996 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.991+0000 7f7f44400640 1 --2- 172.21.15.90:0/3130417009 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e48dd7400 0x555e47f77180 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x555e48dc9740 tx=0x555e48decc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:33.996 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.991+0000 7f7f3c200640 1 -- 172.21.15.90:0/3130417009 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x555e47f304e0 con 0x555e48dd7400 2024-03-24T01:30:33.996 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.991+0000 7f7f3c200640 1 -- 172.21.15.90:0/3130417009 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x555e47f36e00 con 0x555e48dd7400 2024-03-24T01:30:33.996 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.991+0000 7f7f3c200640 1 -- 172.21.15.90:0/3130417009 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x555e47f30680 con 0x555e48dd7400 2024-03-24T01:30:33.996 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.991+0000 7f7f4682c280 1 -- 172.21.15.90:0/3130417009 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x555e47f3f4a0 con 0x555e48dd7400 2024-03-24T01:30:33.997 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.992+0000 7f7f4682c280 1 -- 172.21.15.90:0/3130417009 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555e47f3e960 con 0x555e48dd7400 2024-03-24T01:30:33.999 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.993+0000 7f7f3c200640 1 -- 172.21.15.90:0/3130417009 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x555e48e50c00 con 0x555e48dd7400 2024-03-24T01:30:33.999 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.994+0000 7f7f3c200640 1 --2- 172.21.15.90:0/3130417009 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x555e48e49000 0x555e47f77700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.999 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.994+0000 7f7f3c200640 1 -- 172.21.15.90:0/3130417009 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x555e48e4aa00 con 0x555e48dd7400 2024-03-24T01:30:33.999 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.994+0000 7f7f4682c280 1 --2- 172.21.15.90:0/3130417009 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x555e48e49400 0x555e47f76100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:33.999 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.994+0000 7f7f4682c280 1 -- 172.21.15.90:0/3130417009 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:1 7.f 7:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x555e48e49800 con 0x555e48e49400 2024-03-24T01:30:33.999 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.994+0000 7f7f43000640 1 --2- 172.21.15.90:0/3130417009 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x555e48e49400 0x555e47f76100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:33.999 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.994+0000 7f7f43a00640 1 --2- 172.21.15.90:0/3130417009 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x555e48e49000 0x555e47f77700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.000 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.995+0000 7f7f43000640 1 --2- 172.21.15.90:0/3130417009 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x555e48e49400 0x555e47f76100 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.000 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.995+0000 7f7f43000640 1 -- 172.21.15.90:0/3130417009 <== osd.0 v2:172.21.15.90:6800/307045587 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x555e48c8e000 con 0x555e48e49400 2024-03-24T01:30:34.000 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.995+0000 7f7f4682c280 1 --2- 172.21.15.90:0/3130417009 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x555e48e49c00 0x555e47f75b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.000 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.995+0000 7f7f4682c280 1 -- 172.21.15.90:0/3130417009 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:2 7.14 7:292cf221:::200.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x555e48e49800 con 0x555e48e49c00 2024-03-24T01:30:34.000 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.995+0000 7f7f43a00640 1 --2- 172.21.15.90:0/3130417009 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x555e48e49000 0x555e47f77700 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x555e48dc8810 tx=0x555e47ff2d20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.000 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.995+0000 7f7f44400640 1 --2- 172.21.15.90:0/3130417009 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x555e48e49c00 0x555e47f75b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.000 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.996+0000 7f7f44400640 1 --2- 172.21.15.90:0/3130417009 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x555e48e49c00 0x555e47f75b80 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.000 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.996+0000 7f7f44400640 1 -- 172.21.15.90:0/3130417009 <== osd.3 v2:172.21.15.90:6808/2999931983 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv34 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x555e48c8e500 con 0x555e48e49c00 2024-03-24T01:30:34.000 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.996+0000 7f7f4682c280 1 -- 172.21.15.90:0/3130417009 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:3 7.14 7:2e2fa760:::200.00000001:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x555e48292000 con 0x555e48e49c00 2024-03-24T01:30:34.000 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.997+0000 7f7f44400640 1 -- 172.21.15.90:0/3130417009 <== osd.3 v2:172.21.15.90:6808/2999931983 2 ==== osd_op_reply(3 200.00000001 [read 0~35827 out=35827b] v0'0 uv36 ondisk = 0) v8 ==== 156+0+35827 (crc 0 0 0) 0x555e48c8e500 con 0x555e48e49c00 2024-03-24T01:30:34.000 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.997+0000 7f7f4682c280 1 -- 172.21.15.90:0/3130417009 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:4 7.1b 7:ddb4e4d7:::200.00000002:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x555e48292400 con 0x555e48e49400 2024-03-24T01:30:34.000 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.997+0000 7f7f43000640 1 -- 172.21.15.90:0/3130417009 <== osd.0 v2:172.21.15.90:6800/307045587 2 ==== osd_op_reply(4 200.00000002 [read 0~2147483647] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x555e48c8e000 con 0x555e48e49400 2024-03-24T01:30:34.001 INFO:tasks.workunit.client.0.smithi090.stdout:Events by type: 2024-03-24T01:30:34.001 INFO:tasks.workunit.client.0.smithi090.stdout: LID: 1 2024-03-24T01:30:34.001 INFO:tasks.workunit.client.0.smithi090.stdout: PURGED: 1 2024-03-24T01:30:34.001 INFO:tasks.workunit.client.0.smithi090.stdout: SESSION: 2 2024-03-24T01:30:34.001 INFO:tasks.workunit.client.0.smithi090.stdout: SUBTREEMAP: 1 2024-03-24T01:30:34.001 INFO:tasks.workunit.client.0.smithi090.stdout: UPDATE: 17 2024-03-24T01:30:34.001 INFO:tasks.workunit.client.0.smithi090.stdout:Errors: 0 2024-03-24T01:30:34.002 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.997+0000 7f7f4682c280 1 -- 172.21.15.90:0/3130417009 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x555e48e49400 msgr2=0x555e47f76100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.002 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.997+0000 7f7f4682c280 1 --2- 172.21.15.90:0/3130417009 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x555e48e49400 0x555e47f76100 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.002 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.997+0000 7f7f4682c280 1 -- 172.21.15.90:0/3130417009 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x555e48e49c00 msgr2=0x555e47f75b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.002 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.998+0000 7f7f4682c280 1 --2- 172.21.15.90:0/3130417009 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x555e48e49c00 0x555e47f75b80 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.002 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.998+0000 7f7f4682c280 1 -- 172.21.15.90:0/3130417009 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x555e48e49000 msgr2=0x555e47f77700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.003 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.998+0000 7f7f4682c280 1 --2- 172.21.15.90:0/3130417009 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x555e48e49000 0x555e47f77700 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x555e48dc8810 tx=0x555e47ff2d20 comp rx=0 tx=0).stop 2024-03-24T01:30:34.003 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.998+0000 7f7f4682c280 1 -- 172.21.15.90:0/3130417009 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e48dd7400 msgr2=0x555e47f77180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.003 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.998+0000 7f7f4682c280 1 --2- 172.21.15.90:0/3130417009 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e48dd7400 0x555e47f77180 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x555e48dc9740 tx=0x555e48decc00 comp rx=0 tx=0).stop 2024-03-24T01:30:34.003 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.998+0000 7f7f43000640 1 -- 172.21.15.90:0/3130417009 reap_dead start 2024-03-24T01:30:34.003 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.998+0000 7f7f4682c280 1 -- 172.21.15.90:0/3130417009 shutdown_connections 2024-03-24T01:30:34.003 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.998+0000 7f7f4682c280 1 --2- 172.21.15.90:0/3130417009 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e48dd7400 0x555e47f77180 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.003 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.998+0000 7f7f4682c280 1 -- 172.21.15.90:0/3130417009 >> 172.21.15.90:0/3130417009 conn(0x555e48dd7000 msgr2=0x555e48c8b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:34.003 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.998+0000 7f7f4682c280 1 -- 172.21.15.90:0/3130417009 shutdown_connections 2024-03-24T01:30:34.003 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.998+0000 7f7f4682c280 1 -- 172.21.15.90:0/3130417009 wait complete. 2024-03-24T01:30:34.003 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.998+0000 7f7f4682c280 1 -- 172.21.15.90:0/1373315952 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e47f0ac00 msgr2=0x555e47f74580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.003 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.998+0000 7f7f4682c280 1 --2- 172.21.15.90:0/1373315952 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e47f0ac00 0x555e47f74580 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x555e48d21440 tx=0x555e47ff2e40 comp rx=0 tx=0).stop 2024-03-24T01:30:34.003 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.998+0000 7f7f4682c280 1 -- 172.21.15.90:0/1373315952 shutdown_connections 2024-03-24T01:30:34.003 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.998+0000 7f7f4682c280 1 --2- 172.21.15.90:0/1373315952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555e47f0bc00 0x555e47f75080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.003 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.998+0000 7f7f4682c280 1 --2- 172.21.15.90:0/1373315952 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x555e47f0b800 0x555e47f74b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.003 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.998+0000 7f7f4682c280 1 --2- 172.21.15.90:0/1373315952 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555e47f0ac00 0x555e47f74580 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.003 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.998+0000 7f7f4682c280 1 -- 172.21.15.90:0/1373315952 >> 172.21.15.90:0/1373315952 conn(0x555e47f0b000 msgr2=0x555e48c8a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:34.004 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.998+0000 7f7f4682c280 1 -- 172.21.15.90:0/1373315952 shutdown_connections 2024-03-24T01:30:34.004 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:33.998+0000 7f7f4682c280 1 -- 172.21.15.90:0/1373315952 wait complete. 2024-03-24T01:30:34.004 INFO:tasks.workunit.client.0.smithi090.stderr:+ cephfs-journal-tool --rank=cephfs:0 event get --type=UPDATE --path=/ --inode=0 --frag=0x100 summary 2024-03-24T01:30:34.051 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.050+0000 7f87ed03e280 1 -- start start 2024-03-24T01:30:34.051 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.050+0000 7f87ed03e280 1 Processor -- start 2024-03-24T01:30:34.053 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.051+0000 7f87ed03e280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c144728c00 0x55c144793b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.053 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.051+0000 7f87ed03e280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c144729800 0x55c144793600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.053 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.051+0000 7f87ed03e280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c144729c00 0x55c144793080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.053 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.051+0000 7f87ed03e280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c145510180 con 0x55c144729c00 2024-03-24T01:30:34.053 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.051+0000 7f87ed03e280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c14478a480 con 0x55c144729800 2024-03-24T01:30:34.054 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.051+0000 7f87ed03e280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c14478a900 con 0x55c144728c00 2024-03-24T01:30:34.054 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.051+0000 7f87eb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c144729c00 0x55c144793080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.054 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.051+0000 7f87eb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c144729c00 0x55c144793080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36736/0 (socket says 172.21.15.90:36736) 2024-03-24T01:30:34.054 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.051+0000 7f87eb400640 1 -- 172.21.15.90:0/3405509643 learned_addr learned my addr 172.21.15.90:0/3405509643 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:34.054 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.051+0000 7f87eaa00640 1 --2- 172.21.15.90:0/3405509643 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c144728c00 0x55c144793b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.054 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.051+0000 7f87ea000640 1 --2- 172.21.15.90:0/3405509643 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c144729800 0x55c144793600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.054 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.051+0000 7f87eb400640 1 -- 172.21.15.90:0/3405509643 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c144728c00 msgr2=0x55c144793b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:34.054 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.051+0000 7f87eb400640 1 --2- 172.21.15.90:0/3405509643 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c144728c00 0x55c144793b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.054 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.051+0000 7f87eb400640 1 -- 172.21.15.90:0/3405509643 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c144729800 msgr2=0x55c144793600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:34.054 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.051+0000 7f87eb400640 1 --2- 172.21.15.90:0/3405509643 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c144729800 0x55c144793600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.054 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.051+0000 7f87eb400640 1 -- 172.21.15.90:0/3405509643 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c14475c780 con 0x55c144729c00 2024-03-24T01:30:34.054 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.052+0000 7f87eb400640 1 --2- 172.21.15.90:0/3405509643 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c144729c00 0x55c144793080 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x55c145549350 tx=0x55c1454ff140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c9bd03ec09b30ae server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.054 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.052+0000 7f87e7800640 1 -- 172.21.15.90:0/3405509643 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55c14474e680 con 0x55c144729c00 2024-03-24T01:30:34.055 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.052+0000 7f87e7800640 1 -- 172.21.15.90:0/3405509643 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c144754700 con 0x55c144729c00 2024-03-24T01:30:34.055 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.052+0000 7f87ed03e280 1 -- 172.21.15.90:0/3405509643 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c14475d4a0 con 0x55c144729c00 2024-03-24T01:30:34.055 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.052+0000 7f87ed03e280 1 -- 172.21.15.90:0/3405509643 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x55c14475c000 con 0x55c144729c00 2024-03-24T01:30:34.055 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.052+0000 7f87e7800640 1 -- 172.21.15.90:0/3405509643 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55c14474eb60 con 0x55c144729c00 2024-03-24T01:30:34.055 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.052+0000 7f87e7800640 1 -- 172.21.15.90:0/3405509643 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x55c1455ea600 con 0x55c144729c00 2024-03-24T01:30:34.055 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.052+0000 7f87e7800640 1 -- 172.21.15.90:0/3405509643 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 36) v1 ==== 3310+0+0 (secure 0 0 0) 0x55c144783180 con 0x55c144729c00 2024-03-24T01:30:34.057 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.053+0000 7f87ed03e280 1 Processor -- start 2024-03-24T01:30:34.057 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.053+0000 7f87ed03e280 1 -- start start 2024-03-24T01:30:34.057 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.054+0000 7f87ed03e280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1455ed400 0x55c144792b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.057 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.054+0000 7f87ed03e280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c1455ed800 0x55c144792580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.057 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.054+0000 7f87ed03e280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1455edc00 0x55c144794100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.057 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.054+0000 7f87ed03e280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c14478a180 con 0x55c1455ed400 2024-03-24T01:30:34.057 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.054+0000 7f87ed03e280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c145511500 con 0x55c1455edc00 2024-03-24T01:30:34.058 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.054+0000 7f87ed03e280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c145511380 con 0x55c1455ed800 2024-03-24T01:30:34.058 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.054+0000 7f87ea000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1455ed400 0x55c144792b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.058 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.054+0000 7f87ea000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1455ed400 0x55c144792b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36746/0 (socket says 172.21.15.90:36746) 2024-03-24T01:30:34.058 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.054+0000 7f87ea000640 1 -- 172.21.15.90:0/3277268231 learned_addr learned my addr 172.21.15.90:0/3277268231 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:34.058 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.055+0000 7f87ea000640 1 -- 172.21.15.90:0/3277268231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c1455ed800 msgr2=0x55c144792580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:34.058 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.055+0000 7f87ea000640 1 --2- 172.21.15.90:0/3277268231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c1455ed800 0x55c144792580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.058 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.055+0000 7f87ea000640 1 -- 172.21.15.90:0/3277268231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1455edc00 msgr2=0x55c144794100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:34.058 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.055+0000 7f87ea000640 1 --2- 172.21.15.90:0/3277268231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1455edc00 0x55c144794100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.058 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.055+0000 7f87ea000640 1 -- 172.21.15.90:0/3277268231 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c14475c3c0 con 0x55c1455ed400 2024-03-24T01:30:34.058 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.055+0000 7f87ea000640 1 --2- 172.21.15.90:0/3277268231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1455ed400 0x55c144792b00 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x55c145604690 tx=0x55c145606cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d33f8ed49679701a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.058 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.055+0000 7f87e5000640 1 -- 172.21.15.90:0/3277268231 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55c14474e680 con 0x55c1455ed400 2024-03-24T01:30:34.058 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.055+0000 7f87e5000640 1 -- 172.21.15.90:0/3277268231 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c144754fc0 con 0x55c1455ed400 2024-03-24T01:30:34.058 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.055+0000 7f87e5000640 1 -- 172.21.15.90:0/3277268231 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55c14474eb60 con 0x55c1455ed400 2024-03-24T01:30:34.058 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.056+0000 7f87ed03e280 1 -- 172.21.15.90:0/3277268231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1455ed400 msgr2=0x55c144792b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.059 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.056+0000 7f87ed03e280 1 --2- 172.21.15.90:0/3277268231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1455ed400 0x55c144792b00 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x55c145604690 tx=0x55c145606cc0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.059 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.056+0000 7f87ed03e280 1 -- 172.21.15.90:0/3277268231 shutdown_connections 2024-03-24T01:30:34.059 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.056+0000 7f87ed03e280 1 --2- 172.21.15.90:0/3277268231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1455edc00 0x55c144794100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.059 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.056+0000 7f87ed03e280 1 --2- 172.21.15.90:0/3277268231 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c1455ed800 0x55c144792580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.059 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.056+0000 7f87ed03e280 1 --2- 172.21.15.90:0/3277268231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1455ed400 0x55c144792b00 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.059 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.056+0000 7f87ed03e280 1 -- 172.21.15.90:0/3277268231 >> 172.21.15.90:0/3277268231 conn(0x55c1455ed000 msgr2=0x55c1454a9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:34.059 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.056+0000 7f87ed03e280 1 -- 172.21.15.90:0/3277268231 shutdown_connections 2024-03-24T01:30:34.059 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.056+0000 7f87ed03e280 1 -- 172.21.15.90:0/3277268231 wait complete. 2024-03-24T01:30:34.061 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.056+0000 7f87ed03e280 1 Processor -- start 2024-03-24T01:30:34.061 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.057+0000 7f87ed03e280 1 -- start start 2024-03-24T01:30:34.061 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.057+0000 7f87ed03e280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1455ed400 0x55c144792b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.061 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.057+0000 7f87ed03e280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1455ed800 0x55c144792580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.061 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.057+0000 7f87ed03e280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c1455edc00 0x55c144794100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.061 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.057+0000 7f87ed03e280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c145510780 con 0x55c1455ed800 2024-03-24T01:30:34.061 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.057+0000 7f87ed03e280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c145510900 con 0x55c1455ed400 2024-03-24T01:30:34.062 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.057+0000 7f87ed03e280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c145510600 con 0x55c1455edc00 2024-03-24T01:30:34.062 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.057+0000 7f87eaa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1455ed800 0x55c144792580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.062 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.058+0000 7f87eaa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1455ed800 0x55c144792580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36762/0 (socket says 172.21.15.90:36762) 2024-03-24T01:30:34.062 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.058+0000 7f87eaa00640 1 -- 172.21.15.90:0/2640965771 learned_addr learned my addr 172.21.15.90:0/2640965771 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:34.062 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.058+0000 7f87eaa00640 1 -- 172.21.15.90:0/2640965771 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c1455edc00 msgr2=0x55c144794100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.062 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.058+0000 7f87eaa00640 1 --2- 172.21.15.90:0/2640965771 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c1455edc00 0x55c144794100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.062 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.058+0000 7f87eaa00640 1 -- 172.21.15.90:0/2640965771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1455ed400 msgr2=0x55c144792b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.062 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.058+0000 7f87eaa00640 1 --2- 172.21.15.90:0/2640965771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1455ed400 0x55c144792b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.062 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.058+0000 7f87eaa00640 1 -- 172.21.15.90:0/2640965771 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c14475c780 con 0x55c1455ed800 2024-03-24T01:30:34.062 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.058+0000 7f87eaa00640 1 --2- 172.21.15.90:0/2640965771 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1455ed800 0x55c144792580 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x55c1456052c0 tx=0x55c1454fef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.062 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.058+0000 7f87e3200640 1 -- 172.21.15.90:0/2640965771 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55c14474e9c0 con 0x55c1455ed800 2024-03-24T01:30:34.062 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.058+0000 7f87e3200640 1 -- 172.21.15.90:0/2640965771 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c144754a80 con 0x55c1455ed800 2024-03-24T01:30:34.062 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.058+0000 7f87ed03e280 1 -- 172.21.15.90:0/2640965771 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55c14475d4a0 con 0x55c1455ed800 2024-03-24T01:30:34.063 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.058+0000 7f87e3200640 1 -- 172.21.15.90:0/2640965771 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55c14474e680 con 0x55c1455ed800 2024-03-24T01:30:34.063 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.058+0000 7f87ed03e280 1 -- 172.21.15.90:0/2640965771 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c14475c960 con 0x55c1455ed800 2024-03-24T01:30:34.063 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.060+0000 7f87e3200640 1 -- 172.21.15.90:0/2640965771 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x55c145668c00 con 0x55c1455ed800 2024-03-24T01:30:34.065 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.060+0000 7f87e3200640 1 --2- 172.21.15.90:0/2640965771 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55c145667000 0x55c144a90000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.065 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.061+0000 7f87e3200640 1 -- 172.21.15.90:0/2640965771 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x55c14565ac00 con 0x55c1455ed800 2024-03-24T01:30:34.065 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.061+0000 7f87ed03e280 1 --2- 172.21.15.90:0/2640965771 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55c145667400 0x55c144794680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.065 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.061+0000 7f87ed03e280 1 -- 172.21.15.90:0/2640965771 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:1 7.f 7:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55c145667800 con 0x55c145667400 2024-03-24T01:30:34.065 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.061+0000 7f87ea000640 1 --2- 172.21.15.90:0/2640965771 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55c145667400 0x55c144794680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.065 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.061+0000 7f87ea000640 1 --2- 172.21.15.90:0/2640965771 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55c145667000 0x55c144a90000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.065 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.062+0000 7f87ea000640 1 --2- 172.21.15.90:0/2640965771 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55c145667400 0x55c144794680 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.065 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.062+0000 7f87ea000640 1 -- 172.21.15.90:0/2640965771 <== osd.0 v2:172.21.15.90:6800/307045587 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x55c1454ac000 con 0x55c145667400 2024-03-24T01:30:34.066 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.062+0000 7f87ed03e280 1 --2- 172.21.15.90:0/2640965771 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55c145667c00 0x55c144794c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.066 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.062+0000 7f87ed03e280 1 -- 172.21.15.90:0/2640965771 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:2 7.14 7:292cf221:::200.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55c145664000 con 0x55c145667c00 2024-03-24T01:30:34.066 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.062+0000 7f87ea000640 1 --2- 172.21.15.90:0/2640965771 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55c145667000 0x55c144a90000 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x55c145605890 tx=0x55c145606d20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.066 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.062+0000 7f87eb400640 1 --2- 172.21.15.90:0/2640965771 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55c145667c00 0x55c144794c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.066 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.063+0000 7f87eb400640 1 --2- 172.21.15.90:0/2640965771 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55c145667c00 0x55c144794c00 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.066 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.063+0000 7f87eb400640 1 -- 172.21.15.90:0/2640965771 <== osd.3 v2:172.21.15.90:6808/2999931983 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv34 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x55c1454ac500 con 0x55c145667c00 2024-03-24T01:30:34.066 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.063+0000 7f87ed03e280 1 -- 172.21.15.90:0/2640965771 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:3 7.14 7:2e2fa760:::200.00000001:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55c145664400 con 0x55c145667c00 2024-03-24T01:30:34.066 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.063+0000 7f87eb400640 1 -- 172.21.15.90:0/2640965771 <== osd.3 v2:172.21.15.90:6808/2999931983 2 ==== osd_op_reply(3 200.00000001 [read 0~35827 out=35827b] v0'0 uv36 ondisk = 0) v8 ==== 156+0+35827 (crc 0 0 0) 0x55c1454ac500 con 0x55c145667c00 2024-03-24T01:30:34.066 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.063+0000 7f87ed03e280 1 -- 172.21.15.90:0/2640965771 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:4 7.1b 7:ddb4e4d7:::200.00000002:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55c145664800 con 0x55c145667400 2024-03-24T01:30:34.066 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.064+0000 7f87ea000640 1 -- 172.21.15.90:0/2640965771 <== osd.0 v2:172.21.15.90:6800/307045587 2 ==== osd_op_reply(4 200.00000002 [read 0~2147483647] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x55c1454ac000 con 0x55c145667400 2024-03-24T01:30:34.067 INFO:tasks.workunit.client.0.smithi090.stdout:Events by type: 2024-03-24T01:30:34.067 INFO:tasks.workunit.client.0.smithi090.stdout: UPDATE: 5 2024-03-24T01:30:34.067 INFO:tasks.workunit.client.0.smithi090.stdout:Errors: 0 2024-03-24T01:30:34.068 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.064+0000 7f87ed03e280 1 -- 172.21.15.90:0/2640965771 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55c145667400 msgr2=0x55c144794680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.068 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.064+0000 7f87ed03e280 1 --2- 172.21.15.90:0/2640965771 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55c145667400 0x55c144794680 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.068 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.064+0000 7f87ed03e280 1 -- 172.21.15.90:0/2640965771 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55c145667c00 msgr2=0x55c144794c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.068 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.064+0000 7f87ed03e280 1 --2- 172.21.15.90:0/2640965771 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55c145667c00 0x55c144794c00 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.068 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.064+0000 7f87ed03e280 1 -- 172.21.15.90:0/2640965771 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55c145667000 msgr2=0x55c144a90000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.068 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.064+0000 7f87ed03e280 1 --2- 172.21.15.90:0/2640965771 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55c145667000 0x55c144a90000 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x55c145605890 tx=0x55c145606d20 comp rx=0 tx=0).stop 2024-03-24T01:30:34.069 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.064+0000 7f87ed03e280 1 -- 172.21.15.90:0/2640965771 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1455ed800 msgr2=0x55c144792580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.069 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.064+0000 7f87ed03e280 1 --2- 172.21.15.90:0/2640965771 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1455ed800 0x55c144792580 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x55c1456052c0 tx=0x55c1454fef60 comp rx=0 tx=0).stop 2024-03-24T01:30:34.069 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.064+0000 7f87eaa00640 1 -- 172.21.15.90:0/2640965771 reap_dead start 2024-03-24T01:30:34.069 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.064+0000 7f87eaa00640 1 -- 172.21.15.90:0/2640965771 reap_dead start 2024-03-24T01:30:34.069 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.064+0000 7f87ed03e280 1 -- 172.21.15.90:0/2640965771 shutdown_connections 2024-03-24T01:30:34.069 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.064+0000 7f87ed03e280 1 -- 172.21.15.90:0/2640965771 >> 172.21.15.90:0/2640965771 conn(0x55c1455ed000 msgr2=0x55c1454a9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:34.069 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.064+0000 7f87ed03e280 1 -- 172.21.15.90:0/2640965771 shutdown_connections 2024-03-24T01:30:34.069 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.064+0000 7f87ed03e280 1 -- 172.21.15.90:0/2640965771 wait complete. 2024-03-24T01:30:34.069 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.065+0000 7f87ed03e280 1 -- 172.21.15.90:0/3405509643 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c144729c00 msgr2=0x55c144793080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.069 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.065+0000 7f87ed03e280 1 --2- 172.21.15.90:0/3405509643 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c144729c00 0x55c144793080 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x55c145549350 tx=0x55c1454ff140 comp rx=0 tx=0).stop 2024-03-24T01:30:34.069 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.065+0000 7f87ed03e280 1 -- 172.21.15.90:0/3405509643 shutdown_connections 2024-03-24T01:30:34.069 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.065+0000 7f87ed03e280 1 --2- 172.21.15.90:0/3405509643 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c144729c00 0x55c144793080 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.069 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.065+0000 7f87ed03e280 1 --2- 172.21.15.90:0/3405509643 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c144729800 0x55c144793600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.069 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.065+0000 7f87ed03e280 1 --2- 172.21.15.90:0/3405509643 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55c144728c00 0x55c144793b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.070 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.065+0000 7f87ed03e280 1 -- 172.21.15.90:0/3405509643 >> 172.21.15.90:0/3405509643 conn(0x55c144729000 msgr2=0x55c1454a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:34.070 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.065+0000 7f87ed03e280 1 -- 172.21.15.90:0/3405509643 shutdown_connections 2024-03-24T01:30:34.070 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.065+0000 7f87ed03e280 1 -- 172.21.15.90:0/3405509643 wait complete. 2024-03-24T01:30:34.070 INFO:tasks.workunit.client.0.smithi090.stderr:+ cephfs-journal-tool --rank=cephfs:0 event get json --path /tmp/json.tmp 2024-03-24T01:30:34.116 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.114+0000 7f6659e71280 1 -- start start 2024-03-24T01:30:34.116 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.115+0000 7f6659e71280 1 Processor -- start 2024-03-24T01:30:34.117 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.115+0000 7f6659e71280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558e0ed34c00 0x558e0ed9e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.117 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.115+0000 7f6659e71280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558e0ed35800 0x558e0ed9eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.117 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.115+0000 7f6659e71280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e0ed35c00 0x558e0ed9f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.117 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.115+0000 7f6659e71280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e0fb0e180 con 0x558e0ed34c00 2024-03-24T01:30:34.117 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.115+0000 7f6659e71280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e0fb0e300 con 0x558e0ed35c00 2024-03-24T01:30:34.117 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.115+0000 7f6659e71280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x558e0fb0e480 con 0x558e0ed35800 2024-03-24T01:30:34.118 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.116+0000 7f6658000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558e0ed34c00 0x558e0ed9e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.118 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.116+0000 7f6658000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558e0ed34c00 0x558e0ed9e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36772/0 (socket says 172.21.15.90:36772) 2024-03-24T01:30:34.118 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.116+0000 7f6658000640 1 -- 172.21.15.90:0/736435719 learned_addr learned my addr 172.21.15.90:0/736435719 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:34.118 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.116+0000 7f6658000640 1 -- 172.21.15.90:0/736435719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558e0ed35800 msgr2=0x558e0ed9eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:34.118 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.116+0000 7f6658000640 1 --2- 172.21.15.90:0/736435719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558e0ed35800 0x558e0ed9eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.118 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.116+0000 7f6658000640 1 -- 172.21.15.90:0/736435719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e0ed35c00 msgr2=0x558e0ed9f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:34.118 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.116+0000 7f6658000640 1 --2- 172.21.15.90:0/736435719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e0ed35c00 0x558e0ed9f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.118 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.116+0000 7f6658000640 1 -- 172.21.15.90:0/736435719 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558e0ed68780 con 0x558e0ed34c00 2024-03-24T01:30:34.119 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.116+0000 7f6658000640 1 --2- 172.21.15.90:0/736435719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558e0ed34c00 0x558e0ed9e580 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x558e0fb4f440 tx=0x558e0ee1cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=124c4f0ed683dbc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.119 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.117+0000 7f6654e00640 1 -- 172.21.15.90:0/736435719 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x558e0ed5a680 con 0x558e0ed34c00 2024-03-24T01:30:34.119 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.117+0000 7f6654e00640 1 -- 172.21.15.90:0/736435719 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558e0ed60700 con 0x558e0ed34c00 2024-03-24T01:30:34.120 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.117+0000 7f6654e00640 1 -- 172.21.15.90:0/736435719 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x558e0ed5ab60 con 0x558e0ed34c00 2024-03-24T01:30:34.120 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.117+0000 7f6659e71280 1 -- 172.21.15.90:0/736435719 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558e0ed694a0 con 0x558e0ed34c00 2024-03-24T01:30:34.120 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.117+0000 7f6659e71280 1 -- 172.21.15.90:0/736435719 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x558e0ed68000 con 0x558e0ed34c00 2024-03-24T01:30:34.120 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.117+0000 7f6654e00640 1 -- 172.21.15.90:0/736435719 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x558e0fbfa600 con 0x558e0ed34c00 2024-03-24T01:30:34.120 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.117+0000 7f6654e00640 1 -- 172.21.15.90:0/736435719 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 36) v1 ==== 3310+0+0 (secure 0 0 0) 0x558e0ed8f180 con 0x558e0ed34c00 2024-03-24T01:30:34.120 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.118+0000 7f6659e71280 1 Processor -- start 2024-03-24T01:30:34.120 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.118+0000 7f6659e71280 1 -- start start 2024-03-24T01:30:34.122 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.119+0000 7f6659e71280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e0fbfd400 0x558e0ed9f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.122 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.119+0000 7f6659e71280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558e0fbfd800 0x558e0ed9fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.122 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.119+0000 7f6659e71280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558e0fbfdc00 0x558e0eda0100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.123 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.119+0000 7f6659e71280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e0fb0e600 con 0x558e0fbfdc00 2024-03-24T01:30:34.123 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.119+0000 7f6659e71280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e0fb0e780 con 0x558e0fbfd400 2024-03-24T01:30:34.123 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.119+0000 7f6659e71280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x558e0fb0e900 con 0x558e0fbfd800 2024-03-24T01:30:34.123 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.119+0000 7f6657600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558e0fbfdc00 0x558e0eda0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.123 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.119+0000 7f6657600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558e0fbfdc00 0x558e0eda0100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36784/0 (socket says 172.21.15.90:36784) 2024-03-24T01:30:34.123 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.119+0000 7f6657600640 1 -- 172.21.15.90:0/2571169331 learned_addr learned my addr 172.21.15.90:0/2571169331 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:34.123 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.119+0000 7f6658a00640 1 --2- 172.21.15.90:0/2571169331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e0fbfd400 0x558e0ed9f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.123 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.119+0000 7f6657600640 1 -- 172.21.15.90:0/2571169331 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558e0fbfd800 msgr2=0x558e0ed9fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:34.123 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.119+0000 7f6657600640 1 --2- 172.21.15.90:0/2571169331 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558e0fbfd800 0x558e0ed9fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.123 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.119+0000 7f6657600640 1 -- 172.21.15.90:0/2571169331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e0fbfd400 msgr2=0x558e0ed9f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:34.123 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.120+0000 7f6657600640 1 --2- 172.21.15.90:0/2571169331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e0fbfd400 0x558e0ed9f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.123 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.120+0000 7f6657600640 1 -- 172.21.15.90:0/2571169331 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558e0ed683c0 con 0x558e0fbfdc00 2024-03-24T01:30:34.123 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.120+0000 7f6658a00640 1 --2- 172.21.15.90:0/2571169331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e0fbfd400 0x558e0ed9f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:30:34.123 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.120+0000 7f6657600640 1 --2- 172.21.15.90:0/2571169331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558e0fbfdc00 0x558e0eda0100 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x558e0fb4ea50 tx=0x558e0fc1ad20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3f6cc238fd096ac server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.124 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.120+0000 7f6652600640 1 -- 172.21.15.90:0/2571169331 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x558e0ed5a9c0 con 0x558e0fbfdc00 2024-03-24T01:30:34.124 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.120+0000 7f6652600640 1 -- 172.21.15.90:0/2571169331 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558e0ed60fc0 con 0x558e0fbfdc00 2024-03-24T01:30:34.124 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.120+0000 7f6652600640 1 -- 172.21.15.90:0/2571169331 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x558e0ed5a680 con 0x558e0fbfdc00 2024-03-24T01:30:34.124 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.120+0000 7f6659e71280 1 -- 172.21.15.90:0/2571169331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558e0fbfdc00 msgr2=0x558e0eda0100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.124 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.120+0000 7f6659e71280 1 --2- 172.21.15.90:0/2571169331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558e0fbfdc00 0x558e0eda0100 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x558e0fb4ea50 tx=0x558e0fc1ad20 comp rx=0 tx=0).stop 2024-03-24T01:30:34.124 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.120+0000 7f6659e71280 1 -- 172.21.15.90:0/2571169331 shutdown_connections 2024-03-24T01:30:34.124 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.120+0000 7f6659e71280 1 --2- 172.21.15.90:0/2571169331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558e0fbfdc00 0x558e0eda0100 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.124 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.120+0000 7f6659e71280 1 --2- 172.21.15.90:0/2571169331 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558e0fbfd800 0x558e0ed9fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.124 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.120+0000 7f6659e71280 1 --2- 172.21.15.90:0/2571169331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e0fbfd400 0x558e0ed9f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.124 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.120+0000 7f6659e71280 1 -- 172.21.15.90:0/2571169331 >> 172.21.15.90:0/2571169331 conn(0x558e0fbfd000 msgr2=0x558e0fab5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:34.124 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.121+0000 7f6659e71280 1 -- 172.21.15.90:0/2571169331 shutdown_connections 2024-03-24T01:30:34.124 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.121+0000 7f6659e71280 1 -- 172.21.15.90:0/2571169331 wait complete. 2024-03-24T01:30:34.124 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.121+0000 7f6659e71280 1 Processor -- start 2024-03-24T01:30:34.124 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.121+0000 7f6659e71280 1 -- start start 2024-03-24T01:30:34.126 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.122+0000 7f6659e71280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558e0fbfd400 0x558e0eda1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.127 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.122+0000 7f6659e71280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e0fbfd800 0x558e0eda0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.127 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.122+0000 7f6659e71280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558e0fbfdc00 0x558e0eda0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.127 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.122+0000 7f6659e71280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e0ed96180 con 0x558e0fbfdc00 2024-03-24T01:30:34.127 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.122+0000 7f6659e71280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e0fb0fe00 con 0x558e0fbfd800 2024-03-24T01:30:34.127 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.122+0000 7f6659e71280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x558e0fb0ed80 con 0x558e0fbfd400 2024-03-24T01:30:34.127 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.122+0000 7f6657600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558e0fbfdc00 0x558e0eda0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.127 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.122+0000 7f6657600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558e0fbfdc00 0x558e0eda0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36792/0 (socket says 172.21.15.90:36792) 2024-03-24T01:30:34.127 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.122+0000 7f6657600640 1 -- 172.21.15.90:0/1076806443 learned_addr learned my addr 172.21.15.90:0/1076806443 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:34.127 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.123+0000 7f6657600640 1 -- 172.21.15.90:0/1076806443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558e0fbfd400 msgr2=0x558e0eda1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.127 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.123+0000 7f6657600640 1 --2- 172.21.15.90:0/1076806443 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558e0fbfd400 0x558e0eda1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.127 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.123+0000 7f6657600640 1 -- 172.21.15.90:0/1076806443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e0fbfd800 msgr2=0x558e0eda0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.127 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.123+0000 7f6658000640 1 --2- 172.21.15.90:0/1076806443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e0fbfd800 0x558e0eda0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.128 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.123+0000 7f6657600640 1 --2- 172.21.15.90:0/1076806443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e0fbfd800 0x558e0eda0c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.128 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.123+0000 7f6657600640 1 -- 172.21.15.90:0/1076806443 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558e0ed68780 con 0x558e0fbfdc00 2024-03-24T01:30:34.128 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.123+0000 7f6657600640 1 --2- 172.21.15.90:0/1076806443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558e0fbfdc00 0x558e0eda0680 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x558e0fc18c00 tx=0x558e0fc1ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.128 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.123+0000 7f6650800640 1 -- 172.21.15.90:0/1076806443 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x558e0ed5ab60 con 0x558e0fbfdc00 2024-03-24T01:30:34.128 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.123+0000 7f6650800640 1 -- 172.21.15.90:0/1076806443 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558e0ed60e00 con 0x558e0fbfdc00 2024-03-24T01:30:34.128 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.123+0000 7f6650800640 1 -- 172.21.15.90:0/1076806443 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x558e0ed5a4e0 con 0x558e0fbfdc00 2024-03-24T01:30:34.128 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.123+0000 7f6659e71280 1 -- 172.21.15.90:0/1076806443 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x558e0ed694a0 con 0x558e0fbfdc00 2024-03-24T01:30:34.128 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.123+0000 7f6659e71280 1 -- 172.21.15.90:0/1076806443 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558e0ed68960 con 0x558e0fbfdc00 2024-03-24T01:30:34.128 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.125+0000 7f6650800640 1 -- 172.21.15.90:0/1076806443 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x558e0fc68c00 con 0x558e0fbfdc00 2024-03-24T01:30:34.128 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.125+0000 7f6650800640 1 --2- 172.21.15.90:0/1076806443 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x558e0fb2a400 0x558e0eda1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.128 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.125+0000 7f6650800640 1 -- 172.21.15.90:0/1076806443 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x558e0fb24600 con 0x558e0fbfdc00 2024-03-24T01:30:34.128 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.126+0000 7f6659e71280 1 --2- 172.21.15.90:0/1076806443 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x558e0fb2a800 0x558e0eda0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.128 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.126+0000 7f6659e71280 1 -- 172.21.15.90:0/1076806443 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:1 7.f 7:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x558e0fb2ac00 con 0x558e0fb2a800 2024-03-24T01:30:34.130 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.126+0000 7f6658000640 1 --2- 172.21.15.90:0/1076806443 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x558e0fb2a800 0x558e0eda0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.130 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.126+0000 7f6658a00640 1 --2- 172.21.15.90:0/1076806443 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x558e0fb2a400 0x558e0eda1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.130 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.126+0000 7f6658000640 1 --2- 172.21.15.90:0/1076806443 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x558e0fb2a800 0x558e0eda0100 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.130 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.127+0000 7f6658000640 1 -- 172.21.15.90:0/1076806443 <== osd.0 v2:172.21.15.90:6800/307045587 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x558e0fab8000 con 0x558e0fb2a800 2024-03-24T01:30:34.130 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.127+0000 7f6659e71280 1 --2- 172.21.15.90:0/1076806443 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x558e0fb2b000 0x558e0ed9fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.130 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.127+0000 7f6659e71280 1 -- 172.21.15.90:0/1076806443 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:2 7.14 7:292cf221:::200.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x558e0fb2b400 con 0x558e0fb2b000 2024-03-24T01:30:34.131 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.127+0000 7f6658a00640 1 --2- 172.21.15.90:0/1076806443 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x558e0fb2b000 0x558e0ed9fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.131 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.127+0000 7f6658a00640 1 --2- 172.21.15.90:0/1076806443 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x558e0fb2a400 0x558e0eda1700 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x558e0fb4f320 tx=0x558e0f0a80c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.131 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.127+0000 7f6658a00640 1 --2- 172.21.15.90:0/1076806443 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x558e0fb2b000 0x558e0ed9fb80 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.131 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.128+0000 7f6658a00640 1 -- 172.21.15.90:0/1076806443 <== osd.3 v2:172.21.15.90:6808/2999931983 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv34 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x558e0fab8500 con 0x558e0fb2b000 2024-03-24T01:30:34.131 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.128+0000 7f6659e71280 1 -- 172.21.15.90:0/1076806443 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:3 7.14 7:2e2fa760:::200.00000001:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x558e0fb2b800 con 0x558e0fb2b000 2024-03-24T01:30:34.131 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.128+0000 7f6658a00640 1 -- 172.21.15.90:0/1076806443 <== osd.3 v2:172.21.15.90:6808/2999931983 2 ==== osd_op_reply(3 200.00000001 [read 0~35827 out=35827b] v0'0 uv36 ondisk = 0) v8 ==== 156+0+35827 (crc 0 0 0) 0x558e0fab8500 con 0x558e0fb2b000 2024-03-24T01:30:34.131 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.128+0000 7f6659e71280 1 -- 172.21.15.90:0/1076806443 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:4 7.1b 7:ddb4e4d7:::200.00000002:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x558e0f0ac800 con 0x558e0fb2a800 2024-03-24T01:30:34.131 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.129+0000 7f6658000640 1 -- 172.21.15.90:0/1076806443 <== osd.0 v2:172.21.15.90:6800/307045587 2 ==== osd_op_reply(4 200.00000002 [read 0~2147483647] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x558e0fab8000 con 0x558e0fb2a800 2024-03-24T01:30:34.135 INFO:tasks.workunit.client.0.smithi090.stderr:Wrote output to JSON file '/tmp/json.tmp' 2024-03-24T01:30:34.135 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.133+0000 7f6659e71280 1 -- 172.21.15.90:0/1076806443 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x558e0fb2a800 msgr2=0x558e0eda0100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.135 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.133+0000 7f6659e71280 1 --2- 172.21.15.90:0/1076806443 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x558e0fb2a800 0x558e0eda0100 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.135 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.133+0000 7f6659e71280 1 -- 172.21.15.90:0/1076806443 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x558e0fb2b000 msgr2=0x558e0ed9fb80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.135 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.133+0000 7f6659e71280 1 --2- 172.21.15.90:0/1076806443 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x558e0fb2b000 0x558e0ed9fb80 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.135 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.133+0000 7f6659e71280 1 -- 172.21.15.90:0/1076806443 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x558e0fb2a400 msgr2=0x558e0eda1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.135 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.133+0000 7f6659e71280 1 --2- 172.21.15.90:0/1076806443 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x558e0fb2a400 0x558e0eda1700 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x558e0fb4f320 tx=0x558e0f0a80c0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.136 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.133+0000 7f6659e71280 1 -- 172.21.15.90:0/1076806443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558e0fbfdc00 msgr2=0x558e0eda0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.136 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.133+0000 7f6659e71280 1 --2- 172.21.15.90:0/1076806443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558e0fbfdc00 0x558e0eda0680 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x558e0fc18c00 tx=0x558e0fc1ac00 comp rx=0 tx=0).stop 2024-03-24T01:30:34.136 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.133+0000 7f6657600640 1 -- 172.21.15.90:0/1076806443 reap_dead start 2024-03-24T01:30:34.136 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.133+0000 7f6657600640 1 -- 172.21.15.90:0/1076806443 reap_dead start 2024-03-24T01:30:34.136 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.133+0000 7f6659e71280 1 -- 172.21.15.90:0/1076806443 shutdown_connections 2024-03-24T01:30:34.136 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.133+0000 7f6659e71280 1 -- 172.21.15.90:0/1076806443 >> 172.21.15.90:0/1076806443 conn(0x558e0fbfd000 msgr2=0x558e0fab5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:34.136 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.133+0000 7f6659e71280 1 -- 172.21.15.90:0/1076806443 shutdown_connections 2024-03-24T01:30:34.136 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.133+0000 7f6659e71280 1 -- 172.21.15.90:0/1076806443 wait complete. 2024-03-24T01:30:34.136 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.133+0000 7f6659e71280 1 -- 172.21.15.90:0/736435719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558e0ed34c00 msgr2=0x558e0ed9e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.136 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.133+0000 7f6659e71280 1 --2- 172.21.15.90:0/736435719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558e0ed34c00 0x558e0ed9e580 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x558e0fb4f440 tx=0x558e0ee1cde0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.136 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.134+0000 7f6659e71280 1 -- 172.21.15.90:0/736435719 shutdown_connections 2024-03-24T01:30:34.136 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.134+0000 7f6659e71280 1 --2- 172.21.15.90:0/736435719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e0ed35c00 0x558e0ed9f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.136 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.134+0000 7f6659e71280 1 --2- 172.21.15.90:0/736435719 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x558e0ed35800 0x558e0ed9eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.136 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.134+0000 7f6659e71280 1 --2- 172.21.15.90:0/736435719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558e0ed34c00 0x558e0ed9e580 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.137 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.134+0000 7f6659e71280 1 -- 172.21.15.90:0/736435719 >> 172.21.15.90:0/736435719 conn(0x558e0ed35000 msgr2=0x558e0fab4800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:34.137 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.134+0000 7f6659e71280 1 -- 172.21.15.90:0/736435719 shutdown_connections 2024-03-24T01:30:34.137 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.134+0000 7f6659e71280 1 -- 172.21.15.90:0/736435719 wait complete. 2024-03-24T01:30:34.140 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' '!' -s /tmp/json.tmp ']' 2024-03-24T01:30:34.140 INFO:tasks.workunit.client.0.smithi090.stderr:+ cephfs-journal-tool --rank=cephfs:0 event get binary --path /tmp/binary.tmp 2024-03-24T01:30:34.184 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.183+0000 7f875600f280 1 -- start start 2024-03-24T01:30:34.185 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.183+0000 7f875600f280 1 Processor -- start 2024-03-24T01:30:34.187 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.184+0000 7f875600f280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557576950c00 0x5575769ba580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.187 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.184+0000 7f875600f280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557576951800 0x5575769bab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.187 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.184+0000 7f875600f280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557576951c00 0x5575769bb080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.187 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.184+0000 7f875600f280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55757771e600 con 0x557576950c00 2024-03-24T01:30:34.187 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.184+0000 7f875600f280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55757771e480 con 0x557576951800 2024-03-24T01:30:34.187 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.184+0000 7f875600f280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55757771e300 con 0x557576951c00 2024-03-24T01:30:34.187 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.185+0000 7f8753600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557576950c00 0x5575769ba580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.187 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.185+0000 7f8753600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557576950c00 0x5575769ba580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36812/0 (socket says 172.21.15.90:36812) 2024-03-24T01:30:34.188 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.185+0000 7f8753600640 1 -- 172.21.15.90:0/4154217768 learned_addr learned my addr 172.21.15.90:0/4154217768 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:34.188 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.185+0000 7f8753600640 1 -- 172.21.15.90:0/4154217768 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557576951c00 msgr2=0x5575769bb080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:34.188 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.185+0000 7f8753600640 1 --2- 172.21.15.90:0/4154217768 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557576951c00 0x5575769bb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.188 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.185+0000 7f8753600640 1 -- 172.21.15.90:0/4154217768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557576951800 msgr2=0x5575769bab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:34.188 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.185+0000 7f8753600640 1 --2- 172.21.15.90:0/4154217768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557576951800 0x5575769bab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.188 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.185+0000 7f8753600640 1 -- 172.21.15.90:0/4154217768 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557576984780 con 0x557576950c00 2024-03-24T01:30:34.188 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.185+0000 7f8753600640 1 --2- 172.21.15.90:0/4154217768 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557576950c00 0x5575769ba580 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x55757775f530 tx=0x557576a38de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fc558256b27700d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.188 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.185+0000 7f8750400640 1 -- 172.21.15.90:0/4154217768 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x557576976680 con 0x557576950c00 2024-03-24T01:30:34.188 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.185+0000 7f8750400640 1 -- 172.21.15.90:0/4154217768 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55757697ce00 con 0x557576950c00 2024-03-24T01:30:34.188 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.185+0000 7f8750400640 1 -- 172.21.15.90:0/4154217768 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x557576976b60 con 0x557576950c00 2024-03-24T01:30:34.189 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.185+0000 7f875600f280 1 -- 172.21.15.90:0/4154217768 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5575769854a0 con 0x557576950c00 2024-03-24T01:30:34.189 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.185+0000 7f875600f280 1 -- 172.21.15.90:0/4154217768 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x557576984000 con 0x557576950c00 2024-03-24T01:30:34.189 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.186+0000 7f8750400640 1 -- 172.21.15.90:0/4154217768 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x557577816600 con 0x557576950c00 2024-03-24T01:30:34.189 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.186+0000 7f8750400640 1 -- 172.21.15.90:0/4154217768 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 36) v1 ==== 3310+0+0 (secure 0 0 0) 0x5575769ab180 con 0x557576950c00 2024-03-24T01:30:34.201 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.187+0000 7f875600f280 1 Processor -- start 2024-03-24T01:30:34.201 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.187+0000 7f875600f280 1 -- start start 2024-03-24T01:30:34.201 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.188+0000 7f875600f280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557577819400 0x5575769bb600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.202 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.188+0000 7f875600f280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557577819800 0x5575769bbb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.202 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.188+0000 7f875600f280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557577819c00 0x5575769bc100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.202 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.188+0000 7f875600f280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55757771e180 con 0x557577819c00 2024-03-24T01:30:34.202 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.188+0000 7f875600f280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575769b2900 con 0x557577819800 2024-03-24T01:30:34.202 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.188+0000 7f875600f280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5575769b2480 con 0x557577819400 2024-03-24T01:30:34.202 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.188+0000 7f8752c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557577819c00 0x5575769bc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.202 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.188+0000 7f8752c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557577819c00 0x5575769bc100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36820/0 (socket says 172.21.15.90:36820) 2024-03-24T01:30:34.202 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.188+0000 7f8752c00640 1 -- 172.21.15.90:0/28194791 learned_addr learned my addr 172.21.15.90:0/28194791 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:34.202 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.188+0000 7f8753600640 1 --2- 172.21.15.90:0/28194791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557577819800 0x5575769bbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.202 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.188+0000 7f8754000640 1 --2- 172.21.15.90:0/28194791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557577819400 0x5575769bb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.202 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.188+0000 7f8752c00640 1 -- 172.21.15.90:0/28194791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557577819400 msgr2=0x5575769bb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:34.202 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.189+0000 7f8752c00640 1 --2- 172.21.15.90:0/28194791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557577819400 0x5575769bb600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.202 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.189+0000 7f8752c00640 1 -- 172.21.15.90:0/28194791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557577819800 msgr2=0x5575769bbb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:34.203 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.189+0000 7f8752c00640 1 --2- 172.21.15.90:0/28194791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557577819800 0x5575769bbb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.203 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.189+0000 7f8752c00640 1 -- 172.21.15.90:0/28194791 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5575769843c0 con 0x557577819c00 2024-03-24T01:30:34.203 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.189+0000 7f8753600640 1 --2- 172.21.15.90:0/28194791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557577819800 0x5575769bbb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:30:34.203 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.189+0000 7f8752c00640 1 --2- 172.21.15.90:0/28194791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557577819c00 0x5575769bc100 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x557577834a50 tx=0x557577837020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7305ac40a6ca8a5f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.203 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.189+0000 7f874dc00640 1 -- 172.21.15.90:0/28194791 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x557576976680 con 0x557577819c00 2024-03-24T01:30:34.203 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.189+0000 7f874dc00640 1 -- 172.21.15.90:0/28194791 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55757697ca80 con 0x557577819c00 2024-03-24T01:30:34.203 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.189+0000 7f874dc00640 1 -- 172.21.15.90:0/28194791 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x557576976b60 con 0x557577819c00 2024-03-24T01:30:34.203 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.189+0000 7f875600f280 1 -- 172.21.15.90:0/28194791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557577819c00 msgr2=0x5575769bc100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.203 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.189+0000 7f875600f280 1 --2- 172.21.15.90:0/28194791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557577819c00 0x5575769bc100 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x557577834a50 tx=0x557577837020 comp rx=0 tx=0).stop 2024-03-24T01:30:34.203 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.190+0000 7f875600f280 1 -- 172.21.15.90:0/28194791 shutdown_connections 2024-03-24T01:30:34.203 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.190+0000 7f875600f280 1 --2- 172.21.15.90:0/28194791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557577819c00 0x5575769bc100 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.203 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.190+0000 7f875600f280 1 --2- 172.21.15.90:0/28194791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557577819800 0x5575769bbb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.203 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.190+0000 7f875600f280 1 --2- 172.21.15.90:0/28194791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557577819400 0x5575769bb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.203 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.190+0000 7f875600f280 1 -- 172.21.15.90:0/28194791 >> 172.21.15.90:0/28194791 conn(0x557577819000 msgr2=0x5575776ce800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:34.203 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.190+0000 7f875600f280 1 -- 172.21.15.90:0/28194791 shutdown_connections 2024-03-24T01:30:34.204 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.190+0000 7f875600f280 1 -- 172.21.15.90:0/28194791 wait complete. 2024-03-24T01:30:34.204 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.190+0000 7f875600f280 1 Processor -- start 2024-03-24T01:30:34.204 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.190+0000 7f875600f280 1 -- start start 2024-03-24T01:30:34.204 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.191+0000 7f875600f280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557577819400 0x5575769bd180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.204 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.191+0000 7f875600f280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557577819800 0x5575769bcc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.204 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.191+0000 7f875600f280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557577819c00 0x5575769bc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.204 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.191+0000 7f875600f280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55757771f800 con 0x557577819c00 2024-03-24T01:30:34.204 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.191+0000 7f875600f280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55757771e180 con 0x557577819800 2024-03-24T01:30:34.204 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.191+0000 7f875600f280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55757771ed80 con 0x557577819400 2024-03-24T01:30:34.204 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.191+0000 7f8752c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557577819c00 0x5575769bc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.204 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.191+0000 7f8752c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557577819c00 0x5575769bc680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36834/0 (socket says 172.21.15.90:36834) 2024-03-24T01:30:34.204 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.191+0000 7f8752c00640 1 -- 172.21.15.90:0/1201092143 learned_addr learned my addr 172.21.15.90:0/1201092143 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:34.205 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.192+0000 7f8752c00640 1 -- 172.21.15.90:0/1201092143 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557577819400 msgr2=0x5575769bd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.205 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.192+0000 7f8752c00640 1 --2- 172.21.15.90:0/1201092143 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557577819400 0x5575769bd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.205 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.192+0000 7f8752c00640 1 -- 172.21.15.90:0/1201092143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557577819800 msgr2=0x5575769bcc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.205 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.192+0000 7f8753600640 1 --2- 172.21.15.90:0/1201092143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557577819800 0x5575769bcc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.205 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.192+0000 7f8752c00640 1 --2- 172.21.15.90:0/1201092143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557577819800 0x5575769bcc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.205 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.192+0000 7f8752c00640 1 -- 172.21.15.90:0/1201092143 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557576984780 con 0x557577819c00 2024-03-24T01:30:34.205 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.192+0000 7f8752c00640 1 --2- 172.21.15.90:0/1201092143 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557577819c00 0x5575769bc680 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x557577835710 tx=0x557577836c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.205 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.192+0000 7f874be00640 1 -- 172.21.15.90:0/1201092143 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5575769764e0 con 0x557577819c00 2024-03-24T01:30:34.205 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.192+0000 7f874be00640 1 -- 172.21.15.90:0/1201092143 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55757697cc40 con 0x557577819c00 2024-03-24T01:30:34.205 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.192+0000 7f874be00640 1 -- 172.21.15.90:0/1201092143 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x557576976680 con 0x557577819c00 2024-03-24T01:30:34.205 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.192+0000 7f875600f280 1 -- 172.21.15.90:0/1201092143 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5575769854a0 con 0x557577819c00 2024-03-24T01:30:34.205 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.192+0000 7f875600f280 1 -- 172.21.15.90:0/1201092143 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557576984960 con 0x557577819c00 2024-03-24T01:30:34.205 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.194+0000 7f874be00640 1 -- 172.21.15.90:0/1201092143 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x557577896c00 con 0x557577819c00 2024-03-24T01:30:34.205 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.194+0000 7f874be00640 1 --2- 172.21.15.90:0/1201092143 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x557577893000 0x5575769bd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.206 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.194+0000 7f874be00640 1 -- 172.21.15.90:0/1201092143 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x557577894a00 con 0x557577819c00 2024-03-24T01:30:34.206 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.194+0000 7f875600f280 1 --2- 172.21.15.90:0/1201092143 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x557577893400 0x5575769bc100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.206 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.194+0000 7f875600f280 1 -- 172.21.15.90:0/1201092143 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:1 7.f 7:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x557577893800 con 0x557577893400 2024-03-24T01:30:34.206 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.195+0000 7f8753600640 1 --2- 172.21.15.90:0/1201092143 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x557577893400 0x5575769bc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.206 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.195+0000 7f8754000640 1 --2- 172.21.15.90:0/1201092143 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x557577893000 0x5575769bd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.206 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.195+0000 7f8753600640 1 --2- 172.21.15.90:0/1201092143 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x557577893400 0x5575769bc100 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.206 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.195+0000 7f8753600640 1 -- 172.21.15.90:0/1201092143 <== osd.0 v2:172.21.15.90:6800/307045587 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x5575776d4000 con 0x557577893400 2024-03-24T01:30:34.206 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.196+0000 7f875600f280 1 --2- 172.21.15.90:0/1201092143 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x557577893c00 0x5575769bbb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.206 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.196+0000 7f875600f280 1 -- 172.21.15.90:0/1201092143 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:2 7.14 7:292cf221:::200.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x557576cd4000 con 0x557577893c00 2024-03-24T01:30:34.206 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.196+0000 7f8754000640 1 --2- 172.21.15.90:0/1201092143 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x557577893c00 0x5575769bbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.206 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.196+0000 7f8754000640 1 --2- 172.21.15.90:0/1201092143 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x557577893000 0x5575769bd700 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x5575769837d0 tx=0x557577836e40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.206 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.196+0000 7f8754000640 1 --2- 172.21.15.90:0/1201092143 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x557577893c00 0x5575769bbb80 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.206 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.197+0000 7f8754000640 1 -- 172.21.15.90:0/1201092143 <== osd.3 v2:172.21.15.90:6808/2999931983 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv34 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x5575776d4500 con 0x557577893c00 2024-03-24T01:30:34.207 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.197+0000 7f875600f280 1 -- 172.21.15.90:0/1201092143 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:3 7.14 7:2e2fa760:::200.00000001:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x557576cd4400 con 0x557577893c00 2024-03-24T01:30:34.207 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.197+0000 7f8754000640 1 -- 172.21.15.90:0/1201092143 <== osd.3 v2:172.21.15.90:6808/2999931983 2 ==== osd_op_reply(3 200.00000001 [read 0~35827 out=35827b] v0'0 uv36 ondisk = 0) v8 ==== 156+0+35827 (crc 0 0 0) 0x5575776d4500 con 0x557577893c00 2024-03-24T01:30:34.207 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.197+0000 7f875600f280 1 -- 172.21.15.90:0/1201092143 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:4 7.1b 7:ddb4e4d7:::200.00000002:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x557576cd4800 con 0x557577893400 2024-03-24T01:30:34.207 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.197+0000 7f8753600640 1 -- 172.21.15.90:0/1201092143 <== osd.0 v2:172.21.15.90:6800/307045587 2 ==== osd_op_reply(4 200.00000002 [read 0~2147483647] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x5575776d4000 con 0x557577893400 2024-03-24T01:30:34.207 INFO:tasks.workunit.client.0.smithi090.stderr:Wrote output to binary files in directory '/tmp/binary.tmp' 2024-03-24T01:30:34.207 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.199+0000 7f875600f280 1 -- 172.21.15.90:0/1201092143 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x557577893400 msgr2=0x5575769bc100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.207 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.199+0000 7f875600f280 1 --2- 172.21.15.90:0/1201092143 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x557577893400 0x5575769bc100 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.207 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.199+0000 7f875600f280 1 -- 172.21.15.90:0/1201092143 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x557577893c00 msgr2=0x5575769bbb80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.207 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.199+0000 7f875600f280 1 --2- 172.21.15.90:0/1201092143 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x557577893c00 0x5575769bbb80 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.207 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.199+0000 7f875600f280 1 -- 172.21.15.90:0/1201092143 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x557577893000 msgr2=0x5575769bd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.207 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.199+0000 7f875600f280 1 --2- 172.21.15.90:0/1201092143 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x557577893000 0x5575769bd700 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x5575769837d0 tx=0x557577836e40 comp rx=0 tx=0).stop 2024-03-24T01:30:34.207 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.199+0000 7f875600f280 1 -- 172.21.15.90:0/1201092143 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557577819c00 msgr2=0x5575769bc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.207 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.199+0000 7f875600f280 1 --2- 172.21.15.90:0/1201092143 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557577819c00 0x5575769bc680 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x557577835710 tx=0x557577836c00 comp rx=0 tx=0).stop 2024-03-24T01:30:34.207 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.199+0000 7f8752c00640 1 -- 172.21.15.90:0/1201092143 reap_dead start 2024-03-24T01:30:34.208 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.199+0000 7f8752c00640 1 -- 172.21.15.90:0/1201092143 reap_dead start 2024-03-24T01:30:34.208 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.199+0000 7f875600f280 1 -- 172.21.15.90:0/1201092143 shutdown_connections 2024-03-24T01:30:34.208 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.199+0000 7f875600f280 1 -- 172.21.15.90:0/1201092143 >> 172.21.15.90:0/1201092143 conn(0x557577819000 msgr2=0x5575776ce800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:34.208 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.199+0000 7f875600f280 1 -- 172.21.15.90:0/1201092143 shutdown_connections 2024-03-24T01:30:34.208 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.199+0000 7f875600f280 1 -- 172.21.15.90:0/1201092143 wait complete. 2024-03-24T01:30:34.208 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.200+0000 7f875600f280 1 -- 172.21.15.90:0/4154217768 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557576950c00 msgr2=0x5575769ba580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.208 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.200+0000 7f875600f280 1 --2- 172.21.15.90:0/4154217768 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557576950c00 0x5575769ba580 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x55757775f530 tx=0x557576a38de0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.208 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.200+0000 7f875600f280 1 -- 172.21.15.90:0/4154217768 shutdown_connections 2024-03-24T01:30:34.208 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.200+0000 7f875600f280 1 --2- 172.21.15.90:0/4154217768 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x557576951c00 0x5575769bb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.208 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.200+0000 7f875600f280 1 --2- 172.21.15.90:0/4154217768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557576951800 0x5575769bab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.208 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.200+0000 7f875600f280 1 --2- 172.21.15.90:0/4154217768 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557576950c00 0x5575769ba580 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.208 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.200+0000 7f875600f280 1 -- 172.21.15.90:0/4154217768 >> 172.21.15.90:0/4154217768 conn(0x557576951000 msgr2=0x5575776cf000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:34.208 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.200+0000 7f875600f280 1 -- 172.21.15.90:0/4154217768 shutdown_connections 2024-03-24T01:30:34.208 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.200+0000 7f875600f280 1 -- 172.21.15.90:0/4154217768 wait complete. 2024-03-24T01:30:34.209 INFO:tasks.workunit.client.0.smithi090.stderr:+ '[' '!' -s /tmp/binary.tmp ']' 2024-03-24T01:30:34.209 INFO:tasks.workunit.client.0.smithi090.stderr:+ cephfs-journal-tool --rank=cephfs:0 event recover_dentries summary 2024-03-24T01:30:34.250 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.249+0000 7fcecb046280 1 -- start start 2024-03-24T01:30:34.251 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.249+0000 7fcecb046280 1 Processor -- start 2024-03-24T01:30:34.253 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.250+0000 7fcecb046280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b44649c800 0x55b44573bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.253 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.250+0000 7fcecb046280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b4456d1c00 0x55b44573ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.253 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.250+0000 7fcecb046280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b44649dc00 0x55b44573b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.253 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.250+0000 7fcecb046280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4464ce000 con 0x55b44649dc00 2024-03-24T01:30:34.253 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.250+0000 7fcecb046280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4464ce180 con 0x55b44649c800 2024-03-24T01:30:34.253 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.250+0000 7fcecb046280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b4464ce300 con 0x55b4456d1c00 2024-03-24T01:30:34.253 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.250+0000 7fcec9400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b44649dc00 0x55b44573b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.253 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.250+0000 7fcec9400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b44649dc00 0x55b44573b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36850/0 (socket says 172.21.15.90:36850) 2024-03-24T01:30:34.253 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.250+0000 7fcec9400640 1 -- 172.21.15.90:0/4022602763 learned_addr learned my addr 172.21.15.90:0/4022602763 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:34.253 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.251+0000 7fcec9400640 1 -- 172.21.15.90:0/4022602763 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b4456d1c00 msgr2=0x55b44573ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:34.253 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.251+0000 7fcec9400640 1 --2- 172.21.15.90:0/4022602763 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b4456d1c00 0x55b44573ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.254 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.251+0000 7fcec9400640 1 -- 172.21.15.90:0/4022602763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b44649c800 msgr2=0x55b44573bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:34.254 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.251+0000 7fcec9400640 1 --2- 172.21.15.90:0/4022602763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b44649c800 0x55b44573bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.254 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.251+0000 7fcec9400640 1 -- 172.21.15.90:0/4022602763 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b445704780 con 0x55b44649dc00 2024-03-24T01:30:34.254 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.251+0000 7fcec9400640 1 --2- 172.21.15.90:0/4022602763 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b44649dc00 0x55b44573b600 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x55b4457b2810 tx=0x55b4457b8de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c715c61230a9d3e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.254 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.251+0000 7fcec5800640 1 -- 172.21.15.90:0/4022602763 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55b4456f6b60 con 0x55b44649dc00 2024-03-24T01:30:34.254 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.251+0000 7fcec5800640 1 -- 172.21.15.90:0/4022602763 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b4456fce00 con 0x55b44649dc00 2024-03-24T01:30:34.254 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.251+0000 7fcec5800640 1 -- 172.21.15.90:0/4022602763 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55b4456f6d00 con 0x55b44649dc00 2024-03-24T01:30:34.254 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.251+0000 7fcecb046280 1 -- 172.21.15.90:0/4022602763 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b445705a40 con 0x55b44649dc00 2024-03-24T01:30:34.254 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.251+0000 7fcecb046280 1 -- 172.21.15.90:0/4022602763 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x55b445705860 con 0x55b44649dc00 2024-03-24T01:30:34.254 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.252+0000 7fcec5800640 1 -- 172.21.15.90:0/4022602763 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x55b4464d0400 con 0x55b44649dc00 2024-03-24T01:30:34.254 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.252+0000 7fcec5800640 1 -- 172.21.15.90:0/4022602763 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 36) v1 ==== 3310+0+0 (secure 0 0 0) 0x55b44572b180 con 0x55b44649dc00 2024-03-24T01:30:34.257 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.253+0000 7fcecb046280 1 Processor -- start 2024-03-24T01:30:34.257 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.253+0000 7fcecb046280 1 -- start start 2024-03-24T01:30:34.257 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.253+0000 7fcecb046280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b44649c000 0x55b44573b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.257 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.253+0000 7fcecb046280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b4465a4400 0x55b44573c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.257 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.253+0000 7fcecb046280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4465a4800 0x55b44573c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.257 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.253+0000 7fcecb046280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4464ce480 con 0x55b44649c000 2024-03-24T01:30:34.257 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.253+0000 7fcecb046280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4464ce600 con 0x55b4465a4800 2024-03-24T01:30:34.257 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.253+0000 7fcecb046280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b4464ce780 con 0x55b4465a4400 2024-03-24T01:30:34.257 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.254+0000 7fcec8000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b44649c000 0x55b44573b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.258 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.254+0000 7fcec8000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b44649c000 0x55b44573b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36864/0 (socket says 172.21.15.90:36864) 2024-03-24T01:30:34.258 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.254+0000 7fcec8000640 1 -- 172.21.15.90:0/1523770074 learned_addr learned my addr 172.21.15.90:0/1523770074 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:34.258 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.254+0000 7fcec8000640 1 -- 172.21.15.90:0/1523770074 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b4465a4400 msgr2=0x55b44573c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:34.258 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.254+0000 7fcec8000640 1 --2- 172.21.15.90:0/1523770074 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b4465a4400 0x55b44573c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.258 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.254+0000 7fcec8000640 1 -- 172.21.15.90:0/1523770074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4465a4800 msgr2=0x55b44573c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:34.258 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.254+0000 7fcec8000640 1 --2- 172.21.15.90:0/1523770074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4465a4800 0x55b44573c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.258 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.254+0000 7fcec8000640 1 -- 172.21.15.90:0/1523770074 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b4457043c0 con 0x55b44649c000 2024-03-24T01:30:34.258 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.254+0000 7fcec8000640 1 --2- 172.21.15.90:0/1523770074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b44649c000 0x55b44573b080 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x55b4465a25d0 tx=0x55b4465a6d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca9e80e72dcad819 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.258 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.254+0000 7fcec3000640 1 -- 172.21.15.90:0/1523770074 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55b4456f6b60 con 0x55b44649c000 2024-03-24T01:30:34.258 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.255+0000 7fcec3000640 1 -- 172.21.15.90:0/1523770074 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b4456fc1c0 con 0x55b44649c000 2024-03-24T01:30:34.258 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.255+0000 7fcec3000640 1 -- 172.21.15.90:0/1523770074 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55b4456f6d00 con 0x55b44649c000 2024-03-24T01:30:34.258 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.255+0000 7fcecb046280 1 -- 172.21.15.90:0/1523770074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b44649c000 msgr2=0x55b44573b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.258 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.255+0000 7fcecb046280 1 --2- 172.21.15.90:0/1523770074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b44649c000 0x55b44573b080 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x55b4465a25d0 tx=0x55b4465a6d20 comp rx=0 tx=0).stop 2024-03-24T01:30:34.258 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.255+0000 7fcecb046280 1 -- 172.21.15.90:0/1523770074 shutdown_connections 2024-03-24T01:30:34.259 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.255+0000 7fcecb046280 1 --2- 172.21.15.90:0/1523770074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4465a4800 0x55b44573c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.259 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.255+0000 7fcecb046280 1 --2- 172.21.15.90:0/1523770074 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b4465a4400 0x55b44573c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.259 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.255+0000 7fcecb046280 1 --2- 172.21.15.90:0/1523770074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b44649c000 0x55b44573b080 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.259 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.255+0000 7fcecb046280 1 -- 172.21.15.90:0/1523770074 >> 172.21.15.90:0/1523770074 conn(0x55b44649c400 msgr2=0x55b446451800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:34.259 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.255+0000 7fcecb046280 1 -- 172.21.15.90:0/1523770074 shutdown_connections 2024-03-24T01:30:34.259 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.255+0000 7fcecb046280 1 -- 172.21.15.90:0/1523770074 wait complete. 2024-03-24T01:30:34.259 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.256+0000 7fcecb046280 1 Processor -- start 2024-03-24T01:30:34.259 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.256+0000 7fcecb046280 1 -- start start 2024-03-24T01:30:34.261 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.257+0000 7fcecb046280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b44649c400 0x55b44573d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.261 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.257+0000 7fcecb046280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b4465a5400 0x55b44573d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.261 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.257+0000 7fcecb046280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4465a5000 0x55b44573cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.261 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.257+0000 7fcecb046280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4464cfb00 con 0x55b4465a5400 2024-03-24T01:30:34.261 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.257+0000 7fcecb046280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4464cf980 con 0x55b4465a5000 2024-03-24T01:30:34.261 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.257+0000 7fcecb046280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b4464cec00 con 0x55b44649c400 2024-03-24T01:30:34.261 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.257+0000 7fcec8a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b4465a5400 0x55b44573d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.261 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.257+0000 7fcec8a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b4465a5400 0x55b44573d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36874/0 (socket says 172.21.15.90:36874) 2024-03-24T01:30:34.261 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.257+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 learned_addr learned my addr 172.21.15.90:0/868144061 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:34.262 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.257+0000 7fcec8000640 1 --2- 172.21.15.90:0/868144061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4465a5000 0x55b44573cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.262 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.257+0000 7fcec8000640 1 --2- 172.21.15.90:0/868144061 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b44649c400 0x55b44573d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.262 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.257+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b44649c400 msgr2=0x55b44573d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.262 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.257+0000 7fcec8a00640 1 --2- 172.21.15.90:0/868144061 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b44649c400 0x55b44573d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.262 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.257+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4465a5000 msgr2=0x55b44573cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.262 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.257+0000 7fcec8a00640 1 --2- 172.21.15.90:0/868144061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4465a5000 0x55b44573cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.262 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.257+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b445704780 con 0x55b4465a5400 2024-03-24T01:30:34.262 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.257+0000 7fcec8000640 1 --2- 172.21.15.90:0/868144061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4465a5000 0x55b44573cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:30:34.262 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.258+0000 7fcec8a00640 1 --2- 172.21.15.90:0/868144061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b4465a5400 0x55b44573d180 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x55b4465a32f0 tx=0x55b4465a6f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.262 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.258+0000 7fcec1200640 1 -- 172.21.15.90:0/868144061 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55b4456f6b60 con 0x55b4465a5400 2024-03-24T01:30:34.262 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.258+0000 7fcec1200640 1 -- 172.21.15.90:0/868144061 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b4456fc700 con 0x55b4465a5400 2024-03-24T01:30:34.263 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.258+0000 7fcec1200640 1 -- 172.21.15.90:0/868144061 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55b4456f6d00 con 0x55b4465a5400 2024-03-24T01:30:34.263 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.258+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55b445704000 con 0x55b4465a5400 2024-03-24T01:30:34.263 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.258+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b445705680 con 0x55b4465a5400 2024-03-24T01:30:34.263 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.259+0000 7fcec1200640 1 -- 172.21.15.90:0/868144061 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x55b4465fec00 con 0x55b4465a5400 2024-03-24T01:30:34.265 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.260+0000 7fcec1200640 1 --2- 172.21.15.90:0/868144061 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55b44661d000 0x55b445a48000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.265 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.260+0000 7fcec1200640 1 -- 172.21.15.90:0/868144061 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x55b4465fcc00 con 0x55b4465a5400 2024-03-24T01:30:34.265 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.260+0000 7fcecb046280 1 --2- 172.21.15.90:0/868144061 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55b4465a4c00 0x55b44573c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.265 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.260+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:1 7.f 7:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b4465a4800 con 0x55b4465a4c00 2024-03-24T01:30:34.265 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.261+0000 7fcec8000640 1 --2- 172.21.15.90:0/868144061 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55b4465a4c00 0x55b44573c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.265 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.261+0000 7fcec8000640 1 --2- 172.21.15.90:0/868144061 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55b44661d000 0x55b445a48000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.265 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.261+0000 7fcec8000640 1 --2- 172.21.15.90:0/868144061 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55b4465a4c00 0x55b44573c680 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.265 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.261+0000 7fcec8000640 1 -- 172.21.15.90:0/868144061 <== osd.0 v2:172.21.15.90:6800/307045587 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x55b446454000 con 0x55b4465a4c00 2024-03-24T01:30:34.265 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.261+0000 7fcecb046280 1 --2- 172.21.15.90:0/868144061 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55b4465a4400 0x55b44573c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.265 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.261+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:2 7.14 7:292cf221:::200.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b44661d400 con 0x55b4465a4400 2024-03-24T01:30:34.266 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.262+0000 7fcec9400640 1 --2- 172.21.15.90:0/868144061 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55b4465a4400 0x55b44573c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.266 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.262+0000 7fcec8000640 1 --2- 172.21.15.90:0/868144061 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55b44661d000 0x55b445a48000 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x55b4465a2780 tx=0x55b4465a6cc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.266 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.262+0000 7fcec9400640 1 --2- 172.21.15.90:0/868144061 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55b4465a4400 0x55b44573c100 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.266 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.263+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv34 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x55b446454500 con 0x55b4465a4400 2024-03-24T01:30:34.266 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.263+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:3 7.14 7:2e2fa760:::200.00000001:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b44661d800 con 0x55b4465a4400 2024-03-24T01:30:34.266 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.263+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 2 ==== osd_op_reply(3 200.00000001 [read 0~35827 out=35827b] v0'0 uv36 ondisk = 0) v8 ==== 156+0+35827 (crc 0 0 0) 0x55b446454500 con 0x55b4465a4400 2024-03-24T01:30:34.266 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.263+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:4 7.1b 7:ddb4e4d7:::200.00000002:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b44661dc00 con 0x55b4465a4c00 2024-03-24T01:30:34.266 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.263+0000 7fcec8000640 1 -- 172.21.15.90:0/868144061 <== osd.0 v2:172.21.15.90:6800/307045587 2 ==== osd_op_reply(4 200.00000002 [read 0~2147483647] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x55b446454000 con 0x55b4465a4c00 2024-03-24T01:30:34.266 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.264+0000 7fcecb046280 1 --2- 172.21.15.90:0/868144061 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x55b445a6e000 0x55b44573b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.266 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.264+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:5 7.1f 7:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6e400 con 0x55b445a6e000 2024-03-24T01:30:34.267 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.264+0000 7fcec8a00640 1 --2- 172.21.15.90:0/868144061 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x55b445a6e000 0x55b44573b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.267 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.265+0000 7fcec8a00640 1 --2- 172.21.15.90:0/868144061 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x55b445a6e000 0x55b44573b080 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.267 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.266+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 1 ==== osd_op_reply(5 1.00000000 [omap-get-header out=274b] v0'0 uv2 ondisk = 0) v8 ==== 154+0+274 (crc 0 0 0) 0x55b445a78000 con 0x55b445a6e000 2024-03-24T01:30:34.267 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.266+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:6 7.1f 7:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6e800 con 0x55b445a6e000 2024-03-24T01:30:34.268 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.267+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 2 ==== osd_op_reply(6 1.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv2 ondisk = 0) v8 ==== 154+0+4 (crc 0 0 0) 0x55b445a78000 con 0x55b445a6e000 2024-03-24T01:30:34.268 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.267+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:7 7.13 7:cd5a64a3:::100.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6ec00 con 0x55b4465a4c00 2024-03-24T01:30:34.268 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.267+0000 7fcec8000640 1 -- 172.21.15.90:0/868144061 <== osd.0 v2:172.21.15.90:6800/307045587 3 ==== osd_op_reply(7 100.00000000 [omap-get-header out=274b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+274 (crc 0 0 0) 0x55b446454000 con 0x55b4465a4c00 2024-03-24T01:30:34.268 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.267+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:8 7.13 7:cd5a64a3:::100.00000000:head [omap-get-vals-by-keys in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6f000 con 0x55b4465a4c00 2024-03-24T01:30:34.269 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.268+0000 7fcec8000640 1 -- 172.21.15.90:0/868144061 <== osd.0 v2:172.21.15.90:6800/307045587 4 ==== osd_op_reply(8 100.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+4 (crc 0 0 0) 0x55b446454000 con 0x55b4465a4c00 2024-03-24T01:30:34.269 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.268+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:9 7.14 7:287034c4:::1.00000000.inode:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6f400 con 0x55b4465a4400 2024-03-24T01:30:34.269 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.268+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 3 ==== osd_op_reply(9 1.00000000.inode [read 0~582 out=582b] v0'0 uv4 ondisk = 0) v8 ==== 160+0+582 (crc 0 0 0) 0x55b446454500 con 0x55b4465a4400 2024-03-24T01:30:34.269 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.268+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:10 7.14 7:287034c4:::1.00000000.inode:head [writefull 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6f800 con 0x55b4465a4400 2024-03-24T01:30:34.271 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.270+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 4 ==== osd_op_reply(10 1.00000000.inode [writefull 0~582] v56'37 uv37 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x55b446454500 con 0x55b4465a4400 2024-03-24T01:30:34.271 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.270+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:11 7.1f 7:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6fc00 con 0x55b445a6e000 2024-03-24T01:30:34.272 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.270+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 3 ==== osd_op_reply(11 1.00000000 [omap-get-header out=274b] v0'0 uv2 ondisk = 0) v8 ==== 154+0+274 (crc 0 0 0) 0x55b445a78000 con 0x55b445a6e000 2024-03-24T01:30:34.272 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.271+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:12 7.1f 7:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6ec00 con 0x55b445a6e000 2024-03-24T01:30:34.275 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.275+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 4 ==== osd_op_reply(12 1.00000000 [omap-set-header] v56'6 uv6 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55b445a78000 con 0x55b445a6e000 2024-03-24T01:30:34.275 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.275+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:13 7.1f 7:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=21b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6f000 con 0x55b445a6e000 2024-03-24T01:30:34.277 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.275+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 5 ==== osd_op_reply(13 1.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv6 ondisk = 0) v8 ==== 154+0+4 (crc 0 0 0) 0x55b445a78000 con 0x55b445a6e000 2024-03-24T01:30:34.277 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.275+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:14 7.1f 7:ff5b34d6:::1.00000000:head [omap-set-vals in=509b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b44661dc00 con 0x55b445a6e000 2024-03-24T01:30:34.280 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.280+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 6 ==== osd_op_reply(14 1.00000000 [omap-set-vals] v56'7 uv7 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55b445a78000 con 0x55b445a6e000 2024-03-24T01:30:34.280 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.280+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:15 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b4465a5800 con 0x55b4465a4400 2024-03-24T01:30:34.281 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.280+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 5 ==== osd_op_reply(15 10000000000.00000000 [omap-get-header] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 164+0+0 (crc 0 0 0) 0x55b446454500 con 0x55b4465a4400 2024-03-24T01:30:34.281 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.280+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:16 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6e400 con 0x55b4465a4400 2024-03-24T01:30:34.282 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.281+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 6 ==== osd_op_reply(16 10000000000.00000000 [omap-set-header] v56'9 uv9 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55b446454500 con 0x55b4465a4400 2024-03-24T01:30:34.282 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.281+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:17 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6e800 con 0x55b4465a4400 2024-03-24T01:30:34.283 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.282+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 7 ==== osd_op_reply(17 10000000000.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv9 ondisk = 0) v8 ==== 164+0+4 (crc 0 0 0) 0x55b446454500 con 0x55b4465a4400 2024-03-24T01:30:34.283 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.282+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:18 7.14 7:287034c4:::1.00000000.inode:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6fc00 con 0x55b4465a4400 2024-03-24T01:30:34.283 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.282+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 8 ==== osd_op_reply(18 1.00000000.inode [read 0~582 out=582b] v0'0 uv37 ondisk = 0) v8 ==== 160+0+582 (crc 0 0 0) 0x55b446454500 con 0x55b4465a4400 2024-03-24T01:30:34.283 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.282+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:19 7.14 7:287034c4:::1.00000000.inode:head [writefull 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6ec00 con 0x55b4465a4400 2024-03-24T01:30:34.285 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.284+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 9 ==== osd_op_reply(19 1.00000000.inode [writefull 0~582] v56'38 uv38 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x55b446454500 con 0x55b4465a4400 2024-03-24T01:30:34.285 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.284+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:20 7.1f 7:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6f400 con 0x55b445a6e000 2024-03-24T01:30:34.286 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.285+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 7 ==== osd_op_reply(20 1.00000000 [omap-get-header out=274b] v0'0 uv7 ondisk = 0) v8 ==== 154+0+274 (crc 0 0 0) 0x55b445a78000 con 0x55b445a6e000 2024-03-24T01:30:34.286 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.285+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:21 7.1f 7:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6f800 con 0x55b445a6e000 2024-03-24T01:30:34.290 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.289+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 8 ==== osd_op_reply(21 1.00000000 [omap-set-header] v56'8 uv8 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55b445a78000 con 0x55b445a6e000 2024-03-24T01:30:34.291 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.289+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:22 7.1f 7:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=21b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6e400 con 0x55b445a6e000 2024-03-24T01:30:34.292 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.290+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 9 ==== osd_op_reply(22 1.00000000 [omap-get-vals-by-keys out=509b] v0'0 uv8 ondisk = 0) v8 ==== 154+0+509 (crc 0 0 0) 0x55b445a78000 con 0x55b445a6e000 2024-03-24T01:30:34.292 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.290+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:23 7.1f 7:ff5b34d6:::1.00000000:head [omap-set-vals in=509b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6e800 con 0x55b445a6e000 2024-03-24T01:30:34.296 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.296+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 10 ==== osd_op_reply(23 1.00000000 [omap-set-vals] v56'9 uv9 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55b445a78000 con 0x55b445a6e000 2024-03-24T01:30:34.296 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.296+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:24 7.14 7:287034c4:::1.00000000.inode:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b4465a5800 con 0x55b4465a4400 2024-03-24T01:30:34.296 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.296+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 10 ==== osd_op_reply(24 1.00000000.inode [read 0~582 out=582b] v0'0 uv38 ondisk = 0) v8 ==== 160+0+582 (crc 0 0 0) 0x55b446454500 con 0x55b4465a4400 2024-03-24T01:30:34.296 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.296+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:25 7.14 7:287034c4:::1.00000000.inode:head [writefull 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b44661d400 con 0x55b4465a4400 2024-03-24T01:30:34.298 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.297+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 11 ==== osd_op_reply(25 1.00000000.inode [writefull 0~582] v56'39 uv39 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x55b446454500 con 0x55b4465a4400 2024-03-24T01:30:34.298 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.297+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:26 7.1f 7:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b44661d800 con 0x55b445a6e000 2024-03-24T01:30:34.299 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.299+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 11 ==== osd_op_reply(26 1.00000000 [omap-get-header out=274b] v0'0 uv9 ondisk = 0) v8 ==== 154+0+274 (crc 0 0 0) 0x55b445a69400 con 0x55b445a6e000 2024-03-24T01:30:34.299 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.299+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:27 7.1f 7:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6f000 con 0x55b445a6e000 2024-03-24T01:30:34.303 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.302+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 12 ==== osd_op_reply(27 1.00000000 [omap-set-header] v56'10 uv10 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55b445a69400 con 0x55b445a6e000 2024-03-24T01:30:34.303 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.302+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:28 7.1f 7:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=21b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6f400 con 0x55b445a6e000 2024-03-24T01:30:34.304 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.302+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 13 ==== osd_op_reply(28 1.00000000 [omap-get-vals-by-keys out=509b] v0'0 uv10 ondisk = 0) v8 ==== 154+0+509 (crc 0 0 0) 0x55b445a69400 con 0x55b445a6e000 2024-03-24T01:30:34.304 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.303+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:29 7.1f 7:ff5b34d6:::1.00000000:head [omap-set-vals in=509b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6f800 con 0x55b445a6e000 2024-03-24T01:30:34.308 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.307+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 14 ==== osd_op_reply(29 1.00000000 [omap-set-vals] v56'11 uv11 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55b445a69400 con 0x55b445a6e000 2024-03-24T01:30:34.308 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.307+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:30 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6e400 con 0x55b4465a4400 2024-03-24T01:30:34.308 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.307+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 12 ==== osd_op_reply(30 10000000000.00000000 [omap-get-header out=274b] v0'0 uv9 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55b446454500 con 0x55b4465a4400 2024-03-24T01:30:34.308 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.307+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:31 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b44661dc00 con 0x55b4465a4400 2024-03-24T01:30:34.310 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.309+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 13 ==== osd_op_reply(31 10000000000.00000000 [omap-set-header] v56'10 uv10 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55b446454500 con 0x55b4465a4400 2024-03-24T01:30:34.310 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.309+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:32 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a78000 con 0x55b4465a4400 2024-03-24T01:30:34.311 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.310+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 14 ==== osd_op_reply(32 10000000000.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv10 ondisk = 0) v8 ==== 164+0+4 (crc 0 0 0) 0x55b446454500 con 0x55b4465a4400 2024-03-24T01:30:34.311 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.310+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:33 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=504b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a78400 con 0x55b4465a4400 2024-03-24T01:30:34.313 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.312+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 15 ==== osd_op_reply(33 10000000000.00000000 [omap-set-vals] v56'11 uv11 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55b446454500 con 0x55b4465a4400 2024-03-24T01:30:34.313 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.312+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:34 7.7 7:e2e07930:::10000000001.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a78800 con 0x55b445a6e000 2024-03-24T01:30:34.314 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.313+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 15 ==== osd_op_reply(34 10000000001.00000000 [omap-get-header] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 164+0+0 (crc 0 0 0) 0x55b445a69400 con 0x55b445a6e000 2024-03-24T01:30:34.314 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.313+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:35 7.7 7:e2e07930:::10000000001.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a78c00 con 0x55b445a6e000 2024-03-24T01:30:34.317 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.316+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 16 ==== osd_op_reply(35 10000000001.00000000 [omap-set-header] v56'4 uv4 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55b445a69400 con 0x55b445a6e000 2024-03-24T01:30:34.318 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.316+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:36 7.7 7:e2e07930:::10000000001.00000000:head [omap-get-vals-by-keys in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a79000 con 0x55b445a6e000 2024-03-24T01:30:34.318 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.317+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 17 ==== osd_op_reply(36 10000000001.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv4 ondisk = 0) v8 ==== 164+0+4 (crc 0 0 0) 0x55b445a69400 con 0x55b445a6e000 2024-03-24T01:30:34.319 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.317+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:37 7.14 7:287034c4:::1.00000000.inode:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a79400 con 0x55b4465a4400 2024-03-24T01:30:34.320 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.318+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 16 ==== osd_op_reply(37 1.00000000.inode [read 0~582 out=582b] v0'0 uv39 ondisk = 0) v8 ==== 160+0+582 (crc 0 0 0) 0x55b446454500 con 0x55b4465a4400 2024-03-24T01:30:34.321 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.318+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:38 7.14 7:287034c4:::1.00000000.inode:head [writefull 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a79800 con 0x55b4465a4400 2024-03-24T01:30:34.323 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.320+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 17 ==== osd_op_reply(38 1.00000000.inode [writefull 0~582] v56'40 uv40 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x55b446454500 con 0x55b4465a4400 2024-03-24T01:30:34.323 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.321+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:39 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a79c00 con 0x55b4465a4400 2024-03-24T01:30:34.323 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.321+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 18 ==== osd_op_reply(39 10000000000.00000000 [omap-get-header out=274b] v0'0 uv11 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55b446454500 con 0x55b4465a4400 2024-03-24T01:30:34.323 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.321+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:40 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a78800 con 0x55b4465a4400 2024-03-24T01:30:34.325 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.324+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 19 ==== osd_op_reply(40 10000000000.00000000 [omap-set-header] v56'12 uv12 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55b446454500 con 0x55b4465a4400 2024-03-24T01:30:34.325 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.324+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:41 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a78000 con 0x55b4465a4400 2024-03-24T01:30:34.325 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.324+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 20 ==== osd_op_reply(41 10000000000.00000000 [omap-get-vals-by-keys out=504b] v0'0 uv12 ondisk = 0) v8 ==== 164+0+504 (crc 0 0 0) 0x55b446454500 con 0x55b4465a4400 2024-03-24T01:30:34.326 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.324+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:42 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=504b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a78400 con 0x55b4465a4400 2024-03-24T01:30:34.328 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.327+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 21 ==== osd_op_reply(42 10000000000.00000000 [omap-set-vals] v56'13 uv13 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55b446454500 con 0x55b4465a4400 2024-03-24T01:30:34.328 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.327+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:43 7.7 7:e2e07930:::10000000001.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b44661d800 con 0x55b445a6e000 2024-03-24T01:30:34.329 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.328+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 18 ==== osd_op_reply(43 10000000001.00000000 [omap-get-header out=274b] v0'0 uv4 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55b445a69400 con 0x55b445a6e000 2024-03-24T01:30:34.329 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.328+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:44 7.7 7:e2e07930:::10000000001.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b44661d400 con 0x55b445a6e000 2024-03-24T01:30:34.333 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.332+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 19 ==== osd_op_reply(44 10000000001.00000000 [omap-set-header] v56'5 uv5 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55b445a69400 con 0x55b445a6e000 2024-03-24T01:30:34.333 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.332+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:45 7.7 7:e2e07930:::10000000001.00000000:head [omap-get-vals-by-keys in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b44661dc00 con 0x55b445a6e000 2024-03-24T01:30:34.334 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.333+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 20 ==== osd_op_reply(45 10000000001.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv5 ondisk = 0) v8 ==== 164+0+4 (crc 0 0 0) 0x55b445a69400 con 0x55b445a6e000 2024-03-24T01:30:34.334 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.333+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:46 7.7 7:e2e07930:::10000000001.00000000:head [omap-set-vals in=504b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6e800 con 0x55b445a6e000 2024-03-24T01:30:34.337 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.336+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 21 ==== osd_op_reply(46 10000000001.00000000 [omap-set-vals] v56'6 uv6 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55b445a69400 con 0x55b445a6e000 2024-03-24T01:30:34.337 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.336+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:47 7.12 7:4bec3886:::10000000002.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6f000 con 0x55b445a6e000 2024-03-24T01:30:34.338 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.337+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 22 ==== osd_op_reply(47 10000000002.00000000 [omap-get-header] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 164+0+0 (crc 0 0 0) 0x55b445a69400 con 0x55b445a6e000 2024-03-24T01:30:34.339 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.337+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:48 7.12 7:4bec3886:::10000000002.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6f400 con 0x55b445a6e000 2024-03-24T01:30:34.342 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.341+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 23 ==== osd_op_reply(48 10000000002.00000000 [omap-set-header] v56'2 uv2 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55b445a69400 con 0x55b445a6e000 2024-03-24T01:30:34.342 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.341+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:49 7.12 7:4bec3886:::10000000002.00000000:head [omap-get-vals-by-keys in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6f800 con 0x55b445a6e000 2024-03-24T01:30:34.343 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.342+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 24 ==== osd_op_reply(49 10000000002.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv2 ondisk = 0) v8 ==== 164+0+4 (crc 0 0 0) 0x55b445a69400 con 0x55b445a6e000 2024-03-24T01:30:34.343 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.342+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:50 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6fc00 con 0x55b4465a4400 2024-03-24T01:30:34.344 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.343+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 22 ==== osd_op_reply(50 10000000000.00000000 [omap-get-header out=274b] v0'0 uv13 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55b445a7e000 con 0x55b4465a4400 2024-03-24T01:30:34.344 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.343+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:51 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6ec00 con 0x55b4465a4400 2024-03-24T01:30:34.347 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.346+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 23 ==== osd_op_reply(51 10000000000.00000000 [omap-set-header] v56'14 uv14 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55b445a7e000 con 0x55b4465a4400 2024-03-24T01:30:34.347 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.346+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:52 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6e400 con 0x55b4465a4400 2024-03-24T01:30:34.348 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.347+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 24 ==== osd_op_reply(52 10000000000.00000000 [omap-get-vals-by-keys out=504b] v0'0 uv14 ondisk = 0) v8 ==== 164+0+504 (crc 0 0 0) 0x55b445a7e000 con 0x55b4465a4400 2024-03-24T01:30:34.348 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.347+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:53 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=504b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b4465a5800 con 0x55b4465a4400 2024-03-24T01:30:34.353 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.352+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 25 ==== osd_op_reply(53 10000000000.00000000 [omap-set-vals] v56'15 uv15 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55b445a7e000 con 0x55b4465a4400 2024-03-24T01:30:34.353 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.352+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:54 7.7 7:e2e07930:::10000000001.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6fc00 con 0x55b445a6e000 2024-03-24T01:30:34.354 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.354+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 25 ==== osd_op_reply(54 10000000001.00000000 [omap-get-header out=274b] v0'0 uv6 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55b445a69400 con 0x55b445a6e000 2024-03-24T01:30:34.354 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.354+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:55 7.7 7:e2e07930:::10000000001.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6ec00 con 0x55b445a6e000 2024-03-24T01:30:34.357 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.356+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 26 ==== osd_op_reply(55 10000000001.00000000 [omap-set-header] v56'7 uv7 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55b445a69400 con 0x55b445a6e000 2024-03-24T01:30:34.358 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.356+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:56 7.7 7:e2e07930:::10000000001.00000000:head [omap-get-vals-by-keys in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6e400 con 0x55b445a6e000 2024-03-24T01:30:34.359 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.357+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 27 ==== osd_op_reply(56 10000000001.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv7 ondisk = 0) v8 ==== 164+0+4 (crc 0 0 0) 0x55b445a69400 con 0x55b445a6e000 2024-03-24T01:30:34.359 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.358+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:57 7.7 7:e2e07930:::10000000001.00000000:head [omap-set-vals in=542b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6e800 con 0x55b445a6e000 2024-03-24T01:30:34.363 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.361+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 28 ==== osd_op_reply(57 10000000001.00000000 [omap-set-vals] v56'8 uv8 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55b445a69400 con 0x55b445a6e000 2024-03-24T01:30:34.363 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.362+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:58 7.1f 7:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6f000 con 0x55b445a6e000 2024-03-24T01:30:34.364 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.364+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 29 ==== osd_op_reply(58 1.00000000 [omap-get-header out=274b] v0'0 uv11 ondisk = 0) v8 ==== 154+0+274 (crc 0 0 0) 0x55b445a7e280 con 0x55b445a6e000 2024-03-24T01:30:34.364 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.364+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:59 7.1f 7:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b44661d800 con 0x55b445a6e000 2024-03-24T01:30:34.368 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.368+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 30 ==== osd_op_reply(59 1.00000000 [omap-set-header] v56'12 uv12 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55b445a7e280 con 0x55b445a6e000 2024-03-24T01:30:34.368 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.368+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:60 7.1f 7:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=21b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b44661d400 con 0x55b445a6e000 2024-03-24T01:30:34.370 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.368+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 31 ==== osd_op_reply(60 1.00000000 [omap-get-vals-by-keys out=509b] v0'0 uv12 ondisk = 0) v8 ==== 154+0+509 (crc 0 0 0) 0x55b445a7e280 con 0x55b445a6e000 2024-03-24T01:30:34.370 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.368+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:61 7.1f 7:ff5b34d6:::1.00000000:head [omap-set-vals in=509b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b44661dc00 con 0x55b445a6e000 2024-03-24T01:30:34.373 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.373+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 32 ==== osd_op_reply(61 1.00000000 [omap-set-vals] v56'13 uv13 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55b445a7e280 con 0x55b445a6e000 2024-03-24T01:30:34.374 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.373+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:62 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b44661d800 con 0x55b4465a4400 2024-03-24T01:30:34.375 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.373+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 26 ==== osd_op_reply(62 10000000000.00000000 [omap-get-header out=274b] v0'0 uv15 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55b445a7e000 con 0x55b4465a4400 2024-03-24T01:30:34.375 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.373+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:63 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6f400 con 0x55b4465a4400 2024-03-24T01:30:34.378 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.377+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 27 ==== osd_op_reply(63 10000000000.00000000 [omap-set-header] v56'16 uv16 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55b445a7e000 con 0x55b4465a4400 2024-03-24T01:30:34.378 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.377+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:64 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6f800 con 0x55b4465a4400 2024-03-24T01:30:34.379 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.377+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 28 ==== osd_op_reply(64 10000000000.00000000 [omap-get-vals-by-keys out=504b] v0'0 uv16 ondisk = 0) v8 ==== 164+0+504 (crc 0 0 0) 0x55b445a7e000 con 0x55b4465a4400 2024-03-24T01:30:34.379 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.378+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:65 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=504b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6fc00 con 0x55b4465a4400 2024-03-24T01:30:34.382 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.381+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 29 ==== osd_op_reply(65 10000000000.00000000 [omap-set-vals] v56'17 uv17 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55b445a7e000 con 0x55b4465a4400 2024-03-24T01:30:34.382 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.381+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:66 7.13 7:cd5a64a3:::100.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6ec00 con 0x55b4465a4c00 2024-03-24T01:30:34.383 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.382+0000 7fcec8000640 1 -- 172.21.15.90:0/868144061 <== osd.0 v2:172.21.15.90:6800/307045587 5 ==== osd_op_reply(66 100.00000000 [omap-get-header out=274b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+274 (crc 0 0 0) 0x55b446454000 con 0x55b4465a4c00 2024-03-24T01:30:34.383 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.382+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:67 7.13 7:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6e400 con 0x55b4465a4c00 2024-03-24T01:30:34.387 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.385+0000 7fcec8000640 1 -- 172.21.15.90:0/868144061 <== osd.0 v2:172.21.15.90:6800/307045587 6 ==== osd_op_reply(67 100.00000000 [omap-set-header] v56'3 uv3 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55b446454000 con 0x55b4465a4c00 2024-03-24T01:30:34.387 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.385+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:68 7.13 7:cd5a64a3:::100.00000000:head [omap-get-vals-by-keys in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6e800 con 0x55b4465a4c00 2024-03-24T01:30:34.388 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.387+0000 7fcec8000640 1 -- 172.21.15.90:0/868144061 <== osd.0 v2:172.21.15.90:6800/307045587 7 ==== osd_op_reply(68 100.00000000 [omap-get-vals-by-keys out=523b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+523 (crc 0 0 0) 0x55b446454000 con 0x55b4465a4c00 2024-03-24T01:30:34.388 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.387+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:69 7.13 7:cd5a64a3:::100.00000000:head [omap-set-vals in=507b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6f000 con 0x55b4465a4c00 2024-03-24T01:30:34.391 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.391+0000 7fcec8000640 1 -- 172.21.15.90:0/868144061 <== osd.0 v2:172.21.15.90:6800/307045587 8 ==== osd_op_reply(69 100.00000000 [omap-set-vals] v56'4 uv4 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55b446454000 con 0x55b4465a4c00 2024-03-24T01:30:34.392 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.391+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:70 7.10 7:0bd6d154:::602.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a92000 con 0x55b4465a4400 2024-03-24T01:30:34.393 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.391+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 30 ==== osd_op_reply(70 602.00000000 [omap-get-header out=274b] v0'0 uv7 ondisk = 0) v8 ==== 156+0+274 (crc 0 0 0) 0x55b445a7e500 con 0x55b4465a4400 2024-03-24T01:30:34.393 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.391+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:71 7.10 7:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a92400 con 0x55b4465a4400 2024-03-24T01:30:34.396 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.396+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 31 ==== osd_op_reply(71 602.00000000 [omap-set-header] v56'18 uv18 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55b445a7e500 con 0x55b4465a4400 2024-03-24T01:30:34.396 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.396+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:72 7.10 7:0bd6d154:::602.00000000:head [omap-get-vals-by-keys in=24b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a92800 con 0x55b4465a4400 2024-03-24T01:30:34.397 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.396+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 32 ==== osd_op_reply(72 602.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv18 ondisk = 0) v8 ==== 156+0+4 (crc 0 0 0) 0x55b445a7e500 con 0x55b4465a4400 2024-03-24T01:30:34.398 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.396+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:73 7.10 7:0bd6d154:::602.00000000:head [omap-set-vals in=529b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a92c00 con 0x55b4465a4400 2024-03-24T01:30:34.401 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.400+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 33 ==== osd_op_reply(73 602.00000000 [omap-set-vals] v56'19 uv19 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55b445a7e500 con 0x55b4465a4400 2024-03-24T01:30:34.401 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.400+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:74 7.7 7:e2e07930:::10000000001.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a93000 con 0x55b445a6e000 2024-03-24T01:30:34.402 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.402+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 33 ==== osd_op_reply(74 10000000001.00000000 [omap-get-header out=274b] v0'0 uv8 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55b445a7e280 con 0x55b445a6e000 2024-03-24T01:30:34.402 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.402+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:75 7.7 7:e2e07930:::10000000001.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a93400 con 0x55b445a6e000 2024-03-24T01:30:34.407 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.407+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 34 ==== osd_op_reply(75 10000000001.00000000 [omap-set-header] v56'9 uv9 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55b445a7e280 con 0x55b445a6e000 2024-03-24T01:30:34.407 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.407+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:76 7.7 7:e2e07930:::10000000001.00000000:head [omap-get-vals-by-keys in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a93800 con 0x55b445a6e000 2024-03-24T01:30:34.408 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.407+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 35 ==== osd_op_reply(76 10000000001.00000000 [omap-get-vals-by-keys out=504b] v0'0 uv9 ondisk = 0) v8 ==== 164+0+504 (crc 0 0 0) 0x55b445a7e280 con 0x55b445a6e000 2024-03-24T01:30:34.408 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.407+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:77 7.7 7:e2e07930:::10000000001.00000000:head [omap-rm-keys in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a93c00 con 0x55b445a6e000 2024-03-24T01:30:34.412 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.411+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 36 ==== osd_op_reply(77 10000000001.00000000 [omap-rm-keys] v56'10 uv10 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55b445a7e280 con 0x55b445a6e000 2024-03-24T01:30:34.412 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.411+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:78 7.14 7:287034c4:::1.00000000.inode:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a9c000 con 0x55b4465a4400 2024-03-24T01:30:34.413 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.412+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 34 ==== osd_op_reply(78 1.00000000.inode [read 0~582 out=582b] v0'0 uv40 ondisk = 0) v8 ==== 160+0+582 (crc 0 0 0) 0x55b445a7e500 con 0x55b4465a4400 2024-03-24T01:30:34.413 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.412+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:79 7.14 7:287034c4:::1.00000000.inode:head [writefull 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a9c400 con 0x55b4465a4400 2024-03-24T01:30:34.417 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.416+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 35 ==== osd_op_reply(79 1.00000000.inode [writefull 0~582] v56'41 uv41 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x55b445a7e500 con 0x55b4465a4400 2024-03-24T01:30:34.417 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.416+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:80 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a9c800 con 0x55b4465a4400 2024-03-24T01:30:34.418 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.417+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 36 ==== osd_op_reply(80 10000000000.00000000 [omap-get-header out=274b] v0'0 uv17 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55b445a7e500 con 0x55b4465a4400 2024-03-24T01:30:34.418 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.417+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:81 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a9cc00 con 0x55b4465a4400 2024-03-24T01:30:34.422 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.420+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 37 ==== osd_op_reply(81 10000000000.00000000 [omap-set-header] v56'20 uv20 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55b445a7e500 con 0x55b4465a4400 2024-03-24T01:30:34.422 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.421+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:82 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a9d000 con 0x55b4465a4400 2024-03-24T01:30:34.423 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.421+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 38 ==== osd_op_reply(82 10000000000.00000000 [omap-get-vals-by-keys out=504b] v0'0 uv20 ondisk = 0) v8 ==== 164+0+504 (crc 0 0 0) 0x55b445a7e500 con 0x55b4465a4400 2024-03-24T01:30:34.423 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.422+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:83 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=504b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a9d400 con 0x55b4465a4400 2024-03-24T01:30:34.426 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.426+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 39 ==== osd_op_reply(83 10000000000.00000000 [omap-set-vals] v56'21 uv21 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55b445a7e500 con 0x55b4465a4400 2024-03-24T01:30:34.426 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.426+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:84 7.7 7:e2e07930:::10000000001.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a9d800 con 0x55b445a6e000 2024-03-24T01:30:34.428 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.427+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 37 ==== osd_op_reply(84 10000000001.00000000 [omap-get-header out=274b] v0'0 uv10 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55b445a7e280 con 0x55b445a6e000 2024-03-24T01:30:34.428 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.427+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:85 7.7 7:e2e07930:::10000000001.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a9dc00 con 0x55b445a6e000 2024-03-24T01:30:34.432 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.430+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 38 ==== osd_op_reply(85 10000000001.00000000 [omap-set-header] v56'11 uv11 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55b445a7e280 con 0x55b445a6e000 2024-03-24T01:30:34.432 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.431+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:86 7.7 7:e2e07930:::10000000001.00000000:head [omap-get-vals-by-keys in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a9c000 con 0x55b445a6e000 2024-03-24T01:30:34.433 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.433+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 39 ==== osd_op_reply(86 10000000001.00000000 [omap-get-vals-by-keys out=542b] v0'0 uv11 ondisk = 0) v8 ==== 164+0+542 (crc 0 0 0) 0x55b445a7e280 con 0x55b445a6e000 2024-03-24T01:30:34.433 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.433+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:87 7.7 7:e2e07930:::10000000001.00000000:head [omap-set-vals in=504b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a92000 con 0x55b445a6e000 2024-03-24T01:30:34.438 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.436+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 40 ==== osd_op_reply(87 10000000001.00000000 [omap-set-vals] v56'12 uv12 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55b445a7e280 con 0x55b445a6e000 2024-03-24T01:30:34.438 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.436+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:88 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a92400 con 0x55b4465a4400 2024-03-24T01:30:34.439 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.437+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 40 ==== osd_op_reply(88 10000000000.00000000 [omap-get-header out=274b] v0'0 uv21 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55b445a7e780 con 0x55b4465a4400 2024-03-24T01:30:34.439 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.437+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:89 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a92800 con 0x55b4465a4400 2024-03-24T01:30:34.442 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.442+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 41 ==== osd_op_reply(89 10000000000.00000000 [omap-set-header] v56'22 uv22 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55b445a7e780 con 0x55b4465a4400 2024-03-24T01:30:34.442 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.442+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:90 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a92c00 con 0x55b4465a4400 2024-03-24T01:30:34.443 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.442+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 42 ==== osd_op_reply(90 10000000000.00000000 [omap-get-vals-by-keys out=504b] v0'0 uv22 ondisk = 0) v8 ==== 164+0+504 (crc 0 0 0) 0x55b445a7e780 con 0x55b4465a4400 2024-03-24T01:30:34.443 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.442+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:91 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=504b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6f400 con 0x55b4465a4400 2024-03-24T01:30:34.447 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.446+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 43 ==== osd_op_reply(91 10000000000.00000000 [omap-set-vals] v56'23 uv23 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55b445a7e780 con 0x55b4465a4400 2024-03-24T01:30:34.447 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.446+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:92 7.13 7:cd5a64a3:::100.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6f800 con 0x55b4465a4c00 2024-03-24T01:30:34.448 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.448+0000 7fcec8000640 1 -- 172.21.15.90:0/868144061 <== osd.0 v2:172.21.15.90:6800/307045587 9 ==== osd_op_reply(92 100.00000000 [omap-get-header out=274b] v0'0 uv4 ondisk = 0) v8 ==== 156+0+274 (crc 0 0 0) 0x55b446454000 con 0x55b4465a4c00 2024-03-24T01:30:34.448 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.448+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:93 7.13 7:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6fc00 con 0x55b4465a4c00 2024-03-24T01:30:34.452 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.451+0000 7fcec8000640 1 -- 172.21.15.90:0/868144061 <== osd.0 v2:172.21.15.90:6800/307045587 10 ==== osd_op_reply(93 100.00000000 [omap-set-header] v56'5 uv5 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55b446454000 con 0x55b4465a4c00 2024-03-24T01:30:34.452 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.451+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:94 7.13 7:cd5a64a3:::100.00000000:head [omap-get-vals-by-keys in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b44661d800 con 0x55b4465a4c00 2024-03-24T01:30:34.453 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.452+0000 7fcec8000640 1 -- 172.21.15.90:0/868144061 <== osd.0 v2:172.21.15.90:6800/307045587 11 ==== osd_op_reply(94 100.00000000 [omap-get-vals-by-keys out=507b] v0'0 uv5 ondisk = 0) v8 ==== 156+0+507 (crc 0 0 0) 0x55b446454000 con 0x55b4465a4c00 2024-03-24T01:30:34.453 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.452+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:95 7.13 7:cd5a64a3:::100.00000000:head [omap-set-vals in=507b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b4465a5800 con 0x55b4465a4c00 2024-03-24T01:30:34.458 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.457+0000 7fcec8000640 1 -- 172.21.15.90:0/868144061 <== osd.0 v2:172.21.15.90:6800/307045587 12 ==== osd_op_reply(95 100.00000000 [omap-set-vals] v56'6 uv6 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55b446454000 con 0x55b4465a4c00 2024-03-24T01:30:34.458 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.457+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:96 7.10 7:0bd6d154:::602.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a9d800 con 0x55b4465a4400 2024-03-24T01:30:34.459 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.457+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 44 ==== osd_op_reply(96 602.00000000 [omap-get-header out=274b] v0'0 uv19 ondisk = 0) v8 ==== 156+0+274 (crc 0 0 0) 0x55b445a7e780 con 0x55b4465a4400 2024-03-24T01:30:34.459 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.457+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:97 7.10 7:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a9dc00 con 0x55b4465a4400 2024-03-24T01:30:34.462 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.461+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 45 ==== osd_op_reply(97 602.00000000 [omap-set-header] v56'24 uv24 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55b445a7e780 con 0x55b4465a4400 2024-03-24T01:30:34.462 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.462+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:98 7.10 7:0bd6d154:::602.00000000:head [omap-get-vals-by-keys in=24b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a9c000 con 0x55b4465a4400 2024-03-24T01:30:34.463 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.462+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 46 ==== osd_op_reply(98 602.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv24 ondisk = 0) v8 ==== 156+0+4 (crc 0 0 0) 0x55b445a7e780 con 0x55b4465a4400 2024-03-24T01:30:34.463 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.463+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:99 7.10 7:0bd6d154:::602.00000000:head [omap-set-vals in=529b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a93000 con 0x55b4465a4400 2024-03-24T01:30:34.466 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.465+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 47 ==== osd_op_reply(99 602.00000000 [omap-set-vals] v56'25 uv25 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55b445a7e780 con 0x55b4465a4400 2024-03-24T01:30:34.466 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.465+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:100 7.7 7:e2e07930:::10000000001.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a93400 con 0x55b445a6e000 2024-03-24T01:30:34.467 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.466+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 41 ==== osd_op_reply(100 10000000001.00000000 [omap-get-header out=274b] v0'0 uv12 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55b445a7ea00 con 0x55b445a6e000 2024-03-24T01:30:34.467 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.466+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:101 7.7 7:e2e07930:::10000000001.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a93800 con 0x55b445a6e000 2024-03-24T01:30:34.472 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.472+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 42 ==== osd_op_reply(101 10000000001.00000000 [omap-set-header] v56'13 uv13 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55b445a7ea00 con 0x55b445a6e000 2024-03-24T01:30:34.472 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.472+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:102 7.7 7:e2e07930:::10000000001.00000000:head [omap-get-vals-by-keys in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a93c00 con 0x55b445a6e000 2024-03-24T01:30:34.473 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.472+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 43 ==== osd_op_reply(102 10000000001.00000000 [omap-get-vals-by-keys out=504b] v0'0 uv13 ondisk = 0) v8 ==== 164+0+504 (crc 0 0 0) 0x55b445a7ea00 con 0x55b445a6e000 2024-03-24T01:30:34.473 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.472+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:103 7.7 7:e2e07930:::10000000001.00000000:head [omap-rm-keys in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b44661d400 con 0x55b445a6e000 2024-03-24T01:30:34.477 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.477+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 44 ==== osd_op_reply(103 10000000001.00000000 [omap-rm-keys] v56'14 uv14 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55b445a7ea00 con 0x55b445a6e000 2024-03-24T01:30:34.477 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.477+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:104 7.1f 7:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b44661dc00 con 0x55b445a6e000 2024-03-24T01:30:34.478 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.478+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 45 ==== osd_op_reply(104 1.00000000 [omap-get-header out=274b] v0'0 uv13 ondisk = 0) v8 ==== 154+0+274 (crc 0 0 0) 0x55b445a7ea00 con 0x55b445a6e000 2024-03-24T01:30:34.478 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.478+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:105 7.1f 7:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6ec00 con 0x55b445a6e000 2024-03-24T01:30:34.482 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.481+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 46 ==== osd_op_reply(105 1.00000000 [omap-set-header] v56'14 uv14 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55b445a7ea00 con 0x55b445a6e000 2024-03-24T01:30:34.482 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.481+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:106 7.1f 7:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=21b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6e400 con 0x55b445a6e000 2024-03-24T01:30:34.483 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.483+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 47 ==== osd_op_reply(106 1.00000000 [omap-get-vals-by-keys out=509b] v0'0 uv14 ondisk = 0) v8 ==== 154+0+509 (crc 0 0 0) 0x55b445a7ea00 con 0x55b445a6e000 2024-03-24T01:30:34.483 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.483+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:107 7.1f 7:ff5b34d6:::1.00000000:head [omap-set-vals in=509b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6e800 con 0x55b445a6e000 2024-03-24T01:30:34.487 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.487+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 48 ==== osd_op_reply(107 1.00000000 [omap-set-vals] v56'15 uv15 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55b445a7ea00 con 0x55b445a6e000 2024-03-24T01:30:34.487 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.487+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:108 7.13 7:cd5a64a3:::100.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6f000 con 0x55b4465a4c00 2024-03-24T01:30:34.488 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.487+0000 7fcec8000640 1 -- 172.21.15.90:0/868144061 <== osd.0 v2:172.21.15.90:6800/307045587 13 ==== osd_op_reply(108 100.00000000 [omap-get-header out=274b] v0'0 uv6 ondisk = 0) v8 ==== 156+0+274 (crc 0 0 0) 0x55b445a7ef00 con 0x55b4465a4c00 2024-03-24T01:30:34.488 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.487+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:109 7.13 7:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6f800 con 0x55b4465a4c00 2024-03-24T01:30:34.491 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.490+0000 7fcec8000640 1 -- 172.21.15.90:0/868144061 <== osd.0 v2:172.21.15.90:6800/307045587 14 ==== osd_op_reply(109 100.00000000 [omap-set-header] v56'7 uv7 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55b445a7ef00 con 0x55b4465a4c00 2024-03-24T01:30:34.491 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.490+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:110 7.13 7:cd5a64a3:::100.00000000:head [omap-get-vals-by-keys in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a9c400 con 0x55b4465a4c00 2024-03-24T01:30:34.492 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.491+0000 7fcec8000640 1 -- 172.21.15.90:0/868144061 <== osd.0 v2:172.21.15.90:6800/307045587 15 ==== osd_op_reply(110 100.00000000 [omap-get-vals-by-keys out=507b] v0'0 uv7 ondisk = 0) v8 ==== 156+0+507 (crc 0 0 0) 0x55b445a7ef00 con 0x55b4465a4c00 2024-03-24T01:30:34.492 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.491+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:111 7.13 7:cd5a64a3:::100.00000000:head [omap-set-vals in=507b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a9c800 con 0x55b4465a4c00 2024-03-24T01:30:34.499 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.499+0000 7fcec8000640 1 -- 172.21.15.90:0/868144061 <== osd.0 v2:172.21.15.90:6800/307045587 16 ==== osd_op_reply(111 100.00000000 [omap-set-vals] v56'8 uv8 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55b445a7ef00 con 0x55b4465a4c00 2024-03-24T01:30:34.499 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.499+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:112 7.10 7:0bd6d154:::602.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a9cc00 con 0x55b4465a4400 2024-03-24T01:30:34.500 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.500+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 48 ==== osd_op_reply(112 602.00000000 [omap-get-header out=274b] v0'0 uv25 ondisk = 0) v8 ==== 156+0+274 (crc 0 0 0) 0x55b445aaa000 con 0x55b4465a4400 2024-03-24T01:30:34.500 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.500+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:113 7.10 7:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a9d000 con 0x55b4465a4400 2024-03-24T01:30:34.504 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.504+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 49 ==== osd_op_reply(113 602.00000000 [omap-set-header] v56'26 uv26 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55b445aaa000 con 0x55b4465a4400 2024-03-24T01:30:34.504 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.504+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:114 7.10 7:0bd6d154:::602.00000000:head [omap-get-vals-by-keys in=24b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a9d400 con 0x55b4465a4400 2024-03-24T01:30:34.505 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.505+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 50 ==== osd_op_reply(114 602.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv26 ondisk = 0) v8 ==== 156+0+4 (crc 0 0 0) 0x55b445aaa000 con 0x55b4465a4400 2024-03-24T01:30:34.505 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.505+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:115 7.10 7:0bd6d154:::602.00000000:head [omap-set-vals in=525b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a9d800 con 0x55b4465a4400 2024-03-24T01:30:34.509 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.507+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 51 ==== osd_op_reply(115 602.00000000 [omap-set-vals] v56'27 uv27 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55b445aaa000 con 0x55b4465a4400 2024-03-24T01:30:34.509 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.507+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:116 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a9dc00 con 0x55b4465a4400 2024-03-24T01:30:34.510 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.508+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 52 ==== osd_op_reply(116 10000000000.00000000 [omap-get-header out=274b] v0'0 uv23 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55b445aaa000 con 0x55b4465a4400 2024-03-24T01:30:34.510 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.509+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:117 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6f400 con 0x55b4465a4400 2024-03-24T01:30:34.513 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.513+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 53 ==== osd_op_reply(117 10000000000.00000000 [omap-set-header] v56'28 uv28 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55b445aaa000 con 0x55b4465a4400 2024-03-24T01:30:34.514 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.513+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:118 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a92400 con 0x55b4465a4400 2024-03-24T01:30:34.514 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.513+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 54 ==== osd_op_reply(118 10000000000.00000000 [omap-get-vals-by-keys out=504b] v0'0 uv28 ondisk = 0) v8 ==== 164+0+504 (crc 0 0 0) 0x55b445aaa000 con 0x55b4465a4400 2024-03-24T01:30:34.515 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.513+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:119 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-rm-keys in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a92800 con 0x55b4465a4400 2024-03-24T01:30:34.518 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.518+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 55 ==== osd_op_reply(119 10000000000.00000000 [omap-rm-keys] v56'29 uv29 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55b445aaa000 con 0x55b4465a4400 2024-03-24T01:30:34.518 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.518+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:120 7.1f 7:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a92c00 con 0x55b445a6e000 2024-03-24T01:30:34.519 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.519+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 49 ==== osd_op_reply(120 1.00000000 [omap-get-header out=274b] v0'0 uv15 ondisk = 0) v8 ==== 154+0+274 (crc 0 0 0) 0x55b445a7ea00 con 0x55b445a6e000 2024-03-24T01:30:34.520 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.519+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:121 7.1f 7:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a9c400 con 0x55b445a6e000 2024-03-24T01:30:34.523 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.523+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 50 ==== osd_op_reply(121 1.00000000 [omap-set-header] v56'16 uv16 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55b445a7ea00 con 0x55b445a6e000 2024-03-24T01:30:34.523 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.523+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:122 7.1f 7:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=21b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6ec00 con 0x55b445a6e000 2024-03-24T01:30:34.524 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.524+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 51 ==== osd_op_reply(122 1.00000000 [omap-get-vals-by-keys out=509b] v0'0 uv16 ondisk = 0) v8 ==== 154+0+509 (crc 0 0 0) 0x55b445a7ea00 con 0x55b445a6e000 2024-03-24T01:30:34.525 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.524+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:123 7.1f 7:ff5b34d6:::1.00000000:head [omap-rm-keys in=21b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6e400 con 0x55b445a6e000 2024-03-24T01:30:34.528 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.527+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 52 ==== osd_op_reply(123 1.00000000 [omap-rm-keys] v56'17 uv17 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55b445a7ea00 con 0x55b445a6e000 2024-03-24T01:30:34.528 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.527+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:124 7.13 7:cd5a64a3:::100.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6e800 con 0x55b4465a4c00 2024-03-24T01:30:34.529 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.529+0000 7fcec8000640 1 -- 172.21.15.90:0/868144061 <== osd.0 v2:172.21.15.90:6800/307045587 17 ==== osd_op_reply(124 100.00000000 [omap-get-header out=274b] v0'0 uv8 ondisk = 0) v8 ==== 156+0+274 (crc 0 0 0) 0x55b445a7ef00 con 0x55b4465a4c00 2024-03-24T01:30:34.529 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.529+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:125 7.13 7:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6fc00 con 0x55b4465a4c00 2024-03-24T01:30:34.533 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.533+0000 7fcec8000640 1 -- 172.21.15.90:0/868144061 <== osd.0 v2:172.21.15.90:6800/307045587 18 ==== osd_op_reply(125 100.00000000 [omap-set-header] v56'9 uv9 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55b445a7ef00 con 0x55b4465a4c00 2024-03-24T01:30:34.533 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.533+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:126 7.13 7:cd5a64a3:::100.00000000:head [omap-get-vals-by-keys in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6f000 con 0x55b4465a4c00 2024-03-24T01:30:34.534 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.534+0000 7fcec8000640 1 -- 172.21.15.90:0/868144061 <== osd.0 v2:172.21.15.90:6800/307045587 19 ==== osd_op_reply(126 100.00000000 [omap-get-vals-by-keys out=507b] v0'0 uv9 ondisk = 0) v8 ==== 156+0+507 (crc 0 0 0) 0x55b445a7ef00 con 0x55b4465a4c00 2024-03-24T01:30:34.534 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.534+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:127 7.13 7:cd5a64a3:::100.00000000:head [omap-set-vals in=507b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6f800 con 0x55b4465a4c00 2024-03-24T01:30:34.538 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.538+0000 7fcec8000640 1 -- 172.21.15.90:0/868144061 <== osd.0 v2:172.21.15.90:6800/307045587 20 ==== osd_op_reply(127 100.00000000 [omap-set-vals] v56'10 uv10 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55b445a7ef00 con 0x55b4465a4c00 2024-03-24T01:30:34.538 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.538+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:128 7.10 7:0bd6d154:::602.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b4465a5800 con 0x55b4465a4400 2024-03-24T01:30:34.539 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.539+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 56 ==== osd_op_reply(128 602.00000000 [omap-get-header out=274b] v0'0 uv27 ondisk = 0) v8 ==== 156+0+274 (crc 0 0 0) 0x55b445aaa000 con 0x55b4465a4400 2024-03-24T01:30:34.539 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.539+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:129 7.10 7:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b44661d400 con 0x55b4465a4400 2024-03-24T01:30:34.544 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.543+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 57 ==== osd_op_reply(129 602.00000000 [omap-set-header] v56'30 uv30 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55b445aaa000 con 0x55b4465a4400 2024-03-24T01:30:34.544 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.544+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:130 7.10 7:0bd6d154:::602.00000000:head [omap-get-vals-by-keys in=24b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b44661dc00 con 0x55b4465a4400 2024-03-24T01:30:34.545 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.544+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 58 ==== osd_op_reply(130 602.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv30 ondisk = 0) v8 ==== 156+0+4 (crc 0 0 0) 0x55b445aaa000 con 0x55b4465a4400 2024-03-24T01:30:34.545 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.544+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:131 7.10 7:0bd6d154:::602.00000000:head [omap-set-vals in=521b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b44661d800 con 0x55b4465a4400 2024-03-24T01:30:34.550 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.549+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 59 ==== osd_op_reply(131 602.00000000 [omap-set-vals] v56'31 uv31 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55b445aaa000 con 0x55b4465a4400 2024-03-24T01:30:34.550 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.549+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:132 7.14 7:287034c4:::1.00000000.inode:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a92000 con 0x55b4465a4400 2024-03-24T01:30:34.551 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.551+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 60 ==== osd_op_reply(132 1.00000000.inode [read 0~582 out=582b] v0'0 uv41 ondisk = 0) v8 ==== 160+0+582 (crc 0 0 0) 0x55b445a7e280 con 0x55b4465a4400 2024-03-24T01:30:34.551 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.551+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:133 7.14 7:287034c4:::1.00000000.inode:head [writefull 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a93400 con 0x55b4465a4400 2024-03-24T01:30:34.556 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.556+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 61 ==== osd_op_reply(133 1.00000000.inode [writefull 0~582] v56'42 uv42 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x55b445a7e280 con 0x55b4465a4400 2024-03-24T01:30:34.557 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.556+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:134 7.10 7:0bd6d154:::602.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a93800 con 0x55b4465a4400 2024-03-24T01:30:34.557 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.557+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 62 ==== osd_op_reply(134 602.00000000 [omap-get-header out=274b] v0'0 uv31 ondisk = 0) v8 ==== 156+0+274 (crc 0 0 0) 0x55b445a7e280 con 0x55b4465a4400 2024-03-24T01:30:34.558 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.557+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:135 7.10 7:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a93c00 con 0x55b4465a4400 2024-03-24T01:30:34.562 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.562+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 63 ==== osd_op_reply(135 602.00000000 [omap-set-header] v56'32 uv32 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55b445a7e280 con 0x55b4465a4400 2024-03-24T01:30:34.562 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.562+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:136 7.10 7:0bd6d154:::602.00000000:head [omap-get-vals-by-keys in=24b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a92c00 con 0x55b4465a4400 2024-03-24T01:30:34.564 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.563+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 64 ==== osd_op_reply(136 602.00000000 [omap-get-vals-by-keys out=529b] v0'0 uv32 ondisk = 0) v8 ==== 156+0+529 (crc 0 0 0) 0x55b445a7e280 con 0x55b4465a4400 2024-03-24T01:30:34.564 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.563+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:137 7.10 7:0bd6d154:::602.00000000:head [omap-rm-keys in=24b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b44661d400 con 0x55b4465a4400 2024-03-24T01:30:34.567 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.567+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 65 ==== osd_op_reply(137 602.00000000 [omap-rm-keys] v56'33 uv33 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55b445a7e280 con 0x55b4465a4400 2024-03-24T01:30:34.567 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.567+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:138 7.10 7:0bd6d154:::602.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b4465a5800 con 0x55b4465a4400 2024-03-24T01:30:34.568 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.568+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 66 ==== osd_op_reply(138 602.00000000 [omap-get-header out=274b] v0'0 uv33 ondisk = 0) v8 ==== 156+0+274 (crc 0 0 0) 0x55b445a7e280 con 0x55b4465a4400 2024-03-24T01:30:34.568 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.568+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:139 7.10 7:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a93000 con 0x55b4465a4400 2024-03-24T01:30:34.571 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.571+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 67 ==== osd_op_reply(139 602.00000000 [omap-set-header] v56'34 uv34 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55b445a7e280 con 0x55b4465a4400 2024-03-24T01:30:34.571 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.571+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:140 7.10 7:0bd6d154:::602.00000000:head [omap-get-vals-by-keys in=24b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a92400 con 0x55b4465a4400 2024-03-24T01:30:34.572 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.572+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 68 ==== osd_op_reply(140 602.00000000 [omap-get-vals-by-keys out=529b] v0'0 uv34 ondisk = 0) v8 ==== 156+0+529 (crc 0 0 0) 0x55b445a7e280 con 0x55b4465a4400 2024-03-24T01:30:34.572 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.572+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:141 7.10 7:0bd6d154:::602.00000000:head [omap-rm-keys in=24b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a92800 con 0x55b4465a4400 2024-03-24T01:30:34.576 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.576+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 69 ==== osd_op_reply(141 602.00000000 [omap-rm-keys] v56'35 uv35 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55b445a7e280 con 0x55b4465a4400 2024-03-24T01:30:34.576 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.576+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:142 7.13 7:cd5a64a3:::100.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6f400 con 0x55b4465a4c00 2024-03-24T01:30:34.577 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.577+0000 7fcec8000640 1 -- 172.21.15.90:0/868144061 <== osd.0 v2:172.21.15.90:6800/307045587 21 ==== osd_op_reply(142 100.00000000 [omap-get-header out=274b] v0'0 uv10 ondisk = 0) v8 ==== 156+0+274 (crc 0 0 0) 0x55b445a7ef00 con 0x55b4465a4c00 2024-03-24T01:30:34.577 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.577+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:143 7.13 7:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a9c000 con 0x55b4465a4c00 2024-03-24T01:30:34.581 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.580+0000 7fcec8000640 1 -- 172.21.15.90:0/868144061 <== osd.0 v2:172.21.15.90:6800/307045587 22 ==== osd_op_reply(143 100.00000000 [omap-set-header] v56'11 uv11 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55b445a7ef00 con 0x55b4465a4c00 2024-03-24T01:30:34.581 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.580+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:144 7.13 7:cd5a64a3:::100.00000000:head [omap-get-vals-by-keys in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a9cc00 con 0x55b4465a4c00 2024-03-24T01:30:34.582 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.582+0000 7fcec8000640 1 -- 172.21.15.90:0/868144061 <== osd.0 v2:172.21.15.90:6800/307045587 23 ==== osd_op_reply(144 100.00000000 [omap-get-vals-by-keys out=507b] v0'0 uv11 ondisk = 0) v8 ==== 156+0+507 (crc 0 0 0) 0x55b445a7ef00 con 0x55b4465a4c00 2024-03-24T01:30:34.582 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.582+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:145 7.13 7:cd5a64a3:::100.00000000:head [omap-set-vals in=507b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a9d000 con 0x55b4465a4c00 2024-03-24T01:30:34.586 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.585+0000 7fcec8000640 1 -- 172.21.15.90:0/868144061 <== osd.0 v2:172.21.15.90:6800/307045587 24 ==== osd_op_reply(145 100.00000000 [omap-set-vals] v56'12 uv12 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55b445a7ef00 con 0x55b4465a4c00 2024-03-24T01:30:34.586 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.585+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:146 7.10 7:0bd6d154:::602.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a9d400 con 0x55b4465a4400 2024-03-24T01:30:34.587 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.587+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 70 ==== osd_op_reply(146 602.00000000 [omap-get-header out=274b] v0'0 uv35 ondisk = 0) v8 ==== 156+0+274 (crc 0 0 0) 0x55b445a7e280 con 0x55b4465a4400 2024-03-24T01:30:34.587 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.587+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:147 7.10 7:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a9d800 con 0x55b4465a4400 2024-03-24T01:30:34.591 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.590+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 71 ==== osd_op_reply(147 602.00000000 [omap-set-header] v56'36 uv36 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55b445a7e280 con 0x55b4465a4400 2024-03-24T01:30:34.591 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.590+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:148 7.10 7:0bd6d154:::602.00000000:head [omap-get-vals-by-keys in=24b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a9dc00 con 0x55b4465a4400 2024-03-24T01:30:34.592 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.591+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 72 ==== osd_op_reply(148 602.00000000 [omap-get-vals-by-keys out=521b] v0'0 uv36 ondisk = 0) v8 ==== 156+0+521 (crc 0 0 0) 0x55b445a7e280 con 0x55b4465a4400 2024-03-24T01:30:34.592 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.592+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:149 7.10 7:0bd6d154:::602.00000000:head [omap-set-vals in=521b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a9d400 con 0x55b4465a4400 2024-03-24T01:30:34.595 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.595+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 73 ==== osd_op_reply(149 602.00000000 [omap-set-vals] v56'37 uv37 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55b445a7e280 con 0x55b4465a4400 2024-03-24T01:30:34.595 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.595+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:150 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a9c800 con 0x55b4465a4400 2024-03-24T01:30:34.596 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.596+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 74 ==== osd_op_reply(150 10000000000.00000000 [omap-get-header out=274b] v0'0 uv29 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55b445a7e280 con 0x55b4465a4400 2024-03-24T01:30:34.596 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.596+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:151 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a9c000 con 0x55b4465a4400 2024-03-24T01:30:34.600 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.600+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 75 ==== osd_op_reply(151 10000000000.00000000 [omap-set-header] v56'38 uv38 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55b445a7e280 con 0x55b4465a4400 2024-03-24T01:30:34.600 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.600+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:152 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6e800 con 0x55b4465a4400 2024-03-24T01:30:34.602 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.601+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 76 ==== osd_op_reply(152 10000000000.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv38 ondisk = 0) v8 ==== 164+0+4 (crc 0 0 0) 0x55b445a7e280 con 0x55b4465a4400 2024-03-24T01:30:34.602 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.601+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:153 7.1f 7:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6fc00 con 0x55b445a6e000 2024-03-24T01:30:34.603 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.602+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 53 ==== osd_op_reply(153 1.00000000 [omap-get-header out=274b] v0'0 uv17 ondisk = 0) v8 ==== 154+0+274 (crc 0 0 0) 0x55b445a7ea00 con 0x55b445a6e000 2024-03-24T01:30:34.603 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.603+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:154 7.1f 7:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6f000 con 0x55b445a6e000 2024-03-24T01:30:34.607 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.607+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 54 ==== osd_op_reply(154 1.00000000 [omap-set-header] v56'18 uv18 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x55b445a7ea00 con 0x55b445a6e000 2024-03-24T01:30:34.607 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.607+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:155 7.1f 7:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6f800 con 0x55b445a6e000 2024-03-24T01:30:34.608 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.608+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 55 ==== osd_op_reply(155 1.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv18 ondisk = 0) v8 ==== 154+0+4 (crc 0 0 0) 0x55b445a7ea00 con 0x55b445a6e000 2024-03-24T01:30:34.608 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.608+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:156 7.14 7:287034c4:::1.00000000.inode:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a6f400 con 0x55b4465a4400 2024-03-24T01:30:34.609 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.609+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 77 ==== osd_op_reply(156 1.00000000.inode [read 0~582 out=582b] v0'0 uv42 ondisk = 0) v8 ==== 160+0+582 (crc 0 0 0) 0x55b445a7e280 con 0x55b4465a4400 2024-03-24T01:30:34.609 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.609+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:157 7.14 7:287034c4:::1.00000000.inode:head [writefull 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b4465a5800 con 0x55b4465a4400 2024-03-24T01:30:34.613 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.613+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 78 ==== osd_op_reply(157 1.00000000.inode [writefull 0~582] v56'43 uv43 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x55b445a7e280 con 0x55b4465a4400 2024-03-24T01:30:34.613 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.613+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:158 7.10 7:0bd6d154:::602.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b44661dc00 con 0x55b4465a4400 2024-03-24T01:30:34.614 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.614+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 79 ==== osd_op_reply(158 602.00000000 [omap-get-header out=274b] v0'0 uv37 ondisk = 0) v8 ==== 156+0+274 (crc 0 0 0) 0x55b445a7e280 con 0x55b4465a4400 2024-03-24T01:30:34.614 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.614+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:159 7.10 7:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b44661d800 con 0x55b4465a4400 2024-03-24T01:30:34.617 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.617+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 80 ==== osd_op_reply(159 602.00000000 [omap-set-header] v56'39 uv39 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55b445a7e280 con 0x55b4465a4400 2024-03-24T01:30:34.617 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.617+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:160 7.10 7:0bd6d154:::602.00000000:head [omap-get-vals-by-keys in=24b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b44661d400 con 0x55b4465a4400 2024-03-24T01:30:34.618 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.618+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 81 ==== osd_op_reply(160 602.00000000 [omap-get-vals-by-keys out=525b] v0'0 uv39 ondisk = 0) v8 ==== 156+0+525 (crc 0 0 0) 0x55b445a7e280 con 0x55b4465a4400 2024-03-24T01:30:34.618 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.618+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:161 7.10 7:0bd6d154:::602.00000000:head [omap-rm-keys in=24b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a92000 con 0x55b4465a4400 2024-03-24T01:30:34.622 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.622+0000 7fcec9400640 1 -- 172.21.15.90:0/868144061 <== osd.3 v2:172.21.15.90:6808/2999931983 82 ==== osd_op_reply(161 602.00000000 [omap-rm-keys] v56'40 uv40 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55b445a7e280 con 0x55b4465a4400 2024-03-24T01:30:34.622 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.622+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:162 7.13 7:cd5a64a3:::100.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a92400 con 0x55b4465a4c00 2024-03-24T01:30:34.623 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.623+0000 7fcec8000640 1 -- 172.21.15.90:0/868144061 <== osd.0 v2:172.21.15.90:6800/307045587 25 ==== osd_op_reply(162 100.00000000 [omap-get-header out=274b] v0'0 uv12 ondisk = 0) v8 ==== 156+0+274 (crc 0 0 0) 0x55b445a7e780 con 0x55b4465a4c00 2024-03-24T01:30:34.623 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.623+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:163 7.13 7:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a92800 con 0x55b4465a4c00 2024-03-24T01:30:34.627 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.626+0000 7fcec8000640 1 -- 172.21.15.90:0/868144061 <== osd.0 v2:172.21.15.90:6800/307045587 26 ==== osd_op_reply(163 100.00000000 [omap-set-header] v56'13 uv13 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55b445a7e780 con 0x55b4465a4c00 2024-03-24T01:30:34.627 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.627+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:164 7.13 7:cd5a64a3:::100.00000000:head [omap-get-vals-by-keys in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a92c00 con 0x55b4465a4c00 2024-03-24T01:30:34.628 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.627+0000 7fcec8000640 1 -- 172.21.15.90:0/868144061 <== osd.0 v2:172.21.15.90:6800/307045587 27 ==== osd_op_reply(164 100.00000000 [omap-get-vals-by-keys out=507b] v0'0 uv13 ondisk = 0) v8 ==== 156+0+507 (crc 0 0 0) 0x55b445a7e780 con 0x55b4465a4c00 2024-03-24T01:30:34.628 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.628+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:165 7.13 7:cd5a64a3:::100.00000000:head [omap-set-vals in=507b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a93000 con 0x55b4465a4c00 2024-03-24T01:30:34.631 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.631+0000 7fcec8000640 1 -- 172.21.15.90:0/868144061 <== osd.0 v2:172.21.15.90:6800/307045587 28 ==== osd_op_reply(165 100.00000000 [omap-set-vals] v56'14 uv14 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x55b445a7e780 con 0x55b4465a4c00 2024-03-24T01:30:34.631 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.631+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:166 7.1f 7:fe07bba1:::100.00000000.inode:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a93400 con 0x55b445a6e000 2024-03-24T01:30:34.633 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.633+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 56 ==== osd_op_reply(166 100.00000000.inode [read 0~582 out=582b] v0'0 uv3 ondisk = 0) v8 ==== 162+0+582 (crc 0 0 0) 0x55b445a7ea00 con 0x55b445a6e000 2024-03-24T01:30:34.633 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.633+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:167 7.1f 7:fe07bba1:::100.00000000.inode:head [writefull 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a93800 con 0x55b445a6e000 2024-03-24T01:30:34.638 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.637+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 <== osd.6 v2:172.21.15.120:6816/1454745630 57 ==== osd_op_reply(167 100.00000000.inode [writefull 0~582] v56'19 uv19 ondisk = 0) v8 ==== 162+0+0 (crc 0 0 0) 0x55b445a7ea00 con 0x55b445a6e000 2024-03-24T01:30:34.638 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.637+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:168 7.13 7:c91d4a1d:::mds0_inotable:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445a93c00 con 0x55b4465a4c00 2024-03-24T01:30:34.639 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.638+0000 7fcec8000640 1 -- 172.21.15.90:0/868144061 <== osd.0 v2:172.21.15.90:6800/307045587 29 ==== osd_op_reply(168 mds0_inotable [read 0~34 out=34b] v0'0 uv2 ondisk = 0) v8 ==== 157+0+34 (crc 0 0 0) 0x55b445a7e780 con 0x55b4465a4c00 2024-03-24T01:30:34.639 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.639+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:169 7.13 7:c91d4a1d:::mds0_inotable:head [writefull 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55b445aca000 con 0x55b4465a4c00 2024-03-24T01:30:34.643 INFO:tasks.workunit.client.0.smithi090.stdout:Events by type: 2024-03-24T01:30:34.643 INFO:tasks.workunit.client.0.smithi090.stdout: LID: 1 2024-03-24T01:30:34.643 INFO:tasks.workunit.client.0.smithi090.stdout: PURGED: 1 2024-03-24T01:30:34.643 INFO:tasks.workunit.client.0.smithi090.stdout: SESSION: 2 2024-03-24T01:30:34.643 INFO:tasks.workunit.client.0.smithi090.stdout: SUBTREEMAP: 1 2024-03-24T01:30:34.644 INFO:tasks.workunit.client.0.smithi090.stdout: UPDATE: 17 2024-03-24T01:30:34.644 INFO:tasks.workunit.client.0.smithi090.stdout:Errors: 0 2024-03-24T01:30:34.645 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.643+0000 7fcec8000640 1 -- 172.21.15.90:0/868144061 <== osd.0 v2:172.21.15.90:6800/307045587 30 ==== osd_op_reply(169 mds0_inotable [writefull 0~34] v56'15 uv15 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x55b445a7e780 con 0x55b4465a4c00 2024-03-24T01:30:34.645 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.643+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55b4465a4c00 msgr2=0x55b44573c680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.645 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.643+0000 7fcecb046280 1 --2- 172.21.15.90:0/868144061 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x55b4465a4c00 0x55b44573c680 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.645 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.643+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55b4465a4400 msgr2=0x55b44573c100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.645 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.643+0000 7fcecb046280 1 --2- 172.21.15.90:0/868144061 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x55b4465a4400 0x55b44573c100 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.645 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.643+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x55b445a6e000 msgr2=0x55b44573b080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.645 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.643+0000 7fcecb046280 1 --2- 172.21.15.90:0/868144061 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x55b445a6e000 0x55b44573b080 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.645 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.643+0000 7fcec8a00640 1 -- 172.21.15.90:0/868144061 reap_dead start 2024-03-24T01:30:34.645 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.644+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55b44661d000 msgr2=0x55b445a48000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.645 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.644+0000 7fcecb046280 1 --2- 172.21.15.90:0/868144061 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55b44661d000 0x55b445a48000 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x55b4465a2780 tx=0x55b4465a6cc0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.645 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.644+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b4465a5400 msgr2=0x55b44573d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.646 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.644+0000 7fcecb046280 1 --2- 172.21.15.90:0/868144061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b4465a5400 0x55b44573d180 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x55b4465a32f0 tx=0x55b4465a6f60 comp rx=0 tx=0).stop 2024-03-24T01:30:34.646 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.644+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 shutdown_connections 2024-03-24T01:30:34.646 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.644+0000 7fcecb046280 1 --2- 172.21.15.90:0/868144061 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55b44661d000 0x55b445a48000 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.646 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.644+0000 7fcecb046280 1 --2- 172.21.15.90:0/868144061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b4465a5400 0x55b44573d180 secure :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0x55b4465a32f0 tx=0x55b4465a6f60 comp rx=0 tx=0).stop 2024-03-24T01:30:34.646 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.644+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 >> 172.21.15.90:0/868144061 conn(0x55b44649c000 msgr2=0x55b446451800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:34.646 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.644+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 shutdown_connections 2024-03-24T01:30:34.646 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.644+0000 7fcecb046280 1 -- 172.21.15.90:0/868144061 wait complete. 2024-03-24T01:30:34.647 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.645+0000 7fcecb046280 1 -- 172.21.15.90:0/4022602763 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b44649dc00 msgr2=0x55b44573b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.647 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.645+0000 7fcecb046280 1 --2- 172.21.15.90:0/4022602763 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b44649dc00 0x55b44573b600 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x55b4457b2810 tx=0x55b4457b8de0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.647 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.645+0000 7fcecb046280 1 -- 172.21.15.90:0/4022602763 shutdown_connections 2024-03-24T01:30:34.647 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.645+0000 7fcecb046280 1 --2- 172.21.15.90:0/4022602763 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b44649dc00 0x55b44573b600 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.647 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.645+0000 7fcecb046280 1 --2- 172.21.15.90:0/4022602763 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55b4456d1c00 0x55b44573ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.647 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.645+0000 7fcecb046280 1 --2- 172.21.15.90:0/4022602763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b44649c800 0x55b44573bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.647 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.645+0000 7fcecb046280 1 -- 172.21.15.90:0/4022602763 >> 172.21.15.90:0/4022602763 conn(0x55b4456d1000 msgr2=0x55b446450000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:34.647 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.645+0000 7fcecb046280 1 -- 172.21.15.90:0/4022602763 shutdown_connections 2024-03-24T01:30:34.648 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.645+0000 7fcecb046280 1 -- 172.21.15.90:0/4022602763 wait complete. 2024-03-24T01:30:34.658 INFO:tasks.workunit.client.0.smithi090.stderr:+ cephfs-journal-tool --rank=cephfs:0 event splice summary 2024-03-24T01:30:34.709 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.709+0000 7f216220c280 1 -- start start 2024-03-24T01:30:34.710 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.709+0000 7f216220c280 1 Processor -- start 2024-03-24T01:30:34.711 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.710+0000 7f216220c280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650c41cec00 0x5650c4238580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.711 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.710+0000 7f216220c280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c41cf800 0x5650c4238b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.711 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.710+0000 7f216220c280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5650c41cfc00 0x5650c4239080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.711 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.710+0000 7f216220c280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650c4fb0600 con 0x5650c41cec00 2024-03-24T01:30:34.711 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.710+0000 7f216220c280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650c4fb0480 con 0x5650c41cf800 2024-03-24T01:30:34.711 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.710+0000 7f216220c280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5650c4fb0300 con 0x5650c41cfc00 2024-03-24T01:30:34.711 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.710+0000 7f215f600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650c41cec00 0x5650c4238580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.711 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.710+0000 7f215f600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650c41cec00 0x5650c4238580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36896/0 (socket says 172.21.15.90:36896) 2024-03-24T01:30:34.711 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.710+0000 7f215f600640 1 -- 172.21.15.90:0/4230799635 learned_addr learned my addr 172.21.15.90:0/4230799635 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:34.713 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.710+0000 7f2160000640 1 --2- 172.21.15.90:0/4230799635 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5650c41cfc00 0x5650c4239080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.713 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.710+0000 7f215ec00640 1 --2- 172.21.15.90:0/4230799635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c41cf800 0x5650c4238b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.713 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.710+0000 7f215f600640 1 -- 172.21.15.90:0/4230799635 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5650c41cfc00 msgr2=0x5650c4239080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:34.713 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.710+0000 7f215f600640 1 --2- 172.21.15.90:0/4230799635 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5650c41cfc00 0x5650c4239080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.713 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.710+0000 7f215f600640 1 -- 172.21.15.90:0/4230799635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c41cf800 msgr2=0x5650c4238b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:34.713 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.710+0000 7f215f600640 1 --2- 172.21.15.90:0/4230799635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c41cf800 0x5650c4238b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.713 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.710+0000 7f215f600640 1 -- 172.21.15.90:0/4230799635 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5650c4202780 con 0x5650c41cec00 2024-03-24T01:30:34.713 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.710+0000 7f215ec00640 1 --2- 172.21.15.90:0/4230799635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c41cf800 0x5650c4238b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:30:34.713 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.712+0000 7f215f600640 1 --2- 172.21.15.90:0/4230799635 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650c41cec00 0x5650c4238580 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x5650c4fed4d0 tx=0x5650c4fa7140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62fb067190dd9f15 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.713 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.712+0000 7f215c400640 1 -- 172.21.15.90:0/4230799635 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5650c41f4680 con 0x5650c41cec00 2024-03-24T01:30:34.714 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.712+0000 7f215c400640 1 -- 172.21.15.90:0/4230799635 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5650c41faa80 con 0x5650c41cec00 2024-03-24T01:30:34.714 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.712+0000 7f215c400640 1 -- 172.21.15.90:0/4230799635 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5650c41f4b60 con 0x5650c41cec00 2024-03-24T01:30:34.714 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.712+0000 7f216220c280 1 -- 172.21.15.90:0/4230799635 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5650c42023c0 con 0x5650c41cec00 2024-03-24T01:30:34.714 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.712+0000 7f216220c280 1 -- 172.21.15.90:0/4230799635 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x5650c4203680 con 0x5650c41cec00 2024-03-24T01:30:34.714 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.712+0000 7f215c400640 1 -- 172.21.15.90:0/4230799635 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x5650c42b7000 con 0x5650c41cec00 2024-03-24T01:30:34.714 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.712+0000 7f215c400640 1 -- 172.21.15.90:0/4230799635 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 36) v1 ==== 3310+0+0 (secure 0 0 0) 0x5650c4229180 con 0x5650c41cec00 2024-03-24T01:30:34.716 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.713+0000 7f216220c280 1 Processor -- start 2024-03-24T01:30:34.716 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.713+0000 7f216220c280 1 -- start start 2024-03-24T01:30:34.717 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.714+0000 7f216220c280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650c50a9400 0x5650c4239600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.717 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.714+0000 7f216220c280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5650c50a9800 0x5650c4239b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.717 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.714+0000 7f216220c280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c50a9c00 0x5650c423a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.717 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.714+0000 7f216220c280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650c4fb0180 con 0x5650c50a9400 2024-03-24T01:30:34.717 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.714+0000 7f216220c280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650c4230900 con 0x5650c50a9c00 2024-03-24T01:30:34.717 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.714+0000 7f216220c280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5650c4230480 con 0x5650c50a9800 2024-03-24T01:30:34.717 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.714+0000 7f2160000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650c50a9400 0x5650c4239600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.717 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.715+0000 7f2160000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650c50a9400 0x5650c4239600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36898/0 (socket says 172.21.15.90:36898) 2024-03-24T01:30:34.717 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.715+0000 7f2160000640 1 -- 172.21.15.90:0/123623606 learned_addr learned my addr 172.21.15.90:0/123623606 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:34.717 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.715+0000 7f2160000640 1 -- 172.21.15.90:0/123623606 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5650c50a9800 msgr2=0x5650c4239b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:34.717 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.715+0000 7f2160000640 1 --2- 172.21.15.90:0/123623606 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5650c50a9800 0x5650c4239b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.717 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.715+0000 7f2160000640 1 -- 172.21.15.90:0/123623606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c50a9c00 msgr2=0x5650c423a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:34.717 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.715+0000 7f2160000640 1 --2- 172.21.15.90:0/123623606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c50a9c00 0x5650c423a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.717 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.715+0000 7f2160000640 1 -- 172.21.15.90:0/123623606 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5650c42034a0 con 0x5650c50a9400 2024-03-24T01:30:34.717 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.715+0000 7f2160000640 1 --2- 172.21.15.90:0/123623606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650c50a9400 0x5650c4239600 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x5650c50ba3c0 tx=0x5650c4fa6f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9987c250d9d8cd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.718 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.715+0000 7f2159c00640 1 -- 172.21.15.90:0/123623606 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5650c41f49c0 con 0x5650c50a9400 2024-03-24T01:30:34.718 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.715+0000 7f2159c00640 1 -- 172.21.15.90:0/123623606 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5650c41fae00 con 0x5650c50a9400 2024-03-24T01:30:34.718 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.715+0000 7f2159c00640 1 -- 172.21.15.90:0/123623606 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5650c41f4680 con 0x5650c50a9400 2024-03-24T01:30:34.718 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.716+0000 7f216220c280 1 -- 172.21.15.90:0/123623606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650c50a9400 msgr2=0x5650c4239600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.718 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.716+0000 7f216220c280 1 --2- 172.21.15.90:0/123623606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650c50a9400 0x5650c4239600 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x5650c50ba3c0 tx=0x5650c4fa6f00 comp rx=0 tx=0).stop 2024-03-24T01:30:34.718 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.716+0000 7f216220c280 1 -- 172.21.15.90:0/123623606 shutdown_connections 2024-03-24T01:30:34.718 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.716+0000 7f216220c280 1 --2- 172.21.15.90:0/123623606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c50a9c00 0x5650c423a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.718 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.716+0000 7f216220c280 1 --2- 172.21.15.90:0/123623606 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5650c50a9800 0x5650c4239b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.718 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.716+0000 7f216220c280 1 --2- 172.21.15.90:0/123623606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650c50a9400 0x5650c4239600 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.718 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.716+0000 7f216220c280 1 -- 172.21.15.90:0/123623606 >> 172.21.15.90:0/123623606 conn(0x5650c50a9000 msgr2=0x5650c42d7800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:34.718 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.716+0000 7f216220c280 1 -- 172.21.15.90:0/123623606 shutdown_connections 2024-03-24T01:30:34.718 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.716+0000 7f216220c280 1 -- 172.21.15.90:0/123623606 wait complete. 2024-03-24T01:30:34.720 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.717+0000 7f216220c280 1 Processor -- start 2024-03-24T01:30:34.720 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.717+0000 7f216220c280 1 -- start start 2024-03-24T01:30:34.721 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.718+0000 7f216220c280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c50a9400 0x5650c423b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.721 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.718+0000 7f216220c280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5650c50a9800 0x5650c423ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.721 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.718+0000 7f216220c280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650c50a9c00 0x5650c423a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.721 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.718+0000 7f216220c280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650c4fb1980 con 0x5650c50a9c00 2024-03-24T01:30:34.721 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.718+0000 7f216220c280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650c4fb0180 con 0x5650c50a9400 2024-03-24T01:30:34.721 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.718+0000 7f216220c280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5650c4fb1800 con 0x5650c50a9800 2024-03-24T01:30:34.721 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.718+0000 7f215ec00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650c50a9c00 0x5650c423a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.721 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.718+0000 7f215ec00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650c50a9c00 0x5650c423a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36900/0 (socket says 172.21.15.90:36900) 2024-03-24T01:30:34.721 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.718+0000 7f215ec00640 1 -- 172.21.15.90:0/137583561 learned_addr learned my addr 172.21.15.90:0/137583561 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:34.721 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.718+0000 7f215f600640 1 --2- 172.21.15.90:0/137583561 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5650c50a9800 0x5650c423ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.721 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.718+0000 7f215ec00640 1 -- 172.21.15.90:0/137583561 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5650c50a9800 msgr2=0x5650c423ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.721 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.718+0000 7f215ec00640 1 --2- 172.21.15.90:0/137583561 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5650c50a9800 0x5650c423ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.721 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.718+0000 7f2160000640 1 --2- 172.21.15.90:0/137583561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c50a9400 0x5650c423b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.721 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.718+0000 7f215ec00640 1 -- 172.21.15.90:0/137583561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c50a9400 msgr2=0x5650c423b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.722 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.718+0000 7f215ec00640 1 --2- 172.21.15.90:0/137583561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c50a9400 0x5650c423b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.722 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.719+0000 7f215ec00640 1 -- 172.21.15.90:0/137583561 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5650c4202780 con 0x5650c50a9c00 2024-03-24T01:30:34.722 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.719+0000 7f215f600640 1 --2- 172.21.15.90:0/137583561 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5650c50a9800 0x5650c423ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:30:34.722 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.719+0000 7f215ec00640 1 --2- 172.21.15.90:0/137583561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650c50a9c00 0x5650c423a680 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x5650c50bb5f0 tx=0x5650c50c38c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.722 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.719+0000 7f2157e00640 1 -- 172.21.15.90:0/137583561 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5650c41f49c0 con 0x5650c50a9c00 2024-03-24T01:30:34.722 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.719+0000 7f2157e00640 1 -- 172.21.15.90:0/137583561 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5650c41fac40 con 0x5650c50a9c00 2024-03-24T01:30:34.722 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.719+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5650c42023c0 con 0x5650c50a9c00 2024-03-24T01:30:34.722 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.719+0000 7f2157e00640 1 -- 172.21.15.90:0/137583561 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5650c41f4680 con 0x5650c50a9c00 2024-03-24T01:30:34.722 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.719+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5650c4203680 con 0x5650c50a9c00 2024-03-24T01:30:34.726 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.721+0000 7f2157e00640 1 -- 172.21.15.90:0/137583561 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x5650c50cec00 con 0x5650c50a9c00 2024-03-24T01:30:34.726 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.721+0000 7f2157e00640 1 --2- 172.21.15.90:0/137583561 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x5650c4fcb000 0x5650c423b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.727 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.721+0000 7f2157e00640 1 -- 172.21.15.90:0/137583561 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x5650c5114600 con 0x5650c50a9c00 2024-03-24T01:30:34.727 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.722+0000 7f216220c280 1 --2- 172.21.15.90:0/137583561 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x5650c4fcb400 0x5650c423a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.727 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.722+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:1 7.f 7:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x5650c4fcb800 con 0x5650c4fcb400 2024-03-24T01:30:34.727 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.722+0000 7f215f600640 1 --2- 172.21.15.90:0/137583561 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x5650c4fcb400 0x5650c423a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.727 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.722+0000 7f2160000640 1 --2- 172.21.15.90:0/137583561 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x5650c4fcb000 0x5650c423b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.728 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.723+0000 7f215f600640 1 --2- 172.21.15.90:0/137583561 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x5650c4fcb400 0x5650c423a100 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.728 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.723+0000 7f215f600640 1 -- 172.21.15.90:0/137583561 <== osd.0 v2:172.21.15.90:6800/307045587 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x5650c4f52000 con 0x5650c4fcb400 2024-03-24T01:30:34.728 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.723+0000 7f216220c280 1 --2- 172.21.15.90:0/137583561 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x5650c4fcbc00 0x5650c4239b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.728 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.723+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:2 7.14 7:292cf221:::200.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x5650c454c000 con 0x5650c4fcbc00 2024-03-24T01:30:34.728 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.723+0000 7f2160000640 1 --2- 172.21.15.90:0/137583561 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x5650c4fcb000 0x5650c423b700 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x5650c50ba3c0 tx=0x5650c4fa6e40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.728 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.723+0000 7f2160000640 1 --2- 172.21.15.90:0/137583561 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x5650c4fcbc00 0x5650c4239b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.728 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.724+0000 7f2160000640 1 --2- 172.21.15.90:0/137583561 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x5650c4fcbc00 0x5650c4239b80 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.728 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.724+0000 7f2160000640 1 -- 172.21.15.90:0/137583561 <== osd.3 v2:172.21.15.90:6808/2999931983 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv34 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x5650c4f52500 con 0x5650c4fcbc00 2024-03-24T01:30:34.729 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.724+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:3 7.14 7:2e2fa760:::200.00000001:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x5650c454c400 con 0x5650c4fcbc00 2024-03-24T01:30:34.729 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.725+0000 7f2160000640 1 -- 172.21.15.90:0/137583561 <== osd.3 v2:172.21.15.90:6808/2999931983 2 ==== osd_op_reply(3 200.00000001 [read 0~35827 out=35827b] v0'0 uv36 ondisk = 0) v8 ==== 156+0+35827 (crc 0 0 0) 0x5650c4f52500 con 0x5650c4fcbc00 2024-03-24T01:30:34.729 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.725+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:4 7.1b 7:ddb4e4d7:::200.00000002:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x5650c454c800 con 0x5650c4fcb400 2024-03-24T01:30:34.729 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.725+0000 7f215f600640 1 -- 172.21.15.90:0/137583561 <== osd.0 v2:172.21.15.90:6800/307045587 2 ==== osd_op_reply(4 200.00000002 [read 0~2147483647] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x5650c4f52000 con 0x5650c4fcb400 2024-03-24T01:30:34.729 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.725+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:5 7.14 7:2e2fa760:::200.00000001:head [write 0~48 in=48b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x5650c454cc00 con 0x5650c4fcbc00 2024-03-24T01:30:34.731 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.727+0000 7f2160000640 1 -- 172.21.15.90:0/137583561 <== osd.3 v2:172.21.15.90:6808/2999931983 3 ==== osd_op_reply(5 200.00000001 [write 0~48] v56'44 uv44 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x5650c4f52500 con 0x5650c4fcbc00 2024-03-24T01:30:34.732 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.727+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:6 7.14 7:2e2fa760:::200.00000001:head [write 48~892 in=892b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x5650c454d000 con 0x5650c4fcbc00 2024-03-24T01:30:34.732 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.729+0000 7f2160000640 1 -- 172.21.15.90:0/137583561 <== osd.3 v2:172.21.15.90:6808/2999931983 4 ==== osd_op_reply(6 200.00000001 [write 48~892] v56'45 uv45 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x5650c4f52500 con 0x5650c4fcbc00 2024-03-24T01:30:34.732 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.729+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:7 7.14 7:2e2fa760:::200.00000001:head [write 940~441 in=441b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x5650c454d400 con 0x5650c4fcbc00 2024-03-24T01:30:34.732 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.730+0000 7f2160000640 1 -- 172.21.15.90:0/137583561 <== osd.3 v2:172.21.15.90:6808/2999931983 5 ==== osd_op_reply(7 200.00000001 [write 940~441] v56'46 uv46 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x5650c4f52500 con 0x5650c4fcbc00 2024-03-24T01:30:34.732 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.730+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:8 7.14 7:2e2fa760:::200.00000001:head [write 1381~815 in=815b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x5650c454c000 con 0x5650c4fcbc00 2024-03-24T01:30:34.732 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.732+0000 7f2160000640 1 -- 172.21.15.90:0/137583561 <== osd.3 v2:172.21.15.90:6808/2999931983 6 ==== osd_op_reply(8 200.00000001 [write 1381~815] v56'47 uv47 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x5650c4f52500 con 0x5650c4fcbc00 2024-03-24T01:30:34.732 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.732+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:9 7.14 7:2e2fa760:::200.00000001:head [write 2196~2016 in=2016b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x5650c454c400 con 0x5650c4fcbc00 2024-03-24T01:30:34.735 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.735+0000 7f2160000640 1 -- 172.21.15.90:0/137583561 <== osd.3 v2:172.21.15.90:6808/2999931983 7 ==== osd_op_reply(9 200.00000001 [write 2196~2016] v56'48 uv48 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x5650c4f52500 con 0x5650c4fcbc00 2024-03-24T01:30:34.735 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.735+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:10 7.14 7:2e2fa760:::200.00000001:head [write 4212~1669 in=1669b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x5650c454cc00 con 0x5650c4fcbc00 2024-03-24T01:30:34.737 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.737+0000 7f2160000640 1 -- 172.21.15.90:0/137583561 <== osd.3 v2:172.21.15.90:6808/2999931983 8 ==== osd_op_reply(10 200.00000001 [write 4212~1669] v56'49 uv49 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x5650c4f52500 con 0x5650c4fcbc00 2024-03-24T01:30:34.737 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.737+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:11 7.14 7:2e2fa760:::200.00000001:head [write 5881~2846 in=2846b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x5650c454d800 con 0x5650c4fcbc00 2024-03-24T01:30:34.740 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.740+0000 7f2160000640 1 -- 172.21.15.90:0/137583561 <== osd.3 v2:172.21.15.90:6808/2999931983 9 ==== osd_op_reply(11 200.00000001 [write 5881~2846] v56'50 uv50 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x5650c4f52500 con 0x5650c4fcbc00 2024-03-24T01:30:34.740 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.740+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:12 7.14 7:2e2fa760:::200.00000001:head [write 8727~2256 in=2256b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x5650c454dc00 con 0x5650c4fcbc00 2024-03-24T01:30:34.744 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.743+0000 7f2160000640 1 -- 172.21.15.90:0/137583561 <== osd.3 v2:172.21.15.90:6808/2999931983 10 ==== osd_op_reply(12 200.00000001 [write 8727~2256] v56'51 uv51 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x5650c4f52500 con 0x5650c4fcbc00 2024-03-24T01:30:34.744 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.743+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:13 7.14 7:2e2fa760:::200.00000001:head [write 10983~1954 in=1954b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x5650c4fb3400 con 0x5650c4fcbc00 2024-03-24T01:30:34.749 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.748+0000 7f2160000640 1 -- 172.21.15.90:0/137583561 <== osd.3 v2:172.21.15.90:6808/2999931983 11 ==== osd_op_reply(13 200.00000001 [write 10983~1954] v56'52 uv52 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x5650c4f52500 con 0x5650c4fcbc00 2024-03-24T01:30:34.749 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.749+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:14 7.14 7:2e2fa760:::200.00000001:head [write 12937~4611 in=4611b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x5650c4fb3000 con 0x5650c4fcbc00 2024-03-24T01:30:34.751 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.751+0000 7f2160000640 1 -- 172.21.15.90:0/137583561 <== osd.3 v2:172.21.15.90:6808/2999931983 12 ==== osd_op_reply(14 200.00000001 [write 12937~4611] v56'53 uv53 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x5650c4f52500 con 0x5650c4fcbc00 2024-03-24T01:30:34.751 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.751+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:15 7.14 7:2e2fa760:::200.00000001:head [write 17548~1921 in=1921b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x5650c454d000 con 0x5650c4fcbc00 2024-03-24T01:30:34.754 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.754+0000 7f2160000640 1 -- 172.21.15.90:0/137583561 <== osd.3 v2:172.21.15.90:6808/2999931983 13 ==== osd_op_reply(15 200.00000001 [write 17548~1921] v56'54 uv54 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x5650c4f52500 con 0x5650c4fcbc00 2024-03-24T01:30:34.754 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.754+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:16 7.14 7:2e2fa760:::200.00000001:head [write 19469~3171 in=3171b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x5650c454d400 con 0x5650c4fcbc00 2024-03-24T01:30:34.757 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.756+0000 7f2160000640 1 -- 172.21.15.90:0/137583561 <== osd.3 v2:172.21.15.90:6808/2999931983 14 ==== osd_op_reply(16 200.00000001 [write 19469~3171] v56'55 uv55 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x5650c4f52500 con 0x5650c4fcbc00 2024-03-24T01:30:34.757 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.757+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:17 7.14 7:2e2fa760:::200.00000001:head [write 22640~3176 in=3176b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x5650c454c000 con 0x5650c4fcbc00 2024-03-24T01:30:34.759 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.759+0000 7f2160000640 1 -- 172.21.15.90:0/137583561 <== osd.3 v2:172.21.15.90:6808/2999931983 15 ==== osd_op_reply(17 200.00000001 [write 22640~3176] v56'56 uv56 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x5650c4f52500 con 0x5650c4fcbc00 2024-03-24T01:30:34.759 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.759+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:18 7.14 7:2e2fa760:::200.00000001:head [write 25816~2911 in=2911b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x5650c454c400 con 0x5650c4fcbc00 2024-03-24T01:30:34.762 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.762+0000 7f2160000640 1 -- 172.21.15.90:0/137583561 <== osd.3 v2:172.21.15.90:6808/2999931983 16 ==== osd_op_reply(18 200.00000001 [write 25816~2911] v56'57 uv57 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x5650c4f52500 con 0x5650c4fcbc00 2024-03-24T01:30:34.762 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.762+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:19 7.14 7:2e2fa760:::200.00000001:head [write 28727~599 in=599b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x5650c454cc00 con 0x5650c4fcbc00 2024-03-24T01:30:34.764 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.764+0000 7f2160000640 1 -- 172.21.15.90:0/137583561 <== osd.3 v2:172.21.15.90:6808/2999931983 17 ==== osd_op_reply(19 200.00000001 [write 28727~599] v56'58 uv58 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x5650c4f52500 con 0x5650c4fcbc00 2024-03-24T01:30:34.764 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.764+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:20 7.14 7:2e2fa760:::200.00000001:head [write 29326~599 in=599b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x5650c454d800 con 0x5650c4fcbc00 2024-03-24T01:30:34.766 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.766+0000 7f2160000640 1 -- 172.21.15.90:0/137583561 <== osd.3 v2:172.21.15.90:6808/2999931983 18 ==== osd_op_reply(20 200.00000001 [write 29326~599] v56'59 uv59 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x5650c4f52500 con 0x5650c4fcbc00 2024-03-24T01:30:34.766 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.766+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:21 7.14 7:2e2fa760:::200.00000001:head [write 29925~2265 in=2265b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x5650c454dc00 con 0x5650c4fcbc00 2024-03-24T01:30:34.768 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.768+0000 7f2160000640 1 -- 172.21.15.90:0/137583561 <== osd.3 v2:172.21.15.90:6808/2999931983 19 ==== osd_op_reply(21 200.00000001 [write 29925~2265] v56'60 uv60 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x5650c4f52500 con 0x5650c4fcbc00 2024-03-24T01:30:34.769 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.768+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:22 7.14 7:2e2fa760:::200.00000001:head [write 32190~1138 in=1138b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x5650c454d000 con 0x5650c4fcbc00 2024-03-24T01:30:34.772 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.772+0000 7f2160000640 1 -- 172.21.15.90:0/137583561 <== osd.3 v2:172.21.15.90:6808/2999931983 20 ==== osd_op_reply(22 200.00000001 [write 32190~1138] v56'61 uv61 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x5650c4f52500 con 0x5650c4fcbc00 2024-03-24T01:30:34.772 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.772+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:23 7.14 7:2e2fa760:::200.00000001:head [write 33328~599 in=599b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x5650c454d400 con 0x5650c4fcbc00 2024-03-24T01:30:34.774 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.774+0000 7f2160000640 1 -- 172.21.15.90:0/137583561 <== osd.3 v2:172.21.15.90:6808/2999931983 21 ==== osd_op_reply(23 200.00000001 [write 33328~599] v56'62 uv62 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x5650c4f52500 con 0x5650c4fcbc00 2024-03-24T01:30:34.774 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.774+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:24 7.14 7:2e2fa760:::200.00000001:head [write 33927~173 in=173b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x5650c454c000 con 0x5650c4fcbc00 2024-03-24T01:30:34.777 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.777+0000 7f2160000640 1 -- 172.21.15.90:0/137583561 <== osd.3 v2:172.21.15.90:6808/2999931983 22 ==== osd_op_reply(24 200.00000001 [write 33927~173] v56'63 uv63 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x5650c4f52500 con 0x5650c4fcbc00 2024-03-24T01:30:34.777 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.777+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:25 7.14 7:2e2fa760:::200.00000001:head [write 34100~76 in=76b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x5650c454c400 con 0x5650c4fcbc00 2024-03-24T01:30:34.779 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.779+0000 7f2160000640 1 -- 172.21.15.90:0/137583561 <== osd.3 v2:172.21.15.90:6808/2999931983 23 ==== osd_op_reply(25 200.00000001 [write 34100~76] v56'64 uv64 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x5650c4f52500 con 0x5650c4fcbc00 2024-03-24T01:30:34.780 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.779+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:26 7.14 7:2e2fa760:::200.00000001:head [write 34176~1651 in=1651b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x5650c4fb3400 con 0x5650c4fcbc00 2024-03-24T01:30:34.782 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.782+0000 7f2160000640 1 -- 172.21.15.90:0/137583561 <== osd.3 v2:172.21.15.90:6808/2999931983 24 ==== osd_op_reply(26 200.00000001 [write 34176~1651] v56'65 uv65 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x5650c4f52500 con 0x5650c4fcbc00 2024-03-24T01:30:34.782 INFO:tasks.workunit.client.0.smithi090.stdout:Events by type: 2024-03-24T01:30:34.782 INFO:tasks.workunit.client.0.smithi090.stdout: LID: 1 2024-03-24T01:30:34.782 INFO:tasks.workunit.client.0.smithi090.stdout: PURGED: 1 2024-03-24T01:30:34.782 INFO:tasks.workunit.client.0.smithi090.stdout: SESSION: 2 2024-03-24T01:30:34.782 INFO:tasks.workunit.client.0.smithi090.stdout: SUBTREEMAP: 1 2024-03-24T01:30:34.782 INFO:tasks.workunit.client.0.smithi090.stdout: UPDATE: 17 2024-03-24T01:30:34.782 INFO:tasks.workunit.client.0.smithi090.stdout:Errors: 0 2024-03-24T01:30:34.784 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.782+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x5650c4fcb400 msgr2=0x5650c423a100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.784 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.782+0000 7f216220c280 1 --2- 172.21.15.90:0/137583561 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x5650c4fcb400 0x5650c423a100 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.784 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.782+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x5650c4fcbc00 msgr2=0x5650c4239b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.784 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.782+0000 7f216220c280 1 --2- 172.21.15.90:0/137583561 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x5650c4fcbc00 0x5650c4239b80 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.784 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.782+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x5650c4fcb000 msgr2=0x5650c423b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.784 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.782+0000 7f216220c280 1 --2- 172.21.15.90:0/137583561 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x5650c4fcb000 0x5650c423b700 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x5650c50ba3c0 tx=0x5650c4fa6e40 comp rx=0 tx=0).stop 2024-03-24T01:30:34.784 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.782+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650c50a9c00 msgr2=0x5650c423a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.784 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.782+0000 7f216220c280 1 --2- 172.21.15.90:0/137583561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650c50a9c00 0x5650c423a680 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x5650c50bb5f0 tx=0x5650c50c38c0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.784 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.782+0000 7f215ec00640 1 -- 172.21.15.90:0/137583561 reap_dead start 2024-03-24T01:30:34.784 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.782+0000 7f215ec00640 1 -- 172.21.15.90:0/137583561 reap_dead start 2024-03-24T01:30:34.785 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.782+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 shutdown_connections 2024-03-24T01:30:34.785 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.782+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 >> 172.21.15.90:0/137583561 conn(0x5650c50a9000 msgr2=0x5650c42d7800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:34.785 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.782+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 shutdown_connections 2024-03-24T01:30:34.785 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.782+0000 7f216220c280 1 -- 172.21.15.90:0/137583561 wait complete. 2024-03-24T01:30:34.785 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.783+0000 7f216220c280 1 -- 172.21.15.90:0/4230799635 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650c41cec00 msgr2=0x5650c4238580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.785 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.783+0000 7f216220c280 1 --2- 172.21.15.90:0/4230799635 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650c41cec00 0x5650c4238580 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x5650c4fed4d0 tx=0x5650c4fa7140 comp rx=0 tx=0).stop 2024-03-24T01:30:34.785 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.783+0000 7f216220c280 1 -- 172.21.15.90:0/4230799635 shutdown_connections 2024-03-24T01:30:34.785 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.783+0000 7f216220c280 1 --2- 172.21.15.90:0/4230799635 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5650c41cfc00 0x5650c4239080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.785 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.783+0000 7f216220c280 1 --2- 172.21.15.90:0/4230799635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c41cf800 0x5650c4238b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.785 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.783+0000 7f216220c280 1 --2- 172.21.15.90:0/4230799635 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650c41cec00 0x5650c4238580 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.785 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.783+0000 7f216220c280 1 -- 172.21.15.90:0/4230799635 >> 172.21.15.90:0/4230799635 conn(0x5650c41cf000 msgr2=0x5650c42d7000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:34.785 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.783+0000 7f216220c280 1 -- 172.21.15.90:0/4230799635 shutdown_connections 2024-03-24T01:30:34.785 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.783+0000 7f216220c280 1 -- 172.21.15.90:0/4230799635 wait complete. 2024-03-24T01:30:34.788 INFO:tasks.workunit.client.0.smithi090.stderr:+ cephfs-journal-tool --rank=cephfs:0 journal reset 2024-03-24T01:30:34.836 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.836+0000 7f8bc9941280 1 -- start start 2024-03-24T01:30:34.836 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.836+0000 7f8bc9941280 1 Processor -- start 2024-03-24T01:30:34.837 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.837+0000 7f8bc9941280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561763e36c00 0x561763ea0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.837 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.837+0000 7f8bc9941280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561763e37800 0x561763ea0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.837 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.837+0000 7f8bc9941280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561763e37c00 0x561763ea1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.837 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.837+0000 7f8bc9941280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x561764c1c600 con 0x561763e37c00 2024-03-24T01:30:34.837 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.837+0000 7f8bc9941280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x561764c1c480 con 0x561763e36c00 2024-03-24T01:30:34.838 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.837+0000 7f8bc9941280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x561764c1c300 con 0x561763e37800 2024-03-24T01:30:34.838 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.837+0000 7f8bc8400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561763e37c00 0x561763ea1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.838 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.837+0000 7f8bc8400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561763e37c00 0x561763ea1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36926/0 (socket says 172.21.15.90:36926) 2024-03-24T01:30:34.838 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.837+0000 7f8bc8400640 1 -- 172.21.15.90:0/2062552595 learned_addr learned my addr 172.21.15.90:0/2062552595 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:34.839 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.837+0000 7f8bc7000640 1 --2- 172.21.15.90:0/2062552595 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561763e37800 0x561763ea0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.840 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.837+0000 7f8bc7a00640 1 --2- 172.21.15.90:0/2062552595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561763e36c00 0x561763ea0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.840 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.837+0000 7f8bc8400640 1 -- 172.21.15.90:0/2062552595 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561763e37800 msgr2=0x561763ea0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:34.840 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.838+0000 7f8bc8400640 1 --2- 172.21.15.90:0/2062552595 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561763e37800 0x561763ea0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.840 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.838+0000 7f8bc8400640 1 -- 172.21.15.90:0/2062552595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561763e36c00 msgr2=0x561763ea0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:34.840 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.838+0000 7f8bc8400640 1 --2- 172.21.15.90:0/2062552595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561763e36c00 0x561763ea0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.840 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.838+0000 7f8bc8400640 1 -- 172.21.15.90:0/2062552595 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561763e6a780 con 0x561763e37c00 2024-03-24T01:30:34.840 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.838+0000 7f8bc7a00640 1 --2- 172.21.15.90:0/2062552595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561763e36c00 0x561763ea0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:30:34.840 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.838+0000 7f8bc7000640 1 --2- 172.21.15.90:0/2062552595 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561763e37800 0x561763ea0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:30:34.840 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.838+0000 7f8bc8400640 1 --2- 172.21.15.90:0/2062552595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561763e37c00 0x561763ea1080 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x561764c59440 tx=0x561764c0ede0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75f14f529359e692 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.840 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.838+0000 7f8bc4800640 1 -- 172.21.15.90:0/2062552595 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x561763e5c9c0 con 0x561763e37c00 2024-03-24T01:30:34.840 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.838+0000 7f8bc4800640 1 -- 172.21.15.90:0/2062552595 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561763e62a80 con 0x561763e37c00 2024-03-24T01:30:34.840 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.838+0000 7f8bc4800640 1 -- 172.21.15.90:0/2062552595 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x561763e5c680 con 0x561763e37c00 2024-03-24T01:30:34.840 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.838+0000 7f8bc9941280 1 -- 172.21.15.90:0/2062552595 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561763e6a3c0 con 0x561763e37c00 2024-03-24T01:30:34.840 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.838+0000 7f8bc9941280 1 -- 172.21.15.90:0/2062552595 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x561763e6b680 con 0x561763e37c00 2024-03-24T01:30:34.840 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.838+0000 7f8bc4800640 1 -- 172.21.15.90:0/2062552595 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x561763f1ea00 con 0x561763e37c00 2024-03-24T01:30:34.841 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.839+0000 7f8bc4800640 1 -- 172.21.15.90:0/2062552595 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 36) v1 ==== 3310+0+0 (secure 0 0 0) 0x561763e91180 con 0x561763e37c00 2024-03-24T01:30:34.841 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.839+0000 7f8bc9941280 1 Processor -- start 2024-03-24T01:30:34.841 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.840+0000 7f8bc9941280 1 -- start start 2024-03-24T01:30:34.843 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.840+0000 7f8bc9941280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561764d6a000 0x561763ea1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.843 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.840+0000 7f8bc9941280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561764d6a400 0x561763ea1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.843 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.840+0000 7f8bc9941280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561764d6a800 0x561763ea2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.843 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.840+0000 7f8bc9941280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x561764c1c180 con 0x561764d6a000 2024-03-24T01:30:34.843 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.840+0000 7f8bc9941280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x561763e98900 con 0x561764d6a800 2024-03-24T01:30:34.844 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.840+0000 7f8bc9941280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x561763e98480 con 0x561764d6a400 2024-03-24T01:30:34.844 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.841+0000 7f8bc7000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561764d6a000 0x561763ea1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.844 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.841+0000 7f8bc7000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561764d6a000 0x561763ea1600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36932/0 (socket says 172.21.15.90:36932) 2024-03-24T01:30:34.844 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.841+0000 7f8bc7000640 1 -- 172.21.15.90:0/3388656694 learned_addr learned my addr 172.21.15.90:0/3388656694 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:34.844 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.841+0000 7f8bc7000640 1 -- 172.21.15.90:0/3388656694 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561764d6a400 msgr2=0x561763ea1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:34.844 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.841+0000 7f8bc7000640 1 --2- 172.21.15.90:0/3388656694 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561764d6a400 0x561763ea1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.844 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.841+0000 7f8bc7000640 1 -- 172.21.15.90:0/3388656694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561764d6a800 msgr2=0x561763ea2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:34.844 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.841+0000 7f8bc7000640 1 --2- 172.21.15.90:0/3388656694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561764d6a800 0x561763ea2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.844 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.841+0000 7f8bc7000640 1 -- 172.21.15.90:0/3388656694 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561763e6b4a0 con 0x561764d6a000 2024-03-24T01:30:34.844 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.841+0000 7f8bc7000640 1 --2- 172.21.15.90:0/3388656694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561764d6a000 0x561763ea1600 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x561764d16510 tx=0x561764d1ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48b6996a754b2978 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.844 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.841+0000 7f8bc2000640 1 -- 172.21.15.90:0/3388656694 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x561763e5c9c0 con 0x561764d6a000 2024-03-24T01:30:34.844 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.842+0000 7f8bc2000640 1 -- 172.21.15.90:0/3388656694 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561763e62e00 con 0x561764d6a000 2024-03-24T01:30:34.844 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.842+0000 7f8bc2000640 1 -- 172.21.15.90:0/3388656694 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x561763e5c680 con 0x561764d6a000 2024-03-24T01:30:34.844 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.842+0000 7f8bc9941280 1 -- 172.21.15.90:0/3388656694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561764d6a000 msgr2=0x561763ea1600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.844 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.842+0000 7f8bc9941280 1 --2- 172.21.15.90:0/3388656694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561764d6a000 0x561763ea1600 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x561764d16510 tx=0x561764d1ccc0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.845 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.842+0000 7f8bc9941280 1 -- 172.21.15.90:0/3388656694 shutdown_connections 2024-03-24T01:30:34.845 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.842+0000 7f8bc9941280 1 --2- 172.21.15.90:0/3388656694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561764d6a800 0x561763ea2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.845 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.842+0000 7f8bc9941280 1 --2- 172.21.15.90:0/3388656694 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561764d6a400 0x561763ea1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.845 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.842+0000 7f8bc9941280 1 --2- 172.21.15.90:0/3388656694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561764d6a000 0x561763ea1600 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.845 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.842+0000 7f8bc9941280 1 -- 172.21.15.90:0/3388656694 >> 172.21.15.90:0/3388656694 conn(0x561764c1fc00 msgr2=0x561763f3f800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:34.845 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.842+0000 7f8bc9941280 1 -- 172.21.15.90:0/3388656694 shutdown_connections 2024-03-24T01:30:34.845 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.842+0000 7f8bc9941280 1 -- 172.21.15.90:0/3388656694 wait complete. 2024-03-24T01:30:34.845 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.843+0000 7f8bc9941280 1 Processor -- start 2024-03-24T01:30:34.845 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.843+0000 7f8bc9941280 1 -- start start 2024-03-24T01:30:34.847 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.844+0000 7f8bc9941280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561764d6a000 0x561763ea3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.847 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.844+0000 7f8bc9941280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561764d6a400 0x561763ea2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.847 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.844+0000 7f8bc9941280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561764d6a800 0x561763ea2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.847 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.844+0000 7f8bc9941280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x561764c1d380 con 0x561764d6a800 2024-03-24T01:30:34.847 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.844+0000 7f8bc9941280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x561764c1c180 con 0x561764d6a000 2024-03-24T01:30:34.847 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.844+0000 7f8bc9941280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x561764c1cc00 con 0x561764d6a400 2024-03-24T01:30:34.847 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.844+0000 7f8bc7000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561764d6a800 0x561763ea2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.847 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.844+0000 7f8bc7000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561764d6a800 0x561763ea2680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36938/0 (socket says 172.21.15.90:36938) 2024-03-24T01:30:34.848 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.844+0000 7f8bc7000640 1 -- 172.21.15.90:0/3268841411 learned_addr learned my addr 172.21.15.90:0/3268841411 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:34.848 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.844+0000 7f8bc7000640 1 --2- 172.21.15.90:0/3268841411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561764d6a000 0x561763ea3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.848 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.844+0000 7f8bc7a00640 1 --2- 172.21.15.90:0/3268841411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561764d6a400 0x561763ea2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.848 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.844+0000 7f8bc7000640 1 -- 172.21.15.90:0/3268841411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561764d6a400 msgr2=0x561763ea2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.848 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.844+0000 7f8bc7000640 1 --2- 172.21.15.90:0/3268841411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561764d6a400 0x561763ea2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.848 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.845+0000 7f8bc7000640 1 -- 172.21.15.90:0/3268841411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561764d6a000 msgr2=0x561763ea3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.848 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.845+0000 7f8bc7000640 1 --2- 172.21.15.90:0/3268841411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561764d6a000 0x561763ea3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.848 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.845+0000 7f8bc7000640 1 -- 172.21.15.90:0/3268841411 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561763e6a000 con 0x561764d6a800 2024-03-24T01:30:34.848 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.845+0000 7f8bc7000640 1 --2- 172.21.15.90:0/3268841411 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561764d6a800 0x561763ea2680 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x561764d177d0 tx=0x561764d1cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.848 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.845+0000 7f8bc0200640 1 -- 172.21.15.90:0/3268841411 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x561763e5c4e0 con 0x561764d6a800 2024-03-24T01:30:34.848 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.845+0000 7f8bc0200640 1 -- 172.21.15.90:0/3268841411 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561763e62c40 con 0x561764d6a800 2024-03-24T01:30:34.848 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.845+0000 7f8bc0200640 1 -- 172.21.15.90:0/3268841411 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x561763e5c9c0 con 0x561764d6a800 2024-03-24T01:30:34.848 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.845+0000 7f8bc9941280 1 -- 172.21.15.90:0/3268841411 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x561763e6a960 con 0x561764d6a800 2024-03-24T01:30:34.848 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.845+0000 7f8bc9941280 1 -- 172.21.15.90:0/3268841411 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561763e6b860 con 0x561764d6a800 2024-03-24T01:30:34.848 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.847+0000 7f8bc0200640 1 -- 172.21.15.90:0/3268841411 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x561764d2cc00 con 0x561764d6a800 2024-03-24T01:30:34.850 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.847+0000 7f8bc0200640 1 --2- 172.21.15.90:0/3268841411 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x561764d86400 0x561763ea3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.850 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.847+0000 7f8bc0200640 1 -- 172.21.15.90:0/3268841411 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x561764d7ca00 con 0x561764d6a800 2024-03-24T01:30:34.850 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.848+0000 7f8bc7a00640 1 --2- 172.21.15.90:0/3268841411 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x561764d86400 0x561763ea3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.850 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.849+0000 7f8bc9941280 1 -- start start 2024-03-24T01:30:34.850 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.849+0000 7f8bc9941280 1 Processor -- start 2024-03-24T01:30:34.850 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.849+0000 7f8bc7a00640 1 --2- 172.21.15.90:0/3268841411 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x561764d86400 0x561763ea3700 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x561764c59380 tx=0x5617641b58c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.850 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.850+0000 7f8bc9941280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561764d6b000 0x561763ea2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.850 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.850+0000 7f8bc9941280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561764d87400 0x561763ea1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.850 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.850+0000 7f8bc9941280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561764d87800 0x561763ea1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.850 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.850+0000 7f8bc9941280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x561764c1c900 con 0x561764d87800 2024-03-24T01:30:34.850 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.850+0000 7f8bc9941280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x561764c1d680 con 0x561764d6b000 2024-03-24T01:30:34.850 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.850+0000 7f8bc9941280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x561763e98480 con 0x561764d87400 2024-03-24T01:30:34.852 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.850+0000 7f8bc7000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561764d87800 0x561763ea1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.852 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.850+0000 7f8bc7000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561764d87800 0x561763ea1600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36952/0 (socket says 172.21.15.90:36952) 2024-03-24T01:30:34.852 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.850+0000 7f8bc7000640 1 -- 172.21.15.90:0/2735646727 learned_addr learned my addr 172.21.15.90:0/2735646727 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:34.853 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.850+0000 7f8bc8400640 1 --2- 172.21.15.90:0/2735646727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561764d6b000 0x561763ea2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.853 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.850+0000 7f8bc7000640 1 -- 172.21.15.90:0/2735646727 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561764d87400 msgr2=0x561763ea1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:34.853 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.850+0000 7f8bc7000640 1 --2- 172.21.15.90:0/2735646727 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561764d87400 0x561763ea1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.853 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.850+0000 7f8bc7000640 1 -- 172.21.15.90:0/2735646727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561764d6b000 msgr2=0x561763ea2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:34.853 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.850+0000 7f8bc7000640 1 --2- 172.21.15.90:0/2735646727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561764d6b000 0x561763ea2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.853 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.850+0000 7f8bc7000640 1 -- 172.21.15.90:0/2735646727 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561763e6ba40 con 0x561764d87800 2024-03-24T01:30:34.853 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.850+0000 7f8bc8400640 1 --2- 172.21.15.90:0/2735646727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561764d6b000 0x561763ea2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:30:34.853 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.851+0000 7f8bc7000640 1 --2- 172.21.15.90:0/2735646727 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561764d87800 0x561763ea1600 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x561764dad800 tx=0x561764d1d620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d45767749b37ef79 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.853 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.851+0000 7f8bbc600640 1 -- 172.21.15.90:0/2735646727 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x56176413cd00 con 0x561764d87800 2024-03-24T01:30:34.853 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.851+0000 7f8bbc600640 1 -- 172.21.15.90:0/2735646727 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561763e621c0 con 0x561764d87800 2024-03-24T01:30:34.853 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.851+0000 7f8bc9941280 1 -- 172.21.15.90:0/2735646727 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561763e6b4a0 con 0x561764d87800 2024-03-24T01:30:34.853 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.851+0000 7f8bc9941280 1 -- 172.21.15.90:0/2735646727 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x561763e6a000 con 0x561764d87800 2024-03-24T01:30:34.853 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.851+0000 7f8bbc600640 1 -- 172.21.15.90:0/2735646727 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x56176413cea0 con 0x561764d87800 2024-03-24T01:30:34.853 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.851+0000 7f8bbc600640 1 -- 172.21.15.90:0/2735646727 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x561764216600 con 0x561764d87800 2024-03-24T01:30:34.853 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.851+0000 7f8bbc600640 1 -- 172.21.15.90:0/2735646727 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 36) v1 ==== 3310+0+0 (secure 0 0 0) 0x561763e91500 con 0x561764d87800 2024-03-24T01:30:34.854 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.852+0000 7f8bc9941280 1 --2- 172.21.15.90:0/2735646727 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x561764d87c00 0x56176422c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.854 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.852+0000 7f8bc9941280 1 -- 172.21.15.90:0/2735646727 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:1 7.f 7:f1f69726:::400.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x561764230000 con 0x561764d87c00 2024-03-24T01:30:34.854 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.852+0000 7f8bc8400640 1 --2- 172.21.15.90:0/2735646727 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x561764d87c00 0x56176422c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.855 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.853+0000 7f8bc8400640 1 --2- 172.21.15.90:0/2735646727 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x561764d87c00 0x56176422c000 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=89ba94ccca16016e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.855 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.853+0000 7f8bc8400640 1 -- 172.21.15.90:0/2735646727 <== osd.0 v2:172.21.15.90:6800/307045587 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x561764bba000 con 0x561764d87c00 2024-03-24T01:30:34.855 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.853+0000 7f8bc9941280 1 --2- 172.21.15.90:0/2735646727 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x561764230400 0x56176422c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.855 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.853+0000 7f8bc9941280 1 -- 172.21.15.90:0/2735646727 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:2 7.14 7:292cf221:::200.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v9 -- 0x561764230800 con 0x561764230400 2024-03-24T01:30:34.855 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.853+0000 7f8bc7a00640 1 --2- 172.21.15.90:0/2735646727 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x561764230400 0x56176422c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.856 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.854+0000 7f8bc7a00640 1 --2- 172.21.15.90:0/2735646727 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x561764230400 0x56176422c580 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=502f54a966028e58 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.856 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.854+0000 7f8bc7a00640 1 -- 172.21.15.90:0/2735646727 <== osd.3 v2:172.21.15.90:6808/2999931983 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv34 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x561764bba500 con 0x561764230400 2024-03-24T01:30:34.856 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.854+0000 7f8bba800640 1 -- 172.21.15.90:0/2735646727 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:3 7.14 7:2e2fa760:::200.00000001:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56) v9 -- 0x561764231c00 con 0x561764230400 2024-03-24T01:30:34.856 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.854+0000 7f8bba800640 1 -- 172.21.15.90:0/2735646727 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:4 7.1b 7:ddb4e4d7:::200.00000002:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56) v9 -- 0x561764c38000 con 0x561764d87c00 2024-03-24T01:30:34.856 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.855+0000 7f8bc7a00640 1 -- 172.21.15.90:0/2735646727 <== osd.3 v2:172.21.15.90:6808/2999931983 2 ==== osd_op_reply(3 200.00000001 [stat out=16b] v0'0 uv65 ondisk = 0) v8 ==== 156+0+16 (crc 0 0 0) 0x561764bba500 con 0x561764230400 2024-03-24T01:30:34.856 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.855+0000 7f8bc8400640 1 -- 172.21.15.90:0/2735646727 <== osd.0 v2:172.21.15.90:6800/307045587 2 ==== osd_op_reply(4 200.00000002 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x561764bba000 con 0x561764d87c00 2024-03-24T01:30:34.856 INFO:tasks.workunit.client.0.smithi090.stdout:old journal was 4194304~35827 2024-03-24T01:30:34.856 INFO:tasks.workunit.client.0.smithi090.stdout:new journal start will be 8388608 (4158477 bytes past old end) 2024-03-24T01:30:34.856 INFO:tasks.workunit.client.0.smithi090.stdout:writing journal head 2024-03-24T01:30:34.857 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.855+0000 7f8bc9941280 1 -- 172.21.15.90:0/2735646727 --> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] -- osd_op(unknown.0.0:5 7.14 7:292cf221:::200.00000000:head [writefull 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x561764230c00 con 0x561764230400 2024-03-24T01:30:34.857 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.856+0000 7f8bc7a00640 1 -- 172.21.15.90:0/2735646727 <== osd.3 v2:172.21.15.90:6808/2999931983 3 ==== osd_op_reply(5 200.00000000 [writefull 0~90] v56'66 uv66 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x561764bba500 con 0x561764230400 2024-03-24T01:30:34.857 INFO:tasks.workunit.client.0.smithi090.stdout:writing EResetJournal entry 2024-03-24T01:30:34.858 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.857+0000 7f8bc9941280 1 -- 172.21.15.90:0/2735646727 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:6 7.1b 7:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x561764231000 con 0x561764d87c00 2024-03-24T01:30:34.858 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.857+0000 7f8bc9941280 1 --2- 172.21.15.90:0/2735646727 >> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] conn(0x561764231400 0x56176422cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.858 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.857+0000 7f8bc9941280 1 -- 172.21.15.90:0/2735646727 --> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] -- osd_op(unknown.0.0:7 7.b 7:d5c7a900:::200.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x561764231800 con 0x561764231400 2024-03-24T01:30:34.858 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.857+0000 7f8bc9941280 1 --2- 172.21.15.90:0/2735646727 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x561764c38800 0x56176422d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.859 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.857+0000 7f8bc9941280 1 -- 172.21.15.90:0/2735646727 --> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] -- osd_op(unknown.0.0:8 7.1f 7:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x561764c38c00 con 0x561764c38800 2024-03-24T01:30:34.859 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.857+0000 7f8bc9941280 1 --2- 172.21.15.90:0/2735646727 >> [v2:172.21.15.120:6800/2150486081,v1:172.21.15.120:6801/2150486081] conn(0x561764c39000 0x56176422d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.859 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.857+0000 7f8bc9941280 1 -- 172.21.15.90:0/2735646727 --> [v2:172.21.15.120:6800/2150486081,v1:172.21.15.120:6801/2150486081] -- osd_op(unknown.0.0:9 7.1a 7:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x561764c39400 con 0x561764c39000 2024-03-24T01:30:34.859 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.857+0000 7f8bc9941280 1 -- 172.21.15.90:0/2735646727 --> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] -- osd_op(unknown.0.0:10 7.1e 7:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x561764230800 con 0x561764231400 2024-03-24T01:30:34.859 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.857+0000 7f8bc9941280 1 -- 172.21.15.90:0/2735646727 --> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] -- osd_op(unknown.0.0:11 7.1d 7:bec1251c:::200.00000007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x561764231c00 con 0x561764231400 2024-03-24T01:30:34.859 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.857+0000 7f8bc8400640 1 --2- 172.21.15.90:0/2735646727 >> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] conn(0x561764231400 0x56176422cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.859 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.857+0000 7f8bc7a00640 1 --2- 172.21.15.90:0/2735646727 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x561764c38800 0x56176422d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.859 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.858+0000 7f8bc7000640 1 --2- 172.21.15.90:0/2735646727 >> [v2:172.21.15.120:6800/2150486081,v1:172.21.15.120:6801/2150486081] conn(0x561764c39000 0x56176422d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.859 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.858+0000 7f8bc8400640 1 --2- 172.21.15.90:0/2735646727 >> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] conn(0x561764231400 0x56176422cb00 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=a3e576a4cc0d2e5b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.860 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.858+0000 7f8bc8400640 1 -- 172.21.15.90:0/2735646727 <== osd.0 v2:172.21.15.90:6800/307045587 3 ==== osd_op_reply(6 200.00000002 [delete] v56'7 uv5 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x561764bba000 con 0x561764d87c00 2024-03-24T01:30:34.860 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.859+0000 7f8bc7a00640 1 --2- 172.21.15.90:0/2735646727 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x561764c38800 0x56176422d080 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=1905c5002f7a8ef7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.860 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.859+0000 7f8bc7000640 1 --2- 172.21.15.90:0/2735646727 >> [v2:172.21.15.120:6800/2150486081,v1:172.21.15.120:6801/2150486081] conn(0x561764c39000 0x56176422d600 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=dfe919460153d80c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.860 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.860+0000 7f8bc8400640 1 -- 172.21.15.90:0/2735646727 <== osd.1 v2:172.21.15.90:6816/133441085 1 ==== osd_op_reply(10 200.00000006 [delete] v56'3 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x561764bba000 con 0x561764231400 2024-03-24T01:30:34.861 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.861+0000 7f8bc8400640 1 -- 172.21.15.90:0/2735646727 <== osd.1 v2:172.21.15.90:6816/133441085 2 ==== osd_op_reply(11 200.00000007 [delete] v56'5 uv2 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x561764bba000 con 0x561764231400 2024-03-24T01:30:34.861 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.861+0000 7f8bc8400640 1 -- 172.21.15.90:0/2735646727 <== osd.1 v2:172.21.15.90:6816/133441085 3 ==== osd_op_reply(7 200.00000003 [delete] v56'5 uv1 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x561764bba000 con 0x561764231400 2024-03-24T01:30:34.861 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.861+0000 7f8bba800640 1 -- 172.21.15.90:0/2735646727 --> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] -- osd_op(unknown.0.0:12 7.1b 7:ddb4e4d7:::200.00000002:head [write 0~48 in=48b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x561764c38400 con 0x561764d87c00 2024-03-24T01:30:34.863 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.863+0000 7f8bc8400640 1 -- 172.21.15.90:0/2735646727 <== osd.0 v2:172.21.15.90:6800/307045587 4 ==== osd_op_reply(12 200.00000002 [write 0~48] v56'8 uv8 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x561764bba000 con 0x561764d87c00 2024-03-24T01:30:34.864 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.864+0000 7f8bc7000640 1 -- 172.21.15.90:0/2735646727 <== osd.4 v2:172.21.15.120:6800/2150486081 1 ==== osd_op_reply(9 200.00000005 [delete] v56'3 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x561764bba780 con 0x561764c39000 2024-03-24T01:30:34.865 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.865+0000 7f8bc7a00640 1 -- 172.21.15.90:0/2735646727 <== osd.6 v2:172.21.15.120:6816/1454745630 1 ==== osd_op_reply(8 200.00000004 [delete] v56'20 uv19 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x561764bba500 con 0x561764c38800 2024-03-24T01:30:34.865 INFO:tasks.workunit.client.0.smithi090.stdout:done 2024-03-24T01:30:34.867 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.865+0000 7f8bc9941280 1 -- 172.21.15.90:0/2735646727 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x561764d87c00 msgr2=0x56176422c000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.867 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.865+0000 7f8bc9941280 1 --2- 172.21.15.90:0/2735646727 >> [v2:172.21.15.90:6800/307045587,v1:172.21.15.90:6801/307045587] conn(0x561764d87c00 0x56176422c000 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.868 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.865+0000 7f8bc9941280 1 -- 172.21.15.90:0/2735646727 >> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] conn(0x561764231400 msgr2=0x56176422cb00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.868 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.865+0000 7f8bc9941280 1 --2- 172.21.15.90:0/2735646727 >> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] conn(0x561764231400 0x56176422cb00 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.868 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.865+0000 7f8bc9941280 1 -- 172.21.15.90:0/2735646727 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x561764230400 msgr2=0x56176422c580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.868 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.865+0000 7f8bc9941280 1 --2- 172.21.15.90:0/2735646727 >> [v2:172.21.15.90:6808/2999931983,v1:172.21.15.90:6809/2999931983] conn(0x561764230400 0x56176422c580 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.868 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.865+0000 7f8bc9941280 1 -- 172.21.15.90:0/2735646727 >> [v2:172.21.15.120:6800/2150486081,v1:172.21.15.120:6801/2150486081] conn(0x561764c39000 msgr2=0x56176422d600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.868 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.865+0000 7f8bc9941280 1 --2- 172.21.15.90:0/2735646727 >> [v2:172.21.15.120:6800/2150486081,v1:172.21.15.120:6801/2150486081] conn(0x561764c39000 0x56176422d600 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.868 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.865+0000 7f8bc9941280 1 -- 172.21.15.90:0/2735646727 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x561764c38800 msgr2=0x56176422d080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.868 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.865+0000 7f8bc9941280 1 --2- 172.21.15.90:0/2735646727 >> [v2:172.21.15.120:6816/1454745630,v1:172.21.15.120:6817/1454745630] conn(0x561764c38800 0x56176422d080 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.868 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.865+0000 7f8bc7000640 1 -- 172.21.15.90:0/2735646727 reap_dead start 2024-03-24T01:30:34.868 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.865+0000 7f8bc7000640 1 -- 172.21.15.90:0/2735646727 reap_dead start 2024-03-24T01:30:34.868 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.865+0000 7f8bc9941280 1 -- 172.21.15.90:0/2735646727 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561764d87800 msgr2=0x561763ea1600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.868 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.865+0000 7f8bc9941280 1 --2- 172.21.15.90:0/2735646727 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561764d87800 0x561763ea1600 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x561764dad800 tx=0x561764d1d620 comp rx=0 tx=0).stop 2024-03-24T01:30:34.868 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.865+0000 7f8bc7000640 1 -- 172.21.15.90:0/2735646727 reap_dead start 2024-03-24T01:30:34.868 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.865+0000 7f8bc9941280 1 -- 172.21.15.90:0/2735646727 shutdown_connections 2024-03-24T01:30:34.869 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.865+0000 7f8bc9941280 1 -- 172.21.15.90:0/2735646727 >> 172.21.15.90:0/2735646727 conn(0x561764d6ac00 msgr2=0x561763f41800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:34.869 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.866+0000 7f8bc9941280 1 -- 172.21.15.90:0/2735646727 shutdown_connections 2024-03-24T01:30:34.869 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.866+0000 7f8bc9941280 1 -- 172.21.15.90:0/2735646727 wait complete. 2024-03-24T01:30:34.869 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.866+0000 7f8bc9941280 1 -- 172.21.15.90:0/3268841411 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x561764d86400 msgr2=0x561763ea3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.869 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.866+0000 7f8bc9941280 1 --2- 172.21.15.90:0/3268841411 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x561764d86400 0x561763ea3700 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x561764c59380 tx=0x5617641b58c0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.869 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.866+0000 7f8bc9941280 1 -- 172.21.15.90:0/3268841411 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561764d6a800 msgr2=0x561763ea2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.869 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.866+0000 7f8bc9941280 1 --2- 172.21.15.90:0/3268841411 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561764d6a800 0x561763ea2680 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x561764d177d0 tx=0x561764d1cc00 comp rx=0 tx=0).stop 2024-03-24T01:30:34.869 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.866+0000 7f8bc9941280 1 -- 172.21.15.90:0/3268841411 shutdown_connections 2024-03-24T01:30:34.869 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.866+0000 7f8bc9941280 1 --2- 172.21.15.90:0/3268841411 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x561764d86400 0x561763ea3700 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.869 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.866+0000 7f8bc9941280 1 --2- 172.21.15.90:0/3268841411 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561764d6a800 0x561763ea2680 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.869 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.866+0000 7f8bc9941280 1 --2- 172.21.15.90:0/3268841411 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561764d6a400 0x561763ea2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.869 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.866+0000 7f8bc9941280 1 --2- 172.21.15.90:0/3268841411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561764d6a000 0x561763ea3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.869 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.866+0000 7f8bc9941280 1 -- 172.21.15.90:0/3268841411 >> 172.21.15.90:0/3268841411 conn(0x561764c1fc00 msgr2=0x561763f3f800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:34.869 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.866+0000 7f8bc9941280 1 -- 172.21.15.90:0/3268841411 shutdown_connections 2024-03-24T01:30:34.869 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.866+0000 7f8bc9941280 1 -- 172.21.15.90:0/3268841411 wait complete. 2024-03-24T01:30:34.870 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.867+0000 7f8bc9941280 1 -- 172.21.15.90:0/2062552595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561763e37c00 msgr2=0x561763ea1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.870 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.867+0000 7f8bc9941280 1 --2- 172.21.15.90:0/2062552595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561763e37c00 0x561763ea1080 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x561764c59440 tx=0x561764c0ede0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.870 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.867+0000 7f8bc9941280 1 -- 172.21.15.90:0/2062552595 shutdown_connections 2024-03-24T01:30:34.870 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.867+0000 7f8bc9941280 1 --2- 172.21.15.90:0/2062552595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561763e37c00 0x561763ea1080 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.870 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.867+0000 7f8bc9941280 1 --2- 172.21.15.90:0/2062552595 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x561763e37800 0x561763ea0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.870 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.867+0000 7f8bc9941280 1 --2- 172.21.15.90:0/2062552595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561763e36c00 0x561763ea0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.870 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.867+0000 7f8bc9941280 1 -- 172.21.15.90:0/2062552595 >> 172.21.15.90:0/2062552595 conn(0x561763e37000 msgr2=0x561763f3f000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:34.870 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.867+0000 7f8bc9941280 1 -- 172.21.15.90:0/2062552595 shutdown_connections 2024-03-24T01:30:34.870 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.867+0000 7f8bc9941280 1 -- 172.21.15.90:0/2062552595 wait complete. 2024-03-24T01:30:34.872 INFO:tasks.workunit.client.0.smithi090.stderr:+ cephfs-table-tool all reset session 2024-03-24T01:30:34.928 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.918+0000 7fa6dac39280 1 -- start start 2024-03-24T01:30:34.928 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.918+0000 7fa6dac39280 1 Processor -- start 2024-03-24T01:30:34.928 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.919+0000 7fa6dac39280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3ddd2ec00 0x55e3ddd98580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.928 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.919+0000 7fa6dac39280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e3ddd2f800 0x55e3ddd98b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.928 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.919+0000 7fa6dac39280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3ddd2fc00 0x55e3ddd99080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.928 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.919+0000 7fa6dac39280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3deb18180 con 0x55e3ddd2fc00 2024-03-24T01:30:34.928 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.919+0000 7fa6dac39280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3deb18300 con 0x55e3ddd2ec00 2024-03-24T01:30:34.928 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.919+0000 7fa6dac39280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e3deb18480 con 0x55e3ddd2f800 2024-03-24T01:30:34.929 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.919+0000 7fa6d8e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3ddd2fc00 0x55e3ddd99080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.929 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.919+0000 7fa6d8e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3ddd2fc00 0x55e3ddd99080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36966/0 (socket says 172.21.15.90:36966) 2024-03-24T01:30:34.929 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.919+0000 7fa6d8e00640 1 -- 172.21.15.90:0/1769653729 learned_addr learned my addr 172.21.15.90:0/1769653729 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:34.929 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.919+0000 7fa6d8400640 1 --2- 172.21.15.90:0/1769653729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3ddd2ec00 0x55e3ddd98580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.929 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.919+0000 7fa6d7a00640 1 --2- 172.21.15.90:0/1769653729 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e3ddd2f800 0x55e3ddd98b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.929 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.920+0000 7fa6d8e00640 1 -- 172.21.15.90:0/1769653729 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e3ddd2f800 msgr2=0x55e3ddd98b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:34.929 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.920+0000 7fa6d8e00640 1 --2- 172.21.15.90:0/1769653729 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e3ddd2f800 0x55e3ddd98b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.929 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.920+0000 7fa6d8e00640 1 -- 172.21.15.90:0/1769653729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3ddd2ec00 msgr2=0x55e3ddd98580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:34.929 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.920+0000 7fa6d8e00640 1 --2- 172.21.15.90:0/1769653729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3ddd2ec00 0x55e3ddd98580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.929 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.920+0000 7fa6d8e00640 1 -- 172.21.15.90:0/1769653729 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e3ddd62780 con 0x55e3ddd2fc00 2024-03-24T01:30:34.929 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.920+0000 7fa6d8e00640 1 --2- 172.21.15.90:0/1769653729 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3ddd2fc00 0x55e3ddd99080 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x55e3deb531a0 tx=0x55e3dde17200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc1dc5319c86649b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.929 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.920+0000 7fa6d5200640 1 -- 172.21.15.90:0/1769653729 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e3ddd54680 con 0x55e3ddd2fc00 2024-03-24T01:30:34.930 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.920+0000 7fa6d5200640 1 -- 172.21.15.90:0/1769653729 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e3ddd5aa80 con 0x55e3ddd2fc00 2024-03-24T01:30:34.930 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.920+0000 7fa6d5200640 1 -- 172.21.15.90:0/1769653729 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e3ddd54b60 con 0x55e3ddd2fc00 2024-03-24T01:30:34.930 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.920+0000 7fa6dac39280 1 -- 172.21.15.90:0/1769653729 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e3ddd623c0 con 0x55e3ddd2fc00 2024-03-24T01:30:34.930 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.920+0000 7fa6dac39280 1 -- 172.21.15.90:0/1769653729 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x55e3ddd63680 con 0x55e3ddd2fc00 2024-03-24T01:30:34.930 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.921+0000 7fa6d5200640 1 -- 172.21.15.90:0/1769653729 <== mon.0 v2:172.21.15.90:3300/0 4 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x55e3debf8600 con 0x55e3ddd2fc00 2024-03-24T01:30:34.930 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.921+0000 7fa6d5200640 1 -- 172.21.15.90:0/1769653729 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 36) v1 ==== 3310+0+0 (secure 0 0 0) 0x55e3ddd89180 con 0x55e3ddd2fc00 2024-03-24T01:30:34.930 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.922+0000 7fa6dac39280 1 Processor -- start 2024-03-24T01:30:34.930 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.922+0000 7fa6dac39280 1 -- start start 2024-03-24T01:30:34.930 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.922+0000 7fa6dac39280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3debf5400 0x55e3ddd99600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.930 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.922+0000 7fa6dac39280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3debf5800 0x55e3ddd99b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.930 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.922+0000 7fa6dac39280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e3debf5c00 0x55e3ddd9a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.930 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.922+0000 7fa6dac39280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3deb18600 con 0x55e3debf5400 2024-03-24T01:30:34.930 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.923+0000 7fa6dac39280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3deb18780 con 0x55e3debf5800 2024-03-24T01:30:34.931 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.923+0000 7fa6dac39280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e3deb18900 con 0x55e3debf5c00 2024-03-24T01:30:34.931 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.923+0000 7fa6d7a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3debf5400 0x55e3ddd99600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.931 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.923+0000 7fa6d7a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3debf5400 0x55e3ddd99600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36982/0 (socket says 172.21.15.90:36982) 2024-03-24T01:30:34.931 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.923+0000 7fa6d7a00640 1 -- 172.21.15.90:0/1590441821 learned_addr learned my addr 172.21.15.90:0/1590441821 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:34.931 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.923+0000 7fa6d7a00640 1 -- 172.21.15.90:0/1590441821 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e3debf5c00 msgr2=0x55e3ddd9a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:34.931 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.923+0000 7fa6d7a00640 1 --2- 172.21.15.90:0/1590441821 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e3debf5c00 0x55e3ddd9a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.931 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.923+0000 7fa6d7a00640 1 -- 172.21.15.90:0/1590441821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3debf5800 msgr2=0x55e3ddd99b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:34.931 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.923+0000 7fa6d8400640 1 --2- 172.21.15.90:0/1590441821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3debf5800 0x55e3ddd99b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.931 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.923+0000 7fa6d7a00640 1 --2- 172.21.15.90:0/1590441821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3debf5800 0x55e3ddd99b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.931 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.923+0000 7fa6d7a00640 1 -- 172.21.15.90:0/1590441821 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e3ddd634a0 con 0x55e3debf5400 2024-03-24T01:30:34.931 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.924+0000 7fa6d7a00640 1 --2- 172.21.15.90:0/1590441821 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3debf5400 0x55e3ddd99600 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x55e3dec6a000 tx=0x55e3dde17020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43e46f3704f29723 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.931 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.924+0000 7fa6d2a00640 1 -- 172.21.15.90:0/1590441821 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e3ddd549c0 con 0x55e3debf5400 2024-03-24T01:30:34.931 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.924+0000 7fa6d2a00640 1 -- 172.21.15.90:0/1590441821 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e3ddd5afc0 con 0x55e3debf5400 2024-03-24T01:30:34.932 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.924+0000 7fa6d2a00640 1 -- 172.21.15.90:0/1590441821 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e3ddd54680 con 0x55e3debf5400 2024-03-24T01:30:34.932 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.924+0000 7fa6dac39280 1 -- 172.21.15.90:0/1590441821 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3debf5400 msgr2=0x55e3ddd99600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.932 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.924+0000 7fa6dac39280 1 --2- 172.21.15.90:0/1590441821 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3debf5400 0x55e3ddd99600 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x55e3dec6a000 tx=0x55e3dde17020 comp rx=0 tx=0).stop 2024-03-24T01:30:34.932 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.924+0000 7fa6dac39280 1 -- 172.21.15.90:0/1590441821 shutdown_connections 2024-03-24T01:30:34.932 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.924+0000 7fa6dac39280 1 --2- 172.21.15.90:0/1590441821 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e3debf5c00 0x55e3ddd9a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.932 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.924+0000 7fa6dac39280 1 --2- 172.21.15.90:0/1590441821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3debf5800 0x55e3ddd99b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.932 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.924+0000 7fa6dac39280 1 --2- 172.21.15.90:0/1590441821 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3debf5400 0x55e3ddd99600 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.932 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.924+0000 7fa6dac39280 1 -- 172.21.15.90:0/1590441821 >> 172.21.15.90:0/1590441821 conn(0x55e3debf5000 msgr2=0x55e3dde39000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:34.932 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.924+0000 7fa6dac39280 1 -- 172.21.15.90:0/1590441821 shutdown_connections 2024-03-24T01:30:34.932 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.924+0000 7fa6dac39280 1 -- 172.21.15.90:0/1590441821 wait complete. 2024-03-24T01:30:34.932 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.925+0000 7fa6dac39280 1 Processor -- start 2024-03-24T01:30:34.932 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.925+0000 7fa6dac39280 1 -- start start 2024-03-24T01:30:34.932 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.926+0000 7fa6dac39280 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3debf5400 0x55e3ddd9b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.932 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.926+0000 7fa6dac39280 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3debf5800 0x55e3ddd9ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.933 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.926+0000 7fa6dac39280 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e3debf5c00 0x55e3ddd9a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.933 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.926+0000 7fa6dac39280 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3ddd90900 con 0x55e3debf5400 2024-03-24T01:30:34.933 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.926+0000 7fa6dac39280 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3deb19e00 con 0x55e3debf5800 2024-03-24T01:30:34.933 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.926+0000 7fa6dac39280 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e3deb18d80 con 0x55e3debf5c00 2024-03-24T01:30:34.933 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.926+0000 7fa6d7a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3debf5400 0x55e3ddd9b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.933 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.926+0000 7fa6d7a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3debf5400 0x55e3ddd9b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36988/0 (socket says 172.21.15.90:36988) 2024-03-24T01:30:34.933 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.926+0000 7fa6d7a00640 1 -- 172.21.15.90:0/2791563624 learned_addr learned my addr 172.21.15.90:0/2791563624 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:34.933 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.927+0000 7fa6d7a00640 1 -- 172.21.15.90:0/2791563624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e3debf5c00 msgr2=0x55e3ddd9a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.933 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.927+0000 7fa6d7a00640 1 --2- 172.21.15.90:0/2791563624 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e3debf5c00 0x55e3ddd9a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.933 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.927+0000 7fa6d7a00640 1 -- 172.21.15.90:0/2791563624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3debf5800 msgr2=0x55e3ddd9ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.933 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.927+0000 7fa6d7a00640 1 --2- 172.21.15.90:0/2791563624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3debf5800 0x55e3ddd9ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.933 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.927+0000 7fa6d7a00640 1 -- 172.21.15.90:0/2791563624 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e3ddd62780 con 0x55e3debf5400 2024-03-24T01:30:34.933 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.927+0000 7fa6d7a00640 1 --2- 172.21.15.90:0/2791563624 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3debf5400 0x55e3ddd9b180 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x55e3dec6a330 tx=0x55e3dde16f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.934 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.927+0000 7fa6d0c00640 1 -- 172.21.15.90:0/2791563624 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e3ddd54b60 con 0x55e3debf5400 2024-03-24T01:30:34.934 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.927+0000 7fa6d0c00640 1 -- 172.21.15.90:0/2791563624 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e3ddd5ae00 con 0x55e3debf5400 2024-03-24T01:30:34.934 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.927+0000 7fa6d0c00640 1 -- 172.21.15.90:0/2791563624 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e3ddd544e0 con 0x55e3debf5400 2024-03-24T01:30:34.934 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.927+0000 7fa6dac39280 1 -- 172.21.15.90:0/2791563624 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55e3ddd623c0 con 0x55e3debf5400 2024-03-24T01:30:34.934 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.927+0000 7fa6dac39280 1 -- 172.21.15.90:0/2791563624 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e3ddd63680 con 0x55e3debf5400 2024-03-24T01:30:34.935 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.929+0000 7fa6d0c00640 1 -- 172.21.15.90:0/2791563624 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x55e3deb28c00 con 0x55e3debf5400 2024-03-24T01:30:34.935 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.929+0000 7fa6d0c00640 1 --2- 172.21.15.90:0/2791563624 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55e3deb2e400 0x55e3ddd9b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.935 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.930+0000 7fa6d0c00640 1 -- 172.21.15.90:0/2791563624 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x55e3deb2c600 con 0x55e3debf5400 2024-03-24T01:30:34.935 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.930+0000 7fa6dac39280 1 --2- 172.21.15.90:0/2791563624 >> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] conn(0x55e3deb2e800 0x55e3ddd9a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:34.936 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.930+0000 7fa6dac39280 1 -- 172.21.15.90:0/2791563624 --> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] -- osd_op(unknown.0.0:1 7.b 7:d0630e4c:::mds0_sessionmap:head [omap-clear,omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v9 -- 0x55e3deb2ec00 con 0x55e3deb2e800 2024-03-24T01:30:34.936 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.930+0000 7fa6d7a00640 1 --2- 172.21.15.90:0/2791563624 >> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] conn(0x55e3deb2e800 0x55e3ddd9a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.936 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.930+0000 7fa6d8400640 1 --2- 172.21.15.90:0/2791563624 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55e3deb2e400 0x55e3ddd9b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:34.936 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.930+0000 7fa6d7a00640 1 --2- 172.21.15.90:0/2791563624 >> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] conn(0x55e3deb2e800 0x55e3ddd9a100 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.936 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.931+0000 7fa6d8400640 1 --2- 172.21.15.90:0/2791563624 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55e3deb2e400 0x55e3ddd9b700 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x55e3ddeae7b0 tx=0x55e3de01eae0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:34.936 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.933+0000 7fa6d7a00640 1 -- 172.21.15.90:0/2791563624 <== osd.1 v2:172.21.15.90:6816/133441085 1 ==== osd_op_reply(1 mds0_sessionmap [omap-clear,omap-set-header] v56'6 uv6 ondisk = 0) v8 ==== 201+0+0 (crc 0 0 0) 0x55e3deab2000 con 0x55e3deb2e800 2024-03-24T01:30:34.936 INFO:tasks.workunit.client.0.smithi090.stdout:{ 2024-03-24T01:30:34.936 INFO:tasks.workunit.client.0.smithi090.stdout: "0": { 2024-03-24T01:30:34.936 INFO:tasks.workunit.client.0.smithi090.stdout: "data": {}, 2024-03-24T01:30:34.936 INFO:tasks.workunit.client.0.smithi090.stdout: "result": 0 2024-03-24T01:30:34.936 INFO:tasks.workunit.client.0.smithi090.stdout: } 2024-03-24T01:30:34.936 INFO:tasks.workunit.client.0.smithi090.stdout:} 2024-03-24T01:30:34.937 INFO:tasks.workunit.client.0.smithi090.stdout: 2024-03-24T01:30:34.938 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.933+0000 7fa6dac39280 1 -- 172.21.15.90:0/2791563624 >> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] conn(0x55e3deb2e800 msgr2=0x55e3ddd9a100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.938 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.933+0000 7fa6dac39280 1 --2- 172.21.15.90:0/2791563624 >> [v2:172.21.15.90:6816/133441085,v1:172.21.15.90:6818/133441085] conn(0x55e3deb2e800 0x55e3ddd9a100 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.938 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.933+0000 7fa6dac39280 1 -- 172.21.15.90:0/2791563624 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55e3deb2e400 msgr2=0x55e3ddd9b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.938 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.933+0000 7fa6dac39280 1 --2- 172.21.15.90:0/2791563624 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x55e3deb2e400 0x55e3ddd9b700 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x55e3ddeae7b0 tx=0x55e3de01eae0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.938 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.933+0000 7fa6dac39280 1 -- 172.21.15.90:0/2791563624 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3debf5400 msgr2=0x55e3ddd9b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.938 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.933+0000 7fa6dac39280 1 --2- 172.21.15.90:0/2791563624 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3debf5400 0x55e3ddd9b180 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x55e3dec6a330 tx=0x55e3dde16f60 comp rx=0 tx=0).stop 2024-03-24T01:30:34.938 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.933+0000 7fa6d8400640 1 -- 172.21.15.90:0/2791563624 reap_dead start 2024-03-24T01:30:34.938 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.933+0000 7fa6dac39280 1 -- 172.21.15.90:0/2791563624 shutdown_connections 2024-03-24T01:30:34.938 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.933+0000 7fa6dac39280 1 -- 172.21.15.90:0/2791563624 >> 172.21.15.90:0/2791563624 conn(0x55e3debf5000 msgr2=0x55e3dde39000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:34.939 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.934+0000 7fa6dac39280 1 -- 172.21.15.90:0/2791563624 shutdown_connections 2024-03-24T01:30:34.939 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.934+0000 7fa6dac39280 1 -- 172.21.15.90:0/2791563624 wait complete. 2024-03-24T01:30:34.939 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.934+0000 7fa6dac39280 1 -- 172.21.15.90:0/1769653729 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3ddd2fc00 msgr2=0x55e3ddd99080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:34.939 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.934+0000 7fa6dac39280 1 --2- 172.21.15.90:0/1769653729 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3ddd2fc00 0x55e3ddd99080 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x55e3deb531a0 tx=0x55e3dde17200 comp rx=0 tx=0).stop 2024-03-24T01:30:34.939 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.934+0000 7fa6dac39280 1 -- 172.21.15.90:0/1769653729 shutdown_connections 2024-03-24T01:30:34.939 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.934+0000 7fa6dac39280 1 --2- 172.21.15.90:0/1769653729 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e3ddd2fc00 0x55e3ddd99080 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.939 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.934+0000 7fa6dac39280 1 --2- 172.21.15.90:0/1769653729 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55e3ddd2f800 0x55e3ddd98b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.939 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.934+0000 7fa6dac39280 1 --2- 172.21.15.90:0/1769653729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3ddd2ec00 0x55e3ddd98580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:34.939 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.934+0000 7fa6dac39280 1 -- 172.21.15.90:0/1769653729 >> 172.21.15.90:0/1769653729 conn(0x55e3ddd2f000 msgr2=0x55e3dde38800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:34.939 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.934+0000 7fa6dac39280 1 -- 172.21.15.90:0/1769653729 shutdown_connections 2024-03-24T01:30:34.939 INFO:tasks.workunit.client.0.smithi090.stderr:2024-03-24T01:30:34.934+0000 7fa6dac39280 1 -- 172.21.15.90:0/1769653729 wait complete. 2024-03-24T01:30:34.940 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-24T01:30:34.940 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-24T01:30:35.076 INFO:tasks.workunit:Stopping ['suites/cephfs_journal_tool_smoke.sh'] on client.0... 2024-03-24T01:30:35.076 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-24T01:30:35.979 DEBUG:teuthology.parallel:result is None 2024-03-24T01:30:35.979 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-24T01:30:36.072 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-24T01:30:36.072 DEBUG:teuthology.orchestra.run.smithi090:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-03-24T01:30:36.123 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-03-24T01:30:36.124 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-03-24T01:30:36.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.335+0000 7f30a1a00640 1 Processor -- start 2024-03-24T01:30:36.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.335+0000 7f30a1a00640 1 -- start start 2024-03-24T01:30:36.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.337+0000 7f30a1a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f309c0b3080 0x7f309c15e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:36.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.337+0000 7f30a1a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f309c06e100 0x7f309c06e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:36.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.337+0000 7f30a1a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f309c15e7e0 con 0x7f309c15b790 2024-03-24T01:30:36.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.337+0000 7f30a1a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f309c15bb70 con 0x7f309c06e100 2024-03-24T01:30:36.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.337+0000 7f30a1a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f309c0acd00 con 0x7f309c0b3080 2024-03-24T01:30:36.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.337+0000 7f30a1000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f309c06e100 0x7f309c06e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:36.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.338+0000 7f309b400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f309c0b3080 0x7f309c15e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:36.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.338+0000 7f30a1000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f309c06e100 0x7f309c06e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.90:53058/0 (socket says 172.21.15.90:53058) 2024-03-24T01:30:36.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.338+0000 7f309be00640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f309c15b790 0x7f309c0aea50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.90:54494/0 (socket says 172.21.15.90:54494) 2024-03-24T01:30:36.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.338+0000 7f30a1000640 1 -- 172.21.15.90:0/1252711574 learned_addr learned my addr 172.21.15.90:0/1252711574 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:36.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.338+0000 7f309b400640 1 -- 172.21.15.90:0/1252711574 >> v2:172.21.15.120:3300/0 conn(0x7f309c06e100 msgr2=0x7f309c06e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:36.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.338+0000 7f309b400640 1 --2- 172.21.15.90:0/1252711574 >> v2:172.21.15.120:3300/0 conn(0x7f309c06e100 0x7f309c06e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:36.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.338+0000 7f309b400640 1 -- 172.21.15.90:0/1252711574 >> v1:172.21.15.120:6789/0 conn(0x7f309c15b790 legacy=0x7f309c0aea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:36.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.338+0000 7f309b400640 1 -- 172.21.15.90:0/1252711574 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f309c15eac0 con 0x7f309c0b3080 2024-03-24T01:30:36.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.339+0000 7f30a1000640 1 --2- 172.21.15.90:0/1252711574 >> v2:172.21.15.120:3300/0 conn(0x7f309c06e100 0x7f309c06e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T01:30:36.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.340+0000 7f309b400640 1 --2- 172.21.15.90:0/1252711574 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f309c0b3080 0x7f309c15e0a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f30880097a0 tx=0x7f308802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfb5c30ead5e854c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:36.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.340+0000 7f309aa00640 1 -- 172.21.15.90:0/1252711574 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3088035020 con 0x7f309c0b3080 2024-03-24T01:30:36.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.340+0000 7f309aa00640 1 -- 172.21.15.90:0/1252711574 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3088031810 con 0x7f309c0b3080 2024-03-24T01:30:36.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.340+0000 7f309aa00640 1 -- 172.21.15.90:0/1252711574 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3088031be0 con 0x7f309c0b3080 2024-03-24T01:30:36.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.341+0000 7f30a1a00640 1 -- 172.21.15.90:0/1252711574 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f309c0b3080 msgr2=0x7f309c15e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:36.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.341+0000 7f30a1a00640 1 --2- 172.21.15.90:0/1252711574 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f309c0b3080 0x7f309c15e0a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f30880097a0 tx=0x7f308802b8b0 comp rx=0 tx=0).stop 2024-03-24T01:30:36.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.341+0000 7f30a1a00640 1 -- 172.21.15.90:0/1252711574 shutdown_connections 2024-03-24T01:30:36.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.341+0000 7f30a1a00640 1 --2- 172.21.15.90:0/1252711574 >> v2:172.21.15.120:3300/0 conn(0x7f309c06e100 0x7f309c06e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:36.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.341+0000 7f30a1a00640 1 --2- 172.21.15.90:0/1252711574 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f309c0b3080 0x7f309c15e0a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:36.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.341+0000 7f30a1a00640 1 -- 172.21.15.90:0/1252711574 >> 172.21.15.90:0/1252711574 conn(0x7f309c06b0c0 msgr2=0x7f309c06b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:36.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.341+0000 7f30a1a00640 1 -- 172.21.15.90:0/1252711574 shutdown_connections 2024-03-24T01:30:36.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.341+0000 7f30a1a00640 1 -- 172.21.15.90:0/1252711574 wait complete. 2024-03-24T01:30:36.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.343+0000 7f30a1a00640 1 Processor -- start 2024-03-24T01:30:36.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.343+0000 7f30a1a00640 1 -- start start 2024-03-24T01:30:36.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.344+0000 7f30a1a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f309c06e100 0x7f309c14dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:36.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.344+0000 7f30a1a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f309c0b3080 0x7f309c156620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:36.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.344+0000 7f30a1a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f309c15b790 0x7f309c154a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:36.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.344+0000 7f30a1a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309c0acf30 con 0x7f309c06e100 2024-03-24T01:30:36.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.344+0000 7f30a1a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309c15f080 con 0x7f309c15b790 2024-03-24T01:30:36.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.344+0000 7f30a1a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f309c0ad450 con 0x7f309c0b3080 2024-03-24T01:30:36.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.344+0000 7f309be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f309c06e100 0x7f309c14dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:36.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.344+0000 7f309be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f309c06e100 0x7f309c14dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36994/0 (socket says 172.21.15.90:36994) 2024-03-24T01:30:36.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.344+0000 7f309be00640 1 -- 172.21.15.90:0/3675827583 learned_addr learned my addr 172.21.15.90:0/3675827583 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:36.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.344+0000 7f30a1000640 1 --2- 172.21.15.90:0/3675827583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f309c15b790 0x7f309c154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:36.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.344+0000 7f309b400640 1 --2- 172.21.15.90:0/3675827583 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f309c0b3080 0x7f309c156620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:36.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.344+0000 7f309be00640 1 -- 172.21.15.90:0/3675827583 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f309c0b3080 msgr2=0x7f309c156620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:36.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.344+0000 7f309be00640 1 --2- 172.21.15.90:0/3675827583 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f309c0b3080 0x7f309c156620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:36.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.345+0000 7f309be00640 1 -- 172.21.15.90:0/3675827583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f309c15b790 msgr2=0x7f309c154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:36.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.345+0000 7f309be00640 1 --2- 172.21.15.90:0/3675827583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f309c15b790 0x7f309c154a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:36.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.345+0000 7f309be00640 1 -- 172.21.15.90:0/3675827583 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f309c151a80 con 0x7f309c06e100 2024-03-24T01:30:36.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.345+0000 7f309be00640 1 --2- 172.21.15.90:0/3675827583 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f309c06e100 0x7f309c14dcb0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f308c00c3f0 tx=0x7f308c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:36.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.345+0000 7f3093e00640 1 -- 172.21.15.90:0/3675827583 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308c0074e0 con 0x7f309c06e100 2024-03-24T01:30:36.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.345+0000 7f3093e00640 1 -- 172.21.15.90:0/3675827583 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f308c007680 con 0x7f309c06e100 2024-03-24T01:30:36.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.345+0000 7f3093e00640 1 -- 172.21.15.90:0/3675827583 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308c004a00 con 0x7f309c06e100 2024-03-24T01:30:36.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.345+0000 7f30a1a00640 1 -- 172.21.15.90:0/3675827583 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f309c155580 con 0x7f309c06e100 2024-03-24T01:30:36.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.345+0000 7f30a1a00640 1 -- 172.21.15.90:0/3675827583 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f309c155b40 con 0x7f309c06e100 2024-03-24T01:30:36.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.346+0000 7f30a1a00640 1 -- 172.21.15.90:0/3675827583 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f309c04f400 con 0x7f309c06e100 2024-03-24T01:30:36.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.347+0000 7f3093e00640 1 -- 172.21.15.90:0/3675827583 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f308c01a020 con 0x7f309c06e100 2024-03-24T01:30:36.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.348+0000 7f3093e00640 1 --2- 172.21.15.90:0/3675827583 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f307c06bbd0 0x7f307c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:36.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.348+0000 7f3093e00640 1 -- 172.21.15.90:0/3675827583 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f308c08a560 con 0x7f309c06e100 2024-03-24T01:30:36.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.349+0000 7f309b400640 1 --2- 172.21.15.90:0/3675827583 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f307c06bbd0 0x7f307c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:36.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.350+0000 7f309b400640 1 --2- 172.21.15.90:0/3675827583 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f307c06bbd0 0x7f307c06e090 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f3088036000 tx=0x7f3088008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:36.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.353+0000 7f3093e00640 1 -- 172.21.15.90:0/3675827583 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f308c010300 con 0x7f309c06e100 2024-03-24T01:30:36.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:36.593+0000 7f30a1a00640 1 -- 172.21.15.90:0/3675827583 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) v1 -- 0x7f309c067a70 con 0x7f309c06e100 2024-03-24T01:30:37.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.204+0000 7f3093e00640 1 -- 172.21.15.90:0/3675827583 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v37) v1 ==== 162+0+0 (secure 0 0 0) 0x7f308c058d00 con 0x7f309c06e100 2024-03-24T01:30:37.205 INFO:teuthology.orchestra.run.smithi090.stderr:cephfs marked joinable; MDS may join as newly active. 2024-03-24T01:30:37.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.208+0000 7f30a1a00640 1 -- 172.21.15.90:0/3675827583 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f307c06bbd0 msgr2=0x7f307c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:37.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.208+0000 7f30a1a00640 1 --2- 172.21.15.90:0/3675827583 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f307c06bbd0 0x7f307c06e090 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f3088036000 tx=0x7f3088008030 comp rx=0 tx=0).stop 2024-03-24T01:30:37.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.208+0000 7f30a1a00640 1 -- 172.21.15.90:0/3675827583 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f309c06e100 msgr2=0x7f309c14dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:37.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.208+0000 7f30a1a00640 1 --2- 172.21.15.90:0/3675827583 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f309c06e100 0x7f309c14dcb0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f308c00c3f0 tx=0x7f308c00c8b0 comp rx=0 tx=0).stop 2024-03-24T01:30:37.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.209+0000 7f30a1a00640 1 -- 172.21.15.90:0/3675827583 shutdown_connections 2024-03-24T01:30:37.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.209+0000 7f30a1a00640 1 --2- 172.21.15.90:0/3675827583 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f307c06bbd0 0x7f307c06e090 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:37.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.209+0000 7f30a1a00640 1 --2- 172.21.15.90:0/3675827583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f309c15b790 0x7f309c154a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:37.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.209+0000 7f30a1a00640 1 --2- 172.21.15.90:0/3675827583 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f309c0b3080 0x7f309c156620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:37.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.209+0000 7f30a1a00640 1 --2- 172.21.15.90:0/3675827583 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f309c06e100 0x7f309c14dcb0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:37.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.209+0000 7f30a1a00640 1 -- 172.21.15.90:0/3675827583 >> 172.21.15.90:0/3675827583 conn(0x7f309c06b0c0 msgr2=0x7f309c15be00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:37.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.209+0000 7f30a1a00640 1 -- 172.21.15.90:0/3675827583 shutdown_connections 2024-03-24T01:30:37.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.209+0000 7f30a1a00640 1 -- 172.21.15.90:0/3675827583 wait complete. 2024-03-24T01:30:37.233 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:30:37.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.419+0000 7f262ee00640 1 Processor -- start 2024-03-24T01:30:37.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.419+0000 7f262ee00640 1 -- start start 2024-03-24T01:30:37.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.421+0000 7f262ee00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26280afd50 0x7f26280674a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:37.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.421+0000 7f262ee00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26280b5070 con 0x7f262806c7d0 2024-03-24T01:30:37.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.421+0000 7f262ee00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26280b5350 con 0x7f262815e230 2024-03-24T01:30:37.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.421+0000 7f262ee00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26280b6370 con 0x7f26280afd50 2024-03-24T01:30:37.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.421+0000 7f261f400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f262806c7d0 0x7f26280b0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57772/0 (socket says 172.21.15.90:57772) 2024-03-24T01:30:37.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.421+0000 7f261f400640 1 -- 172.21.15.90:0/2915665563 learned_addr learned my addr 172.21.15.90:0/2915665563 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:30:37.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.422+0000 7f2627e00640 1 --2- 172.21.15.90:0/2915665563 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26280afd50 0x7f26280674a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:37.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.422+0000 7f2627400640 1 -- 172.21.15.90:0/2915665563 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3441693231 0 0) 0x7f26280b5070 con 0x7f262806c7d0 2024-03-24T01:30:37.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.422+0000 7f2627400640 1 -- 172.21.15.90:0/2915665563 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f260c003520 con 0x7f262806c7d0 2024-03-24T01:30:37.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.422+0000 7f2627400640 1 -- 172.21.15.90:0/2915665563 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1891889031 0 0) 0x7f260c003520 con 0x7f262806c7d0 2024-03-24T01:30:37.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.422+0000 7f2627400640 1 -- 172.21.15.90:0/2915665563 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26280afd50 msgr2=0x7f26280674a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:37.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.422+0000 7f2627400640 1 --2- 172.21.15.90:0/2915665563 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26280afd50 0x7f26280674a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:37.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.422+0000 7f2627400640 1 -- 172.21.15.90:0/2915665563 >> v1:172.21.15.120:6789/0 conn(0x7f262815e230 legacy=0x7f26280ad1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:37.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.422+0000 7f2627400640 1 -- 172.21.15.90:0/2915665563 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26280b6090 con 0x7f262806c7d0 2024-03-24T01:30:37.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.422+0000 7f2627400640 1 -- 172.21.15.90:0/2915665563 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f2610001670 con 0x7f262806c7d0 2024-03-24T01:30:37.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.422+0000 7f2627400640 1 -- 172.21.15.90:0/2915665563 >> v1:172.21.15.90:6789/0 conn(0x7f262806c7d0 legacy=0x7f26280b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:37.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.422+0000 7f2627400640 1 --2- 172.21.15.90:0/2915665563 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f260c0039e0 0x7f260c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:37.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.422+0000 7f2627400640 1 --2- 172.21.15.90:0/2915665563 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26280afd50 0x7f260c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:37.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.422+0000 7f2627400640 1 --2- 172.21.15.90:0/2915665563 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f260c004d40 0x7f260c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:37.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.422+0000 7f2627400640 1 -- 172.21.15.90:0/2915665563 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f260c00ba70 con 0x7f260c004d40 2024-03-24T01:30:37.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.422+0000 7f2627400640 1 -- 172.21.15.90:0/2915665563 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f260c00bbf0 con 0x7f260c0039e0 2024-03-24T01:30:37.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.422+0000 7f2627400640 1 -- 172.21.15.90:0/2915665563 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f260c00bdf0 con 0x7f26280afd50 2024-03-24T01:30:37.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.423+0000 7f262ce00640 1 --2- 172.21.15.90:0/2915665563 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f260c004d40 0x7f260c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:37.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.423+0000 7f261f400640 1 --2- 172.21.15.90:0/2915665563 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26280afd50 0x7f260c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:37.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.423+0000 7f2627e00640 1 --2- 172.21.15.90:0/2915665563 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f260c0039e0 0x7f260c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:37.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.423+0000 7f262ce00640 1 -- 172.21.15.90:0/2915665563 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26280afd50 msgr2=0x7f260c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:37.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.423+0000 7f262ce00640 1 --2- 172.21.15.90:0/2915665563 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f26280afd50 0x7f260c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:37.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.423+0000 7f262ce00640 1 -- 172.21.15.90:0/2915665563 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f260c0039e0 msgr2=0x7f260c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:37.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.423+0000 7f262ce00640 1 --2- 172.21.15.90:0/2915665563 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f260c0039e0 0x7f260c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:37.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.423+0000 7f262ce00640 1 -- 172.21.15.90:0/2915665563 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f260c003520 con 0x7f260c004d40 2024-03-24T01:30:37.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.424+0000 7f262ce00640 1 --2- 172.21.15.90:0/2915665563 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f260c004d40 0x7f260c0051c0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f262815e640 tx=0x7f260c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1449e9942fe4d8e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:37.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.425+0000 7f2627400640 1 -- 172.21.15.90:0/2915665563 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2620001a70 con 0x7f260c004d40 2024-03-24T01:30:37.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.425+0000 7f2627400640 1 -- 172.21.15.90:0/2915665563 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2620003b70 con 0x7f260c004d40 2024-03-24T01:30:37.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.425+0000 7f2627400640 1 -- 172.21.15.90:0/2915665563 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26200050d0 con 0x7f260c004d40 2024-03-24T01:30:37.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.425+0000 7f262ee00640 1 -- 172.21.15.90:0/2915665563 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f260c004d40 msgr2=0x7f260c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:37.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.425+0000 7f262ee00640 1 --2- 172.21.15.90:0/2915665563 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f260c004d40 0x7f260c0051c0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f262815e640 tx=0x7f260c005830 comp rx=0 tx=0).stop 2024-03-24T01:30:37.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.425+0000 7f262ce00640 1 -- 172.21.15.90:0/2915665563 reap_dead start 2024-03-24T01:30:37.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.425+0000 7f262ee00640 1 -- 172.21.15.90:0/2915665563 shutdown_connections 2024-03-24T01:30:37.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.425+0000 7f262ee00640 1 -- 172.21.15.90:0/2915665563 >> 172.21.15.90:0/2915665563 conn(0x7f2628099a70 msgr2=0x7f26280b5730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:37.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.425+0000 7f262ee00640 1 -- 172.21.15.90:0/2915665563 shutdown_connections 2024-03-24T01:30:37.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.426+0000 7f262ee00640 1 -- 172.21.15.90:0/2915665563 wait complete. 2024-03-24T01:30:37.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.427+0000 7f262ee00640 1 Processor -- start 2024-03-24T01:30:37.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.427+0000 7f262ee00640 1 -- start start 2024-03-24T01:30:37.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.428+0000 7f262ee00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f260c004d40 0x7f2628157000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:37.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.428+0000 7f262ee00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2628150270 0x7f2628154d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:37.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.428+0000 7f262ee00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2628160b40 0x7f2628160fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:37.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.428+0000 7f262ee00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26280d0f70 con 0x7f2628150270 2024-03-24T01:30:37.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.428+0000 7f262ee00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26280b5e40 con 0x7f260c004d40 2024-03-24T01:30:37.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.428+0000 7f262ee00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26280674a0 con 0x7f2628160b40 2024-03-24T01:30:37.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.429+0000 7f261f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2628150270 0x7f2628154d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:37.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.429+0000 7f261f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2628150270 0x7f2628154d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37006/0 (socket says 172.21.15.90:37006) 2024-03-24T01:30:37.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.429+0000 7f261f400640 1 -- 172.21.15.90:0/44460664 learned_addr learned my addr 172.21.15.90:0/44460664 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:37.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.429+0000 7f2627e00640 1 --2- 172.21.15.90:0/44460664 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f260c004d40 0x7f2628157000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:37.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.429+0000 7f262ce00640 1 --2- 172.21.15.90:0/44460664 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2628160b40 0x7f2628160fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:37.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.429+0000 7f261f400640 1 -- 172.21.15.90:0/44460664 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2628160b40 msgr2=0x7f2628160fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:37.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.429+0000 7f261f400640 1 --2- 172.21.15.90:0/44460664 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2628160b40 0x7f2628160fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:37.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.429+0000 7f261f400640 1 -- 172.21.15.90:0/44460664 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f260c004d40 msgr2=0x7f2628157000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:37.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.429+0000 7f261f400640 1 --2- 172.21.15.90:0/44460664 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f260c004d40 0x7f2628157000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:37.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.429+0000 7f261f400640 1 -- 172.21.15.90:0/44460664 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2628151280 con 0x7f2628150270 2024-03-24T01:30:37.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.430+0000 7f261f400640 1 --2- 172.21.15.90:0/44460664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2628150270 0x7f2628154d90 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f26100072c0 tx=0x7f2610007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:37.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.430+0000 7f2624c00640 1 -- 172.21.15.90:0/44460664 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f261000f8e0 con 0x7f2628150270 2024-03-24T01:30:37.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.430+0000 7f2624c00640 1 -- 172.21.15.90:0/44460664 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2610018070 con 0x7f2628150270 2024-03-24T01:30:37.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.430+0000 7f2624c00640 1 -- 172.21.15.90:0/44460664 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f261000fce0 con 0x7f2628150270 2024-03-24T01:30:37.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.430+0000 7f262ee00640 1 -- 172.21.15.90:0/44460664 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f262815f1d0 con 0x7f2628150270 2024-03-24T01:30:37.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.430+0000 7f262ee00640 1 -- 172.21.15.90:0/44460664 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2628157950 con 0x7f2628150270 2024-03-24T01:30:37.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.431+0000 7f262ee00640 1 -- 172.21.15.90:0/44460664 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2628068060 con 0x7f2628150270 2024-03-24T01:30:37.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.432+0000 7f2624c00640 1 -- 172.21.15.90:0/44460664 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f261001d460 con 0x7f2628150270 2024-03-24T01:30:37.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.433+0000 7f2624c00640 1 --2- 172.21.15.90:0/44460664 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f260406bc00 0x7f260406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:37.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.433+0000 7f2624c00640 1 -- 172.21.15.90:0/44460664 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f261008ae10 con 0x7f2628150270 2024-03-24T01:30:37.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.433+0000 7f2627e00640 1 --2- 172.21.15.90:0/44460664 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f260406bc00 0x7f260406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:37.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.434+0000 7f2627e00640 1 --2- 172.21.15.90:0/44460664 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f260406bc00 0x7f260406e0c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f2618009a10 tx=0x7f2618008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:37.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.437+0000 7f2624c00640 1 -- 172.21.15.90:0/44460664 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f261008e030 con 0x7f2628150270 2024-03-24T01:30:37.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.673+0000 7f262ee00640 1 -- 172.21.15.90:0/44460664 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f262806e6e0 con 0x7f2628150270 2024-03-24T01:30:37.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.674+0000 7f2624c00640 1 -- 172.21.15.90:0/44460664 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f2610059640 con 0x7f2628150270 2024-03-24T01:30:37.675 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:30:37.675 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":38,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4985,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6834/3475131973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":3475131973},{"type":"v1","addr":"172.21.15.90:6834","nonce":3475131973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":4994,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/3151604088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":3151604088},{"type":"v1","addr":"172.21.15.120:6837","nonce":3151604088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5428,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/1239583677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1239583677},{"type":"v1","addr":"172.21.15.120:6836","nonce":1239583677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T01:27:13.961373+0000","modified":"2024-03-24T01:30:37.209094+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6219},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6219":{"gid":6219,"name":"c","rank":0,"incarnation":38,"state":"up:replay","state_seq":1,"addr":"172.21.15.90:6835/226826781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":226826781},{"type":"v1","addr":"172.21.15.90:6835","nonce":226826781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3}]} 2024-03-24T01:30:37.675 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 38 2024-03-24T01:30:37.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.677+0000 7f262ee00640 1 -- 172.21.15.90:0/44460664 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f260406bc00 msgr2=0x7f260406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:37.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.677+0000 7f262ee00640 1 --2- 172.21.15.90:0/44460664 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f260406bc00 0x7f260406e0c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f2618009a10 tx=0x7f2618008040 comp rx=0 tx=0).stop 2024-03-24T01:30:37.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.677+0000 7f262ee00640 1 -- 172.21.15.90:0/44460664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2628150270 msgr2=0x7f2628154d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:37.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.677+0000 7f262ee00640 1 --2- 172.21.15.90:0/44460664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2628150270 0x7f2628154d90 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f26100072c0 tx=0x7f2610007780 comp rx=0 tx=0).stop 2024-03-24T01:30:37.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.677+0000 7f262ee00640 1 -- 172.21.15.90:0/44460664 shutdown_connections 2024-03-24T01:30:37.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.677+0000 7f262ee00640 1 --2- 172.21.15.90:0/44460664 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f260406bc00 0x7f260406e0c0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:37.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.677+0000 7f262ee00640 1 --2- 172.21.15.90:0/44460664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2628150270 0x7f2628154d90 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:37.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.677+0000 7f262ee00640 1 --2- 172.21.15.90:0/44460664 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2628160b40 0x7f2628160fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:37.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.677+0000 7f262ee00640 1 --2- 172.21.15.90:0/44460664 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f260c004d40 0x7f2628157000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:37.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.677+0000 7f262ee00640 1 -- 172.21.15.90:0/44460664 >> 172.21.15.90:0/44460664 conn(0x7f2628099a70 msgr2=0x7f262806ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:37.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.677+0000 7f262ee00640 1 -- 172.21.15.90:0/44460664 shutdown_connections 2024-03-24T01:30:37.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:37.677+0000 7f262ee00640 1 -- 172.21.15.90:0/44460664 wait complete. 2024-03-24T01:30:38.097 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 38, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T01:27:13.961373+0000', 'modified': '2024-03-24T01:30:37.209094+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6219}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6219': {'gid': 6219, 'name': 'c', 'rank': 0, 'incarnation': 38, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.90:6835/226826781', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6833', 'nonce': 226826781}, {'type': 'v1', 'addr': '172.21.15.90:6835', 'nonce': 226826781}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-24T01:30:38.098 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6219:up:replay 2024-03-24T01:30:39.099 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:30:39.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.317+0000 7f8331a00640 1 Processor -- start 2024-03-24T01:30:39.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.317+0000 7f8331a00640 1 -- start start 2024-03-24T01:30:39.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.319+0000 7f8331a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f832c1454e0 0x7f832c0cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:39.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.319+0000 7f8331a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f832c0cc660 con 0x7f832c0ce360 2024-03-24T01:30:39.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.319+0000 7f8331a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f832c0cc920 con 0x7f832c06f000 2024-03-24T01:30:39.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.319+0000 7f8331a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f832c0cc430 con 0x7f832c1454e0 2024-03-24T01:30:39.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.319+0000 7f832b400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f832c0ce360 0x7f832c15c090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57786/0 (socket says 172.21.15.90:57786) 2024-03-24T01:30:39.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.319+0000 7f832b400640 1 -- 172.21.15.90:0/1389313232 learned_addr learned my addr 172.21.15.90:0/1389313232 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:30:39.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.319+0000 7f832aa00640 1 --2- 172.21.15.90:0/1389313232 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f832c1454e0 0x7f832c0cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:39.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.320+0000 7f832a000640 1 -- 172.21.15.90:0/1389313232 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1121889896 0 0) 0x7f832c0cc660 con 0x7f832c0ce360 2024-03-24T01:30:39.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.320+0000 7f832a000640 1 -- 172.21.15.90:0/1389313232 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8310003520 con 0x7f832c0ce360 2024-03-24T01:30:39.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.320+0000 7f832a000640 1 -- 172.21.15.90:0/1389313232 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4106648097 0 0) 0x7f8310003520 con 0x7f832c0ce360 2024-03-24T01:30:39.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.320+0000 7f832a000640 1 -- 172.21.15.90:0/1389313232 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f832c1454e0 msgr2=0x7f832c0cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:39.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.320+0000 7f832a000640 1 --2- 172.21.15.90:0/1389313232 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f832c1454e0 0x7f832c0cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:39.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.320+0000 7f832a000640 1 -- 172.21.15.90:0/1389313232 >> v1:172.21.15.120:6789/0 conn(0x7f832c06f000 legacy=0x7f832c0cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:39.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.320+0000 7f832a000640 1 -- 172.21.15.90:0/1389313232 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f832c0ccc00 con 0x7f832c0ce360 2024-03-24T01:30:39.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.320+0000 7f832a000640 1 -- 172.21.15.90:0/1389313232 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f83240019f0 con 0x7f832c0ce360 2024-03-24T01:30:39.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.320+0000 7f832a000640 1 -- 172.21.15.90:0/1389313232 >> v1:172.21.15.90:6789/0 conn(0x7f832c0ce360 legacy=0x7f832c15c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:39.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.320+0000 7f832a000640 1 --2- 172.21.15.90:0/1389313232 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f832c1454e0 0x7f83100039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:39.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.320+0000 7f832a000640 1 --2- 172.21.15.90:0/1389313232 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8310003f60 0x7f8310004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:39.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.321+0000 7f832a000640 1 --2- 172.21.15.90:0/1389313232 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8310004d40 0x7f83100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:39.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.321+0000 7f832a000640 1 -- 172.21.15.90:0/1389313232 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f831000ba70 con 0x7f8310003f60 2024-03-24T01:30:39.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.321+0000 7f832a000640 1 -- 172.21.15.90:0/1389313232 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f831000bc20 con 0x7f8310004d40 2024-03-24T01:30:39.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.321+0000 7f832a000640 1 -- 172.21.15.90:0/1389313232 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f831000be20 con 0x7f832c1454e0 2024-03-24T01:30:39.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.321+0000 7f8323e00640 1 --2- 172.21.15.90:0/1389313232 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8310003f60 0x7f8310004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:39.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.321+0000 7f8323e00640 1 -- 172.21.15.90:0/1389313232 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f832c1454e0 msgr2=0x7f83100039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:39.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.321+0000 7f832aa00640 1 --2- 172.21.15.90:0/1389313232 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f832c1454e0 0x7f83100039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:39.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.321+0000 7f832b400640 1 --2- 172.21.15.90:0/1389313232 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8310004d40 0x7f83100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:39.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.321+0000 7f8323e00640 1 --2- 172.21.15.90:0/1389313232 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f832c1454e0 0x7f83100039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:39.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.321+0000 7f8323e00640 1 -- 172.21.15.90:0/1389313232 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8310004d40 msgr2=0x7f83100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:39.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.321+0000 7f8323e00640 1 --2- 172.21.15.90:0/1389313232 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8310004d40 0x7f83100051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:39.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.321+0000 7f8323e00640 1 -- 172.21.15.90:0/1389313232 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8310003520 con 0x7f8310003f60 2024-03-24T01:30:39.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.322+0000 7f8323e00640 1 --2- 172.21.15.90:0/1389313232 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8310003f60 0x7f8310004800 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f8314001750 tx=0x7f8310005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61550fe429b9effd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:39.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.323+0000 7f832a000640 1 -- 172.21.15.90:0/1389313232 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8314001370 con 0x7f8310003f60 2024-03-24T01:30:39.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.323+0000 7f832a000640 1 -- 172.21.15.90:0/1389313232 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83140052d0 con 0x7f8310003f60 2024-03-24T01:30:39.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.323+0000 7f832a000640 1 -- 172.21.15.90:0/1389313232 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8314003830 con 0x7f8310003f60 2024-03-24T01:30:39.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.323+0000 7f8331a00640 1 -- 172.21.15.90:0/1389313232 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8310003f60 msgr2=0x7f8310004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:39.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.323+0000 7f8331a00640 1 --2- 172.21.15.90:0/1389313232 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8310003f60 0x7f8310004800 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f8314001750 tx=0x7f8310005830 comp rx=0 tx=0).stop 2024-03-24T01:30:39.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.323+0000 7f832b400640 1 -- 172.21.15.90:0/1389313232 reap_dead start 2024-03-24T01:30:39.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.323+0000 7f8331a00640 1 -- 172.21.15.90:0/1389313232 shutdown_connections 2024-03-24T01:30:39.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.323+0000 7f8331a00640 1 -- 172.21.15.90:0/1389313232 >> 172.21.15.90:0/1389313232 conn(0x7f832c06b2a0 msgr2=0x7f832c06b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:39.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.323+0000 7f8331a00640 1 -- 172.21.15.90:0/1389313232 shutdown_connections 2024-03-24T01:30:39.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.323+0000 7f8331a00640 1 -- 172.21.15.90:0/1389313232 wait complete. 2024-03-24T01:30:39.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.325+0000 7f8331a00640 1 Processor -- start 2024-03-24T01:30:39.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.325+0000 7f8331a00640 1 -- start start 2024-03-24T01:30:39.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.326+0000 7f8331a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8310003f60 0x7f832c0a9360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:39.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.326+0000 7f8331a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f832c1e5b20 0x7f832c1f53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:39.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.326+0000 7f8331a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f832c1f5920 0x7f832c14e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:39.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.326+0000 7f8331a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832c0cd6e0 con 0x7f832c1e5b20 2024-03-24T01:30:39.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.326+0000 7f8331a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832c0cd1c0 con 0x7f8310003f60 2024-03-24T01:30:39.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.326+0000 7f8331a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f832c0cd860 con 0x7f832c1f5920 2024-03-24T01:30:39.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.326+0000 7f8323e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f832c1e5b20 0x7f832c1f53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:39.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.326+0000 7f8323e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f832c1e5b20 0x7f832c1f53e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37034/0 (socket says 172.21.15.90:37034) 2024-03-24T01:30:39.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.326+0000 7f8323e00640 1 -- 172.21.15.90:0/1587269000 learned_addr learned my addr 172.21.15.90:0/1587269000 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:39.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.327+0000 7f832aa00640 1 --2- 172.21.15.90:0/1587269000 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8310003f60 0x7f832c0a9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:39.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.327+0000 7f832b400640 1 --2- 172.21.15.90:0/1587269000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f832c1f5920 0x7f832c14e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:39.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.327+0000 7f8323e00640 1 -- 172.21.15.90:0/1587269000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f832c1f5920 msgr2=0x7f832c14e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:39.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.327+0000 7f8323e00640 1 --2- 172.21.15.90:0/1587269000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f832c1f5920 0x7f832c14e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:39.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.327+0000 7f8323e00640 1 -- 172.21.15.90:0/1587269000 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8310003f60 msgr2=0x7f832c0a9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:39.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.327+0000 7f8323e00640 1 --2- 172.21.15.90:0/1587269000 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8310003f60 0x7f832c0a9360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:39.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.327+0000 7f8323e00640 1 -- 172.21.15.90:0/1587269000 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f832c1f1cf0 con 0x7f832c1e5b20 2024-03-24T01:30:39.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.327+0000 7f832b400640 1 --2- 172.21.15.90:0/1587269000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f832c1f5920 0x7f832c14e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:30:39.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.327+0000 7f8323e00640 1 --2- 172.21.15.90:0/1587269000 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f832c1e5b20 0x7f832c1f53e0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f8314008fd0 tx=0x7f8314001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:39.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.327+0000 7f8322a00640 1 -- 172.21.15.90:0/1587269000 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8314041070 con 0x7f832c1e5b20 2024-03-24T01:30:39.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.327+0000 7f8322a00640 1 -- 172.21.15.90:0/1587269000 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8314000ed0 con 0x7f832c1e5b20 2024-03-24T01:30:39.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.327+0000 7f8331a00640 1 -- 172.21.15.90:0/1587269000 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f832c0a9cb0 con 0x7f832c1e5b20 2024-03-24T01:30:39.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.328+0000 7f8322a00640 1 -- 172.21.15.90:0/1587269000 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8314007a30 con 0x7f832c1e5b20 2024-03-24T01:30:39.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.328+0000 7f8331a00640 1 -- 172.21.15.90:0/1587269000 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f832c0aa270 con 0x7f832c1e5b20 2024-03-24T01:30:39.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.329+0000 7f8331a00640 1 -- 172.21.15.90:0/1587269000 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f832c1f5d30 con 0x7f832c1e5b20 2024-03-24T01:30:39.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.330+0000 7f8322a00640 1 -- 172.21.15.90:0/1587269000 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8314044070 con 0x7f832c1e5b20 2024-03-24T01:30:39.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.330+0000 7f8322a00640 1 --2- 172.21.15.90:0/1587269000 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f830806bbd0 0x7f830806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:39.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.330+0000 7f8322a00640 1 -- 172.21.15.90:0/1587269000 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f83140ac730 con 0x7f832c1e5b20 2024-03-24T01:30:39.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.331+0000 7f832aa00640 1 --2- 172.21.15.90:0/1587269000 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f830806bbd0 0x7f830806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:39.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.332+0000 7f832aa00640 1 --2- 172.21.15.90:0/1587269000 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f830806bbd0 0x7f830806e090 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f831c009a10 tx=0x7f831c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:39.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.335+0000 7f8322a00640 1 -- 172.21.15.90:0/1587269000 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f832c1f5d30 con 0x7f832c1e5b20 2024-03-24T01:30:39.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.570+0000 7f8331a00640 1 -- 172.21.15.90:0/1587269000 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f832c073570 con 0x7f832c1e5b20 2024-03-24T01:30:39.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.571+0000 7f8322a00640 1 -- 172.21.15.90:0/1587269000 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4975 (secure 0 0 0) 0x7f831407af60 con 0x7f832c1e5b20 2024-03-24T01:30:39.572 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:30:39.572 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":40,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4985,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6834/3475131973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":3475131973},{"type":"v1","addr":"172.21.15.90:6834","nonce":3475131973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":4994,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/3151604088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":3151604088},{"type":"v1","addr":"172.21.15.120:6837","nonce":3151604088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5428,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/1239583677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1239583677},{"type":"v1","addr":"172.21.15.120:6836","nonce":1239583677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T01:27:13.961373+0000","modified":"2024-03-24T01:30:38.226436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6219},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6219":{"gid":6219,"name":"c","rank":0,"incarnation":38,"state":"up:rejoin","state_seq":55,"addr":"172.21.15.90:6835/226826781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":226826781},{"type":"v1","addr":"172.21.15.90:6835","nonce":226826781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3}]} 2024-03-24T01:30:39.572 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 40 2024-03-24T01:30:39.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.574+0000 7f8331a00640 1 -- 172.21.15.90:0/1587269000 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f830806bbd0 msgr2=0x7f830806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:39.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.574+0000 7f8331a00640 1 --2- 172.21.15.90:0/1587269000 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f830806bbd0 0x7f830806e090 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f831c009a10 tx=0x7f831c008040 comp rx=0 tx=0).stop 2024-03-24T01:30:39.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.574+0000 7f8331a00640 1 -- 172.21.15.90:0/1587269000 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f832c1e5b20 msgr2=0x7f832c1f53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:39.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.574+0000 7f8331a00640 1 --2- 172.21.15.90:0/1587269000 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f832c1e5b20 0x7f832c1f53e0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f8314008fd0 tx=0x7f8314001120 comp rx=0 tx=0).stop 2024-03-24T01:30:39.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.574+0000 7f8331a00640 1 -- 172.21.15.90:0/1587269000 shutdown_connections 2024-03-24T01:30:39.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.574+0000 7f8331a00640 1 --2- 172.21.15.90:0/1587269000 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f830806bbd0 0x7f830806e090 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:39.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.574+0000 7f8331a00640 1 --2- 172.21.15.90:0/1587269000 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f832c1e5b20 0x7f832c1f53e0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:39.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.574+0000 7f8331a00640 1 --2- 172.21.15.90:0/1587269000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f832c1f5920 0x7f832c14e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:39.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.574+0000 7f8331a00640 1 --2- 172.21.15.90:0/1587269000 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8310003f60 0x7f832c0a9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:39.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.574+0000 7f8331a00640 1 -- 172.21.15.90:0/1587269000 >> 172.21.15.90:0/1587269000 conn(0x7f832c06b2a0 msgr2=0x7f832c0b0410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:39.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.574+0000 7f8331a00640 1 -- 172.21.15.90:0/1587269000 shutdown_connections 2024-03-24T01:30:39.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:39.574+0000 7f8331a00640 1 -- 172.21.15.90:0/1587269000 wait complete. 2024-03-24T01:30:39.995 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 40, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T01:27:13.961373+0000', 'modified': '2024-03-24T01:30:38.226436+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6219}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6219': {'gid': 6219, 'name': 'c', 'rank': 0, 'incarnation': 38, 'state': 'up:rejoin', 'state_seq': 55, 'addr': '172.21.15.90:6835/226826781', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6833', 'nonce': 226826781}, {'type': 'v1', 'addr': '172.21.15.90:6835', 'nonce': 226826781}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-24T01:30:39.996 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6219:up:rejoin 2024-03-24T01:30:40.483 INFO:tasks.ceph.mds.c.smithi090.stderr:2024-03-24T01:30:40.482+0000 7f5b41400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T01:30:40.996 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:30:41.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.220+0000 7f4ddb400640 1 Processor -- start 2024-03-24T01:30:41.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.220+0000 7f4ddb400640 1 -- start start 2024-03-24T01:30:41.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.222+0000 7f4ddb400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4dd40cc150 0x7f4dd40ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:41.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.222+0000 7f4ddb400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4dd406b7d0 0x7f4dd40b3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:41.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.222+0000 7f4ddb400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd40cd490 con 0x7f4dd41603e0 2024-03-24T01:30:41.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.222+0000 7f4ddb400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dd415f900 con 0x7f4dd40cc150 2024-03-24T01:30:41.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.222+0000 7f4ddb400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd415fbe0 con 0x7f4dd406b7d0 2024-03-24T01:30:41.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.223+0000 7f4dd8a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4dd40cc150 0x7f4dd40ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:41.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.223+0000 7f4dcbe00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4dd41603e0 0x7f4dd406b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:34892/0 (socket says 172.21.15.90:34892) 2024-03-24T01:30:41.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.223+0000 7f4dcbe00640 1 -- 172.21.15.90:0/2187269048 learned_addr learned my addr 172.21.15.90:0/2187269048 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:30:41.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.223+0000 7f4dd9400640 1 --2- 172.21.15.90:0/2187269048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4dd406b7d0 0x7f4dd40b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:41.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.223+0000 7f4dd8a00640 1 -- 172.21.15.90:0/2187269048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4dd406b7d0 msgr2=0x7f4dd40b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:41.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.223+0000 7f4dd8a00640 1 --2- 172.21.15.90:0/2187269048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4dd406b7d0 0x7f4dd40b3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:41.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.223+0000 7f4dd8a00640 1 -- 172.21.15.90:0/2187269048 >> v1:172.21.15.90:6789/0 conn(0x7f4dd41603e0 legacy=0x7f4dd406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:41.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.223+0000 7f4dd3e00640 1 -- 172.21.15.90:0/2187269048 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1883343538 0 0) 0x7f4dd40cd490 con 0x7f4dd41603e0 2024-03-24T01:30:41.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.223+0000 7f4dd8a00640 1 -- 172.21.15.90:0/2187269048 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd415fec0 con 0x7f4dd40cc150 2024-03-24T01:30:41.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.223+0000 7f4dd9400640 1 --2- 172.21.15.90:0/2187269048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4dd406b7d0 0x7f4dd40b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:30:41.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.224+0000 7f4dd8a00640 1 --2- 172.21.15.90:0/2187269048 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4dd40cc150 0x7f4dd40ccb60 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f4dc40097a0 tx=0x7f4dc402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16c0c11eb68b08f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:41.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.224+0000 7f4dd3e00640 1 -- 172.21.15.90:0/2187269048 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc4035020 con 0x7f4dd40cc150 2024-03-24T01:30:41.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.225+0000 7f4dd3e00640 1 -- 172.21.15.90:0/2187269048 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc4031830 con 0x7f4dd40cc150 2024-03-24T01:30:41.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.225+0000 7f4dd3e00640 1 -- 172.21.15.90:0/2187269048 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc4031c00 con 0x7f4dd40cc150 2024-03-24T01:30:41.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.225+0000 7f4ddb400640 1 -- 172.21.15.90:0/2187269048 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4dd40cc150 msgr2=0x7f4dd40ccb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:41.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.225+0000 7f4ddb400640 1 --2- 172.21.15.90:0/2187269048 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4dd40cc150 0x7f4dd40ccb60 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f4dc40097a0 tx=0x7f4dc402b8d0 comp rx=0 tx=0).stop 2024-03-24T01:30:41.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.225+0000 7f4ddb400640 1 -- 172.21.15.90:0/2187269048 shutdown_connections 2024-03-24T01:30:41.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.225+0000 7f4ddb400640 1 --2- 172.21.15.90:0/2187269048 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4dd406b7d0 0x7f4dd40b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:41.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.225+0000 7f4ddb400640 1 --2- 172.21.15.90:0/2187269048 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4dd40cc150 0x7f4dd40ccb60 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:41.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.225+0000 7f4ddb400640 1 -- 172.21.15.90:0/2187269048 >> 172.21.15.90:0/2187269048 conn(0x7f4dd40b1770 msgr2=0x7f4dd40b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:41.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.225+0000 7f4ddb400640 1 -- 172.21.15.90:0/2187269048 shutdown_connections 2024-03-24T01:30:41.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.225+0000 7f4ddb400640 1 -- 172.21.15.90:0/2187269048 wait complete. 2024-03-24T01:30:41.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.227+0000 7f4ddb400640 1 Processor -- start 2024-03-24T01:30:41.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.227+0000 7f4ddb400640 1 -- start start 2024-03-24T01:30:41.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.228+0000 7f4ddb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4dd406b7d0 0x7f4dd41f7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:41.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.228+0000 7f4ddb400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4dd40cc150 0x7f4dd40cc530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:41.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.228+0000 7f4ddb400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4dd41603e0 0x7f4dd40cca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:41.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.228+0000 7f4ddb400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd41455b0 con 0x7f4dd406b7d0 2024-03-24T01:30:41.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.228+0000 7f4ddb400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd4145430 con 0x7f4dd40cc150 2024-03-24T01:30:41.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.228+0000 7f4ddb400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd40b3c90 con 0x7f4dd41603e0 2024-03-24T01:30:41.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.228+0000 7f4dd8a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4dd406b7d0 0x7f4dd41f7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:41.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.228+0000 7f4dd8a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4dd406b7d0 0x7f4dd41f7170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33122/0 (socket says 172.21.15.90:33122) 2024-03-24T01:30:41.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.228+0000 7f4dd8a00640 1 -- 172.21.15.90:0/970719182 learned_addr learned my addr 172.21.15.90:0/970719182 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:41.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.229+0000 7f4dcbe00640 1 --2- 172.21.15.90:0/970719182 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4dd40cc150 0x7f4dd40cc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:41.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.229+0000 7f4dd9400640 1 --2- 172.21.15.90:0/970719182 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4dd41603e0 0x7f4dd40cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:41.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.229+0000 7f4dd8a00640 1 -- 172.21.15.90:0/970719182 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4dd41603e0 msgr2=0x7f4dd40cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:41.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.229+0000 7f4dd8a00640 1 --2- 172.21.15.90:0/970719182 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4dd41603e0 0x7f4dd40cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:41.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.229+0000 7f4dd8a00640 1 -- 172.21.15.90:0/970719182 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4dd40cc150 msgr2=0x7f4dd40cc530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:41.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.229+0000 7f4dd8a00640 1 --2- 172.21.15.90:0/970719182 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4dd40cc150 0x7f4dd40cc530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:41.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.229+0000 7f4dd8a00640 1 -- 172.21.15.90:0/970719182 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd41e63d0 con 0x7f4dd406b7d0 2024-03-24T01:30:41.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.229+0000 7f4dcbe00640 1 --2- 172.21.15.90:0/970719182 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4dd40cc150 0x7f4dd40cc530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:30:41.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.229+0000 7f4dd8a00640 1 --2- 172.21.15.90:0/970719182 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4dd406b7d0 0x7f4dd41f7170 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f4dc402bee0 tx=0x7f4dc40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:41.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.229+0000 7f4dd1600640 1 -- 172.21.15.90:0/970719182 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc4041070 con 0x7f4dd406b7d0 2024-03-24T01:30:41.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.229+0000 7f4dd1600640 1 -- 172.21.15.90:0/970719182 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc403aa30 con 0x7f4dd406b7d0 2024-03-24T01:30:41.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.229+0000 7f4dd1600640 1 -- 172.21.15.90:0/970719182 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc4035040 con 0x7f4dd406b7d0 2024-03-24T01:30:41.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.230+0000 7f4ddb400640 1 -- 172.21.15.90:0/970719182 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dd41e66b0 con 0x7f4dd406b7d0 2024-03-24T01:30:41.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.230+0000 7f4ddb400640 1 -- 172.21.15.90:0/970719182 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dd41e6c70 con 0x7f4dd406b7d0 2024-03-24T01:30:41.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.231+0000 7f4ddb400640 1 -- 172.21.15.90:0/970719182 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dd4068060 con 0x7f4dd406b7d0 2024-03-24T01:30:41.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.232+0000 7f4dd1600640 1 -- 172.21.15.90:0/970719182 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4dc4038020 con 0x7f4dd406b7d0 2024-03-24T01:30:41.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.233+0000 7f4dd1600640 1 --2- 172.21.15.90:0/970719182 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4db006bbd0 0x7f4db006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:41.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.233+0000 7f4dd1600640 1 -- 172.21.15.90:0/970719182 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f4dc40ac880 con 0x7f4dd406b7d0 2024-03-24T01:30:41.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.234+0000 7f4dcbe00640 1 --2- 172.21.15.90:0/970719182 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4db006bbd0 0x7f4db006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:41.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.235+0000 7f4dcbe00640 1 --2- 172.21.15.90:0/970719182 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4db006bbd0 0x7f4db006e090 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4142990 tx=0x7f4dbc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:41.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.237+0000 7f4dd1600640 1 -- 172.21.15.90:0/970719182 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4dc407b0b0 con 0x7f4dd406b7d0 2024-03-24T01:30:41.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.498+0000 7f4ddb400640 1 -- 172.21.15.90:0/970719182 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4dd40b3220 con 0x7f4dd406b7d0 2024-03-24T01:30:41.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.499+0000 7f4dd1600640 1 -- 172.21.15.90:0/970719182 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f4dc407a9b0 con 0x7f4dd406b7d0 2024-03-24T01:30:41.499 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:30:41.499 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":41,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4985,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6834/3475131973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":3475131973},{"type":"v1","addr":"172.21.15.90:6834","nonce":3475131973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":4994,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/3151604088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":3151604088},{"type":"v1","addr":"172.21.15.120:6837","nonce":3151604088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5428,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/1239583677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1239583677},{"type":"v1","addr":"172.21.15.120:6836","nonce":1239583677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T01:27:13.961373+0000","modified":"2024-03-24T01:30:40.234926+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6219},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6219":{"gid":6219,"name":"c","rank":0,"incarnation":38,"state":"up:active","state_seq":56,"addr":"172.21.15.90:6835/226826781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":226826781},{"type":"v1","addr":"172.21.15.90:6835","nonce":226826781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6219,"qdb_cluster":[6219]},"id":3}]} 2024-03-24T01:30:41.500 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 41 2024-03-24T01:30:41.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.501+0000 7f4ddb400640 1 -- 172.21.15.90:0/970719182 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4db006bbd0 msgr2=0x7f4db006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:41.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.501+0000 7f4ddb400640 1 --2- 172.21.15.90:0/970719182 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4db006bbd0 0x7f4db006e090 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4142990 tx=0x7f4dbc009040 comp rx=0 tx=0).stop 2024-03-24T01:30:41.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.501+0000 7f4ddb400640 1 -- 172.21.15.90:0/970719182 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4dd406b7d0 msgr2=0x7f4dd41f7170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:41.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.501+0000 7f4ddb400640 1 --2- 172.21.15.90:0/970719182 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4dd406b7d0 0x7f4dd41f7170 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f4dc402bee0 tx=0x7f4dc40046c0 comp rx=0 tx=0).stop 2024-03-24T01:30:41.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.501+0000 7f4ddb400640 1 -- 172.21.15.90:0/970719182 shutdown_connections 2024-03-24T01:30:41.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.501+0000 7f4ddb400640 1 --2- 172.21.15.90:0/970719182 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4db006bbd0 0x7f4db006e090 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:41.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.501+0000 7f4ddb400640 1 --2- 172.21.15.90:0/970719182 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4dd41603e0 0x7f4dd40cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:41.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.501+0000 7f4ddb400640 1 --2- 172.21.15.90:0/970719182 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4dd40cc150 0x7f4dd40cc530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:41.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.501+0000 7f4ddb400640 1 --2- 172.21.15.90:0/970719182 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4dd406b7d0 0x7f4dd41f7170 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:41.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.502+0000 7f4ddb400640 1 -- 172.21.15.90:0/970719182 >> 172.21.15.90:0/970719182 conn(0x7f4dd40b1770 msgr2=0x7f4dd40aded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:41.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.502+0000 7f4ddb400640 1 -- 172.21.15.90:0/970719182 shutdown_connections 2024-03-24T01:30:41.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:41.502+0000 7f4ddb400640 1 -- 172.21.15.90:0/970719182 wait complete. 2024-03-24T01:30:41.922 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 41, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T01:27:13.961373+0000', 'modified': '2024-03-24T01:30:40.234926+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6219}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6219': {'gid': 6219, 'name': 'c', 'rank': 0, 'incarnation': 38, 'state': 'up:active', 'state_seq': 56, 'addr': '172.21.15.90:6835/226826781', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6833', 'nonce': 226826781}, {'type': 'v1', 'addr': '172.21.15.90:6835', 'nonce': 226826781}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6219, 'qdb_cluster': [6219]} 2024-03-24T01:30:41.923 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T01:30:41.923 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-24T01:30:42.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.129+0000 7ff541400640 1 Processor -- start 2024-03-24T01:30:42.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.129+0000 7ff541400640 1 -- start start 2024-03-24T01:30:42.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.131+0000 7ff541400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff53c15e020 0x7ff53c0ceab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:42.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.131+0000 7ff541400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff53c06cd00 0x7ff53c0d30d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:42.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.131+0000 7ff541400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff53c06d410 con 0x7ff53c0d2c40 2024-03-24T01:30:42.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.131+0000 7ff541400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff53c0d27b0 con 0x7ff53c15e020 2024-03-24T01:30:42.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.131+0000 7ff541400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c15e400 con 0x7ff53c06cd00 2024-03-24T01:30:42.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.131+0000 7ff53b400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff53c0d2c40 0x7ff53c06c5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:34896/0 (socket says 172.21.15.90:34896) 2024-03-24T01:30:42.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.131+0000 7ff53b400640 1 -- 172.21.15.90:0/3891806408 learned_addr learned my addr 172.21.15.90:0/3891806408 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:30:42.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.131+0000 7ff53be00640 1 --2- 172.21.15.90:0/3891806408 >> v2:172.21.15.120:3300/0 conn(0x7ff53c15e020 0x7ff53c0ceab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:42.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.131+0000 7ff540a00640 1 --2- 172.21.15.90:0/3891806408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff53c06cd00 0x7ff53c0d30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:42.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.131+0000 7ff53aa00640 1 -- 172.21.15.90:0/3891806408 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1535784967 0 0) 0x7ff53c06d410 con 0x7ff53c0d2c40 2024-03-24T01:30:42.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.132+0000 7ff53aa00640 1 -- 172.21.15.90:0/3891806408 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff520003520 con 0x7ff53c0d2c40 2024-03-24T01:30:42.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.132+0000 7ff53aa00640 1 -- 172.21.15.90:0/3891806408 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2520730854 0 0) 0x7ff520003520 con 0x7ff53c0d2c40 2024-03-24T01:30:42.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.132+0000 7ff53aa00640 1 -- 172.21.15.90:0/3891806408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff53c06cd00 msgr2=0x7ff53c0d30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:42.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.132+0000 7ff53aa00640 1 --2- 172.21.15.90:0/3891806408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff53c06cd00 0x7ff53c0d30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.132+0000 7ff53aa00640 1 -- 172.21.15.90:0/3891806408 >> v2:172.21.15.120:3300/0 conn(0x7ff53c15e020 msgr2=0x7ff53c0ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:42.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.132+0000 7ff53aa00640 1 --2- 172.21.15.90:0/3891806408 >> v2:172.21.15.120:3300/0 conn(0x7ff53c15e020 0x7ff53c0ceab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.132+0000 7ff53aa00640 1 -- 172.21.15.90:0/3891806408 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff53c0cf2e0 con 0x7ff53c0d2c40 2024-03-24T01:30:42.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.132+0000 7ff53aa00640 1 -- 172.21.15.90:0/3891806408 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7ff528001670 con 0x7ff53c0d2c40 2024-03-24T01:30:42.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.132+0000 7ff53aa00640 1 -- 172.21.15.90:0/3891806408 >> v1:172.21.15.90:6789/0 conn(0x7ff53c0d2c40 legacy=0x7ff53c06c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:42.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.132+0000 7ff53aa00640 1 --2- 172.21.15.90:0/3891806408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff53c06cd00 0x7ff5200039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:42.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.132+0000 7ff53aa00640 1 --2- 172.21.15.90:0/3891806408 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff520003f30 0x7ff520004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:42.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.132+0000 7ff53aa00640 1 --2- 172.21.15.90:0/3891806408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff520004d40 0x7ff5200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:42.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.132+0000 7ff53aa00640 1 -- 172.21.15.90:0/3891806408 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c0d27b0 con 0x7ff520003f30 2024-03-24T01:30:42.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.132+0000 7ff53aa00640 1 -- 172.21.15.90:0/3891806408 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c15e400 con 0x7ff520004d40 2024-03-24T01:30:42.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.132+0000 7ff53aa00640 1 -- 172.21.15.90:0/3891806408 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff52000ba10 con 0x7ff53c06cd00 2024-03-24T01:30:42.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.132+0000 7ff53b400640 1 --2- 172.21.15.90:0/3891806408 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff520003f30 0x7ff520004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:42.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.133+0000 7ff53be00640 1 --2- 172.21.15.90:0/3891806408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff53c06cd00 0x7ff5200039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:42.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.133+0000 7ff540a00640 1 --2- 172.21.15.90:0/3891806408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff520004d40 0x7ff5200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:42.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.133+0000 7ff53b400640 1 -- 172.21.15.90:0/3891806408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff53c06cd00 msgr2=0x7ff5200039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:42.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.133+0000 7ff53b400640 1 --2- 172.21.15.90:0/3891806408 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff53c06cd00 0x7ff5200039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.133+0000 7ff53b400640 1 -- 172.21.15.90:0/3891806408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff520004d40 msgr2=0x7ff5200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:42.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.133+0000 7ff53b400640 1 --2- 172.21.15.90:0/3891806408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff520004d40 0x7ff5200051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.133+0000 7ff53b400640 1 -- 172.21.15.90:0/3891806408 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff520003520 con 0x7ff520003f30 2024-03-24T01:30:42.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.134+0000 7ff53b400640 1 --2- 172.21.15.90:0/3891806408 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff520003f30 0x7ff520004800 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7ff52800c210 tx=0x7ff520005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c15980b6b0d9d59 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:42.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.134+0000 7ff53aa00640 1 -- 172.21.15.90:0/3891806408 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff528007d70 con 0x7ff520003f30 2024-03-24T01:30:42.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.134+0000 7ff53aa00640 1 -- 172.21.15.90:0/3891806408 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff528030070 con 0x7ff520003f30 2024-03-24T01:30:42.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.134+0000 7ff53aa00640 1 -- 172.21.15.90:0/3891806408 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52802cd60 con 0x7ff520003f30 2024-03-24T01:30:42.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.135+0000 7ff541400640 1 -- 172.21.15.90:0/3891806408 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff520003f30 msgr2=0x7ff520004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:42.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.135+0000 7ff541400640 1 --2- 172.21.15.90:0/3891806408 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff520003f30 0x7ff520004800 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7ff52800c210 tx=0x7ff520005830 comp rx=0 tx=0).stop 2024-03-24T01:30:42.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.135+0000 7ff540a00640 1 -- 172.21.15.90:0/3891806408 reap_dead start 2024-03-24T01:30:42.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.135+0000 7ff541400640 1 -- 172.21.15.90:0/3891806408 shutdown_connections 2024-03-24T01:30:42.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.135+0000 7ff541400640 1 -- 172.21.15.90:0/3891806408 >> 172.21.15.90:0/3891806408 conn(0x7ff53c0d0560 msgr2=0x7ff53c0d0970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:42.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.135+0000 7ff541400640 1 -- 172.21.15.90:0/3891806408 shutdown_connections 2024-03-24T01:30:42.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.135+0000 7ff541400640 1 -- 172.21.15.90:0/3891806408 wait complete. 2024-03-24T01:30:42.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.137+0000 7ff541400640 1 Processor -- start 2024-03-24T01:30:42.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.137+0000 7ff541400640 1 -- start start 2024-03-24T01:30:42.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.138+0000 7ff541400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff520003f30 0x7ff53c0c6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:42.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.138+0000 7ff541400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff53c0c8d00 0x7ff53c0c9160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:42.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.138+0000 7ff541400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff53c148f50 0x7ff53c1493d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:42.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.138+0000 7ff541400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c0ceff0 con 0x7ff53c0c8d00 2024-03-24T01:30:42.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.138+0000 7ff541400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c0d2440 con 0x7ff53c148f50 2024-03-24T01:30:42.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.138+0000 7ff541400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c0cf5c0 con 0x7ff520003f30 2024-03-24T01:30:42.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.138+0000 7ff53b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff53c0c8d00 0x7ff53c0c9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:42.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.138+0000 7ff53b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff53c0c8d00 0x7ff53c0c9160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33142/0 (socket says 172.21.15.90:33142) 2024-03-24T01:30:42.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.138+0000 7ff53b400640 1 -- 172.21.15.90:0/1177460321 learned_addr learned my addr 172.21.15.90:0/1177460321 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:42.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.138+0000 7ff53be00640 1 --2- 172.21.15.90:0/1177460321 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff520003f30 0x7ff53c0c6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:42.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.139+0000 7ff540a00640 1 --2- 172.21.15.90:0/1177460321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff53c148f50 0x7ff53c1493d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:42.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.139+0000 7ff53b400640 1 -- 172.21.15.90:0/1177460321 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff520003f30 msgr2=0x7ff53c0c6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:42.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.139+0000 7ff53b400640 1 --2- 172.21.15.90:0/1177460321 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff520003f30 0x7ff53c0c6d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.139+0000 7ff53b400640 1 -- 172.21.15.90:0/1177460321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff53c148f50 msgr2=0x7ff53c1493d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:42.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.139+0000 7ff53b400640 1 --2- 172.21.15.90:0/1177460321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff53c148f50 0x7ff53c1493d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.139+0000 7ff53b400640 1 -- 172.21.15.90:0/1177460321 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff53c149ce0 con 0x7ff53c0c8d00 2024-03-24T01:30:42.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.139+0000 7ff53b400640 1 --2- 172.21.15.90:0/1177460321 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff53c0c8d00 0x7ff53c0c9160 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7ff52800c330 tx=0x7ff528000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:42.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.139+0000 7ff533e00640 1 -- 172.21.15.90:0/1177460321 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff528004bf0 con 0x7ff53c0c8d00 2024-03-24T01:30:42.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.139+0000 7ff533e00640 1 -- 172.21.15.90:0/1177460321 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff528004d90 con 0x7ff53c0c8d00 2024-03-24T01:30:42.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.139+0000 7ff533e00640 1 -- 172.21.15.90:0/1177460321 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff528007290 con 0x7ff53c0c8d00 2024-03-24T01:30:42.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.139+0000 7ff541400640 1 -- 172.21.15.90:0/1177460321 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff53c146e00 con 0x7ff53c0c8d00 2024-03-24T01:30:42.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.139+0000 7ff541400640 1 -- 172.21.15.90:0/1177460321 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff53c1473c0 con 0x7ff53c0c8d00 2024-03-24T01:30:42.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.143+0000 7ff533e00640 1 -- 172.21.15.90:0/1177460321 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff528047020 con 0x7ff53c0c8d00 2024-03-24T01:30:42.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.144+0000 7ff545767740 1 Processor -- start 2024-03-24T01:30:42.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.144+0000 7ff533e00640 1 --2- 172.21.15.90:0/1177460321 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff51c06bbd0 0x7ff51c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:42.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.144+0000 7ff545767740 1 -- start start 2024-03-24T01:30:42.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.144+0000 7ff533e00640 1 -- 172.21.15.90:0/1177460321 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x7ff5280ac220 con 0x7ff53c0c8d00 2024-03-24T01:30:42.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.145+0000 7ff540a00640 1 --2- 172.21.15.90:0/1177460321 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff51c06bbd0 0x7ff51c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:42.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.145+0000 7ff545767740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565239406db0 0x565239407190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:42.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.145+0000 7ff545767740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5652393d6dd0 0x565239404020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:42.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.145+0000 7ff545767740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5652393da930 0x565239404560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:42.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.145+0000 7ff545767740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x565239340c90 con 0x5652393da930 2024-03-24T01:30:42.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.145+0000 7ff545767740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652392e9980 con 0x565239406db0 2024-03-24T01:30:42.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.145+0000 7ff545767740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x56523933cd30 con 0x5652393d6dd0 2024-03-24T01:30:42.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.146+0000 7ff540a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5652393da930 0x565239404560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:42.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.146+0000 7ff540a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5652393da930 0x565239404560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33156/0 (socket says 172.21.15.90:33156) 2024-03-24T01:30:42.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.146+0000 7ff540a00640 1 -- 172.21.15.90:0/493236163 learned_addr learned my addr 172.21.15.90:0/493236163 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:42.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.146+0000 7ff53be00640 1 --2- 172.21.15.90:0/493236163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565239406db0 0x565239407190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:42.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.146+0000 7ff53b400640 1 --2- 172.21.15.90:0/493236163 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5652393d6dd0 0x565239404020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:42.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.146+0000 7ff540a00640 1 -- 172.21.15.90:0/493236163 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5652393d6dd0 msgr2=0x565239404020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:42.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.146+0000 7ff540a00640 1 --2- 172.21.15.90:0/493236163 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5652393d6dd0 0x565239404020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.146+0000 7ff540a00640 1 -- 172.21.15.90:0/493236163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565239406db0 msgr2=0x565239407190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:42.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.146+0000 7ff540a00640 1 --2- 172.21.15.90:0/493236163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565239406db0 0x565239407190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.146+0000 7ff540a00640 1 -- 172.21.15.90:0/493236163 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565239301510 con 0x5652393da930 2024-03-24T01:30:42.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.146+0000 7ff53b400640 1 --2- 172.21.15.90:0/493236163 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5652393d6dd0 0x565239404020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:30:42.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.146+0000 7ff540a00640 1 --2- 172.21.15.90:0/493236163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5652393da930 0x565239404560 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7ff53c068a20 tx=0x7ff53401c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b4aae499ac9b0e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:42.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.147+0000 7ff540a00640 1 --2- 172.21.15.90:0/1177460321 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff51c06bbd0 0x7ff51c06e090 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff53c0687d0 tx=0x7ff5340199a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:42.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.147+0000 7ff527e00640 1 -- 172.21.15.90:0/493236163 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff534029070 con 0x5652393da930 2024-03-24T01:30:42.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.147+0000 7ff527e00640 1 -- 172.21.15.90:0/493236163 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff53401a910 con 0x5652393da930 2024-03-24T01:30:42.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.147+0000 7ff527e00640 1 -- 172.21.15.90:0/493236163 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff53401ace0 con 0x5652393da930 2024-03-24T01:30:42.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.147+0000 7ff545767740 1 -- 172.21.15.90:0/493236163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5652393da930 msgr2=0x565239404560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:42.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.147+0000 7ff545767740 1 --2- 172.21.15.90:0/493236163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5652393da930 0x565239404560 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7ff53c068a20 tx=0x7ff53401c000 comp rx=0 tx=0).stop 2024-03-24T01:30:42.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.147+0000 7ff545767740 1 -- 172.21.15.90:0/493236163 shutdown_connections 2024-03-24T01:30:42.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.147+0000 7ff545767740 1 --2- 172.21.15.90:0/493236163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5652393da930 0x565239404560 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.147+0000 7ff545767740 1 --2- 172.21.15.90:0/493236163 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5652393d6dd0 0x565239404020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.147+0000 7ff545767740 1 --2- 172.21.15.90:0/493236163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565239406db0 0x565239407190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.147+0000 7ff545767740 1 -- 172.21.15.90:0/493236163 >> 172.21.15.90:0/493236163 conn(0x5652393d9040 msgr2=0x565239421bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:42.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.148+0000 7ff545767740 1 -- 172.21.15.90:0/493236163 shutdown_connections 2024-03-24T01:30:42.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.148+0000 7ff545767740 1 -- 172.21.15.90:0/493236163 wait complete. 2024-03-24T01:30:42.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.148+0000 7ff545767740 1 -- start start 2024-03-24T01:30:42.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.149+0000 7ff545767740 1 Processor -- start 2024-03-24T01:30:42.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.149+0000 7ff545767740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5652393da930 0x565239421ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:42.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.150+0000 7ff545767740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x565239406db0 0x565239406490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:42.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.150+0000 7ff545767740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652393feda0 0x5652394050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:42.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.150+0000 7ff545767740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652393dbae0 con 0x5652393da930 2024-03-24T01:30:42.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.150+0000 7ff545767740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652392fcdd0 con 0x5652393feda0 2024-03-24T01:30:42.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.150+0000 7ff545767740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5652393daed0 con 0x565239406db0 2024-03-24T01:30:42.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.150+0000 7ff545767740 10 client.0 _ll_register_callbacks cb 0x565239054ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T01:30:42.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.150+0000 7ff53be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5652393da930 0x565239421ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:42.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.150+0000 7ff53be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5652393da930 0x565239421ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33164/0 (socket says 172.21.15.90:33164) 2024-03-24T01:30:42.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.150+0000 7ff53be00640 1 -- 172.21.15.90:0/2804192118 learned_addr learned my addr 172.21.15.90:0/2804192118 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:42.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.150+0000 7ff53be00640 1 -- 172.21.15.90:0/2804192118 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x565239406db0 msgr2=0x565239406490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:42.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.150+0000 7ff53be00640 1 --2- 172.21.15.90:0/2804192118 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x565239406db0 0x565239406490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.150+0000 7ff53be00640 1 -- 172.21.15.90:0/2804192118 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652393feda0 msgr2=0x5652394050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:42.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.150+0000 7ff540a00640 1 --2- 172.21.15.90:0/2804192118 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652393feda0 0x5652394050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:42.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.150+0000 7ff53be00640 1 --2- 172.21.15.90:0/2804192118 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652393feda0 0x5652394050c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.150+0000 7ff53be00640 1 -- 172.21.15.90:0/2804192118 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5652390552d0 con 0x5652393da930 2024-03-24T01:30:42.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.151+0000 7ff53be00640 1 --2- 172.21.15.90:0/2804192118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5652393da930 0x565239421ea0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7ff52c014600 tx=0x7ff52c014ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a63cf912c1172e8e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:42.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.151+0000 7ff524c00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-03-24T01:30:42.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.151+0000 7ff524c00640 1 -- 172.21.15.90:0/2804192118 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c012990 con 0x5652393da930 2024-03-24T01:30:42.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.151+0000 7ff524c00640 1 -- 172.21.15.90:0/2804192118 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x7ff52c012b30 con 0x5652393da930 2024-03-24T01:30:42.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.151+0000 7ff545767740 1 -- 172.21.15.90:0/2804192118 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff53401a910 con 0x5652393da930 2024-03-24T01:30:42.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.151+0000 7ff524c00640 1 -- 172.21.15.90:0/2804192118 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff52c012d30 con 0x5652393da930 2024-03-24T01:30:42.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.151+0000 7ff545767740 10 client.6870 fetch_fsmap learned FSMap version 41 2024-03-24T01:30:42.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.151+0000 7ff545767740 1 -- 172.21.15.90:0/2804192118 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=2+}) v3 -- 0x56523911f6b0 con 0x5652393da930 2024-03-24T01:30:42.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.151+0000 7ff524c00640 1 -- 172.21.15.90:0/2804192118 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff53401a910 con 0x5652393da930 2024-03-24T01:30:42.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.152+0000 7ff524c00640 1 -- 172.21.15.90:0/2804192118 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 41) v1 ==== 3314+0+0 (secure 0 0 0) 0x7ff52c02b020 con 0x5652393da930 2024-03-24T01:30:42.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.152+0000 7ff545767740 10 client.6870 fetch_fsmap finished waiting for FSMap version 41 2024-03-24T01:30:42.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.152+0000 7ff545767740 10 client.6870 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T01:30:42.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.152+0000 7ff545767740 20 client.6870 populate_metadata read hostname 'smithi090' 2024-03-24T01:30:42.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.152+0000 7ff545767740 1 --2- 172.21.15.90:0/2804192118 >> [v2:172.21.15.90:6833/226826781,v1:172.21.15.90:6835/226826781] conn(0x5652394353c0 0x5652393ef070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:42.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.152+0000 7ff545767740 4 client.6870 mds_command: new command op to 6219 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T01:30:42.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.152+0000 7ff545767740 1 -- 172.21.15.90:0/2804192118 --> [v2:172.21.15.90:6833/226826781,v1:172.21.15.90:6835/226826781] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5652392e2520 con 0x5652394353c0 2024-03-24T01:30:42.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.152+0000 7ff53b400640 1 --2- 172.21.15.90:0/2804192118 >> [v2:172.21.15.90:6833/226826781,v1:172.21.15.90:6835/226826781] conn(0x5652394353c0 0x5652393ef070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:42.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.152+0000 7ff524c00640 10 client.6870 ms_handle_connect on v2:172.21.15.90:6833/226826781 2024-03-24T01:30:42.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.152+0000 7ff53b400640 1 --2- 172.21.15.90:0/2804192118 >> [v2:172.21.15.90:6833/226826781,v1:172.21.15.90:6835/226826781] conn(0x5652394353c0 0x5652393ef070 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7da154cddf09751 server_cookie=b38cf5ae0879d3a9 in_seq=0 out_seq=0 2024-03-24T01:30:42.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.154+0000 7ff524c00640 1 -- 172.21.15.90:0/2804192118 <== mds.0 v2:172.21.15.90:6833/226826781 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5652392e2520 con 0x5652394353c0 2024-03-24T01:30:42.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.154+0000 7ff524c00640 10 client.6870 handle_command_reply: tid=0 2024-03-24T01:30:42.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.154+0000 7ff545767740 1 client.6870 shutdown 2024-03-24T01:30:42.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.154+0000 7ff545767740 1 -- 172.21.15.90:0/2804192118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5652393da930 msgr2=0x565239421ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:42.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.154+0000 7ff545767740 1 --2- 172.21.15.90:0/2804192118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5652393da930 0x565239421ea0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7ff52c014600 tx=0x7ff52c014ac0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.154+0000 7ff545767740 1 -- 172.21.15.90:0/2804192118 shutdown_connections 2024-03-24T01:30:42.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.154+0000 7ff545767740 1 --2- 172.21.15.90:0/2804192118 >> [v2:172.21.15.90:6833/226826781,v1:172.21.15.90:6835/226826781] conn(0x5652394353c0 0x5652393ef070 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.154+0000 7ff545767740 1 --2- 172.21.15.90:0/2804192118 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652393feda0 0x5652394050c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.154+0000 7ff545767740 1 --2- 172.21.15.90:0/2804192118 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x565239406db0 0x565239406490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.154+0000 7ff545767740 1 --2- 172.21.15.90:0/2804192118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5652393da930 0x565239421ea0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.154+0000 7ff524c00640 0 client.6870 ms_handle_reset on v2:172.21.15.90:6833/226826781 2024-03-24T01:30:42.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.154+0000 7ff545767740 1 -- 172.21.15.90:0/2804192118 >> 172.21.15.90:0/2804192118 conn(0x5652393d6dd0 msgr2=0x56523940aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:42.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.154+0000 7ff545767740 1 -- 172.21.15.90:0/2804192118 shutdown_connections 2024-03-24T01:30:42.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.154+0000 7ff545767740 1 -- 172.21.15.90:0/2804192118 wait complete. 2024-03-24T01:30:42.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.154+0000 7ff545767740 20 client.6870 trim_cache size 0 max 16384 2024-03-24T01:30:42.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.174+0000 7ff545767740 1 Processor -- start 2024-03-24T01:30:42.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.174+0000 7ff545767740 1 -- start start 2024-03-24T01:30:42.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.175+0000 7ff545767740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5652393da930 0x5652394252c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:42.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.175+0000 7ff545767740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652393feda0 0x565239403d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:42.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.175+0000 7ff545767740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x565239406db0 0x565239404290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:42.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.175+0000 7ff545767740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c1ffdc0 con 0x5652393da930 2024-03-24T01:30:42.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.175+0000 7ff545767740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c148ac0 con 0x5652393feda0 2024-03-24T01:30:42.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.175+0000 7ff545767740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c1ff7c0 con 0x565239406db0 2024-03-24T01:30:42.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.175+0000 7ff53be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5652393da930 0x5652394252c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:42.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.175+0000 7ff53be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5652393da930 0x5652394252c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33170/0 (socket says 172.21.15.90:33170) 2024-03-24T01:30:42.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.175+0000 7ff53be00640 1 -- 172.21.15.90:0/3711877125 learned_addr learned my addr 172.21.15.90:0/3711877125 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:42.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.176+0000 7ff53be00640 1 -- 172.21.15.90:0/3711877125 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x565239406db0 msgr2=0x565239404290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:42.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.176+0000 7ff53be00640 1 --2- 172.21.15.90:0/3711877125 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x565239406db0 0x565239404290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.176+0000 7ff53be00640 1 -- 172.21.15.90:0/3711877125 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652393feda0 msgr2=0x565239403d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:42.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.176+0000 7ff53be00640 1 --2- 172.21.15.90:0/3711877125 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652393feda0 0x565239403d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.176+0000 7ff53be00640 1 -- 172.21.15.90:0/3711877125 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652393fc960 con 0x5652393da930 2024-03-24T01:30:42.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.176+0000 7ff53be00640 1 --2- 172.21.15.90:0/3711877125 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5652393da930 0x5652394252c0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7ff52c000c00 tx=0x7ff52c01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cad000e910a72e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:42.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.176+0000 7ff527400640 1 -- 172.21.15.90:0/3711877125 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c0165a0 con 0x5652393da930 2024-03-24T01:30:42.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.176+0000 7ff527400640 1 -- 172.21.15.90:0/3711877125 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff52c031070 con 0x5652393da930 2024-03-24T01:30:42.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.176+0000 7ff527400640 1 -- 172.21.15.90:0/3711877125 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c0169a0 con 0x5652393da930 2024-03-24T01:30:42.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.177+0000 7ff545767740 1 -- 172.21.15.90:0/3711877125 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5652393da930 msgr2=0x5652394252c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:42.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.177+0000 7ff545767740 1 --2- 172.21.15.90:0/3711877125 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5652393da930 0x5652394252c0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7ff52c000c00 tx=0x7ff52c01a000 comp rx=0 tx=0).stop 2024-03-24T01:30:42.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.177+0000 7ff545767740 1 -- 172.21.15.90:0/3711877125 shutdown_connections 2024-03-24T01:30:42.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.177+0000 7ff545767740 1 --2- 172.21.15.90:0/3711877125 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x565239406db0 0x565239404290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.177+0000 7ff545767740 1 --2- 172.21.15.90:0/3711877125 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652393feda0 0x565239403d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.177+0000 7ff545767740 1 --2- 172.21.15.90:0/3711877125 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5652393da930 0x5652394252c0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.177+0000 7ff545767740 1 -- 172.21.15.90:0/3711877125 >> 172.21.15.90:0/3711877125 conn(0x5652393d6dd0 msgr2=0x5652393db110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:42.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.177+0000 7ff545767740 1 -- 172.21.15.90:0/3711877125 shutdown_connections 2024-03-24T01:30:42.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.177+0000 7ff545767740 1 -- 172.21.15.90:0/3711877125 wait complete. 2024-03-24T01:30:42.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.177+0000 7ff545767740 1 -- start start 2024-03-24T01:30:42.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.178+0000 7ff545767740 1 Processor -- start 2024-03-24T01:30:42.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.178+0000 7ff545767740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5652393da930 0x565239476fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:42.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.179+0000 7ff545767740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652393feda0 0x5652393f0990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:42.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.179+0000 7ff545767740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565239406db0 0x5652393f0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:42.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.179+0000 7ff545767740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652393f01e0 con 0x565239406db0 2024-03-24T01:30:42.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.179+0000 7ff545767740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652393ffec0 con 0x5652393feda0 2024-03-24T01:30:42.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.179+0000 7ff545767740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5652394357e0 con 0x5652393da930 2024-03-24T01:30:42.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.179+0000 7ff545767740 10 client.0 _ll_register_callbacks cb 0x7ff52c009540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T01:30:42.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.179+0000 7ff540a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565239406db0 0x5652393f0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:42.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.179+0000 7ff540a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565239406db0 0x5652393f0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33186/0 (socket says 172.21.15.90:33186) 2024-03-24T01:30:42.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.179+0000 7ff540a00640 1 -- 172.21.15.90:0/2064971614 learned_addr learned my addr 172.21.15.90:0/2064971614 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:42.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.179+0000 7ff540a00640 1 -- 172.21.15.90:0/2064971614 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5652393da930 msgr2=0x565239476fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:42.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.179+0000 7ff540a00640 1 --2- 172.21.15.90:0/2064971614 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5652393da930 0x565239476fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.179+0000 7ff540a00640 1 -- 172.21.15.90:0/2064971614 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652393feda0 msgr2=0x5652393f0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:42.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.179+0000 7ff540a00640 1 --2- 172.21.15.90:0/2064971614 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652393feda0 0x5652393f0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.179+0000 7ff540a00640 1 -- 172.21.15.90:0/2064971614 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5652394218d0 con 0x565239406db0 2024-03-24T01:30:42.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.179+0000 7ff540a00640 1 --2- 172.21.15.90:0/2064971614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565239406db0 0x5652393f0ed0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7ff53401bef0 tx=0x7ff534004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d0c416a6b0630cc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:42.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.180+0000 7ff524c00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-03-24T01:30:42.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.180+0000 7ff524c00640 1 -- 172.21.15.90:0/2064971614 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff534029070 con 0x565239406db0 2024-03-24T01:30:42.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.180+0000 7ff545767740 1 -- 172.21.15.90:0/2064971614 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff52c031070 con 0x565239406db0 2024-03-24T01:30:42.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.180+0000 7ff524c00640 1 -- 172.21.15.90:0/2064971614 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x7ff53402e440 con 0x565239406db0 2024-03-24T01:30:42.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.180+0000 7ff524c00640 1 -- 172.21.15.90:0/2064971614 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff53403b050 con 0x565239406db0 2024-03-24T01:30:42.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.180+0000 7ff545767740 10 client.6876 fetch_fsmap learned FSMap version 41 2024-03-24T01:30:42.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.180+0000 7ff545767740 1 -- 172.21.15.90:0/2064971614 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=2+}) v3 -- 0x565239406230 con 0x565239406db0 2024-03-24T01:30:42.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.180+0000 7ff524c00640 1 -- 172.21.15.90:0/2064971614 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff52c031070 con 0x565239406db0 2024-03-24T01:30:42.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.180+0000 7ff524c00640 1 -- 172.21.15.90:0/2064971614 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 41) v1 ==== 3314+0+0 (secure 0 0 0) 0x7ff53403e020 con 0x565239406db0 2024-03-24T01:30:42.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.181+0000 7ff545767740 10 client.6876 fetch_fsmap finished waiting for FSMap version 41 2024-03-24T01:30:42.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.181+0000 7ff545767740 10 client.6876 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T01:30:42.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.181+0000 7ff545767740 20 client.6876 populate_metadata read hostname 'smithi090' 2024-03-24T01:30:42.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.181+0000 7ff545767740 1 --2- 172.21.15.90:0/2064971614 >> [v2:172.21.15.90:6833/226826781,v1:172.21.15.90:6835/226826781] conn(0x5652394224a0 0x5652393d5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:42.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.181+0000 7ff545767740 4 client.6876 mds_command: new command op to 6219 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T01:30:42.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.181+0000 7ff545767740 1 -- 172.21.15.90:0/2064971614 --> [v2:172.21.15.90:6833/226826781,v1:172.21.15.90:6835/226826781] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56523934e1d0 con 0x5652394224a0 2024-03-24T01:30:42.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.181+0000 7ff53be00640 1 --2- 172.21.15.90:0/2064971614 >> [v2:172.21.15.90:6833/226826781,v1:172.21.15.90:6835/226826781] conn(0x5652394224a0 0x5652393d5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:42.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.181+0000 7ff524c00640 10 client.6876 ms_handle_connect on v2:172.21.15.90:6833/226826781 2024-03-24T01:30:42.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.181+0000 7ff53be00640 1 --2- 172.21.15.90:0/2064971614 >> [v2:172.21.15.90:6833/226826781,v1:172.21.15.90:6835/226826781] conn(0x5652394224a0 0x5652393d5c60 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3f89edc9f2b769f server_cookie=51793a2315cc1149 in_seq=0 out_seq=0 2024-03-24T01:30:42.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.182+0000 7ff524c00640 1 -- 172.21.15.90:0/2064971614 <== mds.0 v2:172.21.15.90:6833/226826781 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x56523934e1d0 con 0x5652394224a0 2024-03-24T01:30:42.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.182+0000 7ff524c00640 10 client.6876 handle_command_reply: tid=0 2024-03-24T01:30:42.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.182+0000 7ff545767740 1 client.6876 shutdown 2024-03-24T01:30:42.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.182+0000 7ff545767740 1 -- 172.21.15.90:0/2064971614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565239406db0 msgr2=0x5652393f0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:42.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.182+0000 7ff545767740 1 --2- 172.21.15.90:0/2064971614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565239406db0 0x5652393f0ed0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7ff53401bef0 tx=0x7ff534004ab0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.182+0000 7ff545767740 1 -- 172.21.15.90:0/2064971614 shutdown_connections 2024-03-24T01:30:42.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.182+0000 7ff545767740 1 --2- 172.21.15.90:0/2064971614 >> [v2:172.21.15.90:6833/226826781,v1:172.21.15.90:6835/226826781] conn(0x5652394224a0 0x5652393d5c60 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.182+0000 7ff545767740 1 --2- 172.21.15.90:0/2064971614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565239406db0 0x5652393f0ed0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.182+0000 7ff545767740 1 --2- 172.21.15.90:0/2064971614 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652393feda0 0x5652393f0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.182+0000 7ff545767740 1 --2- 172.21.15.90:0/2064971614 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5652393da930 0x565239476fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.182+0000 7ff545767740 1 -- 172.21.15.90:0/2064971614 >> 172.21.15.90:0/2064971614 conn(0x5652393d6dd0 msgr2=0x5652393db0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:42.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.182+0000 7ff524c00640 0 client.6876 ms_handle_reset on v2:172.21.15.90:6833/226826781 2024-03-24T01:30:42.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.182+0000 7ff545767740 1 -- 172.21.15.90:0/2064971614 shutdown_connections 2024-03-24T01:30:42.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.182+0000 7ff545767740 1 -- 172.21.15.90:0/2064971614 wait complete. 2024-03-24T01:30:42.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.182+0000 7ff545767740 20 client.6876 trim_cache size 0 max 16384 2024-03-24T01:30:42.186 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-24T01:30:42.186 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "24712165-9839-4fcc-8033-988386efea74", 2024-03-24T01:30:42.186 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 0, 2024-03-24T01:30:42.186 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 6219, 2024-03-24T01:30:42.186 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-03-24T01:30:42.186 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-03-24T01:30:42.186 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-03-24T01:30:42.186 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 4.9688022810000003, 2024-03-24T01:30:42.186 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 41, 2024-03-24T01:30:42.187 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 56, 2024-03-24T01:30:42.187 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 56, 2024-03-24T01:30:42.187 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 213.08276414400001 2024-03-24T01:30:42.187 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-24T01:30:42.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.183+0000 7ff541400640 1 -- 172.21.15.90:0/1177460321 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff51c06bbd0 msgr2=0x7ff51c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:42.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.183+0000 7ff541400640 1 --2- 172.21.15.90:0/1177460321 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff51c06bbd0 0x7ff51c06e090 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff53c0687d0 tx=0x7ff5340199a0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.183+0000 7ff541400640 1 -- 172.21.15.90:0/1177460321 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff53c0c8d00 msgr2=0x7ff53c0c9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:42.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.183+0000 7ff541400640 1 --2- 172.21.15.90:0/1177460321 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff53c0c8d00 0x7ff53c0c9160 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7ff52800c330 tx=0x7ff528000fe0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.183+0000 7ff541400640 1 -- 172.21.15.90:0/1177460321 shutdown_connections 2024-03-24T01:30:42.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.183+0000 7ff541400640 1 --2- 172.21.15.90:0/1177460321 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff51c06bbd0 0x7ff51c06e090 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.183+0000 7ff541400640 1 --2- 172.21.15.90:0/1177460321 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff53c0c8d00 0x7ff53c0c9160 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.183+0000 7ff541400640 1 --2- 172.21.15.90:0/1177460321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff53c148f50 0x7ff53c1493d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.183+0000 7ff541400640 1 --2- 172.21.15.90:0/1177460321 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff520003f30 0x7ff53c0c6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.183+0000 7ff541400640 1 -- 172.21.15.90:0/1177460321 >> 172.21.15.90:0/1177460321 conn(0x7ff53c0d0560 msgr2=0x7ff53c0b8ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:42.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.183+0000 7ff541400640 1 -- 172.21.15.90:0/1177460321 shutdown_connections 2024-03-24T01:30:42.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.183+0000 7ff541400640 1 -- 172.21.15.90:0/1177460321 wait complete. 2024-03-24T01:30:42.198 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T01:30:42.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.390+0000 7fe6fe600640 1 Processor -- start 2024-03-24T01:30:42.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.390+0000 7fe6fe600640 1 -- start start 2024-03-24T01:30:42.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.391+0000 7fe6fe600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe6f80cc150 0x7fe6f80ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:42.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.391+0000 7fe6fe600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6f81603e0 0x7fe6f806b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:42.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.391+0000 7fe6fe600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f80cd490 con 0x7fe6f8145430 2024-03-24T01:30:42.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.391+0000 7fe6fe600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6f815f900 con 0x7fe6f80cc150 2024-03-24T01:30:42.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.391+0000 7fe6fe600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f815fbe0 con 0x7fe6f81603e0 2024-03-24T01:30:42.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.392+0000 7fe6f7e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe6f8145430 0x7fe6f8071990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:34898/0 (socket says 172.21.15.90:34898) 2024-03-24T01:30:42.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.392+0000 7fe6f7e00640 1 -- 172.21.15.90:0/3794070315 learned_addr learned my addr 172.21.15.90:0/3794070315 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:30:42.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.392+0000 7fe6efe00640 1 --2- 172.21.15.90:0/3794070315 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6f81603e0 0x7fe6f806b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:42.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.392+0000 7fe6f7400640 1 --2- 172.21.15.90:0/3794070315 >> v2:172.21.15.120:3300/0 conn(0x7fe6f80cc150 0x7fe6f80ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:42.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.392+0000 7fe6f6a00640 1 -- 172.21.15.90:0/3794070315 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 586397093 0 0) 0x7fe6f80cd490 con 0x7fe6f8145430 2024-03-24T01:30:42.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.392+0000 7fe6f6a00640 1 -- 172.21.15.90:0/3794070315 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6dc003520 con 0x7fe6f8145430 2024-03-24T01:30:42.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.393+0000 7fe6f6a00640 1 -- 172.21.15.90:0/3794070315 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 690461720 0 0) 0x7fe6dc003520 con 0x7fe6f8145430 2024-03-24T01:30:42.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.393+0000 7fe6f6a00640 1 -- 172.21.15.90:0/3794070315 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6f81603e0 msgr2=0x7fe6f806b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:42.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.393+0000 7fe6f6a00640 1 --2- 172.21.15.90:0/3794070315 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6f81603e0 0x7fe6f806b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.393+0000 7fe6f6a00640 1 -- 172.21.15.90:0/3794070315 >> v2:172.21.15.120:3300/0 conn(0x7fe6f80cc150 msgr2=0x7fe6f80ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:42.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.393+0000 7fe6f6a00640 1 --2- 172.21.15.90:0/3794070315 >> v2:172.21.15.120:3300/0 conn(0x7fe6f80cc150 0x7fe6f80ccb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.393+0000 7fe6f6a00640 1 -- 172.21.15.90:0/3794070315 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f815fec0 con 0x7fe6f8145430 2024-03-24T01:30:42.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.393+0000 7fe6f6a00640 1 -- 172.21.15.90:0/3794070315 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7fe6f00019f0 con 0x7fe6f8145430 2024-03-24T01:30:42.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.393+0000 7fe6f6a00640 1 -- 172.21.15.90:0/3794070315 >> v1:172.21.15.90:6789/0 conn(0x7fe6f8145430 legacy=0x7fe6f8071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:42.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.393+0000 7fe6f6a00640 1 --2- 172.21.15.90:0/3794070315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6dc0039b0 0x7fe6dc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:42.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.393+0000 7fe6f6a00640 1 --2- 172.21.15.90:0/3794070315 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6f81603e0 0x7fe6dc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:42.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.393+0000 7fe6f6a00640 1 --2- 172.21.15.90:0/3794070315 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6dc004d40 0x7fe6dc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:42.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.393+0000 7fe6f6a00640 1 -- 172.21.15.90:0/3794070315 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f815f900 con 0x7fe6dc004d40 2024-03-24T01:30:42.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.393+0000 7fe6f6a00640 1 -- 172.21.15.90:0/3794070315 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f815fbe0 con 0x7fe6dc0039b0 2024-03-24T01:30:42.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.393+0000 7fe6f6a00640 1 -- 172.21.15.90:0/3794070315 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6dc00b9c0 con 0x7fe6f81603e0 2024-03-24T01:30:42.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.393+0000 7fe6f7e00640 1 --2- 172.21.15.90:0/3794070315 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6dc004d40 0x7fe6dc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:42.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.394+0000 7fe6f7400640 1 --2- 172.21.15.90:0/3794070315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6dc0039b0 0x7fe6dc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:42.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.394+0000 7fe6efe00640 1 --2- 172.21.15.90:0/3794070315 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6f81603e0 0x7fe6dc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:42.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.394+0000 7fe6f7e00640 1 -- 172.21.15.90:0/3794070315 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6f81603e0 msgr2=0x7fe6dc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:42.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.394+0000 7fe6f7e00640 1 --2- 172.21.15.90:0/3794070315 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6f81603e0 0x7fe6dc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.394+0000 7fe6f7e00640 1 -- 172.21.15.90:0/3794070315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6dc0039b0 msgr2=0x7fe6dc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:30:42.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.394+0000 7fe6f7e00640 1 --2- 172.21.15.90:0/3794070315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6dc0039b0 0x7fe6dc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.394+0000 7fe6f7e00640 1 -- 172.21.15.90:0/3794070315 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6dc003520 con 0x7fe6dc004d40 2024-03-24T01:30:42.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.395+0000 7fe6f7e00640 1 --2- 172.21.15.90:0/3794070315 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6dc004d40 0x7fe6dc0051c0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fe6f000a8c0 tx=0x7fe6dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e58a57e8251ef9f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:42.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.396+0000 7fe6f6a00640 1 -- 172.21.15.90:0/3794070315 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f000a4e0 con 0x7fe6dc004d40 2024-03-24T01:30:42.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.396+0000 7fe6f6a00640 1 -- 172.21.15.90:0/3794070315 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f0034650 con 0x7fe6dc004d40 2024-03-24T01:30:42.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.396+0000 7fe6f6a00640 1 -- 172.21.15.90:0/3794070315 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f0034a40 con 0x7fe6dc004d40 2024-03-24T01:30:42.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.396+0000 7fe6fe600640 1 -- 172.21.15.90:0/3794070315 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6dc004d40 msgr2=0x7fe6dc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:42.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.396+0000 7fe6fe600640 1 --2- 172.21.15.90:0/3794070315 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6dc004d40 0x7fe6dc0051c0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fe6f000a8c0 tx=0x7fe6dc005830 comp rx=0 tx=0).stop 2024-03-24T01:30:42.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.396+0000 7fe6f7e00640 1 -- 172.21.15.90:0/3794070315 reap_dead start 2024-03-24T01:30:42.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.396+0000 7fe6fe600640 1 -- 172.21.15.90:0/3794070315 shutdown_connections 2024-03-24T01:30:42.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.396+0000 7fe6fe600640 1 -- 172.21.15.90:0/3794070315 >> 172.21.15.90:0/3794070315 conn(0x7fe6f80b1770 msgr2=0x7fe6f80b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:42.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.397+0000 7fe6fe600640 1 -- 172.21.15.90:0/3794070315 shutdown_connections 2024-03-24T01:30:42.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.397+0000 7fe6fe600640 1 -- 172.21.15.90:0/3794070315 wait complete. 2024-03-24T01:30:42.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.398+0000 7fe6fe600640 1 Processor -- start 2024-03-24T01:30:42.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.399+0000 7fe6fe600640 1 -- start start 2024-03-24T01:30:42.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.399+0000 7fe6fe600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6dc004d40 0x7fe6f813d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:42.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.399+0000 7fe6fe600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6f81e66d0 0x7fe6f813f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:42.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.399+0000 7fe6fe600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6f81e6b50 0x7fe6f813f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:42.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.400+0000 7fe6fe600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f8072220 con 0x7fe6f81e66d0 2024-03-24T01:30:42.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.400+0000 7fe6fe600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f80720a0 con 0x7fe6f81e6b50 2024-03-24T01:30:42.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.400+0000 7fe6fe600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f80723a0 con 0x7fe6dc004d40 2024-03-24T01:30:42.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.400+0000 7fe6efe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6f81e66d0 0x7fe6f813f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:42.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.400+0000 7fe6efe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6f81e66d0 0x7fe6f813f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33208/0 (socket says 172.21.15.90:33208) 2024-03-24T01:30:42.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.400+0000 7fe6efe00640 1 -- 172.21.15.90:0/3139849536 learned_addr learned my addr 172.21.15.90:0/3139849536 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:30:42.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.400+0000 7fe6f7400640 1 --2- 172.21.15.90:0/3139849536 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6dc004d40 0x7fe6f813d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:42.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.400+0000 7fe6f7e00640 1 --2- 172.21.15.90:0/3139849536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6f81e6b50 0x7fe6f813f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:42.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.400+0000 7fe6efe00640 1 -- 172.21.15.90:0/3139849536 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6dc004d40 msgr2=0x7fe6f813d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:42.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.400+0000 7fe6efe00640 1 --2- 172.21.15.90:0/3139849536 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6dc004d40 0x7fe6f813d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.400+0000 7fe6efe00640 1 -- 172.21.15.90:0/3139849536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6f81e6b50 msgr2=0x7fe6f813f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:42.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.400+0000 7fe6efe00640 1 --2- 172.21.15.90:0/3139849536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6f81e6b50 0x7fe6f813f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.400+0000 7fe6efe00640 1 -- 172.21.15.90:0/3139849536 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f81e7040 con 0x7fe6f81e66d0 2024-03-24T01:30:42.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.401+0000 7fe6efe00640 1 --2- 172.21.15.90:0/3139849536 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6f81e66d0 0x7fe6f813f010 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fe6e0011600 tx=0x7fe6e0011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:42.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.401+0000 7fe6ef400640 1 -- 172.21.15.90:0/3139849536 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e000f920 con 0x7fe6f81e66d0 2024-03-24T01:30:42.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.401+0000 7fe6ef400640 1 -- 172.21.15.90:0/3139849536 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6e000fac0 con 0x7fe6f81e66d0 2024-03-24T01:30:42.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.401+0000 7fe6ef400640 1 -- 172.21.15.90:0/3139849536 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e000fe70 con 0x7fe6f81e66d0 2024-03-24T01:30:42.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.401+0000 7fe6fe600640 1 -- 172.21.15.90:0/3139849536 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6f81e7320 con 0x7fe6f81e66d0 2024-03-24T01:30:42.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.401+0000 7fe6fe600640 1 -- 172.21.15.90:0/3139849536 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6f81e78e0 con 0x7fe6f81e66d0 2024-03-24T01:30:42.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.402+0000 7fe6fe600640 1 -- 172.21.15.90:0/3139849536 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6f8068060 con 0x7fe6f81e66d0 2024-03-24T01:30:42.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.404+0000 7fe6ef400640 1 -- 172.21.15.90:0/3139849536 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe6e0015460 con 0x7fe6f81e66d0 2024-03-24T01:30:42.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.405+0000 7fe6ef400640 1 --2- 172.21.15.90:0/3139849536 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe6d406bbd0 0x7fe6d406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:30:42.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.405+0000 7fe6ef400640 1 -- 172.21.15.90:0/3139849536 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fe6e008b650 con 0x7fe6f81e66d0 2024-03-24T01:30:42.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.406+0000 7fe6f7400640 1 --2- 172.21.15.90:0/3139849536 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe6d406bbd0 0x7fe6d406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:30:42.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.407+0000 7fe6f7400640 1 --2- 172.21.15.90:0/3139849536 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe6d406bbd0 0x7fe6d406e090 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe6f80af040 tx=0x7fe6e8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:30:42.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.410+0000 7fe6ef400640 1 -- 172.21.15.90:0/3139849536 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe6e0019030 con 0x7fe6f81e66d0 2024-03-24T01:30:42.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.648+0000 7fe6fe600640 1 -- 172.21.15.90:0/3139849536 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe6f815f080 con 0x7fe6f81e66d0 2024-03-24T01:30:42.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.649+0000 7fe6ef400640 1 -- 172.21.15.90:0/3139849536 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v41) v1 ==== 53+0+73 (secure 0 0 0) 0x7fe6e0059f20 con 0x7fe6f81e66d0 2024-03-24T01:30:42.649 INFO:teuthology.orchestra.run.smithi090.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T01:30:42.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.651+0000 7fe6fe600640 1 -- 172.21.15.90:0/3139849536 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe6d406bbd0 msgr2=0x7fe6d406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:42.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.651+0000 7fe6fe600640 1 --2- 172.21.15.90:0/3139849536 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe6d406bbd0 0x7fe6d406e090 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe6f80af040 tx=0x7fe6e8008040 comp rx=0 tx=0).stop 2024-03-24T01:30:42.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.651+0000 7fe6fe600640 1 -- 172.21.15.90:0/3139849536 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6f81e66d0 msgr2=0x7fe6f813f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:30:42.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.651+0000 7fe6fe600640 1 --2- 172.21.15.90:0/3139849536 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6f81e66d0 0x7fe6f813f010 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fe6e0011600 tx=0x7fe6e0011ac0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.651+0000 7fe6fe600640 1 -- 172.21.15.90:0/3139849536 shutdown_connections 2024-03-24T01:30:42.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.651+0000 7fe6fe600640 1 --2- 172.21.15.90:0/3139849536 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe6d406bbd0 0x7fe6d406e090 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.651+0000 7fe6fe600640 1 --2- 172.21.15.90:0/3139849536 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6f81e66d0 0x7fe6f813f010 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.651+0000 7fe6fe600640 1 --2- 172.21.15.90:0/3139849536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6f81e6b50 0x7fe6f813f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.651+0000 7fe6fe600640 1 --2- 172.21.15.90:0/3139849536 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe6dc004d40 0x7fe6f813d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:30:42.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.651+0000 7fe6fe600640 1 -- 172.21.15.90:0/3139849536 >> 172.21.15.90:0/3139849536 conn(0x7fe6f80b1770 msgr2=0x7fe6f806b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:30:42.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.651+0000 7fe6fe600640 1 -- 172.21.15.90:0/3139849536 shutdown_connections 2024-03-24T01:30:42.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:42.651+0000 7fe6fe600640 1 -- 172.21.15.90:0/3139849536 wait complete. 2024-03-24T01:30:42.676 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T01:30:42.677 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T01:30:42.677 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi090.front.sepia.ceph.com 2024-03-24T01:30:42.677 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T01:30:42.677 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T01:30:42.677 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T01:30:42.677 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T01:30:42.677 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T01:30:42.677 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T01:30:42.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:30:42.677 DEBUG:teuthology.orchestra.run.smithi090:> ip addr 2024-03-24T01:30:42.725 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T01:30:42.726 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T01:30:42.726 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T01:30:42.726 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:30:42.726 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-03-24T01:30:42.726 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:30:42.726 INFO:teuthology.orchestra.run.smithi090.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T01:30:42.726 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a8 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:30:42.726 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f0 2024-03-24T01:30:42.726 INFO:teuthology.orchestra.run.smithi090.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T01:30:42.726 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a9 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:30:42.726 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f1 2024-03-24T01:30:42.726 INFO:teuthology.orchestra.run.smithi090.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T01:30:42.726 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:66 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:30:42.727 INFO:teuthology.orchestra.run.smithi090.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T01:30:42.727 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:67 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:30:42.727 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.21.15.90/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T01:30:42.727 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft 42139sec preferred_lft 42139sec 2024-03-24T01:30:42.727 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::ec4:7aff:fe8f:d767/64 scope link 2024-03-24T01:30:42.727 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:30:42.727 INFO:teuthology.orchestra.run.smithi090.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T01:30:42.727 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 06:50:1d:bd:ed:70 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:30:42.727 INFO:teuthology.orchestra.run.smithi090.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T01:30:42.727 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:30:42.727 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::a05b:d3ff:fea5:6fd0/64 scope link 2024-03-24T01:30:42.727 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:30:42.727 INFO:teuthology.orchestra.run.smithi090.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T01:30:42.727 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether de:94:0b:7e:c6:41 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:30:42.727 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::dc94:bff:fe7e:c641/64 scope link 2024-03-24T01:30:42.728 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:30:42.728 INFO:teuthology.orchestra.run.smithi090.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T01:30:42.728 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 06:50:1d:bd:ed:70 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:30:42.728 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::450:1dff:febd:ed70/64 scope link 2024-03-24T01:30:42.728 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:30:42.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:30:42.729 DEBUG:teuthology.orchestra.run.smithi090:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T01:30:42.858 INFO:teuthology.orchestra.run.smithi090.stdout:1 2024-03-24T01:30:42.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:30:42.863 DEBUG:teuthology.orchestra.run.smithi090:> route 2024-03-24T01:30:42.920 INFO:teuthology.orchestra.run.smithi090.stdout:Kernel IP routing table 2024-03-24T01:30:42.920 INFO:teuthology.orchestra.run.smithi090.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T01:30:42.920 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T01:30:42.920 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T01:30:42.920 INFO:teuthology.orchestra.run.smithi090.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T01:30:42.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:30:42.921 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2024-03-24T01:30:42.921 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-24T01:30:42.922 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T01:30:42.922 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T01:30:42.922 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T01:30:42.922 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-24T01:30:43.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:30:43.097 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-03-24T01:30:43.144 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T01:30:43.144 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T01:30:43.144 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T01:30:43.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:30:43.144 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set brx.0 up 2024-03-24T01:30:43.279 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-24T01:30:43.279 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T01:30:43.279 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T01:30:43.330 INFO:teuthology.orchestra.run.smithi090.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T01:30:43.330 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T01:30:43.330 DEBUG:teuthology.orchestra.run.smithi090:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T01:30:43.374 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-24T01:30:43.502 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T01:30:43.552 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-24T01:30:43.552 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-24T01:30:43.552 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-24T01:30:43.552 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-24T01:30:43.553 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-24T01:30:43.553 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-24T01:30:43.553 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-24T01:30:43.553 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-24T01:30:43.553 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-24T01:30:43.553 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-24T01:30:43.553 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-24T01:30:43.553 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/selinux 2024-03-24T01:30:43.553 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T01:30:43.553 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-24T01:30:43.553 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-24T01:30:43.553 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-24T01:30:43.553 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-24T01:30:43.553 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T01:30:43.554 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-24T01:30:43.554 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T01:30:43.554 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-24T01:30:43.554 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T01:30:43.554 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T01:30:43.554 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-24T01:30:43.554 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-24T01:30:43.554 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T01:30:43.554 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T01:30:43.554 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T01:30:43.554 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T01:30:43.554 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-24T01:30:43.554 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:30:43.554 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:30:43.554 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:30:43.554 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:30:43.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:30:43.555 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-24T01:30:43.602 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-24T01:30:43.602 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-24T01:30:43.607 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-24T01:30:43.678 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T01:30:43.718 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-24T01:30:43.718 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-24T01:30:43.719 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-24T01:30:43.719 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-24T01:30:43.719 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-24T01:30:43.719 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-24T01:30:43.719 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-24T01:30:43.719 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-24T01:30:43.719 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-24T01:30:43.719 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-24T01:30:43.719 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-24T01:30:43.719 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/selinux 2024-03-24T01:30:43.719 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T01:30:43.719 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-24T01:30:43.719 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-24T01:30:43.719 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-24T01:30:43.720 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-24T01:30:43.720 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T01:30:43.720 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-24T01:30:43.720 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T01:30:43.720 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-24T01:30:43.720 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T01:30:43.720 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T01:30:43.720 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-24T01:30:43.720 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-24T01:30:43.720 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T01:30:43.720 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T01:30:43.720 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T01:30:43.720 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T01:30:43.720 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-24T01:30:43.720 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:30:43.720 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:30:43.720 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:30:43.721 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:30:43.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:30:43.721 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-24T01:30:43.781 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:2024-03-24T01:30:43.781+0000 7f898d95e480 -1 init, newargv = 0x55a5fbb8fbb0 newargc=15 2024-03-24T01:30:43.782 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[62045]: starting ceph client 2024-03-24T01:30:43.790 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[62045]: starting fuse 2024-03-24T01:30:43.808 INFO:teuthology.orchestra.run.smithi090.stdout:40 2024-03-24T01:30:43.808 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-03-24T01:30:43.808 DEBUG:teuthology.orchestra.run.smithi090:> sudo stdin-killer -- python3 -c ' 2024-03-24T01:30:43.808 DEBUG:teuthology.orchestra.run.smithi090:> import glob 2024-03-24T01:30:43.808 DEBUG:teuthology.orchestra.run.smithi090:> import re 2024-03-24T01:30:43.808 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-24T01:30:43.808 DEBUG:teuthology.orchestra.run.smithi090:> import subprocess 2024-03-24T01:30:43.808 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:30:43.808 DEBUG:teuthology.orchestra.run.smithi090:> def _find_admin_socket(client_name): 2024-03-24T01:30:43.808 DEBUG:teuthology.orchestra.run.smithi090:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-24T01:30:43.808 DEBUG:teuthology.orchestra.run.smithi090:> files = glob.glob(asok_path) 2024-03-24T01:30:43.809 DEBUG:teuthology.orchestra.run.smithi090:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-24T01:30:43.809 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:30:43.809 DEBUG:teuthology.orchestra.run.smithi090:> # Given a non-glob path, it better be there 2024-03-24T01:30:43.809 DEBUG:teuthology.orchestra.run.smithi090:> if "*" not in asok_path: 2024-03-24T01:30:43.809 DEBUG:teuthology.orchestra.run.smithi090:> assert(len(files) == 1) 2024-03-24T01:30:43.809 DEBUG:teuthology.orchestra.run.smithi090:> return files[0] 2024-03-24T01:30:43.809 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:30:43.809 DEBUG:teuthology.orchestra.run.smithi090:> for f in files: 2024-03-24T01:30:43.809 DEBUG:teuthology.orchestra.run.smithi090:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T01:30:43.809 DEBUG:teuthology.orchestra.run.smithi090:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T01:30:43.809 DEBUG:teuthology.orchestra.run.smithi090:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T01:30:43.809 DEBUG:teuthology.orchestra.run.smithi090:> contents = proc_f.read() 2024-03-24T01:30:43.809 DEBUG:teuthology.orchestra.run.smithi090:> if mountpoint in contents: 2024-03-24T01:30:43.809 DEBUG:teuthology.orchestra.run.smithi090:> return f 2024-03-24T01:30:43.809 DEBUG:teuthology.orchestra.run.smithi090:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T01:30:43.809 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:30:43.809 DEBUG:teuthology.orchestra.run.smithi090:> print(_find_admin_socket("client.0")) 2024-03-24T01:30:43.809 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-24T01:30:43.994 INFO:teuthology.orchestra.run.smithi090.stdout:/var/run/ceph/ceph-client.0.62045.asok 2024-03-24T01:30:43.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:30:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T01:30:44.012 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.62045.asok 2024-03-24T01:30:44.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:30:44.012 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.62045.asok status 2024-03-24T01:30:44.239 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-24T01:30:44.239 INFO:teuthology.orchestra.run.smithi090.stdout: "metadata": { 2024-03-24T01:30:44.239 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_sha1": "3be9a5c9bc793e11e2800a8c0c696e8b46742033", 2024-03-24T01:30:44.239 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-24T01:30:44.239 INFO:teuthology.orchestra.run.smithi090.stdout: "entity_id": "0", 2024-03-24T01:30:44.240 INFO:teuthology.orchestra.run.smithi090.stdout: "hostname": "smithi090", 2024-03-24T01:30:44.240 INFO:teuthology.orchestra.run.smithi090.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-24T01:30:44.240 INFO:teuthology.orchestra.run.smithi090.stdout: "pid": "62045", 2024-03-24T01:30:44.240 INFO:teuthology.orchestra.run.smithi090.stdout: "root": "/" 2024-03-24T01:30:44.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:30:44.240 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_count": 0, 2024-03-24T01:30:44.240 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_pinned_count": 0, 2024-03-24T01:30:44.240 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 6891, 2024-03-24T01:30:44.240 INFO:teuthology.orchestra.run.smithi090.stdout: "inst": { 2024-03-24T01:30:44.240 INFO:teuthology.orchestra.run.smithi090.stdout: "name": { 2024-03-24T01:30:44.240 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "client", 2024-03-24T01:30:44.240 INFO:teuthology.orchestra.run.smithi090.stdout: "num": 6891 2024-03-24T01:30:44.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:30:44.241 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-24T01:30:44.241 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-24T01:30:44.241 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-03-24T01:30:44.241 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 948351740 2024-03-24T01:30:44.241 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:30:44.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:30:44.241 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-24T01:30:44.241 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-24T01:30:44.241 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-03-24T01:30:44.241 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 948351740 2024-03-24T01:30:44.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:30:44.241 INFO:teuthology.orchestra.run.smithi090.stdout: "inst_str": "client.6891 192.168.0.1:0/948351740", 2024-03-24T01:30:44.241 INFO:teuthology.orchestra.run.smithi090.stdout: "addr_str": "192.168.0.1:0/948351740", 2024-03-24T01:30:44.241 INFO:teuthology.orchestra.run.smithi090.stdout: "inode_count": 1, 2024-03-24T01:30:44.242 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_epoch": 41, 2024-03-24T01:30:44.242 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch": 56, 2024-03-24T01:30:44.242 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch_barrier": 0, 2024-03-24T01:30:44.242 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklisted": false, 2024-03-24T01:30:44.242 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs" 2024-03-24T01:30:44.242 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-24T01:30:44.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:30:44.254 DEBUG:teuthology.orchestra.run.smithi090:> stat --file-system '--printf=%T 2024-03-24T01:30:44.254 DEBUG:teuthology.orchestra.run.smithi090:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-24T01:30:44.302 INFO:teuthology.orchestra.run.smithi090.stdout:fuseblk 2024-03-24T01:30:44.302 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-24T01:30:44.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:30:44.303 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T01:30:44.383 INFO:tasks.workunit:Pulling workunits from ref 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T01:30:44.383 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-24T01:30:44.383 DEBUG:teuthology.orchestra.run.smithi090:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-24T01:30:44.435 INFO:teuthology.orchestra.run.smithi090.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-24T01:30:44.435 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-03-24T01:30:44.435 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 28h/40d Inode: 1 Links: 2 2024-03-24T01:30:44.435 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T01:30:44.435 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-24T01:30:44.435 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-03-24T01:30:44.436 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-03-24 01:28:57.236297296 +0000 2024-03-24T01:30:44.436 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-03-24 01:28:57.236297296 +0000 2024-03-24T01:30:44.436 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: - 2024-03-24T01:30:44.436 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-24T01:30:44.436 DEBUG:teuthology.orchestra.run.smithi090:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-24T01:30:44.530 INFO:tasks.workunit:timeout=1h 2024-03-24T01:30:44.531 INFO:tasks.workunit:cleanup=True 2024-03-24T01:30:44.531 DEBUG:teuthology.orchestra.run.smithi090:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T01:30:44.582 INFO:tasks.workunit.client.0.smithi090.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-24T01:30:59.624 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:30:59.633 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:31:29.765 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:31:29.772 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:31:59.903 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:31:59.909 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:32:17.920 INFO:tasks.workunit.client.0.smithi090.stderr:Updating files: 63% (7765/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-24T01:32:18.724 INFO:tasks.workunit.client.0.smithi090.stderr:Note: switching to '3be9a5c9bc793e11e2800a8c0c696e8b46742033'. 2024-03-24T01:32:18.724 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-03-24T01:32:18.724 INFO:tasks.workunit.client.0.smithi090.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-24T01:32:18.724 INFO:tasks.workunit.client.0.smithi090.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-24T01:32:18.724 INFO:tasks.workunit.client.0.smithi090.stderr:state without impacting any branches by switching back to a branch. 2024-03-24T01:32:18.724 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-03-24T01:32:18.724 INFO:tasks.workunit.client.0.smithi090.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-24T01:32:18.724 INFO:tasks.workunit.client.0.smithi090.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-24T01:32:18.724 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-03-24T01:32:18.724 INFO:tasks.workunit.client.0.smithi090.stderr: git switch -c 2024-03-24T01:32:18.724 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-03-24T01:32:18.725 INFO:tasks.workunit.client.0.smithi090.stderr:Or undo this operation with: 2024-03-24T01:32:18.725 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-03-24T01:32:18.725 INFO:tasks.workunit.client.0.smithi090.stderr: git switch - 2024-03-24T01:32:18.725 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-03-24T01:32:18.725 INFO:tasks.workunit.client.0.smithi090.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-24T01:32:18.725 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-03-24T01:32:18.725 INFO:tasks.workunit.client.0.smithi090.stderr:HEAD is now at 3be9a5c9bc7 dummy commit #1 2024-03-24T01:32:18.732 DEBUG:teuthology.orchestra.run.smithi090:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-24T01:32:18.781 INFO:tasks.workunit.client.0.smithi090.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-24T01:32:18.786 INFO:tasks.workunit.client.0.smithi090.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-24T01:32:18.786 INFO:tasks.workunit.client.0.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-24T01:32:18.876 INFO:tasks.workunit.client.0.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-24T01:32:18.968 INFO:tasks.workunit.client.0.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-24T01:32:19.052 INFO:tasks.workunit.client.0.smithi090.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-24T01:32:19.057 INFO:tasks.workunit.client.0.smithi090.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-24T01:32:19.057 INFO:tasks.workunit.client.0.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-24T01:32:19.135 INFO:tasks.workunit.client.0.smithi090.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-24T01:32:19.144 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:32:19.144 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-24T01:32:19.191 INFO:tasks.workunit:Running workunits matching fs/misc/trivial_sync.sh on client.0... 2024-03-24T01:32:19.192 INFO:tasks.workunit:Running workunit fs/misc/trivial_sync.sh... 2024-03-24T01:32:19.193 DEBUG:teuthology.orchestra.run.smithi090:workunit test fs/misc/trivial_sync.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/trivial_sync.sh 2024-03-24T01:32:21.960 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-24T01:32:21.961 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-24T01:32:22.092 INFO:tasks.workunit:Stopping ['fs/misc/trivial_sync.sh'] on client.0... 2024-03-24T01:32:22.092 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-24T01:32:23.083 DEBUG:teuthology.parallel:result is None 2024-03-24T01:32:23.083 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-24T01:32:23.177 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-24T01:32:23.177 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T01:32:23.177 DEBUG:teuthology.orchestra.run.smithi090:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T01:32:23.278 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T01:32:23.396 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T01:32:23.396 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:32:23.396 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:32:23.450 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:32:23.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.655+0000 7fd74ae00640 1 Processor -- start 2024-03-24T01:32:23.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.655+0000 7fd74ae00640 1 -- start start 2024-03-24T01:32:23.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.657+0000 7fd74ae00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7440ce8b0 0x7fd7440b9800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:23.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.657+0000 7fd74ae00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd7440cec90 0x7fd7440b6c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:23.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.657+0000 7fd74ae00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd7440d1c20 0x7fd7440b8600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:23.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.657+0000 7fd74ae00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7440b7560 con 0x7fd7440d1c20 2024-03-24T01:32:23.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.657+0000 7fd74ae00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7440b7840 con 0x7fd7440cec90 2024-03-24T01:32:23.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.657+0000 7fd74ae00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7440b7b20 con 0x7fd7440ce8b0 2024-03-24T01:32:23.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.657+0000 7fd748e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd7440d1c20 0x7fd7440b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:23.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.657+0000 7fd748e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd7440d1c20 0x7fd7440b8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54022/0 (socket says 172.21.15.90:54022) 2024-03-24T01:32:23.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.657+0000 7fd748e00640 1 -- 172.21.15.90:0/157488470 learned_addr learned my addr 172.21.15.90:0/157488470 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:23.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.657+0000 7fd73b400640 1 --2- 172.21.15.90:0/157488470 >> v2:172.21.15.120:3300/0 conn(0x7fd7440cec90 0x7fd7440b6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:23.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.658+0000 7fd743e00640 1 --2- 172.21.15.90:0/157488470 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7440ce8b0 0x7fd7440b9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:23.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.658+0000 7fd748e00640 1 -- 172.21.15.90:0/157488470 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7440ce8b0 msgr2=0x7fd7440b9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:23.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.658+0000 7fd748e00640 1 --2- 172.21.15.90:0/157488470 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7440ce8b0 0x7fd7440b9800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:23.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.658+0000 7fd748e00640 1 -- 172.21.15.90:0/157488470 >> v2:172.21.15.120:3300/0 conn(0x7fd7440cec90 msgr2=0x7fd7440b6c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:23.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.658+0000 7fd748e00640 1 --2- 172.21.15.90:0/157488470 >> v2:172.21.15.120:3300/0 conn(0x7fd7440cec90 0x7fd7440b6c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:23.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.658+0000 7fd748e00640 1 -- 172.21.15.90:0/157488470 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7440b7e00 con 0x7fd7440d1c20 2024-03-24T01:32:23.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.658+0000 7fd743e00640 1 --2- 172.21.15.90:0/157488470 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7440ce8b0 0x7fd7440b9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:23.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.659+0000 7fd748e00640 1 --2- 172.21.15.90:0/157488470 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7440d1c20 0x7fd7440b8600 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fd7440d4bf0 tx=0x7fd73c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bc5d39f427c35ef server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:23.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.659+0000 7fd743400640 1 -- 172.21.15.90:0/157488470 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73c034070 con 0x7fd7440d1c20 2024-03-24T01:32:23.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.659+0000 7fd743400640 1 -- 172.21.15.90:0/157488470 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd73c030860 con 0x7fd7440d1c20 2024-03-24T01:32:23.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.659+0000 7fd743400640 1 -- 172.21.15.90:0/157488470 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73c030c30 con 0x7fd7440d1c20 2024-03-24T01:32:23.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.660+0000 7fd74ae00640 1 -- 172.21.15.90:0/157488470 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7440d1c20 msgr2=0x7fd7440b8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:23.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.660+0000 7fd74ae00640 1 --2- 172.21.15.90:0/157488470 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7440d1c20 0x7fd7440b8600 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fd7440d4bf0 tx=0x7fd73c02b9a0 comp rx=0 tx=0).stop 2024-03-24T01:32:23.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.660+0000 7fd74ae00640 1 -- 172.21.15.90:0/157488470 shutdown_connections 2024-03-24T01:32:23.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.660+0000 7fd74ae00640 1 --2- 172.21.15.90:0/157488470 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7440d1c20 0x7fd7440b8600 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:23.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.660+0000 7fd74ae00640 1 --2- 172.21.15.90:0/157488470 >> v2:172.21.15.120:3300/0 conn(0x7fd7440cec90 0x7fd7440b6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:23.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.660+0000 7fd74ae00640 1 --2- 172.21.15.90:0/157488470 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7440ce8b0 0x7fd7440b9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:23.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.660+0000 7fd74ae00640 1 -- 172.21.15.90:0/157488470 >> 172.21.15.90:0/157488470 conn(0x7fd744072c10 msgr2=0x7fd744073020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:23.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.660+0000 7fd74ae00640 1 -- 172.21.15.90:0/157488470 shutdown_connections 2024-03-24T01:32:23.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.660+0000 7fd74ae00640 1 -- 172.21.15.90:0/157488470 wait complete. 2024-03-24T01:32:23.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.662+0000 7fd74ae00640 1 Processor -- start 2024-03-24T01:32:23.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.662+0000 7fd74ae00640 1 -- start start 2024-03-24T01:32:23.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.663+0000 7fd74ae00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7440ce8b0 0x7fd74415c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:23.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.663+0000 7fd74ae00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7440cec90 0x7fd7441f6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:23.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.663+0000 7fd74ae00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7440d1c20 0x7fd7440d20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:23.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.663+0000 7fd74ae00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7440b7310 con 0x7fd7440ce8b0 2024-03-24T01:32:23.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.663+0000 7fd74ae00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7440b7190 con 0x7fd7440cec90 2024-03-24T01:32:23.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.663+0000 7fd74ae00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd74415df80 con 0x7fd7440d1c20 2024-03-24T01:32:23.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.663+0000 7fd743e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7440ce8b0 0x7fd74415c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:23.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.663+0000 7fd743e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7440ce8b0 0x7fd74415c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54028/0 (socket says 172.21.15.90:54028) 2024-03-24T01:32:23.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.663+0000 7fd743e00640 1 -- 172.21.15.90:0/4057811803 learned_addr learned my addr 172.21.15.90:0/4057811803 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:23.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.663+0000 7fd743e00640 1 -- 172.21.15.90:0/4057811803 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7440d1c20 msgr2=0x7fd7440d20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:23.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.663+0000 7fd743e00640 1 --2- 172.21.15.90:0/4057811803 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7440d1c20 0x7fd7440d20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:23.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.663+0000 7fd743e00640 1 -- 172.21.15.90:0/4057811803 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7440cec90 msgr2=0x7fd7441f6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:23.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.663+0000 7fd743e00640 1 --2- 172.21.15.90:0/4057811803 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7440cec90 0x7fd7441f6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:23.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.663+0000 7fd743e00640 1 -- 172.21.15.90:0/4057811803 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7441f75a0 con 0x7fd7440ce8b0 2024-03-24T01:32:23.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.664+0000 7fd743e00640 1 --2- 172.21.15.90:0/4057811803 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7440ce8b0 0x7fd74415c090 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fd73400cee0 tx=0x7fd73400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:23.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.664+0000 7fd740c00640 1 -- 172.21.15.90:0/4057811803 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73401a070 con 0x7fd7440ce8b0 2024-03-24T01:32:23.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.664+0000 7fd740c00640 1 -- 172.21.15.90:0/4057811803 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd73400d840 con 0x7fd7440ce8b0 2024-03-24T01:32:23.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.664+0000 7fd740c00640 1 -- 172.21.15.90:0/4057811803 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73400dc10 con 0x7fd7440ce8b0 2024-03-24T01:32:23.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.664+0000 7fd74ae00640 1 -- 172.21.15.90:0/4057811803 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7441f7880 con 0x7fd7440ce8b0 2024-03-24T01:32:23.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.664+0000 7fd74ae00640 1 -- 172.21.15.90:0/4057811803 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd74415f9e0 con 0x7fd7440ce8b0 2024-03-24T01:32:23.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.665+0000 7fd74ae00640 1 -- 172.21.15.90:0/4057811803 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd744068770 con 0x7fd7440ce8b0 2024-03-24T01:32:23.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.666+0000 7fd740c00640 1 -- 172.21.15.90:0/4057811803 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd73401d070 con 0x7fd7440ce8b0 2024-03-24T01:32:23.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.667+0000 7fd740c00640 1 --2- 172.21.15.90:0/4057811803 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd72006bbd0 0x7fd72006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:23.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.667+0000 7fd740c00640 1 -- 172.21.15.90:0/4057811803 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fd73408b4d0 con 0x7fd7440ce8b0 2024-03-24T01:32:23.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.668+0000 7fd73b400640 1 --2- 172.21.15.90:0/4057811803 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd72006bbd0 0x7fd72006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:23.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.669+0000 7fd73b400640 1 --2- 172.21.15.90:0/4057811803 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd72006bbd0 0x7fd72006e090 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd744073620 tx=0x7fd72c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:23.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.677+0000 7fd740c00640 1 -- 172.21.15.90:0/4057811803 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd7340596b0 con 0x7fd7440ce8b0 2024-03-24T01:32:23.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.868+0000 7fd74ae00640 1 -- 172.21.15.90:0/4057811803 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7440b48d0 con 0x7fd7440ce8b0 2024-03-24T01:32:23.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.870+0000 7fd740c00640 1 -- 172.21.15.90:0/4057811803 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+17087 (secure 0 0 0) 0x7fd73405d750 con 0x7fd7440ce8b0 2024-03-24T01:32:23.871 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:32:23.871 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":56,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:29:08.935541+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:27:11.774313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-24T01:27:12.992083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.23","mappings":[{"from":0,"to":1}]},{"pgid":"8.37","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.120:6836/266172244":"2024-03-25T01:29:08.935482+0000","172.21.15.120:6834/266172244":"2024-03-25T01:29:08.935482+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:32:23.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.872+0000 7fd74ae00640 1 -- 172.21.15.90:0/4057811803 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd72006bbd0 msgr2=0x7fd72006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:23.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.872+0000 7fd74ae00640 1 --2- 172.21.15.90:0/4057811803 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd72006bbd0 0x7fd72006e090 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd744073620 tx=0x7fd72c008040 comp rx=0 tx=0).stop 2024-03-24T01:32:23.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.872+0000 7fd74ae00640 1 -- 172.21.15.90:0/4057811803 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7440ce8b0 msgr2=0x7fd74415c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:23.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.872+0000 7fd74ae00640 1 --2- 172.21.15.90:0/4057811803 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7440ce8b0 0x7fd74415c090 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fd73400cee0 tx=0x7fd73400a710 comp rx=0 tx=0).stop 2024-03-24T01:32:23.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.872+0000 7fd74ae00640 1 -- 172.21.15.90:0/4057811803 shutdown_connections 2024-03-24T01:32:23.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.872+0000 7fd74ae00640 1 --2- 172.21.15.90:0/4057811803 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd72006bbd0 0x7fd72006e090 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:23.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.872+0000 7fd74ae00640 1 --2- 172.21.15.90:0/4057811803 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd7440d1c20 0x7fd7440d20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:23.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.872+0000 7fd74ae00640 1 --2- 172.21.15.90:0/4057811803 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7440cec90 0x7fd7441f6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:23.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.872+0000 7fd74ae00640 1 --2- 172.21.15.90:0/4057811803 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd7440ce8b0 0x7fd74415c090 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:23.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.872+0000 7fd74ae00640 1 -- 172.21.15.90:0/4057811803 >> 172.21.15.90:0/4057811803 conn(0x7fd744072c10 msgr2=0x7fd74415d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:23.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.872+0000 7fd74ae00640 1 -- 172.21.15.90:0/4057811803 shutdown_connections 2024-03-24T01:32:23.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:23.873+0000 7fd74ae00640 1 -- 172.21.15.90:0/4057811803 wait complete. 2024-03-24T01:32:23.890 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T01:27:11.774313+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T01:27:12.992083+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T01:32:23.890 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:32:24.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.098+0000 7f0029400640 1 Processor -- start 2024-03-24T01:32:24.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.098+0000 7f0029400640 1 -- start start 2024-03-24T01:32:24.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.100+0000 7f0029400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f002415ba60 0x7f002406b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:24.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.100+0000 7f0029400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f002409a0a0 0x7f00240cd300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:24.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.100+0000 7f0029400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f002415f600 con 0x7f00240b2a70 2024-03-24T01:32:24.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.100+0000 7f0029400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0024160210 con 0x7f002409a0a0 2024-03-24T01:32:24.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.100+0000 7f0029400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f002415f8e0 con 0x7f002415ba60 2024-03-24T01:32:24.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.100+0000 7f001be00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f00240b2a70 0x7f00240ce3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50760/0 (socket says 172.21.15.90:50760) 2024-03-24T01:32:24.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.100+0000 7f001be00640 1 -- 172.21.15.90:0/3017951483 learned_addr learned my addr 172.21.15.90:0/3017951483 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:32:24.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.100+0000 7f0028a00640 1 --2- 172.21.15.90:0/3017951483 >> v2:172.21.15.120:3300/0 conn(0x7f002409a0a0 0x7f00240cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:24.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.101+0000 7f0023e00640 1 --2- 172.21.15.90:0/3017951483 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f002415ba60 0x7f002406b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:24.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.101+0000 7f0023400640 1 -- 172.21.15.90:0/3017951483 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 95791461 0 0) 0x7f002415f600 con 0x7f00240b2a70 2024-03-24T01:32:24.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.101+0000 7f0023400640 1 -- 172.21.15.90:0/3017951483 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0008003520 con 0x7f00240b2a70 2024-03-24T01:32:24.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.101+0000 7f0023400640 1 -- 172.21.15.90:0/3017951483 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1110246817 0 0) 0x7f0008003520 con 0x7f00240b2a70 2024-03-24T01:32:24.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.101+0000 7f0023400640 1 -- 172.21.15.90:0/3017951483 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f002415ba60 msgr2=0x7f002406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:24.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.101+0000 7f0023400640 1 --2- 172.21.15.90:0/3017951483 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f002415ba60 0x7f002406b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:24.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.101+0000 7f0023400640 1 -- 172.21.15.90:0/3017951483 >> v2:172.21.15.120:3300/0 conn(0x7f002409a0a0 msgr2=0x7f00240cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:24.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.101+0000 7f0023400640 1 --2- 172.21.15.90:0/3017951483 >> v2:172.21.15.120:3300/0 conn(0x7f002409a0a0 0x7f00240cd300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:24.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.101+0000 7f0023400640 1 -- 172.21.15.90:0/3017951483 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f002415fbc0 con 0x7f00240b2a70 2024-03-24T01:32:24.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.101+0000 7f0023400640 1 -- 172.21.15.90:0/3017951483 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f000c001670 con 0x7f00240b2a70 2024-03-24T01:32:24.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.101+0000 7f0023400640 1 -- 172.21.15.90:0/3017951483 >> v1:172.21.15.90:6789/0 conn(0x7f00240b2a70 legacy=0x7f00240ce3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:24.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.101+0000 7f0023400640 1 --2- 172.21.15.90:0/3017951483 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00080039b0 0x7f0008003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:24.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.101+0000 7f0023400640 1 --2- 172.21.15.90:0/3017951483 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f002415ba60 0x7f0008004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:24.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.101+0000 7f0023400640 1 --2- 172.21.15.90:0/3017951483 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0008004d40 0x7f00080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:24.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.101+0000 7f0023400640 1 -- 172.21.15.90:0/3017951483 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0024160210 con 0x7f0008004d40 2024-03-24T01:32:24.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.101+0000 7f0023400640 1 -- 172.21.15.90:0/3017951483 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002415f8e0 con 0x7f00080039b0 2024-03-24T01:32:24.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.101+0000 7f0023400640 1 -- 172.21.15.90:0/3017951483 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f000800b9c0 con 0x7f002415ba60 2024-03-24T01:32:24.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.102+0000 7f0028a00640 1 --2- 172.21.15.90:0/3017951483 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0008004d40 0x7f00080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:24.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.102+0000 7f0023e00640 1 --2- 172.21.15.90:0/3017951483 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00080039b0 0x7f0008003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:24.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.102+0000 7f001be00640 1 --2- 172.21.15.90:0/3017951483 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f002415ba60 0x7f0008004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:24.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.102+0000 7f0028a00640 1 -- 172.21.15.90:0/3017951483 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f002415ba60 msgr2=0x7f0008004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:24.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.102+0000 7f0028a00640 1 --2- 172.21.15.90:0/3017951483 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f002415ba60 0x7f0008004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:24.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.102+0000 7f0028a00640 1 -- 172.21.15.90:0/3017951483 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00080039b0 msgr2=0x7f0008003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:24.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.102+0000 7f0028a00640 1 --2- 172.21.15.90:0/3017951483 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00080039b0 0x7f0008003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:24.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.102+0000 7f0028a00640 1 -- 172.21.15.90:0/3017951483 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0008003520 con 0x7f0008004d40 2024-03-24T01:32:24.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.104+0000 7f0028a00640 1 --2- 172.21.15.90:0/3017951483 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0008004d40 0x7f00080051c0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f001c009fc0 tx=0x7f0008005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=712b3fe658352875 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:24.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.104+0000 7f0023400640 1 -- 172.21.15.90:0/3017951483 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f001c039070 con 0x7f0008004d40 2024-03-24T01:32:24.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.104+0000 7f0023400640 1 -- 172.21.15.90:0/3017951483 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f001c02cc60 con 0x7f0008004d40 2024-03-24T01:32:24.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.104+0000 7f0023400640 1 -- 172.21.15.90:0/3017951483 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f001c009c10 con 0x7f0008004d40 2024-03-24T01:32:24.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.105+0000 7f0029400640 1 -- 172.21.15.90:0/3017951483 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0008004d40 msgr2=0x7f00080051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:24.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.105+0000 7f0029400640 1 --2- 172.21.15.90:0/3017951483 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0008004d40 0x7f00080051c0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f001c009fc0 tx=0x7f0008005830 comp rx=0 tx=0).stop 2024-03-24T01:32:24.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.105+0000 7f0028a00640 1 -- 172.21.15.90:0/3017951483 reap_dead start 2024-03-24T01:32:24.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.105+0000 7f0029400640 1 -- 172.21.15.90:0/3017951483 shutdown_connections 2024-03-24T01:32:24.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.105+0000 7f0029400640 1 -- 172.21.15.90:0/3017951483 >> 172.21.15.90:0/3017951483 conn(0x7f0024099a70 msgr2=0x7f00240b2360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:24.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.105+0000 7f0029400640 1 -- 172.21.15.90:0/3017951483 shutdown_connections 2024-03-24T01:32:24.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.105+0000 7f0029400640 1 -- 172.21.15.90:0/3017951483 wait complete. 2024-03-24T01:32:24.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.107+0000 7f0029400640 1 Processor -- start 2024-03-24T01:32:24.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.107+0000 7f0029400640 1 -- start start 2024-03-24T01:32:24.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.108+0000 7f0029400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0008004d40 0x7f0024156060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:24.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.108+0000 7f0029400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0024157a10 0x7f0024157e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:24.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.108+0000 7f0029400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00240b42b0 0x7f00240b4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:24.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.108+0000 7f0029400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00240d0390 con 0x7f0024157a10 2024-03-24T01:32:24.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.108+0000 7f0029400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002415be40 con 0x7f00240b42b0 2024-03-24T01:32:24.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.108+0000 7f0029400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00240d0510 con 0x7f0008004d40 2024-03-24T01:32:24.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.108+0000 7f001be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0024157a10 0x7f0024157e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:24.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.108+0000 7f001be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0024157a10 0x7f0024157e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54038/0 (socket says 172.21.15.90:54038) 2024-03-24T01:32:24.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.108+0000 7f001be00640 1 -- 172.21.15.90:0/220587023 learned_addr learned my addr 172.21.15.90:0/220587023 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:24.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.108+0000 7f0028a00640 1 --2- 172.21.15.90:0/220587023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00240b42b0 0x7f00240b4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:24.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.108+0000 7f0023e00640 1 --2- 172.21.15.90:0/220587023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0008004d40 0x7f0024156060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:24.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.108+0000 7f001be00640 1 -- 172.21.15.90:0/220587023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0008004d40 msgr2=0x7f0024156060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:24.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.108+0000 7f001be00640 1 --2- 172.21.15.90:0/220587023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0008004d40 0x7f0024156060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:24.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.108+0000 7f001be00640 1 -- 172.21.15.90:0/220587023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00240b42b0 msgr2=0x7f00240b4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:24.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.108+0000 7f001be00640 1 --2- 172.21.15.90:0/220587023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00240b42b0 0x7f00240b4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:24.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.108+0000 7f001be00640 1 -- 172.21.15.90:0/220587023 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00240c7a40 con 0x7f0024157a10 2024-03-24T01:32:24.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.109+0000 7f0023e00640 1 --2- 172.21.15.90:0/220587023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0008004d40 0x7f0024156060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:24.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.109+0000 7f001be00640 1 --2- 172.21.15.90:0/220587023 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0024157a10 0x7f0024157e90 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f000c00e780 tx=0x7f000c00ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:24.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.109+0000 7f0020c00640 1 -- 172.21.15.90:0/220587023 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f000c00ee90 con 0x7f0024157a10 2024-03-24T01:32:24.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.109+0000 7f0020c00640 1 -- 172.21.15.90:0/220587023 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f000c018070 con 0x7f0024157a10 2024-03-24T01:32:24.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.109+0000 7f0029400640 1 -- 172.21.15.90:0/220587023 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00240c7d20 con 0x7f0024157a10 2024-03-24T01:32:24.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.109+0000 7f0020c00640 1 -- 172.21.15.90:0/220587023 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f000c00cb10 con 0x7f0024157a10 2024-03-24T01:32:24.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.109+0000 7f0029400640 1 -- 172.21.15.90:0/220587023 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00240c82e0 con 0x7f0024157a10 2024-03-24T01:32:24.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.111+0000 7f0029400640 1 -- 172.21.15.90:0/220587023 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f002404f400 con 0x7f0024157a10 2024-03-24T01:32:24.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.112+0000 7f0020c00640 1 -- 172.21.15.90:0/220587023 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f000c00ccb0 con 0x7f0024157a10 2024-03-24T01:32:24.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.113+0000 7f0020c00640 1 --2- 172.21.15.90:0/220587023 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f000006bda0 0x7f000006e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:24.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.113+0000 7f0020c00640 1 -- 172.21.15.90:0/220587023 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f000c08abc0 con 0x7f0024157a10 2024-03-24T01:32:24.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.113+0000 7f0023e00640 1 --2- 172.21.15.90:0/220587023 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f000006bda0 0x7f000006e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:24.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.115+0000 7f0023e00640 1 --2- 172.21.15.90:0/220587023 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f000006bda0 0x7f000006e260 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f0024158710 tx=0x7f0014008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:24.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.118+0000 7f0020c00640 1 -- 172.21.15.90:0/220587023 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f000c0593f0 con 0x7f0024157a10 2024-03-24T01:32:24.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.388+0000 7f0029400640 1 -- 172.21.15.90:0/220587023 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0024099e80 con 0x7f0024157a10 2024-03-24T01:32:24.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.389+0000 7f0020c00640 1 -- 172.21.15.90:0/220587023 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) v1 ==== 93+0+10 (secure 0 0 0) 0x7f000c058cf0 con 0x7f0024157a10 2024-03-24T01:32:24.389 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:32:24.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.391+0000 7f0029400640 1 -- 172.21.15.90:0/220587023 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f000006bda0 msgr2=0x7f000006e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:24.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.391+0000 7f0029400640 1 --2- 172.21.15.90:0/220587023 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f000006bda0 0x7f000006e260 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f0024158710 tx=0x7f0014008040 comp rx=0 tx=0).stop 2024-03-24T01:32:24.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.391+0000 7f0029400640 1 -- 172.21.15.90:0/220587023 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0024157a10 msgr2=0x7f0024157e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:24.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.391+0000 7f0029400640 1 --2- 172.21.15.90:0/220587023 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0024157a10 0x7f0024157e90 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f000c00e780 tx=0x7f000c00ec40 comp rx=0 tx=0).stop 2024-03-24T01:32:24.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.392+0000 7f0029400640 1 -- 172.21.15.90:0/220587023 shutdown_connections 2024-03-24T01:32:24.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.392+0000 7f0029400640 1 --2- 172.21.15.90:0/220587023 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f000006bda0 0x7f000006e260 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:24.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.392+0000 7f0029400640 1 --2- 172.21.15.90:0/220587023 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0024157a10 0x7f0024157e90 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:24.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.392+0000 7f0029400640 1 --2- 172.21.15.90:0/220587023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00240b42b0 0x7f00240b4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:24.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.392+0000 7f0029400640 1 --2- 172.21.15.90:0/220587023 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0008004d40 0x7f0024156060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:24.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.392+0000 7f0029400640 1 -- 172.21.15.90:0/220587023 >> 172.21.15.90:0/220587023 conn(0x7f0024099a70 msgr2=0x7f00240ae270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:24.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.392+0000 7f0029400640 1 -- 172.21.15.90:0/220587023 shutdown_connections 2024-03-24T01:32:24.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.392+0000 7f0029400640 1 -- 172.21.15.90:0/220587023 wait complete. 2024-03-24T01:32:24.409 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T01:32:24.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.612+0000 7f4e90a00640 1 Processor -- start 2024-03-24T01:32:24.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.613+0000 7f4e90a00640 1 -- start start 2024-03-24T01:32:24.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.615+0000 7f4e90a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e8c15dc20 0x7f4e8c0d1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:24.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.615+0000 7f4e90a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4e8c0d1ff0 0x7f4e8c0b54b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:24.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.615+0000 7f4e90a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f4e8c0d3510 0x7f4e8c0b59f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:24.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.615+0000 7f4e90a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e8c0d2470 con 0x7f4e8c0d1ff0 2024-03-24T01:32:24.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.615+0000 7f4e90a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e8c0d2750 con 0x7f4e8c0d3510 2024-03-24T01:32:24.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.615+0000 7f4e90a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8c0d2a30 con 0x7f4e8c15dc20 2024-03-24T01:32:24.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.615+0000 7f4e89e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4e8c0d1ff0 0x7f4e8c0b54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:24.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.615+0000 7f4e89e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4e8c0d1ff0 0x7f4e8c0b54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54050/0 (socket says 172.21.15.90:54050) 2024-03-24T01:32:24.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.615+0000 7f4e89e00640 1 -- 172.21.15.90:0/2754705633 learned_addr learned my addr 172.21.15.90:0/2754705633 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:24.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.616+0000 7f4e8b200640 1 --2- 172.21.15.90:0/2754705633 >> v2:172.21.15.120:3300/0 conn(0x7f4e8c0d3510 0x7f4e8c0b59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:24.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.616+0000 7f4e8a800640 1 --2- 172.21.15.90:0/2754705633 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e8c15dc20 0x7f4e8c0d1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:24.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.616+0000 7f4e89e00640 1 -- 172.21.15.90:0/2754705633 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e8c15dc20 msgr2=0x7f4e8c0d1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:24.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.616+0000 7f4e89e00640 1 --2- 172.21.15.90:0/2754705633 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e8c15dc20 0x7f4e8c0d1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:24.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.616+0000 7f4e89e00640 1 -- 172.21.15.90:0/2754705633 >> v2:172.21.15.120:3300/0 conn(0x7f4e8c0d3510 msgr2=0x7f4e8c0b59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:24.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.616+0000 7f4e89e00640 1 --2- 172.21.15.90:0/2754705633 >> v2:172.21.15.120:3300/0 conn(0x7f4e8c0d3510 0x7f4e8c0b59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:24.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.616+0000 7f4e89e00640 1 -- 172.21.15.90:0/2754705633 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e8c0d2d10 con 0x7f4e8c0d1ff0 2024-03-24T01:32:24.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.617+0000 7f4e89e00640 1 --2- 172.21.15.90:0/2754705633 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e8c0d1ff0 0x7f4e8c0b54b0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c0d1370 tx=0x7f4e7c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b444bb87481520 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:24.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.618+0000 7f4e89400640 1 -- 172.21.15.90:0/2754705633 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c02ca70 con 0x7f4e8c0d1ff0 2024-03-24T01:32:24.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.618+0000 7f4e89400640 1 -- 172.21.15.90:0/2754705633 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e7c02cc10 con 0x7f4e8c0d1ff0 2024-03-24T01:32:24.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.618+0000 7f4e89400640 1 -- 172.21.15.90:0/2754705633 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c0343f0 con 0x7f4e8c0d1ff0 2024-03-24T01:32:24.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.618+0000 7f4e90a00640 1 -- 172.21.15.90:0/2754705633 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e8c0d1ff0 msgr2=0x7f4e8c0b54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:24.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.618+0000 7f4e90a00640 1 --2- 172.21.15.90:0/2754705633 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e8c0d1ff0 0x7f4e8c0b54b0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c0d1370 tx=0x7f4e7c02bed0 comp rx=0 tx=0).stop 2024-03-24T01:32:24.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.618+0000 7f4e90a00640 1 -- 172.21.15.90:0/2754705633 shutdown_connections 2024-03-24T01:32:24.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.618+0000 7f4e90a00640 1 --2- 172.21.15.90:0/2754705633 >> v2:172.21.15.120:3300/0 conn(0x7f4e8c0d3510 0x7f4e8c0b59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:24.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.618+0000 7f4e90a00640 1 --2- 172.21.15.90:0/2754705633 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e8c0d1ff0 0x7f4e8c0b54b0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:24.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.618+0000 7f4e90a00640 1 --2- 172.21.15.90:0/2754705633 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e8c15dc20 0x7f4e8c0d1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:24.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.618+0000 7f4e90a00640 1 -- 172.21.15.90:0/2754705633 >> 172.21.15.90:0/2754705633 conn(0x7f4e8c072e80 msgr2=0x7f4e8c0d04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:24.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.618+0000 7f4e90a00640 1 -- 172.21.15.90:0/2754705633 shutdown_connections 2024-03-24T01:32:24.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.619+0000 7f4e90a00640 1 -- 172.21.15.90:0/2754705633 wait complete. 2024-03-24T01:32:24.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.620+0000 7f4e90a00640 1 Processor -- start 2024-03-24T01:32:24.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.621+0000 7f4e90a00640 1 -- start start 2024-03-24T01:32:24.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.622+0000 7f4e90a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e8c0d1ff0 0x7f4e8c148c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:24.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.622+0000 7f4e90a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e8c0d3510 0x7f4e8c0b3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:24.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.622+0000 7f4e90a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e8c15dc20 0x7f4e8c156a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:24.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.622+0000 7f4e90a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8c0b94a0 con 0x7f4e8c0d3510 2024-03-24T01:32:24.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.622+0000 7f4e90a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8c15e000 con 0x7f4e8c15dc20 2024-03-24T01:32:24.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.622+0000 7f4e90a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8c0b9620 con 0x7f4e8c0d1ff0 2024-03-24T01:32:24.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.622+0000 7f4e89e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e8c0d3510 0x7f4e8c0b3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:24.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.622+0000 7f4e89e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e8c0d3510 0x7f4e8c0b3ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54066/0 (socket says 172.21.15.90:54066) 2024-03-24T01:32:24.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.622+0000 7f4e89e00640 1 -- 172.21.15.90:0/4207040203 learned_addr learned my addr 172.21.15.90:0/4207040203 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:24.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.623+0000 7f4e8b200640 1 --2- 172.21.15.90:0/4207040203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e8c15dc20 0x7f4e8c156a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:24.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.623+0000 7f4e8a800640 1 --2- 172.21.15.90:0/4207040203 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e8c0d1ff0 0x7f4e8c148c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:24.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.623+0000 7f4e89e00640 1 -- 172.21.15.90:0/4207040203 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e8c0d1ff0 msgr2=0x7f4e8c148c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:24.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.623+0000 7f4e89e00640 1 --2- 172.21.15.90:0/4207040203 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e8c0d1ff0 0x7f4e8c148c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:24.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.623+0000 7f4e89e00640 1 -- 172.21.15.90:0/4207040203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e8c15dc20 msgr2=0x7f4e8c156a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:24.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.623+0000 7f4e89e00640 1 --2- 172.21.15.90:0/4207040203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e8c15dc20 0x7f4e8c156a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:24.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.623+0000 7f4e89e00640 1 -- 172.21.15.90:0/4207040203 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e8c0b4530 con 0x7f4e8c0d3510 2024-03-24T01:32:24.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.623+0000 7f4e8b200640 1 --2- 172.21.15.90:0/4207040203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e8c15dc20 0x7f4e8c156a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:24.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.623+0000 7f4e89e00640 1 --2- 172.21.15.90:0/4207040203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e8c0d3510 0x7f4e8c0b3ce0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c036040 tx=0x7f4e7c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:24.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.624+0000 7f4e82a00640 1 -- 172.21.15.90:0/4207040203 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c043020 con 0x7f4e8c0d3510 2024-03-24T01:32:24.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.624+0000 7f4e82a00640 1 -- 172.21.15.90:0/4207040203 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e7c03e450 con 0x7f4e8c0d3510 2024-03-24T01:32:24.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.624+0000 7f4e82a00640 1 -- 172.21.15.90:0/4207040203 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c009510 con 0x7f4e8c0d3510 2024-03-24T01:32:24.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.624+0000 7f4e90a00640 1 -- 172.21.15.90:0/4207040203 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e8c0b4810 con 0x7f4e8c0d3510 2024-03-24T01:32:24.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.624+0000 7f4e90a00640 1 -- 172.21.15.90:0/4207040203 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e8c0c8c30 con 0x7f4e8c0d3510 2024-03-24T01:32:24.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.625+0000 7f4e90a00640 1 -- 172.21.15.90:0/4207040203 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e8c04f400 con 0x7f4e8c0d3510 2024-03-24T01:32:24.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.626+0000 7f4e82a00640 1 -- 172.21.15.90:0/4207040203 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4e7c046020 con 0x7f4e8c0d3510 2024-03-24T01:32:24.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.627+0000 7f4e82a00640 1 --2- 172.21.15.90:0/4207040203 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4e6806bbd0 0x7f4e6806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:24.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.627+0000 7f4e82a00640 1 -- 172.21.15.90:0/4207040203 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f4e7c0ad540 con 0x7f4e8c0d3510 2024-03-24T01:32:24.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.627+0000 7f4e8a800640 1 --2- 172.21.15.90:0/4207040203 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4e6806bbd0 0x7f4e6806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:24.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.628+0000 7f4e8a800640 1 --2- 172.21.15.90:0/4207040203 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4e6806bbd0 0x7f4e6806e090 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f4e74006fd0 tx=0x7f4e74008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:24.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.631+0000 7f4e82a00640 1 -- 172.21.15.90:0/4207040203 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4e7c04d030 con 0x7f4e8c0d3510 2024-03-24T01:32:24.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.825+0000 7f4e90a00640 1 -- 172.21.15.90:0/4207040203 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4e8c15cee0 con 0x7f4e8c0d3510 2024-03-24T01:32:24.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.825+0000 7f4e82a00640 1 -- 172.21.15.90:0/4207040203 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v56) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4e7c07bd80 con 0x7f4e8c0d3510 2024-03-24T01:32:24.826 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:32:24.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.828+0000 7f4e90a00640 1 -- 172.21.15.90:0/4207040203 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4e6806bbd0 msgr2=0x7f4e6806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:24.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.828+0000 7f4e90a00640 1 --2- 172.21.15.90:0/4207040203 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4e6806bbd0 0x7f4e6806e090 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f4e74006fd0 tx=0x7f4e74008040 comp rx=0 tx=0).stop 2024-03-24T01:32:24.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.828+0000 7f4e90a00640 1 -- 172.21.15.90:0/4207040203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e8c0d3510 msgr2=0x7f4e8c0b3ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:24.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.828+0000 7f4e90a00640 1 --2- 172.21.15.90:0/4207040203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e8c0d3510 0x7f4e8c0b3ce0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c036040 tx=0x7f4e7c004a40 comp rx=0 tx=0).stop 2024-03-24T01:32:24.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.828+0000 7f4e90a00640 1 -- 172.21.15.90:0/4207040203 shutdown_connections 2024-03-24T01:32:24.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.828+0000 7f4e90a00640 1 --2- 172.21.15.90:0/4207040203 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4e6806bbd0 0x7f4e6806e090 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:24.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.828+0000 7f4e90a00640 1 --2- 172.21.15.90:0/4207040203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e8c0d3510 0x7f4e8c0b3ce0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:24.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.828+0000 7f4e90a00640 1 --2- 172.21.15.90:0/4207040203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e8c15dc20 0x7f4e8c156a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:24.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.828+0000 7f4e90a00640 1 --2- 172.21.15.90:0/4207040203 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4e8c0d1ff0 0x7f4e8c148c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:24.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.828+0000 7f4e90a00640 1 -- 172.21.15.90:0/4207040203 >> 172.21.15.90:0/4207040203 conn(0x7f4e8c072e80 msgr2=0x7f4e8c0ce2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:24.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.828+0000 7f4e90a00640 1 -- 172.21.15.90:0/4207040203 shutdown_connections 2024-03-24T01:32:24.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:24.828+0000 7f4e90a00640 1 -- 172.21.15.90:0/4207040203 wait complete. 2024-03-24T01:32:24.846 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T01:32:25.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.047+0000 7fd3aea00640 1 Processor -- start 2024-03-24T01:32:25.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.047+0000 7fd3aea00640 1 -- start start 2024-03-24T01:32:25.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.049+0000 7fd3aea00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd3a8159620 0x7fd3a806b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:25.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.049+0000 7fd3aea00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd3a806b600 0x7fd3a80cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:25.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.049+0000 7fd3aea00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3a80cef80 0x7fd3a80ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:25.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.049+0000 7fd3aea00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3a80b4470 con 0x7fd3a806b600 2024-03-24T01:32:25.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.049+0000 7fd3aea00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3a806bca0 con 0x7fd3a8159620 2024-03-24T01:32:25.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.049+0000 7fd3aea00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a80b36f0 con 0x7fd3a80cef80 2024-03-24T01:32:25.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.049+0000 7fd39f400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd3a806b600 0x7fd3a80cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:25.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.049+0000 7fd39f400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd3a806b600 0x7fd3a80cea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54072/0 (socket says 172.21.15.90:54072) 2024-03-24T01:32:25.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.049+0000 7fd39f400640 1 -- 172.21.15.90:0/1129155388 learned_addr learned my addr 172.21.15.90:0/1129155388 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:25.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.050+0000 7fd39f400640 1 -- 172.21.15.90:0/1129155388 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3a80cef80 msgr2=0x7fd3a80ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:25.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.050+0000 7fd39f400640 1 --2- 172.21.15.90:0/1129155388 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3a80cef80 0x7fd3a80ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:25.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.050+0000 7fd39f400640 1 -- 172.21.15.90:0/1129155388 >> v2:172.21.15.120:3300/0 conn(0x7fd3a8159620 msgr2=0x7fd3a806b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:25.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.050+0000 7fd3a7e00640 1 --2- 172.21.15.90:0/1129155388 >> v2:172.21.15.120:3300/0 conn(0x7fd3a8159620 0x7fd3a806b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:25.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.050+0000 7fd39f400640 1 --2- 172.21.15.90:0/1129155388 >> v2:172.21.15.120:3300/0 conn(0x7fd3a8159620 0x7fd3a806b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:25.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.050+0000 7fd39f400640 1 -- 172.21.15.90:0/1129155388 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a80b3990 con 0x7fd3a806b600 2024-03-24T01:32:25.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.051+0000 7fd39f400640 1 --2- 172.21.15.90:0/1129155388 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3a806b600 0x7fd3a80cea40 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fd3a806c2d0 tx=0x7fd39002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6650ff9706612e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:25.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.051+0000 7fd3a7400640 1 -- 172.21.15.90:0/1129155388 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39002e070 con 0x7fd3a806b600 2024-03-24T01:32:25.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.051+0000 7fd3a7400640 1 -- 172.21.15.90:0/1129155388 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd390037070 con 0x7fd3a806b600 2024-03-24T01:32:25.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.051+0000 7fd3a7400640 1 -- 172.21.15.90:0/1129155388 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd390031a00 con 0x7fd3a806b600 2024-03-24T01:32:25.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.051+0000 7fd3aea00640 1 -- 172.21.15.90:0/1129155388 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3a806b600 msgr2=0x7fd3a80cea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:25.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.051+0000 7fd3aea00640 1 --2- 172.21.15.90:0/1129155388 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3a806b600 0x7fd3a80cea40 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fd3a806c2d0 tx=0x7fd39002b880 comp rx=0 tx=0).stop 2024-03-24T01:32:25.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.052+0000 7fd3aea00640 1 -- 172.21.15.90:0/1129155388 shutdown_connections 2024-03-24T01:32:25.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.052+0000 7fd3aea00640 1 --2- 172.21.15.90:0/1129155388 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3a80cef80 0x7fd3a80ae530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:25.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.052+0000 7fd3aea00640 1 --2- 172.21.15.90:0/1129155388 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3a806b600 0x7fd3a80cea40 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:25.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.052+0000 7fd3aea00640 1 --2- 172.21.15.90:0/1129155388 >> v2:172.21.15.120:3300/0 conn(0x7fd3a8159620 0x7fd3a806b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:25.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.052+0000 7fd3aea00640 1 -- 172.21.15.90:0/1129155388 >> 172.21.15.90:0/1129155388 conn(0x7fd3a80b1770 msgr2=0x7fd3a80b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:25.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.052+0000 7fd3aea00640 1 -- 172.21.15.90:0/1129155388 shutdown_connections 2024-03-24T01:32:25.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.052+0000 7fd3aea00640 1 -- 172.21.15.90:0/1129155388 wait complete. 2024-03-24T01:32:25.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.053+0000 7fd3aea00640 1 Processor -- start 2024-03-24T01:32:25.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.053+0000 7fd3aea00640 1 -- start start 2024-03-24T01:32:25.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.054+0000 7fd3aea00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3a806b600 0x7fd3a8157200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:25.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.054+0000 7fd3aea00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3a80cef80 0x7fd3a815a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:25.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.054+0000 7fd3aea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3a8159620 0x7fd3a8142f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:25.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.054+0000 7fd3aea00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a8159a00 con 0x7fd3a8159620 2024-03-24T01:32:25.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.054+0000 7fd3aea00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a80b3f50 con 0x7fd3a80cef80 2024-03-24T01:32:25.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.054+0000 7fd3aea00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a80cf3f0 con 0x7fd3a806b600 2024-03-24T01:32:25.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.054+0000 7fd3aca00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3a8159620 0x7fd3a8142f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:25.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.055+0000 7fd3aca00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3a8159620 0x7fd3a8142f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54074/0 (socket says 172.21.15.90:54074) 2024-03-24T01:32:25.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.055+0000 7fd3aca00640 1 -- 172.21.15.90:0/43269402 learned_addr learned my addr 172.21.15.90:0/43269402 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:25.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.055+0000 7fd39f400640 1 --2- 172.21.15.90:0/43269402 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3a80cef80 0x7fd3a815a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:25.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.055+0000 7fd3aca00640 1 -- 172.21.15.90:0/43269402 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3a806b600 msgr2=0x7fd3a8157200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:25.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.055+0000 7fd3aca00640 1 --2- 172.21.15.90:0/43269402 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3a806b600 0x7fd3a8157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:25.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.055+0000 7fd3aca00640 1 -- 172.21.15.90:0/43269402 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3a80cef80 msgr2=0x7fd3a815a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:25.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.055+0000 7fd3aca00640 1 --2- 172.21.15.90:0/43269402 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3a80cef80 0x7fd3a815a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:25.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.055+0000 7fd3aca00640 1 -- 172.21.15.90:0/43269402 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a81e4300 con 0x7fd3a8159620 2024-03-24T01:32:25.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.055+0000 7fd3aca00640 1 --2- 172.21.15.90:0/43269402 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3a8159620 0x7fd3a8142f40 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fd3a000b970 tx=0x7fd3a000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:25.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.055+0000 7fd3a4c00640 1 -- 172.21.15.90:0/43269402 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a0019070 con 0x7fd3a8159620 2024-03-24T01:32:25.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.056+0000 7fd3a4c00640 1 -- 172.21.15.90:0/43269402 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3a000cb60 con 0x7fd3a8159620 2024-03-24T01:32:25.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.056+0000 7fd3a4c00640 1 -- 172.21.15.90:0/43269402 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a0014440 con 0x7fd3a8159620 2024-03-24T01:32:25.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.056+0000 7fd3aea00640 1 -- 172.21.15.90:0/43269402 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a81e45e0 con 0x7fd3a8159620 2024-03-24T01:32:25.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.056+0000 7fd3aea00640 1 -- 172.21.15.90:0/43269402 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3a81e4ba0 con 0x7fd3a8159620 2024-03-24T01:32:25.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.057+0000 7fd3aea00640 1 -- 172.21.15.90:0/43269402 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3a804f400 con 0x7fd3a8159620 2024-03-24T01:32:25.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.060+0000 7fd3a4c00640 1 -- 172.21.15.90:0/43269402 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd3a001c070 con 0x7fd3a8159620 2024-03-24T01:32:25.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.061+0000 7fd3a4c00640 1 --2- 172.21.15.90:0/43269402 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd38406bbd0 0x7fd38406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:25.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.061+0000 7fd3a4c00640 1 -- 172.21.15.90:0/43269402 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fd3a008aa00 con 0x7fd3a8159620 2024-03-24T01:32:25.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.061+0000 7fd3a7e00640 1 --2- 172.21.15.90:0/43269402 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd38406bbd0 0x7fd38406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:25.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.063+0000 7fd3a7e00640 1 --2- 172.21.15.90:0/43269402 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd38406bbd0 0x7fd38406e090 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fd398006fd0 tx=0x7fd398008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:25.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.065+0000 7fd3a4c00640 1 -- 172.21.15.90:0/43269402 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd3a00592c0 con 0x7fd3a8159620 2024-03-24T01:32:25.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.265+0000 7fd3aea00640 1 -- 172.21.15.90:0/43269402 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd3a815c810 con 0x7fd3a8159620 2024-03-24T01:32:25.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.266+0000 7fd3a4c00640 1 -- 172.21.15.90:0/43269402 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v56) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd3a0058bc0 con 0x7fd3a8159620 2024-03-24T01:32:25.266 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-24T01:32:25.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.268+0000 7fd3aea00640 1 -- 172.21.15.90:0/43269402 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd38406bbd0 msgr2=0x7fd38406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:25.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.268+0000 7fd3aea00640 1 --2- 172.21.15.90:0/43269402 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd38406bbd0 0x7fd38406e090 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fd398006fd0 tx=0x7fd398008040 comp rx=0 tx=0).stop 2024-03-24T01:32:25.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.268+0000 7fd3aea00640 1 -- 172.21.15.90:0/43269402 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3a8159620 msgr2=0x7fd3a8142f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:25.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.268+0000 7fd3aea00640 1 --2- 172.21.15.90:0/43269402 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3a8159620 0x7fd3a8142f40 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fd3a000b970 tx=0x7fd3a000be30 comp rx=0 tx=0).stop 2024-03-24T01:32:25.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.269+0000 7fd3aea00640 1 -- 172.21.15.90:0/43269402 shutdown_connections 2024-03-24T01:32:25.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.269+0000 7fd3aea00640 1 --2- 172.21.15.90:0/43269402 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd38406bbd0 0x7fd38406e090 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:25.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.269+0000 7fd3aea00640 1 --2- 172.21.15.90:0/43269402 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3a8159620 0x7fd3a8142f40 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:25.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.269+0000 7fd3aea00640 1 --2- 172.21.15.90:0/43269402 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3a80cef80 0x7fd3a815a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:25.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.269+0000 7fd3aea00640 1 --2- 172.21.15.90:0/43269402 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3a806b600 0x7fd3a8157200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:25.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.269+0000 7fd3aea00640 1 -- 172.21.15.90:0/43269402 >> 172.21.15.90:0/43269402 conn(0x7fd3a80b1770 msgr2=0x7fd3a80b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:25.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.269+0000 7fd3aea00640 1 -- 172.21.15.90:0/43269402 shutdown_connections 2024-03-24T01:32:25.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.269+0000 7fd3aea00640 1 -- 172.21.15.90:0/43269402 wait complete. 2024-03-24T01:32:25.289 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T01:32:25.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.498+0000 7ff106000640 1 Processor -- start 2024-03-24T01:32:25.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.498+0000 7ff106000640 1 -- start start 2024-03-24T01:32:25.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.500+0000 7ff106000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff100159620 0x7ff10006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:25.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.500+0000 7ff106000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff1000cf150 0x7ff1000ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:25.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.500+0000 7ff106000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff10006bca0 con 0x7ff10006b600 2024-03-24T01:32:25.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.500+0000 7ff106000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1000b4470 con 0x7ff1000cf150 2024-03-24T01:32:25.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.500+0000 7ff106000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1000b36f0 con 0x7ff100159620 2024-03-24T01:32:25.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.500+0000 7ff0ffe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff1000cf150 0x7ff1000ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:25.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.500+0000 7ff0fea00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff10006b600 0x7ff1000cea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50770/0 (socket says 172.21.15.90:50770) 2024-03-24T01:32:25.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.500+0000 7ff0ffe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff1000cf150 0x7ff1000ae530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54084/0 (socket says 172.21.15.90:54084) 2024-03-24T01:32:25.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.500+0000 7ff0ffe00640 1 -- 172.21.15.90:0/4045125257 learned_addr learned my addr 172.21.15.90:0/4045125257 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:25.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.500+0000 7ff0ff400640 1 --2- 172.21.15.90:0/4045125257 >> v2:172.21.15.120:3300/0 conn(0x7ff100159620 0x7ff10006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:25.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.500+0000 7ff0ffe00640 1 -- 172.21.15.90:0/4045125257 >> v2:172.21.15.120:3300/0 conn(0x7ff100159620 msgr2=0x7ff10006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:25.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.500+0000 7ff0ffe00640 1 --2- 172.21.15.90:0/4045125257 >> v2:172.21.15.120:3300/0 conn(0x7ff100159620 0x7ff10006b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:25.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.501+0000 7ff0ffe00640 1 -- 172.21.15.90:0/4045125257 >> v1:172.21.15.90:6789/0 conn(0x7ff10006b600 legacy=0x7ff1000cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:25.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.501+0000 7ff0ffe00640 1 -- 172.21.15.90:0/4045125257 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1000b3990 con 0x7ff1000cf150 2024-03-24T01:32:25.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.502+0000 7ff0ffe00640 1 --2- 172.21.15.90:0/4045125257 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1000cf150 0x7ff1000ae530 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7ff10006c2d0 tx=0x7ff0f802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a68152a44eae8763 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:25.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.502+0000 7ff0fe000640 1 -- 172.21.15.90:0/4045125257 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f802ca80 con 0x7ff1000cf150 2024-03-24T01:32:25.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.502+0000 7ff0fe000640 1 -- 172.21.15.90:0/4045125257 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0f802cc20 con 0x7ff1000cf150 2024-03-24T01:32:25.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.502+0000 7ff0fe000640 1 -- 172.21.15.90:0/4045125257 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f80343f0 con 0x7ff1000cf150 2024-03-24T01:32:25.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.503+0000 7ff106000640 1 -- 172.21.15.90:0/4045125257 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1000cf150 msgr2=0x7ff1000ae530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:25.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.503+0000 7ff106000640 1 --2- 172.21.15.90:0/4045125257 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1000cf150 0x7ff1000ae530 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7ff10006c2d0 tx=0x7ff0f802bee0 comp rx=0 tx=0).stop 2024-03-24T01:32:25.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.503+0000 7ff106000640 1 -- 172.21.15.90:0/4045125257 shutdown_connections 2024-03-24T01:32:25.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.503+0000 7ff106000640 1 --2- 172.21.15.90:0/4045125257 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1000cf150 0x7ff1000ae530 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:25.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.503+0000 7ff106000640 1 --2- 172.21.15.90:0/4045125257 >> v2:172.21.15.120:3300/0 conn(0x7ff100159620 0x7ff10006b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:25.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.503+0000 7ff106000640 1 -- 172.21.15.90:0/4045125257 >> 172.21.15.90:0/4045125257 conn(0x7ff1000b1770 msgr2=0x7ff1000b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:25.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.503+0000 7ff106000640 1 -- 172.21.15.90:0/4045125257 shutdown_connections 2024-03-24T01:32:25.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.503+0000 7ff106000640 1 -- 172.21.15.90:0/4045125257 wait complete. 2024-03-24T01:32:25.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.505+0000 7ff106000640 1 Processor -- start 2024-03-24T01:32:25.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.505+0000 7ff106000640 1 -- start start 2024-03-24T01:32:25.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.506+0000 7ff106000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff10006b600 0x7ff100144570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:25.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.506+0000 7ff106000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1000cf150 0x7ff10014f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:25.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.506+0000 7ff106000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff100159620 0x7ff10014fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:25.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.506+0000 7ff106000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff100159a00 con 0x7ff10006b600 2024-03-24T01:32:25.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.506+0000 7ff106000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1000b3f50 con 0x7ff1000cf150 2024-03-24T01:32:25.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.506+0000 7ff106000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff10015bfe0 con 0x7ff100159620 2024-03-24T01:32:25.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.506+0000 7ff0ff400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff10006b600 0x7ff100144570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:25.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.506+0000 7ff0ff400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff10006b600 0x7ff100144570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54094/0 (socket says 172.21.15.90:54094) 2024-03-24T01:32:25.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.506+0000 7ff0ff400640 1 -- 172.21.15.90:0/1194595934 learned_addr learned my addr 172.21.15.90:0/1194595934 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:25.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.506+0000 7ff0ff400640 1 -- 172.21.15.90:0/1194595934 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff100159620 msgr2=0x7ff10014fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:25.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.506+0000 7ff0ff400640 1 --2- 172.21.15.90:0/1194595934 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff100159620 0x7ff10014fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:25.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.506+0000 7ff0ff400640 1 -- 172.21.15.90:0/1194595934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1000cf150 msgr2=0x7ff10014f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:25.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.506+0000 7ff0ff400640 1 --2- 172.21.15.90:0/1194595934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1000cf150 0x7ff10014f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:25.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.506+0000 7ff0ff400640 1 -- 172.21.15.90:0/1194595934 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff100144ab0 con 0x7ff10006b600 2024-03-24T01:32:25.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.507+0000 7ff0ff400640 1 --2- 172.21.15.90:0/1194595934 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff10006b600 0x7ff100144570 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7ff0f000d620 tx=0x7ff0f000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:25.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.507+0000 7ff0f7400640 1 -- 172.21.15.90:0/1194595934 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f000b620 con 0x7ff10006b600 2024-03-24T01:32:25.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.507+0000 7ff0f7400640 1 -- 172.21.15.90:0/1194595934 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0f0019070 con 0x7ff10006b600 2024-03-24T01:32:25.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.507+0000 7ff0f7400640 1 -- 172.21.15.90:0/1194595934 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f000ba00 con 0x7ff10006b600 2024-03-24T01:32:25.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.507+0000 7ff106000640 1 -- 172.21.15.90:0/1194595934 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff100153a50 con 0x7ff10006b600 2024-03-24T01:32:25.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.507+0000 7ff106000640 1 -- 172.21.15.90:0/1194595934 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff100154010 con 0x7ff10006b600 2024-03-24T01:32:25.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.508+0000 7ff106000640 1 -- 172.21.15.90:0/1194595934 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff10004f400 con 0x7ff10006b600 2024-03-24T01:32:25.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.509+0000 7ff0f7400640 1 -- 172.21.15.90:0/1194595934 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff0f000bba0 con 0x7ff10006b600 2024-03-24T01:32:25.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.511+0000 7ff0f7400640 1 --2- 172.21.15.90:0/1194595934 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff0e006bbd0 0x7ff0e006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:25.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.511+0000 7ff0f7400640 1 -- 172.21.15.90:0/1194595934 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x7ff0f008b120 con 0x7ff10006b600 2024-03-24T01:32:25.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.511+0000 7ff0fea00640 1 --2- 172.21.15.90:0/1194595934 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff0e006bbd0 0x7ff0e006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:25.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.512+0000 7ff0fea00640 1 --2- 172.21.15.90:0/1194595934 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff0e006bbd0 0x7ff0e006e090 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec000f60 tx=0x7ff0ec007a00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:25.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.514+0000 7ff0f7400640 1 -- 172.21.15.90:0/1194595934 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff100144ab0 con 0x7ff10006b600 2024-03-24T01:32:25.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.744+0000 7ff106000640 1 -- 172.21.15.90:0/1194595934 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff1000b2e90 con 0x7ff10006b600 2024-03-24T01:32:25.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.745+0000 7ff0f7400640 1 -- 172.21.15.90:0/1194595934 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v56) v1 ==== 80+0+122 (secure 0 0 0) 0x7ff0f0059950 con 0x7ff10006b600 2024-03-24T01:32:25.745 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.120:6836/266172244 2024-03-25T01:29:08.935482+0000 2024-03-24T01:32:25.745 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.120:6834/266172244 2024-03-25T01:29:08.935482+0000 2024-03-24T01:32:25.745 INFO:teuthology.orchestra.run.smithi090.stderr:listed 2 entries 2024-03-24T01:32:25.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.747+0000 7ff106000640 1 -- 172.21.15.90:0/1194595934 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff0e006bbd0 msgr2=0x7ff0e006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:25.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.747+0000 7ff106000640 1 --2- 172.21.15.90:0/1194595934 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff0e006bbd0 0x7ff0e006e090 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec000f60 tx=0x7ff0ec007a00 comp rx=0 tx=0).stop 2024-03-24T01:32:25.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.747+0000 7ff106000640 1 -- 172.21.15.90:0/1194595934 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff10006b600 msgr2=0x7ff100144570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:25.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.747+0000 7ff106000640 1 --2- 172.21.15.90:0/1194595934 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff10006b600 0x7ff100144570 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7ff0f000d620 tx=0x7ff0f000dae0 comp rx=0 tx=0).stop 2024-03-24T01:32:25.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.747+0000 7ff106000640 1 -- 172.21.15.90:0/1194595934 shutdown_connections 2024-03-24T01:32:25.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.747+0000 7ff106000640 1 --2- 172.21.15.90:0/1194595934 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff0e006bbd0 0x7ff0e006e090 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:25.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.747+0000 7ff106000640 1 --2- 172.21.15.90:0/1194595934 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff100159620 0x7ff10014fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:25.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.747+0000 7ff106000640 1 --2- 172.21.15.90:0/1194595934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1000cf150 0x7ff10014f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:25.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.747+0000 7ff106000640 1 --2- 172.21.15.90:0/1194595934 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff10006b600 0x7ff100144570 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:25.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.747+0000 7ff106000640 1 -- 172.21.15.90:0/1194595934 >> 172.21.15.90:0/1194595934 conn(0x7ff1000b1770 msgr2=0x7ff10015e740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:25.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.748+0000 7ff106000640 1 -- 172.21.15.90:0/1194595934 shutdown_connections 2024-03-24T01:32:25.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:25.748+0000 7ff106000640 1 -- 172.21.15.90:0/1194595934 wait complete. 2024-03-24T01:32:25.765 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi090.front.sepia.ceph.com... 2024-03-24T01:32:25.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:32:25.766 DEBUG:teuthology.orchestra.run.smithi090:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-24T01:32:25.852 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[62045]: fuse finished with error 0 and tester_r 0 2024-03-24T01:32:25.856 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-03-24T01:32:25.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:32:25.857 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T01:32:26.058 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:32:26.058 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:32:26.109 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T01:32:26.109 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2024-03-24T01:32:26.110 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:32:26.110 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-24T01:32:26.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T01:32:26.207 INFO:teuthology.orchestra.run.smithi120.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-24T01:32:26.207 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:32:26.207 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:32:26.258 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053414,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498340k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249168k,nr_inodes=812292,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498340k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T01:32:26.259 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-03-24T01:32:26.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:32:26.259 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-24T01:32:26.348 INFO:teuthology.orchestra.run.smithi090.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-24T01:32:26.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T01:32:26.349 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:32:26.349 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:32:26.407 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T01:32:26.407 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2024-03-24T01:32:26.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:32:26.408 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-24T01:32:26.454 INFO:teuthology.orchestra.run.smithi120.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-24T01:32:26.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T01:32:26.456 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:32:26.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.668+0000 7f611f600640 1 Processor -- start 2024-03-24T01:32:26.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.669+0000 7f611f600640 1 -- start start 2024-03-24T01:32:26.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.670+0000 7f611f600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6118155030 0x7f611815efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:26.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.670+0000 7f611f600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f611815f500 con 0x7f611806b7d0 2024-03-24T01:32:26.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.670+0000 7f611f600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f611815bf70 con 0x7f61180cdac0 2024-03-24T01:32:26.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.670+0000 7f611f600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f611815b360 con 0x7f6118155030 2024-03-24T01:32:26.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.671+0000 7f611d600640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f611806b7d0 0x7f61180ae530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50778/0 (socket says 172.21.15.90:50778) 2024-03-24T01:32:26.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.671+0000 7f611d600640 1 -- 172.21.15.90:0/2612051720 learned_addr learned my addr 172.21.15.90:0/2612051720 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:32:26.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.671+0000 7f611cc00640 1 --2- 172.21.15.90:0/2612051720 >> v2:172.21.15.120:3300/0 conn(0x7f6118155030 0x7f611815efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:26.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.671+0000 7f6117400640 1 -- 172.21.15.90:0/2612051720 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1709042802 0 0) 0x7f6110001490 con 0x7f611806b7d0 2024-03-24T01:32:26.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.671+0000 7f6117400640 1 -- 172.21.15.90:0/2612051720 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6100003520 con 0x7f611806b7d0 2024-03-24T01:32:26.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.672+0000 7f6117400640 1 -- 172.21.15.90:0/2612051720 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 940748099 0 0) 0x7f6100003520 con 0x7f611806b7d0 2024-03-24T01:32:26.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.672+0000 7f6117400640 1 -- 172.21.15.90:0/2612051720 >> v2:172.21.15.120:3300/0 conn(0x7f6118155030 msgr2=0x7f611815efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:26.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.672+0000 7f6117400640 1 --2- 172.21.15.90:0/2612051720 >> v2:172.21.15.120:3300/0 conn(0x7f6118155030 0x7f611815efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:26.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.672+0000 7f6117400640 1 -- 172.21.15.90:0/2612051720 >> v1:172.21.15.120:6789/0 conn(0x7f61180cdac0 legacy=0x7f611806b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:26.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.672+0000 7f6117400640 1 -- 172.21.15.90:0/2612051720 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f611815b640 con 0x7f611806b7d0 2024-03-24T01:32:26.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.672+0000 7f6117400640 1 -- 172.21.15.90:0/2612051720 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f6110001bc0 con 0x7f611806b7d0 2024-03-24T01:32:26.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.672+0000 7f6117400640 1 -- 172.21.15.90:0/2612051720 >> v1:172.21.15.90:6789/0 conn(0x7f611806b7d0 legacy=0x7f61180ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:26.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.672+0000 7f6117400640 1 --2- 172.21.15.90:0/2612051720 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6100003a00 0x7f6100003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:26.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.672+0000 7f6117400640 1 --2- 172.21.15.90:0/2612051720 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6100004800 0x7f6100004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:26.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.672+0000 7f6117400640 1 --2- 172.21.15.90:0/2612051720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61000051c0 0x7f610000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:26.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.672+0000 7f6117400640 1 -- 172.21.15.90:0/2612051720 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611815b360 con 0x7f6100003a00 2024-03-24T01:32:26.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.672+0000 7f6117400640 1 -- 172.21.15.90:0/2612051720 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f610000c0c0 con 0x7f61000051c0 2024-03-24T01:32:26.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.672+0000 7f6117400640 1 -- 172.21.15.90:0/2612051720 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f610000c2c0 con 0x7f6100004800 2024-03-24T01:32:26.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.672+0000 7f611cc00640 1 --2- 172.21.15.90:0/2612051720 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6100003a00 0x7f6100003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:26.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.672+0000 7f611cc00640 1 -- 172.21.15.90:0/2612051720 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6100004800 msgr2=0x7f6100004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:26.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.672+0000 7f611cc00640 1 --2- 172.21.15.90:0/2612051720 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6100004800 0x7f6100004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:26.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.672+0000 7f611cc00640 1 -- 172.21.15.90:0/2612051720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61000051c0 msgr2=0x7f610000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:26.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.672+0000 7f611cc00640 1 --2- 172.21.15.90:0/2612051720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61000051c0 0x7f610000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:26.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.672+0000 7f611cc00640 1 -- 172.21.15.90:0/2612051720 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6100003520 con 0x7f6100003a00 2024-03-24T01:32:26.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.672+0000 7f611d600640 1 -- 172.21.15.90:0/2612051720 reap_dead start 2024-03-24T01:32:26.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.674+0000 7f611cc00640 1 --2- 172.21.15.90:0/2612051720 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6100003a00 0x7f6100003e20 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f610c009ac0 tx=0x7f6100005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84afe1d45b6916ed server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:26.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.674+0000 7f6117400640 1 -- 172.21.15.90:0/2612051720 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610c039070 con 0x7f6100003a00 2024-03-24T01:32:26.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.674+0000 7f6117400640 1 -- 172.21.15.90:0/2612051720 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f610c02c710 con 0x7f6100003a00 2024-03-24T01:32:26.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.674+0000 7f6117400640 1 -- 172.21.15.90:0/2612051720 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610c009690 con 0x7f6100003a00 2024-03-24T01:32:26.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.675+0000 7f611f600640 1 -- 172.21.15.90:0/2612051720 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6100003a00 msgr2=0x7f6100003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:26.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.675+0000 7f611f600640 1 --2- 172.21.15.90:0/2612051720 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6100003a00 0x7f6100003e20 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f610c009ac0 tx=0x7f6100005830 comp rx=0 tx=0).stop 2024-03-24T01:32:26.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.675+0000 7f611f600640 1 -- 172.21.15.90:0/2612051720 shutdown_connections 2024-03-24T01:32:26.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.675+0000 7f611f600640 1 --2- 172.21.15.90:0/2612051720 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6100003a00 0x7f6100003e20 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:26.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.675+0000 7f611f600640 1 -- 172.21.15.90:0/2612051720 >> 172.21.15.90:0/2612051720 conn(0x7f61180b1770 msgr2=0x7f61180b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:26.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.675+0000 7f611f600640 1 -- 172.21.15.90:0/2612051720 shutdown_connections 2024-03-24T01:32:26.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.675+0000 7f611f600640 1 -- 172.21.15.90:0/2612051720 wait complete. 2024-03-24T01:32:26.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.676+0000 7f611f600640 1 Processor -- start 2024-03-24T01:32:26.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.677+0000 7f611f600640 1 -- start start 2024-03-24T01:32:26.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.677+0000 7f611f600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6100003a00 0x7f61180cbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:26.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.677+0000 7f611f600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61180c8e10 0x7f6118157de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:26.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.677+0000 7f611f600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61181f3ef0 0x7f61181f43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:26.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.677+0000 7f611f600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6118155410 con 0x7f61180c8e10 2024-03-24T01:32:26.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.677+0000 7f611f600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611815bc00 con 0x7f6100003a00 2024-03-24T01:32:26.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.677+0000 7f611f600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f611815af70 con 0x7f61181f3ef0 2024-03-24T01:32:26.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.678+0000 7f6117e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61180c8e10 0x7f6118157de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:26.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.678+0000 7f6117e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61180c8e10 0x7f6118157de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54118/0 (socket says 172.21.15.90:54118) 2024-03-24T01:32:26.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.678+0000 7f6117e00640 1 -- 172.21.15.90:0/4084978210 learned_addr learned my addr 172.21.15.90:0/4084978210 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:26.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.678+0000 7f611d600640 1 --2- 172.21.15.90:0/4084978210 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61181f3ef0 0x7f61181f43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:26.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.678+0000 7f611cc00640 1 --2- 172.21.15.90:0/4084978210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6100003a00 0x7f61180cbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:26.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.678+0000 7f6117e00640 1 -- 172.21.15.90:0/4084978210 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61181f3ef0 msgr2=0x7f61181f43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:26.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.678+0000 7f6117e00640 1 --2- 172.21.15.90:0/4084978210 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61181f3ef0 0x7f61181f43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:26.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.678+0000 7f6117e00640 1 -- 172.21.15.90:0/4084978210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6100003a00 msgr2=0x7f61180cbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:26.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.678+0000 7f6117e00640 1 --2- 172.21.15.90:0/4084978210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6100003a00 0x7f61180cbdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:26.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.678+0000 7f6117e00640 1 -- 172.21.15.90:0/4084978210 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61180cc300 con 0x7f61180c8e10 2024-03-24T01:32:26.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.679+0000 7f6117e00640 1 --2- 172.21.15.90:0/4084978210 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61180c8e10 0x7f6118157de0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f610400c430 tx=0x7f610400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:26.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.679+0000 7f6114c00640 1 -- 172.21.15.90:0/4084978210 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6104017070 con 0x7f61180c8e10 2024-03-24T01:32:26.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.679+0000 7f6114c00640 1 -- 172.21.15.90:0/4084978210 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61040052f0 con 0x7f61180c8e10 2024-03-24T01:32:26.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.679+0000 7f6114c00640 1 -- 172.21.15.90:0/4084978210 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6104003830 con 0x7f61180c8e10 2024-03-24T01:32:26.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.679+0000 7f611f600640 1 -- 172.21.15.90:0/4084978210 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f611815cd80 con 0x7f61180c8e10 2024-03-24T01:32:26.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.679+0000 7f611f600640 1 -- 172.21.15.90:0/4084978210 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f611815d340 con 0x7f61180c8e10 2024-03-24T01:32:26.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.681+0000 7f611f600640 1 -- 172.21.15.90:0/4084978210 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6118068060 con 0x7f61180c8e10 2024-03-24T01:32:26.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.681+0000 7f6114c00640 1 -- 172.21.15.90:0/4084978210 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f61040039d0 con 0x7f61180c8e10 2024-03-24T01:32:26.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.683+0000 7f6114c00640 1 --2- 172.21.15.90:0/4084978210 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f60f806bc00 0x7f60f806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:26.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.683+0000 7f6114c00640 1 -- 172.21.15.90:0/4084978210 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f6104089f50 con 0x7f61180c8e10 2024-03-24T01:32:26.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.683+0000 7f611cc00640 1 --2- 172.21.15.90:0/4084978210 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f60f806bc00 0x7f60f806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:26.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.685+0000 7f611cc00640 1 --2- 172.21.15.90:0/4084978210 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f60f806bc00 0x7f60f806e0c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f610c036000 tx=0x7f610c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:26.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.687+0000 7f6114c00640 1 -- 172.21.15.90:0/4084978210 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f61040588d0 con 0x7f61180c8e10 2024-03-24T01:32:26.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.923+0000 7f611f600640 1 -- 172.21.15.90:0/4084978210 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61180cc6d0 con 0x7f61180c8e10 2024-03-24T01:32:26.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.924+0000 7f6114c00640 1 -- 172.21.15.90:0/4084978210 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f61040581d0 con 0x7f61180c8e10 2024-03-24T01:32:26.926 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:32:26.926 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":41,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4985,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6834/3475131973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":3475131973},{"type":"v1","addr":"172.21.15.90:6834","nonce":3475131973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":4994,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/3151604088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":3151604088},{"type":"v1","addr":"172.21.15.120:6837","nonce":3151604088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5428,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/1239583677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1239583677},{"type":"v1","addr":"172.21.15.120:6836","nonce":1239583677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T01:27:13.961373+0000","modified":"2024-03-24T01:30:40.234926+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6219},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6219":{"gid":6219,"name":"c","rank":0,"incarnation":38,"state":"up:active","state_seq":56,"addr":"172.21.15.90:6835/226826781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":226826781},{"type":"v1","addr":"172.21.15.90:6835","nonce":226826781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6219,"qdb_cluster":[6219]},"id":3}]} 2024-03-24T01:32:26.926 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 41 2024-03-24T01:32:26.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.928+0000 7f611f600640 1 -- 172.21.15.90:0/4084978210 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f60f806bc00 msgr2=0x7f60f806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:26.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.928+0000 7f611f600640 1 --2- 172.21.15.90:0/4084978210 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f60f806bc00 0x7f60f806e0c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f610c036000 tx=0x7f610c030000 comp rx=0 tx=0).stop 2024-03-24T01:32:26.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.928+0000 7f611f600640 1 -- 172.21.15.90:0/4084978210 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61180c8e10 msgr2=0x7f6118157de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:26.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.928+0000 7f611f600640 1 --2- 172.21.15.90:0/4084978210 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61180c8e10 0x7f6118157de0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f610400c430 tx=0x7f610400c8f0 comp rx=0 tx=0).stop 2024-03-24T01:32:26.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.928+0000 7f611f600640 1 -- 172.21.15.90:0/4084978210 shutdown_connections 2024-03-24T01:32:26.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.928+0000 7f611f600640 1 --2- 172.21.15.90:0/4084978210 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f60f806bc00 0x7f60f806e0c0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:26.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.928+0000 7f611f600640 1 --2- 172.21.15.90:0/4084978210 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61180c8e10 0x7f6118157de0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:26.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.928+0000 7f611f600640 1 --2- 172.21.15.90:0/4084978210 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f61181f3ef0 0x7f61181f43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:26.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.928+0000 7f611f600640 1 --2- 172.21.15.90:0/4084978210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6100003a00 0x7f61180cbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:26.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.928+0000 7f611f600640 1 -- 172.21.15.90:0/4084978210 >> 172.21.15.90:0/4084978210 conn(0x7f61180b1770 msgr2=0x7f611815ec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:26.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.928+0000 7f611f600640 1 -- 172.21.15.90:0/4084978210 shutdown_connections 2024-03-24T01:32:26.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:26.928+0000 7f611f600640 1 -- 172.21.15.90:0/4084978210 wait complete. 2024-03-24T01:32:26.950 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:32:27.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.148+0000 7fa2a8a00640 1 Processor -- start 2024-03-24T01:32:27.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.148+0000 7fa2a8a00640 1 -- start start 2024-03-24T01:32:27.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.150+0000 7fa2a8a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa2a415b8c0 0x7fa2a40cf910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:27.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.150+0000 7fa2a8a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2a40cff10 con 0x7fa2a415bca0 2024-03-24T01:32:27.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.150+0000 7fa2a8a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2a40d01f0 con 0x7fa2a415c6d0 2024-03-24T01:32:27.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.150+0000 7fa2a8a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2a40d04d0 con 0x7fa2a415b8c0 2024-03-24T01:32:27.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.150+0000 7fa2a1e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa2a415bca0 0x7fa2a40d1090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50794/0 (socket says 172.21.15.90:50794) 2024-03-24T01:32:27.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.150+0000 7fa2a1e00640 1 -- 172.21.15.90:0/3750671312 learned_addr learned my addr 172.21.15.90:0/3750671312 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:32:27.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.150+0000 7fa2a2800640 1 --2- 172.21.15.90:0/3750671312 >> v2:172.21.15.90:3300/0 conn(0x7fa2a415b8c0 0x7fa2a40cf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:27.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.151+0000 7fa2a1400640 1 -- 172.21.15.90:0/3750671312 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 459149281 0 0) 0x7fa2a40cff10 con 0x7fa2a415bca0 2024-03-24T01:32:27.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.151+0000 7fa2a1400640 1 -- 172.21.15.90:0/3750671312 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa288003520 con 0x7fa2a415bca0 2024-03-24T01:32:27.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.151+0000 7fa2a2800640 1 -- 172.21.15.90:0/3750671312 >> v1:172.21.15.120:6789/0 conn(0x7fa2a415c6d0 legacy=0x7fa2a406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:27.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.151+0000 7fa2a2800640 1 -- 172.21.15.90:0/3750671312 >> v1:172.21.15.90:6789/0 conn(0x7fa2a415bca0 legacy=0x7fa2a40d1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:27.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.151+0000 7fa2a2800640 1 -- 172.21.15.90:0/3750671312 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a415c1b0 con 0x7fa2a415b8c0 2024-03-24T01:32:27.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.152+0000 7fa2a2800640 1 --2- 172.21.15.90:0/3750671312 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2a415b8c0 0x7fa2a40cf910 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fa2940097a0 tx=0x7fa29402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94031c3f605d0c39 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:27.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.152+0000 7fa2a1400640 1 -- 172.21.15.90:0/3750671312 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa294034070 con 0x7fa2a415b8c0 2024-03-24T01:32:27.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.152+0000 7fa2a1400640 1 -- 172.21.15.90:0/3750671312 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa294030610 con 0x7fa2a415b8c0 2024-03-24T01:32:27.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.152+0000 7fa2a1400640 1 -- 172.21.15.90:0/3750671312 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2940309c0 con 0x7fa2a415b8c0 2024-03-24T01:32:27.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.153+0000 7fa2a8a00640 1 -- 172.21.15.90:0/3750671312 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2a415b8c0 msgr2=0x7fa2a40cf910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:27.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.153+0000 7fa2a8a00640 1 --2- 172.21.15.90:0/3750671312 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2a415b8c0 0x7fa2a40cf910 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fa2940097a0 tx=0x7fa29402b960 comp rx=0 tx=0).stop 2024-03-24T01:32:27.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.153+0000 7fa2a8a00640 1 -- 172.21.15.90:0/3750671312 shutdown_connections 2024-03-24T01:32:27.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.153+0000 7fa2a8a00640 1 --2- 172.21.15.90:0/3750671312 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2a415b8c0 0x7fa2a40cf910 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:27.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.153+0000 7fa2a8a00640 1 -- 172.21.15.90:0/3750671312 >> 172.21.15.90:0/3750671312 conn(0x7fa2a40b2410 msgr2=0x7fa2a40b27f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:27.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.153+0000 7fa2a8a00640 1 -- 172.21.15.90:0/3750671312 shutdown_connections 2024-03-24T01:32:27.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.153+0000 7fa2a8a00640 1 -- 172.21.15.90:0/3750671312 wait complete. 2024-03-24T01:32:27.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.155+0000 7fa2a8a00640 1 Processor -- start 2024-03-24T01:32:27.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.156+0000 7fa2a8a00640 1 -- start start 2024-03-24T01:32:27.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.157+0000 7fa2a8a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2a415b8c0 0x7fa2a41479a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:27.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.157+0000 7fa2a8a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2a415bca0 0x7fa2a40cdc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:27.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.157+0000 7fa2a8a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2a415c6d0 0x7fa2a4155310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:27.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.157+0000 7fa2a8a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a415cb00 con 0x7fa2a415c6d0 2024-03-24T01:32:27.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.157+0000 7fa2a8a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a415cc80 con 0x7fa2a415b8c0 2024-03-24T01:32:27.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.157+0000 7fa2a8a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a415ce00 con 0x7fa2a415bca0 2024-03-24T01:32:27.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.158+0000 7fa2a3200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2a415c6d0 0x7fa2a4155310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:27.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.158+0000 7fa2a3200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2a415c6d0 0x7fa2a4155310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54144/0 (socket says 172.21.15.90:54144) 2024-03-24T01:32:27.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.158+0000 7fa2a3200640 1 -- 172.21.15.90:0/1851540523 learned_addr learned my addr 172.21.15.90:0/1851540523 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:27.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.158+0000 7fa2a2800640 1 --2- 172.21.15.90:0/1851540523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2a415b8c0 0x7fa2a41479a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:27.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.158+0000 7fa2a3200640 1 -- 172.21.15.90:0/1851540523 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2a415bca0 msgr2=0x7fa2a40cdc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:27.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.158+0000 7fa2a3200640 1 --2- 172.21.15.90:0/1851540523 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2a415bca0 0x7fa2a40cdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:27.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.158+0000 7fa2a3200640 1 -- 172.21.15.90:0/1851540523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2a415b8c0 msgr2=0x7fa2a41479a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:27.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.158+0000 7fa2a3200640 1 --2- 172.21.15.90:0/1851540523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2a415b8c0 0x7fa2a41479a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:27.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.158+0000 7fa2a3200640 1 -- 172.21.15.90:0/1851540523 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a4147ee0 con 0x7fa2a415c6d0 2024-03-24T01:32:27.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.158+0000 7fa2a3200640 1 --2- 172.21.15.90:0/1851540523 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2a415c6d0 0x7fa2a4155310 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fa29c00c7a0 tx=0x7fa29c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:27.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.159+0000 7fa29aa00640 1 -- 172.21.15.90:0/1851540523 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa29c003b70 con 0x7fa2a415c6d0 2024-03-24T01:32:27.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.159+0000 7fa29aa00640 1 -- 172.21.15.90:0/1851540523 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa29c003d30 con 0x7fa2a415c6d0 2024-03-24T01:32:27.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.159+0000 7fa29aa00640 1 -- 172.21.15.90:0/1851540523 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa29c00aa90 con 0x7fa2a415c6d0 2024-03-24T01:32:27.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.159+0000 7fa2a8a00640 1 -- 172.21.15.90:0/1851540523 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2a40ce150 con 0x7fa2a415c6d0 2024-03-24T01:32:27.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.159+0000 7fa2a8a00640 1 -- 172.21.15.90:0/1851540523 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2a415f3d0 con 0x7fa2a415c6d0 2024-03-24T01:32:27.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.160+0000 7fa2a8a00640 1 -- 172.21.15.90:0/1851540523 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2a404f400 con 0x7fa2a415c6d0 2024-03-24T01:32:27.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.163+0000 7fa29aa00640 1 -- 172.21.15.90:0/1851540523 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa29c00ac30 con 0x7fa2a415c6d0 2024-03-24T01:32:27.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.163+0000 7fa29aa00640 1 --2- 172.21.15.90:0/1851540523 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa28406bbd0 0x7fa28406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:27.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.164+0000 7fa29aa00640 1 -- 172.21.15.90:0/1851540523 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fa29c08ac70 con 0x7fa2a415c6d0 2024-03-24T01:32:27.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.164+0000 7fa2a2800640 1 --2- 172.21.15.90:0/1851540523 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa28406bbd0 0x7fa28406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:27.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.165+0000 7fa2a2800640 1 --2- 172.21.15.90:0/1851540523 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa28406bbd0 0x7fa28406e090 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa294036000 tx=0x7fa294008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:27.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.168+0000 7fa29aa00640 1 -- 172.21.15.90:0/1851540523 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa2a4147ee0 con 0x7fa2a415c6d0 2024-03-24T01:32:27.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.366+0000 7fa2a8a00640 1 -- 172.21.15.90:0/1851540523 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2a40b0700 con 0x7fa2a415c6d0 2024-03-24T01:32:27.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.369+0000 7fa29aa00640 1 -- 172.21.15.90:0/1851540523 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+17087 (secure 0 0 0) 0x7fa29c0594a0 con 0x7fa2a415c6d0 2024-03-24T01:32:27.370 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:32:27.370 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":56,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:29:08.935541+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:27:11.774313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-24T01:27:12.992083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.23","mappings":[{"from":0,"to":1}]},{"pgid":"8.37","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.120:6836/266172244":"2024-03-25T01:29:08.935482+0000","172.21.15.120:6834/266172244":"2024-03-25T01:29:08.935482+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:32:27.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.371+0000 7fa2a8a00640 1 -- 172.21.15.90:0/1851540523 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa28406bbd0 msgr2=0x7fa28406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:27.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.371+0000 7fa2a8a00640 1 --2- 172.21.15.90:0/1851540523 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa28406bbd0 0x7fa28406e090 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa294036000 tx=0x7fa294008030 comp rx=0 tx=0).stop 2024-03-24T01:32:27.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.371+0000 7fa2a8a00640 1 -- 172.21.15.90:0/1851540523 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2a415c6d0 msgr2=0x7fa2a4155310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:27.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.371+0000 7fa2a8a00640 1 --2- 172.21.15.90:0/1851540523 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2a415c6d0 0x7fa2a4155310 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fa29c00c7a0 tx=0x7fa29c00cc60 comp rx=0 tx=0).stop 2024-03-24T01:32:27.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.371+0000 7fa2a8a00640 1 -- 172.21.15.90:0/1851540523 shutdown_connections 2024-03-24T01:32:27.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.371+0000 7fa2a8a00640 1 --2- 172.21.15.90:0/1851540523 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa28406bbd0 0x7fa28406e090 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:27.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.371+0000 7fa2a8a00640 1 --2- 172.21.15.90:0/1851540523 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2a415c6d0 0x7fa2a4155310 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:27.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.371+0000 7fa2a8a00640 1 --2- 172.21.15.90:0/1851540523 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa2a415bca0 0x7fa2a40cdc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:27.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.371+0000 7fa2a8a00640 1 --2- 172.21.15.90:0/1851540523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2a415b8c0 0x7fa2a41479a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:27.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.371+0000 7fa2a8a00640 1 -- 172.21.15.90:0/1851540523 >> 172.21.15.90:0/1851540523 conn(0x7fa2a40b2410 msgr2=0x7fa2a40b27f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:27.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.371+0000 7fa2a8a00640 1 -- 172.21.15.90:0/1851540523 shutdown_connections 2024-03-24T01:32:27.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.371+0000 7fa2a8a00640 1 -- 172.21.15.90:0/1851540523 wait complete. 2024-03-24T01:32:27.391 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T01:27:11.774313+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T01:27:12.992083+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T01:32:27.392 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:32:27.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.588+0000 7fe7b8e00640 1 Processor -- start 2024-03-24T01:32:27.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.588+0000 7fe7b8e00640 1 -- start start 2024-03-24T01:32:27.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.590+0000 7fe7b8e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe7b406cbe0 0x7fe7b40cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:27.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.590+0000 7fe7b8e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7b40d0510 con 0x7fe7b415e0e0 2024-03-24T01:32:27.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.590+0000 7fe7b8e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7b415e4c0 con 0x7fe7b40d00a0 2024-03-24T01:32:27.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.590+0000 7fe7b8e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7b40b1350 con 0x7fe7b406cbe0 2024-03-24T01:32:27.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.590+0000 7fe7a9400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe7b406cbe0 0x7fe7b40cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:27.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.590+0000 7fe7b1e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe7b415e0e0 0x7fe7b40b2080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50810/0 (socket says 172.21.15.90:50810) 2024-03-24T01:32:27.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.590+0000 7fe7b1e00640 1 -- 172.21.15.90:0/895383877 learned_addr learned my addr 172.21.15.90:0/895383877 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:32:27.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.591+0000 7fe7a9400640 1 -- 172.21.15.90:0/895383877 >> v1:172.21.15.120:6789/0 conn(0x7fe7b40d00a0 legacy=0x7fe7b4099c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:27.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.591+0000 7fe7a9400640 1 -- 172.21.15.90:0/895383877 >> v1:172.21.15.90:6789/0 conn(0x7fe7b415e0e0 legacy=0x7fe7b40b2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:27.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.591+0000 7fe7a9400640 1 -- 172.21.15.90:0/895383877 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b406d330 con 0x7fe7b406cbe0 2024-03-24T01:32:27.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.591+0000 7fe7b1400640 1 -- 172.21.15.90:0/895383877 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3626073702 0 0) 0x7fe7a4001110 con 0x7fe7b415e0e0 2024-03-24T01:32:27.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.592+0000 7fe7a9400640 1 --2- 172.21.15.90:0/895383877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7b406cbe0 0x7fe7b40cfb60 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fe79c0097a0 tx=0x7fe79c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb724c282c069144 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:27.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.593+0000 7fe7b1400640 1 -- 172.21.15.90:0/895383877 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe79c035020 con 0x7fe7b406cbe0 2024-03-24T01:32:27.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.593+0000 7fe7b1400640 1 -- 172.21.15.90:0/895383877 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe79c031830 con 0x7fe7b406cbe0 2024-03-24T01:32:27.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.593+0000 7fe7b1400640 1 -- 172.21.15.90:0/895383877 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe79c031c00 con 0x7fe7b406cbe0 2024-03-24T01:32:27.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.593+0000 7fe7b8e00640 1 -- 172.21.15.90:0/895383877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7b406cbe0 msgr2=0x7fe7b40cfb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:27.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.593+0000 7fe7b8e00640 1 --2- 172.21.15.90:0/895383877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7b406cbe0 0x7fe7b40cfb60 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fe79c0097a0 tx=0x7fe79c02b8d0 comp rx=0 tx=0).stop 2024-03-24T01:32:27.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.593+0000 7fe7b8e00640 1 -- 172.21.15.90:0/895383877 shutdown_connections 2024-03-24T01:32:27.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.593+0000 7fe7b8e00640 1 --2- 172.21.15.90:0/895383877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7b406cbe0 0x7fe7b40cfb60 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:27.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.593+0000 7fe7b8e00640 1 -- 172.21.15.90:0/895383877 >> 172.21.15.90:0/895383877 conn(0x7fe7b40b34e0 msgr2=0x7fe7b40b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:27.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.594+0000 7fe7b8e00640 1 -- 172.21.15.90:0/895383877 shutdown_connections 2024-03-24T01:32:27.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.594+0000 7fe7b8e00640 1 -- 172.21.15.90:0/895383877 wait complete. 2024-03-24T01:32:27.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.596+0000 7fe7b8e00640 1 Processor -- start 2024-03-24T01:32:27.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.596+0000 7fe7b8e00640 1 -- start start 2024-03-24T01:32:27.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.598+0000 7fe7b8e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7b406cbe0 0x7fe7b41581a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:27.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.598+0000 7fe7b8e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7b40d00a0 0x7fe7b40b5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:27.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.598+0000 7fe7b8e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7b415e0e0 0x7fe7b415c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:27.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.598+0000 7fe7b8e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b40b1b60 con 0x7fe7b406cbe0 2024-03-24T01:32:27.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.598+0000 7fe7b8e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b40b1880 con 0x7fe7b40d00a0 2024-03-24T01:32:27.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.598+0000 7fe7b8e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b406d040 con 0x7fe7b415e0e0 2024-03-24T01:32:27.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.598+0000 7fe7b1e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7b406cbe0 0x7fe7b41581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:27.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.598+0000 7fe7b1e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7b406cbe0 0x7fe7b41581a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54170/0 (socket says 172.21.15.90:54170) 2024-03-24T01:32:27.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.598+0000 7fe7b1e00640 1 -- 172.21.15.90:0/2867300995 learned_addr learned my addr 172.21.15.90:0/2867300995 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:27.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.599+0000 7fe7a9400640 1 --2- 172.21.15.90:0/2867300995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7b40d00a0 0x7fe7b40b5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:27.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.599+0000 7fe7b1e00640 1 -- 172.21.15.90:0/2867300995 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7b415e0e0 msgr2=0x7fe7b415c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:27.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.599+0000 7fe7b1e00640 1 --2- 172.21.15.90:0/2867300995 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7b415e0e0 0x7fe7b415c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:27.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.599+0000 7fe7b1e00640 1 -- 172.21.15.90:0/2867300995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7b40d00a0 msgr2=0x7fe7b40b5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:27.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.599+0000 7fe7b1e00640 1 --2- 172.21.15.90:0/2867300995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7b40d00a0 0x7fe7b40b5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:27.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.599+0000 7fe7b1e00640 1 -- 172.21.15.90:0/2867300995 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b41f8150 con 0x7fe7b406cbe0 2024-03-24T01:32:27.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.599+0000 7fe7a9400640 1 --2- 172.21.15.90:0/2867300995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7b40d00a0 0x7fe7b40b5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:27.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.599+0000 7fe7b1e00640 1 --2- 172.21.15.90:0/2867300995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7b406cbe0 0x7fe7b41581a0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fe7a400c430 tx=0x7fe7a400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:27.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.599+0000 7fe7aaa00640 1 -- 172.21.15.90:0/2867300995 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a40074e0 con 0x7fe7b406cbe0 2024-03-24T01:32:27.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.599+0000 7fe7aaa00640 1 -- 172.21.15.90:0/2867300995 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a40047e0 con 0x7fe7b406cbe0 2024-03-24T01:32:27.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.599+0000 7fe7aaa00640 1 -- 172.21.15.90:0/2867300995 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4004bd0 con 0x7fe7b406cbe0 2024-03-24T01:32:27.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.599+0000 7fe7b8e00640 1 -- 172.21.15.90:0/2867300995 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7b41f3bb0 con 0x7fe7b406cbe0 2024-03-24T01:32:27.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.600+0000 7fe7b8e00640 1 -- 172.21.15.90:0/2867300995 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7b41f4410 con 0x7fe7b406cbe0 2024-03-24T01:32:27.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.601+0000 7fe7b8e00640 1 -- 172.21.15.90:0/2867300995 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7b404f400 con 0x7fe7b406cbe0 2024-03-24T01:32:27.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.602+0000 7fe7aaa00640 1 -- 172.21.15.90:0/2867300995 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe7a4011070 con 0x7fe7b406cbe0 2024-03-24T01:32:27.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.603+0000 7fe7aaa00640 1 --2- 172.21.15.90:0/2867300995 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe79006bc00 0x7fe79006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:27.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.603+0000 7fe7aaa00640 1 -- 172.21.15.90:0/2867300995 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fe7a408a910 con 0x7fe7b406cbe0 2024-03-24T01:32:27.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.604+0000 7fe7a9400640 1 --2- 172.21.15.90:0/2867300995 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe79006bc00 0x7fe79006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:27.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.606+0000 7fe7a9400640 1 --2- 172.21.15.90:0/2867300995 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe79006bc00 0x7fe79006e0c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fe79c036000 tx=0x7fe79c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:27.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.608+0000 7fe7aaa00640 1 -- 172.21.15.90:0/2867300995 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe7b40d0510 con 0x7fe7b406cbe0 2024-03-24T01:32:27.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.804+0000 7fe7b8e00640 1 -- 172.21.15.90:0/2867300995 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe7b40b39f0 con 0x7fe7b406cbe0 2024-03-24T01:32:27.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.805+0000 7fe7aaa00640 1 -- 172.21.15.90:0/2867300995 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe7a400e280 con 0x7fe7b406cbe0 2024-03-24T01:32:27.807 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:32:27.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.808+0000 7fe7b8e00640 1 -- 172.21.15.90:0/2867300995 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe79006bc00 msgr2=0x7fe79006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:27.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.808+0000 7fe7b8e00640 1 --2- 172.21.15.90:0/2867300995 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe79006bc00 0x7fe79006e0c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fe79c036000 tx=0x7fe79c008030 comp rx=0 tx=0).stop 2024-03-24T01:32:27.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.808+0000 7fe7b8e00640 1 -- 172.21.15.90:0/2867300995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7b406cbe0 msgr2=0x7fe7b41581a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:27.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.808+0000 7fe7b8e00640 1 --2- 172.21.15.90:0/2867300995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7b406cbe0 0x7fe7b41581a0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fe7a400c430 tx=0x7fe7a400c8f0 comp rx=0 tx=0).stop 2024-03-24T01:32:27.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.808+0000 7fe7b8e00640 1 -- 172.21.15.90:0/2867300995 shutdown_connections 2024-03-24T01:32:27.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.808+0000 7fe7b8e00640 1 --2- 172.21.15.90:0/2867300995 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe79006bc00 0x7fe79006e0c0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:27.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.808+0000 7fe7b8e00640 1 --2- 172.21.15.90:0/2867300995 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe7b415e0e0 0x7fe7b415c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:27.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.808+0000 7fe7b8e00640 1 --2- 172.21.15.90:0/2867300995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7b40d00a0 0x7fe7b40b5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:27.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.808+0000 7fe7b8e00640 1 --2- 172.21.15.90:0/2867300995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7b406cbe0 0x7fe7b41581a0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:27.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.808+0000 7fe7b8e00640 1 -- 172.21.15.90:0/2867300995 >> 172.21.15.90:0/2867300995 conn(0x7fe7b40b34e0 msgr2=0x7fe7b415af70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:27.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.808+0000 7fe7b8e00640 1 -- 172.21.15.90:0/2867300995 shutdown_connections 2024-03-24T01:32:27.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:27.808+0000 7fe7b8e00640 1 -- 172.21.15.90:0/2867300995 wait complete. 2024-03-24T01:32:27.828 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T01:32:28.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.039+0000 7ff1dd400640 1 Processor -- start 2024-03-24T01:32:28.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.039+0000 7ff1dd400640 1 -- start start 2024-03-24T01:32:28.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.040+0000 7ff1dd400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff1d80b2e40 0x7ff1d8168c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:28.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.041+0000 7ff1dd400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1d80d20e0 con 0x7ff1d815dfd0 2024-03-24T01:32:28.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.041+0000 7ff1dd400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1d80ced70 con 0x7ff1d80d0b50 2024-03-24T01:32:28.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.041+0000 7ff1dd400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1d80cf050 con 0x7ff1d80b2e40 2024-03-24T01:32:28.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.041+0000 7ff1d7200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff1d80b2e40 0x7ff1d8168c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:28.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.041+0000 7ff1d5e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff1d815dfd0 0x7ff1d81622b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50822/0 (socket says 172.21.15.90:50822) 2024-03-24T01:32:28.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.041+0000 7ff1d5e00640 1 -- 172.21.15.90:0/1683502072 learned_addr learned my addr 172.21.15.90:0/1683502072 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:32:28.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.041+0000 7ff1d7200640 1 -- 172.21.15.90:0/1683502072 >> v1:172.21.15.120:6789/0 conn(0x7ff1d80d0b50 legacy=0x7ff1d80d0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:28.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.041+0000 7ff1d7200640 1 -- 172.21.15.90:0/1683502072 >> v1:172.21.15.90:6789/0 conn(0x7ff1d815dfd0 legacy=0x7ff1d81622b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:28.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.041+0000 7ff1d7200640 1 -- 172.21.15.90:0/1683502072 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d80d2380 con 0x7ff1d80b2e40 2024-03-24T01:32:28.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.041+0000 7ff1d5400640 1 -- 172.21.15.90:0/1683502072 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474802155 0 0) 0x7ff1d80d20e0 con 0x7ff1d815dfd0 2024-03-24T01:32:28.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.043+0000 7ff1d7200640 1 --2- 172.21.15.90:0/1683502072 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1d80b2e40 0x7ff1d8168c80 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0009d90 tx=0x7ff1d002bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2243bf6012841832 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:28.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.043+0000 7ff1d5400640 1 -- 172.21.15.90:0/1683502072 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d0039070 con 0x7ff1d80b2e40 2024-03-24T01:32:28.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.043+0000 7ff1d5400640 1 -- 172.21.15.90:0/1683502072 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1d002cb40 con 0x7ff1d80b2e40 2024-03-24T01:32:28.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.043+0000 7ff1d5400640 1 -- 172.21.15.90:0/1683502072 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d0034440 con 0x7ff1d80b2e40 2024-03-24T01:32:28.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.043+0000 7ff1dd400640 1 -- 172.21.15.90:0/1683502072 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1d80b2e40 msgr2=0x7ff1d8168c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:28.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.043+0000 7ff1dd400640 1 --2- 172.21.15.90:0/1683502072 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1d80b2e40 0x7ff1d8168c80 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0009d90 tx=0x7ff1d002bd90 comp rx=0 tx=0).stop 2024-03-24T01:32:28.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.043+0000 7ff1dd400640 1 -- 172.21.15.90:0/1683502072 shutdown_connections 2024-03-24T01:32:28.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.043+0000 7ff1dd400640 1 --2- 172.21.15.90:0/1683502072 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1d80b2e40 0x7ff1d8168c80 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:28.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.044+0000 7ff1dd400640 1 -- 172.21.15.90:0/1683502072 >> 172.21.15.90:0/1683502072 conn(0x7ff1d8072c10 msgr2=0x7ff1d8073020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:28.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.044+0000 7ff1dd400640 1 -- 172.21.15.90:0/1683502072 shutdown_connections 2024-03-24T01:32:28.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.044+0000 7ff1dd400640 1 -- 172.21.15.90:0/1683502072 wait complete. 2024-03-24T01:32:28.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.045+0000 7ff1dd400640 1 Processor -- start 2024-03-24T01:32:28.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.045+0000 7ff1dd400640 1 -- start start 2024-03-24T01:32:28.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.046+0000 7ff1dd400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1d80b2e40 0x7ff1d80c8720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:28.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.046+0000 7ff1dd400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1d80d0b50 0x7ff1d80c8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:28.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.046+0000 7ff1dd400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1d815dfd0 0x7ff1d81f6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:28.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.046+0000 7ff1dd400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d80b6d80 con 0x7ff1d80b2e40 2024-03-24T01:32:28.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.046+0000 7ff1dd400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d80cf330 con 0x7ff1d80d0b50 2024-03-24T01:32:28.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.046+0000 7ff1dd400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d80b6f00 con 0x7ff1d815dfd0 2024-03-24T01:32:28.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.047+0000 7ff1d6800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1d80b2e40 0x7ff1d80c8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:28.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.047+0000 7ff1d6800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1d80b2e40 0x7ff1d80c8720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54186/0 (socket says 172.21.15.90:54186) 2024-03-24T01:32:28.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.047+0000 7ff1d6800640 1 -- 172.21.15.90:0/3665103557 learned_addr learned my addr 172.21.15.90:0/3665103557 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:28.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.047+0000 7ff1d5e00640 1 --2- 172.21.15.90:0/3665103557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1d80d0b50 0x7ff1d80c8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:28.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.047+0000 7ff1d7200640 1 --2- 172.21.15.90:0/3665103557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1d815dfd0 0x7ff1d81f6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:28.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.047+0000 7ff1d6800640 1 -- 172.21.15.90:0/3665103557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1d815dfd0 msgr2=0x7ff1d81f6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:28.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.047+0000 7ff1d6800640 1 --2- 172.21.15.90:0/3665103557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1d815dfd0 0x7ff1d81f6d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:28.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.047+0000 7ff1d6800640 1 -- 172.21.15.90:0/3665103557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1d80d0b50 msgr2=0x7ff1d80c8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:28.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.047+0000 7ff1d6800640 1 --2- 172.21.15.90:0/3665103557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1d80d0b50 0x7ff1d80c8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:28.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.047+0000 7ff1d6800640 1 -- 172.21.15.90:0/3665103557 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d80c7960 con 0x7ff1d80b2e40 2024-03-24T01:32:28.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.048+0000 7ff1d6800640 1 --2- 172.21.15.90:0/3665103557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1d80b2e40 0x7ff1d80c8720 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7ff1c800c460 tx=0x7ff1c800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:28.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.048+0000 7ff1cea00640 1 -- 172.21.15.90:0/3665103557 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c8016020 con 0x7ff1d80b2e40 2024-03-24T01:32:28.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.048+0000 7ff1cea00640 1 -- 172.21.15.90:0/3665103557 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1c8012870 con 0x7ff1d80b2e40 2024-03-24T01:32:28.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.048+0000 7ff1cea00640 1 -- 172.21.15.90:0/3665103557 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c8012c60 con 0x7ff1d80b2e40 2024-03-24T01:32:28.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.048+0000 7ff1dd400640 1 -- 172.21.15.90:0/3665103557 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1d80c7c40 con 0x7ff1d80b2e40 2024-03-24T01:32:28.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.048+0000 7ff1dd400640 1 -- 172.21.15.90:0/3665103557 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1d80c8200 con 0x7ff1d80b2e40 2024-03-24T01:32:28.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.049+0000 7ff1dd400640 1 -- 172.21.15.90:0/3665103557 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1d804f400 con 0x7ff1d80b2e40 2024-03-24T01:32:28.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.050+0000 7ff1cea00640 1 -- 172.21.15.90:0/3665103557 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff1c8019020 con 0x7ff1d80b2e40 2024-03-24T01:32:28.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.051+0000 7ff1cea00640 1 --2- 172.21.15.90:0/3665103557 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff1b806bbd0 0x7ff1b806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:28.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.051+0000 7ff1cea00640 1 -- 172.21.15.90:0/3665103557 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x7ff1c808aca0 con 0x7ff1d80b2e40 2024-03-24T01:32:28.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.052+0000 7ff1d5e00640 1 --2- 172.21.15.90:0/3665103557 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff1b806bbd0 0x7ff1b806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:28.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.053+0000 7ff1d5e00640 1 --2- 172.21.15.90:0/3665103557 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff1b806bbd0 0x7ff1b806e090 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ff1d80b2ae0 tx=0x7ff1c4007c00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:28.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.060+0000 7ff1cea00640 1 -- 172.21.15.90:0/3665103557 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff1c8020020 con 0x7ff1d80b2e40 2024-03-24T01:32:28.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.256+0000 7ff1dd400640 1 -- 172.21.15.90:0/3665103557 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff1d80c83d0 con 0x7ff1d80b2e40 2024-03-24T01:32:28.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.257+0000 7ff1cea00640 1 -- 172.21.15.90:0/3665103557 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v56) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff1c80182c0 con 0x7ff1d80b2e40 2024-03-24T01:32:28.258 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:32:28.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.260+0000 7ff1dd400640 1 -- 172.21.15.90:0/3665103557 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff1b806bbd0 msgr2=0x7ff1b806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:28.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.260+0000 7ff1dd400640 1 --2- 172.21.15.90:0/3665103557 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff1b806bbd0 0x7ff1b806e090 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ff1d80b2ae0 tx=0x7ff1c4007c00 comp rx=0 tx=0).stop 2024-03-24T01:32:28.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.260+0000 7ff1dd400640 1 -- 172.21.15.90:0/3665103557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1d80b2e40 msgr2=0x7ff1d80c8720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:28.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.260+0000 7ff1dd400640 1 --2- 172.21.15.90:0/3665103557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1d80b2e40 0x7ff1d80c8720 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7ff1c800c460 tx=0x7ff1c800c920 comp rx=0 tx=0).stop 2024-03-24T01:32:28.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.260+0000 7ff1dd400640 1 -- 172.21.15.90:0/3665103557 shutdown_connections 2024-03-24T01:32:28.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.260+0000 7ff1dd400640 1 --2- 172.21.15.90:0/3665103557 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff1b806bbd0 0x7ff1b806e090 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:28.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.260+0000 7ff1dd400640 1 --2- 172.21.15.90:0/3665103557 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff1d815dfd0 0x7ff1d81f6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:28.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.260+0000 7ff1dd400640 1 --2- 172.21.15.90:0/3665103557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1d80d0b50 0x7ff1d80c8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:28.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.260+0000 7ff1dd400640 1 --2- 172.21.15.90:0/3665103557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1d80b2e40 0x7ff1d80c8720 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:28.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.260+0000 7ff1dd400640 1 -- 172.21.15.90:0/3665103557 >> 172.21.15.90:0/3665103557 conn(0x7ff1d8072c10 msgr2=0x7ff1d80d0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:28.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.260+0000 7ff1dd400640 1 -- 172.21.15.90:0/3665103557 shutdown_connections 2024-03-24T01:32:28.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.260+0000 7ff1dd400640 1 -- 172.21.15.90:0/3665103557 wait complete. 2024-03-24T01:32:28.280 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T01:32:28.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.488+0000 7fcc67e00640 1 Processor -- start 2024-03-24T01:32:28.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.489+0000 7fcc67e00640 1 -- start start 2024-03-24T01:32:28.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.490+0000 7fcc67e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcc680b5570 0x7fcc680b5a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:28.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.491+0000 7fcc67e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc680b6f00 0x7fcc680b7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:28.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.491+0000 7fcc67e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc680b7870 con 0x7fcc6815dec0 2024-03-24T01:32:28.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.491+0000 7fcc67e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc6815e2a0 con 0x7fcc680b5570 2024-03-24T01:32:28.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.491+0000 7fcc67e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc680cf2f0 con 0x7fcc680b6f00 2024-03-24T01:32:28.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.491+0000 7fcc5fe00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcc680b5570 0x7fcc680b5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:28.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.491+0000 7fcc5fe00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcc680b5570 0x7fcc680b5a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.90:50846/0 (socket says 172.21.15.90:50846) 2024-03-24T01:32:28.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.491+0000 7fcc5fe00640 1 -- 172.21.15.90:0/2458368039 learned_addr learned my addr 172.21.15.90:0/2458368039 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:28.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.491+0000 7fcc67400640 1 --2- 172.21.15.90:0/2458368039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc680b6f00 0x7fcc680b7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:28.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.492+0000 7fcc5fe00640 1 -- 172.21.15.90:0/2458368039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc680b6f00 msgr2=0x7fcc680b7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:28.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.492+0000 7fcc5fe00640 1 --2- 172.21.15.90:0/2458368039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc680b6f00 0x7fcc680b7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:28.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.492+0000 7fcc5fe00640 1 -- 172.21.15.90:0/2458368039 >> v1:172.21.15.120:6789/0 conn(0x7fcc6815dec0 legacy=0x7fcc680d28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:28.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.492+0000 7fcc5fe00640 1 -- 172.21.15.90:0/2458368039 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc680d1e00 con 0x7fcc680b5570 2024-03-24T01:32:28.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.492+0000 7fcc67400640 1 --2- 172.21.15.90:0/2458368039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc680b6f00 0x7fcc680b7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:32:28.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.493+0000 7fcc5fe00640 1 --2- 172.21.15.90:0/2458368039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc680b5570 0x7fcc680b5a00 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fcc50009870 tx=0x7fcc5002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98285bf9a2bdebe3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:28.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.494+0000 7fcc66000640 1 -- 172.21.15.90:0/2458368039 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc50034070 con 0x7fcc680b5570 2024-03-24T01:32:28.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.494+0000 7fcc66000640 1 -- 172.21.15.90:0/2458368039 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc50030840 con 0x7fcc680b5570 2024-03-24T01:32:28.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.494+0000 7fcc66000640 1 -- 172.21.15.90:0/2458368039 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc50030c10 con 0x7fcc680b5570 2024-03-24T01:32:28.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.495+0000 7fcc67e00640 1 -- 172.21.15.90:0/2458368039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc680b5570 msgr2=0x7fcc680b5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:28.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.495+0000 7fcc67e00640 1 --2- 172.21.15.90:0/2458368039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc680b5570 0x7fcc680b5a00 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fcc50009870 tx=0x7fcc5002b980 comp rx=0 tx=0).stop 2024-03-24T01:32:28.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.495+0000 7fcc67e00640 1 -- 172.21.15.90:0/2458368039 shutdown_connections 2024-03-24T01:32:28.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.495+0000 7fcc67e00640 1 --2- 172.21.15.90:0/2458368039 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc680b6f00 0x7fcc680b7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:28.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.495+0000 7fcc67e00640 1 --2- 172.21.15.90:0/2458368039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc680b5570 0x7fcc680b5a00 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:28.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.495+0000 7fcc67e00640 1 -- 172.21.15.90:0/2458368039 >> 172.21.15.90:0/2458368039 conn(0x7fcc680d0750 msgr2=0x7fcc680d0b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:28.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.495+0000 7fcc67e00640 1 -- 172.21.15.90:0/2458368039 shutdown_connections 2024-03-24T01:32:28.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.495+0000 7fcc67e00640 1 -- 172.21.15.90:0/2458368039 wait complete. 2024-03-24T01:32:28.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.498+0000 7fcc67e00640 1 Processor -- start 2024-03-24T01:32:28.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.499+0000 7fcc67e00640 1 -- start start 2024-03-24T01:32:28.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.500+0000 7fcc67e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc680b5570 0x7fcc681f7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:28.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.500+0000 7fcc67e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc680b6f00 0x7fcc6814f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:28.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.500+0000 7fcc67e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc6815dec0 0x7fcc681605f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:28.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.501+0000 7fcc67e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc680d3170 con 0x7fcc680b5570 2024-03-24T01:32:28.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.501+0000 7fcc67e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc680d2ff0 con 0x7fcc6815dec0 2024-03-24T01:32:28.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.501+0000 7fcc67e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc680ce580 con 0x7fcc680b6f00 2024-03-24T01:32:28.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.501+0000 7fcc66a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc680b5570 0x7fcc681f7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:28.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.501+0000 7fcc66a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc680b5570 0x7fcc681f7c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54198/0 (socket says 172.21.15.90:54198) 2024-03-24T01:32:28.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.501+0000 7fcc66a00640 1 -- 172.21.15.90:0/3492911777 learned_addr learned my addr 172.21.15.90:0/3492911777 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:28.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.501+0000 7fcc67400640 1 --2- 172.21.15.90:0/3492911777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc6815dec0 0x7fcc681605f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:28.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.501+0000 7fcc5fe00640 1 --2- 172.21.15.90:0/3492911777 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc680b6f00 0x7fcc6814f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:28.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.501+0000 7fcc66a00640 1 -- 172.21.15.90:0/3492911777 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc680b6f00 msgr2=0x7fcc6814f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:28.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.501+0000 7fcc66a00640 1 --2- 172.21.15.90:0/3492911777 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc680b6f00 0x7fcc6814f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:28.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.501+0000 7fcc66a00640 1 -- 172.21.15.90:0/3492911777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc6815dec0 msgr2=0x7fcc681605f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:28.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.501+0000 7fcc66a00640 1 --2- 172.21.15.90:0/3492911777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc6815dec0 0x7fcc681605f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:28.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.501+0000 7fcc66a00640 1 -- 172.21.15.90:0/3492911777 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc680c7620 con 0x7fcc680b5570 2024-03-24T01:32:28.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.501+0000 7fcc67400640 1 --2- 172.21.15.90:0/3492911777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc6815dec0 0x7fcc681605f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:28.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.501+0000 7fcc5fe00640 1 --2- 172.21.15.90:0/3492911777 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc680b6f00 0x7fcc6814f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:28.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.502+0000 7fcc66a00640 1 --2- 172.21.15.90:0/3492911777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc680b5570 0x7fcc681f7c40 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fcc5800c400 tx=0x7fcc5800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:28.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.502+0000 7fcc5f400640 1 -- 172.21.15.90:0/3492911777 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc58017070 con 0x7fcc680b5570 2024-03-24T01:32:28.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.502+0000 7fcc5f400640 1 -- 172.21.15.90:0/3492911777 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc580052d0 con 0x7fcc680b5570 2024-03-24T01:32:28.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.502+0000 7fcc67e00640 1 -- 172.21.15.90:0/3492911777 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc681f8490 con 0x7fcc680b5570 2024-03-24T01:32:28.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.502+0000 7fcc5f400640 1 -- 172.21.15.90:0/3492911777 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc58003830 con 0x7fcc680b5570 2024-03-24T01:32:28.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.502+0000 7fcc67e00640 1 -- 172.21.15.90:0/3492911777 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc681f8a50 con 0x7fcc680b5570 2024-03-24T01:32:28.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.503+0000 7fcc67e00640 1 -- 172.21.15.90:0/3492911777 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc68067790 con 0x7fcc680b5570 2024-03-24T01:32:28.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.504+0000 7fcc5f400640 1 -- 172.21.15.90:0/3492911777 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcc5801a070 con 0x7fcc680b5570 2024-03-24T01:32:28.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.505+0000 7fcc5f400640 1 --2- 172.21.15.90:0/3492911777 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcc4406bc00 0x7fcc4406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:28.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.505+0000 7fcc5f400640 1 -- 172.21.15.90:0/3492911777 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fcc58089ff0 con 0x7fcc680b5570 2024-03-24T01:32:28.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.505+0000 7fcc5fe00640 1 --2- 172.21.15.90:0/3492911777 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcc4406bc00 0x7fcc4406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:28.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.507+0000 7fcc5fe00640 1 --2- 172.21.15.90:0/3492911777 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcc4406bc00 0x7fcc4406e0c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fcc50036000 tx=0x7fcc5003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:28.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.514+0000 7fcc5f400640 1 -- 172.21.15.90:0/3492911777 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcc68067790 con 0x7fcc680b5570 2024-03-24T01:32:28.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.711+0000 7fcc67e00640 1 -- 172.21.15.90:0/3492911777 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcc680cdeb0 con 0x7fcc680b5570 2024-03-24T01:32:28.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.712+0000 7fcc5f400640 1 -- 172.21.15.90:0/3492911777 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v56) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcc58058880 con 0x7fcc680b5570 2024-03-24T01:32:28.714 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-24T01:32:28.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.715+0000 7fcc67e00640 1 -- 172.21.15.90:0/3492911777 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcc4406bc00 msgr2=0x7fcc4406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:28.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.715+0000 7fcc67e00640 1 --2- 172.21.15.90:0/3492911777 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcc4406bc00 0x7fcc4406e0c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fcc50036000 tx=0x7fcc5003b040 comp rx=0 tx=0).stop 2024-03-24T01:32:28.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.715+0000 7fcc67e00640 1 -- 172.21.15.90:0/3492911777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc680b5570 msgr2=0x7fcc681f7c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:28.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.715+0000 7fcc67e00640 1 --2- 172.21.15.90:0/3492911777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc680b5570 0x7fcc681f7c40 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fcc5800c400 tx=0x7fcc5800c8c0 comp rx=0 tx=0).stop 2024-03-24T01:32:28.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.716+0000 7fcc67e00640 1 -- 172.21.15.90:0/3492911777 shutdown_connections 2024-03-24T01:32:28.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.716+0000 7fcc67e00640 1 --2- 172.21.15.90:0/3492911777 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcc4406bc00 0x7fcc4406e0c0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:28.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.716+0000 7fcc67e00640 1 --2- 172.21.15.90:0/3492911777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc6815dec0 0x7fcc681605f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:28.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.716+0000 7fcc67e00640 1 --2- 172.21.15.90:0/3492911777 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcc680b6f00 0x7fcc6814f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:28.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.716+0000 7fcc67e00640 1 --2- 172.21.15.90:0/3492911777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc680b5570 0x7fcc681f7c40 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:28.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.716+0000 7fcc67e00640 1 -- 172.21.15.90:0/3492911777 >> 172.21.15.90:0/3492911777 conn(0x7fcc680d0750 msgr2=0x7fcc680b7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:28.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.716+0000 7fcc67e00640 1 -- 172.21.15.90:0/3492911777 shutdown_connections 2024-03-24T01:32:28.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.716+0000 7fcc67e00640 1 -- 172.21.15.90:0/3492911777 wait complete. 2024-03-24T01:32:28.735 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:32:28.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.934+0000 7f584a400640 1 Processor -- start 2024-03-24T01:32:28.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.934+0000 7f584a400640 1 -- start start 2024-03-24T01:32:28.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.935+0000 7f584a400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f58440cc150 0x7f58440ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:28.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.935+0000 7f584a400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f58441603e0 0x7f584406b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:28.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.935+0000 7f584a400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5844145430 0x7f584406b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:28.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.935+0000 7f584a400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58440cd490 con 0x7f58440cc150 2024-03-24T01:32:28.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.935+0000 7f584a400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f584415f900 con 0x7f58441603e0 2024-03-24T01:32:28.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.935+0000 7f584a400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f584415fbe0 con 0x7f5844145430 2024-03-24T01:32:28.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.936+0000 7f5843400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f58440cc150 0x7f58440ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:28.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.936+0000 7f5843400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f58440cc150 0x7f58440ccb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54212/0 (socket says 172.21.15.90:54212) 2024-03-24T01:32:28.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.936+0000 7f5843400640 1 -- 172.21.15.90:0/2590798918 learned_addr learned my addr 172.21.15.90:0/2590798918 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:28.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.936+0000 7f5843e00640 1 --2- 172.21.15.90:0/2590798918 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5844145430 0x7f584406b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:28.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.936+0000 7f583be00640 1 --2- 172.21.15.90:0/2590798918 >> v2:172.21.15.120:3300/0 conn(0x7f58441603e0 0x7f584406b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:28.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.936+0000 7f5843400640 1 -- 172.21.15.90:0/2590798918 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5844145430 msgr2=0x7f584406b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:28.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.936+0000 7f5843400640 1 --2- 172.21.15.90:0/2590798918 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5844145430 0x7f584406b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:28.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.936+0000 7f5843400640 1 -- 172.21.15.90:0/2590798918 >> v2:172.21.15.120:3300/0 conn(0x7f58441603e0 msgr2=0x7f584406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:28.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.936+0000 7f5843400640 1 --2- 172.21.15.90:0/2590798918 >> v2:172.21.15.120:3300/0 conn(0x7f58441603e0 0x7f584406b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:28.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.936+0000 7f5843400640 1 -- 172.21.15.90:0/2590798918 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f584415fec0 con 0x7f58440cc150 2024-03-24T01:32:28.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.937+0000 7f5843400640 1 --2- 172.21.15.90:0/2590798918 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58440cc150 0x7f58440ccb60 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f584406c2d0 tx=0x7f583402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44c1bf5884ece635 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:28.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.938+0000 7f5842a00640 1 -- 172.21.15.90:0/2590798918 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5834035020 con 0x7f58440cc150 2024-03-24T01:32:28.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.938+0000 7f5842a00640 1 -- 172.21.15.90:0/2590798918 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5834031870 con 0x7f58440cc150 2024-03-24T01:32:28.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.938+0000 7f5842a00640 1 -- 172.21.15.90:0/2590798918 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5834031c40 con 0x7f58440cc150 2024-03-24T01:32:28.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.939+0000 7f584a400640 1 -- 172.21.15.90:0/2590798918 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58440cc150 msgr2=0x7f58440ccb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:28.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.939+0000 7f584a400640 1 --2- 172.21.15.90:0/2590798918 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58440cc150 0x7f58440ccb60 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f584406c2d0 tx=0x7f583402b910 comp rx=0 tx=0).stop 2024-03-24T01:32:28.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.939+0000 7f584a400640 1 -- 172.21.15.90:0/2590798918 shutdown_connections 2024-03-24T01:32:28.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.939+0000 7f584a400640 1 --2- 172.21.15.90:0/2590798918 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5844145430 0x7f584406b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:28.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.939+0000 7f584a400640 1 --2- 172.21.15.90:0/2590798918 >> v2:172.21.15.120:3300/0 conn(0x7f58441603e0 0x7f584406b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:28.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.939+0000 7f584a400640 1 --2- 172.21.15.90:0/2590798918 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58440cc150 0x7f58440ccb60 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:28.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.939+0000 7f584a400640 1 -- 172.21.15.90:0/2590798918 >> 172.21.15.90:0/2590798918 conn(0x7f58440b1770 msgr2=0x7f58440b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:28.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.939+0000 7f584a400640 1 -- 172.21.15.90:0/2590798918 shutdown_connections 2024-03-24T01:32:28.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.939+0000 7f584a400640 1 -- 172.21.15.90:0/2590798918 wait complete. 2024-03-24T01:32:28.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.940+0000 7f584a400640 1 Processor -- start 2024-03-24T01:32:28.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.941+0000 7f584a400640 1 -- start start 2024-03-24T01:32:28.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.941+0000 7f584a400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58440cc150 0x7f584415ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:28.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.941+0000 7f584a400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5844145430 0x7f5844143d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:28.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.941+0000 7f584a400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58441603e0 0x7f584415c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:28.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.941+0000 7f584a400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5844071b10 con 0x7f5844145430 2024-03-24T01:32:28.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.941+0000 7f584a400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5844071990 con 0x7f58440cc150 2024-03-24T01:32:28.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.941+0000 7f584a400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5844071c90 con 0x7f58441603e0 2024-03-24T01:32:28.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.942+0000 7f583be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5844145430 0x7f5844143d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:28.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.942+0000 7f583be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5844145430 0x7f5844143d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54226/0 (socket says 172.21.15.90:54226) 2024-03-24T01:32:28.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.942+0000 7f583be00640 1 -- 172.21.15.90:0/2481443528 learned_addr learned my addr 172.21.15.90:0/2481443528 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:28.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.942+0000 7f5843e00640 1 --2- 172.21.15.90:0/2481443528 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58441603e0 0x7f584415c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:28.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.942+0000 7f583be00640 1 -- 172.21.15.90:0/2481443528 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58441603e0 msgr2=0x7f584415c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:28.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.942+0000 7f583be00640 1 --2- 172.21.15.90:0/2481443528 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58441603e0 0x7f584415c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:28.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.942+0000 7f583be00640 1 -- 172.21.15.90:0/2481443528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58440cc150 msgr2=0x7f584415ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:28.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.942+0000 7f5843400640 1 --2- 172.21.15.90:0/2481443528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58440cc150 0x7f584415ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:28.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.942+0000 7f583be00640 1 --2- 172.21.15.90:0/2481443528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58440cc150 0x7f584415ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:28.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.942+0000 7f583be00640 1 -- 172.21.15.90:0/2481443528 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f584415c540 con 0x7f5844145430 2024-03-24T01:32:28.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.943+0000 7f583be00640 1 --2- 172.21.15.90:0/2481443528 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5844145430 0x7f5844143d40 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f582c00c7f0 tx=0x7f582c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:28.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.943+0000 7f583b400640 1 -- 172.21.15.90:0/2481443528 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582c015070 con 0x7f5844145430 2024-03-24T01:32:28.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.943+0000 7f583b400640 1 -- 172.21.15.90:0/2481443528 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f582c007820 con 0x7f5844145430 2024-03-24T01:32:28.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.943+0000 7f583b400640 1 -- 172.21.15.90:0/2481443528 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582c007bd0 con 0x7f5844145430 2024-03-24T01:32:28.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.943+0000 7f584a400640 1 -- 172.21.15.90:0/2481443528 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f584415c7c0 con 0x7f5844145430 2024-03-24T01:32:28.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.943+0000 7f584a400640 1 -- 172.21.15.90:0/2481443528 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f584415cd80 con 0x7f5844145430 2024-03-24T01:32:28.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.944+0000 7f584a400640 1 -- 172.21.15.90:0/2481443528 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5844068060 con 0x7f5844145430 2024-03-24T01:32:28.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.945+0000 7f583b400640 1 -- 172.21.15.90:0/2481443528 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f582c018070 con 0x7f5844145430 2024-03-24T01:32:28.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.946+0000 7f583b400640 1 --2- 172.21.15.90:0/2481443528 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f582006bbd0 0x7f582006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:28.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.946+0000 7f583b400640 1 -- 172.21.15.90:0/2481443528 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f582c08a9d0 con 0x7f5844145430 2024-03-24T01:32:28.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.947+0000 7f5843400640 1 --2- 172.21.15.90:0/2481443528 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f582006bbd0 0x7f582006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:28.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.948+0000 7f5843400640 1 --2- 172.21.15.90:0/2481443528 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f582006bbd0 0x7f582006e090 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f5834036000 tx=0x7f5834008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:28.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:28.951+0000 7f583b400640 1 -- 172.21.15.90:0/2481443528 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f584415c540 con 0x7f5844145430 2024-03-24T01:32:29.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.247+0000 7f584a400640 1 -- 172.21.15.90:0/2481443528 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58440b3220 con 0x7f5844145430 2024-03-24T01:32:29.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.249+0000 7f583b400640 1 -- 172.21.15.90:0/2481443528 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f582c059200 con 0x7f5844145430 2024-03-24T01:32:29.250 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:32:29.250 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":41,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4985,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6834/3475131973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":3475131973},{"type":"v1","addr":"172.21.15.90:6834","nonce":3475131973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":4994,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/3151604088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":3151604088},{"type":"v1","addr":"172.21.15.120:6837","nonce":3151604088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5428,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/1239583677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1239583677},{"type":"v1","addr":"172.21.15.120:6836","nonce":1239583677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T01:27:13.961373+0000","modified":"2024-03-24T01:30:40.234926+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6219},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6219":{"gid":6219,"name":"c","rank":0,"incarnation":38,"state":"up:active","state_seq":56,"addr":"172.21.15.90:6835/226826781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":226826781},{"type":"v1","addr":"172.21.15.90:6835","nonce":226826781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6219,"qdb_cluster":[6219]},"id":3}]} 2024-03-24T01:32:29.250 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 41 2024-03-24T01:32:29.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.251+0000 7f584a400640 1 -- 172.21.15.90:0/2481443528 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f582006bbd0 msgr2=0x7f582006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:29.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.251+0000 7f584a400640 1 --2- 172.21.15.90:0/2481443528 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f582006bbd0 0x7f582006e090 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f5834036000 tx=0x7f5834008030 comp rx=0 tx=0).stop 2024-03-24T01:32:29.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.251+0000 7f584a400640 1 -- 172.21.15.90:0/2481443528 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5844145430 msgr2=0x7f5844143d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:29.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.251+0000 7f584a400640 1 --2- 172.21.15.90:0/2481443528 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5844145430 0x7f5844143d40 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f582c00c7f0 tx=0x7f582c00ccb0 comp rx=0 tx=0).stop 2024-03-24T01:32:29.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.251+0000 7f584a400640 1 -- 172.21.15.90:0/2481443528 shutdown_connections 2024-03-24T01:32:29.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.251+0000 7f584a400640 1 --2- 172.21.15.90:0/2481443528 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f582006bbd0 0x7f582006e090 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:29.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.251+0000 7f584a400640 1 --2- 172.21.15.90:0/2481443528 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5844145430 0x7f5844143d40 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:29.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.251+0000 7f584a400640 1 --2- 172.21.15.90:0/2481443528 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58441603e0 0x7f584415c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:29.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.251+0000 7f584a400640 1 --2- 172.21.15.90:0/2481443528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58440cc150 0x7f584415ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:29.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.251+0000 7f584a400640 1 -- 172.21.15.90:0/2481443528 >> 172.21.15.90:0/2481443528 conn(0x7f58440b1770 msgr2=0x7f584406b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:29.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.251+0000 7f584a400640 1 -- 172.21.15.90:0/2481443528 shutdown_connections 2024-03-24T01:32:29.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.252+0000 7f584a400640 1 -- 172.21.15.90:0/2481443528 wait complete. 2024-03-24T01:32:29.270 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:32:29.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.521+0000 7fa826800640 1 Processor -- start 2024-03-24T01:32:29.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.521+0000 7fa826800640 1 -- start start 2024-03-24T01:32:29.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.523+0000 7fa826800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa8200b2ab0 0x7fa8200b24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:29.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.523+0000 7fa826800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa82006c5b0 con 0x7fa8200cc040 2024-03-24T01:32:29.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.523+0000 7fa826800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa82006c890 con 0x7fa82006cbc0 2024-03-24T01:32:29.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.523+0000 7fa826800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8200d0e20 con 0x7fa8200b2ab0 2024-03-24T01:32:29.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.523+0000 7fa81fe00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa8200cc040 0x7fa8200cc4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50834/0 (socket says 172.21.15.90:50834) 2024-03-24T01:32:29.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.523+0000 7fa81fe00640 1 -- 172.21.15.90:0/633021472 learned_addr learned my addr 172.21.15.90:0/633021472 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:32:29.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.523+0000 7fa81ea00640 1 --2- 172.21.15.90:0/633021472 >> v2:172.21.15.120:3300/0 conn(0x7fa8200b2ab0 0x7fa8200b24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:29.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.523+0000 7fa81e000640 1 -- 172.21.15.90:0/633021472 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 784519752 0 0) 0x7fa82006c5b0 con 0x7fa8200cc040 2024-03-24T01:32:29.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.524+0000 7fa81e000640 1 -- 172.21.15.90:0/633021472 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa804003520 con 0x7fa8200cc040 2024-03-24T01:32:29.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.524+0000 7fa81e000640 1 -- 172.21.15.90:0/633021472 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1746856222 0 0) 0x7fa804003520 con 0x7fa8200cc040 2024-03-24T01:32:29.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.524+0000 7fa81e000640 1 -- 172.21.15.90:0/633021472 >> v2:172.21.15.120:3300/0 conn(0x7fa8200b2ab0 msgr2=0x7fa8200b24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:29.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.524+0000 7fa81e000640 1 --2- 172.21.15.90:0/633021472 >> v2:172.21.15.120:3300/0 conn(0x7fa8200b2ab0 0x7fa8200b24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:29.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.524+0000 7fa81e000640 1 -- 172.21.15.90:0/633021472 >> v1:172.21.15.120:6789/0 conn(0x7fa82006cbc0 legacy=0x7fa82006cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:29.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.524+0000 7fa81e000640 1 -- 172.21.15.90:0/633021472 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8200d07d0 con 0x7fa8200cc040 2024-03-24T01:32:29.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.524+0000 7fa81e000640 1 -- 172.21.15.90:0/633021472 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7fa8180019f0 con 0x7fa8200cc040 2024-03-24T01:32:29.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.524+0000 7fa81e000640 1 -- 172.21.15.90:0/633021472 >> v1:172.21.15.90:6789/0 conn(0x7fa8200cc040 legacy=0x7fa8200cc4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:29.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.524+0000 7fa81e000640 1 --2- 172.21.15.90:0/633021472 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8040039b0 0x7fa804003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:29.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.524+0000 7fa81e000640 1 --2- 172.21.15.90:0/633021472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa804004800 0x7fa804004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:29.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.524+0000 7fa81e000640 1 --2- 172.21.15.90:0/633021472 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa8040051c0 0x7fa80400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:29.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.524+0000 7fa81e000640 1 -- 172.21.15.90:0/633021472 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8200d0e20 con 0x7fa8040039b0 2024-03-24T01:32:29.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.524+0000 7fa81e000640 1 -- 172.21.15.90:0/633021472 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa80400c160 con 0x7fa804004800 2024-03-24T01:32:29.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.524+0000 7fa81e000640 1 -- 172.21.15.90:0/633021472 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa80400c360 con 0x7fa8040051c0 2024-03-24T01:32:29.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.524+0000 7fa81f400640 1 --2- 172.21.15.90:0/633021472 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8040039b0 0x7fa804003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:29.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.525+0000 7fa81ea00640 1 --2- 172.21.15.90:0/633021472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa804004800 0x7fa804004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:29.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.525+0000 7fa81f400640 1 -- 172.21.15.90:0/633021472 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa8040051c0 msgr2=0x7fa80400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:29.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.525+0000 7fa81f400640 1 --2- 172.21.15.90:0/633021472 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa8040051c0 0x7fa80400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:29.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.525+0000 7fa81f400640 1 -- 172.21.15.90:0/633021472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa804004800 msgr2=0x7fa804004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:29.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.525+0000 7fa81f400640 1 --2- 172.21.15.90:0/633021472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa804004800 0x7fa804004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:29.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.525+0000 7fa81f400640 1 -- 172.21.15.90:0/633021472 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa804003520 con 0x7fa8040039b0 2024-03-24T01:32:29.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.525+0000 7fa81fe00640 1 -- 172.21.15.90:0/633021472 reap_dead start 2024-03-24T01:32:29.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.526+0000 7fa81f400640 1 --2- 172.21.15.90:0/633021472 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8040039b0 0x7fa804003dd0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fa810001750 tx=0x7fa804005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d51297541ad32554 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:29.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.526+0000 7fa81e000640 1 -- 172.21.15.90:0/633021472 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa810001370 con 0x7fa8040039b0 2024-03-24T01:32:29.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.526+0000 7fa81e000640 1 -- 172.21.15.90:0/633021472 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8100052d0 con 0x7fa8040039b0 2024-03-24T01:32:29.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.526+0000 7fa81e000640 1 -- 172.21.15.90:0/633021472 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa810003830 con 0x7fa8040039b0 2024-03-24T01:32:29.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.527+0000 7fa826800640 1 -- 172.21.15.90:0/633021472 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8040039b0 msgr2=0x7fa804003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:29.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.527+0000 7fa826800640 1 --2- 172.21.15.90:0/633021472 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8040039b0 0x7fa804003dd0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fa810001750 tx=0x7fa804005830 comp rx=0 tx=0).stop 2024-03-24T01:32:29.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.527+0000 7fa826800640 1 -- 172.21.15.90:0/633021472 shutdown_connections 2024-03-24T01:32:29.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.527+0000 7fa826800640 1 --2- 172.21.15.90:0/633021472 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8040039b0 0x7fa804003dd0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:29.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.527+0000 7fa826800640 1 -- 172.21.15.90:0/633021472 >> 172.21.15.90:0/633021472 conn(0x7fa8200b34e0 msgr2=0x7fa8200b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:29.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.527+0000 7fa826800640 1 -- 172.21.15.90:0/633021472 shutdown_connections 2024-03-24T01:32:29.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.527+0000 7fa826800640 1 -- 172.21.15.90:0/633021472 wait complete. 2024-03-24T01:32:29.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.528+0000 7fa826800640 1 Processor -- start 2024-03-24T01:32:29.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.529+0000 7fa826800640 1 -- start start 2024-03-24T01:32:29.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.529+0000 7fa826800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa8040039b0 0x7fa8201522b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:29.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.529+0000 7fa826800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa8200b5cc0 0x7fa8200c7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:29.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.529+0000 7fa826800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8200b6140 0x7fa8200c8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:29.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.529+0000 7fa826800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8200d12b0 con 0x7fa8200b6140 2024-03-24T01:32:29.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.529+0000 7fa826800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa82006c1d0 con 0x7fa8040039b0 2024-03-24T01:32:29.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.529+0000 7fa826800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8200c8770 con 0x7fa8200b5cc0 2024-03-24T01:32:29.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.530+0000 7fa81fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8200b6140 0x7fa8200c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:29.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.530+0000 7fa81fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8200b6140 0x7fa8200c8230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54234/0 (socket says 172.21.15.90:54234) 2024-03-24T01:32:29.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.530+0000 7fa81fe00640 1 -- 172.21.15.90:0/1612504642 learned_addr learned my addr 172.21.15.90:0/1612504642 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:29.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.530+0000 7fa81f400640 1 --2- 172.21.15.90:0/1612504642 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa8040039b0 0x7fa8201522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:29.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.530+0000 7fa81ea00640 1 --2- 172.21.15.90:0/1612504642 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa8200b5cc0 0x7fa8200c7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:29.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.530+0000 7fa81fe00640 1 -- 172.21.15.90:0/1612504642 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa8200b5cc0 msgr2=0x7fa8200c7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:29.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.530+0000 7fa81fe00640 1 --2- 172.21.15.90:0/1612504642 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa8200b5cc0 0x7fa8200c7cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:29.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.530+0000 7fa81fe00640 1 -- 172.21.15.90:0/1612504642 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa8040039b0 msgr2=0x7fa8201522b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:29.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.530+0000 7fa81fe00640 1 --2- 172.21.15.90:0/1612504642 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa8040039b0 0x7fa8201522b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:29.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.530+0000 7fa81fe00640 1 -- 172.21.15.90:0/1612504642 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8200b6710 con 0x7fa8200b6140 2024-03-24T01:32:29.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.531+0000 7fa81fe00640 1 --2- 172.21.15.90:0/1612504642 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8200b6140 0x7fa8200c8230 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fa818001370 tx=0x7fa81800cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:29.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.531+0000 7fa817400640 1 -- 172.21.15.90:0/1612504642 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa81800cea0 con 0x7fa8200b6140 2024-03-24T01:32:29.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.531+0000 7fa817400640 1 -- 172.21.15.90:0/1612504642 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa81800a610 con 0x7fa8200b6140 2024-03-24T01:32:29.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.531+0000 7fa826800640 1 -- 172.21.15.90:0/1612504642 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8200b69f0 con 0x7fa8200b6140 2024-03-24T01:32:29.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.531+0000 7fa817400640 1 -- 172.21.15.90:0/1612504642 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa81800aa00 con 0x7fa8200b6140 2024-03-24T01:32:29.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.531+0000 7fa826800640 1 -- 172.21.15.90:0/1612504642 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8200b6fb0 con 0x7fa8200b6140 2024-03-24T01:32:29.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.532+0000 7fa826800640 1 -- 172.21.15.90:0/1612504642 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa820068060 con 0x7fa8200b6140 2024-03-24T01:32:29.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.533+0000 7fa817400640 1 -- 172.21.15.90:0/1612504642 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa818011070 con 0x7fa8200b6140 2024-03-24T01:32:29.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.534+0000 7fa817400640 1 --2- 172.21.15.90:0/1612504642 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa80006bbd0 0x7fa80006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:29.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.534+0000 7fa817400640 1 -- 172.21.15.90:0/1612504642 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fa81808a720 con 0x7fa8200b6140 2024-03-24T01:32:29.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.535+0000 7fa81f400640 1 --2- 172.21.15.90:0/1612504642 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa80006bbd0 0x7fa80006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:29.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.536+0000 7fa81f400640 1 --2- 172.21.15.90:0/1612504642 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa80006bbd0 0x7fa80006e090 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fa810033000 tx=0x7fa810009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:29.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.539+0000 7fa817400640 1 -- 172.21.15.90:0/1612504642 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa81801a060 con 0x7fa8200b6140 2024-03-24T01:32:29.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.740+0000 7fa826800640 1 -- 172.21.15.90:0/1612504642 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8200b2f90 con 0x7fa8200b6140 2024-03-24T01:32:29.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.742+0000 7fa817400640 1 -- 172.21.15.90:0/1612504642 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+17087 (secure 0 0 0) 0x7fa818058990 con 0x7fa8200b6140 2024-03-24T01:32:29.744 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:32:29.744 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":56,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:29:08.935541+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:27:11.774313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-24T01:27:12.992083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.23","mappings":[{"from":0,"to":1}]},{"pgid":"8.37","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.120:6836/266172244":"2024-03-25T01:29:08.935482+0000","172.21.15.120:6834/266172244":"2024-03-25T01:29:08.935482+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:32:29.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.745+0000 7fa826800640 1 -- 172.21.15.90:0/1612504642 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa80006bbd0 msgr2=0x7fa80006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:29.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.745+0000 7fa826800640 1 --2- 172.21.15.90:0/1612504642 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa80006bbd0 0x7fa80006e090 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fa810033000 tx=0x7fa810009030 comp rx=0 tx=0).stop 2024-03-24T01:32:29.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.745+0000 7fa826800640 1 -- 172.21.15.90:0/1612504642 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8200b6140 msgr2=0x7fa8200c8230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:29.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.745+0000 7fa826800640 1 --2- 172.21.15.90:0/1612504642 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8200b6140 0x7fa8200c8230 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fa818001370 tx=0x7fa81800cc50 comp rx=0 tx=0).stop 2024-03-24T01:32:29.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.745+0000 7fa826800640 1 -- 172.21.15.90:0/1612504642 shutdown_connections 2024-03-24T01:32:29.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.745+0000 7fa826800640 1 --2- 172.21.15.90:0/1612504642 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa80006bbd0 0x7fa80006e090 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:29.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.745+0000 7fa826800640 1 --2- 172.21.15.90:0/1612504642 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8200b6140 0x7fa8200c8230 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:29.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.745+0000 7fa826800640 1 --2- 172.21.15.90:0/1612504642 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa8200b5cc0 0x7fa8200c7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:29.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.745+0000 7fa826800640 1 --2- 172.21.15.90:0/1612504642 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa8040039b0 0x7fa8201522b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:29.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.745+0000 7fa826800640 1 -- 172.21.15.90:0/1612504642 >> 172.21.15.90:0/1612504642 conn(0x7fa8200b34e0 msgr2=0x7fa820099cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:29.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.745+0000 7fa826800640 1 -- 172.21.15.90:0/1612504642 shutdown_connections 2024-03-24T01:32:29.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.745+0000 7fa826800640 1 -- 172.21.15.90:0/1612504642 wait complete. 2024-03-24T01:32:29.765 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-24T01:32:29.765 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:32:29.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.962+0000 7f49fb200640 1 Processor -- start 2024-03-24T01:32:29.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.962+0000 7f49fb200640 1 -- start start 2024-03-24T01:32:29.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.963+0000 7f49fb200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f49f415dd20 0x7f49f4070010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:29.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.963+0000 7f49fb200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49f40c6550 0x7f49f406b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:29.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.963+0000 7f49fb200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49f4070af0 con 0x7f49f4070550 2024-03-24T01:32:29.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.963+0000 7f49fb200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49f40cf810 con 0x7f49f415dd20 2024-03-24T01:32:29.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.963+0000 7f49fb200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49f40c5790 con 0x7f49f40c6550 2024-03-24T01:32:29.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.964+0000 7f49f9e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f49f415dd20 0x7f49f4070010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:29.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.964+0000 7f49f9e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f49f415dd20 0x7f49f4070010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52814/0 (socket says 172.21.15.90:52814) 2024-03-24T01:32:29.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.964+0000 7f49f9e00640 1 -- 172.21.15.90:0/2347635154 learned_addr learned my addr 172.21.15.90:0/2347635154 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:29.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.964+0000 7f49f9400640 1 --2- 172.21.15.90:0/2347635154 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49f40c6550 0x7f49f406b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:29.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.964+0000 7f49f9e00640 1 -- 172.21.15.90:0/2347635154 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49f40c6550 msgr2=0x7f49f406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:29.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.964+0000 7f49f9e00640 1 --2- 172.21.15.90:0/2347635154 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49f40c6550 0x7f49f406b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:29.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.964+0000 7f49f9e00640 1 -- 172.21.15.90:0/2347635154 >> v1:172.21.15.90:6789/0 conn(0x7f49f4070550 legacy=0x7f49f4161890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:29.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.964+0000 7f49f9e00640 1 -- 172.21.15.90:0/2347635154 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f40c5a70 con 0x7f49f415dd20 2024-03-24T01:32:29.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.964+0000 7f49f9400640 1 --2- 172.21.15.90:0/2347635154 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49f40c6550 0x7f49f406b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:29.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.965+0000 7f49f9e00640 1 --2- 172.21.15.90:0/2347635154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49f415dd20 0x7f49f4070010 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f49e8009870 tx=0x7f49e802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec277552f475275d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:29.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.966+0000 7f49f8a00640 1 -- 172.21.15.90:0/2347635154 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e802f840 con 0x7f49f415dd20 2024-03-24T01:32:29.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.966+0000 7f49f8a00640 1 -- 172.21.15.90:0/2347635154 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49e802f9e0 con 0x7f49f415dd20 2024-03-24T01:32:29.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.966+0000 7f49f8a00640 1 -- 172.21.15.90:0/2347635154 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e802fdb0 con 0x7f49f415dd20 2024-03-24T01:32:29.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.966+0000 7f49fb200640 1 -- 172.21.15.90:0/2347635154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49f415dd20 msgr2=0x7f49f4070010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:29.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.966+0000 7f49fb200640 1 --2- 172.21.15.90:0/2347635154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49f415dd20 0x7f49f4070010 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f49e8009870 tx=0x7f49e802ba20 comp rx=0 tx=0).stop 2024-03-24T01:32:29.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.966+0000 7f49fb200640 1 -- 172.21.15.90:0/2347635154 shutdown_connections 2024-03-24T01:32:29.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.966+0000 7f49fb200640 1 --2- 172.21.15.90:0/2347635154 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49f40c6550 0x7f49f406b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:29.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.966+0000 7f49fb200640 1 --2- 172.21.15.90:0/2347635154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49f415dd20 0x7f49f4070010 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:29.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.966+0000 7f49fb200640 1 -- 172.21.15.90:0/2347635154 >> 172.21.15.90:0/2347635154 conn(0x7f49f40b2480 msgr2=0x7f49f40b2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:29.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.966+0000 7f49fb200640 1 -- 172.21.15.90:0/2347635154 shutdown_connections 2024-03-24T01:32:29.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.966+0000 7f49fb200640 1 -- 172.21.15.90:0/2347635154 wait complete. 2024-03-24T01:32:29.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.968+0000 7f49fb200640 1 Processor -- start 2024-03-24T01:32:29.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.968+0000 7f49fb200640 1 -- start start 2024-03-24T01:32:29.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.969+0000 7f49fb200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49f4070550 0x7f49f415cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:29.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.969+0000 7f49fb200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49f40c6550 0x7f49f40ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:29.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.969+0000 7f49fb200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49f415dd20 0x7f49f41f3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:29.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.969+0000 7f49fb200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f415e100 con 0x7f49f40c6550 2024-03-24T01:32:29.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.969+0000 7f49fb200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f40c6030 con 0x7f49f4070550 2024-03-24T01:32:29.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.969+0000 7f49fb200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49f40d0530 con 0x7f49f415dd20 2024-03-24T01:32:29.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.969+0000 7f49f9400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49f40c6550 0x7f49f40ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:29.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.970+0000 7f49f9400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49f40c6550 0x7f49f40ae8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52816/0 (socket says 172.21.15.90:52816) 2024-03-24T01:32:29.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.970+0000 7f49f9400640 1 -- 172.21.15.90:0/3008462491 learned_addr learned my addr 172.21.15.90:0/3008462491 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:29.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.970+0000 7f49f9400640 1 -- 172.21.15.90:0/3008462491 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49f415dd20 msgr2=0x7f49f41f3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:29.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.970+0000 7f49f9400640 1 --2- 172.21.15.90:0/3008462491 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49f415dd20 0x7f49f41f3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:29.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.970+0000 7f49f9400640 1 -- 172.21.15.90:0/3008462491 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49f4070550 msgr2=0x7f49f415cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:29.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.970+0000 7f49f9400640 1 --2- 172.21.15.90:0/3008462491 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49f4070550 0x7f49f415cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:29.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.970+0000 7f49f9400640 1 -- 172.21.15.90:0/3008462491 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f40aee20 con 0x7f49f40c6550 2024-03-24T01:32:29.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.970+0000 7f49f9400640 1 --2- 172.21.15.90:0/3008462491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49f40c6550 0x7f49f40ae8e0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f49e400c7f0 tx=0x7f49e400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:29.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.970+0000 7f49ee000640 1 -- 172.21.15.90:0/3008462491 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e4015070 con 0x7f49f40c6550 2024-03-24T01:32:29.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.970+0000 7f49ee000640 1 -- 172.21.15.90:0/3008462491 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49e4007a30 con 0x7f49f40c6550 2024-03-24T01:32:29.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.970+0000 7f49ee000640 1 -- 172.21.15.90:0/3008462491 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e4007e00 con 0x7f49f40c6550 2024-03-24T01:32:29.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.971+0000 7f49fb200640 1 -- 172.21.15.90:0/3008462491 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49f40af0a0 con 0x7f49f40c6550 2024-03-24T01:32:29.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.971+0000 7f49fb200640 1 -- 172.21.15.90:0/3008462491 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49f40af660 con 0x7f49f40c6550 2024-03-24T01:32:29.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.972+0000 7f49fb200640 1 -- 172.21.15.90:0/3008462491 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49f404f400 con 0x7f49f40c6550 2024-03-24T01:32:29.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.973+0000 7f49ee000640 1 -- 172.21.15.90:0/3008462491 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f49e4018070 con 0x7f49f40c6550 2024-03-24T01:32:29.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.974+0000 7f49ee000640 1 --2- 172.21.15.90:0/3008462491 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f49d806bc00 0x7f49d806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:29.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.974+0000 7f49ee000640 1 -- 172.21.15.90:0/3008462491 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f49e408ab40 con 0x7f49f40c6550 2024-03-24T01:32:29.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.974+0000 7f49f9e00640 1 --2- 172.21.15.90:0/3008462491 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f49d806bc00 0x7f49d806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:29.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.976+0000 7f49f9e00640 1 --2- 172.21.15.90:0/3008462491 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f49d806bc00 0x7f49d806e0c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f49e8036000 tx=0x7f49e8039040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:29.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:29.978+0000 7f49ee000640 1 -- 172.21.15.90:0/3008462491 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f49f40aee20 con 0x7f49f40c6550 2024-03-24T01:32:30.042 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:32:30.049 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:32:30.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.216+0000 7f49fb200640 1 -- 172.21.15.90:0/3008462491 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49f40afba0 con 0x7f49f40c6550 2024-03-24T01:32:30.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.217+0000 7f49ee000640 1 -- 172.21.15.90:0/3008462491 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f49e40594a0 con 0x7f49f40c6550 2024-03-24T01:32:30.218 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:32:30.218 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":41,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4985,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6834/3475131973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":3475131973},{"type":"v1","addr":"172.21.15.90:6834","nonce":3475131973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":4994,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/3151604088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":3151604088},{"type":"v1","addr":"172.21.15.120:6837","nonce":3151604088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5428,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/1239583677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1239583677},{"type":"v1","addr":"172.21.15.120:6836","nonce":1239583677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T01:27:13.961373+0000","modified":"2024-03-24T01:30:40.234926+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6219},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6219":{"gid":6219,"name":"c","rank":0,"incarnation":38,"state":"up:active","state_seq":56,"addr":"172.21.15.90:6835/226826781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":226826781},{"type":"v1","addr":"172.21.15.90:6835","nonce":226826781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6219,"qdb_cluster":[6219]},"id":3}]} 2024-03-24T01:32:30.218 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 41 2024-03-24T01:32:30.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.219+0000 7f49fb200640 1 -- 172.21.15.90:0/3008462491 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f49d806bc00 msgr2=0x7f49d806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:30.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.219+0000 7f49fb200640 1 --2- 172.21.15.90:0/3008462491 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f49d806bc00 0x7f49d806e0c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f49e8036000 tx=0x7f49e8039040 comp rx=0 tx=0).stop 2024-03-24T01:32:30.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.219+0000 7f49fb200640 1 -- 172.21.15.90:0/3008462491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49f40c6550 msgr2=0x7f49f40ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:30.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.219+0000 7f49fb200640 1 --2- 172.21.15.90:0/3008462491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49f40c6550 0x7f49f40ae8e0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f49e400c7f0 tx=0x7f49e400cbf0 comp rx=0 tx=0).stop 2024-03-24T01:32:30.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.219+0000 7f49fb200640 1 -- 172.21.15.90:0/3008462491 shutdown_connections 2024-03-24T01:32:30.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.219+0000 7f49fb200640 1 --2- 172.21.15.90:0/3008462491 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f49d806bc00 0x7f49d806e0c0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:30.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.219+0000 7f49fb200640 1 --2- 172.21.15.90:0/3008462491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49f40c6550 0x7f49f40ae8e0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:30.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.219+0000 7f49fb200640 1 --2- 172.21.15.90:0/3008462491 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f49f415dd20 0x7f49f41f3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:30.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.219+0000 7f49fb200640 1 --2- 172.21.15.90:0/3008462491 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49f4070550 0x7f49f415cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:30.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.219+0000 7f49fb200640 1 -- 172.21.15.90:0/3008462491 >> 172.21.15.90:0/3008462491 conn(0x7f49f40b2480 msgr2=0x7f49f40b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:30.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.219+0000 7f49fb200640 1 -- 172.21.15.90:0/3008462491 shutdown_connections 2024-03-24T01:32:30.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.219+0000 7f49fb200640 1 -- 172.21.15.90:0/3008462491 wait complete. 2024-03-24T01:32:30.239 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:32:30.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.421+0000 7f586d600640 1 Processor -- start 2024-03-24T01:32:30.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.421+0000 7f586d600640 1 -- start start 2024-03-24T01:32:30.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.423+0000 7f586d600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5868155030 0x7f586815efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:30.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.423+0000 7f586d600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f586806b7d0 0x7f58680ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:30.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.423+0000 7f586d600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f586815f500 con 0x7f58680cdac0 2024-03-24T01:32:30.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.423+0000 7f586d600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f586815bf70 con 0x7f5868155030 2024-03-24T01:32:30.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.423+0000 7f586d600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f586815b360 con 0x7f586806b7d0 2024-03-24T01:32:30.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.423+0000 7f5866a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5868155030 0x7f586815efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:30.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.423+0000 7f5866a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5868155030 0x7f586815efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52826/0 (socket says 172.21.15.90:52826) 2024-03-24T01:32:30.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.423+0000 7f5866a00640 1 -- 172.21.15.90:0/803547000 learned_addr learned my addr 172.21.15.90:0/803547000 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:30.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.424+0000 7f5866a00640 1 -- 172.21.15.90:0/803547000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f586806b7d0 msgr2=0x7f58680ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:30.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.424+0000 7f5866a00640 1 --2- 172.21.15.90:0/803547000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f586806b7d0 0x7f58680ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:30.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.424+0000 7f5866a00640 1 -- 172.21.15.90:0/803547000 >> v1:172.21.15.120:6789/0 conn(0x7f58680cdac0 legacy=0x7f586806b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:30.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.424+0000 7f5866a00640 1 -- 172.21.15.90:0/803547000 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f586815b640 con 0x7f5868155030 2024-03-24T01:32:30.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.425+0000 7f5866a00640 1 --2- 172.21.15.90:0/803547000 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5868155030 0x7f586815efc0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f5858009870 tx=0x7f585802db00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1db86b93f69a858 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:30.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.425+0000 7f5866000640 1 -- 172.21.15.90:0/803547000 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f585802f8c0 con 0x7f5868155030 2024-03-24T01:32:30.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.425+0000 7f5866000640 1 -- 172.21.15.90:0/803547000 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f585802fa60 con 0x7f5868155030 2024-03-24T01:32:30.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.425+0000 7f5866000640 1 -- 172.21.15.90:0/803547000 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f585802fe30 con 0x7f5868155030 2024-03-24T01:32:30.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.426+0000 7f586d600640 1 -- 172.21.15.90:0/803547000 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5868155030 msgr2=0x7f586815efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:30.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.426+0000 7f586d600640 1 --2- 172.21.15.90:0/803547000 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5868155030 0x7f586815efc0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f5858009870 tx=0x7f585802db00 comp rx=0 tx=0).stop 2024-03-24T01:32:30.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.426+0000 7f586d600640 1 -- 172.21.15.90:0/803547000 shutdown_connections 2024-03-24T01:32:30.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.426+0000 7f586d600640 1 --2- 172.21.15.90:0/803547000 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f586806b7d0 0x7f58680ae530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:30.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.426+0000 7f586d600640 1 --2- 172.21.15.90:0/803547000 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5868155030 0x7f586815efc0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:30.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.426+0000 7f586d600640 1 -- 172.21.15.90:0/803547000 >> 172.21.15.90:0/803547000 conn(0x7f58680b1770 msgr2=0x7f58680b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:30.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.426+0000 7f586d600640 1 -- 172.21.15.90:0/803547000 shutdown_connections 2024-03-24T01:32:30.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.426+0000 7f586d600640 1 -- 172.21.15.90:0/803547000 wait complete. 2024-03-24T01:32:30.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.427+0000 7f586d600640 1 Processor -- start 2024-03-24T01:32:30.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.428+0000 7f586d600640 1 -- start start 2024-03-24T01:32:30.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.428+0000 7f586d600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f586806b7d0 0x7f5868144120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:30.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.429+0000 7f586d600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58680cdac0 0x7f5868158020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:30.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.429+0000 7f586d600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5868155030 0x7f58681f7500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:30.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.429+0000 7f586d600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5868155410 con 0x7f5868155030 2024-03-24T01:32:30.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.429+0000 7f586d600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f586815bc00 con 0x7f586806b7d0 2024-03-24T01:32:30.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.429+0000 7f586d600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f586815af70 con 0x7f58680cdac0 2024-03-24T01:32:30.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.429+0000 7f5867400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5868155030 0x7f58681f7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:30.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.429+0000 7f5867400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5868155030 0x7f58681f7500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52830/0 (socket says 172.21.15.90:52830) 2024-03-24T01:32:30.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.429+0000 7f5867400640 1 -- 172.21.15.90:0/3276912310 learned_addr learned my addr 172.21.15.90:0/3276912310 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:30.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.429+0000 7f5866a00640 1 --2- 172.21.15.90:0/3276912310 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f586806b7d0 0x7f5868144120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:30.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.429+0000 7f5867400640 1 -- 172.21.15.90:0/3276912310 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58680cdac0 msgr2=0x7f5868158020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:30.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.429+0000 7f5867400640 1 --2- 172.21.15.90:0/3276912310 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58680cdac0 0x7f5868158020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:30.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.429+0000 7f5867400640 1 -- 172.21.15.90:0/3276912310 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f586806b7d0 msgr2=0x7f5868144120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:30.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.429+0000 7f5867400640 1 --2- 172.21.15.90:0/3276912310 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f586806b7d0 0x7f5868144120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:30.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.429+0000 7f5867400640 1 -- 172.21.15.90:0/3276912310 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5868144660 con 0x7f5868155030 2024-03-24T01:32:30.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.430+0000 7f5867400640 1 --2- 172.21.15.90:0/3276912310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5868155030 0x7f58681f7500 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f586000b9e0 tx=0x7f586000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:30.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.430+0000 7f585f400640 1 -- 172.21.15.90:0/3276912310 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5860019070 con 0x7f5868155030 2024-03-24T01:32:30.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.430+0000 7f585f400640 1 -- 172.21.15.90:0/3276912310 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f586000cb40 con 0x7f5868155030 2024-03-24T01:32:30.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.430+0000 7f586d600640 1 -- 172.21.15.90:0/3276912310 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f586815d0a0 con 0x7f5868155030 2024-03-24T01:32:30.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.430+0000 7f585f400640 1 -- 172.21.15.90:0/3276912310 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5860014440 con 0x7f5868155030 2024-03-24T01:32:30.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.430+0000 7f586d600640 1 -- 172.21.15.90:0/3276912310 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f586815d660 con 0x7f5868155030 2024-03-24T01:32:30.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.431+0000 7f586d600640 1 -- 172.21.15.90:0/3276912310 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5868144ba0 con 0x7f5868155030 2024-03-24T01:32:30.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.432+0000 7f585f400640 1 -- 172.21.15.90:0/3276912310 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f586001c070 con 0x7f5868155030 2024-03-24T01:32:30.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.433+0000 7f585f400640 1 --2- 172.21.15.90:0/3276912310 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f584406bbd0 0x7f584406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:30.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.433+0000 7f585f400640 1 -- 172.21.15.90:0/3276912310 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f586008a9d0 con 0x7f5868155030 2024-03-24T01:32:30.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.434+0000 7f5866a00640 1 --2- 172.21.15.90:0/3276912310 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f584406bbd0 0x7f584406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:30.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.435+0000 7f5866a00640 1 --2- 172.21.15.90:0/3276912310 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f584406bbd0 0x7f584406e090 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f5858038000 tx=0x7f5858008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:30.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.437+0000 7f585f400640 1 -- 172.21.15.90:0/3276912310 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5860059290 con 0x7f5868155030 2024-03-24T01:32:30.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.657+0000 7f586d600640 1 -- 172.21.15.90:0/3276912310 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5868068060 con 0x7f5868155030 2024-03-24T01:32:30.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.657+0000 7f585f400640 1 -- 172.21.15.90:0/3276912310 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f5860058b90 con 0x7f5868155030 2024-03-24T01:32:30.659 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:32:30.659 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":41,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4985,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6834/3475131973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":3475131973},{"type":"v1","addr":"172.21.15.90:6834","nonce":3475131973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":4994,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/3151604088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":3151604088},{"type":"v1","addr":"172.21.15.120:6837","nonce":3151604088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5428,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/1239583677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1239583677},{"type":"v1","addr":"172.21.15.120:6836","nonce":1239583677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T01:27:13.961373+0000","modified":"2024-03-24T01:30:40.234926+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6219},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6219":{"gid":6219,"name":"c","rank":0,"incarnation":38,"state":"up:active","state_seq":56,"addr":"172.21.15.90:6835/226826781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":226826781},{"type":"v1","addr":"172.21.15.90:6835","nonce":226826781}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6219,"qdb_cluster":[6219]},"id":3}]} 2024-03-24T01:32:30.659 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 41 2024-03-24T01:32:30.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.661+0000 7f586d600640 1 -- 172.21.15.90:0/3276912310 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f584406bbd0 msgr2=0x7f584406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:30.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.661+0000 7f586d600640 1 --2- 172.21.15.90:0/3276912310 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f584406bbd0 0x7f584406e090 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f5858038000 tx=0x7f5858008030 comp rx=0 tx=0).stop 2024-03-24T01:32:30.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.661+0000 7f586d600640 1 -- 172.21.15.90:0/3276912310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5868155030 msgr2=0x7f58681f7500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:30.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.661+0000 7f586d600640 1 --2- 172.21.15.90:0/3276912310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5868155030 0x7f58681f7500 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f586000b9e0 tx=0x7f586000bea0 comp rx=0 tx=0).stop 2024-03-24T01:32:30.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.661+0000 7f586d600640 1 -- 172.21.15.90:0/3276912310 shutdown_connections 2024-03-24T01:32:30.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.661+0000 7f586d600640 1 --2- 172.21.15.90:0/3276912310 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f584406bbd0 0x7f584406e090 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:30.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.661+0000 7f586d600640 1 --2- 172.21.15.90:0/3276912310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5868155030 0x7f58681f7500 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:30.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.661+0000 7f586d600640 1 --2- 172.21.15.90:0/3276912310 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f58680cdac0 0x7f5868158020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:30.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.661+0000 7f586d600640 1 --2- 172.21.15.90:0/3276912310 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f586806b7d0 0x7f5868144120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:30.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.661+0000 7f586d600640 1 -- 172.21.15.90:0/3276912310 >> 172.21.15.90:0/3276912310 conn(0x7f58680b1770 msgr2=0x7f58680b0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:30.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.661+0000 7f586d600640 1 -- 172.21.15.90:0/3276912310 shutdown_connections 2024-03-24T01:32:30.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.661+0000 7f586d600640 1 -- 172.21.15.90:0/3276912310 wait complete. 2024-03-24T01:32:30.687 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:32:30.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.876+0000 7f35f1200640 1 Processor -- start 2024-03-24T01:32:30.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.877+0000 7f35f1200640 1 -- start start 2024-03-24T01:32:30.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.878+0000 7f35f1200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f35ec09aca0 0x7f35ec06fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:30.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.878+0000 7f35f1200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f35ec06d400 0x7f35ec06d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:30.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.878+0000 7f35f1200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35ec0702e0 con 0x7f35ec070710 2024-03-24T01:32:30.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.878+0000 7f35f1200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35ec06c1d0 con 0x7f35ec06d400 2024-03-24T01:32:30.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.878+0000 7f35f1200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35ec0cc1d0 con 0x7f35ec09aca0 2024-03-24T01:32:30.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.878+0000 7f35e9e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f35ec070710 0x7f35ec070af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57586/0 (socket says 172.21.15.90:57586) 2024-03-24T01:32:30.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.878+0000 7f35e9e00640 1 -- 172.21.15.90:0/2822411775 learned_addr learned my addr 172.21.15.90:0/2822411775 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:32:30.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.879+0000 7f35ea800640 1 --2- 172.21.15.90:0/2822411775 >> v2:172.21.15.90:3300/0 conn(0x7f35ec06d400 0x7f35ec06d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:30.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.879+0000 7f35e3e00640 1 --2- 172.21.15.90:0/2822411775 >> v2:172.21.15.120:3300/0 conn(0x7f35ec09aca0 0x7f35ec06fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:30.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.879+0000 7f35e9400640 1 -- 172.21.15.90:0/2822411775 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2170108921 0 0) 0x7f35ec0702e0 con 0x7f35ec070710 2024-03-24T01:32:30.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.879+0000 7f35e9400640 1 -- 172.21.15.90:0/2822411775 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35d0003520 con 0x7f35ec070710 2024-03-24T01:32:30.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.879+0000 7f35ea800640 1 -- 172.21.15.90:0/2822411775 >> v2:172.21.15.120:3300/0 conn(0x7f35ec09aca0 msgr2=0x7f35ec06fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:30.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.879+0000 7f35ea800640 1 --2- 172.21.15.90:0/2822411775 >> v2:172.21.15.120:3300/0 conn(0x7f35ec09aca0 0x7f35ec06fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:30.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.879+0000 7f35ea800640 1 -- 172.21.15.90:0/2822411775 >> v1:172.21.15.90:6789/0 conn(0x7f35ec070710 legacy=0x7f35ec070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:30.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.879+0000 7f35ea800640 1 -- 172.21.15.90:0/2822411775 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ec0cc4b0 con 0x7f35ec06d400 2024-03-24T01:32:30.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.880+0000 7f35ea800640 1 --2- 172.21.15.90:0/2822411775 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35ec06d400 0x7f35ec06d890 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f35ec073480 tx=0x7f35e402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=391f0c77e65bc094 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:30.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.881+0000 7f35e9400640 1 -- 172.21.15.90:0/2822411775 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e4039070 con 0x7f35ec06d400 2024-03-24T01:32:30.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.881+0000 7f35e9400640 1 -- 172.21.15.90:0/2822411775 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e402cbf0 con 0x7f35ec06d400 2024-03-24T01:32:30.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.881+0000 7f35e9400640 1 -- 172.21.15.90:0/2822411775 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e4034440 con 0x7f35ec06d400 2024-03-24T01:32:30.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.881+0000 7f35f1200640 1 -- 172.21.15.90:0/2822411775 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35ec06d400 msgr2=0x7f35ec06d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:30.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.881+0000 7f35f1200640 1 --2- 172.21.15.90:0/2822411775 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35ec06d400 0x7f35ec06d890 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f35ec073480 tx=0x7f35e402be40 comp rx=0 tx=0).stop 2024-03-24T01:32:30.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.881+0000 7f35f1200640 1 -- 172.21.15.90:0/2822411775 shutdown_connections 2024-03-24T01:32:30.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.881+0000 7f35f1200640 1 --2- 172.21.15.90:0/2822411775 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35ec06d400 0x7f35ec06d890 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:30.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.881+0000 7f35f1200640 1 --2- 172.21.15.90:0/2822411775 >> v2:172.21.15.120:3300/0 conn(0x7f35ec09aca0 0x7f35ec06fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:30.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.881+0000 7f35f1200640 1 -- 172.21.15.90:0/2822411775 >> 172.21.15.90:0/2822411775 conn(0x7f35ec0b34d0 msgr2=0x7f35ec0b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:30.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.881+0000 7f35f1200640 1 -- 172.21.15.90:0/2822411775 shutdown_connections 2024-03-24T01:32:30.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.881+0000 7f35f1200640 1 -- 172.21.15.90:0/2822411775 wait complete. 2024-03-24T01:32:30.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.883+0000 7f35f1200640 1 Processor -- start 2024-03-24T01:32:30.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.883+0000 7f35f1200640 1 -- start start 2024-03-24T01:32:30.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.884+0000 7f35f1200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35ec06d400 0x7f35ec0c8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:30.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.884+0000 7f35f1200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35ec070710 0x7f35ec15e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:30.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.884+0000 7f35f1200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35ec09aca0 0x7f35ec1605c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:30.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.884+0000 7f35f1200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ec15e140 con 0x7f35ec09aca0 2024-03-24T01:32:30.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.884+0000 7f35f1200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ec0d0f30 con 0x7f35ec06d400 2024-03-24T01:32:30.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.884+0000 7f35f1200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35ec15e2c0 con 0x7f35ec070710 2024-03-24T01:32:30.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.884+0000 7f35ea800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35ec09aca0 0x7f35ec1605c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:30.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.884+0000 7f35ea800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35ec09aca0 0x7f35ec1605c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52860/0 (socket says 172.21.15.90:52860) 2024-03-24T01:32:30.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.884+0000 7f35ea800640 1 -- 172.21.15.90:0/3325736549 learned_addr learned my addr 172.21.15.90:0/3325736549 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:30.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.885+0000 7f35e9e00640 1 --2- 172.21.15.90:0/3325736549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35ec06d400 0x7f35ec0c8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:30.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.885+0000 7f35ea800640 1 -- 172.21.15.90:0/3325736549 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35ec070710 msgr2=0x7f35ec15e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:30.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.885+0000 7f35ea800640 1 --2- 172.21.15.90:0/3325736549 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35ec070710 0x7f35ec15e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:30.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.885+0000 7f35ea800640 1 -- 172.21.15.90:0/3325736549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35ec06d400 msgr2=0x7f35ec0c8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:30.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.885+0000 7f35ea800640 1 --2- 172.21.15.90:0/3325736549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35ec06d400 0x7f35ec0c8ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:30.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.885+0000 7f35ea800640 1 -- 172.21.15.90:0/3325736549 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ec0b55c0 con 0x7f35ec09aca0 2024-03-24T01:32:30.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.885+0000 7f35ea800640 1 --2- 172.21.15.90:0/3325736549 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35ec09aca0 0x7f35ec1605c0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f35e402c450 tx=0x7f35e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:30.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.885+0000 7f35e2000640 1 -- 172.21.15.90:0/3325736549 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e4044020 con 0x7f35ec09aca0 2024-03-24T01:32:30.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.885+0000 7f35e2000640 1 -- 172.21.15.90:0/3325736549 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e403f470 con 0x7f35ec09aca0 2024-03-24T01:32:30.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.885+0000 7f35e2000640 1 -- 172.21.15.90:0/3325736549 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e4039040 con 0x7f35ec09aca0 2024-03-24T01:32:30.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.885+0000 7f35f1200640 1 -- 172.21.15.90:0/3325736549 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35ec0b58a0 con 0x7f35ec09aca0 2024-03-24T01:32:30.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.886+0000 7f35f1200640 1 -- 172.21.15.90:0/3325736549 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35ec0c8ff0 con 0x7f35ec09aca0 2024-03-24T01:32:30.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.887+0000 7f35f1200640 1 -- 172.21.15.90:0/3325736549 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35ec0cb770 con 0x7f35ec09aca0 2024-03-24T01:32:30.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.887+0000 7f35e2000640 1 -- 172.21.15.90:0/3325736549 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f35e4047020 con 0x7f35ec09aca0 2024-03-24T01:32:30.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.889+0000 7f35e2000640 1 --2- 172.21.15.90:0/3325736549 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f35c806bbd0 0x7f35c806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:30.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.889+0000 7f35e2000640 1 -- 172.21.15.90:0/3325736549 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f35e40ad850 con 0x7f35ec09aca0 2024-03-24T01:32:30.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.890+0000 7f35e9e00640 1 --2- 172.21.15.90:0/3325736549 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f35c806bbd0 0x7f35c806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:30.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.891+0000 7f35e9e00640 1 --2- 172.21.15.90:0/3325736549 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f35c806bbd0 0x7f35c806e090 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f35dc000f60 tx=0x7f35dc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:30.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:30.893+0000 7f35e2000640 1 -- 172.21.15.90:0/3325736549 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f35e404e030 con 0x7f35ec09aca0 2024-03-24T01:32:31.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.113+0000 7f35f1200640 1 -- 172.21.15.90:0/3325736549 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35ec09fb10 con 0x7f35ec09aca0 2024-03-24T01:32:31.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.116+0000 7f35e2000640 1 -- 172.21.15.90:0/3325736549 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+17087 (secure 0 0 0) 0x7f35e407c080 con 0x7f35ec09aca0 2024-03-24T01:32:31.117 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:32:31.118 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":56,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:29:08.935541+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:27:11.774313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-24T01:27:12.992083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.23","mappings":[{"from":0,"to":1}]},{"pgid":"8.37","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.120:6836/266172244":"2024-03-25T01:29:08.935482+0000","172.21.15.120:6834/266172244":"2024-03-25T01:29:08.935482+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:32:31.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.119+0000 7f35f1200640 1 -- 172.21.15.90:0/3325736549 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f35c806bbd0 msgr2=0x7f35c806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:31.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.119+0000 7f35f1200640 1 --2- 172.21.15.90:0/3325736549 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f35c806bbd0 0x7f35c806e090 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f35dc000f60 tx=0x7f35dc009040 comp rx=0 tx=0).stop 2024-03-24T01:32:31.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.119+0000 7f35f1200640 1 -- 172.21.15.90:0/3325736549 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35ec09aca0 msgr2=0x7f35ec1605c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:31.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.119+0000 7f35f1200640 1 --2- 172.21.15.90:0/3325736549 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35ec09aca0 0x7f35ec1605c0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f35e402c450 tx=0x7f35e4004a40 comp rx=0 tx=0).stop 2024-03-24T01:32:31.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.119+0000 7f35f1200640 1 -- 172.21.15.90:0/3325736549 shutdown_connections 2024-03-24T01:32:31.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.119+0000 7f35f1200640 1 --2- 172.21.15.90:0/3325736549 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f35c806bbd0 0x7f35c806e090 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:31.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.119+0000 7f35f1200640 1 --2- 172.21.15.90:0/3325736549 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35ec09aca0 0x7f35ec1605c0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:31.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.119+0000 7f35f1200640 1 --2- 172.21.15.90:0/3325736549 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f35ec070710 0x7f35ec15e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:31.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.119+0000 7f35f1200640 1 --2- 172.21.15.90:0/3325736549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35ec06d400 0x7f35ec0c8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:31.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.119+0000 7f35f1200640 1 -- 172.21.15.90:0/3325736549 >> 172.21.15.90:0/3325736549 conn(0x7f35ec0b34d0 msgr2=0x7f35ec0b1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:31.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.119+0000 7f35f1200640 1 -- 172.21.15.90:0/3325736549 shutdown_connections 2024-03-24T01:32:31.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.119+0000 7f35f1200640 1 -- 172.21.15.90:0/3325736549 wait complete. 2024-03-24T01:32:31.143 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T01:32:31.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.324+0000 7f45cdc00640 1 Processor -- start 2024-03-24T01:32:31.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.325+0000 7f45cdc00640 1 -- start start 2024-03-24T01:32:31.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.326+0000 7f45cdc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45c8144680 0x7f45c806d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:31.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.326+0000 7f45cdc00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f45c8145190 0x7f45c8071790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:31.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.326+0000 7f45cdc00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45c806e0f0 con 0x7f45c8071cd0 2024-03-24T01:32:31.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.326+0000 7f45cdc00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45c8144a60 con 0x7f45c8145190 2024-03-24T01:32:31.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.326+0000 7f45cdc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45c80cc190 con 0x7f45c8144680 2024-03-24T01:32:31.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.326+0000 7f45bf400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f45c8145190 0x7f45c8071790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:31.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.326+0000 7f45cd200640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f45c8071cd0 0x7f45c815f5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57590/0 (socket says 172.21.15.90:57590) 2024-03-24T01:32:31.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.326+0000 7f45cd200640 1 -- 172.21.15.90:0/2137646915 learned_addr learned my addr 172.21.15.90:0/2137646915 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:32:31.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.327+0000 7f45c7e00640 1 --2- 172.21.15.90:0/2137646915 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45c8144680 0x7f45c806d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:31.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.327+0000 7f45bf400640 1 -- 172.21.15.90:0/2137646915 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45c8144680 msgr2=0x7f45c806d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:31.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.327+0000 7f45bf400640 1 --2- 172.21.15.90:0/2137646915 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45c8144680 0x7f45c806d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:31.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.327+0000 7f45bf400640 1 -- 172.21.15.90:0/2137646915 >> v1:172.21.15.90:6789/0 conn(0x7f45c8071cd0 legacy=0x7f45c815f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:31.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.327+0000 7f45bf400640 1 -- 172.21.15.90:0/2137646915 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c806e3d0 con 0x7f45c8145190 2024-03-24T01:32:31.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.327+0000 7f45c7400640 1 -- 172.21.15.90:0/2137646915 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3661520755 0 0) 0x7f45c806e0f0 con 0x7f45c8071cd0 2024-03-24T01:32:31.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.328+0000 7f45bf400640 1 --2- 172.21.15.90:0/2137646915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45c8145190 0x7f45c8071790 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f45b00097a0 tx=0x7f45b002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d1f8bbb6fa1b300 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:31.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.328+0000 7f45c7400640 1 -- 172.21.15.90:0/2137646915 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b0035020 con 0x7f45c8145190 2024-03-24T01:32:31.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.328+0000 7f45c7400640 1 -- 172.21.15.90:0/2137646915 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45b0031830 con 0x7f45c8145190 2024-03-24T01:32:31.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.328+0000 7f45c7400640 1 -- 172.21.15.90:0/2137646915 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b0031c00 con 0x7f45c8145190 2024-03-24T01:32:31.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.329+0000 7f45cdc00640 1 -- 172.21.15.90:0/2137646915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45c8145190 msgr2=0x7f45c8071790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:31.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.329+0000 7f45cdc00640 1 --2- 172.21.15.90:0/2137646915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45c8145190 0x7f45c8071790 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f45b00097a0 tx=0x7f45b002b8d0 comp rx=0 tx=0).stop 2024-03-24T01:32:31.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.329+0000 7f45cdc00640 1 -- 172.21.15.90:0/2137646915 shutdown_connections 2024-03-24T01:32:31.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.329+0000 7f45cdc00640 1 --2- 172.21.15.90:0/2137646915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45c8145190 0x7f45c8071790 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:31.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.329+0000 7f45cdc00640 1 --2- 172.21.15.90:0/2137646915 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45c8144680 0x7f45c806d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:31.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.329+0000 7f45cdc00640 1 -- 172.21.15.90:0/2137646915 >> 172.21.15.90:0/2137646915 conn(0x7f45c806b0c0 msgr2=0x7f45c806b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:31.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.329+0000 7f45cdc00640 1 -- 172.21.15.90:0/2137646915 shutdown_connections 2024-03-24T01:32:31.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.329+0000 7f45cdc00640 1 -- 172.21.15.90:0/2137646915 wait complete. 2024-03-24T01:32:31.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.330+0000 7f45cdc00640 1 Processor -- start 2024-03-24T01:32:31.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.331+0000 7f45cdc00640 1 -- start start 2024-03-24T01:32:31.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.331+0000 7f45cdc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45c8071cd0 0x7f45c80ab4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:31.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.331+0000 7f45cdc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45c8144680 0x7f45c80ace90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:31.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.331+0000 7f45cdc00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45c8145190 0x7f45c80a9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:31.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.331+0000 7f45cdc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c806dd30 con 0x7f45c8071cd0 2024-03-24T01:32:31.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.331+0000 7f45cdc00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c80cc470 con 0x7f45c8145190 2024-03-24T01:32:31.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.331+0000 7f45cdc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45c806deb0 con 0x7f45c8144680 2024-03-24T01:32:31.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.332+0000 7f45c7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45c8071cd0 0x7f45c80ab4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:31.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.332+0000 7f45c7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45c8071cd0 0x7f45c80ab4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52884/0 (socket says 172.21.15.90:52884) 2024-03-24T01:32:31.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.332+0000 7f45c7e00640 1 -- 172.21.15.90:0/3036889507 learned_addr learned my addr 172.21.15.90:0/3036889507 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:31.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.332+0000 7f45c7e00640 1 -- 172.21.15.90:0/3036889507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45c8144680 msgr2=0x7f45c80ace90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:31.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.332+0000 7f45c7e00640 1 --2- 172.21.15.90:0/3036889507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45c8144680 0x7f45c80ace90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:31.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.332+0000 7f45c7e00640 1 -- 172.21.15.90:0/3036889507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45c8145190 msgr2=0x7f45c80a9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:31.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.332+0000 7f45c7e00640 1 --2- 172.21.15.90:0/3036889507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45c8145190 0x7f45c80a9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:31.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.332+0000 7f45c7e00640 1 -- 172.21.15.90:0/3036889507 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c80aba30 con 0x7f45c8071cd0 2024-03-24T01:32:31.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.332+0000 7f45c7e00640 1 --2- 172.21.15.90:0/3036889507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45c8071cd0 0x7f45c80ab4f0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f45b800d830 tx=0x7f45b800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:31.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.333+0000 7f45c4c00640 1 -- 172.21.15.90:0/3036889507 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b800b660 con 0x7f45c8071cd0 2024-03-24T01:32:31.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.333+0000 7f45c4c00640 1 -- 172.21.15.90:0/3036889507 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45b8019070 con 0x7f45c8071cd0 2024-03-24T01:32:31.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.333+0000 7f45cdc00640 1 -- 172.21.15.90:0/3036889507 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45c80ad3d0 con 0x7f45c8071cd0 2024-03-24T01:32:31.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.333+0000 7f45c4c00640 1 -- 172.21.15.90:0/3036889507 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b800ba20 con 0x7f45c8071cd0 2024-03-24T01:32:31.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.333+0000 7f45cdc00640 1 -- 172.21.15.90:0/3036889507 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45c80adc30 con 0x7f45c8071cd0 2024-03-24T01:32:31.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.334+0000 7f45cdc00640 1 -- 172.21.15.90:0/3036889507 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45c80ae340 con 0x7f45c8071cd0 2024-03-24T01:32:31.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.335+0000 7f45c4c00640 1 -- 172.21.15.90:0/3036889507 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f45b801e460 con 0x7f45c8071cd0 2024-03-24T01:32:31.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.335+0000 7f45c4c00640 1 --2- 172.21.15.90:0/3036889507 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f45a406bbd0 0x7f45a406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:31.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.336+0000 7f45c4c00640 1 -- 172.21.15.90:0/3036889507 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f45b808b3b0 con 0x7f45c8071cd0 2024-03-24T01:32:31.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.336+0000 7f45bf400640 1 --2- 172.21.15.90:0/3036889507 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f45a406bbd0 0x7f45a406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:31.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.337+0000 7f45bf400640 1 --2- 172.21.15.90:0/3036889507 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f45a406bbd0 0x7f45a406e090 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f45b0036000 tx=0x7f45b0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:31.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.340+0000 7f45c4c00640 1 -- 172.21.15.90:0/3036889507 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f45c80aba30 con 0x7f45c8071cd0 2024-03-24T01:32:31.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.576+0000 7f45cdc00640 1 -- 172.21.15.90:0/3036889507 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f45c804f400 con 0x7f45c8071cd0 2024-03-24T01:32:31.578 INFO:tasks.ceph.mon.a.smithi090.stderr:2024-03-24T01:32:31.577+0000 7fc6f2400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T01:32:31.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.580+0000 7f45c4c00640 1 -- 172.21.15.90:0/3036889507 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v42) v1 ==== 161+0+0 (secure 0 0 0) 0x7f45b8059be0 con 0x7f45c8071cd0 2024-03-24T01:32:31.583 INFO:teuthology.orchestra.run.smithi090.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T01:32:31.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.583+0000 7f45cdc00640 1 -- 172.21.15.90:0/3036889507 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f45a406bbd0 msgr2=0x7f45a406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:31.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.583+0000 7f45cdc00640 1 --2- 172.21.15.90:0/3036889507 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f45a406bbd0 0x7f45a406e090 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f45b0036000 tx=0x7f45b0030000 comp rx=0 tx=0).stop 2024-03-24T01:32:31.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.583+0000 7f45cdc00640 1 -- 172.21.15.90:0/3036889507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45c8071cd0 msgr2=0x7f45c80ab4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:31.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.583+0000 7f45cdc00640 1 --2- 172.21.15.90:0/3036889507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45c8071cd0 0x7f45c80ab4f0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f45b800d830 tx=0x7f45b800dcf0 comp rx=0 tx=0).stop 2024-03-24T01:32:31.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.584+0000 7f45cdc00640 1 -- 172.21.15.90:0/3036889507 shutdown_connections 2024-03-24T01:32:31.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.584+0000 7f45cdc00640 1 --2- 172.21.15.90:0/3036889507 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f45a406bbd0 0x7f45a406e090 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:31.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.584+0000 7f45cdc00640 1 --2- 172.21.15.90:0/3036889507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45c8145190 0x7f45c80a9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:31.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.584+0000 7f45cdc00640 1 --2- 172.21.15.90:0/3036889507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f45c8144680 0x7f45c80ace90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:31.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.584+0000 7f45cdc00640 1 --2- 172.21.15.90:0/3036889507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45c8071cd0 0x7f45c80ab4f0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:31.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.584+0000 7f45cdc00640 1 -- 172.21.15.90:0/3036889507 >> 172.21.15.90:0/3036889507 conn(0x7f45c806b0c0 msgr2=0x7f45c815e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:31.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.584+0000 7f45cdc00640 1 -- 172.21.15.90:0/3036889507 shutdown_connections 2024-03-24T01:32:31.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.584+0000 7f45cdc00640 1 -- 172.21.15.90:0/3036889507 wait complete. 2024-03-24T01:32:31.607 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-24T01:32:31.656 INFO:tasks.ceph.mds.c.smithi090.stdout:starting mds.c at 2024-03-24T01:32:31.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.787+0000 7f0ea9e00640 1 Processor -- start 2024-03-24T01:32:31.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.787+0000 7f0ea9e00640 1 -- start start 2024-03-24T01:32:31.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.788+0000 7f0ea9e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ea40aeeb0 0x7f0ea40b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:31.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.788+0000 7f0ea9e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ea406dac0 con 0x7f0ea415bc50 2024-03-24T01:32:31.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.788+0000 7f0ea9e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ea406dda0 con 0x7f0ea406b7d0 2024-03-24T01:32:31.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.788+0000 7f0ea9e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea40b4da0 con 0x7f0ea40aeeb0 2024-03-24T01:32:31.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.789+0000 7f0ea2000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f0ea415bc50 0x7f0ea406b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57616/0 (socket says 172.21.15.90:57616) 2024-03-24T01:32:31.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.789+0000 7f0ea2000640 1 -- 172.21.15.90:0/3941572569 learned_addr learned my addr 172.21.15.90:0/3941572569 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:32:31.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.789+0000 7f0ea2a00640 1 --2- 172.21.15.90:0/3941572569 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ea40aeeb0 0x7f0ea40b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:31.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.789+0000 7f0ea1600640 1 -- 172.21.15.90:0/3941572569 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3019614618 0 0) 0x7f0ea406dac0 con 0x7f0ea415bc50 2024-03-24T01:32:31.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.789+0000 7f0ea1600640 1 -- 172.21.15.90:0/3941572569 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e88003520 con 0x7f0ea415bc50 2024-03-24T01:32:31.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.789+0000 7f0ea1600640 1 -- 172.21.15.90:0/3941572569 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2105601568 0 0) 0x7f0e88003520 con 0x7f0ea415bc50 2024-03-24T01:32:31.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.789+0000 7f0ea1600640 1 -- 172.21.15.90:0/3941572569 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ea40aeeb0 msgr2=0x7f0ea40b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:31.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.789+0000 7f0ea1600640 1 --2- 172.21.15.90:0/3941572569 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ea40aeeb0 0x7f0ea40b44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:31.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.789+0000 7f0ea1600640 1 -- 172.21.15.90:0/3941572569 >> v1:172.21.15.120:6789/0 conn(0x7f0ea406b7d0 legacy=0x7f0ea40b2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:31.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.789+0000 7f0ea1600640 1 -- 172.21.15.90:0/3941572569 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea406e080 con 0x7f0ea415bc50 2024-03-24T01:32:31.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.789+0000 7f0ea1600640 1 -- 172.21.15.90:0/3941572569 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f0e90001670 con 0x7f0ea415bc50 2024-03-24T01:32:31.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.790+0000 7f0ea1600640 1 -- 172.21.15.90:0/3941572569 >> v1:172.21.15.90:6789/0 conn(0x7f0ea415bc50 legacy=0x7f0ea406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:31.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.790+0000 7f0ea1600640 1 --2- 172.21.15.90:0/3941572569 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e880039e0 0x7f0e88003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:31.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.790+0000 7f0ea1600640 1 --2- 172.21.15.90:0/3941572569 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ea40aeeb0 0x7f0e88004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:31.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.790+0000 7f0ea1600640 1 --2- 172.21.15.90:0/3941572569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e88004d40 0x7f0e880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:31.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.790+0000 7f0ea1600640 1 -- 172.21.15.90:0/3941572569 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea40b4da0 con 0x7f0e880039e0 2024-03-24T01:32:31.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.790+0000 7f0ea1600640 1 -- 172.21.15.90:0/3941572569 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8800bbf0 con 0x7f0e88004d40 2024-03-24T01:32:31.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.790+0000 7f0ea1600640 1 -- 172.21.15.90:0/3941572569 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8800bdf0 con 0x7f0ea40aeeb0 2024-03-24T01:32:31.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.790+0000 7f0ea2a00640 1 --2- 172.21.15.90:0/3941572569 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e880039e0 0x7f0e88003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:31.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.790+0000 7f0ea2a00640 1 -- 172.21.15.90:0/3941572569 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ea40aeeb0 msgr2=0x7f0e88004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:31.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.790+0000 7f0ea2a00640 1 --2- 172.21.15.90:0/3941572569 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ea40aeeb0 0x7f0e88004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:31.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.790+0000 7f0ea2a00640 1 -- 172.21.15.90:0/3941572569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e88004d40 msgr2=0x7f0e880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:31.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.790+0000 7f0ea2a00640 1 --2- 172.21.15.90:0/3941572569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e88004d40 0x7f0e880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:31.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.790+0000 7f0ea2a00640 1 -- 172.21.15.90:0/3941572569 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e88003520 con 0x7f0e880039e0 2024-03-24T01:32:31.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.791+0000 7f0ea2a00640 1 --2- 172.21.15.90:0/3941572569 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e880039e0 0x7f0e88003e20 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f0e9400bd60 tx=0x7f0e88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=509f6636df630a32 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:31.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.791+0000 7f0ea1600640 1 -- 172.21.15.90:0/3941572569 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9403c070 con 0x7f0e880039e0 2024-03-24T01:32:31.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.791+0000 7f0ea1600640 1 -- 172.21.15.90:0/3941572569 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e9402f710 con 0x7f0e880039e0 2024-03-24T01:32:31.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.791+0000 7f0ea1600640 1 -- 172.21.15.90:0/3941572569 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9400b930 con 0x7f0e880039e0 2024-03-24T01:32:31.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.792+0000 7f0ea9e00640 1 -- 172.21.15.90:0/3941572569 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e880039e0 msgr2=0x7f0e88003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:31.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.792+0000 7f0ea9e00640 1 --2- 172.21.15.90:0/3941572569 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e880039e0 0x7f0e88003e20 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f0e9400bd60 tx=0x7f0e88005830 comp rx=0 tx=0).stop 2024-03-24T01:32:31.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.792+0000 7f0ea3400640 1 -- 172.21.15.90:0/3941572569 reap_dead start 2024-03-24T01:32:31.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.792+0000 7f0ea9e00640 1 -- 172.21.15.90:0/3941572569 shutdown_connections 2024-03-24T01:32:31.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.792+0000 7f0ea9e00640 1 -- 172.21.15.90:0/3941572569 >> 172.21.15.90:0/3941572569 conn(0x7f0ea40b1770 msgr2=0x7f0ea40b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:31.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.792+0000 7f0ea9e00640 1 -- 172.21.15.90:0/3941572569 shutdown_connections 2024-03-24T01:32:31.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.792+0000 7f0ea9e00640 1 -- 172.21.15.90:0/3941572569 wait complete. 2024-03-24T01:32:31.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.794+0000 7f0ea9e00640 1 Processor -- start 2024-03-24T01:32:31.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.794+0000 7f0ea9e00640 1 -- start start 2024-03-24T01:32:31.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.795+0000 7f0ea9e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e880039e0 0x7f0ea41564c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:31.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.795+0000 7f0ea9e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ea4155d80 0x7f0ea41f2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:31.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.795+0000 7f0ea9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ea4153f90 0x7f0ea41543f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:31.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.795+0000 7f0ea9e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea406e880 con 0x7f0ea4153f90 2024-03-24T01:32:31.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.795+0000 7f0ea9e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea40af290 con 0x7f0e880039e0 2024-03-24T01:32:31.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.795+0000 7f0ea9e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea406ea00 con 0x7f0ea4155d80 2024-03-24T01:32:31.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.795+0000 7f0ea3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ea4153f90 0x7f0ea41543f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:31.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.795+0000 7f0ea3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ea4153f90 0x7f0ea41543f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52928/0 (socket says 172.21.15.90:52928) 2024-03-24T01:32:31.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.795+0000 7f0ea3400640 1 -- 172.21.15.90:0/2954669766 learned_addr learned my addr 172.21.15.90:0/2954669766 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:31.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.795+0000 7f0ea3400640 1 -- 172.21.15.90:0/2954669766 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ea4155d80 msgr2=0x7f0ea41f2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:31.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.795+0000 7f0ea3400640 1 --2- 172.21.15.90:0/2954669766 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ea4155d80 0x7f0ea41f2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:31.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.795+0000 7f0ea3400640 1 -- 172.21.15.90:0/2954669766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e880039e0 msgr2=0x7f0ea41564c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:31.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.795+0000 7f0ea3400640 1 --2- 172.21.15.90:0/2954669766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e880039e0 0x7f0ea41564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:31.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.795+0000 7f0ea3400640 1 -- 172.21.15.90:0/2954669766 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea4156b50 con 0x7f0ea4153f90 2024-03-24T01:32:31.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.796+0000 7f0ea3400640 1 --2- 172.21.15.90:0/2954669766 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ea4153f90 0x7f0ea41543f0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f0ea415c110 tx=0x7f0e9c00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:31.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.796+0000 7f0e9b400640 1 -- 172.21.15.90:0/2954669766 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c017070 con 0x7f0ea4153f90 2024-03-24T01:32:31.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.796+0000 7f0e9b400640 1 -- 172.21.15.90:0/2954669766 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e9c005d40 con 0x7f0ea4153f90 2024-03-24T01:32:31.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.796+0000 7f0ea9e00640 1 -- 172.21.15.90:0/2954669766 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ea40a85e0 con 0x7f0ea4153f90 2024-03-24T01:32:31.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.796+0000 7f0e9b400640 1 -- 172.21.15.90:0/2954669766 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c00a880 con 0x7f0ea4153f90 2024-03-24T01:32:31.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.796+0000 7f0ea9e00640 1 -- 172.21.15.90:0/2954669766 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ea40a8ba0 con 0x7f0ea4153f90 2024-03-24T01:32:31.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.797+0000 7f0ea9e00640 1 -- 172.21.15.90:0/2954669766 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ea404f400 con 0x7f0ea4153f90 2024-03-24T01:32:31.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.798+0000 7f0e9b400640 1 -- 172.21.15.90:0/2954669766 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0e9c01a070 con 0x7f0ea4153f90 2024-03-24T01:32:31.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.799+0000 7f0e9b400640 1 --2- 172.21.15.90:0/2954669766 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0e8406bbd0 0x7f0e8406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:31.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.799+0000 7f0e9b400640 1 -- 172.21.15.90:0/2954669766 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6331+0+0 (secure 0 0 0) 0x7f0e9c089d10 con 0x7f0ea4153f90 2024-03-24T01:32:31.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.800+0000 7f0ea2a00640 1 --2- 172.21.15.90:0/2954669766 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0e8406bbd0 0x7f0e8406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:31.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.801+0000 7f0ea2a00640 1 --2- 172.21.15.90:0/2954669766 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0e8406bbd0 0x7f0e8406e090 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f0e94039000 tx=0x7f0e94031000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:31.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:31.804+0000 7f0e9b400640 1 -- 172.21.15.90:0/2954669766 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0e9c058620 con 0x7f0ea4153f90 2024-03-24T01:32:32.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.041+0000 7f0ea9e00640 1 -- 172.21.15.90:0/2954669766 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f0ea4157c40 con 0x7f0ea4153f90 2024-03-24T01:32:32.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.591+0000 7f0e9b400640 1 -- 172.21.15.90:0/2954669766 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v43) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0e9c057f20 con 0x7f0ea4153f90 2024-03-24T01:32:32.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.594+0000 7f0ea9e00640 1 -- 172.21.15.90:0/2954669766 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0e8406bbd0 msgr2=0x7f0e8406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:32.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.594+0000 7f0ea9e00640 1 --2- 172.21.15.90:0/2954669766 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0e8406bbd0 0x7f0e8406e090 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f0e94039000 tx=0x7f0e94031000 comp rx=0 tx=0).stop 2024-03-24T01:32:32.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.594+0000 7f0ea9e00640 1 -- 172.21.15.90:0/2954669766 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ea4153f90 msgr2=0x7f0ea41543f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:32.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.594+0000 7f0ea9e00640 1 --2- 172.21.15.90:0/2954669766 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ea4153f90 0x7f0ea41543f0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f0ea415c110 tx=0x7f0e9c00cb90 comp rx=0 tx=0).stop 2024-03-24T01:32:32.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.594+0000 7f0ea9e00640 1 -- 172.21.15.90:0/2954669766 shutdown_connections 2024-03-24T01:32:32.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.594+0000 7f0ea9e00640 1 --2- 172.21.15.90:0/2954669766 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0e8406bbd0 0x7f0e8406e090 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:32.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.594+0000 7f0ea9e00640 1 --2- 172.21.15.90:0/2954669766 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ea4153f90 0x7f0ea41543f0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:32.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.594+0000 7f0ea9e00640 1 --2- 172.21.15.90:0/2954669766 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0ea4155d80 0x7f0ea41f2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:32.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.594+0000 7f0ea9e00640 1 --2- 172.21.15.90:0/2954669766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e880039e0 0x7f0ea41564c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:32.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.594+0000 7f0ea9e00640 1 -- 172.21.15.90:0/2954669766 >> 172.21.15.90:0/2954669766 conn(0x7f0ea40b1770 msgr2=0x7f0ea415c140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:32.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.594+0000 7f0ea9e00640 1 -- 172.21.15.90:0/2954669766 shutdown_connections 2024-03-24T01:32:32.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.594+0000 7f0ea9e00640 1 -- 172.21.15.90:0/2954669766 wait complete. 2024-03-24T01:32:32.615 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-24T01:32:32.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.871+0000 7f30f5800640 1 Processor -- start 2024-03-24T01:32:32.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.871+0000 7f30f5800640 1 -- start start 2024-03-24T01:32:32.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.873+0000 7f30f5800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30f0159620 0x7f30f006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:32.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.873+0000 7f30f5800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f30f006b600 0x7f30f00cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:32.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.873+0000 7f30f5800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f006bca0 con 0x7f30f00cef80 2024-03-24T01:32:32.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.873+0000 7f30f5800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30f00b4470 con 0x7f30f006b600 2024-03-24T01:32:32.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.873+0000 7f30f5800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30f006ba60 con 0x7f30f0159620 2024-03-24T01:32:32.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.873+0000 7f30e6000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f30f006b600 0x7f30f00cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:32.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.873+0000 7f30e6000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f30f006b600 0x7f30f00cea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52944/0 (socket says 172.21.15.90:52944) 2024-03-24T01:32:32.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.873+0000 7f30e6000640 1 -- 172.21.15.90:0/3756190061 learned_addr learned my addr 172.21.15.90:0/3756190061 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:32.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.873+0000 7f30eea00640 1 --2- 172.21.15.90:0/3756190061 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30f0159620 0x7f30f006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:32.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.874+0000 7f30e6000640 1 -- 172.21.15.90:0/3756190061 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30f0159620 msgr2=0x7f30f006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:32.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.874+0000 7f30e6000640 1 --2- 172.21.15.90:0/3756190061 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30f0159620 0x7f30f006b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:32.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.874+0000 7f30e6000640 1 -- 172.21.15.90:0/3756190061 >> v1:172.21.15.90:6789/0 conn(0x7f30f00cef80 legacy=0x7f30f00ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:32.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.874+0000 7f30e6000640 1 -- 172.21.15.90:0/3756190061 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f00b36f0 con 0x7f30f006b600 2024-03-24T01:32:32.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.875+0000 7f30e6000640 1 --2- 172.21.15.90:0/3756190061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30f006b600 0x7f30f00cea40 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f30d8009870 tx=0x7f30d802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ed104bc9dd8d95e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:32.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.875+0000 7f30ee000640 1 -- 172.21.15.90:0/3756190061 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d8035020 con 0x7f30f006b600 2024-03-24T01:32:32.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.875+0000 7f30ee000640 1 -- 172.21.15.90:0/3756190061 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30d8031640 con 0x7f30f006b600 2024-03-24T01:32:32.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.875+0000 7f30ee000640 1 -- 172.21.15.90:0/3756190061 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d802d040 con 0x7f30f006b600 2024-03-24T01:32:32.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.876+0000 7f30f5800640 1 -- 172.21.15.90:0/3756190061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30f006b600 msgr2=0x7f30f00cea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:32.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.876+0000 7f30f5800640 1 --2- 172.21.15.90:0/3756190061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30f006b600 0x7f30f00cea40 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f30d8009870 tx=0x7f30d802b8f0 comp rx=0 tx=0).stop 2024-03-24T01:32:32.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.876+0000 7f30f5800640 1 -- 172.21.15.90:0/3756190061 shutdown_connections 2024-03-24T01:32:32.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.876+0000 7f30f5800640 1 --2- 172.21.15.90:0/3756190061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30f006b600 0x7f30f00cea40 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:32.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.876+0000 7f30f5800640 1 --2- 172.21.15.90:0/3756190061 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30f0159620 0x7f30f006b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:32.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.876+0000 7f30f5800640 1 -- 172.21.15.90:0/3756190061 >> 172.21.15.90:0/3756190061 conn(0x7f30f00b1770 msgr2=0x7f30f00b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:32.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.876+0000 7f30f5800640 1 -- 172.21.15.90:0/3756190061 shutdown_connections 2024-03-24T01:32:32.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.876+0000 7f30f5800640 1 -- 172.21.15.90:0/3756190061 wait complete. 2024-03-24T01:32:32.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.878+0000 7f30f5800640 1 Processor -- start 2024-03-24T01:32:32.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.878+0000 7f30f5800640 1 -- start start 2024-03-24T01:32:32.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.879+0000 7f30f5800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30f006b600 0x7f30f014f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:32.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.879+0000 7f30f5800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30f00cef80 0x7f30f0154fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:32.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.879+0000 7f30f5800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30f0159620 0x7f30f0150f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:32.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.879+0000 7f30f5800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f00b3f50 con 0x7f30f0159620 2024-03-24T01:32:32.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.879+0000 7f30f5800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f00b3c70 con 0x7f30f00cef80 2024-03-24T01:32:32.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.879+0000 7f30f5800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30f0159a00 con 0x7f30f006b600 2024-03-24T01:32:32.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.879+0000 7f30ef400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30f0159620 0x7f30f0150f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:32.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.879+0000 7f30ef400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30f0159620 0x7f30f0150f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52948/0 (socket says 172.21.15.90:52948) 2024-03-24T01:32:32.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.879+0000 7f30ef400640 1 -- 172.21.15.90:0/2149353143 learned_addr learned my addr 172.21.15.90:0/2149353143 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:32.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.879+0000 7f30ef400640 1 -- 172.21.15.90:0/2149353143 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30f006b600 msgr2=0x7f30f014f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:32.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.879+0000 7f30eea00640 1 --2- 172.21.15.90:0/2149353143 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30f006b600 0x7f30f014f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:32.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.879+0000 7f30ef400640 1 --2- 172.21.15.90:0/2149353143 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30f006b600 0x7f30f014f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:32.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.879+0000 7f30ef400640 1 -- 172.21.15.90:0/2149353143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30f00cef80 msgr2=0x7f30f0154fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:32.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.879+0000 7f30e6000640 1 --2- 172.21.15.90:0/2149353143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30f00cef80 0x7f30f0154fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:32.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.879+0000 7f30ef400640 1 --2- 172.21.15.90:0/2149353143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30f00cef80 0x7f30f0154fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:32.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.879+0000 7f30ef400640 1 -- 172.21.15.90:0/2149353143 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f014e760 con 0x7f30f0159620 2024-03-24T01:32:32.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.880+0000 7f30ef400640 1 --2- 172.21.15.90:0/2149353143 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30f0159620 0x7f30f0150f50 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f30e800c710 tx=0x7f30e800cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:32.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.880+0000 7f30e7400640 1 -- 172.21.15.90:0/2149353143 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e8016020 con 0x7f30f0159620 2024-03-24T01:32:32.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.880+0000 7f30e7400640 1 -- 172.21.15.90:0/2149353143 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30e8006d50 con 0x7f30f0159620 2024-03-24T01:32:32.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.880+0000 7f30f5800640 1 -- 172.21.15.90:0/2149353143 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30f014ea40 con 0x7f30f0159620 2024-03-24T01:32:32.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.880+0000 7f30e7400640 1 -- 172.21.15.90:0/2149353143 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e8003bb0 con 0x7f30f0159620 2024-03-24T01:32:32.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.880+0000 7f30f5800640 1 -- 172.21.15.90:0/2149353143 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30f014f2e0 con 0x7f30f0159620 2024-03-24T01:32:32.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.881+0000 7f30f5800640 1 -- 172.21.15.90:0/2149353143 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30f014f4e0 con 0x7f30f0159620 2024-03-24T01:32:32.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.882+0000 7f30e7400640 1 -- 172.21.15.90:0/2149353143 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f30e800a850 con 0x7f30f0159620 2024-03-24T01:32:32.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.883+0000 7f30e7400640 1 --2- 172.21.15.90:0/2149353143 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f30cc06bbd0 0x7f30cc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:32.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.883+0000 7f30e7400640 1 -- 172.21.15.90:0/2149353143 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6331+0+0 (secure 0 0 0) 0x7f30e808a9a0 con 0x7f30f0159620 2024-03-24T01:32:32.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.884+0000 7f30eea00640 1 --2- 172.21.15.90:0/2149353143 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f30cc06bbd0 0x7f30cc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:32.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.885+0000 7f30eea00640 1 --2- 172.21.15.90:0/2149353143 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f30cc06bbd0 0x7f30cc06e090 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f30e0006fd0 tx=0x7f30e0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:32.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:32.887+0000 7f30e7400640 1 -- 172.21.15.90:0/2149353143 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f30f014f4e0 con 0x7f30f0159620 2024-03-24T01:32:33.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.083+0000 7f30f5800640 1 -- 172.21.15.90:0/2149353143 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f30f00b2e90 con 0x7f30f0159620 2024-03-24T01:32:33.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.594+0000 7f30e7400640 1 -- 172.21.15.90:0/2149353143 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v58) v1 ==== 181+0+0 (secure 0 0 0) 0x7f30e8020020 con 0x7f30f0159620 2024-03-24T01:32:33.596 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_metadata' removed 2024-03-24T01:32:33.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.598+0000 7f30f5800640 1 -- 172.21.15.90:0/2149353143 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f30cc06bbd0 msgr2=0x7f30cc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:33.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.598+0000 7f30f5800640 1 --2- 172.21.15.90:0/2149353143 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f30cc06bbd0 0x7f30cc06e090 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f30e0006fd0 tx=0x7f30e0008040 comp rx=0 tx=0).stop 2024-03-24T01:32:33.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.598+0000 7f30f5800640 1 -- 172.21.15.90:0/2149353143 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30f0159620 msgr2=0x7f30f0150f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:33.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.598+0000 7f30f5800640 1 --2- 172.21.15.90:0/2149353143 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30f0159620 0x7f30f0150f50 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f30e800c710 tx=0x7f30e800cbd0 comp rx=0 tx=0).stop 2024-03-24T01:32:33.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.598+0000 7f30f5800640 1 -- 172.21.15.90:0/2149353143 shutdown_connections 2024-03-24T01:32:33.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.598+0000 7f30f5800640 1 --2- 172.21.15.90:0/2149353143 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f30cc06bbd0 0x7f30cc06e090 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:33.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.598+0000 7f30f5800640 1 --2- 172.21.15.90:0/2149353143 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30f0159620 0x7f30f0150f50 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:33.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.598+0000 7f30f5800640 1 --2- 172.21.15.90:0/2149353143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30f00cef80 0x7f30f0154fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:33.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.598+0000 7f30f5800640 1 --2- 172.21.15.90:0/2149353143 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30f006b600 0x7f30f014f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:33.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.598+0000 7f30f5800640 1 -- 172.21.15.90:0/2149353143 >> 172.21.15.90:0/2149353143 conn(0x7f30f00b1770 msgr2=0x7f30f0151990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:33.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.599+0000 7f30f5800640 1 -- 172.21.15.90:0/2149353143 shutdown_connections 2024-03-24T01:32:33.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.599+0000 7f30f5800640 1 -- 172.21.15.90:0/2149353143 wait complete. 2024-03-24T01:32:33.623 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-24T01:32:33.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.823+0000 7f7bbf200640 1 Processor -- start 2024-03-24T01:32:33.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.824+0000 7f7bbf200640 1 -- start start 2024-03-24T01:32:33.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.825+0000 7f7bbf200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7bc00d35b0 0x7f7bc00d1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:33.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.825+0000 7f7bbf200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bc00d2510 con 0x7f7bc015de50 2024-03-24T01:32:33.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.825+0000 7f7bbf200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bc00d27f0 con 0x7f7bc00b5550 2024-03-24T01:32:33.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.825+0000 7f7bbf200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bc00d2ad0 con 0x7f7bc00d35b0 2024-03-24T01:32:33.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.825+0000 7f7bbe800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7bc00d35b0 0x7f7bc00d1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:33.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.825+0000 7f7bb5400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f7bc015de50 0x7f7bc00b6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57636/0 (socket says 172.21.15.90:57636) 2024-03-24T01:32:33.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.825+0000 7f7bb5400640 1 -- 172.21.15.90:0/928645059 learned_addr learned my addr 172.21.15.90:0/928645059 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:32:33.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.826+0000 7f7bbe800640 1 -- 172.21.15.90:0/928645059 >> v1:172.21.15.120:6789/0 conn(0x7f7bc00b5550 legacy=0x7f7bc00b5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:33.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.826+0000 7f7bbe800640 1 -- 172.21.15.90:0/928645059 >> v1:172.21.15.90:6789/0 conn(0x7f7bc015de50 legacy=0x7f7bc00b6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:33.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.826+0000 7f7bbe800640 1 -- 172.21.15.90:0/928645059 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc00d2db0 con 0x7f7bc00d35b0 2024-03-24T01:32:33.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.826+0000 7f7bbd400640 1 -- 172.21.15.90:0/928645059 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3129290955 0 0) 0x7f7bc00d2510 con 0x7f7bc015de50 2024-03-24T01:32:33.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.827+0000 7f7bbe800640 1 --2- 172.21.15.90:0/928645059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7bc00d35b0 0x7f7bc00d1ea0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f7bb8009d90 tx=0x7f7bb802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1696f45f46afdaee server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:33.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.827+0000 7f7bbd400640 1 -- 172.21.15.90:0/928645059 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb8039070 con 0x7f7bc00d35b0 2024-03-24T01:32:33.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.827+0000 7f7bbd400640 1 -- 172.21.15.90:0/928645059 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bb802cc60 con 0x7f7bc00d35b0 2024-03-24T01:32:33.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.827+0000 7f7bbd400640 1 -- 172.21.15.90:0/928645059 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb8034660 con 0x7f7bc00d35b0 2024-03-24T01:32:33.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.828+0000 7f7bbf200640 1 -- 172.21.15.90:0/928645059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7bc00d35b0 msgr2=0x7f7bc00d1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:33.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.828+0000 7f7bbf200640 1 --2- 172.21.15.90:0/928645059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7bc00d35b0 0x7f7bc00d1ea0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f7bb8009d90 tx=0x7f7bb802beb0 comp rx=0 tx=0).stop 2024-03-24T01:32:33.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.828+0000 7f7bbf200640 1 -- 172.21.15.90:0/928645059 shutdown_connections 2024-03-24T01:32:33.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.828+0000 7f7bbf200640 1 --2- 172.21.15.90:0/928645059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7bc00d35b0 0x7f7bc00d1ea0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:33.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.828+0000 7f7bbf200640 1 -- 172.21.15.90:0/928645059 >> 172.21.15.90:0/928645059 conn(0x7f7bc00d0550 msgr2=0x7f7bc00d0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:33.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.828+0000 7f7bbf200640 1 -- 172.21.15.90:0/928645059 shutdown_connections 2024-03-24T01:32:33.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.828+0000 7f7bbf200640 1 -- 172.21.15.90:0/928645059 wait complete. 2024-03-24T01:32:33.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.829+0000 7f7bbf200640 1 Processor -- start 2024-03-24T01:32:33.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.830+0000 7f7bbf200640 1 -- start start 2024-03-24T01:32:33.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.830+0000 7f7bbf200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7bc00b5550 0x7f7bc0148ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:33.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.830+0000 7f7bbf200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7bc00d35b0 0x7f7bc01598e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:33.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.830+0000 7f7bbf200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7bc015de50 0x7f7bc0160610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:33.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.830+0000 7f7bbf200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc00b96c0 con 0x7f7bc015de50 2024-03-24T01:32:33.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.830+0000 7f7bbf200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc00b9540 con 0x7f7bc00b5550 2024-03-24T01:32:33.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.830+0000 7f7bbf200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc00b9840 con 0x7f7bc00d35b0 2024-03-24T01:32:33.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.831+0000 7f7bbe800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7bc015de50 0x7f7bc0160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:33.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.831+0000 7f7bbe800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7bc015de50 0x7f7bc0160610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52960/0 (socket says 172.21.15.90:52960) 2024-03-24T01:32:33.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.831+0000 7f7bbe800640 1 -- 172.21.15.90:0/2694910139 learned_addr learned my addr 172.21.15.90:0/2694910139 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:33.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.831+0000 7f7bbe800640 1 -- 172.21.15.90:0/2694910139 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7bc00d35b0 msgr2=0x7f7bc01598e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:33.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.831+0000 7f7bbde00640 1 --2- 172.21.15.90:0/2694910139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7bc00b5550 0x7f7bc0148ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:33.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.831+0000 7f7bbe800640 1 --2- 172.21.15.90:0/2694910139 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7bc00d35b0 0x7f7bc01598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:33.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.831+0000 7f7bbe800640 1 -- 172.21.15.90:0/2694910139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7bc00b5550 msgr2=0x7f7bc0148ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:33.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.831+0000 7f7bbe800640 1 --2- 172.21.15.90:0/2694910139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7bc00b5550 0x7f7bc0148ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:33.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.831+0000 7f7bbe800640 1 -- 172.21.15.90:0/2694910139 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc015a1e0 con 0x7f7bc015de50 2024-03-24T01:32:33.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.831+0000 7f7bbde00640 1 --2- 172.21.15.90:0/2694910139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7bc00b5550 0x7f7bc0148ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:33.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.832+0000 7f7bbe800640 1 --2- 172.21.15.90:0/2694910139 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7bc015de50 0x7f7bc0160610 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f7bb8037040 tx=0x7f7bb8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:33.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.832+0000 7f7bb6a00640 1 -- 172.21.15.90:0/2694910139 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb8039050 con 0x7f7bc015de50 2024-03-24T01:32:33.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.832+0000 7f7bb6a00640 1 -- 172.21.15.90:0/2694910139 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bb803a040 con 0x7f7bc015de50 2024-03-24T01:32:33.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.832+0000 7f7bbf200640 1 -- 172.21.15.90:0/2694910139 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bc015a4c0 con 0x7f7bc015de50 2024-03-24T01:32:33.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.832+0000 7f7bb6a00640 1 -- 172.21.15.90:0/2694910139 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb803e610 con 0x7f7bc015de50 2024-03-24T01:32:33.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.832+0000 7f7bbf200640 1 -- 172.21.15.90:0/2694910139 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bc015e2d0 con 0x7f7bc015de50 2024-03-24T01:32:33.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.833+0000 7f7bbf200640 1 -- 172.21.15.90:0/2694910139 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bc004f400 con 0x7f7bc015de50 2024-03-24T01:32:33.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.834+0000 7f7bb6a00640 1 -- 172.21.15.90:0/2694910139 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f7bb8047020 con 0x7f7bc015de50 2024-03-24T01:32:33.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.835+0000 7f7bb6a00640 1 --2- 172.21.15.90:0/2694910139 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f7b9c06bbd0 0x7f7b9c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:33.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.835+0000 7f7bb6a00640 1 -- 172.21.15.90:0/2694910139 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5893+0+0 (secure 0 0 0) 0x7f7bb80acfe0 con 0x7f7bc015de50 2024-03-24T01:32:33.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.835+0000 7f7bbde00640 1 --2- 172.21.15.90:0/2694910139 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f7b9c06bbd0 0x7f7b9c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:33.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.837+0000 7f7bbde00640 1 --2- 172.21.15.90:0/2694910139 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f7b9c06bbd0 0x7f7b9c06e090 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f7bb0000f60 tx=0x7f7bb0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:33.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:33.839+0000 7f7bb6a00640 1 -- 172.21.15.90:0/2694910139 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f7bb804e020 con 0x7f7bc015de50 2024-03-24T01:32:34.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.036+0000 7f7bbf200640 1 -- 172.21.15.90:0/2694910139 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7bc00b44a0 con 0x7f7bc015de50 2024-03-24T01:32:34.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.605+0000 7f7bb6a00640 1 -- 172.21.15.90:0/2694910139 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v59) v1 ==== 169+0+0 (secure 0 0 0) 0x7f7bb807ba50 con 0x7f7bc015de50 2024-03-24T01:32:34.607 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_data' removed 2024-03-24T01:32:34.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.609+0000 7f7bbf200640 1 -- 172.21.15.90:0/2694910139 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f7b9c06bbd0 msgr2=0x7f7b9c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:34.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.609+0000 7f7bbf200640 1 --2- 172.21.15.90:0/2694910139 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f7b9c06bbd0 0x7f7b9c06e090 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f7bb0000f60 tx=0x7f7bb0009040 comp rx=0 tx=0).stop 2024-03-24T01:32:34.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.609+0000 7f7bbf200640 1 -- 172.21.15.90:0/2694910139 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7bc015de50 msgr2=0x7f7bc0160610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:34.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.609+0000 7f7bbf200640 1 --2- 172.21.15.90:0/2694910139 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7bc015de50 0x7f7bc0160610 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f7bb8037040 tx=0x7f7bb8004a40 comp rx=0 tx=0).stop 2024-03-24T01:32:34.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.609+0000 7f7bbf200640 1 -- 172.21.15.90:0/2694910139 shutdown_connections 2024-03-24T01:32:34.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.609+0000 7f7bbf200640 1 --2- 172.21.15.90:0/2694910139 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f7b9c06bbd0 0x7f7b9c06e090 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:34.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.609+0000 7f7bbf200640 1 --2- 172.21.15.90:0/2694910139 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7bc015de50 0x7f7bc0160610 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:34.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.609+0000 7f7bbf200640 1 --2- 172.21.15.90:0/2694910139 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f7bc00d35b0 0x7f7bc01598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:34.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.609+0000 7f7bbf200640 1 --2- 172.21.15.90:0/2694910139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7bc00b5550 0x7f7bc0148ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:34.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.609+0000 7f7bbf200640 1 -- 172.21.15.90:0/2694910139 >> 172.21.15.90:0/2694910139 conn(0x7f7bc00d0550 msgr2=0x7f7bc00b37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:34.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.610+0000 7f7bbf200640 1 -- 172.21.15.90:0/2694910139 shutdown_connections 2024-03-24T01:32:34.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.610+0000 7f7bbf200640 1 -- 172.21.15.90:0/2694910139 wait complete. 2024-03-24T01:32:34.636 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-24T01:32:34.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.830+0000 7fc378a00640 1 Processor -- start 2024-03-24T01:32:34.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.831+0000 7fc378a00640 1 -- start start 2024-03-24T01:32:34.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.832+0000 7fc378a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3740af970 0x7fc3740674a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:34.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.832+0000 7fc378a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc3740ad450 0x7fc3740ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:34.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.832+0000 7fc378a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3740cfff0 con 0x7fc374099b50 2024-03-24T01:32:34.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.832+0000 7fc378a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc37415e3e0 con 0x7fc3740ad450 2024-03-24T01:32:34.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.832+0000 7fc378a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3740d02d0 con 0x7fc3740af970 2024-03-24T01:32:34.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.832+0000 7fc372800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc3740ad450 0x7fc3740ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:34.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.833+0000 7fc372800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc3740ad450 0x7fc3740ad8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52966/0 (socket says 172.21.15.90:52966) 2024-03-24T01:32:34.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.833+0000 7fc372800640 1 -- 172.21.15.90:0/3520084069 learned_addr learned my addr 172.21.15.90:0/3520084069 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:34.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.833+0000 7fc372800640 1 -- 172.21.15.90:0/3520084069 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3740af970 msgr2=0x7fc3740674a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:34.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.833+0000 7fc372800640 1 --2- 172.21.15.90:0/3520084069 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3740af970 0x7fc3740674a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:34.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.833+0000 7fc372800640 1 -- 172.21.15.90:0/3520084069 >> v1:172.21.15.120:6789/0 conn(0x7fc374099b50 legacy=0x7fc3740afd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:34.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.833+0000 7fc372800640 1 -- 172.21.15.90:0/3520084069 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3740d05b0 con 0x7fc3740ad450 2024-03-24T01:32:34.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.834+0000 7fc372800640 1 --2- 172.21.15.90:0/3520084069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3740ad450 0x7fc3740ad8e0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fc36c009870 tx=0x7fc36c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=923d4daea47b217d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:34.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.834+0000 7fc371400640 1 -- 172.21.15.90:0/3520084069 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36c035020 con 0x7fc3740ad450 2024-03-24T01:32:34.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.834+0000 7fc371400640 1 -- 172.21.15.90:0/3520084069 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc36c031640 con 0x7fc3740ad450 2024-03-24T01:32:34.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.834+0000 7fc371400640 1 -- 172.21.15.90:0/3520084069 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36c02d040 con 0x7fc3740ad450 2024-03-24T01:32:34.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.835+0000 7fc378a00640 1 -- 172.21.15.90:0/3520084069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3740ad450 msgr2=0x7fc3740ad8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:34.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.835+0000 7fc378a00640 1 --2- 172.21.15.90:0/3520084069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3740ad450 0x7fc3740ad8e0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fc36c009870 tx=0x7fc36c02b8f0 comp rx=0 tx=0).stop 2024-03-24T01:32:34.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.835+0000 7fc378a00640 1 -- 172.21.15.90:0/3520084069 shutdown_connections 2024-03-24T01:32:34.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.835+0000 7fc378a00640 1 --2- 172.21.15.90:0/3520084069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3740ad450 0x7fc3740ad8e0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:34.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.835+0000 7fc378a00640 1 --2- 172.21.15.90:0/3520084069 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3740af970 0x7fc3740674a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:34.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.835+0000 7fc378a00640 1 -- 172.21.15.90:0/3520084069 >> 172.21.15.90:0/3520084069 conn(0x7fc3740b3c90 msgr2=0x7fc3740b4ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:34.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.835+0000 7fc378a00640 1 -- 172.21.15.90:0/3520084069 shutdown_connections 2024-03-24T01:32:34.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.835+0000 7fc378a00640 1 -- 172.21.15.90:0/3520084069 wait complete. 2024-03-24T01:32:34.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.837+0000 7fc378a00640 1 Processor -- start 2024-03-24T01:32:34.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.837+0000 7fc378a00640 1 -- start start 2024-03-24T01:32:34.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.838+0000 7fc378a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc374099b50 0x7fc374148960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:34.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.838+0000 7fc371e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc374099b50 0x7fc374148960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:34.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.838+0000 7fc371e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc374099b50 0x7fc374148960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52978/0 (socket says 172.21.15.90:52978) 2024-03-24T01:32:34.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.838+0000 7fc378a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3740ad450 0x7fc37414c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:34.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.838+0000 7fc371e00640 1 -- 172.21.15.90:0/3235207334 learned_addr learned my addr 172.21.15.90:0/3235207334 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:34.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.838+0000 7fc378a00640 1 --2- 172.21.15.90:0/3235207334 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3740af970 0x7fc3741598d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:34.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.838+0000 7fc378a00640 1 -- 172.21.15.90:0/3235207334 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3740ade20 con 0x7fc374099b50 2024-03-24T01:32:34.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.838+0000 7fc378a00640 1 -- 172.21.15.90:0/3235207334 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3740d0b70 con 0x7fc3740af970 2024-03-24T01:32:34.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.838+0000 7fc378a00640 1 -- 172.21.15.90:0/3235207334 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3740adfa0 con 0x7fc3740ad450 2024-03-24T01:32:34.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.839+0000 7fc371e00640 1 -- 172.21.15.90:0/3235207334 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3740ad450 msgr2=0x7fc37414c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:34.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.839+0000 7fc371e00640 1 --2- 172.21.15.90:0/3235207334 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3740ad450 0x7fc37414c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:34.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.839+0000 7fc371e00640 1 -- 172.21.15.90:0/3235207334 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3740af970 msgr2=0x7fc3741598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:34.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.839+0000 7fc371e00640 1 --2- 172.21.15.90:0/3235207334 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3740af970 0x7fc3741598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:34.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.839+0000 7fc371e00640 1 -- 172.21.15.90:0/3235207334 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3741497b0 con 0x7fc374099b50 2024-03-24T01:32:34.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.839+0000 7fc371e00640 1 --2- 172.21.15.90:0/3235207334 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc374099b50 0x7fc374148960 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fc36400ba50 tx=0x7fc36400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:34.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.839+0000 7fc36aa00640 1 -- 172.21.15.90:0/3235207334 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc364019070 con 0x7fc374099b50 2024-03-24T01:32:34.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.839+0000 7fc36aa00640 1 -- 172.21.15.90:0/3235207334 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc36400cc00 con 0x7fc374099b50 2024-03-24T01:32:34.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.839+0000 7fc378a00640 1 -- 172.21.15.90:0/3235207334 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc374149a90 con 0x7fc374099b50 2024-03-24T01:32:34.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.840+0000 7fc36aa00640 1 -- 172.21.15.90:0/3235207334 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc364014440 con 0x7fc374099b50 2024-03-24T01:32:34.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.840+0000 7fc378a00640 1 -- 172.21.15.90:0/3235207334 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc37414a330 con 0x7fc374099b50 2024-03-24T01:32:34.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.841+0000 7fc378a00640 1 -- 172.21.15.90:0/3235207334 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc374149e80 con 0x7fc374099b50 2024-03-24T01:32:34.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.842+0000 7fc36aa00640 1 -- 172.21.15.90:0/3235207334 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc36401c070 con 0x7fc374099b50 2024-03-24T01:32:34.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.843+0000 7fc36aa00640 1 --2- 172.21.15.90:0/3235207334 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc35006bc00 0x7fc35006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:34.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.843+0000 7fc36aa00640 1 -- 172.21.15.90:0/3235207334 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fc36408a690 con 0x7fc374099b50 2024-03-24T01:32:34.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.843+0000 7fc369400640 1 --2- 172.21.15.90:0/3235207334 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc35006bc00 0x7fc35006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:34.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.844+0000 7fc369400640 1 --2- 172.21.15.90:0/3235207334 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc35006bc00 0x7fc35006e0c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fc37409a430 tx=0x7fc35c007990 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:34.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:34.848+0000 7fc36aa00640 1 -- 172.21.15.90:0/3235207334 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc364059330 con 0x7fc374099b50 2024-03-24T01:32:35.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.045+0000 7fc378a00640 1 -- 172.21.15.90:0/3235207334 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fc37404f400 con 0x7fc374099b50 2024-03-24T01:32:35.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.050+0000 7fc36aa00640 1 -- 172.21.15.90:0/3235207334 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc364058c30 con 0x7fc374099b50 2024-03-24T01:32:35.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.053+0000 7fc378a00640 1 -- 172.21.15.90:0/3235207334 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc35006bc00 msgr2=0x7fc35006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:35.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.053+0000 7fc378a00640 1 --2- 172.21.15.90:0/3235207334 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc35006bc00 0x7fc35006e0c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fc37409a430 tx=0x7fc35c007990 comp rx=0 tx=0).stop 2024-03-24T01:32:35.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.053+0000 7fc378a00640 1 -- 172.21.15.90:0/3235207334 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc374099b50 msgr2=0x7fc374148960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:35.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.053+0000 7fc378a00640 1 --2- 172.21.15.90:0/3235207334 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc374099b50 0x7fc374148960 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fc36400ba50 tx=0x7fc36400bf10 comp rx=0 tx=0).stop 2024-03-24T01:32:35.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.053+0000 7fc378a00640 1 -- 172.21.15.90:0/3235207334 shutdown_connections 2024-03-24T01:32:35.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.053+0000 7fc378a00640 1 --2- 172.21.15.90:0/3235207334 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc35006bc00 0x7fc35006e0c0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:35.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.053+0000 7fc378a00640 1 --2- 172.21.15.90:0/3235207334 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3740af970 0x7fc3741598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:35.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.053+0000 7fc378a00640 1 --2- 172.21.15.90:0/3235207334 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc3740ad450 0x7fc37414c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:35.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.053+0000 7fc378a00640 1 --2- 172.21.15.90:0/3235207334 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc374099b50 0x7fc374148960 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:35.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.053+0000 7fc378a00640 1 -- 172.21.15.90:0/3235207334 >> 172.21.15.90:0/3235207334 conn(0x7fc3740b3c90 msgr2=0x7fc37406ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:35.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.054+0000 7fc378a00640 1 -- 172.21.15.90:0/3235207334 shutdown_connections 2024-03-24T01:32:35.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.054+0000 7fc378a00640 1 -- 172.21.15.90:0/3235207334 wait complete. 2024-03-24T01:32:35.075 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_journal_repair.TestJournalRepair.test_journal_smoke' 2024-03-24T01:32:35.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.281+0000 7fd018600640 1 Processor -- start 2024-03-24T01:32:35.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.281+0000 7fd018600640 1 -- start start 2024-03-24T01:32:35.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.283+0000 7fd018600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0100cdac0 0x7fd01006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:35.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.283+0000 7fd018600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd01006b600 0x7fd0100ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:35.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.283+0000 7fd018600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd01015bf70 con 0x7fd010155030 2024-03-24T01:32:35.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.283+0000 7fd018600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd01006ba70 con 0x7fd01006b600 2024-03-24T01:32:35.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.283+0000 7fd018600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd01015b360 con 0x7fd0100cdac0 2024-03-24T01:32:35.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.283+0000 7fd015c00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd010155030 0x7fd01015efc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57644/0 (socket says 172.21.15.90:57644) 2024-03-24T01:32:35.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.283+0000 7fd015c00640 1 -- 172.21.15.90:0/3648027095 learned_addr learned my addr 172.21.15.90:0/3648027095 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:32:35.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.283+0000 7fd016600640 1 --2- 172.21.15.90:0/3648027095 >> v2:172.21.15.90:3300/0 conn(0x7fd01006b600 0x7fd0100ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:35.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.284+0000 7fd015200640 1 --2- 172.21.15.90:0/3648027095 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0100cdac0 0x7fd01006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:35.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.284+0000 7fd00fe00640 1 -- 172.21.15.90:0/3648027095 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3046099227 0 0) 0x7fd01015bf70 con 0x7fd010155030 2024-03-24T01:32:35.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.284+0000 7fd00fe00640 1 -- 172.21.15.90:0/3648027095 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcff8003520 con 0x7fd010155030 2024-03-24T01:32:35.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.284+0000 7fd016600640 1 -- 172.21.15.90:0/3648027095 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0100cdac0 msgr2=0x7fd01006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:35.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.284+0000 7fd016600640 1 --2- 172.21.15.90:0/3648027095 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0100cdac0 0x7fd01006b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:35.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.284+0000 7fd016600640 1 -- 172.21.15.90:0/3648027095 >> v1:172.21.15.90:6789/0 conn(0x7fd010155030 legacy=0x7fd01015efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:35.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.284+0000 7fd016600640 1 -- 172.21.15.90:0/3648027095 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd01015b640 con 0x7fd01006b600 2024-03-24T01:32:35.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.285+0000 7fd016600640 1 --2- 172.21.15.90:0/3648027095 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd01006b600 0x7fd0100ae530 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fd008009d90 tx=0x7fd00802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b301dec9338e05e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:35.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.285+0000 7fd00fe00640 1 -- 172.21.15.90:0/3648027095 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd008039070 con 0x7fd01006b600 2024-03-24T01:32:35.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.285+0000 7fd00fe00640 1 -- 172.21.15.90:0/3648027095 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd00802cc50 con 0x7fd01006b600 2024-03-24T01:32:35.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.285+0000 7fd00fe00640 1 -- 172.21.15.90:0/3648027095 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd008034440 con 0x7fd01006b600 2024-03-24T01:32:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.286+0000 7fd018600640 1 -- 172.21.15.90:0/3648027095 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd01006b600 msgr2=0x7fd0100ae530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.286+0000 7fd018600640 1 --2- 172.21.15.90:0/3648027095 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd01006b600 0x7fd0100ae530 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fd008009d90 tx=0x7fd00802bea0 comp rx=0 tx=0).stop 2024-03-24T01:32:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.286+0000 7fd018600640 1 -- 172.21.15.90:0/3648027095 shutdown_connections 2024-03-24T01:32:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.286+0000 7fd018600640 1 --2- 172.21.15.90:0/3648027095 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd01006b600 0x7fd0100ae530 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.286+0000 7fd018600640 1 --2- 172.21.15.90:0/3648027095 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd0100cdac0 0x7fd01006b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.286+0000 7fd018600640 1 -- 172.21.15.90:0/3648027095 >> 172.21.15.90:0/3648027095 conn(0x7fd0100b1770 msgr2=0x7fd0100b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.286+0000 7fd018600640 1 -- 172.21.15.90:0/3648027095 shutdown_connections 2024-03-24T01:32:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.286+0000 7fd018600640 1 -- 172.21.15.90:0/3648027095 wait complete. 2024-03-24T01:32:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.287+0000 7fd018600640 1 Processor -- start 2024-03-24T01:32:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.288+0000 7fd018600640 1 -- start start 2024-03-24T01:32:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.288+0000 7fd018600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd01006b600 0x7fd0100ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:35.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.289+0000 7fd018600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0100cdac0 0x7fd010155640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:35.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.289+0000 7fd018600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd010155030 0x7fd0100cb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:35.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.289+0000 7fd018600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd010155410 con 0x7fd010155030 2024-03-24T01:32:35.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.289+0000 7fd018600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01015bc00 con 0x7fd0100cdac0 2024-03-24T01:32:35.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.289+0000 7fd018600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd01015af70 con 0x7fd01006b600 2024-03-24T01:32:35.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.289+0000 7fd016600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd010155030 0x7fd0100cb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:35.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.289+0000 7fd016600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd010155030 0x7fd0100cb3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53002/0 (socket says 172.21.15.90:53002) 2024-03-24T01:32:35.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.289+0000 7fd016600640 1 -- 172.21.15.90:0/2634683528 learned_addr learned my addr 172.21.15.90:0/2634683528 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:35.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.289+0000 7fd015c00640 1 --2- 172.21.15.90:0/2634683528 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd01006b600 0x7fd0100ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:35.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.289+0000 7fd016600640 1 -- 172.21.15.90:0/2634683528 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd01006b600 msgr2=0x7fd0100ca8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:35.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.289+0000 7fd015200640 1 --2- 172.21.15.90:0/2634683528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0100cdac0 0x7fd010155640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:35.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.289+0000 7fd016600640 1 --2- 172.21.15.90:0/2634683528 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd01006b600 0x7fd0100ca8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:35.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.289+0000 7fd016600640 1 -- 172.21.15.90:0/2634683528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0100cdac0 msgr2=0x7fd010155640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:35.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.289+0000 7fd016600640 1 --2- 172.21.15.90:0/2634683528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0100cdac0 0x7fd010155640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:35.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.289+0000 7fd016600640 1 -- 172.21.15.90:0/2634683528 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd010157080 con 0x7fd010155030 2024-03-24T01:32:35.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.290+0000 7fd016600640 1 --2- 172.21.15.90:0/2634683528 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd010155030 0x7fd0100cb3c0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fd008009d90 tx=0x7fd008004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:35.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.290+0000 7fd00d600640 1 -- 172.21.15.90:0/2634683528 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd008039050 con 0x7fd010155030 2024-03-24T01:32:35.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.290+0000 7fd00d600640 1 -- 172.21.15.90:0/2634683528 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd008045070 con 0x7fd010155030 2024-03-24T01:32:35.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.290+0000 7fd00d600640 1 -- 172.21.15.90:0/2634683528 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd008009450 con 0x7fd010155030 2024-03-24T01:32:35.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.290+0000 7fd018600640 1 -- 172.21.15.90:0/2634683528 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd010155fe0 con 0x7fd010155030 2024-03-24T01:32:35.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.290+0000 7fd018600640 1 -- 172.21.15.90:0/2634683528 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0101565a0 con 0x7fd010155030 2024-03-24T01:32:35.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.291+0000 7fd018600640 1 -- 172.21.15.90:0/2634683528 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd01006f300 con 0x7fd010155030 2024-03-24T01:32:35.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.292+0000 7fd00d600640 1 -- 172.21.15.90:0/2634683528 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd00803e610 con 0x7fd010155030 2024-03-24T01:32:35.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.293+0000 7fd00d600640 1 --2- 172.21.15.90:0/2634683528 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcff006bbd0 0x7fcff006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:35.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.293+0000 7fd00d600640 1 -- 172.21.15.90:0/2634683528 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fd0080ad450 con 0x7fd010155030 2024-03-24T01:32:35.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.293+0000 7fd015c00640 1 --2- 172.21.15.90:0/2634683528 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcff006bbd0 0x7fcff006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:35.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.295+0000 7fd015c00640 1 --2- 172.21.15.90:0/2634683528 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcff006bbd0 0x7fcff006e090 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fd0100cafb0 tx=0x7fd004009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:35.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.297+0000 7fd00d600640 1 -- 172.21.15.90:0/2634683528 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd01006f300 con 0x7fd010155030 2024-03-24T01:32:35.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.569+0000 7fd018600640 1 -- 172.21.15.90:0/2634683528 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_journal_repair.TestJournalRepair.test_journal_smoke"]} v 0) v1 -- 0x7fd010068060 con 0x7fd010155030 2024-03-24T01:32:35.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.608+0000 7fd00d600640 1 -- 172.21.15.90:0/2634683528 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_journal_repair.TestJournalRepair.test_journal_smoke"]}]=0 v370) v1 ==== 148+0+0 (secure 0 0 0) 0x7fd0080abe20 con 0x7fd010155030 2024-03-24T01:32:35.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.611+0000 7fd018600640 1 -- 172.21.15.90:0/2634683528 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcff006bbd0 msgr2=0x7fcff006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:35.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.611+0000 7fd018600640 1 --2- 172.21.15.90:0/2634683528 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcff006bbd0 0x7fcff006e090 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fd0100cafb0 tx=0x7fd004009040 comp rx=0 tx=0).stop 2024-03-24T01:32:35.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.611+0000 7fd018600640 1 -- 172.21.15.90:0/2634683528 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd010155030 msgr2=0x7fd0100cb3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:35.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.611+0000 7fd018600640 1 --2- 172.21.15.90:0/2634683528 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd010155030 0x7fd0100cb3c0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fd008009d90 tx=0x7fd008004a40 comp rx=0 tx=0).stop 2024-03-24T01:32:35.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.611+0000 7fd018600640 1 -- 172.21.15.90:0/2634683528 shutdown_connections 2024-03-24T01:32:35.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.611+0000 7fd018600640 1 --2- 172.21.15.90:0/2634683528 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcff006bbd0 0x7fcff006e090 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:35.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.611+0000 7fd018600640 1 --2- 172.21.15.90:0/2634683528 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd010155030 0x7fd0100cb3c0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:35.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.611+0000 7fd018600640 1 --2- 172.21.15.90:0/2634683528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0100cdac0 0x7fd010155640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:35.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.611+0000 7fd018600640 1 --2- 172.21.15.90:0/2634683528 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd01006b600 0x7fd0100ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:35.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.611+0000 7fd018600640 1 -- 172.21.15.90:0/2634683528 >> 172.21.15.90:0/2634683528 conn(0x7fd0100b1770 msgr2=0x7fd01006d990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:35.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.611+0000 7fd018600640 1 -- 172.21.15.90:0/2634683528 shutdown_connections 2024-03-24T01:32:35.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.611+0000 7fd018600640 1 -- 172.21.15.90:0/2634683528 wait complete. 2024-03-24T01:32:35.632 INFO:tasks.cephfs_test_runner:test_journal_smoke (tasks.cephfs.test_journal_repair.TestJournalRepair) ... ok 2024-03-24T01:32:35.632 INFO:tasks.cephfs_test_runner:Starting test: test_reset (tasks.cephfs.test_journal_repair.TestJournalRepair) That after forcibly modifying the backing store, we can get back into 2024-03-24T01:32:35.632 INFO:tasks.cephfs_test_runner:test_reset (tasks.cephfs.test_journal_repair.TestJournalRepair) 2024-03-24T01:32:35.632 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:32:35.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.848+0000 7fad62c00640 1 Processor -- start 2024-03-24T01:32:35.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.848+0000 7fad62c00640 1 -- start start 2024-03-24T01:32:35.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.849+0000 7fad62c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fad5c1469d0 0x7fad5c071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:35.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.849+0000 7fad62c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fad5c146f30 0x7fad5c072220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:35.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.849+0000 7fad62c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad5c0adf80 con 0x7fad5c147480 2024-03-24T01:32:35.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.849+0000 7fad62c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad5c1618a0 con 0x7fad5c146f30 2024-03-24T01:32:35.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.849+0000 7fad62c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad5c162700 con 0x7fad5c1469d0 2024-03-24T01:32:35.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.849+0000 7fad5b400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fad5c146f30 0x7fad5c072220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:35.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.849+0000 7fad5b400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fad5c146f30 0x7fad5c072220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53018/0 (socket says 172.21.15.90:53018) 2024-03-24T01:32:35.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.849+0000 7fad5b400640 1 -- 172.21.15.90:0/3476610656 learned_addr learned my addr 172.21.15.90:0/3476610656 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:35.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.849+0000 7fad5be00640 1 --2- 172.21.15.90:0/3476610656 >> v2:172.21.15.120:3300/0 conn(0x7fad5c1469d0 0x7fad5c071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:35.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.850+0000 7fad5b400640 1 -- 172.21.15.90:0/3476610656 >> v2:172.21.15.120:3300/0 conn(0x7fad5c1469d0 msgr2=0x7fad5c071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:35.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.850+0000 7fad5b400640 1 --2- 172.21.15.90:0/3476610656 >> v2:172.21.15.120:3300/0 conn(0x7fad5c1469d0 0x7fad5c071ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:35.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.850+0000 7fad5b400640 1 -- 172.21.15.90:0/3476610656 >> v1:172.21.15.120:6789/0 conn(0x7fad5c147480 legacy=0x7fad5c06d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:35.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.850+0000 7fad5b400640 1 -- 172.21.15.90:0/3476610656 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad5c161af0 con 0x7fad5c146f30 2024-03-24T01:32:35.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.851+0000 7fad5b400640 1 --2- 172.21.15.90:0/3476610656 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad5c146f30 0x7fad5c072220 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fad5c074710 tx=0x7fad4802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96cb8529bbfb625d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:35.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.851+0000 7fad5aa00640 1 -- 172.21.15.90:0/3476610656 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad48035020 con 0x7fad5c146f30 2024-03-24T01:32:35.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.851+0000 7fad5aa00640 1 -- 172.21.15.90:0/3476610656 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad48031870 con 0x7fad5c146f30 2024-03-24T01:32:35.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.851+0000 7fad5aa00640 1 -- 172.21.15.90:0/3476610656 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad48031c40 con 0x7fad5c146f30 2024-03-24T01:32:35.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.852+0000 7fad62c00640 1 -- 172.21.15.90:0/3476610656 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad5c146f30 msgr2=0x7fad5c072220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:35.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.852+0000 7fad62c00640 1 --2- 172.21.15.90:0/3476610656 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad5c146f30 0x7fad5c072220 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fad5c074710 tx=0x7fad4802b910 comp rx=0 tx=0).stop 2024-03-24T01:32:35.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.852+0000 7fad62c00640 1 -- 172.21.15.90:0/3476610656 shutdown_connections 2024-03-24T01:32:35.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.852+0000 7fad62c00640 1 --2- 172.21.15.90:0/3476610656 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad5c146f30 0x7fad5c072220 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:35.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.852+0000 7fad62c00640 1 --2- 172.21.15.90:0/3476610656 >> v2:172.21.15.120:3300/0 conn(0x7fad5c1469d0 0x7fad5c071ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:35.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.852+0000 7fad62c00640 1 -- 172.21.15.90:0/3476610656 >> 172.21.15.90:0/3476610656 conn(0x7fad5c06b2a0 msgr2=0x7fad5c06b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:35.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.852+0000 7fad62c00640 1 -- 172.21.15.90:0/3476610656 shutdown_connections 2024-03-24T01:32:35.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.852+0000 7fad62c00640 1 -- 172.21.15.90:0/3476610656 wait complete. 2024-03-24T01:32:35.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.853+0000 7fad62c00640 1 Processor -- start 2024-03-24T01:32:35.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.854+0000 7fad62c00640 1 -- start start 2024-03-24T01:32:35.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.854+0000 7fad62c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad5c1469d0 0x7fad5c15d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:35.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.854+0000 7fad62c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad5c146f30 0x7fad5c0b0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:35.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.854+0000 7fad62c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad5c147480 0x7fad5c0b0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:35.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.854+0000 7fad62c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5c162390 con 0x7fad5c147480 2024-03-24T01:32:35.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.854+0000 7fad62c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5c1620b0 con 0x7fad5c146f30 2024-03-24T01:32:35.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.854+0000 7fad62c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad5c0b3c10 con 0x7fad5c1469d0 2024-03-24T01:32:35.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.855+0000 7fad60c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad5c147480 0x7fad5c0b0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:35.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.855+0000 7fad60c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad5c147480 0x7fad5c0b0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53020/0 (socket says 172.21.15.90:53020) 2024-03-24T01:32:35.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.855+0000 7fad60c00640 1 -- 172.21.15.90:0/2781187440 learned_addr learned my addr 172.21.15.90:0/2781187440 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:35.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.855+0000 7fad5be00640 1 --2- 172.21.15.90:0/2781187440 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad5c1469d0 0x7fad5c15d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:35.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.855+0000 7fad5b400640 1 --2- 172.21.15.90:0/2781187440 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad5c146f30 0x7fad5c0b0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:35.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.855+0000 7fad60c00640 1 -- 172.21.15.90:0/2781187440 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad5c1469d0 msgr2=0x7fad5c15d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:35.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.855+0000 7fad60c00640 1 --2- 172.21.15.90:0/2781187440 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad5c1469d0 0x7fad5c15d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:35.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.855+0000 7fad60c00640 1 -- 172.21.15.90:0/2781187440 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad5c146f30 msgr2=0x7fad5c0b0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:35.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.855+0000 7fad60c00640 1 --2- 172.21.15.90:0/2781187440 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad5c146f30 0x7fad5c0b0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:35.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.855+0000 7fad60c00640 1 -- 172.21.15.90:0/2781187440 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad5c0aefb0 con 0x7fad5c147480 2024-03-24T01:32:35.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.855+0000 7fad5b400640 1 --2- 172.21.15.90:0/2781187440 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad5c146f30 0x7fad5c0b0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:35.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.856+0000 7fad60c00640 1 --2- 172.21.15.90:0/2781187440 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad5c147480 0x7fad5c0b0580 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fad5400c780 tx=0x7fad5400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:35.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.856+0000 7fad53e00640 1 -- 172.21.15.90:0/2781187440 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad54017070 con 0x7fad5c147480 2024-03-24T01:32:35.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.856+0000 7fad53e00640 1 -- 172.21.15.90:0/2781187440 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad54003da0 con 0x7fad5c147480 2024-03-24T01:32:35.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.856+0000 7fad53e00640 1 -- 172.21.15.90:0/2781187440 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad540052b0 con 0x7fad5c147480 2024-03-24T01:32:35.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.856+0000 7fad62c00640 1 -- 172.21.15.90:0/2781187440 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad5c0af290 con 0x7fad5c147480 2024-03-24T01:32:35.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.856+0000 7fad62c00640 1 -- 172.21.15.90:0/2781187440 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad5c0af850 con 0x7fad5c147480 2024-03-24T01:32:35.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.857+0000 7fad62c00640 1 -- 172.21.15.90:0/2781187440 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad5c0afa90 con 0x7fad5c147480 2024-03-24T01:32:35.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.858+0000 7fad53e00640 1 -- 172.21.15.90:0/2781187440 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fad54005450 con 0x7fad5c147480 2024-03-24T01:32:35.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.859+0000 7fad53e00640 1 --2- 172.21.15.90:0/2781187440 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fad3c06bbd0 0x7fad3c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:35.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.859+0000 7fad53e00640 1 -- 172.21.15.90:0/2781187440 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fad54089f50 con 0x7fad5c147480 2024-03-24T01:32:35.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.859+0000 7fad5be00640 1 --2- 172.21.15.90:0/2781187440 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fad3c06bbd0 0x7fad3c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:35.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.861+0000 7fad5be00640 1 --2- 172.21.15.90:0/2781187440 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fad3c06bbd0 0x7fad3c06e090 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fad4c006fd0 tx=0x7fad4c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:35.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:35.863+0000 7fad53e00640 1 -- 172.21.15.90:0/2781187440 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fad5c0afa90 con 0x7fad5c147480 2024-03-24T01:32:36.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.064+0000 7fad62c00640 1 -- 172.21.15.90:0/2781187440 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad5c073570 con 0x7fad5c147480 2024-03-24T01:32:36.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.065+0000 7fad53e00640 1 -- 172.21.15.90:0/2781187440 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+13397 (secure 0 0 0) 0x7fad54058b50 con 0x7fad5c147480 2024-03-24T01:32:36.066 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:32:36.067 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":59,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:32:34.596927+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:6835/226826781":"2024-03-25T01:32:31.578842+0000","172.21.15.90:6833/226826781":"2024-03-25T01:32:31.578842+0000","172.21.15.120:6836/266172244":"2024-03-25T01:29:08.935482+0000","172.21.15.120:6834/266172244":"2024-03-25T01:29:08.935482+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:32:36.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.067+0000 7fad62c00640 1 -- 172.21.15.90:0/2781187440 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fad3c06bbd0 msgr2=0x7fad3c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:36.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.067+0000 7fad62c00640 1 --2- 172.21.15.90:0/2781187440 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fad3c06bbd0 0x7fad3c06e090 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fad4c006fd0 tx=0x7fad4c008040 comp rx=0 tx=0).stop 2024-03-24T01:32:36.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.067+0000 7fad62c00640 1 -- 172.21.15.90:0/2781187440 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad5c147480 msgr2=0x7fad5c0b0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:36.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.067+0000 7fad62c00640 1 --2- 172.21.15.90:0/2781187440 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad5c147480 0x7fad5c0b0580 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fad5400c780 tx=0x7fad5400cc40 comp rx=0 tx=0).stop 2024-03-24T01:32:36.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.068+0000 7fad62c00640 1 -- 172.21.15.90:0/2781187440 shutdown_connections 2024-03-24T01:32:36.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.068+0000 7fad62c00640 1 --2- 172.21.15.90:0/2781187440 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fad3c06bbd0 0x7fad3c06e090 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:36.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.068+0000 7fad62c00640 1 --2- 172.21.15.90:0/2781187440 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad5c147480 0x7fad5c0b0580 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:36.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.068+0000 7fad62c00640 1 --2- 172.21.15.90:0/2781187440 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad5c146f30 0x7fad5c0b0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:36.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.068+0000 7fad62c00640 1 --2- 172.21.15.90:0/2781187440 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad5c1469d0 0x7fad5c15d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:36.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.068+0000 7fad62c00640 1 -- 172.21.15.90:0/2781187440 >> 172.21.15.90:0/2781187440 conn(0x7fad5c06b2a0 msgr2=0x7fad5c144080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:36.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.068+0000 7fad62c00640 1 -- 172.21.15.90:0/2781187440 shutdown_connections 2024-03-24T01:32:36.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.068+0000 7fad62c00640 1 -- 172.21.15.90:0/2781187440 wait complete. 2024-03-24T01:32:36.088 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T01:32:36.088 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:32:36.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.282+0000 7f413b600640 1 Processor -- start 2024-03-24T01:32:36.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.282+0000 7f413b600640 1 -- start start 2024-03-24T01:32:36.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.284+0000 7f413b600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f41340cf150 0x7f41340ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:36.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.284+0000 7f413b600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f413406bca0 con 0x7f413406b7d0 2024-03-24T01:32:36.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.284+0000 7f413b600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41340b36f0 con 0x7f4134159620 2024-03-24T01:32:36.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.284+0000 7f413b600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41340b4470 con 0x7f41340cf150 2024-03-24T01:32:36.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.284+0000 7f412be00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f413406b7d0 0x7f41340cea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57658/0 (socket says 172.21.15.90:57658) 2024-03-24T01:32:36.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.284+0000 7f412be00640 1 -- 172.21.15.90:0/3732308640 learned_addr learned my addr 172.21.15.90:0/3732308640 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:32:36.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.285+0000 7f4133e00640 1 -- 172.21.15.90:0/3732308640 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2641326356 0 0) 0x7f413406bca0 con 0x7f413406b7d0 2024-03-24T01:32:36.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.285+0000 7f4139600640 1 --2- 172.21.15.90:0/3732308640 >> v2:172.21.15.120:3300/0 conn(0x7f41340cf150 0x7f41340ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:36.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.285+0000 7f4133e00640 1 -- 172.21.15.90:0/3732308640 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4114003520 con 0x7f413406b7d0 2024-03-24T01:32:36.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.285+0000 7f4133e00640 1 -- 172.21.15.90:0/3732308640 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3108498132 0 0) 0x7f4114003520 con 0x7f413406b7d0 2024-03-24T01:32:36.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.285+0000 7f4133e00640 1 -- 172.21.15.90:0/3732308640 >> v2:172.21.15.120:3300/0 conn(0x7f41340cf150 msgr2=0x7f41340ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:36.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.285+0000 7f4133e00640 1 --2- 172.21.15.90:0/3732308640 >> v2:172.21.15.120:3300/0 conn(0x7f41340cf150 0x7f41340ae530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:36.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.285+0000 7f4133e00640 1 -- 172.21.15.90:0/3732308640 >> v1:172.21.15.120:6789/0 conn(0x7f4134159620 legacy=0x7f413406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:36.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.285+0000 7f4133e00640 1 -- 172.21.15.90:0/3732308640 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41340b3990 con 0x7f413406b7d0 2024-03-24T01:32:36.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.285+0000 7f4133e00640 1 -- 172.21.15.90:0/3732308640 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f4124001670 con 0x7f413406b7d0 2024-03-24T01:32:36.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.285+0000 7f4133e00640 1 -- 172.21.15.90:0/3732308640 >> v1:172.21.15.90:6789/0 conn(0x7f413406b7d0 legacy=0x7f41340cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:36.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.285+0000 7f4133e00640 1 --2- 172.21.15.90:0/3732308640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41140039b0 0x7f4114003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:36.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.285+0000 7f4133e00640 1 --2- 172.21.15.90:0/3732308640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4114004800 0x7f4114004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:36.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.285+0000 7f4133e00640 1 --2- 172.21.15.90:0/3732308640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f41140052e0 0x7f411400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:36.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.285+0000 7f4133e00640 1 -- 172.21.15.90:0/3732308640 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41340b4470 con 0x7f41140052e0 2024-03-24T01:32:36.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.285+0000 7f4133e00640 1 -- 172.21.15.90:0/3732308640 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f411400c1b0 con 0x7f41140039b0 2024-03-24T01:32:36.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.285+0000 7f4133e00640 1 -- 172.21.15.90:0/3732308640 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f411400c3b0 con 0x7f4114004800 2024-03-24T01:32:36.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.286+0000 7f4139600640 1 --2- 172.21.15.90:0/3732308640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f41140052e0 0x7f411400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:36.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.286+0000 7f4139600640 1 -- 172.21.15.90:0/3732308640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4114004800 msgr2=0x7f4114004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:36.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.286+0000 7f412be00640 1 --2- 172.21.15.90:0/3732308640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4114004800 0x7f4114004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:36.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.286+0000 7f4138c00640 1 --2- 172.21.15.90:0/3732308640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41140039b0 0x7f4114003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:36.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.286+0000 7f4139600640 1 --2- 172.21.15.90:0/3732308640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4114004800 0x7f4114004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:36.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.286+0000 7f4139600640 1 -- 172.21.15.90:0/3732308640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41140039b0 msgr2=0x7f4114003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:36.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.286+0000 7f4139600640 1 --2- 172.21.15.90:0/3732308640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41140039b0 0x7f4114003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:36.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.286+0000 7f4139600640 1 -- 172.21.15.90:0/3732308640 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4114003520 con 0x7f41140052e0 2024-03-24T01:32:36.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.286+0000 7f412be00640 1 --2- 172.21.15.90:0/3732308640 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4114004800 0x7f4114004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:36.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.287+0000 7f4139600640 1 -- 172.21.15.90:0/3732308640 reap_dead start 2024-03-24T01:32:36.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.287+0000 7f4139600640 1 --2- 172.21.15.90:0/3732308640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f41140052e0 0x7f411400b9c0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f412c007330 tx=0x7f4114005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc3e8130a630dd92 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:36.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.287+0000 7f4133e00640 1 -- 172.21.15.90:0/3732308640 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412c009be0 con 0x7f41140052e0 2024-03-24T01:32:36.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.287+0000 7f4133e00640 1 -- 172.21.15.90:0/3732308640 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f412c02ccd0 con 0x7f41140052e0 2024-03-24T01:32:36.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.287+0000 7f4133e00640 1 -- 172.21.15.90:0/3732308640 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412c003bd0 con 0x7f41140052e0 2024-03-24T01:32:36.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.288+0000 7f413b600640 1 -- 172.21.15.90:0/3732308640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f41140052e0 msgr2=0x7f411400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:36.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.288+0000 7f413b600640 1 --2- 172.21.15.90:0/3732308640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f41140052e0 0x7f411400b9c0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f412c007330 tx=0x7f4114005830 comp rx=0 tx=0).stop 2024-03-24T01:32:36.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.288+0000 7f413b600640 1 -- 172.21.15.90:0/3732308640 shutdown_connections 2024-03-24T01:32:36.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.288+0000 7f413b600640 1 --2- 172.21.15.90:0/3732308640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f41140052e0 0x7f411400b9c0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:36.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.288+0000 7f413b600640 1 -- 172.21.15.90:0/3732308640 >> 172.21.15.90:0/3732308640 conn(0x7f41340b1770 msgr2=0x7f41340b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:36.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.288+0000 7f413b600640 1 -- 172.21.15.90:0/3732308640 shutdown_connections 2024-03-24T01:32:36.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.288+0000 7f413b600640 1 -- 172.21.15.90:0/3732308640 wait complete. 2024-03-24T01:32:36.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.290+0000 7f413b600640 1 Processor -- start 2024-03-24T01:32:36.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.290+0000 7f413b600640 1 -- start start 2024-03-24T01:32:36.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.291+0000 7f413b600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41140052e0 0x7f41341435b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:36.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.291+0000 7f413b600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41341e4c20 0x7f41341e5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:36.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.291+0000 7f413b600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4134156f20 0x7f41341573b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:36.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.291+0000 7f413b600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4134159a00 con 0x7f4134156f20 2024-03-24T01:32:36.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.291+0000 7f413b600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41340b3f50 con 0x7f41341e4c20 2024-03-24T01:32:36.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.291+0000 7f413b600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f413415bfe0 con 0x7f41140052e0 2024-03-24T01:32:36.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.291+0000 7f4139600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4134156f20 0x7f41341573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:36.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.291+0000 7f4139600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4134156f20 0x7f41341573b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53036/0 (socket says 172.21.15.90:53036) 2024-03-24T01:32:36.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.291+0000 7f4139600640 1 -- 172.21.15.90:0/4133636187 learned_addr learned my addr 172.21.15.90:0/4133636187 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:36.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.292+0000 7f4138c00640 1 --2- 172.21.15.90:0/4133636187 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41140052e0 0x7f41341435b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:36.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.292+0000 7f412be00640 1 --2- 172.21.15.90:0/4133636187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41341e4c20 0x7f41341e5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:36.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.292+0000 7f4139600640 1 -- 172.21.15.90:0/4133636187 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41140052e0 msgr2=0x7f41341435b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:36.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.292+0000 7f4139600640 1 --2- 172.21.15.90:0/4133636187 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41140052e0 0x7f41341435b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:36.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.292+0000 7f4139600640 1 -- 172.21.15.90:0/4133636187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41341e4c20 msgr2=0x7f41341e5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:36.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.292+0000 7f4139600640 1 --2- 172.21.15.90:0/4133636187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41341e4c20 0x7f41341e5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:36.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.292+0000 7f4139600640 1 -- 172.21.15.90:0/4133636187 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41341f4980 con 0x7f4134156f20 2024-03-24T01:32:36.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.292+0000 7f4138c00640 1 --2- 172.21.15.90:0/4133636187 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41140052e0 0x7f41341435b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:36.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.292+0000 7f412be00640 1 --2- 172.21.15.90:0/4133636187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41341e4c20 0x7f41341e5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:36.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.292+0000 7f4139600640 1 --2- 172.21.15.90:0/4133636187 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4134156f20 0x7f41341573b0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f412c02c740 tx=0x7f412c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:36.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.292+0000 7f4131600640 1 -- 172.21.15.90:0/4133636187 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412c044070 con 0x7f4134156f20 2024-03-24T01:32:36.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.292+0000 7f4131600640 1 -- 172.21.15.90:0/4133636187 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f412c0045e0 con 0x7f4134156f20 2024-03-24T01:32:36.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.292+0000 7f4131600640 1 -- 172.21.15.90:0/4133636187 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412c009be0 con 0x7f4134156f20 2024-03-24T01:32:36.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.292+0000 7f413b600640 1 -- 172.21.15.90:0/4133636187 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41340c8ff0 con 0x7f4134156f20 2024-03-24T01:32:36.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.293+0000 7f413b600640 1 -- 172.21.15.90:0/4133636187 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41340c95b0 con 0x7f4134156f20 2024-03-24T01:32:36.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.294+0000 7f413b600640 1 -- 172.21.15.90:0/4133636187 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f413404f400 con 0x7f4134156f20 2024-03-24T01:32:36.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.294+0000 7f4131600640 1 -- 172.21.15.90:0/4133636187 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f412c047070 con 0x7f4134156f20 2024-03-24T01:32:36.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.295+0000 7f4131600640 1 --2- 172.21.15.90:0/4133636187 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f411006bbd0 0x7f411006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:36.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.295+0000 7f4131600640 1 -- 172.21.15.90:0/4133636187 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f412c0ac080 con 0x7f4134156f20 2024-03-24T01:32:36.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.296+0000 7f4138c00640 1 --2- 172.21.15.90:0/4133636187 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f411006bbd0 0x7f411006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:36.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.297+0000 7f4138c00640 1 --2- 172.21.15.90:0/4133636187 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f411006bbd0 0x7f411006e090 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f411c001630 tx=0x7f411c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:36.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.300+0000 7f4131600640 1 -- 172.21.15.90:0/4133636187 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f412c07ac90 con 0x7f4134156f20 2024-03-24T01:32:36.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.496+0000 7f413b600640 1 -- 172.21.15.90:0/4133636187 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f41340b2e90 con 0x7f4134156f20 2024-03-24T01:32:36.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.497+0000 7f4131600640 1 -- 172.21.15.90:0/4133636187 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v59) v1 ==== 93+0+10 (secure 0 0 0) 0x7f412c07a590 con 0x7f4134156f20 2024-03-24T01:32:36.499 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:32:36.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.500+0000 7f413b600640 1 -- 172.21.15.90:0/4133636187 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f411006bbd0 msgr2=0x7f411006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:36.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.500+0000 7f413b600640 1 --2- 172.21.15.90:0/4133636187 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f411006bbd0 0x7f411006e090 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f411c001630 tx=0x7f411c009040 comp rx=0 tx=0).stop 2024-03-24T01:32:36.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.500+0000 7f413b600640 1 -- 172.21.15.90:0/4133636187 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4134156f20 msgr2=0x7f41341573b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:36.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.500+0000 7f413b600640 1 --2- 172.21.15.90:0/4133636187 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4134156f20 0x7f41341573b0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f412c02c740 tx=0x7f412c004ab0 comp rx=0 tx=0).stop 2024-03-24T01:32:36.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.500+0000 7f413b600640 1 -- 172.21.15.90:0/4133636187 shutdown_connections 2024-03-24T01:32:36.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.500+0000 7f413b600640 1 --2- 172.21.15.90:0/4133636187 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f411006bbd0 0x7f411006e090 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:36.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.500+0000 7f413b600640 1 --2- 172.21.15.90:0/4133636187 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4134156f20 0x7f41341573b0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:36.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.500+0000 7f413b600640 1 --2- 172.21.15.90:0/4133636187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41341e4c20 0x7f41341e5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:36.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.500+0000 7f413b600640 1 --2- 172.21.15.90:0/4133636187 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f41140052e0 0x7f41341435b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:36.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.500+0000 7f413b600640 1 -- 172.21.15.90:0/4133636187 >> 172.21.15.90:0/4133636187 conn(0x7f41340b1770 msgr2=0x7f413415e6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:36.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.500+0000 7f413b600640 1 -- 172.21.15.90:0/4133636187 shutdown_connections 2024-03-24T01:32:36.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.500+0000 7f413b600640 1 -- 172.21.15.90:0/4133636187 wait complete. 2024-03-24T01:32:36.520 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_journal_repair.TestJournalRepair.test_reset' 2024-03-24T01:32:36.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.730+0000 7f90df000640 1 Processor -- start 2024-03-24T01:32:36.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.730+0000 7f90df000640 1 -- start start 2024-03-24T01:32:36.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.731+0000 7f90df000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f90d81603e0 0x7f90d806b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:36.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.732+0000 7f90df000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f90d8145430 0x7f90d806b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:36.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.732+0000 7f90df000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d80cd490 con 0x7f90d80cc150 2024-03-24T01:32:36.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.732+0000 7f90df000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90d80cd270 con 0x7f90d81603e0 2024-03-24T01:32:36.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.732+0000 7f90df000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90d815f900 con 0x7f90d8145430 2024-03-24T01:32:36.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.732+0000 7f90d7400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f90d81603e0 0x7f90d806b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:36.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.732+0000 7f90d7400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f90d81603e0 0x7f90d806b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53046/0 (socket says 172.21.15.90:53046) 2024-03-24T01:32:36.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.732+0000 7f90d7e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f90d80cc150 0x7f90d80ccb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57664/0 (socket says 172.21.15.90:57664) 2024-03-24T01:32:36.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.732+0000 7f90d7400640 1 -- 172.21.15.90:0/1608190400 learned_addr learned my addr 172.21.15.90:0/1608190400 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:36.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.732+0000 7f90dd000640 1 --2- 172.21.15.90:0/1608190400 >> v2:172.21.15.120:3300/0 conn(0x7f90d8145430 0x7f90d806b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:36.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.732+0000 7f90d7400640 1 -- 172.21.15.90:0/1608190400 >> v2:172.21.15.120:3300/0 conn(0x7f90d8145430 msgr2=0x7f90d806b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:36.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.733+0000 7f90d7400640 1 --2- 172.21.15.90:0/1608190400 >> v2:172.21.15.120:3300/0 conn(0x7f90d8145430 0x7f90d806b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:36.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.733+0000 7f90d7400640 1 -- 172.21.15.90:0/1608190400 >> v1:172.21.15.90:6789/0 conn(0x7f90d80cc150 legacy=0x7f90d80ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:36.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.733+0000 7f90d7400640 1 -- 172.21.15.90:0/1608190400 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d815fbe0 con 0x7f90d81603e0 2024-03-24T01:32:36.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.733+0000 7f90d6a00640 1 -- 172.21.15.90:0/1608190400 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1105734414 0 0) 0x7f90d80cd490 con 0x7f90d80cc150 2024-03-24T01:32:36.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.734+0000 7f90d7400640 1 --2- 172.21.15.90:0/1608190400 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90d81603e0 0x7f90d806b0c0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f90d806fc10 tx=0x7f90c402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=735ad0e8d1295cec server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:36.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.734+0000 7f90d6a00640 1 -- 172.21.15.90:0/1608190400 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c4034070 con 0x7f90d81603e0 2024-03-24T01:32:36.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.734+0000 7f90d6a00640 1 -- 172.21.15.90:0/1608190400 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c4030860 con 0x7f90d81603e0 2024-03-24T01:32:36.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.734+0000 7f90d6a00640 1 -- 172.21.15.90:0/1608190400 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c4030c30 con 0x7f90d81603e0 2024-03-24T01:32:36.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.735+0000 7f90df000640 1 -- 172.21.15.90:0/1608190400 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90d81603e0 msgr2=0x7f90d806b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:36.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.735+0000 7f90df000640 1 --2- 172.21.15.90:0/1608190400 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90d81603e0 0x7f90d806b0c0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f90d806fc10 tx=0x7f90c402b9a0 comp rx=0 tx=0).stop 2024-03-24T01:32:36.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.735+0000 7f90df000640 1 -- 172.21.15.90:0/1608190400 shutdown_connections 2024-03-24T01:32:36.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.735+0000 7f90df000640 1 --2- 172.21.15.90:0/1608190400 >> v2:172.21.15.120:3300/0 conn(0x7f90d8145430 0x7f90d806b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:36.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.735+0000 7f90df000640 1 --2- 172.21.15.90:0/1608190400 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90d81603e0 0x7f90d806b0c0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:36.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.735+0000 7f90df000640 1 -- 172.21.15.90:0/1608190400 >> 172.21.15.90:0/1608190400 conn(0x7f90d80b1770 msgr2=0x7f90d80b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:36.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.735+0000 7f90df000640 1 -- 172.21.15.90:0/1608190400 shutdown_connections 2024-03-24T01:32:36.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.735+0000 7f90df000640 1 -- 172.21.15.90:0/1608190400 wait complete. 2024-03-24T01:32:36.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.737+0000 7f90df000640 1 Processor -- start 2024-03-24T01:32:36.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.737+0000 7f90df000640 1 -- start start 2024-03-24T01:32:36.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.738+0000 7f90df000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90d80cc150 0x7f90d815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:36.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.738+0000 7f90df000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90d8145430 0x7f90d8139b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:36.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.738+0000 7f90df000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90d81603e0 0x7f90d8157e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:36.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.738+0000 7f90df000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d80b38d0 con 0x7f90d81603e0 2024-03-24T01:32:36.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.738+0000 7f90df000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d80b3750 con 0x7f90d8145430 2024-03-24T01:32:36.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.738+0000 7f90df000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90d80b3a50 con 0x7f90d80cc150 2024-03-24T01:32:36.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.738+0000 7f90dd000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90d81603e0 0x7f90d8157e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:36.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.738+0000 7f90dd000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90d81603e0 0x7f90d8157e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53056/0 (socket says 172.21.15.90:53056) 2024-03-24T01:32:36.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.738+0000 7f90dd000640 1 -- 172.21.15.90:0/158337623 learned_addr learned my addr 172.21.15.90:0/158337623 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:36.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.738+0000 7f90d7e00640 1 --2- 172.21.15.90:0/158337623 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90d80cc150 0x7f90d815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:36.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.738+0000 7f90d7400640 1 --2- 172.21.15.90:0/158337623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90d8145430 0x7f90d8139b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:36.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.739+0000 7f90dd000640 1 -- 172.21.15.90:0/158337623 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90d80cc150 msgr2=0x7f90d815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:36.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.739+0000 7f90dd000640 1 --2- 172.21.15.90:0/158337623 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90d80cc150 0x7f90d815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:36.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.739+0000 7f90dd000640 1 -- 172.21.15.90:0/158337623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90d8145430 msgr2=0x7f90d8139b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:36.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.739+0000 7f90dd000640 1 --2- 172.21.15.90:0/158337623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90d8145430 0x7f90d8139b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:36.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.739+0000 7f90dd000640 1 -- 172.21.15.90:0/158337623 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d813a200 con 0x7f90d81603e0 2024-03-24T01:32:36.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.739+0000 7f90d7400640 1 --2- 172.21.15.90:0/158337623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90d8145430 0x7f90d8139b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:36.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.739+0000 7f90dd000640 1 --2- 172.21.15.90:0/158337623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90d81603e0 0x7f90d8157e40 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f90d0007e40 tx=0x7f90d000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:36.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.739+0000 7f90cfe00640 1 -- 172.21.15.90:0/158337623 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d0014620 con 0x7f90d81603e0 2024-03-24T01:32:36.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.739+0000 7f90cfe00640 1 -- 172.21.15.90:0/158337623 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90d0019070 con 0x7f90d81603e0 2024-03-24T01:32:36.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.739+0000 7f90cfe00640 1 -- 172.21.15.90:0/158337623 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d0014a20 con 0x7f90d81603e0 2024-03-24T01:32:36.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.739+0000 7f90df000640 1 -- 172.21.15.90:0/158337623 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90d813c540 con 0x7f90d81603e0 2024-03-24T01:32:36.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.739+0000 7f90df000640 1 -- 172.21.15.90:0/158337623 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90d813cb00 con 0x7f90d81603e0 2024-03-24T01:32:36.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.740+0000 7f90df000640 1 -- 172.21.15.90:0/158337623 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90d8068060 con 0x7f90d81603e0 2024-03-24T01:32:36.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.741+0000 7f90cfe00640 1 -- 172.21.15.90:0/158337623 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f90d0014bc0 con 0x7f90d81603e0 2024-03-24T01:32:36.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.742+0000 7f90cfe00640 1 --2- 172.21.15.90:0/158337623 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f90b806bc00 0x7f90b806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:36.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.742+0000 7f90cfe00640 1 -- 172.21.15.90:0/158337623 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f90d008b2e0 con 0x7f90d81603e0 2024-03-24T01:32:36.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.743+0000 7f90d7e00640 1 --2- 172.21.15.90:0/158337623 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f90b806bc00 0x7f90b806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:36.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.744+0000 7f90d7e00640 1 --2- 172.21.15.90:0/158337623 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f90b806bc00 0x7f90b806e0c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f90c8000f60 tx=0x7f90c8007b50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:36.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:36.747+0000 7f90cfe00640 1 -- 172.21.15.90:0/158337623 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f90d0059e50 con 0x7f90d81603e0 2024-03-24T01:32:37.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.014+0000 7f90df000640 1 -- 172.21.15.90:0/158337623 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_journal_repair.TestJournalRepair.test_reset"]} v 0) v1 -- 0x7f90d80b3220 con 0x7f90d81603e0 2024-03-24T01:32:37.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.621+0000 7f90cfe00640 1 -- 172.21.15.90:0/158337623 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_journal_repair.TestJournalRepair.test_reset"]}]=0 v372) v1 ==== 143+0+0 (secure 0 0 0) 0x7f90d0059750 con 0x7f90d81603e0 2024-03-24T01:32:37.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.624+0000 7f90df000640 1 -- 172.21.15.90:0/158337623 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f90b806bc00 msgr2=0x7f90b806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:37.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.624+0000 7f90df000640 1 --2- 172.21.15.90:0/158337623 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f90b806bc00 0x7f90b806e0c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f90c8000f60 tx=0x7f90c8007b50 comp rx=0 tx=0).stop 2024-03-24T01:32:37.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.624+0000 7f90df000640 1 -- 172.21.15.90:0/158337623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90d81603e0 msgr2=0x7f90d8157e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:37.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.624+0000 7f90df000640 1 --2- 172.21.15.90:0/158337623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90d81603e0 0x7f90d8157e40 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f90d0007e40 tx=0x7f90d000e4b0 comp rx=0 tx=0).stop 2024-03-24T01:32:37.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.625+0000 7f90df000640 1 -- 172.21.15.90:0/158337623 shutdown_connections 2024-03-24T01:32:37.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.625+0000 7f90df000640 1 --2- 172.21.15.90:0/158337623 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f90b806bc00 0x7f90b806e0c0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:37.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.625+0000 7f90df000640 1 --2- 172.21.15.90:0/158337623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90d81603e0 0x7f90d8157e40 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:37.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.625+0000 7f90df000640 1 --2- 172.21.15.90:0/158337623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90d8145430 0x7f90d8139b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:37.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.625+0000 7f90df000640 1 --2- 172.21.15.90:0/158337623 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f90d80cc150 0x7f90d815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:37.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.625+0000 7f90df000640 1 -- 172.21.15.90:0/158337623 >> 172.21.15.90:0/158337623 conn(0x7f90d80b1770 msgr2=0x7f90d80cd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:37.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.625+0000 7f90df000640 1 -- 172.21.15.90:0/158337623 shutdown_connections 2024-03-24T01:32:37.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.625+0000 7f90df000640 1 -- 172.21.15.90:0/158337623 wait complete. 2024-03-24T01:32:37.647 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-24T01:32:37.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.853+0000 7fa69b200640 1 Processor -- start 2024-03-24T01:32:37.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.853+0000 7fa69b200640 1 -- start start 2024-03-24T01:32:37.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.855+0000 7fa69b200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fa69c06cdd0 0x7fa69c0b6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:37.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.855+0000 7fa69b200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa69c0b2720 0x7fa69c0d33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:37.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.855+0000 7fa69b200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa69c0b9460 con 0x7fa69c15dd90 2024-03-24T01:32:37.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.855+0000 7fa69b200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa69c0b9a20 con 0x7fa69c0b2720 2024-03-24T01:32:37.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.855+0000 7fa69b200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa69c0b9740 con 0x7fa69c06cdd0 2024-03-24T01:32:37.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.855+0000 7fa69a800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa69c0b2720 0x7fa69c0d33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:37.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.855+0000 7fa69a800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa69c0b2720 0x7fa69c0d33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53072/0 (socket says 172.21.15.90:53072) 2024-03-24T01:32:37.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.855+0000 7fa69a800640 1 -- 172.21.15.90:0/3962980597 learned_addr learned my addr 172.21.15.90:0/3962980597 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:37.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.855+0000 7fa693e00640 1 --2- 172.21.15.90:0/3962980597 >> v2:172.21.15.120:3300/0 conn(0x7fa69c06cdd0 0x7fa69c0b6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:37.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.855+0000 7fa69a800640 1 -- 172.21.15.90:0/3962980597 >> v2:172.21.15.120:3300/0 conn(0x7fa69c06cdd0 msgr2=0x7fa69c0b6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:37.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.855+0000 7fa69a800640 1 --2- 172.21.15.90:0/3962980597 >> v2:172.21.15.120:3300/0 conn(0x7fa69c06cdd0 0x7fa69c0b6af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:37.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.855+0000 7fa69a800640 1 -- 172.21.15.90:0/3962980597 >> v1:172.21.15.120:6789/0 conn(0x7fa69c15dd90 legacy=0x7fa69c06c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:37.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.855+0000 7fa69a800640 1 -- 172.21.15.90:0/3962980597 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa69c0d1c10 con 0x7fa69c0b2720 2024-03-24T01:32:37.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.857+0000 7fa69a800640 1 --2- 172.21.15.90:0/3962980597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa69c0b2720 0x7fa69c0d33f0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fa69c0b6580 tx=0x7fa69402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8b91909dd54c9af server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:37.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.857+0000 7fa699400640 1 -- 172.21.15.90:0/3962980597 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69402ca80 con 0x7fa69c0b2720 2024-03-24T01:32:37.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.857+0000 7fa699400640 1 -- 172.21.15.90:0/3962980597 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa69402cc20 con 0x7fa69c0b2720 2024-03-24T01:32:37.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.857+0000 7fa699400640 1 -- 172.21.15.90:0/3962980597 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6940343f0 con 0x7fa69c0b2720 2024-03-24T01:32:37.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.857+0000 7fa69b200640 1 -- 172.21.15.90:0/3962980597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa69c0b2720 msgr2=0x7fa69c0d33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:37.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.857+0000 7fa69b200640 1 --2- 172.21.15.90:0/3962980597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa69c0b2720 0x7fa69c0d33f0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fa69c0b6580 tx=0x7fa69402bee0 comp rx=0 tx=0).stop 2024-03-24T01:32:37.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.857+0000 7fa69b200640 1 -- 172.21.15.90:0/3962980597 shutdown_connections 2024-03-24T01:32:37.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.857+0000 7fa69b200640 1 --2- 172.21.15.90:0/3962980597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa69c0b2720 0x7fa69c0d33f0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:37.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.857+0000 7fa69b200640 1 --2- 172.21.15.90:0/3962980597 >> v2:172.21.15.120:3300/0 conn(0x7fa69c06cdd0 0x7fa69c0b6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:37.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.857+0000 7fa69b200640 1 -- 172.21.15.90:0/3962980597 >> 172.21.15.90:0/3962980597 conn(0x7fa69c072e20 msgr2=0x7fa69c0d3a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:37.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.858+0000 7fa69b200640 1 -- 172.21.15.90:0/3962980597 shutdown_connections 2024-03-24T01:32:37.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.858+0000 7fa69b200640 1 -- 172.21.15.90:0/3962980597 wait complete. 2024-03-24T01:32:37.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.859+0000 7fa69b200640 1 Processor -- start 2024-03-24T01:32:37.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.859+0000 7fa69b200640 1 -- start start 2024-03-24T01:32:37.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.860+0000 7fa69b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa69c06cdd0 0x7fa69c0b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:37.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.860+0000 7fa69b200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa69c0b2720 0x7fa69c0b3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:37.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.860+0000 7fa69b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa69c15dd90 0x7fa69c0b4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:37.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.860+0000 7fa69b200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa69c0d2590 con 0x7fa69c15dd90 2024-03-24T01:32:37.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.860+0000 7fa69b200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa69c0d2410 con 0x7fa69c0b2720 2024-03-24T01:32:37.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.860+0000 7fa69b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa69c0d2710 con 0x7fa69c06cdd0 2024-03-24T01:32:37.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.860+0000 7fa69a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa69c15dd90 0x7fa69c0b4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:37.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.860+0000 7fa69a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa69c15dd90 0x7fa69c0b4520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53084/0 (socket says 172.21.15.90:53084) 2024-03-24T01:32:37.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.860+0000 7fa69a800640 1 -- 172.21.15.90:0/1249908852 learned_addr learned my addr 172.21.15.90:0/1249908852 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:37.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.861+0000 7fa69a800640 1 -- 172.21.15.90:0/1249908852 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa69c06cdd0 msgr2=0x7fa69c0b36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:37.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.861+0000 7fa693e00640 1 --2- 172.21.15.90:0/1249908852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa69c0b2720 0x7fa69c0b3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:37.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.861+0000 7fa69a800640 1 --2- 172.21.15.90:0/1249908852 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa69c06cdd0 0x7fa69c0b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:37.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.861+0000 7fa69a800640 1 -- 172.21.15.90:0/1249908852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa69c0b2720 msgr2=0x7fa69c0b3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:37.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.861+0000 7fa69a800640 1 --2- 172.21.15.90:0/1249908852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa69c0b2720 0x7fa69c0b3fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:37.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.861+0000 7fa69a800640 1 -- 172.21.15.90:0/1249908852 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa69c0b5fd0 con 0x7fa69c15dd90 2024-03-24T01:32:37.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.861+0000 7fa69a800640 1 --2- 172.21.15.90:0/1249908852 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa69c15dd90 0x7fa69c0b4520 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fa694009f40 tx=0x7fa694004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:37.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.861+0000 7fa692000640 1 -- 172.21.15.90:0/1249908852 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa694043020 con 0x7fa69c15dd90 2024-03-24T01:32:37.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.861+0000 7fa692000640 1 -- 172.21.15.90:0/1249908852 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa69403e450 con 0x7fa69c15dd90 2024-03-24T01:32:37.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.861+0000 7fa69b200640 1 -- 172.21.15.90:0/1249908852 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa69c0b62b0 con 0x7fa69c15dd90 2024-03-24T01:32:37.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.861+0000 7fa692000640 1 -- 172.21.15.90:0/1249908852 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa694009510 con 0x7fa69c15dd90 2024-03-24T01:32:37.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.862+0000 7fa69b200640 1 -- 172.21.15.90:0/1249908852 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa69c160a30 con 0x7fa69c15dd90 2024-03-24T01:32:37.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.863+0000 7fa69b200640 1 -- 172.21.15.90:0/1249908852 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa69c04f400 con 0x7fa69c15dd90 2024-03-24T01:32:37.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.864+0000 7fa692000640 1 -- 172.21.15.90:0/1249908852 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa694005d10 con 0x7fa69c15dd90 2024-03-24T01:32:37.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.864+0000 7fa692000640 1 --2- 172.21.15.90:0/1249908852 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa67806bc00 0x7fa67806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:37.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.865+0000 7fa692000640 1 -- 172.21.15.90:0/1249908852 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fa6940acec0 con 0x7fa69c15dd90 2024-03-24T01:32:37.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.865+0000 7fa699e00640 1 --2- 172.21.15.90:0/1249908852 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa67806bc00 0x7fa67806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:37.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.866+0000 7fa699e00640 1 --2- 172.21.15.90:0/1249908852 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa67806bc00 0x7fa67806e0c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa68c009650 tx=0x7fa68c006ba0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:37.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:37.870+0000 7fa692000640 1 -- 172.21.15.90:0/1249908852 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa69404d020 con 0x7fa69c15dd90 2024-03-24T01:32:38.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.064+0000 7fa69b200640 1 -- 172.21.15.90:0/1249908852 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fa69c0c7bb0 con 0x7fa69c15dd90 2024-03-24T01:32:38.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.072+0000 7fa692000640 1 -- 172.21.15.90:0/1249908852 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa69407bac0 con 0x7fa69c15dd90 2024-03-24T01:32:38.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.074+0000 7fa69b200640 1 -- 172.21.15.90:0/1249908852 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa67806bc00 msgr2=0x7fa67806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:38.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.074+0000 7fa69b200640 1 --2- 172.21.15.90:0/1249908852 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa67806bc00 0x7fa67806e0c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa68c009650 tx=0x7fa68c006ba0 comp rx=0 tx=0).stop 2024-03-24T01:32:38.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.074+0000 7fa69b200640 1 -- 172.21.15.90:0/1249908852 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa69c15dd90 msgr2=0x7fa69c0b4520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:38.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.074+0000 7fa69b200640 1 --2- 172.21.15.90:0/1249908852 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa69c15dd90 0x7fa69c0b4520 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fa694009f40 tx=0x7fa694004a40 comp rx=0 tx=0).stop 2024-03-24T01:32:38.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.075+0000 7fa69b200640 1 -- 172.21.15.90:0/1249908852 shutdown_connections 2024-03-24T01:32:38.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.075+0000 7fa69b200640 1 --2- 172.21.15.90:0/1249908852 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa67806bc00 0x7fa67806e0c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:38.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.075+0000 7fa69b200640 1 --2- 172.21.15.90:0/1249908852 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa69c15dd90 0x7fa69c0b4520 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:38.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.075+0000 7fa69b200640 1 --2- 172.21.15.90:0/1249908852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa69c0b2720 0x7fa69c0b3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:38.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.075+0000 7fa69b200640 1 --2- 172.21.15.90:0/1249908852 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa69c06cdd0 0x7fa69c0b36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:38.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.075+0000 7fa69b200640 1 -- 172.21.15.90:0/1249908852 >> 172.21.15.90:0/1249908852 conn(0x7fa69c072e20 msgr2=0x7fa69c0d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:38.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.075+0000 7fa69b200640 1 -- 172.21.15.90:0/1249908852 shutdown_connections 2024-03-24T01:32:38.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.075+0000 7fa69b200640 1 -- 172.21.15.90:0/1249908852 wait complete. 2024-03-24T01:32:38.094 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T01:32:38.094 DEBUG:teuthology.orchestra.run.smithi090:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T01:32:38.204 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T01:32:38.324 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T01:32:38.324 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:32:38.324 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:32:38.380 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053414,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498340k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249168k,nr_inodes=812292,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498340k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T01:32:38.381 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:32:38.381 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:32:38.436 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T01:32:38.437 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:32:38.437 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:32:38.493 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053414,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498340k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249168k,nr_inodes=812292,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498340k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T01:32:38.493 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:32:38.493 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:32:38.550 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T01:32:38.550 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-24T01:32:38.550 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:32:38.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.772+0000 7f4037600640 1 Processor -- start 2024-03-24T01:32:38.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.772+0000 7f4037600640 1 -- start start 2024-03-24T01:32:38.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.773+0000 7f4037600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40300cdac0 0x7f403006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:38.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.773+0000 7f4037600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f403006b600 0x7f40300ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:38.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.773+0000 7f4037600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f403015bf70 con 0x7f4030155030 2024-03-24T01:32:38.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.773+0000 7f4037600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f403006ba70 con 0x7f403006b600 2024-03-24T01:32:38.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.773+0000 7f4037600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f403015b360 con 0x7f40300cdac0 2024-03-24T01:32:38.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.774+0000 7f4034c00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4030155030 0x7f403015efc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57674/0 (socket says 172.21.15.90:57674) 2024-03-24T01:32:38.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.774+0000 7f4034c00640 1 -- 172.21.15.90:0/1667291752 learned_addr learned my addr 172.21.15.90:0/1667291752 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:32:38.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.774+0000 7f402f400640 1 -- 172.21.15.90:0/1667291752 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4023505069 0 0) 0x7f403015bf70 con 0x7f4030155030 2024-03-24T01:32:38.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.774+0000 7f4035600640 1 --2- 172.21.15.90:0/1667291752 >> v2:172.21.15.120:3300/0 conn(0x7f403006b600 0x7f40300ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:38.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.774+0000 7f402f400640 1 -- 172.21.15.90:0/1667291752 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4014003520 con 0x7f4030155030 2024-03-24T01:32:38.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.774+0000 7f402fe00640 1 --2- 172.21.15.90:0/1667291752 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40300cdac0 0x7f403006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:38.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.775+0000 7f402f400640 1 -- 172.21.15.90:0/1667291752 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1872595953 0 0) 0x7f4014003520 con 0x7f4030155030 2024-03-24T01:32:38.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.775+0000 7f402f400640 1 -- 172.21.15.90:0/1667291752 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40300cdac0 msgr2=0x7f403006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:38.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.775+0000 7f402f400640 1 --2- 172.21.15.90:0/1667291752 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40300cdac0 0x7f403006b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:38.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.775+0000 7f402f400640 1 -- 172.21.15.90:0/1667291752 >> v2:172.21.15.120:3300/0 conn(0x7f403006b600 msgr2=0x7f40300ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:38.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.775+0000 7f402f400640 1 --2- 172.21.15.90:0/1667291752 >> v2:172.21.15.120:3300/0 conn(0x7f403006b600 0x7f40300ae530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:38.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.775+0000 7f402f400640 1 -- 172.21.15.90:0/1667291752 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f403015b640 con 0x7f4030155030 2024-03-24T01:32:38.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.775+0000 7f402f400640 1 -- 172.21.15.90:0/1667291752 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f40200019f0 con 0x7f4030155030 2024-03-24T01:32:38.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.775+0000 7f402f400640 1 -- 172.21.15.90:0/1667291752 >> v1:172.21.15.90:6789/0 conn(0x7f4030155030 legacy=0x7f403015efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:38.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.775+0000 7f402f400640 1 --2- 172.21.15.90:0/1667291752 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40300cdac0 0x7f40140039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:38.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.775+0000 7f402f400640 1 --2- 172.21.15.90:0/1667291752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4014003f80 0x7f4014004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:38.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.775+0000 7f402f400640 1 --2- 172.21.15.90:0/1667291752 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4014004d40 0x7f40140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:38.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.775+0000 7f402f400640 1 -- 172.21.15.90:0/1667291752 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403006ba70 con 0x7f4014004d40 2024-03-24T01:32:38.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.775+0000 7f402f400640 1 -- 172.21.15.90:0/1667291752 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403015b360 con 0x7f4014003f80 2024-03-24T01:32:38.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.775+0000 7f402f400640 1 -- 172.21.15.90:0/1667291752 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f401400ba70 con 0x7f40300cdac0 2024-03-24T01:32:38.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.775+0000 7f4035600640 1 --2- 172.21.15.90:0/1667291752 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4014004d40 0x7f40140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:38.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.775+0000 7f4034c00640 1 --2- 172.21.15.90:0/1667291752 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40300cdac0 0x7f40140039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:38.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.775+0000 7f4035600640 1 -- 172.21.15.90:0/1667291752 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40300cdac0 msgr2=0x7f40140039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:38.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.775+0000 7f4035600640 1 --2- 172.21.15.90:0/1667291752 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40300cdac0 0x7f40140039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:38.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.775+0000 7f4035600640 1 -- 172.21.15.90:0/1667291752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4014003f80 msgr2=0x7f4014004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:38.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.776+0000 7f402fe00640 1 --2- 172.21.15.90:0/1667291752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4014003f80 0x7f4014004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:38.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.776+0000 7f4035600640 1 --2- 172.21.15.90:0/1667291752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4014003f80 0x7f4014004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:38.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.776+0000 7f4035600640 1 -- 172.21.15.90:0/1667291752 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4014003520 con 0x7f4014004d40 2024-03-24T01:32:38.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.776+0000 7f4034c00640 1 --2- 172.21.15.90:0/1667291752 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f40300cdac0 0x7f40140039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:38.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.777+0000 7f4035600640 1 --2- 172.21.15.90:0/1667291752 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4014004d40 0x7f40140051c0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f4028009ac0 tx=0x7f4014005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc26d7b543a00fbf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:38.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.777+0000 7f402f400640 1 -- 172.21.15.90:0/1667291752 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4028039070 con 0x7f4014004d40 2024-03-24T01:32:38.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.777+0000 7f402f400640 1 -- 172.21.15.90:0/1667291752 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f402802c710 con 0x7f4014004d40 2024-03-24T01:32:38.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.777+0000 7f402f400640 1 -- 172.21.15.90:0/1667291752 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4028009690 con 0x7f4014004d40 2024-03-24T01:32:38.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.777+0000 7f4037600640 1 -- 172.21.15.90:0/1667291752 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4014004d40 msgr2=0x7f40140051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:38.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.777+0000 7f4037600640 1 --2- 172.21.15.90:0/1667291752 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4014004d40 0x7f40140051c0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f4028009ac0 tx=0x7f4014005830 comp rx=0 tx=0).stop 2024-03-24T01:32:38.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.778+0000 7f4035600640 1 -- 172.21.15.90:0/1667291752 reap_dead start 2024-03-24T01:32:38.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.778+0000 7f4037600640 1 -- 172.21.15.90:0/1667291752 shutdown_connections 2024-03-24T01:32:38.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.778+0000 7f4037600640 1 -- 172.21.15.90:0/1667291752 >> 172.21.15.90:0/1667291752 conn(0x7f40300b1770 msgr2=0x7f40300b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:38.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.778+0000 7f4037600640 1 -- 172.21.15.90:0/1667291752 shutdown_connections 2024-03-24T01:32:38.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.778+0000 7f4037600640 1 -- 172.21.15.90:0/1667291752 wait complete. 2024-03-24T01:32:38.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.780+0000 7f4037600640 1 Processor -- start 2024-03-24T01:32:38.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.780+0000 7f4037600640 1 -- start start 2024-03-24T01:32:38.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.781+0000 7f4037600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4014004d40 0x7f4030150280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:38.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.781+0000 7f4037600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4030156990 0x7f403006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:38.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.781+0000 7f4037600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4030156e10 0x7f403006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:38.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.781+0000 7f4037600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4030155410 con 0x7f4030156e10 2024-03-24T01:32:38.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.781+0000 7f4037600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403015bc00 con 0x7f4014004d40 2024-03-24T01:32:38.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.781+0000 7f4037600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f403015af70 con 0x7f4030156990 2024-03-24T01:32:38.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.781+0000 7f4035600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4030156e10 0x7f403006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:38.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.781+0000 7f4035600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4030156e10 0x7f403006ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53102/0 (socket says 172.21.15.90:53102) 2024-03-24T01:32:38.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.781+0000 7f4035600640 1 -- 172.21.15.90:0/3522328991 learned_addr learned my addr 172.21.15.90:0/3522328991 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:38.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.781+0000 7f4035600640 1 -- 172.21.15.90:0/3522328991 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4030156990 msgr2=0x7f403006d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:38.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.781+0000 7f4035600640 1 --2- 172.21.15.90:0/3522328991 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4030156990 0x7f403006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:38.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.781+0000 7f4035600640 1 -- 172.21.15.90:0/3522328991 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4014004d40 msgr2=0x7f4030150280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:38.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.781+0000 7f4035600640 1 --2- 172.21.15.90:0/3522328991 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4014004d40 0x7f4030150280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:38.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.781+0000 7f4035600640 1 -- 172.21.15.90:0/3522328991 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40301f7c80 con 0x7f4030156e10 2024-03-24T01:32:38.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.782+0000 7f4035600640 1 --2- 172.21.15.90:0/3522328991 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4030156e10 0x7f403006ded0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f4028009ac0 tx=0x7f4028004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:38.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.782+0000 7f402cc00640 1 -- 172.21.15.90:0/3522328991 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4028039070 con 0x7f4030156e10 2024-03-24T01:32:38.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.782+0000 7f402cc00640 1 -- 172.21.15.90:0/3522328991 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4028043070 con 0x7f4030156e10 2024-03-24T01:32:38.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.782+0000 7f402cc00640 1 -- 172.21.15.90:0/3522328991 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4028009690 con 0x7f4030156e10 2024-03-24T01:32:38.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.782+0000 7f4037600640 1 -- 172.21.15.90:0/3522328991 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40300c8d40 con 0x7f4030156e10 2024-03-24T01:32:38.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.782+0000 7f4037600640 1 -- 172.21.15.90:0/3522328991 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40300c9300 con 0x7f4030156e10 2024-03-24T01:32:38.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.783+0000 7f4037600640 1 -- 172.21.15.90:0/3522328991 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4030068060 con 0x7f4030156e10 2024-03-24T01:32:38.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.784+0000 7f402cc00640 1 -- 172.21.15.90:0/3522328991 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4028003850 con 0x7f4030156e10 2024-03-24T01:32:38.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.785+0000 7f402cc00640 1 --2- 172.21.15.90:0/3522328991 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f401006bb70 0x7f401006e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:38.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.785+0000 7f402cc00640 1 -- 172.21.15.90:0/3522328991 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f40280abab0 con 0x7f4030156e10 2024-03-24T01:32:38.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.786+0000 7f4034c00640 1 --2- 172.21.15.90:0/3522328991 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f401006bb70 0x7f401006e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:38.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.787+0000 7f4034c00640 1 --2- 172.21.15.90:0/3522328991 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f401006bb70 0x7f401006e030 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4020001b60 tx=0x7f4020009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:38.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:38.789+0000 7f402cc00640 1 -- 172.21.15.90:0/3522328991 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f402804c020 con 0x7f4030156e10 2024-03-24T01:32:39.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.024+0000 7f4037600640 1 -- 172.21.15.90:0/3522328991 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4030153c90 con 0x7f4030156e10 2024-03-24T01:32:39.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.025+0000 7f402cc00640 1 -- 172.21.15.90:0/3522328991 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f402807a6b0 con 0x7f4030156e10 2024-03-24T01:32:39.025 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:32:39.025 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":43,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4985,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6834/3475131973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":3475131973},{"type":"v1","addr":"172.21.15.90:6834","nonce":3475131973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":4994,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/3151604088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6835","nonce":3151604088},{"type":"v1","addr":"172.21.15.120:6837","nonce":3151604088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5428,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6836/1239583677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1239583677},{"type":"v1","addr":"172.21.15.120:6836","nonce":1239583677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":7014,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/1533808019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":1533808019},{"type":"v1","addr":"172.21.15.90:6835","nonce":1533808019}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[]} 2024-03-24T01:32:39.026 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 43 2024-03-24T01:32:39.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.027+0000 7f4037600640 1 -- 172.21.15.90:0/3522328991 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f401006bb70 msgr2=0x7f401006e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:39.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.027+0000 7f4037600640 1 --2- 172.21.15.90:0/3522328991 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f401006bb70 0x7f401006e030 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4020001b60 tx=0x7f4020009040 comp rx=0 tx=0).stop 2024-03-24T01:32:39.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.027+0000 7f4037600640 1 -- 172.21.15.90:0/3522328991 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4030156e10 msgr2=0x7f403006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:39.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.027+0000 7f4037600640 1 --2- 172.21.15.90:0/3522328991 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4030156e10 0x7f403006ded0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f4028009ac0 tx=0x7f4028004730 comp rx=0 tx=0).stop 2024-03-24T01:32:39.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.028+0000 7f4037600640 1 -- 172.21.15.90:0/3522328991 shutdown_connections 2024-03-24T01:32:39.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.028+0000 7f4037600640 1 --2- 172.21.15.90:0/3522328991 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f401006bb70 0x7f401006e030 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:39.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.028+0000 7f4037600640 1 --2- 172.21.15.90:0/3522328991 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4030156e10 0x7f403006ded0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:39.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.028+0000 7f4037600640 1 --2- 172.21.15.90:0/3522328991 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4030156990 0x7f403006d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:39.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.028+0000 7f4037600640 1 --2- 172.21.15.90:0/3522328991 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4014004d40 0x7f4030150280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:39.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.028+0000 7f4037600640 1 -- 172.21.15.90:0/3522328991 >> 172.21.15.90:0/3522328991 conn(0x7f40300b1770 msgr2=0x7f4030153580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:39.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.028+0000 7f4037600640 1 -- 172.21.15.90:0/3522328991 shutdown_connections 2024-03-24T01:32:39.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.028+0000 7f4037600640 1 -- 172.21.15.90:0/3522328991 wait complete. 2024-03-24T01:32:39.047 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-24T01:32:39.047 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-24T01:32:39.047 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-24T01:32:39.047 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:32:39.047 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-24T01:32:39.048 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-24T01:32:39.048 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-24T01:32:39.048 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:32:39.048 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-24T01:32:39.048 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-24T01:32:39.048 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-24T01:32:39.048 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:32:39.048 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-24T01:32:39.049 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-24T01:32:39.049 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-24T01:32:39.049 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:32:39.063 INFO:tasks.ceph.mds.a:Stopped 2024-03-24T01:32:39.063 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-24T01:32:39.066 INFO:tasks.ceph.mds.c:Stopped 2024-03-24T01:32:39.066 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-24T01:32:39.078 INFO:tasks.ceph.mds.b:Stopped 2024-03-24T01:32:39.078 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-24T01:32:39.081 INFO:tasks.ceph.mds.d:Stopped 2024-03-24T01:32:39.081 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-24T01:32:39.113 INFO:tasks.ceph.mds.a:Started 2024-03-24T01:32:39.114 DEBUG:teuthology.parallel:result is None 2024-03-24T01:32:39.120 INFO:tasks.ceph.mds.c:Started 2024-03-24T01:32:39.120 DEBUG:teuthology.parallel:result is None 2024-03-24T01:32:39.128 INFO:tasks.ceph.mds.b:Started 2024-03-24T01:32:39.128 DEBUG:teuthology.parallel:result is None 2024-03-24T01:32:39.134 INFO:tasks.ceph.mds.d:Started 2024-03-24T01:32:39.135 DEBUG:teuthology.parallel:result is None 2024-03-24T01:32:39.135 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-24T01:32:39.263 INFO:tasks.ceph.mds.c.smithi090.stdout:starting mds.c at 2024-03-24T01:32:39.264 INFO:tasks.ceph.mds.a.smithi090.stdout:starting mds.a at 2024-03-24T01:32:39.287 INFO:tasks.ceph.mds.d.smithi120.stdout:starting mds.d at 2024-03-24T01:32:39.289 INFO:tasks.ceph.mds.b.smithi120.stdout:starting mds.b at 2024-03-24T01:32:39.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.306+0000 7f13d8800640 1 Processor -- start 2024-03-24T01:32:39.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.307+0000 7f13d8800640 1 -- start start 2024-03-24T01:32:39.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.308+0000 7f13d8800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f13d015dc60 0x7f13d00b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:39.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.308+0000 7f13d8800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13d006bc60 0x7f13d00b0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:39.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.308+0000 7f13d8800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d00b5950 con 0x7f13d015e040 2024-03-24T01:32:39.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.308+0000 7f13d8800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13d00b38b0 con 0x7f13d015dc60 2024-03-24T01:32:39.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.308+0000 7f13d8800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13d00b3e70 con 0x7f13d006bc60 2024-03-24T01:32:39.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.308+0000 7f13d5e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f13d015dc60 0x7f13d00b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:39.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.308+0000 7f13d6800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13d006bc60 0x7f13d00b0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:39.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.308+0000 7f13d5400640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f13d015e040 0x7f13d00ad820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.90:37688/0 (socket says 172.21.15.90:37688) 2024-03-24T01:32:39.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.308+0000 7f13d5e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f13d015dc60 0x7f13d00b1f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.90:39112/0 (socket says 172.21.15.90:39112) 2024-03-24T01:32:39.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.308+0000 7f13d5400640 1 -- 172.21.15.90:0/1741922167 learned_addr learned my addr 172.21.15.90:0/1741922167 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:32:39.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.309+0000 7f13d5e00640 1 -- 172.21.15.90:0/1741922167 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13d006bc60 msgr2=0x7f13d00b0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:39.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.309+0000 7f13d5e00640 1 --2- 172.21.15.90:0/1741922167 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13d006bc60 0x7f13d00b0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:39.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.309+0000 7f13d5e00640 1 -- 172.21.15.90:0/1741922167 >> v1:172.21.15.120:6789/0 conn(0x7f13d015e040 legacy=0x7f13d00ad820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:39.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.309+0000 7f13d5e00640 1 -- 172.21.15.90:0/1741922167 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d00b5c30 con 0x7f13d015dc60 2024-03-24T01:32:39.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.309+0000 7f13d4a00640 1 -- 172.21.15.90:0/1741922167 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1460819323 0 0) 0x7f13d00b5950 con 0x7f13d015e040 2024-03-24T01:32:39.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.310+0000 7f13d5e00640 1 --2- 172.21.15.90:0/1741922167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13d015dc60 0x7f13d00b1f60 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f13c40097a0 tx=0x7f13c402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73bd02fead2cd807 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:39.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.310+0000 7f13d4a00640 1 -- 172.21.15.90:0/1741922167 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c40315a0 con 0x7f13d015dc60 2024-03-24T01:32:39.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.310+0000 7f13d4a00640 1 -- 172.21.15.90:0/1741922167 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c4037070 con 0x7f13d015dc60 2024-03-24T01:32:39.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.310+0000 7f13d4a00640 1 -- 172.21.15.90:0/1741922167 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c4031960 con 0x7f13d015dc60 2024-03-24T01:32:39.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.310+0000 7f13d8800640 1 -- 172.21.15.90:0/1741922167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13d015dc60 msgr2=0x7f13d00b1f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:39.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.310+0000 7f13d8800640 1 --2- 172.21.15.90:0/1741922167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13d015dc60 0x7f13d00b1f60 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f13c40097a0 tx=0x7f13c402b840 comp rx=0 tx=0).stop 2024-03-24T01:32:39.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.311+0000 7f13d8800640 1 -- 172.21.15.90:0/1741922167 shutdown_connections 2024-03-24T01:32:39.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.311+0000 7f13d8800640 1 --2- 172.21.15.90:0/1741922167 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13d006bc60 0x7f13d00b0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:39.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.311+0000 7f13d8800640 1 --2- 172.21.15.90:0/1741922167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13d015dc60 0x7f13d00b1f60 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:39.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.311+0000 7f13d8800640 1 -- 172.21.15.90:0/1741922167 >> 172.21.15.90:0/1741922167 conn(0x7f13d006cec0 msgr2=0x7f13d0099a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:39.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.311+0000 7f13d8800640 1 -- 172.21.15.90:0/1741922167 shutdown_connections 2024-03-24T01:32:39.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.311+0000 7f13d8800640 1 -- 172.21.15.90:0/1741922167 wait complete. 2024-03-24T01:32:39.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.312+0000 7f13d8800640 1 Processor -- start 2024-03-24T01:32:39.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.312+0000 7f13d8800640 1 -- start start 2024-03-24T01:32:39.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.313+0000 7f13d8800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13d006bc60 0x7f13d0160880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:39.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.313+0000 7f13d8800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f13d015dc60 0x7f13d01f4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:39.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.313+0000 7f13d8800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13d015e040 0x7f13d015c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:39.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.313+0000 7f13d8800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d00b5730 con 0x7f13d015dc60 2024-03-24T01:32:39.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.313+0000 7f13d8800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d00b55b0 con 0x7f13d015e040 2024-03-24T01:32:39.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.313+0000 7f13d8800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13d00adf30 con 0x7f13d006bc60 2024-03-24T01:32:39.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.313+0000 7f13d5400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f13d015dc60 0x7f13d01f4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:39.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.313+0000 7f13d5400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f13d015dc60 0x7f13d01f4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53130/0 (socket says 172.21.15.90:53130) 2024-03-24T01:32:39.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.313+0000 7f13d5400640 1 -- 172.21.15.90:0/1409245281 learned_addr learned my addr 172.21.15.90:0/1409245281 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:39.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.314+0000 7f13d5e00640 1 --2- 172.21.15.90:0/1409245281 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13d006bc60 0x7f13d0160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:39.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.314+0000 7f13d6800640 1 --2- 172.21.15.90:0/1409245281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13d015e040 0x7f13d015c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:39.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.314+0000 7f13d5400640 1 -- 172.21.15.90:0/1409245281 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13d006bc60 msgr2=0x7f13d0160880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:39.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.314+0000 7f13d5400640 1 --2- 172.21.15.90:0/1409245281 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13d006bc60 0x7f13d0160880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:39.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.314+0000 7f13d5400640 1 -- 172.21.15.90:0/1409245281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13d015e040 msgr2=0x7f13d015c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:39.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.314+0000 7f13d5400640 1 --2- 172.21.15.90:0/1409245281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13d015e040 0x7f13d015c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:39.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.314+0000 7f13d5400640 1 -- 172.21.15.90:0/1409245281 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d015cf00 con 0x7f13d015dc60 2024-03-24T01:32:39.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.314+0000 7f13d5e00640 1 --2- 172.21.15.90:0/1409245281 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13d006bc60 0x7f13d0160880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:32:39.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.314+0000 7f13d5400640 1 --2- 172.21.15.90:0/1409245281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f13d015dc60 0x7f13d01f4810 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f13c000c490 tx=0x7f13c000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:39.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.314+0000 7f13ca000640 1 -- 172.21.15.90:0/1409245281 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c0005ce0 con 0x7f13d015dc60 2024-03-24T01:32:39.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.314+0000 7f13ca000640 1 -- 172.21.15.90:0/1409245281 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c0007070 con 0x7f13d015dc60 2024-03-24T01:32:39.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.314+0000 7f13ca000640 1 -- 172.21.15.90:0/1409245281 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c0007460 con 0x7f13d015dc60 2024-03-24T01:32:39.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.314+0000 7f13d8800640 1 -- 172.21.15.90:0/1409245281 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13d0159ff0 con 0x7f13d015dc60 2024-03-24T01:32:39.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.315+0000 7f13d8800640 1 -- 172.21.15.90:0/1409245281 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13d015a830 con 0x7f13d015dc60 2024-03-24T01:32:39.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.316+0000 7f13d8800640 1 -- 172.21.15.90:0/1409245281 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13d015aa30 con 0x7f13d015dc60 2024-03-24T01:32:39.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.316+0000 7f13ca000640 1 -- 172.21.15.90:0/1409245281 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f13c0007600 con 0x7f13d015dc60 2024-03-24T01:32:39.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.317+0000 7f13ca000640 1 --2- 172.21.15.90:0/1409245281 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f13b406bbd0 0x7f13b406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:39.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.317+0000 7f13ca000640 1 -- 172.21.15.90:0/1409245281 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f13c008a5c0 con 0x7f13d015dc60 2024-03-24T01:32:39.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.318+0000 7f13d5e00640 1 --2- 172.21.15.90:0/1409245281 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f13b406bbd0 0x7f13b406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:39.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.319+0000 7f13d5e00640 1 --2- 172.21.15.90:0/1409245281 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f13b406bbd0 0x7f13b406e090 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f13c4036000 tx=0x7f13c403d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:39.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.322+0000 7f13ca000640 1 -- 172.21.15.90:0/1409245281 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f13d015aa30 con 0x7f13d015dc60 2024-03-24T01:32:39.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.530+0000 7f13d8800640 1 -- 172.21.15.90:0/1409245281 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f13d00732b0 con 0x7f13d015dc60 2024-03-24T01:32:39.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.657+0000 7f13ca000640 1 -- 172.21.15.90:0/1409245281 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v61) v1 ==== 97+0+0 (secure 0 0 0) 0x7f13d0159ff0 con 0x7f13d015dc60 2024-03-24T01:32:39.657 INFO:teuthology.orchestra.run.smithi090.stderr: removed all blocklist entries 2024-03-24T01:32:39.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.661+0000 7f13d8800640 1 -- 172.21.15.90:0/1409245281 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f13b406bbd0 msgr2=0x7f13b406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:39.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.661+0000 7f13d8800640 1 --2- 172.21.15.90:0/1409245281 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f13b406bbd0 0x7f13b406e090 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f13c4036000 tx=0x7f13c403d040 comp rx=0 tx=0).stop 2024-03-24T01:32:39.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.661+0000 7f13d8800640 1 -- 172.21.15.90:0/1409245281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f13d015dc60 msgr2=0x7f13d01f4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:39.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.661+0000 7f13d8800640 1 --2- 172.21.15.90:0/1409245281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f13d015dc60 0x7f13d01f4810 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f13c000c490 tx=0x7f13c000c950 comp rx=0 tx=0).stop 2024-03-24T01:32:39.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.661+0000 7f13d8800640 1 -- 172.21.15.90:0/1409245281 shutdown_connections 2024-03-24T01:32:39.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.661+0000 7f13d8800640 1 --2- 172.21.15.90:0/1409245281 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f13b406bbd0 0x7f13b406e090 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:39.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.661+0000 7f13d8800640 1 --2- 172.21.15.90:0/1409245281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f13d015dc60 0x7f13d01f4810 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:39.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.661+0000 7f13d8800640 1 --2- 172.21.15.90:0/1409245281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13d015e040 0x7f13d015c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:39.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.661+0000 7f13d8800640 1 --2- 172.21.15.90:0/1409245281 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f13d006bc60 0x7f13d0160880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:39.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.661+0000 7f13d8800640 1 -- 172.21.15.90:0/1409245281 >> 172.21.15.90:0/1409245281 conn(0x7f13d006cec0 msgr2=0x7f13d00b16e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:39.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.661+0000 7f13d8800640 1 -- 172.21.15.90:0/1409245281 shutdown_connections 2024-03-24T01:32:39.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.661+0000 7f13d8800640 1 -- 172.21.15.90:0/1409245281 wait complete. 2024-03-24T01:32:39.685 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-24T01:32:39.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.891+0000 7fab84e00640 1 Processor -- start 2024-03-24T01:32:39.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.891+0000 7fab84e00640 1 -- start start 2024-03-24T01:32:39.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.893+0000 7fab84e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fab801603e0 0x7fab8006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:39.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.893+0000 7fab84e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab80145430 0x7fab8006b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:39.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.893+0000 7fab84e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab800cd490 con 0x7fab800cc150 2024-03-24T01:32:39.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.893+0000 7fab84e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab800cd270 con 0x7fab801603e0 2024-03-24T01:32:39.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.893+0000 7fab84e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab8015f900 con 0x7fab80145430 2024-03-24T01:32:39.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.893+0000 7fab7de00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fab800cc150 0x7fab800ccb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40192/0 (socket says 172.21.15.90:40192) 2024-03-24T01:32:39.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.893+0000 7fab7de00640 1 -- 172.21.15.90:0/1933672301 learned_addr learned my addr 172.21.15.90:0/1933672301 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:32:39.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.893+0000 7fab7d400640 1 -- 172.21.15.90:0/1933672301 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2801483604 0 0) 0x7fab800cd490 con 0x7fab800cc150 2024-03-24T01:32:39.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.893+0000 7fab7d400640 1 -- 172.21.15.90:0/1933672301 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab64003520 con 0x7fab800cc150 2024-03-24T01:32:39.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.893+0000 7fab77e00640 1 --2- 172.21.15.90:0/1933672301 >> v2:172.21.15.120:3300/0 conn(0x7fab801603e0 0x7fab8006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:39.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.894+0000 7fab7e800640 1 --2- 172.21.15.90:0/1933672301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab80145430 0x7fab8006b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:39.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.894+0000 7fab7d400640 1 -- 172.21.15.90:0/1933672301 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3667902919 0 0) 0x7fab64003520 con 0x7fab800cc150 2024-03-24T01:32:39.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.894+0000 7fab7d400640 1 -- 172.21.15.90:0/1933672301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab80145430 msgr2=0x7fab8006b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:39.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.894+0000 7fab7d400640 1 --2- 172.21.15.90:0/1933672301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab80145430 0x7fab8006b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:39.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.894+0000 7fab7d400640 1 -- 172.21.15.90:0/1933672301 >> v2:172.21.15.120:3300/0 conn(0x7fab801603e0 msgr2=0x7fab8006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:39.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.894+0000 7fab7d400640 1 --2- 172.21.15.90:0/1933672301 >> v2:172.21.15.120:3300/0 conn(0x7fab801603e0 0x7fab8006b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:39.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.894+0000 7fab7d400640 1 -- 172.21.15.90:0/1933672301 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab8015fbe0 con 0x7fab800cc150 2024-03-24T01:32:39.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.894+0000 7fab7d400640 1 -- 172.21.15.90:0/1933672301 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7fab70001670 con 0x7fab800cc150 2024-03-24T01:32:39.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.894+0000 7fab7d400640 1 -- 172.21.15.90:0/1933672301 >> v1:172.21.15.90:6789/0 conn(0x7fab800cc150 legacy=0x7fab800ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:39.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.894+0000 7fab7d400640 1 --2- 172.21.15.90:0/1933672301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab640039b0 0x7fab64003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:39.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.894+0000 7fab7d400640 1 --2- 172.21.15.90:0/1933672301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab80145430 0x7fab64004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:39.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.894+0000 7fab7d400640 1 --2- 172.21.15.90:0/1933672301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab64004d40 0x7fab640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:39.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.894+0000 7fab7d400640 1 -- 172.21.15.90:0/1933672301 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab800cd270 con 0x7fab64004d40 2024-03-24T01:32:39.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.894+0000 7fab7d400640 1 -- 172.21.15.90:0/1933672301 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8015f900 con 0x7fab640039b0 2024-03-24T01:32:39.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.894+0000 7fab7d400640 1 -- 172.21.15.90:0/1933672301 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab6400baf0 con 0x7fab80145430 2024-03-24T01:32:39.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.894+0000 7fab7e800640 1 --2- 172.21.15.90:0/1933672301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab64004d40 0x7fab640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:39.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.895+0000 7fab77e00640 1 --2- 172.21.15.90:0/1933672301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab80145430 0x7fab64004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:39.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.895+0000 7fab7e800640 1 -- 172.21.15.90:0/1933672301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab80145430 msgr2=0x7fab64004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:39.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.895+0000 7fab7de00640 1 --2- 172.21.15.90:0/1933672301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab640039b0 0x7fab64003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:39.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.895+0000 7fab7e800640 1 --2- 172.21.15.90:0/1933672301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab80145430 0x7fab64004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:39.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.895+0000 7fab7e800640 1 -- 172.21.15.90:0/1933672301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab640039b0 msgr2=0x7fab64003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:39.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.895+0000 7fab7e800640 1 --2- 172.21.15.90:0/1933672301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab640039b0 0x7fab64003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:39.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.895+0000 7fab7e800640 1 -- 172.21.15.90:0/1933672301 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab64003520 con 0x7fab64004d40 2024-03-24T01:32:39.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.895+0000 7fab77e00640 1 --2- 172.21.15.90:0/1933672301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab80145430 0x7fab64004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:39.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.896+0000 7fab7e800640 1 --2- 172.21.15.90:0/1933672301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab64004d40 0x7fab640051c0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fab78007360 tx=0x7fab64005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c1b8627f59d071c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:39.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.896+0000 7fab7d400640 1 -- 172.21.15.90:0/1933672301 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab78039070 con 0x7fab64004d40 2024-03-24T01:32:39.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.896+0000 7fab7d400640 1 -- 172.21.15.90:0/1933672301 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab7802ccb0 con 0x7fab64004d40 2024-03-24T01:32:39.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.896+0000 7fab7d400640 1 -- 172.21.15.90:0/1933672301 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab78009be0 con 0x7fab64004d40 2024-03-24T01:32:39.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.896+0000 7fab84e00640 1 -- 172.21.15.90:0/1933672301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab64004d40 msgr2=0x7fab640051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:39.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.896+0000 7fab84e00640 1 --2- 172.21.15.90:0/1933672301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab64004d40 0x7fab640051c0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fab78007360 tx=0x7fab64005830 comp rx=0 tx=0).stop 2024-03-24T01:32:39.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.896+0000 7fab7e800640 1 -- 172.21.15.90:0/1933672301 reap_dead start 2024-03-24T01:32:39.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.897+0000 7fab84e00640 1 -- 172.21.15.90:0/1933672301 shutdown_connections 2024-03-24T01:32:39.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.897+0000 7fab84e00640 1 -- 172.21.15.90:0/1933672301 >> 172.21.15.90:0/1933672301 conn(0x7fab800b1770 msgr2=0x7fab800b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:39.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.897+0000 7fab84e00640 1 -- 172.21.15.90:0/1933672301 shutdown_connections 2024-03-24T01:32:39.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.897+0000 7fab84e00640 1 -- 172.21.15.90:0/1933672301 wait complete. 2024-03-24T01:32:39.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.898+0000 7fab84e00640 1 Processor -- start 2024-03-24T01:32:39.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.899+0000 7fab84e00640 1 -- start start 2024-03-24T01:32:39.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.899+0000 7fab84e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab64004d40 0x7fab8013c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:39.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.900+0000 7fab84e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab80143120 0x7fab801435a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:39.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.900+0000 7fab84e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab8015c580 0x7fab8015ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:39.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.900+0000 7fab84e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8006b780 con 0x7fab8015c580 2024-03-24T01:32:39.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.900+0000 7fab84e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8006b600 con 0x7fab80143120 2024-03-24T01:32:39.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.900+0000 7fab84e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab8006b900 con 0x7fab64004d40 2024-03-24T01:32:39.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.900+0000 7fab7e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab8015c580 0x7fab8015ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:39.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.900+0000 7fab7e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab8015c580 0x7fab8015ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39558/0 (socket says 172.21.15.90:39558) 2024-03-24T01:32:39.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.900+0000 7fab7e800640 1 -- 172.21.15.90:0/3088600672 learned_addr learned my addr 172.21.15.90:0/3088600672 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:39.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.900+0000 7fab7de00640 1 --2- 172.21.15.90:0/3088600672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab64004d40 0x7fab8013c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:39.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.900+0000 7fab77e00640 1 --2- 172.21.15.90:0/3088600672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab80143120 0x7fab801435a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:39.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.900+0000 7fab7e800640 1 -- 172.21.15.90:0/3088600672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab64004d40 msgr2=0x7fab8013c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:39.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.900+0000 7fab7e800640 1 --2- 172.21.15.90:0/3088600672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab64004d40 0x7fab8013c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:39.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.900+0000 7fab7e800640 1 -- 172.21.15.90:0/3088600672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab80143120 msgr2=0x7fab801435a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:39.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.900+0000 7fab7e800640 1 --2- 172.21.15.90:0/3088600672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab80143120 0x7fab801435a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:39.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.900+0000 7fab7e800640 1 -- 172.21.15.90:0/3088600672 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab80143ae0 con 0x7fab8015c580 2024-03-24T01:32:39.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.900+0000 7fab77e00640 1 --2- 172.21.15.90:0/3088600672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab80143120 0x7fab801435a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:39.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.901+0000 7fab7e800640 1 --2- 172.21.15.90:0/3088600672 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab8015c580 0x7fab8015ca00 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fab78037040 tx=0x7fab78004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:39.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.901+0000 7fab76000640 1 -- 172.21.15.90:0/3088600672 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab78039030 con 0x7fab8015c580 2024-03-24T01:32:39.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.901+0000 7fab76000640 1 -- 172.21.15.90:0/3088600672 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab78043070 con 0x7fab8015c580 2024-03-24T01:32:39.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.901+0000 7fab84e00640 1 -- 172.21.15.90:0/3088600672 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab8015cf40 con 0x7fab8015c580 2024-03-24T01:32:39.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.901+0000 7fab76000640 1 -- 172.21.15.90:0/3088600672 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7802ce60 con 0x7fab8015c580 2024-03-24T01:32:39.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.901+0000 7fab84e00640 1 -- 172.21.15.90:0/3088600672 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab8015d4e0 con 0x7fab8015c580 2024-03-24T01:32:39.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.902+0000 7fab84e00640 1 -- 172.21.15.90:0/3088600672 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab80068060 con 0x7fab8015c580 2024-03-24T01:32:39.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.903+0000 7fab76000640 1 -- 172.21.15.90:0/3088600672 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fab7803d7f0 con 0x7fab8015c580 2024-03-24T01:32:39.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.904+0000 7fab76000640 1 --2- 172.21.15.90:0/3088600672 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fab5c06bbd0 0x7fab5c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:39.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.904+0000 7fab76000640 1 -- 172.21.15.90:0/3088600672 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fab780ac940 con 0x7fab8015c580 2024-03-24T01:32:39.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.904+0000 7fab7de00640 1 --2- 172.21.15.90:0/3088600672 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fab5c06bbd0 0x7fab5c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:39.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.906+0000 7fab7de00640 1 --2- 172.21.15.90:0/3088600672 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fab5c06bbd0 0x7fab5c06e090 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fab70005ab0 tx=0x7fab70009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:39.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:39.908+0000 7fab76000640 1 -- 172.21.15.90:0/3088600672 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fab7807b550 con 0x7fab8015c580 2024-03-24T01:32:40.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.183+0000 7fab84e00640 1 -- 172.21.15.90:0/3088600672 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fab800b3220 con 0x7fab8015c580 2024-03-24T01:32:40.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.184+0000 7fab76000640 1 -- 172.21.15.90:0/3088600672 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fab7807ae50 con 0x7fab8015c580 2024-03-24T01:32:40.185 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:32:40.185 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-24T01:32:40.185 INFO:teuthology.orchestra.run.smithi090.stdout: "auth_dump": [ 2024-03-24T01:32:40.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:32:40.185 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mds.a", 2024-03-24T01:32:40.185 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQBzgP9lEcWmMBAAtUqishOFkZW8T4Hh1Cgj2A==", 2024-03-24T01:32:40.185 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:32:40.185 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-24T01:32:40.185 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow *", 2024-03-24T01:32:40.185 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow *", 2024-03-24T01:32:40.185 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:32:40.185 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:32:40.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:32:40.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:32:40.186 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mds.b", 2024-03-24T01:32:40.186 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQB0gP9lIPZ1FRAAmeGByOKmL/VQTpWm4Y2/Tg==", 2024-03-24T01:32:40.186 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:32:40.186 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-24T01:32:40.186 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow *", 2024-03-24T01:32:40.186 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow *", 2024-03-24T01:32:40.186 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:32:40.186 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:32:40.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:32:40.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:32:40.186 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mds.c", 2024-03-24T01:32:40.186 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQB0gP9lRuOuBBAA3/3PMDba2Tlc6BqaY/499g==", 2024-03-24T01:32:40.186 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:32:40.187 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-24T01:32:40.187 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow *", 2024-03-24T01:32:40.187 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow *", 2024-03-24T01:32:40.187 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:32:40.187 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:32:40.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:32:40.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:32:40.187 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mds.d", 2024-03-24T01:32:40.187 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQB0gP9lOtsLJhAAYUVPFa1eoDHF5MV42s8e6A==", 2024-03-24T01:32:40.187 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:32:40.187 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-24T01:32:40.187 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow *", 2024-03-24T01:32:40.187 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow *", 2024-03-24T01:32:40.187 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:32:40.188 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:32:40.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:32:40.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:32:40.188 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.0", 2024-03-24T01:32:40.188 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQB5gP9lJ2jHGRAAPlZjS638Xdt6yVc+vT2ghw==", 2024-03-24T01:32:40.188 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:32:40.188 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-24T01:32:40.188 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-24T01:32:40.188 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:32:40.188 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:32:40.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:32:40.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:32:40.188 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.1", 2024-03-24T01:32:40.188 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQB8gP9lEN/pIxAAvzVfJNpo7FP6c8KSbZdy9A==", 2024-03-24T01:32:40.188 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:32:40.189 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-24T01:32:40.189 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-24T01:32:40.189 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:32:40.189 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:32:40.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:32:40.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:32:40.189 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.2", 2024-03-24T01:32:40.189 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQB/gP9lKv5mLRAAJXArmoRRVqO07jxWutfMbQ==", 2024-03-24T01:32:40.189 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:32:40.189 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-24T01:32:40.189 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-24T01:32:40.189 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:32:40.189 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:32:40.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:32:40.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:32:40.190 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.3", 2024-03-24T01:32:40.190 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQCCgP9lTZ8YNxAA375ZLJHv6H2CJghmcqfe+A==", 2024-03-24T01:32:40.190 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:32:40.190 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-24T01:32:40.190 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-24T01:32:40.190 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:32:40.190 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:32:40.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:32:40.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:32:40.190 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.4", 2024-03-24T01:32:40.190 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQCKgP9ln1YuBBAAYCNFJGs3fApnhwnY/VYHhw==", 2024-03-24T01:32:40.190 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:32:40.190 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-24T01:32:40.190 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-24T01:32:40.191 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:32:40.191 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:32:40.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:32:40.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:32:40.191 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.5", 2024-03-24T01:32:40.191 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQCNgP9l329zHBAA7VtZVd0XKEK5dzhyzcPFjQ==", 2024-03-24T01:32:40.191 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:32:40.191 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-24T01:32:40.191 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-24T01:32:40.191 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:32:40.191 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:32:40.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:32:40.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:32:40.191 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.6", 2024-03-24T01:32:40.192 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQCQgP9lIv32NBAAZN6sDvBDKUZGB+TnYHO9jA==", 2024-03-24T01:32:40.192 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:32:40.192 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-24T01:32:40.192 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-24T01:32:40.192 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:32:40.192 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:32:40.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:32:40.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:32:40.192 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.7", 2024-03-24T01:32:40.192 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQCUgP9lFpBTBBAAbPF02et1cVIhSfkvJz3R3w==", 2024-03-24T01:32:40.192 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:32:40.192 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-24T01:32:40.192 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-24T01:32:40.192 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:32:40.192 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:32:40.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:32:40.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:32:40.193 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.0", 2024-03-24T01:32:40.193 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQB0gP9lLKWXMxAAwrrRG8zLSylpIEuUgM7q8g==", 2024-03-24T01:32:40.193 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:32:40.193 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-24T01:32:40.193 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow r", 2024-03-24T01:32:40.193 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-24T01:32:40.193 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:32:40.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:32:40.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:32:40.193 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.1", 2024-03-24T01:32:40.193 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQB1gP9lFlgvDRAAJFhGpBldAqv7oKFr2lIXNQ==", 2024-03-24T01:32:40.193 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:32:40.193 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-24T01:32:40.193 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow r", 2024-03-24T01:32:40.193 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow rw", 2024-03-24T01:32:40.194 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow rwx" 2024-03-24T01:32:40.194 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:32:40.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:32:40.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:32:40.194 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.2", 2024-03-24T01:32:40.194 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQB1gP9lsGI+AhAAPLfzrMVwQpf8iGAfVJX1gA==", 2024-03-24T01:32:40.194 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:32:40.194 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-24T01:32:40.194 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow r", 2024-03-24T01:32:40.194 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow rw", 2024-03-24T01:32:40.194 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow rwx" 2024-03-24T01:32:40.194 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:32:40.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:32:40.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:32:40.195 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.3", 2024-03-24T01:32:40.195 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQB1gP9l1LA0GBAAp6dwwZf+25F64cERHm2/Ew==", 2024-03-24T01:32:40.195 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:32:40.195 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-24T01:32:40.195 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow r", 2024-03-24T01:32:40.195 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow rw", 2024-03-24T01:32:40.195 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow rwx" 2024-03-24T01:32:40.195 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:32:40.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:32:40.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:32:40.195 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.admin", 2024-03-24T01:32:40.195 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQBygP9lwVRtIxAAJ7NWfOKFeLoW5nl0TLtR2w==", 2024-03-24T01:32:40.195 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:32:40.195 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow *", 2024-03-24T01:32:40.195 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow *", 2024-03-24T01:32:40.196 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow *", 2024-03-24T01:32:40.196 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:32:40.196 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:32:40.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:32:40.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:32:40.196 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mgr.x", 2024-03-24T01:32:40.196 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQBzgP9l8GPtJBAAgU+pArQqmyMDPwj0w+W/Dg==", 2024-03-24T01:32:40.196 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:32:40.196 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow *", 2024-03-24T01:32:40.196 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile mgr", 2024-03-24T01:32:40.196 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:32:40.196 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:32:40.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:32:40.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:32:40.196 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mgr.y", 2024-03-24T01:32:40.197 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQBzgP9l0KzyGBAA9zVfyPbngjHtBQyp+S37iQ==", 2024-03-24T01:32:40.197 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-24T01:32:40.197 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow *", 2024-03-24T01:32:40.197 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile mgr", 2024-03-24T01:32:40.197 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-24T01:32:40.197 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:32:40.197 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:32:40.197 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-03-24T01:32:40.197 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-24T01:32:40.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.187+0000 7fab84e00640 1 -- 172.21.15.90:0/3088600672 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fab5c06bbd0 msgr2=0x7fab5c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:40.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.187+0000 7fab84e00640 1 --2- 172.21.15.90:0/3088600672 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fab5c06bbd0 0x7fab5c06e090 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fab70005ab0 tx=0x7fab70009040 comp rx=0 tx=0).stop 2024-03-24T01:32:40.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.187+0000 7fab84e00640 1 -- 172.21.15.90:0/3088600672 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab8015c580 msgr2=0x7fab8015ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:40.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.187+0000 7fab84e00640 1 --2- 172.21.15.90:0/3088600672 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab8015c580 0x7fab8015ca00 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fab78037040 tx=0x7fab78004ab0 comp rx=0 tx=0).stop 2024-03-24T01:32:40.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.187+0000 7fab84e00640 1 -- 172.21.15.90:0/3088600672 shutdown_connections 2024-03-24T01:32:40.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.187+0000 7fab84e00640 1 --2- 172.21.15.90:0/3088600672 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fab5c06bbd0 0x7fab5c06e090 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:40.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.187+0000 7fab84e00640 1 --2- 172.21.15.90:0/3088600672 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab8015c580 0x7fab8015ca00 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:40.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.187+0000 7fab84e00640 1 --2- 172.21.15.90:0/3088600672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fab80143120 0x7fab801435a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:40.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.187+0000 7fab84e00640 1 --2- 172.21.15.90:0/3088600672 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fab64004d40 0x7fab8013c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:40.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.187+0000 7fab84e00640 1 -- 172.21.15.90:0/3088600672 >> 172.21.15.90:0/3088600672 conn(0x7fab800b1770 msgr2=0x7fab800ac8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:40.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.187+0000 7fab84e00640 1 -- 172.21.15.90:0/3088600672 shutdown_connections 2024-03-24T01:32:40.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.187+0000 7fab84e00640 1 -- 172.21.15.90:0/3088600672 wait complete. 2024-03-24T01:32:40.207 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:32:40.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.413+0000 7f17dfe00640 1 Processor -- start 2024-03-24T01:32:40.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.413+0000 7f17dfe00640 1 -- start start 2024-03-24T01:32:40.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.415+0000 7f17dfe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f17d80cc150 0x7f17d80ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:40.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.415+0000 7f17dfe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17d81603e0 0x7f17d806b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:40.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.415+0000 7f17dfe00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17d80cd490 con 0x7f17d8145430 2024-03-24T01:32:40.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.415+0000 7f17dfe00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17d815f900 con 0x7f17d80cc150 2024-03-24T01:32:40.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.415+0000 7f17dfe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d815fbe0 con 0x7f17d81603e0 2024-03-24T01:32:40.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.415+0000 7f17dd400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f17d80cc150 0x7f17d80ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:40.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.415+0000 7f17dd400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f17d80cc150 0x7f17d80ccb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39574/0 (socket says 172.21.15.90:39574) 2024-03-24T01:32:40.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.415+0000 7f17dd400640 1 -- 172.21.15.90:0/196169962 learned_addr learned my addr 172.21.15.90:0/196169962 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:40.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.415+0000 7f17dca00640 1 --2- 172.21.15.90:0/196169962 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17d81603e0 0x7f17d806b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:40.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.416+0000 7f17dd400640 1 -- 172.21.15.90:0/196169962 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17d81603e0 msgr2=0x7f17d806b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:40.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.416+0000 7f17dd400640 1 --2- 172.21.15.90:0/196169962 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17d81603e0 0x7f17d806b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:40.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.416+0000 7f17dd400640 1 -- 172.21.15.90:0/196169962 >> v1:172.21.15.120:6789/0 conn(0x7f17d8145430 legacy=0x7f17d8071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:40.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.416+0000 7f17dd400640 1 -- 172.21.15.90:0/196169962 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d815fec0 con 0x7f17d80cc150 2024-03-24T01:32:40.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.417+0000 7f17dd400640 1 --2- 172.21.15.90:0/196169962 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17d80cc150 0x7f17d80ccb60 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f17cc009870 tx=0x7f17cc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af142f2e6ffd2fb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:40.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.418+0000 7f17d7e00640 1 -- 172.21.15.90:0/196169962 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17cc034070 con 0x7f17d80cc150 2024-03-24T01:32:40.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.418+0000 7f17d7e00640 1 -- 172.21.15.90:0/196169962 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17cc030840 con 0x7f17d80cc150 2024-03-24T01:32:40.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.418+0000 7f17d7e00640 1 -- 172.21.15.90:0/196169962 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17cc030c10 con 0x7f17d80cc150 2024-03-24T01:32:40.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.418+0000 7f17dfe00640 1 -- 172.21.15.90:0/196169962 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17d80cc150 msgr2=0x7f17d80ccb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:40.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.418+0000 7f17dfe00640 1 --2- 172.21.15.90:0/196169962 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17d80cc150 0x7f17d80ccb60 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f17cc009870 tx=0x7f17cc02b980 comp rx=0 tx=0).stop 2024-03-24T01:32:40.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.419+0000 7f17dfe00640 1 -- 172.21.15.90:0/196169962 shutdown_connections 2024-03-24T01:32:40.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.419+0000 7f17dfe00640 1 --2- 172.21.15.90:0/196169962 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17d81603e0 0x7f17d806b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:40.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.419+0000 7f17dfe00640 1 --2- 172.21.15.90:0/196169962 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17d80cc150 0x7f17d80ccb60 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:40.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.419+0000 7f17dfe00640 1 -- 172.21.15.90:0/196169962 >> 172.21.15.90:0/196169962 conn(0x7f17d80b1770 msgr2=0x7f17d80b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:40.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.419+0000 7f17dfe00640 1 -- 172.21.15.90:0/196169962 shutdown_connections 2024-03-24T01:32:40.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.419+0000 7f17dfe00640 1 -- 172.21.15.90:0/196169962 wait complete. 2024-03-24T01:32:40.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.420+0000 7f17dfe00640 1 Processor -- start 2024-03-24T01:32:40.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.421+0000 7f17dfe00640 1 -- start start 2024-03-24T01:32:40.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.421+0000 7f17dfe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17d80cc150 0x7f17d80ab230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:40.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.422+0000 7f17dfe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17d8145430 0x7f17d812ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:40.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.422+0000 7f17dfe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17d81603e0 0x7f17d813fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:40.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.422+0000 7f17dfe00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d806b780 con 0x7f17d80cc150 2024-03-24T01:32:40.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.422+0000 7f17dfe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d806b600 con 0x7f17d8145430 2024-03-24T01:32:40.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.422+0000 7f17dfe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d806b900 con 0x7f17d81603e0 2024-03-24T01:32:40.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.422+0000 7f17dd400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17d80cc150 0x7f17d80ab230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:40.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.422+0000 7f17dd400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17d80cc150 0x7f17d80ab230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39590/0 (socket says 172.21.15.90:39590) 2024-03-24T01:32:40.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.422+0000 7f17dd400640 1 -- 172.21.15.90:0/1162968764 learned_addr learned my addr 172.21.15.90:0/1162968764 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:40.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.422+0000 7f17dca00640 1 --2- 172.21.15.90:0/1162968764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17d8145430 0x7f17d812ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:40.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.422+0000 7f17dd400640 1 -- 172.21.15.90:0/1162968764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17d81603e0 msgr2=0x7f17d813fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:40.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.422+0000 7f17dd400640 1 --2- 172.21.15.90:0/1162968764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17d81603e0 0x7f17d813fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:40.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.422+0000 7f17dd400640 1 -- 172.21.15.90:0/1162968764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17d8145430 msgr2=0x7f17d812ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:40.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.422+0000 7f17dd400640 1 --2- 172.21.15.90:0/1162968764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17d8145430 0x7f17d812ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:40.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.422+0000 7f17dd400640 1 -- 172.21.15.90:0/1162968764 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d8131e50 con 0x7f17d80cc150 2024-03-24T01:32:40.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.423+0000 7f17dd400640 1 --2- 172.21.15.90:0/1162968764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17d80cc150 0x7f17d80ab230 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f17cc02b9c0 tx=0x7f17cc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:40.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.423+0000 7f17d5600640 1 -- 172.21.15.90:0/1162968764 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17cc048070 con 0x7f17d80cc150 2024-03-24T01:32:40.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.423+0000 7f17d5600640 1 -- 172.21.15.90:0/1162968764 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17cc030640 con 0x7f17d80cc150 2024-03-24T01:32:40.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.423+0000 7f17dfe00640 1 -- 172.21.15.90:0/1162968764 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17d80ab770 con 0x7f17d80cc150 2024-03-24T01:32:40.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.423+0000 7f17d5600640 1 -- 172.21.15.90:0/1162968764 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17cc034040 con 0x7f17d80cc150 2024-03-24T01:32:40.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.423+0000 7f17dfe00640 1 -- 172.21.15.90:0/1162968764 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17d80ad0e0 con 0x7f17d80cc150 2024-03-24T01:32:40.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.424+0000 7f17dfe00640 1 -- 172.21.15.90:0/1162968764 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17d8068060 con 0x7f17d80cc150 2024-03-24T01:32:40.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.425+0000 7f17d5600640 1 -- 172.21.15.90:0/1162968764 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f17cc04b070 con 0x7f17d80cc150 2024-03-24T01:32:40.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.426+0000 7f17d5600640 1 --2- 172.21.15.90:0/1162968764 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f17b806bbd0 0x7f17b806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:40.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.426+0000 7f17d5600640 1 -- 172.21.15.90:0/1162968764 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f17cc0ac8f0 con 0x7f17d80cc150 2024-03-24T01:32:40.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.427+0000 7f17dca00640 1 --2- 172.21.15.90:0/1162968764 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f17b806bbd0 0x7f17b806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:40.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.428+0000 7f17dca00640 1 --2- 172.21.15.90:0/1162968764 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f17b806bbd0 0x7f17b806e090 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f17c4006fd0 tx=0x7f17c4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:40.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.430+0000 7f17d5600640 1 -- 172.21.15.90:0/1162968764 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f17cc07b620 con 0x7f17d80cc150 2024-03-24T01:32:40.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.630+0000 7f17dfe00640 1 -- 172.21.15.90:0/1162968764 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17d80b3220 con 0x7f17d80cc150 2024-03-24T01:32:40.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.631+0000 7f17d5600640 1 -- 172.21.15.90:0/1162968764 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+13140 (secure 0 0 0) 0x7f17cc030810 con 0x7f17d80cc150 2024-03-24T01:32:40.632 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:32:40.632 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":63,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:32:39.668661+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:32:40.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.634+0000 7f17dfe00640 1 -- 172.21.15.90:0/1162968764 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f17b806bbd0 msgr2=0x7f17b806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:40.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.634+0000 7f17dfe00640 1 --2- 172.21.15.90:0/1162968764 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f17b806bbd0 0x7f17b806e090 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f17c4006fd0 tx=0x7f17c4008040 comp rx=0 tx=0).stop 2024-03-24T01:32:40.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.634+0000 7f17dfe00640 1 -- 172.21.15.90:0/1162968764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17d80cc150 msgr2=0x7f17d80ab230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:40.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.634+0000 7f17dfe00640 1 --2- 172.21.15.90:0/1162968764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17d80cc150 0x7f17d80ab230 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f17cc02b9c0 tx=0x7f17cc004730 comp rx=0 tx=0).stop 2024-03-24T01:32:40.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.634+0000 7f17dfe00640 1 -- 172.21.15.90:0/1162968764 shutdown_connections 2024-03-24T01:32:40.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.634+0000 7f17dfe00640 1 --2- 172.21.15.90:0/1162968764 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f17b806bbd0 0x7f17b806e090 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:40.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.634+0000 7f17dfe00640 1 --2- 172.21.15.90:0/1162968764 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f17d81603e0 0x7f17d813fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:40.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.634+0000 7f17dfe00640 1 --2- 172.21.15.90:0/1162968764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17d8145430 0x7f17d812ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:40.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.634+0000 7f17dfe00640 1 --2- 172.21.15.90:0/1162968764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17d80cc150 0x7f17d80ab230 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:40.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.634+0000 7f17dfe00640 1 -- 172.21.15.90:0/1162968764 >> 172.21.15.90:0/1162968764 conn(0x7f17d80b1770 msgr2=0x7f17d80af570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:40.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.634+0000 7f17dfe00640 1 -- 172.21.15.90:0/1162968764 shutdown_connections 2024-03-24T01:32:40.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.634+0000 7f17dfe00640 1 -- 172.21.15.90:0/1162968764 wait complete. 2024-03-24T01:32:40.652 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T01:32:40.653 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:32:40.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.861+0000 7fc5f8c00640 1 Processor -- start 2024-03-24T01:32:40.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.861+0000 7fc5f8c00640 1 -- start start 2024-03-24T01:32:40.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.863+0000 7fc5f8c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc5f415bc50 0x7fc5f406b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:40.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.863+0000 7fc5f8c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5f406b600 0x7fc5f40b2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:40.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.863+0000 7fc5f8c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5f406dac0 con 0x7fc5f40aeeb0 2024-03-24T01:32:40.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.863+0000 7fc5f8c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5f40b4da0 con 0x7fc5f415bc50 2024-03-24T01:32:40.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.863+0000 7fc5f8c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f406ba40 con 0x7fc5f406b600 2024-03-24T01:32:40.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.863+0000 7fc5f1e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc5f40aeeb0 0x7fc5f40b44c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40196/0 (socket says 172.21.15.90:40196) 2024-03-24T01:32:40.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.863+0000 7fc5f1e00640 1 -- 172.21.15.90:0/478125435 learned_addr learned my addr 172.21.15.90:0/478125435 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:32:40.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.863+0000 7fc5f2800640 1 --2- 172.21.15.90:0/478125435 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5f406b600 0x7fc5f40b2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:40.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.863+0000 7fc5f1400640 1 --2- 172.21.15.90:0/478125435 >> v2:172.21.15.120:3300/0 conn(0x7fc5f415bc50 0x7fc5f406b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:40.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.864+0000 7fc5f0a00640 1 -- 172.21.15.90:0/478125435 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3846049816 0 0) 0x7fc5f406dac0 con 0x7fc5f40aeeb0 2024-03-24T01:32:40.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.864+0000 7fc5f0a00640 1 -- 172.21.15.90:0/478125435 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5d8003520 con 0x7fc5f40aeeb0 2024-03-24T01:32:40.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.864+0000 7fc5f0a00640 1 -- 172.21.15.90:0/478125435 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1624351433 0 0) 0x7fc5d8003520 con 0x7fc5f40aeeb0 2024-03-24T01:32:40.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.864+0000 7fc5f0a00640 1 -- 172.21.15.90:0/478125435 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5f406b600 msgr2=0x7fc5f40b2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:40.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.864+0000 7fc5f0a00640 1 --2- 172.21.15.90:0/478125435 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5f406b600 0x7fc5f40b2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:40.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.864+0000 7fc5f0a00640 1 -- 172.21.15.90:0/478125435 >> v2:172.21.15.120:3300/0 conn(0x7fc5f415bc50 msgr2=0x7fc5f406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:40.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.864+0000 7fc5f0a00640 1 --2- 172.21.15.90:0/478125435 >> v2:172.21.15.120:3300/0 conn(0x7fc5f415bc50 0x7fc5f406b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:40.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.864+0000 7fc5f0a00640 1 -- 172.21.15.90:0/478125435 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f406dda0 con 0x7fc5f40aeeb0 2024-03-24T01:32:40.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.864+0000 7fc5f0a00640 1 -- 172.21.15.90:0/478125435 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7fc5dc001670 con 0x7fc5f40aeeb0 2024-03-24T01:32:40.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.864+0000 7fc5f0a00640 1 -- 172.21.15.90:0/478125435 >> v1:172.21.15.90:6789/0 conn(0x7fc5f40aeeb0 legacy=0x7fc5f40b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:40.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.864+0000 7fc5f0a00640 1 --2- 172.21.15.90:0/478125435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5d80039b0 0x7fc5d8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:40.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.864+0000 7fc5f0a00640 1 --2- 172.21.15.90:0/478125435 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5d8004800 0x7fc5d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:40.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.864+0000 7fc5f0a00640 1 --2- 172.21.15.90:0/478125435 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5f406b600 0x7fc5d80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:40.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.864+0000 7fc5f0a00640 1 -- 172.21.15.90:0/478125435 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f40b4da0 con 0x7fc5d8004800 2024-03-24T01:32:40.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.864+0000 7fc5f0a00640 1 -- 172.21.15.90:0/478125435 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f406ba40 con 0x7fc5d80039b0 2024-03-24T01:32:40.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.864+0000 7fc5f0a00640 1 -- 172.21.15.90:0/478125435 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d800dc50 con 0x7fc5f406b600 2024-03-24T01:32:40.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.864+0000 7fc5f1400640 1 --2- 172.21.15.90:0/478125435 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5d8004800 0x7fc5d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:40.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.865+0000 7fc5f1e00640 1 --2- 172.21.15.90:0/478125435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5d80039b0 0x7fc5d8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:40.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.865+0000 7fc5f2800640 1 --2- 172.21.15.90:0/478125435 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5f406b600 0x7fc5d80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:40.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.865+0000 7fc5f1400640 1 -- 172.21.15.90:0/478125435 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5f406b600 msgr2=0x7fc5d80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:40.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.865+0000 7fc5f1400640 1 --2- 172.21.15.90:0/478125435 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5f406b600 0x7fc5d80051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:40.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.865+0000 7fc5f1400640 1 -- 172.21.15.90:0/478125435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5d80039b0 msgr2=0x7fc5d8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:40.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.865+0000 7fc5f1400640 1 --2- 172.21.15.90:0/478125435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5d80039b0 0x7fc5d8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:40.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.865+0000 7fc5f1400640 1 -- 172.21.15.90:0/478125435 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d8003520 con 0x7fc5d8004800 2024-03-24T01:32:40.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.866+0000 7fc5f1400640 1 --2- 172.21.15.90:0/478125435 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5d8004800 0x7fc5d8004c80 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4009fc0 tx=0x7fc5d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a090c80e80565b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:40.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.866+0000 7fc5f0a00640 1 -- 172.21.15.90:0/478125435 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e4039070 con 0x7fc5d8004800 2024-03-24T01:32:40.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.866+0000 7fc5f0a00640 1 -- 172.21.15.90:0/478125435 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5e402cc60 con 0x7fc5d8004800 2024-03-24T01:32:40.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.866+0000 7fc5f0a00640 1 -- 172.21.15.90:0/478125435 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e4009c10 con 0x7fc5d8004800 2024-03-24T01:32:40.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.867+0000 7fc5f8c00640 1 -- 172.21.15.90:0/478125435 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5d8004800 msgr2=0x7fc5d8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:40.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.867+0000 7fc5f8c00640 1 --2- 172.21.15.90:0/478125435 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5d8004800 0x7fc5d8004c80 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4009fc0 tx=0x7fc5d8005830 comp rx=0 tx=0).stop 2024-03-24T01:32:40.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.867+0000 7fc5f2800640 1 -- 172.21.15.90:0/478125435 reap_dead start 2024-03-24T01:32:40.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.867+0000 7fc5f8c00640 1 -- 172.21.15.90:0/478125435 shutdown_connections 2024-03-24T01:32:40.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.867+0000 7fc5f8c00640 1 -- 172.21.15.90:0/478125435 >> 172.21.15.90:0/478125435 conn(0x7fc5f40b1770 msgr2=0x7fc5f40b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:40.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.867+0000 7fc5f8c00640 1 -- 172.21.15.90:0/478125435 shutdown_connections 2024-03-24T01:32:40.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.867+0000 7fc5f8c00640 1 -- 172.21.15.90:0/478125435 wait complete. 2024-03-24T01:32:40.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.869+0000 7fc5f8c00640 1 Processor -- start 2024-03-24T01:32:40.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.869+0000 7fc5f8c00640 1 -- start start 2024-03-24T01:32:40.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.870+0000 7fc5f8c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5d8004800 0x7fc5f414ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:40.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.870+0000 7fc5f8c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5f40cf8a0 0x7fc5f40ce5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:40.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.870+0000 7fc5f8c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5f414f420 0x7fc5f40b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:40.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.870+0000 7fc5f8c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f40af290 con 0x7fc5f40cf8a0 2024-03-24T01:32:40.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.870+0000 7fc5f8c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f406e360 con 0x7fc5f414f420 2024-03-24T01:32:40.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.870+0000 7fc5f8c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f406e880 con 0x7fc5d8004800 2024-03-24T01:32:40.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.870+0000 7fc5f1400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5f40cf8a0 0x7fc5f40ce5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:40.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.870+0000 7fc5f1400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5f40cf8a0 0x7fc5f40ce5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39602/0 (socket says 172.21.15.90:39602) 2024-03-24T01:32:40.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.870+0000 7fc5f1400640 1 -- 172.21.15.90:0/1780657986 learned_addr learned my addr 172.21.15.90:0/1780657986 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:40.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.871+0000 7fc5f1400640 1 -- 172.21.15.90:0/1780657986 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5d8004800 msgr2=0x7fc5f414ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:40.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.871+0000 7fc5f1400640 1 --2- 172.21.15.90:0/1780657986 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5d8004800 0x7fc5f414ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:40.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.871+0000 7fc5f1400640 1 -- 172.21.15.90:0/1780657986 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5f414f420 msgr2=0x7fc5f40b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:40.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.871+0000 7fc5f2800640 1 --2- 172.21.15.90:0/1780657986 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5f414f420 0x7fc5f40b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:40.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.871+0000 7fc5f1400640 1 --2- 172.21.15.90:0/1780657986 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5f414f420 0x7fc5f40b3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:40.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.871+0000 7fc5f1400640 1 -- 172.21.15.90:0/1780657986 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f414ffe0 con 0x7fc5f40cf8a0 2024-03-24T01:32:40.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.871+0000 7fc5f1400640 1 --2- 172.21.15.90:0/1780657986 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5f40cf8a0 0x7fc5f40ce5e0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fc5e400e230 tx=0x7fc5e4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:40.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.871+0000 7fc5ea000640 1 -- 172.21.15.90:0/1780657986 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e4039050 con 0x7fc5f40cf8a0 2024-03-24T01:32:40.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.871+0000 7fc5ea000640 1 -- 172.21.15.90:0/1780657986 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5e403a040 con 0x7fc5f40cf8a0 2024-03-24T01:32:40.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.871+0000 7fc5ea000640 1 -- 172.21.15.90:0/1780657986 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e4009c10 con 0x7fc5f40cf8a0 2024-03-24T01:32:40.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.871+0000 7fc5f8c00640 1 -- 172.21.15.90:0/1780657986 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5f40ceb20 con 0x7fc5f40cf8a0 2024-03-24T01:32:40.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.872+0000 7fc5f8c00640 1 -- 172.21.15.90:0/1780657986 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5f40cdb00 con 0x7fc5f40cf8a0 2024-03-24T01:32:40.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.873+0000 7fc5f8c00640 1 -- 172.21.15.90:0/1780657986 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5f404f400 con 0x7fc5f40cf8a0 2024-03-24T01:32:40.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.874+0000 7fc5ea000640 1 -- 172.21.15.90:0/1780657986 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc5e4047020 con 0x7fc5f40cf8a0 2024-03-24T01:32:40.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.875+0000 7fc5ea000640 1 --2- 172.21.15.90:0/1780657986 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc5d006bc00 0x7fc5d006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:40.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.875+0000 7fc5ea000640 1 -- 172.21.15.90:0/1780657986 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fc5e40aca90 con 0x7fc5f40cf8a0 2024-03-24T01:32:40.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.876+0000 7fc5f1e00640 1 --2- 172.21.15.90:0/1780657986 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc5d006bc00 0x7fc5d006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:40.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.877+0000 7fc5f1e00640 1 --2- 172.21.15.90:0/1780657986 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc5d006bc00 0x7fc5d006e0c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc000f60 tx=0x7fc5dc000fe0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:40.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:40.880+0000 7fc5ea000640 1 -- 172.21.15.90:0/1780657986 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc5e404e020 con 0x7fc5f40cf8a0 2024-03-24T01:32:41.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.076+0000 7fc5f8c00640 1 -- 172.21.15.90:0/1780657986 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc5f415a410 con 0x7fc5f40cf8a0 2024-03-24T01:32:41.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.077+0000 7fc5ea000640 1 -- 172.21.15.90:0/1780657986 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v63) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc5e407b730 con 0x7fc5f40cf8a0 2024-03-24T01:32:41.077 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:32:41.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.079+0000 7fc5f8c00640 1 -- 172.21.15.90:0/1780657986 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc5d006bc00 msgr2=0x7fc5d006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:41.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.079+0000 7fc5f8c00640 1 --2- 172.21.15.90:0/1780657986 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc5d006bc00 0x7fc5d006e0c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc000f60 tx=0x7fc5dc000fe0 comp rx=0 tx=0).stop 2024-03-24T01:32:41.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.079+0000 7fc5f8c00640 1 -- 172.21.15.90:0/1780657986 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5f40cf8a0 msgr2=0x7fc5f40ce5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:41.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.079+0000 7fc5f8c00640 1 --2- 172.21.15.90:0/1780657986 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5f40cf8a0 0x7fc5f40ce5e0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fc5e400e230 tx=0x7fc5e4009750 comp rx=0 tx=0).stop 2024-03-24T01:32:41.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.079+0000 7fc5f8c00640 1 -- 172.21.15.90:0/1780657986 shutdown_connections 2024-03-24T01:32:41.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.079+0000 7fc5f8c00640 1 --2- 172.21.15.90:0/1780657986 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc5d006bc00 0x7fc5d006e0c0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:41.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.079+0000 7fc5f8c00640 1 --2- 172.21.15.90:0/1780657986 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5f40cf8a0 0x7fc5f40ce5e0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:41.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.079+0000 7fc5f8c00640 1 --2- 172.21.15.90:0/1780657986 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5f414f420 0x7fc5f40b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:41.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.079+0000 7fc5f8c00640 1 --2- 172.21.15.90:0/1780657986 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc5d8004800 0x7fc5f414ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:41.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.079+0000 7fc5f8c00640 1 -- 172.21.15.90:0/1780657986 >> 172.21.15.90:0/1780657986 conn(0x7fc5f40b1770 msgr2=0x7fc5f415c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:41.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.079+0000 7fc5f8c00640 1 -- 172.21.15.90:0/1780657986 shutdown_connections 2024-03-24T01:32:41.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.079+0000 7fc5f8c00640 1 -- 172.21.15.90:0/1780657986 wait complete. 2024-03-24T01:32:41.099 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-24T01:32:41.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.307+0000 7fad0f000640 1 Processor -- start 2024-03-24T01:32:41.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.307+0000 7fad0f000640 1 -- start start 2024-03-24T01:32:41.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.309+0000 7fad0f000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad08144420 0x7fad0806d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:41.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.309+0000 7fad0f000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fad08144800 0x7fad080cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:41.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.309+0000 7fad0f000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad080cceb0 con 0x7fad080cc8f0 2024-03-24T01:32:41.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.309+0000 7fad0f000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad080cd190 con 0x7fad08144800 2024-03-24T01:32:41.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.309+0000 7fad0f000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad080ceb70 con 0x7fad08144420 2024-03-24T01:32:41.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.309+0000 7fad0d200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fad08144800 0x7fad080cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:41.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.309+0000 7fad0e600640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fad080cc8f0 0x7fad080cf8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40198/0 (socket says 172.21.15.90:40198) 2024-03-24T01:32:41.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.309+0000 7fad0e600640 1 -- 172.21.15.90:0/2867084165 learned_addr learned my addr 172.21.15.90:0/2867084165 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:32:41.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.310+0000 7fad0dc00640 1 --2- 172.21.15.90:0/2867084165 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad08144420 0x7fad0806d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:41.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.310+0000 7fad07e00640 1 -- 172.21.15.90:0/2867084165 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2114835396 0 0) 0x7fad080cceb0 con 0x7fad080cc8f0 2024-03-24T01:32:41.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.310+0000 7fad07e00640 1 -- 172.21.15.90:0/2867084165 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facf0003520 con 0x7fad080cc8f0 2024-03-24T01:32:41.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.310+0000 7fad0d200640 1 -- 172.21.15.90:0/2867084165 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad08144420 msgr2=0x7fad0806d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:41.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.310+0000 7fad0d200640 1 --2- 172.21.15.90:0/2867084165 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad08144420 0x7fad0806d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:41.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.310+0000 7fad0d200640 1 -- 172.21.15.90:0/2867084165 >> v1:172.21.15.90:6789/0 conn(0x7fad080cc8f0 legacy=0x7fad080cf8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:41.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.310+0000 7fad0d200640 1 -- 172.21.15.90:0/2867084165 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad080cee50 con 0x7fad08144800 2024-03-24T01:32:41.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.311+0000 7fad0d200640 1 --2- 172.21.15.90:0/2867084165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad08144800 0x7fad080cf370 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7facf40097a0 tx=0x7facf402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f62d8d6878987f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:41.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.311+0000 7fad07e00640 1 -- 172.21.15.90:0/2867084165 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf4038070 con 0x7fad08144800 2024-03-24T01:32:41.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.311+0000 7fad07e00640 1 -- 172.21.15.90:0/2867084165 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf4031700 con 0x7fad08144800 2024-03-24T01:32:41.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.311+0000 7fad07e00640 1 -- 172.21.15.90:0/2867084165 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf4031ad0 con 0x7fad08144800 2024-03-24T01:32:41.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.312+0000 7fad0f000640 1 -- 172.21.15.90:0/2867084165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad08144800 msgr2=0x7fad080cf370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:41.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.312+0000 7fad0f000640 1 --2- 172.21.15.90:0/2867084165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad08144800 0x7fad080cf370 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7facf40097a0 tx=0x7facf402b7a0 comp rx=0 tx=0).stop 2024-03-24T01:32:41.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.312+0000 7fad0f000640 1 -- 172.21.15.90:0/2867084165 shutdown_connections 2024-03-24T01:32:41.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.312+0000 7fad0f000640 1 --2- 172.21.15.90:0/2867084165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad08144800 0x7fad080cf370 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:41.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.312+0000 7fad0f000640 1 --2- 172.21.15.90:0/2867084165 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad08144420 0x7fad0806d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:41.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.312+0000 7fad0f000640 1 -- 172.21.15.90:0/2867084165 >> 172.21.15.90:0/2867084165 conn(0x7fad0806b0c0 msgr2=0x7fad0806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:41.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.312+0000 7fad0f000640 1 -- 172.21.15.90:0/2867084165 shutdown_connections 2024-03-24T01:32:41.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.312+0000 7fad0f000640 1 -- 172.21.15.90:0/2867084165 wait complete. 2024-03-24T01:32:41.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.314+0000 7fad0f000640 1 Processor -- start 2024-03-24T01:32:41.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.314+0000 7fad0f000640 1 -- start start 2024-03-24T01:32:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.315+0000 7fad0f000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad080cc8f0 0x7fad081faf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.315+0000 7fad0f000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad08144420 0x7fad0815d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.315+0000 7fad0f000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad08144800 0x7fad0815a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.315+0000 7fad0f000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad080ce910 con 0x7fad08144800 2024-03-24T01:32:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.315+0000 7fad0f000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad080ce790 con 0x7fad08144420 2024-03-24T01:32:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.315+0000 7fad0f000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad0815f540 con 0x7fad080cc8f0 2024-03-24T01:32:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.315+0000 7fad0e600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad08144800 0x7fad0815a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.315+0000 7fad0e600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad08144800 0x7fad0815a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39622/0 (socket says 172.21.15.90:39622) 2024-03-24T01:32:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.315+0000 7fad0e600640 1 -- 172.21.15.90:0/1453048217 learned_addr learned my addr 172.21.15.90:0/1453048217 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.315+0000 7fad0dc00640 1 --2- 172.21.15.90:0/1453048217 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad080cc8f0 0x7fad081faf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:41.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.316+0000 7fad0d200640 1 --2- 172.21.15.90:0/1453048217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad08144420 0x7fad0815d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:41.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.316+0000 7fad0e600640 1 -- 172.21.15.90:0/1453048217 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad080cc8f0 msgr2=0x7fad081faf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:41.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.316+0000 7fad0e600640 1 --2- 172.21.15.90:0/1453048217 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad080cc8f0 0x7fad081faf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:41.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.316+0000 7fad0e600640 1 -- 172.21.15.90:0/1453048217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad08144420 msgr2=0x7fad0815d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:41.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.316+0000 7fad0e600640 1 --2- 172.21.15.90:0/1453048217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad08144420 0x7fad0815d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:41.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.316+0000 7fad0e600640 1 -- 172.21.15.90:0/1453048217 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad081e84a0 con 0x7fad08144800 2024-03-24T01:32:41.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.316+0000 7fad0d200640 1 --2- 172.21.15.90:0/1453048217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad08144420 0x7fad0815d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:41.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.316+0000 7fad0e600640 1 --2- 172.21.15.90:0/1453048217 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad08144800 0x7fad0815a8a0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fad0000cb10 tx=0x7fad0000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:41.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.316+0000 7fad05600640 1 -- 172.21.15.90:0/1453048217 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad00017070 con 0x7fad08144800 2024-03-24T01:32:41.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.316+0000 7fad05600640 1 -- 172.21.15.90:0/1453048217 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad0000ac10 con 0x7fad08144800 2024-03-24T01:32:41.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.316+0000 7fad05600640 1 -- 172.21.15.90:0/1453048217 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad00012400 con 0x7fad08144800 2024-03-24T01:32:41.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.316+0000 7fad0f000640 1 -- 172.21.15.90:0/1453048217 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad0815e160 con 0x7fad08144800 2024-03-24T01:32:41.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.316+0000 7fad0f000640 1 -- 172.21.15.90:0/1453048217 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad0815e720 con 0x7fad08144800 2024-03-24T01:32:41.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.317+0000 7fad0f000640 1 -- 172.21.15.90:0/1453048217 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad0804f400 con 0x7fad08144800 2024-03-24T01:32:41.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.318+0000 7fad05600640 1 -- 172.21.15.90:0/1453048217 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fad0001a070 con 0x7fad08144800 2024-03-24T01:32:41.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.319+0000 7fad05600640 1 --2- 172.21.15.90:0/1453048217 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7face806bbd0 0x7face806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:41.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.319+0000 7fad05600640 1 -- 172.21.15.90:0/1453048217 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fad0008a650 con 0x7fad08144800 2024-03-24T01:32:41.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.320+0000 7fad0dc00640 1 --2- 172.21.15.90:0/1453048217 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7face806bbd0 0x7face806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:41.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.321+0000 7fad0dc00640 1 --2- 172.21.15.90:0/1453048217 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7face806bbd0 0x7face806e090 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7facfc006fd0 tx=0x7facfc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:41.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.324+0000 7fad05600640 1 -- 172.21.15.90:0/1453048217 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fad081e84a0 con 0x7fad08144800 2024-03-24T01:32:41.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.523+0000 7fad0f000640 1 -- 172.21.15.90:0/1453048217 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fad0806b820 con 0x7fad08144800 2024-03-24T01:32:41.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.524+0000 7fad05600640 1 -- 172.21.15.90:0/1453048217 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v63) v1 ==== 84+0+65 (secure 0 0 0) 0x7fad000592f0 con 0x7fad08144800 2024-03-24T01:32:41.525 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:32:41.525 INFO:teuthology.orchestra.run.smithi090.stdout:[ 2024-03-24T01:32:41.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-24T01:32:41.525 INFO:teuthology.orchestra.run.smithi090.stdout: "poolnum": 1, 2024-03-24T01:32:41.525 INFO:teuthology.orchestra.run.smithi090.stdout: "poolname": ".mgr" 2024-03-24T01:32:41.525 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:32:41.525 INFO:teuthology.orchestra.run.smithi090.stdout:] 2024-03-24T01:32:41.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.526+0000 7fad0f000640 1 -- 172.21.15.90:0/1453048217 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7face806bbd0 msgr2=0x7face806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:41.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.527+0000 7fad0f000640 1 --2- 172.21.15.90:0/1453048217 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7face806bbd0 0x7face806e090 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7facfc006fd0 tx=0x7facfc008040 comp rx=0 tx=0).stop 2024-03-24T01:32:41.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.527+0000 7fad0f000640 1 -- 172.21.15.90:0/1453048217 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad08144800 msgr2=0x7fad0815a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:41.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.527+0000 7fad0f000640 1 --2- 172.21.15.90:0/1453048217 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad08144800 0x7fad0815a8a0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fad0000cb10 tx=0x7fad0000a060 comp rx=0 tx=0).stop 2024-03-24T01:32:41.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.527+0000 7fad0f000640 1 -- 172.21.15.90:0/1453048217 shutdown_connections 2024-03-24T01:32:41.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.527+0000 7fad0f000640 1 --2- 172.21.15.90:0/1453048217 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7face806bbd0 0x7face806e090 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:41.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.527+0000 7fad0f000640 1 --2- 172.21.15.90:0/1453048217 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad08144800 0x7fad0815a8a0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:41.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.527+0000 7fad0f000640 1 --2- 172.21.15.90:0/1453048217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad08144420 0x7fad0815d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:41.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.527+0000 7fad0f000640 1 --2- 172.21.15.90:0/1453048217 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fad080cc8f0 0x7fad081faf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:41.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.527+0000 7fad0f000640 1 -- 172.21.15.90:0/1453048217 >> 172.21.15.90:0/1453048217 conn(0x7fad0806b0c0 msgr2=0x7fad0813f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:41.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.527+0000 7fad0f000640 1 -- 172.21.15.90:0/1453048217 shutdown_connections 2024-03-24T01:32:41.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.527+0000 7fad0f000640 1 -- 172.21.15.90:0/1453048217 wait complete. 2024-03-24T01:32:41.546 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-24T01:32:41.547 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-24T01:32:41.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.761+0000 7fc8bb000640 1 Processor -- start 2024-03-24T01:32:41.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.761+0000 7fc8bb000640 1 -- start start 2024-03-24T01:32:41.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.763+0000 7fc8bb000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8b406b7d0 0x7fc8b41656f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:41.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.763+0000 7fc8bb000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b406dac0 con 0x7fc8b40aeeb0 2024-03-24T01:32:41.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.763+0000 7fc8bb000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b406dda0 con 0x7fc8b415bc50 2024-03-24T01:32:41.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.763+0000 7fc8bb000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b40b4da0 con 0x7fc8b406b7d0 2024-03-24T01:32:41.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.763+0000 7fc8b3e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc8b40aeeb0 0x7fc8b40b44c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40208/0 (socket says 172.21.15.90:40208) 2024-03-24T01:32:41.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.763+0000 7fc8b3e00640 1 -- 172.21.15.90:0/1816850201 learned_addr learned my addr 172.21.15.90:0/1816850201 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:32:41.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.763+0000 7fc8b9000640 1 --2- 172.21.15.90:0/1816850201 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8b406b7d0 0x7fc8b41656f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:41.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.763+0000 7fc8b3400640 1 -- 172.21.15.90:0/1816850201 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2802879723 0 0) 0x7fc8b406dac0 con 0x7fc8b40aeeb0 2024-03-24T01:32:41.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.764+0000 7fc8b3400640 1 -- 172.21.15.90:0/1816850201 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc898003520 con 0x7fc8b40aeeb0 2024-03-24T01:32:41.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.764+0000 7fc8b3400640 1 -- 172.21.15.90:0/1816850201 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1070146526 0 0) 0x7fc898003520 con 0x7fc8b40aeeb0 2024-03-24T01:32:41.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.764+0000 7fc8b3400640 1 -- 172.21.15.90:0/1816850201 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8b406b7d0 msgr2=0x7fc8b41656f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:41.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.764+0000 7fc8b3400640 1 --2- 172.21.15.90:0/1816850201 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8b406b7d0 0x7fc8b41656f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:41.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.764+0000 7fc8b3400640 1 -- 172.21.15.90:0/1816850201 >> v1:172.21.15.120:6789/0 conn(0x7fc8b415bc50 legacy=0x7fc8b406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:41.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.764+0000 7fc8b3400640 1 -- 172.21.15.90:0/1816850201 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b406e080 con 0x7fc8b40aeeb0 2024-03-24T01:32:41.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.764+0000 7fc8b3400640 1 -- 172.21.15.90:0/1816850201 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7fc8a40019f0 con 0x7fc8b40aeeb0 2024-03-24T01:32:41.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.764+0000 7fc8b3400640 1 -- 172.21.15.90:0/1816850201 >> v1:172.21.15.90:6789/0 conn(0x7fc8b40aeeb0 legacy=0x7fc8b40b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:41.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.764+0000 7fc8b3400640 1 --2- 172.21.15.90:0/1816850201 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8b406b7d0 0x7fc898003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:41.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.764+0000 7fc8b3400640 1 --2- 172.21.15.90:0/1816850201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc898003fd0 0x7fc898004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:41.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.764+0000 7fc8b3400640 1 --2- 172.21.15.90:0/1816850201 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc898004d40 0x7fc8980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:41.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.764+0000 7fc8b3400640 1 -- 172.21.15.90:0/1816850201 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b40b4da0 con 0x7fc898004d40 2024-03-24T01:32:41.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.764+0000 7fc8b3400640 1 -- 172.21.15.90:0/1816850201 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc89800bc50 con 0x7fc898003fd0 2024-03-24T01:32:41.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.764+0000 7fc8b3400640 1 -- 172.21.15.90:0/1816850201 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc89800be50 con 0x7fc8b406b7d0 2024-03-24T01:32:41.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.764+0000 7fc8b9000640 1 --2- 172.21.15.90:0/1816850201 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc898004d40 0x7fc8980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:41.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.765+0000 7fc8b3e00640 1 --2- 172.21.15.90:0/1816850201 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8b406b7d0 0x7fc898003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:41.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.765+0000 7fc8ab400640 1 --2- 172.21.15.90:0/1816850201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc898003fd0 0x7fc898004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:41.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.765+0000 7fc8b9000640 1 -- 172.21.15.90:0/1816850201 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8b406b7d0 msgr2=0x7fc898003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:41.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.765+0000 7fc8b9000640 1 --2- 172.21.15.90:0/1816850201 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8b406b7d0 0x7fc898003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:41.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.765+0000 7fc8b9000640 1 -- 172.21.15.90:0/1816850201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc898003fd0 msgr2=0x7fc898004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:41.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.765+0000 7fc8b9000640 1 --2- 172.21.15.90:0/1816850201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc898003fd0 0x7fc898004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:41.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.765+0000 7fc8b9000640 1 -- 172.21.15.90:0/1816850201 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc898003520 con 0x7fc898004d40 2024-03-24T01:32:41.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.765+0000 7fc8ab400640 1 --2- 172.21.15.90:0/1816850201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc898003fd0 0x7fc898004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:41.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.766+0000 7fc8b9000640 1 --2- 172.21.15.90:0/1816850201 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc898004d40 0x7fc8980051c0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac009ac0 tx=0x7fc898005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c71c7f1b2356341 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:41.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.767+0000 7fc8b3400640 1 -- 172.21.15.90:0/1816850201 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac039070 con 0x7fc898004d40 2024-03-24T01:32:41.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.767+0000 7fc8b3400640 1 -- 172.21.15.90:0/1816850201 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8ac02c6b0 con 0x7fc898004d40 2024-03-24T01:32:41.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.767+0000 7fc8b3400640 1 -- 172.21.15.90:0/1816850201 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac009690 con 0x7fc898004d40 2024-03-24T01:32:41.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.767+0000 7fc8bb000640 1 -- 172.21.15.90:0/1816850201 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc898004d40 msgr2=0x7fc8980051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:41.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.767+0000 7fc8bb000640 1 --2- 172.21.15.90:0/1816850201 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc898004d40 0x7fc8980051c0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac009ac0 tx=0x7fc898005830 comp rx=0 tx=0).stop 2024-03-24T01:32:41.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.767+0000 7fc8b9000640 1 -- 172.21.15.90:0/1816850201 reap_dead start 2024-03-24T01:32:41.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.767+0000 7fc8bb000640 1 -- 172.21.15.90:0/1816850201 shutdown_connections 2024-03-24T01:32:41.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.767+0000 7fc8bb000640 1 -- 172.21.15.90:0/1816850201 >> 172.21.15.90:0/1816850201 conn(0x7fc8b40b1770 msgr2=0x7fc8b40b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:41.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.767+0000 7fc8bb000640 1 -- 172.21.15.90:0/1816850201 shutdown_connections 2024-03-24T01:32:41.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.767+0000 7fc8bb000640 1 -- 172.21.15.90:0/1816850201 wait complete. 2024-03-24T01:32:41.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.769+0000 7fc8bb000640 1 Processor -- start 2024-03-24T01:32:41.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.769+0000 7fc8bb000640 1 -- start start 2024-03-24T01:32:41.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.770+0000 7fc8bb000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc898004d40 0x7fc8b4145360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:41.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.770+0000 7fc8bb000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8b40b3090 0x7fc8b41e9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:41.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.770+0000 7fc8bb000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8b41e9590 0x7fc8b41e9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:41.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.770+0000 7fc8bb000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b406e880 con 0x7fc898004d40 2024-03-24T01:32:41.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.770+0000 7fc8bb000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b40af290 con 0x7fc8b40b3090 2024-03-24T01:32:41.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.770+0000 7fc8bb000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b406ea00 con 0x7fc8b41e9590 2024-03-24T01:32:41.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.770+0000 7fc8b3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc898004d40 0x7fc8b4145360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:41.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.771+0000 7fc8b3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc898004d40 0x7fc8b4145360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39628/0 (socket says 172.21.15.90:39628) 2024-03-24T01:32:41.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.771+0000 7fc8b3e00640 1 -- 172.21.15.90:0/2041532523 learned_addr learned my addr 172.21.15.90:0/2041532523 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:41.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.771+0000 7fc8b9000640 1 --2- 172.21.15.90:0/2041532523 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8b41e9590 0x7fc8b41e9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:41.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.771+0000 7fc8ab400640 1 --2- 172.21.15.90:0/2041532523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8b40b3090 0x7fc8b41e9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:41.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.771+0000 7fc8b3e00640 1 -- 172.21.15.90:0/2041532523 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8b41e9590 msgr2=0x7fc8b41e9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:41.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.771+0000 7fc8b3e00640 1 --2- 172.21.15.90:0/2041532523 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8b41e9590 0x7fc8b41e9a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:41.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.771+0000 7fc8b3e00640 1 -- 172.21.15.90:0/2041532523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8b40b3090 msgr2=0x7fc8b41e9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:41.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.771+0000 7fc8b3e00640 1 --2- 172.21.15.90:0/2041532523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8b40b3090 0x7fc8b41e9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:41.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.771+0000 7fc8b3e00640 1 -- 172.21.15.90:0/2041532523 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b41e9f60 con 0x7fc898004d40 2024-03-24T01:32:41.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.771+0000 7fc8ab400640 1 --2- 172.21.15.90:0/2041532523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8b40b3090 0x7fc8b41e9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:41.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.771+0000 7fc8b3e00640 1 --2- 172.21.15.90:0/2041532523 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc898004d40 0x7fc8b4145360 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fc8a4007a30 tx=0x7fc8a400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:41.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.772+0000 7fc8b0c00640 1 -- 172.21.15.90:0/2041532523 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a400ee50 con 0x7fc898004d40 2024-03-24T01:32:41.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.772+0000 7fc8b0c00640 1 -- 172.21.15.90:0/2041532523 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8a4018070 con 0x7fc898004d40 2024-03-24T01:32:41.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.772+0000 7fc8b0c00640 1 -- 172.21.15.90:0/2041532523 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a4013680 con 0x7fc898004d40 2024-03-24T01:32:41.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.772+0000 7fc8bb000640 1 -- 172.21.15.90:0/2041532523 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8b40b3530 con 0x7fc898004d40 2024-03-24T01:32:41.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.772+0000 7fc8bb000640 1 -- 172.21.15.90:0/2041532523 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8b40b3af0 con 0x7fc898004d40 2024-03-24T01:32:41.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.773+0000 7fc8bb000640 1 -- 172.21.15.90:0/2041532523 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8b404f400 con 0x7fc898004d40 2024-03-24T01:32:41.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.774+0000 7fc8b0c00640 1 -- 172.21.15.90:0/2041532523 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc8a4013820 con 0x7fc898004d40 2024-03-24T01:32:41.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.775+0000 7fc8b0c00640 1 --2- 172.21.15.90:0/2041532523 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc89006bbd0 0x7fc89006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:41.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.775+0000 7fc8b0c00640 1 -- 172.21.15.90:0/2041532523 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fc8a408b0a0 con 0x7fc898004d40 2024-03-24T01:32:41.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.775+0000 7fc8ab400640 1 --2- 172.21.15.90:0/2041532523 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc89006bbd0 0x7fc89006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:41.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.777+0000 7fc8ab400640 1 --2- 172.21.15.90:0/2041532523 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc89006bbd0 0x7fc89006e090 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fc8b40cbae0 tx=0x7fc89c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:41.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.779+0000 7fc8b0c00640 1 -- 172.21.15.90:0/2041532523 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc8a4059660 con 0x7fc898004d40 2024-03-24T01:32:41.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:41.976+0000 7fc8bb000640 1 -- 172.21.15.90:0/2041532523 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fc8b415c1e0 con 0x7fc898004d40 2024-03-24T01:32:42.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.675+0000 7fc8b0c00640 1 -- 172.21.15.90:0/2041532523 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v64) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc8a405d700 con 0x7fc898004d40 2024-03-24T01:32:42.676 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_metadata' created 2024-03-24T01:32:42.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.679+0000 7fc8bb000640 1 -- 172.21.15.90:0/2041532523 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc89006bbd0 msgr2=0x7fc89006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:42.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.679+0000 7fc8bb000640 1 --2- 172.21.15.90:0/2041532523 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc89006bbd0 0x7fc89006e090 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fc8b40cbae0 tx=0x7fc89c009040 comp rx=0 tx=0).stop 2024-03-24T01:32:42.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.679+0000 7fc8bb000640 1 -- 172.21.15.90:0/2041532523 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc898004d40 msgr2=0x7fc8b4145360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:42.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.679+0000 7fc8bb000640 1 --2- 172.21.15.90:0/2041532523 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc898004d40 0x7fc8b4145360 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fc8a4007a30 tx=0x7fc8a400e530 comp rx=0 tx=0).stop 2024-03-24T01:32:42.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.679+0000 7fc8bb000640 1 -- 172.21.15.90:0/2041532523 shutdown_connections 2024-03-24T01:32:42.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.679+0000 7fc8bb000640 1 --2- 172.21.15.90:0/2041532523 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc89006bbd0 0x7fc89006e090 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:42.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.679+0000 7fc8bb000640 1 --2- 172.21.15.90:0/2041532523 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc8b41e9590 0x7fc8b41e9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:42.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.679+0000 7fc8bb000640 1 --2- 172.21.15.90:0/2041532523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc8b40b3090 0x7fc8b41e9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:42.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.679+0000 7fc8bb000640 1 --2- 172.21.15.90:0/2041532523 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc898004d40 0x7fc8b4145360 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:42.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.679+0000 7fc8bb000640 1 -- 172.21.15.90:0/2041532523 >> 172.21.15.90:0/2041532523 conn(0x7fc8b40b1770 msgr2=0x7fc8b4071990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:42.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.679+0000 7fc8bb000640 1 -- 172.21.15.90:0/2041532523 shutdown_connections 2024-03-24T01:32:42.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.680+0000 7fc8bb000640 1 -- 172.21.15.90:0/2041532523 wait complete. 2024-03-24T01:32:42.709 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-24T01:32:42.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.912+0000 7fceed200640 1 Processor -- start 2024-03-24T01:32:42.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.912+0000 7fceed200640 1 -- start start 2024-03-24T01:32:42.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.914+0000 7fceed200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcee80aee80 0x7fcee80b33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:42.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.914+0000 7fceed200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcee806b0c0 0x7fcee806b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:42.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.914+0000 7fceed200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee80b3eb0 con 0x7fcee80b3930 2024-03-24T01:32:42.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.914+0000 7fceed200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcee806ba50 con 0x7fcee806b0c0 2024-03-24T01:32:42.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.914+0000 7fceed200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcee806e680 con 0x7fcee80aee80 2024-03-24T01:32:42.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.914+0000 7fcee6800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcee806b0c0 0x7fcee806b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:42.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.914+0000 7fcee6800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcee806b0c0 0x7fcee806b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39644/0 (socket says 172.21.15.90:39644) 2024-03-24T01:32:42.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.914+0000 7fcee6800640 1 -- 172.21.15.90:0/3690216837 learned_addr learned my addr 172.21.15.90:0/3690216837 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:42.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.914+0000 7fcee5e00640 1 --2- 172.21.15.90:0/3690216837 >> v2:172.21.15.120:3300/0 conn(0x7fcee80aee80 0x7fcee80b33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:42.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.915+0000 7fcee6800640 1 -- 172.21.15.90:0/3690216837 >> v2:172.21.15.120:3300/0 conn(0x7fcee80aee80 msgr2=0x7fcee80b33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:42.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.915+0000 7fcee6800640 1 --2- 172.21.15.90:0/3690216837 >> v2:172.21.15.120:3300/0 conn(0x7fcee80aee80 0x7fcee80b33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:42.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.915+0000 7fcee6800640 1 -- 172.21.15.90:0/3690216837 >> v1:172.21.15.120:6789/0 conn(0x7fcee80b3930 legacy=0x7fcee80cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:42.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.915+0000 7fcee6800640 1 -- 172.21.15.90:0/3690216837 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee806e960 con 0x7fcee806b0c0 2024-03-24T01:32:42.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.916+0000 7fcee6800640 1 --2- 172.21.15.90:0/3690216837 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcee806b0c0 0x7fcee806b510 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fcee8072e30 tx=0x7fcee002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cce3eba65b7bb449 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:42.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.916+0000 7fcee4a00640 1 -- 172.21.15.90:0/3690216837 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee002c9f0 con 0x7fcee806b0c0 2024-03-24T01:32:42.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.916+0000 7fcee4a00640 1 -- 172.21.15.90:0/3690216837 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee002cb90 con 0x7fcee806b0c0 2024-03-24T01:32:42.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.916+0000 7fcee4a00640 1 -- 172.21.15.90:0/3690216837 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee00343f0 con 0x7fcee806b0c0 2024-03-24T01:32:42.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.917+0000 7fceed200640 1 -- 172.21.15.90:0/3690216837 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcee806b0c0 msgr2=0x7fcee806b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:42.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.917+0000 7fceed200640 1 --2- 172.21.15.90:0/3690216837 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcee806b0c0 0x7fcee806b510 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fcee8072e30 tx=0x7fcee002be50 comp rx=0 tx=0).stop 2024-03-24T01:32:42.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.917+0000 7fceed200640 1 -- 172.21.15.90:0/3690216837 shutdown_connections 2024-03-24T01:32:42.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.917+0000 7fceed200640 1 --2- 172.21.15.90:0/3690216837 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcee806b0c0 0x7fcee806b510 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:42.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.917+0000 7fceed200640 1 --2- 172.21.15.90:0/3690216837 >> v2:172.21.15.120:3300/0 conn(0x7fcee80aee80 0x7fcee80b33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:42.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.917+0000 7fceed200640 1 -- 172.21.15.90:0/3690216837 >> 172.21.15.90:0/3690216837 conn(0x7fcee80b1770 msgr2=0x7fcee80b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:42.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.917+0000 7fceed200640 1 -- 172.21.15.90:0/3690216837 shutdown_connections 2024-03-24T01:32:42.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.917+0000 7fceed200640 1 -- 172.21.15.90:0/3690216837 wait complete. 2024-03-24T01:32:42.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.919+0000 7fceed200640 1 Processor -- start 2024-03-24T01:32:42.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.920+0000 7fceed200640 1 -- start start 2024-03-24T01:32:42.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.921+0000 7fceed200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcee806b0c0 0x7fcee814fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:42.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.921+0000 7fceed200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcee80aee80 0x7fcee80a8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:42.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.921+0000 7fceed200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcee80b3930 0x7fcee8164150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:42.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.921+0000 7fceed200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee8166ad0 con 0x7fcee80aee80 2024-03-24T01:32:42.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.921+0000 7fceed200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee80af260 con 0x7fcee806b0c0 2024-03-24T01:32:42.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.921+0000 7fceed200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee80ccfb0 con 0x7fcee80b3930 2024-03-24T01:32:42.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.921+0000 7fcee5400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcee80aee80 0x7fcee80a8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:42.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.921+0000 7fcee5400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcee80aee80 0x7fcee80a8ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39658/0 (socket says 172.21.15.90:39658) 2024-03-24T01:32:42.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.921+0000 7fcee5400640 1 -- 172.21.15.90:0/941081609 learned_addr learned my addr 172.21.15.90:0/941081609 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:42.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.921+0000 7fcee5e00640 1 --2- 172.21.15.90:0/941081609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcee806b0c0 0x7fcee814fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:42.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.922+0000 7fcee6800640 1 --2- 172.21.15.90:0/941081609 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcee80b3930 0x7fcee8164150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:42.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.922+0000 7fcee5400640 1 -- 172.21.15.90:0/941081609 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcee80b3930 msgr2=0x7fcee8164150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:42.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.922+0000 7fcee5400640 1 --2- 172.21.15.90:0/941081609 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcee80b3930 0x7fcee8164150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:42.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.922+0000 7fcee5400640 1 -- 172.21.15.90:0/941081609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcee806b0c0 msgr2=0x7fcee814fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:42.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.922+0000 7fcee5400640 1 --2- 172.21.15.90:0/941081609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcee806b0c0 0x7fcee814fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:42.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.922+0000 7fcee5400640 1 -- 172.21.15.90:0/941081609 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee80a97e0 con 0x7fcee80aee80 2024-03-24T01:32:42.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.922+0000 7fcee5400640 1 --2- 172.21.15.90:0/941081609 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcee80aee80 0x7fcee80a8ee0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fced400b3f0 tx=0x7fced400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:42.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.922+0000 7fcede000640 1 -- 172.21.15.90:0/941081609 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced4015020 con 0x7fcee80aee80 2024-03-24T01:32:42.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.923+0000 7fcede000640 1 -- 172.21.15.90:0/941081609 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced4011830 con 0x7fcee80aee80 2024-03-24T01:32:42.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.923+0000 7fcede000640 1 -- 172.21.15.90:0/941081609 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced4011c20 con 0x7fcee80aee80 2024-03-24T01:32:42.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.923+0000 7fceed200640 1 -- 172.21.15.90:0/941081609 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcee80a9ac0 con 0x7fcee80aee80 2024-03-24T01:32:42.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.923+0000 7fceed200640 1 -- 172.21.15.90:0/941081609 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee80aa080 con 0x7fcee80aee80 2024-03-24T01:32:42.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.924+0000 7fceed200640 1 -- 172.21.15.90:0/941081609 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcee804f400 con 0x7fcee80aee80 2024-03-24T01:32:42.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.925+0000 7fcede000640 1 -- 172.21.15.90:0/941081609 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fced4018020 con 0x7fcee80aee80 2024-03-24T01:32:42.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.926+0000 7fcede000640 1 --2- 172.21.15.90:0/941081609 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcec806bc00 0x7fcec806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:42.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.926+0000 7fcede000640 1 -- 172.21.15.90:0/941081609 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5619+0+0 (secure 0 0 0) 0x7fced408a650 con 0x7fcee80aee80 2024-03-24T01:32:42.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.927+0000 7fcee5e00640 1 --2- 172.21.15.90:0/941081609 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcec806bc00 0x7fcec806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:42.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.928+0000 7fcee5e00640 1 --2- 172.21.15.90:0/941081609 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcec806bc00 0x7fcec806e0c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fced8006fd0 tx=0x7fced8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:42.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:42.931+0000 7fcede000640 1 -- 172.21.15.90:0/941081609 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fced401f030 con 0x7fcee80aee80 2024-03-24T01:32:43.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.129+0000 7fceed200640 1 -- 172.21.15.90:0/941081609 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fcee806e250 con 0x7fcee80aee80 2024-03-24T01:32:43.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.683+0000 7fcede000640 1 -- 172.21.15.90:0/941081609 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v65) v1 ==== 170+0+0 (secure 0 0 0) 0x7fced40590f0 con 0x7fcee80aee80 2024-03-24T01:32:43.684 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_data' created 2024-03-24T01:32:43.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.687+0000 7fceed200640 1 -- 172.21.15.90:0/941081609 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcec806bc00 msgr2=0x7fcec806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:43.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.687+0000 7fceed200640 1 --2- 172.21.15.90:0/941081609 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcec806bc00 0x7fcec806e0c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fced8006fd0 tx=0x7fced8008040 comp rx=0 tx=0).stop 2024-03-24T01:32:43.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.687+0000 7fceed200640 1 -- 172.21.15.90:0/941081609 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcee80aee80 msgr2=0x7fcee80a8ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:43.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.687+0000 7fceed200640 1 --2- 172.21.15.90:0/941081609 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcee80aee80 0x7fcee80a8ee0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fced400b3f0 tx=0x7fced400b8b0 comp rx=0 tx=0).stop 2024-03-24T01:32:43.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.687+0000 7fceed200640 1 -- 172.21.15.90:0/941081609 shutdown_connections 2024-03-24T01:32:43.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.687+0000 7fceed200640 1 --2- 172.21.15.90:0/941081609 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcec806bc00 0x7fcec806e0c0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:43.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.687+0000 7fceed200640 1 --2- 172.21.15.90:0/941081609 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcee80aee80 0x7fcee80a8ee0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:43.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.687+0000 7fceed200640 1 --2- 172.21.15.90:0/941081609 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcee80b3930 0x7fcee8164150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:43.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.687+0000 7fceed200640 1 --2- 172.21.15.90:0/941081609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcee806b0c0 0x7fcee814fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:43.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.687+0000 7fceed200640 1 -- 172.21.15.90:0/941081609 >> 172.21.15.90:0/941081609 conn(0x7fcee80b1770 msgr2=0x7fcee806f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:43.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.688+0000 7fceed200640 1 -- 172.21.15.90:0/941081609 shutdown_connections 2024-03-24T01:32:43.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.688+0000 7fceed200640 1 -- 172.21.15.90:0/941081609 wait complete. 2024-03-24T01:32:43.718 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-24T01:32:43.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.895+0000 7f4075600640 1 Processor -- start 2024-03-24T01:32:43.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.895+0000 7f4075600640 1 -- start start 2024-03-24T01:32:43.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.897+0000 7f4075600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f407006b7d0 0x7f40700cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:43.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.897+0000 7f4075600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f407006bca0 con 0x7f4070159620 2024-03-24T01:32:43.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.897+0000 7f4075600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40700b36f0 con 0x7f40700cef80 2024-03-24T01:32:43.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.897+0000 7f4075600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40700b4470 con 0x7f407006b7d0 2024-03-24T01:32:43.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.897+0000 7f4067e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f407006b7d0 0x7f40700cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:43.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.897+0000 7f4067e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f407006b7d0 0x7f40700cea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39660/0 (socket says 172.21.15.90:39660) 2024-03-24T01:32:43.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.897+0000 7f4067e00640 1 -- 172.21.15.90:0/143155853 learned_addr learned my addr 172.21.15.90:0/143155853 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:43.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.897+0000 7f406ea00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4070159620 0x7f407006b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40224/0 (socket says 172.21.15.90:40224) 2024-03-24T01:32:43.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.897+0000 7f4067e00640 1 -- 172.21.15.90:0/143155853 >> v1:172.21.15.120:6789/0 conn(0x7f40700cef80 legacy=0x7f40700ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:43.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.897+0000 7f4067e00640 1 -- 172.21.15.90:0/143155853 >> v1:172.21.15.90:6789/0 conn(0x7f4070159620 legacy=0x7f407006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:43.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.897+0000 7f406e000640 1 -- 172.21.15.90:0/143155853 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3335981290 0 0) 0x7f407006bca0 con 0x7f4070159620 2024-03-24T01:32:43.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.898+0000 7f4067e00640 1 -- 172.21.15.90:0/143155853 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40700b3990 con 0x7f407006b7d0 2024-03-24T01:32:43.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.899+0000 7f4067e00640 1 --2- 172.21.15.90:0/143155853 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f407006b7d0 0x7f40700cea40 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f4058009870 tx=0x7f405802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8361cd6d8666bc76 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:43.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.899+0000 7f406e000640 1 -- 172.21.15.90:0/143155853 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4058035020 con 0x7f407006b7d0 2024-03-24T01:32:43.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.899+0000 7f406e000640 1 -- 172.21.15.90:0/143155853 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4058031870 con 0x7f407006b7d0 2024-03-24T01:32:43.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.899+0000 7f406e000640 1 -- 172.21.15.90:0/143155853 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4058031c40 con 0x7f407006b7d0 2024-03-24T01:32:43.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.899+0000 7f4075600640 1 -- 172.21.15.90:0/143155853 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f407006b7d0 msgr2=0x7f40700cea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:43.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.899+0000 7f4075600640 1 --2- 172.21.15.90:0/143155853 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f407006b7d0 0x7f40700cea40 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f4058009870 tx=0x7f405802b910 comp rx=0 tx=0).stop 2024-03-24T01:32:43.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.899+0000 7f4075600640 1 -- 172.21.15.90:0/143155853 shutdown_connections 2024-03-24T01:32:43.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.899+0000 7f4075600640 1 --2- 172.21.15.90:0/143155853 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f407006b7d0 0x7f40700cea40 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:43.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.899+0000 7f4075600640 1 -- 172.21.15.90:0/143155853 >> 172.21.15.90:0/143155853 conn(0x7f40700b1770 msgr2=0x7f40700b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:43.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.899+0000 7f4075600640 1 -- 172.21.15.90:0/143155853 shutdown_connections 2024-03-24T01:32:43.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.899+0000 7f4075600640 1 -- 172.21.15.90:0/143155853 wait complete. 2024-03-24T01:32:43.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.901+0000 7f4075600640 1 Processor -- start 2024-03-24T01:32:43.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.901+0000 7f4075600640 1 -- start start 2024-03-24T01:32:43.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.902+0000 7f4075600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f407006b7d0 0x7f40701431b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:43.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.902+0000 7f4075600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40700cef80 0x7f40700c9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:43.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.902+0000 7f4075600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4070159620 0x7f4070143c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:43.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.902+0000 7f4075600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40700cf430 con 0x7f40700cef80 2024-03-24T01:32:43.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.902+0000 7f4075600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40700b3f50 con 0x7f407006b7d0 2024-03-24T01:32:43.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.902+0000 7f4075600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4070159a00 con 0x7f4070159620 2024-03-24T01:32:43.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.902+0000 7f4067e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40700cef80 0x7f40700c9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:43.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.902+0000 7f4067e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40700cef80 0x7f40700c9de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39672/0 (socket says 172.21.15.90:39672) 2024-03-24T01:32:43.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.902+0000 7f4067e00640 1 -- 172.21.15.90:0/3596099870 learned_addr learned my addr 172.21.15.90:0/3596099870 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:43.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.903+0000 7f4067e00640 1 -- 172.21.15.90:0/3596099870 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4070159620 msgr2=0x7f4070143c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:43.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.903+0000 7f4067e00640 1 --2- 172.21.15.90:0/3596099870 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4070159620 0x7f4070143c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:43.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.903+0000 7f4067e00640 1 -- 172.21.15.90:0/3596099870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f407006b7d0 msgr2=0x7f40701431b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:43.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.903+0000 7f4067e00640 1 --2- 172.21.15.90:0/3596099870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f407006b7d0 0x7f40701431b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:43.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.903+0000 7f4067e00640 1 -- 172.21.15.90:0/3596099870 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40700c8d40 con 0x7f40700cef80 2024-03-24T01:32:43.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.903+0000 7f4067e00640 1 --2- 172.21.15.90:0/3596099870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40700cef80 0x7f40700c9de0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f4058009870 tx=0x7f40580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:43.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.903+0000 7f4066a00640 1 -- 172.21.15.90:0/3596099870 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40580443f0 con 0x7f40700cef80 2024-03-24T01:32:43.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.903+0000 7f4066a00640 1 -- 172.21.15.90:0/3596099870 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4058044590 con 0x7f40700cef80 2024-03-24T01:32:43.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.903+0000 7f4066a00640 1 -- 172.21.15.90:0/3596099870 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4058044980 con 0x7f40700cef80 2024-03-24T01:32:43.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.904+0000 7f4075600640 1 -- 172.21.15.90:0/3596099870 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40700c9020 con 0x7f40700cef80 2024-03-24T01:32:43.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.904+0000 7f4075600640 1 -- 172.21.15.90:0/3596099870 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40700c95e0 con 0x7f40700cef80 2024-03-24T01:32:43.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.905+0000 7f4075600640 1 -- 172.21.15.90:0/3596099870 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f407004f400 con 0x7f40700cef80 2024-03-24T01:32:43.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.906+0000 7f4066a00640 1 -- 172.21.15.90:0/3596099870 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f405804b060 con 0x7f40700cef80 2024-03-24T01:32:43.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.907+0000 7f4066a00640 1 --2- 172.21.15.90:0/3596099870 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f404c06bbd0 0x7f404c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:43.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.907+0000 7f4066a00640 1 -- 172.21.15.90:0/3596099870 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f40580ace10 con 0x7f40700cef80 2024-03-24T01:32:43.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.907+0000 7f406ea00640 1 --2- 172.21.15.90:0/3596099870 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f404c06bbd0 0x7f404c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:43.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.908+0000 7f406ea00640 1 --2- 172.21.15.90:0/3596099870 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f404c06bbd0 0x7f404c06e090 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f4060000c00 tx=0x7f40600079c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:43.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:43.911+0000 7f4066a00640 1 -- 172.21.15.90:0/3596099870 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f40580b0030 con 0x7f40700cef80 2024-03-24T01:32:44.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.148+0000 7f4075600640 1 -- 172.21.15.90:0/3596099870 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f40700b2e90 con 0x7f40700cef80 2024-03-24T01:32:44.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.745+0000 7f4066a00640 1 -- 172.21.15.90:0/3596099870 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 9 and data pool 10 v46) v1 ==== 173+0+0 (secure 0 0 0) 0x7f405803f070 con 0x7f40700cef80 2024-03-24T01:32:44.746 INFO:teuthology.orchestra.run.smithi090.stderr:new fs with metadata pool 9 and data pool 10 2024-03-24T01:32:44.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.752+0000 7f4075600640 1 -- 172.21.15.90:0/3596099870 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f404c06bbd0 msgr2=0x7f404c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:44.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.752+0000 7f4075600640 1 --2- 172.21.15.90:0/3596099870 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f404c06bbd0 0x7f404c06e090 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f4060000c00 tx=0x7f40600079c0 comp rx=0 tx=0).stop 2024-03-24T01:32:44.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.752+0000 7f4075600640 1 -- 172.21.15.90:0/3596099870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40700cef80 msgr2=0x7f40700c9de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:44.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.752+0000 7f4075600640 1 --2- 172.21.15.90:0/3596099870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40700cef80 0x7f40700c9de0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f4058009870 tx=0x7f40580046c0 comp rx=0 tx=0).stop 2024-03-24T01:32:44.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.753+0000 7f4075600640 1 -- 172.21.15.90:0/3596099870 shutdown_connections 2024-03-24T01:32:44.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.753+0000 7f4075600640 1 --2- 172.21.15.90:0/3596099870 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f404c06bbd0 0x7f404c06e090 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:44.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.753+0000 7f4075600640 1 --2- 172.21.15.90:0/3596099870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40700cef80 0x7f40700c9de0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:44.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.753+0000 7f4075600640 1 --2- 172.21.15.90:0/3596099870 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4070159620 0x7f4070143c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:44.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.753+0000 7f4075600640 1 --2- 172.21.15.90:0/3596099870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f407006b7d0 0x7f40701431b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:44.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.753+0000 7f4075600640 1 -- 172.21.15.90:0/3596099870 >> 172.21.15.90:0/3596099870 conn(0x7f40700b1770 msgr2=0x7f407006dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:44.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.753+0000 7f4075600640 1 -- 172.21.15.90:0/3596099870 shutdown_connections 2024-03-24T01:32:44.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.754+0000 7f4075600640 1 -- 172.21.15.90:0/3596099870 wait complete. 2024-03-24T01:32:44.780 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:32:44.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.958+0000 7f2d95000640 1 Processor -- start 2024-03-24T01:32:44.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.959+0000 7f2d95000640 1 -- start start 2024-03-24T01:32:44.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.960+0000 7f2d95000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d900cdac0 0x7f2d9006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:44.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.960+0000 7f2d95000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d9015bf70 con 0x7f2d9006b600 2024-03-24T01:32:44.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.960+0000 7f2d95000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d9015b360 con 0x7f2d90155030 2024-03-24T01:32:44.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.960+0000 7f2d95000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9006ba70 con 0x7f2d900cdac0 2024-03-24T01:32:44.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.960+0000 7f2d8e800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f2d9006b600 0x7f2d900ae530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40234/0 (socket says 172.21.15.90:40234) 2024-03-24T01:32:44.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.960+0000 7f2d8e800640 1 -- 172.21.15.90:0/1281213439 learned_addr learned my addr 172.21.15.90:0/1281213439 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:32:44.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.961+0000 7f2d85400640 1 --2- 172.21.15.90:0/1281213439 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d900cdac0 0x7f2d9006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:44.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.961+0000 7f2d8d400640 1 -- 172.21.15.90:0/1281213439 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 414625786 0 0) 0x7f2d9015bf70 con 0x7f2d9006b600 2024-03-24T01:32:44.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.961+0000 7f2d8d400640 1 -- 172.21.15.90:0/1281213439 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d74003520 con 0x7f2d9006b600 2024-03-24T01:32:44.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.961+0000 7f2d8d400640 1 -- 172.21.15.90:0/1281213439 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2866116410 0 0) 0x7f2d74003520 con 0x7f2d9006b600 2024-03-24T01:32:44.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.961+0000 7f2d8d400640 1 -- 172.21.15.90:0/1281213439 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d900cdac0 msgr2=0x7f2d9006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:44.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.961+0000 7f2d8d400640 1 --2- 172.21.15.90:0/1281213439 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d900cdac0 0x7f2d9006b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:44.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.961+0000 7f2d8d400640 1 -- 172.21.15.90:0/1281213439 >> v1:172.21.15.120:6789/0 conn(0x7f2d90155030 legacy=0x7f2d9015efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:44.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.961+0000 7f2d8d400640 1 -- 172.21.15.90:0/1281213439 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d9015b640 con 0x7f2d9006b600 2024-03-24T01:32:44.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.961+0000 7f2d8d400640 1 -- 172.21.15.90:0/1281213439 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f2d880019f0 con 0x7f2d9006b600 2024-03-24T01:32:44.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.961+0000 7f2d8d400640 1 -- 172.21.15.90:0/1281213439 >> v1:172.21.15.90:6789/0 conn(0x7f2d9006b600 legacy=0x7f2d900ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:44.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.961+0000 7f2d8d400640 1 --2- 172.21.15.90:0/1281213439 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d740039e0 0x7f2d74003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:44.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.961+0000 7f2d8d400640 1 --2- 172.21.15.90:0/1281213439 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d900cdac0 0x7f2d74004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:44.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.961+0000 7f2d8d400640 1 --2- 172.21.15.90:0/1281213439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d74004d40 0x7f2d740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:44.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.961+0000 7f2d8d400640 1 -- 172.21.15.90:0/1281213439 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9006ba70 con 0x7f2d740039e0 2024-03-24T01:32:44.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.961+0000 7f2d8d400640 1 -- 172.21.15.90:0/1281213439 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7400dc80 con 0x7f2d74004d40 2024-03-24T01:32:44.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.961+0000 7f2d8d400640 1 -- 172.21.15.90:0/1281213439 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7400de80 con 0x7f2d900cdac0 2024-03-24T01:32:44.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.962+0000 7f2d8de00640 1 --2- 172.21.15.90:0/1281213439 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d740039e0 0x7f2d74003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:44.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.962+0000 7f2d8de00640 1 -- 172.21.15.90:0/1281213439 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d900cdac0 msgr2=0x7f2d74004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:44.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.962+0000 7f2d8de00640 1 --2- 172.21.15.90:0/1281213439 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d900cdac0 0x7f2d74004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:44.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.962+0000 7f2d8e800640 1 --2- 172.21.15.90:0/1281213439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d74004d40 0x7f2d740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:44.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.962+0000 7f2d8de00640 1 -- 172.21.15.90:0/1281213439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d74004d40 msgr2=0x7f2d740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:44.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.962+0000 7f2d8de00640 1 --2- 172.21.15.90:0/1281213439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d74004d40 0x7f2d740051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:44.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.962+0000 7f2d8de00640 1 -- 172.21.15.90:0/1281213439 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d74003520 con 0x7f2d740039e0 2024-03-24T01:32:44.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.963+0000 7f2d8de00640 1 --2- 172.21.15.90:0/1281213439 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d740039e0 0x7f2d74003e00 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f2d80001720 tx=0x7f2d74005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39c961bd413af7da server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:44.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.963+0000 7f2d8d400640 1 -- 172.21.15.90:0/1281213439 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d80001370 con 0x7f2d740039e0 2024-03-24T01:32:44.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.963+0000 7f2d8d400640 1 -- 172.21.15.90:0/1281213439 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d800052d0 con 0x7f2d740039e0 2024-03-24T01:32:44.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.963+0000 7f2d8d400640 1 -- 172.21.15.90:0/1281213439 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d80003830 con 0x7f2d740039e0 2024-03-24T01:32:44.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.964+0000 7f2d95000640 1 -- 172.21.15.90:0/1281213439 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d740039e0 msgr2=0x7f2d74003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:44.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.964+0000 7f2d95000640 1 --2- 172.21.15.90:0/1281213439 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d740039e0 0x7f2d74003e00 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f2d80001720 tx=0x7f2d74005830 comp rx=0 tx=0).stop 2024-03-24T01:32:44.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.964+0000 7f2d8e800640 1 -- 172.21.15.90:0/1281213439 reap_dead start 2024-03-24T01:32:44.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.964+0000 7f2d95000640 1 -- 172.21.15.90:0/1281213439 shutdown_connections 2024-03-24T01:32:44.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.964+0000 7f2d95000640 1 -- 172.21.15.90:0/1281213439 >> 172.21.15.90:0/1281213439 conn(0x7f2d900b1770 msgr2=0x7f2d900b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:44.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.964+0000 7f2d95000640 1 -- 172.21.15.90:0/1281213439 shutdown_connections 2024-03-24T01:32:44.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.964+0000 7f2d95000640 1 -- 172.21.15.90:0/1281213439 wait complete. 2024-03-24T01:32:44.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.966+0000 7f2d95000640 1 Processor -- start 2024-03-24T01:32:44.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.966+0000 7f2d95000640 1 -- start start 2024-03-24T01:32:44.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.967+0000 7f2d95000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d740039e0 0x7f2d90145fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:44.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.967+0000 7f2d95000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d901467b0 0x7f2d90149c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:44.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.967+0000 7f2d95000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d9015c160 0x7f2d9015c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:44.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.967+0000 7f2d95000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d90155410 con 0x7f2d901467b0 2024-03-24T01:32:44.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.967+0000 7f2d95000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9015bc00 con 0x7f2d740039e0 2024-03-24T01:32:44.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.967+0000 7f2d95000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9015af70 con 0x7f2d9015c160 2024-03-24T01:32:44.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.967+0000 7f2d85400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d901467b0 0x7f2d90149c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:44.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.967+0000 7f2d85400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d901467b0 0x7f2d90149c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39698/0 (socket says 172.21.15.90:39698) 2024-03-24T01:32:44.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.967+0000 7f2d85400640 1 -- 172.21.15.90:0/1761255128 learned_addr learned my addr 172.21.15.90:0/1761255128 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:44.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.967+0000 7f2d8de00640 1 --2- 172.21.15.90:0/1761255128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d740039e0 0x7f2d90145fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:44.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.968+0000 7f2d8e800640 1 --2- 172.21.15.90:0/1761255128 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d9015c160 0x7f2d9015c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:44.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.968+0000 7f2d85400640 1 -- 172.21.15.90:0/1761255128 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d9015c160 msgr2=0x7f2d9015c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:44.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.968+0000 7f2d85400640 1 --2- 172.21.15.90:0/1761255128 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d9015c160 0x7f2d9015c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:44.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.968+0000 7f2d85400640 1 -- 172.21.15.90:0/1761255128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d740039e0 msgr2=0x7f2d90145fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:44.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.968+0000 7f2d85400640 1 --2- 172.21.15.90:0/1761255128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d740039e0 0x7f2d90145fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:44.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.968+0000 7f2d85400640 1 -- 172.21.15.90:0/1761255128 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d90146510 con 0x7f2d901467b0 2024-03-24T01:32:44.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.968+0000 7f2d8e800640 1 --2- 172.21.15.90:0/1761255128 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d9015c160 0x7f2d9015c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:44.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.968+0000 7f2d85400640 1 --2- 172.21.15.90:0/1761255128 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d901467b0 0x7f2d90149c50 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f2d7800e7c0 tx=0x7f2d7800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:44.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.968+0000 7f2d86a00640 1 -- 172.21.15.90:0/1761255128 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d78017070 con 0x7f2d901467b0 2024-03-24T01:32:44.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.968+0000 7f2d86a00640 1 -- 172.21.15.90:0/1761255128 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d7800c8e0 con 0x7f2d901467b0 2024-03-24T01:32:44.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.968+0000 7f2d86a00640 1 -- 172.21.15.90:0/1761255128 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7800ccb0 con 0x7f2d901467b0 2024-03-24T01:32:44.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.968+0000 7f2d95000640 1 -- 172.21.15.90:0/1761255128 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d90158510 con 0x7f2d901467b0 2024-03-24T01:32:44.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.968+0000 7f2d95000640 1 -- 172.21.15.90:0/1761255128 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d90158230 con 0x7f2d901467b0 2024-03-24T01:32:44.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.970+0000 7f2d95000640 1 -- 172.21.15.90:0/1761255128 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d90068060 con 0x7f2d901467b0 2024-03-24T01:32:44.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.971+0000 7f2d86a00640 1 -- 172.21.15.90:0/1761255128 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2d7801a070 con 0x7f2d901467b0 2024-03-24T01:32:44.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.971+0000 7f2d86a00640 1 --2- 172.21.15.90:0/1761255128 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f2d6c06bc00 0x7f2d6c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:44.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.971+0000 7f2d86a00640 1 -- 172.21.15.90:0/1761255128 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2d7808a980 con 0x7f2d901467b0 2024-03-24T01:32:44.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.972+0000 7f2d8de00640 1 --2- 172.21.15.90:0/1761255128 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f2d6c06bc00 0x7f2d6c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:44.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.973+0000 7f2d8de00640 1 --2- 172.21.15.90:0/1761255128 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f2d6c06bc00 0x7f2d6c06e0c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f2d80033000 tx=0x7f2d80009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:44.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:44.976+0000 7f2d86a00640 1 -- 172.21.15.90:0/1761255128 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2d780592d0 con 0x7f2d901467b0 2024-03-24T01:32:45.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.178+0000 7f2d95000640 1 -- 172.21.15.90:0/1761255128 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d900b3540 con 0x7f2d901467b0 2024-03-24T01:32:45.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.181+0000 7f2d86a00640 1 -- 172.21.15.90:0/1761255128 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+16873 (secure 0 0 0) 0x7f2d78058bd0 con 0x7f2d901467b0 2024-03-24T01:32:45.182 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:32:45.182 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":66,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:32:44.150227+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:32:42.026147+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-24T01:32:43.180712+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:32:45.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.184+0000 7f2d95000640 1 -- 172.21.15.90:0/1761255128 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f2d6c06bc00 msgr2=0x7f2d6c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:45.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.184+0000 7f2d95000640 1 --2- 172.21.15.90:0/1761255128 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f2d6c06bc00 0x7f2d6c06e0c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f2d80033000 tx=0x7f2d80009030 comp rx=0 tx=0).stop 2024-03-24T01:32:45.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.184+0000 7f2d95000640 1 -- 172.21.15.90:0/1761255128 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d901467b0 msgr2=0x7f2d90149c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:45.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.184+0000 7f2d95000640 1 --2- 172.21.15.90:0/1761255128 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d901467b0 0x7f2d90149c50 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f2d7800e7c0 tx=0x7f2d7800ec80 comp rx=0 tx=0).stop 2024-03-24T01:32:45.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.184+0000 7f2d95000640 1 -- 172.21.15.90:0/1761255128 shutdown_connections 2024-03-24T01:32:45.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.184+0000 7f2d95000640 1 --2- 172.21.15.90:0/1761255128 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f2d6c06bc00 0x7f2d6c06e0c0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:45.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.184+0000 7f2d95000640 1 --2- 172.21.15.90:0/1761255128 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d901467b0 0x7f2d90149c50 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:45.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.184+0000 7f2d95000640 1 --2- 172.21.15.90:0/1761255128 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f2d9015c160 0x7f2d9015c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:45.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.184+0000 7f2d95000640 1 --2- 172.21.15.90:0/1761255128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d740039e0 0x7f2d90145fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:45.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.184+0000 7f2d95000640 1 -- 172.21.15.90:0/1761255128 >> 172.21.15.90:0/1761255128 conn(0x7f2d900b1770 msgr2=0x7f2d900ce200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:45.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.184+0000 7f2d95000640 1 -- 172.21.15.90:0/1761255128 shutdown_connections 2024-03-24T01:32:45.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.184+0000 7f2d95000640 1 -- 172.21.15.90:0/1761255128 wait complete. 2024-03-24T01:32:45.203 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:32:45.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.413+0000 7f6e74800640 1 Processor -- start 2024-03-24T01:32:45.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.413+0000 7f6e74800640 1 -- start start 2024-03-24T01:32:45.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.415+0000 7f6e74800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e6c1443d0 0x7f6e6c0cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:45.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.415+0000 7f6e74800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6e6c0b3310 0x7f6e6c0cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:45.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.415+0000 7f6e74800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e6c1447b0 con 0x7f6e6c0cec90 2024-03-24T01:32:45.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.415+0000 7f6e74800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e6c15aef0 con 0x7f6e6c0b3310 2024-03-24T01:32:45.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.415+0000 7f6e74800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c15b110 con 0x7f6e6c1443d0 2024-03-24T01:32:45.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.415+0000 7f6e71e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e6c1443d0 0x7f6e6c0cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:45.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.415+0000 7f6e71400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6e6c0b3310 0x7f6e6c0cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:45.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.415+0000 7f6e71e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e6c1443d0 0x7f6e6c0cceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.90:54560/0 (socket says 172.21.15.90:54560) 2024-03-24T01:32:45.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.416+0000 7f6e71e00640 1 -- 172.21.15.90:0/1418349811 learned_addr learned my addr 172.21.15.90:0/1418349811 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:45.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.416+0000 7f6e71400640 1 -- 172.21.15.90:0/1418349811 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e6c1443d0 msgr2=0x7f6e6c0cceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:45.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.416+0000 7f6e71400640 1 --2- 172.21.15.90:0/1418349811 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e6c1443d0 0x7f6e6c0cceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:45.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.416+0000 7f6e71400640 1 -- 172.21.15.90:0/1418349811 >> v1:172.21.15.120:6789/0 conn(0x7f6e6c0cec90 legacy=0x7f6e6c0cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:45.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.416+0000 7f6e71400640 1 -- 172.21.15.90:0/1418349811 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6c15b3f0 con 0x7f6e6c0b3310 2024-03-24T01:32:45.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.418+0000 7f6e71400640 1 --2- 172.21.15.90:0/1418349811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e6c0b3310 0x7f6e6c0cd3f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c0097a0 tx=0x7f6e5c02b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eab3c90257d8251b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:45.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.418+0000 7f6e70a00640 1 -- 172.21.15.90:0/1418349811 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c031780 con 0x7f6e6c0b3310 2024-03-24T01:32:45.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.418+0000 7f6e70a00640 1 -- 172.21.15.90:0/1418349811 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e5c037070 con 0x7f6e6c0b3310 2024-03-24T01:32:45.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.418+0000 7f6e70a00640 1 -- 172.21.15.90:0/1418349811 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c031b40 con 0x7f6e6c0b3310 2024-03-24T01:32:45.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.419+0000 7f6e74800640 1 -- 172.21.15.90:0/1418349811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e6c0b3310 msgr2=0x7f6e6c0cd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:45.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.419+0000 7f6e74800640 1 --2- 172.21.15.90:0/1418349811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e6c0b3310 0x7f6e6c0cd3f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c0097a0 tx=0x7f6e5c02b820 comp rx=0 tx=0).stop 2024-03-24T01:32:45.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.419+0000 7f6e74800640 1 -- 172.21.15.90:0/1418349811 shutdown_connections 2024-03-24T01:32:45.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.419+0000 7f6e74800640 1 --2- 172.21.15.90:0/1418349811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e6c0b3310 0x7f6e6c0cd3f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:45.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.419+0000 7f6e74800640 1 --2- 172.21.15.90:0/1418349811 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e6c1443d0 0x7f6e6c0cceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:45.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.419+0000 7f6e74800640 1 -- 172.21.15.90:0/1418349811 >> 172.21.15.90:0/1418349811 conn(0x7f6e6c06b2a0 msgr2=0x7f6e6c06b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:45.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.419+0000 7f6e74800640 1 -- 172.21.15.90:0/1418349811 shutdown_connections 2024-03-24T01:32:45.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.419+0000 7f6e74800640 1 -- 172.21.15.90:0/1418349811 wait complete. 2024-03-24T01:32:45.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.421+0000 7f6e74800640 1 Processor -- start 2024-03-24T01:32:45.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.421+0000 7f6e74800640 1 -- start start 2024-03-24T01:32:45.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.422+0000 7f6e74800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e6c0b3310 0x7f6e6c0c5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:45.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.422+0000 7f6e74800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e6c0cec90 0x7f6e6c0ad320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:45.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.422+0000 7f6e74800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e6c1443d0 0x7f6e6c1f4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:45.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.422+0000 7f6e74800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c15bc90 con 0x7f6e6c1443d0 2024-03-24T01:32:45.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.422+0000 7f6e74800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c15b9b0 con 0x7f6e6c0cec90 2024-03-24T01:32:45.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.422+0000 7f6e74800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c15e610 con 0x7f6e6c0b3310 2024-03-24T01:32:45.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.422+0000 7f6e72800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e6c1443d0 0x7f6e6c1f4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:45.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.422+0000 7f6e72800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e6c1443d0 0x7f6e6c1f4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39702/0 (socket says 172.21.15.90:39702) 2024-03-24T01:32:45.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.422+0000 7f6e72800640 1 -- 172.21.15.90:0/2513230529 learned_addr learned my addr 172.21.15.90:0/2513230529 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:45.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.422+0000 7f6e71e00640 1 --2- 172.21.15.90:0/2513230529 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e6c0b3310 0x7f6e6c0c5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:45.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.422+0000 7f6e71400640 1 --2- 172.21.15.90:0/2513230529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e6c0cec90 0x7f6e6c0ad320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:45.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.423+0000 7f6e72800640 1 -- 172.21.15.90:0/2513230529 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e6c0b3310 msgr2=0x7f6e6c0c5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:45.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.423+0000 7f6e72800640 1 --2- 172.21.15.90:0/2513230529 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e6c0b3310 0x7f6e6c0c5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:45.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.423+0000 7f6e72800640 1 -- 172.21.15.90:0/2513230529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e6c0cec90 msgr2=0x7f6e6c0ad320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:45.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.423+0000 7f6e72800640 1 --2- 172.21.15.90:0/2513230529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e6c0cec90 0x7f6e6c0ad320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:45.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.423+0000 7f6e72800640 1 -- 172.21.15.90:0/2513230529 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6c0adb50 con 0x7f6e6c1443d0 2024-03-24T01:32:45.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.423+0000 7f6e72800640 1 --2- 172.21.15.90:0/2513230529 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e6c1443d0 0x7f6e6c1f4050 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f6e6800c7a0 tx=0x7f6e6800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:45.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.423+0000 7f6e66000640 1 -- 172.21.15.90:0/2513230529 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e68003b70 con 0x7f6e6c1443d0 2024-03-24T01:32:45.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.423+0000 7f6e66000640 1 -- 172.21.15.90:0/2513230529 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e68003d30 con 0x7f6e6c1443d0 2024-03-24T01:32:45.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.423+0000 7f6e66000640 1 -- 172.21.15.90:0/2513230529 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6800aa90 con 0x7f6e6c1443d0 2024-03-24T01:32:45.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.423+0000 7f6e74800640 1 -- 172.21.15.90:0/2513230529 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e6c0ade30 con 0x7f6e6c1443d0 2024-03-24T01:32:45.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.424+0000 7f6e74800640 1 -- 172.21.15.90:0/2513230529 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e6c0ad860 con 0x7f6e6c1443d0 2024-03-24T01:32:45.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.425+0000 7f6e74800640 1 -- 172.21.15.90:0/2513230529 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e6c073570 con 0x7f6e6c1443d0 2024-03-24T01:32:45.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.426+0000 7f6e66000640 1 -- 172.21.15.90:0/2513230529 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f6e6800ac30 con 0x7f6e6c1443d0 2024-03-24T01:32:45.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.426+0000 7f6e66000640 1 --2- 172.21.15.90:0/2513230529 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6e5006bbd0 0x7f6e5006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:45.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.427+0000 7f6e66000640 1 -- 172.21.15.90:0/2513230529 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6e6808ab10 con 0x7f6e6c1443d0 2024-03-24T01:32:45.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.427+0000 7f6e71e00640 1 --2- 172.21.15.90:0/2513230529 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6e5006bbd0 0x7f6e5006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:45.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.428+0000 7f6e71e00640 1 --2- 172.21.15.90:0/2513230529 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6e5006bbd0 0x7f6e5006e090 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c0cdf50 tx=0x7f6e6000b450 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:45.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.431+0000 7f6e66000640 1 -- 172.21.15.90:0/2513230529 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f6e680593e0 con 0x7f6e6c1443d0 2024-03-24T01:32:45.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.632+0000 7f6e74800640 1 -- 172.21.15.90:0/2513230529 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e6c04f4e0 con 0x7f6e6c1443d0 2024-03-24T01:32:45.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.636+0000 7f6e66000640 1 -- 172.21.15.90:0/2513230529 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+16873 (secure 0 0 0) 0x7f6e68058ce0 con 0x7f6e6c1443d0 2024-03-24T01:32:45.637 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:32:45.637 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":66,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:32:44.150227+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:32:42.026147+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-24T01:32:43.180712+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:32:45.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.638+0000 7f6e74800640 1 -- 172.21.15.90:0/2513230529 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6e5006bbd0 msgr2=0x7f6e5006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:45.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.638+0000 7f6e74800640 1 --2- 172.21.15.90:0/2513230529 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6e5006bbd0 0x7f6e5006e090 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c0cdf50 tx=0x7f6e6000b450 comp rx=0 tx=0).stop 2024-03-24T01:32:45.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.638+0000 7f6e74800640 1 -- 172.21.15.90:0/2513230529 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e6c1443d0 msgr2=0x7f6e6c1f4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:45.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.638+0000 7f6e74800640 1 --2- 172.21.15.90:0/2513230529 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e6c1443d0 0x7f6e6c1f4050 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f6e6800c7a0 tx=0x7f6e6800cc60 comp rx=0 tx=0).stop 2024-03-24T01:32:45.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.638+0000 7f6e74800640 1 -- 172.21.15.90:0/2513230529 shutdown_connections 2024-03-24T01:32:45.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.638+0000 7f6e74800640 1 --2- 172.21.15.90:0/2513230529 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f6e5006bbd0 0x7f6e5006e090 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:45.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.638+0000 7f6e74800640 1 --2- 172.21.15.90:0/2513230529 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e6c1443d0 0x7f6e6c1f4050 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:45.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.638+0000 7f6e74800640 1 --2- 172.21.15.90:0/2513230529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e6c0cec90 0x7f6e6c0ad320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:45.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.638+0000 7f6e74800640 1 --2- 172.21.15.90:0/2513230529 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6e6c0b3310 0x7f6e6c0c5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:45.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.638+0000 7f6e74800640 1 -- 172.21.15.90:0/2513230529 >> 172.21.15.90:0/2513230529 conn(0x7f6e6c06b2a0 msgr2=0x7f6e6c142c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:45.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.639+0000 7f6e74800640 1 -- 172.21.15.90:0/2513230529 shutdown_connections 2024-03-24T01:32:45.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.639+0000 7f6e74800640 1 -- 172.21.15.90:0/2513230529 wait complete. 2024-03-24T01:32:45.659 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-24T01:32:45.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.868+0000 7fb87ba00640 1 Processor -- start 2024-03-24T01:32:45.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.869+0000 7fb87ba00640 1 -- start start 2024-03-24T01:32:45.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.870+0000 7fb87ba00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb87415bc50 0x7fb87406b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:45.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.870+0000 7fb87ba00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb87406b600 0x7fb8740b2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:45.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.870+0000 7fb87ba00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb87406dac0 con 0x7fb8740aeeb0 2024-03-24T01:32:45.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.870+0000 7fb87ba00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8740b4da0 con 0x7fb87415bc50 2024-03-24T01:32:45.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.870+0000 7fb87ba00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb87406ba40 con 0x7fb87406b600 2024-03-24T01:32:45.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.870+0000 7fb873e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb87415bc50 0x7fb87406b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:45.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.870+0000 7fb873e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb87415bc50 0x7fb87406b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39708/0 (socket says 172.21.15.90:39708) 2024-03-24T01:32:45.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.870+0000 7fb873e00640 1 -- 172.21.15.90:0/997304588 learned_addr learned my addr 172.21.15.90:0/997304588 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:45.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.871+0000 7fb879a00640 1 --2- 172.21.15.90:0/997304588 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb87406b600 0x7fb8740b2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:45.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.871+0000 7fb873e00640 1 -- 172.21.15.90:0/997304588 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb87406b600 msgr2=0x7fb8740b2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:45.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.871+0000 7fb873e00640 1 --2- 172.21.15.90:0/997304588 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb87406b600 0x7fb8740b2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:45.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.871+0000 7fb873e00640 1 -- 172.21.15.90:0/997304588 >> v1:172.21.15.120:6789/0 conn(0x7fb8740aeeb0 legacy=0x7fb8740b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:45.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.871+0000 7fb873e00640 1 -- 172.21.15.90:0/997304588 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87406dda0 con 0x7fb87415bc50 2024-03-24T01:32:45.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.871+0000 7fb879a00640 1 --2- 172.21.15.90:0/997304588 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb87406b600 0x7fb8740b2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:45.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.872+0000 7fb873e00640 1 --2- 172.21.15.90:0/997304588 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb87415bc50 0x7fb87406b0c0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fb86000bb10 tx=0x7fb86002dfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea1851529ca87b9b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:45.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.872+0000 7fb873400640 1 -- 172.21.15.90:0/997304588 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb86003b070 con 0x7fb87415bc50 2024-03-24T01:32:45.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.872+0000 7fb873400640 1 -- 172.21.15.90:0/997304588 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb86002ed50 con 0x7fb87415bc50 2024-03-24T01:32:45.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.872+0000 7fb873400640 1 -- 172.21.15.90:0/997304588 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb860036660 con 0x7fb87415bc50 2024-03-24T01:32:45.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.873+0000 7fb87ba00640 1 -- 172.21.15.90:0/997304588 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb87415bc50 msgr2=0x7fb87406b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:45.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.873+0000 7fb87ba00640 1 --2- 172.21.15.90:0/997304588 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb87415bc50 0x7fb87406b0c0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fb86000bb10 tx=0x7fb86002dfa0 comp rx=0 tx=0).stop 2024-03-24T01:32:45.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.873+0000 7fb87ba00640 1 -- 172.21.15.90:0/997304588 shutdown_connections 2024-03-24T01:32:45.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.873+0000 7fb87ba00640 1 --2- 172.21.15.90:0/997304588 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb87406b600 0x7fb8740b2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:45.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.873+0000 7fb87ba00640 1 --2- 172.21.15.90:0/997304588 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb87415bc50 0x7fb87406b0c0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:45.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.873+0000 7fb87ba00640 1 -- 172.21.15.90:0/997304588 >> 172.21.15.90:0/997304588 conn(0x7fb8740b1770 msgr2=0x7fb8740b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:45.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.873+0000 7fb87ba00640 1 -- 172.21.15.90:0/997304588 shutdown_connections 2024-03-24T01:32:45.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.873+0000 7fb87ba00640 1 -- 172.21.15.90:0/997304588 wait complete. 2024-03-24T01:32:45.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.875+0000 7fb87ba00640 1 Processor -- start 2024-03-24T01:32:45.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.876+0000 7fb87ba00640 1 -- start start 2024-03-24T01:32:45.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.877+0000 7fb87ba00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb87406b600 0x7fb87414b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:45.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.877+0000 7fb87ba00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8740aeeb0 0x7fb8740b3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:45.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.877+0000 7fb87ba00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb87415bc50 0x7fb8740b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:45.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.877+0000 7fb87ba00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8740af290 con 0x7fb87406b600 2024-03-24T01:32:45.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.877+0000 7fb87ba00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87406e360 con 0x7fb8740aeeb0 2024-03-24T01:32:45.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.877+0000 7fb87ba00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb87406e880 con 0x7fb87415bc50 2024-03-24T01:32:45.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.877+0000 7fb879000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb87406b600 0x7fb87414b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:45.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.877+0000 7fb879000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb87406b600 0x7fb87414b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39718/0 (socket says 172.21.15.90:39718) 2024-03-24T01:32:45.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.877+0000 7fb879000640 1 -- 172.21.15.90:0/2166151597 learned_addr learned my addr 172.21.15.90:0/2166151597 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:45.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.877+0000 7fb879a00640 1 --2- 172.21.15.90:0/2166151597 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb87415bc50 0x7fb8740b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:45.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.877+0000 7fb879000640 1 -- 172.21.15.90:0/2166151597 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb87415bc50 msgr2=0x7fb8740b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:45.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.877+0000 7fb873e00640 1 --2- 172.21.15.90:0/2166151597 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8740aeeb0 0x7fb8740b3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:45.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.877+0000 7fb879000640 1 --2- 172.21.15.90:0/2166151597 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb87415bc50 0x7fb8740b41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:45.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.877+0000 7fb879000640 1 -- 172.21.15.90:0/2166151597 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8740aeeb0 msgr2=0x7fb8740b3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:45.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.877+0000 7fb879000640 1 --2- 172.21.15.90:0/2166151597 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8740aeeb0 0x7fb8740b3c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:45.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.877+0000 7fb879000640 1 -- 172.21.15.90:0/2166151597 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87414bf30 con 0x7fb87406b600 2024-03-24T01:32:45.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.878+0000 7fb879000640 1 --2- 172.21.15.90:0/2166151597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb87406b600 0x7fb87414b020 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fb86800b420 tx=0x7fb86800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:45.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.878+0000 7fb870c00640 1 -- 172.21.15.90:0/2166151597 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb868011830 con 0x7fb87406b600 2024-03-24T01:32:45.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.878+0000 7fb870c00640 1 -- 172.21.15.90:0/2166151597 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb868016070 con 0x7fb87406b600 2024-03-24T01:32:45.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.878+0000 7fb870c00640 1 -- 172.21.15.90:0/2166151597 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb868011c10 con 0x7fb87406b600 2024-03-24T01:32:45.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.878+0000 7fb87ba00640 1 -- 172.21.15.90:0/2166151597 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb87414b970 con 0x7fb87406b600 2024-03-24T01:32:45.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.878+0000 7fb87ba00640 1 -- 172.21.15.90:0/2166151597 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb874149030 con 0x7fb87406b600 2024-03-24T01:32:45.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.879+0000 7fb87ba00640 1 -- 172.21.15.90:0/2166151597 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb87404f400 con 0x7fb87406b600 2024-03-24T01:32:45.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.880+0000 7fb870c00640 1 -- 172.21.15.90:0/2166151597 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb86801b460 con 0x7fb87406b600 2024-03-24T01:32:45.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.881+0000 7fb870c00640 1 --2- 172.21.15.90:0/2166151597 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb85406bbd0 0x7fb85406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:45.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.881+0000 7fb870c00640 1 -- 172.21.15.90:0/2166151597 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb86808a800 con 0x7fb87406b600 2024-03-24T01:32:45.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.882+0000 7fb873e00640 1 --2- 172.21.15.90:0/2166151597 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb85406bbd0 0x7fb85406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:45.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.883+0000 7fb873e00640 1 --2- 172.21.15.90:0/2166151597 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb85406bbd0 0x7fb85406e090 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fb860038000 tx=0x7fb860008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:45.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:45.886+0000 7fb870c00640 1 -- 172.21.15.90:0/2166151597 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb868020020 con 0x7fb87406b600 2024-03-24T01:32:46.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:46.129+0000 7fb87ba00640 1 -- 172.21.15.90:0/2166151597 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fb87406c970 con 0x7fb87406b600 2024-03-24T01:32:46.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:46.768+0000 7fb870c00640 1 -- 172.21.15.90:0/2166151597 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v49) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb8680590c0 con 0x7fb87406b600 2024-03-24T01:32:46.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:46.771+0000 7fb87ba00640 1 -- 172.21.15.90:0/2166151597 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb85406bbd0 msgr2=0x7fb85406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:46.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:46.771+0000 7fb87ba00640 1 --2- 172.21.15.90:0/2166151597 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb85406bbd0 0x7fb85406e090 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fb860038000 tx=0x7fb860008040 comp rx=0 tx=0).stop 2024-03-24T01:32:46.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:46.771+0000 7fb87ba00640 1 -- 172.21.15.90:0/2166151597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb87406b600 msgr2=0x7fb87414b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:46.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:46.771+0000 7fb87ba00640 1 --2- 172.21.15.90:0/2166151597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb87406b600 0x7fb87414b020 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fb86800b420 tx=0x7fb86800b8e0 comp rx=0 tx=0).stop 2024-03-24T01:32:46.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:46.771+0000 7fb87ba00640 1 -- 172.21.15.90:0/2166151597 shutdown_connections 2024-03-24T01:32:46.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:46.771+0000 7fb87ba00640 1 --2- 172.21.15.90:0/2166151597 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb85406bbd0 0x7fb85406e090 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:46.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:46.771+0000 7fb87ba00640 1 --2- 172.21.15.90:0/2166151597 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb87415bc50 0x7fb8740b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:46.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:46.771+0000 7fb87ba00640 1 --2- 172.21.15.90:0/2166151597 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8740aeeb0 0x7fb8740b3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:46.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:46.771+0000 7fb87ba00640 1 --2- 172.21.15.90:0/2166151597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb87406b600 0x7fb87414b020 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:46.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:46.771+0000 7fb87ba00640 1 -- 172.21.15.90:0/2166151597 >> 172.21.15.90:0/2166151597 conn(0x7fb8740b1770 msgr2=0x7fb8740ce810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:46.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:46.772+0000 7fb87ba00640 1 -- 172.21.15.90:0/2166151597 shutdown_connections 2024-03-24T01:32:46.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:46.772+0000 7fb87ba00640 1 -- 172.21.15.90:0/2166151597 wait complete. 2024-03-24T01:32:46.792 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-24T01:32:46.793 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-24T01:32:47.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.006+0000 7fdafa800640 1 Processor -- start 2024-03-24T01:32:47.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.006+0000 7fdafa800640 1 -- start start 2024-03-24T01:32:47.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.007+0000 7fdafa800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fdaf41549f0 0x7fdaf4154e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:47.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.007+0000 7fdafa800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaf41553a0 con 0x7fdaf406f0d0 2024-03-24T01:32:47.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.007+0000 7fdafa800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaf415b9d0 con 0x7fdaf406ecf0 2024-03-24T01:32:47.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.007+0000 7fdafa800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaf415bcb0 con 0x7fdaf41549f0 2024-03-24T01:32:47.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.008+0000 7fdaf8a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fdaf406f0d0 0x7fdaf415af80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40246/0 (socket says 172.21.15.90:40246) 2024-03-24T01:32:47.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.008+0000 7fdaf8a00640 1 -- 172.21.15.90:0/104262931 learned_addr learned my addr 172.21.15.90:0/104262931 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:32:47.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.008+0000 7fdaf9e00640 1 --2- 172.21.15.90:0/104262931 >> v2:172.21.15.120:3300/0 conn(0x7fdaf41549f0 0x7fdaf4154e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:47.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.008+0000 7fdaf3e00640 1 -- 172.21.15.90:0/104262931 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3453643104 0 0) 0x7fdaf41553a0 con 0x7fdaf406f0d0 2024-03-24T01:32:47.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.008+0000 7fdaf3e00640 1 -- 172.21.15.90:0/104262931 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdadc003520 con 0x7fdaf406f0d0 2024-03-24T01:32:47.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.008+0000 7fdaf3e00640 1 -- 172.21.15.90:0/104262931 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3589820401 0 0) 0x7fdadc003520 con 0x7fdaf406f0d0 2024-03-24T01:32:47.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.008+0000 7fdaf3e00640 1 -- 172.21.15.90:0/104262931 >> v2:172.21.15.120:3300/0 conn(0x7fdaf41549f0 msgr2=0x7fdaf4154e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:47.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.008+0000 7fdaf3e00640 1 --2- 172.21.15.90:0/104262931 >> v2:172.21.15.120:3300/0 conn(0x7fdaf41549f0 0x7fdaf4154e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:47.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.008+0000 7fdaf3e00640 1 -- 172.21.15.90:0/104262931 >> v1:172.21.15.120:6789/0 conn(0x7fdaf406ecf0 legacy=0x7fdaf40b3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:47.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.008+0000 7fdaf3e00640 1 -- 172.21.15.90:0/104262931 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf40b2930 con 0x7fdaf406f0d0 2024-03-24T01:32:47.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.008+0000 7fdaf3e00640 1 -- 172.21.15.90:0/104262931 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7fdae0001670 con 0x7fdaf406f0d0 2024-03-24T01:32:47.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.008+0000 7fdaf3e00640 1 -- 172.21.15.90:0/104262931 >> v1:172.21.15.90:6789/0 conn(0x7fdaf406f0d0 legacy=0x7fdaf415af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:47.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.009+0000 7fdaf3e00640 1 --2- 172.21.15.90:0/104262931 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdadc0039b0 0x7fdadc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:47.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.009+0000 7fdaf3e00640 1 --2- 172.21.15.90:0/104262931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdadc004800 0x7fdadc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:47.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.009+0000 7fdaf3e00640 1 --2- 172.21.15.90:0/104262931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdadc0051c0 0x7fdadc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:47.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.009+0000 7fdaf3e00640 1 -- 172.21.15.90:0/104262931 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf415bcb0 con 0x7fdadc0039b0 2024-03-24T01:32:47.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.009+0000 7fdaf3e00640 1 -- 172.21.15.90:0/104262931 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdadc00c060 con 0x7fdadc0051c0 2024-03-24T01:32:47.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.009+0000 7fdaf3e00640 1 -- 172.21.15.90:0/104262931 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdadc00c260 con 0x7fdadc004800 2024-03-24T01:32:47.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.009+0000 7fdaf9400640 1 --2- 172.21.15.90:0/104262931 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdadc0039b0 0x7fdadc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:47.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.009+0000 7fdaf9e00640 1 --2- 172.21.15.90:0/104262931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdadc0051c0 0x7fdadc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:47.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.009+0000 7fdaf8a00640 1 --2- 172.21.15.90:0/104262931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdadc004800 0x7fdadc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:47.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.009+0000 7fdaf9400640 1 -- 172.21.15.90:0/104262931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdadc004800 msgr2=0x7fdadc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:47.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.009+0000 7fdaf9400640 1 --2- 172.21.15.90:0/104262931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdadc004800 0x7fdadc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:47.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.009+0000 7fdaf9400640 1 -- 172.21.15.90:0/104262931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdadc0051c0 msgr2=0x7fdadc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:47.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.009+0000 7fdaf9400640 1 --2- 172.21.15.90:0/104262931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdadc0051c0 0x7fdadc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:47.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.009+0000 7fdaf9400640 1 -- 172.21.15.90:0/104262931 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdadc003520 con 0x7fdadc0039b0 2024-03-24T01:32:47.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.009+0000 7fdaf9e00640 1 -- 172.21.15.90:0/104262931 reap_dead start 2024-03-24T01:32:47.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.009+0000 7fdaf8a00640 1 --2- 172.21.15.90:0/104262931 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdadc004800 0x7fdadc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:47.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.010+0000 7fdaf9400640 1 --2- 172.21.15.90:0/104262931 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdadc0039b0 0x7fdadc003dd0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fdae8001e50 tx=0x7fdadc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48d41a1b16acae1b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:47.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.011+0000 7fdaf3e00640 1 -- 172.21.15.90:0/104262931 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae8001a70 con 0x7fdadc0039b0 2024-03-24T01:32:47.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.011+0000 7fdaf3e00640 1 -- 172.21.15.90:0/104262931 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdae8003b70 con 0x7fdadc0039b0 2024-03-24T01:32:47.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.011+0000 7fdaf3e00640 1 -- 172.21.15.90:0/104262931 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae80050d0 con 0x7fdadc0039b0 2024-03-24T01:32:47.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.011+0000 7fdafa800640 1 -- 172.21.15.90:0/104262931 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdadc0039b0 msgr2=0x7fdadc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:47.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.011+0000 7fdafa800640 1 --2- 172.21.15.90:0/104262931 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdadc0039b0 0x7fdadc003dd0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fdae8001e50 tx=0x7fdadc005830 comp rx=0 tx=0).stop 2024-03-24T01:32:47.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.011+0000 7fdafa800640 1 -- 172.21.15.90:0/104262931 shutdown_connections 2024-03-24T01:32:47.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.011+0000 7fdafa800640 1 --2- 172.21.15.90:0/104262931 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdadc0039b0 0x7fdadc003dd0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:47.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.011+0000 7fdafa800640 1 -- 172.21.15.90:0/104262931 >> 172.21.15.90:0/104262931 conn(0x7fdaf406b0c0 msgr2=0x7fdaf406b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:47.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.011+0000 7fdafa800640 1 -- 172.21.15.90:0/104262931 shutdown_connections 2024-03-24T01:32:47.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.011+0000 7fdafa800640 1 -- 172.21.15.90:0/104262931 wait complete. 2024-03-24T01:32:47.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.013+0000 7fdafa800640 1 Processor -- start 2024-03-24T01:32:47.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.013+0000 7fdafa800640 1 -- start start 2024-03-24T01:32:47.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.014+0000 7fdafa800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdadc0039b0 0x7fdaf4149b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:47.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.014+0000 7fdafa800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdaf414aa70 0x7fdaf4153210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:47.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.014+0000 7fdafa800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdaf414aef0 0x7fdaf4153750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:47.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.014+0000 7fdafa800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf40b4720 con 0x7fdadc0039b0 2024-03-24T01:32:47.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.014+0000 7fdafa800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf40b45a0 con 0x7fdaf414aa70 2024-03-24T01:32:47.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.014+0000 7fdafa800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf4154e60 con 0x7fdaf414aef0 2024-03-24T01:32:47.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.014+0000 7fdaf9400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdadc0039b0 0x7fdaf4149b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:47.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.015+0000 7fdaf9400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdadc0039b0 0x7fdaf4149b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39734/0 (socket says 172.21.15.90:39734) 2024-03-24T01:32:47.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.015+0000 7fdaf9400640 1 -- 172.21.15.90:0/4279460341 learned_addr learned my addr 172.21.15.90:0/4279460341 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:47.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.015+0000 7fdaf8a00640 1 --2- 172.21.15.90:0/4279460341 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdaf414aa70 0x7fdaf4153210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:47.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.015+0000 7fdaf9e00640 1 --2- 172.21.15.90:0/4279460341 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdaf414aef0 0x7fdaf4153750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:47.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.015+0000 7fdaf9400640 1 -- 172.21.15.90:0/4279460341 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdaf414aef0 msgr2=0x7fdaf4153750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:47.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.015+0000 7fdaf9400640 1 --2- 172.21.15.90:0/4279460341 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdaf414aef0 0x7fdaf4153750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:47.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.015+0000 7fdaf9400640 1 -- 172.21.15.90:0/4279460341 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdaf414aa70 msgr2=0x7fdaf4153210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:47.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.015+0000 7fdaf9400640 1 --2- 172.21.15.90:0/4279460341 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdaf414aa70 0x7fdaf4153210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:47.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.015+0000 7fdaf9400640 1 -- 172.21.15.90:0/4279460341 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf414d320 con 0x7fdadc0039b0 2024-03-24T01:32:47.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.015+0000 7fdaf9e00640 1 --2- 172.21.15.90:0/4279460341 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdaf414aef0 0x7fdaf4153750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:47.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.015+0000 7fdaf9400640 1 --2- 172.21.15.90:0/4279460341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdadc0039b0 0x7fdaf4149b70 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fdae8000f80 tx=0x7fdae80014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:47.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.016+0000 7fdaf1600640 1 -- 172.21.15.90:0/4279460341 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae8003970 con 0x7fdadc0039b0 2024-03-24T01:32:47.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.016+0000 7fdaf1600640 1 -- 172.21.15.90:0/4279460341 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdae8006b50 con 0x7fdadc0039b0 2024-03-24T01:32:47.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.016+0000 7fdaf1600640 1 -- 172.21.15.90:0/4279460341 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae8007020 con 0x7fdadc0039b0 2024-03-24T01:32:47.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.016+0000 7fdafa800640 1 -- 172.21.15.90:0/4279460341 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaf414d600 con 0x7fdadc0039b0 2024-03-24T01:32:47.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.016+0000 7fdafa800640 1 -- 172.21.15.90:0/4279460341 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaf414dbc0 con 0x7fdadc0039b0 2024-03-24T01:32:47.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.017+0000 7fdafa800640 1 -- 172.21.15.90:0/4279460341 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaf404f400 con 0x7fdadc0039b0 2024-03-24T01:32:47.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.018+0000 7fdaf1600640 1 -- 172.21.15.90:0/4279460341 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fdae8049060 con 0x7fdadc0039b0 2024-03-24T01:32:47.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.019+0000 7fdaf1600640 1 --2- 172.21.15.90:0/4279460341 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fdad406bbd0 0x7fdad406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:47.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.019+0000 7fdaf1600640 1 -- 172.21.15.90:0/4279460341 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fdae80abfa0 con 0x7fdadc0039b0 2024-03-24T01:32:47.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.019+0000 7fdaf8a00640 1 --2- 172.21.15.90:0/4279460341 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fdad406bbd0 0x7fdad406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:47.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.021+0000 7fdaf8a00640 1 --2- 172.21.15.90:0/4279460341 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fdad406bbd0 0x7fdad406e090 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fdaf41541a0 tx=0x7fdaf415bba0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:47.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.024+0000 7fdaf1600640 1 -- 172.21.15.90:0/4279460341 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fdae8033020 con 0x7fdadc0039b0 2024-03-24T01:32:47.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.266+0000 7fdafa800640 1 -- 172.21.15.90:0/4279460341 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fdaf406b640 con 0x7fdadc0039b0 2024-03-24T01:32:47.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.781+0000 7fdaf1600640 1 -- 172.21.15.90:0/4279460341 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v50) v1 ==== 122+0+0 (secure 0 0 0) 0x7fdae807a990 con 0x7fdadc0039b0 2024-03-24T01:32:47.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.785+0000 7fdafa800640 1 -- 172.21.15.90:0/4279460341 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fdad406bbd0 msgr2=0x7fdad406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:47.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.785+0000 7fdafa800640 1 --2- 172.21.15.90:0/4279460341 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fdad406bbd0 0x7fdad406e090 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fdaf41541a0 tx=0x7fdaf415bba0 comp rx=0 tx=0).stop 2024-03-24T01:32:47.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.785+0000 7fdafa800640 1 -- 172.21.15.90:0/4279460341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdadc0039b0 msgr2=0x7fdaf4149b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:47.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.785+0000 7fdafa800640 1 --2- 172.21.15.90:0/4279460341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdadc0039b0 0x7fdaf4149b70 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fdae8000f80 tx=0x7fdae80014a0 comp rx=0 tx=0).stop 2024-03-24T01:32:47.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.786+0000 7fdafa800640 1 -- 172.21.15.90:0/4279460341 shutdown_connections 2024-03-24T01:32:47.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.786+0000 7fdafa800640 1 --2- 172.21.15.90:0/4279460341 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fdad406bbd0 0x7fdad406e090 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:47.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.786+0000 7fdafa800640 1 --2- 172.21.15.90:0/4279460341 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fdaf414aef0 0x7fdaf4153750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:47.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.786+0000 7fdafa800640 1 --2- 172.21.15.90:0/4279460341 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdaf414aa70 0x7fdaf4153210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:47.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.786+0000 7fdafa800640 1 --2- 172.21.15.90:0/4279460341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdadc0039b0 0x7fdaf4149b70 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:47.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.786+0000 7fdafa800640 1 -- 172.21.15.90:0/4279460341 >> 172.21.15.90:0/4279460341 conn(0x7fdaf406b0c0 msgr2=0x7fdaf406d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:47.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.786+0000 7fdafa800640 1 -- 172.21.15.90:0/4279460341 shutdown_connections 2024-03-24T01:32:47.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:47.786+0000 7fdafa800640 1 -- 172.21.15.90:0/4279460341 wait complete. 2024-03-24T01:32:47.807 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-24T01:32:47.807 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:32:48.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.013+0000 7f987b200640 1 Processor -- start 2024-03-24T01:32:48.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.013+0000 7f987b200640 1 -- start start 2024-03-24T01:32:48.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.015+0000 7f987b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f987c15e170 0x7f987c0b6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:48.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.015+0000 7f987b200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f987c06d100 con 0x7f987c0b5590 2024-03-24T01:32:48.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.015+0000 7f987b200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f987c06d3e0 con 0x7f987c15dd90 2024-03-24T01:32:48.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.015+0000 7f987b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f987c0d1ef0 con 0x7f987c15e170 2024-03-24T01:32:48.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.015+0000 7f987a800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f987c0b5590 0x7f987c0d26f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40254/0 (socket says 172.21.15.90:40254) 2024-03-24T01:32:48.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.015+0000 7f987a800640 1 -- 172.21.15.90:0/2319256181 learned_addr learned my addr 172.21.15.90:0/2319256181 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:32:48.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.016+0000 7f9879400640 1 --2- 172.21.15.90:0/2319256181 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f987c15e170 0x7f987c0b6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:48.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.016+0000 7f9878a00640 1 -- 172.21.15.90:0/2319256181 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1571582921 0 0) 0x7f987c06d100 con 0x7f987c0b5590 2024-03-24T01:32:48.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.016+0000 7f9878a00640 1 -- 172.21.15.90:0/2319256181 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9860003520 con 0x7f987c0b5590 2024-03-24T01:32:48.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.016+0000 7f9878a00640 1 -- 172.21.15.90:0/2319256181 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2267335541 0 0) 0x7f9860003520 con 0x7f987c0b5590 2024-03-24T01:32:48.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.016+0000 7f9878a00640 1 -- 172.21.15.90:0/2319256181 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f987c15e170 msgr2=0x7f987c0b6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:48.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.016+0000 7f9878a00640 1 --2- 172.21.15.90:0/2319256181 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f987c15e170 0x7f987c0b6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:48.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.016+0000 7f9878a00640 1 -- 172.21.15.90:0/2319256181 >> v1:172.21.15.120:6789/0 conn(0x7f987c15dd90 legacy=0x7f987c06c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:48.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.016+0000 7f9878a00640 1 -- 172.21.15.90:0/2319256181 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f987c0ce700 con 0x7f987c0b5590 2024-03-24T01:32:48.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.016+0000 7f9878a00640 1 -- 172.21.15.90:0/2319256181 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f9874001670 con 0x7f987c0b5590 2024-03-24T01:32:48.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.016+0000 7f9878a00640 1 -- 172.21.15.90:0/2319256181 >> v1:172.21.15.90:6789/0 conn(0x7f987c0b5590 legacy=0x7f987c0d26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:48.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.016+0000 7f9878a00640 1 --2- 172.21.15.90:0/2319256181 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98600039e0 0x7f9860003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:48.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.016+0000 7f9878a00640 1 --2- 172.21.15.90:0/2319256181 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9860004800 0x7f9860004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:48.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.016+0000 7f9878a00640 1 --2- 172.21.15.90:0/2319256181 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9860005210 0x7f986000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:48.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.016+0000 7f9878a00640 1 -- 172.21.15.90:0/2319256181 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987c0d1ef0 con 0x7f9860004800 2024-03-24T01:32:48.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.016+0000 7f9878a00640 1 -- 172.21.15.90:0/2319256181 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f986000bf30 con 0x7f98600039e0 2024-03-24T01:32:48.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.016+0000 7f9878a00640 1 -- 172.21.15.90:0/2319256181 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f986000c130 con 0x7f9860005210 2024-03-24T01:32:48.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.017+0000 7f9879400640 1 --2- 172.21.15.90:0/2319256181 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9860004800 0x7f9860004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:48.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.017+0000 7f9879e00640 1 --2- 172.21.15.90:0/2319256181 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98600039e0 0x7f9860003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:48.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.017+0000 7f9879400640 1 -- 172.21.15.90:0/2319256181 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9860005210 msgr2=0x7f986000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:48.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.017+0000 7f9879400640 1 --2- 172.21.15.90:0/2319256181 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f9860005210 0x7f986000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:48.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.017+0000 7f9879400640 1 -- 172.21.15.90:0/2319256181 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98600039e0 msgr2=0x7f9860003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:48.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.017+0000 7f9879400640 1 --2- 172.21.15.90:0/2319256181 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98600039e0 0x7f9860003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:48.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.017+0000 7f9879400640 1 -- 172.21.15.90:0/2319256181 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9860003520 con 0x7f9860004800 2024-03-24T01:32:48.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.018+0000 7f9879400640 1 --2- 172.21.15.90:0/2319256181 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9860004800 0x7f9860004c80 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f986c009f90 tx=0x7f9860005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a18ffe1b62bae49 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:48.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.018+0000 7f9878a00640 1 -- 172.21.15.90:0/2319256181 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986c039070 con 0x7f9860004800 2024-03-24T01:32:48.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.018+0000 7f9878a00640 1 -- 172.21.15.90:0/2319256181 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f986c02cbf0 con 0x7f9860004800 2024-03-24T01:32:48.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.018+0000 7f9878a00640 1 -- 172.21.15.90:0/2319256181 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986c009c10 con 0x7f9860004800 2024-03-24T01:32:48.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.019+0000 7f987b200640 1 -- 172.21.15.90:0/2319256181 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9860004800 msgr2=0x7f9860004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:48.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.019+0000 7f987b200640 1 --2- 172.21.15.90:0/2319256181 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9860004800 0x7f9860004c80 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f986c009f90 tx=0x7f9860005830 comp rx=0 tx=0).stop 2024-03-24T01:32:48.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.019+0000 7f987a800640 1 -- 172.21.15.90:0/2319256181 reap_dead start 2024-03-24T01:32:48.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.019+0000 7f987b200640 1 -- 172.21.15.90:0/2319256181 shutdown_connections 2024-03-24T01:32:48.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.019+0000 7f987b200640 1 -- 172.21.15.90:0/2319256181 >> 172.21.15.90:0/2319256181 conn(0x7f987c0d0770 msgr2=0x7f987c0d0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:48.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.019+0000 7f987b200640 1 -- 172.21.15.90:0/2319256181 shutdown_connections 2024-03-24T01:32:48.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.019+0000 7f987b200640 1 -- 172.21.15.90:0/2319256181 wait complete. 2024-03-24T01:32:48.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.021+0000 7f987b200640 1 Processor -- start 2024-03-24T01:32:48.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.021+0000 7f987b200640 1 -- start start 2024-03-24T01:32:48.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.022+0000 7f987b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f987c150e50 0x7f987c0ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:48.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.022+0000 7f987b200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f987c151230 0x7f987c160180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:48.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.022+0000 7f987b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f987c1606c0 0x7f987c1f6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:48.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.022+0000 7f987b200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987c0b6fc0 con 0x7f987c150e50 2024-03-24T01:32:48.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.022+0000 7f987b200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987c0b6e40 con 0x7f987c151230 2024-03-24T01:32:48.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.022+0000 7f987b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f987c0b7140 con 0x7f987c1606c0 2024-03-24T01:32:48.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.022+0000 7f9879e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f987c150e50 0x7f987c0ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:48.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.022+0000 7f9879e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f987c150e50 0x7f987c0ccad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39754/0 (socket says 172.21.15.90:39754) 2024-03-24T01:32:48.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.022+0000 7f9879e00640 1 -- 172.21.15.90:0/2928513507 learned_addr learned my addr 172.21.15.90:0/2928513507 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:48.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.022+0000 7f9879400640 1 --2- 172.21.15.90:0/2928513507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f987c151230 0x7f987c160180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:48.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.022+0000 7f987a800640 1 --2- 172.21.15.90:0/2928513507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f987c1606c0 0x7f987c1f6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:48.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.023+0000 7f9879e00640 1 -- 172.21.15.90:0/2928513507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f987c1606c0 msgr2=0x7f987c1f6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:48.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.023+0000 7f9879e00640 1 --2- 172.21.15.90:0/2928513507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f987c1606c0 0x7f987c1f6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:48.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.023+0000 7f9879e00640 1 -- 172.21.15.90:0/2928513507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f987c151230 msgr2=0x7f987c160180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:48.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.023+0000 7f9879e00640 1 --2- 172.21.15.90:0/2928513507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f987c151230 0x7f987c160180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:48.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.023+0000 7f9879e00640 1 -- 172.21.15.90:0/2928513507 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f987c0cad80 con 0x7f987c150e50 2024-03-24T01:32:48.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.023+0000 7f987a800640 1 --2- 172.21.15.90:0/2928513507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f987c1606c0 0x7f987c1f6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:48.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.023+0000 7f9879400640 1 --2- 172.21.15.90:0/2928513507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f987c151230 0x7f987c160180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:48.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.023+0000 7f9879e00640 1 --2- 172.21.15.90:0/2928513507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f987c150e50 0x7f987c0ccad0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f986400d570 tx=0x7f986400da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:48.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.023+0000 7f9872000640 1 -- 172.21.15.90:0/2928513507 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9864005480 con 0x7f987c150e50 2024-03-24T01:32:48.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.023+0000 7f9872000640 1 -- 172.21.15.90:0/2928513507 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9864017070 con 0x7f987c150e50 2024-03-24T01:32:48.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.023+0000 7f9872000640 1 -- 172.21.15.90:0/2928513507 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9864003810 con 0x7f987c150e50 2024-03-24T01:32:48.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.023+0000 7f987b200640 1 -- 172.21.15.90:0/2928513507 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f987c0cb060 con 0x7f987c150e50 2024-03-24T01:32:48.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.023+0000 7f987b200640 1 -- 172.21.15.90:0/2928513507 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f987c0cb620 con 0x7f987c150e50 2024-03-24T01:32:48.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.024+0000 7f987b200640 1 -- 172.21.15.90:0/2928513507 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f987c04f400 con 0x7f987c150e50 2024-03-24T01:32:48.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.025+0000 7f9872000640 1 -- 172.21.15.90:0/2928513507 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f98640039b0 con 0x7f987c150e50 2024-03-24T01:32:48.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.026+0000 7f9872000640 1 --2- 172.21.15.90:0/2928513507 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f985806bc00 0x7f985806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:48.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.026+0000 7f9872000640 1 -- 172.21.15.90:0/2928513507 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f986408ab30 con 0x7f987c150e50 2024-03-24T01:32:48.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.027+0000 7f9879400640 1 --2- 172.21.15.90:0/2928513507 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f985806bc00 0x7f985806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:48.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.028+0000 7f9879400640 1 --2- 172.21.15.90:0/2928513507 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f985806bc00 0x7f985806e0c0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f986c036000 tx=0x7f986c02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:48.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.031+0000 7f9872000640 1 -- 172.21.15.90:0/2928513507 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f98640593f0 con 0x7f987c150e50 2024-03-24T01:32:48.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.268+0000 7f987b200640 1 -- 172.21.15.90:0/2928513507 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f987c0d1900 con 0x7f987c150e50 2024-03-24T01:32:48.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.269+0000 7f9872000640 1 -- 172.21.15.90:0/2928513507 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f9864058cf0 con 0x7f987c150e50 2024-03-24T01:32:48.270 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:32:48.270 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":50,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5177,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/3458630824","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3458630824},{"type":"v1","addr":"172.21.15.120:6837","nonce":3458630824}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":5614,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/1995195696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1995195696},{"type":"v1","addr":"172.21.15.90:6835","nonce":1995195696}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":7098,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6834/1760094626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":1760094626},{"type":"v1","addr":"172.21.15.90:6834","nonce":1760094626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T01:32:44.150735+0000","modified":"2024-03-24T01:32:47.268247+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7110},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7110":{"gid":7110,"name":"d","rank":0,"incarnation":46,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6835/910380191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":910380191},{"type":"v1","addr":"172.21.15.120:6835","nonce":910380191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7110,"qdb_cluster":[7110]},"id":4}]} 2024-03-24T01:32:48.270 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 50 2024-03-24T01:32:48.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.271+0000 7f987b200640 1 -- 172.21.15.90:0/2928513507 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f985806bc00 msgr2=0x7f985806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:48.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.272+0000 7f987b200640 1 --2- 172.21.15.90:0/2928513507 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f985806bc00 0x7f985806e0c0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f986c036000 tx=0x7f986c02e000 comp rx=0 tx=0).stop 2024-03-24T01:32:48.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.272+0000 7f987b200640 1 -- 172.21.15.90:0/2928513507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f987c150e50 msgr2=0x7f987c0ccad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:48.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.272+0000 7f987b200640 1 --2- 172.21.15.90:0/2928513507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f987c150e50 0x7f987c0ccad0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f986400d570 tx=0x7f986400da30 comp rx=0 tx=0).stop 2024-03-24T01:32:48.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.272+0000 7f987b200640 1 -- 172.21.15.90:0/2928513507 shutdown_connections 2024-03-24T01:32:48.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.272+0000 7f987b200640 1 --2- 172.21.15.90:0/2928513507 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f985806bc00 0x7f985806e0c0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:48.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.272+0000 7f987b200640 1 --2- 172.21.15.90:0/2928513507 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f987c1606c0 0x7f987c1f6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:48.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.272+0000 7f987b200640 1 --2- 172.21.15.90:0/2928513507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f987c151230 0x7f987c160180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:48.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.272+0000 7f987b200640 1 --2- 172.21.15.90:0/2928513507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f987c150e50 0x7f987c0ccad0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:48.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.272+0000 7f987b200640 1 -- 172.21.15.90:0/2928513507 >> 172.21.15.90:0/2928513507 conn(0x7f987c0d0770 msgr2=0x7f987c15d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:48.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.272+0000 7f987b200640 1 -- 172.21.15.90:0/2928513507 shutdown_connections 2024-03-24T01:32:48.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.272+0000 7f987b200640 1 -- 172.21.15.90:0/2928513507 wait complete. 2024-03-24T01:32:48.290 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:32:48.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.492+0000 7ff640e00640 1 Processor -- start 2024-03-24T01:32:48.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.492+0000 7ff640e00640 1 -- start start 2024-03-24T01:32:48.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.493+0000 7ff640e00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff63c0cd6e0 0x7ff63c0cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:48.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.494+0000 7ff640e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff63c0cc660 con 0x7ff63c0ce190 2024-03-24T01:32:48.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.494+0000 7ff640e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff63c0cc920 con 0x7ff63c1454e0 2024-03-24T01:32:48.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.494+0000 7ff640e00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff63c0ce600 con 0x7ff63c0cd6e0 2024-03-24T01:32:48.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.494+0000 7ff63a800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff63c0ce190 0x7ff63c15c090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40266/0 (socket says 172.21.15.90:40266) 2024-03-24T01:32:48.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.494+0000 7ff63a800640 1 -- 172.21.15.90:0/143277691 learned_addr learned my addr 172.21.15.90:0/143277691 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:32:48.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.494+0000 7ff631400640 1 --2- 172.21.15.90:0/143277691 >> v2:172.21.15.120:3300/0 conn(0x7ff63c0cd6e0 0x7ff63c0cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:48.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.494+0000 7ff639400640 1 -- 172.21.15.90:0/143277691 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 368107611 0 0) 0x7ff63c0cc660 con 0x7ff63c0ce190 2024-03-24T01:32:48.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.494+0000 7ff639400640 1 -- 172.21.15.90:0/143277691 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff620003520 con 0x7ff63c0ce190 2024-03-24T01:32:48.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.495+0000 7ff639400640 1 -- 172.21.15.90:0/143277691 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3415036859 0 0) 0x7ff620003520 con 0x7ff63c0ce190 2024-03-24T01:32:48.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.495+0000 7ff639400640 1 -- 172.21.15.90:0/143277691 >> v2:172.21.15.120:3300/0 conn(0x7ff63c0cd6e0 msgr2=0x7ff63c0cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:48.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.495+0000 7ff639400640 1 --2- 172.21.15.90:0/143277691 >> v2:172.21.15.120:3300/0 conn(0x7ff63c0cd6e0 0x7ff63c0cdc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:48.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.495+0000 7ff639400640 1 -- 172.21.15.90:0/143277691 >> v1:172.21.15.120:6789/0 conn(0x7ff63c1454e0 legacy=0x7ff63c06f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:48.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.495+0000 7ff639400640 1 -- 172.21.15.90:0/143277691 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff63c0ccc00 con 0x7ff63c0ce190 2024-03-24T01:32:48.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.495+0000 7ff639400640 1 -- 172.21.15.90:0/143277691 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7ff6340019f0 con 0x7ff63c0ce190 2024-03-24T01:32:48.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.495+0000 7ff639400640 1 -- 172.21.15.90:0/143277691 >> v1:172.21.15.90:6789/0 conn(0x7ff63c0ce190 legacy=0x7ff63c15c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:48.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.495+0000 7ff639400640 1 --2- 172.21.15.90:0/143277691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff620003a00 0x7ff620003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:48.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.495+0000 7ff639400640 1 --2- 172.21.15.90:0/143277691 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff620004800 0x7ff620004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:48.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.495+0000 7ff639400640 1 --2- 172.21.15.90:0/143277691 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6200051c0 0x7ff62000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:48.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.495+0000 7ff639400640 1 -- 172.21.15.90:0/143277691 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63c0ce600 con 0x7ff620004800 2024-03-24T01:32:48.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.495+0000 7ff639400640 1 -- 172.21.15.90:0/143277691 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62000c0c0 con 0x7ff620003a00 2024-03-24T01:32:48.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.495+0000 7ff639400640 1 -- 172.21.15.90:0/143277691 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff62000c2c0 con 0x7ff6200051c0 2024-03-24T01:32:48.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.495+0000 7ff631400640 1 --2- 172.21.15.90:0/143277691 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff620004800 0x7ff620004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:48.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.495+0000 7ff631400640 1 -- 172.21.15.90:0/143277691 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6200051c0 msgr2=0x7ff62000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:48.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.495+0000 7ff639e00640 1 --2- 172.21.15.90:0/143277691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff620003a00 0x7ff620003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:48.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.495+0000 7ff631400640 1 --2- 172.21.15.90:0/143277691 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff6200051c0 0x7ff62000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:48.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.495+0000 7ff631400640 1 -- 172.21.15.90:0/143277691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff620003a00 msgr2=0x7ff620003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:48.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.496+0000 7ff631400640 1 --2- 172.21.15.90:0/143277691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff620003a00 0x7ff620003e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:48.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.496+0000 7ff63a800640 1 -- 172.21.15.90:0/143277691 reap_dead start 2024-03-24T01:32:48.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.496+0000 7ff631400640 1 -- 172.21.15.90:0/143277691 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff620003520 con 0x7ff620004800 2024-03-24T01:32:48.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.497+0000 7ff631400640 1 --2- 172.21.15.90:0/143277691 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff620004800 0x7ff620004c80 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7ff624009ac0 tx=0x7ff620005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f1ed7d1e75b8c56 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:48.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.497+0000 7ff639400640 1 -- 172.21.15.90:0/143277691 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff624039070 con 0x7ff620004800 2024-03-24T01:32:48.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.497+0000 7ff639400640 1 -- 172.21.15.90:0/143277691 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff62402c6b0 con 0x7ff620004800 2024-03-24T01:32:48.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.497+0000 7ff639400640 1 -- 172.21.15.90:0/143277691 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff624009690 con 0x7ff620004800 2024-03-24T01:32:48.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.497+0000 7ff640e00640 1 -- 172.21.15.90:0/143277691 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff620004800 msgr2=0x7ff620004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:48.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.497+0000 7ff640e00640 1 --2- 172.21.15.90:0/143277691 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff620004800 0x7ff620004c80 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7ff624009ac0 tx=0x7ff620005830 comp rx=0 tx=0).stop 2024-03-24T01:32:48.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.497+0000 7ff640e00640 1 -- 172.21.15.90:0/143277691 shutdown_connections 2024-03-24T01:32:48.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.497+0000 7ff640e00640 1 --2- 172.21.15.90:0/143277691 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff620004800 0x7ff620004c80 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:48.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.497+0000 7ff640e00640 1 -- 172.21.15.90:0/143277691 >> 172.21.15.90:0/143277691 conn(0x7ff63c06b2a0 msgr2=0x7ff63c06b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:48.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.498+0000 7ff640e00640 1 -- 172.21.15.90:0/143277691 shutdown_connections 2024-03-24T01:32:48.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.498+0000 7ff640e00640 1 -- 172.21.15.90:0/143277691 wait complete. 2024-03-24T01:32:48.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.499+0000 7ff640e00640 1 Processor -- start 2024-03-24T01:32:48.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.499+0000 7ff640e00640 1 -- start start 2024-03-24T01:32:48.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.500+0000 7ff640e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff620004800 0x7ff63c153d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:48.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.500+0000 7ff640e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff63c13cf10 0x7ff63c142eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:48.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.500+0000 7ff640e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff63c13d370 0x7ff63c1433f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:48.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.500+0000 7ff640e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63c0cd1c0 con 0x7ff620004800 2024-03-24T01:32:48.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.500+0000 7ff640e00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63c0cdc50 con 0x7ff63c13cf10 2024-03-24T01:32:48.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.500+0000 7ff640e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff63c0cddd0 con 0x7ff63c13d370 2024-03-24T01:32:48.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.501+0000 7ff639e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff620004800 0x7ff63c153d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:48.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.501+0000 7ff639e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff620004800 0x7ff63c153d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39762/0 (socket says 172.21.15.90:39762) 2024-03-24T01:32:48.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.501+0000 7ff639e00640 1 -- 172.21.15.90:0/104252227 learned_addr learned my addr 172.21.15.90:0/104252227 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:48.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.501+0000 7ff631400640 1 --2- 172.21.15.90:0/104252227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff63c13cf10 0x7ff63c142eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:48.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.501+0000 7ff63a800640 1 --2- 172.21.15.90:0/104252227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff63c13d370 0x7ff63c1433f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:48.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.501+0000 7ff639e00640 1 -- 172.21.15.90:0/104252227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff63c13d370 msgr2=0x7ff63c1433f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:48.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.501+0000 7ff639e00640 1 --2- 172.21.15.90:0/104252227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff63c13d370 0x7ff63c1433f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:48.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.501+0000 7ff639e00640 1 -- 172.21.15.90:0/104252227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff63c13cf10 msgr2=0x7ff63c142eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:48.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.501+0000 7ff639e00640 1 --2- 172.21.15.90:0/104252227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff63c13cf10 0x7ff63c142eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:48.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.501+0000 7ff639e00640 1 -- 172.21.15.90:0/104252227 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff63c13fa00 con 0x7ff620004800 2024-03-24T01:32:48.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.501+0000 7ff63a800640 1 --2- 172.21.15.90:0/104252227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff63c13d370 0x7ff63c1433f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:48.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.501+0000 7ff639e00640 1 --2- 172.21.15.90:0/104252227 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff620004800 0x7ff63c153d10 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7ff62c00d4e0 tx=0x7ff62c00d9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:48.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.502+0000 7ff632a00640 1 -- 172.21.15.90:0/104252227 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62c005480 con 0x7ff620004800 2024-03-24T01:32:48.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.502+0000 7ff632a00640 1 -- 172.21.15.90:0/104252227 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff62c017070 con 0x7ff620004800 2024-03-24T01:32:48.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.502+0000 7ff640e00640 1 -- 172.21.15.90:0/104252227 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff63c15b770 con 0x7ff620004800 2024-03-24T01:32:48.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.502+0000 7ff632a00640 1 -- 172.21.15.90:0/104252227 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62c003810 con 0x7ff620004800 2024-03-24T01:32:48.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.502+0000 7ff640e00640 1 -- 172.21.15.90:0/104252227 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff63c13f160 con 0x7ff620004800 2024-03-24T01:32:48.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.503+0000 7ff640e00640 1 -- 172.21.15.90:0/104252227 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff63c073570 con 0x7ff620004800 2024-03-24T01:32:48.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.504+0000 7ff632a00640 1 -- 172.21.15.90:0/104252227 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff62c0039b0 con 0x7ff620004800 2024-03-24T01:32:48.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.505+0000 7ff632a00640 1 --2- 172.21.15.90:0/104252227 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff61806bbd0 0x7ff61806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:48.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.505+0000 7ff632a00640 1 -- 172.21.15.90:0/104252227 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff62c08a9b0 con 0x7ff620004800 2024-03-24T01:32:48.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.505+0000 7ff631400640 1 --2- 172.21.15.90:0/104252227 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff61806bbd0 0x7ff61806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:48.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.506+0000 7ff631400640 1 --2- 172.21.15.90:0/104252227 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff61806bbd0 0x7ff61806e090 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ff624036000 tx=0x7ff624030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:48.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.509+0000 7ff632a00640 1 -- 172.21.15.90:0/104252227 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff62c059330 con 0x7ff620004800 2024-03-24T01:32:48.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.706+0000 7ff640e00640 1 -- 172.21.15.90:0/104252227 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff63c0cbfb0 con 0x7ff620004800 2024-03-24T01:32:48.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.709+0000 7ff632a00640 1 -- 172.21.15.90:0/104252227 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+16860 (secure 0 0 0) 0x7ff62c058c30 con 0x7ff620004800 2024-03-24T01:32:48.710 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:32:48.710 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":67,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:32:45.707597+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:32:42.026147+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-24T01:32:43.180712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:32:48.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.711+0000 7ff640e00640 1 -- 172.21.15.90:0/104252227 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff61806bbd0 msgr2=0x7ff61806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:48.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.711+0000 7ff640e00640 1 --2- 172.21.15.90:0/104252227 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff61806bbd0 0x7ff61806e090 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ff624036000 tx=0x7ff624030000 comp rx=0 tx=0).stop 2024-03-24T01:32:48.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.712+0000 7ff640e00640 1 -- 172.21.15.90:0/104252227 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff620004800 msgr2=0x7ff63c153d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:48.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.712+0000 7ff640e00640 1 --2- 172.21.15.90:0/104252227 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff620004800 0x7ff63c153d10 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7ff62c00d4e0 tx=0x7ff62c00d9a0 comp rx=0 tx=0).stop 2024-03-24T01:32:48.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.712+0000 7ff640e00640 1 -- 172.21.15.90:0/104252227 shutdown_connections 2024-03-24T01:32:48.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.712+0000 7ff640e00640 1 --2- 172.21.15.90:0/104252227 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff61806bbd0 0x7ff61806e090 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:48.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.712+0000 7ff640e00640 1 --2- 172.21.15.90:0/104252227 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff63c13d370 0x7ff63c1433f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:48.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.712+0000 7ff640e00640 1 --2- 172.21.15.90:0/104252227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff63c13cf10 0x7ff63c142eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:48.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.712+0000 7ff640e00640 1 --2- 172.21.15.90:0/104252227 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff620004800 0x7ff63c153d10 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:48.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.712+0000 7ff640e00640 1 -- 172.21.15.90:0/104252227 >> 172.21.15.90:0/104252227 conn(0x7ff63c06b2a0 msgr2=0x7ff63c0acda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:48.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.712+0000 7ff640e00640 1 -- 172.21.15.90:0/104252227 shutdown_connections 2024-03-24T01:32:48.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.712+0000 7ff640e00640 1 -- 172.21.15.90:0/104252227 wait complete. 2024-03-24T01:32:48.732 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-24T01:32:48.732 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T01:32:48.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.936+0000 7f8389a00640 1 Processor -- start 2024-03-24T01:32:48.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.936+0000 7f8389a00640 1 -- start start 2024-03-24T01:32:48.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.938+0000 7f8389a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f83840cfab0 0x7f838416ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:48.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.938+0000 7f8389a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83840727c0 con 0x7f8384161f40 2024-03-24T01:32:48.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.938+0000 7f8389a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f838406dda0 con 0x7f83840aeac0 2024-03-24T01:32:48.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.938+0000 7f8389a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83840b3690 con 0x7f83840cfab0 2024-03-24T01:32:48.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.938+0000 7f8383e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f8384161f40 0x7f83840720b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:40274/0 (socket says 172.21.15.90:40274) 2024-03-24T01:32:48.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.938+0000 7f8383e00640 1 -- 172.21.15.90:0/1094902677 learned_addr learned my addr 172.21.15.90:0/1094902677 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:32:48.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.939+0000 7f8389000640 1 --2- 172.21.15.90:0/1094902677 >> v2:172.21.15.90:3300/0 conn(0x7f83840cfab0 0x7f838416ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:48.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.939+0000 7f8383400640 1 -- 172.21.15.90:0/1094902677 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1156245777 0 0) 0x7f83840727c0 con 0x7f8384161f40 2024-03-24T01:32:48.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.939+0000 7f8383400640 1 -- 172.21.15.90:0/1094902677 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8368003520 con 0x7f8384161f40 2024-03-24T01:32:48.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.939+0000 7f8389000640 1 -- 172.21.15.90:0/1094902677 >> v1:172.21.15.120:6789/0 conn(0x7f83840aeac0 legacy=0x7f83840cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:48.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.939+0000 7f8389000640 1 -- 172.21.15.90:0/1094902677 >> v1:172.21.15.90:6789/0 conn(0x7f8384161f40 legacy=0x7f83840720b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:48.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.939+0000 7f8389000640 1 -- 172.21.15.90:0/1094902677 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f838406e080 con 0x7f83840cfab0 2024-03-24T01:32:48.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.940+0000 7f8389000640 1 --2- 172.21.15.90:0/1094902677 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83840cfab0 0x7f838416ceb0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f837c009d90 tx=0x7f837c02bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de470586185ee0df server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:48.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.941+0000 7f8383400640 1 -- 172.21.15.90:0/1094902677 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f837c039070 con 0x7f83840cfab0 2024-03-24T01:32:48.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.941+0000 7f8383400640 1 -- 172.21.15.90:0/1094902677 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f837c02ccf0 con 0x7f83840cfab0 2024-03-24T01:32:48.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.941+0000 7f8383400640 1 -- 172.21.15.90:0/1094902677 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f837c034660 con 0x7f83840cfab0 2024-03-24T01:32:48.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.941+0000 7f8389a00640 1 -- 172.21.15.90:0/1094902677 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83840cfab0 msgr2=0x7f838416ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:48.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.941+0000 7f8389a00640 1 --2- 172.21.15.90:0/1094902677 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83840cfab0 0x7f838416ceb0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f837c009d90 tx=0x7f837c02bf40 comp rx=0 tx=0).stop 2024-03-24T01:32:48.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.941+0000 7f8389a00640 1 -- 172.21.15.90:0/1094902677 shutdown_connections 2024-03-24T01:32:48.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.941+0000 7f8389a00640 1 --2- 172.21.15.90:0/1094902677 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83840cfab0 0x7f838416ceb0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:48.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.941+0000 7f8389a00640 1 -- 172.21.15.90:0/1094902677 >> 172.21.15.90:0/1094902677 conn(0x7f838406b0c0 msgr2=0x7f838406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:48.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.941+0000 7f8389a00640 1 -- 172.21.15.90:0/1094902677 shutdown_connections 2024-03-24T01:32:48.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.941+0000 7f8389a00640 1 -- 172.21.15.90:0/1094902677 wait complete. 2024-03-24T01:32:48.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.943+0000 7f8389a00640 1 Processor -- start 2024-03-24T01:32:48.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.943+0000 7f8389a00640 1 -- start start 2024-03-24T01:32:48.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.944+0000 7f8389a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83840aeac0 0x7f838414c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:48.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.944+0000 7f8389a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83840cfab0 0x7f83841488d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:48.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.944+0000 7f8389a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8384161f40 0x7f8384159050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:48.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.944+0000 7f8389a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838406e920 con 0x7f83840aeac0 2024-03-24T01:32:48.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.944+0000 7f8389a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838406e640 con 0x7f83840cfab0 2024-03-24T01:32:48.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.944+0000 7f8389a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8384162320 con 0x7f8384161f40 2024-03-24T01:32:48.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.944+0000 7f8383e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83840aeac0 0x7f838414c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:48.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.944+0000 7f8383e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83840aeac0 0x7f838414c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39776/0 (socket says 172.21.15.90:39776) 2024-03-24T01:32:48.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.944+0000 7f8383e00640 1 -- 172.21.15.90:0/3158381899 learned_addr learned my addr 172.21.15.90:0/3158381899 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:48.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.945+0000 7f837b400640 1 --2- 172.21.15.90:0/3158381899 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83840cfab0 0x7f83841488d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:48.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.945+0000 7f8383e00640 1 -- 172.21.15.90:0/3158381899 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8384161f40 msgr2=0x7f8384159050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:48.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.945+0000 7f8383e00640 1 --2- 172.21.15.90:0/3158381899 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8384161f40 0x7f8384159050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:48.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.945+0000 7f8383e00640 1 -- 172.21.15.90:0/3158381899 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83840cfab0 msgr2=0x7f83841488d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:48.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.945+0000 7f8383e00640 1 --2- 172.21.15.90:0/3158381899 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83840cfab0 0x7f83841488d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:48.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.945+0000 7f8383e00640 1 -- 172.21.15.90:0/3158381899 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8384148e10 con 0x7f83840aeac0 2024-03-24T01:32:48.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.945+0000 7f8383e00640 1 --2- 172.21.15.90:0/3158381899 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83840aeac0 0x7f838414c570 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f837400c410 tx=0x7f837400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:48.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.945+0000 7f8380c00640 1 -- 172.21.15.90:0/3158381899 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8374017070 con 0x7f83840aeac0 2024-03-24T01:32:48.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.945+0000 7f8380c00640 1 -- 172.21.15.90:0/3158381899 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8374005ce0 con 0x7f83840aeac0 2024-03-24T01:32:48.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.945+0000 7f8380c00640 1 -- 172.21.15.90:0/3158381899 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83740072f0 con 0x7f83840aeac0 2024-03-24T01:32:48.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.946+0000 7f8389a00640 1 -- 172.21.15.90:0/3158381899 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8384163d30 con 0x7f83840aeac0 2024-03-24T01:32:48.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.946+0000 7f8389a00640 1 -- 172.21.15.90:0/3158381899 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8384163ff0 con 0x7f83840aeac0 2024-03-24T01:32:48.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.947+0000 7f8389a00640 1 -- 172.21.15.90:0/3158381899 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f838404f400 con 0x7f83840aeac0 2024-03-24T01:32:48.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.948+0000 7f8380c00640 1 -- 172.21.15.90:0/3158381899 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f837401a070 con 0x7f83840aeac0 2024-03-24T01:32:48.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.949+0000 7f8380c00640 1 --2- 172.21.15.90:0/3158381899 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f836006bbd0 0x7f836006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:48.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.949+0000 7f8380c00640 1 -- 172.21.15.90:0/3158381899 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f837408a070 con 0x7f83840aeac0 2024-03-24T01:32:48.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.949+0000 7f837b400640 1 --2- 172.21.15.90:0/3158381899 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f836006bbd0 0x7f836006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:48.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.951+0000 7f837b400640 1 --2- 172.21.15.90:0/3158381899 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f836006bbd0 0x7f836006e090 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f83840cfa50 tx=0x7f836c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:48.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:48.953+0000 7f8380c00640 1 -- 172.21.15.90:0/3158381899 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8384148e10 con 0x7f83840aeac0 2024-03-24T01:32:49.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.148+0000 7f8389a00640 1 -- 172.21.15.90:0/3158381899 --> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8384159c60 con 0x7f836006bbd0 2024-03-24T01:32:49.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.168+0000 7f8380c00640 1 -- 172.21.15.90:0/3158381899 <== mgr.4104 v2:172.21.15.120:6832/860392634 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246711 (secure 0 0 0) 0x7f836c00d450 con 0x7f836006bbd0 2024-03-24T01:32:49.171 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:32:49.173 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-24T01:32:49.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.174+0000 7f8389a00640 1 -- 172.21.15.90:0/3158381899 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f836006bbd0 msgr2=0x7f836006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.174+0000 7f8389a00640 1 --2- 172.21.15.90:0/3158381899 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f836006bbd0 0x7f836006e090 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f83840cfa50 tx=0x7f836c009040 comp rx=0 tx=0).stop 2024-03-24T01:32:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.174+0000 7f8389a00640 1 -- 172.21.15.90:0/3158381899 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83840aeac0 msgr2=0x7f838414c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.174+0000 7f8389a00640 1 --2- 172.21.15.90:0/3158381899 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83840aeac0 0x7f838414c570 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f837400c410 tx=0x7f837400c8d0 comp rx=0 tx=0).stop 2024-03-24T01:32:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.174+0000 7f8389a00640 1 -- 172.21.15.90:0/3158381899 shutdown_connections 2024-03-24T01:32:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.174+0000 7f8389a00640 1 --2- 172.21.15.90:0/3158381899 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f836006bbd0 0x7f836006e090 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.174+0000 7f8389a00640 1 --2- 172.21.15.90:0/3158381899 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8384161f40 0x7f8384159050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.174+0000 7f8389a00640 1 --2- 172.21.15.90:0/3158381899 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83840cfab0 0x7f83841488d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.174+0000 7f8389a00640 1 --2- 172.21.15.90:0/3158381899 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83840aeac0 0x7f838414c570 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.174+0000 7f8389a00640 1 -- 172.21.15.90:0/3158381899 >> 172.21.15.90:0/3158381899 conn(0x7f838406b0c0 msgr2=0x7f83840b2be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.174+0000 7f8389a00640 1 -- 172.21.15.90:0/3158381899 shutdown_connections 2024-03-24T01:32:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.174+0000 7f8389a00640 1 -- 172.21.15.90:0/3158381899 wait complete. 2024-03-24T01:32:49.194 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":324,"stamp":"2024-03-24T01:32:47.675626+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":256756,"kb_used_data":7304,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717964,"statfs":{"total":773094113280,"available":772831195136,"internally_reserved":0,"allocated":7479296,"data_stored":4046352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12747,"internal_metadata":255315509},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.998314"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.707707+0000","last_change":"2024-03-24T01:32:43.707707+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.707707+0000","last_undegraded":"2024-03-24T01:32:43.707707+0000","last_fullsized":"2024-03-24T01:32:43.707707+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.708057+0000","last_change":"2024-03-24T01:32:43.708057+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.708057+0000","last_undegraded":"2024-03-24T01:32:43.708057+0000","last_fullsized":"2024-03-24T01:32:43.708057+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.713931+0000","last_change":"2024-03-24T01:32:43.713931+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.713931+0000","last_undegraded":"2024-03-24T01:32:43.713931+0000","last_fullsized":"2024-03-24T01:32:43.713931+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.699612+0000","last_change":"2024-03-24T01:32:43.699612+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.699612+0000","last_undegraded":"2024-03-24T01:32:43.699612+0000","last_fullsized":"2024-03-24T01:32:43.699612+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.702612+0000","last_change":"2024-03-24T01:32:43.702612+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.702612+0000","last_undegraded":"2024-03-24T01:32:43.702612+0000","last_fullsized":"2024-03-24T01:32:43.702612+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.702504+0000","last_change":"2024-03-24T01:32:43.702504+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.702504+0000","last_undegraded":"2024-03-24T01:32:43.702504+0000","last_fullsized":"2024-03-24T01:32:43.702504+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.702766+0000","last_change":"2024-03-24T01:32:43.702766+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.702766+0000","last_undegraded":"2024-03-24T01:32:43.702766+0000","last_fullsized":"2024-03-24T01:32:43.702766+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.705564+0000","last_change":"2024-03-24T01:32:43.705564+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.705564+0000","last_undegraded":"2024-03-24T01:32:43.705564+0000","last_fullsized":"2024-03-24T01:32:43.705564+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.716331+0000","last_change":"2024-03-24T01:32:43.716331+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.716331+0000","last_undegraded":"2024-03-24T01:32:43.716331+0000","last_fullsized":"2024-03-24T01:32:43.716331+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.714938+0000","last_change":"2024-03-24T01:32:43.714938+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.714938+0000","last_undegraded":"2024-03-24T01:32:43.714938+0000","last_fullsized":"2024-03-24T01:32:43.714938+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.701980+0000","last_change":"2024-03-24T01:32:43.701980+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.701980+0000","last_undegraded":"2024-03-24T01:32:43.701980+0000","last_fullsized":"2024-03-24T01:32:43.701980+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.712314+0000","last_change":"2024-03-24T01:32:43.712314+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.712314+0000","last_undegraded":"2024-03-24T01:32:43.712314+0000","last_fullsized":"2024-03-24T01:32:43.712314+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.708206+0000","last_change":"2024-03-24T01:32:43.708206+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.708206+0000","last_undegraded":"2024-03-24T01:32:43.708206+0000","last_fullsized":"2024-03-24T01:32:43.708206+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.710509+0000","last_change":"2024-03-24T01:32:43.710509+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.710509+0000","last_undegraded":"2024-03-24T01:32:43.710509+0000","last_fullsized":"2024-03-24T01:32:43.710509+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.711385+0000","last_change":"2024-03-24T01:32:43.711385+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.711385+0000","last_undegraded":"2024-03-24T01:32:43.711385+0000","last_fullsized":"2024-03-24T01:32:43.711385+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.702194+0000","last_change":"2024-03-24T01:32:43.702194+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.702194+0000","last_undegraded":"2024-03-24T01:32:43.702194+0000","last_fullsized":"2024-03-24T01:32:43.702194+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.707315+0000","last_change":"2024-03-24T01:32:43.707315+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.707315+0000","last_undegraded":"2024-03-24T01:32:43.707315+0000","last_fullsized":"2024-03-24T01:32:43.707315+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.698533+0000","last_change":"2024-03-24T01:32:43.698533+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.698533+0000","last_undegraded":"2024-03-24T01:32:43.698533+0000","last_fullsized":"2024-03-24T01:32:43.698533+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.697420+0000","last_change":"2024-03-24T01:32:43.697420+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.697420+0000","last_undegraded":"2024-03-24T01:32:43.697420+0000","last_fullsized":"2024-03-24T01:32:43.697420+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.703125+0000","last_change":"2024-03-24T01:32:43.703125+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.703125+0000","last_undegraded":"2024-03-24T01:32:43.703125+0000","last_fullsized":"2024-03-24T01:32:43.703125+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.707468+0000","last_change":"2024-03-24T01:32:43.707468+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.707468+0000","last_undegraded":"2024-03-24T01:32:43.707468+0000","last_fullsized":"2024-03-24T01:32:43.707468+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.704614+0000","last_change":"2024-03-24T01:32:43.704614+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.704614+0000","last_undegraded":"2024-03-24T01:32:43.704614+0000","last_fullsized":"2024-03-24T01:32:43.704614+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.704122+0000","last_change":"2024-03-24T01:32:43.704122+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.704122+0000","last_undegraded":"2024-03-24T01:32:43.704122+0000","last_fullsized":"2024-03-24T01:32:43.704122+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.711724+0000","last_change":"2024-03-24T01:32:43.711724+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.711724+0000","last_undegraded":"2024-03-24T01:32:43.711724+0000","last_fullsized":"2024-03-24T01:32:43.711724+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.711476+0000","last_change":"2024-03-24T01:32:43.711476+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.711476+0000","last_undegraded":"2024-03-24T01:32:43.711476+0000","last_fullsized":"2024-03-24T01:32:43.711476+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.713052+0000","last_change":"2024-03-24T01:32:43.713052+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.713052+0000","last_undegraded":"2024-03-24T01:32:43.713052+0000","last_fullsized":"2024-03-24T01:32:43.713052+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.701299+0000","last_change":"2024-03-24T01:32:43.701299+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.701299+0000","last_undegraded":"2024-03-24T01:32:43.701299+0000","last_fullsized":"2024-03-24T01:32:43.701299+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.703917+0000","last_change":"2024-03-24T01:32:43.703917+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.703917+0000","last_undegraded":"2024-03-24T01:32:43.703917+0000","last_fullsized":"2024-03-24T01:32:43.703917+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.711697+0000","last_change":"2024-03-24T01:32:43.711697+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.711697+0000","last_undegraded":"2024-03-24T01:32:43.711697+0000","last_fullsized":"2024-03-24T01:32:43.711697+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.709236+0000","last_change":"2024-03-24T01:32:43.709236+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.709236+0000","last_undegraded":"2024-03-24T01:32:43.709236+0000","last_fullsized":"2024-03-24T01:32:43.709236+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.696424+0000","last_change":"2024-03-24T01:32:43.696424+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.696424+0000","last_undegraded":"2024-03-24T01:32:43.696424+0000","last_fullsized":"2024-03-24T01:32:43.696424+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.702852+0000","last_change":"2024-03-24T01:32:43.702852+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.702852+0000","last_undegraded":"2024-03-24T01:32:43.702852+0000","last_fullsized":"2024-03-24T01:32:43.702852+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.701720+0000","last_change":"2024-03-24T01:32:43.701720+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.701720+0000","last_undegraded":"2024-03-24T01:32:43.701720+0000","last_fullsized":"2024-03-24T01:32:43.701720+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.729973+0000","last_change":"2024-03-24T01:32:43.729112+0000","last_active":"2024-03-24T01:32:43.729973+0000","last_peered":"2024-03-24T01:32:43.729973+0000","last_clean":"2024-03-24T01:32:43.729973+0000","last_became_active":"2024-03-24T01:32:43.728882+0000","last_became_peered":"2024-03-24T01:32:43.728882+0000","last_unstale":"2024-03-24T01:32:43.729973+0000","last_undegraded":"2024-03-24T01:32:43.729973+0000","last_fullsized":"2024-03-24T01:32:43.729973+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:19:06.949970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.700467+0000","last_change":"2024-03-24T01:32:43.700467+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.700467+0000","last_undegraded":"2024-03-24T01:32:43.700467+0000","last_fullsized":"2024-03-24T01:32:43.700467+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.726029+0000","last_change":"2024-03-24T01:32:43.723642+0000","last_active":"2024-03-24T01:32:43.726029+0000","last_peered":"2024-03-24T01:32:43.726029+0000","last_clean":"2024-03-24T01:32:43.726029+0000","last_became_active":"2024-03-24T01:32:43.723374+0000","last_became_peered":"2024-03-24T01:32:43.723374+0000","last_unstale":"2024-03-24T01:32:43.726029+0000","last_undegraded":"2024-03-24T01:32:43.726029+0000","last_fullsized":"2024-03-24T01:32:43.726029+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:02.791894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.701118+0000","last_change":"2024-03-24T01:32:43.701118+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.701118+0000","last_undegraded":"2024-03-24T01:32:43.701118+0000","last_fullsized":"2024-03-24T01:32:43.701118+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.724435+0000","last_change":"2024-03-24T01:32:43.722317+0000","last_active":"2024-03-24T01:32:43.724435+0000","last_peered":"2024-03-24T01:32:43.724435+0000","last_clean":"2024-03-24T01:32:43.724435+0000","last_became_active":"2024-03-24T01:32:43.721576+0000","last_became_peered":"2024-03-24T01:32:43.721576+0000","last_unstale":"2024-03-24T01:32:43.724435+0000","last_undegraded":"2024-03-24T01:32:43.724435+0000","last_fullsized":"2024-03-24T01:32:43.724435+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:10:35.120169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.716728+0000","last_change":"2024-03-24T01:32:43.716728+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.716728+0000","last_undegraded":"2024-03-24T01:32:43.716728+0000","last_fullsized":"2024-03-24T01:32:43.716728+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.717511+0000","last_change":"2024-03-24T01:32:43.713108+0000","last_active":"2024-03-24T01:32:43.717511+0000","last_peered":"2024-03-24T01:32:43.717511+0000","last_clean":"2024-03-24T01:32:43.717511+0000","last_became_active":"2024-03-24T01:32:43.712800+0000","last_became_peered":"2024-03-24T01:32:43.712800+0000","last_unstale":"2024-03-24T01:32:43.717511+0000","last_undegraded":"2024-03-24T01:32:43.717511+0000","last_fullsized":"2024-03-24T01:32:43.717511+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:34.647291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.701602+0000","last_change":"2024-03-24T01:32:43.701602+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.701602+0000","last_undegraded":"2024-03-24T01:32:43.701602+0000","last_fullsized":"2024-03-24T01:32:43.701602+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.722195+0000","last_change":"2024-03-24T01:32:43.718182+0000","last_active":"2024-03-24T01:32:43.722195+0000","last_peered":"2024-03-24T01:32:43.722195+0000","last_clean":"2024-03-24T01:32:43.722195+0000","last_became_active":"2024-03-24T01:32:43.717674+0000","last_became_peered":"2024-03-24T01:32:43.717674+0000","last_unstale":"2024-03-24T01:32:43.722195+0000","last_undegraded":"2024-03-24T01:32:43.722195+0000","last_fullsized":"2024-03-24T01:32:43.722195+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:39:22.546505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.700292+0000","last_change":"2024-03-24T01:32:43.700292+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.700292+0000","last_undegraded":"2024-03-24T01:32:43.700292+0000","last_fullsized":"2024-03-24T01:32:43.700292+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.722624+0000","last_change":"2024-03-24T01:32:43.717225+0000","last_active":"2024-03-24T01:32:43.722624+0000","last_peered":"2024-03-24T01:32:43.722624+0000","last_clean":"2024-03-24T01:32:43.722624+0000","last_became_active":"2024-03-24T01:32:43.716994+0000","last_became_peered":"2024-03-24T01:32:43.716994+0000","last_unstale":"2024-03-24T01:32:43.722624+0000","last_undegraded":"2024-03-24T01:32:43.722624+0000","last_fullsized":"2024-03-24T01:32:43.722624+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:14:55.272028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.698786+0000","last_change":"2024-03-24T01:32:43.698786+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.698786+0000","last_undegraded":"2024-03-24T01:32:43.698786+0000","last_fullsized":"2024-03-24T01:32:43.698786+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.722633+0000","last_change":"2024-03-24T01:32:43.719216+0000","last_active":"2024-03-24T01:32:43.722633+0000","last_peered":"2024-03-24T01:32:43.722633+0000","last_clean":"2024-03-24T01:32:43.722633+0000","last_became_active":"2024-03-24T01:32:43.718879+0000","last_became_peered":"2024-03-24T01:32:43.718879+0000","last_unstale":"2024-03-24T01:32:43.722633+0000","last_undegraded":"2024-03-24T01:32:43.722633+0000","last_fullsized":"2024-03-24T01:32:43.722633+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:16:58.498006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.702537+0000","last_change":"2024-03-24T01:32:43.702537+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.702537+0000","last_undegraded":"2024-03-24T01:32:43.702537+0000","last_fullsized":"2024-03-24T01:32:43.702537+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.723762+0000","last_change":"2024-03-24T01:32:43.720754+0000","last_active":"2024-03-24T01:32:43.723762+0000","last_peered":"2024-03-24T01:32:43.723762+0000","last_clean":"2024-03-24T01:32:43.723762+0000","last_became_active":"2024-03-24T01:32:43.720465+0000","last_became_peered":"2024-03-24T01:32:43.720465+0000","last_unstale":"2024-03-24T01:32:43.723762+0000","last_undegraded":"2024-03-24T01:32:43.723762+0000","last_fullsized":"2024-03-24T01:32:43.723762+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:48:15.469032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.703279+0000","last_change":"2024-03-24T01:32:43.703279+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.703279+0000","last_undegraded":"2024-03-24T01:32:43.703279+0000","last_fullsized":"2024-03-24T01:32:43.703279+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.734285+0000","last_change":"2024-03-24T01:32:43.729179+0000","last_active":"2024-03-24T01:32:43.734285+0000","last_peered":"2024-03-24T01:32:43.734285+0000","last_clean":"2024-03-24T01:32:43.734285+0000","last_became_active":"2024-03-24T01:32:43.728892+0000","last_became_peered":"2024-03-24T01:32:43.728892+0000","last_unstale":"2024-03-24T01:32:43.734285+0000","last_undegraded":"2024-03-24T01:32:43.734285+0000","last_fullsized":"2024-03-24T01:32:43.734285+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:31:44.560907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.703626+0000","last_change":"2024-03-24T01:32:43.703626+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.703626+0000","last_undegraded":"2024-03-24T01:32:43.703626+0000","last_fullsized":"2024-03-24T01:32:43.703626+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.718845+0000","last_change":"2024-03-24T01:32:43.715851+0000","last_active":"2024-03-24T01:32:43.718845+0000","last_peered":"2024-03-24T01:32:43.718845+0000","last_clean":"2024-03-24T01:32:43.718845+0000","last_became_active":"2024-03-24T01:32:43.715474+0000","last_became_peered":"2024-03-24T01:32:43.715474+0000","last_unstale":"2024-03-24T01:32:43.718845+0000","last_undegraded":"2024-03-24T01:32:43.718845+0000","last_fullsized":"2024-03-24T01:32:43.718845+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:57:39.379197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.696544+0000","last_change":"2024-03-24T01:32:43.696544+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.696544+0000","last_undegraded":"2024-03-24T01:32:43.696544+0000","last_fullsized":"2024-03-24T01:32:43.696544+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.731495+0000","last_change":"2024-03-24T01:32:43.725497+0000","last_active":"2024-03-24T01:32:43.731495+0000","last_peered":"2024-03-24T01:32:43.731495+0000","last_clean":"2024-03-24T01:32:43.731495+0000","last_became_active":"2024-03-24T01:32:43.724751+0000","last_became_peered":"2024-03-24T01:32:43.724751+0000","last_unstale":"2024-03-24T01:32:43.731495+0000","last_undegraded":"2024-03-24T01:32:43.731495+0000","last_fullsized":"2024-03-24T01:32:43.731495+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:29.157467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.700996+0000","last_change":"2024-03-24T01:32:43.700996+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.700996+0000","last_undegraded":"2024-03-24T01:32:43.700996+0000","last_fullsized":"2024-03-24T01:32:43.700996+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.723022+0000","last_change":"2024-03-24T01:32:43.716683+0000","last_active":"2024-03-24T01:32:43.723022+0000","last_peered":"2024-03-24T01:32:43.723022+0000","last_clean":"2024-03-24T01:32:43.723022+0000","last_became_active":"2024-03-24T01:32:43.716407+0000","last_became_peered":"2024-03-24T01:32:43.716407+0000","last_unstale":"2024-03-24T01:32:43.723022+0000","last_undegraded":"2024-03-24T01:32:43.723022+0000","last_fullsized":"2024-03-24T01:32:43.723022+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:31:19.992387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.695279+0000","last_change":"2024-03-24T01:32:43.695279+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.695279+0000","last_undegraded":"2024-03-24T01:32:43.695279+0000","last_fullsized":"2024-03-24T01:32:43.695279+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.731501+0000","last_change":"2024-03-24T01:32:43.725509+0000","last_active":"2024-03-24T01:32:43.731501+0000","last_peered":"2024-03-24T01:32:43.731501+0000","last_clean":"2024-03-24T01:32:43.731501+0000","last_became_active":"2024-03-24T01:32:43.725142+0000","last_became_peered":"2024-03-24T01:32:43.725142+0000","last_unstale":"2024-03-24T01:32:43.731501+0000","last_undegraded":"2024-03-24T01:32:43.731501+0000","last_fullsized":"2024-03-24T01:32:43.731501+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:43:54.681008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.711736+0000","last_change":"2024-03-24T01:32:43.711736+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.711736+0000","last_undegraded":"2024-03-24T01:32:43.711736+0000","last_fullsized":"2024-03-24T01:32:43.711736+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.724233+0000","last_change":"2024-03-24T01:32:43.720435+0000","last_active":"2024-03-24T01:32:43.724233+0000","last_peered":"2024-03-24T01:32:43.724233+0000","last_clean":"2024-03-24T01:32:43.724233+0000","last_became_active":"2024-03-24T01:32:43.720135+0000","last_became_peered":"2024-03-24T01:32:43.720135+0000","last_unstale":"2024-03-24T01:32:43.724233+0000","last_undegraded":"2024-03-24T01:32:43.724233+0000","last_fullsized":"2024-03-24T01:32:43.724233+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:31:19.992387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.695588+0000","last_change":"2024-03-24T01:32:43.695588+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.695588+0000","last_undegraded":"2024-03-24T01:32:43.695588+0000","last_fullsized":"2024-03-24T01:32:43.695588+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.731473+0000","last_change":"2024-03-24T01:32:43.725306+0000","last_active":"2024-03-24T01:32:43.731473+0000","last_peered":"2024-03-24T01:32:43.731473+0000","last_clean":"2024-03-24T01:32:43.731473+0000","last_became_active":"2024-03-24T01:32:43.724101+0000","last_became_peered":"2024-03-24T01:32:43.724101+0000","last_unstale":"2024-03-24T01:32:43.731473+0000","last_undegraded":"2024-03-24T01:32:43.731473+0000","last_fullsized":"2024-03-24T01:32:43.731473+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:19:06.949970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.699882+0000","last_change":"2024-03-24T01:32:43.699882+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.699882+0000","last_undegraded":"2024-03-24T01:32:43.699882+0000","last_fullsized":"2024-03-24T01:32:43.699882+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.726036+0000","last_change":"2024-03-24T01:32:43.720664+0000","last_active":"2024-03-24T01:32:43.726036+0000","last_peered":"2024-03-24T01:32:43.726036+0000","last_clean":"2024-03-24T01:32:43.726036+0000","last_became_active":"2024-03-24T01:32:43.720338+0000","last_became_peered":"2024-03-24T01:32:43.720338+0000","last_unstale":"2024-03-24T01:32:43.726036+0000","last_undegraded":"2024-03-24T01:32:43.726036+0000","last_fullsized":"2024-03-24T01:32:43.726036+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:48:15.469032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.712601+0000","last_change":"2024-03-24T01:32:43.712601+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.712601+0000","last_undegraded":"2024-03-24T01:32:43.712601+0000","last_fullsized":"2024-03-24T01:32:43.712601+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.723816+0000","last_change":"2024-03-24T01:32:43.722152+0000","last_active":"2024-03-24T01:32:43.723816+0000","last_peered":"2024-03-24T01:32:43.723816+0000","last_clean":"2024-03-24T01:32:43.723816+0000","last_became_active":"2024-03-24T01:32:43.721855+0000","last_became_peered":"2024-03-24T01:32:43.721855+0000","last_unstale":"2024-03-24T01:32:43.723816+0000","last_undegraded":"2024-03-24T01:32:43.723816+0000","last_fullsized":"2024-03-24T01:32:43.723816+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:31:15.858645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.696181+0000","last_change":"2024-03-24T01:32:43.696181+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.696181+0000","last_undegraded":"2024-03-24T01:32:43.696181+0000","last_fullsized":"2024-03-24T01:32:43.696181+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.731711+0000","last_change":"2024-03-24T01:32:43.726876+0000","last_active":"2024-03-24T01:32:43.731711+0000","last_peered":"2024-03-24T01:32:43.731711+0000","last_clean":"2024-03-24T01:32:43.731711+0000","last_became_active":"2024-03-24T01:32:43.726628+0000","last_became_peered":"2024-03-24T01:32:43.726628+0000","last_unstale":"2024-03-24T01:32:43.731711+0000","last_undegraded":"2024-03-24T01:32:43.731711+0000","last_fullsized":"2024-03-24T01:32:43.731711+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:11:39.983567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.697338+0000","last_change":"2024-03-24T01:32:43.697338+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.697338+0000","last_undegraded":"2024-03-24T01:32:43.697338+0000","last_fullsized":"2024-03-24T01:32:43.697338+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.731521+0000","last_change":"2024-03-24T01:32:43.725389+0000","last_active":"2024-03-24T01:32:43.731521+0000","last_peered":"2024-03-24T01:32:43.731521+0000","last_clean":"2024-03-24T01:32:43.731521+0000","last_became_active":"2024-03-24T01:32:43.724956+0000","last_became_peered":"2024-03-24T01:32:43.724956+0000","last_unstale":"2024-03-24T01:32:43.731521+0000","last_undegraded":"2024-03-24T01:32:43.731521+0000","last_fullsized":"2024-03-24T01:32:43.731521+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:31:15.858645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.699266+0000","last_change":"2024-03-24T01:32:43.699266+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.699266+0000","last_undegraded":"2024-03-24T01:32:43.699266+0000","last_fullsized":"2024-03-24T01:32:43.699266+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.723870+0000","last_change":"2024-03-24T01:32:43.721896+0000","last_active":"2024-03-24T01:32:43.723870+0000","last_peered":"2024-03-24T01:32:43.723870+0000","last_clean":"2024-03-24T01:32:43.723870+0000","last_became_active":"2024-03-24T01:32:43.720277+0000","last_became_peered":"2024-03-24T01:32:43.720277+0000","last_unstale":"2024-03-24T01:32:43.723870+0000","last_undegraded":"2024-03-24T01:32:43.723870+0000","last_fullsized":"2024-03-24T01:32:43.723870+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:48:17.796412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.702327+0000","last_change":"2024-03-24T01:32:43.702327+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.702327+0000","last_undegraded":"2024-03-24T01:32:43.702327+0000","last_fullsized":"2024-03-24T01:32:43.702327+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.715972+0000","last_change":"2024-03-24T01:32:43.707322+0000","last_active":"2024-03-24T01:32:43.715972+0000","last_peered":"2024-03-24T01:32:43.715972+0000","last_clean":"2024-03-24T01:32:43.715972+0000","last_became_active":"2024-03-24T01:32:43.706928+0000","last_became_peered":"2024-03-24T01:32:43.706928+0000","last_unstale":"2024-03-24T01:32:43.715972+0000","last_undegraded":"2024-03-24T01:32:43.715972+0000","last_fullsized":"2024-03-24T01:32:43.715972+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:18.017172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.711175+0000","last_change":"2024-03-24T01:32:43.711175+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.711175+0000","last_undegraded":"2024-03-24T01:32:43.711175+0000","last_fullsized":"2024-03-24T01:32:43.711175+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.726365+0000","last_change":"2024-03-24T01:32:43.724088+0000","last_active":"2024-03-24T01:32:43.726365+0000","last_peered":"2024-03-24T01:32:43.726365+0000","last_clean":"2024-03-24T01:32:43.726365+0000","last_became_active":"2024-03-24T01:32:43.722732+0000","last_became_peered":"2024-03-24T01:32:43.722732+0000","last_unstale":"2024-03-24T01:32:43.726365+0000","last_undegraded":"2024-03-24T01:32:43.726365+0000","last_fullsized":"2024-03-24T01:32:43.726365+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:29.157467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.694328+0000","last_change":"2024-03-24T01:32:43.694328+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.694328+0000","last_undegraded":"2024-03-24T01:32:43.694328+0000","last_fullsized":"2024-03-24T01:32:43.694328+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.724217+0000","last_change":"2024-03-24T01:32:43.720278+0000","last_active":"2024-03-24T01:32:43.724217+0000","last_peered":"2024-03-24T01:32:43.724217+0000","last_clean":"2024-03-24T01:32:43.724217+0000","last_became_active":"2024-03-24T01:32:43.719995+0000","last_became_peered":"2024-03-24T01:32:43.719995+0000","last_unstale":"2024-03-24T01:32:43.724217+0000","last_undegraded":"2024-03-24T01:32:43.724217+0000","last_fullsized":"2024-03-24T01:32:43.724217+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:11.809594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.699505+0000","last_change":"2024-03-24T01:32:43.699505+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.699505+0000","last_undegraded":"2024-03-24T01:32:43.699505+0000","last_fullsized":"2024-03-24T01:32:43.699505+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.734071+0000","last_change":"2024-03-24T01:32:43.729534+0000","last_active":"2024-03-24T01:32:43.734071+0000","last_peered":"2024-03-24T01:32:43.734071+0000","last_clean":"2024-03-24T01:32:43.734071+0000","last_became_active":"2024-03-24T01:32:43.729214+0000","last_became_peered":"2024-03-24T01:32:43.729214+0000","last_unstale":"2024-03-24T01:32:43.734071+0000","last_undegraded":"2024-03-24T01:32:43.734071+0000","last_fullsized":"2024-03-24T01:32:43.734071+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:43:54.681008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.705306+0000","last_change":"2024-03-24T01:32:43.705306+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.705306+0000","last_undegraded":"2024-03-24T01:32:43.705306+0000","last_fullsized":"2024-03-24T01:32:43.705306+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.723223+0000","last_change":"2024-03-24T01:32:43.720477+0000","last_active":"2024-03-24T01:32:43.723223+0000","last_peered":"2024-03-24T01:32:43.723223+0000","last_clean":"2024-03-24T01:32:43.723223+0000","last_became_active":"2024-03-24T01:32:43.720192+0000","last_became_peered":"2024-03-24T01:32:43.720192+0000","last_unstale":"2024-03-24T01:32:43.723223+0000","last_undegraded":"2024-03-24T01:32:43.723223+0000","last_fullsized":"2024-03-24T01:32:43.723223+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:19:06.949970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.708484+0000","last_change":"2024-03-24T01:32:43.708484+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.708484+0000","last_undegraded":"2024-03-24T01:32:43.708484+0000","last_fullsized":"2024-03-24T01:32:43.708484+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.726703+0000","last_change":"2024-03-24T01:32:43.725481+0000","last_active":"2024-03-24T01:32:43.726703+0000","last_peered":"2024-03-24T01:32:43.726703+0000","last_clean":"2024-03-24T01:32:43.726703+0000","last_became_active":"2024-03-24T01:32:43.725185+0000","last_became_peered":"2024-03-24T01:32:43.725185+0000","last_unstale":"2024-03-24T01:32:43.726703+0000","last_undegraded":"2024-03-24T01:32:43.726703+0000","last_fullsized":"2024-03-24T01:32:43.726703+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:15:19.800763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.698042+0000","last_change":"2024-03-24T01:32:43.698042+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.698042+0000","last_undegraded":"2024-03-24T01:32:43.698042+0000","last_fullsized":"2024-03-24T01:32:43.698042+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.719313+0000","last_change":"2024-03-24T01:32:43.714827+0000","last_active":"2024-03-24T01:32:43.719313+0000","last_peered":"2024-03-24T01:32:43.719313+0000","last_clean":"2024-03-24T01:32:43.719313+0000","last_became_active":"2024-03-24T01:32:43.714552+0000","last_became_peered":"2024-03-24T01:32:43.714552+0000","last_unstale":"2024-03-24T01:32:43.719313+0000","last_undegraded":"2024-03-24T01:32:43.719313+0000","last_fullsized":"2024-03-24T01:32:43.719313+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:39:25.511297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.693643+0000","last_change":"2024-03-24T01:32:43.693643+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.693643+0000","last_undegraded":"2024-03-24T01:32:43.693643+0000","last_fullsized":"2024-03-24T01:32:43.693643+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.731381+0000","last_change":"2024-03-24T01:32:43.724564+0000","last_active":"2024-03-24T01:32:43.731381+0000","last_peered":"2024-03-24T01:32:43.731381+0000","last_clean":"2024-03-24T01:32:43.731381+0000","last_became_active":"2024-03-24T01:32:43.724051+0000","last_became_peered":"2024-03-24T01:32:43.724051+0000","last_unstale":"2024-03-24T01:32:43.731381+0000","last_undegraded":"2024-03-24T01:32:43.731381+0000","last_fullsized":"2024-03-24T01:32:43.731381+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:31:19.992387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.695082+0000","last_change":"2024-03-24T01:32:43.695082+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.695082+0000","last_undegraded":"2024-03-24T01:32:43.695082+0000","last_fullsized":"2024-03-24T01:32:43.695082+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.718069+0000","last_change":"2024-03-24T01:32:43.715420+0000","last_active":"2024-03-24T01:32:43.718069+0000","last_peered":"2024-03-24T01:32:43.718069+0000","last_clean":"2024-03-24T01:32:43.718069+0000","last_became_active":"2024-03-24T01:32:43.714725+0000","last_became_peered":"2024-03-24T01:32:43.714725+0000","last_unstale":"2024-03-24T01:32:43.718069+0000","last_undegraded":"2024-03-24T01:32:43.718069+0000","last_fullsized":"2024-03-24T01:32:43.718069+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:39:09.134793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.699965+0000","last_change":"2024-03-24T01:32:43.699965+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.699965+0000","last_undegraded":"2024-03-24T01:32:43.699965+0000","last_fullsized":"2024-03-24T01:32:43.699965+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.722608+0000","last_change":"2024-03-24T01:32:43.719188+0000","last_active":"2024-03-24T01:32:43.722608+0000","last_peered":"2024-03-24T01:32:43.722608+0000","last_clean":"2024-03-24T01:32:43.722608+0000","last_became_active":"2024-03-24T01:32:43.718755+0000","last_became_peered":"2024-03-24T01:32:43.718755+0000","last_unstale":"2024-03-24T01:32:43.722608+0000","last_undegraded":"2024-03-24T01:32:43.722608+0000","last_fullsized":"2024-03-24T01:32:43.722608+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:36:56.063762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.703677+0000","last_change":"2024-03-24T01:32:43.703677+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.703677+0000","last_undegraded":"2024-03-24T01:32:43.703677+0000","last_fullsized":"2024-03-24T01:32:43.703677+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.719439+0000","last_change":"2024-03-24T01:32:43.714616+0000","last_active":"2024-03-24T01:32:43.719439+0000","last_peered":"2024-03-24T01:32:43.719439+0000","last_clean":"2024-03-24T01:32:43.719439+0000","last_became_active":"2024-03-24T01:32:43.714216+0000","last_became_peered":"2024-03-24T01:32:43.714216+0000","last_unstale":"2024-03-24T01:32:43.719439+0000","last_undegraded":"2024-03-24T01:32:43.719439+0000","last_fullsized":"2024-03-24T01:32:43.719439+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:51:18.569189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.695893+0000","last_change":"2024-03-24T01:32:43.695893+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.695893+0000","last_undegraded":"2024-03-24T01:32:43.695893+0000","last_fullsized":"2024-03-24T01:32:43.695893+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.724138+0000","last_change":"2024-03-24T01:32:43.719380+0000","last_active":"2024-03-24T01:32:43.724138+0000","last_peered":"2024-03-24T01:32:43.724138+0000","last_clean":"2024-03-24T01:32:43.724138+0000","last_became_active":"2024-03-24T01:32:43.718702+0000","last_became_peered":"2024-03-24T01:32:43.718702+0000","last_unstale":"2024-03-24T01:32:43.724138+0000","last_undegraded":"2024-03-24T01:32:43.724138+0000","last_fullsized":"2024-03-24T01:32:43.724138+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:59:11.903374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":255,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.714595+0000","last_change":"2024-03-24T01:24:01.545855+0000","last_active":"2024-03-24T01:32:43.714595+0000","last_peered":"2024-03-24T01:32:43.714595+0000","last_clean":"2024-03-24T01:32:43.714595+0000","last_became_active":"2024-03-24T01:24:01.543811+0000","last_became_peered":"2024-03-24T01:24:01.543811+0000","last_unstale":"2024-03-24T01:32:43.714595+0000","last_undegraded":"2024-03-24T01:32:43.714595+0000","last_fullsized":"2024-03-24T01:32:43.714595+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_clean_scrub_stamp":"2024-03-24T01:23:59.861564+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:03.682631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574956,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32532,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339308,"statfs":{"total":96636764160,"available":96603451392,"internally_reserved":0,"allocated":1380352,"data_stored":948570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574956,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31948,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339892,"statfs":{"total":96636764160,"available":96604049408,"internally_reserved":0,"allocated":786432,"data_stored":358202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574957,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31948,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339892,"statfs":{"total":96636764160,"available":96604049408,"internally_reserved":0,"allocated":786432,"data_stored":358202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574956,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31948,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339892,"statfs":{"total":96636764160,"available":96604049408,"internally_reserved":0,"allocated":786432,"data_stored":358202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574956,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31952,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339888,"statfs":{"total":96636764160,"available":96604045312,"internally_reserved":0,"allocated":786432,"data_stored":358202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574957,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31952,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339888,"statfs":{"total":96636764160,"available":96604045312,"internally_reserved":0,"allocated":786432,"data_stored":358202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574956,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31948,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339892,"statfs":{"total":96636764160,"available":96604049408,"internally_reserved":0,"allocated":786432,"data_stored":358202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574957,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32528,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339312,"statfs":{"total":96636764160,"available":96603455488,"internally_reserved":0,"allocated":1380352,"data_stored":948570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T01:32:49.198 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T01:32:49.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.445+0000 7f1b3fc00640 1 Processor -- start 2024-03-24T01:32:49.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.445+0000 7f1b3fc00640 1 -- start start 2024-03-24T01:32:49.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.446+0000 7f1b3fc00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1b3815e090 0x7f1b3815e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:49.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.446+0000 7f1b3fc00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1b380b94a0 0x7f1b3806c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:49.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.447+0000 7f1b3fc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b3806cc40 0x7f1b380b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:49.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.447+0000 7f1b3fc00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b380b6c20 con 0x7f1b3815e090 2024-03-24T01:32:49.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.447+0000 7f1b3fc00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b380d1fe0 con 0x7f1b380b94a0 2024-03-24T01:32:49.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.447+0000 7f1b3fc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3815ecd0 con 0x7f1b3806cc40 2024-03-24T01:32:49.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.447+0000 7f1b3d200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1b3815e090 0x7f1b3815e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:49.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.447+0000 7f1b3d200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1b3815e090 0x7f1b3815e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39790/0 (socket says 172.21.15.90:39790) 2024-03-24T01:32:49.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.447+0000 7f1b3d200640 1 -- 172.21.15.90:0/3297318821 learned_addr learned my addr 172.21.15.90:0/3297318821 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:49.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.447+0000 7f1b3dc00640 1 --2- 172.21.15.90:0/3297318821 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b3806cc40 0x7f1b380b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:49.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.447+0000 7f1b37e00640 1 --2- 172.21.15.90:0/3297318821 >> v2:172.21.15.120:3300/0 conn(0x7f1b380b94a0 0x7f1b3806c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:49.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.447+0000 7f1b3d200640 1 -- 172.21.15.90:0/3297318821 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b3806cc40 msgr2=0x7f1b380b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:49.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.447+0000 7f1b3d200640 1 --2- 172.21.15.90:0/3297318821 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b3806cc40 0x7f1b380b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:49.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.447+0000 7f1b3d200640 1 -- 172.21.15.90:0/3297318821 >> v2:172.21.15.120:3300/0 conn(0x7f1b380b94a0 msgr2=0x7f1b3806c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:49.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.447+0000 7f1b3d200640 1 --2- 172.21.15.90:0/3297318821 >> v2:172.21.15.120:3300/0 conn(0x7f1b380b94a0 0x7f1b3806c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:49.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.447+0000 7f1b3d200640 1 -- 172.21.15.90:0/3297318821 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b3815efb0 con 0x7f1b3815e090 2024-03-24T01:32:49.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.447+0000 7f1b37e00640 1 --2- 172.21.15.90:0/3297318821 >> v2:172.21.15.120:3300/0 conn(0x7f1b380b94a0 0x7f1b3806c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:49.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.448+0000 7f1b3d200640 1 --2- 172.21.15.90:0/3297318821 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b3815e090 0x7f1b3815e470 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f1b380ce130 tx=0x7f1b2002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3280398036b3b05 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:49.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.449+0000 7f1b37400640 1 -- 172.21.15.90:0/3297318821 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b20034070 con 0x7f1b3815e090 2024-03-24T01:32:49.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.449+0000 7f1b37400640 1 -- 172.21.15.90:0/3297318821 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b20030860 con 0x7f1b3815e090 2024-03-24T01:32:49.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.449+0000 7f1b37400640 1 -- 172.21.15.90:0/3297318821 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b20030c30 con 0x7f1b3815e090 2024-03-24T01:32:49.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.449+0000 7f1b3fc00640 1 -- 172.21.15.90:0/3297318821 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b3815e090 msgr2=0x7f1b3815e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:49.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.449+0000 7f1b3fc00640 1 --2- 172.21.15.90:0/3297318821 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b3815e090 0x7f1b3815e470 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f1b380ce130 tx=0x7f1b2002b9a0 comp rx=0 tx=0).stop 2024-03-24T01:32:49.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.449+0000 7f1b3fc00640 1 -- 172.21.15.90:0/3297318821 shutdown_connections 2024-03-24T01:32:49.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.449+0000 7f1b3fc00640 1 --2- 172.21.15.90:0/3297318821 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b3806cc40 0x7f1b380b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:49.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.449+0000 7f1b3fc00640 1 --2- 172.21.15.90:0/3297318821 >> v2:172.21.15.120:3300/0 conn(0x7f1b380b94a0 0x7f1b3806c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:49.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.449+0000 7f1b3fc00640 1 --2- 172.21.15.90:0/3297318821 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b3815e090 0x7f1b3815e470 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:49.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.449+0000 7f1b3fc00640 1 -- 172.21.15.90:0/3297318821 >> 172.21.15.90:0/3297318821 conn(0x7f1b38072c40 msgr2=0x7f1b38073050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:49.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.449+0000 7f1b3fc00640 1 -- 172.21.15.90:0/3297318821 shutdown_connections 2024-03-24T01:32:49.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.450+0000 7f1b3fc00640 1 -- 172.21.15.90:0/3297318821 wait complete. 2024-03-24T01:32:49.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.451+0000 7f1b3fc00640 1 Processor -- start 2024-03-24T01:32:49.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.451+0000 7f1b3fc00640 1 -- start start 2024-03-24T01:32:49.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.452+0000 7f1b3fc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b3806cc40 0x7f1b381e6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:49.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.452+0000 7f1b3fc00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b380b94a0 0x7f1b3814d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:49.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.452+0000 7f1b3fc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b3815e090 0x7f1b380cb610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:49.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.452+0000 7f1b3fc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3815e9b0 con 0x7f1b3806cc40 2024-03-24T01:32:49.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.452+0000 7f1b3fc00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b380d22a0 con 0x7f1b380b94a0 2024-03-24T01:32:49.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.452+0000 7f1b3fc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b380b5b80 con 0x7f1b3815e090 2024-03-24T01:32:49.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.452+0000 7f1b3d200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b3806cc40 0x7f1b381e6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:49.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.452+0000 7f1b3d200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b3806cc40 0x7f1b381e6930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39794/0 (socket says 172.21.15.90:39794) 2024-03-24T01:32:49.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.452+0000 7f1b3d200640 1 -- 172.21.15.90:0/1835317107 learned_addr learned my addr 172.21.15.90:0/1835317107 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:49.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.453+0000 7f1b37e00640 1 --2- 172.21.15.90:0/1835317107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b380b94a0 0x7f1b3814d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:49.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.453+0000 7f1b3d200640 1 -- 172.21.15.90:0/1835317107 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b3815e090 msgr2=0x7f1b380cb610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:49.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.453+0000 7f1b3d200640 1 --2- 172.21.15.90:0/1835317107 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b3815e090 0x7f1b380cb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:49.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.453+0000 7f1b3d200640 1 -- 172.21.15.90:0/1835317107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b380b94a0 msgr2=0x7f1b3814d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:49.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.453+0000 7f1b3d200640 1 --2- 172.21.15.90:0/1835317107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b380b94a0 0x7f1b3814d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:49.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.453+0000 7f1b3d200640 1 -- 172.21.15.90:0/1835317107 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b381e4280 con 0x7f1b3806cc40 2024-03-24T01:32:49.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.453+0000 7f1b3d200640 1 --2- 172.21.15.90:0/1835317107 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b3806cc40 0x7f1b381e6930 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f1b2002b9e0 tx=0x7f1b200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:49.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.453+0000 7f1b34c00640 1 -- 172.21.15.90:0/1835317107 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b20034040 con 0x7f1b3806cc40 2024-03-24T01:32:49.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.453+0000 7f1b34c00640 1 -- 172.21.15.90:0/1835317107 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b200433f0 con 0x7f1b3806cc40 2024-03-24T01:32:49.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.453+0000 7f1b34c00640 1 -- 172.21.15.90:0/1835317107 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b200437e0 con 0x7f1b3806cc40 2024-03-24T01:32:49.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.453+0000 7f1b3fc00640 1 -- 172.21.15.90:0/1835317107 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b3814d5a0 con 0x7f1b3806cc40 2024-03-24T01:32:49.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.454+0000 7f1b3fc00640 1 -- 172.21.15.90:0/1835317107 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b380cc140 con 0x7f1b3806cc40 2024-03-24T01:32:49.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.455+0000 7f1b3fc00640 1 -- 172.21.15.90:0/1835317107 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b38068060 con 0x7f1b3806cc40 2024-03-24T01:32:49.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.456+0000 7f1b34c00640 1 -- 172.21.15.90:0/1835317107 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1b2004a060 con 0x7f1b3806cc40 2024-03-24T01:32:49.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.456+0000 7f1b34c00640 1 --2- 172.21.15.90:0/1835317107 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1b1406bbd0 0x7f1b1406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:49.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.457+0000 7f1b34c00640 1 -- 172.21.15.90:0/1835317107 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1b200acf20 con 0x7f1b3806cc40 2024-03-24T01:32:49.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.457+0000 7f1b37e00640 1 --2- 172.21.15.90:0/1835317107 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1b1406bbd0 0x7f1b1406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:49.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.458+0000 7f1b37e00640 1 --2- 172.21.15.90:0/1835317107 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1b1406bbd0 0x7f1b1406e090 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f1b3815ce90 tx=0x7f1b280084d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:49.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.461+0000 7f1b34c00640 1 -- 172.21.15.90:0/1835317107 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1b2003e070 con 0x7f1b3806cc40 2024-03-24T01:32:49.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.648+0000 7f1b3fc00640 1 -- 172.21.15.90:0/1835317107 --> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1b380d3690 con 0x7f1b1406bbd0 2024-03-24T01:32:49.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.666+0000 7f1b34c00640 1 -- 172.21.15.90:0/1835317107 <== mgr.4104 v2:172.21.15.120:6832/860392634 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246711 (secure 0 0 0) 0x7f1b380d3690 con 0x7f1b1406bbd0 2024-03-24T01:32:49.669 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:32:49.672 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-24T01:32:49.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.672+0000 7f1b3fc00640 1 -- 172.21.15.90:0/1835317107 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1b1406bbd0 msgr2=0x7f1b1406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:49.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.672+0000 7f1b3fc00640 1 --2- 172.21.15.90:0/1835317107 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1b1406bbd0 0x7f1b1406e090 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f1b3815ce90 tx=0x7f1b280084d0 comp rx=0 tx=0).stop 2024-03-24T01:32:49.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.672+0000 7f1b3fc00640 1 -- 172.21.15.90:0/1835317107 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b3806cc40 msgr2=0x7f1b381e6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:49.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.672+0000 7f1b3fc00640 1 --2- 172.21.15.90:0/1835317107 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b3806cc40 0x7f1b381e6930 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f1b2002b9e0 tx=0x7f1b200046c0 comp rx=0 tx=0).stop 2024-03-24T01:32:49.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.673+0000 7f1b3fc00640 1 -- 172.21.15.90:0/1835317107 shutdown_connections 2024-03-24T01:32:49.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.673+0000 7f1b3fc00640 1 --2- 172.21.15.90:0/1835317107 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1b1406bbd0 0x7f1b1406e090 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:49.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.673+0000 7f1b3fc00640 1 --2- 172.21.15.90:0/1835317107 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1b3815e090 0x7f1b380cb610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:49.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.673+0000 7f1b3fc00640 1 --2- 172.21.15.90:0/1835317107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b380b94a0 0x7f1b3814d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:49.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.673+0000 7f1b3fc00640 1 --2- 172.21.15.90:0/1835317107 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b3806cc40 0x7f1b381e6930 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:49.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.673+0000 7f1b3fc00640 1 -- 172.21.15.90:0/1835317107 >> 172.21.15.90:0/1835317107 conn(0x7f1b38072c40 msgr2=0x7f1b380ce3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:49.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.673+0000 7f1b3fc00640 1 -- 172.21.15.90:0/1835317107 shutdown_connections 2024-03-24T01:32:49.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.673+0000 7f1b3fc00640 1 -- 172.21.15.90:0/1835317107 wait complete. 2024-03-24T01:32:49.690 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":324,"stamp":"2024-03-24T01:32:47.675626+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":256756,"kb_used_data":7304,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717964,"statfs":{"total":773094113280,"available":772831195136,"internally_reserved":0,"allocated":7479296,"data_stored":4046352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12747,"internal_metadata":255315509},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.998314"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.707707+0000","last_change":"2024-03-24T01:32:43.707707+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.707707+0000","last_undegraded":"2024-03-24T01:32:43.707707+0000","last_fullsized":"2024-03-24T01:32:43.707707+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.708057+0000","last_change":"2024-03-24T01:32:43.708057+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.708057+0000","last_undegraded":"2024-03-24T01:32:43.708057+0000","last_fullsized":"2024-03-24T01:32:43.708057+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.713931+0000","last_change":"2024-03-24T01:32:43.713931+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.713931+0000","last_undegraded":"2024-03-24T01:32:43.713931+0000","last_fullsized":"2024-03-24T01:32:43.713931+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.699612+0000","last_change":"2024-03-24T01:32:43.699612+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.699612+0000","last_undegraded":"2024-03-24T01:32:43.699612+0000","last_fullsized":"2024-03-24T01:32:43.699612+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.702612+0000","last_change":"2024-03-24T01:32:43.702612+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.702612+0000","last_undegraded":"2024-03-24T01:32:43.702612+0000","last_fullsized":"2024-03-24T01:32:43.702612+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.702504+0000","last_change":"2024-03-24T01:32:43.702504+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.702504+0000","last_undegraded":"2024-03-24T01:32:43.702504+0000","last_fullsized":"2024-03-24T01:32:43.702504+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.702766+0000","last_change":"2024-03-24T01:32:43.702766+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.702766+0000","last_undegraded":"2024-03-24T01:32:43.702766+0000","last_fullsized":"2024-03-24T01:32:43.702766+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.705564+0000","last_change":"2024-03-24T01:32:43.705564+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.705564+0000","last_undegraded":"2024-03-24T01:32:43.705564+0000","last_fullsized":"2024-03-24T01:32:43.705564+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.716331+0000","last_change":"2024-03-24T01:32:43.716331+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.716331+0000","last_undegraded":"2024-03-24T01:32:43.716331+0000","last_fullsized":"2024-03-24T01:32:43.716331+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.714938+0000","last_change":"2024-03-24T01:32:43.714938+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.714938+0000","last_undegraded":"2024-03-24T01:32:43.714938+0000","last_fullsized":"2024-03-24T01:32:43.714938+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.701980+0000","last_change":"2024-03-24T01:32:43.701980+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.701980+0000","last_undegraded":"2024-03-24T01:32:43.701980+0000","last_fullsized":"2024-03-24T01:32:43.701980+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.712314+0000","last_change":"2024-03-24T01:32:43.712314+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.712314+0000","last_undegraded":"2024-03-24T01:32:43.712314+0000","last_fullsized":"2024-03-24T01:32:43.712314+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.708206+0000","last_change":"2024-03-24T01:32:43.708206+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.708206+0000","last_undegraded":"2024-03-24T01:32:43.708206+0000","last_fullsized":"2024-03-24T01:32:43.708206+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.710509+0000","last_change":"2024-03-24T01:32:43.710509+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.710509+0000","last_undegraded":"2024-03-24T01:32:43.710509+0000","last_fullsized":"2024-03-24T01:32:43.710509+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.711385+0000","last_change":"2024-03-24T01:32:43.711385+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.711385+0000","last_undegraded":"2024-03-24T01:32:43.711385+0000","last_fullsized":"2024-03-24T01:32:43.711385+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.702194+0000","last_change":"2024-03-24T01:32:43.702194+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.702194+0000","last_undegraded":"2024-03-24T01:32:43.702194+0000","last_fullsized":"2024-03-24T01:32:43.702194+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.707315+0000","last_change":"2024-03-24T01:32:43.707315+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.707315+0000","last_undegraded":"2024-03-24T01:32:43.707315+0000","last_fullsized":"2024-03-24T01:32:43.707315+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.698533+0000","last_change":"2024-03-24T01:32:43.698533+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.698533+0000","last_undegraded":"2024-03-24T01:32:43.698533+0000","last_fullsized":"2024-03-24T01:32:43.698533+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.697420+0000","last_change":"2024-03-24T01:32:43.697420+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.697420+0000","last_undegraded":"2024-03-24T01:32:43.697420+0000","last_fullsized":"2024-03-24T01:32:43.697420+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.703125+0000","last_change":"2024-03-24T01:32:43.703125+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.703125+0000","last_undegraded":"2024-03-24T01:32:43.703125+0000","last_fullsized":"2024-03-24T01:32:43.703125+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.707468+0000","last_change":"2024-03-24T01:32:43.707468+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.707468+0000","last_undegraded":"2024-03-24T01:32:43.707468+0000","last_fullsized":"2024-03-24T01:32:43.707468+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.704614+0000","last_change":"2024-03-24T01:32:43.704614+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.704614+0000","last_undegraded":"2024-03-24T01:32:43.704614+0000","last_fullsized":"2024-03-24T01:32:43.704614+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.704122+0000","last_change":"2024-03-24T01:32:43.704122+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.704122+0000","last_undegraded":"2024-03-24T01:32:43.704122+0000","last_fullsized":"2024-03-24T01:32:43.704122+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.711724+0000","last_change":"2024-03-24T01:32:43.711724+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.711724+0000","last_undegraded":"2024-03-24T01:32:43.711724+0000","last_fullsized":"2024-03-24T01:32:43.711724+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.711476+0000","last_change":"2024-03-24T01:32:43.711476+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.711476+0000","last_undegraded":"2024-03-24T01:32:43.711476+0000","last_fullsized":"2024-03-24T01:32:43.711476+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.713052+0000","last_change":"2024-03-24T01:32:43.713052+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.713052+0000","last_undegraded":"2024-03-24T01:32:43.713052+0000","last_fullsized":"2024-03-24T01:32:43.713052+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.701299+0000","last_change":"2024-03-24T01:32:43.701299+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.701299+0000","last_undegraded":"2024-03-24T01:32:43.701299+0000","last_fullsized":"2024-03-24T01:32:43.701299+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.703917+0000","last_change":"2024-03-24T01:32:43.703917+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.703917+0000","last_undegraded":"2024-03-24T01:32:43.703917+0000","last_fullsized":"2024-03-24T01:32:43.703917+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.711697+0000","last_change":"2024-03-24T01:32:43.711697+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.711697+0000","last_undegraded":"2024-03-24T01:32:43.711697+0000","last_fullsized":"2024-03-24T01:32:43.711697+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.709236+0000","last_change":"2024-03-24T01:32:43.709236+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.709236+0000","last_undegraded":"2024-03-24T01:32:43.709236+0000","last_fullsized":"2024-03-24T01:32:43.709236+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.696424+0000","last_change":"2024-03-24T01:32:43.696424+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.696424+0000","last_undegraded":"2024-03-24T01:32:43.696424+0000","last_fullsized":"2024-03-24T01:32:43.696424+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.702852+0000","last_change":"2024-03-24T01:32:43.702852+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.702852+0000","last_undegraded":"2024-03-24T01:32:43.702852+0000","last_fullsized":"2024-03-24T01:32:43.702852+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.701720+0000","last_change":"2024-03-24T01:32:43.701720+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.701720+0000","last_undegraded":"2024-03-24T01:32:43.701720+0000","last_fullsized":"2024-03-24T01:32:43.701720+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.729973+0000","last_change":"2024-03-24T01:32:43.729112+0000","last_active":"2024-03-24T01:32:43.729973+0000","last_peered":"2024-03-24T01:32:43.729973+0000","last_clean":"2024-03-24T01:32:43.729973+0000","last_became_active":"2024-03-24T01:32:43.728882+0000","last_became_peered":"2024-03-24T01:32:43.728882+0000","last_unstale":"2024-03-24T01:32:43.729973+0000","last_undegraded":"2024-03-24T01:32:43.729973+0000","last_fullsized":"2024-03-24T01:32:43.729973+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:19:06.949970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.700467+0000","last_change":"2024-03-24T01:32:43.700467+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.700467+0000","last_undegraded":"2024-03-24T01:32:43.700467+0000","last_fullsized":"2024-03-24T01:32:43.700467+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.726029+0000","last_change":"2024-03-24T01:32:43.723642+0000","last_active":"2024-03-24T01:32:43.726029+0000","last_peered":"2024-03-24T01:32:43.726029+0000","last_clean":"2024-03-24T01:32:43.726029+0000","last_became_active":"2024-03-24T01:32:43.723374+0000","last_became_peered":"2024-03-24T01:32:43.723374+0000","last_unstale":"2024-03-24T01:32:43.726029+0000","last_undegraded":"2024-03-24T01:32:43.726029+0000","last_fullsized":"2024-03-24T01:32:43.726029+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:02.791894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.701118+0000","last_change":"2024-03-24T01:32:43.701118+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.701118+0000","last_undegraded":"2024-03-24T01:32:43.701118+0000","last_fullsized":"2024-03-24T01:32:43.701118+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.724435+0000","last_change":"2024-03-24T01:32:43.722317+0000","last_active":"2024-03-24T01:32:43.724435+0000","last_peered":"2024-03-24T01:32:43.724435+0000","last_clean":"2024-03-24T01:32:43.724435+0000","last_became_active":"2024-03-24T01:32:43.721576+0000","last_became_peered":"2024-03-24T01:32:43.721576+0000","last_unstale":"2024-03-24T01:32:43.724435+0000","last_undegraded":"2024-03-24T01:32:43.724435+0000","last_fullsized":"2024-03-24T01:32:43.724435+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:10:35.120169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.716728+0000","last_change":"2024-03-24T01:32:43.716728+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.716728+0000","last_undegraded":"2024-03-24T01:32:43.716728+0000","last_fullsized":"2024-03-24T01:32:43.716728+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.717511+0000","last_change":"2024-03-24T01:32:43.713108+0000","last_active":"2024-03-24T01:32:43.717511+0000","last_peered":"2024-03-24T01:32:43.717511+0000","last_clean":"2024-03-24T01:32:43.717511+0000","last_became_active":"2024-03-24T01:32:43.712800+0000","last_became_peered":"2024-03-24T01:32:43.712800+0000","last_unstale":"2024-03-24T01:32:43.717511+0000","last_undegraded":"2024-03-24T01:32:43.717511+0000","last_fullsized":"2024-03-24T01:32:43.717511+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:34.647291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.701602+0000","last_change":"2024-03-24T01:32:43.701602+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.701602+0000","last_undegraded":"2024-03-24T01:32:43.701602+0000","last_fullsized":"2024-03-24T01:32:43.701602+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.722195+0000","last_change":"2024-03-24T01:32:43.718182+0000","last_active":"2024-03-24T01:32:43.722195+0000","last_peered":"2024-03-24T01:32:43.722195+0000","last_clean":"2024-03-24T01:32:43.722195+0000","last_became_active":"2024-03-24T01:32:43.717674+0000","last_became_peered":"2024-03-24T01:32:43.717674+0000","last_unstale":"2024-03-24T01:32:43.722195+0000","last_undegraded":"2024-03-24T01:32:43.722195+0000","last_fullsized":"2024-03-24T01:32:43.722195+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:39:22.546505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.700292+0000","last_change":"2024-03-24T01:32:43.700292+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.700292+0000","last_undegraded":"2024-03-24T01:32:43.700292+0000","last_fullsized":"2024-03-24T01:32:43.700292+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.722624+0000","last_change":"2024-03-24T01:32:43.717225+0000","last_active":"2024-03-24T01:32:43.722624+0000","last_peered":"2024-03-24T01:32:43.722624+0000","last_clean":"2024-03-24T01:32:43.722624+0000","last_became_active":"2024-03-24T01:32:43.716994+0000","last_became_peered":"2024-03-24T01:32:43.716994+0000","last_unstale":"2024-03-24T01:32:43.722624+0000","last_undegraded":"2024-03-24T01:32:43.722624+0000","last_fullsized":"2024-03-24T01:32:43.722624+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:14:55.272028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.698786+0000","last_change":"2024-03-24T01:32:43.698786+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.698786+0000","last_undegraded":"2024-03-24T01:32:43.698786+0000","last_fullsized":"2024-03-24T01:32:43.698786+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.722633+0000","last_change":"2024-03-24T01:32:43.719216+0000","last_active":"2024-03-24T01:32:43.722633+0000","last_peered":"2024-03-24T01:32:43.722633+0000","last_clean":"2024-03-24T01:32:43.722633+0000","last_became_active":"2024-03-24T01:32:43.718879+0000","last_became_peered":"2024-03-24T01:32:43.718879+0000","last_unstale":"2024-03-24T01:32:43.722633+0000","last_undegraded":"2024-03-24T01:32:43.722633+0000","last_fullsized":"2024-03-24T01:32:43.722633+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:16:58.498006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.702537+0000","last_change":"2024-03-24T01:32:43.702537+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.702537+0000","last_undegraded":"2024-03-24T01:32:43.702537+0000","last_fullsized":"2024-03-24T01:32:43.702537+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.723762+0000","last_change":"2024-03-24T01:32:43.720754+0000","last_active":"2024-03-24T01:32:43.723762+0000","last_peered":"2024-03-24T01:32:43.723762+0000","last_clean":"2024-03-24T01:32:43.723762+0000","last_became_active":"2024-03-24T01:32:43.720465+0000","last_became_peered":"2024-03-24T01:32:43.720465+0000","last_unstale":"2024-03-24T01:32:43.723762+0000","last_undegraded":"2024-03-24T01:32:43.723762+0000","last_fullsized":"2024-03-24T01:32:43.723762+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:48:15.469032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.703279+0000","last_change":"2024-03-24T01:32:43.703279+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.703279+0000","last_undegraded":"2024-03-24T01:32:43.703279+0000","last_fullsized":"2024-03-24T01:32:43.703279+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.734285+0000","last_change":"2024-03-24T01:32:43.729179+0000","last_active":"2024-03-24T01:32:43.734285+0000","last_peered":"2024-03-24T01:32:43.734285+0000","last_clean":"2024-03-24T01:32:43.734285+0000","last_became_active":"2024-03-24T01:32:43.728892+0000","last_became_peered":"2024-03-24T01:32:43.728892+0000","last_unstale":"2024-03-24T01:32:43.734285+0000","last_undegraded":"2024-03-24T01:32:43.734285+0000","last_fullsized":"2024-03-24T01:32:43.734285+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:31:44.560907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.703626+0000","last_change":"2024-03-24T01:32:43.703626+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.703626+0000","last_undegraded":"2024-03-24T01:32:43.703626+0000","last_fullsized":"2024-03-24T01:32:43.703626+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.718845+0000","last_change":"2024-03-24T01:32:43.715851+0000","last_active":"2024-03-24T01:32:43.718845+0000","last_peered":"2024-03-24T01:32:43.718845+0000","last_clean":"2024-03-24T01:32:43.718845+0000","last_became_active":"2024-03-24T01:32:43.715474+0000","last_became_peered":"2024-03-24T01:32:43.715474+0000","last_unstale":"2024-03-24T01:32:43.718845+0000","last_undegraded":"2024-03-24T01:32:43.718845+0000","last_fullsized":"2024-03-24T01:32:43.718845+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:57:39.379197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.696544+0000","last_change":"2024-03-24T01:32:43.696544+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.696544+0000","last_undegraded":"2024-03-24T01:32:43.696544+0000","last_fullsized":"2024-03-24T01:32:43.696544+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.731495+0000","last_change":"2024-03-24T01:32:43.725497+0000","last_active":"2024-03-24T01:32:43.731495+0000","last_peered":"2024-03-24T01:32:43.731495+0000","last_clean":"2024-03-24T01:32:43.731495+0000","last_became_active":"2024-03-24T01:32:43.724751+0000","last_became_peered":"2024-03-24T01:32:43.724751+0000","last_unstale":"2024-03-24T01:32:43.731495+0000","last_undegraded":"2024-03-24T01:32:43.731495+0000","last_fullsized":"2024-03-24T01:32:43.731495+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:29.157467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.700996+0000","last_change":"2024-03-24T01:32:43.700996+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.700996+0000","last_undegraded":"2024-03-24T01:32:43.700996+0000","last_fullsized":"2024-03-24T01:32:43.700996+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.723022+0000","last_change":"2024-03-24T01:32:43.716683+0000","last_active":"2024-03-24T01:32:43.723022+0000","last_peered":"2024-03-24T01:32:43.723022+0000","last_clean":"2024-03-24T01:32:43.723022+0000","last_became_active":"2024-03-24T01:32:43.716407+0000","last_became_peered":"2024-03-24T01:32:43.716407+0000","last_unstale":"2024-03-24T01:32:43.723022+0000","last_undegraded":"2024-03-24T01:32:43.723022+0000","last_fullsized":"2024-03-24T01:32:43.723022+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:31:19.992387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.695279+0000","last_change":"2024-03-24T01:32:43.695279+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.695279+0000","last_undegraded":"2024-03-24T01:32:43.695279+0000","last_fullsized":"2024-03-24T01:32:43.695279+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.731501+0000","last_change":"2024-03-24T01:32:43.725509+0000","last_active":"2024-03-24T01:32:43.731501+0000","last_peered":"2024-03-24T01:32:43.731501+0000","last_clean":"2024-03-24T01:32:43.731501+0000","last_became_active":"2024-03-24T01:32:43.725142+0000","last_became_peered":"2024-03-24T01:32:43.725142+0000","last_unstale":"2024-03-24T01:32:43.731501+0000","last_undegraded":"2024-03-24T01:32:43.731501+0000","last_fullsized":"2024-03-24T01:32:43.731501+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:43:54.681008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.711736+0000","last_change":"2024-03-24T01:32:43.711736+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.711736+0000","last_undegraded":"2024-03-24T01:32:43.711736+0000","last_fullsized":"2024-03-24T01:32:43.711736+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.724233+0000","last_change":"2024-03-24T01:32:43.720435+0000","last_active":"2024-03-24T01:32:43.724233+0000","last_peered":"2024-03-24T01:32:43.724233+0000","last_clean":"2024-03-24T01:32:43.724233+0000","last_became_active":"2024-03-24T01:32:43.720135+0000","last_became_peered":"2024-03-24T01:32:43.720135+0000","last_unstale":"2024-03-24T01:32:43.724233+0000","last_undegraded":"2024-03-24T01:32:43.724233+0000","last_fullsized":"2024-03-24T01:32:43.724233+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:31:19.992387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.695588+0000","last_change":"2024-03-24T01:32:43.695588+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.695588+0000","last_undegraded":"2024-03-24T01:32:43.695588+0000","last_fullsized":"2024-03-24T01:32:43.695588+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.731473+0000","last_change":"2024-03-24T01:32:43.725306+0000","last_active":"2024-03-24T01:32:43.731473+0000","last_peered":"2024-03-24T01:32:43.731473+0000","last_clean":"2024-03-24T01:32:43.731473+0000","last_became_active":"2024-03-24T01:32:43.724101+0000","last_became_peered":"2024-03-24T01:32:43.724101+0000","last_unstale":"2024-03-24T01:32:43.731473+0000","last_undegraded":"2024-03-24T01:32:43.731473+0000","last_fullsized":"2024-03-24T01:32:43.731473+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:19:06.949970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.699882+0000","last_change":"2024-03-24T01:32:43.699882+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.699882+0000","last_undegraded":"2024-03-24T01:32:43.699882+0000","last_fullsized":"2024-03-24T01:32:43.699882+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.726036+0000","last_change":"2024-03-24T01:32:43.720664+0000","last_active":"2024-03-24T01:32:43.726036+0000","last_peered":"2024-03-24T01:32:43.726036+0000","last_clean":"2024-03-24T01:32:43.726036+0000","last_became_active":"2024-03-24T01:32:43.720338+0000","last_became_peered":"2024-03-24T01:32:43.720338+0000","last_unstale":"2024-03-24T01:32:43.726036+0000","last_undegraded":"2024-03-24T01:32:43.726036+0000","last_fullsized":"2024-03-24T01:32:43.726036+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:48:15.469032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.712601+0000","last_change":"2024-03-24T01:32:43.712601+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.712601+0000","last_undegraded":"2024-03-24T01:32:43.712601+0000","last_fullsized":"2024-03-24T01:32:43.712601+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.723816+0000","last_change":"2024-03-24T01:32:43.722152+0000","last_active":"2024-03-24T01:32:43.723816+0000","last_peered":"2024-03-24T01:32:43.723816+0000","last_clean":"2024-03-24T01:32:43.723816+0000","last_became_active":"2024-03-24T01:32:43.721855+0000","last_became_peered":"2024-03-24T01:32:43.721855+0000","last_unstale":"2024-03-24T01:32:43.723816+0000","last_undegraded":"2024-03-24T01:32:43.723816+0000","last_fullsized":"2024-03-24T01:32:43.723816+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:31:15.858645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.696181+0000","last_change":"2024-03-24T01:32:43.696181+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.696181+0000","last_undegraded":"2024-03-24T01:32:43.696181+0000","last_fullsized":"2024-03-24T01:32:43.696181+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.731711+0000","last_change":"2024-03-24T01:32:43.726876+0000","last_active":"2024-03-24T01:32:43.731711+0000","last_peered":"2024-03-24T01:32:43.731711+0000","last_clean":"2024-03-24T01:32:43.731711+0000","last_became_active":"2024-03-24T01:32:43.726628+0000","last_became_peered":"2024-03-24T01:32:43.726628+0000","last_unstale":"2024-03-24T01:32:43.731711+0000","last_undegraded":"2024-03-24T01:32:43.731711+0000","last_fullsized":"2024-03-24T01:32:43.731711+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:11:39.983567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.697338+0000","last_change":"2024-03-24T01:32:43.697338+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.697338+0000","last_undegraded":"2024-03-24T01:32:43.697338+0000","last_fullsized":"2024-03-24T01:32:43.697338+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.731521+0000","last_change":"2024-03-24T01:32:43.725389+0000","last_active":"2024-03-24T01:32:43.731521+0000","last_peered":"2024-03-24T01:32:43.731521+0000","last_clean":"2024-03-24T01:32:43.731521+0000","last_became_active":"2024-03-24T01:32:43.724956+0000","last_became_peered":"2024-03-24T01:32:43.724956+0000","last_unstale":"2024-03-24T01:32:43.731521+0000","last_undegraded":"2024-03-24T01:32:43.731521+0000","last_fullsized":"2024-03-24T01:32:43.731521+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:31:15.858645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.699266+0000","last_change":"2024-03-24T01:32:43.699266+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.699266+0000","last_undegraded":"2024-03-24T01:32:43.699266+0000","last_fullsized":"2024-03-24T01:32:43.699266+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.723870+0000","last_change":"2024-03-24T01:32:43.721896+0000","last_active":"2024-03-24T01:32:43.723870+0000","last_peered":"2024-03-24T01:32:43.723870+0000","last_clean":"2024-03-24T01:32:43.723870+0000","last_became_active":"2024-03-24T01:32:43.720277+0000","last_became_peered":"2024-03-24T01:32:43.720277+0000","last_unstale":"2024-03-24T01:32:43.723870+0000","last_undegraded":"2024-03-24T01:32:43.723870+0000","last_fullsized":"2024-03-24T01:32:43.723870+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:48:17.796412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.702327+0000","last_change":"2024-03-24T01:32:43.702327+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.702327+0000","last_undegraded":"2024-03-24T01:32:43.702327+0000","last_fullsized":"2024-03-24T01:32:43.702327+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.715972+0000","last_change":"2024-03-24T01:32:43.707322+0000","last_active":"2024-03-24T01:32:43.715972+0000","last_peered":"2024-03-24T01:32:43.715972+0000","last_clean":"2024-03-24T01:32:43.715972+0000","last_became_active":"2024-03-24T01:32:43.706928+0000","last_became_peered":"2024-03-24T01:32:43.706928+0000","last_unstale":"2024-03-24T01:32:43.715972+0000","last_undegraded":"2024-03-24T01:32:43.715972+0000","last_fullsized":"2024-03-24T01:32:43.715972+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:18.017172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.711175+0000","last_change":"2024-03-24T01:32:43.711175+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.711175+0000","last_undegraded":"2024-03-24T01:32:43.711175+0000","last_fullsized":"2024-03-24T01:32:43.711175+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.726365+0000","last_change":"2024-03-24T01:32:43.724088+0000","last_active":"2024-03-24T01:32:43.726365+0000","last_peered":"2024-03-24T01:32:43.726365+0000","last_clean":"2024-03-24T01:32:43.726365+0000","last_became_active":"2024-03-24T01:32:43.722732+0000","last_became_peered":"2024-03-24T01:32:43.722732+0000","last_unstale":"2024-03-24T01:32:43.726365+0000","last_undegraded":"2024-03-24T01:32:43.726365+0000","last_fullsized":"2024-03-24T01:32:43.726365+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:29.157467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.694328+0000","last_change":"2024-03-24T01:32:43.694328+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.694328+0000","last_undegraded":"2024-03-24T01:32:43.694328+0000","last_fullsized":"2024-03-24T01:32:43.694328+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.724217+0000","last_change":"2024-03-24T01:32:43.720278+0000","last_active":"2024-03-24T01:32:43.724217+0000","last_peered":"2024-03-24T01:32:43.724217+0000","last_clean":"2024-03-24T01:32:43.724217+0000","last_became_active":"2024-03-24T01:32:43.719995+0000","last_became_peered":"2024-03-24T01:32:43.719995+0000","last_unstale":"2024-03-24T01:32:43.724217+0000","last_undegraded":"2024-03-24T01:32:43.724217+0000","last_fullsized":"2024-03-24T01:32:43.724217+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:11.809594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.699505+0000","last_change":"2024-03-24T01:32:43.699505+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.699505+0000","last_undegraded":"2024-03-24T01:32:43.699505+0000","last_fullsized":"2024-03-24T01:32:43.699505+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.734071+0000","last_change":"2024-03-24T01:32:43.729534+0000","last_active":"2024-03-24T01:32:43.734071+0000","last_peered":"2024-03-24T01:32:43.734071+0000","last_clean":"2024-03-24T01:32:43.734071+0000","last_became_active":"2024-03-24T01:32:43.729214+0000","last_became_peered":"2024-03-24T01:32:43.729214+0000","last_unstale":"2024-03-24T01:32:43.734071+0000","last_undegraded":"2024-03-24T01:32:43.734071+0000","last_fullsized":"2024-03-24T01:32:43.734071+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:43:54.681008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.705306+0000","last_change":"2024-03-24T01:32:43.705306+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.705306+0000","last_undegraded":"2024-03-24T01:32:43.705306+0000","last_fullsized":"2024-03-24T01:32:43.705306+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.723223+0000","last_change":"2024-03-24T01:32:43.720477+0000","last_active":"2024-03-24T01:32:43.723223+0000","last_peered":"2024-03-24T01:32:43.723223+0000","last_clean":"2024-03-24T01:32:43.723223+0000","last_became_active":"2024-03-24T01:32:43.720192+0000","last_became_peered":"2024-03-24T01:32:43.720192+0000","last_unstale":"2024-03-24T01:32:43.723223+0000","last_undegraded":"2024-03-24T01:32:43.723223+0000","last_fullsized":"2024-03-24T01:32:43.723223+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:19:06.949970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.708484+0000","last_change":"2024-03-24T01:32:43.708484+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.708484+0000","last_undegraded":"2024-03-24T01:32:43.708484+0000","last_fullsized":"2024-03-24T01:32:43.708484+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.726703+0000","last_change":"2024-03-24T01:32:43.725481+0000","last_active":"2024-03-24T01:32:43.726703+0000","last_peered":"2024-03-24T01:32:43.726703+0000","last_clean":"2024-03-24T01:32:43.726703+0000","last_became_active":"2024-03-24T01:32:43.725185+0000","last_became_peered":"2024-03-24T01:32:43.725185+0000","last_unstale":"2024-03-24T01:32:43.726703+0000","last_undegraded":"2024-03-24T01:32:43.726703+0000","last_fullsized":"2024-03-24T01:32:43.726703+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:15:19.800763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.698042+0000","last_change":"2024-03-24T01:32:43.698042+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.698042+0000","last_undegraded":"2024-03-24T01:32:43.698042+0000","last_fullsized":"2024-03-24T01:32:43.698042+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.719313+0000","last_change":"2024-03-24T01:32:43.714827+0000","last_active":"2024-03-24T01:32:43.719313+0000","last_peered":"2024-03-24T01:32:43.719313+0000","last_clean":"2024-03-24T01:32:43.719313+0000","last_became_active":"2024-03-24T01:32:43.714552+0000","last_became_peered":"2024-03-24T01:32:43.714552+0000","last_unstale":"2024-03-24T01:32:43.719313+0000","last_undegraded":"2024-03-24T01:32:43.719313+0000","last_fullsized":"2024-03-24T01:32:43.719313+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:39:25.511297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.693643+0000","last_change":"2024-03-24T01:32:43.693643+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.693643+0000","last_undegraded":"2024-03-24T01:32:43.693643+0000","last_fullsized":"2024-03-24T01:32:43.693643+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.731381+0000","last_change":"2024-03-24T01:32:43.724564+0000","last_active":"2024-03-24T01:32:43.731381+0000","last_peered":"2024-03-24T01:32:43.731381+0000","last_clean":"2024-03-24T01:32:43.731381+0000","last_became_active":"2024-03-24T01:32:43.724051+0000","last_became_peered":"2024-03-24T01:32:43.724051+0000","last_unstale":"2024-03-24T01:32:43.731381+0000","last_undegraded":"2024-03-24T01:32:43.731381+0000","last_fullsized":"2024-03-24T01:32:43.731381+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:31:19.992387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.695082+0000","last_change":"2024-03-24T01:32:43.695082+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.695082+0000","last_undegraded":"2024-03-24T01:32:43.695082+0000","last_fullsized":"2024-03-24T01:32:43.695082+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.718069+0000","last_change":"2024-03-24T01:32:43.715420+0000","last_active":"2024-03-24T01:32:43.718069+0000","last_peered":"2024-03-24T01:32:43.718069+0000","last_clean":"2024-03-24T01:32:43.718069+0000","last_became_active":"2024-03-24T01:32:43.714725+0000","last_became_peered":"2024-03-24T01:32:43.714725+0000","last_unstale":"2024-03-24T01:32:43.718069+0000","last_undegraded":"2024-03-24T01:32:43.718069+0000","last_fullsized":"2024-03-24T01:32:43.718069+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:39:09.134793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.699965+0000","last_change":"2024-03-24T01:32:43.699965+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.699965+0000","last_undegraded":"2024-03-24T01:32:43.699965+0000","last_fullsized":"2024-03-24T01:32:43.699965+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.722608+0000","last_change":"2024-03-24T01:32:43.719188+0000","last_active":"2024-03-24T01:32:43.722608+0000","last_peered":"2024-03-24T01:32:43.722608+0000","last_clean":"2024-03-24T01:32:43.722608+0000","last_became_active":"2024-03-24T01:32:43.718755+0000","last_became_peered":"2024-03-24T01:32:43.718755+0000","last_unstale":"2024-03-24T01:32:43.722608+0000","last_undegraded":"2024-03-24T01:32:43.722608+0000","last_fullsized":"2024-03-24T01:32:43.722608+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:36:56.063762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.703677+0000","last_change":"2024-03-24T01:32:43.703677+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.703677+0000","last_undegraded":"2024-03-24T01:32:43.703677+0000","last_fullsized":"2024-03-24T01:32:43.703677+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.719439+0000","last_change":"2024-03-24T01:32:43.714616+0000","last_active":"2024-03-24T01:32:43.719439+0000","last_peered":"2024-03-24T01:32:43.719439+0000","last_clean":"2024-03-24T01:32:43.719439+0000","last_became_active":"2024-03-24T01:32:43.714216+0000","last_became_peered":"2024-03-24T01:32:43.714216+0000","last_unstale":"2024-03-24T01:32:43.719439+0000","last_undegraded":"2024-03-24T01:32:43.719439+0000","last_fullsized":"2024-03-24T01:32:43.719439+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:51:18.569189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-03-24T01:32:43.695893+0000","last_change":"2024-03-24T01:32:43.695893+0000","last_active":"2024-03-24T01:32:43.676436+0000","last_peered":"2024-03-24T01:32:43.676436+0000","last_clean":"2024-03-24T01:32:43.676436+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T01:32:43.695893+0000","last_undegraded":"2024-03-24T01:32:43.695893+0000","last_fullsized":"2024-03-24T01:32:43.695893+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.724138+0000","last_change":"2024-03-24T01:32:43.719380+0000","last_active":"2024-03-24T01:32:43.724138+0000","last_peered":"2024-03-24T01:32:43.724138+0000","last_clean":"2024-03-24T01:32:43.724138+0000","last_became_active":"2024-03-24T01:32:43.718702+0000","last_became_peered":"2024-03-24T01:32:43.718702+0000","last_unstale":"2024-03-24T01:32:43.724138+0000","last_undegraded":"2024-03-24T01:32:43.724138+0000","last_fullsized":"2024-03-24T01:32:43.724138+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:59:11.903374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":255,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-24T01:32:43.714595+0000","last_change":"2024-03-24T01:24:01.545855+0000","last_active":"2024-03-24T01:32:43.714595+0000","last_peered":"2024-03-24T01:32:43.714595+0000","last_clean":"2024-03-24T01:32:43.714595+0000","last_became_active":"2024-03-24T01:24:01.543811+0000","last_became_peered":"2024-03-24T01:24:01.543811+0000","last_unstale":"2024-03-24T01:32:43.714595+0000","last_undegraded":"2024-03-24T01:32:43.714595+0000","last_fullsized":"2024-03-24T01:32:43.714595+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_clean_scrub_stamp":"2024-03-24T01:23:59.861564+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:03.682631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574956,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32532,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339308,"statfs":{"total":96636764160,"available":96603451392,"internally_reserved":0,"allocated":1380352,"data_stored":948570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574956,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31948,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339892,"statfs":{"total":96636764160,"available":96604049408,"internally_reserved":0,"allocated":786432,"data_stored":358202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574957,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31948,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339892,"statfs":{"total":96636764160,"available":96604049408,"internally_reserved":0,"allocated":786432,"data_stored":358202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574956,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31948,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339892,"statfs":{"total":96636764160,"available":96604049408,"internally_reserved":0,"allocated":786432,"data_stored":358202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574956,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31952,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339888,"statfs":{"total":96636764160,"available":96604045312,"internally_reserved":0,"allocated":786432,"data_stored":358202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574957,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31952,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339888,"statfs":{"total":96636764160,"available":96604045312,"internally_reserved":0,"allocated":786432,"data_stored":358202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574956,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31948,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339892,"statfs":{"total":96636764160,"available":96604049408,"internally_reserved":0,"allocated":786432,"data_stored":358202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574957,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32528,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339312,"statfs":{"total":96636764160,"available":96603455488,"internally_reserved":0,"allocated":1380352,"data_stored":948570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T01:32:49.693 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T01:32:49.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.928+0000 7fa9fcc00640 1 Processor -- start 2024-03-24T01:32:49.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.928+0000 7fa9fcc00640 1 -- start start 2024-03-24T01:32:49.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.930+0000 7fa9fcc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9f81603e0 0x7fa9f806b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:49.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.930+0000 7fa9fcc00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa9f8145430 0x7fa9f806b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:49.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.930+0000 7fa9fcc00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9f80cd490 con 0x7fa9f80cc150 2024-03-24T01:32:49.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.930+0000 7fa9fcc00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9f80cd270 con 0x7fa9f8145430 2024-03-24T01:32:49.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.930+0000 7fa9fcc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f815f900 con 0x7fa9f81603e0 2024-03-24T01:32:49.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.930+0000 7fa9f5e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa9f80cc150 0x7fa9f80ccb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43118/0 (socket says 172.21.15.90:43118) 2024-03-24T01:32:49.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.930+0000 7fa9f5e00640 1 -- 172.21.15.90:0/3124431081 learned_addr learned my addr 172.21.15.90:0/3124431081 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:32:49.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.930+0000 7fa9f6800640 1 --2- 172.21.15.90:0/3124431081 >> v2:172.21.15.90:3300/0 conn(0x7fa9f8145430 0x7fa9f806b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:49.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.930+0000 7fa9efe00640 1 --2- 172.21.15.90:0/3124431081 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9f81603e0 0x7fa9f806b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:49.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.930+0000 7fa9f5400640 1 -- 172.21.15.90:0/3124431081 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1528324155 0 0) 0x7fa9f80cd490 con 0x7fa9f80cc150 2024-03-24T01:32:49.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.930+0000 7fa9f5400640 1 -- 172.21.15.90:0/3124431081 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9dc003520 con 0x7fa9f80cc150 2024-03-24T01:32:49.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.931+0000 7fa9f6800640 1 -- 172.21.15.90:0/3124431081 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9f81603e0 msgr2=0x7fa9f806b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:49.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.931+0000 7fa9f6800640 1 --2- 172.21.15.90:0/3124431081 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9f81603e0 0x7fa9f806b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:49.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.931+0000 7fa9f6800640 1 -- 172.21.15.90:0/3124431081 >> v1:172.21.15.90:6789/0 conn(0x7fa9f80cc150 legacy=0x7fa9f80ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:49.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.931+0000 7fa9f6800640 1 -- 172.21.15.90:0/3124431081 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f815fbe0 con 0x7fa9f8145430 2024-03-24T01:32:49.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.932+0000 7fa9f6800640 1 --2- 172.21.15.90:0/3124431081 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9f8145430 0x7fa9f806b600 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0009d90 tx=0x7fa9f002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42124d45f6d630f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:49.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.932+0000 7fa9f5400640 1 -- 172.21.15.90:0/3124431081 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f0039070 con 0x7fa9f8145430 2024-03-24T01:32:49.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.932+0000 7fa9f5400640 1 -- 172.21.15.90:0/3124431081 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f002cc50 con 0x7fa9f8145430 2024-03-24T01:32:49.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.932+0000 7fa9f5400640 1 -- 172.21.15.90:0/3124431081 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f0034440 con 0x7fa9f8145430 2024-03-24T01:32:49.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.933+0000 7fa9fcc00640 1 -- 172.21.15.90:0/3124431081 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9f8145430 msgr2=0x7fa9f806b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:49.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.933+0000 7fa9fcc00640 1 --2- 172.21.15.90:0/3124431081 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9f8145430 0x7fa9f806b600 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0009d90 tx=0x7fa9f002bea0 comp rx=0 tx=0).stop 2024-03-24T01:32:49.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.933+0000 7fa9fcc00640 1 -- 172.21.15.90:0/3124431081 shutdown_connections 2024-03-24T01:32:49.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.933+0000 7fa9fcc00640 1 --2- 172.21.15.90:0/3124431081 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9f8145430 0x7fa9f806b600 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:49.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.933+0000 7fa9fcc00640 1 --2- 172.21.15.90:0/3124431081 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9f81603e0 0x7fa9f806b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:49.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.933+0000 7fa9fcc00640 1 -- 172.21.15.90:0/3124431081 >> 172.21.15.90:0/3124431081 conn(0x7fa9f80b1770 msgr2=0x7fa9f80b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:49.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.933+0000 7fa9fcc00640 1 -- 172.21.15.90:0/3124431081 shutdown_connections 2024-03-24T01:32:49.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.933+0000 7fa9fcc00640 1 -- 172.21.15.90:0/3124431081 wait complete. 2024-03-24T01:32:49.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.935+0000 7fa9fcc00640 1 Processor -- start 2024-03-24T01:32:49.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.935+0000 7fa9fcc00640 1 -- start start 2024-03-24T01:32:49.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.936+0000 7fa9fcc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9f80cc150 0x7fa9f815af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:49.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.936+0000 7fa9fcc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9f8145430 0x7fa9f815cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:49.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.936+0000 7fa9fcc00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9f81603e0 0x7fa9f80ada10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:49.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.936+0000 7fa9fcc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f80b38d0 con 0x7fa9f8145430 2024-03-24T01:32:49.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.936+0000 7fa9fcc00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f80b3750 con 0x7fa9f81603e0 2024-03-24T01:32:49.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.936+0000 7fa9fcc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f80b3a50 con 0x7fa9f80cc150 2024-03-24T01:32:49.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.936+0000 7fa9efe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9f8145430 0x7fa9f815cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:49.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.936+0000 7fa9efe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9f8145430 0x7fa9f815cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34432/0 (socket says 172.21.15.90:34432) 2024-03-24T01:32:49.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.936+0000 7fa9efe00640 1 -- 172.21.15.90:0/3463375033 learned_addr learned my addr 172.21.15.90:0/3463375033 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:49.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.936+0000 7fa9efe00640 1 -- 172.21.15.90:0/3463375033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9f80cc150 msgr2=0x7fa9f815af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:49.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.936+0000 7fa9efe00640 1 --2- 172.21.15.90:0/3463375033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9f80cc150 0x7fa9f815af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:49.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.936+0000 7fa9efe00640 1 -- 172.21.15.90:0/3463375033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9f81603e0 msgr2=0x7fa9f80ada10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:49.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.936+0000 7fa9efe00640 1 --2- 172.21.15.90:0/3463375033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9f81603e0 0x7fa9f80ada10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:49.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.936+0000 7fa9efe00640 1 -- 172.21.15.90:0/3463375033 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f81f7600 con 0x7fa9f8145430 2024-03-24T01:32:49.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.937+0000 7fa9efe00640 1 --2- 172.21.15.90:0/3463375033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9f8145430 0x7fa9f815cec0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fa9e000d620 tx=0x7fa9e000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:49.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.937+0000 7fa9ee000640 1 -- 172.21.15.90:0/3463375033 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e000b620 con 0x7fa9f8145430 2024-03-24T01:32:49.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.937+0000 7fa9ee000640 1 -- 172.21.15.90:0/3463375033 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9e0019070 con 0x7fa9f8145430 2024-03-24T01:32:49.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.937+0000 7fa9fcc00640 1 -- 172.21.15.90:0/3463375033 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9f81f78e0 con 0x7fa9f8145430 2024-03-24T01:32:49.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.937+0000 7fa9ee000640 1 -- 172.21.15.90:0/3463375033 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e000ba00 con 0x7fa9f8145430 2024-03-24T01:32:49.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.937+0000 7fa9fcc00640 1 -- 172.21.15.90:0/3463375033 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9f81f7ea0 con 0x7fa9f8145430 2024-03-24T01:32:49.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.938+0000 7fa9fcc00640 1 -- 172.21.15.90:0/3463375033 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9f815c2c0 con 0x7fa9f8145430 2024-03-24T01:32:49.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.939+0000 7fa9ee000640 1 -- 172.21.15.90:0/3463375033 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fa9e000bba0 con 0x7fa9f8145430 2024-03-24T01:32:49.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.940+0000 7fa9ee000640 1 --2- 172.21.15.90:0/3463375033 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa9d406bbd0 0x7fa9d406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:49.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.940+0000 7fa9ee000640 1 -- 172.21.15.90:0/3463375033 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa9e008af90 con 0x7fa9f8145430 2024-03-24T01:32:49.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.940+0000 7fa9f5e00640 1 --2- 172.21.15.90:0/3463375033 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa9d406bbd0 0x7fa9d406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:49.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.941+0000 7fa9f5e00640 1 --2- 172.21.15.90:0/3463375033 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa9d406bbd0 0x7fa9d406e090 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8000f60 tx=0x7fa9e80078f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:49.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:49.944+0000 7fa9ee000640 1 -- 172.21.15.90:0/3463375033 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fa9e0059980 con 0x7fa9f8145430 2024-03-24T01:32:50.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.136+0000 7fa9fcc00640 1 -- 172.21.15.90:0/3463375033 --> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa9f8068060 con 0x7fa9d406bbd0 2024-03-24T01:32:50.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.154+0000 7fa9ee000640 1 -- 172.21.15.90:0/3463375033 <== mgr.4104 v2:172.21.15.120:6832/860392634 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252020 (secure 0 0 0) 0x7fa9f8068060 con 0x7fa9d406bbd0 2024-03-24T01:32:50.158 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:32:50.159 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-24T01:32:50.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.160+0000 7fa9fcc00640 1 -- 172.21.15.90:0/3463375033 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa9d406bbd0 msgr2=0x7fa9d406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:50.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.160+0000 7fa9fcc00640 1 --2- 172.21.15.90:0/3463375033 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa9d406bbd0 0x7fa9d406e090 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8000f60 tx=0x7fa9e80078f0 comp rx=0 tx=0).stop 2024-03-24T01:32:50.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.160+0000 7fa9fcc00640 1 -- 172.21.15.90:0/3463375033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9f8145430 msgr2=0x7fa9f815cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:50.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.160+0000 7fa9fcc00640 1 --2- 172.21.15.90:0/3463375033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9f8145430 0x7fa9f815cec0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fa9e000d620 tx=0x7fa9e000dae0 comp rx=0 tx=0).stop 2024-03-24T01:32:50.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.160+0000 7fa9fcc00640 1 -- 172.21.15.90:0/3463375033 shutdown_connections 2024-03-24T01:32:50.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.160+0000 7fa9fcc00640 1 --2- 172.21.15.90:0/3463375033 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fa9d406bbd0 0x7fa9d406e090 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:50.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.160+0000 7fa9fcc00640 1 --2- 172.21.15.90:0/3463375033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9f8145430 0x7fa9f815cec0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:50.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.160+0000 7fa9fcc00640 1 --2- 172.21.15.90:0/3463375033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9f81603e0 0x7fa9f80ada10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:50.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.160+0000 7fa9fcc00640 1 --2- 172.21.15.90:0/3463375033 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fa9f80cc150 0x7fa9f815af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:50.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.160+0000 7fa9fcc00640 1 -- 172.21.15.90:0/3463375033 >> 172.21.15.90:0/3463375033 conn(0x7fa9f80b1770 msgr2=0x7fa9f80b00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:50.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.160+0000 7fa9fcc00640 1 -- 172.21.15.90:0/3463375033 shutdown_connections 2024-03-24T01:32:50.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.160+0000 7fa9fcc00640 1 -- 172.21.15.90:0/3463375033 wait complete. 2024-03-24T01:32:50.181 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-03-24T01:32:49.676278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":257048,"kb_used_data":7560,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717672,"statfs":{"total":773094113280,"available":772830896128,"internally_reserved":0,"allocated":7741440,"data_stored":4162044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12747,"internal_metadata":255315509},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002027"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.713955+0000","last_change":"2024-03-24T01:32:44.751583+0000","last_active":"2024-03-24T01:32:45.713955+0000","last_peered":"2024-03-24T01:32:45.713955+0000","last_clean":"2024-03-24T01:32:45.713955+0000","last_became_active":"2024-03-24T01:32:44.751290+0000","last_became_peered":"2024-03-24T01:32:44.751290+0000","last_unstale":"2024-03-24T01:32:45.713955+0000","last_undegraded":"2024-03-24T01:32:45.713955+0000","last_fullsized":"2024-03-24T01:32:45.713955+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:14:56.287011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.713664+0000","last_change":"2024-03-24T01:32:44.751982+0000","last_active":"2024-03-24T01:32:45.713664+0000","last_peered":"2024-03-24T01:32:45.713664+0000","last_clean":"2024-03-24T01:32:45.713664+0000","last_became_active":"2024-03-24T01:32:44.751686+0000","last_became_peered":"2024-03-24T01:32:44.751686+0000","last_unstale":"2024-03-24T01:32:45.713664+0000","last_undegraded":"2024-03-24T01:32:45.713664+0000","last_fullsized":"2024-03-24T01:32:45.713664+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:03.806877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117942+0000","last_change":"2024-03-24T01:32:44.761410+0000","last_active":"2024-03-24T01:32:46.117942+0000","last_peered":"2024-03-24T01:32:46.117942+0000","last_clean":"2024-03-24T01:32:46.117942+0000","last_became_active":"2024-03-24T01:32:44.760599+0000","last_became_peered":"2024-03-24T01:32:44.760599+0000","last_unstale":"2024-03-24T01:32:46.117942+0000","last_undegraded":"2024-03-24T01:32:46.117942+0000","last_fullsized":"2024-03-24T01:32:46.117942+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:33:13.594934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.720760+0000","last_change":"2024-03-24T01:32:44.735449+0000","last_active":"2024-03-24T01:32:45.720760+0000","last_peered":"2024-03-24T01:32:45.720760+0000","last_clean":"2024-03-24T01:32:45.720760+0000","last_became_active":"2024-03-24T01:32:44.734765+0000","last_became_peered":"2024-03-24T01:32:44.734765+0000","last_unstale":"2024-03-24T01:32:45.720760+0000","last_undegraded":"2024-03-24T01:32:45.720760+0000","last_fullsized":"2024-03-24T01:32:45.720760+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:12.824577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.720672+0000","last_change":"2024-03-24T01:32:44.731161+0000","last_active":"2024-03-24T01:32:45.720672+0000","last_peered":"2024-03-24T01:32:45.720672+0000","last_clean":"2024-03-24T01:32:45.720672+0000","last_became_active":"2024-03-24T01:32:44.730863+0000","last_became_peered":"2024-03-24T01:32:44.730863+0000","last_unstale":"2024-03-24T01:32:45.720672+0000","last_undegraded":"2024-03-24T01:32:45.720672+0000","last_fullsized":"2024-03-24T01:32:45.720672+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:13:48.874647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117287+0000","last_change":"2024-03-24T01:32:44.741905+0000","last_active":"2024-03-24T01:32:46.117287+0000","last_peered":"2024-03-24T01:32:46.117287+0000","last_clean":"2024-03-24T01:32:46.117287+0000","last_became_active":"2024-03-24T01:32:44.741599+0000","last_became_peered":"2024-03-24T01:32:44.741599+0000","last_unstale":"2024-03-24T01:32:46.117287+0000","last_undegraded":"2024-03-24T01:32:46.117287+0000","last_fullsized":"2024-03-24T01:32:46.117287+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:19:45.922169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.773953+0000","last_change":"2024-03-24T01:32:44.755846+0000","last_active":"2024-03-24T01:32:45.773953+0000","last_peered":"2024-03-24T01:32:45.773953+0000","last_clean":"2024-03-24T01:32:45.773953+0000","last_became_active":"2024-03-24T01:32:44.755533+0000","last_became_peered":"2024-03-24T01:32:44.755533+0000","last_unstale":"2024-03-24T01:32:45.773953+0000","last_undegraded":"2024-03-24T01:32:45.773953+0000","last_fullsized":"2024-03-24T01:32:45.773953+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:00:32.057217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.767176+0000","last_change":"2024-03-24T01:32:44.754059+0000","last_active":"2024-03-24T01:32:45.767176+0000","last_peered":"2024-03-24T01:32:45.767176+0000","last_clean":"2024-03-24T01:32:45.767176+0000","last_became_active":"2024-03-24T01:32:44.753503+0000","last_became_peered":"2024-03-24T01:32:44.753503+0000","last_unstale":"2024-03-24T01:32:45.767176+0000","last_undegraded":"2024-03-24T01:32:45.767176+0000","last_fullsized":"2024-03-24T01:32:45.767176+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:39:23.561488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.118564+0000","last_change":"2024-03-24T01:32:44.761804+0000","last_active":"2024-03-24T01:32:46.118564+0000","last_peered":"2024-03-24T01:32:46.118564+0000","last_clean":"2024-03-24T01:32:46.118564+0000","last_became_active":"2024-03-24T01:32:44.761263+0000","last_became_peered":"2024-03-24T01:32:44.761263+0000","last_unstale":"2024-03-24T01:32:46.118564+0000","last_undegraded":"2024-03-24T01:32:46.118564+0000","last_fullsized":"2024-03-24T01:32:46.118564+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:31:50.925888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.118044+0000","last_change":"2024-03-24T01:32:44.755965+0000","last_active":"2024-03-24T01:32:46.118044+0000","last_peered":"2024-03-24T01:32:46.118044+0000","last_clean":"2024-03-24T01:32:46.118044+0000","last_became_active":"2024-03-24T01:32:44.755708+0000","last_became_peered":"2024-03-24T01:32:44.755708+0000","last_unstale":"2024-03-24T01:32:46.118044+0000","last_undegraded":"2024-03-24T01:32:46.118044+0000","last_fullsized":"2024-03-24T01:32:46.118044+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:33:21.633652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.719972+0000","last_change":"2024-03-24T01:32:44.750963+0000","last_active":"2024-03-24T01:32:45.719972+0000","last_peered":"2024-03-24T01:32:45.719972+0000","last_clean":"2024-03-24T01:32:45.719972+0000","last_became_active":"2024-03-24T01:32:44.750351+0000","last_became_peered":"2024-03-24T01:32:44.750351+0000","last_unstale":"2024-03-24T01:32:45.719972+0000","last_undegraded":"2024-03-24T01:32:45.719972+0000","last_fullsized":"2024-03-24T01:32:45.719972+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:13:48.874647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.767953+0000","last_change":"2024-03-24T01:32:44.752161+0000","last_active":"2024-03-24T01:32:45.767953+0000","last_peered":"2024-03-24T01:32:45.767953+0000","last_clean":"2024-03-24T01:32:45.767953+0000","last_became_active":"2024-03-24T01:32:44.751842+0000","last_became_peered":"2024-03-24T01:32:44.751842+0000","last_unstale":"2024-03-24T01:32:45.767953+0000","last_undegraded":"2024-03-24T01:32:45.767953+0000","last_fullsized":"2024-03-24T01:32:45.767953+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:48:18.811395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.769621+0000","last_change":"2024-03-24T01:32:44.746912+0000","last_active":"2024-03-24T01:32:45.769621+0000","last_peered":"2024-03-24T01:32:45.769621+0000","last_clean":"2024-03-24T01:32:45.769621+0000","last_became_active":"2024-03-24T01:32:44.746271+0000","last_became_peered":"2024-03-24T01:32:44.746271+0000","last_unstale":"2024-03-24T01:32:45.769621+0000","last_undegraded":"2024-03-24T01:32:45.769621+0000","last_fullsized":"2024-03-24T01:32:45.769621+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:33:21.633652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117510+0000","last_change":"2024-03-24T01:32:44.758588+0000","last_active":"2024-03-24T01:32:46.117510+0000","last_peered":"2024-03-24T01:32:46.117510+0000","last_clean":"2024-03-24T01:32:46.117510+0000","last_became_active":"2024-03-24T01:32:44.757289+0000","last_became_peered":"2024-03-24T01:32:44.757289+0000","last_unstale":"2024-03-24T01:32:46.117510+0000","last_undegraded":"2024-03-24T01:32:46.117510+0000","last_fullsized":"2024-03-24T01:32:46.117510+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:41:56.230713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117149+0000","last_change":"2024-03-24T01:32:44.753415+0000","last_active":"2024-03-24T01:32:46.117149+0000","last_peered":"2024-03-24T01:32:46.117149+0000","last_clean":"2024-03-24T01:32:46.117149+0000","last_became_active":"2024-03-24T01:32:44.752869+0000","last_became_peered":"2024-03-24T01:32:44.752869+0000","last_unstale":"2024-03-24T01:32:46.117149+0000","last_undegraded":"2024-03-24T01:32:46.117149+0000","last_fullsized":"2024-03-24T01:32:46.117149+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:48:18.811395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.773869+0000","last_change":"2024-03-24T01:32:44.742945+0000","last_active":"2024-03-24T01:32:45.773869+0000","last_peered":"2024-03-24T01:32:45.773869+0000","last_clean":"2024-03-24T01:32:45.773869+0000","last_became_active":"2024-03-24T01:32:44.742329+0000","last_became_peered":"2024-03-24T01:32:44.742329+0000","last_unstale":"2024-03-24T01:32:45.773869+0000","last_undegraded":"2024-03-24T01:32:45.773869+0000","last_fullsized":"2024-03-24T01:32:45.773869+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:40:03.643169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.714047+0000","last_change":"2024-03-24T01:32:44.745728+0000","last_active":"2024-03-24T01:32:45.714047+0000","last_peered":"2024-03-24T01:32:45.714047+0000","last_clean":"2024-03-24T01:32:45.714047+0000","last_became_active":"2024-03-24T01:32:44.745412+0000","last_became_peered":"2024-03-24T01:32:44.745412+0000","last_unstale":"2024-03-24T01:32:45.714047+0000","last_undegraded":"2024-03-24T01:32:45.714047+0000","last_fullsized":"2024-03-24T01:32:45.714047+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:36:57.078745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.720116+0000","last_change":"2024-03-24T01:32:44.735333+0000","last_active":"2024-03-24T01:32:45.720116+0000","last_peered":"2024-03-24T01:32:45.720116+0000","last_clean":"2024-03-24T01:32:45.720116+0000","last_became_active":"2024-03-24T01:32:44.734813+0000","last_became_peered":"2024-03-24T01:32:44.734813+0000","last_unstale":"2024-03-24T01:32:45.720116+0000","last_undegraded":"2024-03-24T01:32:45.720116+0000","last_fullsized":"2024-03-24T01:32:45.720116+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:15:20.815746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.721498+0000","last_change":"2024-03-24T01:32:44.750885+0000","last_active":"2024-03-24T01:32:45.721498+0000","last_peered":"2024-03-24T01:32:45.721498+0000","last_clean":"2024-03-24T01:32:45.721498+0000","last_became_active":"2024-03-24T01:32:44.750355+0000","last_became_peered":"2024-03-24T01:32:44.750355+0000","last_unstale":"2024-03-24T01:32:45.721498+0000","last_undegraded":"2024-03-24T01:32:45.721498+0000","last_fullsized":"2024-03-24T01:32:45.721498+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:35.662274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.118346+0000","last_change":"2024-03-24T01:32:44.734704+0000","last_active":"2024-03-24T01:32:46.118346+0000","last_peered":"2024-03-24T01:32:46.118346+0000","last_clean":"2024-03-24T01:32:46.118346+0000","last_became_active":"2024-03-24T01:32:44.733687+0000","last_became_peered":"2024-03-24T01:32:44.733687+0000","last_unstale":"2024-03-24T01:32:46.118346+0000","last_undegraded":"2024-03-24T01:32:46.118346+0000","last_fullsized":"2024-03-24T01:32:46.118346+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:50:50.757791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.768693+0000","last_change":"2024-03-24T01:32:44.755202+0000","last_active":"2024-03-24T01:32:45.768693+0000","last_peered":"2024-03-24T01:32:45.768693+0000","last_clean":"2024-03-24T01:32:45.768693+0000","last_became_active":"2024-03-24T01:32:44.754775+0000","last_became_peered":"2024-03-24T01:32:44.754775+0000","last_unstale":"2024-03-24T01:32:45.768693+0000","last_undegraded":"2024-03-24T01:32:45.768693+0000","last_fullsized":"2024-03-24T01:32:45.768693+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:33:13.594934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.118707+0000","last_change":"2024-03-24T01:32:44.741995+0000","last_active":"2024-03-24T01:32:46.118707+0000","last_peered":"2024-03-24T01:32:46.118707+0000","last_clean":"2024-03-24T01:32:46.118707+0000","last_became_active":"2024-03-24T01:32:44.741671+0000","last_became_peered":"2024-03-24T01:32:44.741671+0000","last_unstale":"2024-03-24T01:32:46.118707+0000","last_undegraded":"2024-03-24T01:32:46.118707+0000","last_fullsized":"2024-03-24T01:32:46.118707+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:52:15.663148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117020+0000","last_change":"2024-03-24T01:32:44.740789+0000","last_active":"2024-03-24T01:32:46.117020+0000","last_peered":"2024-03-24T01:32:46.117020+0000","last_clean":"2024-03-24T01:32:46.117020+0000","last_became_active":"2024-03-24T01:32:44.740465+0000","last_became_peered":"2024-03-24T01:32:44.740465+0000","last_unstale":"2024-03-24T01:32:46.117020+0000","last_undegraded":"2024-03-24T01:32:46.117020+0000","last_fullsized":"2024-03-24T01:32:46.117020+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:02:39.321334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.116993+0000","last_change":"2024-03-24T01:32:44.758180+0000","last_active":"2024-03-24T01:32:46.116993+0000","last_peered":"2024-03-24T01:32:46.116993+0000","last_clean":"2024-03-24T01:32:46.116993+0000","last_became_active":"2024-03-24T01:32:44.756763+0000","last_became_peered":"2024-03-24T01:32:44.756763+0000","last_unstale":"2024-03-24T01:32:46.116993+0000","last_undegraded":"2024-03-24T01:32:46.116993+0000","last_fullsized":"2024-03-24T01:32:46.116993+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:59:12.918357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.768651+0000","last_change":"2024-03-24T01:32:44.753078+0000","last_active":"2024-03-24T01:32:45.768651+0000","last_peered":"2024-03-24T01:32:45.768651+0000","last_clean":"2024-03-24T01:32:45.768651+0000","last_became_active":"2024-03-24T01:32:44.750365+0000","last_became_peered":"2024-03-24T01:32:44.750365+0000","last_unstale":"2024-03-24T01:32:45.768651+0000","last_undegraded":"2024-03-24T01:32:45.768651+0000","last_fullsized":"2024-03-24T01:32:45.768651+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:56:30.509999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.767804+0000","last_change":"2024-03-24T01:32:44.747414+0000","last_active":"2024-03-24T01:32:45.767804+0000","last_peered":"2024-03-24T01:32:45.767804+0000","last_clean":"2024-03-24T01:32:45.767804+0000","last_became_active":"2024-03-24T01:32:44.747138+0000","last_became_peered":"2024-03-24T01:32:44.747138+0000","last_unstale":"2024-03-24T01:32:45.767804+0000","last_undegraded":"2024-03-24T01:32:45.767804+0000","last_fullsized":"2024-03-24T01:32:45.767804+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:59:12.918357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.719958+0000","last_change":"2024-03-24T01:32:44.746689+0000","last_active":"2024-03-24T01:32:45.719958+0000","last_peered":"2024-03-24T01:32:45.719958+0000","last_clean":"2024-03-24T01:32:45.719958+0000","last_became_active":"2024-03-24T01:32:44.746359+0000","last_became_peered":"2024-03-24T01:32:44.746359+0000","last_unstale":"2024-03-24T01:32:45.719958+0000","last_undegraded":"2024-03-24T01:32:45.719958+0000","last_fullsized":"2024-03-24T01:32:45.719958+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:36.843294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.713789+0000","last_change":"2024-03-24T01:32:44.749783+0000","last_active":"2024-03-24T01:32:45.713789+0000","last_peered":"2024-03-24T01:32:45.713789+0000","last_clean":"2024-03-24T01:32:45.713789+0000","last_became_active":"2024-03-24T01:32:44.749266+0000","last_became_peered":"2024-03-24T01:32:44.749266+0000","last_unstale":"2024-03-24T01:32:45.713789+0000","last_undegraded":"2024-03-24T01:32:45.713789+0000","last_fullsized":"2024-03-24T01:32:45.713789+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:39:10.149776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117669+0000","last_change":"2024-03-24T01:32:44.751695+0000","last_active":"2024-03-24T01:32:46.117669+0000","last_peered":"2024-03-24T01:32:46.117669+0000","last_clean":"2024-03-24T01:32:46.117669+0000","last_became_active":"2024-03-24T01:32:44.751409+0000","last_became_peered":"2024-03-24T01:32:44.751409+0000","last_unstale":"2024-03-24T01:32:46.117669+0000","last_undegraded":"2024-03-24T01:32:46.117669+0000","last_fullsized":"2024-03-24T01:32:46.117669+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:14:45.725936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.767559+0000","last_change":"2024-03-24T01:32:44.752696+0000","last_active":"2024-03-24T01:32:45.767559+0000","last_peered":"2024-03-24T01:32:45.767559+0000","last_clean":"2024-03-24T01:32:45.767559+0000","last_became_active":"2024-03-24T01:32:44.752378+0000","last_became_peered":"2024-03-24T01:32:44.752378+0000","last_unstale":"2024-03-24T01:32:45.767559+0000","last_undegraded":"2024-03-24T01:32:45.767559+0000","last_fullsized":"2024-03-24T01:32:45.767559+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:36:57.078745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.721483+0000","last_change":"2024-03-24T01:32:44.742966+0000","last_active":"2024-03-24T01:32:45.721483+0000","last_peered":"2024-03-24T01:32:45.721483+0000","last_clean":"2024-03-24T01:32:45.721483+0000","last_became_active":"2024-03-24T01:32:44.742681+0000","last_became_peered":"2024-03-24T01:32:44.742681+0000","last_unstale":"2024-03-24T01:32:45.721483+0000","last_undegraded":"2024-03-24T01:32:45.721483+0000","last_fullsized":"2024-03-24T01:32:45.721483+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:39:10.149776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.116803+0000","last_change":"2024-03-24T01:32:44.744799+0000","last_active":"2024-03-24T01:32:46.116803+0000","last_peered":"2024-03-24T01:32:46.116803+0000","last_clean":"2024-03-24T01:32:46.116803+0000","last_became_active":"2024-03-24T01:32:44.743828+0000","last_became_peered":"2024-03-24T01:32:44.743828+0000","last_unstale":"2024-03-24T01:32:46.116803+0000","last_undegraded":"2024-03-24T01:32:46.116803+0000","last_fullsized":"2024-03-24T01:32:46.116803+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:31:50.925888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.721397+0000","last_change":"2024-03-24T01:32:44.748610+0000","last_active":"2024-03-24T01:32:45.721397+0000","last_peered":"2024-03-24T01:32:45.721397+0000","last_clean":"2024-03-24T01:32:45.721397+0000","last_became_active":"2024-03-24T01:32:44.748297+0000","last_became_peered":"2024-03-24T01:32:44.748297+0000","last_unstale":"2024-03-24T01:32:45.721397+0000","last_undegraded":"2024-03-24T01:32:45.721397+0000","last_fullsized":"2024-03-24T01:32:45.721397+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:19.032155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"66'4","reported_seq":18,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.714111+0000","last_change":"2024-03-24T01:32:43.729112+0000","last_active":"2024-03-24T01:32:45.714111+0000","last_peered":"2024-03-24T01:32:45.714111+0000","last_clean":"2024-03-24T01:32:45.714111+0000","last_became_active":"2024-03-24T01:32:43.728882+0000","last_became_peered":"2024-03-24T01:32:43.728882+0000","last_unstale":"2024-03-24T01:32:45.714111+0000","last_undegraded":"2024-03-24T01:32:45.714111+0000","last_fullsized":"2024-03-24T01:32:45.714111+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:29.157467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.720464+0000","last_change":"2024-03-24T01:32:44.733668+0000","last_active":"2024-03-24T01:32:45.720464+0000","last_peered":"2024-03-24T01:32:45.720464+0000","last_clean":"2024-03-24T01:32:45.720464+0000","last_became_active":"2024-03-24T01:32:44.733182+0000","last_became_peered":"2024-03-24T01:32:44.733182+0000","last_unstale":"2024-03-24T01:32:45.720464+0000","last_undegraded":"2024-03-24T01:32:45.720464+0000","last_fullsized":"2024-03-24T01:32:45.720464+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:19.032155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"66'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.118869+0000","last_change":"2024-03-24T01:32:43.723642+0000","last_active":"2024-03-24T01:32:46.118869+0000","last_peered":"2024-03-24T01:32:46.118869+0000","last_clean":"2024-03-24T01:32:46.118869+0000","last_became_active":"2024-03-24T01:32:43.723374+0000","last_became_peered":"2024-03-24T01:32:43.723374+0000","last_unstale":"2024-03-24T01:32:46.118869+0000","last_undegraded":"2024-03-24T01:32:46.118869+0000","last_fullsized":"2024-03-24T01:32:46.118869+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:16:58.498006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.118687+0000","last_change":"2024-03-24T01:32:44.743311+0000","last_active":"2024-03-24T01:32:46.118687+0000","last_peered":"2024-03-24T01:32:46.118687+0000","last_clean":"2024-03-24T01:32:46.118687+0000","last_became_active":"2024-03-24T01:32:44.743027+0000","last_became_peered":"2024-03-24T01:32:44.743027+0000","last_unstale":"2024-03-24T01:32:46.118687+0000","last_undegraded":"2024-03-24T01:32:46.118687+0000","last_fullsized":"2024-03-24T01:32:46.118687+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:03:37.421881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117972+0000","last_change":"2024-03-24T01:32:43.722317+0000","last_active":"2024-03-24T01:32:46.117972+0000","last_peered":"2024-03-24T01:32:46.117972+0000","last_clean":"2024-03-24T01:32:46.117972+0000","last_became_active":"2024-03-24T01:32:43.721576+0000","last_became_peered":"2024-03-24T01:32:43.721576+0000","last_unstale":"2024-03-24T01:32:46.117972+0000","last_undegraded":"2024-03-24T01:32:46.117972+0000","last_fullsized":"2024-03-24T01:32:46.117972+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:36:56.063762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.118276+0000","last_change":"2024-03-24T01:32:44.755395+0000","last_active":"2024-03-24T01:32:46.118276+0000","last_peered":"2024-03-24T01:32:46.118276+0000","last_clean":"2024-03-24T01:32:46.118276+0000","last_became_active":"2024-03-24T01:32:44.755127+0000","last_became_peered":"2024-03-24T01:32:44.755127+0000","last_unstale":"2024-03-24T01:32:46.118276+0000","last_undegraded":"2024-03-24T01:32:46.118276+0000","last_fullsized":"2024-03-24T01:32:46.118276+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:03:13.359822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117409+0000","last_change":"2024-03-24T01:32:43.713108+0000","last_active":"2024-03-24T01:32:46.117409+0000","last_peered":"2024-03-24T01:32:46.117409+0000","last_clean":"2024-03-24T01:32:46.117409+0000","last_became_active":"2024-03-24T01:32:43.712800+0000","last_became_peered":"2024-03-24T01:32:43.712800+0000","last_unstale":"2024-03-24T01:32:46.117409+0000","last_undegraded":"2024-03-24T01:32:46.117409+0000","last_fullsized":"2024-03-24T01:32:46.117409+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:59:11.903374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.774108+0000","last_change":"2024-03-24T01:32:44.747444+0000","last_active":"2024-03-24T01:32:45.774108+0000","last_peered":"2024-03-24T01:32:45.774108+0000","last_clean":"2024-03-24T01:32:45.774108+0000","last_became_active":"2024-03-24T01:32:44.747117+0000","last_became_peered":"2024-03-24T01:32:44.747117+0000","last_unstale":"2024-03-24T01:32:45.774108+0000","last_undegraded":"2024-03-24T01:32:45.774108+0000","last_fullsized":"2024-03-24T01:32:45.774108+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:24:34.023784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"66'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117538+0000","last_change":"2024-03-24T01:32:43.718182+0000","last_active":"2024-03-24T01:32:46.117538+0000","last_peered":"2024-03-24T01:32:46.117538+0000","last_clean":"2024-03-24T01:32:46.117538+0000","last_became_active":"2024-03-24T01:32:43.717674+0000","last_became_peered":"2024-03-24T01:32:43.717674+0000","last_unstale":"2024-03-24T01:32:46.117538+0000","last_undegraded":"2024-03-24T01:32:46.117538+0000","last_fullsized":"2024-03-24T01:32:46.117538+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:31:49.910904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.713179+0000","last_change":"2024-03-24T01:32:44.746900+0000","last_active":"2024-03-24T01:32:45.713179+0000","last_peered":"2024-03-24T01:32:45.713179+0000","last_clean":"2024-03-24T01:32:45.713179+0000","last_became_active":"2024-03-24T01:32:44.746619+0000","last_became_peered":"2024-03-24T01:32:44.746619+0000","last_unstale":"2024-03-24T01:32:45.713179+0000","last_undegraded":"2024-03-24T01:32:45.713179+0000","last_fullsized":"2024-03-24T01:32:45.713179+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:12.824577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"66'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117671+0000","last_change":"2024-03-24T01:32:43.717225+0000","last_active":"2024-03-24T01:32:46.117671+0000","last_peered":"2024-03-24T01:32:46.117671+0000","last_clean":"2024-03-24T01:32:46.117671+0000","last_became_active":"2024-03-24T01:32:43.716994+0000","last_became_peered":"2024-03-24T01:32:43.716994+0000","last_unstale":"2024-03-24T01:32:46.117671+0000","last_undegraded":"2024-03-24T01:32:46.117671+0000","last_fullsized":"2024-03-24T01:32:46.117671+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:14:44.710953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.721632+0000","last_change":"2024-03-24T01:32:44.733286+0000","last_active":"2024-03-24T01:32:45.721632+0000","last_peered":"2024-03-24T01:32:45.721632+0000","last_clean":"2024-03-24T01:32:45.721632+0000","last_became_active":"2024-03-24T01:32:44.732956+0000","last_became_peered":"2024-03-24T01:32:44.732956+0000","last_unstale":"2024-03-24T01:32:45.721632+0000","last_undegraded":"2024-03-24T01:32:45.721632+0000","last_fullsized":"2024-03-24T01:32:45.721632+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:12.824577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.116267+0000","last_change":"2024-03-24T01:32:43.719216+0000","last_active":"2024-03-24T01:32:46.116267+0000","last_peered":"2024-03-24T01:32:46.116267+0000","last_clean":"2024-03-24T01:32:46.116267+0000","last_became_active":"2024-03-24T01:32:43.718879+0000","last_became_peered":"2024-03-24T01:32:43.718879+0000","last_unstale":"2024-03-24T01:32:46.116267+0000","last_undegraded":"2024-03-24T01:32:46.116267+0000","last_fullsized":"2024-03-24T01:32:46.116267+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:03:12.344839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117429+0000","last_change":"2024-03-24T01:32:44.740874+0000","last_active":"2024-03-24T01:32:46.117429+0000","last_peered":"2024-03-24T01:32:46.117429+0000","last_clean":"2024-03-24T01:32:46.117429+0000","last_became_active":"2024-03-24T01:32:44.740563+0000","last_became_peered":"2024-03-24T01:32:44.740563+0000","last_unstale":"2024-03-24T01:32:46.117429+0000","last_undegraded":"2024-03-24T01:32:46.117429+0000","last_fullsized":"2024-03-24T01:32:46.117429+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:11:10.464938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.719133+0000","last_change":"2024-03-24T01:32:43.720754+0000","last_active":"2024-03-24T01:32:45.719133+0000","last_peered":"2024-03-24T01:32:45.719133+0000","last_clean":"2024-03-24T01:32:45.719133+0000","last_became_active":"2024-03-24T01:32:43.720465+0000","last_became_peered":"2024-03-24T01:32:43.720465+0000","last_unstale":"2024-03-24T01:32:45.719133+0000","last_undegraded":"2024-03-24T01:32:45.719133+0000","last_fullsized":"2024-03-24T01:32:45.719133+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:40:54.360077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.713983+0000","last_change":"2024-03-24T01:32:44.750883+0000","last_active":"2024-03-24T01:32:45.713983+0000","last_peered":"2024-03-24T01:32:45.713983+0000","last_clean":"2024-03-24T01:32:45.713983+0000","last_became_active":"2024-03-24T01:32:44.750572+0000","last_became_peered":"2024-03-24T01:32:44.750572+0000","last_unstale":"2024-03-24T01:32:45.713983+0000","last_undegraded":"2024-03-24T01:32:45.713983+0000","last_fullsized":"2024-03-24T01:32:45.713983+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:39:23.561488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.118621+0000","last_change":"2024-03-24T01:32:43.729179+0000","last_active":"2024-03-24T01:32:46.118621+0000","last_peered":"2024-03-24T01:32:46.118621+0000","last_clean":"2024-03-24T01:32:46.118621+0000","last_became_active":"2024-03-24T01:32:43.728892+0000","last_became_peered":"2024-03-24T01:32:43.728892+0000","last_unstale":"2024-03-24T01:32:46.118621+0000","last_undegraded":"2024-03-24T01:32:46.118621+0000","last_fullsized":"2024-03-24T01:32:46.118621+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:18.017172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117966+0000","last_change":"2024-03-24T01:32:44.747545+0000","last_active":"2024-03-24T01:32:46.117966+0000","last_peered":"2024-03-24T01:32:46.117966+0000","last_clean":"2024-03-24T01:32:46.117966+0000","last_became_active":"2024-03-24T01:32:44.747227+0000","last_became_peered":"2024-03-24T01:32:44.747227+0000","last_unstale":"2024-03-24T01:32:46.117966+0000","last_undegraded":"2024-03-24T01:32:46.117966+0000","last_fullsized":"2024-03-24T01:32:46.117966+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:55.474861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.721185+0000","last_change":"2024-03-24T01:32:43.715851+0000","last_active":"2024-03-24T01:32:45.721185+0000","last_peered":"2024-03-24T01:32:45.721185+0000","last_clean":"2024-03-24T01:32:45.721185+0000","last_became_active":"2024-03-24T01:32:43.715474+0000","last_became_peered":"2024-03-24T01:32:43.715474+0000","last_unstale":"2024-03-24T01:32:45.721185+0000","last_undegraded":"2024-03-24T01:32:45.721185+0000","last_fullsized":"2024-03-24T01:32:45.721185+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:24:00.025674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.721208+0000","last_change":"2024-03-24T01:32:44.748134+0000","last_active":"2024-03-24T01:32:45.721208+0000","last_peered":"2024-03-24T01:32:45.721208+0000","last_clean":"2024-03-24T01:32:45.721208+0000","last_became_active":"2024-03-24T01:32:44.747771+0000","last_became_peered":"2024-03-24T01:32:44.747771+0000","last_unstale":"2024-03-24T01:32:45.721208+0000","last_undegraded":"2024-03-24T01:32:45.721208+0000","last_fullsized":"2024-03-24T01:32:45.721208+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:36.843294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"66'3","reported_seq":17,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.118252+0000","last_change":"2024-03-24T01:32:43.725497+0000","last_active":"2024-03-24T01:32:46.118252+0000","last_peered":"2024-03-24T01:32:46.118252+0000","last_clean":"2024-03-24T01:32:46.118252+0000","last_became_active":"2024-03-24T01:32:43.724751+0000","last_became_peered":"2024-03-24T01:32:43.724751+0000","last_unstale":"2024-03-24T01:32:46.118252+0000","last_undegraded":"2024-03-24T01:32:46.118252+0000","last_fullsized":"2024-03-24T01:32:46.118252+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:35.828311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.720698+0000","last_change":"2024-03-24T01:32:44.737244+0000","last_active":"2024-03-24T01:32:45.720698+0000","last_peered":"2024-03-24T01:32:45.720698+0000","last_clean":"2024-03-24T01:32:45.720698+0000","last_became_active":"2024-03-24T01:32:44.735810+0000","last_became_peered":"2024-03-24T01:32:44.735810+0000","last_unstale":"2024-03-24T01:32:45.720698+0000","last_undegraded":"2024-03-24T01:32:45.720698+0000","last_fullsized":"2024-03-24T01:32:45.720698+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:35.662274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"66'6","reported_seq":20,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.714077+0000","last_change":"2024-03-24T01:32:43.716683+0000","last_active":"2024-03-24T01:32:45.714077+0000","last_peered":"2024-03-24T01:32:45.714077+0000","last_clean":"2024-03-24T01:32:45.714077+0000","last_became_active":"2024-03-24T01:32:43.716407+0000","last_became_peered":"2024-03-24T01:32:43.716407+0000","last_unstale":"2024-03-24T01:32:45.714077+0000","last_undegraded":"2024-03-24T01:32:45.714077+0000","last_fullsized":"2024-03-24T01:32:45.714077+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:43:54.681008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.718926+0000","last_change":"2024-03-24T01:32:44.741777+0000","last_active":"2024-03-24T01:32:45.718926+0000","last_peered":"2024-03-24T01:32:45.718926+0000","last_clean":"2024-03-24T01:32:45.718926+0000","last_became_active":"2024-03-24T01:32:44.741472+0000","last_became_peered":"2024-03-24T01:32:44.741472+0000","last_unstale":"2024-03-24T01:32:45.718926+0000","last_undegraded":"2024-03-24T01:32:45.718926+0000","last_fullsized":"2024-03-24T01:32:45.718926+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:20:55.112893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117614+0000","last_change":"2024-03-24T01:32:43.725509+0000","last_active":"2024-03-24T01:32:46.117614+0000","last_peered":"2024-03-24T01:32:46.117614+0000","last_clean":"2024-03-24T01:32:46.117614+0000","last_became_active":"2024-03-24T01:32:43.725142+0000","last_became_peered":"2024-03-24T01:32:43.725142+0000","last_unstale":"2024-03-24T01:32:46.117614+0000","last_undegraded":"2024-03-24T01:32:46.117614+0000","last_fullsized":"2024-03-24T01:32:46.117614+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:20:54.257187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.767647+0000","last_change":"2024-03-24T01:32:44.750397+0000","last_active":"2024-03-24T01:32:45.767647+0000","last_peered":"2024-03-24T01:32:45.767647+0000","last_clean":"2024-03-24T01:32:45.767647+0000","last_became_active":"2024-03-24T01:32:44.750002+0000","last_became_peered":"2024-03-24T01:32:44.750002+0000","last_unstale":"2024-03-24T01:32:45.767647+0000","last_undegraded":"2024-03-24T01:32:45.767647+0000","last_fullsized":"2024-03-24T01:32:45.767647+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:16:59.512989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.767678+0000","last_change":"2024-03-24T01:32:43.720435+0000","last_active":"2024-03-24T01:32:45.767678+0000","last_peered":"2024-03-24T01:32:45.767678+0000","last_clean":"2024-03-24T01:32:45.767678+0000","last_became_active":"2024-03-24T01:32:43.720135+0000","last_became_peered":"2024-03-24T01:32:43.720135+0000","last_unstale":"2024-03-24T01:32:45.767678+0000","last_undegraded":"2024-03-24T01:32:45.767678+0000","last_fullsized":"2024-03-24T01:32:45.767678+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:11:39.983567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.722236+0000","last_change":"2024-03-24T01:32:44.751131+0000","last_active":"2024-03-24T01:32:45.722236+0000","last_peered":"2024-03-24T01:32:45.722236+0000","last_clean":"2024-03-24T01:32:45.722236+0000","last_became_active":"2024-03-24T01:32:44.750794+0000","last_became_peered":"2024-03-24T01:32:44.750794+0000","last_unstale":"2024-03-24T01:32:45.722236+0000","last_undegraded":"2024-03-24T01:32:45.722236+0000","last_fullsized":"2024-03-24T01:32:45.722236+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:14:56.287011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117416+0000","last_change":"2024-03-24T01:32:43.725306+0000","last_active":"2024-03-24T01:32:46.117416+0000","last_peered":"2024-03-24T01:32:46.117416+0000","last_clean":"2024-03-24T01:32:46.117416+0000","last_became_active":"2024-03-24T01:32:43.724101+0000","last_became_peered":"2024-03-24T01:32:43.724101+0000","last_unstale":"2024-03-24T01:32:46.117416+0000","last_undegraded":"2024-03-24T01:32:46.117416+0000","last_fullsized":"2024-03-24T01:32:46.117416+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:12:55.184668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.713378+0000","last_change":"2024-03-24T01:32:44.750404+0000","last_active":"2024-03-24T01:32:45.713378+0000","last_peered":"2024-03-24T01:32:45.713378+0000","last_clean":"2024-03-24T01:32:45.713378+0000","last_became_active":"2024-03-24T01:32:44.750099+0000","last_became_peered":"2024-03-24T01:32:44.750099+0000","last_unstale":"2024-03-24T01:32:45.713378+0000","last_undegraded":"2024-03-24T01:32:45.713378+0000","last_fullsized":"2024-03-24T01:32:45.713378+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:10:36.135152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"66'8","reported_seq":22,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.719601+0000","last_change":"2024-03-24T01:32:43.720664+0000","last_active":"2024-03-24T01:32:45.719601+0000","last_peered":"2024-03-24T01:32:45.719601+0000","last_clean":"2024-03-24T01:32:45.719601+0000","last_became_active":"2024-03-24T01:32:43.720338+0000","last_became_peered":"2024-03-24T01:32:43.720338+0000","last_unstale":"2024-03-24T01:32:45.719601+0000","last_undegraded":"2024-03-24T01:32:45.719601+0000","last_fullsized":"2024-03-24T01:32:45.719601+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:40:54.360077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117882+0000","last_change":"2024-03-24T01:32:44.762349+0000","last_active":"2024-03-24T01:32:46.117882+0000","last_peered":"2024-03-24T01:32:46.117882+0000","last_clean":"2024-03-24T01:32:46.117882+0000","last_became_active":"2024-03-24T01:32:44.761475+0000","last_became_peered":"2024-03-24T01:32:44.761475+0000","last_unstale":"2024-03-24T01:32:46.117882+0000","last_undegraded":"2024-03-24T01:32:46.117882+0000","last_fullsized":"2024-03-24T01:32:46.117882+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:56:30.509999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"66'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.768808+0000","last_change":"2024-03-24T01:32:43.722152+0000","last_active":"2024-03-24T01:32:45.768808+0000","last_peered":"2024-03-24T01:32:45.768808+0000","last_clean":"2024-03-24T01:32:45.768808+0000","last_became_active":"2024-03-24T01:32:43.721855+0000","last_became_peered":"2024-03-24T01:32:43.721855+0000","last_unstale":"2024-03-24T01:32:45.768808+0000","last_undegraded":"2024-03-24T01:32:45.768808+0000","last_fullsized":"2024-03-24T01:32:45.768808+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:20:54.257187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.721804+0000","last_change":"2024-03-24T01:32:44.728581+0000","last_active":"2024-03-24T01:32:45.721804+0000","last_peered":"2024-03-24T01:32:45.721804+0000","last_clean":"2024-03-24T01:32:45.721804+0000","last_became_active":"2024-03-24T01:32:44.728154+0000","last_became_peered":"2024-03-24T01:32:44.728154+0000","last_unstale":"2024-03-24T01:32:45.721804+0000","last_undegraded":"2024-03-24T01:32:45.721804+0000","last_fullsized":"2024-03-24T01:32:45.721804+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:03.806877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.118235+0000","last_change":"2024-03-24T01:32:43.726876+0000","last_active":"2024-03-24T01:32:46.118235+0000","last_peered":"2024-03-24T01:32:46.118235+0000","last_clean":"2024-03-24T01:32:46.118235+0000","last_became_active":"2024-03-24T01:32:43.726628+0000","last_became_peered":"2024-03-24T01:32:43.726628+0000","last_unstale":"2024-03-24T01:32:46.118235+0000","last_undegraded":"2024-03-24T01:32:46.118235+0000","last_fullsized":"2024-03-24T01:32:46.118235+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:34.647291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.720919+0000","last_change":"2024-03-24T01:32:44.733825+0000","last_active":"2024-03-24T01:32:45.720919+0000","last_peered":"2024-03-24T01:32:45.720919+0000","last_clean":"2024-03-24T01:32:45.720919+0000","last_became_active":"2024-03-24T01:32:44.733404+0000","last_became_peered":"2024-03-24T01:32:44.733404+0000","last_unstale":"2024-03-24T01:32:45.720919+0000","last_undegraded":"2024-03-24T01:32:45.720919+0000","last_fullsized":"2024-03-24T01:32:45.720919+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:10:36.135152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"66'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117934+0000","last_change":"2024-03-24T01:32:43.725389+0000","last_active":"2024-03-24T01:32:46.117934+0000","last_peered":"2024-03-24T01:32:46.117934+0000","last_clean":"2024-03-24T01:32:46.117934+0000","last_became_active":"2024-03-24T01:32:43.724956+0000","last_became_peered":"2024-03-24T01:32:43.724956+0000","last_unstale":"2024-03-24T01:32:46.117934+0000","last_undegraded":"2024-03-24T01:32:46.117934+0000","last_fullsized":"2024-03-24T01:32:46.117934+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:13:47.859663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.721685+0000","last_change":"2024-03-24T01:32:44.743865+0000","last_active":"2024-03-24T01:32:45.721685+0000","last_peered":"2024-03-24T01:32:45.721685+0000","last_clean":"2024-03-24T01:32:45.721685+0000","last_became_active":"2024-03-24T01:32:44.741745+0000","last_became_peered":"2024-03-24T01:32:44.741745+0000","last_unstale":"2024-03-24T01:32:45.721685+0000","last_undegraded":"2024-03-24T01:32:45.721685+0000","last_fullsized":"2024-03-24T01:32:45.721685+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:10:36.135152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117487+0000","last_change":"2024-03-24T01:32:43.721896+0000","last_active":"2024-03-24T01:32:46.117487+0000","last_peered":"2024-03-24T01:32:46.117487+0000","last_clean":"2024-03-24T01:32:46.117487+0000","last_became_active":"2024-03-24T01:32:43.720277+0000","last_became_peered":"2024-03-24T01:32:43.720277+0000","last_unstale":"2024-03-24T01:32:46.117487+0000","last_undegraded":"2024-03-24T01:32:46.117487+0000","last_fullsized":"2024-03-24T01:32:46.117487+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:33:12.579951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.116685+0000","last_change":"2024-03-24T01:32:44.743387+0000","last_active":"2024-03-24T01:32:46.116685+0000","last_peered":"2024-03-24T01:32:46.116685+0000","last_clean":"2024-03-24T01:32:46.116685+0000","last_became_active":"2024-03-24T01:32:44.743079+0000","last_became_peered":"2024-03-24T01:32:44.743079+0000","last_unstale":"2024-03-24T01:32:46.116685+0000","last_undegraded":"2024-03-24T01:32:46.116685+0000","last_fullsized":"2024-03-24T01:32:46.116685+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:03:13.359822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117048+0000","last_change":"2024-03-24T01:32:43.707322+0000","last_active":"2024-03-24T01:32:46.117048+0000","last_peered":"2024-03-24T01:32:46.117048+0000","last_clean":"2024-03-24T01:32:46.117048+0000","last_became_active":"2024-03-24T01:32:43.706928+0000","last_became_peered":"2024-03-24T01:32:43.706928+0000","last_unstale":"2024-03-24T01:32:46.117048+0000","last_undegraded":"2024-03-24T01:32:46.117048+0000","last_fullsized":"2024-03-24T01:32:46.117048+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:39:22.546505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.768153+0000","last_change":"2024-03-24T01:32:44.744400+0000","last_active":"2024-03-24T01:32:45.768153+0000","last_peered":"2024-03-24T01:32:45.768153+0000","last_clean":"2024-03-24T01:32:45.768153+0000","last_became_active":"2024-03-24T01:32:44.742926+0000","last_became_peered":"2024-03-24T01:32:44.742926+0000","last_unstale":"2024-03-24T01:32:45.768153+0000","last_undegraded":"2024-03-24T01:32:45.768153+0000","last_fullsized":"2024-03-24T01:32:45.768153+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:14:45.725936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.767725+0000","last_change":"2024-03-24T01:32:43.724088+0000","last_active":"2024-03-24T01:32:45.767725+0000","last_peered":"2024-03-24T01:32:45.767725+0000","last_clean":"2024-03-24T01:32:45.767725+0000","last_became_active":"2024-03-24T01:32:43.722732+0000","last_became_peered":"2024-03-24T01:32:43.722732+0000","last_unstale":"2024-03-24T01:32:45.767725+0000","last_undegraded":"2024-03-24T01:32:45.767725+0000","last_fullsized":"2024-03-24T01:32:45.767725+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:12:55.184668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.722030+0000","last_change":"2024-03-24T01:32:44.736167+0000","last_active":"2024-03-24T01:32:45.722030+0000","last_peered":"2024-03-24T01:32:45.722030+0000","last_clean":"2024-03-24T01:32:45.722030+0000","last_became_active":"2024-03-24T01:32:44.733302+0000","last_became_peered":"2024-03-24T01:32:44.733302+0000","last_unstale":"2024-03-24T01:32:45.722030+0000","last_undegraded":"2024-03-24T01:32:45.722030+0000","last_fullsized":"2024-03-24T01:32:45.722030+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:39:10.149776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.116796+0000","last_change":"2024-03-24T01:32:43.720278+0000","last_active":"2024-03-24T01:32:46.116796+0000","last_peered":"2024-03-24T01:32:46.116796+0000","last_clean":"2024-03-24T01:32:46.116796+0000","last_became_active":"2024-03-24T01:32:43.719995+0000","last_became_peered":"2024-03-24T01:32:43.719995+0000","last_unstale":"2024-03-24T01:32:46.116796+0000","last_undegraded":"2024-03-24T01:32:46.116796+0000","last_fullsized":"2024-03-24T01:32:46.116796+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:14:55.272028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117369+0000","last_change":"2024-03-24T01:32:44.731492+0000","last_active":"2024-03-24T01:32:46.117369+0000","last_peered":"2024-03-24T01:32:46.117369+0000","last_clean":"2024-03-24T01:32:46.117369+0000","last_became_active":"2024-03-24T01:32:44.730776+0000","last_became_peered":"2024-03-24T01:32:44.730776+0000","last_unstale":"2024-03-24T01:32:46.117369+0000","last_undegraded":"2024-03-24T01:32:46.117369+0000","last_fullsized":"2024-03-24T01:32:46.117369+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:01:46.557014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.769770+0000","last_change":"2024-03-24T01:32:43.729534+0000","last_active":"2024-03-24T01:32:45.769770+0000","last_peered":"2024-03-24T01:32:45.769770+0000","last_clean":"2024-03-24T01:32:45.769770+0000","last_became_active":"2024-03-24T01:32:43.729214+0000","last_became_peered":"2024-03-24T01:32:43.729214+0000","last_unstale":"2024-03-24T01:32:45.769770+0000","last_undegraded":"2024-03-24T01:32:45.769770+0000","last_fullsized":"2024-03-24T01:32:45.769770+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:20:54.097910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.774617+0000","last_change":"2024-03-24T01:32:44.744158+0000","last_active":"2024-03-24T01:32:45.774617+0000","last_peered":"2024-03-24T01:32:45.774617+0000","last_clean":"2024-03-24T01:32:45.774617+0000","last_became_active":"2024-03-24T01:32:44.743592+0000","last_became_peered":"2024-03-24T01:32:44.743592+0000","last_unstale":"2024-03-24T01:32:45.774617+0000","last_undegraded":"2024-03-24T01:32:45.774617+0000","last_fullsized":"2024-03-24T01:32:45.774617+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:57:31.407474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"67'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.768411+0000","last_change":"2024-03-24T01:32:43.720477+0000","last_active":"2024-03-24T01:32:45.768411+0000","last_peered":"2024-03-24T01:32:45.768411+0000","last_clean":"2024-03-24T01:32:45.768411+0000","last_became_active":"2024-03-24T01:32:43.720192+0000","last_became_peered":"2024-03-24T01:32:43.720192+0000","last_unstale":"2024-03-24T01:32:45.768411+0000","last_undegraded":"2024-03-24T01:32:45.768411+0000","last_fullsized":"2024-03-24T01:32:45.768411+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:31:44.560907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.768054+0000","last_change":"2024-03-24T01:32:44.752631+0000","last_active":"2024-03-24T01:32:45.768054+0000","last_peered":"2024-03-24T01:32:45.768054+0000","last_clean":"2024-03-24T01:32:45.768054+0000","last_became_active":"2024-03-24T01:32:44.749471+0000","last_became_peered":"2024-03-24T01:32:44.749471+0000","last_unstale":"2024-03-24T01:32:45.768054+0000","last_undegraded":"2024-03-24T01:32:45.768054+0000","last_fullsized":"2024-03-24T01:32:45.768054+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:41:56.230713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.118232+0000","last_change":"2024-03-24T01:32:43.725481+0000","last_active":"2024-03-24T01:32:46.118232+0000","last_peered":"2024-03-24T01:32:46.118232+0000","last_clean":"2024-03-24T01:32:46.118232+0000","last_became_active":"2024-03-24T01:32:43.725185+0000","last_became_peered":"2024-03-24T01:32:43.725185+0000","last_unstale":"2024-03-24T01:32:46.118232+0000","last_undegraded":"2024-03-24T01:32:46.118232+0000","last_fullsized":"2024-03-24T01:32:46.118232+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:48:17.796412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.721918+0000","last_change":"2024-03-24T01:32:44.742817+0000","last_active":"2024-03-24T01:32:45.721918+0000","last_peered":"2024-03-24T01:32:45.721918+0000","last_clean":"2024-03-24T01:32:45.721918+0000","last_became_active":"2024-03-24T01:32:44.741341+0000","last_became_peered":"2024-03-24T01:32:44.741341+0000","last_unstale":"2024-03-24T01:32:45.721918+0000","last_undegraded":"2024-03-24T01:32:45.721918+0000","last_fullsized":"2024-03-24T01:32:45.721918+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:15:20.815746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.714077+0000","last_change":"2024-03-24T01:32:43.714827+0000","last_active":"2024-03-24T01:32:45.714077+0000","last_peered":"2024-03-24T01:32:45.714077+0000","last_clean":"2024-03-24T01:32:45.714077+0000","last_became_active":"2024-03-24T01:32:43.714552+0000","last_became_peered":"2024-03-24T01:32:43.714552+0000","last_unstale":"2024-03-24T01:32:45.714077+0000","last_undegraded":"2024-03-24T01:32:45.714077+0000","last_fullsized":"2024-03-24T01:32:45.714077+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:11:39.983567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.720541+0000","last_change":"2024-03-24T01:32:44.747784+0000","last_active":"2024-03-24T01:32:45.720541+0000","last_peered":"2024-03-24T01:32:45.720541+0000","last_clean":"2024-03-24T01:32:45.720541+0000","last_became_active":"2024-03-24T01:32:44.747485+0000","last_became_peered":"2024-03-24T01:32:44.747485+0000","last_unstale":"2024-03-24T01:32:45.720541+0000","last_undegraded":"2024-03-24T01:32:45.720541+0000","last_fullsized":"2024-03-24T01:32:45.720541+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:20:55.112893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117921+0000","last_change":"2024-03-24T01:32:43.724564+0000","last_active":"2024-03-24T01:32:46.117921+0000","last_peered":"2024-03-24T01:32:46.117921+0000","last_clean":"2024-03-24T01:32:46.117921+0000","last_became_active":"2024-03-24T01:32:43.724051+0000","last_became_peered":"2024-03-24T01:32:43.724051+0000","last_unstale":"2024-03-24T01:32:46.117921+0000","last_undegraded":"2024-03-24T01:32:46.117921+0000","last_fullsized":"2024-03-24T01:32:46.117921+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:20:54.097910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.721135+0000","last_change":"2024-03-24T01:32:44.741872+0000","last_active":"2024-03-24T01:32:45.721135+0000","last_peered":"2024-03-24T01:32:45.721135+0000","last_clean":"2024-03-24T01:32:45.721135+0000","last_became_active":"2024-03-24T01:32:44.741584+0000","last_became_peered":"2024-03-24T01:32:44.741584+0000","last_unstale":"2024-03-24T01:32:45.721135+0000","last_undegraded":"2024-03-24T01:32:45.721135+0000","last_fullsized":"2024-03-24T01:32:45.721135+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:03.806877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117780+0000","last_change":"2024-03-24T01:32:43.715420+0000","last_active":"2024-03-24T01:32:46.117780+0000","last_peered":"2024-03-24T01:32:46.117780+0000","last_clean":"2024-03-24T01:32:46.117780+0000","last_became_active":"2024-03-24T01:32:43.714725+0000","last_became_peered":"2024-03-24T01:32:43.714725+0000","last_unstale":"2024-03-24T01:32:46.117780+0000","last_undegraded":"2024-03-24T01:32:46.117780+0000","last_fullsized":"2024-03-24T01:32:46.117780+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:56:29.495016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.713744+0000","last_change":"2024-03-24T01:32:44.745758+0000","last_active":"2024-03-24T01:32:45.713744+0000","last_peered":"2024-03-24T01:32:45.713744+0000","last_clean":"2024-03-24T01:32:45.713744+0000","last_became_active":"2024-03-24T01:32:44.745122+0000","last_became_peered":"2024-03-24T01:32:44.745122+0000","last_unstale":"2024-03-24T01:32:45.713744+0000","last_undegraded":"2024-03-24T01:32:45.713744+0000","last_fullsized":"2024-03-24T01:32:45.713744+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:15:20.815746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.116394+0000","last_change":"2024-03-24T01:32:43.719188+0000","last_active":"2024-03-24T01:32:46.116394+0000","last_peered":"2024-03-24T01:32:46.116394+0000","last_clean":"2024-03-24T01:32:46.116394+0000","last_became_active":"2024-03-24T01:32:43.718755+0000","last_became_peered":"2024-03-24T01:32:43.718755+0000","last_unstale":"2024-03-24T01:32:46.116394+0000","last_undegraded":"2024-03-24T01:32:46.116394+0000","last_fullsized":"2024-03-24T01:32:46.116394+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:33:20.618669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.774097+0000","last_change":"2024-03-24T01:32:44.753907+0000","last_active":"2024-03-24T01:32:45.774097+0000","last_peered":"2024-03-24T01:32:45.774097+0000","last_clean":"2024-03-24T01:32:45.774097+0000","last_became_active":"2024-03-24T01:32:44.753568+0000","last_became_peered":"2024-03-24T01:32:44.753568+0000","last_unstale":"2024-03-24T01:32:45.774097+0000","last_undegraded":"2024-03-24T01:32:45.774097+0000","last_fullsized":"2024-03-24T01:32:45.774097+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:05:52.841516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.713120+0000","last_change":"2024-03-24T01:32:43.714616+0000","last_active":"2024-03-24T01:32:45.713120+0000","last_peered":"2024-03-24T01:32:45.713120+0000","last_clean":"2024-03-24T01:32:45.713120+0000","last_became_active":"2024-03-24T01:32:43.714216+0000","last_became_peered":"2024-03-24T01:32:43.714216+0000","last_unstale":"2024-03-24T01:32:45.713120+0000","last_undegraded":"2024-03-24T01:32:45.713120+0000","last_fullsized":"2024-03-24T01:32:45.713120+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:31:15.858645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.722226+0000","last_change":"2024-03-24T01:32:44.744990+0000","last_active":"2024-03-24T01:32:45.722226+0000","last_peered":"2024-03-24T01:32:45.722226+0000","last_clean":"2024-03-24T01:32:45.722226+0000","last_became_active":"2024-03-24T01:32:44.744503+0000","last_became_peered":"2024-03-24T01:32:44.744503+0000","last_unstale":"2024-03-24T01:32:45.722226+0000","last_undegraded":"2024-03-24T01:32:45.722226+0000","last_fullsized":"2024-03-24T01:32:45.722226+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:39:23.561488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.116219+0000","last_change":"2024-03-24T01:32:43.719380+0000","last_active":"2024-03-24T01:32:46.116219+0000","last_peered":"2024-03-24T01:32:46.116219+0000","last_clean":"2024-03-24T01:32:46.116219+0000","last_became_active":"2024-03-24T01:32:43.718702+0000","last_became_peered":"2024-03-24T01:32:43.718702+0000","last_unstale":"2024-03-24T01:32:46.116219+0000","last_undegraded":"2024-03-24T01:32:46.116219+0000","last_fullsized":"2024-03-24T01:32:46.116219+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:41:55.215730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":259,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117100+0000","last_change":"2024-03-24T01:24:01.545855+0000","last_active":"2024-03-24T01:32:46.117100+0000","last_peered":"2024-03-24T01:32:46.117100+0000","last_clean":"2024-03-24T01:32:46.117100+0000","last_became_active":"2024-03-24T01:24:01.543811+0000","last_became_peered":"2024-03-24T01:24:01.543811+0000","last_unstale":"2024-03-24T01:32:46.117100+0000","last_undegraded":"2024-03-24T01:32:46.117100+0000","last_fullsized":"2024-03-24T01:32:46.117100+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_clean_scrub_stamp":"2024-03-24T01:23:59.861564+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:03.682631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574957,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32568,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339272,"statfs":{"total":96636764160,"available":96603414528,"internally_reserved":0,"allocated":1421312,"data_stored":963205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574957,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":815104,"data_stored":372095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574958,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":815104,"data_stored":372083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574957,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31992,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339848,"statfs":{"total":96636764160,"available":96604004352,"internally_reserved":0,"allocated":823296,"data_stored":373673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574957,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31976,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339864,"statfs":{"total":96636764160,"available":96604020736,"internally_reserved":0,"allocated":811008,"data_stored":372061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574958,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31976,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339864,"statfs":{"total":96636764160,"available":96604020736,"internally_reserved":0,"allocated":811008,"data_stored":372061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574957,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32008,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339832,"statfs":{"total":96636764160,"available":96603987968,"internally_reserved":0,"allocated":839680,"data_stored":374437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574958,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32560,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339280,"statfs":{"total":96636764160,"available":96603422720,"internally_reserved":0,"allocated":1404928,"data_stored":962429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T01:32:50.185 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-24T01:32:50.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.434+0000 7f857d800640 1 Processor -- start 2024-03-24T01:32:50.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.434+0000 7f857d800640 1 -- start start 2024-03-24T01:32:50.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.435+0000 7f857d800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f85780b4580 0x7f857806f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:50.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.435+0000 7f857d800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f857815a660 con 0x7f857815a280 2024-03-24T01:32:50.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.436+0000 7f857d800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f857806dac0 con 0x7f857806f080 2024-03-24T01:32:50.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.436+0000 7f857d800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f857806b7d0 con 0x7f85780b4580 2024-03-24T01:32:50.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.436+0000 7f8577400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f85780b4580 0x7f857806f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:50.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.436+0000 7f8577400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f85780b4580 0x7f857806f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34438/0 (socket says 172.21.15.90:34438) 2024-03-24T01:32:50.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.436+0000 7f8577400640 1 -- 172.21.15.90:0/781626619 learned_addr learned my addr 172.21.15.90:0/781626619 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:50.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.436+0000 7f8576a00640 1 --1- 172.21.15.90:0/781626619 >> v1:172.21.15.90:6789/0 conn(0x7f857815a280 0x7f857806b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43120/0 (socket says 172.21.15.90:43120) 2024-03-24T01:32:50.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.436+0000 7f8577400640 1 -- 172.21.15.90:0/781626619 >> v1:172.21.15.120:6789/0 conn(0x7f857806f080 legacy=0x7f85780b3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:50.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.436+0000 7f8577400640 1 -- 172.21.15.90:0/781626619 >> v1:172.21.15.90:6789/0 conn(0x7f857815a280 legacy=0x7f857806b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:50.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.436+0000 7f8577400640 1 -- 172.21.15.90:0/781626619 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f857806dda0 con 0x7f85780b4580 2024-03-24T01:32:50.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.437+0000 7f8577400640 1 --2- 172.21.15.90:0/781626619 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85780b4580 0x7f857806f4e0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f8570009e50 tx=0x7f857002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cdb83aacbd606b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:50.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.438+0000 7f8576000640 1 -- 172.21.15.90:0/781626619 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857002ca80 con 0x7f85780b4580 2024-03-24T01:32:50.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.438+0000 7f8576000640 1 -- 172.21.15.90:0/781626619 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f857002cc20 con 0x7f85780b4580 2024-03-24T01:32:50.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.438+0000 7f8576000640 1 -- 172.21.15.90:0/781626619 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85700343f0 con 0x7f85780b4580 2024-03-24T01:32:50.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.438+0000 7f857d800640 1 -- 172.21.15.90:0/781626619 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85780b4580 msgr2=0x7f857806f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:50.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.438+0000 7f857d800640 1 --2- 172.21.15.90:0/781626619 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85780b4580 0x7f857806f4e0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f8570009e50 tx=0x7f857002bee0 comp rx=0 tx=0).stop 2024-03-24T01:32:50.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.438+0000 7f857d800640 1 -- 172.21.15.90:0/781626619 shutdown_connections 2024-03-24T01:32:50.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.438+0000 7f857d800640 1 --2- 172.21.15.90:0/781626619 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85780b4580 0x7f857806f4e0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:50.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.438+0000 7f857d800640 1 -- 172.21.15.90:0/781626619 >> 172.21.15.90:0/781626619 conn(0x7f85780b1770 msgr2=0x7f85780b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:50.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.438+0000 7f857d800640 1 -- 172.21.15.90:0/781626619 shutdown_connections 2024-03-24T01:32:50.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.438+0000 7f857d800640 1 -- 172.21.15.90:0/781626619 wait complete. 2024-03-24T01:32:50.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.440+0000 7f857d800640 1 Processor -- start 2024-03-24T01:32:50.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.440+0000 7f857d800640 1 -- start start 2024-03-24T01:32:50.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.441+0000 7f857d800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f857806f080 0x7f85781554a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:50.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.441+0000 7f857d800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85780b4580 0x7f857814b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:50.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.441+0000 7f857d800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f857815a280 0x7f8578154810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:50.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.441+0000 7f857d800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85780b4a10 con 0x7f857806f080 2024-03-24T01:32:50.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.441+0000 7f857d800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f857806e360 con 0x7f85780b4580 2024-03-24T01:32:50.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.441+0000 7f857d800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f857806e640 con 0x7f857815a280 2024-03-24T01:32:50.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.441+0000 7f8576a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f857806f080 0x7f85781554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:50.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.441+0000 7f8576a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f857806f080 0x7f85781554a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34452/0 (socket says 172.21.15.90:34452) 2024-03-24T01:32:50.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.441+0000 7f8576a00640 1 -- 172.21.15.90:0/2720511226 learned_addr learned my addr 172.21.15.90:0/2720511226 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:50.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.442+0000 7f8576a00640 1 -- 172.21.15.90:0/2720511226 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f857815a280 msgr2=0x7f8578154810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:50.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.442+0000 7f8576a00640 1 --2- 172.21.15.90:0/2720511226 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f857815a280 0x7f8578154810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:50.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.442+0000 7f8576a00640 1 -- 172.21.15.90:0/2720511226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85780b4580 msgr2=0x7f857814b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:50.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.442+0000 7f856e000640 1 --2- 172.21.15.90:0/2720511226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85780b4580 0x7f857814b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:50.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.442+0000 7f8576a00640 1 --2- 172.21.15.90:0/2720511226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85780b4580 0x7f857814b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:50.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.442+0000 7f8576a00640 1 -- 172.21.15.90:0/2720511226 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8578155b30 con 0x7f857806f080 2024-03-24T01:32:50.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.442+0000 7f856e000640 1 --2- 172.21.15.90:0/2720511226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85780b4580 0x7f857814b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:50.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.442+0000 7f8576a00640 1 --2- 172.21.15.90:0/2720511226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f857806f080 0x7f85781554a0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f856800c430 tx=0x7f856800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:50.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.442+0000 7f856f400640 1 -- 172.21.15.90:0/2720511226 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85680074e0 con 0x7f857806f080 2024-03-24T01:32:50.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.442+0000 7f856f400640 1 -- 172.21.15.90:0/2720511226 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8568007680 con 0x7f857806f080 2024-03-24T01:32:50.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.442+0000 7f856f400640 1 -- 172.21.15.90:0/2720511226 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8568004a00 con 0x7f857806f080 2024-03-24T01:32:50.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.442+0000 7f857d800640 1 -- 172.21.15.90:0/2720511226 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8578154d50 con 0x7f857806f080 2024-03-24T01:32:50.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.442+0000 7f857d800640 1 -- 172.21.15.90:0/2720511226 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85780ae180 con 0x7f857806f080 2024-03-24T01:32:50.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.443+0000 7f857d800640 1 -- 172.21.15.90:0/2720511226 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8578157a80 con 0x7f857806f080 2024-03-24T01:32:50.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.444+0000 7f856f400640 1 -- 172.21.15.90:0/2720511226 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f856801a020 con 0x7f857806f080 2024-03-24T01:32:50.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.445+0000 7f856f400640 1 --2- 172.21.15.90:0/2720511226 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f855406bc00 0x7f855406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:50.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.445+0000 7f856f400640 1 -- 172.21.15.90:0/2720511226 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f856808a4d0 con 0x7f857806f080 2024-03-24T01:32:50.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.446+0000 7f856e000640 1 --2- 172.21.15.90:0/2720511226 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f855406bc00 0x7f855406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:50.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.447+0000 7f856e000640 1 --2- 172.21.15.90:0/2720511226 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f855406bc00 0x7f855406e0c0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f85780b4210 tx=0x7f8560008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:50.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.449+0000 7f856f400640 1 -- 172.21.15.90:0/2720511226 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8578155b30 con 0x7f857806f080 2024-03-24T01:32:50.668 INFO:tasks.ceph.mds.d.smithi120.stderr:2024-03-24T01:32:50.668+0000 7ff94c800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T01:32:50.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.698+0000 7f857d800640 1 -- 172.21.15.90:0/2720511226 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f8578068060 con 0x7f857806f080 2024-03-24T01:32:50.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.699+0000 7f856f400640 1 -- 172.21.15.90:0/2720511226 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1014 (secure 0 0 0) 0x7f857815a660 con 0x7f857806f080 2024-03-24T01:32:50.700 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:32:50.700 INFO:teuthology.orchestra.run.smithi090.stdout:{"fsid":"24712165-9839-4fcc-8033-988386efea74","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":544,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":67,"num_osds":8,"num_up_osds":8,"osd_up_since":1711243438,"num_in_osds":8,"osd_in_since":1711243429,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":24,"data_bytes":592778,"bytes_used":263217152,"bytes_avail":772830896128,"bytes_total":773094113280,"write_bytes_sec":2388,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":50,"id":4,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":4,"rank":0,"name":"d","status":"up:active","gid":7110}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-03-24T01:31:41.652088+0000","services":{}},"progress_events":{}} 2024-03-24T01:32:50.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.702+0000 7f857d800640 1 -- 172.21.15.90:0/2720511226 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f855406bc00 msgr2=0x7f855406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:50.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.702+0000 7f857d800640 1 --2- 172.21.15.90:0/2720511226 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f855406bc00 0x7f855406e0c0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f85780b4210 tx=0x7f8560008040 comp rx=0 tx=0).stop 2024-03-24T01:32:50.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.702+0000 7f857d800640 1 -- 172.21.15.90:0/2720511226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f857806f080 msgr2=0x7f85781554a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:50.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.702+0000 7f857d800640 1 --2- 172.21.15.90:0/2720511226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f857806f080 0x7f85781554a0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f856800c430 tx=0x7f856800c8f0 comp rx=0 tx=0).stop 2024-03-24T01:32:50.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.702+0000 7f857d800640 1 -- 172.21.15.90:0/2720511226 shutdown_connections 2024-03-24T01:32:50.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.702+0000 7f857d800640 1 --2- 172.21.15.90:0/2720511226 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f855406bc00 0x7f855406e0c0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:50.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.702+0000 7f857d800640 1 --2- 172.21.15.90:0/2720511226 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f857815a280 0x7f8578154810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:50.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.702+0000 7f857d800640 1 --2- 172.21.15.90:0/2720511226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85780b4580 0x7f857814b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:50.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.702+0000 7f857d800640 1 --2- 172.21.15.90:0/2720511226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f857806f080 0x7f85781554a0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:50.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.702+0000 7f857d800640 1 -- 172.21.15.90:0/2720511226 >> 172.21.15.90:0/2720511226 conn(0x7f85780b1770 msgr2=0x7f85781594c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:50.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.702+0000 7f857d800640 1 -- 172.21.15.90:0/2720511226 shutdown_connections 2024-03-24T01:32:50.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.702+0000 7f857d800640 1 -- 172.21.15.90:0/2720511226 wait complete. 2024-03-24T01:32:50.719 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-24T01:32:50.719 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T01:32:50.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.927+0000 7f1c2a400640 1 Processor -- start 2024-03-24T01:32:50.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.927+0000 7f1c2a400640 1 -- start start 2024-03-24T01:32:50.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.928+0000 7f1c2a400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c24155030 0x7f1c2415efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:50.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.928+0000 7f1c2a400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f1c240cdac0 0x7f1c2406b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:50.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.928+0000 7f1c2a400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1c2406b600 0x7f1c240ae530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:50.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.928+0000 7f1c2a400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c2406ba70 con 0x7f1c2406b600 2024-03-24T01:32:50.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.928+0000 7f1c2a400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c2415f500 con 0x7f1c240cdac0 2024-03-24T01:32:50.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.928+0000 7f1c2a400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2415bf70 con 0x7f1c24155030 2024-03-24T01:32:50.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.929+0000 7f1c23e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1c2406b600 0x7f1c240ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:50.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.929+0000 7f1c23e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1c2406b600 0x7f1c240ae530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34460/0 (socket says 172.21.15.90:34460) 2024-03-24T01:32:50.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.929+0000 7f1c23e00640 1 -- 172.21.15.90:0/829314963 learned_addr learned my addr 172.21.15.90:0/829314963 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:50.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.929+0000 7f1c23400640 1 --2- 172.21.15.90:0/829314963 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c24155030 0x7f1c2415efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:50.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.929+0000 7f1c1aa00640 1 --2- 172.21.15.90:0/829314963 >> v2:172.21.15.120:3300/0 conn(0x7f1c240cdac0 0x7f1c2406b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:50.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.929+0000 7f1c23e00640 1 -- 172.21.15.90:0/829314963 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c24155030 msgr2=0x7f1c2415efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:50.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.929+0000 7f1c23e00640 1 --2- 172.21.15.90:0/829314963 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c24155030 0x7f1c2415efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:50.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.929+0000 7f1c23e00640 1 -- 172.21.15.90:0/829314963 >> v2:172.21.15.120:3300/0 conn(0x7f1c240cdac0 msgr2=0x7f1c2406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:50.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.929+0000 7f1c23e00640 1 --2- 172.21.15.90:0/829314963 >> v2:172.21.15.120:3300/0 conn(0x7f1c240cdac0 0x7f1c2406b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:50.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.929+0000 7f1c23e00640 1 -- 172.21.15.90:0/829314963 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c2415b360 con 0x7f1c2406b600 2024-03-24T01:32:50.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.930+0000 7f1c23e00640 1 --2- 172.21.15.90:0/829314963 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c2406b600 0x7f1c240ae530 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f1c24072c20 tx=0x7f1c1c02e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eff8fe6fc830a4aa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:50.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.931+0000 7f1c22a00640 1 -- 172.21.15.90:0/829314963 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c02ec80 con 0x7f1c2406b600 2024-03-24T01:32:50.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.931+0000 7f1c22a00640 1 -- 172.21.15.90:0/829314963 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c1c02ee20 con 0x7f1c2406b600 2024-03-24T01:32:50.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.931+0000 7f1c22a00640 1 -- 172.21.15.90:0/829314963 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c036610 con 0x7f1c2406b600 2024-03-24T01:32:50.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.931+0000 7f1c2a400640 1 -- 172.21.15.90:0/829314963 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c2406b600 msgr2=0x7f1c240ae530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:50.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.931+0000 7f1c2a400640 1 --2- 172.21.15.90:0/829314963 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c2406b600 0x7f1c240ae530 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f1c24072c20 tx=0x7f1c1c02e0e0 comp rx=0 tx=0).stop 2024-03-24T01:32:50.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.931+0000 7f1c2a400640 1 -- 172.21.15.90:0/829314963 shutdown_connections 2024-03-24T01:32:50.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.931+0000 7f1c2a400640 1 --2- 172.21.15.90:0/829314963 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c2406b600 0x7f1c240ae530 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:50.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.931+0000 7f1c2a400640 1 --2- 172.21.15.90:0/829314963 >> v2:172.21.15.120:3300/0 conn(0x7f1c240cdac0 0x7f1c2406b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:50.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.932+0000 7f1c2a400640 1 --2- 172.21.15.90:0/829314963 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c24155030 0x7f1c2415efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:50.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.932+0000 7f1c2a400640 1 -- 172.21.15.90:0/829314963 >> 172.21.15.90:0/829314963 conn(0x7f1c240b1770 msgr2=0x7f1c240b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:50.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.932+0000 7f1c2a400640 1 -- 172.21.15.90:0/829314963 shutdown_connections 2024-03-24T01:32:50.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.932+0000 7f1c2a400640 1 -- 172.21.15.90:0/829314963 wait complete. 2024-03-24T01:32:50.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.933+0000 7f1c2a400640 1 Processor -- start 2024-03-24T01:32:50.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.933+0000 7f1c2a400640 1 -- start start 2024-03-24T01:32:50.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.934+0000 7f1c2a400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c2406b600 0x7f1c240b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:50.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.934+0000 7f1c2a400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c240cdac0 0x7f1c240cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:50.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.934+0000 7f1c2a400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c24155030 0x7f1c240cb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:50.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.934+0000 7f1c2a400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2415bc00 con 0x7f1c2406b600 2024-03-24T01:32:50.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.934+0000 7f1c2a400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2415b920 con 0x7f1c24155030 2024-03-24T01:32:50.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.934+0000 7f1c2a400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c24155410 con 0x7f1c240cdac0 2024-03-24T01:32:50.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.935+0000 7f1c23400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c2406b600 0x7f1c240b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:50.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.935+0000 7f1c23400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c2406b600 0x7f1c240b31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34472/0 (socket says 172.21.15.90:34472) 2024-03-24T01:32:50.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.935+0000 7f1c23400640 1 -- 172.21.15.90:0/612344505 learned_addr learned my addr 172.21.15.90:0/612344505 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:50.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.935+0000 7f1c1aa00640 1 --2- 172.21.15.90:0/612344505 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c240cdac0 0x7f1c240cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:50.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.935+0000 7f1c23e00640 1 --2- 172.21.15.90:0/612344505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c24155030 0x7f1c240cb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:50.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.935+0000 7f1c23400640 1 -- 172.21.15.90:0/612344505 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c240cdac0 msgr2=0x7f1c240cb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:50.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.935+0000 7f1c23400640 1 --2- 172.21.15.90:0/612344505 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c240cdac0 0x7f1c240cb0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:50.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.935+0000 7f1c23400640 1 -- 172.21.15.90:0/612344505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c24155030 msgr2=0x7f1c240cb5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:50.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.935+0000 7f1c23400640 1 --2- 172.21.15.90:0/612344505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c24155030 0x7f1c240cb5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:50.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.935+0000 7f1c23400640 1 -- 172.21.15.90:0/612344505 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c241f6e50 con 0x7f1c2406b600 2024-03-24T01:32:50.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.935+0000 7f1c23400640 1 --2- 172.21.15.90:0/612344505 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c2406b600 0x7f1c240b31d0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f1c1400d5f0 tx=0x7f1c1400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:50.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.936+0000 7f1c1be00640 1 -- 172.21.15.90:0/612344505 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c140125f0 con 0x7f1c2406b600 2024-03-24T01:32:50.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.936+0000 7f1c1be00640 1 -- 172.21.15.90:0/612344505 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c14019070 con 0x7f1c2406b600 2024-03-24T01:32:50.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.936+0000 7f1c1be00640 1 -- 172.21.15.90:0/612344505 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c140129d0 con 0x7f1c2406b600 2024-03-24T01:32:50.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.936+0000 7f1c2a400640 1 -- 172.21.15.90:0/612344505 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c241f7bd0 con 0x7f1c2406b600 2024-03-24T01:32:50.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.936+0000 7f1c2a400640 1 -- 172.21.15.90:0/612344505 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c241f76b0 con 0x7f1c2406b600 2024-03-24T01:32:50.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.937+0000 7f1c2a400640 1 -- 172.21.15.90:0/612344505 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c24068060 con 0x7f1c2406b600 2024-03-24T01:32:50.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.938+0000 7f1c1be00640 1 -- 172.21.15.90:0/612344505 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f1c14012b70 con 0x7f1c2406b600 2024-03-24T01:32:50.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.939+0000 7f1c1be00640 1 --2- 172.21.15.90:0/612344505 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1c0006bbd0 0x7f1c0006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:50.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.939+0000 7f1c1be00640 1 -- 172.21.15.90:0/612344505 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1c1408af90 con 0x7f1c2406b600 2024-03-24T01:32:50.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.939+0000 7f1c1aa00640 1 --2- 172.21.15.90:0/612344505 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1c0006bbd0 0x7f1c0006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:50.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.940+0000 7f1c1aa00640 1 --2- 172.21.15.90:0/612344505 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1c0006bbd0 0x7f1c0006e090 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f1c240c90d0 tx=0x7f1c0c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:50.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:50.943+0000 7f1c1be00640 1 -- 172.21.15.90:0/612344505 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f1c140598f0 con 0x7f1c2406b600 2024-03-24T01:32:51.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:51.137+0000 7f1c2a400640 1 -- 172.21.15.90:0/612344505 --> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1c241f7020 con 0x7f1c0006bbd0 2024-03-24T01:32:51.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:51.155+0000 7f1c1be00640 1 -- 172.21.15.90:0/612344505 <== mgr.4104 v2:172.21.15.120:6832/860392634 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252020 (secure 0 0 0) 0x7f1c241f7020 con 0x7f1c0006bbd0 2024-03-24T01:32:51.159 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:32:51.161 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-24T01:32:51.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:51.162+0000 7f1c2a400640 1 -- 172.21.15.90:0/612344505 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1c0006bbd0 msgr2=0x7f1c0006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:51.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:51.162+0000 7f1c2a400640 1 --2- 172.21.15.90:0/612344505 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1c0006bbd0 0x7f1c0006e090 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f1c240c90d0 tx=0x7f1c0c008040 comp rx=0 tx=0).stop 2024-03-24T01:32:51.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:51.162+0000 7f1c2a400640 1 -- 172.21.15.90:0/612344505 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c2406b600 msgr2=0x7f1c240b31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:51.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:51.162+0000 7f1c2a400640 1 --2- 172.21.15.90:0/612344505 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c2406b600 0x7f1c240b31d0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f1c1400d5f0 tx=0x7f1c1400dab0 comp rx=0 tx=0).stop 2024-03-24T01:32:51.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:51.162+0000 7f1c2a400640 1 -- 172.21.15.90:0/612344505 shutdown_connections 2024-03-24T01:32:51.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:51.162+0000 7f1c2a400640 1 --2- 172.21.15.90:0/612344505 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f1c0006bbd0 0x7f1c0006e090 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:51.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:51.162+0000 7f1c2a400640 1 --2- 172.21.15.90:0/612344505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c24155030 0x7f1c240cb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:51.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:51.162+0000 7f1c2a400640 1 --2- 172.21.15.90:0/612344505 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f1c240cdac0 0x7f1c240cb0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:51.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:51.162+0000 7f1c2a400640 1 --2- 172.21.15.90:0/612344505 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c2406b600 0x7f1c240b31d0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:51.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:51.162+0000 7f1c2a400640 1 -- 172.21.15.90:0/612344505 >> 172.21.15.90:0/612344505 conn(0x7f1c240b1770 msgr2=0x7f1c2415e130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:51.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:51.162+0000 7f1c2a400640 1 -- 172.21.15.90:0/612344505 shutdown_connections 2024-03-24T01:32:51.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:51.162+0000 7f1c2a400640 1 -- 172.21.15.90:0/612344505 wait complete. 2024-03-24T01:32:51.181 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-03-24T01:32:49.676278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":257048,"kb_used_data":7560,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717672,"statfs":{"total":773094113280,"available":772830896128,"internally_reserved":0,"allocated":7741440,"data_stored":4162044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12747,"internal_metadata":255315509},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002027"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.713955+0000","last_change":"2024-03-24T01:32:44.751583+0000","last_active":"2024-03-24T01:32:45.713955+0000","last_peered":"2024-03-24T01:32:45.713955+0000","last_clean":"2024-03-24T01:32:45.713955+0000","last_became_active":"2024-03-24T01:32:44.751290+0000","last_became_peered":"2024-03-24T01:32:44.751290+0000","last_unstale":"2024-03-24T01:32:45.713955+0000","last_undegraded":"2024-03-24T01:32:45.713955+0000","last_fullsized":"2024-03-24T01:32:45.713955+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:14:56.287011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.713664+0000","last_change":"2024-03-24T01:32:44.751982+0000","last_active":"2024-03-24T01:32:45.713664+0000","last_peered":"2024-03-24T01:32:45.713664+0000","last_clean":"2024-03-24T01:32:45.713664+0000","last_became_active":"2024-03-24T01:32:44.751686+0000","last_became_peered":"2024-03-24T01:32:44.751686+0000","last_unstale":"2024-03-24T01:32:45.713664+0000","last_undegraded":"2024-03-24T01:32:45.713664+0000","last_fullsized":"2024-03-24T01:32:45.713664+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:03.806877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117942+0000","last_change":"2024-03-24T01:32:44.761410+0000","last_active":"2024-03-24T01:32:46.117942+0000","last_peered":"2024-03-24T01:32:46.117942+0000","last_clean":"2024-03-24T01:32:46.117942+0000","last_became_active":"2024-03-24T01:32:44.760599+0000","last_became_peered":"2024-03-24T01:32:44.760599+0000","last_unstale":"2024-03-24T01:32:46.117942+0000","last_undegraded":"2024-03-24T01:32:46.117942+0000","last_fullsized":"2024-03-24T01:32:46.117942+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:33:13.594934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.720760+0000","last_change":"2024-03-24T01:32:44.735449+0000","last_active":"2024-03-24T01:32:45.720760+0000","last_peered":"2024-03-24T01:32:45.720760+0000","last_clean":"2024-03-24T01:32:45.720760+0000","last_became_active":"2024-03-24T01:32:44.734765+0000","last_became_peered":"2024-03-24T01:32:44.734765+0000","last_unstale":"2024-03-24T01:32:45.720760+0000","last_undegraded":"2024-03-24T01:32:45.720760+0000","last_fullsized":"2024-03-24T01:32:45.720760+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:12.824577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.720672+0000","last_change":"2024-03-24T01:32:44.731161+0000","last_active":"2024-03-24T01:32:45.720672+0000","last_peered":"2024-03-24T01:32:45.720672+0000","last_clean":"2024-03-24T01:32:45.720672+0000","last_became_active":"2024-03-24T01:32:44.730863+0000","last_became_peered":"2024-03-24T01:32:44.730863+0000","last_unstale":"2024-03-24T01:32:45.720672+0000","last_undegraded":"2024-03-24T01:32:45.720672+0000","last_fullsized":"2024-03-24T01:32:45.720672+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:13:48.874647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117287+0000","last_change":"2024-03-24T01:32:44.741905+0000","last_active":"2024-03-24T01:32:46.117287+0000","last_peered":"2024-03-24T01:32:46.117287+0000","last_clean":"2024-03-24T01:32:46.117287+0000","last_became_active":"2024-03-24T01:32:44.741599+0000","last_became_peered":"2024-03-24T01:32:44.741599+0000","last_unstale":"2024-03-24T01:32:46.117287+0000","last_undegraded":"2024-03-24T01:32:46.117287+0000","last_fullsized":"2024-03-24T01:32:46.117287+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:19:45.922169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.773953+0000","last_change":"2024-03-24T01:32:44.755846+0000","last_active":"2024-03-24T01:32:45.773953+0000","last_peered":"2024-03-24T01:32:45.773953+0000","last_clean":"2024-03-24T01:32:45.773953+0000","last_became_active":"2024-03-24T01:32:44.755533+0000","last_became_peered":"2024-03-24T01:32:44.755533+0000","last_unstale":"2024-03-24T01:32:45.773953+0000","last_undegraded":"2024-03-24T01:32:45.773953+0000","last_fullsized":"2024-03-24T01:32:45.773953+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:00:32.057217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.767176+0000","last_change":"2024-03-24T01:32:44.754059+0000","last_active":"2024-03-24T01:32:45.767176+0000","last_peered":"2024-03-24T01:32:45.767176+0000","last_clean":"2024-03-24T01:32:45.767176+0000","last_became_active":"2024-03-24T01:32:44.753503+0000","last_became_peered":"2024-03-24T01:32:44.753503+0000","last_unstale":"2024-03-24T01:32:45.767176+0000","last_undegraded":"2024-03-24T01:32:45.767176+0000","last_fullsized":"2024-03-24T01:32:45.767176+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:39:23.561488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.118564+0000","last_change":"2024-03-24T01:32:44.761804+0000","last_active":"2024-03-24T01:32:46.118564+0000","last_peered":"2024-03-24T01:32:46.118564+0000","last_clean":"2024-03-24T01:32:46.118564+0000","last_became_active":"2024-03-24T01:32:44.761263+0000","last_became_peered":"2024-03-24T01:32:44.761263+0000","last_unstale":"2024-03-24T01:32:46.118564+0000","last_undegraded":"2024-03-24T01:32:46.118564+0000","last_fullsized":"2024-03-24T01:32:46.118564+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:31:50.925888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.118044+0000","last_change":"2024-03-24T01:32:44.755965+0000","last_active":"2024-03-24T01:32:46.118044+0000","last_peered":"2024-03-24T01:32:46.118044+0000","last_clean":"2024-03-24T01:32:46.118044+0000","last_became_active":"2024-03-24T01:32:44.755708+0000","last_became_peered":"2024-03-24T01:32:44.755708+0000","last_unstale":"2024-03-24T01:32:46.118044+0000","last_undegraded":"2024-03-24T01:32:46.118044+0000","last_fullsized":"2024-03-24T01:32:46.118044+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:33:21.633652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.719972+0000","last_change":"2024-03-24T01:32:44.750963+0000","last_active":"2024-03-24T01:32:45.719972+0000","last_peered":"2024-03-24T01:32:45.719972+0000","last_clean":"2024-03-24T01:32:45.719972+0000","last_became_active":"2024-03-24T01:32:44.750351+0000","last_became_peered":"2024-03-24T01:32:44.750351+0000","last_unstale":"2024-03-24T01:32:45.719972+0000","last_undegraded":"2024-03-24T01:32:45.719972+0000","last_fullsized":"2024-03-24T01:32:45.719972+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:13:48.874647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.767953+0000","last_change":"2024-03-24T01:32:44.752161+0000","last_active":"2024-03-24T01:32:45.767953+0000","last_peered":"2024-03-24T01:32:45.767953+0000","last_clean":"2024-03-24T01:32:45.767953+0000","last_became_active":"2024-03-24T01:32:44.751842+0000","last_became_peered":"2024-03-24T01:32:44.751842+0000","last_unstale":"2024-03-24T01:32:45.767953+0000","last_undegraded":"2024-03-24T01:32:45.767953+0000","last_fullsized":"2024-03-24T01:32:45.767953+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:48:18.811395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.769621+0000","last_change":"2024-03-24T01:32:44.746912+0000","last_active":"2024-03-24T01:32:45.769621+0000","last_peered":"2024-03-24T01:32:45.769621+0000","last_clean":"2024-03-24T01:32:45.769621+0000","last_became_active":"2024-03-24T01:32:44.746271+0000","last_became_peered":"2024-03-24T01:32:44.746271+0000","last_unstale":"2024-03-24T01:32:45.769621+0000","last_undegraded":"2024-03-24T01:32:45.769621+0000","last_fullsized":"2024-03-24T01:32:45.769621+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:33:21.633652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117510+0000","last_change":"2024-03-24T01:32:44.758588+0000","last_active":"2024-03-24T01:32:46.117510+0000","last_peered":"2024-03-24T01:32:46.117510+0000","last_clean":"2024-03-24T01:32:46.117510+0000","last_became_active":"2024-03-24T01:32:44.757289+0000","last_became_peered":"2024-03-24T01:32:44.757289+0000","last_unstale":"2024-03-24T01:32:46.117510+0000","last_undegraded":"2024-03-24T01:32:46.117510+0000","last_fullsized":"2024-03-24T01:32:46.117510+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:41:56.230713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117149+0000","last_change":"2024-03-24T01:32:44.753415+0000","last_active":"2024-03-24T01:32:46.117149+0000","last_peered":"2024-03-24T01:32:46.117149+0000","last_clean":"2024-03-24T01:32:46.117149+0000","last_became_active":"2024-03-24T01:32:44.752869+0000","last_became_peered":"2024-03-24T01:32:44.752869+0000","last_unstale":"2024-03-24T01:32:46.117149+0000","last_undegraded":"2024-03-24T01:32:46.117149+0000","last_fullsized":"2024-03-24T01:32:46.117149+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:48:18.811395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.773869+0000","last_change":"2024-03-24T01:32:44.742945+0000","last_active":"2024-03-24T01:32:45.773869+0000","last_peered":"2024-03-24T01:32:45.773869+0000","last_clean":"2024-03-24T01:32:45.773869+0000","last_became_active":"2024-03-24T01:32:44.742329+0000","last_became_peered":"2024-03-24T01:32:44.742329+0000","last_unstale":"2024-03-24T01:32:45.773869+0000","last_undegraded":"2024-03-24T01:32:45.773869+0000","last_fullsized":"2024-03-24T01:32:45.773869+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:40:03.643169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.714047+0000","last_change":"2024-03-24T01:32:44.745728+0000","last_active":"2024-03-24T01:32:45.714047+0000","last_peered":"2024-03-24T01:32:45.714047+0000","last_clean":"2024-03-24T01:32:45.714047+0000","last_became_active":"2024-03-24T01:32:44.745412+0000","last_became_peered":"2024-03-24T01:32:44.745412+0000","last_unstale":"2024-03-24T01:32:45.714047+0000","last_undegraded":"2024-03-24T01:32:45.714047+0000","last_fullsized":"2024-03-24T01:32:45.714047+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:36:57.078745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.720116+0000","last_change":"2024-03-24T01:32:44.735333+0000","last_active":"2024-03-24T01:32:45.720116+0000","last_peered":"2024-03-24T01:32:45.720116+0000","last_clean":"2024-03-24T01:32:45.720116+0000","last_became_active":"2024-03-24T01:32:44.734813+0000","last_became_peered":"2024-03-24T01:32:44.734813+0000","last_unstale":"2024-03-24T01:32:45.720116+0000","last_undegraded":"2024-03-24T01:32:45.720116+0000","last_fullsized":"2024-03-24T01:32:45.720116+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:15:20.815746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.721498+0000","last_change":"2024-03-24T01:32:44.750885+0000","last_active":"2024-03-24T01:32:45.721498+0000","last_peered":"2024-03-24T01:32:45.721498+0000","last_clean":"2024-03-24T01:32:45.721498+0000","last_became_active":"2024-03-24T01:32:44.750355+0000","last_became_peered":"2024-03-24T01:32:44.750355+0000","last_unstale":"2024-03-24T01:32:45.721498+0000","last_undegraded":"2024-03-24T01:32:45.721498+0000","last_fullsized":"2024-03-24T01:32:45.721498+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:35.662274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.118346+0000","last_change":"2024-03-24T01:32:44.734704+0000","last_active":"2024-03-24T01:32:46.118346+0000","last_peered":"2024-03-24T01:32:46.118346+0000","last_clean":"2024-03-24T01:32:46.118346+0000","last_became_active":"2024-03-24T01:32:44.733687+0000","last_became_peered":"2024-03-24T01:32:44.733687+0000","last_unstale":"2024-03-24T01:32:46.118346+0000","last_undegraded":"2024-03-24T01:32:46.118346+0000","last_fullsized":"2024-03-24T01:32:46.118346+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:50:50.757791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.768693+0000","last_change":"2024-03-24T01:32:44.755202+0000","last_active":"2024-03-24T01:32:45.768693+0000","last_peered":"2024-03-24T01:32:45.768693+0000","last_clean":"2024-03-24T01:32:45.768693+0000","last_became_active":"2024-03-24T01:32:44.754775+0000","last_became_peered":"2024-03-24T01:32:44.754775+0000","last_unstale":"2024-03-24T01:32:45.768693+0000","last_undegraded":"2024-03-24T01:32:45.768693+0000","last_fullsized":"2024-03-24T01:32:45.768693+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:33:13.594934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.118707+0000","last_change":"2024-03-24T01:32:44.741995+0000","last_active":"2024-03-24T01:32:46.118707+0000","last_peered":"2024-03-24T01:32:46.118707+0000","last_clean":"2024-03-24T01:32:46.118707+0000","last_became_active":"2024-03-24T01:32:44.741671+0000","last_became_peered":"2024-03-24T01:32:44.741671+0000","last_unstale":"2024-03-24T01:32:46.118707+0000","last_undegraded":"2024-03-24T01:32:46.118707+0000","last_fullsized":"2024-03-24T01:32:46.118707+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:52:15.663148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117020+0000","last_change":"2024-03-24T01:32:44.740789+0000","last_active":"2024-03-24T01:32:46.117020+0000","last_peered":"2024-03-24T01:32:46.117020+0000","last_clean":"2024-03-24T01:32:46.117020+0000","last_became_active":"2024-03-24T01:32:44.740465+0000","last_became_peered":"2024-03-24T01:32:44.740465+0000","last_unstale":"2024-03-24T01:32:46.117020+0000","last_undegraded":"2024-03-24T01:32:46.117020+0000","last_fullsized":"2024-03-24T01:32:46.117020+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:02:39.321334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.116993+0000","last_change":"2024-03-24T01:32:44.758180+0000","last_active":"2024-03-24T01:32:46.116993+0000","last_peered":"2024-03-24T01:32:46.116993+0000","last_clean":"2024-03-24T01:32:46.116993+0000","last_became_active":"2024-03-24T01:32:44.756763+0000","last_became_peered":"2024-03-24T01:32:44.756763+0000","last_unstale":"2024-03-24T01:32:46.116993+0000","last_undegraded":"2024-03-24T01:32:46.116993+0000","last_fullsized":"2024-03-24T01:32:46.116993+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:59:12.918357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.768651+0000","last_change":"2024-03-24T01:32:44.753078+0000","last_active":"2024-03-24T01:32:45.768651+0000","last_peered":"2024-03-24T01:32:45.768651+0000","last_clean":"2024-03-24T01:32:45.768651+0000","last_became_active":"2024-03-24T01:32:44.750365+0000","last_became_peered":"2024-03-24T01:32:44.750365+0000","last_unstale":"2024-03-24T01:32:45.768651+0000","last_undegraded":"2024-03-24T01:32:45.768651+0000","last_fullsized":"2024-03-24T01:32:45.768651+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:56:30.509999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.767804+0000","last_change":"2024-03-24T01:32:44.747414+0000","last_active":"2024-03-24T01:32:45.767804+0000","last_peered":"2024-03-24T01:32:45.767804+0000","last_clean":"2024-03-24T01:32:45.767804+0000","last_became_active":"2024-03-24T01:32:44.747138+0000","last_became_peered":"2024-03-24T01:32:44.747138+0000","last_unstale":"2024-03-24T01:32:45.767804+0000","last_undegraded":"2024-03-24T01:32:45.767804+0000","last_fullsized":"2024-03-24T01:32:45.767804+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:59:12.918357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.719958+0000","last_change":"2024-03-24T01:32:44.746689+0000","last_active":"2024-03-24T01:32:45.719958+0000","last_peered":"2024-03-24T01:32:45.719958+0000","last_clean":"2024-03-24T01:32:45.719958+0000","last_became_active":"2024-03-24T01:32:44.746359+0000","last_became_peered":"2024-03-24T01:32:44.746359+0000","last_unstale":"2024-03-24T01:32:45.719958+0000","last_undegraded":"2024-03-24T01:32:45.719958+0000","last_fullsized":"2024-03-24T01:32:45.719958+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:36.843294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.713789+0000","last_change":"2024-03-24T01:32:44.749783+0000","last_active":"2024-03-24T01:32:45.713789+0000","last_peered":"2024-03-24T01:32:45.713789+0000","last_clean":"2024-03-24T01:32:45.713789+0000","last_became_active":"2024-03-24T01:32:44.749266+0000","last_became_peered":"2024-03-24T01:32:44.749266+0000","last_unstale":"2024-03-24T01:32:45.713789+0000","last_undegraded":"2024-03-24T01:32:45.713789+0000","last_fullsized":"2024-03-24T01:32:45.713789+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:39:10.149776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117669+0000","last_change":"2024-03-24T01:32:44.751695+0000","last_active":"2024-03-24T01:32:46.117669+0000","last_peered":"2024-03-24T01:32:46.117669+0000","last_clean":"2024-03-24T01:32:46.117669+0000","last_became_active":"2024-03-24T01:32:44.751409+0000","last_became_peered":"2024-03-24T01:32:44.751409+0000","last_unstale":"2024-03-24T01:32:46.117669+0000","last_undegraded":"2024-03-24T01:32:46.117669+0000","last_fullsized":"2024-03-24T01:32:46.117669+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:14:45.725936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.767559+0000","last_change":"2024-03-24T01:32:44.752696+0000","last_active":"2024-03-24T01:32:45.767559+0000","last_peered":"2024-03-24T01:32:45.767559+0000","last_clean":"2024-03-24T01:32:45.767559+0000","last_became_active":"2024-03-24T01:32:44.752378+0000","last_became_peered":"2024-03-24T01:32:44.752378+0000","last_unstale":"2024-03-24T01:32:45.767559+0000","last_undegraded":"2024-03-24T01:32:45.767559+0000","last_fullsized":"2024-03-24T01:32:45.767559+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:36:57.078745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.721483+0000","last_change":"2024-03-24T01:32:44.742966+0000","last_active":"2024-03-24T01:32:45.721483+0000","last_peered":"2024-03-24T01:32:45.721483+0000","last_clean":"2024-03-24T01:32:45.721483+0000","last_became_active":"2024-03-24T01:32:44.742681+0000","last_became_peered":"2024-03-24T01:32:44.742681+0000","last_unstale":"2024-03-24T01:32:45.721483+0000","last_undegraded":"2024-03-24T01:32:45.721483+0000","last_fullsized":"2024-03-24T01:32:45.721483+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:39:10.149776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.116803+0000","last_change":"2024-03-24T01:32:44.744799+0000","last_active":"2024-03-24T01:32:46.116803+0000","last_peered":"2024-03-24T01:32:46.116803+0000","last_clean":"2024-03-24T01:32:46.116803+0000","last_became_active":"2024-03-24T01:32:44.743828+0000","last_became_peered":"2024-03-24T01:32:44.743828+0000","last_unstale":"2024-03-24T01:32:46.116803+0000","last_undegraded":"2024-03-24T01:32:46.116803+0000","last_fullsized":"2024-03-24T01:32:46.116803+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:31:50.925888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.721397+0000","last_change":"2024-03-24T01:32:44.748610+0000","last_active":"2024-03-24T01:32:45.721397+0000","last_peered":"2024-03-24T01:32:45.721397+0000","last_clean":"2024-03-24T01:32:45.721397+0000","last_became_active":"2024-03-24T01:32:44.748297+0000","last_became_peered":"2024-03-24T01:32:44.748297+0000","last_unstale":"2024-03-24T01:32:45.721397+0000","last_undegraded":"2024-03-24T01:32:45.721397+0000","last_fullsized":"2024-03-24T01:32:45.721397+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:19.032155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"66'4","reported_seq":18,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.714111+0000","last_change":"2024-03-24T01:32:43.729112+0000","last_active":"2024-03-24T01:32:45.714111+0000","last_peered":"2024-03-24T01:32:45.714111+0000","last_clean":"2024-03-24T01:32:45.714111+0000","last_became_active":"2024-03-24T01:32:43.728882+0000","last_became_peered":"2024-03-24T01:32:43.728882+0000","last_unstale":"2024-03-24T01:32:45.714111+0000","last_undegraded":"2024-03-24T01:32:45.714111+0000","last_fullsized":"2024-03-24T01:32:45.714111+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:29.157467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.720464+0000","last_change":"2024-03-24T01:32:44.733668+0000","last_active":"2024-03-24T01:32:45.720464+0000","last_peered":"2024-03-24T01:32:45.720464+0000","last_clean":"2024-03-24T01:32:45.720464+0000","last_became_active":"2024-03-24T01:32:44.733182+0000","last_became_peered":"2024-03-24T01:32:44.733182+0000","last_unstale":"2024-03-24T01:32:45.720464+0000","last_undegraded":"2024-03-24T01:32:45.720464+0000","last_fullsized":"2024-03-24T01:32:45.720464+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:19.032155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"66'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.118869+0000","last_change":"2024-03-24T01:32:43.723642+0000","last_active":"2024-03-24T01:32:46.118869+0000","last_peered":"2024-03-24T01:32:46.118869+0000","last_clean":"2024-03-24T01:32:46.118869+0000","last_became_active":"2024-03-24T01:32:43.723374+0000","last_became_peered":"2024-03-24T01:32:43.723374+0000","last_unstale":"2024-03-24T01:32:46.118869+0000","last_undegraded":"2024-03-24T01:32:46.118869+0000","last_fullsized":"2024-03-24T01:32:46.118869+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:16:58.498006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.118687+0000","last_change":"2024-03-24T01:32:44.743311+0000","last_active":"2024-03-24T01:32:46.118687+0000","last_peered":"2024-03-24T01:32:46.118687+0000","last_clean":"2024-03-24T01:32:46.118687+0000","last_became_active":"2024-03-24T01:32:44.743027+0000","last_became_peered":"2024-03-24T01:32:44.743027+0000","last_unstale":"2024-03-24T01:32:46.118687+0000","last_undegraded":"2024-03-24T01:32:46.118687+0000","last_fullsized":"2024-03-24T01:32:46.118687+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:03:37.421881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117972+0000","last_change":"2024-03-24T01:32:43.722317+0000","last_active":"2024-03-24T01:32:46.117972+0000","last_peered":"2024-03-24T01:32:46.117972+0000","last_clean":"2024-03-24T01:32:46.117972+0000","last_became_active":"2024-03-24T01:32:43.721576+0000","last_became_peered":"2024-03-24T01:32:43.721576+0000","last_unstale":"2024-03-24T01:32:46.117972+0000","last_undegraded":"2024-03-24T01:32:46.117972+0000","last_fullsized":"2024-03-24T01:32:46.117972+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:36:56.063762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.118276+0000","last_change":"2024-03-24T01:32:44.755395+0000","last_active":"2024-03-24T01:32:46.118276+0000","last_peered":"2024-03-24T01:32:46.118276+0000","last_clean":"2024-03-24T01:32:46.118276+0000","last_became_active":"2024-03-24T01:32:44.755127+0000","last_became_peered":"2024-03-24T01:32:44.755127+0000","last_unstale":"2024-03-24T01:32:46.118276+0000","last_undegraded":"2024-03-24T01:32:46.118276+0000","last_fullsized":"2024-03-24T01:32:46.118276+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:03:13.359822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117409+0000","last_change":"2024-03-24T01:32:43.713108+0000","last_active":"2024-03-24T01:32:46.117409+0000","last_peered":"2024-03-24T01:32:46.117409+0000","last_clean":"2024-03-24T01:32:46.117409+0000","last_became_active":"2024-03-24T01:32:43.712800+0000","last_became_peered":"2024-03-24T01:32:43.712800+0000","last_unstale":"2024-03-24T01:32:46.117409+0000","last_undegraded":"2024-03-24T01:32:46.117409+0000","last_fullsized":"2024-03-24T01:32:46.117409+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:59:11.903374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.774108+0000","last_change":"2024-03-24T01:32:44.747444+0000","last_active":"2024-03-24T01:32:45.774108+0000","last_peered":"2024-03-24T01:32:45.774108+0000","last_clean":"2024-03-24T01:32:45.774108+0000","last_became_active":"2024-03-24T01:32:44.747117+0000","last_became_peered":"2024-03-24T01:32:44.747117+0000","last_unstale":"2024-03-24T01:32:45.774108+0000","last_undegraded":"2024-03-24T01:32:45.774108+0000","last_fullsized":"2024-03-24T01:32:45.774108+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:24:34.023784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"66'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117538+0000","last_change":"2024-03-24T01:32:43.718182+0000","last_active":"2024-03-24T01:32:46.117538+0000","last_peered":"2024-03-24T01:32:46.117538+0000","last_clean":"2024-03-24T01:32:46.117538+0000","last_became_active":"2024-03-24T01:32:43.717674+0000","last_became_peered":"2024-03-24T01:32:43.717674+0000","last_unstale":"2024-03-24T01:32:46.117538+0000","last_undegraded":"2024-03-24T01:32:46.117538+0000","last_fullsized":"2024-03-24T01:32:46.117538+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:31:49.910904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.713179+0000","last_change":"2024-03-24T01:32:44.746900+0000","last_active":"2024-03-24T01:32:45.713179+0000","last_peered":"2024-03-24T01:32:45.713179+0000","last_clean":"2024-03-24T01:32:45.713179+0000","last_became_active":"2024-03-24T01:32:44.746619+0000","last_became_peered":"2024-03-24T01:32:44.746619+0000","last_unstale":"2024-03-24T01:32:45.713179+0000","last_undegraded":"2024-03-24T01:32:45.713179+0000","last_fullsized":"2024-03-24T01:32:45.713179+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:12.824577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"66'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117671+0000","last_change":"2024-03-24T01:32:43.717225+0000","last_active":"2024-03-24T01:32:46.117671+0000","last_peered":"2024-03-24T01:32:46.117671+0000","last_clean":"2024-03-24T01:32:46.117671+0000","last_became_active":"2024-03-24T01:32:43.716994+0000","last_became_peered":"2024-03-24T01:32:43.716994+0000","last_unstale":"2024-03-24T01:32:46.117671+0000","last_undegraded":"2024-03-24T01:32:46.117671+0000","last_fullsized":"2024-03-24T01:32:46.117671+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:14:44.710953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.721632+0000","last_change":"2024-03-24T01:32:44.733286+0000","last_active":"2024-03-24T01:32:45.721632+0000","last_peered":"2024-03-24T01:32:45.721632+0000","last_clean":"2024-03-24T01:32:45.721632+0000","last_became_active":"2024-03-24T01:32:44.732956+0000","last_became_peered":"2024-03-24T01:32:44.732956+0000","last_unstale":"2024-03-24T01:32:45.721632+0000","last_undegraded":"2024-03-24T01:32:45.721632+0000","last_fullsized":"2024-03-24T01:32:45.721632+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:12.824577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.116267+0000","last_change":"2024-03-24T01:32:43.719216+0000","last_active":"2024-03-24T01:32:46.116267+0000","last_peered":"2024-03-24T01:32:46.116267+0000","last_clean":"2024-03-24T01:32:46.116267+0000","last_became_active":"2024-03-24T01:32:43.718879+0000","last_became_peered":"2024-03-24T01:32:43.718879+0000","last_unstale":"2024-03-24T01:32:46.116267+0000","last_undegraded":"2024-03-24T01:32:46.116267+0000","last_fullsized":"2024-03-24T01:32:46.116267+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:03:12.344839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117429+0000","last_change":"2024-03-24T01:32:44.740874+0000","last_active":"2024-03-24T01:32:46.117429+0000","last_peered":"2024-03-24T01:32:46.117429+0000","last_clean":"2024-03-24T01:32:46.117429+0000","last_became_active":"2024-03-24T01:32:44.740563+0000","last_became_peered":"2024-03-24T01:32:44.740563+0000","last_unstale":"2024-03-24T01:32:46.117429+0000","last_undegraded":"2024-03-24T01:32:46.117429+0000","last_fullsized":"2024-03-24T01:32:46.117429+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:11:10.464938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.719133+0000","last_change":"2024-03-24T01:32:43.720754+0000","last_active":"2024-03-24T01:32:45.719133+0000","last_peered":"2024-03-24T01:32:45.719133+0000","last_clean":"2024-03-24T01:32:45.719133+0000","last_became_active":"2024-03-24T01:32:43.720465+0000","last_became_peered":"2024-03-24T01:32:43.720465+0000","last_unstale":"2024-03-24T01:32:45.719133+0000","last_undegraded":"2024-03-24T01:32:45.719133+0000","last_fullsized":"2024-03-24T01:32:45.719133+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:40:54.360077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.713983+0000","last_change":"2024-03-24T01:32:44.750883+0000","last_active":"2024-03-24T01:32:45.713983+0000","last_peered":"2024-03-24T01:32:45.713983+0000","last_clean":"2024-03-24T01:32:45.713983+0000","last_became_active":"2024-03-24T01:32:44.750572+0000","last_became_peered":"2024-03-24T01:32:44.750572+0000","last_unstale":"2024-03-24T01:32:45.713983+0000","last_undegraded":"2024-03-24T01:32:45.713983+0000","last_fullsized":"2024-03-24T01:32:45.713983+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:39:23.561488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.118621+0000","last_change":"2024-03-24T01:32:43.729179+0000","last_active":"2024-03-24T01:32:46.118621+0000","last_peered":"2024-03-24T01:32:46.118621+0000","last_clean":"2024-03-24T01:32:46.118621+0000","last_became_active":"2024-03-24T01:32:43.728892+0000","last_became_peered":"2024-03-24T01:32:43.728892+0000","last_unstale":"2024-03-24T01:32:46.118621+0000","last_undegraded":"2024-03-24T01:32:46.118621+0000","last_fullsized":"2024-03-24T01:32:46.118621+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:18.017172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117966+0000","last_change":"2024-03-24T01:32:44.747545+0000","last_active":"2024-03-24T01:32:46.117966+0000","last_peered":"2024-03-24T01:32:46.117966+0000","last_clean":"2024-03-24T01:32:46.117966+0000","last_became_active":"2024-03-24T01:32:44.747227+0000","last_became_peered":"2024-03-24T01:32:44.747227+0000","last_unstale":"2024-03-24T01:32:46.117966+0000","last_undegraded":"2024-03-24T01:32:46.117966+0000","last_fullsized":"2024-03-24T01:32:46.117966+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:55.474861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.721185+0000","last_change":"2024-03-24T01:32:43.715851+0000","last_active":"2024-03-24T01:32:45.721185+0000","last_peered":"2024-03-24T01:32:45.721185+0000","last_clean":"2024-03-24T01:32:45.721185+0000","last_became_active":"2024-03-24T01:32:43.715474+0000","last_became_peered":"2024-03-24T01:32:43.715474+0000","last_unstale":"2024-03-24T01:32:45.721185+0000","last_undegraded":"2024-03-24T01:32:45.721185+0000","last_fullsized":"2024-03-24T01:32:45.721185+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:24:00.025674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.721208+0000","last_change":"2024-03-24T01:32:44.748134+0000","last_active":"2024-03-24T01:32:45.721208+0000","last_peered":"2024-03-24T01:32:45.721208+0000","last_clean":"2024-03-24T01:32:45.721208+0000","last_became_active":"2024-03-24T01:32:44.747771+0000","last_became_peered":"2024-03-24T01:32:44.747771+0000","last_unstale":"2024-03-24T01:32:45.721208+0000","last_undegraded":"2024-03-24T01:32:45.721208+0000","last_fullsized":"2024-03-24T01:32:45.721208+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:36.843294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"66'3","reported_seq":17,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.118252+0000","last_change":"2024-03-24T01:32:43.725497+0000","last_active":"2024-03-24T01:32:46.118252+0000","last_peered":"2024-03-24T01:32:46.118252+0000","last_clean":"2024-03-24T01:32:46.118252+0000","last_became_active":"2024-03-24T01:32:43.724751+0000","last_became_peered":"2024-03-24T01:32:43.724751+0000","last_unstale":"2024-03-24T01:32:46.118252+0000","last_undegraded":"2024-03-24T01:32:46.118252+0000","last_fullsized":"2024-03-24T01:32:46.118252+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:35.828311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.720698+0000","last_change":"2024-03-24T01:32:44.737244+0000","last_active":"2024-03-24T01:32:45.720698+0000","last_peered":"2024-03-24T01:32:45.720698+0000","last_clean":"2024-03-24T01:32:45.720698+0000","last_became_active":"2024-03-24T01:32:44.735810+0000","last_became_peered":"2024-03-24T01:32:44.735810+0000","last_unstale":"2024-03-24T01:32:45.720698+0000","last_undegraded":"2024-03-24T01:32:45.720698+0000","last_fullsized":"2024-03-24T01:32:45.720698+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:35.662274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"66'6","reported_seq":20,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.714077+0000","last_change":"2024-03-24T01:32:43.716683+0000","last_active":"2024-03-24T01:32:45.714077+0000","last_peered":"2024-03-24T01:32:45.714077+0000","last_clean":"2024-03-24T01:32:45.714077+0000","last_became_active":"2024-03-24T01:32:43.716407+0000","last_became_peered":"2024-03-24T01:32:43.716407+0000","last_unstale":"2024-03-24T01:32:45.714077+0000","last_undegraded":"2024-03-24T01:32:45.714077+0000","last_fullsized":"2024-03-24T01:32:45.714077+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:43:54.681008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.718926+0000","last_change":"2024-03-24T01:32:44.741777+0000","last_active":"2024-03-24T01:32:45.718926+0000","last_peered":"2024-03-24T01:32:45.718926+0000","last_clean":"2024-03-24T01:32:45.718926+0000","last_became_active":"2024-03-24T01:32:44.741472+0000","last_became_peered":"2024-03-24T01:32:44.741472+0000","last_unstale":"2024-03-24T01:32:45.718926+0000","last_undegraded":"2024-03-24T01:32:45.718926+0000","last_fullsized":"2024-03-24T01:32:45.718926+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:20:55.112893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117614+0000","last_change":"2024-03-24T01:32:43.725509+0000","last_active":"2024-03-24T01:32:46.117614+0000","last_peered":"2024-03-24T01:32:46.117614+0000","last_clean":"2024-03-24T01:32:46.117614+0000","last_became_active":"2024-03-24T01:32:43.725142+0000","last_became_peered":"2024-03-24T01:32:43.725142+0000","last_unstale":"2024-03-24T01:32:46.117614+0000","last_undegraded":"2024-03-24T01:32:46.117614+0000","last_fullsized":"2024-03-24T01:32:46.117614+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:20:54.257187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.767647+0000","last_change":"2024-03-24T01:32:44.750397+0000","last_active":"2024-03-24T01:32:45.767647+0000","last_peered":"2024-03-24T01:32:45.767647+0000","last_clean":"2024-03-24T01:32:45.767647+0000","last_became_active":"2024-03-24T01:32:44.750002+0000","last_became_peered":"2024-03-24T01:32:44.750002+0000","last_unstale":"2024-03-24T01:32:45.767647+0000","last_undegraded":"2024-03-24T01:32:45.767647+0000","last_fullsized":"2024-03-24T01:32:45.767647+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:16:59.512989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.767678+0000","last_change":"2024-03-24T01:32:43.720435+0000","last_active":"2024-03-24T01:32:45.767678+0000","last_peered":"2024-03-24T01:32:45.767678+0000","last_clean":"2024-03-24T01:32:45.767678+0000","last_became_active":"2024-03-24T01:32:43.720135+0000","last_became_peered":"2024-03-24T01:32:43.720135+0000","last_unstale":"2024-03-24T01:32:45.767678+0000","last_undegraded":"2024-03-24T01:32:45.767678+0000","last_fullsized":"2024-03-24T01:32:45.767678+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:11:39.983567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.722236+0000","last_change":"2024-03-24T01:32:44.751131+0000","last_active":"2024-03-24T01:32:45.722236+0000","last_peered":"2024-03-24T01:32:45.722236+0000","last_clean":"2024-03-24T01:32:45.722236+0000","last_became_active":"2024-03-24T01:32:44.750794+0000","last_became_peered":"2024-03-24T01:32:44.750794+0000","last_unstale":"2024-03-24T01:32:45.722236+0000","last_undegraded":"2024-03-24T01:32:45.722236+0000","last_fullsized":"2024-03-24T01:32:45.722236+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:14:56.287011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117416+0000","last_change":"2024-03-24T01:32:43.725306+0000","last_active":"2024-03-24T01:32:46.117416+0000","last_peered":"2024-03-24T01:32:46.117416+0000","last_clean":"2024-03-24T01:32:46.117416+0000","last_became_active":"2024-03-24T01:32:43.724101+0000","last_became_peered":"2024-03-24T01:32:43.724101+0000","last_unstale":"2024-03-24T01:32:46.117416+0000","last_undegraded":"2024-03-24T01:32:46.117416+0000","last_fullsized":"2024-03-24T01:32:46.117416+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:12:55.184668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.713378+0000","last_change":"2024-03-24T01:32:44.750404+0000","last_active":"2024-03-24T01:32:45.713378+0000","last_peered":"2024-03-24T01:32:45.713378+0000","last_clean":"2024-03-24T01:32:45.713378+0000","last_became_active":"2024-03-24T01:32:44.750099+0000","last_became_peered":"2024-03-24T01:32:44.750099+0000","last_unstale":"2024-03-24T01:32:45.713378+0000","last_undegraded":"2024-03-24T01:32:45.713378+0000","last_fullsized":"2024-03-24T01:32:45.713378+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:10:36.135152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"66'8","reported_seq":22,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.719601+0000","last_change":"2024-03-24T01:32:43.720664+0000","last_active":"2024-03-24T01:32:45.719601+0000","last_peered":"2024-03-24T01:32:45.719601+0000","last_clean":"2024-03-24T01:32:45.719601+0000","last_became_active":"2024-03-24T01:32:43.720338+0000","last_became_peered":"2024-03-24T01:32:43.720338+0000","last_unstale":"2024-03-24T01:32:45.719601+0000","last_undegraded":"2024-03-24T01:32:45.719601+0000","last_fullsized":"2024-03-24T01:32:45.719601+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:40:54.360077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117882+0000","last_change":"2024-03-24T01:32:44.762349+0000","last_active":"2024-03-24T01:32:46.117882+0000","last_peered":"2024-03-24T01:32:46.117882+0000","last_clean":"2024-03-24T01:32:46.117882+0000","last_became_active":"2024-03-24T01:32:44.761475+0000","last_became_peered":"2024-03-24T01:32:44.761475+0000","last_unstale":"2024-03-24T01:32:46.117882+0000","last_undegraded":"2024-03-24T01:32:46.117882+0000","last_fullsized":"2024-03-24T01:32:46.117882+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:56:30.509999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"66'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.768808+0000","last_change":"2024-03-24T01:32:43.722152+0000","last_active":"2024-03-24T01:32:45.768808+0000","last_peered":"2024-03-24T01:32:45.768808+0000","last_clean":"2024-03-24T01:32:45.768808+0000","last_became_active":"2024-03-24T01:32:43.721855+0000","last_became_peered":"2024-03-24T01:32:43.721855+0000","last_unstale":"2024-03-24T01:32:45.768808+0000","last_undegraded":"2024-03-24T01:32:45.768808+0000","last_fullsized":"2024-03-24T01:32:45.768808+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:20:54.257187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.721804+0000","last_change":"2024-03-24T01:32:44.728581+0000","last_active":"2024-03-24T01:32:45.721804+0000","last_peered":"2024-03-24T01:32:45.721804+0000","last_clean":"2024-03-24T01:32:45.721804+0000","last_became_active":"2024-03-24T01:32:44.728154+0000","last_became_peered":"2024-03-24T01:32:44.728154+0000","last_unstale":"2024-03-24T01:32:45.721804+0000","last_undegraded":"2024-03-24T01:32:45.721804+0000","last_fullsized":"2024-03-24T01:32:45.721804+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:03.806877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.118235+0000","last_change":"2024-03-24T01:32:43.726876+0000","last_active":"2024-03-24T01:32:46.118235+0000","last_peered":"2024-03-24T01:32:46.118235+0000","last_clean":"2024-03-24T01:32:46.118235+0000","last_became_active":"2024-03-24T01:32:43.726628+0000","last_became_peered":"2024-03-24T01:32:43.726628+0000","last_unstale":"2024-03-24T01:32:46.118235+0000","last_undegraded":"2024-03-24T01:32:46.118235+0000","last_fullsized":"2024-03-24T01:32:46.118235+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:34.647291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.720919+0000","last_change":"2024-03-24T01:32:44.733825+0000","last_active":"2024-03-24T01:32:45.720919+0000","last_peered":"2024-03-24T01:32:45.720919+0000","last_clean":"2024-03-24T01:32:45.720919+0000","last_became_active":"2024-03-24T01:32:44.733404+0000","last_became_peered":"2024-03-24T01:32:44.733404+0000","last_unstale":"2024-03-24T01:32:45.720919+0000","last_undegraded":"2024-03-24T01:32:45.720919+0000","last_fullsized":"2024-03-24T01:32:45.720919+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:10:36.135152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"66'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117934+0000","last_change":"2024-03-24T01:32:43.725389+0000","last_active":"2024-03-24T01:32:46.117934+0000","last_peered":"2024-03-24T01:32:46.117934+0000","last_clean":"2024-03-24T01:32:46.117934+0000","last_became_active":"2024-03-24T01:32:43.724956+0000","last_became_peered":"2024-03-24T01:32:43.724956+0000","last_unstale":"2024-03-24T01:32:46.117934+0000","last_undegraded":"2024-03-24T01:32:46.117934+0000","last_fullsized":"2024-03-24T01:32:46.117934+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:13:47.859663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.721685+0000","last_change":"2024-03-24T01:32:44.743865+0000","last_active":"2024-03-24T01:32:45.721685+0000","last_peered":"2024-03-24T01:32:45.721685+0000","last_clean":"2024-03-24T01:32:45.721685+0000","last_became_active":"2024-03-24T01:32:44.741745+0000","last_became_peered":"2024-03-24T01:32:44.741745+0000","last_unstale":"2024-03-24T01:32:45.721685+0000","last_undegraded":"2024-03-24T01:32:45.721685+0000","last_fullsized":"2024-03-24T01:32:45.721685+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:10:36.135152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117487+0000","last_change":"2024-03-24T01:32:43.721896+0000","last_active":"2024-03-24T01:32:46.117487+0000","last_peered":"2024-03-24T01:32:46.117487+0000","last_clean":"2024-03-24T01:32:46.117487+0000","last_became_active":"2024-03-24T01:32:43.720277+0000","last_became_peered":"2024-03-24T01:32:43.720277+0000","last_unstale":"2024-03-24T01:32:46.117487+0000","last_undegraded":"2024-03-24T01:32:46.117487+0000","last_fullsized":"2024-03-24T01:32:46.117487+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:33:12.579951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.116685+0000","last_change":"2024-03-24T01:32:44.743387+0000","last_active":"2024-03-24T01:32:46.116685+0000","last_peered":"2024-03-24T01:32:46.116685+0000","last_clean":"2024-03-24T01:32:46.116685+0000","last_became_active":"2024-03-24T01:32:44.743079+0000","last_became_peered":"2024-03-24T01:32:44.743079+0000","last_unstale":"2024-03-24T01:32:46.116685+0000","last_undegraded":"2024-03-24T01:32:46.116685+0000","last_fullsized":"2024-03-24T01:32:46.116685+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:03:13.359822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117048+0000","last_change":"2024-03-24T01:32:43.707322+0000","last_active":"2024-03-24T01:32:46.117048+0000","last_peered":"2024-03-24T01:32:46.117048+0000","last_clean":"2024-03-24T01:32:46.117048+0000","last_became_active":"2024-03-24T01:32:43.706928+0000","last_became_peered":"2024-03-24T01:32:43.706928+0000","last_unstale":"2024-03-24T01:32:46.117048+0000","last_undegraded":"2024-03-24T01:32:46.117048+0000","last_fullsized":"2024-03-24T01:32:46.117048+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:39:22.546505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.768153+0000","last_change":"2024-03-24T01:32:44.744400+0000","last_active":"2024-03-24T01:32:45.768153+0000","last_peered":"2024-03-24T01:32:45.768153+0000","last_clean":"2024-03-24T01:32:45.768153+0000","last_became_active":"2024-03-24T01:32:44.742926+0000","last_became_peered":"2024-03-24T01:32:44.742926+0000","last_unstale":"2024-03-24T01:32:45.768153+0000","last_undegraded":"2024-03-24T01:32:45.768153+0000","last_fullsized":"2024-03-24T01:32:45.768153+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:14:45.725936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.767725+0000","last_change":"2024-03-24T01:32:43.724088+0000","last_active":"2024-03-24T01:32:45.767725+0000","last_peered":"2024-03-24T01:32:45.767725+0000","last_clean":"2024-03-24T01:32:45.767725+0000","last_became_active":"2024-03-24T01:32:43.722732+0000","last_became_peered":"2024-03-24T01:32:43.722732+0000","last_unstale":"2024-03-24T01:32:45.767725+0000","last_undegraded":"2024-03-24T01:32:45.767725+0000","last_fullsized":"2024-03-24T01:32:45.767725+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:12:55.184668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.722030+0000","last_change":"2024-03-24T01:32:44.736167+0000","last_active":"2024-03-24T01:32:45.722030+0000","last_peered":"2024-03-24T01:32:45.722030+0000","last_clean":"2024-03-24T01:32:45.722030+0000","last_became_active":"2024-03-24T01:32:44.733302+0000","last_became_peered":"2024-03-24T01:32:44.733302+0000","last_unstale":"2024-03-24T01:32:45.722030+0000","last_undegraded":"2024-03-24T01:32:45.722030+0000","last_fullsized":"2024-03-24T01:32:45.722030+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:39:10.149776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.116796+0000","last_change":"2024-03-24T01:32:43.720278+0000","last_active":"2024-03-24T01:32:46.116796+0000","last_peered":"2024-03-24T01:32:46.116796+0000","last_clean":"2024-03-24T01:32:46.116796+0000","last_became_active":"2024-03-24T01:32:43.719995+0000","last_became_peered":"2024-03-24T01:32:43.719995+0000","last_unstale":"2024-03-24T01:32:46.116796+0000","last_undegraded":"2024-03-24T01:32:46.116796+0000","last_fullsized":"2024-03-24T01:32:46.116796+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:14:55.272028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117369+0000","last_change":"2024-03-24T01:32:44.731492+0000","last_active":"2024-03-24T01:32:46.117369+0000","last_peered":"2024-03-24T01:32:46.117369+0000","last_clean":"2024-03-24T01:32:46.117369+0000","last_became_active":"2024-03-24T01:32:44.730776+0000","last_became_peered":"2024-03-24T01:32:44.730776+0000","last_unstale":"2024-03-24T01:32:46.117369+0000","last_undegraded":"2024-03-24T01:32:46.117369+0000","last_fullsized":"2024-03-24T01:32:46.117369+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:01:46.557014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.769770+0000","last_change":"2024-03-24T01:32:43.729534+0000","last_active":"2024-03-24T01:32:45.769770+0000","last_peered":"2024-03-24T01:32:45.769770+0000","last_clean":"2024-03-24T01:32:45.769770+0000","last_became_active":"2024-03-24T01:32:43.729214+0000","last_became_peered":"2024-03-24T01:32:43.729214+0000","last_unstale":"2024-03-24T01:32:45.769770+0000","last_undegraded":"2024-03-24T01:32:45.769770+0000","last_fullsized":"2024-03-24T01:32:45.769770+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:20:54.097910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.774617+0000","last_change":"2024-03-24T01:32:44.744158+0000","last_active":"2024-03-24T01:32:45.774617+0000","last_peered":"2024-03-24T01:32:45.774617+0000","last_clean":"2024-03-24T01:32:45.774617+0000","last_became_active":"2024-03-24T01:32:44.743592+0000","last_became_peered":"2024-03-24T01:32:44.743592+0000","last_unstale":"2024-03-24T01:32:45.774617+0000","last_undegraded":"2024-03-24T01:32:45.774617+0000","last_fullsized":"2024-03-24T01:32:45.774617+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:57:31.407474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"67'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.768411+0000","last_change":"2024-03-24T01:32:43.720477+0000","last_active":"2024-03-24T01:32:45.768411+0000","last_peered":"2024-03-24T01:32:45.768411+0000","last_clean":"2024-03-24T01:32:45.768411+0000","last_became_active":"2024-03-24T01:32:43.720192+0000","last_became_peered":"2024-03-24T01:32:43.720192+0000","last_unstale":"2024-03-24T01:32:45.768411+0000","last_undegraded":"2024-03-24T01:32:45.768411+0000","last_fullsized":"2024-03-24T01:32:45.768411+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:31:44.560907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.768054+0000","last_change":"2024-03-24T01:32:44.752631+0000","last_active":"2024-03-24T01:32:45.768054+0000","last_peered":"2024-03-24T01:32:45.768054+0000","last_clean":"2024-03-24T01:32:45.768054+0000","last_became_active":"2024-03-24T01:32:44.749471+0000","last_became_peered":"2024-03-24T01:32:44.749471+0000","last_unstale":"2024-03-24T01:32:45.768054+0000","last_undegraded":"2024-03-24T01:32:45.768054+0000","last_fullsized":"2024-03-24T01:32:45.768054+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:41:56.230713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.118232+0000","last_change":"2024-03-24T01:32:43.725481+0000","last_active":"2024-03-24T01:32:46.118232+0000","last_peered":"2024-03-24T01:32:46.118232+0000","last_clean":"2024-03-24T01:32:46.118232+0000","last_became_active":"2024-03-24T01:32:43.725185+0000","last_became_peered":"2024-03-24T01:32:43.725185+0000","last_unstale":"2024-03-24T01:32:46.118232+0000","last_undegraded":"2024-03-24T01:32:46.118232+0000","last_fullsized":"2024-03-24T01:32:46.118232+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:48:17.796412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.721918+0000","last_change":"2024-03-24T01:32:44.742817+0000","last_active":"2024-03-24T01:32:45.721918+0000","last_peered":"2024-03-24T01:32:45.721918+0000","last_clean":"2024-03-24T01:32:45.721918+0000","last_became_active":"2024-03-24T01:32:44.741341+0000","last_became_peered":"2024-03-24T01:32:44.741341+0000","last_unstale":"2024-03-24T01:32:45.721918+0000","last_undegraded":"2024-03-24T01:32:45.721918+0000","last_fullsized":"2024-03-24T01:32:45.721918+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:15:20.815746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.714077+0000","last_change":"2024-03-24T01:32:43.714827+0000","last_active":"2024-03-24T01:32:45.714077+0000","last_peered":"2024-03-24T01:32:45.714077+0000","last_clean":"2024-03-24T01:32:45.714077+0000","last_became_active":"2024-03-24T01:32:43.714552+0000","last_became_peered":"2024-03-24T01:32:43.714552+0000","last_unstale":"2024-03-24T01:32:45.714077+0000","last_undegraded":"2024-03-24T01:32:45.714077+0000","last_fullsized":"2024-03-24T01:32:45.714077+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:11:39.983567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.720541+0000","last_change":"2024-03-24T01:32:44.747784+0000","last_active":"2024-03-24T01:32:45.720541+0000","last_peered":"2024-03-24T01:32:45.720541+0000","last_clean":"2024-03-24T01:32:45.720541+0000","last_became_active":"2024-03-24T01:32:44.747485+0000","last_became_peered":"2024-03-24T01:32:44.747485+0000","last_unstale":"2024-03-24T01:32:45.720541+0000","last_undegraded":"2024-03-24T01:32:45.720541+0000","last_fullsized":"2024-03-24T01:32:45.720541+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:20:55.112893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117921+0000","last_change":"2024-03-24T01:32:43.724564+0000","last_active":"2024-03-24T01:32:46.117921+0000","last_peered":"2024-03-24T01:32:46.117921+0000","last_clean":"2024-03-24T01:32:46.117921+0000","last_became_active":"2024-03-24T01:32:43.724051+0000","last_became_peered":"2024-03-24T01:32:43.724051+0000","last_unstale":"2024-03-24T01:32:46.117921+0000","last_undegraded":"2024-03-24T01:32:46.117921+0000","last_fullsized":"2024-03-24T01:32:46.117921+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:20:54.097910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.721135+0000","last_change":"2024-03-24T01:32:44.741872+0000","last_active":"2024-03-24T01:32:45.721135+0000","last_peered":"2024-03-24T01:32:45.721135+0000","last_clean":"2024-03-24T01:32:45.721135+0000","last_became_active":"2024-03-24T01:32:44.741584+0000","last_became_peered":"2024-03-24T01:32:44.741584+0000","last_unstale":"2024-03-24T01:32:45.721135+0000","last_undegraded":"2024-03-24T01:32:45.721135+0000","last_fullsized":"2024-03-24T01:32:45.721135+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:03.806877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117780+0000","last_change":"2024-03-24T01:32:43.715420+0000","last_active":"2024-03-24T01:32:46.117780+0000","last_peered":"2024-03-24T01:32:46.117780+0000","last_clean":"2024-03-24T01:32:46.117780+0000","last_became_active":"2024-03-24T01:32:43.714725+0000","last_became_peered":"2024-03-24T01:32:43.714725+0000","last_unstale":"2024-03-24T01:32:46.117780+0000","last_undegraded":"2024-03-24T01:32:46.117780+0000","last_fullsized":"2024-03-24T01:32:46.117780+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:56:29.495016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.713744+0000","last_change":"2024-03-24T01:32:44.745758+0000","last_active":"2024-03-24T01:32:45.713744+0000","last_peered":"2024-03-24T01:32:45.713744+0000","last_clean":"2024-03-24T01:32:45.713744+0000","last_became_active":"2024-03-24T01:32:44.745122+0000","last_became_peered":"2024-03-24T01:32:44.745122+0000","last_unstale":"2024-03-24T01:32:45.713744+0000","last_undegraded":"2024-03-24T01:32:45.713744+0000","last_fullsized":"2024-03-24T01:32:45.713744+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:15:20.815746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.116394+0000","last_change":"2024-03-24T01:32:43.719188+0000","last_active":"2024-03-24T01:32:46.116394+0000","last_peered":"2024-03-24T01:32:46.116394+0000","last_clean":"2024-03-24T01:32:46.116394+0000","last_became_active":"2024-03-24T01:32:43.718755+0000","last_became_peered":"2024-03-24T01:32:43.718755+0000","last_unstale":"2024-03-24T01:32:46.116394+0000","last_undegraded":"2024-03-24T01:32:46.116394+0000","last_fullsized":"2024-03-24T01:32:46.116394+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:33:20.618669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.774097+0000","last_change":"2024-03-24T01:32:44.753907+0000","last_active":"2024-03-24T01:32:45.774097+0000","last_peered":"2024-03-24T01:32:45.774097+0000","last_clean":"2024-03-24T01:32:45.774097+0000","last_became_active":"2024-03-24T01:32:44.753568+0000","last_became_peered":"2024-03-24T01:32:44.753568+0000","last_unstale":"2024-03-24T01:32:45.774097+0000","last_undegraded":"2024-03-24T01:32:45.774097+0000","last_fullsized":"2024-03-24T01:32:45.774097+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:05:52.841516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.713120+0000","last_change":"2024-03-24T01:32:43.714616+0000","last_active":"2024-03-24T01:32:45.713120+0000","last_peered":"2024-03-24T01:32:45.713120+0000","last_clean":"2024-03-24T01:32:45.713120+0000","last_became_active":"2024-03-24T01:32:43.714216+0000","last_became_peered":"2024-03-24T01:32:43.714216+0000","last_unstale":"2024-03-24T01:32:45.713120+0000","last_undegraded":"2024-03-24T01:32:45.713120+0000","last_fullsized":"2024-03-24T01:32:45.713120+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:31:15.858645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.722226+0000","last_change":"2024-03-24T01:32:44.744990+0000","last_active":"2024-03-24T01:32:45.722226+0000","last_peered":"2024-03-24T01:32:45.722226+0000","last_clean":"2024-03-24T01:32:45.722226+0000","last_became_active":"2024-03-24T01:32:44.744503+0000","last_became_peered":"2024-03-24T01:32:44.744503+0000","last_unstale":"2024-03-24T01:32:45.722226+0000","last_undegraded":"2024-03-24T01:32:45.722226+0000","last_fullsized":"2024-03-24T01:32:45.722226+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:39:23.561488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.116219+0000","last_change":"2024-03-24T01:32:43.719380+0000","last_active":"2024-03-24T01:32:46.116219+0000","last_peered":"2024-03-24T01:32:46.116219+0000","last_clean":"2024-03-24T01:32:46.116219+0000","last_became_active":"2024-03-24T01:32:43.718702+0000","last_became_peered":"2024-03-24T01:32:43.718702+0000","last_unstale":"2024-03-24T01:32:46.116219+0000","last_undegraded":"2024-03-24T01:32:46.116219+0000","last_fullsized":"2024-03-24T01:32:46.116219+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:41:55.215730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":259,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117100+0000","last_change":"2024-03-24T01:24:01.545855+0000","last_active":"2024-03-24T01:32:46.117100+0000","last_peered":"2024-03-24T01:32:46.117100+0000","last_clean":"2024-03-24T01:32:46.117100+0000","last_became_active":"2024-03-24T01:24:01.543811+0000","last_became_peered":"2024-03-24T01:24:01.543811+0000","last_unstale":"2024-03-24T01:32:46.117100+0000","last_undegraded":"2024-03-24T01:32:46.117100+0000","last_fullsized":"2024-03-24T01:32:46.117100+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_clean_scrub_stamp":"2024-03-24T01:23:59.861564+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:03.682631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574957,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32568,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339272,"statfs":{"total":96636764160,"available":96603414528,"internally_reserved":0,"allocated":1421312,"data_stored":963205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574957,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":815104,"data_stored":372095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574958,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":815104,"data_stored":372083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574957,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31992,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339848,"statfs":{"total":96636764160,"available":96604004352,"internally_reserved":0,"allocated":823296,"data_stored":373673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574957,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31976,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339864,"statfs":{"total":96636764160,"available":96604020736,"internally_reserved":0,"allocated":811008,"data_stored":372061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574958,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31976,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339864,"statfs":{"total":96636764160,"available":96604020736,"internally_reserved":0,"allocated":811008,"data_stored":372061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574957,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32008,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339832,"statfs":{"total":96636764160,"available":96603987968,"internally_reserved":0,"allocated":839680,"data_stored":374437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574958,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32560,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339280,"statfs":{"total":96636764160,"available":96603422720,"internally_reserved":0,"allocated":1404928,"data_stored":962429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T01:32:54.186 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T01:32:54.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.405+0000 7f8d1e400640 1 Processor -- start 2024-03-24T01:32:54.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.405+0000 7f8d1e400640 1 -- start start 2024-03-24T01:32:54.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.406+0000 7f8d1e400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8d180cc150 0x7f8d180ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:54.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.407+0000 7f8d1e400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8d1806b7d0 0x7f8d180b3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:54.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.407+0000 7f8d1e400640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d180cd490 con 0x7f8d181603e0 2024-03-24T01:32:54.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.407+0000 7f8d1e400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d1815f900 con 0x7f8d180cc150 2024-03-24T01:32:54.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.407+0000 7f8d1e400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d1815fbe0 con 0x7f8d1806b7d0 2024-03-24T01:32:54.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.407+0000 7f8d17400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8d180cc150 0x7f8d180ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:54.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.407+0000 7f8d17400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8d180cc150 0x7f8d180ccb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34478/0 (socket says 172.21.15.90:34478) 2024-03-24T01:32:54.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.407+0000 7f8d17400640 1 -- 172.21.15.90:0/103966010 learned_addr learned my addr 172.21.15.90:0/103966010 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:54.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.407+0000 7f8d17400640 1 -- 172.21.15.90:0/103966010 >> v2:172.21.15.120:3300/0 conn(0x7f8d1806b7d0 msgr2=0x7f8d180b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:54.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.407+0000 7f8d17400640 1 --2- 172.21.15.90:0/103966010 >> v2:172.21.15.120:3300/0 conn(0x7f8d1806b7d0 0x7f8d180b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:54.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.407+0000 7f8d17400640 1 -- 172.21.15.90:0/103966010 >> v1:172.21.15.120:6789/0 conn(0x7f8d181603e0 legacy=0x7f8d1806b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:54.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.407+0000 7f8d17400640 1 -- 172.21.15.90:0/103966010 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d1815fec0 con 0x7f8d180cc150 2024-03-24T01:32:54.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.408+0000 7f8d17400640 1 --2- 172.21.15.90:0/103966010 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d180cc150 0x7f8d180ccb60 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f8d1806fc10 tx=0x7f8d0802b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44e396eee006f9b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:54.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.409+0000 7f8d16000640 1 -- 172.21.15.90:0/103966010 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d08035020 con 0x7f8d180cc150 2024-03-24T01:32:54.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.409+0000 7f8d16000640 1 -- 172.21.15.90:0/103966010 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d08031890 con 0x7f8d180cc150 2024-03-24T01:32:54.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.409+0000 7f8d16000640 1 -- 172.21.15.90:0/103966010 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d08031c60 con 0x7f8d180cc150 2024-03-24T01:32:54.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.409+0000 7f8d1e400640 1 -- 172.21.15.90:0/103966010 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d180cc150 msgr2=0x7f8d180ccb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:54.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.409+0000 7f8d1e400640 1 --2- 172.21.15.90:0/103966010 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d180cc150 0x7f8d180ccb60 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f8d1806fc10 tx=0x7f8d0802b930 comp rx=0 tx=0).stop 2024-03-24T01:32:54.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.410+0000 7f8d1e400640 1 -- 172.21.15.90:0/103966010 shutdown_connections 2024-03-24T01:32:54.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.410+0000 7f8d1e400640 1 --2- 172.21.15.90:0/103966010 >> v2:172.21.15.120:3300/0 conn(0x7f8d1806b7d0 0x7f8d180b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:54.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.410+0000 7f8d1e400640 1 --2- 172.21.15.90:0/103966010 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d180cc150 0x7f8d180ccb60 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:54.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.410+0000 7f8d1e400640 1 -- 172.21.15.90:0/103966010 >> 172.21.15.90:0/103966010 conn(0x7f8d180b1770 msgr2=0x7f8d180b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:54.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.410+0000 7f8d1e400640 1 -- 172.21.15.90:0/103966010 shutdown_connections 2024-03-24T01:32:54.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.410+0000 7f8d1e400640 1 -- 172.21.15.90:0/103966010 wait complete. 2024-03-24T01:32:54.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.411+0000 7f8d1e400640 1 Processor -- start 2024-03-24T01:32:54.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.411+0000 7f8d1e400640 1 -- start start 2024-03-24T01:32:54.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.412+0000 7f8d1e400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d1806b7d0 0x7f8d1815e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:54.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.412+0000 7f8d1e400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d180cc150 0x7f8d181f3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:54.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.412+0000 7f8d1e400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d181603e0 0x7f8d180cf7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:54.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.412+0000 7f8d1e400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d181455b0 con 0x7f8d180cc150 2024-03-24T01:32:54.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.412+0000 7f8d1e400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d18145430 con 0x7f8d1806b7d0 2024-03-24T01:32:54.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.412+0000 7f8d1e400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d180b3c90 con 0x7f8d181603e0 2024-03-24T01:32:54.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.413+0000 7f8d16a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d180cc150 0x7f8d181f3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:54.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.413+0000 7f8d16a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d180cc150 0x7f8d181f3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34488/0 (socket says 172.21.15.90:34488) 2024-03-24T01:32:54.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.413+0000 7f8d16a00640 1 -- 172.21.15.90:0/1943274138 learned_addr learned my addr 172.21.15.90:0/1943274138 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:54.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.413+0000 7f8d17e00640 1 --2- 172.21.15.90:0/1943274138 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d181603e0 0x7f8d180cf7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:54.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.413+0000 7f8d17400640 1 --2- 172.21.15.90:0/1943274138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d1806b7d0 0x7f8d1815e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:54.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.413+0000 7f8d16a00640 1 -- 172.21.15.90:0/1943274138 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d181603e0 msgr2=0x7f8d180cf7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:54.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.413+0000 7f8d16a00640 1 --2- 172.21.15.90:0/1943274138 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d181603e0 0x7f8d180cf7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:54.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.413+0000 7f8d16a00640 1 -- 172.21.15.90:0/1943274138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d1806b7d0 msgr2=0x7f8d1815e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:54.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.413+0000 7f8d16a00640 1 --2- 172.21.15.90:0/1943274138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d1806b7d0 0x7f8d1815e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:54.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.413+0000 7f8d16a00640 1 -- 172.21.15.90:0/1943274138 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d1815eb00 con 0x7f8d180cc150 2024-03-24T01:32:54.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.413+0000 7f8d17400640 1 --2- 172.21.15.90:0/1943274138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d1806b7d0 0x7f8d1815e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:54.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.413+0000 7f8d17e00640 1 --2- 172.21.15.90:0/1943274138 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d181603e0 0x7f8d180cf7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:54.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.413+0000 7f8d16a00640 1 --2- 172.21.15.90:0/1943274138 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d180cc150 0x7f8d181f3970 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f8d0400b3c0 tx=0x7f8d0400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:54.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.414+0000 7f8d0f400640 1 -- 172.21.15.90:0/1943274138 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d04011830 con 0x7f8d180cc150 2024-03-24T01:32:54.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.414+0000 7f8d0f400640 1 -- 172.21.15.90:0/1943274138 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d04016070 con 0x7f8d180cc150 2024-03-24T01:32:54.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.414+0000 7f8d0f400640 1 -- 172.21.15.90:0/1943274138 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d04011c50 con 0x7f8d180cc150 2024-03-24T01:32:54.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.414+0000 7f8d1e400640 1 -- 172.21.15.90:0/1943274138 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d181f3eb0 con 0x7f8d180cc150 2024-03-24T01:32:54.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.414+0000 7f8d1e400640 1 -- 172.21.15.90:0/1943274138 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d181f8490 con 0x7f8d180cc150 2024-03-24T01:32:54.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.415+0000 7f8d1e400640 1 -- 172.21.15.90:0/1943274138 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d18068060 con 0x7f8d180cc150 2024-03-24T01:32:54.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.416+0000 7f8d0f400640 1 -- 172.21.15.90:0/1943274138 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8d0401b460 con 0x7f8d180cc150 2024-03-24T01:32:54.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.417+0000 7f8d0f400640 1 --2- 172.21.15.90:0/1943274138 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8cf806bbd0 0x7f8cf806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:54.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.417+0000 7f8d0f400640 1 -- 172.21.15.90:0/1943274138 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6246+0+0 (secure 0 0 0) 0x7f8d0408a780 con 0x7f8d180cc150 2024-03-24T01:32:54.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.417+0000 7f8d17400640 1 --2- 172.21.15.90:0/1943274138 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8cf806bbd0 0x7f8cf806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:54.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.418+0000 7f8d17400640 1 --2- 172.21.15.90:0/1943274138 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8cf806bbd0 0x7f8cf806e090 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f8d08036000 tx=0x7f8d0803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:54.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.422+0000 7f8d0f400640 1 -- 172.21.15.90:0/1943274138 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f8d04020020 con 0x7f8d180cc150 2024-03-24T01:32:54.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.616+0000 7f8d1e400640 1 -- 172.21.15.90:0/1943274138 --> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8d180cce80 con 0x7f8cf806bbd0 2024-03-24T01:32:54.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.643+0000 7f8d0f400640 1 -- 172.21.15.90:0/1943274138 <== mgr.4104 v2:172.21.15.120:6832/860392634 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252020 (secure 0 0 0) 0x7f8d180cce80 con 0x7f8cf806bbd0 2024-03-24T01:32:54.647 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:32:54.648 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-24T01:32:54.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.649+0000 7f8d1e400640 1 -- 172.21.15.90:0/1943274138 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8cf806bbd0 msgr2=0x7f8cf806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:54.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.649+0000 7f8d1e400640 1 --2- 172.21.15.90:0/1943274138 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8cf806bbd0 0x7f8cf806e090 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f8d08036000 tx=0x7f8d0803c040 comp rx=0 tx=0).stop 2024-03-24T01:32:54.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.649+0000 7f8d1e400640 1 -- 172.21.15.90:0/1943274138 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d180cc150 msgr2=0x7f8d181f3970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:54.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.649+0000 7f8d1e400640 1 --2- 172.21.15.90:0/1943274138 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d180cc150 0x7f8d181f3970 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f8d0400b3c0 tx=0x7f8d0400b880 comp rx=0 tx=0).stop 2024-03-24T01:32:54.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.649+0000 7f8d1e400640 1 -- 172.21.15.90:0/1943274138 shutdown_connections 2024-03-24T01:32:54.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.649+0000 7f8d1e400640 1 --2- 172.21.15.90:0/1943274138 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8cf806bbd0 0x7f8cf806e090 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:54.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.649+0000 7f8d1e400640 1 --2- 172.21.15.90:0/1943274138 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d180cc150 0x7f8d181f3970 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:54.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.649+0000 7f8d1e400640 1 --2- 172.21.15.90:0/1943274138 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8d181603e0 0x7f8d180cf7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:54.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.649+0000 7f8d1e400640 1 --2- 172.21.15.90:0/1943274138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d1806b7d0 0x7f8d1815e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:54.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.649+0000 7f8d1e400640 1 -- 172.21.15.90:0/1943274138 >> 172.21.15.90:0/1943274138 conn(0x7f8d180b1770 msgr2=0x7f8d18143600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:54.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.649+0000 7f8d1e400640 1 -- 172.21.15.90:0/1943274138 shutdown_connections 2024-03-24T01:32:54.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.649+0000 7f8d1e400640 1 -- 172.21.15.90:0/1943274138 wait complete. 2024-03-24T01:32:54.670 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":328,"stamp":"2024-03-24T01:32:53.822861+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":257048,"kb_used_data":7560,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717672,"statfs":{"total":773094113280,"available":772830896128,"internally_reserved":0,"allocated":7741440,"data_stored":4162044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12747,"internal_metadata":255315509},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.147868"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.713955+0000","last_change":"2024-03-24T01:32:44.751583+0000","last_active":"2024-03-24T01:32:45.713955+0000","last_peered":"2024-03-24T01:32:45.713955+0000","last_clean":"2024-03-24T01:32:45.713955+0000","last_became_active":"2024-03-24T01:32:44.751290+0000","last_became_peered":"2024-03-24T01:32:44.751290+0000","last_unstale":"2024-03-24T01:32:45.713955+0000","last_undegraded":"2024-03-24T01:32:45.713955+0000","last_fullsized":"2024-03-24T01:32:45.713955+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:14:56.287011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.713664+0000","last_change":"2024-03-24T01:32:44.751982+0000","last_active":"2024-03-24T01:32:45.713664+0000","last_peered":"2024-03-24T01:32:45.713664+0000","last_clean":"2024-03-24T01:32:45.713664+0000","last_became_active":"2024-03-24T01:32:44.751686+0000","last_became_peered":"2024-03-24T01:32:44.751686+0000","last_unstale":"2024-03-24T01:32:45.713664+0000","last_undegraded":"2024-03-24T01:32:45.713664+0000","last_fullsized":"2024-03-24T01:32:45.713664+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:03.806877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117942+0000","last_change":"2024-03-24T01:32:44.761410+0000","last_active":"2024-03-24T01:32:46.117942+0000","last_peered":"2024-03-24T01:32:46.117942+0000","last_clean":"2024-03-24T01:32:46.117942+0000","last_became_active":"2024-03-24T01:32:44.760599+0000","last_became_peered":"2024-03-24T01:32:44.760599+0000","last_unstale":"2024-03-24T01:32:46.117942+0000","last_undegraded":"2024-03-24T01:32:46.117942+0000","last_fullsized":"2024-03-24T01:32:46.117942+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:33:13.594934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.720760+0000","last_change":"2024-03-24T01:32:44.735449+0000","last_active":"2024-03-24T01:32:45.720760+0000","last_peered":"2024-03-24T01:32:45.720760+0000","last_clean":"2024-03-24T01:32:45.720760+0000","last_became_active":"2024-03-24T01:32:44.734765+0000","last_became_peered":"2024-03-24T01:32:44.734765+0000","last_unstale":"2024-03-24T01:32:45.720760+0000","last_undegraded":"2024-03-24T01:32:45.720760+0000","last_fullsized":"2024-03-24T01:32:45.720760+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:12.824577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.720672+0000","last_change":"2024-03-24T01:32:44.731161+0000","last_active":"2024-03-24T01:32:45.720672+0000","last_peered":"2024-03-24T01:32:45.720672+0000","last_clean":"2024-03-24T01:32:45.720672+0000","last_became_active":"2024-03-24T01:32:44.730863+0000","last_became_peered":"2024-03-24T01:32:44.730863+0000","last_unstale":"2024-03-24T01:32:45.720672+0000","last_undegraded":"2024-03-24T01:32:45.720672+0000","last_fullsized":"2024-03-24T01:32:45.720672+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:13:48.874647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117287+0000","last_change":"2024-03-24T01:32:44.741905+0000","last_active":"2024-03-24T01:32:46.117287+0000","last_peered":"2024-03-24T01:32:46.117287+0000","last_clean":"2024-03-24T01:32:46.117287+0000","last_became_active":"2024-03-24T01:32:44.741599+0000","last_became_peered":"2024-03-24T01:32:44.741599+0000","last_unstale":"2024-03-24T01:32:46.117287+0000","last_undegraded":"2024-03-24T01:32:46.117287+0000","last_fullsized":"2024-03-24T01:32:46.117287+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:19:45.922169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.773953+0000","last_change":"2024-03-24T01:32:44.755846+0000","last_active":"2024-03-24T01:32:45.773953+0000","last_peered":"2024-03-24T01:32:45.773953+0000","last_clean":"2024-03-24T01:32:45.773953+0000","last_became_active":"2024-03-24T01:32:44.755533+0000","last_became_peered":"2024-03-24T01:32:44.755533+0000","last_unstale":"2024-03-24T01:32:45.773953+0000","last_undegraded":"2024-03-24T01:32:45.773953+0000","last_fullsized":"2024-03-24T01:32:45.773953+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:00:32.057217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.767176+0000","last_change":"2024-03-24T01:32:44.754059+0000","last_active":"2024-03-24T01:32:45.767176+0000","last_peered":"2024-03-24T01:32:45.767176+0000","last_clean":"2024-03-24T01:32:45.767176+0000","last_became_active":"2024-03-24T01:32:44.753503+0000","last_became_peered":"2024-03-24T01:32:44.753503+0000","last_unstale":"2024-03-24T01:32:45.767176+0000","last_undegraded":"2024-03-24T01:32:45.767176+0000","last_fullsized":"2024-03-24T01:32:45.767176+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:39:23.561488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.118564+0000","last_change":"2024-03-24T01:32:44.761804+0000","last_active":"2024-03-24T01:32:46.118564+0000","last_peered":"2024-03-24T01:32:46.118564+0000","last_clean":"2024-03-24T01:32:46.118564+0000","last_became_active":"2024-03-24T01:32:44.761263+0000","last_became_peered":"2024-03-24T01:32:44.761263+0000","last_unstale":"2024-03-24T01:32:46.118564+0000","last_undegraded":"2024-03-24T01:32:46.118564+0000","last_fullsized":"2024-03-24T01:32:46.118564+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:31:50.925888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.118044+0000","last_change":"2024-03-24T01:32:44.755965+0000","last_active":"2024-03-24T01:32:46.118044+0000","last_peered":"2024-03-24T01:32:46.118044+0000","last_clean":"2024-03-24T01:32:46.118044+0000","last_became_active":"2024-03-24T01:32:44.755708+0000","last_became_peered":"2024-03-24T01:32:44.755708+0000","last_unstale":"2024-03-24T01:32:46.118044+0000","last_undegraded":"2024-03-24T01:32:46.118044+0000","last_fullsized":"2024-03-24T01:32:46.118044+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:33:21.633652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.719972+0000","last_change":"2024-03-24T01:32:44.750963+0000","last_active":"2024-03-24T01:32:45.719972+0000","last_peered":"2024-03-24T01:32:45.719972+0000","last_clean":"2024-03-24T01:32:45.719972+0000","last_became_active":"2024-03-24T01:32:44.750351+0000","last_became_peered":"2024-03-24T01:32:44.750351+0000","last_unstale":"2024-03-24T01:32:45.719972+0000","last_undegraded":"2024-03-24T01:32:45.719972+0000","last_fullsized":"2024-03-24T01:32:45.719972+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:13:48.874647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.767953+0000","last_change":"2024-03-24T01:32:44.752161+0000","last_active":"2024-03-24T01:32:45.767953+0000","last_peered":"2024-03-24T01:32:45.767953+0000","last_clean":"2024-03-24T01:32:45.767953+0000","last_became_active":"2024-03-24T01:32:44.751842+0000","last_became_peered":"2024-03-24T01:32:44.751842+0000","last_unstale":"2024-03-24T01:32:45.767953+0000","last_undegraded":"2024-03-24T01:32:45.767953+0000","last_fullsized":"2024-03-24T01:32:45.767953+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:48:18.811395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.769621+0000","last_change":"2024-03-24T01:32:44.746912+0000","last_active":"2024-03-24T01:32:45.769621+0000","last_peered":"2024-03-24T01:32:45.769621+0000","last_clean":"2024-03-24T01:32:45.769621+0000","last_became_active":"2024-03-24T01:32:44.746271+0000","last_became_peered":"2024-03-24T01:32:44.746271+0000","last_unstale":"2024-03-24T01:32:45.769621+0000","last_undegraded":"2024-03-24T01:32:45.769621+0000","last_fullsized":"2024-03-24T01:32:45.769621+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:33:21.633652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117510+0000","last_change":"2024-03-24T01:32:44.758588+0000","last_active":"2024-03-24T01:32:46.117510+0000","last_peered":"2024-03-24T01:32:46.117510+0000","last_clean":"2024-03-24T01:32:46.117510+0000","last_became_active":"2024-03-24T01:32:44.757289+0000","last_became_peered":"2024-03-24T01:32:44.757289+0000","last_unstale":"2024-03-24T01:32:46.117510+0000","last_undegraded":"2024-03-24T01:32:46.117510+0000","last_fullsized":"2024-03-24T01:32:46.117510+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:41:56.230713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117149+0000","last_change":"2024-03-24T01:32:44.753415+0000","last_active":"2024-03-24T01:32:46.117149+0000","last_peered":"2024-03-24T01:32:46.117149+0000","last_clean":"2024-03-24T01:32:46.117149+0000","last_became_active":"2024-03-24T01:32:44.752869+0000","last_became_peered":"2024-03-24T01:32:44.752869+0000","last_unstale":"2024-03-24T01:32:46.117149+0000","last_undegraded":"2024-03-24T01:32:46.117149+0000","last_fullsized":"2024-03-24T01:32:46.117149+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:48:18.811395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.773869+0000","last_change":"2024-03-24T01:32:44.742945+0000","last_active":"2024-03-24T01:32:45.773869+0000","last_peered":"2024-03-24T01:32:45.773869+0000","last_clean":"2024-03-24T01:32:45.773869+0000","last_became_active":"2024-03-24T01:32:44.742329+0000","last_became_peered":"2024-03-24T01:32:44.742329+0000","last_unstale":"2024-03-24T01:32:45.773869+0000","last_undegraded":"2024-03-24T01:32:45.773869+0000","last_fullsized":"2024-03-24T01:32:45.773869+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:40:03.643169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.714047+0000","last_change":"2024-03-24T01:32:44.745728+0000","last_active":"2024-03-24T01:32:45.714047+0000","last_peered":"2024-03-24T01:32:45.714047+0000","last_clean":"2024-03-24T01:32:45.714047+0000","last_became_active":"2024-03-24T01:32:44.745412+0000","last_became_peered":"2024-03-24T01:32:44.745412+0000","last_unstale":"2024-03-24T01:32:45.714047+0000","last_undegraded":"2024-03-24T01:32:45.714047+0000","last_fullsized":"2024-03-24T01:32:45.714047+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:36:57.078745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.720116+0000","last_change":"2024-03-24T01:32:44.735333+0000","last_active":"2024-03-24T01:32:45.720116+0000","last_peered":"2024-03-24T01:32:45.720116+0000","last_clean":"2024-03-24T01:32:45.720116+0000","last_became_active":"2024-03-24T01:32:44.734813+0000","last_became_peered":"2024-03-24T01:32:44.734813+0000","last_unstale":"2024-03-24T01:32:45.720116+0000","last_undegraded":"2024-03-24T01:32:45.720116+0000","last_fullsized":"2024-03-24T01:32:45.720116+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:15:20.815746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.721498+0000","last_change":"2024-03-24T01:32:44.750885+0000","last_active":"2024-03-24T01:32:45.721498+0000","last_peered":"2024-03-24T01:32:45.721498+0000","last_clean":"2024-03-24T01:32:45.721498+0000","last_became_active":"2024-03-24T01:32:44.750355+0000","last_became_peered":"2024-03-24T01:32:44.750355+0000","last_unstale":"2024-03-24T01:32:45.721498+0000","last_undegraded":"2024-03-24T01:32:45.721498+0000","last_fullsized":"2024-03-24T01:32:45.721498+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:35.662274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.118346+0000","last_change":"2024-03-24T01:32:44.734704+0000","last_active":"2024-03-24T01:32:46.118346+0000","last_peered":"2024-03-24T01:32:46.118346+0000","last_clean":"2024-03-24T01:32:46.118346+0000","last_became_active":"2024-03-24T01:32:44.733687+0000","last_became_peered":"2024-03-24T01:32:44.733687+0000","last_unstale":"2024-03-24T01:32:46.118346+0000","last_undegraded":"2024-03-24T01:32:46.118346+0000","last_fullsized":"2024-03-24T01:32:46.118346+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:50:50.757791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.768693+0000","last_change":"2024-03-24T01:32:44.755202+0000","last_active":"2024-03-24T01:32:45.768693+0000","last_peered":"2024-03-24T01:32:45.768693+0000","last_clean":"2024-03-24T01:32:45.768693+0000","last_became_active":"2024-03-24T01:32:44.754775+0000","last_became_peered":"2024-03-24T01:32:44.754775+0000","last_unstale":"2024-03-24T01:32:45.768693+0000","last_undegraded":"2024-03-24T01:32:45.768693+0000","last_fullsized":"2024-03-24T01:32:45.768693+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:33:13.594934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.118707+0000","last_change":"2024-03-24T01:32:44.741995+0000","last_active":"2024-03-24T01:32:46.118707+0000","last_peered":"2024-03-24T01:32:46.118707+0000","last_clean":"2024-03-24T01:32:46.118707+0000","last_became_active":"2024-03-24T01:32:44.741671+0000","last_became_peered":"2024-03-24T01:32:44.741671+0000","last_unstale":"2024-03-24T01:32:46.118707+0000","last_undegraded":"2024-03-24T01:32:46.118707+0000","last_fullsized":"2024-03-24T01:32:46.118707+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:52:15.663148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117020+0000","last_change":"2024-03-24T01:32:44.740789+0000","last_active":"2024-03-24T01:32:46.117020+0000","last_peered":"2024-03-24T01:32:46.117020+0000","last_clean":"2024-03-24T01:32:46.117020+0000","last_became_active":"2024-03-24T01:32:44.740465+0000","last_became_peered":"2024-03-24T01:32:44.740465+0000","last_unstale":"2024-03-24T01:32:46.117020+0000","last_undegraded":"2024-03-24T01:32:46.117020+0000","last_fullsized":"2024-03-24T01:32:46.117020+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:02:39.321334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.116993+0000","last_change":"2024-03-24T01:32:44.758180+0000","last_active":"2024-03-24T01:32:46.116993+0000","last_peered":"2024-03-24T01:32:46.116993+0000","last_clean":"2024-03-24T01:32:46.116993+0000","last_became_active":"2024-03-24T01:32:44.756763+0000","last_became_peered":"2024-03-24T01:32:44.756763+0000","last_unstale":"2024-03-24T01:32:46.116993+0000","last_undegraded":"2024-03-24T01:32:46.116993+0000","last_fullsized":"2024-03-24T01:32:46.116993+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:59:12.918357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.768651+0000","last_change":"2024-03-24T01:32:44.753078+0000","last_active":"2024-03-24T01:32:45.768651+0000","last_peered":"2024-03-24T01:32:45.768651+0000","last_clean":"2024-03-24T01:32:45.768651+0000","last_became_active":"2024-03-24T01:32:44.750365+0000","last_became_peered":"2024-03-24T01:32:44.750365+0000","last_unstale":"2024-03-24T01:32:45.768651+0000","last_undegraded":"2024-03-24T01:32:45.768651+0000","last_fullsized":"2024-03-24T01:32:45.768651+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:56:30.509999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.767804+0000","last_change":"2024-03-24T01:32:44.747414+0000","last_active":"2024-03-24T01:32:45.767804+0000","last_peered":"2024-03-24T01:32:45.767804+0000","last_clean":"2024-03-24T01:32:45.767804+0000","last_became_active":"2024-03-24T01:32:44.747138+0000","last_became_peered":"2024-03-24T01:32:44.747138+0000","last_unstale":"2024-03-24T01:32:45.767804+0000","last_undegraded":"2024-03-24T01:32:45.767804+0000","last_fullsized":"2024-03-24T01:32:45.767804+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:59:12.918357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.719958+0000","last_change":"2024-03-24T01:32:44.746689+0000","last_active":"2024-03-24T01:32:45.719958+0000","last_peered":"2024-03-24T01:32:45.719958+0000","last_clean":"2024-03-24T01:32:45.719958+0000","last_became_active":"2024-03-24T01:32:44.746359+0000","last_became_peered":"2024-03-24T01:32:44.746359+0000","last_unstale":"2024-03-24T01:32:45.719958+0000","last_undegraded":"2024-03-24T01:32:45.719958+0000","last_fullsized":"2024-03-24T01:32:45.719958+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:36.843294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.713789+0000","last_change":"2024-03-24T01:32:44.749783+0000","last_active":"2024-03-24T01:32:45.713789+0000","last_peered":"2024-03-24T01:32:45.713789+0000","last_clean":"2024-03-24T01:32:45.713789+0000","last_became_active":"2024-03-24T01:32:44.749266+0000","last_became_peered":"2024-03-24T01:32:44.749266+0000","last_unstale":"2024-03-24T01:32:45.713789+0000","last_undegraded":"2024-03-24T01:32:45.713789+0000","last_fullsized":"2024-03-24T01:32:45.713789+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:39:10.149776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117669+0000","last_change":"2024-03-24T01:32:44.751695+0000","last_active":"2024-03-24T01:32:46.117669+0000","last_peered":"2024-03-24T01:32:46.117669+0000","last_clean":"2024-03-24T01:32:46.117669+0000","last_became_active":"2024-03-24T01:32:44.751409+0000","last_became_peered":"2024-03-24T01:32:44.751409+0000","last_unstale":"2024-03-24T01:32:46.117669+0000","last_undegraded":"2024-03-24T01:32:46.117669+0000","last_fullsized":"2024-03-24T01:32:46.117669+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:14:45.725936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.767559+0000","last_change":"2024-03-24T01:32:44.752696+0000","last_active":"2024-03-24T01:32:45.767559+0000","last_peered":"2024-03-24T01:32:45.767559+0000","last_clean":"2024-03-24T01:32:45.767559+0000","last_became_active":"2024-03-24T01:32:44.752378+0000","last_became_peered":"2024-03-24T01:32:44.752378+0000","last_unstale":"2024-03-24T01:32:45.767559+0000","last_undegraded":"2024-03-24T01:32:45.767559+0000","last_fullsized":"2024-03-24T01:32:45.767559+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:36:57.078745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.721483+0000","last_change":"2024-03-24T01:32:44.742966+0000","last_active":"2024-03-24T01:32:45.721483+0000","last_peered":"2024-03-24T01:32:45.721483+0000","last_clean":"2024-03-24T01:32:45.721483+0000","last_became_active":"2024-03-24T01:32:44.742681+0000","last_became_peered":"2024-03-24T01:32:44.742681+0000","last_unstale":"2024-03-24T01:32:45.721483+0000","last_undegraded":"2024-03-24T01:32:45.721483+0000","last_fullsized":"2024-03-24T01:32:45.721483+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:39:10.149776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.116803+0000","last_change":"2024-03-24T01:32:44.744799+0000","last_active":"2024-03-24T01:32:46.116803+0000","last_peered":"2024-03-24T01:32:46.116803+0000","last_clean":"2024-03-24T01:32:46.116803+0000","last_became_active":"2024-03-24T01:32:44.743828+0000","last_became_peered":"2024-03-24T01:32:44.743828+0000","last_unstale":"2024-03-24T01:32:46.116803+0000","last_undegraded":"2024-03-24T01:32:46.116803+0000","last_fullsized":"2024-03-24T01:32:46.116803+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:31:50.925888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.721397+0000","last_change":"2024-03-24T01:32:44.748610+0000","last_active":"2024-03-24T01:32:45.721397+0000","last_peered":"2024-03-24T01:32:45.721397+0000","last_clean":"2024-03-24T01:32:45.721397+0000","last_became_active":"2024-03-24T01:32:44.748297+0000","last_became_peered":"2024-03-24T01:32:44.748297+0000","last_unstale":"2024-03-24T01:32:45.721397+0000","last_undegraded":"2024-03-24T01:32:45.721397+0000","last_fullsized":"2024-03-24T01:32:45.721397+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:19.032155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"66'4","reported_seq":18,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.714111+0000","last_change":"2024-03-24T01:32:43.729112+0000","last_active":"2024-03-24T01:32:45.714111+0000","last_peered":"2024-03-24T01:32:45.714111+0000","last_clean":"2024-03-24T01:32:45.714111+0000","last_became_active":"2024-03-24T01:32:43.728882+0000","last_became_peered":"2024-03-24T01:32:43.728882+0000","last_unstale":"2024-03-24T01:32:45.714111+0000","last_undegraded":"2024-03-24T01:32:45.714111+0000","last_fullsized":"2024-03-24T01:32:45.714111+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:29.157467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.720464+0000","last_change":"2024-03-24T01:32:44.733668+0000","last_active":"2024-03-24T01:32:45.720464+0000","last_peered":"2024-03-24T01:32:45.720464+0000","last_clean":"2024-03-24T01:32:45.720464+0000","last_became_active":"2024-03-24T01:32:44.733182+0000","last_became_peered":"2024-03-24T01:32:44.733182+0000","last_unstale":"2024-03-24T01:32:45.720464+0000","last_undegraded":"2024-03-24T01:32:45.720464+0000","last_fullsized":"2024-03-24T01:32:45.720464+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:19.032155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"66'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.118869+0000","last_change":"2024-03-24T01:32:43.723642+0000","last_active":"2024-03-24T01:32:46.118869+0000","last_peered":"2024-03-24T01:32:46.118869+0000","last_clean":"2024-03-24T01:32:46.118869+0000","last_became_active":"2024-03-24T01:32:43.723374+0000","last_became_peered":"2024-03-24T01:32:43.723374+0000","last_unstale":"2024-03-24T01:32:46.118869+0000","last_undegraded":"2024-03-24T01:32:46.118869+0000","last_fullsized":"2024-03-24T01:32:46.118869+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:16:58.498006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.118687+0000","last_change":"2024-03-24T01:32:44.743311+0000","last_active":"2024-03-24T01:32:46.118687+0000","last_peered":"2024-03-24T01:32:46.118687+0000","last_clean":"2024-03-24T01:32:46.118687+0000","last_became_active":"2024-03-24T01:32:44.743027+0000","last_became_peered":"2024-03-24T01:32:44.743027+0000","last_unstale":"2024-03-24T01:32:46.118687+0000","last_undegraded":"2024-03-24T01:32:46.118687+0000","last_fullsized":"2024-03-24T01:32:46.118687+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:03:37.421881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117972+0000","last_change":"2024-03-24T01:32:43.722317+0000","last_active":"2024-03-24T01:32:46.117972+0000","last_peered":"2024-03-24T01:32:46.117972+0000","last_clean":"2024-03-24T01:32:46.117972+0000","last_became_active":"2024-03-24T01:32:43.721576+0000","last_became_peered":"2024-03-24T01:32:43.721576+0000","last_unstale":"2024-03-24T01:32:46.117972+0000","last_undegraded":"2024-03-24T01:32:46.117972+0000","last_fullsized":"2024-03-24T01:32:46.117972+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:36:56.063762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.118276+0000","last_change":"2024-03-24T01:32:44.755395+0000","last_active":"2024-03-24T01:32:46.118276+0000","last_peered":"2024-03-24T01:32:46.118276+0000","last_clean":"2024-03-24T01:32:46.118276+0000","last_became_active":"2024-03-24T01:32:44.755127+0000","last_became_peered":"2024-03-24T01:32:44.755127+0000","last_unstale":"2024-03-24T01:32:46.118276+0000","last_undegraded":"2024-03-24T01:32:46.118276+0000","last_fullsized":"2024-03-24T01:32:46.118276+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:03:13.359822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117409+0000","last_change":"2024-03-24T01:32:43.713108+0000","last_active":"2024-03-24T01:32:46.117409+0000","last_peered":"2024-03-24T01:32:46.117409+0000","last_clean":"2024-03-24T01:32:46.117409+0000","last_became_active":"2024-03-24T01:32:43.712800+0000","last_became_peered":"2024-03-24T01:32:43.712800+0000","last_unstale":"2024-03-24T01:32:46.117409+0000","last_undegraded":"2024-03-24T01:32:46.117409+0000","last_fullsized":"2024-03-24T01:32:46.117409+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:59:11.903374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.774108+0000","last_change":"2024-03-24T01:32:44.747444+0000","last_active":"2024-03-24T01:32:45.774108+0000","last_peered":"2024-03-24T01:32:45.774108+0000","last_clean":"2024-03-24T01:32:45.774108+0000","last_became_active":"2024-03-24T01:32:44.747117+0000","last_became_peered":"2024-03-24T01:32:44.747117+0000","last_unstale":"2024-03-24T01:32:45.774108+0000","last_undegraded":"2024-03-24T01:32:45.774108+0000","last_fullsized":"2024-03-24T01:32:45.774108+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:24:34.023784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"66'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117538+0000","last_change":"2024-03-24T01:32:43.718182+0000","last_active":"2024-03-24T01:32:46.117538+0000","last_peered":"2024-03-24T01:32:46.117538+0000","last_clean":"2024-03-24T01:32:46.117538+0000","last_became_active":"2024-03-24T01:32:43.717674+0000","last_became_peered":"2024-03-24T01:32:43.717674+0000","last_unstale":"2024-03-24T01:32:46.117538+0000","last_undegraded":"2024-03-24T01:32:46.117538+0000","last_fullsized":"2024-03-24T01:32:46.117538+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:31:49.910904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.713179+0000","last_change":"2024-03-24T01:32:44.746900+0000","last_active":"2024-03-24T01:32:45.713179+0000","last_peered":"2024-03-24T01:32:45.713179+0000","last_clean":"2024-03-24T01:32:45.713179+0000","last_became_active":"2024-03-24T01:32:44.746619+0000","last_became_peered":"2024-03-24T01:32:44.746619+0000","last_unstale":"2024-03-24T01:32:45.713179+0000","last_undegraded":"2024-03-24T01:32:45.713179+0000","last_fullsized":"2024-03-24T01:32:45.713179+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:12.824577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"66'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117671+0000","last_change":"2024-03-24T01:32:43.717225+0000","last_active":"2024-03-24T01:32:46.117671+0000","last_peered":"2024-03-24T01:32:46.117671+0000","last_clean":"2024-03-24T01:32:46.117671+0000","last_became_active":"2024-03-24T01:32:43.716994+0000","last_became_peered":"2024-03-24T01:32:43.716994+0000","last_unstale":"2024-03-24T01:32:46.117671+0000","last_undegraded":"2024-03-24T01:32:46.117671+0000","last_fullsized":"2024-03-24T01:32:46.117671+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:14:44.710953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.721632+0000","last_change":"2024-03-24T01:32:44.733286+0000","last_active":"2024-03-24T01:32:45.721632+0000","last_peered":"2024-03-24T01:32:45.721632+0000","last_clean":"2024-03-24T01:32:45.721632+0000","last_became_active":"2024-03-24T01:32:44.732956+0000","last_became_peered":"2024-03-24T01:32:44.732956+0000","last_unstale":"2024-03-24T01:32:45.721632+0000","last_undegraded":"2024-03-24T01:32:45.721632+0000","last_fullsized":"2024-03-24T01:32:45.721632+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:12.824577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.116267+0000","last_change":"2024-03-24T01:32:43.719216+0000","last_active":"2024-03-24T01:32:46.116267+0000","last_peered":"2024-03-24T01:32:46.116267+0000","last_clean":"2024-03-24T01:32:46.116267+0000","last_became_active":"2024-03-24T01:32:43.718879+0000","last_became_peered":"2024-03-24T01:32:43.718879+0000","last_unstale":"2024-03-24T01:32:46.116267+0000","last_undegraded":"2024-03-24T01:32:46.116267+0000","last_fullsized":"2024-03-24T01:32:46.116267+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:03:12.344839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117429+0000","last_change":"2024-03-24T01:32:44.740874+0000","last_active":"2024-03-24T01:32:46.117429+0000","last_peered":"2024-03-24T01:32:46.117429+0000","last_clean":"2024-03-24T01:32:46.117429+0000","last_became_active":"2024-03-24T01:32:44.740563+0000","last_became_peered":"2024-03-24T01:32:44.740563+0000","last_unstale":"2024-03-24T01:32:46.117429+0000","last_undegraded":"2024-03-24T01:32:46.117429+0000","last_fullsized":"2024-03-24T01:32:46.117429+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:11:10.464938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.719133+0000","last_change":"2024-03-24T01:32:43.720754+0000","last_active":"2024-03-24T01:32:45.719133+0000","last_peered":"2024-03-24T01:32:45.719133+0000","last_clean":"2024-03-24T01:32:45.719133+0000","last_became_active":"2024-03-24T01:32:43.720465+0000","last_became_peered":"2024-03-24T01:32:43.720465+0000","last_unstale":"2024-03-24T01:32:45.719133+0000","last_undegraded":"2024-03-24T01:32:45.719133+0000","last_fullsized":"2024-03-24T01:32:45.719133+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:40:54.360077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.713983+0000","last_change":"2024-03-24T01:32:44.750883+0000","last_active":"2024-03-24T01:32:45.713983+0000","last_peered":"2024-03-24T01:32:45.713983+0000","last_clean":"2024-03-24T01:32:45.713983+0000","last_became_active":"2024-03-24T01:32:44.750572+0000","last_became_peered":"2024-03-24T01:32:44.750572+0000","last_unstale":"2024-03-24T01:32:45.713983+0000","last_undegraded":"2024-03-24T01:32:45.713983+0000","last_fullsized":"2024-03-24T01:32:45.713983+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:39:23.561488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.118621+0000","last_change":"2024-03-24T01:32:43.729179+0000","last_active":"2024-03-24T01:32:46.118621+0000","last_peered":"2024-03-24T01:32:46.118621+0000","last_clean":"2024-03-24T01:32:46.118621+0000","last_became_active":"2024-03-24T01:32:43.728892+0000","last_became_peered":"2024-03-24T01:32:43.728892+0000","last_unstale":"2024-03-24T01:32:46.118621+0000","last_undegraded":"2024-03-24T01:32:46.118621+0000","last_fullsized":"2024-03-24T01:32:46.118621+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:18.017172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117966+0000","last_change":"2024-03-24T01:32:44.747545+0000","last_active":"2024-03-24T01:32:46.117966+0000","last_peered":"2024-03-24T01:32:46.117966+0000","last_clean":"2024-03-24T01:32:46.117966+0000","last_became_active":"2024-03-24T01:32:44.747227+0000","last_became_peered":"2024-03-24T01:32:44.747227+0000","last_unstale":"2024-03-24T01:32:46.117966+0000","last_undegraded":"2024-03-24T01:32:46.117966+0000","last_fullsized":"2024-03-24T01:32:46.117966+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:55.474861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.721185+0000","last_change":"2024-03-24T01:32:43.715851+0000","last_active":"2024-03-24T01:32:45.721185+0000","last_peered":"2024-03-24T01:32:45.721185+0000","last_clean":"2024-03-24T01:32:45.721185+0000","last_became_active":"2024-03-24T01:32:43.715474+0000","last_became_peered":"2024-03-24T01:32:43.715474+0000","last_unstale":"2024-03-24T01:32:45.721185+0000","last_undegraded":"2024-03-24T01:32:45.721185+0000","last_fullsized":"2024-03-24T01:32:45.721185+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:24:00.025674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.721208+0000","last_change":"2024-03-24T01:32:44.748134+0000","last_active":"2024-03-24T01:32:45.721208+0000","last_peered":"2024-03-24T01:32:45.721208+0000","last_clean":"2024-03-24T01:32:45.721208+0000","last_became_active":"2024-03-24T01:32:44.747771+0000","last_became_peered":"2024-03-24T01:32:44.747771+0000","last_unstale":"2024-03-24T01:32:45.721208+0000","last_undegraded":"2024-03-24T01:32:45.721208+0000","last_fullsized":"2024-03-24T01:32:45.721208+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:36.843294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"66'3","reported_seq":17,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.118252+0000","last_change":"2024-03-24T01:32:43.725497+0000","last_active":"2024-03-24T01:32:46.118252+0000","last_peered":"2024-03-24T01:32:46.118252+0000","last_clean":"2024-03-24T01:32:46.118252+0000","last_became_active":"2024-03-24T01:32:43.724751+0000","last_became_peered":"2024-03-24T01:32:43.724751+0000","last_unstale":"2024-03-24T01:32:46.118252+0000","last_undegraded":"2024-03-24T01:32:46.118252+0000","last_fullsized":"2024-03-24T01:32:46.118252+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:35.828311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.720698+0000","last_change":"2024-03-24T01:32:44.737244+0000","last_active":"2024-03-24T01:32:45.720698+0000","last_peered":"2024-03-24T01:32:45.720698+0000","last_clean":"2024-03-24T01:32:45.720698+0000","last_became_active":"2024-03-24T01:32:44.735810+0000","last_became_peered":"2024-03-24T01:32:44.735810+0000","last_unstale":"2024-03-24T01:32:45.720698+0000","last_undegraded":"2024-03-24T01:32:45.720698+0000","last_fullsized":"2024-03-24T01:32:45.720698+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:35.662274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"66'6","reported_seq":20,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.714077+0000","last_change":"2024-03-24T01:32:43.716683+0000","last_active":"2024-03-24T01:32:45.714077+0000","last_peered":"2024-03-24T01:32:45.714077+0000","last_clean":"2024-03-24T01:32:45.714077+0000","last_became_active":"2024-03-24T01:32:43.716407+0000","last_became_peered":"2024-03-24T01:32:43.716407+0000","last_unstale":"2024-03-24T01:32:45.714077+0000","last_undegraded":"2024-03-24T01:32:45.714077+0000","last_fullsized":"2024-03-24T01:32:45.714077+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:43:54.681008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.718926+0000","last_change":"2024-03-24T01:32:44.741777+0000","last_active":"2024-03-24T01:32:45.718926+0000","last_peered":"2024-03-24T01:32:45.718926+0000","last_clean":"2024-03-24T01:32:45.718926+0000","last_became_active":"2024-03-24T01:32:44.741472+0000","last_became_peered":"2024-03-24T01:32:44.741472+0000","last_unstale":"2024-03-24T01:32:45.718926+0000","last_undegraded":"2024-03-24T01:32:45.718926+0000","last_fullsized":"2024-03-24T01:32:45.718926+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:20:55.112893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117614+0000","last_change":"2024-03-24T01:32:43.725509+0000","last_active":"2024-03-24T01:32:46.117614+0000","last_peered":"2024-03-24T01:32:46.117614+0000","last_clean":"2024-03-24T01:32:46.117614+0000","last_became_active":"2024-03-24T01:32:43.725142+0000","last_became_peered":"2024-03-24T01:32:43.725142+0000","last_unstale":"2024-03-24T01:32:46.117614+0000","last_undegraded":"2024-03-24T01:32:46.117614+0000","last_fullsized":"2024-03-24T01:32:46.117614+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:20:54.257187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.767647+0000","last_change":"2024-03-24T01:32:44.750397+0000","last_active":"2024-03-24T01:32:45.767647+0000","last_peered":"2024-03-24T01:32:45.767647+0000","last_clean":"2024-03-24T01:32:45.767647+0000","last_became_active":"2024-03-24T01:32:44.750002+0000","last_became_peered":"2024-03-24T01:32:44.750002+0000","last_unstale":"2024-03-24T01:32:45.767647+0000","last_undegraded":"2024-03-24T01:32:45.767647+0000","last_fullsized":"2024-03-24T01:32:45.767647+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:16:59.512989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.767678+0000","last_change":"2024-03-24T01:32:43.720435+0000","last_active":"2024-03-24T01:32:45.767678+0000","last_peered":"2024-03-24T01:32:45.767678+0000","last_clean":"2024-03-24T01:32:45.767678+0000","last_became_active":"2024-03-24T01:32:43.720135+0000","last_became_peered":"2024-03-24T01:32:43.720135+0000","last_unstale":"2024-03-24T01:32:45.767678+0000","last_undegraded":"2024-03-24T01:32:45.767678+0000","last_fullsized":"2024-03-24T01:32:45.767678+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:11:39.983567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.722236+0000","last_change":"2024-03-24T01:32:44.751131+0000","last_active":"2024-03-24T01:32:45.722236+0000","last_peered":"2024-03-24T01:32:45.722236+0000","last_clean":"2024-03-24T01:32:45.722236+0000","last_became_active":"2024-03-24T01:32:44.750794+0000","last_became_peered":"2024-03-24T01:32:44.750794+0000","last_unstale":"2024-03-24T01:32:45.722236+0000","last_undegraded":"2024-03-24T01:32:45.722236+0000","last_fullsized":"2024-03-24T01:32:45.722236+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:14:56.287011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117416+0000","last_change":"2024-03-24T01:32:43.725306+0000","last_active":"2024-03-24T01:32:46.117416+0000","last_peered":"2024-03-24T01:32:46.117416+0000","last_clean":"2024-03-24T01:32:46.117416+0000","last_became_active":"2024-03-24T01:32:43.724101+0000","last_became_peered":"2024-03-24T01:32:43.724101+0000","last_unstale":"2024-03-24T01:32:46.117416+0000","last_undegraded":"2024-03-24T01:32:46.117416+0000","last_fullsized":"2024-03-24T01:32:46.117416+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:12:55.184668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.713378+0000","last_change":"2024-03-24T01:32:44.750404+0000","last_active":"2024-03-24T01:32:45.713378+0000","last_peered":"2024-03-24T01:32:45.713378+0000","last_clean":"2024-03-24T01:32:45.713378+0000","last_became_active":"2024-03-24T01:32:44.750099+0000","last_became_peered":"2024-03-24T01:32:44.750099+0000","last_unstale":"2024-03-24T01:32:45.713378+0000","last_undegraded":"2024-03-24T01:32:45.713378+0000","last_fullsized":"2024-03-24T01:32:45.713378+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:10:36.135152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"66'8","reported_seq":22,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.719601+0000","last_change":"2024-03-24T01:32:43.720664+0000","last_active":"2024-03-24T01:32:45.719601+0000","last_peered":"2024-03-24T01:32:45.719601+0000","last_clean":"2024-03-24T01:32:45.719601+0000","last_became_active":"2024-03-24T01:32:43.720338+0000","last_became_peered":"2024-03-24T01:32:43.720338+0000","last_unstale":"2024-03-24T01:32:45.719601+0000","last_undegraded":"2024-03-24T01:32:45.719601+0000","last_fullsized":"2024-03-24T01:32:45.719601+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:40:54.360077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117882+0000","last_change":"2024-03-24T01:32:44.762349+0000","last_active":"2024-03-24T01:32:46.117882+0000","last_peered":"2024-03-24T01:32:46.117882+0000","last_clean":"2024-03-24T01:32:46.117882+0000","last_became_active":"2024-03-24T01:32:44.761475+0000","last_became_peered":"2024-03-24T01:32:44.761475+0000","last_unstale":"2024-03-24T01:32:46.117882+0000","last_undegraded":"2024-03-24T01:32:46.117882+0000","last_fullsized":"2024-03-24T01:32:46.117882+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:56:30.509999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"66'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.768808+0000","last_change":"2024-03-24T01:32:43.722152+0000","last_active":"2024-03-24T01:32:45.768808+0000","last_peered":"2024-03-24T01:32:45.768808+0000","last_clean":"2024-03-24T01:32:45.768808+0000","last_became_active":"2024-03-24T01:32:43.721855+0000","last_became_peered":"2024-03-24T01:32:43.721855+0000","last_unstale":"2024-03-24T01:32:45.768808+0000","last_undegraded":"2024-03-24T01:32:45.768808+0000","last_fullsized":"2024-03-24T01:32:45.768808+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:20:54.257187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.721804+0000","last_change":"2024-03-24T01:32:44.728581+0000","last_active":"2024-03-24T01:32:45.721804+0000","last_peered":"2024-03-24T01:32:45.721804+0000","last_clean":"2024-03-24T01:32:45.721804+0000","last_became_active":"2024-03-24T01:32:44.728154+0000","last_became_peered":"2024-03-24T01:32:44.728154+0000","last_unstale":"2024-03-24T01:32:45.721804+0000","last_undegraded":"2024-03-24T01:32:45.721804+0000","last_fullsized":"2024-03-24T01:32:45.721804+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:03.806877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.118235+0000","last_change":"2024-03-24T01:32:43.726876+0000","last_active":"2024-03-24T01:32:46.118235+0000","last_peered":"2024-03-24T01:32:46.118235+0000","last_clean":"2024-03-24T01:32:46.118235+0000","last_became_active":"2024-03-24T01:32:43.726628+0000","last_became_peered":"2024-03-24T01:32:43.726628+0000","last_unstale":"2024-03-24T01:32:46.118235+0000","last_undegraded":"2024-03-24T01:32:46.118235+0000","last_fullsized":"2024-03-24T01:32:46.118235+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:34.647291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.720919+0000","last_change":"2024-03-24T01:32:44.733825+0000","last_active":"2024-03-24T01:32:45.720919+0000","last_peered":"2024-03-24T01:32:45.720919+0000","last_clean":"2024-03-24T01:32:45.720919+0000","last_became_active":"2024-03-24T01:32:44.733404+0000","last_became_peered":"2024-03-24T01:32:44.733404+0000","last_unstale":"2024-03-24T01:32:45.720919+0000","last_undegraded":"2024-03-24T01:32:45.720919+0000","last_fullsized":"2024-03-24T01:32:45.720919+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:10:36.135152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"66'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117934+0000","last_change":"2024-03-24T01:32:43.725389+0000","last_active":"2024-03-24T01:32:46.117934+0000","last_peered":"2024-03-24T01:32:46.117934+0000","last_clean":"2024-03-24T01:32:46.117934+0000","last_became_active":"2024-03-24T01:32:43.724956+0000","last_became_peered":"2024-03-24T01:32:43.724956+0000","last_unstale":"2024-03-24T01:32:46.117934+0000","last_undegraded":"2024-03-24T01:32:46.117934+0000","last_fullsized":"2024-03-24T01:32:46.117934+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:13:47.859663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.721685+0000","last_change":"2024-03-24T01:32:44.743865+0000","last_active":"2024-03-24T01:32:45.721685+0000","last_peered":"2024-03-24T01:32:45.721685+0000","last_clean":"2024-03-24T01:32:45.721685+0000","last_became_active":"2024-03-24T01:32:44.741745+0000","last_became_peered":"2024-03-24T01:32:44.741745+0000","last_unstale":"2024-03-24T01:32:45.721685+0000","last_undegraded":"2024-03-24T01:32:45.721685+0000","last_fullsized":"2024-03-24T01:32:45.721685+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:10:36.135152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117487+0000","last_change":"2024-03-24T01:32:43.721896+0000","last_active":"2024-03-24T01:32:46.117487+0000","last_peered":"2024-03-24T01:32:46.117487+0000","last_clean":"2024-03-24T01:32:46.117487+0000","last_became_active":"2024-03-24T01:32:43.720277+0000","last_became_peered":"2024-03-24T01:32:43.720277+0000","last_unstale":"2024-03-24T01:32:46.117487+0000","last_undegraded":"2024-03-24T01:32:46.117487+0000","last_fullsized":"2024-03-24T01:32:46.117487+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:33:12.579951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.116685+0000","last_change":"2024-03-24T01:32:44.743387+0000","last_active":"2024-03-24T01:32:46.116685+0000","last_peered":"2024-03-24T01:32:46.116685+0000","last_clean":"2024-03-24T01:32:46.116685+0000","last_became_active":"2024-03-24T01:32:44.743079+0000","last_became_peered":"2024-03-24T01:32:44.743079+0000","last_unstale":"2024-03-24T01:32:46.116685+0000","last_undegraded":"2024-03-24T01:32:46.116685+0000","last_fullsized":"2024-03-24T01:32:46.116685+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:03:13.359822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117048+0000","last_change":"2024-03-24T01:32:43.707322+0000","last_active":"2024-03-24T01:32:46.117048+0000","last_peered":"2024-03-24T01:32:46.117048+0000","last_clean":"2024-03-24T01:32:46.117048+0000","last_became_active":"2024-03-24T01:32:43.706928+0000","last_became_peered":"2024-03-24T01:32:43.706928+0000","last_unstale":"2024-03-24T01:32:46.117048+0000","last_undegraded":"2024-03-24T01:32:46.117048+0000","last_fullsized":"2024-03-24T01:32:46.117048+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:39:22.546505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.768153+0000","last_change":"2024-03-24T01:32:44.744400+0000","last_active":"2024-03-24T01:32:45.768153+0000","last_peered":"2024-03-24T01:32:45.768153+0000","last_clean":"2024-03-24T01:32:45.768153+0000","last_became_active":"2024-03-24T01:32:44.742926+0000","last_became_peered":"2024-03-24T01:32:44.742926+0000","last_unstale":"2024-03-24T01:32:45.768153+0000","last_undegraded":"2024-03-24T01:32:45.768153+0000","last_fullsized":"2024-03-24T01:32:45.768153+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:14:45.725936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.767725+0000","last_change":"2024-03-24T01:32:43.724088+0000","last_active":"2024-03-24T01:32:45.767725+0000","last_peered":"2024-03-24T01:32:45.767725+0000","last_clean":"2024-03-24T01:32:45.767725+0000","last_became_active":"2024-03-24T01:32:43.722732+0000","last_became_peered":"2024-03-24T01:32:43.722732+0000","last_unstale":"2024-03-24T01:32:45.767725+0000","last_undegraded":"2024-03-24T01:32:45.767725+0000","last_fullsized":"2024-03-24T01:32:45.767725+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:12:55.184668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.722030+0000","last_change":"2024-03-24T01:32:44.736167+0000","last_active":"2024-03-24T01:32:45.722030+0000","last_peered":"2024-03-24T01:32:45.722030+0000","last_clean":"2024-03-24T01:32:45.722030+0000","last_became_active":"2024-03-24T01:32:44.733302+0000","last_became_peered":"2024-03-24T01:32:44.733302+0000","last_unstale":"2024-03-24T01:32:45.722030+0000","last_undegraded":"2024-03-24T01:32:45.722030+0000","last_fullsized":"2024-03-24T01:32:45.722030+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:39:10.149776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.116796+0000","last_change":"2024-03-24T01:32:43.720278+0000","last_active":"2024-03-24T01:32:46.116796+0000","last_peered":"2024-03-24T01:32:46.116796+0000","last_clean":"2024-03-24T01:32:46.116796+0000","last_became_active":"2024-03-24T01:32:43.719995+0000","last_became_peered":"2024-03-24T01:32:43.719995+0000","last_unstale":"2024-03-24T01:32:46.116796+0000","last_undegraded":"2024-03-24T01:32:46.116796+0000","last_fullsized":"2024-03-24T01:32:46.116796+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:14:55.272028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117369+0000","last_change":"2024-03-24T01:32:44.731492+0000","last_active":"2024-03-24T01:32:46.117369+0000","last_peered":"2024-03-24T01:32:46.117369+0000","last_clean":"2024-03-24T01:32:46.117369+0000","last_became_active":"2024-03-24T01:32:44.730776+0000","last_became_peered":"2024-03-24T01:32:44.730776+0000","last_unstale":"2024-03-24T01:32:46.117369+0000","last_undegraded":"2024-03-24T01:32:46.117369+0000","last_fullsized":"2024-03-24T01:32:46.117369+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:01:46.557014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.769770+0000","last_change":"2024-03-24T01:32:43.729534+0000","last_active":"2024-03-24T01:32:45.769770+0000","last_peered":"2024-03-24T01:32:45.769770+0000","last_clean":"2024-03-24T01:32:45.769770+0000","last_became_active":"2024-03-24T01:32:43.729214+0000","last_became_peered":"2024-03-24T01:32:43.729214+0000","last_unstale":"2024-03-24T01:32:45.769770+0000","last_undegraded":"2024-03-24T01:32:45.769770+0000","last_fullsized":"2024-03-24T01:32:45.769770+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:20:54.097910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.774617+0000","last_change":"2024-03-24T01:32:44.744158+0000","last_active":"2024-03-24T01:32:45.774617+0000","last_peered":"2024-03-24T01:32:45.774617+0000","last_clean":"2024-03-24T01:32:45.774617+0000","last_became_active":"2024-03-24T01:32:44.743592+0000","last_became_peered":"2024-03-24T01:32:44.743592+0000","last_unstale":"2024-03-24T01:32:45.774617+0000","last_undegraded":"2024-03-24T01:32:45.774617+0000","last_fullsized":"2024-03-24T01:32:45.774617+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:57:31.407474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"67'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.768411+0000","last_change":"2024-03-24T01:32:43.720477+0000","last_active":"2024-03-24T01:32:45.768411+0000","last_peered":"2024-03-24T01:32:45.768411+0000","last_clean":"2024-03-24T01:32:45.768411+0000","last_became_active":"2024-03-24T01:32:43.720192+0000","last_became_peered":"2024-03-24T01:32:43.720192+0000","last_unstale":"2024-03-24T01:32:45.768411+0000","last_undegraded":"2024-03-24T01:32:45.768411+0000","last_fullsized":"2024-03-24T01:32:45.768411+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:31:44.560907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.768054+0000","last_change":"2024-03-24T01:32:44.752631+0000","last_active":"2024-03-24T01:32:45.768054+0000","last_peered":"2024-03-24T01:32:45.768054+0000","last_clean":"2024-03-24T01:32:45.768054+0000","last_became_active":"2024-03-24T01:32:44.749471+0000","last_became_peered":"2024-03-24T01:32:44.749471+0000","last_unstale":"2024-03-24T01:32:45.768054+0000","last_undegraded":"2024-03-24T01:32:45.768054+0000","last_fullsized":"2024-03-24T01:32:45.768054+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:41:56.230713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.118232+0000","last_change":"2024-03-24T01:32:43.725481+0000","last_active":"2024-03-24T01:32:46.118232+0000","last_peered":"2024-03-24T01:32:46.118232+0000","last_clean":"2024-03-24T01:32:46.118232+0000","last_became_active":"2024-03-24T01:32:43.725185+0000","last_became_peered":"2024-03-24T01:32:43.725185+0000","last_unstale":"2024-03-24T01:32:46.118232+0000","last_undegraded":"2024-03-24T01:32:46.118232+0000","last_fullsized":"2024-03-24T01:32:46.118232+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:48:17.796412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.721918+0000","last_change":"2024-03-24T01:32:44.742817+0000","last_active":"2024-03-24T01:32:45.721918+0000","last_peered":"2024-03-24T01:32:45.721918+0000","last_clean":"2024-03-24T01:32:45.721918+0000","last_became_active":"2024-03-24T01:32:44.741341+0000","last_became_peered":"2024-03-24T01:32:44.741341+0000","last_unstale":"2024-03-24T01:32:45.721918+0000","last_undegraded":"2024-03-24T01:32:45.721918+0000","last_fullsized":"2024-03-24T01:32:45.721918+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:15:20.815746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.714077+0000","last_change":"2024-03-24T01:32:43.714827+0000","last_active":"2024-03-24T01:32:45.714077+0000","last_peered":"2024-03-24T01:32:45.714077+0000","last_clean":"2024-03-24T01:32:45.714077+0000","last_became_active":"2024-03-24T01:32:43.714552+0000","last_became_peered":"2024-03-24T01:32:43.714552+0000","last_unstale":"2024-03-24T01:32:45.714077+0000","last_undegraded":"2024-03-24T01:32:45.714077+0000","last_fullsized":"2024-03-24T01:32:45.714077+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:11:39.983567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.720541+0000","last_change":"2024-03-24T01:32:44.747784+0000","last_active":"2024-03-24T01:32:45.720541+0000","last_peered":"2024-03-24T01:32:45.720541+0000","last_clean":"2024-03-24T01:32:45.720541+0000","last_became_active":"2024-03-24T01:32:44.747485+0000","last_became_peered":"2024-03-24T01:32:44.747485+0000","last_unstale":"2024-03-24T01:32:45.720541+0000","last_undegraded":"2024-03-24T01:32:45.720541+0000","last_fullsized":"2024-03-24T01:32:45.720541+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:20:55.112893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117921+0000","last_change":"2024-03-24T01:32:43.724564+0000","last_active":"2024-03-24T01:32:46.117921+0000","last_peered":"2024-03-24T01:32:46.117921+0000","last_clean":"2024-03-24T01:32:46.117921+0000","last_became_active":"2024-03-24T01:32:43.724051+0000","last_became_peered":"2024-03-24T01:32:43.724051+0000","last_unstale":"2024-03-24T01:32:46.117921+0000","last_undegraded":"2024-03-24T01:32:46.117921+0000","last_fullsized":"2024-03-24T01:32:46.117921+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:20:54.097910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.721135+0000","last_change":"2024-03-24T01:32:44.741872+0000","last_active":"2024-03-24T01:32:45.721135+0000","last_peered":"2024-03-24T01:32:45.721135+0000","last_clean":"2024-03-24T01:32:45.721135+0000","last_became_active":"2024-03-24T01:32:44.741584+0000","last_became_peered":"2024-03-24T01:32:44.741584+0000","last_unstale":"2024-03-24T01:32:45.721135+0000","last_undegraded":"2024-03-24T01:32:45.721135+0000","last_fullsized":"2024-03-24T01:32:45.721135+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:03.806877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117780+0000","last_change":"2024-03-24T01:32:43.715420+0000","last_active":"2024-03-24T01:32:46.117780+0000","last_peered":"2024-03-24T01:32:46.117780+0000","last_clean":"2024-03-24T01:32:46.117780+0000","last_became_active":"2024-03-24T01:32:43.714725+0000","last_became_peered":"2024-03-24T01:32:43.714725+0000","last_unstale":"2024-03-24T01:32:46.117780+0000","last_undegraded":"2024-03-24T01:32:46.117780+0000","last_fullsized":"2024-03-24T01:32:46.117780+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:56:29.495016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.713744+0000","last_change":"2024-03-24T01:32:44.745758+0000","last_active":"2024-03-24T01:32:45.713744+0000","last_peered":"2024-03-24T01:32:45.713744+0000","last_clean":"2024-03-24T01:32:45.713744+0000","last_became_active":"2024-03-24T01:32:44.745122+0000","last_became_peered":"2024-03-24T01:32:44.745122+0000","last_unstale":"2024-03-24T01:32:45.713744+0000","last_undegraded":"2024-03-24T01:32:45.713744+0000","last_fullsized":"2024-03-24T01:32:45.713744+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:15:20.815746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.116394+0000","last_change":"2024-03-24T01:32:43.719188+0000","last_active":"2024-03-24T01:32:46.116394+0000","last_peered":"2024-03-24T01:32:46.116394+0000","last_clean":"2024-03-24T01:32:46.116394+0000","last_became_active":"2024-03-24T01:32:43.718755+0000","last_became_peered":"2024-03-24T01:32:43.718755+0000","last_unstale":"2024-03-24T01:32:46.116394+0000","last_undegraded":"2024-03-24T01:32:46.116394+0000","last_fullsized":"2024-03-24T01:32:46.116394+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:33:20.618669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.774097+0000","last_change":"2024-03-24T01:32:44.753907+0000","last_active":"2024-03-24T01:32:45.774097+0000","last_peered":"2024-03-24T01:32:45.774097+0000","last_clean":"2024-03-24T01:32:45.774097+0000","last_became_active":"2024-03-24T01:32:44.753568+0000","last_became_peered":"2024-03-24T01:32:44.753568+0000","last_unstale":"2024-03-24T01:32:45.774097+0000","last_undegraded":"2024-03-24T01:32:45.774097+0000","last_fullsized":"2024-03-24T01:32:45.774097+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:05:52.841516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.713120+0000","last_change":"2024-03-24T01:32:43.714616+0000","last_active":"2024-03-24T01:32:45.713120+0000","last_peered":"2024-03-24T01:32:45.713120+0000","last_clean":"2024-03-24T01:32:45.713120+0000","last_became_active":"2024-03-24T01:32:43.714216+0000","last_became_peered":"2024-03-24T01:32:43.714216+0000","last_unstale":"2024-03-24T01:32:45.713120+0000","last_undegraded":"2024-03-24T01:32:45.713120+0000","last_fullsized":"2024-03-24T01:32:45.713120+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:31:15.858645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:45.722226+0000","last_change":"2024-03-24T01:32:44.744990+0000","last_active":"2024-03-24T01:32:45.722226+0000","last_peered":"2024-03-24T01:32:45.722226+0000","last_clean":"2024-03-24T01:32:45.722226+0000","last_became_active":"2024-03-24T01:32:44.744503+0000","last_became_peered":"2024-03-24T01:32:44.744503+0000","last_unstale":"2024-03-24T01:32:45.722226+0000","last_undegraded":"2024-03-24T01:32:45.722226+0000","last_fullsized":"2024-03-24T01:32:45.722226+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:43.676436+0000","last_clean_scrub_stamp":"2024-03-24T01:32:43.676436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:39:23.561488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.116219+0000","last_change":"2024-03-24T01:32:43.719380+0000","last_active":"2024-03-24T01:32:46.116219+0000","last_peered":"2024-03-24T01:32:46.116219+0000","last_clean":"2024-03-24T01:32:46.116219+0000","last_became_active":"2024-03-24T01:32:43.718702+0000","last_became_peered":"2024-03-24T01:32:43.718702+0000","last_unstale":"2024-03-24T01:32:46.116219+0000","last_undegraded":"2024-03-24T01:32:46.116219+0000","last_fullsized":"2024-03-24T01:32:46.116219+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:32:42.661453+0000","last_clean_scrub_stamp":"2024-03-24T01:32:42.661453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:41:55.215730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":259,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-24T01:32:46.117100+0000","last_change":"2024-03-24T01:24:01.545855+0000","last_active":"2024-03-24T01:32:46.117100+0000","last_peered":"2024-03-24T01:32:46.117100+0000","last_clean":"2024-03-24T01:32:46.117100+0000","last_became_active":"2024-03-24T01:24:01.543811+0000","last_became_peered":"2024-03-24T01:24:01.543811+0000","last_unstale":"2024-03-24T01:32:46.117100+0000","last_undegraded":"2024-03-24T01:32:46.117100+0000","last_fullsized":"2024-03-24T01:32:46.117100+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T01:23:59.861564+0000","last_clean_scrub_stamp":"2024-03-24T01:23:59.861564+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:03.682631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574957,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32568,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339272,"statfs":{"total":96636764160,"available":96603414528,"internally_reserved":0,"allocated":1421312,"data_stored":963205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574957,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":815104,"data_stored":372095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574958,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":815104,"data_stored":372083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574957,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31992,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339848,"statfs":{"total":96636764160,"available":96604004352,"internally_reserved":0,"allocated":823296,"data_stored":373673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574957,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31976,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339864,"statfs":{"total":96636764160,"available":96604020736,"internally_reserved":0,"allocated":811008,"data_stored":372061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574958,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31976,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339864,"statfs":{"total":96636764160,"available":96604020736,"internally_reserved":0,"allocated":811008,"data_stored":372061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574957,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32008,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339832,"statfs":{"total":96636764160,"available":96603987968,"internally_reserved":0,"allocated":839680,"data_stored":374437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574958,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32560,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339280,"statfs":{"total":96636764160,"available":96603422720,"internally_reserved":0,"allocated":1404928,"data_stored":962429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T01:32:54.674 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-24T01:32:54.674 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-24T01:32:54.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.942+0000 7fcf1ea00640 1 Processor -- start 2024-03-24T01:32:54.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.943+0000 7fcf1ea00640 1 -- start start 2024-03-24T01:32:54.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.944+0000 7fcf1ea00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fcf180aeeb0 0x7fcf180b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:54.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.944+0000 7fcf1ea00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcf1815bc50 0x7fcf1806b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:54.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.944+0000 7fcf1ea00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf1806dac0 con 0x7fcf1806b600 2024-03-24T01:32:54.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.944+0000 7fcf1ea00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf1806ba70 con 0x7fcf1815bc50 2024-03-24T01:32:54.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.944+0000 7fcf1ea00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf180b4da0 con 0x7fcf180aeeb0 2024-03-24T01:32:54.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.945+0000 7fcf17400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcf1815bc50 0x7fcf1806b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:54.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.945+0000 7fcf17400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcf1815bc50 0x7fcf1806b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34504/0 (socket says 172.21.15.90:34504) 2024-03-24T01:32:54.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.945+0000 7fcf17400640 1 -- 172.21.15.90:0/3434541928 learned_addr learned my addr 172.21.15.90:0/3434541928 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:54.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.945+0000 7fcf17e00640 1 --2- 172.21.15.90:0/3434541928 >> v2:172.21.15.120:3300/0 conn(0x7fcf180aeeb0 0x7fcf180b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:54.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.945+0000 7fcf17400640 1 -- 172.21.15.90:0/3434541928 >> v2:172.21.15.120:3300/0 conn(0x7fcf180aeeb0 msgr2=0x7fcf180b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:54.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.945+0000 7fcf17400640 1 --2- 172.21.15.90:0/3434541928 >> v2:172.21.15.120:3300/0 conn(0x7fcf180aeeb0 0x7fcf180b44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:54.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.945+0000 7fcf17400640 1 -- 172.21.15.90:0/3434541928 >> v1:172.21.15.120:6789/0 conn(0x7fcf1806b600 legacy=0x7fcf1815e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:54.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.945+0000 7fcf17400640 1 -- 172.21.15.90:0/3434541928 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf1806dda0 con 0x7fcf1815bc50 2024-03-24T01:32:54.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.945+0000 7fcf17e00640 1 --2- 172.21.15.90:0/3434541928 >> v2:172.21.15.120:3300/0 conn(0x7fcf180aeeb0 0x7fcf180b44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:54.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.946+0000 7fcf17400640 1 --2- 172.21.15.90:0/3434541928 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf1815bc50 0x7fcf1806b0c0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fcf180b4f80 tx=0x7fcf0402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fd8cc28e524c3a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:54.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.947+0000 7fcf16a00640 1 -- 172.21.15.90:0/3434541928 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf04038070 con 0x7fcf1815bc50 2024-03-24T01:32:54.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.947+0000 7fcf16a00640 1 -- 172.21.15.90:0/3434541928 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf0402f050 con 0x7fcf1815bc50 2024-03-24T01:32:54.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.947+0000 7fcf16a00640 1 -- 172.21.15.90:0/3434541928 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf04031970 con 0x7fcf1815bc50 2024-03-24T01:32:54.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.947+0000 7fcf1ea00640 1 -- 172.21.15.90:0/3434541928 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf1815bc50 msgr2=0x7fcf1806b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:54.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.947+0000 7fcf1ea00640 1 --2- 172.21.15.90:0/3434541928 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf1815bc50 0x7fcf1806b0c0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fcf180b4f80 tx=0x7fcf0402b7f0 comp rx=0 tx=0).stop 2024-03-24T01:32:54.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.947+0000 7fcf1ea00640 1 -- 172.21.15.90:0/3434541928 shutdown_connections 2024-03-24T01:32:54.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.947+0000 7fcf1ea00640 1 --2- 172.21.15.90:0/3434541928 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf1815bc50 0x7fcf1806b0c0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:54.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.947+0000 7fcf1ea00640 1 --2- 172.21.15.90:0/3434541928 >> v2:172.21.15.120:3300/0 conn(0x7fcf180aeeb0 0x7fcf180b44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:54.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.947+0000 7fcf1ea00640 1 -- 172.21.15.90:0/3434541928 >> 172.21.15.90:0/3434541928 conn(0x7fcf180b1770 msgr2=0x7fcf180b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:54.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.947+0000 7fcf1ea00640 1 -- 172.21.15.90:0/3434541928 shutdown_connections 2024-03-24T01:32:54.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.947+0000 7fcf1ea00640 1 -- 172.21.15.90:0/3434541928 wait complete. 2024-03-24T01:32:54.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.949+0000 7fcf1ea00640 1 Processor -- start 2024-03-24T01:32:54.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.949+0000 7fcf1ea00640 1 -- start start 2024-03-24T01:32:54.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.950+0000 7fcf1ea00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf1806b600 0x7fcf181452d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:54.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.950+0000 7fcf1ea00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf180aeeb0 0x7fcf18153230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:54.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.950+0000 7fcf1ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf1815bc50 0x7fcf181e7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:54.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.950+0000 7fcf1ea00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf180af290 con 0x7fcf1815bc50 2024-03-24T01:32:54.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.950+0000 7fcf1ea00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1806e360 con 0x7fcf180aeeb0 2024-03-24T01:32:54.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.950+0000 7fcf1ea00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf180b4a00 con 0x7fcf1806b600 2024-03-24T01:32:54.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.951+0000 7fcf1ca00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf1815bc50 0x7fcf181e7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:54.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.951+0000 7fcf1ca00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf1815bc50 0x7fcf181e7670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34508/0 (socket says 172.21.15.90:34508) 2024-03-24T01:32:54.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.951+0000 7fcf1ca00640 1 -- 172.21.15.90:0/2787614340 learned_addr learned my addr 172.21.15.90:0/2787614340 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:54.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.951+0000 7fcf17e00640 1 --2- 172.21.15.90:0/2787614340 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf1806b600 0x7fcf181452d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:54.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.951+0000 7fcf1ca00640 1 -- 172.21.15.90:0/2787614340 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf1806b600 msgr2=0x7fcf181452d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:54.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.951+0000 7fcf1ca00640 1 --2- 172.21.15.90:0/2787614340 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf1806b600 0x7fcf181452d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:54.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.951+0000 7fcf1ca00640 1 -- 172.21.15.90:0/2787614340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf180aeeb0 msgr2=0x7fcf18153230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:54.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.951+0000 7fcf1ca00640 1 --2- 172.21.15.90:0/2787614340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf180aeeb0 0x7fcf18153230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:54.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.951+0000 7fcf1ca00640 1 -- 172.21.15.90:0/2787614340 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf18153b30 con 0x7fcf1815bc50 2024-03-24T01:32:54.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.951+0000 7fcf17e00640 1 --2- 172.21.15.90:0/2787614340 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf1806b600 0x7fcf181452d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:54.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.951+0000 7fcf1ca00640 1 --2- 172.21.15.90:0/2787614340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf1815bc50 0x7fcf181e7670 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fcf1000c780 tx=0x7fcf1000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:54.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.951+0000 7fcf0fe00640 1 -- 172.21.15.90:0/2787614340 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf10017070 con 0x7fcf1815bc50 2024-03-24T01:32:54.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.952+0000 7fcf0fe00640 1 -- 172.21.15.90:0/2787614340 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf10003da0 con 0x7fcf1815bc50 2024-03-24T01:32:54.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.952+0000 7fcf1ea00640 1 -- 172.21.15.90:0/2787614340 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf18153e10 con 0x7fcf1815bc50 2024-03-24T01:32:54.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.952+0000 7fcf0fe00640 1 -- 172.21.15.90:0/2787614340 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf100052b0 con 0x7fcf1815bc50 2024-03-24T01:32:54.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.952+0000 7fcf1ea00640 1 -- 172.21.15.90:0/2787614340 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf18154e40 con 0x7fcf1815bc50 2024-03-24T01:32:54.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.953+0000 7fcf1ea00640 1 -- 172.21.15.90:0/2787614340 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf1804f400 con 0x7fcf1815bc50 2024-03-24T01:32:54.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.954+0000 7fcf0fe00640 1 -- 172.21.15.90:0/2787614340 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcf1001a070 con 0x7fcf1815bc50 2024-03-24T01:32:54.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.955+0000 7fcf0fe00640 1 --2- 172.21.15.90:0/2787614340 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcef806bbd0 0x7fcef806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:54.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.955+0000 7fcf0fe00640 1 -- 172.21.15.90:0/2787614340 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7fcf1008a2b0 con 0x7fcf1815bc50 2024-03-24T01:32:54.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.956+0000 7fcf17e00640 1 --2- 172.21.15.90:0/2787614340 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcef806bbd0 0x7fcef806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:54.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.957+0000 7fcf17e00640 1 --2- 172.21.15.90:0/2787614340 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcef806bbd0 0x7fcef806e090 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fcf181f6ed0 tx=0x7fcf08008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:54.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:54.959+0000 7fcf0fe00640 1 -- 172.21.15.90:0/2787614340 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcf10058bf0 con 0x7fcf1815bc50 2024-03-24T01:32:55.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.226+0000 7fcf1ea00640 1 -- 172.21.15.90:0/2787614340 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fcf180cceb0 con 0x7fcf1815bc50 2024-03-24T01:32:55.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.230+0000 7fcf0fe00640 1 -- 172.21.15.90:0/2787614340 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 190+0+147 (secure 0 0 0) 0x7fcf100584f0 con 0x7fcf1815bc50 2024-03-24T01:32:55.230 INFO:teuthology.orchestra.run.smithi090.stdout:[client.0] 2024-03-24T01:32:55.230 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQB0gP9lLKWXMxAAwrrRG8zLSylpIEuUgM7q8g== 2024-03-24T01:32:55.230 INFO:teuthology.orchestra.run.smithi090.stdout: caps mds = "allow" 2024-03-24T01:32:55.231 INFO:teuthology.orchestra.run.smithi090.stdout: caps mon = "allow r" 2024-03-24T01:32:55.231 INFO:teuthology.orchestra.run.smithi090.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-24T01:32:55.231 INFO:teuthology.orchestra.run.smithi090.stderr:updated caps for client.0 2024-03-24T01:32:55.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.232+0000 7fcf1ea00640 1 -- 172.21.15.90:0/2787614340 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcef806bbd0 msgr2=0x7fcef806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:55.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.232+0000 7fcf1ea00640 1 --2- 172.21.15.90:0/2787614340 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcef806bbd0 0x7fcef806e090 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fcf181f6ed0 tx=0x7fcf08008040 comp rx=0 tx=0).stop 2024-03-24T01:32:55.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.232+0000 7fcf1ea00640 1 -- 172.21.15.90:0/2787614340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf1815bc50 msgr2=0x7fcf181e7670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:55.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.232+0000 7fcf1ea00640 1 --2- 172.21.15.90:0/2787614340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf1815bc50 0x7fcf181e7670 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fcf1000c780 tx=0x7fcf1000cc40 comp rx=0 tx=0).stop 2024-03-24T01:32:55.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.233+0000 7fcf1ea00640 1 -- 172.21.15.90:0/2787614340 shutdown_connections 2024-03-24T01:32:55.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.233+0000 7fcf1ea00640 1 --2- 172.21.15.90:0/2787614340 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fcef806bbd0 0x7fcef806e090 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:55.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.233+0000 7fcf1ea00640 1 --2- 172.21.15.90:0/2787614340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf1815bc50 0x7fcf181e7670 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:55.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.233+0000 7fcf1ea00640 1 --2- 172.21.15.90:0/2787614340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf180aeeb0 0x7fcf18153230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:55.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.233+0000 7fcf1ea00640 1 --2- 172.21.15.90:0/2787614340 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fcf1806b600 0x7fcf181452d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:55.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.233+0000 7fcf1ea00640 1 -- 172.21.15.90:0/2787614340 >> 172.21.15.90:0/2787614340 conn(0x7fcf180b1770 msgr2=0x7fcf18071990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:55.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.233+0000 7fcf1ea00640 1 -- 172.21.15.90:0/2787614340 shutdown_connections 2024-03-24T01:32:55.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.233+0000 7fcf1ea00640 1 -- 172.21.15.90:0/2787614340 wait complete. 2024-03-24T01:32:55.252 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:32:55.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.455+0000 7ff246000640 1 Processor -- start 2024-03-24T01:32:55.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.456+0000 7ff246000640 1 -- start start 2024-03-24T01:32:55.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.457+0000 7ff246000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff2400c5c50 0x7ff240166b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:55.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.457+0000 7ff246000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2400cd4a0 con 0x7ff24015f0f0 2024-03-24T01:32:55.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.457+0000 7ff246000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2400cd760 con 0x7ff2400cd010 2024-03-24T01:32:55.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.457+0000 7ff246000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff240099cd0 con 0x7ff2400c5c50 2024-03-24T01:32:55.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.457+0000 7ff23fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff2400c5c50 0x7ff240166b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:55.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.457+0000 7ff23fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff2400c5c50 0x7ff240166b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34514/0 (socket says 172.21.15.90:34514) 2024-03-24T01:32:55.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.457+0000 7ff23fe00640 1 -- 172.21.15.90:0/592837584 learned_addr learned my addr 172.21.15.90:0/592837584 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:55.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.457+0000 7ff23f400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff24015f0f0 0x7ff24009aa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43134/0 (socket says 172.21.15.90:43134) 2024-03-24T01:32:55.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.458+0000 7ff23fe00640 1 -- 172.21.15.90:0/592837584 >> v1:172.21.15.120:6789/0 conn(0x7ff2400cd010 legacy=0x7ff2400c5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:55.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.458+0000 7ff23fe00640 1 -- 172.21.15.90:0/592837584 >> v1:172.21.15.90:6789/0 conn(0x7ff24015f0f0 legacy=0x7ff24009aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:55.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.458+0000 7ff23fe00640 1 -- 172.21.15.90:0/592837584 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff240099f20 con 0x7ff2400c5c50 2024-03-24T01:32:55.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.459+0000 7ff23fe00640 1 --2- 172.21.15.90:0/592837584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2400c5c50 0x7ff240166b70 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7ff238009870 tx=0x7ff23802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25c96b217e0bc63e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:55.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.459+0000 7ff23e000640 1 -- 172.21.15.90:0/592837584 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff238035020 con 0x7ff2400c5c50 2024-03-24T01:32:55.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.459+0000 7ff23e000640 1 -- 172.21.15.90:0/592837584 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff23802d040 con 0x7ff2400c5c50 2024-03-24T01:32:55.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.459+0000 7ff23e000640 1 -- 172.21.15.90:0/592837584 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff238031a80 con 0x7ff2400c5c50 2024-03-24T01:32:55.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.460+0000 7ff246000640 1 -- 172.21.15.90:0/592837584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2400c5c50 msgr2=0x7ff240166b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:55.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.460+0000 7ff246000640 1 --2- 172.21.15.90:0/592837584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2400c5c50 0x7ff240166b70 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7ff238009870 tx=0x7ff23802b900 comp rx=0 tx=0).stop 2024-03-24T01:32:55.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.460+0000 7ff246000640 1 -- 172.21.15.90:0/592837584 shutdown_connections 2024-03-24T01:32:55.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.460+0000 7ff246000640 1 --2- 172.21.15.90:0/592837584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2400c5c50 0x7ff240166b70 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:55.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.460+0000 7ff246000640 1 -- 172.21.15.90:0/592837584 >> 172.21.15.90:0/592837584 conn(0x7ff2400b34e0 msgr2=0x7ff2400b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:55.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.460+0000 7ff246000640 1 -- 172.21.15.90:0/592837584 shutdown_connections 2024-03-24T01:32:55.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.460+0000 7ff246000640 1 -- 172.21.15.90:0/592837584 wait complete. 2024-03-24T01:32:55.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.462+0000 7ff246000640 1 Processor -- start 2024-03-24T01:32:55.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.462+0000 7ff246000640 1 -- start start 2024-03-24T01:32:55.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.463+0000 7ff246000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2400c5c50 0x7ff24006d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:55.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.463+0000 7ff246000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2400cd010 0x7ff2400c7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:55.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.463+0000 7ff246000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff24015f0f0 0x7ff2400c7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:55.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.463+0000 7ff246000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24015f4d0 con 0x7ff2400cd010 2024-03-24T01:32:55.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.463+0000 7ff246000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24009a4e0 con 0x7ff2400c5c50 2024-03-24T01:32:55.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.463+0000 7ff246000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2401670b0 con 0x7ff24015f0f0 2024-03-24T01:32:55.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.463+0000 7ff23ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2400cd010 0x7ff2400c7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:55.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.463+0000 7ff23ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2400cd010 0x7ff2400c7120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34522/0 (socket says 172.21.15.90:34522) 2024-03-24T01:32:55.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.463+0000 7ff23ea00640 1 -- 172.21.15.90:0/1709934587 learned_addr learned my addr 172.21.15.90:0/1709934587 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:55.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.463+0000 7ff23f400640 1 --2- 172.21.15.90:0/1709934587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2400c5c50 0x7ff24006d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:55.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.464+0000 7ff23ea00640 1 -- 172.21.15.90:0/1709934587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff24015f0f0 msgr2=0x7ff2400c7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:55.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.464+0000 7ff23ea00640 1 --2- 172.21.15.90:0/1709934587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff24015f0f0 0x7ff2400c7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:55.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.464+0000 7ff23ea00640 1 -- 172.21.15.90:0/1709934587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2400c5c50 msgr2=0x7ff24006d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:55.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.464+0000 7ff23ea00640 1 --2- 172.21.15.90:0/1709934587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2400c5c50 0x7ff24006d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:55.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.464+0000 7ff23ea00640 1 -- 172.21.15.90:0/1709934587 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff240157490 con 0x7ff2400cd010 2024-03-24T01:32:55.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.464+0000 7ff23ea00640 1 --2- 172.21.15.90:0/1709934587 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2400cd010 0x7ff2400c7120 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7ff22800ba80 tx=0x7ff22800bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:55.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.464+0000 7ff237400640 1 -- 172.21.15.90:0/1709934587 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22800cc00 con 0x7ff2400cd010 2024-03-24T01:32:55.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.464+0000 7ff237400640 1 -- 172.21.15.90:0/1709934587 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff22800cda0 con 0x7ff2400cd010 2024-03-24T01:32:55.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.464+0000 7ff237400640 1 -- 172.21.15.90:0/1709934587 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff228003bb0 con 0x7ff2400cd010 2024-03-24T01:32:55.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.464+0000 7ff246000640 1 -- 172.21.15.90:0/1709934587 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff24006d970 con 0x7ff2400cd010 2024-03-24T01:32:55.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.464+0000 7ff246000640 1 -- 172.21.15.90:0/1709934587 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff240164ba0 con 0x7ff2400cd010 2024-03-24T01:32:55.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.465+0000 7ff246000640 1 -- 172.21.15.90:0/1709934587 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff240068060 con 0x7ff2400cd010 2024-03-24T01:32:55.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.466+0000 7ff237400640 1 -- 172.21.15.90:0/1709934587 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff228014440 con 0x7ff2400cd010 2024-03-24T01:32:55.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.467+0000 7ff237400640 1 --2- 172.21.15.90:0/1709934587 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff21c06bbd0 0x7ff21c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:55.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.467+0000 7ff237400640 1 -- 172.21.15.90:0/1709934587 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7ff22808abd0 con 0x7ff2400cd010 2024-03-24T01:32:55.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.468+0000 7ff23f400640 1 --2- 172.21.15.90:0/1709934587 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff21c06bbd0 0x7ff21c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:55.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.469+0000 7ff23f400640 1 --2- 172.21.15.90:0/1709934587 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff21c06bbd0 0x7ff21c06e090 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ff230000f60 tx=0x7ff230009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:55.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.472+0000 7ff237400640 1 -- 172.21.15.90:0/1709934587 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff228059490 con 0x7ff2400cd010 2024-03-24T01:32:55.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.709+0000 7ff246000640 1 -- 172.21.15.90:0/1709934587 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2400b43f0 con 0x7ff2400cd010 2024-03-24T01:32:55.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.711+0000 7ff237400640 1 -- 172.21.15.90:0/1709934587 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4981 (secure 0 0 0) 0x7ff228058d90 con 0x7ff2400cd010 2024-03-24T01:32:55.711 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:32:55.711 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":50,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5177,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/3458630824","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3458630824},{"type":"v1","addr":"172.21.15.120:6837","nonce":3458630824}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":5614,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/1995195696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1995195696},{"type":"v1","addr":"172.21.15.90:6835","nonce":1995195696}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":7098,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6834/1760094626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":1760094626},{"type":"v1","addr":"172.21.15.90:6834","nonce":1760094626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T01:32:44.150735+0000","modified":"2024-03-24T01:32:47.268247+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7110},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7110":{"gid":7110,"name":"d","rank":0,"incarnation":46,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6835/910380191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":910380191},{"type":"v1","addr":"172.21.15.120:6835","nonce":910380191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7110,"qdb_cluster":[7110]},"id":4}]} 2024-03-24T01:32:55.712 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 50 2024-03-24T01:32:55.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.713+0000 7ff246000640 1 -- 172.21.15.90:0/1709934587 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff21c06bbd0 msgr2=0x7ff21c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:55.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.713+0000 7ff246000640 1 --2- 172.21.15.90:0/1709934587 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff21c06bbd0 0x7ff21c06e090 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ff230000f60 tx=0x7ff230009040 comp rx=0 tx=0).stop 2024-03-24T01:32:55.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.713+0000 7ff246000640 1 -- 172.21.15.90:0/1709934587 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2400cd010 msgr2=0x7ff2400c7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:55.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.713+0000 7ff246000640 1 --2- 172.21.15.90:0/1709934587 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2400cd010 0x7ff2400c7120 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7ff22800ba80 tx=0x7ff22800bf40 comp rx=0 tx=0).stop 2024-03-24T01:32:55.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.713+0000 7ff246000640 1 -- 172.21.15.90:0/1709934587 shutdown_connections 2024-03-24T01:32:55.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.713+0000 7ff246000640 1 --2- 172.21.15.90:0/1709934587 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff21c06bbd0 0x7ff21c06e090 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:55.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.713+0000 7ff246000640 1 --2- 172.21.15.90:0/1709934587 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2400cd010 0x7ff2400c7120 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:55.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.713+0000 7ff246000640 1 --2- 172.21.15.90:0/1709934587 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff24015f0f0 0x7ff2400c7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:55.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.713+0000 7ff246000640 1 --2- 172.21.15.90:0/1709934587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2400c5c50 0x7ff24006d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:55.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.713+0000 7ff246000640 1 -- 172.21.15.90:0/1709934587 >> 172.21.15.90:0/1709934587 conn(0x7ff2400b34e0 msgr2=0x7ff240154c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:55.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.714+0000 7ff246000640 1 -- 172.21.15.90:0/1709934587 shutdown_connections 2024-03-24T01:32:55.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:55.714+0000 7ff246000640 1 -- 172.21.15.90:0/1709934587 wait complete. 2024-03-24T01:32:56.136 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 50, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T01:32:44.150735+0000', 'modified': '2024-03-24T01:32:47.268247+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7110}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7110': {'gid': 7110, 'name': 'd', 'rank': 0, 'incarnation': 46, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.120:6835/910380191', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6834', 'nonce': 910380191}, {'type': 'v1', 'addr': '172.21.15.120:6835', 'nonce': 910380191}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7110, 'qdb_cluster': [7110]} 2024-03-24T01:32:56.136 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T01:32:56.136 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-24T01:32:56.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.355+0000 7f5e2b200640 1 Processor -- start 2024-03-24T01:32:56.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.355+0000 7f5e2b200640 1 -- start start 2024-03-24T01:32:56.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.356+0000 7f5e2b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e2c144710 0x7f5e2c06d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:56.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.357+0000 7f5e2b200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5e2c071ea0 0x7f5e2c0cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:56.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.357+0000 7f5e2b200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c0cc230 con 0x7f5e2c06dd30 2024-03-24T01:32:56.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.357+0000 7f5e2b200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c06e800 con 0x7f5e2c071ea0 2024-03-24T01:32:56.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.357+0000 7f5e2b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2c06e240 con 0x7f5e2c144710 2024-03-24T01:32:56.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.357+0000 7f5e2a800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5e2c071ea0 0x7f5e2c0cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:56.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.357+0000 7f5e2a800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f5e2c071ea0 0x7f5e2c0cc9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.90:55074/0 (socket says 172.21.15.90:55074) 2024-03-24T01:32:56.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.357+0000 7f5e2a800640 1 -- 172.21.15.90:0/2797194324 learned_addr learned my addr 172.21.15.90:0/2797194324 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:56.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.358+0000 7f5e29e00640 1 --2- 172.21.15.90:0/2797194324 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e2c144710 0x7f5e2c06d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:56.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.358+0000 7f5e2a800640 1 -- 172.21.15.90:0/2797194324 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e2c144710 msgr2=0x7f5e2c06d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:56.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.358+0000 7f5e2a800640 1 --2- 172.21.15.90:0/2797194324 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e2c144710 0x7f5e2c06d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.358+0000 7f5e2a800640 1 -- 172.21.15.90:0/2797194324 >> v1:172.21.15.120:6789/0 conn(0x7f5e2c06dd30 legacy=0x7f5e2c071790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:56.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.358+0000 7f5e2a800640 1 -- 172.21.15.90:0/2797194324 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e2c06e520 con 0x7f5e2c071ea0 2024-03-24T01:32:56.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.359+0000 7f5e2a800640 1 --2- 172.21.15.90:0/2797194324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e2c071ea0 0x7f5e2c0cc9f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f5e24009e50 tx=0x7f5e2402bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a9d1a4cb5e07537 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:56.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.360+0000 7f5e28a00640 1 -- 172.21.15.90:0/2797194324 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2402ca70 con 0x7f5e2c071ea0 2024-03-24T01:32:56.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.360+0000 7f5e28a00640 1 -- 172.21.15.90:0/2797194324 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2402cc10 con 0x7f5e2c071ea0 2024-03-24T01:32:56.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.360+0000 7f5e28a00640 1 -- 172.21.15.90:0/2797194324 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e240343f0 con 0x7f5e2c071ea0 2024-03-24T01:32:56.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.361+0000 7f5e2b200640 1 -- 172.21.15.90:0/2797194324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e2c071ea0 msgr2=0x7f5e2c0cc9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:56.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.361+0000 7f5e2b200640 1 --2- 172.21.15.90:0/2797194324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e2c071ea0 0x7f5e2c0cc9f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f5e24009e50 tx=0x7f5e2402bed0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.361+0000 7f5e2b200640 1 -- 172.21.15.90:0/2797194324 shutdown_connections 2024-03-24T01:32:56.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.361+0000 7f5e2b200640 1 --2- 172.21.15.90:0/2797194324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e2c071ea0 0x7f5e2c0cc9f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.361+0000 7f5e2b200640 1 --2- 172.21.15.90:0/2797194324 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e2c144710 0x7f5e2c06d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.361+0000 7f5e2b200640 1 -- 172.21.15.90:0/2797194324 >> 172.21.15.90:0/2797194324 conn(0x7f5e2c06b0c0 msgr2=0x7f5e2c06b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:56.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.361+0000 7f5e2b200640 1 -- 172.21.15.90:0/2797194324 shutdown_connections 2024-03-24T01:32:56.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.361+0000 7f5e2b200640 1 -- 172.21.15.90:0/2797194324 wait complete. 2024-03-24T01:32:56.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.363+0000 7f5e2b200640 1 Processor -- start 2024-03-24T01:32:56.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.363+0000 7f5e2b200640 1 -- start start 2024-03-24T01:32:56.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.364+0000 7f5e2b200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e2c06dd30 0x7f5e2c15b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:56.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.364+0000 7f5e2b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e2c071ea0 0x7f5e2c1f3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:56.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.364+0000 7f5e2b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e2c144710 0x7f5e2c13c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:56.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.364+0000 7f5e2b200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2c0cd0b0 con 0x7f5e2c144710 2024-03-24T01:32:56.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.364+0000 7f5e2b200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2c0ccf30 con 0x7f5e2c06dd30 2024-03-24T01:32:56.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.364+0000 7f5e2b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2c0cd230 con 0x7f5e2c071ea0 2024-03-24T01:32:56.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.365+0000 7f5e2a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e2c144710 0x7f5e2c13c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:56.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.365+0000 7f5e2a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e2c144710 0x7f5e2c13c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34526/0 (socket says 172.21.15.90:34526) 2024-03-24T01:32:56.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.365+0000 7f5e2a800640 1 -- 172.21.15.90:0/4068421165 learned_addr learned my addr 172.21.15.90:0/4068421165 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:56.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.365+0000 7f5e29e00640 1 --2- 172.21.15.90:0/4068421165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e2c06dd30 0x7f5e2c15b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:56.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.365+0000 7f5e29400640 1 --2- 172.21.15.90:0/4068421165 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e2c071ea0 0x7f5e2c1f3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:56.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.365+0000 7f5e2a800640 1 -- 172.21.15.90:0/4068421165 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e2c071ea0 msgr2=0x7f5e2c1f3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:56.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.365+0000 7f5e2a800640 1 --2- 172.21.15.90:0/4068421165 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e2c071ea0 0x7f5e2c1f3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.365+0000 7f5e2a800640 1 -- 172.21.15.90:0/4068421165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e2c06dd30 msgr2=0x7f5e2c15b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:56.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.365+0000 7f5e2a800640 1 --2- 172.21.15.90:0/4068421165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e2c06dd30 0x7f5e2c15b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.365+0000 7f5e2a800640 1 -- 172.21.15.90:0/4068421165 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e2c1f4a40 con 0x7f5e2c144710 2024-03-24T01:32:56.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.366+0000 7f5e2a800640 1 --2- 172.21.15.90:0/4068421165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e2c144710 0x7f5e2c13c7d0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f5e24036040 tx=0x7f5e24004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:56.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.366+0000 7f5e22000640 1 -- 172.21.15.90:0/4068421165 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e24043020 con 0x7f5e2c144710 2024-03-24T01:32:56.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.366+0000 7f5e22000640 1 -- 172.21.15.90:0/4068421165 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2403e450 con 0x7f5e2c144710 2024-03-24T01:32:56.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.366+0000 7f5e22000640 1 -- 172.21.15.90:0/4068421165 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2403e840 con 0x7f5e2c144710 2024-03-24T01:32:56.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.366+0000 7f5e2b200640 1 -- 172.21.15.90:0/4068421165 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e2c1f4480 con 0x7f5e2c144710 2024-03-24T01:32:56.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.366+0000 7f5e2b200640 1 -- 172.21.15.90:0/4068421165 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e2c0b2d50 con 0x7f5e2c144710 2024-03-24T01:32:56.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.368+0000 7f5e22000640 1 -- 172.21.15.90:0/4068421165 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5e24046020 con 0x7f5e2c144710 2024-03-24T01:32:56.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.369+0000 7f5e22000640 1 --2- 172.21.15.90:0/4068421165 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5e0c06bbd0 0x7f5e0c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:56.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.369+0000 7f5e22000640 1 -- 172.21.15.90:0/4068421165 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7f5e240ad3f0 con 0x7f5e2c144710 2024-03-24T01:32:56.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.370+0000 7f5e29e00640 1 --2- 172.21.15.90:0/4068421165 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5e0c06bbd0 0x7f5e0c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:56.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.370+0000 7f5e339fa740 1 Processor -- start 2024-03-24T01:32:56.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.370+0000 7f5e339fa740 1 -- start start 2024-03-24T01:32:56.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.371+0000 7f5e29e00640 1 --2- 172.21.15.90:0/4068421165 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5e0c06bbd0 0x7f5e0c06e090 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c136920 tx=0x7f5e1c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:56.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.371+0000 7f5e339fa740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56356ef89db0 0x56356ef8a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:56.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.371+0000 7f5e339fa740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56356ef59dd0 0x56356ef87020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:56.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.371+0000 7f5e339fa740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56356ef5d930 0x56356ef87560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:56.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.371+0000 7f5e339fa740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x56356eec3c90 con 0x56356ef5d930 2024-03-24T01:32:56.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.371+0000 7f5e339fa740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x56356ee6c980 con 0x56356ef89db0 2024-03-24T01:32:56.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.371+0000 7f5e339fa740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x56356eebfd30 con 0x56356ef59dd0 2024-03-24T01:32:56.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.371+0000 7f5e2a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56356ef5d930 0x56356ef87560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:56.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.371+0000 7f5e2a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56356ef5d930 0x56356ef87560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34532/0 (socket says 172.21.15.90:34532) 2024-03-24T01:32:56.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.371+0000 7f5e2a800640 1 -- 172.21.15.90:0/872107791 learned_addr learned my addr 172.21.15.90:0/872107791 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:56.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.372+0000 7f5e29e00640 1 --2- 172.21.15.90:0/872107791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56356ef89db0 0x56356ef8a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:56.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.372+0000 7f5e2a800640 1 -- 172.21.15.90:0/872107791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56356ef59dd0 msgr2=0x56356ef87020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:56.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.372+0000 7f5e2a800640 1 --2- 172.21.15.90:0/872107791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56356ef59dd0 0x56356ef87020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.372+0000 7f5e2a800640 1 -- 172.21.15.90:0/872107791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56356ef89db0 msgr2=0x56356ef8a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:56.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.372+0000 7f5e2a800640 1 --2- 172.21.15.90:0/872107791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56356ef89db0 0x56356ef8a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.372+0000 7f5e2a800640 1 -- 172.21.15.90:0/872107791 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56356ee84510 con 0x56356ef5d930 2024-03-24T01:32:56.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.372+0000 7f5e2a800640 1 --2- 172.21.15.90:0/872107791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56356ef5d930 0x56356ef87560 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f5e24077a10 tx=0x7f5e2403c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca5b7ef1d06ba65d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:56.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.372+0000 7f5e16600640 1 -- 172.21.15.90:0/872107791 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2407b5b0 con 0x56356ef5d930 2024-03-24T01:32:56.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.372+0000 7f5e16600640 1 -- 172.21.15.90:0/872107791 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2407f700 con 0x56356ef5d930 2024-03-24T01:32:56.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.372+0000 7f5e16600640 1 -- 172.21.15.90:0/872107791 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e24083cc0 con 0x56356ef5d930 2024-03-24T01:32:56.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.373+0000 7f5e339fa740 1 -- 172.21.15.90:0/872107791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56356ef5d930 msgr2=0x56356ef87560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:56.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.373+0000 7f5e339fa740 1 --2- 172.21.15.90:0/872107791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56356ef5d930 0x56356ef87560 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f5e24077a10 tx=0x7f5e2403c000 comp rx=0 tx=0).stop 2024-03-24T01:32:56.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.373+0000 7f5e339fa740 1 -- 172.21.15.90:0/872107791 shutdown_connections 2024-03-24T01:32:56.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.373+0000 7f5e339fa740 1 --2- 172.21.15.90:0/872107791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56356ef5d930 0x56356ef87560 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.373+0000 7f5e339fa740 1 --2- 172.21.15.90:0/872107791 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56356ef59dd0 0x56356ef87020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.373+0000 7f5e339fa740 1 --2- 172.21.15.90:0/872107791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56356ef89db0 0x56356ef8a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.373+0000 7f5e339fa740 1 -- 172.21.15.90:0/872107791 >> 172.21.15.90:0/872107791 conn(0x56356ef5c040 msgr2=0x56356efa4bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:56.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.373+0000 7f5e339fa740 1 -- 172.21.15.90:0/872107791 shutdown_connections 2024-03-24T01:32:56.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.373+0000 7f5e339fa740 1 -- 172.21.15.90:0/872107791 wait complete. 2024-03-24T01:32:56.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.374+0000 7f5e339fa740 1 -- start start 2024-03-24T01:32:56.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.374+0000 7f5e339fa740 1 Processor -- start 2024-03-24T01:32:56.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.375+0000 7f5e339fa740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56356ef5d930 0x56356ef84260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:56.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.375+0000 7f5e339fa740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56356ef89db0 0x56356ef74d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:56.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.375+0000 7f5e339fa740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56356efa3fa0 0x56356ef75240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:56.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.375+0000 7f5e339fa740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x56356eec3c90 con 0x56356efa3fa0 2024-03-24T01:32:56.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.375+0000 7f5e339fa740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x56356ee7fdd0 con 0x56356ef89db0 2024-03-24T01:32:56.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.375+0000 7f5e339fa740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x56356ef5eae0 con 0x56356ef5d930 2024-03-24T01:32:56.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.375+0000 7f5e339fa740 10 client.0 _ll_register_callbacks cb 0x56356ebd7ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T01:32:56.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.375+0000 7f5e2a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56356efa3fa0 0x56356ef75240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:56.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.375+0000 7f5e2a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56356efa3fa0 0x56356ef75240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34538/0 (socket says 172.21.15.90:34538) 2024-03-24T01:32:56.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.375+0000 7f5e2a800640 1 -- 172.21.15.90:0/1399618987 learned_addr learned my addr 172.21.15.90:0/1399618987 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:56.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.376+0000 7f5e29400640 1 --2- 172.21.15.90:0/1399618987 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56356ef89db0 0x56356ef74d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:56.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.376+0000 7f5e29e00640 1 --2- 172.21.15.90:0/1399618987 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56356ef5d930 0x56356ef84260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:56.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.376+0000 7f5e2a800640 1 -- 172.21.15.90:0/1399618987 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56356ef5d930 msgr2=0x56356ef84260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:56.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.376+0000 7f5e2a800640 1 --2- 172.21.15.90:0/1399618987 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56356ef5d930 0x56356ef84260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.376+0000 7f5e2a800640 1 -- 172.21.15.90:0/1399618987 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56356ef89db0 msgr2=0x56356ef74d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:56.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.376+0000 7f5e2a800640 1 --2- 172.21.15.90:0/1399618987 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56356ef89db0 0x56356ef74d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.376+0000 7f5e2a800640 1 -- 172.21.15.90:0/1399618987 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56356ebd82d0 con 0x56356efa3fa0 2024-03-24T01:32:56.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.376+0000 7f5e29e00640 1 --2- 172.21.15.90:0/1399618987 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56356ef5d930 0x56356ef84260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:56.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.376+0000 7f5e29400640 1 --2- 172.21.15.90:0/1399618987 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56356ef89db0 0x56356ef74d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:56.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.376+0000 7f5e2a800640 1 --2- 172.21.15.90:0/1399618987 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56356efa3fa0 0x56356ef75240 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f5e24083790 tx=0x7f5e24009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9daef37f1b2f0ea5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:56.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.376+0000 7f5e0aa00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-03-24T01:32:56.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.376+0000 7f5e0aa00640 1 -- 172.21.15.90:0/1399618987 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e24083cc0 con 0x56356efa3fa0 2024-03-24T01:32:56.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.376+0000 7f5e0aa00640 1 -- 172.21.15.90:0/1399618987 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7f5e240ad0b0 con 0x56356efa3fa0 2024-03-24T01:32:56.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.376+0000 7f5e339fa740 1 -- 172.21.15.90:0/1399618987 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5e2407f700 con 0x56356efa3fa0 2024-03-24T01:32:56.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.377+0000 7f5e0aa00640 1 -- 172.21.15.90:0/1399618987 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e2407b5b0 con 0x56356efa3fa0 2024-03-24T01:32:56.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.377+0000 7f5e339fa740 10 client.7266 fetch_fsmap learned FSMap version 50 2024-03-24T01:32:56.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.377+0000 7f5e339fa740 1 -- 172.21.15.90:0/1399618987 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=2+}) v3 -- 0x56356eca26b0 con 0x56356efa3fa0 2024-03-24T01:32:56.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.377+0000 7f5e0aa00640 1 -- 172.21.15.90:0/1399618987 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2407f700 con 0x56356efa3fa0 2024-03-24T01:32:56.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.377+0000 7f5e0aa00640 1 -- 172.21.15.90:0/1399618987 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 50) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f5e2402ca80 con 0x56356efa3fa0 2024-03-24T01:32:56.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.377+0000 7f5e339fa740 10 client.7266 fetch_fsmap finished waiting for FSMap version 50 2024-03-24T01:32:56.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.377+0000 7f5e339fa740 10 client.7266 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-24T01:32:56.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.377+0000 7f5e339fa740 20 client.7266 populate_metadata read hostname 'smithi090' 2024-03-24T01:32:56.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.377+0000 7f5e339fa740 1 --2- 172.21.15.90:0/1399618987 >> [v2:172.21.15.120:6834/910380191,v1:172.21.15.120:6835/910380191] conn(0x56356efa2770 0x56356efa2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:56.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.377+0000 7f5e339fa740 4 client.7266 mds_command: new command op to 7110 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T01:32:56.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.377+0000 7f5e339fa740 1 -- 172.21.15.90:0/1399618987 --> [v2:172.21.15.120:6834/910380191,v1:172.21.15.120:6835/910380191] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56356ee65520 con 0x56356efa2770 2024-03-24T01:32:56.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.378+0000 7f5e29e00640 1 --2- 172.21.15.90:0/1399618987 >> [v2:172.21.15.120:6834/910380191,v1:172.21.15.120:6835/910380191] conn(0x56356efa2770 0x56356efa2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:56.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.379+0000 7f5e29e00640 1 --2- 172.21.15.90:0/1399618987 >> [v2:172.21.15.120:6834/910380191,v1:172.21.15.120:6835/910380191] conn(0x56356efa2770 0x56356efa2b50 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1349a9e4485d1 server_cookie=e1d3db6cb089aca4 in_seq=0 out_seq=0 2024-03-24T01:32:56.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.379+0000 7f5e0aa00640 10 client.7266 ms_handle_connect on v2:172.21.15.120:6834/910380191 2024-03-24T01:32:56.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.381+0000 7f5e0aa00640 1 -- 172.21.15.90:0/1399618987 <== mds.0 v2:172.21.15.120:6834/910380191 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56356ee65520 con 0x56356efa2770 2024-03-24T01:32:56.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.381+0000 7f5e0aa00640 10 client.7266 handle_command_reply: tid=0 2024-03-24T01:32:56.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.381+0000 7f5e339fa740 1 client.7266 shutdown 2024-03-24T01:32:56.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.381+0000 7f5e339fa740 1 -- 172.21.15.90:0/1399618987 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56356efa3fa0 msgr2=0x56356ef75240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:56.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.381+0000 7f5e339fa740 1 --2- 172.21.15.90:0/1399618987 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56356efa3fa0 0x56356ef75240 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f5e24083790 tx=0x7f5e24009810 comp rx=0 tx=0).stop 2024-03-24T01:32:56.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.381+0000 7f5e339fa740 1 -- 172.21.15.90:0/1399618987 shutdown_connections 2024-03-24T01:32:56.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.381+0000 7f5e339fa740 1 --2- 172.21.15.90:0/1399618987 >> [v2:172.21.15.120:6834/910380191,v1:172.21.15.120:6835/910380191] conn(0x56356efa2770 0x56356efa2b50 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.381+0000 7f5e339fa740 1 --2- 172.21.15.90:0/1399618987 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56356efa3fa0 0x56356ef75240 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.381+0000 7f5e339fa740 1 --2- 172.21.15.90:0/1399618987 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56356ef89db0 0x56356ef74d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.381+0000 7f5e339fa740 1 --2- 172.21.15.90:0/1399618987 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56356ef5d930 0x56356ef84260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.381+0000 7f5e0aa00640 0 client.7266 ms_handle_reset on v2:172.21.15.120:6834/910380191 2024-03-24T01:32:56.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.381+0000 7f5e339fa740 1 -- 172.21.15.90:0/1399618987 >> 172.21.15.90:0/1399618987 conn(0x56356ef59dd0 msgr2=0x56356efa80d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:56.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.381+0000 7f5e339fa740 1 -- 172.21.15.90:0/1399618987 shutdown_connections 2024-03-24T01:32:56.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.381+0000 7f5e339fa740 1 -- 172.21.15.90:0/1399618987 wait complete. 2024-03-24T01:32:56.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.381+0000 7f5e339fa740 20 client.7266 trim_cache size 0 max 16384 2024-03-24T01:32:56.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.400+0000 7f5e339fa740 1 Processor -- start 2024-03-24T01:32:56.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.400+0000 7f5e339fa740 1 -- start start 2024-03-24T01:32:56.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.400+0000 7f5e339fa740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56356ef5d930 0x56356ef7eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:56.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.400+0000 7f5e339fa740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56356ef89db0 0x56356efa80d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:56.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.400+0000 7f5e339fa740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56356efa3fa0 0x56356efa8610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:56.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.401+0000 7f5e339fa740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2c1ffa50 con 0x56356ef89db0 2024-03-24T01:32:56.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.401+0000 7f5e339fa740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2c0b4240 con 0x56356ef5d930 2024-03-24T01:32:56.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.401+0000 7f5e339fa740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2c1ff450 con 0x56356efa3fa0 2024-03-24T01:32:56.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.401+0000 7f5e29400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56356ef89db0 0x56356efa80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:56.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.401+0000 7f5e29400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56356ef89db0 0x56356efa80d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34548/0 (socket says 172.21.15.90:34548) 2024-03-24T01:32:56.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.401+0000 7f5e29400640 1 -- 172.21.15.90:0/4159687346 learned_addr learned my addr 172.21.15.90:0/4159687346 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:56.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.401+0000 7f5e2a800640 1 --2- 172.21.15.90:0/4159687346 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56356efa3fa0 0x56356efa8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:56.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.401+0000 7f5e29e00640 1 --2- 172.21.15.90:0/4159687346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56356ef5d930 0x56356ef7eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:56.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.401+0000 7f5e29400640 1 -- 172.21.15.90:0/4159687346 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56356efa3fa0 msgr2=0x56356efa8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:56.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.401+0000 7f5e29400640 1 --2- 172.21.15.90:0/4159687346 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56356efa3fa0 0x56356efa8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.401+0000 7f5e29400640 1 -- 172.21.15.90:0/4159687346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56356ef5d930 msgr2=0x56356ef7eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:56.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.401+0000 7f5e29400640 1 --2- 172.21.15.90:0/4159687346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56356ef5d930 0x56356ef7eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.401+0000 7f5e29400640 1 -- 172.21.15.90:0/4159687346 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56356ef81b20 con 0x56356ef89db0 2024-03-24T01:32:56.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.401+0000 7f5e2a800640 1 --2- 172.21.15.90:0/4159687346 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56356efa3fa0 0x56356efa8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:56.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.402+0000 7f5e29400640 1 --2- 172.21.15.90:0/4159687346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56356ef89db0 0x56356efa80d0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f5e180014a0 tx=0x7f5e18001960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a202a7d830be3d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:56.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.402+0000 7f5e15c00640 1 -- 172.21.15.90:0/4159687346 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e180052e0 con 0x56356ef89db0 2024-03-24T01:32:56.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.402+0000 7f5e15c00640 1 -- 172.21.15.90:0/4159687346 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e18018070 con 0x56356ef89db0 2024-03-24T01:32:56.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.402+0000 7f5e15c00640 1 -- 172.21.15.90:0/4159687346 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e18003830 con 0x56356ef89db0 2024-03-24T01:32:56.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.402+0000 7f5e339fa740 1 -- 172.21.15.90:0/4159687346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56356ef89db0 msgr2=0x56356efa80d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:56.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.402+0000 7f5e339fa740 1 --2- 172.21.15.90:0/4159687346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56356ef89db0 0x56356efa80d0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f5e180014a0 tx=0x7f5e18001960 comp rx=0 tx=0).stop 2024-03-24T01:32:56.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.402+0000 7f5e339fa740 1 -- 172.21.15.90:0/4159687346 shutdown_connections 2024-03-24T01:32:56.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.402+0000 7f5e339fa740 1 --2- 172.21.15.90:0/4159687346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56356ef89db0 0x56356efa80d0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.403+0000 7f5e339fa740 1 --2- 172.21.15.90:0/4159687346 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56356efa3fa0 0x56356efa8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.403+0000 7f5e339fa740 1 --2- 172.21.15.90:0/4159687346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56356ef5d930 0x56356ef7eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.403+0000 7f5e339fa740 1 -- 172.21.15.90:0/4159687346 >> 172.21.15.90:0/4159687346 conn(0x56356ef59dd0 msgr2=0x56356ef5ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:56.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.403+0000 7f5e339fa740 1 -- 172.21.15.90:0/4159687346 shutdown_connections 2024-03-24T01:32:56.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.403+0000 7f5e339fa740 1 -- 172.21.15.90:0/4159687346 wait complete. 2024-03-24T01:32:56.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.403+0000 7f5e339fa740 1 -- start start 2024-03-24T01:32:56.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.403+0000 7f5e339fa740 1 Processor -- start 2024-03-24T01:32:56.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.404+0000 7f5e339fa740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56356ef5d930 0x56356f01c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:56.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.404+0000 7f5e339fa740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56356ef89db0 0x56356efa80d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:56.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.404+0000 7f5e339fa740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56356efa3fa0 0x56356efa8610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:56.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.404+0000 7f5e339fa740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x56356eec3c90 con 0x56356ef89db0 2024-03-24T01:32:56.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.404+0000 7f5e339fa740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x56356ee7fdd0 con 0x56356ef5d930 2024-03-24T01:32:56.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.404+0000 7f5e339fa740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x56356efb55a0 con 0x56356efa3fa0 2024-03-24T01:32:56.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.404+0000 7f5e339fa740 10 client.0 _ll_register_callbacks cb 0x7f5e24092070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T01:32:56.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.405+0000 7f5e29400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56356ef89db0 0x56356efa80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:56.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.405+0000 7f5e29400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56356ef89db0 0x56356efa80d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34560/0 (socket says 172.21.15.90:34560) 2024-03-24T01:32:56.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.405+0000 7f5e29400640 1 -- 172.21.15.90:0/1212555585 learned_addr learned my addr 172.21.15.90:0/1212555585 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:56.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.405+0000 7f5e29e00640 1 --2- 172.21.15.90:0/1212555585 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56356ef5d930 0x56356f01c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:56.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.405+0000 7f5e2a800640 1 --2- 172.21.15.90:0/1212555585 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56356efa3fa0 0x56356efa8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:56.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.405+0000 7f5e29400640 1 -- 172.21.15.90:0/1212555585 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56356efa3fa0 msgr2=0x56356efa8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:56.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.405+0000 7f5e29400640 1 --2- 172.21.15.90:0/1212555585 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56356efa3fa0 0x56356efa8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.405+0000 7f5e29400640 1 -- 172.21.15.90:0/1212555585 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56356ef5d930 msgr2=0x56356f01c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:56.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.405+0000 7f5e29400640 1 --2- 172.21.15.90:0/1212555585 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56356ef5d930 0x56356f01c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.405+0000 7f5e29400640 1 -- 172.21.15.90:0/1212555585 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56356efa3560 con 0x56356ef89db0 2024-03-24T01:32:56.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.405+0000 7f5e2a800640 1 --2- 172.21.15.90:0/1212555585 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56356efa3fa0 0x56356efa8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:56.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.405+0000 7f5e29400640 1 --2- 172.21.15.90:0/1212555585 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56356ef89db0 0x56356efa80d0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f5e18001eb0 tx=0x7f5e18000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d20aa36f28b87306 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:56.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.405+0000 7f5e0aa00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-03-24T01:32:56.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.406+0000 7f5e0aa00640 1 -- 172.21.15.90:0/1212555585 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e18003e30 con 0x56356ef89db0 2024-03-24T01:32:56.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.406+0000 7f5e0aa00640 1 -- 172.21.15.90:0/1212555585 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7f5e18015900 con 0x56356ef89db0 2024-03-24T01:32:56.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.406+0000 7f5e339fa740 1 -- 172.21.15.90:0/1212555585 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5e18018070 con 0x56356ef89db0 2024-03-24T01:32:56.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.406+0000 7f5e0aa00640 1 -- 172.21.15.90:0/1212555585 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e18015b60 con 0x56356ef89db0 2024-03-24T01:32:56.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.406+0000 7f5e339fa740 10 client.7272 fetch_fsmap learned FSMap version 50 2024-03-24T01:32:56.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.406+0000 7f5e339fa740 1 -- 172.21.15.90:0/1212555585 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=2+}) v3 -- 0x56356f01de70 con 0x56356ef89db0 2024-03-24T01:32:56.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.406+0000 7f5e0aa00640 1 -- 172.21.15.90:0/1212555585 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e18018070 con 0x56356ef89db0 2024-03-24T01:32:56.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.406+0000 7f5e0aa00640 1 -- 172.21.15.90:0/1212555585 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 50) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f5e18015d00 con 0x56356ef89db0 2024-03-24T01:32:56.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.406+0000 7f5e339fa740 10 client.7272 fetch_fsmap finished waiting for FSMap version 50 2024-03-24T01:32:56.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.407+0000 7f5e339fa740 10 client.7272 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-24T01:32:56.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.407+0000 7f5e339fa740 20 client.7272 populate_metadata read hostname 'smithi090' 2024-03-24T01:32:56.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.407+0000 7f5e339fa740 1 --2- 172.21.15.90:0/1212555585 >> [v2:172.21.15.120:6834/910380191,v1:172.21.15.120:6835/910380191] conn(0x56356efbefe0 0x56356f01cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:56.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.407+0000 7f5e339fa740 4 client.7272 mds_command: new command op to 7110 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T01:32:56.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.407+0000 7f5e339fa740 1 -- 172.21.15.90:0/1212555585 --> [v2:172.21.15.120:6834/910380191,v1:172.21.15.120:6835/910380191] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56356ef5a5d0 con 0x56356efbefe0 2024-03-24T01:32:56.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.407+0000 7f5e29e00640 1 --2- 172.21.15.90:0/1212555585 >> [v2:172.21.15.120:6834/910380191,v1:172.21.15.120:6835/910380191] conn(0x56356efbefe0 0x56356f01cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:56.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.408+0000 7f5e29e00640 1 --2- 172.21.15.90:0/1212555585 >> [v2:172.21.15.120:6834/910380191,v1:172.21.15.120:6835/910380191] conn(0x56356efbefe0 0x56356f01cdf0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb2ba48b8fd453b7 server_cookie=d268e0f87d74e1f in_seq=0 out_seq=0 2024-03-24T01:32:56.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.408+0000 7f5e0aa00640 10 client.7272 ms_handle_connect on v2:172.21.15.120:6834/910380191 2024-03-24T01:32:56.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.408+0000 7f5e0aa00640 1 -- 172.21.15.90:0/1212555585 <== mds.0 v2:172.21.15.120:6834/910380191 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+319 (crc 0 0 0) 0x56356ef5a5d0 con 0x56356efbefe0 2024-03-24T01:32:56.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.408+0000 7f5e0aa00640 10 client.7272 handle_command_reply: tid=0 2024-03-24T01:32:56.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.408+0000 7f5e339fa740 1 client.7272 shutdown 2024-03-24T01:32:56.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.409+0000 7f5e339fa740 1 -- 172.21.15.90:0/1212555585 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56356ef89db0 msgr2=0x56356efa80d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:56.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.409+0000 7f5e339fa740 1 --2- 172.21.15.90:0/1212555585 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56356ef89db0 0x56356efa80d0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f5e18001eb0 tx=0x7f5e18000bc0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.409+0000 7f5e339fa740 1 -- 172.21.15.90:0/1212555585 shutdown_connections 2024-03-24T01:32:56.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.409+0000 7f5e339fa740 1 --2- 172.21.15.90:0/1212555585 >> [v2:172.21.15.120:6834/910380191,v1:172.21.15.120:6835/910380191] conn(0x56356efbefe0 0x56356f01cdf0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.409+0000 7f5e339fa740 1 --2- 172.21.15.90:0/1212555585 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56356ef89db0 0x56356efa80d0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.409+0000 7f5e339fa740 1 --2- 172.21.15.90:0/1212555585 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x56356efa3fa0 0x56356efa8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.409+0000 7f5e339fa740 1 --2- 172.21.15.90:0/1212555585 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56356ef5d930 0x56356f01c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.409+0000 7f5e0aa00640 0 client.7272 ms_handle_reset on v2:172.21.15.120:6834/910380191 2024-03-24T01:32:56.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.409+0000 7f5e339fa740 1 -- 172.21.15.90:0/1212555585 >> 172.21.15.90:0/1212555585 conn(0x56356ef59dd0 msgr2=0x56356ef5ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:56.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.409+0000 7f5e339fa740 1 -- 172.21.15.90:0/1212555585 shutdown_connections 2024-03-24T01:32:56.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.409+0000 7f5e339fa740 1 -- 172.21.15.90:0/1212555585 wait complete. 2024-03-24T01:32:56.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.409+0000 7f5e339fa740 20 client.7272 trim_cache size 0 max 16384 2024-03-24T01:32:56.413 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-24T01:32:56.413 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "24712165-9839-4fcc-8033-988386efea74", 2024-03-24T01:32:56.413 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 0, 2024-03-24T01:32:56.413 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 7110, 2024-03-24T01:32:56.413 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-03-24T01:32:56.413 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-03-24T01:32:56.413 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-03-24T01:32:56.413 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 11.659041322, 2024-03-24T01:32:56.413 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 50, 2024-03-24T01:32:56.413 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 69, 2024-03-24T01:32:56.413 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 67, 2024-03-24T01:32:56.413 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 17.117151176 2024-03-24T01:32:56.413 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-24T01:32:56.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.410+0000 7f5e2b200640 1 -- 172.21.15.90:0/4068421165 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5e0c06bbd0 msgr2=0x7f5e0c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:56.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.410+0000 7f5e2b200640 1 --2- 172.21.15.90:0/4068421165 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5e0c06bbd0 0x7f5e0c06e090 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c136920 tx=0x7f5e1c008040 comp rx=0 tx=0).stop 2024-03-24T01:32:56.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.410+0000 7f5e2b200640 1 -- 172.21.15.90:0/4068421165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e2c144710 msgr2=0x7f5e2c13c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:56.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.410+0000 7f5e2b200640 1 --2- 172.21.15.90:0/4068421165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e2c144710 0x7f5e2c13c7d0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f5e24036040 tx=0x7f5e24004a40 comp rx=0 tx=0).stop 2024-03-24T01:32:56.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.410+0000 7f5e2b200640 1 -- 172.21.15.90:0/4068421165 shutdown_connections 2024-03-24T01:32:56.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.410+0000 7f5e2b200640 1 --2- 172.21.15.90:0/4068421165 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5e0c06bbd0 0x7f5e0c06e090 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.410+0000 7f5e2b200640 1 --2- 172.21.15.90:0/4068421165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e2c144710 0x7f5e2c13c7d0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.410+0000 7f5e2b200640 1 --2- 172.21.15.90:0/4068421165 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e2c071ea0 0x7f5e2c1f3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.410+0000 7f5e2b200640 1 --2- 172.21.15.90:0/4068421165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e2c06dd30 0x7f5e2c15b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.410+0000 7f5e2b200640 1 -- 172.21.15.90:0/4068421165 >> 172.21.15.90:0/4068421165 conn(0x7f5e2c06b0c0 msgr2=0x7f5e2c1416e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:56.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.410+0000 7f5e2b200640 1 -- 172.21.15.90:0/4068421165 shutdown_connections 2024-03-24T01:32:56.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.410+0000 7f5e2b200640 1 -- 172.21.15.90:0/4068421165 wait complete. 2024-03-24T01:32:56.428 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T01:32:56.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.639+0000 7ff988400640 1 Processor -- start 2024-03-24T01:32:56.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.639+0000 7ff988400640 1 -- start start 2024-03-24T01:32:56.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.641+0000 7ff988400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9800cc150 0x7ff9800ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:56.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.641+0000 7ff988400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff9801603e0 0x7ff98006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:56.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.641+0000 7ff988400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff980145430 0x7ff98006b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:56.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.641+0000 7ff988400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9800cd490 con 0x7ff980145430 2024-03-24T01:32:56.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.641+0000 7ff988400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff98015f900 con 0x7ff9801603e0 2024-03-24T01:32:56.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.641+0000 7ff988400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98015fbe0 con 0x7ff9800cc150 2024-03-24T01:32:56.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.641+0000 7ff986400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff980145430 0x7ff98006b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:56.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.641+0000 7ff986400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff980145430 0x7ff98006b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34576/0 (socket says 172.21.15.90:34576) 2024-03-24T01:32:56.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.641+0000 7ff986400640 1 -- 172.21.15.90:0/2852102447 learned_addr learned my addr 172.21.15.90:0/2852102447 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:56.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.641+0000 7ff985a00640 1 --2- 172.21.15.90:0/2852102447 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9800cc150 0x7ff9800ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:56.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.641+0000 7ff985000640 1 --2- 172.21.15.90:0/2852102447 >> v2:172.21.15.120:3300/0 conn(0x7ff9801603e0 0x7ff98006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:56.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.642+0000 7ff986400640 1 -- 172.21.15.90:0/2852102447 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9800cc150 msgr2=0x7ff9800ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:56.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.642+0000 7ff986400640 1 --2- 172.21.15.90:0/2852102447 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9800cc150 0x7ff9800ccb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.642+0000 7ff986400640 1 -- 172.21.15.90:0/2852102447 >> v2:172.21.15.120:3300/0 conn(0x7ff9801603e0 msgr2=0x7ff98006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:56.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.642+0000 7ff986400640 1 --2- 172.21.15.90:0/2852102447 >> v2:172.21.15.120:3300/0 conn(0x7ff9801603e0 0x7ff98006b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.642+0000 7ff986400640 1 -- 172.21.15.90:0/2852102447 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff98015fec0 con 0x7ff980145430 2024-03-24T01:32:56.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.643+0000 7ff986400640 1 --2- 172.21.15.90:0/2852102447 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff980145430 0x7ff98006b600 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7ff98006fd20 tx=0x7ff97802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f7662cb2cc819d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:56.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.643+0000 7ff97fe00640 1 -- 172.21.15.90:0/2852102447 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff97802ca70 con 0x7ff980145430 2024-03-24T01:32:56.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.643+0000 7ff97fe00640 1 -- 172.21.15.90:0/2852102447 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff97802cc10 con 0x7ff980145430 2024-03-24T01:32:56.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.643+0000 7ff97fe00640 1 -- 172.21.15.90:0/2852102447 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9780343f0 con 0x7ff980145430 2024-03-24T01:32:56.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.644+0000 7ff988400640 1 -- 172.21.15.90:0/2852102447 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff980145430 msgr2=0x7ff98006b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:56.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.644+0000 7ff988400640 1 --2- 172.21.15.90:0/2852102447 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff980145430 0x7ff98006b600 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7ff98006fd20 tx=0x7ff97802bed0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.644+0000 7ff988400640 1 -- 172.21.15.90:0/2852102447 shutdown_connections 2024-03-24T01:32:56.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.644+0000 7ff988400640 1 --2- 172.21.15.90:0/2852102447 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff980145430 0x7ff98006b600 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.644+0000 7ff988400640 1 --2- 172.21.15.90:0/2852102447 >> v2:172.21.15.120:3300/0 conn(0x7ff9801603e0 0x7ff98006b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.644+0000 7ff988400640 1 --2- 172.21.15.90:0/2852102447 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9800cc150 0x7ff9800ccb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.644+0000 7ff988400640 1 -- 172.21.15.90:0/2852102447 >> 172.21.15.90:0/2852102447 conn(0x7ff9800b1770 msgr2=0x7ff9800b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:56.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.644+0000 7ff988400640 1 -- 172.21.15.90:0/2852102447 shutdown_connections 2024-03-24T01:32:56.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.644+0000 7ff988400640 1 -- 172.21.15.90:0/2852102447 wait complete. 2024-03-24T01:32:56.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.646+0000 7ff988400640 1 Processor -- start 2024-03-24T01:32:56.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.646+0000 7ff988400640 1 -- start start 2024-03-24T01:32:56.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.647+0000 7ff988400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9800cc150 0x7ff98013d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:56.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.647+0000 7ff988400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff980145430 0x7ff9800ad720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:56.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.647+0000 7ff988400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9801603e0 0x7ff9800adc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:56.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.647+0000 7ff988400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980071b10 con 0x7ff980145430 2024-03-24T01:32:56.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.647+0000 7ff988400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980071990 con 0x7ff9800cc150 2024-03-24T01:32:56.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.647+0000 7ff988400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff980071c90 con 0x7ff9801603e0 2024-03-24T01:32:56.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.647+0000 7ff985000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff980145430 0x7ff9800ad720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:56.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.647+0000 7ff985000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff980145430 0x7ff9800ad720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34592/0 (socket says 172.21.15.90:34592) 2024-03-24T01:32:56.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.647+0000 7ff985000640 1 -- 172.21.15.90:0/186872174 learned_addr learned my addr 172.21.15.90:0/186872174 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:56.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.647+0000 7ff985a00640 1 --2- 172.21.15.90:0/186872174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9800cc150 0x7ff98013d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:56.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.647+0000 7ff985000640 1 -- 172.21.15.90:0/186872174 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9801603e0 msgr2=0x7ff9800adc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:56.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.647+0000 7ff986400640 1 --2- 172.21.15.90:0/186872174 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9801603e0 0x7ff9800adc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:56.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.647+0000 7ff985000640 1 --2- 172.21.15.90:0/186872174 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9801603e0 0x7ff9800adc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.647+0000 7ff985000640 1 -- 172.21.15.90:0/186872174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9800cc150 msgr2=0x7ff98013d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:56.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.647+0000 7ff985000640 1 --2- 172.21.15.90:0/186872174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9800cc150 0x7ff98013d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.647+0000 7ff985000640 1 -- 172.21.15.90:0/186872174 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff98015c0a0 con 0x7ff980145430 2024-03-24T01:32:56.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.647+0000 7ff986400640 1 --2- 172.21.15.90:0/186872174 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9801603e0 0x7ff9800adc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:56.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.647+0000 7ff985a00640 1 --2- 172.21.15.90:0/186872174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9800cc150 0x7ff98013d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:56.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.648+0000 7ff985000640 1 --2- 172.21.15.90:0/186872174 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff980145430 0x7ff9800ad720 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7ff96c00d5f0 tx=0x7ff96c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:56.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.648+0000 7ff97d600640 1 -- 172.21.15.90:0/186872174 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96c0125f0 con 0x7ff980145430 2024-03-24T01:32:56.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.648+0000 7ff97d600640 1 -- 172.21.15.90:0/186872174 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff96c019070 con 0x7ff980145430 2024-03-24T01:32:56.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.648+0000 7ff97d600640 1 -- 172.21.15.90:0/186872174 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96c0129d0 con 0x7ff980145430 2024-03-24T01:32:56.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.648+0000 7ff988400640 1 -- 172.21.15.90:0/186872174 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff98013c1d0 con 0x7ff980145430 2024-03-24T01:32:56.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.648+0000 7ff988400640 1 -- 172.21.15.90:0/186872174 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff98015c5f0 con 0x7ff980145430 2024-03-24T01:32:56.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.649+0000 7ff988400640 1 -- 172.21.15.90:0/186872174 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff980068060 con 0x7ff980145430 2024-03-24T01:32:56.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.650+0000 7ff97d600640 1 -- 172.21.15.90:0/186872174 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff96c012b70 con 0x7ff980145430 2024-03-24T01:32:56.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.651+0000 7ff97d600640 1 --2- 172.21.15.90:0/186872174 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff96006bbd0 0x7ff96006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:56.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.651+0000 7ff97d600640 1 -- 172.21.15.90:0/186872174 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7ff96c08b0c0 con 0x7ff980145430 2024-03-24T01:32:56.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.652+0000 7ff985a00640 1 --2- 172.21.15.90:0/186872174 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff96006bbd0 0x7ff96006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:56.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.653+0000 7ff985a00640 1 --2- 172.21.15.90:0/186872174 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff96006bbd0 0x7ff96006e090 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7ff974006fd0 tx=0x7ff974008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:56.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.656+0000 7ff97d600640 1 -- 172.21.15.90:0/186872174 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff96c0598f0 con 0x7ff980145430 2024-03-24T01:32:56.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.892+0000 7ff988400640 1 -- 172.21.15.90:0/186872174 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff9800ccbf0 con 0x7ff980145430 2024-03-24T01:32:56.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.893+0000 7ff97d600640 1 -- 172.21.15.90:0/186872174 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v50) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff96c0591f0 con 0x7ff980145430 2024-03-24T01:32:56.894 INFO:teuthology.orchestra.run.smithi090.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T01:32:56.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.896+0000 7ff988400640 1 -- 172.21.15.90:0/186872174 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff96006bbd0 msgr2=0x7ff96006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:56.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.896+0000 7ff988400640 1 --2- 172.21.15.90:0/186872174 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff96006bbd0 0x7ff96006e090 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7ff974006fd0 tx=0x7ff974008040 comp rx=0 tx=0).stop 2024-03-24T01:32:56.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.896+0000 7ff988400640 1 -- 172.21.15.90:0/186872174 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff980145430 msgr2=0x7ff9800ad720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:56.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.896+0000 7ff988400640 1 --2- 172.21.15.90:0/186872174 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff980145430 0x7ff9800ad720 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7ff96c00d5f0 tx=0x7ff96c00dab0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.896+0000 7ff988400640 1 -- 172.21.15.90:0/186872174 shutdown_connections 2024-03-24T01:32:56.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.896+0000 7ff988400640 1 --2- 172.21.15.90:0/186872174 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff96006bbd0 0x7ff96006e090 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.896+0000 7ff988400640 1 --2- 172.21.15.90:0/186872174 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff980145430 0x7ff9800ad720 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.896+0000 7ff988400640 1 --2- 172.21.15.90:0/186872174 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff9801603e0 0x7ff9800adc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.896+0000 7ff988400640 1 --2- 172.21.15.90:0/186872174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9800cc150 0x7ff98013d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:56.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.896+0000 7ff988400640 1 -- 172.21.15.90:0/186872174 >> 172.21.15.90:0/186872174 conn(0x7ff9800b1770 msgr2=0x7ff9800b0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:56.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.896+0000 7ff988400640 1 -- 172.21.15.90:0/186872174 shutdown_connections 2024-03-24T01:32:56.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:56.896+0000 7ff988400640 1 -- 172.21.15.90:0/186872174 wait complete. 2024-03-24T01:32:56.916 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T01:32:56.916 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T01:32:56.916 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi090.front.sepia.ceph.com 2024-03-24T01:32:56.916 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T01:32:56.916 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T01:32:56.916 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T01:32:56.916 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T01:32:56.917 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T01:32:56.917 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T01:32:56.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:32:56.917 DEBUG:teuthology.orchestra.run.smithi090:> ip addr 2024-03-24T01:32:56.965 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T01:32:56.965 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T01:32:56.965 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T01:32:56.965 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:32:56.965 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-03-24T01:32:56.965 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:32:56.965 INFO:teuthology.orchestra.run.smithi090.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T01:32:56.966 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a8 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:32:56.966 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f0 2024-03-24T01:32:56.966 INFO:teuthology.orchestra.run.smithi090.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T01:32:56.966 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a9 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:32:56.966 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f1 2024-03-24T01:32:56.966 INFO:teuthology.orchestra.run.smithi090.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T01:32:56.966 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:66 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:32:56.966 INFO:teuthology.orchestra.run.smithi090.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T01:32:56.966 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:67 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:32:56.966 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.21.15.90/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T01:32:56.966 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft 42005sec preferred_lft 42005sec 2024-03-24T01:32:56.967 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::ec4:7aff:fe8f:d767/64 scope link 2024-03-24T01:32:56.967 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:32:56.967 INFO:teuthology.orchestra.run.smithi090.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T01:32:56.967 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 06:50:1d:bd:ed:70 brd ff:ff:ff:ff:ff:ff 2024-03-24T01:32:56.967 INFO:teuthology.orchestra.run.smithi090.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T01:32:56.967 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:32:56.967 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::a05b:d3ff:fea5:6fd0/64 scope link 2024-03-24T01:32:56.967 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:32:56.967 INFO:teuthology.orchestra.run.smithi090.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T01:32:56.967 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether de:94:0b:7e:c6:41 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:32:56.967 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::dc94:bff:fe7e:c641/64 scope link 2024-03-24T01:32:56.967 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:32:56.967 INFO:teuthology.orchestra.run.smithi090.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T01:32:56.968 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 06:50:1d:bd:ed:70 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:32:56.968 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::450:1dff:febd:ed70/64 scope link 2024-03-24T01:32:56.968 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-24T01:32:56.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:32:56.969 DEBUG:teuthology.orchestra.run.smithi090:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T01:32:57.057 INFO:teuthology.orchestra.run.smithi090.stdout:1 2024-03-24T01:32:57.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:32:57.062 DEBUG:teuthology.orchestra.run.smithi090:> route 2024-03-24T01:32:57.113 INFO:teuthology.orchestra.run.smithi090.stdout:Kernel IP routing table 2024-03-24T01:32:57.114 INFO:teuthology.orchestra.run.smithi090.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T01:32:57.114 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T01:32:57.114 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T01:32:57.114 INFO:teuthology.orchestra.run.smithi090.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T01:32:57.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:32:57.115 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2024-03-24T01:32:57.115 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-24T01:32:57.115 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T01:32:57.115 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T01:32:57.115 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T01:32:57.115 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-24T01:32:57.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:32:57.296 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-03-24T01:32:57.343 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T01:32:57.343 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T01:32:57.344 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T01:32:57.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:32:57.344 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set brx.0 up 2024-03-24T01:32:57.431 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-24T01:32:57.431 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T01:32:57.431 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T01:32:57.485 INFO:teuthology.orchestra.run.smithi090.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T01:32:57.485 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T01:32:57.485 DEBUG:teuthology.orchestra.run.smithi090:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T01:32:57.536 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-24T01:32:57.634 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T01:32:57.686 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-24T01:32:57.686 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-24T01:32:57.686 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-24T01:32:57.686 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-24T01:32:57.686 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-24T01:32:57.687 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-24T01:32:57.687 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-24T01:32:57.687 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-24T01:32:57.687 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-24T01:32:57.687 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-24T01:32:57.687 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-24T01:32:57.687 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/selinux 2024-03-24T01:32:57.687 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T01:32:57.687 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-24T01:32:57.687 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-24T01:32:57.687 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-24T01:32:57.687 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-24T01:32:57.687 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T01:32:57.687 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-24T01:32:57.687 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T01:32:57.688 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-24T01:32:57.688 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T01:32:57.688 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T01:32:57.688 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-24T01:32:57.688 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-24T01:32:57.688 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T01:32:57.688 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T01:32:57.688 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T01:32:57.688 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T01:32:57.688 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-24T01:32:57.688 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:32:57.688 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:32:57.688 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:32:57.688 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:32:57.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:32:57.689 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-24T01:32:57.739 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-24T01:32:57.739 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-24T01:32:57.743 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-24T01:32:57.818 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T01:32:57.860 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-24T01:32:57.860 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-24T01:32:57.860 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-24T01:32:57.860 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-24T01:32:57.860 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-24T01:32:57.860 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-24T01:32:57.860 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-24T01:32:57.860 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-24T01:32:57.860 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-24T01:32:57.860 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-24T01:32:57.861 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-24T01:32:57.861 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/selinux 2024-03-24T01:32:57.861 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T01:32:57.861 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-24T01:32:57.861 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-24T01:32:57.861 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-24T01:32:57.861 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-24T01:32:57.861 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T01:32:57.861 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-24T01:32:57.861 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T01:32:57.861 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-24T01:32:57.861 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T01:32:57.861 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T01:32:57.861 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-24T01:32:57.861 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-24T01:32:57.862 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T01:32:57.862 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T01:32:57.862 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T01:32:57.862 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T01:32:57.862 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-24T01:32:57.862 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:32:57.862 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:32:57.862 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:32:57.862 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:32:57.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:32:57.863 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-24T01:32:57.927 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:2024-03-24T01:32:57.927+0000 7f9fd543a480 -1 init, newargv = 0x558f14272cd0 newargc=15 2024-03-24T01:32:57.927 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[65114]: starting ceph client 2024-03-24T01:32:57.938 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[65114]: starting fuse 2024-03-24T01:32:57.948 INFO:teuthology.orchestra.run.smithi090.stdout:40 2024-03-24T01:32:57.948 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-03-24T01:32:57.948 DEBUG:teuthology.orchestra.run.smithi090:> sudo stdin-killer -- python3 -c ' 2024-03-24T01:32:57.948 DEBUG:teuthology.orchestra.run.smithi090:> import glob 2024-03-24T01:32:57.948 DEBUG:teuthology.orchestra.run.smithi090:> import re 2024-03-24T01:32:57.948 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-24T01:32:57.948 DEBUG:teuthology.orchestra.run.smithi090:> import subprocess 2024-03-24T01:32:57.948 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:32:57.948 DEBUG:teuthology.orchestra.run.smithi090:> def _find_admin_socket(client_name): 2024-03-24T01:32:57.949 DEBUG:teuthology.orchestra.run.smithi090:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-24T01:32:57.949 DEBUG:teuthology.orchestra.run.smithi090:> files = glob.glob(asok_path) 2024-03-24T01:32:57.949 DEBUG:teuthology.orchestra.run.smithi090:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-24T01:32:57.949 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:32:57.949 DEBUG:teuthology.orchestra.run.smithi090:> # Given a non-glob path, it better be there 2024-03-24T01:32:57.949 DEBUG:teuthology.orchestra.run.smithi090:> if "*" not in asok_path: 2024-03-24T01:32:57.949 DEBUG:teuthology.orchestra.run.smithi090:> assert(len(files) == 1) 2024-03-24T01:32:57.949 DEBUG:teuthology.orchestra.run.smithi090:> return files[0] 2024-03-24T01:32:57.949 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:32:57.949 DEBUG:teuthology.orchestra.run.smithi090:> for f in files: 2024-03-24T01:32:57.949 DEBUG:teuthology.orchestra.run.smithi090:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T01:32:57.949 DEBUG:teuthology.orchestra.run.smithi090:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T01:32:57.949 DEBUG:teuthology.orchestra.run.smithi090:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T01:32:57.949 DEBUG:teuthology.orchestra.run.smithi090:> contents = proc_f.read() 2024-03-24T01:32:57.949 DEBUG:teuthology.orchestra.run.smithi090:> if mountpoint in contents: 2024-03-24T01:32:57.949 DEBUG:teuthology.orchestra.run.smithi090:> return f 2024-03-24T01:32:57.949 DEBUG:teuthology.orchestra.run.smithi090:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T01:32:57.949 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:32:57.949 DEBUG:teuthology.orchestra.run.smithi090:> print(_find_admin_socket("client.0")) 2024-03-24T01:32:57.950 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-24T01:32:58.144 INFO:teuthology.orchestra.run.smithi090.stdout:/var/run/ceph/ceph-client.0.65114.asok 2024-03-24T01:32:58.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T01:32:58.161 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.65114.asok 2024-03-24T01:32:58.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:32:58.161 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.65114.asok status 2024-03-24T01:32:58.388 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-24T01:32:58.389 INFO:teuthology.orchestra.run.smithi090.stdout: "metadata": { 2024-03-24T01:32:58.389 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_sha1": "3be9a5c9bc793e11e2800a8c0c696e8b46742033", 2024-03-24T01:32:58.389 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-24T01:32:58.389 INFO:teuthology.orchestra.run.smithi090.stdout: "entity_id": "0", 2024-03-24T01:32:58.389 INFO:teuthology.orchestra.run.smithi090.stdout: "hostname": "smithi090", 2024-03-24T01:32:58.389 INFO:teuthology.orchestra.run.smithi090.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-24T01:32:58.389 INFO:teuthology.orchestra.run.smithi090.stdout: "pid": "65114", 2024-03-24T01:32:58.389 INFO:teuthology.orchestra.run.smithi090.stdout: "root": "/" 2024-03-24T01:32:58.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:32:58.389 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_count": 0, 2024-03-24T01:32:58.389 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_pinned_count": 0, 2024-03-24T01:32:58.389 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 7284, 2024-03-24T01:32:58.390 INFO:teuthology.orchestra.run.smithi090.stdout: "inst": { 2024-03-24T01:32:58.390 INFO:teuthology.orchestra.run.smithi090.stdout: "name": { 2024-03-24T01:32:58.390 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "client", 2024-03-24T01:32:58.390 INFO:teuthology.orchestra.run.smithi090.stdout: "num": 7284 2024-03-24T01:32:58.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:32:58.390 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-24T01:32:58.390 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-24T01:32:58.390 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-03-24T01:32:58.390 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1071194461 2024-03-24T01:32:58.390 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-24T01:32:58.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:32:58.390 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-24T01:32:58.390 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-24T01:32:58.390 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-03-24T01:32:58.390 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1071194461 2024-03-24T01:32:58.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-24T01:32:58.391 INFO:teuthology.orchestra.run.smithi090.stdout: "inst_str": "client.7284 192.168.0.1:0/1071194461", 2024-03-24T01:32:58.391 INFO:teuthology.orchestra.run.smithi090.stdout: "addr_str": "192.168.0.1:0/1071194461", 2024-03-24T01:32:58.391 INFO:teuthology.orchestra.run.smithi090.stdout: "inode_count": 1, 2024-03-24T01:32:58.391 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_epoch": 50, 2024-03-24T01:32:58.391 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch": 69, 2024-03-24T01:32:58.391 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch_barrier": 0, 2024-03-24T01:32:58.391 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklisted": false, 2024-03-24T01:32:58.391 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs" 2024-03-24T01:32:58.391 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-24T01:32:58.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:32:58.403 DEBUG:teuthology.orchestra.run.smithi090:> stat --file-system '--printf=%T 2024-03-24T01:32:58.403 DEBUG:teuthology.orchestra.run.smithi090:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-24T01:32:58.458 INFO:teuthology.orchestra.run.smithi090.stdout:fuseblk 2024-03-24T01:32:58.458 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-24T01:32:58.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:32:58.458 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T01:32:58.550 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2024-03-24T01:32:58.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.766+0000 7f50d9c00640 1 Processor -- start 2024-03-24T01:32:58.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.767+0000 7f50d9c00640 1 -- start start 2024-03-24T01:32:58.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.768+0000 7f50d9c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f50d415da30 0x7f50d415de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:58.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.768+0000 7f50d9c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50d40af140 con 0x7f50d406b0c0 2024-03-24T01:32:58.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.768+0000 7f50d9c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50d415e550 con 0x7f50d40acea0 2024-03-24T01:32:58.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.768+0000 7f50d9c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50d40b2c70 con 0x7f50d415da30 2024-03-24T01:32:58.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.769+0000 7f50d2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f50d415da30 0x7f50d415de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:58.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.769+0000 7f50d2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f50d415da30 0x7f50d415de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34596/0 (socket says 172.21.15.90:34596) 2024-03-24T01:32:58.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.769+0000 7f50d2a00640 1 -- 172.21.15.90:0/1577410707 learned_addr learned my addr 172.21.15.90:0/1577410707 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:58.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.769+0000 7f50d2a00640 1 -- 172.21.15.90:0/1577410707 >> v1:172.21.15.120:6789/0 conn(0x7f50d40acea0 legacy=0x7f50d40aea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:58.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.769+0000 7f50d2a00640 1 -- 172.21.15.90:0/1577410707 >> v1:172.21.15.90:6789/0 conn(0x7f50d406b0c0 legacy=0x7f50d4161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:58.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.769+0000 7f50d2a00640 1 -- 172.21.15.90:0/1577410707 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d415e830 con 0x7f50d415da30 2024-03-24T01:32:58.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.770+0000 7f50d2a00640 1 --2- 172.21.15.90:0/1577410707 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50d415da30 0x7f50d415de10 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f50c4009870 tx=0x7f50c402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fa07cd3644503ca server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:58.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.770+0000 7f50d2000640 1 -- 172.21.15.90:0/1577410707 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c4034070 con 0x7f50d415da30 2024-03-24T01:32:58.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.770+0000 7f50d2000640 1 -- 172.21.15.90:0/1577410707 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50c4030860 con 0x7f50d415da30 2024-03-24T01:32:58.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.770+0000 7f50d2000640 1 -- 172.21.15.90:0/1577410707 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c4030c30 con 0x7f50d415da30 2024-03-24T01:32:58.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.771+0000 7f50d9c00640 1 -- 172.21.15.90:0/1577410707 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50d415da30 msgr2=0x7f50d415de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:58.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.771+0000 7f50d9c00640 1 --2- 172.21.15.90:0/1577410707 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50d415da30 0x7f50d415de10 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f50c4009870 tx=0x7f50c402b9a0 comp rx=0 tx=0).stop 2024-03-24T01:32:58.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.771+0000 7f50d9c00640 1 -- 172.21.15.90:0/1577410707 shutdown_connections 2024-03-24T01:32:58.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.771+0000 7f50d9c00640 1 --2- 172.21.15.90:0/1577410707 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50d415da30 0x7f50d415de10 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:58.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.771+0000 7f50d9c00640 1 -- 172.21.15.90:0/1577410707 >> 172.21.15.90:0/1577410707 conn(0x7f50d40b1770 msgr2=0x7f50d40b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:58.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.771+0000 7f50d9c00640 1 -- 172.21.15.90:0/1577410707 shutdown_connections 2024-03-24T01:32:58.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.771+0000 7f50d9c00640 1 -- 172.21.15.90:0/1577410707 wait complete. 2024-03-24T01:32:58.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.773+0000 7f50d9c00640 1 Processor -- start 2024-03-24T01:32:58.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.773+0000 7f50d9c00640 1 -- start start 2024-03-24T01:32:58.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.774+0000 7f50d9c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50d406b0c0 0x7f50d4150290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:58.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.774+0000 7f50d9c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50d40acea0 0x7f50d4143420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:58.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.774+0000 7f50d9c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50d415da30 0x7f50d40b0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:58.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.774+0000 7f50d9c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d415edf0 con 0x7f50d40acea0 2024-03-24T01:32:58.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.774+0000 7f50d9c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d415f160 con 0x7f50d415da30 2024-03-24T01:32:58.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.774+0000 7f50d9c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50d40ad300 con 0x7f50d406b0c0 2024-03-24T01:32:58.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.774+0000 7f50ca000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50d40acea0 0x7f50d4143420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:58.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.774+0000 7f50ca000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50d40acea0 0x7f50d4143420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34602/0 (socket says 172.21.15.90:34602) 2024-03-24T01:32:58.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.774+0000 7f50ca000640 1 -- 172.21.15.90:0/2019873589 learned_addr learned my addr 172.21.15.90:0/2019873589 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:58.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.774+0000 7f50d2a00640 1 --2- 172.21.15.90:0/2019873589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50d406b0c0 0x7f50d4150290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:58.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.775+0000 7f50ca000640 1 -- 172.21.15.90:0/2019873589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50d406b0c0 msgr2=0x7f50d4150290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:58.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.775+0000 7f50ca000640 1 --2- 172.21.15.90:0/2019873589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50d406b0c0 0x7f50d4150290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:58.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.775+0000 7f50d3400640 1 --2- 172.21.15.90:0/2019873589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50d415da30 0x7f50d40b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:58.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.775+0000 7f50ca000640 1 -- 172.21.15.90:0/2019873589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50d415da30 msgr2=0x7f50d40b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:58.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.775+0000 7f50ca000640 1 --2- 172.21.15.90:0/2019873589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50d415da30 0x7f50d40b0550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:58.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.775+0000 7f50ca000640 1 -- 172.21.15.90:0/2019873589 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d4154ba0 con 0x7f50d40acea0 2024-03-24T01:32:58.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.775+0000 7f50d2a00640 1 --2- 172.21.15.90:0/2019873589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50d406b0c0 0x7f50d4150290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:58.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.775+0000 7f50ca000640 1 --2- 172.21.15.90:0/2019873589 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50d40acea0 0x7f50d4143420 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f50bc00c430 tx=0x7f50bc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:58.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.775+0000 7f50cb400640 1 -- 172.21.15.90:0/2019873589 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50bc011790 con 0x7f50d40acea0 2024-03-24T01:32:58.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.775+0000 7f50cb400640 1 -- 172.21.15.90:0/2019873589 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50bc017070 con 0x7f50d40acea0 2024-03-24T01:32:58.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.775+0000 7f50cb400640 1 -- 172.21.15.90:0/2019873589 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50bc011bb0 con 0x7f50d40acea0 2024-03-24T01:32:58.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.775+0000 7f50d9c00640 1 -- 172.21.15.90:0/2019873589 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50d4154e80 con 0x7f50d40acea0 2024-03-24T01:32:58.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.775+0000 7f50d9c00640 1 -- 172.21.15.90:0/2019873589 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50d4155440 con 0x7f50d40acea0 2024-03-24T01:32:58.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.776+0000 7f50d9c00640 1 -- 172.21.15.90:0/2019873589 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50d404f400 con 0x7f50d40acea0 2024-03-24T01:32:58.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.778+0000 7f50cb400640 1 -- 172.21.15.90:0/2019873589 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f50bc01c460 con 0x7f50d40acea0 2024-03-24T01:32:58.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.779+0000 7f50cb400640 1 --2- 172.21.15.90:0/2019873589 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f50b006bbd0 0x7f50b006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:58.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.779+0000 7f50cb400640 1 -- 172.21.15.90:0/2019873589 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7f50bc08ac50 con 0x7f50d40acea0 2024-03-24T01:32:58.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.780+0000 7f50d2a00640 1 --2- 172.21.15.90:0/2019873589 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f50b006bbd0 0x7f50b006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:58.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.781+0000 7f50d2a00640 1 --2- 172.21.15.90:0/2019873589 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f50b006bbd0 0x7f50b006e090 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f50c4036000 tx=0x7f50c403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:58.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:58.784+0000 7f50cb400640 1 -- 172.21.15.90:0/2019873589 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f50bc059480 con 0x7f50d40acea0 2024-03-24T01:32:59.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.025+0000 7f50d9c00640 1 -- 172.21.15.90:0/2019873589 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f50d40d05d0 con 0x7f50d40acea0 2024-03-24T01:32:59.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.240+0000 7f50cb400640 1 -- 172.21.15.90:0/2019873589 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v51) v1 ==== 105+0+0 (secure 0 0 0) 0x7f50bc058d80 con 0x7f50d40acea0 2024-03-24T01:32:59.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.244+0000 7f50d9c00640 1 -- 172.21.15.90:0/2019873589 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f50b006bbd0 msgr2=0x7f50b006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:59.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.244+0000 7f50d9c00640 1 --2- 172.21.15.90:0/2019873589 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f50b006bbd0 0x7f50b006e090 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f50c4036000 tx=0x7f50c403b040 comp rx=0 tx=0).stop 2024-03-24T01:32:59.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.244+0000 7f50d9c00640 1 -- 172.21.15.90:0/2019873589 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50d40acea0 msgr2=0x7f50d4143420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:59.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.244+0000 7f50d9c00640 1 --2- 172.21.15.90:0/2019873589 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50d40acea0 0x7f50d4143420 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f50bc00c430 tx=0x7f50bc00c8f0 comp rx=0 tx=0).stop 2024-03-24T01:32:59.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.244+0000 7f50d9c00640 1 -- 172.21.15.90:0/2019873589 shutdown_connections 2024-03-24T01:32:59.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.244+0000 7f50d9c00640 1 --2- 172.21.15.90:0/2019873589 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f50b006bbd0 0x7f50b006e090 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:59.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.244+0000 7f50d9c00640 1 --2- 172.21.15.90:0/2019873589 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50d40acea0 0x7f50d4143420 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:59.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.244+0000 7f50d9c00640 1 --2- 172.21.15.90:0/2019873589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50d415da30 0x7f50d40b0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:59.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.244+0000 7f50d9c00640 1 --2- 172.21.15.90:0/2019873589 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f50d406b0c0 0x7f50d4150290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:59.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.244+0000 7f50d9c00640 1 -- 172.21.15.90:0/2019873589 >> 172.21.15.90:0/2019873589 conn(0x7f50d40b1770 msgr2=0x7f50d4159850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:59.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.244+0000 7f50d9c00640 1 -- 172.21.15.90:0/2019873589 shutdown_connections 2024-03-24T01:32:59.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.244+0000 7f50d9c00640 1 -- 172.21.15.90:0/2019873589 wait complete. 2024-03-24T01:32:59.270 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:32:59.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.456+0000 7f2164800640 1 Processor -- start 2024-03-24T01:32:59.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.456+0000 7f2164800640 1 -- start start 2024-03-24T01:32:59.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.458+0000 7f2164800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f215c0723f0 0x7f215c0b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:59.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.458+0000 7f2164800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f215c0adf80 con 0x7f215c146f30 2024-03-24T01:32:59.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.458+0000 7f2164800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f215c161af0 con 0x7f215c1469d0 2024-03-24T01:32:59.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.458+0000 7f2164800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f215c1618a0 con 0x7f215c0723f0 2024-03-24T01:32:59.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.458+0000 7f2161400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f215c0723f0 0x7f215c0b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:59.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.458+0000 7f2162800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f215c146f30 0x7f215c0b3140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43150/0 (socket says 172.21.15.90:43150) 2024-03-24T01:32:59.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.458+0000 7f2162800640 1 -- 172.21.15.90:0/3983028921 learned_addr learned my addr 172.21.15.90:0/3983028921 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:32:59.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.459+0000 7f2161400640 1 -- 172.21.15.90:0/3983028921 >> v1:172.21.15.120:6789/0 conn(0x7f215c1469d0 legacy=0x7f215c071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:32:59.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.459+0000 7f2161400640 1 -- 172.21.15.90:0/3983028921 >> v1:172.21.15.90:6789/0 conn(0x7f215c146f30 legacy=0x7f215c0b3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:59.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.459+0000 7f2161400640 1 -- 172.21.15.90:0/3983028921 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f215c161dd0 con 0x7f215c0723f0 2024-03-24T01:32:59.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.460+0000 7f2161400640 1 --2- 172.21.15.90:0/3983028921 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f215c0723f0 0x7f215c0b2c00 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f214c0097a0 tx=0x7f214c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f45a3028c8cbe22 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:59.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.460+0000 7f2160a00640 1 -- 172.21.15.90:0/3983028921 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f214c0315a0 con 0x7f215c0723f0 2024-03-24T01:32:59.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.460+0000 7f2160a00640 1 -- 172.21.15.90:0/3983028921 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f214c037070 con 0x7f215c0723f0 2024-03-24T01:32:59.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.460+0000 7f2160a00640 1 -- 172.21.15.90:0/3983028921 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f214c031960 con 0x7f215c0723f0 2024-03-24T01:32:59.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.461+0000 7f2164800640 1 -- 172.21.15.90:0/3983028921 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f215c0723f0 msgr2=0x7f215c0b2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:59.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.461+0000 7f2164800640 1 --2- 172.21.15.90:0/3983028921 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f215c0723f0 0x7f215c0b2c00 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f214c0097a0 tx=0x7f214c02b840 comp rx=0 tx=0).stop 2024-03-24T01:32:59.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.461+0000 7f2164800640 1 -- 172.21.15.90:0/3983028921 shutdown_connections 2024-03-24T01:32:59.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.461+0000 7f2164800640 1 --2- 172.21.15.90:0/3983028921 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f215c0723f0 0x7f215c0b2c00 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:59.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.461+0000 7f2164800640 1 -- 172.21.15.90:0/3983028921 >> 172.21.15.90:0/3983028921 conn(0x7f215c06b2a0 msgr2=0x7f215c06b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:59.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.461+0000 7f2164800640 1 -- 172.21.15.90:0/3983028921 shutdown_connections 2024-03-24T01:32:59.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.461+0000 7f2164800640 1 -- 172.21.15.90:0/3983028921 wait complete. 2024-03-24T01:32:59.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.463+0000 7f2164800640 1 Processor -- start 2024-03-24T01:32:59.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.463+0000 7f2164800640 1 -- start start 2024-03-24T01:32:59.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.464+0000 7f2164800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f215c0723f0 0x7f215c15cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:59.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.464+0000 7f2164800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f215c1469d0 0x7f215c1f7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:59.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.464+0000 7f2164800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f215c146f30 0x7f215c0afee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:59.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.464+0000 7f2164800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f215c162390 con 0x7f215c0723f0 2024-03-24T01:32:59.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.464+0000 7f2164800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f215c162700 con 0x7f215c146f30 2024-03-24T01:32:59.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.464+0000 7f2164800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f215c147480 con 0x7f215c1469d0 2024-03-24T01:32:59.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.464+0000 7f2161e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f215c0723f0 0x7f215c15cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:59.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.464+0000 7f2161e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f215c0723f0 0x7f215c15cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34622/0 (socket says 172.21.15.90:34622) 2024-03-24T01:32:59.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.464+0000 7f2161e00640 1 -- 172.21.15.90:0/1380561961 learned_addr learned my addr 172.21.15.90:0/1380561961 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:32:59.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.464+0000 7f2162800640 1 --2- 172.21.15.90:0/1380561961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f215c146f30 0x7f215c0afee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:59.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.464+0000 7f2161400640 1 --2- 172.21.15.90:0/1380561961 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f215c1469d0 0x7f215c1f7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:59.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.464+0000 7f2161e00640 1 -- 172.21.15.90:0/1380561961 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f215c1469d0 msgr2=0x7f215c1f7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:59.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.464+0000 7f2161e00640 1 --2- 172.21.15.90:0/1380561961 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f215c1469d0 0x7f215c1f7fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:59.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.464+0000 7f2161e00640 1 -- 172.21.15.90:0/1380561961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f215c146f30 msgr2=0x7f215c0afee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:59.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.465+0000 7f2161e00640 1 --2- 172.21.15.90:0/1380561961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f215c146f30 0x7f215c0afee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:59.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.465+0000 7f2161e00640 1 -- 172.21.15.90:0/1380561961 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f215c0ae6d0 con 0x7f215c0723f0 2024-03-24T01:32:59.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.465+0000 7f2162800640 1 --2- 172.21.15.90:0/1380561961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f215c146f30 0x7f215c0afee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:32:59.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.465+0000 7f2161e00640 1 --2- 172.21.15.90:0/1380561961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f215c0723f0 0x7f215c15cb30 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f215000c460 tx=0x7f215000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:59.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.465+0000 7f2156000640 1 -- 172.21.15.90:0/1380561961 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2150016020 con 0x7f215c0723f0 2024-03-24T01:32:59.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.465+0000 7f2156000640 1 -- 172.21.15.90:0/1380561961 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2150012870 con 0x7f215c0723f0 2024-03-24T01:32:59.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.465+0000 7f2164800640 1 -- 172.21.15.90:0/1380561961 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f215c0b0e70 con 0x7f215c0723f0 2024-03-24T01:32:59.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.465+0000 7f2156000640 1 -- 172.21.15.90:0/1380561961 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2150012c60 con 0x7f215c0723f0 2024-03-24T01:32:59.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.465+0000 7f2164800640 1 -- 172.21.15.90:0/1380561961 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f215c15d330 con 0x7f215c0723f0 2024-03-24T01:32:59.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.466+0000 7f2164800640 1 -- 172.21.15.90:0/1380561961 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f215c073570 con 0x7f215c0723f0 2024-03-24T01:32:59.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.467+0000 7f2156000640 1 -- 172.21.15.90:0/1380561961 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f2150019020 con 0x7f215c0723f0 2024-03-24T01:32:59.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.468+0000 7f2156000640 1 --2- 172.21.15.90:0/1380561961 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f214006bbd0 0x7f214006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:32:59.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.468+0000 7f2156000640 1 -- 172.21.15.90:0/1380561961 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7f215008a980 con 0x7f215c0723f0 2024-03-24T01:32:59.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.469+0000 7f2161400640 1 --2- 172.21.15.90:0/1380561961 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f214006bbd0 0x7f214006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:32:59.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.470+0000 7f2161400640 1 --2- 172.21.15.90:0/1380561961 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f214006bbd0 0x7f214006e090 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f214c036000 tx=0x7f214c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:32:59.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.473+0000 7f2156000640 1 -- 172.21.15.90:0/1380561961 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f2150020030 con 0x7f215c0723f0 2024-03-24T01:32:59.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.712+0000 7f2164800640 1 -- 172.21.15.90:0/1380561961 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f215c04f4e0 con 0x7f215c0723f0 2024-03-24T01:32:59.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.713+0000 7f2156000640 1 -- 172.21.15.90:0/1380561961 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+4997 (secure 0 0 0) 0x7f2150059240 con 0x7f215c0723f0 2024-03-24T01:32:59.714 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:32:59.714 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":52,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5177,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/3458630824","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3458630824},{"type":"v1","addr":"172.21.15.120:6837","nonce":3458630824}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":5614,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/1995195696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1995195696},{"type":"v1","addr":"172.21.15.90:6835","nonce":1995195696}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T01:32:44.150735+0000","modified":"2024-03-24T01:32:59.247092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7110,"mds_1":7098},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7098":{"gid":7098,"name":"a","rank":1,"incarnation":52,"state":"up:creating","state_seq":1,"addr":"172.21.15.90:6834/1760094626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":1760094626},{"type":"v1","addr":"172.21.15.90:6834","nonce":1760094626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7110":{"gid":7110,"name":"d","rank":0,"incarnation":46,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6835/910380191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":910380191},{"type":"v1","addr":"172.21.15.120:6835","nonce":910380191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7110,"qdb_cluster":[7110]},"id":4}]} 2024-03-24T01:32:59.714 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 52 2024-03-24T01:32:59.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.716+0000 7f2164800640 1 -- 172.21.15.90:0/1380561961 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f214006bbd0 msgr2=0x7f214006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:59.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.716+0000 7f2164800640 1 --2- 172.21.15.90:0/1380561961 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f214006bbd0 0x7f214006e090 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f214c036000 tx=0x7f214c008030 comp rx=0 tx=0).stop 2024-03-24T01:32:59.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.716+0000 7f2164800640 1 -- 172.21.15.90:0/1380561961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f215c0723f0 msgr2=0x7f215c15cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:32:59.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.716+0000 7f2164800640 1 --2- 172.21.15.90:0/1380561961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f215c0723f0 0x7f215c15cb30 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f215000c460 tx=0x7f215000c920 comp rx=0 tx=0).stop 2024-03-24T01:32:59.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.716+0000 7f2164800640 1 -- 172.21.15.90:0/1380561961 shutdown_connections 2024-03-24T01:32:59.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.716+0000 7f2164800640 1 --2- 172.21.15.90:0/1380561961 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f214006bbd0 0x7f214006e090 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:59.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.716+0000 7f2164800640 1 --2- 172.21.15.90:0/1380561961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f215c146f30 0x7f215c0afee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:59.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.716+0000 7f2164800640 1 --2- 172.21.15.90:0/1380561961 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f215c1469d0 0x7f215c1f7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:59.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.717+0000 7f2164800640 1 --2- 172.21.15.90:0/1380561961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f215c0723f0 0x7f215c15cb30 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:32:59.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.717+0000 7f2164800640 1 -- 172.21.15.90:0/1380561961 >> 172.21.15.90:0/1380561961 conn(0x7f215c06b2a0 msgr2=0x7f215c0b2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:32:59.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.717+0000 7f2164800640 1 -- 172.21.15.90:0/1380561961 shutdown_connections 2024-03-24T01:32:59.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:32:59.717+0000 7f2164800640 1 -- 172.21.15.90:0/1380561961 wait complete. 2024-03-24T01:33:00.140 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 52, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T01:32:44.150735+0000', 'modified': '2024-03-24T01:32:59.247092+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 7110, 'mds_1': 7098}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7098': {'gid': 7098, 'name': 'a', 'rank': 1, 'incarnation': 52, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.90:6834/1760094626', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6833', 'nonce': 1760094626}, {'type': 'v1', 'addr': '172.21.15.90:6834', 'nonce': 1760094626}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_7110': {'gid': 7110, 'name': 'd', 'rank': 0, 'incarnation': 46, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.120:6835/910380191', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6834', 'nonce': 910380191}, {'type': 'v1', 'addr': '172.21.15.120:6835', 'nonce': 910380191}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7110, 'qdb_cluster': [7110]} 2024-03-24T01:33:00.140 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7098:up:creating 2024-03-24T01:33:00.197 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:33:00.206 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:33:00.669 INFO:tasks.ceph.mds.d.smithi120.stderr:2024-03-24T01:33:00.669+0000 7ff94c800640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-24T01:33:01.141 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:33:01.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.341+0000 7ff2a7200640 1 Processor -- start 2024-03-24T01:33:01.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.341+0000 7ff2a7200640 1 -- start start 2024-03-24T01:33:01.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.343+0000 7ff2a7200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff2a815df80 0x7ff2a8162330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:01.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.343+0000 7ff2a7200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2a80ce4b0 con 0x7ff2a80b5c40 2024-03-24T01:33:01.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.343+0000 7ff2a7200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2a80ce790 con 0x7ff2a815dba0 2024-03-24T01:33:01.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.343+0000 7ff2a7200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2a8162870 con 0x7ff2a815df80 2024-03-24T01:33:01.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.343+0000 7ff2a6800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff2a815df80 0x7ff2a8162330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:01.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.343+0000 7ff2a6800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff2a815df80 0x7ff2a8162330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54342/0 (socket says 172.21.15.90:54342) 2024-03-24T01:33:01.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.343+0000 7ff29fe00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff2a80b5c40 0x7ff2a80d1d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:58842/0 (socket says 172.21.15.90:58842) 2024-03-24T01:33:01.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.343+0000 7ff2a6800640 1 -- 172.21.15.90:0/3264767033 learned_addr learned my addr 172.21.15.90:0/3264767033 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:33:01.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.343+0000 7ff2a6800640 1 -- 172.21.15.90:0/3264767033 >> v1:172.21.15.120:6789/0 conn(0x7ff2a815dba0 legacy=0x7ff2a80b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:33:01.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.343+0000 7ff2a6800640 1 -- 172.21.15.90:0/3264767033 >> v1:172.21.15.90:6789/0 conn(0x7ff2a80b5c40 legacy=0x7ff2a80d1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:33:01.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.344+0000 7ff2a6800640 1 -- 172.21.15.90:0/3264767033 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a80cea70 con 0x7ff2a815df80 2024-03-24T01:33:01.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.344+0000 7ff2a5400640 1 -- 172.21.15.90:0/3264767033 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 496251742 0 0) 0x7ff2a80ce4b0 con 0x7ff2a80b5c40 2024-03-24T01:33:01.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.345+0000 7ff2a6800640 1 --2- 172.21.15.90:0/3264767033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2a815df80 0x7ff2a8162330 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0009e80 tx=0x7ff2a002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0eb72121f70f1de server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:33:01.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.345+0000 7ff2a5400640 1 -- 172.21.15.90:0/3264767033 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a002ca70 con 0x7ff2a815df80 2024-03-24T01:33:01.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.345+0000 7ff2a5400640 1 -- 172.21.15.90:0/3264767033 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2a002cc10 con 0x7ff2a815df80 2024-03-24T01:33:01.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.345+0000 7ff2a5400640 1 -- 172.21.15.90:0/3264767033 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a00343f0 con 0x7ff2a815df80 2024-03-24T01:33:01.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.345+0000 7ff2a7200640 1 -- 172.21.15.90:0/3264767033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2a815df80 msgr2=0x7ff2a8162330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:33:01.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.345+0000 7ff2a7200640 1 --2- 172.21.15.90:0/3264767033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2a815df80 0x7ff2a8162330 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0009e80 tx=0x7ff2a002bed0 comp rx=0 tx=0).stop 2024-03-24T01:33:01.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.345+0000 7ff2a7200640 1 -- 172.21.15.90:0/3264767033 shutdown_connections 2024-03-24T01:33:01.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.345+0000 7ff2a7200640 1 --2- 172.21.15.90:0/3264767033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2a815df80 0x7ff2a8162330 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:01.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.345+0000 7ff2a7200640 1 -- 172.21.15.90:0/3264767033 >> 172.21.15.90:0/3264767033 conn(0x7ff2a80d0530 msgr2=0x7ff2a80d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:33:01.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.346+0000 7ff2a7200640 1 -- 172.21.15.90:0/3264767033 shutdown_connections 2024-03-24T01:33:01.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.346+0000 7ff2a7200640 1 -- 172.21.15.90:0/3264767033 wait complete. 2024-03-24T01:33:01.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.347+0000 7ff2a7200640 1 Processor -- start 2024-03-24T01:33:01.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.347+0000 7ff2a7200640 1 -- start start 2024-03-24T01:33:01.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.348+0000 7ff2a7200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2a80b5c40 0x7ff2a80c7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:01.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.348+0000 7ff2a7200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2a815dba0 0x7ff2a815c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:01.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.348+0000 7ff2a7200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2a815df80 0x7ff2a81ed1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:01.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.348+0000 7ff2a7200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a80b2ae0 con 0x7ff2a80b5c40 2024-03-24T01:33:01.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.348+0000 7ff2a7200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a80b2960 con 0x7ff2a815dba0 2024-03-24T01:33:01.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.348+0000 7ff2a7200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a80b2c60 con 0x7ff2a815df80 2024-03-24T01:33:01.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.348+0000 7ff2a5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2a80b5c40 0x7ff2a80c7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:01.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.348+0000 7ff2a5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2a80b5c40 0x7ff2a80c7fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54350/0 (socket says 172.21.15.90:54350) 2024-03-24T01:33:01.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.348+0000 7ff2a5e00640 1 -- 172.21.15.90:0/3236242784 learned_addr learned my addr 172.21.15.90:0/3236242784 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:33:01.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.349+0000 7ff2a5e00640 1 -- 172.21.15.90:0/3236242784 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2a815df80 msgr2=0x7ff2a81ed1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:33:01.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.349+0000 7ff2a5e00640 1 --2- 172.21.15.90:0/3236242784 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2a815df80 0x7ff2a81ed1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:01.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.349+0000 7ff2a5e00640 1 -- 172.21.15.90:0/3236242784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2a815dba0 msgr2=0x7ff2a815c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:33:01.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.349+0000 7ff2a5e00640 1 --2- 172.21.15.90:0/3236242784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2a815dba0 0x7ff2a815c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:01.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.349+0000 7ff2a5e00640 1 -- 172.21.15.90:0/3236242784 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a80c88f0 con 0x7ff2a80b5c40 2024-03-24T01:33:01.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.349+0000 7ff2a5e00640 1 --2- 172.21.15.90:0/3236242784 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2a80b5c40 0x7ff2a80c7fc0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7ff29800c460 tx=0x7ff29800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:33:01.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.349+0000 7ff29e000640 1 -- 172.21.15.90:0/3236242784 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff298016020 con 0x7ff2a80b5c40 2024-03-24T01:33:01.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.349+0000 7ff29e000640 1 -- 172.21.15.90:0/3236242784 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff298012870 con 0x7ff2a80b5c40 2024-03-24T01:33:01.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.349+0000 7ff2a7200640 1 -- 172.21.15.90:0/3236242784 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2a80c8bd0 con 0x7ff2a80b5c40 2024-03-24T01:33:01.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.349+0000 7ff29e000640 1 -- 172.21.15.90:0/3236242784 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff298012c60 con 0x7ff2a80b5c40 2024-03-24T01:33:01.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.350+0000 7ff2a7200640 1 -- 172.21.15.90:0/3236242784 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2a80c9190 con 0x7ff2a80b5c40 2024-03-24T01:33:01.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.351+0000 7ff2a7200640 1 -- 172.21.15.90:0/3236242784 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2a804f400 con 0x7ff2a80b5c40 2024-03-24T01:33:01.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.351+0000 7ff29e000640 1 -- 172.21.15.90:0/3236242784 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff298019020 con 0x7ff2a80b5c40 2024-03-24T01:33:01.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.353+0000 7ff29e000640 1 --2- 172.21.15.90:0/3236242784 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff28406bbd0 0x7ff28406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:01.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.353+0000 7ff29e000640 1 -- 172.21.15.90:0/3236242784 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7ff29808a990 con 0x7ff2a80b5c40 2024-03-24T01:33:01.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.353+0000 7ff29fe00640 1 --2- 172.21.15.90:0/3236242784 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff28406bbd0 0x7ff28406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:01.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.355+0000 7ff29fe00640 1 --2- 172.21.15.90:0/3236242784 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff28406bbd0 0x7ff28406e090 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7ff290007fd0 tx=0x7ff290007960 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:33:01.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.358+0000 7ff29e000640 1 -- 172.21.15.90:0/3236242784 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff298020030 con 0x7ff2a80b5c40 2024-03-24T01:33:01.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.593+0000 7ff2a7200640 1 -- 172.21.15.90:0/3236242784 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2a80b64b0 con 0x7ff2a80b5c40 2024-03-24T01:33:01.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.594+0000 7ff29e000640 1 -- 172.21.15.90:0/3236242784 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+5000 (secure 0 0 0) 0x7ff298059240 con 0x7ff2a80b5c40 2024-03-24T01:33:01.595 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:33:01.595 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":53,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5177,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/3458630824","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3458630824},{"type":"v1","addr":"172.21.15.120:6837","nonce":3458630824}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":5614,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/1995195696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1995195696},{"type":"v1","addr":"172.21.15.90:6835","nonce":1995195696}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T01:32:44.150735+0000","modified":"2024-03-24T01:33:00.250575+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7110,"mds_1":7098},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7098":{"gid":7098,"name":"a","rank":1,"incarnation":52,"state":"up:active","state_seq":7,"addr":"172.21.15.90:6834/1760094626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":1760094626},{"type":"v1","addr":"172.21.15.90:6834","nonce":1760094626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7110":{"gid":7110,"name":"d","rank":0,"incarnation":46,"state":"up:active","state_seq":3,"addr":"172.21.15.120:6835/910380191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":910380191},{"type":"v1","addr":"172.21.15.120:6835","nonce":910380191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7110,"qdb_cluster":[7098,7110]},"id":4}]} 2024-03-24T01:33:01.595 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 53 2024-03-24T01:33:01.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.597+0000 7ff2a7200640 1 -- 172.21.15.90:0/3236242784 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff28406bbd0 msgr2=0x7ff28406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:33:01.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.597+0000 7ff2a7200640 1 --2- 172.21.15.90:0/3236242784 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff28406bbd0 0x7ff28406e090 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7ff290007fd0 tx=0x7ff290007960 comp rx=0 tx=0).stop 2024-03-24T01:33:01.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.597+0000 7ff2a7200640 1 -- 172.21.15.90:0/3236242784 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2a80b5c40 msgr2=0x7ff2a80c7fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:33:01.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.597+0000 7ff2a7200640 1 --2- 172.21.15.90:0/3236242784 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2a80b5c40 0x7ff2a80c7fc0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7ff29800c460 tx=0x7ff29800c920 comp rx=0 tx=0).stop 2024-03-24T01:33:01.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.597+0000 7ff2a7200640 1 -- 172.21.15.90:0/3236242784 shutdown_connections 2024-03-24T01:33:01.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.597+0000 7ff2a7200640 1 --2- 172.21.15.90:0/3236242784 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff28406bbd0 0x7ff28406e090 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:01.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.597+0000 7ff2a7200640 1 --2- 172.21.15.90:0/3236242784 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff2a815df80 0x7ff2a81ed1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:01.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.597+0000 7ff2a7200640 1 --2- 172.21.15.90:0/3236242784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2a815dba0 0x7ff2a815c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:01.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.597+0000 7ff2a7200640 1 --2- 172.21.15.90:0/3236242784 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2a80b5c40 0x7ff2a80c7fc0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:01.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.597+0000 7ff2a7200640 1 -- 172.21.15.90:0/3236242784 >> 172.21.15.90:0/3236242784 conn(0x7ff2a80d0530 msgr2=0x7ff2a80b7dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:33:01.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.597+0000 7ff2a7200640 1 -- 172.21.15.90:0/3236242784 shutdown_connections 2024-03-24T01:33:01.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:01.597+0000 7ff2a7200640 1 -- 172.21.15.90:0/3236242784 wait complete. 2024-03-24T01:33:02.018 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 53, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T01:32:44.150735+0000', 'modified': '2024-03-24T01:33:00.250575+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 7110, 'mds_1': 7098}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7098': {'gid': 7098, 'name': 'a', 'rank': 1, 'incarnation': 52, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.90:6834/1760094626', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6833', 'nonce': 1760094626}, {'type': 'v1', 'addr': '172.21.15.90:6834', 'nonce': 1760094626}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_7110': {'gid': 7110, 'name': 'd', 'rank': 0, 'incarnation': 46, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.120:6835/910380191', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6834', 'nonce': 910380191}, {'type': 'v1', 'addr': '172.21.15.120:6835', 'nonce': 910380191}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7110, 'qdb_cluster': [7098, 7110]} 2024-03-24T01:33:02.018 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2024-03-24T01:33:02.018 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-24T01:33:02.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.220+0000 7fe999600640 1 Processor -- start 2024-03-24T01:33:02.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.220+0000 7fe999600640 1 -- start start 2024-03-24T01:33:02.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.221+0000 7fe999600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe9940b5550 0x7fe9940d1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.221+0000 7fe999600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe99415de50 0x7fe9940b9540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.221+0000 7fe999600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9940d35b0 0x7fe9940b5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.221+0000 7fe999600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9940b9a80 con 0x7fe9940b5550 2024-03-24T01:33:02.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.221+0000 7fe999600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9940b5e70 con 0x7fe99415de50 2024-03-24T01:33:02.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.221+0000 7fe999600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9940d2510 con 0x7fe9940d35b0 2024-03-24T01:33:02.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.221+0000 7fe993e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe9940b5550 0x7fe9940d1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.221+0000 7fe993e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe9940b5550 0x7fe9940d1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54356/0 (socket says 172.21.15.90:54356) 2024-03-24T01:33:02.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.221+0000 7fe993e00640 1 -- 172.21.15.90:0/1013653919 learned_addr learned my addr 172.21.15.90:0/1013653919 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:33:02.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.222+0000 7fe998c00640 1 --2- 172.21.15.90:0/1013653919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9940d35b0 0x7fe9940b5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.222+0000 7fe98be00640 1 --2- 172.21.15.90:0/1013653919 >> v2:172.21.15.120:3300/0 conn(0x7fe99415de50 0x7fe9940b9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.222+0000 7fe993e00640 1 -- 172.21.15.90:0/1013653919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9940d35b0 msgr2=0x7fe9940b5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:33:02.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.222+0000 7fe993e00640 1 --2- 172.21.15.90:0/1013653919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9940d35b0 0x7fe9940b5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.222+0000 7fe993e00640 1 -- 172.21.15.90:0/1013653919 >> v2:172.21.15.120:3300/0 conn(0x7fe99415de50 msgr2=0x7fe9940b9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:33:02.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.222+0000 7fe993e00640 1 --2- 172.21.15.90:0/1013653919 >> v2:172.21.15.120:3300/0 conn(0x7fe99415de50 0x7fe9940b9540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.222+0000 7fe993e00640 1 -- 172.21.15.90:0/1013653919 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9940d27f0 con 0x7fe9940b5550 2024-03-24T01:33:02.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.223+0000 7fe993e00640 1 --2- 172.21.15.90:0/1013653919 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9940b5550 0x7fe9940d1ea0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fe994068a10 tx=0x7fe98402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9120b692b0e41e41 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:33:02.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.223+0000 7fe993400640 1 -- 172.21.15.90:0/1013653919 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98402e070 con 0x7fe9940b5550 2024-03-24T01:33:02.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.223+0000 7fe993400640 1 -- 172.21.15.90:0/1013653919 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe984037070 con 0x7fe9940b5550 2024-03-24T01:33:02.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.223+0000 7fe993400640 1 -- 172.21.15.90:0/1013653919 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe984031a00 con 0x7fe9940b5550 2024-03-24T01:33:02.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.224+0000 7fe999600640 1 -- 172.21.15.90:0/1013653919 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9940b5550 msgr2=0x7fe9940d1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:33:02.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.224+0000 7fe999600640 1 --2- 172.21.15.90:0/1013653919 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9940b5550 0x7fe9940d1ea0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fe994068a10 tx=0x7fe98402b880 comp rx=0 tx=0).stop 2024-03-24T01:33:02.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.224+0000 7fe999600640 1 -- 172.21.15.90:0/1013653919 shutdown_connections 2024-03-24T01:33:02.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.224+0000 7fe999600640 1 --2- 172.21.15.90:0/1013653919 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9940d35b0 0x7fe9940b5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.224+0000 7fe999600640 1 --2- 172.21.15.90:0/1013653919 >> v2:172.21.15.120:3300/0 conn(0x7fe99415de50 0x7fe9940b9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.224+0000 7fe999600640 1 --2- 172.21.15.90:0/1013653919 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9940b5550 0x7fe9940d1ea0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.224+0000 7fe999600640 1 -- 172.21.15.90:0/1013653919 >> 172.21.15.90:0/1013653919 conn(0x7fe9940d0550 msgr2=0x7fe9940d0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:33:02.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.224+0000 7fe999600640 1 -- 172.21.15.90:0/1013653919 shutdown_connections 2024-03-24T01:33:02.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.224+0000 7fe999600640 1 -- 172.21.15.90:0/1013653919 wait complete. 2024-03-24T01:33:02.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.225+0000 7fe999600640 1 Processor -- start 2024-03-24T01:33:02.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.226+0000 7fe999600640 1 -- start start 2024-03-24T01:33:02.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.226+0000 7fe999600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9940b5550 0x7fe99414c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.226+0000 7fe999600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9940d35b0 0x7fe99415f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.226+0000 7fe999600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe99415de50 0x7fe99415fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.226+0000 7fe999600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9940d3090 con 0x7fe9940d35b0 2024-03-24T01:33:02.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.226+0000 7fe999600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9940d2db0 con 0x7fe99415de50 2024-03-24T01:33:02.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.226+0000 7fe999600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe99406c5d0 con 0x7fe9940b5550 2024-03-24T01:33:02.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.227+0000 7fe98be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9940d35b0 0x7fe99415f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.227+0000 7fe98be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9940d35b0 0x7fe99415f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54364/0 (socket says 172.21.15.90:54364) 2024-03-24T01:33:02.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.227+0000 7fe98be00640 1 -- 172.21.15.90:0/2753560451 learned_addr learned my addr 172.21.15.90:0/2753560451 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:33:02.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.227+0000 7fe98be00640 1 -- 172.21.15.90:0/2753560451 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9940b5550 msgr2=0x7fe99414c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:33:02.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.227+0000 7fe98be00640 1 --2- 172.21.15.90:0/2753560451 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9940b5550 0x7fe99414c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.227+0000 7fe98be00640 1 -- 172.21.15.90:0/2753560451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe99415de50 msgr2=0x7fe99415fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:33:02.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.227+0000 7fe98be00640 1 --2- 172.21.15.90:0/2753560451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe99415de50 0x7fe99415fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.227+0000 7fe98be00640 1 -- 172.21.15.90:0/2753560451 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9940c9400 con 0x7fe9940d35b0 2024-03-24T01:33:02.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.227+0000 7fe98be00640 1 --2- 172.21.15.90:0/2753560451 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9940d35b0 0x7fe99415f820 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fe97c00d5f0 tx=0x7fe97c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:33:02.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.227+0000 7fe990c00640 1 -- 172.21.15.90:0/2753560451 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97c0125f0 con 0x7fe9940d35b0 2024-03-24T01:33:02.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.228+0000 7fe990c00640 1 -- 172.21.15.90:0/2753560451 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe97c019070 con 0x7fe9940d35b0 2024-03-24T01:33:02.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.228+0000 7fe990c00640 1 -- 172.21.15.90:0/2753560451 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97c0129d0 con 0x7fe9940d35b0 2024-03-24T01:33:02.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.228+0000 7fe999600640 1 -- 172.21.15.90:0/2753560451 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9940c96e0 con 0x7fe9940d35b0 2024-03-24T01:33:02.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.228+0000 7fe999600640 1 -- 172.21.15.90:0/2753560451 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9940c9ca0 con 0x7fe9940d35b0 2024-03-24T01:33:02.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.230+0000 7fe990c00640 1 -- 172.21.15.90:0/2753560451 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe97c012b70 con 0x7fe9940d35b0 2024-03-24T01:33:02.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.231+0000 7fe990c00640 1 --2- 172.21.15.90:0/2753560451 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe97006bbd0 0x7fe97006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.231+0000 7fe990c00640 1 -- 172.21.15.90:0/2753560451 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7fe97c08b130 con 0x7fe9940d35b0 2024-03-24T01:33:02.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.231+0000 7fe99d9a3740 1 Processor -- start 2024-03-24T01:33:02.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.231+0000 7fe99d9a3740 1 -- start start 2024-03-24T01:33:02.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.231+0000 7fe993e00640 1 --2- 172.21.15.90:0/2753560451 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe97006bbd0 0x7fe97006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.232+0000 7fe99d9a3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55618ecf3db0 0x55618ecf4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.232+0000 7fe99d9a3740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55618ecc3dd0 0x55618ecf1020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.232+0000 7fe99d9a3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55618ecc7930 0x55618ecf1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.232+0000 7fe99d9a3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55618ec2dc90 con 0x55618ecc7930 2024-03-24T01:33:02.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.232+0000 7fe99d9a3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55618ebd6980 con 0x55618ecf3db0 2024-03-24T01:33:02.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.232+0000 7fe99d9a3740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55618ec29d30 con 0x55618ecc3dd0 2024-03-24T01:33:02.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.233+0000 7fe998c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55618ecc7930 0x55618ecf1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.233+0000 7fe998c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55618ecc7930 0x55618ecf1560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54368/0 (socket says 172.21.15.90:54368) 2024-03-24T01:33:02.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.233+0000 7fe998c00640 1 -- 172.21.15.90:0/3733335757 learned_addr learned my addr 172.21.15.90:0/3733335757 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:33:02.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.233+0000 7fe993e00640 1 --2- 172.21.15.90:0/3733335757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55618ecf3db0 0x55618ecf4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.233+0000 7fe98be00640 1 --2- 172.21.15.90:0/3733335757 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55618ecc3dd0 0x55618ecf1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.233+0000 7fe998c00640 1 -- 172.21.15.90:0/3733335757 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55618ecc3dd0 msgr2=0x55618ecf1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:33:02.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.233+0000 7fe998c00640 1 --2- 172.21.15.90:0/3733335757 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55618ecc3dd0 0x55618ecf1020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.233+0000 7fe998c00640 1 -- 172.21.15.90:0/3733335757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55618ecf3db0 msgr2=0x55618ecf4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:33:02.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.233+0000 7fe998c00640 1 --2- 172.21.15.90:0/3733335757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55618ecf3db0 0x55618ecf4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.233+0000 7fe998c00640 1 -- 172.21.15.90:0/3733335757 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55618ebee510 con 0x55618ecc7930 2024-03-24T01:33:02.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.233+0000 7fe993e00640 1 --2- 172.21.15.90:0/2753560451 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe97006bbd0 0x7fe97006e090 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe984036000 tx=0x7fe984030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:33:02.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.233+0000 7fe998c00640 1 --2- 172.21.15.90:0/3733335757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55618ecc7930 0x55618ecf1560 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fe98c009e70 tx=0x7fe98c007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1a094e4e49eaa87 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:33:02.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.233+0000 7fe983e00640 1 -- 172.21.15.90:0/3733335757 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98c019070 con 0x55618ecc7930 2024-03-24T01:33:02.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.233+0000 7fe983e00640 1 -- 172.21.15.90:0/3733335757 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe98c012710 con 0x55618ecc7930 2024-03-24T01:33:02.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.234+0000 7fe983e00640 1 -- 172.21.15.90:0/3733335757 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98c012ae0 con 0x55618ecc7930 2024-03-24T01:33:02.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.234+0000 7fe99d9a3740 1 -- 172.21.15.90:0/3733335757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55618ecc7930 msgr2=0x55618ecf1560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:33:02.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.234+0000 7fe99d9a3740 1 --2- 172.21.15.90:0/3733335757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55618ecc7930 0x55618ecf1560 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fe98c009e70 tx=0x7fe98c007700 comp rx=0 tx=0).stop 2024-03-24T01:33:02.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.234+0000 7fe99d9a3740 1 -- 172.21.15.90:0/3733335757 shutdown_connections 2024-03-24T01:33:02.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.234+0000 7fe99d9a3740 1 --2- 172.21.15.90:0/3733335757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55618ecc7930 0x55618ecf1560 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.234+0000 7fe99d9a3740 1 --2- 172.21.15.90:0/3733335757 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55618ecc3dd0 0x55618ecf1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.234+0000 7fe99d9a3740 1 --2- 172.21.15.90:0/3733335757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55618ecf3db0 0x55618ecf4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.234+0000 7fe99d9a3740 1 -- 172.21.15.90:0/3733335757 >> 172.21.15.90:0/3733335757 conn(0x55618ecc6040 msgr2=0x55618ed0ebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:33:02.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.234+0000 7fe99d9a3740 1 -- 172.21.15.90:0/3733335757 shutdown_connections 2024-03-24T01:33:02.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.234+0000 7fe99d9a3740 1 -- 172.21.15.90:0/3733335757 wait complete. 2024-03-24T01:33:02.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.235+0000 7fe99d9a3740 1 -- start start 2024-03-24T01:33:02.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.235+0000 7fe99d9a3740 1 Processor -- start 2024-03-24T01:33:02.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.236+0000 7fe99d9a3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55618ecc7930 0x55618ecf7110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.236+0000 7fe99d9a3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55618ecf3db0 0x55618ecea7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.236+0000 7fe99d9a3740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55618ecf5230 0x55618ece94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.236+0000 7fe99d9a3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55618ecc8ae0 con 0x55618ecf3db0 2024-03-24T01:33:02.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.236+0000 7fe99d9a3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55618ebe9dd0 con 0x55618ecc7930 2024-03-24T01:33:02.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.236+0000 7fe99d9a3740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55618ecc7ed0 con 0x55618ecf5230 2024-03-24T01:33:02.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.236+0000 7fe99d9a3740 10 client.0 _ll_register_callbacks cb 0x55618e941ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T01:33:02.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.236+0000 7fe98be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55618ecf3db0 0x55618ecea7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.236+0000 7fe98be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55618ecf3db0 0x55618ecea7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54376/0 (socket says 172.21.15.90:54376) 2024-03-24T01:33:02.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.236+0000 7fe98be00640 1 -- 172.21.15.90:0/3189830769 learned_addr learned my addr 172.21.15.90:0/3189830769 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:33:02.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.237+0000 7fe98be00640 1 -- 172.21.15.90:0/3189830769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55618ecf5230 msgr2=0x55618ece94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:33:02.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.237+0000 7fe98be00640 1 --2- 172.21.15.90:0/3189830769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55618ecf5230 0x55618ece94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.237+0000 7fe98be00640 1 -- 172.21.15.90:0/3189830769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55618ecc7930 msgr2=0x55618ecf7110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:33:02.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.237+0000 7fe993e00640 1 --2- 172.21.15.90:0/3189830769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55618ecc7930 0x55618ecf7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.237+0000 7fe98be00640 1 --2- 172.21.15.90:0/3189830769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55618ecc7930 0x55618ecf7110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.237+0000 7fe98be00640 1 -- 172.21.15.90:0/3189830769 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55618e9422d0 con 0x55618ecf3db0 2024-03-24T01:33:02.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.237+0000 7fe993e00640 1 --2- 172.21.15.90:0/3189830769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55618ecc7930 0x55618ecf7110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:33:02.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.237+0000 7fe98be00640 1 --2- 172.21.15.90:0/3189830769 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55618ecf3db0 0x55618ecea7c0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fe97c089340 tx=0x7fe97c059aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=807cf79a16d5afdb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:33:02.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.237+0000 7fe980c00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-03-24T01:33:02.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.237+0000 7fe980c00640 1 -- 172.21.15.90:0/3189830769 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97c087060 con 0x55618ecf3db0 2024-03-24T01:33:02.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.237+0000 7fe980c00640 1 -- 172.21.15.90:0/3189830769 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7fe97c0558d0 con 0x55618ecf3db0 2024-03-24T01:33:02.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.237+0000 7fe99d9a3740 1 -- 172.21.15.90:0/3189830769 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe98c012710 con 0x55618ecf3db0 2024-03-24T01:33:02.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.237+0000 7fe980c00640 1 -- 172.21.15.90:0/3189830769 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe97c071170 con 0x55618ecf3db0 2024-03-24T01:33:02.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.237+0000 7fe99d9a3740 10 client.7323 fetch_fsmap learned FSMap version 53 2024-03-24T01:33:02.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.237+0000 7fe99d9a3740 1 -- 172.21.15.90:0/3189830769 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=2+}) v3 -- 0x55618ea0c6b0 con 0x55618ecf3db0 2024-03-24T01:33:02.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.238+0000 7fe980c00640 1 -- 172.21.15.90:0/3189830769 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe98c012710 con 0x55618ecf3db0 2024-03-24T01:33:02.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.238+0000 7fe980c00640 1 -- 172.21.15.90:0/3189830769 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 53) v1 ==== 3334+0+0 (secure 0 0 0) 0x7fe97c013070 con 0x55618ecf3db0 2024-03-24T01:33:02.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.238+0000 7fe99d9a3740 10 client.7323 fetch_fsmap finished waiting for FSMap version 53 2024-03-24T01:33:02.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.238+0000 7fe99d9a3740 10 client.7323 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T01:33:02.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.238+0000 7fe99d9a3740 20 client.7323 populate_metadata read hostname 'smithi090' 2024-03-24T01:33:02.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.238+0000 7fe99d9a3740 1 --2- 172.21.15.90:0/3189830769 >> [v2:172.21.15.90:6833/1760094626,v1:172.21.15.90:6834/1760094626] conn(0x55618ed22600 0x55618ed229e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.238+0000 7fe99d9a3740 4 client.7323 mds_command: new command op to 7098 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T01:33:02.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.238+0000 7fe99d9a3740 1 -- 172.21.15.90:0/3189830769 --> [v2:172.21.15.90:6833/1760094626,v1:172.21.15.90:6834/1760094626] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55618ebcf520 con 0x55618ed22600 2024-03-24T01:33:02.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.238+0000 7fe993e00640 1 --2- 172.21.15.90:0/3189830769 >> [v2:172.21.15.90:6833/1760094626,v1:172.21.15.90:6834/1760094626] conn(0x55618ed22600 0x55618ed229e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.239+0000 7fe993e00640 1 --2- 172.21.15.90:0/3189830769 >> [v2:172.21.15.90:6833/1760094626,v1:172.21.15.90:6834/1760094626] conn(0x55618ed22600 0x55618ed229e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b24a56d113eccf7c server_cookie=e64f5da8afe20a95 in_seq=0 out_seq=0 2024-03-24T01:33:02.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.239+0000 7fe980c00640 10 client.7323 ms_handle_connect on v2:172.21.15.90:6833/1760094626 2024-03-24T01:33:02.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.240+0000 7fe980c00640 1 -- 172.21.15.90:0/3189830769 <== mds.1 v2:172.21.15.90:6833/1760094626 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55618ebcf520 con 0x55618ed22600 2024-03-24T01:33:02.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.240+0000 7fe980c00640 10 client.7323 handle_command_reply: tid=0 2024-03-24T01:33:02.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.240+0000 7fe99d9a3740 1 client.7323 shutdown 2024-03-24T01:33:02.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.240+0000 7fe99d9a3740 1 -- 172.21.15.90:0/3189830769 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55618ecf3db0 msgr2=0x55618ecea7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:33:02.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.240+0000 7fe99d9a3740 1 --2- 172.21.15.90:0/3189830769 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55618ecf3db0 0x55618ecea7c0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fe97c089340 tx=0x7fe97c059aa0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.240+0000 7fe99d9a3740 1 -- 172.21.15.90:0/3189830769 shutdown_connections 2024-03-24T01:33:02.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.240+0000 7fe99d9a3740 1 --2- 172.21.15.90:0/3189830769 >> [v2:172.21.15.90:6833/1760094626,v1:172.21.15.90:6834/1760094626] conn(0x55618ed22600 0x55618ed229e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.240+0000 7fe99d9a3740 1 --2- 172.21.15.90:0/3189830769 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55618ecf3db0 0x55618ecea7c0 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.240+0000 7fe99d9a3740 1 --2- 172.21.15.90:0/3189830769 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55618ecf5230 0x55618ece94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.240+0000 7fe99d9a3740 1 --2- 172.21.15.90:0/3189830769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55618ecc7930 0x55618ecf7110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.240+0000 7fe99d9a3740 1 -- 172.21.15.90:0/3189830769 >> 172.21.15.90:0/3189830769 conn(0x55618ecc3dd0 msgr2=0x55618ed100d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:33:02.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.240+0000 7fe980c00640 0 client.7323 ms_handle_reset on v2:172.21.15.90:6833/1760094626 2024-03-24T01:33:02.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.240+0000 7fe99d9a3740 1 -- 172.21.15.90:0/3189830769 shutdown_connections 2024-03-24T01:33:02.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.240+0000 7fe99d9a3740 1 -- 172.21.15.90:0/3189830769 wait complete. 2024-03-24T01:33:02.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.241+0000 7fe99d9a3740 20 client.7323 trim_cache size 0 max 16384 2024-03-24T01:33:02.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.258+0000 7fe99d9a3740 1 Processor -- start 2024-03-24T01:33:02.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.259+0000 7fe99d9a3740 1 -- start start 2024-03-24T01:33:02.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.259+0000 7fe99d9a3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55618ecc7930 0x55618ecf6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.259+0000 7fe99d9a3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55618ecf3db0 0x55618ecf7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.259+0000 7fe99d9a3740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55618ecf5230 0x55618ed0e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.259+0000 7fe99d9a3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9942008c0 con 0x55618ecf3db0 2024-03-24T01:33:02.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.259+0000 7fe99d9a3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9941fed40 con 0x55618ecc7930 2024-03-24T01:33:02.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.259+0000 7fe99d9a3740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9941ff310 con 0x55618ecf5230 2024-03-24T01:33:02.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.260+0000 7fe98be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55618ecf3db0 0x55618ecf7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.260+0000 7fe98be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55618ecf3db0 0x55618ecf7220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54390/0 (socket says 172.21.15.90:54390) 2024-03-24T01:33:02.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.260+0000 7fe98be00640 1 -- 172.21.15.90:0/4060057812 learned_addr learned my addr 172.21.15.90:0/4060057812 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:33:02.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.260+0000 7fe98be00640 1 -- 172.21.15.90:0/4060057812 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55618ecf5230 msgr2=0x55618ed0e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:33:02.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.260+0000 7fe98be00640 1 --2- 172.21.15.90:0/4060057812 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55618ecf5230 0x55618ed0e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.260+0000 7fe98be00640 1 -- 172.21.15.90:0/4060057812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55618ecc7930 msgr2=0x55618ecf6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:33:02.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.260+0000 7fe98be00640 1 --2- 172.21.15.90:0/4060057812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55618ecc7930 0x55618ecf6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.260+0000 7fe98be00640 1 -- 172.21.15.90:0/4060057812 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55618ecf4500 con 0x55618ecf3db0 2024-03-24T01:33:02.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.260+0000 7fe98be00640 1 --2- 172.21.15.90:0/4060057812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55618ecf3db0 0x55618ecf7220 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fe97c059b10 tx=0x7fe97c06f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7050c9ac4e8f4cd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:33:02.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.261+0000 7fe983400640 1 -- 172.21.15.90:0/4060057812 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97c07c070 con 0x55618ecf3db0 2024-03-24T01:33:02.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.261+0000 7fe983400640 1 -- 172.21.15.90:0/4060057812 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe97c071170 con 0x55618ecf3db0 2024-03-24T01:33:02.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.261+0000 7fe983400640 1 -- 172.21.15.90:0/4060057812 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97c061970 con 0x55618ecf3db0 2024-03-24T01:33:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.261+0000 7fe99d9a3740 1 -- 172.21.15.90:0/4060057812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55618ecf3db0 msgr2=0x55618ecf7220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:33:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.261+0000 7fe99d9a3740 1 --2- 172.21.15.90:0/4060057812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55618ecf3db0 0x55618ecf7220 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fe97c059b10 tx=0x7fe97c06f000 comp rx=0 tx=0).stop 2024-03-24T01:33:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.261+0000 7fe99d9a3740 1 -- 172.21.15.90:0/4060057812 shutdown_connections 2024-03-24T01:33:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.261+0000 7fe99d9a3740 1 --2- 172.21.15.90:0/4060057812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55618ecf3db0 0x55618ecf7220 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.261+0000 7fe99d9a3740 1 --2- 172.21.15.90:0/4060057812 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55618ecf5230 0x55618ed0e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.261+0000 7fe99d9a3740 1 --2- 172.21.15.90:0/4060057812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55618ecc7930 0x55618ecf6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.261+0000 7fe99d9a3740 1 -- 172.21.15.90:0/4060057812 >> 172.21.15.90:0/4060057812 conn(0x55618ecc3dd0 msgr2=0x55618ece9370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:33:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.261+0000 7fe99d9a3740 1 -- 172.21.15.90:0/4060057812 shutdown_connections 2024-03-24T01:33:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.261+0000 7fe99d9a3740 1 -- 172.21.15.90:0/4060057812 wait complete. 2024-03-24T01:33:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.262+0000 7fe99d9a3740 1 -- start start 2024-03-24T01:33:02.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.262+0000 7fe99d9a3740 1 Processor -- start 2024-03-24T01:33:02.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.263+0000 7fe99d9a3740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55618ecc7930 0x55618eced5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.263+0000 7fe99d9a3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55618ecf3db0 0x55618ecea540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.263+0000 7fe99d9a3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55618ecf5230 0x55618eceaa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.263+0000 7fe99d9a3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x55618ecec830 con 0x55618ecf5230 2024-03-24T01:33:02.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.263+0000 7fe99d9a3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55618ecf5670 con 0x55618ecf3db0 2024-03-24T01:33:02.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.263+0000 7fe99d9a3740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x55618ecee4e0 con 0x55618ecc7930 2024-03-24T01:33:02.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.263+0000 7fe99d9a3740 10 client.0 _ll_register_callbacks cb 0x7fe97c00b1e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T01:33:02.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.264+0000 7fe998c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55618ecf5230 0x55618eceaa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.264+0000 7fe998c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55618ecf5230 0x55618eceaa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54406/0 (socket says 172.21.15.90:54406) 2024-03-24T01:33:02.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.264+0000 7fe998c00640 1 -- 172.21.15.90:0/1764055607 learned_addr learned my addr 172.21.15.90:0/1764055607 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:33:02.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.264+0000 7fe993e00640 1 --2- 172.21.15.90:0/1764055607 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55618ecc7930 0x55618eced5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.264+0000 7fe998c00640 1 -- 172.21.15.90:0/1764055607 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55618ecc7930 msgr2=0x55618eced5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:33:02.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.264+0000 7fe98be00640 1 --2- 172.21.15.90:0/1764055607 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55618ecf3db0 0x55618ecea540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.264+0000 7fe998c00640 1 --2- 172.21.15.90:0/1764055607 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55618ecc7930 0x55618eced5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.264+0000 7fe998c00640 1 -- 172.21.15.90:0/1764055607 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55618ecf3db0 msgr2=0x55618ecea540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:33:02.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.264+0000 7fe998c00640 1 --2- 172.21.15.90:0/1764055607 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55618ecf3db0 0x55618ecea540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.264+0000 7fe998c00640 1 -- 172.21.15.90:0/1764055607 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55618ed780a0 con 0x55618ecf5230 2024-03-24T01:33:02.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.264+0000 7fe993e00640 1 --2- 172.21.15.90:0/1764055607 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55618ecc7930 0x55618eced5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:33:02.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.264+0000 7fe998c00640 1 --2- 172.21.15.90:0/1764055607 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55618ecf5230 0x55618eceaa80 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fe98c017040 tx=0x7fe98c012b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f438520d6a62953 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:33:02.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.264+0000 7fe980c00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-03-24T01:33:02.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.265+0000 7fe980c00640 1 -- 172.21.15.90:0/1764055607 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98c022020 con 0x55618ecf5230 2024-03-24T01:33:02.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.265+0000 7fe99d9a3740 1 -- 172.21.15.90:0/1764055607 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe97c071170 con 0x55618ecf5230 2024-03-24T01:33:02.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.265+0000 7fe980c00640 1 -- 172.21.15.90:0/1764055607 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7fe98c012c90 con 0x55618ecf5230 2024-03-24T01:33:02.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.265+0000 7fe980c00640 1 -- 172.21.15.90:0/1764055607 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe98c02c050 con 0x55618ecf5230 2024-03-24T01:33:02.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.265+0000 7fe99d9a3740 10 client.7329 fetch_fsmap learned FSMap version 53 2024-03-24T01:33:02.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.265+0000 7fe99d9a3740 1 -- 172.21.15.90:0/1764055607 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=2+}) v3 -- 0x55618ed120d0 con 0x55618ecf5230 2024-03-24T01:33:02.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.265+0000 7fe980c00640 1 -- 172.21.15.90:0/1764055607 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe97c071170 con 0x55618ecf5230 2024-03-24T01:33:02.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.265+0000 7fe980c00640 1 -- 172.21.15.90:0/1764055607 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 53) v1 ==== 3334+0+0 (secure 0 0 0) 0x7fe98c003990 con 0x55618ecf5230 2024-03-24T01:33:02.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.265+0000 7fe99d9a3740 10 client.7329 fetch_fsmap finished waiting for FSMap version 53 2024-03-24T01:33:02.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.265+0000 7fe99d9a3740 10 client.7329 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T01:33:02.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.265+0000 7fe99d9a3740 20 client.7329 populate_metadata read hostname 'smithi090' 2024-03-24T01:33:02.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.265+0000 7fe99d9a3740 1 --2- 172.21.15.90:0/1764055607 >> [v2:172.21.15.90:6833/1760094626,v1:172.21.15.90:6834/1760094626] conn(0x55618ed760f0 0x55618ecf57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.265+0000 7fe99d9a3740 4 client.7329 mds_command: new command op to 7098 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T01:33:02.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.265+0000 7fe99d9a3740 1 -- 172.21.15.90:0/1764055607 --> [v2:172.21.15.90:6833/1760094626,v1:172.21.15.90:6834/1760094626] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55618ecebd40 con 0x55618ed760f0 2024-03-24T01:33:02.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.266+0000 7fe993e00640 1 --2- 172.21.15.90:0/1764055607 >> [v2:172.21.15.90:6833/1760094626,v1:172.21.15.90:6834/1760094626] conn(0x55618ed760f0 0x55618ecf57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.266+0000 7fe993e00640 1 --2- 172.21.15.90:0/1764055607 >> [v2:172.21.15.90:6833/1760094626,v1:172.21.15.90:6834/1760094626] conn(0x55618ed760f0 0x55618ecf57f0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cc554eb1b7f0f5e4 server_cookie=b19f53355fa5315a in_seq=0 out_seq=0 2024-03-24T01:33:02.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.266+0000 7fe980c00640 10 client.7329 ms_handle_connect on v2:172.21.15.90:6833/1760094626 2024-03-24T01:33:02.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.266+0000 7fe980c00640 1 -- 172.21.15.90:0/1764055607 <== mds.1 v2:172.21.15.90:6833/1760094626 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55618ecebd40 con 0x55618ed760f0 2024-03-24T01:33:02.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.266+0000 7fe980c00640 10 client.7329 handle_command_reply: tid=0 2024-03-24T01:33:02.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.266+0000 7fe99d9a3740 1 client.7329 shutdown 2024-03-24T01:33:02.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.267+0000 7fe99d9a3740 1 -- 172.21.15.90:0/1764055607 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55618ecf5230 msgr2=0x55618eceaa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:33:02.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.267+0000 7fe99d9a3740 1 --2- 172.21.15.90:0/1764055607 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55618ecf5230 0x55618eceaa80 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fe98c017040 tx=0x7fe98c012b20 comp rx=0 tx=0).stop 2024-03-24T01:33:02.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.267+0000 7fe99d9a3740 1 -- 172.21.15.90:0/1764055607 shutdown_connections 2024-03-24T01:33:02.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.267+0000 7fe99d9a3740 1 --2- 172.21.15.90:0/1764055607 >> [v2:172.21.15.90:6833/1760094626,v1:172.21.15.90:6834/1760094626] conn(0x55618ed760f0 0x55618ecf57f0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.267+0000 7fe99d9a3740 1 --2- 172.21.15.90:0/1764055607 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55618ecf5230 0x55618eceaa80 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.267+0000 7fe99d9a3740 1 --2- 172.21.15.90:0/1764055607 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55618ecf3db0 0x55618ecea540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.267+0000 7fe99d9a3740 1 --2- 172.21.15.90:0/1764055607 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x55618ecc7930 0x55618eced5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.267+0000 7fe99d9a3740 1 -- 172.21.15.90:0/1764055607 >> 172.21.15.90:0/1764055607 conn(0x55618ecc3dd0 msgr2=0x55618ecf6ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:33:02.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.267+0000 7fe980c00640 0 client.7329 ms_handle_reset on v2:172.21.15.90:6833/1760094626 2024-03-24T01:33:02.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.267+0000 7fe99d9a3740 1 -- 172.21.15.90:0/1764055607 shutdown_connections 2024-03-24T01:33:02.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.267+0000 7fe99d9a3740 1 -- 172.21.15.90:0/1764055607 wait complete. 2024-03-24T01:33:02.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.267+0000 7fe99d9a3740 20 client.7329 trim_cache size 0 max 16384 2024-03-24T01:33:02.273 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-24T01:33:02.273 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "24712165-9839-4fcc-8033-988386efea74", 2024-03-24T01:33:02.273 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 1, 2024-03-24T01:33:02.274 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 7098, 2024-03-24T01:33:02.274 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-03-24T01:33:02.274 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-03-24T01:33:02.274 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-03-24T01:33:02.274 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 3.0160254580000001, 2024-03-24T01:33:02.274 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 53, 2024-03-24T01:33:02.274 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 69, 2024-03-24T01:33:02.274 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 69, 2024-03-24T01:33:02.274 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 22.999271659000001 2024-03-24T01:33:02.275 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-24T01:33:02.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.268+0000 7fe999600640 1 -- 172.21.15.90:0/2753560451 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe97006bbd0 msgr2=0x7fe97006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:33:02.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.268+0000 7fe999600640 1 --2- 172.21.15.90:0/2753560451 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe97006bbd0 0x7fe97006e090 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe984036000 tx=0x7fe984030000 comp rx=0 tx=0).stop 2024-03-24T01:33:02.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.268+0000 7fe999600640 1 -- 172.21.15.90:0/2753560451 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9940d35b0 msgr2=0x7fe99415f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:33:02.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.268+0000 7fe999600640 1 --2- 172.21.15.90:0/2753560451 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9940d35b0 0x7fe99415f820 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fe97c00d5f0 tx=0x7fe97c00dab0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.268+0000 7fe999600640 1 -- 172.21.15.90:0/2753560451 shutdown_connections 2024-03-24T01:33:02.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.268+0000 7fe999600640 1 --2- 172.21.15.90:0/2753560451 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe97006bbd0 0x7fe97006e090 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.268+0000 7fe999600640 1 --2- 172.21.15.90:0/2753560451 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9940d35b0 0x7fe99415f820 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.268+0000 7fe999600640 1 --2- 172.21.15.90:0/2753560451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe99415de50 0x7fe99415fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.268+0000 7fe999600640 1 --2- 172.21.15.90:0/2753560451 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe9940b5550 0x7fe99414c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.268+0000 7fe999600640 1 -- 172.21.15.90:0/2753560451 >> 172.21.15.90:0/2753560451 conn(0x7fe9940d0550 msgr2=0x7fe9940b8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:33:02.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.268+0000 7fe999600640 1 -- 172.21.15.90:0/2753560451 shutdown_connections 2024-03-24T01:33:02.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.268+0000 7fe999600640 1 -- 172.21.15.90:0/2753560451 wait complete. 2024-03-24T01:33:02.287 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-24T01:33:02.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.479+0000 7f8dfd400640 1 Processor -- start 2024-03-24T01:33:02.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.479+0000 7f8dfd400640 1 -- start start 2024-03-24T01:33:02.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.481+0000 7f8dfd400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8df80b5550 0x7f8df80d1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.481+0000 7f8dfd400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8df815de50 0x7f8df80b9540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.481+0000 7f8dfd400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f8df80d35b0 0x7f8df80b5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.481+0000 7f8dfd400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8df80b9a80 con 0x7f8df80b5550 2024-03-24T01:33:02.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.481+0000 7f8dfd400640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8df80b5e70 con 0x7f8df80d35b0 2024-03-24T01:33:02.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.481+0000 7f8dfd400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8df80d2510 con 0x7f8df815de50 2024-03-24T01:33:02.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.481+0000 7f8df7e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8df80b5550 0x7f8df80d1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.481+0000 7f8df7e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8df80b5550 0x7f8df80d1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54418/0 (socket says 172.21.15.90:54418) 2024-03-24T01:33:02.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.481+0000 7f8df7e00640 1 -- 172.21.15.90:0/3431700493 learned_addr learned my addr 172.21.15.90:0/3431700493 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:33:02.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.481+0000 7f8df7400640 1 --2- 172.21.15.90:0/3431700493 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8df815de50 0x7f8df80b9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.481+0000 7f8df7e00640 1 -- 172.21.15.90:0/3431700493 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8df815de50 msgr2=0x7f8df80b9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:33:02.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.481+0000 7f8df7e00640 1 --2- 172.21.15.90:0/3431700493 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8df815de50 0x7f8df80b9540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.481+0000 7f8df7e00640 1 -- 172.21.15.90:0/3431700493 >> v2:172.21.15.120:3300/0 conn(0x7f8df80d35b0 msgr2=0x7f8df80b5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:33:02.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.481+0000 7f8dfca00640 1 --2- 172.21.15.90:0/3431700493 >> v2:172.21.15.120:3300/0 conn(0x7f8df80d35b0 0x7f8df80b5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.481+0000 7f8df7e00640 1 --2- 172.21.15.90:0/3431700493 >> v2:172.21.15.120:3300/0 conn(0x7f8df80d35b0 0x7f8df80b5930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.481+0000 7f8df7e00640 1 -- 172.21.15.90:0/3431700493 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df80d27f0 con 0x7f8df80b5550 2024-03-24T01:33:02.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.483+0000 7f8df7e00640 1 --2- 172.21.15.90:0/3431700493 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8df80b5550 0x7f8df80d1ea0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f8df808c2b0 tx=0x7f8de802b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77678968ff403613 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:33:02.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.483+0000 7f8df6a00640 1 -- 172.21.15.90:0/3431700493 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de80317d0 con 0x7f8df80b5550 2024-03-24T01:33:02.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.483+0000 7f8df6a00640 1 -- 172.21.15.90:0/3431700493 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de8037070 con 0x7f8df80b5550 2024-03-24T01:33:02.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.483+0000 7f8df6a00640 1 -- 172.21.15.90:0/3431700493 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de8031b90 con 0x7f8df80b5550 2024-03-24T01:33:02.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.483+0000 7f8dfd400640 1 -- 172.21.15.90:0/3431700493 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8df80b5550 msgr2=0x7f8df80d1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:33:02.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.483+0000 7f8dfd400640 1 --2- 172.21.15.90:0/3431700493 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8df80b5550 0x7f8df80d1ea0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f8df808c2b0 tx=0x7f8de802b870 comp rx=0 tx=0).stop 2024-03-24T01:33:02.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.484+0000 7f8dfd400640 1 -- 172.21.15.90:0/3431700493 shutdown_connections 2024-03-24T01:33:02.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.484+0000 7f8dfd400640 1 --2- 172.21.15.90:0/3431700493 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8df815de50 0x7f8df80b9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.484+0000 7f8dfd400640 1 --2- 172.21.15.90:0/3431700493 >> v2:172.21.15.120:3300/0 conn(0x7f8df80d35b0 0x7f8df80b5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.484+0000 7f8dfd400640 1 --2- 172.21.15.90:0/3431700493 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8df80b5550 0x7f8df80d1ea0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.484+0000 7f8dfd400640 1 -- 172.21.15.90:0/3431700493 >> 172.21.15.90:0/3431700493 conn(0x7f8df80d0550 msgr2=0x7f8df80d0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:33:02.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.484+0000 7f8dfd400640 1 -- 172.21.15.90:0/3431700493 shutdown_connections 2024-03-24T01:33:02.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.484+0000 7f8dfd400640 1 -- 172.21.15.90:0/3431700493 wait complete. 2024-03-24T01:33:02.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.485+0000 7f8dfd400640 1 Processor -- start 2024-03-24T01:33:02.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.485+0000 7f8dfd400640 1 -- start start 2024-03-24T01:33:02.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.486+0000 7f8dfd400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8df80b5550 0x7f8df81f3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.486+0000 7f8dfd400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8df80d35b0 0x7f8df80cc190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.486+0000 7f8dfd400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8df815de50 0x7f8df80cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.486+0000 7f8dfd400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df80d3090 con 0x7f8df80d35b0 2024-03-24T01:33:02.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.486+0000 7f8dfd400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df80d2db0 con 0x7f8df815de50 2024-03-24T01:33:02.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.486+0000 7f8dfd400640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8df806c5d0 con 0x7f8df80b5550 2024-03-24T01:33:02.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.487+0000 7f8df7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8df80d35b0 0x7f8df80cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.487+0000 7f8df7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8df80d35b0 0x7f8df80cc190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54422/0 (socket says 172.21.15.90:54422) 2024-03-24T01:33:02.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.487+0000 7f8df7400640 1 -- 172.21.15.90:0/2531687215 learned_addr learned my addr 172.21.15.90:0/2531687215 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:33:02.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.487+0000 7f8df7e00640 1 --2- 172.21.15.90:0/2531687215 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8df80b5550 0x7f8df81f3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.487+0000 7f8dfca00640 1 --2- 172.21.15.90:0/2531687215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8df815de50 0x7f8df80cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.487+0000 7f8df7400640 1 -- 172.21.15.90:0/2531687215 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8df80b5550 msgr2=0x7f8df81f3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:33:02.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.487+0000 7f8df7400640 1 --2- 172.21.15.90:0/2531687215 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8df80b5550 0x7f8df81f3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.487+0000 7f8df7400640 1 -- 172.21.15.90:0/2531687215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8df815de50 msgr2=0x7f8df80cc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:33:02.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.487+0000 7f8df7400640 1 --2- 172.21.15.90:0/2531687215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8df815de50 0x7f8df80cc6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.487+0000 7f8df7400640 1 -- 172.21.15.90:0/2531687215 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df81f3cc0 con 0x7f8df80d35b0 2024-03-24T01:33:02.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.487+0000 7f8df7400640 1 --2- 172.21.15.90:0/2531687215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8df80d35b0 0x7f8df80cc190 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f8de400c820 tx=0x7f8de400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:33:02.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.488+0000 7f8defe00640 1 -- 172.21.15.90:0/2531687215 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de4015070 con 0x7f8df80d35b0 2024-03-24T01:33:02.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.488+0000 7f8defe00640 1 -- 172.21.15.90:0/2531687215 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de4004020 con 0x7f8df80d35b0 2024-03-24T01:33:02.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.488+0000 7f8dfd400640 1 -- 172.21.15.90:0/2531687215 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8df8150da0 con 0x7f8df80d35b0 2024-03-24T01:33:02.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.488+0000 7f8defe00640 1 -- 172.21.15.90:0/2531687215 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de40043f0 con 0x7f8df80d35b0 2024-03-24T01:33:02.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.488+0000 7f8dfd400640 1 -- 172.21.15.90:0/2531687215 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8df81507e0 con 0x7f8df80d35b0 2024-03-24T01:33:02.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.491+0000 7f8defe00640 1 -- 172.21.15.90:0/2531687215 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8de4007830 con 0x7f8df80d35b0 2024-03-24T01:33:02.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.492+0000 7f8defe00640 1 --2- 172.21.15.90:0/2531687215 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8dd806bbd0 0x7f8dd806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.492+0000 7f8defe00640 1 -- 172.21.15.90:0/2531687215 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7f8de408aa80 con 0x7f8df80d35b0 2024-03-24T01:33:02.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.492+0000 7f8e0176a740 1 Processor -- start 2024-03-24T01:33:02.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.492+0000 7f8e0176a740 1 -- start start 2024-03-24T01:33:02.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.492+0000 7f8df7e00640 1 --2- 172.21.15.90:0/2531687215 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8dd806bbd0 0x7f8dd806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.493+0000 7f8e0176a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618ba04cdb0 0x5618ba04d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.493+0000 7f8e0176a740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5618ba01cdd0 0x5618ba04a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.493+0000 7f8e0176a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ba020930 0x5618ba04a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.493+0000 7f8e0176a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618b9f86c90 con 0x5618ba04cdb0 2024-03-24T01:33:02.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.493+0000 7f8e0176a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618b9f2f980 con 0x5618ba020930 2024-03-24T01:33:02.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.493+0000 7f8e0176a740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5618b9f82d30 con 0x5618ba01cdd0 2024-03-24T01:33:02.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.493+0000 7f8df7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618ba04cdb0 0x5618ba04d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.493+0000 7f8df7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618ba04cdb0 0x5618ba04d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54432/0 (socket says 172.21.15.90:54432) 2024-03-24T01:33:02.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.493+0000 7f8df7e00640 1 -- 172.21.15.90:0/1097839229 learned_addr learned my addr 172.21.15.90:0/1097839229 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:33:02.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.493+0000 7f8dfca00640 1 --2- 172.21.15.90:0/1097839229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ba020930 0x5618ba04a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.494+0000 7f8df7400640 1 --2- 172.21.15.90:0/1097839229 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5618ba01cdd0 0x5618ba04a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.494+0000 7f8df7e00640 1 -- 172.21.15.90:0/1097839229 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5618ba01cdd0 msgr2=0x5618ba04a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:33:02.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.494+0000 7f8df7e00640 1 --2- 172.21.15.90:0/1097839229 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5618ba01cdd0 0x5618ba04a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.494+0000 7f8df7e00640 1 -- 172.21.15.90:0/1097839229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ba020930 msgr2=0x5618ba04a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:33:02.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.494+0000 7f8df7e00640 1 --2- 172.21.15.90:0/1097839229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ba020930 0x5618ba04a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.494+0000 7f8df7e00640 1 -- 172.21.15.90:0/1097839229 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5618b9f47510 con 0x5618ba04cdb0 2024-03-24T01:33:02.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.494+0000 7f8dfca00640 1 --2- 172.21.15.90:0/1097839229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ba020930 0x5618ba04a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:33:02.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.494+0000 7f8df7e00640 1 --2- 172.21.15.90:0/2531687215 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8dd806bbd0 0x7f8dd806e090 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f8de8036000 tx=0x7f8de803d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:33:02.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.494+0000 7f8df7e00640 1 --2- 172.21.15.90:0/1097839229 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618ba04cdb0 0x5618ba04d190 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f8de8000c00 tx=0x7f8de8031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=892f1fcac4e72ef server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:33:02.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.494+0000 7f8de3e00640 1 -- 172.21.15.90:0/1097839229 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de804c070 con 0x5618ba04cdb0 2024-03-24T01:33:02.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.494+0000 7f8de3e00640 1 -- 172.21.15.90:0/1097839229 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de8037040 con 0x5618ba04cdb0 2024-03-24T01:33:02.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.494+0000 7f8de3e00640 1 -- 172.21.15.90:0/1097839229 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de8048a10 con 0x5618ba04cdb0 2024-03-24T01:33:02.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.495+0000 7f8e0176a740 1 -- 172.21.15.90:0/1097839229 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618ba04cdb0 msgr2=0x5618ba04d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:33:02.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.495+0000 7f8e0176a740 1 --2- 172.21.15.90:0/1097839229 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618ba04cdb0 0x5618ba04d190 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f8de8000c00 tx=0x7f8de8031000 comp rx=0 tx=0).stop 2024-03-24T01:33:02.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.495+0000 7f8e0176a740 1 -- 172.21.15.90:0/1097839229 shutdown_connections 2024-03-24T01:33:02.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.495+0000 7f8e0176a740 1 --2- 172.21.15.90:0/1097839229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ba020930 0x5618ba04a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.495+0000 7f8e0176a740 1 --2- 172.21.15.90:0/1097839229 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5618ba01cdd0 0x5618ba04a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.495+0000 7f8e0176a740 1 --2- 172.21.15.90:0/1097839229 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618ba04cdb0 0x5618ba04d190 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.495+0000 7f8e0176a740 1 -- 172.21.15.90:0/1097839229 >> 172.21.15.90:0/1097839229 conn(0x5618ba01f040 msgr2=0x5618ba067bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:33:02.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.495+0000 7f8e0176a740 1 -- 172.21.15.90:0/1097839229 shutdown_connections 2024-03-24T01:33:02.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.495+0000 7f8e0176a740 1 -- 172.21.15.90:0/1097839229 wait complete. 2024-03-24T01:33:02.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.496+0000 7f8e0176a740 1 -- start start 2024-03-24T01:33:02.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.496+0000 7f8e0176a740 1 Processor -- start 2024-03-24T01:33:02.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.497+0000 7f8e0176a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618ba020930 0x5618ba078b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.497+0000 7f8e0176a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ba04cdb0 0x5618ba04e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.497+0000 7f8e0176a740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5618ba0780d0 0x5618ba053900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.497+0000 7f8e0176a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618ba021ae0 con 0x5618ba020930 2024-03-24T01:33:02.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.497+0000 7f8e0176a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618b9f42dd0 con 0x5618ba04cdb0 2024-03-24T01:33:02.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.497+0000 7f8e0176a740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5618ba020ed0 con 0x5618ba0780d0 2024-03-24T01:33:02.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.497+0000 7f8e0176a740 10 client.0 _ll_register_callbacks cb 0x5618b9c9aac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T01:33:02.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.497+0000 7f8df7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618ba020930 0x5618ba078b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.497+0000 7f8df7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618ba020930 0x5618ba078b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54442/0 (socket says 172.21.15.90:54442) 2024-03-24T01:33:02.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.497+0000 7f8df7e00640 1 -- 172.21.15.90:0/1698313065 learned_addr learned my addr 172.21.15.90:0/1698313065 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:33:02.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.497+0000 7f8dfca00640 1 --2- 172.21.15.90:0/1698313065 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5618ba0780d0 0x5618ba053900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.498+0000 7f8df7400640 1 --2- 172.21.15.90:0/1698313065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ba04cdb0 0x5618ba04e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.498+0000 7f8df7e00640 1 -- 172.21.15.90:0/1698313065 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5618ba0780d0 msgr2=0x5618ba053900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:33:02.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.498+0000 7f8df7e00640 1 --2- 172.21.15.90:0/1698313065 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5618ba0780d0 0x5618ba053900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.498+0000 7f8df7e00640 1 -- 172.21.15.90:0/1698313065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ba04cdb0 msgr2=0x5618ba04e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:33:02.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.498+0000 7f8df7e00640 1 --2- 172.21.15.90:0/1698313065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ba04cdb0 0x5618ba04e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.498+0000 7f8df7e00640 1 -- 172.21.15.90:0/1698313065 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5618b9c9b2d0 con 0x5618ba020930 2024-03-24T01:33:02.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.498+0000 7f8dfca00640 1 --2- 172.21.15.90:0/1698313065 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5618ba0780d0 0x5618ba053900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:33:02.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.498+0000 7f8df7e00640 1 --2- 172.21.15.90:0/1698313065 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618ba020930 0x5618ba078b00 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f8de8000c00 tx=0x7f8de8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c912569b81a1fa30 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:33:02.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.498+0000 7f8de0c00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-03-24T01:33:02.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.498+0000 7f8de0c00640 1 -- 172.21.15.90:0/1698313065 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de8058070 con 0x5618ba020930 2024-03-24T01:33:02.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.498+0000 7f8de0c00640 1 -- 172.21.15.90:0/1698313065 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7f8de8032870 con 0x5618ba020930 2024-03-24T01:33:02.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.498+0000 7f8e0176a740 1 -- 172.21.15.90:0/1698313065 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8de8037040 con 0x5618ba020930 2024-03-24T01:33:02.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.498+0000 7f8de0c00640 1 -- 172.21.15.90:0/1698313065 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8de804c030 con 0x5618ba020930 2024-03-24T01:33:02.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.498+0000 7f8e0176a740 10 client.7341 fetch_fsmap learned FSMap version 53 2024-03-24T01:33:02.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.498+0000 7f8e0176a740 1 -- 172.21.15.90:0/1698313065 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=2+}) v3 -- 0x5618b9d656b0 con 0x5618ba020930 2024-03-24T01:33:02.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.499+0000 7f8de0c00640 1 -- 172.21.15.90:0/1698313065 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de8037040 con 0x5618ba020930 2024-03-24T01:33:02.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.499+0000 7f8de0c00640 1 -- 172.21.15.90:0/1698313065 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 53) v1 ==== 3334+0+0 (secure 0 0 0) 0x7f8de8066080 con 0x5618ba020930 2024-03-24T01:33:02.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.499+0000 7f8e0176a740 10 client.7341 fetch_fsmap finished waiting for FSMap version 53 2024-03-24T01:33:02.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.499+0000 7f8e0176a740 10 client.7341 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-24T01:33:02.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.499+0000 7f8e0176a740 20 client.7341 populate_metadata read hostname 'smithi090' 2024-03-24T01:33:02.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.499+0000 7f8e0176a740 1 --2- 172.21.15.90:0/1698313065 >> [v2:172.21.15.120:6834/910380191,v1:172.21.15.120:6835/910380191] conn(0x5618ba0514f0 0x5618ba0518d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.499+0000 7f8e0176a740 4 client.7341 mds_command: new command op to 7110 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T01:33:02.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.499+0000 7f8e0176a740 1 -- 172.21.15.90:0/1698313065 --> [v2:172.21.15.120:6834/910380191,v1:172.21.15.120:6835/910380191] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5618b9f28520 con 0x5618ba0514f0 2024-03-24T01:33:02.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.500+0000 7f8df7400640 1 --2- 172.21.15.90:0/1698313065 >> [v2:172.21.15.120:6834/910380191,v1:172.21.15.120:6835/910380191] conn(0x5618ba0514f0 0x5618ba0518d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.500+0000 7f8df7400640 1 --2- 172.21.15.90:0/1698313065 >> [v2:172.21.15.120:6834/910380191,v1:172.21.15.120:6835/910380191] conn(0x5618ba0514f0 0x5618ba0518d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f90d3bbac367c6a8 server_cookie=efdd0d7073ca3c32 in_seq=0 out_seq=0 2024-03-24T01:33:02.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.500+0000 7f8de0c00640 10 client.7341 ms_handle_connect on v2:172.21.15.120:6834/910380191 2024-03-24T01:33:02.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.502+0000 7f8de0c00640 1 -- 172.21.15.90:0/1698313065 <== mds.0 v2:172.21.15.120:6834/910380191 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5618b9f28520 con 0x5618ba0514f0 2024-03-24T01:33:02.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.502+0000 7f8de0c00640 10 client.7341 handle_command_reply: tid=0 2024-03-24T01:33:02.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.502+0000 7f8e0176a740 1 client.7341 shutdown 2024-03-24T01:33:02.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.502+0000 7f8e0176a740 1 -- 172.21.15.90:0/1698313065 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618ba020930 msgr2=0x5618ba078b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:33:02.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.502+0000 7f8e0176a740 1 --2- 172.21.15.90:0/1698313065 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618ba020930 0x5618ba078b00 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f8de8000c00 tx=0x7f8de8004730 comp rx=0 tx=0).stop 2024-03-24T01:33:02.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.503+0000 7f8e0176a740 1 -- 172.21.15.90:0/1698313065 shutdown_connections 2024-03-24T01:33:02.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.503+0000 7f8e0176a740 1 --2- 172.21.15.90:0/1698313065 >> [v2:172.21.15.120:6834/910380191,v1:172.21.15.120:6835/910380191] conn(0x5618ba0514f0 0x5618ba0518d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.503+0000 7f8e0176a740 1 --2- 172.21.15.90:0/1698313065 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5618ba0780d0 0x5618ba053900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.503+0000 7f8e0176a740 1 --2- 172.21.15.90:0/1698313065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ba04cdb0 0x5618ba04e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.503+0000 7f8e0176a740 1 --2- 172.21.15.90:0/1698313065 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618ba020930 0x5618ba078b00 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.503+0000 7f8e0176a740 1 -- 172.21.15.90:0/1698313065 >> 172.21.15.90:0/1698313065 conn(0x5618ba01cdd0 msgr2=0x5618ba06b0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:33:02.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.503+0000 7f8de0c00640 0 client.7341 ms_handle_reset on v2:172.21.15.120:6834/910380191 2024-03-24T01:33:02.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.503+0000 7f8e0176a740 1 -- 172.21.15.90:0/1698313065 shutdown_connections 2024-03-24T01:33:02.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.503+0000 7f8e0176a740 1 -- 172.21.15.90:0/1698313065 wait complete. 2024-03-24T01:33:02.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.503+0000 7f8e0176a740 20 client.7341 trim_cache size 0 max 16384 2024-03-24T01:33:02.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.521+0000 7f8e0176a740 1 Processor -- start 2024-03-24T01:33:02.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.521+0000 7f8e0176a740 1 -- start start 2024-03-24T01:33:02.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.522+0000 7f8e0176a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618ba020930 0x5618ba0181b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.522+0000 7f8e0176a740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5618ba04cdb0 0x5618ba07ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.522+0000 7f8e0176a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ba0780d0 0x5618ba021020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.522+0000 7f8e0176a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df82006b0 con 0x5618ba020930 2024-03-24T01:33:02.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.522+0000 7f8e0176a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df813f7d0 con 0x5618ba0780d0 2024-03-24T01:33:02.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.522+0000 7f8e0176a740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8df813fdd0 con 0x5618ba04cdb0 2024-03-24T01:33:02.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.522+0000 7f8df7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618ba020930 0x5618ba0181b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.522+0000 7f8df7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618ba020930 0x5618ba0181b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54448/0 (socket says 172.21.15.90:54448) 2024-03-24T01:33:02.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.522+0000 7f8df7e00640 1 -- 172.21.15.90:0/239028224 learned_addr learned my addr 172.21.15.90:0/239028224 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:33:02.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.522+0000 7f8df7e00640 1 -- 172.21.15.90:0/239028224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5618ba04cdb0 msgr2=0x5618ba07ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:33:02.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.522+0000 7f8df7e00640 1 --2- 172.21.15.90:0/239028224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5618ba04cdb0 0x5618ba07ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.522+0000 7f8df7e00640 1 -- 172.21.15.90:0/239028224 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ba0780d0 msgr2=0x5618ba021020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:33:02.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.522+0000 7f8df7e00640 1 --2- 172.21.15.90:0/239028224 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ba0780d0 0x5618ba021020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.522+0000 7f8df7e00640 1 -- 172.21.15.90:0/239028224 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5618ba051110 con 0x5618ba020930 2024-03-24T01:33:02.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.523+0000 7f8df7e00640 1 --2- 172.21.15.90:0/239028224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618ba020930 0x5618ba0181b0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f8de8006da0 tx=0x7f8de8053040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=565034727e071fbb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:33:02.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.523+0000 7f8de3400640 1 -- 172.21.15.90:0/239028224 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de8058040 con 0x5618ba020930 2024-03-24T01:33:02.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.523+0000 7f8de3400640 1 -- 172.21.15.90:0/239028224 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de8037040 con 0x5618ba020930 2024-03-24T01:33:02.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.523+0000 7f8de3400640 1 -- 172.21.15.90:0/239028224 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de8051650 con 0x5618ba020930 2024-03-24T01:33:02.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.523+0000 7f8e0176a740 1 -- 172.21.15.90:0/239028224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618ba020930 msgr2=0x5618ba0181b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:33:02.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.523+0000 7f8e0176a740 1 --2- 172.21.15.90:0/239028224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618ba020930 0x5618ba0181b0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f8de8006da0 tx=0x7f8de8053040 comp rx=0 tx=0).stop 2024-03-24T01:33:02.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.523+0000 7f8e0176a740 1 -- 172.21.15.90:0/239028224 shutdown_connections 2024-03-24T01:33:02.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.523+0000 7f8e0176a740 1 --2- 172.21.15.90:0/239028224 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ba0780d0 0x5618ba021020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.523+0000 7f8e0176a740 1 --2- 172.21.15.90:0/239028224 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5618ba04cdb0 0x5618ba07ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.523+0000 7f8e0176a740 1 --2- 172.21.15.90:0/239028224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618ba020930 0x5618ba0181b0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.523+0000 7f8e0176a740 1 -- 172.21.15.90:0/239028224 >> 172.21.15.90:0/239028224 conn(0x5618ba01cdd0 msgr2=0x5618ba047940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:33:02.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.524+0000 7f8e0176a740 1 -- 172.21.15.90:0/239028224 shutdown_connections 2024-03-24T01:33:02.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.524+0000 7f8e0176a740 1 -- 172.21.15.90:0/239028224 wait complete. 2024-03-24T01:33:02.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.524+0000 7f8e0176a740 1 -- start start 2024-03-24T01:33:02.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.524+0000 7f8e0176a740 1 Processor -- start 2024-03-24T01:33:02.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.525+0000 7f8e0176a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618ba020930 0x5618ba04d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.525+0000 7f8e0176a740 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5618ba04cdb0 0x5618ba04df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.525+0000 7f8e0176a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ba0780d0 0x5618ba067be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.525+0000 7f8e0176a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618ba021ae0 con 0x5618ba020930 2024-03-24T01:33:02.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.525+0000 7f8e0176a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618b9f42dd0 con 0x5618ba0780d0 2024-03-24T01:33:02.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.525+0000 7f8e0176a740 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x5618ba07b8a0 con 0x5618ba04cdb0 2024-03-24T01:33:02.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.525+0000 7f8e0176a740 10 client.0 _ll_register_callbacks cb 0x7f8de802bdd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T01:33:02.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.525+0000 7f8df7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618ba020930 0x5618ba04d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.525+0000 7f8df7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618ba020930 0x5618ba04d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54452/0 (socket says 172.21.15.90:54452) 2024-03-24T01:33:02.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.525+0000 7f8df7e00640 1 -- 172.21.15.90:0/507546586 learned_addr learned my addr 172.21.15.90:0/507546586 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:33:02.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.526+0000 7f8df7e00640 1 -- 172.21.15.90:0/507546586 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5618ba04cdb0 msgr2=0x5618ba04df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:33:02.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.526+0000 7f8df7e00640 1 --2- 172.21.15.90:0/507546586 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5618ba04cdb0 0x5618ba04df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.526+0000 7f8df7e00640 1 -- 172.21.15.90:0/507546586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ba0780d0 msgr2=0x5618ba067be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:33:02.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.526+0000 7f8df7e00640 1 --2- 172.21.15.90:0/507546586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ba0780d0 0x5618ba067be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.526+0000 7f8df7e00640 1 -- 172.21.15.90:0/507546586 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5618ba0da870 con 0x5618ba020930 2024-03-24T01:33:02.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.526+0000 7f8df7e00640 1 --2- 172.21.15.90:0/507546586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618ba020930 0x5618ba04d190 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f8de80090b0 tx=0x7f8de8048000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46f55ac510605b54 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:33:02.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.526+0000 7f8de0c00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-03-24T01:33:02.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.526+0000 7f8de0c00640 1 -- 172.21.15.90:0/507546586 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de806d070 con 0x5618ba020930 2024-03-24T01:33:02.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.526+0000 7f8de0c00640 1 -- 172.21.15.90:0/507546586 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7f8de8051b90 con 0x5618ba020930 2024-03-24T01:33:02.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.526+0000 7f8e0176a740 1 -- 172.21.15.90:0/507546586 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8de8037040 con 0x5618ba020930 2024-03-24T01:33:02.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.527+0000 7f8de0c00640 1 -- 172.21.15.90:0/507546586 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8de8078050 con 0x5618ba020930 2024-03-24T01:33:02.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.527+0000 7f8e0176a740 10 client.7347 fetch_fsmap learned FSMap version 53 2024-03-24T01:33:02.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.527+0000 7f8e0176a740 1 -- 172.21.15.90:0/507546586 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=2+}) v3 -- 0x5618ba0da5f0 con 0x5618ba020930 2024-03-24T01:33:02.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.527+0000 7f8de0c00640 1 -- 172.21.15.90:0/507546586 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de8037040 con 0x5618ba020930 2024-03-24T01:33:02.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.527+0000 7f8de0c00640 1 -- 172.21.15.90:0/507546586 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 53) v1 ==== 3334+0+0 (secure 0 0 0) 0x7f8de807b020 con 0x5618ba020930 2024-03-24T01:33:02.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.527+0000 7f8e0176a740 10 client.7347 fetch_fsmap finished waiting for FSMap version 53 2024-03-24T01:33:02.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.527+0000 7f8e0176a740 10 client.7347 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-24T01:33:02.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.527+0000 7f8e0176a740 20 client.7347 populate_metadata read hostname 'smithi090' 2024-03-24T01:33:02.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.527+0000 7f8e0176a740 1 --2- 172.21.15.90:0/507546586 >> [v2:172.21.15.120:6834/910380191,v1:172.21.15.120:6835/910380191] conn(0x5618ba0d68e0 0x5618ba0d6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.527+0000 7f8e0176a740 4 client.7347 mds_command: new command op to 7110 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T01:33:02.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.527+0000 7f8e0176a740 1 -- 172.21.15.90:0/507546586 --> [v2:172.21.15.120:6834/910380191,v1:172.21.15.120:6835/910380191] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5618b9f941d0 con 0x5618ba0d68e0 2024-03-24T01:33:02.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.528+0000 7f8df7400640 1 --2- 172.21.15.90:0/507546586 >> [v2:172.21.15.120:6834/910380191,v1:172.21.15.120:6835/910380191] conn(0x5618ba0d68e0 0x5618ba0d6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.528+0000 7f8df7400640 1 --2- 172.21.15.90:0/507546586 >> [v2:172.21.15.120:6834/910380191,v1:172.21.15.120:6835/910380191] conn(0x5618ba0d68e0 0x5618ba0d6cc0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6327ca3685fc0535 server_cookie=ac03ad0b9a9fa575 in_seq=0 out_seq=0 2024-03-24T01:33:02.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.528+0000 7f8de0c00640 10 client.7347 ms_handle_connect on v2:172.21.15.120:6834/910380191 2024-03-24T01:33:02.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.529+0000 7f8de0c00640 1 -- 172.21.15.90:0/507546586 <== mds.0 v2:172.21.15.120:6834/910380191 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5618b9f941d0 con 0x5618ba0d68e0 2024-03-24T01:33:02.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.529+0000 7f8de0c00640 10 client.7347 handle_command_reply: tid=0 2024-03-24T01:33:02.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.529+0000 7f8e0176a740 1 client.7347 shutdown 2024-03-24T01:33:02.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.529+0000 7f8e0176a740 1 -- 172.21.15.90:0/507546586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618ba020930 msgr2=0x5618ba04d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:33:02.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.529+0000 7f8e0176a740 1 --2- 172.21.15.90:0/507546586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618ba020930 0x5618ba04d190 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f8de80090b0 tx=0x7f8de8048000 comp rx=0 tx=0).stop 2024-03-24T01:33:02.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.529+0000 7f8e0176a740 1 -- 172.21.15.90:0/507546586 shutdown_connections 2024-03-24T01:33:02.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.529+0000 7f8e0176a740 1 --2- 172.21.15.90:0/507546586 >> [v2:172.21.15.120:6834/910380191,v1:172.21.15.120:6835/910380191] conn(0x5618ba0d68e0 0x5618ba0d6cc0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.529+0000 7f8e0176a740 1 --2- 172.21.15.90:0/507546586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ba0780d0 0x5618ba067be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.529+0000 7f8e0176a740 1 --2- 172.21.15.90:0/507546586 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x5618ba04cdb0 0x5618ba04df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.529+0000 7f8e0176a740 1 --2- 172.21.15.90:0/507546586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618ba020930 0x5618ba04d190 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.529+0000 7f8e0176a740 1 -- 172.21.15.90:0/507546586 >> 172.21.15.90:0/507546586 conn(0x5618ba01cdd0 msgr2=0x5618ba04ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:33:02.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.529+0000 7f8de0c00640 0 client.7347 ms_handle_reset on v2:172.21.15.120:6834/910380191 2024-03-24T01:33:02.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.529+0000 7f8e0176a740 1 -- 172.21.15.90:0/507546586 shutdown_connections 2024-03-24T01:33:02.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.529+0000 7f8e0176a740 1 -- 172.21.15.90:0/507546586 wait complete. 2024-03-24T01:33:02.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.530+0000 7f8e0176a740 20 client.7347 trim_cache size 0 max 16384 2024-03-24T01:33:02.533 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-24T01:33:02.533 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "24712165-9839-4fcc-8033-988386efea74", 2024-03-24T01:33:02.533 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 0, 2024-03-24T01:33:02.533 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 7110, 2024-03-24T01:33:02.533 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-03-24T01:33:02.533 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-03-24T01:33:02.533 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-03-24T01:33:02.533 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 17.779571019999999, 2024-03-24T01:33:02.533 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 53, 2024-03-24T01:33:02.534 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 69, 2024-03-24T01:33:02.534 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 69, 2024-03-24T01:33:02.534 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 23.237702423000002 2024-03-24T01:33:02.534 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-24T01:33:02.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.530+0000 7f8dfd400640 1 -- 172.21.15.90:0/2531687215 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8dd806bbd0 msgr2=0x7f8dd806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:33:02.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.530+0000 7f8dfd400640 1 --2- 172.21.15.90:0/2531687215 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8dd806bbd0 0x7f8dd806e090 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f8de8036000 tx=0x7f8de803d040 comp rx=0 tx=0).stop 2024-03-24T01:33:02.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.530+0000 7f8dfd400640 1 -- 172.21.15.90:0/2531687215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8df80d35b0 msgr2=0x7f8df80cc190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:33:02.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.530+0000 7f8dfd400640 1 --2- 172.21.15.90:0/2531687215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8df80d35b0 0x7f8df80cc190 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f8de400c820 tx=0x7f8de400cce0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.530+0000 7f8dfd400640 1 -- 172.21.15.90:0/2531687215 shutdown_connections 2024-03-24T01:33:02.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.530+0000 7f8dfd400640 1 --2- 172.21.15.90:0/2531687215 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f8dd806bbd0 0x7f8dd806e090 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.530+0000 7f8dfd400640 1 --2- 172.21.15.90:0/2531687215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8df80d35b0 0x7f8df80cc190 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.530+0000 7f8dfd400640 1 --2- 172.21.15.90:0/2531687215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8df815de50 0x7f8df80cc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.530+0000 7f8dfd400640 1 --2- 172.21.15.90:0/2531687215 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f8df80b5550 0x7f8df81f3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.530+0000 7f8dfd400640 1 -- 172.21.15.90:0/2531687215 >> 172.21.15.90:0/2531687215 conn(0x7f8df80d0550 msgr2=0x7f8df80b3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:33:02.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.530+0000 7f8dfd400640 1 -- 172.21.15.90:0/2531687215 shutdown_connections 2024-03-24T01:33:02.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.530+0000 7f8dfd400640 1 -- 172.21.15.90:0/2531687215 wait complete. 2024-03-24T01:33:02.546 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable false 2024-03-24T01:33:02.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.728+0000 7f3b21a00640 1 Processor -- start 2024-03-24T01:33:02.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.728+0000 7f3b21a00640 1 -- start start 2024-03-24T01:33:02.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.729+0000 7f3b21a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f3b1c15dd90 0x7f3b1c0b6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.729+0000 7f3b21a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b1c0b9460 con 0x7f3b1c06cdd0 2024-03-24T01:33:02.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.729+0000 7f3b21a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b1c0b9740 con 0x7f3b1c0b2720 2024-03-24T01:33:02.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.729+0000 7f3b21a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b1c0b9a20 con 0x7f3b1c15dd90 2024-03-24T01:33:02.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.730+0000 7f3b21000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f3b1c06cdd0 0x7f3b1c0d2410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:58854/0 (socket says 172.21.15.90:58854) 2024-03-24T01:33:02.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.730+0000 7f3b21000640 1 -- 172.21.15.90:0/3786532123 learned_addr learned my addr 172.21.15.90:0/3786532123 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:33:02.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.730+0000 7f3b1be00640 1 --2- 172.21.15.90:0/3786532123 >> v2:172.21.15.120:3300/0 conn(0x7f3b1c15dd90 0x7f3b1c0b6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.730+0000 7f3b1aa00640 1 -- 172.21.15.90:0/3786532123 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2045859732 0 0) 0x7f3b1c0b9460 con 0x7f3b1c06cdd0 2024-03-24T01:33:02.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.730+0000 7f3b1aa00640 1 -- 172.21.15.90:0/3786532123 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b00003520 con 0x7f3b1c06cdd0 2024-03-24T01:33:02.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.730+0000 7f3b1aa00640 1 -- 172.21.15.90:0/3786532123 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 824974528 0 0) 0x7f3b00003520 con 0x7f3b1c06cdd0 2024-03-24T01:33:02.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.730+0000 7f3b1aa00640 1 -- 172.21.15.90:0/3786532123 >> v2:172.21.15.120:3300/0 conn(0x7f3b1c15dd90 msgr2=0x7f3b1c0b6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:33:02.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.730+0000 7f3b1aa00640 1 --2- 172.21.15.90:0/3786532123 >> v2:172.21.15.120:3300/0 conn(0x7f3b1c15dd90 0x7f3b1c0b6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.730+0000 7f3b1aa00640 1 -- 172.21.15.90:0/3786532123 >> v1:172.21.15.120:6789/0 conn(0x7f3b1c0b2720 legacy=0x7f3b1c06c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:33:02.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.730+0000 7f3b1aa00640 1 -- 172.21.15.90:0/3786532123 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b1c0d1c10 con 0x7f3b1c06cdd0 2024-03-24T01:33:02.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.731+0000 7f3b1aa00640 1 -- 172.21.15.90:0/3786532123 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f3b140019f0 con 0x7f3b1c06cdd0 2024-03-24T01:33:02.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.731+0000 7f3b1aa00640 1 -- 172.21.15.90:0/3786532123 >> v1:172.21.15.90:6789/0 conn(0x7f3b1c06cdd0 legacy=0x7f3b1c0d2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:33:02.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.731+0000 7f3b1aa00640 1 --2- 172.21.15.90:0/3786532123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b000039b0 0x7f3b00003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.731+0000 7f3b1aa00640 1 --2- 172.21.15.90:0/3786532123 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b00004800 0x7f3b00004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.731+0000 7f3b1aa00640 1 --2- 172.21.15.90:0/3786532123 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b000051c0 0x7f3b0000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.731+0000 7f3b1aa00640 1 -- 172.21.15.90:0/3786532123 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1c0b9a20 con 0x7f3b000051c0 2024-03-24T01:33:02.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.731+0000 7f3b1aa00640 1 -- 172.21.15.90:0/3786532123 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0000c060 con 0x7f3b000039b0 2024-03-24T01:33:02.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.731+0000 7f3b1aa00640 1 -- 172.21.15.90:0/3786532123 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0000c260 con 0x7f3b00004800 2024-03-24T01:33:02.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.731+0000 7f3b21000640 1 --2- 172.21.15.90:0/3786532123 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b000051c0 0x7f3b0000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.731+0000 7f3b1be00640 1 --2- 172.21.15.90:0/3786532123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b000039b0 0x7f3b00003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.731+0000 7f3b21000640 1 -- 172.21.15.90:0/3786532123 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b00004800 msgr2=0x7f3b00004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:33:02.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.731+0000 7f3b1b400640 1 --2- 172.21.15.90:0/3786532123 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b00004800 0x7f3b00004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.731+0000 7f3b21000640 1 --2- 172.21.15.90:0/3786532123 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b00004800 0x7f3b00004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.731+0000 7f3b21000640 1 -- 172.21.15.90:0/3786532123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b000039b0 msgr2=0x7f3b00003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:33:02.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.731+0000 7f3b21000640 1 --2- 172.21.15.90:0/3786532123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b000039b0 0x7f3b00003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.731+0000 7f3b21000640 1 -- 172.21.15.90:0/3786532123 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b00003520 con 0x7f3b000051c0 2024-03-24T01:33:02.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.731+0000 7f3b1be00640 1 --2- 172.21.15.90:0/3786532123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b000039b0 0x7f3b00003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:33:02.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.732+0000 7f3b21000640 1 -- 172.21.15.90:0/3786532123 reap_dead start 2024-03-24T01:33:02.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.732+0000 7f3b21000640 1 --2- 172.21.15.90:0/3786532123 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b000051c0 0x7f3b0000b9c0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f3b1400a840 tx=0x7f3b00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c2f1669de166fdb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:33:02.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.733+0000 7f3b1aa00640 1 -- 172.21.15.90:0/3786532123 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1400a460 con 0x7f3b000051c0 2024-03-24T01:33:02.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.733+0000 7f3b1aa00640 1 -- 172.21.15.90:0/3786532123 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b14034650 con 0x7f3b000051c0 2024-03-24T01:33:02.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.733+0000 7f3b1aa00640 1 -- 172.21.15.90:0/3786532123 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b14034a40 con 0x7f3b000051c0 2024-03-24T01:33:02.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.733+0000 7f3b21a00640 1 -- 172.21.15.90:0/3786532123 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b000051c0 msgr2=0x7f3b0000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:33:02.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.733+0000 7f3b21a00640 1 --2- 172.21.15.90:0/3786532123 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b000051c0 0x7f3b0000b9c0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f3b1400a840 tx=0x7f3b00005830 comp rx=0 tx=0).stop 2024-03-24T01:33:02.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.733+0000 7f3b21a00640 1 -- 172.21.15.90:0/3786532123 shutdown_connections 2024-03-24T01:33:02.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.733+0000 7f3b21a00640 1 --2- 172.21.15.90:0/3786532123 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b000051c0 0x7f3b0000b9c0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.733+0000 7f3b21a00640 1 -- 172.21.15.90:0/3786532123 >> 172.21.15.90:0/3786532123 conn(0x7f3b1c072e20 msgr2=0x7f3b1c0d3a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:33:02.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.733+0000 7f3b21a00640 1 -- 172.21.15.90:0/3786532123 shutdown_connections 2024-03-24T01:33:02.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.733+0000 7f3b21a00640 1 -- 172.21.15.90:0/3786532123 wait complete. 2024-03-24T01:33:02.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.735+0000 7f3b21a00640 1 Processor -- start 2024-03-24T01:33:02.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.735+0000 7f3b21a00640 1 -- start start 2024-03-24T01:33:02.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.736+0000 7f3b21a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b000051c0 0x7f3b1c0cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.736+0000 7f3b21a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b1c153d10 0x7f3b1c157290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.736+0000 7f3b21a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b1c152890 0x7f3b1c159b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.736+0000 7f3b21a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1c0b6c70 con 0x7f3b000051c0 2024-03-24T01:33:02.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.736+0000 7f3b21a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1c0b6af0 con 0x7f3b1c152890 2024-03-24T01:33:02.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.736+0000 7f3b21a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1c0b6df0 con 0x7f3b1c153d10 2024-03-24T01:33:02.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.736+0000 7f3b1be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b000051c0 0x7f3b1c0cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.736+0000 7f3b1be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b000051c0 0x7f3b1c0cdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54478/0 (socket says 172.21.15.90:54478) 2024-03-24T01:33:02.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.736+0000 7f3b1be00640 1 -- 172.21.15.90:0/3771873811 learned_addr learned my addr 172.21.15.90:0/3771873811 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:33:02.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.737+0000 7f3b1b400640 1 --2- 172.21.15.90:0/3771873811 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b1c153d10 0x7f3b1c157290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.737+0000 7f3b1be00640 1 -- 172.21.15.90:0/3771873811 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b1c153d10 msgr2=0x7f3b1c157290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:33:02.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.737+0000 7f3b1be00640 1 --2- 172.21.15.90:0/3771873811 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b1c153d10 0x7f3b1c157290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.737+0000 7f3b1be00640 1 -- 172.21.15.90:0/3771873811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b1c152890 msgr2=0x7f3b1c159b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:33:02.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.737+0000 7f3b1be00640 1 --2- 172.21.15.90:0/3771873811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b1c152890 0x7f3b1c159b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:02.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.737+0000 7f3b1be00640 1 -- 172.21.15.90:0/3771873811 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b1c1577d0 con 0x7f3b000051c0 2024-03-24T01:33:02.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.737+0000 7f3b1be00640 1 --2- 172.21.15.90:0/3771873811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b000051c0 0x7f3b1c0cdec0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c0108b0 tx=0x7f3b0c010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:33:02.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.737+0000 7f3b13e00640 1 -- 172.21.15.90:0/3771873811 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b0c00f040 con 0x7f3b000051c0 2024-03-24T01:33:02.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.737+0000 7f3b13e00640 1 -- 172.21.15.90:0/3771873811 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b0c00eb00 con 0x7f3b000051c0 2024-03-24T01:33:02.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.737+0000 7f3b13e00640 1 -- 172.21.15.90:0/3771873811 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b0c01c400 con 0x7f3b000051c0 2024-03-24T01:33:02.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.737+0000 7f3b21a00640 1 -- 172.21.15.90:0/3771873811 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b1c0b4cb0 con 0x7f3b000051c0 2024-03-24T01:33:02.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.737+0000 7f3b21a00640 1 -- 172.21.15.90:0/3771873811 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b1c0b5270 con 0x7f3b000051c0 2024-03-24T01:33:02.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.738+0000 7f3b21a00640 1 -- 172.21.15.90:0/3771873811 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b1c04f400 con 0x7f3b000051c0 2024-03-24T01:33:02.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.739+0000 7f3b13e00640 1 -- 172.21.15.90:0/3771873811 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f3b0c01c5a0 con 0x7f3b000051c0 2024-03-24T01:33:02.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.740+0000 7f3b13e00640 1 --2- 172.21.15.90:0/3771873811 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3afc06bc00 0x7f3afc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:33:02.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.740+0000 7f3b13e00640 1 -- 172.21.15.90:0/3771873811 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7f3b0c08b370 con 0x7f3b000051c0 2024-03-24T01:33:02.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.741+0000 7f3b1b400640 1 --2- 172.21.15.90:0/3771873811 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3afc06bc00 0x7f3afc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:33:02.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.742+0000 7f3b1b400640 1 --2- 172.21.15.90:0/3771873811 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3afc06bc00 0x7f3afc06e0c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f3b08000c00 tx=0x7f3b08009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:33:02.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.745+0000 7f3b13e00640 1 -- 172.21.15.90:0/3771873811 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f3b1c1577d0 con 0x7f3b000051c0 2024-03-24T01:33:02.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:02.981+0000 7f3b21a00640 1 -- 172.21.15.90:0/3771873811 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "false"} v 0) v1 -- 0x7f3b1c1f3ef0 con 0x7f3b000051c0 2024-03-24T01:33:03.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:03.282+0000 7f3b13e00640 1 -- 172.21.15.90:0/3771873811 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "false"}]=0 cephfs marked not joinable; MDS cannot join as newly active. v54) v1 ==== 170+0+0 (secure 0 0 0) 0x7f3b0c018030 con 0x7f3b000051c0 2024-03-24T01:33:03.283 INFO:teuthology.orchestra.run.smithi090.stderr:cephfs marked not joinable; MDS cannot join as newly active. 2024-03-24T01:33:03.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:03.285+0000 7f3b21a00640 1 -- 172.21.15.90:0/3771873811 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3afc06bc00 msgr2=0x7f3afc06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:33:03.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:03.285+0000 7f3b21a00640 1 --2- 172.21.15.90:0/3771873811 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3afc06bc00 0x7f3afc06e0c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f3b08000c00 tx=0x7f3b08009040 comp rx=0 tx=0).stop 2024-03-24T01:33:03.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:03.285+0000 7f3b21a00640 1 -- 172.21.15.90:0/3771873811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b000051c0 msgr2=0x7f3b1c0cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:33:03.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:03.285+0000 7f3b21a00640 1 --2- 172.21.15.90:0/3771873811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b000051c0 0x7f3b1c0cdec0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c0108b0 tx=0x7f3b0c010d70 comp rx=0 tx=0).stop 2024-03-24T01:33:03.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:03.285+0000 7f3b21a00640 1 -- 172.21.15.90:0/3771873811 shutdown_connections 2024-03-24T01:33:03.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:03.285+0000 7f3b21a00640 1 --2- 172.21.15.90:0/3771873811 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f3afc06bc00 0x7f3afc06e0c0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:03.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:03.285+0000 7f3b21a00640 1 --2- 172.21.15.90:0/3771873811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3b1c152890 0x7f3b1c159b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:03.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:03.285+0000 7f3b21a00640 1 --2- 172.21.15.90:0/3771873811 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f3b1c153d10 0x7f3b1c157290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:03.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:03.285+0000 7f3b21a00640 1 --2- 172.21.15.90:0/3771873811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b000051c0 0x7f3b1c0cdec0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:33:03.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:03.285+0000 7f3b21a00640 1 -- 172.21.15.90:0/3771873811 >> 172.21.15.90:0/3771873811 conn(0x7f3b1c072e20 msgr2=0x7f3b1c15f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:33:03.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:03.286+0000 7f3b21a00640 1 -- 172.21.15.90:0/3771873811 shutdown_connections 2024-03-24T01:33:03.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:03.286+0000 7f3b21a00640 1 -- 172.21.15.90:0/3771873811 wait complete. 2024-03-24T01:33:03.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:33:03.304 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'mkdir {alpha,bravo} && touch {alpha,bravo}/file') 2024-03-24T01:33:03.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:33:03.387 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 0 alpha/) 2024-03-24T01:33:03.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:33:03.447 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 bravo/) 2024-03-24T01:33:03.500 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json get subtrees 2024-03-24T01:33:03.712 INFO:teuthology.orchestra.run.smithi120.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"13","version":"9","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":2,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":1},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-24T01:33:03.713 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 2, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 1, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "13", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "9" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T01:33:03.713 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/alpha', 0), ('/bravo', 1)] 2024-03-24T01:33:05.714 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json get subtrees 2024-03-24T01:33:05.942 INFO:teuthology.orchestra.run.smithi120.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"13","version":"13","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-24T01:33:05.943 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 3, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "13", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T01:33:05.943 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/alpha', 0), ('/bravo', 1)] 2024-03-24T01:33:07.944 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json get subtrees 2024-03-24T01:33:08.177 INFO:teuthology.orchestra.run.smithi120.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"13","version":"13","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-24T01:33:08.178 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 3, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "13", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T01:33:08.178 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/alpha', 0), ('/bravo', 1)] 2024-03-24T01:33:10.179 DEBUG:teuthology.orchestra.run.smithi120:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json get subtrees 2024-03-24T01:33:10.451 INFO:teuthology.orchestra.run.smithi120.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"18","version":"18","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":0,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/alpha","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/bravo","dirfrag":"0x10000000001","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}}] 2024-03-24T01:33:10.451 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "18", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "18" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/alpha", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "5", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 0, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000001", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/bravo", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2024-03-24T01:33:10.452 INFO:tasks.cephfs.cephfs_test_case:[('/alpha', 0), ('/bravo', 1)] =?= [('/alpha', 0), ('/bravo', 1)] 2024-03-24T01:33:10.452 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:33:10.452 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:33:10.504 DEBUG:teuthology.orchestra.run.smithi090:> stdin-killer -- python3 -c ' 2024-03-24T01:33:10.504 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-24T01:33:10.504 DEBUG:teuthology.orchestra.run.smithi090:> import uuid 2024-03-24T01:33:10.504 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:33:10.504 DEBUG:teuthology.orchestra.run.smithi090:> n = 1000 2024-03-24T01:33:10.504 DEBUG:teuthology.orchestra.run.smithi090:> create_hard_links = False 2024-03-24T01:33:10.504 DEBUG:teuthology.orchestra.run.smithi090:> if 0 > 0: 2024-03-24T01:33:10.504 DEBUG:teuthology.orchestra.run.smithi090:> create_hard_links = True 2024-03-24T01:33:10.504 DEBUG:teuthology.orchestra.run.smithi090:> path = "/home/ubuntu/cephtest/mnt.0/alpha/file" 2024-03-24T01:33:10.504 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:33:10.504 DEBUG:teuthology.orchestra.run.smithi090:> dpath = os.path.dirname(path) 2024-03-24T01:33:10.504 DEBUG:teuthology.orchestra.run.smithi090:> fnameprefix = os.path.basename(path) 2024-03-24T01:33:10.504 DEBUG:teuthology.orchestra.run.smithi090:> os.makedirs(dpath, exist_ok=True) 2024-03-24T01:33:10.504 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:33:10.505 DEBUG:teuthology.orchestra.run.smithi090:> try: 2024-03-24T01:33:10.505 DEBUG:teuthology.orchestra.run.smithi090:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-24T01:33:10.505 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:33:10.505 DEBUG:teuthology.orchestra.run.smithi090:> for i in range(n): 2024-03-24T01:33:10.505 DEBUG:teuthology.orchestra.run.smithi090:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-24T01:33:10.505 DEBUG:teuthology.orchestra.run.smithi090:> with open(fpath, '"'"'w'"'"') as f: 2024-03-24T01:33:10.505 DEBUG:teuthology.orchestra.run.smithi090:> f.write(f"{i}") 2024-03-24T01:33:10.505 DEBUG:teuthology.orchestra.run.smithi090:> if False: 2024-03-24T01:33:10.505 DEBUG:teuthology.orchestra.run.smithi090:> f.flush() 2024-03-24T01:33:10.505 DEBUG:teuthology.orchestra.run.smithi090:> os.fsync(f.fileno()) 2024-03-24T01:33:10.505 DEBUG:teuthology.orchestra.run.smithi090:> if False: 2024-03-24T01:33:10.505 DEBUG:teuthology.orchestra.run.smithi090:> os.unlink(fpath) 2024-03-24T01:33:10.505 DEBUG:teuthology.orchestra.run.smithi090:> if False: 2024-03-24T01:33:10.505 DEBUG:teuthology.orchestra.run.smithi090:> os.fsync(dirfd) 2024-03-24T01:33:10.505 DEBUG:teuthology.orchestra.run.smithi090:> if create_hard_links: 2024-03-24T01:33:10.505 DEBUG:teuthology.orchestra.run.smithi090:> for j in range(0): 2024-03-24T01:33:10.505 DEBUG:teuthology.orchestra.run.smithi090:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-24T01:33:10.506 DEBUG:teuthology.orchestra.run.smithi090:> if False: 2024-03-24T01:33:10.506 DEBUG:teuthology.orchestra.run.smithi090:> os.fsync(dirfd) 2024-03-24T01:33:10.506 DEBUG:teuthology.orchestra.run.smithi090:> finally: 2024-03-24T01:33:10.506 DEBUG:teuthology.orchestra.run.smithi090:> os.close(dirfd) 2024-03-24T01:33:10.506 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-24T01:33:27.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:33:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T01:33:27.029 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:33:27.029 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:33:27.114 DEBUG:teuthology.orchestra.run.smithi090:> stdin-killer -- python3 -c ' 2024-03-24T01:33:27.114 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-24T01:33:27.114 DEBUG:teuthology.orchestra.run.smithi090:> import uuid 2024-03-24T01:33:27.114 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:33:27.114 DEBUG:teuthology.orchestra.run.smithi090:> n = 1000 2024-03-24T01:33:27.115 DEBUG:teuthology.orchestra.run.smithi090:> create_hard_links = False 2024-03-24T01:33:27.115 DEBUG:teuthology.orchestra.run.smithi090:> if 0 > 0: 2024-03-24T01:33:27.115 DEBUG:teuthology.orchestra.run.smithi090:> create_hard_links = True 2024-03-24T01:33:27.115 DEBUG:teuthology.orchestra.run.smithi090:> path = "/home/ubuntu/cephtest/mnt.0/bravo/file" 2024-03-24T01:33:27.115 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:33:27.115 DEBUG:teuthology.orchestra.run.smithi090:> dpath = os.path.dirname(path) 2024-03-24T01:33:27.115 DEBUG:teuthology.orchestra.run.smithi090:> fnameprefix = os.path.basename(path) 2024-03-24T01:33:27.115 DEBUG:teuthology.orchestra.run.smithi090:> os.makedirs(dpath, exist_ok=True) 2024-03-24T01:33:27.115 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:33:27.115 DEBUG:teuthology.orchestra.run.smithi090:> try: 2024-03-24T01:33:27.115 DEBUG:teuthology.orchestra.run.smithi090:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-24T01:33:27.115 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-24T01:33:27.115 DEBUG:teuthology.orchestra.run.smithi090:> for i in range(n): 2024-03-24T01:33:27.115 DEBUG:teuthology.orchestra.run.smithi090:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-24T01:33:27.115 DEBUG:teuthology.orchestra.run.smithi090:> with open(fpath, '"'"'w'"'"') as f: 2024-03-24T01:33:27.115 DEBUG:teuthology.orchestra.run.smithi090:> f.write(f"{i}") 2024-03-24T01:33:27.116 DEBUG:teuthology.orchestra.run.smithi090:> if False: 2024-03-24T01:33:27.116 DEBUG:teuthology.orchestra.run.smithi090:> f.flush() 2024-03-24T01:33:27.116 DEBUG:teuthology.orchestra.run.smithi090:> os.fsync(f.fileno()) 2024-03-24T01:33:27.116 DEBUG:teuthology.orchestra.run.smithi090:> if False: 2024-03-24T01:33:27.116 DEBUG:teuthology.orchestra.run.smithi090:> os.unlink(fpath) 2024-03-24T01:33:27.116 DEBUG:teuthology.orchestra.run.smithi090:> if False: 2024-03-24T01:33:27.116 DEBUG:teuthology.orchestra.run.smithi090:> os.fsync(dirfd) 2024-03-24T01:33:27.116 DEBUG:teuthology.orchestra.run.smithi090:> if create_hard_links: 2024-03-24T01:33:27.116 DEBUG:teuthology.orchestra.run.smithi090:> for j in range(0): 2024-03-24T01:33:27.116 DEBUG:teuthology.orchestra.run.smithi090:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-24T01:33:27.116 DEBUG:teuthology.orchestra.run.smithi090:> if False: 2024-03-24T01:33:27.116 DEBUG:teuthology.orchestra.run.smithi090:> os.fsync(dirfd) 2024-03-24T01:33:27.116 DEBUG:teuthology.orchestra.run.smithi090:> finally: 2024-03-24T01:33:27.116 DEBUG:teuthology.orchestra.run.smithi090:> os.close(dirfd) 2024-03-24T01:33:27.116 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-24T01:33:27.326 INFO:tasks.ceph.mds.a.smithi090.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el9/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f25fd800640 time 2024-03-24T01:33:27.327285+0000 2024-03-24T01:33:27.327 INFO:tasks.ceph.mds.a.smithi090.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el9/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T01:33:27.328 INFO:tasks.ceph.mds.a.smithi090.stderr: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T01:33:27.328 INFO:tasks.ceph.mds.a.smithi090.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f2604781fba] 2024-03-24T01:33:27.329 INFO:tasks.ceph.mds.a.smithi090.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x182178) [0x7f2604782178] 2024-03-24T01:33:27.329 INFO:tasks.ceph.mds.a.smithi090.stderr: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x402) [0x55fa5e101392] 2024-03-24T01:33:27.329 INFO:tasks.ceph.mds.a.smithi090.stderr: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3d3) [0x55fa5dfe7513] 2024-03-24T01:33:27.329 INFO:tasks.ceph.mds.a.smithi090.stderr: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xbc) [0x55fa5dfe51dc] 2024-03-24T01:33:27.329 INFO:tasks.ceph.mds.a.smithi090.stderr: 6: (Server::handle_client_openc(boost::intrusive_ptr const&)+0xd16) [0x55fa5dff8f26] 2024-03-24T01:33:27.329 INFO:tasks.ceph.mds.a.smithi090.stderr: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6b6) [0x55fa5dfe6786] 2024-03-24T01:33:27.329 INFO:tasks.ceph.mds.a.smithi090.stderr: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55fa5dfd9f77] 2024-03-24T01:33:27.329 INFO:tasks.ceph.mds.a.smithi090.stderr: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55fa5df6055b] 2024-03-24T01:33:27.329 INFO:tasks.ceph.mds.a.smithi090.stderr: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55fa5df60a2c] 2024-03-24T01:33:27.329 INFO:tasks.ceph.mds.a.smithi090.stderr: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x55fa5df45465] 2024-03-24T01:33:27.329 INFO:tasks.ceph.mds.a.smithi090.stderr: 12: (DispatchQueue::entry()+0x53a) [0x7f260498009a] 2024-03-24T01:33:27.329 INFO:tasks.ceph.mds.a.smithi090.stderr: 13: /usr/lib64/ceph/libceph-common.so.2(+0x4142c1) [0x7f2604a142c1] 2024-03-24T01:33:27.329 INFO:tasks.ceph.mds.a.smithi090.stderr: 14: /lib64/libc.so.6(+0x9f802) [0x7f2603e9f802] 2024-03-24T01:33:27.329 INFO:tasks.ceph.mds.a.smithi090.stderr: 15: /lib64/libc.so.6(+0x3f450) [0x7f2603e3f450] 2024-03-24T01:33:27.329 INFO:tasks.ceph.mds.a.smithi090.stderr:*** Caught signal (Aborted) ** 2024-03-24T01:33:27.330 INFO:tasks.ceph.mds.a.smithi090.stderr: in thread 7f25fd800640 thread_name:ms_dispatch 2024-03-24T01:33:27.330 INFO:tasks.ceph.mds.a.smithi090.stderr:2024-03-24T01:33:27.328+0000 7f25fd800640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el9/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f25fd800640 time 2024-03-24T01:33:27.327285+0000 2024-03-24T01:33:27.330 INFO:tasks.ceph.mds.a.smithi090.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el9/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T01:33:27.330 INFO:tasks.ceph.mds.a.smithi090.stderr: 2024-03-24T01:33:27.330 INFO:tasks.ceph.mds.a.smithi090.stderr: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T01:33:27.330 INFO:tasks.ceph.mds.a.smithi090.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f2604781fba] 2024-03-24T01:33:27.330 INFO:tasks.ceph.mds.a.smithi090.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x182178) [0x7f2604782178] 2024-03-24T01:33:27.330 INFO:tasks.ceph.mds.a.smithi090.stderr: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x402) [0x55fa5e101392] 2024-03-24T01:33:27.330 INFO:tasks.ceph.mds.a.smithi090.stderr: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3d3) [0x55fa5dfe7513] 2024-03-24T01:33:27.330 INFO:tasks.ceph.mds.a.smithi090.stderr: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xbc) [0x55fa5dfe51dc] 2024-03-24T01:33:27.330 INFO:tasks.ceph.mds.a.smithi090.stderr: 6: (Server::handle_client_openc(boost::intrusive_ptr const&)+0xd16) [0x55fa5dff8f26] 2024-03-24T01:33:27.330 INFO:tasks.ceph.mds.a.smithi090.stderr: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6b6) [0x55fa5dfe6786] 2024-03-24T01:33:27.330 INFO:tasks.ceph.mds.a.smithi090.stderr: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55fa5dfd9f77] 2024-03-24T01:33:27.331 INFO:tasks.ceph.mds.a.smithi090.stderr: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55fa5df6055b] 2024-03-24T01:33:27.331 INFO:tasks.ceph.mds.a.smithi090.stderr: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55fa5df60a2c] 2024-03-24T01:33:27.331 INFO:tasks.ceph.mds.a.smithi090.stderr: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x55fa5df45465] 2024-03-24T01:33:27.331 INFO:tasks.ceph.mds.a.smithi090.stderr: 12: (DispatchQueue::entry()+0x53a) [0x7f260498009a] 2024-03-24T01:33:27.331 INFO:tasks.ceph.mds.a.smithi090.stderr: 13: /usr/lib64/ceph/libceph-common.so.2(+0x4142c1) [0x7f2604a142c1] 2024-03-24T01:33:27.331 INFO:tasks.ceph.mds.a.smithi090.stderr: 14: /lib64/libc.so.6(+0x9f802) [0x7f2603e9f802] 2024-03-24T01:33:27.331 INFO:tasks.ceph.mds.a.smithi090.stderr: 15: /lib64/libc.so.6(+0x3f450) [0x7f2603e3f450] 2024-03-24T01:33:27.331 INFO:tasks.ceph.mds.a.smithi090.stderr: 2024-03-24T01:33:27.331 INFO:tasks.ceph.mds.a.smithi090.stderr: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T01:33:27.332 INFO:tasks.ceph.mds.a.smithi090.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f2603e54db0] 2024-03-24T01:33:27.332 INFO:tasks.ceph.mds.a.smithi090.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f2603ea154c] 2024-03-24T01:33:27.332 INFO:tasks.ceph.mds.a.smithi090.stderr: 3: raise() 2024-03-24T01:33:27.332 INFO:tasks.ceph.mds.a.smithi090.stderr: 4: abort() 2024-03-24T01:33:27.332 INFO:tasks.ceph.mds.a.smithi090.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f2604782014] 2024-03-24T01:33:27.332 INFO:tasks.ceph.mds.a.smithi090.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x182178) [0x7f2604782178] 2024-03-24T01:33:27.332 INFO:tasks.ceph.mds.a.smithi090.stderr: 7: (Locker::set_xlocks_done(MutationImpl*, bool)+0x402) [0x55fa5e101392] 2024-03-24T01:33:27.332 INFO:tasks.ceph.mds.a.smithi090.stderr: 8: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3d3) [0x55fa5dfe7513] 2024-03-24T01:33:27.332 INFO:tasks.ceph.mds.a.smithi090.stderr: 9: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xbc) [0x55fa5dfe51dc] 2024-03-24T01:33:27.332 INFO:tasks.ceph.mds.a.smithi090.stderr: 10: (Server::handle_client_openc(boost::intrusive_ptr const&)+0xd16) [0x55fa5dff8f26] 2024-03-24T01:33:27.332 INFO:tasks.ceph.mds.a.smithi090.stderr: 11: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6b6) [0x55fa5dfe6786] 2024-03-24T01:33:27.332 INFO:tasks.ceph.mds.a.smithi090.stderr: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55fa5dfd9f77] 2024-03-24T01:33:27.332 INFO:tasks.ceph.mds.a.smithi090.stderr: 13: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55fa5df6055b] 2024-03-24T01:33:27.332 INFO:tasks.ceph.mds.a.smithi090.stderr: 14: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55fa5df60a2c] 2024-03-24T01:33:27.333 INFO:tasks.ceph.mds.a.smithi090.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x55fa5df45465] 2024-03-24T01:33:27.333 INFO:tasks.ceph.mds.a.smithi090.stderr: 16: (DispatchQueue::entry()+0x53a) [0x7f260498009a] 2024-03-24T01:33:27.333 INFO:tasks.ceph.mds.a.smithi090.stderr: 17: /usr/lib64/ceph/libceph-common.so.2(+0x4142c1) [0x7f2604a142c1] 2024-03-24T01:33:27.333 INFO:tasks.ceph.mds.a.smithi090.stderr: 18: /lib64/libc.so.6(+0x9f802) [0x7f2603e9f802] 2024-03-24T01:33:27.333 INFO:tasks.ceph.mds.a.smithi090.stderr: 19: /lib64/libc.so.6(+0x3f450) [0x7f2603e3f450] 2024-03-24T01:33:27.333 INFO:tasks.ceph.mds.a.smithi090.stderr:2024-03-24T01:33:27.331+0000 7f25fd800640 -1 *** Caught signal (Aborted) ** 2024-03-24T01:33:27.333 INFO:tasks.ceph.mds.a.smithi090.stderr: in thread 7f25fd800640 thread_name:ms_dispatch 2024-03-24T01:33:27.333 INFO:tasks.ceph.mds.a.smithi090.stderr: 2024-03-24T01:33:27.333 INFO:tasks.ceph.mds.a.smithi090.stderr: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T01:33:27.333 INFO:tasks.ceph.mds.a.smithi090.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f2603e54db0] 2024-03-24T01:33:27.333 INFO:tasks.ceph.mds.a.smithi090.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f2603ea154c] 2024-03-24T01:33:27.333 INFO:tasks.ceph.mds.a.smithi090.stderr: 3: raise() 2024-03-24T01:33:27.333 INFO:tasks.ceph.mds.a.smithi090.stderr: 4: abort() 2024-03-24T01:33:27.334 INFO:tasks.ceph.mds.a.smithi090.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f2604782014] 2024-03-24T01:33:27.334 INFO:tasks.ceph.mds.a.smithi090.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x182178) [0x7f2604782178] 2024-03-24T01:33:27.334 INFO:tasks.ceph.mds.a.smithi090.stderr: 7: (Locker::set_xlocks_done(MutationImpl*, bool)+0x402) [0x55fa5e101392] 2024-03-24T01:33:27.334 INFO:tasks.ceph.mds.a.smithi090.stderr: 8: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3d3) [0x55fa5dfe7513] 2024-03-24T01:33:27.334 INFO:tasks.ceph.mds.a.smithi090.stderr: 9: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xbc) [0x55fa5dfe51dc] 2024-03-24T01:33:27.334 INFO:tasks.ceph.mds.a.smithi090.stderr: 10: (Server::handle_client_openc(boost::intrusive_ptr const&)+0xd16) [0x55fa5dff8f26] 2024-03-24T01:33:27.334 INFO:tasks.ceph.mds.a.smithi090.stderr: 11: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6b6) [0x55fa5dfe6786] 2024-03-24T01:33:27.334 INFO:tasks.ceph.mds.a.smithi090.stderr: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55fa5dfd9f77] 2024-03-24T01:33:27.334 INFO:tasks.ceph.mds.a.smithi090.stderr: 13: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55fa5df6055b] 2024-03-24T01:33:27.335 INFO:tasks.ceph.mds.a.smithi090.stderr: 14: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55fa5df60a2c] 2024-03-24T01:33:27.335 INFO:tasks.ceph.mds.a.smithi090.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x55fa5df45465] 2024-03-24T01:33:27.335 INFO:tasks.ceph.mds.a.smithi090.stderr: 16: (DispatchQueue::entry()+0x53a) [0x7f260498009a] 2024-03-24T01:33:27.335 INFO:tasks.ceph.mds.a.smithi090.stderr: 17: /usr/lib64/ceph/libceph-common.so.2(+0x4142c1) [0x7f2604a142c1] 2024-03-24T01:33:27.335 INFO:tasks.ceph.mds.a.smithi090.stderr: 18: /lib64/libc.so.6(+0x9f802) [0x7f2603e9f802] 2024-03-24T01:33:27.335 INFO:tasks.ceph.mds.a.smithi090.stderr: 19: /lib64/libc.so.6(+0x3f450) [0x7f2603e3f450] 2024-03-24T01:33:27.335 INFO:tasks.ceph.mds.a.smithi090.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T01:33:27.335 INFO:tasks.ceph.mds.a.smithi090.stderr: 2024-03-24T01:33:27.390 INFO:tasks.ceph.mds.a.smithi090.stderr: -1> 2024-03-24T01:33:27.328+0000 7f25fd800640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el9/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f25fd800640 time 2024-03-24T01:33:27.327285+0000 2024-03-24T01:33:27.390 INFO:tasks.ceph.mds.a.smithi090.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el9/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T01:33:27.390 INFO:tasks.ceph.mds.a.smithi090.stderr: 2024-03-24T01:33:27.390 INFO:tasks.ceph.mds.a.smithi090.stderr: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T01:33:27.391 INFO:tasks.ceph.mds.a.smithi090.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f2604781fba] 2024-03-24T01:33:27.391 INFO:tasks.ceph.mds.a.smithi090.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x182178) [0x7f2604782178] 2024-03-24T01:33:27.391 INFO:tasks.ceph.mds.a.smithi090.stderr: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x402) [0x55fa5e101392] 2024-03-24T01:33:27.391 INFO:tasks.ceph.mds.a.smithi090.stderr: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3d3) [0x55fa5dfe7513] 2024-03-24T01:33:27.391 INFO:tasks.ceph.mds.a.smithi090.stderr: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xbc) [0x55fa5dfe51dc] 2024-03-24T01:33:27.391 INFO:tasks.ceph.mds.a.smithi090.stderr: 6: (Server::handle_client_openc(boost::intrusive_ptr const&)+0xd16) [0x55fa5dff8f26] 2024-03-24T01:33:27.391 INFO:tasks.ceph.mds.a.smithi090.stderr: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6b6) [0x55fa5dfe6786] 2024-03-24T01:33:27.391 INFO:tasks.ceph.mds.a.smithi090.stderr: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55fa5dfd9f77] 2024-03-24T01:33:27.391 INFO:tasks.ceph.mds.a.smithi090.stderr: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55fa5df6055b] 2024-03-24T01:33:27.391 INFO:tasks.ceph.mds.a.smithi090.stderr: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55fa5df60a2c] 2024-03-24T01:33:27.391 INFO:tasks.ceph.mds.a.smithi090.stderr: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x55fa5df45465] 2024-03-24T01:33:27.391 INFO:tasks.ceph.mds.a.smithi090.stderr: 12: (DispatchQueue::entry()+0x53a) [0x7f260498009a] 2024-03-24T01:33:27.391 INFO:tasks.ceph.mds.a.smithi090.stderr: 13: /usr/lib64/ceph/libceph-common.so.2(+0x4142c1) [0x7f2604a142c1] 2024-03-24T01:33:27.391 INFO:tasks.ceph.mds.a.smithi090.stderr: 14: /lib64/libc.so.6(+0x9f802) [0x7f2603e9f802] 2024-03-24T01:33:27.392 INFO:tasks.ceph.mds.a.smithi090.stderr: 15: /lib64/libc.so.6(+0x3f450) [0x7f2603e3f450] 2024-03-24T01:33:27.392 INFO:tasks.ceph.mds.a.smithi090.stderr: 2024-03-24T01:33:27.392 INFO:tasks.ceph.mds.a.smithi090.stderr: 0> 2024-03-24T01:33:27.331+0000 7f25fd800640 -1 *** Caught signal (Aborted) ** 2024-03-24T01:33:27.392 INFO:tasks.ceph.mds.a.smithi090.stderr: in thread 7f25fd800640 thread_name:ms_dispatch 2024-03-24T01:33:27.392 INFO:tasks.ceph.mds.a.smithi090.stderr: 2024-03-24T01:33:27.392 INFO:tasks.ceph.mds.a.smithi090.stderr: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T01:33:27.392 INFO:tasks.ceph.mds.a.smithi090.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f2603e54db0] 2024-03-24T01:33:27.392 INFO:tasks.ceph.mds.a.smithi090.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f2603ea154c] 2024-03-24T01:33:27.392 INFO:tasks.ceph.mds.a.smithi090.stderr: 3: raise() 2024-03-24T01:33:27.392 INFO:tasks.ceph.mds.a.smithi090.stderr: 4: abort() 2024-03-24T01:33:27.392 INFO:tasks.ceph.mds.a.smithi090.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f2604782014] 2024-03-24T01:33:27.392 INFO:tasks.ceph.mds.a.smithi090.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x182178) [0x7f2604782178] 2024-03-24T01:33:27.392 INFO:tasks.ceph.mds.a.smithi090.stderr: 7: (Locker::set_xlocks_done(MutationImpl*, bool)+0x402) [0x55fa5e101392] 2024-03-24T01:33:27.392 INFO:tasks.ceph.mds.a.smithi090.stderr: 8: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3d3) [0x55fa5dfe7513] 2024-03-24T01:33:27.393 INFO:tasks.ceph.mds.a.smithi090.stderr: 9: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xbc) [0x55fa5dfe51dc] 2024-03-24T01:33:27.393 INFO:tasks.ceph.mds.a.smithi090.stderr: 10: (Server::handle_client_openc(boost::intrusive_ptr const&)+0xd16) [0x55fa5dff8f26] 2024-03-24T01:33:27.393 INFO:tasks.ceph.mds.a.smithi090.stderr: 11: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6b6) [0x55fa5dfe6786] 2024-03-24T01:33:27.393 INFO:tasks.ceph.mds.a.smithi090.stderr: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55fa5dfd9f77] 2024-03-24T01:33:27.393 INFO:tasks.ceph.mds.a.smithi090.stderr: 13: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55fa5df6055b] 2024-03-24T01:33:27.393 INFO:tasks.ceph.mds.a.smithi090.stderr: 14: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55fa5df60a2c] 2024-03-24T01:33:27.393 INFO:tasks.ceph.mds.a.smithi090.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x55fa5df45465] 2024-03-24T01:33:27.393 INFO:tasks.ceph.mds.a.smithi090.stderr: 16: (DispatchQueue::entry()+0x53a) [0x7f260498009a] 2024-03-24T01:33:27.393 INFO:tasks.ceph.mds.a.smithi090.stderr: 17: /usr/lib64/ceph/libceph-common.so.2(+0x4142c1) [0x7f2604a142c1] 2024-03-24T01:33:27.393 INFO:tasks.ceph.mds.a.smithi090.stderr: 18: /lib64/libc.so.6(+0x9f802) [0x7f2603e9f802] 2024-03-24T01:33:27.393 INFO:tasks.ceph.mds.a.smithi090.stderr: 19: /lib64/libc.so.6(+0x3f450) [0x7f2603e3f450] 2024-03-24T01:33:27.393 INFO:tasks.ceph.mds.a.smithi090.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T01:33:27.393 INFO:tasks.ceph.mds.a.smithi090.stderr: 2024-03-24T01:33:27.410 INFO:tasks.ceph.mds.a.smithi090.stderr: -1> 2024-03-24T01:33:27.328+0000 7f25fd800640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el9/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f25fd800640 time 2024-03-24T01:33:27.327285+0000 2024-03-24T01:33:27.410 INFO:tasks.ceph.mds.a.smithi090.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el9/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T01:33:27.410 INFO:tasks.ceph.mds.a.smithi090.stderr: 2024-03-24T01:33:27.410 INFO:tasks.ceph.mds.a.smithi090.stderr: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T01:33:27.410 INFO:tasks.ceph.mds.a.smithi090.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f2604781fba] 2024-03-24T01:33:27.410 INFO:tasks.ceph.mds.a.smithi090.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x182178) [0x7f2604782178] 2024-03-24T01:33:27.410 INFO:tasks.ceph.mds.a.smithi090.stderr: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x402) [0x55fa5e101392] 2024-03-24T01:33:27.411 INFO:tasks.ceph.mds.a.smithi090.stderr: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3d3) [0x55fa5dfe7513] 2024-03-24T01:33:27.411 INFO:tasks.ceph.mds.a.smithi090.stderr: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xbc) [0x55fa5dfe51dc] 2024-03-24T01:33:27.411 INFO:tasks.ceph.mds.a.smithi090.stderr: 6: (Server::handle_client_openc(boost::intrusive_ptr const&)+0xd16) [0x55fa5dff8f26] 2024-03-24T01:33:27.411 INFO:tasks.ceph.mds.a.smithi090.stderr: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6b6) [0x55fa5dfe6786] 2024-03-24T01:33:27.411 INFO:tasks.ceph.mds.a.smithi090.stderr: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55fa5dfd9f77] 2024-03-24T01:33:27.411 INFO:tasks.ceph.mds.a.smithi090.stderr: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55fa5df6055b] 2024-03-24T01:33:27.411 INFO:tasks.ceph.mds.a.smithi090.stderr: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55fa5df60a2c] 2024-03-24T01:33:27.411 INFO:tasks.ceph.mds.a.smithi090.stderr: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x55fa5df45465] 2024-03-24T01:33:27.411 INFO:tasks.ceph.mds.a.smithi090.stderr: 12: (DispatchQueue::entry()+0x53a) [0x7f260498009a] 2024-03-24T01:33:27.411 INFO:tasks.ceph.mds.a.smithi090.stderr: 13: /usr/lib64/ceph/libceph-common.so.2(+0x4142c1) [0x7f2604a142c1] 2024-03-24T01:33:27.411 INFO:tasks.ceph.mds.a.smithi090.stderr: 14: /lib64/libc.so.6(+0x9f802) [0x7f2603e9f802] 2024-03-24T01:33:27.411 INFO:tasks.ceph.mds.a.smithi090.stderr: 15: /lib64/libc.so.6(+0x3f450) [0x7f2603e3f450] 2024-03-24T01:33:27.411 INFO:tasks.ceph.mds.a.smithi090.stderr: 2024-03-24T01:33:27.411 INFO:tasks.ceph.mds.a.smithi090.stderr: 0> 2024-03-24T01:33:27.331+0000 7f25fd800640 -1 *** Caught signal (Aborted) ** 2024-03-24T01:33:27.412 INFO:tasks.ceph.mds.a.smithi090.stderr: in thread 7f25fd800640 thread_name:ms_dispatch 2024-03-24T01:33:27.412 INFO:tasks.ceph.mds.a.smithi090.stderr: 2024-03-24T01:33:27.412 INFO:tasks.ceph.mds.a.smithi090.stderr: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T01:33:27.412 INFO:tasks.ceph.mds.a.smithi090.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f2603e54db0] 2024-03-24T01:33:27.412 INFO:tasks.ceph.mds.a.smithi090.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f2603ea154c] 2024-03-24T01:33:27.412 INFO:tasks.ceph.mds.a.smithi090.stderr: 3: raise() 2024-03-24T01:33:27.412 INFO:tasks.ceph.mds.a.smithi090.stderr: 4: abort() 2024-03-24T01:33:27.412 INFO:tasks.ceph.mds.a.smithi090.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f2604782014] 2024-03-24T01:33:27.412 INFO:tasks.ceph.mds.a.smithi090.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x182178) [0x7f2604782178] 2024-03-24T01:33:27.412 INFO:tasks.ceph.mds.a.smithi090.stderr: 7: (Locker::set_xlocks_done(MutationImpl*, bool)+0x402) [0x55fa5e101392] 2024-03-24T01:33:27.412 INFO:tasks.ceph.mds.a.smithi090.stderr: 8: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3d3) [0x55fa5dfe7513] 2024-03-24T01:33:27.412 INFO:tasks.ceph.mds.a.smithi090.stderr: 9: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xbc) [0x55fa5dfe51dc] 2024-03-24T01:33:27.412 INFO:tasks.ceph.mds.a.smithi090.stderr: 10: (Server::handle_client_openc(boost::intrusive_ptr const&)+0xd16) [0x55fa5dff8f26] 2024-03-24T01:33:27.412 INFO:tasks.ceph.mds.a.smithi090.stderr: 11: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6b6) [0x55fa5dfe6786] 2024-03-24T01:33:27.412 INFO:tasks.ceph.mds.a.smithi090.stderr: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55fa5dfd9f77] 2024-03-24T01:33:27.413 INFO:tasks.ceph.mds.a.smithi090.stderr: 13: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55fa5df6055b] 2024-03-24T01:33:27.413 INFO:tasks.ceph.mds.a.smithi090.stderr: 14: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55fa5df60a2c] 2024-03-24T01:33:27.413 INFO:tasks.ceph.mds.a.smithi090.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x55fa5df45465] 2024-03-24T01:33:27.413 INFO:tasks.ceph.mds.a.smithi090.stderr: 16: (DispatchQueue::entry()+0x53a) [0x7f260498009a] 2024-03-24T01:33:27.413 INFO:tasks.ceph.mds.a.smithi090.stderr: 17: /usr/lib64/ceph/libceph-common.so.2(+0x4142c1) [0x7f2604a142c1] 2024-03-24T01:33:27.413 INFO:tasks.ceph.mds.a.smithi090.stderr: 18: /lib64/libc.so.6(+0x9f802) [0x7f2603e9f802] 2024-03-24T01:33:27.413 INFO:tasks.ceph.mds.a.smithi090.stderr: 19: /lib64/libc.so.6(+0x3f450) [0x7f2603e3f450] 2024-03-24T01:33:27.413 INFO:tasks.ceph.mds.a.smithi090.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T01:33:27.413 INFO:tasks.ceph.mds.a.smithi090.stderr: 2024-03-24T01:33:27.694 INFO:tasks.ceph.mds.a.smithi090.stderr:daemon-helper: command crashed with signal 6 2024-03-24T01:33:30.361 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:33:30.369 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:33:30.898 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-03-24T01:33:37.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~7s 2024-03-24T01:33:44.120 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~13s 2024-03-24T01:33:50.731 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~20s 2024-03-24T01:33:57.343 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~26s 2024-03-24T01:34:00.501 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:34:00.510 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:34:03.955 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~33s 2024-03-24T01:34:10.567 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~40s 2024-03-24T01:34:17.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~46s 2024-03-24T01:34:23.789 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~53s 2024-03-24T01:34:30.399 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~60s 2024-03-24T01:34:30.634 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:34:30.642 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:34:37.010 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~66s 2024-03-24T01:34:43.622 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~73s 2024-03-24T01:34:50.232 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~79s 2024-03-24T01:34:56.846 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~86s 2024-03-24T01:35:00.913 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:35:00.921 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:35:03.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~93s 2024-03-24T01:35:10.068 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~99s 2024-03-24T01:35:16.681 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~106s 2024-03-24T01:35:23.292 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~112s 2024-03-24T01:35:29.904 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~119s 2024-03-24T01:35:31.045 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:35:31.054 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:35:36.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~126s 2024-03-24T01:35:43.127 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~132s 2024-03-24T01:35:49.739 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~139s 2024-03-24T01:35:56.351 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~145s 2024-03-24T01:36:01.179 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:36:01.187 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:36:02.962 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~152s 2024-03-24T01:36:09.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~159s 2024-03-24T01:36:16.185 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~165s 2024-03-24T01:36:22.796 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~172s 2024-03-24T01:36:29.407 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~179s 2024-03-24T01:36:31.429 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:36:31.437 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:36:36.019 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~185s 2024-03-24T01:36:42.630 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~192s 2024-03-24T01:36:49.241 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~198s 2024-03-24T01:36:55.853 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~205s 2024-03-24T01:37:01.559 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:37:01.567 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:37:02.465 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~212s 2024-03-24T01:37:09.076 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~218s 2024-03-24T01:37:15.687 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~225s 2024-03-24T01:37:22.299 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~231s 2024-03-24T01:37:28.911 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~238s 2024-03-24T01:37:31.691 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:37:31.699 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:37:35.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~245s 2024-03-24T01:37:42.133 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~251s 2024-03-24T01:37:48.744 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~258s 2024-03-24T01:37:55.354 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~264s 2024-03-24T01:38:01.829 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:38:01.837 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:38:01.963 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~271s 2024-03-24T01:38:08.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~278s 2024-03-24T01:38:15.185 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~284s 2024-03-24T01:38:21.797 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~291s 2024-03-24T01:38:28.410 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~298s 2024-03-24T01:38:31.967 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:38:31.976 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:38:35.021 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~304s 2024-03-24T01:38:35.021 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-03-24T01:38:35.022 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:38:35.022 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:38:35.079 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:38:35.089 INFO:teuthology.orchestra.run.smithi090.stderr:Traceback (most recent call last): 2024-03-24T01:38:35.089 INFO:teuthology.orchestra.run.smithi090.stderr: File "", line 20, in 2024-03-24T01:38:35.089 INFO:teuthology.orchestra.run.smithi090.stderr:ConnectionAbortedError: [Errno 103] Software caused connection abort: '/home/ubuntu/cephtest/mnt.0/bravo/file_1' 2024-03-24T01:38:35.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2024-03-24T01:38:35.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T01:38:35.107 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T01:38:35.107 DEBUG:teuthology.orchestra.run.smithi090:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T01:38:35.248 DEBUG:teuthology.orchestra.run.smithi120:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T01:38:35.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.276+0000 7f962b200640 1 Processor -- start 2024-03-24T01:38:35.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.276+0000 7f962b200640 1 -- start start 2024-03-24T01:38:35.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.277+0000 7f962b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f962c158280 0x7f962c0b3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:35.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.277+0000 7f962b200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f962c0b3670 con 0x7f962c15ae10 2024-03-24T01:38:35.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.277+0000 7f962b200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f962c15b2d0 con 0x7f962c06e900 2024-03-24T01:38:35.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.277+0000 7f962b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f962c15b5b0 con 0x7f962c158280 2024-03-24T01:38:35.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.278+0000 7f9623e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f962c15ae10 0x7f962c0cccd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:37076/0 (socket says 172.21.15.90:37076) 2024-03-24T01:38:35.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.278+0000 7f9623e00640 1 -- 172.21.15.90:0/3456214680 learned_addr learned my addr 172.21.15.90:0/3456214680 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:38:35.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.278+0000 7f9629e00640 1 --2- 172.21.15.90:0/3456214680 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f962c158280 0x7f962c0b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:35.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.278+0000 7f9629400640 1 -- 172.21.15.90:0/3456214680 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2397712215 0 0) 0x7f962c0b3670 con 0x7f962c15ae10 2024-03-24T01:38:35.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.278+0000 7f9629400640 1 -- 172.21.15.90:0/3456214680 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9610003520 con 0x7f962c15ae10 2024-03-24T01:38:35.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.278+0000 7f9629e00640 1 -- 172.21.15.90:0/3456214680 >> v1:172.21.15.120:6789/0 conn(0x7f962c06e900 legacy=0x7f962c071be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:35.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.278+0000 7f9629e00640 1 -- 172.21.15.90:0/3456214680 >> v1:172.21.15.90:6789/0 conn(0x7f962c15ae10 legacy=0x7f962c0cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:35.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.278+0000 7f9629e00640 1 -- 172.21.15.90:0/3456214680 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f962c15b890 con 0x7f962c158280 2024-03-24T01:38:35.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.279+0000 7f9629e00640 1 --2- 172.21.15.90:0/3456214680 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f962c158280 0x7f962c0b3130 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f961c0097d0 tx=0x7f961c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b5ec45567dcdef1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:35.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.280+0000 7f9629400640 1 -- 172.21.15.90:0/3456214680 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f961c035020 con 0x7f962c158280 2024-03-24T01:38:35.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.280+0000 7f9629400640 1 -- 172.21.15.90:0/3456214680 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f961c031890 con 0x7f962c158280 2024-03-24T01:38:35.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.280+0000 7f9629400640 1 -- 172.21.15.90:0/3456214680 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f961c031c60 con 0x7f962c158280 2024-03-24T01:38:35.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.280+0000 7f962b200640 1 -- 172.21.15.90:0/3456214680 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f962c158280 msgr2=0x7f962c0b3130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:35.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.280+0000 7f962b200640 1 --2- 172.21.15.90:0/3456214680 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f962c158280 0x7f962c0b3130 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f961c0097d0 tx=0x7f961c02b930 comp rx=0 tx=0).stop 2024-03-24T01:38:35.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.280+0000 7f962b200640 1 -- 172.21.15.90:0/3456214680 shutdown_connections 2024-03-24T01:38:35.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.280+0000 7f962b200640 1 --2- 172.21.15.90:0/3456214680 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f962c158280 0x7f962c0b3130 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:35.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.280+0000 7f962b200640 1 -- 172.21.15.90:0/3456214680 >> 172.21.15.90:0/3456214680 conn(0x7f962c06b0c0 msgr2=0x7f962c06b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:35.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.280+0000 7f962b200640 1 -- 172.21.15.90:0/3456214680 shutdown_connections 2024-03-24T01:38:35.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.280+0000 7f962b200640 1 -- 172.21.15.90:0/3456214680 wait complete. 2024-03-24T01:38:35.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.282+0000 7f962b200640 1 Processor -- start 2024-03-24T01:38:35.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.282+0000 7f962b200640 1 -- start start 2024-03-24T01:38:35.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.283+0000 7f962b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f962c06e900 0x7f962c140060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:35.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.283+0000 7f962b200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f962c158280 0x7f962c155ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:35.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.283+0000 7f962b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f962c15ae10 0x7f962c156000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:35.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.283+0000 7f962b200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f962c158660 con 0x7f962c15ae10 2024-03-24T01:38:35.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.283+0000 7f962b200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f962c0cd3e0 con 0x7f962c158280 2024-03-24T01:38:35.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.283+0000 7f962b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f962c1587e0 con 0x7f962c06e900 2024-03-24T01:38:35.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.284+0000 7f962a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f962c15ae10 0x7f962c156000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:35.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.284+0000 7f9623e00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f962c158280 0x7f962c155ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:35.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.284+0000 7f9629e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f962c06e900 0x7f962c140060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:35.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.284+0000 7f962a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f962c15ae10 0x7f962c156000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38220/0 (socket says 172.21.15.90:38220) 2024-03-24T01:38:35.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.284+0000 7f962a800640 1 -- 172.21.15.90:0/1372912275 learned_addr learned my addr 172.21.15.90:0/1372912275 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:38:35.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.284+0000 7f962a800640 1 -- 172.21.15.90:0/1372912275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f962c06e900 msgr2=0x7f962c140060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:35.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.284+0000 7f962a800640 1 --2- 172.21.15.90:0/1372912275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f962c06e900 0x7f962c140060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:35.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.284+0000 7f962a800640 1 -- 172.21.15.90:0/1372912275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f962c158280 msgr2=0x7f962c155ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:35.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.284+0000 7f962a800640 1 --2- 172.21.15.90:0/1372912275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f962c158280 0x7f962c155ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:35.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.284+0000 7f962a800640 1 -- 172.21.15.90:0/1372912275 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f962c1f4c50 con 0x7f962c15ae10 2024-03-24T01:38:35.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.284+0000 7f962a800640 1 --2- 172.21.15.90:0/1372912275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f962c15ae10 0x7f962c156000 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f962400c7a0 tx=0x7f962400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:35.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.285+0000 7f9622000640 1 -- 172.21.15.90:0/1372912275 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9624003b70 con 0x7f962c15ae10 2024-03-24T01:38:35.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.285+0000 7f9622000640 1 -- 172.21.15.90:0/1372912275 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9624003d30 con 0x7f962c15ae10 2024-03-24T01:38:35.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.285+0000 7f9622000640 1 -- 172.21.15.90:0/1372912275 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962400aa30 con 0x7f962c15ae10 2024-03-24T01:38:35.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.285+0000 7f962b200640 1 -- 172.21.15.90:0/1372912275 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f962c1f4f30 con 0x7f962c15ae10 2024-03-24T01:38:35.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.285+0000 7f962b200640 1 -- 172.21.15.90:0/1372912275 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f962c1f54f0 con 0x7f962c15ae10 2024-03-24T01:38:35.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.286+0000 7f962b200640 1 -- 172.21.15.90:0/1372912275 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f962c157a80 con 0x7f962c15ae10 2024-03-24T01:38:35.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.287+0000 7f9622000640 1 -- 172.21.15.90:0/1372912275 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f962400abd0 con 0x7f962c15ae10 2024-03-24T01:38:35.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.288+0000 7f9622000640 1 --2- 172.21.15.90:0/1372912275 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f960806bbd0 0x7f960806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:35.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.288+0000 7f9622000640 1 -- 172.21.15.90:0/1372912275 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7f962408a9f0 con 0x7f962c15ae10 2024-03-24T01:38:35.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.289+0000 7f9629e00640 1 --2- 172.21.15.90:0/1372912275 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f960806bbd0 0x7f960806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:35.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.289+0000 7f9629e00640 1 --2- 172.21.15.90:0/1372912275 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f960806bbd0 0x7f960806e090 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f961c036000 tx=0x7f961c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:35.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.294+0000 7f9622000640 1 -- 172.21.15.90:0/1372912275 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f962c157a80 con 0x7f962c15ae10 2024-03-24T01:38:35.352 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T01:38:35.352 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:38:35.353 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:38:35.401 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:38:35.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.493+0000 7f962b200640 1 -- 172.21.15.90:0/1372912275 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f962c04f400 con 0x7f962c15ae10 2024-03-24T01:38:35.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.495+0000 7f9622000640 1 -- 172.21.15.90:0/1372912275 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+17101 (secure 0 0 0) 0x7f9624058c60 con 0x7f962c15ae10 2024-03-24T01:38:35.496 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:38:35.496 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":69,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:32:54.821325+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:32:42.026147+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-24T01:32:43.180712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.1e","mappings":[{"from":7,"to":2}]},{"pgid":"10.10","mappings":[{"from":4,"to":3}]},{"pgid":"10.1d","mappings":[{"from":4,"to":3}]},{"pgid":"10.1f","mappings":[{"from":7,"to":3}]},{"pgid":"10.37","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:38:35.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.498+0000 7f962b200640 1 -- 172.21.15.90:0/1372912275 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f960806bbd0 msgr2=0x7f960806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:35.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.498+0000 7f962b200640 1 --2- 172.21.15.90:0/1372912275 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f960806bbd0 0x7f960806e090 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f961c036000 tx=0x7f961c008030 comp rx=0 tx=0).stop 2024-03-24T01:38:35.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.498+0000 7f962b200640 1 -- 172.21.15.90:0/1372912275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f962c15ae10 msgr2=0x7f962c156000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:35.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.498+0000 7f962b200640 1 --2- 172.21.15.90:0/1372912275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f962c15ae10 0x7f962c156000 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f962400c7a0 tx=0x7f962400cc60 comp rx=0 tx=0).stop 2024-03-24T01:38:35.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.498+0000 7f962b200640 1 -- 172.21.15.90:0/1372912275 shutdown_connections 2024-03-24T01:38:35.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.498+0000 7f962b200640 1 --2- 172.21.15.90:0/1372912275 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f960806bbd0 0x7f960806e090 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:35.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.498+0000 7f962b200640 1 --2- 172.21.15.90:0/1372912275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f962c15ae10 0x7f962c156000 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:35.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.498+0000 7f962b200640 1 --2- 172.21.15.90:0/1372912275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f962c158280 0x7f962c155ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:35.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.498+0000 7f962b200640 1 --2- 172.21.15.90:0/1372912275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f962c06e900 0x7f962c140060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:35.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.498+0000 7f962b200640 1 -- 172.21.15.90:0/1372912275 >> 172.21.15.90:0/1372912275 conn(0x7f962c06b0c0 msgr2=0x7f962c0b3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:35.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.498+0000 7f962b200640 1 -- 172.21.15.90:0/1372912275 shutdown_connections 2024-03-24T01:38:35.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.498+0000 7f962b200640 1 -- 172.21.15.90:0/1372912275 wait complete. 2024-03-24T01:38:35.521 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T01:32:42.026147+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T01:32:43.180712+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T01:38:35.521 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:38:35.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.639+0000 7f0b97c00640 1 Processor -- start 2024-03-24T01:38:35.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.639+0000 7f0b97c00640 1 -- start start 2024-03-24T01:38:35.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.640+0000 7f0b97c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0b900cc150 0x7f0b900ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:35.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.640+0000 7f0b97c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b900cd490 con 0x7f0b901603e0 2024-03-24T01:38:35.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.640+0000 7f0b97c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b9015f900 con 0x7f0b9006b7d0 2024-03-24T01:38:35.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.640+0000 7f0b97c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b9015fbe0 con 0x7f0b900cc150 2024-03-24T01:38:35.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.641+0000 7f0b95200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0b900cc150 0x7f0b900ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:35.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.641+0000 7f0b95200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0b900cc150 0x7f0b900ccb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38222/0 (socket says 172.21.15.90:38222) 2024-03-24T01:38:35.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.641+0000 7f0b95200640 1 -- 172.21.15.90:0/745463134 learned_addr learned my addr 172.21.15.90:0/745463134 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:38:35.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.641+0000 7f0b95200640 1 -- 172.21.15.90:0/745463134 >> v1:172.21.15.120:6789/0 conn(0x7f0b9006b7d0 legacy=0x7f0b900b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:38:35.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.641+0000 7f0b95200640 1 -- 172.21.15.90:0/745463134 >> v1:172.21.15.90:6789/0 conn(0x7f0b901603e0 legacy=0x7f0b9006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:35.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.641+0000 7f0b95200640 1 -- 172.21.15.90:0/745463134 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b9015fec0 con 0x7f0b900cc150 2024-03-24T01:38:35.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.641+0000 7f0b8f400640 1 -- 172.21.15.90:0/745463134 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 273340584 0 0) 0x7f0b900cd490 con 0x7f0b901603e0 2024-03-24T01:38:35.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.642+0000 7f0b95200640 1 --2- 172.21.15.90:0/745463134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b900cc150 0x7f0b900ccb60 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f0b84009870 tx=0x7f0b8402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7a6367a83f02ea9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:35.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.643+0000 7f0b8f400640 1 -- 172.21.15.90:0/745463134 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b84034070 con 0x7f0b900cc150 2024-03-24T01:38:35.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.643+0000 7f0b8f400640 1 -- 172.21.15.90:0/745463134 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b84030860 con 0x7f0b900cc150 2024-03-24T01:38:35.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.643+0000 7f0b8f400640 1 -- 172.21.15.90:0/745463134 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b84030c30 con 0x7f0b900cc150 2024-03-24T01:38:35.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.644+0000 7f0b97c00640 1 -- 172.21.15.90:0/745463134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b900cc150 msgr2=0x7f0b900ccb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:35.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.644+0000 7f0b97c00640 1 --2- 172.21.15.90:0/745463134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b900cc150 0x7f0b900ccb60 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f0b84009870 tx=0x7f0b8402b9a0 comp rx=0 tx=0).stop 2024-03-24T01:38:35.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.644+0000 7f0b97c00640 1 -- 172.21.15.90:0/745463134 shutdown_connections 2024-03-24T01:38:35.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.644+0000 7f0b97c00640 1 --2- 172.21.15.90:0/745463134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b900cc150 0x7f0b900ccb60 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:35.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.644+0000 7f0b97c00640 1 -- 172.21.15.90:0/745463134 >> 172.21.15.90:0/745463134 conn(0x7f0b900b1770 msgr2=0x7f0b900b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:35.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.644+0000 7f0b97c00640 1 -- 172.21.15.90:0/745463134 shutdown_connections 2024-03-24T01:38:35.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.644+0000 7f0b97c00640 1 -- 172.21.15.90:0/745463134 wait complete. 2024-03-24T01:38:35.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.645+0000 7f0b97c00640 1 Processor -- start 2024-03-24T01:38:35.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.646+0000 7f0b97c00640 1 -- start start 2024-03-24T01:38:35.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.646+0000 7f0b97c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b9006b7d0 0x7f0b9013c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:35.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.647+0000 7f0b97c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b900cc150 0x7f0b9006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:35.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.647+0000 7f0b97c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b901603e0 0x7f0b901f74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:35.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.647+0000 7f0b97c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b901455b0 con 0x7f0b900cc150 2024-03-24T01:38:35.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.647+0000 7f0b97c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b90145430 con 0x7f0b901603e0 2024-03-24T01:38:35.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.647+0000 7f0b97c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b900b3e60 con 0x7f0b9006b7d0 2024-03-24T01:38:35.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.647+0000 7f0b8fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b900cc150 0x7f0b9006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:35.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.647+0000 7f0b8fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b900cc150 0x7f0b9006dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38224/0 (socket says 172.21.15.90:38224) 2024-03-24T01:38:35.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.647+0000 7f0b8fe00640 1 -- 172.21.15.90:0/3710250540 learned_addr learned my addr 172.21.15.90:0/3710250540 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:38:35.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.647+0000 7f0b95c00640 1 --2- 172.21.15.90:0/3710250540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b901603e0 0x7f0b901f74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:35.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.647+0000 7f0b95200640 1 --2- 172.21.15.90:0/3710250540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b9006b7d0 0x7f0b9013c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:35.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.647+0000 7f0b8fe00640 1 -- 172.21.15.90:0/3710250540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b9006b7d0 msgr2=0x7f0b9013c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:35.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.647+0000 7f0b8fe00640 1 --2- 172.21.15.90:0/3710250540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b9006b7d0 0x7f0b9013c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:35.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.647+0000 7f0b8fe00640 1 -- 172.21.15.90:0/3710250540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b901603e0 msgr2=0x7f0b901f74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:35.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.647+0000 7f0b8fe00640 1 --2- 172.21.15.90:0/3710250540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b901603e0 0x7f0b901f74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:35.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.647+0000 7f0b8fe00640 1 -- 172.21.15.90:0/3710250540 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b901f7c40 con 0x7f0b900cc150 2024-03-24T01:38:35.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.647+0000 7f0b95c00640 1 --2- 172.21.15.90:0/3710250540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b901603e0 0x7f0b901f74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:38:35.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.648+0000 7f0b8fe00640 1 --2- 172.21.15.90:0/3710250540 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b900cc150 0x7f0b9006dd50 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f0b7c00c410 tx=0x7f0b7c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:35.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.648+0000 7f0b8cc00640 1 -- 172.21.15.90:0/3710250540 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7c017070 con 0x7f0b900cc150 2024-03-24T01:38:35.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.648+0000 7f0b8cc00640 1 -- 172.21.15.90:0/3710250540 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b7c0035f0 con 0x7f0b900cc150 2024-03-24T01:38:35.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.648+0000 7f0b8cc00640 1 -- 172.21.15.90:0/3710250540 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7c003a00 con 0x7f0b900cc150 2024-03-24T01:38:35.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.648+0000 7f0b97c00640 1 -- 172.21.15.90:0/3710250540 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b901f7f20 con 0x7f0b900cc150 2024-03-24T01:38:35.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.648+0000 7f0b97c00640 1 -- 172.21.15.90:0/3710250540 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b900acf00 con 0x7f0b900cc150 2024-03-24T01:38:35.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.650+0000 7f0b97c00640 1 -- 172.21.15.90:0/3710250540 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b90068060 con 0x7f0b900cc150 2024-03-24T01:38:35.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.650+0000 7f0b8cc00640 1 -- 172.21.15.90:0/3710250540 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f0b7c01a070 con 0x7f0b900cc150 2024-03-24T01:38:35.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.651+0000 7f0b8cc00640 1 --2- 172.21.15.90:0/3710250540 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0b7006bc00 0x7f0b7006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:35.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.651+0000 7f0b8cc00640 1 -- 172.21.15.90:0/3710250540 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7f0b7c08a140 con 0x7f0b900cc150 2024-03-24T01:38:35.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.652+0000 7f0b95200640 1 --2- 172.21.15.90:0/3710250540 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0b7006bc00 0x7f0b7006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:35.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.653+0000 7f0b95200640 1 --2- 172.21.15.90:0/3710250540 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0b7006bc00 0x7f0b7006e0c0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f0b84036000 tx=0x7f0b8403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:35.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.655+0000 7f0b8cc00640 1 -- 172.21.15.90:0/3710250540 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f0b7c058970 con 0x7f0b900cc150 2024-03-24T01:38:35.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.710+0000 7fc1aa000640 1 Processor -- start 2024-03-24T01:38:35.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.710+0000 7fc1aa000640 1 -- start start 2024-03-24T01:38:35.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.711+0000 7fc1aa000640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc1a40aeeb0 0x7fc1a40b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:35.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.711+0000 7fc1aa000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1a406b7d0 0x7fc1a40b2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:35.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.711+0000 7fc1aa000640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1a406dac0 con 0x7fc1a415bc50 2024-03-24T01:38:35.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.711+0000 7fc1aa000640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1a40b4da0 con 0x7fc1a40aeeb0 2024-03-24T01:38:35.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.711+0000 7fc1aa000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a406dda0 con 0x7fc1a406b7d0 2024-03-24T01:38:35.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.712+0000 7fc1a3400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc1a40aeeb0 0x7fc1a40b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:35.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.712+0000 7fc1a3e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1a406b7d0 0x7fc1a40b2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:35.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.712+0000 7fc19aa00640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fc1a415bc50 0x7fc1a406b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.90:56068/0 (socket says 172.21.15.90:56068) 2024-03-24T01:38:35.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.712+0000 7fc1a3400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fc1a40aeeb0 0x7fc1a40b44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.90:39196/0 (socket says 172.21.15.90:39196) 2024-03-24T01:38:35.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.712+0000 7fc19aa00640 1 -- 172.21.15.90:0/729322567 learned_addr learned my addr 172.21.15.90:0/729322567 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:38:35.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.712+0000 7fc1a2a00640 1 -- 172.21.15.90:0/729322567 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3748544987 0 0) 0x7fc1a406dac0 con 0x7fc1a415bc50 2024-03-24T01:38:35.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.713+0000 7fc1a2a00640 1 -- 172.21.15.90:0/729322567 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc188003520 con 0x7fc1a415bc50 2024-03-24T01:38:35.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.713+0000 7fc1a3400640 1 -- 172.21.15.90:0/729322567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1a406b7d0 msgr2=0x7fc1a40b2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:38:35.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.713+0000 7fc1a3400640 1 --2- 172.21.15.90:0/729322567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1a406b7d0 0x7fc1a40b2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:35.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.713+0000 7fc1a3400640 1 -- 172.21.15.90:0/729322567 >> v1:172.21.15.120:6789/0 conn(0x7fc1a415bc50 legacy=0x7fc1a406b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:35.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.713+0000 7fc1a3400640 1 -- 172.21.15.90:0/729322567 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a406e080 con 0x7fc1a40aeeb0 2024-03-24T01:38:35.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.714+0000 7fc1a3400640 1 --2- 172.21.15.90:0/729322567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1a40aeeb0 0x7fc1a40b44c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc1940097a0 tx=0x7fc19402b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b85b21a5ecb763b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:35.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.714+0000 7fc1a2a00640 1 -- 172.21.15.90:0/729322567 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc194038070 con 0x7fc1a40aeeb0 2024-03-24T01:38:35.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.714+0000 7fc1a2a00640 1 -- 172.21.15.90:0/729322567 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc194031700 con 0x7fc1a40aeeb0 2024-03-24T01:38:35.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.714+0000 7fc1a2a00640 1 -- 172.21.15.90:0/729322567 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc194031ad0 con 0x7fc1a40aeeb0 2024-03-24T01:38:35.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.715+0000 7fc1aa000640 1 -- 172.21.15.90:0/729322567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1a40aeeb0 msgr2=0x7fc1a40b44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:35.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.715+0000 7fc1aa000640 1 --2- 172.21.15.90:0/729322567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1a40aeeb0 0x7fc1a40b44c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc1940097a0 tx=0x7fc19402b7a0 comp rx=0 tx=0).stop 2024-03-24T01:38:35.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.715+0000 7fc1aa000640 1 -- 172.21.15.90:0/729322567 shutdown_connections 2024-03-24T01:38:35.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.715+0000 7fc1aa000640 1 --2- 172.21.15.90:0/729322567 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1a406b7d0 0x7fc1a40b2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:35.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.715+0000 7fc1aa000640 1 --2- 172.21.15.90:0/729322567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1a40aeeb0 0x7fc1a40b44c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:35.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.715+0000 7fc1aa000640 1 -- 172.21.15.90:0/729322567 >> 172.21.15.90:0/729322567 conn(0x7fc1a40b1770 msgr2=0x7fc1a40b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:35.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.715+0000 7fc1aa000640 1 -- 172.21.15.90:0/729322567 shutdown_connections 2024-03-24T01:38:35.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.715+0000 7fc1aa000640 1 -- 172.21.15.90:0/729322567 wait complete. 2024-03-24T01:38:35.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.717+0000 7fc1aa000640 1 Processor -- start 2024-03-24T01:38:35.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.717+0000 7fc1aa000640 1 -- start start 2024-03-24T01:38:35.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.718+0000 7fc1aa000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1a406b7d0 0x7fc1a41509f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:35.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.718+0000 7fc1aa000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1a40aeeb0 0x7fc1a40ad940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:35.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.718+0000 7fc1aa000640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1a415bc50 0x7fc1a414ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:35.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.718+0000 7fc1aa000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a40b4a00 con 0x7fc1a406b7d0 2024-03-24T01:38:35.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.718+0000 7fc1aa000640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a40af290 con 0x7fc1a415bc50 2024-03-24T01:38:35.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.718+0000 7fc1aa000640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a40b4b80 con 0x7fc1a40aeeb0 2024-03-24T01:38:35.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.718+0000 7fc1a3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1a406b7d0 0x7fc1a41509f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:35.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.718+0000 7fc1a3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1a406b7d0 0x7fc1a41509f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38240/0 (socket says 172.21.15.90:38240) 2024-03-24T01:38:35.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.718+0000 7fc1a3400640 1 -- 172.21.15.90:0/3586672334 learned_addr learned my addr 172.21.15.90:0/3586672334 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:38:35.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.719+0000 7fc19aa00640 1 --2- 172.21.15.90:0/3586672334 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1a40aeeb0 0x7fc1a40ad940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:35.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.719+0000 7fc1a3e00640 1 --2- 172.21.15.90:0/3586672334 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1a415bc50 0x7fc1a414ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:35.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.719+0000 7fc1a3400640 1 -- 172.21.15.90:0/3586672334 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1a40aeeb0 msgr2=0x7fc1a40ad940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:35.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.719+0000 7fc1a3400640 1 --2- 172.21.15.90:0/3586672334 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1a40aeeb0 0x7fc1a40ad940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:35.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.719+0000 7fc1a3400640 1 -- 172.21.15.90:0/3586672334 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1a415bc50 msgr2=0x7fc1a414ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:35.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.719+0000 7fc1a3400640 1 --2- 172.21.15.90:0/3586672334 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1a415bc50 0x7fc1a414ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:35.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.719+0000 7fc1a3400640 1 -- 172.21.15.90:0/3586672334 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a41f6fd0 con 0x7fc1a406b7d0 2024-03-24T01:38:35.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.719+0000 7fc1a3400640 1 --2- 172.21.15.90:0/3586672334 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1a406b7d0 0x7fc1a41509f0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fc194036040 tx=0x7fc1940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:35.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.720+0000 7fc19be00640 1 -- 172.21.15.90:0/3586672334 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc194042070 con 0x7fc1a406b7d0 2024-03-24T01:38:35.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.720+0000 7fc19be00640 1 -- 172.21.15.90:0/3586672334 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc19403d890 con 0x7fc1a406b7d0 2024-03-24T01:38:35.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.720+0000 7fc1aa000640 1 -- 172.21.15.90:0/3586672334 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1a41f3fb0 con 0x7fc1a406b7d0 2024-03-24T01:38:35.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.720+0000 7fc19be00640 1 -- 172.21.15.90:0/3586672334 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc194038040 con 0x7fc1a406b7d0 2024-03-24T01:38:35.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.720+0000 7fc1aa000640 1 -- 172.21.15.90:0/3586672334 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1a41f47f0 con 0x7fc1a406b7d0 2024-03-24T01:38:35.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.721+0000 7fc1aa000640 1 -- 172.21.15.90:0/3586672334 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1a41f71a0 con 0x7fc1a406b7d0 2024-03-24T01:38:35.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.722+0000 7fc19be00640 1 -- 172.21.15.90:0/3586672334 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fc19403b070 con 0x7fc1a406b7d0 2024-03-24T01:38:35.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.723+0000 7fc19be00640 1 --2- 172.21.15.90:0/3586672334 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc18006bbd0 0x7fc18006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:35.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.723+0000 7fc19be00640 1 -- 172.21.15.90:0/3586672334 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7fc1940ac370 con 0x7fc1a406b7d0 2024-03-24T01:38:35.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.723+0000 7fc19aa00640 1 --2- 172.21.15.90:0/3586672334 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc18006bbd0 0x7fc18006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:35.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.724+0000 7fc19aa00640 1 --2- 172.21.15.90:0/3586672334 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc18006bbd0 0x7fc18006e090 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fc1a40cbae0 tx=0x7fc18c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:35.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.727+0000 7fc19be00640 1 -- 172.21.15.90:0/3586672334 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fc1a41f3fb0 con 0x7fc1a406b7d0 2024-03-24T01:38:35.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.860+0000 7f0b97c00640 1 -- 172.21.15.90:0/3710250540 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b900ad0d0 con 0x7f0b900cc150 2024-03-24T01:38:35.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.862+0000 7f0b8cc00640 1 -- 172.21.15.90:0/3710250540 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+17101 (secure 0 0 0) 0x7f0b7c058270 con 0x7f0b900cc150 2024-03-24T01:38:35.863 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:38:35.863 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":69,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:32:54.821325+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:32:42.026147+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-24T01:32:43.180712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.1e","mappings":[{"from":7,"to":2}]},{"pgid":"10.10","mappings":[{"from":4,"to":3}]},{"pgid":"10.1d","mappings":[{"from":4,"to":3}]},{"pgid":"10.1f","mappings":[{"from":7,"to":3}]},{"pgid":"10.37","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:38:35.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.864+0000 7f0b97c00640 1 -- 172.21.15.90:0/3710250540 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0b7006bc00 msgr2=0x7f0b7006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:35.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.864+0000 7f0b97c00640 1 --2- 172.21.15.90:0/3710250540 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0b7006bc00 0x7f0b7006e0c0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f0b84036000 tx=0x7f0b8403b040 comp rx=0 tx=0).stop 2024-03-24T01:38:35.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.864+0000 7f0b97c00640 1 -- 172.21.15.90:0/3710250540 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b900cc150 msgr2=0x7f0b9006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:35.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.864+0000 7f0b97c00640 1 --2- 172.21.15.90:0/3710250540 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b900cc150 0x7f0b9006dd50 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f0b7c00c410 tx=0x7f0b7c00c8d0 comp rx=0 tx=0).stop 2024-03-24T01:38:35.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.864+0000 7f0b97c00640 1 -- 172.21.15.90:0/3710250540 shutdown_connections 2024-03-24T01:38:35.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.864+0000 7f0b97c00640 1 --2- 172.21.15.90:0/3710250540 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f0b7006bc00 0x7f0b7006e0c0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:35.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.864+0000 7f0b97c00640 1 --2- 172.21.15.90:0/3710250540 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b900cc150 0x7f0b9006dd50 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:35.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.864+0000 7f0b97c00640 1 --2- 172.21.15.90:0/3710250540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b901603e0 0x7f0b901f74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:35.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.864+0000 7f0b97c00640 1 --2- 172.21.15.90:0/3710250540 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f0b9006b7d0 0x7f0b9013c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:35.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.864+0000 7f0b97c00640 1 -- 172.21.15.90:0/3710250540 >> 172.21.15.90:0/3710250540 conn(0x7f0b900b1770 msgr2=0x7f0b900b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:35.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.865+0000 7f0b97c00640 1 -- 172.21.15.90:0/3710250540 shutdown_connections 2024-03-24T01:38:35.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.865+0000 7f0b97c00640 1 -- 172.21.15.90:0/3710250540 wait complete. 2024-03-24T01:38:35.883 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T01:32:42.026147+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T01:32:43.180712+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T01:38:35.883 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:38:35.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.933+0000 7fc1aa000640 1 -- 172.21.15.90:0/3586672334 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc1a4157820 con 0x7fc1a406b7d0 2024-03-24T01:38:35.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.934+0000 7fc19be00640 1 -- 172.21.15.90:0/3586672334 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v69) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc194030070 con 0x7fc1a406b7d0 2024-03-24T01:38:35.934 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:38:35.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.936+0000 7fc1aa000640 1 -- 172.21.15.90:0/3586672334 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc18006bbd0 msgr2=0x7fc18006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:35.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.936+0000 7fc1aa000640 1 --2- 172.21.15.90:0/3586672334 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc18006bbd0 0x7fc18006e090 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fc1a40cbae0 tx=0x7fc18c009040 comp rx=0 tx=0).stop 2024-03-24T01:38:35.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.936+0000 7fc1aa000640 1 -- 172.21.15.90:0/3586672334 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1a406b7d0 msgr2=0x7fc1a41509f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:35.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.936+0000 7fc1aa000640 1 --2- 172.21.15.90:0/3586672334 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1a406b7d0 0x7fc1a41509f0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fc194036040 tx=0x7fc1940046c0 comp rx=0 tx=0).stop 2024-03-24T01:38:35.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.937+0000 7fc1aa000640 1 -- 172.21.15.90:0/3586672334 shutdown_connections 2024-03-24T01:38:35.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.937+0000 7fc1aa000640 1 --2- 172.21.15.90:0/3586672334 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fc18006bbd0 0x7fc18006e090 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:35.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.937+0000 7fc1aa000640 1 --2- 172.21.15.90:0/3586672334 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1a415bc50 0x7fc1a414ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:35.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.937+0000 7fc1aa000640 1 --2- 172.21.15.90:0/3586672334 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fc1a40aeeb0 0x7fc1a40ad940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:35.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.937+0000 7fc1aa000640 1 --2- 172.21.15.90:0/3586672334 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1a406b7d0 0x7fc1a41509f0 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:35.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.937+0000 7fc1aa000640 1 -- 172.21.15.90:0/3586672334 >> 172.21.15.90:0/3586672334 conn(0x7fc1a40b1770 msgr2=0x7fc1a4159ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:35.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.937+0000 7fc1aa000640 1 -- 172.21.15.90:0/3586672334 shutdown_connections 2024-03-24T01:38:35.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:35.937+0000 7fc1aa000640 1 -- 172.21.15.90:0/3586672334 wait complete. 2024-03-24T01:38:35.954 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T01:38:36.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.081+0000 7fd3afc00640 1 Processor -- start 2024-03-24T01:38:36.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.081+0000 7fd3afc00640 1 -- start start 2024-03-24T01:38:36.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.082+0000 7fd3afc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3a8072190 0x7fd3a80b4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:36.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.082+0000 7fd3afc00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd3a81631f0 0x7fd3a8072570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:36.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.082+0000 7fd3afc00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3a8072ab0 con 0x7fd3a8165d60 2024-03-24T01:38:36.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.082+0000 7fd3afc00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3a80b34b0 con 0x7fd3a81631f0 2024-03-24T01:38:36.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.082+0000 7fd3afc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a80b3790 con 0x7fd3a8072190 2024-03-24T01:38:36.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.082+0000 7fd3adc00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd3a8165d60 0x7fd3a81661f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:37088/0 (socket says 172.21.15.90:37088) 2024-03-24T01:38:36.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.082+0000 7fd3adc00640 1 -- 172.21.15.90:0/2542968637 learned_addr learned my addr 172.21.15.90:0/2542968637 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:38:36.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.083+0000 7fd39fe00640 1 --2- 172.21.15.90:0/2542968637 >> v2:172.21.15.120:3300/0 conn(0x7fd3a81631f0 0x7fd3a8072570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:36.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.083+0000 7fd3ad200640 1 --2- 172.21.15.90:0/2542968637 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3a8072190 0x7fd3a80b4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:36.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.083+0000 7fd3a7e00640 1 -- 172.21.15.90:0/2542968637 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 51004628 0 0) 0x7fd3a8072ab0 con 0x7fd3a8165d60 2024-03-24T01:38:36.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.083+0000 7fd3a7e00640 1 -- 172.21.15.90:0/2542968637 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd38c003520 con 0x7fd3a8165d60 2024-03-24T01:38:36.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.083+0000 7fd3a7e00640 1 -- 172.21.15.90:0/2542968637 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2427697337 0 0) 0x7fd38c003520 con 0x7fd3a8165d60 2024-03-24T01:38:36.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.083+0000 7fd3a7e00640 1 -- 172.21.15.90:0/2542968637 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3a8072190 msgr2=0x7fd3a80b4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:38:36.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.083+0000 7fd3a7e00640 1 --2- 172.21.15.90:0/2542968637 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3a8072190 0x7fd3a80b4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.083+0000 7fd3a7e00640 1 -- 172.21.15.90:0/2542968637 >> v2:172.21.15.120:3300/0 conn(0x7fd3a81631f0 msgr2=0x7fd3a8072570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:38:36.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.083+0000 7fd3a7e00640 1 --2- 172.21.15.90:0/2542968637 >> v2:172.21.15.120:3300/0 conn(0x7fd3a81631f0 0x7fd3a8072570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.083+0000 7fd3a7e00640 1 -- 172.21.15.90:0/2542968637 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a80b3a70 con 0x7fd3a8165d60 2024-03-24T01:38:36.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.083+0000 7fd3a7e00640 1 -- 172.21.15.90:0/2542968637 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7fd3a00019f0 con 0x7fd3a8165d60 2024-03-24T01:38:36.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.083+0000 7fd3a7e00640 1 -- 172.21.15.90:0/2542968637 >> v1:172.21.15.90:6789/0 conn(0x7fd3a8165d60 legacy=0x7fd3a81661f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:36.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.083+0000 7fd3a7e00640 1 --2- 172.21.15.90:0/2542968637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd38c0039b0 0x7fd38c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:36.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.083+0000 7fd3a7e00640 1 --2- 172.21.15.90:0/2542968637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd38c004800 0x7fd38c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:36.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.083+0000 7fd3a7e00640 1 --2- 172.21.15.90:0/2542968637 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3a8072190 0x7fd38c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:36.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.083+0000 7fd3a7e00640 1 -- 172.21.15.90:0/2542968637 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a80b34b0 con 0x7fd38c004800 2024-03-24T01:38:36.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.083+0000 7fd3a7e00640 1 -- 172.21.15.90:0/2542968637 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a80b3790 con 0x7fd38c0039b0 2024-03-24T01:38:36.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.083+0000 7fd3a7e00640 1 -- 172.21.15.90:0/2542968637 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd38c00ba10 con 0x7fd3a8072190 2024-03-24T01:38:36.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.084+0000 7fd39fe00640 1 --2- 172.21.15.90:0/2542968637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd38c004800 0x7fd38c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:36.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.084+0000 7fd3ad200640 1 --2- 172.21.15.90:0/2542968637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd38c0039b0 0x7fd38c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:36.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.084+0000 7fd39fe00640 1 -- 172.21.15.90:0/2542968637 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3a8072190 msgr2=0x7fd38c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:38:36.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.084+0000 7fd39fe00640 1 --2- 172.21.15.90:0/2542968637 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3a8072190 0x7fd38c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.084+0000 7fd39fe00640 1 -- 172.21.15.90:0/2542968637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd38c0039b0 msgr2=0x7fd38c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:38:36.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.084+0000 7fd39fe00640 1 --2- 172.21.15.90:0/2542968637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd38c0039b0 0x7fd38c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.084+0000 7fd39fe00640 1 -- 172.21.15.90:0/2542968637 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd38c003520 con 0x7fd38c004800 2024-03-24T01:38:36.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.085+0000 7fd39fe00640 1 --2- 172.21.15.90:0/2542968637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd38c004800 0x7fd38c004c80 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fd390009ac0 tx=0x7fd38c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e23c09e39e6e0075 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:36.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.085+0000 7fd3a7e00640 1 -- 172.21.15.90:0/2542968637 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd390039070 con 0x7fd38c004800 2024-03-24T01:38:36.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.085+0000 7fd3a7e00640 1 -- 172.21.15.90:0/2542968637 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd39002c710 con 0x7fd38c004800 2024-03-24T01:38:36.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.085+0000 7fd3a7e00640 1 -- 172.21.15.90:0/2542968637 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd390009690 con 0x7fd38c004800 2024-03-24T01:38:36.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.086+0000 7fd3afc00640 1 -- 172.21.15.90:0/2542968637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd38c004800 msgr2=0x7fd38c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:36.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.086+0000 7fd3afc00640 1 --2- 172.21.15.90:0/2542968637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd38c004800 0x7fd38c004c80 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fd390009ac0 tx=0x7fd38c005830 comp rx=0 tx=0).stop 2024-03-24T01:38:36.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.086+0000 7fd3adc00640 1 -- 172.21.15.90:0/2542968637 reap_dead start 2024-03-24T01:38:36.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.086+0000 7fd3afc00640 1 -- 172.21.15.90:0/2542968637 shutdown_connections 2024-03-24T01:38:36.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.086+0000 7fd3afc00640 1 -- 172.21.15.90:0/2542968637 >> 172.21.15.90:0/2542968637 conn(0x7fd3a80d00f0 msgr2=0x7fd3a80d04d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:36.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.086+0000 7fd3afc00640 1 -- 172.21.15.90:0/2542968637 shutdown_connections 2024-03-24T01:38:36.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.086+0000 7fd3afc00640 1 -- 172.21.15.90:0/2542968637 wait complete. 2024-03-24T01:38:36.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.088+0000 7fd3afc00640 1 Processor -- start 2024-03-24T01:38:36.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.088+0000 7fd3afc00640 1 -- start start 2024-03-24T01:38:36.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.089+0000 7fd3afc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd38c004800 0x7fd3a80cbc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:36.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.089+0000 7fd3afc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3a80cdc80 0x7fd3a81fb310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:36.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.089+0000 7fd3afc00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3a80ce100 0x7fd3a81fb850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:36.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.089+0000 7fd3afc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a80b3180 con 0x7fd38c004800 2024-03-24T01:38:36.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.089+0000 7fd3afc00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a80b4030 con 0x7fd3a80ce100 2024-03-24T01:38:36.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.089+0000 7fd3afc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a8163670 con 0x7fd3a80cdc80 2024-03-24T01:38:36.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.089+0000 7fd3ad200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd38c004800 0x7fd3a80cbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:36.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.089+0000 7fd3ad200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd38c004800 0x7fd3a80cbc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38248/0 (socket says 172.21.15.90:38248) 2024-03-24T01:38:36.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.089+0000 7fd3ad200640 1 -- 172.21.15.90:0/531302317 learned_addr learned my addr 172.21.15.90:0/531302317 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:38:36.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.090+0000 7fd39fe00640 1 --2- 172.21.15.90:0/531302317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3a80cdc80 0x7fd3a81fb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:36.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.090+0000 7fd3ad200640 1 -- 172.21.15.90:0/531302317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3a80cdc80 msgr2=0x7fd3a81fb310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:36.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.090+0000 7fd3ad200640 1 --2- 172.21.15.90:0/531302317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3a80cdc80 0x7fd3a81fb310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.090+0000 7fd3ad200640 1 -- 172.21.15.90:0/531302317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3a80ce100 msgr2=0x7fd3a81fb850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:36.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.090+0000 7fd3adc00640 1 --2- 172.21.15.90:0/531302317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3a80ce100 0x7fd3a81fb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:36.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.090+0000 7fd3ad200640 1 --2- 172.21.15.90:0/531302317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3a80ce100 0x7fd3a81fb850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.090+0000 7fd3ad200640 1 -- 172.21.15.90:0/531302317 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a80ce640 con 0x7fd38c004800 2024-03-24T01:38:36.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.090+0000 7fd39fe00640 1 --2- 172.21.15.90:0/531302317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3a80cdc80 0x7fd3a81fb310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:38:36.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.090+0000 7fd3ad200640 1 --2- 172.21.15.90:0/531302317 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd38c004800 0x7fd3a80cbc40 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fd39800f5b0 tx=0x7fd39800fa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:36.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.090+0000 7fd3a5600640 1 -- 172.21.15.90:0/531302317 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd398019020 con 0x7fd38c004800 2024-03-24T01:38:36.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.090+0000 7fd3a5600640 1 -- 172.21.15.90:0/531302317 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd398014870 con 0x7fd38c004800 2024-03-24T01:38:36.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.090+0000 7fd3afc00640 1 -- 172.21.15.90:0/531302317 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a80ce920 con 0x7fd38c004800 2024-03-24T01:38:36.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.090+0000 7fd3a5600640 1 -- 172.21.15.90:0/531302317 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd398014c60 con 0x7fd38c004800 2024-03-24T01:38:36.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.091+0000 7fd3afc00640 1 -- 172.21.15.90:0/531302317 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3a815c210 con 0x7fd38c004800 2024-03-24T01:38:36.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.092+0000 7fd3afc00640 1 -- 172.21.15.90:0/531302317 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3a804f400 con 0x7fd38c004800 2024-03-24T01:38:36.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.093+0000 7fd3a5600640 1 -- 172.21.15.90:0/531302317 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd39801c020 con 0x7fd38c004800 2024-03-24T01:38:36.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.093+0000 7fd3a5600640 1 --2- 172.21.15.90:0/531302317 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd38406bbd0 0x7fd38406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:36.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.093+0000 7fd3a5600640 1 -- 172.21.15.90:0/531302317 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7fd39808b3f0 con 0x7fd38c004800 2024-03-24T01:38:36.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.094+0000 7fd39fe00640 1 --2- 172.21.15.90:0/531302317 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd38406bbd0 0x7fd38406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:36.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.095+0000 7fd39fe00640 1 --2- 172.21.15.90:0/531302317 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd38406bbd0 0x7fd38406e090 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fd390036000 tx=0x7fd390030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:36.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.102+0000 7fd3a5600640 1 -- 172.21.15.90:0/531302317 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd398023020 con 0x7fd38c004800 2024-03-24T01:38:36.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.146+0000 7f754fa00640 1 Processor -- start 2024-03-24T01:38:36.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.147+0000 7f754fa00640 1 -- start start 2024-03-24T01:38:36.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.148+0000 7f754fa00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f75480aeeb0 0x7f75480b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:36.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.148+0000 7f754fa00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f754806b7d0 0x7f75480b2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:36.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.148+0000 7f754fa00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f754806dac0 con 0x7f754815bc50 2024-03-24T01:38:36.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.148+0000 7f754fa00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75480b4da0 con 0x7f75480aeeb0 2024-03-24T01:38:36.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.148+0000 7f754fa00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f754806dda0 con 0x7f754806b7d0 2024-03-24T01:38:36.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.148+0000 7f754d000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f75480aeeb0 0x7f75480b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:36.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.148+0000 7f754d000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f75480aeeb0 0x7f75480b44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38254/0 (socket says 172.21.15.90:38254) 2024-03-24T01:38:36.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.148+0000 7f754d000640 1 -- 172.21.15.90:0/1236758193 learned_addr learned my addr 172.21.15.90:0/1236758193 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:38:36.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.148+0000 7f754da00640 1 --2- 172.21.15.90:0/1236758193 >> v2:172.21.15.120:3300/0 conn(0x7f754806b7d0 0x7f75480b2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:36.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.149+0000 7f754d000640 1 -- 172.21.15.90:0/1236758193 >> v2:172.21.15.120:3300/0 conn(0x7f754806b7d0 msgr2=0x7f75480b2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:38:36.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.149+0000 7f754d000640 1 --2- 172.21.15.90:0/1236758193 >> v2:172.21.15.120:3300/0 conn(0x7f754806b7d0 0x7f75480b2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.149+0000 7f754d000640 1 -- 172.21.15.90:0/1236758193 >> v1:172.21.15.120:6789/0 conn(0x7f754815bc50 legacy=0x7f754806b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:36.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.149+0000 7f754d000640 1 -- 172.21.15.90:0/1236758193 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f754806e080 con 0x7f75480aeeb0 2024-03-24T01:38:36.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.150+0000 7f754d000640 1 --2- 172.21.15.90:0/1236758193 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75480aeeb0 0x7f75480b44c0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f75480cd4e0 tx=0x7f753c02da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c55d616fd5df5f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:36.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.150+0000 7f7547400640 1 -- 172.21.15.90:0/1236758193 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753c02f850 con 0x7f75480aeeb0 2024-03-24T01:38:36.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.150+0000 7f7547400640 1 -- 172.21.15.90:0/1236758193 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f753c02f9f0 con 0x7f75480aeeb0 2024-03-24T01:38:36.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.150+0000 7f7547400640 1 -- 172.21.15.90:0/1236758193 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753c02fdc0 con 0x7f75480aeeb0 2024-03-24T01:38:36.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.151+0000 7f754fa00640 1 -- 172.21.15.90:0/1236758193 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75480aeeb0 msgr2=0x7f75480b44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:36.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.151+0000 7f754fa00640 1 --2- 172.21.15.90:0/1236758193 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75480aeeb0 0x7f75480b44c0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f75480cd4e0 tx=0x7f753c02da90 comp rx=0 tx=0).stop 2024-03-24T01:38:36.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.151+0000 7f754fa00640 1 -- 172.21.15.90:0/1236758193 shutdown_connections 2024-03-24T01:38:36.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.151+0000 7f754fa00640 1 --2- 172.21.15.90:0/1236758193 >> v2:172.21.15.120:3300/0 conn(0x7f754806b7d0 0x7f75480b2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.151+0000 7f754fa00640 1 --2- 172.21.15.90:0/1236758193 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75480aeeb0 0x7f75480b44c0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.151+0000 7f754fa00640 1 -- 172.21.15.90:0/1236758193 >> 172.21.15.90:0/1236758193 conn(0x7f75480b1770 msgr2=0x7f75480b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:36.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.151+0000 7f754fa00640 1 -- 172.21.15.90:0/1236758193 shutdown_connections 2024-03-24T01:38:36.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.151+0000 7f754fa00640 1 -- 172.21.15.90:0/1236758193 wait complete. 2024-03-24T01:38:36.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.153+0000 7f754fa00640 1 Processor -- start 2024-03-24T01:38:36.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.153+0000 7f754fa00640 1 -- start start 2024-03-24T01:38:36.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.154+0000 7f754fa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f754806b7d0 0x7f7548150bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:36.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.154+0000 7f754fa00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75480aeeb0 0x7f7548155c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:36.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.154+0000 7f754fa00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f754815bc50 0x7f754815c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:36.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.154+0000 7f754fa00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75480b4a00 con 0x7f754806b7d0 2024-03-24T01:38:36.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.154+0000 7f754fa00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75480af290 con 0x7f75480aeeb0 2024-03-24T01:38:36.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.154+0000 7f754fa00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75480b4b80 con 0x7f754815bc50 2024-03-24T01:38:36.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.154+0000 7f754d000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f754806b7d0 0x7f7548150bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:36.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.154+0000 7f754d000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f754806b7d0 0x7f7548150bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38262/0 (socket says 172.21.15.90:38262) 2024-03-24T01:38:36.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.154+0000 7f754d000640 1 -- 172.21.15.90:0/1357845594 learned_addr learned my addr 172.21.15.90:0/1357845594 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:38:36.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.154+0000 7f7547e00640 1 --2- 172.21.15.90:0/1357845594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75480aeeb0 0x7f7548155c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:36.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.154+0000 7f754da00640 1 --2- 172.21.15.90:0/1357845594 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f754815bc50 0x7f754815c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:36.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.155+0000 7f754d000640 1 -- 172.21.15.90:0/1357845594 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f754815bc50 msgr2=0x7f754815c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:36.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.155+0000 7f754d000640 1 --2- 172.21.15.90:0/1357845594 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f754815bc50 0x7f754815c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.155+0000 7f754d000640 1 -- 172.21.15.90:0/1357845594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75480aeeb0 msgr2=0x7f7548155c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:36.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.155+0000 7f754d000640 1 --2- 172.21.15.90:0/1357845594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75480aeeb0 0x7f7548155c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.155+0000 7f754d000640 1 -- 172.21.15.90:0/1357845594 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75480ad420 con 0x7f754806b7d0 2024-03-24T01:38:36.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.155+0000 7f7547e00640 1 --2- 172.21.15.90:0/1357845594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75480aeeb0 0x7f7548155c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:38:36.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.155+0000 7f754d000640 1 --2- 172.21.15.90:0/1357845594 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f754806b7d0 0x7f7548150bd0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f753c009960 tx=0x7f753c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:36.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.155+0000 7f7544c00640 1 -- 172.21.15.90:0/1357845594 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753c047070 con 0x7f754806b7d0 2024-03-24T01:38:36.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.155+0000 7f7544c00640 1 -- 172.21.15.90:0/1357845594 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f753c0397f0 con 0x7f754806b7d0 2024-03-24T01:38:36.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.155+0000 7f754fa00640 1 -- 172.21.15.90:0/1357845594 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75480ad700 con 0x7f754806b7d0 2024-03-24T01:38:36.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.155+0000 7f7544c00640 1 -- 172.21.15.90:0/1357845594 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753c039be0 con 0x7f754806b7d0 2024-03-24T01:38:36.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.155+0000 7f754fa00640 1 -- 172.21.15.90:0/1357845594 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f754814f650 con 0x7f754806b7d0 2024-03-24T01:38:36.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.156+0000 7f754fa00640 1 -- 172.21.15.90:0/1357845594 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f754804f400 con 0x7f754806b7d0 2024-03-24T01:38:36.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.157+0000 7f7544c00640 1 -- 172.21.15.90:0/1357845594 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f753c042410 con 0x7f754806b7d0 2024-03-24T01:38:36.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.158+0000 7f7544c00640 1 --2- 172.21.15.90:0/1357845594 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f752806bbd0 0x7f752806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:36.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.158+0000 7f7544c00640 1 -- 172.21.15.90:0/1357845594 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7f753c0ac370 con 0x7f754806b7d0 2024-03-24T01:38:36.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.159+0000 7f7547e00640 1 --2- 172.21.15.90:0/1357845594 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f752806bbd0 0x7f752806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:36.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.160+0000 7f7547e00640 1 --2- 172.21.15.90:0/1357845594 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f752806bbd0 0x7f752806e090 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f7534000f60 tx=0x7f7534009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:36.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.163+0000 7f7544c00640 1 -- 172.21.15.90:0/1357845594 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f753c07acd0 con 0x7f754806b7d0 2024-03-24T01:38:36.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.294+0000 7fd3afc00640 1 -- 172.21.15.90:0/531302317 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd3a80c58e0 con 0x7fd38c004800 2024-03-24T01:38:36.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.294+0000 7fd3a5600640 1 -- 172.21.15.90:0/531302317 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v69) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd398059cb0 con 0x7fd38c004800 2024-03-24T01:38:36.295 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:38:36.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.297+0000 7fd3afc00640 1 -- 172.21.15.90:0/531302317 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd38406bbd0 msgr2=0x7fd38406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:36.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.297+0000 7fd3afc00640 1 --2- 172.21.15.90:0/531302317 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd38406bbd0 0x7fd38406e090 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fd390036000 tx=0x7fd390030000 comp rx=0 tx=0).stop 2024-03-24T01:38:36.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.297+0000 7fd3afc00640 1 -- 172.21.15.90:0/531302317 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd38c004800 msgr2=0x7fd3a80cbc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:36.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.297+0000 7fd3afc00640 1 --2- 172.21.15.90:0/531302317 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd38c004800 0x7fd3a80cbc40 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fd39800f5b0 tx=0x7fd39800fa70 comp rx=0 tx=0).stop 2024-03-24T01:38:36.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.297+0000 7fd3afc00640 1 -- 172.21.15.90:0/531302317 shutdown_connections 2024-03-24T01:38:36.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.297+0000 7fd3afc00640 1 --2- 172.21.15.90:0/531302317 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd38406bbd0 0x7fd38406e090 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.297+0000 7fd3afc00640 1 --2- 172.21.15.90:0/531302317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3a80ce100 0x7fd3a81fb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.297+0000 7fd3afc00640 1 --2- 172.21.15.90:0/531302317 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3a80cdc80 0x7fd3a81fb310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.297+0000 7fd3afc00640 1 --2- 172.21.15.90:0/531302317 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd38c004800 0x7fd3a80cbc40 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.297+0000 7fd3afc00640 1 -- 172.21.15.90:0/531302317 >> 172.21.15.90:0/531302317 conn(0x7fd3a80d00f0 msgr2=0x7fd3a81638e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:36.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.297+0000 7fd3afc00640 1 -- 172.21.15.90:0/531302317 shutdown_connections 2024-03-24T01:38:36.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.297+0000 7fd3afc00640 1 -- 172.21.15.90:0/531302317 wait complete. 2024-03-24T01:38:36.315 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T01:38:36.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.360+0000 7f754fa00640 1 -- 172.21.15.90:0/1357845594 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f75480b1d60 con 0x7f754806b7d0 2024-03-24T01:38:36.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.360+0000 7f7544c00640 1 -- 172.21.15.90:0/1357845594 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v69) v1 ==== 104+0+11 (secure 0 0 0) 0x7f753c07a5d0 con 0x7f754806b7d0 2024-03-24T01:38:36.361 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:38:36.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.363+0000 7f754fa00640 1 -- 172.21.15.90:0/1357845594 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f752806bbd0 msgr2=0x7f752806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:36.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.363+0000 7f754fa00640 1 --2- 172.21.15.90:0/1357845594 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f752806bbd0 0x7f752806e090 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f7534000f60 tx=0x7f7534009040 comp rx=0 tx=0).stop 2024-03-24T01:38:36.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.363+0000 7f754fa00640 1 -- 172.21.15.90:0/1357845594 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f754806b7d0 msgr2=0x7f7548150bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:36.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.363+0000 7f754fa00640 1 --2- 172.21.15.90:0/1357845594 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f754806b7d0 0x7f7548150bd0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f753c009960 tx=0x7f753c0046c0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.363+0000 7f754fa00640 1 -- 172.21.15.90:0/1357845594 shutdown_connections 2024-03-24T01:38:36.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.363+0000 7f754fa00640 1 --2- 172.21.15.90:0/1357845594 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f752806bbd0 0x7f752806e090 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.363+0000 7f754fa00640 1 --2- 172.21.15.90:0/1357845594 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f754815bc50 0x7f754815c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.363+0000 7f754fa00640 1 --2- 172.21.15.90:0/1357845594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75480aeeb0 0x7f7548155c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.363+0000 7f754fa00640 1 --2- 172.21.15.90:0/1357845594 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f754806b7d0 0x7f7548150bd0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.363+0000 7f754fa00640 1 -- 172.21.15.90:0/1357845594 >> 172.21.15.90:0/1357845594 conn(0x7f75480b1770 msgr2=0x7f754815afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:36.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.363+0000 7f754fa00640 1 -- 172.21.15.90:0/1357845594 shutdown_connections 2024-03-24T01:38:36.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.363+0000 7f754fa00640 1 -- 172.21.15.90:0/1357845594 wait complete. 2024-03-24T01:38:36.379 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T01:38:36.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.502+0000 7ff8f4c00640 1 Processor -- start 2024-03-24T01:38:36.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.503+0000 7ff8f4c00640 1 -- start start 2024-03-24T01:38:36.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.504+0000 7ff8f4c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff8f00b8600 0x7ff8f00b8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:36.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.505+0000 7ff8f4c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f00b8f50 con 0x7ff8f015dc90 2024-03-24T01:38:36.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.505+0000 7ff8f4c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f00b7290 con 0x7ff8f00b97c0 2024-03-24T01:38:36.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.505+0000 7ff8f4c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8f00b7560 con 0x7ff8f00b8600 2024-03-24T01:38:36.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.505+0000 7ff8efe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff8f00b8600 0x7ff8f00b8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:36.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.505+0000 7ff8efe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff8f00b8600 0x7ff8f00b8a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38276/0 (socket says 172.21.15.90:38276) 2024-03-24T01:38:36.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.505+0000 7ff8efe00640 1 -- 172.21.15.90:0/671877417 learned_addr learned my addr 172.21.15.90:0/671877417 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:38:36.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.505+0000 7ff8efe00640 1 -- 172.21.15.90:0/671877417 >> v1:172.21.15.120:6789/0 conn(0x7ff8f00b97c0 legacy=0x7ff8f00d1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:36.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.505+0000 7ff8efe00640 1 -- 172.21.15.90:0/671877417 >> v1:172.21.15.90:6789/0 conn(0x7ff8f015dc90 legacy=0x7ff8f00b6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:36.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.505+0000 7ff8efe00640 1 -- 172.21.15.90:0/671877417 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f00b7840 con 0x7ff8f00b8600 2024-03-24T01:38:36.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.506+0000 7ff8efe00640 1 --2- 172.21.15.90:0/671877417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8f00b8600 0x7ff8f00b8a10 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8009e50 tx=0x7ff8e802c090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f2269a8dc07ddc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:36.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.507+0000 7ff8ee000640 1 -- 172.21.15.90:0/671877417 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e802cc30 con 0x7ff8f00b8600 2024-03-24T01:38:36.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.507+0000 7ff8ee000640 1 -- 172.21.15.90:0/671877417 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8e802cdd0 con 0x7ff8f00b8600 2024-03-24T01:38:36.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.507+0000 7ff8ee000640 1 -- 172.21.15.90:0/671877417 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e8034610 con 0x7ff8f00b8600 2024-03-24T01:38:36.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.507+0000 7ff8f4c00640 1 -- 172.21.15.90:0/671877417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8f00b8600 msgr2=0x7ff8f00b8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:36.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.507+0000 7ff8f4c00640 1 --2- 172.21.15.90:0/671877417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8f00b8600 0x7ff8f00b8a10 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8009e50 tx=0x7ff8e802c090 comp rx=0 tx=0).stop 2024-03-24T01:38:36.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.507+0000 7ff8f4c00640 1 -- 172.21.15.90:0/671877417 shutdown_connections 2024-03-24T01:38:36.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.507+0000 7ff8f4c00640 1 --2- 172.21.15.90:0/671877417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8f00b8600 0x7ff8f00b8a10 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.507+0000 7ff8f4c00640 1 -- 172.21.15.90:0/671877417 >> 172.21.15.90:0/671877417 conn(0x7ff8f0072be0 msgr2=0x7ff8f0072ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:36.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.507+0000 7ff8f4c00640 1 -- 172.21.15.90:0/671877417 shutdown_connections 2024-03-24T01:38:36.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.508+0000 7ff8f4c00640 1 -- 172.21.15.90:0/671877417 wait complete. 2024-03-24T01:38:36.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.509+0000 7ff8f4c00640 1 Processor -- start 2024-03-24T01:38:36.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.509+0000 7ff8f4c00640 1 -- start start 2024-03-24T01:38:36.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.510+0000 7ff8f4c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8f00b8600 0x7ff8f00c7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:36.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.510+0000 7ff8f4c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8f00b97c0 0x7ff8f0142bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:36.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.510+0000 7ff8f4c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8f015dc90 0x7ff8f015e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:36.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.510+0000 7ff8f4c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f00b80e0 con 0x7ff8f00b8600 2024-03-24T01:38:36.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.510+0000 7ff8f4c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f00b7e00 con 0x7ff8f00b97c0 2024-03-24T01:38:36.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.510+0000 7ff8f4c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f00d2260 con 0x7ff8f015dc90 2024-03-24T01:38:36.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.510+0000 7ff8ef400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8f00b8600 0x7ff8f00c7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:36.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.510+0000 7ff8ef400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8f00b8600 0x7ff8f00c7bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38286/0 (socket says 172.21.15.90:38286) 2024-03-24T01:38:36.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.510+0000 7ff8ef400640 1 -- 172.21.15.90:0/2845378268 learned_addr learned my addr 172.21.15.90:0/2845378268 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:38:36.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.511+0000 7ff8eea00640 1 --2- 172.21.15.90:0/2845378268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8f00b97c0 0x7ff8f0142bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:36.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.511+0000 7ff8efe00640 1 --2- 172.21.15.90:0/2845378268 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8f015dc90 0x7ff8f015e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:36.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.511+0000 7ff8ef400640 1 -- 172.21.15.90:0/2845378268 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8f015dc90 msgr2=0x7ff8f015e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:36.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.511+0000 7ff8ef400640 1 --2- 172.21.15.90:0/2845378268 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8f015dc90 0x7ff8f015e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.511+0000 7ff8ef400640 1 -- 172.21.15.90:0/2845378268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8f00b97c0 msgr2=0x7ff8f0142bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:36.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.511+0000 7ff8ef400640 1 --2- 172.21.15.90:0/2845378268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8f00b97c0 0x7ff8f0142bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.511+0000 7ff8ef400640 1 -- 172.21.15.90:0/2845378268 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f00c8250 con 0x7ff8f00b8600 2024-03-24T01:38:36.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.511+0000 7ff8eea00640 1 --2- 172.21.15.90:0/2845378268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8f00b97c0 0x7ff8f0142bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:38:36.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.511+0000 7ff8ef400640 1 --2- 172.21.15.90:0/2845378268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8f00b8600 0x7ff8f00c7bc0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7ff8e000c450 tx=0x7ff8e000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:36.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.512+0000 7ff8e7400640 1 -- 172.21.15.90:0/2845378268 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e00052d0 con 0x7ff8f00b8600 2024-03-24T01:38:36.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.512+0000 7ff8e7400640 1 -- 172.21.15.90:0/2845378268 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8e0005470 con 0x7ff8f00b8600 2024-03-24T01:38:36.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.512+0000 7ff8f4c00640 1 -- 172.21.15.90:0/2845378268 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f0154f90 con 0x7ff8f00b8600 2024-03-24T01:38:36.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.512+0000 7ff8e7400640 1 -- 172.21.15.90:0/2845378268 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e0012870 con 0x7ff8f00b8600 2024-03-24T01:38:36.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.512+0000 7ff8f4c00640 1 -- 172.21.15.90:0/2845378268 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f0155550 con 0x7ff8f00b8600 2024-03-24T01:38:36.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.513+0000 7ff8f4c00640 1 -- 172.21.15.90:0/2845378268 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f004f400 con 0x7ff8f00b8600 2024-03-24T01:38:36.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.514+0000 7ff8e7400640 1 -- 172.21.15.90:0/2845378268 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff8e0012a10 con 0x7ff8f00b8600 2024-03-24T01:38:36.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.515+0000 7ff8e7400640 1 --2- 172.21.15.90:0/2845378268 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff8d006bbd0 0x7ff8d006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:36.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.515+0000 7ff8e7400640 1 -- 172.21.15.90:0/2845378268 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7ff8e008a960 con 0x7ff8f00b8600 2024-03-24T01:38:36.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.515+0000 7ff8eea00640 1 --2- 172.21.15.90:0/2845378268 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff8d006bbd0 0x7ff8d006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:36.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.516+0000 7ff8eea00640 1 --2- 172.21.15.90:0/2845378268 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff8d006bbd0 0x7ff8d006e090 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ff8f00b44c0 tx=0x7ff8dc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:36.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.524+0000 7ff8e7400640 1 -- 172.21.15.90:0/2845378268 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff8e0059190 con 0x7ff8f00b8600 2024-03-24T01:38:36.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.614+0000 7f017dc00640 1 Processor -- start 2024-03-24T01:38:36.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.615+0000 7f017dc00640 1 -- start start 2024-03-24T01:38:36.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.616+0000 7f017dc00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f017815bc50 0x7f017806b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:36.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.616+0000 7f017dc00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f017806dac0 con 0x7f017806b600 2024-03-24T01:38:36.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.616+0000 7f017dc00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f017806dda0 con 0x7f01780aeeb0 2024-03-24T01:38:36.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.616+0000 7f017dc00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f017806ba70 con 0x7f017815bc50 2024-03-24T01:38:36.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.616+0000 7f016fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f017815bc50 0x7f017806b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:36.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.616+0000 7f0177400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f017806b600 0x7f01780b2890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:37104/0 (socket says 172.21.15.90:37104) 2024-03-24T01:38:36.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.616+0000 7f0177400640 1 -- 172.21.15.90:0/1602201803 learned_addr learned my addr 172.21.15.90:0/1602201803 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:38:36.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.617+0000 7f016fe00640 1 -- 172.21.15.90:0/1602201803 >> v1:172.21.15.120:6789/0 conn(0x7f01780aeeb0 legacy=0x7f01780b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:36.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.617+0000 7f016fe00640 1 -- 172.21.15.90:0/1602201803 >> v1:172.21.15.90:6789/0 conn(0x7f017806b600 legacy=0x7f01780b2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:36.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.617+0000 7f016fe00640 1 -- 172.21.15.90:0/1602201803 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f017806e080 con 0x7f017815bc50 2024-03-24T01:38:36.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.618+0000 7f016fe00640 1 --2- 172.21.15.90:0/1602201803 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f017815bc50 0x7f017806b0c0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f01600097a0 tx=0x7f016002da50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53488234ad97dcb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:36.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.618+0000 7f0176000640 1 -- 172.21.15.90:0/1602201803 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0160031870 con 0x7f017815bc50 2024-03-24T01:38:36.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.618+0000 7f0176000640 1 -- 172.21.15.90:0/1602201803 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0160031a10 con 0x7f017815bc50 2024-03-24T01:38:36.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.618+0000 7f0176000640 1 -- 172.21.15.90:0/1602201803 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0160031de0 con 0x7f017815bc50 2024-03-24T01:38:36.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.619+0000 7f017dc00640 1 -- 172.21.15.90:0/1602201803 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f017815bc50 msgr2=0x7f017806b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:36.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.619+0000 7f017dc00640 1 --2- 172.21.15.90:0/1602201803 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f017815bc50 0x7f017806b0c0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f01600097a0 tx=0x7f016002da50 comp rx=0 tx=0).stop 2024-03-24T01:38:36.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.619+0000 7f017dc00640 1 -- 172.21.15.90:0/1602201803 shutdown_connections 2024-03-24T01:38:36.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.619+0000 7f017dc00640 1 --2- 172.21.15.90:0/1602201803 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f017815bc50 0x7f017806b0c0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.619+0000 7f017dc00640 1 -- 172.21.15.90:0/1602201803 >> 172.21.15.90:0/1602201803 conn(0x7f01780b1770 msgr2=0x7f01780b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:36.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.619+0000 7f017dc00640 1 -- 172.21.15.90:0/1602201803 shutdown_connections 2024-03-24T01:38:36.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.619+0000 7f017dc00640 1 -- 172.21.15.90:0/1602201803 wait complete. 2024-03-24T01:38:36.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.621+0000 7f017dc00640 1 Processor -- start 2024-03-24T01:38:36.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.621+0000 7f017dc00640 1 -- start start 2024-03-24T01:38:36.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.622+0000 7f017dc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f017806b600 0x7f01781f3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:36.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.622+0000 7f017dc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01780aeeb0 0x7f01781e5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:36.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.622+0000 7f017dc00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f017815bc50 0x7f01781e6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:36.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.622+0000 7f017dc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01780af290 con 0x7f01780aeeb0 2024-03-24T01:38:36.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.622+0000 7f017dc00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01780b4da0 con 0x7f017815bc50 2024-03-24T01:38:36.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.622+0000 7f017dc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f017806e880 con 0x7f017806b600 2024-03-24T01:38:36.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.622+0000 7f016fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01780aeeb0 0x7f01781e5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:36.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.622+0000 7f016fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01780aeeb0 0x7f01781e5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38296/0 (socket says 172.21.15.90:38296) 2024-03-24T01:38:36.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.622+0000 7f016fe00640 1 -- 172.21.15.90:0/2669748075 learned_addr learned my addr 172.21.15.90:0/2669748075 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:38:36.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.622+0000 7f0176a00640 1 --2- 172.21.15.90:0/2669748075 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f017806b600 0x7f01781f3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:36.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.623+0000 7f0177400640 1 --2- 172.21.15.90:0/2669748075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f017815bc50 0x7f01781e6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:36.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.623+0000 7f016fe00640 1 -- 172.21.15.90:0/2669748075 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f017806b600 msgr2=0x7f01781f3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:36.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.623+0000 7f016fe00640 1 --2- 172.21.15.90:0/2669748075 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f017806b600 0x7f01781f3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.623+0000 7f016fe00640 1 -- 172.21.15.90:0/2669748075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f017815bc50 msgr2=0x7f01781e6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:36.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.623+0000 7f016fe00640 1 --2- 172.21.15.90:0/2669748075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f017815bc50 0x7f01781e6090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.623+0000 7f016fe00640 1 -- 172.21.15.90:0/2669748075 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01781e33b0 con 0x7f01780aeeb0 2024-03-24T01:38:36.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.623+0000 7f016fe00640 1 --2- 172.21.15.90:0/2669748075 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01780aeeb0 0x7f01781e5b50 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f016002e060 tx=0x7f01600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:36.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.623+0000 7f016ea00640 1 -- 172.21.15.90:0/2669748075 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016003ad20 con 0x7f01780aeeb0 2024-03-24T01:38:36.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.623+0000 7f016ea00640 1 -- 172.21.15.90:0/2669748075 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f016002f400 con 0x7f01780aeeb0 2024-03-24T01:38:36.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.623+0000 7f017dc00640 1 -- 172.21.15.90:0/2669748075 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01781e3690 con 0x7f01780aeeb0 2024-03-24T01:38:36.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.623+0000 7f016ea00640 1 -- 172.21.15.90:0/2669748075 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016002f7f0 con 0x7f01780aeeb0 2024-03-24T01:38:36.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.623+0000 7f017dc00640 1 -- 172.21.15.90:0/2669748075 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01780b00c0 con 0x7f01780aeeb0 2024-03-24T01:38:36.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.625+0000 7f017dc00640 1 -- 172.21.15.90:0/2669748075 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f017806b0c0 con 0x7f01780aeeb0 2024-03-24T01:38:36.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.625+0000 7f016ea00640 1 -- 172.21.15.90:0/2669748075 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f016002f990 con 0x7f01780aeeb0 2024-03-24T01:38:36.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.626+0000 7f016ea00640 1 --2- 172.21.15.90:0/2669748075 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f015406bbd0 0x7f015406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:36.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.626+0000 7f016ea00640 1 -- 172.21.15.90:0/2669748075 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7f01600ad1d0 con 0x7f01780aeeb0 2024-03-24T01:38:36.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.627+0000 7f0176a00640 1 --2- 172.21.15.90:0/2669748075 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f015406bbd0 0x7f015406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:36.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.628+0000 7f0176a00640 1 --2- 172.21.15.90:0/2669748075 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f015406bbd0 0x7f015406e090 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f0168000f60 tx=0x7f0168009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:36.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.631+0000 7f016ea00640 1 -- 172.21.15.90:0/2669748075 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f017806b0c0 con 0x7f01780aeeb0 2024-03-24T01:38:36.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.723+0000 7ff8f4c00640 1 -- 172.21.15.90:0/2845378268 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff8f00b6d50 con 0x7ff8f00b8600 2024-03-24T01:38:36.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.724+0000 7ff8e7400640 1 -- 172.21.15.90:0/2845378268 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v69) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff8e0058a90 con 0x7ff8f00b8600 2024-03-24T01:38:36.724 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:38:36.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.726+0000 7ff8f4c00640 1 -- 172.21.15.90:0/2845378268 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff8d006bbd0 msgr2=0x7ff8d006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:36.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.726+0000 7ff8f4c00640 1 --2- 172.21.15.90:0/2845378268 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff8d006bbd0 0x7ff8d006e090 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ff8f00b44c0 tx=0x7ff8dc009040 comp rx=0 tx=0).stop 2024-03-24T01:38:36.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.726+0000 7ff8f4c00640 1 -- 172.21.15.90:0/2845378268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8f00b8600 msgr2=0x7ff8f00c7bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:36.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.726+0000 7ff8f4c00640 1 --2- 172.21.15.90:0/2845378268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8f00b8600 0x7ff8f00c7bc0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7ff8e000c450 tx=0x7ff8e000c910 comp rx=0 tx=0).stop 2024-03-24T01:38:36.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.727+0000 7ff8f4c00640 1 -- 172.21.15.90:0/2845378268 shutdown_connections 2024-03-24T01:38:36.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.727+0000 7ff8f4c00640 1 --2- 172.21.15.90:0/2845378268 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff8d006bbd0 0x7ff8d006e090 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.727+0000 7ff8f4c00640 1 --2- 172.21.15.90:0/2845378268 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff8f015dc90 0x7ff8f015e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.727+0000 7ff8f4c00640 1 --2- 172.21.15.90:0/2845378268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8f00b97c0 0x7ff8f0142bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.727+0000 7ff8f4c00640 1 --2- 172.21.15.90:0/2845378268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8f00b8600 0x7ff8f00c7bc0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.727+0000 7ff8f4c00640 1 -- 172.21.15.90:0/2845378268 >> 172.21.15.90:0/2845378268 conn(0x7ff8f0072be0 msgr2=0x7ff8f00b49d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:36.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.727+0000 7ff8f4c00640 1 -- 172.21.15.90:0/2845378268 shutdown_connections 2024-03-24T01:38:36.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.727+0000 7ff8f4c00640 1 -- 172.21.15.90:0/2845378268 wait complete. 2024-03-24T01:38:36.745 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T01:38:36.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.831+0000 7f017dc00640 1 -- 172.21.15.90:0/2669748075 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f017804f400 con 0x7f01780aeeb0 2024-03-24T01:38:36.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.831+0000 7f016ea00640 1 -- 172.21.15.90:0/2669748075 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v69) v1 ==== 100+0+11 (secure 0 0 0) 0x7f016003f070 con 0x7f01780aeeb0 2024-03-24T01:38:36.832 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-24T01:38:36.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.833+0000 7f017dc00640 1 -- 172.21.15.90:0/2669748075 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f015406bbd0 msgr2=0x7f015406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:36.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.833+0000 7f017dc00640 1 --2- 172.21.15.90:0/2669748075 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f015406bbd0 0x7f015406e090 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f0168000f60 tx=0x7f0168009040 comp rx=0 tx=0).stop 2024-03-24T01:38:36.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.833+0000 7f017dc00640 1 -- 172.21.15.90:0/2669748075 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01780aeeb0 msgr2=0x7f01781e5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:36.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.833+0000 7f017dc00640 1 --2- 172.21.15.90:0/2669748075 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01780aeeb0 0x7f01781e5b50 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f016002e060 tx=0x7f01600046c0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.833+0000 7f017dc00640 1 -- 172.21.15.90:0/2669748075 shutdown_connections 2024-03-24T01:38:36.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.833+0000 7f017dc00640 1 --2- 172.21.15.90:0/2669748075 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f015406bbd0 0x7f015406e090 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.833+0000 7f017dc00640 1 --2- 172.21.15.90:0/2669748075 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01780aeeb0 0x7f01781e5b50 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.833+0000 7f017dc00640 1 --2- 172.21.15.90:0/2669748075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f017815bc50 0x7f01781e6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.834+0000 7f017dc00640 1 --2- 172.21.15.90:0/2669748075 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f017806b600 0x7f01781f3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.834+0000 7f017dc00640 1 -- 172.21.15.90:0/2669748075 >> 172.21.15.90:0/2669748075 conn(0x7f01780b1770 msgr2=0x7f01780ade40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:36.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.834+0000 7f017dc00640 1 -- 172.21.15.90:0/2669748075 shutdown_connections 2024-03-24T01:38:36.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.834+0000 7f017dc00640 1 -- 172.21.15.90:0/2669748075 wait complete. 2024-03-24T01:38:36.858 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T01:38:36.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.944+0000 7fd326a00640 1 Processor -- start 2024-03-24T01:38:36.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.944+0000 7fd326a00640 1 -- start start 2024-03-24T01:38:36.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.946+0000 7fd326a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd32015dec0 0x7fd3200d28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:36.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.946+0000 7fd326a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd3200d2e20 0x7fd3200b5570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:36.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.946+0000 7fd326a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3200cf2f0 con 0x7fd3200b5ab0 2024-03-24T01:38:36.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.946+0000 7fd326a00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3200b5f00 con 0x7fd3200d2e20 2024-03-24T01:38:36.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.946+0000 7fd326a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd32015e2a0 con 0x7fd32015dec0 2024-03-24T01:38:36.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.946+0000 7fd325600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd32015dec0 0x7fd3200d28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:36.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.946+0000 7fd324c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fd3200d2e20 0x7fd3200b5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:36.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.947+0000 7fd326000640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7fd3200b5ab0 0x7fd3200b6f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.90:56102/0 (socket says 172.21.15.90:56102) 2024-03-24T01:38:36.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.947+0000 7fd326000640 1 -- 172.21.15.90:0/3299639951 learned_addr learned my addr 172.21.15.90:0/3299639951 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:38:36.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.947+0000 7fd325600640 1 --2- 172.21.15.90:0/3299639951 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd32015dec0 0x7fd3200d28e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.90:56050/0 (socket says 172.21.15.90:56050) 2024-03-24T01:38:36.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.947+0000 7fd325600640 1 -- 172.21.15.90:0/3299639951 >> v2:172.21.15.120:3300/0 conn(0x7fd3200d2e20 msgr2=0x7fd3200b5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:38:36.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.947+0000 7fd325600640 1 --2- 172.21.15.90:0/3299639951 >> v2:172.21.15.120:3300/0 conn(0x7fd3200d2e20 0x7fd3200b5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.947+0000 7fd325600640 1 -- 172.21.15.90:0/3299639951 >> v1:172.21.15.120:6789/0 conn(0x7fd3200b5ab0 legacy=0x7fd3200b6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:36.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.947+0000 7fd325600640 1 -- 172.21.15.90:0/3299639951 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3200d1e00 con 0x7fd32015dec0 2024-03-24T01:38:36.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.947+0000 7fd324c00640 1 --2- 172.21.15.90:0/3299639951 >> v2:172.21.15.120:3300/0 conn(0x7fd3200d2e20 0x7fd3200b5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T01:38:36.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.948+0000 7fd325600640 1 --2- 172.21.15.90:0/3299639951 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd32015dec0 0x7fd3200d28e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd3140097a0 tx=0x7fd31402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9d066b9bad1597f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:36.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.949+0000 7fd31fe00640 1 -- 172.21.15.90:0/3299639951 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd314035020 con 0x7fd32015dec0 2024-03-24T01:38:36.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.949+0000 7fd31fe00640 1 -- 172.21.15.90:0/3299639951 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd314031810 con 0x7fd32015dec0 2024-03-24T01:38:36.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.949+0000 7fd31fe00640 1 -- 172.21.15.90:0/3299639951 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd314031be0 con 0x7fd32015dec0 2024-03-24T01:38:36.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.949+0000 7fd326a00640 1 -- 172.21.15.90:0/3299639951 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd32015dec0 msgr2=0x7fd3200d28e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:36.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.949+0000 7fd326a00640 1 --2- 172.21.15.90:0/3299639951 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd32015dec0 0x7fd3200d28e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd3140097a0 tx=0x7fd31402b8b0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.949+0000 7fd326a00640 1 -- 172.21.15.90:0/3299639951 shutdown_connections 2024-03-24T01:38:36.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.949+0000 7fd326a00640 1 --2- 172.21.15.90:0/3299639951 >> v2:172.21.15.120:3300/0 conn(0x7fd3200d2e20 0x7fd3200b5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.949+0000 7fd326a00640 1 --2- 172.21.15.90:0/3299639951 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd32015dec0 0x7fd3200d28e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.949+0000 7fd326a00640 1 -- 172.21.15.90:0/3299639951 >> 172.21.15.90:0/3299639951 conn(0x7fd3200d0750 msgr2=0x7fd3200d0b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:36.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.950+0000 7fd326a00640 1 -- 172.21.15.90:0/3299639951 shutdown_connections 2024-03-24T01:38:36.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.950+0000 7fd326a00640 1 -- 172.21.15.90:0/3299639951 wait complete. 2024-03-24T01:38:36.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.951+0000 7fd326a00640 1 Processor -- start 2024-03-24T01:38:36.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.951+0000 7fd326a00640 1 -- start start 2024-03-24T01:38:36.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.952+0000 7fd326a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3200b5ab0 0x7fd3201f4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:36.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.952+0000 7fd326a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3200d2e20 0x7fd3200c7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:36.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.952+0000 7fd326a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd32015dec0 0x7fd3201f3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:36.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.952+0000 7fd326a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3200b7790 con 0x7fd32015dec0 2024-03-24T01:38:36.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.952+0000 7fd326a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3200b7610 con 0x7fd3200b5ab0 2024-03-24T01:38:36.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.952+0000 7fd326a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3200b7910 con 0x7fd3200d2e20 2024-03-24T01:38:36.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.953+0000 7fd326000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd32015dec0 0x7fd3201f3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:36.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.953+0000 7fd324c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3200d2e20 0x7fd3200c7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:36.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.953+0000 7fd326000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd32015dec0 0x7fd3201f3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38312/0 (socket says 172.21.15.90:38312) 2024-03-24T01:38:36.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.953+0000 7fd326000640 1 -- 172.21.15.90:0/4276284430 learned_addr learned my addr 172.21.15.90:0/4276284430 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:38:36.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.953+0000 7fd325600640 1 --2- 172.21.15.90:0/4276284430 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3200b5ab0 0x7fd3201f4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:36.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.953+0000 7fd326000640 1 -- 172.21.15.90:0/4276284430 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3200d2e20 msgr2=0x7fd3200c7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:36.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.953+0000 7fd326000640 1 --2- 172.21.15.90:0/4276284430 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3200d2e20 0x7fd3200c7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.953+0000 7fd326000640 1 -- 172.21.15.90:0/4276284430 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3200b5ab0 msgr2=0x7fd3201f4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:36.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.953+0000 7fd326000640 1 --2- 172.21.15.90:0/4276284430 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3200b5ab0 0x7fd3201f4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:36.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.953+0000 7fd326000640 1 -- 172.21.15.90:0/4276284430 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3200c7d70 con 0x7fd32015dec0 2024-03-24T01:38:36.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.953+0000 7fd325600640 1 --2- 172.21.15.90:0/4276284430 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3200b5ab0 0x7fd3201f4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:38:36.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.953+0000 7fd326000640 1 --2- 172.21.15.90:0/4276284430 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd32015dec0 0x7fd3201f3b70 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fd31800cbf0 tx=0x7fd31800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:36.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.954+0000 7fd31d600640 1 -- 172.21.15.90:0/4276284430 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd318017070 con 0x7fd32015dec0 2024-03-24T01:38:36.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.954+0000 7fd31d600640 1 -- 172.21.15.90:0/4276284430 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd31800ad40 con 0x7fd32015dec0 2024-03-24T01:38:36.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.954+0000 7fd31d600640 1 -- 172.21.15.90:0/4276284430 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd318012600 con 0x7fd32015dec0 2024-03-24T01:38:36.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.954+0000 7fd326a00640 1 -- 172.21.15.90:0/4276284430 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3200c8050 con 0x7fd32015dec0 2024-03-24T01:38:36.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.954+0000 7fd326a00640 1 -- 172.21.15.90:0/4276284430 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3200c8610 con 0x7fd32015dec0 2024-03-24T01:38:36.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.955+0000 7fd326a00640 1 -- 172.21.15.90:0/4276284430 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd320067790 con 0x7fd32015dec0 2024-03-24T01:38:36.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.956+0000 7fd31d600640 1 -- 172.21.15.90:0/4276284430 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fd31801a070 con 0x7fd32015dec0 2024-03-24T01:38:36.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.957+0000 7fd31d600640 1 --2- 172.21.15.90:0/4276284430 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd30006bbd0 0x7fd30006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:36.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.957+0000 7fd31d600640 1 -- 172.21.15.90:0/4276284430 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7fd31808a950 con 0x7fd32015dec0 2024-03-24T01:38:36.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.958+0000 7fd325600640 1 --2- 172.21.15.90:0/4276284430 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd30006bbd0 0x7fd30006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:36.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.959+0000 7fd325600640 1 --2- 172.21.15.90:0/4276284430 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd30006bbd0 0x7fd30006e090 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd3140097a0 tx=0x7fd31403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:36.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:36.962+0000 7fd31d600640 1 -- 172.21.15.90:0/4276284430 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fd320067790 con 0x7fd32015dec0 2024-03-24T01:38:37.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.038+0000 7fda61800640 1 Processor -- start 2024-03-24T01:38:37.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.039+0000 7fda61800640 1 -- start start 2024-03-24T01:38:37.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.040+0000 7fda61800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fda5c0b94e0 0x7fda5c0ce890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:37.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.040+0000 7fda61800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fda5c0cedd0 0x7fda5c0d23e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:37.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.040+0000 7fda61800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda5c1620a0 con 0x7fda5c15df40 2024-03-24T01:38:37.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.040+0000 7fda61800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda5c0d1c70 con 0x7fda5c0b94e0 2024-03-24T01:38:37.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.040+0000 7fda61800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda5c162380 con 0x7fda5c0cedd0 2024-03-24T01:38:37.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.040+0000 7fda5a000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fda5c0b94e0 0x7fda5c0ce890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:37.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.040+0000 7fda5a000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fda5c0b94e0 0x7fda5c0ce890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38318/0 (socket says 172.21.15.90:38318) 2024-03-24T01:38:37.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.040+0000 7fda5a000640 1 -- 172.21.15.90:0/2117433536 learned_addr learned my addr 172.21.15.90:0/2117433536 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:38:37.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.040+0000 7fda5b400640 1 --2- 172.21.15.90:0/2117433536 >> v2:172.21.15.120:3300/0 conn(0x7fda5c0cedd0 0x7fda5c0d23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:37.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.040+0000 7fda5a000640 1 -- 172.21.15.90:0/2117433536 >> v2:172.21.15.120:3300/0 conn(0x7fda5c0cedd0 msgr2=0x7fda5c0d23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:38:37.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.040+0000 7fda5a000640 1 --2- 172.21.15.90:0/2117433536 >> v2:172.21.15.120:3300/0 conn(0x7fda5c0cedd0 0x7fda5c0d23e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:37.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.040+0000 7fda5a000640 1 -- 172.21.15.90:0/2117433536 >> v1:172.21.15.120:6789/0 conn(0x7fda5c15df40 legacy=0x7fda5c161890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:38:37.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.041+0000 7fda5a000640 1 -- 172.21.15.90:0/2117433536 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda5c162660 con 0x7fda5c0b94e0 2024-03-24T01:38:37.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.041+0000 7fda5a000640 1 --2- 172.21.15.90:0/2117433536 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda5c0b94e0 0x7fda5c0ce890 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fda5c0d05f0 tx=0x7fda4802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=569eea01dbb00a07 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:37.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.042+0000 7fda59600640 1 -- 172.21.15.90:0/2117433536 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda48034070 con 0x7fda5c0b94e0 2024-03-24T01:38:37.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.042+0000 7fda59600640 1 -- 172.21.15.90:0/2117433536 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda48030860 con 0x7fda5c0b94e0 2024-03-24T01:38:37.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.042+0000 7fda59600640 1 -- 172.21.15.90:0/2117433536 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda48030c30 con 0x7fda5c0b94e0 2024-03-24T01:38:37.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.042+0000 7fda61800640 1 -- 172.21.15.90:0/2117433536 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda5c0b94e0 msgr2=0x7fda5c0ce890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:37.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.042+0000 7fda61800640 1 --2- 172.21.15.90:0/2117433536 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda5c0b94e0 0x7fda5c0ce890 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fda5c0d05f0 tx=0x7fda4802b9a0 comp rx=0 tx=0).stop 2024-03-24T01:38:37.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.042+0000 7fda61800640 1 -- 172.21.15.90:0/2117433536 shutdown_connections 2024-03-24T01:38:37.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.042+0000 7fda61800640 1 --2- 172.21.15.90:0/2117433536 >> v2:172.21.15.120:3300/0 conn(0x7fda5c0cedd0 0x7fda5c0d23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:37.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.042+0000 7fda61800640 1 --2- 172.21.15.90:0/2117433536 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda5c0b94e0 0x7fda5c0ce890 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:37.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.042+0000 7fda61800640 1 -- 172.21.15.90:0/2117433536 >> 172.21.15.90:0/2117433536 conn(0x7fda5c06c9f0 msgr2=0x7fda5c06ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:37.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.043+0000 7fda61800640 1 -- 172.21.15.90:0/2117433536 shutdown_connections 2024-03-24T01:38:37.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.043+0000 7fda61800640 1 -- 172.21.15.90:0/2117433536 wait complete. 2024-03-24T01:38:37.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.044+0000 7fda61800640 1 Processor -- start 2024-03-24T01:38:37.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.044+0000 7fda61800640 1 -- start start 2024-03-24T01:38:37.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.045+0000 7fda61800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda5c0b94e0 0x7fda5c0b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:37.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.045+0000 7fda61800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda5c0cedd0 0x7fda5c157220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:37.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.045+0000 7fda61800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda5c15df40 0x7fda5c15f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:37.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.045+0000 7fda61800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5c0d2aa0 con 0x7fda5c0cedd0 2024-03-24T01:38:37.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.045+0000 7fda61800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5c0d2920 con 0x7fda5c0b94e0 2024-03-24T01:38:37.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.045+0000 7fda61800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda5c0d2c20 con 0x7fda5c15df40 2024-03-24T01:38:37.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.045+0000 7fda5a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda5c0cedd0 0x7fda5c157220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:37.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.045+0000 7fda5a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda5c0cedd0 0x7fda5c157220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38320/0 (socket says 172.21.15.90:38320) 2024-03-24T01:38:37.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.045+0000 7fda5a000640 1 -- 172.21.15.90:0/1422099236 learned_addr learned my addr 172.21.15.90:0/1422099236 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:38:37.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.045+0000 7fda5b400640 1 --2- 172.21.15.90:0/1422099236 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda5c15df40 0x7fda5c15f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:37.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.046+0000 7fda5aa00640 1 --2- 172.21.15.90:0/1422099236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda5c0b94e0 0x7fda5c0b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:37.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.046+0000 7fda5a000640 1 -- 172.21.15.90:0/1422099236 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda5c15df40 msgr2=0x7fda5c15f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:37.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.046+0000 7fda5a000640 1 --2- 172.21.15.90:0/1422099236 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda5c15df40 0x7fda5c15f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:37.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.046+0000 7fda5a000640 1 -- 172.21.15.90:0/1422099236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda5c0b94e0 msgr2=0x7fda5c0b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:37.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.046+0000 7fda5a000640 1 --2- 172.21.15.90:0/1422099236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda5c0b94e0 0x7fda5c0b46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:37.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.046+0000 7fda5a000640 1 -- 172.21.15.90:0/1422099236 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda5c0cb860 con 0x7fda5c0cedd0 2024-03-24T01:38:37.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.046+0000 7fda5b400640 1 --2- 172.21.15.90:0/1422099236 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda5c15df40 0x7fda5c15f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:38:37.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.046+0000 7fda5a000640 1 --2- 172.21.15.90:0/1422099236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda5c0cedd0 0x7fda5c157220 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fda4802b9e0 tx=0x7fda48004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:37.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.046+0000 7fda52a00640 1 -- 172.21.15.90:0/1422099236 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda48048070 con 0x7fda5c0cedd0 2024-03-24T01:38:37.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.046+0000 7fda52a00640 1 -- 172.21.15.90:0/1422099236 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda48030660 con 0x7fda5c0cedd0 2024-03-24T01:38:37.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.046+0000 7fda52a00640 1 -- 172.21.15.90:0/1422099236 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda48034040 con 0x7fda5c0cedd0 2024-03-24T01:38:37.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.046+0000 7fda61800640 1 -- 172.21.15.90:0/1422099236 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda5c0cbb40 con 0x7fda5c0cedd0 2024-03-24T01:38:37.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.046+0000 7fda61800640 1 -- 172.21.15.90:0/1422099236 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda5c0cc100 con 0x7fda5c0cedd0 2024-03-24T01:38:37.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.048+0000 7fda61800640 1 -- 172.21.15.90:0/1422099236 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda5c04f400 con 0x7fda5c0cedd0 2024-03-24T01:38:37.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.048+0000 7fda52a00640 1 -- 172.21.15.90:0/1422099236 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fda48037070 con 0x7fda5c0cedd0 2024-03-24T01:38:37.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.049+0000 7fda52a00640 1 --2- 172.21.15.90:0/1422099236 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fda3c06bbd0 0x7fda3c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:37.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.049+0000 7fda52a00640 1 -- 172.21.15.90:0/1422099236 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7fda480acb30 con 0x7fda5c0cedd0 2024-03-24T01:38:37.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.050+0000 7fda5aa00640 1 --2- 172.21.15.90:0/1422099236 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fda3c06bbd0 0x7fda3c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:37.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.051+0000 7fda5aa00640 1 --2- 172.21.15.90:0/1422099236 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fda3c06bbd0 0x7fda3c06e090 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fda4c001500 tx=0x7fda4c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:37.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.053+0000 7fda52a00640 1 -- 172.21.15.90:0/1422099236 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fda4807b510 con 0x7fda5c0cedd0 2024-03-24T01:38:37.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.176+0000 7fd326a00640 1 -- 172.21.15.90:0/4276284430 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd32004f400 con 0x7fd32015dec0 2024-03-24T01:38:37.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.177+0000 7fd31d600640 1 -- 172.21.15.90:0/4276284430 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v69) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd318058bc0 con 0x7fd32015dec0 2024-03-24T01:38:37.177 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-24T01:38:37.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.179+0000 7fd326a00640 1 -- 172.21.15.90:0/4276284430 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd30006bbd0 msgr2=0x7fd30006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:37.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.179+0000 7fd326a00640 1 --2- 172.21.15.90:0/4276284430 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd30006bbd0 0x7fd30006e090 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd3140097a0 tx=0x7fd31403c040 comp rx=0 tx=0).stop 2024-03-24T01:38:37.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.179+0000 7fd326a00640 1 -- 172.21.15.90:0/4276284430 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd32015dec0 msgr2=0x7fd3201f3b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:37.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.179+0000 7fd326a00640 1 --2- 172.21.15.90:0/4276284430 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd32015dec0 0x7fd3201f3b70 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fd31800cbf0 tx=0x7fd31800a430 comp rx=0 tx=0).stop 2024-03-24T01:38:37.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.179+0000 7fd326a00640 1 -- 172.21.15.90:0/4276284430 shutdown_connections 2024-03-24T01:38:37.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.179+0000 7fd326a00640 1 --2- 172.21.15.90:0/4276284430 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fd30006bbd0 0x7fd30006e090 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:37.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.179+0000 7fd326a00640 1 --2- 172.21.15.90:0/4276284430 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd32015dec0 0x7fd3201f3b70 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:37.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.179+0000 7fd326a00640 1 --2- 172.21.15.90:0/4276284430 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fd3200d2e20 0x7fd3200c7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:37.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.179+0000 7fd326a00640 1 --2- 172.21.15.90:0/4276284430 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3200b5ab0 0x7fd3201f4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:37.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.179+0000 7fd326a00640 1 -- 172.21.15.90:0/4276284430 >> 172.21.15.90:0/4276284430 conn(0x7fd3200d0750 msgr2=0x7fd320153c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:37.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.180+0000 7fd326a00640 1 -- 172.21.15.90:0/4276284430 shutdown_connections 2024-03-24T01:38:37.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.180+0000 7fd326a00640 1 -- 172.21.15.90:0/4276284430 wait complete. 2024-03-24T01:38:37.198 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T01:38:37.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.284+0000 7fda61800640 1 -- 172.21.15.90:0/1422099236 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fda5c161350 con 0x7fda5c0cedd0 2024-03-24T01:38:37.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.285+0000 7fda52a00640 1 -- 172.21.15.90:0/1422099236 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v69) v1 ==== 80+0+0 (secure 0 0 0) 0x7fda4807ae10 con 0x7fda5c0cedd0 2024-03-24T01:38:37.285 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-24T01:38:37.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.288+0000 7fda61800640 1 -- 172.21.15.90:0/1422099236 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fda3c06bbd0 msgr2=0x7fda3c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:37.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.288+0000 7fda61800640 1 --2- 172.21.15.90:0/1422099236 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fda3c06bbd0 0x7fda3c06e090 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fda4c001500 tx=0x7fda4c009040 comp rx=0 tx=0).stop 2024-03-24T01:38:37.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.288+0000 7fda61800640 1 -- 172.21.15.90:0/1422099236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda5c0cedd0 msgr2=0x7fda5c157220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:37.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.288+0000 7fda61800640 1 --2- 172.21.15.90:0/1422099236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda5c0cedd0 0x7fda5c157220 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fda4802b9e0 tx=0x7fda48004730 comp rx=0 tx=0).stop 2024-03-24T01:38:37.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.288+0000 7fda61800640 1 -- 172.21.15.90:0/1422099236 shutdown_connections 2024-03-24T01:38:37.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.288+0000 7fda61800640 1 --2- 172.21.15.90:0/1422099236 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fda3c06bbd0 0x7fda3c06e090 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:37.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.288+0000 7fda61800640 1 --2- 172.21.15.90:0/1422099236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda5c0cedd0 0x7fda5c157220 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:37.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.288+0000 7fda61800640 1 --2- 172.21.15.90:0/1422099236 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fda5c15df40 0x7fda5c15f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:37.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.288+0000 7fda61800640 1 --2- 172.21.15.90:0/1422099236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda5c0b94e0 0x7fda5c0b46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:37.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.288+0000 7fda61800640 1 -- 172.21.15.90:0/1422099236 >> 172.21.15.90:0/1422099236 conn(0x7fda5c06c9f0 msgr2=0x7fda5c0b6cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:37.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.289+0000 7fda61800640 1 -- 172.21.15.90:0/1422099236 shutdown_connections 2024-03-24T01:38:37.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.289+0000 7fda61800640 1 -- 172.21.15.90:0/1422099236 wait complete. 2024-03-24T01:38:37.311 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:38:37.311 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:38:37.354 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:38:37.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.387+0000 7f5e7de00640 1 Processor -- start 2024-03-24T01:38:37.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.387+0000 7f5e7de00640 1 -- start start 2024-03-24T01:38:37.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.388+0000 7f5e7de00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5e7815de10 0x7f5e780acea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:37.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.389+0000 7f5e7de00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e780af140 con 0x7f5e7815da30 2024-03-24T01:38:37.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.389+0000 7f5e7de00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e7815e550 con 0x7f5e7806b0c0 2024-03-24T01:38:37.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.389+0000 7f5e7de00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e780b2c70 con 0x7f5e7815de10 2024-03-24T01:38:37.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.389+0000 7f5e76a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f5e7815da30 0x7f5e780aea30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:37110/0 (socket says 172.21.15.90:37110) 2024-03-24T01:38:37.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.389+0000 7f5e76a00640 1 -- 172.21.15.90:0/2692101344 learned_addr learned my addr 172.21.15.90:0/2692101344 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:38:37.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.389+0000 7f5e76000640 1 --2- 172.21.15.90:0/2692101344 >> v2:172.21.15.90:3300/0 conn(0x7f5e7815de10 0x7f5e780acea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:37.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.389+0000 7f5e75600640 1 -- 172.21.15.90:0/2692101344 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 768762945 0 0) 0x7f5e780af140 con 0x7f5e7815da30 2024-03-24T01:38:37.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.389+0000 7f5e76000640 1 -- 172.21.15.90:0/2692101344 >> v1:172.21.15.120:6789/0 conn(0x7f5e7806b0c0 legacy=0x7f5e78161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:38:37.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.389+0000 7f5e76000640 1 -- 172.21.15.90:0/2692101344 >> v1:172.21.15.90:6789/0 conn(0x7f5e7815da30 legacy=0x7f5e780aea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:37.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.389+0000 7f5e76000640 1 -- 172.21.15.90:0/2692101344 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e7815e830 con 0x7f5e7815de10 2024-03-24T01:38:37.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.390+0000 7f5e76000640 1 --2- 172.21.15.90:0/2692101344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e7815de10 0x7f5e780acea0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f5e600097a0 tx=0x7f5e6002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b7f233a81c98974 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:37.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.391+0000 7f5e75600640 1 -- 172.21.15.90:0/2692101344 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e60035020 con 0x7f5e7815de10 2024-03-24T01:38:37.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.391+0000 7f5e75600640 1 -- 172.21.15.90:0/2692101344 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e60031830 con 0x7f5e7815de10 2024-03-24T01:38:37.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.391+0000 7f5e75600640 1 -- 172.21.15.90:0/2692101344 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e60031c00 con 0x7f5e7815de10 2024-03-24T01:38:37.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.391+0000 7f5e7de00640 1 -- 172.21.15.90:0/2692101344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e7815de10 msgr2=0x7f5e780acea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:37.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.391+0000 7f5e7de00640 1 --2- 172.21.15.90:0/2692101344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e7815de10 0x7f5e780acea0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f5e600097a0 tx=0x7f5e6002b8d0 comp rx=0 tx=0).stop 2024-03-24T01:38:37.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.391+0000 7f5e7de00640 1 -- 172.21.15.90:0/2692101344 shutdown_connections 2024-03-24T01:38:37.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.391+0000 7f5e7de00640 1 --2- 172.21.15.90:0/2692101344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e7815de10 0x7f5e780acea0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:37.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.391+0000 7f5e7de00640 1 -- 172.21.15.90:0/2692101344 >> 172.21.15.90:0/2692101344 conn(0x7f5e780b1770 msgr2=0x7f5e780b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:37.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.392+0000 7f5e7de00640 1 -- 172.21.15.90:0/2692101344 shutdown_connections 2024-03-24T01:38:37.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.392+0000 7f5e7de00640 1 -- 172.21.15.90:0/2692101344 wait complete. 2024-03-24T01:38:37.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.393+0000 7f5e7de00640 1 Processor -- start 2024-03-24T01:38:37.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.393+0000 7f5e7de00640 1 -- start start 2024-03-24T01:38:37.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.394+0000 7f5e7de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e7806b0c0 0x7f5e78143340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:37.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.394+0000 7f5e7de00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e7815da30 0x7f5e7814ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:37.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.394+0000 7f5e7de00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e7815de10 0x7f5e7815c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:37.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.394+0000 7f5e7de00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e780ad3e0 con 0x7f5e7806b0c0 2024-03-24T01:38:37.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.394+0000 7f5e7de00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e7815f160 con 0x7f5e7815de10 2024-03-24T01:38:37.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.394+0000 7f5e7de00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e7815edf0 con 0x7f5e7815da30 2024-03-24T01:38:37.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.394+0000 7f5e76a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e7806b0c0 0x7f5e78143340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:37.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.395+0000 7f5e76a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e7806b0c0 0x7f5e78143340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38346/0 (socket says 172.21.15.90:38346) 2024-03-24T01:38:37.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.395+0000 7f5e76a00640 1 -- 172.21.15.90:0/2465281207 learned_addr learned my addr 172.21.15.90:0/2465281207 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:38:37.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.395+0000 7f5e76000640 1 --2- 172.21.15.90:0/2465281207 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e7815da30 0x7f5e7814ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:37.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.395+0000 7f5e77400640 1 --2- 172.21.15.90:0/2465281207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e7815de10 0x7f5e7815c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:37.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.395+0000 7f5e76a00640 1 -- 172.21.15.90:0/2465281207 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e7815da30 msgr2=0x7f5e7814ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:37.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.395+0000 7f5e76a00640 1 --2- 172.21.15.90:0/2465281207 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e7815da30 0x7f5e7814ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:37.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.395+0000 7f5e76a00640 1 -- 172.21.15.90:0/2465281207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e7815de10 msgr2=0x7f5e7815c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:37.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.395+0000 7f5e76a00640 1 --2- 172.21.15.90:0/2465281207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e7815de10 0x7f5e7815c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:37.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.395+0000 7f5e76a00640 1 -- 172.21.15.90:0/2465281207 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e781518f0 con 0x7f5e7806b0c0 2024-03-24T01:38:37.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.395+0000 7f5e77400640 1 --2- 172.21.15.90:0/2465281207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e7815de10 0x7f5e7815c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:38:37.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.395+0000 7f5e76a00640 1 --2- 172.21.15.90:0/2465281207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e7806b0c0 0x7f5e78143340 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f5e6800c4d0 tx=0x7f5e6800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:37.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.396+0000 7f5e6f400640 1 -- 172.21.15.90:0/2465281207 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e68018070 con 0x7f5e7806b0c0 2024-03-24T01:38:37.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.396+0000 7f5e6f400640 1 -- 172.21.15.90:0/2465281207 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e68007500 con 0x7f5e7806b0c0 2024-03-24T01:38:37.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.396+0000 7f5e6f400640 1 -- 172.21.15.90:0/2465281207 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e68004a20 con 0x7f5e7806b0c0 2024-03-24T01:38:37.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.396+0000 7f5e7de00640 1 -- 172.21.15.90:0/2465281207 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e78150850 con 0x7f5e7806b0c0 2024-03-24T01:38:37.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.396+0000 7f5e7de00640 1 -- 172.21.15.90:0/2465281207 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e781510f0 con 0x7f5e7806b0c0 2024-03-24T01:38:37.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.397+0000 7f5e7de00640 1 -- 172.21.15.90:0/2465281207 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e7804f400 con 0x7f5e7806b0c0 2024-03-24T01:38:37.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.398+0000 7f5e6f400640 1 -- 172.21.15.90:0/2465281207 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f5e6801b070 con 0x7f5e7806b0c0 2024-03-24T01:38:37.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.399+0000 7f5e6f400640 1 --2- 172.21.15.90:0/2465281207 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5e5406bbd0 0x7f5e5406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:37.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.399+0000 7f5e6f400640 1 -- 172.21.15.90:0/2465281207 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7f5e68089ff0 con 0x7f5e7806b0c0 2024-03-24T01:38:37.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.399+0000 7f5e76000640 1 --2- 172.21.15.90:0/2465281207 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5e5406bbd0 0x7f5e5406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:37.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.400+0000 7f5e76000640 1 --2- 172.21.15.90:0/2465281207 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5e5406bbd0 0x7f5e5406e090 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f5e60036000 tx=0x7f5e6003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:37.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.403+0000 7f5e6f400640 1 -- 172.21.15.90:0/2465281207 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f5e781518f0 con 0x7f5e7806b0c0 2024-03-24T01:38:37.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.561+0000 7fe3aa200640 1 Processor -- start 2024-03-24T01:38:37.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.561+0000 7fe3aa200640 1 -- start start 2024-03-24T01:38:37.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.562+0000 7fe3aa200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fe3a40b5c40 0x7fe3a4162330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:37.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.562+0000 7fe3aa200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe3a415df80 0x7fe3a40d1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:37.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.562+0000 7fe3aa200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a40ce4b0 con 0x7fe3a415dba0 2024-03-24T01:38:37.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.562+0000 7fe3aa200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3a4162870 con 0x7fe3a415df80 2024-03-24T01:38:37.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.562+0000 7fe3aa200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3a40ce790 con 0x7fe3a40b5c40 2024-03-24T01:38:37.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.563+0000 7fe3a9800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe3a415df80 0x7fe3a40d1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:37.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.563+0000 7fe3a9800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe3a415df80 0x7fe3a40d1d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38350/0 (socket says 172.21.15.90:38350) 2024-03-24T01:38:37.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.563+0000 7fe3a9800640 1 -- 172.21.15.90:0/840952943 learned_addr learned my addr 172.21.15.90:0/840952943 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:38:37.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.563+0000 7fe3a3e00640 1 --2- 172.21.15.90:0/840952943 >> v2:172.21.15.120:3300/0 conn(0x7fe3a40b5c40 0x7fe3a4162330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:37.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.563+0000 7fe3a9800640 1 -- 172.21.15.90:0/840952943 >> v2:172.21.15.120:3300/0 conn(0x7fe3a40b5c40 msgr2=0x7fe3a4162330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:38:37.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.563+0000 7fe3a9800640 1 --2- 172.21.15.90:0/840952943 >> v2:172.21.15.120:3300/0 conn(0x7fe3a40b5c40 0x7fe3a4162330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:37.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.563+0000 7fe3a9800640 1 -- 172.21.15.90:0/840952943 >> v1:172.21.15.120:6789/0 conn(0x7fe3a415dba0 legacy=0x7fe3a40b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:37.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.563+0000 7fe3a9800640 1 -- 172.21.15.90:0/840952943 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a40cea70 con 0x7fe3a415df80 2024-03-24T01:38:37.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.564+0000 7fe3a9800640 1 --2- 172.21.15.90:0/840952943 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3a415df80 0x7fe3a40d1d10 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fe39c009e50 tx=0x7fe39c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23b1eb4fdcd85b64 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:37.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.564+0000 7fe3a3400640 1 -- 172.21.15.90:0/840952943 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39c02ca80 con 0x7fe3a415df80 2024-03-24T01:38:37.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.564+0000 7fe3a3400640 1 -- 172.21.15.90:0/840952943 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe39c02cc20 con 0x7fe3a415df80 2024-03-24T01:38:37.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.564+0000 7fe3a3400640 1 -- 172.21.15.90:0/840952943 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39c0343f0 con 0x7fe3a415df80 2024-03-24T01:38:37.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.565+0000 7fe3aa200640 1 -- 172.21.15.90:0/840952943 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3a415df80 msgr2=0x7fe3a40d1d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:37.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.565+0000 7fe3aa200640 1 --2- 172.21.15.90:0/840952943 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3a415df80 0x7fe3a40d1d10 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fe39c009e50 tx=0x7fe39c02bee0 comp rx=0 tx=0).stop 2024-03-24T01:38:37.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.565+0000 7fe3aa200640 1 -- 172.21.15.90:0/840952943 shutdown_connections 2024-03-24T01:38:37.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.565+0000 7fe3aa200640 1 --2- 172.21.15.90:0/840952943 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3a415df80 0x7fe3a40d1d10 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:37.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.565+0000 7fe3aa200640 1 --2- 172.21.15.90:0/840952943 >> v2:172.21.15.120:3300/0 conn(0x7fe3a40b5c40 0x7fe3a4162330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:37.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.565+0000 7fe3aa200640 1 -- 172.21.15.90:0/840952943 >> 172.21.15.90:0/840952943 conn(0x7fe3a40d0530 msgr2=0x7fe3a40d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:37.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.565+0000 7fe3aa200640 1 -- 172.21.15.90:0/840952943 shutdown_connections 2024-03-24T01:38:37.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.565+0000 7fe3aa200640 1 -- 172.21.15.90:0/840952943 wait complete. 2024-03-24T01:38:37.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.567+0000 7fe3aa200640 1 Processor -- start 2024-03-24T01:38:37.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.567+0000 7fe3aa200640 1 -- start start 2024-03-24T01:38:37.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.567+0000 7fe3aa200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3a40b5c40 0x7fe3a40ca080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:37.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.567+0000 7fe3aa200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3a415dba0 0x7fe3a4159250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:37.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.568+0000 7fe3aa200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a415df80 0x7fe3a415aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:37.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.568+0000 7fe3aa200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a40b2ae0 con 0x7fe3a40b5c40 2024-03-24T01:38:37.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.568+0000 7fe3aa200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a40b2960 con 0x7fe3a415df80 2024-03-24T01:38:37.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.568+0000 7fe3aa200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a40b2c60 con 0x7fe3a415dba0 2024-03-24T01:38:37.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.568+0000 7fe3a8e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3a40b5c40 0x7fe3a40ca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:37.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.568+0000 7fe3a8e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3a40b5c40 0x7fe3a40ca080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38362/0 (socket says 172.21.15.90:38362) 2024-03-24T01:38:37.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.568+0000 7fe3a8e00640 1 -- 172.21.15.90:0/1483419623 learned_addr learned my addr 172.21.15.90:0/1483419623 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:38:37.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.568+0000 7fe3a3e00640 1 --2- 172.21.15.90:0/1483419623 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3a415dba0 0x7fe3a4159250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:37.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.568+0000 7fe3a8e00640 1 -- 172.21.15.90:0/1483419623 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3a415dba0 msgr2=0x7fe3a4159250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:37.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.568+0000 7fe3a8e00640 1 --2- 172.21.15.90:0/1483419623 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3a415dba0 0x7fe3a4159250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:37.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.568+0000 7fe3a9800640 1 --2- 172.21.15.90:0/1483419623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a415df80 0x7fe3a415aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:37.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.568+0000 7fe3a8e00640 1 -- 172.21.15.90:0/1483419623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a415df80 msgr2=0x7fe3a415aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:37.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.568+0000 7fe3a8e00640 1 --2- 172.21.15.90:0/1483419623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a415df80 0x7fe3a415aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:37.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.568+0000 7fe3a8e00640 1 -- 172.21.15.90:0/1483419623 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a40c8b30 con 0x7fe3a40b5c40 2024-03-24T01:38:37.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.568+0000 7fe3a3e00640 1 --2- 172.21.15.90:0/1483419623 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3a415dba0 0x7fe3a4159250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:38:37.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.569+0000 7fe3a8e00640 1 --2- 172.21.15.90:0/1483419623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3a40b5c40 0x7fe3a40ca080 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fe39800c430 tx=0x7fe39800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:37.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.569+0000 7fe3a0c00640 1 -- 172.21.15.90:0/1483419623 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3980052d0 con 0x7fe3a40b5c40 2024-03-24T01:38:37.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.569+0000 7fe3a0c00640 1 -- 172.21.15.90:0/1483419623 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe398005490 con 0x7fe3a40b5c40 2024-03-24T01:38:37.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.569+0000 7fe3aa200640 1 -- 172.21.15.90:0/1483419623 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3a40c8e10 con 0x7fe3a40b5c40 2024-03-24T01:38:37.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.569+0000 7fe3a0c00640 1 -- 172.21.15.90:0/1483419623 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe398007260 con 0x7fe3a40b5c40 2024-03-24T01:38:37.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.569+0000 7fe3aa200640 1 -- 172.21.15.90:0/1483419623 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3a40c93d0 con 0x7fe3a40b5c40 2024-03-24T01:38:37.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.570+0000 7fe3aa200640 1 -- 172.21.15.90:0/1483419623 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3a404f400 con 0x7fe3a40b5c40 2024-03-24T01:38:37.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.572+0000 7fe3a0c00640 1 -- 172.21.15.90:0/1483419623 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fe398007400 con 0x7fe3a40b5c40 2024-03-24T01:38:37.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.573+0000 7fe3a0c00640 1 --2- 172.21.15.90:0/1483419623 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe38406bbd0 0x7fe38406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:37.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.573+0000 7fe3a0c00640 1 -- 172.21.15.90:0/1483419623 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7fe39808a1a0 con 0x7fe3a40b5c40 2024-03-24T01:38:37.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.573+0000 7fe3a3e00640 1 --2- 172.21.15.90:0/1483419623 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe38406bbd0 0x7fe38406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:37.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.574+0000 7fe3a3e00640 1 --2- 172.21.15.90:0/1483419623 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe38406bbd0 0x7fe38406e090 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fe3a40b9b80 tx=0x7fe390008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:37.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.578+0000 7fe3a0c00640 1 -- 172.21.15.90:0/1483419623 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fe398058a70 con 0x7fe3a40b5c40 2024-03-24T01:38:37.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.611+0000 7f5e7de00640 1 -- 172.21.15.90:0/2465281207 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f5e7806e250 con 0x7f5e7806b0c0 2024-03-24T01:38:37.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.611+0000 7f5e6f400640 1 -- 172.21.15.90:0/2465281207 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v69) v1 ==== 80+0+0 (secure 0 0 0) 0x7f5e68058860 con 0x7f5e7806b0c0 2024-03-24T01:38:37.612 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-24T01:38:37.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.615+0000 7f5e7de00640 1 -- 172.21.15.90:0/2465281207 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5e5406bbd0 msgr2=0x7f5e5406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:37.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.615+0000 7f5e7de00640 1 --2- 172.21.15.90:0/2465281207 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5e5406bbd0 0x7f5e5406e090 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f5e60036000 tx=0x7f5e6003c040 comp rx=0 tx=0).stop 2024-03-24T01:38:37.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.615+0000 7f5e7de00640 1 -- 172.21.15.90:0/2465281207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e7806b0c0 msgr2=0x7f5e78143340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:37.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.615+0000 7f5e7de00640 1 --2- 172.21.15.90:0/2465281207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e7806b0c0 0x7f5e78143340 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f5e6800c4d0 tx=0x7f5e6800c990 comp rx=0 tx=0).stop 2024-03-24T01:38:37.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.615+0000 7f5e7de00640 1 -- 172.21.15.90:0/2465281207 shutdown_connections 2024-03-24T01:38:37.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.615+0000 7f5e7de00640 1 --2- 172.21.15.90:0/2465281207 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f5e5406bbd0 0x7f5e5406e090 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:37.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.615+0000 7f5e7de00640 1 --2- 172.21.15.90:0/2465281207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e7815de10 0x7f5e7815c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:37.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.615+0000 7f5e7de00640 1 --2- 172.21.15.90:0/2465281207 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f5e7815da30 0x7f5e7814ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:37.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.615+0000 7f5e7de00640 1 --2- 172.21.15.90:0/2465281207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e7806b0c0 0x7f5e78143340 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:37.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.615+0000 7f5e7de00640 1 -- 172.21.15.90:0/2465281207 >> 172.21.15.90:0/2465281207 conn(0x7f5e780b1770 msgr2=0x7f5e7806f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:37.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.615+0000 7f5e7de00640 1 -- 172.21.15.90:0/2465281207 shutdown_connections 2024-03-24T01:38:37.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.615+0000 7f5e7de00640 1 -- 172.21.15.90:0/2465281207 wait complete. 2024-03-24T01:38:37.635 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi090.front.sepia.ceph.com... 2024-03-24T01:38:37.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:38:37.635 DEBUG:teuthology.orchestra.run.smithi090:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-24T01:38:37.761 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-03-24T01:38:37.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:38:37.762 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T01:38:37.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.778+0000 7fe3aa200640 1 -- 172.21.15.90:0/1483419623 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3a40d17c0 con 0x7fe3a40b5c40 2024-03-24T01:38:37.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.780+0000 7fe3a0c00640 1 -- 172.21.15.90:0/1483419623 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+17101 (secure 0 0 0) 0x7fe398058370 con 0x7fe3a40b5c40 2024-03-24T01:38:37.780 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:38:37.781 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":69,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:32:54.821325+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:32:42.026147+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-24T01:32:43.180712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.1e","mappings":[{"from":7,"to":2}]},{"pgid":"10.10","mappings":[{"from":4,"to":3}]},{"pgid":"10.1d","mappings":[{"from":4,"to":3}]},{"pgid":"10.1f","mappings":[{"from":7,"to":3}]},{"pgid":"10.37","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:38:37.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.782+0000 7fe3aa200640 1 -- 172.21.15.90:0/1483419623 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe38406bbd0 msgr2=0x7fe38406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:37.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.782+0000 7fe3aa200640 1 --2- 172.21.15.90:0/1483419623 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe38406bbd0 0x7fe38406e090 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fe3a40b9b80 tx=0x7fe390008040 comp rx=0 tx=0).stop 2024-03-24T01:38:37.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.782+0000 7fe3aa200640 1 -- 172.21.15.90:0/1483419623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3a40b5c40 msgr2=0x7fe3a40ca080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:37.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.782+0000 7fe3aa200640 1 --2- 172.21.15.90:0/1483419623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3a40b5c40 0x7fe3a40ca080 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fe39800c430 tx=0x7fe39800c8f0 comp rx=0 tx=0).stop 2024-03-24T01:38:37.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.782+0000 7fe3aa200640 1 -- 172.21.15.90:0/1483419623 shutdown_connections 2024-03-24T01:38:37.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.782+0000 7fe3aa200640 1 --2- 172.21.15.90:0/1483419623 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fe38406bbd0 0x7fe38406e090 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:37.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.782+0000 7fe3aa200640 1 --2- 172.21.15.90:0/1483419623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a415df80 0x7fe3a415aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:37.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.782+0000 7fe3aa200640 1 --2- 172.21.15.90:0/1483419623 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fe3a415dba0 0x7fe3a4159250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:37.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.782+0000 7fe3aa200640 1 --2- 172.21.15.90:0/1483419623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe3a40b5c40 0x7fe3a40ca080 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:37.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.782+0000 7fe3aa200640 1 -- 172.21.15.90:0/1483419623 >> 172.21.15.90:0/1483419623 conn(0x7fe3a40d0530 msgr2=0x7fe3a40b7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:37.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.782+0000 7fe3aa200640 1 -- 172.21.15.90:0/1483419623 shutdown_connections 2024-03-24T01:38:37.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:37.783+0000 7fe3aa200640 1 -- 172.21.15.90:0/1483419623 wait complete. 2024-03-24T01:38:37.799 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T01:32:42.026147+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T01:32:43.180712+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T01:38:37.800 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:38:37.805 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:38:37.805 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:38:37.855 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T01:38:37.855 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2024-03-24T01:38:37.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:38:37.855 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-24T01:38:37.946 INFO:teuthology.orchestra.run.smithi120.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-24T01:38:37.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T01:38:37.947 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:38:37.947 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:38:37.993 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053414,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498340k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249168k,nr_inodes=812292,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498340k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T01:38:37.993 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-03-24T01:38:37.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:38:37.993 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-24T01:38:38.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.048+0000 7f3884600640 1 Processor -- start 2024-03-24T01:38:38.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.048+0000 7f3884600640 1 -- start start 2024-03-24T01:38:38.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.049+0000 7f3884600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f387c0d3e70 0x7f387c0d1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:38.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.049+0000 7f3884600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f387c0d4250 0x7f387c15d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:38.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.049+0000 7f3884600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f387c0d2740 con 0x7f387c15dd00 2024-03-24T01:38:38.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.049+0000 7f3884600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f387c0d2a20 con 0x7f387c0d3e70 2024-03-24T01:38:38.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.049+0000 7f3884600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f387c0d2d00 con 0x7f387c0d4250 2024-03-24T01:38:38.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.050+0000 7f3881c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f387c0d3e70 0x7f387c0d1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:38.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.050+0000 7f3882600640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f387c15dd00 0x7f387c0b56f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:37114/0 (socket says 172.21.15.90:37114) 2024-03-24T01:38:38.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.050+0000 7f3882600640 1 -- 172.21.15.90:0/858138845 learned_addr learned my addr 172.21.15.90:0/858138845 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:38:38.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.050+0000 7f3881200640 1 --2- 172.21.15.90:0/858138845 >> v2:172.21.15.120:3300/0 conn(0x7f387c0d4250 0x7f387c15d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:38.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.050+0000 7f3881c00640 1 -- 172.21.15.90:0/858138845 >> v2:172.21.15.120:3300/0 conn(0x7f387c0d4250 msgr2=0x7f387c15d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:38:38.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.050+0000 7f3881c00640 1 --2- 172.21.15.90:0/858138845 >> v2:172.21.15.120:3300/0 conn(0x7f387c0d4250 0x7f387c15d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:38.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.050+0000 7f3881c00640 1 -- 172.21.15.90:0/858138845 >> v1:172.21.15.90:6789/0 conn(0x7f387c15dd00 legacy=0x7f387c0b56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:38.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.050+0000 7f3881c00640 1 -- 172.21.15.90:0/858138845 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f387c0d2fe0 con 0x7f387c0d3e70 2024-03-24T01:38:38.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.051+0000 7f3881c00640 1 --2- 172.21.15.90:0/858138845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f387c0d3e70 0x7f387c0d1dd0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f38700097a0 tx=0x7f387002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93168bf68e25fb2c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:38.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.052+0000 7f387be00640 1 -- 172.21.15.90:0/858138845 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3870035020 con 0x7f387c0d3e70 2024-03-24T01:38:38.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.052+0000 7f387be00640 1 -- 172.21.15.90:0/858138845 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3870031830 con 0x7f387c0d3e70 2024-03-24T01:38:38.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.052+0000 7f387be00640 1 -- 172.21.15.90:0/858138845 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3870031c00 con 0x7f387c0d3e70 2024-03-24T01:38:38.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.052+0000 7f3884600640 1 -- 172.21.15.90:0/858138845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f387c0d3e70 msgr2=0x7f387c0d1dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:38.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.052+0000 7f3884600640 1 --2- 172.21.15.90:0/858138845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f387c0d3e70 0x7f387c0d1dd0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f38700097a0 tx=0x7f387002b8d0 comp rx=0 tx=0).stop 2024-03-24T01:38:38.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.052+0000 7f3884600640 1 -- 172.21.15.90:0/858138845 shutdown_connections 2024-03-24T01:38:38.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.052+0000 7f3884600640 1 --2- 172.21.15.90:0/858138845 >> v2:172.21.15.120:3300/0 conn(0x7f387c0d4250 0x7f387c15d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:38.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.052+0000 7f3884600640 1 --2- 172.21.15.90:0/858138845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f387c0d3e70 0x7f387c0d1dd0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:38.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.052+0000 7f3884600640 1 -- 172.21.15.90:0/858138845 >> 172.21.15.90:0/858138845 conn(0x7f387c072c10 msgr2=0x7f387c073020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:38.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.052+0000 7f3884600640 1 -- 172.21.15.90:0/858138845 shutdown_connections 2024-03-24T01:38:38.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.052+0000 7f3884600640 1 -- 172.21.15.90:0/858138845 wait complete. 2024-03-24T01:38:38.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.054+0000 7f3884600640 1 Processor -- start 2024-03-24T01:38:38.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.054+0000 7f3884600640 1 -- start start 2024-03-24T01:38:38.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.055+0000 7f3884600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f387c0d3e70 0x7f387c152440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:38.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.055+0000 7f3884600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f387c0d4250 0x7f387c14ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:38.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.055+0000 7f3884600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f387c15dd00 0x7f387c15f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:38.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.055+0000 7f3884600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f387c0b7e20 con 0x7f387c0d3e70 2024-03-24T01:38:38.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.055+0000 7f3884600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f387c15e170 con 0x7f387c0d4250 2024-03-24T01:38:38.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.055+0000 7f3884600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f387c0d37c0 con 0x7f387c15dd00 2024-03-24T01:38:38.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.055+0000 7f3881c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f387c0d3e70 0x7f387c152440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:38.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.055+0000 7f3881c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f387c0d3e70 0x7f387c152440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38382/0 (socket says 172.21.15.90:38382) 2024-03-24T01:38:38.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.055+0000 7f3881c00640 1 -- 172.21.15.90:0/4146075275 learned_addr learned my addr 172.21.15.90:0/4146075275 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:38:38.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.056+0000 7f3881c00640 1 -- 172.21.15.90:0/4146075275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f387c15dd00 msgr2=0x7f387c15f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:38.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.056+0000 7f3881200640 1 --2- 172.21.15.90:0/4146075275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f387c0d4250 0x7f387c14ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:38.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.056+0000 7f3882600640 1 --2- 172.21.15.90:0/4146075275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f387c15dd00 0x7f387c15f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:38.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.056+0000 7f3881c00640 1 --2- 172.21.15.90:0/4146075275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f387c15dd00 0x7f387c15f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:38.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.056+0000 7f3881c00640 1 -- 172.21.15.90:0/4146075275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f387c0d4250 msgr2=0x7f387c14ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:38.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.056+0000 7f3881c00640 1 --2- 172.21.15.90:0/4146075275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f387c0d4250 0x7f387c14ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:38.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.056+0000 7f3881c00640 1 -- 172.21.15.90:0/4146075275 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f387c152980 con 0x7f387c0d3e70 2024-03-24T01:38:38.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.056+0000 7f3881200640 1 --2- 172.21.15.90:0/4146075275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f387c0d4250 0x7f387c14ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:38:38.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.056+0000 7f3881c00640 1 --2- 172.21.15.90:0/4146075275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f387c0d3e70 0x7f387c152440 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f387002bee0 tx=0x7f38700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:38.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.056+0000 7f3879600640 1 -- 172.21.15.90:0/4146075275 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3870041070 con 0x7f387c0d3e70 2024-03-24T01:38:38.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.056+0000 7f3884600640 1 -- 172.21.15.90:0/4146075275 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f387c157bd0 con 0x7f387c0d3e70 2024-03-24T01:38:38.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.056+0000 7f3879600640 1 -- 172.21.15.90:0/4146075275 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f387003b920 con 0x7f387c0d3e70 2024-03-24T01:38:38.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.056+0000 7f3879600640 1 -- 172.21.15.90:0/4146075275 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3870035040 con 0x7f387c0d3e70 2024-03-24T01:38:38.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.056+0000 7f3884600640 1 -- 172.21.15.90:0/4146075275 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f387c153100 con 0x7f387c0d3e70 2024-03-24T01:38:38.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.058+0000 7f3884600640 1 -- 172.21.15.90:0/4146075275 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f387c153430 con 0x7f387c0d3e70 2024-03-24T01:38:38.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.058+0000 7f3879600640 1 -- 172.21.15.90:0/4146075275 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f387004c060 con 0x7f387c0d3e70 2024-03-24T01:38:38.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.059+0000 7f3879600640 1 --2- 172.21.15.90:0/4146075275 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f385c06bbd0 0x7f385c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:38.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.059+0000 7f3879600640 1 -- 172.21.15.90:0/4146075275 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7f38700ac6f0 con 0x7f387c0d3e70 2024-03-24T01:38:38.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.060+0000 7f3881200640 1 --2- 172.21.15.90:0/4146075275 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f385c06bbd0 0x7f385c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:38.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.061+0000 7f3881200640 1 --2- 172.21.15.90:0/4146075275 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f385c06bbd0 0x7f385c06e090 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f387c1f7010 tx=0x7f3868008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:38.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.065+0000 7f3879600640 1 -- 172.21.15.90:0/4146075275 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f387007af30 con 0x7f387c0d3e70 2024-03-24T01:38:38.077 INFO:teuthology.orchestra.run.smithi090.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-24T01:38:38.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T01:38:38.078 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:38:38.079 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:38:38.127 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T01:38:38.127 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2024-03-24T01:38:38.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:38:38.127 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-24T01:38:38.217 INFO:teuthology.orchestra.run.smithi120.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-24T01:38:38.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T01:38:38.219 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:38:38.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.260+0000 7f3884600640 1 -- 172.21.15.90:0/4146075275 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f387c04f400 con 0x7f387c0d3e70 2024-03-24T01:38:38.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.261+0000 7f3879600640 1 -- 172.21.15.90:0/4146075275 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v69) v1 ==== 93+0+10 (secure 0 0 0) 0x7f387007a830 con 0x7f387c0d3e70 2024-03-24T01:38:38.261 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:38:38.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.263+0000 7f3884600640 1 -- 172.21.15.90:0/4146075275 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f385c06bbd0 msgr2=0x7f385c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:38.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.263+0000 7f3884600640 1 --2- 172.21.15.90:0/4146075275 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f385c06bbd0 0x7f385c06e090 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f387c1f7010 tx=0x7f3868008040 comp rx=0 tx=0).stop 2024-03-24T01:38:38.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.263+0000 7f3884600640 1 -- 172.21.15.90:0/4146075275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f387c0d3e70 msgr2=0x7f387c152440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:38.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.263+0000 7f3884600640 1 --2- 172.21.15.90:0/4146075275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f387c0d3e70 0x7f387c152440 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f387002bee0 tx=0x7f38700046c0 comp rx=0 tx=0).stop 2024-03-24T01:38:38.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.263+0000 7f3884600640 1 -- 172.21.15.90:0/4146075275 shutdown_connections 2024-03-24T01:38:38.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.263+0000 7f3884600640 1 --2- 172.21.15.90:0/4146075275 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f385c06bbd0 0x7f385c06e090 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:38.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.263+0000 7f3884600640 1 --2- 172.21.15.90:0/4146075275 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f387c15dd00 0x7f387c15f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:38.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.263+0000 7f3884600640 1 --2- 172.21.15.90:0/4146075275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f387c0d4250 0x7f387c14ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:38.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.263+0000 7f3884600640 1 --2- 172.21.15.90:0/4146075275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f387c0d3e70 0x7f387c152440 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:38.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.263+0000 7f3884600640 1 -- 172.21.15.90:0/4146075275 >> 172.21.15.90:0/4146075275 conn(0x7f387c072c10 msgr2=0x7f387c15c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:38.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.263+0000 7f3884600640 1 -- 172.21.15.90:0/4146075275 shutdown_connections 2024-03-24T01:38:38.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.264+0000 7f3884600640 1 -- 172.21.15.90:0/4146075275 wait complete. 2024-03-24T01:38:38.284 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T01:38:38.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.424+0000 7f4c76c00640 1 Processor -- start 2024-03-24T01:38:38.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.424+0000 7f4c76c00640 1 -- start start 2024-03-24T01:38:38.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.425+0000 7f4c76c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c700ceb30 0x7f4c70168dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:38.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.426+0000 7f4c76c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c70161dd0 con 0x7f4c7015ded0 2024-03-24T01:38:38.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.426+0000 7f4c76c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c700d38b0 con 0x7f4c700d10c0 2024-03-24T01:38:38.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.426+0000 7f4c76c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c700d2410 con 0x7f4c700ceb30 2024-03-24T01:38:38.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.426+0000 7f4c6fe00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4c7015ded0 0x7f4c700d09b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:37128/0 (socket says 172.21.15.90:37128) 2024-03-24T01:38:38.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.426+0000 7f4c6fe00640 1 -- 172.21.15.90:0/1455731301 learned_addr learned my addr 172.21.15.90:0/1455731301 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:38:38.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.426+0000 7f4c74c00640 1 --2- 172.21.15.90:0/1455731301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c700ceb30 0x7f4c70168dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:38.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.426+0000 7f4c6f400640 1 -- 172.21.15.90:0/1455731301 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1815155352 0 0) 0x7f4c60001110 con 0x7f4c7015ded0 2024-03-24T01:38:38.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.427+0000 7f4c6f400640 1 -- 172.21.15.90:0/1455731301 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c54003520 con 0x7f4c7015ded0 2024-03-24T01:38:38.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.427+0000 7f4c6f400640 1 -- 172.21.15.90:0/1455731301 <== mon.1 v1:172.21.15.120:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1110535596 0 0) 0x7f4c700d38b0 con 0x7f4c700d10c0 2024-03-24T01:38:38.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.427+0000 7f4c6f400640 1 -- 172.21.15.90:0/1455731301 --> v1:172.21.15.120:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c60001110 con 0x7f4c700d10c0 2024-03-24T01:38:38.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.427+0000 7f4c74c00640 1 -- 172.21.15.90:0/1455731301 >> v1:172.21.15.120:6789/0 conn(0x7f4c700d10c0 legacy=0x7f4c700ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:38.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.427+0000 7f4c6f400640 1 -- 172.21.15.90:0/1455731301 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4231975657 0 0) 0x7f4c54003520 con 0x7f4c7015ded0 2024-03-24T01:38:38.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.427+0000 7f4c74c00640 1 -- 172.21.15.90:0/1455731301 >> v1:172.21.15.90:6789/0 conn(0x7f4c7015ded0 legacy=0x7f4c700d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:38.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.427+0000 7f4c74c00640 1 -- 172.21.15.90:0/1455731301 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c700cf000 con 0x7f4c700ceb30 2024-03-24T01:38:38.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.428+0000 7f4c74c00640 1 --2- 172.21.15.90:0/1455731301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c700ceb30 0x7f4c70168dd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4c68009dc0 tx=0x7f4c6802bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a89d6c6d4c15e67 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:38.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.429+0000 7f4c6f400640 1 -- 172.21.15.90:0/1455731301 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c68039070 con 0x7f4c700ceb30 2024-03-24T01:38:38.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.429+0000 7f4c6f400640 1 -- 172.21.15.90:0/1455731301 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c6802c910 con 0x7f4c700ceb30 2024-03-24T01:38:38.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.429+0000 7f4c6f400640 1 -- 172.21.15.90:0/1455731301 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c60001840 con 0x7f4c700ceb30 2024-03-24T01:38:38.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.429+0000 7f4c76c00640 1 -- 172.21.15.90:0/1455731301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c700ceb30 msgr2=0x7f4c70168dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:38.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.429+0000 7f4c76c00640 1 --2- 172.21.15.90:0/1455731301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c700ceb30 0x7f4c70168dd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4c68009dc0 tx=0x7f4c6802bde0 comp rx=0 tx=0).stop 2024-03-24T01:38:38.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.429+0000 7f4c76c00640 1 -- 172.21.15.90:0/1455731301 shutdown_connections 2024-03-24T01:38:38.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.429+0000 7f4c76c00640 1 --2- 172.21.15.90:0/1455731301 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c700ceb30 0x7f4c70168dd0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:38.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.429+0000 7f4c76c00640 1 -- 172.21.15.90:0/1455731301 >> 172.21.15.90:0/1455731301 conn(0x7f4c7006c9c0 msgr2=0x7f4c7006cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:38.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.429+0000 7f4c76c00640 1 -- 172.21.15.90:0/1455731301 shutdown_connections 2024-03-24T01:38:38.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.429+0000 7f4c76c00640 1 -- 172.21.15.90:0/1455731301 wait complete. 2024-03-24T01:38:38.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.431+0000 7f4c76c00640 1 Processor -- start 2024-03-24T01:38:38.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.431+0000 7f4c76c00640 1 -- start start 2024-03-24T01:38:38.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.432+0000 7f4c76c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c700ceb30 0x7f4c7015ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.432+0000 7f4c76c00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c700d10c0 0x7f4c700b55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.432+0000 7f4c76c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c7015ded0 0x7f4c700c7500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.432+0000 7f4c76c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c700d20a0 con 0x7f4c7015ded0 2024-03-24T01:38:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.432+0000 7f4c76c00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c700d1dc0 con 0x7f4c700d10c0 2024-03-24T01:38:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.432+0000 7f4c76c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7015e2b0 con 0x7f4c700ceb30 2024-03-24T01:38:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.433+0000 7f4c74c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c7015ded0 0x7f4c700c7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.433+0000 7f4c74c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c7015ded0 0x7f4c700c7500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38392/0 (socket says 172.21.15.90:38392) 2024-03-24T01:38:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.433+0000 7f4c74c00640 1 -- 172.21.15.90:0/2502777680 learned_addr learned my addr 172.21.15.90:0/2502777680 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:38:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.433+0000 7f4c6fe00640 1 --2- 172.21.15.90:0/2502777680 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c700ceb30 0x7f4c7015ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.433+0000 7f4c67e00640 1 --2- 172.21.15.90:0/2502777680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c700d10c0 0x7f4c700b55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.433+0000 7f4c74c00640 1 -- 172.21.15.90:0/2502777680 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c700ceb30 msgr2=0x7f4c7015ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.433+0000 7f4c74c00640 1 --2- 172.21.15.90:0/2502777680 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c700ceb30 0x7f4c7015ffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.433+0000 7f4c74c00640 1 -- 172.21.15.90:0/2502777680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c700d10c0 msgr2=0x7f4c700b55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.433+0000 7f4c74c00640 1 --2- 172.21.15.90:0/2502777680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c700d10c0 0x7f4c700b55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:38.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.433+0000 7f4c74c00640 1 -- 172.21.15.90:0/2502777680 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c70160810 con 0x7f4c7015ded0 2024-03-24T01:38:38.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.433+0000 7f4c67e00640 1 --2- 172.21.15.90:0/2502777680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c700d10c0 0x7f4c700b55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:38:38.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.433+0000 7f4c74c00640 1 --2- 172.21.15.90:0/2502777680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c7015ded0 0x7f4c700c7500 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f4c6802c3f0 tx=0x7f4c68004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:38.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.434+0000 7f4c6cc00640 1 -- 172.21.15.90:0/2502777680 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c68039050 con 0x7f4c7015ded0 2024-03-24T01:38:38.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.434+0000 7f4c6cc00640 1 -- 172.21.15.90:0/2502777680 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c68045070 con 0x7f4c7015ded0 2024-03-24T01:38:38.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.434+0000 7f4c6cc00640 1 -- 172.21.15.90:0/2502777680 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c68009450 con 0x7f4c7015ded0 2024-03-24T01:38:38.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.434+0000 7f4c76c00640 1 -- 172.21.15.90:0/2502777680 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c70160af0 con 0x7f4c7015ded0 2024-03-24T01:38:38.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.434+0000 7f4c76c00640 1 -- 172.21.15.90:0/2502777680 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c700cb7d0 con 0x7f4c7015ded0 2024-03-24T01:38:38.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.435+0000 7f4c76c00640 1 -- 172.21.15.90:0/2502777680 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c700cb9d0 con 0x7f4c7015ded0 2024-03-24T01:38:38.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.436+0000 7f4c6cc00640 1 -- 172.21.15.90:0/2502777680 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f4c68004650 con 0x7f4c7015ded0 2024-03-24T01:38:38.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.437+0000 7f4c6cc00640 1 --2- 172.21.15.90:0/2502777680 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4c4c06bc00 0x7f4c4c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:38.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.437+0000 7f4c6cc00640 1 -- 172.21.15.90:0/2502777680 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7f4c680adf00 con 0x7f4c7015ded0 2024-03-24T01:38:38.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.437+0000 7f4c6fe00640 1 --2- 172.21.15.90:0/2502777680 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4c4c06bc00 0x7f4c4c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:38.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.438+0000 7f4c6fe00640 1 --2- 172.21.15.90:0/2502777680 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4c4c06bc00 0x7f4c4c06e0c0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f4c600010e0 tx=0x7f4c60009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:38.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.446+0000 7f4c6cc00640 1 -- 172.21.15.90:0/2502777680 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f4c700cb9d0 con 0x7f4c7015ded0 2024-03-24T01:38:38.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.535+0000 7f837dc00640 1 Processor -- start 2024-03-24T01:38:38.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.536+0000 7f837dc00640 1 -- start start 2024-03-24T01:38:38.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.537+0000 7f837dc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f837806b7d0 0x7f83781656f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:38.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.537+0000 7f837dc00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f837806dac0 con 0x7f837815bc50 2024-03-24T01:38:38.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.537+0000 7f837dc00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f837806dda0 con 0x7f83780aeeb0 2024-03-24T01:38:38.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.537+0000 7f837dc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83780b4da0 con 0x7f837806b7d0 2024-03-24T01:38:38.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.537+0000 7f8376000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f837815bc50 0x7f837806b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:37142/0 (socket says 172.21.15.90:37142) 2024-03-24T01:38:38.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.537+0000 7f8376000640 1 -- 172.21.15.90:0/1685850390 learned_addr learned my addr 172.21.15.90:0/1685850390 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:38:38.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.538+0000 7f8377400640 1 --2- 172.21.15.90:0/1685850390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f837806b7d0 0x7f83781656f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:38.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.538+0000 7f8375600640 1 -- 172.21.15.90:0/1685850390 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 833700711 0 0) 0x7f837806dac0 con 0x7f837815bc50 2024-03-24T01:38:38.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.538+0000 7f8375600640 1 -- 172.21.15.90:0/1685850390 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f835c003520 con 0x7f837815bc50 2024-03-24T01:38:38.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.538+0000 7f8375600640 1 -- 172.21.15.90:0/1685850390 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3057614160 0 0) 0x7f835c003520 con 0x7f837815bc50 2024-03-24T01:38:38.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.538+0000 7f8375600640 1 -- 172.21.15.90:0/1685850390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f837806b7d0 msgr2=0x7f83781656f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:38:38.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.538+0000 7f8375600640 1 --2- 172.21.15.90:0/1685850390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f837806b7d0 0x7f83781656f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:38.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.538+0000 7f8375600640 1 -- 172.21.15.90:0/1685850390 >> v1:172.21.15.120:6789/0 conn(0x7f83780aeeb0 legacy=0x7f83780b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:38.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.538+0000 7f8375600640 1 -- 172.21.15.90:0/1685850390 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f837806e080 con 0x7f837815bc50 2024-03-24T01:38:38.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.538+0000 7f8375600640 1 -- 172.21.15.90:0/1685850390 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f8360001670 con 0x7f837815bc50 2024-03-24T01:38:38.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.538+0000 7f8375600640 1 -- 172.21.15.90:0/1685850390 >> v1:172.21.15.90:6789/0 conn(0x7f837815bc50 legacy=0x7f837806b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:38.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.538+0000 7f8375600640 1 --2- 172.21.15.90:0/1685850390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f835c0039e0 0x7f835c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:38.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.538+0000 7f8375600640 1 --2- 172.21.15.90:0/1685850390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f837806b7d0 0x7f835c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:38.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.538+0000 7f8375600640 1 --2- 172.21.15.90:0/1685850390 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f835c004d40 0x7f835c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:38.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.539+0000 7f8375600640 1 -- 172.21.15.90:0/1685850390 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83780b4da0 con 0x7f835c004d40 2024-03-24T01:38:38.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.539+0000 7f8375600640 1 -- 172.21.15.90:0/1685850390 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f835c00dd60 con 0x7f835c0039e0 2024-03-24T01:38:38.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.539+0000 7f8375600640 1 -- 172.21.15.90:0/1685850390 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f835c00df30 con 0x7f837806b7d0 2024-03-24T01:38:38.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.539+0000 7f8377400640 1 --2- 172.21.15.90:0/1685850390 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f835c004d40 0x7f835c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:38.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.539+0000 7f8376a00640 1 --2- 172.21.15.90:0/1685850390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f835c0039e0 0x7f835c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:38.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.539+0000 7f8377400640 1 -- 172.21.15.90:0/1685850390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f837806b7d0 msgr2=0x7f835c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:38:38.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.539+0000 7f8376000640 1 --2- 172.21.15.90:0/1685850390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f837806b7d0 0x7f835c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:38.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.539+0000 7f8377400640 1 --2- 172.21.15.90:0/1685850390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f837806b7d0 0x7f835c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:38.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.539+0000 7f8377400640 1 -- 172.21.15.90:0/1685850390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f835c0039e0 msgr2=0x7f835c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:38:38.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.539+0000 7f8377400640 1 --2- 172.21.15.90:0/1685850390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f835c0039e0 0x7f835c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:38.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.539+0000 7f8377400640 1 -- 172.21.15.90:0/1685850390 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f835c003520 con 0x7f835c004d40 2024-03-24T01:38:38.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.539+0000 7f8376a00640 1 --2- 172.21.15.90:0/1685850390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f835c0039e0 0x7f835c003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:38:38.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.539+0000 7f8376000640 1 --2- 172.21.15.90:0/1685850390 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f837806b7d0 0x7f835c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:38:38.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.540+0000 7f8377400640 1 --2- 172.21.15.90:0/1685850390 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f835c004d40 0x7f835c0051c0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f837000e170 tx=0x7f835c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14772277e113b555 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:38.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.541+0000 7f8375600640 1 -- 172.21.15.90:0/1685850390 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8370039070 con 0x7f835c004d40 2024-03-24T01:38:38.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.541+0000 7f8375600640 1 -- 172.21.15.90:0/1685850390 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f837002cbf0 con 0x7f835c004d40 2024-03-24T01:38:38.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.541+0000 7f8375600640 1 -- 172.21.15.90:0/1685850390 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8370009c10 con 0x7f835c004d40 2024-03-24T01:38:38.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.541+0000 7f837dc00640 1 -- 172.21.15.90:0/1685850390 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f835c004d40 msgr2=0x7f835c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:38.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.541+0000 7f837dc00640 1 --2- 172.21.15.90:0/1685850390 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f835c004d40 0x7f835c0051c0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f837000e170 tx=0x7f835c005830 comp rx=0 tx=0).stop 2024-03-24T01:38:38.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.541+0000 7f8377400640 1 -- 172.21.15.90:0/1685850390 reap_dead start 2024-03-24T01:38:38.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.541+0000 7f837dc00640 1 -- 172.21.15.90:0/1685850390 shutdown_connections 2024-03-24T01:38:38.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.541+0000 7f837dc00640 1 -- 172.21.15.90:0/1685850390 >> 172.21.15.90:0/1685850390 conn(0x7f83780b1770 msgr2=0x7f83780b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:38.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.541+0000 7f837dc00640 1 -- 172.21.15.90:0/1685850390 shutdown_connections 2024-03-24T01:38:38.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.541+0000 7f837dc00640 1 -- 172.21.15.90:0/1685850390 wait complete. 2024-03-24T01:38:38.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.543+0000 7f837dc00640 1 Processor -- start 2024-03-24T01:38:38.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.543+0000 7f837dc00640 1 -- start start 2024-03-24T01:38:38.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.544+0000 7f837dc00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f835c004d40 0x7f8378154460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:38.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.544+0000 7f837dc00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83780ae0c0 0x7f837815cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:38.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.544+0000 7f837dc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8378156bb0 0x7f8378157030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:38.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.544+0000 7f837dc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f837806e880 con 0x7f8378156bb0 2024-03-24T01:38:38.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.544+0000 7f837dc00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83780af290 con 0x7f835c004d40 2024-03-24T01:38:38.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.544+0000 7f837dc00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f837806ea00 con 0x7f83780ae0c0 2024-03-24T01:38:38.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.545+0000 7f8377400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8378156bb0 0x7f8378157030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:38.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.545+0000 7f8377400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8378156bb0 0x7f8378157030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38414/0 (socket says 172.21.15.90:38414) 2024-03-24T01:38:38.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.545+0000 7f8377400640 1 -- 172.21.15.90:0/3246537328 learned_addr learned my addr 172.21.15.90:0/3246537328 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:38:38.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.545+0000 7f8376a00640 1 --2- 172.21.15.90:0/3246537328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f835c004d40 0x7f8378154460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:38.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.545+0000 7f8376000640 1 --2- 172.21.15.90:0/3246537328 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83780ae0c0 0x7f837815cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:38.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.545+0000 7f8377400640 1 -- 172.21.15.90:0/3246537328 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83780ae0c0 msgr2=0x7f837815cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:38.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.545+0000 7f8377400640 1 --2- 172.21.15.90:0/3246537328 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83780ae0c0 0x7f837815cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:38.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.545+0000 7f8377400640 1 -- 172.21.15.90:0/3246537328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f835c004d40 msgr2=0x7f8378154460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:38.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.545+0000 7f8377400640 1 --2- 172.21.15.90:0/3246537328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f835c004d40 0x7f8378154460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:38.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.545+0000 7f8377400640 1 -- 172.21.15.90:0/3246537328 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f837815d400 con 0x7f8378156bb0 2024-03-24T01:38:38.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.545+0000 7f8376a00640 1 --2- 172.21.15.90:0/3246537328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f835c004d40 0x7f8378154460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:38:38.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.545+0000 7f8377400640 1 --2- 172.21.15.90:0/3246537328 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8378156bb0 0x7f8378157030 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f8370009f90 tx=0x7f8370009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:38.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.546+0000 7f836ea00640 1 -- 172.21.15.90:0/3246537328 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8370045070 con 0x7f8378156bb0 2024-03-24T01:38:38.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.546+0000 7f836ea00640 1 -- 172.21.15.90:0/3246537328 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f837003a040 con 0x7f8378156bb0 2024-03-24T01:38:38.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.546+0000 7f837dc00640 1 -- 172.21.15.90:0/3246537328 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f837815d6e0 con 0x7f8378156bb0 2024-03-24T01:38:38.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.546+0000 7f836ea00640 1 -- 172.21.15.90:0/3246537328 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8370039070 con 0x7f8378156bb0 2024-03-24T01:38:38.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.546+0000 7f837dc00640 1 -- 172.21.15.90:0/3246537328 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f837815dca0 con 0x7f8378156bb0 2024-03-24T01:38:38.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.547+0000 7f837dc00640 1 -- 172.21.15.90:0/3246537328 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8378157570 con 0x7f8378156bb0 2024-03-24T01:38:38.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.548+0000 7f836ea00640 1 -- 172.21.15.90:0/3246537328 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f8370048070 con 0x7f8378156bb0 2024-03-24T01:38:38.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.549+0000 7f836ea00640 1 --2- 172.21.15.90:0/3246537328 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f835406bc00 0x7f835406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:38.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.549+0000 7f836ea00640 1 -- 172.21.15.90:0/3246537328 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7f83700ac560 con 0x7f8378156bb0 2024-03-24T01:38:38.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.550+0000 7f8376a00640 1 --2- 172.21.15.90:0/3246537328 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f835406bc00 0x7f835406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:38.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.551+0000 7f8376a00640 1 --2- 172.21.15.90:0/3246537328 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f835406bc00 0x7f835406e0c0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f8368001850 tx=0x7f8368009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:38.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.554+0000 7f836ea00640 1 -- 172.21.15.90:0/3246537328 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f837007a860 con 0x7f8378156bb0 2024-03-24T01:38:38.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.688+0000 7f4c76c00640 1 -- 172.21.15.90:0/2502777680 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c700d18d0 con 0x7f4c7015ded0 2024-03-24T01:38:38.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.689+0000 7f4c6cc00640 1 -- 172.21.15.90:0/2502777680 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+5050 (secure 0 0 0) 0x7f4c6803f3b0 con 0x7f4c7015ded0 2024-03-24T01:38:38.690 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:38:38.690 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":58,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5177,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/3458630824","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3458630824},{"type":"v1","addr":"172.21.15.120:6837","nonce":3458630824}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":5614,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/1995195696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1995195696},{"type":"v1","addr":"172.21.15.90:6835","nonce":1995195696}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":58,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T01:32:44.150735+0000","modified":"2024-03-24T01:36:32.782070+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7110,"mds_1":7098},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7098":{"gid":7098,"name":"a","rank":1,"incarnation":52,"state":"up:active","state_seq":7,"addr":"172.21.15.90:6834/1760094626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":1760094626},{"type":"v1","addr":"172.21.15.90:6834","nonce":1760094626}]},"join_fscid":-1,"laggy_since":"2024-03-24T01:33:46.240858+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7110":{"gid":7110,"name":"d","rank":0,"incarnation":46,"state":"up:active","state_seq":60,"addr":"172.21.15.120:6835/910380191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":910380191},{"type":"v1","addr":"172.21.15.120:6835","nonce":910380191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7110,"qdb_cluster":[7110,7098]},"id":4}]} 2024-03-24T01:38:38.690 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 58 2024-03-24T01:38:38.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.692+0000 7f4c76c00640 1 -- 172.21.15.90:0/2502777680 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4c4c06bc00 msgr2=0x7f4c4c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:38.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.692+0000 7f4c76c00640 1 --2- 172.21.15.90:0/2502777680 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4c4c06bc00 0x7f4c4c06e0c0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f4c600010e0 tx=0x7f4c60009040 comp rx=0 tx=0).stop 2024-03-24T01:38:38.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.692+0000 7f4c76c00640 1 -- 172.21.15.90:0/2502777680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c7015ded0 msgr2=0x7f4c700c7500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:38.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.692+0000 7f4c76c00640 1 --2- 172.21.15.90:0/2502777680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c7015ded0 0x7f4c700c7500 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f4c6802c3f0 tx=0x7f4c68004a40 comp rx=0 tx=0).stop 2024-03-24T01:38:38.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.692+0000 7f4c76c00640 1 -- 172.21.15.90:0/2502777680 shutdown_connections 2024-03-24T01:38:38.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.692+0000 7f4c76c00640 1 --2- 172.21.15.90:0/2502777680 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f4c4c06bc00 0x7f4c4c06e0c0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:38.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.692+0000 7f4c76c00640 1 --2- 172.21.15.90:0/2502777680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c7015ded0 0x7f4c700c7500 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:38.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.692+0000 7f4c76c00640 1 --2- 172.21.15.90:0/2502777680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c700d10c0 0x7f4c700b55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:38.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.692+0000 7f4c76c00640 1 --2- 172.21.15.90:0/2502777680 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f4c700ceb30 0x7f4c7015ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:38.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.692+0000 7f4c76c00640 1 -- 172.21.15.90:0/2502777680 >> 172.21.15.90:0/2502777680 conn(0x7f4c7006c9c0 msgr2=0x7f4c7015bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:38.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.692+0000 7f4c76c00640 1 -- 172.21.15.90:0/2502777680 shutdown_connections 2024-03-24T01:38:38.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.693+0000 7f4c76c00640 1 -- 172.21.15.90:0/2502777680 wait complete. 2024-03-24T01:38:38.710 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:38:38.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.752+0000 7f837dc00640 1 -- 172.21.15.90:0/3246537328 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8378068ad0 con 0x7f8378156bb0 2024-03-24T01:38:38.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.753+0000 7f836ea00640 1 -- 172.21.15.90:0/3246537328 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v69) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8378068ad0 con 0x7f8378156bb0 2024-03-24T01:38:38.753 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:38:38.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.755+0000 7f837dc00640 1 -- 172.21.15.90:0/3246537328 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f835406bc00 msgr2=0x7f835406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:38.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.755+0000 7f837dc00640 1 --2- 172.21.15.90:0/3246537328 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f835406bc00 0x7f835406e0c0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f8368001850 tx=0x7f8368009040 comp rx=0 tx=0).stop 2024-03-24T01:38:38.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.755+0000 7f837dc00640 1 -- 172.21.15.90:0/3246537328 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8378156bb0 msgr2=0x7f8378157030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:38.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.755+0000 7f837dc00640 1 --2- 172.21.15.90:0/3246537328 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8378156bb0 0x7f8378157030 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f8370009f90 tx=0x7f8370009750 comp rx=0 tx=0).stop 2024-03-24T01:38:38.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.755+0000 7f837dc00640 1 -- 172.21.15.90:0/3246537328 shutdown_connections 2024-03-24T01:38:38.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.755+0000 7f837dc00640 1 --2- 172.21.15.90:0/3246537328 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f835406bc00 0x7f835406e0c0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:38.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.755+0000 7f837dc00640 1 --2- 172.21.15.90:0/3246537328 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8378156bb0 0x7f8378157030 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:38.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.755+0000 7f837dc00640 1 --2- 172.21.15.90:0/3246537328 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f83780ae0c0 0x7f837815cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:38.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.755+0000 7f837dc00640 1 --2- 172.21.15.90:0/3246537328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f835c004d40 0x7f8378154460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:38.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.755+0000 7f837dc00640 1 -- 172.21.15.90:0/3246537328 >> 172.21.15.90:0/3246537328 conn(0x7f83780b1770 msgr2=0x7f83781563b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:38.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.755+0000 7f837dc00640 1 -- 172.21.15.90:0/3246537328 shutdown_connections 2024-03-24T01:38:38.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.755+0000 7f837dc00640 1 -- 172.21.15.90:0/3246537328 wait complete. 2024-03-24T01:38:38.773 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T01:38:38.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.906+0000 7f72c0800640 1 Processor -- start 2024-03-24T01:38:38.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.906+0000 7f72c0800640 1 -- start start 2024-03-24T01:38:38.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.908+0000 7f72c0800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72b806b7d0 0x7f72b8166a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:38.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.908+0000 7f72c0800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72b80cd490 con 0x7f72b81603e0 2024-03-24T01:38:38.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.908+0000 7f72c0800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72b815f900 con 0x7f72b80cc150 2024-03-24T01:38:38.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.908+0000 7f72c0800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72b80cd270 con 0x7f72b806b7d0 2024-03-24T01:38:38.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.908+0000 7f72bd400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f72b81603e0 0x7f72b806b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:37144/0 (socket says 172.21.15.90:37144) 2024-03-24T01:38:38.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.908+0000 7f72bd400640 1 -- 172.21.15.90:0/3780605029 learned_addr learned my addr 172.21.15.90:0/3780605029 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:38:38.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.908+0000 7f72be800640 1 --2- 172.21.15.90:0/3780605029 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72b806b7d0 0x7f72b8166a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:38.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.909+0000 7f72bca00640 1 -- 172.21.15.90:0/3780605029 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3653983558 0 0) 0x7f72b80cd490 con 0x7f72b81603e0 2024-03-24T01:38:38.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.909+0000 7f72bca00640 1 -- 172.21.15.90:0/3780605029 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72a0003520 con 0x7f72b81603e0 2024-03-24T01:38:38.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.909+0000 7f72be800640 1 -- 172.21.15.90:0/3780605029 >> v1:172.21.15.120:6789/0 conn(0x7f72b80cc150 legacy=0x7f72b80ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:38.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.909+0000 7f72be800640 1 -- 172.21.15.90:0/3780605029 >> v1:172.21.15.90:6789/0 conn(0x7f72b81603e0 legacy=0x7f72b806b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:38.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.909+0000 7f72bca00640 1 -- 172.21.15.90:0/3780605029 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1322006283 0 0) 0x7f72a0003520 con 0x7f72b81603e0 2024-03-24T01:38:38.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.909+0000 7f72be800640 1 -- 172.21.15.90:0/3780605029 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b815fbe0 con 0x7f72b806b7d0 2024-03-24T01:38:38.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.910+0000 7f72be800640 1 --2- 172.21.15.90:0/3780605029 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72b806b7d0 0x7f72b8166a80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f72b4009dc0 tx=0x7f72b402bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b89a1cfab9639c1a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:38.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.910+0000 7f72bca00640 1 -- 172.21.15.90:0/3780605029 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b4039070 con 0x7f72b806b7d0 2024-03-24T01:38:38.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.911+0000 7f72bca00640 1 -- 172.21.15.90:0/3780605029 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72b402c910 con 0x7f72b806b7d0 2024-03-24T01:38:38.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.911+0000 7f72bca00640 1 -- 172.21.15.90:0/3780605029 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a8001670 con 0x7f72b806b7d0 2024-03-24T01:38:38.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.911+0000 7f72c0800640 1 -- 172.21.15.90:0/3780605029 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72b806b7d0 msgr2=0x7f72b8166a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:38.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.911+0000 7f72c0800640 1 --2- 172.21.15.90:0/3780605029 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72b806b7d0 0x7f72b8166a80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f72b4009dc0 tx=0x7f72b402bde0 comp rx=0 tx=0).stop 2024-03-24T01:38:38.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.911+0000 7f72c0800640 1 -- 172.21.15.90:0/3780605029 shutdown_connections 2024-03-24T01:38:38.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.911+0000 7f72c0800640 1 --2- 172.21.15.90:0/3780605029 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72b806b7d0 0x7f72b8166a80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:38.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.911+0000 7f72c0800640 1 -- 172.21.15.90:0/3780605029 >> 172.21.15.90:0/3780605029 conn(0x7f72b80b1770 msgr2=0x7f72b80b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:38.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.911+0000 7f72c0800640 1 -- 172.21.15.90:0/3780605029 shutdown_connections 2024-03-24T01:38:38.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.911+0000 7f72c0800640 1 -- 172.21.15.90:0/3780605029 wait complete. 2024-03-24T01:38:38.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.913+0000 7f72c0800640 1 Processor -- start 2024-03-24T01:38:38.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.913+0000 7f72c0800640 1 -- start start 2024-03-24T01:38:38.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.914+0000 7f72c0800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72b806b7d0 0x7f72b8144400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:38.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.914+0000 7f72c0800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72b80cc150 0x7f72b80cf7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:38.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.914+0000 7f72c0800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72b81603e0 0x7f72b80cfd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:38.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.914+0000 7f72c0800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b81455b0 con 0x7f72b806b7d0 2024-03-24T01:38:38.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.914+0000 7f72c0800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b8145430 con 0x7f72b80cc150 2024-03-24T01:38:38.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.914+0000 7f72c0800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72b8166fc0 con 0x7f72b81603e0 2024-03-24T01:38:38.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.914+0000 7f72bde00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72b806b7d0 0x7f72b8144400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:38.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.915+0000 7f72bde00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72b806b7d0 0x7f72b8144400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38422/0 (socket says 172.21.15.90:38422) 2024-03-24T01:38:38.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.915+0000 7f72bde00640 1 -- 172.21.15.90:0/1966697638 learned_addr learned my addr 172.21.15.90:0/1966697638 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:38:38.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.915+0000 7f72bd400640 1 --2- 172.21.15.90:0/1966697638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72b80cc150 0x7f72b80cf7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:38.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.915+0000 7f72be800640 1 --2- 172.21.15.90:0/1966697638 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72b81603e0 0x7f72b80cfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:38.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.915+0000 7f72bde00640 1 -- 172.21.15.90:0/1966697638 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72b81603e0 msgr2=0x7f72b80cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:38.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.915+0000 7f72bde00640 1 --2- 172.21.15.90:0/1966697638 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72b81603e0 0x7f72b80cfd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:38.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.915+0000 7f72bde00640 1 -- 172.21.15.90:0/1966697638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72b80cc150 msgr2=0x7f72b80cf7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:38.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.915+0000 7f72bde00640 1 --2- 172.21.15.90:0/1966697638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72b80cc150 0x7f72b80cf7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:38.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.915+0000 7f72bde00640 1 -- 172.21.15.90:0/1966697638 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b80acd70 con 0x7f72b806b7d0 2024-03-24T01:38:38.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.915+0000 7f72bd400640 1 --2- 172.21.15.90:0/1966697638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72b80cc150 0x7f72b80cf7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T01:38:38.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.916+0000 7f72bde00640 1 --2- 172.21.15.90:0/1966697638 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72b806b7d0 0x7f72b8144400 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f72ac00c480 tx=0x7f72ac00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:38.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.916+0000 7f72b2000640 1 -- 172.21.15.90:0/1966697638 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ac0052d0 con 0x7f72b806b7d0 2024-03-24T01:38:38.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.916+0000 7f72b2000640 1 -- 172.21.15.90:0/1966697638 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72ac005470 con 0x7f72b806b7d0 2024-03-24T01:38:38.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.916+0000 7f72c0800640 1 -- 172.21.15.90:0/1966697638 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72b80ad050 con 0x7f72b806b7d0 2024-03-24T01:38:38.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.916+0000 7f72b2000640 1 -- 172.21.15.90:0/1966697638 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ac012870 con 0x7f72b806b7d0 2024-03-24T01:38:38.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.916+0000 7f72c0800640 1 -- 172.21.15.90:0/1966697638 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72b80ad610 con 0x7f72b806b7d0 2024-03-24T01:38:38.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.917+0000 7f72c0800640 1 -- 172.21.15.90:0/1966697638 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72b8068060 con 0x7f72b806b7d0 2024-03-24T01:38:38.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.918+0000 7f72b2000640 1 -- 172.21.15.90:0/1966697638 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f72ac012a10 con 0x7f72b806b7d0 2024-03-24T01:38:38.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.919+0000 7f72b2000640 1 --2- 172.21.15.90:0/1966697638 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f729c06bc00 0x7f729c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:38.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.919+0000 7f72b2000640 1 -- 172.21.15.90:0/1966697638 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7f72ac08a890 con 0x7f72b806b7d0 2024-03-24T01:38:38.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.920+0000 7f72bd400640 1 --2- 172.21.15.90:0/1966697638 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f729c06bc00 0x7f729c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:38.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.921+0000 7f72bd400640 1 --2- 172.21.15.90:0/1966697638 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f729c06bc00 0x7f729c06e0c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f72b813fd50 tx=0x7f72a8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:38.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:38.924+0000 7f72b2000640 1 -- 172.21.15.90:0/1966697638 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f72ac0591f0 con 0x7f72b806b7d0 2024-03-24T01:38:39.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.012+0000 7f71dd200640 1 Processor -- start 2024-03-24T01:38:39.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.013+0000 7f71dd200640 1 -- start start 2024-03-24T01:38:39.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.014+0000 7f71dd200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71d80aeeb0 0x7f71d80b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:39.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.014+0000 7f71dd200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f71d815bc50 0x7f71d806b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:39.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.014+0000 7f71dd200640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71d806dac0 con 0x7f71d806b600 2024-03-24T01:38:39.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.014+0000 7f71dd200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71d806ba70 con 0x7f71d815bc50 2024-03-24T01:38:39.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.014+0000 7f71dd200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71d80b4da0 con 0x7f71d80aeeb0 2024-03-24T01:38:39.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.015+0000 7f71d5400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f71d815bc50 0x7f71d806b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:39.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.015+0000 7f71d6800640 1 --1- >> v1:172.21.15.120:6789/0 conn(0x7f71d806b600 0x7f71d815e590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.120:6789/0 says I am v1:172.21.15.90:56182/0 (socket says 172.21.15.90:56182) 2024-03-24T01:38:39.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.015+0000 7f71d6800640 1 -- 172.21.15.90:0/1166217553 learned_addr learned my addr 172.21.15.90:0/1166217553 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:38:39.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.015+0000 7f71d5e00640 1 --2- 172.21.15.90:0/1166217553 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71d80aeeb0 0x7f71d80b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:39.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.015+0000 7f71d5e00640 1 -- 172.21.15.90:0/1166217553 >> v2:172.21.15.120:3300/0 conn(0x7f71d815bc50 msgr2=0x7f71d806b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:38:39.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.015+0000 7f71d5e00640 1 --2- 172.21.15.90:0/1166217553 >> v2:172.21.15.120:3300/0 conn(0x7f71d815bc50 0x7f71d806b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.015+0000 7f71d5e00640 1 -- 172.21.15.90:0/1166217553 >> v1:172.21.15.120:6789/0 conn(0x7f71d806b600 legacy=0x7f71d815e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:39.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.015+0000 7f71d5e00640 1 -- 172.21.15.90:0/1166217553 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d806dda0 con 0x7f71d80aeeb0 2024-03-24T01:38:39.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.016+0000 7f71d5e00640 1 --2- 172.21.15.90:0/1166217553 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71d80aeeb0 0x7f71d80b44c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f71c80097a0 tx=0x7f71c802b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbec1980a34f303e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:39.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.017+0000 7f71d4a00640 1 -- 172.21.15.90:0/1166217553 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c8038070 con 0x7f71d80aeeb0 2024-03-24T01:38:39.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.017+0000 7f71d4a00640 1 -- 172.21.15.90:0/1166217553 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71c8031700 con 0x7f71d80aeeb0 2024-03-24T01:38:39.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.017+0000 7f71d4a00640 1 -- 172.21.15.90:0/1166217553 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c8031ad0 con 0x7f71d80aeeb0 2024-03-24T01:38:39.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.018+0000 7f71dd200640 1 -- 172.21.15.90:0/1166217553 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71d80aeeb0 msgr2=0x7f71d80b44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:39.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.018+0000 7f71dd200640 1 --2- 172.21.15.90:0/1166217553 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71d80aeeb0 0x7f71d80b44c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f71c80097a0 tx=0x7f71c802b7a0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.018+0000 7f71dd200640 1 -- 172.21.15.90:0/1166217553 shutdown_connections 2024-03-24T01:38:39.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.018+0000 7f71dd200640 1 --2- 172.21.15.90:0/1166217553 >> v2:172.21.15.120:3300/0 conn(0x7f71d815bc50 0x7f71d806b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.018+0000 7f71dd200640 1 --2- 172.21.15.90:0/1166217553 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71d80aeeb0 0x7f71d80b44c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.018+0000 7f71dd200640 1 -- 172.21.15.90:0/1166217553 >> 172.21.15.90:0/1166217553 conn(0x7f71d80b1770 msgr2=0x7f71d80b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:39.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.018+0000 7f71dd200640 1 -- 172.21.15.90:0/1166217553 shutdown_connections 2024-03-24T01:38:39.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.018+0000 7f71dd200640 1 -- 172.21.15.90:0/1166217553 wait complete. 2024-03-24T01:38:39.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.020+0000 7f71dd200640 1 Processor -- start 2024-03-24T01:38:39.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.020+0000 7f71dd200640 1 -- start start 2024-03-24T01:38:39.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.021+0000 7f71dd200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71d806b600 0x7f71d81f31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:39.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.021+0000 7f71dd200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71d80aeeb0 0x7f71d80ce3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:39.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.021+0000 7f71dd200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71d815bc50 0x7f71d81e5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:39.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.021+0000 7f71dd200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d80af290 con 0x7f71d80aeeb0 2024-03-24T01:38:39.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.021+0000 7f71dd200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d806e360 con 0x7f71d806b600 2024-03-24T01:38:39.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.021+0000 7f71dd200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71d80b4a00 con 0x7f71d815bc50 2024-03-24T01:38:39.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.021+0000 7f71d5400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71d80aeeb0 0x7f71d80ce3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:39.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.021+0000 7f71d5400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71d80aeeb0 0x7f71d80ce3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38438/0 (socket says 172.21.15.90:38438) 2024-03-24T01:38:39.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.021+0000 7f71d5400640 1 -- 172.21.15.90:0/1300768790 learned_addr learned my addr 172.21.15.90:0/1300768790 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:38:39.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.021+0000 7f71d5e00640 1 --2- 172.21.15.90:0/1300768790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71d806b600 0x7f71d81f31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:39.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.021+0000 7f71d6800640 1 --2- 172.21.15.90:0/1300768790 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71d815bc50 0x7f71d81e5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:39.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.022+0000 7f71d5400640 1 -- 172.21.15.90:0/1300768790 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71d815bc50 msgr2=0x7f71d81e5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:39.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.022+0000 7f71d5400640 1 --2- 172.21.15.90:0/1300768790 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71d815bc50 0x7f71d81e5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.022+0000 7f71d5400640 1 -- 172.21.15.90:0/1300768790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71d806b600 msgr2=0x7f71d81f31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:39.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.022+0000 7f71d5400640 1 --2- 172.21.15.90:0/1300768790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71d806b600 0x7f71d81f31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.022+0000 7f71d5400640 1 -- 172.21.15.90:0/1300768790 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d81f3700 con 0x7f71d80aeeb0 2024-03-24T01:38:39.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.022+0000 7f71d5400640 1 --2- 172.21.15.90:0/1300768790 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71d80aeeb0 0x7f71d80ce3b0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f71c400e8a0 tx=0x7f71c400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:39.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.022+0000 7f71ce000640 1 -- 172.21.15.90:0/1300768790 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c4017070 con 0x7f71d80aeeb0 2024-03-24T01:38:39.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.022+0000 7f71ce000640 1 -- 172.21.15.90:0/1300768790 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71c400c940 con 0x7f71d80aeeb0 2024-03-24T01:38:39.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.022+0000 7f71dd200640 1 -- 172.21.15.90:0/1300768790 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71d80cee00 con 0x7f71d80aeeb0 2024-03-24T01:38:39.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.022+0000 7f71ce000640 1 -- 172.21.15.90:0/1300768790 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c400cd30 con 0x7f71d80aeeb0 2024-03-24T01:38:39.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.023+0000 7f71dd200640 1 -- 172.21.15.90:0/1300768790 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71d81e3360 con 0x7f71d80aeeb0 2024-03-24T01:38:39.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.024+0000 7f71dd200640 1 -- 172.21.15.90:0/1300768790 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71d804f400 con 0x7f71d80aeeb0 2024-03-24T01:38:39.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.026+0000 7f71ce000640 1 -- 172.21.15.90:0/1300768790 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f71c401a070 con 0x7f71d80aeeb0 2024-03-24T01:38:39.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.029+0000 7f71ce000640 1 --2- 172.21.15.90:0/1300768790 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f71b806bbd0 0x7f71b806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:39.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.029+0000 7f71ce000640 1 -- 172.21.15.90:0/1300768790 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7f71c408b250 con 0x7f71d80aeeb0 2024-03-24T01:38:39.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.029+0000 7f71d5e00640 1 --2- 172.21.15.90:0/1300768790 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f71b806bbd0 0x7f71b806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:39.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.030+0000 7f71d5e00640 1 --2- 172.21.15.90:0/1300768790 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f71b806bbd0 0x7f71b806e090 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f71c8035000 tx=0x7f71c8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:39.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.031+0000 7f71ce000640 1 -- 172.21.15.90:0/1300768790 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f71c4059540 con 0x7f71d80aeeb0 2024-03-24T01:38:39.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.126+0000 7f72c0800640 1 -- 172.21.15.90:0/1966697638 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72b80b2040 con 0x7f72b806b7d0 2024-03-24T01:38:39.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.128+0000 7f72b2000640 1 -- 172.21.15.90:0/1966697638 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+17101 (secure 0 0 0) 0x7f72ac058af0 con 0x7f72b806b7d0 2024-03-24T01:38:39.130 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:38:39.130 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":69,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:32:54.821325+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:32:42.026147+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-24T01:32:43.180712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.1e","mappings":[{"from":7,"to":2}]},{"pgid":"10.10","mappings":[{"from":4,"to":3}]},{"pgid":"10.1d","mappings":[{"from":4,"to":3}]},{"pgid":"10.1f","mappings":[{"from":7,"to":3}]},{"pgid":"10.37","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:38:39.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.131+0000 7f72c0800640 1 -- 172.21.15.90:0/1966697638 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f729c06bc00 msgr2=0x7f729c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:39.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.131+0000 7f72c0800640 1 --2- 172.21.15.90:0/1966697638 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f729c06bc00 0x7f729c06e0c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f72b813fd50 tx=0x7f72a8009040 comp rx=0 tx=0).stop 2024-03-24T01:38:39.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.131+0000 7f72c0800640 1 -- 172.21.15.90:0/1966697638 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72b806b7d0 msgr2=0x7f72b8144400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:39.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.131+0000 7f72c0800640 1 --2- 172.21.15.90:0/1966697638 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72b806b7d0 0x7f72b8144400 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f72ac00c480 tx=0x7f72ac00c940 comp rx=0 tx=0).stop 2024-03-24T01:38:39.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.131+0000 7f72c0800640 1 -- 172.21.15.90:0/1966697638 shutdown_connections 2024-03-24T01:38:39.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.131+0000 7f72c0800640 1 --2- 172.21.15.90:0/1966697638 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f729c06bc00 0x7f729c06e0c0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.131+0000 7f72c0800640 1 --2- 172.21.15.90:0/1966697638 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f72b81603e0 0x7f72b80cfd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.131+0000 7f72c0800640 1 --2- 172.21.15.90:0/1966697638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72b80cc150 0x7f72b80cf7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.131+0000 7f72c0800640 1 --2- 172.21.15.90:0/1966697638 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72b806b7d0 0x7f72b8144400 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.131+0000 7f72c0800640 1 -- 172.21.15.90:0/1966697638 >> 172.21.15.90:0/1966697638 conn(0x7f72b80b1770 msgr2=0x7f72b80b0080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:39.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.131+0000 7f72c0800640 1 -- 172.21.15.90:0/1966697638 shutdown_connections 2024-03-24T01:38:39.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.131+0000 7f72c0800640 1 -- 172.21.15.90:0/1966697638 wait complete. 2024-03-24T01:38:39.151 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T01:23:59.548370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T01:32:42.026147+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T01:32:43.180712+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T01:38:39.151 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T01:38:39.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.234+0000 7f71dd200640 1 -- 172.21.15.90:0/1300768790 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f71d80cf320 con 0x7f71d80aeeb0 2024-03-24T01:38:39.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.235+0000 7f71ce000640 1 -- 172.21.15.90:0/1300768790 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v69) v1 ==== 100+0+11 (secure 0 0 0) 0x7f71c405d5e0 con 0x7f71d80aeeb0 2024-03-24T01:38:39.235 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-24T01:38:39.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.237+0000 7f71dd200640 1 -- 172.21.15.90:0/1300768790 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f71b806bbd0 msgr2=0x7f71b806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:39.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.237+0000 7f71dd200640 1 --2- 172.21.15.90:0/1300768790 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f71b806bbd0 0x7f71b806e090 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f71c8035000 tx=0x7f71c8008030 comp rx=0 tx=0).stop 2024-03-24T01:38:39.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.237+0000 7f71dd200640 1 -- 172.21.15.90:0/1300768790 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71d80aeeb0 msgr2=0x7f71d80ce3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:39.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.237+0000 7f71dd200640 1 --2- 172.21.15.90:0/1300768790 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71d80aeeb0 0x7f71d80ce3b0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f71c400e8a0 tx=0x7f71c400ed60 comp rx=0 tx=0).stop 2024-03-24T01:38:39.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.237+0000 7f71dd200640 1 -- 172.21.15.90:0/1300768790 shutdown_connections 2024-03-24T01:38:39.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.237+0000 7f71dd200640 1 --2- 172.21.15.90:0/1300768790 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f71b806bbd0 0x7f71b806e090 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.237+0000 7f71dd200640 1 --2- 172.21.15.90:0/1300768790 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f71d80aeeb0 0x7f71d80ce3b0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.237+0000 7f71dd200640 1 --2- 172.21.15.90:0/1300768790 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f71d815bc50 0x7f71d81e5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.237+0000 7f71dd200640 1 --2- 172.21.15.90:0/1300768790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71d806b600 0x7f71d81f31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.237+0000 7f71dd200640 1 -- 172.21.15.90:0/1300768790 >> 172.21.15.90:0/1300768790 conn(0x7f71d80b1770 msgr2=0x7f71d80ade30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:39.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.237+0000 7f71dd200640 1 -- 172.21.15.90:0/1300768790 shutdown_connections 2024-03-24T01:38:39.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.238+0000 7f71dd200640 1 -- 172.21.15.90:0/1300768790 wait complete. 2024-03-24T01:38:39.254 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T01:38:39.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.348+0000 7f6339800640 1 Processor -- start 2024-03-24T01:38:39.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.349+0000 7f6339800640 1 -- start start 2024-03-24T01:38:39.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.350+0000 7f6339800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f633415bda0 0x7f63340cdb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:39.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.350+0000 7f6339800640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f63340cff70 0x7f63340d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:39.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.350+0000 7f6339800640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f633415c180 con 0x7f63340cc530 2024-03-24T01:38:39.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.350+0000 7f6339800640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63340cd6d0 con 0x7f63340cff70 2024-03-24T01:38:39.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.350+0000 7f6339800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63340cf7f0 con 0x7f633415bda0 2024-03-24T01:38:39.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.351+0000 7f6333400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f63340cff70 0x7f63340d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:39.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.351+0000 7f6332a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f633415bda0 0x7f63340cdb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:39.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.351+0000 7f6333400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f63340cff70 0x7f63340d0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.90:39426/0 (socket says 172.21.15.90:39426) 2024-03-24T01:38:39.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.351+0000 7f6332a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f633415bda0 0x7f63340cdb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.90:56216/0 (socket says 172.21.15.90:56216) 2024-03-24T01:38:39.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.351+0000 7f6333400640 1 -- 172.21.15.90:0/865515810 learned_addr learned my addr 172.21.15.90:0/865515810 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:38:39.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.351+0000 7f6332a00640 1 -- 172.21.15.90:0/865515810 >> v2:172.21.15.120:3300/0 conn(0x7f63340cff70 msgr2=0x7f63340d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:38:39.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.351+0000 7f6332a00640 1 --2- 172.21.15.90:0/865515810 >> v2:172.21.15.120:3300/0 conn(0x7f63340cff70 0x7f63340d0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.351+0000 7f6332a00640 1 -- 172.21.15.90:0/865515810 >> v1:172.21.15.120:6789/0 conn(0x7f63340cc530 legacy=0x7f6334160280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:39.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.351+0000 7f6332a00640 1 -- 172.21.15.90:0/865515810 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63340ccac0 con 0x7f633415bda0 2024-03-24T01:38:39.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.352+0000 7f6332a00640 1 --2- 172.21.15.90:0/865515810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f633415bda0 0x7f63340cdb60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f63240097a0 tx=0x7f632402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=539c4156209b4138 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:39.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.353+0000 7f6331600640 1 -- 172.21.15.90:0/865515810 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63240315a0 con 0x7f633415bda0 2024-03-24T01:38:39.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.353+0000 7f6331600640 1 -- 172.21.15.90:0/865515810 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6324037070 con 0x7f633415bda0 2024-03-24T01:38:39.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.353+0000 7f6331600640 1 -- 172.21.15.90:0/865515810 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6324031960 con 0x7f633415bda0 2024-03-24T01:38:39.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.353+0000 7f6339800640 1 -- 172.21.15.90:0/865515810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f633415bda0 msgr2=0x7f63340cdb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:39.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.353+0000 7f6339800640 1 --2- 172.21.15.90:0/865515810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f633415bda0 0x7f63340cdb60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f63240097a0 tx=0x7f632402b840 comp rx=0 tx=0).stop 2024-03-24T01:38:39.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.354+0000 7f6339800640 1 -- 172.21.15.90:0/865515810 shutdown_connections 2024-03-24T01:38:39.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.354+0000 7f6339800640 1 --2- 172.21.15.90:0/865515810 >> v2:172.21.15.120:3300/0 conn(0x7f63340cff70 0x7f63340d0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.354+0000 7f6339800640 1 --2- 172.21.15.90:0/865515810 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f633415bda0 0x7f63340cdb60 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.354+0000 7f6339800640 1 -- 172.21.15.90:0/865515810 >> 172.21.15.90:0/865515810 conn(0x7f63340b3480 msgr2=0x7f63340b3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:39.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.354+0000 7f6339800640 1 -- 172.21.15.90:0/865515810 shutdown_connections 2024-03-24T01:38:39.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.354+0000 7f6339800640 1 -- 172.21.15.90:0/865515810 wait complete. 2024-03-24T01:38:39.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.356+0000 7f6339800640 1 Processor -- start 2024-03-24T01:38:39.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.356+0000 7f6339800640 1 -- start start 2024-03-24T01:38:39.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.357+0000 7f6339800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63340cc530 0x7f63340c4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:39.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.357+0000 7f6339800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63340cff70 0x7f6334158b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:39.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.357+0000 7f6339800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f633415bda0 0x7f633415c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:39.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.357+0000 7f6339800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63340cd360 con 0x7f63340cc530 2024-03-24T01:38:39.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.357+0000 7f6339800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63340cd080 con 0x7f633415bda0 2024-03-24T01:38:39.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.357+0000 7f6339800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63340c54e0 con 0x7f63340cff70 2024-03-24T01:38:39.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.357+0000 7f6332a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63340cc530 0x7f63340c4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:39.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.357+0000 7f6332a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63340cc530 0x7f63340c4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38448/0 (socket says 172.21.15.90:38448) 2024-03-24T01:38:39.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.357+0000 7f6332a00640 1 -- 172.21.15.90:0/4179675841 learned_addr learned my addr 172.21.15.90:0/4179675841 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:38:39.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.358+0000 7f6333400640 1 --2- 172.21.15.90:0/4179675841 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f633415bda0 0x7f633415c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:39.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.358+0000 7f6332000640 1 --2- 172.21.15.90:0/4179675841 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63340cff70 0x7f6334158b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:39.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.358+0000 7f6332a00640 1 -- 172.21.15.90:0/4179675841 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63340cff70 msgr2=0x7f6334158b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:39.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.358+0000 7f6332a00640 1 --2- 172.21.15.90:0/4179675841 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63340cff70 0x7f6334158b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.358+0000 7f6332a00640 1 -- 172.21.15.90:0/4179675841 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f633415bda0 msgr2=0x7f633415c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:39.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.358+0000 7f6332a00640 1 --2- 172.21.15.90:0/4179675841 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f633415bda0 0x7f633415c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.358+0000 7f6332a00640 1 -- 172.21.15.90:0/4179675841 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f633415c8b0 con 0x7f63340cc530 2024-03-24T01:38:39.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.358+0000 7f6332a00640 1 --2- 172.21.15.90:0/4179675841 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63340cc530 0x7f63340c4600 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f6324034040 tx=0x7f63240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:39.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.359+0000 7f632aa00640 1 -- 172.21.15.90:0/4179675841 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6324049070 con 0x7f63340cc530 2024-03-24T01:38:39.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.359+0000 7f632aa00640 1 -- 172.21.15.90:0/4179675841 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6324032430 con 0x7f63340cc530 2024-03-24T01:38:39.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.359+0000 7f632aa00640 1 -- 172.21.15.90:0/4179675841 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6324032800 con 0x7f63340cc530 2024-03-24T01:38:39.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.359+0000 7f6339800640 1 -- 172.21.15.90:0/4179675841 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f633415cb30 con 0x7f63340cc530 2024-03-24T01:38:39.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.359+0000 7f6339800640 1 -- 172.21.15.90:0/4179675841 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f633415d0f0 con 0x7f63340cc530 2024-03-24T01:38:39.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.360+0000 7f6339800640 1 -- 172.21.15.90:0/4179675841 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f633404f400 con 0x7f63340cc530 2024-03-24T01:38:39.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.361+0000 7f632aa00640 1 -- 172.21.15.90:0/4179675841 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f63240329a0 con 0x7f63340cc530 2024-03-24T01:38:39.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.362+0000 7f632aa00640 1 --2- 172.21.15.90:0/4179675841 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f631006bbd0 0x7f631006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:39.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.362+0000 7f632aa00640 1 -- 172.21.15.90:0/4179675841 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7f632402f020 con 0x7f63340cc530 2024-03-24T01:38:39.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.362+0000 7f6332000640 1 --2- 172.21.15.90:0/4179675841 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f631006bbd0 0x7f631006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:39.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.364+0000 7f6332000640 1 --2- 172.21.15.90:0/4179675841 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f631006bbd0 0x7f631006e090 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f633415a830 tx=0x7f631c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:39.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.367+0000 7f632aa00640 1 -- 172.21.15.90:0/4179675841 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f633415c8b0 con 0x7f63340cc530 2024-03-24T01:38:39.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.443+0000 7fb971800640 1 Processor -- start 2024-03-24T01:38:39.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.443+0000 7fb971800640 1 -- start start 2024-03-24T01:38:39.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.444+0000 7fb971800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb96c0720a0 0x7fb96c0ceaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:39.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.444+0000 7fb971800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb96c0cf030 0x7fb96c0b3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:39.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.444+0000 7fb971800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb96c0b2890 con 0x7fb96c1452b0 2024-03-24T01:38:39.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.444+0000 7fb971800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb96c072990 con 0x7fb96c0cf030 2024-03-24T01:38:39.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.444+0000 7fb971800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb96c0b2b70 con 0x7fb96c0720a0 2024-03-24T01:38:39.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.445+0000 7fb96b400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb96c0cf030 0x7fb96c0b3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:39.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.445+0000 7fb96aa00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fb96c1452b0 0x7fb96c071990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:37146/0 (socket says 172.21.15.90:37146) 2024-03-24T01:38:39.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.445+0000 7fb96aa00640 1 -- 172.21.15.90:0/4061722731 learned_addr learned my addr 172.21.15.90:0/4061722731 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:38:39.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.445+0000 7fb96b400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb96c0cf030 0x7fb96c0b3930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38456/0 (socket says 172.21.15.90:38456) 2024-03-24T01:38:39.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.445+0000 7fb96a000640 1 --2- 172.21.15.90:0/4061722731 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb96c0720a0 0x7fb96c0ceaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:39.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.445+0000 7fb96b400640 1 -- 172.21.15.90:0/4061722731 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb96c0720a0 msgr2=0x7fb96c0ceaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:38:39.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.445+0000 7fb96b400640 1 --2- 172.21.15.90:0/4061722731 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb96c0720a0 0x7fb96c0ceaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.445+0000 7fb96b400640 1 -- 172.21.15.90:0/4061722731 >> v1:172.21.15.90:6789/0 conn(0x7fb96c1452b0 legacy=0x7fb96c071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:39.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.445+0000 7fb96b400640 1 -- 172.21.15.90:0/4061722731 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb96c0b2e50 con 0x7fb96c0cf030 2024-03-24T01:38:39.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.445+0000 7fb969600640 1 -- 172.21.15.90:0/4061722731 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3299273166 0 0) 0x7fb96c0b2890 con 0x7fb96c1452b0 2024-03-24T01:38:39.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.446+0000 7fb96b400640 1 --2- 172.21.15.90:0/4061722731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb96c0cf030 0x7fb96c0b3930 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fb964009d90 tx=0x7fb96402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75e5e446b849683 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:39.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.447+0000 7fb969600640 1 -- 172.21.15.90:0/4061722731 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb964039070 con 0x7fb96c0cf030 2024-03-24T01:38:39.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.447+0000 7fb969600640 1 -- 172.21.15.90:0/4061722731 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb96402cc50 con 0x7fb96c0cf030 2024-03-24T01:38:39.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.447+0000 7fb969600640 1 -- 172.21.15.90:0/4061722731 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb964034440 con 0x7fb96c0cf030 2024-03-24T01:38:39.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.448+0000 7fb971800640 1 -- 172.21.15.90:0/4061722731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb96c0cf030 msgr2=0x7fb96c0b3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:39.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.448+0000 7fb971800640 1 --2- 172.21.15.90:0/4061722731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb96c0cf030 0x7fb96c0b3930 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fb964009d90 tx=0x7fb96402bea0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.448+0000 7fb971800640 1 -- 172.21.15.90:0/4061722731 shutdown_connections 2024-03-24T01:38:39.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.448+0000 7fb971800640 1 --2- 172.21.15.90:0/4061722731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb96c0cf030 0x7fb96c0b3930 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.448+0000 7fb971800640 1 --2- 172.21.15.90:0/4061722731 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb96c0720a0 0x7fb96c0ceaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.448+0000 7fb971800640 1 -- 172.21.15.90:0/4061722731 >> 172.21.15.90:0/4061722731 conn(0x7fb96c0cf7e0 msgr2=0x7fb96c0cfbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:39.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.448+0000 7fb971800640 1 -- 172.21.15.90:0/4061722731 shutdown_connections 2024-03-24T01:38:39.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.448+0000 7fb971800640 1 -- 172.21.15.90:0/4061722731 wait complete. 2024-03-24T01:38:39.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.449+0000 7fb971800640 1 Processor -- start 2024-03-24T01:38:39.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.450+0000 7fb971800640 1 -- start start 2024-03-24T01:38:39.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.450+0000 7fb971800640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb96c0720a0 0x7fb96c13dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:39.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.450+0000 7fb971800640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb96c0cf030 0x7fb96c139280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:39.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.450+0000 7fb971800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb96c1452b0 0x7fb96c13f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:39.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.451+0000 7fb971800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb96c0cf4c0 con 0x7fb96c1452b0 2024-03-24T01:38:39.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.451+0000 7fb971800640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb96c0b3410 con 0x7fb96c0720a0 2024-03-24T01:38:39.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.451+0000 7fb971800640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb96c145690 con 0x7fb96c0cf030 2024-03-24T01:38:39.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.451+0000 7fb96b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb96c1452b0 0x7fb96c13f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:39.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.451+0000 7fb96b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb96c1452b0 0x7fb96c13f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38464/0 (socket says 172.21.15.90:38464) 2024-03-24T01:38:39.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.451+0000 7fb96b400640 1 -- 172.21.15.90:0/3821023452 learned_addr learned my addr 172.21.15.90:0/3821023452 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:38:39.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.451+0000 7fb96aa00640 1 --2- 172.21.15.90:0/3821023452 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb96c0720a0 0x7fb96c13dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:39.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.451+0000 7fb96a000640 1 --2- 172.21.15.90:0/3821023452 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb96c0cf030 0x7fb96c139280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:39.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.451+0000 7fb96b400640 1 -- 172.21.15.90:0/3821023452 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb96c0cf030 msgr2=0x7fb96c139280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:39.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.451+0000 7fb96b400640 1 --2- 172.21.15.90:0/3821023452 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb96c0cf030 0x7fb96c139280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.451+0000 7fb96b400640 1 -- 172.21.15.90:0/3821023452 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb96c0720a0 msgr2=0x7fb96c13dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:39.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.451+0000 7fb96b400640 1 --2- 172.21.15.90:0/3821023452 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb96c0720a0 0x7fb96c13dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.451+0000 7fb96b400640 1 -- 172.21.15.90:0/3821023452 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb96c1397c0 con 0x7fb96c1452b0 2024-03-24T01:38:39.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.452+0000 7fb96b400640 1 --2- 172.21.15.90:0/3821023452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb96c1452b0 0x7fb96c13f4e0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fb964037040 tx=0x7fb964004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:39.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.452+0000 7fb962a00640 1 -- 172.21.15.90:0/3821023452 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb964044030 con 0x7fb96c1452b0 2024-03-24T01:38:39.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.452+0000 7fb962a00640 1 -- 172.21.15.90:0/3821023452 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb96402ca50 con 0x7fb96c1452b0 2024-03-24T01:38:39.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.452+0000 7fb962a00640 1 -- 172.21.15.90:0/3821023452 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb964039040 con 0x7fb96c1452b0 2024-03-24T01:38:39.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.452+0000 7fb971800640 1 -- 172.21.15.90:0/3821023452 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb96c13fd30 con 0x7fb96c1452b0 2024-03-24T01:38:39.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.452+0000 7fb971800640 1 -- 172.21.15.90:0/3821023452 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb96c1402f0 con 0x7fb96c1452b0 2024-03-24T01:38:39.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.453+0000 7fb971800640 1 -- 172.21.15.90:0/3821023452 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb96c04f400 con 0x7fb96c1452b0 2024-03-24T01:38:39.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.454+0000 7fb962a00640 1 -- 172.21.15.90:0/3821023452 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fb964047070 con 0x7fb96c1452b0 2024-03-24T01:38:39.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.455+0000 7fb962a00640 1 --2- 172.21.15.90:0/3821023452 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb94c06bbd0 0x7fb94c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:39.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.455+0000 7fb962a00640 1 -- 172.21.15.90:0/3821023452 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7fb9640ad2d0 con 0x7fb96c1452b0 2024-03-24T01:38:39.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.455+0000 7fb96aa00640 1 --2- 172.21.15.90:0/3821023452 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb94c06bbd0 0x7fb94c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:39.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.456+0000 7fb96aa00640 1 --2- 172.21.15.90:0/3821023452 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb94c06bbd0 0x7fb94c06e090 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fb95c000f60 tx=0x7fb95c007c00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:39.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.465+0000 7fb962a00640 1 -- 172.21.15.90:0/3821023452 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fb96404e070 con 0x7fb96c1452b0 2024-03-24T01:38:39.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.571+0000 7f6339800640 1 -- 172.21.15.90:0/4179675841 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f63340b4370 con 0x7f63340cc530 2024-03-24T01:38:39.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.571+0000 7f632aa00640 1 -- 172.21.15.90:0/4179675841 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v69) v1 ==== 93+0+10 (secure 0 0 0) 0x7f632407a3c0 con 0x7f63340cc530 2024-03-24T01:38:39.572 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-24T01:38:39.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.574+0000 7f6339800640 1 -- 172.21.15.90:0/4179675841 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f631006bbd0 msgr2=0x7f631006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:39.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.574+0000 7f6339800640 1 --2- 172.21.15.90:0/4179675841 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f631006bbd0 0x7f631006e090 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f633415a830 tx=0x7f631c009040 comp rx=0 tx=0).stop 2024-03-24T01:38:39.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.574+0000 7f6339800640 1 -- 172.21.15.90:0/4179675841 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63340cc530 msgr2=0x7f63340c4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:39.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.574+0000 7f6339800640 1 --2- 172.21.15.90:0/4179675841 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63340cc530 0x7f63340c4600 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f6324034040 tx=0x7f63240046c0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.574+0000 7f6339800640 1 -- 172.21.15.90:0/4179675841 shutdown_connections 2024-03-24T01:38:39.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.574+0000 7f6339800640 1 --2- 172.21.15.90:0/4179675841 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f631006bbd0 0x7f631006e090 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.574+0000 7f6339800640 1 --2- 172.21.15.90:0/4179675841 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f633415bda0 0x7f633415c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.574+0000 7f6339800640 1 --2- 172.21.15.90:0/4179675841 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f63340cff70 0x7f6334158b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.574+0000 7f6339800640 1 --2- 172.21.15.90:0/4179675841 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63340cc530 0x7f63340c4600 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.574+0000 7f6339800640 1 -- 172.21.15.90:0/4179675841 >> 172.21.15.90:0/4179675841 conn(0x7f63340b3480 msgr2=0x7f633415b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:39.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.574+0000 7f6339800640 1 -- 172.21.15.90:0/4179675841 shutdown_connections 2024-03-24T01:38:39.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.574+0000 7f6339800640 1 -- 172.21.15.90:0/4179675841 wait complete. 2024-03-24T01:38:39.594 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T01:38:39.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.677+0000 7fb971800640 1 -- 172.21.15.90:0/3821023452 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb96c144890 con 0x7fb96c1452b0 2024-03-24T01:38:39.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.678+0000 7fb962a00640 1 -- 172.21.15.90:0/3821023452 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v69) v1 ==== 80+0+0 (secure 0 0 0) 0x7fb96407bb90 con 0x7fb96c1452b0 2024-03-24T01:38:39.679 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-24T01:38:39.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.681+0000 7fb971800640 1 -- 172.21.15.90:0/3821023452 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb94c06bbd0 msgr2=0x7fb94c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:39.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.681+0000 7fb971800640 1 --2- 172.21.15.90:0/3821023452 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb94c06bbd0 0x7fb94c06e090 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fb95c000f60 tx=0x7fb95c007c00 comp rx=0 tx=0).stop 2024-03-24T01:38:39.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.681+0000 7fb971800640 1 -- 172.21.15.90:0/3821023452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb96c1452b0 msgr2=0x7fb96c13f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:39.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.681+0000 7fb971800640 1 --2- 172.21.15.90:0/3821023452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb96c1452b0 0x7fb96c13f4e0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fb964037040 tx=0x7fb964004a40 comp rx=0 tx=0).stop 2024-03-24T01:38:39.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.681+0000 7fb971800640 1 -- 172.21.15.90:0/3821023452 shutdown_connections 2024-03-24T01:38:39.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.681+0000 7fb971800640 1 --2- 172.21.15.90:0/3821023452 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fb94c06bbd0 0x7fb94c06e090 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.681+0000 7fb971800640 1 --2- 172.21.15.90:0/3821023452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb96c1452b0 0x7fb96c13f4e0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.681+0000 7fb971800640 1 --2- 172.21.15.90:0/3821023452 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fb96c0cf030 0x7fb96c139280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.681+0000 7fb971800640 1 --2- 172.21.15.90:0/3821023452 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb96c0720a0 0x7fb96c13dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.681+0000 7fb971800640 1 -- 172.21.15.90:0/3821023452 >> 172.21.15.90:0/3821023452 conn(0x7fb96c0cf7e0 msgr2=0x7fb96c142cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:39.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.681+0000 7fb971800640 1 -- 172.21.15.90:0/3821023452 shutdown_connections 2024-03-24T01:38:39.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.681+0000 7fb971800640 1 -- 172.21.15.90:0/3821023452 wait complete. 2024-03-24T01:38:39.700 ERROR:tasks.daemonwatchdog.daemon_watchdog:ignoring exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/daemonwatchdog.py", line 61, in bark mount.umount_wait(force=True) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/fuse_mount.py", line 399, in umount_wait self.umount(cleanup=False) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/fuse_mount.py", line 320, in umount if self.is_blocked(): File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/mount.py", line 197, in is_blocked return self.addr in output TypeError: 'in ' requires string as left operand, not NoneType 2024-03-24T01:38:39.701 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:38:39.701 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:38:39.742 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053414,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498340k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249168k,nr_inodes=812292,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498340k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T01:38:39.743 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.2 is not mounted at ubuntu@smithi090.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.2 2024-03-24T01:38:39.743 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-03-24T01:38:39.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:38:39.743 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-24T01:38:39.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.795+0000 7ff97b200640 1 Processor -- start 2024-03-24T01:38:39.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.795+0000 7ff97b200640 1 -- start start 2024-03-24T01:38:39.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.796+0000 7ff97b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff97406cbe0 0x7ff9740cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:39.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.796+0000 7ff97b200640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7ff9740d00a0 0x7ff974099c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:39.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.796+0000 7ff97b200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff97415e4c0 con 0x7ff97415e0e0 2024-03-24T01:38:39.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.796+0000 7ff97b200640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9740b1350 con 0x7ff9740d00a0 2024-03-24T01:38:39.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.796+0000 7ff97b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9740d0530 con 0x7ff97406cbe0 2024-03-24T01:38:39.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.797+0000 7ff973e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff97415e0e0 0x7ff9740b2080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:37162/0 (socket says 172.21.15.90:37162) 2024-03-24T01:38:39.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.797+0000 7ff973e00640 1 -- 172.21.15.90:0/1661343479 learned_addr learned my addr 172.21.15.90:0/1661343479 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:38:39.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.797+0000 7ff979200640 1 --2- 172.21.15.90:0/1661343479 >> v2:172.21.15.120:3300/0 conn(0x7ff9740d00a0 0x7ff974099c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:39.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.797+0000 7ff973400640 1 --2- 172.21.15.90:0/1661343479 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff97406cbe0 0x7ff9740cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:39.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.797+0000 7ff972a00640 1 -- 172.21.15.90:0/1661343479 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2676202154 0 0) 0x7ff97415e4c0 con 0x7ff97415e0e0 2024-03-24T01:38:39.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.797+0000 7ff972a00640 1 -- 172.21.15.90:0/1661343479 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff958003520 con 0x7ff97415e0e0 2024-03-24T01:38:39.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.797+0000 7ff979200640 1 -- 172.21.15.90:0/1661343479 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff97406cbe0 msgr2=0x7ff9740cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:38:39.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.797+0000 7ff979200640 1 --2- 172.21.15.90:0/1661343479 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff97406cbe0 0x7ff9740cfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.797+0000 7ff979200640 1 -- 172.21.15.90:0/1661343479 >> v1:172.21.15.90:6789/0 conn(0x7ff97415e0e0 legacy=0x7ff9740b2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:39.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.797+0000 7ff979200640 1 -- 172.21.15.90:0/1661343479 --> v2:172.21.15.120:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff97406d330 con 0x7ff9740d00a0 2024-03-24T01:38:39.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.799+0000 7ff979200640 1 --2- 172.21.15.90:0/1661343479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9740d00a0 0x7ff974099c30 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff96c009d90 tx=0x7ff96c02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f775098bebbb366f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:39.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.799+0000 7ff972a00640 1 -- 172.21.15.90:0/1661343479 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96c039070 con 0x7ff9740d00a0 2024-03-24T01:38:39.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.799+0000 7ff972a00640 1 -- 172.21.15.90:0/1661343479 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff96c02cc50 con 0x7ff9740d00a0 2024-03-24T01:38:39.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.799+0000 7ff972a00640 1 -- 172.21.15.90:0/1661343479 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96c034440 con 0x7ff9740d00a0 2024-03-24T01:38:39.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.800+0000 7ff97b200640 1 -- 172.21.15.90:0/1661343479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9740d00a0 msgr2=0x7ff974099c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:39.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.800+0000 7ff97b200640 1 --2- 172.21.15.90:0/1661343479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9740d00a0 0x7ff974099c30 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff96c009d90 tx=0x7ff96c02bea0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.800+0000 7ff97b200640 1 -- 172.21.15.90:0/1661343479 shutdown_connections 2024-03-24T01:38:39.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.800+0000 7ff97b200640 1 --2- 172.21.15.90:0/1661343479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9740d00a0 0x7ff974099c30 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.800+0000 7ff97b200640 1 --2- 172.21.15.90:0/1661343479 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff97406cbe0 0x7ff9740cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.800+0000 7ff97b200640 1 -- 172.21.15.90:0/1661343479 >> 172.21.15.90:0/1661343479 conn(0x7ff9740b34e0 msgr2=0x7ff9740b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:39.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.800+0000 7ff97b200640 1 -- 172.21.15.90:0/1661343479 shutdown_connections 2024-03-24T01:38:39.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.800+0000 7ff97b200640 1 -- 172.21.15.90:0/1661343479 wait complete. 2024-03-24T01:38:39.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.802+0000 7ff97b200640 1 Processor -- start 2024-03-24T01:38:39.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.802+0000 7ff97b200640 1 -- start start 2024-03-24T01:38:39.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.803+0000 7ff97b200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff97406cbe0 0x7ff974154ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:39.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.803+0000 7ff97b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9740d00a0 0x7ff9740b6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:39.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.803+0000 7ff97b200640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff97415e0e0 0x7ff9740b1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:39.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.803+0000 7ff97b200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9740b1b60 con 0x7ff9740d00a0 2024-03-24T01:38:39.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.803+0000 7ff97b200640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9740b1880 con 0x7ff97415e0e0 2024-03-24T01:38:39.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.803+0000 7ff97b200640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff97406d040 con 0x7ff97406cbe0 2024-03-24T01:38:39.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.803+0000 7ff973400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9740d00a0 0x7ff9740b6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:39.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.803+0000 7ff973400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9740d00a0 0x7ff9740b6320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38470/0 (socket says 172.21.15.90:38470) 2024-03-24T01:38:39.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.803+0000 7ff973400640 1 -- 172.21.15.90:0/3509026286 learned_addr learned my addr 172.21.15.90:0/3509026286 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:38:39.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.803+0000 7ff979200640 1 --2- 172.21.15.90:0/3509026286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff97415e0e0 0x7ff9740b1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:39.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.803+0000 7ff973400640 1 -- 172.21.15.90:0/3509026286 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff97406cbe0 msgr2=0x7ff974154ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:39.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.803+0000 7ff973400640 1 --2- 172.21.15.90:0/3509026286 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff97406cbe0 0x7ff974154ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.803+0000 7ff973400640 1 -- 172.21.15.90:0/3509026286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff97415e0e0 msgr2=0x7ff9740b1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:39.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.803+0000 7ff973400640 1 --2- 172.21.15.90:0/3509026286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff97415e0e0 0x7ff9740b1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.803+0000 7ff973400640 1 -- 172.21.15.90:0/3509026286 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9740ae820 con 0x7ff9740d00a0 2024-03-24T01:38:39.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.804+0000 7ff973400640 1 --2- 172.21.15.90:0/3509026286 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9740d00a0 0x7ff9740b6320 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7ff96000e870 tx=0x7ff96000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:39.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.804+0000 7ff96be00640 1 -- 172.21.15.90:0/3509026286 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff960017070 con 0x7ff9740d00a0 2024-03-24T01:38:39.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.804+0000 7ff96be00640 1 -- 172.21.15.90:0/3509026286 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff96000c940 con 0x7ff9740d00a0 2024-03-24T01:38:39.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.804+0000 7ff97b200640 1 -- 172.21.15.90:0/3509026286 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9740aeb60 con 0x7ff9740d00a0 2024-03-24T01:38:39.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.804+0000 7ff96be00640 1 -- 172.21.15.90:0/3509026286 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96000cd30 con 0x7ff9740d00a0 2024-03-24T01:38:39.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.804+0000 7ff97b200640 1 -- 172.21.15.90:0/3509026286 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff974155400 con 0x7ff9740d00a0 2024-03-24T01:38:39.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.805+0000 7ff97b200640 1 -- 172.21.15.90:0/3509026286 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff97404f400 con 0x7ff9740d00a0 2024-03-24T01:38:39.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.807+0000 7ff96be00640 1 -- 172.21.15.90:0/3509026286 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7ff96001a070 con 0x7ff9740d00a0 2024-03-24T01:38:39.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.807+0000 7ff96be00640 1 --2- 172.21.15.90:0/3509026286 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff95406bc00 0x7ff95406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:39.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.807+0000 7ff96be00640 1 -- 172.21.15.90:0/3509026286 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7ff96008b3a0 con 0x7ff9740d00a0 2024-03-24T01:38:39.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.808+0000 7ff973e00640 1 --2- 172.21.15.90:0/3509026286 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff95406bc00 0x7ff95406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:39.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.809+0000 7ff973e00640 1 --2- 172.21.15.90:0/3509026286 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff95406bc00 0x7ff95406e0c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7ff964000f60 tx=0x7ff9640078f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:39.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.812+0000 7ff96be00640 1 -- 172.21.15.90:0/3509026286 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7ff9740aeb60 con 0x7ff9740d00a0 2024-03-24T01:38:39.826 INFO:teuthology.orchestra.run.smithi090.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-24T01:38:39.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T01:38:39.827 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:38:39.827 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:38:39.878 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T01:38:39.879 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.1 is not mounted at ubuntu@smithi120.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1 2024-03-24T01:38:39.879 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2024-03-24T01:38:39.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:38:39.879 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-24T01:38:39.966 INFO:teuthology.orchestra.run.smithi120.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-24T01:38:39.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T01:38:39.968 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:38:39.968 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:38:39.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.994+0000 7ff97b200640 1 -- 172.21.15.90:0/3509026286 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff9740b4310 con 0x7ff9740d00a0 2024-03-24T01:38:39.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.995+0000 7ff96be00640 1 -- 172.21.15.90:0/3509026286 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v69) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff960059c70 con 0x7ff9740d00a0 2024-03-24T01:38:39.995 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-24T01:38:39.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.997+0000 7ff97b200640 1 -- 172.21.15.90:0/3509026286 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff95406bc00 msgr2=0x7ff95406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:39.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.997+0000 7ff97b200640 1 --2- 172.21.15.90:0/3509026286 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff95406bc00 0x7ff95406e0c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7ff964000f60 tx=0x7ff9640078f0 comp rx=0 tx=0).stop 2024-03-24T01:38:39.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.997+0000 7ff97b200640 1 -- 172.21.15.90:0/3509026286 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9740d00a0 msgr2=0x7ff9740b6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:40.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.997+0000 7ff97b200640 1 --2- 172.21.15.90:0/3509026286 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9740d00a0 0x7ff9740b6320 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7ff96000e870 tx=0x7ff96000ed30 comp rx=0 tx=0).stop 2024-03-24T01:38:40.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.997+0000 7ff97b200640 1 -- 172.21.15.90:0/3509026286 shutdown_connections 2024-03-24T01:38:40.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.997+0000 7ff97b200640 1 --2- 172.21.15.90:0/3509026286 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7ff95406bc00 0x7ff95406e0c0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:40.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.997+0000 7ff97b200640 1 --2- 172.21.15.90:0/3509026286 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9740d00a0 0x7ff9740b6320 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:40.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.997+0000 7ff97b200640 1 --2- 172.21.15.90:0/3509026286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff97415e0e0 0x7ff9740b1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:40.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.997+0000 7ff97b200640 1 --2- 172.21.15.90:0/3509026286 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7ff97406cbe0 0x7ff974154ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:40.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.997+0000 7ff97b200640 1 -- 172.21.15.90:0/3509026286 >> 172.21.15.90:0/3509026286 conn(0x7ff9740b34e0 msgr2=0x7ff9740cef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:40.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.998+0000 7ff97b200640 1 -- 172.21.15.90:0/3509026286 shutdown_connections 2024-03-24T01:38:40.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:39.998+0000 7ff97b200640 1 -- 172.21.15.90:0/3509026286 wait complete. 2024-03-24T01:38:40.013 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T01:38:40.014 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.3 is not mounted at ubuntu@smithi120.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.3 2024-03-24T01:38:40.014 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi120.front.sepia.ceph.com 2024-03-24T01:38:40.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:38:40.014 DEBUG:teuthology.orchestra.run.smithi120:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-24T01:38:40.016 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T01:38:40.105 INFO:teuthology.orchestra.run.smithi120.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-24T01:38:40.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T01:38:40.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.228+0000 7f30cbe00640 1 Processor -- start 2024-03-24T01:38:40.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.229+0000 7f30cbe00640 1 -- start start 2024-03-24T01:38:40.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.230+0000 7f30cbe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30cc144e10 0x7f30cc06e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:40.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.230+0000 7f30cbe00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30cc0cccc0 con 0x7f30cc144990 2024-03-24T01:38:40.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.230+0000 7f30cbe00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30cc0ccfa0 con 0x7f30cc1445b0 2024-03-24T01:38:40.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.230+0000 7f30cbe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30cc0cca70 con 0x7f30cc144e10 2024-03-24T01:38:40.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.231+0000 7f30c3e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f30cc144990 0x7f30cc06de50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47036/0 (socket says 172.21.15.90:47036) 2024-03-24T01:38:40.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.231+0000 7f30c3e00640 1 -- 172.21.15.90:0/1963560907 learned_addr learned my addr 172.21.15.90:0/1963560907 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:38:40.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.231+0000 7f30cb400640 1 --2- 172.21.15.90:0/1963560907 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30cc144e10 0x7f30cc06e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:40.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.231+0000 7f30ca000640 1 -- 172.21.15.90:0/1963560907 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 657234175 0 0) 0x7f30cc0cccc0 con 0x7f30cc144990 2024-03-24T01:38:40.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.231+0000 7f30ca000640 1 -- 172.21.15.90:0/1963560907 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30b0003520 con 0x7f30cc144990 2024-03-24T01:38:40.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.231+0000 7f30ca000640 1 -- 172.21.15.90:0/1963560907 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1738696200 0 0) 0x7f30b0003520 con 0x7f30cc144990 2024-03-24T01:38:40.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.231+0000 7f30ca000640 1 -- 172.21.15.90:0/1963560907 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30cc144e10 msgr2=0x7f30cc06e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:38:40.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.231+0000 7f30ca000640 1 --2- 172.21.15.90:0/1963560907 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30cc144e10 0x7f30cc06e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:40.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.231+0000 7f30ca000640 1 -- 172.21.15.90:0/1963560907 >> v1:172.21.15.120:6789/0 conn(0x7f30cc1445b0 legacy=0x7f30cc0cfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:40.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.231+0000 7f30ca000640 1 -- 172.21.15.90:0/1963560907 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30cc0cf390 con 0x7f30cc144990 2024-03-24T01:38:40.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.231+0000 7f30ca000640 1 -- 172.21.15.90:0/1963560907 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2764314971 0 0) 0x7f30b4001670 con 0x7f30cc144990 2024-03-24T01:38:40.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.231+0000 7f30ca000640 1 -- 172.21.15.90:0/1963560907 >> v1:172.21.15.90:6789/0 conn(0x7f30cc144990 legacy=0x7f30cc06de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:40.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.232+0000 7f30ca000640 1 --2- 172.21.15.90:0/1963560907 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30b00039e0 0x7f30b0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:40.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.232+0000 7f30ca000640 1 --2- 172.21.15.90:0/1963560907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30b0004390 0x7f30b0004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:40.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.232+0000 7f30ca000640 1 --2- 172.21.15.90:0/1963560907 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30cc144e10 0x7f30b0005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:40.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.232+0000 7f30ca000640 1 -- 172.21.15.90:0/1963560907 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30cc0cca70 con 0x7f30b00039e0 2024-03-24T01:38:40.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.232+0000 7f30ca000640 1 -- 172.21.15.90:0/1963560907 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b000ba70 con 0x7f30b0004390 2024-03-24T01:38:40.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.232+0000 7f30ca000640 1 -- 172.21.15.90:0/1963560907 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30b000bc70 con 0x7f30cc144e10 2024-03-24T01:38:40.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.232+0000 7f30caa00640 1 --2- 172.21.15.90:0/1963560907 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30b00039e0 0x7f30b0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:40.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.232+0000 7f30c3e00640 1 --2- 172.21.15.90:0/1963560907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30b0004390 0x7f30b0004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:40.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.232+0000 7f30cb400640 1 --2- 172.21.15.90:0/1963560907 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30cc144e10 0x7f30b0005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:40.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.232+0000 7f30caa00640 1 -- 172.21.15.90:0/1963560907 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30cc144e10 msgr2=0x7f30b0005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:38:40.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.232+0000 7f30caa00640 1 --2- 172.21.15.90:0/1963560907 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30cc144e10 0x7f30b0005160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:40.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.232+0000 7f30caa00640 1 -- 172.21.15.90:0/1963560907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30b0004390 msgr2=0x7f30b0004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:38:40.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.232+0000 7f30caa00640 1 --2- 172.21.15.90:0/1963560907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30b0004390 0x7f30b0004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:40.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.232+0000 7f30caa00640 1 -- 172.21.15.90:0/1963560907 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b0003520 con 0x7f30b00039e0 2024-03-24T01:38:40.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.232+0000 7f30c3e00640 1 --2- 172.21.15.90:0/1963560907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30b0004390 0x7f30b0004860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:38:40.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.233+0000 7f30caa00640 1 --2- 172.21.15.90:0/1963560907 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30b00039e0 0x7f30b0003dc0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f30bc001870 tx=0x7f30b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f2ae30814936bfe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:40.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.233+0000 7f30ca000640 1 -- 172.21.15.90:0/1963560907 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc037070 con 0x7f30b00039e0 2024-03-24T01:38:40.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.233+0000 7f30ca000640 1 -- 172.21.15.90:0/1963560907 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30bc0052d0 con 0x7f30b00039e0 2024-03-24T01:38:40.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.233+0000 7f30ca000640 1 -- 172.21.15.90:0/1963560907 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc001440 con 0x7f30b00039e0 2024-03-24T01:38:40.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.234+0000 7f30cbe00640 1 -- 172.21.15.90:0/1963560907 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30b00039e0 msgr2=0x7f30b0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:40.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.234+0000 7f30cbe00640 1 --2- 172.21.15.90:0/1963560907 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30b00039e0 0x7f30b0003dc0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f30bc001870 tx=0x7f30b0005830 comp rx=0 tx=0).stop 2024-03-24T01:38:40.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.234+0000 7f30cb400640 1 -- 172.21.15.90:0/1963560907 reap_dead start 2024-03-24T01:38:40.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.234+0000 7f30cbe00640 1 -- 172.21.15.90:0/1963560907 shutdown_connections 2024-03-24T01:38:40.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.234+0000 7f30cbe00640 1 -- 172.21.15.90:0/1963560907 >> 172.21.15.90:0/1963560907 conn(0x7f30cc06b0c0 msgr2=0x7f30cc06b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:40.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.234+0000 7f30cbe00640 1 -- 172.21.15.90:0/1963560907 shutdown_connections 2024-03-24T01:38:40.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.234+0000 7f30cbe00640 1 -- 172.21.15.90:0/1963560907 wait complete. 2024-03-24T01:38:40.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.236+0000 7f30cbe00640 1 Processor -- start 2024-03-24T01:38:40.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.236+0000 7f30cbe00640 1 -- start start 2024-03-24T01:38:40.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.237+0000 7f30cbe00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30cc1424a0 0x7f30cc0c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:40.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.237+0000 7f30cbe00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30cc142880 0x7f30cc1f7e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:40.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.237+0000 7f30cbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30cc13b4d0 0x7f30cc0ca8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:40.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.237+0000 7f30cbe00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30cc06e560 con 0x7f30cc13b4d0 2024-03-24T01:38:40.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.237+0000 7f30cbe00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30cc0cf700 con 0x7f30cc1424a0 2024-03-24T01:38:40.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.237+0000 7f30cbe00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30cc06e6e0 con 0x7f30cc142880 2024-03-24T01:38:40.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.237+0000 7f30cb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30cc13b4d0 0x7f30cc0ca8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:40.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.237+0000 7f30cb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30cc13b4d0 0x7f30cc0ca8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55394/0 (socket says 172.21.15.90:55394) 2024-03-24T01:38:40.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.237+0000 7f30cb400640 1 -- 172.21.15.90:0/2587357371 learned_addr learned my addr 172.21.15.90:0/2587357371 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:38:40.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.237+0000 7f30caa00640 1 --2- 172.21.15.90:0/2587357371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30cc1424a0 0x7f30cc0c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:40.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.238+0000 7f30c3e00640 1 --2- 172.21.15.90:0/2587357371 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30cc142880 0x7f30cc1f7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:40.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.238+0000 7f30cb400640 1 -- 172.21.15.90:0/2587357371 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30cc142880 msgr2=0x7f30cc1f7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:40.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.238+0000 7f30cb400640 1 --2- 172.21.15.90:0/2587357371 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30cc142880 0x7f30cc1f7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:40.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.238+0000 7f30cb400640 1 -- 172.21.15.90:0/2587357371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30cc1424a0 msgr2=0x7f30cc0c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:40.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.238+0000 7f30cb400640 1 --2- 172.21.15.90:0/2587357371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30cc1424a0 0x7f30cc0c8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:40.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.238+0000 7f30cb400640 1 -- 172.21.15.90:0/2587357371 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30cc0c90c0 con 0x7f30cc13b4d0 2024-03-24T01:38:40.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.238+0000 7f30cb400640 1 --2- 172.21.15.90:0/2587357371 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30cc13b4d0 0x7f30cc0ca8e0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f30cc06d810 tx=0x7f30c400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:40.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.238+0000 7f30c2a00640 1 -- 172.21.15.90:0/2587357371 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c401c070 con 0x7f30cc13b4d0 2024-03-24T01:38:40.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.238+0000 7f30c2a00640 1 -- 172.21.15.90:0/2587357371 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30c4017400 con 0x7f30cc13b4d0 2024-03-24T01:38:40.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.238+0000 7f30c2a00640 1 -- 172.21.15.90:0/2587357371 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c40177f0 con 0x7f30cc13b4d0 2024-03-24T01:38:40.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.238+0000 7f30cbe00640 1 -- 172.21.15.90:0/2587357371 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30cc0c93a0 con 0x7f30cc13b4d0 2024-03-24T01:38:40.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.239+0000 7f30cbe00640 1 -- 172.21.15.90:0/2587357371 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30cc13b950 con 0x7f30cc13b4d0 2024-03-24T01:38:40.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.240+0000 7f30cbe00640 1 -- 172.21.15.90:0/2587357371 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30cc13bc10 con 0x7f30cc13b4d0 2024-03-24T01:38:40.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.241+0000 7f30c2a00640 1 -- 172.21.15.90:0/2587357371 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f30c4005d10 con 0x7f30cc13b4d0 2024-03-24T01:38:40.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.242+0000 7f30c2a00640 1 --2- 172.21.15.90:0/2587357371 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f30a806bc00 0x7f30a806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:40.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.242+0000 7f30c2a00640 1 -- 172.21.15.90:0/2587357371 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7f30c408ad10 con 0x7f30cc13b4d0 2024-03-24T01:38:40.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.242+0000 7f30caa00640 1 --2- 172.21.15.90:0/2587357371 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f30a806bc00 0x7f30a806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:40.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.243+0000 7f30caa00640 1 --2- 172.21.15.90:0/2587357371 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f30a806bc00 0x7f30a806e0c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f30bc034000 tx=0x7f30bc009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:40.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.246+0000 7f30c2a00640 1 -- 172.21.15.90:0/2587357371 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f30cc13bc10 con 0x7f30cc13b4d0 2024-03-24T01:38:40.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.445+0000 7f30cbe00640 1 -- 172.21.15.90:0/2587357371 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f30cc04f400 con 0x7f30cc13b4d0 2024-03-24T01:38:40.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.446+0000 7f30c2a00640 1 -- 172.21.15.90:0/2587357371 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v69) v1 ==== 100+0+11 (secure 0 0 0) 0x7f30c4012070 con 0x7f30cc13b4d0 2024-03-24T01:38:40.446 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-24T01:38:40.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.448+0000 7f30cbe00640 1 -- 172.21.15.90:0/2587357371 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f30a806bc00 msgr2=0x7f30a806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:40.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.448+0000 7f30cbe00640 1 --2- 172.21.15.90:0/2587357371 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f30a806bc00 0x7f30a806e0c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f30bc034000 tx=0x7f30bc009030 comp rx=0 tx=0).stop 2024-03-24T01:38:40.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.448+0000 7f30cbe00640 1 -- 172.21.15.90:0/2587357371 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30cc13b4d0 msgr2=0x7f30cc0ca8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:40.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.448+0000 7f30cbe00640 1 --2- 172.21.15.90:0/2587357371 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30cc13b4d0 0x7f30cc0ca8e0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f30cc06d810 tx=0x7f30c400f4a0 comp rx=0 tx=0).stop 2024-03-24T01:38:40.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.448+0000 7f30cbe00640 1 -- 172.21.15.90:0/2587357371 shutdown_connections 2024-03-24T01:38:40.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.448+0000 7f30cbe00640 1 --2- 172.21.15.90:0/2587357371 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f30a806bc00 0x7f30a806e0c0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:40.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.448+0000 7f30cbe00640 1 --2- 172.21.15.90:0/2587357371 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30cc13b4d0 0x7f30cc0ca8e0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:40.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.448+0000 7f30cbe00640 1 --2- 172.21.15.90:0/2587357371 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f30cc142880 0x7f30cc1f7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:40.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.448+0000 7f30cbe00640 1 --2- 172.21.15.90:0/2587357371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30cc1424a0 0x7f30cc0c8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:40.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.448+0000 7f30cbe00640 1 -- 172.21.15.90:0/2587357371 >> 172.21.15.90:0/2587357371 conn(0x7f30cc06b0c0 msgr2=0x7f30cc0cfb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:40.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.448+0000 7f30cbe00640 1 -- 172.21.15.90:0/2587357371 shutdown_connections 2024-03-24T01:38:40.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.448+0000 7f30cbe00640 1 -- 172.21.15.90:0/2587357371 wait complete. 2024-03-24T01:38:40.468 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:38:40.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.667+0000 7f489be00640 1 Processor -- start 2024-03-24T01:38:40.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.667+0000 7f489be00640 1 -- start start 2024-03-24T01:38:40.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.668+0000 7f489be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f489c06c860 0x7f489c0cf210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:40.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.668+0000 7f489be00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f489c0d1b20 con 0x7f489c0d1620 2024-03-24T01:38:40.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.668+0000 7f489be00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f489c0d1e00 con 0x7f489c06cc40 2024-03-24T01:38:40.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.668+0000 7f489be00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f489c0d2730 con 0x7f489c06c860 2024-03-24T01:38:40.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.669+0000 7f489aa00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f489c06c860 0x7f489c0cf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:40.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.669+0000 7f489a000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f489c0d1620 0x7f489c0d2bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47040/0 (socket says 172.21.15.90:47040) 2024-03-24T01:38:40.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.669+0000 7f489a000640 1 -- 172.21.15.90:0/4173561961 learned_addr learned my addr 172.21.15.90:0/4173561961 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-24T01:38:40.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.669+0000 7f489aa00640 1 -- 172.21.15.90:0/4173561961 >> v1:172.21.15.120:6789/0 conn(0x7f489c06cc40 legacy=0x7f489c0b6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:40.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.669+0000 7f489aa00640 1 -- 172.21.15.90:0/4173561961 >> v1:172.21.15.90:6789/0 conn(0x7f489c0d1620 legacy=0x7f489c0d2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:40.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.669+0000 7f4899600640 1 -- 172.21.15.90:0/4173561961 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3676504176 0 0) 0x7f489c0d1b20 con 0x7f489c0d1620 2024-03-24T01:38:40.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.669+0000 7f489aa00640 1 -- 172.21.15.90:0/4173561961 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f489c0d20e0 con 0x7f489c06c860 2024-03-24T01:38:40.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.671+0000 7f489aa00640 1 --2- 172.21.15.90:0/4173561961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f489c06c860 0x7f489c0cf210 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f488c0097a0 tx=0x7f488c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cf13eb090d173fa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:40.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.671+0000 7f4899600640 1 -- 172.21.15.90:0/4173561961 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488c034070 con 0x7f489c06c860 2024-03-24T01:38:40.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.671+0000 7f4899600640 1 -- 172.21.15.90:0/4173561961 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f488c030610 con 0x7f489c06c860 2024-03-24T01:38:40.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.671+0000 7f4899600640 1 -- 172.21.15.90:0/4173561961 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488c0309e0 con 0x7f489c06c860 2024-03-24T01:38:40.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.672+0000 7f489be00640 1 -- 172.21.15.90:0/4173561961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f489c06c860 msgr2=0x7f489c0cf210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:40.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.672+0000 7f489be00640 1 --2- 172.21.15.90:0/4173561961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f489c06c860 0x7f489c0cf210 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f488c0097a0 tx=0x7f488c02b960 comp rx=0 tx=0).stop 2024-03-24T01:38:40.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.672+0000 7f489be00640 1 -- 172.21.15.90:0/4173561961 shutdown_connections 2024-03-24T01:38:40.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.672+0000 7f489be00640 1 --2- 172.21.15.90:0/4173561961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f489c06c860 0x7f489c0cf210 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:40.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.672+0000 7f489be00640 1 -- 172.21.15.90:0/4173561961 >> 172.21.15.90:0/4173561961 conn(0x7f489c0d0560 msgr2=0x7f489c0d0970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:40.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.672+0000 7f489be00640 1 -- 172.21.15.90:0/4173561961 shutdown_connections 2024-03-24T01:38:40.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.672+0000 7f489be00640 1 -- 172.21.15.90:0/4173561961 wait complete. 2024-03-24T01:38:40.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.673+0000 7f489be00640 1 Processor -- start 2024-03-24T01:38:40.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.674+0000 7f489be00640 1 -- start start 2024-03-24T01:38:40.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.674+0000 7f489be00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f489c06c860 0x7f489c0c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:40.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.674+0000 7f489be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f489c06cc40 0x7f489c1f7010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:40.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.674+0000 7f489be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f489c0d1620 0x7f489c1509a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:40.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.674+0000 7f489be00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f489c0c7300 con 0x7f489c0d1620 2024-03-24T01:38:40.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.674+0000 7f489be00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f489c0c7180 con 0x7f489c06c860 2024-03-24T01:38:40.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.674+0000 7f489be00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f489c0d32d0 con 0x7f489c06cc40 2024-03-24T01:38:40.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.675+0000 7f489b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f489c0d1620 0x7f489c1509a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:40.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.675+0000 7f489b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f489c0d1620 0x7f489c1509a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55412/0 (socket says 172.21.15.90:55412) 2024-03-24T01:38:40.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.675+0000 7f489b400640 1 -- 172.21.15.90:0/4242404926 learned_addr learned my addr 172.21.15.90:0/4242404926 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:38:40.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.675+0000 7f489aa00640 1 --2- 172.21.15.90:0/4242404926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f489c06c860 0x7f489c0c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:40.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.675+0000 7f489a000640 1 --2- 172.21.15.90:0/4242404926 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f489c06cc40 0x7f489c1f7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:40.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.675+0000 7f489b400640 1 -- 172.21.15.90:0/4242404926 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f489c06cc40 msgr2=0x7f489c1f7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:40.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.675+0000 7f489b400640 1 --2- 172.21.15.90:0/4242404926 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f489c06cc40 0x7f489c1f7010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:40.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.675+0000 7f489b400640 1 -- 172.21.15.90:0/4242404926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f489c06c860 msgr2=0x7f489c0c7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:40.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.675+0000 7f489b400640 1 --2- 172.21.15.90:0/4242404926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f489c06c860 0x7f489c0c7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:40.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.675+0000 7f489b400640 1 -- 172.21.15.90:0/4242404926 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f489c1f7980 con 0x7f489c0d1620 2024-03-24T01:38:40.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.675+0000 7f489aa00640 1 --2- 172.21.15.90:0/4242404926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f489c06c860 0x7f489c0c7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:38:40.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.675+0000 7f489a000640 1 --2- 172.21.15.90:0/4242404926 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f489c06cc40 0x7f489c1f7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T01:38:40.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.676+0000 7f489b400640 1 --2- 172.21.15.90:0/4242404926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f489c0d1620 0x7f489c1509a0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f489400c7a0 tx=0x7f489400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:40.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.676+0000 7f4892a00640 1 -- 172.21.15.90:0/4242404926 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4894003b70 con 0x7f489c0d1620 2024-03-24T01:38:40.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.676+0000 7f4892a00640 1 -- 172.21.15.90:0/4242404926 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4894003d30 con 0x7f489c0d1620 2024-03-24T01:38:40.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.676+0000 7f4892a00640 1 -- 172.21.15.90:0/4242404926 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489400aa90 con 0x7f489c0d1620 2024-03-24T01:38:40.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.676+0000 7f489be00640 1 -- 172.21.15.90:0/4242404926 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f489c1f7c60 con 0x7f489c0d1620 2024-03-24T01:38:40.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.676+0000 7f489be00640 1 -- 172.21.15.90:0/4242404926 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f489c0c7990 con 0x7f489c0d1620 2024-03-24T01:38:40.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.677+0000 7f489be00640 1 -- 172.21.15.90:0/4242404926 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f489c04f400 con 0x7f489c0d1620 2024-03-24T01:38:40.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.678+0000 7f4892a00640 1 -- 172.21.15.90:0/4242404926 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7f489400ac30 con 0x7f489c0d1620 2024-03-24T01:38:40.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.679+0000 7f4892a00640 1 --2- 172.21.15.90:0/4242404926 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f487c06bbd0 0x7f487c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:40.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.679+0000 7f4892a00640 1 -- 172.21.15.90:0/4242404926 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7f489408abf0 con 0x7f489c0d1620 2024-03-24T01:38:40.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.679+0000 7f489aa00640 1 --2- 172.21.15.90:0/4242404926 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f487c06bbd0 0x7f487c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:40.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.680+0000 7f489aa00640 1 --2- 172.21.15.90:0/4242404926 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f487c06bbd0 0x7f487c06e090 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f488c036000 tx=0x7f488c03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:40.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.683+0000 7f4892a00640 1 -- 172.21.15.90:0/4242404926 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7f48940594a0 con 0x7f489c0d1620 2024-03-24T01:38:40.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.904+0000 7f489be00640 1 -- 172.21.15.90:0/4242404926 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f489c0c7b60 con 0x7f489c0d1620 2024-03-24T01:38:40.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.905+0000 7f4892a00640 1 -- 172.21.15.90:0/4242404926 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+5050 (secure 0 0 0) 0x7f4894058da0 con 0x7f489c0d1620 2024-03-24T01:38:40.906 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:38:40.906 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":58,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5177,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/3458630824","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":3458630824},{"type":"v1","addr":"172.21.15.120:6837","nonce":3458630824}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":5614,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/1995195696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1995195696},{"type":"v1","addr":"172.21.15.90:6835","nonce":1995195696}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":58,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T01:32:44.150735+0000","modified":"2024-03-24T01:36:32.782070+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7110,"mds_1":7098},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7098":{"gid":7098,"name":"a","rank":1,"incarnation":52,"state":"up:active","state_seq":7,"addr":"172.21.15.90:6834/1760094626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":1760094626},{"type":"v1","addr":"172.21.15.90:6834","nonce":1760094626}]},"join_fscid":-1,"laggy_since":"2024-03-24T01:33:46.240858+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7110":{"gid":7110,"name":"d","rank":0,"incarnation":46,"state":"up:active","state_seq":60,"addr":"172.21.15.120:6835/910380191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":910380191},{"type":"v1","addr":"172.21.15.120:6835","nonce":910380191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7110,"qdb_cluster":[7110,7098]},"id":4}]} 2024-03-24T01:38:40.907 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 58 2024-03-24T01:38:40.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.907+0000 7f489be00640 1 -- 172.21.15.90:0/4242404926 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f487c06bbd0 msgr2=0x7f487c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:40.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.907+0000 7f489be00640 1 --2- 172.21.15.90:0/4242404926 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f487c06bbd0 0x7f487c06e090 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f488c036000 tx=0x7f488c03b040 comp rx=0 tx=0).stop 2024-03-24T01:38:40.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.907+0000 7f489be00640 1 -- 172.21.15.90:0/4242404926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f489c0d1620 msgr2=0x7f489c1509a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:40.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.907+0000 7f489be00640 1 --2- 172.21.15.90:0/4242404926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f489c0d1620 0x7f489c1509a0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f489400c7a0 tx=0x7f489400cc60 comp rx=0 tx=0).stop 2024-03-24T01:38:40.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.907+0000 7f489be00640 1 -- 172.21.15.90:0/4242404926 shutdown_connections 2024-03-24T01:38:40.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.907+0000 7f489be00640 1 --2- 172.21.15.90:0/4242404926 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7f487c06bbd0 0x7f487c06e090 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:40.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.907+0000 7f489be00640 1 --2- 172.21.15.90:0/4242404926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f489c0d1620 0x7f489c1509a0 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:40.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.907+0000 7f489be00640 1 --2- 172.21.15.90:0/4242404926 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f489c06cc40 0x7f489c1f7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:40.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.907+0000 7f489be00640 1 --2- 172.21.15.90:0/4242404926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f489c06c860 0x7f489c0c7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:40.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.907+0000 7f489be00640 1 -- 172.21.15.90:0/4242404926 >> 172.21.15.90:0/4242404926 conn(0x7f489c0d0560 msgr2=0x7f489c0b7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:40.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.907+0000 7f489be00640 1 -- 172.21.15.90:0/4242404926 shutdown_connections 2024-03-24T01:38:40.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:40.907+0000 7f489be00640 1 -- 172.21.15.90:0/4242404926 wait complete. 2024-03-24T01:38:40.927 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T01:38:41.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.124+0000 7fccf8600640 1 Processor -- start 2024-03-24T01:38:41.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.124+0000 7fccf8600640 1 -- start start 2024-03-24T01:38:41.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.125+0000 7fccf8600640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7fccf00cc150 0x7fccf00ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:41.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.125+0000 7fccf8600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fccf01603e0 0x7fccf006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:41.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.125+0000 7fccf8600640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccf00cd490 con 0x7fccf0145430 2024-03-24T01:38:41.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.125+0000 7fccf8600640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccf015f900 con 0x7fccf00cc150 2024-03-24T01:38:41.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.125+0000 7fccf8600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccf015fbe0 con 0x7fccf01603e0 2024-03-24T01:38:41.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.126+0000 7fccf5200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fccf01603e0 0x7fccf006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:41.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.126+0000 7fccf5200640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fccf01603e0 0x7fccf006b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3301/0 says I am v2:172.21.15.90:44704/0 (socket says 172.21.15.90:44704) 2024-03-24T01:38:41.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.126+0000 7fccf5200640 1 -- 172.21.15.90:0/512795489 learned_addr learned my addr 172.21.15.90:0/512795489 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:38:41.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.126+0000 7fccf5c00640 1 --2- 172.21.15.90:0/512795489 >> v2:172.21.15.120:3300/0 conn(0x7fccf00cc150 0x7fccf00ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:41.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.126+0000 7fccf5200640 1 -- 172.21.15.90:0/512795489 >> v2:172.21.15.120:3300/0 conn(0x7fccf00cc150 msgr2=0x7fccf00ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:38:41.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.126+0000 7fccf5200640 1 --2- 172.21.15.90:0/512795489 >> v2:172.21.15.120:3300/0 conn(0x7fccf00cc150 0x7fccf00ccb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:41.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.126+0000 7fccf5200640 1 -- 172.21.15.90:0/512795489 >> v1:172.21.15.120:6789/0 conn(0x7fccf0145430 legacy=0x7fccf0071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:41.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.126+0000 7fccf5200640 1 -- 172.21.15.90:0/512795489 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf015fec0 con 0x7fccf01603e0 2024-03-24T01:38:41.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.128+0000 7fccf5200640 1 --2- 172.21.15.90:0/512795489 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fccf01603e0 0x7fccf006b0c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fccdc009870 tx=0x7fccdc02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21b8db34d79be090 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:41.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.128+0000 7fccefe00640 1 -- 172.21.15.90:0/512795489 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccdc034070 con 0x7fccf01603e0 2024-03-24T01:38:41.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.128+0000 7fccefe00640 1 -- 172.21.15.90:0/512795489 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccdc030840 con 0x7fccf01603e0 2024-03-24T01:38:41.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.128+0000 7fccefe00640 1 -- 172.21.15.90:0/512795489 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccdc030c10 con 0x7fccf01603e0 2024-03-24T01:38:41.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.129+0000 7fccf8600640 1 -- 172.21.15.90:0/512795489 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fccf01603e0 msgr2=0x7fccf006b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:41.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.129+0000 7fccf8600640 1 --2- 172.21.15.90:0/512795489 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fccf01603e0 0x7fccf006b0c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fccdc009870 tx=0x7fccdc02b980 comp rx=0 tx=0).stop 2024-03-24T01:38:41.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.129+0000 7fccf8600640 1 -- 172.21.15.90:0/512795489 shutdown_connections 2024-03-24T01:38:41.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.129+0000 7fccf8600640 1 --2- 172.21.15.90:0/512795489 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fccf01603e0 0x7fccf006b0c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:41.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.129+0000 7fccf8600640 1 --2- 172.21.15.90:0/512795489 >> v2:172.21.15.120:3300/0 conn(0x7fccf00cc150 0x7fccf00ccb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:41.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.129+0000 7fccf8600640 1 -- 172.21.15.90:0/512795489 >> 172.21.15.90:0/512795489 conn(0x7fccf00b1770 msgr2=0x7fccf00b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:41.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.129+0000 7fccf8600640 1 -- 172.21.15.90:0/512795489 shutdown_connections 2024-03-24T01:38:41.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.129+0000 7fccf8600640 1 -- 172.21.15.90:0/512795489 wait complete. 2024-03-24T01:38:41.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.131+0000 7fccf8600640 1 Processor -- start 2024-03-24T01:38:41.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.131+0000 7fccf8600640 1 -- start start 2024-03-24T01:38:41.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.132+0000 7fccf8600640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fccf00cc150 0x7fccf015b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:41.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.132+0000 7fccf8600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fccf0145430 0x7fccf01f78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:41.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.132+0000 7fccf8600640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fccf01603e0 0x7fccf00ad930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:41.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.132+0000 7fccf8600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf006b780 con 0x7fccf0145430 2024-03-24T01:38:41.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.132+0000 7fccf8600640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf006b600 con 0x7fccf01603e0 2024-03-24T01:38:41.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.132+0000 7fccf8600640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccf006b900 con 0x7fccf00cc150 2024-03-24T01:38:41.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.132+0000 7fccf5200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fccf0145430 0x7fccf01f78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:41.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.132+0000 7fccf5200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fccf0145430 0x7fccf01f78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55422/0 (socket says 172.21.15.90:55422) 2024-03-24T01:38:41.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.132+0000 7fccf5200640 1 -- 172.21.15.90:0/790523571 learned_addr learned my addr 172.21.15.90:0/790523571 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:38:41.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.132+0000 7fccf5c00640 1 --2- 172.21.15.90:0/790523571 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fccf00cc150 0x7fccf015b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:41.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.133+0000 7fccf6600640 1 --2- 172.21.15.90:0/790523571 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fccf01603e0 0x7fccf00ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:41.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.133+0000 7fccf5c00640 1 -- 172.21.15.90:0/790523571 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fccf01603e0 msgr2=0x7fccf00ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:41.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.133+0000 7fccf5c00640 1 --2- 172.21.15.90:0/790523571 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fccf01603e0 0x7fccf00ad930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:41.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.133+0000 7fccf5c00640 1 -- 172.21.15.90:0/790523571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fccf0145430 msgr2=0x7fccf01f78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:41.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.133+0000 7fccf5c00640 1 --2- 172.21.15.90:0/790523571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fccf0145430 0x7fccf01f78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:41.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.133+0000 7fccf5c00640 1 -- 172.21.15.90:0/790523571 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf015d230 con 0x7fccf00cc150 2024-03-24T01:38:41.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.133+0000 7fccf5200640 1 --2- 172.21.15.90:0/790523571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fccf0145430 0x7fccf01f78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T01:38:41.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.133+0000 7fccf5c00640 1 --2- 172.21.15.90:0/790523571 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fccf00cc150 0x7fccf015b430 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcce400e870 tx=0x7fcce400ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:41.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.133+0000 7fcced600640 1 -- 172.21.15.90:0/790523571 <== mon.2 v2:172.21.15.120:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce4017070 con 0x7fccf00cc150 2024-03-24T01:38:41.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.134+0000 7fcced600640 1 -- 172.21.15.90:0/790523571 <== mon.2 v2:172.21.15.120:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce400c940 con 0x7fccf00cc150 2024-03-24T01:38:41.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.134+0000 7fccf8600640 1 -- 172.21.15.90:0/790523571 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccf01f7e10 con 0x7fccf00cc150 2024-03-24T01:38:41.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.134+0000 7fcced600640 1 -- 172.21.15.90:0/790523571 <== mon.2 v2:172.21.15.120:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce400cd10 con 0x7fccf00cc150 2024-03-24T01:38:41.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.134+0000 7fccf8600640 1 -- 172.21.15.90:0/790523571 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccf015c990 con 0x7fccf00cc150 2024-03-24T01:38:41.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.135+0000 7fccf8600640 1 -- 172.21.15.90:0/790523571 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccf0068060 con 0x7fccf00cc150 2024-03-24T01:38:41.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.136+0000 7fcced600640 1 -- 172.21.15.90:0/790523571 <== mon.2 v2:172.21.15.120:3301/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fcce401a070 con 0x7fccf00cc150 2024-03-24T01:38:41.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.137+0000 7fcced600640 1 --2- 172.21.15.90:0/790523571 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fccd006bc00 0x7fccd006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:41.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.137+0000 7fcced600640 1 -- 172.21.15.90:0/790523571 <== mon.2 v2:172.21.15.120:3301/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7fcce408ab60 con 0x7fccf00cc150 2024-03-24T01:38:41.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.137+0000 7fccf5200640 1 --2- 172.21.15.90:0/790523571 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fccd006bc00 0x7fccd006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:41.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.138+0000 7fccf5200640 1 --2- 172.21.15.90:0/790523571 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fccd006bc00 0x7fccd006e0c0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fccdc009870 tx=0x7fccdc03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:41.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.152+0000 7fcced600640 1 -- 172.21.15.90:0/790523571 <== mon.2 v2:172.21.15.120:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fcce4058e50 con 0x7fccf00cc150 2024-03-24T01:38:41.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.353+0000 7fccf8600640 1 -- 172.21.15.90:0/790523571 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fccf00b3220 con 0x7fccf00cc150 2024-03-24T01:38:41.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.357+0000 7fcced600640 1 -- 172.21.15.90:0/790523571 <== mon.2 v2:172.21.15.120:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+17101 (secure 0 0 0) 0x7fcce405cef0 con 0x7fccf00cc150 2024-03-24T01:38:41.358 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-24T01:38:41.358 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":69,"fsid":"24712165-9839-4fcc-8033-988386efea74","created":"2024-03-24T01:23:46.212633+0000","modified":"2024-03-24T01:32:54.821325+0000","last_up_change":"2024-03-24T01:23:58.837072+0000","last_in_change":"2024-03-24T01:23:49.609376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T01:23:59.548370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-24T01:32:42.026147+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-24T01:32:43.180712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aeced55e-300f-4d89-89ec-f59579c3f96c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6801","nonce":307045587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6803","nonce":307045587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6807","nonce":307045587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":307045587},{"type":"v1","addr":"172.21.15.90:6805","nonce":307045587}]},"public_addr":"172.21.15.90:6801/307045587","cluster_addr":"172.21.15.90:6803/307045587","heartbeat_back_addr":"172.21.15.90:6807/307045587","heartbeat_front_addr":"172.21.15.90:6805/307045587","state":["exists","up"]},{"osd":1,"uuid":"f7214f99-b216-43b3-91b9-67ce92c611cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6818","nonce":133441085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6821","nonce":133441085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6828","nonce":133441085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6823","nonce":133441085},{"type":"v1","addr":"172.21.15.90:6825","nonce":133441085}]},"public_addr":"172.21.15.90:6818/133441085","cluster_addr":"172.21.15.90:6821/133441085","heartbeat_back_addr":"172.21.15.90:6828/133441085","heartbeat_front_addr":"172.21.15.90:6825/133441085","state":["exists","up"]},{"osd":2,"uuid":"9cb0e2f5-39b8-454e-87b5-4e3eb784ab24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6817","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6820","nonce":865053620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6824","nonce":865053620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6831","nonce":865053620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6827","nonce":865053620},{"type":"v1","addr":"172.21.15.90:6829","nonce":865053620}]},"public_addr":"172.21.15.90:6820/865053620","cluster_addr":"172.21.15.90:6824/865053620","heartbeat_back_addr":"172.21.15.90:6831/865053620","heartbeat_front_addr":"172.21.15.90:6829/865053620","state":["exists","up"]},{"osd":3,"uuid":"4d8e413a-693a-4587-925f-2f5263a49617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6809","nonce":2999931983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6811","nonce":2999931983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6815","nonce":2999931983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":2999931983},{"type":"v1","addr":"172.21.15.90:6813","nonce":2999931983}]},"public_addr":"172.21.15.90:6809/2999931983","cluster_addr":"172.21.15.90:6811/2999931983","heartbeat_back_addr":"172.21.15.90:6815/2999931983","heartbeat_front_addr":"172.21.15.90:6813/2999931983","state":["exists","up"]},{"osd":4,"uuid":"898478a5-2b9d-4f13-8bdd-cb28aafe1f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6801","nonce":2150486081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6803","nonce":2150486081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6807","nonce":2150486081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2150486081},{"type":"v1","addr":"172.21.15.120:6805","nonce":2150486081}]},"public_addr":"172.21.15.120:6801/2150486081","cluster_addr":"172.21.15.120:6803/2150486081","heartbeat_back_addr":"172.21.15.120:6807/2150486081","heartbeat_front_addr":"172.21.15.120:6805/2150486081","state":["exists","up"]},{"osd":5,"uuid":"35fe45dc-a2aa-44de-a90b-a3fe9e274fed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6825","nonce":1739969887}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6827","nonce":1739969887}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6831","nonce":1739969887}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":1739969887},{"type":"v1","addr":"172.21.15.120:6829","nonce":1739969887}]},"public_addr":"172.21.15.120:6825/1739969887","cluster_addr":"172.21.15.120:6827/1739969887","heartbeat_back_addr":"172.21.15.120:6831/1739969887","heartbeat_front_addr":"172.21.15.120:6829/1739969887","state":["exists","up"]},{"osd":6,"uuid":"9af87141-0549-43a1-9350-abb0db6108f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6817","nonce":1454745630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6819","nonce":1454745630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6823","nonce":1454745630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":1454745630},{"type":"v1","addr":"172.21.15.120:6821","nonce":1454745630}]},"public_addr":"172.21.15.120:6817/1454745630","cluster_addr":"172.21.15.120:6819/1454745630","heartbeat_back_addr":"172.21.15.120:6823/1454745630","heartbeat_front_addr":"172.21.15.120:6821/1454745630","state":["exists","up"]},{"osd":7,"uuid":"424703a0-ba08-4d01-9d48-b38eabdde4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6809","nonce":1045862942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6811","nonce":1045862942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6815","nonce":1045862942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1045862942},{"type":"v1","addr":"172.21.15.120:6813","nonce":1045862942}]},"public_addr":"172.21.15.120:6809/1045862942","cluster_addr":"172.21.15.120:6811/1045862942","heartbeat_back_addr":"172.21.15.120:6815/1045862942","heartbeat_front_addr":"172.21.15.120:6813/1045862942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.689505+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.712373+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.983561+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.959885+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.697138+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.733960+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.680332+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T01:23:54.969301+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.1e","mappings":[{"from":7,"to":2}]},{"pgid":"10.10","mappings":[{"from":4,"to":3}]},{"pgid":"10.1d","mappings":[{"from":4,"to":3}]},{"pgid":"10.1f","mappings":[{"from":7,"to":3}]},{"pgid":"10.37","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T01:38:41.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.359+0000 7fccf8600640 1 -- 172.21.15.90:0/790523571 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fccd006bc00 msgr2=0x7fccd006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:41.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.359+0000 7fccf8600640 1 --2- 172.21.15.90:0/790523571 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fccd006bc00 0x7fccd006e0c0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fccdc009870 tx=0x7fccdc03b040 comp rx=0 tx=0).stop 2024-03-24T01:38:41.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.359+0000 7fccf8600640 1 -- 172.21.15.90:0/790523571 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fccf00cc150 msgr2=0x7fccf015b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:41.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.359+0000 7fccf8600640 1 --2- 172.21.15.90:0/790523571 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fccf00cc150 0x7fccf015b430 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcce400e870 tx=0x7fcce400ed30 comp rx=0 tx=0).stop 2024-03-24T01:38:41.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.359+0000 7fccf8600640 1 -- 172.21.15.90:0/790523571 shutdown_connections 2024-03-24T01:38:41.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.359+0000 7fccf8600640 1 --2- 172.21.15.90:0/790523571 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fccd006bc00 0x7fccd006e0c0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:41.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.359+0000 7fccf8600640 1 --2- 172.21.15.90:0/790523571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fccf0145430 0x7fccf01f78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:41.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.359+0000 7fccf8600640 1 --2- 172.21.15.90:0/790523571 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fccf01603e0 0x7fccf00ad930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:41.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.359+0000 7fccf8600640 1 --2- 172.21.15.90:0/790523571 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fccf00cc150 0x7fccf015b430 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:41.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.359+0000 7fccf8600640 1 -- 172.21.15.90:0/790523571 >> 172.21.15.90:0/790523571 conn(0x7fccf00b1770 msgr2=0x7fccf00739c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:41.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.359+0000 7fccf8600640 1 -- 172.21.15.90:0/790523571 shutdown_connections 2024-03-24T01:38:41.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.359+0000 7fccf8600640 1 -- 172.21.15.90:0/790523571 wait complete. 2024-03-24T01:38:41.380 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-24T01:38:41.380 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T01:38:41.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.584+0000 7fce57a00640 1 Processor -- start 2024-03-24T01:38:41.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.585+0000 7fce57a00640 1 -- start start 2024-03-24T01:38:41.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.586+0000 7fce57a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce500af040 0x7fce5006b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:41.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.586+0000 7fce57a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fce501630f0 0x7fce5015f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:41.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.586+0000 7fce57a00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce50165ed0 con 0x7fce5006b600 2024-03-24T01:38:41.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.586+0000 7fce57a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce5006ba60 con 0x7fce501630f0 2024-03-24T01:38:41.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.586+0000 7fce57a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce501661b0 con 0x7fce500af040 2024-03-24T01:38:41.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.586+0000 7fce55a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fce501630f0 0x7fce5015f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:41.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.587+0000 7fce55a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fce501630f0 0x7fce5015f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55430/0 (socket says 172.21.15.90:55430) 2024-03-24T01:38:41.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.587+0000 7fce55a00640 1 -- 172.21.15.90:0/2614265698 learned_addr learned my addr 172.21.15.90:0/2614265698 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:38:41.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.587+0000 7fce55000640 1 --2- 172.21.15.90:0/2614265698 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce500af040 0x7fce5006b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:41.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.587+0000 7fce55a00640 1 -- 172.21.15.90:0/2614265698 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce500af040 msgr2=0x7fce5006b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:38:41.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.587+0000 7fce55a00640 1 --2- 172.21.15.90:0/2614265698 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce500af040 0x7fce5006b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:41.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.587+0000 7fce55a00640 1 -- 172.21.15.90:0/2614265698 >> v1:172.21.15.120:6789/0 conn(0x7fce5006b600 legacy=0x7fce501629e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T01:38:41.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.587+0000 7fce55a00640 1 -- 172.21.15.90:0/2614265698 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce50166490 con 0x7fce501630f0 2024-03-24T01:38:41.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.588+0000 7fce55a00640 1 --2- 172.21.15.90:0/2614265698 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce501630f0 0x7fce5015f2a0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fce48009e50 tx=0x7fce4802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8840d1398dcf3e6d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:41.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.588+0000 7fce4fe00640 1 -- 172.21.15.90:0/2614265698 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4802ca80 con 0x7fce501630f0 2024-03-24T01:38:41.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.589+0000 7fce4fe00640 1 -- 172.21.15.90:0/2614265698 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce4802cc20 con 0x7fce501630f0 2024-03-24T01:38:41.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.589+0000 7fce4fe00640 1 -- 172.21.15.90:0/2614265698 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce480343f0 con 0x7fce501630f0 2024-03-24T01:38:41.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.589+0000 7fce57a00640 1 -- 172.21.15.90:0/2614265698 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce501630f0 msgr2=0x7fce5015f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:41.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.589+0000 7fce57a00640 1 --2- 172.21.15.90:0/2614265698 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce501630f0 0x7fce5015f2a0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fce48009e50 tx=0x7fce4802bee0 comp rx=0 tx=0).stop 2024-03-24T01:38:41.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.589+0000 7fce57a00640 1 -- 172.21.15.90:0/2614265698 shutdown_connections 2024-03-24T01:38:41.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.589+0000 7fce57a00640 1 --2- 172.21.15.90:0/2614265698 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce501630f0 0x7fce5015f2a0 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:41.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.589+0000 7fce57a00640 1 --2- 172.21.15.90:0/2614265698 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce500af040 0x7fce5006b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:41.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.589+0000 7fce57a00640 1 -- 172.21.15.90:0/2614265698 >> 172.21.15.90:0/2614265698 conn(0x7fce500b1770 msgr2=0x7fce500b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T01:38:41.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.589+0000 7fce57a00640 1 -- 172.21.15.90:0/2614265698 shutdown_connections 2024-03-24T01:38:41.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.590+0000 7fce57a00640 1 -- 172.21.15.90:0/2614265698 wait complete. 2024-03-24T01:38:41.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.591+0000 7fce57a00640 1 Processor -- start 2024-03-24T01:38:41.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.591+0000 7fce57a00640 1 -- start start 2024-03-24T01:38:41.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.592+0000 7fce57a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 0x7fce501f7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:41.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.592+0000 7fce57a00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce500af040 0x7fce50164c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:41.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.592+0000 7fce57a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce501630f0 0x7fce501f51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:41.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.592+0000 7fce57a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5015f960 con 0x7fce501630f0 2024-03-24T01:38:41.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.592+0000 7fce57a00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5015f7e0 con 0x7fce500af040 2024-03-24T01:38:41.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.592+0000 7fce57a00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce5015fae0 con 0x7fce5006b600 2024-03-24T01:38:41.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.593+0000 7fce55a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce501630f0 0x7fce501f51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:41.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.593+0000 7fce55000640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 0x7fce501f7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:41.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.593+0000 7fce55a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce501630f0 0x7fce501f51c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55434/0 (socket says 172.21.15.90:55434) 2024-03-24T01:38:41.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.593+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 learned_addr learned my addr 172.21.15.90:0/1565633459 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-24T01:38:41.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.593+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce500af040 0x7fce50164c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:41.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.593+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 msgr2=0x7fce501f7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:41.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.593+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 0x7fce501f7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:41.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.593+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce500af040 msgr2=0x7fce50164c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T01:38:41.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.593+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce500af040 0x7fce50164c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:41.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.593+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce501f5700 con 0x7fce501630f0 2024-03-24T01:38:41.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.594+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce501630f0 0x7fce501f51c0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fce48036040 tx=0x7fce48004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:41.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.594+0000 7fce4d600640 1 -- 172.21.15.90:0/1565633459 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce48043020 con 0x7fce501630f0 2024-03-24T01:38:41.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.594+0000 7fce4d600640 1 -- 172.21.15.90:0/1565633459 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce4803e450 con 0x7fce501630f0 2024-03-24T01:38:41.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.594+0000 7fce4d600640 1 -- 172.21.15.90:0/1565633459 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce48009510 con 0x7fce501630f0 2024-03-24T01:38:41.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.594+0000 7fce57a00640 1 -- 172.21.15.90:0/1565633459 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce501f8360 con 0x7fce501630f0 2024-03-24T01:38:41.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.594+0000 7fce57a00640 1 -- 172.21.15.90:0/1565633459 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce501f8920 con 0x7fce501630f0 2024-03-24T01:38:41.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.595+0000 7fce57a00640 1 -- 172.21.15.90:0/1565633459 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce50068060 con 0x7fce501630f0 2024-03-24T01:38:41.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.596+0000 7fce4d600640 1 -- 172.21.15.90:0/1565633459 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 90136+0+0 (secure 0 0 0) 0x7fce48046020 con 0x7fce501630f0 2024-03-24T01:38:41.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.597+0000 7fce4d600640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 0x7fce2c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:41.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.597+0000 7fce4d600640 1 -- 172.21.15.90:0/1565633459 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6246+0+0 (secure 0 0 0) 0x7fce480ad3e0 con 0x7fce501630f0 2024-03-24T01:38:41.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.597+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 0x7fce2c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:38:41.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.598+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 0x7fce2c06e090 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fce40006fd0 tx=0x7fce40008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:38:41.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.602+0000 7fce4d600640 1 -- 172.21.15.90:0/1565633459 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+185025 (secure 0 0 0) 0x7fce4804d030 con 0x7fce501630f0 2024-03-24T01:38:41.716 INFO:tasks.ceph.osd.0:Sent signal 15 2024-03-24T01:38:41.717 INFO:tasks.ceph.osd.1:Sent signal 15 2024-03-24T01:38:41.717 INFO:tasks.ceph.osd.2:Sent signal 15 2024-03-24T01:38:41.717 INFO:tasks.ceph.osd.3:Sent signal 15 2024-03-24T01:38:41.717 INFO:tasks.ceph.osd.4:Sent signal 15 2024-03-24T01:38:41.717 INFO:tasks.ceph.osd.5:Sent signal 15 2024-03-24T01:38:41.718 INFO:tasks.ceph.osd.6:Sent signal 15 2024-03-24T01:38:41.718 INFO:tasks.ceph.osd.7:Sent signal 15 2024-03-24T01:38:41.718 INFO:tasks.ceph.mds.c:Sent signal 15 2024-03-24T01:38:41.718 INFO:tasks.ceph.mds.b:Sent signal 15 2024-03-24T01:38:41.718 INFO:tasks.ceph.mds.d:Sent signal 15 2024-03-24T01:38:41.719 INFO:tasks.ceph.mon.a:Sent signal 15 2024-03-24T01:38:41.719 INFO:tasks.ceph.mon.b:Sent signal 15 2024-03-24T01:38:41.719 INFO:tasks.ceph.mon.c:Sent signal 15 2024-03-24T01:38:41.719 INFO:tasks.ceph.mgr.y:Sent signal 15 2024-03-24T01:38:41.719 INFO:tasks.ceph.mgr.x:Sent signal 15 2024-03-24T01:38:41.720 INFO:tasks.ceph.osd.4.smithi120.stderr:2024-03-24T01:38:41.718+0000 7f4d6be00640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37275) UID: 0 2024-03-24T01:38:41.720 INFO:tasks.ceph.osd.4.smithi120.stderr:2024-03-24T01:38:41.718+0000 7f4d6be00640 -1 osd.4 69 *** Got signal Terminated *** 2024-03-24T01:38:41.720 INFO:tasks.ceph.osd.4.smithi120.stderr:2024-03-24T01:38:41.718+0000 7f4d6be00640 -1 osd.4 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T01:38:41.721 INFO:tasks.ceph.osd.5.smithi120.stderr:2024-03-24T01:38:41.718+0000 7f7445e00640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37270) UID: 0 2024-03-24T01:38:41.721 INFO:tasks.ceph.osd.6.smithi120.stderr:2024-03-24T01:38:41.718+0000 7fd67d400640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37273) UID: 0 2024-03-24T01:38:41.721 INFO:tasks.ceph.osd.6.smithi120.stderr:2024-03-24T01:38:41.718+0000 7fd67d400640 -1 osd.6 69 *** Got signal Terminated *** 2024-03-24T01:38:41.721 INFO:tasks.ceph.osd.6.smithi120.stderr:2024-03-24T01:38:41.719+0000 7fd67d400640 -1 osd.6 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T01:38:41.721 INFO:tasks.ceph.osd.5.smithi120.stderr:2024-03-24T01:38:41.718+0000 7f7445e00640 -1 osd.5 69 *** Got signal Terminated *** 2024-03-24T01:38:41.721 INFO:tasks.ceph.osd.5.smithi120.stderr:2024-03-24T01:38:41.718+0000 7f7445e00640 -1 osd.5 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T01:38:41.722 INFO:tasks.ceph.osd.7.smithi120.stderr:2024-03-24T01:38:41.719+0000 7fe729000640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37277) UID: 0 2024-03-24T01:38:41.722 INFO:tasks.ceph.osd.7.smithi120.stderr:2024-03-24T01:38:41.719+0000 7fe729000640 -1 osd.7 69 *** Got signal Terminated *** 2024-03-24T01:38:41.722 INFO:tasks.ceph.osd.7.smithi120.stderr:2024-03-24T01:38:41.719+0000 7fe729000640 -1 osd.7 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T01:38:41.722 INFO:tasks.ceph.mds.b.smithi120.stderr:2024-03-24T01:38:41.719+0000 7f3b91a00640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mds -f --cluster ceph -i b (PID: 44765) UID: 0 2024-03-24T01:38:41.722 INFO:tasks.ceph.mds.b.smithi120.stderr:2024-03-24T01:38:41.719+0000 7f3b91a00640 -1 mds.b *** got signal Terminated *** 2024-03-24T01:38:41.723 INFO:tasks.ceph.mds.d.smithi120.stderr:2024-03-24T01:38:41.719+0000 7ff951800640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mds -f --cluster ceph -i d (PID: 44766) UID: 0 2024-03-24T01:38:41.723 INFO:tasks.ceph.mds.d.smithi120.stderr:2024-03-24T01:38:41.719+0000 7ff951800640 -1 mds.d *** got signal Terminated *** 2024-03-24T01:38:41.723 INFO:tasks.ceph.mon.c.smithi120.stderr:2024-03-24T01:38:41.720+0000 7f973f600640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i c (PID: 36704) UID: 0 2024-03-24T01:38:41.723 INFO:tasks.ceph.mon.c.smithi120.stderr:2024-03-24T01:38:41.720+0000 7f973f600640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-24T01:38:41.724 INFO:tasks.ceph.mon.b.smithi120.stderr:2024-03-24T01:38:41.720+0000 7f574c400640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i b (PID: 36702) UID: 0 2024-03-24T01:38:41.724 INFO:tasks.ceph.mon.b.smithi120.stderr:2024-03-24T01:38:41.720+0000 7f574c400640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2024-03-24T01:38:41.724 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-24T01:38:41.720+0000 7f198ee00640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i x (PID: 36706) UID: 0 2024-03-24T01:38:41.724 INFO:tasks.ceph.mgr.x.smithi120.stderr:2024-03-24T01:38:41.720+0000 7f198ee00640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-24T01:38:41.725 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-03-24T01:38:41.716+0000 7f320e600640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 37189) UID: 0 2024-03-24T01:38:41.725 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-03-24T01:38:41.716+0000 7f320e600640 -1 osd.0 69 *** Got signal Terminated *** 2024-03-24T01:38:41.725 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-03-24T01:38:41.716+0000 7f320e600640 -1 osd.0 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T01:38:41.725 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-03-24T01:38:41.717+0000 7f9ad4400640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 37200) UID: 0 2024-03-24T01:38:41.726 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-03-24T01:38:41.717+0000 7f9ad4400640 -1 osd.1 69 *** Got signal Terminated *** 2024-03-24T01:38:41.726 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-03-24T01:38:41.717+0000 7f9ad4400640 -1 osd.1 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T01:38:41.726 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-03-24T01:38:41.717+0000 7f1005e00640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 37207) UID: 0 2024-03-24T01:38:41.726 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-03-24T01:38:41.717+0000 7f1005e00640 -1 osd.2 69 *** Got signal Terminated *** 2024-03-24T01:38:41.726 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-03-24T01:38:41.717+0000 7f1005e00640 -1 osd.2 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T01:38:41.727 INFO:tasks.ceph.osd.3.smithi090.stderr:2024-03-24T01:38:41.717+0000 7fc7f5e00640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 37204) UID: 0 2024-03-24T01:38:41.727 INFO:tasks.ceph.osd.3.smithi090.stderr:2024-03-24T01:38:41.717+0000 7fc7f5e00640 -1 osd.3 69 *** Got signal Terminated *** 2024-03-24T01:38:41.727 INFO:tasks.ceph.osd.3.smithi090.stderr:2024-03-24T01:38:41.717+0000 7fc7f5e00640 -1 osd.3 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T01:38:41.727 INFO:tasks.ceph.mds.c.smithi090.stderr:2024-03-24T01:38:41.718+0000 7fc58ea00640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mds -f --cluster ceph -i c (PID: 63798) UID: 0 2024-03-24T01:38:41.727 INFO:tasks.ceph.mds.c.smithi090.stderr:2024-03-24T01:38:41.718+0000 7fc58ea00640 -1 mds.c *** got signal Terminated *** 2024-03-24T01:38:41.728 INFO:tasks.ceph.mon.a.smithi090.stderr:2024-03-24T01:38:41.718+0000 7fc6f9200640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 36953) UID: 0 2024-03-24T01:38:41.728 INFO:tasks.ceph.mon.a.smithi090.stderr:2024-03-24T01:38:41.719+0000 7fc6f9200640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-24T01:38:41.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.721+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce501630f0 msgr2=0x7fce501f51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-24T01:38:41.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.721+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce501630f0 msgr2=0x7fce501f51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-24T01:38:41.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.721+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce501630f0 0x7fce501f51c0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fce48036040 tx=0x7fce48004a40 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T01:38:41.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.721+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce501630f0 0x7fce501f51c0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fce48036040 tx=0x7fce48004a40 comp rx=0 tx=0).stop 2024-03-24T01:38:41.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.721+0000 7fce4d600640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce501630f0 msgr2=0x7fce501f51c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-24T01:38:41.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.721+0000 7fce4d600640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce501630f0 0x7fce501f51c0 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:41.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.721+0000 7fce4d600640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 0x7fce2c0764b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:41.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.721+0000 7fce4d600640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce500af040 0x7fce2c078cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:41.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.721+0000 7fce4d600640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 0x7fce2c07bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:41.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.721+0000 7fce4d600640 1 -- 172.21.15.90:0/1565633459 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c07c640 con 0x7fce2c079700 2024-03-24T01:38:41.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.721+0000 7fce4d600640 1 -- 172.21.15.90:0/1565633459 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c07c7c0 con 0x7fce500af040 2024-03-24T01:38:41.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.721+0000 7fce4d600640 1 -- 172.21.15.90:0/1565633459 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c07c990 con 0x7fce5006b600 2024-03-24T01:38:41.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.721+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 msgr2=0x7fce2c07bb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:38:41.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.721+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 0x7fce2c07bb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:38:41.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.721+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 msgr2=0x7fce2c0764b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:38:41.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.721+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce500af040 msgr2=0x7fce2c078cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:38:41.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.722+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 0x7fce2c0764b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:38:41.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.722+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce500af040 0x7fce2c078cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:38:41.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.778+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 msgr2=0x7fce2c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 20 2024-03-24T01:38:41.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.778+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 msgr2=0x7fce2c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-24T01:38:41.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.778+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 0x7fce2c06e090 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fce40006fd0 tx=0x7fce40008040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T01:38:41.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.778+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 0x7fce2c06e090 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fce40006fd0 tx=0x7fce40008040 comp rx=0 tx=0).stop 2024-03-24T01:38:41.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.778+0000 7fce4d600640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 msgr2=0x7fce2c06e090 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-24T01:38:41.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.778+0000 7fce4d600640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 0x7fce2c06e090 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:41.921 INFO:tasks.ceph.mgr.y.smithi090.stderr:daemon-helper: command crashed with signal 15 2024-03-24T01:38:41.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.922+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 msgr2=0x7fce2c07bb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:38:41.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.922+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 0x7fce2c07bb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:38:41.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.923+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce500af040 msgr2=0x7fce2c078cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:38:41.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.923+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce500af040 0x7fce2c078cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:38:41.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.923+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 msgr2=0x7fce2c0764b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:38:41.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:41.923+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 0x7fce2c0764b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:38:42.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:42.323+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 msgr2=0x7fce2c07bb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:38:42.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:42.323+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 0x7fce2c07bb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:38:42.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:42.324+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 msgr2=0x7fce2c0764b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:38:42.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:42.324+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce500af040 msgr2=0x7fce2c078cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:38:42.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:42.324+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 0x7fce2c0764b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:38:42.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:42.324+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce500af040 0x7fce2c078cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:38:42.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:42.597+0000 7fce46a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 0x7fce501f51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:42.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:42.598+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 msgr2=0x7fce501f51c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:6832/860392634 2024-03-24T01:38:42.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:42.598+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 0x7fce501f51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:38:42.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:42.799+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 msgr2=0x7fce501f51c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:6832/860392634 2024-03-24T01:38:42.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:42.799+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 0x7fce501f51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:38:43.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:43.124+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 msgr2=0x7fce2c07bb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:38:43.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:43.124+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 0x7fce2c07bb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:38:43.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:43.125+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 msgr2=0x7fce2c0764b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:38:43.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:43.125+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 0x7fce2c0764b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:38:43.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:43.125+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce500af040 msgr2=0x7fce2c078cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:38:43.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:43.126+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce500af040 0x7fce2c078cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:38:43.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:43.200+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 msgr2=0x7fce501f51c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:6832/860392634 2024-03-24T01:38:43.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:43.200+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 0x7fce501f51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:38:44.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:44.002+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 msgr2=0x7fce501f51c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:6832/860392634 2024-03-24T01:38:44.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:44.002+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 0x7fce501f51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:38:44.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:44.593+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 msgr2=0x7fce2c0764b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T01:38:44.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:44.593+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 0x7fce2c0764b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:44.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:44.593+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce500af040 msgr2=0x7fce2c078cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T01:38:44.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:44.593+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce500af040 0x7fce2c078cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:44.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:44.593+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 msgr2=0x7fce2c07bb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T01:38:44.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:44.593+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 0x7fce2c07bb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:44.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:44.593+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 0x7fce1c005620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:44.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:44.593+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce500af040 0x7fce1c007c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:44.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:44.593+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 0x7fce1c00a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:44.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:44.593+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c07c640 con 0x7fce2c079700 2024-03-24T01:38:44.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:44.593+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c07c7c0 con 0x7fce500af040 2024-03-24T01:38:44.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:44.593+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c07c990 con 0x7fce5006b600 2024-03-24T01:38:44.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:44.593+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 msgr2=0x7fce1c005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:38:44.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:44.593+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 0x7fce1c005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:38:44.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:44.593+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce500af040 msgr2=0x7fce1c007c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:38:44.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:44.593+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 msgr2=0x7fce1c00a350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:38:44.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:44.593+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce500af040 0x7fce1c007c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:38:44.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:44.593+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 0x7fce1c00a350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:38:44.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:44.794+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 msgr2=0x7fce1c00a350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:38:44.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:44.794+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 msgr2=0x7fce1c005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:38:44.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:44.794+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce500af040 msgr2=0x7fce1c007c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:38:44.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:44.794+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 0x7fce1c00a350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:38:44.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:44.794+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 0x7fce1c005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:38:44.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:44.794+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce500af040 0x7fce1c007c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:38:45.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:45.195+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 msgr2=0x7fce1c00a350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:38:45.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:45.195+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce500af040 msgr2=0x7fce1c007c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:38:45.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:45.195+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 0x7fce1c00a350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:38:45.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:45.195+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 msgr2=0x7fce1c005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:38:45.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:45.195+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce500af040 0x7fce1c007c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:38:45.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:45.195+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 0x7fce1c005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:38:45.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:45.603+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 msgr2=0x7fce501f51c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:6832/860392634 2024-03-24T01:38:45.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:45.603+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 0x7fce501f51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T01:38:45.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:45.996+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 msgr2=0x7fce1c00a350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:38:45.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:45.996+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 0x7fce1c00a350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:38:45.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:45.996+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 msgr2=0x7fce1c005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:38:45.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:45.996+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce500af040 msgr2=0x7fce1c007c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:38:45.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:45.997+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 0x7fce1c005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:38:45.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:45.997+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce500af040 0x7fce1c007c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:38:47.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:47.599+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 msgr2=0x7fce1c00a350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:38:47.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:47.599+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 0x7fce1c00a350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T01:38:47.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:47.599+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 msgr2=0x7fce1c005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:38:47.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:47.599+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce500af040 msgr2=0x7fce1c007c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:38:47.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:47.599+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 0x7fce1c005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T01:38:47.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:47.599+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce500af040 0x7fce1c007c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T01:38:48.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:48.804+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 msgr2=0x7fce501f51c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:6832/860392634 2024-03-24T01:38:48.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:48.804+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 0x7fce501f51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-24T01:38:50.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:50.802+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 msgr2=0x7fce1c00a350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:38:50.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:50.802+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 0x7fce1c00a350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-24T01:38:50.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:50.803+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce500af040 msgr2=0x7fce1c007c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:38:50.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:50.803+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 msgr2=0x7fce1c005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:38:50.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:50.803+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce500af040 0x7fce1c007c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-24T01:38:50.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:50.803+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 0x7fce1c005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-24T01:38:51.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:51.344+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 msgr2=0x7fce1c00a350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T01:38:51.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:51.344+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 0x7fce1c00a350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:51.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:51.344+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce500af040 msgr2=0x7fce1c007c30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T01:38:51.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:51.344+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce500af040 0x7fce1c007c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:51.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:51.344+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 msgr2=0x7fce1c005620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T01:38:51.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:51.344+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 0x7fce1c005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:38:51.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:51.344+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 0x7fce1c00d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:51.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:51.344+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 0x7fce1c0081f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:51.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:51.344+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 0x7fce1c003610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:38:51.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:51.344+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c07c640 con 0x7fce2c079700 2024-03-24T01:38:51.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:51.344+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c07c7c0 con 0x7fce1c00ad10 2024-03-24T01:38:51.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:51.344+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c07c990 con 0x7fce5006b600 2024-03-24T01:38:51.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:51.344+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 msgr2=0x7fce1c0081f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:38:51.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:51.344+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 msgr2=0x7fce1c00d100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:38:51.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:51.344+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:38:51.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:51.344+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 0x7fce1c0081f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:38:51.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:51.344+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 0x7fce1c00d100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:38:51.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:51.344+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:38:51.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:51.545+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:38:51.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:51.545+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:38:51.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:51.545+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 msgr2=0x7fce1c0081f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:38:51.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:51.545+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 msgr2=0x7fce1c00d100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:38:51.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:51.545+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 0x7fce1c0081f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:38:51.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:51.545+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 0x7fce1c00d100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:38:51.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:51.946+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:38:51.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:51.946+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:38:51.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:51.947+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 msgr2=0x7fce1c0081f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:38:51.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:51.947+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 msgr2=0x7fce1c00d100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:38:51.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:51.947+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 0x7fce1c0081f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:38:51.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:51.947+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 0x7fce1c00d100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:38:52.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:52.747+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:38:52.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:52.747+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:38:52.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:52.748+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 msgr2=0x7fce1c0081f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:38:52.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:52.748+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 msgr2=0x7fce1c00d100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:38:52.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:52.748+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 0x7fce1c0081f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:38:52.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:52.748+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 0x7fce1c00d100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:38:54.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:54.349+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:38:54.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:54.349+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T01:38:54.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:54.350+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 msgr2=0x7fce1c0081f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:38:54.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:54.350+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 0x7fce1c0081f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T01:38:54.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:54.350+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 msgr2=0x7fce1c00d100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:38:54.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:54.350+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 0x7fce1c00d100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T01:38:55.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:55.206+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 msgr2=0x7fce501f51c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:6832/860392634 2024-03-24T01:38:55.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:55.206+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 0x7fce501f51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-03-24T01:38:57.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:57.551+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:38:57.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:57.552+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-24T01:38:57.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:57.554+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 msgr2=0x7fce1c00d100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:38:57.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:57.554+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 msgr2=0x7fce1c0081f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:38:57.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:57.554+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 0x7fce1c00d100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-24T01:38:57.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:38:57.554+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 0x7fce1c0081f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-24T01:39:01.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:01.469+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 msgr2=0x7fce1c0081f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T01:39:01.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:01.469+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce5006b600 0x7fce1c0081f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:39:01.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:01.469+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 msgr2=0x7fce1c00d100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T01:39:01.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:01.469+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 0x7fce1c00d100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:39:01.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:01.469+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T01:39:01.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:01.469+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:39:01.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:01.470+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:39:01.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:01.470+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 0x7fce1c00a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:39:01.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:01.470+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 0x7fce1c003610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:39:01.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:01.470+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c07c640 con 0x7fce2c079700 2024-03-24T01:39:01.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:01.470+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c07c7c0 con 0x7fce1c00ad10 2024-03-24T01:39:01.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:01.470+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c07c990 con 0x7fce1c005cc0 2024-03-24T01:39:01.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:01.470+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 msgr2=0x7fce1c00a200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:39:01.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:01.470+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:39:01.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:01.470+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 0x7fce1c00a200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:39:01.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:01.470+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:39:01.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:01.470+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:39:01.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:01.470+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:39:01.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:01.671+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 msgr2=0x7fce1c00a200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:39:01.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:01.671+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:39:01.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:01.671+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:39:01.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:01.671+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 0x7fce1c00a200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:39:01.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:01.671+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:39:01.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:01.671+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:39:02.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:02.072+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 msgr2=0x7fce1c00a200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:39:02.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:02.072+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:39:02.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:02.072+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:39:02.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:02.072+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 0x7fce1c00a200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:39:02.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:02.072+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:39:02.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:02.072+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:39:02.103 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:39:02.110 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:39:02.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:02.873+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 msgr2=0x7fce1c00a200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:39:02.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:02.873+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:39:02.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:02.873+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:39:02.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:02.873+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 0x7fce1c00a200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:39:02.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:02.873+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:39:02.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:02.873+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:39:04.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:04.475+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 msgr2=0x7fce1c00a200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:39:04.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:04.476+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:39:04.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:04.476+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 0x7fce1c00a200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T01:39:04.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:04.476+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:39:04.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:04.476+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T01:39:04.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:04.476+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T01:39:07.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:07.679+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 msgr2=0x7fce1c00a200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:39:07.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:07.679+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:39:07.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:07.679+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:39:07.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:07.679+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 0x7fce1c00a200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-24T01:39:07.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:07.680+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-24T01:39:07.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:07.680+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-24T01:39:08.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:08.007+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 msgr2=0x7fce501f51c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:6832/860392634 2024-03-24T01:39:08.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:08.007+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 0x7fce501f51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-24T01:39:14.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:14.086+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 msgr2=0x7fce1c00a200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:39:14.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:14.086+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 0x7fce1c00a200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-03-24T01:39:14.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:14.087+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:39:14.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:14.087+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:39:14.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:14.087+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-03-24T01:39:14.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:14.087+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-03-24T01:39:16.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:16.658+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T01:39:16.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:16.658+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:39:16.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:16.658+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T01:39:16.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:16.658+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:39:16.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:16.658+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 msgr2=0x7fce1c00a200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T01:39:16.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:16.658+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce2c079700 0x7fce1c00a200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:39:16.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:16.658+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c006880 0x7fce1c006c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:39:16.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:16.658+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 0x7fce1c003610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:39:16.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:16.658+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:39:16.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:16.658+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c07c640 con 0x7fce1c006880 2024-03-24T01:39:16.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:16.658+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c07c7c0 con 0x7fce1c00ad10 2024-03-24T01:39:16.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:16.658+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c07c990 con 0x7fce1c005cc0 2024-03-24T01:39:16.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:16.658+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c006880 msgr2=0x7fce1c006c60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:39:16.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:16.658+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:39:16.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:16.658+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:39:16.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:16.658+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c006880 0x7fce1c006c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:39:16.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:16.658+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:39:16.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:16.658+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:39:16.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:16.859+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c006880 msgr2=0x7fce1c006c60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:39:16.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:16.859+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:39:16.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:16.859+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:39:16.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:16.859+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c006880 0x7fce1c006c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:39:16.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:16.859+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:39:16.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:16.859+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:39:17.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:17.260+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:39:17.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:17.260+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c006880 msgr2=0x7fce1c006c60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:39:17.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:17.260+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:39:17.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:17.260+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:39:17.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:17.260+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c006880 0x7fce1c006c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:39:17.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:17.260+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:39:18.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:18.061+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:39:18.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:18.061+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c006880 msgr2=0x7fce1c006c60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:39:18.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:18.061+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:39:18.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:18.061+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:39:18.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:18.061+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:39:18.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:18.061+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c006880 0x7fce1c006c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:39:19.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:19.663+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:39:19.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:19.663+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:39:19.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:19.663+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c006880 msgr2=0x7fce1c006c60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:39:19.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:19.663+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T01:39:19.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:19.663+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T01:39:19.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:19.664+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c006880 0x7fce1c006c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T01:39:22.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:22.867+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c006880 msgr2=0x7fce1c006c60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:39:22.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:22.867+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:39:22.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:22.867+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:39:22.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:22.867+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c006880 0x7fce1c006c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-24T01:39:22.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:22.867+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-24T01:39:22.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:22.867+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-24T01:39:23.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:23.008+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 msgr2=0x7fce501f51c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:6832/860392634 2024-03-24T01:39:23.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:23.008+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 0x7fce501f51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-24T01:39:29.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:29.274+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:39:29.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:29.274+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-03-24T01:39:29.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:29.274+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c006880 msgr2=0x7fce1c006c60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:39:29.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:29.274+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c006880 0x7fce1c006c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-03-24T01:39:29.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:29.274+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:39:29.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:29.274+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-03-24T01:39:32.248 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:39:32.256 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:39:38.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:38.017+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 msgr2=0x7fce501f51c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:6832/860392634 2024-03-24T01:39:38.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:38.017+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 0x7fce501f51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-24T01:39:39.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:39.440+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T01:39:39.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:39.440+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:39:39.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:39.440+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T01:39:39.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:39.440+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c00ad10 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:39:39.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:39.440+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c006880 msgr2=0x7fce1c006c60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T01:39:39.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:39.440+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c006880 0x7fce1c006c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:39:39.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:39.440+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 0x7fce1c00a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:39:39.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:39.440+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c0076d0 0x7fce1c011120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:39:39.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:39.440+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:39:39.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:39.440+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c07c640 con 0x7fce1c0072f0 2024-03-24T01:39:39.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:39.440+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c07c7c0 con 0x7fce1c0076d0 2024-03-24T01:39:39.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:39.440+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c07c990 con 0x7fce1c005cc0 2024-03-24T01:39:39.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:39.440+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 msgr2=0x7fce1c00a1e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:39:39.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:39.440+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:39:39.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:39.440+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c0076d0 msgr2=0x7fce1c011120 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:39:39.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:39.440+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 0x7fce1c00a1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:39:39.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:39.440+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:39:39.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:39.440+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c0076d0 0x7fce1c011120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:39:39.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:39.641+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:39:39.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:39.641+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 msgr2=0x7fce1c00a1e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:39:39.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:39.641+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c0076d0 msgr2=0x7fce1c011120 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:39:39.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:39.641+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:39:39.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:39.641+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c0076d0 0x7fce1c011120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:39:39.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:39.641+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 0x7fce1c00a1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:39:40.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:40.042+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:39:40.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:40.042+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c0076d0 msgr2=0x7fce1c011120 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:39:40.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:40.042+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 msgr2=0x7fce1c00a1e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:39:40.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:40.042+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:39:40.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:40.042+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c0076d0 0x7fce1c011120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:39:40.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:40.042+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 0x7fce1c00a1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:39:40.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:40.843+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:39:40.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:40.843+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 msgr2=0x7fce1c00a1e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:39:40.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:40.843+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c0076d0 msgr2=0x7fce1c011120 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:39:40.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:40.843+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:39:40.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:40.843+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c0076d0 0x7fce1c011120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:39:40.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:40.843+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 0x7fce1c00a1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:39:42.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:42.445+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:39:42.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:42.445+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 msgr2=0x7fce1c00a1e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:39:42.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:42.445+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T01:39:42.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:42.445+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c0076d0 msgr2=0x7fce1c011120 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:39:42.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:42.445+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c0076d0 0x7fce1c011120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T01:39:42.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:42.445+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 0x7fce1c00a1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T01:39:45.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:45.648+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:39:45.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:45.648+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 msgr2=0x7fce1c00a1e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:39:45.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:45.648+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c0076d0 msgr2=0x7fce1c011120 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:39:45.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:45.648+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-24T01:39:45.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:45.648+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c0076d0 0x7fce1c011120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-24T01:39:45.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:45.648+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 0x7fce1c00a1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-24T01:39:52.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:52.055+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:39:52.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:52.055+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c0076d0 msgr2=0x7fce1c011120 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:39:52.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:52.055+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 msgr2=0x7fce1c00a1e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:39:52.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:52.055+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-03-24T01:39:52.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:52.055+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 0x7fce1c00a1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-03-24T01:39:52.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:52.055+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c0076d0 0x7fce1c011120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-03-24T01:39:53.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:53.018+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 msgr2=0x7fce501f51c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:6832/860392634 2024-03-24T01:39:53.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:39:53.018+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 0x7fce501f51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-24T01:40:02.392 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:40:02.401 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:40:04.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:04.868+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:40:04.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:04.868+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-24T01:40:04.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:04.869+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 msgr2=0x7fce1c00a1e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:40:04.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:04.869+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c0076d0 msgr2=0x7fce1c011120 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:40:04.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:04.869+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 0x7fce1c00a1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-24T01:40:04.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:04.869+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c0076d0 0x7fce1c011120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-24T01:40:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:08.022+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 msgr2=0x7fce501f51c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:6832/860392634 2024-03-24T01:40:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:08.022+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 0x7fce501f51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-24T01:40:09.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:09.440+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T01:40:09.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:09.440+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:40:09.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:09.440+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c0076d0 msgr2=0x7fce1c011120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T01:40:09.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:09.440+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c0076d0 0x7fce1c011120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:40:09.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:09.440+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 msgr2=0x7fce1c00a1e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T01:40:09.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:09.440+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 0x7fce1c00a1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:40:09.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:09.440+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c006ce0 0x7fce1c003610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:40:09.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:09.440+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:40:09.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:09.440+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 0x7fce1c0081d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:40:09.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:09.440+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c07c640 con 0x7fce1c0072f0 2024-03-24T01:40:09.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:09.440+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c07c7c0 con 0x7fce1c006ce0 2024-03-24T01:40:09.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:09.440+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c07c990 con 0x7fce1c005cc0 2024-03-24T01:40:09.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:09.440+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 msgr2=0x7fce1c0081d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:40:09.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:09.440+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 0x7fce1c0081d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:40:09.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:09.440+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:40:09.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:09.440+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c006ce0 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:40:09.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:09.440+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:40:09.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:09.440+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c006ce0 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:40:09.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:09.641+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 msgr2=0x7fce1c0081d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:40:09.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:09.641+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 0x7fce1c0081d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:40:09.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:09.641+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:40:09.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:09.641+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c006ce0 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:40:09.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:09.641+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:40:09.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:09.641+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c006ce0 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:40:10.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:10.042+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 msgr2=0x7fce1c0081d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:40:10.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:10.042+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 0x7fce1c0081d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:40:10.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:10.042+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c006ce0 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:40:10.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:10.042+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:40:10.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:10.042+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c006ce0 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:40:10.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:10.042+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:40:10.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:10.843+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 msgr2=0x7fce1c0081d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:40:10.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:10.843+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 0x7fce1c0081d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:40:10.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:10.844+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c006ce0 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:40:10.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:10.844+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:40:10.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:10.844+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c006ce0 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:40:10.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:10.844+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:40:12.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:12.445+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 msgr2=0x7fce1c0081d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:40:12.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:12.445+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 0x7fce1c0081d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T01:40:12.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:12.446+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:40:12.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:12.446+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c006ce0 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:40:12.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:12.446+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T01:40:12.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:12.446+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c006ce0 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T01:40:15.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:15.648+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 msgr2=0x7fce1c0081d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:40:15.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:15.648+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 0x7fce1c0081d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-24T01:40:15.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:15.649+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:40:15.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:15.649+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-24T01:40:15.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:15.649+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c006ce0 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:40:15.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:15.650+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c006ce0 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-24T01:40:22.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:22.055+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 msgr2=0x7fce1c0081d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:40:22.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:22.055+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 0x7fce1c0081d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-03-24T01:40:22.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:22.056+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:40:22.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:22.056+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-03-24T01:40:22.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:22.056+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c006ce0 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:40:22.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:22.057+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c006ce0 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-03-24T01:40:23.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:23.023+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 msgr2=0x7fce501f51c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:6832/860392634 2024-03-24T01:40:23.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:23.024+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 0x7fce501f51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-24T01:40:32.555 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:40:32.563 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:40:34.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:34.867+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 msgr2=0x7fce1c0081d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:40:34.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:34.867+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 0x7fce1c0081d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-24T01:40:34.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:34.870+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:40:34.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:34.870+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c006ce0 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:40:34.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:34.870+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-24T01:40:34.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:34.870+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c006ce0 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-24T01:40:38.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:38.028+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 msgr2=0x7fce501f51c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:6832/860392634 2024-03-24T01:40:38.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:38.028+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:6832/860392634,v1:172.21.15.120:6833/860392634] conn(0x7fce2c06bbd0 0x7fce501f51c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-24T01:40:39.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:39.441+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T01:40:39.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:39.441+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:40:39.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:39.441+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c006ce0 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T01:40:39.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:39.441+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c006ce0 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:40:39.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:39.441+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 msgr2=0x7fce1c0081d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T01:40:39.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:39.441+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 0x7fce1c0081d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:40:39.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:39.441+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c006ce0 0x7fce1c003610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:40:39.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:39.441+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:40:39.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:39.441+0000 7fce47400640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 0x7fce1c0081d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:40:39.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:39.441+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c07c640 con 0x7fce1c0072f0 2024-03-24T01:40:39.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:39.441+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c07c7c0 con 0x7fce1c006ce0 2024-03-24T01:40:39.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:39.441+0000 7fce47400640 1 -- 172.21.15.90:0/1565633459 --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c07c990 con 0x7fce1c005cc0 2024-03-24T01:40:39.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:39.441+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:40:39.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:39.441+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c006ce0 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:40:39.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:39.441+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:40:39.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:39.441+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c006ce0 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:40:39.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:39.441+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 msgr2=0x7fce1c0081d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:40:39.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:39.441+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 0x7fce1c0081d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:40:39.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:39.642+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 msgr2=0x7fce1c0081d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:40:39.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:39.642+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c006ce0 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:40:39.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:39.642+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:40:39.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:39.642+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 0x7fce1c0081d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:40:39.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:39.642+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c006ce0 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:40:39.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:39.642+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:40:40.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:40.043+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 msgr2=0x7fce1c0081d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:40:40.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:40.043+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 0x7fce1c0081d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:40:40.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:40.043+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:40:40.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:40.043+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c006ce0 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:40:40.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:40.043+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:40:40.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:40.043+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c006ce0 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:40:40.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:40.844+0000 7fce55000640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 msgr2=0x7fce1c0081d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:40:40.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:40.844+0000 7fce55000640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce1c0072f0 0x7fce1c0081d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:40:40.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:40.844+0000 7fce47e00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c006ce0 msgr2=0x7fce1c003610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:40:40.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:40.844+0000 7fce55a00640 1 -- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 msgr2=0x7fce1c0060a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:40:40.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:40.844+0000 7fce47e00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce1c006ce0 0x7fce1c003610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:40:40.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:40.844+0000 7fce55a00640 1 --2- 172.21.15.90:0/1565633459 >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7fce1c005cc0 0x7fce1c0060a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:40:41.493 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-24T01:40:41.495 INFO:tasks.cephfs_test_runner:That after forcibly modifying the backing store, we can get back into ... ERROR 2024-03-24T01:40:41.497 INFO:tasks.cephfs_test_runner:ERROR 2024-03-24T01:40:41.498 INFO:tasks.cephfs_test_runner: 2024-03-24T01:40:41.498 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-24T01:40:41.498 INFO:tasks.cephfs_test_runner:ERROR: test_reset (tasks.cephfs.test_journal_repair.TestJournalRepair) 2024-03-24T01:40:41.498 INFO:tasks.cephfs_test_runner:That after forcibly modifying the backing store, we can get back into 2024-03-24T01:40:41.498 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-24T01:40:41.498 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-24T01:40:41.498 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/test_journal_repair.py", line 177, in test_reset 2024-03-24T01:40:41.498 INFO:tasks.cephfs_test_runner: self.mount_a.create_n_files("bravo/file", 1000) 2024-03-24T01:40:41.499 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/mount.py", line 1368, in create_n_files 2024-03-24T01:40:41.499 INFO:tasks.cephfs_test_runner: self.run_python(pyscript) 2024-03-24T01:40:41.499 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/mount.py", line 776, in run_python 2024-03-24T01:40:41.499 INFO:tasks.cephfs_test_runner: p.wait() 2024-03-24T01:40:41.499 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait 2024-03-24T01:40:41.499 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-03-24T01:40:41.499 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-03-24T01:40:41.499 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-03-24T01:40:41.499 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 1: 'stdin-killer -- python3 -c \'\nimport os\nimport uuid\n\nn = 1000\ncreate_hard_links = False\nif 0 > 0:\n create_hard_links = True\npath = "/home/ubuntu/cephtest/mnt.0/bravo/file"\n\ndpath = os.path.dirname(path)\nfnameprefix = os.path.basename(path)\nos.makedirs(dpath, exist_ok=True)\n\ntry:\n dirfd = os.open(dpath, os.O_DIRECTORY)\n\n for i in range(n):\n fpath = os.path.join(dpath, f"{fnameprefix}_{i}")\n with open(fpath, \'"\'"\'w\'"\'"\') as f:\n f.write(f"{i}")\n if False:\n f.flush()\n os.fsync(f.fileno())\n if False:\n os.unlink(fpath)\n if False:\n os.fsync(dirfd)\n if create_hard_links:\n for j in range(0):\n os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") \n if False:\n os.fsync(dirfd)\nfinally:\n os.close(dirfd)\n\'' 2024-03-24T01:40:41.499 INFO:tasks.cephfs_test_runner: 2024-03-24T01:40:41.499 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-24T01:40:41.499 INFO:tasks.cephfs_test_runner:ERROR: test_reset (tasks.cephfs.test_journal_repair.TestJournalRepair) 2024-03-24T01:40:41.499 INFO:tasks.cephfs_test_runner:That after forcibly modifying the backing store, we can get back into 2024-03-24T01:40:41.499 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-24T01:40:41.500 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-24T01:40:41.500 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/cephfs_test_case.py", line 208, in tearDown 2024-03-24T01:40:41.500 INFO:tasks.cephfs_test_runner: self.mds_cluster.delete_all_filesystems() 2024-03-24T01:40:41.500 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 500, in delete_all_filesystems 2024-03-24T01:40:41.500 INFO:tasks.cephfs_test_runner: Filesystem(ctx=self._ctx, fscid=fs['id']).destroy() 2024-03-24T01:40:41.500 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 855, in destroy 2024-03-24T01:40:41.500 INFO:tasks.cephfs_test_runner: if self.dead(): 2024-03-24T01:40:41.500 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 563, in dead 2024-03-24T01:40:41.500 INFO:tasks.cephfs_test_runner: return not bool(self.get_mds_map()) 2024-03-24T01:40:41.500 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 927, in get_mds_map 2024-03-24T01:40:41.500 INFO:tasks.cephfs_test_runner: status = self.status() 2024-03-24T01:40:41.500 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 402, in status 2024-03-24T01:40:41.500 INFO:tasks.cephfs_test_runner: return FSStatus(self.mon_manager, epoch) 2024-03-24T01:40:41.500 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 88, in __init__ 2024-03-24T01:40:41.500 INFO:tasks.cephfs_test_runner: self.map = json.loads(self.get_ceph_cmd_stdout(*cmd)) 2024-03-24T01:40:41.500 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_test_case.py", line 55, in get_ceph_cmd_stdout 2024-03-24T01:40:41.501 INFO:tasks.cephfs_test_runner: return self.run_ceph_cmd(**kwargs).stdout.getvalue() 2024-03-24T01:40:41.501 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd 2024-03-24T01:40:41.501 INFO:tasks.cephfs_test_runner: return self.mon_manager.run_cluster_cmd(**kwargs) 2024-03-24T01:40:41.501 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd 2024-03-24T01:40:41.501 INFO:tasks.cephfs_test_runner: return self.controller.run(**kwargs) 2024-03-24T01:40:41.501 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run 2024-03-24T01:40:41.501 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-03-24T01:40:41.501 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run 2024-03-24T01:40:41.501 INFO:tasks.cephfs_test_runner: r.wait() 2024-03-24T01:40:41.501 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait 2024-03-24T01:40:41.501 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-03-24T01:40:41.501 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-03-24T01:40:41.501 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-03-24T01:40:41.501 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json' 2024-03-24T01:40:41.501 INFO:tasks.cephfs_test_runner: 2024-03-24T01:40:41.502 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-24T01:40:41.502 INFO:tasks.cephfs_test_runner:Ran 3 tests in 927.518s 2024-03-24T01:40:41.502 INFO:tasks.cephfs_test_runner: 2024-03-24T01:40:41.502 INFO:tasks.cephfs_test_runner:FAILED (errors=2) 2024-03-24T01:40:41.502 INFO:tasks.cephfs_test_runner: 2024-03-24T01:40:41.502 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-24T01:40:41.502 INFO:tasks.cephfs_test_runner:ERROR: test_reset (tasks.cephfs.test_journal_repair.TestJournalRepair) 2024-03-24T01:40:41.502 INFO:tasks.cephfs_test_runner:That after forcibly modifying the backing store, we can get back into 2024-03-24T01:40:41.502 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-24T01:40:41.502 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-24T01:40:41.502 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/test_journal_repair.py", line 177, in test_reset 2024-03-24T01:40:41.502 INFO:tasks.cephfs_test_runner: self.mount_a.create_n_files("bravo/file", 1000) 2024-03-24T01:40:41.503 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/mount.py", line 1368, in create_n_files 2024-03-24T01:40:41.503 INFO:tasks.cephfs_test_runner: self.run_python(pyscript) 2024-03-24T01:40:41.503 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/mount.py", line 776, in run_python 2024-03-24T01:40:41.503 INFO:tasks.cephfs_test_runner: p.wait() 2024-03-24T01:40:41.503 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait 2024-03-24T01:40:41.503 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-03-24T01:40:41.503 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-03-24T01:40:41.503 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-03-24T01:40:41.503 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 1: 'stdin-killer -- python3 -c \'\nimport os\nimport uuid\n\nn = 1000\ncreate_hard_links = False\nif 0 > 0:\n create_hard_links = True\npath = "/home/ubuntu/cephtest/mnt.0/bravo/file"\n\ndpath = os.path.dirname(path)\nfnameprefix = os.path.basename(path)\nos.makedirs(dpath, exist_ok=True)\n\ntry:\n dirfd = os.open(dpath, os.O_DIRECTORY)\n\n for i in range(n):\n fpath = os.path.join(dpath, f"{fnameprefix}_{i}")\n with open(fpath, \'"\'"\'w\'"\'"\') as f:\n f.write(f"{i}")\n if False:\n f.flush()\n os.fsync(f.fileno())\n if False:\n os.unlink(fpath)\n if False:\n os.fsync(dirfd)\n if create_hard_links:\n for j in range(0):\n os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") \n if False:\n os.fsync(dirfd)\nfinally:\n os.close(dirfd)\n\'' 2024-03-24T01:40:41.503 INFO:tasks.cephfs_test_runner: 2024-03-24T01:40:41.503 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-24T01:40:41.503 INFO:tasks.cephfs_test_runner:ERROR: test_reset (tasks.cephfs.test_journal_repair.TestJournalRepair) 2024-03-24T01:40:41.503 INFO:tasks.cephfs_test_runner:That after forcibly modifying the backing store, we can get back into 2024-03-24T01:40:41.503 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-24T01:40:41.504 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-24T01:40:41.504 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/cephfs_test_case.py", line 208, in tearDown 2024-03-24T01:40:41.504 INFO:tasks.cephfs_test_runner: self.mds_cluster.delete_all_filesystems() 2024-03-24T01:40:41.504 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 500, in delete_all_filesystems 2024-03-24T01:40:41.504 INFO:tasks.cephfs_test_runner: Filesystem(ctx=self._ctx, fscid=fs['id']).destroy() 2024-03-24T01:40:41.504 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 855, in destroy 2024-03-24T01:40:41.504 INFO:tasks.cephfs_test_runner: if self.dead(): 2024-03-24T01:40:41.504 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 563, in dead 2024-03-24T01:40:41.504 INFO:tasks.cephfs_test_runner: return not bool(self.get_mds_map()) 2024-03-24T01:40:41.504 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 927, in get_mds_map 2024-03-24T01:40:41.504 INFO:tasks.cephfs_test_runner: status = self.status() 2024-03-24T01:40:41.504 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 402, in status 2024-03-24T01:40:41.504 INFO:tasks.cephfs_test_runner: return FSStatus(self.mon_manager, epoch) 2024-03-24T01:40:41.504 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 88, in __init__ 2024-03-24T01:40:41.504 INFO:tasks.cephfs_test_runner: self.map = json.loads(self.get_ceph_cmd_stdout(*cmd)) 2024-03-24T01:40:41.504 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_test_case.py", line 55, in get_ceph_cmd_stdout 2024-03-24T01:40:41.505 INFO:tasks.cephfs_test_runner: return self.run_ceph_cmd(**kwargs).stdout.getvalue() 2024-03-24T01:40:41.505 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd 2024-03-24T01:40:41.505 INFO:tasks.cephfs_test_runner: return self.mon_manager.run_cluster_cmd(**kwargs) 2024-03-24T01:40:41.505 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd 2024-03-24T01:40:41.505 INFO:tasks.cephfs_test_runner: return self.controller.run(**kwargs) 2024-03-24T01:40:41.505 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run 2024-03-24T01:40:41.505 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-03-24T01:40:41.505 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run 2024-03-24T01:40:41.505 INFO:tasks.cephfs_test_runner: r.wait() 2024-03-24T01:40:41.505 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait 2024-03-24T01:40:41.505 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-03-24T01:40:41.505 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-03-24T01:40:41.505 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-03-24T01:40:41.505 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json' 2024-03-24T01:40:41.505 INFO:tasks.cephfs_test_runner: 2024-03-24T01:40:41.506 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_reset (tasks.cephfs.test_journal_repair.TestJournalRepair), test_reset (tasks.cephfs.test_journal_repair.TestJournalRepair) 2024-03-24T01:40:41.737 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=5a14705d930742f8a51d83f32eb23f2b Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_reset (tasks.cephfs.test_journal_repair.TestJournalRepair), test_reset (tasks.cephfs.test_journal_repair.TestJournalRepair) 2024-03-24T01:40:41.739 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-03-24T01:40:41.749 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-24T01:40:41.832 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-24T01:40:41.832 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:40:41.832 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:40:41.893 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053414,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498340k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249168k,nr_inodes=812292,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498340k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T01:40:41.893 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-24T01:40:41.893 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:40:41.946 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053414,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498340k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249168k,nr_inodes=812292,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498340k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T01:40:41.946 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:40:41.946 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:40:42.003 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T01:40:42.003 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-24T01:40:42.004 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T01:40:42.056 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053418,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249172k,nr_inodes=812293,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498348k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T01:40:42.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:40:42.057 DEBUG:teuthology.orchestra.run.smithi120:> ip netns list 2024-03-24T01:40:42.109 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-24T01:40:42.109 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-03-24T01:40:42.110 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:40:42.111 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T01:40:42.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:40:42.199 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-24T01:40:42.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:40:42.285 DEBUG:teuthology.orchestra.run.smithi120:> sudo ip link delete ceph-brx 2024-03-24T01:40:42.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:40:42.398 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-03-24T01:40:42.452 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T01:40:42.452 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T01:40:42.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:40:42.454 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T01:40:42.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:40:42.540 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T01:40:42.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T01:40:42.627 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link delete ceph-brx 2024-03-24T01:40:42.736 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-24T01:40:42.746 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-24T01:40:42.747 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T01:40:42.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:42.932+0000 7f62a9e00640 1 Processor -- start 2024-03-24T01:40:42.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:42.933+0000 7f62a9e00640 1 -- start start 2024-03-24T01:40:42.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:42.934+0000 7f62a9e00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62a41603e0 0x7f62a406b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:40:42.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:42.934+0000 7f62a9e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a40cd490 con 0x7f62a4145430 2024-03-24T01:40:42.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:42.934+0000 7f62a9e00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a415f900 con 0x7f62a40cc150 2024-03-24T01:40:42.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:42.934+0000 7f62a9e00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f62a41603e0 2024-03-24T01:40:42.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:42.934+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62a4145430 legacy=0x7f62a40b3750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:40:42.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:42.934+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62a41603e0 msgr2=0x7f62a406b0c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:40:42.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:42.934+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f62a40cc150 legacy=0x7f62a40ccb60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:40:42.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:42.935+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62a41603e0 0x7f62a406b0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:40:43.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:43.135+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62a4145430 legacy=0x7f62a40b3750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:40:43.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:43.136+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f62a40cc150 legacy=0x7f62a40ccb60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:40:43.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:43.136+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62a41603e0 msgr2=0x7f62a406b0c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:40:43.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:43.136+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62a41603e0 0x7f62a406b0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:40:43.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:43.536+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62a4145430 legacy=0x7f62a40b3750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:40:43.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:43.537+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f62a40cc150 legacy=0x7f62a40ccb60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:40:43.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:43.537+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62a41603e0 msgr2=0x7f62a406b0c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:40:43.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:43.537+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62a41603e0 0x7f62a406b0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:40:44.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:44.336+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62a4145430 legacy=0x7f62a40b3750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:40:44.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:44.337+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f62a40cc150 legacy=0x7f62a40ccb60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:40:44.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:44.337+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62a41603e0 msgr2=0x7f62a406b0c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:40:44.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:44.337+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62a41603e0 0x7f62a406b0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:40:45.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:45.934+0000 7f62a0c00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62a41603e0 msgr2=0x7f62a406b0c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:40:45.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:45.934+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f62a41603e0 0x7f62a406b0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:40:45.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:45.934+0000 7f62a0c00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f62a40cc150 legacy=0x7f62a40ccb60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:40:45.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:45.934+0000 7f62a0c00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62a4145430 legacy=0x7f62a40b3750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:40:45.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:45.934+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:40:45.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:45.934+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f62800061a0 0x7f6280008660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:40:45.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:45.934+0000 7f62a0c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f62800036d0 2024-03-24T01:40:45.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:45.934+0000 7f62a0c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6280008d10 con 0x7f62800061a0 2024-03-24T01:40:45.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:45.934+0000 7f62a2a00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:40:45.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:45.934+0000 7f629be00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f62800061a0 msgr2=0x7f6280008660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:40:45.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:45.934+0000 7f62a2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:40:45.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:45.934+0000 7f629be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f62800061a0 0x7f6280008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:40:46.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:46.135+0000 7f62a2a00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:40:46.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:46.135+0000 7f62a2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:40:46.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:46.135+0000 7f629be00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f62800061a0 msgr2=0x7f6280008660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:40:46.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:46.135+0000 7f629be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f62800061a0 0x7f6280008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:40:46.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:46.536+0000 7f62a2a00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:40:46.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:46.536+0000 7f62a2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:40:46.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:46.536+0000 7f629be00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f62800061a0 msgr2=0x7f6280008660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:40:46.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:46.536+0000 7f629be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f62800061a0 0x7f6280008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:40:47.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:47.337+0000 7f62a2a00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:40:47.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:47.337+0000 7f62a2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:40:47.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:47.337+0000 7f629be00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f62800061a0 msgr2=0x7f6280008660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:40:47.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:47.337+0000 7f629be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f62800061a0 0x7f6280008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:40:48.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:48.935+0000 7f62a0c00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f62800061a0 msgr2=0x7f6280008660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:40:48.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:48.935+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f62800061a0 0x7f6280008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:40:48.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:48.935+0000 7f62a0c00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:40:48.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:48.935+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:40:48.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:48.935+0000 7f62a3400640 1 -- reap_dead start 2024-03-24T01:40:48.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:48.935+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:40:48.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:48.935+0000 7f62a0c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a415fbe0 con 0x7f62a4145430 2024-03-24T01:40:48.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:48.935+0000 7f62a0c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a40cd490 con 0x7f62a40cc150 2024-03-24T01:40:48.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:48.935+0000 7f62a0c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f62800036d0 2024-03-24T01:40:48.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:48.935+0000 7f629be00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:40:48.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:48.935+0000 7f629be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:40:48.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:48.935+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62a4145430 legacy=0x7f628000e7c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:40:48.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:48.935+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f62a40cc150 legacy=0x7f628000b050 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:40:49.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:49.136+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62a4145430 legacy=0x7f628000e7c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:40:49.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:49.136+0000 7f629be00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:40:49.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:49.136+0000 7f629be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:40:49.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:49.136+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f62a40cc150 legacy=0x7f628000b050 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:40:49.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:49.537+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62a4145430 legacy=0x7f628000e7c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:40:49.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:49.537+0000 7f629be00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:40:49.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:49.537+0000 7f629be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:40:49.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:49.537+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f62a40cc150 legacy=0x7f628000b050 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:40:50.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:50.338+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62a4145430 legacy=0x7f628000e7c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:40:50.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:50.338+0000 7f629be00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:40:50.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:50.338+0000 7f629be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:40:50.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:50.338+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f62a40cc150 legacy=0x7f628000b050 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:40:51.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:51.935+0000 7f62a0c00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:40:51.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:51.935+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:40:51.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:51.935+0000 7f62a0c00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f62a40cc150 legacy=0x7f628000b050 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:40:51.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:51.935+0000 7f62a0c00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62a4145430 legacy=0x7f628000e7c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:40:51.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:51.935+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000fff0 0x7f6280010460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:40:51.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:51.935+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280010ad0 0x7f6280006610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:40:51.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:51.935+0000 7f62a0c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f628000fff0 2024-03-24T01:40:51.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:51.935+0000 7f62a0c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6280008d10 con 0x7f6280010ad0 2024-03-24T01:40:51.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:51.935+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000fff0 msgr2=0x7f6280010460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:40:51.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:51.935+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280010ad0 msgr2=0x7f6280006610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:40:51.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:51.935+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280010ad0 0x7f6280006610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:40:51.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:51.935+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000fff0 0x7f6280010460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:40:52.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:52.137+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280010ad0 msgr2=0x7f6280006610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:40:52.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:52.137+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000fff0 msgr2=0x7f6280010460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:40:52.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:52.137+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280010ad0 0x7f6280006610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:40:52.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:52.137+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000fff0 0x7f6280010460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:40:52.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:52.538+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000fff0 msgr2=0x7f6280010460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:40:52.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:52.538+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280010ad0 msgr2=0x7f6280006610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:40:52.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:52.538+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000fff0 0x7f6280010460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:40:52.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:52.538+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280010ad0 0x7f6280006610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:40:53.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:53.340+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000fff0 msgr2=0x7f6280010460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:40:53.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:53.340+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280010ad0 msgr2=0x7f6280006610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:40:53.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:53.340+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000fff0 0x7f6280010460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:40:53.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:53.340+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280010ad0 0x7f6280006610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:40:54.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:54.935+0000 7f62a0c00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280010ad0 msgr2=0x7f6280006610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:40:54.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:54.935+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280010ad0 0x7f6280006610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:40:54.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:54.935+0000 7f62a0c00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000fff0 msgr2=0x7f6280010460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:40:54.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:54.935+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000fff0 0x7f6280010460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:40:54.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:54.935+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:40:54.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:54.935+0000 7f62a3400640 1 -- reap_dead start 2024-03-24T01:40:54.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:54.935+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:40:54.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:54.935+0000 7f62a0c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a415f900 con 0x7f6280006d50 2024-03-24T01:40:54.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:54.935+0000 7f62a0c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f62800036d0 2024-03-24T01:40:54.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:54.935+0000 7f62a0c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6280008d10 con 0x7f6280006610 2024-03-24T01:40:54.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:54.935+0000 7f62a2a00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:40:54.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:54.935+0000 7f62a2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:40:54.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:54.935+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:40:54.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:54.935+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f6280006d50 legacy=0x7f628000eef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:40:54.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:54.935+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:40:55.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:55.136+0000 7f62a2a00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:40:55.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:55.136+0000 7f62a2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:40:55.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:55.136+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f6280006d50 legacy=0x7f628000eef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:40:55.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:55.136+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:40:55.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:55.136+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:40:55.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:55.537+0000 7f62a2a00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:40:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:55.538+0000 7f62a2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:40:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:55.538+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f6280006d50 legacy=0x7f628000eef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:40:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:55.538+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:40:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:55.538+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:40:56.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:56.339+0000 7f62a2a00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:40:56.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:56.339+0000 7f62a2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:40:56.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:56.339+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f6280006d50 legacy=0x7f628000eef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:40:56.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:56.339+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:40:56.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:56.339+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:40:57.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:57.936+0000 7f62a0c00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:40:57.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:57.936+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:40:57.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:57.936+0000 7f62a0c00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:40:57.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:57.936+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:40:57.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:57.936+0000 7f62a0c00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f6280006d50 legacy=0x7f628000eef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:40:57.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:57.936+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000e010 0x7f628000e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:40:57.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:57.936+0000 7f62a0c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a415fbe0 con 0x7f6280008690 2024-03-24T01:40:57.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:57.936+0000 7f62a0c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f628000e010 2024-03-24T01:40:57.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:57.936+0000 7f629be00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000e010 msgr2=0x7f628000e420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:40:57.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:57.936+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280008690 legacy=0x7f628000f600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:40:57.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:57.937+0000 7f629be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000e010 0x7f628000e420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:40:58.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:58.138+0000 7f629be00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000e010 msgr2=0x7f628000e420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:40:58.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:58.138+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280008690 legacy=0x7f628000f600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:40:58.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:58.138+0000 7f629be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000e010 0x7f628000e420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:40:58.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:58.539+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280008690 legacy=0x7f628000f600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:40:58.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:58.539+0000 7f629be00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000e010 msgr2=0x7f628000e420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:40:58.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:58.539+0000 7f629be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000e010 0x7f628000e420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:40:59.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:59.340+0000 7f629be00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000e010 msgr2=0x7f628000e420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:40:59.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:59.340+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280008690 legacy=0x7f628000f600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:40:59.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:40:59.340+0000 7f629be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000e010 0x7f628000e420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:41:00.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:00.936+0000 7f62a0c00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000e010 msgr2=0x7f628000e420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:00.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:00.936+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000e010 0x7f628000e420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:41:00.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:00.936+0000 7f62a0c00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280008690 legacy=0x7f628000f600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:00.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:00.936+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:41:00.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:00.936+0000 7f62a3400640 1 -- reap_dead start 2024-03-24T01:41:00.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:00.936+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:41:00.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:00.936+0000 7f62a0c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a415f900 con 0x7f6280006d50 2024-03-24T01:41:00.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:00.936+0000 7f62a0c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f62800036d0 2024-03-24T01:41:00.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:00.936+0000 7f62a0c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6280008d10 con 0x7f6280006610 2024-03-24T01:41:00.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:00.937+0000 7f629be00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:00.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:00.937+0000 7f629be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:41:00.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:00.937+0000 7f62a2a00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:00.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:00.937+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f6280006d50 legacy=0x7f628000eef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:00.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:00.937+0000 7f62a2a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:41:01.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:01.138+0000 7f629be00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:01.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:01.138+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f6280006d50 legacy=0x7f628000eef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:01.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:01.138+0000 7f629be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:41:01.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:01.138+0000 7f62a2a00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:01.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:01.138+0000 7f62a2a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:41:01.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:01.539+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f6280006d50 legacy=0x7f628000eef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:01.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:01.539+0000 7f629be00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:01.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:01.539+0000 7f629be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:41:01.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:01.539+0000 7f62a2a00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:01.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:01.539+0000 7f62a2a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:41:02.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:02.340+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f6280006d50 legacy=0x7f628000eef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:02.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:02.340+0000 7f629be00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:02.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:02.340+0000 7f629be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:41:02.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:02.340+0000 7f62a2a00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:02.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:02.340+0000 7f62a2a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:41:02.711 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:41:02.720 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:41:03.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:03.937+0000 7f62a0c00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:03.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:03.937+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:41:03.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:03.937+0000 7f62a0c00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:03.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:03.937+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:41:03.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:03.937+0000 7f62a0c00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f6280006d50 legacy=0x7f628000eef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:03.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:03.937+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280008690 0x7f628000f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:41:03.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:03.937+0000 7f62a0c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a415fbe0 con 0x7f628000e3f0 2024-03-24T01:41:03.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:03.937+0000 7f62a0c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f6280008690 2024-03-24T01:41:03.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:03.937+0000 7f629be00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f628000e3f0 legacy=0x7f628000d000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:03.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:03.937+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280008690 msgr2=0x7f628000f600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:03.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:03.937+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280008690 0x7f628000f600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:41:04.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:04.138+0000 7f629be00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f628000e3f0 legacy=0x7f628000d000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:04.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:04.138+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280008690 msgr2=0x7f628000f600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:04.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:04.138+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280008690 0x7f628000f600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:41:04.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:04.539+0000 7f629be00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f628000e3f0 legacy=0x7f628000d000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:04.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:04.539+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280008690 msgr2=0x7f628000f600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:04.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:04.539+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280008690 0x7f628000f600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:41:05.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:05.340+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280008690 msgr2=0x7f628000f600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:05.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:05.340+0000 7f629be00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f628000e3f0 legacy=0x7f628000d000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:05.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:05.340+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280008690 0x7f628000f600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:41:06.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:06.937+0000 7f62a0c00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280008690 msgr2=0x7f628000f600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:06.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:06.937+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280008690 0x7f628000f600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:41:06.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:06.937+0000 7f62a0c00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f628000e3f0 legacy=0x7f628000d000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:06.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:06.937+0000 7f62a3400640 1 -- reap_dead start 2024-03-24T01:41:06.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:06.937+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000e800 0x7f628000d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:41:06.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:06.937+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f628000f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:41:06.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:06.937+0000 7f62a0c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a415f900 con 0x7f628000e3f0 2024-03-24T01:41:06.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:06.937+0000 7f62a0c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f62800036d0 2024-03-24T01:41:06.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:06.937+0000 7f62a0c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6280008d10 con 0x7f628000e800 2024-03-24T01:41:06.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:06.937+0000 7f629be00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f628000f600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:06.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:06.937+0000 7f629be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f628000f600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:41:06.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:06.938+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000e800 msgr2=0x7f628000d710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:06.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:06.938+0000 7f62a3400640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f628000e3f0 legacy=0x7f628000d000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:06.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:06.938+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000e800 0x7f628000d710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:41:07.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:07.138+0000 7f629be00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f628000f600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:07.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:07.138+0000 7f629be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f628000f600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:41:07.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:07.139+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000e800 msgr2=0x7f628000d710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:07.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:07.139+0000 7f62a3400640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f628000e3f0 legacy=0x7f628000d000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:07.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:07.139+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000e800 0x7f628000d710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:41:07.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:07.539+0000 7f629be00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f628000f600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:07.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:07.539+0000 7f629be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f628000f600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:41:07.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:07.540+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000e800 msgr2=0x7f628000d710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:07.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:07.540+0000 7f62a3400640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f628000e3f0 legacy=0x7f628000d000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:07.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:07.540+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000e800 0x7f628000d710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:41:08.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:08.340+0000 7f629be00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f628000f600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:08.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:08.340+0000 7f629be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f628000f600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:41:08.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:08.341+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000e800 msgr2=0x7f628000d710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:08.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:08.341+0000 7f62a3400640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f628000e3f0 legacy=0x7f628000d000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:08.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:08.341+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000e800 0x7f628000d710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:41:09.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:09.937+0000 7f62a0c00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000e800 msgr2=0x7f628000d710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:09.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:09.937+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000e800 0x7f628000d710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:41:09.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:09.937+0000 7f62a0c00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f628000f600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:09.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:09.937+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f628000f600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:41:09.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:09.937+0000 7f62a0c00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f628000e3f0 legacy=0x7f628000d000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:09.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:09.937+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:41:09.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:09.937+0000 7f62a0c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a415fbe0 con 0x7f6280006d50 2024-03-24T01:41:09.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:09.937+0000 7f62a0c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f6280006610 2024-03-24T01:41:09.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:09.938+0000 7f62a2a00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f6280006d50 legacy=0x7f628000ebe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:09.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:09.938+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:09.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:09.938+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:41:10.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:10.139+0000 7f62a2a00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f6280006d50 legacy=0x7f628000ebe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:10.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:10.139+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:10.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:10.139+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:41:10.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:10.540+0000 7f62a2a00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f6280006d50 legacy=0x7f628000ebe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:10.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:10.540+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:10.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:10.540+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:41:11.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:11.341+0000 7f62a2a00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f6280006d50 legacy=0x7f628000ebe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:11.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:11.341+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:11.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:11.341+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:41:12.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:12.938+0000 7f62a0c00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:12.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:12.938+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:41:12.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:12.938+0000 7f62a0c00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f6280006d50 legacy=0x7f628000ebe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:12.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:12.938+0000 7f62a3400640 1 -- reap_dead start 2024-03-24T01:41:12.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:12.938+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:41:12.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:12.938+0000 7f62a0c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a415f900 con 0x7f6280006d50 2024-03-24T01:41:12.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:12.938+0000 7f62a0c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a415fbe0 con 0x7f628000e3f0 2024-03-24T01:41:12.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:12.938+0000 7f62a0c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f62800036d0 2024-03-24T01:41:12.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:12.938+0000 7f629be00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:12.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:12.938+0000 7f629be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:41:12.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:12.938+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f6280006d50 legacy=0x7f6280015650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:12.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:12.938+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f628000e3f0 legacy=0x7f6280012f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:13.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:13.139+0000 7f629be00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:13.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:13.139+0000 7f629be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:41:13.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:13.139+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f6280006d50 legacy=0x7f6280015650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:13.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:13.139+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f628000e3f0 legacy=0x7f6280012f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:13.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:13.540+0000 7f629be00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:13.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:13.540+0000 7f629be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:41:13.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:13.540+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f6280006d50 legacy=0x7f6280015650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:13.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:13.540+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f628000e3f0 legacy=0x7f6280012f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:14.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:14.341+0000 7f629be00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:14.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:14.341+0000 7f629be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:41:14.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:14.341+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f6280006d50 legacy=0x7f6280015650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:14.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:14.341+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f628000e3f0 legacy=0x7f6280012f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:15.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:15.938+0000 7f62a0c00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:15.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:15.938+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:41:15.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:15.938+0000 7f62a0c00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f628000e3f0 legacy=0x7f6280012f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:15.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:15.938+0000 7f62a0c00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f6280006d50 legacy=0x7f6280015650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:15.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:15.938+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:41:15.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:15.938+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f628000e7d0 0x7f628000ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:41:15.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:15.938+0000 7f62a0c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f6280006610 2024-03-24T01:41:15.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:15.938+0000 7f62a0c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6280008d10 con 0x7f628000e7d0 2024-03-24T01:41:15.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:15.939+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f628000e7d0 msgr2=0x7f628000ebb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:15.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:15.939+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:15.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:15.939+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:41:15.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:15.939+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f628000e7d0 0x7f628000ebb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:41:16.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:16.140+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f628000e7d0 msgr2=0x7f628000ebb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:16.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:16.140+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:16.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:16.140+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:41:16.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:16.140+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f628000e7d0 0x7f628000ebb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:41:16.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:16.541+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:16.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:16.541+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f628000e7d0 msgr2=0x7f628000ebb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:16.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:16.541+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:41:16.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:16.541+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f628000e7d0 0x7f628000ebb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:41:17.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:17.342+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f628000e7d0 msgr2=0x7f628000ebb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:17.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:17.342+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:17.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:17.342+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f628000e7d0 0x7f628000ebb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:41:17.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:17.342+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:41:18.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:18.939+0000 7f62a0c00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f628000e7d0 msgr2=0x7f628000ebb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:18.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:18.939+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f628000e7d0 0x7f628000ebb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:41:18.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:18.939+0000 7f62a0c00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:18.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:18.939+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:41:18.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:18.939+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:41:18.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:18.939+0000 7f62a3400640 1 -- reap_dead start 2024-03-24T01:41:18.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:18.939+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:41:18.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:18.939+0000 7f62a0c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a40cd490 con 0x7f628000e3f0 2024-03-24T01:41:18.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:18.939+0000 7f62a0c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f62800036d0 2024-03-24T01:41:18.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:18.939+0000 7f62a0c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6280008d10 con 0x7f6280006610 2024-03-24T01:41:18.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:18.939+0000 7f629be00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:18.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:18.939+0000 7f629be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:41:18.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:18.939+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f628000e3f0 legacy=0x7f6280012f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:18.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:18.939+0000 7f62a3400640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:18.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:18.939+0000 7f62a3400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:41:19.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:19.140+0000 7f629be00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:19.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:19.140+0000 7f629be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:41:19.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:19.140+0000 7f62a3400640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:19.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:19.140+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f628000e3f0 legacy=0x7f6280012f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:19.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:19.140+0000 7f62a3400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:41:19.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:19.541+0000 7f629be00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:19.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:19.541+0000 7f629be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:41:19.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:19.541+0000 7f62a3400640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:19.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:19.541+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f628000e3f0 legacy=0x7f6280012f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:19.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:19.541+0000 7f62a3400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:41:20.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:20.342+0000 7f629be00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:20.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:20.342+0000 7f629be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:41:20.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:20.342+0000 7f62a3400640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:20.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:20.342+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f628000e3f0 legacy=0x7f6280012f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:20.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:20.342+0000 7f62a3400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:41:21.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:21.939+0000 7f62a0c00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:21.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:21.939+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:41:21.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:21.939+0000 7f62a0c00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:21.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:21.939+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:41:21.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:21.939+0000 7f62a0c00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f628000e3f0 legacy=0x7f6280012f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:21.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:21.939+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 0x7f6280007490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:41:21.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:21.939+0000 7f62a0c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a415f900 con 0x7f62800079d0 2024-03-24T01:41:21.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:21.939+0000 7f62a0c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f6280006d30 2024-03-24T01:41:21.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:21.939+0000 7f629be00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800079d0 legacy=0x7f628000e7d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:21.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:21.939+0000 7f62a2a00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 msgr2=0x7f6280007490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:21.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:21.939+0000 7f62a2a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 0x7f6280007490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:41:22.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:22.140+0000 7f629be00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800079d0 legacy=0x7f628000e7d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:22.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:22.140+0000 7f62a2a00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 msgr2=0x7f6280007490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:22.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:22.140+0000 7f62a2a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 0x7f6280007490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:41:22.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:22.541+0000 7f629be00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800079d0 legacy=0x7f628000e7d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:22.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:22.541+0000 7f62a2a00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 msgr2=0x7f6280007490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:22.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:22.541+0000 7f62a2a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 0x7f6280007490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:41:23.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:23.342+0000 7f629be00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800079d0 legacy=0x7f628000e7d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:23.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:23.342+0000 7f62a2a00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 msgr2=0x7f6280007490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:23.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:23.342+0000 7f62a2a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 0x7f6280007490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:41:24.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:24.939+0000 7f62a0c00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 msgr2=0x7f6280007490 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:24.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:24.939+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 0x7f6280007490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:41:24.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:24.939+0000 7f62a0c00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800079d0 legacy=0x7f628000e7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:24.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:24.939+0000 7f62a3400640 1 -- reap_dead start 2024-03-24T01:41:24.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:24.939+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 0x7f6280007490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:41:24.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:24.939+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:41:24.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:24.939+0000 7f62a0c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a40cd490 con 0x7f628000e3f0 2024-03-24T01:41:24.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:24.939+0000 7f62a0c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f62800036d0 2024-03-24T01:41:24.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:24.939+0000 7f62a0c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6280008d10 con 0x7f6280006d30 2024-03-24T01:41:24.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:24.940+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f628000e3f0 legacy=0x7f6280012f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:24.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:24.940+0000 7f62a3400640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:24.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:24.940+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 msgr2=0x7f6280007490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:24.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:24.940+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 0x7f6280007490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:41:24.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:24.940+0000 7f62a3400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:41:25.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:25.141+0000 7f62a3400640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:25.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:25.141+0000 7f62a3400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:41:25.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:25.141+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 msgr2=0x7f6280007490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:25.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:25.141+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f628000e3f0 legacy=0x7f6280012f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:25.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:25.141+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 0x7f6280007490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:41:25.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:25.542+0000 7f62a3400640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:25.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:25.542+0000 7f62a3400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:41:25.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:25.542+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 msgr2=0x7f6280007490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:25.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:25.542+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f628000e3f0 legacy=0x7f6280012f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:25.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:25.542+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 0x7f6280007490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:41:26.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:26.343+0000 7f62a3400640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:26.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:26.343+0000 7f62a3400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:41:26.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:26.343+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 msgr2=0x7f6280007490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:26.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:26.343+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f628000e3f0 legacy=0x7f6280012f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:26.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:26.343+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 0x7f6280007490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:41:27.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:27.940+0000 7f62a0c00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 msgr2=0x7f6280007490 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:27.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:27.940+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 0x7f6280007490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:41:27.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:27.940+0000 7f62a0c00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:27.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:27.940+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:41:27.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:27.940+0000 7f62a0c00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f628000e3f0 legacy=0x7f6280012f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:27.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:27.940+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:41:27.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:27.940+0000 7f62a0c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a415f900 con 0x7f6280007a90 2024-03-24T01:41:27.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:27.940+0000 7f62a0c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f6280006610 2024-03-24T01:41:27.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:27.940+0000 7f629be00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f6280007a90 legacy=0x7f628000e7d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:27.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:27.940+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:27.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:27.940+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:41:28.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:28.141+0000 7f629be00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f6280007a90 legacy=0x7f628000e7d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:28.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:28.141+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:28.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:28.141+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:41:28.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:28.542+0000 7f629be00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f6280007a90 legacy=0x7f628000e7d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:28.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:28.542+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:28.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:28.542+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:41:29.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:29.343+0000 7f629be00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f6280007a90 legacy=0x7f628000e7d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:29.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:29.343+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:29.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:29.343+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:41:30.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:30.940+0000 7f62a0c00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:30.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:30.940+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:41:30.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:30.940+0000 7f62a0c00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f6280007a90 legacy=0x7f628000e7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:30.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:30.940+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:41:30.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:30.940+0000 7f62a3400640 1 -- reap_dead start 2024-03-24T01:41:30.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:30.940+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006d30 0x7f62800103d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:41:30.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:30.940+0000 7f62a0c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a40cd490 con 0x7f6280006610 2024-03-24T01:41:30.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:30.940+0000 7f62a0c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f62800036d0 2024-03-24T01:41:30.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:30.940+0000 7f62a0c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6280008d10 con 0x7f6280006d30 2024-03-24T01:41:30.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:30.940+0000 7f62a2a00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:30.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:30.940+0000 7f62a2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:41:30.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:30.940+0000 7f629be00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280006610 legacy=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:30.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:30.940+0000 7f62a3400640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006d30 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:30.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:30.941+0000 7f62a3400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006d30 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:41:31.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:31.141+0000 7f62a2a00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:31.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:31.141+0000 7f62a2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:41:31.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:31.141+0000 7f629be00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280006610 legacy=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:31.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:31.142+0000 7f62a3400640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006d30 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:31.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:31.142+0000 7f62a3400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006d30 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:41:31.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:31.542+0000 7f62a2a00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:31.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:31.542+0000 7f62a2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:41:31.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:31.542+0000 7f629be00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280006610 legacy=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:31.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:31.543+0000 7f62a3400640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006d30 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:31.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:31.543+0000 7f62a3400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006d30 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:41:32.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:32.343+0000 7f62a2a00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:32.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:32.343+0000 7f62a2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:41:32.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:32.343+0000 7f629be00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280006610 legacy=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:32.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:32.344+0000 7f62a3400640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006d30 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:32.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:32.344+0000 7f62a3400640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006d30 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:41:33.092 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:41:33.101 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:41:33.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:33.941+0000 7f62a0c00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006d30 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:33.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:33.941+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006d30 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:41:33.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:33.941+0000 7f62a0c00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:33.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:33.941+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:41:33.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:33.941+0000 7f62a0c00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280006610 legacy=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:33.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:33.941+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007460 0x7f6280007f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:41:33.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:33.941+0000 7f62a0c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a415f900 con 0x7f6280007a90 2024-03-24T01:41:33.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:33.941+0000 7f62a0c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f6280007460 2024-03-24T01:41:33.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:33.941+0000 7f62a2a00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f6280007a90 legacy=0x7f628000d000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:33.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:33.941+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007460 msgr2=0x7f6280007f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:33.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:33.941+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007460 0x7f6280007f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:41:34.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:34.142+0000 7f62a2a00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f6280007a90 legacy=0x7f628000d000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:34.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:34.142+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007460 msgr2=0x7f6280007f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:34.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:34.142+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007460 0x7f6280007f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:41:34.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:34.543+0000 7f62a2a00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f6280007a90 legacy=0x7f628000d000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:34.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:34.543+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007460 msgr2=0x7f6280007f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:34.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:34.543+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007460 0x7f6280007f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:41:35.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:35.344+0000 7f62a2a00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f6280007a90 legacy=0x7f628000d000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:35.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:35.344+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007460 msgr2=0x7f6280007f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:35.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:35.344+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007460 0x7f6280007f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:41:36.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:36.941+0000 7f62a0c00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007460 msgr2=0x7f6280007f10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:36.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:36.941+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007460 0x7f6280007f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:41:36.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:36.941+0000 7f62a0c00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f6280007a90 legacy=0x7f628000d000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:36.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:36.941+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:41:36.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:36.941+0000 7f62a3400640 1 -- reap_dead start 2024-03-24T01:41:36.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:36.941+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006d30 0x7f62800103d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:41:36.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:36.941+0000 7f62a0c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a40cd490 con 0x7f6280006610 2024-03-24T01:41:36.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:36.941+0000 7f62a0c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f62800036d0 2024-03-24T01:41:36.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:36.941+0000 7f62a0c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6280008d10 con 0x7f6280006d30 2024-03-24T01:41:36.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:36.942+0000 7f62a2a00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:36.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:36.942+0000 7f629be00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006d30 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:36.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:36.942+0000 7f62a2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:41:36.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:36.942+0000 7f62a3400640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280006610 legacy=0x7f628000d710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:36.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:36.942+0000 7f629be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006d30 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:41:37.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:37.143+0000 7f62a2a00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:37.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:37.143+0000 7f62a2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:41:37.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:37.143+0000 7f629be00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006d30 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:37.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:37.143+0000 7f62a3400640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280006610 legacy=0x7f628000d710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:37.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:37.143+0000 7f629be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006d30 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:41:37.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:37.544+0000 7f62a2a00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:37.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:37.544+0000 7f62a2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:41:37.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:37.544+0000 7f62a3400640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280006610 legacy=0x7f628000d710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:37.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:37.544+0000 7f629be00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006d30 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:37.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:37.544+0000 7f629be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006d30 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:41:38.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:38.345+0000 7f62a2a00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:38.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:38.345+0000 7f62a2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:41:38.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:38.345+0000 7f629be00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006d30 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:38.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:38.345+0000 7f62a3400640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280006610 legacy=0x7f628000d710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:38.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:38.345+0000 7f629be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006d30 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:41:39.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:39.942+0000 7f62a0c00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006d30 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:39.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:39.942+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006d30 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:41:39.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:39.942+0000 7f62a0c00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 msgr2=0x7f6280005ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:39.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:39.942+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62800036d0 0x7f6280005ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:41:39.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:39.942+0000 7f62a0c00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280006610 legacy=0x7f628000d710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:39.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:39.942+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007a90 0x7f6280007f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:41:39.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:39.942+0000 7f62a0c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a415f900 con 0x7f6280007460 2024-03-24T01:41:39.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:39.942+0000 7f62a0c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f6280007a90 2024-03-24T01:41:39.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:39.942+0000 7f629be00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f6280007460 legacy=0x7f628000d000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:39.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:39.942+0000 7f62a2a00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007a90 msgr2=0x7f6280007f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:39.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:39.942+0000 7f62a2a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007a90 0x7f6280007f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:41:40.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:40.143+0000 7f629be00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f6280007460 legacy=0x7f628000d000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:40.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:40.143+0000 7f62a2a00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007a90 msgr2=0x7f6280007f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:40.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:40.143+0000 7f62a2a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007a90 0x7f6280007f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:41:40.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:40.544+0000 7f629be00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f6280007460 legacy=0x7f628000d000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:40.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:40.544+0000 7f62a2a00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007a90 msgr2=0x7f6280007f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:40.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:40.544+0000 7f62a2a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007a90 0x7f6280007f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:41:41.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:41.345+0000 7f629be00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f6280007460 legacy=0x7f628000d000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:41.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:41.345+0000 7f62a2a00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007a90 msgr2=0x7f6280007f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:41.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:41.345+0000 7f62a2a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007a90 0x7f6280007f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:41:42.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:42.942+0000 7f62a0c00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007a90 msgr2=0x7f6280007f10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:42.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:42.942+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007a90 0x7f6280007f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:41:42.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:42.942+0000 7f62a0c00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f6280007460 legacy=0x7f628000d000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:42.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:42.942+0000 7f62a3400640 1 -- reap_dead start 2024-03-24T01:41:42.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:42.942+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006d30 0x7f62800103d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:41:42.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:42.942+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007a90 0x7f6280007f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:41:42.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:42.942+0000 7f62a0c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a40cd490 con 0x7f628000f400 2024-03-24T01:41:42.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:42.942+0000 7f62a0c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f6280006d30 2024-03-24T01:41:42.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:42.942+0000 7f62a0c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6280008d10 con 0x7f6280007a90 2024-03-24T01:41:42.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:42.942+0000 7f62a2a00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f628000f400 legacy=0x7f628001c390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:42.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:42.942+0000 7f629be00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006d30 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:42.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:42.942+0000 7f629be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006d30 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:41:42.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:42.942+0000 7f62a3400640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007a90 msgr2=0x7f6280007f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:42.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:42.942+0000 7f62a3400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007a90 0x7f6280007f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:41:43.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:43.143+0000 7f62a2a00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f628000f400 legacy=0x7f628001c390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:43.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:43.143+0000 7f629be00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006d30 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:43.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:43.143+0000 7f629be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006d30 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:41:43.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:43.144+0000 7f62a3400640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007a90 msgr2=0x7f6280007f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:43.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:43.144+0000 7f62a3400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007a90 0x7f6280007f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:41:43.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:43.544+0000 7f62a2a00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f628000f400 legacy=0x7f628001c390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:43.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:43.544+0000 7f629be00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006d30 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:43.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:43.544+0000 7f629be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006d30 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:41:43.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:43.545+0000 7f62a3400640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007a90 msgr2=0x7f6280007f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:43.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:43.545+0000 7f62a3400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007a90 0x7f6280007f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:41:44.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:44.345+0000 7f62a2a00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f628000f400 legacy=0x7f628001c390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:44.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:44.345+0000 7f629be00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006d30 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:44.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:44.345+0000 7f629be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006d30 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:41:44.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:44.346+0000 7f62a3400640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007a90 msgr2=0x7f6280007f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:44.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:44.346+0000 7f62a3400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007a90 0x7f6280007f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:41:45.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:45.941+0000 7f62a0c00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007a90 msgr2=0x7f6280007f10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:45.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:45.941+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007a90 0x7f6280007f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:41:45.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:45.941+0000 7f62a0c00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006d30 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:45.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:45.941+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006d30 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:41:45.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:45.941+0000 7f62a0c00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f628000f400 legacy=0x7f628001c390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:45.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:45.941+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f62800056c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:41:45.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:45.941+0000 7f62a0c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a415f900 con 0x7f6280005c00 2024-03-24T01:41:45.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:45.941+0000 7f62a0c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f6280006610 2024-03-24T01:41:45.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:45.942+0000 7f62a2a00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 msgr2=0x7f62800056c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:45.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:45.942+0000 7f62a2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f62800056c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:41:45.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:45.942+0000 7f629be00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000d000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:46.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:46.143+0000 7f62a2a00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 msgr2=0x7f62800056c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:46.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:46.143+0000 7f62a2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f62800056c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:41:46.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:46.143+0000 7f629be00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000d000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:46.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:46.544+0000 7f62a2a00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 msgr2=0x7f62800056c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:46.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:46.544+0000 7f62a2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f62800056c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:41:46.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:46.544+0000 7f629be00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000d000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:47.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:47.346+0000 7f62a2a00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 msgr2=0x7f62800056c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:47.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:47.346+0000 7f62a2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f62800056c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:41:47.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:47.346+0000 7f629be00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000d000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:48.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:48.942+0000 7f62a0c00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 msgr2=0x7f62800056c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:48.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:48.942+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f62800056c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:41:48.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:48.942+0000 7f62a0c00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000d000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:48.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:48.942+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007a90 0x7f6280007f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:41:48.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:48.942+0000 7f62a3400640 1 -- reap_dead start 2024-03-24T01:41:48.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:48.942+0000 7f62a0c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a40cd490 con 0x7f628000f400 2024-03-24T01:41:48.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:48.942+0000 7f62a0c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a415f900 con 0x7f6280005c00 2024-03-24T01:41:48.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:48.942+0000 7f62a0c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f6280007a90 2024-03-24T01:41:48.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:48.942+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f628000f400 legacy=0x7f628000d710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:48.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:48.942+0000 7f629be00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000d000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:48.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:48.942+0000 7f62a2a00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007a90 msgr2=0x7f6280007f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:48.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:48.942+0000 7f62a2a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007a90 0x7f6280007f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:41:49.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:49.143+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f628000f400 legacy=0x7f628000d710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:49.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:49.143+0000 7f629be00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000d000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:49.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:49.143+0000 7f62a2a00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007a90 msgr2=0x7f6280007f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:49.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:49.143+0000 7f62a2a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007a90 0x7f6280007f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:41:49.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:49.544+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f628000f400 legacy=0x7f628000d710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:49.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:49.544+0000 7f629be00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000d000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:49.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:49.544+0000 7f62a2a00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007a90 msgr2=0x7f6280007f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:49.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:49.544+0000 7f62a2a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007a90 0x7f6280007f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:41:50.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:50.345+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f628000f400 legacy=0x7f628000d710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:50.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:50.345+0000 7f629be00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000d000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:50.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:50.345+0000 7f62a2a00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007a90 msgr2=0x7f6280007f10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:50.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:50.345+0000 7f62a2a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007a90 0x7f6280007f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:41:51.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:51.943+0000 7f62a0c00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007a90 msgr2=0x7f6280007f10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:51.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:51.943+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007a90 0x7f6280007f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:41:51.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:51.943+0000 7f62a0c00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000d000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:51.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:51.943+0000 7f62a0c00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f628000f400 legacy=0x7f628000d710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:51.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:51.943+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006d30 0x7f62800103d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:41:51.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:51.943+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 0x7f62800036b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:41:51.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:51.943+0000 7f62a0c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f6280006d30 2024-03-24T01:41:51.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:51.943+0000 7f62a0c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6280008d10 con 0x7f6280006610 2024-03-24T01:41:51.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:51.943+0000 7f62a2a00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006d30 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:51.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:51.943+0000 7f629be00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 msgr2=0x7f62800036b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:51.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:51.943+0000 7f62a2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006d30 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:41:51.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:51.943+0000 7f629be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 0x7f62800036b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:41:52.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:52.144+0000 7f62a2a00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006d30 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:52.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:52.144+0000 7f62a2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006d30 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:41:52.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:52.144+0000 7f629be00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 msgr2=0x7f62800036b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:52.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:52.144+0000 7f629be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 0x7f62800036b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:41:52.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:52.545+0000 7f62a2a00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006d30 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:52.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:52.545+0000 7f62a2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006d30 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:41:52.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:52.545+0000 7f629be00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 msgr2=0x7f62800036b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:52.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:52.545+0000 7f629be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 0x7f62800036b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:41:53.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:53.346+0000 7f62a2a00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006d30 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:53.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:53.346+0000 7f62a2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006d30 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:41:53.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:53.346+0000 7f629be00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 msgr2=0x7f62800036b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:53.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:53.346+0000 7f629be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 0x7f62800036b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:41:54.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:54.943+0000 7f62a0c00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 msgr2=0x7f62800036b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:54.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:54.943+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 0x7f62800036b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:41:54.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:54.943+0000 7f62a0c00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006d30 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:54.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:54.943+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006d30 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:41:54.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:54.943+0000 7f62a3400640 1 -- reap_dead start 2024-03-24T01:41:54.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:54.943+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f62800036b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:41:54.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:54.943+0000 7f62a0c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a415fbe0 con 0x7f628000f400 2024-03-24T01:41:54.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:54.943+0000 7f62a0c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a40cd490 con 0x7f6280005c00 2024-03-24T01:41:54.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:54.943+0000 7f62a0c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f6280006610 2024-03-24T01:41:54.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:54.943+0000 7f629be00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f628000f400 legacy=0x7f628000d710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:54.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:54.943+0000 7f62a3400640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 msgr2=0x7f62800036b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:54.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:54.943+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000d000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:54.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:54.943+0000 7f62a3400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f62800036b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:41:55.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:55.144+0000 7f629be00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f628000f400 legacy=0x7f628000d710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:55.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:55.144+0000 7f62a3400640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 msgr2=0x7f62800036b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:55.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:55.144+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000d000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:55.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:55.144+0000 7f62a3400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f62800036b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:41:55.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:55.545+0000 7f629be00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f628000f400 legacy=0x7f628000d710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:55.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:55.545+0000 7f62a3400640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 msgr2=0x7f62800036b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:55.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:55.545+0000 7f62a3400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f62800036b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:41:55.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:55.545+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000d000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:56.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:56.346+0000 7f629be00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f628000f400 legacy=0x7f628000d710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:41:56.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:56.347+0000 7f62a3400640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 msgr2=0x7f62800036b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:41:56.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:56.347+0000 7f62a3400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f62800036b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:41:56.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:56.347+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000d000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:41:57.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:57.943+0000 7f62a0c00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 msgr2=0x7f62800036b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:57.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:57.943+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f62800036b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:41:57.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:57.944+0000 7f62a0c00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000d000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:57.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:57.944+0000 7f62a0c00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f628000f400 legacy=0x7f628000d710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:41:57.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:57.944+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 0x7f62800103d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:41:57.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:57.944+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280007a90 0x7f6280007e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:41:57.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:57.944+0000 7f62a0c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f6280007a90 2024-03-24T01:41:57.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:57.944+0000 7f62a0c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6280008d10 con 0x7f6280006d30 2024-03-24T01:41:57.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:57.944+0000 7f62a2a00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:57.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:57.944+0000 7f629be00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280007a90 msgr2=0x7f6280007e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:57.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:57.944+0000 7f62a2a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:41:57.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:57.944+0000 7f629be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280007a90 0x7f6280007e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:41:58.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:58.145+0000 7f62a2a00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:58.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:58.145+0000 7f629be00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280007a90 msgr2=0x7f6280007e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:58.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:58.145+0000 7f62a2a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:41:58.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:58.145+0000 7f629be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280007a90 0x7f6280007e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:41:58.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:58.546+0000 7f629be00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280007a90 msgr2=0x7f6280007e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:58.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:58.546+0000 7f62a2a00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:58.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:58.546+0000 7f629be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280007a90 0x7f6280007e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:41:58.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:58.546+0000 7f62a2a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:41:59.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:59.347+0000 7f62a2a00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:41:59.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:59.347+0000 7f629be00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280007a90 msgr2=0x7f6280007e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:41:59.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:59.347+0000 7f62a2a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:41:59.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:41:59.347+0000 7f629be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280007a90 0x7f6280007e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:42:00.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:00.944+0000 7f62a0c00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 msgr2=0x7f62800103d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:42:00.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:00.944+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 0x7f62800103d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:42:00.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:00.944+0000 7f62a0c00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280007a90 msgr2=0x7f6280007e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:42:00.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:00.944+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280007a90 0x7f6280007e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:42:00.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:00.944+0000 7f62a3400640 1 -- reap_dead start 2024-03-24T01:42:00.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:00.944+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000eb00 0x7f628000c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:42:00.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:00.944+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 0x7f6280007e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:42:00.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:00.944+0000 7f62a0c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a415f900 con 0x7f6280005c00 2024-03-24T01:42:00.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:00.944+0000 7f62a0c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f628000eb00 2024-03-24T01:42:00.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:00.944+0000 7f62a0c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6280008d10 con 0x7f6280006d30 2024-03-24T01:42:00.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:00.944+0000 7f629be00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000eb00 msgr2=0x7f628000c000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:42:00.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:00.944+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:42:00.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:00.944+0000 7f629be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000eb00 0x7f628000c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:42:00.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:00.944+0000 7f62a3400640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 msgr2=0x7f6280007e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:42:00.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:00.945+0000 7f62a3400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 0x7f6280007e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:42:01.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:01.145+0000 7f629be00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000eb00 msgr2=0x7f628000c000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:42:01.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:01.145+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:42:01.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:01.145+0000 7f629be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000eb00 0x7f628000c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:42:01.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:01.146+0000 7f62a3400640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 msgr2=0x7f6280007e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:42:01.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:01.146+0000 7f62a3400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 0x7f6280007e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:42:01.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:01.546+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:42:01.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:01.546+0000 7f629be00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000eb00 msgr2=0x7f628000c000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:42:01.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:01.546+0000 7f629be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000eb00 0x7f628000c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:42:01.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:01.547+0000 7f62a3400640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 msgr2=0x7f6280007e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:42:01.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:01.547+0000 7f62a3400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 0x7f6280007e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:42:02.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:02.347+0000 7f629be00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000eb00 msgr2=0x7f628000c000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:42:02.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:02.347+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:42:02.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:02.347+0000 7f629be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000eb00 0x7f628000c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:42:02.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:02.348+0000 7f62a3400640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 msgr2=0x7f6280007e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:42:02.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:02.348+0000 7f62a3400640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 0x7f6280007e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:42:03.237 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:42:03.246 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:42:03.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:03.944+0000 7f62a0c00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 msgr2=0x7f6280007e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:42:03.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:03.944+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 0x7f6280007e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:42:03.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:03.944+0000 7f62a0c00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000eb00 msgr2=0x7f628000c000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:42:03.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:03.944+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000eb00 0x7f628000c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:42:03.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:03.944+0000 7f62a0c00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:42:03.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:03.944+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f62800056c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:42:03.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:03.944+0000 7f62a0c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a415fbe0 con 0x7f62800103d0 2024-03-24T01:42:03.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:03.944+0000 7f62a0c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f6280006610 2024-03-24T01:42:03.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:03.945+0000 7f62a2a00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 msgr2=0x7f62800056c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:42:03.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:03.945+0000 7f62a2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f62800056c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:42:03.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:03.945+0000 7f629be00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800103d0 legacy=0x7f628000eee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:42:04.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:04.146+0000 7f62a2a00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 msgr2=0x7f62800056c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:42:04.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:04.146+0000 7f629be00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800103d0 legacy=0x7f628000eee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:42:04.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:04.146+0000 7f62a2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f62800056c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:42:04.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:04.547+0000 7f62a2a00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 msgr2=0x7f62800056c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:42:04.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:04.547+0000 7f629be00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800103d0 legacy=0x7f628000eee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:42:04.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:04.547+0000 7f62a2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f62800056c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:42:05.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:05.348+0000 7f629be00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800103d0 legacy=0x7f628000eee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:42:05.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:05.348+0000 7f62a2a00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 msgr2=0x7f62800056c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:42:05.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:05.348+0000 7f62a2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f62800056c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:42:06.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:06.945+0000 7f62a0c00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 msgr2=0x7f62800056c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:42:06.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:06.945+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f62800056c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:42:06.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:06.945+0000 7f62a0c00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800103d0 legacy=0x7f628000eee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:42:06.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:06.945+0000 7f62a3400640 1 -- reap_dead start 2024-03-24T01:42:06.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:06.945+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f62800056c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:42:06.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:06.945+0000 7f62a0c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a415f900 con 0x7f62800103d0 2024-03-24T01:42:06.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:06.945+0000 7f62a0c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a415fbe0 con 0x7f6280005c00 2024-03-24T01:42:06.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:06.945+0000 7f62a0c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f6280006610 2024-03-24T01:42:06.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:06.945+0000 7f629be00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800103d0 legacy=0x7f628000eee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:42:06.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:06.945+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:42:06.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:06.945+0000 7f62a3400640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 msgr2=0x7f62800056c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:42:06.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:06.945+0000 7f62a3400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f62800056c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:42:07.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:07.146+0000 7f62a3400640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 msgr2=0x7f62800056c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:42:07.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:07.146+0000 7f629be00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800103d0 legacy=0x7f628000eee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:42:07.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:07.146+0000 7f62a3400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f62800056c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:42:07.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:07.146+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:42:07.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:07.547+0000 7f62a3400640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 msgr2=0x7f62800056c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:42:07.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:07.547+0000 7f629be00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800103d0 legacy=0x7f628000eee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:42:07.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:07.547+0000 7f62a3400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f62800056c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:42:07.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:07.547+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:42:08.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:08.348+0000 7f62a3400640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 msgr2=0x7f62800056c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:42:08.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:08.348+0000 7f629be00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800103d0 legacy=0x7f628000eee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:42:08.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:08.348+0000 7f62a3400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f62800056c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:42:08.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:08.348+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:42:09.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:09.945+0000 7f62a0c00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 msgr2=0x7f62800056c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:42:09.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:09.945+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f62800056c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:42:09.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:09.945+0000 7f62a0c00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:42:09.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:09.945+0000 7f62a0c00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800103d0 legacy=0x7f628000eee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:42:09.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:09.945+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000eb00 0x7f6280007e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:42:09.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:09.945+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 0x7f628000c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:42:09.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:09.945+0000 7f62a0c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f628000eb00 2024-03-24T01:42:09.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:09.945+0000 7f62a0c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6280008d10 con 0x7f6280006d30 2024-03-24T01:42:09.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:09.945+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000eb00 msgr2=0x7f6280007e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:42:09.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:09.946+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000eb00 0x7f6280007e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:42:09.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:09.946+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 msgr2=0x7f628000c000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:42:09.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:09.946+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 0x7f628000c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:42:10.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:10.147+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 msgr2=0x7f628000c000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:42:10.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:10.147+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000eb00 msgr2=0x7f6280007e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:42:10.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:10.147+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 0x7f628000c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:42:10.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:10.147+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000eb00 0x7f6280007e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:42:10.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:10.548+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 msgr2=0x7f628000c000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:42:10.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:10.548+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000eb00 msgr2=0x7f6280007e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:42:10.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:10.548+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 0x7f628000c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:42:10.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:10.548+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000eb00 0x7f6280007e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:42:11.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:11.349+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000eb00 msgr2=0x7f6280007e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:42:11.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:11.349+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 msgr2=0x7f628000c000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:42:11.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:11.349+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000eb00 0x7f6280007e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:42:11.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:11.349+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 0x7f628000c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:42:12.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:12.945+0000 7f62a0c00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 msgr2=0x7f628000c000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:42:12.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:12.945+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006d30 0x7f628000c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:42:12.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:12.945+0000 7f62a0c00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000eb00 msgr2=0x7f6280007e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:42:12.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:12.945+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000eb00 0x7f6280007e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:42:12.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:12.945+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000f5f0 0x7f628000d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:42:12.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:12.945+0000 7f62a3400640 1 -- reap_dead start 2024-03-24T01:42:12.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:12.945+0000 7f62a0c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a40cd490 con 0x7f62800103d0 2024-03-24T01:42:12.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:12.945+0000 7f62a0c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a415f900 con 0x7f6280005c00 2024-03-24T01:42:12.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:12.945+0000 7f62a0c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f628000f5f0 2024-03-24T01:42:12.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:12.946+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:42:12.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:12.946+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800103d0 legacy=0x7f628000eee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:42:12.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:12.946+0000 7f629be00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000f5f0 msgr2=0x7f628000d550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:42:12.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:12.946+0000 7f629be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000f5f0 0x7f628000d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:42:13.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:13.147+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800103d0 legacy=0x7f628000eee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:42:13.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:13.147+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:42:13.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:13.147+0000 7f629be00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000f5f0 msgr2=0x7f628000d550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:42:13.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:13.147+0000 7f629be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000f5f0 0x7f628000d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:42:13.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:13.548+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800103d0 legacy=0x7f628000eee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:42:13.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:13.548+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:42:13.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:13.548+0000 7f629be00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000f5f0 msgr2=0x7f628000d550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:42:13.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:13.548+0000 7f629be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000f5f0 0x7f628000d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:42:14.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:14.349+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800103d0 legacy=0x7f628000eee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:42:14.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:14.349+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:42:14.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:14.349+0000 7f629be00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000f5f0 msgr2=0x7f628000d550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:42:14.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:14.349+0000 7f629be00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000f5f0 0x7f628000d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:42:15.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:15.946+0000 7f62a0c00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000f5f0 msgr2=0x7f628000d550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:42:15.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:15.946+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000f5f0 0x7f628000d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:42:15.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:15.946+0000 7f62a0c00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:42:15.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:15.946+0000 7f62a0c00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800103d0 legacy=0x7f628000eee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:42:15.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:15.946+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 0x7f628000c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:42:15.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:15.946+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006cf0 0x7f6280007e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:42:15.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:15.946+0000 7f62a0c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f6280006cf0 2024-03-24T01:42:15.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:15.946+0000 7f62a0c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6280008d10 con 0x7f6280006610 2024-03-24T01:42:15.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:15.946+0000 7f62a2a00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 msgr2=0x7f628000c000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:42:15.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:15.946+0000 7f629be00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006cf0 msgr2=0x7f6280007e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:42:15.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:15.946+0000 7f62a2a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 0x7f628000c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:42:15.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:15.946+0000 7f629be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006cf0 0x7f6280007e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:42:16.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:16.147+0000 7f629be00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006cf0 msgr2=0x7f6280007e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:42:16.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:16.147+0000 7f62a2a00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 msgr2=0x7f628000c000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:42:16.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:16.147+0000 7f629be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006cf0 0x7f6280007e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:42:16.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:16.147+0000 7f62a2a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 0x7f628000c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:42:16.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:16.548+0000 7f629be00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006cf0 msgr2=0x7f6280007e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:42:16.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:16.548+0000 7f629be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006cf0 0x7f6280007e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:42:16.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:16.548+0000 7f62a2a00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 msgr2=0x7f628000c000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:42:16.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:16.548+0000 7f62a2a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 0x7f628000c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:42:17.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:17.349+0000 7f629be00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006cf0 msgr2=0x7f6280007e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:42:17.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:17.349+0000 7f629be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006cf0 0x7f6280007e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:42:17.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:17.349+0000 7f62a2a00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 msgr2=0x7f628000c000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:42:17.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:17.349+0000 7f62a2a00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 0x7f628000c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:42:18.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:18.946+0000 7f62a0c00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 msgr2=0x7f628000c000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:42:18.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:18.946+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280006610 0x7f628000c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:42:18.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:18.946+0000 7f62a0c00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006cf0 msgr2=0x7f6280007e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:42:18.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:18.946+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006cf0 0x7f6280007e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:42:18.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:18.946+0000 7f62a3400640 1 -- reap_dead start 2024-03-24T01:42:18.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:18.946+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f628000c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:42:18.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:18.946+0000 7f62a0c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a415fbe0 con 0x7f62800103d0 2024-03-24T01:42:18.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:18.946+0000 7f62a0c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a40cd490 con 0x7f6280005c00 2024-03-24T01:42:18.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:18.946+0000 7f62a0c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f6280006610 2024-03-24T01:42:18.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:18.946+0000 7f62a2a00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800103d0 legacy=0x7f628000eee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:42:18.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:18.946+0000 7f62a3400640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 msgr2=0x7f628000c000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:42:18.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:18.946+0000 7f629be00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:42:18.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:18.946+0000 7f62a3400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f628000c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:42:19.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:19.147+0000 7f62a3400640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 msgr2=0x7f628000c000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:42:19.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:19.147+0000 7f62a2a00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800103d0 legacy=0x7f628000eee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:42:19.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:19.147+0000 7f62a3400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f628000c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:42:19.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:19.148+0000 7f629be00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:42:19.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:19.548+0000 7f62a3400640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 msgr2=0x7f628000c000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:42:19.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:19.548+0000 7f62a2a00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800103d0 legacy=0x7f628000eee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:42:19.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:19.548+0000 7f62a3400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f628000c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:42:19.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:19.549+0000 7f629be00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:42:20.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:20.349+0000 7f62a2a00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800103d0 legacy=0x7f628000eee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:42:20.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:20.349+0000 7f62a3400640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 msgr2=0x7f628000c000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:42:20.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:20.349+0000 7f62a3400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f628000c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:42:20.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:20.350+0000 7f629be00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:42:21.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:21.947+0000 7f62a0c00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 msgr2=0x7f628000c000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:42:21.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:21.947+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f628000c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:42:21.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:21.947+0000 7f62a0c00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:42:21.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:21.947+0000 7f62a0c00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800103d0 legacy=0x7f628000eee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:42:21.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:21.947+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000f5f0 0x7f62800056a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:42:21.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:21.947+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 0x7f628000d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:42:21.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:21.947+0000 7f62a0c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f628000f5f0 2024-03-24T01:42:21.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:21.947+0000 7f62a0c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6280008d10 con 0x7f6280007e70 2024-03-24T01:42:21.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:21.947+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000f5f0 msgr2=0x7f62800056a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:42:21.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:21.947+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000f5f0 0x7f62800056a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:42:21.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:21.947+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 msgr2=0x7f628000d550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:42:21.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:21.947+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 0x7f628000d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:42:22.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:22.148+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000f5f0 msgr2=0x7f62800056a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:42:22.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:22.148+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 msgr2=0x7f628000d550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:42:22.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:22.148+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000f5f0 0x7f62800056a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:42:22.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:22.148+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 0x7f628000d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:42:22.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:22.549+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000f5f0 msgr2=0x7f62800056a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:42:22.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:22.549+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 msgr2=0x7f628000d550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:42:22.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:22.549+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000f5f0 0x7f62800056a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:42:22.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:22.549+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 0x7f628000d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:42:23.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:23.350+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 msgr2=0x7f628000d550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:42:23.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:23.350+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000f5f0 msgr2=0x7f62800056a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:42:23.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:23.350+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 0x7f628000d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:42:23.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:23.350+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000f5f0 0x7f62800056a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:42:24.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:24.947+0000 7f62a0c00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 msgr2=0x7f628000d550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:42:24.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:24.947+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 0x7f628000d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:42:24.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:24.947+0000 7f62a0c00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000f5f0 msgr2=0x7f62800056a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:42:24.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:24.947+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000f5f0 0x7f62800056a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:42:24.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:24.947+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f628000c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:42:24.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:24.947+0000 7f62a3400640 1 -- reap_dead start 2024-03-24T01:42:24.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:24.947+0000 7f62a0c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a415f900 con 0x7f62800103d0 2024-03-24T01:42:24.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:24.947+0000 7f62a0c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a415fbe0 con 0x7f6280005c00 2024-03-24T01:42:24.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:24.947+0000 7f62a0c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f6280006610 2024-03-24T01:42:24.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:24.947+0000 7f629be00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 msgr2=0x7f628000c000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:42:24.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:24.947+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800103d0 legacy=0x7f628000eee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:42:24.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:24.947+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:42:24.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:24.947+0000 7f629be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f628000c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:42:25.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:25.148+0000 7f629be00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 msgr2=0x7f628000c000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:42:25.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:25.148+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800103d0 legacy=0x7f628000eee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:42:25.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:25.148+0000 7f629be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f628000c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:42:25.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:25.148+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:42:25.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:25.549+0000 7f629be00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 msgr2=0x7f628000c000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:42:25.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:25.549+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800103d0 legacy=0x7f628000eee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:42:25.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:25.549+0000 7f629be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f628000c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:42:25.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:25.549+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:42:26.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:26.350+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800103d0 legacy=0x7f628000eee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:42:26.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:26.350+0000 7f629be00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 msgr2=0x7f628000c000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:42:26.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:26.350+0000 7f629be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f628000c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:42:26.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:26.350+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:42:27.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:27.947+0000 7f62a0c00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 msgr2=0x7f628000c000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:42:27.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:27.947+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f628000c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:42:27.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:27.947+0000 7f62a0c00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280005c00 legacy=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:42:27.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:27.947+0000 7f62a0c00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800103d0 legacy=0x7f628000eee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:42:27.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:27.947+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000f5f0 0x7f6280005670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:42:27.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:27.947+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 0x7f628000d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:42:27.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:27.947+0000 7f62a0c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f628000f5f0 2024-03-24T01:42:27.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:27.947+0000 7f62a0c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6280008d10 con 0x7f6280007e70 2024-03-24T01:42:27.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:27.947+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000f5f0 msgr2=0x7f6280005670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:42:27.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:27.947+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000f5f0 0x7f6280005670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:42:27.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:27.947+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 msgr2=0x7f628000d550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:42:27.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:27.948+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 0x7f628000d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:42:28.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:28.148+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000f5f0 msgr2=0x7f6280005670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:42:28.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:28.148+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000f5f0 0x7f6280005670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:42:28.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:28.149+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 msgr2=0x7f628000d550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:42:28.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:28.149+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 0x7f628000d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:42:28.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:28.549+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000f5f0 msgr2=0x7f6280005670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:42:28.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:28.549+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000f5f0 0x7f6280005670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:42:28.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:28.550+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 msgr2=0x7f628000d550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:42:28.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:28.550+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 0x7f628000d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:42:29.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:29.350+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000f5f0 msgr2=0x7f6280005670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:42:29.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:29.350+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000f5f0 0x7f6280005670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:42:29.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:29.351+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 msgr2=0x7f628000d550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:42:29.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:29.351+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 0x7f628000d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:42:30.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:30.948+0000 7f62a0c00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 msgr2=0x7f628000d550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:42:30.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:30.948+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 0x7f628000d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:42:30.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:30.948+0000 7f62a0c00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f628000f5f0 msgr2=0x7f6280005670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:42:30.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:30.948+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f628000f5f0 0x7f6280005670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:42:30.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:30.948+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f628000c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:42:30.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:30.948+0000 7f62a3400640 1 -- reap_dead start 2024-03-24T01:42:30.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:30.948+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 0x7f628000d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:42:30.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:30.948+0000 7f62a0c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a40cd490 con 0x7f62800103d0 2024-03-24T01:42:30.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:30.948+0000 7f62a0c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f6280006610 2024-03-24T01:42:30.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:30.948+0000 7f62a0c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6280008d10 con 0x7f6280007e70 2024-03-24T01:42:30.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:30.948+0000 7f62a2a00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 msgr2=0x7f628000c000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:42:30.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:30.948+0000 7f62a2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f628000c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:42:30.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:30.948+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800103d0 legacy=0x7f628000eee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:42:30.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:30.948+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 msgr2=0x7f628000d550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:42:30.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:30.948+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 0x7f628000d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:42:31.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:31.149+0000 7f62a2a00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 msgr2=0x7f628000c000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:42:31.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:31.149+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800103d0 legacy=0x7f628000eee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:42:31.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:31.149+0000 7f62a2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f628000c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:42:31.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:31.149+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 msgr2=0x7f628000d550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:42:31.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:31.149+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 0x7f628000d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:42:31.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:31.550+0000 7f62a2a00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 msgr2=0x7f628000c000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:42:31.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:31.550+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800103d0 legacy=0x7f628000eee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:42:31.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:31.550+0000 7f62a2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f628000c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:42:31.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:31.550+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 msgr2=0x7f628000d550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:42:31.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:31.550+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 0x7f628000d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:42:32.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:32.351+0000 7f62a2a00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 msgr2=0x7f628000c000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:42:32.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:32.351+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800103d0 legacy=0x7f628000eee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:42:32.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:32.351+0000 7f62a2a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f628000c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:42:32.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:32.351+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 msgr2=0x7f628000d550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:42:32.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:32.351+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 0x7f628000d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:42:33.383 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:42:33.391 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:42:33.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:33.948+0000 7f62a0c00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 msgr2=0x7f628000d550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:42:33.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:33.948+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 0x7f628000d550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:42:33.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:33.948+0000 7f62a0c00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 msgr2=0x7f628000c000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:42:33.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:33.948+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280006610 0x7f628000c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:42:33.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:33.948+0000 7f62a0c00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800103d0 legacy=0x7f628000eee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:42:33.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:33.948+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280005c00 0x7f6280005670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:42:33.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:33.948+0000 7f62a0c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a415f900 con 0x7f628000f5f0 2024-03-24T01:42:33.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:33.948+0000 7f62a0c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f6280005c00 2024-03-24T01:42:33.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:33.948+0000 7f629be00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f628000f5f0 legacy=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:42:33.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:33.948+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280005c00 msgr2=0x7f6280005670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:42:33.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:33.948+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280005c00 0x7f6280005670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:42:34.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:34.149+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280005c00 msgr2=0x7f6280005670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:42:34.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:34.149+0000 7f629be00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f628000f5f0 legacy=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:42:34.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:34.149+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280005c00 0x7f6280005670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:42:34.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:34.550+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280005c00 msgr2=0x7f6280005670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:42:34.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:34.550+0000 7f629be00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f628000f5f0 legacy=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:42:34.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:34.550+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280005c00 0x7f6280005670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:42:35.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:35.351+0000 7f629be00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f628000f5f0 legacy=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:42:35.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:35.351+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280005c00 msgr2=0x7f6280005670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:42:35.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:35.351+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280005c00 0x7f6280005670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:42:36.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:36.948+0000 7f62a0c00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280005c00 msgr2=0x7f6280005670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:42:36.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:36.948+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280005c00 0x7f6280005670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:42:36.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:36.948+0000 7f62a0c00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f628000f5f0 legacy=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:42:36.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:36.948+0000 7f62a3400640 1 -- reap_dead start 2024-03-24T01:42:36.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:36.948+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 0x7f6280017a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:42:36.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:36.948+0000 7f62a0c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a40cd490 con 0x7f62800103d0 2024-03-24T01:42:36.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:36.948+0000 7f62a0c00640 1 -- --> v1:172.21.15.120:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a415f900 con 0x7f6280016f50 2024-03-24T01:42:36.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:36.948+0000 7f62a0c00640 1 -- --> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f6280007e70 2024-03-24T01:42:36.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:36.948+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280016f50 legacy=0x7f6280017330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:42:36.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:36.949+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 msgr2=0x7f6280017a40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:42:36.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:36.949+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800103d0 legacy=0x7f628000eee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:42:36.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:36.949+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 0x7f6280017a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:42:37.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:37.150+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800103d0 legacy=0x7f628000eee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:42:37.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:37.150+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 msgr2=0x7f6280017a40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:42:37.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:37.150+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280016f50 legacy=0x7f6280017330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:42:37.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:37.150+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 0x7f6280017a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:42:37.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:37.551+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800103d0 legacy=0x7f628000eee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:42:37.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:37.551+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 msgr2=0x7f6280017a40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:42:37.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:37.551+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280016f50 legacy=0x7f6280017330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:42:37.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:37.551+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 0x7f6280017a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:42:38.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:38.352+0000 7f62a3400640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800103d0 legacy=0x7f628000eee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-24T01:42:38.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:38.352+0000 7f62a2a00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280016f50 legacy=0x7f6280017330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.120:6789/0 2024-03-24T01:42:38.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:38.352+0000 7f629be00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 msgr2=0x7f6280017a40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3301/0 2024-03-24T01:42:38.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:38.352+0000 7f629be00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 0x7f6280017a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:42:39.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:39.949+0000 7f62a0c00640 1 -- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 msgr2=0x7f6280017a40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:42:39.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:39.949+0000 7f62a0c00640 1 --2- >> [v2:172.21.15.120:3301/0,v1:172.21.15.120:6790/0] conn(0x7f6280007e70 0x7f6280017a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:42:39.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:39.949+0000 7f62a0c00640 1 -- >> v1:172.21.15.120:6789/0 conn(0x7f6280016f50 legacy=0x7f6280017330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:42:39.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:39.949+0000 7f62a0c00640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7f62800103d0 legacy=0x7f628000eee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T01:42:39.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:39.949+0000 7f62a0c00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 0x7f628000e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:42:39.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:39.949+0000 7f62a0c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280005c00 0x7f6280005670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:42:39.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:39.949+0000 7f62a0c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a40cd270 con 0x7f6280005c00 2024-03-24T01:42:39.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:39.949+0000 7f62a0c00640 1 -- --> v2:172.21.15.120:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6280008d10 con 0x7f6280006610 2024-03-24T01:42:39.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:39.949+0000 7f629be00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280005c00 msgr2=0x7f6280005670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:42:39.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:39.949+0000 7f629be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280005c00 0x7f6280005670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:42:39.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:39.949+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 msgr2=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:42:39.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:39.949+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 0x7f628000e3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T01:42:40.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:40.150+0000 7f629be00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280005c00 msgr2=0x7f6280005670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:42:40.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:40.150+0000 7f629be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280005c00 0x7f6280005670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:42:40.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:40.150+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 msgr2=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:42:40.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:40.150+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 0x7f628000e3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T01:42:40.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:40.551+0000 7f629be00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280005c00 msgr2=0x7f6280005670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:42:40.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:40.551+0000 7f629be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280005c00 0x7f6280005670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:42:40.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:40.551+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 msgr2=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:42:40.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:40.551+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 0x7f628000e3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T01:42:41.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:41.352+0000 7f629be00640 1 -- >> v2:172.21.15.90:3300/0 conn(0x7f6280005c00 msgr2=0x7f6280005670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.90:3300/0 2024-03-24T01:42:41.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:41.352+0000 7f629be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6280005c00 0x7f6280005670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:42:41.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:41.352+0000 7f62a2a00640 1 -- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 msgr2=0x7f628000e3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.120:3300/0 2024-03-24T01:42:41.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-24T01:42:41.352+0000 7f62a2a00640 1 --2- >> v2:172.21.15.120:3300/0 conn(0x7f6280006610 0x7f628000e3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T01:42:42.835 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-24T01:42:42.835 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-03-24T01:42:42.846 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-24T01:42:42.846 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-24T01:42:42.846 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:42:42.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T01:42:42.848 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a' 2024-03-24T01:42:42.858 INFO:tasks.ceph.mds.a:Stopped 2024-03-24T01:42:42.858 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-24T01:42:42.858 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:42:42.858 INFO:tasks.ceph.mds.c:Stopped 2024-03-24T01:42:42.858 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-24T01:42:42.858 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:42:42.859 INFO:tasks.ceph.mds.b:Stopped 2024-03-24T01:42:42.859 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-24T01:42:42.859 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:42:42.859 INFO:tasks.ceph.mds.d:Stopped 2024-03-24T01:42:42.859 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-24T01:42:42.859 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-24T01:42:42.860 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:42:42.860 INFO:tasks.ceph.osd.0:Stopped 2024-03-24T01:42:42.860 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-24T01:42:42.860 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:42:42.860 INFO:tasks.ceph.osd.1:Stopped 2024-03-24T01:42:42.860 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-24T01:42:42.860 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:42:42.861 INFO:tasks.ceph.osd.2:Stopped 2024-03-24T01:42:42.861 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-24T01:42:42.861 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:42:42.861 INFO:tasks.ceph.osd.3:Stopped 2024-03-24T01:42:42.861 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-24T01:42:42.861 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:42:42.861 INFO:tasks.ceph.osd.4:Stopped 2024-03-24T01:42:42.862 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-24T01:42:42.862 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:42:42.862 INFO:tasks.ceph.osd.5:Stopped 2024-03-24T01:42:42.862 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-24T01:42:42.862 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:42:42.862 INFO:tasks.ceph.osd.6:Stopped 2024-03-24T01:42:42.862 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-24T01:42:42.862 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:42:42.863 INFO:tasks.ceph.osd.7:Stopped 2024-03-24T01:42:42.863 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-24T01:42:42.863 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-24T01:42:42.863 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:42:42.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T01:42:42.863 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y' 2024-03-24T01:42:42.873 INFO:tasks.ceph.mgr.y:Stopped 2024-03-24T01:42:42.873 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-24T01:42:42.873 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:42:42.874 INFO:tasks.ceph.mgr.x:Stopped 2024-03-24T01:42:42.874 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-24T01:42:42.874 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-24T01:42:42.874 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:42:42.874 INFO:tasks.ceph.mon.a:Stopped 2024-03-24T01:42:42.874 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-24T01:42:42.874 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:42:42.875 INFO:tasks.ceph.mon.b:Stopped 2024-03-24T01:42:42.875 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-24T01:42:42.875 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T01:42:42.875 INFO:tasks.ceph.mon.c:Stopped 2024-03-24T01:42:42.875 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-24T01:42:42.875 DEBUG:teuthology.orchestra.run.smithi090:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'bad backtrace on directory inode' | egrep -v 'error reading table object' | egrep -v 'Metadata damage detected' | egrep -v 'slow requests are blocked' | egrep -v 'Behind on trimming' | egrep -v 'error reading sessionmap' | head -n 1 2024-03-24T01:42:42.986 INFO:teuthology.orchestra.run.smithi090.stdout:2024-03-24T01:36:29.300666+0000 mds.d (mds.0) 1 : cluster [WRN] 1 slow requests, 1 included below; oldest blocked for > 181.976948 secs 2024-03-24T01:42:42.987 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2024-03-24T01:42:42.987 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi090.front.sepia.ceph.com 2024-03-24T01:42:42.987 DEBUG:teuthology.orchestra.run.smithi090:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-24T01:42:43.200 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi090.front.sepia.ceph.com 2024-03-24T01:42:43.200 DEBUG:teuthology.orchestra.run.smithi090:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-24T01:42:43.369 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi090.front.sepia.ceph.com 2024-03-24T01:42:43.369 DEBUG:teuthology.orchestra.run.smithi090:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-24T01:42:43.523 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi090.front.sepia.ceph.com 2024-03-24T01:42:43.523 DEBUG:teuthology.orchestra.run.smithi090:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-24T01:42:43.693 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi120.front.sepia.ceph.com 2024-03-24T01:42:43.693 DEBUG:teuthology.orchestra.run.smithi120:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-24T01:42:44.072 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi120.front.sepia.ceph.com 2024-03-24T01:42:44.072 DEBUG:teuthology.orchestra.run.smithi120:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-24T01:42:44.293 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi120.front.sepia.ceph.com 2024-03-24T01:42:44.293 DEBUG:teuthology.orchestra.run.smithi120:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-24T01:42:44.430 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi120.front.sepia.ceph.com 2024-03-24T01:42:44.430 DEBUG:teuthology.orchestra.run.smithi120:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-24T01:42:44.566 INFO:tasks.ceph:Archiving mon data... 2024-03-24T01:42:44.568 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616453/data/mon.a.tgz 2024-03-24T01:42:44.568 DEBUG:teuthology.orchestra.run.smithi090:> mktemp 2024-03-24T01:42:44.626 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/tmp.avAuwQnUvB 2024-03-24T01:42:44.626 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.avAuwQnUvB 2024-03-24T01:42:44.814 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0666 /tmp/tmp.avAuwQnUvB 2024-03-24T01:42:44.964 DEBUG:teuthology.orchestra.remote:smithi090:/tmp/tmp.avAuwQnUvB is 303KB 2024-03-24T01:42:45.073 DEBUG:teuthology.orchestra.run.smithi090:> rm -fr /tmp/tmp.avAuwQnUvB 2024-03-24T01:42:45.128 DEBUG:teuthology.misc:Transferring archived files from smithi120:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616453/data/mon.b.tgz 2024-03-24T01:42:45.128 DEBUG:teuthology.orchestra.run.smithi120:> mktemp 2024-03-24T01:42:45.186 INFO:teuthology.orchestra.run.smithi120.stdout:/tmp/tmp.0bJJZnKAQY 2024-03-24T01:42:45.187 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.0bJJZnKAQY 2024-03-24T01:42:45.369 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 0666 /tmp/tmp.0bJJZnKAQY 2024-03-24T01:42:45.519 DEBUG:teuthology.orchestra.remote:smithi120:/tmp/tmp.0bJJZnKAQY is 312KB 2024-03-24T01:42:45.584 DEBUG:teuthology.orchestra.run.smithi120:> rm -fr /tmp/tmp.0bJJZnKAQY 2024-03-24T01:42:45.639 DEBUG:teuthology.misc:Transferring archived files from smithi120:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616453/data/mon.c.tgz 2024-03-24T01:42:45.639 DEBUG:teuthology.orchestra.run.smithi120:> mktemp 2024-03-24T01:42:45.688 INFO:teuthology.orchestra.run.smithi120.stdout:/tmp/tmp.SHchICFadQ 2024-03-24T01:42:45.689 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.SHchICFadQ 2024-03-24T01:42:45.855 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 0666 /tmp/tmp.SHchICFadQ 2024-03-24T01:42:46.004 DEBUG:teuthology.orchestra.remote:smithi120:/tmp/tmp.SHchICFadQ is 312KB 2024-03-24T01:42:46.075 DEBUG:teuthology.orchestra.run.smithi120:> rm -fr /tmp/tmp.SHchICFadQ 2024-03-24T01:42:46.127 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-24T01:42:46.128 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-24T01:42:46.135 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-24T01:42:46.288 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-24T01:42:46.345 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-24T01:42:46.346 INFO:tasks.ceph:Archiving crash dumps... 2024-03-24T01:42:46.347 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/lib/ceph/crash to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616453/remote/smithi090/crash 2024-03-24T01:42:46.348 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-24T01:42:46.490 DEBUG:teuthology.misc:Transferring archived files from smithi120:/var/lib/ceph/crash to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616453/remote/smithi120/crash 2024-03-24T01:42:46.491 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-24T01:42:46.586 INFO:tasks.ceph:Shutting down logrotate 2024-03-24T01:42:46.587 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:42:46.598 DEBUG:teuthology.orchestra.run.smithi120:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T01:42:46.786 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-24T01:42:46.891 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-24T01:42:46.992 INFO:tasks.ceph:Compressing logs... 2024-03-24T01:42:46.993 DEBUG:teuthology.orchestra.run.smithi090:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T01:42:47.007 DEBUG:teuthology.orchestra.run.smithi120:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T01:42:47.093 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63586.log 2024-03-24T01:42:47.095 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52229.log 2024-03-24T01:42:47.095 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.63586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42589.log 2024-03-24T01:42:47.096 INFO:teuthology.orchestra.run.smithi090.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.63586.log.gz 2024-03-24T01:42:47.097 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48710.log 2024-03-24T01:42:47.097 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52229.log.gz 2024-03-24T01:42:47.097 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54439.log 2024-03-24T01:42:47.098 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42589.log.gz 2024-03-24T01:42:47.100 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48710.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48710.log.gz 2024-03-24T01:42:47.100 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.56445.log 2024-03-24T01:42:47.100 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.54439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45448.log 2024-03-24T01:42:47.101 INFO:teuthology.orchestra.run.smithi090.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54439.log.gz 2024-03-24T01:42:47.103 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.0.56445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55246.log 2024-03-24T01:42:47.103 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64837.log 2024-03-24T01:42:47.103 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45448.log.gz 2024-03-24T01:42:47.104 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-24T01:42:47.105 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36980.log 2024-03-24T01:42:47.107 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.55246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51857.log 2024-03-24T01:42:47.107 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55246.log.gz 2024-03-24T01:42:47.107 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.64837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43184.log 2024-03-24T01:42:47.107 INFO:teuthology.orchestra.run.smithi090.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.64837.log.gz 2024-03-24T01:42:47.107 INFO:teuthology.orchestra.run.smithi090.stderr: 92.7% -- replaced with /var/log/ceph/ceph-client.0.56445.log.gz 2024-03-24T01:42:47.107 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53230.log 2024-03-24T01:42:47.108 INFO:teuthology.orchestra.run.smithi090.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51857.log.gz 2024-03-24T01:42:47.108 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51563.log 2024-03-24T01:42:47.112 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36940.log 2024-03-24T01:42:47.112 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.36980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37060.log 2024-03-24T01:42:47.112 INFO:teuthology.orchestra.run.smithi120.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36980.log.gz 2024-03-24T01:42:47.113 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.36940.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36940.log.gz 2024-03-24T01:42:47.113 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37020.log 2024-03-24T01:42:47.113 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.37060.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-24T01:42:47.113 INFO:teuthology.orchestra.run.smithi120.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37060.log.gz 2024-03-24T01:42:47.113 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43184.log.gz 2024-03-24T01:42:47.114 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.53230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60806.log 2024-03-24T01:42:47.114 INFO:teuthology.orchestra.run.smithi090.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53230.log.gz 2024-03-24T01:42:47.114 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50219.log 2024-03-24T01:42:47.114 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51563.log.gz 2024-03-24T01:42:47.114 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.60806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42632.log 2024-03-24T01:42:47.114 INFO:teuthology.orchestra.run.smithi090.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.60806.log.gz 2024-03-24T01:42:47.115 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.50219.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-24T01:42:47.115 INFO:teuthology.orchestra.run.smithi090.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50219.log.gz 2024-03-24T01:42:47.118 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49451.log 2024-03-24T01:42:47.118 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42632.log.gz 2024-03-24T01:42:47.119 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48080.log 2024-03-24T01:42:47.119 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59149.log 2024-03-24T01:42:47.119 INFO:teuthology.orchestra.run.smithi090.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49451.log.gz 2024-03-24T01:42:47.119 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61614.log 2024-03-24T01:42:47.121 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.37020.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-03-24T01:42:47.121 INFO:teuthology.orchestra.run.smithi120.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37020.log.gz 2024-03-24T01:42:47.121 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-24T01:42:47.121 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37100.log 2024-03-24T01:42:47.121 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-24T01:42:47.122 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.37100.log: 93.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-24T01:42:47.122 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-24T01:42:47.122 INFO:teuthology.orchestra.run.smithi120.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37100.log.gz 2024-03-24T01:42:47.122 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.43535.log 2024-03-24T01:42:47.122 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-24T01:42:47.124 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.3.43535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.43083.log 2024-03-24T01:42:47.124 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-mgr.x.log: 92.3% -- replaced with /var/log/ceph/ceph-client.3.43535.log.gz 2024-03-24T01:42:47.127 INFO:teuthology.orchestra.run.smithi090.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48080.log.gz 2024-03-24T01:42:47.128 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.59149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51815.log 2024-03-24T01:42:47.128 INFO:teuthology.orchestra.run.smithi090.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.59149.log.gz 2024-03-24T01:42:47.128 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.61614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48374.log 2024-03-24T01:42:47.128 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61614.log.gz 2024-03-24T01:42:47.128 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66565.log 2024-03-24T01:42:47.128 INFO:teuthology.orchestra.run.smithi090.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51815.log.gz 2024-03-24T01:42:47.129 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66389.log 2024-03-24T01:42:47.129 INFO:teuthology.orchestra.run.smithi090.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48374.log.gz 2024-03-24T01:42:47.129 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.66565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44848.log 2024-03-24T01:42:47.129 INFO:teuthology.orchestra.run.smithi090.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.66565.log.gz 2024-03-24T01:42:47.129 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.66389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47996.log 2024-03-24T01:42:47.129 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66389.log.gz 2024-03-24T01:42:47.129 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44848.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44848.log.gz 2024-03-24T01:42:47.130 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56868.log 2024-03-24T01:42:47.130 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.47996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60173.log 2024-03-24T01:42:47.130 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47996.log.gz 2024-03-24T01:42:47.130 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.56868.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56868.log.gz 2024-03-24T01:42:47.130 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50335.log 2024-03-24T01:42:47.135 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.1.43083.log: 91.2% -- replaced with /var/log/ceph/ceph-client.1.43083.log.gz 2024-03-24T01:42:47.135 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-24T01:42:47.135 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-24T01:42:47.135 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37140.log 2024-03-24T01:42:47.135 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36900.log 2024-03-24T01:42:47.135 INFO:teuthology.orchestra.run.smithi120.stderr: 93.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-24T01:42:47.137 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.60173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60740.log 2024-03-24T01:42:47.137 INFO:teuthology.orchestra.run.smithi090.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.60173.log.gz 2024-03-24T01:42:47.137 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.50335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50093.log 2024-03-24T01:42:47.137 INFO:teuthology.orchestra.run.smithi090.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50335.log.gz 2024-03-24T01:42:47.137 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.60740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52355.log 2024-03-24T01:42:47.138 INFO:teuthology.orchestra.run.smithi090.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.60740.log.gz 2024-03-24T01:42:47.138 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.50093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47844.log 2024-03-24T01:42:47.138 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50093.log.gz 2024-03-24T01:42:47.138 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42370.log 2024-03-24T01:42:47.138 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52355.log.gz 2024-03-24T01:42:47.138 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.47844.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47844.log.gz 2024-03-24T01:42:47.138 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48878.log 2024-03-24T01:42:47.139 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52583.log 2024-03-24T01:42:47.139 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42370.log.gz 2024-03-24T01:42:47.139 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66871.log 2024-03-24T01:42:47.139 INFO:teuthology.orchestra.run.smithi090.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48878.log.gz 2024-03-24T01:42:47.142 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48458.log 2024-03-24T01:42:47.142 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52583.log.gz 2024-03-24T01:42:47.142 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.66871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36958.log 2024-03-24T01:42:47.143 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66871.log.gz 2024-03-24T01:42:47.145 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.37140.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37140.log.gz 2024-03-24T01:42:47.145 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-03-24T01:42:47.145 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37180.log 2024-03-24T01:42:47.145 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-24T01:42:47.146 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/ceph-client.admin.37180.log: 87.7%/var/log/ceph/ceph-client.admin.36900.log: /var/log/ceph/ceph-mon.c.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36900.log.gz 2024-03-24T01:42:47.146 INFO:teuthology.orchestra.run.smithi120.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37180.log.gz 2024-03-24T01:42:47.149 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64011.log 2024-03-24T01:42:47.149 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48458.log.gz 2024-03-24T01:42:47.150 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48248.log 2024-03-24T01:42:47.150 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.64011.log: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.36958.log.gz 2024-03-24T01:42:47.150 INFO:teuthology.orchestra.run.smithi090.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64011.log.gz 2024-03-24T01:42:47.150 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52144.log 2024-03-24T01:42:47.150 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44064.log 2024-03-24T01:42:47.150 INFO:teuthology.orchestra.run.smithi090.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48248.log.gz 2024-03-24T01:42:47.150 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58873.log 2024-03-24T01:42:47.151 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52144.log.gz 2024-03-24T01:42:47.151 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46612.log 2024-03-24T01:42:47.151 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44064.log.gz 2024-03-24T01:42:47.151 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.58873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58495.log 2024-03-24T01:42:47.151 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58873.log.gz 2024-03-24T01:42:47.156 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53062.log 2024-03-24T01:42:47.156 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46612.log.gz 2024-03-24T01:42:47.156 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.58495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45449.log 2024-03-24T01:42:47.156 INFO:teuthology.orchestra.run.smithi090.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58495.log.gz 2024-03-24T01:42:47.156 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.53062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51773.log 2024-03-24T01:42:47.157 INFO:teuthology.orchestra.run.smithi090.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53062.log.gz 2024-03-24T01:42:47.157 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45449.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45449.log.gz 2024-03-24T01:42:47.157 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66712.log 2024-03-24T01:42:47.157 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59041.log 2024-03-24T01:42:47.157 INFO:teuthology.orchestra.run.smithi090.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51773.log.gz 2024-03-24T01:42:47.157 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.66712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44932.log 2024-03-24T01:42:47.157 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66712.log.gz 2024-03-24T01:42:47.158 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.59041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66544.log 2024-03-24T01:42:47.158 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59041.log.gz 2024-03-24T01:42:47.158 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60299.log 2024-03-24T01:42:47.158 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44932.log.gz 2024-03-24T01:42:47.161 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.66544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48752.log 2024-03-24T01:42:47.161 INFO:teuthology.orchestra.run.smithi090.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.66544.log.gz 2024-03-24T01:42:47.161 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.60299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45785.log 2024-03-24T01:42:47.161 INFO:teuthology.orchestra.run.smithi090.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60299.log.gz 2024-03-24T01:42:47.161 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48752.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48752.log.gz 2024-03-24T01:42:47.162 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64393.log 2024-03-24T01:42:47.162 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57333.log 2024-03-24T01:42:47.162 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45785.log.gz 2024-03-24T01:42:47.162 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.64393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59887.log 2024-03-24T01:42:47.163 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64393.log.gz 2024-03-24T01:42:47.164 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.57333.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57333.log.gz 2024-03-24T01:42:47.164 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46528.log 2024-03-24T01:42:47.164 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.59887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55864.log 2024-03-24T01:42:47.166 INFO:teuthology.orchestra.run.smithi090.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59887.log.gz 2024-03-24T01:42:47.166 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46528.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46528.log.gz 2024-03-24T01:42:47.166 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52806.log 2024-03-24T01:42:47.166 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.55864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60410.log 2024-03-24T01:42:47.166 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55864.log.gz 2024-03-24T01:42:47.167 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66087.log 2024-03-24T01:42:47.167 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52806.log.gz 2024-03-24T01:42:47.168 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.60410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63334.log 2024-03-24T01:42:47.168 INFO:teuthology.orchestra.run.smithi090.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60410.log.gz 2024-03-24T01:42:47.169 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.66087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59989.log 2024-03-24T01:42:47.169 INFO:teuthology.orchestra.run.smithi090.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.66087.log.gz 2024-03-24T01:42:47.170 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.63334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46150.log 2024-03-24T01:42:47.170 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63334.log.gz 2024-03-24T01:42:47.171 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.59989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52271.log 2024-03-24T01:42:47.171 INFO:teuthology.orchestra.run.smithi090.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.59989.log.gz 2024-03-24T01:42:47.172 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46150.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46150.log.gzgzip 2024-03-24T01:42:47.172 INFO:teuthology.orchestra.run.smithi090.stderr: -5 --verbose -- /var/log/ceph/ceph-client.0.53657.log 2024-03-24T01:42:47.174 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48164.log 2024-03-24T01:42:47.174 INFO:teuthology.orchestra.run.smithi090.stderr: 88.1%/var/log/ceph/ceph-client.0.53657.log: -- replaced with /var/log/ceph/ceph-client.admin.52271.log.gz 2024-03-24T01:42:47.174 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48668.log 2024-03-24T01:42:47.176 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48164.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.48164.log.gz 2024-03-24T01:42:47.176 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49393.log 2024-03-24T01:42:47.176 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61289.log 2024-03-24T01:42:47.177 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49393.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48668.log.gz 2024-03-24T01:42:47.178 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49393.log.gz 2024-03-24T01:42:47.178 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59947.log 2024-03-24T01:42:47.179 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.61289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61343.log 2024-03-24T01:42:47.180 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.59947.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.61289.log.gz 2024-03-24T01:42:47.180 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59947.log.gz 2024-03-24T01:42:47.181 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54930.log 2024-03-24T01:42:47.182 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53104.log 2024-03-24T01:42:47.183 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.61343.log: /var/log/ceph/ceph-client.admin.54930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44764.log 2024-03-24T01:42:47.183 INFO:teuthology.orchestra.run.smithi090.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.61343.log.gz 2024-03-24T01:42:47.184 INFO:teuthology.orchestra.run.smithi090.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.54930.log.gz 2024-03-24T01:42:47.184 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.53104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49781.log 2024-03-24T01:42:47.184 INFO:teuthology.orchestra.run.smithi090.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53104.log.gz 2024-03-24T01:42:47.185 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44764.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44764.log.gz 2024-03-24T01:42:47.185 INFO:teuthology.orchestra.run.smithi090.stderr: 93.3% -- replaced with /var/log/ceph/ceph-client.0.53657.log.gz 2024-03-24T01:42:47.187 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54776.log 2024-03-24T01:42:47.187 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57730.log 2024-03-24T01:42:47.192 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.54776.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49781.log.gz 2024-03-24T01:42:47.193 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66188.log 2024-03-24T01:42:47.193 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.57730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57856.log 2024-03-24T01:42:47.193 INFO:teuthology.orchestra.run.smithi090.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57730.log.gz 2024-03-24T01:42:47.193 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.66188.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.54776.log.gz 2024-03-24T01:42:47.193 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57772.log 2024-03-24T01:42:47.193 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.57856.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66188.log.gz 2024-03-24T01:42:47.194 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57856.log.gz 2024-03-24T01:42:47.194 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54565.log 2024-03-24T01:42:47.194 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.57772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55922.log 2024-03-24T01:42:47.195 INFO:teuthology.orchestra.run.smithi120.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-03-24T01:42:47.198 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57772.log.gz 2024-03-24T01:42:47.198 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.54565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66649.log 2024-03-24T01:42:47.198 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54565.log.gz 2024-03-24T01:42:47.198 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.55922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66913.log 2024-03-24T01:42:47.198 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.66649.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55922.log.gz 2024-03-24T01:42:47.199 INFO:teuthology.orchestra.run.smithi090.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.66649.log.gz 2024-03-24T01:42:47.199 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48416.log 2024-03-24T01:42:47.199 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.66913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44722.log 2024-03-24T01:42:47.199 INFO:teuthology.orchestra.run.smithi090.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.66913.log.gz 2024-03-24T01:42:47.199 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51731.log 2024-03-24T01:42:47.199 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48416.log.gz 2024-03-24T01:42:47.201 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52439.log 2024-03-24T01:42:47.201 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51731.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44722.log.gz 2024-03-24T01:42:47.201 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51731.log.gz 2024-03-24T01:42:47.201 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42674.log 2024-03-24T01:42:47.201 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48206.log 2024-03-24T01:42:47.202 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52439.log.gz 2024-03-24T01:42:47.203 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42674.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42674.log.gz 2024-03-24T01:42:47.203 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45446.log 2024-03-24T01:42:47.204 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51101.log 2024-03-24T01:42:47.204 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45446.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48206.log.gz 2024-03-24T01:42:47.204 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45446.log.gz 2024-03-24T01:42:47.204 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62681.log 2024-03-24T01:42:47.205 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51605.log 2024-03-24T01:42:47.205 INFO:teuthology.orchestra.run.smithi090.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51101.log.gz 2024-03-24T01:42:47.206 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.62681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54661.log 2024-03-24T01:42:47.206 INFO:teuthology.orchestra.run.smithi090.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62681.log.gz 2024-03-24T01:42:47.207 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48542.log 2024-03-24T01:42:47.208 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.54661.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51605.log.gz 2024-03-24T01:42:47.209 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54661.log.gz 2024-03-24T01:42:47.209 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64097.log 2024-03-24T01:42:47.209 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61121.log 2024-03-24T01:42:47.210 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48542.log.gz 2024-03-24T01:42:47.210 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.64097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55822.log 2024-03-24T01:42:47.211 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64097.log.gz 2024-03-24T01:42:47.212 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.61121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57940.log 2024-03-24T01:42:47.213 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.55822.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.61121.log.gz 2024-03-24T01:42:47.213 INFO:teuthology.orchestra.run.smithi090.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55822.log.gz 2024-03-24T01:42:47.213 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52397.log 2024-03-24T01:42:47.214 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.57940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62834.log 2024-03-24T01:42:47.214 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57940.log.gz 2024-03-24T01:42:47.215 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50377.log 2024-03-24T01:42:47.215 INFO:teuthology.orchestra.run.smithi090.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.52397.log.gz 2024-03-24T01:42:47.216 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.62834.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62834.log.gz 2024-03-24T01:42:47.216 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45111.log 2024-03-24T01:42:47.217 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.50377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62513.log 2024-03-24T01:42:47.217 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45111.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50377.log.gz 2024-03-24T01:42:47.218 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45111.log.gz 2024-03-24T01:42:47.218 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52523.log 2024-03-24T01:42:47.219 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.62513.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.62513.log.gz 2024-03-24T01:42:47.219 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54856.log 2024-03-24T01:42:47.220 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51983.log 2024-03-24T01:42:47.221 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52523.log.gz 2024-03-24T01:42:47.221 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.54856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50729.log 2024-03-24T01:42:47.221 INFO:teuthology.orchestra.run.smithi090.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.54856.log.gz 2024-03-24T01:42:47.222 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48038.log 2024-03-24T01:42:47.222 INFO:teuthology.orchestra.run.smithi090.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.51983.log.gz 2024-03-24T01:42:47.223 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.50729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56168.log 2024-03-24T01:42:47.224 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50729.log.gz 2024-03-24T01:42:47.224 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47332.log 2024-03-24T01:42:47.224 INFO:teuthology.orchestra.run.smithi090.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48038.log.gz 2024-03-24T01:42:47.225 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.56168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42758.log 2024-03-24T01:42:47.225 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56168.log.gz 2024-03-24T01:42:47.227 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.47332.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47332.log.gz 2024-03-24T01:42:47.227 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60914.log 2024-03-24T01:42:47.228 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57147.log 2024-03-24T01:42:47.228 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.60914.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42758.log.gz 2024-03-24T01:42:47.229 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.47740.log 2024-03-24T01:42:47.229 INFO:teuthology.orchestra.run.smithi090.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.60914.log.gz 2024-03-24T01:42:47.230 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.57147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66331.log 2024-03-24T01:42:47.230 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57147.log.gz 2024-03-24T01:42:47.231 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.2.47740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64183.log 2024-03-24T01:42:47.231 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.66331.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66331.log.gz 2024-03-24T01:42:47.232 INFO:teuthology.orchestra.run.smithi090.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.2.47740.log.gz 2024-03-24T01:42:47.232 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45776.log 2024-03-24T01:42:47.233 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.64183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54893.log 2024-03-24T01:42:47.233 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64183.log.gz 2024-03-24T01:42:47.234 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61370.log 2024-03-24T01:42:47.234 INFO:teuthology.orchestra.run.smithi090.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45776.log.gz 2024-03-24T01:42:47.235 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.54893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43334.log 2024-03-24T01:42:47.235 INFO:teuthology.orchestra.run.smithi090.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.54893.log.gz 2024-03-24T01:42:47.235 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.61370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45435.log 2024-03-24T01:42:47.236 INFO:teuthology.orchestra.run.smithi090.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.61370.log.gz 2024-03-24T01:42:47.237 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43334.log: 88.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.43334.log.gz 2024-03-24T01:42:47.237 INFO:teuthology.orchestra.run.smithi090.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.55696.log 2024-03-24T01:42:47.238 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61316.log 2024-03-24T01:42:47.239 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.55696.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45435.log.gz 2024-03-24T01:42:47.240 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55696.log.gz 2024-03-24T01:42:47.240 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57501.log 2024-03-24T01:42:47.240 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.61316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50009.log 2024-03-24T01:42:47.242 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.57501.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57501.log.gz 2024-03-24T01:42:47.242 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66101.log 2024-03-24T01:42:47.242 INFO:teuthology.orchestra.run.smithi090.stderr: 92.9% -- replaced with /var/log/ceph/ceph-client.admin.61316.log.gz 2024-03-24T01:42:47.244 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.50009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50771.log 2024-03-24T01:42:47.245 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.66101.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50009.log.gz 2024-03-24T01:42:47.245 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.66101.log.gz 2024-03-24T01:42:47.245 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37222.log 2024-03-24T01:42:47.246 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.50771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62918.log 2024-03-24T01:42:47.246 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50771.log.gz 2024-03-24T01:42:47.248 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37222.log: gzip 87.7% -5 -- replaced with /var/log/ceph/ceph-client.admin.37222.log.gz 2024-03-24T01:42:47.248 INFO:teuthology.orchestra.run.smithi090.stderr: --verbose -- /var/log/ceph/ceph-client.admin.52671.log 2024-03-24T01:42:47.249 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.62918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45774.log 2024-03-24T01:42:47.249 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52671.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62918.log.gz 2024-03-24T01:42:47.250 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52671.log.gz 2024-03-24T01:42:47.250 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54818.log 2024-03-24T01:42:47.252 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57291.log 2024-03-24T01:42:47.252 INFO:teuthology.orchestra.run.smithi090.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45774.log.gz 2024-03-24T01:42:47.252 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.54818.log: 89.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.54818.log.gz 2024-03-24T01:42:47.252 INFO:teuthology.orchestra.run.smithi090.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.64351.log 2024-03-24T01:42:47.252 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.57291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42974.log 2024-03-24T01:42:47.252 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.64351.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57291.log.gz 2024-03-24T01:42:47.254 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.64351.log.gz 2024-03-24T01:42:47.254 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62876.log 2024-03-24T01:42:47.254 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56994.log 2024-03-24T01:42:47.255 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42974.log.gz 2024-03-24T01:42:47.255 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.62876.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.62876.log.gz 2024-03-24T01:42:47.255 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50891.log 2024-03-24T01:42:47.285 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.56994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50461.log 2024-03-24T01:42:47.286 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.50891.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56994.log.gz 2024-03-24T01:42:47.286 INFO:teuthology.orchestra.run.smithi090.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50891.log.gz 2024-03-24T01:42:47.286 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46402.log 2024-03-24T01:42:47.286 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.50461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51017.log 2024-03-24T01:42:47.286 INFO:teuthology.orchestra.run.smithi090.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50461.log.gz 2024-03-24T01:42:47.287 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45441.log 2024-03-24T01:42:47.287 INFO:teuthology.orchestra.run.smithi090.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46402.log.gz 2024-03-24T01:42:47.287 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63044.log 2024-03-24T01:42:47.287 INFO:teuthology.orchestra.run.smithi090.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51017.log.gz 2024-03-24T01:42:47.287 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44680.log 2024-03-24T01:42:47.288 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45441.log.gz 2024-03-24T01:42:47.288 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.63044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46108.log 2024-03-24T01:42:47.288 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63044.log.gz 2024-03-24T01:42:47.288 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44680.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-24T01:42:47.288 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44680.log.gz 2024-03-24T01:42:47.289 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46108.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46108.log.gz 2024-03-24T01:42:47.289 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50933.log 2024-03-24T01:42:47.289 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63723.log 2024-03-24T01:42:47.290 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.50933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52313.log 2024-03-24T01:42:47.290 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50933.log.gz 2024-03-24T01:42:47.290 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.63723.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.63723.log.gz 2024-03-24T01:42:47.291 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50051.log 2024-03-24T01:42:47.291 INFO:teuthology.orchestra.run.smithi090.stderr: 89.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-24T01:42:47.291 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56048.log 2024-03-24T01:42:47.291 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52313.log.gz 2024-03-24T01:42:47.292 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.50051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62555.log 2024-03-24T01:42:47.292 INFO:teuthology.orchestra.run.smithi090.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50051.log.gz 2024-03-24T01:42:47.292 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.56048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48122.log 2024-03-24T01:42:47.292 INFO:teuthology.orchestra.run.smithi090.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.56048.log.gz 2024-03-24T01:42:47.293 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.62555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50645.log 2024-03-24T01:42:47.293 INFO:teuthology.orchestra.run.smithi090.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.62555.log.gz 2024-03-24T01:42:47.293 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48122.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48122.log.gz 2024-03-24T01:42:47.293 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.55523.log 2024-03-24T01:42:47.293 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.50645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64687.log 2024-03-24T01:42:47.294 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50645.log.gz 2024-03-24T01:42:47.294 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.0.55523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49967.log 2024-03-24T01:42:47.294 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.64687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.65114.log 2024-03-24T01:42:47.295 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64687.log.gz 2024-03-24T01:42:47.295 INFO:teuthology.orchestra.run.smithi090.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.0.55523.log.gz 2024-03-24T01:42:47.295 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49088.log 2024-03-24T01:42:47.295 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49967.log.gz 2024-03-24T01:42:47.295 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.0.65114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63146.log 2024-03-24T01:42:47.296 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49577.log 2024-03-24T01:42:47.296 INFO:teuthology.orchestra.run.smithi090.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49088.log.gz 2024-03-24T01:42:47.296 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.63146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51311.log 2024-03-24T01:42:47.296 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63146.log.gz 2024-03-24T01:42:47.297 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61768.log 2024-03-24T01:42:47.297 INFO:teuthology.orchestra.run.smithi090.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49577.log.gz 2024-03-24T01:42:47.297 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45783.log 2024-03-24T01:42:47.297 INFO:teuthology.orchestra.run.smithi090.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51311.log.gz 2024-03-24T01:42:47.297 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.61768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46696.log 2024-03-24T01:42:47.298 INFO:teuthology.orchestra.run.smithi090.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.61768.log.gz 2024-03-24T01:42:47.298 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52978.log 2024-03-24T01:42:47.298 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45783.log.gz 2024-03-24T01:42:47.299 INFO:teuthology.orchestra.run.smithi120.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-24T01:42:47.316 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46318.log 2024-03-24T01:42:47.317 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52978.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.46696.log.gz 2024-03-24T01:42:47.317 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63926.log 2024-03-24T01:42:47.317 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52978.log.gz 2024-03-24T01:42:47.317 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49865.log 2024-03-24T01:42:47.318 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46318.log: 88.1%/var/log/ceph/ceph-client.admin.63926.log: -- replaced with /var/log/ceph/ceph-client.admin.46318.log.gz 2024-03-24T01:42:47.318 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63926.log.gz 2024-03-24T01:42:47.318 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55964.log 2024-03-24T01:42:47.318 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43226.log 2024-03-24T01:42:47.318 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49865.log.gz 2024-03-24T01:42:47.319 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.55964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44266.log 2024-03-24T01:42:47.319 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55964.log.gz 2024-03-24T01:42:47.319 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62792.log 2024-03-24T01:42:47.319 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43226.log.gz 2024-03-24T01:42:47.320 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51521.log 2024-03-24T01:42:47.320 INFO:teuthology.orchestra.run.smithi090.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44266.log.gz 2024-03-24T01:42:47.320 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.62792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44022.log 2024-03-24T01:42:47.320 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62792.log.gz 2024-03-24T01:42:47.320 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58705.log 2024-03-24T01:42:47.321 INFO:teuthology.orchestra.run.smithi090.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51521.log.gz 2024-03-24T01:42:47.321 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50687.log 2024-03-24T01:42:47.321 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44022.log.gz 2024-03-24T01:42:47.321 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.58705.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58705.log.gz 2024-03-24T01:42:47.321 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43766.log 2024-03-24T01:42:47.322 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.50687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60257.log 2024-03-24T01:42:47.322 INFO:teuthology.orchestra.run.smithi090.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50687.log.gz 2024-03-24T01:42:47.322 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64477.log 2024-03-24T01:42:47.322 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.60257.log: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.43766.log.gz 2024-03-24T01:42:47.322 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63376.log 2024-03-24T01:42:47.323 INFO:teuthology.orchestra.run.smithi090.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.60257.log.gz 2024-03-24T01:42:47.323 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.64477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51941.log 2024-03-24T01:42:47.323 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.64477.log.gz 2024-03-24T01:42:47.323 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.63376.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.63376.log.gz 2024-03-24T01:42:47.324 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61148.log 2024-03-24T01:42:47.324 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51479.log 2024-03-24T01:42:47.324 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51941.log.gz 2024-03-24T01:42:47.324 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.61148.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.61148.log.gz 2024-03-24T01:42:47.324 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63002.log 2024-03-24T01:42:47.325 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44554.log 2024-03-24T01:42:47.325 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51479.log.gz 2024-03-24T01:42:47.325 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.63002.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.63002.log.gz 2024-03-24T01:42:47.325 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61403.log 2024-03-24T01:42:47.326 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49493.log 2024-03-24T01:42:47.326 INFO:teuthology.orchestra.run.smithi090.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44554.log.gz 2024-03-24T01:42:47.326 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.61403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56826.log 2024-03-24T01:42:47.326 INFO:teuthology.orchestra.run.smithi090.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.61403.log.gz 2024-03-24T01:42:47.326 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55738.log 2024-03-24T01:42:47.326 INFO:teuthology.orchestra.run.smithi090.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.49493.log.gz 2024-03-24T01:42:47.327 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.56826.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56826.log.gz 2024-03-24T01:42:47.327 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63628.log 2024-03-24T01:42:47.327 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.55738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49004.log 2024-03-24T01:42:47.327 INFO:teuthology.orchestra.run.smithi090.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55738.log.gz 2024-03-24T01:42:47.328 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.63628.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63628.log.gz 2024-03-24T01:42:47.328 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57814.log 2024-03-24T01:42:47.341 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48920.log 2024-03-24T01:42:47.341 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49004.log.gz 2024-03-24T01:42:47.341 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.57814.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57814.log.gz 2024-03-24T01:42:47.342 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65541.log 2024-03-24T01:42:47.342 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64267.log 2024-03-24T01:42:47.342 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48920.log.gz 2024-03-24T01:42:47.342 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.65541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55780.log 88.7% 2024-03-24T01:42:47.343 INFO:teuthology.orchestra.run.smithi090.stderr: -- replaced with /var/log/ceph/ceph-client.admin.65541.log.gz 2024-03-24T01:42:47.343 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.64267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60073.log 2024-03-24T01:42:47.343 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64267.log.gz 2024-03-24T01:42:47.343 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.55780.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55780.log.gz 2024-03-24T01:42:47.343 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50419.log 2024-03-24T01:42:47.344 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.60073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42289.log 2024-03-24T01:42:47.344 INFO:teuthology.orchestra.run.smithi090.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60073.log.gz 2024-03-24T01:42:47.344 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.50419.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50419.log.gz 2024-03-24T01:42:47.344 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49351.log 2024-03-24T01:42:47.345 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55092.log 2024-03-24T01:42:47.345 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42289.log.gz 2024-03-24T01:42:47.345 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58453.log 2024-03-24T01:42:47.345 INFO:teuthology.orchestra.run.smithi090.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49351.log.gz 2024-03-24T01:42:47.345 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.55092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56910.log 2024-03-24T01:42:47.346 INFO:teuthology.orchestra.run.smithi090.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55092.log.gz 2024-03-24T01:42:47.346 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.58453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61262.log 2024-03-24T01:42:47.346 INFO:teuthology.orchestra.run.smithi090.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58453.log.gz 2024-03-24T01:42:47.346 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.56910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46192.log 2024-03-24T01:42:47.347 INFO:teuthology.orchestra.run.smithi090.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.56910.log.gz 2024-03-24T01:42:47.347 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.61262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45784.log 2024-03-24T01:42:47.347 INFO:teuthology.orchestra.run.smithi090.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.61262.log.gz 2024-03-24T01:42:47.347 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42802.log 2024-03-24T01:42:47.347 INFO:teuthology.orchestra.run.smithi090.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46192.log.gz 2024-03-24T01:42:47.347 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50603.log 2024-03-24T01:42:47.348 INFO:teuthology.orchestra.run.smithi090.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45784.log.gz 2024-03-24T01:42:47.348 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57898.log 2024-03-24T01:42:47.348 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42802.log.gz 2024-03-24T01:42:47.348 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.50603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63544.log 2024-03-24T01:42:47.349 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50603.log.gz 2024-03-24T01:42:47.349 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.57898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46824.log 2024-03-24T01:42:47.349 INFO:teuthology.orchestra.run.smithi090.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57898.log.gz 2024-03-24T01:42:47.349 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.63544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58035.log 2024-03-24T01:42:47.350 INFO:teuthology.orchestra.run.smithi090.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63544.log.gz 2024-03-24T01:42:47.362 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60860.log 2024-03-24T01:42:47.362 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46824.log.gz 2024-03-24T01:42:47.362 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.58035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59083.log 2024-03-24T01:42:47.363 INFO:teuthology.orchestra.run.smithi090.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.58035.log.gz 2024-03-24T01:42:47.363 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.60860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53188.log 2024-03-24T01:42:47.363 INFO:teuthology.orchestra.run.smithi090.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.60860.log.gz 2024-03-24T01:42:47.363 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.59083.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-24T01:42:47.364 INFO:teuthology.orchestra.run.smithi090.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.59083.log.gz 2024-03-24T01:42:47.364 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.53188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50503.log 2024-03-24T01:42:47.364 INFO:teuthology.orchestra.run.smithi090.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.53188.log.gz 2024-03-24T01:42:47.364 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66955.log 2024-03-24T01:42:47.364 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.50503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57105.log 2024-03-24T01:42:47.365 INFO:teuthology.orchestra.run.smithi090.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50503.log.gz 2024-03-24T01:42:47.365 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.66955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45772.log 2024-03-24T01:42:47.365 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.66955.log.gz 2024-03-24T01:42:47.365 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.57105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51143.log 2024-03-24T01:42:47.365 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45772.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57105.log.gz 2024-03-24T01:42:47.365 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45104.log 2024-03-24T01:42:47.366 INFO:teuthology.orchestra.run.smithi090.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.45772.log.gz 2024-03-24T01:42:47.366 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54684.log 2024-03-24T01:42:47.366 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51143.log.gz 2024-03-24T01:42:47.366 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48290.log 2024-03-24T01:42:47.367 INFO:teuthology.orchestra.run.smithi090.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45104.log.gz 2024-03-24T01:42:47.367 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.54684.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.54684.log.gz 2024-03-24T01:42:47.367 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42547.log 2024-03-24T01:42:47.367 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43938.log 2024-03-24T01:42:47.367 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42547.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48290.log.gz 2024-03-24T01:42:47.368 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42547.log.gz 2024-03-24T01:42:47.368 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62960.log 2024-03-24T01:42:47.368 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63418.log 2024-03-24T01:42:47.368 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.62960.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43938.log.gz 2024-03-24T01:42:47.368 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43142.log 2024-03-24T01:42:47.368 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62960.log.gz 2024-03-24T01:42:47.374 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.63418.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63418.log.gz 2024-03-24T01:42:47.375 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51899.log 2024-03-24T01:42:47.375 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66997.log 2024-03-24T01:42:47.375 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43142.log.gz 2024-03-24T01:42:47.375 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44148.log 2024-03-24T01:42:47.376 INFO:teuthology.orchestra.run.smithi090.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51899.log.gz 2024-03-24T01:42:47.376 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.66997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44890.log 2024-03-24T01:42:47.376 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44148.log: 95.2% -- replaced with /var/log/ceph/ceph-client.admin.66997.log.gz 2024-03-24T01:42:47.377 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45786.log 2024-03-24T01:42:47.377 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44148.log.gz 2024-03-24T01:42:47.377 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50561.log 2024-03-24T01:42:47.378 INFO:teuthology.orchestra.run.smithi090.stderr: 89.4%/var/log/ceph/ceph-client.admin.45786.log: -- replaced with /var/log/ceph/ceph-client.admin.44890.log.gz 2024-03-24T01:42:47.378 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42466.log 2024-03-24T01:42:47.378 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45786.log.gz 2024-03-24T01:42:47.378 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.50561.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50561.log.gz 2024-03-24T01:42:47.379 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54171.log 2024-03-24T01:42:47.379 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61208.log 2024-03-24T01:42:47.379 INFO:teuthology.orchestra.run.smithi090.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42466.log.gz 2024-03-24T01:42:47.380 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.54171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53272.log 2024-03-24T01:42:47.380 INFO:teuthology.orchestra.run.smithi090.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54171.log.gz 2024-03-24T01:42:47.394 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.61208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54481.log 2024-03-24T01:42:47.394 INFO:teuthology.orchestra.run.smithi090.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.61208.log.gz 2024-03-24T01:42:47.394 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.53272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45450.log 2024-03-24T01:42:47.399 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53272.log.gz 2024-03-24T01:42:47.399 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.54481.log: gzip -5 --verbose -- 88.3% /var/log/ceph/ceph-client.admin.52852.log 2024-03-24T01:42:47.400 INFO:teuthology.orchestra.run.smithi090.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54481.log.gz 2024-03-24T01:42:47.400 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45450.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45450.log.gz 2024-03-24T01:42:47.400 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58915.log 2024-03-24T01:42:47.400 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55180.log 2024-03-24T01:42:47.401 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52852.log.gz/var/log/ceph/ceph-client.admin.58915.log: 2024-03-24T01:42:47.401 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51227.log 2024-03-24T01:42:47.401 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58915.log.gz 2024-03-24T01:42:47.402 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.55180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57207.log 2024-03-24T01:42:47.402 INFO:teuthology.orchestra.run.smithi090.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.55180.log.gz 2024-03-24T01:42:47.402 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48836.log 2024-03-24T01:42:47.403 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51227.log.gz 2024-03-24T01:42:47.403 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.57207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51689.log 2024-03-24T01:42:47.403 INFO:teuthology.orchestra.run.smithi090.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.57207.log.gz 2024-03-24T01:42:47.403 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45107.log 2024-03-24T01:42:47.403 INFO:teuthology.orchestra.run.smithi090.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.48836.log.gz 2024-03-24T01:42:47.403 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51689.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.30942.log 2024-03-24T01:42:47.403 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51689.log.gz 2024-03-24T01:42:47.403 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58999.log 2024-03-24T01:42:47.404 INFO:teuthology.orchestra.run.smithi090.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45107.log.gz 2024-03-24T01:42:47.404 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph.tmp-client.admin.30942.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.30942.log.gz 2024-03-24T01:42:47.404 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50849.log 2024-03-24T01:42:47.404 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.58999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48962.log 2024-03-24T01:42:47.404 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58999.log.gz 2024-03-24T01:42:47.404 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.50849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49309.log 2024-03-24T01:42:47.404 INFO:teuthology.orchestra.run.smithi090.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50849.log.gz 2024-03-24T01:42:47.404 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64141.log 2024-03-24T01:42:47.405 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48962.log.gz 2024-03-24T01:42:47.405 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49309.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49309.log.gz 2024-03-24T01:42:47.405 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63969.log 2024-03-24T01:42:47.409 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.64141.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64141.log.gz 2024-03-24T01:42:47.409 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48332.log 2024-03-24T01:42:47.410 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.63969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64225.log 2024-03-24T01:42:47.410 INFO:teuthology.orchestra.run.smithi090.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.63969.log.gz 2024-03-24T01:42:47.410 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65253.log 2024-03-24T01:42:47.410 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48332.log.gz 2024-03-24T01:42:47.410 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.64225.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64225.log.gz 2024-03-24T01:42:47.410 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61660.log 2024-03-24T01:42:47.410 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.65253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49661.log 2024-03-24T01:42:47.410 INFO:teuthology.orchestra.run.smithi090.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.65253.log.gz 2024-03-24T01:42:47.411 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.61660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66247.log 2024-03-24T01:42:47.411 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61660.log.gz 2024-03-24T01:42:47.411 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46444.log 2024-03-24T01:42:47.411 INFO:teuthology.orchestra.run.smithi090.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49661.log.gz 2024-03-24T01:42:47.411 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.66247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51395.log 2024-03-24T01:42:47.411 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46444.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.66247.log.gz 2024-03-24T01:42:47.411 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-24T01:42:47.411 INFO:teuthology.orchestra.run.smithi090.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46444.log.gz 2024-03-24T01:42:47.411 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45098.log 2024-03-24T01:42:47.412 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51395.log.gz 2024-03-24T01:42:47.412 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64309.log 2024-03-24T01:42:47.412 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46654.log 2024-03-24T01:42:47.412 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45098.log.gz 2024-03-24T01:42:47.412 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.64309.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.64309.log.gz 2024-03-24T01:42:47.412 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42888.log 2024-03-24T01:42:47.414 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48794.log 2024-03-24T01:42:47.414 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46654.log.gz 2024-03-24T01:42:47.414 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42888.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42888.log.gz 2024-03-24T01:42:47.414 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57375.log 2024-03-24T01:42:47.414 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52627.log 2024-03-24T01:42:47.414 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48794.log.gz 2024-03-24T01:42:47.414 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.57375.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57375.log.gz 2024-03-24T01:42:47.414 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45112.log 2024-03-24T01:42:47.416 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56090.log 2024-03-24T01:42:47.417 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52627.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52627.log.gz 2024-03-24T01:42:47.417 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45112.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45112.log.gz 2024-03-24T01:42:47.417 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63291.log 2024-03-24T01:42:47.417 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.56090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64729.log 2024-03-24T01:42:47.418 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56090.log.gz 2024-03-24T01:42:47.418 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.63291.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.63291.log.gz -5 2024-03-24T01:42:47.418 INFO:teuthology.orchestra.run.smithi090.stderr: --verbose -- /var/log/ceph/ceph-client.admin.60833.log 2024-03-24T01:42:47.419 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.64729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58747.log 2024-03-24T01:42:47.420 INFO:teuthology.orchestra.run.smithi090.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.64729.log.gz 2024-03-24T01:42:47.420 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.60833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49619.log 2024-03-24T01:42:47.420 INFO:teuthology.orchestra.run.smithi090.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.60833.log.gz 2024-03-24T01:42:47.421 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.58747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44806.log 2024-03-24T01:42:47.422 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58747.log.gz 2024-03-24T01:42:47.423 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49619.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49619.log.gz 2024-03-24T01:42:47.423 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49907.log 2024-03-24T01:42:47.424 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44434.log 2024-03-24T01:42:47.425 INFO:teuthology.orchestra.run.smithi090.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44806.log.gz 2024-03-24T01:42:47.425 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58115.log 2024-03-24T01:42:47.425 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49907.log.gz 2024-03-24T01:42:47.426 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65409.log 2024-03-24T01:42:47.426 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44434.log.gz 2024-03-24T01:42:47.427 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.58115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46066.log 2024-03-24T01:42:47.428 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58115.log.gz 2024-03-24T01:42:47.428 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.65409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63803.log 2024-03-24T01:42:47.429 INFO:teuthology.orchestra.run.smithi090.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.65409.log.gz 2024-03-24T01:42:47.430 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60980.log 2024-03-24T01:42:47.430 INFO:teuthology.orchestra.run.smithi090.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46066.log.gz 2024-03-24T01:42:47.431 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.63803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60131.log 2024-03-24T01:42:47.431 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63803.log.gz 2024-03-24T01:42:47.432 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.60980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45746.log 2024-03-24T01:42:47.432 INFO:teuthology.orchestra.run.smithi090.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.60980.log.gz 2024-03-24T01:42:47.434 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.60131.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60131.log.gz 2024-03-24T01:42:47.434 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51185.log 2024-03-24T01:42:47.434 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42846.log 2024-03-24T01:42:47.435 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51185.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45746.log.gz 2024-03-24T01:42:47.436 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44596.log 2024-03-24T01:42:47.436 INFO:teuthology.orchestra.run.smithi090.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51185.log.gz 2024-03-24T01:42:47.437 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42846.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42846.log.gz 2024-03-24T01:42:47.437 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58281.log 2024-03-24T01:42:47.438 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.44596.log: -- /var/log/ceph/ceph-client.admin.61519.log 2024-03-24T01:42:47.439 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.58281.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44596.log.gz 2024-03-24T01:42:47.440 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43016.log 2024-03-24T01:42:47.440 INFO:teuthology.orchestra.run.smithi090.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.58281.log.gz 2024-03-24T01:42:47.440 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.61519.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61519.log.gz 2024-03-24T01:42:47.441 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47886.log 2024-03-24T01:42:47.442 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52186.log 2024-03-24T01:42:47.443 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43016.log: /var/log/ceph/ceph-client.admin.47886.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43016.log.gz 2024-03-24T01:42:47.443 INFO:teuthology.orchestra.run.smithi090.stderr: 89.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.47886.log.gz -5 2024-03-24T01:42:47.443 INFO:teuthology.orchestra.run.smithi090.stderr: --verbose -- /var/log/ceph/ceph-client.admin.46486.log 2024-03-24T01:42:47.444 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49183.log 2024-03-24T01:42:47.444 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52186.log.gz 2024-03-24T01:42:47.445 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44392.log 2024-03-24T01:42:47.446 INFO:teuthology.orchestra.run.smithi090.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46486.log.gz 2024-03-24T01:42:47.447 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46276.log 2024-03-24T01:42:47.447 INFO:teuthology.orchestra.run.smithi090.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49183.log.gz 2024-03-24T01:42:47.447 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44392.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44392.log.gz 2024-03-24T01:42:47.448 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63231.log 2024-03-24T01:42:47.448 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46360.log 2024-03-24T01:42:47.449 INFO:teuthology.orchestra.run.smithi090.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46276.log.gz 2024-03-24T01:42:47.450 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.63231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.62045.log 2024-03-24T01:42:47.451 INFO:teuthology.orchestra.run.smithi090.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63231.log.gz 2024-03-24T01:42:47.451 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46360.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46360.log.gz 2024-03-24T01:42:47.451 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58579.log 2024-03-24T01:42:47.452 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.0.62045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54997.log 2024-03-24T01:42:47.452 INFO:teuthology.orchestra.run.smithi120.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-24T01:42:47.453 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.58579.log: gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58579.log.gz -5 2024-03-24T01:42:47.454 INFO:teuthology.orchestra.run.smithi090.stderr: --verbose -- /var/log/ceph/ceph-client.admin.62597.log 2024-03-24T01:42:47.455 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.54997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62639.log 2024-03-24T01:42:47.455 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54997.log.gz 2024-03-24T01:42:47.456 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.62597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45114.log 2024-03-24T01:42:47.456 INFO:teuthology.orchestra.run.smithi090.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.62597.log.gz 2024-03-24T01:42:47.457 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.62639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51647.log 2024-03-24T01:42:47.458 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62639.log.gz 2024-03-24T01:42:47.458 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45114.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45114.log.gz 2024-03-24T01:42:47.459 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-24T01:42:47.459 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64771.log 2024-03-24T01:42:47.460 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9%/var/log/ceph/ceph-mgr.y.log: -- replaced with /var/log/ceph/ceph-client.admin.51647.log.gz 2024-03-24T01:42:47.461 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43980.log 2024-03-24T01:42:47.462 INFO:teuthology.orchestra.run.smithi090.stderr: 93.5% -- replaced with /var/log/ceph/ceph-client.0.62045.log.gz 2024-03-24T01:42:47.463 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.64771.log: gzip -5 --verbose 91.2% -- /var/log/ceph/ceph-client.admin.48500.log -- replaced with /var/log/ceph/ceph-client.admin.64771.log.gz 2024-03-24T01:42:47.463 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-24T01:42:47.464 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43980.log: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61034.log 2024-03-24T01:42:47.464 INFO:teuthology.orchestra.run.smithi090.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43980.log.gz 2024-03-24T01:42:47.466 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64435.log 2024-03-24T01:42:47.466 INFO:teuthology.orchestra.run.smithi090.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48500.log.gz 2024-03-24T01:42:47.467 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.61034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49739.log 2024-03-24T01:42:47.468 INFO:teuthology.orchestra.run.smithi090.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.61034.log.gz 2024-03-24T01:42:47.468 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.64435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53020.log 2024-03-24T01:42:47.468 INFO:teuthology.orchestra.run.smithi090.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.64435.log.gz 2024-03-24T01:42:47.470 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42716.log 2024-03-24T01:42:47.470 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49739.log.gz 2024-03-24T01:42:47.471 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.53020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46024.log 2024-03-24T01:42:47.471 INFO:teuthology.orchestra.run.smithi090.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53020.log.gz 2024-03-24T01:42:47.473 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66001.log 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42716.log.gz 2024-03-24T01:42:47.473 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-24T01:42:47.473 INFO:teuthology.orchestra.run.smithi090.stderr: 94.6% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-24T01:42:47.474 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46024.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46024.log.gz 2024-03-24T01:42:47.474 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48584.log 2024-03-24T01:42:47.475 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.66001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45443.log 2024-03-24T01:42:47.476 INFO:teuthology.orchestra.run.smithi120.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-24T01:42:47.476 INFO:teuthology.orchestra.run.smithi090.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.66001.log.gz 2024-03-24T01:42:47.476 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48584.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48584.log.gz 2024-03-24T01:42:47.477 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58831.log 2024-03-24T01:42:47.477 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57688.log 2024-03-24T01:42:47.478 INFO:teuthology.orchestra.run.smithi090.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45443.log.gz 2024-03-24T01:42:47.478 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.58831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65475.log 2024-03-24T01:42:47.479 INFO:teuthology.orchestra.run.smithi090.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58831.log.gz 2024-03-24T01:42:47.480 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.57688.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57688.log.gz 2024-03-24T01:42:47.480 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44638.log 2024-03-24T01:42:47.481 INFO:teuthology.orchestra.run.smithi090.stderr:gzip/var/log/ceph/ceph-client.admin.65475.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.61702.log 2024-03-24T01:42:47.482 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44638.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.65475.log.gz 2024-03-24T01:42:47.482 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.44638.log.gz 2024-03-24T01:42:47.482 INFO:teuthology.orchestra.run.smithi090.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.60031.log 2024-03-24T01:42:47.483 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45431.log 2024-03-24T01:42:47.484 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.61702.log: /var/log/ceph/ceph-client.admin.60031.log: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57249.log 2024-03-24T01:42:47.485 INFO:teuthology.orchestra.run.smithi090.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.61702.log.gz 2024-03-24T01:42:47.485 INFO:teuthology.orchestra.run.smithi090.stderr: -- replaced with /var/log/ceph/ceph-client.admin.60031.log.gz 2024-03-24T01:42:47.487 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50135.log 2024-03-24T01:42:47.487 INFO:teuthology.orchestra.run.smithi090.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45431.log.gz 2024-03-24T01:42:47.488 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.57249.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57249.log.gz 2024-03-24T01:42:47.488 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64053.log 2024-03-24T01:42:47.489 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.50135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65304.log 2024-03-24T01:42:47.489 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.64053.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50135.log.gz 2024-03-24T01:42:47.490 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-24T01:42:47.490 INFO:teuthology.orchestra.run.smithi090.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.64053.log.gz 2024-03-24T01:42:47.492 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.65304.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.65304.log.gz 2024-03-24T01:42:47.492 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43100.log 2024-03-24T01:42:47.492 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67231.log 2024-03-24T01:42:47.493 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57417.log 2024-03-24T01:42:47.493 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43100.log.gz 2024-03-24T01:42:47.495 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.67231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50177.log 2024-03-24T01:42:47.496 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.57417.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.57417.log.gz 2024-03-24T01:42:47.496 INFO:teuthology.orchestra.run.smithi090.stderr: 95.7% -- replaced with /var/log/ceph/ceph-client.admin.67231.log.gz 2024-03-24T01:42:47.496 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63086.log 2024-03-24T01:42:47.498 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.50177.log: gzip -5 --verbose -- 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50177.log.gz 2024-03-24T01:42:47.498 INFO:teuthology.orchestra.run.smithi090.stderr: /var/log/ceph/ceph-client.admin.60887.log 2024-03-24T01:42:47.499 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.63086.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63086.log.gz 2024-03-24T01:42:47.500 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49225.log 2024-03-24T01:42:47.501 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.60887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50293.log 2024-03-24T01:42:47.501 INFO:teuthology.orchestra.run.smithi090.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.60887.log.gz 2024-03-24T01:42:47.502 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49225.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49225.log.gz 2024-03-24T01:42:47.502 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57646.log 2024-03-24T01:42:47.503 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.50293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58663.log 2024-03-24T01:42:47.504 INFO:teuthology.orchestra.run.smithi090.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50293.log.gz 2024-03-24T01:42:47.505 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.57646.log: gzip -5 --verbose -- 89.1% -- replaced with /var/log/ceph/ceph-client.admin.57646.log.gz 2024-03-24T01:42:47.505 INFO:teuthology.orchestra.run.smithi090.stderr: /var/log/ceph/ceph-client.admin.65367.log 2024-03-24T01:42:47.508 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.58663.log: 88.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45982.log 2024-03-24T01:42:47.508 INFO:teuthology.orchestra.run.smithi090.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58663.log.gz 2024-03-24T01:42:47.509 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.65367.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65367.log.gz 2024-03-24T01:42:47.509 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52894.log 2024-03-24T01:42:47.512 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54339.log 2024-03-24T01:42:47.512 INFO:teuthology.orchestra.run.smithi090.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45982.log.gz/var/log/ceph/ceph-client.admin.52894.log: 2024-03-24T01:42:47.513 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54297.log 2024-03-24T01:42:47.513 INFO:teuthology.orchestra.run.smithi090.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52894.log.gz 2024-03-24T01:42:47.513 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.54339.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54339.log.gz 2024-03-24T01:42:47.513 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51353.log 2024-03-24T01:42:47.517 INFO:teuthology.orchestra.run.smithi090.stderr:gzip/var/log/ceph/ceph-client.admin.54297.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.52764.log 2024-03-24T01:42:47.517 INFO:teuthology.orchestra.run.smithi090.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54297.log.gz 2024-03-24T01:42:47.517 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51353.log: gzip 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51353.log.gz 2024-03-24T01:42:47.517 INFO:teuthology.orchestra.run.smithi090.stderr: -5 --verbose -- /var/log/ceph/ceph.log 2024-03-24T01:42:47.518 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52764.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52764.log.gz 2024-03-24T01:42:47.518 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64645.log 2024-03-24T01:42:47.519 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53380.log 2024-03-24T01:42:47.519 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.64645.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64645.log.gz 2024-03-24T01:42:47.520 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5/var/log/ceph/ceph-client.admin.53380.log: --verbose -- 87.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-24T01:42:47.520 INFO:teuthology.orchestra.run.smithi090.stderr: /var/log/ceph/ceph-client.admin.56006.log 2024-03-24T01:42:47.521 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53380.log.gz 2024-03-24T01:42:47.521 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54213.log 2024-03-24T01:42:47.522 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.56006.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56006.log.gz 2024-03-24T01:42:47.522 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49267.log 2024-03-24T01:42:47.523 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.54213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64603.log 2024-03-24T01:42:47.525 INFO:teuthology.orchestra.run.smithi090.stderr: 89.0%/var/log/ceph/ceph-client.admin.49267.log: -- replaced with /var/log/ceph/ceph-client.admin.54213.log.gz 2024-03-24T01:42:47.525 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49267.log.gz 2024-03-24T01:42:47.525 INFO:teuthology.orchestra.run.smithi090.stderr: -- /var/log/ceph/ceph-client.admin.66017.log 2024-03-24T01:42:47.526 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.64603.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.64603.log.gz 2024-03-24T01:42:47.526 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52936.log 2024-03-24T01:42:47.528 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.66017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46570.log 2024-03-24T01:42:47.528 INFO:teuthology.orchestra.run.smithi090.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.66017.log.gz 2024-03-24T01:42:47.529 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57459.log 2024-03-24T01:42:47.529 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52936.log.gz 2024-03-24T01:42:47.530 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46570.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46570.log.gz 2024-03-24T01:42:47.530 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-24T01:42:47.531 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.57459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45113.log 2024-03-24T01:42:47.532 INFO:teuthology.orchestra.run.smithi090.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57459.log.gz/var/log/ceph/ceph-osd.2.log: 2024-03-24T01:42:47.533 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54397.log 2024-03-24T01:42:47.534 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45113.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45113.log.gz 2024-03-24T01:42:47.535 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53314.log 2024-03-24T01:42:47.535 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.54397.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54397.log.gz 2024-03-24T01:42:47.536 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51437.log 2024-03-24T01:42:47.539 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.53314.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.53314.log.gz 2024-03-24T01:42:47.539 INFO:teuthology.orchestra.run.smithi090.stderr:gzip/var/log/ceph/ceph-client.admin.51437.log: -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-03-24T01:42:47.540 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51437.log.gz 2024-03-24T01:42:47.540 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65923.log 2024-03-24T01:42:47.543 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-mds.c.log: /var/log/ceph/ceph-client.admin.65923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61175.log 2024-03-24T01:42:47.544 INFO:teuthology.orchestra.run.smithi090.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.65923.log.gz 2024-03-24T01:42:47.546 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48626.log 2024-03-24T01:42:47.548 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.61175.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.61175.log.gz 2024-03-24T01:42:47.552 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.48626.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48626.log.gz 2024-03-24T01:42:47.552 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60947.log 2024-03-24T01:42:47.554 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.60947.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.60947.log.gz 2024-03-24T01:42:47.554 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49535.log 2024-03-24T01:42:47.557 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49535.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49535.log.gz 2024-03-24T01:42:47.558 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55050.log 2024-03-24T01:42:47.560 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.55050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44308.log 2024-03-24T01:42:47.562 INFO:teuthology.orchestra.run.smithi090.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55050.log.gz 2024-03-24T01:42:47.563 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58238.log 2024-03-24T01:42:47.564 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44308.log.gz 2024-03-24T01:42:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54739.log 2024-03-24T01:42:47.567 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.58238.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58238.log.gz 2024-03-24T01:42:47.569 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66628.log 2024-03-24T01:42:47.574 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.66628.log: /var/log/ceph/ceph-client.admin.54739.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.54739.log.gz 2024-03-24T01:42:47.574 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61094.log 2024-03-24T01:42:47.575 INFO:teuthology.orchestra.run.smithi090.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66628.log.gz 2024-03-24T01:42:47.577 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43058.log 2024-03-24T01:42:47.580 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43058.log: /var/log/ceph/ceph-client.admin.61094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52481.log 2024-03-24T01:42:47.583 INFO:teuthology.orchestra.run.smithi090.stderr: 91.0% 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61094.log.gz 2024-03-24T01:42:47.583 INFO:teuthology.orchestra.run.smithi090.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43058.log.gz 2024-03-24T01:42:47.584 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63188.log 2024-03-24T01:42:47.588 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.63188.log: /var/log/ceph/ceph-client.admin.52481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44512.log 2024-03-24T01:42:47.590 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52481.log.gz 2024-03-24T01:42:47.590 INFO:teuthology.orchestra.run.smithi090.stderr: -- replaced with /var/log/ceph/ceph-client.admin.63188.log.gz 2024-03-24T01:42:47.592 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54523.log 2024-03-24T01:42:47.595 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.54523.log: /var/log/ceph/ceph-client.admin.44512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66796.log 2024-03-24T01:42:47.597 INFO:teuthology.orchestra.run.smithi090.stderr: 88.1% 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44512.log.gz 2024-03-24T01:42:47.598 INFO:teuthology.orchestra.run.smithi090.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54523.log.gz 2024-03-24T01:42:47.603 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52063.log 2024-03-24T01:42:47.603 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.52063.log: /var/log/ceph/ceph-client.admin.66796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.59444.log 2024-03-24T01:42:47.605 INFO:teuthology.orchestra.run.smithi090.stderr: 87.8% 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66796.log.gz 2024-03-24T01:42:47.605 INFO:teuthology.orchestra.run.smithi090.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52063.log.gz 2024-03-24T01:42:47.606 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66469.log 2024-03-24T01:42:47.610 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.66469.log: /var/log/ceph/ceph-client.0.59444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43268.log 2024-03-24T01:42:47.613 INFO:teuthology.orchestra.run.smithi090.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.66469.log.gz 2024-03-24T01:42:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43268.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.43268.log.gz 2024-03-24T01:42:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57543.log 2024-03-24T01:42:47.618 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46234.log 2024-03-24T01:42:47.620 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.57543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49046.log 2024-03-24T01:42:47.620 INFO:teuthology.orchestra.run.smithi090.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57543.log.gz 2024-03-24T01:42:47.621 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.46234.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46234.log.gz 2024-03-24T01:42:47.624 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61235.log 2024-03-24T01:42:47.625 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.49046.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49046.log.gz 2024-03-24T01:42:47.628 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.61235.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.61235.log.gz 2024-03-24T01:42:47.629 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53146.log 2024-03-24T01:42:47.630 INFO:teuthology.orchestra.run.smithi090.stderr: 93.5% -- replaced with /var/log/ceph/ceph-client.0.59444.log.gz 2024-03-24T01:42:47.631 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60713.log 2024-03-24T01:42:47.634 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.53146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66733.log 2024-03-24T01:42:47.635 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.60713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44206.log 2024-03-24T01:42:47.635 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53146.log.gz 2024-03-24T01:42:47.636 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.66733.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.60713.log.gz 2024-03-24T01:42:47.636 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66733.log.gz 2024-03-24T01:42:47.637 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.47228.log 2024-03-24T01:42:47.641 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44206.log: /var/log/ceph/ceph-client.0.47228.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44206.log.gz 2024-03-24T01:42:47.642 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64519.log 2024-03-24T01:42:47.644 INFO:teuthology.orchestra.run.smithi090.stderr: 91.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51269.log 2024-03-24T01:42:47.645 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.64519.log: -- replaced with /var/log/ceph/ceph-client.0.47228.log.gz 2024-03-24T01:42:47.646 INFO:teuthology.orchestra.run.smithi090.stderr:gzip 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64519.log.gz 2024-03-24T01:42:47.646 INFO:teuthology.orchestra.run.smithi090.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.66270.log 2024-03-24T01:42:47.650 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.51269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45100.log 2024-03-24T01:42:47.650 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.66270.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.66270.log.gz 2024-03-24T01:42:47.650 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51059.log 2024-03-24T01:42:47.651 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51269.log.gz 2024-03-24T01:42:47.654 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.45100.log: gzip/var/log/ceph/ceph-client.admin.51059.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45100.log.gz 2024-03-24T01:42:47.655 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51059.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.63502.log 2024-03-24T01:42:47.655 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-24T01:42:47.656 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63460.log 2024-03-24T01:42:47.658 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.63502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44106.log 2024-03-24T01:42:47.659 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63502.log.gz 2024-03-24T01:42:47.660 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66165.log 2024-03-24T01:42:47.664 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.63460.log: /var/log/ceph/ceph-client.admin.44106.log: 88.1% 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44106.log.gz -- replaced with /var/log/ceph/ceph-client.admin.63460.log.gz 2024-03-24T01:42:47.664 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-24T01:42:47.666 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.66165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49823.log 2024-03-24T01:42:47.669 INFO:teuthology.orchestra.run.smithi090.stderr: 88.1%/var/log/ceph/ceph-client.admin.49823.log: -- replaced with /var/log/ceph/ceph-client.admin.66165.log.gz 2024-03-24T01:42:47.669 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44350.log 2024-03-24T01:42:47.671 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54255.log 2024-03-24T01:42:47.671 INFO:teuthology.orchestra.run.smithi090.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49823.log.gz 2024-03-24T01:42:47.672 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.44350.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44350.log.gz 2024-03-24T01:42:47.674 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.54255.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54255.log.gz 2024-03-24T01:42:47.675 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58789.log 2024-03-24T01:42:47.677 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.58789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42930.log 2024-03-24T01:42:47.678 INFO:teuthology.orchestra.run.smithi090.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58789.log.gz 2024-03-24T01:42:47.680 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52722.log 2024-03-24T01:42:47.684 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42930.log: /var/log/ceph/ceph-client.admin.52722.log: 89.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50975.log 2024-03-24T01:42:47.685 INFO:teuthology.orchestra.run.smithi090.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42930.log.gz 2024-03-24T01:42:47.686 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52722.log.gz 2024-03-24T01:42:47.689 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.50975.log: /var/log/ceph/ceph-client.admin.57603.log 2024-03-24T01:42:47.692 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.57603.log: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56952.log 2024-03-24T01:42:47.693 INFO:teuthology.orchestra.run.smithi090.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50975.log.gz 2024-03-24T01:42:47.694 INFO:teuthology.orchestra.run.smithi090.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57603.log.gz 2024-03-24T01:42:47.696 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58365.log 2024-03-24T01:42:47.699 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.58365.log: /var/log/ceph/ceph-client.admin.56952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61007.log 2024-03-24T01:42:47.702 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56952.log.gz 2024-03-24T01:42:47.702 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.61007.log: -- replaced with /var/log/ceph/ceph-client.admin.58365.log.gz 2024-03-24T01:42:47.703 INFO:teuthology.orchestra.run.smithi090.stderr: 91.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63884.log 2024-03-24T01:42:47.704 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61067.log 2024-03-24T01:42:47.706 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.63884.log: -- replaced with /var/log/ceph/ceph-client.admin.61007.log.gz 2024-03-24T01:42:47.707 INFO:teuthology.orchestra.run.smithi090.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.63884.log.gz 2024-03-24T01:42:47.708 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58323.log 2024-03-24T01:42:47.710 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58957.log 2024-03-24T01:42:47.712 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.61067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60686.log 2024-03-24T01:42:47.713 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.58323.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58323.log.gz 2024-03-24T01:42:47.716 INFO:teuthology.orchestra.run.smithi090.stderr: 91.2%/var/log/ceph/ceph-client.admin.60686.log: -- replaced with /var/log/ceph/ceph-client.admin.61067.log.gz 2024-03-24T01:42:47.719 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.58957.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.60686.log.gz 2024-03-24T01:42:47.719 INFO:teuthology.orchestra.run.smithi090.stderr: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37315.log 2024-03-24T01:42:47.719 INFO:teuthology.orchestra.run.smithi090.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58957.log.gz 2024-03-24T01:42:47.721 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64561.log 2024-03-24T01:42:47.722 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37315.log.gz 2024-03-24T01:42:47.726 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.64561.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.64561.log.gz 2024-03-24T01:42:47.726 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61572.log 2024-03-24T01:42:47.727 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.61572.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61572.log.gz 2024-03-24T01:42:47.728 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-03-24T01:42:47.732 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58621.log 2024-03-24T01:42:47.737 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58196.log 2024-03-24T01:42:47.738 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.58621.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58621.log.gz 2024-03-24T01:42:47.741 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60215.log 2024-03-24T01:42:47.742 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.58196.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.58196.log.gz 2024-03-24T01:42:47.743 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55138.log 2024-03-24T01:42:47.744 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.60215.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60215.log.gz 2024-03-24T01:42:47.747 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60773.log 2024-03-24T01:42:47.748 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.55138.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55138.log.gz 2024-03-24T01:42:47.750 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58537.log 2024-03-24T01:42:47.753 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58409.log 2024-03-24T01:42:47.754 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.60773.log: /var/log/ceph/ceph-client.admin.58537.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.58537.log.gz 2024-03-24T01:42:47.754 INFO:teuthology.orchestra.run.smithi090.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.60773.log.gz 2024-03-24T01:42:47.760 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.58409.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.58409.log.gz 2024-03-24T01:42:47.775 INFO:teuthology.orchestra.run.smithi090.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-03-24T01:42:47.789 INFO:teuthology.orchestra.run.smithi090.stderr: 93.5% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-03-24T01:42:47.889 INFO:teuthology.orchestra.run.smithi090.stderr: 90.3% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-24T01:42:47.998 INFO:teuthology.orchestra.run.smithi090.stderr: 93.0% -- replaced with /var/log/ceph/ceph-client.0.65114.log.gz 2024-03-24T01:42:48.338 INFO:teuthology.orchestra.run.smithi120.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-24T01:42:48.519 INFO:teuthology.orchestra.run.smithi120.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-24T01:42:48.627 INFO:teuthology.orchestra.run.smithi120.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-24T01:42:48.712 INFO:teuthology.orchestra.run.smithi090.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-24T01:42:48.773 INFO:teuthology.orchestra.run.smithi120.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-24T01:42:48.888 INFO:teuthology.orchestra.run.smithi090.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-24T01:42:48.905 INFO:teuthology.orchestra.run.smithi090.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-24T01:42:49.123 INFO:teuthology.orchestra.run.smithi120.stderr: 94.2% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-03-24T01:42:49.127 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-24T01:42:49.127 INFO:teuthology.orchestra.run.smithi120.stderr:real 0m2.064s 2024-03-24T01:42:49.127 INFO:teuthology.orchestra.run.smithi120.stderr:user 0m8.058s 2024-03-24T01:42:49.128 INFO:teuthology.orchestra.run.smithi120.stderr:sys 0m0.734s 2024-03-24T01:42:49.144 INFO:teuthology.orchestra.run.smithi090.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-24T01:42:49.148 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-24T01:42:49.148 INFO:teuthology.orchestra.run.smithi090.stderr:real 0m2.097s 2024-03-24T01:42:49.148 INFO:teuthology.orchestra.run.smithi090.stderr:user 0m7.278s 2024-03-24T01:42:49.149 INFO:teuthology.orchestra.run.smithi090.stderr:sys 0m1.766s 2024-03-24T01:42:49.150 INFO:tasks.ceph:Archiving logs... 2024-03-24T01:42:49.150 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616453/remote/smithi090/log 2024-03-24T01:42:49.151 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T01:42:50.560 DEBUG:teuthology.misc:Transferring archived files from smithi120:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616453/remote/smithi120/log 2024-03-24T01:42:50.561 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T01:42:51.612 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-03-24T01:42:51.624 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-24T01:42:51.633 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-03-24T01:42:51.638 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-24T01:42:51.638 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T01:42:51.649 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T01:42:51.795 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-24T01:42:51.805 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-24T01:42:51.805 DEBUG:teuthology.orchestra.run.smithi090:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T01:42:51.813 DEBUG:teuthology.orchestra.run.smithi120:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T01:42:51.858 INFO:teuthology.orchestra.run.smithi090.stderr:bash: line 1: ntpq: command not found 2024-03-24T01:42:51.866 INFO:teuthology.orchestra.run.smithi120.stderr:bash: line 1: ntpq: command not found 2024-03-24T01:42:51.868 INFO:teuthology.orchestra.run.smithi090.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T01:42:51.868 INFO:teuthology.orchestra.run.smithi090.stdout:=============================================================================== 2024-03-24T01:42:51.868 INFO:teuthology.orchestra.run.smithi090.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 50 +1298us[+1298us] +/- 92ms 2024-03-24T01:42:51.868 INFO:teuthology.orchestra.run.smithi090.stdout:^* hv02.front.sepia.ceph.com 3 6 377 52 -1304us[-1292us] +/- 60ms 2024-03-24T01:42:51.868 INFO:teuthology.orchestra.run.smithi090.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 +809us[ +809us] +/- 68ms 2024-03-24T01:42:51.868 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T01:42:51.875 INFO:teuthology.orchestra.run.smithi120.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T01:42:51.876 INFO:teuthology.orchestra.run.smithi120.stdout:=============================================================================== 2024-03-24T01:42:51.876 INFO:teuthology.orchestra.run.smithi120.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 115 +1301us[+1258us] +/- 113ms 2024-03-24T01:42:51.876 INFO:teuthology.orchestra.run.smithi120.stdout:^* hv02.front.sepia.ceph.com 3 7 377 48 -1362us[-1405us] +/- 60ms 2024-03-24T01:42:51.876 INFO:teuthology.orchestra.run.smithi120.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 115 +828us[ +785us] +/- 67ms 2024-03-24T01:42:51.876 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T01:42:51.877 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-24T01:42:51.887 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-24T01:42:51.887 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-24T01:42:51.896 DEBUG:teuthology.orchestra.run.smithi090:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T01:42:52.141 DEBUG:teuthology.orchestra.run.smithi120:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T01:42:52.345 DEBUG:teuthology.orchestra.run.smithi090:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T01:42:52.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T01:42:52.452 DEBUG:teuthology.orchestra.run.smithi120:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T01:42:52.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T01:42:52.554 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-24T01:42:52.563 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-24T01:42:52.572 INFO:teuthology.task.internal:Duration was 1785.609401 seconds 2024-03-24T01:42:52.572 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-24T01:42:52.580 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-24T01:42:52.580 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T01:42:52.587 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T01:42:52.746 INFO:teuthology.orchestra.run.smithi090.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T01:42:52.752 INFO:teuthology.orchestra.run.smithi120.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T01:42:53.131 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-24T01:42:53.131 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi090.front.sepia.ceph.com 2024-03-24T01:42:53.132 DEBUG:teuthology.orchestra.run.smithi090:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T01:42:53.195 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi120.front.sepia.ceph.com 2024-03-24T01:42:53.196 DEBUG:teuthology.orchestra.run.smithi120:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T01:42:53.268 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-24T01:42:53.268 DEBUG:teuthology.orchestra.run.smithi090:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T01:42:53.275 DEBUG:teuthology.orchestra.run.smithi120:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T01:42:53.368 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-24T01:42:53.368 DEBUG:teuthology.orchestra.run.smithi090:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T01:42:53.373 DEBUG:teuthology.orchestra.run.smithi120:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T01:42:53.565 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-24T01:42:53.575 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-24T01:42:53.575 DEBUG:teuthology.orchestra.run.smithi090:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T01:42:53.669 DEBUG:teuthology.orchestra.run.smithi120:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T01:42:53.753 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-24T01:42:53.763 DEBUG:teuthology.orchestra.run.smithi090:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T01:42:53.771 DEBUG:teuthology.orchestra.run.smithi120:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T01:42:53.874 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = core 2024-03-24T01:42:53.877 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern = core 2024-03-24T01:42:53.934 DEBUG:teuthology.orchestra.run.smithi090:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T01:42:53.982 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi090.front.sepia.ceph.com, flagging run as failed 2024-03-24T01:42:53.983 DEBUG:teuthology.orchestra.run.smithi120:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T01:42:54.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T01:42:54.036 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-24T01:42:54.046 INFO:teuthology.task.internal:Transferring archived files... 2024-03-24T01:42:54.046 DEBUG:teuthology.misc:Transferring archived files from smithi090:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616453/remote/smithi090 2024-03-24T01:42:54.047 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T01:43:02.436 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2024-03-24T01:43:02.503 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616453/remote/smithi090/coredump/1711244007.63804.core.gz: gzip compressed data, was "1711244007.63804.core", last modified: Sun Mar 24 01:42:54 2024, max compression, original size modulo 2^32 269058048 2024-03-24T01:43:02.503 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ... 2024-03-24T01:43:03.604 INFO:teuthology.task.internal: core looks like: /tmp/tmpnbbl378n: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'ceph-mds -f --cluster ceph -i a', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/bin/ceph-mds', platform: 'x86_64' 2024-03-24T01:43:03.632 INFO:teuthology.task.internal: dump_program: ceph-mds -f --cluster ceph -i a 2024-03-24T01:43:03.632 DEBUG:teuthology.orchestra.run.smithi090:> which 'ceph-mds -f --cluster ceph -i a' 2024-03-24T01:43:03.690 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/which: invalid option -- 'f' 2024-03-24T01:43:03.691 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/which: unrecognized option '--cluster' 2024-03-24T01:43:03.691 INFO:teuthology.orchestra.run.smithi090.stderr:/usr/bin/which: no a in (/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin) 2024-03-24T01:43:03.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T01:43:03.692 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph-mds 2024-03-24T01:43:03.692 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph 2024-03-24T01:43:03.693 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 345, in fetch_binaries_for_coredumps remote_path = remote.sh(['which', dump_program]).rstrip() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 1: "which 'ceph-mds -f --cluster ceph -i a'" 2024-03-24T01:43:03.711 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-24T01:43:03.722 INFO:teuthology.task.internal:Not uploading archives. 2024-03-24T01:43:03.722 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-24T01:43:03.772 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-24T01:43:03.772 DEBUG:teuthology.orchestra.run.smithi090:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T01:43:03.785 DEBUG:teuthology.orchestra.run.smithi120:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T01:43:03.832 INFO:teuthology.orchestra.run.smithi090.stdout: 393249 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 24 01:42 /home/ubuntu/cephtest 2024-03-24T01:43:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: 393250 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 24 01:42 /home/ubuntu/cephtest/archive 2024-03-24T01:43:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: 393252 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 01:42 /home/ubuntu/cephtest/archive/syslog 2024-03-24T01:43:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: 393412 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Mar 24 01:13 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-03-24T01:43:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: 393254 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Mar 24 01:13 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-03-24T01:43:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: 393253 204 -rw-r--r-- 1 ubuntu ubuntu 206220 Mar 24 01:42 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T01:43:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: 393401 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 01:42 /home/ubuntu/cephtest/archive/audit 2024-03-24T01:43:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: 393413 196 -rw------- 1 ubuntu ubuntu 198613 Mar 24 01:42 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-03-24T01:43:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: 393251 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 01:33 /home/ubuntu/cephtest/archive/coredump 2024-03-24T01:43:03.834 INFO:teuthology.orchestra.run.smithi090.stdout: 406944 34264 -rw------- 1 root root 269058048 Mar 24 01:33 /home/ubuntu/cephtest/archive/coredump/1711244007.63804.core 2024-03-24T01:43:03.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T01:43:03.835 INFO:teuthology.orchestra.run.smithi090.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-03-24T01:43:03.836 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-03-24T01:43:03.850 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-24T01:43:03.862 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-24T01:43:03.873 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi090 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-03-24T01:43:03.873 INFO:teuthology.nuke:Checking targets against current locks 2024-03-24T01:43:03.881 INFO:teuthology.orchestra.run.smithi120.stdout: 393229 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 24 01:42 /home/ubuntu/cephtest 2024-03-24T01:43:03.881 INFO:teuthology.orchestra.run.smithi120.stdout: 393230 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 24 01:42 /home/ubuntu/cephtest/archive 2024-03-24T01:43:03.881 INFO:teuthology.orchestra.run.smithi120.stdout: 393232 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 01:42 /home/ubuntu/cephtest/archive/syslog 2024-03-24T01:43:03.881 INFO:teuthology.orchestra.run.smithi120.stdout: 393402 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Mar 24 01:13 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-03-24T01:43:03.882 INFO:teuthology.orchestra.run.smithi120.stdout: 393242 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Mar 24 01:13 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-03-24T01:43:03.882 INFO:teuthology.orchestra.run.smithi120.stdout: 393241 188 -rw-r--r-- 1 ubuntu ubuntu 190235 Mar 24 01:42 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T01:43:03.882 INFO:teuthology.orchestra.run.smithi120.stdout: 524291 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 01:42 /home/ubuntu/cephtest/archive/audit 2024-03-24T01:43:03.882 INFO:teuthology.orchestra.run.smithi120.stdout: 524573 136 -rw------- 1 ubuntu ubuntu 135860 Mar 24 01:42 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-03-24T01:43:03.883 INFO:teuthology.orchestra.run.smithi120.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-03-24T01:43:03.902 DEBUG:teuthology.nuke:shortname: smithi090 2024-03-24T01:43:03.902 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T01:43:03.920 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616453', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 01:02:38.676443', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDPxTpKXePPQNHpR4U7WKV34yOJfd7d/mNOkZ153u5OYNUU/BVD7SyJJJSic+YiPf10ceGPv0TJhVYnK5d2enao='} 2024-03-24T01:43:03.925 DEBUG:teuthology.nuke:shortname: smithi120 2024-03-24T01:43:03.926 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T01:43:03.942 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi120.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616453', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 01:02:38.675107', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIsD5/kDR1BWRDWfmlrWw9avR8IC90oP1y5hOVOfAiJXGeLToZTQvDj9Ol/SdQ0iTE0wDvajJKkjTfhdzedsbTY='} 2024-03-24T01:43:03.964 INFO:teuthology.orchestra.console.smithi090:Power off 2024-03-24T01:43:03.965 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T01:43:03.989 DEBUG:teuthology.orchestra.console.smithi090:power off output: Chassis Power Control: Down/Off 2024-03-24T01:43:03.989 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T01:43:04.005 INFO:teuthology.orchestra.console.smithi120:Power off 2024-03-24T01:43:04.006 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T01:43:04.017 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on 2024-03-24T01:43:04.033 DEBUG:teuthology.orchestra.console.smithi120:power off output: Chassis Power Control: Down/Off 2024-03-24T01:43:04.034 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T01:43:04.056 DEBUG:teuthology.orchestra.console.smithi120:check power output: Chassis Power is on 2024-03-24T01:43:08.018 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T01:43:08.057 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T01:43:08.130 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on 2024-03-24T01:43:08.167 DEBUG:teuthology.orchestra.console.smithi120:check power output: Chassis Power is on 2024-03-24T01:43:12.134 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T01:43:12.168 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T01:43:12.246 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is off 2024-03-24T01:43:12.280 DEBUG:teuthology.orchestra.console.smithi120:check power output: Chassis Power is off 2024-03-24T01:43:12.347 INFO:teuthology.orchestra.console.smithi090:Power off completed 2024-03-24T01:43:12.381 INFO:teuthology.orchestra.console.smithi120:Power off completed 2024-03-24T01:43:12.486 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/journal-repair} duration: 1785.6094012260437 failure_reason: 'Test failure: test_reset (tasks.cephfs.test_journal_repair.TestJournalRepair), test_reset (tasks.cephfs.test_journal_repair.TestJournalRepair)' flavor: default mon.a-kernel-sha1: 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 mon.b-kernel-sha1: 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 owner: mchangir sentry_event: https://sentry.ceph.com/organizations/ceph/?query=5a14705d930742f8a51d83f32eb23f2b status: fail success: false 2024-03-24T01:43:12.487 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T01:43:12.572 INFO:teuthology.run:FAIL